COVID19 Update: 40 million API requests, grant, stability

It’s been a short while since the last update on the API, but the silence has not meant nothing is happening - if anything, it’s the opposite. API usage has skyrocketed and continues to climb.

Stability

The chief focal area has been stability. Around the second week of April we released a few new features (all status queries, from and to dates, world totals) as well as worked on performance improvements. The focus since then has been on service stability.

The first area of service stability is the application infrastructure. Due to the increase in load from usage we’ve had to adapt as we go along. This has brought us to our latest iteration, which is having both MySQL and Redis be a part of the managed infrastructure. Redis was on the same box as the application, and due to the heavy caching we are doing, Redis was causing high memory usage which would then cause issues with the application. The application itself is super lightweight and can handle enormous load. Since moving to a managed Redis service, the application has been serving requests with zero issues.

The second area of service stability is the data. We continue to struggle with data integrity. I would say the main cause of this is the way we handle data across three concerns: normal/standard data, group/aggregated data, US data. Due to the three different types of data, it becomes complex to ensure that all routes are handling the different types of data in the correct way. This might be due for a refactor, where we pull the different types of data in to different tables and explicitly treat them differently, but we will see if the issues continue.

There is always a line between adding features and maintaining the system. Currently we have a few contributors getting involved which is great and hopefully will allow us to scale well.

Emergent Ventures

This is a bit late to formally announce but Emergent Ventures awarded the COVID19 API a grant to assist with the building and running of the service. The grant money has been hugely helpful in getting the service to where it is today, and will continue to help scale and run the service. A personal thanks to Tyler Cowen for seeing the value of the API so early on.

We also received a small grant from Zak Slayback at the 1517Fund, which has also helped us cover some costs.

GoRemoteFest

There was at a recent event online called GoRemoteFest where I did a talk on building the COVID19 API. You watch my session below:

Moving Forward

The API crossed the 40 million API request mark yesterday. With the application being used in more and more services, websites and applications - including by IBM in their Call For COVID tutorials - the focus is on ensuring that it is always available and accurate. We’d love to include more data (currently looking at testing data from Our World In Data) but only once we are fully stable.

Get involved: we have a growing Slack community (almost 400 members now!) so join if you want to showcase your work or keep updated.