You hear a lot about doing APIs at scale in our space. Many folks dismiss web APIs because they feel they won’t scale, and aren’t performing at the scale they envision. The majority of these discussions focus on how do you scale large operations of Twitter, Facebook, or Google scope. A single organization operating API infrastructure at scale, distributed across many geographical regions, supporting millions of users. There are plenty of discussions going on regarding the technology, business, and politics of doing APIs at this scale. I find myself thinking in similar ways, but more federated version of this, where the latest technology might not always be the right answer.
My Human Services Data API (HDSA) work is the best example I have of this. Where I’m having to keep the technology, and API definition bar as low as possible to onboard as many people as I possibly can, but then eventually, be able to aggregate large amounts of data across many federated instance. I have 3,144 counties, and 19,354 cities to consider. They should all be speaking a common schema when it comes to the sharing of human services data. Something that is easier said, than done. When you get on the ground you realize many of them are stuck in 1990s, or early 2000s edition of the web, and just do not have the resources needed to move things forward. They can’t afford the latest SaaS service, and they can’t drop the ball, or thousands, or millions of people will suffer–the stakes are high.
When I go into large companies, who have a large teams, and significant number of resources, the conversation around scale is much different. Sure, there is distributed scale. Sure, there is volume scale. However, most times the distribution and volume exists within a single company or organization. A single command and control structure. However, I’m talking about federated distribution and volume, with no single command and control structure. I’m facing not just how you deliver technology across all these nodes, but how do you train, consider extremely short budgets, and other aspects of ensuring things get done consistently, reliably, without disruption. Cities aren’t the only example of this in our world. I’m also seeing the same across state and federal agencies, as well as k-12, and higher educational institutions. Again, where the technological bar is low, but the stakes are high.
This real world is just a different game than tech culture likes to admit. I can’t always take what I learn in the tech sector and immediately apply on the ground in the mainstream worlds. Some of it applies, but honestly I’ve had to throw a lot of seemingly sensible decision out lately, opting for much simpler, web based solutions, that I’m confident staff can be trained on, and can realistically be implemented in existing IT environments. If we keep moving fast and breaking things, and some point we are going to have to stop, pick up some of the pieces, and help take care of the folks we’ve left behind.
