Shadow API: Launching An API Before Someone Else Does

In a perfect world, every government agency, non-profit organization, and company would have an API, and the content and data available via their website, mobile applications, and other systems, would be accessible in an easy, machine readable way. This perfect world does not exist. Even though many government agencies, non-profit organizations, and companies are getting hip to the world of APIs, there is still massive amounts of work to be done, and I'm afraid if we want to see APIs across our government, we are going to have to do much of it ourselves.

I was listening to an NPR story on the release of files detailing Nebraska's homesteading history, a project between the National Archives and National Park Service, today. I wish I could be at all federal agencies, in all meetings, advocating for an API first approach to these projects--I can’t. So, while we can celebrate the fact that the records around Nebraska’s history have been digitized, and the website is very nice, and usable for humans, THERE IS NO API!

This is when the private sector has to step in and help our federal government realize the importance and potential of an API--we need a shadow API for the homestead records database. While the website is great for human consumption, we also need a simple, machine readable API that other applications can put to use. It would be nice if the National Archives and National Park Service had made the API a priority, but I think this is where the private sector needs to step in, and make it happen.

I have direct orders from my super secret White House contact(s) to challenge you, Joe and Jane citizen to see who can launch an API faster, around content or data that is already public (no hacking)—the federal government, or you the average tech savvy citizen. I won’t give up my sources, but I’m dead serious when I say the White House asked me to put the word out on this.

Data sources like Nebraskas homestead records, are well structured web searches, and not that difficult to scrape. A savvy programmer could easily write a script to do it, and hand roll a simple API for others access the records. I also think a tech savvy non-programmer, could make happen by employing a modern API deployment tools like ScraperWiki,,, or Kimono Labs. I don’t imagine it taking more than a day or two, to scrape the data, and deploy a simple API interface from the results.

This isn’t a new concept, something API devlopers have been doing a while. The first Instagram API was a shadow API, deployed by independent 3rd party developer, who reverse engineered the mobile application. Kimono Labs also just launched an unofficial World Cup API, allowing for some pretty cool applications to be built on top of the shadow API. Shadow APIs have been being deployed for years, we just need to make them more of a mainstream, acceptable way to make resources available via APIs, real fast!

If you know of a shadow API you would like to see built, or you are available to deploy shadow APIs, let me know. There are some things I’d like to see in quality shadow API deployments around government data (maybe another story), and I can help you get the word out when the time is right. In my opinion, if content or data is already available via a government website, or as a full data dowload, there should be an API for it. Period.

What do you think? Who is going to build my Nebraska homestead record API, before I can harass the National Archives and National Park Service to do it?

Disclosure: I'm not going to disclose my confidential sources at the White House, don't even ask or render me to a black ops site.