I always find an API providers business model to be very telling about the company’s overall strategy when it comes to APIs. I’m currently navigating the difference between two big API providers, trying to balance my needs spread across very different approaches to offering up API resources. I’m working to evolve and refine my API search algorithms and I find myself having to do a significant amount of work due to the differences between GitHub and Microsoft Search. Ironically, they are both owned by the same company, but we all know their business models are seeking alignment as we speak, and I suspect my challenges with GitHub API is probably a result of this alignment.
The challenges with integrating with GitHub and Microsoft APIs are pretty straightforward, and something I find myself battling regularly when integrating with many different APIs. My use of each platform is pretty simple. I am looking for APIs. The solutions are pretty simple, and robust. I can search for code using the GitHub Search API, and I can search for websites using the Bing Search API. Both produce different types of results, but what both produce is of value to me. The challenge comes in when I can pay for each API call with Bing, and I do not have that same option with GitHub. I am also noticing much tighter restriction on how many calls I can make to the GitHub APIs. With Bing I can burst, depending on how much money I want to spend, but with GitHub I have no relief value—I can only make X calls a minute, per IP, per user.
This is a common disconnect in the world of APIs, and something I’ve written a lot about. GitHub (Microsoft) has a more “elevated” business model, with the APIs being just an enabler of that business model. Where Bing (Microsoft) is going with a much more straightforward API monetization strategy—pay for what you use. In this comparison my needs are pretty straightforward—-both providers have data I want, and I’m willing to pay for it. However, there is an additional challenge. I’m also using GitHub to manage the underlying application for my project. Meaning after I pull search results from GitHub and Bing, and run them through my super top secret, magical, and proprietary refinement algorithm, I publish the refined results to a GitHub repository, and manage the application in real time using Git, and GitHub APIs—which counts against my API usage.
I used to manage all my static sites and applications 100% on GitHub, using the APIs to orchestrate the data behind each Jekyll-driven site. For the last five years I’ve run API Evangelist, and waves of simple data-driven static applications on GitHub like this. It has been a good ride. A free ride. One I fear is coming to a close. I can no longer deploy static data-driven Jekyll apps on the platform, and confidently manage using the GitHub API anymore. It is something I do not expect to continue getting for free. I’d be happy to pay for my account on a per organization, per repo, and per API call basis. In the end, I’ll probably begin just relying on Git for bulk builds of each application I run on GitHub, and eventually begin migrating them to my own servers, running Jekyll on my own, and custom developing an API for managing the more granular changes across hundreds of micro applications that run on Jekyll using YAML data. It would be nice for GitHub to notice this type of application development as part of their business model, but I’m guessing it isn’t mainstream enough for folks to adopt, and GitHub to cater to.
Getting back to the search portion of this post. I am finding myself writing a scheduling algorithm that spread out my API calls across a 24 hour period. I guess I can also leverage the different GitHub accounts I have access to and maybe spread the harvesting across a couple EC2 instance, but I’d rather just do what Bing offers me, and put in my credit card. I am sure there are other ways I can find to circumvent the GitHub API rate limits, but why? I would rather just be above board and put in my credit card to be able to scale how I’m using the platform. One of the biggest challenges to API integration at scale in the future will be API providers who do not offer relief valves for their consumers. Significantly increasing the investment required to integrate with an API in a meaningful way, making it much more difficult to seamless use just a handful of APIs, let alone hundreds or thousands of them. This challenge is nothing new, and just one example of how the business of APIs can get in the way of the technology of APIs—-slowing things down along the way.