It Needs to Scale

I hear this frequently about my APIs.json work-—it needs to scale. People struggle with understanding why I would hand-craft an APIs.json search index for other peoples APIs. Trust me, I sure wish they would. My work only happens so fast. But, at least the work still gets done, and I learn a lot along the way, while simultaneously hardening the surface area of APIs.json. Plus, I enjoy doing it. I find it relaxing to come home after a full day’s work, discover new APIs, and hand-craft indexes of their operations. I don’t mind people bringing up the question of scale, but I do find it interesting that it seems to be the default response, and something that leaves me wondering why people see the API world this way-—I also can’t help but think it has a lot to do with why people also believe in AI so passionately, and why everything needs investment.

The current reality of the API universe leaves me asking why everything has to scale? I am pretty confident that at some point I will get the APIs.json specification, and the APIs.io search index up to a point where there is enough value for others to begin maintaining their own APIs.json. But, why does this have to be the focus today? I am just focused on finding new APIs and profiling them at the moment. I am interested in understanding the properties they use to define their API operations, and the individual API operations they make available. I want to know more about the industries in which companies, organizations, institutions, and government agencies operate APIs in, and what types of resources they make available are endlessly fascinating. Sure I wish there was just a magic index where every API provider maintained their own portion of the index according to my rules. But this isn’t the case, and there is so much else to focus on while I do this work. I have been hearing this rhetoric for over 10 years. I have hand-crafted three separate APIs.json indexes during this time. 1) API Stack, 2) Streamdata, 3) Postman. I have also seen other people craft proprietary indexes in this time. Each project sought scale, and none have “solved” the API discovery problem, or even contributed to it in any sustainable way. The biggest obstacle to any effort reaching critical mass is scale and the distraction of venture capital investment. I wish now I had just continued building my API Stack search index, but the need to pay my bills got in the way, and the short sightedness and misaligned motives got in the way of other efforts. Scale does not always mean success—-you have to give a shit about the index of APIs and do the work, something not all API providers are capable of.

It is going to take me some time (year or more) to get my work where it needs to be to be of any value to anyone else. And I don’t feel that the web search model translates directly to api search. And I also don’t think that AI is going to abstract away the need for high quality API indexes. It will only increase the demand. If I could work full time on this I would make much quicker headway, but with the 1-2 hours maybe 1-2 evenings a week, it is going to take time. But, I am making good progress. I feel like I have the framework and scaffolding for how it will all work at scale, I just need to do the work to properly index public APIs. There is no automated way to do this (yet). Sorry, I wish there were. I am automating what I can. Yes, ideally API providers would produce their own APIs.json indexes, but this isn’t the case. So until this happens I am going to keep doing the work. I appreciate folks letting me know that it isn’t sustainable, but I am acutely aware of this fact, and just doing everything I can to move the conversation forward.

I can see the distributed scale that is possible with API Commons, APIs.json, and APIs in my head. It is just the difference between that and what is represented in GitHub repositories and via the API Commons, APIs.json, and APIs.io domains. I see the scale, but honestly I don’t think it is the same scale y’all are concerned with, or thinking is needed to solve this. Answer me this. Did Google “fix” web search and discovery? No. They successfully developed a centralized advertising revenue engine. Google built an impressive search algorithm and inventory of web pages, and produced a convincing narrative that they were going to help you find what you need and that they would do no evil. I think Google was successful. They did interesting things with web search at scale, but they lost their way, and became just about advertising, surveillance, generating revenue. I guess revenue scale is the true metric here. I guess I see several dimensions of scale needed. First, federated web scale with API Commons, with API providers publishing their own API Indexes complete with rich machine-readable API Common defined properties—automating onboarding and operations around APIs, as well as each individual API. I think a piece of that is what people are talking about when they tell me what I am doing won’t scale. However, I also think there is a scale required with the search index itself—this is APis.io. You see, I don’t trust that API providers will always produce the best quality API indexes. Through experience I know that the people operating public API portals for enterprises don’t always care about doing them well, and more success just means more work for them, so not having an APIs.json is probably better. I think the search will have scale and be automated, and rely on a federated suite of indexes maintained by API providers, but also API curators who give shit about the indexes they are building. This is why I have broken up APIs.io into federated search ones powered by GitHub, and added the network and overlays properties to APIs.json—-governing the realities that exist at each individual dimensions.

I don’t trust API providers, and I don’t trust API service providers anymore. I don’t trust them to be good stewards of API search indexes. I am willing to walk this line with APIs.io only because I trust that my partner in this will argue with me to do the right thing, and when more funding is needed, he’ll continue to do this until, well, he can’t anymore. But I don’t trust anyone else. Sorry. Well, maybe one or two people out there I trust enough to know they will walk the line and go the distance, but most of you. NAH! Trust is difficult to scale. This is why I am keeping one of the main search nodes as well as the mining and rating of API patterns more of a proprietary dictatorship, while I also invest in the API Commons. I don’t trust commercial or open-source anymore. I trust in my negotiated vision of API Commons, APIs.json, and APIs.io, balanced through my completely selfish and ego-driven vision of API Evangelist. Across this I do see opportunities for scalability, but also doing small things really well in a way that scales through federation. I envision a more nuanced scalability that will take time to build. I am weary of the current off the shelf notion of scalability being prescribed for API discovery. I don’t think there is a precedent for what will work, and I am hoping that I have a vision that just might work. We’ll see.