Why Is API On-Boarding And Authentication Still So Hard?
I was teaching a class to business users yesterday and they were very curious about being able to play with some of the public APIs that exist, but for me, once again I found myself struggling with how to help them get over the hurdle of signing up for an API and getting the key or token you need to begin making your first call to an API. Even with having some ready to go Postman Collections that help introduce business users to some very well known APIs like Facebook, Twitter, and Instagram, I still can’t on-board them without pushing them to setup a developer account, create a new application, and obtain a set of keys or tokens. Something that is a regular hurdle for developers to jump over, and one that will keep most “normals” out of the conversation.
This bothers me. After over a decade of evangelizing the streamlining of API on-boarding processes, and listening to API management and other API service providers profess how they’ll reduce friction when it comes to putting APIs to work—clearly nothing has changed. Few APIs are truly open, meaning you don’t need any API keys or tokens to access, and the rest of the APIs use a mix of different authentication strategies to secure their APIs—introducing friction for developers and non-developers. This isn’t just a problem for non-developers, it is also a problem for developers, making it pretty challenging to go from learning about a new API to actually making a call to that API. Even with my experience, I find it very difficult to on-board with most APIs—this is a problem.
When it comes to putting APIs to work, there are a handful of common areas where friction exist, making it difficult and in many cases impossible to actually make an API call, even once they gone through the hard work of finding an API and learning about what it does:
- Sign-Up - Finding where you can signup for an API, and then going through a process of signing up can introduce a lot of friction for any user. There are many issues at play here, ranging from the amount of information required, needing approval, or just having complex multi-step processes to signup for an API account—pushing most users to just walk away from on-boarding.
- Account - Adding to the sign-up friction many API platforms have a separation in platform accounts that can cause confusion. Twitter is a good example of this. There used to be ore parity between your Twitter amount and your Twitter developer account, but these seem to be separating, and most users do not realize there is a separate account area to manage your API access.
- Keys - Next, for the APIs that require API keys for general access to an API, the process isn’t always clear where you obtain those keys even after signing up for an account. Some of this has been standardized by API management providers, but it can take me anywhere from seconds to days to figure out where the API keys are for the APIs I am on-boarding with making it a real challenge.
- Applications - Many platforms require you to setup an application and explain what you are going to do before you can obtain keys or tokens. When OAuth is involved an added dimension is introduced, making a process that is too challenging for most developers and impossible for non-developers. Where other platforms like GitHub allow you to add a simple app token for each user account, bypassing the entire process for personal use.
- Key or Token Placement - Even once you have signed up for an account, added an application, and obtained your key or token, where you put these tokens will vary from platform to platform. Do you put them in a query parameter or a header? Is it custom header or standard header? ?What is different between secret, keys, or tokens? There are many questions to be answered before you can make your first API call.
These are the main areas of friction when it comes to going from discovering an API to making your first API call. While some API providers have made this easier and very frictionless, many API providers insist on making it as hard as possible in often unintentional ways, but sometimes I suspect it is on purpose. The API management providers promised to make this easy, but because of the competitive nature of this game, they needed up each doing it in their own way, refusing to support any sort of common standard, leaving us with an even more confusing range of API on-boarding and authentication solutions. Demonstrating how venture backed API services and tooling will often do very little to actually drive interoperability and solve these common areas of friction developers and non-developers face every day in their work.
Thinking Through Solutions
Ok, so how do we fix all of this. It isn’t easy. We’d have to provide a single open source API management solution for all API providers to use, and convince them to use it. Both things aren’t going to happen. As the first proponent of there being an open source API management solution back in 2011 and 12, and living in a reality where there are now many different open source API management solutions, we won’t get every API provider using the same solution. In my opinion, this comes down to patterns and standards, and we need to begin establishing common solutions that API providers and API tooling and service providers can implement to help streamline the friction. These are some of the areas we need to collectively invest in when it comes to reducing friction:
- Sign-Up - There needs to be a set of sign-up forms in a variety of language and platforms for API providers to use.
- OpenID - There needs to be dead simple implementation for authentication using existing platform providers.
- Account - We need a set of open source account management forms and screen that standardize how accounts work.
- Applications - We need a set of open source solutions for managing applications, keys, and tokens.
- Usage - We need a set of open source solutions for reporting upon usage for each API developer.
Ok, yawwwwn. This is basically open source API management—which we have. If every API provider was the same it would reduce friction for developers when on-boarding. This is what I have been evangelizing for almost ten years. Let’s all use the common API building blocks when it comes to publishing our APIs. Emulate what Twilio, Stripe, and others do, and it will help make API on-boarding easier, where developers do not have to learn about yet another process each time they go to use another API. Ideally, also use OpenID so you don’t have to create yet another account just to kick the tires of an API, and allow developers to use my GitHub or Google account to signup and manage all of our API accounts. This is a solution, but it clearly isn’t the solution that will move the needle, because I’ve been evangelizing this approach for waaaaay too long.
If I have to add in some elements to help move this conversation forward, I would focus on a few newer ways that I feel represents API management 2.0, and allow API providers to begin thinking beyond their portal and API silo. The problem with most API providers is that even though they’ve gotten over the hurdle of doing a public API, they are still operating within their bubble, and do not think about the bigger world of APIs. Everyone wants to build their API, and API consumers will just come, sign-up, and the magic will just happen. They find it difficult to think about the fact that they are just one of thousands of APIs operating in a sea of APIs, and we need some patterns and standards that help us make our APIs more interoperable and usable at scale—here is what I am thinking might help move the needle on this.
- APIs - Every API provider needs APIs. Duh. They do! Well, they also need APIs for their API infrastructure. I should be able to signup and create an account with an API. I should be able to manage my account, applications, keys, and usage via APIs. I should be able to do everything I’m expected to do via an API portal via an API.
- Personal Tokens - I should be automatically given, or able to create a personal access token as part of my account and use to make API calls on behalf of my account, without setting up an application, and doing the OAuth dance required of most platforms. Every API should have this as a feature within the primary account for their platforms—not some separate area just for developers.
- Applications - We need to evolve the definition of what an “application” is within the modern API ecosystem. It isn’t just a web or mobile application anymore. We need personal applications, integration applications, device applications, and many other types. This is one area that API management providers have dropped the ball in their quest of exists and revenue, and not innovating in helping evolve the definition of what application means.
- Pricing - We need a machine readable definition for API service composition, access tiers, and the pricing of resources to emerge. The cost of usage, or the lack of pricing is one of the biggest challenges in on-boarding new API consumers, and if this is going to be automated and scaled, we are going to need a machine readable representation of the business of APIs, complete with tooling and services to help you manage.
- Terms of Service - One of the biggest areas of friction for this reality is the TOS for API providers, and that platforms want to have the direct relationship with each developer, restricting aggregators, marketplaces, and other brokers from signing up new users in automated way via an API or other mechanism. API providers need to be able to allow more streamlined TOS for their platforms, getting out of the way of scaled usage and adoption of their resources.
- Trust - There has to be some sort of trust network if things re going to scaled and automated. You need to know that your newly setup API accounts aren’t just bots and are trustworthy. So we need some way to validate new accounts, as well as accounts over time, ensuring that our platform users actually benefitting the platform overall. This is something that can be done through both a technical and business solution, but needs standardization.
These are what I’d consider to be the API management 2.0 investments that need to occur to fix the illnesses we experience with on-boarding developers and non-developers with APIs. Ironically it will be APIs for APIs that will help scale and automate this problem outside of the bubble for each individual API provider. I know that every API provider wants to be the next Twilio or Stripe, but 98.9% of the APIs will end up just being a commodity in a sea of APIs, and if your API won’t allow for on-boarding via an API you won’t last in the next decade. We need API providers to evolve the technology of their APIs with APIs, the business of their APIs with machine readability and APIs, and the legal side of your API operations with more scalable, flexible terms of service. We need more investment in these areas by API service providers. This work has been left on the table after the latest round of investment, acquisitions, and exists. Just as mainstream companies began on-boarding with the world of APIs.
If the next generation of developers are going to be successful on-boarding with not just 10+ APIs, but with 100+, or 1000+ APIs, we have to automate and standardize all of this. If we are going to empower business users to integrate and automate their operations using APIs in low-code and no-code ways, we are going to have to automate and standardize all of this. It shouldn’t be this hard. The technical solutions are already out there. This isn’t a technical challenge. This is a business challenge. The desires of venture capital, and the captive power of this realm has pushed API service providers to miss what needs to happen. Resulting in waves of API startups that aren’t about actually providing API solutions. They are just providing solutions that help investors find an exit, and building walled gardens and silos. We need more investment in the API life cycle tooling we already have, and more participation in API standards development that isn’t just about the selfish needs of a single company, but the robustness of the entire API economy. Let’s stop just talking about the API economy, and actually invest in what is needed to do this at scale—we all will be much better for it, and there will be plenty of money to be made along the way.