I have been hearing about the challenges with keeping API portals, catalogs, directories, networks, hubs, marketplaces, and other similar constructs with the needed metadata, and adequately providing an up to date snapshot of enterprise API resources and capabilities for the last decade. It is a conversation that is growing louder due to the number of APIs in operation, and the number of enterprise organizations trying to get a handle on their sprawling API landscape. There are several realities converging on this very visible layer of our increasingly critical dimension of our API operations—the “portal”. Let me try to walk through each of these layers of this API onion to help me make sense of it, and help me better articulate how we can begin peeling back the layers and moving forward without too many tears.
The API Portal as a Construct
It is important to remember that the current construct of an API portal, and the Venn diagram of API landing page, documentation, catalog, and base for many definitions of “developer experience” is a construct that didn’t originate, but was cultivated by API management service providers. This means that many of our conversations around this problem is framed, shaped, and almost always has a dependency on the vendor, or more likely vendors we are involved with. Compare this to the early days of the web, where our web experience was framed by the likes of AOL and YAHOO. The early days of APIs (which is just the next iteration of the web), and you see that there was a lot of money to be made owning these portals and doorways, and while this concept isn’t going anywhere, it is time to embrace what the future of the developer or builder experience will be.
A Very Sprawling Landscape
The sheer number of APIs that exist across the enterprise is simply too much for the current concept of an API portal or catalog can keep up with. The only way we will be able to keep up is to automate the creation and updating of catalogs, and leveraging layers of machine readable artifacts that are outputs from a variety of manual and automated activity across many different roles and stages of the API lifecycle. Expecting developers to keep the increasing amounts of AjPI metadata up to date on their own just won’t scale, and will increasingly be out of scope for the skills API developers have. API sustainment, and evolution of legacy APIs (any APIs created before today), just isn’t priority or attractive work for many teams, and if APIs don’t get the metadata and documentation love they need in the moment, it is unlikely they will down the road.
Lots of Anxiety About API Visibility
Across the enterprise there is a lot of anxiety about whether an API is only available privately, accessible to one or many partners, or available for wider public access. This anxiety and lack of clarity around the visibility of APIs reveals itself in the richness of metadata for documentation, and catalogs. Developers often think that an API doesn’t require as much rigor if it isn’t being made available publicly to a wide audience, even if there is a chance that the visibility will widen upon release. Teams are usually just delivering an API for a specific application, integration, or partner, and aren’t concerned with the wider availability and visibility of an API, leaving metadata, documentation, and other resources an after thought. And without clear direction and guidance in this area, teams will assume the narrowest use possible of the APIs they are bringing to life.
No Common API Lifecycle Definition
Most teams just see the developer lifecycle, meaning an API gets developed and deployed. When other teams and business stakeholders work to widen this to include testing, security, or even design, you will get push back from teams of developers who don’t value or prioritize the needs of consumers outside their view. Without a common definition of what the API lifecycle is, the need for distribution and the more business facing aspects of producing APIs will always be an unknown unknown, or known unknown. The distribution of an API to the proper product catalog and developer portal is a business concern that is just out of view of developers whose job it is to develop and deploy code that powers applications and integrations. Without a clear definition of what the lifecycle is, development teams will rarely see the importance of the distribution of their APIs for others to use.
Growing Number of Roles Involved
As the value and importance of APIs to business increases, the number of business stakeholders entering the picture, and so does the complexity in understand who is responsible for the distribution of APIs to private, partner, and public portals. The lack of clarity and understanding around ownership across every stage of the API Lifecycle is contributing to growing confusion around defining, designing, but also observing, and distributing APIs across where they will be accessed via consumers. Who’s job is it to add an API to the product catalog and portal, but also sometimes more importantly, who’s job is it to update with each minor or major release of an API? It is common for API catalogs and portals to be updated when an API is released, but it is with each future release that we’ll see roles shift, change, and ultimately leaving API documentation and other resources out of sync.
The Pace of Change Occurring
The number of APIs that exist across the enterprise has significantly increased, but the pace of change that is occurring across this APIs is also dramatically increasing. Some teams find it easy to properly define, document, document an API when it is new, but then fall short with ongoing releases. API change almost always happens within the moment while responding to instability, errors, customer requests, and the growing demands of leadership. The pace of change happening across the enterprise as part of their digital transformation will continue to introduce “drift” at scale between the original intend of API stakeholders, and the reality that plays out on the ground over time.
A Bias Toward Producer View
It is common for API producers to think, well, like API producers. As an API producer you spend a lot of time thinking about what you are needing, but will struggle with understanding and empathizing with what your consumers are needing. Metadata, documentation, and making sure an API is properly represented in the API catalog and portal is an API consumer concern, and something that may not rise in priority during the mad rush to get an API produced and out the door. The more APIs you produce, the more you see things as a producer. In our busy and fast-paced operations it can be difficult to empathize with consumers, and where we begin trimming and cutting corners are usually at the metadata, documentation, and other work that will benefit consumers.
Shifting Developer Anxiety Left
With testing, design, security, and other stages of the lifecycle we are increasingly shifting work left, which often times is creating more work for developers. As this work piles up, it is common for developers to have to choose along the way which balls they will be dropping. In this reality it is common to drop or ignore the more consumer facing items, making documentation and the distribution of APIs something easy to ignore, and wait until someone complains downstream. In today’s fast-paced operations developers are asked to do more with less, so we have to work harder to better equip, enable, and guide them to make the right decisions when it comes to defining, describing, and documenting their APIs.
A Lack of a Product Mindset
Could you imagine if a company came together and decided they were going to offer a product catalog without any standards, guidelines, or communication. Imagine if 100 people were told to go develop a product and then just add your product into the product catalog before publishing without review. Think about the jumbled mess of products you’d have in the catalog, with different ways of describing products, that do a wide range of things. API portals and product catalogs being out of alignment with what is happening across a business is due to a lack of seeing APIs as a product, deserving of all the attention needed to not just develop and deploy and API, but also sell, market, and establish feedback loop with customers.
Using a Contract to Drive Process
Without a machine readable contract available for each API it becomes very difficult to ground the contribution of developers, let alone other stakeholders through the API lifecycle. OpenAPI and AsyncAPI are widely seen as simply about documenting an API, but in reality they provide a scaffolding for all of the metadata and other details needed to move an API forward. OpenAPI and AsyncAPI provide a structure to defining the summaries and descriptions for API documentation, but they also provide an opportunity to define pricing, SLA, rate limits, and many other details of API operations. API contracts drive the API lifecycle, providing a single source of truth for all stakeholders to contribute to a collective set of desired outcomes like publishing to the API catalog and portal.
The Need for More Automation
Our API portals and product catalogs are in desperate need of more automation, helping leverage the API contracts for each API to aggregate and layer on the details of what an API does and the operations around it, but then also scale the publishing and distribution of information where it is needed. We have gotten really good at automating the deployment and testing of our APIs, but we also need to get better at publishing. distributing, and keeping our APIs information up to date. Automation is the only way we are going to be able to keep our catalogs of APIs up to date with the pace of change that is occurring across operations. Automation is how we help developers and other stakeholders do more with less, while keeping our API catalog and portals an accurate representation of what is going on across teams.
Investing in More Enablement
The lack of metadata, documentation, and other resources across our API operations is the direct result of a lack of enablement provided to teams by leadership. In most cases, when an API catalog is not up to date for an API, or the documentation in the API portal is out of alignment with what is actually in production, it is because teams weren’t given the support they need. Teams just don’t think about the need to publish metadata, or it was the thing that slipped their mind in the rush of the day. There are many opportunities to enable API developers as they develop and deploy their APIs to augment with richer metadata data that powers catalogs and documentation through standards, specifications, and tooling. There are also many opportunities to enable API developers by bringing API product managers into the conversation, helping developers do what they do best, while brining more alignment between developers, customers, and the business, by using product managers as a bridge.
Accounting for Business, Gateway, and Product Drift
API artifacts like OpenAPI and AsyncAPI, combined with source control, CI/CD, and our gateways provide a rich opportunity for identifying when drift occurs from the original intent of why an API was brought to life. The infrastructure we use to deliver our APIs, and the presence of machine readable API artifacts, provide us with a significant opportunity to identify and correct the business and technical drift of each API as it progresses through all stages of the API lifecycle. Our infrastructure provides us with signals early on that we’ll need rich descriptions, tags, and other metadata for our APIs further on down the road, or possibly before we begin iterating on the next version. Infrastructure APIs, and machine readable artifacts provide us with rich and automated ways to manage drift as it occurs, rather than waiting until the drift becomes painful and impacts operations downstream.
Design-led, Code-led, and Proxy-led
There is a lot of dogma across teams when it comes to how APIs are indeed brought to life, which in reality should all be used to help keep API artifacts, catalogs, and portals up to date reflecting business and production realities. When it comes to keeping API artifacts, catalogs, and portals up to date and accurate, we are going to have to invest in trying to define and design APIs using contracts early on, while also committing to generate API contracts and updates to API contracts from code in many situations. However, both approaches are also going to need perpetual syncing with what is actually in production using proxies, APMs, and other existing outputs, allowing us to identify drift, bring more alignment, and ensure our human readable and machine readable artifiacts provides an accurate representation of what is happening–otherwise the humans and the machines will drift further apart in our understanding of how things work.
A Seamless Developer and Business Experience
Reflecting the inability for the concept of a portal and reference documentation to keep pace with the number of APIs, as well as the change across APIs, most API consumers will deamand a more meaningful experience when it comes to API engagement. Not all teams see the benefits of publishing APIs to the product catalog or developer portal when consumers are discovering APIs via blog posts, videos walkthroughs, workshops, messaging apps, and the other places we experience on a daily basis as part of our online work. I enjoy a well designed API portal, but I am an API nerd. Most API consumers, especially the growing number of business stakeholders aren’t going to feel like an API portal is speaking to them. As the number of and importance of APIs continues to expand, and the number of business stakeholders grows, the way we discover, onboard, and stay in tune with APIs is dramatically shifting.
Other Closing Thoughts
I struggle with the concept of the API catalog and portal. It feels very 1999 in terms of the web. Meaning that we used to go to a single destination to find websites, but now we understand there are many different channels we can tune into a single domain, let alone many domains. I am just not sure that our API product catalogs and portals being incomplete is the actual problem here. I feel like the real problem lies in the fact that we aren’t treating our APIs as products, and we insist on creating them adhoc to support specific applications, integrations, partners, and projects, rather than seeing them as a single product catalog or morei mportantly set of experie4nces. I think API developers neglecting to keep metadata up to date for APIs, resulting in incomplete and out of data API catalogs and portals is just a symptom of a greater challenge with properly “seeing” and prioritizing APIs as part of our business operations. Ultimately I feel like the define and design stages at the beginning of the API Lifecycle, as well as the observability and distribution stages towards the other end are where these problems should be solved—-which means it isn’t developers who should be alleviating this pain.
When you rely on API developers to maintain the metadata for each API in the product catalog and portal you end up with APIs that have descriptions like “This API provides access to the information in the database for X”, which is just a greater symptom of illnesses that exist along the enterprise digital supply chain. However, even once we bring in the product management and product marketing experience needed across the API lifecycle, I think we need to be making sure to ask the hard questions about the role of our product catalog and portals as part of our wider developer or builder experience. We need to make that we understand who our consumers are and if the product catalog and portal is how they will be discovering, exploring, and learning about our APIs. In todays world, it is more likely that we’ll need blog posts, white papers, videos, workshops, and other richer experiences that introduce consumers to our API resources and capabilities. Expecting the diverse range of stakeholders involved with integrating and applying our API resources and capabilities to find and learn about our APIs via product catalogs and portals provided by our API management providers might not make as much sense as it used just a few years back.
