API Governance Models In The Public and Private Sector
19 Jul 2018
This is a report for the Department of Veterans Affairs microconsulting project, “Governance Models in Public and Private Sector”. Providing an overview of API governance to help the VA, “understand, with the intention to adopt, best practices from the private and public sector, specifically for prioritizing APIs to build, standards to which to build APIs, and making the APIs usable by external consumers.” Pulling together several years of research conducted by industry analyst API Evangelist, as well as phone interviews with API practitioners from large enterprise organizations who are implementing API governance on the ground across the public and private sector, conducted by Skylight Digital.
We’ve assembled this report to reflect the interview conversations we had with leaders from the space, helping provide a walk through of the types or roles and software architecture being employed to implement governance at large organizations. Then we walk through governance as it pertains to identifying possible APIs, developing standards around the delivery of APIs, how organizations are moving APIs into production, as well as presenting them to their consumers. Wrapping up with an overview of formal API governance details, as well as an acknowledgement that most API governance is rarely ever a fully formed initiative at this point in time. Providing a narrative for API governance, with a wealth of bulleted elements that can be considered, and assembled in the service of helping govern the API efforts across any large enterprise.
Roles Within An Organization
There are many roles being used by organizations who are leading the conversation around the delivery of high quality, industry changing APIs. Defining the personalities that are needed to make change across large organizations when it comes to delivering APIs consistently at scale. While there may be many names for the specific roles leading the charge it is clear that these people are bringing a unique blend of skills to an organization, with an emphasis in a couple of key areas:
- Leadership - Providing leadership for teams when it comes to APIs.
- Innovation - A focus on innovation using APIs across the organization.
- Communication - Facilitating communication across all teams, and projects.
- Advisory - Acting as an advisor to existing leadership and management.
- Strategy - Helping existing teams develop, evolve, and realize their strategy.
- Success - Focusing on helping existing teams be successful when it comes to APIs.
- Architect - Bringing a wide variety of software architectural skills to the table.
- Coaching - Being a coach to existing teams, and decision makers across the organization.
Bringing together a unique set of skills that range from the technical to deep knowledge of the business domain, into a concentrated, although sometimes distributed effort to bring change across an organization using APIs. Along with these roles, many large organizations are investing in new types of structure to help develop talent, take charge of new ideas, and move forward the enterprise wide API strategy, with a handful of common characteristics:
- Labs - Treating API efforts as if it is a laboratory creating new experiments.
- Center - Making it a center for API thinking, ideation, and for access to information.
- Centralized - Keeping all efforts in a single group or organization within larger entity.
- Distributed - Emphasis on keeping API knowledge distributed and not centralized at all.
- Global - Acknowledging that APIs will need to be a global initiative for larger organizations.
- Excellence - Focusing on the organization bring excellence to how APIs are delivered.
- Embedded - Making sure there is API knowledge and expertise embedded in every group.
Combing a unique set of skills and personalities, into a focused organization that takes the reins when it comes to leading API change, and digital transformation at an organization. While many of these efforts emphasize a center, or centralized presence, many are also realizing the importance of embedded and distributed approach, ensuring that talent, and ideas grow within existing teams, and are not seen as just some new, external, isolated group or initiative.
There clearly is not a single role or organization structure that brings success to API efforts at scale across the enterprise, however there are clear patterns being applied in the early stages that can be emulated. Helping ensure that API knowledge and expertise is available and accessible by all groups across an organization, and all its geographic regions, ensuring that the entire enterprise is part of the conversation and moving forward in unison.
Software Architecture Design
Governance is all about shaping and crafting the way we design and architect software, leveraging the web, and specifically web APIs to help drive the web, mobile, device, and network applications we depend on. There are a number of healthy, and not so healthy patterns across the landscape for considering as we look to shape and transform our software architecture, begin honest about the forces that influence what software is, does, and what it will ultimately become.
Software architecture is always a product of its environment, being influenced by a number of factors that already exist within any given domain. We are seeing a number of factors influence how large enterprises are investing and defining their software architecture. Here are a handful of the top areas of consideration when it comes to how the domain an enterprise exists within impact architecture:
- Resources - The types of digital resources an enterprise already possess will drive software architecture, defining how it works, grows, expands, and shifts.
- Schema - Existing schema define how data is stored, and often gathered and syndicated–even if this is abstracted away through other systems, it is still influencing architectural decisions at all levels.
- Process - Existing business process are already in motion, driving current architecture, and is something that cannot immediately be changed, without having echoes of impact on future architectural decisions.
- Industry - External industry factors are always emerging to shift how software architecture is crafted, providing design, development, and operational factors that need to be considered as architecture is refactored.
- Regulatory - Beyond more organic industry influences, there are regulatory, legal, and other government considerations that will shift how software architecture will ultimately operate.
- Definitions - The access and availability of machine readable definitions, schema, process, and other guiding structural elements that can help make software architecture operate more efficiently, or less efficiently in the absence of standardization, and portability.
Domain expertise, awareness, and structure will always shape software architecture, and the decision making process that surrounds it. Making it an imperative for there to be an investment in internal capacity as well as leveraging external expertise and vendors when it comes to shaping the enterprise architectural landscape. Without the proper internal capacity, domain knowledge can be minimized, weakening the overall architecture of the digital infrastructure nutrients an enterprise will need to move forward.
We can never escape the past when it comes to the software architectural decisions we make, and it is important that we don’t just see legacy as a negative, and also view legacy as a historical artifact that should move forward. Maybe not always the same legacy code should be in forward motion, but the wisdom, knowledge, and lessons learned around the enterprise legacy should be on display. Here are a handful of the legacy considerations we’ve identified through our discussions.
- Systems - Existing systems in operation have a significant influence over all current and future architectural decisions, making legacy system consideration a top player when it comes to decision making around software architecture conversations.
- People - Senior staff who operate and sustain legacy system, or were around when they were developed possess a significant amount of power when it comes to influencing any new system architecture, and what gets invested in or not.
- Partners - External partners who have significant history with the enterprise possess a great deal of voting power when it comes to what software architecture gets adopted or not.
- Trauma - Legacy trauma from historical outages, breaches, and bad architectural decisions will continue to influence the future, especially when legacy teams still have influence over future projects.
Systems, people, partners, and bad decisions made in the past will continue to drive, and often times haunt each wave of software architectural shifts. This influence cannot be ignored, abandoned, and needs to be transformed into positive effects on next generation investment in software architecture. Change will be inevitable, and legacy technical and cultural debt needs to be addressed, but not at the cost of repeating the mistakes of the past.
After legacy concerns, we all live in the reality have been given, and it is something that will continue to shape how we define our architecture. Throughout our discussions with companies, institutions, and government agencies regarding the challenges they face, and the current forces that shape their software architecture decisions, we found several recurring themes regarding contemporary considerations that were making the largest impact:
- Talent Available - The talent available for designing, developing, and deploying of API infrastructure dictates what is possible at all stages.
- Offshore Workers - The offshoring of work changes the governance dynamics, and requires strong processes, and a different focus when it comes to execution.
- Mainstream Awareness - Keeping software architectural practices in alignment with mainstream practices helps shape software architecture decisions, allowing them to move forward at a healthier pace.
- Internal Capacity - It has been stated several times that doing APIs at scale across the enterprise would not be possible without investing in internal capacity over outsourcing, or depending on vendor solutions.
Modern practices continue shaping how we deliver our software architecture, defining how we govern the evolution of our infrastructure, and find the resources and talent to make it happen. Keeping software architecture practices in alignment with contemporary approaches helps streamline the road map, how teams work with partners, and can outsource and work with external entities to help get the job done as efficiently as possible.
The technology we adopts helps define and govern how software architecture is delivered, and evolved. There are many evolution trends in software architecture that has moved the conversation forward, allowing teams to be more agile, consistent, and efficient in doing what they do. As we studied the architectural approaches of leading API providers across the landscape, and engaged in conversations with a handful of them, we found several technologically defined views of how software architecture is influencing future generations and iterations.
- Vendors - Specific vendors have their owning guiding principles to how software architecture gets defined, delivered, and governed. Often times given an outsized role in dictating what happens next.
- Frameworks - Software and programming language frameworks dictate specific patterns, and govern how software is delivered, and lead the conversation on how it evolves. Software frameworks can possess a significant amount of dogma that will have a gravity all its own when it comes to evolving into the future.
- Cloud Platforms - Amazon, Google, and Microsoft (Azure) have a strong voice in how software architecture is defined in the current climate, providing us with the services and tooling to govern the lifecycle. This control over how we define our infrastructure is only going to increase with their market dominance in the digital realm.
- Continuous Integration / Deployment - CI/CD services and tooling have established a new way of defining software architecture, and establishing a pipeline approach to moving it forward, building in the hooks needed to govern every step of its evolution. Reducing the cycles from annual down to monthly, weekly, and even daily cycles of change.
- Source Control - Github, Gitlab, and Bitbucket are defining how software is delivered, providing the vehicle for moving code forward, the hooks for governing each commit, and step forward any infrastructure makes as it is versioned, and evolved.
These areas are increasingly governing how we design, develop, deploy, and manage our infrastructure. Providing us with the scaffolding we need to hang our technological infrastructure on, and gives us the knobs and levers we can pull to consistently orchestrate, and move forward increasingly complex and large enterprise software infrastructure, across many teams, and geographic regions. The decisions we make around the technology we use will stick with us for years, and continue to influence decisions even after it is gone.
When it comes to delivering software architecture, not everything is governed by the technical components, and much of what gets delivered and moved forward will defined by the business side of the equation. The amount of investment by a business into its overall IT, as well as more progressive groups, will determine what gets done, and what doesn’t. With the following elements of the business governing software architecture in several cases::
- Budgets - How much money is allocated for a team to work when it comes to defining, deploying, managing, and iterating upon software architecture.
- Investors - Many groups are influenced, driven, and even restricted by outside investors, determining what software architecture is prioritized, and even dictating the decisions around what is put to work.
- Partners - External partners with strong influence over the business discussions that drive software infrastructure decision play a big role in the governance, or lack of governance involved.
- Public Image - Often times the decisions that go into software architecture, and the governance of how it moves forward will be driven by the public image concerns around company, and its stakeholders.
- Culture - The culture of a business will drive decisions being made when it comes to developing, managing, and governing software architecture, which can be more challenging to move forward than the technology in many cases.
The governance of software architecture has to be in alignment with business objectives of an enterprise. Many groups choose to begin their API journeys based upon trends, or the desire of a small group, and have encountered significant friction when trying to bring in alignment with the wider enterprise business objectives. Groups that addressed business considerations earlier on in their strategy have done much better when it came to reducing friction, and eliminating obstacles from their road map.
Almost every discussion we’ve had around governance of software infrastructure has included mentions of the importance of observability across next generation iterations. Software designed, delivered, and supported in the darkness or in isolation either fails, or is destined to become the next generation of technical debt. There were several areas of emphasis when it came to ensuring the API driven infrastructure sees the light of day from day one, and continues to operate in a way that everyone involved can see what is happening.
- Out in Open - Groups who operate out in the open, sharing their progress actively with other teams, and encouraging a transparent process find higher levels of success, adoption, and consistency across their architectural efforts.
- Co-Discovery - Ensuring that before work begins, teams are working together to discovery new ideas, learn about alternative software solutions, and working together to create buy-in, and ultimately make decisions around what gets adopted.
- Collaborative - While identified as sometimes being slower than traditional, more isolated efforts, teams who encouraged cross-team collaboration saw that their architectural decisions were sounder, more stable, and had more longevity.
- Open Source - Following open source software development practices, and working with existing open source solutions helps ensure that enterprise software architecture last longer, has more support, and follows common standards over other more proprietary approaches.
- Publicly - When it makes sense from a privacy and security standpoint, groups often articulate that being public by default helps ensure project teams behave differently, enjoy more accountability, and often attract external talent, domain expertise, and publicly opinion along the way.
Enterprise organizations that push for observability by default find that teams tend to work better together, and have a more open attitude. Attracting the right personalities, encouraging regular communication, and thinking externally by default, not as something that happens down the road. Bringing much needed sunlight and observability into processes that can often be very complex and abstract, and pushing things to speak to a wider audience beyond developer and IT groups.
Having a shared process that can be communicated across teams, going beyond just technical teams, and is something that business groups, partners, 3rd party, and all other stakeholders can follow and participate in, is a regular part of newer, API-centric, software delivery life cycles. Possessing several core elements that help ensure the process for defining, designing, delivering and evolving software architecture is shared by all.
- Contract - Crafting, maintaining, and consistently applying a common machine readable contract that is available in YAML format, is a common approach to ensuring there is a contract that can be used across all architectural projects, defining each solutions as an independent, business service.
- Pipeline - Extending the machine readable service contracts with YAML defined pipeline definitions that ensure all software is delivered in a consistent, reproducible manner across many disparate teams.
- Versioning - Establishing a common approach to versioning code, definitions, and other artifacts, providing a common semantic approach to governing how architecture is evolved in a shared manner that everyone can follow.
Historically the software development and operation lifecycle is owned by IT, and development groups. Modern approaches to delivering software at scale is a shared process, including internal business and technical stakeholders, while also sharing the process with external partners, 3rd party developers, and the public. Bringing software architecture out of the shadows, and conducting it on the open web, making it more inclusive amongst all stakeholders, but done in a way that respects privacy and security along the way.
Identifying Potential APIs
Once the architectural foundations have been laid, there are many ways in which large enterprises begin identifying the potential APIs that should be designed, deployed, and evolved supporting the many applications that will be depending on the underlying platform architecture. Depending on the organization and it’s priorities, the reasons for how new APIs are born will vary, resulting in different lifecycles, and resulting services being delivered across internal groups, partner stakeholders, and 3rd party developers.
Throughout this research, we identified that there is no single approach to identifying which APIs should be delivered, but we did work to understand a variety of approaches in use across the landscape, and by the practitioners interviewed for this project. Establishing some common areas for answering the questions around what should be an API, why are we doing APIs, leading us to the how of doing APIs, and uncovering the pragmatic reasoning behind web, mobile, device, and other applications that APIs are driving across the landscape.
Our existing realities drive the need for APIs, and reflect where we should be looking to provide new services for internal stakeholders, partners, and potentially as new revenue streams for 3rd party developers. While some APIs may be entirely new solutions, it is most likely that APIs will be born out of the realities we are already dealing with on a daily basis, based upon the digital solutions we depend on each day. We identified the most common realities that enterprise group face when it comes to their present day digital transformation challenges.
- Database - The existing databases in operation are the number one place groups are identifying potential resources for deployment of APIs. Exposing historically accumulated digital assets using the web, and making them available for use in new applications, to partners, and driving new types of data products for generating the next generation of revenue streams.
- Website - Our existing websites reflect the last 20 years of the digital evolution of our enterprises, representing the digital resources we’ve accumulated and identified as being important for sharing with partners and the public. HTML representations of our digital assets are always the 101 of API deployment, and understand what should also be available as JSON, and other more sophisticated representations.
- Integrations - Existing software integrations, system to system integrations, are represent a rich area for understanding how digital resources are delivered, accessed, and made available throughout existing applications and systems. Providing an important landscape for mapping out when understand what should be turned into APIs, having a more consistent process applied, eliminating custom integrations, and standardizing how systems speak with one another.
- Applications - Exiting web, mobile, desktop and other applications can have a variety of backend system connectivity solutions, but also might have a more custom, bespoke approach to doing APIs that is off the radar when it comes to governing how infrastructure evolves. Providing another rich area for mapping out the connections behind the common applications in use, understanding the internal, partner, and 3rd party APIs and other connections they use.
- Services - APIs have been around for a while, and exist in a variety of formats. Legacy web services, RPC, FTP, and other API and messaging formats should be mapped out and included as part of the potential API evolutionary landscape. Taking existing services, and evolving them in a consistent manner with all other API-driven services, leveraging web technologies to consistently deliver and manage digital resources.
- Spreadsheets - The majority of business in the world still happens within the spreadsheet. These portable data stores are emailed, shared, and spread around the enterprise, and represent a rich source of information when it comes to understanding which resources should be published as APIs.
You can’t govern what isn’t mapped out and known. It becomes increasingly difficult to govern software infrastructure that exists across many open and proprietary formats, and delivered as custom one-off solutions. Governance begins with a known landscape, and the greatest impedance to functional governance across organizations are the unknowns. Not knowing a solution exists, or its architectural approach being something that isn’t part of the bigger picture, leaving it to be a lone actor, in a larger landscape of known services operating in concert.
Another reason for having an open, very public approach to selecting, delivering, and operating software infrastructure, is that it establishes a public presence, across web properties, social networks, and other platforms where enterprise organizations can build community. There are a number of ways to identify potential new API resources by just being public, engaging with the community, and establishing API delivery life cycles that involve having a public presence.
- User Requests - Actively soliciting web, mobile, and developer feedback, both internally and externally is a great way to learn about potentially new API resource opportunities. Leveraging existing users as a source of insight when it comes to what services would make applications better, and demonstrating the importance of investing in an API literate user base.
- Partner Requests - Actively working with partners, conducting regular meetings regarding digital assets and transformation, seeking feedback on what types of services would improve upon existing solutions, and strengthen partner relations. Investing in existing partners, and using them as a way to evolve the API road map, and increase their dependency on enterprise resources.
- Public Feedback - Engaging with the public via websites, social networks, forums, and at events, to understand what opportunities are out there when it comes to delivering new API resources. Tapping public awareness around specific topics, within particular domains, and considering suggestions for new APIs outside the enterprise firewall and the traditional box.
- Media Coverage - Tuning on to popular tech blogs, mainstream media, and other public outlets to study and understand what opportunities are emerging for the delivery of new API services. Tuning into popular trends when it comes what is happening with APIs, or business sectors that might not have caught up to some of the modern approaches to delivering APIs.
- Feedback Loops - Cultivating trusted feedback loops with existing users, social networks, and private messaging platforms. Investing in long term feedback loops that tap the knowledge and domain expertise of trusted professionals, who can bring fresh ideas to the table when it comes to which new APIs can benefit the enterprise.
- Negative Consequences - One significant thing to note with having a public presence, is that not everything will be positive. There are serious privacy, security, and safety concerns when operating your APIs out in the public, and there should be plenty of consideration for how to do it in a healthy way. Acknowledging that not everyone in the public domain will have the enterprise’s best interest in mind.
It isn’t easy soliciting feedback from the general public when it comes to determining the direction a platform road map should head. However, with some investment, curation, and cultivation, a more reliable source of insight regarding the direction of an API platform can be established. The API community across the public and private sector has grown significantly over the last decade, providing a wealth of knowledge talent that can be tapped, if you know where to look for it.
Moving beyond the “where to look for opportunities for APIs”, and moving into the why to find, as well as how to prioritize which resources should be turned into APIs, we hear a lot about investing in the overall improvement of the enterprise when it came to the motivations behind governance. Looking at many of the common incentives behind doing APIs, but also doing it in a more consistent and scalable way that supports the mission and forward motion of the enterprise.
- Optimization - Seeking to optimize how applications are delivered, and services provided across teams. Providing consistent services that can be used across many internal groups, across partners, and that will improve the lives of 3rd party developers.
- Common Patterns - Doing APIs, and pushing for governance to help identify common patterns across how software is designed, delivered, and managed. Working to extract the existing patterns in use, help standardize and establish the common patterns, and reenforce their usage across teams, and distributed groups.
- Reusability - Encouraging reusability is the number one improvement we heard from different groups, and see across the landscape. Governing how software is not just delivered, but maximized, reused–ensuring the enterprise is maximizing software spend, as well as the revenue from services it delivers.
- Acceleration - Investing in governance to help accelerate how applications are delivered, measuring, standardizing, and optimizing along the way to improve existing efforts, as well as new projects on the horizon. Increasing the speed of not just new services, but how the services are able to be put to work by developers and integrators.
- Efficiency - Setting into motion patterns and processes that increase overall efficiency around how services are delivered, and how they enable teams to deliver on new projects, applications, integrations. Allowing IT, developers, and business users to benefit from an API focus.
- Flexibility - Increasing the flexibility of how applications operate, and teams are able to work together and deliver across the enterprise. Encouraging the design and development of APIs that work together, and flexibly achieve organizational objectives.
Providing a set of criteria that can be used to help prioritize which APIs get identified for delivery, and for evolution and versioning. If API resources help deliver on any of these areas, their benefit to the enterprise is increased, and it should be bumped up the list when looking for API opportunities. Always looking for how the enterprise can be improved upon, while also understanding which specific resources should be targeted for wrapping, and exposing as simple web APIs that can be used for both internal and external use cases.
The API journey is always full of challenges, and these areas of friction should be identified, and incorporated into the criteria for identifying new API solutions, as well as determining which APIs should be invested in and evolved. While some challenges can be minimized and overcome, many can also cause unnecessary friction throughout the API roadmap, making challenges something to be considered when putting together any API release strategy.
- Education - What education is required when it comes to acquiring the resources behind any potential API, developing and standing up a proper API, as well as deploying, managing, and supporting an API. What challenges can be foreseen, and identified early on, helping weigh what investment in education, training, and learning along the API journey for any set of services.
- Maturity - Understanding early on, and putting together a plan on what maturity will look like for any service, acknowledging that every service will begin in a juvenile state, and take time to harden, mature, and become something that is dependable, reliable, and usable in a production environment.
- Isolation - Identifying resources that are being developed, maintained, and operated in isolation, and working to move them out into the mainstream. While also ensuring that any new services being developed avoid isolation, and are developed, evolved, and managed out in the open, ensuring that services never operate alone.
- Management - Including management in discussions around which resources should be developed into APIs, including leadership in all conversations involving the targeting, evolution, and even deprecation of API services. Ensuring that the prioritization of API development is always on the radar of management, and there is a general awareness regarding the delivery of services.
- Consistency - Realizing that while consistency is the goal, it may be an elusive, non-stop chase to actually realize consistency across teams. It should be a goal, but also realistically understanding that it won’t be easy to achieve, and while we want to strive for perfection, sometimes there will be good enough achieved for some services.
- Reusability - Similar to consistency, reusability is an obvious goal, and should be worked towards, but it will also be elusive, and not always reliably achieved over time. There might still be redundancy for some services, and overlapping aspects of delivering services, while some areas reusability will be achievable.
- Build It And They Will Come - There has been a significant amount of reflection regarding targeting, developing, and publishing APIs that were not needed based upon an “if you build it, they will come” mentality–where most often, nobody came, and the work was in vain.
Challenges are a fact of life in the delivery of software, and evolving complex systems using APIs. Identifying challenges should be a natural part of targeting resources for delivery as APIs. Challenges can increase fiction when delivering service, and should be carefully evaluated before tackling the development of any new services. It is easy to identify the potential of new APIs, but it takes a more seasoned eye to understand potential challenges.
New ideas for APIs will become numerous once you begin looking. Based upon existing resources, applications, and the feedback of internal groups, partners, and the public. Along with all of the possibilities that come along, a standardized, pragmatic approach to understanding the potential, value, as well as challenges with each potential idea should be part of the equation.
Defining Data Models & Standards
To help realize and deliver upon governance at scale it will take heavy investment in standardizing data models, and incorporating existing patterns and standards throughout the API delivery lifecycle. Many enterprise API development groups are streamlining and standardization the delivery of APIs through the adoption and development of standards across operations, which is something that is also contributing to adoption, integration and removing friction for application developers.
The adoption of common data models, interfaces, media types, and web standards helps contribute to the delivery of consistent APIs at scale, but they can also prove to be a challenge for some teams, and even been seen as a threat by others. There are a number of ways in which teams are pushing for standardization across their operations, and helping achieve more consistency, reuse, and the desired results across operations. Reflecting one of the strengths of web APIs, in that they employ web standards to achieve wider adoption, and the delivery of valuable resources at web scale.
A suite of approaches have emerged in the last decade for designing, developing, evolving, and applying common API patterns across the API lifecycle. These standardized approaches to defining and delivering APIs, using common machine readable specifications, and widely used patterns, have become central to API governance discussions. Providing the fuel for the growth of the API sector to serve mobile applications, as well as the growth of other emerging channels like voice, bot automation, and the connecting of everyday objects to the net. Helping the enterprise get more organized about how services are delivered across the organization at scale.
- Resource-Defined - RESTful design patterns have provided a simple approach to taking corporate resources and defining them as an intuitive, reusable, potentially web-scale stack of API resources that can be used across a variety of applications. REST provides a philosophy that can be adopted across the enterprise to help organize digital resources as a reusable stack of resources that can be discovered and put to use across many channels.
- Schema-Driven - JSON Schema is being used to take a variety of schema and standardize them for use in RESTful API resource delivery. Providing a reusable blueprint that can be used across the request and response model for all APIs. Deriving, and standardizing existing schema in use, and making available for usage in in newly developed, and evolving APIs allows for teams to achieve many of the objectives set out as part of modern API strategies.
- Domain Driven - The business domain is used across the enterprise for guiding the identification, development, evolution, and standardization of a variety of API definitions in use across the enterprise. Lines of business, industry definitions, and a focus on the domain helps establish areas of concern, and the separation of services, allowing for the decoupling of enterprise resources used across systems, but working in unison to deliver a single set of business objectives.
- Legacy Abstraction - Continue movements to decouple, redefine, and evolve legacy systems is pushing forward the identification of common patterns, and pushing to map, transform, and give them new life as newer web APIs. Taking legacy databases, system interfaces, and distilling the wisdom that exists across them, to help drive the development of common standards.
- Vocabularies - API development teams are establishing common vocabularies based upon the standardized language already in use, but also essentially taking the slang that is used in bespoke systems and helping tame it, and add it to the common lexicon when it makes sense. Providing a standard language that can be used across the enterprise to talk about services, resources, and digital assets.
- Discovery - Many groups expressed challenges around standards not seeing the desired adoption because other teams could not find existing schema, definitions, and other existing standards. Emphasizing the importance of comprehensive, actively maintained, and evangelized catalog of core definitions across the enterprise. Providing a single, or distributed location where everyone can find and publish their common definitions.
The definitions coming out of existing API development efforts are being organized into catalog, and discovery systems that can be used to guide governance efforts. Mapping out the known landscape across the enterprise, and turning it into the common patterns that can be reused across the design, development, and operation of the next generation of APIs. Distilling down the essence of the enterprise so that it can become the building blocks of an API program, while also allowing each stop along the lifecycle to be quantified, measured, and considered as part of a wider governance strategy.
Doing Business On The Web
APis are built on the web. They use, and benefit from over 25 years of the evolution of the web. There are a number of elements to consider when working to identify and define common standards for use across the governance of any API program. While the API strategy should be rooted in definitions derived from the core of the enterprise, secondarily it should be embracing the web, and employing common patterns that make the web work to use as the foundation for delivering APIs.
- Web Standards - The web is the foundation for the delivery of APIs. Most APIs will use HTTP as a transport, and be employing URLs, HTTP verbs, headers, parameters, and other common web standards. Web standards should be part of any governance strategy to help establish common patterns and definitions for use across operations.
- Media Types - Media types are a fundamental part of the web, and help establish message formats that will be widely recognized outside the enterprise, encouraging the reuse and adoption of APIs that employ common media types. Allowing consumers to negotiate the format that makes the most sense to their team, and the types of applications they are looking to develop.
- Industry Schema - Industry level schema are emerging and maturing for use across API operations. Specifications like FHIR, PSD2, and other schema, along with API design patterns are evolving to help support industry focused API operations, while encouraging reuse and interoperability across disparate groups.
- Open Source - The usage of open source software, tooling, specifications, and processes are helping deliver on the API vision across the enterprise. Web APIs reflect the open source ethos, and plays well with the delivery of web APIs. Encouraging reuse, adoption, and bringing the observability necessary to help APIs succeed.
APIs are all about doing business on the web. The web provides the platform in which any API program will operate. When it comes to defining schema, standards, and common patterns for use across API operations, the web is always the beginning of the conversation. While enterprise defined patterns will always be front and center, the standards used to operate the web should always trump localized definitions, and be given priority whenever possible. Don’t reinvent the wheel when it comes to the web, always reuse and implement what is already known.
Under The Influence
When learning about new standards, and considering which standards to adopt, it can be easy to find yourself under the influence of specific vendors, competing standards, programming communities, and other factors. Careful evaluation of standards is important, and an awareness of what some of the common elements are that may shift your opinions one way or another, or even obfuscate what is real and prevent you from achieving objectives.
- Caught in Trends - Avoid getting caught up in the trend cycles that can often make it difficult to understand the hype around specific specifications. Do your research, understand best practices and adoption levels, and make the sensible decisions around the impact to your own efforts.
- External Entities - While engaging with external entities, understand what their priorities are when it comes to standards and specifications. Consider what affinity may exist between enterprise objectives, and any external entity that is engaged with, and makes sure there is the right alignment, and influences are pushing efforts in the right direction.
- Internal Demands - Similar to external entities, understand what the internal teams priorities are and don’t always assume internal requests will have the overall enterprise objectives in mind. Fully understanding what the awareness and motivation are around the implementation of specific standards, and how the fit into the overall strategy.
- Feedback Loops - Ensure that feedback looks are diverse, and provide a wealth of opinions around what types of standards and specifications should be supported, providing the widest possible view of the landscape when it comes to adoption and investment.
- Organic Change - Keeping an eye on vendor induced standards adoption over a more organic approach to the growth of standards, internally, as well as the outside community. Working to understand when a standard is artificially inflated, amplified, for alternative objectives beyond its core mission.
There are plenty of currents to get caught up in when it comes to identifying, defining, and evolving standards. Not all will bear fruit, or realize the type of adoption they need to be successful. Establishing a balanced view of the landscape across internal, and external actors, while keeping counsel with a diverse set of voices can help ensure you understand which API specifications, standards, and definitions will help move the enterprise forward.
Taking The Lead
While there are a number of ready to use standards available for the web, and organically grown out of the API community, these standards won’t always find their way into the enterprise. Leading organizations demonstrate that it takes a structured effort to define, disseminate, educate, and evolve standards across large organizations, with a number of proven tactics for taking the lead when it comes to standardizing API infrastructure across the enterprise.
- Workshops - Organizing, conducting, and growing the number of workshops held to introduce individuals across many teams to a variety of common standards and specifications.
- Discussions - Formalizing discussions around emerging standards, and those that are in use, to help push forward awareness, and adoption of standardized approaches across groups.
- Collaboration - Push teams to work together when it comes to sharing the standards in use, showcasing the investment they’ve made, and working together to understand the tooling, services, and standards being used.
- Event Storming - Putting event storming, a rapid, lightweight group modeling technique to help accelerate the identification, evolution, and adoption of standards that meet specific team’s needs.
- Influencers - Identifying, investing in, and cultivating influencers who exist within current groups, and encourage them to evangelize and help spread the good word about standards across the enterprise.
- Ask Questions - Always be asking questions about the standards, or lack of standards in use across the enterprise, pushing the conversation forward at all times when it comes to standards.
- Challenge Assumptions - Making sure teams don’t get complacent, and the status quo is always being challenged, and that the internal domain should always be rising to a higher level of standardization whenever possible.
It takes standards bodies to move forward common standards at the web and industry levels, and it takes the same approach to push forward the adoption and usage of standards within the enterprise. Leading enterprise organizations are able to quantify, measure and evolve the infrastructure in a more organized way through the adoption of common schemas, specifications, and standards. Providing a common vocabulary for all teams to use when designing, deploying, and managing services that can be used consistently across the enterprise, and its public interests.
Development to Production
After understanding the roles needed to realize governance, more about the underlying platform architecture that is needed, how organizations can identify where the API opportunities are, and making sure groups are putting standards to work, we scrutinized how groups are moving APIs from development to production in a more structured way. Governing how teams are efficiently moving APIs from idea and design, to actually putting services to work in a production environment at scale across large teams. Documenting the lifecycle of a service, and the common elements of how enterprises are getting the job done on a regular basis.
To be able to deliver APIs at scale in a consistent way teams are relying on a well honed, well defined lifecycle that has been defined, proven, and evolved by senior teams. Forcing structure and rigor throughout the evolution of all services, putting governance in front of teams, and forcing them to execute in a consistent way if they expect to reach a production reality with their services. Focusing on a handful of structured formats for imposing governance at the deployment levels.
- Contract - Requiring ALL services begin with a machine readable OpenAPI contract defining the entire surface area of the API and its schema. Leveraging the contract as the central truth for what the service will deliver, and how governance will be measured throughout the lifecycle of the service.
- Process - Providing a well defined process for all developers laying out how any service moves from design to production, with as much detail regarding each step along the way. Helping all developers understand what they will face as they work to move their services forward in the enterprise.
- Scorecard - Having a machine readable checklist and scorecard, with tooling to help each developer fork or establish an instance for their service. Providing a checklist of everything they need to consider, that allows them to check off what has been done, what is left to be done, and providing a definition that can be used to define and report upon governance along the way.
- Cycles - Provide a variety of cycles that every service will need to go to before they will be production worthy, forcing developers to iterate upon their services, harden and mature them before they will be considered ready for production.
- Reviews - Require all services go through a series of lifecycle reviews by other teams, pushing service owners to present their work to each review team, and work with them to satisfy any concerns, and make sure it meets all governance criteria.
- Clinics - Providing developers with a variety of clinics where they can receive feedback on their work, improve upon their service, and improve the health of their work before submitting it for inclusion in a production environment.
Enterprise organizations that provide structure for API development teams find it much easier to realize their governance aspirations. The scaffolding is already there to think about the consistency of services, and the face to face, and virtual scrutiny of services helps provide the environment for governance practices to be executed, enforced, and evolved before any service reaches a production state. A well defined API deployment lifecycle will help contribute to a well defined API governance practice.
One sign of enterprise groups who are further along in their governance journeys is when there are virtualized environments being put to use. Requiring all API developers to mock and iterate upon their APIs in a virtualized environment, presenting them as if they are real, before they ever are given a license to write any code, let alone reach a production state for their services.
- Mocking - Creating mock APIs for all endpoints, virtualizing every aspect of the surface area of an API, allowing a service to be iterated upon early on in its lifecycle.
- Data - Requiring virtualized and synthesized data be present for all mocked APIs, returning realistic data with responses, reflecting behavior encountered in a production environment.
- Sandbox - Providing a complete labs and sandbox environment for developers to publish their mocked APIs into, reflecting what they’ll encounter in a production environment, but done in a much more safer and secure way.
Virtualized environments provide an important phase in the journey for APIs moving from concept to reality. Establishing a safe environment for developers to iterate upon their work, encounter many of the challenges they’ll face in a public environment, without any of the potential for harm to users or the platform. Ensuring that when a service is ready for development, most of the rough edges have been worked out of the service contract, and for the team behind.
One of the most significant ways in which we’ve found enterprise groups governing the evolution of their APIs is through the technology they employ. This technology is providing much of the structure and discipline that organizations are depending on to help ensure that APIs are being developed, and ultimately deployed in a consistent manner. Bringing most of the governance to the table for some organizations who haven’t begun moving their governance strategy forward as a formal approach.
- Authentication - Requiring standard-based approaches to authentication using Basic Auth, API keys, OAuth, and JWT. Ensuring the teams understand when to use which protocol, and how to properly configure, and use as part of larger API governance strategy.
- Framework - Relying on the programming frameworks in use to inject discipline into the process, dictating the governance of how APIs are delivered before they are ready for a production environment.
- Gateway - Applying the policies and structure necessary to govern API services as they are made available in a production environment. Many groups also had a sandbox or development edition of their gateway emulating many of the same elements that will be found in a production world.
- Management - Similar to the gateway, groups are relying on their API management layers to help govern what APIs do, providing transformations, policy templates, and a wide variety of other standardization that occurs prior to being made available in production sense.
- Vendor - The reliance on technology to deliver governance at the API deployment level gives a lot of control to vendors when it comes to governing the API lifecycle. If a vendor doesn’t provide a specific way of doing things, it may not exist within some groups. Dictating what is governance for many enterprise groups.
- Tooling - Most groups have an arsenal of open source, and custom developed tooling for helping push code from development to production, validating, scanning, transforming, shaping, and hardening code and interfaces to be ready for production usage.
- Encryption - Requiring encryption by default for storage, and in transport, using technology to ensure security is a default parameter for everything that is exposed publicly. Reducing the possibility of a breach, and minimizing the damage when one does occur.
Demonstrating how important the technological choices we make, and the architectural planning we discussed earlier is to the overall API governance conversations. The services, tooling, and applications we adopt will either contribute, or will not contribute to our governance practices. Potentially enforcing governance for all APIs as they move from development to production, in a way that teams cannot circumvent, and often times don’t even notice is occurring behind the scenes.
Augmenting the core technology, there are a number of orchestration practices we found that help quantify and enforce governance on the road from development to production. Dictating how code, artifacts, and other elements included as part of the API lifecycle move forward, evolve, or possibly get held back until specific requirements are met to meet wider governance criteria.
- Pipeline - CI/CD services, tooling, and mindset have introduced a pipeline workflow for many teams, standardizing the API delivery process as an executable, repeatable, measurable pipeline that can be realized across any team.
- Stages - The defining of clear stages that exist after development, but before production, requiring quality assurance, security reviews, compliance audits, and other relevant governance practices to be realized.
- Hooks - Well defined pre and post commit hooks for all service repositories, requiring that governance is applied throughout a service’s pipeline, and are default for all services, no matter which organization they emerge from.
- Devops - Pushing that all teams are competent, and skilled enough to execute on behalf of their services from beginning to end, owning and executing at every stage of the life cycle. Reducing the need for reliance on special teams, and eliminating bottlenecks.
- Logging - Identifying the logging capabilities of the entire stack for each service being delivered. Making sure logging is turned on for everything, and all logs are shipped to a central location for auditing, and when possible real time analysis and response.
Orchestration provides some clarity on the automation side of moving services from development to production, while also enforcing governance along the way. Allowing for an assembly line delivery of consistent services, and the iteration of each version, in alignment with the overall governance strategy. Reducing the chance for human error, and increasing the chance for consistent execution of the enterprise API strategy at scale across many different teams.
The Legal Department
Beyond the technology, the legal department should have a significant influence over APIs going from development to production. Providing a structured framework that can generally apply across all services easily, but also providing granular level control over the fine tuning of legal documents for specific services and use cases. With a handful of clear building blocks in use to help govern the delivery of APIs from the legal side of the equation.
- Terms of Services - Have universally applicable, and modular, as well as possibly machine readable, and human readable terms of service governing all services from a central location.
- Security Policy - Provide a comprehensive security policy that governs how services are secured, reflecting the technologies, checklists, tooling, and reviews that are in use by all team members, providing an overview for all stakeholders to understand.
- Licensing - Establish clear code, data, and interface licensing to be used across the entire API stack, allowing developers to properly license their services, as well as understand the provenance for the systems and services they depend on.
- Server Level Agreements - Have universally applicable, modular, as well as possibly machine readable, and human readable service level agreement (SLA) that can be applied across all services, measured, and reported upon as part of wider governance strategy.
- Scopes - Define and publish the OAuth access scopes as part of a formal set of legal guidance regarding what data is accessible via services, and the role users and developers play in managing scope that is defined by the platform.
The legal department will play an important role in governing APIs as they move from development to production, and there needs to be clear guidance for all developers regarding what is required. Similar to the technical, and business elements of delivering services, the legal components need to be easy to access, understand, and apply, but also make sure and protect the interests of everyone involved with the delivery and operation of enterprise services.
Making APIs Available to Consumers
The next step in the life cycle of properly governed APIs is making them available to consumer after they’ve been published to a production environment. The governing of APIs is not limited to the technical side of things, and this is where we begin understanding how to consistently deliver, measure, and understand the impact of API resources across the many consumers who are integrating the valuable resources into their applications. Shining a light on the business and politics of how digital assets are being put to use across the enterprise.
This portion of this governance research is intended to provide a basic list of the building blocks used by enterprise groups to help reduce friction when putting APIs to work, but also make sense of how consumers are using API resources, establishing a feedback loop that guide the road map for the future of the platform. Taking us back to the beginning of this research and informing how we should be targeting the development of new APIs, the evolution of existing services, and in many cases the deprecation and archiving of services. Ensuring governance goes well beyond the technical details, and making sure they are benefitting the platform, as well as consumers.
Making your APIs available to consumers requires doing a lot of research on who you are marketing them to, and positioning yourself to speak to an intended audience. Tailoring not just the design of your APIs, but the overall presentation, messaging, and even portal, documentation, and other building blocks to speak to a particular audience. For many API providers, APIs might be made available to multiple audience, in a variety of ways, based upon knowing their customers, and presenting exactly the resources they are needing to get a specific job done.
- Studies - Conducting regular stories about what internal, partner, and public user groups are using, and needing when it comes to developing applications, and integrating systems.
- Landscape - Establishing an understanding of the industry landscape for the area being targeted by services, and regularly tuning into and refining the understanding of what consumers are using across that landscape.
- B2B - Positioning the API to speak to a B2