Establishing an API-First Reference Implementation
06 Apr 2020
I do a lot of API blah blah blah’ing about abstract technical concepts. Sometimes I am able to craft a coherent narratives around some complex technology goings on, but most of the time I am just practicing. Workshopping different concepts until I find one that will make an impact on the way folks see APIs. One of the challenges I face in my storytelling is that I operate too far into the abstract, and not making the rubber meet the road enough. Another challenger I face is going too far down the rabbit hole with a particular companies implementation, which usually turns down the volume significantly on my storytelling, because most companies, organizations, institutions, and government agencies aren’t equipped to be 100% transparent about what they are doing. After a decade of storytelling exploration I find that operating somewhere in between the abstract and the real world is the best place to be, resulting in me desiring a reference implementation that I could use as an anchor for my storytelling, helping keep me grounded when it comes to how I talk about APIs.
Welcome Union Fashion
One of my co-workers at Postman had created a fictional company called Union Fashion when he started working on our solutions engineering team, but hadn’t put much more work into the project since. When I heard about it sounded exactly like what I was looking for. An e-commerce reference implementation that a wide audience could relate with, providing us with a model API implementation that we could use across webinars, workshops, and other storytelling channels. I’m big on building on the work of others, so I adopted Union Fashion, and I am working to define the fictional company as an API-First approach to operating a common real world business.
Repo) (Docs) - Defining all of the products that Union Fashion offers.
Each API will get fully fleshed out as part of an API-First design effort, where I am guessing there will be other APIs added along the way. The goal of this is to come out of the other end with some fully functioning APIs that I can actually use as part of selling products online, which I can simultaneously also use for workshops and training on how APIs can be done well using a consistent API life cycle and set of governance practices.
Planning For The Reference Implementation Out In the Open
I was given an “enterprise webinar” to do back in February, shortly after I learned about Union Fashion. Before I thought much about it I declared that I would plan the Union Fashion API out in the open on GitHub using Postman, and make it center of my webinar, and eventually turn it into a series of webinars showing how to deliver APIs using an API-first approach. Proposing the following outline when it came to planning for how to deliver APIs for Union Fashion, shaping each part of the webinar series alongside building our reference e-commerce implementation.
- Initial Planning (Webinar Video)- Getting the ball rolling with an API-first approach to development.
- Build & Test Planning (Webnar Video) - In planning phase currently, determining the process for how we will be building and testing our infrastructure.
- Deploy & Monitor Planning - In planning phase currently, shaping how each team will be delivering their API infrastructure across the organization.
- GTM & Sustainment -In planning phase currently, and helping make sure we have a formal approach to how we are going living with our infrastructure.
- Developer Portal - Publishing a portal for Union Fashion, providing a dedicated area for developers to find all the essential building blocks of an API platform.
I added the developer portal portion just now. I haven’t scheduled that one, but Deploy & Monitor and GTM & Sustainment are both scheduled for end of April and beginning of May. With the initial planning and Build & Test having already occurred—with videos published to YouTube. My intention is to keep planning Union Fashion out in the open, while telling the story of it all via a series of webinars, and posts here on this blog, and eventually on the Postman blog. Right now I am just looking to get a handle on the overall tone for the project, and hammer out some of the technical details of my implementation. Since it is a reference implementation I am particularly eager to keep it simple, standardized, and reflecting the best of what I have to offer—while still being perpetually under construction.
The Infrastructure Behind Union Fashion
I am looking for Union Fashion to be a real world implementation that others can actually implement using common infrastructure. I am beginning with a mostly AWS focused stack to help me get the Union Fashion APIs stood up, leveraging Postman and GitHub as the backbone of the implementation, while also relying on a handful of AWS APIs to make the Union Fashion API factory operate.
- Postman (API) (Docs) - We use Postman as our API life cycle manager, helping deliver on a variety of stops along the life cycle for each API, and orchestrate how everything works using APIs.
- GitHub (API) (Docs) - We use GitHub as the underlying workspace for each API, working in sync with Postman workspaces to make the core definition of each API available across the entire life cycle.
- AWS API Gateway (API) (Docs) - All internal, partner, and public APIs are published using the AWS API Gateway, providing a content management layer across all APIs being delivered.
- AWS DynamoDB (API) (Docs) - For simpler data APIs we are using AWS DynamoDB for the persistent data storage behind APIs, allowing for simple NoSQL data storage for each individual APIs.
- AWS RDS Aurora (API) (Docs) - For more complex data APIs we are using AWS RDS Aurora for persistent storage behind APIs, allowing for more complex relational data storage for each API.
- AWS Lambda (API) (Docs) - For the serverless compute layer for many APIs we are using Lambda as the scalable power behind each API, allowing us to scale APIs at the most atopic level.
- AWS S3 (API) (Docs) - Using AWS S3 for the storage of all images, videos, documents, and other heavy objects that are made available via Union Fashion APIs, giving each API it's own object store.
- AWS Cloudtrail (API) (Docs) - We are using AWS CloudTrail for the logging layer behind each API, gathering the exhaust of the database, storage, compute, and gateway layers of the APIs.
- AWS Identity & Access Management (IAM) (API) (Docs) - AWS IAM is used to provide access to APIs for all the underlying services they are using across the AWS platform.
- CloudFlare (API) (Docs) - We are using CloudFlare as the DNS and certificate provider behind Union Fashion, providing addressing and encryption for all of the APIs being delivered.
- Twitter (API) (Docs) - We are using the Twitter API for updates, support, and other communication with the Union Square community.
I almost have the entire AWS stack up and running, minus some Lambda and DNS work, and once I am done I will be extending this to use Azure. Expanding the Union Fashion approach to delivering APIs to include a multi-cloud view of the landscape, allowing us to deploy APIs in two separate clouds. Something we’ll expand to Google and beyond down the road. Expanding the infrastructure we are using to deploy and operate APIs, while always keeping a strong, well documented list of API-driven infrastructure to help deliver APIs throughout the entire API life cycle.
A Well Defined API Life Cycle for Union Fashion
Union Fashion was my opportunity to move my very abstract narrative around the API life cycle from a bulleted list to something you can actually apply on the ground. Using the APIs of the infrastructure providers listed above I actually crafted a single API life cycle collection that would help me not just quantify the API life cycle for Union Fashion, it would let me actually execute upon it. Resulting in the following outline for the Union Fashion API life cycle, helping guide each API forward in a standardized way.
- Define (Docs) - Providing guidelines for how APIs should be defined.
- Design (Docs) - Training and guidance about how to design APIs.
- Mocking (Docs) - The ability to mock an API to assist in it's delivery.
- Documentation (Docs) - Always having consistent up to date documentation.
- Database (NoSQL) (Docs) - Ensuring there is always persistent data storage.
- Database (SQL) (Docs) - Ensuring there is always persistent data storage.
- Storage (Docs) - Providing a place to put objects as part of API operations.
- Compute (Docs) - Having the appropriate amount of compute behind each API.
- Pipeline (Docs) - Making the delivery of code behind each API repeatable.
- Deployment (Docs) - Making sure that APIs are deployed in repeatable ways.
- Management (Docs) - Ensuring that every API is being managed consistently.
- Logging (Docs) - Establishing a centralized logging strategy across APIs.
- Encryption (Docs) - Always making encryption the default response across APIs.
- DNS (Docs) - Having a strategy for how DNS is handled across groups and APIs.
- Portal (Docs) - Ensuring there is always a doorway to get at all APIs.
- Testing (Docs) - Providing a consistent approach to testing all APIs.
- Security (Docs) - Scanning and auditing the security landscape for each API.
- Monitoring (Docs) - Having monitors on a schedule ensuring the quality of APIs.
- Support (Docs) - Requiring there be support for every API being operated.
- Communications (Docs) - Having consistent communication in place for APIs.
- Analytics (Docs) - Establishing a layer for understanding what is happening.
- Deprecation (Docs) - Making sure there is a plan for how all APIs will be sunset.
It is all a work in progress, and many of the requests in the collection are just relying on hacked together scripts, but it demonstrates the actual API life cycle I am using to design, deploy, and sustain APIs as part of Union Fashion in a well documented, executable, and machine readable way. I am actually using each of these steps to define, design, and move forward each API across a consistent life cycle. There are a lot of steps missing, and lots of edges to grind down and make smooth, but it goes further than any other API life cycle project I’ve developed to date, getting me closer to a fully automated series of API life cycle outcomes.
Governing The API Life Cycle For APIs at Union Fashion
Complimenting the guard rails that a standardized API life cycle collection bring to the table I wanted to be able to similarly govern how each of the APIs were moving along across the API life cycle. Not just understanding the design of each of the APIs, but actually understand important other stops required for consistently delivering APIs at scale across an organization. So I got to work creating a Postman collection that would help me govern each API I was developing, helping keep an eye on the following areas:
- Definition (Docs) - The guidelines in place when it comes to defining each API.
- Design (Docs) - The guidelines in place for helping design better APIs.
- Mocks (Docs) - Looking at how mocking is used across the API life cycle.
- Development (Docs)- Helping standardize how APIs are being developed.
- Production (Docs) - Standardizing how APIs are made available in live environment.
- Management (Docs) - Using a consistent way to manage access and usage of all APIs.
- Testing (Docs) - Establishing a common approach to delivering testing across APIs
- Monitoring (Docs) - Making sure there are monitors for some of the key collections.
- Support (Docs) - Being consistent in how each of the APIs are being supported.
- Licensing (Docs) - Looking at the licensing of each API and its support resources.
- Reporting (Docs) - Reporting on governance for each API, providing an overview.
My work on this collection began with defining not just the API, but defining all of the artifacts and infrastructure that would be used across the governance process. Then I started with API design, which is the stop along the API life cycle most people associate with API governance, but I wanted to quickly move on to also service other stops along the API life cycle. There are definitely many more stops I can service, but the API governance collection is a pretty good start when it comes demonstrating how Postman, collections, and environments can be used to test more than just the surface area of the API—you can also govern the entire API life cycle.
A Dance Between APIs, Collections, Environments, and Workspaces
My goal with Union Fashion is to push the boundaries of how Postman is used. Helping folks realize it can be used to do more than just making calls to APIs or testing APIs, and that it can be applied as a complete API life cycle solution. What makes it all possible is a dance that is occuring between the central OpenAPI definition as well as the collections and environments that exist within workspaces for each API. There is a lot more work to be done to polish the rough edges and make it easier for new users to on-board with these concepts but it is a damn good start. It is the most polished version of what I see in my head when it comes to the API life cycle and governance, but have only been just talking about. I am hoping eventually this will become a dance choreography that others can follow when it comes to crafting their own approach to doing APIs. Going well beyond what I am doing here.
An Exercise In Self-Service API-First Training and Storytelling
I feel like Union Fashion is allowing me to anchor my vision of how to realize the API life cycle and governance, making it more tangible. However, one of my biggest challenges here is doing all of this out in the open, and keep everything I am doing accessible to the widest possible audience. My goal is to make Union Fashion be a self-service reference implementation that anyone can reverse engineer and apply in their world. I am not saying that my approach to the life cycle and governance will work for every organization, but I am hoping it will provide a new way of thinking about your API life cycle and governance through a series of self-service material, and storytelling via blog posts, webinars, videos, and other formats, with the following currently in play.
- Overview - A quick video walk through of the Union Fashion project.
- Life Cycle - A quick video walk through of the Union Fashion API life cycle.
- Governance - A quick video walk through of the Union Fashion API life cycle.
I will be re-recording this screen capture videos as I move each area forward. I am also hopeful I will get better at recording them in the future, as I am pretty stiff at the moment. I am looking to develop more of a flow between the actual work I am doing on Union Fashion APIs, while also telling the story of how I did it. Evolving the Union Fashion APIs, how I deliver APIs, as well as how I tell stories and help others learn about Union Fashion and doing APIs in a consistent way using Postman, GitHub, and a variety of other services.
Why Does This Even Matter in the First Place?
It is always important for me to step back and look at my work through the lens of the average person who may not be aware of why APIs matter in the first place. There is a lot here to consider when it comes to shaping the future of your own API operations, and it always helps to step back and sell folks on why APIs matter in the first place, and why doing APIs in a consistent way also counts. Union Fashion is doing APIs in order to realize the following outcomes that will help the (fictitious) company better meet its business objectives as a fashion provider in the e-commerce space.
- Allow for more easier publishing of data, content, and algorithms to known web and mobile applications.
- Ensure that data content, and algorithms are immediately available for publishing to unknown applications.
- More easily integrate all data, content, and algorithms into external 3rd party and partner systems.
- Reduce overall overhead and costs involved with delivering new web, mobile, device, and network applications.
- Increase the overall quality and velocity of new features and functionality being delivered across applications.
There are many more benefits of doing APIs, and going API first, but these reflect the areas can will make a significant change in how businesses, organizations, institutions, and government agencies do what they do. Helping them more effectively utilize Internet connected applications, and iterate upon the technology that they have in place. Helping be more consistent and efficient in how API infrastructure, and the applications that rely on it, ensuring that technology meets the needs of the organizations putting them to work.
An API-First E-Commerce Reference Implementation
I am looking to come out of this work with a real world API implementation that I can reference across my storytelling. I am looking to establish an open source set of blueprints that anyone can pick up, learn from, reverse engineer and apply the parts and pieces that are relevant to them in their world. I don’t expect Union Fashion to be perfect, but I do expect that it should be thoughtful, observable, and teachable. Helping show how to do APIs right and wrong, and learn from our mistakes out in the open. Working real hard to make sure Union Fashion is accessible to the API curious all the way to the API expert, and helps provide a solid set of blueprints that others can follow.
Union Fashion is just getting started and is being planned out in the open on GitHub. If you see a mistake, something incomplete, or a feature you’d like to see covered, feel free to submit a GitHub issue for the overall project, or for a specific API. It is all published to GitHub, and I am using GitHub issues as the task list and road map for the project. With every step forward I will be stepping back to better understand how I can document, and tell stories about what is happening on the blog (once I have one), and via video walk-throughs and webinars which will all be published to YouTube. I am hoping to make it a familiar reference implementation that the community can learn from and contribute to, helping push forward and stabilize how we talk about and deliver APIs across our organizations.