This is the narrative from one of the last webinars I did for Postman oin my API-first e-commerce reference implementation Union Fashion. I always try to write what I am going to be saying furing a webinar so that it is loaded up in my brain in a natural way---think Neo learning Kung Fu before sparring with Morpheus. Anyways, here is the narrative, with accompanying video embedded below, helping share the narrative behind how I am building Union Fashion, trying to learn, grow, and expand how I use APIs out in the open, so that others can learn along the way.
Jeff Jones the VP of Engineering at Union Fashion has been working to get more organized about how they deliver applications using APIs at the company. In our last webinar episode we looked at how Union Fashion was building and testing APIs, moving their operations towards an API first way of delivering application infrastructure. The build and test planning session was all about ensuring Jeff could lay the foundation for organizational change at Union Fashion by making sure his teams had the proper planning, and everyone was in agreement on an API-first way of delivering APIs behind the web and mobile applications they were needing to run their business—investing in the following areas to help define how they build and test APIs before actually building them:
- Organizational - Making sure the team was well defined, including having dedicated workspaces for each API.
- APIs - We are making sure that common API definitions at the center of the conversation for each API being developed.
- Collections - Being very structured in how collections are define and used to power different stops along the API life cycle.
- Environments - We made sure secrets and some key value pairs were abstracted away from each of the API collections.
- Mocks - Mock servers are published from all APIs, using a specific collection, making each API being developed more tangible.
- Documentation - Documentation is also published for each API from the beginning making it more accessible and usable.
- Tests - Establish contract tests for each API, making sure that each API is properly tested to meet its design requirements.
- Feedback Loops - Establishing feedback loops around each API, ensuring there is proper feedback and communication.
This allows the team at Union Fashion to employ an API design first approach to developing APIs, defining, mocking, documenting, testing, and iterating upon each API before it is ever actually deployed. Allowing teams to be more efficient when it comes to how APIs are brought to life, while making sure APIs better meet the needs of stakeholders as early on as possible, including those who are building applications and other integrations using each potential API. The build and test approach Union Fashion has been developing is a great way to bring each API to life but eventually we will need to actually deploy each API, so that it can be put to work within applications and system integrations.
Deploying and Monitoring Union Fashion APIs
Using our build and test API first methodology we have established five OpenAPI definitions reflecting the core commerce services we will be needing, including products, baskets, orders, users, and search. There will be other APIs that will emerge along the way, but in this first sprintd we were able to define a robust build and test strategy for developing APIs in a consistent way without writing code, while actually producing the first version of five APIs we will need. Now we need to actually deploy the APIs we designed using a standardized, consistent, and scalable way for standing each API up, and making available to consumers using common cloud infrastructure.
To help lay the groundwork for our deploy and monitor strategy we got to work on defining a set of processes which developers can put to use when deploying an API into any stage of delivery, from development to staging to production. Laying the foundation for Union Fashion’s official deploy and monitors strategy, helping formalize how APIs are delivered across the organization. Making sure API infrastructure at Union Fashion all follows the healthiest and most widely agreed upon patterns for delivering consistent and reliable APIs.
Ensuring Each API is Versioned
Union Fashion employs Semantic versioning (https://semver.org/) for each API definition, ensuring that each API is evolved in a way that is fully documented, but also helps ensure backward compatibility, and easy reuse across the organization by making sure each version is available within the repository for each API being developed.
- API Version - Making sure to iterate the version of each API and derivative collection, managing change across APIs.
- Version Tag - Applying vision tagging to each of the collections associated with the API being moved forward.
- Change Log - Evaluate change log activity for each service, making sure you are in tune with what has been happening across an API.
- Major Version Sync - Ensure that latest version is synced to GitHub, keeping all changes made in Postman or on GitHub reflected in both locations.
Managing change for each of the APIs we are delivering as part of Union Fashion is critical to ensuring the reliability of each of the APIs we are making available to consumers. Having a plan for managing change rather than responding to it after it has already occurred.
Actual Deployment of Each API
With each of our APIs well defined using OpenAPI, with supporting documentation and testing to ensure the API operates as intended, we have reached the point in the evolution of our API resource where we need to actually bring the to life using common cloud infrastructure. Union Fashion currently has two blueprints for delivering APIs using AWS, providing a basic, and more robust “serverless” model for delivering API infrastructure in a consistent and repeatable way that is contract-driven and fully tested to ensure each API is living up to its intended contract.
- Basic NoSQL (AWS) - Providing a standardized and repeatable way to deploy basic NoSQL backed API resources.
- OpenAPI - Leveraging the OpenAPI contract to define, prepare, and build each API being deployed.
- Table - Pulling the table name from the schema available as part of the OpenAPI definition from the API.
- Publish - Publishing the OpenAPI definition to the AWS API Gateway, using it as the truth for deployment.
- Deploy - Actually deploying the API published to the AWS API Gateway into a specific stage of development.
- Serverless SQL (AWS) - Providing a standardized and repeatable way to deploy basic serverless SQL backed API resources.
- OpenAPI - Leveraging the OpenAPI contract to define, prepare, and build each API being deployed.
- Table - Pulling the table name from the schema available as part of the OpenAPI definition from the API.
- Layers - Pulling and deploying the AWS Lambda layers package needed to satisfy Node.js dependencies.
- Scripts - Dynamically generating JavaScript for each API method that is defined as part of the OpenAPI.
- Packages - Pulling, zipping, and publishing AWS Lambda deployment packages for each OpenAPI method.
- Functions - Deploying the package for each OpenAPI method and publishing individual AWS Lambda functions.
- Publish - Publishing the OpenAPI definition to the AWS API Gateway, using it as the truth for deployment.
- Deploy - Actually deploying the API published to the AWS API Gateway into a specific stage of development.
There are additional deployment patterns being developed, providing delivering more functional and algorithmic APIs, as well as to replicate these patterns across additional cloud platforms, including Azure and Google. Deployment collections can be executed incrementally or collectively running API deployment collections for each of the patterns listed, deploying APIs individually, or as part of a wider API life cycle process.
Management of Each API
Each API requires a minimal level of management ensuring that each one is properly secured, and requires an API key to access. Each API should have its own usage plan with the AWS API Gateway, and be configured with a rate limit for the consumption of each API, limiting what consumers can access per second, minute, day, or other metrics.
- Usage Plan - Publishing a usage plan with access limitations defined, governing each API being deployed.
- API Key - Generating an API key and associated it with the plan, allowing each API to be securely accessed.
Each API should have a default API usage plan with associated key, then each additional consumer should have their API keys associated with the APIs plan, keeping each individual API within its own definition of operation—defined by the API management layer. Ensuring that each API is properly protected from access by unknown actors, but also limits the amount any one consumer can access of each API resource, during a particular time frame.
Establishing Environments
Each API needs an environment to define each stage of its evolution, including a mock representation for each stage. Providing a place to store key / value pairs that are needed to define individual API as they move through each stage of development. Ensuring that there is a snapshot of the environment need to operate each version and stage of an API.
- Stage
- Name - Publishing of an environment for managing the deployment of the API, naming for the stage.
- BaseURL - Provide a key / value for the base_url , which will be used when deploying and testing of the deployed API.
- Mock
- Name - Publishing of an environment for managing the mock for thi API, naming for the stage.
- BaseURL - Provide a key / value for the base_url , which will be used when working with the mock.
The presence of an environment for each stage of an APIs development reflect the maturity and evolution of each API within its workspace. Providing the definition of how far along it has moved from from design to development to production. Providing a machine readable map of each stage of evolution, for each version of an API.
Monitoring API Contract and Performance
Each API should possess a monitor for each of its test collections. Making sure that contract and performance testing occurs on a regular basis. Taking the tests established as part of the build and test process cycle and running them against each API deployed as part of development, staging, and production environments. Ensuring that all APIs are regularly monitored to ensure they are living up to expectations and meeting the agreed upon service level agreement (SLA).
- Contract Testing
- Name - Apply name to the monitor matching it with the API and type of testing being implemented.
- Version - Make sure the monitor has the proper version tag applied associated it with current version.
- Environment - Using the proper environment stage for the API contract being tested and monitored.
- Schedule - Applying the required schedule for the monitoring of the API contract testing.
- Region - Monitoring the contract of each API from the required geographic regions.
- Performance Testing
- Name - Apply name to the monitor matching it with the API and type of testing being implemented.
- Version - Make sure the monitor has the proper version tag applied associated it with current version.
- Environment - Using the proper environment stage for the API performance being tested and monitored.
- Schedule - Applying the required schedule for the monitoring of the API performance testing.
- Region - Monitoring the performance of each API from the required geographic regions.
- Service Level Agreement - Establish benchmarks for contract and performance testing.
- Results - Ensure that the results of contract and performance testing are being stored in the environment.
- Publish - Publishing of contract and performance testing results to the GitHub repository for each API.
Additional test suites, and integration testing will be developed, but basic contract and performance testing is mandatory for all Union Fashion APIs being deployed. Making sure each API lives up to its contract, meets the level of service needed, and does this across every geographic region that matters in an ongoing way, maintaining a consistent level of service across all APIs, no matter what teams are delivering them.
Mocking of Each API Being Deployed
There should be a mock server for each stage of API evolution, ensuring that there is a mock representation for use during testing, as well as safe and easy playing with by consumers as they are learning what an API does, without having to be exposed to live data—providing a virtualized instance of each API for quick access and use.
- Deploy - Establish a mock representation of the current stage, providing a mocked server for this version.
- Share - Share the URL of mock server with rest of team, making sure it is know that there is a separate mock.
- Environment - Add the mock URL and keys to the environment, ensuring the mocked version can be accessed.
Mock APIs reduce the overhead on development, staging, and production environments, while helping alleviate anxiety with consumers when it comes to working with live data, or event development and staging data that might be needed for testing. Ensuring there is always a sandbox present for API consumers and making it easier for consumers to play around with each API.
Ensuring There Is Always Up to Date Documentation
There should always be up to date documentation published for each API, making sure the collection behind documentation has proper name, description, and examples of the value each API request delivers. With each stage of an APIs development, the following elements should be addressed as soon as any stage of an API is deployed.
- Publish - Publish documentation for each service being developed, making sure the document is up to date.
- Environment - Make sure there is an environment available for each stage of an APIs development.
- Share - Share the URL of documentation with team members, making sure everyone has the link they need.
The publishing of API documentation for each stage of an APIs development should be a deliberate step at the end of the deployment cycle. Making sure that documentation represents the current state of any fully deployed and monitored API, keeping the visual representation for each API is always up to date and providing the most value possible.
Establish Feedback Loops for Each API
There should be an independent feedback loop in place for each API being deployed, allowing for stakeholders in the deliver of each API can comment on the API contract, as well as the derivative collections used behind documentation, mock servers, and tests. Ensuring API consumers have a channel to provide feedback, that can be considered as part of the evolution of each API.
- API Comments - Tune into comments available for each API, helping keep discussions self-contained for each API being developed.
- Collection Comments - Tune into comments available for each collection, helping keep discussions self-contained for each API derivative.
- Issues - Tune into GitHub issues for each service, leveraging the repository issues as public and private communication cycles.
The feedback loop for each API should be used to stay in tune with what is happening as each API is being deployed, ensuring that all the concerns and feedback of stakeholders have been considered before an API is deployed. It should also be regularly visited upon a well established schedule, making sure all feedback has been considered, and if possible, the comments and issues should be automatically tracked for evaluation of feedback lop activity and engagement across teams.
A Standardized and Repeatable Way to Deliver API Infrastructure
We now have a standardized way of deploying APIs at Union Fashion. This process will evolve with each API that is delivered using the process, but it also provides us with a base approach for delivering the infrastructure we need to operate online as a business. Combined with our API-First approach to building and testing APIs, our approach to deploying and monitoring APIs will make sure the API contracts defined are delivered as intended using common cloud infrastructure. Helping stabilize the infrastructure Union Fashion uses behind its web and mobile applications, and system integrations.
This concludes the deploy and monitor portion of our API develop life cycles. This process will help us move APIs from idea to production more efficiently, and will set that stage for us to bring our API to market, no matter whether they are strictly for internal use, access by trusted partners, or for the general public. This process is meant to show what is possible when it comes to quantifying how we deploy and monitor our API infrastructure, but can and should be evolved to consider the needs of future APIs that will be delivered. Continuing to push how we move APIs forward by making sure each individual step of the process is as well through out and planned as we possibly can. Allowing us to be more flexible, efficient, and scalable in how we deliver our API-driven application infrastructure.
