The API Evangelist Blog
This blog represents the thoughts I have while I'm research the world of APIs. I share what I'm working each week, and publish daily insights on a wide range of topics from design to depcration, and spanning the technology, business, and politics of APIs. All of this runs on Github, so if you see a mistake, you can either fix by submitting a pull request, or let me know by submitting a Github issue for the repository.
I am working on delivering a handful of new APIs, which I will be turning into products. I will be prototyping, developing, and operating them in production environments for myself, and for a handful of customers. To help guide my workflow, I’ve crafted this API lifecycle definition to help direct my efforts in an ongoing lifecycle approach.
Define - Define the problem and solution in a human and machine readable way.
- Repository - Establish a Github repository.
- README - Craft a README for the repository.
- Title - Provide title for the service.
- Description - Provide concise description for the service.
- Goals - Establish goals for the service.
- Schema - Organize loose, and JSON schema for the service.
- OpenAPI - Establish an OpenAPI for the service.
- Assertions - Craft a set of assertions for the service.
- Team - Define the team behind the service.
Design - Establish a base set of design practices for the service.
- Versioning - Determine how the code, schema, and the API be versioned.
- Base Path - Set the base path for the service.
- Path(s) - Define a set of resource paths for the service.
- Verb(s) - Define which HTTP, and other verbs will be used for the service.
- Parameters - Define a list of query parameters in use to work with service.
- Headers - Define the HTTP Headers that will be used to work with the service.
- Response(s) - Provide a resulting message and associated schema definition for the service.
- Media Types - Define whether service will return CSV, JSON, and / or XML responses.
- Status Codes - Define the available status code for each responses.
- Pagination - Define how pagination will be handled for requests and responses.
- Sorting - Define how sorting will be handled for requests and responses.
Database - Establish the base for how data will be managed behind the service.
- Platform - Define which platform is in use to manage the database.
- Schema - Establish the schema used for database based upon definition provided.
- Location - Define where the database is located that supports services.
- Logs - Define where the database logs are located that support services.
- Backup - Define the database backup process and location for service.
- Encryption - Define the encryption layer for the service database.
Storage - Establish how all objects will be stored for the service.
- Platform - Define which platform is used to store objects.
- Location - Define where objects are stored behind the service.
- Access - Quantify how objects access is provided behind the service.
- Backup - Define the backup process for objects behind the service.
- Encryption - Define the encryption layer for stored objects.
DNS - Establish the DNS layer for this service.
- Platform - Define which platform is used to operate DNS.
- Prototype - Provide host for prototype of service.
- Mock - Provide host for mock of service.
- Development - Provide host for development version of service.
- Production - Provide host for production version of service.
- Portal - Provide host for the portal for this service.
- Encryption - Define the encryption layer for service in transport.
Mocking - Provide a mock representation of this service.
- Paths - Providing virtualized paths for the API driving service.
- Data - Providing synthesized data behind each API response for service.
Deployment - Define the deployment scaffolding for this service.
- Platform - Define the platform used to deploy this service.
- Framework - Define the code framework used to deploy service.
- Gateway - Define the gateway used to deploy service.
- Function - Define the function(s) used to deploy service.
- Containers - Define the container used to deploy service.
- Pipeline - Define the pipeline in place to build and deploys service.
- Location - Define where the service is deployed to.
Orchestration - Define how the service will be orchestrated.
- Build - Define the build process for this service.
- Hooks - Detail the pre and post commit hooks in use for this service.
- Jobs - Define the jobs being executed as part of this service operations.
- Events - Define the events that are in play to help operate this service.
- Schedule - Details of the schedules used to orchestrate this service.
Dependencies - Providing details of the dependencies that exist for this service.
- Service - Details of other services this service depends upon.
- Software - Details of other software this service depends upon.
- People - Details of other people this service depends upon.
Authentication - Details regarding authentication in use for this service.
- Type - Define whether this service uses Basic Auth, API Keys, JWT, or OAuth for authentication.
- Overview - Provide a location of the page that delivers an overview of this services authentication.
Management - Define the management layer for this service’s API.
- Platform - Defining the platform used for the API management layer.
- Administration - Provide a location for administrating the management layer.
- Signup - Provide a location for users to signup for access to this service.
- Login - Provide a location for users to login to access to this service.
- Account - Provide a location for users to access a dashboard for this service.
- Applications - Provide the location of applications that are approved to use service.
Logging - Define the logging layer for supporting this service.
- Database - Define the logging for the database layer.
- API - Define the logging for the API access layer.
- DNS - Define the logging for the DNS layer.
- Shipping - Define how longs are shipped or centralized for auditing.
Monetization - Define the costs associated with the delivery of this service.
- Acquisition - Provide costs associated with acquisition of resources behind service.
- Development - Provide costs associated with the development of this service.
- Operation - Provide costs associated with the operation of this service.
- Value - Provide a description of the value delivered by this service.
Plans - Define the operational plan for the business of this service.
- Tiers - Define the tiers of access in place to support this service.
- Elements - Define the elements of access for each tier for this service.
- Paths - Define which API paths are available as part of each tier or service.
- Metrics - Provide a list of metrics being used to measure service access.
- Timeframes - Define the timeframes in use to measure access to this service.
- Limits - Define what limitations and constraints are in place for this service.
- Pricing - Define the monetary value in place to define the price for this service.
Portal - Define the public or private portal in use to present this service.
- Hosting - Provide details on where this service portal is hosted.
- Template - Define which graphical UI and brand template is in use for this portal.
- Analytics - Define which analytics package is used to measure traffic for portal.
Documentation - Provide documentation for this service within portal.
- Overview - Publish concise over for this service’s documentation.
- Paths - Publish an interactive list of API paths available for service.
- Examples - Provide as many examples of API requests in a variety of languages.
- Definitions - Publish a list of schema definitions in use by this service.
- Errors - Provide a list of available errors users will encounter for this service.
Getting Started - Provide a getting started page for this service within portal.
- Overview - Provide an introduction to the getting started process for this service.
- Signup - Provide a link to where users can signup for this service.
- Authentication - Provide a link to the authentication overview for this service.
- Documentation - Provide a link to the documentation for this service.
- SDKs - Provide a link to where users can find SDKs and code libraries for this service.
- FAQ - Provide a link to to the frequently asked questions for this service.
- Support - Provide a link to where users can get support for this service.
SDKs - Providing code samples, libraries, or complete software development kits (SDKs).
- PHP - Provide a PHP SDK.
- Python - Provide a Python SDK.
- Ruby - Provide a Ruby SDK.
- Go - Provide a Go SDK.
- Java - Provide a Java SDK.
- C# - Provide a C# SDK.
- Node.js - Provide a Node.js SDK.
FAQ - Publish a list of the frequently asked questions (FAQ) for this service.
- Categories - Break all questions down by logical categories.
- Questions - Publish a list of questions with answers within each category.
- Ask Question - Provide a form for users to ask a new question.
Road Map - Provide a road map for the future of this service.
- Private - Publish a private, internal version of entries for the road map.
- Public - Publish a publicly available version of entries for the road map.
- Suggest - Provide a mechanism for users to make suggestions for the road map.
Issues - Provide a list of currently known issues for this service.
- Entries - Publish a list of all known issues currently outstanding.
- Report - Provide a mechanism for users to report issues.
Change Log - Providing a historical list of what has changed for this service.
- Outline - Publish a list of all road map and issue entries that have been satisfied for this service.
Communication - Establish a communication strategy for this service.
- Blog - Provide a simple blog and update mechanism for this service.
- Twitter - Provide the Twitter handle that is used as part of this service.
- Github - Provide the Github account or organization behind this service.
- Internal - Provide a location where internal communication is available.
- External - Provide a location where public communication is available.
Support - Establish the support apparatus in place for this service.
- Email - Define the email account used to support this service.
- Issues - Provide a URL to the repository issues to support this service.
- Tickets - Provide a URL to the ticketing system used to support this service.
Licensing - Provide a set of licensing in place for this service.
- Server - Define how all backend server code is licensed for this service.
- Data - Define how all data and schema is licensed for this service.
- API - Define how the API definition is licensed for this service.
- SDK - Define how all client code is licensed for this service.
Legal - Provide a set of legal documents guiding the service.
- Terms of Service - Provide a terms of service for this service to operate within.
- Service Level Agreement - Provide a service level agreement (SLA) for this service to operate within.
Monitoring - Defining the uptime monitoring for this service.
- Monitors - Establish the monitors for this service.
- Status - Provide real time details of monitor activity.
Testing - Defining the testing for this service.
- Assertions - Provide details of the assertions being tested for.
- Results - Provide real time details of testing activity.
Performance - Defining the performance monitoring for this service.
- Tests - Provide details of the performance testing in place for this service.
- Results - Provide real time details of performance testing activity.
Security - Defining the security practices in place for this service.
- Overview - Provide the URL of the security practices overview page.
- Policies - Define the security, IAM, and other policies are in place for this service.
- Tests - Define the security tests that are conducted for this service.
- Results - Provide real time details of security testing activity.
Discovery - Defining the discovery aspects for this service.
- API Discovery - Publish an API Discovery (APIs.json) index for the project.
- OpenAPI - Provide URL for all OpenAPI definitions and index in API discovery index.
- Postman Collection - Provide URL for all Postman Collections and index in API discovery index.
Analysis - Define the analysis in play for this service.
- Traffic - Document traffic to the service portal.
- Usage - Document usage of APIs for the service.
- Activity - Document other activity around the service.
- SLA - Document whether the SLA was met for this service.
Stages - Define the stages that are applied to this lifecycle outline.
- Prototype - When a prototype of this service is being developed.
- Development - When a production instance of service is being developed.
- Production - When a production instance of service is being operated.
Maintenance - Define the maintenance cycles applied to this lifecycle outline.
- Daily - Provide a version of this outline that should be considered daily.
- Weekly - Provide a version of this outline that should be considered weekly.
- Monthly - Provide a version of this outline that should be considered monthly.
- Releases - Provide a version of this outline that should be considered for each release.
- Governance - Provide an outline of how this outline is measured, reported, and evolved.
This outline gets executed differently depending on the service stage and maintenance cycle being executed. It is meant to provide a master checklist to consider from day one, and every other time this service is being versioned, maintained, and considered as part of my overall operations. Providing a living checklist, and scorecard rubric for how well this service is doing, depending on stage and maintenance dimensions.
Ultimately the API Discovery (APIs.json) document is the heartbeat for this service checklist, which resides in the root of the Github repository will provide the machine readable index for reporting and governance, while also driving the human readable interface that is accessible via the service portal, which is driven by Jekyll running in Github Pages. Which is something I will publish more about next, as part of a working portal for one of the first services coming off the assembly line.
You don’t usually find me writing about API acquisitions unless I have a relationship with the company, or there are other interesting aspects of the acquisition that makes it noteworthy. This acquisition of Elastic Beam by Ping Identity has a little of both for me, as I’ve been working with the Elastic Beam team for over a year now, and I’ve been interested in what Ping Identity is up to because of some research I am doing around open banking in the UK, and the concept of an industry level API identity and access management, as well as API management layer. All of which makes for an interesting enough mix for me to want to quantify here on the blog and load up in my brain, and share with my readers.
From the press release, “Ping Identity, the leader in Identity Defined Security, today announced the acquisition of API cybersecurity provider Elastic Beam and the launch of PingIntelligence for APIs.” Which I think reflects some of the evolution of API security I’ve been seeing in the space, moving being just API management, and also being about security from the outside-in. The newly combined security solution, PingIntelligence for APIs, focuses in on automated API discovery, threat detection & blocking, API deception & honeypot, traffic visibility & reporting, and self-learning–merging the IAM, API management, and API security realms for me, into a single approach to addressing security that is focused on the world of APIs.
While I find this an interesting intersection for the world of APIs in general, where I’m really intrigued by the potential is when it comes to the pioneering open banking API efforts coming out of the UK, and the role Ping Identity has played. “Ping’s IAM solution suite, the Ping Identity Platform, will provide the hub for Open Banking, where all UK banks and financial services organizations, and third-party providers (TPPs) wanting to participate in the open banking ecosystem, will need to go through an enrollment and verification process before becoming trusted identities stored in a central Ping repository.” Which provides an industry level API management blueprint I think is worth tuning into.
Back in March, I wrote about the potential of the identity, access management, API management, and directory for open banking in the UK to be a blueprint for an industry level approach to securing APIs in an observable way. Where all the actors in an API ecosystem have to be registered and accessible in a transparent way through the neutral 3rd party directory, before they can provide or access APIs. In this case it is banking APIs, but the model could apply to any regulated industry, including the world of social media which I wrote about a couple months back as well after the Cambridge Analytics / Facebook shitshow. Bringing API management and security out into the open, making it more observable and accountable, which is the way it should be in my opinion–otherwise we are going to keep seeing the same games being played we’ve seen with high profile breaches like Equifax, and API management lapses like we see at Facebook.
This is why I find the Ping Identity acquisition of ElasticBeam interesting and noteworthy. The acquisition reflect the evolving world of API security, but also has real world applications as part of important models for how we need to be conducting API operations at scale. ElasticBeam is a partner of mine, and I’ve been talking with them and the Ping Identity team since the acquisition. I’ll keep talking with them about their road map, and I’ll keep understanding how they apply to the world of API management and security. I feel the acquisition reflects the movement in API security I’ve been wanting to see for a while, moving us beyond just authentication and API management, looking at API security through an external lens, exploring the potential of machine learning, but also not leaving everything we’ve learned so far behind.
There are a lot of people making money off of the acquisition, organization, and providing access to data in our digital world. While I quietly tune into what the data monetization trends are, I am also actively looking for interesting approaches to generating revenue from data, but specifically with an eye on revenue sharing opportunities for the owners or stewards of that data. You know as opposed to just the exploitation of people’s data, and generating of revenue without them knowing, or including them in the conversation. To help counteract this negative aspect of the data economy, I’m always looking to highlight (potentially) more positive outcomes when it comes to making money from data.
I was recently profiling the API of the people intelligence platform LotaData, and I came across their data monetization program, which provides an interesting look at how platforms can help data stewards generate revenue from data, but in a way that makes it accessible to individuals looking to monetize their own data as well. “LotaData’s AI platform transforms raw location signals into ‘People Intelligence’ for monetization, usually based upon the follow key attributes: latitude, longitude, timestamp, deviceID, and accuracy.”
Representing activity at a location and/or point in time, allowing LotaData’s to understand what is happening at specific places at scale, and develop meaningful insights and behavioral segments that other companies and government agencies will want to buy into. Some of the examples they provide are:
- Commuting daily on CalTrain from Palo Alto to San Francisco
- Mid-week date night at Nopa on the way back from work
- Sweating it out at Soul Cycle on Saturday mornings
- Taking the dog out for a walk on Sunday afternoons
- Season ticket holder for Warriors games at the Oakland Arena
LotaData’s location-based insights and segments are entirely inferred from raw location signals, emphasizing that they do not access or collect any personally identifiable information (PII) from mobile phones–stating that they “do not and never will collect PII such as name, email, phone number, date of birth, national identifier, credit cards, or other sensitive information”. Essentially walking on the light side of the whole data acquisition and monetization game, and playing the honest card when it comes to the data economy.
When it comes to the monetization of data, LotaData enables marketers, brands, city governments and enterprise businesses to purchase location-based insights–providing an extensive network of data buyers who are ready to purchase the insights generated from this type of data. Then the revenue generated from the sale of an insight is split proportionately and shared with the app developers who contributed their app data. With the SDK agreement with LotaData governing the payment terms, conditions and schedule for sharing revenue. However, if you are unable to integrated LotaData’s SDK in a mobile app for any reason, they can offer you alternative ways to share and monetize your location data:<p></p>
- Geo-Context API - The Geo-Context API is a simple script that you can embed in your mobile web sites and web apps. The script collects location data with explicit notice and permission obtained from end users.
- Bulk Data Transfer - Customers that are proficient in collecting location signals from their mobile apps, websites or other services, can easily upload their historical location archives to LotaData’s cloud for analyzing, inferring and monetizing mobile user segments. The data can be transferred to LotaData by configuring the appropriate access policies for AWS S3 buckets.
- Integration - LotaData can integrate with CRM and in-house data warehouse systems to ingest custom datasets or usage logs for deep analysis, enrichment and segmentation. Questions?
Providing a pretty compelling model for data providers to monetize their location based data. It is something I’ll be exploring more regarding how individuals can aggregate their own personal or professional data, as well as take advantage of the geo context API, bulk data transfer, or other integration opportunities. I have no idea how much money an individual or company could make from publishing data to LotaData, but the model provides an interesting approach that I think is worth exploring. It would be interesting to run a 30 to 90 test of tracking all of my location data, uploading it to LotaData, and then sharing the revenue details about what I can make with through a single provider like LotaData, as well as explore other potential providers so that you could sell your location data multiple times.
In a world where our data is the new oil, I’m interested in any way that I can help level the playing field, and seeing how we can put more control back into the device owners hands. Allowing mobile phone, wearable, drone, automobile, and other connected device owners to aggregate and monetize their own data in a personal or professional capacity. Helping us all better understand the value of our own bits, and potentially generating some extra cash from its existence. I don’t think any of us are going to get rich doing this, but if we can put a little cash back in our own pockets, and limit the exploitation of our bits by other companies and device manufacturers, it might change the game to be a little more in our favor.
It is funny work with companies, organizations, institutions, and government agencies of all shapes and sizes, and learn all the weird practices they have, and the strange belief systems they’ve established. One day I will be talking to a 3 person startup, the next day I’ll be talking with a large bank, and after that I’ll be working with a group at a massive government agency. I have to be mindful of my time, make sure I’m meeting my mission, having an impact, as well as paying my bills, but for the most part I don’t have any entrenched rules about who I will talk to, or who I will share my knowledge with.
One thing I chuckle at regularly is when I come across large organizations who will gladly talk with me, and tap my knowledge, but won’t work with some of the startups I work with, or the conferences I produce, because they are “too small”. They can’t waste their time working with small startups because it won’t bring the scope of revenue they need to justify the relationships, but they’ll gladly talk to me and welcome the exposure and knowledge I might bring. Sometimes I feel like telling organizations, “sorry you are just to large to work with, you are almost guaranteed to fail at this whole API thing, why should I bother?” I think I’ll say it sometimes jokingly, but not really interested in truly being a dick at that level.
Most large organizations can’t figure out how to work with me in any long term anyway, because they are too bureaucratic and slow moving. Other large organizations have no problem figuring out how to get me past legal, and getting me paid, but some just can’t figure it out. I had one large enterprise group who follows my work, wanted to get me in really badly, but their on-boarding team needed proof that I was the API Evangelist going back every year since 2010, a letter from client, tax returns, or other proof that I was who I say I was–just so I could share my knowledge with them. Um, ok? You really are going to put up so many barriers to people coming into your organization and sharing knowledge? I’m guessing you aren’t going be very good at this whole API thing, with these types of barriers in the way.
I know I can’t change the way large organization behave, but know I can influence their behavior. I’ve done it before, and I’ll keep doing it. Especially when large organizations reach out to me, asking to help them in their journey. At 99% of them I will have no impact, but it is the other 1% that I’m hoping to influence in some way. I can also regularly point out how silly their organizations are, even if the people I’m working with are well aware of the state of things. I know it isn’t how ALL large organizations have to behave because I do a lot of business with large entities, who are able to get me through legal, and able to pay me without problems. Somewhere along the way, certain organizations have made the decision to be more bureaucratic and the trick is going to be how do you begin unwinding this–this is what the API journey is all about.
I import and work with a number of OpenAPI definitions that I come across in the wild. When I come across a version 1.2, 2.0, 3.0 OpenAPI, I import them into my API monitoring system for publishing as part of my research. After the initial import of any OpenAPI definition, the first thing I look for is the consistent in the naming of paths, the availability of summary, descriptions, as well as tags. The naming conventions used is paths is all over the place, some are cleaner than others. Most have a summary, with fewer having descriptions, but I’d say about 80% of them do not have any tags available for each API path.
Tags for each API path are essential to labeling the value a resource delivers. I’m surprised that API providers don’t see the need for applying these tags. I’m guessing it is because they don’t have to work with many external APIs, and really haven’t put much thought into other people working with their OpenAPI definition beyond it just driving their own documentation. Many people still see OpenAPI as simply a driver of API documentation on their portal, and not as an API discovery, or complete lifecycle solution that is portable beyond their platform. Not considering how tags applied to each API resource will help others index, categorize, and organize APIs based upon the value in delivers.
I have a couple of algorithms that help me parse the path, summary, and description to generate tags for each path, but it is something I’d love for API providers to think more deeply about. It goes beyond just the resources available via each path, and the tags should reflect the overall value an API delivers. If it is a product, event, messaging, or other resource, I can extract a tag from the path, but the path doesn’t always provide a full picture, and I regularly find myself adding more tags to each API(if I have the time). This means that many of the APIs I’m profiling, and adding to my API Stack, API Gallery, and other work isn’t as complete with metadata as they possibly could be. Something API providers should be more aware of, and helping define as part of their hand crafting, or auto-generation of OpenAPI definitions.
Tag your APIs as part of your OpenAPI definitions! I know that many API providers are still auto-generating from a system, but once they have published the latest copy, make sure you load up in one of the leading API design tools, and give that last little bit of polish. Think of it as that last bit of API editorial workflow that ensures your API definitions speak to the widest possible audience, and are as coherent as it possibly can. Your API definitions tell a story about the resources you are making available, and the tags help provide a much more precise way to programmatically interpret what APIs actually deliver. Without them APIs might not properly show up in search engine and Github searches, or render coherently in other API services and tooling. OpenAPI tags are an essential part of defining and organizing your API resources–give them the attention they deserve.
I gave a talk early in in June at POST/CON 2018 in San Francisco. The conference was a great mix of discussions reflecting the Postman community. You can find all the talks on Google, including mine about moving towards a modern AP lifecycle.
You can find all the stop along what I consider to be a modern API lifecycle on the home page of API Evangelist, with links to any of my research, services, tooling, and other storytelling I’ve done in each area.
Thanks again to Postman for having me out!
I am shifting my long running API operations from a PHP / EC2 based implementation to a more efficient Node.js / Lambda based solution, and I promised James Higginbotham (@launchany) that I’d share a breakdown of my process with him a month or so back. I’m running 100+, to bursts of 1000+ long running API requests for a variety of purposes, and it helps me to tell the narrative behind my code, introducing some coherence into the why and how of what I’m doing, while also sharing with others along the way. I had covered my earlier process a little bit in a story a few months ago, but as I was migrating the process, I wanted to further flesh out, and make sure I wasn’t mad.
The base building block of each long running API request I am making is HTTP. The only difference between these API requests, and any others I am making on a daily basis, is that they are long running–I am keeping them alive for seconds, minutes, and historically hours. My previous version of this work ran as long running server side jobs using PHP, which I monitored and kept alive as long as I possibly could. My next generation scripts will have a limit of 5 minutes per API request, because of constraints imposed by Lambda, but I am actually find this to be a positive constraint, and something that will be helping me orchestrate my long running API requests more efficiently–making them work on a schedule, and respond to events.
Ok, so why am I running these API calls? A variety of reasons. I’m monitoring a Github repository, waiting for changes. I’m monitoring someone’s Twitter account, or a specific Tweet, looking for a change, like a follow, favorite, or retweet. Maybe I’m wanting to know when someone asks a new question about Kafka on Stack Overflow, or Reddit. Maybe I’m wanting to understand the change schedule for a financial markets API over the course of a week. No matter the reason, they are all granular level events that are occurring across publicly available APIs that I am using to keep an eye on what is happening across the API sector. Ideally all of these API platforms would have webhook solutions that would allow for me to define and subscribe to specific events that occur via their platform, but they don’t–so I am doing it from the outside-in, augmenting their platform with some externally event-driven architecture.
An essential ingredient in what I am doing is Streamdata.io. Which provides me a way to proxy any existing JSON API, and turn into a long running / streaming API connection using Server-Sent Events (SSE). Another essential ingredient of this is that I can choose to get my responses as JSON PATCH, which only sends me what has changed after the initial API response comes over the pipes. I don’t receive any data, unless something has changed, so I can proxy Github, Twitter, Stack Overflow, Reddit, and other APIs, and tailor my code to just respond to the differential updates I receive with each incremental update. I can PATCH the update to my initial response, but more importantly I can take some action based upon the incremental change, triggering an event, sending a webhook, or any other action I need based upon the change in the API space time continuum I am looking for.
My previous scripts would get deployed individually, and kept alive for as long as I directed the jobs manager. It was kind of a one size fits all approach, however now that I’m using Lambda, each script will run for 5 minutes when triggered, and then I can schedule to run again every 5 minutes–repeating the cycle for as long as I need, based upon what I’m trying to accomplish. However, now I can trigger each long running API request based upon a schedule, or based upon other events I’m defining, leveraging AWS Cloudwatch as the logging mechanism, and AWS Cloudwatch Events as the event-driven layer. I am auto-generating each Node.js Lambda script using OpenAPI definitions for each API, with a separate environment layer driving authentication, and then triggering, running, and scaling the API streams as I need, updating my AWS S3 Lake(s) and AWS RDS databases, and pushing other webhook or notifications as I need.
I am relying heavily on Streamdata.io for the long running / streaming layer on top of any existing JSON API, as well as doing the differential heavy lifting. Every time I trigger a long running API request, I’ll have to do a diff between it’s initial response, and the previous one, but every incremental update for the next 4:59 is handled by Streamdata.io. Then AWS Lambda is doing the rest of the triggering, scaling, logging, scheduling, and event management in a way more efficient way than I was previously with my long running PHP scripts running as background jobs on a Linux EC2 server. It is a significant step up in efficiency and scalability for me, allowing me to layer on an event-driven layer on top of existing 3rd party API infrastructure I am depending on to keep me informed of what is going on, and keep my network of API Evangelist research moving forward.
I’m doing a lot of thinking regarding how JSON PATCH can be applied because of my work with Streamdata.io. When you proxy an existing JSON API with Streamdata.io, after the initial response, every update sent over the wire is articulated as a JSON PATCH update, showing only what has changed. It is an efficient, and useful way to show what has changed with any JSON API response, while being very efficient about what you transmit with each API response, reducing polling, and taking advantage of HTTP caching.
As I’m writing an OpenAPI diff solution, helping understand the differences between OpenAPI definitions I’m importing, and allowing me to understand what has changed over time, I can’t help but think that JSON PATCH would be a great way to articulate change of the surface area of an API over time–that is, if everyone loyally used OpenAPI as their API contract. Providing an OpenAPI diff using JSON PATCH would be a great way to articulate an API road map, and tooling could be developed around it to help API providers publish their road map to their portal, and push out communications with API consumers. Helping everyone understand exactly what is changing in way that could be integrated into existing services, tooling, and systems–making change management a more real time, “pipelinable” (making this word up) affair.
I feel like this could help API providers better understand and articulate what might be breaking changes. There could be tooling and services that help quantify the scope of changes during the road map planning process, and teams could submit OpenAPI definitions before they ever get to work writing code, helping them better see how changes to the API contract will impact the road map. Then the same tooling and services could be used to articulate the road map to consumers, as the road map becomes approved, developed, and ultimately rolled out. With each OpenAPI JSON PATCH moving from road map to change log, keeping all stakeholders up to speed on what is happening across all API resources they depend on–documenting everything along the way.
I am going to think more about this as I evolve my open API lifecycle. How I can iterate a version of my OpenAPI definitions, evaluate the difference, and articulate each update using JSON PATCH. Since more of my API lifecycle is machine readable, I’m guessing I’m going to be able to use this approach beyond just the surface area of my API. I’m going to be able to use it to articulate the changes in my API pricing and plans, as well as licensing, terms of service, and other evolving elements of my operations. It is a concept that will take some serious simmering on the back burners of my platform, but a concept I haven’t been able to shake. So I might as well craft some stories about the approach, and see what I can move forward as I continue to define, design, and iterate on the APIs that drive my platform and API research forward.
It is tough to help developers think outside of the world they operate within. Most software is still developed and managed within silos, knowing it’s inner workings will never be seen by anyone outside of the team. This mode of operation is a rich environment for poor code quality, and teams with port communication. This is one of the reasons I’ve embraced web APIs, after running software development teams since the 1990s, I’ve been put in charge of some pretty dysfunctional teams, and some pretty unwieldy legacy codebases, so once I started working out in the open using web APIs, I did’t want to go back. Web APIs aren’t the cure for all of our technology problems, but it does begin to let some sunlight in on some messed up ways of doing things.
One common illness I still see trickling out of API operations are developers not using plain language. Speaking in acronyms, code, and other cryptic ways of articulating the resources they are exposing. I came across a set of API resources for managing a DEG the other day. You could add, updated, delete and get DEGs. You can also pull analytics, history, and other elements of a DEG. I spent about 10-15 minutes looking around their developer portal, documentation, and even Googling, but never could figure out what a DEG was. Nowhere in their documentation did they ever tell consumers what a DEG was, you just had to be in the know I guess. The API designer (if that occurred) and developer had never stopped to consider that maybe someone would stumble across their very public API and not know what a DEG was. Demonstrating how us developers have trouble thinking outside our silos, and thinking about what others will need.
There is no reason that your API paths shouldn’t be plain language, using common words. I’m not even talking about good RESTful resource design, I’m simply talking about looking at the URI for an API and being able to understand what it is because it used words we can understand. If you have trouble pausing, and stepping back, and thinking what some random 3rd party developer will interpret your API paths as, I recommend printing them out and sharing them with someone that isn’t on your team, and familiar with the resources you work with. Even if your APIs aren’t going to be public, someday you will be gone, and maybe your documentation isn’t up to date, and someone will have to reverse engineer what your API does. There is no reason your API should hide what it does, and not speak for itself, providing an intuitive, plain language description fo the value it possesses.
I look at hundreds of APIs each month. I push myself to understand what an API does in seconds, or minutes. When I spend 10-15 minutes unsuccessfully to understand what an API does, there is a problem with its design. I’m not talking about good API design, I’m just talking about coherent API design. There is no reason you should have an acronym in your API path. I don’t care how short-lived, or internal you view this API. These are often times the APIs that end up sticking around for generations, and becoming part of the technical debt future teams will have to tackle. Don’t be part of the problem in the future. Speak in plain language, and make your API paths speak for themselves. Make them speak to as wide as possible audience as you can. Make them reach outside of your developer circles, and become something any human can copy and paste, and put to work as part of their daily routine.
When you operate your application within the API ecosystem of a large platform, depending on the platform, you might have to worry about the platform operator copying, and emulating what you do. Twitter has long been accused of sharecropping within their ecosystem, and other larger platforms have come out with similar features to what you can find within their API communities. Not all providers take the ideas, it is also very common for API platforms to acquire talent, features, and applications from their ecosystems–something that Twitter has done regularly. Either way, API ecosystems are the R&D, and innovation labs for many platforms, where the latest features get proven.
As the technology playing field has consolidated across three major cloud providers, AWS, Azure, and Google, this R&D and innovation zone, has become more of a cloud kill zone for API providers. Where the cloud giants can see the traction you are getting, and decide whether or not they want to launch a competing solution behind the scenes. Investors are tuning into this new cloud kill zone, and in many cases opting not to invest in startups who operate on a cloud platform, afraid that the cloud giant will just come along and copy a service, and begin directly competing with companies operating within their own ecosystem. Making it a kill zone for API providers, who can easily be assimilated into the AWS, Azure, or Google stack, and left helpless do anything but wither on the vine, and die.
Much like other API ecosystems, AWS, Azure, and Google all have the stats on who is performing across their platforms, and they know which solutions developers are demanding. Factoring in the latest growth trends into their own road maps, and making the calculations around whether they will be investing in their own solutions, or working to partner, and eventually acquire a company operating with this new kill zone. The 1000 lb cloud gorillas carry a lot of weight in regards to whether or not they choose to partner and acquire, or just crush a startup. I’m guessing there are a lot of factors they consider along the way that will contribute to whether or not they play nicely or not. There are no rules to this game, and they really can do whatever they want with as much market share and control over the resources as they all possess. It will be interesting to begin tracking on acquisitions and partnerships across all players to better understand the score.
I wrote last year about how the API space is in the tractor beam of the cloud providers now, and it is something I think will only continue in coming years. It will be hard to deploy, scale, and operate your API without doing it on one of the cloud platforms, or multiple cloud platforms, forcing all API providers to operate within the cloud kill zone. Exposing all new ideas to share their analytics with their platform overlords, and open them up for being copied, or at least hopefully acquired. Which is something that will stunt investment in new APIs, making it harder for them to scale and grow on the business side of things. Any way you look at it, the cloud providers have the upper hand when it comes to cherry picking the best ideas and features, with AWS having a significant advantage in the game with their dominant cloud market position. It will be pretty hard to do APIs in the next decade without AWS, Azure, and Google knowing what you are doing, and having the last vote in whether you are successful or not.
I was profiling the volume of API from the Internet of Things platform Predix this last week. Luckily they have OpenAPI definitions for each of the APIs, something that makes my life a lot easier. As, they have a wealth of APIs available, doing an amazing amount of work when it comes to connecting devices to the Internet–I love their enthusiasm for putting out APIs. My only critical feedback for them after working my way through their API definitions, is they should invest some time to develop an API design guide, and distribute across their teams. The wild variances in definition and design of their APIs made me stumble a number of times while learning about what they do.
While looking through the definitions for the Predix APIs, I found many inconsistent patterns between them, and you could tell that they had different teams (or individuals) working across the suite of APIs. The inconsistencies ranged from the naming, description, and how the meta data was provided for each API, all the way to acronyms used in API paths, and other things that prevented me from understand what an API did all together. While I am stoked they provide OpenAPI definitions for all of their APIs, I still struggled to understand what was possible with many of their APIs. It kind of feels like they need an external editor to review each API definition before it leaves the door, as well as some sort of automated validation using JSON schema, that would work against a common set of API design standards.
I can tell that Predix has an extremely powerful stack of Internet of Things API resources. They have insight, predictive, and event-driven layers, and a wealth of resources for device operators to put to work. They just need another layer of API design polish on their APIs, as well as ensuring their API documentation reflects this design polish, helping bring it all home. If they did, I’m guessing they would see their adoption numbers increase. It can be tough to come into someone’s world and understand the value they bring to the table, even with simple API resources, but with something as robust and complex as what Predix is up to, even an experienced integrator like me is having trouble getting up to speed on what was possible. A little API design and documentation polish would go a long way to reduce the friction for new consumers getting up to speed.
I struggle with making sure some of my writing gets the editing love before it gets out the door. I also struggle with making sure my own API definitions and designs get the love they need before they see the light of day. As a one person show I just do not have the resources it always takes to deliver at the scope I need. So I fully understand the challenge of small startups when it comes to investing in proper API design across their operations–you just don’t always have the time to slow down and invest in a common API design guide, and the training and awareness across teams. I don’t want to shame the Predix team, as I can tell they’ve invested a lot into their APIs. I just want to make sure they understand that a little investment in API design will go a long ways in helping them better achieve their goals as an Internet of Things API provider.
About half of the teams I work with on microservices strategy are beginning to freak out about the number repositories they have, and someone is regularly bringing up the subject of having a mono repo. Which is usually a sign for me that a group is not ready for doing the hard work involved with microservices, but also shows a lack of ability to think, act, and respond to things in a distributed way. It can be a challenge to manage many different repositories, but with a decoupled awareness of the sprawl that can exist, and some adjustments and aggregation to your strategy it can be doable, even for a small team.
The most import part of sticking to multiple repositories is for the sake of the code. Keeping services decoupled in reality, not just name is extremely important. Allowing the code behind each service to have its own repository, and build a pipeline that keeps things more efficient, nimble, and fast. Each service you layer into a mono repo will be one more chunk of time needed when it comes to builds, and understanding what is going on with the codebase. I know there are a growing number of strategies for managing mono repos efficiently, but it is something that will begin to work against your overall decoupling efforts, and you are better off having a distributed strategy in place, because code is only the first place you’ll have to battle centralization, in the name of a more distributed reality.
Github, Gitlab, and Bitbucket all have an API, which makes all of your repositories accessible in a programmatic way. If you are building microservices, and working towards a distributed way of doing things, it seems like you should be using APIs to aggregate and automate your reality. It is pretty easy to setup an organization for each grouping of microservices, and setup a single master or control repository where you can aggregate information, and activity across all repository–using Github Pages (or other static implementation) as a central dashboard, and command center for managing and containing microservice sprawl. Your repository structure should reflect your wider microservices organization strategy, and all the moving parts should be allowed to operated in a distribution fashion, not just the code, but also the conversation, support, and other essential elements.
I’m spending more time learning about Kubernetes, and studying how microservices are being orchestrated. Like other aspects of the API world, I’m going to focus on not just the code, but also the other communications, support, dependencies, security, testing, and critical building blocks of delivering APIs. I feel like many folks Im talking with are getting hung up on the distributed nature of everything else, while trying to distribute and decouple their code base. Microservices are definitely not easy to do, and decoupling isn’t an automatic solution to all our problems. From what I am seeing, it is opening up more problems than it is solving in some of the organizations I am working with, and causing a lot of anxiety about the scope of what teams will have to tackle when trying to find success with microservices across their increasingly distributed organizations.
We are building up to the 9th edition of API Strategy & Practice (APIStrat) happening in Nashville, Tennessee this September 24th through 26th. As part of the build up we are looking for sponsors to help make the event happen, bringing the API community together once again to share stories from the trenches, and discuss healthy practices that are allowing companies, organizations, institutions, and government agencies make an impact when it comes to their API operations.
The 2017 edition of APIStrat in Portland, OR was a huge success, and help complete the transition of APIStrat to be part of the OpenAPI Initiative (OAI). After seven editions, and four years of operation exclusively by 3Scale and API Evangelist, the event has matured and will continue growing under the guidance of the OAI, and the community that has evolved around the OpenAPI specification. Presenting an opportunity for other API providers, and API service providers to get involved by joining as an OAI member and / or sponsoring APIStrat, and joining the conversation that has been going on in the community since early 2013.
You can download the APIStrat conference prospectus from the Linux Foundation / OAI event website, and there is a form to submit to learn more about sponsoring. You can also email [email protected] if you’d like to get plugged in. Feel free to also reach out to me as well, as I’m in charge of trying to drum up sponsors, and expand our base beyond just the OAI membership, and the companies who stepped up last year. Helping API providers and service providers understand what a community event APIStrat is, and help it differentiate from the other API, and tech-focused conferences that are happening.
I’m definitely biased, as I help start and grow the conference, but after running tech events for over a decade, it was important to me that APIStrat grow into a community event about ideas, and less about vendors and product pitches. It is a great opportunity for API providers, and API service and tooling providers to actually rub elbows with developers who are building on top of their APIs, and putting their tools and services to work. The keynotes, sessions, and workshops are always great, but the hallway conversations are always where the magic happens for me. Please step up and help make sure the event continues to grow, and help sponsor APIStrat in Nashville. If you do, I promise to cover your APIs here on the blog, and help tell the story of the impact you are making on the community leading up to the event this September in Tennessee!!
GraphQL folks keep on with the GraphQL vs REST narratives, rather than a REST and / or GraphQL narrative lately with a recent burger meme/narrative. Continuing to demonstrate their narrow view of the landscape, and revealing the short lived power of an adversarial approach to community building. I get why people do this, because they feel they are being clever, and that the click response from the echo chamber re-enforces it, but ultimately it is something that won’t move the conversation forward, but it does get them kudos within their community–which is what many of them live for.
I’ll start with the usual disclaimer. I actually like GraphQL, and prescribe it as part of my API toolbox. However, rather than a REST vs GraphQL approach, I sell it as REST and GraphQL, depending on the developer audience we are trying to reach with our efforts. Whether or not you use GraphQL on your platform is completely based upon knowing your developers, and working with a group that understands the resources before offered–something the GraphQL community continues their failure to see. Also their adversarial marketing tactics has lost me several GraphQL projects in government because it comes off as being a trend, and not something that will be around very long.
With that said, I think this meme tells a great story about GraphQL, and demonstrates the illnesses of not the technology, but the ideology and beliefs of the community. I had a couple of thoughts after seeing the Tweet, and reviewing the replies:
1) I thought it was an anti-GraphQL meme at first. Demonstrating that you can build a horrible burger with some very well known ingredients. Spoofing on the burger emoji drama that has been going on in recent years.. I mean, is the lettuce the plate in the GraphQL burger?
2) Like GraphQL, the food choices demonstrate that GraphQL works well in very controlled environments. Where there are known ingredients, and your clients/customers/developers know the ingredients, and know what they want. Hell yeah GraphQL is a better choice in this environment. The problem is you are selling it as a a better solution than REST in general. I hate to tell you, but most of the business getting done in the world IS NOT FAST FOOD.
3) The meme demonstrates the whole fast food, limited world view of many technologists who work with known ingredients, and think everyone is just like them. This tool works for me, and everyone is just like me, so what I use is cool, and everyone should use the tools and the process that I do. A common perspective out of the white bread (bun?) world of technology.
4) Let’s take this GraphQL meme and begin applying it to an Ethiopian, Greek, or French menu. Let’s take it and apply to a BBQ, catering, or maybe home cooked family gathering. Try applying it when you get a food basket from your local community supported agriculture (CSA), where you have no idea the ingredients that are coming, and you’ll have to adjust based upon the season and whatever is available to you that week. Maybe do the same for a food shelter and pantry, does everyone get it their way?
5) There are some restaurants in New York I’d love to take you to, and have you ask for it your way. I’d love to see you get yelled out of the place when you think you know more than the chef, and you always should have things your way. Really, you know more than someone who has been cooking for years, and your fast food loving, unsophisticated tastes are going to dictate what gets served? Get outta here!!
6) I love API to restaurant menu analogies. I wrote one to support the Oracle v Google copyright case, which the Google lawyer referenced in the latest round. There are many ways you can use restaurants and food to make API comparisons, and educate people about the potential of APIs. I’m sorry though, this one just wasn’t sophisticated enough to really bring home the potential of APIs, and it was more about reflecting this same unsophisticated approach of people marketing and telling stories around GraphQL.
I’ll say it again, and again, and again. I’m not anti-GraphQL. I’m against ya’ll saying it is a replacement for REST. Stop it. It’s dumb. It shows your lack of awareness of the larger API world. It shows you live in tech isolation, where you think everyone wants it your way. Most developers I know do not have a clue as to what they want. They don’t understand the existing schema being used, and need menus, and hand-crafted buffets. Sure, there are development groups who know exactly what they need, and have a full grasp on the schema and resource models being used, but this isn’t EVERYONE!! Stop it. I get GraphQL, but I’m getting tired of coming across new APIs I don’t understand at all, and being expected to just know what I want. I love GraphQL for Github because I KNOW GITHUB. I don’t love GraphQL for the OpenStates API, because I have no clue what the schema and model is for their API–please do the extra work to document your resources, and provide me intelligent, well-crafted paths to get at your valuable data.
Instead of bashing REST, how about thinking more about REST as a starter. Having feedback loops in place to get to your audience. Sure, if it is all internal development, in service of a known group of React developers, go for it–use GraphQL! However, if it is a public API, start with REST, establish feedback loops, and get to know your audience. If enough developers are requesting a query language (GraphQL isn’t the only show in town), and it makes sense in your roadmap, then offer GraphQL alongside REST, but not instead of REST. GraphQL works in a known known, and sometimes a known unknown environment, but not in an unknown unknown environment. The community needs to wake up and realize this. Stop selling it as a replacement to REST, and realize it is just another tool in the API toolbox. Y’all are just hurting your cause, and running some people off with this regular REST v GraphQL storytelling. In the end, you are just showing your lack of knowledge and respect for the web–just like Facebook does.
P.S. Anyone who has their feelings hurt by this post, needs to get out more. Maybe change jobs, move to a new city and industry. You need to see and experience more than you have currently.
It has been a fascinating and eye opening experience sitting at the intersection of tech startups and the web, mobile, and device applications they’ve built over the last decade. In 2010 I was captivated by the power of APIs to deliver resources to developers, and end-users. In 2018, I’m captivated by the power of APIs to mine end-users like they are just a resource, with the assistance of the developer class. A dominant white male class of people who are more than willing to look the other way when exploitation occurs, and make for the perfect “tools” to be exploited by the wealthy investor class.
While I do not have much hope for diversity efforts in tech, or the bro culture waking up, I do have hope for city and state/provincial lawmakers to wake up to the exploitation that is going on. I’ve seen hints of cities waking up to the mining that has been occurring by Facebook and Google over the last decade. The open exploitation and monetization of a city’s and state’s most precious resources–their constituents. While some cities are still swooning over having Amazon set up shop, or Facebook to build a data center, these company’s web, mobile, and device applications have infiltrated their districts been probing, mining, extracting, and shipping value back to offshore corporate headquarters.
You can see this occurring with Google Maps, which has long been a darling of the API community. We were all amazed at the power of this new mapping resource, something us developers could never have built on our own. We all integrated it into our websites, and embedded it into our mobile applications. We could use it to navigate and find where we were going, completely unaware of the power of the application to mine data from our local transit authorities, businesses, as well as track the location of all of us at each moment. Google Maps was the perfect trojan horse to invade our local communities, extract value, only leaving us with a handful of widgets and embeddable apps to keep us hooked, and working for the Google machine–always giving as little back as possible.
Facebook is probably the highest profile example, connecting our families and communities, while it also disrupted our local news, and information channels, as well as take control over our elections. While connecting us all at the local level, we failed to see we were being connected to the Facebook corporate machine, reminiscence of the Matrix movie of the 1990s. Now we are just mindlessly scrolling, clicking, and emotionally responding, where we are simultaneously being mined, tracked, influenced, nudged, and directed. Something that was once done out in the open for many years through a public API program, but is slowly being closed up and done privately behind closed doors, so that a new regulatory show can be performed to demonstrate that Facebook really cares.
I’m spending more time in Europe, having conversations with regulators and business leaders about a more sensible future driven by APIs. Having conversations with city leaders about the value of their data, content, and algorithms. Discussing the value of their constituents personal data, privacy, and security. Talking about the imperialist nature of Facebook, Google, Twitter, Amazon, and Microsoft, and how they invade, conquer, then extract value from our communities. Helping mayors, governors, and other lawmakers realize the value they have before it is gone, and helping them realize that they can take control over their digital resources using APIs, and gain an upper hand in the conversations that are already occurring across the web.
API versioning is almost always one of the top attended discussions at conferences I help organize, and one of the first questions I get in the QA sessions at workshops I conduct. People want to understand the “right way” to version, when it my experience there is rarely ever a “right way” to version your APIs. There are commonly held practices regarding sensible ways to version your APIs, as well as dominant patterns for how you version APIs, but there isn’t any 100% solid answer to the question, despite what many folks might say.
In my experience, the most commonly held approach to properly versioning your APIs (if you are going to), is to put the major and minor version in your header and / or combine it with content-type negotiation via your header. However, even with this knowledge being widely held, the most dominant pattern for versioning your APIs is sticking it in the URL of your API. I know many API providers who put the version in the header, despite many on their team fully being aware that it is something that should be put in the header. So, why is this? Why do people still do it the “wrong way”, even though then know how to do it the “right way”?
I feel like this phenomenon reflects the wider API space, and how upside down many API belief systems are. People put the version in the URL because it is easier for them, and it is easier for their developers to understand. While headers are a native aspect of developing using the web, they are still very foreign and unknown to most developers. While this shows the lack of web literacy that is rampant amongst developers, it also demonstrates why simple web APIs have dominated the landscape–they are easy for a wide segment of developers to understand. An aspect of why this whole API thing has worked that many technologists overlook, and take for granted as they try to push the next trend or solution on the sector.
While conducting workshops, I always teach the more sensible patterns around versioning, but I can’t always sell them as the “right way”. Because I don’t see a “right way”. I see people trying to get a job done, and reach a wide audience. I see people trying to keep things simple for their developers, and taking the path of least resistance. I see a whole lot of web literacy education that needs to occur across the tech sectors and in school. I just don’t see any perfect answer to the API versioning debate. I see a whole lot of interesting and useful patterns, and I see people doing the best they can with what they have. Reflecting why APIs work so well, because they are scrappy, often times simple, and allow people to get business done on the web using low cost, easy to understand approaches to making resources available.
People love to hate in the API space. Ok, I guess its not exclusive to the API space, but it is a significant aspect of the community. I receive a regular amount of people hating on my work, for no reason at all. I also see people doing it to others in the API space on a regular basis. It always makes me sad to see, and have always worked to try to be as nice as I can to counteract the male negativity and competitive tone that often exists. While I feel bad for the people on the receiving end of all of this, I often times feel bad for the people on the giving end of things, as they are often not the most informed and up to speed folks, who seem to enjoy opening their mouth before they understand what is happening.
One thing I notice regularly, is that these same people like to bash on is OpenAPI (fka Swagger). I regularly see people (still) say how bad of an idea it is, and how it has done nothing for the API space. One common thread I see with these folks, which prevents me from saying anything to them, is that it is clear they really don’t have an informed view of what OpenAPI is. Most people spend a few minutes looking it, maybe read a few blog posts, and then establish their opinions about what it is, or what it isn’t. I regularly find people who are using it as part of their work, and don’t actually understand the scope of the specification and tooling, so when someone is being vocal about it and doesn’t use actually it, it is usually pretty clear pretty quickly how uninformed they are about the specification, tooling, and scope of the community.
I’ve been tracking on it since 2011, and I still have trouble finding OpenAPI specifications, and grasping all of the ways it is being used. When you are a sideline pundit, you are most likely seeing about 1-2% of what OpenAPI does–I am a full time pundit in the game and I see about 60%. The first sign that someone isn’t up to speed is they still call it Swagger. The second sign is they often refer to it as documentation. Thirdly, they often refer to code generation with Swagger as a failure. All three of these views date someone’s understanding to about a 2013 level. If someone is forming assumptions, opinions, and making business decisions about OpenAPI, and being public about it, I’d hate to see what the rest of their technology views look like. In the end, I just don’t even feel like picking on them, challenging them on their assumptions, because their regular world is probably already kicking their ass on a regular basis–no assistance is needed.
I do not feel OpenAPI is the magical solution to fix all the challenges the API space, but it does help reduce friction at almost every stop along the API lifecycle. In my experience, 98% of the people who are hating on it do not have a clue what OpenAPI is, or what it does. I used to challenge folks, and try to educate them. Over the years I’ve converted a lot of folks from skeptics to believers, but in 2018, I think I’m done. If someone is openly criticizing it, I’m guessing it is more about their relationship to tech, and their lack of awareness of delivering APIs at scale, and they probably exist in a pretty entrenched position because of their existing view of the landscape–they don’t need me piling on. However, if people aren’t aware of the landscape, and ask questions about how OpenAPI works, I’m always more than happy to help open their eyes to how the API definition is serving almost every stop along the API lifecycle from design to deprecation, and everything in between.
When it comes to providing data, content, and even ML and AI models via APIs, having a public platform will become a competitive advantage. I know that many companies see it as giving away something, especially when your resources and business model are not defensible, but in reality having a publicly available, 24/7 operational, self-service solution will give you an edge over your more proprietary approaches to making resources available on the web. Sure, your competition will be able to often get in there without friction, but so will your customers–how many customers vs. competitors do you have?
I know many companies believe in the power of a sales team to be able to squeeze every last penny out of would be customers, but a sales only approach leaves a significant amount of self-service revenue on the table. Throughout the course of our busy days, many IT decision makers just do not have the time for the phone calls and lunches involved with the traditional sales process. Sure, there are some IT decision makers who fill their schedule with these types of conversations, but there are a growing number who depend on self-service, SaaS approaches to getting business done on a daily basis–look at the growth of Amazon Web Services over the last decade if you need a reference point.
If you think a public API platform involves giving away your intellectual property in 2018, you are severely behind the times on where the sector has been headed for about a decade. Far enough behind that you may not be able to play catch up at the speed in which things are shifting. A public portal, documentation, and other resources does not mean you are giving anything away. Even having a free tier doesn’t mean that you are giving away the farm. Modern API management solutions allow you to generate leads, let developers kick the tires, while also still being able to charge what the market will bear for your data, content, and algorithms. You can also still have a sales force that will swoop in on leads, and close the deals when it makes sense.
Even with a self-service API, and robust documentation, code samples and SDKs, API providers still have to work hard to reduce friction when on boarding–providing OpenAPI definitions, Postman collections, connectors, plugins, and platform specific development kits to make integration quick and painless. If you don’t even have a public self-service presence you are just getting in the way of integration, and a growing number of your customers will just choose to go with your competitors who have opted to get out their way. The companies who don’t have self-service in their DNA won’t be able to compete in the new landscape, making it essential to be able to do business out in the open, in a self-service way, essential to staying competitive in the new API-driven landscape.
My friend James Higginbotham (@launchany) was sharing his frustration with being able to stay in tune with changes to a variety of APIs. Like me, James works to stay in tune with a variety of signals available via platforms like Twitter, Github, and other commonly used services. These platforms don’t always properly signal when things are updated, changed, or advanced, making it difficult to understand the granular changes that occur like likes, votes, edits, and other common events that occur via highly active platforms.
This challenge is why the evolution towards a more event-driven approach to operating an API platform is not just more efficient, it gives users what they need. Using event-driven architectural approaches like Webhooks, and real times streams. This is one of the reasons I’m interested in what Streamdata.io does, beyond them helping support me financially, is that they allow me to focus on the event-driven shift that is occurring with many leading API providers, and needs to be brought to the attention of other platforms. Helping API providers be more efficient in what they are doing, while also meeting the needs of the most demanding customers like James and myself.
It is easy to think Streamdata.io is just about streaming real time data. This is definitely a large aspect of what the SaaS solution does, but the approach to using Server-Sent Events (SSE), with incremental updates using JSON Patch adds another useful dimension when it comes to understanding what has changed. You can proxy an existing HTTP API that returns a JSON response using Streamdata.io, and the first response will look just like any other, but every pushed response after that will be a JSON Patch of just what has changed. Doing the heavy lifting of figuring out what has changed in each API response and only sending you the difference, and allowing you to focus only on what has changed, and not having to rely on timestamps, and other signals within the JSON response to understand what the difference is from the previous API response.
Using Streamdata.io you don’t have to keep polling an API asking if things have changed, you just proxy the API and you get pushed changes via an HTTP stream. You also don’t have to sort through each response and try to understand what changed, you just take the JSON Patch response, and it tells you what has changed. I’m going to create a draft blueprint for James of how to do this, that he can use across a variety of APIs to establish multiple API connections using long running, server-side API streams for a variety of topics. Allowing him to monitor many different APIs, and stay in tune with what changes as efficiently as possible. Once I craft a generic blueprint, I’m going to apply to Twitter and see if I can increase the efficiency of my Twitter monitoring, by turning their REST APIs into real time feeds using Streamdata.io.
One thing I see a lot from API service providers who are selling their services to the API sector, is that once they find success servicing one stop along the API lifecycle, they often want to service other additional stops. I don’t have a problem with API service providers delivering across multiple stops along the API lifecycle, however I do caution of trying to expand across too many stops, and potentially doing any of them poorly, rather than partnering with other more specialized API service providers to help you focus on what you do best.
I’m a big advocate for encouraging API providers to service one to five stops along the API lifecycle well, and then partner for helping deliver the rest of the stops. I know that all your investors are encouraging to take as many pieces of the puzzle as you possibly can, but there is more money in doing a handful of things really well, over doing many things poorly. Try to be an expert in a handful of specialized areas, over being a generalist. Then make sure your platform is as interoperable as possible, while investing in your partner program to attract the best of breed API service providers to your platform.
This balance between focusing on a handful of stops and partnering is why I emphasize and study common approaches to delivering plugins. All platforms should invest in plugin infrastructure, to allow for extending their reach beyond the stops that a platform services. Feature creep, and platform bloat is a real challenge, especially when you have investors whispering in your ear to keep building, and a very vocal, but often long-tail group of users demanding solutions to their unique problems. Plugin and connector architecture is how you help manage this reality, and provide a relief valve for delivering too many features as part of your platform, while also bringing in potential partners who can help extend what your platforms in a way that allows you to keep doing what you do best.
I see a big push going on from many legacy API service providers, as well as some of the next generation of startups bringing services and tooling to the space. I feel like many people desire a single solution to do everything, but then fail to realize that every platform that has attempted this in the past ends up failing, because you can’t be everything to everyone. I want my API service providers to stick to doing a handful of things well, but then acknowledge that I will also be using several other tools to what get I need accomplished on a daily basis. Ideally all of my tools are interoperable with import and export capabilities, as well as a suite of API driven connectors, and plugins that allow me to keep all of my services and tooling working together in concert. For this reality to occur we all have to resist the temptation to lock our customers in, and put down delusions that we can serve all stops along a modern API lifecycle all by ourselves.
I’m fascinated by the dominating power of programming languages. There are many ideological forces at play in the technology sector, but the dogma that exists within each programming language community continues to amaze me. The potential absence of programming language dogma within the world of APIs is one of the reasons I feel it has been successful, but alas, other forms of dogma tends to creep in around specific API approaches and philosophies, making API evangelism and adoption always a challenge.
The absence of programming languages in the API design, management, and testing discussion is why they have been so successful. People in these disciplines have ben focused on the language agnostic aspects of just doing business with APIs. It is also one of the reasons the API deployment conversation still is still so fragmented, with so many ways of getting things done. When it comes to API deployment, everyone likes to bring their programming language beliefs to the table, and let it affect how we actually deliver this API, and in my opinion, why API gateways have the potential to make a comeback, and even excel when it comes to playing the role of API intermediary, proxy, and gateway.
Programming language dogma is why many groups have so much trouble going API first. They see APIs as code, and have trouble transcending the constraints of their development environment. I’ve seen many web or HTTP APIs called Java API, Python APIs, or reflect a specific language style. It is hard for developers to transcend their primary programming language, and learn multiple languages, or think in a language agnostic way. It is not easy for us to think out of our boxes, and consider external views, and empathize with people who operate within other programming or platform dimensions. It is just easier to see the world through our lenses, making the world of APIs either illogical, or something we need to bend to our way of doing things.
I’m in the process of evolving from my PHP and Node.js realm to a Go reality. I’m not abandoning the PHP world because many of my government and institutional clients still operate in this world, and I’m using Node.js for a lot of serverless API stuff I’m doing. However I can’t ignore the Go buzz I keep coming stumbling upon. I also feel like it is time for a paradigm shift, forcing me out of my comfort zone and push me to think in a new language. This is something I like to do every five years, shifting my reality, keeping me on my toes, and forcing me to not get too comfortable. I find that this keeps me humble and thinking across programming languages, which is something that helps me realize the power of APIs, and how they transcend programming languages, and make data, content, algorithms, and other resources more accessible via the web.
I spend a lot of time talking about the SalesForce API, using it as a reference for where the API evolution began 18 years ago, but it has been a long time since I’ve actually worked with the SalesForce API. Getting up and running with any API, especially iconic APIs that we all should be familiar with, is always an enlightening experience for me. Going from zero to understanding what is going on and actually achieving the API call(s) you want, is really what this game is all about.
As part of some work I’m doing with Streamdata.io I needed to be able to add new leads into SalesForce, and I thought it would be a good time for me to get back into the saddle with the SalesForce REST API–so I volunteered to tackle the integration. The SalesForce API wasn’t as easy to get up and running as many simpler APIs I onboard with is, as the API docs isn’t as modern as I’d expect, and what you need is buried behind multiple clicks. Once you find what you are looking for, and click numerous times, you begin to get a feel for what is going on, and the object model in use becomes a little more accessible.
In addition to finding what you need with the SalesForce REST API, you have to make sure you have a handle on the object structure and nuance of SalesForce itself. For this story, I am just working with one object–Leads. I’m using PHP to work with the API, and to begin I wanted to be able to get leads, to be able to see which leads I currently have in the system:
I will add pagination, and other elements in the future. For now, I just wanted to be able to get the latest leads I have in the system to help with with some checks on what is being added. Now that I can check to see what leads are in the system, I wanted to be able to add a lead, with the following script:
I am only displaying some of the default fields available for this example, and you can add other custom fields based upon which values you wish to add. Once I have added my lead, I wanted to be able to update with a PATCH API call:
Now I am able to add, update, and get any leads I’m working with via the SalesForce API. The project gave me a good refresher for what is possible with the SalesForce API. The API is extremely powerful, and something I want to be up to speed on so that I can intelligently respond to questions I get. I wish the SalesForce API team would spend some time modernizing their API portal and documentation, providing a more coherent separation between the different flavors of their API, and provide OpenAPI driven documentation, as well as Postman Collections. It would have saved me hours of working through their API docs, and playing around with different API calls in Postman before I was able to successfully OAuth, and make my first call against the accounts and leads API endpoints.
While I think SalesForce remains a worthwhile API to showcase when I talk about the history of APIs, and the power of providing web APIs, their overall documentation and approach is beginning to fall behind the times. SalesForce possesses many of the building blocks I recommend other API providers operate, and are very advanced in some of their support and training efforts, but their documentation, which is the biggest pain point for developers, leaves a lot to be desired. I’m used to having to jump through hurdles to get up and running APIs, so the friction for me was probably less than a newer API developer would experience. I could see some of the domain instance url, versioning, and available API paths proving to be a significant hurdle if you didn’t understand what was going on. Something that could be significantly minimized with some simpler, more modern API docs, and OpenAPI and Postman Collections available.
This report summarizes Skylight’s evaluation of the VA’s public datasets, which exist within the va.gov web domain, as well as an analysis of what types of data representatives of the Veteran community expressed would be most useful and valuable to Veterans and their supporters if made more digitally accessible and available by the VA. This report also outlines potential resources that can be turned into application programming interfaces (APIs) as part of the VA’s Lighthouse platform initiative, and actions the VA should consider to move forward successfully.
APIs are the next evolution in the web, and shouldn’t be thought of as the latest tech trend or vendor solution. The first phase of the web was about delivering data and content to humans using a browser. The second phase of the web is about delivering that same data and content to other applications and algorithms using APIs.
With that said, the purpose of this landscape analysis is, in effect, to assist the VA in evaluating the data and content that they’ve made available during the first phase of the web. This, in turn, will help set the stage for the VA to make smart investments in phase two of their web presence.
The VA has already signaled they’re committed to investing in the second phase of their web presence with the announcement of the Lighthouse API platform initiative. Our landscape analysis will help ensure that the Lighthouse program is aware of what types of data and content that the VA has already identified as important to serving the Veteran community. This visibility will allow the Lighthouse program to bring these resources into alignment with the development and operation of their API platform.
To help the VA evaluate the landscape that defines their web presence, we employed a “low-hanging-fruit” process that involved identifying the resources that exist across their web properties. That process relied on a spidering script, which we ran for two weeks (and continue to run). To begin the process, we seeded the script by giving it the root URL for the va.gov domain. The script then proceeded to:
Parse every URL on the page and store it in a database;
Count every table on the page, and the number of rows that exist in the table;
Count every form that exists on the page; and
Extract the title from the meta tags for each page.
The script then iterated and repeated this for every URL it found on any web page, working to identify each of the following types of data resources:
HTML table with more than 10 rows
The script ignored any URLs external to the seed domain (va.gov) and many common web objects (for example, images, Word docs, and videos).
As each page was processed, the script tried to identify potential data resources to deliver as an API by parsing several elements from them:
The title of the page a file was published on,
The name of the file itself, and
Occasionally a sample of the data.
We took the list of words extracted from this process, and sorted and grouped them by the number of times the word appeared, helping us understand the overall presence of each potential resource. Sometimes this produced a lot of meaningless words, but we worked to filter those out, leaving only the meaningful data resources.
After running the script for a couple of weeks, we spidered nearly 1/3 of the URLs (out a total of 4M+) targeted for processing. That was enough progress to start painting an interesting picture of the VA’s web presence and existing data resources, as described in the sections that follow.
VA’s web presence
The VA has a sprawling web presence, spanning multiple domains and subdomains. We focused our analysis on everything within the va.gov domain. In the future, we can extend the analysis to other domains, but for now we focused on the core VA web presence.
The VA’s web presence is spread across a mix of domain levels, including top-level domain and subdomain levels — program, state/region, and city. Domains play a role in providing addresses in the browser so that users can find the resources they need, as well as providing similar addressing for applications to find the resources they need via APIs. The VA domain sprawl reflects the growth of the VA’s web presence, and the lack of overall strategy when it comes to providing web address and routing to all VA resources. The current strategy (or lack thereof) represents the need for location- and program-related resource discovery, whether it’s in the browser, or for other applications via APIs.
We assume there are other domains that haven’t been indexed by our spider, as we were only able to index less than 1/3 of the targeted URLs during our two-week timebox. We can continue indexing and updating numbers beyond this period in order to paint an even more complete picture. Ideally, this would extend beyond the core va.gov domain. Domain and subdomains play an important role in determining how APIs will be accessed, and have a downstream impact on the overall API design, affecting both API path and parameter design. This makes domain and subdomains a top-level consideration early on in the VA’s API journey.
After the top-level domains of va.gov and www.va.gov, the most common approach to defining domains is by program, providing the addressing needed for organizing information by relevant programs. We have identified 133 individual program-related domains.
While there aren’t consistent naming conventions used in crafting these subdomains, it does demonstrate the prominence of programs, research, and other related groupings used across the VA web presence for organizing resources.
Beyond program-related domains, state/region level domains are being used to organize data and content for presentation to consumers. Only 22 subdomains are represented currently, but the practice demonstrates the prominence of these locations when it comes to organizing information.
Some states are just paths within the top-level VA domains, while others exist within regional subdomains, with the rest possessing their own subdomain. This demonstrates the importance of states and regions, but also the inconsistency of how domains or paths are used to organize information.
Lastly, you find many city-related subdomains being used to organize data and content, providing another dimension on how resources are being organized, while demonstrating the dominance of specific cities. We have identified 120 individual city-related domains.
Like states, there isn’t a consistent pattern in which cities have their own subdomain, with others existing as a path within state subdomains or top-level domains. The approach to using cities as part of subdomain DNS addressing further demonstrates the importance of location when it comes to the organization of data and content.
As part of the spidering the va.gov domain across the 278 subdomains that exist, over 4M individual URLs were identified, with slightly less than 1/3 of these URLs evaluated for potential data sources to-date. Across these URLs, we took the base path and grouped them by the number of pages and data files that exist.
While there are many other paths in use across the VA websites, these paths reflect the top paths in use to deliver data and content. Providing a look at what the most relevant resources are when it comes to providing web access to data and content, which is something that should be considered when delivering the same data and content to other applications.
After assessing the titles of HTML pages and the names of files, it’s clear there’s no consistent vocabulary in use across VA resources. This, combined with the use of key phrases, acronyms, and singular and plural variances, make it difficult to cleanly identify resources. We opted to use just keywords over phrases and to not expand acronyms as part of the process due to the difficulty in consistently identifying resources.
Even with the difficulties in identifying some resources, we were still able to paint a fairly compelling picture of the resources being exposed as common data formats across VA web properties. That’s because we were able to isolate, group, and identify words that are most commonly associated with resources. From there, we were able to establish some resource lists, which we have organized visually as tag clouds and tag lists.
Data is available across VA websites in a variety of formats. We focused on a handful of easy to identify formats, reflecting the low-hanging-fruit aspect of this landscape analysis. While there’s data locked up in simple text files and zipped packages, we chose to look for the easiest to identify and the easiest to publish data sources. Data that’s published by humans usually take the form of CSV files, spreadsheets, and HTML tables. Data that’s published by systems usually take the form of JSON and XML.
Each of type of format that we targeted provides a different story as to the type of resources being published. Publication implies that those resources carry some level of value and importance to VA stakeholders, and, potentially, to Veterans, their supporters, and other consumers of this information. We worked to harvest all the data available from several formats, but also worked to identify the top resources available from each type.
We discovered 534 CSV files containing a variety of data. By parsing the titles of the web pages these CSV files were linked from, and the names of some of the files, we identified handful of top resource types present across these files.
CSV files tell a particular story because they were most likely published by people working at the VA, who exported the files from spreadsheets and made them available on the website for a reason. This makes them relevant to the VA’s API conversation. You can view a list of CSV resources, as well as a complete list of CSV files on the GitHub repository.
We identified 6,077 spreadsheets containing a variety of data. After parsing these files for semantic meaning, we identified handful of top resource types present across these files.
Similar to CSV files, the presence of spreadsheets tell a very human story. Spreadsheets are the #1 source of data on the web, and reflects the data management and publishing practices across the VA. After evaluating what types of resources are available across these spreadsheets, we have been considering the use of spreadsheets as a data source, as well as a data publishing tool. You can view a list of XLS/XLSX resources, as well as a complete list of XLS/XLSX files on the GitHub repository.
We identified 467 JSON files containing a variety of data. Unlike the CSV and spreadsheet data sources, JSON files likely represent a more modern systems approach to publishing data and a whole another set of data sources, which should be considered when deploying APIs.
JSON reflects the latest evolution of data publishing at the VA. But they are only a small subset of the data being made available across VA web properties. This implies they have only become a recent priority when it comes to publishing data in a format that is consumable by developers and computers. You can view a list of JSON resources, as well as a complete list of JSON files on the GitHub repository.
We found 3,099 XML files containing a variety of data. Like JSON files, XML files represent system-generated publication of data. Unlike JSON, however, XML reflects an older systems approach to data publication. And are likely being generated by legacy systems that’ll be important to interface with over the course of the VA’s API journey.
XML represents a large portion of the data being published across VA web properties. This list of priority resources represents a significant part of the system-based publishing of data occurring at the VA. And provides a large snapshot of the systems that should be evolved as part of the deployment of APIs. You can view a list of XML resources, as well as a complete list of XML files on the GitHub repository.
We identified 8,393 pages that had tables on them with over 10 rows. These tables represent potentially valuable data and should be considered as part of the VA’s API deployment conversations.
While HTML tables tell a story about top resources that VA stakeholders thought website users needed access to, these tables also represent data that was published with potential search engine optimization (SEO) in mind. In other words, someone wanted the data to be indexed by search engines in order to make it more readily accessible. You can view a list of table resources, as well as a complete list of pages containing tables on the GitHub repository.
We identified 9,439 pages with more than one form present, which is usually just a basic search. Similar to HTML tables, these forms provide a window into how the VA is making data available for users to search, explore, and consume in the browser. This, in turn, tells another story of what types of resources are published to VA websites.
HTML forms often times provide a search mechanism for other table, CSV, JSON, XML, and spreadsheet resources, many of which are listed in the sections above. HTML forms tell their own story as to how and why data are being published across VA websites. And offer another source of resources that are being made available and should be considered as part of the VA’s API deployment efforts.
The only external source of data that we analyzed was data.gov, which hosts a number of VA data resources. While somewhat out-of-date, the VA datasets on data.gov tell an important part of the story that should be considered as part of the Lighthouse efforts. There are a lot of lessons to be learned from how data.gov has been used, beyond just understanding what resources have been published there.
The resources published to data.gov reflect the VA’s recent past when it comes to making data resources available and accessible via manual downloads and APIs. We think that the most important lesson that the VA should take away from its experience with data.gov is that the VA should own all the data and API resources and syndicate them as part of other external efforts. That way the VA owns the full scope of the effort, which will ultimately result in the VA being more invested in API operations. You can view a list of data.gov resources, as well as a complete list of data files on the GitHub.
Humanizing the data
To give us a more human perspective on what types of data resources are most valuable to Veterans and their supporters, we facilitated a series of online workshops using Mural. About 50 people total participated across all three workshops, with about 40% reporting as Veterans and 60% non-Veterans. During these workshops, we employed the KJ technique for establishing group priorities. The KJ technique relies on a focus question to drive the results of the workshop. We used the following focus question:
“What types of data, content, and other resources would be most useful to Veterans and their supporters if the VA could make them more available and accessible on the web, mobile devices, and other platforms?”
The following images capture the results of each workshop:
The yellow cards represent all the ideas, in response to the focus question, that everyone brainstormed. As you can see, these yellow cards were organized into like groups. The blue cards represent descriptive labels that participants gave to each group. The black circles with numbers represent the votes that the participants casted when asked which group labels they thought best answered the focus question. We weighted Veteran votes 2x more heavily than votes from non-Veterans.
You may notice things that seem out of place in the final results (for example, yellow cards that look like they belong to another category). This is largely due to the timeboxed nature of the activities. In other words, not everything could be made perfect, but that doesn’t detract from the overall usefulness of the results.
Given the fact that the results were spread across three different workshop sessions, we took the additional step of normalizing the groupings and merging the votes.
Directory of Services/Resources – 34 votes
Mental Health – 21 votes
Personal Healthcare Data – 20 votes
Personalized Self-Service Portal – 20 votes
Benefits – 13 votes
Peer Support Networking – 8 votes
Family Support Networking – 9 votes
Real-Time Status – 8 votes
Patient Experience Data – 7 votes
Military-to-Civilian Transition – 7 votes
Ratings and Calculators – 6 votes
Appointments – 6 votes
Medical Healthcare – 5 votes
Housing Assistance – 5 votes
Public Accountability and Awareness – 4 votes
Service History Data – 4 votes
Veteran Status Verification – 0 votes
Statistical Analysis and Machine Learning – 0 votes
Metadata Support – 0 votes
Documentation – 0 votes
It’s entirely possible that these groupings could be further normalized, or even some of the ideas within the original groups split out into separate groups. Some groupings could even be disregarded as irrelevant (for example, Metadata Support). However, we didn’t want to dilute the results of what the participants came up with. Somewhat surprising is the low number of votes for Medical Health. That may be a result of lacking the right type of participant representation in the workshops, at least for that particular category.
Summary of the landscape analysis
The landscape analysis, which only processed about 1/3 of the URLs targeted for spidering over the course of a two-week period, revealed about 20,216 data files. This work produced a lot of data to wrangle and make sense of. Each of the data formats made available tell their own story about what types of data has been published to VA websites and why. The number of times a resource has been published using a particular data format (CSV, XML, JSON, XLS/XLSX, etc.) serves as a vote for making that resource available and accessible on the web.
Despite the huge amount of information to work with, we believe that our analysis provides valuable insight into some of the most relevant data resources, based on years of publication to VA websites. The top resources identified from all of the URLs, file formats, tables, and forms all point to data resources that should be considered turning into APIs. If these data resources were considered a priority when publishing to the VA’s websites, then there’s a good chance that they should be considered priorities when it comes to publishing via APIs as part of the Lighthouse initiative.
Lighthouse program considerations moving forward
Resources to prioritize
After spending time with all of the data uncovered during the landscape analysis, we began to see patterns emerge from across all the resources being published to the VA’s web properties, as well as those resources identified by people during our facilitated workshops. So based on analysis of the available data, we recommend that the VA Lighthouse program give consideration to prioritizing the following 25 resources:
Healthcare Facilities – Up-to-date information on hospitals, clinics, and other healthcare facilities.
Organizations – Details on any type of organization that services Veterans and their families.
Services – Services being offered by the VA, healthcare facilities, and other organizations.
Programs – Programs being offered by the VA, healthcare facilities, and other organizations.
Resources – Content, video, and other resources providing healthcare, outpatient, and other relevant content.
Schedules – The schedules of healthcare facilities, organizations, services, and programs being offered.
Events – Calendar and details of relevant events that service Veterans around the country.
Benefits – Details of the benefits being offered to Veterans, including elements of the process involved.
Performance – Performance details for the healthcare facilities, organizations, services, and programs.
Insurance – Home, auto, and healthcare insurance information that Veterans can take advantage of.
Loans – Information on home, auto, and other types of loans available to Veterans and their families.
Grants – Grants for education, businesses, projects, and other Veteran-focused efforts.
Education – Educational opportunities and information available to Veterans and their families.
Training – Specific training opportunities available that Veterans can take advantage of.
Jobs – Job postings that Veterans can apply to and use to guide their career.
Human Resources – VA human resources and related information in support of VA employees and Veterans.
Forms – Directory, access, and management of forms and the data that’s stored within them.
Budgets – Budget information on healthcare facilities, organizations, programs, and services.
Statistics – Statistics and data on all aspects of VA operations, and anything that impacts Veterans.
Cemeteries – Details of the cemeteries, and the Veterans who are laid to rest at all locations.
News – News that impacts Veterans from across any source and is relevant to the community.
Press – Press releases from the VA and related organizations and programs.
Research – Information and other resources produced as part of specific Veteran-related research.
Surveys – Centralized organization, access to, and the results of Veteran and program-related surveys.
FOIA – Process and information related to Freedom of Information Act (FOIA) efforts occurring at VA.
These resources represent what was harvested and analyzed as part of our landscape analysis, merging many of the patterns present across individual datasets. They’re organized using a REST-centric approach to turning data into API resources, which allows for data access via HTTP. Many of the keywords identified as part of the landscape analysis have been rolled-up into higher level areas — such as PTSD, mental health, and suicide — would exist across services, programs, and resources.
These suggested resources are derived from about 65% of the top-level resources identified across all the top paths, file formats, tables, and forms. They represent a nice cross-section of resources across all the data formats, but also reflect the general web presence of the VA. Our list also provides a coherent stack of resources that could be developed, deployed, and maintained in support of the central veteran APIs, offering personalized and generalized data experiences that would benefit Veterans and their families.
Centralize focus on the Veteran
From a data perspective, the most important resource above all is the Veteran and their personal data. Therefore, the identity and healthcare record of a Veteran should be at the front and center of any API deployed as part of the Lighthouse API platform initiative. This requires full knowledge and accurate information about a Veteran. In others words, in order for the Lighthouse’s APIs to work well, there must be a robust identity and access management in place, as well as detailed, layered, portable, and usable Veteran profiles.
One thing that became evident during our work is the need for greater personalization of data across almost every resource that we identified. Where there’s value in having general information available (for example, medical facilities), this data becomes exponentially more valuable when it’s personalized, localized, and made more relevant to the Veteran who is browsing, searching, engaging. Therefore, there are two types of engagement models with the resources that we propose: (1) general access without knowledge of the Veteran and (2) personalized knowledge of the Veteran via custom configuration settings that determine the relevancy of the data and content when these are made available via APIs and within applications.
There are existing portal efforts such as vets.gov that are available as part of the VA’s online presence. The personalization efforts occurring there should be reflected across the design and operation of the Lighthouse’s APIs. By designing APIs to operate in generalized or personalized mode, this would empower API developers to act on behalf of a Veteran using OAuth tokens. If a token’s present, each API will act in a more personalized manner and allow for localization based upon a Veteran’s preferences and history of interactions. This personalization layer should act as a bridge between the core healthcare record of a Veteran and the other resources that we outlined above.
Writing, not just reading data
Many of the resources that we identified represent read-only access to data and content. It’s important to note that getting access to data and content is useful; however, a significant portion of the resources that we harvested and gathered through conversations with the community will require the ability to write information via APIs. Forms, surveys, and other feedback loops will need to allow for APIs that not only GET data, but also POST and PUT data as part of their operations. This additional operations will round-off the Lighthouse’s stack of resources, helping to ensure that services provide a two-way street for engaging with the VA community.
In addition to reading, the ability to write data and content will be a deciding factor in whether applications built on top of the APIs will deliver meaningful value to Veterans and their supporters. If information is only being pushed outwards, many applications will be seen as having little to no value to users, developers, and operators of the Lighthouse API platform. To help ensure meaningful value is delivered to everyone involved, all applications should be capable of sharing usage data and feed analytics and support feedback loops between users and the platform operators. With the ability to write data, the APIs will lack meaning and substance, and will contribute to lack of adoption and integration.
Focus on the source of the data
A common misconception in conducting a landscape analysis such as the one we performed is to assume that the data discovered can be published via any APIs that are deployed as part of the next phase of work. That’s rarely the situation, because most of the discovered data is just published snapshots derived from existing data sources. This is certainly the case with the VA. Much of the data we discovered is unusable in its current state due to lack of normalization, duplication, being out of date, and other noise and clutter. Many of the XML and JSON files identified provider a much cleaner option for transforming into web APIs. However, with any resource identified, it’s more desirable to integrate the original source of data than relying on published snapshots.
Even after coming to a consensus on the data resources to transform into APIs, the next phase of work should focus on identifying the data sources for each of the targeted resource areas, and not relying on published data that already exist across websites. While it’s tempting, and sometimes necessary to rely on published data for the source of API data, it increases the chance that an API will eventually become dormant, out-of-date, and cause many of the issues that we’ve seen play out with the existing VA datasets. Our landscape analysis came at the resource prioritization from an external, public perspective. We recommend a subsequent, more internal landscape analysis to identify the data sources for important resource types emerging from this landscaping effort.
Improve domain management
Moving forward, it’d be logical to have a standardized approach to naming subdomains for both web and API properties in support of VA operations. Establishing a common approach to naming city, state, regional, program, research, and other resource areas would provide human- and machine-readable access to these resources. This might be difficult to do for web properties with so much legacy infrastructure, but the API platform provides an opportunity to establish a standardized approach moving forward.
Leverage common data formats
Our landscape analysis revealed a lack of a consistency when it comes to vocabulary, schema, and data formats. Most of the data published is derived from an existing system or represents a human-directed process. There’s a significant amount of fluff and noise surrounding these valuable data and a lack of consistent naming and field types.
Based upon the resources that we’ve identified for your consideration, there are a handful of existing data formats the VA should consider. Some of these are already underway, while others are not currently reflected in the Lighthouse’s efforts, but are used by other government entities to publish data in a consistent manner.
Fast Healthcare Interoperability Requirements (FHIR) – FHIR is already in-motion at the VA, but worth highlighting here. FHIR provides an anchor for why common data schema formats are relevant to other resources beyond Veteran healthcare records.
Open Referral (211) – Open Referral is a common schema and API specification for defining human services, including organizations, locations, and services, along with all the supporting information and metadata that goes with this core set of resources.
Open311 – Open311 is a common data format for reporting problems and issues at the municipal level, but can easily be adopted for establishing feedback loops at any level of government. It provides a common schema for how large volumes of information get submitted via API infrastructure.
Schema.org – A common schema vocabulary that provides object definitions for almost every resource identified throughout this landscape analysis, and the recommended list of resources above.
There are undoubtedly other open data formats that can be leveraged. Common microformats and other RFCs should also be considered, but these can be addressed during the define and design stages of the API development lifecycle, once individual resources have been decided upon. Common formats help ensure resources are interoperable and reusable across VA groups; they also help bring teams together to speak in a common language, using a common dictionary, which will go a long ways to standardizing how data is published and consumed.
Improve analytical information
We had hoped there would have been more analytical information available to help rank the resource data that we identified. We did incorporate the ranking information available from data.gov as an input into our resource prioritization. However, we relied mostly on publication frequency and the overall occurrence of each keyword to help weight relevance. The existence of a word in a path, title, and file name gives it a weight, which can be amplified for every occurrence, providing us with adequate levels of prioritization, grouping, and organization to help us understand each topics importance. If a topic exists frequently across VA web properties, and exists as a sectional grouping, and title of data file, it has importance and relevance.
The lack of analytics, or access to current analytics, across existing VA data sources demonstrates the importance of having a consistent and comprehensive analytics strategy across the VA’s data. There should be download counts for all machine-readable files. And, more importantly, real-time analytics for the consumption of this data via simple web APIs. There should be regional- and program-related ata. We should have personalized data that reflects what’s most important to Veterans. We should understand what’s relevant what isn’t through strategically-designed analytics across web and API operations. The lack of analytics is why we’re working to identify relevant data sources, so those can be made more available and analytics become the default — not an afterthought.
Continue refining the landscape analysis
Our landscape analysis produced a lot of information that was messy and difficult to work with. We can continue to make another pass, which would involve refining indexes, optimizing title and filename parsing, and developing key phrase, plural word, and other dictionaries to make the results much more refined.
There was a lot of data to harvest, process, and make sense of in a two-week sprint. However, we feel that we were able to do a good job of making sense of what was captured. Another sprint could easily be spent sorting through all of the data targeted, separating quality datasets from the more messier ones. Creating a dictionary to translate words and rehydrate acronyms would be very useful to help make further sense of what’s available in CSV, XLS/XLSX, JSON, and XML files. More work could also be done around forms: identifying the types available; defining their search mechanisms; defining what input parameters they allow (whether GET or POST); and unlocking further details on how they store data. Form and table data often have a direct connection to backend databases, which make them more valuable than some of the published data files.
All of the data from the landscape analysis has been published to GitHub, minus the primary index of harvested and processed URLs. Those are too big to publish as JSON to GitHub, but we’ll evaluate how best to provide access to each site index using a solution such as Amazon AMI. We also started experimenting with a secondary spider solution in order to generate an index of the VA website and can publish those indexes as separate GitHub repositories within a single GitHub repository when completed. We feel like these newer indexes could provide a much richer approach to understanding the data and content across the VA’s web properties. And allow for other researchers and analysts to fork and work to make sense of the data that they contain.
Incorporate user research
It’s critical that any further landscape analysis focused on uncovering valuable data resources from across the VA’s web presence is combined with user research activities, such as the series of design workshops that we conducted. Doing so will provide a human perspective on what’s most important to Veterans and their supporters.
We strongly encourage the Lighthouse program to conduct a similar workshop activity to the one that we ran, leveraging the VA’s much stronger outreach capability in order to attract an even larger and more diversified representation of the Veteran community’s data resource needs.
We also recommend that the Lighthouse program consider using the service blueprinting technique as a way to help identify and prioritize specific APIs for deployment. For example, a service blueprint could be created for a specific interaction that Veterans have with the VA, such as trying to find information on healthcare facilities. It’s likely that any service blueprints you want to create could be acquired by chunking the work into multiple micro-purchases. At the very least, we recommend trying to do at least one as an experiment. Once specific APIs are identified, you could then map them against a 2x2 prioritization matrix, based on how high they score against two main criteria: Veteran Experience Impact (y-axis) and Readiness to Execute (x-axis).
Conclusion: this journey is just beginning
The landscape analysis for the VA doesn’t end here. Just like the resulting API effort, the evaluation of the VA’s web presence should be an ongoing process. Work should continue to help identify what datasets are being published to the VA’s web properties, and to incorporate these datasets into API operations or to replace them with API-driven solutions. In the long term, there shouldn’t be any tables, forms, CSV files, JSON files, XML files, or XLS/XLSX files without a direct connection to the API platform. Eventually all data should be derived from a federated, but standardized, set of API platforms that are designed, deployed, and managed consistently as part of the VA Lighthouse effort.
Hopefully the work conducted here provides a base of resources for the Lighthouse program to consider as it moves forward. Ideally, everything uncovered as part of this work eventually becomes an API, or part of a suite of APIs. We understand that this won’t be a reality anytime soon, but we worked diligently to uncover the most valuable resources and to provide a concise list of data resources that could be turned into APIs and used to begin driving web, mobile, and desktop applications that serve Veterans and their families. There’s a wealth of resources available to Veterans across the VA’s websites. The challenge now is how do we ensure these resources deliver value consistently across many platforms? A simple, consistent, and usable API stack is the answer.
Lessons to share
This project is one of the VA’s first experiments using the microconsulting model in support of the Lighthouse initiative. Sharing what went well, what didn’t, and what could we have been done better — all in the name of continuous improvement — is the responsibility of everyone involved in order to make not only the Lighthouse initiative a success, but the microconsulting model as well. With that said, here’s what we have to share:
We thoroughly enjoyed working on this as a micro-project. We felt that the short-timeboxed, tightly-scoped nature of the work focused our efforts on executing only the most essential activities, giving even more meaning to inherently impactful work. As Parkinson’s Law states, “work expands so as to fill the time available for its completion.”
Looking back, our approach to this project involved some known unknowns (and some unknown unknowns) from a technical standpoint. In particular, the question of how well our spidering process could scale to handle the VA’s enormous web footprint. It would have been best to propose conducting an agile “spike” activity as a small micro-project in order to gain risk-reducing knowledge.
For micro-projects under a tight schedule and for which there are external dependencies (for example, scheduling interviews or workshops with external participants), some lead time may be necessary before formally kicking off the project.
Those people who participated in our facilitated workshops expressed extreme gratitude for the opportunity to contribute to the progress of the Lighthouse program. Working in the open and co-creating with the public will not only foster an engaged community of supporters, but will also lead to better quality outcomes.
While our workshop activities were extremely valuable in giving us a human perspective on our landscape analysis, we felt that there could have been even greater representation from the Veteran community. We should have been more proactive about leveraging the VA’s outreach capability to draw in an even more dense and diverse group of Veterans and their supporters.
In my world, OpenAPI is always a primary actor, and the tooling and services that put it to work are always secondary. However, I’d say that 80% of the people I talk with are the opposite, putting OpenAPI tooling in a primary role, and the OpenAPI specification in a secondary role. This is the primary reason that many still see Swagger tooling as the value, and haven’t made the switch to the concept of OpenAPI, or understand the separation between the specification and the tooling.
Another way in which you can see the importance of OpenAPI tooling is the slow migration of OpenAPI 2.0 to 3.0 users. Many folks I’ve talked to about OpenAPI 3.0 tell me that they haven’t made the jump because of the lack of tooling available for the specification. This isn’t always about the external services and tooling that supports OpenAPI 3.0, it is also about the internal tooling that supports it. It demonstrates the importance of tooling when it comes to the evolution, and adoption of OpenAPI. It demonstrates the need for the OAI community to keep investing in the development and evangelism of tooling for the latest version.
I am going to work to invest more time into rounding up OpenAPI tooling, and getting to know the developers behind them, as I prepare APIStrat in Nashville, TN. I’m also going to invest in my own migration to OpenAPI 3.0. The reason I haven’t evolved isn’t because of lack tooling, it is because of a lack of time, and the cognitive load involved with thinking new ways. I fully grasp the differences between 2.0 and 3.0, but I just don’t have intuitive knowledge of 3.0 in the way I do for 2.0. I’ve spent hundreds of hours developing around 2.0, and I just don’t have the time in my schedule to make similar investment in 3.0–soon!
If you need to get up to speed on the latest when it comes to OpenAPI 3.0 tooling I recommend checking out OpenAPI.Tools from Matt Trask (@matthewtrask) and Crashy McCiderface (aka Phil Sturgeon) (@philsturgeon). It is the best source of OpenAPI tooling out there right now. If you are still struggling with the migration from 2.0 to 3.0, or would like to see a specific solution developed on top of OpenAPI 3.0, I’d love to hear from you. I’m working to help shape the evolution of the OpenAPI tooling conversation, as well as tell stories about what tools are available, or should be available, and how they are can be put to work on the ground at companies, organizations, institutions, and government agencies.
After eight years of educating people about sensible API security and management, I’m always amazed at how many people I come across who still think public web APIs are about giving away access to your data, content, and algorithms for free. I regularly come across very smart people who say they’d be doing APIs, but they depend on revenue from selling their data and content, and wouldn’t benefit from just putting it online for everyone to download for free.
I wonder when we stopped thinking the web was not about giving everything away for free? It is something I’m going to have to investigate a little more. For me, it shows how much education we still have ahead of us when it comes to informing people about what APIs are, and how to properly manage them. Which is a problem, when many of the companies I’m talking to are most likely doing APIs to drive internal systems, and public mobile applications. They are either unaware of the APIs that already exist across their organization, or think that because they don’t have a public developer portal showcasing their APIs, that they are much more private and secure than if they were openly offering them to partners and the public.
Web API management has been around for over a decade now. Requiring ALL developers to authenticate when accessing any APIs, and the ability to put APIs into different access tiers, limit that the rate of consumption, while logging and billing for all API consumption isn’t anything new. Amazon has been extremely public about their AWS efforts, and the cloud isn’t a secret. The fact that smart business leaders see all of this and do not see that APIs are driving it all represents a disconnect amongst business leadership. It is something I’m going to be testing out a little bit more to see what levels of knowledge exist across many fortune 1000 companies, helping paint of picture of how they view the API landscape, and help me quantify their API literacy.
Educating business leaders about APIs has been a part of my mission since I started API Evangelist in 2010. It is something that will continue to be a focus of mine. This lack of awareness is why we end up with damaging incidents like the Equifax breach, and the Cambridge Analytica / Facebook scandal. Its how we end up with so many trolls on Twitter, and an out of balance API ecosystems across federal, state, and municipal governments. It is a problem that we need to address in the industry, and work to help educate business leaders around common patterns for securing and managing our API resources. I think this process always begins with education and API literacy, but is a symptom of the disconnect around storytelling about public vs private APIs, when in reality there are just APIs that are secured and managed properly, or not.
It is APIStrat time again! This time it is in Nashville, Tennessee! We are in the early stages of the event, but we are getting close to the deadline of the call for papers. We’ve assembled another rockstar ensemble for this round to help us steer the event, and review talk submissions once the CFP process has closed. I just wanted to take a moment and recognize the folks who are helping out and make sure they get the recognition they deserve.
First up are the six members of the APIStrat steering committee, playing different leadership roles in the conference, making sure everything gets done by September:
- Darrel Miller (@darrel_miller)– Workshop Chair - Microsoft
- Erin McKean (@emckean) – Program Chair - IBM
- Kin Lane (@kinlane) – Marketing Chair - API Evangelist
- Lorinda Brandon (@lindybrandon) – Community Chair - Capital One DevExchange
- Steven Willmott (@njyx) – Chair - Red Hat.
- Tim Burks (@timburks) – Program Vice Chair - Google
Then we have assembled nineteen folks on the program committee who will be reviewing your talk submissions before you can get on stage at APIStrat in Nashville:
- Amy Palamountain (@ammeep) - GitHub
- Ash Hathaway (@Ash_Hathaway) - Pivotal
- Carly Thelander (@CarlyNThelander) - MilkRun
- David Biesack (@davidbiesack) - Apiture
- Gail Frederick (@screaminggeek) - eBay
- George Atala (@gmatala) - Quantero Capital
- Glenn Block (@gblock) - Auth0 https://auth0.com/
- Gregory Koberger (@gkoberger) - ReadMe
- James Higginbotham (@launchany) - LaunchAny
- Jennifer Riggins (@jkriggins) - the eBranding Ninja
- Joey French (@josephtfrench) - Intrinio
- Kyle Dallaire (@kyledallaire) - Capital One Digital
- Melissa Jurkoic (@melissa_jurkoic) - Amadeus Hospitality
- Phil Sturgeon (@philsturgeon) - WeWork
- Sai Vennam (@sai_vennam) - IBM https://www.ibm.com/
- Shelby Switzer (@switzerly) - Healthify
- Skip Hovsmith (@skiphovsmith) - Critical Blue
- Taylor Barnett (@taylor_atx) - Stoplight
- Tessa Mero (@TessaMero) - Cisco
- Yina Arenas (@yina_arenas) - Microsoft Graph
Than you to everyone for helping do the hard work of making sure APIStrat not only continues, but continues to represent the wider API community. Everyone is doing this work because they care about the community, and want to make the event as good as, or better than it has been in the past. This is the 9th edition of APIStrat, spanning New York, San Francisco, Amsterdam, Chicago, Berlin, Austin, Boston, Portland, and now Nashville! It has been a pretty wild ride.
While we have everyone we need for these committees, we still need help in other areas. First, get your talk submitted before the CFP closes next week. Second, we need your financial support, so make sure you consider sponsoring APIStrat, and help make sure Nashville rocks. Beyond that we can use some help spreading the word. We are looking to grow the event beyond the usual 500 threshold, helping expand participation in the event, as well as the OpenAPI Initiative. If you want to help, feel free to ping me anytime, and I’ll see you in Nashville.
I’m kicking off a busy week of travel and talks this week in DC with a discussion about delivering microservices at federal agencies at DevNation Federal on Tuesday, June 5th, 2018. I was invited by Red Hat to come speak about the work I’m doing as API Evangelist across federal agencies. You can find me in the afternoon lineup, sharing my stories title “The Tech, Business, and Politics of APIs In Federal Government”. Focusing on information gathered as part of my research, workshops, and consulting across the public and private sector.
My talk reflects my work to motivate federal agencies to do APIs over the last five years, and help pollinate the ideas and practices I gather from across the private sector, and understand which ones will work in the public sphere. Not everything about doing APIs at startups and in the enterprise translates perfectly to delivering APIs in the federal government, but there are many practices that will help agencies better serve the people. My goal is to open up discussion with government employees and contractors, to help figure out what works and what doesn’t–sharing stories along the way.
Let me know if you are going to be at DevNation Federal let me know. I am happy to make some time to talk, and hear what you are up to with APIs. I depend on these hallway conversations to populate my blog with stories. I’ll be in town around noon, and there until around 5 or so until I head over to the DC API Meetup for my second talk of the day. Thanks to Red Hat for having me out. I enjoy doing talks for Red Hat events, as they tend to reflect more of the audience I’m looking for, with a focus on more open source, and a little less proprietary focus when it comes to delivering government technology. I’ll see you in Washington D.C. on Tuesday!
After I speak at DevNation Federal in Washington DC this Tuesday, I am going to give a similar talk at the DC API API User Group that evening. I love going to the Meetups in DC, partly because my good friend Gray Brooks runs the event, but also because I’ve been working to jumpstart API conversations in Washington DC since 2012 when I held the first DC edition of API Craft. I was on a mission to jumpstart API Craft gatherings around the country that year, and it makes me happy to see the API Meetup culture continuing to thrive in DC, where other places it has died out.
At the DC API Meetup I’ll be giving a variation of my talk that I’m giving earlier that day at DevNation Federal. Talking about the technology, business, and politics of doing APIs, with an emphasis on a consistent and repeatable API lifecycle. I’ll be reworking my regular material in light of current projects I’m working on at the federal level including with the VA, FDIC, HHS, and beyond. Sharing stories about how a microservice approach can help make government services more agile, flexible, and delivered in smaller more bite sized chunks–helping move the IT conversation forward across federal agencies.
If you can’t make it to DevNation Federal, I recommend you head out to the DC API User Group later that evening. I’d love to get a chance to hang out with you and talk about APIs. I’m always impressed with the folks who turn out for the DC API Meetup, consistently providing a fresh opportunity to discuss APIs and the impact they are making across the federal government. Organizer Gray Brooks has his finger on the pulse of what is going on across agencies, way beyond what I am capable of from the outside-in. I look forward to hanging out in DC, and hope you can make it out Tuesday to talk some APIs with me.
I’ve been evaluating API management providers, and this important stop along the API lifecycle in which they serve for eight years now. It is a space that I’m very familiar with, and have enjoyed watching it mature, evolve, and become something that is more standardized, and lately more commoditized. I’ve enjoyed watching the old guard (3Scale, Apigee, and Mashery) be acquired, and API management be baked into the cloud with AWS, Azure, and Google. I’ve also had fun learning about Kong, Tyk, and the next generation API management providers as they grow and evolve, as well as some of the older players like Axway as they work to retool so that they can compete and even lead the charge in the current environment.
I am renewing efforts to study what each of the API management solutions provide, pushing forward my ongoing API management research, understanding what the current capacity of the active providers are, and potentially they are pushing forward the conversation. One of the things I’m extremely interested in learning more about is the connector, plugin, and extensibility opportunities that exist with each solution. Functionality that allows other 3rd party API service providers to inject their valuable services into the management layer of APIs, bringing other stops along the API lifecycle into management layer, allowing API providers to do more than just what their API management solution delivers. Turning the API management layer into much more than just authentication, service plan management, logging, analytics, and billing.
Over the last year I’ve been working with API security provider ElasticBeam to help make sense of what is possible at the API management layer when it comes to securing our APIs. ElasticBeam can analyze the surface area of an API, as well as the DNS, web, API management, web server, and database logs for potential threats, and apply their machine learning models in real time. Without direct access at the API management layer, ElasticBeam is still valuable but cannot respond in real-time to threats, shutting down keys, blocking request, and other threats being leveraged against our API infrastructure. Sure, you can still respond after the fact based upon what ElasticBeam learns from scanning all of your logs, but without being able to connect directly into your API management layer, the effectiveness of their security solution is significantly diminished.
Complimenting, but also contrasting ElasticBeam, I’m also working with Streamdata.io to help understand how they can be injected at the API management layer, adding an event-driven architectural layer to any existing API. The first part of this would involve turning high volume APIs into real time streams using Server-Sent Events (SSE). With future advancements focused on topical streaming, webhooks, and WebSub enhancements to transform simple request and response APIs into event-driven streams of information that only push what has changed to subscribers. Like ElasticBeam, Streamdata.io would benefit being directly baked into the API management layer as a connector or plugin, augmenting the API management layer with a next generation event-driven layer that would compliment what any API management solution brings to the table.
Without an extensible connector or plugin layer at the API management layer you can’t inject additional services like security with ElasticBeam, or event-driven architecture like Streamdata.io. I’m going to be looking for this type of extensibility as I profile the features of all of the active API management providers. I’m looking to understand the core features each API management provider brings to the table, but I’m also looking to understand how modern these API management solutions are when it comes to seamlessly working with other stops along the API lifecycle, and specifically how these other stops can be serviced by other 3rd party providers. Similar to my regular rants about API service providers always having APIs, you are going to hear me rant more about API service providers needing to have connector, plugin, and other extensibility features. API management service providers can put their APIs to work driving this connector and plugin infrastructure, but it should allow for more seamless interaction and benefits for their customers, that are brought to the table by their most trusted partners.
I conducted an API workshop at Genscape in Boston yesterday. I thoroughly enjoyed the conversation with the technical folks there, and found their pragmatic, yet educated views on APIs refreshing. I spent the day going through the usual stops along the API lifecycle, but found some of our breakout conversations during the API design section to be what has stuck with me on my train ride back to New York. Specifically the discussions around versioning, content negotiation and the overall design of paths, parameters, and potentially providing query language layers on top of APIs.
After going through much of my textbook API design patterns for paths, and parameters, then working my through content negotiation and headers, we kept finding ourselves talking about what their clients were capable of. Bringing the API design discussion back around to why REST is still dominate in my opinion–simplicity, and lowering the bar for consumers. Time after time we found ourselves talking about their target consumers, the spreadsheet wielding data analyst, and how the bar needed to be kept low to make sure their needs were being met. Sure, we can quickly get academic and theoretical with the design practices, but if they fall on deaf ears, and API consumers do not adopt and use the API-driven tools, does it matter?
I know that all of us API “thawt liters” would rather folks just do things the right way, but this isn’t always the reality on the ground. Most people within real world businesses don’t have the time or luxury to learn the right way of doing things, and take the time to disrupt their flow with new ways of doing things. Most of the time we need to just get people the data and other resources they need, in the tooling they are comfortable with (spreadsheet cough cough), and get out of their way. Sure, we should still be introducing people to new concepts, and working to strike a balance when it comes to the API design patterns we adopt, but it should be about striking a balance between reality on the ground, and the future we want to see.
During the workshop we kept coming back around to simple, plain language, URIs that provide flexibility with path variables, as well as potentially relevant query parameters, that allow people to get CSV and JSON representations of the resources they need. With Excel as the target client, once again I find myself minimizing header usage, and maximizing the paths, and simplifying the data models in which folks can retrieve via APIs. I see this in industry after industry, and across the government agencies I am working with. I know that we all want our APIs to reflect the best possible patterns, and leverage the latest technology, but many of the people who will be potentially consuming our APIs, just want to get at the resources we are serving up, and do not have the time and the bandwidth to get on board with anything too far beyond their current mode of getting business done.
None of this will shift me evangelizing the “proper way to design APIs”, but it reminds me (once again), at how immovable the business world can actually be. APIs are having a significant impact on how we develop web, mobile, desktop, and device applications, but one of the reasons web APIs have found so much success is that they are simple, scrappy, and flexible. Being able to serve up valuable data and other resources through simple URLs, allowing anyone to take them and put to use in the application of their choice has fed the explosion in the API we see across the landscape today. It is why we still see just as many poorly designed APIs in 2018 as we saw in 2010. It is because the best design doesn’t always win. Sometimes you just need the right design for the job, and the one that will make sense to the audience who will be consuming it. It is why hypermedia, GraphQL, and other design patterns will continue to be the choice of some practitioners, but simple, plain, REST will keep dominating the landscape.
A question I get regularly in my API workshops is, “how should we be organizing all of our microservices?” To which I always recommend they tune into what the API Academy team is up to, and then I dance around give a long winded answer about how hard it is for me to answer that. I think in response, I’m going to start asking for a complete org chart for their operations, list of all their database schema, and a list of all their clients and the industries they are operating in. It will still be a journey for them, or me to answer that question, but maybe this response will help them understand the scope of what they are asking.
I wish I could provide simple answers for folks when it came to how they should be naming, grouping, and organizing their microservices. I just don’t have enough knowledge about their organization, clients, and the domains in which they operate to provide a simple answer. It is another one of those API journeys an organization will have to embark on, and find their own way forward. It would take so much time for me to get to know an organization, its culture, resources, and how they are being put to use, I hesitate to even provide any advice, short of pointing them to what the API Academy team publishes books, and provides talks on. They are the only guidance I know that goes beyond the hyped of definition of microservices, and actually gets at the root of how you do it within specific domains, and tackle the cultural side of the conversation.
A significant portion of my workshops lately have been about helping groups think about delivering services using a consistent API lifecycle, and showing them the potential for API governance if they can achieve this consistency. Clearly I need to back up a bit, and address some of the prep work involved with making sure they have an organizational chart, all of the schema they can possibly bring to the table, existing architecture and services in play, as well as much detail on the clients, industries, and domains in which they operate. Most of my workshops I’m going in blind, not knowing who will all be there, but I think I need a section dedicated to the business side of doing microservices, before I ever begin talking about the technical details of delivering microservices, otherwise I will keep getting questions like this that I can’t answer.
Another area that is picking up momentum for me in these discussions is a symptom of of the lack of API discovery, and directly related to the other areas I just mentioned. You need to be able to deliver APIs along a lifecycle, but more importantly you need to be able to find the services, schema, and people behind them, as well as coherently speak to who will be consuming them. Without a comprehensive discovery, and the ability to understand all of these dependencies, organizations will never be able to find the success they desire with microservices. They won’t be any better than the monolithic way many organizations have been doing things to date, it will just be much more distributed complexity, which will achieve the same results as the monolithic systems that are in place today.
I know, time flies and we are all very busy, but have you submitted your talk for APIStrat Nashville? The call for papers closes next Friday, June 8, 2018 at 11:59 PM PST! This is the second edition of the conference being run by the OpenAPI community, and is something you aren’t going to want to miss. We are bringing together the usual API community suspects, but also working to build upon the work that is going on in the fast growing OpenAPI community. The conference continues to be its usual voice of the API community, and won’t be all about the OpenAPI specification, but it it will definitely be where you want to be showcasing any groundbreaking work you are doing around the specification.
The CFP process for APIStrat Nashville asks three main questions:
- How will the audience benefit from your presentation?
- Why should YOU be the one to give this talk? You have a unique story. Tell it.
- Be prepared to explain how this fits into the API ecosystem.
After answering these questions we are looking for talks in the following areas of doing APIs: API Design, API For the Greater Good, API Management, API SDKs & Clients, API Security, API Success Stories, API Transformations, API Testing, API Usability, At The Protocol Level, Digital Transformation, GraphQL, Hypermedia, Machine Learning, Microservices, REST, RPC systems, and Standards & Definitions. However, don’t let these topics put you in a box. We want you to think about whatever you think will make the biggest impact on the audience, and hopefully the wider API community.
Do not wait until the last moment to get your talk submitted! Last year, as the program chair I spoke with a number of people who had waited too long, and didn’t get their talk in front of the program committee early on. As soon as the submission are close they will begin spending time reviewing the talks, and while we may consider accepting late submissions, they are unlikely to get the attention that the earlier submissions will. If you have any questions about past events, or what might speak to this years audience, feel free to reach out to me personally. While I’m not in charge of the program committee this year, I’m still happy to help answer questions, and share my thoughts on what will catch their attention. Here is the link to the APIStrat CFP page for when you are ready, and I look forward to hearing what you have to say this year in Nashville!
I’m kicking off a micro-procurement project with the Department of Veterans Affairs (VA) this week. I’m going to to be conducting one of my API low hanging fruit campaigns for them, where I help them identify the best possible data sets available across their public websites for turning into APIs. The project is one of many small projects the federal agency is putting out there to begin working on their agency wide API platform they are calling Lighthouse. Laying the groundwork for better serving veterans through a robust stack of microservices that each do one thing really well, but can be used in concert to deliver the applications the agency needs to meet their mission.
While not specifically a project to develop a microservice. The landscape analysis project is an API focused research project, that is testing a new procurement model called microprocurement. At my government consulting agency partnership Skylight Digital, my partner in crime Chris Cairns has been pushing for a shift in how government tackles technology projects, pushing them to do in smaller chunks that essentially can be put on the credit card in less than 10K increments. So far we’ve been doing consulting, research, and training related projects like how to create a bug bounty program, and internal evangelism strategies. Now we are kicking our campaign into high gear and pushing more agencies to think about microprocurement for microservices–the VA was the first agency to buy into this new way of thinking about how government IT gets delivered.
I am working with all of my partners to help me craft potential services that would fit into the definition of a microprocurement project. I’ve been working to educate people about the process so that more API experts are on hand to respond when agencies publish microprocurement RFPs on Github like the VA did, but I also want to make sure and have a suite of microprocurement packages that federal agencies can choose from as well. I’ve been working with my partner Dave O’Neill over at APIMetrics to provide me with some detail on an API testing microprocurement package that federal agencies could buy into for less than 10K, providing the follow value:
- API discovery and creation of a Postman collection
- Annual monitoring for up to 10 APIs including the following:
- Weekly and monthly emailed quality statements – CASC scores and SLOs/SLA attainment
- Interface to named operations tooling for alerts
- Public dashboards for sharing API status and SLA with all stakeholders.
Providing a pretty compelling annual API monitoring package that government agencies could put on the credit card, and help ensure the quality of service provided across federal agencies when it comes to providing APIs. Something that is desperately needed across federal agencies who in my experience are mostly conducting “API nonitoring”, where they are not monitoring anything. Ideally, ALL federal agencies provide an SLA with the public and private APIs they are serving up, and relying on outside entities like APIMetrics to be doing the monitoring from an external perspective, and potentially even different regions of the US–ensuring everyone has access to government services.
I’ll publish anther story after I kick off my work with the VA. I’ll also keep beating the drum about microprocurement opportunities like this one with APIMetrics. If you are an API service provider who has an interesting package you think federal agencies would be interested in–let me know. Also, if you are a government agency and would like to learn more about how to conduct microprocurement projects feel free to drop me a line, or reach out to Chris Cairns (@cscairns) over at Skylight Digital–he can set you up. We are just getting going with this types of projects, but I’m pretty optimistic about the potential they can bring to the table when it comes to delivering IT projects–an approach that reflects the API way of doing things in small, meaningful chunks, and won’t break the bank.
Most of this API stuff isn’t technical. For APIs to work at scale within a single company, a wider industry, or across government agencies, you need people who are committed to evangelism, coaching, and communication around everything that is occurring across API ecosystems. We often times get caught up in our work, and operating within our silos and forget to email, call, and just walk next door sometimes to share stories of what is going on. If you are a regular reader of my work you know how I feel about storytelling, and just how important it is to all of this working or not. Which is why I like to make sure that I showcase the storyelling of other evangelists who are working their magic, and spreading API knowledge within their domains.
I’m a big fan of what the GSA is up to with APIs across the federal government, and specifically what my hero Gray Brooks does to support API efforts acorss multiple federal agencies. He does a lot of legwork, but one thing that has stood out for me is his work to establish an API community of practice, and share relevant API stories on the US Government APIs Google Group. To help highlight what he is up to, and hep amplify the knowledge he puts out there, I wanted to share his latest post from the Google Group in it’s entirety because it is, well, beautiful.
I hope your week has wrapped up well so far. As I mentioned back in April, this and other efforts have been formalized as the API Community of Practice, part of Digital.gov's communities of practice program. I hope you all will put this forum to use, sharing your thoughts, questions, and ideas about federal APIs you're working on or thinking about. In the meantime, here's some odds and ends that you all may find interesting.
- One of the API programs here at GSA put out a call for developer input ahead of some further work on the sam.gov API. It's not surprise that the wonderful API Evangelist took note and offered both praise and some wise counsel. It's well worth a read and offers good advice to us all, if we'll take it.
- FOIA.gov launched its developer hub, which has a lot to it, including the ability to submit a FOIA request via API.
- If you want to see a really neat set of examples of developer engagement from an agency, check out the pair of A+ listserves set up by CMS for the Blue Button and Quality Payment Program APIs. This is a great example for us all to follow.
- Here are some APIs that are either new (or at least are new to me!) - NASA's Mission Design API, CPSC's Violations & NEISS API, and USDA's MARS API.
- A digital services team in the New Zealand government is keeping a public list of NZ government APIs as they find them.
- Also, it's just worth noting the success of CFPB's open source API documentation project, DOCter. That good work not only was used for their API documentation, but has been re-used by GSA for its API documentation template and is now employed by many of the agency's APIs. Regulations.gov and Interior are two more agencies putting it to good use. A good . reminder for us too all just reuse each other's stuff! (Interested in more examples you can quickly and easily fork and re-use? Here's a bunch more.)
- NIST including API requirements and specs in some its efforts.
Last but not least, there's a bunch to see from VA and HHS/CMS that's a pretty big deal. This is the first time that I've been seeing agency principles actively talking and giving speeches about APIs and API strategies. This is 100% what needs to be happening across government and I def. recommend diving into these efforts and trying to drive the same at your agencies. VA is the largest civilian agency in the federal government. If they can be doing this, so can any of us! VA
- Back in March, the VA secretary gave a speech driving an open API pledge among healthcare providers. More coverage here.
- VA has launched a new developer hub around the effort, called Lighthouse.
- One cool sub-project is a set of micropurchases they are trying in this area.
- The National Coordinator for Health Information Technology at HHS is writing blogposts like "APIs: A Path to Putting Patients at the Center" and a senior advisor to him is writing to announce a Secure API Server challenge that gave out $50,000 in prizes.
- Meanwhile, the CMS administrator gave a big speech about APIs at the same event as the VA secretary, also worth checking out.
- As part of those efforts, the US Digital Service team at CMS helped roll out v2 of the Blue Botton API.
- More on this rollout here and here. The first link has a great interview I strongly recommend.
Anyhoo, I'm sure there's much more going on that this. Definitely share anything you all are finding or working on as it comes up. In the meantime, take care can have a good holiday weekend.
"name": "api.data.gov - API analytics for federal agencies",
"name": "US Government API listserve",
It is tough to directly measure the ROI from the work Gray Brooks does in a purely business sense, but this type of outreach and storytelling is ESSENTIAL to APIs working across federal agencies. Gray is paying attention to what is happening across federal agencies and doing the hard work to share relevant stories across teams, agencies, and with the general public. While it would be valuable for each federal agency to do this themselves, it isn’t something that is always in the DNA of each agency, and many teams just don’t have the time or resources to publish stories on a regular basis–making Gray’s work all the more important to the overall health of APIs at the federal level.
We need more Gray Brooks. We tried to clone him at a super top secret agency and failed. He is just too special. His work reflects what needs to exist at every single federal government agency, as well as at the state and municipal levels. We also need this kind of dedicated evangelism and storytelling to exist within industries with healthcare, education, energy, and other specialized API evangelist uncovering the meaningful stories, and working to make them more public. If you aren’t tuned into the Government API Google Group I recommend tuning in. If you don’t know Gray Brooks I recommend going to one of the DC Web API User Group which Gray operates–you’ll likely get a chance to meet him in person, and hear stories of the API change he is making first hand.
I was on a call for a federal government API platform project with my partner in crime Chris Cairns (@cscairns) of Skylight.Digital. We were back channeling in our Slack channel during the call, when he said, “I always imagine you participating in these things, finding topics you haven’t covered or emphasized from a certain angle, and then writing a blog post in real time.” He was right, I had taken notes on a couple of new angles regarding the testing, monitoring, and understanding performance of APIs involved with federal government projects.
The single conference call resulted in about seven potential stories in my notebook. I’m guessing that only four of them will actually end up being published. However, the conversation does a good job at highlighting my style for generating stories on the blog, which is something that allows me to publish 3-5 posts a day–keeping things as active as I possibly can, generating traffic, and bringing attention to my work. If you’ve ever worked closely with me, you know that I can turn anything into a story, even a story like this, about how I create stories. ;-)
Think of API Evangelist as my public workbench where I work through the projects I’m tackling each day. It is how I make sense of my research, the projects I am working on, and the conversations I am having. All of which generates SEO exhaust, which brings more attention to my work, extends its reach, and ultimately brings in more work. It is a cycle that helps me work through my ideas in a way that forces me to make them more coherent (hopefully) along the way, while also making them immediately available to my partners, customers, and readers.
Chris has been pushing this concept forward and advocating that we be public with as many of the Skylight.Digital responses and proposals as we possibly can, which resulted in me publishing both of our responses to the first, and the second Department of Veteran Affairs (VA) RFIs. He has also been pushing other Skylight.Digital partners to be public with their proposals and responses, which folks were skeptical about at first, but then it started making sense when they see the effect it has, and the publicity it can generate.
This approach to storytelling isn’t something you can do effectively right out of the gate. It takes practice, and regular exercising–I have eight years of practice. However, I feel it is something that anyone can do eventually. You just have to find your own way of approaching it, and work on establishing your own voice and style. It is something that I’ve found to be essential to how I do business, but also something I find to be personally rewarding. I enjoy working through my ideas, and telling stories. It is always the one aspect of my work that I look forward to doing each day, and I feel like something is missing the days I don’t get to craft any posts. I really enjoy that every moment has the potential to be an API story, it really makes each day an adventure for me.
I have been intrigued by the people who reached out after I published a story about my partner program. One of the companies that reached out was the API integration agency Bridge Software, who reflects the next generation of software development groups who are emerging to focus exclusively on API integration.
I’ve had several calls with the Bridge Software team this week, discussing the possible ways in which we can work together. I’m regularly in need of software development resources that I can refer projects to, and I’m also interested in hearing more interesting stories about how businesses are integrating with APIs. Of course, Bridge Software is looking for more customers, which is something I can definitely help with. So we started brainstorming more about how they can regularly feed me stories that I can develop into posts for API Evangelist, and I can help send new business their way.
One of the most valuable resources I possess as the API Evangelist, is access to real world stories from the trenches of API providers and integrators. These sources of stories have been the bread and butter for API Evangelist for eight years. I have depended on the stories that businesses share with me to generate the 3,276 posts I have published on the blog since 2010. So I am stoked to have a new source of fresh stories, from a team who is neck deep in API integrations each day, and will be actively gathering stories from their regular developer team meetings.
I’m looking forward to working with the Bridge Software team, and getting new stories from the API integration trenches. I’m also looking forward to assisting them with finding new customers, and helping them with deliver on their API integration needs. If you are looking for help with API integrations, Bridge Software has a pretty sophisticated approach to delivering integrations that has test driven development at its core. Feel free to reach out to the team, but make sure and tell them where you heard about them, to get special treatment. ;-) Also, make sure you stay tuned for more stories about the API integrations stories they will be sharing with me on a regular basis, which I will be weaving into my regular cadence of storytelling here on the blog.
I’m working to define APIs at the Department of Veterans Affairs (VA) on several fronts right now. I’ve provided not just one, but two detailed reponses to their Lighthouse API platform RFIs. Additionally, I’m providing guidance to different teams who are working on a variety of projects occurring simultaneously to serve veterans who receive support from the agency.
As I do this work, I am hyper aware of the privatization machinations of the current administration, and the potential for APIs to serve these desires. APIs aren’t good, bad, nor are they neutral. APIs reflect the desires of their owners, and the good or bad they can inflict is determined by their consumers. Because APIs are so abstract, and are often buried within web and mobile applications, it can be difficult to see what they are truly doing on a day to day basis. This is why we have API management solutions in place to help paint a picture of activity in real time, but as we’ve seen with the Facebook / Cambridge Analytica, it requires the API provider to be paying attention, and to possess incentives that ensure they will actually care about paying attention, and responding to negative events.
By re-defining existing VA internal systems as APIs, we are opening up the possibility that digital resources can be extracted, and transferred externally. Of course, if these systems remain the sources of data, VA power and control can be retained, but it also opens up the possibilities for power to eventually be transferred externally–reversing the polarities of government power and putting it into private hands. Depending on your politics, this could be good, or it could be bad. Personally, I’m a fan of a balance being struck, allowing government to do what it does best, and allowing private, commercial entities to help fuel and participate in what is happening–both with proper oversight. This reflects the potential benefits APIs can bring, but this good isn’t present by default, and could shift one way or the other at any moment.
Early on, APIs appeared as a promising way to deliver value to external developers. They held the promise of delivering unprecedented resources to developers, thing they would not normally be able to get access to. Facebook promised democratization using it’s social platform. While silently mining and extracting value from it’s application end-users, executed by often unknowing, but sometimes complicit developers. A similar transfer of power has occurred with Google Maps, which promised unprecedented access to maps for developers, as well as web and mobile maps for cities, businesses, and other interests. Over time we’ve seen that Google Maps APIs have become very skilled at extracting value from cities, business, and end-users, again executed unknowingly, and knowingly by developers. This process has been playing out over and over throughout the last decade of the open data evolution occurring across government at all levels.
Facebook and Google have done this at the platform level, extracting value and maintaining control within their operations, but the open data movement, Facebook and Google included, have been efficiently extracting value from government, and giving as little back as it possibly can. Sure, one can argue that government benefits from the resulting applications, but this model doesn’t ensure that the data and content stewards within government retain ownership or control over data, and benefit from revenue generated from the digital resources–leaving them struggling with tight budgets, and rarely improving upon the resources. This situation is what I fear will happen at the Department of Veterans Affairs (VA), allowing privatization supporters to not actually develop applications that benefit veterans, but really focusing on extracting value possessed by the VA, capturing it, and generating revenue from it, without ever giving back to the VA–never being held accountable for actually delivering value to veterans.
I have no way of testing the political motivations of people that I am working with. I also don’t want to slow or stop the potential of APIs being used to help veterans, and ensure their health and insurance information can become more interoperable. I just want to be mindful of this reality, and revisit it often. Openly discussing it as I progress in my work, and as the VA Lighthouse platform evolves. Sometimes I regret all the API evangelism I have done, as I witness the damage they can do. However, the cat is out of the bag, and I’d rather be involved, part of the conversation, and pushing for observability, transparency, and open dialogue about the realities of opening up government digital resources to private entities via APIs. Allowing people who truly care about serving veterans to help police what is going on, maximize the value and control over digital resources into the hands of people who are truly serving veteran’s interests, and minimizing access by those who are just looking to make money at veteran’s expense.
The General Services Administration (GSA) has an open call for help to test the new FedBizOpps API. Setting a pretty compelling precedent for releasing APIs in the federal government, slowly bringing federal agencies out of their shell, and moving the API conversation forward in government. Hopefully it will be something that other federal agencies, and other levels of government will consider as they move forward on their API journeys.
It is good to see federal agencies reaching out to ask for help in making sure APIs are well designed, deliver value, and operate as expected. In my experience, most government agencies are gun-shy when it comes to seeking outside help, and accepting criticism when it comes to their work. The GSA is definitely the most progressive on this front, but they are leading by example, showing other agencies what is possible, and something that hopefully will spread. Something I am always keen to support with my storytelling here on API Evangelist.
One thing I’d caution federal agencies on when seeking outside feedback in this area, is to be mindful of fatigue in the private sector when it comes to working for free. API providers have been encouraging developers to document, test, develop code libraries, and other essential aspects of operating APIs for years now, and many developers are growing weary of this exploitation by companies who can afford to pay, but choose not to. While it is good that government is getting on the API bandwagon here, but be aware that you are about 5+ years behind, and the tides are shifting.
I’d recommend thinking about how micro-procurement models emerging at the Department of Veterans Affairs (VA) and other agencies can be applied to testing, performance, security, and other needs agencies will have. Think about how you can create $500, $1,000, and other opportunities for professional testers to make money. Of course, you are going to have to establish a way of vetting developers, but once you do, you are going to get a higher level work than you would for free. It might take more effort to lay the groundwork, but it will be something that will pay off down the road with a community of external professionals who can hep tackle micro-tasks that emerge while developing and operating government APIs.
Overall, I’m stoked to see stuff like this come out of the government. I’m just eager to see it spread to other federal agencies, shifting how digital services are delivered. I’m also eager to see more mock or virtualized APIs exposed publicly, long before any code is actually written. Bringing in outside opinions early on, allowing APIs to be more efficiently delivered–ensuring they are more mature, without the costly evolutionary and versioning process. I’m going to push more projects I’m involved in to follow GSA’s lead, and solicit outside testers to join in and provide feedback at critical steps in the API delivery process. The more agencies do this, the more likely they will to have a trusted group of outside developers, to step up when they need help, and have tasks like this to accomplish.
I’ve been on five separate government reflated projects lately where finding modular OpenAPI-driven open source tooling has been a top priority. All of these projects are microservice-focused and OpenAPI-driven, and are investing significant amounts of time looking open source tools that will help with design governance, monitoring, testing, and security, and interact with the Jenkins pipeline. Helping government agencies find success as their API journey picks up speed, and the number of APIs grows exponentially.
Selling to the federal government can be a long journey in itself. They can’t always use the SaaS solutions many of us fire up to get the job done in our startup or enterprise lives. Increasingly government agencies are depending on open source solutions to help them move projects forward. Every agency I’m working with is using OpenAPI (Swagger) to drive their API lifecycle. While not all have gone design (define) first, they are using them as the contract for mocking, documentation, testing, monitoring, and security. The teams I’m working with are investing a lot of energy looking for, vetting, and testing out different open source modules on Github–with varying degrees of success.
Ideally, there was an OpenAPI-driven marketplace, or federated set of marketplaces like OpenAPI.Tools. I’ve had one for a while, but haven’t kept up to date–I will invest some time / resources into it soon. My definition of an OpenAPI tool marketplace would be that it is OpenAPI-driven, and open source. I’m fine with there being other marketplaces of OpenAPI-driven services, but I want a way to get at just the actively maintained open source tools. When it comes to serving government this is an important, and meaningful distinction. I’d also like to encourage many of the project owners to ensure there is CI/CD integration, as well as make sure their projects are actively supported, and they are willing to entertain commercial implementations.
While there wouldn’t always be direct commercial opportunities for open source tooling owners to engage with federal agencies, there would be through contractors and subcontractors. Working for federal agencies is a maze of forms and hoop jumping, but working with contractors can be pretty straightforward if you find the right ones. I don’t think you will get rich developing OpenAPI-driven tooling that serves the API lifecycle, but I think with the right solutions, support, and team behind them, you can make a decent living developing them. Especially as the lifecycle expands, and the number of services being delivered grows, the need for specialized, OpenAPI-driven tools to apply across the API lifecycle is only going to increase. Making it something I’ll be writing more stories about as I hear more stories from the API trenches.
I’m going to try and spend time working with Phil Sturgeon (@philsturgeon) and Matt Trask (@matthewtrask) on API.Tools, as well as give my own toolbox some love. If you have an open source OpenAPI-driven tool you’d like to get some attention feel free to ping me, and make sure its part of API.Tools. Also, if you have a directory, catalog, or marketplace of tools you’d like to showcase, ping me as well, I’m all about supporting diversity of choice in the space. I have multiple federal agencies ear right now when it comes to delivering along the API lifecycle, and I’m happy to point agencies and their contractors to specific tools, if it makes sense. Like I said, there won’t always be direct revenue opportunities, but they are implementations that will undoubtedly lead to commercial opportunities in the form of consulting, advising, and development opportunities with the contractors and subcontractors who are delivering on federal agency contracts.
I’ve had a huge demand for putting on API seminars for a variety of enterprise groups lately. Helping bring my API 101, history, lifecycle, and governance knowledge to the table. I’ve conducted seminars in the UK and France in April, and have more in Virginia, Massachusetts, and California in May. I’ve been working with a variety of companies, institutions, and government agencies to plan even more internal seminars, which is something that is proving to be challenging for some groups who are just getting started on their API journey.
While groups are keen on me coming visit to share my API knowledge, some of them are having trouble getting me through legal, get me into their payment systems, and making everyone’s schedules work. Some of them don’t have a problem figuring it out, while others are facing significant amounts of bureaucracy and friction due to the size, complexity, and legacy of their organizations. To help simplify the process for organizations to participate in my seminars I am going to begin planning a rolling schedule of seminars beginning in New York City.
Every two weeks, on Tuesday and Wednesday I’ll be planning to do a 2-day API lifecycle seminars in NYC. I’m going to be focusing on delivering 12 stops along the API lifecycle, targeting my readers and customers who have already embarked on their API journey:
- Definition - The basics of OpenAPI, Postman, and using Github to manage API definitions.
- Design - Entry level API design, including touching on API governance and guidance.
- Deployment - Looking at the big world of how APIs can deployed on-premise, and the clouds.
- Virtualization - Understanding mocking, and API, as well as data virtualization.
- Authentication - Thinking about the common patterns of authentication for API consumption.
- Management - Covering the services, tools, and approaches to managing APIs in operation.
- Discussion - Open discussion about anything covered throughout the day, and beyond.
- Documentation - Demonstrating how to deliver portals, documentation, and other resources.
- Testing - Providing information on the monitoring and testing of API infrastructure.
- Security - Walking through API security beyond just API management and authentication.
- Support - Discussing the importance of providing direct and indirect support for APIs.
- Evangelism - Looking at how to evangelize your APIs internally and externally.
- Integration - Thinking briefly about about API integration concerns for API providers.
- Discussion - Open discussion about anything covered throughout the day, and beyond.
I’m kicking off this seminar series in New York City, with the following dates proposed to get the ball rolling:
- June 12th & 13th 2018 from 8:30 AM to 4:30 PM each day.
- June 26th & 27th 2018 from 8:30 AM to 4:30 PM each day.
I’m going to be charging $1,000.00 per person for the two day seminars. I’m setting the minimum bar for attendance to be 5 people, otherwise I won’t be scheduling the seminar, and will be pushing forward to the next dates. I’ll make sure and let everyone know at least 2 weeks in advance if we don’t get the expected attendance. All seminars are open to anyone who would like to attend, but I am also happy to conduct private group seminars, with minimum of five people in attendance. My goal is to conduct a regular cadence of seminars, that people know they can plan on, and participate in as it works for their schedule, but also provide more stability for my schedule as well.
The objective with these seminars is to make it easier for companies, institutions, and government agencies to participate in my seminars, while also forcing them out of their bubbles. I’m finding the toll of me coming onsite can vary widely, and the value to seminar attendees is lower when they aren’t forced to leave their bubbles. So, I am looking to force people out of their usual domains, so that they begin thinking about how they’ll be doing APIs, and playing nicely with others–even if they are only intended for internal use. I find the process helps you to think outside your normal daily operations, and is easier for me to not have to navigate coming on site to deliver seminars.
I’ve published both initial dates to Eventbrite to test the waters, and we’ll see if I can shift some of the demand I’m getting to this format. I will also keep pushing dates out every other week, and in July I’ll probably start looking at dates in Washington DC, then London, Paris, and probably some other west coast US locations. If you have interest in participating in my API lifecycle seminars and / or have specific requests for venues I’d love to hear from you. I’m looking to continue formalizing my process as well as schedule, and bringing my seminars wherever they are need, but doing so in a more organized fashion that helps you get your team what they need, but will also be something that is more sustainable for me in the long term.
I’m a big fan of API sandboxes, labs, and other virtualization environments. API sandboxes should be default in heavily regulated industries like banking. I also support the virtualization of schema and data used across API operations, like I am doing at the Department of Veterans Affairs (VA), with synthetic healthcare data. I’m very interested in anything that moves forward the API virtualization conversation, so I found the live sporting API simulations over at SportRadar very interesting.
SportRadar’s “live simulations give you the opportunity to test your code against a simulation of live data before the preseason starts or any time! Our simulation system replays select completed games allowing you to view our API feeds as if they were happening live.” Here are the details of their NFL Official API simulations that run every day:
- 11:00 am - Data is reset for the day’s simulations.
- 1:00 pm - PST week 1 games will run – Oakland at Houston, Detroit at Seattle, Miami at Pittsburgh, and New York at Green Bay
- 2:00 pm – PST week 2 games will run - Seattle at Atlanta, Houston at New England, Green Bay at Dallas, and Pittsburgh at Kansas City
- 3:00 pm – PST week 3 games will run – Green Bay at Atlanta and Pittsburgh at New England
- 4:00 pm – PST week 4 games will run – New England vs Atlanta
Providing a pretty compelling evolution in the concept API virtualization when it comes to event data, or data that will happen on a schedule. I wanted to write up this story to make sure I bookmarked this as part of my API virtualization research. If I don’t write about it, it doesn’t happen. As I’m working to include API sandboxes, lab environments, and other API virtualization approaches in my consulting, I’m keen to add new dimensions like API simulation, which provide great material for workshops, presentations, and talks.
I feel like ALL APIs should have some sort of sandbox to play in while developing. Not just heavily regulated industries, or those with sensitive production data and content. It can be stressful to have to develop against a live environment, and having a realistic sandbox, labs, and simulations which provide complete copies of production APIs, along with real world synthetic data can help developers be more successful in getting up and running. I’ll keep profiling interesting approaches like this one out of SportRadar, and add to my API virtualization research, helping round off my toolbox when it comes to helping API providers develop their sandbox environments.
I’ve been evaluating microservices at scale, using their OpenAPI definitions to provide API design guidance to each team using Github. I just finished a round of work where I took 20 microservices, and evaluated each OpenAPI definition individually, and made design suggestions via an updated OpenAPI definition that I submitted as a Github issue. I was going to submit as a pull request, but I really want the API design guidance to be a suggestions, as well as a learning experience, so I didn’t want them to feel like they had to merge all my suggestions.
After going through all 20 OpenAPI definitions, I took all of them and dumped them into a single local repository where I was running Jekyll via localhost. Then using Liquid I created a handful of web pages for looking at the APIs across all microservices in a single page–publishing two separate reports:
- API Paths - An alphabetical list of the APIs for all 20 services, listing the paths, verbs, parameters, and headers in a single bulleted list for evaluation.
- API Schema - An alphabetical list of the APIs for all 20 services, listing the schema definitions, properties, descriptions, and types in a single bulleted list for evaluation.
While each service lives in its own Github repository, and operates independently of each other, I wanted to see them all together, to be able to evaluate their purpose and design patterns from the 100K level. I wanted to see if there was any redundancy in purpose, and if any services overlapped in functionality, as well as trying to see the potential when all API resources were laid out on a single page. It can be hard to see the forest for the trees when working with microservices, and publishing everything as a single document helps flatten things out, and is the equivalent of climbing to the top of the local mountain to get a better view of things.
Jekyll makes this process pretty easy to do by dumping all OpenAPI definitions into a single collection, which I can then navigate easily using Liquid on a single HTML page. Providing a comprehensive report of the functionality across all microservices. I am calling this my microservices monolith report, as it kind of reassembles all the individual microservices into a single view, letting me see the collective functionality available across the project. I’m currently evaluating these service from an API governance perspective, but at a later date I will be taking a look from the API integration perspective, and trying to understand if the services will work in concert to help deliver on a specific application objective.
I’ve written about how the startup community has driven the value exchange that occurs at the API management layer down a two lane road with API monetizatin and plans. To generate the value that investors have been looking for, we have ended up with a free, pro, enterprise approach to measuring value exchanged with API integration, when in reality there is so much more going on here. Something that really becomes evident when you begin evaluating the API conversations going on across government at all levels.
In conversation after conversation I have with government API folk I hear that they don’t need API management reporting, analysis, and billing features. There is a perception that government isn’t selling access to APIs, so API management measurement isn’t really needed. Missing out on a huge opportunity to be measuring, analyzing, and reporting upon API usage, and requiring a huge amount of re-education on my part to help API providers within government to understand how they need to be measuring value exchange at the API management level.
Honestly, I’d love to see government agencies actually invoicing API consumers at all levels for their usage. Even if the invoices aren’t in dollars. Measuring API usage, reporting, quantifying, and sending an invoice for usage each month would help bring awareness to how government digital resources are being put to use (or not). If all data, content, and algorithms in use across federal, state, and municipal government were available via APIs, then measured, rate limited, and reported upon across all internal, partner, and public consumers–the awareness around the value of government digital assets would increase significantly.
I’m not saying we should charge for all access to government data. I’m saying we should be measuring and reporting upon all access to government data. The APIs, and a modern API management layer is how you do this. We have the ability to measure the value exchanged and generated around government digital resources, and we shouldn’t let misconceptions around how the private sector measures and generates revenue at the API layer interfere with government agencies benefitting from this approach. If you are publishing an API at a government agency, I’d love to learn more about how you are managing access to your APIs, and how you are reporting upon the reading and writing of data across applications, and learn more about how you are sharing and communicating around these consumption reports.
I checked out the Github repositories for twenty microservices of one of my clients recently, looking understand what is being accomplished across all these services as they work independently to accomplish a single collective objective. I’m being contracted with to help come in blindly and provide feedback on the design of the APIs being exposed across services, and help provide guidance on their API lifecycle, as well as eventually API governance when things have matured to that level. Right now we are addressing pretty fundamental definition and design issues, but eventually we’ll hopefully graduate to the next level.
A complete and up to date README for each microservice is essential to understanding what is going on with a service, and a robust OpenAPI definition is critical to breaking down the details of what each API delivers. When you aren’t part of each service’s development team it can be difficult to understand what each service does, but with an up to date README and OpenAPI, you can get up to speed pretty quickly. If an service is well documented via its README, and the API is well designed, and the surface area is reflected in it’s OpenAPI, you can go from not knowing what a service does to, understanding its value within hopefully minutes, not hours.
When each service possesses an OpenAPI it becomes possible to evaluate what they deliver at scale. You can take all APIs, their paths, headers, parameters, and schema and out them in different ways so that you can begin to paint a picture of what they deliver in aggregate. Bringing all the disparate services back together to perform together in a sort of monolith concert, while still acknowledging they all do their own thing independently. Allowing us to look at how many different service can be used in concert to deliver a single application, or potentially a variety of application instances. Thinking critically about each independent service, but more importantly how they all work together.
I feel like many groups are still struggling with decomposing their monolithic systems into separate services, and while some are doing so in a domain-driven way, few are beginning to invest in understanding how they move forward with services in concert to deliver on application needs. Many of the groups I’m working with are so focused on decomposing and tearing down, they aren’t thinking too critically about how they will make all of this begin work together again. I see monolith systems working like a massive church organ which take a lot of maintenance, and require a single (or handful) of knowledgeable operators to play. Where microservices are much more like an orchestra, where every individual player has a role, but they play in concert, directed by a conductor. I feel like most groups I’m talking with are just beginning the process of hiring a conductor, and have a bunch of musicians roaming around–not quite ready to play any significant productions quite yet.
The topic of API discovery has been picking up momentum in 2018. It is something I’ve worked on for years, but with the number of microservices emerging out there, it is something I’m seeing become a concern amongst providers. It is also something I’m seeing more potential vendor chatter, looking to provide more services and tooling to help alleviate API discovery pain. Even with all this movement, there is still a lot of education and discussion to occur on the subject to help bring people up to speed on what is API discovery.
The most common view of what is API discovery, is when you need to find an API for developing an application. You have a need for a resource in your application, and you need to look across your internal and partner resources to find what you are looking for. Beyond that, you will need to search for publicly available API resources, using Google, Github, ProgrammableWeb, and other common ways to find popular APIs. This is definitely the most prominent perspective when it comes to API discovery, but it isn’t the only dimension of this problem. There are several dimensions to this stop along the API lifecycle, that I’d like to flesh out further, so that I can better articulate across conversations I am having.
Another area that gets lumped in with API discovery is the concept of service discovery, or how your APIs will find their backend services that they use to make the magic happen. Service discovery focuses on the initial discovery, connectivity, routing, and circuit breaker patterns involved with making sure an API is able to communicate with any service it depends on. With the growth of microservices there are a number of solutions like Consul that have emerged, and cloud providers like AWS are evolving their own service discovery mechanisms. Providing one dimension to the API discovery conversation, but different from, and often confused with front-end API discovery and how developers and applications find services.
One of the least discussed areas of API discovery, but is one that is picking up momentum, is finding APIs when you are developing APIs, to make sure you aren’t building something that has already been developed. I come across many organizations who have duplicate and overlapping APIs that do similar things due to lack of communication and a central directory of APIs. I’m getting asked by more groups regarding how they can be conducting API discovery by default across organizations, sniffing out APIs from log files, on Github, and other channels in use by existing development teams. Many groups just haven’t been good at documenting and communicating around what has been developed, as well as beginning new projects without seeing what already exists–something that will only become a great problem as the number of microservices grows.
The other dimension of API discovery I’m seeing emerge is discovery in the service of governance. Understand what APIs exist across teams so that definitions, schema, and other elements can be aggregated, measured, secured, and governed. EVERY organization I work with is unaware of all the data sources, web services, and APIs that exist across teams. Few want to admit it, but it is a reality. The reality is that you can’t govern or secure what you don’t know you have. Things get developed so rapidly, and baked into web, mobile, desktop, network, and device applications so regularly, that you just can’t see everything. Before companies, organizations, institutions, and government agencies are going to be able to govern anything, they are going to have begin addressing the API discovery problem that exists across their teams.
API discovery is a discipline that is well over a decade old. It is one I’ve been actively working on for over 5 years. It is something that is only now getting the discussion it needs, because it is a growing concern. It will be come a major concern with each passing day of the microservice evolution. People are jumping on the microservices bandwagon without any coherent way to organize schema, vocabulary, or API definitions. Let alone any strategy for indexing, cataloging, sharing, communicating, and registering services. I’m continuing my work on APIs.json, and the API Stack, as well as pushing forward my usage of OpenAPI, Postman, and AsyncAPI, which all contribute to API discovery. I’m going to continue thinking about how we can publish open source directories, catalogs, and search engines, and even some automated scanning of logs and other ways to conduct discovery in the background. Eventually, we will begin to find more solutions that work–it will just take time.
The more I study where the API management, gateway, and proxy layer is headed, the less I’m seeing a front-end or a backend, I’m seeing just a node. A node that can connect to existing databases, or what is traditionally considered a backend system, but also can just as easily proxy and be a gateway to any existing API. A lot has been talked about when it comes to API gateways deploying APIs from an existing database. There has also been considerable discussion around proxying existing internal APIs or web services, so that we can deploy newer APIs. However, I don’t think there has been near as much discussion around proxying other 3rd party public APIs–which flips the backend and frontend discuss on its head for me.
After profiling the connector and plugin marketplaces for a handful of the leading API management providers I am seeing a number of API deployment opportunities for Twilio, Twitter, SendGrid, etc. Allowing API providers to publish API facades for commonly used public APIs, and obfuscate away the 3rd party provider, and make the API your own. Allowing providers to build a stack of APIs from existing backend systems, private APIs and services, as well as 3rd party public APIs. Shifting gateways and proxies from being API deployment and management gatekeepers for backend systems, to being nodes of connectivity for any system, service, and API that you can get your hand on. Changing how we think about designing, deploying, and managing APIs at scale.
I feel like this conversation is why API deployment is such a hard thing to discuss. It can mean so many different things, and be accomplished in so many ways. It can be driven by a databases, or strictly using code, or be just taking an existing API and turning it into something new. I don’t think it is something that is well understood amongst developer circles, let alone in the business world. An API gateway can be integration just as much as it can be about deployment. It can be both simultaneously. Further complexing what APIs are all about, but also making the concept of the API gateway a more powerful concept, continuing to renew this relic of our web services past, into something that will accommodate the future.
What I really like about this notion, is that it ensures we will all be API providers as well as consumers. The one-sided nature of API providers in recent years has always frustrated me. It has allowed people to stay isolated within their organizations, and not see the world from the API consumer perspective. While API gateways as a node won’t bring everyone out of their bubble, it will force some API providers to experience more pain than they have historically. Understanding what it takes to to not just provide APIs, but what it takes to do so in a more reliable, consistent and friendly way. If we all feel the pain our integration and have to depend on each other, the chances we will show a little more empathy will increase.
I was in a conference session about an API design tool today, and someone asked if you could get at the OpenAPI definition behind the solution. They said yes, but quickly also said that the definition is boring and that you don’t want to be in there, you want to be in the interface. I get that service providers want you to focus on their interface, but we shouldn’t be burying, or abstracting away the API contract for APIs, we should always be educating people about it, an bringing it front and center in any service, tooling, or conversation.
Technology folks burying or devaluing the OpenAPI definition with business users is common, but I also see technology folks doing it to each other. Reducing OpenAPI to be just another machine readable artifact alongside other components of delivering API infrastructure today. I think this begins with people not understanding what OpenAPI is, but I think it is sustained by people’s view of what is technological magic and should remain in the hands of the wizards, and what should be accessible to a wider audience. If you limit who has access and knowledge, you can usually maintain a higher level of control, so they use your interface in the case of a vendor, or they come to you develop and build an API in the case of a developer.
There is nothing in a YAML OpenAPI definition that business users won’t be able to understand. OpenAPIs aren’t anymore boring than a Word document or Spreadsheet. If you are a stakeholder in the service, you should be able to read, understand, and engage with the OpenAPI contract. If we teach people to be afraid of the OpenAPI definitions we are repeating the past, and maintaining the canyon that can exist between business and IT/Developer groups. If you are in the business of burying the OpenAPI definition, I’m guessing you don’t understand the portable API lifecycle potential of this API contract, and simply see it as a config, documentation, or other technical artifiact. Or you are just in the business of maintaining control and power by being the gatekeeper for the API contract, similar to how we see database people defend their domain.
Please do not devalue or hide away the OpenAPI contract. It isn’t your secret sauce. It isn’t boring. It isn’t too technical. It is the contract for how a service will work, that will speak to business and technical groups. It is the contract that all the services and tools you will use along the API lifecycle will understand. It is fine to have the OpenAPI right behind the scenes, but always provide a button, link, or other way to quickly see the latest version, and definitely do not scare people away or devalue it when you are talking. If you are doing APIs, you should be encouraging, and investing in everyone being able to have a conversation around the API contract behind any service you are putting forward.
I’ve had a few API groups contact me lately regarding the challenges they are facing when it comes to supporting organizational, or industry-wide API definitions and schema. They were eager to support common definitions and schema that have been standardized, but were getting frustrated by not being able to do everything they wanted, and having to live within the constraints introduced by the standardized definitions. Which is something that doesn’t get much discussion by those of use who are advocating for standardization of APIs and schema.
Web APIs Come With Their Own Constraints
We all want more developers to use our APIs. However, with more usage, comes more responsibility. Also, to get more usage our APIs need to speak to a wider audience, something that common API definitions and schema will help with. This is why web APIs are working, because they speak to a wider audience, however with this architectural decision we are making some tradeoffs, and accepting some constraints in how we do our APIs. This is why REST is just one tool in our toolbox, so we can use the right tool, establish the right set of constraints, to allow our APIs to be successful. The wider our API toolbox, the wider the number options we will have available when it comes to how we design our APIs, and what schema we can employ
Allowing For Content Negotiation By Consumers
One way I’ve encouraged folks to help alleviate some of the pain around the adoption of common API definitions and schema is to provide content negotiation to consumers, allowing them to obtain the response they are looking for. If people want the standardized approaches they can choose those, and if they want something more precise, or custom they can choose that. This also allows API providers to work around the API standards that have become bottlenecks, while still supporting them where they matter. Having the best of both worlds, where you are supporting the common approach, but still able to do what you want when you feel it is important. Allowing for experimentation as well as standardization using the same APIs.
Participate In Standards Body and Process
Another way to help move things forward is to participate in the standards body that is moving an API definition or schema forward. Make sure you have a seat at the table so that you can present your case for where the problems are, and how to improve on the design, definition, and schema being evolved. Taking a lead in creating the world you want to see when it comes to API and schema standards, and not just sitting back being frustrated because it doesn’t do what you want it to do. Having a role in the standards body, and actively participating in the process isn’t easy, and it can be time consuming, but it can be worth it down the road and helping you better achieve your goals when it comes to your APIs operating as you aspect, as well as the wider community and industry you are serving.
Delivering APIs at scale won’t be easy. To reach a wide audience with your API it helps to be speaking a common vocabulary. This doesn’t always allow you to move as fast as you’d like, and do everything exactly as you envision. You will have to compromise. Operate within constraints. However, it can be worth it. Not just for your organization, but for the overall health of your community, and the industry you operate in. You never know, with a little patience, collaboration, and communication, you might learn even new approaches to defining your APIs and schema that you didn’t think about in isolation. Also, experimentation with new patterns will still be important, even while working to standardize things. In the end, a balance between standardized and custom will make the most sense, and hopefully alleviate your frustrations in moving things forward.
I’m always fascinated by the API design debate, and how many entrenched positions there are when it comes to the right way of doing it. Personally, I don’t see any right way of doing it, I just see many different ways to put the responsibility on the provider or the consumer shoulders, or sharing the load between them. I spend a lot of time profiling APIs, crafting OpenAPI definitions that try to capture 100% of the surface area of an API. Something that is pretty difficult to do when you aren’t the provider, and you are just working from existing static documentation. It is just hard to find every parameter and potential value, exhaustively detailing what is possible when you use an API.
When designing APIs I tend to lean towards exposing the surface area of an API in the path. This is my personal preference for when I’m using an API, but I also do this to try and make APIs more accessible to non-developers. However, I regularly get folks who freak out at how many API paths I have, preferring to have the complexity at the parameter level. This conversation continues with the GraphQL and other query language folks who prefer to craft more complex queries that can be passed in the body, to define exactly what is desired. I do not feel there is a right way of doing this, but it does reflect what I said early about balancing the load between provider and consumer.
The burden for defining and designing the surface area of an API resides in the providers court–only the provider truly knows the entire surface area. Then I’d add that when you offload the responsibility to the consumer using GraphQL, you are limiting who will be able to craft a query, putting API access beyond the reach of business users, and many developers. I feel that exposing the surface area of an API in the URL makes sense to a lot of people, and puts it all out in the open. Unless you are going to provide every possible enum value for all parameters, this is the only way to make 100% of the surface area of an API visible and known. However, depending on the complexity of an API, this is something that can get pretty unwieldy pretty quickly, making parameters the next stop be defining and designing the surface area of your API.
I know my view of API design doesn’t sync with many API believers, across many different disciplines. I’m not concerned with that. I’m happy to hear all the pros and cons of any approach. My objective is to lower the bar for entry into the API game, not raise the bar, or hide the bar. I’m all for pushing API providers to be more responsible for defining the surface area of their API, and not just offloading it on consumers to do all the work, unless they implicitly ask for it. In the end, I’m just a fan of simple, elegantly designed APIs that are intuitive and well documented using OpenAPI. I want ALL APIs to be accessible to everyone, even non-developers. I want them accessible to developers, minimizing the load on them to understand what is happening, and what all the possibilities are. I just don’t want to spend too much time on-boarding with an API. I just want to go from discovery to exploration in as little time as possible.
|<< Prev||Next >>|