The API Evangelist Blog
This blog represents the thoughts I have while I'm research the world of APIs. I share what I'm working each week, and publish daily insights on a wide range of topics from design to depcration, and spanning the technology, business, and politics of APIs. All of this runs on Github, so if you see a mistake, you can either fix by submitting a pull request, or let me know by submitting a Github issue for the repository.
My friend James Higginbotham (@launchany) was sharing his frustration with being able to stay in tune with changes to a variety of APIs. Like me, James works to stay in tune with a variety of signals available via platforms like Twitter, Github, and other commonly used services. These platforms don’t always properly signal when things are updated, changed, or advanced, making it difficult to understand the granular changes that occur like likes, votes, edits, and other common events that occur via highly active platforms.
This challenge is why the evolution towards a more event-driven approach to operating an API platform is not just more efficient, it gives users what they need. Using event-driven architectural approaches like Webhooks, and real times streams. This is one of the reasons I’m interested in what Streamdata.io does, beyond them helping support me financially, is that they allow me to focus on the event-driven shift that is occurring with many leading API providers, and needs to be brought to the attention of other platforms. Helping API providers be more efficient in what they are doing, while also meeting the needs of the most demanding customers like James and myself.
It is easy to think Streamdata.io is just about streaming real time data. This is definitely a large aspect of what the SaaS solution does, but the approach to using Server-Sent Events (SSE), with incremental updates using JSON Patch adds another useful dimension when it comes to understanding what has changed. You can proxy an existing HTTP API that returns a JSON response using Streamdata.io, and the first response will look just like any other, but every pushed response after that will be a JSON Patch of just what has changed. Doing the heavy lifting of figuring out what has changed in each API response and only sending you the difference, and allowing you to focus only on what has changed, and not having to rely on timestamps, and other signals within the JSON response to understand what the difference is from the previous API response.
Using Streamdata.io you don’t have to keep polling an API asking if things have changed, you just proxy the API and you get pushed changes via an HTTP stream. You also don’t have to sort through each response and try to understand what changed, you just take the JSON Patch response, and it tells you what has changed. I’m going to create a draft blueprint for James of how to do this, that he can use across a variety of APIs to establish multiple API connections using long running, server-side API streams for a variety of topics. Allowing him to monitor many different APIs, and stay in tune with what changes as efficiently as possible. Once I craft a generic blueprint, I’m going to apply to Twitter and see if I can increase the efficiency of my Twitter monitoring, by turning their REST APIs into real time feeds using Streamdata.io.
One thing I see a lot from API service providers who are selling their services to the API sector, is that once they find success servicing one stop along the API lifecycle, they often want to service other additional stops. I don’t have a problem with API service providers delivering across multiple stops along the API lifecycle, however I do caution of trying to expand across too many stops, and potentially doing any of them poorly, rather than partnering with other more specialized API service providers to help you focus on what you do best.
I’m a big advocate for encouraging API providers to service one to five stops along the API lifecycle well, and then partner for helping deliver the rest of the stops. I know that all your investors are encouraging to take as many pieces of the puzzle as you possibly can, but there is more money in doing a handful of things really well, over doing many things poorly. Try to be an expert in a handful of specialized areas, over being a generalist. Then make sure your platform is as interoperable as possible, while investing in your partner program to attract the best of breed API service providers to your platform.
This balance between focusing on a handful of stops and partnering is why I emphasize and study common approaches to delivering plugins. All platforms should invest in plugin infrastructure, to allow for extending their reach beyond the stops that a platform services. Feature creep, and platform bloat is a real challenge, especially when you have investors whispering in your ear to keep building, and a very vocal, but often long-tail group of users demanding solutions to their unique problems. Plugin and connector architecture is how you help manage this reality, and provide a relief valve for delivering too many features as part of your platform, while also bringing in potential partners who can help extend what your platforms in a way that allows you to keep doing what you do best.
I see a big push going on from many legacy API service providers, as well as some of the next generation of startups bringing services and tooling to the space. I feel like many people desire a single solution to do everything, but then fail to realize that every platform that has attempted this in the past ends up failing, because you can’t be everything to everyone. I want my API service providers to stick to doing a handful of things well, but then acknowledge that I will also be using several other tools to what get I need accomplished on a daily basis. Ideally all of my tools are interoperable with import and export capabilities, as well as a suite of API driven connectors, and plugins that allow me to keep all of my services and tooling working together in concert. For this reality to occur we all have to resist the temptation to lock our customers in, and put down delusions that we can serve all stops along a modern API lifecycle all by ourselves.
I’m fascinated by the dominating power of programming languages. There are many ideological forces at play in the technology sector, but the dogma that exists within each programming language community continues to amaze me. The potential absence of programming language dogma within the world of APIs is one of the reasons I feel it has been successful, but alas, other forms of dogma tends to creep in around specific API approaches and philosophies, making API evangelism and adoption always a challenge.
The absence of programming languages in the API design, management, and testing discussion is why they have been so successful. People in these disciplines have ben focused on the language agnostic aspects of just doing business with APIs. It is also one of the reasons the API deployment conversation still is still so fragmented, with so many ways of getting things done. When it comes to API deployment, everyone likes to bring their programming language beliefs to the table, and let it affect how we actually deliver this API, and in my opinion, why API gateways have the potential to make a comeback, and even excel when it comes to playing the role of API intermediary, proxy, and gateway.
Programming language dogma is why many groups have so much trouble going API first. They see APIs as code, and have trouble transcending the constraints of their development environment. I’ve seen many web or HTTP APIs called Java API, Python APIs, or reflect a specific language style. It is hard for developers to transcend their primary programming language, and learn multiple languages, or think in a language agnostic way. It is not easy for us to think out of our boxes, and consider external views, and empathize with people who operate within other programming or platform dimensions. It is just easier to see the world through our lenses, making the world of APIs either illogical, or something we need to bend to our way of doing things.
I’m in the process of evolving from my PHP and Node.js realm to a Go reality. I’m not abandoning the PHP world because many of my government and institutional clients still operate in this world, and I’m using Node.js for a lot of serverless API stuff I’m doing. However I can’t ignore the Go buzz I keep coming stumbling upon. I also feel like it is time for a paradigm shift, forcing me out of my comfort zone and push me to think in a new language. This is something I like to do every five years, shifting my reality, keeping me on my toes, and forcing me to not get too comfortable. I find that this keeps me humble and thinking across programming languages, which is something that helps me realize the power of APIs, and how they transcend programming languages, and make data, content, algorithms, and other resources more accessible via the web.
I spend a lot of time talking about the SalesForce API, using it as a reference for where the API evolution began 18 years ago, but it has been a long time since I’ve actually worked with the SalesForce API. Getting up and running with any API, especially iconic APIs that we all should be familiar with, is always an enlightening experience for me. Going from zero to understanding what is going on and actually achieving the API call(s) you want, is really what this game is all about.
As part of some work I’m doing with Streamdata.io I needed to be able to add new leads into SalesForce, and I thought it would be a good time for me to get back into the saddle with the SalesForce REST API–so I volunteered to tackle the integration. The SalesForce API wasn’t as easy to get up and running as many simpler APIs I onboard with is, as the API docs isn’t as modern as I’d expect, and what you need is buried behind multiple clicks. Once you find what you are looking for, and click numerous times, you begin to get a feel for what is going on, and the object model in use becomes a little more accessible.
In addition to finding what you need with the SalesForce REST API, you have to make sure you have a handle on the object structure and nuance of SalesForce itself. For this story, I am just working with one object–Leads. I’m using PHP to work with the API, and to begin I wanted to be able to get leads, to be able to see which leads I currently have in the system:
I will add pagination, and other elements in the future. For now, I just wanted to be able to get the latest leads I have in the system to help with with some checks on what is being added. Now that I can check to see what leads are in the system, I wanted to be able to add a lead, with the following script:
I am only displaying some of the default fields available for this example, and you can add other custom fields based upon which values you wish to add. Once I have added my lead, I wanted to be able to update with a PATCH API call:
Now I am able to add, update, and get any leads I’m working with via the SalesForce API. The project gave me a good refresher for what is possible with the SalesForce API. The API is extremely powerful, and something I want to be up to speed on so that I can intelligently respond to questions I get. I wish the SalesForce API team would spend some time modernizing their API portal and documentation, providing a more coherent separation between the different flavors of their API, and provide OpenAPI driven documentation, as well as Postman Collections. It would have saved me hours of working through their API docs, and playing around with different API calls in Postman before I was able to successfully OAuth, and make my first call against the accounts and leads API endpoints.
While I think SalesForce remains a worthwhile API to showcase when I talk about the history of APIs, and the power of providing web APIs, their overall documentation and approach is beginning to fall behind the times. SalesForce possesses many of the building blocks I recommend other API providers operate, and are very advanced in some of their support and training efforts, but their documentation, which is the biggest pain point for developers, leaves a lot to be desired. I’m used to having to jump through hurdles to get up and running APIs, so the friction for me was probably less than a newer API developer would experience. I could see some of the domain instance url, versioning, and available API paths proving to be a significant hurdle if you didn’t understand what was going on. Something that could be significantly minimized with some simpler, more modern API docs, and OpenAPI and Postman Collections available.
This report summarizes Skylight’s evaluation of the VA’s public datasets, which exist within the va.gov web domain, as well as an analysis of what types of data representatives of the Veteran community expressed would be most useful and valuable to Veterans and their supporters if made more digitally accessible and available by the VA. This report also outlines potential resources that can be turned into application programming interfaces (APIs) as part of the VA’s Lighthouse platform initiative, and actions the VA should consider to move forward successfully.
APIs are the next evolution in the web, and shouldn’t be thought of as the latest tech trend or vendor solution. The first phase of the web was about delivering data and content to humans using a browser. The second phase of the web is about delivering that same data and content to other applications and algorithms using APIs.
With that said, the purpose of this landscape analysis is, in effect, to assist the VA in evaluating the data and content that they’ve made available during the first phase of the web. This, in turn, will help set the stage for the VA to make smart investments in phase two of their web presence.
The VA has already signaled they’re committed to investing in the second phase of their web presence with the announcement of the Lighthouse API platform initiative. Our landscape analysis will help ensure that the Lighthouse program is aware of what types of data and content that the VA has already identified as important to serving the Veteran community. This visibility will allow the Lighthouse program to bring these resources into alignment with the development and operation of their API platform.
To help the VA evaluate the landscape that defines their web presence, we employed a “low-hanging-fruit” process that involved identifying the resources that exist across their web properties. That process relied on a spidering script, which we ran for two weeks (and continue to run). To begin the process, we seeded the script by giving it the root URL for the va.gov domain. The script then proceeded to:
Parse every URL on the page and store it in a database;
Count every table on the page, and the number of rows that exist in the table;
Count every form that exists on the page; and
Extract the title from the meta tags for each page.
The script then iterated and repeated this for every URL it found on any web page, working to identify each of the following types of data resources:
HTML table with more than 10 rows
The script ignored any URLs external to the seed domain (va.gov) and many common web objects (for example, images, Word docs, and videos).
As each page was processed, the script tried to identify potential data resources to deliver as an API by parsing several elements from them:
The title of the page a file was published on,
The name of the file itself, and
Occasionally a sample of the data.
We took the list of words extracted from this process, and sorted and grouped them by the number of times the word appeared, helping us understand the overall presence of each potential resource. Sometimes this produced a lot of meaningless words, but we worked to filter those out, leaving only the meaningful data resources.
After running the script for a couple of weeks, we spidered nearly 1/3 of the URLs (out a total of 4M+) targeted for processing. That was enough progress to start painting an interesting picture of the VA’s web presence and existing data resources, as described in the sections that follow.
VA’s web presence
The VA has a sprawling web presence, spanning multiple domains and subdomains. We focused our analysis on everything within the va.gov domain. In the future, we can extend the analysis to other domains, but for now we focused on the core VA web presence.
The VA’s web presence is spread across a mix of domain levels, including top-level domain and subdomain levels — program, state/region, and city. Domains play a role in providing addresses in the browser so that users can find the resources they need, as well as providing similar addressing for applications to find the resources they need via APIs. The VA domain sprawl reflects the growth of the VA’s web presence, and the lack of overall strategy when it comes to providing web address and routing to all VA resources. The current strategy (or lack thereof) represents the need for location- and program-related resource discovery, whether it’s in the browser, or for other applications via APIs.
We assume there are other domains that haven’t been indexed by our spider, as we were only able to index less than 1/3 of the targeted URLs during our two-week timebox. We can continue indexing and updating numbers beyond this period in order to paint an even more complete picture. Ideally, this would extend beyond the core va.gov domain. Domain and subdomains play an important role in determining how APIs will be accessed, and have a downstream impact on the overall API design, affecting both API path and parameter design. This makes domain and subdomains a top-level consideration early on in the VA’s API journey.
After the top-level domains of va.gov and www.va.gov, the most common approach to defining domains is by program, providing the addressing needed for organizing information by relevant programs. We have identified 133 individual program-related domains.
While there aren’t consistent naming conventions used in crafting these subdomains, it does demonstrate the prominence of programs, research, and other related groupings used across the VA web presence for organizing resources.
Beyond program-related domains, state/region level domains are being used to organize data and content for presentation to consumers. Only 22 subdomains are represented currently, but the practice demonstrates the prominence of these locations when it comes to organizing information.
Some states are just paths within the top-level VA domains, while others exist within regional subdomains, with the rest possessing their own subdomain. This demonstrates the importance of states and regions, but also the inconsistency of how domains or paths are used to organize information.
Lastly, you find many city-related subdomains being used to organize data and content, providing another dimension on how resources are being organized, while demonstrating the dominance of specific cities. We have identified 120 individual city-related domains.
Like states, there isn’t a consistent pattern in which cities have their own subdomain, with others existing as a path within state subdomains or top-level domains. The approach to using cities as part of subdomain DNS addressing further demonstrates the importance of location when it comes to the organization of data and content.
As part of the spidering the va.gov domain across the 278 subdomains that exist, over 4M individual URLs were identified, with slightly less than 1/3 of these URLs evaluated for potential data sources to-date. Across these URLs, we took the base path and grouped them by the number of pages and data files that exist.
While there are many other paths in use across the VA websites, these paths reflect the top paths in use to deliver data and content. Providing a look at what the most relevant resources are when it comes to providing web access to data and content, which is something that should be considered when delivering the same data and content to other applications.
After assessing the titles of HTML pages and the names of files, it’s clear there’s no consistent vocabulary in use across VA resources. This, combined with the use of key phrases, acronyms, and singular and plural variances, make it difficult to cleanly identify resources. We opted to use just keywords over phrases and to not expand acronyms as part of the process due to the difficulty in consistently identifying resources.
Even with the difficulties in identifying some resources, we were still able to paint a fairly compelling picture of the resources being exposed as common data formats across VA web properties. That’s because we were able to isolate, group, and identify words that are most commonly associated with resources. From there, we were able to establish some resource lists, which we have organized visually as tag clouds and tag lists.
Data is available across VA websites in a variety of formats. We focused on a handful of easy to identify formats, reflecting the low-hanging-fruit aspect of this landscape analysis. While there’s data locked up in simple text files and zipped packages, we chose to look for the easiest to identify and the easiest to publish data sources. Data that’s published by humans usually take the form of CSV files, spreadsheets, and HTML tables. Data that’s published by systems usually take the form of JSON and XML.
Each of type of format that we targeted provides a different story as to the type of resources being published. Publication implies that those resources carry some level of value and importance to VA stakeholders, and, potentially, to Veterans, their supporters, and other consumers of this information. We worked to harvest all the data available from several formats, but also worked to identify the top resources available from each type.
We discovered 534 CSV files containing a variety of data. By parsing the titles of the web pages these CSV files were linked from, and the names of some of the files, we identified handful of top resource types present across these files.
CSV files tell a particular story because they were most likely published by people working at the VA, who exported the files from spreadsheets and made them available on the website for a reason. This makes them relevant to the VA’s API conversation. You can view a list of CSV resources, as well as a complete list of CSV files on the GitHub repository.
We identified 6,077 spreadsheets containing a variety of data. After parsing these files for semantic meaning, we identified handful of top resource types present across these files.
Similar to CSV files, the presence of spreadsheets tell a very human story. Spreadsheets are the #1 source of data on the web, and reflects the data management and publishing practices across the VA. After evaluating what types of resources are available across these spreadsheets, we have been considering the use of spreadsheets as a data source, as well as a data publishing tool. You can view a list of XLS/XLSX resources, as well as a complete list of XLS/XLSX files on the GitHub repository.
We identified 467 JSON files containing a variety of data. Unlike the CSV and spreadsheet data sources, JSON files likely represent a more modern systems approach to publishing data and a whole another set of data sources, which should be considered when deploying APIs.
JSON reflects the latest evolution of data publishing at the VA. But they are only a small subset of the data being made available across VA web properties. This implies they have only become a recent priority when it comes to publishing data in a format that is consumable by developers and computers. You can view a list of JSON resources, as well as a complete list of JSON files on the GitHub repository.
We found 3,099 XML files containing a variety of data. Like JSON files, XML files represent system-generated publication of data. Unlike JSON, however, XML reflects an older systems approach to data publication. And are likely being generated by legacy systems that’ll be important to interface with over the course of the VA’s API journey.
XML represents a large portion of the data being published across VA web properties. This list of priority resources represents a significant part of the system-based publishing of data occurring at the VA. And provides a large snapshot of the systems that should be evolved as part of the deployment of APIs. You can view a list of XML resources, as well as a complete list of XML files on the GitHub repository.
We identified 8,393 pages that had tables on them with over 10 rows. These tables represent potentially valuable data and should be considered as part of the VA’s API deployment conversations.
While HTML tables tell a story about top resources that VA stakeholders thought website users needed access to, these tables also represent data that was published with potential search engine optimization (SEO) in mind. In other words, someone wanted the data to be indexed by search engines in order to make it more readily accessible. You can view a list of table resources, as well as a complete list of pages containing tables on the GitHub repository.
We identified 9,439 pages with more than one form present, which is usually just a basic search. Similar to HTML tables, these forms provide a window into how the VA is making data available for users to search, explore, and consume in the browser. This, in turn, tells another story of what types of resources are published to VA websites.
HTML forms often times provide a search mechanism for other table, CSV, JSON, XML, and spreadsheet resources, many of which are listed in the sections above. HTML forms tell their own story as to how and why data are being published across VA websites. And offer another source of resources that are being made available and should be considered as part of the VA’s API deployment efforts.
The only external source of data that we analyzed was data.gov, which hosts a number of VA data resources. While somewhat out-of-date, the VA datasets on data.gov tell an important part of the story that should be considered as part of the Lighthouse efforts. There are a lot of lessons to be learned from how data.gov has been used, beyond just understanding what resources have been published there.
The resources published to data.gov reflect the VA’s recent past when it comes to making data resources available and accessible via manual downloads and APIs. We think that the most important lesson that the VA should take away from its experience with data.gov is that the VA should own all the data and API resources and syndicate them as part of other external efforts. That way the VA owns the full scope of the effort, which will ultimately result in the VA being more invested in API operations. You can view a list of data.gov resources, as well as a complete list of data files on the GitHub.
Humanizing the data
To give us a more human perspective on what types of data resources are most valuable to Veterans and their supporters, we facilitated a series of online workshops using Mural. About 50 people total participated across all three workshops, with about 40% reporting as Veterans and 60% non-Veterans. During these workshops, we employed the KJ technique for establishing group priorities. The KJ technique relies on a focus question to drive the results of the workshop. We used the following focus question:
“What types of data, content, and other resources would be most useful to Veterans and their supporters if the VA could make them more available and accessible on the web, mobile devices, and other platforms?”
The following images capture the results of each workshop:
The yellow cards represent all the ideas, in response to the focus question, that everyone brainstormed. As you can see, these yellow cards were organized into like groups. The blue cards represent descriptive labels that participants gave to each group. The black circles with numbers represent the votes that the participants casted when asked which group labels they thought best answered the focus question. We weighted Veteran votes 2x more heavily than votes from non-Veterans.
You may notice things that seem out of place in the final results (for example, yellow cards that look like they belong to another category). This is largely due to the timeboxed nature of the activities. In other words, not everything could be made perfect, but that doesn’t detract from the overall usefulness of the results.
Given the fact that the results were spread across three different workshop sessions, we took the additional step of normalizing the groupings and merging the votes.
Directory of Services/Resources – 34 votes
Mental Health – 21 votes
Personal Healthcare Data – 20 votes
Personalized Self-Service Portal – 20 votes
Benefits – 13 votes
Peer Support Networking – 8 votes
Family Support Networking – 9 votes
Real-Time Status – 8 votes
Patient Experience Data – 7 votes
Military-to-Civilian Transition – 7 votes
Ratings and Calculators – 6 votes
Appointments – 6 votes
Medical Healthcare – 5 votes
Housing Assistance – 5 votes
Public Accountability and Awareness – 4 votes
Service History Data – 4 votes
Veteran Status Verification – 0 votes
Statistical Analysis and Machine Learning – 0 votes
Metadata Support – 0 votes
Documentation – 0 votes
It’s entirely possible that these groupings could be further normalized, or even some of the ideas within the original groups split out into separate groups. Some groupings could even be disregarded as irrelevant (for example, Metadata Support). However, we didn’t want to dilute the results of what the participants came up with. Somewhat surprising is the low number of votes for Medical Health. That may be a result of lacking the right type of participant representation in the workshops, at least for that particular category.
Summary of the landscape analysis
The landscape analysis, which only processed about 1/3 of the URLs targeted for spidering over the course of a two-week period, revealed about 20,216 data files. This work produced a lot of data to wrangle and make sense of. Each of the data formats made available tell their own story about what types of data has been published to VA websites and why. The number of times a resource has been published using a particular data format (CSV, XML, JSON, XLS/XLSX, etc.) serves as a vote for making that resource available and accessible on the web.
Despite the huge amount of information to work with, we believe that our analysis provides valuable insight into some of the most relevant data resources, based on years of publication to VA websites. The top resources identified from all of the URLs, file formats, tables, and forms all point to data resources that should be considered turning into APIs. If these data resources were considered a priority when publishing to the VA’s websites, then there’s a good chance that they should be considered priorities when it comes to publishing via APIs as part of the Lighthouse initiative.
Lighthouse program considerations moving forward
Resources to prioritize
After spending time with all of the data uncovered during the landscape analysis, we began to see patterns emerge from across all the resources being published to the VA’s web properties, as well as those resources identified by people during our facilitated workshops. So based on analysis of the available data, we recommend that the VA Lighthouse program give consideration to prioritizing the following 25 resources:
Healthcare Facilities – Up-to-date information on hospitals, clinics, and other healthcare facilities.
Organizations – Details on any type of organization that services Veterans and their families.
Services – Services being offered by the VA, healthcare facilities, and other organizations.
Programs – Programs being offered by the VA, healthcare facilities, and other organizations.
Resources – Content, video, and other resources providing healthcare, outpatient, and other relevant content.
Schedules – The schedules of healthcare facilities, organizations, services, and programs being offered.
Events – Calendar and details of relevant events that service Veterans around the country.
Benefits – Details of the benefits being offered to Veterans, including elements of the process involved.
Performance – Performance details for the healthcare facilities, organizations, services, and programs.
Insurance – Home, auto, and healthcare insurance information that Veterans can take advantage of.
Loans – Information on home, auto, and other types of loans available to Veterans and their families.
Grants – Grants for education, businesses, projects, and other Veteran-focused efforts.
Education – Educational opportunities and information available to Veterans and their families.
Training – Specific training opportunities available that Veterans can take advantage of.
Jobs – Job postings that Veterans can apply to and use to guide their career.
Human Resources – VA human resources and related information in support of VA employees and Veterans.
Forms – Directory, access, and management of forms and the data that’s stored within them.
Budgets – Budget information on healthcare facilities, organizations, programs, and services.
Statistics – Statistics and data on all aspects of VA operations, and anything that impacts Veterans.
Cemeteries – Details of the cemeteries, and the Veterans who are laid to rest at all locations.
News – News that impacts Veterans from across any source and is relevant to the community.
Press – Press releases from the VA and related organizations and programs.
Research – Information and other resources produced as part of specific Veteran-related research.
Surveys – Centralized organization, access to, and the results of Veteran and program-related surveys.
FOIA – Process and information related to Freedom of Information Act (FOIA) efforts occurring at VA.
These resources represent what was harvested and analyzed as part of our landscape analysis, merging many of the patterns present across individual datasets. They’re organized using a REST-centric approach to turning data into API resources, which allows for data access via HTTP. Many of the keywords identified as part of the landscape analysis have been rolled-up into higher level areas — such as PTSD, mental health, and suicide — would exist across services, programs, and resources.
These suggested resources are derived from about 65% of the top-level resources identified across all the top paths, file formats, tables, and forms. They represent a nice cross-section of resources across all the data formats, but also reflect the general web presence of the VA. Our list also provides a coherent stack of resources that could be developed, deployed, and maintained in support of the central veteran APIs, offering personalized and generalized data experiences that would benefit Veterans and their families.
Centralize focus on the Veteran
From a data perspective, the most important resource above all is the Veteran and their personal data. Therefore, the identity and healthcare record of a Veteran should be at the front and center of any API deployed as part of the Lighthouse API platform initiative. This requires full knowledge and accurate information about a Veteran. In others words, in order for the Lighthouse’s APIs to work well, there must be a robust identity and access management in place, as well as detailed, layered, portable, and usable Veteran profiles.
One thing that became evident during our work is the need for greater personalization of data across almost every resource that we identified. Where there’s value in having general information available (for example, medical facilities), this data becomes exponentially more valuable when it’s personalized, localized, and made more relevant to the Veteran who is browsing, searching, engaging. Therefore, there are two types of engagement models with the resources that we propose: (1) general access without knowledge of the Veteran and (2) personalized knowledge of the Veteran via custom configuration settings that determine the relevancy of the data and content when these are made available via APIs and within applications.
There are existing portal efforts such as vets.gov that are available as part of the VA’s online presence. The personalization efforts occurring there should be reflected across the design and operation of the Lighthouse’s APIs. By designing APIs to operate in generalized or personalized mode, this would empower API developers to act on behalf of a Veteran using OAuth tokens. If a token’s present, each API will act in a more personalized manner and allow for localization based upon a Veteran’s preferences and history of interactions. This personalization layer should act as a bridge between the core healthcare record of a Veteran and the other resources that we outlined above.
Writing, not just reading data
Many of the resources that we identified represent read-only access to data and content. It’s important to note that getting access to data and content is useful; however, a significant portion of the resources that we harvested and gathered through conversations with the community will require the ability to write information via APIs. Forms, surveys, and other feedback loops will need to allow for APIs that not only GET data, but also POST and PUT data as part of their operations. This additional operations will round-off the Lighthouse’s stack of resources, helping to ensure that services provide a two-way street for engaging with the VA community.
In addition to reading, the ability to write data and content will be a deciding factor in whether applications built on top of the APIs will deliver meaningful value to Veterans and their supporters. If information is only being pushed outwards, many applications will be seen as having little to no value to users, developers, and operators of the Lighthouse API platform. To help ensure meaningful value is delivered to everyone involved, all applications should be capable of sharing usage data and feed analytics and support feedback loops between users and the platform operators. With the ability to write data, the APIs will lack meaning and substance, and will contribute to lack of adoption and integration.
Focus on the source of the data
A common misconception in conducting a landscape analysis such as the one we performed is to assume that the data discovered can be published via any APIs that are deployed as part of the next phase of work. That’s rarely the situation, because most of the discovered data is just published snapshots derived from existing data sources. This is certainly the case with the VA. Much of the data we discovered is unusable in its current state due to lack of normalization, duplication, being out of date, and other noise and clutter. Many of the XML and JSON files identified provider a much cleaner option for transforming into web APIs. However, with any resource identified, it’s more desirable to integrate the original source of data than relying on published snapshots.
Even after coming to a consensus on the data resources to transform into APIs, the next phase of work should focus on identifying the data sources for each of the targeted resource areas, and not relying on published data that already exist across websites. While it’s tempting, and sometimes necessary to rely on published data for the source of API data, it increases the chance that an API will eventually become dormant, out-of-date, and cause many of the issues that we’ve seen play out with the existing VA datasets. Our landscape analysis came at the resource prioritization from an external, public perspective. We recommend a subsequent, more internal landscape analysis to identify the data sources for important resource types emerging from this landscaping effort.
Improve domain management
Moving forward, it’d be logical to have a standardized approach to naming subdomains for both web and API properties in support of VA operations. Establishing a common approach to naming city, state, regional, program, research, and other resource areas would provide human- and machine-readable access to these resources. This might be difficult to do for web properties with so much legacy infrastructure, but the API platform provides an opportunity to establish a standardized approach moving forward.
Leverage common data formats
Our landscape analysis revealed a lack of a consistency when it comes to vocabulary, schema, and data formats. Most of the data published is derived from an existing system or represents a human-directed process. There’s a significant amount of fluff and noise surrounding these valuable data and a lack of consistent naming and field types.
Based upon the resources that we’ve identified for your consideration, there are a handful of existing data formats the VA should consider. Some of these are already underway, while others are not currently reflected in the Lighthouse’s efforts, but are used by other government entities to publish data in a consistent manner.
Fast Healthcare Interoperability Requirements (FHIR) – FHIR is already in-motion at the VA, but worth highlighting here. FHIR provides an anchor for why common data schema formats are relevant to other resources beyond Veteran healthcare records.
Open Referral (211) – Open Referral is a common schema and API specification for defining human services, including organizations, locations, and services, along with all the supporting information and metadata that goes with this core set of resources.
Open311 – Open311 is a common data format for reporting problems and issues at the municipal level, but can easily be adopted for establishing feedback loops at any level of government. It provides a common schema for how large volumes of information get submitted via API infrastructure.
Schema.org – A common schema vocabulary that provides object definitions for almost every resource identified throughout this landscape analysis, and the recommended list of resources above.
There are undoubtedly other open data formats that can be leveraged. Common microformats and other RFCs should also be considered, but these can be addressed during the define and design stages of the API development lifecycle, once individual resources have been decided upon. Common formats help ensure resources are interoperable and reusable across VA groups; they also help bring teams together to speak in a common language, using a common dictionary, which will go a long ways to standardizing how data is published and consumed.
Improve analytical information
We had hoped there would have been more analytical information available to help rank the resource data that we identified. We did incorporate the ranking information available from data.gov as an input into our resource prioritization. However, we relied mostly on publication frequency and the overall occurrence of each keyword to help weight relevance. The existence of a word in a path, title, and file name gives it a weight, which can be amplified for every occurrence, providing us with adequate levels of prioritization, grouping, and organization to help us understand each topics importance. If a topic exists frequently across VA web properties, and exists as a sectional grouping, and title of data file, it has importance and relevance.
The lack of analytics, or access to current analytics, across existing VA data sources demonstrates the importance of having a consistent and comprehensive analytics strategy across the VA’s data. There should be download counts for all machine-readable files. And, more importantly, real-time analytics for the consumption of this data via simple web APIs. There should be regional- and program-related ata. We should have personalized data that reflects what’s most important to Veterans. We should understand what’s relevant what isn’t through strategically-designed analytics across web and API operations. The lack of analytics is why we’re working to identify relevant data sources, so those can be made more available and analytics become the default — not an afterthought.
Continue refining the landscape analysis
Our landscape analysis produced a lot of information that was messy and difficult to work with. We can continue to make another pass, which would involve refining indexes, optimizing title and filename parsing, and developing key phrase, plural word, and other dictionaries to make the results much more refined.
There was a lot of data to harvest, process, and make sense of in a two-week sprint. However, we feel that we were able to do a good job of making sense of what was captured. Another sprint could easily be spent sorting through all of the data targeted, separating quality datasets from the more messier ones. Creating a dictionary to translate words and rehydrate acronyms would be very useful to help make further sense of what’s available in CSV, XLS/XLSX, JSON, and XML files. More work could also be done around forms: identifying the types available; defining their search mechanisms; defining what input parameters they allow (whether GET or POST); and unlocking further details on how they store data. Form and table data often have a direct connection to backend databases, which make them more valuable than some of the published data files.
All of the data from the landscape analysis has been published to GitHub, minus the primary index of harvested and processed URLs. Those are too big to publish as JSON to GitHub, but we’ll evaluate how best to provide access to each site index using a solution such as Amazon AMI. We also started experimenting with a secondary spider solution in order to generate an index of the VA website and can publish those indexes as separate GitHub repositories within a single GitHub repository when completed. We feel like these newer indexes could provide a much richer approach to understanding the data and content across the VA’s web properties. And allow for other researchers and analysts to fork and work to make sense of the data that they contain.
Incorporate user research
It’s critical that any further landscape analysis focused on uncovering valuable data resources from across the VA’s web presence is combined with user research activities, such as the series of design workshops that we conducted. Doing so will provide a human perspective on what’s most important to Veterans and their supporters.
We strongly encourage the Lighthouse program to conduct a similar workshop activity to the one that we ran, leveraging the VA’s much stronger outreach capability in order to attract an even larger and more diversified representation of the Veteran community’s data resource needs.
We also recommend that the Lighthouse program consider using the service blueprinting technique as a way to help identify and prioritize specific APIs for deployment. For example, a service blueprint could be created for a specific interaction that Veterans have with the VA, such as trying to find information on healthcare facilities. It’s likely that any service blueprints you want to create could be acquired by chunking the work into multiple micro-purchases. At the very least, we recommend trying to do at least one as an experiment. Once specific APIs are identified, you could then map them against a 2x2 prioritization matrix, based on how high they score against two main criteria: Veteran Experience Impact (y-axis) and Readiness to Execute (x-axis).
Conclusion: this journey is just beginning
The landscape analysis for the VA doesn’t end here. Just like the resulting API effort, the evaluation of the VA’s web presence should be an ongoing process. Work should continue to help identify what datasets are being published to the VA’s web properties, and to incorporate these datasets into API operations or to replace them with API-driven solutions. In the long term, there shouldn’t be any tables, forms, CSV files, JSON files, XML files, or XLS/XLSX files without a direct connection to the API platform. Eventually all data should be derived from a federated, but standardized, set of API platforms that are designed, deployed, and managed consistently as part of the VA Lighthouse effort.
Hopefully the work conducted here provides a base of resources for the Lighthouse program to consider as it moves forward. Ideally, everything uncovered as part of this work eventually becomes an API, or part of a suite of APIs. We understand that this won’t be a reality anytime soon, but we worked diligently to uncover the most valuable resources and to provide a concise list of data resources that could be turned into APIs and used to begin driving web, mobile, and desktop applications that serve Veterans and their families. There’s a wealth of resources available to Veterans across the VA’s websites. The challenge now is how do we ensure these resources deliver value consistently across many platforms? A simple, consistent, and usable API stack is the answer.
Lessons to share
This project is one of the VA’s first experiments using the microconsulting model in support of the Lighthouse initiative. Sharing what went well, what didn’t, and what could we have been done better — all in the name of continuous improvement — is the responsibility of everyone involved in order to make not only the Lighthouse initiative a success, but the microconsulting model as well. With that said, here’s what we have to share:
We thoroughly enjoyed working on this as a micro-project. We felt that the short-timeboxed, tightly-scoped nature of the work focused our efforts on executing only the most essential activities, giving even more meaning to inherently impactful work. As Parkinson’s Law states, “work expands so as to fill the time available for its completion.”
Looking back, our approach to this project involved some known unknowns (and some unknown unknowns) from a technical standpoint. In particular, the question of how well our spidering process could scale to handle the VA’s enormous web footprint. It would have been best to propose conducting an agile “spike” activity as a small micro-project in order to gain risk-reducing knowledge.
For micro-projects under a tight schedule and for which there are external dependencies (for example, scheduling interviews or workshops with external participants), some lead time may be necessary before formally kicking off the project.
Those people who participated in our facilitated workshops expressed extreme gratitude for the opportunity to contribute to the progress of the Lighthouse program. Working in the open and co-creating with the public will not only foster an engaged community of supporters, but will also lead to better quality outcomes.
While our workshop activities were extremely valuable in giving us a human perspective on our landscape analysis, we felt that there could have been even greater representation from the Veteran community. We should have been more proactive about leveraging the VA’s outreach capability to draw in an even more dense and diverse group of Veterans and their supporters.
In my world, OpenAPI is always a primary actor, and the tooling and services that put it to work are always secondary. However, I’d say that 80% of the people I talk with are the opposite, putting OpenAPI tooling in a primary role, and the OpenAPI specification in a secondary role. This is the primary reason that many still see Swagger tooling as the value, and haven’t made the switch to the concept of OpenAPI, or understand the separation between the specification and the tooling.
Another way in which you can see the importance of OpenAPI tooling is the slow migration of OpenAPI 2.0 to 3.0 users. Many folks I’ve talked to about OpenAPI 3.0 tell me that they haven’t made the jump because of the lack of tooling available for the specification. This isn’t always about the external services and tooling that supports OpenAPI 3.0, it is also about the internal tooling that supports it. It demonstrates the importance of tooling when it comes to the evolution, and adoption of OpenAPI. It demonstrates the need for the OAI community to keep investing in the development and evangelism of tooling for the latest version.
I am going to work to invest more time into rounding up OpenAPI tooling, and getting to know the developers behind them, as I prepare APIStrat in Nashville, TN. I’m also going to invest in my own migration to OpenAPI 3.0. The reason I haven’t evolved isn’t because of lack tooling, it is because of a lack of time, and the cognitive load involved with thinking new ways. I fully grasp the differences between 2.0 and 3.0, but I just don’t have intuitive knowledge of 3.0 in the way I do for 2.0. I’ve spent hundreds of hours developing around 2.0, and I just don’t have the time in my schedule to make similar investment in 3.0–soon!
If you need to get up to speed on the latest when it comes to OpenAPI 3.0 tooling I recommend checking out OpenAPI.Tools from Matt Trask (@matthewtrask) and Crashy McCiderface (aka Phil Sturgeon) (@philsturgeon). It is the best source of OpenAPI tooling out there right now. If you are still struggling with the migration from 2.0 to 3.0, or would like to see a specific solution developed on top of OpenAPI 3.0, I’d love to hear from you. I’m working to help shape the evolution of the OpenAPI tooling conversation, as well as tell stories about what tools are available, or should be available, and how they are can be put to work on the ground at companies, organizations, institutions, and government agencies.
After eight years of educating people about sensible API security and management, I’m always amazed at how many people I come across who still think public web APIs are about giving away access to your data, content, and algorithms for free. I regularly come across very smart people who say they’d be doing APIs, but they depend on revenue from selling their data and content, and wouldn’t benefit from just putting it online for everyone to download for free.
I wonder when we stopped thinking the web was not about giving everything away for free? It is something I’m going to have to investigate a little more. For me, it shows how much education we still have ahead of us when it comes to informing people about what APIs are, and how to properly manage them. Which is a problem, when many of the companies I’m talking to are most likely doing APIs to drive internal systems, and public mobile applications. They are either unaware of the APIs that already exist across their organization, or think that because they don’t have a public developer portal showcasing their APIs, that they are much more private and secure than if they were openly offering them to partners and the public.
Web API management has been around for over a decade now. Requiring ALL developers to authenticate when accessing any APIs, and the ability to put APIs into different access tiers, limit that the rate of consumption, while logging and billing for all API consumption isn’t anything new. Amazon has been extremely public about their AWS efforts, and the cloud isn’t a secret. The fact that smart business leaders see all of this and do not see that APIs are driving it all represents a disconnect amongst business leadership. It is something I’m going to be testing out a little bit more to see what levels of knowledge exist across many fortune 1000 companies, helping paint of picture of how they view the API landscape, and help me quantify their API literacy.
Educating business leaders about APIs has been a part of my mission since I started API Evangelist in 2010. It is something that will continue to be a focus of mine. This lack of awareness is why we end up with damaging incidents like the Equifax breach, and the Cambridge Analytica / Facebook scandal. Its how we end up with so many trolls on Twitter, and an out of balance API ecosystems across federal, state, and municipal governments. It is a problem that we need to address in the industry, and work to help educate business leaders around common patterns for securing and managing our API resources. I think this process always begins with education and API literacy, but is a symptom of the disconnect around storytelling about public vs private APIs, when in reality there are just APIs that are secured and managed properly, or not.
It is APIStrat time again! This time it is in Nashville, Tennessee! We are in the early stages of the event, but we are getting close to the deadline of the call for papers. We’ve assembled another rockstar ensemble for this round to help us steer the event, and review talk submissions once the CFP process has closed. I just wanted to take a moment and recognize the folks who are helping out and make sure they get the recognition they deserve.
First up are the six members of the APIStrat steering committee, playing different leadership roles in the conference, making sure everything gets done by September:
- Darrel Miller (@darrel_miller)– Workshop Chair - Microsoft
- Erin McKean (@emckean) – Program Chair - IBM
- Kin Lane (@kinlane) – Marketing Chair - API Evangelist
- Lorinda Brandon (@lindybrandon) – Community Chair - Capital One DevExchange
- Steven Willmott (@njyx) – Chair - Red Hat.
- Tim Burks (@timburks) – Program Vice Chair - Google
Then we have assembled nineteen folks on the program committee who will be reviewing your talk submissions before you can get on stage at APIStrat in Nashville:
- Amy Palamountain (@ammeep) - GitHub
- Ash Hathaway (@Ash_Hathaway) - Pivotal
- Carly Thelander (@CarlyNThelander) - MilkRun
- David Biesack (@davidbiesack) - Apiture
- Gail Frederick (@screaminggeek) - eBay
- George Atala (@gmatala) - Quantero Capital
- Glenn Block (@gblock) - Auth0 https://auth0.com/
- Gregory Koberger (@gkoberger) - ReadMe
- James Higginbotham (@launchany) - LaunchAny
- Jennifer Riggins (@jkriggins) - the eBranding Ninja
- Joey French (@josephtfrench) - Intrinio
- Kyle Dallaire (@kyledallaire) - Capital One Digital
- Melissa Jurkoic (@melissa_jurkoic) - Amadeus Hospitality
- Phil Sturgeon (@philsturgeon) - WeWork
- Sai Vennam (@sai_vennam) - IBM https://www.ibm.com/
- Shelby Switzer (@switzerly) - Healthify
- Skip Hovsmith (@skiphovsmith) - Critical Blue
- Taylor Barnett (@taylor_atx) - Stoplight
- Tessa Mero (@TessaMero) - Cisco
- Yina Arenas (@yina_arenas) - Microsoft Graph
Than you to everyone for helping do the hard work of making sure APIStrat not only continues, but continues to represent the wider API community. Everyone is doing this work because they care about the community, and want to make the event as good as, or better than it has been in the past. This is the 9th edition of APIStrat, spanning New York, San Francisco, Amsterdam, Chicago, Berlin, Austin, Boston, Portland, and now Nashville! It has been a pretty wild ride.
While we have everyone we need for these committees, we still need help in other areas. First, get your talk submitted before the CFP closes next week. Second, we need your financial support, so make sure you consider sponsoring APIStrat, and help make sure Nashville rocks. Beyond that we can use some help spreading the word. We are looking to grow the event beyond the usual 500 threshold, helping expand participation in the event, as well as the OpenAPI Initiative. If you want to help, feel free to ping me anytime, and I’ll see you in Nashville.
I’m kicking off a busy week of travel and talks this week in DC with a discussion about delivering microservices at federal agencies at DevNation Federal on Tuesday, June 5th, 2018. I was invited by Red Hat to come speak about the work I’m doing as API Evangelist across federal agencies. You can find me in the afternoon lineup, sharing my stories title “The Tech, Business, and Politics of APIs In Federal Government”. Focusing on information gathered as part of my research, workshops, and consulting across the public and private sector.
My talk reflects my work to motivate federal agencies to do APIs over the last five years, and help pollinate the ideas and practices I gather from across the private sector, and understand which ones will work in the public sphere. Not everything about doing APIs at startups and in the enterprise translates perfectly to delivering APIs in the federal government, but there are many practices that will help agencies better serve the people. My goal is to open up discussion with government employees and contractors, to help figure out what works and what doesn’t–sharing stories along the way.
Let me know if you are going to be at DevNation Federal let me know. I am happy to make some time to talk, and hear what you are up to with APIs. I depend on these hallway conversations to populate my blog with stories. I’ll be in town around noon, and there until around 5 or so until I head over to the DC API Meetup for my second talk of the day. Thanks to Red Hat for having me out. I enjoy doing talks for Red Hat events, as they tend to reflect more of the audience I’m looking for, with a focus on more open source, and a little less proprietary focus when it comes to delivering government technology. I’ll see you in Washington D.C. on Tuesday!
After I speak at DevNation Federal in Washington DC this Tuesday, I am going to give a similar talk at the DC API API User Group that evening. I love going to the Meetups in DC, partly because my good friend Gray Brooks runs the event, but also because I’ve been working to jumpstart API conversations in Washington DC since 2012 when I held the first DC edition of API Craft. I was on a mission to jumpstart API Craft gatherings around the country that year, and it makes me happy to see the API Meetup culture continuing to thrive in DC, where other places it has died out.
At the DC API Meetup I’ll be giving a variation of my talk that I’m giving earlier that day at DevNation Federal. Talking about the technology, business, and politics of doing APIs, with an emphasis on a consistent and repeatable API lifecycle. I’ll be reworking my regular material in light of current projects I’m working on at the federal level including with the VA, FDIC, HHS, and beyond. Sharing stories about how a microservice approach can help make government services more agile, flexible, and delivered in smaller more bite sized chunks–helping move the IT conversation forward across federal agencies.
If you can’t make it to DevNation Federal, I recommend you head out to the DC API User Group later that evening. I’d love to get a chance to hang out with you and talk about APIs. I’m always impressed with the folks who turn out for the DC API Meetup, consistently providing a fresh opportunity to discuss APIs and the impact they are making across the federal government. Organizer Gray Brooks has his finger on the pulse of what is going on across agencies, way beyond what I am capable of from the outside-in. I look forward to hanging out in DC, and hope you can make it out Tuesday to talk some APIs with me.
I’ve been evaluating API management providers, and this important stop along the API lifecycle in which they serve for eight years now. It is a space that I’m very familiar with, and have enjoyed watching it mature, evolve, and become something that is more standardized, and lately more commoditized. I’ve enjoyed watching the old guard (3Scale, Apigee, and Mashery) be acquired, and API management be baked into the cloud with AWS, Azure, and Google. I’ve also had fun learning about Kong, Tyk, and the next generation API management providers as they grow and evolve, as well as some of the older players like Axway as they work to retool so that they can compete and even lead the charge in the current environment.
I am renewing efforts to study what each of the API management solutions provide, pushing forward my ongoing API management research, understanding what the current capacity of the active providers are, and potentially they are pushing forward the conversation. One of the things I’m extremely interested in learning more about is the connector, plugin, and extensibility opportunities that exist with each solution. Functionality that allows other 3rd party API service providers to inject their valuable services into the management layer of APIs, bringing other stops along the API lifecycle into management layer, allowing API providers to do more than just what their API management solution delivers. Turning the API management layer into much more than just authentication, service plan management, logging, analytics, and billing.
Over the last year I’ve been working with API security provider ElasticBeam to help make sense of what is possible at the API management layer when it comes to securing our APIs. ElasticBeam can analyze the surface area of an API, as well as the DNS, web, API management, web server, and database logs for potential threats, and apply their machine learning models in real time. Without direct access at the API management layer, ElasticBeam is still valuable but cannot respond in real-time to threats, shutting down keys, blocking request, and other threats being leveraged against our API infrastructure. Sure, you can still respond after the fact based upon what ElasticBeam learns from scanning all of your logs, but without being able to connect directly into your API management layer, the effectiveness of their security solution is significantly diminished.
Complimenting, but also contrasting ElasticBeam, I’m also working with Streamdata.io to help understand how they can be injected at the API management layer, adding an event-driven architectural layer to any existing API. The first part of this would involve turning high volume APIs into real time streams using Server-Sent Events (SSE). With future advancements focused on topical streaming, webhooks, and WebSub enhancements to transform simple request and response APIs into event-driven streams of information that only push what has changed to subscribers. Like ElasticBeam, Streamdata.io would benefit being directly baked into the API management layer as a connector or plugin, augmenting the API management layer with a next generation event-driven layer that would compliment what any API management solution brings to the table.
Without an extensible connector or plugin layer at the API management layer you can’t inject additional services like security with ElasticBeam, or event-driven architecture like Streamdata.io. I’m going to be looking for this type of extensibility as I profile the features of all of the active API management providers. I’m looking to understand the core features each API management provider brings to the table, but I’m also looking to understand how modern these API management solutions are when it comes to seamlessly working with other stops along the API lifecycle, and specifically how these other stops can be serviced by other 3rd party providers. Similar to my regular rants about API service providers always having APIs, you are going to hear me rant more about API service providers needing to have connector, plugin, and other extensibility features. API management service providers can put their APIs to work driving this connector and plugin infrastructure, but it should allow for more seamless interaction and benefits for their customers, that are brought to the table by their most trusted partners.
I conducted an API workshop at Genscape in Boston yesterday. I thoroughly enjoyed the conversation with the technical folks there, and found their pragmatic, yet educated views on APIs refreshing. I spent the day going through the usual stops along the API lifecycle, but found some of our breakout conversations during the API design section to be what has stuck with me on my train ride back to New York. Specifically the discussions around versioning, content negotiation and the overall design of paths, parameters, and potentially providing query language layers on top of APIs.
After going through much of my textbook API design patterns for paths, and parameters, then working my through content negotiation and headers, we kept finding ourselves talking about what their clients were capable of. Bringing the API design discussion back around to why REST is still dominate in my opinion–simplicity, and lowering the bar for consumers. Time after time we found ourselves talking about their target consumers, the spreadsheet wielding data analyst, and how the bar needed to be kept low to make sure their needs were being met. Sure, we can quickly get academic and theoretical with the design practices, but if they fall on deaf ears, and API consumers do not adopt and use the API-driven tools, does it matter?
I know that all of us API “thawt liters” would rather folks just do things the right way, but this isn’t always the reality on the ground. Most people within real world businesses don’t have the time or luxury to learn the right way of doing things, and take the time to disrupt their flow with new ways of doing things. Most of the time we need to just get people the data and other resources they need, in the tooling they are comfortable with (spreadsheet cough cough), and get out of their way. Sure, we should still be introducing people to new concepts, and working to strike a balance when it comes to the API design patterns we adopt, but it should be about striking a balance between reality on the ground, and the future we want to see.
During the workshop we kept coming back around to simple, plain language, URIs that provide flexibility with path variables, as well as potentially relevant query parameters, that allow people to get CSV and JSON representations of the resources they need. With Excel as the target client, once again I find myself minimizing header usage, and maximizing the paths, and simplifying the data models in which folks can retrieve via APIs. I see this in industry after industry, and across the government agencies I am working with. I know that we all want our APIs to reflect the best possible patterns, and leverage the latest technology, but many of the people who will be potentially consuming our APIs, just want to get at the resources we are serving up, and do not have the time and the bandwidth to get on board with anything too far beyond their current mode of getting business done.
None of this will shift me evangelizing the “proper way to design APIs”, but it reminds me (once again), at how immovable the business world can actually be. APIs are having a significant impact on how we develop web, mobile, desktop, and device applications, but one of the reasons web APIs have found so much success is that they are simple, scrappy, and flexible. Being able to serve up valuable data and other resources through simple URLs, allowing anyone to take them and put to use in the application of their choice has fed the explosion in the API we see across the landscape today. It is why we still see just as many poorly designed APIs in 2018 as we saw in 2010. It is because the best design doesn’t always win. Sometimes you just need the right design for the job, and the one that will make sense to the audience who will be consuming it. It is why hypermedia, GraphQL, and other design patterns will continue to be the choice of some practitioners, but simple, plain, REST will keep dominating the landscape.
A question I get regularly in my API workshops is, “how should we be organizing all of our microservices?” To which I always recommend they tune into what the API Academy team is up to, and then I dance around give a long winded answer about how hard it is for me to answer that. I think in response, I’m going to start asking for a complete org chart for their operations, list of all their database schema, and a list of all their clients and the industries they are operating in. It will still be a journey for them, or me to answer that question, but maybe this response will help them understand the scope of what they are asking.
I wish I could provide simple answers for folks when it came to how they should be naming, grouping, and organizing their microservices. I just don’t have enough knowledge about their organization, clients, and the domains in which they operate to provide a simple answer. It is another one of those API journeys an organization will have to embark on, and find their own way forward. It would take so much time for me to get to know an organization, its culture, resources, and how they are being put to use, I hesitate to even provide any advice, short of pointing them to what the API Academy team publishes books, and provides talks on. They are the only guidance I know that goes beyond the hyped of definition of microservices, and actually gets at the root of how you do it within specific domains, and tackle the cultural side of the conversation.
A significant portion of my workshops lately have been about helping groups think about delivering services using a consistent API lifecycle, and showing them the potential for API governance if they can achieve this consistency. Clearly I need to back up a bit, and address some of the prep work involved with making sure they have an organizational chart, all of the schema they can possibly bring to the table, existing architecture and services in play, as well as much detail on the clients, industries, and domains in which they operate. Most of my workshops I’m going in blind, not knowing who will all be there, but I think I need a section dedicated to the business side of doing microservices, before I ever begin talking about the technical details of delivering microservices, otherwise I will keep getting questions like this that I can’t answer.
Another area that is picking up momentum for me in these discussions is a symptom of of the lack of API discovery, and directly related to the other areas I just mentioned. You need to be able to deliver APIs along a lifecycle, but more importantly you need to be able to find the services, schema, and people behind them, as well as coherently speak to who will be consuming them. Without a comprehensive discovery, and the ability to understand all of these dependencies, organizations will never be able to find the success they desire with microservices. They won’t be any better than the monolithic way many organizations have been doing things to date, it will just be much more distributed complexity, which will achieve the same results as the monolithic systems that are in place today.
I know, time flies and we are all very busy, but have you submitted your talk for APIStrat Nashville? The call for papers closes next Friday, June 8, 2018 at 11:59 PM PST! This is the second edition of the conference being run by the OpenAPI community, and is something you aren’t going to want to miss. We are bringing together the usual API community suspects, but also working to build upon the work that is going on in the fast growing OpenAPI community. The conference continues to be its usual voice of the API community, and won’t be all about the OpenAPI specification, but it it will definitely be where you want to be showcasing any groundbreaking work you are doing around the specification.
The CFP process for APIStrat Nashville asks three main questions:
- How will the audience benefit from your presentation?
- Why should YOU be the one to give this talk? You have a unique story. Tell it.
- Be prepared to explain how this fits into the API ecosystem.
After answering these questions we are looking for talks in the following areas of doing APIs: API Design, API For the Greater Good, API Management, API SDKs & Clients, API Security, API Success Stories, API Transformations, API Testing, API Usability, At The Protocol Level, Digital Transformation, GraphQL, Hypermedia, Machine Learning, Microservices, REST, RPC systems, and Standards & Definitions. However, don’t let these topics put you in a box. We want you to think about whatever you think will make the biggest impact on the audience, and hopefully the wider API community.
Do not wait until the last moment to get your talk submitted! Last year, as the program chair I spoke with a number of people who had waited too long, and didn’t get their talk in front of the program committee early on. As soon as the submission are close they will begin spending time reviewing the talks, and while we may consider accepting late submissions, they are unlikely to get the attention that the earlier submissions will. If you have any questions about past events, or what might speak to this years audience, feel free to reach out to me personally. While I’m not in charge of the program committee this year, I’m still happy to help answer questions, and share my thoughts on what will catch their attention. Here is the link to the APIStrat CFP page for when you are ready, and I look forward to hearing what you have to say this year in Nashville!
I’m kicking off a micro-procurement project with the Department of Veterans Affairs (VA) this week. I’m going to to be conducting one of my API low hanging fruit campaigns for them, where I help them identify the best possible data sets available across their public websites for turning into APIs. The project is one of many small projects the federal agency is putting out there to begin working on their agency wide API platform they are calling Lighthouse. Laying the groundwork for better serving veterans through a robust stack of microservices that each do one thing really well, but can be used in concert to deliver the applications the agency needs to meet their mission.
While not specifically a project to develop a microservice. The landscape analysis project is an API focused research project, that is testing a new procurement model called microprocurement. At my government consulting agency partnership Skylight Digital, my partner in crime Chris Cairns has been pushing for a shift in how government tackles technology projects, pushing them to do in smaller chunks that essentially can be put on the credit card in less than 10K increments. So far we’ve been doing consulting, research, and training related projects like how to create a bug bounty program, and internal evangelism strategies. Now we are kicking our campaign into high gear and pushing more agencies to think about microprocurement for microservices–the VA was the first agency to buy into this new way of thinking about how government IT gets delivered.
I am working with all of my partners to help me craft potential services that would fit into the definition of a microprocurement project. I’ve been working to educate people about the process so that more API experts are on hand to respond when agencies publish microprocurement RFPs on Github like the VA did, but I also want to make sure and have a suite of microprocurement packages that federal agencies can choose from as well. I’ve been working with my partner Dave O’Neill over at APIMetrics to provide me with some detail on an API testing microprocurement package that federal agencies could buy into for less than 10K, providing the follow value:
- API discovery and creation of a Postman collection
- Annual monitoring for up to 10 APIs including the following:
- Weekly and monthly emailed quality statements – CASC scores and SLOs/SLA attainment
- Interface to named operations tooling for alerts
- Public dashboards for sharing API status and SLA with all stakeholders.
Providing a pretty compelling annual API monitoring package that government agencies could put on the credit card, and help ensure the quality of service provided across federal agencies when it comes to providing APIs. Something that is desperately needed across federal agencies who in my experience are mostly conducting “API nonitoring”, where they are not monitoring anything. Ideally, ALL federal agencies provide an SLA with the public and private APIs they are serving up, and relying on outside entities like APIMetrics to be doing the monitoring from an external perspective, and potentially even different regions of the US–ensuring everyone has access to government services.
I’ll publish anther story after I kick off my work with the VA. I’ll also keep beating the drum about microprocurement opportunities like this one with APIMetrics. If you are an API service provider who has an interesting package you think federal agencies would be interested in–let me know. Also, if you are a government agency and would like to learn more about how to conduct microprocurement projects feel free to drop me a line, or reach out to Chris Cairns (@cscairns) over at Skylight Digital–he can set you up. We are just getting going with this types of projects, but I’m pretty optimistic about the potential they can bring to the table when it comes to delivering IT projects–an approach that reflects the API way of doing things in small, meaningful chunks, and won’t break the bank.
Most of this API stuff isn’t technical. For APIs to work at scale within a single company, a wider industry, or across government agencies, you need people who are committed to evangelism, coaching, and communication around everything that is occurring across API ecosystems. We often times get caught up in our work, and operating within our silos and forget to email, call, and just walk next door sometimes to share stories of what is going on. If you are a regular reader of my work you know how I feel about storytelling, and just how important it is to all of this working or not. Which is why I like to make sure that I showcase the storyelling of other evangelists who are working their magic, and spreading API knowledge within their domains.
I’m a big fan of what the GSA is up to with APIs across the federal government, and specifically what my hero Gray Brooks does to support API efforts acorss multiple federal agencies. He does a lot of legwork, but one thing that has stood out for me is his work to establish an API community of practice, and share relevant API stories on the US Government APIs Google Group. To help highlight what he is up to, and hep amplify the knowledge he puts out there, I wanted to share his latest post from the Google Group in it’s entirety because it is, well, beautiful.
I hope your week has wrapped up well so far. As I mentioned back in April, this and other efforts have been formalized as the API Community of Practice, part of Digital.gov's communities of practice program. I hope you all will put this forum to use, sharing your thoughts, questions, and ideas about federal APIs you're working on or thinking about. In the meantime, here's some odds and ends that you all may find interesting.
- One of the API programs here at GSA put out a call for developer input ahead of some further work on the sam.gov API. It's not surprise that the wonderful API Evangelist took note and offered both praise and some wise counsel. It's well worth a read and offers good advice to us all, if we'll take it.
- FOIA.gov launched its developer hub, which has a lot to it, including the ability to submit a FOIA request via API.
- If you want to see a really neat set of examples of developer engagement from an agency, check out the pair of A+ listserves set up by CMS for the Blue Button and Quality Payment Program APIs. This is a great example for us all to follow.
- Here are some APIs that are either new (or at least are new to me!) - NASA's Mission Design API, CPSC's Violations & NEISS API, and USDA's MARS API.
- A digital services team in the New Zealand government is keeping a public list of NZ government APIs as they find them.
- Also, it's just worth noting the success of CFPB's open source API documentation project, DOCter. That good work not only was used for their API documentation, but has been re-used by GSA for its API documentation template and is now employed by many of the agency's APIs. Regulations.gov and Interior are two more agencies putting it to good use. A good . reminder for us too all just reuse each other's stuff! (Interested in more examples you can quickly and easily fork and re-use? Here's a bunch more.)
- NIST including API requirements and specs in some its efforts.
Last but not least, there's a bunch to see from VA and HHS/CMS that's a pretty big deal. This is the first time that I've been seeing agency principles actively talking and giving speeches about APIs and API strategies. This is 100% what needs to be happening across government and I def. recommend diving into these efforts and trying to drive the same at your agencies. VA is the largest civilian agency in the federal government. If they can be doing this, so can any of us! VA
- Back in March, the VA secretary gave a speech driving an open API pledge among healthcare providers. More coverage here.
- VA has launched a new developer hub around the effort, called Lighthouse.
- One cool sub-project is a set of micropurchases they are trying in this area.
- The National Coordinator for Health Information Technology at HHS is writing blogposts like "APIs: A Path to Putting Patients at the Center" and a senior advisor to him is writing to announce a Secure API Server challenge that gave out $50,000 in prizes.
- Meanwhile, the CMS administrator gave a big speech about APIs at the same event as the VA secretary, also worth checking out.
- As part of those efforts, the US Digital Service team at CMS helped roll out v2 of the Blue Botton API.
- More on this rollout here and here. The first link has a great interview I strongly recommend.
Anyhoo, I'm sure there's much more going on that this. Definitely share anything you all are finding or working on as it comes up. In the meantime, take care can have a good holiday weekend.
"name": "api.data.gov - API analytics for federal agencies",
"name": "US Government API listserve",
It is tough to directly measure the ROI from the work Gray Brooks does in a purely business sense, but this type of outreach and storytelling is ESSENTIAL to APIs working across federal agencies. Gray is paying attention to what is happening across federal agencies and doing the hard work to share relevant stories across teams, agencies, and with the general public. While it would be valuable for each federal agency to do this themselves, it isn’t something that is always in the DNA of each agency, and many teams just don’t have the time or resources to publish stories on a regular basis–making Gray’s work all the more important to the overall health of APIs at the federal level.
We need more Gray Brooks. We tried to clone him at a super top secret agency and failed. He is just too special. His work reflects what needs to exist at every single federal government agency, as well as at the state and municipal levels. We also need this kind of dedicated evangelism and storytelling to exist within industries with healthcare, education, energy, and other specialized API evangelist uncovering the meaningful stories, and working to make them more public. If you aren’t tuned into the Government API Google Group I recommend tuning in. If you don’t know Gray Brooks I recommend going to one of the DC Web API User Group which Gray operates–you’ll likely get a chance to meet him in person, and hear stories of the API change he is making first hand.
I was on a call for a federal government API platform project with my partner in crime Chris Cairns (@cscairns) of Skylight.Digital. We were back channeling in our Slack channel during the call, when he said, “I always imagine you participating in these things, finding topics you haven’t covered or emphasized from a certain angle, and then writing a blog post in real time.” He was right, I had taken notes on a couple of new angles regarding the testing, monitoring, and understanding performance of APIs involved with federal government projects.
The single conference call resulted in about seven potential stories in my notebook. I’m guessing that only four of them will actually end up being published. However, the conversation does a good job at highlighting my style for generating stories on the blog, which is something that allows me to publish 3-5 posts a day–keeping things as active as I possibly can, generating traffic, and bringing attention to my work. If you’ve ever worked closely with me, you know that I can turn anything into a story, even a story like this, about how I create stories. ;-)
Think of API Evangelist as my public workbench where I work through the projects I’m tackling each day. It is how I make sense of my research, the projects I am working on, and the conversations I am having. All of which generates SEO exhaust, which brings more attention to my work, extends its reach, and ultimately brings in more work. It is a cycle that helps me work through my ideas in a way that forces me to make them more coherent (hopefully) along the way, while also making them immediately available to my partners, customers, and readers.
Chris has been pushing this concept forward and advocating that we be public with as many of the Skylight.Digital responses and proposals as we possibly can, which resulted in me publishing both of our responses to the first, and the second Department of Veteran Affairs (VA) RFIs. He has also been pushing other Skylight.Digital partners to be public with their proposals and responses, which folks were skeptical about at first, but then it started making sense when they see the effect it has, and the publicity it can generate.
This approach to storytelling isn’t something you can do effectively right out of the gate. It takes practice, and regular exercising–I have eight years of practice. However, I feel it is something that anyone can do eventually. You just have to find your own way of approaching it, and work on establishing your own voice and style. It is something that I’ve found to be essential to how I do business, but also something I find to be personally rewarding. I enjoy working through my ideas, and telling stories. It is always the one aspect of my work that I look forward to doing each day, and I feel like something is missing the days I don’t get to craft any posts. I really enjoy that every moment has the potential to be an API story, it really makes each day an adventure for me.
I have been intrigued by the people who reached out after I published a story about my partner program. One of the companies that reached out was the API integration agency Bridge Software, who reflects the next generation of software development groups who are emerging to focus exclusively on API integration.
I’ve had several calls with the Bridge Software team this week, discussing the possible ways in which we can work together. I’m regularly in need of software development resources that I can refer projects to, and I’m also interested in hearing more interesting stories about how businesses are integrating with APIs. Of course, Bridge Software is looking for more customers, which is something I can definitely help with. So we started brainstorming more about how they can regularly feed me stories that I can develop into posts for API Evangelist, and I can help send new business their way.
One of the most valuable resources I possess as the API Evangelist, is access to real world stories from the trenches of API providers and integrators. These sources of stories have been the bread and butter for API Evangelist for eight years. I have depended on the stories that businesses share with me to generate the 3,276 posts I have published on the blog since 2010. So I am stoked to have a new source of fresh stories, from a team who is neck deep in API integrations each day, and will be actively gathering stories from their regular developer team meetings.
I’m looking forward to working with the Bridge Software team, and getting new stories from the API integration trenches. I’m also looking forward to assisting them with finding new customers, and helping them with deliver on their API integration needs. If you are looking for help with API integrations, Bridge Software has a pretty sophisticated approach to delivering integrations that has test driven development at its core. Feel free to reach out to the team, but make sure and tell them where you heard about them, to get special treatment. ;-) Also, make sure you stay tuned for more stories about the API integrations stories they will be sharing with me on a regular basis, which I will be weaving into my regular cadence of storytelling here on the blog.
I’m working to define APIs at the Department of Veterans Affairs (VA) on several fronts right now. I’ve provided not just one, but two detailed reponses to their Lighthouse API platform RFIs. Additionally, I’m providing guidance to different teams who are working on a variety of projects occurring simultaneously to serve veterans who receive support from the agency.
As I do this work, I am hyper aware of the privatization machinations of the current administration, and the potential for APIs to serve these desires. APIs aren’t good, bad, nor are they neutral. APIs reflect the desires of their owners, and the good or bad they can inflict is determined by their consumers. Because APIs are so abstract, and are often buried within web and mobile applications, it can be difficult to see what they are truly doing on a day to day basis. This is why we have API management solutions in place to help paint a picture of activity in real time, but as we’ve seen with the Facebook / Cambridge Analytica, it requires the API provider to be paying attention, and to possess incentives that ensure they will actually care about paying attention, and responding to negative events.
By re-defining existing VA internal systems as APIs, we are opening up the possibility that digital resources can be extracted, and transferred externally. Of course, if these systems remain the sources of data, VA power and control can be retained, but it also opens up the possibilities for power to eventually be transferred externally–reversing the polarities of government power and putting it into private hands. Depending on your politics, this could be good, or it could be bad. Personally, I’m a fan of a balance being struck, allowing government to do what it does best, and allowing private, commercial entities to help fuel and participate in what is happening–both with proper oversight. This reflects the potential benefits APIs can bring, but this good isn’t present by default, and could shift one way or the other at any moment.
Early on, APIs appeared as a promising way to deliver value to external developers. They held the promise of delivering unprecedented resources to developers, thing they would not normally be able to get access to. Facebook promised democratization using it’s social platform. While silently mining and extracting value from it’s application end-users, executed by often unknowing, but sometimes complicit developers. A similar transfer of power has occurred with Google Maps, which promised unprecedented access to maps for developers, as well as web and mobile maps for cities, businesses, and other interests. Over time we’ve seen that Google Maps APIs have become very skilled at extracting value from cities, business, and end-users, again executed unknowingly, and knowingly by developers. This process has been playing out over and over throughout the last decade of the open data evolution occurring across government at all levels.
Facebook and Google have done this at the platform level, extracting value and maintaining control within their operations, but the open data movement, Facebook and Google included, have been efficiently extracting value from government, and giving as little back as it possibly can. Sure, one can argue that government benefits from the resulting applications, but this model doesn’t ensure that the data and content stewards within government retain ownership or control over data, and benefit from revenue generated from the digital resources–leaving them struggling with tight budgets, and rarely improving upon the resources. This situation is what I fear will happen at the Department of Veterans Affairs (VA), allowing privatization supporters to not actually develop applications that benefit veterans, but really focusing on extracting value possessed by the VA, capturing it, and generating revenue from it, without ever giving back to the VA–never being held accountable for actually delivering value to veterans.
I have no way of testing the political motivations of people that I am working with. I also don’t want to slow or stop the potential of APIs being used to help veterans, and ensure their health and insurance information can become more interoperable. I just want to be mindful of this reality, and revisit it often. Openly discussing it as I progress in my work, and as the VA Lighthouse platform evolves. Sometimes I regret all the API evangelism I have done, as I witness the damage they can do. However, the cat is out of the bag, and I’d rather be involved, part of the conversation, and pushing for observability, transparency, and open dialogue about the realities of opening up government digital resources to private entities via APIs. Allowing people who truly care about serving veterans to help police what is going on, maximize the value and control over digital resources into the hands of people who are truly serving veteran’s interests, and minimizing access by those who are just looking to make money at veteran’s expense.
The General Services Administration (GSA) has an open call for help to test the new FedBizOpps API. Setting a pretty compelling precedent for releasing APIs in the federal government, slowly bringing federal agencies out of their shell, and moving the API conversation forward in government. Hopefully it will be something that other federal agencies, and other levels of government will consider as they move forward on their API journeys.
It is good to see federal agencies reaching out to ask for help in making sure APIs are well designed, deliver value, and operate as expected. In my experience, most government agencies are gun-shy when it comes to seeking outside help, and accepting criticism when it comes to their work. The GSA is definitely the most progressive on this front, but they are leading by example, showing other agencies what is possible, and something that hopefully will spread. Something I am always keen to support with my storytelling here on API Evangelist.
One thing I’d caution federal agencies on when seeking outside feedback in this area, is to be mindful of fatigue in the private sector when it comes to working for free. API providers have been encouraging developers to document, test, develop code libraries, and other essential aspects of operating APIs for years now, and many developers are growing weary of this exploitation by companies who can afford to pay, but choose not to. While it is good that government is getting on the API bandwagon here, but be aware that you are about 5+ years behind, and the tides are shifting.
I’d recommend thinking about how micro-procurement models emerging at the Department of Veterans Affairs (VA) and other agencies can be applied to testing, performance, security, and other needs agencies will have. Think about how you can create $500, $1,000, and other opportunities for professional testers to make money. Of course, you are going to have to establish a way of vetting developers, but once you do, you are going to get a higher level work than you would for free. It might take more effort to lay the groundwork, but it will be something that will pay off down the road with a community of external professionals who can hep tackle micro-tasks that emerge while developing and operating government APIs.
Overall, I’m stoked to see stuff like this come out of the government. I’m just eager to see it spread to other federal agencies, shifting how digital services are delivered. I’m also eager to see more mock or virtualized APIs exposed publicly, long before any code is actually written. Bringing in outside opinions early on, allowing APIs to be more efficiently delivered–ensuring they are more mature, without the costly evolutionary and versioning process. I’m going to push more projects I’m involved in to follow GSA’s lead, and solicit outside testers to join in and provide feedback at critical steps in the API delivery process. The more agencies do this, the more likely they will to have a trusted group of outside developers, to step up when they need help, and have tasks like this to accomplish.
I’ve been on five separate government reflated projects lately where finding modular OpenAPI-driven open source tooling has been a top priority. All of these projects are microservice-focused and OpenAPI-driven, and are investing significant amounts of time looking open source tools that will help with design governance, monitoring, testing, and security, and interact with the Jenkins pipeline. Helping government agencies find success as their API journey picks up speed, and the number of APIs grows exponentially.
Selling to the federal government can be a long journey in itself. They can’t always use the SaaS solutions many of us fire up to get the job done in our startup or enterprise lives. Increasingly government agencies are depending on open source solutions to help them move projects forward. Every agency I’m working with is using OpenAPI (Swagger) to drive their API lifecycle. While not all have gone design (define) first, they are using them as the contract for mocking, documentation, testing, monitoring, and security. The teams I’m working with are investing a lot of energy looking for, vetting, and testing out different open source modules on Github–with varying degrees of success.
Ideally, there was an OpenAPI-driven marketplace, or federated set of marketplaces like OpenAPI.Tools. I’ve had one for a while, but haven’t kept up to date–I will invest some time / resources into it soon. My definition of an OpenAPI tool marketplace would be that it is OpenAPI-driven, and open source. I’m fine with there being other marketplaces of OpenAPI-driven services, but I want a way to get at just the actively maintained open source tools. When it comes to serving government this is an important, and meaningful distinction. I’d also like to encourage many of the project owners to ensure there is CI/CD integration, as well as make sure their projects are actively supported, and they are willing to entertain commercial implementations.
While there wouldn’t always be direct commercial opportunities for open source tooling owners to engage with federal agencies, there would be through contractors and subcontractors. Working for federal agencies is a maze of forms and hoop jumping, but working with contractors can be pretty straightforward if you find the right ones. I don’t think you will get rich developing OpenAPI-driven tooling that serves the API lifecycle, but I think with the right solutions, support, and team behind them, you can make a decent living developing them. Especially as the lifecycle expands, and the number of services being delivered grows, the need for specialized, OpenAPI-driven tools to apply across the API lifecycle is only going to increase. Making it something I’ll be writing more stories about as I hear more stories from the API trenches.
I’m going to try and spend time working with Phil Sturgeon (@philsturgeon) and Matt Trask (@matthewtrask) on API.Tools, as well as give my own toolbox some love. If you have an open source OpenAPI-driven tool you’d like to get some attention feel free to ping me, and make sure its part of API.Tools. Also, if you have a directory, catalog, or marketplace of tools you’d like to showcase, ping me as well, I’m all about supporting diversity of choice in the space. I have multiple federal agencies ear right now when it comes to delivering along the API lifecycle, and I’m happy to point agencies and their contractors to specific tools, if it makes sense. Like I said, there won’t always be direct revenue opportunities, but they are implementations that will undoubtedly lead to commercial opportunities in the form of consulting, advising, and development opportunities with the contractors and subcontractors who are delivering on federal agency contracts.
I’ve had a huge demand for putting on API seminars for a variety of enterprise groups lately. Helping bring my API 101, history, lifecycle, and governance knowledge to the table. I’ve conducted seminars in the UK and France in April, and have more in Virginia, Massachusetts, and California in May. I’ve been working with a variety of companies, institutions, and government agencies to plan even more internal seminars, which is something that is proving to be challenging for some groups who are just getting started on their API journey.
While groups are keen on me coming visit to share my API knowledge, some of them are having trouble getting me through legal, get me into their payment systems, and making everyone’s schedules work. Some of them don’t have a problem figuring it out, while others are facing significant amounts of bureaucracy and friction due to the size, complexity, and legacy of their organizations. To help simplify the process for organizations to participate in my seminars I am going to begin planning a rolling schedule of seminars beginning in New York City.
Every two weeks, on Tuesday and Wednesday I’ll be planning to do a 2-day API lifecycle seminars in NYC. I’m going to be focusing on delivering 12 stops along the API lifecycle, targeting my readers and customers who have already embarked on their API journey:
- Definition - The basics of OpenAPI, Postman, and using Github to manage API definitions.
- Design - Entry level API design, including touching on API governance and guidance.
- Deployment - Looking at the big world of how APIs can deployed on-premise, and the clouds.
- Virtualization - Understanding mocking, and API, as well as data virtualization.
- Authentication - Thinking about the common patterns of authentication for API consumption.
- Management - Covering the services, tools, and approaches to managing APIs in operation.
- Discussion - Open discussion about anything covered throughout the day, and beyond.
- Documentation - Demonstrating how to deliver portals, documentation, and other resources.
- Testing - Providing information on the monitoring and testing of API infrastructure.
- Security - Walking through API security beyond just API management and authentication.
- Support - Discussing the importance of providing direct and indirect support for APIs.
- Evangelism - Looking at how to evangelize your APIs internally and externally.
- Integration - Thinking briefly about about API integration concerns for API providers.
- Discussion - Open discussion about anything covered throughout the day, and beyond.
I’m kicking off this seminar series in New York City, with the following dates proposed to get the ball rolling:
- June 12th & 13th 2018 from 8:30 AM to 4:30 PM each day.
- June 26th & 27th 2018 from 8:30 AM to 4:30 PM each day.
I’m going to be charging $1,000.00 per person for the two day seminars. I’m setting the minimum bar for attendance to be 5 people, otherwise I won’t be scheduling the seminar, and will be pushing forward to the next dates. I’ll make sure and let everyone know at least 2 weeks in advance if we don’t get the expected attendance. All seminars are open to anyone who would like to attend, but I am also happy to conduct private group seminars, with minimum of five people in attendance. My goal is to conduct a regular cadence of seminars, that people know they can plan on, and participate in as it works for their schedule, but also provide more stability for my schedule as well.
The objective with these seminars is to make it easier for companies, institutions, and government agencies to participate in my seminars, while also forcing them out of their bubbles. I’m finding the toll of me coming onsite can vary widely, and the value to seminar attendees is lower when they aren’t forced to leave their bubbles. So, I am looking to force people out of their usual domains, so that they begin thinking about how they’ll be doing APIs, and playing nicely with others–even if they are only intended for internal use. I find the process helps you to think outside your normal daily operations, and is easier for me to not have to navigate coming on site to deliver seminars.
I’ve published both initial dates to Eventbrite to test the waters, and we’ll see if I can shift some of the demand I’m getting to this format. I will also keep pushing dates out every other week, and in July I’ll probably start looking at dates in Washington DC, then London, Paris, and probably some other west coast US locations. If you have interest in participating in my API lifecycle seminars and / or have specific requests for venues I’d love to hear from you. I’m looking to continue formalizing my process as well as schedule, and bringing my seminars wherever they are need, but doing so in a more organized fashion that helps you get your team what they need, but will also be something that is more sustainable for me in the long term.
I’m a big fan of API sandboxes, labs, and other virtualization environments. API sandboxes should be default in heavily regulated industries like banking. I also support the virtualization of schema and data used across API operations, like I am doing at the Department of Veterans Affairs (VA), with synthetic healthcare data. I’m very interested in anything that moves forward the API virtualization conversation, so I found the live sporting API simulations over at SportRadar very interesting.
SportRadar’s “live simulations give you the opportunity to test your code against a simulation of live data before the preseason starts or any time! Our simulation system replays select completed games allowing you to view our API feeds as if they were happening live.” Here are the details of their NFL Official API simulations that run every day:
- 11:00 am - Data is reset for the day’s simulations.
- 1:00 pm - PST week 1 games will run – Oakland at Houston, Detroit at Seattle, Miami at Pittsburgh, and New York at Green Bay
- 2:00 pm – PST week 2 games will run - Seattle at Atlanta, Houston at New England, Green Bay at Dallas, and Pittsburgh at Kansas City
- 3:00 pm – PST week 3 games will run – Green Bay at Atlanta and Pittsburgh at New England
- 4:00 pm – PST week 4 games will run – New England vs Atlanta
Providing a pretty compelling evolution in the concept API virtualization when it comes to event data, or data that will happen on a schedule. I wanted to write up this story to make sure I bookmarked this as part of my API virtualization research. If I don’t write about it, it doesn’t happen. As I’m working to include API sandboxes, lab environments, and other API virtualization approaches in my consulting, I’m keen to add new dimensions like API simulation, which provide great material for workshops, presentations, and talks.
I feel like ALL APIs should have some sort of sandbox to play in while developing. Not just heavily regulated industries, or those with sensitive production data and content. It can be stressful to have to develop against a live environment, and having a realistic sandbox, labs, and simulations which provide complete copies of production APIs, along with real world synthetic data can help developers be more successful in getting up and running. I’ll keep profiling interesting approaches like this one out of SportRadar, and add to my API virtualization research, helping round off my toolbox when it comes to helping API providers develop their sandbox environments.
I’ve been evaluating microservices at scale, using their OpenAPI definitions to provide API design guidance to each team using Github. I just finished a round of work where I took 20 microservices, and evaluated each OpenAPI definition individually, and made design suggestions via an updated OpenAPI definition that I submitted as a Github issue. I was going to submit as a pull request, but I really want the API design guidance to be a suggestions, as well as a learning experience, so I didn’t want them to feel like they had to merge all my suggestions.
After going through all 20 OpenAPI definitions, I took all of them and dumped them into a single local repository where I was running Jekyll via localhost. Then using Liquid I created a handful of web pages for looking at the APIs across all microservices in a single page–publishing two separate reports:
- API Paths - An alphabetical list of the APIs for all 20 services, listing the paths, verbs, parameters, and headers in a single bulleted list for evaluation.
- API Schema - An alphabetical list of the APIs for all 20 services, listing the schema definitions, properties, descriptions, and types in a single bulleted list for evaluation.
While each service lives in its own Github repository, and operates independently of each other, I wanted to see them all together, to be able to evaluate their purpose and design patterns from the 100K level. I wanted to see if there was any redundancy in purpose, and if any services overlapped in functionality, as well as trying to see the potential when all API resources were laid out on a single page. It can be hard to see the forest for the trees when working with microservices, and publishing everything as a single document helps flatten things out, and is the equivalent of climbing to the top of the local mountain to get a better view of things.
Jekyll makes this process pretty easy to do by dumping all OpenAPI definitions into a single collection, which I can then navigate easily using Liquid on a single HTML page. Providing a comprehensive report of the functionality across all microservices. I am calling this my microservices monolith report, as it kind of reassembles all the individual microservices into a single view, letting me see the collective functionality available across the project. I’m currently evaluating these service from an API governance perspective, but at a later date I will be taking a look from the API integration perspective, and trying to understand if the services will work in concert to help deliver on a specific application objective.
I’ve written about how the startup community has driven the value exchange that occurs at the API management layer down a two lane road with API monetizatin and plans. To generate the value that investors have been looking for, we have ended up with a free, pro, enterprise approach to measuring value exchanged with API integration, when in reality there is so much more going on here. Something that really becomes evident when you begin evaluating the API conversations going on across government at all levels.
In conversation after conversation I have with government API folk I hear that they don’t need API management reporting, analysis, and billing features. There is a perception that government isn’t selling access to APIs, so API management measurement isn’t really needed. Missing out on a huge opportunity to be measuring, analyzing, and reporting upon API usage, and requiring a huge amount of re-education on my part to help API providers within government to understand how they need to be measuring value exchange at the API management level.
Honestly, I’d love to see government agencies actually invoicing API consumers at all levels for their usage. Even if the invoices aren’t in dollars. Measuring API usage, reporting, quantifying, and sending an invoice for usage each month would help bring awareness to how government digital resources are being put to use (or not). If all data, content, and algorithms in use across federal, state, and municipal government were available via APIs, then measured, rate limited, and reported upon across all internal, partner, and public consumers–the awareness around the value of government digital assets would increase significantly.
I’m not saying we should charge for all access to government data. I’m saying we should be measuring and reporting upon all access to government data. The APIs, and a modern API management layer is how you do this. We have the ability to measure the value exchanged and generated around government digital resources, and we shouldn’t let misconceptions around how the private sector measures and generates revenue at the API layer interfere with government agencies benefitting from this approach. If you are publishing an API at a government agency, I’d love to learn more about how you are managing access to your APIs, and how you are reporting upon the reading and writing of data across applications, and learn more about how you are sharing and communicating around these consumption reports.
I checked out the Github repositories for twenty microservices of one of my clients recently, looking understand what is being accomplished across all these services as they work independently to accomplish a single collective objective. I’m being contracted with to help come in blindly and provide feedback on the design of the APIs being exposed across services, and help provide guidance on their API lifecycle, as well as eventually API governance when things have matured to that level. Right now we are addressing pretty fundamental definition and design issues, but eventually we’ll hopefully graduate to the next level.
A complete and up to date README for each microservice is essential to understanding what is going on with a service, and a robust OpenAPI definition is critical to breaking down the details of what each API delivers. When you aren’t part of each service’s development team it can be difficult to understand what each service does, but with an up to date README and OpenAPI, you can get up to speed pretty quickly. If an service is well documented via its README, and the API is well designed, and the surface area is reflected in it’s OpenAPI, you can go from not knowing what a service does to, understanding its value within hopefully minutes, not hours.
When each service possesses an OpenAPI it becomes possible to evaluate what they deliver at scale. You can take all APIs, their paths, headers, parameters, and schema and out them in different ways so that you can begin to paint a picture of what they deliver in aggregate. Bringing all the disparate services back together to perform together in a sort of monolith concert, while still acknowledging they all do their own thing independently. Allowing us to look at how many different service can be used in concert to deliver a single application, or potentially a variety of application instances. Thinking critically about each independent service, but more importantly how they all work together.
I feel like many groups are still struggling with decomposing their monolithic systems into separate services, and while some are doing so in a domain-driven way, few are beginning to invest in understanding how they move forward with services in concert to deliver on application needs. Many of the groups I’m working with are so focused on decomposing and tearing down, they aren’t thinking too critically about how they will make all of this begin work together again. I see monolith systems working like a massive church organ which take a lot of maintenance, and require a single (or handful) of knowledgeable operators to play. Where microservices are much more like an orchestra, where every individual player has a role, but they play in concert, directed by a conductor. I feel like most groups I’m talking with are just beginning the process of hiring a conductor, and have a bunch of musicians roaming around–not quite ready to play any significant productions quite yet.
The topic of API discovery has been picking up momentum in 2018. It is something I’ve worked on for years, but with the number of microservices emerging out there, it is something I’m seeing become a concern amongst providers. It is also something I’m seeing more potential vendor chatter, looking to provide more services and tooling to help alleviate API discovery pain. Even with all this movement, there is still a lot of education and discussion to occur on the subject to help bring people up to speed on what is API discovery.
The most common view of what is API discovery, is when you need to find an API for developing an application. You have a need for a resource in your application, and you need to look across your internal and partner resources to find what you are looking for. Beyond that, you will need to search for publicly available API resources, using Google, Github, ProgrammableWeb, and other common ways to find popular APIs. This is definitely the most prominent perspective when it comes to API discovery, but it isn’t the only dimension of this problem. There are several dimensions to this stop along the API lifecycle, that I’d like to flesh out further, so that I can better articulate across conversations I am having.
Another area that gets lumped in with API discovery is the concept of service discovery, or how your APIs will find their backend services that they use to make the magic happen. Service discovery focuses on the initial discovery, connectivity, routing, and circuit breaker patterns involved with making sure an API is able to communicate with any service it depends on. With the growth of microservices there are a number of solutions like Consul that have emerged, and cloud providers like AWS are evolving their own service discovery mechanisms. Providing one dimension to the API discovery conversation, but different from, and often confused with front-end API discovery and how developers and applications find services.
One of the least discussed areas of API discovery, but is one that is picking up momentum, is finding APIs when you are developing APIs, to make sure you aren’t building something that has already been developed. I come across many organizations who have duplicate and overlapping APIs that do similar things due to lack of communication and a central directory of APIs. I’m getting asked by more groups regarding how they can be conducting API discovery by default across organizations, sniffing out APIs from log files, on Github, and other channels in use by existing development teams. Many groups just haven’t been good at documenting and communicating around what has been developed, as well as beginning new projects without seeing what already exists–something that will only become a great problem as the number of microservices grows.
The other dimension of API discovery I’m seeing emerge is discovery in the service of governance. Understand what APIs exist across teams so that definitions, schema, and other elements can be aggregated, measured, secured, and governed. EVERY organization I work with is unaware of all the data sources, web services, and APIs that exist across teams. Few want to admit it, but it is a reality. The reality is that you can’t govern or secure what you don’t know you have. Things get developed so rapidly, and baked into web, mobile, desktop, network, and device applications so regularly, that you just can’t see everything. Before companies, organizations, institutions, and government agencies are going to be able to govern anything, they are going to have begin addressing the API discovery problem that exists across their teams.
API discovery is a discipline that is well over a decade old. It is one I’ve been actively working on for over 5 years. It is something that is only now getting the discussion it needs, because it is a growing concern. It will be come a major concern with each passing day of the microservice evolution. People are jumping on the microservices bandwagon without any coherent way to organize schema, vocabulary, or API definitions. Let alone any strategy for indexing, cataloging, sharing, communicating, and registering services. I’m continuing my work on APIs.json, and the API Stack, as well as pushing forward my usage of OpenAPI, Postman, and AsyncAPI, which all contribute to API discovery. I’m going to continue thinking about how we can publish open source directories, catalogs, and search engines, and even some automated scanning of logs and other ways to conduct discovery in the background. Eventually, we will begin to find more solutions that work–it will just take time.
The more I study where the API management, gateway, and proxy layer is headed, the less I’m seeing a front-end or a backend, I’m seeing just a node. A node that can connect to existing databases, or what is traditionally considered a backend system, but also can just as easily proxy and be a gateway to any existing API. A lot has been talked about when it comes to API gateways deploying APIs from an existing database. There has also been considerable discussion around proxying existing internal APIs or web services, so that we can deploy newer APIs. However, I don’t think there has been near as much discussion around proxying other 3rd party public APIs–which flips the backend and frontend discuss on its head for me.
After profiling the connector and plugin marketplaces for a handful of the leading API management providers I am seeing a number of API deployment opportunities for Twilio, Twitter, SendGrid, etc. Allowing API providers to publish API facades for commonly used public APIs, and obfuscate away the 3rd party provider, and make the API your own. Allowing providers to build a stack of APIs from existing backend systems, private APIs and services, as well as 3rd party public APIs. Shifting gateways and proxies from being API deployment and management gatekeepers for backend systems, to being nodes of connectivity for any system, service, and API that you can get your hand on. Changing how we think about designing, deploying, and managing APIs at scale.
I feel like this conversation is why API deployment is such a hard thing to discuss. It can mean so many different things, and be accomplished in so many ways. It can be driven by a databases, or strictly using code, or be just taking an existing API and turning it into something new. I don’t think it is something that is well understood amongst developer circles, let alone in the business world. An API gateway can be integration just as much as it can be about deployment. It can be both simultaneously. Further complexing what APIs are all about, but also making the concept of the API gateway a more powerful concept, continuing to renew this relic of our web services past, into something that will accommodate the future.
What I really like about this notion, is that it ensures we will all be API providers as well as consumers. The one-sided nature of API providers in recent years has always frustrated me. It has allowed people to stay isolated within their organizations, and not see the world from the API consumer perspective. While API gateways as a node won’t bring everyone out of their bubble, it will force some API providers to experience more pain than they have historically. Understanding what it takes to to not just provide APIs, but what it takes to do so in a more reliable, consistent and friendly way. If we all feel the pain our integration and have to depend on each other, the chances we will show a little more empathy will increase.
I was in a conference session about an API design tool today, and someone asked if you could get at the OpenAPI definition behind the solution. They said yes, but quickly also said that the definition is boring and that you don’t want to be in there, you want to be in the interface. I get that service providers want you to focus on their interface, but we shouldn’t be burying, or abstracting away the API contract for APIs, we should always be educating people about it, an bringing it front and center in any service, tooling, or conversation.
Technology folks burying or devaluing the OpenAPI definition with business users is common, but I also see technology folks doing it to each other. Reducing OpenAPI to be just another machine readable artifact alongside other components of delivering API infrastructure today. I think this begins with people not understanding what OpenAPI is, but I think it is sustained by people’s view of what is technological magic and should remain in the hands of the wizards, and what should be accessible to a wider audience. If you limit who has access and knowledge, you can usually maintain a higher level of control, so they use your interface in the case of a vendor, or they come to you develop and build an API in the case of a developer.
There is nothing in a YAML OpenAPI definition that business users won’t be able to understand. OpenAPIs aren’t anymore boring than a Word document or Spreadsheet. If you are a stakeholder in the service, you should be able to read, understand, and engage with the OpenAPI contract. If we teach people to be afraid of the OpenAPI definitions we are repeating the past, and maintaining the canyon that can exist between business and IT/Developer groups. If you are in the business of burying the OpenAPI definition, I’m guessing you don’t understand the portable API lifecycle potential of this API contract, and simply see it as a config, documentation, or other technical artifiact. Or you are just in the business of maintaining control and power by being the gatekeeper for the API contract, similar to how we see database people defend their domain.
Please do not devalue or hide away the OpenAPI contract. It isn’t your secret sauce. It isn’t boring. It isn’t too technical. It is the contract for how a service will work, that will speak to business and technical groups. It is the contract that all the services and tools you will use along the API lifecycle will understand. It is fine to have the OpenAPI right behind the scenes, but always provide a button, link, or other way to quickly see the latest version, and definitely do not scare people away or devalue it when you are talking. If you are doing APIs, you should be encouraging, and investing in everyone being able to have a conversation around the API contract behind any service you are putting forward.
I’ve had a few API groups contact me lately regarding the challenges they are facing when it comes to supporting organizational, or industry-wide API definitions and schema. They were eager to support common definitions and schema that have been standardized, but were getting frustrated by not being able to do everything they wanted, and having to live within the constraints introduced by the standardized definitions. Which is something that doesn’t get much discussion by those of use who are advocating for standardization of APIs and schema.
Web APIs Come With Their Own Constraints
We all want more developers to use our APIs. However, with more usage, comes more responsibility. Also, to get more usage our APIs need to speak to a wider audience, something that common API definitions and schema will help with. This is why web APIs are working, because they speak to a wider audience, however with this architectural decision we are making some tradeoffs, and accepting some constraints in how we do our APIs. This is why REST is just one tool in our toolbox, so we can use the right tool, establish the right set of constraints, to allow our APIs to be successful. The wider our API toolbox, the wider the number options we will have available when it comes to how we design our APIs, and what schema we can employ
Allowing For Content Negotiation By Consumers
One way I’ve encouraged folks to help alleviate some of the pain around the adoption of common API definitions and schema is to provide content negotiation to consumers, allowing them to obtain the response they are looking for. If people want the standardized approaches they can choose those, and if they want something more precise, or custom they can choose that. This also allows API providers to work around the API standards that have become bottlenecks, while still supporting them where they matter. Having the best of both worlds, where you are supporting the common approach, but still able to do what you want when you feel it is important. Allowing for experimentation as well as standardization using the same APIs.
Participate In Standards Body and Process
Another way to help move things forward is to participate in the standards body that is moving an API definition or schema forward. Make sure you have a seat at the table so that you can present your case for where the problems are, and how to improve on the design, definition, and schema being evolved. Taking a lead in creating the world you want to see when it comes to API and schema standards, and not just sitting back being frustrated because it doesn’t do what you want it to do. Having a role in the standards body, and actively participating in the process isn’t easy, and it can be time consuming, but it can be worth it down the road and helping you better achieve your goals when it comes to your APIs operating as you aspect, as well as the wider community and industry you are serving.
Delivering APIs at scale won’t be easy. To reach a wide audience with your API it helps to be speaking a common vocabulary. This doesn’t always allow you to move as fast as you’d like, and do everything exactly as you envision. You will have to compromise. Operate within constraints. However, it can be worth it. Not just for your organization, but for the overall health of your community, and the industry you operate in. You never know, with a little patience, collaboration, and communication, you might learn even new approaches to defining your APIs and schema that you didn’t think about in isolation. Also, experimentation with new patterns will still be important, even while working to standardize things. In the end, a balance between standardized and custom will make the most sense, and hopefully alleviate your frustrations in moving things forward.
I’m always fascinated by the API design debate, and how many entrenched positions there are when it comes to the right way of doing it. Personally, I don’t see any right way of doing it, I just see many different ways to put the responsibility on the provider or the consumer shoulders, or sharing the load between them. I spend a lot of time profiling APIs, crafting OpenAPI definitions that try to capture 100% of the surface area of an API. Something that is pretty difficult to do when you aren’t the provider, and you are just working from existing static documentation. It is just hard to find every parameter and potential value, exhaustively detailing what is possible when you use an API.
When designing APIs I tend to lean towards exposing the surface area of an API in the path. This is my personal preference for when I’m using an API, but I also do this to try and make APIs more accessible to non-developers. However, I regularly get folks who freak out at how many API paths I have, preferring to have the complexity at the parameter level. This conversation continues with the GraphQL and other query language folks who prefer to craft more complex queries that can be passed in the body, to define exactly what is desired. I do not feel there is a right way of doing this, but it does reflect what I said early about balancing the load between provider and consumer.
The burden for defining and designing the surface area of an API resides in the providers court–only the provider truly knows the entire surface area. Then I’d add that when you offload the responsibility to the consumer using GraphQL, you are limiting who will be able to craft a query, putting API access beyond the reach of business users, and many developers. I feel that exposing the surface area of an API in the URL makes sense to a lot of people, and puts it all out in the open. Unless you are going to provide every possible enum value for all parameters, this is the only way to make 100% of the surface area of an API visible and known. However, depending on the complexity of an API, this is something that can get pretty unwieldy pretty quickly, making parameters the next stop be defining and designing the surface area of your API.
I know my view of API design doesn’t sync with many API believers, across many different disciplines. I’m not concerned with that. I’m happy to hear all the pros and cons of any approach. My objective is to lower the bar for entry into the API game, not raise the bar, or hide the bar. I’m all for pushing API providers to be more responsible for defining the surface area of their API, and not just offloading it on consumers to do all the work, unless they implicitly ask for it. In the end, I’m just a fan of simple, elegantly designed APIs that are intuitive and well documented using OpenAPI. I want ALL APIs to be accessible to everyone, even non-developers. I want them accessible to developers, minimizing the load on them to understand what is happening, and what all the possibilities are. I just don’t want to spend too much time on-boarding with an API. I just want to go from discovery to exploration in as little time as possible.
I’m turning different APIs into topical streams using Streamdata.io. I have been profiling hundreds of different APIs as part of my work to build out the Streamdata.io API Gallery, and as I’m creating OpenAPI definitions for each API, I’m realizing the potential for event and topical driven streams across many existing web APIs. One thing I am doing after profiling each API is that I benchmark them to see how often the data changes, applying what we are calling StreamRank to each API path. Then I try to make sure all the parameters, and even enum values for each parameter are represented for each API definition, helping me see the ENTIRE surface area of an API. Which is something that really illuminates the possibilities surrounding each API.
After profiling the Stack Exchange Questions API, I began to see how much functionality and data is buried within a single API endpoint, and was something I wanted to expose and make much easier to access. Taking a single OpenAPI definition for the Stack Exchange Questions API:
Then exploding it into 25 separate tech topic streaming APIs. Taking the top 25 enum value for the tags parameter for the Stack Overflow site, and exploding into 25 separate streaming API resources. To do this, I’m taking each OpenAPI definition, and generating an AsyncAPI definition to represent each possible stream:
I’m not 100% sure I’m properly generating the AsyncAPI currently, as I’m still learning about how to use the topics and streams collections properly. However, the OpenAPI definition above is meant to represent the web API resource, and the AsyncAPI definition is meant to represent the streaming edition of the same resource. Something that can be replicated for any tag, or any site that is available via the Stack Exchange API. Turning the existing Stack Exchange API into streaming topic APIs, that people can subscribe to only the topics they are interested in receiving updates.
At this point I’m just experimenting with what is possible with OpenAPI and AsyncAPI specifications, and understanding what I can do with some of the existing APIs I am already using each day. I’m going to try and turn this into a prototype, delivering streaming APIs for all 25 of the top Stack Overflow tags. To demonstrate what is possible on Stack Exchange, but also to establish a proof of concept that I can apply to other APIs like Reddit, Github, and others. Then eventually automating the creation of streaming topic APIs using the OpenAPI definitions for common APIs, and the Streamdata.io service.
I received an email from my CRON job API (EasyCRON) provider this morning about discontinuing the usage of social logins to their service with Gmail, Facebook, etc. Something that I think is a sign of things to come in response to the recent (and continued) bad behavior by many of the leading technology platforms. EasyCRON gave the following response for removing social logins from their service:
In order to: 1) prevent any confusion that could be caused by using third party platform’s authentication API, 2) decouple from those data greedy platforms, 3) and keep our system simple, we decide to stop using third party platform’s authentication API on EasyCron.
As much as I prefer one click login using my Github or Google (rarely use Facebook), I can’t argue with their logic. I’m a big fan of minimizing data sharing, surveillance, and of course keeping things simple. I wish we could trust tech companies to be good citizens with our data, but they seem to prefer repeatedly demonstrating they are more concerned with profits than they are about us.
I’m still on board with using Github login for my infrastructure related services, but will definitely stop using Google, Twitter, Facebook, and any other purely advertising related service for authentication purposes. As much of a pain in the ass as it might be to create yet another login, it is more of a pain in the ass to be surveilled, and exploited on a daily basis by technology platforms. I wish people would have more respect, and be better behaved, but things are what they are, and we should respond accordingly.
Thanks for fucking this one up for us all Facebook, Google, Twitter, and other platforms who just can’t see the big picture over their short term profit. I am going to stop advising API platforms utilize social logins as part of their API management and on-boarding process, due to it putting developers and consumers at risk for exploitation.
One of the arguments I hear for why API providers should be employing GraphQL is that they should just get out of developers way, and let them build their own queries so that they can just get exactly the data that they want. As an application developer we know what we want from your API, do not have us make many different calls, to multiple endpoints–just give us one API and let us ask for exactly what we need. It is an eloquent, logical argument when you operate and live within a “known bubble”, and you know exactly what you want. Now, ask yourself, will every API consumer know what they want? Maybe in some scenarios, every API consumer will know what they want, but in most situations, developers will not have a clue what they want, need, or what an API does.
This is where the GraphQL as a replacement for REST argument begins breaking down. In a narrowly defined bubble, where every developers knows the schema, knows GraphQL, and knows they want–GraphQL can make a lot of sense. In the autodidact alpha developer startup world this argument makes a lot of sense, and gets a lot of traction. However, not everyone lives in this world, and in this real world, API design can become very important. Helping people understand what is possible, learn the schema behind an API, and become more familiar with an API, until they have a better understanding of what they might want. I’m not saying GraphQL doesn’t have a place when you have a significant portion of your audience knowing what they want, I’m just saying that you shouldn’t leave everyone else behind.
To further turn this argument on its head, as a developer, if I know what I want, why make me build a query at all? Just give me a single URL with what I want! Don’t make me do the heavy lifting, and work to craft a query, just give me a single URL with minimal authentication, and let me get what I need in one click. Sure, it will take some time before you can craft enough URLs to meet everyone’s needs, but it will be worth it for those who you can. You can always craft new paths based upon requests, and yes, you can also augment your web APIs with a GraphQL endpoint for those neediest, most demanding developers who love building queries, and know exactly what they want. I guess my point is that there are a lot of definitions of knowing what you want, and how APIs can satisfy that, and not everyone will be in the same mindset as you are.
Now I have to bury in this last paragraph the fact that I’m not anti-GraphQL. I am anti-GraphQL being sold as a replacement for simpler, resource centered web APIs (aka REST). So if you are going to come at me with the Y U HATE GraphQL response–don’t. I’m just trying to show GraphQL believers that they are leaving some people behind with a GraphQL only approach, unless you are 100% sure that ALL your API consumers know GraphQL, know your schema, and know what they want. GraphQL is an important tool in the API toolbox, but it isn’t the one size fits all tool it is often sold as. After much contemplation, and working on the ground within enterprise groups, I am trying to put GraphQL to work in more use cases where it makes sense, and before I can do this I have to push back much of the misinformation that has been peddled about it, and undo the damage I’me seeing on the ground.
I have been studying API management for eight years now. I’ve spent a lot of time understanding the approach of leading API providers, and the services and tools put out there by API service providers from 3Scale to Tyk, and how the cloud service providers like AWS are baking API management into their clouds. API management isn’t the most exciting aspect of doing APIs, but I feel it is one of the most important elements of doing APIs, delivering on the business and politics of doing APIs, which can often make or break a platform and the applications that depend on it.
Employing a common API management solution, and having a solid plan in place, takes time and investment. To do it properly takes lots of regular refinement, investment, and work. It is something that will often seem unnecessary–having to review applications, get to know developers, and consider how they fit into the picture picture. Making it something that can easily be to pushed aside for other tasks on a regular basis–until it is too late. This is frequently the case when your API access isn’t properly aligned with your business model, and there are no direct financial strings attached attached to new API users, or a line distinguishing active from inactive users, let alone governing what they can do with data, content, and algorithms that are made accessible via APIs.
The API management layer is where the disconnect between API providers and consumers occur. It is also where the connection, and meaningful engagement occurs when done right. In most cases API providers aren’t being malicious, they are just investing in the platform as their leadership has directed, and acting upon the stage their business model has set into motion. If your business model is advertising, then revenue is not directly connected to your API consumption, and you just turn on the API faucet to let in as many consumers as you can possibly attract. Your business model doesn’t require that you play gatekeeper, qualify, or get to know your API developers, or the applications they are developing. This is what we’ve seen with Facebook, Twitter, and other platforms who are experiencing challenges at the API management layer lately, due to a lack of management over the last 5+ years.
There was no incentive for Facebook or Twitter to review applications, audit their consumption, or get to know their usage patterns. The business model is an indirect one, so investment at the API management layer is not a priority. They only respond to situations once it begins to hurt their image, and potentially rise up to begin to hurt their bottom line. The problem with this type of behavior is that other API providers see this and think this is a problem with doing APIs, and do not see it as a problem with not doing API management. That having a business model which is sensibly connected to your API usage, and a team who is actively managing and engaging with your API consumers is how you deal with these challenges. If you manage your API properly, you are dealing with negative situations early on, as opposed to waiting until there is a media shitstorm before you start reigning in API consumption on your platform.
Sensible API management is something I’ve been writing about for eight years. It is something I will continue to write about for many more years, helping companies understand the importance of doing it correctly, and being proactive rather than reactive. I’ll also be regularly writing about the subject to help API consumers understand the dangers of using platforms that do not have a clear business model, as it is usually a sign of this API management disconnect I’m talking about. It is something that will ultimately cause problems down the road, and contribute to platform instability, and APIs potentially being limited or shuttered as part of these reactionary responses we are seeing from Facebook and Twitter currently. I know that my views of API management are out of sync with popular notions of how you incentivize exponential growth on a platform, but my views are in alignment with healthy API ecosystems, successful API consumers, and reliable API providers who are around for the long term.
I have been working on several fronts to help with API efforts at the Department of Veterans Affairs (VA) this year, and one of them is helping quantify the deployment of a lab API environment for the platform. The VA doesn’t want it called it a sandbox, so they are calling it a lab, but the idea is to provide an environment where developers can work with APIs, see data just like they would in a live environment, but not actually have access to live patient data before they can prove their applications are reviewed and meet requirements.
One of the projects being used to help deliver data within this environment is called Synthea. Providing the virtualized data will be made available through VA labs API–here is the description of what they do from their website:
Synthea is an open-source, synthetic patient generator that models the medical history of synthetic patients. Our mission is to provide high-quality, synthetic, realistic but not real, patient data and associated health records covering every aspect of healthcare. The resulting data is free from cost, privacy, and security restrictions, enabling research with Health IT data that is otherwise legally or practically unavailable.
Synthea data contains a complete medical history, including medications, allergies, medical encounters, and social determinants of health, providing data can be used without concern for legal or privacy restrictions by developers to support a variety of data standards, including HL7 FHIR, C-CDA and CSV. Perfect work loading up into sandbox and lab API environments, allowing to developers to safely play around with building healthcare applications, without actually touching production patient data.
I’ve been looking for solutions like this for other industries. Synthea even has a patient data generator available on Github, which is something I’d love to see for every industry. Sandbox and labs environment should be default for any API, especially APIs operating within heavily regulated industries. I think Synthea provides a pretty compelling model for the virtualization of API data, and I will be referencing it as part of my work in hopes of incentivizing someone to fork it and use to provide something we can use as part of any API implementation.
I’ve been reading about all the work Facebook and Twitter have been doing over the last couple of weeks to begin asserting more control over their API applications. I’m not talking about the deprecation of APIs, that is a separate post. I’m focusing on them reviewing applications that have access to their API, and shutting off access to the ones who are’t adding value to the platform and violating the terms of service. Doing the hard work to maintain a level of quality on the platform, which is something they should have been doing all along.
I don’t want to diminish the importance of the work they are doing, but it really is something that should have been done along the way–not just when something goes wrong. This kind of behavior really sets the wrong tone across the API sector, and people tend to focus on the thing that went wrong, rather than the best practices of what you should be doing to maintain quality across API operations. Other API providers will hesitate launching public APIs because they’ll not want to experience the same repercussions as Facebook and Twitter have, completely overlooking the fact that you can have public APIs, and maintain control along the way. Setting the wrong precedent for API providers to emulate, and damaging the overall reputation of operating public APIs.
Facebook and Twitter have both had the tools all along to police the applications using their APIs. The problem is the incentives to do so, and to prioritize these efforts isn’t there, due to an imbalance with their business model, and a lack of diversity in their leadership. When you have a bunch of white dudes with a libertarian ethos pushing a company towards profitability with a advertising driven business model, investing in quality control at the API management layer just isn’t a priority. You want as may applications, users, and activity as you possibly can, and when you don’t see the abuse, harassment, and other illnesses, there really is no problem from your vantage point. That is, until you get called out in the press, or are forced to testify in front of congress. The reasons us white dudes get away with this is that there are no repercussions, we just get to ignore until it becomes a problem, apologize, perform a little bit to show we care, and wait until the next problem occurs.
This is the wrong API model to put out there. API providers need to see the benefits of properly reviewing applications that want access to their APIs, and the value of setting a higher bar for how applications use the API. There should be regular reviews of active APIs, and audits of how they are accessing, storing, and putting resources to work. This isn’t easy work, or inexpensive to do properly. It isn’t something you can put off until you get in trouble. It is something that should be done from the beginning, and conducted regularly, as part of the operations of a well funded team. You can have public APIs for a platform, and avoid privacy, security, and other shit-shows. If you need an example of doing it well, look at Slack, who has a public API that is successful, even with a high level of bot automation, but somehow manages to stay out of the spotlight for doing dumb things. It is because their API management practices are in better alignment with their business model–the incentives are there.
For the next 3-5 years I’m going to have to hear from companies who aren’t doing public APIs, because they don’t want to make the same mistake as Facebook and Twitter. All because Facebook and Twitter have been able to get away with such bad behavior for so long, avoid doing the hard work of managing their API platforms, and receive so much bad press. All in the name of growth and profits at all cost. Now, I’m going to have to write a post every six months showcasing Facebook and Twitter as pioneers for how NOT to run your platforms, explaining the importance of healthy API management practices, and investing in your API teams so they have the resources to do it properly. I’d rather have positive role models to showcase rather than poorly behaved role models who I have to work overtime to change perception and alter API provider’s behavior. As an API community let’s learn from what has happened and invest properly in our API management layers, properly screen and get to know who is building application on our resources, and regularly tune into and audit their behavior. Yes, it takes more investment, time, and resources, but in the end we’ll all be better off for it.
I have several projects right now that are needed a baseline for what is expected of microservices developers when it comes to the README for their Github repository. Each microservice should be a self-contained entity, with everything needed to operate the service within a single Github repository. Making the README the front door for the service, and something that anyone engaging with a service will depend on to help them understand what the service does, and where to get at anything needed to operate the service.
Here is a general outline of the elements that should be present in a README for each microservice, providing as much of an overview as possible for each service:
- Title - A concise title for the service that fits the pattern identified and in use across all services.
- Description - Less than 500 words that describe what a service delivers, providing an informative, descriptive, and comprehensive overview of the value a service brings to the table.
- Documentation - Links to any documentation for the service including any machine readable definitions like an OpenAPI definition or Postman Collection, as well as any human readable documentation generated from definitions, or hand crafted and published as part of the repository.
- Requirements - An outline of what other services, tooling, and libraries needed to make a service operate, providing a complete list of EVERYTHING required to work properly.
- Setup - A step by step outline from start to finish of what is needed to setup and operate a service, providing as much detail as you possibly for any new user to be able to get up and running with a service.
- Testing - Providing details and instructions for mocking, monitoring, and testing a service, including any services or tools used, as well as links or reports that are part of active testing for a service.
- Configuration - An outline of all configuration and environmental variables that can be adjusted or customized as part of service operations, including as much detail on default values, or options that would produce different known results for a service.
- Road Map - An outline broken into three groups, 1) planned additions, 2) current issues, 3) change log. Providing a simple, descriptive outline of the road map for a service with links to any Github issues that support what the plan is for a service.
- Discussion - A list of relevant discussions regarding a service with title, details, and any links to relevant Github issues, blog posts, or other updates that tell a story behind the work that has gone into a service.
- Owner - The name, title, email, phone, or other relevant contact information for the owner, or owners of a service providing anyone with the information they need to reach out to person who is responsible for a service.
That represent ten things that each service should contain in the README, providing what is needed for ANYONE to understand what a service does. At any point in time someone should be able to land on the README for a service and be able to understand what is happening without having to reach out to the owner. This is essential for delivering individual services, but also delivery of service at scale across tens, or hundreds of services. If you want to know what a service does, or what the team behind the service is thinking you just have to READ the README to get EVERYTHING you need.
It is important to think outside your bubble when crafting and maintaining a README for each microservice. If it is not up to date, or lacking relevant details, it means the service will potentially be out of sync with other services, and introduce problems into the ecosystem. The README is a simple, yet crucial aspect of delivering services, and it should be something any service stakeholder can read and understand without asking questions. Every service owner should be stepping up to the plate and owning this aspect of their service development, and professionally owning this representation of what their service delivers. In a microservices world each service doesn’t hide in the shadows, it puts it best foot forward and proudly articulates the value it delivers or it should be deprecated and go away.
p></p>When I talk to government agencies, non-profit organizations, or organizations doing internal APIs I regularly receive pushback about the information I share regarding API management, service composition, rate limiting, access tiers, and the other essential ingredients of the business of APIs. People tell me they aren’t selling access to their APIs, they don’t need to be measuring APIs like publicly available commercial APIs do. They don’t need a free, pro, and other tiers of access, and they won’t be invoicing their consumers like the majority of APIs I showcase from the API universe.
This phenomenon is just one of the many negative side effects of venture capital, and Silicon Valley startups driving the conversation around APIs. Most people can’t even see the value exchange that potentially happens at the API management layer, and only see one way commercial revenue. When in reality API management is all about measuring and developing an awareness of the value exchanged around all of an organizations digital assets. With charging consumers for access just one the many ways in which value exchange can be measured, quantified, and reporting upon, by invoicing the consumer. There are so many other ways in which value can be exchanged, incentivized, measured, and reported upon, but many people have tuned into a single dominant narrative, and are completely unaware there is more than one way of doing things.
Each API request can be recorded at the API management level. Whether it originated internally, from partners, or through public applications. Each resource path, HTTP verb (GET, POST, PUT, PATCH, DELETE), application and end-user token(s) can is recorded. The value generated by ANY API resource, and the exchange with consumers can be measured. Charging for GETs is such a narrow, and unimaginative view of what is possible. Just because many tech startups are interested in charging to GET data, and extracting value by encouraging unrestrained POST, PUT, PATCH, doesn’t mean this is how all API platforms should operate. It doesn’t reflect the value exchange many organizations are looking to facilitate, and honestly can become dangerous if not balanced with healthy privacy, security, and an ethical business model.
Service composition, policies, and analysis at the API management layer is one of the most important areas of API operations that organizations should be investing in and maturing. Not only are some organization I meet not investing heavily in this area, they are completely unaware of the best practices, and solutions provided by the API management tooling they’ve put into place. I blame startup API culture for this. Investors have had a significant amount of control over what API startups present to their customers, thus have controlled the narrative around how you do API management, and deliver the business of your APIs. I’ve had conversations with API management company leadership about measuring value exchange at government agencies, and it was a completely new concept to them. Something they hadn’t ever considered, because they were held capture by this dominant narrative.
I’m hoping with APIs moving more mainstream, and API management being baked into the cloud, we can begin moving beyond the damage that has been done by this narrative. I’m going to be telling more stories about what is possible at the API management layer, and looking for innovative approaches to showcase in my work. Hopefully helping everyone get back to more meaningful value exchange around our digital assets, rather than the one way exploitation that has been occurring for the last decade, creating monsters like Twitter and Facebook, and making venture capital folks wealthier. Allowing us to just get down to doing real business, balanced value exchanges between platforms, developers, and users. Shifting the narrative from one way of doing business that is just a cover for exploitative business models, and extracting as much value as you can with APIs and giving nothing back in return.
I’m always amazed what large organizations can achieve. They definitely have more resources, more momentum, and do way more than any single individual can achieve. However, as an individual actor I’m always also amazed at how large organization culture seems to always work to defend itself from change, and actively work to grind down API efforts I’m involved in. Even before I walk in the front door there are processes in place that work to ensure any effective API strategy gets ground down so that it won’t be as effective.
NDAs, intellectual property, background checks, schedules, logistics, leadership buy-in, physical security, legacy incidents, software, processes all seem to begin doing their work. Grinding down on anything new. I would ask folks to leave their baggage at home, but in most cases they aren’t even aware of the number, or the size of the suitcases they travel with. This is how things are done! This is the way things are! This is our standard way of operating! Even when we just want to have a conversation. Even when I’m just looking to share some knowledge with you and your team. I haven’t even begun talking about getting paid, I just want to share some wisdom with y’all, and help your team figure all of this out.
The machine isn’t setup for this time of knowledge sharing. This is why API can be so hard to achieve for some. The machine is setup to extract, aggregate, store, and control value–not share it. So when the machine encounters another entity that just wants to share, no string attached, it just doesn’t want it was designed to do. Even though many have made the conscious effort to open up doors and invite people in to share knowledge, innovate, and explore, the vacuum already in place will often consume anyone involved, sucking the oxygen out of the room, and oxidizing anything new, pulling nutrients and value into the central system. It is what the machine was built to do, and it is hard to reverse the course of the machine, no matter how may holes you poke into the exterior.
This is why so many APIs become extraction engines, rather than delivering resources to applications and users as was intended and promised. It is why Facebook is in the situation they are in. It is why so many API efforts will fail, not deliver as expected, or become harmful to developers and end-users. I watch many well meaning folks try to do APIs, completely unaware of the large machine they are in service of. I don’t want to be in the business of telling people not to do APIs, but in some of the conversations I’m having I feel like I should. APIs are not good by default. Nor are they bad. They are simply a reflection of your organizations existing culture and practices. If things are a mess internally, or your organization is primarily in the value extraction business, then APIs probably aren’t the best option. I think you’ll find they just create more problems for you, for developers, and end-users.
I haven’t published any stories this week on API Evangelist. I’ve been on the road in Lyon and Paris, France. Giving talks, and conducting workshops about my API lifecycle and governance work. Often times when I go on the road I try to pre-populate the blog with stories, but I’ve been so busy lately with travel and projects that I just didn’t have the time. Resulting in the blog not resembling its usual stream of API rants and stories. While this bothers me, I understand the balance between virtual API evangelism and the need to be present in-person from time to time.
While I feel like I can reach more people virtually, I feel like at least 30% of the time I should be present in-person. It helps re-enforce what I know, and allows me to evolve my work, and learn knew things by exercising my API knowledge on the ground in the trenches within existing organizations, and at conferences, Meetups, and workshops. While exhausting, and often costly, in-person gatherings help build relationships, allowing me to establish deeper connections with people. Helping my work penetrate the thick bubbles that exist around us in our personal and professional lives. I’m always exhausted after traveling and shaking so many hands, but ultimately it is worth it if I do it in a thoughtful and logical way.
In-person experiences are valuable, but I still feel that consistent, smart, and a syndicated virtual experience can reach more people, and make a more significant impact. It costs a lot less that traveling and attending conferences too. A robust presence isn’t easy to setup, and takes time to establish, but once in motion it can be the most effective way to build an audience. After eight years of doing API Evangelist, with some of it exclusively operating on the road, I can say that the sustained virtual presence will have the biggest impact, and provide a much more evergreen exposure that will keep on producing results even when you aren’t working. I can take a week off like I just did, and my traffic numbers keep growing, as long as I do not take too much time off, and neglect my work for more than a week or two–even when I took 3 months off a while back, my numbers and presence wasn’t damaged to badly.
In the end, it is all a balance. After a road trip I’m always happy to be home, and feel like I want to say no to other invitations. I feel like I could do so much more if I just had interrupted time at home doing the work I feel matters. While this is true to some degree, I definitely grow and learn a lot by talking with people at companies, organizations, institutions, and government agencies, as well as connecting at the right conferences and Meetups. It is a balance I have to assess from month to month, and have to put serious thought into which events and in-person engagements will make a difference. It is something I don’t think there is a perfect formula for, and is something that evolves, flows, and sometimes dries up depending on the season. It is something I just have to keep trusting my gut, while also forcing myself out of my comfort zone as much as I can possibly handle.
I’m participating in a meeting today where one of the agenda items will be discussing the different ways in which the team can deal with increasingly large API responses. I don’t feel there is a perfect response to this question, and the answer should depend on a variety of considerations, but I wanted to think through some of the possibilities, and make sure the answers were on the tip of my tongue. It helps to exercise these things regularly in my storytelling so when I need to recall them, they are just beneath the surface, ready to bring forward.
Reduce Size Pagination
I’d say the most common approach to send over large amounts of data is to break things down into smaller chunks, based upon rows being sent over, and paginate the responses. Sending over a default amount (ie. 10,25,100), and require consumers either ask for larger amount, as well as request each additional page of results in a separate API request. Providing consumers with a running count of how many pages, what the current page is, and the ability to paginate forward and backward through the pages with each request. It is common to send pagination parameters through the query parameters, but some providers prefer handle it through headers (ie. Github).
Organizing Using Hypermedia
Another approach, which usually augments and extends pagination is using common hypermedia formats for messaging. To paginate results, many API providers use hypermedia media types as a message format, because the media types allow for easy linking to paginate results, as well as providing the relevant parameters in the body of the response. Additionally, hypermedia would further allow you to intelligently break down large responses into different collections, beyond just simple pagination. Then use the linking that is native to hypermedia to provide meaningful links with relations to the different collections of potential responses. Allowing API consumers to obtain all, or just the portions of information they are looking for.
Exactly What They Need With Schema Filtering
Another way to break things down, while putting the control into the API consumers hands, is to allow for schema filtering. Providing parameters and other ways for API consumers to dictate which fields they would like to return with each API response. Reducing or expanding the schema that gets returned based upon which fields are selected by the consumer. There are simpler examples of doing this with a fields parameter, all the way to more holistic approaches using query languages like GraphQL that let you provide a schema of which response you want returned via the URL or the parameter of each API request. Providing a very consumer-centric approach to ensuring only what is needed is transmitted via each API call.
Defining Specific Responses Using The Prefer Header
Another lesson known approach is to use the Prefer Header to allow API consumers to request which representation of a resource they would prefer, based upon some pre-determined definitions. Each API request would pass in a value for the Prefer Header, providing definitions like simple, complete, or other variation of response defined by the API provider. Keeping API responses based upon known schema and scopes defined by the API provider, but still allowing API consumers to select from the type of response they would like to receive. Balancing the scope of API responses between both the needs of API providers, and API consumers.
Using Caching To Make Response More Efficient
Beyond breaking up API calls into different types of responses, we can start focusing on the performance of the API, and making sure HTTP caching is used. Keeping API responses as standardized as possible while leveraging CDN, web server, and HTTP to ensure that each response is being cached as much as it makes sense. Ensuring that an API provider is thinking about, measuring, and responding to how frequent or infrequent data and content is changing. Helping make each API response as fast as it possibly can by leverage the web as a transport.
More Efficiency Through Compression
Beyond caching, HTTP Compression can be used to further reduce the surface area of API responses. DEFLATE and GZIP are the two most common approaches to compression, helping make sure API responses are as efficient as they possibly can. Compressing each API call to make sure only the least amount of bytes are transmitted across the wire.
Breaking Things Down With Chunked Responses
Moving beyond breaking things down, and efficiency approaches using the web, we can start looking at different HTTP standards for making the transport of API responses more efficient. Chunked transfers are one way to send API responses in not just a single API response, but break it down into an appropriate number of chunks, and send them in order. API consumers can make a request and receive large volumes of data in separate chunks that are reassembled on the client side. Leveraging HTTP to make sure large amounts of data is able to be received in smaller, more efficient API calls.
Switch To Providing More Streaming Responses
Beyond chunk responses, streaming using HTTP with standards like Server-Sent Events (SSE) can be used to deliver large volumes of data as streams. Allowing volumes of data and content to be streamed in logical series, with API consumers receiving as individual updates in a continuous, long-running HTTP request. Only receiving the data that has been requested, as well as any incremental updates, changes, or other events that have been subscribed to as part of the establishment of streams. Providing a much more real time approach to making sure large amounts of data can be sent as efficiently as possible to API consumers.
Moving Forward With HTTP/2
Everything we’ve discussed until now leverages the HTTP 1.1 standard, but there are also increased efficiencies available with the HTTP/2 release of the standard. HTTP/2 has delivered a number of improvements on caching and streaming, and handling the threading of of multiple messages within a single API request. Allowing for single, or bi-directional API requests and responses to exist simultaneously. When combined with existing approaches to pagination, hypermedia, and query languages, or using serialization formats like Protocol Buffers, further efficiency gains can be realized, while staying within the HTTP realm, but moving forward to use the latest version of the standard.
This is just a summary look at the different ways to help deliver large amounts of data using APIs. Depending on the conversations I have today I may dive in deeper into all of these approaches and provide more examples of how this can be done. I might do some round ups of other stories and tutorials I’ve curated on these subjects, aggregating the advice of other API leaders. I just wanted to spend a few moments thinking through possibilities so I can facilitate some intelligent conversations around the different approaches out there. While also sharing with my readers, helping them understand what is possible when it comes to making large amounts of data available via APIs.
I wrote about Werner Vogel of Amazon’s post considering the impact of cloud regions a couple weeks back. I feel that his post captured an aspect of doing business in the cloud that isn’t discussed enough, and one that will continue to drive not just the business of APIs, but also increasingly the politics of APIs. Amidst increasing digital nationalism, and growing regulation of not just the pipes, but also platforms, understanding where your APIs are operating, and what networks you are using will become very important to doing business at a global scale.
It is an area I’m adding to my list of machine readable API definitions I’d like to add to the APIs.json stack. The goal with APIs.json is to provide a single index where we can link to all the essential building blocks of a APIs operations, with OpenAPI being the first URI, which provides a machine readable definition of the surface area of the APIs. Shortly after establishing the APIs.json specification, we also created API Commons, which is designed to be a machine readable specification for describing the licensing applied to an API, in response to the Oracle v Google API copyright case. Beyond that, there hasn’t been many other machine readable resources, beyond some existing API driven solutions used as part of API operations like Github and Twitter. There are other API definitions like Postman Collections and API Blueprint that I reference, but they are in the same silo as OpenAPI operates within.
Most of the resources we link to are still human-centered URLs like documentation, pricing, terms of service, support, and other essential building blocks of API operations. However, the goal is to evolve as many of these as possible towards being more machine readable. I’d like to see pricing, terms of services, and aspects of support become machine readable, allowing them to become more automated and understood not just at discovery, but also at runtime. I’m envisioning that regions should be added to this list of currently human readable building blocks that should eventually become machine readable. I feel like we are going to be needing to make runtime decisions regarding API regions, and we will need major cloud providers like Amazon, Azure, and Google to describe their regions in a machine readable way–something that API providers can reflect in their own API definitions, depending on which regions they operate in.
At the application and client level, we are going to need to be able to quantify, articulate, and switch between different regions depending on the user, type of resources being consumed, and business being conducted. While this can continue being manual for a while, at some point we are going to need it to become machine readable so it can become part of the API discovery, as well as integration layers. I do not know what this machine readable schema will look like, but I’m sure it will be defined based upon what AWS, Azure, and Google are already up to. However, it will quickly need to become a standard that is owned by some governing body, and overseen by the community and not just vendors. I just wanted to plant the seed, and it is something I’m hoping will grow over time, but I’m sure it will take 5-10 years before something takes roots, based upon my experience with OpenAPI, APIs.json, and API Commons.
Google has been really moving forward with their development, and storytelling around gRPC. Their high speed to approach to doing APIs that uses HTTP/2 as a transport, and protocol buffers (ProtoBuf) as its serialized message format. Even with all this motion forward they aren’t leaving everyone doing basic web APIs behind, and are actively supporting both approaches across all new Google APIs, as well as in their services and tooling for deploying APIs in the Google Cloud–supporting two-speed APIs side by side, across their platform.
When you are using Google Cloud Endpoints to deploy and manage your APIs, you can choose to offer a more RESTful edition, as well as a more advanced gRPC edition. They’ve continued to support this approach across their service features, and tooling, by now also documenting your APIs. As part of their rollout of a supporting API portal and documentation for your Google Cloud Endpoints, you can automatically document both flavors of your APIs. Making a strong case for considering to offer both types of APIs, depending on the types of use cases you are looking to solve, and the types of developers you are catering to.
In my experience, simpler web APIs are ideal for people just getting going on their API journey, and will accommodate the evolution of 60-75% of the API deployment needs out there. Where some organizations further along in their API journey, and those providing B2B solutions, will potentially need higher performance, higher volume, gRPC APIs. Making what Google is offering with their cloud API infrastructure a pretty compelling option for helping mature API providers shift gears, or even helping folks understand that they’ll be able to shift gears down the road. You get an API deployment and management solution that simultaneously supports both speeds, but also the other supporting features, services, and tooling like documentation delivers at both speeds.
Normally I am pretty skeptical of single provider / community approaches to delivering alternative approaches to APIs. It is one of the reasons I still hold reservations around GraphQL. However with Google and gRPC they have put HTTP/2 to work, and the messaging format is open source. While the approach is definitely all Google, they have embraced the web, which I don’t see out of the Facebook led GraphQL community. I still questions Google’s motives, not because they are up to anything shady, but I’m just skeptical of EVERY company’s motivations when it comes to APIs. After eight years of doing this I don’t trust anyone to not be completely self serving. However, I’ve been tuned into gRPC for some time now and I haven’t seen any signs that make me nervous, and they keep delivering beneficial features like they did with this documentation, keeping me writing stories and showcasing what they are doing.
APIs are hard to talk about across large groups of people, while ensuring everyone is on the page. APIs don’t have much a visual side to them, providing a tangible reference for everyone to use by default. This is where OpenAPI comes in, helping us “see” an API, and establish a human and machine readable document that we can produce, pass around, and use as a reference to what an API does. OpenAPI makes me feel like I have a handle on what an API does, in a way that can actually have a conversation around with other people–without it, things are much fuzzier.
Many folks associate OpenAPI with documentation, code generation, or some other tooling or service that uses the specification–putting their emphasis on the tangible thing, over the definition. While working on projects, I spend a lot of time educating folks about what OpenAPI is, what it is not, and how it can facilitate communication across teams and API stakeholders. While this work can be time consuming, and a little frustrating sometimes, it is worth it. A little education, and OpenAPI adoption can go a long way to moving projects along, because (almost) everyone involved is able to be actively involved in moving API operations forward.
Without OpenAPI it is hard to consistently design API paths, as well as articulate the headers, parameters, status codes, and responses being applied across many APIs, and teams. If I ask, “are we using the sort parameter across APIs?” If there is no OpenAPI, I can’t get an immediate or timely answer, it is something that might not be reliably answered. Making OpenAPI a pretty critical conversation and collaboration driver across the API projects I’m working on. I am not even getting to the part where we are deploying, managing, documenting, or testing APIs. I’m just talking about APIs in general, and making sure everyone involved in a meeting is on the same page when we are talking about one or many APIs.
Almost every API I’m working on starts as a basic OpenAPI, even with just a title and description, published to a Github, Gitlab, Bitbucket, or other repository. Then I usually add schema definitions, which drive conversations about how the schema will be accessed, as we add paths, parameters, and other details of the requests, and responses for each API. With OpenAPI acting as the guide throughout the process, ensuring we are all on the same page, and ensuring all stakeholders, as well as myself have a handle on what is going on with each API being developed. Without OpenAPI, we can never quite be sure we are all talking about the same thing, let alone having a machine readable definition that we can all take back to our corners to get work done.
The Department of Veterans Affairs (VA) is listening to my advice around how to execute their API strategy and adopting a micro-approach to not just delivering services, but also to the business of moving the platform forward at the federal agency. I’ve responded to round one, and round two of the RFI’s, and now they have submitted a handful of work statements on Github, so I wanted to provide an official response, share my thoughts on each of the work statements, and actually bid for the work.
First, Here is The VA Background
The Lighthouse program is moving VA towards an Application Programming Interface (API) first driven digital enterprise, which will establish the next generation open management platform for Veterans and accelerate transformation in VA’s core functions, such as Health, Benefits, Burial and Memorials. This platform will be a system for designing, developing, publishing, operating, monitoring, analyzing, iterating and optimizing VA’s API ecosystem.
Next, What The VA Considers The Play
As the Lighthouse Product Owner, I must have a repeatable process to communicate with internal and external stakeholders the availability of existing, new, and future APIs so that the information can be consumed for the benefit of the Veteran. This outreach/evangelism effort may be different depending on the API type.
Then, What The VA Considers The Deliverable
A defined and repeatable strategy/process to evangelize existing, new, and future APIs to VA’s stakeholders as products. This may be in the form of charts, graphics, narrative, or combination thereof. Ultimately, VA wants the best format to accurately communicate the process/strategy. This strategy/process may be unique to each type of API.
Here Is My Official Response To The Statement
API Evangelism is always something that is more about people, than it is about the technology, and should always be something that speaks to not just developers, being inclusive to all stakeholders involved in, and being served by a platform. Evangelism is all about striking the right balance around storytelling about what is happening across a platform, providing a perfect blend of sales and marketing that expands the reach of the platform, while also properly showcasing the technical value APIs bring to the table.
For the scope of this micro engagement around the VA API platform, I recommend focusing on delivering a handful of initiatives involved with getting the word out about what the API platforms, while also encouraging feedback, but all in an easily measurable way:
- Blogging - No better way to get the word out than an active, informative, and relevant blog with an RSS feed that can be subscribed to.
- Twitter - Augmenting the platform blog with a Twitter account that can help amplify platform storytelling in a way that can directly engage with users.
- Github - Make sure the platform is publishing code, content, and other resources to Github repositories for the platform, and engaging with the Github community around these repos.
- Newsletter - Publishing a weekly newsletter that includes blog posts, other relevant links to things happening on a platform, as well as in the wider community.
- Feedback Loop - Aggregating, responding to, supporting, and organizing email, Twitter, Github, and other feedback from a platform and report back to stakeholders, as well as incorporate into regular storytelling.
For the scope of this project, there really isn’t room to do much else. Daily storytelling, Twitter, and Github engagement, as well as a weekly newsletter would absorb the scope of the project for a 30-60 day period. There would be no completion date for this type of work, as it is something that should go on in perpetuity. However, the number of blog posts, tweets, repos, followers, likes, subscribers, and other metrics could be tracked and reported upon weekly providing a clear definition of what work has been accomplished, and the value from the overall effort over any timeframe.
Evangelism isn’t rocket science. It just takes someone who cares about he platform’s mission, and the developers, and end-users being served by the platform. With a little passion, and technical curiosity, a platform can become alive with activity. A little search engine and social media activity can go a long way towards bringing in new users, keeping existing users engaged and encouraging increased level of activities, internally and externally around platform operations. With simple KPIs in place, and a simple reporting process in operation, the evangelism around a platform can be executed, quantified, and scaled across several individuals, as well as rolling teams of contractors.
Some Closing Thoughts On This Project
That concludes my response to the work statement. Evangelism is something I know. I’ve been studying and doing it for 8 years straight. Simple, consistent, informative evangelism is why I’m in a position to respond to this project out of the VA. It is how many of these ideas have been planted at the VA, through storytelling I’ve been investing in since I worked at the VA in 2013. A single page response doesn’t allow much space to cover all the details, but an active blog, Twitter, Github, and newsletter with a structured feedback loop in place can provide the proper scaffolding needed not to just execute a single cycle of evangelism, but guide many, hopefully rolling contracts to deliver evangelism for the platform in an ongoing fashion. Hopefully bringing fresh ideas, individuals, storytelling and outreach to the platform. Which can significantly amplify awareness around the APIs being exposed by the agency, and helping the platform better deliver on the mission to better serve veterans.
My Response To The VA Microconsulting Work Statement On API Landscape Analysis and Near-term Roadmap17 Apr 2018
The Department of Veterans Affairs (VA) is listening to my advice around how to execute their API strategy and adopting a micro-approach to not just delivering services, but also to the business of moving the platform forward at the federal agency. I’ve responded to round one, and round two of the RFI’s, and now they have submitted a handful of work statements on Github, so I wanted to provide an official response, share my thoughts on each of the work statements, and actually bid for the work.
First, Here is The VA Background
The Lighthouse program is moving VA towards an Application Programming Interface (API) first driven digital enterprise, which will establish the next generation open management platform for Veterans and accelerate transformation in VA’s core functions, such as Health, Benefits, Burial and Memorials. This platform will be a system for designing, developing, publishing, operating, monitoring, analyzing, iterating and optimizing VA’s API ecosystem.
Next, What The VA Considers The Play
As the VA Lighthouse Product Owner I have identified two repositories of APIs and publicly available datasets containing information with varying degrees of usefulness to the public. If there are other publicly facing datasets or APIs that would be useful they can be included. I need an evaluation performed to identify and roadmap the five best candidates for public facing APIs.
The two known sources of data are:
- Data.gov - https://catalog.data.gov/organization/va-gov and www.va.gov/data
- Vets.gov - https://github.com/department-of-veterans-affairs/vets-api
Then, What The VA Considers The Deliverable
Rubric/logical analysis for determining prioritization of APIs Detail surrounding how the top five were prioritized based on the rubric/analysis (Not to exceed 2 pages unless approved by the Government) Deliverables shall be submitted to VA’s GitHub Repo.
Here Is My Official Response To The Statement
To describe my approach to this work statement, and describe what I’ve been already doing in this area for the last five years, I’m going to go with a more logical analysis, as opposed a rubric. There is already too much structured data in all of these efforts, and I’m a big fan of making the process more human. I’ve been working to identify the most valuable data, content, and other resources across the VA, as well as other federal agencies since I worked in DC back in2013, something that has inspired my Knight Foundation funded work with Adopta.Agency.
Always Start With Low Hanging Fruit When it comes to where you should start with understanding what data and content should be turned into APIs, you always begin with the public website for a government agency. If the data and content is already published to the website, it means that there has been demand for accessing the information, and it has already been through a rigorous process to make publicly available. My well developed low hanging fruit process involves spidering of all VA web properties identifying any page containing links to XML, CSV, and other machine readable formats, as well as any page with a table containing over 25 rows, and any forms providing search, filtering, and access to data. The low hanging fruit portion of this will help identify potentially valuable sources of data beyond what is already available at va.gov/data, data.gov, and on Github–expanding the catalog of APIs that we should be targeting for publishing, going well beyond just va.gov/data, data.gov, and Github.
Tap Existing Government Metrics
To help understand what data available at va.gov/data, data.gov, on Github, as well as across all VA web properties, you need to reach out to existing players who are working to track activity across the properties. analytics.usa.gov is a great place to start to understand what people are looking at and searching for across the existing VA web properties. While talking with the folks at the GSA, I would also be talking to them about any available statistics for relevant data on data.gov, understanding what is being downloaded and access when it comes to VA data, but also any other related data from other agencies. Then I’d stop and look at the metrics available on Github for any data stored in with repositories, taking advantage of the metrics built into the social coding platform. Tapping into as many of the existing metrics being tracked on across the VA, as well as other federal agencies.
Talk To Veterans And Their Supporters
Next, I’d spend time on Twitter, Facebook, Github, and Linkedin talking to veterans, veteran service organizations (VSO), and people who advocate for veterans. This work would go a long way towards driving other platform outreach and evangelism described in the other statement of work I responded to. I’d spend a significant amount of time asking veterans and their supports what data, content, and other resources they most use, and would be most valuable if were made more available in web, mobile, and other applications. Social media would provide a great start to help understand what information should be made available as APIs, then you could also spend time reviewing the web and mobile properties of organizations that support veterans, taking note of what statistics, data, and other information they are publishing to help them achieve their mission, and better serve veterans.
Start Doing The Work
Lastly, I’d start doing the work. Based upon what I’ve learned from my Adopta.Agency work, each dataset I identified I’d publish my profiling to an individual Github repository. Publishing a README file describing the data, where it is located, and a quick snapshot of what is needed to turn the dataset into an API. No matter where the data ended up being in the list of priorities there would be a seed planted, which might attract its own audience, stewards, and interested parties that might want to move the conversation forward–internally, or externally to VA operations. EVERY public dataset at the VA should have its own Github repository seed, and then eventually it’s corresponding API to help make sure the latest version of the data is available for integration in applications. There is no reason that the work to deploy APIs cannot be started as part of the identifying and prioritization of data across the VA. Let’s plant the seed to ensure all data at the VA eventually becomes an API, no matter what it’s priority level is this year.
Some Closing Thoughts On This Project
It is tough to understand where to begin with an API effort at any large institution. Trust me, I tried to do at the VA. I was the person who originally setup the VA Github organization, and began taking inventory of the data now available on data.gov, va.gov/data, and on Github. To get the job done you have to do a lot of legwork to identify where all the valuable data is. Then you have to do the hard work of talking to internal data stewards, as well as individuals and organizations across the public landscape who are serving veterans. This is a learning process unto itself, and will go a long way towards helping you understand which datasets truly matter, and should be a priority for API deployment, helping the VA better achieve its mission in serving veterans.
I’m a big fan of building portable micro applications using this approach. The only road block for me to make them easier to use by other users, is the ability to fork them into their own Github account with a single click. I have a Github OAuth token for any authenticated user, and with the right scopes I can easily fork a self-contained web application into their account or organization, giving them full control over their own version of the application. The problem is that each user has to enable Github Pages for the repository before they can view the application. It is a simple, but pretty significant hurdle for many users, and is something that prevents me from developing more applications that work like this.
It has been almost a year and half since I first wrote my post questioning that GraphQL folks didn’t want to do the hard work of API design, which I also clarified that I was keeping my mind open regarding the approach to delivering APIs. I’ve covered several GraphQL implementations since then, as well as my post on waiting the GraphQL assault out-to which I received a stupid amount of, “dude you just don’t get it!”, and “why you gotta be so mean?” responses.
GraphQL Is A Tool In My Toolbox
I’ll start with my official stance on GraphQL. It is a tool in my API toolbox. When evaluating projects, and making recommendations regarding what technology to use, it exists alongside REST, Hypermedia, gRPC, Server-Sent Events (SSE), Websockets, Kafka, and other tools. I’m actively discussing the options with my clients, helping them understand the pros and cons of each approach, and working to help them define their client use cases, and when it makes sense I’m recommending a query language layer like GraphQL. When there are a large number of data points, and a knowledgeable, known group of developers being targeted for building web, mobile, and other client applications, it can make sense. I’m finding GraphQL to be a great option to augment a full stack of web APIs, and in many cases a streaming API, and other event-driven architectural considerations.
GraphQL Does Not Replace REST
Despite what many of the pundits and startups would like to be able to convince everyone of, GraphQL will not replace REST. Sorry, it doesn’t reach as wide of an audience as REST does, and it still keeps APIs in the database, and data literate club. It does make some very complex things simpler, but it also makes some very simple things more complex. I’ve reviewed almost 5 brand new GraphQL APIs lately where the on-boarding time was in the 1-2 hour range, rather than minutes with many other more RESTful APIs. GraphQL augments RESTful approaches, and in some cases it can replace them, but it will not entirely do away with the simplicity of REST. Sorry, I know you don’t want to understand the benefits REST brings to the table, and desperately want there to be a one size fits all solution, but it just isn’t the reality on the ground. Insisting that GraphQL will replace REST does everyone more harm than good, hurts the overall API community, and will impede GraphQL adoption–regardless of what you want to believe.
The GraphQL Should Embrace The Web
One of my biggest concerns around the future of GraphQL is its lack of a focus on the web. If you want to see the adoption you envision, it has to be bigger than Facebook, Apollo, and the handful of cornerstone implementations like Github, Pinterest, and others. If you want to convert others into being believers, then propose the query language to become part of the web, and improve web caching so that it rises to the level of the promises being made about GraphQL. I know underneath the Facebook umbrella of GraphQL and React it seems like everything revolves around you, but there really is a bigger world out there. There are more mountains to conquer, and Facebook won’t be around forever. To many of the folks I’m trying to educate about GraphQL, they can’t help but see the shadow of Facebook, and their lack of respect for the web. GraphQL may move fast, and break some things, but it won’t have the longevity all y’all believe so passionately in, if you don’t change your tune. I’ve been doing this 30 years now, and seen a lot of technology come and go. My recommendation is to embrace the web, bake GraphQL into what is already happening, and we’ll all be better off for it.
GraphQL Does Not Do What OpenAPI Does
I hear a lot of pushback on my OpenAPI work, telling me that GraphQL does everything that OpenAPI does, and more! No, no it doesn’t. By saying that you are declaring that you don’t have a clue what OpenAPI does, and that you are just pushing a trend, and have very little interest in the viability of the APIs I’m building, or my client’s needs. There is an overlap in what GraphQL delivers, and what OpenAPI does, but they both have their advantages, and honestly OpenAPI has a lot more tooling, adoption, and enjoys a diverse community of support. I know in your bubble that GraphQL and React dominates the landscape, but on my landscape it is just a blip, and there are other tools in the toolbox to consider. Along with the web, the GraphQL should be embracing the OpenAPI community, understanding the overlap, and reaching out to the OAI to help define the GraphQL layer for the specification. Both communities would benefit from this work, rather than trying to replace something you don’t actually understand.
GraphQL Dogma Continues To Get In The Way
I am writing this post because I had another GraphQL believer mess up my chances for it being in the roadmap, because of their overzealous beliefs, and allowing dogma to drive their contribution to the conversation. This is the 3rd time I’ve had this happen on high profile projects, and while the GraphQL conversation hasn’t been ruled out, it has been slowed, pushed back, and taken off the table, due to pushing the topic in unnecessary ways. The conversation unfortunately wasn’t about why GraphQL was a good option, the conversations was dominated by GraphQL being THE SOLUTION, and how it was going to fix everything web service and API that came before it. This combined with inadequate questions regarding IP concerns conveyed, and GraphQL being a “Facebook solution”, set back the conversations. In each of these situations I stepped in to help regulate the conversations, and answer questions, but the damage was already done, and leadership was turned off to the concept of GraphQL because of overzealous, dogmatic beliefs in this relevant technology. Which brings me back to why I’m pushing back on GraphQL, not because I do not get the value it brings, but because the rhetoric around how it is being sold and pushed.
No, I Do Get What GraphQL Does
This post will result in the usual number of Tweets, DMs, and emails telling me I just don’t get GraphQL. I do. I’ve installed and played with it, and hacked against several implementations. I get it. It makes sense. I see why folks feel like it is useful. The database guy in me gets why it is a viable solution. It has a place in the API toolbox. However, the rhetoric around it is preventing me from putting it to use in some relevant projects. You don’t need to convince me of its usefulness. I see it. I’m also not interested in convincing y’all of the merits of REST, Hypermedia, gRPC, or the other tools in the toolbox. I’m interested in applying the tools as part of my work, and the tone around GraphQL over the last year or more hasn’t been helping the cause. So, please don’t tell me I don’t get what GraphQL does, I do. I don’t think y’all get the big picture of the API space, and how to help ensure GraphQL is in it for the long haul, and achieving the big picture adoption y’all envision.
Let’s Tone Down The Rhetoric
I’m writing about this because the GraphQL rhetoric got in my way recently. I’m still waiting out the GraphQL assault. I’m not looking to get on the bandwagon, or argue with the GraphQL cult, I’m just looking to deliver on the projects I’m working on. I don’t need to be schooled on the benefits of GraphQL, and what the future will hold. I’m just looking to get business done, and support the needs of my clients. I know that the whole aggressive trends stuff works in Silicon Valley, and the startup community. But in government, and some of the financial, insurance, and healthcare spaces I’ve been putting GraphQL on the table, the aggressive rhetoric isn’t working. It is working against the adoption of the query language. Honestly it isn’t just the rhetoric, I don’t feel the community is doing enough to satisfy the wider IP, and Facebook connection to help make the sale. Anyways, I’m just looking to vent, and push back on the rhetoric aound GraphQL replacing REST. After a year and a half I’m convinced of the utility of GraphQL, however the wider tooling, and messaging around it still hasn’t won me over.
|<< Prev||Next >>|