The API Evangelist Blog
This blog represents the thoughts I have while I'm research the world of APIs. I share what I'm working each week, and publish daily insights on a wide range of topics from design to depcration, and spanning the technology, business, and politics of APIs. All of this runs on Github, so if you see a mistake, you can either fix by submitting a pull request, or let me know by submitting a Github issue for the repository.
I am working my way through a variety of API design considerations for the Human Services Data API (HSDA)that I’m working on with Open Referral. I was working through my thoughts on how I wanted to approach the filtering of the underlying data schema of the API, and Shelby Switzer (@switzerly) suggested I follow Irakli Nadareishvili’s advice and consider using RFC 7240 -the Prefer Header for HTTP, instead of some of the commonly seen approaches to filtering which fields are returned in an API response.
I find this approach to be of interest for this Human Services Data API implementation because I want to lean on API design, over providing parameters for consumers to dial in the query they are looking for. While I’m not opposed to going down the route of providing a more parameter based approach to defining API responses, in the beginning I want to carefully craft endpoints for specific use cases, and I think the usage of the HTTP Prefer Header helps extend this to the schema, allowing me to craft simple, full, or specialized representations of the schema for a variety of potential use cases. (ie. mobile, voice, bot)
It adds a new dimension to API design for me. Since I’ve been using OpenAPI I’ve gotten better at considering the schema alongside the surface area of the APIs I design, showing how it is used in the request and response structure of my APIs. I like the idea of providing tailored schema in responses over allowing consumers to dynamically filter the schema that is returned using request parameters. At some point, I can see embracing a GraphQL approach to this, but I don’t think that human service data stewards will always know what they want, and we need to invest in a thoughtful set design patterns that reflect exactly the schema they will need.
Early on in this effort, I like allowing API consumers to request minimal, standard or full schema for human service organizations, locations, and services, using the Prefer header, over adding another set of parameters that filter the fields–it reduces the cognitive load for them in the beginning. Before I introduce any more parameters to the surface area, I want to better understand some of the other aspects of taxonomy and metadata proposed as part of HSDS. At this point, I’m just learning about the Prefer header, and suggesting it as a possible solution for allowing human services API consumers to have more control over the schema that is returned to them, without too much overhead.
I am going through my API design checklist for the Human Services Data API work I am doing. I’m trying to make sure I’m not forgetting anything before I propose a v1.1 OpenAPI draft, so I pulled together a simple checklist I wanted to share with other stakeholders, and hopefully also help keep me focused.
First, to support my API design work I got to work on these areas for defining the HSDS schema and the HSDA definition:
- JSON Schema - I generated a JSON Schema from the HSDS documentation.
- OpenAPI - I crafted an OpenAPI for the API, generating GET, POST, PUT, and DELETE methods for 100% of the schema, and reflective its use in the API request and response.
- Github Repo - I published it all in a Github repository for sharing with stakeholders, and programmatic usage across any tooling and applications being developed.
Then I reviewed the core elements of my API design to make sure I had everything I wanted to cover in this cycle, with the resources we have:
- Domain(s) - Right now I’m going with api.example.com, and developer.example.com for the portal.
- Versioning - I know many of my friends are gonna give me grief, but I’m putting versioning in the URL, keeping things front and center, and in alignment with the versioning of the schema.
- Paths - Really not much to consider here as the paths are derived from the schema definitions, providing a pretty simple, and intuitive design for paths–will continue adding guidance for future APIs.
- Verbs - A major part of this release was making sure 100% of the surface area of the HSDS schema add the ability to POST, PUT, and DELETE, as well as just GET a response. I’m not addressing PATCH in this cycle, but it is on the roadmap.
- Parameters - There are only a handful of query parameters present in the primary paths (organizations, locations, services), and a robust set for use in /search. Other than that, everything is mostly defined through path parameters, keeping things cleanly separated between path and query.
- Headers - I’m only using headers for authentication. I’m also considering using the HTTP Prefer Header for schema filtering, but nothing else currently.
- Actions - Nothing to do here either, as the API is pretty CRUD at this point, and I’m awaiting more community feedback before I add any more detailed actions beyond what is possible with the default verbs–when relevant I will add guidance to this area of the design.
- Body - All POST and PUT methods use the body for request transport. There are no other uses of the body across the current design.
- Pagination - I am just going with what is currently in place as part of v1.0 for the API, which uses page and per_page for handling this.
- Data Filtering - The parameters for core resources (organizations, locations, and services all have a query parameter for filtering data, and the search path has a set of parameters for filtering data returned in response. Not adding anything new for this version.
- Schema Filtering - I am taking Irakli Nadareishvili’s advice and going to go with RFC 7240 - Prefer Header for HTTP, and craft some different representations when it comes to filtering the schema is returned.
- Sorting - There is no sorting currently. I did some research in this area, but not going to make any recommendations until I hear more requests from consumers, and the community.
- Operation ID - I went with camelCase for all API operation IDs, providing a unique reference to be included in the OpenaPI.
- Requirements - Going through and making sure all the required fields are reflected in the definitions for the OpenAPI.
- Status Codes - Currently I’m going to just reflect the 200 HTTP status code. I don’t want to overwhelm folks with this release and I would like to accumulate more resources so I can invest in a proper HTTP status code strategy.
- Error Responses - Along with the status code work I will define a core set of definitions to be used across a variety of responses and HTTP statuses.
- Media Types - While not a requirement, I would like to encourage implementors to offer four default media types: application/json, application/xml, text/csv, and text/html.
After being down in the weeds I wanted to step back and just think about some of the common sense aspects of API design:
- Granularity - I think the API provides a very granular approach to getting at the HSDS schema. If I just want a phone number for a location, and I know its location id I can get it. It’s CRUD, but it’s a very modular CRUD that reflects the schema.
- Simplicity - I worked hard to keep things as simple as possible, and not running away with adding dimensions to the data, or adding on the complexity of the taxonomy that will come with future iterations and some of the more operational level APIs that are present in the current metadata portion of the schema.
- Readability - While lengthy, the design of the API is readable and scannable. Maybe I’m biased, but I think the documentation flows, and anyone can read and get an idea of the possibilities with the human *services API.
- Relationships - There really isn’t much sophistication in the relationships present in the HSDA. Organizations, locations, and services are related, but you need to construct your own paths to navigate these relationships. I intentionally kept the schema flat, as this is a minor release. Hypermedia and other design patterns are being considered for future releases–this is meant to be a basic API to get at the entire HSDS schema.
I have a functioning demo of this v1.1 release, reflecting most of the design decisions I listed above. While not a complete API design list, it provides me with a simple checklist to apply to this iteration of the Human Services Data API (HSDA). Since this design is more specification than actual implementation, this API design checklist can also act as guidance for vendors and practitioners when designing their own APIs beyond the HSDS schema.
Next, I’m going to tackle some of the API management, portal, and other aspects of operating a human services API. I’m looking to push my prototype to be a living blueprint for providers to go from schema and OpenAPI to a fully functioning API with monitoring, documentation, and everything else you will need. The schema and OpenAPI are just the seeds to be used at every step of a human services API life cycle.
When I suggest modern approaches to API management be applied to public data I always get a few open data folks who push back saying that public data shouldn’t be locked up, and needs to always be publicly available–as the open data gods intended. I get it, and I agree that public data should be easily accessible, but there are increasingly a number of unintended consequences that data stewards need to consider before they publish public data to the web in 2017.
I’m going through this exercise with my recommendations and guidance for municipal 211 operators when it comes to implementing Open Referral’s Human Services Data API (HSDA). The schema and API definition centers around the storage and access to organizations, locations, services, contacts, and other key data for human services offered in any city–things like mental health resources, suicide assistance, food banks, and other things we humans need on a day to day basis.
This data should be publicly available, and easy to access. We want people to find the resources they need at the local level–this is the mission. However, once you get to know the data, you start understanding the importance of not everything being 100% public by default. When you come across listings Muslim faith, and LGBTQ services, or possibly domestic violence shelters, and needle exchanges. They are numerous types of listings where we need to be having sincere discussions around security and privacy concerns, and possibly think twice about publishing all or part of a dataset.
This is where modern approaches to API management can lend a hand. Where we can design specific endpoints, that pull specialized information for specific groups of people, and define who has access through API rate limiting. Right now my HSDA implementation has two access groups, public and private. Every GET path is publicly available, and if you want to POST, PUT, or DELETE data you will need an API key. As I consider my API management guidance for implementors, I’m adding a healthy dose of the how and why of privacy and security using existing API management solutions and practice.
I am not interested in playing decider when it comes to what data is public, private, and requires approval before getting access. I’m merely thinking about how API management can be applied in the name of privacy and security when it comes to public data, and how I can put tools in the hands of data stewards, and API providers that help them make the decision about what is public, private, and more tightly controlled. The trick with all of this is how transparent should providers be with the limits and restrictions imposed, and communicate the offical stance with all stakeholders appropriately when it comes to public data privacy and security.
I am stepping back to today and thinking about a pretty long list of API design considerations for the Human Services Data API (HSDA), providing guidance for municipal 211 who are implementing an API. I’m making simple API design decisions from how I define query parameters all the way to hypermedia decisions for the version 2.0 of the HSDA API.
There are a ton of things I want to do with this API design. I really want folks involved with municipal 211 operations to be adopting it, helping ensure their operations are interoperable, and I can help incentivize developers to build some interesting applications. As I think through the laundry list of things I want, I keep coming back to my audience of practitioners, you know the people on the ground with 211 operations that I want to adopt an API way of doing things.
My target audience isn’t steeped in API. They are regular folks trying to get things done on a daily basis. This move from v1.0 to v.1 is incremental. It is not about anything big. Primarily this move was to make sure the API reflected 100% of the surface area of the Human Services Data Specification (HSDS), keeping in sync with the schema’s move from v1.0 to v1.1, and not much more. I need to onboard folks with the concept of HSDS, and what access looks like using the API–I do not have much more bandwidth to do much else.
I want to avoid moving too fast for my API audience. I can see version 2,3, even 4 in my head as the API architect and designer, but am I think of me, or my potential consumers? I’m going to seize this opportunity to educate my target audience about APIs using the road map for the API specification. I have a huge amount of education of 211 operators, as well as the existing vendors who sell them software when it comes to APIs. This stepping back has pulled the reigns in on my API design strategy, encouraging me to keep things as simple as possible for this iteration, and helping me think more thoughtfully about what I will be releasing as part of each incremental, as well as major releases the future will hold.
I get regular waves of folks who chime in anytime I push on one of the hot-button topics on my site like hypermedia and OpenAPI. I have a couple of messages in my inbox regarding some recent stories I’ve done about OpenAPI recently, and how it isn’t sustainable, and we should be putting hypermedia practices to work. I’m still working on my responses, but I wanted to think through some of my thoughts here on the blog before I respond–I like to simmer on these things, releasing the emotional exhaust before I respond.
When it comes to the arguments from the hypermedia folks, the short answer is that I agree. I think many of the APIs I’m seeing designed using OpenAPI would benefit from some hypermedia patterns. However, there is such a big gap between where people are, and where we need to be for hypermedia to become the default, and getting people there is easier said than done. I view OpenAPI as a scaffolding or bridge to transport API designers, developers, and architects at scale from where we are, to where we need to be–at scale.
I wish I could snap my fingers and everyone understood how the web works and understood the pros and cons of each of the leading hypermedia types. Many developers do not even understand how the web works. Hell, I’m still learning new things every day, and I’ve been doing this full time for seven years. Most developers still do not even properly include and use HTTP status codes in their simple API designs, let alone understand the intricate relationship possibilities between their resources, and the clients that will be consuming them. Think about it, as developer, I don’t even have time, budget or care to articulate the details of why a response failed, and you expect that I have the time, budget, are care about link relations, and the evolution of the clients build on top of my APIs? Really?
I get it. You want everyone to see the world like you do. You are seeing us headed down a bad road, and want to do something about it. So do I. I’m doing something about it full time, telling stories about hypermedia APIs that exist out there, and even crafting my own hypermedia implementations, and telling the story around them. What are you doing to help equip folks, and educate them about best practices? I think once you hit the road doing this type of storytelling in a sustained way, you will see things differently, and begin to see how much investment we need in basic web literacy–something the OpenAPI spec is helping break down for folks, and providing them with a bridge to incrementally get from where they are at, to where they need to be.
Folks need to learn about consistent design patterns for their requests and responses. Think about getting their schema house in order. Think through content negotiation in a consistent way. Be able to see plainly that they only use three HTTP status codes 200, 404, and 500. OpenAPI provides a scaffolding for API developers to begin thinking about design, learn how the web works, and begin to share and communicate around what they’ve learned using a common language. It also allows them to learn from other API designers who are further along in their journey or just have had success with a single API pattern.
For you (the hypermedia guru), OpenAPI is redundant, meaningless, and unnecessary. For many others, it provides them with a healthy blueprint to follow and allows you to be spoon fed web literacy, and good API design practices in the increments your daily job, and current budget allows. Few API practitioners have the luxury of immersing them in deep thought about the architectural styles and the design of network-based software architectures or wading through W3C standards. They just need to get their job done, and the API deployed, with very little time for much else.
It is up to us leaders in the space to help provide them with the knowledge they’ll need. Not just in book format. Not everyone learns this way. We need to also point out the existing healthy patterns in use already. We need to invest in more blueprints, prototypes, and working examples of good API design. I think my default response for folks who tell me that OpenAPI isn’t a solution will be to ask them to send me a link to all the working examples and stories of the future they envision. I think until you’ve paid your dues in this area, you won’t see how hard it is to educate the masses and actually get folks headed in the right direction at scale.
I often get frustrated with the speed at which things move when I’m purely looking at things from what I want and what I know and understand. However, when I step back and think about what it will truly take to get the masses of developers and business users on-boarded with the API economy–the OpenAPI specification makes a lot more sense as a scaffolding for helping us get where we need. Oh, and also I watch Darrel Miller (@darrel_miller) push on the specification on a daily basis, and I know things are going to be ok.
I wrote earlier this year that I would like us all to focus more on our schema and definitions of our data we use across API operations. Since then I’ve been keeping an eye out for any other interesting signs in this area like Postman with their data editor, and now I’ve come across the Streams Manager for inspecting the data schema of your event collections in Keen IO..
With Streams Manager you can:
- Inspect and review the data schema for each of your event collections
- Review the last 10 events for each of your event collections
- Delete event collections that are no longer needed
- Inspect the trends across your combined data streams over the last 30-day period
Keen IO provides us with an interesting approach to getting in tune with the schema across your event collections. I’d like to see more of this across the API lifecycle. I understand that companies like Runscope, Stoplight, Postman, and others already let us peek inside of each API call, which includes a look at the schema in play. This is good, but I’d like to see more schema management solutions at this layer helping API providers from design to deprecation.
In 2017 we all have an insane amount of bits and bytes flowing around us in our daily business operations. APIs are only enabling this to grow, opening up access to our bits to our partners and on the open web. We need more solutions like Keen’s Stream Manager, but for every layer of the API stack, allowing us to get our schema house in order, and make sense of the growing data bits we are producing, managing, and sharing.
Box has gone all in on OpenAPI. They have published an OpenAPI for their document and storage API on Github, where it can be used in a variety of tools and services, as well as be maintained as part of the Box platform operations. Adding to the number of high-profile APIs managing their OpenAPI definitions on Github, like Box, and the NY Times.
As part of their OpenaPI release, Box published a blog post that touches on all the major benefits of having an OpenAPI, like forking on Github for integration into your workflow, generating documentation and visualizations, code, mock APIs, and even monitoring and testing using Runscope. It’s good to see a major API provider drinking the OpenAPI Kool-Aid, and working to reduce friction for their developers.
I would love to not be in the business of crafting complete OpenAPIs for API providers. I would like every single API provider to be maintaining their own OpenAPI on Github like Box and [NY Times(http://apievangelist.com/2017/03/01/new-york-times-manages-their-openapi-using-github/) does. Then I (we) could spend time indexing, curating, and developing interesting tooling and visualizations to help us tell stories around APIs, helping developers and business owners understand what is possible with the growing number of APIs available today.
One layer of my API research is dedicated to keeping track on what is going on with API software development kits (SDK). I have been looking at trends in SDK evolution as part of continuous integration and deployment, increased analytics at the SDK layer, and SDKs getting more specialized in the last year. This is a conversation that Google is continuing to move forward by focusing on enhanced storage, logging, and analytics at the SDK level.
Google provides a nice example of how API providers are increasing the number of available resources at the SDK layer, beyond just handling API requests and responses, and authentication. I’ll try to carve out some time to paint a bigger picture of what Google is up to with SDKs. All their experience developing and supporting SDKs across hundreds of public APIs seems to be coming to a head with the Google Cloud SDK effort(s).
I’m optimistic about the changes at the SDK level, so much I’m even advising APIMATIC on their strategy, but I’m also nervous about some of the negative consequences that will come with expanding this layer of API integration–things like latency, surveillance, privacy, and security are top of mind. I’ll keep monitoring what is going on, and do deeper dives into SDK approaches when I have the time and $money$, and if nothing else I will just craft regular stories about how SDKs are evolving.
Disclosure: I am an advisor of APIMATIC.
I was just messing around with a friend online about monitoring of our monitoring tools, where I said that I have a monitor setup to monitor whether or not I care about monitoring. I was half joking, but in reality, giving a shit is actually a pretty critical component of monitoring when you think about it. Nobody monitors something they don’t care about. While monitoring in the world of APIsn might mean a variety of things, I’m guessing that caring about those resources is a piece of every single monitoring configuration.
This has come up before in conversation with my friend Dave O’Neill of APIMetrics, where he tells stories of developers signing up for their service, running the reports they need to satisfy management or customers, then they turn off the service. I think this type of behavior exists at all levels, with many reasons why someone truly doesn’t care about a service actually performing as promised, and doing what it takes to rise to the occasion–resulting in the instability, and unreliability that APIs that gets touted in the tech blogosphere.
There are many reasons management or developers will not truly care when it comes to monitoring the availability, reliability, and security of an API. Demonstrating yet another aspect of the API space that is more business and politics, than it is ever technical. We are seeing this play out online with the flakiness of websites, applications, devices, and the networks we depend on daily, and the waves of breaches, vulnerabilities, and general cyber(in)security. This is a human problem, not a technical, but there are many services and tools that can help mitigate people not caring.
I was reading about Google’s TensorFlowLite in Techcrunch, and their mention of Facebook’s Caffe2Go, and I was reminded of a conversation I was having with the Oxford Dictionaries API team a couple months ago.
The OED and other dictionary and language content API teams wanted to learn more about on-device API deployment, so their dictionaries could become the default. I have asked when we will have containers natively on our routers a while ago, but I’d also like to add to that request–when will we have a stack of containers on device where we can deploy API resources that can be used by applications, and augment the existing on-device hardware and OS APIs?
API providers should be able to deploy their APIs exactly here they are needed. API deployment, management, monitoring, logging, and analytics should exist by default in these micro-containerized environments on any devices. Whether it’s our mobile phones, our automobiles, or the weather, solar, or other industrial device integration, we are going to new API-driven data, ML, AI, augmented, and other resources on-device, in a localized environment.
I am in the process of eliminating the MySQL backend behind much of my research, eliminating a business expense, as well as an unnecessary complexity in my architecture. There really is no reason for the data I use in my business to be in a database. Nothing I track on tends to go beyond 10K rows, with most of the tables actually being less than 100 rows–perfect for spreadsheets, and my new static approach to delivering APIs, and websites for my research.
The time had come to update some of the products on my website, and I thought my product page was a perfect candidate for this approach, providing me with the following elements:
This simple approach to publishing static APIs using Google Sheets and Github is working well for little data like this–I am all about the little data, while everyone else is excited about big data. ;-) I even have the beginning of some documentation and an updated APIs.json for my website.
Next, I’ll work through the rest of my projects, organizations, tools, and other data I track on as part of my API research. I’ll be publishing a complete snapshot of this data at API Evangelist, as well as subsets of it at each of the individual research projects. When I’m done I’ll have a nice static stack of APIs for all of my research, easily managed via Google Sheets, and YAML on Github.
One of my clients asked me for fifteen bullet points of what I’d say to help convince folks at his company that APIs are the future, and have potentially viable business models. While helping convince people of the market value of APIs is not really my game anymore, I’m still interested in putting on my business of APIs hat, and playing this game to see what I can brainstorm to convince folks to be more open with their APIs.
Here are the fifteen stories from the API space that I would share with folks to help them understand the potential.
- Web - Remember asking about the viability about the web? That was barely 20 years ago. APIs are just the next iteration of the web, and instead of just delivering HTML to humans for viewing in the browser, it is about sharing machine-readable versions for use in mobile, devices, and other types of applications.
- Cloud - The secret to Amazon’s success has been APIs. Their ability to disrupt retail commerce, and impact almost every other business sector with the cloud was API-driven.
- Mobile - APIs are how data, content, and algorithms are delivered to mobile devices, as well as provides developers with access to other device capabilities like the camera, GPS, and other essential aspects of our ubiquitous mobile devices.
- SalesForce - SalesForce has disrupted the CRM and sales market with it’s API-driven approach to software since 2001, generating 50% of its revenue via APIs.
- Twitter - Well, maybe Twitter is not the poster child of revenue, but I do think they provide an example of how an API can create a viable ecosystem where there is money to be made building businesses. There are numerous successful startups born out of the Twitter API ecosystem, and Twitter itself is a great example of what is possible with APIs–both good and bad.
- Twilio - Twilio is the poster child for how you do APIs right, build a proper business, then go public. Twilio has transformed messaging with their voice and SMS API solutions and provides a solid blueprint for building a viable business using APIs.
- Netflix - While the Netflix public API was widely seen as a failure, APIs have long driven internal and partner growth at Netflix, and has enabled the company to scale beyond the data center, into the cloud, then dominate and transform an industry at a global level.
- Apigee - The poster child for an API sector IPO, and then acquisition by Google–demonstrating the value of API management to the leading tech companies, as they position themselves for ongoing battle in the clouds.
- Automobiles - Most of the top automobile manufacturers have publicly available API programs and are actively courting other leading API providers like Facebook, Pandora, Youtube, and others–acknowledging the role APIs will play in the automobile experience of tomorrow.
- iPaaS - Integration platform as service providers like IFTTT and Zapier put APIs in the hands of the average business users, allowing them to orchestrate their presence and operations across the growing number of platforms we depend on each day.
- Big Data - Like it or not, data is often seen as the new oil, elevating the value of data, and increasing investment into acquiring this valuable data. APIs are the pipes for everything big data, analysis, visualization, prediction, and everything you need to quantify markets using data.
- Investments - Investment in API-focused companies have not slowed in a decade. VC’s are waking up to the competitive advantage of having APIs, and their portfolios are continuing to reflect the value APIs bring to the table.
- Acquisitions - Google buying Apigee, Red Hat buying 3Scale, Oracle buying Apiary, are just a few of the recent high profile API related acquisitions. Successful API startups are a valuable acquisition target, continuing to show the viability of building API businesses.
- Regulations - We are beginning to see the government move beyond just open data and getting into the API regulations and policy arena, with banking and PSD2 in Europe, FHIR in healthcare from Health and Human Services in the U.S., and other movements within a variety of sectors. As the value and importance of APIs grows, the need for government to step in and provide guidance for existing industries like taxicabs with Uber, or newer media outlets like Facebook or Twitter.
- Elections - As we are seeing in the UK, US, EU, and beyond, elections are being driven by social media, primarily Twitter, and Facebook, which are API-driven. This is all being fueled by advertising, which is primarily API driven via Facebook, Twitter, and Google.
If I was sitting in a room full of executives, these are the fifteen points I’d bring up to drive a conversation around the business viability of doing APIs. There are many other stories I’d include, but I would want to work from a variety of perspectives that would speak to leadership across a potentially wide variety of business sectors.
Honestly, APIs really don’t excite me at VC scale, but I get it. Personally, I think Amazon, Google, Azure, and a handful of API rock stars are going to dominate the conversation moving forward. However, there will still be a lot of opportunities in top performing business sectors, as well as in the long tail for the rest of us little guys. Hopefully, in the cracks, we can still innovate, while the tech giants partner with other industry giants and continue to gobble up API startups along the way.
I’m profiling a number of drone APIs lately and I came across some interesting APIs out of Parrot. Not all of the APIs are for drones, but I thought they were clean and simple examples of what IoT APIs can look like.
The API for the Parrot Sequoia camera can be controlled over USB, WIFI, allowing you to change settings, calibrate the sensors, trigger image capture and manage memory, and files.
Here are the paths for the device:
- /capture: to get the Sequoia capture state, start and stop a capture
- /config: to get and set the configuration of the camera
- /status: to get all information about the Sequoia physical state
- /calibration: to get the calibration status, start and stop a calibration
- /storage: to get informations about memory
- /file: to get files and folders information
- /download: to download files
- /delete: to delete files and folders
- /version: to get serial number and software version
- /wifi: to get the Sequoia SSID
- /manualmode: to get and set ISO and exposure manually
- /websocket: to use WebSocket notifications on asynchronous events
I like the simple use of API design to express what is possible with an IoT device and that a small hand-held deployable camera and sensor can be defined in this way. While you still need some coding skills to bring any integration life, anyone could land on the API page and pretty quickly understand what is possible with the device API.
When I come across simple approaches to IoT devices using web APIs I try to write about them, adding them to my research when it comes to IoT APIs. It gives me an easy way to find it again in the future, but also hopefully provides IoT manufacturers some examples of how you can do this as simply and effectively as you can.
I need as many examples of how APIs can be in the cloud, or even on a device like the Parrot Sequoia API.
There are some pretty proven API solutions out there these days. I had to explain to someone a call the other day that in 2017 you shouldn’t ever roll your own API signup, registration, rate limiting, reporting, logging, and other API management features–there are too many proven API management solutions on the market these days (cough, 3Scale, Restlet, DreamFactory, or Tyk)
As a penny-pinching small business owner who is also a programmer, I am always struggling with the question of whether I should be buying or building. However, when it comes to some of the more proven, well-laid API sectors–I know better. One of these areas I will never develop my own tooling is when it comes to analytics. There are just too many quality analytics solutions available out there who are API-driven.
One of these is Keen.io, who describe themselves as “APIs for capturing, analyzing, and embedding event data in everything you build”, provided an example of this in action, in their unstoppable API era post:
One of Keen’s largest customers stealthily uses Keen’s APIs to white label an in-app analytics suite for their Fortune 500 customers. Their delivery manager probably made the point most succinctly: “It would have taken our team two years to deliver what we’ve done with Keen in nine weeks.”
As a developer, we like to think we can doing anything on our own, and we often also underestimate the time it will take to accomplish something. I think I’m going to go through the high grade, well-established resources available in my API Stack research, and see which areas I feel contain solid API providers, and I would think twice about reinventing the wheel in.
Something like analytics always seems like it is easy to do at first, but then once you are down the rabbit hole trying to do at scale, and things begin to change. I prefer to not get distracted by the nuts and bolts of things like analytics, and be allowed to focus on what I’m good at. The best part of APIs is that you get to look good doing what you do, but it is something that can also be enhanced and augmented by other experts like Keen.io who are rocking what they do.
A recent addition to my API research is the concept of making batch API requests. I was reminded of this during a webinar I did with Cloud Elements when they cited batch API requests as an area needing improvement in their State of API Integration report. I had also recently come across several batch APIs while profiling the Google API stack, so I already had the topic in my notebook, but Cloud Element pushed me to add the topic to my research.
Here are a handful of batch API implementations I am working through, to better understand how providers are approaching the problem:
As I do, in my approach to API research, I will process the common patterns I come across in each of these implementations, then add as building blocks in my API design research, hopefully providing some details API providers can consider early on in the API lifecycle. I’m not looking to tell people how to deliver batch APIs–I am just looking to shine a light on how the successful APIs are already doing it.
I feel like batch APIs are a response to more APIs, and less direct database access or full data download availability. While modular, simple APIs that do one thing well works in many situations, sometimes you need to move large amounts of resources around and make API requests that do more than just update a single resource or database record. I’ll file this research under API design, but I’ll migrate make a mention of it at the database and other levels, as I identify the variances in how bulk API requests are being made and the solutions they are providing.
We are inching closer to a final release of version 3.0 for the OpenAPI specification, with the official version currently set at 3.0.0-rc1. We are beginning to see tooling emerge, and services like APIMATIC are already supporting version 3.0 when it comes to SDK generation, as well their API Transformer conversion tool.
I am working on an OpenAPI validation solutions tailored specifically for municipal API deployments and was working with the JSON Schema for version 2.0 of the API specification. I wanted to help make my work be as ready for the future of the API specification and wanted to see if there was a JSON Schema for version 3.0 of the OpenAPI specification. I couldn’t find anything in the new branch of the repository, so I set out seeing if anyone else has been working on it.
While I was searching I saw Tim Burks share that he and Mike Ralphson were working on one over at the Google Gnostic project. It looks like it is still a work in progress, but it provides us with a starting point to work from.
I will keep an eye out on the OpenAPI repo for when they add an official JSON Schema for version 3.0. As I’ve learned more about JSON Schema, the more I’m learning about how it helps harmonize and stabilizing tooling being developed around any schema. Without it, you get some pretty unreliable results, but with it, you can achieve some pretty scalable consistency across API tooling and services–something the API space needs as much as it can get in 2017.
I was having a discussion with an investor today about the potential of algorithmic-centered API marketplaces. I’m not talking about API marketplaces like Mashape, I’m more talking about ML API marketplaces like Algorithmia. This conversation spans multiple areas of my API lifecycle research, so I wanted to explore my thoughts on the subject some more.
I really do not get excited about API marketplaces when you think just about API discovery–how do I find an API? We need solutions in this area, but I feel good implementations will immediately move from useful to commodity, with companies like Amazon already pushing this towards a reality.
There are a handful of key factors for determining who ultimately wins the API Machine Learning (ML) marketplace game:
- Always Modular - Everything has to be decoupled and deliver micro value. Vendors will be tempted to build in dependency and emphasize relationships and partnerships, but the smaller and more modular will always win out.
- Easy Multi-Cloud - Whatever is available in a marketplace has to be available on all major platforms. Even if the marketplace is AWS, each unit of compute has to be transferrable to Google or Azure cloud without ANY friction.
- Enterprise Ready - The biggest failure of API marketplaces has always been being public. On-premise and private cloud API ML marketplaces will always be more successful that their public counterparts. The marketplace that caters to the enterprise will do well.
- Financial Engine - The key to markets are their financial engines. This is one area AWS is way ahead of the game, with their approach to monetizing digital bits, and their sophisticated market creating pricing calculators for estimating and predicting costs gives them a significant advantage. Whichever marketplaces allows for innovation at the financial engine level will win.
- Definition Driven - Marketplaces of the future will have to be definition driven. Everything has to have a YAML or JSON definition, from the API interface, and schema defining inputs and outputs, to the pricing, licensing, TOS, and SLA. The technology, business, and politics of the marketplace needs to be defined in a machine-readable way that can be measured, exchanged, and syndicated as needed.
Google has inroads into this realm with their GSuite Marketplace, and Play Marketplaces, but it feels more fragmented than Azure and AWS approaches. None of them are as far along as Algorithmia when it comes to specifically ML focused APIs. In coming months I will invest more time into mapping out what is available via marketplaces, trying to better understand their contents–whether application, SaaS, and data, content, or algorithmic API.
I feel like many marketplace conversations often get lost in the discovery layer. In my opinion, there are many other contributing factors beyond just finding things. I talked about the retail and wholesale economics of Algorithmia’s approach back in January, and I continue to think the economic engine will be one of the biggest factors in any API ML marketplace success–how it allows marketplace vendors to understand, experiment, and scale the revenue part of things without giving up to big of a slice of the pie.
Beyond revenue, the modularity and portability will be equally important as the financial engine, providing vital relief valves for some of the classic silo and walled garden effects we’ve seen the impact the success of previous marketplace efforts. I’ll keep studying the approach of smaller providers like Algorithmia, as well as those of the cloud giants, and see where all of this goes. It is natural to default to AWS lead when it comes to the cloud, but I’m continually impressed with what I’m seeing out of Azure, as well as feel that Google has a significant advantage when it comes to TensorFlow, as well as their overall public API experience–we will see.
I saw the news that Google’s Spanner Database is ready for prime time, and I wanted to connect it with a note I took at the Google Analyst Summit a few months back–that gRPC is the heart of the database solution. I’m not intimate with the Spanner architecture, approach, or codebase yet, but the API focus, both gRPC core, and REST APIs for a database platform are very interesting.
My first programming job was in 1987, developing COBOL databases. I’ve watched the database world evolve, contributing to my interest in APIs, and I have to say Google Spanner isn’t something I anticipated. Databases have always been where you start deploying an API, but Spanner feels like something new, where the database and the API are one, and the way the database does everything internally and externally is done via APIs (gRPC).
Now that Spanner Database is ready for prime time, I will invest some more time in standing up an instance of it and get to work playing with what is possible with the REST APIs. I also want to push forward my grPC education by hacking on this side of the database’s interface. Spanner feels like a pretty seismic shift in how we do APIs, and how we do them at scale–when you combine this with the elasticity of the cloud, and the simplicity of RESTful interfaces I think there is a lot of potential.
I’m comparing five separate vendor API implementations with the Human Services API standard I’m working on at the moment. I’m looking to push version 1.0 of the API towards a 1.1 with some incremental, forward-thinking changes.
During This phase of the project, I’m looking to get as much feedback on the API interface from commercial vendors. The Human Services schema is being moved forward by a separate, but overlapping group, and has already gone through a feedback phase, and has officially released version 1.1 of the schema–I’m looking to do the same for the API.
Even though the Human Services schema is present, the purpose of the API definition is to open up discussion about what access to that data looks like, with the OpenAPI for the Human Services API acting as a distributed and repeatable contract governing how we access publicly available human services data.
The contract provided by the Human Services API defines how stakeholders can access organizations, locations, services. The Human Services schema defines how human services data is stored, and with the assistance of the API will be defined in transit for every request made, as well as the response that is given.
If we are going to get thousands of municipalities exchanging data with each other, as well as with the growing number of applications and systems they are using to serve the public, we will need a shared definition for how data is stored, as well as accessed by everyone involved. As I prepare responses to vendors involved in the feedback loop, I just wanted to gather my thoughts regarding the separation between the schema efforts and the API efforts.
The folks behind the OpenAPI Spec driven, interactive API documentation ReDoc, have also developed an OpenAPI generator that helps you manage your OpenAPI Spec deployment using Yeoman. If you aren’t familiar with Yeoman, it is modern scaffolding for web apps, which is all about helping you manage the quick and consistent deployment of APIs following existing best practices.
ReDoc’s Yeoman generator leverages all the benefits of pushing your OpenAPI Specs and API documentation on Github:
- Community Engagement - Allows for engagement with API consumers via Github’s native infrastructure, and issue management.
- Hosting on GitHub Pages - You are offloading the hosting and bandwidth to Github, and their CDN, significantly reducing overhead.
- Exposure in Github Community - Your API documentation and presence is available on Github, via trending, search, and other discovery mechanisms.
- Life Cycle Management - Github provides you with version history, branching, and continuous integration options, matching with your existing workflows.
- Familiar Environment - People increasingly understand Github as an environment where you can engage with code, as well as the community around the tooling and content published in this environment.
ReDoc’s Yeoman generator helps to create a GitHub repo with the following features:
- Possibility to split a big Swagger spec into smaller files and bundle it for deployment
- Continuous integration/deployment on Travis
- Code samples as separate files
- Swagger spec is validated after each commit
- Swagger spec + ReDoc deployed to Github Pages
- Live editing in your editor or swagger-editor
All the documentation for ReDoc’s Yeoman generator is available on the Github repository. The Yeoman generator demonstrates how OpenAPI Spec operates as more than just a machine readable core to API documentation. It acts as a machine-readable core to all stops along the API life cycle defining how you design, deploy, manage, and document your APIs.
ReDoc shows us the role OpenAPI Spec can play in driving your APIs documentation, but then turns things up a notch or two, demonstrating that when you publish your OpenAPI Spec to Github and drive your documentation, it can become a Github-driven engine for any services and tooling you will need as part of API operations—from design all the way to deprecation. ReDoc’s OpenAPI Spec generator will help you bootstrap the deployment your OpenAPI, which will end up being the heartbeat of your API.
All of my websites have run 100% on Github for the last three years. The core of my API industry research is always in JSON or YAML, stored in individual project-based Github repositories. I leverage Jekyll for the page and other content collections (blogs, news, etc.). Since 2011 I’ve used my own homebrew CMS system, making it accommodate the switch to a more static presence on Github.
Over the weekend I ditched my CMS and lit up a new CMS I came across called Siteleaf, which has all the core features I need: Github, Jekyll, Amazon S3, and API. This is how I manage a couple hundred API research sites, and the images, video, and other heavy objects I store using Amazon S3–these services and tools are critical to my business.
I am writing this post in Siteleaf. I do not have it setup for publishing across all of my websites, but so far it has worked for getting up 3 blog posts on API Evangelist, providing me with a successful test run. Siteleaf represents how I think software tools should be built, providing a simple useful application, that leverages Github, common storage like AWS S3, Dropbox, and others, and of course, they should ALL have an API.
I am going through my entire infrastructure lately, quantifying the products and services that API Evangelist offers, and the partnerships that make everything go round. As I do in my work as the API Evangelist, I’m looking to work through my thoughts here on the blog, and this week I have an interesting topic on the workbench–the API Evangelist remarketing tags.
According to Google, remarketing tags are: “To show ads to people who have visited your desktop or mobile website, add the remarketing tag to your website. The tag is a short snippet of code that adds your website visitors to remarketing lists; you can then target these lists with your ads. If your website has a Google Analytics tag, you can use this tag instead and skip adding the AdWords remarketing tag.”
Over the last couple of years, I’ve had 3Scale’s remarketing tags on my site. 3Scale has been my number one supporter for the last three years, and API Evangelist wouldn’t even be a thing without them. They have provided me with financial support each month for a variety of reasons. We’ve never fully itemized what is in exchange for this support, primarily it has been to just invest in me being API Evangelist and help 3Scale be successful, but one of the things on the table for the last couple of years has been that I published the 3Scale remarketing tags on my network of sites.
So, if you visited API Evangelist in the last three years, it is likely you saw 3Scale ads wherever you went on the web. As I’m evaluating all the products and services I offer, quantify my partnerships, and identify the value that API Evangelist brings to the table, I find myself thinking deeply about this practice. Should I be selling my visitors data like this? If you visit API Evangelist, should I allow my partners to target you with advertising on other sites you visit? As I think about the challenges I face in accepting money from startups, I’m forced to look at myself in the mirror as I think about selling remarketing tag access on my website(s).
Ok, I take the high road. I don’t sell remarketing tags to my partners. I even take the Google tracking off my website. I’m happy with tracking my traffic at the DNS level, I do not need to do this. Except for the fact that it brings in revenue, I need to pay my server bills, rent, and everything else that keeps me doing API Evangelist. It isn’t easy making a living being independent in the tech space, and I need every little bit of revenue I possibly can. It’s not an insignificant amount of money either. I make a couple thousand dollars a month this way, as the traffic to my website is pretty high quality, at least in the eyes of an API service provider looking to sell you their warez.
I am still thinking on this one. I need to make a decision this month regarding what I am going to do with the remarketing tags for API Evangelist. Do I want to sell them entirely to one partner? Do I want to sell individual ones for each area of my research, like design, deployment, management, monitoring, and the other 70 areas of the API lifecycle I track on? IDK. It is a shitty place to be, with so few options for monetization of my work, beyond just the current advertising industrial complex (AIC)™. I’d love to hear your thoughts, either publicly or privately. It’s a tough decision for me because if I choose to not do it, I risk API Evangelist going away, and me having to go get a job. shudder shudder again I’m exploring other ways of generating revenue, things like creating classes, and other content I can sell access to, so hopefully I can just walk away from remarketing tags without it killing me off.
I saw that WhoAPI launched a Domain Score API recently, helping put a value on whether or not you can trust a domain. The example they have in their blog post applies a domain score to the email addresses for any developer signing up for the Domain Score API–pretty useful stuff.
I do not know anything about the algorithm behind the domain scoring API, or what data it pulls from, but I think the concept is definitely applicable in today’s online environment. I wrote about Best Buy requiring their developers to register with an email at their business domain, not some general email service, and this seems like another layer of security you could add to this process.
With the current climate online, services like these are going to be increasingly valuable in the day to day operations of businesses, and not just for registering for an API. I’m going to start building a collection of these types of APIs and publish them as an individual stack–not sure what I’ll call it yet, but hopefully, it will be something I can leverage as part of a future API ranking system.
Showcasing the integrations that are possible with your API via your API developer portal is an increasingly important way to demonstrate the usefulness of your API resources. Companies like Amazon, DataDog, Intercom, and other leading providers showcase other systems their solution is already integrated with. When it comes to API solutions, applications aren't just web and mobile, they are often system to system integrations with many of the SaaS and other software solutions that companies are already using in their operations.
As I do with other types of signals coming out of the API space I'd like to see more API providers publish a listing of possible integrations, and I've created a simple API solution for managing an API integration page that can run in any Jekyll environment and possesses an API core. Using my approach to managing simple datasets, and publishing a human and machine interface using Github, I published an API integration page solution. Anyone can fork the project, reskin the look & feel of the website, and update the listing of integrations that are possible using a Google spreadsheet, then publishing the latest data using a Github repo and site.
My objective is to help API providers better manage and communicate the integrations that are possible via their API platform. I want them doing this in an organized and standardized way, helping their API consumers understand what is possible. I also want there to be a machine-readable definition of these integrations, so that analysts like me can more easily aggregate and connect the dots when it comes to what is possible with APIs. Right now, I am just tracking the URL of API providers integration pages, but I'd love it if the contents were also machine-readable and available via Github and API for forking and integration into other applications.
My API integration page and API template can be forked and run standalone, or you can copy the listing or icon pages, APIs, and YAML data core into a separate project. Each project has an OpenAPI defining the API surface area, and an APIs.json index for the entire project, as well as a YAML and JSON representation of API integrations present. I haven't had time to document everything 100%, so there is some awareness of Github, Jekyll, Liquid, and OpenAPI needed to get up and running. If you need help setting up a project, I'm adding these types of implementations to my list of services I offer, and I'm happy to help setup one up, and populate it for you, and even help you manage--for a fee of course. ;-)
I spend a lot of time talking about API providers, companies who have a public APIs. Occasionally, you will also hear me talk integration platform as a service (iPaaS) providers, companies like Zapier and Datafire who focus on providing a platform that connects you with many different API integration possibilities. These companies are a valuable player in the API ecosystem because they acknowledge that we usually do not just need one API, we will almost always need to integrate with many APIs, and they provide tools for developers, and non-developers to deliver API solutions that can leverage multiple individual APIs in a variety of business workflows.
I just got off a call with Sean Matthews of Left Hook Digital, an integration service provider who "efficiently build, maintain, and grow their integration options through a diversified iPaaS presence." This is the other half of the API integration coin I have been looking for, actual people who will help you navigate the crazy world of API integration, as well as the growing number of API integration and aggregation platforms and tooling that have been emerging, and evolving. I've been looking for people to help my readers navigate this world of API integration gray space in between full automation and custom development.
I've been looking for people to help small businesses, organizations, institutions, and government agencies understand how they can better leverage API aggregation providers like Cloud Elements, and integration platform as a service provider (iPaaS) like Zapier. Both platforms provide a wealth of services and tooling, but there still needs to be a person who is knowledgeable of these platforms who are willing to talk to a company or organization about which API-driven services they use, and what the possibilities and limitations around integrations are.
In addition to talking to your average company about integration, I've also been in need of knowledgeable folks to help API providers better leverage aggregation and integration platforms in their own operations. API providers are in desperate need of API design knowledge, helping to make sure their APIs reflect common patterns already in use across the sector, by leading providers, reducing friction in the API integration process. API providers also need help defining, implementing, managing, and communicating what the aggregation and implementation possibilities are with their platforms are.
Sean and I will be continuing our conversation about API integration, exploring how we can work together to tell stories, and craft more definitions of what is possible when it comes to API integration. If you are a company, organization, institution, or agency looking to better understand the API integration space, or an API provider looking to get a handle on it, feel free to reach out to Left Hook (tell them I sent you). Also, if you are an integration expert, or would like to be plugged into a larger network of API integration experts, where your talents can be leveraged, and applied in paid projects, feel free to reach out.
I'm in the design and development phase of my own API Evangelist integration with Zapier, just so I can better articulate what is possible. Cloud Elements has long been an API Evangelist partner, so I'm really excited to see what Left Hook is up to. I'm looking forward to continuing to help define this layer of the API lifecycle, which I think it is one of the more important aspects of the API economy, enabling everyday problem owners to find API driven solutions, and put them to work with no, or limit coding knowledge required.
I was trying to fit some new content into my website, and I couldn't make it fit within the layout. Then I remembered I had also taken off a section of my work because it looked like crap on my iPad a couple weeks back. It is a sign I've outgrown the current layout of my website when I can't publish my new work, as well as be an adequate archive for my historical research. Thankfully, my website is a pretty modular Jekyll implementation, so once I found the right look, it was only a couple hours worth of work to give it a full overhaul.
The new look and feel for API Evangelist remind me of the original look for the site back in the day, but with a more modern touch. It's responsive and has one of those little icon menus that follow you as your scroll. Something that bothered me at first, but has grown on me a little bit. While my old site was responsive, over the last couple years I managed to screw a few things up as I worked to make it fit all my crazy ideas. This new layout feels bigger, with much more sensible information architecture--something I'm hoping will accommodate what I'm looking to do with API Evangelist over the next couple of months.
The overhaul of the UI for API Evangelist also gave me an opportunity to shift how I manage the YAML data across all my research, moving from a MySQL database to Google Sheets. I'm using my evolved approach to using Github, Jekyll, and Google Sheets to deploy new data-driven API research, complete with HTML, YAML, Atom, and JSON output, depending on who I am looking to serve. All my short form content (blog posts) is managed using Jekyll within each repository, and my long form content (guides) are available via PDF--it's a nice marriage between content and data, that jives with my workflow.
I feel like the new look brings out the scope of work across my existing research. I feel like it will also allow me to grow, adding in my industry guides for each of my research areas, while also having room for new things like videos, and other API training material that I am working on. The layout lends itself well to displaying images, video, and other more interactive ways of publishing stories using data, which fits with the path I'm headed down with my storytelling and evangelism. I have to say, making sure my projects are modular and using Jekyll and Github as the core has seriously improved my ability to overhaul a large number of sites, data, and content in a short amount of time.
I have used the analogy of public lands when talking about access to, and monetization around public data resources, for a number of years. While not a perfect analogy, it provides me with a very tangible, and relatable way to help people understand access to, and the value of public data resources that can often be very abstract and difficult to see. Conveniently, some of the stories about public data, and policy I've worked on in the Federal Government involved public data that was actually about public lands, and more specifically national parks, and other resources in the Recreational Information Database (RIDB).
While discussing this work on a conference call the other day, someone thought that using the analogy of public lands when talking about public data didn't always work because public lands were a limited resource--a national park is only so big. It was not endless like digital data and content can be. I urged them to think beyond just the land, but the elements of the land, and a variety of resources made available via its existence. Things like water, minerals, fish, timber, and other resources that are often very valuable. While not all these resources are endless, some are, and others are renewable if managed properly, sharing many useful considerations for applying to how we manage access to our digital public resources.
I feel like there are many lessons present in applying the public lands analogy to public data. Depending on who you talk to, you will receive different views over ownership and access to public lands in the United States. Similarly, depending on who you talk to, you will receive different views over ownership and access to public data. It is easy to think of digital public assets as infinite, but similar to public lands, this will vary from resource to resource, and in some cases public data could be very limited, involving things like quarterly budgets, executive summaries, and other aggregate views of what is going on--not everything digital is infinitely scalable.
There are other analogies I am exploring when it comes to helping people understand public data, and how we discuss access to these resources. I think public roads, and other infrastructure also offers some interesting potential when it comes to quantifying public data, and its value to commercial interests, as well as to the larger public. I'll keep exploring this concept as I have time, and find ways of helping paint a clearer picture of public data using common physical elements of our world we are all familiar with. I'm looking to develop a robust toolbox of analogies I can use in my storytelling involving public data, and how we make it visible, usable, and even generating vital revenue fo budget-starved public data stewards across city, county, state, and federal government agencies, and the organizations, institutions, and companies that help them in their efforts.
It is a hustle to do API Evangelist. I've been lucky to have the support of 3Scale since 2013, without them API Evangelist would not have survived. I'm also thankful for the community stepping up last year to keep the site up and running, keeping it community focused thing, and not just yet another vendor mouthpiece. I make my money providing four ad slots on the site, by selling guides and white papers, and by consulting and creating content for others. It is a hustle that I enjoy much more than having a regular job, even though it is often more precarious, and unpredictable regarding what the future might hold.
Taking money from companies always creates a paradox for me. People read my stories because they tend to be vendor neutral and focus on ideas, and usable API-centric concepts. While I do write about specific companies, products, services, and tooling, I primarily try to talk about the solutions they provide, the ideas and stories behind them, steering clear of just being a cheerleader for specific vendor solutions. It's hard, and something I'm not always successful at, but I have primarily defined my brand by sticking to this theory.
This approach is at odds with what most people want to give me money for. 3Scale has long supported me and invested in me being me, doing what I do--which is rare. Most companies just want me to write about them, even if they understand the API Evangelist brand. They are giving me money, and in exchange, I should write about them, and their products and services. They often have no regard to the fact that this will hurt my brand, and run my readers off, falling short of actually achieving what they are wanting to achieve. I get the desire to advertise and market your warez, but the ability for companies to be their own worst enemy in this process is always fascinating to me.
I get regular waves of folks who want to give me money. I'm talking with and fending off a couple at the moment. So I wanted to think through this paradox (again), and talk about it out loud. It just doesn't make sense for me to take money from a company to write blog posts, white papers, and guides about their products and services. However, I feel like I can take money from companies to write blog posts, white papers, and guides about an industry or topic related to the problems they provide solutions for, or possesses a significant audience that might be interested in their products and services. I consider this underwriting and sponsorship of my API Evangelist research, where they receive branding, references, and other exposure opportunities along the way.
Ideally, all branding, reference, and exposure elements are measurable through the tracking of impressions and links. What was the reach? What is the scope of exposure? It is difficult to sustain any relationship without measuring success and both parties are unable to articulate and justify the financial underwriting and support. In some cases, I get paid a finder's fee for referrals, but this can be very difficult to track on and validate--something that requires a company to be pretty ethical, and truly invested in partnerships with smaller brands like me. I prefer to rely on this, as opposed to any sort of affiliate or reseller style tracking systems. I like companies that ask their customers, "how did you learn about us?", as they tend to actually care about their employees, partners, other stakeholders at a higher level.
Sometimes I take an advisor, or even technology investor role in startups, taking on a larger stake in outcomes, but these are something that is very rare. I have a manila file folder filled with stock options, and stakes I have in companies that never reached an exit, or when they did I was written out of things--when I do get a check from startup founders, I'm always floored. This does happen, but is truly a rare occurrence. I wish there were more decoupled, plug and play opportunities to invest in startups as an advisor, researcher, analyst, and storyteller, but alas the system isn't really set up for this type of thinking--people prefer the big money plays, over smaller, more meaningful contributions--it's about the money man.
Anyways, every time I visit this conversation in my head I come back to the same place. I'm happy to take money from companies to generate short form and long form content about an industry or topic. If there are finder fees for referrals, great! I leave it up to you to track on and come back to me with the details, and any specific event--while I will stay focused on what I do best, the thing you are investing in me to do. I'm mildly interested in opportunities to become more invested in your companies overall success, honestly, I just don't trust that the system will deliver in this area, and is more often just looking to extract value from me. I have seen too much in my 30-year career. However, I always welcome folks who want to prove me wrong! ;-)
In the end, my mission stays the same. I'm interested in studying where the API space has been, where it is at, and where it might be going. Then, I am interested in telling stories from what I learn in this process. If you want to invest in any of this, let me know. I could use your help.
I used to keep track of Swagger vendor extensions are part of my previous research around what was formerly known as Swagger. It is something I'm reviving as part of my OpenAPI Toolbox work, profiling the OpenAPI extensions I come across in the course of my work.
While profiling the Azure as part of my API Stack research I came across Microsoft Flow, and noticed that they use OpenAPI as part of the configuration of the integration platform as a service (iPaaS) solution, and have four specific extensions defined:
- x-ms-summary - Title of the entity.
- x-ms-visibility - Determines the user facing visibility of the entity.
- x-ms-dynamicvalues - Enables populating a dropdown for collecting input parameters to an operation.
- x-ms-dynamic-schema - This is a hint to the flow designer that the schema for this parameter or response is dynamic in nature.
According to the Microsoft Flow website: "To use custom APIs in Microsoft Flow, you must provide a Swagger definition, which is a language-agnostic machine-readable document describing the API's operations and parameters. In addition to the out-of-the-box Swagger specification, there are some extensions available when creating a custom API for Microsoft Flow."
I like that they use OpenAPI for this layer, and think that their extension of the OpenAPI specification is worth noting. My goal is to aggregate all the extensions I can as part of the OpenAPI Toolbox, encouraging reuse of existing patterns that have not made it into the specification. I also want providers to understand that they can extend the specification from outside the working group, delivering interesting features in the services and tooling you develop. I don't think that folks always understand the role they can play in helping define the OpenAPI specification just by developing interesting implementations that use OpenAPI as the common way to define and model your APIs.
I'm always looking for simpler, and cheaper ways of doing APIs that can help anyone easily manage data while making it available in both a human and machine readable way--preferably something developers and non-developers both will find useful. I've pushed forward my use of Github when it comes to managing simple datasets, and have a new approach I want to share, and potentially use across other projects.
You can find a working example of this in action with my OpenAPI Toolbox, where I'm looking to manage and share a listing of tooling that is built on top of the OpenAPI specification. Like the rest of my API research, I am looking manage the data in a simple and cheap way that I can offload the storage, compute, and bandwidth to other providers, preferably ones that don't cost me a dime. While not a solution that would work in every API scenario, I am pretty happy with the formula I've come up with for my OpenAPI Toolbox.
Data Storage and Management In Google Sheets
The data used in the OpenAPI Toolbox comes from a public Google Sheet. I manage all the tools in the toolbox via this spreadsheet, tracking title, description, URL, image, organization, types, tags, and license using the spreadsheet. I have two separate worksheets, one of which tracks details on the organizations, and the other keeping track of each individual tool in the toolbox. This allows for the data to be managed by anyone I give access to the sheet using Google Docs, offloading storage and data management to Google. Sure, it has its limitations, but for simple datasets, it works better than a database in my opinion.
Website and Basic API Hosting Using Github
First, and foremost the data in the OpenAPI Toolbox is meant to be accessible by any human on the web. Github, using their Github Pages solution, combined with the static website tool Jekyll, provides a rich environment for managing this data-driven toolbox. Jekyll provides the ability to store YAML data in its _data folder, which I can then use across static HTML pages which display the data using Liquid syntax. This approach to managing data allows for easy publishing of static representations in HTML, JSON, and YAML, making the data easily consumable by humans and machines, in an environment that is version controlled, forkable, and free for publicly available projects.
HTML Toolbox For Humans To Browse The Toolbox
Jekyll provides a static website that acts as the public face for the OpenAPI Toolbox. The home page provides icon links to each of the types of tools I have indexed, as well as to specific tags I've selected, such as the programming language of each tool. Each page of the website is an HTML page that uses Liquid to display data stored in the central YAML store. Liquid handles the filtering of data by type, tags, or any other criteria I choose. Soon I will be adding a search, and other ways to browse the data in the toolbox as the data store grows, and I obtain more data points to slice and dice things on.
JSON API To Put To Use Across Other Applications
To provide the API-esque functionality I also use Liquid to display data from the YAML data store, but instead of publishing as HTML, I publish as JSON, essentially providing a static API facade. The primary objective of this type of implementation is to allow for GET requests on a variety of machine-readable paths for the toolbox. I published a full JSON listing of the entire toolbox, as well as more precise paths for getting at types of tools, and specific programming language tools. Similar to the human side of the toolbox, I will be adding more paths as more data points become available in the OpenAPI toolbox data store.
Documentation Using Liquid and OpenAPI Definition
Project Support and Road Map Using Github Issues
As with all of my projects I am using the underlying issue management system to help me manage support and the roadmap for the project. Anyone can submit an issue regarding a tool they'd like to see in the toolbox, regarding API integration, or possibly new APIs they would like to see published. I can use the Github issue management to handle general support requests, and communication around the project, as well as incrementally manage the data, schema, website, and API for the toolbox.
Indexed In Machine Readable Way With APIs.json
The entire project is indexed using APIs.json, providing metadata for the project as well as other indexes for the API, support, and other aspects of operating the project. APIs.json is meant to provide a machine readable index for not just the API, which is already defined using OpenAPI, but for the rest of the project, including documentation and support, and eventually a road map, blog, and other supporting elements. Using the APIs.json index, other systems can easily discover the API, and programmatically access the data via the APIs, or even access the repository for the spreadsheet via the Github API, or the Google Sheet via its API--all the information is available in the APIs.json for use.
A Free Forkable Way To Manage Simple Data And APIs
This approach to doing APIs won't' be something you will want to do for every API implementation, but for simple data-driven projects like the OpenAPI Toolbox, it works great. Google Sheets and Github are both free, so I am offloading the storage, hosting, and bandwidth to another provider, while I am managing things in a way that any tech-savvy user could engage with. Anyone could manage entries in the toolbox using the Google Sheet and even engage with humans, and other applications via the published project toolbox.
I am going to continue evolving this approach to fit some of my other data-driven projects. I really like having all my projects self-contained as individual repositories, and the public side of things running using Jekyll--the entire API Evangelist network runs this way. I also like having the data managed in individual Google Sheets like this. it gives me simple data stores that I can easily manage with the help of other data stewards. Each of the resulting projects exists as a static representation of each data set--in this case an OpenAPI toolbox. I have many other toolboxes, toolkits, curriculum, and API research areas that are data driven, and will benefit from this approach.
What really makes me smile about this is that each project has an API representation of its core. Sure, I don't have POST, PUT, and DELETE capabilities for these APIs, or even advanced search capabilities, but for projects that are heavily read only--this works just fine. If you think about it though, I can easily access the data for reading and writing using the Google Sheets or Github APIs, depending on which layer I want to get access at. Really I am just looking to allow for easy management of data using Google Sheets, and simple publishing as YAML, JSON, and HTML, so that humans can browse, as well as put to use in other applications.
I am wanting to profile the world of machine learning APIs, similar to what I've done with Amazon, Google, Microsoft, Facebook, and the rest of my API Stack research, but I'm in need of some investment to help make sure I can properly carve out the time to conduct the research, and publish a resulting guide that provides an overview of the space, when done. After profiling the tech giants, I'm seeing some interesting shifts in the landscape when it comes to machine learning and would like to spend time profiling the rest of the landscape beyond just the bigcos.
While I am interested in mapping out the landscape of the machine learning API space, I don't have the time to make every project happen, and also pay the bills. To help make sure this project happens, I am looking for someone to step up and sponsor the research. In exchange for your brand and link on the resulting guide and any blog posts along the way, I'm looking for some cash investment to help me make ends meet, while I am doing the work.
When I profile a segment of the API space I focus on documenting the company behind, including all their relevant signals like Twitter and Github, but I also document the available API using OpenAPI, and the pricing, and other relevant building blocks. The resulting short form (blog posts), and long form (guide) provides an overview of the landscape when it comes to how APIs are being used as part of the latest machine learning wave. If your company is looking to better understand machine learning, and how APIs are playing a role, or maybe just looking to get some exposure with my audience who would read the blog posts and the resulting guide, let me know--I could use your help.
I added another type of tool to my OpenAPI Toolbox, this time it is extensions. They used to be called Swagger vendor extensions, and now they are simply called OpenAPI extensions, which allow any implementor to extend the schema outside the current version of the API specification. All you do to add an OpenAPI extension is prepend x- to any value that you wish to include in your OpenAPI, and the validator will overlook it as part of the specification.
I have a whole list of vendor extensions I'd like to add, but I've started with a handful from Microsoft Flow, and my friends over at APIMATIC. Two great examples of how OpenAPI extensions can be used in the API lifecycle. In this case, one is for integration platform as a service (iPaaS), and the other is SDK generation and continuous integration. Both vendors needed to extend the specification to meet a specific need, so they just extended it as required--you can find the extensions in the new section of the toolbox.
My goal in adding the section to the OpenAPI toolbox is to highlight how people are evolving the specification outside the core working group. While some of the extensions are pretty unique, some of them have a potential common purpose. I will be adding some discovery focused extensions next week from the OpenAPI directory APIs.guru, which I will be adopting and using in my own definitions to help me articulate the provenance of any OpenAPI definition in my catalog(s). Plus, I find it to be a learning experience to see how different vendors are putting them to work.
If you know of any OpenAPI extensions that are not in the toolbox currently feel free to submit an issue on the Github repository for the project. I'd like to evolve the collection to be a comprehensive look at how OpenAPI extensions are being used across the sector, from a diverse number of providers. I'm going to be teaching my own OpenAPI crawler to identify extensions within any OpenAPI it comes across, and automatically submit an issue on the toolbox, rapidly expanding the discovery of how they are used across a variety of implementations in coming months.
I keep an eye on several thousand companies as part of my research into the API space and publish over a thousand of these profiles in my API Stack project. Across the over 1,100 companies, organizations, institutions, and government agencies I'm regularly running into a growing number of signals that tune me into what is going on with each API provider, or service provider.
Here are the almost 100 types of signals I am tuning into as I keep an eye on the world of APIs, each contributing to my unique awareness of what is going on with everything API.
- Account Settings (x-account-settings) - Does an API provider allow me to manage the settings for my account?
- Android SDK (x-android-sdk) - Is there an Android SDK present?
- Angular (x-angularjs) - Is there an Angular SDK present?
- API Explorer (x-api-explorer) - Does a provider have an interactive API explorer?
- Application Gallery (x-application-gallery) - Is there a gallery of applications build on an API available?
- Application Manager (x-application-manager) - Does the platform allow me to management my APIs?
- Authentication Overview (x-authentication-overview) - Is there a page dedicated to educating users about authentication?
- Base URL for API (x-base-url-for-api) - What is the base URL(s) for the API?
- Base URL for Portal (x-base-url-for-portal) - What is the base URL for the developer portal?
- Best Practices (x-best-practices) - Is there a page outlining best practices for integrating with an API?
- Billing history (x-billing-history) - As a developer, can I get at the billing history for my API consumption?
- Blog (x-blog) - Does the API have a blog, either at the company level, but preferably at the API and developer level as well?
- Blog RSS Feed (x-blog-rss-feed) - Is there an RSS feed for the blog?
- Branding page (x-branding-page) - Is there a dedicated branding page as part of API operations?
- Buttons (x-buttons) - Are there any embeddable buttons available as part of API operations.
- C# SDK (x-c-sharp) - Is there a C# SDK present?
- Case Studies (x-case-studies) - Are there case studies available, showcasing implementations on top of an API?
- Change Log (x-change-log) - Does a platform provide a change log?
- Chrome Extension (x-chrome-extension) - Does a platform offer up open-source or white label chrome extensions?
- Code builder (x-code-builder) - Is there some sort of code generator or builder as part of platform operations?
- Code page (x-code-page) - Is there a dedicated code page for all the samples, libraries, and SDKs?
- Command Line Interface (x-command-line-interface) - Is there a command line interface (CLI) alongside the API?
- Community Supported Libraries (x-community-supported-libraries) - Is there a page or section dedicated to code that is developed by the API and developer community?
- Compliance (x-compliance) - Is there a section dedicated to industry compliance?
- Contact form (x-contact-form) - Is there a contact form for getting in touch?
- Crunchbase (x-crunchbase) - Is there a Crunchbase profile for an API or its company?
- Dedicated plans pricing page (x-dedicated-plans--pricing-page)
- Deprecation policy (x-deprecation-policy) - Is there a page dedicated to deprecation of APIs?
- Developer Showcase (x--developer-showcase) - Is there a page that showcases API developers?
- Documentation (x-documentation) - Where is the documentation for an API?
- Drupal (x-drupal) - Is there Drupal code, SDK, or modules available for an API?
- Email (x-email) - Is an email address available for a platform?
- Embeddable page (x-embeddable-page) - Is there a page of embeddable tools available for a platform?
- Error response codes (x-error-response-codes) - Is there a listing or page dedicated to API error responses?
- Events (x-events) - Is there a calendar of events related to platform operations?
- Facebook (x-facebook) - Is there a Facebook page available for an API?
- Faq (x-faq) - Is there an FAQ section available for the platform?
- Forum (x-forum) - Does a provider have a forum for support and asynchronous conversations?
- Forum rss (x-forum-rss) - If there is a forum, does it have an RSS feed?
- Getting started (x-getting-started) - Is there a getting started page for an API?
- Github (x-github) - Does a provider have a Github account for the API or company?
- Glossary (x-glossary) - Is there a glossary of terms available for a platform?
- Heroku (x-heroku) - Are there Heroku SDKs, or deployment solutions?
- How-To Guides (x-howto-guides) - Does a provider offer how-to guides as part of operations?
- Interactive documentation (x-interactive-documentation) - Is there interactive documentation available as part of operatoins?
- IoS SDK (x-ios-sdk) - Is there an IoS SDK for Objective-C or Swift?
- Issues (x-issues) - Is there an issue management page or repo for the platform?
- Java SDK (x-java) - Is there a Java SDK for the platform?
- Joomla (x-joomla) - Is there Joomla plug for the platform?
- Knowledgebase (x-knowledgebase) - Is there a knowledgebase for the platform?
- Labs (x-labs) - Is there a labs environment for the API platform?
- Licensing (x-licensing) - Is there licensing for the API, schema, and code involved?
- Message Center (x-message-center) - Is there a messaging center available for developers?
- Mobile Overview (x-mobile-overview) - Is there a section or page dedicated to mobile applications?
- Node.js (x-nodejs) - Is there a Node.js SDK available for the API?
- Oauth Scopes (x-oauth-scopes) - Does a provider offer details on the available OAuth scopes?
- Openapi spec (x-openapi-spec) - Is there an OpenAPI available for the API?
- Overview (x-overview) - Does a platform have a simple, concise description of what they do?
- Paid support plans (x-paid-support-plans) - Are there paid support plans available for a platform?
- Postman Collections (x-postman) - Are there any Postman Collections available?
- Partner (x-partner) - Is there a partner program available as part of API operations?
- Phone (x-phone) - Does a provider publish a phone number?
- PHP SDK (x-php) - Is there a PHP SDK available for an API?
- PubSub (x-pubsubhubbub) - Does a platform provide a PubSub feed?
- Python SDK (x-python) - Is there a Python SDK for an API?
- Rate Limiting (x-rate-limiting) - Does a platform provide information on API rate limiting?
- Real Time Solutions (x-real-time-page) - Are there real-time solutions available as part of the platform?
- Road Map (x-road-map) - Does a provider share their roadmap publicly?
- Ruby SDK (x-ruby) - Is there a Ruby SDK available for the API?
- Sandbox (x-sandbox) - Is there a sandbox for the platform?
- Security (x-security) - Does a platform provide an overview of security practices?
- Self-Service registration (x-self-service-registration) - Does a platform allow for self-service registration?
- Service Level Agreement (x-service-level-agreement) - Is an SLA available as part of platform integration?
- Slideshare (x-slideshare) - Does a provider publish talks on Slideshare?
- Stack Overflow (x-stack-overflow) - Does a provider actively use Stack Overflow as part of platform operations?
- Starter Projects (x-starter-projects) - Are there start projects available as part of platform operations?
- Status Dashboard (x-status-dashboard) - Is there a status dashboard available as part of API operations.
- Status History (x-status-history) - Can you get at the history involved with API operations?
- Status RSS (x-status-rss) - Is there an RSS feed available as part of the platform status dashboard?
- Support Page (x-support-overview-page) - Is there a page or section dedicated to support?
- Terms of Service (x-terms-of-service-page) - Is there a terms of service page?
- Ticket System (x-ticket-system) - Does a platform offer a ticketing system for support?
- Tour (x-tour) - Is a tour available to walk a developer through platforms operations?
- Trademarks (x-trademarks) - Is there details about trademarks, and how to use them?
- Twitter (x-twitter) - Does a platform have a Twitter account dedicated to the API or even company?
- Videos (x-videos) - Is there a page, YouTube, or other account dedicated to videos about the API?
- Webhooks (x-webhook) - Are there webhooks available for an API?
- Webinars (x-webinars) - Does an API conduct webinars to support operations?
- White papers (x-white-papers) - Does a platform provide white papers as part of operations?
- Widgets (x-widgets) - Are there widgets available for use as part of integration?
- Wordpress (x-wordpress) - Are there WordPress plugins or code available?
There are hundreds of other building blocks I track on as part of API operations, but this list represents the most common, that often have dedicated URLs available for exploring, and have the most significant impact on API integrations. You'll notice there is an x- representation for each one, which I use as part of APIs.json indexes for all the APIs I track on. Some of these signal types are machine readable like OpenAPIs or a Blog RSS, with others machine readable because there is another API behind, like Twitter or Github, but most of them are just static pages, where a human (me) can visit and stay in tune with signals.
I have two primary objectives with this work: 1) identify the important signals, that impact integration, and will keep me and my readers in tune with what is going on, and 2) identify the common channels, and help move the more important ones to be machine-readable, allowing us to scale the monitoring of important signals like pricing and terms of service. My API Stack research provides me wit a nice listing of APIs, as well as more individualized stacks like Microsoft, Google, Microsoft, and Facebook, or even industry stacks like SMS, Email, and News. It also provides me with a wealth of signals we can tune into better understand the scope and health of the API sector, and any individual business vertical that is being touched by APIs.
I have seen a number of incarnations when it comes to making public data available on the Internet, from startup implementations like earlier InfoChimps, U.S. Federal Government efforts like Dataa.gov, and Socrata. Recently, Andrew Nicklin (@technickle), the Director of Data Practices at the Center for Government Excellence at Johns Hopkins University pointed out a version I haven't come across yet, the public / private sector hybrid.
Publicly-operated data markets. This is an extremely interesting approach, because it provides a few other benefits beyond making government data accessible. With this approach, a government offers a public data market as a platform on which it and third-parties make a variety of data available, some for free and some at a premium. Because they are operating it, the government gains the ability to apply taxes or fees to data-access transactions (and this could be through any or all of the models suggested above), but it also gets an opportunity to regulate the market itself by establishing ground rules to protect privacy, public interest, and so on. Smart Copenhagen appears to be moving in this direction, and Smart Dubai may evolve towards this as well. (These platforms also present the opportunity for revenue generation through advertising, even if it’s just advertising other datasets to their repeat customers.)
You can see this in action at the City Data Exchange in Copenhagen, Denmark, where the city has partnered with the Hitachi Insight Group to provide a data marketplace where public and private sector data can coexist, and as Andrew suggests, even allow for fee-based access (key point). Providing us with an evolving definition of what data marketplaces can be. I've seen a lot of data marketplace come and go, and this model, with a realistic business model for both public and private sector partners, provides me with a little bit of hope for providing access to public data in a sustainable way.
I will be keeping an eye on the City Data Exchange, and look for other similar examples. All the city data portals I track on are either public sector or private sector--no hybrids. Also, in this paradigm, only the private sector data marketplaces allow for fee-based access. I'll be adding this concept to my monetization of public data using APIs research, and see if I can discover any other examples in the wild, and maybe influence new partnerships between public and private sector organizations like we are seeing in Copenhagen. I think it is something we should be exploring in a very open and observable way, learning what works, and what doesn't, so that other cities can begin to develop strategies for managing public data, and even generating much-needed revenue when it makes sense.
I get why SaaS, and API providers offer a handful of pricing plans and tiers for their platforms, but it isn't something I personally care for as an API consumer. I've studied thousands of plans and pricing for API providers, and have to regularly navigate 50+ plans for my own API operations, and I just prefer having access to a wide range of API resources, across many different companies, with a variety of usage limitations and pricing based upon each individual resources. I really am getting tired of having to choose between bronze, gold, or platinum, and often getting priced out completely because I can scale to the next tier as a user.
I understand that companies like putting users into buckets, something that makes revenue predictable from month to month, or year to year, but as we consumer more APIs from many different providers, it would help reduce the complexity for us API consumers if you flattened the landscape. I really don't want to have to learn the difference between each of my provider's tiers. I just want access to THAT resource via an API, at a fair price--something that scales infinitely if at all possible (I want it all). Ultimately, I do not feel like API plans and tiers will scale to API economy levels. I think as API providers, we are still being pretty self-centered, and thinking about pricing as we see it, and we need to open up and think about how our API consumers will view us in a growing landscape of service providers--otherwise, someone else will.
As I pick up my earlier API pricing work, which has two distinct components: 1) all API resources and pricing available for a platform 2) the details of plans and tiers which a complex list of resources, features, and pricing fit into. It would be much easier to just track resources, the features they have, and the unit price available for each API. Then we could let volume, time-based agreements, and other aspects of the API contract help us quantify the business of APIs, without limiting things to just a handful of API contract plans and tiers, expanding the ways we can do business using APIs.
As an API provider, I get that a SaaS model has worked well to quantify predictable revenue in a way that makes sense to consumers, but after a decade or more, as we move into a more serverless, microservices, devops world, it seems like we should be thinking in a more modular way when it comes to the business of our APIs. I'm sure all you bean counters can get out of your comfort zone for a bit, and change up how you quantify access to your API resources, following the lead of API pioneers like Amazon, and just provide a master list of API, CLI, and console resources available for a competitive price.
I have been profiling the Microsoft Azure platform over the last couple of weeks, and I found their approach to talking about the regions that were available was worth taking note of. I haven't actually assessed who has more regions, but Azure's approach seems to be pretty advanced, even if AWS might possess more regions (gut feeling). By profiling these cloud services and their available APIs using OpenAPI I am hoping to eventually develop a machine-readable approach to comparing which providers are available within which regions.
Google has a regions page, but it doesn't feel as forward leaning as AWS and Azures. It is interesting to watch how each of these providers is handling the availability of API services in a variety of regions across North and South America, Europe, Asia, Africa, and the Middle East. I've been watching how providers are thinking about the availability of API resources in different geographic regions for a while, but after seeing Azure evolve in this area, it is something I'll keep a closer eye on it moving forward.
Increasing the number of available regions is definitely the biggest concern for providers, something that small providers will be able to piggyback on and expand using as the top cloud providers grow and expand their regions. API providers and API service providers should be expanding the number of regions available, but everyone involved needs to also get more organized about how they communicate with customers about which regions available--region availability should be communicated at the highest level, like we see with the AWS, Google, and Azure deployment pages, but should also work to articulate which regions are available at the individual API level.
As data and algorithmic nationalism continue to grow, we are going to see more focus from providers when it comes to enabling their customer's deployment and operation of APIs into exactly the region they need. I"m guessing with the evolution of software-defined networking (SDN), we are going to see more control over the transport and routing of the data, content, and other resources we are making available via our regionally deployed APIs. Along with other channels and building blocks that I tune into I will start working to define a schema for tracking on regions, allowing me to index which APIs are available in specific regions using APIs.json.
When you are an individual in a sea of tech giants, and startups who are moving technical conversations forward, it can be easy to just sit back, stay quiet, and go with the flow. As a single person, it feels like our voice will not be heard, or even listened to when it comes to moving forward standards and specifications like the OpenAPI, but in reality, every single voice that speaks up is important, and has the potential to bring a new perspective regarding what the future should hold when it comes to the roadmap.
If you are building any services or tooling that supports version 2.0 of the OpenAPI specification and will be looking to evolve your services or tooling to support version 3.0, you need to make sure and share your views. No matter where you are in the development of your tooling, planning or even deployment, you should make sure you gather and share your thoughts with the OpenAPI Initiative (OAI)--they have a form for tooling developers to submit their feedback and details about what you are up to.
Whether or not you submit your OAI tooling and service plans via the form they provide, you should be also telling your story on your blog. You don't have to have a big audience for your blog, you just need to make sure and publicly share the details of your tools and services, and your perspective of both the OpenAPI 2.0 and 3.0 versions. If you tell your story on your blog, and Tweet or email a link to me, I may even craft my own story based on your perspective, and publish to API Evangelist, and put in my OpenAPI Toolbox. Storytelling around the specification plays an important role in helping evolve the specification, as well as help onboard other folks to the API specification format.
As the only individual in the OAI, I can testify that I often feel like my voice is too small to make a difference. This is not true. Whether it's via the Open API Github repo, directly via OpenAPI tooling feedback forms, or even via our blogs on the open Internet, your perspective on OpenAPI matters. Makes sure it gets heard--if you don't step up and share via one of these open channels, you are guaranteeing that you won't be heard, and your views definitely will not matter. Make sure you step up, there is too much at stake when it comes to API definitions right now.
ReDoc is the responsive, three-panel, OpenAPI specification driven documentation for your API that you were looking for. Swagger UI is still reigning king when it comes to API documentation generated using the OpenAPI Spec, but ReDoc provides a simple, attractive, and clean alternative to documentation.
ReDoc is deployable to any web page with just two tags--with the resulting documentation looking attractive on both web and mobile devices. Now you can have it all, your API documentation looking good, interactive, and driven by a machine-readable definition that will help you keep everything up to date.
All you need to fire up ReDoc is two lines of HTML on your web page:
The quickest way to deploy ReDoc is using the CDN step shown above, but they also provide bower or npm solutions, if that is your desire. There is also a Yeoman generator to help you share your OpenAPIs that are central of your web application operation, something we will write about in future posts here on the blog.
ReDoc leverages a custom HTML tag, and provides you with a handful of attributes for defining, and customizing their documentation, including specurl, scroll-y-offset, suppress-warnings, lazy-rendering, hid-hostname, and expand-responses--providing some quick ways to get exactly what you need, on any web page.
There is a handful of APIs who have put ReDocs to use as API documentation for their platform:
There also provide a live demo of ReDoc, allowing you to kick the tires some more before you deploy, and make sure it does what you will need it to before you fork.
ReDoc provides a simple, OpenAPI spec compliant way of delivering attractive, interactive, responsive and up to date documentation that can be deployed anywhere, including integration into your existing continuous integration, and API lifecycle. ReDoc reflects a new generation of very modular, plug and play API tooling that can be put to use immediately as part of an OpenAPI Spec-driven web, mobile, and device application development cycle(s).
When thinking about generating revenue generated from APIs it is easy to focus on directly charging for any digital resource being made available via the API. If it's an image, we charge per API call, and maybe the amount of MB transferred. If it's messaging, we charge per message. There are plenty of existing examples out there regarding how you directly charge for data, content, or algorithms using APIs, and an API way of doing business--look to Amazon, Twilio, and other pioneers.
Where there are fewer examples and less open discussions, is around the value of the operation level of APIs,Â and making these data available via APIs--yes APIs for APIs. Modern approaches to doing APIs are all about requiring each application to use an API key with each call they make, the logging of each request and response, possessing the identifying key for each application. This is how API providers are developing an awareness of who is accessing resources, how they are being put them to use, and specific details about each application, and maybe even the users involved.
Sometimes the value generated at this layer doesn't exist. Due to restrictive access models, and direct revenue models, there isn't much going on operationally, so there isn't much value generated. However, when there is heavy usage around APIs, the exhaust of the API management layer can become increasingly valuable. What are people searching for? What are applications most popular? Which geographic regions are the most active? There is a pretty lengthy laundry list of valuable data points being applied across modern API operations, that are helping API providers better understand what is going on, that aren't often being included as part of the API road map, and future revenue models.
Ok, let me pause here for a moment. I identify the value being generated at this layer because I see existing providers reaching this realization in their operations, as well as wanting to help other providers see the potential being achieved by successful API providers. I also acknowledge there is a lot of exploitation, abuse, and surveillance going on at this level, which is one of the biggest reasons I'm encouraging more transparency, observability, and discussion about this layer. I want API providers to understand the potential, but I also want API consumers and the end users of their applications to understanding what is going on at the API operational layer as well.Â
The current model I'm looking at through this lens currently is around my Open Referral Human Services Data Specification (HSDS) work, where I'm trying to help define the operational layer of human services APIs, as well as the direct API access to this critical public data. I am asking the question of how stewards of this very important data at the municipal level leverage APIs to make their valuable resources more accessible, and put to work where it is most needed, while also being able to generate and harness the valuable particles generated as part of an API exhaust system. What are people searching for? How are demographics evolving in a city, and how can city services shift and evolve too. Making the operational layer available via API so that it is available to key decision makers, even if those are private sector decisions makers who are required to pay for access to this intelligence--bringing critical new revenue streams for data stewards.
Let's pause again and be honest about the privacy concerns here. Access at this layer needs an additional level of scrutiny and care, over the direct access layers. Examples I'm concerned with can be seen in searches for Muslim religious services, or possibly LGBTQ services, and other information that could be used to violate the privacy and rights of application developers and end users. There are numerous privacy and security concerns at this level, but the inverse of these concerns also highlight the value of a data access exhaust system at this level. This is important information, that can provide some real time signals for both the public and private sector to consider more deeply.
I am purposely using the word exhaust here, specifically as a noun, as I don't think people are considering this layer, and may often see log files, and other ways data being generated in this as way as an unusable byproduct and output of web and mobile operations. I want to help people see the potential dangers of exhaust from API-centric data operations, but also understand that when it is captured, it can become valuable, similar to natural gas capture from recycling or landfill operations. There are some toxic aspects of API-driven data operations, but when measured and controlled, and made more observable, the dangerous aspects can be mitigated, and you might also find ways that other reuse and extraction that can also occur along the way.Â
I'm involved in some very interesting conversations with public data folks who are trying to push forward the conversation around sensible revenue generation by cities, counties, state, and the federal government using public data. I'm learning a lot from these conversations, resulting in the expansion and evolution my perceptions of how the API layer can help the government develop new revenue streams through making public data more accessible.Â
I have long been a proponent of using modern API management infrastructure to help government agencies generate revenue using public data. I would also add that I'm supportive of the crafting of sensible approaches to developing applications on top of public data and API in ways that generate a fair profit for private sector actors. I am also in favor of free and unfettered access to data, and observability into the platform operations, as well as ALL commercial interests developing applications on top of public data and APIs. I'm only in favor of this, when the right amount of observability is present--otherwise digital good olÂ boy networks form, and the public will lose.
API management is the oldest area of my API research, expanding into my other work to eventually defineÂ documentation, SDKs, communication, support, monetization, and API plans. This is where you define the business of API operations, organizing APIs into coherent catalogs, where you can then work to begin establishing a wider monetization strategy, as well as tiers and plans that govern access to data, content, and algorithms being made available via APIs. This is the layer of API operations I'm focusing on when helping government agencies better understand how they can get more in tune with their data resources, and identify potential partnerships and other applications that might establish new revenue streams.
A portion of thisÂ conversation that I am having was involved in the story from Anthony Williams about maybe government data shouldn't always be free, where the topic of taxation came up. One possible analogy for public data access and monetization was brought up as a reference to theÂ Vehicle-miles Traveled (VMT) tax, injecting the concept of taxation to my existing thoughts on revenue generation using API management. I've considered affiliate and reseller aspects to the API management layer before, applying percentage based revenue and payments on top of API access, but never thought about a government taxation layer existing here.
I thought my stance on revenue generation on public data using API management was controversial before, adding in concepts of taxation to the discussion is really going to invigorate folks who are in opposition to my argument. I'm sure there is a libertarian free web, open data advocate, smaller government Venn diagram in there somewhere. I'm not too concerned, as the monetization is already going on, I'm simply talking about making it more observable, and in favor of revenue generation for data stewards and government agencies. I'm confident that most won't folks in opposition won't even read this paragraph, as it's buried in the middle of this post. ;-)
I take no stance on which data, content, or algorithms should be taxed, or what that tax rate should be. I leave this to data stewards and policy makers. My objective is to just introduce folks to the concept, and marry with the existing approaches to using APIs to develop digital products and services in the private sector. However, if I was wearing my policy maker hat I would suggest thinking about this as a digital VAT tax, "that is collected incrementally, based on the surplus value, added to the price on the work at each stage of production, which is usually implemented as a destination-based tax, where the tax rate is based on the location of the customer."
My thoughts on a government tax at the API management layer are at an early stage. I am just exploring the concept on my blog--this is what I do as the API Evangelist. I'd love to hear your thoughts, on your blog. I am merely suggesting a digital VAT tax at the API contract layer around public data and APIs when commercial activities are involved. Eventually, I could see the concept spread to other sectors as the API economy becomes a reality, but I feel that public data provides us with a rich test bed for a concept like this. I'm considering reframing my argument about charging for commercial access to public data using APIs as taxing commercial usage of public data using APIs, allowing for tax revenue to fund future investment in public data and API efforts.
As I remove my API Evangelist hat and think about this concept, I'm not 100% sure if I'm in agreement with my argument. It will take a lot more polishing before I'm convinced that taxation should be included in the API management layer. I'll keep exploring, and play with a variety of potential use cases, and see if I can build a case for API taxation when public data is involved, and applications are generating surplus value in the API economy.Â
Profiling APIs always provides me with a nice bulleted list of what a company does or doesn't do. In my work as the API Evangelist, I can read marketing and communications to find out what a company does, but I find that profiling their APIs provides a more honest view of what is going on. The lack of a public API always sets the tone for how I view what a company is up to, but when there is a public API, profiling it always provides a nice distillation of what a company does, in a nice bulleted list I can share with my readers.
When I profile the APIs of companies like Amazon, Google, and Microsoft, I come out of it with a nice bulleted list of what is possible, but when I go even further, making sure each API profile has accompanying schema definitions, a nice list of what data company begins to emerge. When I profile an API using OpenAPI I always start by profiling the request layer of an API, the paths, parameters, and other elements. Next, I get to work describing the schema definitions of data used in these requests, as well as the structure of the responses--providing me with a nice bulleted list of the data that a company has.
You can see this in action with my Facebook API profiling work. There is a bulleted list of what is possible (API definition), as well as what data is sent, received, and stored (API schema). This work provides me with a nice look at the data Facebook gathers and stores about everyone. It is FAR from a complete picture of the data Facebook gathers, but it does provide us with a snapshot to consider, as well as a model we can ask Facebook to share more schema about the data points that they track. API and data specification formats like JSON Schema, and OpenAPI provides us with a toolbox to help us quantify and share the details of what data a company has, and what is possible when it comes to using this data in web, mobile, and device based applications.
I fully aware of the boldness of this statement, but I feel that ALL companies should have a public API definition, including a catalog of the schema for data in use. Ideally, this schema would employ commonly used standards like Schema.org, but just having a machine-readable catalog of the schema would go a long way to helping pull back the curtain of how companies are using our data. I am not asking for companies to make data public, I am asking for companies to make the schema for this data public, showing what they track and store about us. I know many people view this as intellectual property, but in an increasingly un/insecure online world of digital privacy, we are going to have to begin pulling back the curtain a little bit, otherwise, a rich environment for exploitation and abuse will continue to develop.
It has been over a year since I've pushed any API definitions to my API Stack research, but I finally was able to prioritize time this week to make sure it was updated with the latest profiles I have in my API monitoring system. I pushed 1,173 companies who are doing interesting things with APIs. Not all of them have a traditional API program, but most of them do. It isn't all of the API related companies in my tracking system, but it's definitely the core group of what I'm watching.
Each API is profiled with an APIs.json file, providing an index of the name, description, tags, and other metadata, but also provides the URLs for documentation, Github, Twitter, and other key aspects of API operations. When there is an OpenAPI present, I publish a copy of it, with a URL for the resource, within each index. I have a bunch more OpenAPI definitions to publish, but they will take some considerable cleaning up before they are ready for prime time--look for regular updates now that I have the bulk of it updated.
My API Stack runs 100% on Github, like the rest of my research. All the data behind is available in APIs.jsonÂ and OpenAPI, both as JSON and YAML--take your pick. I prefer YAML being the core of everything and stored in the _data folder, but I tend to also put up JSON and HTML facades on this data when it makes sense. The homepage for the project is just an HTML listing done in Liquid, driven from the YAML in the _data folder, making it easy to play with different views of the APIs indexed as part of the collection.
At over 1000 APIs the repository is getting pretty unwieldy to commit each time, and i've begun the process of breaking up most of the larger APIs into their own repositories for individual management, projects like AWS, Google, Microsoft, and Facebook. I will keep most of the lesser known APIs as part of the main API Stack collection, but will continue to slice off the APIs, as well as individual stacks of APIs like email, SMS, and news into their own repositories for indepdenteÂ managment and evolution. I'm happy to finally have this updated, as I'm starting to find some really rich indexes of OpenAPI stashes on Github, and I want to start ingesting them as part of my larger index, something I didn't want to do until this was updated.
It made me happy to read the Rise of Non “VC compatible” SaaS Companies, and see that there are more sensible discussions going on around how to develop SaaS business, something that I hope spreads into the specifically API as a product startups and API service providers. I know that many of my readers think I'm anti-VC--I am not. Or may I'm anti-startup--I am not. I'm anti-VC and anti-startup ideology becoming the dominant religion, pushing out a lot of really good people and ideas who can't play that game.
When it comes to Silicon Valley, if you push back, you get excluded from the club, and there are waves of people who step up to tell you "not all startups are bad" or "not all VCs are bad"--I wish I could help you understand how this response makes you look. Of course, they aren't all bad, but there are bad ones, and there is a lot of rhetoric that this is the ONLY way you can build technology when it isn't. There are plenty of ways you can develop technology, and build a business without the VC approach, or the cult of the startup.
There are more instructions you should follow in the rise of the non-VC compatible SaaS companies story, but the author outlines four types of SaaS companies, which I think applies nicely to APIi companies, as many of them will be SaaS providers:
- Funded SaaS: companies which finance their business with VCs a.k.a equity against money. From early stage startups with no revenue to companies going public with hundreds of millions of dollars of ARR, the range is extremely wide.
- Bootstrapped “scaling” SaaS companies: SaaS companies which manage to pass the $10M ARR threshold without VC money. Ex: Mailchimp or Atlassian (which raise VC money but at a very late stage) have reached the hundreds of millions of dollars of ARR without VC money. These “unicorns among unicorns” are very rare.
- Bootstrapped SaaS companies: bootstrapped companies which manage to reach the $300k — $10M ARR range without VC money.
- Bootstrapped Micro SaaS: “1 to 3” people companies which operate in the $1k — $300k ARR range, without VC money.
There are some ideas that should go VC, but there are even more that should not. I want to see more talk like this about the funding realities of an API startups. A sensible discussion about what the goals are, how fast a company should grow, and whether the company is building a business to develop software solutions that we sell to customers, or a business to sell some large enterprise--hell, maybe even go IPO. These are all viable options for your business, I'm just looking for more discussion about the priorities. One more thing, you should be having this discussion out in the open with the customers you are supposedly selling your products and services to--this is the important part, not just the having of the discussion.
I'm not trying to get in the way of you getting super filthy rich. I'm just trying to strike a balance between you building a company, and the stability and availability of your APIs, and API tools and services, in an industry I depend on to make my living. You see, APIs have gotten a bad wrap for not being stable, and going away at any time. This isn't an API thing, this is a VC funded startup thing. When you are telling your companies that you are building a business, with products and services to purchase, and then everyone bakes your solutions into their solutions, and you go away--that sucks. If you tell people the truth from the beginning and are honest and communicative about what your plans are, people can build and plan accordingly--the problem isn't APIs going away, it is startup founders not caring.
I am just trying to strike a balance between your business aspirations, and those of the people I help convince to use your APIs. Many of them aren't trying to get rich. They are just trying to build a business, and get their work done with your service, tool, and API. Let's start having more honest and open conversation about the variety of business models available to use when building API startups, and remember that not everything is a VC-worthy idea, sometimes you are just a viable feature for regular business owners like me.
I am helping a customer think through their decision-making process around the adoption of a new API service, and while I am doing this I am spending the time to think through my own API adoption process. I like having checklists to consider when making new purchasing and integration decision. Sometimes I have an immediate need which is driven by emotion, and it can help to step back and think through a more rational checklist I established for myself on a previous date.
When I am approaching a new API that I think might move beyond just playing around, and actually have a viable business use case in my operations, I think through the following areas:
- Define Value - What value is being created by an API I'm looking to use.
- Define Needs - What needs do I have which using an API will solve.
- Define Options - What other solutions are there beyond this API.
- Think About Today - Is this an immediate need I have with days or weeks.
- Think About Tomorrow - Is this a long term need that will go on for years.
- Vet Company & People - More detail about the company, people, and investors.
- Define Partners - What does my the partnership landscape look like for the API.
- What Things Cost - What are things going to cost be for using an API.
- What You Can Afford - Can I really afford using this service, or not use.
- Develop Continuity Plan - What is the plan for ensuring stable operations using API.
- Develop Exit Plan - How will I severe a relationship and replace or deprecate need.
Sometimes I do not have everything in order before I pull the trigger. I may not completely though through what I will do when an API goes away, but I like at least having some notes written down about my frame of mind when I was integrating with any API. Having the checklist to think about at the time of integration and purchase, as well as my notes about evaluation around a vendor provides me with something I can revisit each month when paying the bill, or may as I encounter a new service, or possibly instability or lack of access to an API.
I am using this as part of my own API operations, but I'm also helping a client think through their API purchasing decisions, and hopefully, make the process a much healthier and educated one. I'm hoping this helps put much of the burden of API stability on the shoulders of us people who are actually making the decision, allowing us to be more mindful of who we integrate with, and set more informed and honest expectations around what APIs do, and where some of the current business models in place might help, or hurt our plans.
I learn a lot by studying APIs. One of the ways I get to know what an API does is by creating an OpenAPI for the API, which helps define all of the paths available for an API--helping me understand what is possible. After defining the API requests that are possible, ensuring there are simple descriptions for each path, I also like to make sure all the data that is being passed back and forth via an API is also defined in the OpenAPI--giving me a good snapshot of what data is stored behind an API.
In my current Facebook API definition, there is a total of 271 paths, with 90 objects defined using 654 data points. The machine-readable OpenAPI definition tells me what data is stored and transmitted, and what actions I can take involving these items. The Facebook API definition isn't complete, but it does provide me with a nice view of the Facebook Graph API landscape, and since the project is hosted on Github anyone can help me continue adding to and expanding on the definition, as well as the documentation and other tools I'm building on top of the definition.
There are three parts to my research: 1) to define Facebooks API operation from a technical and business perspective 2) understand which of my bits Facebook tracks about me and my business(es), and how I can take more control over these bits with the Facebook API, and 3) understand how Facebook can be orchestrated and manipulated as part of a social engineering toolbox. Facebook isn't on my list of top places I hang out online, but I grasp its magnitude, and importance to my business and my career, and our democracy--so I work to understand WTF is going on with it.
I currently post stories to Facebook from my content management system. When I blog something, if I want it on Facebook, I can schedule it, and my system posts the story using the Facebook API. I'm looking to expand this to my photos, video, and the other bits I make available online. Profiling an API like this helps me understand the wider Facebook API landscape, better understanding what is possible, but it also helps me get my personal and professional world in order when it comes to one of the most important social networks out there right now.
I work to develop OpenAPI definitions for 3rd party APIs because it helps me understand what is being offered by a company. Even when I'm able to autogenerate an OpenAPI for an API, or come across an existing one, I still spend time going through the finer details of what an API does, or doesn't do. I find the process to be one of the best ways to learn about an API, stopping short of actually integrating with it.
Over the last couple of months, I've aggregated, generated, and crafted OpenAPI and APIs.json definitions for the top three cloud API providers out there. I wanted to be able to easily see the surface area for as many of the APIs as I could find for these three companies:
- Amazon - 2222 paths (or methods) across 65 of the Amazon Web Services - you can find the APIs.json, and OpenAPI behind in the Github repository.
- Google - 2089 paths across 75 of the Google services I'm profiling - you can find the APIs.json, and OpenAPI behind in the Github repository.
- Microsoft - 2109 paths across 41 of the Microsoft and Azure services. - you can find the APIs.json, and OpenAPI behind in the Github repository.
I learned a lot about all three providers in the process. I filled my notebook with stories about their approaches. I also ended up with three separate Github repositories with APIs.json indexed OpenAPI definitions for as many of their APIs as I could process. They are far from complete, but I feel like they paint a pretty interesting picture of the API landscape across these three tech giants.
I am working to update my OpenAPI definitions for AWS, Google, and Microsoft using some other OpenAPIs I've discovered on Github. When a new OpenAPI has entirely new paths available, I just insert them, but when it has an existing path I have to think more critically about what is next. Sometimes I dismiss the metadata about the API path as incomplete or lower quality than the one I have already. Other times the content is actually more superior than mine, and I incorporate it into my work. Now I'm also finding that in some cases I want to keep my representation, as well as the one I discovered, side by side--both having value.
This is one reason I'm not 100% sold on the fact that just API providers should be crafting their own OpenAPis--sure, the API space would be waaaaaay better if ALL API providers had machine readable OpenAPIs for all their services, but I would want it to end here. You see, API providers are good (sometimes) at defining what their API does, but they often suck at telling you what is possible--which is why they are doing APIs. I have a lot of people who push back on me creating OpenAPIs for popular APIs, telling me that API providers should be the ones doing the hard work, otherwise it doesn't matter. I'm just not sold that this is the case, and there is an opportunity for evolving the definition of an API by external entities using OpenAPI.
To help me explore this idea, and push the boundaries of how I use OpenAPI in my API storytelling, I wanted to frame this in the context of the Amazon EC2 API, which allows me to deploy a single unit of compute into the cloud using an API, a pretty fundamental component of our digital worlds. To make any call against the Amazon EC2 I send all my calls to a single base URL:
With this API call I pass in the "action" I'd like to be taken:
Along with this base action parameter, I pass in a handful of other parameters to further define things:
Amazon has never been known for superior API design, but it gets the job done. With this single API call I can launch a server in the clouds. When I was first able to do this with APIs, is when the light really went on in my head regarding the potential of APIs. However, back to my story on expressing what an API does, as well as what is possible using OpenAPI. AWS has done an OK job at expressing what Amazon EC2 API does, however they suck at expressing what is possible. This is where API consumers like me step up with OpenAPI and provide some alternative representations of what is possible with the highly valuable API.
When I define the Amazon EC2 API using the OpenAPI specification I use the following:
title: Amazon EC2
summary: The Amazon EC2 service
- in: query
The AWS API design pattern doesn't lend itself to reuse when it comes to documentation and storytelling, but I'm always looking for an opportunity to push the boundaries, and I'm able to better outline all available actions, as individual API paths by appending the action parameter to the path:
title: Amazon EC2
summary: Run a new Amazon EC2 instance
Now I'm able to describe all 228 actions you can take with the single Amazon EC2 API path as separate paths in any OpenAPI generated API documentation and tooling. I can give them unique summaries, descriptions, and operationId. OpenAPI allows me to describe what is possible with an API, going well beyond what the API provider was able to define. I've been using this approach to better quantify the surface area of APIs like Amazon, Flickr, and others who use this pattern for a while now, but as I was looking to update my work, I wanted to take this concept even further.
While appending query parameters to the path definition has allowed me to expand how I describe the surface area of an API using OpenAPI, I'd rather keep these parameters defined properly using the OpenAPI specification, and define an alternative way to make the path unique. To do this, I am exploring the usage of #bookmarks, to help make duplicate API paths more unqiue in the eyes of the schema validators, but invisible to the server side of things--something like this:
title: Amazon EC2
summary: Run a new Amazon EC2 instance
- in: query
I am considering how we can further make the path unique, by predefining other parameters using default or enum:
title: Amazon EC2
summary: Run a new Amazon EC2 website instance
description: The ability to launch a new website running on its own Amazon EC2 instance, from a predefined AWS AMI.
- in: query
- in: query
I am still drawing in the lines of what the API provider has given me, but I'm now augmenting with a better summary and description of what is possible using OpenAPI, which can now be reflected in documentation and other tooling that is OpenAPI compliant. I can even prepopulate the default values, or available options using enum settings, tailoring to my team, company, or other specific needs. Taking an existing API definition beyond its provider interpretation of what it does, and getting to work on being more creative around what is possible.
Let me know how incoherent this is. I can't tell sometimes. Maybe I need more examples of this in action. I feel like it might be a big piece of the puzzle that has been missing for me regarding how we tell stories about what is possible with APIs. When it comes to API definitions, documentation, and discovery I feel like we are chained to a provider's definition of what is possible, when in reality this shouldn't be what drives the conversation. There should be definitions, documentation, and discovery documents created by API providers that help articulate what an API does, but more importantly, there should be a wealth of definitions, documentation, and discovery documents created by API consumers that help articulate what is possible.
Many of the problems that APIs are often associated with API adoption can often be mitigated via more communication. I track on a number of ways the successful API providers are communicating around their API efforts, but I also like it when API designers and architects communicate through the actual technical design of their APIs. One example of this can be found in the IETF draft submission for The Sunset HTTP Header, by Erik Wilde.
"This specification defines the Sunset HTTP response header field, which indicates that a URI is likely to become unresponsive at a specified point in the future."
In his original post, nothing lasts Forever (and some things disappear according to a schedule), Erik shows us that a little bit of embedded communication like this in the design of our APIs can help make API consumers more in tune with what is going on. It is tough to get people's attention sometimes, and I think sometimes when us engineers are heads down in the code we tune out the regular channels, and baking things in the request and response structure of the API can help out a lot.
I like the concept of baking API communication and literacy into our operations through good API design. This is one aspect of hypermedia API design that I think often gets overlooked by the haterz. I'll keep adding these little building blocks to my API communications research, helping develop ways we can better communicate with our API consumers through our API design--in real time.
I was firing up my low hanging fruit engine for a customer today, pulling down their entire website, giving them suggestions on where they should start with their API efforts, so I felt it was a good time to blog about this process again. Getting started with APIs can be a challenging question for some organizations, especially when they don't have executive and/or IT buy-in to the concept. To assist folks who find themselves in this situation, I have a process which I call "low-hanging fruit", designed to help jumpstart the API conversation in a safer way.
When someone approaches me with the question: Where do I start with APIs at our company, organization, institution, or government agency? If they do not have a specific project in mind or have the buy-in of IT yet, I always recommend started with the low hanging fruit on your website, and targeting these four areas:
- Existing Spreadsheets - If you there is a CSV, Microsoft Excel Spreadsheet, or Google sheet on the public website, it should be available as an API for searching, and usage in other applications.
- Existing Data Files - If there are existing XML, JSON, or other machine-readable data files already available on the public website, they should be available as an API for searching, versioning, and usage in other applications.
- Existing Tables - Any page that has a table over 25 rows should be reviewed, and consider for publishing as a simple API, for use in all applications.
- Existing Forms - Any page that has a form should be reviewed, and consider for publishing the content available within as a simple API, for use in all applications.
Begin here. Target these four areas across your websites. Identify the data source. See if you can get access to the source. If not, scrape the data. Either way, get the data published as a simple API. As soon as you launch your first API, make sure you also publish a basic API portal on Github, using Github pages, and publish your first API there. Get to work on additional APIs, and repeat until you have a robust catalog of APIs in your developer portal. Get to work evolving each API, improving how it acquires and delivers its data, and develop new types of applications and partnerships around the data you've made available. #repeat
The data and content available on your public website have two important characteristics: 1) it has already passed legal and approved for being available publicly, and 2) there is a value in it being made available, as it's already on the public website for a reason. Learn about APIs and the API life cycle within this process. Practice. Grow. Learn. Low hanging fruit gives you a simple, safe place to begin your API journey, learning how it all works in an environment that will be more forgiving when you make mistakes, and hopefully will provide some pretty useful solutions that you can use to demonstrate the value of doing APIs at your company, organization, institution, or government agency.
When I get asked by folks about where they should start with APIs, I always start with the low hanging fruit on their websites--if it is publicly available as HTML on your website, it should be available as JSON or YAML via an API. After that, I recommend people start with the software and systems a company is already using and might have APIs or data outputs. Your existing technology most likely has APIs or the ability to publish machine-readable data, and you are just not taking advantage of their capabilities as part of any larger API strategy.
I recommend launching a new developer portal using Github and get to work inventorying all of the software, systems, website, and services you use. It doesn't have to be public, you can make your repository private if you have any concerns. Profile each of the technological solutions you are using on a regular basis, and see if it has an API or not. If it doesn't, why the hell are you using it? Ok, well maybe that is another discussion. If it has an API, add it to your API list, and set it aside for profiling in more detail, developing a better understanding what is possible.
After reviewing all your existing software, and now possessing a list of companies who do have APIs, let's get to work profiling them, and bring together into a coherent 3rd party API strategy for your operations:
- Developer Portal - Where do we find documentation and other resources for each API?
- Pay for Service - Are we paying for a service? What are the pricing and plans for each API?
- Documentation - Where is the API documentation for each API we are including in our catalog?
- OpenAPI Definition - Create or obtain an OpenAPI definition for each API, allowing it to be used in operations.
- Support Information - Where do we get support for an API? Are there multiple options, or possibly paid solutions?
- Road Map & Change Log - Where do we find the roadmap, status, change log, and other aspects of API operations?
- Code & SDKs - Where do we find code and other open tools for an API? Which are forked, copied, or implemented for our operations?
- iPaaS Solutions - Can we identify and share common integration patterns, and put to use services like Zapier as part of our operations?
APIs are not just about providing APIs, it is also about consuming APIs. Google Apps, SalesForce, Office 365, and many of the software solutions you are already putting to work have APIs. There should be a single portal you can go to and understand what is possible when it comes to APIs You should be able to find code, tools, and communication around existing usage, integration, and other projects at your company. API portals are often confused as something you do for a single API, providing direct access to data, content, and algorithms, but there is no reason a portal can't also contain access to other 3rd party APIs that also benefit your operations.
The public or private API portal for your company, organization, institution, or government agency doesn't have to only possess your APIs. It makes perfect sense for you to aggregate the APIs from across the software solutions you use, cultivating a central place to find everything regarding 3rd party API integration at your company. In addition to low hanging fruit, it provides a great way to kick off an API effort at your company, focusing on the digital resources that are already being used, allowing you to learn more about an API way of life before you consider tackling more complicated API efforts.
|<< Prev||Next >>|