• Twitter
• Github
• LinkedIn
• RSS

I am Kin Lane, the API Evangelist...

This is my online domain where I work to understand the world of the Application Programming Interfaces, also known as APIs. This new way of sharing data using the web is touching almost every aspect of our increasingly digital lives, providing access to the bits and bytes that make our personal and professional worlds go round.

API Evangelist is my full time research into how APIs are impacting our worlds (both good and bad), where I work to better understand the technology, business, and politics of the APIs that are driving our websites, mobile, and devices that seem to be everywhere.

API Evangelist is a network of data driven projects and APIs which I curate and manage as part of this ongoing research, hoping to provide easy access to the moving parts of my work. Everything you see here runs on Github, making everything forkable, and resuable for both humans and machines.

API Evangelist Partners

These are my partners who invest in API Evangelist each month, helping underwrite my research, and making sure I'm able to keep monitoring the API space as I do.

Streamdata.io

Streamdata is a software vendor making real-time data accessible to all by operating a proxy turning request / response APIs into feeds of real-time events.

3Scale

3scale makes it easy to open, secure, distribute, control and monetize APIs, that is built with performance, customer control and excellent time-to-value in mind.

Latest From Blog: Delivering Large API Responses As Efficiently As Possible

20 Apr 2018

I’m participating in a meeting today where one of the agenda items will be discussing the different ways in which the team can deal with increasingly large API responses. I don’t feel there is a perfect response to this question, and the answer should depend on a variety of considerations, but I wanted to think through some of the possibilities, and make sure the answers were on the tip of my tongue. It helps to exercise these things regularly in my storytelling so when I need to recall them, they are just beneath the surface, ready to bring forward.

Reduce Size Pagination
I’d say the most common approach to send over large amounts of data is to break things down into smaller chunks, based upon rows being sent over, and paginate the responses. Sending over a default amount (ie. 10,25,100), and require consumers either ask for larger amount, as well as request each additional page of results in a separate API request. Providing consumers with a running count of how many pages, what the current page is, and the ability to paginate forward and backward through the pages with each request. It is common to send pagination parameters through the query parameters, but some providers prefer handle it through headers (ie. Github).

Organizing Using Hypermedia
Another approach, which usually augments and extends pagination is using common hypermedia formats for messaging. To paginate results, many API providers use hypermedia media types as a message format, because the media types allow for easy linking to paginate results, as well as providing the relevant parameters in the body of the response. Additionally, hypermedia would further allow you to intelligently break down large responses into different collections, beyond just simple pagination. Then use the linking that is native to hypermedia to provide meaningful links with relations to the different collections of potential responses. Allowing API consumers to obtain all, or just the portions of information they are looking for.

Exactly What They Need With Schema Filtering
Another way to break things down, while putting the control into the API consumers hands, is to allow for schema filtering. Providing parameters and other ways for API consumers to dictate which fields they would like to return with each API response. Reducing or expanding the schema that gets returned based upon which fields are selected by the consumer. There are simpler examples of doing this with a fields parameter, all the way to more holistic approaches using query languages like GraphQL that let you provide a schema of which response you want returned via the URL or the parameter of each API request. Providing a very consumer-centric approach to ensuring only what is needed is transmitted via each API call.

Defining Specific Responses Using The Prefer Header
Another lesson known approach is to use the Prefer Header to allow API consumers to request which representation of a resource they would prefer, based upon some pre-determined definitions. Each API request would pass in a value for the Prefer Header, providing definitions like simple, complete, or other variation of response defined by the API provider. Keeping API responses based upon known schema and scopes defined by the API provider, but still allowing API consumers to select from the type of response they would like to receive. Balancing the scope of API responses between both the needs of API providers, and API consumers.

Using Caching To Make Response More Efficient
Beyond breaking up API calls into different types of responses, we can start focusing on the performance of the API, and making sure HTTP caching is used. Keeping API responses as standardized as possible while leveraging CDN, web server, and HTTP to ensure that each response is being cached as much as it makes sense. Ensuring that an API provider is thinking about, measuring, and responding to how frequent or infrequent data and content is changing. Helping make each API response as fast as it possibly can by leverage the web as a transport.

More Efficiency Through Compression
Beyond caching, HTTP Compression can be used to further reduce the surface area of API responses. DEFLATE and GZIP are the two most common approaches to compression, helping make sure API responses are as efficient as they possibly can. Compressing each API call to make sure only the least amount of bytes are transmitted across the wire.

Breaking Things Down With Chunked Responses
Moving beyond breaking things down, and efficiency approaches using the web, we can start looking at different HTTP standards for making the transport of API responses more efficient. Chunked transfers are one way to send API responses in not just a single API response, but break it down into an appropriate number of chunks, and send them in order. API consumers can make a request and receive large volumes of data in separate chunks that are reassembled on the client side. Leveraging HTTP to make sure large amounts of data is able to be received in smaller, more efficient API calls.

Switch To Providing More Streaming Responses
Beyond chunk responses, streaming using HTTP with standards like Server-Sent Events (SSE) can be used to deliver large volumes of data as streams. Allowing volumes of data and content to be streamed in logical series, with API consumers receiving as individual updates in a continuous, long-running HTTP request. Only receiving the data that has been requested, as well as any incremental updates, changes, or other events that have been subscribed to as part of the establishment of streams. Providing a much more real time approach to making sure large amounts of data can be sent as efficiently as possible to API consumers.

Moving Forward With HTTP/2
Everything we’ve discussed until now leverages the HTTP 1.1 standard, but there are also increased efficiencies available with the HTTP/2 release of the standard. HTTP/2 has delivered a number of improvements on caching and streaming, and handling the threading of of multiple messages within a single API request. Allowing for single, or bi-directional API requests and responses to exist simultaneously. When combined with existing approaches to pagination, hypermedia, and query languages, or using serialization formats like Protocol Buffers, further efficiency gains can be realized, while staying within the HTTP realm, but moving forward to use the latest version of the standard.

This is just a summary look at the different ways to help deliver large amounts of data using APIs. Depending on the conversations I have today I may dive in deeper into all of these approaches and provide more examples of how this can be done. I might do some round ups of other stories and tutorials I’ve curated on these subjects, aggregating the advice of other API leaders. I just wanted to spend a few moments thinking through possibilities so I can facilitate some intelligent conversations around the different approaches out there. While also sharing with my readers, helping them understand what is possible when it comes to making large amounts of data available via APIs.

• Read The Blog

API 101

I have a lot of information regarding APIs on my network of sites, but I'm always trying to make sure I have a sufficient amount of 101, or getting started information for people who are just learning what an API is, and what is possible.

I have some information available to help walk you through the concept of an API, and how they have been expanding over the last 15 years across all the online services you have become familiar with on the web and mobile phone.

• Getting Started With APIs

API Lifecycle

These are the 68 steps in a modern API lifecycle I am keeping track of, trying to understand how APIs are being delivered, from define to deprecation.

	Definitions From the simplest definition of what a service does, all the way to the complete OpenAPI definition for each service.
	Design The consistent design of each service leverage existing patterns, as well as the web to deliver each service.
	Versioning Plan for handling changes, and the inevitable evolution of each service, its definitions, and supporting code.
	DNS The domain addressing being used for routing, management and auditing of all service traffic.
	Database The backend database schema, infrastructure, and other relevant information regarding how data is managed.
	Compute How is the underlying compute delivered for each service using containers, serverless, cloud, and on-premise infrastructure.
	Storage What does storage of all heavy objects, media, and other assets involved with the delivery of services.
	Deployment Details services developed, and deployed, including frameworks and other libraries being used.
	Orchestration Defining what the build, syndication, and orchestration of service deployments and integrations look like, and are executed.
	Dependencies Identifying all backend service, code and infrastructure dependencies present for each service.
	Search Strategy for understanding what search will look like for each individual service, and play a role in larger, more comprehensive search across services.
	Proxy What proxies are in place to translate, cache, stream, and make services more efficient and secure?
	Gateway What gateways are in place to defend, protect, route, translate, and act as gatekeeper for service operations?
	Virtualization Details how how services and their underlying data are mocked, virtualized, sandboxed, and made available for non-production usage.
	Authentication What is involved with authentication across all services, including key-based approaches, basic authentication, JWT, and OAuth solutions.
	Management Information about how services are composed, limited, measured, reported upon, and managed consistently across all services.
	Logging What is being logged as part of service operations, and what is required to participate in overall logging strategy?
	Portal The strategy for how all services are required to be published to one or many public, private, and partner developer portals.
	Documentation The requirements for what documentation is expected as part of each service presence, defining what the services delivers.
	Support Relevant support channels, points of contact, and best practices for receiving support as an API consumer.
	Communications Information about the communication strategy around each service, and how blogs, social, and other channels should be leveraged.
	Road Map Details on a services road map, and what the future will hold, providing individual service, as well as larger organizational details on what is next.
	Issues Expectations, communications, and transparency around what the current bugs, issues, and other active problems exist on a platform.
	Change Log Translating the road map, and issues into a log of activity that has occurred for each service, providing a history of the service.
	Monitoring What is required when it comes to monitoring the availability and activity of each individual service.
	Testing The tactical, as well as strategic testing that is involved with each service, ensuring it is meeting service level agreements.
	Performance How is the performance of each service measured and report upon, providing a benchmark for the quality of service.
	Observability What does observability of the service look like, providing transparency and accountability using all of its existing outputs?
	Caching What caching exists at the server, CDN, and DNS layers for each service to provide a higher level of performance?
	Encryption Details regarding what is expected regarding encryption on disk, as well as in transport for each service.
	Security Detailed strategy and processes regarding how each service is secured on a regular basis as part of operations.
	Terms of Service (TOS) Considerations, and legal requirements applied to each service as part of the overall, or individual terms of services.
	Privacy Details regarding the privacy of platform owners, developers, and end users as it pertains to service usage.
	Service Level Agreements (SLA) Details regarding what service levels are required to be met when it comes to partner and public engagements.
	Licensing Information about backend server code, API surface area, data, and client licensing in play.
	Branding What branding requirements are in place for the platforms and its partners when it comes to the service.
	Regulation Information regarding regulations in place that effect service operations, and are required as part of its usage.
	Discovery How are services catalogued and made discoverable, making them accessible to other systems, developers, as well as to internal, partner, or public groups.
	Client Information about clients being used to interface with and work with the service, allowing it to be put to use without code.
	Command Line Interface The command line interface (CLI) tooling being used to developer or consume a service.
	SDKs What software development kits (SDK) are generated, or developed and maintained as part of a service’s operation?
	Plugin What platform plugins are developed and maintained as part of a services operations, allowing it to work with other platforms and services.
	IDE Are there integrated development environment (IDE) integrations, libraries, plugins, or availability considerations for each service?
	Browsers Are there any browser plugins, add-ons, bookmarklets and integrations used as part of each service’s operation?
	Embeddable Information about any embeddable badges, buttons, widgets, and other JavaScript enabled solutions built on top of a service?
	Bots What type of automation and bot implementations are in development or being supported as part of a service’s operation?
	Visualization Are there specific visualizations that exist to help present the resources available within any service?
	Analysis How are logs, APIs, and other aspects of a service being used as part of wider analysis, and analytics strategy?
	Aggregation Are there any aggregation solutions in place that involve the service, and depend on its availability?
	Integration What 3rd party integrations are available for working with a service in existing integration platforms like IFTTT, and Zapier.
	Network Information about networks that are setup, used, and allocated for each service governing how it can be accessed and consumed.
	Regions Which regions does a service operate within, making it available in specific regions, and jurisdictions–also which regions is it not allowed to operate within.
	Webhooks Are there webhooks employed to respond to events that occur via the service, pushing data, and notification externally to consumers?
	Migration What solutions are available for migrating an API between regions, cloud environments, and on-premise?
	Backup What types of backups are in place to bring redundancy to the database, server, storage, and other essential aspects of a service’s operations?
	Real Time Are there Server-Sent Events (SSE), Websocket, Kafka, and other real time aspects of a service’s delivery?
	Voice Are there any voice or speech enablement, integrating services with Alex, Siri, Google Home, or other conversational interface?
	Spreadsheets What types of spreadsheet integrations, connectors, and publishing solutions are available for a service?
	Investment Where do the funds for operating a service come from within a company, from external organizations, or VC funds?
	Monetization What does it cost to operate a service, breaking down the one time and recurring costs involved with delivering the service.
	Plans Outline of plans involved with defining, measuring, reporting, and quantifying value generated around a service’s operation.
	Partners An overview of partner program involved with a service’s operation, and how a wider partner strategy affects a service’s access and consumption.
	Certification Are there certification channels available for applications and developers, defining the knowledge required to competently operate and integrate a service.
	Evangelism What does internal, public, and partner evangelism efforts and requirements look like for a service, and its overall presence?
	Showcase How are developers and applications using a service showcased, and presented to the community, demonstrating the valuable around a service?
	Deprecation What are the plans for deprecating a service, involved the road map and communication around the individual and overall deprecation of service(s).
	Training What training materials need to be developed, or already exist to support the service.
	Governance How are all steps measured, quantified, aggregated, reported upon, and audited as part of a larger quality of service effort for each service.

API Evangelist Weekly Digest

A digest of all the blog posts from here on API Evangelist assembled into a single email newsletter arriving in your inbox each Monday morning.

Email Address *

I am responsible for this newsletter, so anything you need to know if found right here on API Evangelist. ;-)

REST API Notes Newsletter

A carefully curated newsletter from Matthew Reinbold, providing a succinct list of only the most important web API news each week, allowing you to stay more informed of what is happening in the API space in less time.

Email Address *

More information about REST API Notes email newsletter is available via their website.

API Developer Weekly

The API Developer Weekly is a weekly newsletter hyper-focused on the business, design, development, and deployment of APIs for web and mobile apps.

Subscribe to our mailing list

More information about API Developer Weekly is available on their website.

API Evangelist Partners

Get in touch

Please contact me with any specific questions through any of these channels.

	Connect With Me On Email
	Connect With Me On Twitter
	Connect With Me On Github