What is an API?
An API is a digital interface for sharing data, content, and algorithms with web, mobile, and device applications using the Internet, building on web technologies to make digital resources available across many different applications. APIs have been around since computers and their networks first gained a foothold back in the 1960s, but with the rise of the web since 2000, a new breed of APIs have emerged which has changed how we build and use technology, and introduced entirely new ways of doing business, but sadly, they have also introduced entirely new ways of exploiting an destabilizing the physical world around us. APIs aren’t the latest techno solutionism, although they are oftentimes billed as that, they are the digital currents that flow around us each day. APIs power the web and mobile applications we depend on each day, while also steadily working to redefine our physical worlds by connecting everything to the Internet—reshaping our virtual and physical worlds, while also remaking who we are as humans along the way.
Websites Are for Humans
The web as we know it has been evolving for over 50 years, but in the 1990s it became something we were able to access in our homes and businesses, laying the foundation for the ubiquitous web of applications we now access in our homes via laptops, televisions, and other appliances, as well as our cars, in our businesses, and across our communities in the form of security cameras, traffic infrastructure, digital signage and much more. Websites are hypertext markup language (HTML) that are designed to be rendered for humans in a browser, making the text, images, and other media consumable by humans using their eyes and ears, and navigated using our fingers via touch screens, keyboards, and increasingly voice controls. Websites provide us with a user interface that each person can put the web to use in their personal or professional worlds, connecting us with the digital world that has emerged before us, and increasingly throughout our physical world as well.
APIs are for All Other Applications
As the web has expanded over the last twenty years beyond simple web pages accessed via desktop computers, the builders of the web realized they needed to develop ways in which data, content, and algorithms could be reused across many different web and mobile devices. To accomplish this, developers began adapting web technologies using HTTP to make data, content, and algorithms available in much more standardized and machine readable ways so that the data and content could be used across many destinations. Separating the user interface from the data, content, media, or algorithms, widening how and where the digital resources could be applied, using them to power their primary web, mobile, and device applications, but also making the increasingly valuable digital resources available to partners and 3rd party developers, generating entirely new revenue streams from data, content, media, and algorithms which had limited value before. Resulting in a layer of application programming interfaces (APIs) on top of the existing low cost web infrastructure companies were already using to deliver their existing web applications.
History of API - Commerce
The early web API pioneers were all interested in expanding the reach of their commerce networks, making their products and services available to affiliates, resellers, partners, and 3rd party developers looking to build the next killer application. Salesforce, eBay, and Amazon were the first technology companies who began investing in API infrastructure in the early 2000s, setting into motion a whole new way to make digital resources available via the web. Translating the sale of physical products and services from our physical worlds into the online world were the most straightforward and widely understood way to apply the web to reach new customers, but it also quickly translated to more distributed approaches to selling products and services through many different applications and digital networks via simple web APIs. By 2005 a significant portion of Salesforce, eBay, and Amazon’s revenues were beginning to flow through their APIs, laying the commercial foundation for what many folks like to call the API economy, but many who were tuned into this shift would soon realize there were some other critical ingredients needed to get us where we are today.
History of API - Social
As the commerce phase of the API evolution was being set into motion another aspect of the API economy was taking shape, leveraging APIs to not just sell products and services, but to further define how humans use the web to stay connected and get their information, showing the social value of APIs. In 2003 the image sharing platform Flickr realized they could rapidly expand the reach of their platform to the growing blogosphere by offering 3rd party developers API access to their increasingly valuable media platform—realizing this was much more than just a technical advancement, but also a business one they dubbed business development 2.0. Shortly after Flickr realized the potential of APIs the world saw both Twitter and Facebook introduce two API-driven seismic shifts that we are still working to understand the impacts of in 2020. Much of the growth of Twitter and Facebook over the last fifteen years has been built on top of their public APIs, providing a self-service way for partners and 3rd party developers to extend the reach of the network, helping build buttons, badges, widgets web, mobile, bots, and device applications that would extend, augment, and strengthen the importance of these social networks, and as we’ve seen in the last decade also the automation, orchestration, and introduction of bots who continue to define the tone and balance of these online platforms.
History of API - Cloud
While the social phase of the API evolution was ramping up, one of the early API pioneers forever changed how we’d deploy our software infrastructure using APIs, with the introduction of what we now call “the cloud” from Amazon Web Services. After finding success with APIs to support their commercial ambitions Amazon had internalized APIs in a way that would demonstrate how you could use web APIs to do more than sell products, blog, and share posts with your friends on a social network. With the introduction of AWS S3 and EC2 in 2006, Amazon showed us that you could also deploy global infrastructure using APIs, opening the floodgates of the types of digital resources you could make available through web APIs. It wasn’t just data, content, and media anymore. You could use web APIs to deploy servers, store previously unheard of amounts of data, as well as configure the DNS and network for your applications in real time. This API-driven cloud gave us the modular resources we would need to build applications for the future, but it also gave us the scale and scope we needed to deliver the distributed applications we would need to do business in the digital age. Adding another important dimension to the API conversation that would make the next seismic shift possible, moving the Internet from servers in data centers into the cloud, and then into our pockets with mobile applications.
History of API - Mobile
With the introduction of the iPhone in 2007 and the Android mobile phone shortly after, a whole new world of applications emerged. Moving applications from our desktops and laptops into our hands, mobilizing common applications and platforms we used so that we could take them with us, expanding the reach of the web to wherever we could get a cellular signal. Developers quickly realize that they could deliver the same data, content, media, and algorithms they used in their web applications to the new mobile applications they were delivering. Realizing that they could deliver digital resources once using APIs then reuse over and over in many different web and mobile applications, while also making them available for use by trusted partners, and 3rd party developers when it m made sense. Using web APIs to deliver the digital resources needed to power mobile applications allowed application providers to use the same low cost web infrastructure they used for their websites to deliver mobile applications, helping reduce overhead and become more agile and nimble when it comes to delivering new applications, but also iterate upon each version of existing applications. The table for the API economy was being set, and APIs were becoming the preferred approach developers needed to deliver the digital resources used in web and mobile applications, but soon would expand to anything that could be connected to the Internet, pushing APIs beyond the digital world, lighting up common everyday physical objects with the ability to read and write data via wireless networks.
History of API - Devices
APIs demonstrated that data could provide a rich experience via mobile devices, and it didn’t take too long for developers to push these capabilities to a new breed of Internet connected devices, giving birth to a new wave of wearable devices, home automation appliances, digital signage, and the connecting of everyday objects from our personal and business lives to the Internet. Wearable activity trackers, home cameras, thermostats, television, and other Internet connected devices are all using APIs to send and receive data, establishing entirely new sources of rich and increasingly valuable data from entirely new digitally enabled objects. Giving birth to an Internet of things that would produce entirely new markets for home, surveillance, environmental, agriculture, and other emerging API-driven device data. Devices were telling us the temperature in our homes or the wind speeds on the farm, helping us measure, track, and understand our physical world using Internet connected devices that send data back and forth using expanding wireless networks that were covering every corner of our physical world. APIs weren’t just for expanding your affiliate networks to sell more products, or to publish an image to your blog. You could now deploy global infrastructure to support the needs of humans via mobile applications as well as a growing number of automated connected devices that were being used to further blur the line between the online and offline worlds.
Every Online Shift In Last Twenty Years Have APIs Behind It
Every major online shift of the last twenty years have had APIs behind it. E-commerce, social networks, the cloud, mobile phones, and the Internet of things has changed every business sector, and has forever changed how we live our lives. All of these shifts have contributed to making things like ridesharing and food delivery something we now take for granted. APIs have changed how we buy things, how we communicate and connect online and offline. APIs are transforming every existing business sector, while also creating entirely new business sectors, disrupting many aspects of our lives from how we date, to how we vote in an election. APIs allow for the dismantling of what was, and reassembling it into entirely new digital assets that can be bought, sold, and turn people’s everyday behaviors and activities into something that also can be bought and sold online. While the cloud and their APIs exist just beyond the view of the average consumer or citizen, the commercial, social, mobility, and device enablement is front and center for everyone to help quantify the impact APIs are having in their lives. Each of these seismic shifts introduced by APIs are large by themselves, but it really is the collective momentum they all bring to the table together. Mobile and device applications would be much more difficult to deliver without the cloud, and many of these new types of applications become more of the natural flow of our live because of the social connectivity woven into them. All working together to shift more of our everyday lives online so that they can be quantified, tracked, monetized, and all contributing to a real or perceived forward motion enabled by the Internet.
APIs Are What Makes a Platform
APIs are the Internet enabled pipes behind each of the major platforms dominating the business and political landscape today. Amazon, Facebook, Twitter, Uber, Netflix, Instagram, WhatsApp, Pinterest, Salesforce, Expedia, and other top brands are using APIs to power their businesses and their growth. The public APIs are just the tip of the iceberg when it comes to APIs on their platforms, and for every public API from these companies publish, there are probably 100 others internally and providing partner access to valuable digital assets. While the growth of these platforms over the last twenty years have been driven by their public APIs, internal APIs are also a critical aspect of how their business operates. Enterprise capabilities are increasingly being measured by the quantity and adoption of internal APIs. While these brands have emerged as part of the last wave of Internet enabled businesses over the last twenty years, a growing number of more traditional businesses are in the process of also redefining themselves as platforms. Remaking businesses that have been around for fifty or a hundred years, breaking down operations into small API defined bits, then making them available across all of the enterprise applications where they are needed to conduct businesses on any given day. APIs are what turn a web application into a platform, making an application or suite of applications much bigger than the sums of its parts by ensuring digital capabilities are available across many different channels and domains.
Delivery Digital Resources to Web, Mobile, and Device
Most businesses are doing APIs, they just were not doing them in any organized way by 2010, but the introduction of mobile increased the number of domains in which companies were having to deliver data, content, media, and other resources to power a growing number of applications. Web applications were also growing much more complex by 2010, but it was the need to support one or more mobile applications that really pushed APIs into the foreground. Businesses needed to get more organized about not just how they delivered data, content, media, and algorithms to the growing number of applications, but they also needed to get more organized about how they were creating, storing, and making sense of the growing amount of data being generated from all of these new endpoints. Then alongside all of this happening companies were getting asked by partners for access to the same digital resources and data exhaust, while also facing pressure from new players who were being much more public with their API-driven performance, further leveling up the urgency when it comes to doing APIs, but also ensuring they are as performant and easy to put to use as they possibly can internally and externally.
Delivering Digital Resources to Partners & 3rd Party Developers
As the value of data, content, media, and algorithms grows on any platform, the demand for those digital resources increases. As popular platforms like Salesforce, Flickr, Twitter, Facebook, Google Maps, and others grew, the demand for access to their valuable resources by partners and 3rd party developers increased dramatically. This wasn’t exclusive to these platforms. Other companies who were finding success with their web, mobile, and device applications were facing the same challenges. As platforms grew and expanded, so did the demand for access to those resources in a standardized, and oftentimes developers demanded it in a self-service way. Companies needed to meet their own needs for delivering web, mobile, and devices applications, but also the needs of partners and developers who were looking to build the next generation of applications. Amidst all of these real world demand on business to deliver and manage data, content, media, and algorithms across multiple channels, and their desire to satisfy the needs of partners and 3rd party developers, companies are having to get more strategic and organized in how they manage their own digital resources, resulting in many benefits, as well as a whole industry of services and tooling that help companies, organizations, institutions, and government agencies manage their API infrastructure.
Organizational Efficiency, Agility, Reuse, Traceability, and Provenance
There are many more reasons enterprise organizations do APIs, but over the years we have seen some consistent benefits that organizations realize when it comes to operating their businesses using APIs to engage with partners, customers, and the public. Being able to quickly and efficiently discover and apply digital resources where they are needed bring a certain agility to teams when it comes to being able to deliver new web, mobile, and device applications, and more predictably evolve existing ones to meet the changing needs of a platform. Having an organization's digital capabilities defined as APIs, helps improve efficiency and the agility of an organization, but it also allows for the reuse of digital resources, which allows for better traceability, provenance, and management of dependencies across the applications and integrations that depend on them. APIs provide more visibility into who is using data, content, media, and algorithms, by establishing a real-time feedback loop between the backend systems, and the applications putting resources to work, or generating data, content, and media from platform users. APIs aren’t just about developing web, mobile, and device applications, they are about managing the digital resources applied across applications in a more organized way, while developing more awareness of what is actually going on under the hood of a platform and the organization behind.
Recapping the API Fundamentals
To help illustrate what an API is, let’s look at a simple products SaaS platform, where our digital resources are products that we want to sell via an e-commerce platform. To access our products via the web you would simply go to https://example.com/products/ in your browser and you would see a list of products rendered using HTML. Our products API will use a similar URL pattern ofhttps://api.example.com/products, but instead of returning HTML for rendering in a browser, it returns either XML, JSON, or a CSV. Allowing the product catalog to be used to generate HTML, or be rendered into any other application, or integrated into other systems for other uses. The HTML product listing can be easily used in browsers, the CSV product listing can easily be used in a spreadsheet, and the XML and JSON can easily be applied by developers into a wide variety of use cases. This approach can be applied to any digital resource, not just products. For the purposes of this discussion we are focusing on our resource being a product, but this could just as easily be a task list, videos, images, payments, or any other resources that are being made available online. APIs allow us to consistently define a variety of digital resources in a consistent way that leverages HTTP to make data, content, media, and algorithms available using the same low cost infrastructure that powers the web. Early API providers realized quickly that they were going to need their resources available in more than one place on the web.d After building out large web sites or applications, they created affiliate and reseller networks, or were looking to cultivate, grow, and benefit from growing developer communities who were using their APIs in their applications. As early API providers realized the potential of APIs, and continued their investment in developing them they quickly realized that they were going to have to get a little more organized about how they managed their API infrastructure if they were to maximize value within their developer communities, and across their platforms.
The Emergence of API Management in 2006
As APIs were getting started as part of the commerce, social, cloud, and mobile phases of evolution a new approach to managing APIs emerged to help API providers do better APIs. Companies like Mashery, Apigee, and 3Scale emerged to provide a standardized suite of tooling to help secure, manage, and develop an awareness of how APIs were being put to work. By 2012 API management was a viable segment of the tech sector, and by 2016, after some consolidation, API management had become baked into the cloud, and is something that is available as part of AWS, Azure, Google, and other cloud platforms. There are a number of features provided by API management platforms that help API providers better operate their APIs, including securing, rate limiting, and reporting upon API usage, but more importantly API management is about defining and generating value being generated using APIs. API management when used properly helps API providers develop more awareness about how APIs are being used, but also being able to properly manage the value exchanged between a platform and the applications and integrations that exist on top of a platform. Making API management a pretty significant part of the overall conversation when it comes to what is happening across an organizations applications, as well as engagements with partners and the public.
API Management Requires Everyone to Have an API Key
One of the tenets of API management is that they help API Providers secure all of their APIs, requiring all consumers signup to receive a key or token that they need to pass in with each API call they make. Providing a single token that is present in all API requests and responses which is attributable to each API consumer. Allowing access to all digital resources be tracked, rate limited, reported upon, and used to better understand what consumers are doing with the valuable data, content, media, and algorithms being accessed via APIs, while efficiently tracking all data, content, and media being generated by consumers across web, mobile, and device applications. API keys control access to APIs, but this key also acts as a fingerprint on every action a user makes via a platform, allowing their usage to be understood on an individual consumer level, or wider as part of personas, groups, and larger subsections of consumers who are putting APIs to work. API keys have become a ubiquitous way for API providers to providers to define who has access to resources, and how they are putting them to work across applications, but they have also become commonplace for API consumers, and are an expected part on the API onboarding process, when the APIs and the underlying API management layer offers value to consumers.
Giving End Users a Voice with OAuth and Access Scope
Most APIs only require partners and 3rd party developers to signup for access and receive their API keys before they can access API resources, but once they have keys they can usually obtain access to most resources. However, there is an additional layer of security that some API providers employ when there is sensitive platform user data being accessed via an API. Over the last decade OAuth has emerged as a standard for providing not just a set of tokens for accessing an API, but also the scope of usage that is required when it comes to API access to data, content, and media. Facebook, Twitter, Salesforce, and other common API-driven platforms use OAuth to govern access.giviing their end users a voice when it comes to opening up 3rd party access to their data via a platform. Allowing them to authorize the issuing of keys to a 3rd party developer to be able to access their account data, content, and media. While the APIs, and the scope of access are defined by the platform, the end users has a say in who is able to access their data via the platform, initiating what is considered to be the “OAuth dance” to give access to an external platform or application. While imperfect, OAuth provides the best solution we have for brokering the access to valuable platform data by 3rd party developers, while keeping end users in the equation--helping automated and manage the access of valuable resources using publicly accessible web APIs.
Defining the Business of APIs Using Service Plans and API Composition
Since the beginning, API management has provided the ability to group APIs into different access plans, composing exactly the right mix of API defined digital resources and rate limits for consumers. You see this evolution branded on the approaches of software as a service (SaaS) solutions with their free, pro, team, enterprise, and other mix of self-service or direct engagement pricing plans and tiers. API management allows you to put one or many APIs into a plan, put usage limitations on the plan, and then apply it to one or many users, allowing API providers to compose plans for each type of API consumer persona, often providing free public tiers, but then also crafting additional plans for other levels of customers, partners, or even for internal usage across different groups and teams. In my experience all APIs should only be accessed via a well defined, metered, and reported upon usage plan, requiring all developers whether internal, partner, or public 3rd party to obtain API keys that are associated with a well defined API access plan. No API should go without being measured as part of a plan, and no API consumer should be accessing digital resources across an organization without having a key. There can be limited use free and public access to some resources without signing up, but once you need access to more resources, and higher volumes of each resource, history has shown us that you want to have visibility into what digital resources are being made available and who has access, and are putting these resources to work. Without this visibility into how we are putting API resources to work across web, mobile, and device applications, it becomes increasingly difficult to secure and define the value being delivered.
The Role HTTP Methods LIke GET, POST, PUT, DELETE Play
Each page we load on the web uses what is called an HTTP GET—the technical standard for retrieving an HTML page. GET is the most common HTTP method used today, which is all about consumption of data, content, and media. However, there are numerous other HTTP methods available which allow for not just retrieving, but creating, updating, deleting, and other verbs we may need to use in web and API design. The second most common HTTP verb is POST, which is what we use to create a digital resource. We POST to the web and via APIs all the time. We POST contact forms. We POST Tweets. We POST our location. APIs are really good at using HTTP methods, and employ POST (Create), GET (Read), PUT (Update), and DELETE (Delete) to manage the state of data, content, media, and algorithms using the web. When you require that all API consumers must possess a key to access APIs, this ensures that all consumption can be attributed back to each individual consumer, opening up the ability to report on usage by HTTP method, and see and understanding who your consumers are, and who your creators are. Providing one of many dimensions opened up my using API management, but one that tends to reflect a very meaningful to determining who is creating value and who is just consuming.
API Awareness and Reporting from API Management
Layered on top of the authentication, service composition, and rate limiting, API management solutons provide a suite of reporting for API providers to tune into what is happening across the APIs they are serving up. Reporting upon multiple dimensions of consumption, quantifies what each user is accessing, but also breaks things down by the API resource, HTTP method, errors, time frames, geographic region, and many other dimensions that help API providers develop an awareness of what resources are getting used and how they are applied in applications. This awareness is the biggest benefit of doing API management because it makes the API landscape across an organization more observable, helping providers make sense of the growing sprawl that is enterprise API infrastructure. This awareness isn’t just at the API provider side of the conversation, and most API management solutions allow API providers to share reporting with API consumers as well, helping them understand their own usage across their applications and integrations. Helping provide dashboards for API providers and API consumers to see API usage, visualizing the very digital and abstract API layer that exists behind everything we do online each day. Ensure we are in tune with the API infrastructure we depend on each day, and are able to quantify the value that is being generated each day.
Invoicing for Value, Usage and Consumption via API Management
The final feature of modern API management solutions that help define the value being generated by APIs each day is the ability to invoice each API consumer for their usage each month, charging them varying rates based upon their API plan, and how much they have used. As the types of API resources being made available have expanded over the last twenty years, the approaches to pricing, invoicing, and generating revenue from these API resources have expanded as well. While many API providers charge based upon the number of API calls made, many other providers have pushed the boundaries of API monetization to include per megabyte transferred or stored, by timeframe, or other dimension of each resource being served up. The ability to develop and evolve different API plans, meter, report, and then invoice upon how API consumers access APIs has created entirely new types of business and revenue streams from existing businesses. Think about what Amazon has done with the cloud by making servers, storage, databases, and other infrastructure available as a pay for what you use model. APIs have created entirely new ways to generate revenue from social media and other user data or user generated content, with providers like Twitter giving away low volume data access for free, but then charging high volume data consumers based upon what they consume. Invoicing for API usage isn’t just about charging for API usage, and can be used to signal value exchanged, without actually exchanging money, or even be used to pay those who contribute, while charging those who just consume.
Considering the API Management Fundamentals
API management is software you stand up in front of your APIs to secure access to your API resources, meter, measure, and report upon their usage. You do this with what is called a gateway, proxy, or sometimes a connector. API management has evolved to include other elements over the years, but continues to revlve around a portal where publish your APIs, a registration and login flow for developers to sign up when accessing a service, then providing them with the keys and sometimes code they need to actually consume an API. All API consumption is measured, rate limited, and reporting upon in real-time, as well as according to designated billing and subscription cycles. API management stands in between an API provider's digital resources and an API consumers desktop, web, or mobile application and integration. We can stand up an API management solution in front of our products API we showcased earlier, requiring all developers register and obtain keys before they can use, select an access tier, and in some cases put in a credit card to be billed for what they use. API management secures digital resources from unwanted usage, defining the acceptable limits of value being exchanged, rewarding desired behavior, and blocking or shutting off undesired behavior. Leading API providers have mastered the API management layer to benefit their bottom line, and drive the lion share of the value inward towards their companies. Attracting developers with valuable digital resources, but making sure the platform wins when it comes to generating value and revenue. Striking a balance between making digital resources available to exactly the audience you wish, protecting yourself from unwanted access and usage, while also developing insight and awareness about how consumers are putting resources to work.
The Beginning of an Open Data Movement
As the API economy was getting its footing with commerce, social, cloud, and mobile, another movement was occurring that was focused entirely on making public data available from city, county, state, and federal government agencies as well as non-government organizations, non-profits, and other institutions. Around 2010 you began to see more emphasis on publishing data on the web as compressed downloads, or as XML, JSON, CSV, and other machine readable formats. This movement was born out of the tech industry in the San Francisco Bay Area, but quickly spread to cities around the United States, with similar movements emerging in Europe, Australia, and around the globe. Government agencies were looking to stimulate innovation across the private sector with this data, and the tech sector was looking to build the next killer app on the data being made publicly available. By 2013, governments in both the US and UK were mandating that government agencies publish data, and in 2020, while there has been many ups and downs in the open data movement, many public entities are still publishing data online as download and via APIs.
Considering Public Data Download Versus API Consumption
Even before the popular open data movement was picking up speed by 2010, making data available as a single large, or series of large downloaded compressed files was common. When data was available as smaller datasets, they might also be made available as downloadable machine readable files. Along the way, some data stewards who were aware of the emerging world of APIs were also making data available as simple web APIs, returning XML or JSON responses, adding a more advanced query layer. Downloading data as files proved to be effective for developers who have the resources to download and process them, but the process remained disconnected from the stewards of the data, and rarely possessed any kind of feedback loop that would help improve, evolve, and move data forward to meet the needs of consumers—it was a one way street. Most agencies, organizations, and institutions publishing data were unaware of the wider API movement, and the benefits of establishing a connection with consumers. Data consumers with the most technical resources tended to also be the most vocal about keeping things as they were, pushing back on the need for APIs because they had the resources to download and process the data, and didn’t want any friction when it came to accessing the growing wealth of public data resources that were being made available.
Government Mandating Machine Readable by Default
In 2013, then President of the United States Barack Obama issued a directive that top level cabinet agencies In the United States would have to go machine readable by default when it came to publishing data publicly. Instead of publishing data as spreadsheets and PDFs, they needed to publish it as XML, JSON, or CSV. This would have a significant effect on federal agencies, but also trickling down to states and cities, as well as helping lead when it comes to other countries around the world. Demonstrating that by making data available in machine readable ways you would be able maximize the value and usage of the data in both the public and private sector. Making data available in standardized machine readable formats would extend the reach of research, markets, and the many other efforts underway in the public sector. Leveraging the web to make data available for use in web, mobile, and device applications, helping lead the way when it comes to ensuring the web would continue to be an engine for the economy and society. Using the public sector to drive innovation in the private sector, which would then in turn fuel the growth and impact of government through a healthy economy and a growth in tax revenue. Helping make all levels of government machines readable by default so that data can be easily shared between systems, stakeholders, and constituents across the web, mobile, and device applications they depend upon.
Realizing Open Data and Open APIs Are Open for Business
When people speak of open data they most often refer to open being about access. Meaning it is openly available on the web for anyone to download. While this is the intent of most open data providers, it isn’t the intent of many open data consumers. Many wanted it open so that they could freely use these valuable public resources for use in their commercial applications, allowing them to tap this new data resource like it was a mineral, forest, or oil and gas natural resource that had become available in this growing public sector digital landscape. Open data meant open for business when it came to the digital economy, and the growing number of entrepreneurs who needed the public sector to power their apps, providing the life blood of the tech sector, continuing to build on fifty years of public sector investment in the ever growing private tech sector. This arrangement has always benefited the technology sector, but as the value of data increases, the appetite for public data only grew. Resulting in a vocal minority that is extremely passionate about public data being made available, but rarely motivated to contribute back, pay for data, or have anything getting in their way when it comes to accessing the digital resources they had developed an appetite for. Which has had a chilling effect on the overall growth and velocity of the public data movement, actually working against private sector innovation, but is something that the tech sector isn’t always open to seeing and understanding. There has been a lot of money made with commercial applications of public data, but because there isn’t a feedback loop in place, little value ends up back at the source, again operating as a one way street.
Considering Commercial Usage of Public Data With Google Maps
One of the most significant examples of the value extraction that occurs on public data resources can be found with Google Maps. The ubiquitous mapping solution is built on public data, and continues to harvest public data resources to fuel the commerce solution without giving much back beyond "free" usage of the service. Google Maps depends on federal, state, and municipal data to operate, dominating the conversation around travel, transit, neighborhoods, businesses, and many other dimensions of the public sphere. After over decade of growth, Google Maps is a juggernaut, making it something that government agencies, institutions, and organizations often need, even if they simultaneously are feeding data to the giant Google Maps machine. Real time transit feeds, census, business registry, and other public data is essential to the Google Maps machine continuing to operate and grow to cover the entire world at the global level, but all the way down to the neighborhoods within our communities. The often cash strapped agencies and organizations that provide Google with public data are not given any support from Google, a commercial entity, even if continuing to provide the data requires significant resources. It makes sense for public data to remain freely available, but when you have the resources Google does it doesn’t make sense for things to be such a one way street, with all the value being generated for the public sector flowing into and enriching Google. Public sector data should provide a solid foundation for private sector tech innovation, and help stabilize the data life blood of the leading platforms and applications, but this shouldn’t just be flowing one way. Yes, taxes play a significant role in funding the creation of public data, but in the current state of corporate taxes this isn’t a fair argument, and with public data being a living, evolving, and expanding landscape, there needs to be some value going back into the public sphere for it all to keep growing and expanding.
API Realizations From the United States Census Bureau
As long as the United States Census Bureau has been conducting its surveys of the American population it has also been making the data from it available to other government agencies, institutions, organizations, and commercial enterprise entities. In 2011 when I asked the Census Bruearu why they don’t have an API for the massive amount of data it possesses, they simply responded that they are meant to be neutral stewards of the survey data and they couldn’t place any of their own opinions on the public data, something that applying API design might very well do. However, after launching their API shortly after they realized that there was a whole other class of consumers that existed, ones who had lacked the resources to download and process the massive data survey files, and were not able to get at exactly the slice of the census data pie that they needed. With the new web APIs, spreadsheet connectors, and other API driven census survey resources, the Census Bureau was able to reach an entirely new audience. More importantly they were able to establish a feedback loop with consumers that did not exist before with consumers who simply downloaded the large data files for the expansive surveys. When asked if they knew of the interesting projects like the Google Flu Project and other applications that were built on census data, the US Census Bureau said they had no visibility into how the data was being used—something that changed with the introduction of a feedback loop around the current Census APIs. The US Census Bureau had realized the awareness making data available as web APIs introduces, bringing consumers closer, and planting the seeds for more value exchange to be flowing both ways, from the Census Bureau to consumers of the survey data, and then back to the Census Bureau in the form of discussion—eventually, with more investment it could be something significantly more than just feedback from developers.
Managing Public Data Like We Manage Public Lands
In the United States we enjoy an amazing patchwork of public lands and parks that are owned by all citizens and managed by our Department of Interior. These lands are available to every citizen to use. While some are free to use, most require parking and other fees to take advantage of the trails, roads, parking lots, and other ways we engage with public parks and lands. While public lands are available to every citizen, if you are McDonalds and you want to begin selling burgers at the Grand Canyon you have to obtain permits and even share revenue with the federal government. Why are digital public assets different than physical public assets? Public lands require ongoing maintenance even though they already exist. Public data requires maintenance and further investment to keep fresh, alive, and evolving. While Google may pay more taxes than the average citizen, it doesn’t pay more into the government to fund public data than the value it extracts, comparatively to the value extracted by individual citizens. Current views on public data from the tech sector are exploitative and extractive, and do not respect or give back to the valuable digital commons that are emerging. Similar to the reasons why public lands exist in the first place, we need to protect the value that exists within public data resources. There should always be a commercial aspect of why public data exists in the first place, but like public lands it has to be a managed revenue sharing relationship that allows for a shared value expanson between the public and private sector. Times have changed, and it is time that we begin to look at our public data differently, otherwise like our environment we might find us in an unsustainable state.
Stepping Back to Look At The Public API Management Fundamentals
Shifting gears beyond our example products API, helping illustrate the potential of APIs coupled with API management in the public sector, let’s consider how a 311 API can be offered by a municipality or other civic organization, providing programmatic access to non-emergency incidents that occur across a city and the neighborhoods within. Our digital resource in this case is a 311 service request, providing access to a list of requests that have been made via official municipal 311 channels (ie. Phone or Web). Now that we have a published 311 API providing a listing of service requests, we can also stand up an API management solution in front of it, not because we are looking to make a profit, but because we are looking to secure resources, and maximize the value exchange that is going on between municipal API providers and the different types of API consumers that would be interested in the data. We will still offer a limit on free public API calls by IP address, limiting to 100 per day, then requiring users who want more to signup and get access to a free tier that is rate limited at a much higher rate of 1000 per day. We will also then establish higher tiers of access that require API consumers pay for access to APIs beyond 1000 requests per day. Also providing some additional API endpoints providing access to historical data, multi-city sources, or other valuable dimensions for companies and researchers looking to understand what is happening within communities at scale. There is a wealth of knowledge within the 311 service requests across communities, telling stories of class, race, corruption, and what people are needing to achieve a certain quality of life. This data should be available across every city, and the stewards of this valuable data should be able to serve their constituents with free access to the data, but then also generate much needed revenue from commercial providers who are looking to make a buck off of public data. Modern approaches to delivering APIs and API management solutions provide us with the mechanisms we need to make public data available, but also protect the value that it possesses. Ironically, using the same mechanisms that commercial providers are using to protect their own digital resources, and maximize value generation occurring via their own platform.
In 2010 It Was All Tech Leaders Doing APIs
In 2010, when I first started studying the API sector full time I was studying leading API providers like Salesforce, Amazon, Twitter, and Facebook. Enterprise organizations were still doing the much more rigid version of APIs called service oriented architecture (SOA), where the next generation tech companies were beginning to embrace the web and leveraging HTTP APIs to power their platforms. But then, “API as a product” companies like Twilio, Stripe, and others had emerged to show the potential of generating entirely new types of revenue from APIs. These are the companies in which the world of APIs were cultivated from, providing the first looks at how APIs can be used to make not just data available via simple web APIs, but how you could transform existing digital objects into small bite size digital commodities that can be exchanged via HTTP, at a pricing model that was affordable to a wide audience. These are just a handful of the companies who showed that APIs could be used to connect everyday physical objects to the web, helping us better measure weather, water quality, our health and activity. However it would still be a few years before this new formula for defining digital services and making them available to the market would break out of the tech echo chamber, and be something that people would be talking about beyond Silicon Valley. The early days were all about defining a blueprint that was simple enough that others could follow, refining, distilling, and standardizing how we define our digital assets, then securely and cost effectively make them available to the masses.
By 2016 APIs Were Going Mainstream Across All Business Sectors
By 2016, I wasn’t just studying and talking to the tech sector elite when it came to doing APIs. I was talking to the Capital One, Mutual of Omaha, Ford Motor Company, Center for Medicaid and Medicare (CMS), and other public and private sector enterprise entities. The mainstream had woken up to the need for doing web APIs, and doing them well across a large enterprise organization. I do not spend my days convincing people that they should be doing APIs, everyone is doing them. I spend my days educating people about what APIs are, and what the benefits of doing them well are. In a digital world all types of businesses are being forced to become a technology company, and are required to redefine who they are in a market that operates entirely on the web. APIs are a ubiquitous aspect of doing business today, but unfortunately many still don’t fully see the API pipes used beneath everything we do each day. Many developers who are building web, mobile, and device applications with APIs do not fully see and understand what APIs are. APIs have gone mainstream, but there is still a lot of work to be done when it comes to helping companies, organizations, institutions, and government agencies understand how to do APIs simply and consistently, making them something everyone can access and put to work as part of business operations, or within your own personal realm to take more control over your own professional life.
In 2020 APIs Are Making a Mark on Everyone’s Life
In 2020, everyone is being impacted by APIs. APIs power all those apps on our phones, connecting our homes and cars, and are perpetually automating and connecting the world around us. APIs impact our world at the local level by helping us order food for delivery, all the way up to influencing how we vote in an election. APIs are impacting everyone’s life and defining who we are in this new digital world, bit by bit, but also increasingly defining who we are in the physical world. APIs are much like our financial system, where not everyone needs to understand how the entire banking system works, but everyone should have a handle on where their personal data, content, and media lives, and who has access to, much in the same way we have a handle on where our cash, credit, and debt lives, and who has access to this financial layer of our world. Our personal and professional lives are increasingly defined by APIs, and the more awareness of APIs we have, the more control we will have over creating, managing, moving around, and even deleting our digital bits when we want. With an awareness of this layer of our life we will posses more control over who we are online and offline, without it, we are open to exploitation, abuse, and influencing by the myriad of people and corporations who are wielding power online today—giving over a piece of who we are to an increasingly minority group of tech elite.
APIs Are Behind Every Application We Are Using
Every application on our desktops, laptops, and mobile phones use APIs to communicate with the platform behind them. If you proxy your laptop or mobile phone with a piece of software or hardware that records every bit of traffic coming and going, you will see a steady stream of API calls being made behind the scenes. Hundreds or thousands of API calls are made each day with or without you consciously triggering each request. Once you turn on software like Charles Proxy you will see a steady stream of API calls being made from each application even when you aren’t doing anything. Chatting with their platforms, syncing information, pinging home, and doing a variety of other tasks you had no idea were even occurring. Then once you actually begin to surf a variety of web pages you’ll see multiple API calls occurring for each page you visit, calling content and advertising APIs, populating widgets and other elements on the screen. Web and mobile applications behave the same way, using APIs to publish content, and submit your messages, upload your images and videos, allowing you to do what you do on the web each day. It is no exaggeration to say that APIs are everywhere in 2020. Parking meters, gas pumps, cash registers, signs, surveillance cameras, and other common objects are connected to the Internet using APIs, and while everyone doesn’t have to understand how it all works, there should be at least enough awareness that APIs exist to be able to understand that our data and personal information is being shared.
All Of The Top Brands You Know Are Using APIs
If you look down the list of Fortune 500 companies you will see all of the companies working to redefine themselves using APIs in 2020. Starbucks, Nike, Ford, McDonalds, AT&T, Fedex, and others are actively using APIs to deliver the web and mobile apps they are using to expand the reach of their businesses. Some are even embracing public API models, allowing 3rd party developers and trusted partners to take data, content, and media and build entirely new applications with the valuable resources—syndicating, extending, and franchising their valuable digital resources. Top brands and platforms are using APIs to define their businesses, but they are also using them to define their competition and customers. APIs are how we define, track, measure, and reach our customers in a digital world. Management, automation, and orchestration of advertising and marketing is becoming commonplace within any business, demonstrating another evolving API landscape across enterprise organizations that may not immediately reflect internal IT APIs, external public APIs, but are actively defining brands trying to compete online, and the markets in which they are battling for. APIs are how brands are iterating on proven business models lie marketing, advertising, communication, lobbying, finances, and other bedrock aspects of how we do business, pushing them to reflect a much more fast moving stream of ongoing digital transformations that brands must keep up with in order to stay relevant.
With Many Online Platform We Are The Customers
APIs helped usher in an entirely new way to sell software called software as a service (SaaS), defined by their tiered, pay for what you use pricing plans, which were just a reflection of the API management defined pricing plans we defined earlier. All of this expansion increased the number and types of software services available across the landscape while widening the number of services we subscribe to as part of our regular business operations or personal experience. All SaaS companies are using APIs to deliver their services, and a majority of them offer up partner and 3rd party access to those APIs, with APIs also defining the ability for the many different services you use to work with each other via API integration. In this new SaaS economy we are the customer. We are directly sold a service plan that we put in our credit card, and we get the level of service we paid for, subscribing to it for an agreed upon period of time. We are the customer in this SaaS driven world. APIs are defining the underlying resources being delivered, allowing us to access the CRM, document, image, video, or other service as a subscription through a useful web and mobile application, while ensuring they integrate intelligently with each other using their developer APIs. Shifting forever how software gets built, sold, maintained, and evolved, allowing us to subscribe to the services we desire, and effectively manage our digital resources across many different platforms.
With Some Online Platforms We Are The Product
If a platform is offering their services for free via a web or mobile application and via an API, it is likely that instead of being the customer, end users are actually the product. Platforms are increasingly using web and mobile applications to GET, POST, PUT, and DELETE the digital bits of their end users, while then leveraging APIs to make the user generated content, profile, location, and other data available to partners and 3rd party developers. On these free API-driven platforms, the end users are the product. They are what is being harvested and sold. Keeping platform users on a digital hamster wheel, generating valuable digital resources that can then be organized, enriched, and then sold and used for a variety of applicaiton. There are many ways in which our personal data and online activity is organized, aggregated, and packaged up for sale to the highest bidder online. APIs play a significant role in helping define this landscape ranging from powerful advertising APIs on Facebook, Twitter, and Google, to these 3rd party APIs that allow for surveilling, defining, and targeting platform users across the web, and increasingly our physical worlds. All by oneself our data isn’t worth a whole lot, but in aggregate, and over time, this value becomes very significant to defining emerging markets and the next generation of businesses. Leveraging APIs to produce and access the rich data, content, media, and algorithms that are the future commodities of the digital economy.
GET, POST, PUT, DELETE Are Used To Define Us
API providers use HTTP methods such as GET, POST, PUT, and DELETE to define their digital capabilities. As an API consumer, GET, POST, PUT, and DELETE define who I am and what I am building on top of the digital resources I am consuming from one or many different API providers. As an end user of these platforms and applications, GET, POST, PUT, and DELETE are how my digital self is put to work each day on the digital hamster wheel. Defining what we create and consume as we move through our days. Each API defines how we express ourselves online. The POST of a Tweet on Twitter. The POST of a video on Youtube. The GET our bank account balance. The PUT of our LinkedIn profile when we are looking for a new job. We DELETE that Instagram photo from when we drank too much last night. HTTP methods define an API provider, and which methods an API consumer puts to use will tell a story about what they are looking to do across their applications. These two dimensions of the API conversation have the most control over what is happening, and unfortunately most end users are left to just operate within whatever they are given, letting each platform and application developer define who they are, and what becomes of the data, content, and media that gets generated each day. Depending on the type of API resource being served up, such as a message, image, or video, and the platform making it available, each HTTP method can mean a variety of things. Adding a video on Youtube from a birthday party versus from a black lives matter protest. An image of your baby versus an image of your car as it drives through an intersection after being captured via a traffic camera. HTTP methods via APIs are shaping our behavior each day, and defining who we are, as we work and live in an Internet connected world. To help you see the importance of this conversation, think about those psychological tests Cambridge Analytica employed to harvest the data of millions of Facebook users via their API. That should help illuminate the psychological implications of GET, POST, PUT, and DELETE via APIs at scale each day.
APIs Allow Us To Define Personal Digital Assets
APIs are defining our personal digital assets from our bank accounts to our family photos. Our personal lives are spread across Facebook, Instagram, and TikTok. The digital exhaust from our days are captured via the desktop, web, and mobile applications we operated each day. Creating text, images, video, audio, location information, and other virtual bits that tell our narrative each day, and can be used to potentially predict our futures. More and more of our lives are being stored in the cloud and in the datacenter rather than in our closets, photo albums, and garages. The last twenty years of our lives has increasingly been captured, digitized, and made accessible online by us, or by companies, institutions, and government agencies targeting us. APIs define our digital reality as it is created and transmitted into the cloud for further processing, packaging up, and used in other systems and applications. APIs are being used to capture, define, and guide us throughout the world each day. And this is just our personal world. APIs are increasingly being used to capture, define, and direct our professional lives as well. In both our personal and professional capacities an awareness of APIs, and the ability to put them to work can have a profound impact on how much control we have over our personal and professional worlds. APIs can give us an edge in getting hired for a job, or being able to find success with a project at work. APIs can help us take more control over our health care data and our finances. Knowing that a SaaS solution you use has an API, then realizing you can use it to keep two separate services in sync, will incrementally give you more control over your online reality at home and work.
APIs Allow Us To Define Our Corporate Digital Assets
APIs define enterprise digital assets and capabilities. They provide the menu of what is possible when it comes to a company doing business online, and the maturity of APIs and the overall lifecycle employed to deliver APIs will define how fast a company is able to evolve, pivot, and respond to changing business conditions. If you want to see the important role APIs play as the menu for what is possible with an enterprise organization just look at Facebook or Amazon APIs. An API menu for how you change the world, and dominate the technological landscape. APIs are how you break down the monolithic business processes and software development of the last fifty years and you begin redefining, organizing, automating, and orchestrating with them in new and innovative ways. API management is how companies map out this API landscape that exists across an organization, and optimize how those API resources are made available to internal, partner, and public entities for integrating into other systems and applications. APIs are how businesses move their enterprise organizations forward. APIs are how businesses work with their partners. APIs are how businesses define and engage with their customers. The savviest companies out there today have been using APIs to transform themselves to not just stay relevant, but stay ahead of the pack for the last twenty years. It is how Amazon went from an e-commerce bookseller to dominating so many different business verticals while also powering half of the Internet with their Amazon Web Services. APIs are how Twitter goes from a fun social media platform to the global nervous system for the planet. APIs are how corporations are defining themselves, and redefining global markets today.
APIs Are Defining Our Institutional Digital Assets
Government agencies, universities, non-governmental organizations, and other institutions have all been leveraging the web for a greater part of the last twenty-five years, and those who are further along in their journey have realized that APIs are essential to their mission as well.. Most government agencies have APIs of some kind, and you will see more APIs emerging out of universities, first via their libraries, then from IT, faculty, and even student organizations and projects. All of this movement is just one slice of the digital pie when it comes to institutional adoption of APIs, where just like the rest of us, they also are increasingly dependent on APIs for the software as a service (SaaS) solutions they use each day. Leveraging Microsoft, Google, Salesforce, and other API-driven platforms to operate each day. Institutions in 2020 are both API providers and consumers, but many are only becoming aware of this recently. Most institutions are 5-10 years behind the rest of the private sector when it comes to Internet technology, leaving them ripe for vendor lock-in and extraction of value when it comes to the digital resources they possess. It is hard for institutions to say no to Google or Microsoft. Institutions find it difficult to avoid usage of social media, file sharing, and other popular tech platforms. Which leaves them open to some of the same predatory platform practices that the rest of us face when it comes to mining and harvesting of our data, just the stakes are much higher when you are a municipality, federal agency, or higher education institution. In 2020 all institutions are doing APIs, but like many companies they just aren’t doing them well, or as part of any overarching strategy, leaving institutions behind when it comes to the digital evolution.
APIs Are Used To Define Our Public Digital Assets
The public commons began changing substantially after it moved online. In the early days of the World Wide Web (WWW) things were much more free and open, and the web held such promise when it came to information sharing that benefits all, instead of just a handful of corporate entities. While there is a growing number of public data APIs emerging on a regular basis today, we are not seeing the coverage, maturity, and scale that we will need to have the desired impact on both the economy and the institutions behind them. Public data platforms rarely see the investment they need to be successful, with many commercial entities taking more than they give back in direct investment, as well as indirect tax funding of the public commons via trusted institutions. Public data and content is a vital and an increasingly valuable resource in free markets, but we don’t always see the investment and protection required to keep these resources available in a sustainable way while ensuring the value generation is going to the widest possible audience it can. If high value public data makes its way into the markets, it is just as likely to be captured and locked up by commercial entities, than it is to be successfully published as self-service, tiered access resources ensuring those who can’t afford access can get what they need, and those with more resources can also get what they need, but they also are required contribute back in some way that helps ensure an public API is sustainable for every stakeholder. Public data doesn’t just mean all data should just be publicly available. Public data means that it should be accessible and benefit the entire public. We’ve only scratched the surface when it comes to enriching the public commons with rich data, content, media, and algorithms, and with the right investment and protections in place, similar to the ones we’ve public in place for our physical public assets, we can ensure public digital assets continue to be published, evolve, and benefit both the public as well as the private sector.
“Open” and “Public” Can Mean Many Things on the Web
The words “open” and “public” mean different things to different people, both on and offline. Open and public lands will elicit different responses from individuals you ask from Wyoming to New York, as well as if you are asking a small outdoor tour operator and a big oil and gas company. It is even more diverse in a digital word. Google views US Census data very differently than a small grassroots organization trying to improve their local community or region. Open in many entrepreneurial circles means open for business and free in the monetary sense, where others might see it more about access and ability to use. Open source, open data, and open APIs are seen by the majority of consumers as something you take and use, and to a much smaller group as something you contribute back to. It is easy to see the web as about giving, because all the users of the web are getting the information they seek. When in reality there lies a lot more nuance to how data, content, and media gets created, shared, bought, and sold. It is easy to pretend that open or public data should remain free of charge, but data is rarely ever in a finished state, and it costs money to gather, organize, maintain, evolve, and serve up public data. Yes, taxes pay for this to occur, but like public lands and other physical infrastructure, there is further value to be generated and maintained. Value that shouldn’t just be siphoned off to a few, but be something that benefits everyone. Companies are using APIs and public APIs to redefine themselves in a digital age, employing not just APIs, but sophisticated API management practices to generate as much possible value as they can from their digital resources, while keeping the lion share of the value for themselves. There is no reason that public institutions can’t do the same to manage their own digital resources, but also leverage API management to maximize the value exchange, while also keeping the lion share of the value within the institution, helping them better achieve their mission.
Acknowledging the Role Public Resources Play In The Evolution Of Internet Technology
Everything we do on the web today owes a debt to public resources. The Internet was born out of federal government funding, and built on the backs of public universities. Despite popular liberatarian belief that the web is some free market utopia, it is built primarily on the backs of publicly subsidized infrastructure and programs, from our telecommunications network to grant money making companies like Google possible. The technology sector depends on public resources, and it is important that we acknowledge the role that digital data, content, media, algorithms, and other public resources play in making all of this working. Think about what GPS has done for the tech sector. That is what strong and healthy public data commons can do for the future of our economy. APIs are how corporations are redefining the landscape, and it will be how organizations, institutions, and government agencies redefine the landscape as well. Allowing the public sector to employ some of the same practices and tooling to deliver value around digital resources, empowering the next generation of applications, while also generating enough revenue to sufficiently fund the future we all envision. Public digital resources are a critical base for many commercial applications, and it also provides an essential counterbalance for end users of these applications that doesn’t always exist in free markets--requiring other non-commercial entities to step in and help keep everything balanced.
APIs are defining who we are in 2020. Enterprise organizations are aware of this. Developers are aware of this. End users and consumers of online technology are not so aware that every moment of every day is being defined as a never ending stream of API requests. It is widely recognized that data is seen as the “new oil” by entrepreneurs. An endless resource that can be mined, extracted, and transformed into an endless array of new products. The only problem with this analogy is that tihs new resource exists in each of us and the public institutions that we give rise to as a society, which means there is good money to be made in mindlessly mining and extracting value from our personal lives and the public commons in which we all depend on to live our lives. I am not advocating that we employ APIs for everything in our personal worlds and the public space because I think APIs are a good idea. I am advocating this because we have to become more API aware and literate to help us better navigate the digital world around us. APIs already exist--we need to be more aware of them. I am also advocating that we use APIs and some of the same API management practices use by the private sector to help quantify and protect public digital resources from exploitation by commercial interests. Our economy depends on a steady flow of data and other resources from the public sector, and APIs are how we are going to continue to quantify, defend, and evolve this value generation, ensuring that our collective public resources continue to benefit everyone, and not just the few technically savvy entities dominating the global business landscape in 2020.
