The API Evangelist Blog
This blog represents the thoughts I have while I'm research the world of APIs. I share what I'm working each week, and publish daily insights on a wide range of topics from design to depcration, and spanning the technology, business, and politics of APIs. All of this runs on Github, so if you see a mistake, you can either fix by submitting a pull request, or let me know by submitting a Github issue for the repository.
I like to pick worrisome patents from my API patent research and share them on my blog regularly. Last week I talk about Patent #US9300759 B1: API Calls With Dependencies and today I want to talk about patent #US09471283: Generating virtualized application programming interface (API) implementation from narrative API documentation, which according to its abstract is:
A virtualized Application Program Interface (API) implementation is generated based upon narrative API documentation that includes sentences that describe the API, by generating programming statements for the virtualized API implementation based on parsing the narrative API documentation and generating the virtualized API implementation based on upon the programming statements for the virtualized API implementation. The parsing of the narrative documentation may use a natural language parser and a domain-specific ontology for the API that may be obtained or created for the API. The virtualized API implementation may be generated using an API virtualizer.
I generally won't talk smack about folks filing patents, except I'm a pretty strong believer that the API interface and the supporting elements that make an API do the API thing should be left out. All the elements present in this patent like virtualization, documentation, and narrative need to be open--this is how things work. Just when we are all beginning to get good when it comes to crafting useful APIs, learning to speak in complete sentences with multiple APIs, we are going to patent the process of telling stories with APIs? Sorry, it just doesn't compute for me. I know this is what ya'll do at bigcos, but it's counterproductive.
Instead of filing this patent I wish they would have taken their idea and launched as open source tool, then also launch an accompanying service running in the cloud, and get to work letting us crafting narratives around our APIs. As I've said before, these patents really mean nothing, and it will all come down to keeping an eye on the court filings using the CourtListener API for any cases that are being litigated using API related patents.
It won't stop me from complaining, though. I just wish folks would spend their time on more productive things, instead of slowing down the world of APIs.
I was getting ready to do some work on a developer portal for a project I'm working on and I wanted to stop, step back and try to define what the goals in launching this portal are. As the technologist on this project, it is easy for me to impose my belief in why I am launching this portal (to publish documentation), but I feel it is important that we get the perspective of all stakeholders--so, I asked everyone involved what the goals were.
In the short term, the goals are to engage these groups around the API resources:
- Engage Third-Parties - Bring in new, and stimulate existing usage of data made available via APIs.
- Internal Departments - Ensure the internal groups are also engaged, and part of the outreach.
While incentivizing the following things between engaged stakeholders:
- Aware - Help make people aware of available resources, and what is being done with them.
- Conversations - Stimulate focused conversations among stakeholders around data and APIs.
- Issues - Focus on solving tangible issues that actually make an impact in the community.
Long term the goals to stimulate projects that matter, and can possibly bring in sustainable relationships and revenue that will help ensure the platform is up and running, and serving the target audiences. It takes money to do APIs properly, keeping the servers running, code evolving, active outreach and support in the community, and a passionate, engaged community to develop valuable projects.
I know all of this sounds overly simplified, but it is actually the thought process I'm applying to this project. I would say it is oversimplified to say that we are just launching a portal so that people will use our APIs. We need some basic goals about who we are reaching out to, and what we are trying to accomplish--we can evolve, and continue to define from there. When I lay out the outline for this project's portal I will make sure each element that I include speaks to these goals, or I will not put it in the version 1.0 of the portal.
Next, I want to make sure my minimum viable API portal definition speaks to the widest possible audience, and not just to developers. Then I will get to work on setting up the first version of the portal for use in this API project, and we'll consider what's next.
I'm spending some cycles on discovering what "cybersecurity" or "security" API solutions are out there, but specifically looking at threat information related to operating on the web. First up on the list is Facebook's ThreatExchange API, and I wanted to go through and break down what they offer via their API as I work to define an OpenAPI Spec, and their overall API operations as I populate an APIs.json file.This process helps me better understand what Facebook is offering in this area, as well as producing a machine readable definition that I can use across the rest of my research.
So, what is Facebook ThreatExchange?
Learn about threats. Share threat information back. Everyone becomes more secure. Most threat intelligence solutions suffer because the data is too hard to standardize and verify. Facebook created the ThreatExchange platform so that participating organizations can share threat data using a convenient, structured, and easy-to-use API that provides privacy controls to enable sharing with only desired groups.
- Scale your intelligence - Threats like malware and phishing will often attack many targets. Safeguard yourself using shared intelligence from other participants.
- A better way to share - Utilize a structured platform to send and receive information about threats.
- Join ThreatExchange - Learn about threats. Share threat information back. Everybody becomes more secure. *in beta
I'm wanting to understand Facebook's motivations behind doing the ThreatExchange API, and better understand the technical, business, and political aspects of providing a platform like this. When profiling a platform I always start with the endpoints, as I feel like they give the most honest accounting of what is going on.
- /malware_analyses - Search for malware samples by hash and other metadata.
- /malware_families - Search for malware families by name and other metadata.
- /threat_descriptors - Enables searching for indicators of compromise descriptors.
- /threat_indicators - Enables searching for indicators of compromise.
- /threat_tags - Enables searching for threat tags.
- /threat_exchange_members - Returns a list of current members of the ThreatExchange.
Next, I look at the underlying data and object model, and since Facebook uses their Graph model for their ThreatConnect API, it is pretty easy to pick up and get running with things--here are the graph objects used as part of the ThreatExchange:
- Malware - A sample of the malware.
- MalwareFamily - A group, cluster or similar type grouping of Malware.
- ThreatDescriptor - The subjective context provided by a ThreatExchangeMember for a ThreatIndicator
- ThreatExchangeMember - A participant within ThreatExchange.
- ThreatIndicator - An indicator of compromise.
- ThreatTags - A label to group threat objects together.
Facebook also allows for "types" to be applied across the ThreatConnect objects, adding additional dimensions to the objects available via the API:
- AttackType - The kind of attack a ThreatIndicator object was associated with.
- IndicatorType - The kind of indicator being described by a ThreatIndicator object.
- MalwareAnalysisType - A description of the type of malware, see MalwareAnalysisType.
- MalwareFamilyType - A description of the type of logic or automation used to create a MalwareFamily object.
- PrecisionType - Defines how accurately the threat intelligence detects its intended target, victim or actor.
- PrivacyType - Defines who can access the threat intelligence.
- ReviewStatusType - >A description of how the threat intelligence was vetted.
- SeverityType - A description of the dangerousness of the threat associated with a ThreatIndicator object. The order of the values below is ordered from least severe to most severe.
- SignatureType - The kind or format of signature described by a ThreatIndicator object with ThreatType of
- ShareLevelType (aka Traffic Light Protocol or TLP) - A designation of how any object in ThreatExchange may be re-shared both within and outside of ThreatExchange, based on the US-CERT's Traffic Light Protocol.
- StatusType - A description of the maliciousness of any object within ThreatExchange.
- ThreatType - The kind of threat a ThreatIndicator object is expressing.
I was pleased to see webhook infrastructure available, making things a two-way street, and a little more responsive and real-time. I would also consider a streaming technology at this layer eventually, making it easier to manage at scale.
- Webhooks - ThreatExchange Webhooks allow you to receive information in real-time for MalwareAnalyses, ThreatDescriptors, and MalwareFamilies.
It was also good to see privacy considerations right out of the box. If you are going to get other companies to participate at this level and share real-world threat information, there have to be granular controls to help mediate the politics of all of this. The Facebook approach also doesn't just deal with identity, groups, and access, it also dictates the sharing of information--I am guessing a logging layer at this level will be needed in the future, with an API for it as well, to enable 3rd party auditing.
- Privacy Controls - All submissions to the ThreatExchange API allow for limiting the visibility of any Malware and ThreatDescriptor objects. Currently, ThreatExchange supports three levels of visibility: allow all members; allow a ThreatPrivacyGroup; and allow a whitelist of specific members. The desired privacy setting on an object is specified by the values at the time of a create or edit submission to the API. Privacy settings can also be changed retroactively for data you've already submitted.
- Reshare - All submissions to the ThreatExchange API allow for defining how the data can be re-shared by its recipients. The level of re-sharing is applied via the share_level attribute.
This is where this API becomes an exchange. With the required privacy and sharing controls in place, the submitting of new data, connecting the dots between data, and evolving the history of everything in real time can occur, allowing trusted 3rd parties to not just access data, but confidently share it with the ThreatExchange.
- Submitting New Data - You may submit data to the graph via an HTTP POST request the following URL: https://graph.facebook.com/v2.8/threat_descriptors
- Submitting Connections - ThreatExchange supports creating connections (aka edges) between ThreatIndicator objects to express relationships. Examples of when this can be useful are for describing URL re-direct chains or domain to IP address relationships.
- Editing Existing Data - The ThreatExchange API allows for editing existing ThreatIndicator objects. As with all Facebook Graph APIs, editing is performed via an HTTP POST request to the object's unique ID URL.
- Deleting Data - ThreatExchange currently supports true deletes only for connections between ThreatIndicator objects to express relationships. Examples of when this can be useful are for describing URL re-direct chains or domain to IP address relationships.
After that, there are some more support level elements present. I track on these elements as part of my indexing with an APIs.json, so that anyone can easily find the code, examples, and other supporting resources without clicking around, and digging through the documentation.
There are more general elements of the Facebook Developer community present as well, but this provides a nice roundup of all the elements that make up the Facebook ThreatExchange API, providing one possible blueprint for how we share threat information associated with operating any platform on the web. I'm going to be looking at other solutions in this same space, but I wanted to profile Facebook's effort in this area first, as they probably have the most insight and investment in this area.
What Are Facebook's Motives With ThreatExchange?
One thing I want to learn more about is why Facebook would want to do the ThreatExchange. I'm sure they truly want to share the wealth of data they have on threats they have there, but I'm sure they equally want incentivize other tech giants to share their data as well, sweetening the entire pot. I would add that they might also be doing this to alleviate any future regulatory pressures that may unfold around cybersecurity legislation. All we can do is monitor what is going on and see how much Facebook, and others invest into ThreatExcahgne before we will know the full extent of the motivations behind the platform.
A Possible Blueprint For Wider Threat Exchange Model
This post is part of my research into security and cybersecurity. As with the other areas of my research I am looking for the common building blocks of how APIs are being used to understand, map out, defend, and share threat information. I'm happy to see Facebook being proactive in this area, but this feels like it needs to become an open standard and wider blueprint that can be operated by an independent party, with trusted arrangements from each provider regarding what is being submitted and shared--for it to work properly. I have a lot more research to conduct before I make this determination, but it is where I'm headed with my thinking in this area.
Researcher and Journalist Access To Data On Exchange
I applied for access to the Facebook ThreatExchange API, but I'm unsure if I will be given access, being that I am just a blogger. This opens up a wider question, though, regarding opening up this valuable information to researchers, journalists, and 3rd party auditors. I talked about the need for API access tiers and potentially service level agreements for research and journalists previously, and threat information is definitely an area we need to deeply consider who has access to this important data, in a similar way. We need to be ensuring the privacy of companies, organizations, institutions, and government agencies are protected, but we also need to allow for a wider understanding of this problem to be established, with more eyes on the data.
Further Assessment of APIs and Threat Information Solutions
I am just getting going with this research, and it will take some time for me to paint a complete picture of the landscape when it comes to online threat data, and how APIs are being used. I'll publish each profile as I complete them, and eventually, I'd like to have a common definition of all the API paths, and underlying schema being employed as part of the leading threat data sharing platforms. If we are going to scale the effort to address this growing problem, we are going to need a wealth of APIs in operation, and ensure they are all speaking a common dialect. I need to develop a better understanding of the politics around the operation of a threat data API, but I'm guessing it is something we don't want in the hands of any single, leading tech provider.
Silicon Valley startups and entrepreneurs love to point out that they are trying to make the world a better place. Over a 25+ year career, I have fallen for the belief that I was improving a situation through technology. Hell, I still do this regularly as the API Evangelist, stating that a certain approach to opening up access to data, content, and algorithms can make things better, when in numerous situations it will not. I walk a fine line with this and I hope that I'm a little more critical about where technology should be applied, and focus primarily on making existing technology more accessible using APIs--not starting new ones.
When you are critical of technology in the current climate, there are plenty of folks you like to push back on you, leaning on the fact that they are trying to make the world a better place. Not sure where this line of bullshit first began but is something I should look into. I mean, when you hang out with enough entrepreneurs you really begin see through the bullshit and hype, and you know that this is all about them selling you something, and then selling you and your data as part of an exit via acquisition or going public. It rarely ever has anything to do with making the world a better place, what was promised as part of the marketing, and helping the average end-user.
This is where my entrepreneur friends have stopped reading and lean on the fact that they actually believe they are trying to make the world a better place, and their firm belief that Kin is an annoying hippie. You know I love you. You are a special snowflake. But, you are a minority, I am talking about the other 95% of the system you choose to be willfully ignorant of. If you want some evidence of this in the wild, take a look at all the world saving, innovative, revolution startups out there and how many are dedicated to terms of service?
You know, those little legal documents we agree to for our usage of every device in our life, and every single application we use in our professional and personal worlds. The terms of service touch ALL of our lives, and act as a central force throughout our days--kind of like gravity, except it's a corporate force, not anything natural--pushing down on everything we do. As important as terms of services are, and how big of a role they play in our lives, where is the innovation around these legal documents? You know, the kind that assists the average citizen make sense of all of this bullshit and makes their life better?
There are Terms of Service Didn't Read, Docracy, and TOSBack, but they just don't have the resources to tackle this properly. There are some service providers who do a good job of helping users make sense of their own terms of service, but there is NO INNOVATION when it comes to terms of service. There is nothing that serves the end-users over the platforms. With all of the innovation going on out there why isn't there more investment, and solutions in the area of terms of service? I mean, if we are trying to make our user's lives easier with technology, and not more complicated, why isn't there more technology to make sense of this legalese that is governing our lives?
Hint, it is because we aren't out for the greater good. With all the money flowing around out there, I just can't believe there isn't the money to do this right. Provide a suite modern tooling and a wealth of legal resources to tackle this problem properly. If nothing else, let's throw TOS Didn't Read, Docracy, and EFF's TOSBAck more money. Maybe there should be a "guilt tax" imposed on every terms of service owner to pay for an English, human translation of the legal documents that guide their platforms. Maybe there should be a machine-readable schema enforced at the regulatory level, making sure a suite of tools can process, and make sense of TOS changes in real-time.
Maybe some day we will stop just saying we are trying to help the world with our technology, and actually invest in what truly does.
This is one of those regular public service announcements that if at all possible, you should be requiring SSL for all your API calls. I recently got an email from the IBM Watson team telling me that they would be enforcing encryption on all calls to the Alchemy API in February.
SSL is something I've started enforcing on my own internal APIs. I do not have wide usage of my APIs by third-party providers, but I do have a variety of systems making calls to my APIs, transmitting some potentially sensitive information--luckily nothing too serious, as I'm just a simple API Evangelist.
Encryption is an area I research regularly, hoping to stay in tune (as much as I can) with where discussions are going when it comes to encryption and API operations. Much of it doesn't apply to the average API provider but requiring encryption for API calls, and emailing your developers when and if you do begin enforcing, is what I'd consider an essential building block for all API providers.
I'll keep crafting a unique blog post each time I receive on of these emails from the APIs I depend on. Hopefully some day soon, all APIs will be SSL by default.
Integration Platform as a Service (iPaaS) solutions are something I've been tracking on for a while, and an area I haven't seen too much innovation in, except by Zapier for most of that time. I'm a big fan of what IFTTT enables, but I'm not a big fan of companies who build services that depend on APIs but do not offer APIs in turn, so you don't find me highlighting them as an iPaaS solution.
Instead, you'll find me cheering for Zapier, who has an API, and even though I wish they had more APIs, I am grateful they paying it forward a little bit. I wish we had better solutions, but the politics of API operations seems to slow the evolution of iPaaS, usually leaving me disappointed.
That was until recently when some of my favorite API hackers released DataFire:
DataFire is an open source integration framework - think Grunt for APIs, or Zapier for the command line. It is built on top of open standards such as RSS and Open API. Flows can be run locally, on AWS Lambda, Google Cloud, or Azure via the Serverless framework, or on DataFire.io.
"DataFire natively supports over 250 public APIs including: • Slack • GitHub • Twilio • Trello • Spotify • Instagram • Gmail • Google Analytics • YouTube, as well as MongoDB, RSS feeds, and custom integrations." They have a sample flows available as an individual Github repositories. Integrations can be added by the URL of an Open API (Swagger) specification or an RSS feed, you can also specify --raml, --io_docs, --wadl, or --api_blueprint.
DataFire is new, so it has a lot of maturing to do as an API framework, but it has EVERYTHING that iPaaS solutions should have at its core in my opinion. It's API definition-driven, its open source, and there is a cloud version that any non-developer user can put to use. DataFire is encouraging everyone to share each of the flows as machine readable templates, each as their own Github repository so that anyone can fork, modify, and put to work. #IMPORTANT
This is the future of iPaaS. There is lots of money to be made in the sector, empowering average business, professional, and individual users when it comes to managing their own bits on the web--if current providers get out of the way. The problem with current solutions is that they work too hard to capture the exhaust of these workflows and limit their execution to specific walled garden platforms. DataFire acknowledges that these flows will need to be executed across all the leading cloud providers, orchestrated in serverless environments, as well as more managed level of service in a single, premium cloud edition.
DataFire is the iPaaS solution I've been waiting to see emerge and will be investing time into learning more about how it works, developing integrations and flows, and telling stories about what others are doing with it. DataFire and DataFire.io needs a lot of polishing before it is ready for use by the average, non-technical IFTTT or Zapier user, but I don't think it won't take much to get it there pretty quickly. I'm stoked to finally have an iPaaS solution that I can get behind 100%.
In addition to using the news of Medium's downsizing as a moment to stop and think about who owns our bits, I wanted to point out what a missed opportunity the Medium API is. Having an API is no guarantee of success, and after $132M in 3 Rounds from 21 Investors, I'm not sure an API can even help out, but it is fun to speculate about what might be possible if Medium had robust API in operation.
Medium has an API, but it is just a Github repository, with reference to a handful of paths allowing you to get details on yourself, the publications you are part of, and post entries to the site. There are no APIs for allowing me to get the posts of other users, or publications, let alone any of the analytics, or traffic for this. I'm guessing there is no API vision or champion at Medium, which results in the simple, restrictive API we see today.
Many media and content companies see APIs as giving away all the value they are trying to monetize, and are unaware of the control that modern approaches to API management bring to the table. Many people see the pain that other API pioneers have suffered like Twitter, and want to avoid the same problems, completely unaware that many of Twitter's ecosystem problems were Twitter-induced and not inflicted by 3rd party developer.
If Medium had a proper developer portal, complete set of API paths, proper OAuth controls, and other API management tooling, they could open up innovation in content delivery, publishing, analytics, visualizations, voice, bot, and the numerous of other areas where APIs are changing how we consume, create, and engage with information. I get that control over the user experience is key to the Medium model, but there are examples of how this can be done well, and still have an API. The best part is it only costs you the operation of the API operations.
I do not think more money will save Medium. I think they have to innovate. I think part of this innovation needs to come from the passionate base of users they have already developed. I've seen Medium carefully plot out a path forward when it comes their trusted partners, and there is no reason this type of quality control can't be replicated when it comes to API consumers, as well as the applications and integrations that they develop. The Medium API just seems like a really huge missed opportunity to me, but of course, I'm a little biased.
As I approach my seventh year as the API Evangelist and find myself squarely in 2017, I wanted to take a moment to better understand, and articulate why I still believe in APIs. To be the API Evangelist I have to believe in this, or I just can't do it. It is how my personality works--if I am not interested, or believe in something, you will never find me doing it for a living, let alone as obsessively as I have delivered API Evangelist.
First, What Does API Mean To Me?
There are many, many interpretations, and incarnations of "API" out there. I have a pretty wide definition of what is API, one that spans the technical, business, and politics of APIs. API does not equal REST, although it does employ the same Internet technologies used to drive the web. API is not the latest vendor solution or even standard. The web delivers HTML for humans to consume in the browser, and web APIs deliver machine-readable media types (XML, JSON, Atom, CSV, etc.) for use in other applications. When I say applications, I do not just mean the web, mobile, and devices applications--I mean other applications, as in "the action of putting something into operation". An API has to find harmony between the technical, business, and political aspects of API operations and strike a balance between platform, 3rd party developer, and end-user needs--with every stakeholder being well informed along the way.
I Still Believe Early My API Vision
When I close my eyes I still believe in the API vision that captured my attention in 2005 using the Delicious API, again in 2006 with the Amazon S3 and EC2 APIs, and with the Twitter API in 2007. Although today, I better understand that a significant portion of this early vision was very naive, and too trusting in the fact that people (API providers and consumers) would do the right thing with APIs. APIs use Internet technology to make data, content, and algorithms more accessible and observable, securely using the web. APIs can keep private information safe, and ensure only those who should have access do. I believe that an API-first approach can make companies, organizations, institutions, and government agencies more agile, flexible, and ultimately more successful. APIs can bring change, and make valuable resources more accessible and usable. When done right.
APIs Are Not Going Away Or Being Replaced
Yes, many other technologies are coming along to evolve how we exchange data, content, and algorithms on the web, but doing this as a platform, in a more open, transparent, self-service, and machine-readable way is not going anywhere. I try not to argue in favor of any technological dogma like REST, Hypermedia, GraphQL, and beyond, but I do try to be aware and make recommendations of the benefits, and consequences along the way. Having a website or mobile applications for your business, organization, institution, government agency, and often times individuals isn't going to go away anytime soon. Neither will sharing the same data, content, and algorithms available for use in applications in a machine readable way, for use by 3rd party groups using Internet technology. The term API has gained significant mindshare in the tech space, enjoys use as an acronym in the titles of leading news publications, and has taken root in the vocabulary of average folks, as well as the entrepreneurs, developers, and technology companies who helped popularized the term.
APIs Are Not Good, Nor Bad, Nor Neutral--People And Companies Are
APIs get blamed for a lot of things. They can go away at any time. They are unstable. They are not secure. The list goes on and on, and many also like to think that I blindly evangelize that APIs will make the world better all by themselves--I do not. APIs are a reflection of the individuals, companies, organizations, institutions, and agencies that operate them. In the last seven years, I have seen some very badly behaved API providers, consumers, and companies who are selling services to the sector. Going forward, I will never again be surprised again by how badly behaved folks can be when using APIs, and the appetite for an API-driven surveillance economy that is fueled by greed and fear.
API Specifications, Schema, and Scopes Are The Most Critical Part
I track on over 75 stops along my definition of the API life cycle, and many of them are very important, but my API definition research, which encompasses the world of API specifications, schema, and scopes will play the most significant role in the future of the web, mobile, and connected device technology. This is why I will be dedicating the majority of my bandwidth to this area in 2017, and beyond. Other areas will get attention from me, but API specifications, schema, and scopes touch on every other aspect of the API sector, as well as almost EVERY business vertical driving our economy today. Industry level standards like PSD2 for the banking industry will become clearer in coming years, as a result of the hard work that is going on right now with API definitions.
Help Ensure The Algorithms That Are Increasingly Impacting Our Lives Are Observable
APIs provide access to data and content, but they also provide some observability into the black box algorithms that are increasingly governing our world. While APIs can't provide 100% visibility into what algorithms do, or do not do, they can provide access to the inputs, and the outputs of the algorithms, making them more observable. When you combine with modern approaches to API management, it can do this securely, and in a way that considers any end-user, as well as developer and platform interests. I'm going to keep investing in fine tuning my argument for APIs to be used as part of artificial intelligence, machine learning, and the other classifications of algorithms that are being put to use across most business sectors. It's a relatively new area of my API research, but something I will invest more time, and resources into to help push the conversation forward.
More API Storytelling Is Needed, And The Most Rewarding Part Of What I Do
I've done an OK job at instigating other API service providers, API operators, integrators, and individuals to tell their story. I feel that my biggest impact on the space has been producing the @APIStrat conference with my partner in crime Steve Willmott at 3Scale / Red Hat, which is focused on giving people across the API sector a diverse, and inclusive place to tell their story, on stage and in the hallways. Next, I'd say my regular storytelling on my blog, like my evergreen post on the secret to Amazon's success, has had the biggest impact, while also being the most rewarding and nourishing part for me personally. I've heard stories about my Amazon blog post being framed on the wall in a bank in Europe, baked into the home page of the IT Portal for government agencies, and many other scenarios. Stories matter. We need more of it. I am optimistic regarding much of the new writing I'm seeing on Medium, but this storytelling being mostly isolated to Medium worries me.
Keep On, Keep'n On With API Evangelist The Best I Can
I still believe in APIs. Over the last seven years, my belief in what APIs can do has not diminished. My belief in what people are capable of doing with them has tremendously. I enjoy staying in tune with what is going on, and trying to distil it down into something people can use in their own API operations. I think the biggest areas of concern for the API life cycle in coming years will be in the areas of API definitions, security, terms of services, privacy, discovery, intellectual property, and venture capital.
I just needed to refresh my argument of why I still believe in APIs. Honestly, 2015 and 2016 really stretched my faith when it came to APIs. Once I realized it was primarily a crisis of faith in humans and not in APIs, I was able to find a path forward again with API Evangelist. Really, I couldn't ask for a better career focus. It keeps me reading, learning, and thinking deeply about something that touches all of us, and possesses some really meaningful areas that could make an actual impact on people's lives. Things like Open Referral, my work on FAFSA, EPA, National Park, Census, Energy, API Commons, and APIs.json, keep me feeling that APIs can make a positive impact on how our world works.
In 2017 I don't think I need to spend much time convincing people to do APIs. They are already doing this. There is more going on than I can keep track of. I think I need to focus on convincing people to do APIs as open and observable as they possibly can. Push for a balanced approach to not just the technology, but the business and legal side of platform operations, in a way that protects the platform's operations, but also in a way that is looking out for 3rd party developer, and end-user interests. I still believe in APIs, but only if they possess the right balance which I talk about regularly, otherwise they are just another approach to technology that we are letting run amok in our lives.
I am working with Open Referral to evolve the schema for the delivery of human services, as well as helping craft a first draft of the OpenAPI Spec for the API definition. The governing organization is looking to take this to the next level, but there are also a handful of the leading commercial providers at the table, as well other groups closer to the municipalities who are implementing and managing Open211 human service implementations.
I was working with Open Referral on this before checking out this last summer, and would like to help steward the process, and definition(s) forward further in 2017. This means that we need to speak using a common language when hammering out this specification and be using a common platform where we can record changes, and produce a resulting living document. I will be borrowing from existing work I've done on API definitions, schema, and scope across the API space, and putting together a formal process design specifically for the very public process of defining, and delivering human services at the municipal level.
I use OpenAPI Spec (openapis.org) as an open, machine readable format to drive this process. It is the leading standard for defining APIs in 2017, and now is officially part of the Linux Foundation. OpenAPI Spec provides all stakeholders in the process with a common language when describing the Open Referral in JSON Schema, as well as the surface area of the API that handles responses & requests made of the underlying schema.
I have an OpenAPI Spec from earlier work on this project, with the JSON version of the machine-readable definition, as well as a YAML edition--OpenAPI Spec allows for JSON or YAML editions, which helps the format speak to a wider, even less technical audience. These current definitions are not complete agreed upon definitions for the human services specification, and are just meant to jumpstart the conversation at this point.
OpenAPI Spec provides us with a common language to use when communicating around the API definition and design process with all stakeholders, in a precise, and machine readable way. OpenAPI Spec can be used to define the master Open Referral definition, as well as the definition of each individual deployment or integration. This opens up the opportunity to conduct a "diff" between the definitions, showing what is compatible, and what is not, at any point in time.
The platform I will be using to facilitate this discussion is Github, which provides the version control, "diff", communication, and user interactions that will be required throughout the lifecycle of the Open Referral specification. Allowing each path, parameter, response, request, and other elements to be discussed independently, with all history present. Github also provides an interesting opportunity for developing other tools, like I have for annotating the API definition as part of the process.
This approach to defining a common data schema and API definition requires that all stakeholders involved become fluent in OpenAPI Spec, and JSON Schema, but is something that I've done successfully with other technical, as well as non-technical teams. This process allows us all to all be on the same page with all discussion around the Open Referral API definition and schema, with the ability to invite and include new participants in the conversation at any time using Github's existing services.
Once a formal draft API specification + underlying JSON schema for Open Referral is established, it will become the machine readable contract and act as a central source of truth regarding the API definition as well as the data model schema. It is a contract that humans can follow, as well as be used to drive almost every other stop along the API life cycle like deployment, mocking, management, testing, monitoring, SDKs, documentation, and more.
This process is not unique to Open Referral. I want to be as public with the process to help other people, who are working to define data schema, understand what is possible when you use APIs, OpenAPI Spec, JSON Schema, and Github. I am also looking to reach the people who do the hard work of delivering human services on the ground in cities and help them understand what is possible with Open Referral. Some day I hope to have a whole suite of server-side, and client-side tooling developed around the standard, empowering cities, organizations, and even commercial groups deliver human services more effectively.
I have a minimum definition for what I consider to be a good portal for an API, and was spending some time thinking about a baseline definition for the API developer account portion of that portal, as well as potentially any other authenticated, and validated platform user. I want a baseline user account definition that I could use as aa base, and the best one out there off the top of my head would be from Google.
To support my work I went through my Google account page and outlined the basic building blocks of the Google account:
- Sign-in & Security - Manage your account access and security settings
- Signing in to Google - Control your password and account access, along with backup options if you get locked out of your account.
- Password & sign-in method - Your password protects your account. You can also add a second layer of protection with 2-Step Verification, which sends a single-use code to your phone for you to enter when you sign in.
- Password - Manage your password.
- 2-Step Verification - Manage 2-Step verification.
- App Passwords - Create and manage application passwords.
- Account recovery options - If you forget your password or cannot access your account, we will use this information to help you get back in.
- Account Recovery Email - The email to send recovery instructions.
- Account Recovery Phone - The email to send recovery instructions.
- Security Question - A secret question to use as verification during recovery.
- Password & sign-in method - Your password protects your account. You can also add a second layer of protection with 2-Step Verification, which sends a single-use code to your phone for you to enter when you sign in.
- Device activity & notifications - Review which devices have accessed your account, and control how you want to receive alerts if Google thinks something suspicious might be happening.
- Recent security events - Review security events from the past 28 days.
- Recently used devices - Check when and where specific devices have accessed your account.
- Security alerts settings - Decide how we should contact you to let you know of a change to your account’s security settings or if we notice something suspicious.
- Connected apps & sites - Keep track of which apps and sites you have approved to connect to your account and remove ones you no longer use or trust.
- Apps connected to your account - Make sure you still use these apps and want to keep them connected.
- Saved passwords - Use Google Smart Lock to remember passwords for apps & sites you use Chrome & Android
- Signing in to Google - Control your password and account access, along with backup options if you get locked out of your account.
- Personal Info & Privacy - Decide which privacy settings are right for you
- Your personal info - Manage this basic information — your name, email, and phone number.
- Manage your Google activity - You have lots of tools for controlling and reviewing your Google activity. These help you decide how to make Google services work better for you.
- Activity controls - Tell Google which types of data you’d like to save to improve your Google experience.
- Review activity - Here’s where you can find and easily review your Google activity.
- Ads Settings - You can control the information Google uses to show you ads.
- Control your content - You are in control of the content in your Google Account, even if you stop using Google products or decide to delete your account altogether.
- Copy or move your content - You can make a copy of the content in your account at any time, and use it for another service.
- Download your data - Create an archive with a copy of your data from Google products.
- Assign an account trustee - Approve a family member or friend to download some of your account content in the event it is left unattended for an amount of time you've specified.
- Data Awareness - We want you to understand what data we collect and use.
- Account Preferences - Manage options for language, storage, and accessibility. Set up your data storage and how you interact with Google.
- Language & Input Tools - Set Google services on the web to work in the language of your choice.
- Accessibility - Adjust Google on the web to match your assistive technology needs.
- Your Google Drive storage - Your Google account comes with free Google Drive storage to share across Gmail, Google Photos, Google Docs and more. If this isn't enough, you can add more storage here.
- Delete your account or services - If you are no longer interested in using specific Google services like Gmail or Google+, you can delete them here. You can even delete your entire Google Account.
- Delete a Google service - A Google Account offers many services. Some of these services can be deleted from your account individually.
- Delete your Google Account - You're trying to delete your Google Account, which provides access to various Google services. You'll no longer be able to use any of those services, and your account and data will be lost.
These are all building blocks I will add to my API management research, with an overlap with my API portal research. I'm not sure how many of them I will end up recommending as part of my formal guide, but it provides a nice set of things that seem like they SHOULD be present in all online services we use. Google also had two other tools present here, that overlap with my security and privacy research:
- Security Checkup - Protect your account in just a few minutes by reviewing your security settings and activity.
- Privacy Checkup - Take this quick checkup to review important privacy settings and adjust them to your preference.
I am going to be going through Google's privacy and security sections, grabbing any relevant building blocks that providers should be considering as part their API operations as well. For now, I'm just going to add this to the list of things I think should be present in the accounts of third party platform users, whether they are a developer, partner, or otherwise.
I would also like to consider what other providers offer accounts features I'd like to emulate. Like Amazon, Dropbox, and other leading providers. I would also like to take another look at what the API management providers like 3Scale offer in this area. Eventually, I want to have an industry guide that API providers can follow when thinking about what they should be offering as part of their user accounts.
I am evaluating Shutterstock as a new destination for some of my photos and videos. I've been a Shutterstock user for their stock images, but I'm just getting going being a publisher. I thought it was worth noting that as part of their sign up process they require me to upload a copy of my state issued identification before I can sell photos or images as a Shutterstock publisher.
This is something I've encountered with other affiliate , partner, and verified solutions. I've also had domains expire, go into limbo, and I have to fax in or upload my identification. It isn't something I haven't seen with many API providers yet, but I'm guessing it will be something we'll see more of with API providers further locking down their valuable resources.
I am not sure how I feel about it being a regular part of the partner and developer validation process--I will have to think about it more. I'm just adding to the list of items I consider as part of the API management process. It makes sense to certify and establish trust with developers, but I'm not 100% sure this is the way to do it in the digital age. IDK, I will consider more, and keep an eye out for other examples of this with other API providers.
Shutterstock publisher isn't necessarily connected directly to the API, but once I'm approved I will be uploading, and managing my account via their API, so it is a developer validation process for me. The topic of developer validation and trust keeps coming up in other discussions for me, and with the increasing number of APIs we are all developing with, it seems like we are going to need a more streamlined, multi-platform, and an API-driven solution to tackle this.
For me, it would be nice if this solution was associated with my Github account, which plays a central role in all of my integrations. When possible, I create my developer accounts using my Github authentication. It would be nice if I had some sort of validation, and trust ranking based upon my Github profile, something that would increase the more APIs I use, and establish trust with.
I will file this away under my API management, authentication, and partner research, and will look for other examples of it in the wild--especially at the partner layer. Certifying that developers are truly human, or possibly truly a valid corporate entity seems like it is something that will only grow in the bot-infested Internet landscape we are building.
I regularly talk about the evolving world of API SDKs, showcasing what API service providers like APIMATIC are up to when it comes to orchestration, integration, other dimensions of providing client code for API integrations. A new example of this that I have found in the wild is from communication and support API platform Intercom, with their publishing of Docker images of their API SDKs. This overlaps my SDK research with the influence that containerization is having on the the world of providing and integrating with APIs.
Intercom provides Docker images for their Ruby, Node, Go, and PHP API SDKs. It's a new approach to making API code available to API consumers that I haven't seen before, (potentially) making their integrations easier, and quicker. I like their approach to providing the containers and specifically the fact they are looking for feedback on whether or not having SDK Docker containers offer any benefit to developers. I'm guessing this won't benefit all API integrators, but for those who have successfully adopted a containerized way of life, it might streamline the process and overall time to integration.
I just wanted to have a reference on my blog to their approach. I'll keep an eye on their operations, and see if their SDK Docker images become something that gets some traction when it comes to SDK deliver. Since they are sharing the story on their blog, maybe they'll also provide us with an update in a couple months regarding whether developers found it useful or not. If nothing else, their story has reminded me to keep profiling Intercom, and other similar API providers, as part of my wider API communication and support research.
Evernote employees have not read, and do not read, your note content. We only access notes under strictly limited circumstances: where we have your permission, or to comply with our legal obligations.Your privacy, and your trust in Evernote are the most important things to us. They are at the heart of our company, and we will continue to focus on that now and in the future.
While I am thankful for their change of heart, I wanted to take a moment to point out the wider online environment that incentivizes this type of behavior. This isn't a single situation with Evernote reading our notes, this is the standard mindset for startups operating online, and via our mobile devices in 2017. This is just one situation that was called out, resulting in a change of heart by the platform. Our digital bits are being harvested in the name of machine learning and artificial intelligence across the platforms we depend on daily for our business and personal lives.
In these startup's quest for profit, and ultimately their grand exit, they are targeting our individual and business bits. Use this free web application. Use this free mobile application. Plug this device in at home on your network. Let our machine learning evaluate your database for insights. Let me read your most personal and intimate thoughts in your diary, journal, and notebook. I will provide you with entertainment, convenience, and insight that you would never be able to achieve on your own, without the magic of artificial intelligence.
In my experience, the position a company takes on their API provides an early warning system for these situations. Evernote sent all the wrong signals to their developer community years ago, letting me know it was not a platform I could depend on, and trust for my business operations, let alone my personal, private thoughts. I was able to get off the platform successfully, but the trick in all of this is identifying other platforms who are primed for making similar decisions and helping the average users understand the motives behind all of these "solutions" we let into our lives, so they can make an educated decision on their own.
I'm working on the next evolution in my API research, and I'm investing more time and energy into the design of the guides I produce as a result of each area of my research. I've long produced a 20+ page PDF dumps of the leading areas of my research like API design, definitions, deployment, and management, but with the next wave of industry guides, I want to polish my approach a little more.
The biggest critique I get from folks about the API industry guides I produce is that they provide too much information, aren't always polished enough, and sometimes contain some obvious errors. I'm getting better at editing, but this only goes so far, and I'm bringing in a second pair of eyes to review things before they go out. Another thing I'm introducing into the process is the use am of professional design software (Adobe InDesign) rather than just relying on PDF's generated from my internal system with a little HTML spit shine.
While it is taking me longer to dust off my Adobe skills than I anticipated, I am finding the design process to be extremely valuable. I've often dismissed the fact that my API research needed to look good, and that it is more important that it is just available publicly on my websites. This is fine and is something that will continue, but I'm finding a more formal design process is helping me think through all of the material, better understand what is valuable, what is important, and hopefully better tell a story about why it is relevant to the API space. It is helping me take my messy backend data and content, and present it in a more coherent and useful way.
As I'm formalizing my approach using my API definition guide, I'm moving on to my API design guide, and I can't help but see the irony in learning the value of design while publishing a guide on API design, where I highlight the benefits of API design to make sense of our growing digital assets. Once I've applied this new approach to my definition, design, deployment, DNS, and management guides I am going to go back to my API design for these resources, and take what I've learned and applied to the way I share the raw data and content. The lesson that stands out most at the moment, is that less is more, and simple speaks volumes.
I understand that companies file for patents to build their portfolios, and assert their stance in their industry, and when necessary use patents as leverage in negotiations, and in a court of law. There are a number of things that I feel patents logically apply to, but I have trouble understanding why folks insist on patenting things that make the web work, and this whole API thing work.
One such filing is patent number US9300759 B1: API Calls With Dependencies, which is defined as:
Techniques are disclosed for a client-and-server architecture where the client makes asynchronous API calls to the client. Where the client makes multiple asynchronous API calls, and where these API calls have dependencies (i.e., a result of one call is used as a parameter in a second call), the client may send the server these multiple asynchronous API calls before execution of a call has completed. The server may then execute these multiple asynchronous API calls, using a result generated from one call as a parameter to another call.
Maybe I live in a different dimension than everyone else, but this doesn't sound unique, novel, and really just feels like folks are mapping out all the things that are working on the web and filing for patents. I found this patent while going through the almost 1300 API related patents in Amazon Web Services portfolio. Many of their patents make sense to me, but this one felt like it didn't belong.
When I read these patents I worry about the future of the web. Ultimately I can only monitor the courts for API related patent litigation, and keep an eye out for new cases, as this is where the whole API patent game is going to play out. I'll keep squawking every time I read a patent that doesn't just belong, and when I see any new legal cases I will help shine a light on what is going on.
The importance of a machine readable API definition has grown significantly over the last couple of years, with a lot of attention being spent (rightfully so) on helping educate API providers of the value of having an OpenAPI Spec, API Blueprint, or another format. This is something I want to continue contributing to in 2017, but I also want to also shine a light on the importance of having your data schema well defined.
When you look through the documentation of many API providers, some of them provide an example request which might give hints at the underlying data model, but you rarely ever see API providers openly share their schema in any usable format. You do come across some of a complete OpenAPI Spec or API Blueprints from time to time, but usually, when you find API definitions, the schema definition portion is incomplete.
Not having your schema well defined, shareable, and machine-readable is one of the contributing factors to a lack of common, shared schema in the API space. We have healthy examples of this in action with Schema.org, but for some reason, many of us don't bring schema front and center in our API operations. We are accepting them as input for our API requests, and returning them with API responses, but don't always share examples of this in our documentation, complete sections in our API definitions, or share JSON Schema as part of our developer resources. All contributing to a lack of consistency within a single API operation, as well as the wider industry.
I am going to spend more time in 2017 talking to people about the schema they use in their API operations and shining a light on existing schema that has been published by API providers. I will be also continuing to support important schema like Open Referral that helps streamline how our world works. It is no secret that when we speak using common schema things work smoother and that we will make sense to a wider audience. I hope in 2017 we can invest more in defining and sharing the schema we put to use, and reusing some the most common examples out there.
I spend a lot of time studying and thinking about the "digital bits" that we move around the Internet. Personally, and professionally I am dedicated to quantifying, and understanding those bits that are the most important to us as individuals, professionals, and business owners. Like many other folks who work in the tech sector I have always been good at paying attention to the digital bits, I am just not as good at others when monetizing these bits, adding to my own wealth.
When you talk about this world in the world as much as I have, you see just a handful of responses. Most "normals" aren't very interested in things at this level--they just want to benefit from the Internet and aren't really interested in how it works. People who are associated with the tech sector and understand the value of these bits, often do not see them as "your" bits, they seem them as their bits--something they can extract value from, and generate revenue. Then there are always a handful of "normals" who are interested in understanding more, because of security and privacy concerns, as well as a handful of tech sector folks who actually care about the humans enough to balance the desire to just make profits.
The Imbalance In All OF This Is What Fascinates Me
The majority of the "normals" don't care about the complexity of the bits they generate, as well as who has access to them. Folks in the tech sector love to tell me regularly that people don't care about this stuff, they just want the convenience, and for it all to work. However, they are also overwhelmingly interested in the bits you generate each day because there is plenty of money to be made extracting insights from your bits, and selling those insights, as well as the raw bits to other companies so they can do the same. This is why EVERYTHING is being connected to the Internet--the convenience factor is just to incentivize you enough so that you plug it in.
There is a reason this latest wave of tech barons like Facebook, Twitter, and Google are getting wealthy--it is generated from the exhaust from our personal and professional lives. Facebook doesn't make money from me just having a Facebook account, they make money off me regularly using Facebook, sharing details of my life via my home and work computers, and mobile devices. This "exhaust" from our daily lives is why Silicon Valley is looking to invest money in each wave of startups, and why corporations, law enforcement, and government agencies are looking to get their hands on it. If these bits are so important, valuable and sought after, why shouldn't the average citizen be more engaged as part of this? I mean they are our bits, right?
Where the imbalance really comes in for me in all of this, is that technologists agree that these bits are valuable. Many even hold them up to be the source of intelligence for the next generation of algorithms that are increasingly governing our lives. Some even hold up these bits to almost a religious status, and that someday humans will be able to be simply uploaded to the Internet because these bits reflect our human soul. However, the average person shouldn't worry themselves with these bits, do not have any rights to access these bits, let alone be able to assert any sort of control and ownership over these bits and bytes? It is troubling that this is the norm for everything right now.
I operate somewhere in the pragmatic and realistic middle I guess. I do not believe these bits and bytes represent our soul, being, or any other religious stance. I do believe they are a digital reflection of ourselves, though. I do think that my thoughts in my Evernote notebook should not be purely seen as a commodity to enrich their algorithms or the photos of my daughter on Instagram being open for use in advertising--they are my bits. I am fine with the platform that I use generating revenue from my activity, however, I do expect them these bits as mine, and this is a shared partnership when it comes to being a steward of my bits--something I feel has severely gotten out of balance in the current investment climate.
Who Do These Bits Belong To?
Only in recent years have I seen more tangible discussion around who these bits belong to. Startup founders I've discussed this with feel pretty strongly that these bits wouldn't exist if it wasn't for their platform--so they belong to them. Increasingly the ISP, cable, and mobile network providers feel they have some right to this information as well. Local and the federal government has also pushed pretty hard to clarify that they should have unfettered access to these bits. Sadly, there aren't more discussions that actually include end-users, the average citizen, and "normals" in these discussions, but I guess this is how power works right? Let's keep them in the dark, while we work this shit out for them.
I guess that I am looking to help shift this balance by being a very vocal "average citizen", and active participant in the tech sector, one who cares about privacy and security. This is why I do API Evangelist--to shine a light on this layer of our increasingly digital world, and be transparent about my own world, so that I can help educate other "normals", and people who care, about this version of our digital self that is being generated, cultivated, and often exploited online without any respect for us as individual human beings. To help quantify the digital version of myself, I wanted to walk through my footprint, and share it with others.
What Makes Up The Digital Version Of Kin Lane?
Alongside studying the world of APIs, and how the bits and bytes are being moved around the Interwebz, I regularly try to assess my own online presence, define, a regularly redefine who is Kin Lane on the Internet--this is how I make money, and pay my rent, so it is very important to me. I am always eager to dive in and quantify this presence, because the more I am aware of this digital presence, the more I am able to make it work in the service of what I want, over what other people, companies, and the government want for me. Let's take a stroll through the core services that define my digital self in 2017.
Twitter (@kinlane & @apievangelist)
Starting with the most public version of myself, I'd say Twitter is one of the most important digital performances I participate in each day. I have 10 accounts I perform on, Tweeting about what I see, what I write, and many other elements of my personal and professional life. Here are some of the key bits and bytes:
- Profile - My Twitter profile, and account details.
- Tweets - My words, thoughts, and observations of the world around me.
- Messages - My private messages with people I know.
- Media - Photos and videos that I have often taken myself.
- Lists / Collection - Curation of the world of Twitter as I see it.
- Location - Where I am when I'm engaging with my network.
- Friends - The people that I know and connect with on Twitter.
- Links - Links to important other important sites in my world.
I get that Twitter needs to generate revenue by providing insights about the relationships I have, the role I play in trending topics, and other interesting insights they can extract in aggregate. However, these are my words, thoughts, messages, photos, and friends. Even though these activities connect in the cloud at twitter.com, they are occurring in my home and my physical world via my laptop, tablet, and mobile phone.
Github, Github Pages, and Github Gists (@kinlane, @apievangelist)
Twitter is an important channel for me, but I would say that Github plays the most critical role in my professional world of all the services I depend on. I host the majority of my public presence using Github Pages, and I manage all the API driven code that runs my world on Github. Aside from Twitter, I'd say that Github is one of the most important social, and communication channel in my world, used to keep track on what individuals and companies are up to.--here are the bits I track on via Github:
- Profile - My Github profiles for Kin Lane and API Evangelist.
- Organizations - For each of my major project areas I have separate organizations.
- Users - The people that I work with, have relationships with, or just stalk them on Github.
- Projects - I'm beginning to organize my work into projects instead of organizations.
- Repositories - All my public websites, backend APIs, and code that I write lives as repos.
- Issues - I leverage Issues as intended by Github, but also as a tasking system for myself.
- Gists - I use Github Gists as part of my storytelling, sharing code, data, and other bits of code.
- Search - I manually, and as part of automated work conduct regular searches across Github.
When it comes to what matters in my world as the API Evangelist, Github is the most critical. Many providers have dialed in how to game Twitter, LinkedIn, Facebook, and other popular channels, but it is much harder to fake a presence on Github. This is where I track on companies and individuals who are doing some the most valuable work on the web--the most important part is it's out in the open.
Facebook (@kinlane & @apievangelist, & @dronerecovery)
I would put Facebook more in the personal category, than a professional one, as I have never really found a voice for API Evangelist on Facebook. I tend to use the platform for coordinating with my friends and family, but I am increasingly opening up to the wider public and including folks from my professional world. This doesn't change the conversation around the value of these bits and bytes in my daily life:
- Profile - My Facebook profile, details, and pages.
- Posts - My words, thoughts, and observations of the world around me.
- Friends - My friends, and family, and their networks.
- Images - The photos and images I post to Facebook.
- Videos - The videos and animations I publish to Facebook.
- Links - Links to important other important sites in my world.
- Pages - The versions of this detail that get posted to my Facebook pages.
I do not publish much else to Facebook. I don't check-in, or really use it as part of my business operations, so what I do is pretty basic. I am spending more time crafting how-to API videos, and other content and media that is focused on a Facebook audience, so this will change over time, making my perspective on ownership and control over my bits and bytes even more important in coming years.
YouTube (@kinlane, @dronerecovery, @algorotoscope)
Historically most of my videos on Youtube have been speaking at conferences, universities, and other gatherings. I've been slowly shifting this, as I hope to produce more how-to API content, as well as significantly shifted with the introduction of my Drone Recovery and Algorotoscope work. Youtube's role in my personal and professional world is only going to continue to expand:
- Profile - My Youtube profiles, detail, and history.
- Videos - The videos I publish on Youtube.
- Channels - How I organize things on Youtube.
- Playlists - The curation of content I publish, and discovery.
- Comments - The comments on the videos I've published.
- Search - My search history on Youtube.
I do not have a lot of experience working with Youtube in a professional capacity and do not spend a huge amount of time on Youtube watching videos, but its network effect is undeniable. While Twitter and Facebook are growing in significance when it comes to my video bits, Youtube is still number one.
While not a very public part of my online self, I use Dropbox a lot for managing a more human side of my online storage of files, videos, and other digital objects. I leverage Dropbox as sharing for planning my conference(s), working on video projects, and a variety of other projects, across many different teams. Here are some of the bits I manage with Dropbox:
- Profile - My profile, details, and Dropbox account.
- Files - Everything I'm storing on Dropbox.
- Users - The users I've added to teams and collaborated with.
- Sharing - The management of sharing files.
I'm increasingly using the Dropbox API as a way to automate my document, image, video, and other media processing workflows in a way that allows me to plug humans into different stops along the way. It helps me maximize control over the media and other objects I need throughout the work on a daily basis.
Google has been a significant player in helping me define my digital self since 2005. However, in recent years I've made an effort to reduce my dependence on Google, but it is something I'm not able to ever do completely. I see Google as the poster child for why we should be mindful of our digital self, the bits we manage online, and why we should be skeptical of free software and tools for our businesses. Here are the bits I'm leveraging Google to manage:
- Profile - My overall Google profile for personal and my business domains.
- Email - Email for my personal and business profiles going back to 2005.
- Documents - All my files for my personal and business profiles going back years.
- Calendar - My secondary calendar, that is tied to my local calendar.
- Search - My search history, and since Google is my primary search engine--significant.
- Location - Where I am at, where I'm going, and where I've been.
- Translation - Translating of near real-time conversations I have with people around the world.
- Analytics - The analytics for all of my public domains.
- Forums - Conversations from a handful of groups I'm part of and run for other projects.
- News - I use Google News as primary news aggregate, with personalized sections and searches.
- Hangouts - I still use Google Hangouts with folks around the world.
I would point the finger at shifting the landscape from business software to free, surveillance-style business models. I also point the finger at Google for shifting the overall focus of a free, open, and democratic Internet, towards something that is about making money, generating leads and clicks, and fueling disinformation networks--"Index The Worst Parts of the World and Human Race".
I like Slack. I get it. However I find it to be a little too noisy and busy, and I find that I can only handle about three active channels at any point in time. Even with all the noise, it is a pretty interesting approach to messaging, and one of my favorite API platforms, so I can't help but put the platform to use in my business and a personal world in helping me manage the following bits:
- Profile - My Slack profile that spans multiple channels.
- Channels - The various Slack channels I'm part of or manage.
- Files - Any file I share as part of my conversations on Slack.
- Groups - The different groups I tune into and participate in.
- Search - My search history across the channels I participate in.
- Bots - Any automate assistant or other tools I've added.
I'm not a big believer in bots and automated approaches to messaging, but I get why others might be into it. I'm a little more critical about where and how I automate things in my world, and a little biased because I tend to write the code that assists me in operating my world. When it comes to bots, I'm hoping that users are more critical about which bots they give access to, and bot developers provide more transparency into the data they collect, and their security practices.
I have my own conference that I help produce, as well as have played a role in other conferences, so EventBrite can play a pretty significant role in managing the bits for this part of my professional world. Eventbrite helps me manage my bits for use across numerous events but also helps me manage my relationship with the bits of my conference attendees, which are very important to me.
- Profile - My profile and account on Eventbrite.
- Events - The events I've attended and produced on the platform.
- Venues - Venues of the events I'm watching, attending, or operating in.
- Users - The users who are attending events I'm involved with.
- Orders - The orders made for those attending my events.
- Reports - Reports on the details of each event I've managed.
- Media - The media I've uploaded and managed to support events.
Eventbrite is an interesting convergence of the physical and digital worlds for me. Tracking on the details of the types of events I attend, which are primarily tech events, but occasionally also other more entertainment, political, and other types of events. If you've ever run a lot of conferences as I have, you understand the importance, value, and scope of information you are tracking about everyone involved. A significant portion of the conference's business model is based on sponsors having access to these bits.
when it comes to smaller gatherings, I depend on Meetup to help me manage my involvement. I actively use Meetup to stay in tune with many of the API Meetup groups I frequent, as well as keep tabs on what other interesting types of gatherings are available out there. Here are the bits I track as part of Meetup usage:
- Profile - My Meetup profile, account, and messages.
- Groups - The Meetup groups that operate in various places.
- Events - The Meetup gatherings I've attended or thrown.
- Locations - The different cities where Meetups occur.
- Venues - The venues I've attended and thrown Meetups.
- Members - The members of Meetup groups I'm involved in.
- Comments - The conversations that occur within a group and part of events.
- Pictures - Any pictures that get uploaded in support of an event.
I am hyper aware oft he information Meetup tracks from my own usage when attending Meetups, but also my experience running Meetups. I also use the Meetup API to do research on locations, and the different areas that are related to the API industry--so I know what data is available, or not when it comes to profiling users.
I use CloudFlare to manage the DNS for all my domains. They provide me with a professional, API driven set of tools for managing the addressing of my online presence. Most importantly, CloudFlare helps me manage the portion of my digital presence that I own the most, and allows me to extract the most value from the bits I generate each day.
- Profile - My CloudFlare profile, and details of my domain registration.
- Firewall - My firewall settings for my domains.
- DNS - The addressing for all my public and private domains.
- Certificates - The certificates I use to encrypt data and content in transit.
- Analytics - The analytics of traffic, threats, and other details of this layer.
This is a very important service for me, my business and brand. Choosing the right service to manage the frontline for your online presence is increasingly important these days. CloudFlare plays an important role when it comes to securing my bits, and making sure I can operate safely on the Internet, and stay functioning as a viable independent business.
I use Gumroad to help me sell my white papers, essays, guides, images, video, and other digital goods. They provide a simple way for me to create new products from the digital goods I produce, and give me an interface, as well as an API for adding, managing, and selling things that are as part of my operations.
- Profile - My GumRoad account, profile, and history.
- Papers - All my white papers I sell via Gumroad.
- Guides - All the guides I sell via Gumroad.
- Videos - All the videos I sell via Gumroad.
- Images - All the images and collections I sell via Gumroad.
- Offer Codes - Offer codes I generate for different campaigns.
- Sales - All the sales I've made across my sites, and Gumroad store.
- Subscribers - Any subscribers I have to subscription based products.
I like Gumroad because it allows me to sell digital goods one my site, and they just get out of the way. They aren't looking to build a network effect, just empower you to sell your digital goods by providing embeddable tools, and easy to use ordering, checkout, and payment solutions--they make money when you sell things and are successful.
A lot of what I do targets a business audience, making LinkedIn a pretty important social and content platform for me to operate on. I do not agree with LinkedIn's business strategy, and much of what they have done to restrict their developer community makes them of lower value to me, but I still use them to manage a lot of bits online.
- Profile - My LinkedIn profile and resume.
- People - People I know and have relationships with.
- Companies - The companies and organizations I follow.
- Jobs - Employment postings that I keep an eye on, and include in work.
- Groups - The LInkedIn Groups I am part of and manage.
- News - Any news I curate and share via the platform.
- Essays - The essays I write and share via LinkedIn publishing.
Like Facebook, I'm looking to expand my business usage of LinkedIn, and minimize its role in my personal world. Their lack of APIs for some of the important bits I manage on the platform makes it hard for me to expand and automate this very much. As a result, it tends to be just a destination for publishing news and engage with people via messaging and groups.
Amazon & Amazon Web Services
Amazon is another cornerstone of my operations. They provide me with the core elements of my business operations, and are what I'd consider to be a wholesale provider of some the bits I put to work across my operations. I guess I have to consider the consumer relationship I have with Amazon as well, and the role they play in making sure physical goods arrive on my doorstep, as well as the digital bits I manage on their platform.
- Profile - My Amazon and Amazon Web Services Account.
- Compute - All my compute resources operate in AWS.
- Storage - Amazon is my primary storage account.
- Database - I run several MySQL database on Amazon.
- Shopping - I do a lot of shopping for my personal and professional worlds here.
- Books - I buy and publish books using Amazon publishing.
- Guides - I am looking to expand my publishing of industry guides on Amazon.
Amazon is a very tough service for me to replace. I keep everything I run there as generic as possible, staying away from some of their newer, more proprietary services, but when it comes to what I can do at scale, and the costs associated with it--you can't beat Amazon. Aside from Amazon Web Services, I depend on Amazon for my camera equipment, printer, and other things that help me generate and evolve my bits both on, and offline.
This is is a very specialized service I use, but has provided a core part of my world for over three years now. I use Pinboard to bookmark news and blog posts I read, images, videos, and any other digital bit of others that I keep track on. Pinboard is my memory, and notebook for everything I read and curate across the Web.
- Profile - My public Pinboard profile, and the private account.
- Links - All the links I've curated on Pinboard.
- Tags - How I've tagged everythign I've curated -- linked to my internal tagging.
I have a different relationship with almost every service provider listed on this page, and in some of the cases I have a love/hate relationship with the platform--I need their service, but I don't agree with all of their policies and practices. This IS NOT the case with Pinboard. I absolutely love Pinboard. It is a service that does one thing and does it well, and I'm happy to pay for the service they offer. They do not have any surprises when it comes to helping me manage my bits. They are extremely clear about our partner relationship.
This is where I publish the audio from my world. I've published recorded webinars, audio versions of talks I've given, and Audrey and I publish our Tech Gypsies podcast here. Like Youtube, SoundCloud is a place I'm looking to expand on how I use the platform, to manage the audio side of my digital self with these bits:
- Profile - My SoundCloud profile and account.
- Tracks - Any audio tracks I've published to SoundCloud.
- Playlists - The playlists I've created for my work and others.
- Comments - The ocmments I've made, or have been made on my work.
- User - Any users I've engaged with, liked, followed, or otherwise.
At Tech Gypsies (my parent company) been publishing our podcast to SoundCloud each week since April of 2016, and is something we are going to keep doing in 2017. I'm looking to capture more sounds that I can publish here, to accompany some of my videos. I'm also looking at some of the music and sounds of other artists for inclusion in some of my work.
It took me a while to come around to the network effects of publishing on Medium, but I'm there. While I'm not nearly as active here as I am on my other domains, I do syndicate certain pieces here to Medium to tap into these benefits.
- Profile - My Medium profile which is linked to my Twitter profile.
- Users - Anyone that I've followed or has follwed me.
- Publications - The publications I've created or participated in.
- Posts - Any posts I've published to Medium.
- Images - The images I include with any of my posts.
After a recent announcement that they would be downsizing at Medium, the benefits of my approach to managing my bits were clear. Many folks have told me I should move my blogs to Medium, and while I'll keep investing time into the platform, it will never receive the same attention as my own properties, because of the revenue they bring to the table.
I have been a long time Flickr user, and was a paid customer for many years. I still use the platform to manage certain collections of images, including my APIStrat conference, but its dominance as an image and photo manage channel has been reduced due to concerns about the stability of it's parent company Yahoo.
- Profile - My profile and account for Flickr.
- Photos - The photos I've uploaded to Flickr.
- Collections - The collections I've created of my photos.
- People - The people I follow, and engage with around my photos.
- Places - The places where photos have been taken.
- Tags - The tags I apply to photos I've published.
In addition to managing my own digital presence using Flickr, I also use it to discover photos and people, by navigating tags and conducting searches via the site and API. This service contributes pretty significantly to my digital presence because I use them in my blog posts, and other publishing I do online (licensing allowing).
I swore off Instagram when they changed their terms of service temporarily so that they could use your photos in advertising without your permission--it is something they have backed off, but I still kind of lost faith in them. I still try to maintain the platform by publishing photos there, and I've recently setup an account for my photography, drone, and algorotoscope work, so I expect my usage will grow again.
- Profile - My Instagram profile and account.
- Users - The users I follow, and who follow me.
- Images - The photos I have taken and published.
- Video - The videos I have taken and published.
- Comments - The comments I've made, and are made on my work.
- Likes - Anything I've liked, and flagged as being intersting.
- Tags - How I tag the images and video I publish.
- Locations - The locations where I've taken photos and videos.
I really like Instagram as a solution. My only problem with it really is that they are part of the Facebook ecosystem. As an image management solution, and social network around photos I think its a great idea, I'm just not a big fan of their priorities when it comes to licensing and surveillance.
My primary payment platform for my business is still Paypal. If I am building an application, or scaling a business I'd be leveraging Stripe, Dwolla, and other leading providers, but because there is still a human element in my payment scenarios I heavily use Paypal.
- Profile - My personal, and business profile on Paypal.
- Payments - Any payments I've made and received.
- Payors - People and companies who have paid me money.
- Payees- People and companies who I've paid money.
- Invoices - The invoices associated with payments.
My Paypal is a look at the financial side of my operations. It helps me centralize my money in a way that helps me work with a variety of services, as well as the underlying bank account(s). I can integrate into my site, and use the embeddable tooling that they provide to integrating payments into my sites, and the applications or tooling I develop.
The majority of automation that occurs on my platform is hand coded because I have these skills. However, there are many services that the integration platform as a service provider Zapier offers which I just can't ignore. Zapier makes integration between common API driven services dead simple--something any non-developer can put to use, and even programmers like me can put to work to save time
- Services - All of the services I've integrated with Zapier to help automate things.
- Authentication - The authentication tokens I've approved for integration to occur.
- Connections - The formulas employed from connection to connection.
Zapier has an API, but it doesn't provide control over my account, services, and the connections, or as they call them--zaps, as I'd like to see. It is a much better solution than IFTTT, who takes a proprietary stance about your services and connections, but in my opinion, we will need to evolve more solutions like DataFire.io to help more normals make sense of all of this.
The Core Services I Depend On
There are numerous other services I use, but these 20+ are the core services I depend on to make my personal and professional world work. These are the services I depend on to make a living. Some of these services I pay for, and there is a sensible and fair terms of service in place. Other services I do not pay for and put to use because of the network, and other positive effects...even if there is often a term of service in place that extracts value from my data and content--hopefully the positives outweigh the negative for me in the end.
Now we come to the most important part of my digital self. The portion of it where I get to have 100% control over how things work, and benefit from all of the value that is generated--this is my domain, where I am a domain expert, across a handful of web domains that I own and operate.
- API Evangelist - Where I write about the technology, business, and politics of APIs.
- Kin Lane - My personal blog, where I talk about tech, politics, and anything else I want.
- API.Report - Where I publish the press releases I process each week from the API space.
- Stack.Network - My research into the API operations of leading API providers.
- Drone Recovery - Photos and video from my drone work, in an urban and natural environment.
- Algorithmic Rotoscope - Applying machine learning textures and filters to images and video.
In my world, all roads should begin here and end here. The majority of my work starts here and then gets syndicated to other channels. When this isn't possible, I make sure that I can get my content back out via API, scraping, or good old manual work. My objective in all of this work is to define my entire digital footprint and paint a picture of not just the services I depend on for my online self, but also the valuable little bits that get created, moved around, and ultimately monetized by someone (hopefully that is me).
Identifying The Bits
When you look at my digital self I would say that the two most defined bits of my existence are the blog post and the Tweet. These are the most public bits I produce most frequently, and probably what I'm best known for which also results in me making money. I began focusing on the digital bits I generate and manage as part of my presence on a regular basis because I am the API Evangelist, and I saw companies using APIs to make money off of all of our bits. Amongst it all, I managed to make a living talking about all of this and generate bits (blogs, guides, white papers, talks) that people are interested in, and sometimes they are interested enough to pay me money.
I am not looking to identify all of my bits just so that I can make money off them, and I'm not looking to deprive companies who work hard to develop platforms and useful tools from sharing the value generated by my traffic, data, content, photos, videos, and other bits I generate. I'm just looking to assert as much control over these digital bits as I possibly can because, well....they are mine. Here are some of the most common bits I'm producing, managing, and in some cases making a living from across these services I use to define my digital self:
All of these digital bits exist for a variety of reasons. Some are more meta, some are casual, while others help me be more informed, organized, and ultimately becomes my research which drives the bits that make me money. I want to respect the digital bits of others that I put to use in my work (images, videos, quotes), and I would like folks to be respectful of my digital bits. I could use some help driving traffic to my sites, and I'm happy to help create unique content, media, and other activities to drive traffic to the services I use, but I want it to be fair, equitable, and acknowledge that these are my digital bits, regardless of the platform, tool, and device where it was generated--it is quite likely that occurred on my laptop, tablet, or mobile phone in my home.
The API Driven Marketplace That Is My Digital Self
I understand that I probably think about this stuff a little too much. More than the average person. It is my job, but I can't shake that the average person might want to be a little more aware of this layer of their increasingly digital life. Especially so if you are looking to make a living off your work as an independent progressional (some day), and successfully develop your career in the digital jungle, where everyone is looking to extract as much value from your digital bits, before you ever even can benefit yourself. As a professional, I want to maximize my control over the traffic and other value generated by my labor, asserting control over my digital bits whenever, and wherever I can.
I notice my digital bits flowing around online at this level because this is what I do as API Evangelist, and I'm working to automate as much of it as I can, maximize my digital footprint and presence, while retaining as much control over all of my bits as I possibly can. All of the platforms listed above have APIs. If possible, I only depend on services that have APIs. I don't do this just because of what I do for a living, I do this because it allows me to automate, and syndicate my presence and work more efficiently and effectively. I have active integrations with Facebook, Twitter, Github, and all of the services listed above. I always have a queue of new integrations I need to tackle, and when I do not have time to code, I try to use Zapier or other integration as a service solution.
All of my work is conducted within my domain, within my workshop. Each blog post I write, each image or video I create, and guide, essay, and white paper I produce is crafted on my desktop or the workshops that are kinlane.com, apievangelist.com, and my other web domains, where I operate my open, publicly available digital workbench. Then I use the API for each of the service listed above to then push my bits to each channel, syndicating to Twitter, LinkedIn, Facebook Medium, and to my network of sites using Github. I know the APIs of these services intimately, including the bits that are exchanged with each transaction. I have written code to integrate with, and crafted OpenAPI Specs for each of the services listed above--I can close my eyes and see these bits being exchanged daily, and think constantly about how they are exchanged, bought, sold, and leveraged across the platforms above. I know, I have a problem.
I'm sure that all of this sounds a little OCD to some. Most folks I talk to about this stuff do not care and dismiss as out of their realm. Most of the entrepreneurs who get things at this level are too busy making money from any value being generated, they do not have much interest in everyone else understanding all of this. In my opinion, you will either want to understand things at this level, and assert control over your bits, or you will work on these other platforms, allowing others to benefit from what you do online, in your car, and in your home. I don't mind other people generating value from my digital bits, especially if they provide me with valuable services, and the details of the relationship are disclosed, I am informed, and we are in agreement before entering.
I am just getting to know my digital self. Sadly I do not fully understand the terms of service that guide all of these relationships as well as I know the technical details of our relationship. Now that I have more of a handle on which core services I depend on, as well as the digital bits that are exchanged in these digital relationships I have entered into. I'm currently working on profiling the business models, pricing and plans available for each of these services I depend on. After that, I'd like to take a better look at the terms of service, privacy policies, and other legal aspects of these relationships. Maybe then I'll understand more about how my digital bits are being bought and sold on the open market, and what dimensions of my digital self really exist.
I came across a pretty interesting post on using APIs for research, and the benefits, and challenges that researchers face when depending on APIs. It was another side of API stability and availability that I hadn't considered too much lately. Social media platforms like Twitter and Facebook are rich with findings to be studied across almost any discipline. I regularly find social media API studies at universities from areas like healthcare and Zika virus, algorithmic intellectual property protection, all the way up to US Navy surveillance programs that are studying Twitter.
APIs are being used for research, but there are rarely API platform plans crafted with research in mind. Flexible rate limits, custom terms of service, that give them access to the data they need. I'm assuming that some companies have behind the scenes deals with some universities, or larger enterprise research groups (IBM, etc), as well as government agencies, and police agencies. The problem with this, is that 1) there is no virtual public front door to walk through and understand research levels of access, and 2) the details of partnerships are not publicly, equitable, and auditable by journalists, and other groups.
The author of this essay provides a lot of details regarding what it is like to depend on APIs for your research. Some of them could put your career in jeopardy if the terms of service, and access levels change before you could finish your research, or dissertation. I'm not sure what the responsibility of API providers should be when it comes to making their resources available for research, but it is something I will be exploring further. I will be reaching out to researchers about their API usage, but will also be helping encourage API providers to share their side of things, and maybe eventually formalize how API providers make their valuable resources available for important research.
I have used Shutterstock for some time now when it comes stock images but I've only recently started playing around with their publishing program, hoping to make some money from some of my photos and videos. As with any other channel that I am considering for inclusion in my line-up of tools and services, I am spending time going through their platform and evaluate the tech, business, and political considerations of adding any new service to work into my world.
First, a service should always have an API. This isn't just because of what I do for a living and my obsession with APIs. This is so that I can integrate seamlessly with my existing operations. Another side of this argument is that I will always be able to get my data and content out of a system, but I am working to be a little more proactive than that. I want my system, that operates within my domain to be the lead, and any new channel I adopt only play second fiddle. In this scenario, each photo or video that I publish to Shutterstock will live within my image and video systems and then with the Shutterstock API, I will publish to the Shutterstock domain as I deem worthy.
The Shutterstock API (potentially) gives me more access and control over my digital bits and allows me to do more with fewer resources. I do not have to depend on APIs, or a platform's data portability to get my data and content out, I've always possessed this control and ownership from the beginning. Then this control and ownership is now exercised and strengthened in three areas: technology, business, and politics. I technical have control over my bits. I have business control over where they are sold, by whom, and how much of the action they get. I have political control, and when I want to change, evolve, or end the relationship I can do what I think is best for me, and my API Evangelist business.
I still have a lot of reading and learning to do when it comes to understanding the legal terms of services, and the details of my Shutterstock partnership, but a company having an API is increasingly the first step of any new business relationship for me. If I can help it, I will not add any new business relationship into my world unless there is an API. Of course, there are deviations from this, but as a single operator small business, this is critical for me if I expect to be able to scale the technical side of my operations, while also meeting the business and legal requirements I have in place to help me achieve success in my business.
I've been working with Algorithmia to manage a large number of images as part of my algorithmic rotoscope side project, and they have a really nice omni-platform approach to allowing me to manage my images and other files I am using in my machine learning workflows. Images, files, and the input and output of heavy object is an essential part of almost any machine learning task, and Algorithmia makes easy to do across the storage platforms we use the most (hopefully).
Algorithmia provides you with local data storage--pretty standard stuff, but they also allow you to connect your Amazon S3 account, or your Dropbox account, and connect to specific folders, buckets, while helping you handle all of your permissions. Maybe I have my blinders on with this because I heavily use Amazon S3 as me default online storage, and Dropbox is my secondary store, but I think the concept still is worth sharing..
This allows me to seamlessly manage the objects, documents, files, and other images I store across my operation as part of my machine learning workflow. Algorithmia even provides you with an intuitive way of referencing files, by allowing each Data URI to uniquely identifies files and directories, with each composed of a protocol and a path, with each service having its own unique protocol:
- data:// Algorithmia hosted data
- dropbox:// Dropbox default connected accounts
- S3:// Amazon S3 default connected account
This approach dramatically simplifies my operations when working with files, and allows me to leverage the API driven storage services I am already putting to work, while also taking advantage of the growing number of algorithms available to me in Algorithmia's catalog. In my algorithmic rotoscope project I am breaking videos into individual images, producing 60 images per second of video, and uploading to Amazon S3. Once images are uploaded, I can then run Algorithmia's Deep Filter algorithm against all images, sometimes thousands of images, using their text models, or any of the 25+ I've trained myself.
This approach is not limited to just video and images, this is generic to any sort of API driven machine learning orchestration. Just swap out video and images, with mapping, content, or other resource, and then find the relevant machine learning workflow you need to apply, and get to work. While I am having fun playing with my drone videos and texture filters, the approach can be just as easily applied to streamline any sort of marchine learning workflow.
One additional benefit of storing data this way is I've found Dropbox to be a really amazing layer for including humans in the workflow. I leverage Amazon S3 for my wholesale, compute grade storage, but Dropbox is where I publish images, videos, and documents that I need to put in front of humans, or include them in the machine learning workflow. I find this gives them a role in the process, in a way that gives them control over the data, images, videos, and other objects, on a platform they are already comfortable with. I'd encourage Algorithmia, and other providers to also consider including Google Drive as part of this--it would go a long way logically connected with the human portion of the wokflows.
Anyways, I thought Algorithmia's approach to storage was interesting, worth highlight, and something that other providers might consider implementing themselves.
I just finished crafting API definitions for 66 of the Amazon Web Services. You can find it all on Github, indexed with an APIs.json. While I wish all API providers would do this hard work on their, I do enjoy the process because it forces me to learn a lot of each API, and the details of what providers are up to. I learned quite a bit about Amazon Web Services going through the over 2000 paths that are available across the 66 services.
The Importance Of Consistency Across Teams
When you bounce from service to service within the AWS ecosystem you can tell that consistency is a challenge for Amazon. Consistency is lacking in API design, documentation, and other critical areas. This is something that is actually getting worse with some of their newer projects. While the older AWS APIs aren't the best possible design because they are: "?Action= based", at least they are consistent, and the documentation is using the same template. Some of the newer APIs are better designed, but their documentation is all over the place, and they are deviating from the consistency that seemed to exist with some of the older API efforts.
Clear Picture Of Essential Building Blocks
There are a variety of building blocks employed in support of AWS APIs, but there is a pretty clear definition of what are considered to be the essential building blocks that exist across ALL AWs APIs:
- Documentation - Overall, developer, and API documentation to support the services.
- Getting Started - What you need to get up and going with any of the AWS solutions.
- Frequently Asked Questions - A list of the frequently asked questions asked of each service.
- Pricing - The pricing for using each service, with some providing a calculator to assist.
Amazon also provides a centralized blog, code, support, and what I'd consider to be essential building blocks, and some of the individual services do a good job linking to these resources, but these four are present across ALL of the AWS services, making them clearly considered to be essential.
Relationship Between CLI and API
I think the relationship between CLI and API isn't discussed enough in the API sector but is something that is clearly strong across the AWS ecosystem. I'm seeing more API providers also offer a CLI alongside their API to support different developer tastes, but I think AWS does a good job investing equally in both approaches to putting AWS resources to work. In some cases, I'd say the CLI is better documented than the API, but this wasn't always the case--for the most part they were equally invested in.
API First And Console Second
Another area I think Amazon provides an interesting case study is when it comes to the relationship between their human interface vs their API and CLI solutions. Many companies launch their human interface and secondarily provide the one for programmatic access, where Amazon delivered the API and CLI first, and their console came second. With current releases, this seems like it is in sync, but in early days they were API first. I appreciated the AWS teams that provided me a link directly to the AWS console, dropping right into the human interface for the API I'm working with. I have a ranking score of 1-3 for how coupled a company's API is with their human interface, and I'd put Amazon as a 2, with a moderate amount of coupling--with a ranking of 1 meaning that they are well linked (tightly coupled).
Meet Other Folks Doing Interesting Things
One of the reasons I'm so transparent with all of my work is that it tends to alert folks to what I'm working on, and is something that attracts like-minded individuals who are headed in a similar direction. Shortly after tweeting out my work, Mike Ralphson (@PermittedSoc) shared his Github repository of OpenAPI Specs. This will save me a ton of work in verifying paths, making sure header, parameters, errors, and the underlying data model actually gets completed. I will be setting up scripts to keep my definitions in sync with his collection, as well as other folks collections that I'm keeping an eye on.
Change Will Come With New Products & Services
I knew that AWS had released a number of new products at their annual conference this year, but I haven't had time to dive in. It was interesting to learn about their efforts in the area of machine learning, and Internet of Things. I also got a good look at their authentication, encryption, identity, access management, and other security-related efforts. I feel like this will continue to be an important offering for all the 1000lb gorilla tech giants--security. Us mere mortals will not be able to muster the resources to do at scale, and AWS scale companies will need a buffet of security solutions for API providers.
I'm going to continue refining my Amazon Web Services Stack, but I'm going to also get to work on a similar one for Google and Microsoft. Once I have these three tech giants profiled from this API standpoint, I will step back and see what I can do to compare, and better understand where things are headed. This is tedious work, but I find it worthwhile because it is something that continues to push my understanding of the space forward. As I've said before, crafting an API definition of an API is one of the best ways to get to know an API in an intimate way, second to actually writing some code and integrating with an API for realz.
I enjoy having conversations with "normals" about APIs, especially when they approach me after doing a great deal of research, and are pretty knowledgeable about the landscape, even if they may lack deeper awareness around the technical details. These conversations are important to me because it is these folks that will make the biggest impact with APIs--it won't be the entrepreneurs, developers, architects, and us true believers.
While having one of these conversations yesterday, the topic of API design came up, and we were talking about the differences between seemingly similar APIs like Flickr and Instagram, or maybe Twitter and Facebook. I was asked, "why are these APIs are so different? I thought the whole thing with APIs is that they are interoperable, and make integration easier?" << I love getting asked this because it helps me see the API space for what it is, not the delusion that many of us API believers are peddling.
So why are the designs of APIs so different, even between seemingly similar APIs?
- Integration - APIs make integration into web, mobile, and devices apps easier. It will also make integration with other systems easier. However, very few API providers truly want their APIs to work seamlessly with the competition!
- Silos - Many API providers operate in silos, and I have encountered teams who do almost no due diligence on existing API design patterns, standards, or even looking at the potential competition, and what already exists before crafting their API design strategy.
- Intellectual Property - Not many folks see the separate between their API design, the naming, ordering, and structure of the interface, and their backend API code, resulting in some distorted views of what is proprietary and what is not.
- Venture Capital - The investors in many of the companies behind APIs are not interested in being open and interoperable with others in their industry, resulting in a pretty narrow, and selfish focus when it comes to API design patterns.
These are just a handful of the reasons why APIs are so different. It can be hard for people not immersed in the world of technology to cut through the hype, walled garden belief systems, and delusions we peddle in the world of APIs. What makes it even worse, is when you see APIs become a facade for a variety of industries, and begin masking the unique hype, belief systems, and delusions that exist in these verticals. #FUN
When you are only focused on the technology of APIs this all seems simple, and pretty straightforward. Once you begin layering in the business of APIs things get more complicated, but are doable. It is when you start considering the politics of APIs you begin to see the darker motivations behind doing APIs, seeing more of the illnesses that plague the API sector, and infecting each vertical it touches. All of this contributes to many APIs never living up to the hype, or even pragmatic expectations, and will continue to hurt the bottom line of companies who are doing APIs.
I was doing some planning around a potential business model for commercial implementations of OpenReferral, which provides Open211 open data and API services for cities, allowing citizens to find local services, and I had separated out two types of metrics: 1) API calls 2) API traffic. My partner in crime on the project asked me what the difference was, looking for some clarification on how it might possibly contribute to the bottom line of municipalities looking to fund this important open data work.
So, what is the difference between API call and API traffic in this context?
- API Call - This is the measurement of each call made to the API by web, mobile, and device applications.
- API Traffic - This is the measurement of each click made via URLs / URIs served up as part of any API response.
In this context, we are looking to provide municipalities, non-profit organizations, and even commercial efforts that are delivering 211 services in cities around the world. I am not suggesting that every bit of revenue and profit be squeezed out of the operation of these important services, I am simply suggesting that there are ways to generate revenue that can become important in keeping services up and running, and impact the quality of that services--it takes money to do this stuff right.
Think of API traffic like an affiliate program or in service of lead generation. This approach requires the usage of some sort of URL shortener services so that you can broker, and measure each click made on a link served up by an API. This opens up other security and privacy concerns we should think about, but it does provides a potential layer for generating valuable traffic to internal, and partner web and mobile applications. This is just one of several approaches I am considering when we are thinking about monetization of open data using APIs.
I look at a lot of API portals and developer areas , and experience a number of innovative approaches from startups, as well as a handful of leading API providers, but the Lufthansa Airlines API portal (which recently came across on my radar) I feel represents the next wave of API providers, as the mainstream business world wakes up to the importance of doing business online in a machine readable way. Their developer program isn't anything amazing, it's pretty run of the mill, but I think it represents the minimum bar for SMB and SMEs out there in 2017.
The Lufthansa developer portal has all the basics including documentation, getting started, an application showcase, blog, and they are using Github, Stack Overflow, Twitter, and have a service status page. They provide APIs for their core business resources including cities, airports, aircraft, and the schedule of their flights. Honestly, it is a pretty boring, mundane representation of an API, something you are unlikely to find written up in Techcrunch, but this is why I like it. In 2017, we are getting down to the boring business of doing business on the web (maybe security will come soon?).
I'm hoping this is what 2017 is all about when it comes to APIs--where average small businesses and enterprises getting their API operations up and running. Its is like 2002 for websites, and 2012 for mobile--APIs are what you do if you are doing business online in 2017. They aren't the latest tech trend or fad, it is about acknowledging there are many applications and systems that will need to integrate with your resources, and having simple, low cost, machine-readable APIs is how you do this. Let's all get down to business in 2017, and leave behind the hype when it comes to the API life cycle.
This is my walk-through of the concepts involved with the monetization of public data using APIs. In this work I am not advocating that companies should be mindlessly profiting from publicly available data, my intent is to provide a framework for organizations to think through the process of generating revenue from commercial access to public data, acknowledging that it costs money to aggregate, serve up, and keep data up to date and usable for the greater public good--if public data is not accessible, accurate, and up to date it is of no use to anyone.
I have long argued that companies and even government agencies should be able to charge for commercial access to public data and be able to generate revenue to cover operational costs, and even produce much-needed funds that can be applied to the road map. My work in this has been referenced in existing projects, such as the Department of Interior and Forest Service looking to generate revenue from commercial access and usage of public data generated by the national parks systems. In my opinion, this conversation around generating revenue from publicly available digital assets should be occurring right alongside the existing conversations that already are going on around publicly available physical assets.
Building Upon The Monetization Strategies Of Leading Cloud Providers
My thoughts around generating revenue from public open data is built upon monitoring the strategies of leading online platforms like Amazon Web Services, Google, and others. In 2001 a new approach to providing access to digital resources began to emerge from Internet companies like Amazon and Salesforce, and by 2016, it has become a common way for companies to do business online, providing metered, secure access to valuable corporate and end-users data, content, and other algorithmic resources. This research looks to combine these practices into a single guide that public data stewards can consider as they look to fund their important work.
Do not get me wrong, there are many practices of leading tech providers that I am not looking to replicate when it comes to providing access to public data, let alone generating revenue. Much of the illness in the tech space right now is due to the usage of APIs, it is due to a lack of creative approaches to monetizing digital assets like data and content, and terms of service that do not protect the interest of users. My vantage point is the result of six years studying the technology, business, and politics of the API sector, while also working actively on open data projects within city, state, and federal government--I'm looking to strike a balance between these two worlds.
Using Common Government Services As A Target For Generating Much-Needed Revenue
For this research, I am going to use a common example of public data, public services. I am focusing in this area specifically to help develop a strategy for Open Referral but it is also a potential model that I can see working beyond just public services. I am looking to leverage my existing Open Referral work to help push this concept forward, but at the same time, I am hoping it will also provide me with some public data examples that are familiar to all of my readers, giving me with some relevant ways to explain some potentially abstract concepts like APIs to the average folk we need to convince.
For the sake of this discussion things down and focus on three areas of public data, which could be put to work in any city across the country:
- Organizations - The business listings and details for public and private sector agencies, businesses, organizations, and institutions.
- Locations - The details of specific locations which organizations are providing access to public services.
- Services - The listings and details of public services offered at the municipal, county, state, or federal levels of government.
Open Referral is a common definition for describing public services organizations, locations, and services, allowing the government, organizations, institutions, and companies to share data in a common way, which focuses on helping them better serve their constituents--this is what public data is all about, right? The trick is getting all players at the table to speak a common language, one that serves their constituents, and allows them to also make money.
While some open data people may snicker at me suggesting that revenue should be generated on top of open data describing public services, the reality is that this is already occurring--there are numerous companies in this space. The big difference is it is currently being done within silos, locked up in databases, and only accessible to those with the privileges and the technical expertise required. I am looking to bring the data, and the monetization out of the shadows, and expand on it in a transparent way that benefits everybody involved.
Using APIs To Make Public Data More Accessible and Usable In A Collaborative Way
Publicly available data plays a central role in driving websites, mobile applications, and system to system integrations, but simply making this data available for download only serves a small portion of these needs, and often does so in a very disconnected way, establishing data silos where data is duplicated, and the accuracy of data is often in question. Web APIs are increasingly being used to make data not just available for downloading, but also allow it to be updated, and deleted in a secure way, by trusted parties.
For this example I am looking provide three separate API paths, which will give access to our public services data:
- http://example.com/organizations/ - Returns JSON or XML listing and details of organizations for use in other applications.
- http://example.com/locations/ - Returns JSON or XML listing and details of organizational locations for use in other applications.
- http://example.com/services/ - Returns JSON or XML listing and details of public services for use in other applications.
A website provides HTML information for humans, and web APIs provides machine readable representations of the same data, making it open for use in a single website, but also potentially multiple websites, mobile applications, visualizations, and other important use cases. The mandate for public data should ensure it isn't available on a single website but is as many scenarios that empower end-users as is possible. This is what APIs excel at, but is also something that takes resources to do properly, making the case for generating revenue to properly fund the operations of APIs in the service of the public good.
The Business of Public Data Using Modern Approaches to API Management
One of the common misconceptions of public web APIs is that they are open to anyone with access to the Internet, with no restrictions. This might be the case for some APIs, but increasingly government agency, organizations, and institutions are making public data available securely using common API management practices defined by the Internet pioneers like Salesforce, Amazon, and Google over the last decade.
API management practices provide some important layers on top of public data resources, allowing for a greater understanding and control over how data is accessed and put to use. I want to provide an overview of how this works before I dive into the details of this approach by outlining some of the tenets of an API management layer:
- Users - Requiring users to register, establishing a unique account for associated all API and public data activity.
- Applications - Requiring users to define the application (the web, mobile, visualization, etc.) and other viable information regarding their access to the public data.
- Keys - Issuing of unique API keys for each application, requiring their inclusion in all consumption of public data via the API.
- Service Composition - Placement of public data resource (organizations, locations, services) into tiers, defining which APIs different users have access to and the scope of that access.
- Resource Specific - Allowing access to specific data resources to a select audience.
- Read / Write - Restricting write access to select users and applications.
- Data Specific - Limiting which data is returned, filtering based on who is requesting it.
- Rate Limits - All APIs are rate limited, allowing for different levels of access to public data resources, which can be defined in alignment with the costs associated with operations.
- Logging - Each API call is logged, with required user application keys, as well as details of the request and response associated with each API call.
- Analytics - The presence of a variety of visual layers that establish an awareness of who is accessing public data APIs, what they are accessing, and details on how and where it is being applied.
These seven areas provide some very flexible variables which can be applied to the technical, business, and politics of providing access to public data using the Internet. Before you can access the organizations, locations, and service information via this example public services API you will need to be a registered user, with an approved application, possessing valid API keys. Each call to the API will contain these keys, identifying which tier of access an application is operating within, which API paths are available, the rate limits in existence, and logging of everything you consume and add so it can be included as part of any operational analytics.
This layer enables more control over public data assets, while also ensuring data is available and accessible. When done thoughtfully, this can open up entirely new approaches to monetization of commercial usage by allowing for increased rate limits, performance, and service level agreements, which can be used to help fund the public data's mandate to be accessible by the public, researchers, and auditors.
Providing The Required Level Of Control Over Public Data Access
Understandably, there concerns when it comes to publishing data on the Internet. Unless you have experience working with modern approaches to delivering APIs it can be easy to focus on losing control over your data when publishing on the web--when in reality data stewards of public data can gain more control over their data when using APIs over just publishing for a complete download. There are some distinct ways that API providers are leveraging modern API management practices to evolve greater levels of control over who accesses data, and how it is put to use.
I wanted to highlight what can be brought to the table by employing APIs in service of public data, to help anyone make the argument for why providing machine readable data via APIs is just as important as having a website in 2016:
- Awareness - Requiring all data to be accessed via APIs which required keys to be used for ALL applications, combined with a comprehensive logging strategy, brings a new level of awareness regarding which data is accessed, and how it is being used, or not used.
- Security - While API keys are primarily used for logging and analytics, it also ensures that all public data resources are secured, providing tiered levels of access to 3rd parties based upon trust, contributing value to the data, and overall platform involvement--allowing data to be securely made available on the open web.
- Quality Control - APIs act as central gatekeeper regarding how data is updated, evolved, and deleted, allowing for a centralized, yet also potentially distributed, and collaborative approach to ensuring public data is accurate, possessing a high level of quality control.
- Terms of Service - All API usage is governed by the legal terms of service laid out as part of platform operations, requiring all users to respect and follow terms of service if they expect to maintain their public data API keys.
- Governance - Opening up the overall management of the availability, usability, integrity, and security of the public data which may include oversight from governing body or council, a defined set of procedures, and a plan to execute those procedures.
- Provenance - Enabling visibility into the origins, history, and oversight of public data, helping establish the chain of custody regarding shared use of valuable data across platform operations.
- Observability - Allowing for the observability of data resources, and its contributors and consumers using existing platform outputs and mechanisms, enabling high levels of awareness through the API management framework employed as part of platform operations, meeting service level agreements, and expected availability.
It is important to discuss, and quantify this control layer of any public data being made available via APIs if we are going to talk about monetization. Having APIs is not enough to ensure platform success, and sometimes too strict of control can suffocate consumption and contribution, but a lack of some control elements can also have a similar effect, encouraging the type of consumption and contribution that might not benefit a platform's success. A balanced approach to control, with a sensible approach to management and monetization, has helped API pioneers like Amazon achieve new levels of innovation, and domination using APIs--some of this way of thinking can be applied to public data by other organizations.
Enabling and Ensuring Access To Public Data For Everyone It Touches
Providing access to data through a variety of channels for commercial and non-commercial purposes is what modern API management infrastructure is all about. Shortly after possessing a website became normal operating procedure for companies, organizations, institutions, and government agencies, web APIs began to emerge to power networks of distributed websites, embeddable widgets, and then mobile applications for many different service providers. APIs can provide access to public data, while modern API management practices ensure that access is balanced and in alignment with platform objectives--resulting in the desired level of control discussed above.
There are a number of areas of access that can be opened up by employing APIs in the service of public data:
- Internal - APIs can be used by all internal efforts, powering websites, mobile applications, and other systems. The awareness, logging, and other benefits can just as easily be applied to internal groups, helping understand how resources are used (or not used) across internal groups.
- Partner - After internal access to data resources, access levels can be crafted to support partner tiers of access, which might include access to special APIs, read and write capabilities, and relaxing of rate limits. These levels often include service level agreements, additional support options, as well as other benefits.
- Public - Data can be made publicly available using the web, while also maintaining the quality and security of the data, keep the access as frictionless as possible, while ensuring things stay up and running, and of expected quality and availability.
- Privacy - Even with publicly available data there is a lot to consider when it comes to the privacy of organizations, locations, and services involved, but also the logging, and tracking associated with platform operations.
- Transparency - One important characteristic of API platform is transparency in the API management layer, being public with the access tiers, resources available, and how a platform operates--without necessary transparency, consumers can become distrustful of the data.
- Licensing - Ideally all data and all schema in this scenario would be licensed as CC0, putting them into the public domain, but if there are license considerations, these requirements can be included along with each API response, as well as in platform documentation.
- Platform Meta API - APIs do not just provide access to the public data, they also provide access to the API management layer for the public data. Modern API management allows for API access to the platform in the several important ways:
- Users - Providing API access to user's data and usage.
- Apps - Providing API access to applicaiton level data and usage.
- Tiers - Providing API access to platform tiers and details.
- Logs - Providing API access to the platform log files.
- Billing - Providing API access to the platform billing for access.
- Analytics - Providing API access to the analytics derived from logs, billing, and usage.
- Observability - An API management layer on top of public data makes data access observable, allowing platform operators, government agencies, and potentially 3rd party and independent auditors--observability will define both the control as well as access to vital public data resources.
In a world that is increasingly defined by data, access to quality data is important and easy, secure access via the Internet is part of the DNA of public data in this world. API management provides a coherent way to define access to public data, adhering to the mandate that the data is accessible, while also striking a balance to ensure the quality, reliability, and completeness of the public data.
There has been a lot of promises made in the past about what open or public data can do by default when in reality opening up data is not a silver bullet for public services, and there is a lot more involved in successfully operating a sustained public data operation. APIs help ensure data resources are made available publicly, while also opening up some new revenue generation opportunities, helping ensure access is sustainable and continues to provide value--hopefully find a balance between public good and any sensible commercial aspirations that may exist.
APIs Open Up Many Potential Applications That Support the Mission
As doing business on the web became commonplace in the early 21st century, Amazon was realizing that they could enable the sales of their books and other products on the websites of their affiliate partners by using APIs. In 2016 there are many additional applications being developed on top of APIs, with delivering public data to multiple web sites being just the beginning.
- Web - It is common for websites to pull from a database. Increasingly APIs are being used to drive not just a single website, but networks, and distributed websites that draw data and content from a variety of sources.
- Mobile - APIs are used to make data and content available across a variety of mobile applications, on different platforms.
- Embeddable - Delivering data to buttons, badges, bookmarklets, and widgets that can be embedded across a variety of websites, and applications.
- Conversational - Using data in conversational interfaces like for bots, messaging, and voice-enabled applications.
- Visualizations - Including data in visualizations, showing API consumption, and platform usage around public data.
- iPaaS / ETL - Enabling the migration of public data to and from other external 3rd party platforms using traditional ETL, or more modern iPaaS solutions powered via the API.
- Webhooks - Notifying external systems of relevant events (location or service update) by pushing to URLs via what is called a webhook.
- Spreadsheets - Publishing of data to Microsoft Excel or Google Spreadsheet using the public data APIs, as well as spreadsheet APIs.
This is just an overview of the number of ways in which a single, or multiple APIs can be used to deliver public data to many different endpoints, all in service of a single mission. When you consider this in support of public services, a bigger picture of how APIs and public data can be used to better serve the population--the first step always involved standardized, well-planned set of APIs being made available.
The Monetization Requirements Around Public Data API Operations
This is where we get to the portion of this discussion that is specifically about monetization of the operations around publishing and maintaining high-quality sources of public data. Before a sensible monetization strategy can be laid out, we need to be able to quantify what it costs to operate the platform and generate the expected value from everyone at the table.
What are the hard costs that should be considered when operating a public data platform and looking to establish some reasonable monetization objectives?
- Data Acquisition - What one-time, and recurring costs are associated with acquiring data. This might include ongoing payouts to API contributors who are adding, updating, and validating data via the API.
- Discover - What was spent to discover data, and identify its use on the platform.
- Negotiate - What time to I have in actually getting access to something.
- Licensing - Are there licensing costs or fees involved in the acquisition of data.
- Development - What one-time, and recurring costs are associated with platform development.
- Normalization - What does it take me to clean up, and normalize a data set, or across content. This is usually the busy janitorial work necessary.
- Validation - What is involved with validating that data is accurate correct, providing sources, and following up on references.
- Database - How much work is being put putting into setting up the database, maintaining, backing up, and delivering optimal levels of performance.
- Server - Defining the amount of work put into setting up, and configuring the server(s) to operate an API, including where it goes in the overall operations plan.
- Coding - How much work goes into actually coding an API. Ideally, open source frameworks are employed to reduce overhead, maintenance, and the resources needed to launch new endpoints.
- Operational - What one-time, and recurring costs are associated with platform development.
- Compute - What costs are associated with providing server compute capacity to process and deliver public data via APIs.
- Storage -What costs are associated with on the disk storage, for both the database and other images, video, and related objects.
- Network - How much bandwidth in / out is an API using to get the job done, as well as any other network overhead.
- Management - What percentage of API management resources is dedicated to the API. A flat percentage of API management overhead until usage history exists.
- Monitoring - What percentage of the API monitoring, testing, and performance service budget is dedicated to this API. How large is the surface area for monitoring?
- Security - What does it cost to secure a single API, as part of the larger overall operations? Does internal resource spend time, or is this a 3rd party service.
Understand The Value Being Generated By Public Data
Now that we understand some of our hard costs, let's have an honest conversation about what value is being generated? First, public data has to offer value, or why are we doing all this hard work? Second, nobody is going to pay for anything if it doesn't offer any value. Let's stop for a moment and think about why we are doing all of this in the first place, and what value is worthy of carving off to drive monetization efforts.
- Direct Value Generation - What direct value is being generated by the public data platform operations.
- Usage - How is usage wielded as part of value generation? Is value about the increased usage of a resource, or possible value generated by a minimum usage of a resource? Usage is an important dimension of determining how value is generated as part of API operations.
- Users - How is the value generated on a per user level? Are more users valuable? or possibly more targeted users? Teams, groups, and many other ways to determine how users impact positively or negatively the value generated from platform usage.
- Relationships - How can relationships between users, or companies be applied to value generated? Will access to more relationships positively or negatively impact how value is generated for the platform and consumers?
- Data Acquisition - Is the acquisition of data part of the generation of value via the public data platform, encouraging the growth of data.
- Applications - Is value generated looked at on a per application basis? Does having multiple applications impact the value generate? Coming up with interesting ways to understand how applications impact platform value for everyone.
- Integrations - What external integrations are available? How can these be leveraged to enhance the value for consumers? Are some integrations part of base operations, where others are accessible at higher levels, or on a one-off basis.
- Support - Is access to support something that impacts the value being generated? Does access to support resources introduce the optimal levels of value consumers are looking for? How is support wielded within overall platform monetization?
- Service Level Agreements - Are we measuring the number of contracts signed, and partner agreements in place? And how we are delivering against those agreements?
- Revenue - What revenue opportunities for the ecosystem around an API and its operation, sharing in the money made from platform operations. Obviously, anything beyond operating costs should be applied to expansion of efforts.
- Indirect Value - What are some of the indirect value being generated by the public data platform operations.
- Marketing Vehicle - Having an API is cool these days, and some APIs are worth just having because of the PR value, and discussion.
- Traffic Generation - The API exists solely for distributing links to the web and mobile applications, driving traffic to specific properties - is this tracked as part of other analytics?
- Brand Awareness - Applying a brand strategy, and using the API to incentivize publishers to extend the reach of the brand and ultimately the platform - can we quantify this?
- Analysis - How can analytics be leveraged as part of API value generation? Are analytics part of the base of operations, or are they an added value incentive for consumers, and platform operators.
- Competitiveness - Is the public data effort more agile, flexible, and competitive because it has an API and can deliver on new integrations, partnerships, and to new platforms easier, and more cost effectively?
- Public Service - Making data available for use on many different web, mobile, and other applications demonstrates a commitment to public service, and the good public data can do.
While there may be other hard costs associated, as well as areas of value being generated, this should provide a simple checklist that any open data provider can use as a starting blueprint. Additional costs can be included on in these existing areas or added on as new areas as deemed relevant--this is just about getting the monetization conversation going.
There are two main objectives in this exercise: 1) understanding the hard costs and value associated with operations 2) assembling into a coherent list so that we can explain to others as part of transparency efforts. When it comes to the business of public data, it is more than just generating revenue, it is about being upfront and honest about why we are doing this, and how it is done--mitigating the political risk involved with doing business with public resources out in the open.
Putting Together A Working Plan Involving Public Data
With an understanding of the hard costs of providing a public data platform and an awareness of the intended value to be generated via operations, we can now look at what details would be involved in a plan for executing this monetization strategy. API management practices are architected for metering, measuring, and limiting access to data, content, and algorithmic resources in service of a coherent, transparent public data monetization strategy.
Here is a core framework of API management that can be applied to public data that can be used to drive monetization efforts:
- Access - What are the default access levels for public data access via the API.
- Self-Service - Public access to the platform via user registration, or 3rd party authentication like Twitter, Facebook, or Github.
- Approval - Access level(s) that require the approval of user or application before they are able to access platform resources.
- Tiers - What are the multiple tiers of access to all data resources available via API.
- Public - Define the default public access for the platform, with a free, limited access tier that is obtainable via a self-service registration without approval.
- Contributor - Providing a tier of access to contribute content, validate and management data on the platform.
- Service Provider - Providing a tier of access for service providers involved with public data operations.
- Internal - Access tier for internal groups, used by all websites, mobile applications, and system integrations.
- Partner - Tier(s) of access design for data, and other partners involved in the management of public data resources.
- Commercial - Access tier(s) for commercial usage of public data resources with higher levels of access for set fees.
- Non-Commercial - Access tier(s) for non-commercial usage of public data resources with specific access waving fees.
- Government - A set of API resources is available specifically for government access.
- Auditor - Access across APIs specifically designed for external 3rd party auditors.
- Elements - What are the core elements that make up the service composition for the monetization plan(s).
- Paths - Establishing plans based upon the availability and access to different API endpoints, including platform meta API.
- Read / Write - Restricting read and write access to public data to specific tiers, limiting who writes data to only trusted applications.
- Time Frames - What are the timeframes that impact the public data / API monetization plan(s) and consumption.
- Daily - What are the details for managing, guiding, and restricting plan entries each day.
- Weekly - What are the details for managing, guiding, and restricting plan entries in weekly timeframes.
- Monthly - What are the details for managing, guiding, and restricting plan entries on a monthly basis.
- Metrics - What is being measured to quantify value generated, providing a framework to understand monetization possibilities.
- API Calls - Each call to the API is measured, providing the cornerstone of monetizing access and contribution to public data--remember not all calls will cost, some will add value with contributions.
- URL Clicks - Each click on a URL served up via the API drive data and content is measured, providing details on value delivered to internal and external websites--URL shortener required for this.
- Searches - All searches conducted via the API are measured, providing details on what users are looking for.
- Users - Measure user acquisitions and history to keep track of the value of each platform user.
- Applications - Measure the number of applications added, with details of activity to understand value generated.
- Limits - What are the limitations imposed across all tiers of access as part of the API monetization plan.
- API Calls - How many API calls any single application can make during a specific time frame.
- IP Address - Which IP addresses you can request data from, limiting the scope of serving data.
- Spend - How much any user can spend during a given time period, for each user or application.
- Pricing - What prices are set for different aspects of the monetizing the platform.
- Tiers - What are the base prices established for each tier of API access.
- Unit - What are the default unit prices of per API call access for each tier.
- Support - What charges are in place for receiving support for platform applications.
- SLA - What costs are associated with delivering specific quality or levels of service and availability?
These are the moving parts of a public data monetization strategy. It allows any public data resources to be made available on the web, enabling self-service access to data 24/7. However, it does it in a way that requires accountability by ALL consumers, whether they are internal, partner, or the public at large. This API management scaffolding allows for the frictionless access to public data resources by the users and applications that are identified as worthwhile, and imposing limits, and fees for higher volume and commercial levels of access.
Speaking To A Wide Audience With This Public Data Monetization Research
I purposely wrote this document to speak to as wide as possible audience as possible. In my experience working with public data across numerous different industries, there can be a wide variety of actors involved in the public data stewardship pipeline. My objective is to get more public data accessible via web APIs, and generating revenue to help fund this is one of my biggest concerns. I am not looking to incentivize people in making unwarranted profits on top of public data, this is already going on. My goal is open up the silos of public data out there right now, make them more accessible, while opening up the opportunity for delivering to a variety of applications, while also funding this important process.
I wanted to help anyone reading this to craft a coherent argument for generating much-needed revenue from public data, whether they are trying to convince a government agency, non-profit organization, institution, or a commercial company. Public data needs to be available in a machine-readable way for use in a variety of applications in 2016--something that takes resources and collaboration. APIs are not another vendor solution, they are the next step in the evolution of the web, where we don't just make data available for humans by publishing as HTML--we need the raw data available for use in many different applications.
I was playing around with Algorithmia for a story about their business model back in December, when I got sucked into playing with their DeepFilter service, resulting in a 4-week long distraction which ultimately became what I am calling my algorithmic rotoscope work. After weeks of playing around, I have a good grasp of what it takes to separate videos into individual images, applying the Algorithmia machine learning filters, and reassembling them as videos. I also have several of my own texture filters created now using the AWS AMI and process provided Algorithmia--you can learn more about algorithmic rotoscope, and details of what I did via the Github project updatese.
The project has been a great distraction from what I should be doing. After the election, I just did not feel like doing my regular writing, scheduling of Tweets, processing of press releases, and the other things I do on a regular basis. Algorithmic Rotoscope provided a creative, yet a still very API focused project to take my mind off things during the holidays. It was a concept I couldn't get out of my head, which is always a sign for me that I should be working on a project. The work was more involved than I anticipated, but after a couple weeks of tinkering, I have the core process for applying filters to videos working well, allowing me to easily apply the algorithmic textures.
Other than just being a distraction, this project has been a great learning experience for me, with several aspects keeping me engaged:
- Algorithmia's Image Filters - Their very cool DeepFilter service, which allows you to apply artistically and stylish filters to your images using their API or CLI, providing over 30 filters you can use right away.
- Training Style Transfer Models - Firing up an Amazon GPU instance, look through art books and find interesting pieces that can be used to train the machine learning models, so you can define your own filters.
- Applying Filters To Images - I spent hours playing with Algorithmia's filters, applying to my photo library, experimenting, and playing around with what looks good, and what is possible.
- Applying Filters To Videos - Applying Algorithmia's, and my own filters video I have laying around, especially what is possible when applied to the GB's of drone video I have laying around, something that is only going to grow.
Why is this an API story? Well, first of all, it uses the Algorithmia API, but I also developed the separation of the videos, applying filters to images, and reassembling the videos as an API. It isn't anything that is production stable, but I've processed thousands of images, many minutes of video, and made over 100K API calls to Algorithmia. Next, I am going to write-up Algorithmia's business model, using my algorithmic rotoscope work as a hypothetical API-driven business--helping me think through the economics of building a SaaS or retail API solution on top of Algorithmia.
Beyond being an API story, it has been a lot of fun to engineer, and play with. I still have a GPU instance fired up, training filters, as well as recording more drone and other video footage specifically so I can apply some of the new filters I've produced. I have no intention of doing it as a business. algorithmic rotoscope is just a side project, that I hope will continue to be a creative distraction for me, and give me another reason to keep flying drones, and getting away from the computer when I can. In the end I am learning a lot about drones, videography, and machine learning, but the best of all it has helped me regain my writing mojo--with this being the first post I've written on API Evangelist since LAST YEAR! ;-)
I got sucked into a month long project applying machine learning filters to video over the holidays. The project began with me doing the research on the economics behind Algorithmia's machine learning services, specifically the DeepFilter algorithm in their catalog. My algorithmic rotoscope work applying Algorithmia's Deep Filters to images and drone videos has given me a hands-on view of Algorithmia's approach to algorithms, and APIs, and the opportunity to think pretty deeply about the economics of all of this. I think Algorithmia's vision of all of this has a lot of potential for not just image filters, but any sort of algorithmic and machine learning API.
Retail Algorithmic and Machine Learning APIs
Using Algorithmia is pretty straightforward. With their API or CLI you can make calls to a variety of algorithms in their catalog, in this case their DeepFilter solution. All I do is pass them the URL of an image, what I want the new filtered image to be called, and the name of the filter that I want to be applied. Algorithmia provides an API explorer you can copy & paste the required JSON into, or they also provide a demo application for you to use--no JSON required.
Training Your Own Style Transfer Models Using Their AWS AMI
The first "rabbit hole" concept I fell into when doing the research on Algorithmia's model was their story on creating your own style transfer models, providing you step by step details on how to train them, including a ready to go AWS AMI that you can run as a GPU instance. At first, I thought they were just cannibalizing their own service, but then I realized it was much more savvier than that. They were offloading much of the costly compute resources needed to create the models, but the end product still resulted in using their Deep Filter APIs.
Developing My Own API Layer For Working With Images and Videos
Once I had experience using Algorithmia's deep filter via their API, and had produced a handful of my own style transfer models, I got to work designing my own process for uploading and applying the filters to images, then eventually separating out videos into individual images, applying the filters, then reassembling them into videos. The entire process, start to finish is a set of APIs, with a couple of them simply acting as a facade for Algorithmia's file upload, download, and DeepFilter APIs. It provided me with a perfect hypothetical business for thinking through the economics of building on top of Algorithmia's platform.
Defining My Hard Costs of Algorithmia's Service and the AWS Compute Needed
Algorithmia provides a pricing calculator along with each of their algorithms, allowing you to easily predict your costs. They charge you per API call, and the compute usage by the second. Each API has its own calculator, and average runtime duration costs, so I'm easily able to calculate a per image cost to apply filters--something that exponentially grows when you are applying to 60 frames (images) per second of video. Similarly, when it comes to training filter models using AWS EC2 GUP instance, I have a per hour charge for compute, storage costs, and (now) a pretty good idea of how many hours it takes to make a single filter.
All of this gives me some pretty solid numbers to work with when trying to build a viable business built on top of Algorithmia. In theory, when my customers use my algorithmic rotoscope image or video interface, as well as the API, I can cover my operating costs, and generate a healthy profit by charging a per image cost for applying a machine learning texture filter. What I really think is innovative about Algorithmia's approach is that they are providing an AWS AMI to offload much of the "heavy compute lifting", with all roads still leading back to using their service. It is a model that could quickly shift algorithmic API consumers to be more wholesale / volume consumers, from being just a retail level API consumer.
My example of this focuses on images and video, but this model can be applied to any type of algorithmically fueled APIs. It provides me with a model of how you can safely open source the process behind your algorithms as AWS AMI and actually drive more business to your APIs by evolving your API consumers into wholesale API consumers. In my experience, many API providers are very concerned with malicious users reverse engineering their algorithms via their APIs, when in reality, in true API fashion, there are ways you can actually open up your algorithms, make them more accessible, and deployable, while still helping contribute significantly to your bottom line.
I am reworking the API Evangelist developer area, and shifting most of my content to be available as YAML and JSON data on the Github repositories that drive my network of sites. I'm doing this partly because I am not in the business of managing and growing an API community, and because there are some really badly behaved people out there that I'm just not really interested in having keys to my internal network. I am happy to open up read only access to my work publicly and write capabilities to my trusted partners, but having self-service access to my server(s) just isn't fun in the current online climate.
I get it when folks want to keep their valuable data, content, and algorithm under lock and key, and require developers to build a relationship before they get access or increased levels of consumption. However, this hasn't always been the tune I'm whistling. There are plenty of examples of me telling API providers to provide self-service access to their resources in the past--well, we've fucked that off, with our bad behavior as API consumers. It's not that the concept won't work, it is just that it won't work with the number of assholes on the web these days it just isn't a good idea.
Even with keeping the lock and key on our API resources, we can still be public with our API operations--there are many positive reasons for doing so. SEO is probably the first, I mean how are people going to find your APIs if you can't Google for them. Providing information to the press, and making the resources that support your APIs self-service can reduce the workload when you do give people access. Some examples of this can be found when looking at the Target vs. Pandora developer programs, both required approval to get access, but Target is much more open than Pandora with their overall story.
Target really doesn't have that much more than Pandora does, but they have a blog and link to their Github, and of course their jobs page. I'm still going to publish the documentation for my own APIs, and go further than Target has, but I have to give them credit for being at least a little more creative, and public than Pandora. Like I said I won't be pushing back on providers when they do not have self-service public access to their API resources anymore, but I will be encouraging y'all to be creative when thinking about the public presence of your operations and engaging in some meaningful storytelling via a publicly available portal.
Talking to people, and telling stories on a regular basis always pushes me to evolve my understanding of how people see (or don't see) APIs, and pushes me to keep shifting the way I tell stories. I've always felt that that education about APIs should be relevant to the users, but usually this centers around making it familiar, and speak to whoever I am helping onboard. After talking to some folks recently at @APIStrat, I'm adding to my thinking on this, and focusing on making my stories more precise for folks I talk with.
One of the reasons I really like APIs is that they are so versatile. You can take and a piece of data, content, or an algorithm (code), and wrap with an API, and provide read and write access via the Internet. However, I think the average person does not thrive in this environment and need an explanation that is more precise. It doesn't help them to know that APIs can do anything, they need it to be relevant, but also providing a single solution that they can wrap their heads around, and apply in their world.
I am just sharing this thinking as I'm working to add it to my storytelling toolbox. I'm really committed to helping people understand what APIs are, so they can help push back on platforms to have APIs, as well as be more open with existing APIs. It doesn't do any good to confuse people with an unlimited number of API scenarios. We should be dialing in our storytelling so that we can help onboard them with the concept, and increase the number of folks who understand that they can take control over their own information on the platforms they depend on each day.
My friend and partner in crime Nicolas Grenié (@picsoung), and operator of our open source API search engine APIs.io, just let me know that the Taiwanese government just added an APIs.json file for their government open data site. Adding to the other authoritative (added by owner) government API indexes like from Trade.gov in the United States.
We haven't' had a lot of time to move forward the APIs.json index lately, but honestly it doesn't need much pushing forward at this point in time. Our primary objective is to continue getting adoption like this, and not radically shift the specification until we have more feedback from the community, across a large number of API operators.
You can learn more about the specification we've been working on over at apisjson.org, and see other existing implements from Fibit, Plivo, and Actuity Scheduling. Once you've created an APIs.json definition for your API operations you can add to the APIs.io search engine using the submission form, or using the APIs.io API. We are gearing up for another push forward regarding tooling being developed on the specification over the holidays and will have more information in 2017.
I tried to get back to normal last week on API Evangelist -- I failed. The previous week was @APIStrat in Boston, which was a success. It was the Presidential election that caused me to swerve and put things into the ditch. I was devastated and saddened by the results. Not because my party lost, but because we chose someone who ran on such a racially, and religiously charged platform, that was so threatening to women.
It is easy to mistake what I do as the API Evangelist, as being a voice for the startup community -- cheerleading APIs in the service of the seemingly endless wave of tech companies coming out of Silicon Valley. While I do pay attention to the technology, and business of how these companies are using APIs, making sure the content, data, and algorithms they employ are transparent and observable by partners, 3rd party developers, end-users, and industry regulators is my primary objective.
My mission is not about open APIs because open always makes things better, or simply open for business. I believe corporate, government and institutional data, content, and algorithmic resources should be as transparent and observable as possible, by those who are impacted by their existence. Meaning if you are collecting data about me, I should know what you are collecting, and have access to it. I should be able to move it around or delete it. Trusted regulators and auditors should also be able to peek behind the curtains of the algorithms that are increasingly impacting our world, and make sure they size up with the claims being made.
I have spent six years pushing on startups and the enterprise to be more transparent and inclusive with their resources by employing APIs. During this time I did the same for the city, state, and the federal government. I've also extended this to higher educational institutions. With Donald Trump in office, this does not change, it just ups the stakes for me. In this climate, being transparent about the data we collect and share, and how the algorithms operate will only be more critical.
I am nervous about the future. Much of the existing rhetoric around algorithms and the surveillance economy has worried me, but all of this in the hands of a sexist, racist, and Islamophobically charged administration and climate scare the hell out of me. In 2017 I will continue to cover the technology and business of APIs, but as I have already been doing, I will be spending a larger portion of my time talking about the politics of APIs, including terms of service, privacy, and the observability of the systems and algorithms that are fueling our increasingly Internet-connected world. As always, I will need your support -- thank you!
There aren't too many startups doing interesting things in the API space right now. One of the exceptions is Stoplight.io. I am working really hard to find some of the good things in the API space to focus on, in hopes of not being too dark with my writing, after the election. Stoplight's new Scenarios is something new, something progressive, and I think could have a larger impact down the road--making it worth covering here on API Evangelist.
Stoplight Scenarios is billed as "Test, automate, and debug web APIs + AWS Lambda". You can make API and AWS Lambda calls, and test, automate, and debug the responses -- pretty standard stuff we are seeing across several API service providers. Where it gets interesting for me is that you can daisy chain these requests together into a variety of scenarios. You can do this with a single API, or you can do it across a variety of disparate APIs--making for some pretty valuable "juju" in my opinion.
As I am learning about a new API, I am often frustrated by having to connect the dots between many different paths, adding, then pulling, then updating, and other common "scenarios" I will want to accomplish. I would love for API providers to do this heavy lifting for me, and provide me with a variety of the common scenarios I need as an API consumer to get up and going. This is just the local possibilities around using scenarios, I'll explore the possibilities between many distributed APIs in future posts.
If you haven't played with Stoplight before, I recommend heading over there. Their API design, definition, mocking, and documentation tooling is leading edge stuff in the world of APIs right now. The Stoplight Scenarios just ups the value their platform brings to the table, continuing to make them on of the few bright spots in the API world for me these days. Let me know what you think after you play with more.
The 7th edition of API Strategy & Practice wrapped up last week. It has been difficult to gather my thoughts with the election going on, but I wanted to shift my attention back to the API community for a bit. Steve Willmott and I started APIStrat back in 2012 to help establish an open, vendor-neutral community to discuss the technology, business, and politics of APIs -- we more than succeeded!
It always warms my heart when people come up to me and share that they didn't see anything different about APIStrat, from other events before they attended and that they do now. Steve and I worked our asses off to make it a rich, inclusive, and meaningful conversation around APIs. It makes me very happy to hear people seeing the event as we intended, and going home with a headful of API knowledge.
One thing I heard from several of the core members who have been there since the first one, and is something that has lingered in my mind, not just because of APIStrat, but also because of where are at in the wider API space -- is that the API world is expanding, and there are concerns regarding how we can maintain community. I consider our hard work in evangelizing APIs to be a success, as space is expanding like never before. Everybody is doing APIs, it isn't just a niche anymore. We are at the point we were with the web in 2001, and folks aren't asking if they should be doing APIs anymore--they are just doing them.
Many of us in the original group are tired. We've been at this for a while. Many of the younger ones are still easily excited and distracted by new technology. There are many new entrants who just need to hear the same old stories we've been telling for years so that they can just get up to speed. Many people expressed their concerns around maintaining community in the face of this evolution, and growth in the space. It leaves a pretty big smile on my face that we are having trouble scaling the API community -- maybe we should try micro-communities or a dev ops approach...no, no a containerized API community! ;-)
I don't know the answer. I'm confident we will find a way forward. I'm super thankful for the community we've built. It literally saved my ass this summer. I just wanted to put these thoughts out there, and let folks know I heard, and Steve and I care.
The 7th edition of API Strategy & Practice Conference happened last week. While I wasn't fully engaged throughout the planning process for this edition, due to my summer being disrupted, I wanted to take the time to share some of what happened to make it more of an inclusive technology event. There is a lot of people who are "interested" in making their events more diverse and inclusive, but APIStrat is "committed" to this (thanks, Charles Ashley III @CAsh_The3rd), and here are some of what we did.
- Strong Female Lead - Put a woman in charge. Period. She will set a good tone.
- Invite Women To Speak - Work to ONLY invite women when getting started.
- Invite People of Color To Speak - Work to ONLY invite people of color when getting started.
- Have Code Of Conduct - Make there is a code of conduct present, and communicated well.
- Enforce Code of Conduct - Sadly, we had to do this round, but it sets the right tone.
- No Manels - Do not have any panels where you only have men.
- Numbers - Know your numbers, and work every moment to increase them wherever you can.
- Repeat, Rinse - Repeat all of this at all levels, session chairs, keynotes, reg counter, etc.
We are nowhere near where we want to be with making #APIStrat a truly inclusive event, but we are making improvements with each edition. We have our checklist and have been building on it with each event. It takes a leadership team that is committed to this. Steve, Lorinda, Amelia, and the team delivered this round -- I wish I could take credit. I saw more women, diverse faces, and topics at #APIStrat this round -- leaving me very, please.
If you are running a tech conference, please put in the extra work. You can't just give this 5 or 10% effort. You literally have to invite NO WHITE MEN to your event for the first couple of waves of outreach. You need to do some serious homework on the women and people of color who are making an impact across the space. If you do the work, set the right tone, it will go far beyond just being able to pat yourself on the back--the ideas and conversations will be richer for it.
Difficult To Keep My Attention
When I was young I was always curious when it came to technology. I set up the entire computer lab for my 7th-grade math teacher back in 1983. I programmed computers all through high school, even having a job programming the software used by schools in the State of Oregon in COBOL. I was really good at school until I wasn't interested. If I got bored, which I did growing up in rural Oregon, I tended not to pay attention in school. Eventually, it got kicked out of high school in my senior year, and I ended up getting into drugs, and a lot of trouble. From 1990 through 1995 I spent my time traveling the country partying and dealing drugs until I finally hit a wall and needed a way out. To get my life out of the ditch I turned to what I already knew, the outdoors (I grew up out in the woods), and computers. I was good at paying attention to the bits and bytes in the emerging world of personal computing, and I would leverage technology to get me out of this mess I found myself in.
Attention To Family & Career
After spending a summer in the Oregon wilderness getting clean and healthy, I moved to the nearest city and got to work building a career. By the first dot com bubble, I had found success, married a young lady, and had a beautiful baby girl. I had left my troubled past behind. It was a period in my life where there the harder I worked, the better I felt. I had a good job and bought a house (two), but my attention always seemed to be on finding further business success, a sort of chronic entrepreneurial condition, resulting in having at least one, and often times multiple startup projects going on at any point in time. In total, I had almost 14 separate startups with only two I can even remotely consider a success. Maybe I was too early? Maybe I was just paying attention to the wrong details, and ignoring what mattered to people around me--most critically, to my now ex-wife.
Attention To My New Partner In Crime
In 2009, after putting my startup ambitions behind me, and moving on with my life after a divorce, I met an amazing lady. She had a teenage son, and I had a tween daughter. She worked in education technology, and I worked in tech. What we were paying attention to seemed to be compatible. After we connected she increased her focus on the world of educational technology and encouraged me to increase my focus to something that mattered to me. I got to work exploring what that would be--at the time I was finding success using this new thing called the "cloud". I knew it was going to be significant in the future, but I also knew there was more to it, beyond just things being in the "cloud".
Attention To The Business of APIs
I saw the commercial potential of making digital resources like compute and storage available in the cloud using APIs and increasingly saw entrepreneurs deliver the resources needed for mobile application development using APIs. What did both cloud and mobile have in common? APIs! This is where I would pay attention. As I dug deeper, I noticed there were a number folks paying attention to the technical merits of web APIs, aka REST, but that very few people were paying attention to the business side of API operations. I was watching companies like Twilio, SendGrid, Stripe, and others demonstrate that there was more to this API thing than just having a RESTful API on the open web, and that through sensible identity and access management (API keys, OAuth), rate limiting, metrics and reporting, aka API service composition, that you could develop some pretty interesting business models. I felt I had hit on something significant, and would focus all my attention on the business of APIs.
Attention To Bankruptcy & Addiction
Things were going reasonably well. I was paying attention to the fast growing world of APIs. The only problem was that I hadn't been paying attention to the finances as much as I probably should have, and my previous startups and my divorce were finally catching up with me. I had nowhere else to go, I had to declare bankruptcy. Around the same time, my partner's son began to have problems in his world, involving depression, and pharmaceutical addiction. He needed some assistance. We tried a change of scenery, moving him to another city in another state, hoping to shift his attention to a new environment, and job. At this point in our career, we were living 100% on the road, traveling to events, evangelizing APIs and education technology, but we did what we could to help him find some balance.
Attention To The Politics of APIs
My hard work paying attention to the world of APIs was paying off. I was acquiring new partners, sponsors, and tackling more projects that were focused on the business of APIs, telling the stories of how startups were making an impact across a variety business sector. As I paid more attention to the business of APIs I began to see there was more to APIs, and that things like terms of service, copyright, service level agreements, security, and even regulation were beginning to define the space. My work ended up getting me invited to work at the White House, helping out the Obama team with open data and API strategy across the federal government, and helping the Electronic Frontier Foundation with the Oracle v Google API copyright case. I was still paying very much paying attention to the business of APIs, but increasingly I was also paying more attention to what I had come to call the politics of APIs.
Attending To My Health & Well Being
It can easy to get caught up in work, especially when you are an API Evangelist. It can also become a distraction from your actual life, and the material world around you. API Evangelist was a character for me, a persona, but I was living on the road, spending every waking moment as this character--resulting in me paying attention to very little else. During this time I was present, front and center in the API space, but I was neglecting to pay attention to my own health, something that finally caught up with me. Because of the Affordable Care Act (thanks, Obama!) I went to the doctor early this year, and he said that I would not live beyond 50 if I didn't stop drinking alcohol. I quit drinking immediately. Then shortly after this happened, my partner's son showed up on our doorstep. He had been kicked out of the treatment center we had placed him in 8 months earlier, after a continued struggle with his depression and addiction. I was sober. He was using opiates again. We had no money. What could we do?
Attention In A Material World
I did the only thing I knew how to do. I did what I had done before and headed up to the woods to get clean. We rented a truck and drove to Yosemite, and deep into the Sierra Nevada mountains of California. We found ourselves on Peterson Mountain on the border of California and Nevada digging for quartz crystals (another place from my past), after which we would make our way north through North California, and on into Oregon. We avoided cities, and spend our days hiking and clearing trails. We put signs back up. We often were paying attention to trails that seemed forgotten by humans and were just melting back into nature. We rock hopped in rivers and creeks. Swam in the snow melt. We walked 10-15 miles most days, paying attention to the kid's health, and as I would learn--my own health and sanity along the way. We talked about getting high, the online world, having oxycontin delivered to your door enabled by the Fedex API, being paid for with help from the Bitcoin API. Convenience and online culture. We walked. We walked. We walked. We walked until we slept each day.
Attention To The Digital World
While I was marching the kid around the mountains, it was an emotionally and physically exhausting time for both us, and through all of this I didn't want to neglect (our) more digital selves. The kid is your average 20 something who spends much of his time online, and gaming. While out in the wilderness I wanted to still attend to the digital parts of his (our) personality and I settled in on drones as a viable distraction. Drones worked. We could fly them on the trail, capture video, and as I would come to learn, we could also pay attention to much more around us using this new and wonderful Internet and radio connected device. We chose the DJI Phantom 3 Pro drone which has an iPad as the visual experience for the radio controller. Something that when you put in the kid's hands he'd head into a bush, under a rock overhang, and focus 100% of his attention to flying and experiencing the world through the drone's camera. Contrasting his approach, when you put the controller in my hand, I would always keep one eye towards the sky, paying attention to this very expensive, physical object flying through the canyons, over the lakes, and down the river.
Attention At The Intersection Of Physical & Virtual
Drones are an amazing piece of tech. They give you an entirely new view of the world. It is a 4K video of the world wrapped in longitude, latitude, altitude, airspeed, and a wealth of other data. The physical drone has its own APIs. The guidance systems have APIs. The camera has its own APIs. The battery has its own APIs. The radio controller has its own APIs. The mobile interface for the controller has its own APIs. It uses APIs to acquire GPS and mapping information. When a wireless network is present it can stream live to Youtube and Facebook using APIs. It receives updates in the controller app regarding nearby airports, military bases, weather, and forest fires using APIs. Data is captured every second when operating and can be uploaded to the cloud synchronously or asynchronously using APIs. It is an amazing Internet of Things (IoT) device to consider when it comes to API usage, as well as just for flying through mountain meadows, and through river canyons.
The Digitial Demanding Our Attention
This summer we were able to successfully escape out into the woods, at the tops of mountains, and down into the deep river valleys. However, even though we were often hundreds of miles into the wilderness, the digital world seemed to seek us out. The battery would need a firmware upgrade. The controller was out of sync guidance firmware. Where is the damn cell phone charging cable? The batteries (with firmware)--were they warm enough to operate? When we'd get to the end of trails, and I would grab a cell phone to call for a ride, or communicate that we were safe, the cell service was so bad, we'd wait for hours sometimes for push notifications, alerts, syncs, and other digital noise to complete before we could gather enough bandwidth to send a simple SMS. We eventually learned to remove all but the essential applications, so the demands for our attention were reduced to only what we needed to survive and get from point A to B, and to acquire supplies.
Attending To My Online World
As we approached 3 months in the woods, it was time to come back. The kid settled into an apartment in Portland OR, and I went back to Los Angeles, CA. I turned on all the regular channels, my feeds, email, Github, Twitter, Facebook, LinkedIn, Medium, Slack, Disqus, Reddit, DZone, Lanyrd, Meetup, Stack Overflow, SoundCloud, Tumblr, and others. Everything was demanding my attention. Read Me! We want to work with you! Our tech is the next thing! Come speak at our event! This is the next big thing! By 2020 everyone will be doing this! I realized how much I had been paying attention to, how much momentum I had built up. I realized how much of an API driven assault I was under each day. Now I needed to understand what was actually worthy of my attention, bend the concept of real time to my personal definition, and establish a sort of "air gap" between me and the digital world that had an unending appetite for my attention.
Attention On The Drone
Even back in the real world, the drone continued to linger in my mind, working to get my attention. I was learning more about the device and it's APIs, and how it consumed APIs in the cloud, and how it used APIs to publish video, images, and other data to the clouds. I learned about how companies like Airmap were providing updates to my DJI drones from federal government agencies like the Department of Interior. I also learned how these agencies had themselves discovered that their fleets of DJI drones were updating video, images, and all this data to Chinese servers. DOH! Drones seemed to be capturing everybody's attention, as they interfered with forest fires, flew in front of commercial airlines, and gave us a new perspective of our both the material and digital worlds. Providing us with an entirely new way to gather data, develop awareness, and pay attention to infrastructure, nature, or even a police protest. Drones are capturing our attention, but it also feels like they were now also paying attention to us, whether we like it or not.
Attending To What Matters
I consider myself a success. I do what I enjoy for a living, and make more than enough money to survive. I have an amazing partner in crime. I have a strong and smart teenage daughter. None of this success is due to any single event along the way. My success is because I always realized that all if this is a journey and never a single destination. It is not the money. It is not about a lack of money. Not any single piece technology. Not about any of the startups I have helped build or worked with. Not even API Evangelist. APIs simple have given me lens to look through as I pay attention to the "seemingly fast-paced" world of web, mobile, and now these amazing devices we are connecting to the web. No single piece of technology is what truly matters, not even the API, it all depends on what I do with them, and how we empower (or hurt) humans with them. This is a human story. This is my story. It is not a technology story. It is my personal journey. The technology is just one of the tools in my toolbox. The usefulness of my tools depends on which ones I choose to use (or not), how I care for them, and the knowledge behind how I put these tools to work. Taking a moment with each application, to always consider what really matters.
Definition Of Attention
To close I wanted to borrow from my partner in crime Audrey Watters (@audreywatters) on talk a couple of months ago on attention, where she said: According to the OED, “attention” – derived from the Latin “attendere,” to attend – means “The action, fact, or state of attending or giving heed; earnest direction of the mind, consideration, or regard; especially in the phrase to pay or give attention. The mental power or faculty of attending; especially with attract, call, draw, arrest, fix, etc.” “Attention” is a noun, but it refers to an action and/or a state of being. Attention is a mental activity. An earnest activity – which I particularly like. Attention is a military activity. It refers to how we stand and how we see and how we think.
I was reading the latest Yahoo transparency report, as well as the Tumblr. When a company releases their latest version of this data, it tends to prompt me to take a look at some of the other providers who have them, like Google and Twitter. I am interested in what I can learn from these reports, about what the government is up to, as well as the incentives behind each platform publishing their reports. I fascinated by studying the process and approach of each company, tracking on what some of the common building blocks so that I can include in my API transparency research.
The addition of CSV downloads of information requests by Twitter is worth noting, and adding to my list of building blocks. Platform providers are going to have to consider adding machine-readable versions of their transparency reports to help address the growing number of requests they will bet getting in this area. Providing CSV, JSON, or even YAML access to transparency report information is going to become important for us to understand how this legal and very political layer of our worlds is evolving.
The number of government and court requests for information are up. The number of companies publishing transparency reports in response to public demand for transparency is increasing. Providing machine readable representations of this data, as well as evolving a common API and data schema definition is the only way we are going to be able to manage this growth, and make sense of the data across platform providers. I will keep adding company transparency reports to my research, and take the time to aggregate the common elements in play across them, and see if I can't contribute to this common API and schema definition for use by providers
|Photo: Drones and Society|
I went down to the police department in Hermosa Beach and filed my application for a drone permit. It's been two weeks and I haven't heard back. When I get done with @APIStrat I will go down there and talk to them again, and probably have to resubmit my application. Hermosa Beach is purported to have some of the strictest laws. I'm submitting mine so I can just play with my drone, and program it on the beach without getting into trouble.
Before I fly my drone I use the B4UFLY mobile app from the Federal Aviation Administration to know whether I should fly my drone or not. It tells me whether I'm near an airport, military base, other location or event. This helps, but it doesn't help me with the legal side of things, what the laws are in my area, and any assistance in acquiring permits and approval. The industry is young, I'm sure eager startups out there are already working their asses off to aggregate the data, and serving it up via APIs for use in mobile and drone applications.
As an operator, I am going to need the laws applying to the drone, bu also laws applying to video, and other data gathered. Can I be filming? What are private / public property data and privacy laws? I am gathering video, weather, temperature, and other crop data over commercial farms, am I allowed to keep it? What are the rules with other types of imaging like infrared? Are there any noise ordinances? There are numerous considerations when flying a drone for both personal and commercial scenarios.
To be an informed pilot I need all this at run(flight)time. I guess I might also need some of it before I head out to my flight location, but also whatever I can also get at flight time. The drone sector seems to begin the process of tackling many of the urgent questions facing the industry. The FAA has stepped up with their guidance, and companies like Airmap are also getting involved with the flight restriction level, as well as weather. We will need the same layer for the legal side of things, helping us receive up to date details on the laws impacting me flying drones, filming, and gathering data--maybe systems to help us manage.
Sounds like another drone API opportunity. I'd say the three top things impacting me when operating my drone(s) right now are 1) flight restrictions 2) weather, and 3) legal. I'm sure if you pulled together the data, provided a simple commercial API, you could do well delivering it to drone application developers and other platforms.
I wrote about transparent data transfer control APIs at the IoT device level the other day, so it made me happy to come across Dowse, an on/off switch for home devices while monitoring things today. This Internet of Things privacy and control device sounds like something that should be present in every home and business network.
I am still learning about it, but here is a short description from the post by WaaG, that I think describes it well:
The Dowse box gives you back control. Dowse is free and open source software. There is no big company behind it. By installing the Dowse box you will get insight down to the network layer of what is actually going on in your home. Who is talking to who, what, where and when? You can see which device connects to which company and you can turn that communication off. Or allow it.
This reflects the transparency I would like to see at the network level for every home and business. With the recent wave of IoT launched DDOS attacks we are going to need more discoverability, awareness, transparency, and control at the home and business and home network and Universal Plug and Play level.
I'm not sure saying the Dowse box is the solution, but it provides us with one possible blueprint, similar to what I was talking about in my post on transparent data transfer control. In this IoT age, we are going to need device awareness to be a default, and transparency about which devices are talking to the Internet, when and what data they are transferring, and possibly even more security auditing and observability solutions.
It seems like the Dowse box concept should be baked into all home and business networking. I am well aware that a version of this does exist at the network administration level for most modern networking solutions, but I'm talking about the next generation, end-user friendly, mobile edition that makes devices monitoring, awareness, and control dead simple for anyone.
Of course, this layer would have an API to allow for the extending, and customizing of all of this. Now that I am coming across examples of this type of control at the IoT network level, I will keep scratching, and I'm confident I'll find other examples. Maybe enough thtat Ithatadd a section to my Internet of Things and APIs research.
I thought the microservices platform Nanoscale.io have an interesting argument for why you would upgrade to a paid plan. On their pricing page, after they break down each of the pricing plans they provide you with four reasons of why you would want to upgrade from their free tier.
- More powerful APIs - Your nanoscale.io hosted microservices can run longer and perform more complex operations, or access slower source systems, without timing out.
- Get premium support - Having a problem with nanoscale.io? Think you may have found a bug? Submit your inquiry and get a guaranteed response from our technical team.
- Influence product roadmap - We are open to new feature consideration, and give preference to paid accounts to influence which ones get built sooner.
- Deploy anywhere - Upgrade your downloadable nanoscale.io server with a production license to deploy your microservices on any infrastructure you choose.
More power and support seems like no-brainers. Being able to influence the roadmap is a compelling reason and something I would pay money for! ;-) The deploy anywhere I think is a sign of the future, not just for how you will buy services for your API, this is how you will deploy your APIs for your consumers. In cloud. On-premise. On-device. You can consume our API anywhere you want--if you pay for it!
I have been aggregating the plans and pricing of API providers, and service providers for a while now. People are getting better at providing a decent breakdown of their API plans, but they aren't always that good at articulating the reasons why you would go from free to a paid plan. I think this is an area of stress, and concern for many API providers, and an area they could use more examples to follow from out in the wild.
Adrian Cockroft (@adrianco) turned me on to a DNS aggregation solution the other day while I was working on updating the API definitions for the API providers that are included in my API DS research. It was a very appropriate day for thinking deeply about aggregate DNS, with the DDOS attack against Dyn going on.
Denominator is a portable Java library for manipulating DNS clouds. It has pluggable backends, including AWS Route53, Neustar Ultra, DynECT, Rackspace Cloud DNS, and OpenStack Designate. Here is a good post on it from back in 2013, describing it as a multi-vendor interface for DNS.
There doesn't look to be a lot of activity around the project in the last year, but it provides a good model for what I'd like to eventually see across all the major stops along the API lifecycle. I picture a wealth of aggregate tooling like Denominator that can act as a broker between API service providers and help switch, migrate, and sync between providers whether you are deploying, managing, testing, monitoring, or dialing in your DNS.
As I read the multiple investigations into what happened with the DDOS attack on Dyn last week, it seems relevant to learn more about aggregate DNS API solutions like Denominator. I will spend some time looking for other similar open tooling that is vendor-neutral, as well as vendor-switchable. We are going to need open source circuit breakers like this to help route, switch, migrate, and sync DNS across many service providers in this volatile landscape.
Managing developers access to an API is API management 101. Managing the relationships between developers, and allowing for multiple users associated with an API application isn't something I have seen before. Slack just added the ability to add what they call app collaborators--"adding any existing member of your Slack team as a Collaborator, including members and Guest Accounts."
The functionality felt a lot like the social aspects that made Github more attractive than just Git. When it comes to developing messaging apps and bots I can see a social layer doing pretty well. "Once a Collaborator is added, they’ll receive a notification from Slackbot letting them know they now have access to your app." Smells like an opportunity for API management providers to bake into user management solution, and if you wanted to take it to the next level, add a Slack messaging layer as option.
Its a small feature, but I think it is one of the things that made Github work, and could have a similar impact at the API management level, allowing for more engagement between users who are working together on API integration. I am going to add it as a building block for my API management research, even if I haven't seen it elsewhere. It is something I'd like to see more of, and maybe if I plant the bug, more providers will implement.
I am diving deep into the DJi drone developer platform, and one of the elements of the DJi Drone Guidance API that caught my attention was the data transfer control methods. In this situation, the transfer control methods are designed for just the data being sent as part of the drone guidance systems, but I think it provides a blueprint that can be used across almost any IoT device connectivity.
DJI provides four methods for managing the drone data transfer control:
- Start Transfer - Inform guidance to start data transfer.
- Stop Transfer - Inform guidance to stop data transfer.
- Release Transfer - Release the data transfer thread.
- Wait For Board Ready - Set callback function handler for hen data from guidance comes, it will be called by data transfer thread.
The "wait for board ready" method acts as a sort of web hook, that can notify any application build on the API that data is now being transferred, opening up the possibilities for notifying a device owner, and operator that data is being transferred. To me, this can be a critical aspect of building trust that our devices have our best interests in mind, providing some essential transparency in the data layer of the IoT space.
Data transfer control APIs for IoT devices like this will not ensure healthy data practices. This implementation is designed to provide transfer control capabilities to the developer, it is now up to the developer to include the end users in this process. Many current mobile application business models do not incentivize this type of transparency, as you do not want end users, and often times 3rd party developers involved in data gathering, and revenue generation this valuable "exhaust" of Internet-connected devices.
I am hoping this evolves and changes as the Internet matures, and the number of connected devices increases. We need transparency at the device data transfer level, and we need all humans involved / impacted to be a literate, active, and will participate in the IoT data flow. I know many entrepreneurs think the end user isn't sophisticated enough to be involved, but this is more about your desire to keep any value generated for yourself than it is about the end-users capacity to grasp this stuff.
Many folks see me simply as a cheerleader for APIs when in reality I am more of an evangelist for the bad that can happen with APIs. I believe that sharing of data, content, and algorithms using web APIs has the potential for good, but in reality, they are often be used for doing some pretty shady shit.
An example of this is found in my inbox this morning, and I'm sure is something everyone will encounter at some point in their daily lives. It is an email for an undelivered Fedex package, which I know better than to click on, but sadly I think it is one that many folks will fall for.
Why do they fall for this? Because the email potentially has relevance, as I just ordered a handful of packages from Amazon, which were being shipped via Fedex (I do not order much online). Using the FedEx API, anyone can query the status of a package. I'm assuming that there are folks out there who are scanning for the presence of delivering notifications--I'm not up to speed on the details of how you can do this. I'm unsure if they can get my email alongside this information, but I don't think this matters. I think they can correlate data about where I live, and the fact I'm receiving packages--whether the email came from API, or through other forms intelligence, it doesn't matter.
My point is more around the fact that APIs are increasingly opening up signals about our daily lives, providing a wealth of context for phishing campaigns, increasing the chance that people will fall for these attacks. My solution to this problem does not involve a knee-jerk response to providing APIs, I am just looking to just warn API providers that they should be monitoring for this type of behavior on top of an API, and we should help the average email users and Amazon package receiver that these dangers exist. Everyone should pause and think deeply about each link or file attachment we click on--no matter how relevant it might seem in our daily flow.
I am going through the DJi DJI drone developer area which has three distinct SDKs, which allow us to leverage a variety of APIs that make the drone magic happen. I'm still wrapping my head around the intersection of drones and APIs, and this is my attempt to distil down what I'm finding in their developer area, and absorb some of what is going across the industry. This is not meant to be a complete list. It is meant for my learning, and hopefully yours along the way.
There are a variety of devices being connected to the Internet, but other than the automobile I don't think there is another object that is as complex as the drone. I'm fascinated by what is possible with this device, and the variety of APIs it has, the interaction with the RC controller, mobile device, and with other resources the clouds. I personally fly a DJI drone, so I am going through the DJI developer area, learning about their three SDKs, as they seem to be the ecosystem furthest along in their understanding the API potential -- think Twitter for IoT.
The DJI Onboard SDK
This SDK allows for communication with the DJI flight controller over a direct serial connection, to monitor and control aircraft flight behavior with the Onboard API while utilizing the built-in Intelligent Navigation Modes to create autonomous flight paths and maneuvers. Some of the actions for the onboard SDK are:
- Activation - Before you start exploring DJI Onboard SDK functionality via our ROS examples, you will need to go through the "Activation" process.
- Obtain/Release Flight Control - Managing the process to get flight control.
- Take Off - Initiate a take-off for the drone.
- Landing - Tell the device to land.
- Go Home - Tell the device to go home.
- Gimbal Control - Manage gimbal for camera.
- Altitude Control - Manage the altitude for the drone.
- Photo Taking - Allow for taking photos
- Start/Stop Video Recording - Start and stop the video for camera.
- Virtual RC Control - Control the drone through serial port by simulated channel values.
- Broadcast Frequency Control - Manage which frequency drone is broadcasting on.
- Arm/Disarm Control - Arm or disram the controls.
- Timestamp Synchronization - Synchronize the timestamp -
- Native Waypoint - Manage the waypoints for mission.
- Hotpoint - Manage the hotpoint for circling.
- Local Navigation - Go to specified local position
- Global Navigation - Go to specified global position
- Waypoint Navigation - Manage flying through a series of GPS coordinates)
- WebSocket With Baidu Map (for navigation) - Real time mapping.
- MAVLink And QGroundStation - Managing the vehicle to air communication.
These are the actions that onboard SDK open up, but the SDK has other layers, that go beyond the drone itself, and is more about the space and environment around a drone, and its interaction with this world.
Light Detection and Ranging (LiDAR) sensors are for commercial UAS applications, such as 3D aerial mapping, surveying, inspection, collision avoidance, and autonomous navigation in potentially either indoor and outdoor environments. There are three distinct elements of the drones LiDAR system:
- Lidar API - Supporting the computation of point cloud and logging LiDAR data in real-time into a standard LAS (LiDAR Aerial Survey) or PCAP (packet capture) file
- Lidar Simulator - A LiDAR simulator for playing back PCAP files in real-time via the same Ethernet output as a real LiDAR for facilitating development and debugging for the system integration
- Lidar Logging - An API based example to demonstrate how to use the API of real-time point cloud computation and LAS/PCAP logging. It can also be used for the use case of the integration of Velodyne LiDAR with M100 without the onboard SDK
LiDAR opens up another universe within the DJI onboard SDK, allowing for access the system through API, manage the logging around activity, also simulating common navigation elements in the environment.
uAvionix ADS-B Receiver
The pingRX ADS-B receiver allows the drone to receive real-time traffic information broadcasted by other manned or unmanned aircraft, as well as temporary flight restriction (TFR) information broadcasted by the government. With this type of situation awareness, the onboard embedded system (OES) will be able to make some safety-critical decisions like collision avoidance or self-separation.
Precision Trajectory Mission Planning
With the Onboard SDK Precision Trajectory Mission Planning suite, DJI developers can now plan complex missions without having to use GPS waypoints. The new DJI Precision Trajectory Mission Planning library has the flexibility to deal with complicated trajectories, issues with GPS accuracy and cases when GPS is simply unavailable.
- Trajectory following library that can autonomously execute preplanned smooth spiral trajectories
- SketchUp plugin to visualize trajectories, import 3D CAD models and geolocate the scene
- Configurable speed, start/end radii and pitch for the spiral
- Start your drone from anywhere - real-time path planning to get to the trajectory's GPS location
- Integration with DJI Assistant 2 to visualize simulations of the drone following the trajectory in the SketchUp scene
That is a pretty robust SDK. I'm taking the time to learn about each action, as well as the more communication and mission planning components separately. I can tell that they are having trouble to keep the large amounts of functionality and features coherent, and organized in the documentation, one of the reasons I'm breaking out separately here on my blog.
DJI Guidance SDK
Guidance SDK enables allows you to develop vision-based applications, by granting you full control over drone guidance. You can access all output data for any device using the DJI Guidance SDK--they break things down into five separate groups.
- Reset Config - Clear subscribed configure, if you want to subscribe data different from last time.
- Initialize Transfer - Initialize Guidance and create data transfer thread.
- IMU - Subscribe Inertia Measurement Unit (IMU) data.
- Ultrasonic - Subscribe ultrasonic data.
- Velocity - Subscribe velocity data, i.e. velocity of Guidance in body coordinate system.
- Obstacle Distance - Subscribe obstacle distance.
- Depth Image - Subscribe rectified greyscale image.
- Disparity Image - Subscribe disparity image, which can be filtered with functions such as filterSpeckles.
- Greyscale Image - Subscribe rectified greyscale image.
- Motion - Subscribe global motion data, i.e. velocity and position of Guidance in global coordinate system.
Set Callback and Exposure
- Event Handler - Set callback function handler. When data from Guidance comes, it will be called by data transfer thread.
- Exposure Parameters - Get stereo calibration parameters.
- Get Online Status - Get the online status of Guidance sensors.
- Get Sterio Calibration - Get stereo calibration parameters.
- Get Device Type - Get the type of devices. Currently only support one type of device: Guidance
- Get Image Size - Get the size of image data.
- Start Transfer - Inform guidance to start data transfer.
- Stop Transfer - Inform guidance to stop data transfer.
- Release Transfer - Release the data transfer thread.
- Wait For Board Ready - Set callback function handler. When data from guidance comes, it will be called by data transfer thread.
This SDK seems to be real time senses of the drone, allowing you to develop the experience you need to be in control, and guiding the device.
One of the thing that captivates me about the whole drone thing is its data collection capacity. I'm still learning about what is possible with the drone itself, but I know that much of the value generated by these flights will be based on the data that is gathered, as well as the images and video recorded. Here are the data points I have found so far in the DJI documentation for the DJI Guidance SDK.
- Error Code - Enumerates possible error codes. When error occurs, usually an error code will be given, and the developer can reference this enum to find the error type.
- Velocity Data - Velocity in body frame. The unit is millimeter/second and the frequency is 10 Hz.
- Obstacle Distance Data - Obstacle distance from five Guidance Sensors. The unit is centimeter and the frequency is 20 Hz.
- IMU Data - IMU data including accelerometer (in unit of acceleration of gravity g) and gyroscope (in quaternion format) data. The frequency is 20 Hz.
- Motion Data - Pose and velocity data including quaternion orientation, position in the global frame, velocity in the global frame.
- Ultrasonic Data - Outputs ultrasonic data from five Guidance Sensors, including obstacle distance (in unit of meter) and reliability of the data. The frequency is 20 Hz.
- Greyscale Image - Outputs Greyscale images for five directions. The image size is 320*240 bytes for individual sensor. The default frequency is 20 Hz and can be scaled down using API functions.
- Depth Image - Outputs depth images for five directions. The image size is 320*240*2 bytes for each direction. The default frequency is 20 Hz and can be scaled down using API functions.
- Disparity Image - Outputs disparity images for five directions. This data is useful when developers want to further refine the disaprity images using functions like speckle filter.
This data is generated constantly by a drone, and you have control over this transfer process through the DJI Guidance SDK. I'm thinking I need to aggregate some JSON schemas for this data, to better help me understand the depth and relationships in this data. There is a lot going on here, and a wealth of data to consider in a wide range of scenarios.
DJI Drone Mobile SDK
I use the DJI drone application to operate my two drones. The DJI Drone Mobile SDK is where you can get to work crafting your own custom application, to deliver exactly the drone operation experience you want. This is what the iPhone application was for mobile, but this is for the consumer and commercial drone world. There are a wealth of areas you can develop around in this SDK:
- Flight Controller - The flight controller is an onboard computer that combines control information from the pilot with sensor information to adjust the thrust at each propellor and fly the aircraft as desired.
- Camera - The camera captures photos and videos. Many different modes of operation, resolutions, frame rates, exposure settings, picture settings and file types can be selected. Cameras have local storage to hold the media which will typically be an SD card, and in some cases an SSD (solid state drive).
- Gimbal - Cameras fixed to an aircraft will record images that pitch and roll with the aircraft as it moves. Multi rotor aircraft need to pitch and roll simply to move horizontally, and so getting a stable horizontal shot is not possible.
- Airlink - AirLink describes the wireless link between aircraft, remote controllers, handheld cameras and mobile devices.
- Remote Controller - The remote controller allows manual flight, gimbal and camera control, and provides a robust wireless control link for aircraft. The mobile device can connect to the remote controller to communicate to the aircraft and receive the live video stream from the camera.
- Smart Battery - Smart batteries provide the energy required to run the products. Together with the flight controller, the smart battery can estimate remaining flight time and provide warnings when low battery thresholds are crossed. Batteries are easily swapped between flights, extending product use considerably.
- Missions - Missions can be used to easily automate flight. There are many different mission types that offer different product behavior. Some missions can be uploaded to and managed by the aircraft, while other missions are managed from the mobile device.
- SDK Manager - Application registration to use the DJI Mobile SDK, product connection, debugging and logging services are handled through the SDK manager class DJISDKManager.
I think that this stack of features speaks for itself. Providing a wealth of valuable API-driven resources to think about. This blows my mind as I begin to think about the possibilities for developing drone applications but gets even better when you think about how this also applies to the rest of the IoT world. Flight controller might not apply universally, but cameras, batteries, remote control, and the network are all ubiquitous with other IoT devices, and in my world should be considered beyond just drones.
I just needed to wrap my head around what programmatic resources are available to me as a DJI drone operator and developer. Next, I will be diving in and learning about some of the more interesting layers of this drone ecosystem, but first I am more interested in spending time looking through the platform API and SDK resources for other drone platforms, as well as some of the data solution providers like Airmap, and other physical components providers like FLIR for imaging, and LumeCube for lighting. I am always having to pick my battles on how deep on want to go down each rabbit hole or stay at the high level for a wider perspective.
There is a lot going on here. I find drones fascinating from a technical perspective, and terrifying when it comes to surveillance, privacy, policing, and some of the other bad behavior we've seen recently. Like other areas of the tech space, I think APIs are important for not just managing devices and the experience, but also provide transparency, logging, auditing, and other observability considerations when it comes to the Internet of Things space.
I get why people are interested in voice-enabled solutions like Alexa and Siri. I'm personally not a fan of speaking to get what I want, but I get the attraction for others. Similarly, I get why people are interested in bot enabled solutions like Facebook and Slack are bringing to the table, but I'm personally not a fan of the human-led noise in both of these channels, let alone automating this mayhem with bots.
In short, I'm not 100% on board that voice and bots will be as revolutionary as promised. I do think they will have a significant impact and are worthy of paying attention to, but when it comes to API driven conversational interfaces, I'm putting my money on push driven approaches to making API magic happen. Approaches like Push by Zapier, and Webtask.io, where you can initiate a single, or chain of API driven events from the click on a button in the browser, in a web page, on my mobile phone, or hell, using the Amazon Dash button approach.
These web tasks operate in an asynchronous way, making them more conversational-esque. Allowing those of us who are anti-social, and have adequate air gapped our social and messaging channels, and haven't fully subscribed to the surveillance economy, alternate solutions. These mediums could even facilitate a back and forth, passing machine readable values, until the desired result has been achieved. Some conversations could be predefined or saved, allowing me to trigger using a button at any point (ie. reorder that product from Amazon, retweet that article from last week). I'm not saying I don't want to have an API-enabled conversation, I'm just not sure I want a speaker or bot always present to get what I need to get done in my day.
I understand that I am not the norm. There are plenty of folks who have no problem with devices listening around their home or business, and are super excited when it comes to engaging with half-assed artificial intelligence wich accomplish basic tasks in our world(s). But I can't be that far out in left field. I'm thinking the landscape for conversational interfaces will be diverse, with some voice, some chat, and hopefully also some asynchronous approaches to having conversations that can be embedded anywhere across our virtual and physical worlds.
One of the things I love about my world as the API Evangelist is the time I get diving into rabbit holes and learning about different areas where technology is being applied. I do not always agree with the business motivations behind what is going on, which can result in some often pretty shady situations, but I enjoy stepping back and understanding the data, API and approaches behind what is going on.
I was doing some research on drone APIs recently, and as I was falling down the rabbit hole, I found myself reading about drones being a source of alpha? WTF is alpha? I had no idea and wanted to learn more about what an alpha generation platform was, and how drones and APIs are playing a role--here is the definition I found:
An alpha generation platform is a technology solution used in algorithmic trading to develop quantitative financial models, or trading strategies, that generate consistent alpha, or absolute returns. The process of alpha generation refers to generating excess returns.
The article that triggered this is about drones generating alpha was focused on the data generation, and communication capabilities of drones, and how they can be used in trading algorithms. Companies are looking to fly over crops, and predict yields, aggregate data, and add to the resources that available in the "alpha generation platform".
I'm not convinced of the reality of this approach, but it does provide for some interesting scenarios as I am learning about the data drones can gather, how this transfer occurs to the cloud, and be put to work using existing approaches to video, image, analysis, and visualization APIs. Also, it provides fuel for my alternate design fiction writing, where I explore the possibilities of technology, both good and bad.
From what I understand, this type of data would be considered "dirty" in the alpha workflow, in the same category of other data, you might gather from Amazon sales, Twitter sentiment, and other common API implementations. It's experimental, and not proven (yet) to produce the consistent or absolute returns I guess. What the article is discussing, are just early discussions, speculations, and desires of people looking to create a market--interesting fairy tales to tune into. Well, this is how markets operate right? On stories?
I'm adding high-frequency trading and alpha generation to my list of sectors potentially being impacted by drones. Another area I'm diving into that intersects with this work are satellite APIs. Satellite and drones intersect as drones are being used to fill in where satellite imagery can fall short (weather, low level buildings, etc.), but satellites are also being used as an alpha generation tool for high-frequency trading. I'm guessing drones and satellites are going to overlap with many other sectors like agriculture, mining, and the environment.
Lots of API and drone goodness to keep my mind busy.
Internet-connected devices generate data. The most recent wave of mobile devices has opened up an unprecedented world of data generation and harvesting from the network, device, and application layers. The location data, photos, videos, and other valuable exhaust from these devices is why there has been so much investment in technology, and why we are seeing continued investment in the Internet of Things (IoT).
When it came to mobile phones this opportunity was new, and it isn't always clear that we are the product when it comes to making money off connecting these devices to the Internet. People aren't always aware of how much data they are generating, and how much this data is generating revenue for the latest generation of entrepreneurs--because it's new. Things have moved along, and it's not a secret anymore that devices connected to the Internet generating data has the potential to be very valuable in the right situation.
I have been historically frustrated with people's lack of awareness of this, but I'm hoping that with each wave of technology that comes in the future, we will get smarter about this, and stop being the product when we can, and begin demanding a piece of this action (if we do it at all). If our wearable fitness device is used in any healthcare study, if that weather, water, or pollution sensor in our yard is generating revenue, we should get a piece of the action. If a device in our homes or businesses is generating data, we should be a more aware of, and willing participant in this new supply chain.
The average person may not always care about their privacy in the surveillance economy, but maybe they'll care about lost opportunities for making money. At the consumer level this isn't always a coherent argument, but as you approach the work at home world, and into the professional territory, it can begin making more sense. Not all weather and pollution monitors might make sense for this, but when you start thinking about participating in clinical trials and operating your own drone fleet in farm country, revenue share models for Internet-connected devices might seem a little more realistic.
Silicon Valley prefers an unaware, data-illiterate consumer, and even business class of users. I prefer a more aware, and data literate society, which is one reason I'm so big on APIs. Opening up 3rd party access, metering, and potentially revenue share opportunities for business, professionals, and individuals is one of the reasons I think APIs are important for all of this to work. I will keep studying how APIs are being used to generate, transmit, and generate revenue for platform providers, and device operators, so that I can hopefully keep pulling back the curtain on how the sausage is being made so that more of us can play in the game, and get a piece of the action.
I am learning about observability from reading Stripes post on Veneur, a high performance and global aggregation for Datadog. While the math of it all is over my head, the definition makes a lot of sense and provides me with a nice Venn diagram overlap across several areas of my API research, including testing, monitoring, logging, analysis, and visualization.
Formally, a system is said to be observable if, for any possible sequence of state and control vectors, the current state can be determined in finite time using only the outputs (this definition is slanted towards the state space representation). Less formally, this means that from the system's outputs it is possible to determine the behavior of the entire system. If a system is not observable, this means the current values of some of its states cannot be determined through output sensors.
Stripe provides a great technical breakdown of the tools, services used to establish observability as part of their system operations, but I wanted to step back, and think about observability through a business and political lens. The business imperative for observability might seem clear, as you want as much visibility and awareness into your API operations as you possibly can, so you can provide a reliable level of service. I am thinking about the incentives for extending this observability beyond internal groups, to partners, developers, regulators, or the public--encouraging transparent observability.
This moves into the area of API and algorithmic transparency you hear me ranting about regularly, and the benefits APIs bring to the table when you apply in areas like policing and surveillance, or other more regulated sectors of our society. When you take the assertions applied as part of API testing and monitoring practices, and you link them up to this observability stack definition, and open things up beyond just core technical groups, I think we are moving into new territory where we can begin to observe the real-time state of the systems that are increasingly making critical decisions in our lives.
I saw the potential for observability and awareness in the API layer back in 2010 with the logging, analytics, visualizations aspects of early API management solutions like 3Scale. In 2016, I'm thinking these benefits are going well beyond just the business of APIs, and can provide some of the critical transparency we are going to need to understand the behavior of increasingly complex corporate, institutional, and governmental systems, that in many cases we will have to be able to understand from the outside in using existing "output sensors".
In the future, we should be able to define any system as observable or not, when APIs are in play.
As I curate the interesting news from across the API space each week I tag things to put them into different buckets. At the end of each week, I look through each bucket, deciding which area(s) I will be writing about each week. I am always trying to identify patterns and evolve the different areas of my research. One of the areas I'm considering adding as a formal area of API research is when it comes to certification.
I have been seeing the area come up quite a bit, making me think it is something I should be thinking more about, and researching as its own project. This week there were four items thought caught my attention:
- BigML Certifications are Here! - Professional certification on machine learning APIs, from a leading API provider.
- Better Business Bureau yanks Wells Fargo's accreditation - Ok, not quite certification, but a classic representation and story to consider.
- Why cybersecurity certifications suck - A look into why certifications can be better, and applied in a critical area of the online world.
- Eligible Announces SOC2 Certification - Certification than an insurance API provider on their availability, processing integrity, confidentiality, or privacy.
These posts came after another four items from last week, providing, even more, to think about when it comes to APIs and certification:
- A new certification program for Open Source Hardware - I certify that I am truly open source, not that faux open source you see a lot of--we are the real deal!
- Online Dante Certification Program Now Available - Professional certification on audio APIs, from an older school software provider.
- Transform your business; become a Google Certified Professional Data Engineer - Professional certification that you know your stuff when it comes to data, specifically data in the Google ecosystem--data engineers are hot.
- IBM launches Watson application developer certification - Another professional certification for machine learning and artificial intelligence, this time from big blue.
There is a lot going on here when it comes to the concept of certification. I guess the BigML, Dante, and IBM certifications are more of what had in my head entering this post, but thinking about the shady shit that Wells Fargo has been up to, and thoughts on certifying that you are "ethical open source", or "honest open source" is fascinating to ponder as part of the big picture.
Another thing that stands out for me in these certification stories, is that there are two separate certifications for machine learning across the last two week. I can't help but think that there is some sort of purification going on in this ritual, or a sort of validation of the thing behind the certification, meaning that if you certify that professionals know this thing, then the thing, must be a thing. Right? IDK, I just can't help but think more of the motivations behind doing certifications, and the psychology behind how they are perceived (right or wrong)
I'm inches away from adding certification as an official API research area, so I can keep a closer eye on it. I've never been a big certification guy, but I understand why they exist, and why some people invest in them. If I add it as a research area, I'll dive in deeper to map out which API certifications exist out there, and which ones might provide a good model for other API providers to look at, or maybe even what to consider NOT doing when crafting your certification program.
I am seeing more people asking that we put on the brakes when it comes to technology, looking to slow the adoption of new technology, in favor of mastery of the existing, and getting our house in order with the technology we already in play. One of the core tenets of my message as the API Evangelist centers on the importance of doing what we are already doing and doing it better. You can see this message present in my 2014 recommendation on an API strategy for the US federal government--do more of what is already in motion, don't disrupt by just doing the new.
I have seen a lot of API technology float by in the last six years of doing API Evangelist. I can still get excited by some of it, but far less than I did in 2010 when I first started. This is partly because I'm tired, but mostly it is because I've seen a lot of shit float by, and it has to be meaningful in some way to get me excited. I am fairly willing to keep an open mind when it comes to microservices, DevOps, GraphQL, voice, bots, drones, and the other technological frontlines, but I refuse to accept that technology has to mindlessly march forward. This is more about selling us new things than it is every about truly bringing us real solutions to everyday problems.
Whether it's addressing technical debt and monoliths, the security concerns of the Internet of Things, or anything in between, we should always work to step back and ask if the new technology will actually provide a solution, or create three new problems for each solution it brings. I am not anti new tech, I love new shiny tech objects, but I think for our own sanity we should learn to be more thoughtful. New technology trends can be exciting, and fun to play with, but when it comes to production environments, and getting important things done, maybe it should stay in the R&D lab or sandbox until it matures.
Maybe we can spend 2017 just taking care of what we already have in motion, and put a moratorium on any new and exciting projects until we get better at testing, usability, security, and other areas we seem to regularly suffer from in our operations. Maybe we can just do all those API implementations that we have the queue, over the next version of existing resources. IDK, if nothing else, maybe we can just a breather and allow us to play catch up a little, and ponders on what matters most.
|<< Prev||Next >>|