10 Aug 2016
The messaging platform Slack made waves when they launched their Slack Fund as part of their API release, putting up $80M to invest in developers who were interested in building cool things on the API. Slack has continued telling their story, talking about how they have invested some of the fund and are being pretty transparent about which API integrations made the cut.
After reading about the Slack Fund I wanted to see which other APIs were investing in their communities like this. Next, I came across the Alexa Fund where Amazon is investing in developers building voice-enabled apps, giving their Echo platform the boost of applications it will need to find success. After poking around the AWS platform, you also find they have also had their AWS Grants program, investing in the best of breed projects that use the AWS platform to give back to the world.
I came across a number of other fund announcements about available funds, only to find the pages gone, but MailChimp had an interesting accounting of their fund which they started in November of 2010, pledging $1M to "provide developers with funding assistance to build integrations on our email platform". I also found another fund developer fund out of Cisco, to encourage development with Cisco Spark and Tropo, and the other APIs they provide at Cisco DevNet, that was noteworthy.
There are not enough examples of API investment funds out there for me to add the concepts as what I'd consider being a common API building block, but there are enough out there, especially from leading pioneers, to keep me paying attention. API developer funds don't seem like something you have to go as big as AWS or Slack did, but it seems to me, if you provide a small fund like MailChimp did, it could incentivize development significantly--at least make your launch be fewer crickets.
10 Aug 2016
I am always looking for simple advice I can give to API providers to make their developers and would-be developers integration as friction-less as possible. When it comes to making the code available for your API I recommend providing a wealth of samples, libraries, and full blown SDKs for your developers. Also make sure these code resources are as easy to find within your developer portal, as well as being openly licensed and available on your API platform's Github profile.
Adding to this advice I would also recommend that you take the top 10 programming languages and platforms out there, and perform a quick Google search for the key phrase, and see what comes up, on a regular basis. I tend to find interesting conversations around an API when I do this. Inversely it can also provide some interesting signals that an API isn't very active within specific communities, and there probably isn't much existing support.
For me, searching for API [name] + [programming language] or [platform name] = some interesting reflections of activity around an API. I can also imagine that this would provide a way for API providers to learn where the gaps in their outreach are, and give them another way to reach their developers or potential developers--demonstrating that SEO plays a significant role in successful API operations.
09 Aug 2016
I am still learning about the recent API release out of Apple, and working to keep up with the volume of resources coming out of the Amazon Alexa ecosystem, but one thing I know for sure is Amazon is way ahead of the game, and Apple will have to invest a lot more resources if they expect to catch up. Apple may have the device advantage, but Amazon is leaps and bounds ahead when it comes to how you develop a community in support of voice-driven applications.
I wouldn't be surprised if Apple's release was in response to the waves Amazon is making with their voice platform. When it comes to the platform battles between tech giants, they can throw the necessary resources at things to play catch up, but it is something that other smaller players will not be able to always navigate. I'm reviewing a number of device-focused API efforts lately, and the ones who launched with an API are always ahead in the game.
Learning to do APIs well takes time, and learning to grow your API community of developers and partners will always take time to do properly--you just can't throw resources at it to make things scale properly. You might as well kick things off with an API and platform approach, even if it is in beta, and limited--at least you are getting going in a limited environment. Siri has been out there for a while now, but not as a platform, and I'm going to wager this will be a significant handicap.
What is your approach to technology? Is it more like Siri, or is it more like Alexa?
09 Aug 2016
I was reading a post from the Electronic Frontier Foundation asking some great questions about having a safety protocol for things like the DARPA's Cyber Grand Challenge. It is good to see EFF leading this conversation, providing us with some important things to consider as part of any security related work, whether it's in the cyber theater or just guiding your ethics as a small business, analyst or freelance hacker (I happen to be all three).
To hopefully contribute somewhat to the conversation, I wanted to think through some of the questions I ask when defining and helping folks secure their slice of the API landscape. There are two distinct portions of the conversation I wanted to chime in on, speaking to the areas that I find the most friction with when mapping out, and securing infrastructure.
The Dangers Of Being An API Cartographer In The Shadows Behind Mobile And Internet of Things
Only a small fraction of web APIs available today are intentionally open to the public. The majority of web APIs out there are developed to support web, mobile, or device-based apps, delivering resources on the open Internet, but not meant to be viewed, accessed, and documented by 3rd parties.
API developers and architects heavily focused on the end deliverable, the web, mobile, system, or device integration, they do not consider the API to be open to the public--thus perceived as a private API, even though it uses public infrastructure. Many people frown on even mapping out this shadow API world, let along making and API definition or documentation publicly available.
If I can publicly get access to a website, application, or device, I can map out the APIs, simply by routing traffic on my computer through a proxy. Before we can ensure this portion of our public infrastructure is secure, we must map it out, and we must change popular opinion to view this infrastructure as public, and begin securing it like it is, sharing maps of the infrastructure, and having an "open" conversation about it--not creating friction for those who do.
A Fine Line Between Scanning For API Vulnerabilities And Executing a DDoS Attack On A Domain
Once I have a map of the landscape, I can begin looking for vulnerabilities. Establishing the map (API definition) is a grey zone for me, but actually scanning that map looking for even the most common of vulnerabilities is black, and a no-go zone for me. Before I can scan any domain used as part of API infrastructure, I will need the approval of the domain owner. Even a low-level security scan for the most common vulnerabilities will come across as a potential DDoS attack--requiring deeper considerations before doing it.
Much like the EFF sis looking to encourage a conversation around a safety protocol for cyber challenge related activity, I'd like to encourage a conversation about mapping and scanning of publicly available API infrastructure. I'd like to discuss the importance of having public maps be available, much like we do with physical infrastructure, both public and private. Then I'd like to also talk about how we can collectively ensure this infrastructure is secured properly, and have a sensible conversation about this, on a regular basis.
I am not suggesting anything wild here. I do not get in trouble for driving by our local hospital, my child's school, or the local water plant. I shouldn't get in trouble for reporting a hole in the fence, or suspicious activity that I see when I drive or walk by either. I should also be able to be reassured that the hospital, school, and local water utility have a solid security plan, and have my best interest in mind when it comes to my healthcare data, student information, and water usage and billing information. Why is the virtual space so different?
I am just putting this out there, as I do. I walk this ethical line on a daily basis as I monitor the API space, and wanted to make my stance known. I am going to keep mapping out the known API universe, I feel comfortable pushing the boundaries of this grey zone. I will also be scanning, and helping secure API infrastructure using my API security service Sapience, but of course only when a customer asks me to. ;-)
09 Aug 2016
According to Wikipedia, in mathematics and computer science, an algorithm is a self-contained step-by-step set of operations to be performed. Algorithms perform a calculation, data processing, and/or automated reasoning tasks. These often very abstract inventions are increasingly guiding our personal and professional worlds from deciding what we read on Facebook, scanning the videos uploaded to your Twitter and Vine feed, to whether or not you might commit a crime again, and even predict potential police misconduct.
I see the concept of an algorithm applied in many different ways, some are very simple and easy to understand, and some are not so easy to understand, and operate as black boxes. One of the reasons I like APIs so much is the possibility of introducing some transparency into how algorithms work (or don't). APIs do not give visibility into the inner workings of algorithms by default, but when they are done right they can significantly contribute to a wider understanding of how they work.
As with other areas of the API space that I research, I'm intrigued with algorithms, how they are being used, and the role APIs can play in the wider conversation. I added an APIs and algorithms research project, to help be understand how algorithms are being wielded, the news, companies, and code and schemas which are framing the conversation. The information I aggregate here will fuel my overall perspective of the intersection of API and algorithms, and maybe yours too.
09 Aug 2016
When I talk about how much I believe in APIs, all I am really saying is how much I believe in the web. The web is how humans are consuming and sharing data, content, and algorithms with each other using the Internet, and APIs are how systems, applications, and devices are doing this via the Internet.
I do not advocate for any single API, company, tooling, or technology. I am evangelizing for an open web. This is why I am always a little ashamed at how incomplete my understanding is of the core concepts that make up the web and the specifications that are driving it. Thankfully there are people like Erik Wilde (@dret) who work tirelessly to help us understand them, and make available in a simple way like he did with his Web Concepts project.
Erik has organized 13 Web Concepts for use to learn about, detailing 445 distinct specifications across the International Organization for Standardization (ISO), Internet Engineering Task Force (IETF), Java Community Process (JCP) and the World Wide Web Consortium (W3C). There are a number of these specifications I'm already aware of, but there are numerous others I am not, and that just isn't right.
To help me increase my awareness and hopefully yours as well, I'm going to regularly tweet out individual concepts, and their specifications, linking to Erik's work. I encourage you to bookmark his work and get to work improving your web literacy where possible, and consider forking his work and sharing within your organization as well. It is important that we are all aware of these core building blocks of the web, and are reusing existing concepts and specifications across our work, wherever possible.
08 Aug 2016
I wrote a story last week about how Airmap is positioning itself to be an API broker within the fast growing Drone industry. I also stumbled across the satellite imagery API out of Astro Digital, and specifically their 3rd Party Real-Time Data API which is designed "to combine mass amounts of data, like adding live weather data on top of imagery"--resembling the layer of the drone industry which Airmap is investing in.
Airmap is delivering geo-intelligence for applications used when flying drones, Astro Digital has an API which injects geo-intelligence at the satellite image layer--I guess the satellite layer would be one layer up from the drone layer when it comes to APIs providing access to our physical world. Which leaves me considering all the other potential layers of our physical world being opened up using APIs.
Security cameras, sensors, and other Internet-connected devices are rapidly opening up our physical world using APIs. The API solutions being opened up by Airmap and Astro Digital are allowing us to access, inject, remix, and make decisions in real time. I find this intriguing because there are so many dimensions present beyond just a single resource being made available via an API, and allows for true multi-API integration, while also employing APIs to direct physical activity like drone flights.
Beyond the myriad of API implementations that are available within this new landscape, I feel it is also is something that demonstrates the opportunity surrounding weather data, government open data (FAA situation data), and other relevant data, content, media, and algorithms being opened up today. I find it endlessly fascinating to try and keep up with the different ways that API-driven resources are being used to open up and generate revenue, taking advantage of opportunities that are emerging as this new API landscape expands both digitally and physically.
08 Aug 2016
In 2006 Amazon introduced me to the idea that you could deploy global infrastructure using web APIs. This was when web APIs went from the hobby level (ie. photos, social, links) workbench, and became a tool I could actually use in my business toolbox. Web APIs weren't just used or fun and games anymore, opening up my eyes to the endless possibilities when you make the web programmable using APIs.
In 2016, I see a number of APIs show up on my monitoring radar, and honestly, most of them are not that notable--then you come across the solutions like the dead simple satellite imagery API from Astro Digital. I'm increasingly finding myself tracking on the collision APIs are having with our physical world, so I see a lot of interesting (and some not so) APIs, but I find the notion that you can get the latest picture of a specific place on earth is a pretty significant milestone for web APIs.
For me the Astro Digital satellite imagery API demonstrates for me the power of the Uniform Resource Locator (URL) and the Uniform Resource Identifier (URI)--I can get a snapshot of any location on earth, and manipulate my view of it using a single URI, from within any web or mobile application. This is just one of many APIs available to the average developer today, providing access to a wealth of high-value resources, that used to only be accessible to larger companies who could afford it.
I just wanted to pause for a moment, and take notice of some of the important resources that are being made available via APIs today, because I feel like the landscape is shifting pretty quickly with the introduction of these valuable resources. When companies small companies can work with the same resources as large enterprises and institutions, interesting things are possible.
05 Aug 2016
I have done a lot of reading in the last week, catching up on my monitoring of the API space. I have read a couple of posts about the reliability of APIs, and the overall viability of building applications, and businesses based upon them. A couple of the posts were focusing on the shuttering of ThinkUp, but a couple others were just part of the regular flow of these stories that question whether we can depend on APIs or not--nothing new, as this is a regular staple of bloggers and the wider tech blogosphere.
My official stance on this line of thinking is that I would not want to be building a business, and application that depends on leading API platforms like Twitter, Facebook, Instagram, and others, but I will happily build a business, applications, and system integration on APIs. You see, this isn't an API issue, it is a business and vendor viability issue. As with other sectors, there are badly behaved business, and there are well-behaved businesses--I try to choose to do business with the well-behaved one's (can't always achieve this, but I try).
I find the ongoing desire of the startup culture to point out how unreliable APIs are, while simultaneously supporting the overall business tone set by venture capital investment, often delusionally blind levels of support, is just not reconcilable. I'm not saying all VC investment is bad, so don't even take this down that road. I am saying that the quest for VC investment, and the shift in priorities once VC investment is acquired, then further shift with each additional round, and the final exit is setting a tone that is completely at odds with API reliability.
The problem really begins when APIs become the front-end for this blame. If I depend on vendors for my brick and mortar store, and the delivery trucks don't reliably bring my products, I don't talk about how you can't depend on trucks--I find new vendors. Of course, I can't find new vendors if they can't be replaced like Twitter and Facebook, but this is a whole other conversation, although it is also one that is a symptom of the tone being set by the VC currents (this is a business conversation). Blaming APIs instead of raising questions about the business ethics bar being set by venture capital shows the blinding power of greed, as the tech community refuses to blame VC $$, and shifts this to being about the viability of APIs, because I will get my VC $$ some day too bro!
I am not saying APIs are always good. I'm just saying they aren't bad. Hell, they aren't neutral. They are a simply a reflection of the business behind them, as well as being a reflection of the industry they operate in. Stop blaming them for businesses not giving a shit about developers, and the end-users. Maybe we could start changing the tone by admitting the #1 priority is always set by VC $$, and not by our API community, or even our end-users and customers, and all this shit is out of whack.
05 Aug 2016
I've seen quite a few incomplete API efforts in my time as the API Evangelist. It is why I aggregate the most common building blocks I see across successful API providers like Twilio and organize them into blueprints anyone can follow like my minimum viable API footprint definition. It doesn't take much effort to pull together a basic API presence like this and is something that can go a long ways to help onboard new API consumers.
There are a number of interesting APIs out there like the AirControl API, which gives you programmatic access to enterprise-grade wifi and wireless network solutions from Ubiquiti Networks, but then quickly reduces a pretty robust API effort to being just a PDF document. I still keep an eye on these APIs like this because they offer value, and give a look at how ubiquitous web APIs are becoming. It also demonstrates how much more work I have ahead of me when it comes to educating the coming wave of device API providers.
Another thing this demonstrates for me is that this wave of Internet-connected device APIs will also suffer from many of the same illnesses that API providers have had when deploying APIs for mobile applications. These APIs are often hastily designed and deployed, where documentation, authentication, and security are often an afterthought or not even a thought at all.
I am working on reworking some of my forkable Github templates for deploying API portals. I am going to create a minium viable API portal solution that can be easily forked, helping API providers quickly launch a more coherent presence without much effort. I'm going to also create a complete API portal solution that takes into consideration the complete list of building blocks for an API presence that I've gathered from across the API landscape.
05 Aug 2016
As the number of available API definitions out there grows, I am increasingly coming across variations of APIs that I already have included in my API Stack. It can be tedious to try and sync these with existing copies, so I wanted to take a look and see if there was anything already available out there, that would help provide a diff for either OpenAPI Spec, RAML, or API Blueprint.
The most common one I found referenced was a Ruby one for OpenAPI Spec from Civis Analytics. Next, I found a pretty interesting web tool, which provides its Node.js source code on Github, including a CLI edition. After these two solutions, I only found one more for RAML, but couldn't find anything for API Blueprint.
I wanted to think beyond just the common API definition formats and look for diff tools that would work just for JSON. I found Diff which was pretty interesting, as well as DiffSync which provided a real-time diff for JSON. I haven't tried any of them out, I am just trying to compile a list of what is out there before I dive into what their capabilities are.
I have a master database of API definitions where I can just import API definitions that I find, and it will add any new paths, parameters, and schemas it comes across. I would much rather have the ability to just run each version through a diff tool, and be able to select which elements I want to have merged. For now, I will just implement a custom layer to my API monitoring system, but I would like to see API diff move forward similar to we saw the conversation evolve with the introduction of API Transformer.
05 Aug 2016
I first came across Airmap as I was learning about them acting as the middle man for DJI drone updates via the Department of Interior. After this story, I added them to my database of companies that I track on. Then as I was monitoring my feeds I saw they were now working with IBM to get weather information for use by drone manufacturers, operators, and those impacted by drone activity.
Airmap positions themselves as "airspace Intelligence", offering low-altitude airspace management solution for unmanned aircraft. When you consider the drone as just one of many devices being connected to the Internet, the role intelligence providers like Airmap will play when it comes to system to system integrations, and web or mobile application development in each space--the company seems to be well-positioned for success in the drone industry.
Airmap is working hard to define the physical space, the constants, as well as some of the most ephemeral layers of our physical world like weather, forest fires, and events that are increasingly impacting our online reality via the web and our mobile phones. Airmap is aggregating valuable data, like the updates they get from the Department of Interior (open data) to the partner data like they will be ingesting like the weather data from IBM.
Airmap provides access to their flight conditions and other airspace data from Airspace via API(s), and has established some pretty significant manufacturer relationships, as well as open and partner data sources--providing the beginnings of a pretty interesting Internet of things business blueprint for what I'd consider to be a sort of API broker. When you consider the influence some of these API brokers have in the governance of airspace around nuclear power plants or even the Olympics, and govern an assortment of Internet-connected devices--the API opportunity appears to be pretty interesting.
04 Aug 2016
I was happy to stumble across the Better Business Bureau API the other day. I was working on a piece about "the API economy", and looking for real world examples of how APIs could actually contribute positively (or negatively) to the actual economy, and found the useful API.
The Better Business Bureau provides five API endpoints for consumers:
- Organization Search - This endpoint will allow you to search for Organizations that are known to the BBB. Organizations may include businesses or charities.
- Organization Collections - BBB Partners will need to be able to identify collections of Organizations that they care about and be able to manage those collections and receive Organization info and updates on them as needed.
- Bulk Retrieval - This section defines the message specification that will be sent back to the API partners who request a bulk data transfer.
- Push Notifications - BBB Partners have the need to be notified when BBB data updates occur for the records or a collection of records they are currently subscribed to.
- BBB Investigation/Application for Accreditation - This endpoint allows submission of organizations for BBB investigation and application for BBB accreditation.
They provide a pretty valuable service, which is significantly amplified when there is an API present. Being able to find out which companies are accredited or have complaints against them in real time has endless potential for impacting real world decisions that may influence the economy one way or the other. Being able to automate this process for a digital age, convinced me that this should be in the list of APIs that reflect my thoughts on the wider API economy.
I will be including the Better Business Bureau API in my overall API ranking research, which is something I've explored in the past, but will be spending some more time thinking in the context of the API economy. I will also profile the BBB API, to contribute to the design patterns I will be considering when it comes to designing business and API ranking, investigation, and accreditation resources.
04 Aug 2016
I was reading some of the blog posts, Reddit and Hacker News threads about Microsoft's release of their API design guide. While many of the comments are technically correct, the tone in which they are delivered, and the intent behind their delivery is one of the biggest contributing factors to why we have such wild inconsistencies in API design across the landscape.
API Evangelist was born in 2010 out of frustration with the Restafarian pundit class. I made it my mission to help tell stories of the technical, business, and politics of APIs in an inclusive way, that didn't make people feel stupid for what they didn't understand. Even after six years of doing this, I am still learning things about APIs and being regularly reminded that there are few absolutes when it comes API design, definition, deployment, management, or any of the other 50+ stops along the API life cycle that I track on.
I stopped sharing my blog posts to Hacker News, Reddit, and DZone (reversed after they invested in change) because of hostile responses to my thoughts. I cannot imagine what it is like to work hard on your API design strategy at a company, then share publicly, only to get flamed because it wasn't REST compliant. Many folks will never even attempt this for fear of being flamed, and those that do, will often never make the mistake twice.
This is the opposite of what we want. We want EVERYONE organizing, publishing, and sharing their API design patterns so that we can all learn from each other. We don't want to make people feel stupid for what they do not know. We want to encourage everyone to learn, grow, evolve, and build on each other's design patterns and work from across the API sector.
Let's work to encourage more companies to craft, publish, and share their API design patterns. Let's please stop REST shaming folks, it didn't work in 2010, and it doesn't work in 2016. Making people feel dumb will never get us to a place where API developers and architects are truly HTTP literate, and well informed of existing patterns, let alone actually putting them to work across their organizations.
04 Aug 2016
I saw the potential for collaboration when it came to using web APIs back around 2004 and 2005. I was seeing innovative companies opening up their digital assets to the world using low-cost, efficient Internet technology like HTTP, opening things up for potentially interesting approaches to collaboration around the development of web and mobile applications on top of valuable digital resources. This approach has brought us valuable platforms like Amazon Web Services and SalesForce.
Common API discussions tend to focus on providing APIs to an ecosystem of developers and encouraging the development of web and mobile applications, widgets, visualizations, and other integrations that benefit the platform. In the course of these operations, it is also customary to gather feedback from the community and work to evolve the APIs design, available resources, and even the underlying data model--extending collaboration to also be about the APIs, and underlying resources, in addition to just building things on top of the API.
This approach to designing, defining, and deploying APIs, and then also web and mobile applications on top of these APIs is nothing new, and is something that I have been tracking on for over the last six years. The transparency that can be injected into the evolution of data, content, and potentially the "algorithms behind" with APIs is significant, which is how it became such a big part of my professional mission, and fueling my drive to spread the "gospel" whenever and wherever I can.
Ok, so how can APIs contribute to algorithmic transparency? To fully grasp where I am taking this, you need to understand that APIs can be used as an input and output for data, content, as well as algorithms. Let's use Twitter as an example. Using Twitter and the Twitter API I can read and write data about myself, or any user, using the /account and /users API endpoints--providing the content and data portion of what I am talking about.
When it comes to the algorithm portion, Twitter API has several methods, such as GET statuses/user_timeline, GET statuses/home_timeline and GET search/tweets, which return a "timeline of Tweet data". In 2006 this timeline was just the latest Tweets from the users you follow, in sequential order. In 2016, you will get "content powered by a variety of signals". In short, the algorithm that drives the Twitter timeline is pretty complicated, with a number of things to consider:
- Your home timeline displays a stream of Tweets from accounts you have chosen to follow on Twitter.
- You may see suggested content powered by a variety of signals.
- Tweets you are likely to care about most will show up first in your timeline.
- You may see a summary of the most interesting Tweets you received since your last visit
- You may also see content such as promoted Tweets or Retweets in your timeline.
- Additionally, when we identify a Tweet, an account to follow, or other content that's popular or relevant, we may add it to your timeline.
There are a number of considerations that would go into any one timeline response--this is Twitter's algorithm. While I technically have access to this algorithm via three separate API endpoints, there really isn't much algorithmic transparency present, beyond their overview in the support section. Most companies are going to claim this is their secret sauce and their intellectual property. That is fine, I don't have a problem with y'all being secretive about this, even though I will always push you to be more open, as well as leave the API layer out of your patents you use to pretect your algorithms.
Algorithmic transparency with APIs is not something that should be applied to all APIs in my opinion, but for regulated industries, and truly open API solutions, transparency can go a long way, and bring a number of benefits. All Twitter (and any other API provider) has to do is add parameters, and corresponding that open up the variables of the underlying algorithm for each endpoint. What goes into considerations about "what I care about", constitutes "interesting", and what makes things "popular or relevant"? Twitter will never do this, but other API providers can.
It is up to each API provider to decide how transparent they are going to be with their algorithms. The ideal solution when it comes to transparency is that the algorithm is documented and shared along with supporting code on Github, like Chicago, did for their food inspection algorithm. This opens up the algorithm, and the code behind for evaluation by 3rd parties, potentially improving upon it, as well as validating the logic behind--potentially opening up a conversation about the life of the algorithm.
There are a number of common reasons I have seen for companies and developers not opening up their algorithms:
- It truly is secret sauce, and too much was invested to just share with the world.
- It is crap, and the creator doesn't want anyone to know there is nothing behind.
- There are malicious things going on behind the scenes that they do not want to be public.
- Insecurities about coding abilities, security practices and logic applied to the algorithms.
- Exist in competitive space with lots of bad actors, and may want to limit this behavior.
- What is accomplished isn't really that defensible, and the only advantage is to keep hidden.
I have no problem making an argument for algorithmic transparency when it comes to regulated industries, like financial, healthcare, and education. I think it should be default in all civic, non-profit, and other similar scenarios where the whole stack should just be open sourced, and available on Github. You won't find me pushing back to hard on the startups unless I see some wild claims about the magic behind, or I see evidence of exploitation, then you will hear me rant about this some more.
Algorithmic transparency can help limit algorithmic exploitation and the other shady shit that is going on behind the scenes on a regular basis these days. I have added an algorithm section to my research, and as I see more talk about the magic of algorithms, and how these amazing creations are changing our world--I am going to be poking around a bit, and probably asking to see more algorithmic transparency when I think it makes sense.
04 Aug 2016
I find myself thinking about what the terms of service we agree to for online services are doing to our lives. Whether we see the effects or not, they are guiding almost everything we do in our personal and professional worlds. They are something that began on our desktop computers, migrated to our more mobile laptops, then deeper into our personal lives via our mobile smartphones.
We rarely understand what we are agreeing to when we check the box for these terms of service, which leaves me really surprised to see us accept the application of terms of service to everyday objects we depend on, as part of the Internet of Things movement. As physical objects in our lives get connected to the Internet, they begin to generate valuable data, requiring software to manage--all being directed by the terms of service set forth by the manufacturer.
You can see this playing out as manufacturers like GM and John Deer are limiting what you can do with their physically connected equipment. These terms of service are changing the meaning of ownership, and evolving te physical things we purchase into subscriptions, instead of outright ownership. Following the tone of the conversation set by terms of service which is guiding software, this new breed of terms of service being applied to physical objects in a way that heavily benefits manufacturers, shifting as much the risk as possible to the consumers, leaving many of the benefits as possible for themselves.
In the rapidly expanding online and mobile worlds we (consumers) have almost no tools that help us manage the terms of service we are agreeing to, and almost no leverage when it comes to how these are crafted, changed, and applied. Why the hell would we keep this rapid pace continuing into the physical world, as well as our online worlds? It just doesn't pencil out? Especially when you consider how we give up so much of our privacy, security, and ownership as in agreeing to these terms of service.
How terms of service are applied to virtual and increasingly physical objects that are being connected to the Internet are much like algorithms, as they are often coded black boxes that we do not always understand. I will keep tracking on them the best I can, but as a community, we need a lot more resources if we are going to shift the balance of all of this more in the favor of the average consumer.
03 Aug 2016
I am working my way through Title 45, which is the principle set of rules and regulations issued by federal agencies of the United States regarding public welfare. I've made my way through the security standards, and the breach notification portions, but can't help but feel that much of the "uses and disclosures of protected health information" section should be closely evaluated for consideration beyond healthcare.
I am currently processing this, for use as building blocks in my privacy research:
(a) Standard. A covered entity or business associate may not use or disclose protected health information, except as permitted or required by this subpart or by subpart C of part 160 of this subchapter.
(1) Covered entities: Permitted uses and disclosures. A covered entity is permitted to use or disclose protected health information as follows:
(i) To the individual;
(ii) For treatment, payment, or health care operations, as permitted by and in compliance with §164.506;
(iii) Incident to a use or disclosure otherwise permitted or required by this subpart, provided that the covered entity has complied with the applicable requirements of §§164.502(b), 164.514(d), and 164.530(c) with respect to such otherwise permitted or required use or disclosure;
(iv) Except for uses and disclosures prohibited under §164.502(a)(5)(i), pursuant to and in compliance with a valid authorization under §164.508;
(v) Pursuant to an agreement under, or as otherwise permitted by, §164.510; and
(vi) As permitted by and in compliance with this section, §164.512, §164.514(e), (f), or (g).
(2) Covered entities: Required disclosures. A covered entity is required to disclose protected health information:
(i) To an individual, when requested under, and required by §164.524 or §164.528; and
(ii) When required by the Secretary under subpart C of part 160 of this subchapter to investigate or determine the covered entity's compliance with this subchapter.
(3) Business associates: Permitted uses and disclosures. A business associate may use or disclose protected health information only as permitted or required by its business associate contract or other arrangement pursuant to §164.504(e) or as required by law. The business associate may not use or disclose protected health information in a manner that would violate the requirements of this subpart, if done by the covered entity, except for the purposes specified under §164.504(e)(2)(i)(A) or (B) if such uses or disclosures are permitted by its contract or other arrangement.
(4) Business associates: Required uses and disclosures. A business associate is required to disclose protected health information:
(i) When required by the Secretary under subpart C of part 160 of this subchapter to investigate or determine the business associate's compliance with this subchapter.
(ii) To the covered entity, individual, or individual's designee, as necessary to satisfy a covered entity's obligations under §164.524(c)(2)(ii) and (3)(ii) with respect to an individual's request for an electronic copy of protected health information.
(5) Prohibited uses and disclosures.
(i) Use and disclosure of genetic information for underwriting purposes: Notwithstanding any other provision of this subpart, a health plan, excluding an issuer of a long-term care policy falling within paragraph (1)(viii) of the definition ofhealth plan, shall not use or disclose protected health information that is genetic information for underwriting purposes. For purposes of paragraph (a)(5)(i) of this section, underwriting purposes means, with respect to a health plan:
(A) Except as provided in paragraph (a)(5)(i)(B) of this section:
(1) Rules for, or determination of, eligibility (including enrollment and continued eligibility) for, or determination of, benefits under the plan, coverage, or policy (including changes in deductibles or other cost-sharing mechanisms in return for activities such as completing a health risk assessment or participating in a wellness program);
(2) The computation of premium or contribution amounts under the plan, coverage, or policy (including discounts, rebates, payments in kind, or other premium differential mechanisms in return for activities such as completing a health risk assessment or participating in a wellness program);
(3) The application of any pre-existing condition exclusion under the plan, coverage, or policy; and
(4) Other activities related to the creation, renewal, or replacement of a contract of health insurance or health benefits.
(B) Underwriting purposes does not include determinations of medical appropriateness where an individual seeks a benefit under the plan, coverage, or policy.
(ii) Sale of protected health information:
(A) Except pursuant to and in compliance with §164.508(a)(4), a covered entity or business associate may not sell protected health information.
(B) For purposes of this paragraph, sale of protected health information means:
(1) Except as provided in paragraph (a)(5)(ii)(B)(2) of this section, a disclosure of protected health information by a covered entity or business associate, if applicable, where the covered entity or business associate directly or indirectly receives remuneration from or on behalf of the recipient of the protected health information in exchange for the protected health information.
(2) Sale of protected health information does not include a disclosure of protected health information:
(i) For public health purposes pursuant to §164.512(b) or §164.514(e);
(ii) For research purposes pursuant to §164.512(i) or §164.514(e), where the only remuneration received by the covered entity or business associate is a reasonable cost-based fee to cover the cost to prepare and transmit the protected health information for such purposes;
(iii) For treatment and payment purposes pursuant to §164.506(a);
(iv) For the sale, transfer, merger, or consolidation of all or part of the covered entity and for related due diligence as described in paragraph (6)(iv) of the definition of health care operations and pursuant to §164.506(a);
(v) To or by a business associate for activities that the business associate undertakes on behalf of a covered entity, or on behalf of a business associate in the case of a subcontractor, pursuant to §§164.502(e) and 164.504(e), and the only remuneration provided is by the covered entity to the business associate, or by the business associate to the subcontractor, if applicable, for the performance of such activities;
(vi) To an individual, when requested under §164.524 or §164.528;
(vii) Required by law as permitted under §164.512(a); and
(viii) For any other purpose permitted by and in accordance with the applicable requirements of this subpart, where the only remuneration received by the covered entity or business associate is a reasonable, cost-based fee to cover the cost to prepare and transmit the protected health information for such purpose or a fee otherwise expressly permitted by other law.
(b) Standard: Minimum necessary— Minimum necessary applies. When using or disclosing protected health information or when requesting protected health information from another covered entity or business associate, a covered entity or business associate must make reasonable efforts to limit protected health information to the minimum necessary to accomplish the intended purpose of the use, disclosure, or request.
(2) Minimum necessary does not apply. This requirement does not apply to:
(i) Disclosures to or requests by a health care provider for treatment;
(ii) Uses or disclosures made to the individual, as permitted under paragraph (a)(1)(i) of this section or as required by paragraph (a)(2)(i) of this section;
(iii) Uses or disclosures made pursuant to an authorization under §164.508;
(iv) Disclosures made to the Secretary in accordance with subpart C of part 160 of this subchapter;
(v) Uses or disclosures that are required by law, as described by §164.512(a); and
(vi) Uses or disclosures that are required for compliance with applicable requirements of this subchapter.
(c) Standard: Uses and disclosures of protected health information subject to an agreed upon restriction. A covered entity that has agreed to a restriction pursuant to §164.522(a)(1) may not use or disclose the protected health information covered by the restriction in violation of such restriction, except as otherwise provided in §164.522(a).
(d) Standard: Uses and disclosures of de-identified protected health information—(1) Uses and disclosures to create de-identified information. A covered entity may use protected health information to create information that is not individually identifiable health information or disclose protected health information only to a business associate for such purpose, whether or not the de-identified information is to be used by the covered entity.
(2) Uses and disclosures of de-identified information. Health information that meets the standard and implementation specifications for de-identification under §164.514(a) and (b) is considered not to be individually identifiable health information, i.e., de-identified. The requirements of this subpart do not apply to information that has been de-identified in accordance with the applicable requirements of §164.514, provided that:
(i) Disclosure of a code or other means of record identification designed to enable coded or otherwise de-identified information to be re-identified constitutes disclosure of protected health information; and
(ii) If de-identified information is re-identified, a covered entity may use or disclose such re-identified information only as permitted or required by this subpart.
(e)(1) Standard: Disclosures to business associates. (i) A covered entity may disclose protected health information to a business associate and may allow a business associate to create, receive, maintain, or transmit protected health information on its behalf, if the covered entity obtains satisfactory assurance that the business associate will appropriately safeguard the information. A covered entity is not required to obtain such satisfactory assurances from a business associate that is a subcontractor.
(ii) A business associate may disclose protected health information to a business associate that is a subcontractor and may allow the subcontractor to create, receive, maintain, or transmit protected health information on its behalf, if the business associate obtains satisfactory assurances, in accordance with §164.504(e)(1)(i), that the subcontractor will appropriately safeguard the information.
(2) Implementation specification: Documentation. The satisfactory assurances required by paragraph (e)(1) of this section must be documented through a written contract or other written agreement or arrangement with the business associate that meets the applicable requirements of §164.504(e).
(f) Standard: Deceased individuals. A covered entity must comply with the requirements of this subpart with respect to the protected health information of a deceased individual for a period of 50 years following the death of the individual.
(g)(1) Standard: Personal representatives. As specified in this paragraph, a covered entity must, except as provided in paragraphs (g)(3) and (g)(5) of this section, treat a personal representative as the individual for purposes of this subchapter.
(2) Implementation specification: Adults and emancipated minors. If under applicable law a person has authority to act on behalf of an individual who is an adult or an emancipated minor in making decisions related to health care, a covered entity must treat such person as a personal representative under this subchapter, with respect to protected health information relevant to such personal representation.
(3)(i) Implementation specification: Unemancipated minors. If under applicable law a parent, guardian, or other person acting in loco parentis has authority to act on behalf of an individual who is an unemancipated minor in making decisions related to health care, a covered entity must treat such person as a personal representative under this subchapter, with respect to protected health information relevant to such personal representation, except that such person may not be a personal representative of an unemancipated minor, and the minor has the authority to act as an individual, with respect to protected health information pertaining to a health care service, if:
(A) The minor consents to such health care service; no other consent to such health care service is required by law, regardless of whether the consent of another person has also been obtained; and the minor has not requested that such person be treated as the personal representative;
(B) The minor may lawfully obtain such health care service without the consent of a parent, guardian, or other person acting in loco parentis, and the minor, a court, or another person authorized by law consents to such health care service; or
(C) A parent, guardian, or other person acting in loco parentis assents to an agreement of confidentiality between a covered health care provider and the minor with respect to such health care service.
(ii) Notwithstanding the provisions of paragraph (g)(3)(i) of this section:
(A) If, and to the extent, permitted or required by an applicable provision of State or other law, including applicable case law, a covered entity may disclose, or provide access in accordance with §164.524 to, protected health information about an unemancipated minor to a parent, guardian, or other person acting in loco parentis;
(B) If, and to the extent, prohibited by an applicable provision of State or other law, including applicable case law, a covered entity may not disclose, or provide access in accordance with §164.524 to, protected health information about an unemancipated minor to a parent, guardian, or other person acting in loco parentis; and
(C) Where the parent, guardian, or other person acting in loco parentis, is not the personal representative under paragraphs (g)(3)(i)(A), (B), or (C) of this section and where there is no applicable access provision under State or other law, including case law, a covered entity may provide or deny access under §164.524 to a parent, guardian, or other person acting in loco parentis, if such action is consistent with State or other applicable law, provided that such decision must be made by a licensed health care professional, in the exercise of professional judgment.
(4) Implementation specification: Deceased individuals. If under applicable law an executor, administrator, or other person has authority to act on behalf of a deceased individual or of the individual's estate, a covered entity must treat such person as a personal representative under this subchapter, with respect to protected health information relevant to such personal representation.
(5) Implementation specification: Abuse, neglect, endangerment situations. Notwithstanding a State law or any requirement of this paragraph to the contrary, a covered entity may elect not to treat a person as the personal representative of an individual if:
(i) The covered entity has a reasonable belief that:
(A) The individual has been or may be subjected to domestic violence, abuse, or neglect by such person; or
(B) Treating such person as the personal representative could endanger the individual; and
(ii) The covered entity, in the exercise of professional judgment, decides that it is not in the best interest of the individual to treat the person as the individual's personal representative.
(h) Standard: Confidential communications. A covered health care provider or health plan must comply with the applicable requirements of §164.522(b) in communicating protected health information.
(i) Standard: Uses and disclosures consistent with notice. A covered entity that is required by §164.520 to have a notice may not use or disclose protected health information in a manner inconsistent with such notice. A covered entity that is required by §164.520(b)(1)(iii) to include a specific statement in its notice if it intends to engage in an activity listed in §164.520(b)(1)(iii)(A)-(C), may not use or disclose protected health information for such activities, unless the required statement is included in the notice.
(j) Standard: Disclosures by whistleblowers and workforce member crime victims—(1) Disclosures by whistleblowers.A covered entity is not considered to have violated the requirements of this subpart if a member of its workforce or a business associate discloses protected health information, provided that:
(i) The workforce member or business associate believes in good faith that the covered entity has engaged in conduct that is unlawful or otherwise violates professional or clinical standards, or that the care, services, or conditions provided by the covered entity potentially endangers one or more patients, workers, or the public; and
(ii) The disclosure is to:
(A) A health oversight agency or public health authority authorized by law to investigate or otherwise oversee the relevant conduct or conditions of the covered entity or to an appropriate health care accreditation organization for the purpose of reporting the allegation of failure to meet professional standards or misconduct by the covered entity; or
(B) An attorney retained by or on behalf of the workforce member or business associate for the purpose of determining the legal options of the workforce member or business associate with regard to the conduct described in paragraph (j)(1)(i) of this section.
(2) Disclosures by workforce members who are victims of a crime. A covered entity is not considered to have violated the requirements of this subpart if a member of its workforce who is the victim of a criminal act discloses protected health information to a law enforcement official, provided that:
(i) The protected health information disclosed is about the suspected perpetrator of the criminal act; and
(ii) The protected health information disclosed is limited to the information listed in §164.512(f)(2)(i).
[65 FR 82802, Dec. 28, 2000, as amended at 67 FR 53267, Aug. 14, 2002; 78 FR 5696, Jan. 25, 2013]
I am working on each portion of this and considering how I can add. I do not see any reason why this can't be applied to other areas of our information, like social, electricity, and beyond. I get that health care information has been granted a special place in our digital worlds, but I think we need to asking ourselves where else this might also apply.
It is tough to digest this in its full legal form. This is why I work to break them down as plain English building blocks that I can include alongside my research. If we can make turn this into an easy to read checklist that any data steward could use as best practices when crafting their own approach tot he disclosure of our information, maybe more companies will step up to the plate.
03 Aug 2016
I had heard about the Zika virus research that was going on at the University of Wisconsin listening to an NPR episode this last spring. I finally had the time to dig into the topic a little more, and learn more about where the research is at, and some of the software behind the sharing and collaboration around the research.
The urgency in getting the raw data and results of the research out to the wider scientific community caught my attention and the potential for applying API related approaches seems pretty huge. When it comes to mission-critical research that could impact thousands or millions of people, it seems like there should be a whole suite of open tooling that people can employ to ensure sharing and collaboration are real time and frictionless.
As I dug into the Zika virus research, I was happy to find the LabKey technology employed to publish the research. I do not know much about them yet, but I was happy to see the open source community solution, developer resources including a web API for integrating with research that is published using the platform. There are plenty of improvements I'd like to see added to the API and developer efforts, but it is a damn good start when it comes to making important scientific research much more shareable and collaborative.
I'll spend more time learning more about what LabKey currently offers, and then I'll work to establish some sort of next steps blueprint that would employ other modern API approaches to help ensure important research can be made more real-time, aggregated, interoperable, and shareable using technology like definitions, Webhooks, iPaaS, and other common areas of a modern API effort.
When it comes to research, scientists should have a wealth of open tooling and resources that make their work a collaborative and shareable process by default, but with as much control as they desire--something modern web API solutions excel at. I added LabKey to a new research area dedicated to science. I will spend more time going through space, and see what guides, blueprints, and other resources I can pull together to assist researchers in their API journey.
03 Aug 2016
I've been reading a number of stories about concerns within the Department of Interior about the usage of DJI drones as part of their UAV operations. Whether or not the federal agency actually issued a ban on using the drones is another story, regardless I wanted to extract this concept and apply to the wider world of Internet of Things, in which drones are definitely a poster child for.
In our race to connect devices to the Internet, at the consumer and industrial level, what considerations are being made regarding what data is collected, how this information is transmitted (via which networks), and ultimately where is data being stored (which country). Obviously, this is something the federal government is beginning to think about, but I'm guessing hasn't fully formed an approach to dealing with this problem consistently across agencies.
People love to dismiss individual consumer right to know about their data at this level, as nobody cares enough, but at the corporate, institutional, and agency level we better be taking things a little more seriously. If you are uploading your organizational intelligence unknowingly to servers out of the country, or in jurisdictions where it might be easily shared with your competitors or rival factions--you might want to be thinking about switching providers. ;-)
As platform providers how will we be certifying the chain of custody for the Internet-connected devices we are manufacturing, and within the applications and system they employ? As device consumers, how will we ensure that ALL the devices we are employing are in sync with our organizational mission and governance?
So many interesting questions being asked right now, as we race to wire up our physical world to the Internet...
03 Aug 2016
I wish every time you came across facts and statistics in any news story, blog post, report, and beyond, there would be an interactive experience like you get with the US Census Bureau's QuickFacts tool.
QuickFacts provides statistics for all states and counties, and for cities and towns with a population of 5,000 or more, and provides everything I think you need to make the case behind any facts and statistics you are presenting.
Using the tool you get a table, chart, map, data download, email results, and share via popular social networks as well as embeddable tooling. The QuickFacts experience should be a default for all storytelling around open data.
The Census team has clearly worked very hard to make a wealth of very complicated data much more accessible, usable, and easily understood through proven approaches to telling stories using open data.
If you are in the business of publishing open data and APIs and building analytics and visualization tools, follow the lead of the US Census Bureau, and develop easy to use API tooling like the QuickFacts solution.