Continuing My Struggle For Reciprocity As ETL Evolves Into The Cloud As iPaaS

Early on in 2013, I started a research project to keep an eye on a specific type of API driven service provider, like IFTTT and Zapier, who were enabling individuals and businesses to move data around in the cloud. This new wave of startups was moving from what we traditionally called ETL in the enterprise, which was about extracting, transforming, and loading data between various systems, into the cloud era. 

I'm an old enterprise database guy, and ETL has been an essential tool in my toolbox for quite some time--according to Wikipedia ETL is:

Extract, Transform and Load (ETL) refers to a process in database usage and especially in data warehousing that performs: Data extraction – extracts data from homogeneous or heterogeneous data sources.

The IT teams that I have historically been a part of have employed ETL to make sure data was where it was needed within the enterprise. As IT began its evolution to the web, the need to migrate data between systems outside the firewall increased. We were increasingly extracting, transforming, and loading data via FTP, web services, and web APIs outside the firewall, and even migrating data between systems that exclusively existed in the cloud.

Increasingly the data we were migrating was using web APIs and began employing a more granular approach to how authentication occurred for each "extract and load"--oAuth. With the growth of shadow IT, and the adoption of software as a service (SaaS) solutions, individuals, and businesses were needing to move documents, media, and other vital data or content between the cloud services which we were increasingly depending on.

APIs were enabling savvy tech users to migrate and sync their data between systems, and startups like IFTTT and Zapier saw the opportunity and jumped in with their new offerings. I do not talk about IFTTT, as they choose to not pay all of this forward by offering an API, and also opt to ignore the transparency that APIs bring to the table--so I will simply refer to Zapier as my example of this new breed of service provider. ;-) 

In addition to evolving beyond FTP and ODBC as the primary channels, and being about the migration of information in the cloud, the other significant characteristic that stood out for me, is this new approach was also paying attention to the individual needs of stakeholders, owners, and the people who the migration of information was important to. This information was also increasingly being migrated between multiple systems and adhering to the terms of service, as well as the privacy of each party involved.

When I launched my research back in 2013, I called it reciprocity, which the dictionary defined as:

  • the quality or state of being reciprocal : mutual dependence, action, or influence
  • a mutual exchange of privileges; specifically : a recognition by one of two countries or institutions of the validity of licenses or privileges granted by the other

When I looked in the thesaurus, reciprocity also had a definition of "interchange" with synonyms of cooperation, exchange, mutuality and reciprocation. Reciprocity is also a synonym of connection with a definition of “person who aids another in achieving goal”. With synonyms being an acquaintance, agent, ally, associate, association, contact, friend, go-between, intermediary, kin, kindred, kinship, mentor, messenger, network, reciprocity, relation, relative, and sponsor.

All of these terms apply to what I was seeing unfold with this new generation of ETL providers. ETL was moving into the clouds, and out from behind the firewall, using the open web, cloud platforms and APIs, and now we have to rethink ETL, and make it accessible to the masses--putting it within reach of the everyday problem owners.

After three years, as I've seen this area continue to grow, but the growth in my website traffic to this research was not in alignment with what I saw in other areas. Over time I realized the area had been dubbed integration platform as a service (iPaaS), and while I have resisted using this term for a while, as I was wanting to emphasize the human aspect of this, I am now giving in. Reciprocity was the only term I have ever tried to push on the API community and have to admit it will be the last term or phrase I ever try to brand in this way.

Sadly iPaaS has become the leading acronym to talk about this evolution, and because IT vendors and analysts couldn't give a shit about the users who's information is being moved around, they have defined it simply as:

Integration Platform as a Service (iPaaS) is a suite of cloud services enabling development, execution and governance of integration flow connecting any combination of on premises and cloud-based processes, services, applications and data within the individual or across multiple organizations.

In classic IT fashion, no emphasis has been placed on the humans in this equation. In the early days of web APIs, startups had focused on what the people who were using their solution needed. Then with each wave of VC investment into the space, with enterprise vendors shifting their focus to this new world, and the industry analyst pundits realizing the value that lies in this new era, we are going back to the old ways of thinking about IT--one that rarely ever focuses on the human aspect of the bits and bytes we are shuffling around.

I am losing this same fight in almost every area of the API space that I keep an eye on. I'm not naive to think I can cut through the noise of the space, and truly take on the IT and developer class's obsessive belief in technology, and the blindness that comes from a focus on the money, but I do believe I can at least influence some of the conversations. This is why I'll keep trying to make sure there is reciprocity across the API space, and iPaaS pays attention to the human aspect of integration, migration, and keeping our increasingly online world in sync.