I was being asked for more information about how I profile APIs, and deal with the many differences I come across. It isn’t easy navigating the differences between different APIs, and come out with a standard API definition (OpenAPI or Postman collection) that you can use across different stops along the API life cycle. I’m pretty agile and flexible in how I approach profiling different APIs, with a variety of tools and tricks I use to vacuum up as much details as I possibly can with as little manual labor as I possibly can. The example for profiling that was thrown at me was the TypeForm API, which is a pretty sophisticated API, but will still need some massaging to create an acceptable set of API definitions.
First thing I do is search for an OpenAPI definition, hopefully published to GitHub or prominently linked off their documentation, but I will settle having to sniff out from behind an APIs documentation. TypeForm doesn’t have an OpenAPI or Swagger available (from what we can tell). Next, I go looking for a Postman collection. Boom!! Typeform has a Postman collection. The question now is why hasn’t Typeform published their Postman collection to the Postman Network? I will Tweet at them. Ok, now I have a machine readable definition for the Typeform API that I can import into my API monitoring system—which is just an API that I use in Postman to import a Postman collection (head explodes).
My Postman collection import API grabs as many elements from the Postman collection definition as it can, normalizing the paths, parameters, and other details for an API. I am always adding to what my API is capable of, but it does a pretty good job of giving me what I need to begin to profile the surface area of an API.
Now I have all of the paths imported into my monitoring system. However, I am still at the mercy of how much work an API provider has invested into their Postman collection. Depending on the provider and the API definition format being important the differences can vary widely, so I have been developing different APIs that help me clean up and deal with the common types of data that is missing or needs work.
I am always working to flesh out as many of the common details of an API as I can, but always stop short of doing too much manual labor when it comes to crafting a complete enough API definition. My goal is to just get as much cleaned up and normalized as I can, and leave what I can do left undone. When it comes to the Typeform API, they didn’t provide descriptions for their API paths. Descriptions exist on the documentation, but would need to be scraped, or manually added to either the Postman collection or OpenAPI. I just add a GitHub issue for the todo item, and move on. Ideally Typeform does this work, or someone from the API community submits as a pull request on the API definitions I have published to GitHub. Beyond descriptions, I am able to clean up paths, generate an operation id which is useful in SDK and code generation, and apply tags to each path based upon the words in the path.
Then I go ahead and publish both an OpenAPI and Postman collection to the GitHub repository I have for Typeform. This allows me to independently track each API provider, separately manage issues, and engage with the community, and hopefully Typeform in a central place.The API definitions are far from complete, but they provide me with a good start. I can queue up some other tasks that I need accomplished, pushing the API definitions further along when i have time. For my needs, having the Typeform API paths and parameters using OpenAPI and Postman collections, then the Typeform API operations defined as an APIs.json is a damn good start. I’ll make time in the future to continue fleshing out additional details, or hopefully the community and Typeform steps in and does it for me.
The next major step in profiling the Typeform API is producing JSON schema for the request bodies and responses. Because there wasn’t an OpenAPI available, there is no JSON schema defined. It isn’t always a guarantee you’ll have these defined when importing OpenAPI, but they aren’t part of the Postman collection. For my needs when it comes to API discovery, the schema is a nice to have, but not required, so I am happy to leave on my todo list. When it comes to producing JSON schema for APIs I am profiling I will usually approach in two different ways:
- Scrape From Documentation - If there are examples present as part of API documentation I can usually scrape, satisfying two needs: 1) adding examples to OpenAPI and Postman collections, and 2) generate standalone JSON Schema, and add to the OpenAPI definitions .
- Run In Postman - Sign up for a Typeform key, and actually make API calls within Postman, then I can use a solution like Optic (https://www.useoptic.com/) to autogenerate the OpenAPI (and JSON Schema) for each API call I make, then run the diff against my existing OpenAPI.
For now, I have added an issue to the GitHub repository for my Typeform work. https://github.com/api-evangelist/typeform/issues/2 I am happy to have the Typeform API in my index, and published as part my research. I can continue fleshing out my profile for Typform, and publish as part of my API Evangelist Network, at a later date in the future. Or maybe someone comes along and adds the JSON schema to the GitHub repository—doing the work for me. ;-) This is why I am publishing all of my API definitions to GitHub. So that we can all stop creating our own API definitions, and we can start sharing our work. I am going to better work at attribution and keeping in sync with existing efforts like APIs.guru, helping build on the existing hard work other folks have already done. Hopefully folks can already build on top of the work I have done as I continue to publish what I have in my database to GitHub as part of my API Evangelist Network research.
My goal here is to keep layering on the API definitions I’ve come across and producing a “complete enough” API definition for the APIs I’m profiling and including in my research. It is also to continue convincing API providers to take ownership over their own API definitions. I’m happy to accept pull requests from API providers, give them administrator access to my repositories, or simply keep in sync from an authoritative set of OpenAPI and Postman collections. However, I feel it is pretty important to do this out in the open, encourage other API service providers and community members to also contribute and help define the APIs available across the public API commons. Helping community-source the discovery, profiling, and deprecation of APIs that are available on the web, and going beyond what each API provider is capable of, ensuring there is human and machine readable resources available for each industry that is being impacted by APIs today.
