Thinking Differently When Approaching OpenAPI Diffs And Considering How To Layer Each Potential Change

I have a lot of OpenAPI definitions, covering about 2,000 separate entities. For each entity, I often have multiple OpenAPIs, and I am finding more all the time. One significant challenge I have in all of this centers around establishing a master “truth” OpenAPI, or series of definitive OpenAPIs for each entity. I can never be sure that I have a complete definition of any given API, so I want to keep vacuuming up any OpenAPI, Swagger, Postman, or other artifact I can, and compare it with the “truth” copy” I have on indexed. Perpetually layering the additions and changes I come across while scouring the Internet for signs of API life. This perpetual update of API definitions in my index isn’t easy, and any tool that I develop to assist me will be in need constant refinement and evolution to be able to make sense of the API fragments I’m finding across the web.

There are many nuances of API design, as well as the nuances of how the OpenAPI specification is applied when quantifying the design of an API, making the process of doing a “diff” between two OpenAPI definitions very challenging. Rendering common “diff” tools baked into GitHub, and other solutions ineffective when it comes to understanding the differences between two API definitions that may represent a single API. These are some of the things I’m considering as I’m crafting my own OpenAPI “diff” tooling:

  • Host - How the host is stored, defined, and applied across sandbox, production, and other implementations injects challenges.
  • Base URL - How OpenAPI define their base url versus their host will immediately cause problems in how diffs are established.
  • Path - Adding even more instability, many paths will often conflict with host and base URL, providing different fragments that show as differences.
  • Verbs - Next I take account of the verbs available for any path, understanding what the differences are in methods applied.
  • Summary - Summaries are difficult to diff, and almost always have to be evaluated and weighted by a human being.
  • Description - Descriptions are difficult to diff, and almost always have to be evaluated and weighted by a human being.
  • Operation ID - These are usually autogenerated by tooling, and rarely reflect a provider defined standard, making them worthless in “diff”.
  • Query Properties - Evaluating query parameters individually is essential to a granular level diff between OpenAPI definitions.
  • Path Properties - Evaluating path parameters individually is essential to a granular level diff between OpenAPI definitions.
  • Headers - Evaluating headers individually is essential to a granular level diff between OpenAPI definitions.
  • Tags - Most providers do not tag their APIs, and they are often not included, and rarely provide much value when applying a “diff”.
  • **Request Bodies - Request bodies provide a significant amount of friction for diffs depending on the complexity and design of an API.
  • Responses - Responses often provide an incomplete view of an API, and rarely are robust enough to impact the “diff” view.
  • Status Codes - Status codes should be evaluated on an individual basis, providing a variety of ways to articulate these statuses.
  • Content Types - Content types these days are often application/json, but do provide some opportunities to define unique characteristics.
  • Schema Objects - Schema is often not defined, and rarely used as part of a diff unless OpenAPIs are generated from log, HAR, and other files.
  • Schema Properties - Schema properties are rarely present in OpenAPIs, making them not something that comes up on the “diff” radar.
  • Security Definitions - Security definitions are the holy grail of automating API indexing, but are rarely present in OpenAPI, and only in Postman Collections.
  • References - The use of $ref, or absence of $ref and doing everything inline poses massive challenges to coherently considering “diff” results.
  • Scope - The size of the OpenAPI snippet being applied as part of a “diff” helps narrow what needs to be considered by a human or machine.

This reflects the immediate concerns I have approaching the development of a custom “diff” tool for OpenAPI. First I am just trying to establish a strategy for stripping back the layers of OpenAPI definitions, and established a sort of layered user interface for me to manually accept or reject changes to an OpenAPI. An interface that will also allow me to define a sort of rules vocabulary for increasingly automating the decision making process. I’d love it if eventually the diff tool would show me just a single diff, present me with the change it thinks I should make, and allow me to just agree and move to the next “diff”. I have a lot of work to get things to this point.

Like API search, I feel like API diff is something I have to reduce to its basics, and then fumble my way towards finding an acceptable solution. I don’t feel there is a single “diff” tool for JSON or YAML that will have the eye that I demand for analyzing, presenting, and either manually or automatically merging a diff. Like the other layers of my API search engine, diff is something I need to think through, iterate upon, and repeat until I come up with something that helps me merge “diffs” efficiently across thousands of APIs, and hopefully eventually automates and abstract away the most common differences between the APIs that I am spidering and indexing. Like every other area it is something I’m only working on when I have time, but something I will eventually come out the other end with a usable OpenAPI diff tool, that can help me make sense of all the API definitions I’m bombarded with on a daily basis.