Doing A Diff Between Available Web, Mobile and Public APIs
22 Aug 2019
I spend a lot of time running web and mobile applications through a proxy to reverse engineer their APIs. I generally use Charles Proxy for routing my desktop, web, and mobile traffic through, which then automatically saves a JSON dump of sessions every five minutes, and syncs with Dropbox via my shared folders. From there I have a schedule service that will look in the shared Dropbox folder every hour, sift through the Charles Proxy JSON dump, and look for JSON, XML, CSV, and other common machine readable formats—which are then converted into OpenAPI definitions. Allowing me to reverse engineer desktop, web, and mobile applications as I use them, and map the API surface area for these targeted applications.
Recently I started playing with doing the same thing as part of my use of Postman. You can use the built-in proxy in the Postman native apps or use the Interceptor extension for the Postman app. Postman walks you through how to configure your laptop, mobile, and Postman application, and ultimately capture HTTP requests and save them to history or as a Postman Collection. Doing essentially the same thing I’m doing, but doing it with the Postman application, and leveraging collections instead of OpenAPI. I’d say there are pros and cons to both approaches, but Postman gives me the ability to manage environments, workspaces, and other essential concepts that would help take my API profiling work to the next level.
One of the benefits of working with Postman collections is you get all the benefits of using the Postman app. Things like the built-in proxy for capturing traffic, but also the history, generate, fork, merge, and diff collections. My work profiling APIs is all about reverse engineering desktop, web, and mobile applications, as well as quickly translating API documentation in machine readable API definitions, like OpenAPI and Postman. When profiling an API, most of the time I have the API documentation open in one browser window, and my Postman application open—-operating them both in split screen. I’ll just take the paths and parameters from the documentation and enter them into my Postman application, and fidget with things until I make a successful API call. Then I repeat for every single path until I have a complete Postman Collection for a single APIs. Merging my more automated proxy profiling with my manual API documentation profiling makes a lot of sense—-then I can manage Postman Collections as part of Postman workspaces, complete with Postman environment definitions.
Postman gives me forking, merging, diff, and conflict resolution for Postman Collections. This is essential for mapping out the API landscape for a single API provider over time, but I find it is also useful for doing a diff between available web, mobile, and public APIs. Helping me paint a complete picture of a platform’s APIs by reverse engineering their APIs behind their web and mobile, then taking their own Postman Collection, OpenAPI definition, or manually create a Postman Collection from their API documentation, allowing me to create a diff between each of their channels. Painting a complete picture of a provider’s public API infrastructure, but also be showing the prioritization of resources when it comes to the different channels. I find this to be a pretty compelling way to paint a picture of not just tech companies, but almost any company in 2019 who has a web, mobile, and public API infrastructure. Wait, does having an API make you a tech company? IDK. We’ll answer that question another time.
I enjoy learning new ways to accomplish existing tasks I’m working on daily. Sometimes it helps me to shift my view to a new service or tool so that I can gain a few new bells and whistles, but also the process forces me to look at things differently. This shift can push me into entirely new territory when it comes to understanding more about the technology, business, or politics of operating APIs. There are a lot of games played in the world of APIs, and even more being played in the shadows behind web and mobile applications. I need as much help as I can to help me shine a light into these dark corners of the web, which reside directly behind the desktop, web, browser, mobile, and device applications we are using in our personal and professional lives. Having a diff of the various APIs in use behind these channels goes a long way towards helping us understand the motivations of the companies behind these applications, pushing us be more aware of who the good or bad actors potentially are.