Concern Around Working With Many Github Repositories
16 Jul 2018
I’m regularly fascinated with API development teams I work with expressing their concern with working with many Github repositories. With all of the complexity I watch teams embrace when it comes to frameworks, scaffolding, continuous integration, deployment, and orchestration solutions, I’m lost on why many Github repositories suddenly become such a challenge. A Github organization is a folder, and a Github repository is a folder, that you can check out locally, on your server, or work with via an API, or Git. It is a distributed file store, that you can orchestrate with programmatically, and can be as logical or illogical as you design it to be.
I work really hard to keep technical complexity to a minimum in my world, but this means limiting unnecessary vendor lock-in, and tech just because it is the latest trend. For me, Git is just a distributed file system, with version control built in, with Github providing a nice API and network effect layer that makes it compelling. Referencing a Github folder is just a matter of using its org and repo name, checking it out, and working with the standardized layout of information I have published there. Allowing me to work with, and orchestrate thousands of separate folders (repos), across almost 50 organizations (folders), in a consistent way, as a one person team. Something that has taken me about four years to setup, and fine tune, but has become essential to what I do on a daily basis.
I’m not saying working with hundreds or thousands of individual Github repositories can’t be complex, or doesn’t take a significant amount of work. I’m just saying I’m intrigued by how technologists who manage large systems, adopt complex frameworks for delivering simple web solutions, and regularly make other complex technological investments, draw the line here. I see Github as a robust file store, with two doorways, 1) Git, and 2) API. I have a standardized structure to what I store on Github, something that is similar to my Amazon S3 store, or my backend of databases, and is something that takes discipline to maintain and keep from being unwieldy, but if I do the hard work it is possible. I’m guessing folks who see Github as a lot of work are seeing it through a web or desktop UI lens, and haven’t stopped to think about through an API or CLI lens–I find my AWS, Google, and other platforms to be complicated if I only look at them through the UI.
I enjoy being able to checkout and work with the relevant repositories in my world, and automate my backend and front end systems to automate with the same repositories. My blog schedules, checks out, and publishes posts across 250 repositories. My curation system pulls from my Feedly every day, organizing what I’ve bookmarked and tagged across almost 500 repositories. My API system updates and publishes APIs.json, OpenAPI, and Postman collections across almost 4,000 separate repositories, as I discovery, profile, and make sense of the API landscape. I don’t see this as complexity, I see it as pretty simple, Git, Jekyll, HTML, CSS, JS, JSON, and YAML driven goodness. Something I can easily migrate off of Github if I wanted, and run on AWS, Google, Azure, or my own server infrastructure. For now I’m enjoying the network effect provided by Github, and the power of their API when it comes to more granular changes, and tuning into valuable signals that are available via the social platform.
The conversations I’m having around the complexities of managing many Github repositories are a reminder for me of how technological complexity is relative. Some will see opportunity, while others see complexity, and vice versa. If you understand something, there is a lesser chance you will see complexity–you’ve made the investment already. For many, Github is a burden. I see it as liberating, and something to be orchestrated, providing me with some of the most significant signals I can find online–rivaling Twitter when it comes to driving my work. I can see managing Github through the web or desktop interface being pretty cumbersome, but once you’ve elevated beyond these tools, and work with repositories using the API and CLI, taking full advantage of the Git in Github, the landscape changes, and become less complex, and a more empowering experience. Something I don’t think everyone will fully realize, and be able to get beyond their view of Github as complex, difficult, and challenging. And, thats ok.