The Open Source Community Tooling Built on Protocol Buffers

Moving on in my API specification research I am continuing beyond just gRPC and looking at how Protocol Buffers are used across the space. GitHub provides a pretty good way to pull a snapshot of the community around Protocol Buffers, by measuring the engagement with open source tooling being built on top of the specification. Here are the top open source tools I pulled from this round of research into Protocol Buffers, organized by what they deliver.

Specification

  • protobuf - (forks: 11358) (stars: 42046) (watchers: 42046) - protocol buffers - google's data interchange format

Serialization

  • flatbuffers - (forks: 2251) (stars: 14468) (watchers: 14468) - flatbuffers: memory efficient serialization library
  • protostuff - (forks: 248) (stars: 1419) (watchers: 1419) - java serialization library, proto compiler, code generator
  • openrtb - (forks: 155) (stars: 353) (watchers: 353) - openrtb model for java and other languages via protobuf; helper openrtb libraries for java including json serialization

Language Libraries

  • protobuf - (forks: 1260) (stars: 6714) (watchers: 6714) - go support for google's protocol buffers
  • protobuf.js - (forks: 1047) (stars: 6730) (watchers: 6730) - protocol buffers for javascript (& typescript).
  • protobuf net - (forks: 852) (stars: 2939) (watchers: 2939) - protocol buffers library for idiomatic .net
  • protoactor go - (forks: 339) (stars: 3192) (watchers: 3192) - proto actor - ultra fast distributed actors for go, c# and java/kotlin
  • rust protobuf - (forks: 214) (stars: 1290) (watchers: 1290) - rust implementation of google protocol buffers
  • prost - (forks: 132) (stars: 1061) (watchers: 1061) - prost! a protocol buffers implementation for the rust language
  • php protobuf - (forks: 264) (stars: 861) (watchers: 861) - php protobuf - google's protocol buffers for php
  • protobuf swift - (forks: 141) (stars: 913) (watchers: 913) - google protocolbuffers for apple swift
  • lua protobuf - (forks: 202) (stars: 800) (watchers: 800) - a lua module to work with google protobuf
  • brpc java - (forks: 166) (stars: 527) (watchers: 527) - java implementation for baidu rpc, multi-protocol & high performance rpc.
  • grpclib - (forks: 49) (stars: 525) (watchers: 525) - pure-python grpc implementation for asyncio
  • Protobuf PHP - (forks: 162) (stars: 459) (watchers: 459) - php implementation of google's protocol buffers with a protoc plugin compiler
  • pyrobuf - (forks: 57) (stars: 477) (watchers: 477) - a cython alternative to google's python protobuf library
  • protobuf objc - (forks: 310) (stars: 328) (watchers: 328) - protocol buffers for objective c
  • erlang_protobuffs - (forks: 184) (stars: 382) (watchers: 382) - an implementation of google's protocol buffers for erlang, based on ngerakines/erlang_protobuffs.
  • gpb - (forks: 124) (stars: 404) (watchers: 404) - a google protobuf implementation for erlang
  • protobuf - (forks: 94) (stars: 382) (watchers: 382) - a pure ruby implementation of google's protocol buffers
  • protobuf elixir - (forks: 68) (stars: 356) (watchers: 356) - a pureelixir implementation of google protobuf
  • protobuf go - (forks: 34) (stars: 331) (watchers: 331) - go support for google's protocol buffers
  • gruf - (forks: 38) (stars: 325) (watchers: 325) - grpc ruby framework
  • protobuf - (forks: 97) (stars: 238) (watchers: 238) - runtime library for dart protobufs
  • protoc jar maven plugin - (forks: 44) (stars: 157) (watchers: 157) - protocol buffers protobuf maven plugin - based on protoc-jar multi-platform executable protoc jar

Frameworks

  • twirp - (forks: 197) (stars: 4073) (watchers: 4073) - a simple rpc framework with protobuf service definitions
  • sofa rpc - (forks: 906) (stars: 3119) (watchers: 3119) - sofarpc is a high-performance, high-extensibility, production-level java rpc framework.
  • sofa pbrpc - (forks: 650) (stars: 1939) (watchers: 1939) - a light-weight rpc implement of google protobuf rpc framework.
  • ananas - (forks: 78) (stars: 301) (watchers: 301) - a c++11 rpc framework based on future and protobuf, with utility: timer, ssl, future/promise, log, coroutine, etc
  • Nebula - (forks: 92) (stars: 277) (watchers: 277) - nebula is a powerful framwork for building highly concurrent, distributed, and resilient message-driven applications for c++.
  • molecule - (forks: 7) (stars: 314) (watchers: 314) - molecule is a go library for parsing protobufs in an efficient and zero-allocation manner.
  • clojure protobuf - (forks: 70) (stars: 205) (watchers: 205) - google protocol buffers wrapper for clojure.
  • navi pbrpc - (forks: 61) (stars: 163) (watchers: 163) - a protobuf based high performance rpc framework leveraging full-duplexing and asynchronous io with netty

Command Line Interface

  • grpcurl - (forks: 147) (stars: 3205) (watchers: 3205) - like curl, but for grpc: command-line tool for interacting with grpc servers

Plugins

  • swift protobuf - (forks: 251) (stars: 3043) (watchers: 3043) - plugin and runtime library for using protobuf with swift
  • vim clang format - (forks: 84) (stars: 764) (watchers: 764) - vim plugin for clang-format, a formatter for c, c++, obj-c, java, javascript, typescript and protobuf.
  • protobuf adapter - (forks: 1) (stars: 186) (watchers: 186) - google protocol buffers adapter for casbin

Converters

  • rejoiner - (forks: 122) (stars: 2982) (watchers: 2982) - generates a unified graphql schema from grpc microservices and other protobuf sources
  • json2pb - (forks: 113) (stars: 212) (watchers: 212) - python/c++ implementation of json to/from protobuf convertor
  • protobuf to dict - (forks: 99) (stars: 176) (watchers: 176) - a small python library for creating dicts from protocol buffers. useful as an intermediate step before serialization (e.g. to json).

Registries

  • schema registry - (forks: 736) (stars: 1234) (watchers: 1234) - confluent schema registry for kafka

Transformations

  • rq - (forks: 45) (stars: 1553) (watchers: 1553) - record query - a tool for doing record analysis and transformation

Code Generators

  • protostuff - (forks: 248) (stars: 1419) (watchers: 1419) - java serialization library, proto compiler, code generator
  • protobuf - (forks: 95) (stars: 256) (watchers: 256) - c# code generator for reading and writing the protocol buffers format

Gaming

  • nano - (forks: 216) (stars: 1291) (watchers: 1291) - lightweight, facility, high performance golang based game server framework

3D Modeling & Printing

  • CuraEngine - (forks: 637) (stars: 1069) (watchers: 1069) - curaengine is a powerful, fast and robust engine for processing 3d models into 3d printing instruction for ultimaker and other gcode based 3d printers. it is part of the larger open source project called "cura".

Documentation

  • protoc gen doc - (forks: 242) (stars: 1188) (watchers: 1188) - documentation generator plugin for google protocol buffers

Toolchains

  • jprotobuf - (forks: 218) (stars: 609) (watchers: 609) - a very useful utility library for java programmer using google protobuf

Fuzzing

  • pbtk - (forks: 80) (stars: 655) (watchers: 655) - a toolset for reverse engineering and fuzzing protobuf-based apps

Generators

  • openapi2proto - (forks: 71) (stars: 594) (watchers: 594) - a tool for generating protobuf v3 schemas and grpc service definitions from openapi specifications
  • protobuf dynamic - (forks: 24) (stars: 165) (watchers: 165) - protocol buffers dynamic schema - create protobuf schemas programmatically
  • protofuzz - (forks: 27) (stars: 160) (watchers: 160) - google protocol buffers message generator
  • python betterproto - (forks: 18) (stars: 164) (watchers: 164) - clean, modern, python 3.7+ code generator & library for protobuf 3 and async grpc

Benchmarking

  • cpp serializers - (forks: 82) (stars: 484) (watchers: 484) - benchmark comparing various data serialization libraries (thrift, protobuf etc.) for c++

Command Line Interface

  • polyglot - (forks: 49) (stars: 474) (watchers: 474) - a universal grpc command line client

File Generator

  • bazel gazelle - (forks: 128) (stars: 424) (watchers: 424) - gazelle is a bazel build file generator for bazel projects. it natively supports go and protobuf, and it may be extended to support new languages and custom rule sets.

Lighting

  • ola - (forks: 151) (stars: 379) (watchers: 379) - the open lighting architecture - the travel adaptor for the lighting industry

Mapping

  • Leaflet.VectorGrid - (forks: 133) (stars: 373) (watchers: 373) - display gridded vector data (sliced geojson or protobuf vector tiles) in leaflet 1.0.0

Change Management

  • protolock - (forks: 20) (stars: 341) (watchers: 341) - protocol buffer companion tool. track your .proto files and prevent changes to messages and services which impact api compatibility.

Parsers

  • proto - (forks: 46) (stars: 308) (watchers: 308) - parser for google protocolbuffers definition

Binary

  • jackson dataformats binar - (forks: 67) (stars: 187) (watchers: 187) - uber-project for standard jackson binary format backends: avro, cbor, protobuf, smile

Validation

  • protoc gen lint - (forks: 14) (stars: 209) (watchers: 209) - a plug-in for google's protocol buffers (protobufs) compiler to lint .proto files for style violations.

It is tought to organize these and make sense of what each tool is doing. Protocol buffers get used in similar ways to OpenAPI, JSON Schema, and other formats, but how tooling gets build can be a little different. I'll keep going through these and tagging them, helping me make sense of what is happening. I am just looking to get a better handle on what is going on, and how Protocol Buffers are being put to work across the API space.