The reasons for this advice are valid: HTTP/JSON is trivial to set up, inspect in DevTools, cache, proxy, and hit from a browser. gRPC and Protobuf do give you schemas, generated clients, compact and fast binary payloads, and streaming.

Why can’t we have the best of both?

You can get the best of both worlds. With ConnectRPC.

(Disclosure: I work at Buf, which maintains ConnectRPC, but I absolutely had all of these opinions beforehand.)

Browser support

gRPC relies on HTTP/2 features that browser APIs refuse to expose to JavaScript, including trailers. So gRPC can’t be used with web browsers.

gRPC-Web tried to bridge the gap by putting trailers as another frame type inside the body of the HTTP responses. gRPC-Web ultimately failed as a project. Its own roadmap says it can no longer deliver new modern solutions, will not add new features, and recommends gRPC-Gateway instead. I cover the reasons in detail in gRPC-Web Failed the Web.

Transcoding is the other common workaround that enables you to have gRPC in the backend and an HTTP/JSON API in the frontend leveraging tools like gRPC-Gateway and Envoy. You annotate Protobuf files with google.api.http rules, and the proxy translates inbound requests to gRPC and the outgoing responses back to HTTP/JSON. This works fine, but now the browser-facing API goes through another representation. The Protobuf schema defines the gRPC service, google.api.http defines how that service maps onto HTTP, and tooling may then generate an OpenAPI description and another client from that.

• An extra required hop between the browser and the gRPC backend.
• Another HTTP mapping to define and maintain alongside the RPC service.
• Browser clients no longer talk directly to the RPC service described by the Protobuf schema.

The Connect response

Connect sidesteps the proxy layer entirely by having the application server handle three protocols natively: standard gRPC, gRPC-Web, and Connect’s own HTTP-based protocol. Existing gRPC clients can keep talking standard gRPC, while browsers talk directly to the application server over plain HTTP with JSON or binary Protobuf.

Connect-Web also supports server-streaming RPCs right in the browser using the Fetch API. Outside the browser, the protocol handles client and bidirectional streaming over HTTP/2.

Ad-hoc requests

A dead simple requirement for any API is the “send a coworker a curl command” test. This should be trivial. Standard gRPC fails this spectacularly. gRPC-Web also fails it.

Generated SDKs are actually pretty awesome, but they can get in the way when you want to quickly reproduce an issue or verify a change from the terminal. With gRPC, you have to install grpcurl, verify reflection is enabled in that environment, or manually pass proto files before calling an API dynamically.

The Connect way

A unary Connect request is just an HTTP POST. For quick debugging, you can use ProtoJSON:

Content-Type: application/json

That makes terminal testing straightforward:

curl -X POST \
  https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say \
  -H "Connect-Protocol-Version: 1" \
  -H "Content-Type: application/json" \
  -d '{"sentence":"Hello"}'

No special CLIs, no need for server reflection. The server handles JSON deserialization according to the Protobuf schema, and the response prints clean, readable ProtoJSON. This example actually works, by the way. You can copy/paste it into your terminal and try.

Errors behave predictably too. Like many RPC systems, standard gRPC returns an HTTP 200 OK for every response and puts the actual failures away in grpc-status trailers, which means edge proxies and access logs often report failed calls as perfectly healthy traffic. Connect uses standard HTTP statuses for unary calls: NotFound maps to 404, InvalidArgument maps to 400, and existing observability tools see unary call failures without custom parsers.

The same pragmatism applies to HTTP verbs. gRPC mandates POST for everything, even simple lookups or queries. Connect supports GET requests for methods flagged as side-effect free:

rpc Say(SayRequest) returns (SayResponse) {
  option idempotency_level = NO_SIDE_EFFECTS;
}

Connect clients can call these endpoints over GET by placing serialized parameters into the query string:

curl "https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say?message=%7B%22sentence%22%3A%22Hello%22%7D&encoding=json&connect=v1"

The payload lives in the URL (as JSON or base64-encoded Protobuf), which means standard web caching actually works. Return a standard Cache-Control header, and any standard CDN or browser cache will respect it.

You also aren’t locked into Protobuf on the client side. Tools like protoc-gen-connect-openapi, which I created, can generate an OpenAPI spec from your proto definitions. If another team needs OpenAPI specs for their tooling, you can export them without rebuilding your backend architecture around them.

Use ConnectRPC

The usual REST vs gRPC discourse assumes you have to choose between ordinary HTTP semantics and a schema-driven RPC API. Connect doesn’t make you choose.

Instead of asking whether to use REST or gRPC, why not ConnectRPC?

That matters for a protocol like ConnectRPC. Connect already tries to map RPCs onto ordinary HTTP instead of fighting the grain of the web. It supports HTTP GET for side-effect-free unary RPCs, which is great for caching, but GET forces structured request payloads into query parameters.

ConnectRPC should support QUERY as the body-carrying counterpart to GET for those same side-effect-free unary calls. Caching support across the broader web can mature over time, but the protocol shape is already useful today: structured request data belongs in the request body, not squeezed into a URL.

Why Connect Got It Right

One of the best design decisions in the Connect protocol is that it leans into standard HTTP semantics. Unlike traditional gRPC, which demands HTTP/2 and relies heavily on trailing headers, Connect maps naturally onto the HTTP infrastructure most teams already run.

That pays off operationally:

• Meaningful HTTP Status Codes: Errors map directly to standard HTTP statuses, allowing metrics to work without a specialized gRPC proxy.
• Standard Compression: Traffic relies on standard Content-Encoding headers (like gzip or brotli) already built into your infrastructure.
• No Trailers Required: Requests pass cleanly through standard load balancers, firewalls, and HTTP/1.1 proxies without requiring end-to-end HTTP/2.
• Native Ecosystem Integration: The protocol plugs directly into Go’s standard net/http stack, allowing you to reuse standard middleware, multiplexers, and observability tools.

Building on this foundation, Connect allows any unary RPC marked as side-effect free (NO_SIDE_EFFECTS in Protobuf) to be invoked via HTTP GET, unlocking caching at the CDN or proxy layer.

The Problem with GET and Query Parameters

To make GET work with complex schema definitions, the protocol has to perform significant gymnastics. Because GET request bodies have no defined semantics and are routinely ignored or rejected by intermediate proxies, Connect is forced to cram structured payloads into the URL.

For a simple JSON request, the client must serialize the payload, URL-encode it, and append it as a query parameter:

GET /connectrpc.greet.v1.GreetService/Greet?connect=v1&encoding=json&message=%7B%22name%22%3A%22Buf%22%7D HTTP/1.1
Host: demo.connectrpc.com

If you use binary Protobuf or compression, the overhead increases further, requiring base64 encoding along with additional control parameters. Shoving these complex payloads into URLs creates immediate practical problems for production systems:

• Bloated URLs: Complex requests easily hit maximum URL length limits enforced by load balancers, reverse proxies, and older browsers.
• Leaky Logs: Query parameters show up in plain text in standard Nginx or Apache access logs, WAF dashboards, and observability tools. If a request contains sensitive filter criteria, teams are forced to write custom masking rules to scrub their logs.
• Encoding Friction: Maintaining a separate serialization path just for GET requests introduces branching logic into the codebase. Clients and servers must implement special handling to treat this specific verb entirely differently than the rest of the API surface.

Enter HTTP QUERY

QUERY gives HTTP the method shape this use case has been missing. Semantically, it is defined as a safe, idempotent method. Mechanically, it operates like a POST, allowing a standard request body.

If ConnectRPC adopts QUERY, the entire query parameter encoding scheme can be dropped. A QUERY request looks exactly like a POST request on the wire, keeping the payload in the HTTP body natively encoded as application/json or application/proto.

Here is how a proposed JSON-based QUERY wire request looks:

QUERY /connectrpc.greet.v1.GreetService/Greet HTTP/1.1
Host: demo.connectrpc.com
Content-Type: application/json
Connect-Protocol-Version: 1

{"name":"Buf"}

And for binary Protobuf:

QUERY /connectrpc.greet.v1.GreetService/Greet HTTP/1.1
Host: demo.connectrpc.com
Content-Type: application/proto
Connect-Protocol-Version: 1

<binary protobuf>

This is the core protocol win: QUERY lets Connect reuse the normal unary POST body format instead of maintaining a specialized GET query-encoding path.

Caching Demands QUERY-Aware Infrastructure

From a network caching perspective, QUERY is not simply “GET with a body.” RFC 10008 specifies that for a QUERY response to be cached, the cache key must incorporate the request body content alongside related metadata.

Most existing HTTP caching infrastructure is built strictly around request metadata such as the method, scheme, host, path, query string, and selected headers. Supporting QUERY requires intermediate proxies, gateways, and CDNs to inspect and hash request bodies. Until major CDNs document first-class support for body-aware cache keys, QUERY caching should be treated as experimental outside infrastructure you control directly.

The Browser Story

Browser support is not a reason to avoid QUERY; it is a reason to implement it deliberately.

Because QUERY is not a CORS-safelisted method, cross-origin browser clients require the server or gateway to allow it in preflight responses. This is a standard deployment requirement for modern APIs. Many Connect-Web deployments already require custom CORS configuration for protocol headers, content types, and credentials.

Adopting QUERY gives web clients a clean way to express search forms, filtered list views, reporting queries, and batch reads without resorting to URL hacks.

A Pragmatic Rollout Strategy

We cannot flip a switch and expect a new HTTP verb to work across the public internet immediately. The web is built on middleboxes, load balancers, strict firewalls, and managed WAFs that are inherently suspicious of unfamiliar traffic and will likely reject QUERY requests as malformed for some time.

However, many modern backend servers accept arbitrary verbs without complaint. You can verify how your current stack handles unfamiliar methods right now using httpbin.io:

$ curl -X QUERY [https://httpbin.io/status/200](https://httpbin.io/status/200) -w "%{http_code}"
200

To prove the server simply accepts the string token, you can test an arbitrary string:

$ curl -X YEET [https://httpbin.io/status/200](https://httpbin.io/status/200) -w "%{http_code}"
200

While this does not prove full RFC 10008 compliance, it confirms that the initial deployment barrier at the server application layer is low. The harder problem is teaching intermediaries, caches, and client libraries what QUERY actually means.

The most effective early deployments will target paths where engineering teams control the entire network hop: internal service-to-service traffic and browser-facing APIs behind configurable API gateways. Protocols like Connect are the ideal starting point because they can expose QUERY as an opt-in transport mechanism while the broader networking ecosystem catches up.

What QUERY Should Not Replace Yet

While QUERY is the cleaner protocol shape, a practical rollout requires clear boundaries:

• GET remains useful for small, URL-friendly requests: For simple payloads that fit naturally in a URI, GET continues to offer mature browser integration, native CDN caching, and effortless manual debugging.
• POST remains the compatibility fallback: When requests have side effects, or when traffic must route through uncooperative legacy intermediaries, POST remains the universal standard.
• QUERY starts as opt-in: Support should initially be opt-in for unary RPCs explicitly marked NO_SIDE_EFFECTS. This option should be exposed to both server-side and web clients, backed by clear deployment guidance for CORS, gateways, and cache behavior.

The QUERY method solves a real, persistent networking problem. It is time to put it to work.

When you’re building with gRPC or ConnectRPC, this problem gets both easier and harder. It’s easier because you have a strict schema (the Protobuf contract) to guide you. It’s harder because writing binary payloads or mock servers that conform to those schemas by hand is tedious.

In FauxRPC, I built a tool to generate fake data from Protobuf schemas dynamically. But I wanted to go further. I wanted to make mocks a natural byproduct of running your development environment. That is why I introduced Proxy Mode and Auto-Recording.

By placing FauxRPC in front of a real upstream service, it acts as a smart proxy: intercepting traffic, forwarding it to the upstream server, and writing out reusable mock stubs to disk. It even generates intelligent matching rules automatically.

Here is how it works under the hood.

The Auto-Recording Workflow

Rather than thinking about mocks as a separate coding chore, FauxRPC integrates mock generation directly into your existing development flow:

1. Start FauxRPC in proxy mode in front of a staging or local backend server.
2. Interact with your application normally in the browser or via API clients.
3. Capture traffic automatically as FauxRPC records every request and response into the stubs/ directory.
4. Commit the generated stubs to Git along with your code changes.
5. Run your CI test suite against FauxRPC using those recorded stubs, ensuring fast, offline, and reproducible test runs.

Zero-Configuration Reflection

Normally, running a mock server requires you to supply the schema files. You have to pass paths to your .proto files or compiled descriptor sets (.binpb). That’s fine for a CI environment, but during local development, dragging files around and keeping them in sync is a pain.

If you start FauxRPC in proxy mode without specifying a schema:

fauxrpc run --proxy-to=localhost:8080 --record-dir=stubs/

FauxRPC uses gRPC Server Reflection to discover the schema dynamically from the upstream server on startup.

Behind the scenes, FauxRPC performs a few key steps during this schema discovery phase:

1. It connects to the upstream and queries the reflection service using ListServices to list all endpoints.
2. For each service, it fetches the file descriptors using FileContainingSymbol.
3. It recursively collects the raw file descriptors (FileDescriptorProto), deduplicating common dependencies (like google/protobuf/timestamp.proto).
4. It compiles the gathered descriptors into a FileDescriptorSet and registers them in FauxRPC’s registry.

This means FauxRPC is ready to handle, translate, and mock any method defined on the upstream server with absolutely zero configuration.

Because the schema is discovered directly from the running service, the proxy automatically stays in sync with upstream changes. No descriptor files need to be exported, committed, or manually refreshed—as your API evolves, FauxRPC adapts instantly.

The Multi-Protocol Proxy Engine

FauxRPC is designed to work as a multi-protocol translator. A frontend might communicate using ConnectRPC (over JSON), while the upstream service is a pure gRPC service using binary Protobuf.

To achieve this, FauxRPC sets up a ConnectRPC client using a custom dynamicProtoCodec for proxying. This codec uses Go’s protobuf reflection (dynamicpb.Message) to encode and decode payloads on the fly using the schemas retrieved during the reflection phase.

• Unary calls: The proxy reads the request, forwards it using the client, intercepts the response, and writes it back.
• Streaming calls (client, server, and bidi): To handle streams, FauxRPC uses FrameTracker objects. The tracker forwards frames in real time so there is no added latency, but it keeps a copy of the sequence in memory for logging and recording.
• Metadata and Headers: When forwarding headers via copyHeaders, FauxRPC filters out protocol-level headers (like content-type, grpc-status, and Connect-specific protocol headers) to let the underlying transport layer handle them cleanly. It also automatically masks sensitive headers (e.g. Authorization, Cookie, X-API-Key) with ***** to avoid leaking production keys or user credentials into logs or stubs.

Graceful Fallbacks for Parallel Teams

In a fast-moving team, schemas are often updated before the code is ready. A backend engineer might merge a .proto change adding a new endpoint, but the actual implementation won’t land for days. Normally, this blocks the frontend engineers who need that endpoint to build the UI.

FauxRPC’s Fallback Mode solves this problem directly.

If the upstream server returns an Unimplemented error code, the proxy doesn’t just pass that error to the client. Instead, it enters fallback mode:

1. It checks the local stub database to see if a mock stub matches the request.
2. If no stub is found, it automatically generates realistic fake data (leveraging protovalidate annotations if they are present in the schema).
3. The fake response is returned to the client.

To the frontend, the endpoint looks and acts like it’s fully implemented. The frontend team keeps coding, completely unblocked by backend delays.

Auto-Recording Stubs

Writing mock stubs by hand is the worst part of API mocking. In proxy mode, FauxRPC can automate this entirely when you pass the --record-dir flag:

stubs/
└── connectrpc.eliza.v1.ElizaService/
    ├── Say.json
    └── Introduce.json

As you interact with your upstream service, FauxRPC intercepts the requests and responses, automatically writing them to disk as structured JSON or YAML files matching your service hierarchy.

For example, a recorded stub might look like this:

{
  "id": "c85d8869-ad10-449e-ba63-2287f7401c10",
  "target": "connectrpc.eliza.v1.ElizaService/Say",
  "active_if": "req.sentence == \"hello\"",
  "content": {
    "sentence": "Hello! How can I help you today?"
  }
}

Because these files are saved directly in your codebase, you can commit them to Git and immediately use them as your mock suite in CI/CD pipelines or integration tests. There is no extra setup: just run your application, click around your frontend, and your API mock stubs are generated for you.

A Protobuf-Native Dashboard for Curation

Having an automated directory of stubs is a massive time saver, but you still need a way to easily curate and manage them. The developer dashboard in FauxRPC (served at http://localhost:6660/fauxrpc when running with --dashboard) gives you exactly that.

The dashboard captures your live traffic history so you can view the raw JSON payload of any logged request. From there, you can copy a pre-compiled FauxRPC YAML stub, along with its generated active_if matcher, directly to your clipboard. This gives you the ability to carefully curate your stub directory while still completely bypassing the need to hand-write them yourself.

This dashboard interface is powered by Protodocs, which displays native protobuf schemas directly inside FauxRPC. Because it understands protobuf natively, you can view your packages, messages, and services, and use a built-in API explorer to run test calls (including unary and streaming APIs) directly from the browser without needing complex local Envoy or gRPC-Web proxy configurations.

Generating Intelligent CEL Matchers

When you are curating these recorded stubs, they are only useful if they match the right request parameters. If you search for user ID 12, you want the stub that returns Alice. If you search for user ID 42, you want the stub that returns Bob.

Whether auto-saved to disk or copied from the dashboard, FauxRPC automatically compiles request shapes into Common Expression Language (CEL) matching rules by examining the request metadata and payload:

1. It ranges over the fields set on the request message.
2. It ignores complex fields (nested messages, lists, and maps) to keep the generated rules readable.
3. For primitive fields (like strings, booleans, and integers), it formats the values into CEL literals.
4. It joins these field checks with &&.

For example, if you send a request where name is “Alice” and age is 30, FauxRPC automatically generates:

active_if: req.name == "Alice" && req.age == 30

When a subsequent request comes in, FauxRPC compiles and evaluates this expression against the request payload. If it evaluates to true, the stub is served.

Recorded Stub Formats

FauxRPC formats these stubs into three distinct kinds based on the call type:

Unary / Client-Streaming Success

For successful unary calls, it records the response payload:

{
  "id": "c85d8869-ad10-449e-ba63-2287f7401c10",
  "target": "connectrpc.eliza.v1.ElizaService/Say",
  "active_if": "req.sentence == \"hello\"",
  "content": {
    "sentence": "Hello! How can I help you today?"
  },
  "priority": 10
}

Unary / Client-Streaming Error

If the upstream returned an error, FauxRPC records the status code and message so you can mock error states:

{
  "id": "18fd7d9e-108c-4a37-bcfc-fa8d7a12ad4f",
  "target": "connectrpc.eliza.v1.ElizaService/Say",
  "active_if": "req.sentence == \"trigger error\"",
  "error_code": 3,
  "error_message": "invalid sentence structure",
  "priority": 10
}

Server-Streaming / Bidirectional Streaming

For streaming APIs, it records the sequence of frames, mimicking latency with a default delay, and captures trailing errors:

{
  "id": "a97df11b-7a31-4b10-8b4b-6f81e3a1f810",
  "target": "connectrpc.eliza.v1.ElizaService/Introduce",
  "active_if": "req.name == \"Bob\"",
  "stream": {
    "items": [
      { "content": { "sentence": "Hi Bob!" }, "delay": "100ms" },
      { "content": { "sentence": "I am Eliza." }, "delay": "100ms" },
      { "error": { "code": 5, "message": "session lost" } }
    ]
  },
  "priority": 10
}

Summary

FauxRPC’s proxy and recording capabilities bridge the gap between static mock servers and real-world backend services. By discovering schemas through reflection, handling dynamic protocol translation, falling back to fake data, and generating intelligent CEL matchers automatically, it takes the busywork out of API mocking.

Traditional mocks drift because they are maintained separately from the systems they represent. By recording real traffic, discovering schemas automatically, and falling back to generated data when necessary, FauxRPC keeps mock environments aligned with production behavior while dramatically reducing the amount of manual work required. It turns mocks from a maintenance chore into a natural byproduct of running your development environment.

Today, ConnectRPC isn’t just a library. It is the core of a toolchain that makes traditional protoc workflows look completely dated. Companies like Anthropic are using it in production to power their SDKs, even maintaining their own ConnectRPC library in Rust.

Let’s look at how far things have come and how tools like Buf Remote Plugins, Protobuf SDKs, FauxRPC, and native HTTP/3 are changing API development.

Code Generation

One of my biggest complaints in Working with Protobuf in 2024 was the compatibility matrix from hell. Managing local installations of protoc, protoc-gen-go, and half a dozen other plugins was a miserable onboarding experience. If one person had a slightly different version of a plugin, the generated code drifted, and the CI build would fail for reasons that took twenty minutes to track down.

We can finally stop doing that. Buf Remote Plugins effectively killed the “it works on my machine” version of protoc. By pointing buf.gen.yaml to remote plugins on the Buf Schema Registry (BSR), we get deterministic, zero-install code generation.

# buf.gen.yaml in 2026
version: v2
plugins:
  - remote: buf.build/connectrpc/go:v1.19.1
    out: gen/go
    opt: paths=source_relative
  - remote: buf.build/protocolbuffers/go:v1.34.1
    out: gen/go
    opt: paths=source_relative

Your CI pipeline doesn’t need a bloated custom Docker image packed with binaries anymore. You just need the buf CLI. New hires clone the repo, run one command, and they’re done. It’s the level of “it just works” that we should have had a decade ago.

First-Class IDE Support

Writing Protobuf used to feel like coding in a glorified Notepad. We lacked the basic editor intelligence that almost every other major language enjoys.

That changed in early 2026 when Buf released a production-grade Language Server Protocol (LSP) server for Protobuf. It’s bundled directly into the buf CLI, which means whether you use VSCode or Neovim, you finally get go-to-definition and reference finding that actually works.

The LSP is workspace-aware, too. You can cmd-click an imported message from a third-party library and jump straight to the definition on the BSR without manually syncing files. It also catches syntax errors and duplicate modifiers before you even try to compile, which saves you from that annoying “context switch to terminal, run build, see error, switch back” loop.

Format, Lint, and Breaking Changes

The Buf CLI also provides the kind of guardrails that keep a team from moving into “legacy debt” territory too quickly.

If you’ve ever sat through a PR review where someone spent ten comments arguing about whether a field should be camelCase or snake_case, buf fmt and buf lint are for you. They end the debate. You run the command, the code is formatted, and the team moves on to actually solving problems.

The real winner is buf breaking. In a microservices setup, accidentally deleting a field or changing a data type in your schema is a great way to wake up the on-call engineer. By running buf breaking in CI, you verify the current schema against previous commits. It catches destructive changes before they hit the main branch, ensuring your contracts stay stable without requiring a human to manually audit every .proto change.

Data Validation

Validation has historically been a tedious chore. Writing endless if req.Age < 0 or if req.Email == "" checks in every single handler is a waste of time and a magnet for bugs.

protovalidate (which recently hit v1.0) moves those rules directly into the Protobuf schema. Since it’s built on Google’s Common Expression Language (CEL), you can do more than just check for nulls; you can write complex cross-field logic, like ensuring a “start date” is always before an “end date.”

By dropping the protovalidate interceptor into your server, requests are automatically validated before they touch your business logic. But the real “aha!” moment is the frontend. Your TypeScript client can run these same rules in the browser before the request even leaves. No more maintaining a separate Zod or Yup schema that inevitably gets out of sync with the backend. One source of truth, enforced everywhere.

Docs and Mocks

Sharing a gRPC endpoint used to be a pain; you couldn’t just hand someone a cURL command and expect it to work. ConnectRPC solved that fundamental issue by supporting standard HTTP/1.1 and JSON. But to truly treat these services like REST APIs, we needed the documentation tooling to match. That is why I spent part of 2024 working on protoc-gen-connect-openapi.

Now, Self-Documenting Connect Services are essentially the default for me. Because ConnectRPC skips binary framing for unary calls and uses standard HTTP status codes, we can generate an OpenAPI spec directly from the Protobuf definitions. You can spin up a Swagger UI directly from your server, let external users test with JSON, and keep your strict internal contracts intact.

We’ve also mostly solved the “waiting for the backend” bottleneck. FauxRPC uses your Protobuf descriptors to spin up a mock server in seconds. When you pair it with protovalidate, the fake data is actually realistic enough to build a frontend against. Some teams are even running FauxRPC in Testcontainers for integration tests, which is much cleaner than trying to manage a “staging” backend for every test run.

Why gRPC-Web Failed

To understand why ConnectRPC won the frontend, you have to look at the history of the protocol. Native gRPC relies on HTTP/2 trailers for status codes, but browsers do not expose those trailers to JavaScript. This originally made gRPC effectively unusable on the web.

The official solution to this problem was gRPC-Web. Its intended goal was straightforward: allow developers to use gRPC directly from web applications. As far as that specific goal goes, it was a success. You could finally make gRPC calls from a browser.

But there is a big difference between a technical success and a widely adopted standard. gRPC-Web never truly took off for a number of reasons. First, it was fundamentally unfriendly to modern infrastructure. It required a separate proxy (usually Envoy) just to translate the frontend requests into something the gRPC backend could understand. This added immediate operational overhead to every project.

Worse, it preserved the most frustrating parts of gRPC. Every single request returned a 200 OK, regardless of whether the server crashed or the resource was missing. It was a baffling design choice that broke the internet’s existing contract for observability. You could not rely on standard load balancer metrics, standard browser dev tools, or your generic APM to see if your site was actually healthy. You were forced to use specialized, protocol-aware tooling just to perform basic debugging. If I have to open a dedicated “gRPC-aware” network tab just to see why a login failed, I feel like it hasn’t actually earned the “web” part of gRPC-Web name.

ConnectRPC stepped in and completely erased the proxy requirement. It also fixed the integration issues with the traditional web. A unary JSON request in ConnectRPC acts exactly like a standard REST call. If a resource is missing, you get a real 404 Not Found, and your existing monitoring stack just works. It gave frontend developers the familiar, straightforward debugging experience they actually wanted while keeping the strict schema safety that backend teams need.

Why it’s my default choice

In 2024, ConnectRPC was about making gRPC more approachable. Now, the underlying protocol is almost an implementation detail. We get the benefits of typed schemas and code generation, but the friction of the “gRPC tax” is gone.

If you’re still hand-rolling JSON/REST APIs or wrestling with legacy gRPC-go stubs and Envoy proxies, it’s time to move on. The tools are ready, the workflow is better, and your on-call engineer will thank you.

In today’s interconnected world, APIs (Application Programming Interfaces) are the glue that connects computers. They allow different applications to talk to each other, share data, and perform actions. However, traditional methods of creating APIs often lead to frustrating challenges: breaking changes in JSON APIs, silent failures due to missing fields, frontend and backend drift, or schema mismatches that result in the classic “works on my machine” excuse.

Imagine a real-world scenario where the backend team renames a userId field to user_id and deploys their changes. Instantly, the frontend checkout process breaks in production because the API had no strict enforcement to catch the mismatch.

This is where contract-based APIs come in. A contract-based API is one where the schema is defined first in a formal specification, and both client and server are generated or validated against that contract. They reduce ambiguity and enforce consistency across services.

The Power of Pre-defined API Contracts

A contract-based API defines exactly what data can be exchanged, in what format, and what actions can be performed. This strict, pre-defined agreement unlocks several immediate advantages:

• Improved Developer Experience: Developers on both sides (client and server) have a clear understanding of what is expected, making integration smoother.
• Automated Documentation: Contracts serve as self-documenting artifacts. This reduces the need for manual documentation maintenance and ensures the docs stay in sync with the actual API implementation.
• Reduced Errors: Mismatched data formats or API changes become less likely, leading to fewer bugs. Contracts act as a validation layer that catches potential issues early.
• Easier Integration: Contracts act as a single source of truth. Developers can quickly understand how to interact with the API without extensive back and forth communication.
• Streamlined Development: These APIs often enable tools to automatically generate code for both client and server implementations. This eliminates manual boilerplate so you can focus on core logic.

Protobuf: The Language of APIs

In modern distributed systems, the foundation of many contract-based APIs lies in Protocol Buffers (protobuf). It is a language-neutral data format specifically designed for structured messages.

Unlike JSON, which is a text-based format designed to be human-readable, Protobuf is a binary format. This means you trade the ability to natively read the raw data in transit for significant performance gains:

• Smaller Message Sizes: Protobuf messages are compact and efficient, which leads to faster transmission and reduced bandwidth usage.
• Faster Parsing: Parsing binary protobuf messages is significantly faster compared to traditional formats like JSON or XML.
• Built-in Versioning: Protobuf uses field numbers (the = 1, = 2 in the code below) to identify data. This allows for excellent backward and forward compatibility. You can add new fields without breaking older clients that do not know about them yet.
• Cross-language Compatibility: Protobuf definitions are language-agnostic. Code for interacting with the API can be generated for almost any modern programming language.

Because the data is binary, you cannot simply open your browser’s network tab and read the payloads by default. You will usually need to rely on modern browser extensions (like the gRPC-Web or Connect dev tools) to decode the traffic. It also requires setting up specialized tooling and build steps to compile the generated code.

Here is a basic example of a .proto file defining messages for a user and an address:

syntax = "proto3";

message User {
  string name = 1;
  int32 id = 2;
  string email = 3;
  Address address = 4;
}

message Address {
  string street = 1;
  string city = 2;
  string state = 3;
  string zip = 4;
}

In this example, the User message has fields for name, ID, email, and an Address message. These defined structures ensure consistent data exchange between applications.

Key idea: Protobuf relies on immutable field numbers instead of field names. This golden rule guarantees backward and forward compatibility.

gRPC: Building APIs on a Solid Foundation

gRPC (gRPC Remote Procedure Call) is a high-performance framework that builds upon protobuf’s strengths. It provides a powerful way to implement remote procedure calls, allowing applications to interact using clients generated for each language.

Introducing Services and Request/Response Types with gRPC

We can expand the .proto file to define a service called UserService with methods for user management:

syntax = "proto3";

service UserService {
  rpc CreateUser(CreateUserRequest) returns (User) {}
  rpc GetUser(GetUserRequest) returns (User) {}
}

message CreateUserRequest {
  User user = 1;
}

message GetUserRequest {
  int32 id = 1;
}

This example defines a UserService with two methods: CreateUser and GetUser. Each method takes a specific request message and returns a response.

Notice how clear the intention is. A helpful mental model to contrast modern APIs is:

• REST is resource-oriented (relying on URLs and HTTP verbs).
• gRPC is action-oriented (relying on explicit methods).

A reader of this spec does not have to map vague HTTP verbs like “POST” to actions like “create.” Also, these method names are greppable. It is trivial to locate every use of CreateUser across several repositories, making refactoring and impact analysis much easier.

Server Reflection

Another powerful feature of the gRPC ecosystem is Server Reflection. This allows clients or debugging tools (like Postman or grpcurl) to query the server at runtime to discover the available services and methods. This eliminates the need to distribute .proto files to developers just so they can explore the API structure.

Distributing API Contracts

Defining a contract is only half the battle. How do the frontend and backend teams actually share that .proto file? If the schema is not easily accessible, the contract is useless.

In practice, teams usually solve this distribution problem in one of three ways:

1. Monorepos: Storing the backend, frontend, and API definitions in a single repository so all code shares the same source of truth.
2. Package Managers: Generating the client SDKs in a CI/CD pipeline and publishing them as internal NPM, Maven, or Go packages.
3. Schema Registries: Using dedicated tools like the Buf Schema Registry to manage, version, and distribute Protobuf files securely across an organization.

What about public APIs?

Historically, strict RPC contracts were tough for external, public-facing APIs. If your primary consumers were third-party developers, handing them a raw Protobuf file or expecting them to set up gRPC clients caused massive friction. They just wanted to use standard REST with JSON.

This is where tools like ConnectRPC shine. ConnectRPC allows you to define your API using Protobuf, but it automatically exposes endpoints that support standard HTTP/1.1 and JSON serialization as a fallback format.

This hybrid approach also solves the local debugging problem. You can configure ConnectRPC to use JSON during local development specifically so you can read the network tab in plain text, and then flip it to highly efficient binary for production. In practice, you write Protobuf once, and get both gRPC and REST/JSON APIs for free.

Even better, because the source of truth is still Protobuf, you can use ecosystem plugins to automatically generate an OpenAPI specification directly from your .proto files. You get a highly maintainable, contract-driven architecture on the backend, while your external users can still curl standard REST endpoints, read plain JSON, and explore your API via a generated Swagger UI. It offers the best of both worlds without compromising the developer experience on either side.

Key idea: Tools like ConnectRPC allow you to maintain strict internal Protobuf contracts while exposing standard REST/JSON APIs to external consumers.

Alternatives

While Protobuf and gRPC are a powerful duo, there are other contract-based API solutions to consider depending on your architecture:

• OpenAPI (Swagger): Contracts are not exclusive to RPC. You can use OpenAPI to define strict contracts for RESTful services. However, a harsh reality of the industry is that OpenAPI specs often drift from the actual code because they are bolted on after the fact. To make OpenAPI truly safe, teams must rely on strict framework integration (like FastAPI in Python or tsoa in Node) where the code generates the spec, or vice versa.
• GraphQL: Arguably the most mainstream contract-driven API paradigm for frontend developers. Its strictly typed schema defines the exact shape of the available data. Unlike gRPC, which has fixed responses, GraphQL allows the client to dictate the exact payload it wants to receive.
• Twirp: Developed by Twitch, Twirp is a lightweight RPC framework built on top of Protobuf and HTTP/1.1. It shares similarities with ConnectRPC but focuses on absolute simplicity. It avoids the complexity of HTTP/2 and gRPC streams while still providing generated clients, making it an excellent alternative if full gRPC is overkill for your needs.
• Thrift: Originally developed at Facebook, Thrift is a language-neutral protocol for defining service contracts similar to Protobuf. It is often found in large-scale data environments and supports various RPC protocols.
• tRPC: This tool defines the API schema directly in TypeScript code to be reused on both the client and the server. While it often pairs with libraries like Zod for runtime validation, it lacks true language-agnostic safety across the network boundary since it relies entirely on a TypeScript ecosystem.
• Avro: This format uses JSON-like schemas but stores data in a compact binary format. It is a staple in the Apache Kafka ecosystem for streaming data pipelines. It handles schema evolution differently than Protobuf (often sending the schema alongside the data), making it highly flexible for dynamic systems.

When NOT to Use API Contracts

While these tools are powerful, they are not a silver bullet. You should reconsider using strict API contracts if:

• You are building small projects or MVPs: The initial setup, code generation, and boilerplate overhead might slow down your speed of delivery when rapid iteration is the top priority.
• Simplicity for external consumers outweighs strict contracts: If you are building a straightforward public API and are not using a hybrid tool like ConnectRPC, raw JSON over REST remains the path of least resistance for third-party developers.
• Your team lacks tooling maturity: Implementing gRPC or Protobuf requires solid CI/CD pipelines and a team that is comfortable managing build steps, code generation, and backward-compatible schema evolutions.

Key idea: Strict API contracts add overhead and may not be suitable for small MVPs, simple public APIs, or teams lacking tooling maturity.

Conclusion

Contract-based APIs offer a significant advantage in building robust and scalable communication between applications. Protobuf and gRPC provide a powerful combination for defining clear contracts and generating highly efficient code.

As a general rule of thumb: if you are building an early-stage prototype, stick to what is fast and familiar. But if you are scaling a complex system across multiple teams and services, contract-based APIs transition from a nice-to-have to an absolute necessity. Once multiple teams depend on your API, contracts stop being optional. They are how you avoid chaos.

Imagine staring at a production log with a generic error in a function called processData(). You search the codebase, only to find forty different functions with that exact name spread across five repositories. This is the opposite of searchable code.

Greppability^[3] is a measure of how easily a human can find specific logic using simple text search. Modern IDEs are great, but greppability is the safety net for when they fail: during code reviews, in terminal sessions, while scanning traces, or when navigating a massive mesh of microservices.

Navigation is the Job

Developers spend a considerable amount of effort on program comprehension, which includes reading and understanding source code. In fact, research shows that programmers spend roughly 70% of their time^[1] on code comprehension.

When you use a generic name like entity, you increase the “search cost"^[2] of the workday. Research indicates that when programmers deal with high comprehension effort, they navigate and make edits at a significantly slower rate. Choosing specific names like CustomerBillingRecord ensures your search result is a surgical strike rather than a list of a hundred collisions.

Spanning Across Repos

The real power of greppability shows up in polyglot environments. When your infrastructure spans multiple repositories and languages, like Go, TypeScript, and Python, your IDE “Jump to Definition” often stops at the edge of the current project.

In these distributed systems, a unique string is the only universal bridge. A specific domain term, like SubscriptionRenewalWebhook, connects a frontend UI component to a backend service implementation across repo boundaries. If both use the same unique name, you can track a feature across the entire stack in seconds without needing a specialized indexer for every language you use.

Key idea: Greppability turns your codebase into a searchable database. By treating unique strings as the “primary keys” of your architecture, you bypass the limitations of IDEs and language boundaries, especially in multi-repo environments where traditional navigation tools often break.

How to Write for Grep

• The Log-to-Code Pipeline: If a method handles a critical business action, name it something distinct. If a system crashes at 3:00 AM and the log says Error in CalculateRegionalTax(), you have found the bug before you have even opened your editor.
• Avoid Generic Names: Names like data, info, manager, or entity are search poison. orderValidationLogic is longer, but it is a direct hit for a search engine.
• Establish a Naming Hierarchy: Focus your “naming budget” where the scope is widest. Filenames, class names, and API methods must be unique. Local variables inside a five-line function, like i or buf, can stay generic because their search boundary is tiny.
• Beware of Dynamic Magic: If your language uses reflection or string interpolation to call methods, such as this.call("prefix_" + action), you have killed the ability to grep for the implementation.

The gRPC Search Advantage

This is where RPC-based designs offer a distinct advantage over REST. In a REST architecture, searching for an “update” action usually requires a two-step mental grep: find the path (/users/:id) and then filter by the HTTP verb (PUT).

In gRPC, the method name is a globally unique string.

In a gRPC environment, UpdateUserBio() appears identically in your .proto file, your server code, your client code, and your monitoring dashboard. It is the ultimate greppable identifier.

Summary

Greppability is an engineering-first term. It moves naming away from subjective aesthetics and toward functional operability. When you tell a teammate that a name is not very greppable, you are not critiquing their style: you are pointing out a future debugging bottleneck.

Protobuf includes a feature known as unknown fields. They act as a safety net when systems encounter data they weren’t explicitly built to handle. Here is a breakdown of what they are and why they matter.

What are Protobuf Unknown Fields?

Your .proto file defines the expected structure, fields, and data types. But what happens when you parse a message and it contains fields that aren’t in your current .proto definition?

These extra pieces of data are called unknown fields.

At a lower level, unknown fields are field numbers and wire types that exist in the serialized message but are not defined in the current schema.

This mechanism is what enables forward compatibility: an older version of your software can safely read, process, and forward data produced by a newer version of the schema without crashing or losing the new data.

Key idea: Unknown fields enable forward compatibility by default.

Preserving Unknown Data

A key aspect of unknown fields is how they behave during message manipulation.

If you receive a message with unknown fields and forward it to another system, Protobuf defaults to forwarding the unknown fields alongside the known ones. This ensures the receiving system gets the complete payload.

If this didn’t happen, you could accidentally clear field values set by another part of the system.

This forwarding capability also applies when persisting messages, as long as they remain in binary Protobuf format. If you store and later reload the binary payload, the unknown fields are preserved.

Key idea: Binary Protobuf preserves unknown fields end-to-end.

Historical Note: Protobuf v3 initially tried to simplify the specification by removing several proto2 features, but real-world usage forced them to walk the biggest ones back. Early versions of proto3 dropped unknown fields entirely, but this was reversed in v3.5. Similarly, proto3 initially removed the optional keyword, but brought it back in v3.15 after developers struggled to distinguish between a field being unset and a field just having a zero value, which is a classic programming mistake.

Comparison to JSON

Consider a scenario where a new field, email, is added to a user object. The backend is updated, but the frontend is not.

The issue in JSON systems is not JSON itself, it’s typed deserialization.

{
  "user": {
    "id": "0edc0903-9e31-47be-adad-1dfc434ca2d3",
    "name": "Bob",
    "email": "bob@example.com"
  }
}

If the frontend maps this into a typed structure:

class User {
  id: string;
  name: string;
}

The unknown field (email) is dropped during deserialization. When the object is sent back:

{
  "user": {
    "id": "0edc0903-9e31-47be-adad-1dfc434ca2d3",
    "name": "Bob"
  }
}

The email field is lost.

Key idea: Typed JSON pipelines often drop unknown fields during reserialization.

Protobuf Behavior

With Protobuf, the same scenario behaves differently.

Even if the frontend does not know about the email field, it is preserved internally:

Symbol(@bufbuild/protobuf/unknown-fields): [
  {0: {no:3, wire_type:2, data: Uint8Array(14)}}
]

(Note: This specific Symbol representation is how the @bufbuild/protobuf implementation manages it under the hood. Other JS/TS generators might expose this data slightly differently, but the underlying concept remains the same.)

• no: 3 → field number (email)
• wire_type: 2 → length-delimited (used for strings)
• data → raw encoded value

When the message is re-encoded:

1:LEN {"0edc0903-9e31-47be-adad-1dfc434ca2d3"}
2:LEN {"Bob"}
3:LEN {"bob@example.com"}

The unknown field survives the round trip.

Key idea: Unknown fields are preserved even when not understood.

Why We Can’t “Guess” the Type

You might wonder why we can’t just look at the raw data in an unknown field and “guess” what it is. The reason is that Protobuf uses a very limited set of wire types that are shared across many different high-level data types.

If you see an unknown field with Wire Type 0, you don’t know if it represents the number 150, the boolean true, or an enum value.

More critically, if you see Wire Type 2, you have no way of knowing if the data is a string, a bytes buffer, or a complex nested message. If you try to “guess” and parse a string as a sub-message, you will likely get gibberish or a parsing error. This is why the schema is strictly required to understand the data, while the binary format is sufficient to preserve it.

The Middleware Advantage

Unknown fields shine in internal, middleware-heavy architectures.

Example:

• API Gateway reads id for routing
• Logging service reads trace_id
• Downstream service understands full schema including new fields

Intermediate services can safely:

1. Unmarshal using an older schema
2. Read known fields
3. Forward the message unchanged

No coordination is required when new fields are added upstream.

Key idea: Internal middleware can stay stable while schemas evolve.

Observability: A Signal for Upgrades

Beyond just forwarding data safely, unknown fields provide a highly valuable observability metric.

When an API gateway or a downstream service detects unknown fields in incoming payloads, it is a clear telemetry signal: a client or upstream service is sending extra information because it is using a newer schema.

Instead of crashing or silently dropping the data, the service can log the presence of these unknown fields. You can use this data to trigger alerts, track the rollout progress of new features across your architecture, and pinpoint exactly which legacy services are lagging behind and due for an upgrade.

A Note on JSON Serialization and Object Re-use

There are a couple of important exceptions where unknown fields get lost.

First, unknown field preservation applies only to binary Protobuf serialization. If you convert from binary to JSON (e.g., using protojson in Go or toJson in TypeScript), unknown fields are dropped during the encoding process. When unmarshaling JSON back into Protobuf, many libraries are strictly configured by default. For example, Go’s protojson.Unmarshal will throw a hard error if it encounters unknown fields in the JSON payload unless you explicitly bypass it by passing DiscardUnknown: true. JSON simply isn’t designed to carry this extra payload without a strict schema map.

Second, preserving these fields during binary serialization requires that you re-use the exact same object for re-serialization. If you read a message, pull out the known fields, and map them into a freshly created object to send downstream, the unknown fields tied to the original object will be left behind.

Key idea: Binary preserves and JSON drops. Always re-use the original object if you want to keep unknown fields intact.

Databases and Security

The theoretical elegance of unknown fields often collides with the messy reality of databases and security perimeters. In practice, relying on unknown fields breaks down entirely in a few critical scenarios.

First, let us consider database persistence. If clients are trying to store extra data, and a backend service parses a Protobuf message to map it to standard relational database columns, those unknown fields are absolutely gone. There is no magic column for data your database schema does not know about.

The only way to achieve true end-to-end preservation is to store the entire serialized Protobuf message directly in the database as a BLOB. Some teams do this, but blindly storing data you haven’t validated and don’t even recognize is highly dangerous.

Allowing unknown fields to propagate unchecked from external sources is a significant security risk. While they are a powerful tool inside clearly defined, trusted internal pipelines, accepting them from the open web opens your system up to data smuggling. It allows malicious actors to sneak unvalidated payloads into unknown fields to bypass validation layers that only inspect known schema structures. If your systems blindly unmarshal, store, and forward this data, older services act as unwitting mules for malicious input.

Because of these exact risks, the standard security posture is to aggressively filter at the edge. API Gateways and ingress proxies should explicitly discard unknown fields before the data ever reaches internal microservices.

Conclusion

Unknown fields provide a powerful mechanism for forward compatibility in distributed systems. They allow internal systems to evolve independently, act as a clear signal for required upgrades, reduce coordination overhead, and simplify middleware design.

However, they are not a substitute for validation, schema discipline, or proper security boundaries. Use them intentionally in trusted internal pipelines, but never trust them at the edge.

The encoding you use (binary protobuf or transcoded JSON) dramatically changes the rules of what constitutes a “safe” or “breaking” change to your API. A change that is perfectly harmless for a protobuf client can completely break a JSON client. Let’s dig into this more.

How Encodings Work

First, a quick refresher on how each format represents data. Consider this simple protobuf message:

syntax = "proto3";

package my_service.v1;

message User {
  // A unique identifier for the user.
  int64 user_id = 1;
  // The user's full name.
  string name = 2;
}

Protobuf: It’s All About the Numbers

On the wire, the binary protobuf encoding doesn’t care about the field names (user_id, name). It only cares about the field numbers (1, 2) and their wire types. A simplified view of the encoded data is a series of key-value pairs where the key is the field number. I dig into this further in my gRPC from Scratch series, where I discuss the binary protobuf encoding.

Because of this, you can rename a field in your .proto file, and as long as the field number and type remain the same, it’s a non-breaking change for protobuf clients.

JSON: It’s All About the Names

When a gRPC gateway or library transcodes this message to JSON, it produces a standard JSON object. JSON is also a perfectly valid encoding to use with gRPC. By default, it uses the protobuf field names (converted to lowerCamelCase) as the JSON keys:

{
  "userId": 12345,
  "name": "Alex"
}

Since JSON clients are coupled to these names, changing them will inevitably break the integration. JSON clients are coupled to field names, not field numbers. This fundamental difference is the source of many potential compatibility issues.

Analyzing API Changes: Breaking vs. Non-Breaking

Let’s look at common changes you might make to a .proto file and see their impact on each encoding.

The Solution: Decouple Names with json_name

So, how do you refactor your .proto field names without breaking your JSON clients? The protobuf specification provides a simple and elegant solution: the json_name field option.

This option lets you explicitly set the JSON key for a field, decoupling it from the .proto field name.

Let’s revise our User message. Suppose we want to rename name to full_name for clarity in our Go or Python code, but we can’t break existing JSON clients that rely on the "name" key.

syntax = "proto3";

package my_service.v1;

message User {
  int64 user_id = 1 [json_name = "userId"];

  // The field is now 'full_name' in code, but will still be 'name' in JSON.
  string full_name = 2 [json_name = "name"];
}

With json_name = "name", we’ve instructed the transcoder to do the following:

1. For Protobuf: Continue using field number 2. The field name full_name is used by the code generator.
2. For JSON: Always use the key "name" during serialization, regardless of what the .proto field is called.

Now, you are free to change the full_name field to something else (e.g., user_display_name) in the future, and your JSON contract remains stable.

Automating Your Safety Net with buf breaking

Remembering all these nuanced rules across different encodings is difficult and error-prone. This is where automated tooling becomes essential. The popular Buf toolchain includes a powerful command, buf breaking, designed specifically for this problem.

The buf breaking command compares your current .proto files against a previous state (like your main git branch) and reports any changes that would break your API consumers. Crucially, it understands that “breaking” means different things to different clients. You can configure it to check against multiple compatibility strategies.

In your buf.yaml configuration file, you can specify which rule sets to check against:

• FILE: Checks for backward-incompatible changes at the .proto file level, like deleting a field or changing a field number. This protects your protobuf-based clients.
• WIRE_JSON: Checks for backward-incompatible changes for the JSON wire format. This catches things like renaming a field without using json_name. This protects your JSON-based clients.
• PACKAGE: Checks for source-code-level breaking changes in the generated stubs for languages like Go and Java. This protects the developers using your generated code.

A typical configuration for a service with both gRPC and JSON clients might look like this:

# buf.yaml
version: v2
breaking:
  use:
    - FILE
    - WIRE_JSON

By integrating buf breaking into your CI/CD pipeline, you can automatically prevent developers from merging changes that would break any of your consumers, whether they speak protobuf or JSON.

Conclusion

Evolving an API for both protobuf and JSON clients is a recipe for a very specific kind of headache, the kind that pages you at 3 AM. You’ve got protobuf, which only cares about numbers, and JSON, which only cares about names. A “safe” refactor for one is a production-breaking slap in the face for the other. This is where a schema-first approach, backed by powerful schema-aware tooling, isn’t just a good idea; it’s the only thing keeping you from questioning all your life choices.

Protobuf’s semantics, like the json_name option, give you a powerful escape hatch. It makes certain refactors, like renaming a field for internal clarity, trivial if you have the right tooling in place. You can change your code without your JSON clients ever knowing you touched a thing. This decoupling is a superpower, but only if you use it correctly.

And that’s the catch: don’t rely on developers’ goldfish sized memory or manual code reviews to enforce these complex, conflicting rules. That’s how you break production at 3 AM. Instead, let the robots do the heavy lifting. Integrating a tool like buf breaking into your CI pipeline is like having an unblinking, unforgiving guardian for your API. It understands the different breaking change rules for both protobuf and JSON and will stop a bad change before it ever gets merged. This is the real strength of a schema-first workflow: it makes complex refactors not just possible, but safe. You can merge with confidence and keep all your clients (binary or JSON) happy.

But, why?

The standard HTTP methods serve us well, except the ones that don’t… but that’s a topic for another day. GET is great for simple data retrieval, but its reliance on URL parameters makes it cumbersome for complex queries or large sets of input parameters. URLs have practical length limits, and embedding deeply nested structures is awkward.

This often leads developers to use POST for what are semantically read-only query operations. Many widely-used protocols effectively do this:

• GraphQL typically uses POST requests with JSON bodies to send queries.
• gRPC and gRPC-Web typically rely exclusively on POST.
• Older protocols like SOAP and XML-RPC almost exclusively use POST to encapsulate their operations, including data retrieval. This has been an issue for a long time!

While using POST works, it’s a compromise. POST traditionally implies an action that might change state on the server, which means intermediaries (read: caches) and even client-side logic might treat these “query-via-POST” requests with undue caution, forgoing caching or automatic retries that would be safe for a truly read-only operation.

This is precisely the problem that a dedicated safe method with a request body aims to solve. The IETF HTTP Working Group is discussing such a method in a draft titled “A Safe HTTP Method with a Request Content”. As the draft states:

The QUERY method provides a solution that spans the gap between the use of GET and POST. As with POST, the input to the query operation is passed along within the content of the request rather than as part of the request URI. Unlike POST, however, the method is explicitly safe and idempotent, allowing functions like caching and automatic retries to operate.

While the draft uses “QUERY” as a candidate name (among others), the core idea is what we’re exploring: a method for safe, idempotent data retrieval that can carry a payload. For the rest of this article, we’ll continue to use “QUERY” to represent this concept and show how you can implement such a custom method in Go today.

It’s crucial to remember that until such a method is formally standardized and widely adopted, using custom HTTP methods can impact interoperability. However, for internal APIs, tightly controlled systems, or as a forward-looking experiment, they can be very useful.

Server-Side with Go

Go 1.22 introduced enhancements to http.ServeMux that allow you to register handlers for specific HTTP methods and paths more directly. Let’s build a server that handles our custom QUERY method:

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
)

func queryHandler(w http.ResponseWriter, r *http.Request) {
	// The new ServeMux handles method checking based on registration.

	body, err := io.ReadAll(r.Body)
	if err != nil {
		http.Error(w, "Error reading request body", http.StatusInternalServerError)
		return
	}
	defer r.Body.Close()

	// In a real application, you'd parse the query from the body
	// (e.g., JSON) and fetch data accordingly.
	fmt.Fprintf(w, "Received your QUERY request with body: %s\n", string(body))
	log.Printf("Handled QUERY request with body: %s", string(body))
}

func main() {
	mux := http.NewServeMux()
	// Register handler specifically for QUERY method on /data path
	mux.HandleFunc("QUERY /data", queryHandler)

	log.Println("Server starting on port 8080, handling custom QUERY method...")
	if err := http.ListenAndServe(":8080", mux); err != nil {
		log.Fatal(err)
	}
}

In this server:

1. We use mux.HandleFunc("QUERY /data", queryHandler) to directly associate the queryHandler with our custom QUERY HTTP method for the /data path.
2. The queryHandler reads the request body, where the complex query parameters would reside.

Congratulations, we’ve made an API that uses the QUERY method. Now let’s write a client to pair with this server.

Client-Side, also with Go

Here’s how a Go client can send a QUERY request:

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"strings"
)

func main() {
	queryPayload := `{"filters": {"status": "active", "category": "electronics"}, "fields": ["name", "price"]}`
	client := &http.Client{}

	// Create a new request with the custom "QUERY" method
	req, err := http.NewRequest("QUERY", "http://localhost:8080/data", strings.NewReader(queryPayload))
	if err != nil {
		log.Fatalf("Error creating request: %v", err)
	}
	req.Header.Set("Content-Type", "application/json") // Important for body processing

	resp, err := client.Do(req)
	if err != nil {
		log.Fatalf("Error sending request: %v", err)
	}
	defer resp.Body.Close()

	fmt.Println("Response Status:", resp.Status)
	body, err := io.ReadAll(resp.Body)
	if err != nil {
		log.Fatalf("Error reading response body: %v", err)
	}
	fmt.Println("Response Body:", string(body))
}

The client uses http.NewRequest("QUERY", ...) to specify the custom method and sends the queryPayload in the request body.

Client-Side with cURL

Although web browsers won’t randomly make QUERY calls, many other tools can use QUERY. Many also support arbitrary HTTP verbs, although this isn’t typically leveraged due to filters from load balancers, firewalls, etc.

Anyway, here’s what that request looks like with cURL:

$ curl -X QUERY -d '{"filters": {"status": "active", "category": "electronics"}, "
fields": ["name", "price"]}' http://localhost:8080/data -v
* Host localhost:8080 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8080...
* Connected to localhost (::1) port 8080
> QUERY /data HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 89
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 89 bytes
< HTTP/1.1 200 OK
< Date: Sat, 31 May 2025 15:47:45 GMT
< Content-Length: 129
< Content-Type: text/plain; charset=utf-8
< 
Received your QUERY request with body: {"filters": {"status": "active", "category": "electronics"}, "fields": ["name", "price"]}
* Connection #0 to host localhost left intact

QUERY vs. GET and POST - A Clearer Separation

Let’s summarize the distinctions with our (potentially future-standard) QUERY method in mind:

• GET: Used for retrieving data. Parameters are typically sent in the URL. GET requests must be safe and idempotent. They generally don’t have a body.
• POST: Used for submitting data to be processed, often resulting in a change of state or side effects on the server (e.g., creating a new resource). Parameters are sent in the request body. Not necessarily safe or idempotent.
• QUERY (custom/proposed): Used to request data (like GET) but with the ability to send a complex query in the request body (like POST). Crucially, it is defined as safe and idempotent (like GET). This explicitly tells intermediaries and clients that the request has no side effects and can be cached or retried automatically.

Caching Behavior: GET, POST, and QUERY

HTTP caching is vital for performance. A method’s characteristics (especially safety and idempotency) directly influence its cacheability.

1. GET

• Behavior: GET requests are inherently cacheable. Caches readily store and serve responses to GET requests if caching headers (like Cache-Control, Expires, ETag) allow. This is a foundational aspect of HTTP performance.

2. POST

• Behavior: POST requests are generally not cacheable by default. Since POST can have side effects, caching responses could lead to unintended consequences or stale data if the action isn’t repeated.

3. QUERY

The IETF draft emphasizes that a method like QUERY is “explicitly safe and idempotent, allowing functions like caching and automatic retries to operate.” This is key.

• Behavior (Current Custom Method): Because our QUERY method is non-standard today, caches will not cache it by default. They typically only automatically consider standard safe methods like GET. To make responses to a custom QUERY request cacheable:
  • The server must send explicit caching headers (e.g., Cache-Control: public, max-age=3600). These headers signal the cacheability of the response.
  • Intermediary caches (CDNs, reverse proxies) might need specific configuration to recognize and cache responses for this custom method.
  • The Vary HTTP header is important if the response depends on the request body content.
• Behavior (Future Standardized Method): If a method like QUERY becomes a recognized HTTP standard, caches would likely treat it similarly to GET for caching purposes, provided it’s implemented according to its safe and idempotent semantics. This would be a major advantage, allowing complex queries with bodies to be cached as effectively as GET requests.

Key Takeaway for QUERY Caching

The intent of a QUERY method is to be cacheable. If using a custom method today, you must provide explicit caching directives. The ongoing standardization effort aims to make this caching behavior more automatic and universally understood, unlocking performance benefits for complex, body-inclusive queries that are currently often forced into less cache-friendly POST requests. How long will this process take? Who knows! These kinds of changes can take decades.

What’s Next?

The discussion around a safe HTTP method with a request body is an exciting development. By understanding its purpose and experimenting with custom methods like QUERY in Go, we can better appreciate the nuances of HTTP and prepare for potential future standards that will make our APIs more robust and performant.

Some projects have already taken to adding support for the QUERY method. Take a look at NodeJS: Support for ‘QUERY’ method, which added support last year. As demonstrated earlier in this article, using QUERY is already possible in Go. This general support is often true for other languages and libraries as well, since custom HTTP methods are a feature of the HTTP specification.

The other aspect of moving this forward is a bit more nebulous: bureaucracy. There is still work and review to be done to graduate the draft into an official RFC from the IETF. But fear not. There’s actually steady changes being made to the draft document. You can tell this from the git history on the httpwg’s repo. At the time of writing May 19th was when the last change was made, so I’m certain that this hasn’t been forgotten about.

How it works

First, define your validation rules using protovalidate’s constraint annotations within your Protobuf definitions. For guidance, you should reference the protovalidate documentation. Fauxrpc will take care of the rest, automatically validating each request against these rules. If a request fails validation, a detailed error message will be returned, guiding you toward quickly fixing the issue.

For this example, we’re going to use a simple service where you can call with a name and it will return a greeting. Here’s what that would look like in protobuf:

greet.proto

syntax = "proto3";

package greet.v1;

import "buf/validate/validate.proto";

message GreetRequest {
  string name = 1 [(buf.validate.field).string = {min_len: 3, max_len: 20}];
}

message GreetResponse {
  string greeting = 1 [(buf.validate.field).string.example = "Hello, user!"];
}

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse) {}
}

The buf.validate.field annotations are from protovalidate. The section (buf.validate.field).string = {min_len: 3, max_len: 20} ensures that the name field is between 3 and 20 characters. (buf.validate.field).string.example = "Hello, user!" is declaring what an example response might look like. This mostly serves as documentation but it also influences other tools, including protoc-gen-connect-openapi and, now, FauxRPC! We’ll see how FauxRPC uses both of these constraints in a second. First, let’s start up the FauxRPC server.

To use protovalidate with FauxRPC, we need to leverage another tool that’s extremely helpful for working with protobuf definitions. This is because we used protovalidate, a dependency. Usually dependencies are hard to deal with in protobuf because the default protobuf tooling just doesn’t manage dependencies at all. However, the Buf CLI and the BSR can help us here.

First, create a new buf.yaml file in the same directory as greet.proto with the contents:

version: v2
deps:
  - buf.build/bufbuild/protovalidate

Now we’re just a few commands away from having a mock service running:

# Get Buf CLI to pull our new dependency
$ buf dep update
# Build our protobuf file (and dependencies) into a protobuf "image": https://buf.build/docs/build/overview/
$ buf build . -o greet.binpb
# Start FauxRPC with this image
$ fauxrpc run --schema=greet.binpb

Now let’s try some requests against this new service:

$ buf curl --http2-prior-knowledge -d '{}' http://127.0.0.1:6660/greet.v1.GreetService/Greet
{
   "code": "invalid_argument",
   "message": "validation error:\n - name: value length must be at least 3 characters [string.min_len]",
   "details": [
      {
         "type": "buf.validate.Violations",
         "value": "CkIKBG5hbWUSDnN0cmluZy5taW5fbGVuGip2YWx1ZSBsZW5ndGggbXVzdCBiZSBhdCBsZWFzdCAzIGNoYXJhY3RlcnM",
         "debug": {
            "violations": [
               {
                  "fieldPath": "name",
                  "constraintId": "string.min_len",
                  "message": "value length must be at least 3 characters"
               }
            ]
         }
      }
   ]
}

Oh, duh! We hit the length constraint we added for the name field, so our empty object {} isn’t good enough anymore. We need a name and it needs to have between 3 and 20 characters. Let’s try once more:

$ buf curl --http2-prior-knowledge -d '{"name": "Bob"}' http://127.0.0.1:6660/greet.v1.GreetService/Greet
{
  "greeting": "Hello, user!"
}

Great, we received a response! And the response is populated using our (buf.validate.field).string.example annotation. This will allow us to stand up a fake service with a few simple commands that has request validation built in and will use the constraints to make realistic fake data.

This shows how protovalidate uses protovalidate constraints both for input validation and fake data generation.

On a related note, I am personally excited to eventually have Typescript Support for protovalidate. That would allow us to use protovalidate for both clean frontend UX and robust server side validation. In my eyes, this would completely solve the problem of having duplicate and inconsistent validation rules between the frontend and backend. Backend needs these rules to ensure system reliability and consistency. Frontend needs these rules for better UX.

Benefits for you

FauxRPC and protobuf synergizes well with the model-driven API design with protobuf. From a single API definition you have strongly typed definitions that has support for many programming languages, powerful validation constraints, examples of what each field looks like. And FauxRPC lets you experiment with all of that before writing a single line of application code. This means:

• Reduced development time: Spend less time debugging data issues and more time building amazing features.
• Increased confidence: Trust that your RPCs are handling data correctly, leading to more stable and reliable applications.
• Improved collaboration: Make it easier for teams to work together by ensuring everyone adheres to the same data standards.

Ready to give it a try?

Ready to experience the power of FauxRPC and protovalidate?

• Update to the latest version of FauxRPC: github.com/sudorandom/fauxrpc/releases/latest
• Learn more about protovalidate: github.com/bufbuild/protovalidate
• Explore the FauxRPC documentation: fauxrpc.com

For reference, all of the code in the article is available here.

Generated Code

To get started, I have to talk about how ugly the code generated from protobuf definitions is. It has historically been verbose, complex, and difficult to navigate. Even though it’s not meant to be hand-edited, this can impact code readability and maintainability, especially when integrating gRPC into larger projects. This has actually improved a lot recently in most languages but even so, there are some rough edges.

Language-specific Quirks

Protobuf and gRPC’s initial implementations often diverged from language-specific norms, especially in their HTTP handling. This stemmed partly from the decision to mandate HTTP/2 support, a decision that has since proven to limit gRPC’s reach into the web frontend. We know now from gRPC-Web that trailers aren’t a hard requirement for a protocol like gRPC. In the aftermath of this decision, we are now left with a need to evolve the language implementations of protobuf and gRPC to be more idiomatic for each language.

For Go, avoiding the net/http package is a rough decision because it makes it harder to use gRPC endpoints alongside other kinds of HTTP APIs and to re-use HTTP middleware. They eventually added a ServeHTTP() interface to grpc-go as an experimental way to use the HTTP server from the Go standard library but using that method results in a significant loss of performance. Maybe they did it for performance reasons? If so, it’s definitely a tradeoff that has split gRPC from the rest of the Go ecosystem.

Sometimes language quirks actually impact how you design protobuf types. If you follow the style recommendations from Buf, the names of enums are expected to be prefixed an upper-snake-case version of the enum name, like so:

enum FooBar {
  FOO_BAR_UNSPECIFIED = 0;
  FOO_BAR_FIRST_VALUE = 1;
  FOO_BAR_SECOND_VALUE = 2;
}

This is described better in the buf lint rule description for ENUM_VALUE_PREFIX but the style guide is like this because of C++ scoping rules with enums, which makes it impossible to have two enum values in the same package with the same enum value name. While this convention originated from C++ scoping rules, it affects how you should design all protobuf files. Why would scoping inside of the enum not be enough for the C++ compiler to generate unique names? Why is this flaw something that impacts the style guide and, in effect, all target languages? To me, this is kind of ugly, because quirks of some language implementations are bubbling up in unintuitive ways.

The generated code isn’t even that fast

One benefit of generated code is that you can generate code that no sane human would write in order to get some performance optimizations. However, if you look at some of the code generated from protobuf you’ll see runtime reflection used a lot. Why? In a way, I am saying the generated code isn’t ugly enough. Let’s look at a concrete example. Be warned that this will be a very Go-specific section because most of my experience with protobufs is in Go. However, the same strategy has been applied in most languages.

Let’s take a look at super a simple example in Go. Here’s the protobuf:

message Hello {
  string name = 1;
}

Here’s the type generated by protoc:

type Hello struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
}

// With these methods, contents are stripped
func (*Hello) Reset()
func (*Hello) String() string
func (*Hello) ProtoMessage()
func (*Hello) ProtoReflect() protoreflect.Message

There’s actually no Marshal() or Unmarshal() functions defined specifically for this type. This means that runtime reflection is used to make serialization work. Reflection is generally seen as slower, because it is slower. I find it strange that optimized, type-specific serialization code isn’t being generated for Go. That said, you can actually get this by using a separate protoc plugin called vtprotobuf that will generate specialized marshal and unmarshal functions for each protobuf type. It also allows for using type-specific memory pools, which can also help reduce allocations and improve performance. From my own testing just adding vtprotobuf with zero code changes can improve performance by 2-4%. This is essentially a “free” 2-4%, so it’s super strange to me that this wouldn’t be part of the standard compiler. You may not like it, but this is what peak performance looks like. Anyway, this project needs more love and support.

Note that there are other efforts which claim outrageous improvements over what the standard protobuf library does. They do make tradeoffs to achieve these performance gains, but many times the extra complexity is worth it.

You might have read this section and thought “well, this would increase the amount of code being generated and increase binary or package sizes and in some environments, you might not want that. That’s true amd that’s why protobuf has an optimize_for option, so you can annotate one of the following:

• option optimize_for = SPEED; - more verbose, faster code
• option optimize_for = CODE_SIZE; - smaller code
• option optimize_for = LITE_RUNTIME; - intended to run on a smaller runtime that omits features like descriptors and reflection.

See the full description for optimize_for on the official protobuf documentation. While these options exist, they aren’t actually used for most target languages. In the future I would totally like to see most of vtprotobuf be rolled into the standard protobuf compiler for Go and be used if optimize_for = SPEED. Integrating vtprotobuf-like optimizations into the standard protobuf compiler could offer significant performance gains for Go and there are potentially similar opportunities in other languages as well.

Required Fields

The maintainers of protobuf learned some hard lessons with required fields. They felt like they misstepped so badly, that they made a new version of protobuf, proto3, just to remove required fields from the spec. Why? The author of the “Required considered harmful” manifesto talks about this in a lengthy hacker news comment, but the important bit is:

Real-world practice has also shown that quite often, fields that originally seemed to be “required” turn out to be optional over time, hence the “required considered harmful” manifesto. In practice, you want to declare all fields optional to give yourself maximum flexibility for change.

This is echoed by the official style guide of protobufs, where they recommend adding a comment indicating that a field is required. If we’re talking about getting a message from A to B, I totally agree with this line of thinking. However, just because the fields that are considered “required” change over time doesn’t mean required fields don’t exist. There still needs to be code that enforces this requirement and I’d rather not write this code, to be honest. Therefore, I think the best way of handling required fields without writing a bunch of null checks everywhere is by using protovalidate or a similar library that has protobuf options that allow you to annotate which fields are required. Then there is code on the server and/or client that can enforce these requirements using a library. In my opinion, this has the best of both worlds: you can still declare required fields in a way that doesn’t completely break message integrity.

I don’t like this:

message User {
  int32 age = 1; // required.
}

I do like this:

message User {
  int32 age = 1 [(buf.validate.field).required = true];
}

I’m a big fan of protovalidate and I’ve used it a good amount and have contributed to it. Generally, I think custom options for protobuf fields is an untapped superpower of protobufs.

Failure to Launch

While gRPC has undeniable advantages, its learning curve can be steep. Getting started with protobuf, understanding the tooling, and setting up the necessary infrastructure can be intimidating for newcomers, making the initial adoption hurdle higher than with simpler JSON-based APIs. Why is it steep? Well, it introduces non-idiomatic tooling to most languages. There are some examples of language support that make protobuf generation seamless. Grpc.Tools for .NET is one shining example, showing how protobuf tooling can be more integrated into standard language tooling. We need more of this.

The steep learning curve doesn’t help when many people who use and rely on protobuf and gRPC actively don’t want gRPC to extend to the frontend and think that pushing in this direction will lead to uninformed people encroaching on the domain of the backend, where only they are smart enough to work. This is elitist gate-keeping and is unfortunately prevalent in this industry. I believe gRPC has as much of a place in web frontends as much as it does in microservices.

I’ve learned a lot by helping others work with protobuf. You may see me on Buf’s slack channel or on related discussions because I truly have gotten a lot out of it. Many article ideas have come directly from answering questions there. If I see a problem often enough, I may end up writing an article about it. I believe the protobuf and gRPC community needs more of this attitude.

I believe the steep learning curve (which can be helped with tooling), coupled with some resistance from backend developers (which can be helped by… having empathy?), has slowed its broader adoption in web development.

gRPC Has a History

gRPC’s initial focus on microservices and its close ties to HTTP/2 hindered its widespread adoption in web development. Even with the advent of gRPC-Web, there’s still a perception that it’s not a first-class citizen in the frontend ecosystem. The lack of robust integration with popular frontend libraries like TanStack Query further solidifies this notion to me.

I think there’s a real chance to get more frontend developers excited about gRPC with improved tooling. There’s a giant industry-wide conversation happening right now around where the line between “frontend” and “backend” meet and I think no matter the outcome, we’re going to see more typescript code using gRPC.

The “g” in gRPC

While the gRPC project claims that the “g” in gRPC is a backronym that stands for “gRPC”, it originally stood for Google, because it was Google who developed and released both protobuf and gRPC.

There’s always a lingering question about Google’s long-term commitment to gRPC and protobuf. Will they continue to invest in these open-source projects, or could they pull the plug if priorities shift? Remember that Google has recently layed off much of the Flutter, Dart and Python teams. The protobuf community is growing, but would it be self-sustaining enough to survive such a scenario?

It’s Not Finished

Others have said that gRPC is immature, not because of its age but by how developed the ecosystem is. I tend to agree, because it’s missing features and tools that I would have expected from a mature ecosystem.

The missing package manager

Sharing protobuf definitions across multiple projects or repositories is a constant struggle without specialized tools. While solutions like Bazel, Pants, and Buf’s BSR exist, my experience with protobuf “in the real world” is… mixed. There are prominent open source projects, some by Google, that have bash scripts scrapped together to download dependencies before evoking protoc manually. Just imagine a programming language with no solution for managing dependencies. That’s insane. I think both Bazel and Buf tooling solve this problem pretty well but I’m just frustrated that every repo I come across that uses protobuf solves the problem in the most bespoke way possible. The community needs to come together to improve this. There is an open-source repo called Buffrs that appears to be tackling this problem. I haven’t used it personally but it looks decent so far.

Related to dependencies, I do want to call out that Google’s “well-known” protobuf types get special privilege of being built into protoc. While these types are incredibly useful and invaluable, their privilege makes it hard for other libraries of useful protobuf types to exist and thrive. Just building these protobuf definitions into protoc (and other tooling) is a cop out for not having a real and consistent story for dependency management.

Editor Support

Editor integration for protobuf code generation leaves a lot to be desired. It would be immensely helpful if editors could intelligently link generated code back to its protobuf source. This would provide a more seamless experience, but the tooling just isn’t smart enough yet. Also, I think everyone needs to run with Buf’s editor support. Having a linter and autoformatter built into your editor is the expected from developers nowadays. And with protobuf, there are extremely real reasons to follow the advice of the linter.

Projects like tRPC showcase the benefits of tight integration and opinionated design choices—something that protobuf, by its nature, can’t fully replicate. However, I remain hopeful that the protobuf ecosystem can evolve to offer a similarly streamlined developer experience.

Ugly Documentation

I’ve never seen documentation generated from protobuf that wasn’t super ugly. I think since gRPC has historically been a backend service, the backend devs never bothered to put any real effort into making pretty documentation output using a protoc plugin. I’ve solved this problem by making a protoc plugin that generates OpenAPI from given protobuf files. Then I use one of the many beautiful tools for displaying the OpenAPI spec. This was, by far, much easier than getting me to make a decent design. Another side benefit for generating OpenAPI from protobuf is the ability to tap into that ecosystem since there’s more to it than just documentation.

Let’s look at a real example. This is a document generated using one of the few tools for generating documentation from protobuf, protoc-gen-doc:

Compare it to some of the OpenAPI tooling. This was generated using Elements, but there are many, many other alternatives that look equally as polished:

It’s kind-of not fair to point at a single plugin and say that the default template doesn’t look as good as OpenAPI alternatives, because you actually do have more flexibility with protoc-gen-doc. It allows you to specify your own template so it could look as beautiful as you want. However, this does line up with my point: the tooling is more finished and polished in the REST world than gRPC. This is a fixable problem, but we need to get frontend devs and designers excited about gRPC or backend engineers need to start sharpening their design skills.

I also want to note that OpenAPI/Swagger interfaces often have a way to test endpoints directly from the documentation website. This is completely missing from equivalent tools in the gRPC world. Additionally, with most OpenAPI documentation tools you can clearly see which fields are required and will display constraints on fields that have them. So not only is it prettier, it’s more functional as well.

Conclusion

gRPC, while a powerful tool in many ways, still has room to grow. The less-than-ideal aspects of generated code, coupled with the challenges of dependency management and evolving protobuf schemas, can create friction for developers. The lack of intuitive editor integration and the historical focus on backend services have also hindered its wider adoption in web development.

However, I think the future of gRPC is bright and can be far less ugly. The community is actively addressing these challenges, developing tools like the buf CLI, protovalidate and protoc-gen-connect-openapi to bridge the gaps and enhance the developer experience. As gRPC matures and its ecosystem expands, we can anticipate improved tooling, better editor support, and a smoother integration into the frontend world.

Traditional Workflow

Here’s what a typical workflow can look like when working with protobufs:

1. Define the Protobuf File: You create a .proto file, defining your message structures, enums, and services (if using gRPC).
2. Compile with protoc: You use the protoc compiler to generate code in your desired languages (e.g., Go, Java, Python, Rust, etc.) from your .proto files.
3. Implement and Use: You implement the server-side logic (if applicable) using the generated server stubs and utilize the generated client code to interact with your protobuf-based services.

Let’s visualize this traditional workflow to highlight these pain points:

What’s Missing?

While this workflow gets the job done, it leaves a lot of room for improvement:

• Error-Prone Build Steps: The process often involves manual compilation and code generation, increasing the risk of human errors. Most people introduce a Makefile or a bash script to handle calling protoc, but even this can be error-prone and difficult to maintain. Some believe this doesn’t exist or some magical build system “handles it for you”, but I assure you, it exists and it’s not hard to find plenty of examples of this.
• This workflow doesn’t include a standardized way to handle protobuf dependencies. Traditionally, this involves the Makefile or bash script that I just mentioned to download protobufs from various online repositories or manually copying them between projects. This is bad, but the Protobuf/gRPC projects don’t really give much guidance on how to reuse protobufs, so people have just done it themselves in wildly different ways.
• Lack of Consistency: Without proper tooling, it’s challenging to maintain consistent formatting and styling across multiple .proto files.
• Breaking Changes: Manually tracking changes to your protobufs and ensuring backward compatibility can be tedious and error-prone.
• No Mocking or Prototyping: It takes a good amount of effort to make a gRPC service, even if you just want to use mock data.

Fortunately, the Protobuf ecosystem has evolved dramatically in the last few years, and most of the advancement is from third-parties (not Google), which is exciting. So let’s cover parts of the modern workflow that you won’t see in a gRPC or protobuf tutorial.

The next generation of protobuf tooling

The Protobuf ecosystem has seen a surge of innovation in recent years, with many third-party tools emerging to address the limitations of the traditional workflow. Let’s delve into some of these tools.

JSON to Proto

JSON to Proto is a online tool that simplifies the creation of protobuf definitions if you are brand new to protobufs. I wouldn’t recommend using this for the long haul, but it can be a good way to get started quickly by pasting in sample JSON data, and the tool will generate a corresponding .proto file for you. This is particularly useful when you’re starting with existing JSON data or want to quickly prototype a protobuf schema. Learn more from the Github repo.

Protobuf Pal

Protobuf Pal is a browser-based protobuf editor designed to streamline the creation and editing of .proto files. It offers features like syntax highlighting, error checking, and auto-completion, making it easier to write valid protobuf definitions.

Buf CLI

Buf is a comprehensive toolkit designed to make working with protobufs a breeze.

The Buf CLI offers several powerful features:

• buf generate: A replacement for protoc that provides consistent code generation across different languages and environments. Read more here.
• buf lint: A linter that helps you maintain clean and consistent protobuf definitions, catching potential issues early. Read more here.
• buf format: An opinionated formatter to ensure consistent styling across your .proto files. Read more here.
• buf curl: is useful not only for development but also for testing. You can use it to send gRPC requests from your terminal, making it easy to verify the behavior of your services. It can craft requests using server reflection, local protobuf files, descriptor files or from a reference to the buf.build registry. Read more here.

$ buf curl --list-methods https://demo.connectrpc.com
connectrpc.eliza.v1.ElizaService/Converse
connectrpc.eliza.v1.ElizaService/Introduce
connectrpc.eliza.v1.ElizaService/Say
$ buf curl -d '{"sentence":"Hello! I need some help, doc"}' https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say
{
  "sentence": "Hello...I'm glad you could drop by today."
}

You can use the buf CLI directly on the CLI (which is great when you’re playing around or adding it to a continuous integration process) or integrated into your code editor, which is where the real magic happens, anyway.

Buf Schema Registry (BSR)

The Buf Schema Registry (BSR) is the missing package manager for Protobufs, but it does a bit more than that. Not only does it allow you to push versioned schemas to one place, but it also has cool features like automatically generated SDKs. The idea is that you can just import the package in your favorite language and magically you have your server stubs and clients in the language of your choice (as long as it’s Go, Typescript/Javascript, Java/Kotlin, Swift, Python or Rust).

With the BSR, you no longer need bespoke Makefile or bash scripts to pull down protobuf dependencies.

Testing

This next set of tools are good for testing live gRPC endpoints. Obviously, gRPCurl is an amazing tool for this, but let’s discover some tools that are a bit newer to the scene.

Buf Studio

Buf Studio is an interactive web UI for all your gRPC and Protobuf services stored on the Buf Schema Registry. With Buf Studio you can craft gRPC/gRPC-Web/Connect requests using images on the buf registry. Buf Studio uses those protobuf schemas to support autocompletion of these requests, which is super cool. It can also use an agent that is built into the Buf CLI to proxy requests from internal networks, making this web-based tool a bit more flexible.

Postman

Postman, a popular API testing tool, now supports gRPC. You can leverage its familiar interface to construct and send gRPC requests, making it a convenient option for testing your protobuf services.

Insomnia

Insomnia is another API testing platform that has added gRPC support. Similar to Postman, it allows you to design and execute gRPC requests within its user-friendly environment.

Insomnia’s UI takes a little getting used to but once gRPC is set up, it gets easier.

k6

k6 is a powerful load testing tool that can be used to simulate heavy traffic on your gRPC services. It helps you identify performance bottlenecks and ensure your services can handle real-world loads.

import { Client, StatusOK } from 'k6/net/grpc';
import { check, sleep } from 'k6';

const client = new Client();
client.load(['definitions'], 'eliza.proto');

export default () => {
  client.connect('127.0.0.1:10000', {});

  const data = { sentence: 'Hello, doc!' };
  const response = client.invoke('connectrpc.eliza.v1.ElizaService/Say', data);

  check(response, {
    'status is OK': (r) => r && r.status === StatusOK,
  });

  console.log(JSON.stringify(response.message));

  client.close();
  sleep(1);
};

FauxRPC

I couldn’t write this article without promoting my own tool, FauxRPC. FauxRPC is a tool that enables developers to quickly generate mock gRPC servers from protobuf definitions, facilitating early API development and testing. By incorporating FauxRPC into your workflow, you can easily create realistic mock services that simulate real gRPC server behavior, allowing you to test your client implementations and identify potential issues without the need for a fully implemented backend. This streamlined prototyping and testing process ultimately fosters faster iteration and more robust API development.

With a single command, you can have a server running with fake data!

$ buf build buf.build/connectrpc/eliza -o eliza.binpb
$ fauxrpc run --schema=eliza.binpb
FauxRPC (0.0.16 (97e4c8caf9d3c22387a393180e00bce40b2834c6) @ 2024-08-22T18:42:56Z; go1.22.4)
Listening on http://127.0.0.1:6660
OpenAPI documentation: http://127.0.0.1:6660/fauxrpc.openapi.html

Example Commands:
$ buf curl --http2-prior-knowledge http://127.0.0.1:6660 --list-methods
$ buf curl --http2-prior-knowledge http://127.0.0.1:6660/[METHOD_NAME]

Learn more about it in my previous post announcing it or the documentation website.

New Workflow

This enhanced workflow empowers developers to iterate faster, catch errors earlier, and ensure API stability, ultimately leading to more robust and maintainable protobuf-based applications. Let’s explore how these tools fit into the enhanced workflow:

Note that this workflow has one more “find issues and iterate” connections right after writing the protobuf file. That’s because these new tools will help you find issues with your protobuf schemas earlier in a more automated way.

Also note that with FauxRPC, frontend developers (or whatever is using the generated clients for the service) can start working on their part before the backend developer is finished with their work. Frontend devs no longer have to come up with mock APIs (which rapidly get outdated with reality) just to get started on their frontend work. Integration work on another service which uses this protobuf can happen before the backend implementation is completed. Everyone can work in parallel and the better the schema is (with protovalidate constraints) the better the fake data will be.

More ways to start making protobuf files

Developers can still define services, messages, and enums directly in the .proto file. However, tools like JSON-to-Proto and Protobuf Pal provide visual aids and assistance in creating and editing .proto files, reducing errors and improving productivity. I wouldn’t recommend these tools all the time, but they are probably good for quickly getting a prototype going quickly.

Automated Code Generation and Management

Buf’s buf generate command streamlines the code generation process with a set of declarative configuration files, ensuring consistency across different languages and platforms. Buf’s dependency management capabilities eliminate the need for manual scripts to handle protobuf dependencies.

Enhanced Quality Control

• Buf’s buf lint enforces coding standards and best practices, catching potential issues early in development.
• Buf’s buf format automatically formats your .proto files, ensuring consistency and readability.
• Buf’s buf breaking change detection helps you avoid introducing changes that could disrupt existing clients.

All of these put together mean that issues in your API schema are discovered automatically, as soon as possible.

Streamlined Testing and Prototyping

Buf’s buf curl and Buf Studio’s built-in gRPC client enable quick testing and interaction with your services. FauxRPC facilitates rapid prototyping by allowing you to mock gRPC services without implementing the actual server-side logic.

Conclusion

The traditional protobuf workflow, while functional, can involve manual steps and potential pitfalls. By incorporating tools like the Buf CLI, JSON-to-Proto, Protobuf Pal, and FauxRPC into your development process, you can significantly enhance your productivity and ensure the quality of your protobuf definitions. Note that I haven’t come close to outlining all of the different tools that you can use with Protobuf. So, don’t hesitate to explore these tools and discover how they can transform your Protobuf development experience!

Why FauxRPC?

• Faster Development & Testing: Work independently without relying on fully functional backend services.
• Isolation & Control: Test frontend components in isolation with controlled fake data.
• Multi-Protocol Support: Supports multiple protocols (gRPC, gRPC-Web, Connect, and REST).
• Prototyping & Demos: Create prototypes and demos quickly without building the full backend. Fake it till you make it.
• Improved Collaboration: Bridge the gap between frontend and backend teams.
• Plays well with others: Test data from FauxRPC will try to automatically follow any protovalidate constraints that are defined.

How it Works

FauxRPC leverages your Protobuf definitions to generate fake services that mimic the behavior of real ones. You can easily configure the fake data returned, allowing you to simulate various scenarios and edge cases. It takes in *.proto files or protobuf descriptors (in binpb, json, txtpb, yaml formats), then it automatically starts up a server that can speak gRPC/gRPC-Web/Connect and REST (as long as there are google.api.http annotations defined). Descriptors contain all of the information found in a set of .proto files. You can generate them with protoc or the buf build command.

Get Started

FauxRPC is available as an open-source project. Check out the documentation and examples to get started. Here’s a quick overview, but be sure to check the official documentation for the most up-to-date instructions:

Install via source

go install github.com/sudorandom/fauxrpc/cmd/fauxrpc@latest

Pre-built binaries

Binaries are built for several platforms for each release. See the latest ones on the releases page.

Use Descriptors

Make an example.proto file (or use a file that already exists):

syntax = "proto3";

package greet.v1;

message GreetRequest {
  string name = 1;
}

message GreetResponse {
  string greeting = 1;
}

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse) {}
}

Create a descriptors file and use it to start the FauxRPC server:

$ buf build ./example.proto -o ./example.binpb
$ fauxrpc run --schema=./example.binpb
2024/08/17 08:01:19 INFO Listening on http://127.0.0.1:6660
2024/08/17 08:01:19 INFO See available methods: buf curl --http2-prior-knowledge http://127.0.0.1:6660 --list-methods

Done! It’s that easy. Now you can call the service with any tooling that supports gRPC, gRPC-Web, or connect. So buf curl, grpcurl, Postman, Insomnia all work fine!

$ buf curl --http2-prior-knowledge http://127.0.0.1:6660/greet.v1.GreetService/Greet
{
  "greeting": "dream"
}

Server Reflection

If there’s an existing gRPC service running that you want to emulate, you can use server reflection to start the FauxRPC service:

$ fauxrpc run --schema=https://demo.connectrpc.com

From BSR (Buf Schema Registry)

Buf has a schema registry where many schemas are hosted. Here’s how to use FauxRPC using images from the registry.

$ buf build buf.build/bufbuild/registry -o bufbuild.registry.json
$ fauxrpc run --schema=./bufbuild.registry.json

Multiple Sources

You can define this --schema option as many times as you want. That means you can add services from multiple descriptors and even mix and match from descriptors and from server reflection:

$ fauxrpc run --schema=https://demo.connectrpc.com --schema=./example.binpb

Multi-protocol Support

The multi-protocol support is based on ConnectRPC. So with FauxRPC, you get gRPC, gRPC-Web and Connect out of the box. However, FauxRPC does one thing more. It allows you to use google.api.http annotations to present a JSON/HTTP API, so you can gRPC and REST together! This is normally done with an additional service that runs in-between the outside world and your actual gRPC service but with FauxRPC you get the so-called transcoding from HTTP/JSON to gRPC all in the same package. Here’s a concrete example:

syntax = "proto3";

package http.service;

import "google/api/annotations.proto";

service HTTPService {
  rpc GetMessage(GetMessageRequest) returns (Message) {
    option (google.api.http) = {get: "/v1/{name=messages/*}"};
  }
}
message GetMessageRequest {
  string name = 1; // Mapped to URL path.
}
message Message {
  string text = 1; // The resource content.
}

Again, we start the service by building the descriptors and using

$ buf build ./httpservice.proto -o ./httpservice.binpb
$ fauxrpc run --schema=httpservice.binpb

Now that we have the server running we can test this with the “normal” curl:

$ curl http://127.0.0.1:6660/v1/messages/123456
{"text":"Retro."}⏎

Sweet. You can now easily support REST alongside gRPC. If you are wondering how to do this with “real” services, look into vanguard-go. This library is doing the real heavy lifting.

What does the fake data look like?

You might be wondering what actual responses look like. FauxRPC’s fake data generation is continually improving so these details might change as time goes on. It uses a library called fakeit to generate fake data. Because protobufs have pretty well-defined types, we can easily generate data that technically matches the types. This works well for most use cases, but FauxRPC tries to be a little bit better. If you annotate your protobuf files with protovalidate constraints, FauxRPC will try its best to generate data that matches these constraints. Let’s look at some examples!

syntax = "proto3";

package greet.v1;

message GreetRequest {
  string name = 1;
}

message GreetResponse {
  string greeting = 1;
}

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse) {}
}

With FauxRPC, you will get any kind of word, so it might look like this:

{
  "greeting": "sufficient"
}

This is fine, but for the RPC, we know a bit more about the type being returned. We know that it sends a greeting back that looks like “Hello, [name]”. So here’s what the same protobuf file might look like with protovalidate constraints:

Now let’s see what this looks like with protovalidate constraints:

syntax = "proto3";

import "buf/validate/validate.proto";

package greet.v1;

message GreetRequest {
  string name = 1 [(buf.validate.field).string = {min_len: 3, max_len: 100}];
}

message GreetResponse {
  string greeting = 1 [(buf.validate.field).string.pattern = "^Hello, [a-zA-Z]+$"];
}

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse) {}
}

With this new protobuf file, this is what FauxRPC might output now:

{
  "greeting": "Hello, TWXxF"
}

This shows how protovalidate constraints enable FauxRPC to generate more realistic and contextually relevant fake data, aligning it closer to the expected behavior of your actual services. As another example, I will show one of Buf’s services used to manage users, buf.registry.owner.v1.UserService. Here’s what the UserRef message looks like:

message UserRef {
  option (buf.registry.priv.extension.v1beta1.message).request_only = true;
  oneof value {
    option (buf.validate.oneof).required = true;
    // The id of the User.
    string id = 1 [(buf.validate.field).string.tuuid = true];
    // The name of the User.
    string name = 2 [(buf.validate.field).string = {
      min_len: 2
      max_len: 32
      pattern: "^[a-z][a-z0-9-]*[a-z0-9]$"
    }];
  }
}

So let’s make our descriptors for this service, start the FauxRPC server and make our example request:

$ buf build buf.build/bufbuild/registry -o bufbuild.registry.binpb
$ fauxrpc run --schema=./bufbuild.registry.binpb
$ buf curl --http2-prior-knowledge http://127.0.0.1:6660/buf.registry.owner.v1.UserService/ListUsers
{
  "nextPageToken": "Food truck.",
  "users": [
    {
      "id": "c4468393f926400d8880a264df9c284a",
      "createTime": "2012-03-06T12:15:03.239463070Z",
      "updateTime": "1990-10-29T13:12:31.224347086Z",
      "name": "jexox",
      "type": "USER_TYPE_STANDARD",
      "description": "Tattooed taxidermy.",
      "url": "http://www.productexploit.name/synergies/target"
    },
    {
      "id": "0e4ca24f4ff54761b109daab0da1bea2",
      "createTime": "1955-05-16T02:37:30.643378679Z",
      "updateTime": "1923-08-28T04:28:43.330711919Z",
      "name": "ya0",
      "type": "USER_TYPE_STANDARD",
      "state": "USER_STATE_INACTIVE",
      "description": "Helvetica.",
      "url": "https://www.centralengage.info/markets/scale/e-commerce/exploit",
      "verificationStatus": "USER_VERIFICATION_STATUS_UNVERIFIED"
    }
  ]
}

Hopefully, this gives you a good idea of what the output might look like. The better your validation rules, the better the FauxRPC data will be.

What’s left?

FauxRPC is already great for some use cases but it’s not “done” as there’s more to do to make it better. I have plans to add the ability to configure stubs for each RPC method. This will allow you to define specific responses or behaviors for each RPC, giving you more control over the simulated service. I hope this will make it easier to iterate on protobuf designs without needing to actually implement services until later.

Stay Tuned

I made a documentation website to organize documentation. I think it looks pretty good for how quickly I threw it together. The code for FauxRPC lives on GitHub at sudorandom/fauxrpc. It’s a little thin now but there’s a lot that I can write about in there. I’m actively developing FauxRPC and have many exciting features planned for the future. This is early on for this project but it has come together as a coherent and useful program for me extremely quickly. So please try it out and let me know your feedback and suggestions. Stay tuned for updates!

… and don’t forget to star the repo on GitHub. It helps more than you know!

At the time of writing, HTTP/3 is supported by 30.4% of the top 10 million websites. This market penetration is astounding, but it seems like all of this progress has been possible almost exclusively by work on browsers, load balancers and CDN providers. What about the backend? How’s HTTP/3 doing there? The answer, sadly, is not as incredible.

Because of this, I have been very interested in HTTP/3 in the context of gRPC. While gRPC has been instrumental in driving the adoption of HTTP/2, the same isn’t true for HTTP/3 even though HTTP/3 promises several benefits that all seem to apply exceptionally well to gRPC services.

In this post, we’ll dive into what HTTP/3 is and explore the compelling reasons why it’s an ideal fit for gRPC applications. We’ll uncover the technical advancements that promise to make HTTP/3 faster, more reliable, and more secure. But we won’t stop at theory; we’ll get our hands dirty with practical examples in Go, demonstrating how to set up and test gRPC servers and clients over HTTP/3.

By the end of this journey, you’ll have a solid understanding of the benefits HTTP/3 brings to gRPC, the tools available to start using it today, and the potential it holds for the future of API development. So, fasten your seatbelts and get ready to experience the next generation of network protocols!

All code examples are available at sudorandom/example-connect-http3 on GitHub.

Why HTTP/3

gRPC has had a lot of success pushing the world into HTTP/2 but there are some advantages to pushing even further and adopting the new major version of HTTP, HTTP/3. Let’s discuss these advantages before delving into code examples. So why should we use HTTP/3?

Faster Connection Establishment

HTTP/3 is built on QUIC (Quick UDP Internet Connections). You can think of it as a similar protocol to TCP but as the name suggests, it is built on top of UDP. While TCP has historically been the foundation of web communication, offering sequential packet ordering, congestion control and retransmissions, the needs of web browsers have evolved to require some different features and tradeoffs. All three main features of TCP—sequential packet ordering, congestion control, and retransmissions—can sometimes hinder the performance of modern web applications. Acknowledging these trade-offs led to the development of QUIC and HTTP/3, which both leverage UDP.

As the internet evolved better security practices, we’ve layered TLS on top of TCP, so a TCP connection gets established first and then an HTTP/gRPC/etc. request can be made. This layering made it much easier to slowly adopt TLS but it has also made it slower to establish connections that need TLS because it added more round trips between the server and client before a connection can be “established”. Each round trip introduces latency, delaying the time it takes for a request to reach the server and receive a response. Here’s what it looks like with HTTP/1.1 and HTTP/2:

sequenceDiagram
    actor Client

    rect rgb(47,75,124)
        Client ->> Server: TCP SYN
        Server ->> Client: TCP SYN-ACK
        Client ->> Server: TCP ACK
    end

    rect rgb(102,81,145)
        Client ->> Server: TLS Client Hello
        Server ->> Client: TLS Server Hello
        Client ->> Server: TLS Server Finished
        Server ->> Client: TLS Client Finished
    end

    rect rgb(200,80,96)
        Client ->> Server: HTTP Request
        Server ->> Client: HTTP Response
    end
  

Yes, this process involves three round trips before the client even sends a request. Viewed in this way, this seems shockingly slow to me. By combining TLS 1.3, QUIC and HTTP/3, we can do away with several of those round trips. So with HTTP/3 it typically looks like this with only a single round trip before we can issue our request:

sequenceDiagram
	    actor Client

    rect rgb(47,75,124)
        Client ->> Server: QUIC
        Server ->> Client: QUIC
        Client ->> Server: QUIC
    end

    rect rgb(200,80,96)
        Client ->> Server: HTTP Request
        Server ->> Client: HTTP Response
    end
  

But wait, we can actually go further. With a feature called 0-RTT, the number of round trips for a new connection can actually be ZERO. The idea is that the client can presume that a connection will be established and send the request based on that presumption. 0-RTT quite literally removes the need to do a round trip before making the first request. It looks like this:

sequenceDiagram
    actor Client

    rect rgb(47,75,124)
        Client ->> Server: QUIC
    end
    rect rgb(200,80,96)
        Client ->> Server: HTTP Request
    end

    rect rgb(47,75,124)
        Server ->> Client: QUIC
    end
    rect rgb(200,80,96)
        Server ->> Client: HTTP Response
    end
  

Note that 0-RTT requires the client to have some cached information about the server so it’s not always possible to use. Also, there are still some security concerns, mostly focused on the potential for denial of service attacks.

Head-of-line Blocking

HTTP/2 has allowed gRPC to be quite good at multiplexing multiple requests and streams onto a single connection. This wasn’t possible with HTTP/1.1. However, there is an issue that can arise due to TCP’s guarantee of delivering packets in order even if they arrive out of order. This is an issue because packets for a request could be waiting for retransmissions from another request that is using the same connection. From gRPC’s point of view, these requests are independent things, so there’s no need to wait but TCP does not know about these separate streams, so we end up experiencing a so-called head-of-line blocking issue.

HTTP/3 solves the head-of-line blocking issue by avoiding TCP altogether. Instead, it is built on top of a protocol called QUIC which is built on top of UDP. QUIC is aware of multiple streams so it knows when it is appropriate to deliver packets without having this head-of-line blocking behavior. This makes HTTP/3 much better when dealing with unreliable networks. With its heavy use of streams, gRPC in particular would greatly benefit from the elimination of the head-of-line blocking issue.

Encryption is Required

When creating HTTP/2 there was a lot of disagreement over if it should require TLS. Many have argued that this requirement would hurt the adoption of HTTP/2. Further, they argue that HTTP/2 offers a lot of benefits that can be attained without the use of TLS. As a result, we now have h2c, which is a way to use HTTP/2 without encryption.

These arguments might have worked for HTTP/2 but they didn’t hold up for HTTP/3. HTTP/3 requires encryption and usage of the TLS 1.3 protocol. There are several reasons for this, but the most compelling one is that TLS 1.3 working with QUIC allows for fewer (and sometimes zero) round trips to negotiate a new connection before making a request, which is even faster than HTTP/1.1 without TLS. It’s a double win by being both more secure and, theoretically, much faster.

Experimentation

When researching this topic, I discovered that HTTP/3 with gRPC is a bit of a complicated story. I covered this a bit in my gRPC: The Good Parts post but the gist is that no decision has been made to officially support HTTP/3 throughout the entire gRPC ecosystem. There is an open issue to discuss this and there is an official proposal also discussing the idea. Some projects are closer to HTTP/3 than others, so here are the implementations that I have found where you can make gRPC over HTTP/3 work:

• C#: grpc-dotnet is the pioneer here, already containing an implementation of an HTTP/3 transport for gRPC.
• Rust: Tonic with the Hyper transport appears to be able to support this, although I’m not sure if there are good examples of this in the wild yet.
• Go: ConnectRPC for Go uses the standard library http.Handlers, so any http server implementation can be used, including the transport available in quic-go.
• Cronet: Cronet, designed primarily for mobile clients (Android and iOS), provides a way to utilize Chrome’s network stack, including its QUIC and HTTP/3 support. This can be particularly useful for building gRPC clients with HTTP/3 capabilities in mobile environments.

If you know about other gRPC implementations that can work with HTTP/3 (client or server), let me know. These are only the ones I know about.

When learning something new I always like getting my hands dirty by using it. I feel like this is the best way to learn. Since Go is my current working language, I decided to explore HTTP/3 using ConnectRPC in Go. I’m including the full examples in this article because I’ve looked and haven’t really been able to find good examples for this. I hope it’s helpful for others. The full working examples are on a git repo that I made for this post. Examples will have things like imports omitted for brevity but the full source is linked under each example.

Example server in Go

Let’s explore how easy it is to set up a gRPC server with HTTP/3 support using Go. The key players here are:

• quic-go: This library provides a robust implementation of the QUIC protocol in Go, enabling HTTP/3 communication.
• ConnectRPC: connect-go, the Go implementation of ConnectRPC allows us to define gRPC services using familiar Go HTTP handlers (http.Handler), which greatly simplifies the integration process.

I am using the Eliza service as the service to implement with ConnectRPC. This is a simple service intended to help demonstrate ConnectRPC. The implementation is un-important, so I omitted it but my implementation just echos back whatever the user sends.

func main() {
	mux := http.NewServeMux()
	// Implementation is only in the full source
	mux.Handle(elizav1connect.NewElizaServiceHandler(&server{}))

	addr := "127.0.0.1:6660"
	log.Printf("Starting connectrpc on %s", addr)
	h3srv := http3.Server{
		Addr:    addr,
		Handler: mux,
	}
	if err := h3srv.ListenAndServeTLS("cert.crt", "cert.key"); err != nil {
		log.Fatalf("error: %s", err)
	}
}

That’s it! With this minimal setup, we now have a gRPC server that supports HTTP/3! I just used an http3.Server instance instead of http.Server. http3.Server does have different options related to the differences between QUIC and TCP. Note that it is possible to run an http.Server alongside http3.Server using the same address with the same port. How is this possible? Because HTTP/3 uses UDP and the ports are completely separate from TCP ports. Here’s an example:

func main() {
	mux := http.NewServeMux()
	mux.Handle(elizav1connect.NewElizaServiceHandler(&server{}))

	addr := "127.0.0.1:6660"
	log.Printf("Starting connectrpc on %s", addr)
	h3srv := http3.Server{
		Addr:    addr,
		Handler: mux,
	}

	srv := http.Server{
		Addr:    addr,
		Handler: h2c.NewHandler(mux, &http2.Server{}),
	}

	eg, _ := errgroup.WithContext(context.Background())
	eg.Go(func() error {
		return h3srv.ListenAndServeTLS("cert.crt", "cert.key")
	})
	eg.Go(func() error {
		return srv.ListenAndServeTLS("cert.crt", "cert.key")
	})
	if err := eg.Wait(); err != nil {
		log.Fatalf("error: %s", err)
	}
}

This code demonstrates running multiple HTTP servers concurrently, providing support for HTTP/1.1, HTTP/2, and HTTP/3 on the same port by leveraging TCP for HTTP/1.1 and HTTP/2 and UDP for HTTP/3.

Oh, and note that this requires a certificate and a key because HTTP/3 requires TLS. Here’s the command that I used to create a self-signed cert for testing:

openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 \
    -subj "/C=DK/L=Copenhagen/O=kmcd/CN=local.kmcd.dev" \
    -keyout cert.key  -out cert.crt

That’s it! With just a few lines of code, we now have a gRPC server that supports HTTP/3!

Testing with an example client in Go

Now that we have a server running we need some way to test it, so let’s use the standard library http.Client with some quic-go magic sprinkled on top:

const (
	url        = "https://127.0.0.1:6660/connectrpc.eliza.v1.ElizaService/Say"
	reqBody = `{"sentence": "Hello World!"}`
)

func main() {
	roundTripper := &http3.RoundTripper{
		TLSClientConfig: &tls.Config{
			// we need this because our certificate is self signed
			InsecureSkipVerify: true,
		},
	}
	defer roundTripper.Close()
	client := &http.Client{
		Transport: roundTripper,
	}

	log.Println("connect: ", url)
	log.Println("send: ", reqBody)
	req, err := http.NewRequest("POST", url, strings.NewReader(reqBody))
	if err != nil {
		log.Fatalf("error: %s", err)
	}
	req.Header.Add("Content-Type", "application/json")

	resp, err := client.Do(req)
	if err != nil {
		log.Fatalf("error: %s", err)
	}

	body, err := io.ReadAll(resp.Body)
	if err != nil {
		log.Fatalf("error: %s", err)
	}
	defer resp.Body.Close()

	log.Println("recv: ", string(body))
}

In the case of the client, we only need to define a http3.RoundTripper instance and pass that into a completely normal http.Client instance. That’s… quite literally it. Everything else should be the same.

2024/07/06 12:57:24 connect:  https://127.0.0.1:6660/connectrpc.eliza.v1.ElizaService/Say
2024/07/06 12:57:24 send:  {"sentence": "Hello World!"}
2024/07/06 12:57:24 recv:  {"sentence":"Hello World!"}

This works as expected! In this next example, I’m calling the service using a client built with ConnectRPC, demonstrating how it can work seamlessly over HTTP/3 as well, just by configuring http.Client a little differently.

const url = "https://127.0.0.1:6660"
var reqBody = &elizav1.SayRequest{Sentence: "Hello World!"}

func main() {
	roundTripper := &http3.RoundTripper{
		TLSClientConfig: &tls.Config{
			InsecureSkipVerify: true,
		},
	}
	defer roundTripper.Close()
	client := &http.Client{
		Transport: roundTripper,
	}

	svcClient := elizav1connect.NewElizaServiceClient(client, url, connect.WithGRPC())

	log.Println("connect: ", url)
	log.Println("send: ", reqBody)
	resp, err := svcClient.Say(context.Background(), connect.NewRequest(reqBody))
	if err != nil {
		log.Fatalf("error: %s", err)
	}

	log.Println("recv: ", resp.Msg)
}

It works the same, but the output is a bit different since protobuf types print slightly differently from JSON:

2024/07/06 12:57:12 connect:  https://127.0.0.1:6660
2024/07/06 12:57:12 send:  sentence:"Hello World!"
2024/07/06 12:57:14 recv:  sentence:"Hello World!"

Now we have a server and a client that talks HTTP/3! Amazing!

Testing HTTP/3 servers with curl

I wanted to test this with a “real” HTTP/3 client because maybe there are some weird issues where quic-go servers only work against quic-go clients. To be extra sure, I wanted to validate this further with a well-established HTTP/3 client. So I reached for curl, the de-facto tool for calling HTTP APIs from the command line. So I immediately tried to just run curl --http3:

$ curl \
  --json '{"sentence": "Hello World!"}' \
  --http3 -k -v \
  https://127.0.0.1:6660/connectrpc.eliza.v1.ElizaService/Say
curl: option --http3: the installed libcurl version doesn't support this
curl: try 'curl --help' or 'curl --manual' for more information

Wait, what? What’s happening?? I thought curl supported HTTP/3! What gives? Well, the story isn’t that simple. The curl CLI might support HTTP/3 but the libraries that it uses also need to be on the correct version to make it all work. To get around that, I installed curl using Cloudflare’s homebrew formula which will install everything needed to get curl to work with HTTP/3. So let’s try it out:

$ curl \
  --json '{"sentence": "Hello World!"}' \
  --http3 -k -v \
  https://127.0.0.1:6660/connectrpc.eliza.v1.ElizaService/Say
*   Trying 127.0.0.1:6660...
* Server certificate:
*  subject: C=DK; L=Copenhagen; O=kmcd; CN=local.kmcd.dev
*  start date: Jul  6 06:14:48 2024 GMT
*  expire date: Jul  6 06:14:48 2025 GMT
*  issuer: C=DK; L=Copenhagen; O=kmcd; CN=local.kmcd.dev
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Connected to 127.0.0.1 (127.0.0.1) port 6660
* using HTTP/3
* [HTTP/3] [0] OPENED stream for https://127.0.0.1:6660/connectrpc.eliza.v1.ElizaService/Say
* [HTTP/3] [0] [:method: POST]
* [HTTP/3] [0] [:scheme: https]
* [HTTP/3] [0] [:authority: 127.0.0.1:6660]
* [HTTP/3] [0] [:path: /connectrpc.eliza.v1.ElizaService/Say]
* [HTTP/3] [0] [user-agent: curl/8.9.0-DEV]
* [HTTP/3] [0] [content-type: application/json]
* [HTTP/3] [0] [accept: application/json]
* [HTTP/3] [0] [content-length: 28]
> POST /connectrpc.eliza.v1.ElizaService/Say HTTP/3
> Host: 127.0.0.1:6660
> User-Agent: curl/8.9.0-DEV
> Content-Type: application/json
> Accept: application/json
> Content-Length: 28
>
* upload completely sent off: 28 bytes
< HTTP/3 200
< content-type: application/json
< accept-encoding: gzip
< date: Sat, 06 Jul 2024 13:17:12 GMT
< content-length: 27
<
* Connection #0 to host 127.0.0.1 left intact
{"sentence":"Hello World!"}

Success! You can see that HTTP/3 is being used by inspecting the verbose logging. This flexes our server with HTTP/3 but you may be wondering what this has to do with gRPC because I’m only using basic HTTP requests with JSON and you’re totally right. The previous examples only leverage one of the three protocols that ConnectRPC provides by default, the connect protocol. Thus far, I haven’t validated if the other two protocols, gRPC or gRPC-Web, really work using this transport. To test that, we’ll need more gRPC-centric tooling.

Adding HTTP/3 to the Buf CLI

I also wanted to test with gRPC-specific tooling to ensure gRPC and gRPC-Web worked as expected. So I added support for HTTP/3 with the buf CLI because it supports calling services using Connect, gRPC and gRPC-Web. This ended up being pretty easy and looks similar to the ConnectRPC example client above. I have an open PR here and if you want a sneak peak you can build the Buf CLI from my branch. Below, I show how I tested this new feature with ConnectRPC’s demo website:

$ buf curl \
    --http3 \
    --schema=buf.build/connectrpc/eliza \
    -d '{"sentence": "Hello world!"}' \
    https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say
{
  "sentence": "Hello...I'm glad you could drop by today."
}

Success! HTTP/3 works now! When I tested this, I was surprised that demo.connectrpc.com actually supported HTTP/3, but it does! By default, buf curl will use the connect protocol, so we didn’t test anything new. So let’s try gRPC-Web next:

$buf curl \
    --http3 \
    --protocol=grpcweb \
    --schema=buf.build/connectrpc/eliza \
    -d '{"sentence": "Hello world!"}' \
    https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say
{
  "sentence": "Hello, how are you feeling today?"
}

And now the classic gRPC:

$ buf curl \
    --http3 \
    --protocol=grpc \
    --schema=buf.build/connectrpc/eliza \
    -d '{"sentence": "Hello world!"}' \
    https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say
{
   "code": "internal",
   "message": "protocol error: no Grpc-Status trailer: unexpected EOF"
}

Wait, what? Why is it complaining about missing HTTP trailers? What gives?

HTTP trailers are a recurring issue for gRPC. HTTP trailers are a special type of HTTP header that is sent at the very end of a message body after all the data has been transmitted. They are useful for sending metadata that cannot be determined until the entire message is known, such as checksums, signatures, or other end-of-message signals. gRPC relies on trailers to return status codes and error details.

To explain why I am not receiving the HTTP trailers, I had to dig into quic-go’s HTTP/3 implementation.

Adding trailer support to quic-go

When I looked into quic-go, I discovered that it doesn’t support HTTP trailers yet. There is an issue and a related pull request. But the issue is four years old, the PR is two years old, and both are still open after all of this time. This seemed quite crazy to me at first, but, on reflection, I realized that gRPC is the most popular thing that currently uses HTTP trailers. However, grpc-go directly uses HTTP/2 support from golang.org/x/net/http2 instead of using net/http so it would require a good amount of work to get HTTP/3 support through quic-go. Therefore, this issue probably isn’t on the radar of anyone working on grpc-go. However, it is trivial (as seen above) to use quic-go with ConnectRPC, so I think this is a unique situation where progress can be made quickly.

So I ended up implementing trailer support for clients and I submitted my own PR. I don’t think it is quite ready yet but when I use my branch of quic-go with my branch of the buf CLI, gRPC actually works with HTTP/3!

$ buf curl \
    --http3 \
    --protocol=grpc \
    --schema=buf.build/connectrpc/eliza \
    -d '{"sentence": "Hello world!"}' \
    https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say
{
  "sentence": "Hello there...how are you today?"
}

There are a few issues that I have with my PR. I based it off of the earlier PR but many things have changed with the codebase that actually make it harder to implement this feature. However, it’s super encouraging that this code seems to work without too much fuss.

Experiment Results

My experimentation shows that while Go doesn’t yet have full, native support for gRPC over HTTP/3, there are practical workarounds available today. Both the gRPC-Web and Connect protocols function seamlessly over HTTP/3, and in fact, ConnectRPC may already be leveraging HTTP/3 in production environments where infrastructure allows. I discovered this firsthand with the ConnectRPC demo website, which will connect using HTTP/3 from the browser to the load balancer (after a few requests so the browser knows HTTP/3 is available).

Conclusion

In this post, we’ve explored the exciting potential of HTTP/3 to supercharge gRPC performance. We dove into the key advantages of HTTP/3, such as faster connection establishment, elimination of head-of-line blocking, and mandatory encryption. By getting our hands dirty with practical examples in Go, we’ve seen firsthand how HTTP/3 can be seamlessly integrated into gRPC services using tools like ConnectRPC and Buf.

While the gRPC ecosystem’s full adoption of HTTP/3 is in its early stages, the benefits are clear, and the tools are already available in some libraries and tools. As developers, we have the opportunity to push this technology forward and shape the future of high-performance, secure communication.

I encourage you to experiment with HTTP/3 and gRPC in your own projects. Explore different implementations, measure the performance gains, and don’t be afraid to dive into the code if you run into issues. Your active engagement with this evolving technology can directly contribute to the ongoing development of gRPC and HTTP/3. Although widespread adoption of HTTP/3 for gRPC on the backend is still in its early stages, if you have the flexibility to control both server and client components, or are working with browser-based clients, you might find compelling use cases for it even today.

I’d love to hear about your experiences with HTTP/3 and gRPC. Have you seen significant performance improvements? Or perhaps you’ve found that QUIC is slower without the kernel-level optimizations that TCP can take advantage of? What challenges have you encountered while experimenting with this exciting technology? Let’s share our experiences and findings because even though HTTP/3 is still finding its footing in this context, there’s a lot we can learn from each other.

Let’s dive into the key advantages that make gRPC a powerful tool for modern web development.

Performance

This one might be a little controversial, but Protocol Buffers is, indeed, faster than JSON and XML. This continues to be demonstrated over and over again. Protobuf is able to be faster for these reasons:

• Field names are not included in the message. Instead, protobuf uses numbers to distinguish fields. In most cases you’ll see field numbers take one or two bytes on the wire when it can be much more than that depending on your JSON field names.
• Protobuf’s VARINT type allows for small scale integers to take up a single byte, even if it’s an int64. Realistically we really don’t use that many large numbers so these savings can add up. Again, it’s much better than ASCII-encoded numbers being used for each digit.
• There’s no real winning with strings or byte arrays but compression is still supported with gRPC so at worst this aspect is even with HTTP/JSON.

I’ve personally seen 50% data transfer savings by switching to protobuf encoding with realistic payloads.

There are some haters of protobuf encoding, and that’s perfectly fine. The only “fatal flaw” that I’ve actually been annoyed with is “map values cannot be other maps.” It does seem like that should be possible, even when you consider the internal representation of a map:

map<key_type, value_type> map_field = N;

transforms into:

message MapFieldEntry {
  key_type key = 1;
  value_type value = 2;
}

repeated MapFieldEntry map_field = N;

It’s super frustrating because I don’t understand why value_type can’t be a map. The solution to this problem is just to make your own wrapper type to use as the value that contains a map. It is kind of annoying and this does come up semi-often. Crap, this was supposed to be a positive article. Let’s get back on track.

I think the protobuf encoding is better than JSON in many ways. However, I understand that sometimes you just want JSON, and with gRPC you absolutely can just use JSON. gRPC still has a few binary framing bytes before each message that won’t be human readable but if you’re really concerned with those check out the ConnectRPC section below.

Most gRPC implementations also let you define your own encoding, so it may possible to insert your own favorite encoding if you want to push the limits.

Strongly Typed Contracts

Say goodbye to the guesswork of loosely typed APIs. gRPC’s protobuf definitions create rock-solid contracts between client and server. This translates to:

• Fewer errors: Clear expectations for data types reduce the chance of mismatched data.
• Better code generation: Automatic generation of client and server code in various languages saves time and effort.
• Smoother development cycles: Consistent contracts make it easier to evolve your API without breaking existing clients.
• Generated Documentation: Automatic generation of documentation means that your documentation will never be out of sync with your API.

API contracts are very powerful. For more on this topic, I’ve written an article discussing API contracts called Building APIs with Contracts.

Streaming Support

Streaming support is arguably the best and most unique feature for gRPC. It does away with needing to frequently poll for updates in many scenarios which make it a good candidate for:

• Chat applications: Seamlessly handle messages flowing back and forth.
• Live updates: Push updates to clients as soon as they happen.
• Any scenario where constant communication is key: From gaming to financial data, gRPC’s streaming capabilities open up a world of possibilities.

If you come from the networking world, you might know that gNMI (which is based on gRPC) is the replacement for SNMP. Instead of polling network devices for the same data every minute, you can now use gNMI to subscribe to counters. I’ve written more about this in a post called Why you should use gNMI over SNMP in 2024.

Cross-Language Support

gRPC doesn’t care what programming language you prefer. Thanks to code generation tools, you can seamlessly work with gRPC in a wide range of languages, including:

• Go
• Rust
• Java
• Python
• C#
• Node.js
• Ruby
• …and many more!

This promotes flexibility, collaboration, and the ability to choose the right tool for the job. Currently, I believe gRPC has such a large momentum with language support that it’s hard for me to consider an alternative that doesn’t also speak gRPC. Many alternatives that have similar benefits to gRPC have mediocre support for a handful of languages at best.

Pioneered HTTP/2

gRPC was a driving force behind the adoption of HTTP/2, a major upgrade to the web’s underlying protocol. This means you get all the benefits of HTTP/2’s:

• Multiplexing: Multiple requests and responses can share a single connection, improving efficiency.
• Header compression: Smaller headers mean faster transmission.
• Overall performance improvements: HTTP/2 is simply a faster, more efficient way to communicate over the web.

HTTP/3

There’s some movement on HTTP/3 support for gRPC. There is an open proposal created by the dotnet gRPC library maintainers and there is an open issue to discuss actually adding HTTP/3 to the gRPC spec. Frustratingly, there hasn’t been a lot of movement on the official gRPC repo to add support directly to any of their implementations, but as you can see from the thread, there’s a lot of interest and a lot of people making prototypes that prove the concept.

This is likely an incomplete list but here are the packages that you can likely use HTTP/3 with today:

• The standard grpc library for C#, dotnet-grpc (ref)
• It may already be possible in rust with Tonic with the Hyper HTTP transport (ref)
• It’s possible in Go if you use ConnectRPC with quic-go - I don’t have a link for this, but I’ve tested this out myself. This is a topic for a future post!
• This is untested but I believe many gRPC-Web implementations in the browser might “just work” with HTTP/3 as well as long as the browsers are informed of the support via the ALT-SVC header and the servers support it.

As more servers and clients support HTTP/3 they should see faster connection establishment times, complete removal of the head-of-line blocking problem and much better recovery from packet loss. There’s a long way to go here, but there is progress.

Bridging the Gap

If you’re looking to gradually adopt gRPC or need to support existing REST clients, there are several options available today!

JSON/HTTP Transcoding

Tools like gRPC-Gateway, Google Cloud Endpoints and Envoy can expose REST-like interfaces while still reaping the benefits of gRPC on the backend. You can define a service that looks like this:

syntax = "proto3";
package your.service.v1;
option go_package = "github.com/yourorg/yourprotos/gen/go/your/service/v1";

import "google/api/annotations.proto";

message StringMessage {
  string value = 1;
}

service YourService {
  rpc Echo(StringMessage) returns (StringMessage) {
    option (google.api.http) = {
      post: "/v1/example/echo"
      body: "*"
    };
  }
}

And get a REST-like endpoint where you can make this request:

curl -XPOST '{"value": "my value!"}' http://localhost:8000/v1/example/echo

This is pretty amazing because it’s doing a lot of the hard work for you and you can now support many different REST APIs without writing any additional code. This is a simple example here but there are many options, like being able to populate message fields from components of the path.

gRPC-Web

One of the big limitations of gRPC is that it doesn’t work on the web with web browsers due to limited support of HTTP trailers. Browsers support receiving trailers but there isn’t yet a way to retrieve those trailers from javascript. Yes, this is incredibly frustrating, especially since there are many small use cases where trailer support would be amazing to have.

The gRPC-Web protocol gives browsers the ability to use gRPC, which drastically improves the story of contract-based services in gRPC. It also allows for HTTP/1.1 clients to work with gRPC. Some platforms (I’m looking at you, Unity) still don’t support HTTP/2, even though it’s 2024 and the HTTP/2 spec was created nearly a decade ago.

ConnectRPC

ConnectRPC automatically generates JSON/HTTP APIs from your gRPC definitions while also maintaining compatibility with gRPC and gRPC-Web. This HTTP protocol, called Connect, follows HTTP standards more closely. For example, the Content-Coding header, Content-Length header, HTTP status codes, etc. all work as expected for unary RPC calls. That means you can run this normal-looking curl command and talk to a gRPC service:

curl --header "Content-Type: application/json" \
    --data '{"sentence": "I feel happy."}' \
    https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say

Twirp

Twirp is very similar to ConnectRPC. It was developed by Twitch, and is another framework that can help bridge the gap between gRPC and REST. Twirp’s approach is to use protobufs to generate an alternative protocol that also aligns more with HTTP conventions. It doesn’t also support gRPC and gRPC-Web. Implementing those alongside twirp is left as an exercise for the user if you want to interoperate with other gRPC tooling.

Tooling

I have mentioned that gRPC tooling isn’t that great. I still agree with that if we’re talking about the “out of the box” tooling from the gRPC project. However, the community is much bigger than the gRPC Authors and someone finally made the protobuf code generation a lot better.

Buf CLI

Buf (the company) has made a client called Buf CLI, which I’m going to refer to as just “buf” from here on out.

protoc is the official compiler for protobufs, which has plugins for many languages, frameworks, documentation and other kinds of outputs. Buf completely replaces protoc by using the same protoc plugins that protoc uses. How is it better? It adds a set of config files for defining the structure of your protobuf files, including external protobuf dependencies and external plugins using the Buf Schema Registry. Instead of random makefile directives or bash scripts, we now have a well-defined config file for defining how the protobuf is built, which is amazing.

Similarly, buf curl provides a convenient way to interact with gRPC services, much like the popular tool grpcurl.

In addition to replacing existing tooling with easier-to-use versions, buf also implements some extremely useful functions. I first started using buf by using buf lint, which helps enforce some common rules and practices that developers should follow when making their protobuf files. Soon after, I started using buf breaking which will report on breaking changes being made to protobuf files that may break clients. Both have easy-to-use Github actions and were pretty painless to set up.

Adding buf into the mix can greatly improve your workflow with protobufs, especially when working in a larger team or working with other teams.

Third-party protoc plugins, libraries and tools

There’s so many plugins now. I even made one and to be honest, plugins aren’t that hard to make. I think this is the way that API development should work where you base API services off of a contract and generate everything from that same contract. No typos. No confusion over what methods exist. No arguing over REST semantics that has never been clear to anyone.

• protoc-gen-doc: Builds gRPC documentation in several formats. The default styling honestly doesn’t look the prettiest but it does allow you to specify a custom template which has been amazing for me to generate something custom without requiring an entire plugin.
• protoc-gen-connect-openapi: This is my plugin. It generates OpenAPIv3 specs for your ConnectRPC services and has support for protovalidate, gnostic OpenAPIv3 annotations, and gRPC-Gateway annotations.
• protovalidate: Protovalidate allows you to embed validation rules in your protobuf files which can be used by an associated library to enforce those rules. A large complaint that people have with gRPC is that it’s hard to use whenever every single field is optional. Now you can replace a lot of validation code, including required fields, with protobuf options. I’m anxiously awaiting typescript support so validation logic can be shared on web frontends and backends, which, to me, is the “holy grail” feature of a contract-driven service.

In addition to these libraries and plugins, more tools that you know and love from HTTP are supporting gRPC like Postman, Insomnia and k6.

The availability of numerous third-party plugins underscores the fact that gRPC is more than just a framework – it’s a dynamic ecosystem that fosters innovation and empowers developers to customize their workflows to meet their specific requirements.

Conclusion

gRPC offers a compelling set of advantages for modern web development.

Its performance, strong typing, streaming capabilities, cross-language support, and HTTP/2 foundation make it a powerful tool for building efficient and scalable APIs. With various adoption options available, you can gradually incorporate gRPC into your projects and experience its benefits firsthand.

The growing community and active development around gRPC suggest a bright future for this technology. If you’re looking to build fast, reliable, and future-proof APIs, gRPC is a tool that deserves a serious look. Dive in, explore the ecosystem, and discover how gRPC can revolutionize the way you approach API development.

Learning Curve

Let’s start out extremely nit-picky. So-called unary RPCs are calls where the client sends a single request to the server and gets a single response back. Why does gRPC have to use such a non-standard term for this that only mathematicians have an intuitive understanding of? I have to explain the term every time I use it. And I’m a little tired of it.

Speaking of unary RPCs, the implementation is more complicated than it needs to be. While gRPC’s streaming capabilities are powerful, they have introduced complexity for simple RPC calls that don’t require streaming. This hurts the ability to inspect gRPC calls because now there is framing on every unary RPC which only makes sense for streaming. Protobuf encoding is complicated enough so let’s not add extra gRPC framing where it isn’t needed. Also, it doesn’t pass my “send a friend a cURL example” test for any web API. It’s just super annoying to explain to someone how to use gRPC. I’ve said “okay, but is server reflection enabled?” so many times. I’m just tired of it.

This complexity also bleeds into the tooling with the mandatory code generation step. This can be a hurdle, especially for dynamic languages where runtime flexibility is valued. Additionally, some developers might be hesitant to adopt a technology that necessitates an extra build step. We already need 20 build steps for modern web development, so it’s sometimes hard to justify one more.

Compatibility with the Web

The reliance on HTTP/2 initially limited gRPC’s reach, as not all platforms and browsers fully supported it. This has improved over time, but it still poses a challenge in some environments. But even with HTTP/2 support, browsers have avoided adding a way to process HTTP trailers so browsers today still cannot use “original” gRPC. gRPC-Web has acted as a plaster for this issue by avoiding the use of trailers, but it often requires “extra stuff” like running a proxy that supports gRPC-Web. Which is annoying.

Late Adoption of HTTP/3: The delay in embracing HTTP/3 might have hindered gRPC’s ability to take full advantage of the protocol’s performance and efficiency benefits. I have personally been affected by the head-of-line blocking issue that can happen when using gRPC with HTTP/2 and it would be so nice to be able to completely do away with this issue by being able to use HTTP/3 with gRPC. It’s strange to see a framework that pushed many languages to support HTTP/2 struggling to do the same thing with HTTP/3.

JSON Mapping and Prototext

Another area where the “timing” was wrong was the lack of a standardized JSON mapping early on. It has made gRPC less accessible for developers accustomed to JSON-based APIs and I don’t think it ever recovered from that stigma. Having a mapping between protobuf types and JSON simplifies integration and interoperability with existing tools and systems. You would not believe how happy web developers can get when you say “yeah, this is a super-efficient binary format… but you can set this flag and get JSON back if you want to debug.” They get unreasonably excited. Unreasonably. excited. Anyway, now that protobuf has standard rules for mapping protobuf types to JSON (and the other way) I feel like the protobuf text format is an unnecessary complexity. I don’t see a use-case for the text format now that we have JSON. So let’s throw the text format away. We don’t need it and I’m down to pretend like it never existed if everyone else is. Cool?

Finite Message Sizes

Most Protobuf encoders/decoders expect to fully parse an entire message and give the full response to the consumer but memory is finite and sometimes you might want larger messages. Sometimes you want to stream parts of these larger messages somewhere else and not keep the entire message in memory. Therefore, if you want to, for example, upload large files you’re going to need to implement some kind of chunking. While chunking is a reasonable solution for handling large files, the absence of a standardized approach within gRPC might lead to inconsistent implementations and increased development effort.

As a demonstration, here’s what it may look like to upload a file with gRPC:

syntax = "proto3";

package file_service;

service FileService {
   rpc Upload(stream UploadRequest) returns(UploadResponse);
}

message UploadRequest {
    string file_name = 1;
    bytes chunk = 2;
}

message UploadResponse {
  string etag = 1;
}

This is both a strength and weakness of protobufs. This concept is super easy to define in protobuf but in practice, the code to properly implement this can be cumbersome and error-prone. And while Google, the creator of gRPC, has figured out solutions for their APIs, the lack of a standardized approach leaves others to reinvent the wheel.

You might be thinking “Google uses gRPC in most of their APIs so obviously they’ve done this” and you’d be right. They actually have a gRPC and HTTP version for downloading (potentially large) files. We can compare the gRPC and HTTP versions directly and gRPC is BY FAR more complex. Go ahead and compare the linked code. I’ll wait.

ded internet theory

I see a lot of gRPC/protobuf communities that are devoid of activity. The lack of visible activity on some websites might create the impression that gRPC is stagnant or less actively maintained. This could deter potential adopters and contribute to slower community growth. This might be a case of too many options, making it difficult to find someone to nerd out about gRPC outside of GitHub issues where such enthusiasm might be perceived as annoying.

Bad tooling

For the longest time, when I saw that a codebase uses protobuf I found a weird script that downloads random protobuf files in super custom ways and places them in random paths and then makes a series of super complex calls to protoc. Only google would think not solving dependency management is the solution to dependency management. Google has its own extremely Google-y way of managing dependencies that we peasants can only dream of using.

It can be (and is) better

While I’ve been critical of gRPC, I hope my comments come across as constructive. Those who have read this far down in the article get to know that many of these issues are already fixed or at least on the way to being fixed!

• Several gRPC implementations already support HTTP/3. ConnectRPC makes it pretty easy to use HTTP/3 with gRPC (I’ll follow up on this in a future post).
• Since the protobuf spec has a canonical mapping to/from JSON I no longer have to worry about the text format. I really do hope that everyone forgets that it exists. There’s only room for so many text-based formats. I wasn’t joking about that. This is the last time I’m acknowledging its existence.
• The gRPC community is actually alive and well if you know where to look. For example, the buf slack has been a great resource for me. You may find me hanging out and answering questions fairly often.
• The Buf CLI is an amazing tool for gRPC. It completely replaces protoc but also adds linting, breaking change detection, curl for gRPC, integration with the Buf Schema Registry (wow, real dependency management!) and more! In addition, more tools that you know and love from HTTP support gRPC like Postman, Insomnia and k6.

Despite gRPC’s undeniable successes, it’s important to acknowledge the framework’s shortcomings to ensure its continued evolution and improvement. By addressing its learning curve, compatibility issues, lack of standardization, and community engagement, we can unlock gRPC’s full potential and make it a more accessible and user-friendly tool for all developers.

One crucial optimization for internet-facing gRPC services is customizing the behavior towards unknown fields. I’ve talked about unknown fields in a previous post, so read that one if unknown fields are still a mystery to you and then come back here. By default, protobuf messages can contain fields that are not defined in the current version of the proto schema. While convenient for development and can help with forward compatibility, this poses a security risk in a public environment.

Here’s why you should consider dropping unknown fields when exposing gRPC to the internet:

• Preventing Malicious Data: Unknown fields can be exploited by malicious actors to inject unexpected data into your service. This could lead to potential security vulnerabilities like code injection or unexpected behavior.
• Ensuring Compatibility: Uncontrolled unknown fields can cause compatibility issues if your clients are using different versions of the proto schema. Dropping them enforces stricter adherence to the defined message format.
• Improving Performance: Skipping unknown fields during message parsing can lead to performance gains, especially when dealing with large datasets.

How to Drop Unknown Fields

Here is how you can drop unknown fields while using the standard proto.UnmarshalOptions struct provided by the google.golang.org/protobuf/proto package. Here’s how to do it in your Go code:

import (
	"google.golang.org/protobuf/proto"
	...
)

// Configure unmarshalling options to discard unknown fields
opts := proto.UnmarshalOptions{
	DiscardUnknown: true,
}

// Use the options when unmarshalling incoming messages
msg := &MyMessage{}
err := proto.Unmarshal(data, msg, opts)
if err != nil {
	// Handle error
}

By setting the DiscardUnknown field to true in the proto.UnmarshalOptions struct before unmarshalling incoming messages, you ensure that any unknown fields are ignored. This helps mitigate the security risks associated with unknown fields while processing internet-facing gRPC requests.

How to Drop Unknown Fields in Connect RPC Servers

package main

import (
	"log"
	"net/http"

	"golang.org/x/net/http2"
	"golang.org/x/net/http2/h2c"
	"go.akshayshah.org/connectproto"
)

func main() {
	greeter := &GreetServer{}
	mux := http.NewServeMux()
	path, handler := greetv1connect.NewGreetServiceHandler(
		greeter,
		// Add an option that customizes protobuf marshalling/unmarshalling behavior
		connectproto.WithBinary(
			proto.MarshalOptions{},
			proto.UnmarshalOptions{DiscardUnknown: true},
		),
		// Add an option to customize JSON marshalling/unmachalling
		connectproto.WithJSON(
			protojson.MarshalOptions{},
			protojson.UnmarshalOptions{DiscardUnknown: true},
		)
	)
	mux.Handle(path, handler)
	log.Fatal(http.ListenAndServe(
		"localhost:9000",
		h2c.NewHandler(mux, &http2.Server{}),
	))
}

In this example, connectproto.WithBinary ensures only messages with defined fields are processed, enhancing the security of your gRPC service. connectproto.WithJSON does the same thing but with JSON.

Additional Considerations

While dropping unknown fields is a valuable security practice, it’s important to consider potential trade-offs:

• Backward compatibility: Clients using older versions of the proto schema will encounter errors if they rely on previously defined unknown fields.
• Logging and Debugging: Dropping unknown fields might make it harder to identify the source of unexpected behavior during development or debugging.

In such cases, it’s recommended to document these trade-offs and have a clear versioning policy for your gRPC service and client applications.

Conclusion

Exposing gRPC services to the internet requires careful security considerations. By customizing protobuf encoding options, specifically by dropping unknown fields using proto.UnmarshalOptions, you can significantly improve the security posture of your service. Remember to weigh the benefits against potential drawbacks and implement a solution that aligns with your specific needs.

Objects? More like “objnoxious”

Imagine building a social media API endpoint to retrieve a user’s feed. Using a single REST object to represent a feed item can get messy. This object would need to encompass:

• User information (username, profile picture)
• Post content (text, images, videos)
• User interactions (likes, comments, shares) with timestamps
• Additional data like post visibility or author verification status

This “feed” object becomes bloated, especially if the feed contains many posts. Fetching and updating this complex object for every feed interaction can be inefficient. You can split the object up into many child objects but you are likely creating the need for clients to make more requests and greatly increasing the complexity of the API.

This highlights a limitation of REST: forcing real-world entities (like a social media feed) into rigid object structures with a strict hierarchy can lead to cumbersome data management.

Versioning is weird

Let’s say you introduce a new field to your product data model in a REST API. Versioning in the path (e.g., /api/v2/products) forces you to update every single endpoint URL that uses that data. Versioning each path by adding a query parameter (e.g., /api/products?version=2) or header seems more targeted, but what if only a specific endpoint needs the new version? Do you maintain a list of versions per endpoint? Do you bump the version for the entire API and default to the latest version? This is open to interpretation and every solution seems awkward to me.

Limited Options, Clever Solution

REST only offers a handful of methods (GET, POST, etc.) to handle data. There are 9 methods in total. Let’s talk about each one.

• CONNECT
• DELETE
• GET
• HEAD
• OPTIONS
• PATCH
• POST
• PUT
• TRACE

Most of these are used in niche, hyper-specific ways. I suspect most developers don’t know what HEAD, TRACE, OPTIONS, and CONNECT do. None of these are incredibly useful when developing web APIs. So let’s ditch them. Let’s also ditch PATCH because it’s just PUT in a trenchcoat.

Now we have: GET, POST, DELETE, and PUT. Sweet, we’re left with enough methods to make a CRUD application. Roughly speaking:

• Create = POST
• Read = GET
• Update = PUT
• Delete = DELETE

Wow! Such simple. So elegant. I’m sure glad that everything web developers do boils down to these 4 simple actions… Oh wait, that’s totally wrong. There are so many more actions you can do to an object. Just think of how crazy it would be if you were using a programming language where you could only have four pre-defined methods in each class. Think of all of the extra container classes and weird abstractions we’d make on top of that. This sounds like utter insanity and is exactly what REST gives us.

Let’s stop pretending that there are only 9 (but actually 4) things you can do to a resource.

Inefficiency of JSON

JSON is slow and inefficient. It’d be insane if we built the internet around this format. Wait, we did? Really? JSON is wasteful in many ways. It’s text-based, which has an inherent cost in payload size and processing. JSON also will include key names over and over again and the length of the keys directly translates to longer payloads. This is not ideal if we’re trying to save on data transfer. Protobuf, on the other hand, is a compact and efficient binary format specifically designed for data serialization. Even factoring in gzip, JSON loses out to encoded protobuf in pretty much every way: CPU usage, memory usage, message size and speed. This just shows that there are better formats than JSON.

OpenAPI?

OpenAPI is a specification for describing RESTful APIs. It acts as a contract between API providers and consumers, defining the available resources, their properties, and the allowed operations (GET, POST, PUT, DELETE) for each. A common way OpenAPI is used is by generating OpenAPI specifications from the source of the backend service. Depending on the level of library integration these tools can automatically discover the HTTP method, route, request and response types, etc. This can help keep documentation up-to-date compared to manually creating OpenAPI spec, which is pretty incredible.

Okay, now here’s where I get philosophical. If we’re going to have a declarative specification for our APIs, I believe the specification should be the source of truth rather than the output. In my mind, OpenAPI specification should be the very first thing you write and agree upon and the servers and clients should be. However, many people don’t do this because the tooling isn’t amazing. Developers have grown fond of specific libraries and frameworks to develop our APIs so the target for generating language/framework/library code is vast and appears to be an incredibly hard problem. OpenAPI tries to solve so many problems at once. It’s coming in after we’ve designed our APIs and is trying to describe what’s already there. It’s an afterthought.

I’ve attempted to go down the route of using OpenAPI to generate server stubs and clients. It didn’t end well. There were too many issues generating clients and servers, even when the OpenAPI specification was valid. So I had to edit our OpenAPI spec to fit the limitations of the code generators just to get code generation to work. And that was just the beginning of my problems. I contend that this is a natural result of the design goals of the project. OpenAPI was not designed for generating code this way as a priority, so with many complex scenarios, it can be unclear how to map the spec to the semantics of the language/framework/library. OpenAPI wasn’t designed for that, it was only designed to describe the API, not the server or client that produces or consumes it. Now consider just how many “targets” for client and server stubs you want. Now consider that target libraries and the OpenAPI spec itself are evolving. This results in a compatibility matrix from hell.

So what other options do we have?

REST has served us well, but the modern web demands more. Let’s explore the exciting alternatives that offer a range of benefits and functionalities:

GraphQL

The “Choose Your Own Adventure” API. Still with the social media app example, imagine fetching only the user’s name and profile picture for the feed view, and then requesting their full profile and friend list separately when a user clicks on their profile. GraphQL allows you to specify exactly the data you need for each view, reducing unnecessary data transfer.

This method also has its downsides like making the backend API extremely complex.

gRPC

The “Cut the Drama” API. Consider a mobile game that communicates with a game server. gRPC allows you to define remote procedures (like attackEnemy or usePowerUp) that the client can call directly on the server. This removes the need for complex REST resource mapping and makes the communication intent clear. There are variants of gRPC like ConnectRPC that allow for leveraging of HTTP GET requests so you can fully leverage browser caching.

WebSockets

Need real-time updates like a live chat or stock ticker? WebSockets offer a persistent two-way communication channel, ideal for constantly flowing data between client and server. This is different from REST’s request-response cycle, allowing for a more dynamic connection.

Server-Sent Events (SSE)

SSE allows the server to push updates to the client without the client needing to constantly ask. Imagine live sports scores or social media notifications. SSE is simpler to implement than WebSockets, but is one-way (server to client).

Here’s a conclusion that summarizes the key points and offers a final thought:

The Verdict: REST vs. the Rest

REST APIs have served us faithfully for years, but as our applications become more complex and data-hungry, it’s worth considering the alternatives. GraphQL offers flexibility in data fetching, gRPC provides clear and efficient communication, WebSockets enable bidirectional real-time data flow and great browser support, and SSE simplifies server-to-client updates.

The choice ultimately depends on your specific needs. But remember, the API landscape is ever-evolving. HTTP/2 and HTTP/3 have opened up some new functionality that we have yet to fully tap into with our API designs. So, keep an open mind, explore the options, and don’t be afraid to break free from the comfy (but maybe slightly threadbare) jeans of REST when a more fitting alternative emerges.

When dogfooding our services it became painfully obvious that our API was extremely hard to use. When making tooling we had to go through extremely complex ordering APIs that were designed for internal use and were exposed publicly for convenience. I preferred to do this work in Python at the time, so I used our public API bindings. This was the current state of the project on github. Let me point out some of the issues I had:

• Didn’t work with Python 3.
• Used class variables incorrectly, making it impossible to use multiple instances of the client at the same time.
• Had an awkward use of dictionary accessors that made things more confusing as a user.
• Had absolutely zero unit tests.
• The Python module name used upper-case characters.

So… Immediately, I wanted to improve things. However, this was the first open-source project that I would modify that had a non-trivial number of users. Because of this, I learned to make strategic changes that followed the pattern of:

• Add several unit tests for the part of the code that
• Add a new interface that
• Use the new interface in the implementation of the old interface (to reduce code duplication)
• Add deprecation warnings for the old interface.
• After enough time, bump the major version of the library and remove all code that is deprecated.

This pattern can be incredibly tedious. However, if you are maintaining a project used by many then you need to worry about upgrading. You need to write release notes. You need to give enough examples. You need to provide an upgrade path when you want to make breaking changes. This is super boring work, but it’s the difference between “when I upgrade this package everything breaks” and “when I upgrade this package, I get more cool new stuff”.

For perspective, here’s what the code looks like now. Note that there are now almost 2,000 unit tests. Note that the tests are run with several different versions of Python. And you should also note that there’s an entire command line client in the repo as well!

The CLI was also created from the same motivations. For us, it makes automating and testing things much easier. Instead of clicking through a virtual machine creation web form for 20 minutes I could just copy/paste a command that I’ve run before that could specify everything I would have to type or select anyway. In fact, here’s an example of that!

$ slcli vs create --hostname=example --domain=softlayer.com -f B1_1X2X25 -o DEBIAN_LATEST_64  --datacenter=ams01 --billing=hourly
This action will incur charges on your account. Continue? [y/N]: y
    :..........:.................................:......................................:...........................:
    :    ID    :               FQDN              :                 guid                 :         Order Date        :
    :..........:.................................:......................................:...........................:
    : 70112999 : testtesttest.test.com : 1abc7afb-9618-4835-89c9-586f3711d8ea : 2019-01-30T17:16:58-06:00 :
    :..........:.................................:......................................:...........................:
    :.........................................................................:
    :                            OrderId: 12345678                            :
    :.......:.................................................................:
    :  Cost : Description                                                     :
    :.......:.................................................................:
    :   0.0 : Debian GNU/Linux 9.x Stretch/Stable - Minimal Install (64 bit)  :
    :   0.0 : 25 GB (SAN)                                                     :
    :   0.0 : Reboot / Remote Console                                         :
    :   0.0 : 100 Mbps Public & Private Network Uplinks                       :
    :   0.0 : 0 GB Bandwidth Allotment                                        :
    :   0.0 : 1 IP Address                                                    :
    :   0.0 : Host Ping and TCP Service Monitoring                            :
    :   0.0 : Email and Ticket                                                :
    :   0.0 : Automated Reboot from Monitoring                                :
    :   0.0 : Unlimited SSL VPN Users & 1 PPTP VPN User per account           :
    :   0.0 : 2 GB                                                            :
    :   0.0 : 1 x 2.0 GHz or higher Core                                      :
    : 0.000 : Total hourly cost                                               :
    :.......:.................................................................:

Here we’re creating a VM inside of the Amsterdam data center that uses the latest Debian image with 2GB of RAM, a single CPU core and 25GB of disk space. All by copying a single command. This is super powerful. If you’re wondering why everything is free it’s because our account had special billing. 😛

After you create a VM, you can also list the running instances to see it:

$ slcli vs list
:.........:............:....................:.......:........:................:..............:....................:
:    id   : datacenter :       host         : cores : memory :   primary_ip   :  backend_ip  : active_transaction :
:.........:............:....................:.......:........:................:..............:....................:
: 1234567 :   sjc01    :  test.example.com  :   4   :   4G   :    12.34.56    :   65.43.21   :         -          :
:.........:............:....................:.......:........:................:..............:....................:

The story is the same for several other products. Here’s a list of what products are supported today:

• Account Management
• Block Storage
• Bandwidth Pools
• CDN
• Dedicated Hosts
• DNS
• Email
• File Storage
• Firewall
• Global IP
• Dedicated Hardware
• Disk Images
• IPSec
• Licenses
• Load Balancers
• NAS
• Object Storage
• Ordering/Quotes
• SSH Keys and Certificates
• Security Groups
• Subnets
• Support Tickets
• Users
• VLANs
• Virtual Servers

It’s remarkable how far this project has come. This started as a way to create some VMs with a script but it became an extremely important interface for the entire suite of cloud products. Slowly, the CLI grew to what it is today. This wasn’t a result of just me. Instead, several members of mine and other people’s teams contributed to the project. There were periods when I didn’t write a lot of code but would spend most of my time reviewing and guiding others to make their contributions. I learned a lot about the importance of enforcing a single style and keeping the quality of code high. Essentially, the ways customers interfaced was through the website, API or the CLI.

The CLI also drove more usage of the Python client. It encouraged this growth in several ways:

• It showcased what was possible with the API in a way that the documentation just can’t do.
• It acted as a good reference for “how can I do this with the API”. The more we added to the CLI, the fewer extra examples we needed to make. It is very important that the CLI was also open source for this reason.
• It had a verbose flag that showed the API calls that were being made, greatly increasing visibility into how it works.

Here’s an example of what a command looks like when running a command using the verbose flag.

$ slcli -v vs detail 74397127
Calling: SoftLayer_Virtual_Guest::getObject(id=74397127, mask='id,globalIdentifier,fullyQualifiedDomainName,hostname,domain', filter='None', args=(), limit=None, offset=None))
Calling: SoftLayer_Virtual_Guest::getReverseDomainRecords(id=77460683, mask='', filter='None', args=(), limit=None, offset=None))
:..................:..............................................................:
:       name       :                            value                             :
:..................:..............................................................:
:  execution_time  :                          2.020334s                           :
:    api_calls     :        SoftLayer_Virtual_Guest::getObject (1.515583s)        :
:                  : SoftLayer_Virtual_Guest::getReverseDomainRecords (0.494480s) :
:     version      :                   softlayer-python/v5.7.2                    :
:  python_version  :           3.7.3 (default, Mar 27 2019, 09:23:15)             :
:                  :              [Clang 10.0.1 (clang-1001.0.46.3)]              :
: library_location : /Users/chris/Code/py3/lib/python3.7/site-packages/SoftLayer  :
:..................:..............................................................:

The more v characters you add, the more verbose the output gets. If you use -vvv then you will get the equivalent cURL commands to make the same API calls, which should be clear enough for any developer to make a client against the API.

$ slcli -vvv account summary
curl -u $SL_USER:$SL_APIKEY -X GET -H "Accept: */*" -H "Accept-Encoding: gzip, deflate, compress"  'https://api.softlayer.com/rest/v3.1/SoftLayer_Account/getObject.json?objectMask=mask%5B%0A++++++++++++nextInvoiceTotalAmount%2C%0A++++++++++++pendingInvoice%5BinvoiceTotalAmount%5D%2C%0A++++++++++++blockDeviceTemplateGroupCount%2C%0A++++++++++++dedicatedHostCount%2C%0A++++++++++++domainCount%2C%0A++++++++++++hardwareCount%2C%0A++++++++++++networkStorageCount%2C%0A++++++++++++openTicketCount%2C%0A++++++++++++networkVlanCount%2C%0A++++++++++++subnetCount%2C%0A++++++++++++userCount%2C%0A++++++++++++virtualGuestCount%0A++++++++++++%5D'

In summary, this was an incredibly successful side project. What started as a small script for internal use turned into a Swiss army knife that was a completely new way to access all products that SoftLayer offered. I learned so much about maintaining an open-source project, choosing reliable libraries to build on, code quality/style, and so much more.

References:

• Github: https://github.com/softlayer/softlayer-python
• Documentation: https://softlayer-python.readthedocs.io/en/latest/

After testing out a few different ideas I ended up deciding on the SFTP protocol. Object Storage doesn’t translate 1:1 with a “normal” filesystem but it was close enough. This might seem a strange decision in modern days where everyone has S3 integration but this was done as a way to integrate with older products and give customers a much more familiar feel with interacting with their data. I decided to call the project SwFTP.

graph LR
    cyberduck((Cyberduck)) -- SFTP --> swftp((SwFTP))
    filezilla((FileZilla)) -- SFTP --> swftp((SwFTP))
    client((SFTP/FTP Client)) -- SFTP or FTP --> swftp((SwFTP))
    swftp((SwFTP)) -- HTTPS --> swift((Swift))
  

I was all set. I needed to just… implement an SFTP server. Oof. That’s actually very challenging. You see, the SFTP protocol has lived for over a decade at that point so it has gone through several RFC drafts and it supports various extensions… And it’s built on top of SSH, which is also fairly complex to implement. So I needed to use something more proven. I needed a library that I could plug in OpenStack Swift integration into without needing to implement the wire protocol itself. Again, at the time this kind of ability was fairly rare. I was very experienced with Python and a bit with Go (nowadays I would totally write this project in Go, using this SFTP library and implementing the fs.Walker interface). That would have made my life so much simpler. But no, I was essentially stuck with Python’s Twisted. Twisted has a library called “Conche” which implements the SSH protocol and it allows you to hook into the SFTP subsystem, which is exactly what I needed. Twisted seemed to be the best option at the time, but it was (and still is) very hard to work with. The failure modes can be very complex. Plus, the SFTP protocol is fairly complex and SFTP clients will behave in vastly different ways. For instance, some clients, in order to maximize throughput, will concurrently send several batches of data at once when uploading a file without waiting for the acknowledgment to be received. We can’t behave similarly with an object storage API call so I needed to force the concurrent write requests to queue up properly until it was their turn to be sent down the wire to Swift. This, alone, is complex but I was also using an unfamiliar framework that has completely different sync primitives than I’ve used in the past… So I found this work to be very challenging… but in the end, it was very rewarding.

SwFTP was never the only project I was working on so my attention was split multiple ways. Despite that, after a year of working on this project, it was finally good enough to deploy to production as a supported service. Testing took a long time because, as I said, many SFTP clients behave very differently. I am happy about where I got the functional test framework since I was able to easily write code that would reproduce errors that we saw when testing, including some super complicated cases of race conditions. From this experience, I’ve learned some very important lessons about functional and manual testing.

All-in-all, SwFTP ended up being a success and a lot of data was transferred using this service. Thanks to my manager and support from others in the company I was able to perform all of this iteration and development as an open source project. There were no SoftLayer-specific implementation details included here so others could (and did) deploy the project for their own OpenStack Swift clusters.

By the way, if you’re having issues pronouncing “SwFTP” in your head then you aren’t alone. I used to call it something like “Swefteepee”.

References:

• SwFTP Github - https://github.com/softlayer/swftp
• Conch (SSH library) - https://docs.twisted.org/en/stable/api/twisted.conch.html
• Writing a client using Conch - https://docs.twisted.org/en/twisted-18.9.0/conch/howto/conch_client.html