In today’s interconnected world, APIs (Application Programming Interfaces) are the glue that connects computers. They allow different applications to talk to each other, share data, and perform actions. However, traditional methods of creating APIs often lead to frustrating challenges: breaking changes in JSON APIs, silent failures due to missing fields, frontend and backend drift, or schema mismatches that result in the classic “works on my machine” excuse.

Imagine a real-world scenario where the backend team renames a userId field to user_id and deploys their changes. Instantly, the frontend checkout process breaks in production because the API had no strict enforcement to catch the mismatch.

This is where contract-based APIs come in. A contract-based API is one where the schema is defined first in a formal specification, and both client and server are generated or validated against that contract. They reduce ambiguity and enforce consistency across services.

The Power of Pre-defined API Contracts

A contract-based API defines exactly what data can be exchanged, in what format, and what actions can be performed. This strict, pre-defined agreement unlocks several immediate advantages:

• Improved Developer Experience: Developers on both sides (client and server) have a clear understanding of what is expected, making integration smoother.
• Automated Documentation: Contracts serve as self-documenting artifacts. This reduces the need for manual documentation maintenance and ensures the docs stay in sync with the actual API implementation.
• Reduced Errors: Mismatched data formats or API changes become less likely, leading to fewer bugs. Contracts act as a validation layer that catches potential issues early.
• Easier Integration: Contracts act as a single source of truth. Developers can quickly understand how to interact with the API without extensive back and forth communication.
• Streamlined Development: These APIs often enable tools to automatically generate code for both client and server implementations. This eliminates manual boilerplate so you can focus on core logic.

Protobuf: The Language of APIs

In modern distributed systems, the foundation of many contract-based APIs lies in Protocol Buffers (protobuf). It is a language-neutral data format specifically designed for structured messages.

Unlike JSON, which is a text-based format designed to be human-readable, Protobuf is a binary format. This means you trade the ability to natively read the raw data in transit for significant performance gains:

• Smaller Message Sizes: Protobuf messages are compact and efficient, which leads to faster transmission and reduced bandwidth usage.
• Faster Parsing: Parsing binary protobuf messages is significantly faster compared to traditional formats like JSON or XML.
• Built-in Versioning: Protobuf uses field numbers (the = 1, = 2 in the code below) to identify data. This allows for excellent backward and forward compatibility. You can add new fields without breaking older clients that do not know about them yet.
• Cross-language Compatibility: Protobuf definitions are language-agnostic. Code for interacting with the API can be generated for almost any modern programming language.

Because the data is binary, you cannot simply open your browser’s network tab and read the payloads by default. You will usually need to rely on modern browser extensions (like the gRPC-Web or Connect dev tools) to decode the traffic. It also requires setting up specialized tooling and build steps to compile the generated code.

Here is a basic example of a .proto file defining messages for a user and an address:

syntax = "proto3";

message User {
  string name = 1;
  int32 id = 2;
  string email = 3;
  Address address = 4;
}

message Address {
  string street = 1;
  string city = 2;
  string state = 3;
  string zip = 4;
}

In this example, the User message has fields for name, ID, email, and an Address message. These defined structures ensure consistent data exchange between applications.

Key idea: Protobuf relies on immutable field numbers instead of field names. This golden rule guarantees backward and forward compatibility.

gRPC: Building APIs on a Solid Foundation

gRPC (gRPC Remote Procedure Call) is a high-performance framework that builds upon protobuf’s strengths. It provides a powerful way to implement remote procedure calls, allowing applications to interact using clients generated for each language.

Introducing Services and Request/Response Types with gRPC

We can expand the .proto file to define a service called UserService with methods for user management:

syntax = "proto3";

service UserService {
  rpc CreateUser(CreateUserRequest) returns (User) {}
  rpc GetUser(GetUserRequest) returns (User) {}
}

message CreateUserRequest {
  User user = 1;
}

message GetUserRequest {
  int32 id = 1;
}

This example defines a UserService with two methods: CreateUser and GetUser. Each method takes a specific request message and returns a response.

Notice how clear the intention is. A helpful mental model to contrast modern APIs is:

• REST is resource-oriented (relying on URLs and HTTP verbs).
• gRPC is action-oriented (relying on explicit methods).

A reader of this spec does not have to map vague HTTP verbs like “POST” to actions like “create.” Also, these method names are greppable. It is trivial to locate every use of CreateUser across several repositories, making refactoring and impact analysis much easier.

Server Reflection

Another powerful feature of the gRPC ecosystem is Server Reflection. This allows clients or debugging tools (like Postman or grpcurl) to query the server at runtime to discover the available services and methods. This eliminates the need to distribute .proto files to developers just so they can explore the API structure.

Distributing API Contracts

Defining a contract is only half the battle. How do the frontend and backend teams actually share that .proto file? If the schema is not easily accessible, the contract is useless.

In practice, teams usually solve this distribution problem in one of three ways:

1. Monorepos: Storing the backend, frontend, and API definitions in a single repository so all code shares the same source of truth.
2. Package Managers: Generating the client SDKs in a CI/CD pipeline and publishing them as internal NPM, Maven, or Go packages.
3. Schema Registries: Using dedicated tools like the Buf Schema Registry to manage, version, and distribute Protobuf files securely across an organization.

What about public APIs?

Historically, strict RPC contracts were tough for external, public-facing APIs. If your primary consumers were third-party developers, handing them a raw Protobuf file or expecting them to set up gRPC clients caused massive friction. They just wanted to use standard REST with JSON.

This is where tools like ConnectRPC shine. ConnectRPC allows you to define your API using Protobuf, but it automatically exposes endpoints that support standard HTTP/1.1 and JSON serialization as a fallback format.

This hybrid approach also solves the local debugging problem. You can configure ConnectRPC to use JSON during local development specifically so you can read the network tab in plain text, and then flip it to highly efficient binary for production. In practice, you write Protobuf once, and get both gRPC and REST/JSON APIs for free.

Even better, because the source of truth is still Protobuf, you can use ecosystem plugins to automatically generate an OpenAPI specification directly from your .proto files. You get a highly maintainable, contract-driven architecture on the backend, while your external users can still curl standard REST endpoints, read plain JSON, and explore your API via a generated Swagger UI. It offers the best of both worlds without compromising the developer experience on either side.

Key idea: Tools like ConnectRPC allow you to maintain strict internal Protobuf contracts while exposing standard REST/JSON APIs to external consumers.

Alternatives

While Protobuf and gRPC are a powerful duo, there are other contract-based API solutions to consider depending on your architecture:

• OpenAPI (Swagger): Contracts are not exclusive to RPC. You can use OpenAPI to define strict contracts for RESTful services. However, a harsh reality of the industry is that OpenAPI specs often drift from the actual code because they are bolted on after the fact. To make OpenAPI truly safe, teams must rely on strict framework integration (like FastAPI in Python or tsoa in Node) where the code generates the spec, or vice versa.
• GraphQL: Arguably the most mainstream contract-driven API paradigm for frontend developers. Its strictly typed schema defines the exact shape of the available data. Unlike gRPC, which has fixed responses, GraphQL allows the client to dictate the exact payload it wants to receive.
• Twirp: Developed by Twitch, Twirp is a lightweight RPC framework built on top of Protobuf and HTTP/1.1. It shares similarities with ConnectRPC but focuses on absolute simplicity. It avoids the complexity of HTTP/2 and gRPC streams while still providing generated clients, making it an excellent alternative if full gRPC is overkill for your needs.
• Thrift: Originally developed at Facebook, Thrift is a language-neutral protocol for defining service contracts similar to Protobuf. It is often found in large-scale data environments and supports various RPC protocols.
• tRPC: This tool defines the API schema directly in TypeScript code to be reused on both the client and the server. While it often pairs with libraries like Zod for runtime validation, it lacks true language-agnostic safety across the network boundary since it relies entirely on a TypeScript ecosystem.
• Avro: This format uses JSON-like schemas but stores data in a compact binary format. It is a staple in the Apache Kafka ecosystem for streaming data pipelines. It handles schema evolution differently than Protobuf (often sending the schema alongside the data), making it highly flexible for dynamic systems.

When NOT to Use API Contracts

While these tools are powerful, they are not a silver bullet. You should reconsider using strict API contracts if:

• You are building small projects or MVPs: The initial setup, code generation, and boilerplate overhead might slow down your speed of delivery when rapid iteration is the top priority.
• Simplicity for external consumers outweighs strict contracts: If you are building a straightforward public API and are not using a hybrid tool like ConnectRPC, raw JSON over REST remains the path of least resistance for third-party developers.
• Your team lacks tooling maturity: Implementing gRPC or Protobuf requires solid CI/CD pipelines and a team that is comfortable managing build steps, code generation, and backward-compatible schema evolutions.

Key idea: Strict API contracts add overhead and may not be suitable for small MVPs, simple public APIs, or teams lacking tooling maturity.

Conclusion

Contract-based APIs offer a significant advantage in building robust and scalable communication between applications. Protobuf and gRPC provide a powerful combination for defining clear contracts and generating highly efficient code.

As a general rule of thumb: if you are building an early-stage prototype, stick to what is fast and familiar. But if you are scaling a complex system across multiple teams and services, contract-based APIs transition from a nice-to-have to an absolute necessity. Once multiple teams depend on your API, contracts stop being optional. They are how you avoid chaos.

First off, the full source code can be found on github. If it helps, feel free to download, run, and modify as you see fit!

Why Unit Test?

Before we dive in, let’s address the “why.” Unit testing your ConnectRPC servers brings a multitude of benefits:

• Isolation: Focus on testing individual components in isolation, making it easier to pinpoint and fix issues.
• Speed: Unit tests execute quickly, providing fast feedback during development.
• Refactoring Confidence: When you have solid unit tests, you can refactor your code with confidence, knowing that the tests will catch any unintended consequences.
• Documentation: Well-written unit tests can serve as living documentation, illustrating how your code is meant to be used.
• Bug Prevention: A good suite of unit tests can help you catch bugs early on, before they become harder and more expensive to fix.

Testing Strategies with ConnectRPC

ConnectRPC, built upon the Protocol Buffers ecosystem, offers a couple of primary approaches to unit testing:

1. Direct Service Testing: This is ideal for unit testing but it’s not always possible. You directly call the methods of your service implementation (typically a struct in Go), bypassing any client and server networking.
2. Server Testing: This approach creates an actual ConnectRPC server with net/http/httptest. It’s helpful when you want to test the interactions between your client and server code but is usually “overkill” unless you’re wanting to test interceptors or HTTP middleware.

Hands-On: Our example service

Here is the protobuf file that we’re using for our example:

syntax = "proto3";

package greet.v1;

option go_package = "example/gen/greet/v1;greetv1";

message GreetRequest {
  string name = 1;
}

message GreetResponse {
  string greeting = 1;
}

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse) {}
}

And here is the resulting server implementation:

type greeterService struct{}

var _ greetv1connect.GreetServiceHandler = (*greeterService)(nil)

func (g *greeterService) Greet(ctx context.Context, req *connect.Request[greetv1.GreetRequest]) (*connect.Response[greetv1.GreetResponse], error) {
	if req.Msg.Name == "" {
		return nil, errors.New("missing name")
	}

	// Simulate some network call that takes 10ms
	select {
	case <-ctx.Done():
		return nil, ctx.Err()
	case <-time.After(10 * time.Millisecond):
	}
	return connect.NewResponse(&greetv1.GreetResponse{Greeting: "Hello, " + req.Msg.Name}), nil
}

Here we defined our greetv1connect.GreetServiceHandler implementation. It implements the Greet method defined in the protobuf file alove. Since this is for demonstration purposes, all we do is check to see if the given name is empty, sleep for 10 milliseconds to simulate a network call and returns the greeting as "Hello, {name}".

A keep observer might notice that this file contains our first “test”. The line var _ greetv1connect.GreetServiceHandler = (*greeterService)(nil) is a way of doing a type assertion in Go. It ensures that your greeterService struct correctly implements the GreeterService interface defined by the protobuf file above. This relies on a trick of the Go syntax that will try to bind a variable _ using the greetv1connect.GreetServiceHandler type. If the given greeterService pointer doesn’t implement the interface then the compiler should complain about what specific methods are missing and which method signatures don’t match.

Hands-On: Direct Service Testing Example

Let’s write some unit tests for a simple ConnectRPC service:

func TestGreet(t *testing.T) {
	service := &greeterService{}
	response, err := service.Greet(context.Background(), connect.NewRequest(&greetv1.GreetRequest{Name: "Alice"}))
	if err != nil {
		t.Fatalf("Greet failed: %v", err)
	}
	if response.Msg.Greeting != "Hello, Alice" {
		t.Errorf("Unexpected greeting: got %q, want %q", response.Msg.Greeting, "Hello, Alice")
	}
}

Explanation: The TestGreet function creates an instance of your greeterService and directly calls its Greet method. We then assert that the response matches our expectations. This is, by far, the simplest method for testing a ConnectRPC service.

Hands-On: Table-Driven Tests with Testify

Now that we wrote a single unit test, the next example will show you how to utilize table tests in order to easily write more test cases. You will see code that looks like this in well-tested Go repositories.

func TestGreetTable(t *testing.T) {
	service := &greeterService{}
	cancelledCtx, cancel := context.WithCancel(context.Background())
	cancel()

	testCases := []struct {
		name    string
		ctx     context.Context
		req     *connect.Request[greetv1.GreetRequest]
		want    *connect.Response[greetv1.GreetResponse]
		wantErr string
	}{
		{
			name:    "Success",
			req:     connect.NewRequest(&greetv1.GreetRequest{Name: "Bob"}),
			want:    connect.NewResponse(&greetv1.GreetResponse{Greeting: "Hello, Bob"}),
			wantErr: "",
		},
		{
			name:    "Empty Name",
			req:     connect.NewRequest(&greetv1.GreetRequest{}),
			want:    nil, // Expecting an error
			wantErr: "missing name",
		},
		{
			name:    "Context Cancelled",
			ctx:     cancelledCtx,
			req:     connect.NewRequest(&greetv1.GreetRequest{Name: "Alice"}),
			want:    nil,
			wantErr: "context canceled",
		},
	}

	for _, tc := range testCases {
		t.Run(tc.name, func(t *testing.T) {
			ctx := tc.ctx
			if ctx == nil {
				ctx = context.Background()
			}

			got, err := service.Greet(ctx, tc.req)
			if tc.wantErr != "" {
				require.Error(t, err)
				assert.Contains(t, err.Error(), tc.wantErr)
			} else {
				require.NoError(t, err)
				assert.Equal(t, tc.want, got)
			}
		})
	}
}

Explanation:

1. Table Setup: A testCases slice defines scenarios with varying inputs (req), expected outputs (want), and potential errors (wantErr).
2. Context Cancellation: The test case “Context Cancelled” simulates a cancelled context by creating a context with context.WithCancel and immediately calling cancel().
3. Testify Assertions: The require package is used for assertions that should stop the test if they fail (e.g., requiring an error). The assert package is used for assertions that are not critical for continuing the test. Typically, errors during test setup use require and assertions on the results of the test use assert.

Hands-On: Server Testing Example

Here’s how you can test the same service using net/http/httptest server:

func TestGreetWithServer(t *testing.T) {
	mux := http.NewServeMux()
	mux.Handle(greetv1connect.NewGreetServiceHandler(&greeterService{}))

	server := httptest.NewServer(mux)
	t.Cleanup(func() { server.Close() })

	client := greetv1connect.NewGreetServiceClient(http.DefaultClient, server.URL)

	response, err := client.Greet(context.Background(), connect.NewRequest(&greetv1.GreetRequest{Name: "Alice"}))
	if err != nil {
		t.Fatalf("Greet failed: %v", err)
	}
	if response.Msg.Greeting != "Hello, Alice" {
		t.Errorf("Unexpected greeting: got %q, want %q", response.Msg.Greeting, "Hello, Alice")
	}
}

• Server Setup: We create a ConnectRPC handler and start it with httptest.NewServer(mux).
• Client Setup: We create a ConnectRPC client that connects to the server that we just created.
• Test Interaction: We use the client to call the Greet method and assert the response, just like in the direct service testing example.

Conclusion: Test with Confidence

In this guide, we’ve explored the “why” and “how” of unit testing your ConnectRPC services. By embracing unit testing as a core part of your development workflow, you’ll create more robust, reliable, and maintainable RPC systems. Remember, effective testing isn’t just about fixing bugs – it’s about building confidence in your codebase and enabling you to iterate and evolve your services with ease.

The full source code can be found on github.

Next Steps:

• Go Beyond the Basics: Explore more advanced testing techniques, such as mocking dependencies for more complex scenarios.
• Integrate with Your CI/CD: Automate your unit tests to run as part of your continuous integration and continuous delivery (CI/CD) pipeline for immediate feedback on code changes.
• Share Your Knowledge: Help the ConnectRPC community grow by sharing your own testing strategies and experiences!

Ready to put your newfound knowledge into action? Start writing those unit tests and watch your ConnectRPC projects thrive!

For marshaling and unmarshaling entire protobuf messages, use the google.golang.org/protobuf/proto package instead.

Am I going to listen to this solid advice? No! I want to know how this works! No reflection and no reliance on generated code. All of the code in this post are taken from unit tests available here so feel free to download the tests and play around with it locally. Now, let’s get started.

Wire Types

I discuss in my Inspecting Protobuf Messages post that protobuf only has a small handful of types. Here they are again:

For today we’re only going to implement some of the LEN and VARINT wire types. Protobuf messages are a series of key-value pairs. Keys are “field numbers” and values are one of the types in the table above. To save space, protobuf decided to encode both the field number and wire type into a single byte. The lower three bits are the wire type and the other 5 (and maybe more, more on that later) are used for the field number. This is what the “Tag-Length” part looks like for field number 1 with the VARINT wire type:

0000 1000

You can shift the three least significant bits off to get the protobuf wire type and the rest is used for the field number. It is at this point that the bitwise operators start becoming useful:

(field_number << 3) | wire_type

So… 3 bits for the wire type leaves room for 8 different options so protobuf has room for two more wire types before they have to break compatibility with older versions or add another byte to describe more types. And 5 bits for the field number which leaves room for…… 32 different fields??? What?! You can’t have a message that has more than 32 fields?! Why is no one talking about this glaring limitation where the number of fields is limited to a four-year-old’s counting ability?! Well, obviously this is not true and, at this point, I am now forced to explain what protobuf refers to as Base 128 Varints.

Big numbers

In the previous example, we saw VARINT take a single byte. What does it look like when your number is too big? Where does the variableness part of VARINT come in? The protobuf encoding uses what it calls Base-128 Variable Integers in several places. “Base 128” means you can count to 127 before rolling over to the next “digit” (or in this case, byte). The most significant bit is used as a continuation bit, which is a signal that there’s at least one more byte worth of data to complete this integer. Let’s decode one for practice:

11000000 11000100 00000111   // Original inputs.
 1000000  1000100  0000111   // Drop continuation bits.
 0000111  1000100  1000000   // Convert to big-endian.
 000011110001001000000       // Concatenate.
 (1 × 2^16) + (1 × 2^15) +   // Convert binary to decimal
 (1 × 2^14) + (1 × 2^13) +   // because we're not a computer.
 (1 × 2^9) + (1 × 2^6)
 = 123456                    // Interpret as an unsigned 64-bit integer.

Next, I will show you some go code that can do this VARINT encoding process for us. Note that this code is actually in Go’s standard library and makes reference to protocol-buffers directly in the documentation; (source).

// AppendUvarint appends the varint-encoded form of x,
// as generated by [PutUvarint], to buf and returns the extended buffer.
func AppendUvarint(buf []byte, x uint64) []byte {
	for x >= 0x80 {
		buf = append(buf, byte(x)|0x80)
		x >>= 7
	}
	return append(buf, byte(x))
}

What’s happening here? Let’s break it down:

• The core logic is in a for loop that continues as long as x is greater than or equal to 128 (represented by 0x80 in hexadecimal). Why 128? That’s the highest number (in hexadecimal) that you can count to only using 7 bits. Remember that the most significant bit (MSB) is used as a continuation bit so we can only use 7 out 8 of the available bits in each byte.
  • The next line appends the uint64 argument truncated to a byte, so the last 8 bits. One of those bits isn’t actually used because of the |0x80 part of the line. This combines our extracted 7 bits with 0x80 to ensure that the continuation bit is set to true.
  • The next line uses the right shift operator to shift 7 bits off of our current uint64 value because we’ve “dealt with” these bits already. Next, we check the for-loop condition again until we get a number lower than 128.
• Finally, we append the final byte as-is because we know that it’s less than 128 which assures us that the MSB is not set, so we are done building this integer.

Next, we have the decoding code for VARINT. That looks like this (source):

// Uvarint decodes a uint64 from buf and returns that value and the
// number of bytes read (> 0). If an error occurred, the value is 0
// and the number of bytes n is <= 0 meaning:
//
//	n == 0: buf too small
//	n  < 0: value larger than 64 bits (overflow)
//	        and -n is the number of bytes read
func Uvarint(buf []byte) (uint64, int) {
	var x uint64
	var s uint
	for i, b := range buf {
		if i == MaxVarintLen64 {
			// Catch byte reads past MaxVarintLen64.
			// See issue https://golang.org/issues/41185
			return 0, -(i + 1) // overflow
		}
		if b < 0x80 {
			if i == MaxVarintLen64-1 && b > 1 {
				return 0, -(i + 1) // overflow
			}
			return x | uint64(b)<<s, i + 1
		}
		x |= uint64(b&0x7f) << s
		s += 7
	}
	return 0, 0
}

• It loops through each byte:
  • Checks for overflow (reading past expected max size).
  • Handles the last byte (if the continuation bit is 0):
    • Validates it and returns the decoded value.
  • Processes continuation bytes:
    • Extracts data bits and combines them with the accumulated value, shifting them to their correct position. The expression b&0x7f ensures that the continuation bit is not included in the result by using a bitwise AND with the current byte and 0x7f, which looks like this in binary 01111111.
    • s keeps track of the total bits already processed. It increments by 7 because that’s how many bits we can use for each byte because of the continuation bit.
• If the loop finishes without a valid ending, it signals an error.

As a short aside, you should probably know that the modern protobuf library doesn’t use these functions from the encoding/binary standard library package. I suspect that they were initially used but the current protowire implementation doesn’t use for loops at all because it has performed an optimization technique called loop unrolling. Here’s AppendVarint and ConsumeVarint that are now used in the modern protobuf library. That code looks insane but it is likely a bit faster than the version of the code that I just showed you.

As a test for comprehension of this last section, you should now understand that smaller numbers take up less room on the wire. That’s true for all numeric types except for the fixed-length types: I64 and I32. Encoding 20 in protobbuf takes a single byte on the wire but 1234 would take two bytes.

Integers

Now that we know how the VARINT wire-type works we now have enough raw material to write some protobuf packets. Again, we need three things to make a message with a single field: Field number, wire type, and the encoded value. As mentioned earlier, the field number and wire type are merged into a single VARINT value using the formula: (field_number << 3) | wire_type. This essentially means that the the least significant bits are reserved for the wire type and the rest are used for the field number, and we expand using the Base-128 varint method above if the field number needs more bits to be represented. Now, let’s write a full field in protobuf! Here’s how we encode a message with a single int32 field:

1: 1234

var buf []byte
buf = protowire.AppendVarint(buf, uint64(1<<3)|uint64(0))
buf = protowire.AppendVarint(buf, 1234)

And… that’s it. We’ve fully encoded probably the simplest (non-empty) protobuf message. We can check that it works by running a test with the “real” protobuf library:

func TestEncodeRaw(t *testing.T) {
	var buf []byte
	buf = protowire.AppendVarint(buf, uint64(1<<3)|uint64(0))
	buf = protowire.AppendVarint(buf, 1234)

	res := gen.TestMessage{}
	require.NoError(t, proto.Unmarshal(buf, &res))
	assert.Equal(t, int32(1234), res.IntValue)
}

By the way, these tests are using a protobuf file that looks like this:

syntax = "proto3";

package example;

message TestMessage {
  // Basic types
  int32 int_value = 1;
  uint64 uint_value = 2;
  bool bool_value = 3;
  string string_value = 4;

  // Repeated fields (varint encoding)
  repeated int32 repeated_int_value = 10;

  // Nested message
  message NestedMessage {
    string nested_string = 1;
  }
  NestedMessage nested_message = 11;
}

All of the field numbers and corresponding types in this article match up to this protobuf file.

Okay, let’s show what it looks like to read this message:

tagNumber, protoType, n := protowire.ConsumeTag(b)
...
b = b[n:]
i, n := protowire.ConsumeVarint(b)

Note that I omitted error handling and in a real scenario we need to check the tagNumber and protoType to decide how to read this field.

Strings/byte arrays

Next, we’re going to start with strings and byte arrays. These use the LEN wire type. This type is a little more complex than VARINT. It’s composed of a VARINT that tells us the size of the LEN field followed by the actual content in bytes. For LEN this content can be a string, a byte array, an embedded message or packed repeated fields. Here’s what a field looks like:

• Field Tag Byte (field number along with the field type, set to 2 for the LEN type)
• Byte size of our content as a VARINT
• The actual content

Encode:

var buf []byte
buf = protowire.AppendTag(buf, protowire.Number(4), protowire.BytesType)
buf = protowire.AppendString(buf, "Hello World!")

Decode:

tagNumber, protoType, n := protowire.ConsumeTag(b)
b = b[n:]
i, n := protowire.ConsumeVarint(b)

Integer Arrays (Packed)

Packed repeated fields are a space-saving optimization for integer arrays. It’s a feature that’s enabled by default for most repeated scalar types when using the proto3 syntax. Instead of encoding each element individually as a separate field/value pair an entire array is packed into a single LEN type. Therefore it looks like the following for repeated int32 types:

• VARINT of our field number with the three least significant bits being reserved for field type for LEN which is 2. This is the
• The raw integer data, also encoded with VARINT for our int32 type, one after another

Here’s an example of how to encode a packed repeated integer field:

arr := []int32{100002130, 2, 3, 4, 5}
var buf, buf2 []byte
buf = protowire.AppendVarint(buf, uint64(10<<3)|uint64(2))
for i := 0; i < len(arr); i++ {
	buf2 = protowire.AppendVarint(buf2, uint64(arr[i]))
}
buf = protowire.AppendVarint(buf, uint64(len(buf2)))
buf = append(buf, buf2...)

Notice that we are writing the list of int32 values to a second buffer. I do this so that we can know the size of the encoded/packed int32 values so that I can properly set the size for the encapsulating LEN wire type.

Decoding a packed repeated field is similar. First, you read the field tag (like always). Then we are using protowire.ConsumeBytes that reads the entire LEN value into a byte array. Then you read varint values until you exhaust the buffer. Remember that varint values have that continuation but so the protowire.ConsumeVarint function knows when to finish reading each varint value.

Here’s what that looks like in code:

tagNumber, protoType, n := protowire.ConsumeTag(b)
...
b = b[n:]
int32buf, n := protowire.ConsumeBytes(b)
b = b[n:]
...
res := []int32{}
for len(int32buf) > 0 {
	v, n := protowire.ConsumeVarint(int32buf)
	int32buf = int32buf[n:]
	res = append(res, int32(v))
}

Conclusion

In this part of the series, we’ve taken a deep dive into the world of manual protobuf encoding. We explored a few wire types and the primitives that they are built on, like Base-128 varints, and built functions to encode and decode basic protobuf data types. While for most use cases, you’ll probably want to leverage the efficiency and convenience of generated code and libraries like golang/protobuf, understanding how to manually encode these messages can be a valuable asset for debugging, creating a protobuf library implementation in your favorite new language or for possibly creating your own binary encoding that’s better in some way than protobuf.

In the next part of the series, we will put more pieces together with a layer on top of our encoding code and integrate this protobuf library into the client and server that we made in parts 1 and 2. Build a real gRPC client and server that uses protobufs for data exchange!

One crucial optimization for internet-facing gRPC services is customizing the behavior towards unknown fields. I’ve talked about unknown fields in a previous post, so read that one if unknown fields are still a mystery to you and then come back here. By default, protobuf messages can contain fields that are not defined in the current version of the proto schema. While convenient for development and can help with forward compatibility, this poses a security risk in a public environment.

Here’s why you should consider dropping unknown fields when exposing gRPC to the internet:

• Preventing Malicious Data: Unknown fields can be exploited by malicious actors to inject unexpected data into your service. This could lead to potential security vulnerabilities like code injection or unexpected behavior.
• Ensuring Compatibility: Uncontrolled unknown fields can cause compatibility issues if your clients are using different versions of the proto schema. Dropping them enforces stricter adherence to the defined message format.
• Improving Performance: Skipping unknown fields during message parsing can lead to performance gains, especially when dealing with large datasets.

How to Drop Unknown Fields

Here is how you can drop unknown fields while using the standard proto.UnmarshalOptions struct provided by the google.golang.org/protobuf/proto package. Here’s how to do it in your Go code:

import (
	"google.golang.org/protobuf/proto"
	...
)

// Configure unmarshalling options to discard unknown fields
opts := proto.UnmarshalOptions{
	DiscardUnknown: true,
}

// Use the options when unmarshalling incoming messages
msg := &MyMessage{}
err := proto.Unmarshal(data, msg, opts)
if err != nil {
	// Handle error
}

By setting the DiscardUnknown field to true in the proto.UnmarshalOptions struct before unmarshalling incoming messages, you ensure that any unknown fields are ignored. This helps mitigate the security risks associated with unknown fields while processing internet-facing gRPC requests.

How to Drop Unknown Fields in Connect RPC Servers

package main

import (
	"log"
	"net/http"

	"golang.org/x/net/http2"
	"golang.org/x/net/http2/h2c"
	"go.akshayshah.org/connectproto"
)

func main() {
	greeter := &GreetServer{}
	mux := http.NewServeMux()
	path, handler := greetv1connect.NewGreetServiceHandler(
		greeter,
		// Add an option that customizes protobuf marshalling/unmarshalling behavior
		connectproto.WithBinary(
			proto.MarshalOptions{},
			proto.UnmarshalOptions{DiscardUnknown: true},
		),
		// Add an option to customize JSON marshalling/unmachalling
		connectproto.WithJSON(
			protojson.MarshalOptions{},
			protojson.UnmarshalOptions{DiscardUnknown: true},
		)
	)
	mux.Handle(path, handler)
	log.Fatal(http.ListenAndServe(
		"localhost:9000",
		h2c.NewHandler(mux, &http2.Server{}),
	))
}

In this example, connectproto.WithBinary ensures only messages with defined fields are processed, enhancing the security of your gRPC service. connectproto.WithJSON does the same thing but with JSON.

Additional Considerations

While dropping unknown fields is a valuable security practice, it’s important to consider potential trade-offs:

• Backward compatibility: Clients using older versions of the proto schema will encounter errors if they rely on previously defined unknown fields.
• Logging and Debugging: Dropping unknown fields might make it harder to identify the source of unexpected behavior during development or debugging.

In such cases, it’s recommended to document these trade-offs and have a clear versioning policy for your gRPC service and client applications.

Conclusion

Exposing gRPC services to the internet requires careful security considerations. By customizing protobuf encoding options, specifically by dropping unknown fields using proto.UnmarshalOptions, you can significantly improve the security posture of your service. Remember to weigh the benefits against potential drawbacks and implement a solution that aligns with your specific needs.

However, all hope is not lost. Even if you just have a binary protobuf file with no knowledge of the corresponding protobuf file we can still get some information out of it. Let me introduce a tool called Protoscope. Protoscope is a tool for inspecting protobuf binary. It can do this with or without the protobuf files or the equivalent descriptor set (but it can do a better job with the protobuf data).

Install Protoscope

Okay, let’s hit the ground running. Here’s how to install protoscope (requires go).

go install github.com/protocolbuffers/protoscope/cmd/protoscope@latest

Using Protoscope

If you have a binary protobuf file, here’s what you can run to get protoscope output:

# protoscope [filename]
# variety.pb contains binary protobuf content.
$ protoscope -explicit-wire-types variety.pb
1:LEN {"Hello World!"}
2:VARINT 3
3:VARINT 175
4:LEN {
  `736563726574fa43bcab0ddfd7f3582699331e7ebbe267196804216a885fad5a3b0b01da25577220`
  `f583a5ac18b5c28516de341db3a7b44226e21ed85a6cdb571019fbee016574b8b99cd4ceab728ddd`
  `34a3e0b54605f7c7d1181ee3e13f4d9a07655f6ec843e74a997fd4b8ab87dc61754a60bd513d0121`
  `e4ad1fdc9e07a632`
}
5:LEN {`01020304`}
6:LEN {
  1:LEN {"Fluffy"}
  2:VARINT 1
}
7:VARINT 921
8:VARINT 1
9:I32 1.2345i32   # 0x3f9e0419i32
10:LEN {
  1:LEN {"key"}
  2:LEN {1:LEN {"Fred"}}
}
19:LEN {`ffffffffffffffffff01feffffffffffffffff01fdffffffffffffffff01fcffffffffffffffff01`}

# Often times you might see binary files encoded with hexadecimal.
$ cat variety.pb.hex
0a0c48656c6c6f20576f726c6421100318af01228001736563726574fa43bcab0ddfd7f3582699331e7ebbe267196804216a885fad5a3b0b01da25577220f583a5ac18b5c28516de341db3a7b44226e21ed85a6cdb571019fbee016574b8b99cd4ceab728ddd34a3e0b54605f7c7d1181ee3e13f4d9a07655f6ec843e74a997fd4b8ab87dc61754a60bd513d0121e4ad1fdc9e07a6322a0401020304320a0a06466c75666679100138990740014d19049e3f520d0a036b657912060a04467265649a0128ffffffffffffffffff01feffffffffffffffff01fdffffffffffffffff01fcffffffffffffffff01

# We can use xxd to convert it to binary then pipe it into protoscope
$ xxd -r -ps variety.pb.hex | protoscope -explicit-wire-types
... [same as above] ...

Note that with the examples above protoscope needs to guess types because we did not pass in the -descriptor-set and -message-type options. Why does protoscope need to guess types? One thing that will help with understanding this topic is knowing that protobuf encoding only has 6 types. And two of them aren’t even used in the latest version. You may be saying to yourself “I remember seeing a table of protobuf types and it had way more than 6!” and you would be correct. Although protobufs support many types they are all serialized into 6 “wire types.”:

How protobuf encodes each protobuf type into a wire type differs depending on the type. The full explanation exists in the programming guide for the protobuf encoding. I recommend reading through it to fully understand the implications of using the protoscope tool.

Strings

Let’s see what it looks like with a trivial example that is sourced from connectrpc.eliza.v1.SayRequest (protoscope -explicit-wire-types eliza.SayRequest.pb):

1:LEN {"Hello World!"}

What does this tell us? Well, it tells us the message has a single field with field number 1. It also tells us the value of the string is "World". That’s pretty incredible compared to the nothing we knew about this collection of bytes a second ago! Let me quickly explain protobuf field numbers. The 1 in this example is a field number and it corresponds to the protobuf field number. For repeated values, you may see this number appear multiple times. But notice how the name is completely missing. That’s because protobuf doesn’t want to waste resources transmitting or storing metadata like that. It would be fair to summarize protobuf as a list of key/value pairs. The key is the field number and the values are one of the basic types in protobuf.

For the record, here’s what connectrpc.eliza.v1.SayRequest looks like:

message SayRequest {
	string sentence = 1;
}

Disclaimer: Protoscope has to guess types in a lot of instances because the protobuf encoding has only 6 wire types (two deprecated): VARINT, I64, LEN, and I32 are used with modern protobuf files. In the example above, field 1 could have been a string, byte array, an embedded message, or packed repeated fields. Protoscope guessed that it was a string and showed it to us as a string. It can guess wrong.

More Strings

Okay, now we’re going to look at a message derived from a different type: connectrpc.eliza.v1.IntroduceRequest (protoscope -explicit-wire-types eliza.IntroduceRequest.pb):

1:LEN {"Hello World!"}

Wait, what? It’s the exact same? Yep. If the field numbers and types match there’s no distinguishable difference when protobuf is encoded into binary. Here’s the protobuf type:

message IntroduceRequest {
	string name = 1;
}

Notice the message contains a single string field which is similar to `SayRequest`` above, but there are a few notable differences. The message name and the field name are different but since those two things are never transmitted over the wire with protobuf we can’t tell the difference between these two message types without prior knowledge. This flexibility allows you to make certain significant changes to your protobuf file without changing what is encoded… but you do have to follow some rules. These rules make more sense with more knowledge of the protobuf encoding.

Bytes

Okay, now let’s look at a new type: bytes. Let’s take a look at what that looks like with a byte array (protoscope -explicit-wire-types bytes.pb)):

4:LEN {
  `7365637265745b060fd327e7efb49cb89b479b65b71043859c2bafbd2d712fbcea2a759b230ceed4`
  `af7177eef821cd43935bdc74b682aa939ad99379b6a0e9c4e156b42691bf5e7cb7c8194eea230de4`
  `8981314872d7286920d6c5d2799546ce6131391ecd75edf27c17f413e257f50f9834454566c3439d`
  `7d2e52204aa57ba7`
}

What you’re seeing here is a hexadecimal representation of the bytes in our field. In this example, we mostly have random data. But if you pass this hexadecimal text through a hex-to-string converter you may notice that the beginning text, 736563726574, decodes to secret. Even though there is ASCII string content in the byte array, protoscope still treats this as a byte array, not a string. Here’s a byte array with a string that says Hello World! as the content (protoscope -explicit-wire-types bytes2.pb):

4:LEN {"Hello World!"}

Wait, what? Protoscope renders the text as text! What gives? Protoscope is, again, guessing the type of the data. It notices that all of the included bytes are in the ASCII range so it renders the content as text. Could this be the wrong thing to do? Maybe!

Numbers

Now let’s at numbers represented in protobuf. You may see some… odd things. We’ll break it down field by field (protoscope -explicit-wire-types numbers.pb).

2:VARINT 3
3:VARINT 175
5:LEN {`01020304`}
7:VARINT 921
8:VARINT 1
9:I32 1.2345i32   # 0x3f9e0419i32
19:LEN {`ffffffffffffffffff01feffffffffffffffff01fdffffffffffffffff01fcffffffffffffffff01`}

Refer to this table to see the actual protobuf types and the intended values. You will notice that several of them don’t match up with what protoscope outputs at all.

• 2: Enum fields are encoded as numbers on the wire. because of this, the name of the enum value may be unknown to you without the protobuf file.
• 3: uint32 types look like what you’d expect! Nice.
• 5: This is the first super weird one. Why is it shown as a string? This has to do with packed repeated fields. The protobuf encoding packs repeated primitive types into a single LEN field (instead of using VARINT, I64 or I32 as normal). Therefore, protoscope may simply represent this as a string or byte array because it can’t tell the difference on the wire.
• 7: int64 types also look like what you’d expect! Nice.
• 8: Booleans are encoded as false = 0 and true = 1.
• 9: In this case, protobufs treated the float correctly and we get the correct value.
• 19: This has to be the weirdest case. This looks so strange because it’s a result of using repeated int32 type, which is packed with negative values. Protoscope thinks this value looks like a LEN wire type with binary data in it. It guessed the type incorrectly this time.

Submessages and Maps

In protobuf you can put messages instead of other messages, so let’s look at what that looks like from protoscope’s perspective (protoscope -explicit-wire-types submessages.pb):

6:LEN {
  1:LEN {"Harey"}
  2:VARINT 1
}
14:LEN {
  1:LEN {"buf.build/connectrpc/eliza/connectrpc.eliza.v1.SayRequest"}
  2:LEN {1:LEN {"Hello World!"}}
}
17:LEN {}
18:LEN {1:LEN {"alpha"}}
18:LEN {1:LEN {"beta"}}

From the protoscope output above you might also notice that we have two field 18 values. That is because submessages cannot be packed as primitive types can. The “unpacked” way of representing a repeated value in the protobuf encoding is to simply write the field multiple times with different values. Simple. So field 18 is likely a repeated submessage field.

Now let’s look at maps: (protoscope -explicit-wire-types maps.pb)

10:LEN {
  1:LEN {"key"}
  2:LEN {
    1:LEN {"Corgi"}
    2:VARINT 1
  }
}
20:LEN {
  1:LEN {"Knock, knock"}
  2:LEN {"who's there?"}
}
20:LEN {
  1:LEN {"Java"}
  2:LEN {"Coffee, not code."}
}

Wait, what? This looks a lot like submessages! There’s a reason for that! Maps ARE submessages in the protobuf encoding. Here’s basically what the encoder is doing.

A map that looks like this:

message TestWithMap {
  map<string, int32> name_to_age = 7;
}

… is converted into a submessage that looks like this:

message TestWithMap {
  message name_to_age_Entry {
    optional string key = 1;
    optional int32 value = 2;
  }
  repeated name_to_age_Entry name_to_age = 7;
}

This shows that protobuf is a very practical encoding that re-uses basic concepts to support more complex structures.

Summary

In this blog post, we delved into the intricacies of inspecting binary Protobuf messages using the Protoscope tool. We highlighted its ability to decipher binary data even without the corresponding Protobuf files. We also covered the six wire types used in Protobuf encoding and explored various scenarios involving strings, bytes, numbers, submessages, and maps.

I didn’t cover all of the weird edge cases. There are features in the, now deprecated, proto2 format that I didn’t show. However, I hope that I’ve shown that you can get something from a binary protobuf file. This, alone, is quite impressive for a binary format. You would usually have a very hard time understanding anything without knowledge of the specific binary protocol. This demonstrates how protobufs takes some of the benefits you might get from text-based encodings (composability, support for “unknown” fields, some amount of discoverability) with the performance of binary formats (speed, reduced size) but protobuf does bring in an extra ingredient: contracts. Because protobuf files are the source of truth for the format and type-safe serialization code, gRPC client code, gRPC server code, and documentation can all be generated from protobuf files this shows the strength of the format… which is why you should try to never be in a situation where you NEED to use protoscope. You should always have a descriptor set or the protobuf files nearby to decode these messages.

For a more extensive overview of the protobuf binary encoding refer to the official documentation.