Usage without generic methods

Here’s how a user uses heap today:

package main

import (
	"container/heap"
	"fmt"
)

type IntHeap []int

func (h IntHeap) Len() int           { return len(h) }
func (h IntHeap) Less(i, j int) bool { return h[i] < h[j] }
func (h IntHeap) Swap(i, j int)      { h[i], h[j] = h[j], h[i] }
func (h *IntHeap) Push(x any)        { *h = append(*h, x.(int)) }

func (h *IntHeap) Pop() any {
	old := *h
	n := len(old)
	x := old[n-1]
	*h = old[0 : n-1]
	return x
}

func main() {
	h := &IntHeap{2, 1, 5}
	heap.Init(h)
	heap.Push(h, 3)
	fmt.Printf("minimum: %d\n", (*h)[0])
	for h.Len() > 0 {
		fmt.Printf("%d ", heap.Pop(h))
	}
}

And, in case you missed it, Push, Pop, and so on use any. So, you could write heap.Push(h, 3) and then heap.Push(h, "foo") and it’d compile! But you’d get a panic at runtime:

$ go build .
# aww
$ go run .
panic: interface conversion: interface {} is string, not int

Overall, it’s definitely usable and great not to have to re-implement heap bubbling, but the UX is a bit clunky. It’s not great to lose types, and it’s not great to have to implement 5 functions every time I have a different type of heap.

Usage with generic methods

With generic methods, a differently-defined heap (which we’ll look at shortly) could just be:

package main

import (
	"container/heap"
	"fmt"
)

func main() {
	h := &heap.Heap[int]{2, 1, 5}
	h.Init()
	h.Push(3)

	fmt.Printf("minimum: %d\n", (*h)[0])
	for h.Len() > 0 {
		fmt.Printf("%d ", h.Pop())
	}
	fmt.Println()
}

So much easier! No more having to define 5 methods per type: we just declare a heap[] and use it.

And, we’re using generics instead of any, so we have types on our methods. If we try to do h.Push("foo"), we get a nice compiler error:

$ go build .
# example.com/foo
./main.go:20:9: cannot use "foo" (untyped string constant) as int value in argument to h.Push
# yay!

Implementing heap with generic methods

And finally, here’s how the heap package looks with generic methods. Mostly the same as it is, except we define Push and Pop directly on a heap type rather than as free functions that take an interface. And, cmp aleady gives us cmp.Ordered as the type constraint, so we can just use that directly. The bubbling logic remains the same:

import "cmp"

type Heap[T cmp.Ordered] []T

func (h Heap[T]) Len() int           { return len(h) }
func (h Heap[T]) Less(i, j int) bool { return h[i] < h[j] }
func (h Heap[T]) Swap(i, j int)      { h[i], h[j] = h[j], h[i] }

func (h *Heap[T]) Init() {
	n := len(*h)
	for i := n/2 - 1; i >= 0; i-- {
		h.down(i, n)
	}
}

func (h *Heap[T]) Push(x T) {
	*h = append(*h, x)
	h.up(len(*h) - 1)
}

func (h *Heap[T]) Pop() T {
	n := len(*h) - 1
	h.Swap(0, n)
	h.down(0, n)
	x := (*h)[n]
	*h = (*h)[:n]
	return x
}

func (h *Heap[T]) Remove(i int) T {
	n := len(*h) - 1
	if n != i {
		h.Swap(i, n)
		if !h.down(i, n) {
			h.up(i)
		}
	}
	x := (*h)[n]
	*h = (*h)[:n]
	return x
}

func (h *Heap[T]) Fix(i int) {
	if !h.down(i, len(*h)) {
		h.up(i)
	}
}

func (h *Heap[T]) up(j int) {
	for {
		i := (j - 1) / 2
		if i == j || !h.Less(j, i) {
			break
		}
		h.Swap(i, j)
		j = i
	}
}

func (h *Heap[T]) down(i0, n int) bool {
	i := i0
	for {
		j1 := 2*i + 1
		if j1 >= n || j1 < 0 {
			break
		}
		j := j1
		if j2 := j1 + 1; j2 < n && h.Less(j2, j1) {
			j = j2
		}
		if !h.Less(j, i) {
			break
		}
		h.Swap(i, j)
		i = j
	}
	return i > i0
}

So, I often find myself starting applications by writing graceful shutdown logic. And since most of the application programming that I do uses HTTP and gRPC servers, I end up writing graceful shutdown for those fairly frequently.

I thought I’d codify that here for myself, and for anyone else interested in seeing how to gracefully shut down your gRPC and HTTP servers in Go.

The key parts are:

1. We use signal.NotifyContext to attach SIGINT/SIGTERM to context cancellation.
2. We then feed that context to an errgroup.WithContext to create context cancellation aware goroutines.
3. We use that errgroup to spawn goroutines to serve HTTP/gRPC.
4. We use that errgroup to spawn HTTP/gRPC goroutines that watch for context cancellation and begin graceful shutdown.
   • Apart from the cancellation provided by signal.NotifyContext, any other reason for cancellation also triggers graceful shutdown. So this is a nice generic way to wire up your application for shutdown due to any reason you might care to express.
5. We bound graceful shutdown to a timeout, after which we forcefully shut down.
   • This part is take it or leave it: Kubernetes will send SIGKILL after whatever grace period it allows. So, you could just use that mechanism. I personally prefer to own shutdown as much as I can, though, so that I can make sure to call all the logs/metrics/tracing flushing.

package main

import (
	"context"
	"errors"
	"flag"
	"fmt"
	"log/slog"
	"net"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"

	"golang.org/x/sync/errgroup"
	"google.golang.org/grpc"

	pb "google.golang.org/grpc/examples/helloworld/helloworld"
)

var httpPort = flag.Int("httpPort", 7000, "The port to serve HTTP on")
var grpcPort = flag.Int("grpcPort", 8000, "The port to server gRPC on")

const httpShutdownTimeout = 5 * time.Second // How long HTTP shutdown is allowed to take.
const grpcShutdownTimeout = 5 * time.Second // How long gRPC shutdown is allowed to take.

func main() {
	flag.Parse()

	// SIGINT/SIGTERM signals to our application will result in the ctx being cancelled.
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	if err := run(ctx); err != nil {
		slog.Error("application exited with 1", "error", err)
		// Make sure to flush logs/metrics/traces/etc by this point! And keep in
		// mind that defer doesn't run when you os.Exit(1), so you might need
		// to invoke them manually depending on where you performed init.
		os.Exit(1)
	}
	slog.Info("application exited with 0")
}

func run(ctx context.Context) error {
	// Start with port binding.
	httpListener, err := net.Listen("tcp", fmt.Sprintf(":%d", *httpPort))
	if err != nil {
		return fmt.Errorf("http listen on port %d failed: %v", *httpPort, err)
	}
	defer httpListener.Close()

	grpcServerListener, err := net.Listen("tcp", fmt.Sprintf(":%d", *grpcPort))
	if err != nil {
		return fmt.Errorf("grpc listen on port %d failed: %v", *grpcPort, err)
	}
	defer grpcServerListener.Close()

	eg, gCtx := errgroup.WithContext(ctx)

	// HTTP server.
	mux := http.NewServeMux()
	mux.HandleFunc("/foo", func(w http.ResponseWriter, _ *http.Request) {
		if _, err := fmt.Fprintln(w, "bar"); err != nil {
			slog.Error("/foo failed during write", "error", err)
		}
	})
	httpServer := &http.Server{Handler: mux}

	// Start HTTP serve and shutdown goroutines.
	eg.Go(func() error { // HTTP serve.
		slog.Info(fmt.Sprintf("HTTP server starting at port :%d", *httpPort))
		if err := httpServer.Serve(httpListener); err != nil && !errors.Is(err, http.ErrServerClosed) {
			slog.Error("HTTP server failed", "error", err)
			return err
		}
		slog.Info("HTTP server shut down")
		return nil
	})
	eg.Go(func() error { // HTTP graceful -> forceful shutdown.
		<-gCtx.Done()
		shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), httpShutdownTimeout)
		defer shutdownCancel()
		slog.Info("HTTP server gracefully shutting down")
		if err := httpServer.Shutdown(shutdownCtx); err != nil {
			slog.Error("HTTP server shutdown failed. forcefully shutting down")
			return errors.Join(err, httpServer.Close())
		}
		return nil
	})

	// gRPC server.
	grpcServer := grpc.NewServer()
	pb.RegisterGreeterServer(grpcServer, &fooService{})

	// Start gRPC serve and shutdown goroutines.
	eg.Go(func() error { // gRPC serve.
		slog.Info(fmt.Sprintf("gRPC server starting at port :%d", *grpcPort))
		if err := grpcServer.Serve(grpcServerListener); err != nil {
			slog.Error("gRPC server failed", "error", err)
			return err
		}
		slog.Info("gRPC server shut down")
		return nil
	})
	eg.Go(func() error { // gRPC graceful -> forceful shutdown.
		<-gCtx.Done()
		stopped := make(chan struct{})
		go func() {
			slog.Info("gRPC server gracefully shutting down")
			grpcServer.GracefulStop()
			close(stopped)
		}()
		select {
		case <-stopped:
		case <-time.After(grpcShutdownTimeout):
			slog.Error("gRPC server forcefully shutting down")
			grpcServer.Stop()
			return fmt.Errorf("gRPC server was unable to gracefully shut down")
		}
		return nil
	})

	// This returns nil if shutdown was successful, or the first err for any
	// shutdown issue.
	return eg.Wait()
}

type fooService struct {
	pb.UnimplementedGreeterServer
}

Let’s run it and send it a SIGINT:

$ go run .
2026/08/18 09:19:39 INFO HTTP server starting at port :7000
2026/08/18 09:19:39 INFO gRPC server starting at port :8000
^C2026/08/18 09:19:40 INFO gRPC server gracefully shutting down
2026/08/18 09:19:40 INFO HTTP server gracefully shutting down
2026/08/18 09:19:40 INFO HTTP server shut down
2026/08/18 09:19:40 INFO gRPC server shut down
2026/08/18 09:19:40 INFO application exited with 0

Each language maintained a style guide: a list of best practice and style decisions, curated by the hundreds or thousands of readability experts for that language.

I was a C++ and Go readability reviewer, and contributed some tidbits to the Go styleguide, as well as to many discussions about how to write Go. The styleguide was a great place to put best practices for the language, but also a great place to go to learn, since best practices were usually followed with an explanation. That way, newcomers to the language could ask for a change and simultaneously reference the styleguide as the high-quality explanation for the change request.

Google has very generously open sourced the styleguides for its languages at https://google.github.io/styleguide/. I highly encourage practitioners of any of the languages there to give them a read. I can greatly recommend the C++ and Go ones, but I’m sure the others are good too.

For the last 6 months at Netflix, when asking Claude to write code, I’ve found it varying degrees of helpful to have Claude read those styleguides, and apply edits based on advice in them. It’s not been perfect - far from it, unfortunately. But it’s generally been better than not having it do so.

So, I thought I’d open source it: https://github.com/jeanbza/claude-extensions.

To install it, simply run the following inside a Claude session:

/plugin marketplace add jeanbza/claude-extensions
/plugin install google-go-review@jeanbza
/reload-plugins

To use it, simply run /jeanbza:google-go-review (or cpp, or py - the other two I’ve added so far).

There’s more for me to do, in terms of prompt engineering: Claude frequently does a poor job at “Handle errors” for example. (My pet theory is that Claude doesn’t bother to check the signatures) I’ll keep working on it, both for myself and whomever else wants to use it. I’m also working on a project to go a bit lower down the stack and try to inject some evaluators into its reward model with lifecycle hooks. But for now, I hope it’s helpful to you as it’s been for me!

For example, a common LSC is to migrate all codebases at a company from libraryX@v1 to libraryX@v2, changing code along the way to account for differences in API between v1 and v2. Due to their breadth, LSCs have high cognition cost, owing to the high amount of contexts you have to reason about. But also due to their breadth, they tend to have bounded problem spaces: the wider a change has to be applied, the more generic the change must be. A change that can’t be generically expressed is not an LSC, but instead a series of individual changes.

AI agents are one way to attempt to tackle LSCs. Goals can be expressed generically and handed to AIs as prompts to work on all target codebases. But, when AI agents are set loose to perform LSCs, LSCs’ high cognition cost compounds with AI non-determinism to result in low confidence in the changes being applied, incur high costs, and produce results which are unauditable.

This article presents a different approach, based on a simple observation: most changes that can be expressed generically can also be expressed mechanically. AI agents explore and solve the LSC’s problem space in an evaluation flywheel, and encode their results in a metaprogramming program which itself is used to perform the LSC in a way that is deterministic, leads to higher confidence, is cheap, and is as auditable as ordinary source code.

Context

At Google and now at Netflix, I find myself frequently having to perform large scale changes to code: changes to hundreds or thousands of codebases. For example, at Google, when I rewrote the internal Go status library (similar external version here) to support Go 1.13 error wrapping/unwrapping, Google’s internal one-version-of-every-library + monorepo setup meant that I had to also go make this work for ~56k Go programs. That involved making changes to hundreds of programs with non-conforming / Hyrum’s-Law type oddities at the same time.

Most recently at Netflix, I’ve been working on problems requiring LSCs over our thousands of Go projects. An example recent, long-running LSC is periodic remediation of vulnerable dependencies.

Doing this kind of work in 2026 is dramatically faster with AI Agents, but surprisingly there’s a good lesson to be learned from the old way of doing it which can be paired with AI agents.

Before AI agents, we used to perform these changes by hand, and quickly learned instead to build metaprogramming programs: programs which modified programs.

With AI agents, it’s natural to think to replace this with hordes of AI sub-agents performing code change on each repository, with some shared prompt or goal. However, it turns out to still be more advantageous to write a program for the LSC, albeit now with an AI agent to explore the problem space and encode the solution into that program.

The end result is a small program which, when run on all the target repositories, solves the LSC goal in a deterministic, cheap, and auditable manner.

Let’s take a look in closer detail.

Evaluation framework

This technique starts with an evaluation framework and a discovery / actuate / evaluate loop. Using the vulnerability remediation project above, the evaluation framework is largely self-evident. The goal is to remediate all Go repositories of vulnerable dependencies. For each repository,

• Go’s govulncheck reports whether there are any vulnerabilities. (Goal: 0 fixable vulnerabilities)
• The output and exit code of whatever program we write tells us whether we were successful, and if not what failed. (Goal: no failures)
• A PR’s CI/CD log tells us whether a PR to perform that remediation is valid, and if not what failed. (Goal: valid PR)

This is all we need to begin our loop.

Discovery loop

With a sufficiently large enough set of repositories (which we have at Netflix, or is generally available in GitHub), and enough compute, and enough tokens, and a bounded problem space, we can solve the problem space and produce a single solution.

We produce a solution to the problem space in the form of a program. That program remediates vulnerabilities in whatever repository it is run in.

We encode the problem space, as we discover it, as a series of txtar tests. That becomes the early signal in our evaluation framework; it doubles up as a regression test mechanism; and it triples up as audit documentation (This is how the program is built to behave under scenario X, Y, Z).

As the agent explores the space, it encodes new situations it finds as txtar tests, writes code to solve the expanded problem scope, verifies it first against the txtar test and then by attempting to send PRs to its test repository cohort, looking at failure and build logs, and re-assessing. When it reaches quiescence, it expands its test cohort exponentially.

When we applied this strategy to the vulnerability remediation problem, encoding the problem space as txtar texts, we found fewer and fewer new cases as we expanded the test pool:

Deterministic solution

The end result is a single program which can be run on a repository to deterministically upgrade dependencies to remediate vulnerabilities. And since it’s a program and not an LLM, we also benefit being able to audit its behaviour through normal source control history. Both the determinism and auditability increase confidence in its use in large scale changes considerably.

The other major benefit is cost and time: the program runs in <2s, and costs $0+a bit of CPU.

I ran a benchmark against 50 repositories, using the program in one test and an AI agent (Claude Sonnet 5, medium effort) in the other set, both tasked with remediating vulnerabilities. On average, the AI agent took ~50k¹ tokens ($0.15/run) and 4m²/run. Across ~2000 of Go repos, that’s $750/run. We run this large scale change every 6h to remediate vulnerabilities quickly: that means we’d pay ~$0.66M/yr. At Google’s scale with 30x-40x those Go repos, that’s more like $20M/yr.

And, that’s a minimum: as developers 10x their velocity, they also 10x the repos they create, which 10x this cost. The program by comparison is ~$0 and will forever stay that way.

Long tail

There may be a long tail of issues to deal with. In the cases we’ve dealt with so far, this has not been the case, but it’s fair to imagine that there might be.

Of course, two simple solutions exist:

• Run an AI agent as a backstop.
• Keep the discovery loop infrastructure and allow long-tail findings to continually improve the program.

A better solution

It’s attractive to throw AI Agents at every problem, since they’re so fast at exploring problems at producing solutions. But in the case of LSCs, the way that you apply AI Agents makes a difference.

At Netflix we’ve had good success marrying the speed and ability of AI agents to explore the unknown, along with the observation that most LSCs have bounded problem spaces whose solutions are generically and mechanically expressable, to have AI agents create LSC programs.

The result is a solution that runs in seconds instead of hours, costs nothing instead of hundreds of thousands a year, behaves the same way on the nth repository as it did on the first, and can be reviewed the way we review everything else: by reading the source and the logs.

I’ll detail that another time. For now, I’m going to focus on a much simpler way to solve this if you have lesser constraints than what we do. This is based on Postgres but would work for most relational databases. I’ve not load tested this and couldn’t really tell you how it scales, but it should be “good enough” for most queueing under ~1k QPS and <1M messages (and probably fair amount higher, but again, haven’t load tested).

Note: This was used at our golang-index as well as in Go’s pkgsite Postgres queue.

Schema

-- This is the queue table.
CREATE TABLE IF NOT EXISTS queue_tasks (
    -- Bookkeeping.
    id          BIGSERIAL PRIMARY KEY,
    
    -- Each consumer/dequeue-er needs a unique task name. appID+threadID for example.
    task_name   TEXT UNIQUE NOT NULL,
    
    -- [...] Whatever payload you need for each message goes here.

    -- Queueing.
    created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    started_at  TIMESTAMPTZ

    -- Optionally also add completed_at and maybe success/failure columns if you
    -- want a history; otherwise just delete on finish.
);
CREATE INDEX IF NOT EXISTS idx_queue_tasks_started_created
ON queue_tasks (started_at, created_at);

Enqueueing

Enqueue with an atomic test-and-set query:

INSERT INTO queue_tasks (...) -- All but created_at.
VALUES (...)
ON CONFLICT (task_name) DO NOTHING; -- Another consumer won the race; no-op.

Dequeueing

Dequeue with a test for stalled workers:

WITH next_task AS (
    SELECT id
    FROM queue_tasks
    WHERE started_at IS NULL
        -- Allow accepting work from stalled workers.
       OR started_at + INTERVAL '5 minutes' < NOW()
    ORDER BY created_at ASC
    LIMIT 1
    FOR UPDATE SKIP LOCKED
)
UPDATE queue_tasks
SET started_at = NOW()
WHERE id = (SELECT id FROM next_task)
RETURNING id, started_at

And then to complete work, you’d simply DELETE the row.

With priority

Simple add a priority field of your choosing and then ORDER BY priority instead of ORDER BY created_at.

With batch dequeue

To batch dequeue simply change LIMIT 1 to LIMIT n.

With lease extension

If you need lease extension, you can add leased_until and change the stall check to:

WITH next_task AS (
    SELECT id
    FROM queue_tasks
    WHERE started_at IS NULL
        -- Allow accepting work from stalled workers; but now we used
        -- leased_until, which workers are expected to periodically lease
        -- extend.
       OR leased_until < NOW()
    ORDER BY created_at ASC
    LIMIT 1
    FOR UPDATE SKIP LOCKED
)
UPDATE queue_tasks
SET started_at = NOW(),
    leased_until = NOW() + INTERVAL '5 minutes' -- Or however long you want leases.
WHERE id = (SELECT id FROM next_task)
RETURNING id, started_at

And then have your workers periodically update leased_until to some time in the near future.

With max consumers

If you need to limit work to a maximum amount of in-flight work/consumers, you can do:

WITH active_workers AS (
    SELECT COUNT(*) AS current_count
    FROM queue_tasks
    WHERE started_at IS NOT NULL
      AND started_at + INTERVAL '5 minutes' >= NOW()
),
next_task AS (
    SELECT id
    FROM queue_tasks
    WHERE (SELECT current_count FROM active_workers) < :n
      AND (started_at IS NULL OR started_at + INTERVAL '5 minutes' < NOW())
    ORDER BY created_at ASC
    LIMIT 1
    FOR UPDATE SKIP LOCKED
)
UPDATE queue_tasks
SET started_at = NOW()
WHERE id = (SELECT id FROM next_task)
RETURNING id, started_at

Conclusion

SQL databases are pretty easy to turn into queueing databases, and it’s fairly easy to tweak your SQL scripts to make the queue behave the way you want them to.

If you already rely on a SQL database and need to add queueing, consider just doing it in your pre-existing SQL database rather than adding a whole new system dependency.

I’ve lately been working with a new college grad on my team, and he asked a good question about how often we should be abstracting code into functions. I thought about it a while and came up with this answer. I’m somewhat satisfied by it, but if you have a better way to frame this, please reach out to let me know.

Cost

All abstractions come with a cost:

• They introduce complexity.
• They demand context switching.
  • For functions: This cost can be very high when trying to understand a high-level function that requires context switching between many nested sub-, sub-sub-, sub-sub-sub-, […] -functions.
• They increase the size of the mental map required to understand the code.
• They may make debugging harder (mostly as a result of all the above).
• They may impact build and runtime performance.
• See also: there are no zero-cost abstractions.

So, their cost should be justified by their value.

Objective value

Functions are objectively valuable in a few situations:

• When they’re used to deduplicate logic (my personal bar is logic found in 3 places, but some people go for 2).
• When they’re used to extract some logic that needs to be called from elsewhere (ex a library; although this often falls into the prior bullet).
• When they’re used to extract some logic that you want to individually test.
• (There are probably a few other objectively useful cases)

Subjective value

Functions are subjectively valuable when they’re used just for the sake of abstraction. That subjectivity is what this article is about. Let’s dive into that subjectivity.

Let’s imagine a 1000 line program whose logic has no objective reasons to introduce functions: there’s no duplication, there’s no logic that needs to be called from elsewhere, there are no tests, etc.

One type of person might prefer 0 functions: a single main func that is 1000 lines long, each line a valuable statement of business logic. Every thing the program does can immediately be grokked in perfect context. Debugging is extremely easy: you can go to the faulting line and perfectly work backwards to see how you arrived at the fault. And, it’s hard to misuse sub-functions: there are none!

Another type of person might prefer so many functions that every business logic line gets its own function, the caller of which gets its own function, the caller of which gets its own function, […], and main is just 2-5 function calls long. (This used to be a really popular thing in Javascript and Ruby, but I’ve noticed a lot of pushback on that kind of thing recently) The advantage here is that it’s really easy to get a quick, vague sense of what main does: A, B, C. Ok but what actually does A do? Oh, it does x, y, z. Ok… but what does x actually do? Oh, it does i, ii, iii, and iv. Ok….. what does i actually do? […] Oh, it sends a GET request to /foo.

So those are two extremes. In between the extremes is a spectrum.

I think the people who tend to want to understand a program completely are more on the top part of the spectrum (less abstractions to dig through when trying to understand completely) and people who tend to want to understand a program quickly are more on the bottom part of the spectrum (more abstractions to gain vague understanding quickly).

Illustrating subjectivity

To illustrate that a little more:

Person #1 wants to understand program #2 completely. Because of all the abstractions, he has to do a depth first search and collect all leaf nodes (the valuable business logic) into a mental list before he’s able to follow a path from main start to main end. (And, every time he returns, he has to perform some of that DFS again as the mapping inevitably falls out of his mind each time he context switches)

Person #2 wants to understand program #1 quickly. Because of the lack of abstractions, she has to try to map sections of code into logical groupings in her mind, and can only reason about the order of those groupings mentally. (She may make a series of notes or flow charts outside of code to help cope) (And, every time she returns, she has to perform some re-grouping as the mapping inevitably falls out of her head each time she context switches)

At Netflix, we use the spotless linter. The main gripe I have with it is that it’s not integrated with any of our IDEs. In contrast, in Go, gofmt is integrated with all my IDEs and linting happens as I program; with spotless I have to remember to run ./gradlew spotlessApply before I push my changes, or else my PR’s build will break when it gets to the lint check step.

When I’m working with jj rev chains, it’s annoying to jj edit <rev> && ./gradlew spotlessApply each and every rev before sending them all off to be CI/CD’d with jj git push --all.

So, I made this little script. I hope you’ll find it useful too:

#!/bin/bash
set -e
for i in `jj log -r "mutable()" --no-graph -T 'change_id ++ "\n"' | tac`; do
  jj edit $i
  ./gradlew spotlessApply
  ./gradlew spotbugsMain
done
jj log

Of course, sub out the for loop with whatever you want to do at each rev.

It makes use of jj’s fantastic templating language to walk the graph from oldgest to newest change, linting as it goes.

I’ve been using jj instead of git, and it’s been great. One thing I really miss using git, though, is a nice terminal prompt that I get from bash-it. So, I whipped up one for jj.

Notice: The prompt now prints |jj:tymnqrpn - readme| and |jj:sttyyssy|. The coloured short letters (ty, s) are the jj short prefix, the full 8 characters are the short rev, and the optional - readme show the bookmark associated with the rev if one exists.

I thought I’d post it here if anyone else is looking for this:

1. Create ~/.bash_it/custom/jj.plugins.bash with:

    # ~/.bash_it/custom/jj.plugins.bash
   
    # Set jj prompt to be enabled by default
    SCM_PROMPT_SHOW_JJ_PRIVATE_INFO="${SCM_PROMPT_SHOW_JJ_PRIVATE_INFO:-true}"
   
    # jj repo check
    function is_jj_repo {
    if [ -d ".jj" ]; then
        return 0
    else
        return 1
    fi
    }
   
    # jj prompt
    function jj_prompt_info {
    if [ "$SCM_PROMPT_SHOW_JJ_PRIVATE_INFO" = "true" ] && is_jj_repo; then
        local jj_info
        local short_hash
        local jj_bookmarks
   
        # Get an 8-character short hash and bookmarks
        short_hash=$(jj log -r @ --no-graph --template 'change_id.short(8)' 2>/dev/null)
        jj_bookmarks=$(jj log -r @ --no-graph --template 'bookmarks.map(|b| b.name())' 2>/dev/null)
   
        # Exit if we couldn't get a hash
        if [[ -z "$short_hash" ]]; then
        return
        fi
   
        # --- FINAL DYNAMIC HIGHLIGHTING ---
        # Get the shortest unique prefix using the template you found
        local prefix=$(jj log -r @ --no-graph --template 'self.change_id().shortest()' 2>/dev/null)
   
        # Fallback: if the command fails, default to the first character
        if [[ -z "$prefix" ]]; then
        prefix="${short_hash:0:1}"
        fi
   
        # Determine the rest of the hash by removing the prefix
        local rest_of_hash="${short_hash#$prefix}"
        local colored_hash="$MAGENTA$prefix$WHITE$rest_of_hash"
   
        # Build the base info string: always start with the hash
        jj_info="$colored_hash"
   
        # If bookmarks exist, append them
        if [[ -n "$jj_bookmarks" ]]; then
        jj_info="$jj_info - $jj_bookmarks"
        fi
   
        # Wrap in parentheses
        jj_info="($jj_info)"
   
        # --- ASTERISK LOGIC (UNCHANGED) ---
        local first_bookmark="${jj_bookmarks%% *}"
        if [[ -n "$first_bookmark" ]]; then
        local any_diff
        any_diff=$(jj diff --summary --from "${first_bookmark}@origin" 2>/dev/null)
        if [[ -n "$any_diff" ]]; then
            jj_info="$jj_info*"
        fi
        fi
   
        echo -e "${SCM_THEME_PROMPT_PREFIX}${jj_info}${SCM_THEME_PROMPT_SUFFIX}"
    fi
    }
   

2. Update ~/.bash_it/themes/<theme>/<theme>.bash (ex ~/.bash_it/themes/sexy/sexy.bash), amending the prompt_command:

    function prompt_command() {
    local scm_info=""
   
    # 1. Check if it is a JJ repo first (Priority)
    # We use the 'is_jj_repo' function defined in your plugin.
    if type is_jj_repo &>/dev/null && is_jj_repo; then
        # If JJ, show ONLY JJ info (suppress Git)
        scm_info="\[$CYAN\]\$(jj_prompt_info)"
           
    # 2. If not JJ, check if it is a Git repo
    elif [[ -n $(git branch 2> /dev/null) ]]; then
        # If Git, show " on " + Git info
        scm_info=" on \[$PURPLE\]\$(parse_git_branch)"
    fi
   
    # Build the PS1 using the selected scm_info
    PS1="\[${BOLD}${MAGENTA}\]\u \[$WHITE\]at \[$ORANGE\]\h \[$WHITE\]in \[$GREEN\]\w\[$WHITE\]${scm_info}\[$WHITE\]\n\$ \[$RESET\]"
   
    if [ "$SEXY_THEME_SHOW_PYTHON" = true ] ; then
        PS1="\[${BOLD}${WHITE}\]$(env_prompt) "$PS1
    fi
    }
   

And then you get some nice jj prompts!

I’m also partial to the following jj config edit --user:

[revsets]
log = "present(@) | ancestors(mine() & immutable_heads().., 2) | trunk()"

That does a good job showing me my commits when I jj log.

Local protos

The following covers local protos that you own.

That you expect to be imported

If you expect others to import your proto, declare option go_package and generate Go bindings to that location.

This defines the single source of truth for Go generated code for your proto, making it easy for others to use your proto without needing to generate and store the Go bindings themselves.

For example:

$ head protos/server/server.proto
syntax = "proto3";

package server;

option go_package = "github.com/username/myrepo/protos/server";
$ tree
.
|____protos
| |____server
| | |____server.pb.go
| | |____server.proto

Note: Public directories and releasing a tag. Remember that these Go bindings are intended to be import-able, so they should be in a non-internal/ directory. And, if your repo has tagged releases, remember to tag a new release when you generate for the first time, so that others can begin depending on your proto-generated code.

That you don’t expect to be imported

Even if you don’t expect others to import your proto, it’s still best to treat it as if it will be imported.

But, if you’re certain you’ll never want anybody else to import it, give it an option go_package that generates to an internal/ directory in your Go module and add a comment above explaining that it is not meant to be depended upon.

For example:

$ head protos/server/server.proto
syntax = "proto3";

package server;

// This proto is not intended to be depended upon, and as such always generates
// into an internal/ directory.
option go_package = "github.com/username/myrepo/internal/protogen/server";
$ tree
.
|____protos
| |____server
| | |____server.proto
|____internal
| |____protogen
| | |____server
| | | |____server.pb.go

Foreign protos

The following covers foreign protos that you don’t own.

WARNING: Never copy .protos. What follows provides nuance about whether or not to store Go bindings for a proto in your module. However, you should never copy and store a foreign .proto in your repo.

With a single source of truth

If a proto declares option go_package and provides its Go bindings at that location, that is the single source of truth for its generated Go code and you should import the single source of truth instead of generating and storing your own Go bindings.

If you are generating with protoc, that means you should not provide --go_opt=M for the proto.

If you are generating with buf, that means you should not include the proto in managed.override.

Looking at a concrete example

Here’s a concrete example of that:

• github.com/googleapis/google/cloud/speech/v1/cloud_speech.proto is a proto that defines option go_package = "cloud.google.com/go/speech/apiv1/speechpb;speechpb";.
• Accordingly, its generated Go code exists at github.com/google-cloud-go/speech/apiv1/speechpb.
  • Note: cloud.google.com/go is an alias for github.com/googleapis/google-cloud-go.
• If you want to use it,
  • ✅ You should import it as speechpb cloud.google.com/go/speech/apiv1/speechpb.
  • ❌ You should not generate (or store) cloud_speech.pb.go.

What if a proto declares option go_package, but I can’t import it?

The proto is incorrectly configured. You should reach out to the proto owners to have them either remove the option go_package or generate the Go bindings at the expected location.

Without a single source of truth

If a foreign proto does not declare option go_package, you have two options:

1. If you can: convince the maintainers to add option go_package, generate their Go bindings at that location, and make it available in a Go module. This is the best option.

   However, this can be a big ask for non-Go teams that aren’t used to maintaining any Go code.And, if it’s a widely used proto there may need to be considerable thought given to how to migrate all existing dependers. So, it may be more pragmatic to:

2. Otherwise: Generate and store their Go bindings in your module. This is risky, since it opens you and others up to the runtime panic described in Proto global registry.

   To mitigate that, store the Go bindings an internal/ directory. If any of your code depends on these Go bindings and is accessible to other modules (is not in main or _test package), then it should also be in an internal/ directory. See the discussion on transitive dependencies in Proto global registry.

Doing so won’t reduce your chance of running into this issue, but it does prevent anyone else accidentally depending on your copy of the Go bindings.

Proto global registry

All protocol buffers declarations linked into a Go binary are inserted into an in-memory global registry. If two protobuf declarations linked into a Go binary have the same name, then this leads to a namespace conflict. Worse, this error is a runtime error, making it easy to find its way into a deployment.

Generating Go code with protoc

Generating code with protoc is best for protos that have few or no dependencies.

Simple code generation

Imagine two simple protos:

// protos/user/user.proto
syntax = "proto3";

package user;

option go_package = "github.com/username/myrepo/protos/user";

message User {
    string name = 1;
}

// protos/server/server.proto
syntax = "proto3";

package server;

option go_package = "github.com/username/myrepo/protos/server";

import "user/user.proto";

message Server {
    repeated user.User users = 1;
}

Generate Go code (.pb.gos) from this simple proto with protoc:

$ protoc protos/**/*.proto \
    --go_out=protos \
    -I protos/ \
    --go_opt=paths=source_relative
$ tree
.
|____protos
| |____server
| | |____server.pb.go
| | |____server.proto
| |____user
| | |____user.pb.go
| | |____user.proto

Warning: This approach allows others to depend on your generated Go code.

This example signs you up to maintaining generated Go code at github.com/username/myrepo/protos/server and [...]/protos/user. Whether or not this is a good idea is discussed in “Storing generated code”. For now suffice it to say that instead of declaring option go_package you could instead provide --go_opt=M<proto>=<go import>. And, instead of generating your .pb.gos into the publicly import-able protos/, you could generate them into an internal/ directory which is not publicly import-able.

If you need to produce gRPC language bindings, add the gRPC option equivalents --go-grpc_out and --go-grpc_opt:

$ protoc protos/**/*.proto \
    --go_out=protos \
    --go-grpc_out=protos \
    -I protos/ \
    --go_opt=paths=source_relative \
    --go-grpc_opt=paths=source_relative

Foreign protos

Part of your dependency graph may include protos in other repos:

// protos/user/user.proto
syntax = "proto3";

package user;

option go_package = "github.com/username/myrepo/protos/user";

// Not in this repo. Also not repo-rooted, which we'll discuss below. We could
// change this if we owned user.proto, but if this occurred in a foreign proto
// that we can't change then we'd have to work around it.
import "non/repo/rooted/import.proto";

message User {
    string name = 1;
}

You’ll need to git clone the foreign protos and include them with -I during your protoc invocation.

You’ll may also need to account for quirks, such as:

• Dealing with imports not rooted at its repo root. Consider your code A importing B, and B importing another proto C, but not at C’s repo root. Instead, B imports C at at $REPO-ROOT/some/nested/dir. Since you don’t own B (another team / company does), you can’t change B. Get around this with -I rooted at the place the import expects, as shown below.

• Dealing with imports that do not define option go_package. When protos in your dependency graph don’t define option go_package, you’ll have to tell protoc which go_package to use with --go_opt=M<proto>=<go import path>. You’ll probably have to generate the foreign proto’s .pb.go into your module and point the aforementioned Go import path there, under the assumption that if they haven’t defined an option go_package then it’s also likely they are not generating and publishing Go bindings for you to use.

TMP=$(mktemp -d)
git clone https://github.com/foreignorg/foreignrepo.git "$TMP/foreign"
mkdir -p internal/protogen
protoc protos/**/*.proto \
    non/repo/rooted/import.proto \
    --go_out=internal/protogen \
    -I protos \
    --go_opt=Muser/user.proto=github.com/username/myrepo/internal/protogen/user \
    --go_opt=Mserver/server.proto=github.com/username/myrepo/internal/protogen/server \
    -I $TMP/path/to/expected/import/root \
    --go_opt=Mnon/repo/rooted/import.proto=github.com/foreignorg/foreignrepo/path/to/expected/import/root \
    --go_opt=paths=source_relative

Which when run would generate:

$ tree
.
|____internal
| |____protogen
| | |____server
| | | |____server.pb.go
| | |____user
| | | |____user.pb.go
| | |____non
| | | |____repo
| | | | |____rooted
| | | | | |____import.pb.go

Note that all the generated code was generated into internal/. That involved overwriting user.proto and server.proto’s option go_package with --go_opt=M. That’s because different generated versions of import.proto may not exist (directly or transitively) in any Go import graph due to Go’s global proto registry. So, internal/ is used to prevent anybody from depending on these .pb.gos. Read more on this in “Storing generated code”.

Large dependency graphs

Using protoc becomes progressively more unwieldy the larger your proto dependency graph becomes, since you have to specify all transitive dependencies at protoc time.

Let’s now look at a better tool for managing larger dependency graphs.

Generating Go code with buf

Generating code with buf is better for protos that have more than a few dependencies.

Simple code generation

Imagine two simple protos:

// protos/user/user.proto
syntax = "proto3";

package user;

option go_package = "github.com/username/myrepo/protos/user";

message User {
    string name = 1;
}

// protos/server/server.proto
syntax = "proto3";

package server;

option go_package = "github.com/username/myrepo/protos/server";

import "user/user.proto";

message Server {
    repeated user.User users = 1;
}

Generate Go code (.pb.gos) from these protos with buf by adding:

• A buf.yaml:

    # buf.yaml
    # For details on buf.yaml configuration, visit https://buf.build/docs/configuration/v2/buf-yaml
    version: v2
    lint:
      use:
        - STANDARD
    breaking:
      use:
        - FILE
    modules:
      - path: protos/
  

• A buf.gen.yaml:

    # buf.gen.yaml
    # For details on buf.yaml configuration, visit https://buf.build/docs/configuration/v2/buf-gen-yaml
    version: v2
    plugins:
      - remote: buf.build/protocolbuffers/go:v1.36.6
        out: protos
        opt:
          - paths=source_relative
  

Generate code with buf generate:

$ buf generate
$ tree
.
|____protos
| |____server
| | |____server.pb.go
| | |____server.proto
| |____user
| | |____user.pb.go
| | |____user.proto

Warning: This approach allows others to depend on your generated Go code.

This example signs you up to maintaining generated Go code at github.com/username/myrepo/protos/server and [...]/protos/user. Whether or not this is a good idea is discussed in “Storing generated code”. For now suffice it to say that instead of declaring option go_package you could instead provide --go_opt=M<proto>=<go import>. And, instead of generating your .pb.gos into the publicly import-able protos/, you could generate them into an internal/ directory which is not publicly import-able.

If you need to produce gRPC language bindings, add the gRPC plugin to buf.gen.yaml:

# buf.gen.yaml
version: v2
plugins:
  - remote: buf.build/protocolbuffers/go:v1.36.6
    out: protos
    opt:
      - paths=source_relative
  - remote: buf.build/grpc/go:v1.5.1
    out: protos
    opt:
      - paths=source_relative

Foreign protos

Part of your dependency graph may include protos in other repos:

// protos/user/user.proto
syntax = "proto3";

package user;

option go_package = "github.com/username/myrepo/protos/user";

// Not in this repo. Also not repo-rooted, which we'll discuss below. We could
// change this if we owned user.proto, but if this occurred in a foreign proto
// that we can't change then we'd have to work around it.
import "non/repo/rooted/import.proto";

message User {
    string name = 1;
}

Foreign protos that are neither in your repo nor the Buf Schema Registry (BSR) can be imported, but in a roundabout way. You’ll need to git clone them yourself and include them as modules in buf.yaml.

Before we do so, let’s quickly recap proto quirks you might run into, described above in “Generating code with protoc: Foreign protos”. We’ll need to contend with the fact that the import.proto import was not rooted at its repo root, and that import.proto does not define option go_package.

# buf.yaml
version: v2
lint:
  use:
    - STANDARD
breaking:
  use:
    - FILE
modules:
  - path: protos/
  - path: tmp/path/to/expected/import/root

The non-repo rooted import is handled by specifying the path at which the import "non/repo/rooted/..."; import works.

To handle the lack of option go_package, we’ll need to turn on managed mode in buf.gen.yaml:

# buf.gen.yaml
version: v2
plugins:
  - remote: buf.build/protocolbuffers/go:v1.36.6
    out: internal/protogen
    opt:
      - paths=source_relative
managed:
  enabled: true
  override:
    - file_option: go_package_prefix
      path: non/repo/rooted/
      value: github.com/foreignorg/foreignrepo/path/to/expected/import/root
    - file_option: go_package_prefix
      path: user/user.proto
      value: github.com/username/myrepo/internal/protogen
    - file_option: go_package_prefix
      path: server/server.proto
      value: github.com/username/myrepo/internal/protogen

Next, let’s perform the git clone and tmp/ management:

git clone https://github.com/foreignorg/foreignrepo.git tmp/foreign
buf generate

Which when run generates:

$ tree
.
|____internal
| |____protogen
| | |____server
| | | |____server.pb.go
| | |____user
| | | |____user.pb.go
| | |____non
| | | |____repo
| | | | |____rooted
| | | | | |____import.pb.go

Note that all the generated code was generated into internal/. That involved overwriting user.proto and server.proto’s option go_package with --go_opt=M. That’s because different generated versions of import.proto may not exist (directly or transitively) in any Go import graph due to Go’s global proto registry. So, internal/ is used to prevent anybody from depending on these .pb.gos. Read more on this in “Storing generated code”.

Concurrent git clones

If you have many git clone statements, consider using & and wait to run them concurrently:

git clone ... &
git clone ... &
git clone ... &
wait
buf generate