What’s your first move?

I watched this happen last week.

A Node + TypeScript service had a 15-minute CI. Unit tests with coverage, around 1,200 of them.

And the reasoning that shows up in the moment is always the same: if it’s slow, something is wrong, so it must be possible to optimize. Upgrade TypeScript to the Go version, which typechecks much faster. Swap eslint and prettier for ox, which are much faster.

None of that is false. TypeScript in Go is absurdly faster. Oxlint is faster than eslint.

And it still wasn’t going to fix anything.

TypeScript 7 really is that fast

The tsc you know was a TypeScript compiler written in TypeScript, running on Node. Version 7 is a faithful port of it to Go, shipped as a native binary, with type-checking parallelized across 4 threads.

Microsoft’s numbers are not modest: VS Code went from 125.7s to 10.6s, Sentry from 139.8s to 15.7s. In the editor, opening a file with errors dropped from ~17.5s to under 1.3s.

The emitted JavaScript is the same. The type system is the same. This is a compile-time story, full stop.

One detail if you plan to migrate: 7.0 has no programmatic API, and it only arrives in 7.1. Any tool that embeds the compiler - typescript-eslint, Volar, and by extension Vue, Svelte, Astro, Angular - stays on TypeScript 6.

The Japanese knife and the pot of beans

Amdahl’s Law explains why swapping the compiler was never going to move the needle:

p is the fraction of the work you improved, s is how much you improved it. The painful part is the limit: as s approaches infinity, the maximum speedup of the whole system becomes 1 / (1 - p).

In other words: the part you didn’t optimize is your ceiling. And it doesn’t move.

Think about Sunday lunch. It takes you 40 minutes. You buy an expensive Japanese knife that chops onions 10x faster, and the marketing wasn’t lying. Except chopping onions took 2 minutes, and now it takes 12 seconds.

Lunch still takes a little over 38 minutes.

Because what takes time is the pot of beans on the stove. And beans are not a chopping problem, they are a waiting problem. No knife solves waiting.

They did the upgrade

Everything went up at once: eslint 8.57 -> oxlint 1.76, prettier 3.0 -> oxfmt 0.61, TypeScript 5.9 -> TypeScript 7.

The new engines delivered the promised 6x to 10x. tsc 7 compiles the entire project in 1.22s. What eats the difference is the fixed overhead of each invocation: spinning up the process and loading the tool costs more than the actual work.

Net result: 5.91 seconds. Out of 900.

And note that those four steps together were 21s in a 900s pipeline, which puts p at 2.3%. If the tools were infinitely fast, zero cost:

In the impossible scenario, CI drops from 15m00s to 14m39s. That was the ceiling, and it was available before a single line of code was written, from one thirty-second division.

Microsoft’s numbers aren’t a lie, either. Slack cut CI type-checking from 7.5 minutes to 1.25 minutes, but there it’s a dedicated tsc step, where the compiler is 100% of the work. Same compiler, same engine gain, opposite outcome: the difference isn’t in the tool, it’s in the denominator.

Measuring costs five minutes

GitHub Actions already shows the duration of every step in the UI, for free. Open the last run and read it before forming any theory. Then divide the step’s time by the total: that is the maximum you can gain by attacking it. If it comes out to 2%, you just saved yourself a week.

If you need to dig deeper, the tools already exist and are criminally underused:

• jest --verbose gives you the duration per file. Three files usually account for half the time.
• jest --detectOpenHandles finds promises that never resolve and timers that never clear, which leave the runner sitting around waiting for the event loop to drain after the tests already finished. It has existed for years.

Knuth wrote this in 1974, on the same page of Structured Programming with go to Statements that gave us the “premature optimization is the root of all evil” everyone quotes at half length:

“It is often a mistake to make a priori judgments about what parts of a program are really critical, since the universal experience of programmers who have been using measurement tools has been that their intuitive guesses fail.”

The problem is the jump

None of this means “don’t upgrade.” Do upgrade - TypeScript 7 is an impressive piece of engineering and I plan to migrate everything I can. Those 6 seconds don’t move CI, but they do move the agent loop, which runs typecheck and lint dozens of times an hour. That is a real gain, just in a different number.

What’s wrong is something else: going from “it’s slow” straight to “swap it for something faster,” without the thirty-second division in between. Tool speed is not system speed.

And swapping tools is comfortable because it looks productive: there’s a PR, there’s a changelog, there’s a nice benchmark to show. Measuring first looks like bureaucracy.

But measuring is what separates engineering from cheering.

Thanks for reading!

• Announcing TypeScript 7.0 - the official announcement, with the benchmarks and the full list of breaking changes.
• TypeScript 7 Went Native: What Actually Changes and What Doesn’t - a good read on what the migration does not change.
• Systems Performance, by Brendan Gregg. If you only read one chapter, read chapter 2.

Integer i = 1, j = 1;
i == j        // true

Integer x = 1996, y = 1996;
x == y        // false

So you did what everyone does: you searched for it.

You found the answer fast, because the question has been on Stack Overflow since November 2009 with a couple hundred votes. The explanation is always the same: Integer x = 1996 doesn’t call new Integer(1996), it calls Integer.valueOf(1996). And valueOf keeps a cache of ready-made instances from -128 to 127. Inside that range you get the same object back every time. Outside it, a fresh object per call.

Fair enough. You filed away “compare wrappers with equals”, closed the tab, and moved on.

And it’s easy to dismiss this case. Boxing is your choice: use int, stop comparing wrappers with ==, and the problem evaporates. Interview trivia, lab curiosity.

But what happens when there’s no primitive to fall back on?

LocalDate d1 = LocalDate.of(1996, 1, 23);
LocalDate d2 = d1.plusYears(30);      // 2026-01-23
LocalDate d3 = d2.minusYears(30);     // 1996-01-23

d1.equals(d3)   // true
d1 == d3        // false

LocalDate has no literal syntax. It has no primitive version. If you need a date, you’re using an object.

And there’s no cache in this story to blame. d1 and d3 are the same date, same year, same month, same day. equals agrees. == doesn’t.

So “use equals” is the medicine, not the diagnosis. The question nobody answered is why == compares memory addresses in the first place.

It has an answer. On July 31st, 2026, a commit of 208 thousand lines landed in the OpenJDK and changed that answer for the first time since Java 1.0.

Identity is being able to tell two identical things apart

Before Java, think about two things in the physical world.

Nobody cares which twenty-dollar bill they got as change. If I swap yours for another one while you look away, nothing happened. Two twenties are interchangeable.

Now your car. You lend it to a neighbor and you want THAT car back, not an identical one.

The difference between those two cases is what we call identity: the ability to distinguish two things that hold exactly the same content.

And here’s the thing. Identity is only useful for things that change.

The car matters individually because it accumulates history: mileage, a dent, an empty tank. If two cars were frozen and never changed, it wouldn’t matter which one you got back. It becomes a twenty-dollar bill.

That gives you the one sentence everything else follows from: identity is a capability only mutable data uses.

You already know this by another name

If you do DDD, this distinction is nothing new. It’s Entity versus Value Object, an idea Ward Cunningham was already describing in 1994 and that Fowler catalogued before Evans put it at the center of modeling.

Entity is the car. Order #4712 is still the same order after changing status three times.

Value Object is the twenty-dollar bill. $50.00 is $50.00, and an EmailAddress is the string it carries.

Except in DDD this was always design discipline and nothing more. You wrote Money immutable, no setters, equals by hand, and documented that it was a value object. The team understood. The compiler didn’t. The JVM even less.

At runtime, your value object was an entity like every other one: its own address, a header, an identity, and == lying to you. You drew the distinction in the domain diagram and paid full price in memory.

That gap is what JEP 401 closes. For the first time value is a word the compiler reads, not a comment in the documentation.

Java assumed everything was a car

Here’s the decision Java made: every object has identity. No exceptions, since 1995.

It was a design choice, and it was reasonable at the time, because objects in Java were born mutable by default.

The problem is that it became a law of physics for the language. And laws of physics have consequences that fall out by gravity, whether you want them or not.

== compares addresses because the address is the identity. Two distinct objects have to live in distinct places, otherwise you can’t tell them apart.

Every object carries a header, and it exists because identity needs somewhere to live. That’s where lock state and the identity hash sit.

You don’t have to take my word for it. JOL, short for Java Object Layout, is an OpenJDK tool that reads the layout the JVM actually chose for an object, field by field, instead of estimating it. I ran it against a LocalDate on JDK 28:

Sixteen bytes of object to carry six bytes of date. The header alone is bigger than the data, and there are still two bytes of padding on top.

And that’s already the slim version. On JDK 28 HotSpot enables compact object headers by default, folding the class pointer into the mark word. Run with -XX:-UseCompactObjectHeaders and the same LocalDate goes back to 24 bytes. The JVM was already fighting this cost from another angle, and even after shrinking the header it remains the largest piece of the object.

Hold on to that figure, because the header comes back at the end of this post. The lock lives in it, and that’s why synchronized is about to stop working.

An array of objects is an array of pointers. If every element needs its own address, the array can’t hold the data. It holds the path to it.

Compare an int[5] with a LocalDate[5], which is the example the JEP itself uses:

The int array is a single block. The LocalDate one doesn’t hold dates, it holds pointers, and each object ended up wherever the allocator found room, carrying its own header along. Walking that array is a sequence of memory jumps with a cache miss on each one.

Measured with JOL: 32 bytes against 80. Two and a half times the memory to represent the same thing.

All of it to carry an int and two bytes of useful information per date.

The Integer cache is a workaround for not paying identity

Now the surprise from the beginning makes sense.

Identity costs. Every new is an allocation, a header, an address, and one more object for the GC to visit later. Since boxing int happens constantly, somebody decided it was worth reusing the most common instances instead of creating a new object every time.

Hence the cache from -128 to 127.

The catch is that reusing an instance means reusing an identity. And identity is observable through ==.

In other words: the Integer gotcha is the model showing through. An allocation optimization leaked into the semantics of the language, and it could only leak because == talks about identity instead of value.

You weren’t confused. The model was strange. I’ve written before about why Java feels hard (in Portuguese), and a good part of the answer is exactly this: the language asks you to understand old decisions nobody tells you about.

But immutable data never needed it

LocalDate is immutable. Integer is immutable. So are Optional, Duration, BigDecimal, and the Money class you wrote last week.

None of them is a car. They’re all twenty-dollar bills.

You’ve never, in any code you’ve written, needed to know which instance of January 23rd, 1996 you were holding. The JVM had no way to know that, so it charged identity to everyone, at full price, as a guarantee.

That’s what JEP 401 fixes. It gives you a way out.

JEP 401 lets you opt out of identity

Commit cc278dbb implements two JEPs at once, both as preview in JDK 28: JEP 401 (Value Objects) and JEP 539 (Strict Field Initialization). That’s 208,011 lines added, 13,161 removed, 300 files, 64 co-authors and 14 reviewers. It’s the largest Project Valhalla delivery so far.

And the API for it’s one word:

jshell> value record Point(int x, int y) {}
|  created record Point

jshell> Point p = new Point(17, 3)
p ==> Point[x=17, y=3]

jshell> Objects.hasIdentity(p)
$3 ==> false

jshell> new Point(17, 3) == p
$4 ==> true

The value modifier gives you three things for free. Fields become final, the class becomes final, and == starts comparing field by field.

The terms look alike and mean different things:

• value class is what you declare, with the modifier
• value object is an instance of it, the object without identity
• Value Objects is the name of the feature, and Project Valhalla is the umbrella

The platform has already migrated 30 classes:

Which answers the surprise from the opening. Real output, run on a build with JEP 401 enabled:

Integer 1996 == 1996      : true
d1 == d3                  : true
Objects.hasIdentity(d1)   : false
Objects.hasIdentity("abcd"): true

Keep an eye on that last line. It comes back at the end.

Writing your own

If your data is already a record, it’s one word:

value record Point(int x, int y) {}

A record is transparent: its fields are exactly the constructor components. When you store state one way and expose it another, say money as a long of cents, you need a plain value class:

value class EURCurrency {
    private long cs;  // implicitly final
    public EURCurrency(long e, int c) { cs = e * 100 + c; }
    public long euros() { return cs / 100; }
    public int cents() { return (int) cs % 100; }
}

The value modifier closes doors, and the compiler is direct about which ones:

error: cannot assign a value to final variable x
error: cannot inherit from final V
error: The concrete class Base is not allowed to be a super class
       of the value class E either directly or indirectly

Fields become final, the class becomes final, and inheriting from a class with identity would mean inheriting the identity along with it. You still get hierarchy: you can implement interfaces, and you can extend an abstract value class, which is how Integer and BigInteger now coexist under Number.

The rule that catches most people is the constructor one. A value object has to be complete before anyone can observe it, so the entire body runs before super(), and this doesn’t exist there yet:

value class Name {
    String name;
    int length;
    private int strLength() { return name.length(); }

    Name(String n) {
        name = n;
        length = strLength();   // error: reference to strLength() may only
    }                           // appear after an explicit constructor invocation
}

The way out is making the method static, or calling super() by hand after setting every field. And with preview enabled this applies to all records, value or not, so a record that uses this in its canonical constructor stops compiling.

As for what to mark, the rule is short: immutable state, and you never need to distinguish two instances holding the same content. If you model with DDD, your value objects package is the first place to look. What stays out is anything mutable, anything used as a lock, and anything holding sensitive data, since == compares private fields.

Without identity, the JVM no longer needs to hand out addresses

Go back to the list of consequences above and invert each one.

If the object has no identity, it doesn’t need to be distinguishable. If it doesn’t need to be distinguishable, it doesn’t need its own address. Which buys the JVM two freedoms.

Flattening means putting the fields directly inside the array or the field that references the object:

No pointers, no headers, all contiguous, with the first bit saying whether the reference is null. The JEP says this array may end up with performance characteristics similar to an int[].

That’s what the JEP describes. I wanted to watch it happen.

I allocated a LocalDate[] of two million slots, all distinct dates, and measured the heap. Same program, same JDK, same machine, changing only --enable-preview:

Eight bytes per element, which is exactly the 64-bit word the JEP predicted.

And you don’t need to trust my heap measurement to accept it, because the arithmetic closes on its own: if each element were still a pointer to a separate object, the objects alone would take two million times 16 bytes, the minimum size of an object on the heap. That’s 32 MB. It doesn’t fit in 16.8.

The objects aren’t there. Only the values.

Now, a caveat worth more than the measurement: none of this is in the spec. JEP 401 doesn’t even have a Specification section, and it states outright that flattening and scalarization are “optimizations, not language features”, done at the discretion of the JVM. Guaranteeing memory layout is a declared non-goal.

What the JEP guarantees is semantics. A value object has no identity, == compares fields, synchronizing throws. That’s the contract.

The contiguous array is permission, not a promise. The JEP removes what was stopping the JVM from flattening, and each implementation decides whether it does. I showed you one that did.

Scalarization is the next step, inside the JIT. When the object sits in a local variable or a parameter, it gets decomposed into loose values. The compiled plusYears stops taking a pointer and starts taking (boolean isNull, int year, byte month, byte day), returning another tuple like it.

The object simply never exists in memory.

Escape analysis already did something similar for ordinary objects, but a single code path comparing identity makes the optimization evaporate. With a value class the guarantee is static, and it crosses method boundaries.

Less allocation is less GC

This is where it shows up in your Grafana.

Every object the JVM doesn’t allocate is an object the GC doesn’t have to mark, sweep or move. A LocalDate[] of a million slots stops being a million live objects on the heap and becomes a block of memory.

That loop you wrote without thinking, creating a LocalDate per iteration only to throw it away, stops generating garbage. It isn’t that the GC got faster: there’s nothing left to collect. And contiguous data is still data the CPU fetches with fewer cache misses, which usually matters more than the allocation time itself.

And since it isn’t a promise, there are ways for it not to happen. Three things get in the way in practice:

• A mutable field has a 64-bit ceiling, because reads and writes need to be atomic. A LocalDateTime doesn’t fit and goes back to being a pointer.
• Object kills flattening. Integer[] is flattenable, Object[] isn’t, and erased generics fall in the same bucket. It changes no semantics, only layout.
• Old code needs recompiling, because the JVM relies on a new class file attribute to learn in time that a class is a value class.

What you lose is exactly what depended on identity

And there’s a logic to the price: the same design decision charging you on the way out what it charged on the way in. Everything that breaks is something that needed to distinguish instances. I ran each one to get the real message:

synchronized via Object : java.lang.IdentityException:
                          Cannot synchronize on an instance of value class java.time.LocalDate
d.notify()              : java.lang.IllegalMonitorStateException: java.time.LocalDate
new WeakReference<>(d)  : java.lang.IdentityException:
                          java.time.LocalDate is not an identity class
weakHashMap.put(d, "x") : java.lang.IdentityException:
                          java.time.LocalDate is not an identity class

The lock lives in the header, in that mark word from earlier, so synchronized stops working. All of Java Concurrency in Practice assumes any object can serve as a lock, and now it can’t. wait and notify fall with it, since they depend on that same lock, and so do WeakHashMap and all of java.lang.ref, because a weak reference needs to point at one specific instance.

Beyond that, some things keep working but not the way you expect.

== now compares internal fields, so it can diverge from your equals, which might look at something else. It also became an operation with a cost, because the comparison is recursive and a deep tree of value objects can hit StackOverflowError. And since it reads private fields, it became an inference channel. The JEP says it plainly: value objects weren’t designed to protect sensitive data.

Even == on identity objects got marginally more expensive, because the if_acmpeq bytecode now needs an extra test to detect value objects. The identity path became a fast path, but it exists, and it’s charged to code that uses none of this.

The missing piece: JEP 539

There was still a hole. A value object promises its value never changes, but in Java a field can be read before it’s initialized, holding 0 or null.

The JEP’s example is a circular dependency:

class App {
    public static final long appID = Log.currentPID();
    public static void main() {
        IO.println("App[" + appID + "] has started");
        Log.log("Completed 'main'");
    }
}

class Log {
    private static final String prefix = "App[" + App.appID + "]: ";
    public static void log(String msg) { IO.println(prefix + msg); }
    public static long currentPID() { return ProcessHandle.current().pid(); }
}

Running it prints:

App[8145] has started
App[0]: Completed 'main'

Two reads of the same final field, two different values. Log gets initialized in the middle of initializing App, reads appID holding the default 0, and bakes that zero into prefix. And here’s the nasty part: if Log were initialized first, the bug would disappear. It’s the kind of bug that vanishes when you go looking for it.

A final field that yields two different values destroys the entire premise of a value object. That’s why JEP 539 introduces the ACC_STRICT_INIT flag: a field marked with it has no default value and must be written before any read. javac marks every field of a value class with it, which is why both JEPs landed in the same commit.

If you go run that example, don’t expect it to fix itself: I turned on --enable-preview and App[0] is still there. Imposing strict initialization on existing code is a declared non-goal of JEP 539, so only value class fields get the flag.

Trying it today

There’s a logistics trap here, and I only found it because I went and ran things.

The obvious path is grabbing the JDK 28 early access from jdk.java.net/28. It doesn’t work yet. Build 9 shipped on July 31st, 2026, the same day as the integration, and it was cut before that landed: value record is a syntax error, Objects.hasIdentity doesn’t exist, and Integer 1996 == 1996 is still false.

What runs today is Valhalla’s own early access, at jdk.java.net/valhalla. Build 27-jep401ea3+1-1 implements JEP 401, and it’s where I ran everything in this post that produces output.

javac --release 27 --enable-preview Demo.java
java --enable-preview Demo

Preview has to be on at both ends, and you can’t pick the identity version of LocalDate in that mode: it’s all or nothing.

What didn’t change

Two things got left behind, and both are funny.

The Integer cache still exists. The doc/value-class-preview.md that shipped with the commit says it was kept on purpose, for performance, and that it now has no semantic impact whatsoever. The workaround that created the gotcha keeps running under the hood. You just can’t observe it anymore.

And String didn’t migrate. The class has identity dependencies in its API and its implementation, so Objects.hasIdentity("abcd") still returns true. Java’s most famous gotcha, == on String, is still standing.

Past that, the rest is foundation. JEP 402 will improve primitive boxing on top of this, and JEP 218 will let generics specialize layout when parameterized with a value class, which is List<int> without boxing.

But the big change already happened, and it’s conceptual before it’s technical.

== stopped asking “do you two live at the same address?” and started asking “can you two be told apart?”. For a car, the answer is still the address. For a twenty-dollar bill, it’s now the value.

That surprise you had at the beginning never had “use equals” as its answer, let alone “use a primitive”. For LocalDate there was never a primitive to use.

The answer was that the date never needed identity, and you had been paying for it all along.

Thanks for reading!

References

• JEP 401: Value Objects and JEP 539: Strict Field Initialization, both Integrated in JDK 28
• Commit cc278dbb, July 31st, 2026, and the doc/value-class-preview.md that shipped with it
• Valhalla early access, build 27-jep401ea3+1-1, used for the measurements in this post
• The original Stack Overflow question, from 2009

EVAL PASSED - gate model "qwen2.5:3b": acc 95.1%, answer 95.7%

I stared at that line for a week feeling pretty good about myself. Then I changed a prompt, ran it again, got 94.6%, and asked the obvious question: is that worse than yesterday, or is that noise?

I scrolled up to check. My terminal had eaten yesterday.

That’s it. That’s the entire reason I put Langfuse in front of a Go agent. Not because I wanted a dashboard, but because a number you can’t compare to last week is not a measurement, it’s a vibe.

The agent is Ava, a research PoC in Go: a voice agent that runs opinion polls out loud and knows when to hang up. She exists because a team here had a voice agent that didn’t know when a conversation was over, so I built the dumbest PoC that worked instead of writing a spec. One afternoon, no document.

This post is what happened after that afternoon, once “it works” stopped being enough and I wanted to know whether it kept working.

And Go is where the first small wall shows up, because Langfuse ships SDKs for Python and JS and nothing for us. Turns out that’s fine. What follows is the whole path: what an eval even is, the crude one I wrote first, and how it got to Langfuse with nothing but the standard OpenTelemetry SDK and net/http.

First, how do you know an LLM app is any good?

If you write a function that adds two numbers, the test is obvious:

if Add(2, 2) != 4 {
    t.Fatal("math is broken")
}

Now write that test for a model. Same input, run it twice, get two different sentences. Both correct. != is useless to you.

This is the part that trips people up coming from normal software: the model is not the unit under test, the behavior is. You are not asserting that the output is a specific string. You are asserting that the output has a property you care about. So the whole job becomes: pick the property, and find a way to score it.

Chip Huyen catalogs the ways to do that in AI Engineering. I ended up using three of them, and I’d learn those three before touching any platform.

1. Functional correctness

Did the system do the thing? Not “does the text look nice” - did it work.

This is the strongest kind of eval and always the one to reach for first, because there’s no interpretation involved. If you ask a model to write gcd(a, b), you don’t grade the code, you run it and check that gcd(15, 20) returns 5. It’s how LeetCode grades you and how HumanEval grades models.

Ava’s version: on every reply, a classifier decides what the conversation does next - advance, re-read the question, ask for clarification, or hang up. That decision has a right answer, so the eval is a comparison:

turn, err := classifier.ClassifyTurn(ctx, c.q, c.reply)
if turn.Intent == c.want {
    report.correct++
}

Boring. == on a label. That is a feature. The classifier’s output is constrained enough to be checked exactly, and everything downstream of that decision is a state machine I can test like any other Go code. When you can push a fuzzy thing into a small set of labels, do it - you get your == back.

2. Similarity against reference data

Some things have no single right answer. Ava’s sign-off is a personalized callback to what the respondent said. There are a thousand good ones.

Here you compare the output against reference data: a labeled corpus of (input, expected) pairs, where expected is a reference answer rather than a label. And “compare” splits in two.

Lexical similarity works on the words themselves. Overlap, edit distance, BLEU, ROUGE. Cheap, deterministic, no model in the loop, and it has no idea that “pricey” and “expensive” mean the same thing.

Semantic similarity works on meaning. You embed both texts into vectors and measure the angle between them, so paraphrase scores high. Costs an embedding call per comparison, and returns a float you now have to pick a threshold for.

In Go both are the same shape of function:

// Lexical: words in common, no model involved.
func WordOverlap(got, reference string) float64

// Semantic: embed both, measure the angle between the vectors.
func CosineSimilarity(ctx context.Context, e Embedder, got, reference string) (float64, error)

Then you score a case with whichever one fits, against a threshold you own:

score := WordOverlap(got, c.reference)
if score < 0.6 {
    report.miss(c, score)
}

I went lexical, and not because I benchmarked anything. I only needed one property: did the agent talk about things the respondent actually said? That question is about which words showed up, so meaning was never the axis.

A dozen lines of strings.Fields and a map[string]bool answered it. UnsupportedWords(line, answers) gives me the words in the sign-off that appear in neither the respondent’s answers nor the agent’s own register. A cheap groundedness check: it doesn’t ask whether the sentence is right, only whether anything in it was invented.

It caught a real one. I added a nice example closing line to the prompt, and the 3B model copied it verbatim into all 13 cases, cheerfully telling a respondent who had only said “Vanilla.” all about lavender. Every exact-match score still read 100%, because none of them were checking whether the words were earned.

There’s a tax, and it’s specifically the lexical tax: a legitimate paraphrase (“scent” for “smell”) lands in that list as a violation. Semantic scoring would forgive it, at the price of an embedding call and a threshold I’d have to defend. Either way I report the number and never gate on it. It tells me where to go look and nothing more.

3. AI as a judge

And then there’s the stuff no string comparison will ever reach. Ava says a short acknowledgment before the next question so she doesn’t sound like a form. Is “Lavender, nice one” a good ack? You know instantly. Your code has no idea.

So you ask a model, constrain it to {"good": bool, "reason": string}, and parse the JSON. In Go that’s an API call and a json.Unmarshal, and there’s nothing clever about it.

Two rules I’d hand anyone doing this. Pin one judge model for every model you evaluate, or your scores stop being comparable to each other. And never let the judge fail your build - it’s a paid, non-deterministic dependency, and a judge outage should not turn CI red.

Cost and reach go up as you move down that list. Trust goes the other way. So gate on the first kind, and merely watch the other two.

The rustic eval: a slice, a loop, and an exit code

Here’s the part I wish someone had said to me earlier: an eval is not a platform. It’s three things.

A dataset:

type evalCase struct {
    q       string
    reply   string
    want    llm.Intent
    clarity llm.Clarity
}

var dataset = []evalCase{
    {"What's your favorite scent?", "Vanilla, definitely.", llm.IntentAnswer, clear},
    {"What could we do better?", "Nothing that comes to my mind actually.", llm.IntentAnswer, clear},
    {"How do you like it?", "(coughing)", llm.IntentUnintellig, na},
    // ~80 of these, hand-labeled
}

A scorer - the loop above, in a worker pool. And a threshold:

minAcc := flag.Float64("min-acc", 0.90, "minimum overall intent accuracy to pass")
minAns := flag.Float64("min-answer", 0.95, "minimum valid-answer acceptance to pass")

// ...

if gate.acc() >= *minAcc && gate.ansRate() >= *minAns {
    fmt.Printf("\nEVAL PASSED - gate model %q: acc %.1f%%, answer %.1f%%\n", ...)
    return
}
os.Exit(1)

go run ./cmd/eval, and a non-zero exit code when the agent’s behavior regresses. That’s a real eval. Zero dependencies, and it caught actual bugs.

NOTA: the two thresholds are deliberately different. Misreading an answer as something else loses a real answer, so it’s gated hard. A flat acknowledgment is cosmetic, so it’s reported and never blocks. Gate on what loses data.

I’m not showing you the crude version because it’s charming. I’m showing it because everything Langfuse gave me afterwards is a view over exactly these three pieces. The scorer is disposable. The dataset is the asset.

Once you see that, the platform reads as storage rather than magic. Which is all I ever wanted from it, since my actual problem was that yesterday’s run no longer existed.

Two doors into Langfuse, and neither one is an SDK

Langfuse has no Go SDK. What it has is two HTTP surfaces, and between them they cover everything:

1. An OTLP endpoint at /api/public/otel/v1/traces. This is the officially supported path for any language without an SDK - their OpenTelemetry docs say it outright: “For other languages, use the native OpenTelemetry API for your language and export spans to Langfuse.” You point the standard OpenTelemetry Go SDK at it. This carries traces.
2. A REST API at /api/public/* for the things OpenTelemetry has no concept of: datasets, experiment runs, and scores. Plain net/http. The Public API docs cover auth and conventions; the full API reference is the page you’ll actually keep open.

That’s the whole architecture. internal/obs in my project is two files, one per door.

Door 1: traces over OTLP

The entire configuration is one exporter:

func Init(ctx context.Context) (shutdown func(context.Context) error, enabled bool, err error) {
    noop := func(context.Context) error { return nil }
    pk := strings.TrimSpace(os.Getenv("LANGFUSE_PUBLIC_KEY"))
    sk := strings.TrimSpace(os.Getenv("LANGFUSE_SECRET_KEY"))
    if pk == "" || sk == "" {
        return noop, false, nil // no creds: tracing off, everything is a no-op
    }

    auth := base64.StdEncoding.EncodeToString([]byte(pk + ":" + sk))
    exp, err := otlptracehttp.New(ctx,
        otlptracehttp.WithEndpointURL(Host()+"/api/public/otel/v1/traces"),
        otlptracehttp.WithHeaders(map[string]string{
            "Authorization":                "Basic " + auth,
            "x-langfuse-ingestion-version": "4",
        }),
    )
    if err != nil {
        return noop, false, err
    }

    res, err := sdkresource.Merge(sdkresource.Default(),
        sdkresource.NewSchemaless(semconv.ServiceName("voicesurvey")))
    if err != nil {
        return noop, false, err
    }
    tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp), sdktrace.WithResource(res))
    otel.SetTracerProvider(tp)
    return tp.Shutdown, true, nil
}

Three things in there are worth more than the rest.

Auth is Basic auth over your key pair. Public key as user, secret key as password, base64, done. No token dance.

x-langfuse-ingestion-version: 4 is the current ingestion contract, and their docs are blunt about it: include it “so that new data appears in real time.” Send it. Without it the endpoint falls back to older mapping behavior and your spans land looking subtly wrong.

NewSchemaless, not NewWithAttributes. This one cost me an afternoon. If you pin your own semconv schema URL on the resource, it conflicts with the one the SDK’s default resource already carries, and Merge fails your whole init on a schema mismatch. Schemaless attributes merge cleanly and survive SDK upgrades.

And notice what happens with no credentials: the global tracer stays a noop, so every instrumented call in the codebase costs nothing. The PoC still runs fully offline, which is not a small thing when your gate model is local.

Instrumenting: wrap, don’t edit

Now, what gets traced? I did not sprinkle spans through the agent. Every LLM caller in the project is an interface, so tracing is a decorator:

func TraceClassifier(inner llm.Classifier, model string) llm.Classifier {
    return &tracedClassifier{inner: inner, model: model}
}

func (t *tracedClassifier) ClassifyTurn(ctx context.Context, question, reply string) (llm.Turn, error) {
    ctx, span := otel.Tracer("voicesurvey").Start(ctx, "classify_turn")
    defer span.End()

    input, _ := json.Marshal(map[string]string{"question": question, "reply": reply})
    span.SetAttributes(
        attribute.String("gen_ai.request.model", t.model),
        attribute.String("langfuse.observation.type", "generation"),
        attribute.String("langfuse.observation.input", string(input)),
        attribute.String("langfuse.trace.name", "classify_turn"),
        attribute.String("langfuse.trace.metadata.prompt_version", llm.ClassifyPromptVersion()),
    )

    turn, err := t.inner.ClassifyTurn(ctx, question, reply)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return turn, err
    }
    output, _ := json.Marshal(turn)
    span.SetAttributes(attribute.String("langfuse.observation.output", string(output)))
    return turn, nil
}

One line at the call site turns it on: cl = obs.TraceClassifier(cl, name). The wrapper is a pure pass-through, and it never touches the Turn or the error. A wrapper that can alter a result is a bug you’ll eventually blame on the model.

The langfuse.* attribute prefix is the part you can’t guess from the OTel docs. Those are Langfuse’s own conventions: observation.type makes the span render as a generation with input/output panels instead of a bare timing bar, and any langfuse.trace.metadata.* key becomes a filterable field in the UI.

I stamp prompt_version on every single call. Without it a prompt edit is invisible, because old and new output land in one pile with no axis to split them.

This is what those attributes look like once they land. One classify_turn trace, 0.31s, from a Go program with no SDK:

Every field in that screenshot came from a SetAttributes call. Nothing was configured in the UI. And notice the left sidebar while you’re there, because it settles the misconception I’ll get to below: Evaluation and Prompt Management are two separate sections. Neither depends on the other.

Why traces, and not just numbers

Here’s what I underestimated. I went to Langfuse for run history and got something I hadn’t asked for.

A voice conversation isn’t one model call. It’s transcribe, classify, write the acknowledgment, synthesize, and around again, twenty turns deep. When a conversation feels wrong, the aggregate accuracy number is useless. It tells me how often something broke and never what happened.

A trace is the whole turn as one nested object: inputs, outputs, latency per step, in order, tagged with the session id. Scoring a conversation turns out to be much easier than scoring a sentence, because the sentence finally has context around it. The respondent’s previous answer sits right there, above the decision that misread it.

And latency stops hiding. In a voice call, TTS and transcription time is dead air, and it never shows up in the LLM spans. So those get traced too, with a tiny helper for steps that aren’t LLM calls:

op := obs.StartOp(ctx, "tts")
op.In(text).Bytes("audio.bytes", len(pcm))
defer op.End()

Those spans carry the session id, which is what turns twenty separate traces back into one readable conversation:

24 traces for one poll. That’s the number that convinced me a dashboard of averages was never going to be enough.

The thing I got wrong: evals live on top of traces

Now the misconception that kept me from starting, which is the real reason I wanted to write this post.

I walked into this assuming that to run evals in Langfuse I’d have to move my prompts into Langfuse first. Adopt their prompt management, version prompts in their UI, let their runner execute them. That’s a real migration, and it’s the kind of price that makes you close the tab and decide the terminal is fine actually.

It’s not true. Evaluation in Langfuse hangs off traces. The unit is a Score, and the score data model is the page that settled it for me: a name, a value, a data type, and exactly one subject - a trace, an observation, a session, or a dataset run. Trace is listed as the common case. Nothing about a score cares where your prompt lives. You emit the trace from Go, you attach scores to it, and you’re doing evals.

You don’t hand a restaurant critic your recipe. They eat the dish.

Prompt management only becomes a prerequisite for one specific thing: asking Langfuse to execute a prompt itself against a dataset. I never wanted that. My prompt lives in Go, next to the code that depends on it, and it stays there.

Door 2: scores over REST

A score is a small POST. The whole client is net/http:

// Score is one judgment. Exactly ONE subject must be set.
type Score struct {
    Name         string
    Value        float64
    DataType     string // NUMERIC or BOOLEAN
    Comment      string
    TraceID      string
    DatasetRunID string
    ID           string // supply it and the write is idempotent
}

Attaching a per-case verdict to the trace that produced it:

c.Score(ctx, obs.Score{
    ID:       obs.StableID(runName, itemID(cr.c), "intent_correct"),
    Name:     "intent_correct",
    Value:    correct,
    DataType: "BOOLEAN",
    TraceID:  cr.traceID,
    Comment:  fmt.Sprintf("want %s, got %s", cr.c.want, cr.got.Intent),
})

That single call is what turned my scrolling terminal into something usable. In the UI I filter to intent_correct = 0 and read only the misses, each one clickable straight through to the full conversation that produced it.

To capture the trace id, the tracing wrapper drops it into a ref carried on the context:

var ref obs.TraceRef
ctx = obs.WithTraceRef(ctx, &ref)
turn, err := cl.ClassifyTurn(ctx, c.q, c.reply)
outcomes[i] = caseResult{c: c, got: turn, err: err, traceID: ref.ID()}

And the offline side - my hand-written dataset - maps onto Langfuse’s dataset experiments the same way. The corpus becomes a dataset, each model’s pass becomes a run, each case becomes a run item that links the dataset item to the trace it produced. Aggregate metrics attach to the run instead of a trace:

c.Score(ctx, obs.Score{
    ID:           obs.StableID(runName, "intent_accuracy"),
    Name:         "intent_accuracy",
    Value:        r.acc(),
    DataType:     "NUMERIC",
    DatasetRunID: runID,
})

Which makes the point concrete: even an offline eval run goes through tracing, because a run item requires a trace id. There’s no path into Langfuse that skips the traces.

And this is the payoff for the whole exercise, the thing my terminal could never give me:

That’s the sign-off eval from earlier, the one UnsupportedWords belongs to. Four runs of the same 13 cases, each row tagged with the prompt hash that produced it, lined up so a change is something I read instead of something I remember. The baseline row and the row after the fix are both still there weeks later, which is the entire thing my terminal failed to do.

And that’s what finally answers the question I opened with, which storage alone never could. Is 94.6% worse than yesterday, or is it noise? is really two questions, and the one that matters is whether the prompt moved.

Two rows carrying the same fingerprint and different scores: that’s noise, and the honest response is to widen the dataset or stop reading that decimal place. Two rows with different fingerprints and different scores: that’s your change, and now you can argue about it.

The history is what let me ask the question next week. The fingerprint is what made it answerable at all.

Four gotchas that will bite you

Flush before you link. The OTLP batch exporter is asynchronous, and the run-item endpoint rejects a trace it hasn’t ingested yet. So force a flush first, and treat a 404 as retryable:

if err := obs.Flush(ctx); err != nil {
    return fmt.Errorf("flush traces: %w", err)
}

Retry the 404, and only the 404. A 404 there means nothing was created, so retrying is safe. That endpoint is otherwise not idempotent - the server mints the id - so no other status is ever worth retrying.

Learn which endpoints upsert. Datasets upsert by name, dataset items upsert by id, scores upsert by supplied id. So I derive ids from a content hash and re-push the whole corpus on every run without creating a single duplicate:

func itemID(c evalCase) string {
    return obs.StableID("turn-classifier", c.q, c.reply)
}

Edit dataset.go and only the cases that actually changed change upstream. Run items are the exception, so they’re posted exactly once and never blind-retried.

You now own prompt versioning. This is the bill for keeping prompts in Go, and it arrives quietly. Langfuse’s prompt management would have versioned them for me. My prompts are Go string constants, so nothing versions them but me, and a score I can’t attribute to a prompt is back to being a vibe.

My first instinct was a constant to bump by hand. Terrible idea: I would edit the prompt, forget the bump, and quietly attribute new output to the old version. Worse than no version at all, because it looks trustworthy.

So the version is derived from the prompt instead:

// ClassifyPromptVersion is a short, stable fingerprint of the classifier's
// instructions: the system prompt plus the few-shot anchors. Content-addressed,
// so it cannot drift out of sync the way a hand-bumped number would.
func ClassifyPromptVersion() string {
    _, shots := classifyPrompt("", "")
    h := sha256.New()
    h.Write([]byte(classifySystem))
    for _, m := range shots {
        h.Write([]byte(m.Role))
        h.Write([]byte(m.Content))
    }
    return hex.EncodeToString(h.Sum(nil))[:12]
}

Twelve hex characters, and that’s the 79bec7b42725 in the trace screenshot above. Change a single word of the prompt and the fingerprint changes with it, whether I remembered to think about it or not.

Two details worth stealing. Hash the few-shot examples too, not just the system prompt, because an edited example changes behavior every bit as much as an edited instruction. And the test asserts only that the value is stable across calls and 12 characters long, never what the value is - a test pinning the hash would fail on every legitimate prompt edit, which is the fastest way to teach yourself to ignore a red test.

The trade is real and I’d make it again: an opaque 79bec7b42725 is less readable than v3, and it cannot lie to me.

What I’d tell a friend starting this in Go

The absence of an SDK looked like a blocker and was a two-file inconvenience. internal/obs: one file for OTLP traces, one for the REST client. That’s the whole integration.

So, in order:

1. Write the dataset first. Twenty labeled cases in a []struct{} beats any tool you haven’t picked yet. The cases are the asset; everything else is downstream of them.
2. Score exactly, where you can. Push fuzzy outputs into small label sets and get your == back. Reach for similarity and judges only for what genuinely resists that, and don’t gate on them.
3. Point the standard OTel SDK at /api/public/otel/v1/traces. Basic auth, x-langfuse-ingestion-version: 4, NewSchemaless. You are done in 30 lines.
4. Instrument by wrapping interfaces, never by editing call sites. And keep the wrapper a pure pass-through.
5. Your prompts can stay in Go. Scores hang on traces. Fingerprint the prompt yourself and stamp it on every span, and you keep both the prompt and its history where the code is.

Ava’s numbers still print in the terminal, exactly like before. The difference is that they no longer vanish when I close it.

Thanks for reading!

Book recommendation: AI Engineering, by Chip Huyen - the evaluation chapters are the clearest treatment of this I’ve found, and the taxonomy in the first half of this post comes from there.

For a single prompt, this step is instant. You don’t even notice it happened.

But you almost never tokenize just one thing. Think about RAG: you take your company’s documents and tokenize every one of them before indexing. Or a fine-tune: you tokenize the entire dataset before training. Or an eval running over millions of examples. That invisible step turns into machine-hours.

And nobody times it. We accept slow tokenization as if it were a law of physics - “that’s the price of working with LLMs at scale”. It isn’t. When I finally measured it, there was a 200x sitting there, waiting for someone to look.

That 200x lives in a specific place: pipelines that tokenize in bulk and write the ids to disk. If your case is a user’s prompt in production, the math changes. And between those two ends sits TTFT, the time to first token, where tokenization shows up again. We’ll get to all of them.

First: how much of the time is tokenization?

Before selling a solution, it’s worth measuring the problem. Optimizing something that carries no weight is wasted time - we’ll come back to that at the end.

I set up the simplest possible scenario: 281,664 real text documents, 863 MB from the fineweb dataset. Read from disk, run through the Qwen3-8B tokenizer, write out the token ids. That’s it - no model running anywhere in the middle.

nanoGPT’s prepare.py does exactly this: it tokenizes the corpus once and writes the ids to a .bin file that training reads later. Tokenizing again every epoch would be waste.

Then I timed each step. The result:

Tokenization was 99.4% of the time. It wasn’t a bottleneck among others. It was the whole pipeline. Reading and writing nearly a gigabyte each was noise next to it.

That changes the game. When one step eats 99% of the time, optimizing it stops being polish and becomes priority number one.

SO: killing tokenization speeds up the entire pipeline

Enter Gigatoken, a tokenizer Marcel Roed wrote in Rust. Its promise is aggressive: up to ~1000x faster than HuggingFace’s tokenizers.

Too round a number, so I got suspicious. I ran my own benchmark - same corpus, same tokenizer, on my machine (a MacBook M4 Max). Tokenization only:

About 200x on my machine. Not the 1000x from their README, but that 1000x came from a 144-core server - the more cores, the wider the gap opens. On my laptop, 200x is enough to hurt.

And the whole pipeline, I/O and all? It dropped from 147 seconds to under 2. 88x faster end to end. Two and a half minutes of waiting became less than two seconds, on the same corpus, with the same result.

NOTE: the most important number in those tables isn’t the time. It’s that both produced exactly the same 190,429,497 tokens. Byte for byte. One of them is 200x faster doing identical work. That’s not a shortcut, that’s engineering.

This is not premature optimization

An objection popped into my head before it could pop into yours: isn’t swapping tokenizers to shave off seconds the very picture of premature optimization?

That’s what I used to think, and it’s why I had never timed this step. HuggingFace’s tokenizers helps you not look, too: it’s written in Rust, it’s multithreaded, it’s maintained by serious people. Any reasonable person glances at it and says “this is optimized”. And there were still 200x on the table.

Premature optimization would have been tuning the I/O that took 0.8 seconds. Attacking the step that eats 99% of the time is the obvious move, and it only looks bold because nobody had checked the clock before.

What does exist, and nobody has named it, is the inverse problem: late optimization - accepting a bottleneck as untouchable without ever having timed it. “Tokenization is slow” was exactly that. A wall everybody saw, until someone saw a door.

“But didn’t you pick the slowest tokenizer on purpose?”

Fair question. If I had tested only Qwen, you’d have every right to be suspicious. So I tested more.

With GPT-2, the classic: HuggingFace took 129 seconds, Gigatoken took 0.22. The 193,502,159 tokens, once again, identical. Switching tokenizers within HuggingFace doesn’t close the gap - it widens it.

But the test that really matters is against tiktoken, from OpenAI. It’s the tokenizer with a reputation for speed, the one running behind GPT-4. And it is fast: it did the same corpus in 12 seconds, about 10x faster than HuggingFace.

Gigatoken did it in 0.22 seconds. 57x faster than the fast tokenizer, with token-for-token identical output.

And here’s the interesting part: HuggingFace being slow doesn’t mean “tokenization is slow”. tiktoken does the same tokenization 10x faster just by swapping one internal mechanism. The bottleneck has a specific name, and it isn’t BPE.

How? SIMD in the part nobody was looking at

The bottleneck’s name is pretokenization.

Before the actual tokenization algorithm (BPE), there’s a mundane step: splitting the text into pre-chunks. Almost everybody does it with a regular expression. And a regex scanning gigabytes of text is slow - it crawls along more or less character by character, hunting for the pattern.

What Marcel did was replace the regex with an implementation that does the same thing with SIMD. And this is where the magic falls apart.

Imagine you need to find every comma in a thousand-page text. You can read it word by word, looking. Or you can open the text into 64 parallel columns and ask all at once: “is there a comma in any of these 64 characters right now?”. Same result. Very different time.

That’s SIMD. Single Instruction, Multiple Data. One instruction, many pieces of data. The processor grabs 16, 32, 64 bytes and runs the same operation on all of them in a single cycle.

NOTE: if you want to pull on this thread, SIMD is an entire axis of concurrency most of us ignore - it isn’t threads or processes, it’s the processor’s own ALU working in batches. Paul Butcher dedicates a chapter to it in Seven Concurrency Models in Seven Weeks (data parallelism), and Mitchell Hashimoto has a great essay arguing that every developer should know SIMD. But for what matters here, one thing is enough: this is the technique that turns 146 seconds into 0.7.

The honest part: 200x on one step is not 200x on your life

Now the warning that separates this post from a sales brochure.

I said tokenization was 99% of the time. Notice: 99% of the time of THAT test. That holds for any pipeline where no model runs alongside: you read text, tokenize, write the ids. That’s all I measured, and there tokenization dominates on its own.

The moment a model enters the same pipeline, the math changes. In a RAG you tokenize and then push each chunk through an embedding model - that forward pass is real work, and tokenization’s slice shrinks.

And if you’re tokenizing a user’s prompt before calling a model in production, the picture flips completely. During generation itself, tokenization is a crumb next to the forward pass - a 200x there changes almost nothing, because it was never the problem.

The narrow exception is the time to first token, TTFT. Gigatoken’s own author showed, in the Hacker News discussion, that on smaller models you can shave 5-10% off TTFT. It’s real - but it’s a different universe from the 88x in preprocessing. The size of the prize depends on where tokenization sits in your equation. That’s Amdahl’s Law.

The total gain from optimizing one part is capped by how much that part weighed in the whole. Optimizing 99% of the time transforms everything. Optimizing 1% changes nothing, no matter how spectacular the 200x looks.

I learned that the hard way. This year I entered the rinha de backend (a Brazilian backend performance contest) and, for the first time in my life, wrote a SIMD kernel by hand - the application did a lot of repetitive math, the perfect candidate. The math got much faster. And the final gain was disappointing.

Because the math was never my bottleneck. The time lived in everything running around it. I had brilliantly accelerated a piece that carried no weight. The same law that makes Gigatoken look like magic tripped me up in the rinha - just in the opposite direction.

What remains

The real work isn’t “install Gigatoken”. It’s measuring where your time lives before you start optimizing.

If you do data preprocessing for LLMs and have never timed tokenization, time it. If your path looks like the test’s - read text, tokenize, write the ids - it may be eating almost everything. And if it is, there’s a 200x sitting there:

pip install gigatoken

Its API mimics the tokenizers you already use, so the swap itself is one line. How much of that gain shows up on your end is another conversation, and it depends on how much of your time was in tokenization. Only the stopwatch can answer that.

But if tokenization is a crumb in your pipeline, save Gigatoken for the right day and go hunt the real bottleneck. Because in the end it’s always the same story: time lives somewhere specific, and it’s almost never where we think it does.

Thanks for reading!

The product is neat. Someone opens a link, an AI asks a sequence of questions by voice, and at the end it all turns into structured data. A conversational survey. The kind of thing you send to a client and they answer by talking, no form involved.

The missing piece was the ending. The agent asked the questions, got the answers, and just stayed there. Floating. A human feels when it’s time to wrap up a chat. The machine doesn’t.

Lately I’ve been taking this kind of problem and building a PoC to leave as an example - a running case my colleagues can open, understand, and adapt to their own context. That’s what I did here.

I could have done what’s fashionable now: sit down and write a giant document before touching any code. A spec. Requirements, state diagram, the fifteen academic ways to detect end of conversation, acceptance criteria for each one. Spec-driven development, which is being sold as the grown-up way to work with AI.

I didn’t. And that’s what this post is about: why the simplest possible thing solved it, and why the big document would have slowed me down.

The only document worth writing came before everything - and it wasn’t a spec

The first thing I did was ask for research. One prompt, written in a hurry, typos and all:

claude, my teammates are working on a project that is an AI agent, a voice AI agent. The agent goes through a sequence of questions and has some interactions in the middle. The agent, however, cannot understand when to finish a conversation by itself. Websearch algorithms, models and all techniques that can be used for it. Fan out agents, look into HuggingFace, GitHub, Kaggle, wherever you find useful to look into.

Look at what this document is. It doesn’t describe what I was going to build. It maps what the world already knows about the problem: how LiveKit, Pipecat, and Vapi end a call, which endpointing models exist, what others have already tried and where they broke.

That’s the distinction almost nobody draws properly.

Research reduces an unknown you cannot guess from your chair. It’s external knowledge, something that already exists outside your head, waiting for you to go get it.

A spec tries to guess an unknown that only the running system can answer. It’s an internal guess dressed up as certainty.

One you collect. The other you invent. And guess which of the two everyone spends their first day producing.

I saved the result:

save all this research and reference to a file, in a new folder.

And only after having the map in hand did I ask for the prototype:

plan a PoC on this. A voice agent that runs on browser, with a human voice. It should go through a list of AI-crafted questions. The backend team is in Go, so it’s preferable to keep it that way.

The first real line of code was born 37 minutes after the first prompt. No PRD. No alignment meeting about the schema. Research, a half-page plan, and hands on the keyboard.

That prompt was not a Google search

When I ask for research like that, Claude Code doesn’t open Google and paste the first result. That prompt of mine up there became a /deep-research, a command that ships with the tool. I described the problem in sloppy English; Claude picked the tool and ran it.

Under the hood, /deep-research is a workflow: a script that orchestrates several subagents, each on a slice of the task, storing results outside the main context window so it doesn’t clog up. It runs five phases - the logic any good human researcher would follow, except in parallel and without laziness:

1. Scope - takes my question and breaks it into about 5 different angles. One goes after the state of the art, another the academic papers, another the skeptical/contrarian view, another the practical implementation. Angles that don’t overlap.
2. Search - fires 5 search agents in parallel, one per angle. Five simultaneous searches, not one.
3. Fetch - merges everything, removes duplicate URLs, and actually fetches the ~15 best sources. Not the SERP snippet: the whole page. From each one, it extracts falsifiable claims, with the direct quote backing each one.
4. Verify - and here’s the trick. Every claim goes through 3 skeptical reviewers, each with an explicit order: try to refute this. If 2 out of 3 knock it down, the claim dies. Marketing, cherry-picked benchmarks, forums, old papers from a fast-moving field: all of it gets filtered before it reaches me.
5. Synthesize - merges the semantic duplicates, ranks by confidence, and spits out a report with cited sources.

Now compare that with what you do on Google.

You type a query, get ten blue links, open eight tabs, read half of three, forget two, and end up saving a bookmark you’ll never revisit. One query. One angle. Zero verification. You are the loop - the tired one, the one who skips the seventh tab because enough is enough.

In Claude Code the loop is the machine, which doesn’t get tired at the seventh tab or the fiftieth. And there’s a difference deeper than speed: on Google you search for pages; in the workflow you search for verified claims. One hands you a pile of tabs. The other hands you a report where every sentence has already been beaten up by three skeptics.

The result of that afternoon is open in the project’s RESEARCH.md. It has 45 distinct sources - arXiv, GitHub, HuggingFace, LiveKit and Vapi docs - organized by theme, with the open source turn-taking models separated from the academic datasets separated from the vendor tools. That came out of one prompt.

Sit down and try to assemble that same bibliography by hand, on Google, in one afternoon. Good luck. You’ll get to the tenth tab and give up - and the 35 missing sources were exactly the ones that would have saved you from reinventing endpointing from scratch.

The real design was written by the bugs

Here’s the part no spec would have caught, because it’s impossible for it to catch.

The real product was born from me talking to the thing and complaining about what was bad. Prompt by prompt, each one fired right after I spoke into the microphone and heard the agent get it wrong:

I’m still speaking and she asks if I’m still there, but then she could get my answer…

VAD timeout too short. She would cut me off mid-sentence to ask if I was still there. That wasn’t in the spec because it couldn’t be. You only find that out with a microphone in your hand and half a sentence in your mouth.

she only says “that’s everything I wanted to ask” and it cuts. Why is this happening?

The goodbye cut off in the middle of its own goodbye.

it repeated an already-answered question.

She was re-asking something I had already answered, because a classifier mislabeled my answer.

And the most subjective one of all, the one that really ties the knot:

this intro is not like a human would talk.

How do you write the rule “the opening has to sound human” in a document? You don’t. That was the whole point of the product - fluid, not robotic - and that is unspecifiable in prose. You don’t feel a PRD. You feel a conversation.

Each of these defects was discovered floating in front of the prototype, not predicted in a document. And it couldn’t have gone any other way. A voice agent is made of what happens in the gap between speech and response, and that gap doesn’t fit in a bullet point.

The eval became the spec - only growing backwards

Every time I found a new behavior, the request was the same:

add this to the eval.

That’s how the “spec” was born. It didn’t predict the behaviors. It accumulated them as they showed up. The document grew backwards, from the reality that ran, not from my Friday-afternoon imagination.

Notice the inversion. In spec-driven, the document comes first and reality tries to catch up. In what I did, reality came first and the document ran after it to record what had already proven itself true.

The same went for writing things down:

keep it documented, every single step must be documented and revalidated on every single change.

Docs after validation, describing what a run had proven. Not before, describing what I hoped would work. The project’s CLAUDE.md only showed up at the very end, when there was something to describe.

What was left at the end

A PoC good enough to solve the team’s internal problem.

It runs in the browser. It generates the questions from a preset, the agent runs the survey with a human voice, reacts to what the person says, and - the thing that mattered from the start - knows when to end. Including when the person disappears halfway through and she has to detect the silence and finish on her own, with dignity.

No hundred-page document. A tight loop, repeated to exhaustion: build, test live, listen to what breaks, fix it, pin it in the eval. Again. And again.

It was fast not despite being simple. It was fast because it was simple.

Why simplicity was too effective

There’s a famous Wigner essay about the unreasonable effectiveness of mathematics in the natural sciences. The idea is that a simple tool sometimes explains far more than it had any right to explain. That was exactly the feeling here.

And the dumb trial-and-error loop that handled everything has a name: it’s just a chat with Claude Code open next to the browser. Short prompt, eyes on the prototype, short prompt again. No ceremony. A problem the big-document crowd would treat as a two-week project became an afternoon of conversation.

If you got this far looking for the secret trick, I owe you a disappointment: there isn’t one. No clever architecture, no magic prompt, no secret technique. I described the problem in sloppy English, let the tool do the research, built the dumbest thing that worked, and kept fixing whatever broke in front of me. I didn’t do anything fancy - and that’s the entire post. Simplicity wasn’t a detail of the path. It was the path.

And the reason is kind of obvious once you stop to look.

Spec-driven tries to turn the hardest unknowns - the ones that only exist in the behavior of the running system - into confident prose, before the system exists. It spends the most expensive bullet early, at the exact moment you know the least about the problem.

I only front-loaded the one unknown that could be front-loaded: the domain research, the knowledge that already existed out there. The rest of the unknowns - the design ones - I let resolve themselves, empirically, in the friction with the microphone.

It’s not that documents are useless. It’s that a document is good for recording what you discovered, and terrible for pretending you already know. Its time is after the first contact with reality, not before.

And yes, I’m aware this is a PoC, not a product running in production with an SLA and a pager going off at three in the morning. But the order doesn’t change with the size of the thing. Product, task, one-afternoon prototype: you start with research, understand what you want to do, and then you do it. That was literally the arc of this post. Spec-driven doesn’t invert that order - it just pushes the understanding part to before its time, while it’s still a guess.

The spec-driven crowd will spend day one writing an exquisite section about the barge-in timeout - which a real microphone corrects in ninety seconds.

You probably don’t need that. Research what can be researched, build the dumbest thing that works, and let reality write the rest of the spec for you. It writes better, and it never delays the delivery.

Thanks for reading!

The context: I built a World Cup betting pool for my team (a “bolão”, the classic Brazilian prediction pool), with everyone guessing scores from the terminal, over SSH. Then we put an AI in the game, just for fun. It placed bets, commented on the pool, teased whoever was doing badly. We named it BETanIA.

And the problem showed up in the commentary. Every time it opened its mouth, it rediscovered the same facts with the same astonishment. It teased the same player about the same bad guess three days in a row, as if it were news. It dragged a two-week-old result into the conversation with the excitement of someone who had just watched the ball hit the net.

The intuitive reaction - and maybe yours right now - is: give it more memory.

That’s not where I went. I had already worked on an AI assistant for veterinarians, and that’s where I first ran head-on into this business of managing agent context and memory. So it clicked fast: more memory would only push the problem down the road - what was missing was a smarter architecture for its memory.

Add to that some time poking around how Claude Code handles context in long sessions, and I already had a suspicion: the problem is almost never a lack of memory, it’s what you do with it.

BETanIA didn’t need more memory. It needed to forget properly.

The goldfish

There’s a cliché about goldfish having a three-second memory. It’s not true about the fish, but it’s a perfect description of an LLM call.

Each invocation of the model is a brand-new mind that has never seen your data. It wakes up, looks at whatever you put in the context, answers, and dies. The next call remembers nothing.

So a commentator that only receives the current standings does what a goldfish would do: discovers everything again, every time. That’s not a character. It’s a slot machine that pays out in repeated jokes.

This is one of the two pits of agent memory. And it’s the obvious one: forgetting what matters. Amnesia.

But there’s a second pit, on the opposite side, and that one is far less intuitive.

Funes, the man who couldn’t think

Borges has a short story called Funes the Memorious. A young man, Ireneo Funes, falls off a horse and, after the accident, can no longer forget anything.

Every leaf of every tree he has ever seen. The exact shape of every cloud. The exact wording of a page read once. Everything, permanently, in full detail.

It sounds like a superpower, but it’s a curse.

Borges’ point is that Funes cannot think. Thinking is generalizing - it’s seeing a dog in profile at 3:14 pm and the same dog head-on at 3:15 pm and calling both of them “dog”.

To think is to forget differences, to generalize, to abstract.

Funes can’t. To him those are two distinct, incomparable things, because they in fact are different. Drowning in the specific, he loses the ability to abstract.

Here’s the part that matters to anyone building agents: every long-running agent is on its way to becoming Funes.

BETanIA’s diary grew by one paragraph per match. A World Cup has more than a hundred games. Push all of it forward, forever, and the context stops being a notebook and becomes a haystack.

You don’t want either one. Not the goldfish that forgets what matters, not the Funes that remembers everything and locks up. Good design lives in between, and it has a name.

The right question

Every agent project eventually runs into the question “how much should my agent remember?”.

It’s the wrong question.

The right question is: what is safe to forget?

And here comes the thing I want you to take from this post. This is not an implementation detail you sort out later. Deciding what can be thrown away is the memory design. It’s context engineering in its purest form.

Best of all: you already use a tool that does this all the time, right in front of you, and you probably never noticed.

Claude Code forgets on purpose

Open a long session in Claude Code. Two hours, dozens of tool calls, files read, commands executed. At some point the context would fill up and the whole thing would break.

It doesn’t break. Because Claude Code forgets, in layers.

The cheapest layer it calls microcompact. That tool result from 90 minutes ago - the giant log it has already read and already analyzed - it no longer needs the raw text. So it swaps the content for a [Old tool result cleared to save context] and moves on. Cost: zero model calls.

When that’s not enough, it goes up a layer: it calls the model to summarize the entire conversation and replaces the history with the summary plus the most recent messages.

Notice what happens in both layers. It throws away the text and keeps the conclusion. It forgets the 250KB log, remembers that “the database connection pool was exhausted, I restarted the pods”. The information that matters survives; the raw material rots away.

This is not a hack to save tokens. It’s the only way an agent can last two hours without becoming Funes.

A notebook, not a tape

When I had to build BETanIA’s memory from scratch, I landed in the same place - only the hard way.

Think about how a real sports commentator prepares. He doesn’t rewatch the whole season before going on air. He shows up with a small notebook: one line per game, a few facts per player, who’s feuding with whom.

The skill is not remembering everything. It’s keeping notes good enough that the past stays available without being relived.

That’s what BETanIA got. A diary: when a match ends, one model call condenses everything - the score, everyone’s guess, the best live comments - into a single paragraph. The story of that match. The paragraph is kept; the rest can rot.

And the diary can be compacted. One call reads the whole diary and merges everything into a single narrative, weighted toward recency: the old rounds become “the early chaos when nobody could get Brazil right”, the latest matches keep their detail.

In the final database backup, the 104 matches of the World Cup live in exactly three diary entries: one merged narrative carrying the entire tournament up to the semifinals, plus the last two matches in full detail. A month of football, compressed, but never lost.

And “compressed” here is not a figure of speech. In that first entry, rounds that were once dramatic become a single clause. A real excerpt, verbatim from the database (it writes in English):

The tournament’s most democratic disasters are worth cataloguing: Jul 5 saw Norway eliminate Brazil 1-2 for a thirteen-pick, thirteen-zero collective wipeout; Jun 30 handed all fifteen players unified zeroes on Germany-Paraguay.

Each of those was an entire match, with a minute-by-minute score and the standings dancing at every goal. One sentence is what remains. That’s what compacting well looks like: the detail disappears, the shape stays.

A notebook, not a cassette tape of everything that ever happened. Funes keeps the tape. And Funes would never win the pool.

Not everything can be compressed

Now, the part that separates engineering from a quick hack. It’s tempting to think the rule is “compress everything you can”. It isn’t.

BETanIA has a second compaction, with the opposite contract - and the contrast is the real lesson.

Besides the diary, I kept notes with facts about the players. One of them is a Gentoo user who compiles his own kernel. That kind of thing. When those notes grow, they get compacted too - but the prompt literally says “PRESERVE every distinct fact”. Merge duplicates, group what’s similar, but nothing can be lost.

Why? Because those facts are non-reproducible truth.

If it forgets that someone compiles his own kernel, no amount of standings data will ever bring that back. A match summary, on the other hand, can be tolerated if it comes out sloppy - the core information is still there.

Same operation, two contracts. A summary can lose detail. A fact cannot.

And here the whole rule collapses into one sentence, which I left written in the project’s docs:

Non-reproducible truth gets persisted. Reconstructible presentation gets thrown away and rebuilt later.

The pool standings, for example, are never stored. They’re computed on the fly, from the guesses and the results. Storing them would create a second copy that can drift from the truth. So: throw it away, rebuild it when someone asks.

That criterion - store or rebuild - is what organized BETanIA’s entire memory into a shelf, each type with a rule for how long it lives:

In the end, only two shelves hold actual truth: the persistent one (the diary, the player facts) and the append-only log. The rest is presentation it rebuilds on its own - and the derived kind, like the standings, is never stored at all.

Deciding which of your memories is fact and which is presentation - that’s not implementation. That’s the design.

Forgetting properly takes work

I don’t want to sell you the idea that forgetting is just hitting delete. Forgetting well is dated and deliberate, and I learned both the hard way, with bugs in production.

First: timestamp everything. The first diary entries were text with no notion of time, so it treated a two-week-old result as if it happened today - the problem from the beginning of this post. The fix was stamping every entry with [Jun 22] and opening every prompt with a “today is…” line. The model needs an anchor to figure out what “old” means. Give it the anchor, explicitly.

Second: don’t reconstruct the past. If the memory boots up mid-tournament, it adopts the already-finished matches as “settled” without writing the story of any of them. Without that, the first boot fires off thirty fresh little stories about old matches - and now you’ve manufactured a Funes with vivid false memories on day one.

In other words: forgetting is not the absence of a memory system. Forgetting is the memory system. The hard part isn’t storing. It’s deciding what goes, when, and what must never go.

Funes never wins the pool

In the end, BETanIA won the World Cup pool. It finished first, 50 points ahead of the runner-up - which was me, the guy who built it.

Let me be honest about one thing: memory is not why it won. It won on the guesses - web search, no ego, no favorite team. A bot with no memory at all would have scored the same points.

Memory is why nobody minded losing to it. The difference between a tool that spits out results and a character that shares a story with the team.

And that continuity doesn’t come from a bigger context window. It comes from the opposite. Every technique in this post is the same move repeated: condense the truth at the moment it’s about to become unrecoverable, store it small, date it, and hand the model only what the current task needs.

Funes kept everything and couldn’t think. The goldfish kept nothing and couldn’t remember. Your agent needs to sit in between - and sitting in between is a design decision, not an accident.

If you’re building an agent right now, stop asking how much it should remember. Ask what it can forget. It’s the same question we ask about our own heads without noticing - and Borges had answered it decades before a single token existed.

All of BETanIA’s code is on GitHub, memory design doc included. If you enjoy looking at the guts of one of these things, that’s the place.

Thanks for reading!

The detail that explains the 6,755 commits: Jarred Sumner, Bun’s creator, didn’t write it by hand. He orchestrated around 50 Claude Code workflows running non-stop for 11 days.

And days before merging, he was still treating the whole thing as an experiment - “There’s a very high chance all this code gets thrown out completely.” It got merged anyway. What unlocked it was empirical: when 100% of the test suite passed on every platform, his opinion went from “worth trying” to “I’m merging this”.

In the post about the migration, Jarred presents a list of bugs that lived in the codebase - use-after-free, double-free, memory leak - all from the same class of problem, which he claims is easier to prevent in Rust.

Let’s go through these bugs and what they mean.

The class of problem

Bun is a JavaScript runtime. It manages two worlds of memory at the same time:

1. GC-managed memory - JS objects, strings, ArrayBuffers that JavaScriptCore’s garbage collector controls
2. Manually managed memory - pointers, buffers, C/C++/Zig handles that you allocate and free yourself

The problem is when the two worlds mix. JS code can come back in the middle of a native operation (reentrancy). Callbacks like valueOf() and toString() can run arbitrary code. Errors can exit through paths nobody tested. And then:

• You free memory that is still in use (use-after-free)
• You free the same memory twice (double-free)
• You forget to free (memory leak)

Let’s take each one.

Use-after-free

In garbage-collected languages, when you no longer need an object, it just goes away. The GC handles everything.

In C, there is no GC. You ask the system for memory with malloc() and give it back with free(). After free(), that address is no longer yours. But the pointer still points there.

// Compile: gcc -fsanitize=address -g uaf.c -o uaf && ./uaf
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

char* cache;   // a reference stashed away for later reuse

void processa(char* msg) {
    printf("Processing: %s\n", msg);
    cache = msg;    // stash the pointer
    free(msg);      // and give the memory back to the system
}

int main() {
    char* msg = malloc(100);
    strcpy(msg, "hello world");

    processa(msg);

    // later, another piece of code uses what's in the cache:
    printf("From cache: %s\n", cache);   // use-after-free! cache points to freed memory
    return 0;
}

It compiles without errors. It may even seem to work. But the behavior is undefined - it can print garbage, it can crash, it can look fine for months and blow up in production.

AddressSanitizer (-fsanitize=address) catches this kind of bug on the spot. Run with it and you’ll see:

==12345==ERROR: AddressSanitizer: heap-use-after-free on address 0x602000000010

Diagnosis vs prevention

Great. Amazing tool. So why not stay in C/C++ with sanitizers?

Because a sanitizer is diagnosis. Rust is prevention.

You might think that passing -fsanitize=address to the compiler makes this a compile-time thing. It doesn’t: the compiler doesn’t analyze whether the bug can exist, it just instruments the binary with checks that run along with the program. The flag is a compile flag; the detection happens at runtime.

A sanitizer runs the code and tells you: “here, on this specific input, on this specific execution path, at that moment, you accessed freed memory”. If the path with the bug never runs, the sanitizer catches nothing. It’s a testing tool - it only finds bugs on the paths you actually execute.

Rust’s borrow checker is static analysis. It tells you: “this code can have a use-after-free, no matter the input, the path, or the moment”. It catches the bug before the code exists. Before compiling. Before running. Before production.

Static analysis is not exclusive to Rust - your editor may have caught the example above. But in C it’s heuristic: it nails the easy, straight-line case and misses the hard ones. The borrow checker is neither optional nor a guess - it either proves the code is safe, or it doesn’t compile.

It’s the difference between a blood test and a vaccine. A sanitizer detects the disease after it shows up. Rust keeps it from showing up.

How this bit Bun

In Bun, the reentrancy bugs were non-deterministic. They depend on timing, on which callback runs first, on how many requests are in flight. A sanitizer can run a thousand tests and catch nothing, because the timing never lined up. The borrow checker catches it every time, because the problem is structural - two mutable references to the same data at the same time is not a timing issue, it’s a design issue.

A good share of the bugs listed in Bun are use-after-free. The most common pattern is reentrancy: JS code comes back in the middle of a native operation and invalidates the state the operation was using. Example: a hashmap that grows and reallocates everything internally, leaving dangling pointers to the old memory.

The real bugs:

• node:zlib - heap-use-after-free when calling .reset() while an async .write() is still running on the threadpool. valueOf()/toString() as the attack vector.
• node:http2 - reentrant JS callbacks (session.request() inside a listener) trigger a hashmap rehash, invalidating internal stream pointers.
• UDPSocket.send()/sendMany() - valueOf() detaches the ArrayBuffer between capturing the payload and actually sending it.
• Buffer#copy/Buffer#fill - valueOf() detaches/resizes the ArrayBuffer during argument coercion.

Double-free

If use-after-free is accessing memory after freeing it, double-free is freeing the same memory twice. But it’s not as simple as free(ptr); free(ptr); in the same function. Nobody does that. The problem is when two owners don’t know about each other.

// Compile: gcc -fsanitize=address -g df.c -o df && ./df
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

typedef struct {
    int fd;
    char name[64];
} Pipe;

// Function A: schedules an async close
// In practice, this would be libuv scheduling a callback for the next tick
void close_pipe_async(Pipe* pipe) {
    printf("  [async] Close scheduled. Callback will run later...\n");
    // The on_pipe_close callback will run on the next tick of the event loop
    // and will call free(pipe) as well
}

// Function B: the async close callback (runs later)
void on_pipe_close(Pipe* pipe) {
    printf("  [callback] Freeing pipe...\n");  // doesn't touch pipe->fd: it's already been freed
    free(pipe);  // Second free - but the scope already freed it!
}

void spawn_subprocess() {
    Pipe* pipe = malloc(sizeof(Pipe));
    pipe->fd = 42;
    strcpy(pipe->name, "stdout");

    // Schedule the async close - the callback will free the pipe later
    close_pipe_async(pipe);

    // But at the end of the scope, the pipe is freed too
    // In Zig/C, it's easy to forget the callback already owns it
    printf("  [scope] Freeing pipe (fd=%d)...\n", pipe->fd);
    free(pipe);  // First free

    // ... later, on the next tick of the event loop:
    on_pipe_close(pipe);  // Second free - DOUBLE-FREE!
}

int main() {
    spawn_subprocess();
    return 0;
}

Two different functions, two paths that both believe they own the same pointer. In practice, that’s how double-free happens: an async path that frees, and a sync path that also frees, and neither knows about the other.

In Bun, this bug came from the exact pattern the code above shows: uv_close schedules an async close, and the scope that called close also frees the pointer. Jarred’s adversarial review caught this bug before the merge - the fix was Box::leak(pipe) to transfer ownership to the callback (commit f0a454376c7).

Memory leak

A leak is allocating memory and never giving it back. It doesn’t crash. It just keeps eating RAM until the process dies or the OS kills it.

The most common pattern: error paths. You test the happy path. Nobody tests the error path.

// Compile: gcc -g leak.c -o leak && ./leak
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int read_config(const char* path) {
    char* buffer = malloc(4096);
    char* temp = malloc(1024);

    if (strcmp(path, "invalid") == 0) {
        // Error! But buffer and temp were allocated and never freed.
        // On the happy path, there's a free() at the end.
        // Here? Forgotten.
        return -1;
    }

    // ... process config ...
    free(buffer);
    free(temp);
    return 0;
}

int main() {
    read_config("invalid");
    // buffer and temp leaked. If this runs on a 24/7 server,
    // that's 5 KB per call. A thousand calls = 5 MB. A million = 5 GB.
    return 0;
}

Bun’s most subtle bug was a reference count underflow: the reference counter went below zero and wrapped into a huge number (4,294,967,295 in unsigned). The GC thought there were still billions of references and never collected the object. fs.watch() leaked permanently because of this.

The real leaks:

• crypto.scrypt - callback buffers and protected password/salt never freed when the output buffer allocation fails.
• SSLWrapper.init - the strdup of the passphrase leaked on error paths.
• tlsSocket.setSession() - every call leaked an SSL_SESSION (~6.5 KB). Missing SSL_SESSION_free after d2i_SSL_SESSION.
• fs.watch() - reference count underflow pinned watchers as GC roots permanently. Never collected, even after .close().
• DuplexUpgradeContext - full leak via tls.connect({ socket: duplex }). Never freed.

Bun improved its integration with LeakSanitizer to track native memory allocations - but as we saw in the use-after-free section, sanitizers are diagnosis, not prevention.

RAII: the concept that solves all of this

By now it sounds like C is a nightmare and we should all give up. But the solution exists, and it’s older than many people reading this post.

RAII is a pattern from the early days of C++, named by Bjarne Stroustrup (the creator of the language). The name is terrible: Resource Acquisition Is Initialization. The idea is simple: the resource is acquired when the object is constructed and released when it’s destroyed. Automatic. Nothing to forget.

Let’s see the difference in practice.

Without RAII (C)

In C, you are responsible for every malloc() and every free(). Every error path is a place where you can forget.

// Compile: gcc -g no_raii.c -o no_raii && ./no_raii
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

typedef struct {
    char* data;
    size_t len;
} Buffer;

Buffer* buf_create(size_t size) {
    Buffer* b = malloc(sizeof(Buffer));
    if (!b) return NULL;

    b->data = malloc(size);
    if (!b->data) {
        free(b);      // have to remember to free b if data fails
        return NULL;
    }
    b->len = size;
    return b;
}

void buf_destroy(Buffer* b) {
    if (b) {
        free(b->data);
        free(b);
    }
}

int process(const char* path) {
    Buffer* buf = buf_create(4096);
    if (!buf) return -1;

    Buffer* extra = buf_create(1024);
    if (!extra) {
        buf_destroy(buf);   // have to remember
        return -1;
    }

    if (strcmp(path, "bad") == 0) {
        buf_destroy(extra);  // have to remember
        buf_destroy(buf);    // have to remember
        return -1;
    }

    printf("  OK: %s\n", path);
    buf_destroy(extra);
    buf_destroy(buf);
    return 0;
}

int main() {
    process("good");
    process("bad");
    return 0;
}

3 resources, 2 error paths, 6 places where you have to remember to free. Forget one? Leak. It scales linearly with the number of resources.

With RAII (C++)

In C++, destructors run automatically when the object goes out of scope. Even on error paths. Even with exceptions.

// Compile: g++ -g raii.cpp -o raii && ./raii
#include <iostream>
#include <memory>
#include <string>

class Buffer {
    std::unique_ptr<char[]> data;
    size_t len;
public:
    explicit Buffer(size_t size)
        : data(std::make_unique<char[]>(size)), len(size) {
        std::cout << "  + Buffer allocated (" << len << " bytes)\n";
    }

    ~Buffer() {
        std::cout << "  - Buffer freed (" << len << " bytes)\n";
    }

    Buffer(const Buffer&) = delete;
    Buffer& operator=(const Buffer&) = delete;
};

int process(const std::string& path) {
    Buffer buf(4096);
    Buffer extra(1024);

    if (path == "bad") {
        std::cout << "  Error! Bailing out...\n";
        return -1;
        // buf and extra are DESTROYED automatically here.
        // Even on the error path. Impossible to forget.
    }

    std::cout << "  OK: " << path << "\n";
    return 0;
    // buf and extra are DESTROYED automatically here too.
}

int main() {
    std::cout << "=== Happy path ===\n";
    process("good");

    std::cout << "\n=== Error path ===\n";
    process("bad");

    return 0;
}

Run it and see:

=== Happy path ===
  + Buffer allocated (4096 bytes)
  + Buffer allocated (1024 bytes)
  OK: good
  - Buffer freed (1024 bytes)
  - Buffer freed (4096 bytes)

=== Error path ===
  + Buffer allocated (4096 bytes)
  + Buffer allocated (1024 bytes)
  Error! Bailing out...
  - Buffer freed (1024 bytes)
  - Buffer freed (4096 bytes)

The destructors run on every path. Happy, sad, exception, early return. The compiler guarantees it. There is no way to forget.

The same idea fixes the SSL_SESSION bug that leaked 6.5 KB per call in Bun:

// Compile: g++ -g ssl_session.cpp -o ssl_session && ./ssl_session
#include <iostream>
#include <cstdlib>

class SSLSession {
    void* session;
public:
    explicit SSLSession() {
        session = malloc(6500);  // simulates d2i_SSL_SESSION
        std::cout << "  + SSL_SESSION allocated\n";
    }
    ~SSLSession() {
        free(session);  // simulates SSL_SESSION_free
        std::cout << "  - SSL_SESSION freed\n";
    }
    SSLSession(const SSLSession&) = delete;
};

void set_session() {
    SSLSession sess;
    // sess is destroyed when it goes out of scope
    // IMPOSSIBLE to forget to free
}

int main() {
    set_session();  // allocates and frees automatically, no hand-written SSL_SESSION_free
    return 0;
}

If the developer forgot to call SSL_SESSION_free() by hand, it leaked. With RAII, the destructor does it automatically. It’s the compiler guaranteeing what human memory forgets.

RAII in Rust: Drop + borrow checker

Rust adopts RAII through Drop - the equivalent of a C++ destructor. The difference is that Rust goes further: the borrow checker prevents use-after-free at compile time.

In C++, RAII solves leaks and double-free. But use-after-free is still possible - a raw pointer can point to memory the destructor already released. In Rust, the borrow checker closes that door: it won’t let two mutable references to the same data coexist.

That http2 reentrancy bug - taking a mutable reference to the stream and, midway through, letting JS mutate the same structure - doesn’t even compile in Rust. The compiler refuses with error[E0499]: cannot borrow as mutable more than once at a time. It’s not discipline, it’s not convention, it’s not a test you have to remember to run: the code simply doesn’t get past the compiler.

What in Zig was a non-deterministic crash in production, in Rust becomes a deterministic compile error.

The Bun team even tried to emulate this in Zig, with homegrown smart pointers:

fn foo(a_ptr: SharedPtr(TCPSocket)) !void {
  const a: *TCPSocket = a_ptr.get();
  defer a_ptr.deref();

  const b = try do_something_with_a(a);
  defer b.deref();

  // ...
}

Notice that every resource requires a hand-written defer - and someone has to remember to write each one. Jarred himself admits it:

“Homegrown smart pointers offer worse ergonomics than Rust, with none of the guarantees.”

Summary

Adapted from the table in Jarred’s post, the cleanup mechanism of each language:

And what each concept solves in practice:

Strip away the noise and Bun’s migration is about one thing: the class of bug that shows up when GC and manual memory mix, and which tool prevents it by construction. RAII takes care of leaks and double-free. The borrow checker takes care of use-after-free from reentrancy.

A counterpoint is worth mentioning. Andrew Kelley, Zig’s creator, responded to the migration saying the problem was never the language - it was code quality. And he has a point: TigerBeetle writes Zig without these bugs. You can avoid all of this in Zig. Jarred himself admits he doesn’t blame Zig.

But “you can avoid it with discipline” and “the compiler won’t let it happen” are different things. It’s the same difference from the beginning of this post: the sanitizer you need to run on the right path, versus the borrow checker that catches it every time. Kelley bets on the team; Rust bets on not needing a perfect team for this class of bug.

The bugs listed in Jarred’s post? In safe Rust, almost none would survive. Use-after-free and double-free become compile errors - the borrow checker and ownership refuse them. The leaks, Drop prevents on its own.

Thanks for reading!

Yesterday I opened Claude Code and sent a prompt. A single prompt. No back and forth, no corrections, no guiding. I pasted it and went to make coffee:

create a compiler from Brainfuck to JVM in Node.js with no dependencies. Full Node.js.

the interface should be ./compiler somefile.bf SomeFile

i should be able to run SomeFile just by running "java SomeFile"

every brainfuck command should be accepted and the produced .class must not have a warning.
You must test everything, from the parser to the generation of the .class.

When I came back, three minutes later, the compiler was done. 51 tests passing, a valid .class, Hello World running with no warnings on stderr. One prompt. Three minutes.

My first reaction was to laugh. The second was to go quiet.

What it generated

Claude’s compiler is a single file, ~300 lines. Parser, constant pool, bytecode generation, StackMapTable, .class assembly - all in one place. It works. All eight Brainfuck commands compile correctly to JVM bytecode. The loops generate ifeq/goto with correct offsets. The append_frame in the StackMapTable declares the right local variables. javap -v shows a valid class.

The test suite is decent: it tests the parser in isolation, the constant pool, the bytecode generation, the StackMapTable, the structure of the .class, and runs 19 integration tests that compile Brainfuck, execute it with java, and check the output. Hello World, nested loops, input, cell wrapping, all 8 commands together.

It works. No tricks, no hacks. It even got the append_frame right on the first try - fine, on the second. On the first it used same_frame and got a VerifyError, exactly like I did. But it fixed it by itself in seconds.

Where mine is better

BrainJuck is not just a compiler that works. It’s a compiler that thinks.

Claude’s parser takes +++ and generates three separate iadds. My parser combines them into an increment(3) and emits a sipush 3 + iadd. Less bytecode, less work for the JVM.

BrainJuck has an IR layer between parsing and code generation. That means I can add optimizations without touching the bytecode generator. Claude went straight from parsing to bytecode - it works, but it’s rigid.

Mine tracks the pointer position at compile time. >>><< becomes a move_head(1) with an absolute position. Claude’s generates five separate iincs. Again - it works, but it’s naive.

The architecture has real separation: index.js for the parser, class_generator.js for .class assembly, helpers/jvm.js for the opcodes. Claude stuffed everything into one file. For 300 lines, that’s fine. For evolving the thing, it’s not.

Where Claude’s is better

The test coverage. I have to admit it.

I have 6 unit tests and 1 integration test. Claude generated 51 tests covering each layer individually. It tests edge cases I hadn’t even thought of - the empty loop [], consecutive loops [][], cell wrapping, an empty program.

My integration test is solid - it compiles Hello World, runs it with java, checks stdout and stderr. But it’s one scenario. Claude tested 19 different integration scenarios.

What actually matters

This is where things get honest.

Claude’s compiler works. If someone came to me with “I need a Brainfuck-to-JVM compiler by tomorrow” and I used Claude, the job would be delivered. Nobody would look at the generated .class and know it was made in 3 minutes.

But I would know nothing.

I wouldn’t know that the constant pool is 1-indexed and that entry 0 doesn’t exist. I wouldn’t know that baload sign-extends to int. I wouldn’t know that slot 0 of the local variables is reserved for the method’s arguments. I wouldn’t know that the StackMapTable’s offset_delta is calculated relative to the previous frame, not to the start of the method. I wouldn’t know the difference between same_frame and append_frame, or why the first frame of a method with branches has to be an append_frame if you declared local variables beyond the signature.

None of it. My head would be empty.

When I was debugging BrainJuck at two in the morning, comparing hex dumps against the JVM spec, getting the offset math wrong for the tenth time - that was knowledge going in. Every VerifyError was a lesson. Every wrong byte I found in the hex dump was a new connection in my brain.

Claude didn’t have to debug any of that. It already knew. It was trained on the JVM spec, on thousands of similar implementations, on decades of accumulated knowledge. For Claude, generating a correct StackMapTable is interpolation. For me, it was the final boss of the project.

It’s not about the AI being bad

I use AI to write code. I use Claude Code practically every day. I’m not against it, I don’t think it will destroy the profession, I’m not afraid of losing my job. Let that be clear.

But there is a difference between using AI to speed up work you understand and using AI to do work you don’t understand. In the first case, you gain time. In the second, you gain an illusion.

BrainJuck took weeks. Claude’s compiler took 3 minutes. The end result is similar - both generate a valid .class, both compile Hello World, both pass the JVM verifier. But after those weeks, I know how the JVM works on the inside. I can read bytecode, I know what a VerifyError means, I can dissect a .class with xxd. Those weeks gave me something no prompt does.

Claude’s compiler is better tested than mine. It’s faster to produce. If I wanted, I could take its code and improve it - add the optimizations mine has, split it into modules, evolve the architecture.

But I would never have the foundation to do that if I hadn’t gone through those weeks first.

The real test

If I hand you Claude’s compiler and ask you to add an optimization - collapsing [-] into a direct clear cell in the bytecode - could you do it? If a new VerifyError shows up, would you know where to start?

If you built your own, the answer is yes. If AI built it for you, the honest answer is probably no.

That’s the point. It’s not that AI writes bad code. The code is good. It’s that good code you don’t understand is exactly as useful as bad code you don’t understand. Either way, when it breaks, you’re lost.

If you’re learning

If you’re a developer and you want to understand compilers, virtual machines, bytecode - build one by hand. At least once. It doesn’t have to be Brainfuck, it doesn’t have to be the JVM. Pick any simple language and compile it to any target. What matters is going through the process.

Once you understand it, then yes - use AI to move faster, to test more scenarios, to explore variations. AI is an absurd tool when you know what you’re doing. When you don’t, it’s just a generator of false confidence.

BrainJuck is on GitHub. Read the code, play with it, break it. And if you feel like it, build your own.

Thanks for reading!

It doesn’t. And to convince you of that, I went into the source code of OpenCode to see what actually happens under the hood.

But first, I need to explain how LLMs really work. Without that, skills make no sense.

LLMs do nothing

An LLM is a function. Text goes in, text comes out. It doesn’t access the internet, doesn’t read files, doesn’t execute code. It predicts the next token based on what it received.

That is literally all.

When you send a message to Claude or GPT and it “reads a file” or “searches the web”, it’s not the model doing that. It’s the system around it. The model only generates text. The one that acts is the program orchestrating the conversation.

Tools: giving the model hands

For an LLM to interact with the real world, we use tools (or function calling). The flow is:

1. You send a message along with a list of available tools - each with a name, a description, and parameters
2. The model analyzes the message and decides whether it needs to use a tool
3. If so, it responds asking for the execution: { "tool": "read_file", "args": { "path": "src/main.ts" } }
4. The host system (not the model) executes the tool and returns the result
5. The model receives the result and continues generating the response

The model never executes anything. It only asks for execution. The agent is the one that runs it.

Agents: the loop that connects everything

An agent is that loop. Simplified to the extreme:

while not done:
    response = llm.generate(messages, tools)
    if response has tool_call:
        result = execute(tool_call)
        messages.append(result)
    else:
        return response

The agent keeps the history, injects the tool definitions, executes the calls, and feeds the model with the results. OpenCode does exactly that in packages/opencode/src/session/prompt.ts.

If you want to truly understand agents, study this loop. Everything else is implementation detail.

How OpenCode registers tools

In OpenCode, every tool implements a Tool.Info interface defined in packages/opencode/src/tool/tool.ts:

export interface Info<Parameters, M> {
  id: string
  init: (ctx?) => Promise<{
    description: string
    parameters: Parameters
    execute(args, ctx): Promise<{ title, metadata, output }>
  }>
}

Every tool has an id, a description, the parameters it accepts, and an execute function. The ToolRegistry in packages/opencode/src/tool/registry.ts gathers all of them - built-in, custom, and from plugins - and hands them to the model on every interaction.

The built-in tools are registered like this:

return [
  ReadTool, GlobTool, GrepTool, EditTool, WriteTool,
  BashTool, TaskTool, WebFetchTool, SkillTool,
  // ...
]

Notice that SkillTool in the middle. Remember that name.

Now we can talk: what is a skill?

A skill in OpenCode is a markdown file named SKILL.md with a YAML frontmatter:

---
name: agents-sdk
description: Build AI agents on Cloudflare Workers using the Agents SDK
---

# Cloudflare Agents SDK

Here go detailed instructions, code examples,
references, best practices...

That’s it. A .md file with a name and a description.

The code that discovers these files lives in packages/opencode/src/skill/skill.ts. It scans for SKILL.md in global directories (~/.claude/skills/, ~/.agents/skills/), project directories (.opencode/skills/), custom paths, and even remote URLs.

The trick: SkillTool is just a tool

The SkillTool in packages/opencode/src/tool/skill.ts is a tool like any other. In its init() function, it:

1. Scans all the available SKILL.md files
2. Builds its own description listing what it found:

<available_skills>
  <skill>
    <name>agents-sdk</name>
    <description>Build AI agents on Cloudflare Workers...</description>
  </skill>
</available_skills>

That description goes to the model along with the other tools. When the model decides it needs a skill, it makes a regular tool call:

{ "tool": "skill", "args": { "name": "agents-sdk" } }

The SkillTool receives that call, reads the corresponding SKILL.md, and returns its content. The model uses those instructions to continue the work.

Read that again. It’s the same flow as any tool. The model asks, the system reads a file, the content goes back into the context.

Think of it this way: if the model can call read_file to read a code file, why can’t it call a tool to read an instructions file? That’s exactly what SkillTool does. The only difference is the convention - a standardized place to put reusable instructions that the model pulls on demand.

How to create a skill, for real

1. Create a folder inside .opencode/skills/ (or .claude/skills/, depending on the agent)
2. Put a SKILL.md inside it with name and description in the frontmatter
3. Write the instructions in markdown

Done. There is no step 4.

The model will see your skill’s short description in the list of available tools. If it finds it relevant to what it’s doing, it will call the tool and read the full content. If it doesn’t, it ignores it. You don’t force anything.

The full content only enters the context when the model asks for it. The descriptions are lightweight; the heavy markdown stays out until it’s needed.

One tip on what to put in a skill: Sean Goedecke wrote about generating skills after solving the problem, not before. The idea is that the LLM writes better skills after it has already iterated on the solution, because then it distills what it learned. It makes sense - you don’t write good documentation before understanding the problem.

Conclusion

A skill is a markdown file that a tool reads when the model asks for it. The same tool calling that lets the model read files or run commands is what lets it load a skill. There is no framework, no runtime, no magic.

If you know how to write markdown, you know how to create skills.

Thanks for reading!