He thinks that distrust is a sensible place to start. Agents can be wrong, so if you let them touch business operations, you need safeguards around their decisions.

Let’s start with why he doesn’t trust them.

Your talk at WAD San Jose starts with a strong statement: “I don’t trust AI agents.” What exactly shouldn’t developers trust about them?

Darko: The reason I don’t trust my AI agents is that they’re ultimately probabilistic systems. They’re not deterministic. When you give agents serious responsibilities, such as handling refunds, selling things or dealing with sensitive customer data, relying on a probabilistic system is a problem. Agents by themselves, without guardrails around them, are problematic.

I’ve spent a lot of time teaching developers how to approach this. Building agents is more than having an agent and access to an LLM. It’s about building the system around it, with guardrails and everything you need to make the agent a little more secure.

What’s the biggest difference between an AI agent that works in a demo and one that is actually ready for production?

Darko: Believe it or not, cost. When we build an agent for a demo, we might pick the biggest model because we want to make sure the demo works.

The challenge comes when you start scaling that out. When an agent handles hundreds of thousands of customer requests in different ways, cost becomes an issue. Optimization isn’t something you worry about in a demo. You don’t worry about prompt caching, RAG or those things.

In an actual production system, you start to consider cost. Beyond safety and guardrails, that’s the biggest difference between a demo agent and a real-life production agent.

When building an agent, what safety layer should developers add first: guardrails, permissions, observability or something else?

Darko: The answer is yes to all of those things, because proper defense is built in layers. A single layer, a single guardrail, may not prevent everything. You may want guardrails, MCP input validation and deterministic policy checks around certain tool calls. All of those things are important.

As I mentioned in my talk, observability becomes incredibly important because you really don’t know what your agents are doing, especially at that scale. Being able to observe them and run evaluations periodically or in real time gives you a lot more peace of mind. So, to answer the question of which one to add, the answer is yes.

How do you know when an AI agent has made a mistake if its output still looks perfectly reasonable?

Darko: You have to combine a few things, and we go back to observability. Implementing automated tracing is critical so you can trace what’s going on in real time. But tracing doesn’t stop at the agent. You also need to trace the backend system the agent works with.

If you start detecting anomalies, such as an increasing number of small or large orders, or something unusual, you need to look at both the agent and the backend. Combining those signals into a proper anomaly detection system is how you could potentially catch certain things happening in real time.

It’s never just an agent. It’s an agent plus everything else behind it.

How much autonomy should we actually give AI agents in production? Where should developers draw the line?

Darko: Ultimately, responsibility for what happens, for a system or code that goes into production, rests with the developer.

How much autonomy should we give agents? It depends on the blast radius of a change or of a system going down. Is this a simple chatbot that tells customers the opening times of your business? Sure. If something goes wrong, it’s not horrible.

But if the system handles banking information or sensitive data, or processes purchases, you may want to reduce its autonomy. It’s not only about the code or change being submitted; it’s also about the system you’re submitting it to. Unfortunately, the answer here is: it depends.

If a developer is shipping their first production AI agent tomorrow, what are the three things they absolutely need to have in place?

Darko: Number one, proper observability. They need to be able to observe and monitor what their agent is doing, run evaluations against the models and their decisions and choices, and constantly monitor those things for changes.

Number two, policies. A policy language such as Cedar lets you put policies in front of tool execution so you can prevent certain tools from executing.

Number three, consider an established harness. It’s tempting to build your own because we’re all developers, but I encourage developers to consider something like the Strands SDK, LangGraph or LangChain. It makes things easier and lets you focus on your business problem and spend less time figuring out how to wrangle the agent.

The post You shouldn’t trust AI agents in production appeared first on ShiftMag.

In one PR, a colleague replaced the legacy JSP gateway-edit page with a React single-page app. Because that meant changing the UI stack, the backend had to move with it: new REST controllers, a new application service, new tests, a Maven frontend build, and an early commit that deleted everything nobody referenced anymore.

The diff, against master:

1186 files changed, 19809 insertions(+), 473971 deletions(-)

Bitbucket helpfully informed me that only the first 1000 files are shown. That number was the whole reason this turned into a project of its own.

So I sat down and asked myself: how do I actually do this well?

Review fails through lack of coverage, not lack of skill

Doing it manually was not realistic. I’d read for days, hold a shrinking amount of the change in my head, get tired, and still miss things, and the things I’d miss would be the boring, important ones (a field that quietly lost its validation or an action that no longer exists).

The failure mode of manual review at this size is not “I review it wrong”, it’s “I review 15% of it and quietly convince myself that was enough”.

The obvious alternative, throwing the diff at an LLM and asking “review this”, is worse. The diff alone doesn’t fit in the context window. And even if it did, the agent has no baseline: “is this line correct?” is a much weaker question than “does this behave like the thing it replaces?” When you’re replacing a UI, the interesting question is never does the new code look right, it’s does the new code still do what the old code did. You can’t answer that from the diff.

I needed an approach that kept the context small enough for the agent to be reliable, while still covering the whole change.

What I had going for me (and what I didn’t)

Before choosing an approach, it’s worth being honest about the shape of the thing you’re reviewing.

In my favor: this is a relatively small CRUD service with only a handful of flows. A list page, a copy flow, an edit form, not a sprawling distributed system with a dozen integration points. That matters, because it means the surface a rewrite can break is finite and enumerable. Users had already been testing the new UI on IO (Integrating Environment), so the obvious, everyday path (open a gateway, change a field, save) had human eyes on it before the PR ever reached me.

Which is exactly why I didn’t trust it. People test the happy path. They click through the flow that works, confirm the screen looks right, and move on. Nobody spends their afternoon feeding a 4000-character string into a 100-column field, or pulling the session out from under a save.

So I deliberately weighted the review toward the unhappy paths: missing validations, error handling, what happens when a write fails halfway through. A green light from testers on IO told me the good path worked but told me almost nothing about the bad ones.

And one thing working against me: I already knew, from weeks of troubleshooting this service, that it had holes: plenty of its fields were not properly validated or handled to begin with. So “the new code matches the old code” was never going to be enough on its own. If the baseline is leaky, faithfully reproducing the baseline reproduces the leaks.

That’s part of why the review ended up covering DB constraints and failure modes rather than stopping at parity, and a big reason so many of the findings turned out to be inherited rather than new.

Divide and conquer

Two things made the difference.

First, clone both sides side by side. I checked out master and the PR branch into two separate directories:

./master/infobip-gateway-edit      ← the old JSP page, the baseline
./pr_branch/infobip-gateway-edit   ← the React/Bepo SPA

That sounds trivial but it changes the whole exercise. Now “what did master do here?” is a file read, not a guess. Every claim the review makes can be traced back to a line in the old code. The agent is comparing two concrete states, and not reasoning a diff.

(I also cloned the two SQL repositories that hold the table definitions, because part of the review ended up being “does the new code respect the actual database constraints?”. More on that later.)

Second, break the review into small steps and do one per session. This is the part that made the agent reliable. A single “review this PR” prompt would have produced a confident, shallow wall of text. Sixteen focused prompts produced something I could actually trust, because each one had a narrow scope, a written-down baseline, and a bounded amount of context to hold.

The prompt I gave the agent did not say “review the changes”. It asked a narrower, more provable question, and it did so in very concrete terms.

The instruction that started it

Here is the first prompt in full: it names what we’re checking, the order, the output files, and how we’ll work together.

Two things are doing most of the work here. The prompt leads with a negative instruction (don’t review the diff), which steers the agent away from the obvious-but-wrong approach. And it hands over a concrete first output (three named files), a stopping rule, and a coordination protocol (stop, update the plan, report back). None of that is clever; it’s just specific. That specificity is what kept sixteen sessions coherent.

Why the plan lives in a file

The move that mattered most wasn’t clever. It was one of the earliest practices in agentic coding: a plan file (REVIEW_PLAN.md) with a status table at the top:

The workflow was:

1. Agent does one step.
2. Agent stops, updates the status table, and reports back.
3. I read the result and decide if it’s good enough.
4. I compact the session (context reset), give the green light, and the agent starts the next step.

Two things come out of this. The agent never has to hold more than one step in mind, which keeps the context (and therefore the hallucination risk) small. And I never have to hold the whole PR in mind either, because the plan file does that for me. If I come back a week later, REVIEW_PLAN.md tells me exactly where we are. (This turned out to matter: I later re-checked comments after the developer pushed new commits, and the plan file meant I didn’t have to reconstruct anything.)

The other side of the workflow: the intermediate documents are baselines. MASTER_FIELDS_DOCS.md lists every field on the old page with its type, options, and visibility rules. MASTER_ACTIONS_DOCS.md lists 17 groups of user actions.

MASTER_VALIDATIONS_DOCS.md lists 47 numbered validations (V1–V47). Each entry names its source file and line, so the parity check in a later step is mechanical rather than interpretive: you can diff a document against the new code, which is much more reliable than asking an agent to “look for missing validations”.

Building on the baseline (phase 2)

Once the three parity axes were done (fields, actions, validations), I had a solid, evidence-backed answer to “did the rewrite lose anything?” That’s the question a UI rewrite can actually get wrong in a way users notice.

But parity isn’t correctness. A rewrite can keep every field and every rule and still be broken in ways the old page wasn’t. So I extended the plan with a second phase, still small steps, still one per session, this time walking the application like an engineer building it from scratch and asking which parts have to work for this to be safe to ship?

I kicked off that phase with a second prompt, the same shape as the first: a concrete goal, the scope I was deliberately excluding, and an ordered list of steps.

I wrote these steps the way I’d audit a service: transactions, then failure paths, then who’s allowed in, then the contract, then the runtime, then the edges. Each one is a standalone unit of “does this part work?”, so the agent could check it in isolation without needing the other fifteen steps in context.

Two of the steps earned their place in ways I didn’t predict:

• DB constraints (step 7) turned into the single most valuable step. Comparing the form metadata against the actual DDL found that 43 text columns had no length enforcement anywhere, client or server, so an over-long value would travel all the way to SQL Server and come back as “String or binary data would be truncated”, with the save handler flattening every database error into one generic message. There’s no way to find that by reading a diff; you find it by resolving each form field to its column and checking the constraint is enforced somewhere.
• Caching (step 13) found that the new SPA loads its bundles from fixed, unversioned URLs, so a deploy could leave browsers running last week’s JavaScript against this week’s backend. That one was real, and it got fixed. (Which is its own small lesson: a review that produces at least one concrete, fixed bug has paid for itself.)

What I didn’t review on purpose

I ignored code quality and organization. Not because it doesn’t matter, but because I had to choose, and at 1186 files I couldn’t do everything (see “the line” below). What I prioritised was app correctness and the things a user would notice. These include a missing field, a missing validation, a save that half-commits, an error page instead of an error message or a bundle that won’t refresh.

I’m confident there is dead code left in that PR. There is certainly some code that could be written more clearly, or factored differently. I flagged the small amount of dead code I tripped over while looking for correctness issues (unused imports, unreferenced configuration properties), but I didn’t go looking for it.

That was a conscious trade, not an oversight.

Tools

I used Pi with an AI model served from our own infrastructure.

I keep Pi deliberately bare (no MCPs and no skills) because I also use it with models running locally on my Mac, where RAM limits how much context I can hold, so I keep the harness lean to leave as much of the window as possible for the actual work. This review ran against the company-hosted model, so that constraint didn’t really bite here, but the bare setup is what I’m used to.

When a task genuinely needs tooling, I reach for other agents (Codex, Claude Code) that have it wired up. For this review what I needed wasn’t tools; it was a patient reader with one small, well-defined job per session, and that’s what the step-by-step structure provided.

What the review produced

All told, the process produced 16 review steps, 13 findings documents, and a set of PR-ready comments. Not every step produced a comment: several produced an explicit “no regression, no comment needed” verdict, which is just as valuable, because it’s a documented answer instead of a silent omission. A few of the things it turned up:

• A real caching bug in the new SPA bundle URLs (found, commented, fixed by the developer).
• A dropped keystore on copy-without-reupload (the copy path read the source gateway and then discarded it, so the row ended up with the old filename and password but a NULL keystore blob).
• A blank page when the SPA is opened at a URL that isn’t a copy URL: the URL parser throws before the promise .catch that renders the “cannot open the gateway” message, and there’s no error boundary, so the operator gets nothing.
• The dead action envelope field, and assorted smaller items.
• And on the other side, a genuine improvement worth calling out: the branch is stricter than master on server-side bounds: 18 @Range checks and the GatewaySpeed @DecimalMin/Max rules now run on save where master had them dormant. I asked the developer to confirm the stricter rejections were intentional, but it’s a good change.

A handful of the findings resolved in the developer’s favour, too: two “missing” grids turned out to be non-functional on master in the first place (their buttons were disabled and the permission helper always returned false), and one authorization concern from the plan turned out not to be a real gate at all.

Part of doing this well is being willing to be wrong in writing, and the baseline documents made that checkable rather than a matter of opinion.

The line you have to draw

Somewhere in a PR this size you have to decide you’re done. You will not read everything. You will miss nuances. There is dead code you didn’t flag and duplication you didn’t mention.

That’s not failure. It’s the job. The alternative is holding a 1186-file change to an impossible standard and never shipping the review. So I drew the line at app correctness, accepted that code-quality nits would survive, and moved on, because other things were waiting for my attention.

If you take one thing from this: scope the review to a small number of axis you can actually prove, write the baseline down, and let the structure (not willpower) keep you honest.

Credit where it’s due

A list of findings can give the wrong impression, so I want to be explicit about something: most of what this review flagged was not introduced by this PR. It was copied, faithfully, from the old master implementation.

The non-atomic copy (step 8), the generic “database error” that names no field (step 7), the audit that is best-effort and swallows its own failures (step 15), the permission helper that always returned false, the grids whose buttons did nothing (steps 5/10), the lengthless text columns: the new code reproduces the old behaviour, warts and all.

That is exactly what a rewrite is supposed to do, and finding inherited behaviour is not the same as finding a defect. I said as much in the write-ups: several findings are marked “parity” or “no regression” rather than “bug” for precisely that reason.

Which reframes the numbers. In a PR this size (a full UI stack replacement, plus the backend changes to support it), the count of genuinely new defects was small. Most of the review’s output is a map of what the old system already did, not a list of things the rewrite broke.

Large part of that is the developer’s work. He kept parity deliberately, wrote tests that pin behaviour, and fixed the real issues (the caching bug, the keystore drop, the case-sensitivity bug in the new field check) as they came up. But I also think it’s a genuine sign of how far LLM-assisted development has come.

Producing a large, coherent, mostly-correct rewrite, where the majority of what a careful reviewer finds is the old code’s pre-existing behaviour rather than new mistakes, is not something I would have expected a couple of years ago. The human still matters: someone had to guide the change, land it, and decide what was worth reviewing.

But the baseline quality of the large machine-written change was high, and the remaining edge cases were the exceptions, not the rule.

Caveats

This is one developer’s approach to one specific PR. It is not a general methodology and it isn’t a claim that everyone should review big PRs this way.

It worked well here for a particular reason: this PR was a rewrite of an existing behaviour, which gives you a natural baseline (the old code) and a natural set of questions (does the new code still do all of it?). A 1000-file PR that adds a new feature has no such baseline, and the same divide-and-conquer axes wouldn’t transfer directly.

A PR that’s mostly dependency bumps or generated code would want something else again.

Treat it as a worked example you can borrow from, not a rulebook. The transferable part isn’t the sixteen steps. It’s the habits: clone both sides, pick a small number of provable axes, write the plan down, keep each session narrow, and stop when you’ve covered what you decided mattered.

The post How to Review a 1,000+ File PR Without Losing Your Mind appeared first on ShiftMag.

I love this style: a function’s expected failures are spelled out in its signature as a sealed type, rather than hidden in its body.

A couple of years ago, I tried something very similar in a toy project, but I never saw the contract benefit. The style wasn’t the problem, and doing more of it wasn’t the fix.

This essay responds to a JetBrains blog post about typed domain errors. I agree with its diagnosis, though I’m clearer about what not to do than about the ideal approach. I also share what I got wrong, in the hope you can avoid the same mistakes.

The blog post I’m responding to

Sergey Chernov’s “Signatures, be true: domain errors and functional handling in Kotlin” makes a claim I strongly agree with: a signature like fun signDocument(id, code): Unit that throws in secret is lying about what it can do, and the fix is to put the expected failures in the type.

His core contribution is the three-category taxonomy, and it’s the most useful part of the post.

1. API client misuse, e.g. malformed JSON, a missing header or an out-of-sequence request. Collapse these to a coarse 4xx and keep them out of the domain model. Access control is a partial exception. A bare authentication failure is transport-level and stays out, but an ownership or permission decision that depends on business state often belongs in the union, because the caller needs the specific reason and not just a 403.
2. Unexpected exceptions, e.g. the database being down, a timeout, a dropped network or a bug. Throw these, and let the global handler turn them into a 500 with a stack trace and an on-call page.
3. Domain errors, e.g. a wrong sign code, a closed window or an already-signed document. This is the only category that belongs in the contract, as a sealed type.

A sealed type, in case that term is unfamiliar, is a closed hierarchy where the compiler knows every subtype. That is what makes a when over it exhaustive, with no else branch, and it’s the whole game.

Everything else in the post is implementation. It covers a sealed interface for the error union, an Either<DomainError, T> return, exhaustive when, HTTP mapping only at the route boundary (the Ktor layer that owns the routes). It also publishes the error types alongside the API client, and its central recommendation is one narrow union per public method, rather than one shared union across a class. Here Either is just a value that is either a Left failure or a Right success. The line I would keep is “the contract is the unit of review”, which means you read the types rather than the 200 lines underneath. That is doubly true when an AI wrote the implementation.

The post also skips over how to compose heterogeneous error unions, which is real work I had to figure out on my own. Real flows fail per step with different unions. A plain either { } block, which chains fallible steps and short-circuits on the first failure, infers a single left type from all of them. Mixing unrelated domain errors therefore gives you their least upper bound, meaning the most specific common supertype the compiler can find. When the types are unrelated, that is Any, which is the open-error problem the post warns about. A shared sealed interface is different, since it stays closed, but a union that broad is the contract bloat the post warns about elsewhere.

The post is also quiet about how to observe legitimate domain values. It says even less about the cost of mis-categorizing infrastructure failures as domain errors, which I ran into myself. The same open-type reasoning is why Result<T> is a poor fit for domain modeling. Its failure side is Throwable, so a when over the failure value always needs an else and the contract gains nothing.

A note on the boundary: API-client misuse vs validation

One line the taxonomy leaves murky is the line between API client misuse and validation. I read that line as two different things the post merges.

The first is structural or transport validation, such as malformed JSON, a missing field or a wrong wire type. This is genuine client misuse. A healthy client never produces it and no designed screen is needed, so a coarse 400 is enough.

The second is semantic validation, where a well-formed field violates a real rule and carries a meaningful message, like TooShortUsername or NoDigitsInPassword. Here the client benefits from knowing exactly which field is wrong and why, and often from getting all of the problems at once rather than just the first one.

Where the exact line falls is not well-defined, as it depends on the use case and on the implementor’s preference. Some APIs deliberately collapse semantic validation to a coarse 400. Others return rich, accumulated field errors, which is what my validation tree did and it’s the part I still like. My own leaning is to let structural junk stay a coarse 400 at the framework boundary. Semantic validation can come back as a value, accumulated, but kept as its own validation-result type rather than folded into the business domain union.

One wrinkle deserves naming. A rule like “this email is already taken” looks like validation, but it reads persisted state, so it’s really a domain conflict and belongs in the domain union, not in the request-validation result. It’s a fair criticism of the taxonomy that the first category needs this much boundary work, but it’s still a better starting point than nothing. I won’t pretend there is a single correct division here. It’s a judgment call for each API.

My case study: a good design at the wrong layer

In 2024 I built a Ktor/Kotlin/Arrow backend (instakt, now an unmaintained toy project) and went fully functional on error handling, but with no grouping by use case. This is how the style gets misused, and the taxonomy above is the frame I wish I had back then, because without it I mislabeled infrastructure failures as domain outcomes.

The design was coherent in principle. One wide root union DomainError held a large tree of cases, and my database bucket alone had about 26 of them. Services mostly returned EitherNel<DomainError, T>, though a few methods returned a non-accumulating Either<DomainError, T> instead. EitherNel is an Either whose failure side is a NonEmptyList, which is how several errors get accumulated.

A single total mapper DomainError -> HttpStatusCode owned all of the exhaustiveness. It handled every case and returned a status, while the response body carried the error detail. It looked elegant and it compiled. Reading it back, three things were wrong.

The most obvious problem was that the per-method contract was gone. Every method returned the whole DomainError union. A signature like upload(...): EitherNel<DomainError, ContentMetadata> told a reader almost nothing about which failures that one call could produce. The blog’s whole point, that the signature tells you what can go wrong for this call, was silently surrendered. Because the central mapper owned exhaustiveness, I avoided the impossible-branch work, but the signature stopped telling the truth.

The problem I most regret was observability. I had modeled infrastructure failures, a failed persist or a failed mail send, as union values. My exception<Throwable> handler logged only cause.message with no stack trace. Between the two, infrastructure failures were turned into a status by the mapper and never reached the global handler at all. That meant no log entry, no trace and no error-rate spike, which is exactly why the blog says unexpected exceptions should throw. Grouping errors by use case would not have fixed this, and I want to be explicit about that. It was a separate mistake that needs a separate fix.

The last problem was bloat. A shared root collects one more case from every operation, so there was no natural place to stop. Discovery got worse too, because every new failure landed next to unrelated ones.

What was actually good: validation

The part that aged best is validation, and that’s not a coincidence. Validation is the one concern where the design matched the shape of the problem. A form wants all of its errors at once rather than the first one, so the accumulating checks served a genuine UX need. Nowhere else in the project was accumulation load-bearing. Field validation is also the purest domain concern, because it judges the request itself and never needs the database, storage or infrastructure.

The nested tree earns itself here too, since field-level messages and group-level routing are two different consumers of the same structure. Each check is also a self-describing type such as ContentSizeIsValid, so a failure names exactly what was checked, even if the surrounding method signatures were still too wide to be useful.

The lesson I draw is that error handling works best when it’s shaped by the operation’s structure. In my experience independent checks want to aggregate, while sequential business steps want to short-circuit or narrow per use case. Validation happened to be the only thing that was structurally an aggregate problem, so it came out right almost by accident. Aggregate independent checks (like validation); short-circuit sequential business steps with a narrow, use-case-specific outcome. Validation happened to be the only thing that was structurally an aggregation problem, so it came out right almost by accident.

What I’d try next

I haven’t tested these in production, so treat them as hypotheses, not a proven design. The change I’d make first is to group by use case. By use case I mean the operation the caller asked for, not the function that happens to serve it. Uploading a profile photo and deleting a document are different use cases even when they share the same repository, and each one gets its own closed union of the failures it can produce. The wide union then lives only at the transport layer, where a single mapper to HTTP status genuinely belongs.

That leaves the composition question. A narrow service union widens into the transport union through a small total mapping at the boundary, and that mapping is where the composition happens. One function there turns each UploadError into the matching case of the wide union, as in UploadError -> DomainError, while the signature itself stays narrow, something like fun upload(...): UploadError. When a flow runs several steps, its union collects the cases those steps can produce that the caller must distinguish. That mapping is total, so adding a case stops the boundary compiling until it is handled, and it works the same whether the result is wrapped in an Either or a hand-written outcome type. It is deliberate and named, which separates it from the inferred collapse I described earlier, and it is acceptable only because the broad union never reaches a service signature. The cost is more up-front ceremony, a set of small types and a mapping per use case, but I think it’s worth it because the contract becomes readable, and I’d keep each mapping next to the type it converts so a new case stays a local edit.

For the result type, I now prefer plain when plus hand-written per-use-case outcome types over Either. I tried Either, and while it reads fine, the machinery around it wears on me. The cost is the Arrow either { } builder plus the chained calls that unwrap a success or short-circuit on a failure. It is not the Either type itself, which is a small sealed pair. As the years pass I’ve come to prefer the simplicity of a when and a set of small named types. This is a matter of taste, but it’s where I lean today. I’d also match the mechanism to the shape of the work. In my experience validation wants to accumulate, while a business flow wants to short-circuit through a narrow outcome type. I see no reason to force one mechanism everywhere.

Observability needs its own attention, and it matters more in this style, not less. Going from exceptions to domain errors means you lose the stack trace unless you deliberately keep it. So I’d log expected domain failures at info or debug with a correlation ID. The global handler would keep the error-level logging for thrown exceptions, so nothing gets logged twice. I’d still throw for genuine unexpected exceptions, the ones that cannot be categorized as domain or client. The rule I hold myself to is to build domain errors from data I already have and never wrap raw database or network failures into the union. That rule is still coarse. A bounced email is a domain outcome, while an unreachable mail server is not. The real test is whether a correct client could still hit the failure and act on it.

Narrow unions are also self-documenting in a way a wide one is not. A signature that returns a narrow, per-use-case error type tells a reader what can fail, while one that returns DomainError tells them almost nothing. For context, an early proposal in the Kotlin Evolution and Enhancement Process, KEEP-0441 “Rich Errors, aka Error Union Types”, would add error-union types to the language. It’s still at the design review stage, so it isn’t accepted or promised, and it isn’t an Arrow-specific idea. The error contract also has to survive releases. The concrete hazard is that adding a case to a shared sealed type is a breaking change for every client whose when is exhaustive. It also breaks serialization when an unknown case arrives from a newer producer. I’d treat an unknown incoming case as a defined fallback rather than a crash. This matters far less when the consumers are ours to rebuild in lockstep, as with internal models, and much more when a customer-facing connector is involved.

On conventions I don’t think there is one right answer, since it always depends on the use case. I like names that are clear and descriptive. I’d use a data class whenever an error carries data, and object or data object when it does not. A data object needs Kotlin 1.9 or later, and a data class requires at least one property. I lean toward carrying data where I can, because an error that carries a payload is easier to debug and test than one that carries nothing. For nesting, validation seems to benefit from it, while in general I’d strive for flat, per-use-case errors, though there’s no universal rule and it has to be tested in context. Testing gets easier too, because you assert on the returned error value directly instead of catching a thrown type.

Where I land

I still reach for this style, only with clearer boundaries. The blog’s taxonomy and its “contract is the unit of review” idea are worth keeping. Its flat “one narrow union per method” advice is a reasonable default at the level of the operation, though not for every helper function. A wide union with a central mapper still has a home at the transport layer.

The honest summary of my two years is that a coherent wide-union pipeline is a defensible transport design, but it was the wrong layer for it. I can’t promise the approach I’d try next is the right architecture either, because I haven’t built it yet. What I’m confident about is the failure mode. My first attempt taught me that the hard way, and I hope it saves you the tuition.

The post Two years later, I know what I got wrong about typed domain errors and what I’d keep appeared first on ShiftMag.

As someone who has spent nearly 20 years following developers, founders, and tech entrepreneurship, I spoke with Amir after his talk about what developers need to know to become founders. We discussed how AI is changing software development, junior hiring, product building, and what one person can achieve.

Developers are not doomed

What happens to developers now that AI can write code? It has probably become one of the most common questions at tech conferences over the past few years, ever since the AI revolution began. Naturally, I opened my conversation with Amir by asking exactly that.

Amir has programmed since childhood and spent over 20 years developing his own Vim configuration, which he says became one of the world’s most popular. Today, he barely uses the editor the same way and rarely reviews code in detail (except when security, models, or other areas need his attention).

Still, he doesn’t think developers are becoming irrelevant. Quite the opposite:

My contrarian view on this is: developers are still super critical, because I don’t think code was ever the objective. What is objective? Solving problems for customers, building systems.

In other words, writing code was never the whole point. Code serves as a tool, while solving customer problems and building systems remain the real goals.

What changes now is the level at which developers work. Managing agents, configuring them, coordinating multiple systems, and working with what Amir calls agent swarms all matter more and more.

That shift also dramatically changes the leverage a single developer can have:

I think the difference between a good developer and an amazing developer is not a 10x anymore. It’s 1000x.

Juniors may have an advantage that seniors do not

That raises another question: how can someone who enters the industry today learn what good engineering actually looks like?

Amir sees a potential advantage for junior developers here. Unlike seniors, they do not carry 20 years of habits that they first need to unlearn – they can start from first principles and treat agents, agent swarms, or systems that build other systems as a natural part of development, instead of trying to retrofit them into a workflow they have followed in the same way for years.

But that advantage has its limits: AI models still cannot replace the kind of experience that helps developers recognize whether a solution is actually good:

This said, I do still think the models are not good enough for a lot. Especially architecture.

Customers are still P1 for founders

AI may let one person build more, but a founder’s main job is still the same.

I asked Amir what developers should focus on when they also have to think about design, marketing, sales, hiring, and fundraising. Once again, he came back to the basics: customers:

Again, I’m actually not sure how much has changed, because I think in the end what you’re building for is customers, it’s other people, it’s empowering them. And you need to find out: how can you get customers? How can you build something useful? And that is maybe the P1. Everything else, attending conferences, getting investment, should not be a P1.

He says new founders often reverse that order. They start with incubators, try to raise money, or look for a co-founder, and only then ask whether anyone actually needs what they are building.

None of those things lack value. They simply do not belong at P1:

But P1 is finding customers, finding something useful, building something useful, trying it out, learning from that, getting kicked in your face. And then you go back and you rebuild and you learn. And also this loop is very critical.

In other words, build something, put it in front of users, make mistakes, learn from them, and build again. Amir sees that loop as one of the key parts of building a company.

Do not blindly copy Silicon Valley

From the very beginning, Todoist followed a different organizational model from the classic Silicon Valley playbook, but Amir did not choose that path because he had a grand theory about the future of work.

The company went remote because he lived in Chile and simply could not find enough people to hire locally.

At one point, he actually planned to move to Silicon Valley and start the company there, but then he met his wife and changed his plans.

That experience shaped another piece of advice he gives developers who want to become founders: instead of copying someone else’s formula, start with your own circumstances, abilities, and strengths:

I do think you need to work in what kind of environment you have, what kind of skills you have, what kind of strengths you have, and then work from that.

For that reason, founders should not treat his remote model as a template that automatically works for every company.

The same goes for following other founders too closely:

You follow what I have done, or you see something on X or some other founder, and then you just blindly follow them. That’s probably not going to lead to a very good outcome.

If you start an AI startup today, you may already be too late

AI has become impossible to ignore, swallowing the tech industry’s attention. Almost every startup wants to become AI-native, conversations turn to agents, and everyone looks to Silicon Valley again.

So I asked Amir whether developers outside this wave should feel they’re missing out. He gave one of the most direct answers of our conversation:

In a lot of cases, if you actually start an AI startup right now, you’re probably too late.

For him, understanding technological S-curves matters much more: founders need to recognize which stage a particular technology has reached.

Ideally, he says, you enter a trend before exponential growth begins. AI may already be deep into that curve, while space could still be near the start.

The reverse is true too: in mature markets, making a significant impact is much harder:

For instance, if you do SaaS right now, that is freaking mature. You have very, very entrenched companies. Of course, maybe by some miracle you can do something that’s 10x, but it’s highly unlikely.

Rather than chase the hottest field, founders should understand where the market and their company sit on the S-curve and where they want to compete. The same applies to fundraising: know what game you’re choosing to play.

Before taking VC money, you need to know which game you are playing

Todoist has remained bootstrapped, and when I asked how he thinks about company financing today, Amir returned to the same starting point: founders who lack traction, customers, proof that the product works, or a personal track record will struggle to raise money.

His preferred order therefore remains the same. First, build, learn, and find customers. Only then decide whether fundraising makes sense at all.

But another part of that decision matters even more. VC money comes with very specific expectations about what company success should look like:

VCs want to see billion-dollar outcomes. If you don’t have a realistic way to achieve that, even if you lie, say, okay, this is our path, but you don’t really believe it, at some point they are going to push you to their direction, and then they’re going to mess up your company, because that is what you’re signed up for.

The problem, then, does not lie in the VC model itself. The problem appears when a founder accepts that kind of capital without actually wanting to build the kind of company that the VC game requires.

That is one of the reasons Amir has never raised money for Todoist:

Why I have never raised money for Todoist is because I don’t want to build a billion-dollar ARR business.

Bootstrapping is like climbing Everest without oxygen

That does not mean Amir sees bootstrapping as an easier or more relaxed alternative to the VC path.

I also wanted to know whether people have romanticized bootstrapping in recent years. Founder stories often present it as a way to avoid investor pressure, keep control, and build a company at your own pace.

Amir dismantles that image:

But I think bootstrapping is incredibly hard. It’s not the easy path. What I actually tell my team is: we are climbing Mount Everest without oxygen, and with oxygen is the capital. That is what you’re signing up for.

So the real question is not whether bootstrapping beats VC funding. Founders need to ask what they actually want from their company and from their own lives.

Not every company fits the VC path. At the same time, bootstrapping comes with serious constraints of its own.

Todoist removed “balance” from its core value

Building a company for 20 years doesn’t mean 20 years of growth. Amir recalled that Todoist and Buffer followed a similar path: years of growth, then a plateau and slight decline.

As he puts it, growth can make you think you’re “the freaking smartest.” When it stops, you realize how much needs rethinking. At Todoist, that meant major culture changes, including dropping one word from a core value:

We actually had this core value called ambition and balance. And then we just removed balance.

Amir says that at one point he became deeply unhappy with how things worked inside his own company. He did not think the company showed enough ambition, he did not like the level of output, and he felt that the team no longer pushed boundaries.

The cultural shift went beyond words. During that process, the company let some people go, while others chose to leave:

And I think right now we have a lot more calibrated company because of that.

He sees an important lesson in that experience for new founders. The culture you create at the beginning, the first people you hire, and the first core values you define can have a huge impact on the company you eventually build.

Once the company grows, changing those foundations becomes much harder.

Todoist now wants to become a suite of apps

At the end of our conversation, I wanted to know what comes next, and Amir described two very different possible futures for Todoist.

In one, he could cut 50 percent of the team, generate much more profit and, as he joked, perhaps buy himself a small yacht, while in the other, the company would take on a much bigger problem, which is ultimately the path they chose:

Or we can just say, okay, let’s freaking solve a much, much bigger problem. So for us, we are going to create a suite of apps.

The first product in that expansion will focus on automations:

Honestly, I think what we have done technologically, but also on product side, will be really, really cool for people to see, because everything is natural language. And you tell what you want to build, and then the system builds it for you. And it’s quite affordable. So automations will be the first leak. The other one is team communication.

When Amir mentioned team communication, I asked whether that meant a return for Twist, Doist’s asynchronous team communication app, which the company developed as an alternative to real-time workplace chat.

Amir replied:

It’s coming back in a very strong form.

That fits with the broader direction Amir described for Todoist, as the company now plans to expand beyond its core product and build a wider suite of apps.

The post Todoist Founder: Developers Should Focus on Solving Customer Problems appeared first on ShiftMag.

Things get more complex at NVIDIA. A job title does not always reveal an engineer’s exact position in the engineering organization, and a level upgrade does not always come with a new title.

NVIDIA uses a deep technical career ladder. This structure lets engineers reach high internal levels without moving into people management. However, NVIDIA does not publish a simple chart mapping each job title to a specific internal level. Job postings show that single titles overlap across multiple levels. An engineer’s scope of responsibility, problem complexity, and cross-team impact reflect their true seniority far better than a title.

To understand NVIDIA’s engineering career ladder in practice, I reviewed its current job postings and compared the responsibilities and levels listed.

How NVIDIA measures seniority

After reviewing NVIDIA job postings, I found that the Senior Software Engineer title can cover several internal levels.

The CUDA UMD-GPU Kernel Scheduling posting lists salary ranges for Levels 3 and 4, while the AI Inference Performance posting lists ranges for Levels 4 and 5.

The CUDA posting also notes multiple openings for candidates with different experience levels. Together, these postings show that the title spans at least Levels 3, 4, and 5. Job titles and internal levels are not interchangeable.

Public databases like Levels.fyi map NVIDIA’s structure into a simple series of individual contributor (IC) levels. However, candidates should view those maps as external estimates rather than official company policy.

Evaluating how an engineer’s scope of responsibility expands with each step provides a much clearer picture.

Moving from task execution to problem ownership

Early in the career ladder, a Software Engineer must master their core tech stack, implement clearly defined features, and deliver clean, maintainable code. As experience grows, responsibilities shift.

Managers evaluate engineers on how independently they understand a problem, analyze system constraints, propose solutions, and ship code to production.

This shift marks a core transition that repeats across higher ladder levels: detailed assignments decrease, and open-ended problems requiring self-directed solutions increase.

What NVIDIA expects from senior engineer – beyond writing good code

At the Senior level, strong technical skills are expected. Engineers must also decide what to build and why.

A Senior Software Engineer makes technical decisions, evaluate system-wide trade-offs, identify future risks, collaborate across teams, and shape system architecture.

NVIDIA’s job postings I lokeed at highlight these expectations. For example, the Senior Software Engineer (AI Inference Performance) role spans AI models, serving software, distributed runtimes, CUDA kernels, and GPU architecture. Senior engineers lead end-to-end performance analyses, resolve bottlenecks, and collaborate across specialized teams.

Similarly, the Senior Software Engineer (CUDA UMD – GPU Kernel Scheduling) role lists core responsibilities such as architecting new features, coordinating multi-team development, and influencing the future CUDA API and programming model.

At this level, the difference between two engineers with the same title shows up in system ownership rather than job titles.

There is another internal detail that makes this distinction particularly interesting. According to a source at NVIDIA I spoke with, engineers generally do not see their colleagues’ IC levels unless they are in management:

No one sees a colleague’s level outside of management. That gets a bit interesting during performance reviews because someone might be excellent for IC3, but I would expect more from them if they were IC4.

That means two engineers can carry the same visible title while being evaluated against different internal expectations. Someone’s performance might look exceptional at one level but simply meet the bar at the next.

Senior Staff engineers work across teams

The next major jump happens when an engineer’s impact crosses team boundaries.

NVIDIA hires roles such as Senior Staff Software Engineer proving that an advanced layer of IC roles exists between Senior and Principal levels.

At this stage, understanding a single service or codebase is not enough. Engineers must manage cross-system dependencies, identify long-term technical risks, and align decisions across teams with competing priorities.

Influence without authority becomes critical at this level. A Senior Staff Engineer does not manage team members or issue top-down commands. Instead, their technical reputation, logic, and judgment drive alignment across engineering teams.

The same NVIDIA source described this progression in much simpler terms:

The IC level determines whether you have the ability, with or without supervision, to handle a subfeature of something bigger, handle a feature, handle a feature that requires coordination with one team, or handle a feature where you coordinate multiple teams as the lead. And then the holy grail: steering the company and the industry, something like IC7 or IC8.

That description captures the logic behind the ladder more clearly than titles alone. Each step expands the scope of ownership, from a contained piece of work to increasingly complex technical efforts that span teams and, at the highest levels, influence the direction of the company and the wider industry.

As engineers move up the technical ladder, their role shifts from individual task execution to creating conditions that help large teams make better technical decisions.

Principal Engineers plan strategy years in advance

At the Principal level, engineers help shape technical direction across the company. They improve existing architectures, plan future systems, identify organization-wide bottlenecks, and lead major changes from strategy and design through implementation and deployment. In infrastructure roles, this can include setting a multi-year vision for automation and configuration management.

They remain close to the technology, though they may spend less time writing production code. Their work can include prototyping, reviewing critical code, solving scaling problems, and weighing architectural trade-offs.

In compute infrastructure, a Principal Engineer may guide the design of a global platform with thousands of nodes and tens of thousands of virtual machines and containers. They may also help build the operational foundation for advanced AI models.

Their impact comes less from the amount of code they write and more from guiding important technical decisions while staying close enough to the technology to make those decisions well.

Distinguished Engineers combine deep expertise with company-wide influence

Above the Principal level, NVIDIA places Distinguished Engineers. At this stage, impact extends across entire business units, platforms, and technology strategies.

For example, the current job posting for this role requires at least 18 years of relevant experience and a proven record of company-wide technical leadership at the Principal or Distinguished level.

Distinguished Engineers improve system reliability, scalability, and performance by setting technical standards and reviewing architecture. They work across platform, hardware, infrastructure, and service teams to turn recurring problems into lasting software improvements.

The role combines deep technical expertise with influence across the organization. They spot important issues early, set technical direction, and help teams act on it.

Career growth means more responsibility and technical impact

Removing job titles reveals a clear career trajectory:

• Early Career: Delivering reliable work and completing assigned tasks.
• Mid-Level: Solving complex problems independently without pre-defined blueprints.

• Senior & Staff: Shaping large systems, coordinating technical strategy across teams, and forecasting long-term trade-offs.
• Principal & Distinguished: Setting technical strategy and guiding large engineering teams without direct managerial authority.

A Principal Engineer is not just a Senior Engineer who knows more programming languages or GPU architecture. A Distinguished Engineer is not just a Principal Engineer with more experience. Each role brings broader responsibilities and a longer-term view, including thinking about the architectures the company will need in three to five years.

NVIDIA engineers can grow their careers without becoming managers

In some organizations, engineers move into management to grow their influence and compensation. Technical career ladders offer another path: Principal and Distinguished Engineers can shape company decisions while continuing to focus on technology.

This technical path fits NVIDIA’s mission. The company builds GPU architectures, CUDA software, AI infrastructure, networking systems, autonomous platforms, and large-scale distributed systems. Solving these engineering challenges requires deep technical ownership that reaches beyond individual teams or products.

At NVIDIA, seniority reflects an engineer’s ability to spot critical engineering challenges, map out technical strategies, and help teams deliver effective solutions.

A top engineer looks beyond today’s questions to help the company solve tomorrow’s biggest technical challenges.

The post At NVIDIA, “Senior Software Engineer” can mean three different levels appeared first on ShiftMag.

Chris has seen his share of enthusiastic implementations that don’t always work out. Funnily enough, it’s not always the clients and partners that get carried away with AI; sometimes it’s Atlassian. Chris noted that the company is just like many others, trying to figure out the impact of AI as they go on. The one thing to point out is the usage of agents:

I think about agents inside Atlassian, and as a team member adopting them, it’s really interesting to see how quickly they get spun up, but then how quickly they only have only one or two uses.

Then you even get to the point where there’s almost like a clean up process. Many agents, duplicated names and activities, and you have to clean those up. You almost need agents to clean up agents.

Chris told us that, in Atlassian, they’re trying to move fast and react to what customers need and want. This often results in making services and face challenges in duplication.

Don’t use AI selfishly

Despite AI agents and tools’ ability to understand almost everything about software, building things with AI is something most of us do by ourselves. We create tools and use AI to write code we’re working on, but rarely we go and seek how we could work with others. You could argue that using AI has become a bit of a selfish thing; we’re all trying to see what we can do on our own. Chris told us that teams that legitimately change the way they work with AI are usually the ones who use it to collaborate with others.

Maybe they’re opening up more of an interface that’s like self-serve to another part of the organization. You know, you think like QA management and test and development, that interfance, how can we make that better?

It’s not about what we’re coding in our IDE, it’s more about how that work is then moving to the next part of the organization. So I really look for where are connections across that whole development lifecycle, not just the coding bit.

AI is magnifying some challenges

An interesting trend in software development today is adding AI for the sake of AI, thus adding more steps in an otherwise very streamlined process. Chris said that we, as humans, need to be very intentional in finding steps where AI can help. We should not just try and apply AI to every single part of every development lifecycle. Doing that just causes something Chris calls “drag”:

Where it causes drag is when poor quality comes down the pipe. We’ve known for years that it costs a lot more to fix poor quality later in the lifecycle than earlier, and I just think we’re magnifying that challenge with AI because it’s so convincing.

Humans can be a little bit lazy sometimes, said Chris, and added that we might let a lot of things pass through unintentionally. Those things get further down the pipeline and AI then causes a real drag because it’s harder to find and fix, and it takes more effort from everyone.

It’s challenging to lead in the AI era

Engineering leaders today, such as higher-level managers and executives, do not have it easy today. Chris echoed this statement, saying that he “feels for engineering leaders at the moment”, and adding that there’s a whole other layer of dealing with both the technical and human challenges of this AI era.

In terms of where leaders go wrong, Chris used a high reliance on data as an example. In his opinion, with more data of all sorts available today, leaders tend to rotate more towards it than towards qualitative metrics:

Managers can defend and they can articulate well with data. There is a risk that you over-rotate purely to data and not those qualitative metrics. So, engineering managers need to also keep qualitative metrics, developer surveys and developer pulses.

That’s how you stay close to what the feeling is on the ground, from both a performance perspective but also a human perspective.

Ask your AI to challenge you

Using AI tools without being skeptical and critical of them might be the single worst trend not just in software development today, but in almost every industry (and in personal lives of many). Chris noted that it’s really dangerous how convincingly wrong AI tools and agents might be, and without knowing how to properly work with them, you might be a victim of their hallucinations without ever noticing it. Chris explained:

A practice I see many teams do is you use AI to challenge you. You have to have a viewpoint and a hypothesis about what you’re doing and why you’re doing it and how you’re approaching it. Then you use AI to challenge that.

Guaranteed, it will come up with things you didn’t think of. It will come up with elements of code or something in the code that you wouldn’t have seen, and it would not have been detected had it not been for AI.

Ask how you could use AI to help other teams

Wrapping up our chat with Chris, we asked him how would he advise those working with AI every day to make the most of their experience, and collaboration with AI tools:

A practical change might be to take some time out of your week, draw a circle and put all the teams that you work with around that circle, and then start to pull your top three or five teams that are critical to that flow of work, and ask yourself, how could we be using AI better in our relationship with those teams?

If you switch your thought process related to AI from thinking how can the technology make your job easier to how it can make working with others easier, it might just be the thing you need to get the edge. Just don’t go using it without questioning the things it says.

The post Chris Davidson, Atlassian: “Ask AI to challenge you” appeared first on ShiftMag.

I soon learned that college did not prepare me for the job, got some kind of idea of the skills and knowledge required to ship production software, and figured out that the more I knew the less stressful the job would be.

So over the course of many years, much against my will, I became an expert. Then AI came into the picture, and nothing fundamentally changed, because the job of a software engineer is to help achieve business goals by leveraging their understanding of the technology, and now there was yet another technology to learn.

Get ahead of security issues

I have decided to focus on making the job less miserable for me. My original approach, once I learned how to use tools such as Claude, was to do the same amount of work as before and then just chill.

Nobody ever complained about my performance, so I figured if I just did the same amount of stuff in less time, I would have time for the things I enjoyed, like drinking coffee with other slackers on the balcony or writing pieces such as this one.

This felt good for a while, but still there were annoying things that needed to be done. One of them is upgrading packages to mitigate security issue. I don’t know about you, but I fall asleep each time I need to read a CVE Record.

So I get AI to write me a short horror story written from the perspective of a sarcastic and frustrated application security engineer whose warnings companies have ignored at their own peril.

The story always has some snippets that help me understand how this exploit might happen on services I am responsible for, and it always ends up with my company going bankrupt, this really makes the lessons stick and motivates me to fix the issues.

Sure I could completely automate the bumping, but where is fun in that, and what do I learn?

The production can be cruel

The other thing I don’t like is the production; the production is cruel. You surround your code with all kinds of tests, you create the production like environment with simulated traffic, you use feature flags, you deploy small increments often, and still production finds the ways to surprise you, and it has a tendency not to do that during business hours.

So you do what you must, add a bunch of observability, create runbooks and alerts. Now, with all that telemetry in place and a coding agent I can finally engage with production properly: not reactively when something goes wrong, but every day I can chat with the production, ask it about its concerns, about what is going on lately, if something feels wrong.

I do this by invoking the skill that gathers all the data and then reports it with a certain tone of voice. And I’ll tell you one thing, gaze long enough into production telemetry and the production gazes back.

Chat instead of jot

One last example, and I envy the juniors for having the ability to do this from the start, is the change in the way how I learn stuff. When I was starting, I used to jot down the things and concepts I did not understand, I would look them up in articles and books, I would go through half-baked internet tutorials, blog posts and man pages, and still come out with questionable understanding.

Now I start a chat session with an AI model, give him a personality, most often it is an old and grumpy but well-mannered distinguished engineer who explains topics with a lot of examples and then asks challenging questions while throwing casual insults, he asks me a challenging question and then roasts my answer before proceeding with the next lesson.

Hope you find some of these ideas useful, because I feel much of the talk has been on how to use AI to speed things up, and too little of the talk is about how to use AI to make software more reliable or the process of building and maintaining it more enjoyable.

The post I Talk to Production Every Morning. It Talks Back. appeared first on ShiftMag.

What’s the earliest reduced version we can put in front of real customers to validate against something we actually want to learn?

At worst, it’s just a date and an announcement with nothing clearly scoped behind it, so that gap needs to be closed before anyone starts building toward it.

That’s where things usually start to go wrong.

Business, product, and engineering often mean different things by “early release”

Once one customer flow is split across several teams, an early release becomes real coordination work.

Someone has to lead it end to end, and that usually falls to no one. The reduced scope drifts and nobody is coordinating the whole release. With each new team that holds a certain piece of the project, coordinating them gets harder. The fix is good management, and it requires someone to actually lead the release.

Part of the trouble is that the same words mean different things to different people who are, of course, the part of the same meeting. When the business asks for an early release, it usually means a reduced scope of something, made to be more of a barebones product than a full-sized one. This is something that’s enough to validate assumptions and deliver value while the rest is still being built. 

To most engineers, “done” means complete, so “early release” gets heard as “the whole thing, but fast.” Put an MVP, the smallest version that tests your least-sure assumption in the mix alongside full GA, the finished product for everyone, and one phrase is now pointing at three different targets.

Nobody draws the line between them, so scope drifts toward GA by default: a team builds one target while the plan assumed another, and that drift is what sinks these releases. 

How one customer journey becomes nobody’s problem 

Multi-team releases break along a predictable line, and it was mapped a long time ago. In 1968, Melvin Conway published a paper called How Do Committees Invent? with an observation that later became known as Conway’s Law. The law states that organizations that design systems are constrained to produce designs that copy their own communication structure.  

Applied to a release, it predicts the break before anyone writes code: split one customer journey across five teams that mostly talk within their own walls, and the software inherits the same boundaries the organization has. The release breaks at the boundary between two teams, where the calls that need the whole picture don’t belong to anyone.

Put that prediction next to a real feature. I kept it deliberately small here, just to picture the shape; the real ones are bigger and messier.  

A customer sends across dozens of destinations, each with its own registration rules. Provisioning sets the customer up to send, sender registration gets each sender approved per destination. The hard cases raise questions neither team can answer alone: when a sender is fine in one country and refused in the next, does provisioning collect a fallback, does registration substitute one, or does the campaign ship partial?  

No owners means no decisions

Since there’s no owner, there is no decision.

Instead, each team codes to its own assumption and passes its own tests, and the assumptions only collide in some cases that cross between teams. The first customer with a mixed set of destinations goes live approved in some and silently refused in others. This is a fix that looks obvious at this size, but at real scale it’s dozens of these calls across five teams, and the someone who should make them is still no one. 

It starts when the date is set before the scope or target customer is clear. Then teams start building without knowing where to stop. One team works on an edge case for full GA, while another builds for customers who aren’t onboarded yet.

“Early release” becomes “the whole thing” slowly, one decision at a time. It stays invisible because each team is doing good work on its own. The drift only exists at the level nobody is watching, which is the whole end-to-end journey, or the big picture. A release that goes out with no defined scope, no chosen customer, and no one leading it pushes the risk onto whoever is downstream.

Cutting scope on purpose is different. You choose what to leave out and handle the rest manually, so the customer still gets value without needing the unfinished parts.

The case against shared ownership

Shared ownership across those teams is where the whole falls through. Everybody owns a piece, which really means no one owns the whole.

When multiple teams each hold a piece of a customer journey, someone has to lead the end-to-end release. This should ideally be one or two people who know the flow in real detail: which service does what, where the boundaries between teams are, what a real customer actually experiences from first call to final result.  

A steering committee can set priorities and handle escalations, but it can’t lead the release day to day. A manager who only sees team status reports can’t do that either. The details that decide the release depend on how the pieces fit together, and neither of them is close enough to manage that.

In practice, that leadership is coordination work, not a box on the chart. The owner learns the whole flow and keeps it in their head. They draw the reduced-scope line and publish it, so every team is working from the same picture of what “first version” means.  

They watch for symptoms of a team drifting, which can include hardening a GA edge case or generalizing too early in the process. They decide where “reduced scope” ends and “next phase” begins and call the release. Most orgs already have someone near this role. Often, that’s a product manager or a TPM. 

Try making one person responsible for the release end to end

Define the reduced scope and target customer before you set the date. Write it down. Give it one owner who understands the whole flow. And when the first customer hits a gap, fix it in the shared path so the next customer doesn’t run into it too.

The reason it doesn’t happen is that leading five teams end to end is hard, unglamorous management work, and in most organizations it isn’t any single manager’s job. Each manager runs one team and looks after that team’s part of the release. The coordination across teams is no one’s job.

Some organizations avoid this because leading the whole is already built into someone’s job, or the structure is flat enough that the whole stays visible. The teams aren’t weak and nobody is being lazy; the job that needs doing simply isn’t anyone’s job. The fix is to make one person responsible for the release end to end, and to give them the room to actually lead it. 

Sometimes, though, one person genuinely can’t lead the whole thing, however skilled they are. In a matrix organization, an agency where the client signs off every step, or a regulated field where the approval path is set from outside, the flow crosses boundaries a single coordinator can’t reach across. 

If your structure genuinely prevents one person from leading end to end, no release process will fix it, because the real constraint is how the organization is wired. The release is only where that constraint first becomes visible. 

The post Why Early Releases Often Fail in Multi-Team Organizations appeared first on ShiftMag.

This article looks at those five individual-contributor levels. It uses Netflix job descriptions, the company’s culture and compensation policies, and reported pay data from Levels.fyi to show the kind of work associated with each level. Those hiring examples should not be treated as Netflix’s internal promotion criteria.

Engineering levels and compensation

Levels.fyi labels L3 through L7 as Engineer, Engineer II, Senior Engineer, Staff Engineer, and Principal Engineer. Netflix’s recruiting titles vary, sometimes using only a number. The examples below describe particular advertised roles; they are not a uniform specification for every engineer at that level.

Based on the last information checked on October 9, 2026, the individual US level pages on Levels.fyi displayed average annual total compensation of approximately $214K at L3, $321K at L4, $509K at L5, $704K at L6, and $1.14M at L7.

Separately, the company overview reported a median of $430K across US software engineer submissions. What should be noted is these are reported compensation figures, not Netflix’s official pay bands or a representative census of its employees.

Netflix’s engineering listings generally describe annual compensation without a separate bonus. Its Work Life Philosophy says eligible employees choose each year how much compensation to receive as salary or stock options. Under its Personal Top of Market approach, the company assesses an individual’s market value rather than defining the market through fixed bands and grades. The advertised ranges still depend on the role and can vary by location.

For example, the L4 DevOps listing specifies $250K to $413K, and the L6 Developer Productivity listing specifies $600K to $1.066M. The published L7 Growth Engineering description gives a range of $240K to $1.2M. Such overlapping ranges are a reason to check the specific role and offer: they cannot be used as a standard salary schedule for L4, L6, or L7.

L3 and L4 Experience requirements and project ownership

Netflix has a dedicated US new-graduate program that includes technical roles, onboarding, and training. Its careers page says new-graduate openings typically appear in September or October, depending on hiring needs. Levels.fyi identifies L3 as entry level, but that does not mean every Software Engineer 3 vacancy is suitable for someone straight out of university.

The Software Engineer 3 role at Ink, for example, asks for three to five years of software engineering experience, preferably in animation production. It involves working with senior colleagues on tools and platforms for AI-assisted production. That is a specific hiring requirement for a specialist role, not a universal experience threshold for L3.

Speaking about L4, the DevOps and CI/CD posting asks for at least four years of professional experience or equivalent practical experience. It describes work on defined parts of larger platform initiatives, with guidance from senior engineers and a manager. An Android L4 posting expects ownership from project scoping through delivery, while still stressing the importance of team support.

Together, those examples suggest increasing independence at L4, with the amount of guidance depending on the role. Project ownership and support from more experienced colleagues appear together in these descriptions.

L5 Senior engineering and system ownership

Globally popular streaming platform has advertised Senior Software Engineer positions at L5. A useful example is its Analysis role in the Experimentation Platform team, which combines responsibility for production systems with input into architecture and the team’s roadmap.

The archived Analysis listing describes workflows processing billions of events each day. Its responsibilities include developing experiment-analysis tooling, responding to incidents, improving reliability, and agreeing data and API contracts with other teams. The engineer is also expected to understand how product teams use experiments. These details show that the job extends beyond implementing a predefined feature.

In this role, senior responsibility includes accounting for the operational consequences of a design and the needs of the teams using it. Netflix’s emphasis on independent judgment provides context for that work, but the Culture Memo applies across the company rather than defining separate expectations for L5.

L6 Staff engineering across teams

At L6, Netflix explicitly advertises Staff Software Engineer roles. The AIMS Platform Systems examples make the wider scope concrete: the engineer works on infrastructure used by other engineering and research teams.

The Developer Productivity role covers development environments, build and test systems, CI/CD, and tools used to move research into production experiments. It also calls for measuring build times and iteration speed, supporting adoption, and setting shared technical standards.

The Staff ML role focuses on observability, infrastructure cost, and reliability, alongside contributing to architecture and migration planning with other teams.

These examples suggest that Staff responsibility includes getting shared systems adopted and making them useful across AIMS. The developer productivity posting makes that test concrete: the tools need to improve how other teams build and run experiments. The work combines hands-on engineering with coordination across teams.

L7 Technical direction across an organization

Levels.fyi calls Netflix L7 Principal Engineer, while the Growth Engineering job description uses Software Engineer L7. Its responsibilities are defined around the Growth Engineering organization, so the scope is more informative than the title alone.

The published role reports to a Senior Director and asks for at least 15 years of professional software development experience as an individual contributor. It includes setting architectural direction across Growth Engineering, connecting business goals to technology investments, and working with senior leaders and other L7 engineers. The experience requirement belongs to this vacancy, not to a published rule for every L7 role.

The remit is broad technical leadership within Growth Engineering. It combines architectural decisions with continued expertise in production systems and mentoring other engineers, including helping managers support their teams’ development.

Comparing level names across companies

Level numbers are company-specific. Levels.fyi places Google’s Senior Software Engineer title at L5 and Staff at L6, while Amazon uses L6 for Senior SDE and L7 for Principal SDE. Meta uses E-level labels in the same database. These naming conventions can help a candidate navigate vacancies, but they do not establish equivalent responsibilities or hiring standards.

The distinction matters most above Senior. For example, Google’s L7 is Senior Staff and its Principal title appears at L8, whereas Levels.fyi labels Netflix L7 Principal. A shared title or similar number does not show where an engineer would be placed after changing companies. The role’s technical remit and the employer’s assessment remain necessary context.

How culture affects engineering work

Netflix’s Culture Memo highlights judgment, candid feedback, and giving employees enough context to make decisions. It also expects managers to stay involved and coach their teams. Autonomy should therefore not be mistaken for an absence of support, nor treated as a benefit reserved for senior levels.

The keeper test is part of that performance culture. Managers are asked whether they would try to retain someone or choose to hire them again. If the answer is no, Netflix says it favors ending the employment relationship promptly. The memo also says to assess a person’s overall record, discuss performance regularly, and support employees through short-term setbacks.

For a candidate, the practical next step is to ask how the hiring team defines scope at the advertised level, what support is available, and how progression is assessed. Public postings can help frame that conversation. They cannot supply a guaranteed promotion timetable or the terms of an individual offer.

The post Netflix Engineering Levels Explained: L3 to L7 Career Ladder, Scope, and Pay appeared first on ShiftMag.

In my team in Infobip, we handle message sending and serve as an endpoint for OTT providers such as WhatsApp, Viber, and Apple. Because of that, message sending is our core responsibility, and we need to be aware of any bottlenecks.

Initially, our service was measured at around 1,000 requests per second, but we didn’t know why.

From time to time, this raised concerns in the team: we should understand better where the bottleneck is and what causes it. Once we know that, we can make informed decisions to address client needs – for example, answering whether we can handle the traffic they expect to send. That was partly what led to this investigation.

How to run the tests

Application

Initially, with no experience in performance testing, I viewed the service as a set of methods – essentially a stack of calls. I assumed I could analyze it with a profiler. I also knew I shouldn’t run it on my computer, because other applications would affect the service’s performance.

That turned out to be a bad assumption. I’d suggest starting by measuring the application as a whole to get a broader view of its performance.

Integration

Since our application forwards messages further down the infrastructure and ultimately to the provider, it had to be adapted so that it would not send real traffic, while preserving as much of the original logic as possible.

Initial tests of this version showed that the service could handle a request in 0 ms, which seemed unrealistic. I realized that I needed to introduce a delay based on production, where the rounded average request processing time is around 50 ms.

Initial measurements

Having established that the service handles one request in 50 ms, we can estimate that it can process about 20 requests per second. Since the service is Jetty-based, it has roughly 200 threads available by default, which gives a theoretical peak of around 4,000 requests per second. With that in place, the service can now be tested.

Testing environment

• DO NOT run such tests on a local machine.
• Dedicated machine on our testing env was created for it.
• K6 was chosen as test suite.
• Rundeck to automate and parameterize remotely run tests.

Endpoints to be tested against:

• /mapi-mock/1/dummy – performs Thread.sleep(50) – so it is blocking code run on Jetty thread pool
• /mapi-mock/1/dummyReactor – also performs Thread.sleep(50)but publishing result on boundedElastic thread pool – it is still blocking code, but in that case one thread of reactor’s pool will be used
• /mapi-mock/1/messages – non-blocking code which sends request through WebClient to real service
• /mapi-mock/1/messagesReactor – non-blocking code which sends request through WebClient but publishing it on boundedElastic

JVM tuning

Make sure the correct memory parameter is used. In my case, HEAP_PERCENT was set to 60, meaning the JVM heap used 60% of the machine’s RAM. That allowed me to focus on machine RAM only, and the heap size would be adjusted automatically. This is not ideal, but it was good enough for our tests.

Metrics gathering

Initially, I ran the tests for 40 seconds, but I forgot that the metrics were collected every 30 seconds. That explains why I didn’t see any issues in the service metrics during the first run. When I extended the test duration to 180 seconds (a value I borrowed from similar tests) everything became visible.

You can also observe the machine directly with tools like htop or top, or any other tool that shows current CPU and memory usage. In the end, you can use whichever tool helps you complete the task.

One obvious but important rule: do not change two variables at the same time unless you know what you are doing. If you do, you won’t know which parameter had the effect. Change one thing, test it, then change the next thing, and repeat until you understand what is happening.know what is going on.

Gathering measurements

Initially, I decided to send up to 5k requests toward the service – it is below theoretical machine efficiency, but higher than what was experienced in the past. Lower CPU and RAM measurements are somewhat weird: 4 GB of RAM is not recommended as a base machine setup, because there is less than 2 GB of RAM left for the operating system, where not only our service lives, but also supporting applications. But both measurements for 2 CPU showed that it is way too low for handling any traffic efficiently and it showed quite high CPU and RAM usage, meaning that it could be improved. I finished the tests with 5k requests/s, when performance was quite near predicted performance.

The theoretical peak was reached, so it was time to check whether it could be improved. I tried switching from Jetty pool to Reactor pool, but it turned out (my bad for not remembering Project Reactor documentation by heart) that to reach the initial performance I had to configure the machine with 20 CPU, because Reactor constructs the pool to have 10 * CPU threads. So when I matched theoretical performance again, but now with the Reactor pool, I thought about what could be done next to improve. The last thing to check was how non-blocking code would perform, so I started using the /mapi-mock/1/messages* endpoints for testing. As one would predict, it was hit. The application peaked at 7,000 requests/s, but with quite large demands for hardware. So the last thing to check was to try to find reasonable settings to have big enough performance with not much resources assigned.

Now we are focusing on the non-blocking endpoint, so it more reflects the real-case scenario, and on limiting resources. The first thing that is apparent is that when it comes to non-blocking code, there is no difference between Jetty thread pool and boundedElastic thread pool (the latter results in some queuing because the rule of 10 * CPU threads is always true). We can see performance downgrade with CPUs removal, but not apparently linear. I had to say “stop” somewhere, and I decided to stop with 8 CPU and 8 GB of RAM, for which we received ~3.5k requests/s. Good enough with not much resources assigned. As we can scale horizontally, adding more machines will use more resources, but give more resiliency back.

 The thing with VT…

As you can see, I focused on measuring reactive vs non-reactive code.

It was our go-to architecture at that time, while VT was being adopted slowly (we were afraid of thread pinning, so we waited until it was addressed in JDK 24 to start adoption). After I published it internally, I was asked specifically about VT – how the app would perform when configured to use this not-so-new-but-fixed feature.

It turned out that VT gave the app a boost. With optimal configuration (8 CPU and 8 GB of RAM) the app was able to handle 10k requests/s. Why is that? Any code, regardless of whether it is blocking or not, will be run on a virtual thread, which ideally should not block a system thread, leading to the case where system threads are used only for real logic and not for waiting. VT can improve applications where there is a lot of blocking code. In the rest of the cases, it is “just” syntactic freedom, no more looking at which operator do I need?

The outcome

When I performed the tests, I observed that just switching to boundedElastic (i.e. “use Reactor and you will get higher TPS”) didn’t increase throughput – it actually reduced it. Not remembering the Project Reactor documentation by heart, I was surprised, but after some research I found that boundedElastic provides 10 * CPU threads, which gave us a total of 60 threads to service all requests previously handled by 200 Jetty threads.

When the CPU count was adjusted to match 200 Jetty threads (20 CPUs) we reached the expected throughput. But it was wasteful in terms of resources. In the Reactor world, people often say that blocking is bad, and my service was actually doing exactly that, on purpose. I thought it was time to switch to a different endpoint.

So I was able to nearly reach the estimated performance of the service, which was good. That allowed me to move on and investigate other ways of improving it, such as using non-blocking code or VT.

What I’ve learned

It is always about finding the bottleneck and finding out if we can improve it, for example by adding resources or redesigning it. If it is not to be improved, then this is the peak capacity of the service.

Sometimes questions from clients can actually help. It is just like with a new person joining the team: they will ask obvious questions about all-known solutions, but if you want to respond accurately, you have to dig deeper, and sometimes you find out that this simple question leads to an investigation and even to questioning whether it is correct at all.

Also, be aware of architecture. In that scenario, we tested a service which accepts requests, so it is one of the first applications, or APIs, which the client experiences. Being first means that there is the rest of the system, which can in turn have throttling applied. What does it mean? Even if we accept 10k requests/s, sometimes we cannot send them immediately, because an independent system cannot handle, for example, more than 2k requests/s. So we can be fast for clients, in the sense that we accept their traffic as fast as they want, but we should always remember that this is not the whole picture, and the client should be informed about it.

In my test there was no DB, just API-to-API calls. Having any limited, and even worse blocking, resource like a DB in the way of processing a request will definitely impact its performance.

The post How 5,000 RPS Exposed the Real Bottleneck in Our Jetty Messaging Service appeared first on ShiftMag.