I’ve known Mik Kersten for a while, his 2018 book Project to Product and the whole concept of moving from individual managed projects to continuously developed products has been part of my core advice to enterprises for many years. I was a guest on his podcast, and we had so much fun finding out how aligned our thinking was that I came back for a second chat. More recently, Mik is part of Gene Kim’s IT Revolution forum and we meet each year to write papers on interesting topics. In 2025 Mik and I were on the same team and he mentioned he was writing a new book, Output to Outcome: An Operating Model for AI. Of course I offered to review it as he wrote it, and I was able to read a largely complete copy in December 2025. This turned out to be very frustrating, because my reaction was that everyone needed to read this immediately, as we were already struggling with the problems that it breaks down and shows how to solve. However it’s a book, and they take a long time to deliver even when the words are basically done, so you’ve all had to wait about 8 months to get it. My contribution is this ‘praise quote’:

“We are in a chaotic environment driven by a rate of change that current systems and businesses cannot absorb. The climate crisis, pandemics, and wars disrupt business plans and supply chains, and the pace of AI- driven development means that tools are obsolete before they stabilize. We have to reorganize to focus on outcomes, and rebuild everything to deliver them using new tools that are building themselves. This is the definitive guidebook for that transition.” — Adrian Cockcroft, Tech Advisor, Formerly of AWS and Netflix

The perspective that I got in advance from Mik has informed a lot of my work this year, including a 2026 IT Revolution paper on Agentic Continuous Decision Intelligence that will release in September or so, and my recent blog post Entering No Mans Land.

Here’s an excerpt of what I wrote for a draft of that paper:

In the last few years, new challenges have arrived. The pace of technological change has accelerated to the point that it’s no longer possible for enterprises to adopt AI technologies and tools before they become obsolete, leading to “enterprise indigestion.” Supply chain shocks from pandemics, wars, and rapid policy changes, such as tariffs, are creating chaotic conditions. Hypothesis testing is insufficient, and a different operating model is needed to act-sense-respond with “wartime” rules of engagement and guardrails constraining allowable automated actions. The complicated and complex modes can be optimized because the rate of change is manageable, but in the chaotic mode, with unpredictable change, the main concern is survival, and managing contingency reserve buffers that would be optimized away in less chaotic times becomes a priority. Managing enough capacity to survive shocks while responding quickly is often the winning strategy.

This new challenge for businesses is that they need to operate in different modes as their environments and markets change between complicated, complex, and chaotic behaviors and responses. Maintaining an appropriate balance between efficiency, resilience, and agility for each part of the business is the winning strategy. These modes are described by the Cynefin framework, developed by David Snowden, and Mik Kersten’s Output to Outcome book explains how to apply them using AI. Continuous Decision Intelligence is needed to make sense of what to do in all cases, but the speed of response needed to survive today is driving much more integration and automated decision-making via agentic CDI.

You can get your copy (releasing July 14th 2026) from the web site: https://outputtooutcome.org/

[Updated for Fable-5] Every few weeks a new frontier model tops the leaderboards, and the implicit advice is “upgrade.” Sites like llm-stats.com rank models well across many benchmarks — but they answer a question most engineering teams aren’t actually asking. They hold the stack constant: one prompt, one harness, a fixed benchmark. They don’t tell you whether the newest model is worth 4× the cost in Rust, how reliably each model gets a Go MCP server completely right, or how long any of it takes.

Those are the variables that decide a real project. So I built retort to measure them properly — with statistical Design of Experiments, the same technique you’d use to tune a manufacturing process. Vary the factors you care about (here: programming language × model version× tooling — and, newly, the coding agent itself), run a factorial grid on a real task, score every cell, and let the analysis tell you which factors actually matter. Nine experiments, 258 scored runs, two tasks, eight languages, four Claude models (plus a fast-mode variant and a next-tier model, Fable 5). Here’s what came out.

The metric that matters: how often is it completely right?

Most code scores grade on a curve — 80% test coverage, a clean linter run, a plausible-looking diff. But for code you intend to ship, “mostly implemented” is a failure, not a B+. So retort’s headline metric is pass-proportion: run a stack N times and count the fraction whose output fully implements the spec — every requirement on a fixed checklist, with tests that actually execute, verified by an independent evaluator.

Read it as the probability that a single run comes out completely correct. 3 of 3 → 1.00, 2 of 3 → 0.66, 1 of 3 → 0.33. A run that misses even one requirement counts as a fail, not a 0.9. That’s a deliberately harsh bar, and it’s the one that matters when you’re deciding whether to trust an agent with a feature.

The headline: newer is more reliable — and you pay for it

Aggregated per model per task:

Three things jump out:

1. Newer genuinely is more reliable — and the gap is enormous on hard tasks. Opus-4.8 produced a completely-correct result 100% of the time, on both tasks. The older, cheaper models (4.6 and Sonnet) got the hard task fully right only about half the time. On a difficult task, the cheap model is a coin-flip — it’ll look fine in a demo and bite you in review.
2. You pay through the nose for that reliability. Opus-4.8 was ~3× slower and ~4× more expensive than 4.6/Sonnet on the hard task. Reliability isn’t free; it’s a line item, and it grows fast across model generations.
3. Opus-4.7 is the value sweet spot, and on easy work the newest model is pure overhead. On the REST API, 4.7 and 4.8 are tied at 100% — so paying for 4.8 there buys you nothing but a slower, costlier run. On the hard task, 4.7 reaches 85% for noticeably less money than 4.8’s 100%.
4. Fast mode is the same reliability at the highest price — and so is a tier above 4.8. Opus-4.8 fast matches 4.8’s 1.00/1.00 and trims wall-clock, but its 2× per-token rate makes it one of the costliest rows ($8.72/run on the hard task) — it buys latency, not value. And Claude Fable 5, a model a whole tier above 4.8 at that same 2× rate, also lands at 1.00/1.00 — because where 4.8 is already perfect there’s simply no reliability left to buy. It ends up the priciest and slowest option, with nothing to show for it (more below).

The controlled view: same cells, two models

Those aggregates mix experiments, so the firm conclusions come from the within-experiment comparisons — identical language/tooling cells, run with two models, three replicates each.

Hard task (Brazil, 6 languages × {4.7, 4.8}): opus-4.8 passed 18/18 cells; opus-4.7 passed 15/18 — it dropped to 2-of-3 on Go and 1-of-3 on Rust. So the newer model didn’t just win on average; it closed specific, repeatable failure modes. But it took ~1040 s vs ~710 s per run and cost ~$5.6 vs ~$4.6.

Easy task (REST API, 6 languages × {4.7, 4.8}): both models passed essentially everything (1.00). The only measurable difference was that 4.8 was ~50% slower (243 s vs 165 s) and a bit pricier. Identical result, higher bill.

The pattern is consistent: each model generation buys you reliability on hard problems, and charges you time and money for it everywhere. If your work is routine, the premium is wasted; if it’s genuinely hard, it may be the difference between “ship it” and “rewrite it.”

Fast mode: speed you pay double for

Opus-4.8 ships a fast mode (the /fast toggle — same model weights, faster token output), and it's billed at 2× the standard per-token rate: $10/$50 vs $5/$25 per million input/output tokens, per the announcement. So the real question isn't "is it faster?" — it's "is the speed worth double the price?" I re-ran the same languages on both tasks with fast mode on. Reliability was untouched — every cell held pass-proportion 1.00, identical to regular 4.8 — but the economics are not what a casual reading suggests:

Two things stand out. On the easy task, fast mode genuinely shaves wall-clock — roughly 20–40% — but at the 2× rate it still costs more in dollars (python: 26% faster, but 48% pricier). On the hard task it’s the worst of both worlds: you pay about double and you don’t even get the speed — Go and Python fast runs were slower than regular, because a reasoning-bound task is gated by the model thinking, not by how fast it emits tokens.

So fast mode buys latency, not savings, and only on routine work. The honest rule is: turn it on when a human is waiting on a quick task and you’ll happily pay double to wait less; leave it off for anything hard, where it’s pure overhead. (It’s also a clean illustration of why you separate “speed” from “capability” as factors — averaging them together would have hidden that the premium pays off in exactly one quadrant and nowhere else.)

A confession is owed here, because it’s the whole point of the project: my first pass at this section concluded fast mode was cheaper — a “free lunch.” It wasn’t; I’d trusted the cost number the CLI reported, which (I later confirmed by probe) prices fast-mode tokens at the standard rate and silently omits the 2× premium. The conclusion flipped completely once the cost was corrected. Measure, then check that what you measured is real — including when the measurement flatters the answer you were hoping for.

A tier above 4.8: does paying even more buy reliability?

Fast mode raised an obvious follow-up. It charges the $10/$50 rate — double Opus 4.8 — for the same model, faster. But what about a genuinely higher model? Claude Fable 5 sits a tier above Opus 4.8 and is priced at that same $10/$50 rate (the CLI prices it natively, so no correction needed). If 4.8 is the reliability frontier, does stepping up a tier — at the same premium fast mode charges — actually buy you anything? I ran Fable 5 on the identical grid: both tasks, the four shared languages, three replicates each.

The answer is a clean no:

Fable 5 passed 12 of 12 cells on each task — a perfect 1.00, exactly matching Opus 4.8. That’s the catch: where 4.8 already gets it completely right every time, there is no reliability headroom for a better model to capture. The ceiling is the ceiling. So the higher tier delivers an identical pass-proportion while costing roughly double the dollars, and — on the hard task — running slower than regular 4.8 (≈1039 s vs ≈947 s), making it the priciest and slowest option I measured. Fast mode at least buys latency on easy work; a tier-up here buys nothing measurable at all.

This isn’t a knock on Fable 5 — it’s a statement about the task. Both of these problems are inside Opus 4.8’s reliable envelope, and you can’t out-reliable 1.00. The place a higher tier would earn its premium is a task hard enough that 4.8 itself drops below 1.00 — and the honest read of this data is that neither task here is that hard. Which is exactly the decision the per-task framing is built to expose: the right model isn’t the highest one, it’s the cheapest one that clears your task’s reliability bar. For these two tasks, that’s plain Opus 4.8 (or, on the easy one, something cheaper still).

Two more languages: Erlang and Elixir

I added the two big BEAM languages to the REST-API matrix (Erlang, Elixir, on Opus-4.7 and 4.8). They slot straight in at the top: 1.00 on pass-proportion, test coverage, and code quality — every cell, both models. They’re the most uniformly clean stacks I measured on the easy task, and Elixir on 4.8 was the cheapest-and-fastest of the pair ($0.85, 207 s). Two languages that weren’t in the original grid, measured and ranked in an afternoon — which is the whole point of treating language as just another factor you can add.

It’s not just the model — it’s the language and the task

Average a model over everything and you hide the most useful signal. Break reliability down by language and task and it swings wildly:

Look at the spread:

• Rust flips completely: 0.96 on the easy task, 0.50 on the hard one. The agents write clean, well-typed Rust for a CRUD API, but the harder knowledge-graph task trips them up half the time.
• Java runs the other way: 0.80 on the hard task but only 0.52 on the easy one — counter-intuitive until you see how it fails (over-engineered scaffolding that misses a small requirement on the simple task).
• TypeScript and Python are the all-rounders: strong on both (0.92–1.00 and 0.86–0.90).
• And code quality barely moves across tasks within a language — Go and Java sit at 1.00 regardless of difficulty — even as their reliability swings. Clean code and complete code are not the same thing, and they’re driven by different factors (see below).

There is no single “best language.” “Use Rust, it’s rigorous” is good advice for a service and bad advice for the hard task; the only way to know is to run your task.

What the variance actually comes from (ANOVA)

Because this is a designed experiment, I can decompose where each metric’s variation comes from — language vs. model vs. tooling — with a type-II ANOVA (cost and time log-transformed, since they scale multiplicatively). The separation of concerns is almost suspiciously clean:

Stated plainly: language governs how clean the code is, the task governs how much it costs, and the model governs how reliably it’s correct. Three different factors, three different knobs. The practical consequence is sharp — reaching for a newer model to get “better code” is mostly wasted money. It doesn’t write cleaner code (the language already decided that); it writes more reliable code, and charges you time and dollars for it. You can only see that by varying the whole stack and doing the statistics, instead of reading one number off a leaderboard.

(The beads issue-tracker tooling I tested showed up in exactly one place — extra cost and time, with no quality or reliability payoff — which is why it was dropped from the later experiments. Worth remembering the next time someone suggests bolting more scaffolding onto an agent "to be safe.")

Which stacks are actually production-ready?

ANOVA tells you what moves each metric; the stack maturity view tells you which specific stacks you’d actually trust. retort maturityscores each language × model × tooling × task combination into a lifecycle phase from its reliability, reproducibility, and completion rate. Of 103 stacks in the combined data, 67 are "production" (ship it), 18 "trial", 12 "screening", and 6 "candidate" (avoid). Every stack I added in this round — fast mode on both tasks, Erlang and Elixir — landed in production.

The interesting part is the bottom of the list, because it’s not random: the entire immature tail is the hard task, and overwhelmingly the hard task with beads tooling. On Brazil, plain stacks average 0.88 maturity (18 of them production-ready); the same stacks with beadsaverage 0.54, and only two stay production-ready. Even Opus-4.8 — the model that aces Brazil bare — drops to "candidate" once you add the tooling. So beads isn't merely wasted money on a hard task; it actively destabilizes the run. That's a much stronger statement than the ANOVA's "+10% cost," and it's the kind of thing you only see when you score whole stacks instead of averaging a metric.

A word on failures — and trusting your own harness

A strict bar (“a run only passes if its tests actually execute and it implements the whole spec”) is the only honest way to score this — but it cuts both ways, because sometimes a failure is your measurement, not the model. Adding Erlang and Elixir and fast mode surfaced three such cases worth being candid about:

• Elixir looked like a total failure — 0% on every run. It wasn’t: the models wrote valid Elixir (a sample project runs 17 tests, 0 failures). My scorer invoked the test suite with a mix sub-command syntax that a recent Elixir release had removed, so the tests never ran and the gate failed them all. One-line fix; all six runs then scored a clean 1.00.
• The newest runs reported $0.00. A refactor of the agent-runner had quietly stopped parsing the cost telemetry for runs that didn't pin an explicit agent name — the model ran and billed, but the number was dropped on the floor. Fixed and regression-tested.
• The re-scorer silently did nothing on the two newest experiments because a database query compared against SQL NULL (which is never equal to anything) for designs that had no tooling factor.

And the discipline kept paying off. When I went back to re-run a batch of old beads-tooling false-failures under the fixed harness, the rerun job itself broke — it never launched the model, and stamped every cell as a failure in ~1–4 seconds for $0. Worse, it overwrote the good runs it was meant to repair: one experiment dropped from 36 completed runs to 18. Two things saved the data. First, the runner snapshots each DB before a rerun, so I could restore from the .pre-rerun.bak files and lose nothing. Second — and this is the part that generalizes — the failures were obviously the harness's, not the model's, on sight: a real model failure on the hard task burns minutes of wall-clock and real dollars before it fails the gate, while these died instantly for nothing. That single tell — time and cost spent — is the cheapest harness-vs-model lie detector I have, and it caught a corruption that would otherwise have silently halved an experiment.

None of these were model failures; they were all mine, and all are now fixed (or, in the rerun’s case, rolled back) — the genuine signal restored intact. The genuine failures, once the harness was honest, fell exactly where the rest of the data predicts: the hard task, with the cheaper models or the extra tooling. The meta-lesson is the same discipline the whole project is built on — measure, then check that what you measured is real before you draw a conclusion from it.

The factor I haven’t varied yet: the prompt

There’s a large lever I deliberately held constant: the prompt. Every run got the same terse “implement TASK.md” instruction. But how you ask plausibly moves reliability as much as which model you pick — and it’s nearly free to change. Does a test-first prompt, or one with a worked example, or a “list the requirements before you code” preamble, lift a cheap model’s hard-task pass rate from 0.5 toward the expensive model’s 1.0? If so, a better prompt could be worth more than a model upgrade, at a fraction of the cost. retort treats prompt as just another factor, so the next study writes itself: prompt × model on a hard task. That's the experiment I'd run next, and it's the one with the most direct impact on a real engineering budget.

Beyond the model: varying the agent itself

There’s a second constant I’ve started to relax. Every run above used one agent — Claude Code (claude -p) — and varied the modelinside it. But the agent is its own variable: the harness around the model (its tools, its file-editing loop, its planning, its prompt scaffolding) plausibly moves results as much as the weights do. So the obvious question is whether a different agent — same class of task, different vendor — lands in a different place.

retort now treats agent as a first-class factor. I added a Google Gemini adapter (it shells out to the gemini CLI exactly the way the Claude path shells out to claude -p), so you can put agent: [claude-code, gemini] straight into the factor grid and let the same ANOVA decompose how much of quality, reliability, and cost is the agent versus the language versus the task. Building it was a good demonstration of why you run things rather than trust them: the integration looked done in a unit test, but the first live run caught two things the test couldn't — the CLI was reporting tokens under different field names than I'd assumed (so cost would've been silently wrong), and it quietly refuses to act autonomously in an "untrusted" folder until you pass an explicit flag. Both fixed against the real CLI's behavior.

What I don’t have yet is the cross-agent data: the free-tier Gemini quota hit a capacity wall before a single cell finished, so the comparison itself is still pending a quota reset or a paid key. But the scaffold is wired and validated, and the more interesting point stands — once you can vary the agent, “which coding agent” becomes a measurable question on your task, not a Twitter argument.

So how should you actually choose?

The data suggests a simple decision procedure:

1. Classify the task. Is it routine (CRUD, glue, well-trodden patterns) or genuinely hard (novel domain, many interacting requirements)?
2. Easy task → optimize for cost/speed. Almost every model fully implements it, so take the cheapest fast one — here that’s Opus-4.7, with 4.6 and Sonnet close behind. Paying for the newest model is wasted.
3. Hard task → pay for reliability if you need it right. Opus-4.8 was the only model that got the hard task completely right every time. If a half-chance of a subtly-incomplete implementation is unacceptable, that premium is the cost of trust.
4. Pick the language for quality, not the model. If you have latitude, Go/Java/Rust score top marks for code quality on these tasks — but check reliability for your specific task, because that’s where languages diverge.
5. Don’t add tooling for its own sake. It cost time and money here and changed nothing else.

How it’s measured

Each run gets its own isolated workspace; the agent implements the task, and the code is then built and tested in place. The spec check is the strict part: an independent evaluator verifies the code against a fixed requirement checklist for the task, and a run only counts as a pass if it implements all of it and its tests actually run. To keep that grading reproducible, the checklist is pinned (so the denominator is constant across runs), a strong model does the judging, and a borderline result gets a second opinion before it’s recorded. Every number above is that gate applied across all 258 scored runs — not a hand-picked sample. Per-experiment tables and the combined dataset are in the README and master.csv.

Try it on your own stack

The point of retort isn’t my numbers — it’s that you can get yours, on your task or codebase, in an afternoon:

$ claude
> clone and install https://github.com/adrianco/retort here
> then compare opus 4.6/4.7/4.8 across Go, Rust and Python on this task

Claude designs the experiment, installs the toolchains, runs the cells (resuming across usage-limit windows), and scores each one for whether it actually implements the spec. Watch it live with retort monitor; roll it all up with retort aggregate and run the ANOVA with retort report effects.

Leaderboards tell you which model wins in the abstract. Retort tells you which stack wins for the code you’re shipping — how reliably, how fast, and for how much. Sometimes the answer is the newest model; sometimes it’s the one that’s four times cheaper. You won’t know until you measure it.

Code, data, and full per-run results: github.com/adrianco/retort

What happens when it’s possible to build a company without any middle management people? Teams of agents managed by agents guided by desired outcomes derived from a purpose. It’s not possible to convert an existing company to this model, but I’m seeing new startups forming in months around the ideas of a handful of people that in the past would have taken tens or hundreds of people years to build.

I’m also seeing projects take off with a single author, that would have been a company a few years ago. They are open source partly because they didn’t cost much to make so they don’t need revenue from sales as a product.

Reuven Cohen built Claude Flow and developed it into Ruflow amongst many other projects.

Steve Yegge has built Gastown and is now moving on to Gas City. https://steve-yegge.medium.com/gas-town-from-clown-show-to-v1-0-c239d9a407ec

OpenClaw and similar agentic automation systems have been started by one person and extended by their user base. I have recently been trying out https://instar.sh and contributed iMessage support to the project.

That’s for developer tooling, with no need to hire a management team to build something that deeply affects the industry.

I’m also looking at how agents can extend continuous decision intelligence by automating the collection of business metrics, deciding what to do about it, and making whatever changes are needed. When there is high certainty, without human intervention, but with guard rails and rules of engagement that direct the human domain expert to be in the loop only when needed.

The companies that operate in NoMan’s land will be created from scratch. Existing companies have too much baggage and can’t get there by firing employees. These new companies are built around a domain expert with a sense of purpose and are created by defining the outcomes they want and having the platform build it.

This isn’t a comfortable future for middle managers to contemplate, but unless your company purpose explicitly includes developing and employing people (eg consulting businesses) the people are in support of the purpose.

Will it work? I’m leaning into the future (as usual) but it doesn’t need to work for everyone to get started as a pattern, it will do the usual innovators dilemma thing of solving for simple cases and growing from there.

Edit: I’m going to add more relevant examples below as I hear about them.

https://paperclip.ing/ creates a zero human company. It was created in March 2026 and is open source with 50k stars. Here’s more background about it: https://pub.towardsai.net/paperclip-the-open-source-operating-system-for-zero-human-companies-2c16f3f22182

https://cofounder.co/ lets you run an entire company with agents.

https://savas.me/2026/06/04/the-boat-is-in-the-water-spring-voyage-is-open-source/ — a collaboration platform for humans and agents

https://jiegou.ai/ — operations layer for AI

There are more companies in this space that I’m aware of that aren’t public yet…

Advent(ure) in System Seeing

Think of a situation you’d like to explore with a systems lens as we practice various systems approaches and views. It’s good (since these 15–20 minutes of daily journaling add up) if it’s something that matters to you to explore and understand, and begin to shape responses to. Write a few paragraphs describing the situation. We’ve been doing “warm-up” workouts in systems and seeing (observing, noticing, seeing more deeply), and now we’re shifting to applying system lenses and practices to a situation of particular interest to you.

(Note: I’m writing this blog post in the UK and my computer has decided that I should be using UK centred or English specialised spelling for some reason.)

The situation I will describe is one that I’ve encountered a few times. It’s around the establishment, funding and goals of a systems performance team. My early career was as a performance tuning specialist, I wrote a popular book on Sun Performance Tuning in the 1990s, and worked with many people who specialise in this area.

Performance tuning teams operate on two distinct metrics, speed and cost. They make computer systems respond more quickly, which has an indirect effect on business value, and make them run more efficiently, which saves money by doing more with less resources. For poorly tuned systems, work on either of these metrics tends to improve both metrics, but situations also occur where a tradeoff between cost and speed can be made. In addition, resilience and stability tend to improve when systems are well tuned, although those are not usually a direct goal of a performance tuning team.

Business justification to create and fund a performance tuning team occurs when there is an emergency issue that needs to be fixed, or more proactively, a significantly large spend that needs to be optimised.

The system that I aim to setup is to get the performance team to be recognised as a profit centre not a cost centre. Most organisations are under-invested in performance work, so adding headcount/spend generates a high return on investment. Convincing the CFO to make an exception and dedicate extra headcount to an obscure engineering team deep in the organisation is left as an exercise to the reader.

An anecdote, back in 2011 Netflix was scaling up streaming rapidly on AWS but hadn’t put any effort into performance optimization. Then there was a botched attempt to spin off the DVD business into a new brand called Quickster, and Netflix had a bad quarter, with customers cancelling their subscriptions. It looked like we would make a loss for the quarter, but it was going to be close. The cloud team looked at the AWS bill and spent a few weeks optimising and tidying up our spend, saving several million dollars over that quarter, and Netflix ended up just above break-even, by an amount less than the cloud team had saved. We also effectively flat-lined our AWS spend for a few quarters even though the streaming business traffic more than doubled. Soon after that we created a dedicated performance team to create tools, find problems that needed fixing, and work with whoever owned the code to get it fixed.

In the datacenter world, performance teams don’t save money immediately, because they just delay the next purchase cycle, but in a cloud environment, you can get savings on next month’s bill.

The workload and goals of a performance team should be organised into a backlog of tuning project ideas. Those ideas get fleshed out into projects and rated using a methodology like Reach, Impact, Confidence, Effort (RICE) as described in the recent book on fixing developer experience Frictionless, by Nicole Forsgren and Abi Noda. In the case of performance projects, the incremental financial savings in each quarter should be enough to justify the headcount needed to make the projects happen. If the CFO wants more savings sooner, then more performance engineers are needed, and more pressure needs to be put on other teams to work on prioritising delivering performance over features on a regular basis.

Finally, I’d like to point out that in my experience most developers are not taught systematic performance tuning methodologies, and that many people working in the field of performance tuning, including myself, have physics degrees. Physics experiments teach data collection, mathematical modelling, statistics, and the scientific method of hypothesis testing. Good performance tuning engineers are a “special breed” and may not be great developers, so the typical developer oriented interview and hiring processes can exclude them. It’s hard to interview for people that can solve problems you don’t understand. My favourite interview question centres around queueing theory, Little’s Law, and how many threads a server is going to use.

Advent(ure) in System Seeing

We have an alternative here, where you can either draw a rich picture again (but with a new focus), or draw an actor map. They capture overlapping details, but Rich Pictures include interactions, whereas Actor Maps gathers more notes on the perspectives, orientations, and so on, of the actors (and other stakeholders). So, either:

• draw a Rich Picture that tells a visual story of the situation (that you described on Day 11). Identify who is involved (actors, groups, organizations, systems) and draw interactions among them, labelling the interactions and also jotting notes on the picture to capture the roles, cares and concerns, and orientations of the people, groups or entities. (See Rich Pictures on Day 4 to refresh.)

or

• draw an Actor Map, where you briefly summarize the situation (described on Day 11) in the center of your map (in a cloud to make it distinct). (See diagram above.) Identify actors (individuals or groups) who are involved in and influence the situation, and add them around the (summary) situation cloud. (If there are many, use your judgment about who to focus on.) Alongside each actor (or group), cluster notes about the actor. These notes may include: brief description of the situation from their perspective; what the actor or group cares about (as it relates to the situation) and what they are concerned/worried about; how they orient to the situation; how they contribute to the situation; and what their expections are. Repeat, for each actor (or group) that you’ve identified to focus on. (You could do this with notes in a table, if you really want a break from pictures. However, seeing the different perspectives around the circle, allows us to notice shared interests and potential conflicts (of interest, orientation, etc.).)

The image at the top of this post is the result. My additional thoughts are around the worries and blockers, and how to make performance optimisation part of the culture of the organisation, rather than a one-off event. It’s up to the CTO and senior management to publicly celebrate successes and goal the product organisations with latency and efficiency goals as well as feature delivery goals. It’s up to the performance team, working with the product teams, to carefully test and gradually roll out change so they don’t break the product. It’s up to the cloud economics team to estimate possible gains, then validate those estimates against actual measured results, and to calibrate their own estimation models. Platform management needs to propose the concept of the performance team as a profit center to both technical and financial management, to secure the resources and establish the measurements of results that will create a self sustaining system, with positive funding feedback, until the available project backlog doesn’t justify additional headcount.

Advent(ure) in System Seeing

Day 10: Draw Your Org 3 Ways

1. Draw an organization that you have been part of (where you work, community organization, etc.), at least 3 different ways. Especially when getting ideas flowing, have fun with it… do the obvious, the playful, etc.

2. Jot down reflections on what you learned about your org from doing this. What do these different views illuminate? What do you learn looking across them?

3. Now “go meta” and think about the views you drew:

What first occurred to you to draw? What do our customary views of organizations suggest? What views are helpful? Why?

Of course, the classic, as org structure doodles (all grown up into a comic) go, is Manu Cornet’s! (Which manages to be an “org structure chart” that is about power, communication, and org culture.)

“One of the hardest and most valuable things you can do as a company is the following:

Have a fully up to date org chart

Have a diagram that is not the org chart that accurately reflects how work flows through the company

Have an up to date and accurate diagram and explanation of what the company does and how it does it (architecture, revenue funnels, business value streams, code-bases)“

— Hazel Weakly

The inverted tree oriented org chart shown as the image for this blog is my favorite way to lay out an organization because it emphasizes the servant-leader principles that the twigs and leaves are where the work is being done, and managements job is to support and grow those profit center parts that reach out to customers. Products could even be shown as fruit on the tree to extend the analogy. The tree also shows the support functions as roots, cost centers keeping the tree stable, that are fed by the profits generated by the leaves and fruit. By showing management in a supporting role, it changes the conversation and cultural attitudes. The servant leadership ideas were first described by Robert Greenleaf in 1970.

I’ve always had problems making an org chart that looks nice, and like a tree, with the right positions in the right places. Maybe with some AI coding and image generation help this would be easier to draw.

One organization I work with as an advisor uses Slack as the primary communication channel, with very little email. Their org chart is embedded in Slack, so it’s trivial to figure out for anyone you are talking to, who they work for and what part of the org that is. The slack channels form the communication structure of the company and I think this works well.

When I worked for Sun Microsystems they had an online web based employee database. However when the company started to shrink and lay people off via a “Reduction in Force” (RIF) in 2001–2004, the HR department was trying to keep the details confidential. The RIF’d employee had their email and work access shut down, and was sent home on full pay for a few months, but their org was dissolved so they appeared differently in the database. Some people figured out that they could download copies of the employee database and track who was being RIF’d when. What followed was some back and forth as HR tried to find ways to hide the information and the engineers found ways to uncover it again.

There was a time at Netflix, in 2010 or so, when engineering was small enough to fit in one building, and there was an office-based in-person culture. There was a conscious effort to lay out the teams across the office such that the structure of the product was laid out in sequence. This meant that the user experience designers were at one end of the building, next to the front end developers, who were next to the API team, then the various personalization teams, then the database and platform teams. At the time, the systems thinking approach was to maximize internal informal communication channels, and to have less reliance on written documentation, which would slow things down. I think it worked at that scale, but for a fast growing organization, we out-grew it quickly as we needed more space for people. Netflix is now a globally distributed organization, and I recommend reading No Rules Rules by Reed Hastings and Erin Meyer as it describes how they made that transition.

Advent(ure) in System Seeing

Day 8: Context Changes Everything

Before we get into today’s activity, a word about where we’re headed. We’ve been practicing systems sensing and sense making: zooming in to see the system and its parts and interrelationships (day 1), zooming out to see the system in its vaious contexts (day 2), and shifting perspective to consider what is taking shape (day 3). We used Rich Pictures to explore some of the infrastructure we rely on, in a form that draws out people, organizations and systems involved, but also their interactions, and concerns (day 4). We considered a dynamic model of shaping influences in a situation, mapping causal networks and considering feedback loops (day 7). And we started to draw out some of the ideas about systems that have been percolating (day 5). We will again step back and gather thoughts about central concept in systems seeing (today, day 8).

We were getting practice with some of the perspectives and tools of system understanding, but the focus, while hopefully interesting, was likely generally outside our current set of concerns. We will shortly shift focus to applying the practices and tools of system seeing to a situation you care about. I’m mentioning that now, so you can start to think about what situation or system you would like to focus on. This may mean we have fewer folk sharing their work done, because its personal or work sensitive. Still, the part where we step back and reflect on what we learned may be shareable. Ok, back to today’s activity.

Today’s Activity: On Day 5 we created an overview zine page and optionally one on another topic. Today, we will return to the idea of creating a page (or two) of a zine. The topic is up to you, but I’ll suggest the focus is Context (or environment or situation). Again, you’re drafting ideas for the design of a zine page on context (or another concept that is fundamental to systems, like purpose). Why do we consider (system) context, when we’re talking about systems? What are associated concepts? Is there an explanatory diagram that helps convey why context is important to consider, or what is useful to draw out and make visible? Think back across the work we’ve done (bicycle in context (day 2), changes and impacts as ebikes become more prevalent (day 3), dependencies on other systems and infrastructure (day 4), causes and effects that shape the situation (day 7)), and draw out lessons.

“We fail more often not because we fail to solve the problem we face but because we fail to face the right problem.” — Russell Ackoff (via Jamshid Gharajedaghi in Systems Thinking)

“the meaning of the system can only be found in its context, so what you discern as ‘context’ is inextricably part of what you discern as ‘system’.” — Patrick Hoverstadt The Grammar of Systems

I am so un-skilled in art, that I failed art class in high school (although I got high marks in technical drawing/draftmanship). I also don’t have access to my iPad and Apple Pencil today and am trying to do this on my laptop, so I decided to see if I could manage some agentic AI skills to produce something I liked. I’d heard of Google’s Nano Banana tool, so I decided to try Google Gemini image generation. Here is my initial prompt:

Design a page that is part of a zine on systems thinking, and the page focuses on the topic of context

This is what Gemini thought of that:

Exploring the Concept

I’m currently breaking down the request: “Context in Systems Thinking.” I’m focusing on defining “context” in this specific field, considering elements like environment, boundaries, and perspective. My aim is to explain why context is so vital in Systems Thinking, and how it shapes the outcomes we observe. I’m moving forward with a strong emphasis on these core concepts.

Visualizing Context’s Role

I’m starting to build a visual metaphor to represent context. I’m focusing on a “frame” around a system, like a gear mechanism. Outside the frame, I’m depicting elements that influence the system but aren’t part of it, to demonstrate the influence of the environment and boundaries. I’m also considering how perspective alters the view within the frame. The goal is a zine-style page that clearly communicates the importance of these relationships in Systems Thinking.

Visualizing Context’s Role

I’m starting to build a visual metaphor to represent context. I’m focusing on a “frame” around a system, like a gear mechanism. Outside the frame, I’m depicting elements that influence the system but aren’t part of it, to demonstrate the influence of the environment and boundaries. I’m also considering how perspective alters the view within the frame. The goal is a zine-style page that clearly communicates the importance of these relationships in Systems Thinking.

Designing the Layout

I’m currently working on the layout. I’ve placed the visual metaphor — the framed gear mechanism — prominently in the middle of the page. Surrounding it is a chaotic environment, visually contrasting the system’s clarity. Supporting text, including key slogans like “SHIFT THE FRAME, SHIFT THE MEANING” are being added around the central image. I’m focused on creating a balanced and visually engaging composition within the zine aesthetic.

Defining Key Visual Elements

I’m now zeroing in on the visual elements. The “frame” metaphor will prominently feature a gear mechanism encased within a clearly defined border, representing the system’s boundary. Outside this frame, I’m sketching a chaotic, diverse environment to illustrate the influences that the system interacts with. Adding the lens element, signifying perspective, will change what’s inside. Finally, I’m working with the layout of text in relation to the main visual.

Constructing a Visual Narrative

I’m now integrating the text and visual components. My main focus is on the central metaphor: the “frame” around the system. I will portray a clear distinction between the orderly system within and the chaotic environment outside. I’m focusing on text placement in the context of the zine aesthetic. My aim is to make the core idea of “CONTEXT” the focal point.

\n\n I am finalizing the zine page. I’ve designed the visual elements and the layout. I’m aiming for a clear visual representation of how context influences understanding. It highlights the importance of perspective. My final objective is to create a well-balanced visual narrative using my key elements.

Conceptualizing the Narrative

I’m now integrating the text and visual components. My main focus is on the central metaphor: the “frame” around the system. I will portray a clear distinction between the orderly system within and the chaotic environment outside. I’m focusing on text placement in the context of the zine aesthetic. My aim is to make the core idea of “CONTEXT” the focal point.

Examining Conceptual Integrity

I’m currently working to determine if the generated content precisely aligns with the user’s initial prompt. The initial look confirms the ‘zine page’ feel and raw textures. The abstract concepts are clearly visualized.

Analyzing Alignment & Focus

I’m now carefully checking the alignment. The image mirrors the requested “zine page” aesthetic. The key concept of ‘context in systems thinking’ is represented clearly through the metaphor of a boundary. Additional elements, like the eye and magnifying glass, further support the central theme and help reinforce the key concepts of the page.

The analysis seems reasonable, I left it thinking about that for several minutes and it failed to produce any image. I stopped it and complained and it re-loaded nano-banana and produced the image immediately.

Looks pretty good to me, apart from the “CUDN…” label and the “-Unknown” quote attribution. I tried looking up the quote, and there isn’t a clear attribution for it, but it seems on point, so I asked for some tidy up and ended up with this image. Good enough, and much better looking than my attempts.

Think back across the work we’ve done (bicycle in context (day 2), changes and impacts as ebikes become more prevalent (day 3), dependencies on other systems and infrastructure (day 4), causes and effects that shape the situation (day 7)), and draw out lessons.

The context that changed and allowed eBikes to be practical is the development of lightweight low-cost energy dense Lithium batteries. A lead-acid battery would be too heavy and have too short a range to be practical. Laptop computers and phones drove the initial development of Lithium batteries, but cars have taken over as the biggest driver of that technology, with eBikes and other use cases benefiting from a slipstream effect.

Day 9: Relationships and Interactions

Advent(ure) in System Seeing

In a couple of days we’ll switch gears, and each focus on a situation of our choosing for most of the remaining activities. But today, we’re going to watch a lovely short ( 4 minute) video about … trees! Jot down notes (and sketches) as you watch it. With any time that remains, draft a zine page on Relationships or Interactions (as relates to systems and ecologies of systems).

The Secret Language of Trees, by Camille DeFrenne and Suzanne Simard

Everything within a system is changed by being part of that system, every element, every relationship is changed.” — Patrick Hoverstadt The Grammar of Systems

“They all together make a certain situation, but they constitute that situation through their relation to one another. If you change one, usually some, if not all, of the others are changed.” — Mary Parker Follett, The Illusion of Final Authority

To understand a system (and ecology or system of systems), we need to understand not only the components but their relationships and interactions.

Notes on the video: recent research has found that tree roots are much more complex than expected, and have a symbiotic relationship with a large number of specialized kinds of fungus. The fungus takes nutrients from the soil and passes it into the tree in exchange for sugars that the tree makes. The fungal networks extend far beyond the roots of each tree, and actually connect trees together, such that mature trees support the development of young trees, even though their root systems don’t overlap.

Nutrients flow from high concentrations to low concentrations, and this is enough to pass them over long distances via fungal filaments that are strands of single cells.

Thinking about this some more, when a tree dies, it leaves roots without a sugar supply, but other trees will keep the fungus alive in that area, and help to support the replacement trees that are growing in the new clearing, so all the system needs to operate is the flow concentration gradient and trees that are close enough to each other for the fungal networks to connect.

My attempts to get Nano Banana to fix the image and text it came up with failed, so here is a somewhat messed up zine page attempt.

Advent(ure) in System Seeing

Ask those who are interested in systems to recommend a book, and chances are good that they’ll recommend Donella Meadows’ Thinking in Systems: A Primer. It’s wonderful! She was a great teacher, and a couple of her lectures survived the decades and are online. So, let’s go to Meadows herself to learn about Causal Loop Diagrams.

The whole lecture is great, but to keep to 15 minutes, start at minute 18:59. Take sketchnotes as you watch. (And stop after 15 to 20 minutes.)

I’ve been running behind and catching up but haven’t been looking ahead to see what the next few days challenges are about. I read Donella Meadows book a decade or so ago and hadn’t seen this video before, and didn’t know that we were going to talk about feedback loops next. I watched the entire video, it was worth it.

https://www.youtube.com/watch?v=XL_lOoomRTA

Some notes I made while watching.

There is a lot of detailed diagram syntax that she uses that I haven’t seen used except by her. Another book on Systems Thinking by Jamshid Gharajedaghi is one I studied more closely and I think had some different notation. It’s more focused on purposeful business and technology systems, and was written a few decades later than Donella’s book.

The feedback loop examples are good. The positive feedback of annual interest on bank balances is small and slow acting, so it doesn’t have time to become unstable. The stock market positive feedback of people buying into a stock that is going up (greed), and selling a stock that is going down (scared) is much more chaotic, and has additional external systems loops that have negative feedback that limit upside when people get scared that its gone too far, and limit downside by stopping trading entirely.

Sadly Donella passed away in 2001, but I was lucky enough to meet Jamshid about a decade ago. I sent him a copy of a presentation I had made about Systems for Innovation that referred to his work, and he invited me to give an in-person talk to his students at Villanova.

Day 7: Give Causal Loop Diagrams a Spin

Advent(ure) in System Seeing

On Day 6 we learned about Causal Loop Diagrams from Donella Meadows. Today we’ll get a little practice using causal loop diagrams to understand interacting effects in a system. Deciding what to explore can be the hard part, so here are some options (pick one, or explore a tricky/sticky challenge you face):

the effect of adding lanes to a highway (induced traffic)

bringing wolves back to Yellowstone (watch 4 minute video)

the “flywheel effect” (eg etsy flywheel) to explore growth strategy (where value increases when other people also consume the same brand or participate on the same platform)

technical debt (in the colloquial sense): how does it arise, how is it sustained, what can we do to avoid it or to get out of vicious cycles?

explore impact of genai or agentic ai on understanding and skills

I’ve been using Agentic AI to supply developer skills, so decided to explore that topic. My first attempt was really about defining the purpose of understanding and skills: doing things successfully, by understanding them and being able to manipulate them to a satisfactory outcome. To do that humans have have an understanding (a model) of what they are trying to do, and remember their education, training and experience as they work on a task. Agentic AI tends to have a much broader training set than any one human can remember, but use simpler models than humans. The result of the skill is fed back as human experience and agentic context, and some iteration may be needed to get the successful outcome.

Given my own experience, I felt that this didn’t capture the systemic difference I’ve seen, so I started again with a different approach.

A skill is needed to perform a task, so I decided to try a task flow model. The task may be something the human is not skilled in (for me, github commands, programming languages I don’t know), which the human would have to research and study and then execute and iterate until it worked. I estimate that humans code at about 1000 tokens/hr of finished output when working on tasks, and personally, I often run out of time before I’ve even finished the research and study phases needed to get something done.

A skilled human would skip the research and study, and be more likely to execute correctly first try than an unskilled human, so would read/write a lot fewer tokens to get the work done.

Agentic AI knows how to do just about any well documented developer task, but also I recently tried to use a brand new open source software package and it read the blog post and repo documentation and used it successfully first time. Agents code at about a million tokens/hr, and the number of tokens they need to get work done is dropping rapidly.

I represent an unskilled human, who doesn’t have enough time to become skilled in git commands, Python libraries I’ve never seen before, coding in Swift etc. I have enough developer management skill to direct an agent to get things done in far less time than even a skilled human would take, and can iterate a few times if needed to get the results I need.

The negative feedback loops in the case of skills are primarily the evaluation of whether the result is good enough yet, and what went wrong that needs to be fixed. There’s an outer feedback loop I haven’t shown which evaluates whether Agentic AI works well on a particular task (is it sufficiently skilful) assigns work to skilled humans or agents as needed, and builds a skills model that increases the success rate of agents by giving them smaller tasks that align with stronger skills. For example I currently expect code to work first time for agents coding in Python, and to have syntax and compile time errors to work through for agents coding in Swift.

Only a day or so behind Ruth Malan now…

Advent(ure) in System Seeing

Let’s gather some of the thoughts that have been percolating around systems. Let’s assume that you are designing the introductory page of a zine on systems, so the frame you’re working within is about the size on 5x3 index card. I’m suggesting a “zine,” because they can be playful (or serious), have graphic elements (or not), etc., but the key is the small size as constraint. At this point, we’re designing the page, jotting down ideas (and images) that will go on the page, and then drafting the page.

Some ideas for the page: a characterization of systems. A list of the key ideas in systems. Some images that illustrate your characterization. Some examples.

If that falls out quickly, pick a key concept like Boundaries, and do the same thing, drafting ideas for the design of a zine page on system boundaries. Think of analogies that suggest visuals and insights, like cell walls and containers. Why are boundaries important, and what role do they play (in systems or our conceptions of systems)? Etc.

Once again, reserve some time (in your 15–20 minutes) to step back and reflect on what you learned.

Feedback affects the ability of a system to respond to change, and whether that response is stable or unstable. It’s mathematically defined as part of control theory, and complicated systems can be modelled accurately. Complex (and chaotic) systems don’t have exact analytical solutions but the same feedback principles apply.

Feedback starts with system that has an input and an output, and a desired value of that output. The difference between the actual output and the desired output is the error. A feedback algorithm looks at the error and changes the input to hopefully improve the output. The feedback system may also anticipate a desired change in the state and blend that with the error as it exerts control.

Feedback is characterised by two values, the strength of the feedback and the delay that occurs before the feedback affects the output. The most common mistake people make is that they don’t think about or measure the delay. Since the input is changing all the time, the time it takes to apply the feedback must be much faster than the rate of change of the input and output.

The strength and direction of the feedback affects how much it changes the input. Negative feedback takes the a proportion of the error and subtracts it from the input. Positive feedback occurs when the error is added to the input, and that creates an unstable system. Too high levels of feedback may cause overshoot and “ringing”, and too low levels may take too long for the system to correct the error. If you get it just right, a perfectly damped system minimises the error without oscillating.

If the feedback takes too long to take effect it acts like an input that is inappropriate to the current state of the system, and will have an effect that is sometimes positive and sometimes negative, and the system becomes unstable.

In the steering example, the inertia of a car is a stock of kinetic energy that depends upon the mass and speed of the car, and higher inertia slows the response of the car to inputs.

Advent(ure) in System Seeing

Day 4: Going Behind the Scenes

We want to continue, for a while, to focus on concepts and systems that are at least somewhat in everyday encounter (and not private or “business sensitive”) so that folk can generously share their work if they’re willing to contribute to community learning that way.

Rich Pictures (example above) come from Peter Checkland’s work on soft systems, and are useful to explore the interaction of people (roles, groups, etc.) and organizations and systems of various kinds. They are informal “visual stories” depicting the relationships and flows of interactions among entities involved in some situation of interest.

Draw a Rich Picture, exploring all that happens “behind the scenes” when you fill a kettle or pot with water. Unravel the story of that pot of water, going further and further back. Perhaps you get to rain drops. But also teasing out other systems that are involved in getting water to you. Perhaps you get to metering and payments. To prompt more ideas to mind, what if you go to draw a pot of water, and there is no water, what could have failed? The graphical language is informal. Identify who (people, organizations, systems in the landscape of this frame) is involved (with sketches, annotating to add what their purpose and role is as it relates to this situation, what they care about, their concerns, and so forth), how they interact (using arrows and labels). Stick figures and block buildings are fine! The key is to have fun.

Continue adding to your picture, using judgment not to over-clutter, but to draw in people, groups, organizations, systems and their interactions and roles and concerns, as they seem significant to you, to understand the various systems in the background, managing water getting to your pot or kettle.

Once again, reserve some time (in your 15–20 minutes) to step back and reflect on what you learned. What do we learn about what we take for granted? About infrastructure and dependencies? About system properties like resilience? How does this relate to our work?

“[Infrastructure] “Becomes visible upon breakdown. The normally invisible quality of working infrastructure becomes visible when it breaks: the server is down, the bridge washes out, there is a power blackout.” — Susan Leigh Star, The Ethnography of Infrastructure

There’s a long chain of dependencies in water supply. Recently I’ve read stories that the capital city of Iran is being abandoned because there isn’t enough water, and that water is being polluted by agricultural fertilizer runoff. Climate change is disrupting rain patterns, over-use is draining aquifers, old mains pipes fail and leak, and some houses may have lead pipes or contaminated water supply.

Our house has an integrated softener and filter that I have to top up with salt regularly, and I’m the chef in our house, so I’m usually the one filling a pot with water. We used to live in a house in the Los Gatos Mountains of California that had its own well, pumps and storage tanks, and after large local earthquakes, the wells often run dry for a while, or may never return to use.

Advent(ure) in System Seeing

Day 2: Zoom Out to See the World Around a Bicycle

Draw a Concept Map or Bubble Diagram to explore the world (context) around bicycles. Zoom out, so that “bicycle” is just one bubble in the center of the diagram; what’s around it?

For example, bicycles are sold and repaired. They are ridden in different contexts, like commuting and recreation. Seek out different contexts across the lifecycle, not only across use situations and environments.

Step back and reflect on what you learned. Why is it useful to see a system in context, and in these various contexts that are relevant, in different ways at different points?

The history of bicyles conditions the marketplace of what riders expect, and within the limits of safety regulations, manufacturers try to meet the needs and innovate for different kinds of riders so that they can sell new bikes for competitions, recreational riding and daily use like commuting.

The recent rise in functionality and popularity of e-bikes has driven a higher price point and new demand for bikes, as it enables extended commuting over longer distances for more people.

Day 3: Explore Impact of Change

We’ll stay with bicycles for one more journaling session. So far, we’ve “zoomed in” on the parts and relationships that consititute a bicycle, and we’ve “zoomed out” to the various ecologies of systems around bicycles. Now we’ll look at what’s changing and impacting (one or more of) these ecosystems.

A Futures Wheel is a useful in exploring the impact of a change or trend. The change or trend is put in a circle at the center. In a ring of circles around that center, we identify direct effects or consequences of that change (see image above). Indirect consequences are identified and positioned in the outer (third) ring of circles.

Use a Futures Wheel to explore a change that is impacting cycling or the bicycle industry, in terms of direct and indirect effects. One example would be the trend toward ebikes.

Step back and reflect on what you learned.

I already anticipated this question in principle at the end of my Day 2 answer. Here’s my futures wheel for eBikes.

eBikes are a significant carbon offsetting trend, replacing car journeys for more people, over more difficult terrain and longer distances.