One old dragon that’s reared its head again in this “age of AI” is the very wrongheaded notion that productivity == code. Managers aim to maximise the amount of code their dev teams produce, and so they maximise the time devs spend writing code.
Let me tell you a story about the value of code.
When I first started contracting, I worked on a Point Of Sale system upgrade for a major retailer. I was stuck on a feature, and just couldn’t wrap my head around the use case because I’d never actually seen their existing system in operation.
So I walked to a local branch on the high street, showed them my security pass, and asked if I could observe and – when the cashier wasn’t busy – ask questions.
They told me they could do better than that. Upstairs they had a room with a working till in it, which they used to train new staff. They also used it to test system updates, which was the first time I’d seen what we now call a “model office” for software testing.
We were able to run through the use case scenarios I was stuck on, with them showing me how to use the POS system to achieve specific goals. The mist cleared. It was like I’d been reading a book on how to ride a bicycle that had no pictures, and then someone gave me a bicycle.
Enlightened, I walked back to the office and changed about three lines of code. That was all the coding I did that day. Three lines, in 8 hours.
But if I hadn’t made that trip and seen for myself, and had a chance to talk to a department manager in that store, those three lines would have been applying special offers wrong. (If only the person who wrote the spec had done this in the first place…)
Now multiply that error by 250 branches nationwide. I potentially saved my client a bunch of money and embarrassment with that 3-line change.
Now, I consider that a productive day.
But had I been measured on my contribution by lines of code, or commits, or features finished, it would have been seen as a very unproductive day by my manager.
I may have felt pressured to stay at my workstation, bashing out more code, compounding the mistake and costing my client more money.
A very teachable moment for me early in my career. The lightbulb pinged: some code is worth more than others, and coding and productivity aren’t the same thing.
We could even argue that coding is the interruption. What’s the shortcut in IntelliJ that tells you if you’re writing the wrong code? Oh, that’s right. There isn’t one.
When I’m heads-down-coding, I’m not seeing, I’m not asking, and I’m not learning about the problem. To do that, I have to get up from my desk, go to where the problem is and/or the people I need to ask are, and have a conversation. Let the dog see the rabbit.
This takes time. And if we’re producing code faster than we can validate it – either by exploring the problem ourselves, or learning from user feedback if our release cycles are fast enough – then we’re piling assumptions on top of assumptions.
I’ve seen so many times how 10 lines of code can end up being worth £millions, and 10,000 ends up being worthless.
If productivity, in reality, is a measure of how much net value we create, then that learning feedback loop is where the real productivity happens, and not at our desks punching keys. Coding is when we’re least productive.
“The age of the syntax writer is over”, hailed a post on LinkedIn. (Why is it always LinkedIn?)
I say: welcome to the age of the syntax READER!
If I presented you with a bowl of candy and told you only 1% had broken glass in them, what percentage would you check before giving them to your kids?
The fact is that a 1% error rate would be an order-of-magnitude improvement in the reliability of LLM-generated code, and one we shouldn’t expect for a very long time – if ever – as scaling hits the wall of diminishing returns at exponentially increasing cost.
The need to check generated code isn’t going to go away, and therefore neither is the need to understand it.
And, given the error rates involved, we may need to understand all of it. At least, anyone who doesn’t want to be handing candy-covered shards to their users will.
“But Jason, we don’t need to check machine code or assembly language generated by compilers.”
That’s a category mistake. When was the last time a compiler misinterpreted your source code, or hallucinated output? Compiler boo-boos are very rare.
If my compiler randomly misinterpreted my source code just 1% of the time, then, yes, I’d check all of the generated machine code for anything that was going to be put in the hands of significant numbers of users. And that means I’d need to be able to understand that machine code.
Now for the fun part.
Decades of studies into program comprehension show that one of the best predictors of a person’s ability to understand code is how much experience they have writing it.
Cognitively, we don’t engage with syntax and semantics more deeply than when we’re writing the code ourselves. (Which is why I strongly discourage students from copying and pasting – it stunts their growth.)
In my blog series The AI-Ready Software Developer, I talk about how a mountain of “comprehension debt” can rapidly accumulate as “AI” produces code far faster than we can wrap our heads around it. I call this “LGTM-speed”.
I recommend “staying sharp” where code comprehension is concerned, as well as slowing down code generation to the speed of comprehension. When you’re drinking from a firehose, the limit isn’t the firehose.
This implies that we’re not limited by how many tokens/second “AI” coding assistants can predict, but by how many tokens/second we can understand. That’s the thing we need to optimise.
The best way – the only way, really – to maintain good code comprehension is to write it regularly.
We need to keep our hand in to make sure we don’t get caught in a trap where comprehension debt balloons as our ability to comprehend withers.
That leads to a situation where serious, show-stopping – potentially business-stopping – errors become much more likely to make it into releases, and there’s nobody on the team who can fix them.
And in that scenario, who are they gonna call? The “AI-native developer” who boasts they haven’t written code in months, or the developer who was debugging that kind of code just this morning?
And as a teacher of TDD for some quarter of a century now, you can probably imagine that I’ve heard every reason for not doing TDD under the Sun. (And some more reasons under the Moon.)
“It won’t work with our tech stack” is one of the most common, and one of the most easily addressed. I’ve done and seen done TDD on all of the tech stacks, at all levels of abstraction from 4GLs down through assembly language to the hardware design itself. If you can invoke it and get an output, you can automatically test it. And if you can automatically test it, you can write that test first.
(Typically, what they really mean is that the architecture of the framework(s) they’re using doesn’t make unit testing easy. That’s about separation of concerns, though, and usually work-aroundable.)
The second most common reason I hear is perhaps the more puzzling: “But how can I write tests first if I don’t know what the code’s supposed to do?”
The implication here is that developers are writing solution code without a clear idea of what they expect it to do – that they’re retrofitting intent to implementations.
I find that hard to imagine. When I write code, I “hear the tune” in my head, so to speak. The intended meaning is clear to me. When I run it, my understanding might turn out to be wrong. But there is an expectation of what the code will do: I think it’s going to do X.
My best guess is that we all kind of sort of have those inner expectations when we write code. The code has meaning to us, even if we turn out to have understood it wrong when we run it.
So I could perhaps rephrase “How can I write tests first if I don’t know what the code’s supposed to do?” to articulate what might actually be happening:
“How do I express what I want the code to do before I’ve seen that code?”
Take this example of code that calculates the total of items in a shopping basket:
class Basket:
def __init__(self, items):
self.items = items
def total(self):
sum = 0.0
for item in self.items:
sum += item.price * item.quantity
return sum
When I write this code, in my head – often subconsciously – I have expectations about what it’s going to do. I start by declaring a sum of zero, because an empty basket will have a total of zero.
Then, for every item in the basket, I add that item’s price multiplied by it’s quantity to the sum.
So, in my head, there’s an expectation that if the basket had one item with a quantity of one, the total would equal just the price of that item.
If that item had a quantity of two, then the total would be the price multiplied by two.
If there were two items, the total would be the sum of price times quantity of both items.
And so on.
You’ll notice that my thinking isn’t very abstract. I’m thinking more with examples than with symbols.
No items.
One item with quantity of one.
One item with quantity of two.
Two items.
If you asked me to write unit tests for the total function, these examples might form the basis of them.
A test-driven approach just flips the script. I start by listing examples of what I expect the function to do, and then – one example at a time – I write a failing test, write the simplest code to pass the test, and then refactor if I need to before moving on to the next example.
What I’m doing – and this is part of the art of Test-Driven Development – is externalising the subconscious expectations I would no doubt have as I write the total function’s implementation.
Importantly, I’m not doing it in the abstract – “the total of the basket is the sum of price times quantity for all of its items”.
I’m using concrete examples, like the total of an empty basket, or the total of a single item of quantity one.
“But, Jason, surely it’s six of one and half-a-dozen of the other whether we write the tests first or write the implementation first. Why does it matter?”
The psychology of it’s very interesting. You may have heard life coaches and business gurus tell their audience to visualise their goal – picture themselves in their perfect home, or sipping champagne on their yacht, or making that acceptance speech, or destabilising western democracy. It’s good to have goals.
When people set out with a clear goal, we’re much more likely to achieve it. It’s a self-fulfilling prophecy.
We make outcomes visible and concrete by adding key details – how many bedrooms does your perfect home have? How big is the yacht? Which Oscar did you win? How little regulation will be applied to your business dealings?
What should the total of a basket with no items be? What should the total of a basket with a single item with price 9.99 and quantity 1 be?
We precisely describe the “what” – the desired properties of the outcome – and work our way backwards directly to the “how”? What would be the simplest way of achieving that outcome?
class Basket:
def __init__(self, items):
self.items = items
def total(self):
if len(self.items) > 0:
return self.items[0].price
return 0.0
Then we move on to the next outcome – the next example:
If we enforce that items must have a price >= 0.0 and an integer quantity > 0, this code should cover any list of items, including an empty list, with any price and any quantity.
And our unit tests cover every outcome. If I were to break this code so that, say, an empty basket causes an error to be thrown, one of these tests would fail. I’d know straight away that I’d broken it.
This is another self-fulfilling prophecy of starting with the outcome and working directly backwards to the simplest way of achieving it – we end up with the code we need, and only the code we need, and we end up with tests that give us high assurance after every change that those outcomes are still being satisfied.
Which means that if I were to refactor the design of the total function:
If I write the code and then write tests for it, several things tend to happen:
I may end up with code I didn’t actually need, and miss code I did need
I may well miss important cases, because unit tests? Such a chore when the work’s already done! I just wanna ship it!
It’s not safe to refactor the new code without those tests, so I have to leave that until the end, and – well, yeah. Refactoring? Such a chore! etc etc etc.
The tests I choose – the “what” – are now being driven by my design – the “how”. I’m asking “What test do I need to cover that branch?” and not “What branch do I need to pass that test?”
And finally, there’s the issue of design methodology. Any effective software design methodology is usually usage-driven. We don’t start by asking “What does this feature do?” We start by asking “How will this feature be used?”
What the feature does is a consequence of how it will be used. We don’t build stuff and then start looking for use cases for it. Well, I don’t, anyway.
In a test-driven approach, my tests are the first users of the total function. That’s what my tests are about – user outcomes. I’m thinking about the design from the user’s – the external – perspective and driving the design of my code from the outside in.
I’m not thinking “How am I going to test this total function?” I’m thinking “How will the user know the total cost of the basket?” and my tests reveal the need for a total function. I use it in the test, and that tells me I need it.
“Test-driven”. In case you were wondering what that meant.
When we design code from the user’s perspective, we’re far more likely to end up with useful code. And when we design code with tests playing the role of the user, we’re far more likely to end up with code that works.
One final question: if I find myself asking “What is this function supposed to do?”, is that a cue for me to start writing code in the hope that somebody will find a use for it?
Or is that my cue to go and speak to someone who understands the user’s needs?
Here’s my hypothesis (and I’ve seen real-world examples with client teams that make me ask this question):
Dev teams bring “AI” coding assistants into their daily workflows. Very quickly, much more code starts hitting the feedback loops in their process: testing, code review, integration, user feedback.
This starts to overwhelm those loops. Delays get longer, more Bad Stuff leaks through into production, systems get less stable and teams end up spending more and more time playing whack-a-mole with issues.
Far from making these teams faster overall, the traffic jams get worse and their journeys take even longer.
So some teams adapt*. They reduce batch sizes going into the feedback loops: fewer changes being tested, reviewed and integrated at a time, in tighter feedback loops.
We know that if you tighten these feedback loops, three things tend to happen:
1. Lead times shrink
2. Reliability improves
3. Cost of change goes down
My hypothesis is that when we see positive systemic impact from “AI” code generation, it’s actually more attributable to the team adapting to it, and not directly to the “AI” itself.
“AI” code generation load-tests your dev process and forces you to address the worst bottlenecks.
Basically, “Your Waterfall is showing”.
* And, of course, some teams run in the exact opposite direction, getting even more Waterfall. Silly Billies!
It just so happens that I specialise in helping development teams build the technical skills needed to shrink lead times, improve reliability and lower cost of change – with and without AI.
Some of the marketing choices made by the “AI” industry over the last few years have seemed a little… odd.
The latest is a “breakthrough” in “agentic AI” coding heralded by Cursor, in which they claim that a 3+ million-lines-of-code (MLOC) web browser was generated by 100 or so agents in a week.
It certainly sounds impressive, and many of the usual AI boosters have been amplifying it online as “proof” that agentic software development works at scale.
But don’t start ordering your uniform to fight in the Butlerian Jihad just yet. They might be getting a little ahead of themselves.
Did 100 agents generate 3 MLOC in about a week? It would appear so, yes. So that part of the claim’s probably true.
Did 100 agents generate a working web browser? Well, I couldn’t get it to work. And, apparently, other developers couldn’t get it to work.
And while you’re looking at the repo – and it surprises me it didn’t occur to them that anybody might – you might want to hop over to the Action performance metrics on the Insights page.
An 88% job failure rate is very high. It’s kind of indicative of a code base that doesn’t work. And looking at the CI build history on the Actions page, it appears it wasn’t working for a long time. I couldn’t go back far enough to find out when it became a sea of red builds.
Curiously, near to the end, builds suddenly started succeeding. Did the agents “fix” the build in the same way they sometimes “fix” failing tests, I wonder? If you’re a software engineering researcher, I suspect there’s at least one PhD project hiding in the data.
But, true to form, it ended on a broken build and what does indeed appear to be broken software.
The total time GitHub spent running builds on this repo was 143,911 minutes. That’s 4 months of round-the-clock builds in about a week.
This strongly suggests that builds were happening in parallel, and that strongly suggests agents were checking in changes on top of each other. It also suggests agents were pulling changes while CI builds were in progress.
This is Continuous Integration 101. While a build is in progress, the software’s like Schrödinger’s Cat – simultaneously working and not working. Basically, we don’t know if the changes being tested in that check-in have broken the software.
The implication is, if our goal is to keep the code working, that nobody else should push or pull changes until they know the build’s green. And this means that builds shouldn’t be happening in parallel on the same code base.
Your dev team – agentic or of the meat-puppet variety – may be a 100-lane motorway, but a safe CI pipeline remains a garden gate.
The average job queuing time in the Action performance metrics illustrates what happens when a 100-lane motorway meets a garden gate.
And the 88% build failure rate illustrates what happens when motorists don’t stop for it.
The other fact staring us in the face is that the agents could not have been doing what Kent Beck calls “Clean Check-ins” – only checking in code that’s passing the tests.
They must have been pulling code from broken builds to stay in sync, and pushing demonstrably broken code (if they were running the tests, of course).
In the real world, when the build breaks and we can’t fix it quickly, we roll back to the previous working version – the last green build. Their agentic pile-on doesn’t appear to have done this. It broke, and they just carried on 88% of the time.
Far from proving that agentic software development works at scale, this experiment has proved my point. You can’t outrun a bottleneck.
If the agents had been constrained to producing software that works, all their check-ins would have had to go in single file – one at a time through the garden gate.
That’s where those 143,911 total build minutes tell a very different story. That’s the absolute minimum time it would have taken – with no slip-ups, no queueing etc -to produce a working web browser on that scale.
Realistically, with real-world constraints and LLMs’ famous unreliability – years, if ever. I strongly suspect it just wouldn’t be possible, and this experiment has just strengthened that case.
Who cares how fast we can generate broken code?
The discipline of real Continuous Integration – that results in working, shippable software – is something we explore practically with a team CI & CD exercise on my 3-day Code Craft training workshop. If you book it by January 31st 2026, you could save £thousands with our 50% off deal.
The value proposition of Large Language Models is that they might boost our productivity as programmers (when we use them with good engineering discipline). And there’s no doubting that there are things we can do faster using this technology.
It would be a mistake, though, to assume that we can do everything faster using them.
I’ve watched many developers prompting, say, Claude or Cursor asking them to perform tasks that they could have done much faster – and more reliably – themselves using “classical” tools or just typing the damn code instead of a prompt.
For example, there’s been times when I’ve seen developers writing prompts like “Claude, please extract lines 23-29 into a new method called foo that returns the value of x” when their IDE could do that with a few keystrokes.
In these moments, the tool isn’t making them more productive. It’s making them less productive. So we might, when we find ourselves doing it – and I certainly have – pause to reflect on why.
It could be that we just don’t know the easier way. You might be surprised at how many developers haven’t even looked at the refactoring menu in their IDE, for example. Or that we know there’s an easier way, but don’t want to take the time to learn it.
In the latter case, it’s true that it would probably take them longer the first time. So they continue doing it the long way. Arrested development – often under time pressure, or perceived time pressure – is a common condition in our profession.
But in many cases, it seems performative. We know there’s a quicker, easier way, but we feel we need to show that it can be done – a bit like those people who insist you can cook anything in a microwave. Yes, technically you can, but is that always the best or the easiest option?
Someone calling themselves an “AI engineer” or “AI-native” might feel the need to signal to the people around them that they can indeed cook anything in the proverbial microwave.
And then it ceases to be about productivity. It’s about making a point, and demonstrating prowess to peers, superiors and random strangers on LinkedIn. The technology has become part of their professional identity.
Sacrificing real productivity in service to a specific technology or a technique is nothing new, of course. Software developers have been applying the “if all you’ve got is a hammer” principle for many decades – “I don’t know how we’re going to solve this problem, but we’re going to do it with microservices” sort of thing.
Quite often, these decisions – conscious or unconscious – seem to be career and status-driven. If “AI-native” is hot in the job market, that’s what we want on our CV. “AI when it makes sense” is not hot right now. It may be rational, but it’s less in-demand.
I’m still very much unfashionably rational, having sustained a long career by avoiding getting pigeonholed in the many fads and fashions that have come and gone. I’m interested in what’s real and in what works.
You never know. One day that might catch on.
If you want to hone your “classical” software engineering skills for the times when those are the better option, as well as learn how to apply engineering principles to “AI”-assisted development in an evidence-based approach that more and more developers are discovering gets better results – if it’s better results you’re after, of course – then check out my training website for details of courses and coaching, and oodles of free learning resources.
Software development’s essentially a learning process. Most of the value in a product or system’s added in response to user and eventually market feedback.
With each iteration we get the design less wrong. With each iteration, we learn.
The effect of batch size on learning is profound.
I urge teams to work on the basis that every design decision is guesswork until it hits the real world. We can’t know with certainty that we made the right decisions.
Getting user feedback is the only meaningful mechanism we have to “turn the cards over” and found out if we guessed right. In this sense, learning is characterised as reducing or eliminating uncertainty in product design. Teams who do this faster will tend to out-learn their competition.
Imagine trying to guess a random 4-digit number in one go vs. guessing one digit at a time.
In both approaches, we start with the same odds of guessing it right: 1/10,000. But with each guess, the uncertainty collapses orders of magnitude faster when we’re guessing one digit at a time. The latter approach out-learns the former.
Even if we had an “AI” random 4-digit number generator that enabled us to make 10x as many guesses in the same time, guessing one digit at a time would still out-learn us.
The chances of a complete solution delivered in a single pass – guessing all 4 digits in one go – being even on the same continent as correct are vanishingly remote, and we learn very little because of the nature of user feedback.
If I deliver 50 changes (e.g., new features) in a single release and ask users “waddaya think?”, I won’t get meaningful feedback about all 50 changes.
Most likely I’ll get general feedback of the “LGTM” or “meh” variety, and maybe some specific feedback about things that stood out. (Bugs in a release tend to overshadow anything else, for example – the proverbial fly in the soup. “Waddaya think of the soup?” “There’s a fly in it!”)
If I deliver ONE change, they’ll probably have something meaningful to say about it. We can at least observe what impact that one change has on user behaviour (e.g., engagement, completing tasks etc).
So we learn faster when we iterate fewer changes into the hands of users at a time. This inevitably forces us to apply the brakes on the creation of code, because we need to wait for feedback, and we need to do that often.
I see many posts here from folks claiming to have generated entire applications in days or even hours using LLM-based coding tools. That’s the equivalent of “guessing all 4 digits at a time using an ‘AI’ 4-digit number generator”. That’s an entire application – hundreds of design decisions – created without any user feedback.
Creating an entire application in a single pass is every bit as “Big Design Up-Front” as wireframing or modeling the whole thing in UML in advance. And assumptions and guesses in your early decisions get compounded in later decisions, piling up uncertainty under a mountain of interconnected complexity. Failure is almost inevitable.
This is another potential solution to the Gorman Paradox.
Where are all the “AI”-generated apps? In the bin.
It just so happens that I train and mentor teams in the technical practices that enable them to learn faster from user and market feedback. I know, right! What are the chances?
And it also just so happens that any Codemanship training course booked by January 31st 2026 is HALF-PRICE.Which is nice.
In my previous blog post talking about the preciseness of software specifications, I used an example from one of my training workshops to illustrate the value in adding clarity when we have a shared understanding of the problem domain.
Now, when many developers see a UML class diagram – especially ones who lived through the age of Big Architecture in the 1990s – they immediately draw connotations of BDUF (Big Design Up-Front). And to be fair, it’s understandable how visual modeling, and UML in particular, gained that reputation, with it’s association with heavyweight model-driven development processes.
But teams who reject visual modeling outright because it’s “not Agile” are throwing the baby out with the BDUF bathwater.
I recounted in my post how providing a basic domain model with the requirements dramatically reduced misinterpretations in the training exercise.
And I’ve seen it have the same effect in real projects, too. As a tech lead I would often take on the responsibility of creating visual models based on our actual code and displaying them prominently in the team space. As the code evolved, I’d regularly update the models so they were a slightly-lagging but mostly accurate reflection of our design.
Domain models – the business concepts and their relationships – have proven to be the most useful things to share, helping to keep the team on the same page in our understanding of what it is we’re actually talking about.
Most importantly, there’s no hint of BDUF in sight. I describe the domain concepts that are pertinent to the test cases we’re working on. The model grows as our software grows, working in vertical slices in tight feedback loops, and never getting ahead of ourselves. We don’t model the entire problem domain, just the concepts we need for the functionality we’re working on.
In this sense, to describe our approach to design as “domain-driven” might be misleading. The domain doesn’t drive the design, user needs do. And user needs dictate what domain concepts our design needs.
Let’s examine the original requirements:
• Add item – add an item to an order. An order item has a product and a quantity. There must be sufficient stock of that product to fulfil the order
• Total including shipping – calculate the total amount payable for the order, including shipping to the address
• Confirm – when an order is confirmed, the stock levels of every product in the items are adjusted by the item quantity, and then the order is added to the sales history.
I’d tackle these one at a time. The domain model for Add Item would look like:
Note that Product price isn’t pertinent to this use case, so it’s not in the model.
When I start working on the next use case, Total Including Shipping, the domain model evolves.
And it evolves again to handle the Confirm use case.
And the level of detail in the model, again, is only what’s pertinent. We do not need to know about the getters and the setters and all that low-level malarkey. We can look at the code to get the details. Otherwise it just becomes visual clutter, making the models less useful as communication tools.
Another activity in which visual modeling can really help is as an aid to collaborative design.
I’ve seen so many times developers or pairs picking up different requirements and going off into their respective corners, designing in isolation and coming up with some serious trainwrecks – duplicated concepts, mismatched architectures, conflicting assumptions, and so on.
It’s the classic “Borrow a video”/”Return a video” situation, where we end up with two versions of the same conceptual model that don’t connect.
It’s especially risky early in the life of a software product, when a basic architecture hasn’t been established yet and everything’s up in the air.
I’ve found it very helpful in those early stages to get everybody around a whiteboard and lay out designs for their specific requirement that’s part of the same model. So if somebody’s already added a Rental class, they add their behaviour around that, and not around their own rental concept.
As the code grows, maintaining a picture of what’s in it – especially domain concepts – gives the team a shared map of what things are and where things go, and a shared vocabulary for discussing and reasoning about problems together.
This is part of the wider discipline of Continuous Architecture, where understanding, planning, evaluating and steering software design is happening throughout the day.
The opposite of Big Design Up-Front.
If your team wants to level up their capability to rapidly, reliably and sustainably evolve working software to meeting changing business needs, check out my live, instructor-led and very hands-on training workshops.
Increasingly, I see people who’ve been struggling with LLM-based coding assistants reaching the conclusion that what’s needed is “better” specifications.
If you were to ask me what might make a specification “better”, I’d probably say:
Less ambiguous – less open to multiple valid interpretations
More complete – fewer gaps where expected system behaviour and other properties are left undefined
More consistent – fewer contradictions (e.g., Requirement #1: “Users can opt in to notifications”, Requirement #77: “By default, notifications must be on”)
Of these three factors, ambiguity is top of my list. It can mask contradictions and paper over gaps. When requirements are ambiguous, that takes us into physicist Wolfgang Pauli’s “not even wrong” territory.
It’s hard to know what the software’s supposed to do, and hard to know when it’s not doing it. This is why so many testers tell me that a large part of their job is figuring out what the requirements were in the first place. (Pro tip: bring them into those discussions.)
An ideal software specification therefore has no ambiguity. It’s not open to multiple interpretations. This enables us to spot gaps and inconsistencies more easily. But more importantly, it enables us to know with certainty when the software doesn’t conform to the specification.
We can never know, of course, that it always conforms to the specification. That would require infinite testing in most cases. But it only needs one test to refute it – and that requires the specification to be refutable.
So I guess when I talk about a “better” specification, I’m talking mostly about refutability.
“Precise”. You Keep Using That Word.
Refutability requires precision. And this is where our natural languages let us down. Try as we might to articulate rules in “precise English” or “precise French” or “precise Cantonese”, these languages haven’t evolved for precision.
Language entropy – the tendency of natural language statements to have multiple valid interpretations, and therefore uncertain meaning – is pretty inescapable.
For completely unambiguous statements, we need a formal language – a language with precisely-defined syntax – with formal semantics that precisely define how that syntax is to be interpreted. Statements made with these can have one – and only one – interpretation. It’s possible to know with certainty when an example contradicts it.
Computer programmers are very familiar with these formal systems. Programming languages are formal languages, and compilers and interpreters endow them with formal semantics – with precise meaning.
I half-joke, when product managers and software designers ask me where they can find good examples of complete software specifications to look on GitHub. It’s full of them.
It’s only half a joke because it’s literally true that program source code is a program specification, not an actual program. It expresses all of the rules of a program in a formal language that are then interpreted into lower-level formal languages like x86 assembly language or machine code. These in turn are interpreted into even lower-level representations, until eventually they’re interpreted by the machine itself – the ultimate arbiter of meaning.
It’s turtles all the way down, and given a specific stack of turtles, meaning – hardware failures notwithstanding – is completely predictable. The same source code, compiled by the same compiler, executed by the same CPU, will produce the same observable behaviour.
So we have a specification that’s refutable and predictable. The same rules will produce the same behaviour every time, and we can know with certainty when examples break the rules.
But, of course, a computer program does what it does. It will always conform to its program specification, expressed in Java or Python or – okay, maybe not JavaScript – or Go. That doesn’t mean it’s the right program.
So we need to take a step back from the program. Sure, it does what it does. But what is it supposed to do?
Remember those turtles? Well, it would be a mistake if we believed the program source code is at the top of the stack. In order to meaningfully test if we wrote the right program code, we need another formal specification (and I use those words most accurately) that describes the desired properties of the program without being part of the program itself.
Let’s think of a simple example. If I have a program that withdraws money from a bank account, and me and my customer agree that withdrawal amounts must be more than zero, and the account needs to have sufficient funds to cover it, we might specify that withdrawals should only happen when that’s true.
In informal language, a precondition of any withdrawal is that the amount must be greater than zero, and the balance must be greater than or equal to the amount being withdrawn. If the withdraw function is invoked when that condition isn’t met, the program is wrong.
To remove any ambiguity, I would wish to express that in a formal language. I could do it in a programming language. I could insert an assertion at the start of the withdraw function that checks the condition and e.g., throws an exception of it’s not satisfied, or halts execution during testing and reports an error.
e.g. in Python “defensive programming” (we can talk in another blog post about what terrible UX design this is – yes, UX design. In the code. Bazinga!)
def withdraw(self, amount):
if amount <= 0:
raise InvalidAmountError()
if self.balance < amount:
raise InsufficientFundsError()
self.balance -= amount
e.g., using inline assertions that are checked during testing
def withdraw(self, amount):
assert amount > 0
assert self.balance >= amount
self.balance -= amount
These approaches are fine, but they’re not a great way to establish what those rules are with our customer in the first place. Are we going to sit down with them and start writing code to capture the requirements?
In the late 1980s, formal languages started to appear specifically with the aim of creating precise external specifications of correct behaviour that aren’t part of the code at all.
The first I used was Z. Z was a notation founded on predicate logic and set theory. Here’s an artist’s impression of a Z specification that ChatGPT hallucinated for me.
Not the most customer-friendly of notations. Other formal specification languages attempted to be more “business-friendly”, like the Object Constraint Language:
context BankAccount::withdraw(amount: Real)
pre: amount > 0
pre: balance >= amount
post: balance = balance@pre - amount
These OCL constraints were designed to extend UML models to make their meaning more precise. I remember being told that it was designed to be used by business people. I found that naivety endearing.
To cut a long story short, while formal specification certainly found a home in the niche of high-integrity and critical systems engineering, that same snow never settled on the plains of business and requirements analysis and everyday software development. We were expecting business stakeholders to become programmers. That rarely works out.
But for a time, I used formal specifications – luckily, my customers were electronics engineers and not marketing executives, so most already had programming experience.
Tests As Specifications
We’d firm up a specification using a combination of Z and the Object Modeling Technique (UML wasn’t a thing then) describing precisely what a feature or a function needed to do.
Then I’d analyse that specification and choose test examples.
BankAccount:: withdraw
Example #1: invalid amount
amount = 0
Outcome:
throws InvalidAmountError
Example #2: valid amount and sufficient funds
amount = 50.0
balance = 50.0
Outcome:
balance = 0.0
Example #3: insufficient funds
amount = 50.01
balance = 50.0
Outcome:
throws InsufficientFundsError
It turned out that business stakeholders can much more easily understand specific examples than general rules expressed in formal languages. So we flipped the script, and explored examples first, and then generalised them to a formal specification.
It was when I first started learning about “test-first design”, one of the practices of the earliest documented versions of Extreme Programming, that the lightbulb moment came.
If we’ve got tests, do we need the formal specifications at all? Maybe we could cut out the middle-man and go straight to the tests?
This often works well – exploring the precise meaning of requirements using test examples – with non-programming stakeholders.
And many people are discovering that including test examples in our prompts helps LLMs match more accurately by reducing the search space of code patterns. It turns out that models are trained on code samples that have been paired with usage examples (tests, basically), so including examples in the prompt gives them more to match on.
So, if you were to ask me what might make a specification for LLM code generation “better”, I’d definitely say “tests”. (And there was you thinking it was the LLM’s job to dream up tests.)
Visualising The Gaps
That helps reduce ambiguity and the risk of misinterpretation, but what of completeness and consistency?
This is where some kind of generalisation is really needed, but it doesn’t have take us down the Z or OCL road. What we really need is a way to visualise the state space of the problem.
One simple technique I’ve used to good effect is a decision table. This helps me to see how the rules of a function or an action map to different outcomes.
Here, I’ve laid out all the possible combinations of conditions and mapped them to specific outcomes. There’s one simplification we can make – if the amount isn’t greater than zero, we don’t care if the account has sufficient funds.
That maps exactly on to my three original test cases, so I’m confident they’re a complete description of this withdraw function.
Mapping it out like this and exploring test cases encourages us to clarify exactly what the customer expects to happen. When the amount is greater than the balance, exactly what should the software do? It forces us and our customers to consider details that probably wouldn’t have come up otherwise.
Other tools we can use to visualise system behaviour and rules include Venn diagrams (have we tested every part of the diagram?), state transition diagrams and state transition tables (have we tested every transition from every state?), logic flow diagrams (have we tested every branch and every path?), and good old-fashioned truth tables – the top half of a decision table.
Isn’t This Testing?
“But, Jason, this sounds awfully like what testers do!”
Yup 🙂
Tests are to specifications what experiments are to hypotheses.
If I say “It should throw an error when the account holder tries to withdraw more than their balance” before any code’s been written to do that, I’m specifying what should happen. Hypothesis.
If I try to withdraw £100 from an account with a balance of £99, then that’s a test of whether the software satisfies it’s specification. It’s a test of what does happen. Experiment.
This is why I strongly recommend teams bring testing experts into requirements discussions. You’re far more likely to get a complete specification when someone in the room is thinking “Ah, but what if A and B, but not C?”
You can, of course, learn to think more like a tester. I did, so it can’t be that hard.
But there’s really no substitute for someone with deep and wide testing experience in the room.
If a function or a feature is straightforward, we can probably figure out what test cases we’d need to cover in our heads. My initial guesses at tests for the withdraw function were pretty good, it turned out.
But when they’re not straightforward, or when the scenario’s high risk, I’ve found these techniques very valuable.
As a bottom line, I’ve found that tests of some kind are table stakes. They’re the least I’ll include in my specification.
Shared Language
Another thing I’ve found that helps to minimise misinterpretations is establishing a shared model of the concepts we’re talking about in our specifications.
In a training exercise I run often, pairs are asked to use Test-Driven Development to create a simple online retail program. They’re given a set of requirements expressed in plain English and the idea is that they agree tests with the customer (one of them plays that role) to pin down what they think the requirements mean.
e.g.
• Add item – add an item to an order. An order item has a product and a quantity. There must be sufficient stock of that product to fulfil the order
• Total including shipping – calculate the total amount payable for the order, including shipping to the address
• Confirm – when an order is confirmed, the stock levels of every product in the items are adjusted by the item quantity, and then the order is added to the sales history.
A couple of years back, I changed the exercise by giving them a “walking skeleton” – essentially a “Hello, world!” project for their tech stack with a dummy test and a CI build script set up and ready to go – to get them started.
And in that project I added a bare-bones domain model – just classes, fields and relationships – that modeled the concepts used in the requirements.
In UML, it looked something like this.
Before I added a domain model, pairs would come up with distinctly different interpretations of the requirements.
With the addition of a domain model, 90% of pairs would land on pretty much the same interpretation. Such is the power of a shared conceptual model of what it is we’re actually talking about.
It doesn’t need to be code or a UML diagram – but some expression in some form we hopefully can all understand of the concepts in our requirements and how they’re related evidently cuts out a lot of misunderstandings.
Precision In UX & UI Design
And, of course, if we’re trying to describe a user interface, pictures can really help there. Wireframes and mock-ups are great, but if we’re trying to describe dynamic behaviour – what happens when I click that button? – I highly recommend storyboards.
A storyboard is just a sequence of snapshots of the UI in specific test scenarios that illustrates what happens with each user interaction. Here’s a great example.
It’s another way of visualising a test case, just from the user’s perspective. In that sense, it can be a powerful tool in user experience design, helping stakeholders to come to a shared understanding of the user’s journey, and potentially revealing problems with the design early.
Precision != BDUF
Before anybody jumps in with accusations of Big Design Up-Front (BDUF), a quick reminder that I would never suggest trying to specify everything, then implement it, then test it, then merge and release it in one pass. I trust you know me better than that.
When clarity’s needed, I have a pretty well-stocked toolbox of techniques for providing it, as and when it’s needed in a highly iterative process delivering working software in thin slices – one feature at a time, one scenario at a time, one outcome at a time, and one example at a time. Solving one problem at a time in tight feedback loops.
Taking small steps with continuous feedback and opportunities to steer is highly compatible with working with LLM-based coding assistants. It’s actually kind of essential, really. Folks talking about specifying e.g., a whole feature “precisely” and then leaving the agent(s) to get on with it are… Well, you probably know what I think. I’ve seen those trains come off the rails so many times.
And with each step, I stay on-task. I’ll rarely, for example, model domain concepts that aren’t involved in the test cases I’m working on. I’m not one of these “First, I model ALL THE THINGS, then I think about the user’s goals” guys.
And using tests as specifications goes hand-in-glove with a test-driven approach to development, which you may have heard I’m quite partial to.
Believe it or not, agility and precision are completely compatible. How precise you’re being, and the size of the steps you’re taking that end in user feedback from working software, are orthogonal concerns. If you look in the original XP books, you’ll even find – gasp! – UML diagrams.
Hopefully you get some ideas about the kinds of things we can include in a specification to make it more precise, more complete and more consistent.
But at the very least, you might begin to rethink just how good your current specifications actually are.
Prompts Aren’t Code and LLMs Aren’t Compilers
One final thought. The formal systems of computer programming – programming languages, compilers, machine code and so on – and the “turtles” in an LLM-based stack are very different.
Prompts – even expressed in formal languages – aren’t code, and LLMs aren’t compilers. They will rarely produce the exact same output given the exact same input. It’s a category mistake to believe otherwise.
This means that no matter how precise our inputs are, they will not be processed precisely or predictably. Expect surprises.
But less ambiguity will – and I’ve tested this a lot – reduce the number of surprises. And refutability gives us a way to spot the brown M&Ms in the output more easily.
You’ve probably heard of “clean code” (and the “clean coder”, and “clean architecture”, and other things Bob Martin has added the word “clean” in front of to get another book out of it).
In this dawning age of “AI”-assisted software development, I’d like to propose clean contexts.
What is a “clean context”? Well, I’m glad you asked.
A clean context:
Addresses one problem – one failing test, one code quality rule, one refactoring etc.
Is small enough to stay inside the model’s effective context limit – which is going to be orders of magnitude smaller than the advertised maximum context
Uses clear and consistent shared language – if you’ve been calling it “sales tax”, don’t suddenly start calling it “VAT”
Clarifies with examples that can be used as success criteria (i.e., tests) – The code samples used in training were paired with usage examples, so it improves matching
Only contains information pertinent to the task – don’t divert the model’s attention (literally)
Only contains accurate information (“ground truth”) – the code and the architecture as it is now (not a bunch of changes back when you asked the tool to summarise it), the test failure message, the mutation testing results and so on. Ground your interactions in reality.
Only contains working code – if the model breaks the code, don’t feed it back to it. It can’t tell broken code from working code, and you’ll pollute the context. Revert and try again. The exception to this is bug fixes, of course. But if the model introduced the bug – git reset –hard
Contains code that doesn’t go outside the model’s data distribution – LLMs famously choke on code that lacks clarity, is overly complex and lacks separation of concerns because it’s far outside the distribution of examples they were trained on. When it comes to gnarly legacy code, I’ve had more success breaking it down myself initially before letting Claude loose on it. Y’know, like how an adult bird chews the food first before feeding it to its chicks.
And remember that a prompt isn’t the entire context. Claude Code and Cursor will use static analysis to determine what source code needs to be added. Context files may be added (e.g., CLAUDE.md). And of course, everything in the conversation- your (or your agent’s) prompts and the model’s responses – are all part of the context. When an LLM “hallucinates”, that becomes part of the context, and the model has no way of determining fact from its own fiction. It’s all just context to a language model.
This is why I purge and then construct a new, task-specific context with each interaction. Many users are reporting how much more accurate LLMs tend to be with a fresh context.
Our goal with a clean context is to minimise ambiguity and the risk of misinterpretation, to minimise attention dilution and context drift, context pollution and context “rot”, and as much as possible, stay within the LLM’s training data distribution.
Basically, we’re aiming to maximise the chances of an accurate prediction from the LLM, and spend less time cleaning up mistakes and digging the tool out of “doom loops”.
Importantly, working in small steps – solving one problem at a time – opens up many more opportunities after each step to get feedback from testing, code review and merging, so clean contexts are highly compatible with much more iterative approaches.
Just as Continuous Delivery enables us to make progress by putting one foot in front of the other, ensuring a working product after every step, we also aim to start every step with a clean context that significantly reduces the risk of a stumble.
Codemanship's Blog 
