For the past several years, AI safety has largely been framed as an alignment problem. How do we ensure that models behave according to human intentions? How do we reduce hallucinations? How do we constrain unwanted behavior? Those are important questions, but they increasingly feel like yesterday’s questions.

Today’s AI systems are no longer isolated models answering prompts. They are becoming teams of collaborating agents that plan, reason, use tools, modify software, and increasingly execute real business workflows. As they become embedded in our organizations, AI safety stops being solely a machine learning problem and starts becoming an organizational one.

The challenge is no longer simply making an AI system produce the right answer. It is deciding who has the authority to decide when the answer matters, and this is what we discuss in this article.

AI Systems Are Beginning to Look Like Organizations

One of the most interesting developments in AI over the last year has been the rise of multi-agent systems. Rather than relying on a single model, today’s most capable systems increasingly divide work among specialized agents that critique one another, share context, and coordinate on complex tasks. This isn’t simply an engineering trick. It works because it mirrors something humans have been refining for centuries: collaboration.

In recent research, my colleagues and I have explored the idea that these systems succeed because they absorb patterns found throughout human organizations. Diverse perspectives, constructive disagreement, independent judgment, shared context, and mechanisms for correction improve both human decision-making and multi-agent AI systems.

Let’s look at why AI agents mirror organizations, and what that means for AI safety.

Organizations Already Know How to Manage Complexity

Long before large language models existed, organizational theorists were wrestling with a similar problem. How do you control a system that is simply too complex for any one individual to understand? One of the most influential answers came from Stafford Beer’s Viable Systems Model, developed in the 1970s. Its insight is remarkably simple.

Leadership can never process every piece of information flowing through an organization. If every operational detail reached the CEO, nothing useful would ever happen. Instead, organizations succeed because authority is distributed downward while information is filtered upward. People closest to the work make local decisions. Only the information requiring intervention reaches leadership.

Beer referred to this filtering process as attenuation. The result is not less control; it is better control. The organization becomes capable of managing complexity precisely because it recognizes that not every decision belongs at the top.

The Missing Piece in Agentic AI

This is where today’s discussion around AI safety often falls short. Organizations are rapidly automating operational work using AI agents. But many assume that judgment can be automated alongside execution. That assumption deserves much more scrutiny.

The operational work may well be delegated to agents. The judgment that determines what matters cannot simply disappear. As I remarked during a recent discussion:

“Accounting is the numbers. Accountability is the human authority and the judgment.”

Those two ideas are increasingly confused. AI is becoming exceptionally good at accounting. It can summarize logs, correlate alerts, generate reports, and execute workflows at extraordinary speed. But accountability remains fundamentally different. Someone must still own the decision. Someone must still carry the authority.

Why Judgment Matters

Another foundational idea helps explain why. The Good Regulator Theorem argues that an effective regulator must contain a model of the system it regulates. Humans naturally do this. Security engineers understand not only software, but also organizations, priorities, risk tolerance, previous incidents, customers, deadlines, and the personalities of the people making decisions.

Those models shape countless judgments every day. Should this finding interrupt production? Does this require executive attention? Is this genuinely critical, or simply noisy? These are not deterministic calculations. They are contextual judgments, and context is what organizations depend upon. The more autonomous AI systems become, the more valuable this judgment becomes, not less.

The Judgment Layer

This is why I increasingly think about AI systems in terms of what I call the judgment layer. The judgment layer is not another scanner. It is not another model. It is the layer responsible for deciding what deserves human attention, how information should be presented, what can safely be automated, and where human authority must remain. During our internal discussions, I summarized it this way:

“We need the separation of the judgment layer, the authority of the AI augmented engineer, and this is how Trent technology delivers that.”

That separation matters. Without it, organizations risk replacing human judgment with automated confidence. With it, AI becomes something much more valuable: an amplifier of human expertise rather than a substitute for it.

Why This Matters for AI Security

Security provides perhaps the clearest example of why this distinction matters. Modern security teams are overwhelmed by data. Every scanner, cloud platform, code repository, compliance framework, and runtime system generates alerts. The problem is rarely a lack of information. The problem is deciding what actually deserves action.

Traditional security tooling largely solves the accounting problem. It produces findings, generates dashboards, and raises alerts. But security engineers still spend most of their time exercising judgment. They determine whether a vulnerability is actually exploitable in the context of a specific architecture. They understand whether a critical vulnerability is isolated behind multiple trust boundaries or whether a seemingly modest finding exposes an entire customer environment.

That judgment cannot be captured by severity scores alone. As organizations increasingly deploy AI agents, this challenge becomes even greater. Agentic systems introduce new forms of complexity, new attack surfaces, and new interactions between models, tools, and workflows. Static rules become increasingly insufficient. What organizations need is not another source of alerts. They need an AI system that understands context well enough to filter, prioritize, explain, and recommend, all while leaving authority exactly where it belongs. With the human security engineer.

Building AI Around Human Authority

This philosophy has shaped how we think about Trent, augmented AI Security Engineer. Our goal has never been to replace experienced security engineers. It has been to make their expertise scalable.

Trent continuously builds context across code, infrastructure, architecture, threat models, documentation, and existing security tooling. It reasons across that context, filters noise, prioritizes genuine risks, recommends remediations, and verifies outcomes.

But the final authority never moves, because the security engineer remains accountable, able to make decisions, disagreeing, even overruling. In other words, Trent, the augmented AI Security Engineer does not replace the judgment layer; instead it strengthens it.

Rather than asking humans to supervise every automated action, Trent aims to ensure that human expertise is applied precisely where it creates the most value: at the moments that genuinely require judgment. That is a fundamentally different vision of AI safety. It is one built not on removing humans from decision-making, but on preserving their authority while allowing AI to absorb the complexity surrounding it.

Rethinking AI Safety

As AI systems become more capable, the limiting factor will not be computation. It will be judgment. The organizations that succeed will not be those that remove humans from the loop. They will be the ones that preserve human authority while using AI to make better decisions, faster. Real-world AI safety needs to be about more than just alignment. It needs to be about ensuring that, in increasingly autonomous systems, judgment remains exactly where it belongs.

a self-adjudicating system cannot seek external arbitration to determine its next state.

I like the idea that some tautologies are useful. And I think this one is. I’ve been applying it to the dynamical system I’m working on called “The Inaccessible Game”. The aim is to create an information theoretic zero-player game which has rules that are emergent. I think the tautology is a route to a selection principle for the game.

Why would a game need to seek external adjudication? Well imagine a situation where the next move in the game could not be resolved within the game’s internal rules. Such a game would either stall, or have to seek external adjudication. And how could such a situation arise? Well, one way is by falling victim to impredicative circularity. A famous example was given by Bertrand Russell and it’s become known as the barber paradox.

Imagine there’s a village and the village has a barber. The barber shaves only those people in the village who don’t shave themselves. The question is, does the barber shave themself?

Could prohibiting impredicative circularity provide a selection principle for a self-adjudicating game?

This question leads to an impredicative circularity, if the barber shaves themselves, then they don’t, so they do and then they don’t etc …

So the tautology above suggests that a self-adjudicating system should ensure that such a situation doesn’t arise. How can we do that?

I call the selection principle the “No Barber Principle”. The idea is that a self-adjudicating system should not have external structure that adjudicates and therefore it should prohibit internal rules that would require such adjudication.

Is this principle useful? My best attempt at showing it can be is given in the latest paper in the inaccessible game series http://arxiv.org/abs/2604.21945. The paper argues that by blocking copying of variables, we can guarantee that Russell’s paradox can’t arise. It builds on a category theoretic explanation due to William Lawvere.¹ An amazing paper from 1969 that connects Russell to Turing, Gödel and Cantor.²

The no-cloning condition is a sufficient condition rather than the necessary one, but I’ve been quite taken with the idea that in a self-adjudicating system copying is prohibited … this gives the work a foundational reason to prefer von Neumann entropy over Shannon.

India’s AI momentum: the future belongs to developers who can balance automation with judgment, security, and system-level thinking.

On the surface of February’s India AI Impact Summit, when listening to founders, engineers, policymakers, students, I didn’t see any fear. I saw excitement and momentum. But if we scratch below the surface, into the nitty-gritty of same founders, engineers, policymakers and students. They are still asking a hard question:

What does all of this mean for the systems we build and the responsibilities we carry?

What was clear from the main summit day is that India is “standing on its own two feet.”

The self-reliance they are developing goes beyond national policy. It applies to their engineering philosophy. We are at a moment where it is easy to outsource thinking, to let AI models decide, to let automation abstract away our judgment. India has realized that sovereignty in AI is not about owning every chip or writing every line of code from scratch. It is about choosing where to intervene. This is a lesson that other countries and companies would do well to learn from.

For developers and security leaders, that means asking:

• Where do we place human oversight?

• Where do we enforce transparency?

• How do we ensure systems remain steerable?

India sees AI as a route to lifting its population out of poverty. That implies that their dependence on technology is inevitable. So, their question is not whether they depend on AI, but  how do they depend on AI?

AI Amplifies Both Capability and Risk

There’s anxiety in developer communities that AI coding tools, automation frameworks, and generative systems will erode hard-earned expertise and make the systems potentially less robust. But AI is not a substitute for individual knowledge, it’s an amplifier of our knowledge.

If you understand distributed systems deeply, AI allows you to tune those systems more effectively. If you understand security architectures deeply, AI helps you imagine stress test a wider range of threat models. If you understand data pipelines deeply, AI helps curate more data and deliver better business intelligence.

But garbage in, garbage out. The danger isn’t that experts become obsolete. The danger is shallow understanding becomes amplified at scale. For developers, that means fundamentals matter more, not less. Poor understanding of security and architecture was always a recipe for downstream technical debt. AI amplification makes these stakes much higher. Architecture matters more than ever before. Security matters more than ever before. Overall, engineering judgment matters more than ever before.

I see this firsthand at the company I co-founded, Trent AI, where we’re designing a system with explicit governance layers, model monitoring, auditability, override mechanisms, and transparent documentation. Control isn’t an afterthought. It’s an architectural decision.

I’d suggest that India’s social calculus is an optimism based on knowledge of the challenges its population faces today. For established economies and incumbent companies, that social calculus pans out differently. The question of what it means feels more like a threat than an opportunity. None of us work at our best when we feel threatened. This means that valid doubts can be clumsily expressed. Skepticism seems obstructive instead of constructive.

However, those in our organizations who are most skeptical are often those with the deepest understanding. It tends to be our more experienced employees who have the deepest domain expertise. Ironically, the challenge of amplification means that assimilating their knowledge has become more important judicious deployment of the AI amplifier. We need to understand the essence of their arguments. We need to translate their objections into improved processes. Resolving this conundrum is urgent in cybersecurity. AI amplification can improve offensive capabilities as much as defensive. Amplified offensive capabilities require amplified defensive capabilities. The attackers are not waiting to operationalise AI.

What Does This All Mean for Younger Developers and Students?

So integrating our experienced workers is key. But what does this all mean for younger developers and students? My advice to them is consistent: don’t abandon your trajectory. Broaden and deepen your knowledge.

AI amplification means we expect more from less. As we delegate tasks to automatic code generation, we expect more code from each developer. This is bringing challenges in integration. The bottleneck on code creation is replaced with a bottleneck on code review. More code makes blunders are harder to spot. We need to be making less blunders per line of code and improving the ability of reviewers to spot blunders in larger code bases. On both the sides of generation and review this requires broader and deeper knowledge from our developers. This implies accelerated learning curves for young developers. We will expect systems-level thinking from them far sooner. That means they will need to improve on both breadth and depth. The human developer has to retain responsibility for the code-agent work they are managing. They will need to learn when to delegate and when to intervene. To be effective all our developers will need to be operating as principal engineers as rapidly as they can manage.

The Future Still Requires Judgment

AI can amplify our human intelligence, but that means it can amplify our ignorance. Modern software engineering requires broader, deeper developers. It requires that we each understand its not just about how we produce code, but how we integrate code. We steer that not just through our technical architectures, but through the social architectures that govern them.

In many ways India is ahead of us. Not because they don’t see the same problems that we do, but because they are aware of the opportunities. Success in this era involves balancing the optimism and skepticism. We can do that by not losing sight of what we are trying to deliver, whether as a country or a company. The advantage India has is that their opportunity is so big, it is impossible to lose sight of.

There’s a pattern I keep seeing, and it’s not new, but AI is about to put it on steroids.

It usually starts with something that looks like progress. Someone builds a spreadsheet, or a script, or now, an agent. It makes their job easier, dramatically easier, and in that moment, everything feels right. We’re automating the boring stuff. We’re being efficient. We’re being modern. But there’s a second act to this story that organizations are very bad at noticing.

The Excel Dead End

One of the clearest examples I’ve seen came from Formula One. About ten years ago, I was consulting with a legendary racing brand’s strategy team, known for their unmatched excellence. But at the time, they’d just missed out on a World Championship, and strategy was blamed. When a new head of strategy came in, I asked what had been happening before. The answer was simple and alarming. The entire race strategy had been run in Excel. Not “supported by” Excel. Run in Excel. One spreadsheet, owned by one person, deeply clever, deeply complex, and completely opaque to everyone else. And here’s the thing: that wasn’t incompetence. It was mastery.

The engineer who built it knew Excel inside out. He’d encoded years of expertise into that file. But because only he understood it, the spreadsheet became inseparable from the person. As long as he owned it, he was irreplaceable, until he wasn’t. That spreadsheet wasn’t just a tool. It was a claim on power.

This pattern shows up everywhere: finance, supply chains, operations. Anywhere smart people are rewarded for making their own lives easier. Excel is an amazing tool, but it’s also a data isolation format. It doesn’t standardize best practice. It doesn’t build institutional memory. It quietly turns individual efficiency into organizational fragility.

I’ve written more about what Excel’s limitations reveal about superficial automation.

Agents Are the New Excel

Now replace “spreadsheet” with “agent.” We’re entering a world where individuals can orchestrate powerful agentic workflows: one agent for banking, one for investments, one for legal advice, another for work tasks. As long as you feel safe giving them access, the productivity gains are enormous.

But ask the same question we never asked with Excel: What happens when those workflows need to be understood, shared, secured, or handed over? Agents will repeat Excel’s trajectory, except faster, and at much larger scale. Someone will build something extraordinary. And no one else will know how it works.

When Local Optimization Breaks the System

The most dangerous failures don’t look dangerous at the time. During COVID in the UK, testing data was flowing through a pipeline that, somewhere along the way, included an Excel spreadsheet, for convenience, for speed. That spreadsheet hit its maximum row limit. For a period of time, new test results simply stopped being counted. National-level decisions were being made on incomplete data, and no one noticed right away.

A single ‘easy’ tool quietly became the weakest link in a national system. This wasn’t malice. It wasn’t stupidity. It was a perfectly rational local decision that had catastrophic systemic consequences, and that’s the pattern.

The Real Problem Isn’t the Black Box

People often say this is a “black box” issue. It’s not, really. Most systems log everything. You can trace what called what. The problem is more subtle: no one designs these systems for future understanding. They’re built to deploy quickly. To solve today’s problem. Not to be debugged by someone else, six months later, when the context has changed.

That was true in Ferrari. It was true in supply chains. It will absolutely be true for agentic AI. The failure isn’t opacity; it’s the absence of responsibility for future comprehension.

Open Loops, Closed Loops, and Why Prediction Fails

There’s another idea that matters here: open loop versus closed loop systems. In open loop systems, you make predictions without changing the world. You can get incredibly good accuracy that way. This is why so many models look amazing in tests.

Closed loop systems are different. Once you deploy them, they start interacting with humans, and humans adapt. Driverless cars are a good example. One car behaves well. A thousand cars behave… differently. And once humans realize there’s no driver inside, they change their behavior too.

Now extrapolate that to markets, security tooling, or organizational decision-making. Deployment changes the environment. Accuracy alone doesn’t survive feedback. This is why things become unpredictable, not because we’re bad at modeling, but because we’re dealing with complex, adaptive systems.

The 80/20 Inversion and Deferred Pain

AI feels like it’s eliminating drudgery. And it is, temporarily. If I can write code ten times faster, I don’t end up with nine-tenths less pain. I end up writing ten times more code. The pain resurfaces elsewhere, often at 2am, debugging something no one fully understands.

AI doesn’t remove the hard parts. It postpones them and concentrates them. This is the real inversion of the cost model. Writing is cheap. Understanding is expensive.

I explored this dynamic in more depth in Speed Isn’t Productivity.

Why Cybersecurity Sees This First

This is why I keep coming back to cybersecurity. Cybersecurity is where deferred consequences show up most clearly. People don’t think about it while they’re building. The costs arrive later, at scale, when they’re hardest to absorb.

And crucially, cybersecurity forces you to think both locally and systemically. About individual actions and how they compose into something larger. Healthy cybersecurity is often just healthy systems thinking, enforced by reality. That’s why this isn’t just a product problem, or an AI problem, or an engineering problem. It’s a societal one.

The Pattern to Watch

If there’s one thing to take away, it’s this:

• Tools that make individuals more powerful

• Without mechanisms for sharing, traceability, and handover

• Will always create hidden risk

Excel taught us that lesson, and we mostly ignored it. Agents are about to teach it again, but much faster. The question isn’t whether this will happen; it’s whether we notice early enough to do something about it.

The demon appears in an 1814 paper. Here is a 1902 English translation.

We may regard the present state of the universe as the effect of its past and the cause of its future. An intellect which at a certain moment would know all forces that set nature in motion, and all positions of all items of which nature is composed, if this intellect were also vast enough to submit these data to analysis, it would embrace in a single formula the movements of the greatest bodies of the universe and those of the tiniest atom; for such an intellect nothing would be uncertain and the future just like the past would be present before its eyes.²

Laplace’s demon working away on the natural laws, data and computation necessary for deterministic prediction

Hawking was writing before the internet, and before the source was widely available, so he may be forgiven for not noticing that the demon is not Laplace’s point. It is a straw man. Within two pages Laplace goes on to explain:

The curve described by a simple molecule of air or vapour is regulated in a manner just as certain as the planetary orbits; the only difference between them is that which comes from our ignorance.³

Laplace’s real point is that we need probability to deal with this ignorance. I’ve come to think about this as Laplace’s gremlin, and wrote about it extensively in The Atomic Human. But as a short cut to the thinking, we might think about the irreducibility of each of the pillars of the demon. Laplace is reducing prediction to a combination of laws, data and compute. The reason that perfect prediction isn’t possible is lack of availability of the laws, the data and the irreducibility of the compute.

The notion of computational irreducibility comes to us from Wolfram,⁴ but I’m tempted to extend it here to the laws and the data. That gives me the following idea. If we can somehow prove that any or all of the components of Laplace’s demon are irreducible, does that give us an impossibility theorem for demon-like intelligences?

I don’t know, but I’m having fun playing with the idea!

The connection between these ideas and the inaccessible game may not yet be clear. But the problem I’ve encountered when looking at other approaches, such as the thermodynamics of information,⁵ is that it’s very difficult to get the primitives of energy and time into the discussion (which I view as resources the demon may have access to) in a way that calibrates them with information (a third form of resource). This is based on an intuition that suggests that … to the extent that intelligence can be defined … it feels like it needs to be done in the context of these three resources … time, energy and information.

The idea in the inaccessible game is to try and better understand how they interrelate by starting with simple information-derived rules and seeing what might be emergent.

The first paper hints at the way energy might emerge. The second paper, that came out on arXiv last week https://arxiv.org/abs/2601.12576 suggests that the game’s clock might be entropy itself. It’s called “The Origin of the Inaccessible Game” and it builds on the first paper by looking at a `natural origin’ for the game where the multi-information is maximised and joint entropy is zero. This triggers some paradoxes and moves the game away from Shannon entropy to von Neumann entropy. But it does allow us to identify a distinguishable trajectory in the information dynamics and associate that trajectory with the internal entropy clock … access to external clocks is prohibited by the information isolation condition.

I’ve been very much enjoying studying these ideas and seeing the direction they take me. But for the moment I want to emphasise that they are only suggestive. I’ve had to learn maths I wasn’t deeply familiar with before and relate it to my own mathematical intuitions (that mainly come from working with information theory and stochastic processes in machine learning). I also have a sense of where I’m trying to take the maths. Together this all feels like a recipe for clumsy errors.

With that in mind I’ve tried to spot such errors, but would welcome suggestions for where I may have gone wrong.

Friday musings from the uncomfortable edge of LLM efficiency.

Over the last year, I’ve found myself circling the same unease in different conversations, with different people, in different contexts. It comes up when talking about large language models, about agents, about productivity, and inevitably about security. On the surface, everything feels faster. And yet, I’m not convinced we’re actually getting more done. What I think is happening is more subtle, and potentially more dangerous.

Speed Isn’t Productivity

The 80/20 Trap of LLMs

We all know the 80/20 rule, even if we don’t always define it consistently. Roughly speaking, 80% of the visible work often takes 20% of the time, while the remaining 20% of the work consumes the other 80% of the effort. LLMs are astonishingly good at compressing that first part.

First drafts. Boilerplate code. Initial reports. Scaffolding. The shell of a house. Suddenly, what used to take days takes hours or minutes, and that feels like productivity. But here’s the problem: that first 80% was never the hard part.

When an LLM produces a draft in minutes, it hasn’t removed the work of finalization. It’s simply pushed it downstream, and this is the hidden cost of making things easier. In some cases, it’s made it harder, because the process of doing the work was also how we learned the landscape, spotted the edge cases, and developed judgment. So, we normalize the numbers and say:

“Look, we’re five times faster.”

But what we’ve actually done is make the easy 80% almost free, while making the hard 20% dangerously invisible, and invisible work is the easiest work to ignore. That’s how a quality deficit starts to form. **

Quality Is the Real Bottleneck

Security, Finalization, and Artificial Indolence

Quality work is cognitively hard and deeply unglamorous. Security. Testing. Reconciliation. Documentation. Explanation. The final pass where you ask, “Is this actually right?” Humans have always avoided this work when they could. And LLMs trained on human behavior, inherit that same indolence.

I see this all the time when I’m doing what I half-jokingly call “vibe science.” I’ll brainstorm ideas with a model, explore references, sketch arguments. It’s powerful and creative. But when I ask it to do something genuinely tedious, like working through a full analytic gradient instead of jumping to a numerical shortcut, it behaves like a grumpy graduate student.

“Do we really need to do that?”

That’s not super intelligence. That’s super-indolence. What’s fascinating is that the model often seems to be reasoning based on how painful a task would be for a human, not on what it could do cheaply as a machine. It has an embedded sense of task difficulty, and it avoids the hard parts unless you force it not to.

I learned the importance of this kind of work long before AI. As a teenager, I worked for an accountant who insisted that the books had to reconcile exactly. Not “close enough.” If the last two digits didn’t match, something was wrong. And every time I thought he was being pedantic, pulling on that thread revealed a real problem.

Later, working offshore as an engineer, the most important part of the job happened after everyone was exhausted, sitting down to finalize the report, annotate it properly, and explain anything that went wrong. That’s the work that separates something that looks done from something that is done.

Security failures are quality failures. Finalization is where professional judgment lives, and skipped quality doesn’t disappear, it compounds. What we often call “slow” work is simply the work that actually matters.

Small Omissions Become Big Failures

From L4 Errors to Q4 Outcomes

At Amazon, I used to describe a particular class of problem as an “L4 to Q4 issue.” An L4 is a junior engineer. Q4 is your fourth-quarter results. An L4 to Q4 problem is when a small, perfectly understandable mistake made low in the system only becomes visible when it shows up in the company’s financials. By the time you notice it, it’s far too late to fix cheaply. This isn’t about blame. Junior engineers will make mistakes, and that’s normal. What matters is whether the system is designed to catch those mistakes early.

Here’s the uncomfortable parallel: LLMs currently behave a lot like junior engineers. They’re fast. They’re confident. They’re often right. And when they’re wrong, the error can propagate silently, especially if we’ve stripped away the quality checks in the name of efficiency.

AI scales output, yes, but it also scales mistakes. If leadership incentives focus only on visible speed, more code, more features, more throughput, while quality work is deferred or handed off to “someone else,” the gap widens. And by the time the consequences show up, security incidents, insurance premiums, regulatory scrutiny, or missed quarters, then organization has already taken on more debt than it can repay.

This isn’t an individual failure; it’s a systems failure. And it’s why guardrails can’t be bolted on at the end. They must be embedded into how we build, from the moment of creation.

The Real Opportunity

Now, I’m not saying this means we should slow down or reject LLMs. Quite the opposite. The real opportunity is to use them to make quality work cheaper earlier, to integrate testing, security, review, and documentation into the same moment we generate the first draft. If we do that, we don’t just move faster; we move better. But if we confuse speed with productivity, we risk building a thousand beautifully rendered shells, houses that look complete, until someone tries to live in them. By then, the cost of fixing what we skipped may already be existential, and that’s the part we need to reckon with now, not in Q4.

Musings while traveling and speaking…

Recently, after speaking at an industry event, it got me thinking about the evolution of agent architectures, these emerging systems of interconnected sub-agents that plan, remember, and act on our behalf. The excitement is palpable. Yet the more I see of them, the more they remind me of an unlikely genre: the self-help book.

Like those books, agents depend on carefully written instructions: prompts, plans, and checklists, that shape their behavior. They don’t “understand” these words as we do, but they follow them with remarkable diligence. It’s the Checklist Manifesto for machines: whatever you do, follow the plan.

That’s powerful, but also deeply fragile and there’s a hidden cost to making things easier. If the entire control of an intelligent system comes down to what’s written in text, the potential for prompt insertion, for subtle manipulation through words, becomes enormous. These vulnerabilities aren’t a niche concern; they’re the new frontier of cybersecurity.

Search on Steroids

When ChatGPT first appeared, some thought it marked the end of search. In fact, the opposite is happening. What we’re building is search on steroids, a world where agents fire off hundreds of queries, across sub-agents and databases, to assemble knowledge in real time.

It reminds me of Xerox’s early dream of the paperless office. When the GUI arrived, paper consumption actually skyrocketed, because easier access to information made us print more. In the same way, these models are amplifying our dependence on the traditional stack, search engines, databases, APIs, and command-line tools.

LLMs are not replacing infrastructure. They’re intensifying its use. Which means that our ground game: our systems, our boundaries, our security, matter more than ever.

Architecture Over Intelligence

We could stop all model innovation today and still have extraordinary power at our fingertips. The bottleneck isn’t the intelligence of the model; it’s the architecture around it. Where is the agent allowed to go? What systems can it access? Who’s responsible when it goes wrong?

These are classical software-engineering questions, now made urgent by the scale and speed of automated reasoning. Building safely means putting creative system designers hand-in-hand with security experts. It means designing not just for intelligence, but for containment.

The Human Context

No matter how capable these systems become, they lack one crucial thing: context. They have no embarrassment, no social stake, no sense of consequence. They don’t learn from mistakes in the way humans do. That absence matters, especially in complex systems where human judgment, intuition, and shared norms prevent cascading failures.

It’s why humans still need to sit at the heart of decision-making. Context is not optional. It’s the glue that holds intelligence, artificial or otherwise, together.

Rethinking Organizations

This transformation isn’t just technical; it’s organizational. Conway’s Law tells us that companies design systems that mirror their communication structures, but LLMs disrupt that. A small team can now build what once took dozens. One engineer can see and understand an entire codebase.

At Trent AI, we’re exploring what it means to be AI-native, where creativity, security, and human capital combine in new ways. It’s not about replacing people with agents; it’s about amplifying human potential while securing the systems that enable it.

Securing Your Agentic Software

We want to help make every software product secure by design, delivering invisible, automatic security for the AI age, empowering teams to innovate freely. Our goal is to ensure that as organizations adopt agentic workflows and autonomous systems, safety and compliance are embedded from the start, not bolted on later.

This is how we see the next evolution of digital architecture: a world where intelligent systems are trusted, accountable, secure, and where developers, designers, and security experts work as one.

The Road Ahead

The future of intelligent systems won’t be defined by who builds the smartest model, but by who builds the most resilient architecture. One that acknowledges the limits of machines and the irreplaceable value of human context.

An advert for a Fuelless car as rendered by ChatGPT after being fed the blogpost

Well, of course that didn’t happen. But I sometimes wonder if what we’re seeing today 100 years later is the modern equivalent of that. This week it was announced that Yann’s leaving the FAIR lab he founded. And one can’t help but wonder if the billions that Zuckerberg is now investing in superintelligence are the modern equivalent of arguing for perpetual motion.¹

Why might this be the case? Well, that’s where entropy comes in. The second law of thermodynamics tells us that entropy always increases. So we can’t have motion without entropy production. How might we make an equivalent statement for the bizarre claims around superintelligence? Some inspiration comes from Maxwell’s demon, an “intelligent” entity which operates against the laws of thermodynamics. The inspiration comes because the demon suggests that for the second law to hold there must be a relationship between the demon’s decisions and thermodynamic entropy. One of the resolutions comes from Landauer’s principle, the notion that erasure of information requires heat dissipation.

I’ve been scratching my head about this for a few years now, but on Monday I shared an arXiv paper² that captures some of the directions I’ve been taking. By considering an axiomatic game where entropy is maximised I’ve been exploring how the dynamics emerge. I admit, it seems a long way from my original motivation, but its just been one of those threads that when you start pulling on it you end up having to go further back. I’m relieved to have the paper out, not because I know it’s all correct, I’m pretty sure I’ve misunderstood lots of things and made some clumsy mathematical errors. But I’m hoping the foundation is still solid enough to build on.

The idea is for an “inaccessible game” which is information-isolated from observers. The assumption is that inside this game the dynamics take the form of “information relaxation” which turns out to be equivalent to constrained entropy production. The nice thing is that resulting dynamics turn out to have a structure that resembles GENERIC, a formalism that is at the heart of modern non-equilibrium thermodynamics.

By building on the similarities to GENERIC one can show an energy-information equivalance that leads to a principle equivalent to Landauer’s.

All this gives me the feeling that the work is on the right track, but it’s clearly still a long way from its destination. My hope is that some friendly folks will be interested enough to take a look and help me tidy it up a bit!

I’ve no doubt that AI technologies will transform our world just as much as the automobile has. But I also have no doubt that the promise of superintelligence is just as silly as the promise of perpetual motion. Maybe the insights from the inaccessible game could provide one way of understanding that.

Originally inspired by a real bug and an even realer problem.

A Morning Routine Interrupted

The other morning, I was doing something ordinary. I walked past my mum’s allotment, checked the rain gauge, and as I do every day, tried to record the reading in Microsoft Excel on my iPhone.

Except this time, it didn’t work.

At first, I thought it was just me, maybe I’d done something wrong. But then someone sent me a link to The Register, confirming that the recent Excel deployment on iPhone and iPad had indeed broken for many users.

It was frustrating, yes — but also revealing. Because buried in that article was a claim that caught my attention: some major tech companies are now saying that up to 30% of their code is being written through AI-assisted or “vibe” coding.

That made me pause…and this is in addition to the recent claim discussed in our previous article, that a quarter of Y Combinator startups say 95%+ of their code is AI-generated.

The Hidden Cost of Superficial Automation

What if what we’re seeing with Excel isn’t just a bug, but an early sign of a much deeper issue?

I like to call it superficial automation.

Superficial automation happens when we assume the job our people are doing is simply to write code, file tickets, or push deployments. So, we replace that visible layer with an LLM or an AI assistant.

But the truth is, those people are doing something much deeper:

• Understanding edge cases

• Spotting subtle issues before they ship

• Intuitively preventing cascading failures

• Protecting customers from exactly this kind of problem

By automating the surface, the typing, the syntax, the visible effort, we risk losing the invisible expertise that keeps systems stable.

When the Surface Looks Smooth, but the Foundation Shifts

If Big Tech is already deep into AI-generated coding, we are seeing the first cracks, and AI agents will repeat Excel’s trajectory, except faster and at larger scale. Bugs slip through. Security checks get skipped. Context that once lived in engineers’ heads now gets abstracted away by an LLM that doesn’t truly understand consequences. And because these companies often limit themselves to their own in-house tools, their constraints can blind them to risks developing outside their ecosystem.

That’s why this is an opportunity for smaller, more agile teams; the ones who adopt AI thoughtfully, not superficially. They can design workflows that enhance judgment, not replace it.

Structural Automation: The Better Path Forward

This is why I’m excited at the work we’re doing at Trent AI, what I’d call structural automation, an automation that goes deep, not just wide.

Rather than using AI to replace human insight, we use it to amplify it, especially in cybersecurity and software reliability. The goal isn’t just faster code; it’s better, safer systems.

Because what’s the point of saving developer hours if the result is a broken Excel for millions of users?

A Raindrop Reminder

For now, my rainfall reading lives in my head. (24.4 mm, for the record) But that small inconvenience reminded me of something bigger, that progress in AI isn’t about replacing human intelligence; it’s about learning where it truly matters.

If we automate only the surface, we’ll keep running into hidden cracks. If we automate with understanding, we can build secure systems that stand the test of time.

In Goethe’s poem The Sorcerer’s Apprentice, a young sorcerer learns one of their master’s spells and deploys it to assist in his chores. Unfortunately, he cannot control it. The poem was popularised by Paul Dukas’s musical composition, in 1940 Disney used the composition in the film Fantasia. Mickey Mouse plays the role of the hapless apprentice who deploys the spell but cannot control the results.

When it comes to our software systems, the same thing is happening. The Harvard Law professor, Jonathan Zittrain calls the phenomenon intellectual debt. In intellectual debt, like the sorcerer’s apprentice, a software system is created but it cannot be explained or controlled by its creator. The phenomenon comes from the difficulty of building and maintaining large software systems: the complexity of the whole is too much for any individual to understand, so it is decomposed into parts. Each part is constructed by a smaller team. The approach is known as separation of concerns, but it has the unfortunate side effect that no individual understands how the whole system works. When this goes wrong, the effects can be devastating. We saw this in the recent Horizon scandal, where neither the Post Office or Fujitsu were able to control the accounting system they had deployed, and we saw it when Facebook’s systems were manipulated to spread misinformation in the 2016 US election.

When Disney’s Fantasia was released, the philosopher Karl Popper was in exile in New Zealand. He wrote The Open Society and its Enemies when his hometown of Vienna was under Nazi rule. The book defends the political system of liberal democracy against totalitarianism. For Popper, the open society is one characterised by institutions that can engage in the pragmatic pursuit of solutions to social and political problems. Those institutions are underpinned by professions: lawyers, the accountants, civil administrators. To Popper these “piecemeal social engineers” are the pragmatic solution to how a society solves political and social problems.

In 2019 Mark Zuckerberg wrote an op-ed in the Washington Post calling for regulation of social media. He was repeating the realisation of Goethe’s apprentice, he had released a technology he couldn’t control. In Goethe’s poem, the master returns, “Besen, besen! Seid’s gewesen” he calls, and order is restored, but back in the real world the role of the master is played by Popper’s open society. Unfortunately, those institutions have been undermined by the very spell that these modern apprentices have cast. The book, the letter, the ledger, each of these has been supplanted in our modern information infrastructure by the computer. The modern scribes are software engineers, and their guilds are the big tech companies. Facebook’s motto was to “move fast and break things”. Their software engineers have done precisely that and the apprentice has robbed the master of his powers. This is a desperate situation, and it’s getting worse. The latest to reprise the apprentice’s role are Sam Altman and OpenAI who dream of “general intelligence” solutions to societal problems which OpenAI will develop, deploy, and control. Popper worried about the threat of totalitarianism to our open societies, today’s threat is a form of information totalitarianism which emerges from the way these companies undermine our institutions.

So, what to do? If we value the open society, we must expose these modern apprentices to scrutiny. Open development processes are critical here, Fujitsu would never have got away with their claims of system robustness for Horizon if the software they were using was open source. We also need to re-empower the professions, equipping them with the resources they need to have a critical understanding of these technologies. That involves redesigning the interface between these systems and the humans that empowers civil administrators to query how they are functioning. This is a mammoth task. But recent technological developments, such as code generation from large language models, offer a route to delivery.

The open society is characterised by institutions that collaborate with each otherin the pragmatic pursuit of solutions to social problems. The large tech companies that have thrived because of the open society are now putting that ecosystem in peril. For the open society to survive it needs to embrace open development practices that enable Popper’s piecemeal social engineers to come back together and chant “Besen, besen! Seid’s gewesen.” Before it is too late for the master to step in and deal with the mess the apprentice has made.

With thanks to Sylvie Delacroix and Jess Montgomery for comments and suggestions.