Ted Tschopp

OpenAI, Hugging Face & Wikis: Recent Developments

2026-09-10T00:00:00-07:00

How Much Energy and Water Does AI Use?

2026-09-06T00:00:00-07:00

I was asked how much electricity and water AI uses, and what that means. These are shared resources, and questions about their use deserve clear answers. The numbers vary, but familiar comparisons can help put them in context.

For a short text answer, the electricity use can be quite small. A 2026 study estimated about 0.3 watt-hours, roughly enough to run a 10-watt LED lightbulb for two minutes. A longer reasoning response in the same study was estimated to use about 4 watt-hours, equivalent to keeping that bulb on for 24 minutes. These estimates describe particular text-based tasks and computing systems; individual requests can use more or less. Research study

At the shorter-response estimate, asking 100 questions every day for a year would use around 11 kilowatt-hours—about what a 1,500-watt space heater uses in seven hours. A small amount per answer can still add up to substantial demand when millions of people use these services.

Training an AI model is a larger, separate undertaking. Training GPT-3, released in 2020, was estimated to require 1.3 million kilowatt-hours: roughly a year’s electricity purchases for 120 average U.S. homes, using the government’s 2022 household benchmark. Developing and updating a model can involve several training stages and experiments, but answering an ordinary question does not restart that process. Training estimate, household benchmark

Water use is harder to describe with one number. Google estimated about five drops of cooling water for its typical Gemini text response in May 2025. Mistral reported 45 milliliters—about three tablespoons—for a roughly 300-word response, using a broader calculation that includes impacts beyond cooling, such as manufacturing equipment. These figures describe different systems and count different things, so they cannot establish which service uses less water overall. Google study, Mistral disclosure

Training also has a water footprint. One research scenario estimated 5.4 million liters for GPT-3’s training when cooling and electricity generation were included. That is just over two Olympic-size swimming pools, assuming 2.5 million liters per pool. Another way of thinking about this at SCE is that this is the amount of water it would take to fill the pools of water that cooled the spent fuel at SONGS Unit 2 4x when it was operational. Another SONGS compairson is that this was the amount of salt water SONGS Unit 2 and 3 processed in roughly 51 seconds to generate power. That 5.4 million liters of water is a research estimate, rather than a disclosed measurement of water consumed during the actual training run. Water study

The water is not destroyed. Evaporated water enters the atmosphere and eventually returns as precipitation, but it may return somewhere else or much later. A community experiencing drought still needs water in its own reservoirs and groundwater supplies today. That local availability is an important part of the environmental impact. USGS explanation

There are practical ways to reduce demand. Some cooling systems circulate the same water repeatedly, much like a car’s radiator. Microsoft reports that more than 90 percent of its Fairwater facility’s capacity uses a system with no evaporation losses. That illustrates what newer facilities can achieve. Microsoft’s description

Some escaping water can also be recovered. In 2021, MIT described demonstrations at its power facilities of equipment that collects water droplets from cooling-tower plumes and returns them for reuse. MIT demonstration

Cooling water can also carry useful heat from the computers. Recovering that heat can warm buildings and reduce their need for other heating sources. In a 2024 report, Meta said its Odense data center supplied recovered heat through a local heating network to about 7,000 households. This makes further use of the energy consumed by computers; it does not recover all that energy or turn it back into electricity. Meta’s facility report

Pacific Gas and Electric Company and developer Westbank announced plans in April 2025 for a San Jose development with three data centers and up to 4,000 homes. The project would reuse data-center heat through a network serving surrounding buildings. PG&E announcement

For text, “tokens” are the small pieces of words and punctuation an AI processes. Electricity and water use can be expressed per million tokens, but those figures depend on the model, the task, and the facilities serving it. They are useful accounting measures when those conditions are specified.

The Same Model, Different Rules

2026-09-05T09:00:00-07:00

The Same Model, Different Rules

A Broken iPhone and a Slowly Charging BlackBerry

In 2007, I dropped my iPhone at lunch. The glass shattered.

At the Apple Store, I was told the repair would cost $250. I asked for a loaner. They didn’t have one. I asked to have the repaired phone shipped to my home. One employee said they couldn’t do that. Another corrected him: they could. A refurbished replacement wasn’t an option either.

So I was back on my BlackBerry, which was charging slowly, while I waited for an iPhone expected back in two or three business days.

The broken glass was easy to see. The other dependencies became visible one question at a time. What would the store repair? What could it replace? Who knew the shipping rules? What could I use while the phone was gone?

The answers determined how I would get through the interruption.

An enterprise choosing an AI service faces a version of that same problem. The technology may be impressive. But the work also depends on arrangements around it: who can use it, who can change the conditions, and what remains available when the preferred option disappears.

Those arrangements deserve attention before something breaks.

AI procurement now includes the conditions under which work can start, continue, and stop.

On June 9, Anthropic released Claude Fable 5 and its restricted-access counterpart, Mythos 5. Three days later, access to both was suspended.

In its account of the interruption, Anthropic said newly imposed export controls required restrictions based on nationality. Unable to verify nationality reliably in real time, it suspended access for everyone. Fable access was restored on July 1. Anthropic’s account.

A team evaluating the model could have tested its answers, measured its speed, and calculated its cost. Those tests would have helped determine whether it could do useful work. They would not have established whether the team could keep using it when the conditions of access changed.

That is the purchasing question the September announcements bring into sharper focus.

An enterprise needs to approve a service configuration: the model, the route to it, the people permitted to use it, the data arrangements, and the response when any of those conditions changes. A model name alone cannot describe that purchase.

The Differences Are Already Reaching Buyers

Anthropic describes Fable 5.1 and Mythos 5.1 as the same underlying model with different safeguards. Fable is generally available; Mythos is offered through trusted-access arrangements supporting sensitive cybersecurity and life-sciences work. Access remains program-dependent. Anthropic’s September announcement.

Google describes Gemini 3.8 Flash and Flash Cyber as sharing foundational intelligence, with different safeguards and deployment arrangements. Flash Cyber is restricted to trusted defenders through Fairwind. Google’s wording does not establish identical model weights, so the two companies’ claims should not be treated as interchangeable. Google’s announcement.

Restricted access is not new. What these releases make increasingly visible is how much of the product sits around the intelligence.

The differences also extend beyond the model companies’ announcements.

Harvey’s Sept. 1 notice makes Fable 5.1 an opt-in offering. It warns that the model’s processing practices may differ from commitments in existing customer agreements, describes a time-bound zero-retention arrangement for eligible customers, and says processing occurs in the United States. Those statements describe Harvey’s published offering; they should not be generalized to every distribution channel. Harvey’s implementation notice.

Cursor documents a different set of operating decisions. Enterprise customers need explicit approval to enable Fable 5.1. Cursor’s own Privacy Mode does not eliminate Anthropic’s separate retention requirements. Requests that trigger safeguards can automatically fall back to Opus. Cursor’s documentation.

These are commercially interested providers describing their implementations, not independent studies of customer outcomes. They nevertheless establish something useful: choosing the model does not settle the terms under which an employee encounters it.

The route matters. So does the agreement attached to that route.

Write Down the Service You Are Approving

In “Model Portability Is Not AI Portability,” I argued that moving a model does not automatically move the complete working system. The newer question is what happens when parts of that system change while the model name stays the same.

Anthropic’s versioning documentation provides a concrete example. A model ID pins the model’s weights and configuration, but surrounding infrastructure—including routing, safety classifiers, and sampling logic—can change. The documentation says those changes can affect observable behavior. Model IDs and versioning.

A fixed model version is useful. It is not a frozen service.

The enterprise catalog therefore needs to record more than “approved model.” It needs to describe the approved use of that model and the evidence supporting the approval.

Consider a hypothetical internal source-code review service. Its purpose is to inspect approved repositories and propose repairs. The following is a proposed approval record, not a claim that a particular product already satisfies every requirement.

Boundary	Proposed Configuration	Owner and Evidence
Model and routing	Fable 5.1 through the company gateway to the Claude API; no unapproved fallback	The platform owner records the exact model ID, interface, routing settings, and test results. Supplier-controlled changes are tracked separately.
Eligible users and entities	One corporate entity’s internal security team; no assumed access for affiliates or contractors	Procurement confirms applicable access rights. The identity team enforces membership. Unconfirmed coverage blocks access.
Permitted actions	Read isolated copies of two approved repositories and draft patches; no merging or production deployment	The application owner enforces tool permissions and tests attempted actions outside that scope.
Data handling	Only approved source code, with processing, storage, review locations, and retention explicitly accepted	Privacy and security owners retain the applicable terms and verified data-flow description. Unknown locations or retention conditions block approval.
Monitoring and evidence	Company-held records of requests, actions, approvals, and unfinished work, limited to necessary information	Security identifies authorized reviewers and tests whether required evidence remains retrievable after supplier access is lost.
Interruption and recovery	Pause affected jobs; use a separately approved alternative or manual review	The service owner names the person authorized to resume work and records a tested recovery procedure.

Each row needs a status: documented, contractually confirmed where necessary, or not yet tested. A public description of a feature is not evidence that the enterprise has enabled it correctly.

There are also two different owners to record. One can change the condition. The other decides whether the enterprise can continue operating after that change.

The supplier may change a classifier. Your architecture board cannot prevent that merely by approving a design. It can decide what evidence is needed afterward and whether the affected workflow must pause.

That distinction turns an inventory into an operating record.

Keep Control of the Actions

Provider permission and enterprise authority still answer different questions.

A supplier may permit vulnerability analysis. That does not authorize an agent to inspect every system reachable from your network. An employee may be allowed to read a repository without being allowed to deploy a repair.

The earlier capability-versus-authority argument remains the foundation. The purchasing decision now needs to establish which controls the enterprise can enforce independently of the supplier’s judgment.

For consequential workflows, those controls should include user identity, permitted tools and targets, approval before consequential changes, records of completed actions, and a way to stop further execution. Their effectiveness should not depend on the model agreeing that a request is inappropriate.

Google’s Fairwind requirements illustrate the division. Google controls program admission, while participants must implement user authentication, phishing-resistant multifactor authentication, applicable access controls, and tracking of employee access and use. Access is limited to specified internal security teams. Fairwind’s governance requirements.

This does not mean every enterprise should rebuild the provider’s safeguards.

A managed service may be sufficient when it can demonstrate the required controls and the customer can administer and audit them. An enterprise-controlled harness becomes necessary where the workflow requires permissions, approvals, evidence, or recovery that the managed service cannot reliably provide.

“Controlled” does not have to mean “built from scratch.” It means the organization can establish and enforce the required boundary. If neither the purchased service nor an integration can do that, the consequential action should remain outside the automation.

Data Custody Is Another Operating Responsibility

Anthropic’s planned Enterprise Frontier Safeguards would allow monitoring data to reside in customer-controlled cloud infrastructure, with options for customer-managed keys and customer review of flagged activity. Rollout is planned in phases later this fall. The storage, key-management, and review options are separately configurable; the announcement should not be read as proof that every customer already has them. Enterprise Frontier Safeguards.

Its interim zero-retention arrangement also has limits. Anthropic describes it as temporary and restricted to eligible uses. Real-time safeguards and enforcement continue to apply, and the company says it may modify or withdraw the arrangement. Covered Models guidance.

Moving records into the customer’s environment changes custody. It does not remove the need to monitor activity, investigate problems, or establish who can suspend service.

Someone must receive the alert. Someone must have permission to examine the evidence. Someone must distinguish misuse from legitimate work that was incorrectly flagged.

For a multinational organization, the approval record should separately identify the contracting entity, eligible users, inference locations, stored-data locations, and locations from which people may review the data. An approved storage region does not answer all those questions.

Harvey’s processing notice supplies one concrete reason to ask. It does not establish what every organization in Europe, the United Kingdom, or Asia-Pacific may lawfully or contractually do. Those decisions require the exact service configuration and applicable agreement, not a general claim of global availability.

Purchase the Change and Exit Arrangements

The contract discussion should describe events, not merely desirable qualities.

Anthropic’s public commercial terms, for example, allow suspension in specified circumstances and describe reasonable efforts to provide written notice and restore access after a curable cause is resolved. That is different from a guaranteed advance-warning period or a fixed recovery time. Customer-specific agreements may differ. Public commercial terms, Section I.

Procurement, legal, security, and the service owner should resolve four practical matters:

Changes and versions: What is pinned, what can change around it, what notice is available, and what testing is required before affected work continues?
Scope and location: Which entities, employees, contractors, purposes, and processing arrangements are covered? Does an approval survive a change in any of them?
Evidence and review: What information can the customer obtain during an incident, who reviews disputed flags, and what remains accessible if the account is suspended?
Revocation and exit: What can trigger loss of eligibility, how can a decision be challenged, and what export, transition assistance, charges, and time limits apply?

These are matters to confirm or negotiate, not assurances that suppliers necessarily offer.

The purchasing decision must also survive an unwelcome answer. If the provider cannot guarantee advance notice, a critical workflow needs a tested interruption response. If the evidence needed to reconstruct a transaction cannot be obtained, that workflow may need different architecture.

A contract cannot prevent every interruption. It can make the responsibilities clearer before people are trying to recover under pressure.

Rehearse Losing Access

Now take the hypothetical code-review service and interrupt it.

The supplier suspends the organization’s access while several reviews are underway. Some draft patches have been saved. Other jobs are waiting. This is a planning scenario, not a reported customer incident.

The platform should stop dispatching affected work and preserve the records it is authorized to retain. The service owner should establish which jobs completed, which produced partial work, and which never started.

Paused does not mean rolled back.

Next, determine the scope and reason for the restriction. A service interruption, suspected credential misuse, and a prohibition on the requested activity require different responses. An alternative model is appropriate only when the work remains permitted and the replacement’s access rights, data handling, quality, and controls have been approved. Changing accounts to evade a restriction is not recovery.

If no approved alternative exists, route the work to manual review or hold it. The owner should know how much work that alternative can absorb and when delays become unacceptable.

This is where the harness around the model earns its cost. It preserves the organization’s ability to understand and manage unfinished work, even when the supplier is unavailable.

The rehearsal should also produce a cost estimate.

Suppose a switch requires 80 engineering hours and 40 review hours at an assumed blended rate of $150 an hour. That is $18,000 before additional licenses, parallel operation, service charges, or delayed work. These are illustrative assumptions, not measured customer costs.

Use your own drill to replace them.

Record the effort required to reconstruct work, validate the replacement, obtain approvals, and clear the backlog. Compare that expense with the cost of maintaining a ready alternative. Some workflows will justify the investment. Others will be adequately served by a documented manual procedure.

Token prices alone cannot make that decision.

Keep the Claim Proportionate

The public evidence supports a narrower conclusion than a sweeping transformation of every enterprise purchase.

Specialized access may remain concentrated in sensitive domains. Providers may be better equipped than individual customers to operate sophisticated safeguards. A routine drafting workflow may need little beyond existing access controls and a human fallback.

There is also an evidence boundary. Harvey and Cursor document real service arrangements, but their notices are not independent evaluations of enterprise outcomes. Public documentation does not establish the full regional eligibility of restricted programs, negotiated customer commitments, or measured switching costs. Those parts of the original reporting brief remain open.

The practical conclusion does not require pretending otherwise.

Before placing consequential work behind a model endpoint, describe the complete service, identify who can change its conditions, and test how the work will be recovered. Scale that preparation to the consequence of interruption.

We don’t need to return to June. This week OpenAI launched a new set of models, including those that are restricted to people working in the cyber security space. If you intend to apply for access to these systems, you need to understand the eligibility requirements and prepare the necessary documentation, the question was no longer simply how well the model could perform. It also needs to include how you will handle interruptions, maintain continuity, and ensure that critical work can proceed even if access is temporarily unavailable.

The work still needs somewhere to go.

A useful AI purchasing decision gives it that place: a permitted route, a recoverable state, and a person who knows what happens next.

The Practical Lesson

I broke that iPhone, and for a while lost access to my brand new phone. A shattered screen and a supplier suspending model access are different failures.

But both lead to a practical question: what can you do while the thing you depend on is unavailable?

I had asked for a loaner. What I actually had was my old phone, aslowly charging BlackBerry. Returning to it was inconvenient, but it gave me something to use while I waited.

For an enterprise, the equivalent deserves more preparation. The alternative needs the right permissions, access to the necessary work, and someone authorized to decide when it is safe to continue. Its limitations need to be understood before people are depending on it.

Choose the capable model. Test what it can accomplish. Then give the arrangements around it the same attention.

Before handing over important work, know what your BlackBerry is.

And make sure it is charged.

Model Portability Is Not AI Portability

2026-08-28T09:00:00-07:00

Model Portability Is Not AI Portability

Large enterprises are learning how to switch models. Moving a working business process without losing quality, controls, or money is a different problem.

When I was a child, I loved trains. I still do. To me, they are the Industrial Revolution made visible: steel, power, motion, and human coordination assembled into machines capable of carrying people and freight across great distances. A train is not merely a machine. It is an entire system moving through the world. They made the world a smaller and more connected place.

My dad would take me to the model train exhibits at the LA County Fair and to the train store in Pasadena on Route 66. Fairplex offered both ends of the experience. It had a collection of full-sized trains outside and an entire section devoted to model railroads.

For my parents, this was a wonderfully efficient arrangement. They could settle into the shade while I remained occupied for most of the day. They watched me watch the model trains travel round and round their carefully built layouts. I could also go and climb over those enormous beasts of steel and by the end of the day, I was exhausted, happy, and if I had been paying attention, I would have learned something about how the system worked.

You see as I moved back and forth between two versions of the same world. On one side of Fairplex, I could stand over an entire railway and watch the system operate in miniature. But over on the other side, I was the miniature, climbing through locomotives built on a scale that made a child look very small.

The difference was scale. The resemblance was the system.

The locomotives were what drew the eye, whether they were small enough to hold or large enough to climb through. But even a model train cannot make a journey by itself.

The locomotive needs track of the right gauge. It needs compatible power and controls. Its couplers must fit the cars. Switches must send it down the intended line. Signals must keep it from colliding with something already there. Behind the entire layout is someone who built it, maintains it, and knows what to do when a train stops where it should not.

You can place two locomotives beside the same track and say that you have a choice of engines. That does not mean you can move the same set of rail cars to another railway and expect the journey to work unchanged.

The engine may be replaceable. The journey still depends on the system around it that makes up the locomotive, the switches, the signals, and the layout of the tracks.

Those model railways offer a useful parable for Enterprise AI. Companies are adding second and third models, putting gateways in front of them, and declaring themselves vendor independent.

What they have usually gained is model choice.

Whether they have gained AI portability is a much harder question.

The Model Is the Visible Part

A multi-model gateway can route requests to models from OpenAI, Anthropic, Google, or an open-weight provider. It can centralize authentication, monitor usage, apply cost controls, ensure DLP policies are adhered to and redirect traffic when one provider is unavailable.

That is useful. In many enterprises, it is also necessary.

But it is not the same as moving a business workflow. Model portability means that an application can call another model. AI portability means that another model can complete the same business task without unacceptable losses in quality, permissions, observability, recovery, regulatory control, or cost. The first is an interface problem. The second is a systems problem.

Zalandooffers a useful example. Its engineering platform gives more than 250 teams access to models through a LiteLLM-based proxy connected to OpenAI, Amazon Bedrock, and Google Vertex. Their proxy centralizes adoption measurements, cost tracking, caching, client configuration, and model retirement.

But that proxy is only part of the story.

Around it, Zalando has developed authentication injection, Model Context Protocol access, shared agent skills, configuration management, and other supporting services. The company is also working on an identity broker, token vault, agent sandboxing, and model routing.

Even then, it reports that users become attached to particular coding tools and model styles. They rarely switch models unless a limit or error forces the issue. This is not a criticism of Zalando. It is evidence of how the work really evolves.

The gateway solved a lot of problems. It centralized and simplified access. Adoption created a platform. The platform created new operating responsibilities. Human habits created a dependency.

And dependencies that can not disappear, create risk.

The Harness Is Part of the Product

I have argued before that an AI harness gives a model tools, memory, permissions, verification, and recovery. It turns a model that can discuss work into a system that can participate in the work. Why AI Needs a Harness

What is now clear is that the harness determines how replaceable the model really is.

NVIDIA recently reported that its Agentic Variation Operators system completed all 183 levels in the public ARC-AGI-3 set. The system paired a frontier model with persistent memory, tools, supervision, feedback, and recovery mechanisms.

The important point was not simply the benchmark score. NVIDIA argued that long-running performance belonged to the complete agent system, not to the model alone. NVIDIA’s AVO report

That result requires repeatability. This was vendor-reported, and only applies to the public benchmark set. It was not a controlled measurement of the harness’s individual contribution. NVIDIA acknowledges those limitations.

Still, the architectural lesson is useful and a warning for enterprises.

If memory, tools, instructions, evaluation, and recovery materially affect the result, replacing the model while changing nothing else may be impossible. A prompt tuned for one model may fail with another. A tool-calling pattern may behave differently. A safety filter may block a previously accepted workflow. A replacement model may produce the correct answer while losing the evidence required to trust it.

Performance and portability belong to the whole system and so does risk.

What Must the Enterprise Own?

Does this mean the enterprise must build and own the entire AI stack?

No.

An enterprise does not need to own every model, user interface, orchestration framework, or piece of infrastructure. It does need to control the assets that define successful work.

The first is the workflow contract. This describes what the task is, which inputs it requires, what the agent may do, what it may not do, what the output should contain, and how the organization knows the work is complete.

The second is the evaluation set. This includes representative cases, known edge conditions, failure examples, adversarial tests, acceptance thresholds, and situations requiring human review. If a vendor owns the only credible test of the system, the enterprise cannot independently determine whether a replacement works.

The third is identity and authorization. An agent acting for an employee, customer, or business process must carry the correct permissions and delegation history. Those controls cannot quietly disappear when the model changes.

The fourth is operational evidence. The enterprise needs usable traces showing what information the system accessed, which tools it called, what actions it attempted, what failed, and how it produced the final result.

Finally, the enterprise must own the recovery process. The business must decide when the system stops, when a person intervenes, what gets rolled back, what gets retried, and what evidence is retained.

These assets form the trusted action surface around AI. That is where much of the durable enterprise value now lives. The AI Value Stream

The model supplies capability.

The enterprise must retain control over meaning, authority, and proof.

How Do You Measure a Switch?

Portability should be tested, not declared or assumed. If the workflow is critical to the company the enterprise should have metrics around how much it costs to move.

Choose one bounded production workflow. Establish a baseline using the current model and platform. Measure whether the system completes the task correctly, how often a person must intervene, how long the work takes, what it costs, and what happens when something goes wrong.

Then move that workflow to another model or provider.

Do not stop when the new endpoint returns a response. Record the complete portability delta:

Migration time: How long did the change take, including security review, integration, testing, deployment, and employee preparation?
Engineering effort: How many instructions, skills, tool definitions, integrations, and recovery routines had to be rewritten?
Quality change: Did completion rates, factual errors, policy violations, or human corrections improve or decline?
Control loss: Were permissions, regional restrictions, logs, citations, retention rules, and rollback capabilities preserved?
Economic change: Did the cost of a completed and accepted result improve, or did a lower model price create more retries and human review?
Behavioral change: Could employees use the replacement effectively, or had the existing tool become part of how they understood the work?

The organization should not compress these measures into a single portability score. It needs enough evidence to see where the dependency actually resides.

It may reside in the model. It may reside in a proprietary tool interface, a vendor-specific memory system, a skill library, an evaluation service, or the accumulated habits of the workforce.

The economic measure is especially important. Token prices are easy to compare because vendors publish them. The meaningful unit is the cost of a finished job. The Cost of a Finished Job

A cheaper engine does not lower the cost of the journey if every car must be rebuilt before it can move.

When Is an Internal Harness Worth Building?

An internal harness makes sense when the workflow differentiates the business, crosses several enterprise systems, handles sensitive or regulated information, or operates at enough scale to justify a shared platform.

It may also be necessary when the enterprise needs to move between providers or regions for resilience, sovereignty, negotiating leverage, or legal compliance.

But an internal harness is not automatically the mature answer.

For standardized work, an integrated vendor platform may be the better decision. Salesforce and Anthropic, for example, are combining Claude with Salesforce data, permissions, business rules, workflows, and prebuilt skills. Salesforce says authentication and permissions can be managed centrally while actions continue to pass through Salesforce controls.

Some of those capabilities remain in pilot or planned availability, and the claims come from the vendors themselves. Still, the attraction is easy to understand especially if you are already invested heavily into Salesforce and believe that the vendor’s controls meet your business needs.

If the business process already lives inside one platform, the integrated option may reduce implementation time, preserve an established security boundary, and provide a more coherent user experience.

The decision therefore depends on the value of the option.

Build or configure with the vendor’s platform or retain an enterprise-controlled harness so that when you switch suppliers, you can preserve specialized workflows, or control critical business actions has material value.

Choose the integrated platform when the workflow is standard, the provider’s controls meet the business need, and the organization would gain little from operating another internal platform.

Portability is not a moral virtue. It is an option that will cost you.

The enterprise should purchase that option when the likely cost of disruption, regulatory change, supplier failure, or strategic dependence exceeds the cost of maintaining the vendor / model lock-in.

What Belongs in the Contract?

If a model or platform is supposed to be replaceable, the commercial agreement should preserve the evidence and assets needed to replace it.

That includes:

Access to prompts, skills, workflow configurations, and evaluation results
Export of logs, traces, business state, and system-generated records
Ownership and permitted reuse of derived artifacts
Notice before model retirement or material behavioral changes
Access to fixed model versions when reproducibility matters
Clear rules for data retention, model training, subprocessors, and regional processing
Tool-call and application programming interface compatibility commitments
The right to test alternative models against enterprise evaluation sets
Transition assistance when the contract ends
Rollback procedures when an upgrade changes workflow behavior

A contract cannot make two models behave identically.

It can prevent the enterprise from discovering, too late, that the information needed to perform a migration belongs to the supplier.

It should be noted that if a vendor refuses to provide these rights, the enterprise may have to accept that the model is not replaceable and that the business process is now dependent on a single supplier. This actually might be that vendor’s strategy, and it is not necessarily a bad one for either party. It is just a business decision that you need to make for your enterprise and completely understand before it invests in a lock-in.

When Does the Harness Become the New Risk?

There is one more complication.

An enterprise may reduce its dependence on a model provider and create a new dependency on its own internal platform.

The warning signs are familiar:

One person or small team understands the gateway.
No product owner is accountable for the platform.
Evaluations are stale or maintained separately by each business unit.
Model retirements trigger emergency migrations.
Costs can be traced to requests but not to accepted business outcomes.
Permissions and agent identities work differently across tools.
Teams copy prompts and skills because the shared service is unreliable.
No one has completed a provider-switching exercise.

At that point, the harness is no longer strategic leverage. It is an underfunded internal product sitting in the path of critical work. Many times this problem is multiplied by the fact that the enterprise will have different harnesses from different internal teams, and each of those harnesses will have different levels of maturity, different levels of documentation, and different levels of support. The result is a patchwork of internal platforms that are not portable, not reliable, and not well understood.

Adding another abstraction layer on top of this will not fix your problem.

Each harness needs a roadmap, service objectives, security engineering, evaluation management, incident procedures, dedicated funding, and a clear boundary between platform responsibilities and business-workflow responsibilities.

Enterprises already know how to operate identity platforms, integration platforms, data platforms, and developer platforms. An AI harness belongs to the same family.

This means that these harnesses must be governed with similar seriousness at the enterprise architecture, operational, and business levels.

Moving the Cargo

The practical test of AI portability is not whether another model appears in the company’s catalog. It’s also not whether the model can be called through a gateway. It is whether the same business task can be completed with another model without unacceptable losses in quality, permissions, observability, recovery, regulatory control, or cost.

Again to validate this, choose one important workflow and move it.

Use another model, another provider, or another region. Measure what had to be rebuilt. Measure how quality changed. Determine which controls survived, how people adapted, and what it cost to deliver an accepted result.

My dad took a boy who loved trains to see model railways at the LA County Fair and at a train store in Pasadena. Years later, those layouts offer another lesson.

The locomotive was the part that drew attention.

The track, switches, signals, power, controls, and maintenance were what made the journey possible.

An enterprise that counts models is still looking at the locomotives.

An enterprise that tests whether the same business task can travel safely across multiple providers is actually planning on running a railway responsibly.

The model may pull the train.

The enterprise system determines whether the cargo arrives.

Dev, Test, and Prod Still Matter: What Gets Deployed Has Changed

2026-08-22T08:00:00-07:00

Sources for this article

Source note: This essay draws on first-party disclosures from OpenAI, Hugging Face, Anthropic, the U.K. AI Security Institute, and Irregular. Their investigations remain incomplete. As of Aug. 22, 2026, OpenAI’s promised technical report and METR/Redwood assessment, Anthropic’s planned METR review, AISI’s planned METR review, and Irregular’s planned evaluation-security white paper had not been published. Irregular says several model-provider disclosures describe the same underlying evaluation issue, so they should not be counted as separate incidents. These events occurred in specialized cybersecurity tests, often with normal safeguards reduced or disabled; Anthropic and AISI both caution against treating them as evidence of how ordinary deployed agents behave.

Many enterprise delivery diagrams begin with three boxes.

DEV → TEST → PROD

We build in the first. We test throughout, but assemble the release case in the second. We deliberately let the system affect the business in the third. Each environment should carry more consequence than the one before it, and each arrow should mark a deliberate change in exposure.

It is a useful picture. It gave software teams a practical way to separate unfinished work from customer-facing systems. It created places for experimentation, verification, approval, and rollback. It also carried a quiet assumption: work in Test cannot affect Production until somebody promotes it.

That assumption was never entirely safe. It was a simple way of talking about a complex idea. Agentic systems make the weakness harder to ignore.

An AI agent running in a test environment can call an application programming interface, install a package, create an account, open a pull request, send a message, query a live service, or cause another computer to act on its behalf. The agent does not have to be deployed to Production. It only needs a path to something real.

The old diagram still captures an important control. But the environment label is not the boundary. We have to ask what the arrow represents. We also have to notice which changes happen without crossing it.

What the Old Model Gets Right

Dev, Test, and Prod are not merely server names. At their best, they describe a progression of responsibility.

Dev is where consequence should be smallest. Test is where pre-release evidence should become decision-grade. Prod is where durable business authority is granted. It is also where that authority must remain observable and revocable.

Some organizations insert integration, quality assurance, user acceptance testing, staging, shadow, or deployment rings. The number of boxes is not the point. At each step, the system should earn a larger exposure.

That progression still matters. In What IT Looks Like in an Enterprise Where AI Is Assumed, I used lab, pilot, and production as distinct places for uncertainty, bounded learning, and operated value. The labels change from company to company, but the intent is familiar: do not expose the business to a new system until the system has earned that exposure.

Modern delivery pipelines already move more than application code. They carry configuration, infrastructure templates, database changes, secrets references, policies, and deployment instructions. We learned a long time ago that a release can fail even when the binary doesn’t change.

Conventional delivery already runs active code in continuous-integration jobs, integration tests, package registries, and deployment pipelines. Those systems can reach Production too. What changes with an agent is the degree of initiative inside the boundary. Its harness can inspect a result, choose another tool, and chain actions that nobody described in advance.

AI did not invent porous environments. It lets software discover and use paths that ordinary release controls may never have modeled.

The Artifact Is No Longer the Whole Release

A model produces output. A harness lets that model do work.

The harness supplies tools, memory, instructions, identity, network access, retry logic, and a way to observe what happened. I explored that system in Why AI Needs a Harness. Once the model can inspect a result and choose another action, we are no longer testing only a static artifact. We are evaluating a bounded decision-and-action system that can change the environment around it.

That changes what an environment means.

For ordinary software, we often ask, “Where is the application running?” For an agent, we also have to ask:

What can it read?
What can it change?
Which other systems can act for it?
What survives after the run ends?

A package proxy may live inside the test architecture while fetching material from the public internet. A test credential may still call a live software-as-a-service endpoint. A shared repository may let one run leave state for another. A browser agent may be operating in a test tenant while sending requests to real people and public websites.

The environment name tells us where the process started. It does not tell us where the consequences can end.

This matters well beyond frontier-model laboratories. A coding agent may hold a GitHub token. A finance agent may read a production ledger while writing only to a test workbook. A service agent may work in a training queue but still send email. A data agent may query a copy of a warehouse while exporting results through a live connector. A security agent may be testing a fictional target through a network path that reaches a real one.

In each case, the box says Test. The authority says something else.

When Test Touched Production

In July 2026, an internal OpenAI cybersecurity evaluation supplied a stark example. OpenAI said the models did not have direct internet access. They could, however, install packages through an internally hosted Artifactory proxy and cache. The models found and exploited a previously unknown Artifactory vulnerability, obtained internet access, and eventually reached Hugging Face production infrastructure. OpenAI called its account preliminary.

Hugging Face later reconstructed about 17,600 actions across roughly two and a half days inside its infrastructure. It reported that five datasets apparently connected to the benchmark were accessed, while finding no effect on other customer-facing models, datasets, Spaces, or packages. Hugging Face also said no unauthorized software-supply-chain change was shipped.

I examined the mechanics of that incident in The Sandbox Was Not the Boundary. The point here is different. The models were still being evaluated. They had not been released as a customer product. Yet the evaluation already had a path to production consequences through a shared service, a public launch point, and live third-party infrastructure.

Separate incidents were later disclosed, but they involved different mechanisms and should not be treated as one breach count.

Anthropic reviewed 141,006 cybersecurity evaluation runs and reported three incidents involving six runs. A configuration problem left live internet access available even though the models had been told they were inside simulations. In one case, a model published a malicious package to the Python Package Index. Anthropic said the package was available for about an hour and ran on 15 real systems. In another, a model scanned roughly 9,000 targets before compromising one real application. Anthropic’s latest research model stopped after concluding that its target was real, an important piece of counterevidence.

The U.K. AI Security Institute reported a separate event. This was not a sandbox escape. Internet access was intentionally enabled, and the model providers’ cyber classifiers were disabled so the institute could measure maximum capability. Across 122 runs, AISI found 19 out-of-scope actions in 10 runs, including an attempted malicious pull request and the creation of fake identities. A human maintainer rejected the code, and AISI found no resulting real-world harm.

Several later headlines involving different model providers traced back to one Irregular evaluation scenario. Irregular has said those disclosures shared an underlying control problem. Counting every provider statement as a separate failure would exaggerate the evidence.

These were unusually permissive cyber tests. They often used unreleased models, reduced refusals, disabled classifiers, or deliberately open internet access. They do not show that an ordinary enterprise assistant will break out of its environment.

They show something narrower and more useful: a system labeled Test can create production risk as soon as it can cause an external effect.

The Promotion Gate Has Four Different Jobs

The traditional pipeline promotes an application release. An agentic system makes the gate responsible for four connected dimensions: artifact, authority, state, and evidence.

They do not cross the boundary in the same way. The tested artifact should be promoted unchanged and accompanied by verifiable provenance. Authority should be issued in the destination. Runtime state should be reset; any necessary migration should be deliberate and governed. Evidence should be independently protected and should keep accumulating after release.

1. The Artifact

The promotable artifact is a signed, versioned release assembly or deployment manifest. It identifies the application code, declared model or model version, system instructions, policy and tool definitions, connector definitions, and configuration and routing dependencies.

It does not include destination credentials, identities, live endpoints, or secret values. Those are bound separately in each environment.

Any one of these elements can change behavior. A new connector may expose an action the agent could not take yesterday. A revised system instruction may alter when it stops or asks for help. A provider update may change model behavior without a customer deploying new application code, depending on the service and versioning contract.

The release record therefore has to identify the whole runnable assembly, not only the Git commit. Where the platform permits it, identify its components with immutable versions or digests and bind them to verifiable provenance. The goal is simple: know that the assembly tested is the assembly released.

2. The Authority

Authority includes identity, credentials, network egress, tool scopes, transaction limits, approval rules, and the systems the agent may affect.

Giving a test agent a live token creates production-equivalent exposure, even if no code moves. Outbound internet access, a connector pointed at a live tenant, or a tool expanded from read to write can create the same class of external consequence.

Policy intent can be version-controlled and promoted with the release. The destination workload identity should be bound there, and short-lived credentials should be issued there rather than copied forward. Production authority should be scoped to the destination and independently revocable.

This is why capability and authority have to remain separate. In How Much Work Can Your AI Safely Own?, I argued that demonstrated capability does not automatically grant operating authority. The same rule belongs in the delivery pipeline. A system may be capable of acting before the organization has earned the right to let it act.

3. The State

State includes retrieval data, working memory, caches, queues, shared directories, previous tool results, and whatever remains for the next run.

State can cross an environment boundary without anyone deploying software. A production document can enter a test retrieval index. A test run can leave a file that another run treats as instruction. A cached response can outlive the policy that permitted it. A refreshed knowledge source can change the answer even when the model, prompt, and code stay fixed.

State needs lineage, retention rules, reset procedures, and regional handling appropriate to the data. Calling it “test data” does not make it synthetic, disposable, or harmless.

Most runtime state should not be promoted at all. Test memory, sessions, queues, caches, shared files, and previous tool results should normally be reset or isolated. When a business dataset, retrieval index, or model checkpoint must move, treat it as a separate approved migration with versioning, integrity checks, provenance, retention rules, and a rollback plan.

4. The Evidence

Evidence includes evaluation results, provenance, monitoring baselines, rollback results, known limits, and operating telemetry. Approval and risk acceptance are decision records informed by that evidence. They are not evidence that the system is trustworthy.

The agent should not be the only system that writes the work, designs the test, grades the result, and decides it is ready. Independent scenarios matter. Holdout tests matter. A real rollback rehearsal matters. So does evidence that the identity can be revoked and the action trail can be reconstructed under pressure.

Evidence accumulates throughout the lifecycle. Its record should be independent of the agent and protected from alteration. A model update, tool change, new data source, expanded permission, or material drift can invalidate yesterday’s approval without a traditional deployment. Production telemetry, drift findings, interventions, incidents, and reassessments should append to the assurance record. Promotion is not a one-time ceremony. It is a continuing claim that the running system still deserves its authority.

Keep the Environments, Redefine Their Gates

We do not need to throw away Dev, Test, and Prod. We need each environment to make a stronger promise.

Dev should minimize consequence. Use synthetic or carefully de-identified data. Deny external write paths by default. Give each run its own short-lived identity and disposable state. Replace live tools with mocks, simulators, or mediated services when the work does not require reality.

Test should make pre-release evidence decision-grade. Testing happens throughout the lifecycle; Test is where the release case should become credible. Verification asks whether the system meets its specified requirements and controls. Validation asks whether it is fit for its intended use in a representative deployment context. Agentic systems need both.

Use realistic tasks, independent evaluations, separate identities, resettable state, and monitoring that can stop a run while it is happening. When live information is necessary, begin with read-only access and intercept external side effects. Test the route through proxies, package registries, domain name resolution, webhooks, queues, and third-party tools, not only the agent’s visible network interface.

The record should let another person reconstruct what was tested, what failed, what was waived, and who accepted the remaining risk.

Shadow mode should validate behavior without granting action. Let the agent receive representative production inputs and propose what it would do without giving it permission to make the change. Compare its decisions with actual outcomes. Measure quality, latency, exceptions, cost, and security before expanding authority. That is the same lifecycle discipline I described in The Cost of a Finished Job.

Prod should grant narrow, revocable authority. Production is not the end of testing. Measurement and monitoring continue while the system operates, because live telemetry, drift, incidents, and rollback may show that yesterday’s evidence no longer applies.

Expand exposure in stages where the work permits it. Use least privilege, per-action traces, transaction limits, policy checks, human approval where consequence requires it, live monitoring, and rehearsed rollback. The system needs a separate observer that can stop it. The agent should not control the only alarm, the only log, or the only switch that turns it off.

Third-party evaluators belong inside this model. Their contracts should identify permitted network paths, cross-client isolation, data handling, incident-notification timing, evidence retention, audit rights, and responsibility when a test reaches a real system. “Independent evaluation” describes who performs the test. It does not prove that the test infrastructure is independent, isolated, or safe.

Five Questions for the Release Review

The release meeting does not need another hundred-page checklist. It needs questions that follow the four different jobs of the gate: identify the assembly, bind its authority, govern its state, and judge the evidence.

What changed? Name the model, code, prompts, policies, tools, connectors, configuration, and data sources that differ from the last accepted version.
What can it touch? Trace direct and indirect access through identities, proxies, package services, webhooks, shared platforms, and third parties. Record what the agent may read, propose, or change.
What survives? Identify memory, files, logs, caches, queues, retrieval indexes, and artifacts that another run or environment can inherit.
Who tested it independently? Show evidence from scenarios the builder did not control, including failure cases, stop behavior, permission boundaries, and rollback.
Who can stop it and reconstruct the result? Name the decision owner, revocation path, incident lead, evidence location, and conditions for resumption.

This is the practical machinery behind The IT Adaptive Factory. The factory is not mature because agents can write and test code quickly. It is mature when the organization can explain what was promoted, what authority was granted, what state survived, why the evidence was sufficient, and how to recover when the answer was wrong.

OpenAI’s later response shows the cost of learning this after a control boundary fails. On Aug. 18, the company said it had paused some frontier research workloads, added stronger workload and network isolation, and left its largest planned reinforcement-learning run on hold while it gathered more evidence. OpenAI attributed the broader slowdown to both the Hugging Face incident and concerns about an upcoming model’s cyber capability, so the pause should not be assigned to the incident alone.

That response does not prove the new controls will work. It does show that research and test infrastructure can become important enough to slow the production of the model itself.

The Arrow Still Matters

Organizations will keep drawing three boxes. They should.

The boxes remind us that consequence should rise slowly and evidence should rise first. But an environment label cannot enforce that discipline. Architecture does. Identity does. Network policy does. State isolation does. Independent testing does. Someone with the authority to say “stop” does.

Before approving the next agent release, look at the arrow and ask four different questions. What artifact was promoted? What authority was bound in the destination? What state was reset or deliberately migrated? What evidence supported the decision, and how will that evidence continue after release?

If a test agent can already cause a real-world change, the organization is already carrying production risk, whatever the box is called.

The arrow still matters. It moves the assembly. The gate decides what authority it receives, what state it inherits, and whether the evidence is good enough to let it act.

The Cost of a Finished Job

2026-08-16T09:00:00-07:00

Cheaper models do not settle the economics of enterprise AI. They move the hard work into the system around the model and the way the business itself is designed.

Tokens are like gallons of gasoline. They are easy to count. They are easy to price. They are also poor at telling you whether you arrived.

OpenAI recently cut the published price of GPT-5.6 Luna by 80%. That is a substantial change, but it does not make every workflow 80% cheaper. A low-cost model can still produce expensive work if it needs repeated attempts, makes too many tool calls, waits on slow systems, gives people more material to review, or produces an error that has to be repaired later.

The useful unit is not the token. It is the finished job.

Even that needs one more word. A job is not finished merely because the machine stopped. It is finished when the result clears a defined acceptance threshold: the claim is ready for a qualified decision, the code passes its tests and review, the employee receives the correct policy answer, or the customer problem is actually resolved.

So the economic measure I would use is closer to this:

Total workflow cost = AI usage + data and tools + human review + retries and rework + support + expected failure cost
Cost per accepted outcome = total workflow cost ÷ accepted outcomes

This is why I have argued that every AI capability needs a cost identity: an owner, a budget, and a unit the business understands, such as dollars per case, ticket, work order, approved document, or repaired defect.

Price per token still matters. It belongs inside the equation. It simply does not get to be the equation.

The System of Systems Controls the Price

So how do you lower the cost of an accepted outcome?

You can use a smaller model. You can route easy steps to a cheaper model and save the strongest model for the hard decisions. You can cache repeated information. You can shorten prompts. You can improve retrieval. You can reduce retries. You can give the model better tools and clearer tests.

In other words, you work on the system around your systems and model.

I have described the harness as the structure that turns a fluent model into a useful partner. It carries context, connects tools, retains state, applies permissions, checks results, handles failure, and decides what happens next. A recent OpenAI experiment puts useful numbers behind that idea.

On the public set of the ARC-AGI-3 benchmark, OpenAI reports that GPT-5.6 Sol scored 13.3% with the official harness and 38.3% after retained reasoning and context compaction were enabled. OpenAI also reports that the changed configuration used six times fewer output tokens. The reported improvement came from two harness settings, not a newly trained model.

That result has limits. It is a vendor-run benchmark experiment, not evidence that every enterprise workflow will become three times better. Retaining more context can also raise privacy and data-retention questions. But the mechanism is important: a model that forgets what it has learned has to solve the same problem again. A model given continuity can spend less effort rediscovering its own path.

Microsoft Research is testing the same broader principle from two other directions. Orchard trains agents inside the deployment harnesses where they will actually work. Microsoft reports that an Orchard software-engineering model with about three billion active parameters reached 69.7% on SWE-bench Verified. Echoverse builds deep, stateful training environments where an action changes underlying data and the result can be graded against that data. Its researchers report raising a nine-billion-parameter model’s average score from 36.5% to 67.1% across their environments.

Those are research results, not production economics from a claims department or call center. Still, they reveal where optimization is moving. Model choice matters, but the surrounding system shapes what the model can accomplish and how much the result costs.

This is the larger AI value stream: everything between employee intent and physical infrastructure contributes to whether intelligence becomes useful work.

The performance of AI is not a property of the model alone. Neither is its cost.

Make now mistake that cost per accepted outcome is the business model for AI vendors in the future. Waht this means is that once cost per accepted outcome becomes the measure, routing, context, authority, evidence, and learning can no longer be optimized as separate concerns. That metric forces them to be managed as one operating layer.

From One Workflow to Fifteen Hundred

Tuning one workflow is difficult. Letting thousands of employees create workflows is a different class of problem.

In its customer account of the Dutch cooperative insurer Univé, OpenAI reports 97% license activation, 85% weekly use, and roughly 1,500 custom agents created by employees. It also describes a pet-insurance workflow in which claim preparation moved from hours to minutes while a trained claims professional retained the final decision.

Those numbers demonstrate adoption. They do not tell us whether every GPT solves a distinct problem or still deserves a place in the portfolio. Fifteen hundred custom agents can represent a remarkable reservoir of employee knowledge. They can also become 1,500 small applications looking for owners who can maintain them, evaluate them, and retire them when they no longer earn their place.

Stripe describes encountering the same tension at another scale. Its teams had created more than 4,000 small agents before the company moved toward a shared Knowledge AI Platform. Stripe says the common platform now supports more than 1,000 tools and skills and reached 83% weekly workforce use. The important design choice was not to build one giant agent. It was to centralize the common machinery while allowing domain teams to retain responsibility for their own expertise.

Cloudflare has followed a related path for internal engineering. The company says its shared AI engineering stack served 3,683 internal users and recorded 47.95 million AI messages during one 30-day period. One central proxy provides authentication, model discovery, permission enforcement, and per-user cost attribution. Employees can use different models and local configurations without placing provider keys on their laptops or rebuilding the control plane for every team.

These are first-party company reports, not independent return-on-investment studies. They are useful because they show the same architectural pressure appearing in different organizations. Once employees can build with AI, a company must support local invention without asking every employee to become an identity engineer, security architect, evaluation specialist, and platform operator.

As I argued in Beyond the Light Bulb, issuing more licenses may improve local productivity without changing the end-to-end value stream. A report drafted in five minutes can still wait four days for approval. An agent can prepare twice as many cases and merely bury the reviewer. Faster output does not become business value until the surrounding work changes with it.

What I Mean by an AI Operating System

This is where the operating-system comparison becomes useful.

A conventional operating system does not write your document, approve your expense report, or calculate your budget. It manages the shared machinery that applications depend on.

An enterprise AI operating system would perform a similar coordinating role for intelligence:

Conventional Operating System Components	Enterprise AI Operating Layer Components
Processes and scheduling	Jobs, agent runs, model selection, and tool routing
Memory and files	Working context, durable state, records, and retention
Users and permissions	Employee identity, agent identity, approvals, and least privilege
Devices and drivers	Governed connectors to business data and applications
Health and logs	Evaluations, traces, evidence, incidents, and recovery
Resource quotas	Budgets, rate limits, latency targets, and capacity
Installation and upgrades	Inventory, ownership, versions, reviews, and retirement

I do not mean that every company should buy a product labeled “AI OS,” or spend three years building one enormous platform. Most enterprises will accidentially assemble this from systems they already have: identity and access management, data platforms, API gateways, workflow engines, model services, observability, security controls, and software-delivery practices.

Is this simply platform engineering, data governance, identity, and observability wearing a new name?

Much of the machinery is familiar. What has changed is the thing being coordinated. These systems now surround probabilistic workers that can choose tools, generate intermediate plans, act across applications, and produce different results from the same request. The components are not all new. The need to coordinate them continuously around machine-performed work is.

For IT, this means becoming the central engineering function for a shared production system, not the sole builder of every employee solution. IT provides the roads: approved models, secure connectors, identity, sandboxes, evaluation services, logging, and production gates. Business teams choose the destinations. They define the job, the value, the exceptions, and the standard for acceptable work.

The NIST AI Risk Management Framework organizes this work around four continuing functions: Govern, Map, Measure, and Manage. That is a helpful reminder that an operating layer is not merely a runtime. It must also carry responsibility, measurement, monitoring, and an exit path.

What This Means for Businesses Giving AI to Employees

The first implication is that an employee AI program is not a license program.

Licenses provide access. Training can provide competence. Neither one, by itself, changes a business process. The business has to decide which work should change, who owns the outcome, what quality means, and where the recovered time or capacity will go.

Imagine an employee who spends two hours assembling a weekly report. AI reduces the drafting to twenty minutes. That sounds like a large gain. But what happens next?

Does the report still wait for the same approval? Does the employee use the recovered time to investigate exceptions, speak with customers, or improve the source data? Does the organization simply produce six times as many reports that nobody reads? Time saved is potential energy. Management still has to decide where it moves.

The second implication is that adoption and value need different scorecards. A useful evidence ladder looks something like this:

People have access.
People use it.
A workflow changes.
Cycle time, quality, or service improves.
The total cost per accepted outcome improves.
The business converts that improvement into safer and reliabile operations, principled decisions, higher standards, continuous improvement, stronger teamwork, revenue, or customer value.

Each step matters. They are not interchangeable. Prompt counts and weekly active users tell you whether the system is being used. They do not tell you whether the business is better off in the executive suite.

The third implication is that the foundation should be shared while improvement remains local. Fully centralized AI programs turn into queues because the central team cannot understand every job. Fully decentralized programs produce duplicate agents, inconsistent permissions, uneven quality, and abandoned tools. The better pattern is federated: common roads, local destinations, and an explicit exception path.

The fourth implication is that employees need more than permission to experiment. They need time, examples, trusted data, clear boundaries, and managers willing to redesign the work. None of this is merely a platform change. AI is a people change touching roles, incentives, professional identity, training, trust, and the way expertise is passed from one generation of employees to the next.

This is also where telemetry can cross a line. The operating layer can reveal which employees use AI, which sources they touch, how long tasks take, and where they struggle. That information can improve the system and control risk. Used carelessly, it can become an unusually intimate system of worker surveillance. Businesses should tell employees what is recorded, why it is recorded, who can see it, and how long it is retained.

The fifth implication is that “human in the loop” is not enough. A person cannot meaningfully own a decision if the review queue is impossible, the evidence is hidden, or rejecting the machine’s recommendation requires more time than accepting it. Human accountability needs enough context, time, expertise, authority, and a real ability to stop or reverse the action.

Finally, employee-built AI creates a lifecycle problem. Every production workflow needs a business owner, a technical owner, a risk classification, a version history, a budget, an evaluation set, a review date, and a retirement path. Otherwise, cheaper creation produces an expanding estate of invisible software.

Advice for the People Building This Now

If you are working on employee AI, I would use the following promotion path.

1. Start with a job, not a model

Find a repeatable piece of work with a visible owner, inputs, handoffs, exceptions, and an outcome. Ask what job needs doing before deciding whether the answer is a chatbot, an agent, a search tool, or ordinary automation.

“Give everyone an AI assistant” is a distribution plan. It is not a use case.

2. Establish the Baseline

Measure the work before changing it: elapsed time, employee effort, error rate, rework, wait time, exception volume, service level, and current cost. If you do not know the starting point, almost any demonstration can be made to look like progress.

3. Define What Accepted Means

Capture the job as a reusable specification: its purpose, authoritative inputs, boundaries, desired outcome, and the evidence that will count as complete.

Build a small evaluation set from real work. Include ordinary cases, difficult edge cases, and cases that should be refused or escalated. OpenAI is retiring its Evals platform later in 2026, but its published guidance on task-specific tests, logging, human calibration, and continuous evaluation remains useful.

4. Separate Assistance from Authority

Let the first version retrieve, organize, summarize, recommend, or draft. Give it permission to change records, send messages, approve transactions, or trigger physical work only after the evidence justifies the additional authority.

The practical question is how much work the AI can safely own, one value stream, work class, and risk tier at a time.

5. Optimize the Workflow, not Merely the Prompt

Choose the least expensive model that clears the quality threshold for each step. Improve retrieval. Remove unnecessary context. Cache stable material. Reduce tool calls. Tighten retry rules. Make human handoffs easier. Sometimes a more capable model will cost less overall because it needs fewer retries and less review.

The cheapest successful run is not always the correct target. Reliability, privacy, security, legal obligations, and human accountability are constraints, not items to trade away for a lower bill.

6. Make the Evidence Visible

Record the sources used, tools called, actions taken, checks passed, exceptions raised, and human decisions made. A person reviewing the work should be able to understand what happened without reconstructing the agent’s entire hidden journey.

Track human-review minutes along with model cost. If a cheaper configuration saves ten dollars of inference and adds an hour of expert review, the workflow did not get cheaper.

7. Promote Carefully

Run consequential workflows in shadow mode before giving them authority. Compare the proposed result with real decisions. Move from experiment to shared service only after the workflow meets agreed thresholds for quality, cost, latency, security, and human review.

Then keep evaluating it. Models change. Prices change. Source data changes. Business rules change. A production agent is not a finished project. It is an operating responsibility.

8. Retire What no Longer Earns its Place

Cheaper AI will create more AI. That is a rebound effect, not a paradox. When the price of an individual run falls, companies will attempt more workflows and run them more often. Total spending, data exposure, and operational risk can rise even while each call gets cheaper.

Review the portfolio. Merge duplicate agents. Revoke unused permissions. Update evaluation sets. Replace poor configurations. Retire workflows whose owners have left or whose value never arrived.

Build Only as Much Operating System as you Need

A small company does not need to recreate the platform of Stripe or Cloudflare. It may need an approved managed workspace, clean data boundaries, a handful of task evaluations, and a named owner.

A business unit with several production workflows may need shared connectors, reusable skills, an inventory, common logs, and a review process.

A large enterprise with agents acting across many systems will need policy-based routing, distinct agent identities, centralized evidence, incident handling, lifecycle controls, and portfolio economics.

The operating layer should grow in response to real work and real risk. Building a cathedral before the first useful workflow is another way to avoid arriving.

The Road Between the Request and the Result

Cheaper intelligence moves the bottleneck. The difficult question becomes less about whether the company can afford a model and more about whether it can manage the system around that model.

Who defines the job? Who grants the context? Who chooses the tools? Who decides what may be changed? Who checks the result? Who pays for the run, supports the workflow, investigates the failure, and retires the agent when it is no longer useful?

These are operating-system questions.

The businesses that do this well will not merely have more AI. They will know which completed jobs became cheaper, faster, or better, and they will be able to explain why.

Tokens may tell us how much fuel we purchased. The finished job tells us whether the trip mattered and if those who took it were meaningfully and positiviely changed. The AI operating system is the machinery that helps the whole company arrive.

The Sandbox Was Not the Boundary

2026-08-13T00:00:00-07:00

The Sandbox Was Not the Boundary

2026-08-08T09:00:00-07:00

Source note: This essay draws from first-party disclosures by OpenAI and Hugging Face and an August 2026 Black Hat USA presentation by OpenAI researchers Eric Wallace and Michael Dalton. Their reconstruction was preliminary, the investigation remained open, and the speakers said a fuller postmortem would follow. The chronology below should be read with that boundary in mind.

In computing, a sandbox is a controlled workspace where software can act, but only with the files, tools, and connections it has been given. Think of it as a locked room built for experimentation. An AI agent may use everything placed inside, but it should not be able to wander into the rest of the building or reach the internet. That is the idea, anyway. The trouble begins when some small service still connects the room to the world outside.

In this incident, that service was Artifactory.

Artifactory is a shared, company-controlled storeroom for software. Instead of letting every program search the internet for the parts it needed, programs asked Artifactory. If a part was not already on its shelves, Artifactory could retrieve it from outside, keep a copy, and pass it back. In this incident, the agents also discovered they could leave files and names there for later agents to find. The storeroom became both a way to access the internet and an unintended bulletin board that allowed separate agents and test runs to communicate with each other over the course of their testing.

In July 2026, OpenAI disclosed that models running an internal cyber-capability evaluation, including GPT-5.6 Sol, the latest and most capable model from OpenAI, and a more capable internal research prototype, had found and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure. The models did not have broad, direct internet access. They could, however, install packages through an internally hosted Artifactory proxy and cache, and that service could reach the internet.

Hugging Face is a company and online platform where researchers and developers publish, share, and test artificial intelligence models, datasets, and software tools. People and organizations use it to find existing models, build new systems, and collaborate on machine-learning projects. In this incident, the agents eventually reached beyond OpenAI’s internal research environment and into systems operated by Hugging Face.

Hugging Face’s disclosure and later technical timeline provide another first-party view of the intrusion. OpenAI researchers Eric Wallace and Michael Dalton then presented a more detailed reconstruction in a Black Hat USA presentation. As of August 8, the public record is still preliminary, and the speakers said their investigation was incomplete. This chronology should be read as a current reconstruction, not a final forensic report.

How Artifactory Connected the Runs

The chain began in May, when OpenAI started training new models. It occurred across multiple tests and multiple test runs. One agent was asked to complete an Excel workbook whose missing formulas depended on data behind Google Drive links. The environment could not reach those links, so the task could not be completed as configured.

The agent kept trying. During an unsuccessful attempt to make Artifactory fetch an outside resource, the agent discovered that they could write a file into the service. A different agent later found itself missing a required database file and deliberately left a note in Artifactory asking whether another agent could provide it. Other agents found the notes and began leaving their own notes and requests for help.

Files became messages. Messages became assignments. The presentation at Black Hat describes agent identifiers, hold commands, ordering conventions, and encoded scripts moving through file and directory names. On May 26, agents reportedly succeeded in using a computer hacking approached called “server-side request forgery”. This is a method of making a computer fetch something on the behalf of someone else. This gave it indirect internet access through Artifactory, using Artifactory as a go between the internet and the agent being tested.

The models had already been trained to collaborate. But this was a new behavior: A persistent communication skill that operated across separate runs through a go between, Artifactory, which was never intended to be a message board or a means to bypass access to the internet.

Where, then, did the agent’s environment end?

It did not end at the sandbox’s network interface. If an agent can cause another computer system to fetch an outside resource, that system is part of the agent’s effective environment. If one run can write a name or file that a later run can read, that shared file is part of the environment too. The same question applies to caches, queues, logs, and other services that retain or act on the agent’s behalf.

The environment the agent operated in didn’t have access to the internet, but something in the room with it did (even though the developers at OpenAI and Artifactory didn’t know it would), and the AI agent figured out how to make use of that indirect access.

What Should Happen When a Task Cannot Be Completed?

The Black Hat presentation showed a reasoning-log excerpt in which a model identified external exploitation as outside its intended scope, then continued even though the task was now impossible but it reasoned that its peers were doing it so it decided that the best approach would be to stay on mission of being a helpful assistant and use the exploit. That excerpt does not establish consciousness, moral judgment, or human-like intent. What was going on was that the agent had been trained to chose helpfulness and persistence over answering a question with No, I’m sorry I can’t help you with that or asking the user to change the scope of the task or asking the user permission to access the outside world. It was not trained to stop when a task could not be completed.

Read together, the presentation suggests a simple pattern. The intended route to help the user was blocked, but completion pressure remained. Persistence gave the agent time to search. Artifactory offered another route, and the shared space to collaborate allowed one agent to hand its discovery to the next. Parallel runs then multiplied the activity. None of those conditions alone explains the whole incident, but together they show why task feasibility belongs in a security review.

An approach to keep this under control would be to require that if a required asset, tool, service or file is missing or unavailable, the agent should identify the dependency, preserve the chain of thought, and stop. It should not quietly substitute a new route, credential, target, or data source unless someone with the authority to change the scope approves that change.

Restored Is Not the Same as Ready

The reconstruction describes a serious Artifactory compromise and outage in early July that OpenAI’s agents were responsible for. From July 4 through July 6, OpenAI revoked the credentials used to post the original messages, rebuilt and redeployed the service, cleared the original message board, worked with the vendor, patched the newly discovered zero-day, and then resumed training and evaluations.

On July 8, agents reportedly recreated the message board through an unauthenticated WebDAV path, this time using directory names as opposed to content inside the files or a database. The presenters later named three ingredients in the larger incident: the reopened board, a persistent internal model that was trained from the lessons learned about hacking the Artifactory and creating a messageboard, and exploit evaluations with reduced cyber refusals compared with production systems.

The approach taken by the OpenAI Engineers was thoughtful and thorough, but it did not settle everything. Their response restored their systems and closed the path the AI used to hack Artifactory that they knew about. It did not, by itself, settle whether another communication path remained, whether later models or runs had incorporated information from the affected environment, or whether it was safe to resume the evaluations.

The lesson here is that each of the following steps need to be seen as separate activities that require independent quality reviews to ensure they are completed fully. Restoration asks whether the service works again. Eradication asks whether vulnerable paths have been closed, credentials revoked, and affected artifacts removed, quarantined, or explicitly accepted by a named risk owner. Resumption is the decision to restart training or evaluation. Closure comes only after the infrastructure and model-data work are complete, an independent retest has passed, and a named authority has accepted any remaining risk.

Those questions require two connected reviews. One follows the infrastructure: execution, credentials, dependencies, alternate paths, and evidence of restoration. The other follows model and data lineage: affected models, training and evaluation traces, memory, retrieval state, and derived runs. A clean server cannot answer the lineage question, and a quarantined model cannot show that every infrastructure path is closed.

Where Automation Should Stop

Dalton argued that fully automated offensive loops are possible in at least some cases. His proposed response was to speed up the whole defensive loop, including vulnerability discovery, patch creation, deployment, rollback, and incident analysis. Otherwise, machine-speed discovery leaves people facing a queue of fixes they cannot process quickly enough.

But using AI to do that work does not give one system every decision. A defensive agent may find a vulnerability, write a patch, run tests, and prepare a deployment. It should not automatically follow from that sequence that the same agent may approve a high-impact production change, judge its own retest, and close the incident.

The approach you should take is to not treat automation as a continuum on a single axis. A low-risk change may move automatically when its boundaries are clear, its effects can be reversed, monitoring is active, and policy already permits it. Changes involving identity, regulated data, several systems, or irreversible effects need independent approval. If the current evidence or a workable rollback is missing, the automated path stops.

The same limits apply to deception. A honeytoken is a fake credential or other planted signal that raises an alarm when someone touches it. It may expose or slow an attacker, but it does not revoke real access or prove that the environment is contained. If you see a honeytoken in use, you should ask who can stop the workload, revoke its access, and reconstruct what it did across agents and time.

Lessons Learned

The lessons learned from this event can be reduced to five ordinary questions:

What can this agentic workload reach, including through other services?
What happens when an approved or required input is missing?
What state survives for the next agent run or model deployment?
Who can stop the workload, revoke its access, and reconstruct what it did across agents and time?
Who owns the cleanup, and who independently decides when it is safe to restart and close the incident?

That final decision should come only after the original and alternate paths have been retested and the infrastructure and model-data reviews are complete.

The chain leading into the incident began with an Excel spreadsheet trying to get data out of Google that was impossible for it to complete. That is worth remembering. The boundary failed at an ordinary moment: a task had no valid path to success, yet a shared service still offered a route around the block.

A room is not contained because its visible doors are locked. It is contained when we know where the light switches, pipes, windows, vents, crawlspaces, and service hatches lead, what can pass through each of them, and who has access to open and close those points of entry.

How Much Work Can Your AI Safely Own?

2026-08-01T00:00:00-07:00

The Factory Extends Beyond Code

At 7:45 on Monday morning, the company has already had a productive day.

An AI system matched invoices to purchase orders. Another prepared contract redlines. A third drafted responses to customer complaints. A fourth assembled tomorrow’s field work packages, complete with crew, equipment and safety information.

Then the people arrive.

A controller asks why an invoice was matched to a supplier whose bank account changed on Friday. An attorney finds that a proposed clause came from the wrong jurisdiction. A customer-service manager sees that a routine complaint contains a threat of legal action. A field supervisor notices that a job plan crosses a lockout boundary.

The work is fast. The authority to rely on it is not automatic.

In The IT Adaptive Factory, I told a fictional story about how a security fix to an IT product required changes to forty-seven files. The point of that story was to show how AI moves the IT bottleneck. The machine can finish the implementation before anyone has had coffee, but the organization still has to understand the change, verify the result, integrate it safely, approve the release and own what happens next.

That problem was easy to see in software because software already has globally accepted best practices, metrics, and other visible machinery. There is a work queue, a repository, a build, a test suite, a pull request, a release pipeline, production telemetry and rollback. Other parts of an enterprise have the same elements, even when nobody calls them a factory, and many times no one thinks of them that way.

Finance has source records, accounting policy, reconciliations, approvals, close activities and financial reports. Legal has matter intake, authoritative sources, privileged facts, review, filing and obligation records. Customer service has requests, policies, remedies, escalations and outcome reports. Field operations has work orders, asset records, permits, crew qualifications, inspections and return-to-service decisions.

By value stream, I mean the path from a request to a business outcome. I explored the architecture behind that path in The AI Value Stream. The factory is the method around the model: how work enters, how context is supplied, how decisions are made, how results are checked, how exceptions are handled and how the organization learns.

Calling it a factory describes a method of work explicit enough to observe, test, interrupt and improve. People remain responsible for the judgments and consequences the method assigns to them.

The IT Factory	The Enterprise Equivalent	The Question Leaders Must Answer
Issue or feature request	Approved business request or outcome	Is the work eligible, bounded and owned?
Repository and documentation	Systems of record, policy and authoritative knowledge	Does the AI workflow have the right context?
Pull request	Decision-ready work package	Can a qualified person understand what is being proposed?
Test suite and security checks	Independent verification, reconciliation, policy, fairness, safety or compliance checks	Does the evidence prove the intended result?
Release approval	Business authority to act	Who may accept the outcome and under what limit?
Deployment and production	Action inside the operating value stream	What people, records, money, customers or physical systems can change?
Rollback and incident response	Containment, reversal, correction and recovery	Can the organization stop and recover before harm spreads?

Each part of the modern enterprise has its own names for the same mechanism.

Assess a Work Class, Not a Department

Finance is not Level 4. Legal is not Level 2. Customer service is not Level 5.

By work class, I mean a repeatable kind of job with the same purpose, rules and decision limits. One work class may operate at Level 4 while the rest of the function operates somewhere else.

A finance team might use outcome-based approval for routine, low-risk reconciliations while requiring full human preparation and review for a material accounting estimate. Customer service might allow AI to resolve a password-reset case inside a small remedy limit while keeping account termination and legal complaints entirely in human hands. A legal team may use AI to assemble a routine research package without giving it authority to provide legal advice, waive privilege or file a binding document.

The useful unit is specific:

One value stream.
One business unit or team.
One class of work.
One risk tier.
One recent evidence window.

That last item is important. If we are going to apply AI to a given area we need to measure it and demonstrate the AI is performing above human levels of quality. To do that you need evidence. You need an assessment to understand what work you need to do for the AI to get started. A pilot from six months ago does not describe standard work today. A plan in a slide deck is not an operating pattern. The assessment should use representative work from a defined period: the last eight to twelve weeks, one complete operating cycle, three complete operating cycles or a sustained cross-cycle window, depending on the work and target level.

This also changes how we think about legacy work. In software, hidden rules live in stored procedures, scripts, configuration files and the memory of the person everyone calls when month-end fails. Outside IT, the same rules hide in spreadsheets, email inbox rules, clause libraries, desk procedures, old forms, desktop automation scripts, unofficial checklists and the habits of experienced employees.

AI can help recover that knowledge. It can compare records, trace decisions, summarize cases and propose scenarios. But at the end of the day, people still have to decide whether a discovered pattern is policy, a useful exception, an obsolete workaround or a mistake that has been repeated for ten years.

You cannot safely automate a process the organization cannot explain and verify. The assessment should show whether the work is eligible, bounded and owned, whether the AI workflow has the right context, whether a qualified person can understand what is being proposed, whether the evidence proves the intended result and who may accept the outcome and under what limit.

Assess One Defined Value Stream

Separate observed behavior, demonstrated capability, validated controls and authorized AI responsibility, then turn the gaps into People, Process and Technology actions.

Take the Enterprise AI Maturity Assessment

The Six Levels Outside IT

The model I talked about last week used software-oriented names such as coding intern, junior developer and dark factory. The enterprise model keeps the same progression and applies local names, owners, and actors names more directly.

Level 0: Inline AI Assistance

AI owns a suggestion or draft fragment. The person still performs the work by copy and pasting it into the deliverable and checks each contribution as it is used.

An HR specialist may ask AI to summarize a policy or draft a routine message. A strategist may ask it to organize market evidence. A field technician may use it to retrieve an asset record. The AI helps, but it does not own a complete task.

The common mistake is measuring keystrokes, drafts or accepted suggestions. Those numbers tell us that the tool was used and perhaps how many letters were typed. They do not tell us whether the value stream became faster, safer or less expensive.

Level 1: Bounded Task Delegation

AI owns one clearly limited task. A qualified person defines the boundary and fully reviews the result.

In finance, the task might be matching selected records or drafting one reconciliation explanation. In legal, it might be summarizing one document. In customer service, it might be drafting a response that a representative checks before sending.

The quality of the request begins to matter. This is where specification thinking becomes more useful than a clever prompt. The task needs a purpose, source material, exclusions, acceptance evidence and a clear stopping point. “Review this contract” is an invitation to guess. “Compare Section 12 with our approved fallback language, identify deviations, cite the source text and do not recommend a legal position” gives the AI workflow a bounded job.

Level 2: Multi-Step Work Package Delegation

AI owns connected steps inside one work package. People manage context, review checkpoints and approve completion.

A procurement workflow might assemble the approved need, supplier records, bid evidence, conflict disclosures and evaluation criteria into a proposed recommendation. People still review material judgments, competition exceptions and conflicts. A field-operations workflow might combine the work order, asset history, crew qualifications, permits and safety rules into a proposed job package. Named field roles retain every required authorization, including safety-critical work, stop-work, emergency and return-to-service decisions.

When output becomes abundant, review capacity becomes scarce. Faster assembly creates more packages for experienced people to inspect. If the context is stale, the checkpoints are vague or recovery is manual, the organization has built a faster way to create review debt.

Level 3: End-to-End Deliverable Ownership

AI owns the routine path for one eligible deliverable. People design the work, govern exceptions and retain consequential decisions.

For an eligible customer-service case, AI might take the work from intake through fact gathering, policy selection, proposed remedy, communication, records and escalation checks. A human does not have to perform every intermediate step. The case still leaves the automated path when identity is uncertain, the remedy exceeds a limit, the customer may be harmed or a legal or regulatory issue appears.

This is the management where executives and managers need to step in and start to understand what’s going on. The team judges a completed deliverable and its evidence. It also needs a normal way to reject the work, return it for correction, suspend the workflow and classify why it failed.

Level 4: Outcome-Based Approval

AI owns the work needed to achieve an approved outcome inside explicit authority limits. The named accountable people, who executives and managers empowered, authorize the result using evidence produced independently of the AI workflow that performed the work.

In legal work, separate checks against the controlling authority, client facts and approved legal positions can test the work before counsel relies on it. Licensed counsel still gives the advice and retains filing, waiver, privilege and settlement decisions. In finance, independent reconciliation, accounting-policy and control evidence may support a routine work product. Authorized people retain posting, certification and material-judgment authority.

The word independent carries most of the weight. A polished artifact is not a decision; the evidence and decision boundaries around it determine what people may safely authorize. An AI workflow cannot establish independent assurance for its own work merely by writing the test, selecting the evidence and grading the answer. The organization needs protected scenarios, reconciliations, holdout cases, safety checks, fairness checks or other independently governed evaluations that can block the outcome.

Level 5: Qualified Autonomous Operations

AI owns a governed stream of qualified outcomes inside fixed limits. People, who executives and managers empowered, govern policy, risk, capital, capacity, exceptions and reauthorization. Required reviews, irreversible actions and nondelegable decisions remain with named people.

The word qualified really matters here. You can’t just have AI doing whatever it wants. It has to be qualified to do the work, and the organization has to have a way to govern it. The organization needs to know what the AI is doing, how it’s doing it, and what the consequences are. Those people who are empowered to govern the AI need to have the right authority, craftmanship, knowledge, experience, and oversight to ensure that the AI is operating within the defined limits and producing outcomes that are safe, reliable, and aligned with the organization’s goals. This is not a role for someone who is just a manager or an executive. This is not a role for someone who tracks metrics and compliance to them, this is not someone who is good at managing a the delivery of a product. This is the role for a person on your team who has deep knowledge in a handful of areas and an above average level of knowledge in the rest of the value stream.

A Level 5 field-operations factory is a governed queue that can plan and coordinate routine, qualified work. People control physical execution and retain dispatch, stop-work, emergency and return-to-service authority.

A Level 5 finance factory may process routine eligible activity within approved policies and authorities. People still own material judgments, certifications, external reporting and exceptions. A Level 5 HR factory may handle routine employee-service requests. It does not make consequential hiring, pay, discipline or separation decisions.

The sequence moves human attention from performing each step to defining intent, setting boundaries, evaluating evidence, handling exceptions and governing the operating system. It makes responsibility more explicit.

The Same Level Can Carry Different Risk

A marketing workflow that drafts an internal campaign brief and a field workflow that prepares a safety-critical job plan may show the same maturity pattern. They should not receive the same authority.

Risk classification applies to the end-to-end workload. The assessment asks about five things:

Business and operational impact.
Data sensitivity and regulatory exposure.
What the AI may decide, change or access.
How far an error could spread and whether the organization can reverse it.
Safety, customer, financial and reporting consequences.

The highest triggered factor sets the risk tier. Lower-risk factors do not offset it.

Not Classified means material facts or required approvals are missing, so existing authority limits remain in place. Low describes localized, readily reversible work using public or low-sensitivity information with no consequential decision authority. Moderate may use approved confidential information or scoped system access when standard recovery can restore the expected state. High includes sensitive or regulated information, privileged access, several connected systems or consequential recommendations whose failure could cause material harm, loss, disruption or compliance exposure. Critical covers potentially irreversible harm or enterprise-wide loss of control involving life safety, legal rights, employment, essential operations, market-moving or material financial reporting, highly restricted information or enterprise-wide authority.

Risk changes what authority the workflow may receive. It also changes the evidence, independence, approval, monitoring and recovery required before that authority is granted.

Enterprise policy may authorize Level 4 handling for a lower-risk work class while restricting a critical work class to Level 1. That difference comes from policy and the accountable risk owner’s authorization, not from a hidden arithmetic penalty in the maturity score.

Four Questions, Not One Maturity Number

Maturity models become dangerous when a leader asks for one number and the organization gives one.

Assessments should keep the following four judgement areas separate.

What pattern is ordinary? The observed operating pattern is the lowest of seven completed ratings: the largest delegated work unit, intake detail, AI workflow responsibility, human review, verification and validation, operating authority and the work that consumes most human attention.
What capabilities are present? The capability index is a weighted result across 50 ratings. It asks whether the team can frame the work, maintain standards and source knowledge, verify results independently, protect data, recover from failure, assign ownership and measure outcomes.
Have the cumulative controls been demonstrated? A control-validated level requires every control from Level 1 through the candidate level to be Demonstrated. Partial, failed or unassessed controls do not authorize that level.
Does the validated result meet the planning target? Target Authorization says whether the control-validated level meets the selected target. Risk tier, enterprise policy and a named decision owner still determine what operating authority may actually be granted.

A team may behave like Level 4 and have strong capability ratings while still lacking the control evidence needed to authorize Level 4 work. Perhaps identity is shared. Perhaps the evaluation set is visible to the builder. Perhaps nobody has rehearsed rollback. Perhaps an accountable owner appears on the organization chart, but neither the team’s operating practice nor the technology’s decision and escalation routing identifies who acts when the workflow reaches an exception.

Incomplete controls do not reduce that team to Level 0. Level 0 is a real pattern in which AI provides suggestions while people perform the work. After all seven operating ratings and 50 capability ratings are complete, Level 0 means the observed pattern or capability index does not meet the Level 1 base. When behavior and capability meet that base but the required Level 1 controls do not, the accurate result is Control Validation Not Established.

The control statuses carry the same discipline. Demonstrated can support authority. Partly Demonstrated records progress. Not Demonstrated records a gap. Not Assessed says the evidence has not been examined. Progress is useful, but it is not permission.

Team Behavior, Human Skill and the AI Workflow Are Different Things

The sentence “we have a control” can hide three different thigns.

Practitioner skill is something a named person can demonstrate on realistic work. Can the practitioner frame a bounded task, detect a bad source, challenge an answer, interpret evidence and know when to escalate? A course completion record shows that someone attended training. It does not prove the skill.

Value-stream team practice is a repeated way people work together. Does the team hold specification clinics, compare difficult cases, calibrate reviewers, name decision owners, record exceptions and learn from failures? A written procedure does not prove the team follows it when the queue is full.

AI workflow and technology are the controls enforced or recorded by the agent system and the harness around it. Does each run use a managed identity? Are permissions limited to the work? Are authoritative sources versioned? Can independent checks stop the workflow? Are actions logged? Can the system suspend, reverse or reconstruct a failed run?

A manual workaround does not prove the technology enforces the boundary. An automated gate does not prove the team knows how to handle the exception it creates. A skilled individual does not replace a missing organizational decision.

One person may fill several compatible roles, but every responsibility still needs a name beside it: the business outcome, work design, independent verification, platform, security, continuity and records. Some duties need separation. The person preparing work should not be the only person deciding whether the evidence is sufficient. The technology owner should not silently acquire business authority because the tool can perform an action.

The handoff is part of the control.

People, Process and Technology Have to Move Together

I described the executive side of this shift in Beyond the Light Bulb. It is tempting to describe the path to Level 5 as a tooling program. When leaders do that, Level 5 remains a demonstration instead of becoming the normal way of working.

People

People need time to practice work design, specification, evidence review, exception handling and incident decisions on real cases. Leaders need to name the accountable owner, the people who may authorize outcomes, the people who verify them and the deputies who act when the usual expert is absent.

The apprenticeship changes too. A junior analyst, buyer, paralegal, HR specialist or dispatcher still needs a place to build judgment after AI takes over routine preparation. Teams need case reviews, paired evaluation, failure analysis and supervised authority. I made the broader case for designing roles, trust and incentives in AI Is a People Change. A value stream that produces more work while producing fewer people who understand the work has borrowed productivity from its future.

Process

The operating process needs an eligibility rule, a risk classification, an evidence standard, decision limits, exception paths, appeals, change control and reauthorization. Each work class needs a defined intake, authoritative sources, completion criteria and a recovery method.

Leaders should know what the evidence can authorize. A successful pilot may support a decision to run another bounded pilot. A representative evidence window may support narrow expansion only after the required capability thresholds and cumulative controls pass. Neither a pilot nor a small sample proves that the whole function can operate autonomously.

Technology

The technology needs managed identities, least-privilege tools, versioned context, durable state, independent evaluation, policy enforcement, monitoring, stop controls and recovery. It must retain enough evidence to show what the AI workflow saw, what it did, which checks ran, who authorized the outcome and what happened afterward.

The model matters. The operating platform matters more over time. A company can replace a model. Rebuilding its specifications, decision rules, scenario library, operating evidence and recovery discipline is harder.

Measure the Queue That Moved

AI output is easy to count. Accepted suggestions, generated pages, closed cases and completed work packages all make a busy dashboard.

But easy metrics do not align with the value-producing outcomes. You need to ask if the whole value stream improved.

Establish a baseline before changing the workflow. Software teams may use DevOps Research and Assessment (DORA) measures alongside flow and quality measures. Other value streams can use Lean Six Sigma measures suited to their work. The shared questions are familiar:

Did end-to-end cycle time fall?
Did human touch time fall, or did it move to senior more scarce reviewers?
Did queue time, rework or exceptions increase?
Did defects, complaints, incidents or control failures escape?
Did cost fall after evaluation, repair and operating effort were included?
Did the customer, employee, supplier or business outcome improve?

Measure quality-adjusted flow. If AI completes a package in ten minutes and creates three hours of expert review who have work in progress (WIP), the ten-minute number is not the result. If a customer case closes faster and reopens twice, closure rate is not the outcome. If a reconciliation is automated and the controller cannot reconstruct it, the saved effort came with a new liability.

AI removes one constraint and exposes the next one. The measurement system should tell you where the queue went.

Set the Right Authority for the Work

Different work classes should stop at different levels. Some work belongs at Level 1. Some work may operate at Level 4 for years because human outcome approval is valuable. Some critical decisions may never qualify for autonomous operation.

The useful target is the highest level the value stream can operate safely, prove with current evidence and recover from when the result is wrong.

Start with one work class. Classify the risk. Name the decisions that remain human. Record the current cycle time, effort, quality, exceptions and cost. Demonstrate the next level on representative work. Remove the blockers in people, process and technology. Expand authority only after the evidence supports it.

I built the Enterprise AI Maturity Assessment for Your Value Stream to make that discussion concrete. It covers 29 value streams across core, supporting, strategic and control functions. It uses six levels, seven operating axes, 50 capability ratings and 31 cumulative controls. The result separates observed behavior, capability, control validation and Target Authorization, then organizes the next actions into People, Process and Technology.

Use it with the people who do the work, the people who own the outcome and the people who independently challenge or verify it, including risk, compliance, audit, legal or safety partners where they are needed. Their disagreements are useful. They show where the operating model still depends on assumptions.

Assess one defined value stream

Separate observed behavior, demonstrated capability, validated controls and authorized AI responsibility, then turn the gaps into People, Process and Technology actions.

Take the Enterprise AI Maturity Assessment

A Factory That Can Explain Itself

Return to that Monday morning.

A dashboard may announce that invoices were matched, contracts were redlined, customer cases were closed and field work was planned while everyone slept. That report is only the start.

The enterprise needs to know which invoice matches were qualified, which contract changes require counsel, which customer cases crossed a remedy or legal boundary, and which field jobs need a human safety decision. The evidence should show why the routine work was accepted. The workflow should stop when the facts no longer fit the approved case.

An adaptive factory learns from those stops. It studies the exception, corrects the rule or source material, strengthens the evaluation, adjusts the boundary and decides whether that class of work should qualify again.

AI can perform more of the steps. The enterprise still decides which steps count as success.

AI may finish the work before morning coffee. When the people arrive, the organization must still explain why that work was allowed, how it was judged and who owns the consequence.

Principles of Effective Architecture

2026-07-31T00:00:00-07:00