RFC: replace Headless with a minimal Eval kernel

[Astro-Han]

Problem

packages/headless duplicates Maka execution outside Runtime Host and has accumulated TaskRun, autonomous loops, self-check, specialized workflows, and multiple entry points.

The operative problem is authority, not size or missing abstraction: Headless constructs and owns runtime concepts — sessions, turns, backends, tool surfaces, Agent Graph behavior, continuation, events — outside Runtime Host.

Proposal

Delete packages/headless and replace it with a minimal @maka/eval.

flowchart TD
    CLI["maka eval"] --> E["Experiment kernel"]
    E --> C["Cells<br/>task × repetition × subject"]

    C --> X["Executor<br/>Harbor or Pier"]
    X --> M["Maka subject"]
    X --> O["External subject"]

    M --> R["Runtime Host<br/>ephemeral session"]

    M --> K["Result"]
    O --> K

Loading

Experiment = benchmark + executor + subjects + tasks + repetitions

A/B is simply a two-subject experiment.

Principles:

• All Maka runs go through Runtime Host.
• Eval depends on the Runtime Host client/protocol only, never on Runtime implementation, and never constructs Runtime objects.
• Subjects are either maka or generic external.
• Harbor and Pier are executors, not separate workflows.
• maka eval is the only public CLI.
• One fully expanded spec defines each experiment.
• Infrastructure-failed cells can be replaced individually; the earliest valid attempt wins.
• Results contain only score, usage, cost, duration, status, and artifacts.

Three concepts that Headless conflates stay distinct:

repetition   = a new experimental sample
infra retry  = a replacement attempt for the same cell
continuation = subject-internal behavior owned by Runtime Host

Delete

Do not port or preserve:

• TaskRun
• Autonomous Loop
• Heavy/Self-check
• specialized A/B workflows
• duplicated runtime, session, and event models
• maka-headless
• compatibility and legacy result readers

Heavy/Self-check is deleted outright. It is subject behavior, not an Eval workflow — but it is not preserved as a subject setting either. Comparing self-check on/off is a two-subject experiment, not a kernel feature.

Runtime Host must also shed Eval concepts, and these reach past Runtime Host into Runtime and Core:

• benchmark_deadline as a stop-session source in packages/runtime (session-manager.ts, message-authority.ts, session-projection-helpers.ts) and packages/runtime-host (root-turn-coordinator.ts). session-projection-helpers normalizes it to benchmark.deadline and projects it. Its only producers are in Headless.
• Eval artifact layout hardcoded into context compaction: /maka-task-run/, /exports/harbor-, /task-run.json and a self-check verifier pattern in packages/runtime/src/active-full-compact-facts.ts.

Removing the package also removes its host seams: the @maka/headless dependency and maka inspect in packages/cli, the headless tool host id in packages/core/src/tool-catalog.ts, and the headless build steps in the root package.json. maka eval replaces the existing eval-router entry point rather than adding a new one.

Two look-alikes must survive, to avoid over-deletion during execution:

• subject in packages/runtime-host/src/protocol/task-ledger.ts is a task title field, unrelated to Eval subjects.
• The Goal tools in packages/runtime (GoalSet / GoalPause / GoalResume) are product autonomous execution — subject behavior owned by Runtime Host, not the Headless Autonomous Loop.

This is an intentional breaking cutover. No existing experiment needs resume support.

Done when

• packages/headless is gone.
• @maka/eval has no build-time dependency on Runtime implementation and constructs no Runtime objects.
• No benchmark, repetition, score, or A/B concept remains in Runtime, Runtime Host, or Core — including the benchmark_deadline stop source, Eval artifact paths in compaction heuristics, and the headless tool host id.
• The current cohort runs from one declarative multi-arm spec.
• Maka, competitors, and Maka setting variants use the same kernel.
• A single failed cell can be retried without rerunning the cohort.
• Adding a subject does not require changing the kernel.

Review

Please identify any current Headless capability that is both necessary and not represented above. Otherwise, delete it rather than migrate it.

[likun666661]

I think the core of this RFC is the separation of two different domains that Headless currently mixes together.

First, packages/headless has become a second Maka runtime. This is an authority problem, not primarily a code-size or abstraction problem. It constructs and owns runtime concepts such as sessions, turns, backends, tool surfaces, Agent Graph behavior, continuation, and events outside Runtime Host.

The invariant should be:

@maka/eval does not execute Maka.
@maka/eval asks Runtime Host to execute a Maka subject.

Ideally this is also a dependency boundary: Eval may depend on the Runtime Host client/protocol, but must not depend on Runtime implementation or construct Runtime objects directly. Runtime Host remains the sole authority for Maka execution.

Second, there is still a legitimate evaluation domain. Removing the second Runtime must not erase the benchmark kernel. Eval owns:

Experiment -> Cells -> Attempts -> Results

It answers what to run, for which subjects and tasks, how many repetitions, on which executor, and how results are selected and compared. It does not own Session, Turn, Agent Graph, continuation, self-check, or agent lifecycle.

This also makes three currently conflated concepts distinct:

repetition   = a new experimental sample
infra retry  = a replacement attempt for the same cell
continuation = subject-internal execution behavior owned by Runtime Host

From that boundary:

• TaskRun is an experiment cell incorrectly modeled as another runtime run.
• Autonomous Loop mixes experiment retry/repetition with runtime continuation.
• Heavy/Self-check is subject behavior or a subject setting, not an Eval workflow.
• A/B is simply a two-subject experiment.
• Harbor and Pier are executor adapters.
• Competitors are external subject adapters.

The resulting architectural invariants should be:

1. Runtime Host is the sole authority for Maka execution.
2. Eval owns only experiment semantics.
3. Runtime Host contains no benchmark, repetition, score, or A/B concepts.
4. One cell requests one subject execution; how that execution completes belongs to the subject.
5. Adding a subject, benchmark, or executor adds an adapter without changing the kernel.

So the cutover should extract the valid Eval domain and delete the duplicated Runtime domain, rather than treating existing Headless capabilities as things to migrate by default.

[Astro-Han]

Agreed on the framing, and I've adopted it as the basis for the final version of the issue.

Your reframing is the correction I needed: I described this as duplication causing size, drift, and CI cost, but the operative problem is authority. That also replaces the filter in my Review question — the test is not "is this capability necessary" but "which domain does it belong to". Under that test, nothing currently in Headless is both necessary and unrepresented, so the cutover stands as a deletion, not a migration.

Three things folded into the issue.

1. The dependency boundary becomes an acceptance criterion.

My original wording ("all Maka runs go through Runtime Host") only constrained runtime behavior. Making it a build-time boundary is stronger and mechanically enforceable, so it is now in Done when: @maka/eval may depend on the Runtime Host client/protocol only, never on Runtime implementation, and never constructs Runtime objects.

2. The reverse invariant is already violated, and more broadly than I assumed.

I had only constrained the Eval side. Your point 3 has concrete outstanding items today, and they reach past Runtime Host into Runtime and Core:

• benchmark_deadline is a stop-session source in packages/runtime (session-manager.ts, message-authority.ts, session-projection-helpers.ts) and packages/runtime-host (root-turn-coordinator.ts). session-projection-helpers normalizes it to benchmark.deadline and projects it, so a benchmark concept reaches the session event stream. Its only producers are in Headless, so after the cutover it is a dead enum and a dead projection branch.
• packages/runtime/src/active-full-compact-facts.ts hardcodes Eval artifact layout — /maka-task-run/, /exports/harbor-, /task-run.json — into context-compaction heuristics, plus self-check in a verifier pattern.
• packages/core/src/tool-catalog.ts treats headless as a first-class tool host id, so every surface definition carries a headless field.

All of this is in scope for the cutover, not follow-up work.

Two look-alikes should explicitly survive, to avoid over-deletion during execution: subject in runtime-host/src/protocol/task-ledger.ts is a task title field, unrelated to Eval subjects; and the Goal tools in packages/runtime (GoalSet / GoalPause / GoalResume) are product autonomous execution — subject behavior owned by Runtime Host, not the Headless Autonomous Loop.

3. Heavy/Self-check is deleted outright, not preserved as a subject setting.

We agree it leaves Eval. To be explicit: it is not kept as a subject setting either. Concretely it is self_check_plan_submit and self_check_submit — MakaTools constructed in Headless, plus a completion gate, policy, evidence, finalization, and export path. So it is a clear instance of the authority problem: Headless building a tool surface outside Runtime Host.

But its evidence contracts (HeavyTaskSelfCheckPlanAuditSummary and friends) have no consumer on the Runtime Host side. "Move it to a subject setting" would turn a pure deletion into a rebuild, for a capability that grew to serve benchmark scoring. If we want it later, the right shape is a distinct subject variant compared against baseline through the same kernel — which is exactly what the two-subject experiment model gives us for free, with no Eval-side switch.

Your repetition / infra retry / continuation split is in the issue verbatim. It was implicit in the original spec and easy to lose during execution: deleting Autonomous Loop must not delete cell-replacement semantics.