refactor: replace Headless with minimal Eval kernel

[Astro-Han]

Summary

Replace packages/headless with a minimal @maka/eval domain and make Runtime Host the sole authority for Maka execution.

• model evaluation as Experiment -> Cells -> Attempts -> Results; a cell is task x repetition x subject
• execute Maka only through the Runtime Host client/protocol; Runtime Host owns Session, Turn, continuation, tools, events, lifecycle, cancellation, and usage settlement
• execute generic external subjects through executor-owned Harbor/Pier native Trial lifecycles
• keep the current 89-task, four-arm cohort in one fully expanded Experiment spec under one executor, benchmark, budget, and verifier
• preserve immutable append-only attempts, single-cell replacement, and earliest-valid result selection
• expose only maka eval, and delete Headless, TaskRun, Autonomous Loop, Heavy/Self-check, specialized A/B workflows, legacy readers, duplicate runtime models, and their reverse dependencies

Fixes #2598

Verification

• clean affected builds: Code Mode, Core, Storage, MCP, Runtime, Runtime Host, Eval, and CLI
• Runtime Graph focused tests: 26 passed
• Runtime Host hosted-execution/composition focused tests: 17 passed
• Eval: 18 TypeScript tests and 4 Python tests passed
• Windows recovery, CI planning, and CLI build-order focused scripts: 13 passed, 1 Windows-only skip
• npm run format:check: 1,483 files checked
• git diff --check

Repository-wide tests were not run locally. No live-provider/Docker four-arm smoke was run. The pinned Harbor/Pier native contracts were checked manually in isolated environments during development; checked-in automated tests cover the local framework/relay boundaries without claiming a live framework or provider integration proof.

Breaking change

This is an intentional cutover with no compatibility layer, legacy result reader, resume path, or result migration. Existing Headless experiments cannot resume. Environment variables carry only credentials and machine-local paths; the fully expanded Experiment spec remains semantic authority.

Review focus

The diff is +5,632 / -131,081. Review the authority invariant first: Eval owns experiment semantics but must not construct Runtime objects or own Session, Turn, continuation, tools, events, or agent lifecycle. Runtime, Runtime Host, and Core must not acquire benchmark, repetition, score, or A/B concepts from Eval.

Fresh-eye review after rebasing onto current main found no reproducible production P0/P1. Remaining observations are P2 test-protection gaps, so no further production changes were made under the agreed stop rule.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — described under Summary above
• No

[M4n5ter]

English

The overall direction is strong. Deleting Headless as a second Runtime authority and limiting @maka/eval to Experiment/Cell/Attempt/Result semantics matches the Runtime Host architecture well. Maka execution crosses the public Runtime Host boundary, while Harbor/Pier retain ownership of Trial preparation, verification, and finalization.

I found one concrete regression, one intent-dependent artifact question, and one important opportunity to simplify the Host integration.

1. Preserve maka inspect for Runtime Host state

This PR removes the entire inspect command because it includes Headless TaskRun support. However, the same command also provides valid Session/AgentRun inspection through the Runtime Host control plane.

Please remove only the TaskRun and Headless-root branches. Session/AgentRun inspection, execution.inspect.resolve/query, and live Runtime Host inspection remain useful Runtime Host capabilities and should not be removed as Headless residue.

2. Clarify the intended contract of runtime_host_execution

The checked-in experiment creates Runtime Host roots under /tmp/maka-runtime-hosts inside a Docker environment configured with delete: true. The persisted Eval result records:

{ kind: 'runtime_host_execution', executionId }

After Trial finalization deletes the container, the corresponding Runtime Host root and its Session/Turn/Run data no longer exist. Whether this needs a larger fix depends on the intended meaning:

• If runtime_host_execution is intended to be a durable, inspectable artifact, the Host root must be persisted or an immutable execution artifact must be exported before finalization.
• If it is only correlation or diagnostic metadata, it should not be represented as an artifact. The simplest first-version solution is to remove it and rely on the native Harbor/Pier Trial artifacts.

I do not think the first version needs a new Eval-owned trace schema. The important part is that the result contract accurately reflects what remains available after the Trial has completed.

3. Collapse the hosted-execution adapter onto canonical Host authority

HostHostedExecutionRunner currently composes five protocol handlers, constructs a synthetic ConnectionContext, polls turn.query every 25 ms, and waits for global Host residency through an optional waitForResidencies port with a no-op fallback.

Runtime Host already has HostedExecutionAuthority, exact completion/settlement promises, and waitForHostedExecutionTerminal(). The hosted-execution protocol adapter should reuse those canonical signals through a narrow application service. This would remove the polling loop, global residency dependency, synthetic context, and optional lifecycle port.

The Domain Module should also be named hosted-execution, not eval-execution: Runtime Host provides a generic capability and should not know which consumer requested it.

The append-only Attempt model, earliest-valid-result rule, dedicated Host root per Maka attempt, and fail-closed writer lock are appropriately simple for the first version. I would keep those parts as they are unless a concrete need justifies more machinery.

中文

整体方向很好。删除作为第二套 Runtime authority 的 Headless，并将 @maka/eval 限定在 Experiment、Cell、Attempt 和 Result 语义内，符合 Runtime Host 架构。Maka execution 经过公开的 Runtime Host 边界，而 Harbor/Pier 继续拥有 Trial preparation、verification 和 finalization。

我发现了一个明确的功能回退、一个取决于作者意图的 artifact 问题，以及一项重要的 Host 集成简化机会。

1. 保留面向 Runtime Host 状态的 maka inspect

这个 PR 因为 inspect command 包含 Headless TaskRun 支持，而删除了整个命令。但同一个命令还通过 Runtime Host control plane 提供有效的 Session/AgentRun 检查能力。

请只删除 TaskRun 和 Headless root 相关分支。Session/AgentRun inspection、execution.inspect.resolve/query 和 live Runtime Host inspection 都是仍然有价值的 Runtime Host 能力，不应作为 Headless 残留一并删除。

2. 明确 runtime_host_execution 的预期契约

仓库内的实验配置在设置了 delete: true 的 Docker 环境中，将 Runtime Host root 创建在 /tmp/maka-runtime-hosts。持久化的 Eval result 中记录了：

{ kind: 'runtime_host_execution', executionId }

Trial finalization 删除容器后，相应的 Runtime Host root 以及其中的 Session、Turn、Run 数据也会消失。是否需要较大的修复，取决于作者对该字段的设计意图：

• 如果 runtime_host_execution 表示一份应当持久存在、可以事后检查的 artifact，那么需要持久化 Host root，或者在 finalization 前导出不可变的 execution artifact。
• 如果它只是用于关联或诊断的 metadata，就不应表示为 artifact。第一版最简单的方案是删除它，继续依赖 Harbor/Pier 原生持久化的 Trial artifacts。

我认为第一版没有必要再设计一套由 Eval 拥有的 trace schema。关键是 result contract 应准确反映 Trial 完成后仍然真实存在的内容。

3. 将 hosted-execution adapter 收敛到 canonical Host authority

HostHostedExecutionRunner 当前组合了五个 protocol handlers、构造 synthetic ConnectionContext、每 25ms 轮询一次 turn.query，并通过带有 no-op fallback 的 optional waitForResidencies port 等待全局 Host residency。

Runtime Host 已经拥有 HostedExecutionAuthority、精确的 completion/settlement promises，以及 waitForHostedExecutionTerminal()。Hosted-execution protocol adapter 应通过一个窄应用服务复用这些 canonical signals。这样可以删除 polling loop、全局 residency 依赖、synthetic context 和 optional lifecycle port。

Domain Module 也应命名为 hosted-execution，而不是 eval-execution：Runtime Host 提供的是通用能力，不应知道调用它的是哪个消费者。

Append-only Attempt、earliest-valid-result、每个 Maka attempt 使用专属 Host root，以及 fail-closed writer lock，对第一版来说都足够简单合理。除非出现明确需求，否则我建议保持这些部分不变，不再增加额外机制。

[Astro-Han]

Thanks for the thoughtful review. I checked the three points against the current Issue contract and HEAD.

On maka inspect, I’m keeping its removal because #2598 explicitly makes maka eval the only public Eval CLI and calls for deleting the old inspect entry point. The Runtime Host inspection protocol (execution.inspect.resolve/query) remains intact. If a public Host inspection CLI is needed, it can be rebuilt separately without Headless or Eval semantics.

I addressed the other two concrete boundary issues:

• 109fb08f4 removes runtime_host_execution, since the Trial deletes that Host root and the ID did not reference a durable artifact.
• 79810edaf renames the Runtime Host module from eval-execution to hosted-execution.

I’m deferring the larger runner rewrite. The existing HostedExecutionAuthority settles a Root Turn, while this boundary must also account for Host-owned continuations, usage sealing, and verifier-environment resources. Collapsing onto it directly would broaden this PR and risk premature settlement. The earlier no-op lifecycle fallback is already gone on current HEAD.

Focused Eval and Runtime Host tests, Biome, and git diff --check pass.

中文对照

感谢细致的审查。我根据当前 Issue 约定和最新 HEAD 核对了这三点。

关于 maka inspect，本 PR 仍会删除它，因为 #2598 明确规定 maka eval 是唯一公共 Eval CLI，并要求删除旧 inspect 入口。Runtime Host 的 execution.inspect.resolve/query 协议仍完整保留。若以后需要公共 Host inspection CLI，可以在不引入 Headless 或 Eval 语义的前提下独立重建。

另外两个具体边界问题已经处理：

• 109fb08f4 删除了 runtime_host_execution。Trial 会删除对应 Host root，因此该 ID 并不指向持久 artifact。
• 79810edaf 将 Runtime Host 模块从 eval-execution 改名为 hosted-execution。

更大的 runner 重构暂缓。现有 HostedExecutionAuthority 只封口 Root Turn，而这里还需要处理 Host 拥有的 continuation、usage seal 和 verifier environment resources。直接收敛过去会扩大本 PR，并可能重新造成过早结算。早期 review 提到的 no-op lifecycle fallback 在当前 HEAD 中也已不存在。

Eval、Runtime Host 聚焦测试、Biome 和 git diff --check 均已通过。

AI disclosure: This reply was drafted and posted with OpenAI Codex assistance. The author reviewed the decisions; the cited changes and focused validation were verified locally.

[Astro-Han]

@M4n5ter, @likun666661, I rebased #2605 onto the latest main. The new head is dcd10847e; local checks pass, and CI is running.

Since you both reviewed it earlier, would you mind giving the rebased version one last look and approving it if it matches #2598?

The earlier feedback is addressed or documented in the thread.

中文对照

@M4n5ter、@likun666661，我已经把 #2605 rebase 到最新 main。新的 HEAD 是 dcd10847e，本地验证通过，CI 正在运行。

两位之前都审查过这个 PR，方便的话，麻烦再看一下 rebase 后的版本。如果符合 #2598，请 approve。

之前 review 提到的问题都已处理，或在 thread 中说明了取舍。

AI disclosure: OpenAI Codex helped draft and post this comment. I chose the reviewers and verified the rebase and local checks.

[M4n5ter]

English

Approved. The updated PR matches the authority boundary in #2598: Eval owns experiment semantics, while Runtime Host remains the sole authority for Maka execution

I am withdrawing my earlier maka inspect finding. #2598 explicitly defines maka eval as the only public Eval CLI and includes removal of the old inspect entry point, so that deletion is intentional rather than an accidental Runtime Host regression

The two concrete boundary issues are resolved cleanly: the ephemeral runtime_host_execution artifact is gone, and the Domain Module is now named hosted-execution. Deferring the larger runner rewrite is also reasonable because this boundary currently has to settle continuations, usage, and verifier-visible environment resources together

One non-blocking follow-up is worth recording. After hosted.execution.start has already returned a settled completed or failed projection, runHostedExecution() performs a separate diagnostics request to decide whether runtime resources remain. If that observation request loses the connection, the catch path replaces the already-settled result with indeterminate, so Eval skips verification even though execution itself completed

The simplest follow-up is likely subtraction: for settled completed or failed results, close the connection and always call releaseToEnvironment(). An owned ephemeral Host with no residencies already exits with idleGraceMs = 0; a Host with a runtime-resource residency remains available to the verifier. This would remove the diagnostics round trip, the environmentResourcesRemain state, and the control-flow dependency on the runtime-resource label. If the diagnostics query is retained, its failure should at least not overwrite a settled execution result

This does not block approval because the current production caller runs one dedicated Host root inside the Trial environment, and the updated validation and CI are green

中文

已批准。更新后的 PR 符合 #2598 规定的 authority 边界：Eval 只拥有实验语义，Runtime Host 仍是 Maka execution 的唯一 authority

我撤回之前关于 maka inspect 的 finding。#2598 明确规定 maka eval 是唯一公共 Eval CLI，并将删除旧 inspect 入口列入范围，因此这属于有意删除，而不是 Runtime Host 功能的意外回退

另外两个具体边界问题已经干净地解决：短暂存在的 runtime_host_execution artifact 已删除，Domain Module 也已改名为 hosted-execution。暂缓更大的 runner 重写同样合理，因为当前边界确实需要共同封口 continuation、usage 和 verifier 可见的 environment resource

还有一项不阻塞合并的后续改进值得记录。hosted.execution.start 已经返回 completed 或 failed 的 settled projection 后，runHostedExecution() 又发起了一次独立 diagnostics 请求来判断是否仍有 runtime resource。如果这次观察请求断线，catch 路径会把已经确定的结果覆盖成 indeterminate，导致 Eval 在 execution 实际已经完成的情况下跳过 verifier

最简单的后续修法可能是继续做减法：对 settled 的 completed 或 failed 结果，关闭连接后始终调用 releaseToEnvironment()。没有 residency 的 owned ephemeral Host 已经会通过 idleGraceMs = 0 自行退出；存在 runtime-resource residency 的 Host 则会继续供 verifier 使用。这样可以删除 diagnostics 往返、environmentResourcesRemain 状态以及控制流对 runtime-resource 字符串标签的依赖。如果仍保留 diagnostics 查询，至少不应让它的失败覆盖已经 settled 的 execution result

这不阻塞本次批准，因为当前生产调用方在 Trial 环境内为每个 Attempt 使用独立 Host root，并且更新后的验证与 CI 均已通过

AI disclosure: OpenAI Codex assisted with this re-review and approval text. The reviewer verified the current head, addressed changes, and CI state before submission

[liugddx]

对抗性审查结论：建议修改后合入

这个 PR 的架构方向是对的：将 Eval 收敛到 Experiment → Cells → Attempts → Results，并把 Session / Turn / continuation / tools / usage settlement 留在 Runtime Host，符合单一 authority 与奥卡姆剃刀。删除 Headless 而不保留兼容层也符合本次明确的 breaking cutover。

不过，以下问题会使新公开的唯一 Eval 路径在部分场景不符合其契约，建议修复或明确收窄承诺后再合入。

P1 — --cell 与 subject_failed 的公开契约冲突

• packages/eval/src/runner.ts:99
• packages/eval/src/result.ts:68-75
• packages/eval/README.md:19

README 说 --cell 可替换一个 “failed or indeterminate cell”。但 subject_failed 被 selectCellResult() 视为有效结果；随后 runExperiment() 直接跳过该 cell。因此：先产生一个 subject_failed，再运行：

maka eval run <spec.json> --out <existing-run-dir> --cell <cell-id>

不会执行 subject，也不会追加 sequence 2 attempt。

而且即使仅放宽 runner，当前 earliest-valid selection 仍会选择第一条 subject_failed，所以新 attempt 也不会成为 cell result。需要先定义“显式 rerun”与“最早有效结果”之间的语义：

• 如果 subject_failed 是有效 benchmark 测量，最小修复是把 README 收窄为仅重试 infra_failed / indeterminate；
• 如果产品要求人工重跑 subject failure，则需要显式 supersession/rerun policy，而不是隐式绕过 earliest-valid invariant。

请添加回归测试覆盖预置 subject_failed 后传入 --cell 的行为。

P1 — 没有真实、锁定版本的 Harbor/Pier Trial 合约测试

• 生产链：packages/eval/src/harness-executor.ts:194-233 → packages/eval/harbor/run_trial.py:10-30
• 现有测试：packages/eval/src/__tests__/harness-executor.test.ts、packages/eval/harbor/test_relay_agent.py

现有测试使用手写 fake executable、Node TCP peer 或 fake harbor/pier Python module；没有通过实际锁定版本运行 TrialConfig.model_validate_json()、Trial.create()、Trial.run()、Agent import、native verifier/finalizer 和 result.json 契约。

这意味着任一真实框架契约不兼容都可能让完整 cohort 全部失败，而 CI 仍然全绿。请为 Harbor、Pier 各增加一个离线、无真实 provider 凭据的最小 native Trial smoke，覆盖 relay subject execution、verification 与 finalization。

P1 — Windows host 上原生 Harbor/Pier runner 会在 Trial 创建前失败，或至少应 fail-fast

• packages/eval/harbor/run_trial.py:20-22
• packages/eval/src/harness-executor.ts:213-220, 402-414

runner 无条件使用 loop.add_signal_handler(SIGINT/SIGTERM, ...)，Windows 默认 asyncio event loop 不支持该 API；同时清理路径依赖 POSIX signal 语义。Windows 用户运行 maka eval run ... 会在真正创建 Trial 前失败并将 cell 记为 infra failure。

请二选一：

1. 明确 Eval native framework execution 仅支持 POSIX host，并在 CLI/executor 入口 fail-fast 给出可操作诊断；或
2. 实现 Windows 兼容的 signal/cancellation 路径，并添加 Windows 启动/取消 smoke。

P2 — abort 在 canonical terminal state 之后仍会改写结果

• packages/runtime-host/src/server/hosted-execution-runner.ts:73-75

当 turn 已经由 Runtime Host 确定为 completed 或 failed，但 AbortSignal 在等待 residency settlement 或读取 usage 时触发，当前代码会基于 signal.aborted 把投影改为 cancelled；真实 failed 的 failureReason 也会被抹掉。

终态一旦由 Runtime Host 确定，后续 abort 应只影响清理策略（例如等待全部 residencies），不能覆盖 canonical business result。请增加 completed/failed terminal snapshot 后触发 abort 的回归测试。

P2 — wrapper / cohort 测试给出了过强的信心

• packages/eval/src/__tests__/cli-cohort.test.ts:70-116
• packages/eval/src/__tests__/cli-cohort.test.ts:155-170

wrapper 测试对子进程使用 /usr/bin/true，并只在退出后检查文件；它没有证明 child 运行期间实际收到 OPENAI_BASE_URL / ANTHROPIC_BASE_URL、Reasonix 配置和 containment policy。应使用 recording fixture，在 child 生命周期内断言环境变量和策略/配置文件可见，退出后再断言临时 credential 被删除。

cohort 测试只检查数量并验证前四个 cell。建议至少遍历完整 cohort 执行 adapter/spec validation；最好让上述真实 framework smoke 覆盖 task path、verifier 与 finalizer 的契约。

非问题 / 不建议做的重构

• 不建议让 Eval 直接构造 Runtime Session/Turn，或把 Experiment/Cell/Score 概念引入 Runtime/Core；这会破坏本 PR 最重要的 authority boundary。
• 不建议为已删除 Headless API 再加兼容层；会重新引入双 authority 和历史读写路径。
• 未发现可高置信安全删除的新增测试。问题是关键边界 mock 过多而非测试冗余。

在修复上述 P1 并补足关键真实依赖证明后，我认为该 PR 的核心架构可以保留，无需再做大规模重构。

[Astro-Han]

@liugddx Thanks for taking the time to review this, especially after the PR had already merged. I checked each point against the merged code and the local run records. There are a few useful follow-ups here, but I do not think any of them is P1.

The --cell behavior is intentional. #2598 only allows replacement for infra_failed and indeterminate attempts. A subject_failed attempt is a valid result and stays selected. The README says “failed,” which is too broad. I will fix that wording and add one regression test. No supersession policy is needed.

The PR description missed an important detail: Harbor 0.20.0 was run end to end locally. The smoke covered the real Trial lifecycle, relay Agent, verifier, finalizer, and result.json. Eleven attempts completed with reward 1.0, and a larger Harbor run is in progress. So the Harbor smoke finding does not hold. Pier has not had the same run, but that is missing evidence rather than a known failure. I do not plan to open a generic Harbor/Pier smoke Issue.

The Windows problem is real, but I would rate it P2. It fails before the Trial starts and does not corrupt results. For the first version, we can reject non-POSIX hosts with a clear error.

The abort-after-terminal case is also real and belongs in #2662. Once Runtime Host has returned completed or failed, a later abort should not change that result.

I will keep the follow-up small: #2662, plus one Eval PR for the README wording, the --cell test, and the POSIX check. I do not plan to expand the wrapper or cohort tests without a concrete failure.

中文对照

@liugddx 感谢你在 PR 合并后仍花时间做这轮审查。我重新对照了合并后的代码和本地运行记录。这里有几项值得处理，但我认为都不到 P1。

--cell 的行为是有意设计。#2598 只允许替换 infra_failed 和 indeterminate attempt；subject_failed 是有效结果，仍应被选中。README 写成了 “failed”，范围太宽。我会修正文案并补一条回归测试，不需要引入 supersession policy。

PR 描述漏掉了一个重要信息：Harbor 0.20.0 已经在本地完整跑过。Smoke 经过了真实 Trial lifecycle、relay Agent、verifier、finalizer 和 result.json，其中 11 个 attempt 完成并得到 1.0 reward；更大的 Harbor 实验也正在运行。因此 Harbor smoke 这条 finding 不成立。Pier 没有跑过同等验证，但这只是缺少证据，不是已经发现故障。我不会为此开一个笼统的 Harbor/Pier smoke Issue。

Windows 问题成立，但我会定为 P2。它发生在 Trial 启动前，不会污染结果。第一版在非 POSIX host 上报清楚错误即可。

Terminal 后 abort 的问题也成立，应并入 #2662。Runtime Host 一旦返回 completed 或 failed，后续 abort 不应再改变结果。

后续会保持精简：处理 #2662，再开一个小型 Eval PR，修正 README、补 --cell 测试和 POSIX 检查。没有具体故障前，不扩大 wrapper 或 cohort 测试。

AI disclosure: OpenAI Codex helped trace the reported paths and draft this reply. I checked the conclusions against the merged source, #2598, and the retained Harbor run records.