feat(eval): add the DeepSeek Harness benchmark arm

[Astro-Han]

Summary

Adds DeepSeek Harness as an Eval subject, so its Terminal-Bench 2.1 numbers come from the same executor, metering proxy, and egress policy as every other arm.

It is carried as a complete harness profile, not a patch over a stock one: harbor/deepseek-harness-profile/ declares no bundles, so every entry the model can observe is named in one file and an upstream release adding a plugin cannot widen this arm's tool surface.

One deliberate deviation: upstream configures neither thinking nor reasoningEffort, and the adapter default resolves to reasoning_effort=high on the wire, which would leave this arm reasoning less than every other arm.

Refs #2947

Three changes that reach beyond this arm

• 7fa6d1fc5 — the pinned toolchain fingerprint bound nothing, for every external arm. Verification compared the pinned constant against the value the tree reported for itself in manifest.json, then checked files against that tree's own checksums.sha256 — both pass for arbitrary content carrying an internally consistent manifest. The digest is now recomputed from the manifest on disk.
• 055327076 — the relay's process-group teardown is deleted, for every arm. What the verifier scores is the environment the task was left in, so tearing it down because Eval classified the run as failed makes the measurement depend on the classification, for the condemned half of the sample only. It also never reached a PTY-hosted service, and nothing waited on those processes. 3256b97e2 finishes this on the cancellation path.
• 4852209e5 — the price table read 0.145 / 0.29 / 0.0029 per million input / output / cache-read tokens. DeepSeek publishes 0.14 / 0.28 / 0.0028, which model-pricing.generated.ts already held; [codex] docs(eval): publish nine-arm Terminal-Bench 2.1 leaderboard #3004's costs are 3.45% over, ranking unchanged. 97f117765 also makes Eval price every arm from that one table instead of taking the Maka subject's own figure, kept as a subject-reported-cost artifact so a drift stays visible.

The rest of four review rounds is in the commit history with its reasoning.

Blocking the head-to-head cohort

Both arms run in one task group, one container of each per task at the same moment. #2962 and #2963 must merge first — scheduling, not a request for changes here.

#2963 overlaps in three places, better resolved deliberately than at merge time: it adds isSuccessfulInferenceResponse to the admission branch this PR rewrote into a settled report(); it reads the provider-usage checkpoint from the relay that this branch reads from the host side (schema is now maka.external_provider_usage.v2, exact-key decoder, so a v1 file is rejected rather than misread); and it fences task prompts with --, which bfc248bcf also does here.

Real-machine findings, ported in f7f972349

@hqhq1025 found two defects on VMSS hitting this arm only: a five-minute deadline that killed apt-get mid-dpkg while it waited on an interactive tzdata prompt (DEBIAN_FRONTEND and TZ are now set for every arm), and the harness killing the task's own background services before the verifier ran. DSH goes 61/89 → 65/89, the row in #3004. Those four cells were recovered by the deadline fix, not the service fix — every other arm passes them under a relay that killed each process group at exit.

Review response

@hqhq1025's five findings, all addressed: destructive --out (refuses reserved paths outright, otherwise only removes a directory carrying this build's marker); identity did not pin the dependency tree (npm ci from a lockfile shipped inside the toolchain, version cross-checked); Undici's 300s inactivity timeout (headersTimeout: 0, bodyTimeout: 0); metering read before requests settle (eight getters replaced by one report(), plus a checkpoint the host recovers when the wrapper is killed); profile admission not exhaustive (Record<ExternalProfile, Preparer>, so a missing preparer is a compile error).

Two not acted on. Fail-soft checkpoint writes do weaken "persist before forwarding the chunk", but the alternative fails every transient write error into an infrastructure failure on a request billed regardless. The three-way classification does treat the same evidence differently in two branches, because the evidence differs: in one the relay observed the subject exit, in the other the execution call threw and the subject may never have started.

Verification

Composition compared against upstream request-for-request through a recording proxy: tool names and schemas, system prompt, message sequence, model, max_tokens, thinking, reasoning_effort, stream — all identical.

Harbor trial, sqlite-with-gcov, real container and credentials: completed, 740s, score 1, 43/43 requests admitted, usageComplete: true, real toolchain identity. models and toolNames observed on the wire, not asserted from config.

Also run: 35/35 Eval Node tests across the affected files, test_relay_lifecycle.py (14, 2 skipped), test_relay_contract.py (18), typecheck, lint, format. Not run: the full 89-task cohort — the head-to-head has not been run by anyone yet.

Operating notes

The pinned fingerprint is the digest of checksums.sha256, recomputed from the manifest on disk at verification time. The pinned build is @hqhq1025's /mnt/deepswe/toolchains/deepseek-harness-0.1.0-rc.6-bullseye-v3, linux/amd64 including the compiled pty.node — copy it to the benchmark host rather than rebuilding, since native modules need not come out byte-identical elsewhere and verification fails closed until constant and tree agree. A rebuild re-pins:

node scripts/prepare-deepseek-harness-toolchain.mjs --out <dir> --write

Then export MAKA_EVAL_DEEPSEEK_HARNESS_TOOLCHAIN_PATH. Task groups hold every subject for one task, so each arm adds a container; this spec raises its own concurrency limit accordingly.

Deferred

• Provider finish reasons are not captured for this arm — a stall and a timeout are indistinguishable after the fact. Needs a schema fix(eval): harden external prompt and admission handling #2963 is changing concurrently; costs comparability nothing today.
• checksums.sha256 skips symlinks and verification is one-directional — existing design shared by all eight arms.
• Interrupted-settlement classification is only covered on the recovery side — closing it needs an integration test with a real toolchain.
• DeepSeek moves this model to peak / off-peak billing at 16:00 UTC on 16 August, at 3–5× current rates. A static price table stops being able to express cost then.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — described under Summary above
• No

[hqhq1025]

I found five blockers / follow-ups before using this arm for the full cohort:

1. Destructive --out handling (scripts/prepare-deepseek-harness-toolchain.mjs:54-58). The script resolves the user-provided path and immediately removes it recursively. --out . would delete the repository before any validation. Please build into a temporary sibling and atomically publish it, and refuse /, the repo root/cwd, or a non-empty directory without a toolchain marker.

2. The toolchain identity does not pin the installed dependency tree (prepare-deepseek-harness-toolchain.mjs:27-47,63-70,94-104). npm i @deepseek-ai/dsh@0.1.0-rc.6 resolves many ^ transitive ranges without a checked-in lockfile. The fingerprint covers only the top-level spec; checksums.sha256 is generated by the same build and is not anchored by the checked-in fingerprint. Two different dependency trees can therefore be admitted under the same benchmark identity. Please use a reviewed lockfile + npm ci and bind its digest (or the complete tree digest) into the pinned identity.

3. The provider metering proxy still inherits Undici's 300-second headers/body inactivity timeout (packages/eval/src/harbor-external-subject.ts:635-663). A valid long-reasoning request can be aborted locally before the benchmark-native deadline and become an infra failure. Eval provider transports need explicit inactivity-timeout policy, with the benchmark timeout remaining authoritative.

4. Admission/usage is read before active proxy requests settle, and usage exists only in memory (harbor-external-subject.ts:79-89,715-718; external-subject.ts:87-97). If the child exits or the framework timeout fires while the final provider request is still active, classification can record admittedRequests=0 / usage=null; the later close() waits too late. A hard teardown also loses the final stdout frame. Please settle active proxy operations before classification, persist metering incrementally, and recover it after trial finalization.

5. Registry-derived profile admission is still not exhaustive (harbor-external-subject.ts:339-407). Any future key added to TOOLCHAIN_IDENTITIES is accepted by isExternalProfile, but an omitted prepareProfile branch silently falls into the final ZCode else. Please make ZCode an explicit branch and end with a satisfies never failure.

CI is green and the focused smoke is useful, but these paths are not exercised by that smoke and can change full-cohort reliability or cause destructive local behavior.

[hqhq1025]

I fixed the interrupted-settlement usage gap on top of this PR. The implementation is available as commit 86091aefc on hqhq1025:codex/pr2971-failure-usage.

Root cause: external provider usage only lived in the wrapper process until the final result frame was emitted. Framework timeout, wrapper interruption, or result-frame loss could therefore discard already-confirmed usage. The first real-machine pass also found that /logs/agent files are created under umask 077, so a nominal 0644 create mode still produced a root-owned 0600 checkpoint that the host Eval process could not read.

Fix:

• atomically persist provider metering at request start and settlement;
• explicitly chmod 0644 the credential-free checkpoint after rename;
• recover complete or lower-bound usage/cost from the trial's live agent/ directory;
• record usageComplete, in-flight/missing request counts, and complete vs lower-bound provenance;
• retain verified toolchain identity on recovery paths so scored attempts remain reusable;
• continue to verifier after admitted model work even when the final result frame is missing.

Verification:

• Eval Node tests: 33/33 PASS.
• Python relay/policy/artifact tests: 28/28 PASS.
• Abrupt-wrapper regression test passes and confirms a parseable checkpoint survives SIGKILL.
• Real VMSS success smoke: score=1, 416,361 tokens, $0.0070178898.
• Real VMSS fault smoke: killed the DeepSeek Harness wrapper after the first measured request. The result frame was missing, but verifier still ran, score=0 was recorded, and usage recovered as 1,505 tokens / $0.000058957. CLI finished with cells=1, incomplete=0.
• Fault smoke output: /mnt/deepswe/eval-runs/deepseek-harness-usage-86091aefc-kill
• Deployed bundle: /mnt/deepswe/maka-eval-pr2971-usage-86091aefc

The real-machine result should be treated as authoritative for the container ownership and Harbor lifecycle behavior.

[Astro-Han]

Thanks — all five are addressed. Pushed as three commits, plus one that predates the review but changes what several of these findings touch, so I'll start there.

The arm no longer patches a stock profile. It previously inherited headless and disabled eighteen plugins. Reviewing the disables meant never reviewing the defaults of what stayed, and two of those defaults had silently drifted from upstream: the sandbox ran workspace-write instead of danger-full-access, and every request carried a runtime-context snapshot that upstream suppresses. The profile now declares no bundles, so the tree composes over an empty entry list and every observable entry is named in one file. That is what surfaced both deviations — no amount of review of the disable list would have.

5 — exhaustive profile admission. I went with a Record<ExternalProfile, Preparer> rather than satisfies never. Adding a profile to TOOLCHAIN_IDENTITIES without a preparer now fails at the table (Property 'newarm' is missing in type ...), so there is no runtime path to reach a guard through. Verified by injecting exactly that.

This one earned its keep immediately: moving the preparers into a module-level const put them in the temporal dead zone of the wrapper's own top-level code. It typechecks clean and dies at runtime with Cannot access 'PROFILE_PREPARERS' before initialization. The end-to-end wrapper test caught it; the table now sits above its caller.

4 — metering read before requests settle. Replaced the eight getters with one report() that settles active operations before snapshotting, so an unsettled read is no longer expressible. The recovery half — surviving a SIGKILL — I have deliberately left out: it needs a new persistence contract between the wrapper and the executor, which is not specific to this arm. The framework already records a missing reward as an infrastructure failure. Happy to file it separately.

3 — undici's 300s inactivity timeout. The dispatcher now sets headersTimeout: 0 and bodyTimeout: 0 explicitly on both the Agent and ProxyAgent paths. A reasoning model at max effort can legitimately go quiet for longer than that, and the benchmark's own agent timeout should be the only deadline.

1 — destructive --out. It now refuses /, the repo root, the working directory and the home directory outright, and otherwise only removes a directory that is either empty or holds a previous build (manifest.json present). Verified against all four reserved paths and against a non-empty unrelated directory, whose contents survive.

I did not add build-to-temp-then-publish. It removes no failure mode the guard leaves open, and the publish step still has to delete the previous build, so the destructive operation moves rather than disappears.

2 — identity did not pin the dependency tree. Dependencies now install with npm ci from a reviewed lockfile that ships inside the toolchain, and the build refuses to start when the lockfile and the spec disagree about the harness version. That check fires before docker runs, so a mismatch cannot produce a manifest describing something other than what was installed.

The complete-tree digest half of your suggestion was already in the branch: the fingerprint is sha256(checksums.sha256) over every regular file. An earlier revision had it as a hash of the build script's own spec literal, which made verifyToolchainDirectory compare a value against itself — that is fixed.

The lockfile is 9,035 of the diff's 10,636 lines. Another 900 are the task entries, byte-identical to the eight-arm spec (the CLI has no subject filter, so a single-arm cohort needs its own file). Roughly 700 lines are actual logic.

Also still open, and not from this round: checksums.sha256 skips symlinks, and verification is one-directional — it walks the manifest and never sweeps for extra files on disk. That is the existing design in toolchain-verification.ts and is shared by all eight arms, so I would rather fix it there than only here.

Re-verified end to end after the changes: Harbor trial on sqlite-with-gcov returns completed / score: 1, 43 requests all admitted with usageComplete: true, models: ["deepseek-v4-flash"], toolNames: ["bash", "str_replace_editor"], real toolchain identity sha256:9be54a86…. Eval suites 31/31, typecheck, lint and format clean.

[Astro-Han]

Ported as 81322bda6, with you as the author. Thanks — this is exactly the half I said I would file separately, and the umask finding is one I could not have reached from a laptop.

cherry-pick was not usable: 86091aefc sits on d55b631b5, and the three commits I pushed afterwards rewrote the same region — the eight getters became one settled report(), prepareProfile became a registry with a different signature, and the dispatcher line changed. So I split it by file.

Applied unchanged. external-subject.ts (all of the recovery, provenance and validation), both test files, and the trialPath line in harness-executor.ts — git apply --3way took them clean.

Rewritten. harbor-external-subject.ts. Rather than restore the getters, I extracted a snapshot() that both readers share: report() calls it after settling, the checkpoint calls it at an arbitrary moment. The fields are defined once, so a settled report and an on-disk checkpoint cannot drift apart in meaning, and inFlightRequests becomes the only thing that distinguishes them — zero by construction in a report, possibly positive in a checkpoint. Your separate settle() is then redundant; report() awaits checkpointWrites alongside the active set.

The checkpoint invariants your decoder enforces hold by construction here, because every counter mutation is synchronous with no await between the paired increments — requests/inFlightRequests, admittedRequests/usageRequests, and the settlement pair. A concurrent request cannot observe a half-updated snapshot.

One change to your version. I first put the opening persistCheckpoint() outside the try. If that write throws, the finally never runs, inFlightRequests stays elevated for the rest of the process, and every later usageComplete is false. Your placement inside the try is right; I moved it back.

Verified locally: 33/33 Eval Node tests, including your abrupt-exit regression test running against the rewritten proxy. Typecheck, lint and format clean. I did not reproduce your fault smoke — your VMSS run stands as the authority for the container ownership and Harbor lifecycle behavior.

The README now documents the checkpoint and the complete/lower-bound distinction, so the deferral note in my previous comment no longer applies.

[Astro-Han]

One more thing on the branch: experiments/terminal-bench-2.1-deepseek-v4-flash-maka-vs-deepseek-harness.json (36f8f11b8).

Comparing the single-arm harness spec against the eight-arm cohort means comparing two runs on two occasions, which carries a difference no setting removes. This spec puts both arms in one task group instead — each task runs one Maka container and one harness container at the same moment, same executor, metering proxy, egress policy, verifier and budget. Benchmark, tasks, verifier, budget and executor config are taken unchanged from the existing specs; only the subject list, the toolchain mounts and the concurrency limit differ. 89 tasks, 178 cells, 64 groups so the pair meets the same 128 concurrent trials the eight-arm cohort reaches at its declared 16.

Would you be willing to run it on the VMSS host? I have no benchmark machine on my side — the alternative is standing one up from scratch, which costs more in setup than the run does. You already have the pier environment, the egress proxy image and both toolchains there.

Two things worth flagging if you do:

The toolchain has to be built on whichever machine runs the cohort, and the fingerprint currently pinned in toolchain-verification.ts is from my laptop. Native modules are compiled during the build, so yours will differ and verification will fail closed until they agree:

node scripts/prepare-deepseek-harness-toolchain.mjs --out <dir> --write
npx tsc -p packages/eval/tsconfig.json

--write re-pins it. That value — the one the cohort actually ran under — is what should be committed, so this wants to happen before the PR merges rather than after.

The build compiles node-pty from source inside a pinned linux/amd64 container. Mine ran under Docker Desktop's amd64 emulation; yours will be native, so that step is the one I would expect to behave differently if anything does. It fails loudly inside the container rather than producing a partial toolchain.

If a subset would be more useful than the full 89 first — say 20 tasks at lower concurrency to see the shape before committing the machine time — say so and I will cut one.

[Astro-Han]

@hqhq1025 three things for when you pick this up — two of them are about your PRs, not this one.

Your #2963 and my 81322bda6 solve the same problem at different points in the lifecycle. Both read /logs/agent/<profile>.provider-usage.json under schema maka.external_provider_usage.v1. Yours reads it from the relay with download_file before the container is torn down; mine (ported from your 86091aefc) reads it host-side from the trial's agent/ directory after the fact. These look complementary — relay-side rescues the file before teardown destroys it, host-side survives a wrapper that never got to emit a frame — but they should be reconciled deliberately, not by whoever merges second. I have no stake in which shape wins.

#2963 also adds isSuccessfulInferenceResponse to the admission branch this PR rewrote into a single settled report(), so that hunk will conflict textually. The rewrite does not change admission semantics, only when the snapshot is taken, so your change should drop in cleanly once rebased.

Correction to what I said earlier about the fingerprint. I told you it has to be rebuilt on whichever machine runs the cohort. That is only true of a rebuild. The build product is already linux/amd64 — I checked, bin/node is ELF x86-64 and pty.node is compiled — so it can be copied to the benchmark host and the fingerprint pinned in toolchain-verification.ts stays valid. Rebuilding also works and --write re-pins it, but it is not required, and copying is the cheaper path.

The head-to-head cohort now on the branch is blocked on three of your and our open PRs, which the PR body now spells out: #2960 (the isolation contract is bypassable and contamination rules never matched the hostname), #2962 (the Maka arm gets cut off by a local idle timeout), and #2963 (admission classification). Running before those land produces numbers we would have to discard. #2967 can land after.

So there is no rush on the run itself — the blocking work is those three, not machine time. No hurry on any of this tonight.

One smaller thing from your #2963 that I applied here: fencing the task prompt with --. For this arm it is consistency rather than repair — dsh combines allowUnknownOption with passThroughOptions, so a dash-leading prompt already reached the profile verbatim (checked against the commander build the toolchain ships). It only matters when the prompt exactly matches a known option, where --help alone would print usage and exit.

[Astro-Han]

@hqhq1025 morning summary, so you do not have to reconstruct last night from the thread.

Nothing needs running yet. The head-to-head cohort is blocked on code, not on machine time, so there is no rush on the VMSS host.

Status of the four as of now:

#2960	blocks the run	CHANGES_REQUESTED, waiting on @M4n5ter's re-review
#2962	blocks the run	approved (pullrequestreview-4929701530), CI red on e2e and storybook
#2963	blocks the run	needs you — see below
#2967	does not block	can land after; it changes what we can detect, not what the subject does

#2963 is the one that needs you. It has been at eeec9529a since 14:33 yesterday with no commits or replies after the review, and there is an unanswered P0 on it. I left a comment there as a downstream consumer rather than as a reviewer: the admission widening in harbor-external-subject.ts:762-763 drops upstream.ok as a conjunct, and in this arm admittedRequests is what separates an infrastructure retry from a recorded zero. Widening it turns attempts that would be retried into silent zeros, and in the paired cohort it does that to one arm only, since the Maka arm never goes through this proxy. Two ways out suggested there; I have no stake in which.

Why the other two block: #2960 — the isolation contract is bypassable and contamination rules never matched the hostname, so numbers taken before it cannot rule out contamination. Note the scope is narrower than "then it's clean": per that PR's author, the fixed filter addresses a subject that stumbles onto an answers page, while #2976 and #2977 leave channels outside the audited path entirely, so it stays a blocklist rather than a proof of isolation. #2962 — the Maka arm gets cut off by its own idle timeout, which would depress one arm for reasons unrelated to the arm.

On this PR since your last look: the interrupted-settlement port (81322bda6, yours, rebased onto the settled report()), the head-to-head spec, -- fencing on the task prompt, and a README section stating what the paired cohort does not guarantee — pairing is not preserved by the framework, and loss is biased toward the arm that ran longer, so results have to be re-paired by task rather than averaged per arm.

Also, correcting myself from yesterday: the toolchain does not have to be rebuilt on the benchmark host. The build product is already linux/amd64 including the compiled pty.node, so copying it keeps the pinned fingerprint valid. Rebuilding works too and --write re-pins it, but it is not required.

[hqhq1025]

Real-machine follow-up from the 89-task DSH run: I found and fixed two Eval-specific reliability defects, then reran every affected cell.

Root causes:

1. The maka-eval profile inherited a 300-second timeout for both terminal-bash layers. Under cohort load, apt-get install was killed while dpkg was configuring tzdata, leaving iU packages. Six verifiers then failed before tests with dpkg was interrupted, missing curl, and missing uvx.
2. DSH owns its persistent PTY process tree and kills every descendant during normal shutdown. A task-created nohup Flask service was reachable before DSH exit but gone before the shared-environment verifier. The outer Eval relay also quiesced the complete subject process group.
3. Once the artificial five-minute kill was removed, the underlying package install was shown to be waiting for interactive tzdata, not consuming CPU. The Eval profile therefore also needs a noninteractive package-install environment.

Fix: commit 28ebe0949 on hqhq1025:codex/dsh-eightarm-integration.

• DSH Bash timeout is 3,900,000 ms, beyond Terminal-Bench's longest native subject deadline.
• DSH Eval sets DEBIAN_FRONTEND=noninteractive and TZ=Etc/UTC.
• The toolchain build applies a fail-closed patch to @deepseek-ai/dsh-subprocess-local: only when DSH_PRESERVE_BACKGROUND_PROCESSES=1, normal PTY shutdown closes the shell without killing background descendants.
• The Eval relay preserves the process group only for successful DSH exits. Failed exits, cancellation, transport loss, and framework timeout still quiesce it.
• The patched toolchain is fingerprinted as sha256:a0882b448718ddfb7b64e33a12369c92b0064baf8388fe08a8ff64fe3dd98896.

Regression verification:

• Eval Node: 36/36 PASS.
• Python relay/policy/artifact suites: 33/33 PASS.
• Added red-green coverage for the command deadline, noninteractive package environment, DSH subprocess patch, successful background handoff, and failed-exit cleanup.
• Lint, format, and Eval typecheck PASS.

Real VMSS results using complete-attempt replacement:

• hf-model-inference: 0 -> 1; Flask remained available to verifier.
• polyglot-c-py: 0 -> 1.
• merge-diff-arc-agi-task: 0 -> 1.
• regex-log: 0 -> 1.
• dna-assembly: remains 0; verifier ran and found a real primer-clamp error.
• torch-pipeline-parallelism: remains 0; verifier ran and found a real implementation error.
• extract-moves-from-video: remains 0; native 1800-second timeout, then verifier confirmed no solution file.

All seven verifier logs are free of the former dpkg/curl/uvx setup failure. DSH moves from 61/89 (68.5%) to 65/89 (73.0%).

Artifacts:

• Bundle: /mnt/deepswe/maka-eval-dsh-verifier-fix-74c98a6ac (.bundle-commit = 28ebe0949)
• Toolchain: /mnt/deepswe/toolchains/deepseek-harness-0.1.0-rc.6-bullseye-v3
• Service canary: /mnt/deepswe/eval-runs/deepseek-v4-flash-dsh-service-canary-v3-74c98a6ac
• Six-cell retry: /mnt/deepswe/eval-runs/deepseek-v4-flash-dsh-verifier-fix-retry-6-v4-28ebe0949

[Astro-Han]

Ported both fixes here as f7f972349, rebased from your 28ebe0949. Thanks for running this down on real hardware — the dpkg/tzdata chain is not something the local harness would ever have shown.

Four conflicts, none of them mechanical, so the resolutions are worth naming:

• harbor-external-subject.ts — your branch still has the if/else chain; this one has been rewritten into a PROFILE_PREPARERS table, so the three env values went into the table's deepseek-harness entry rather than into an else arm.
• relay_agent.py — _preserve_process_group_on_exit landed next to _require_constrained_subject, which arrived with fix(eval): make the subject isolation contract hold #2960 after your base. Independent, both kept.
• README.md — the 3-way pulled in a paragraph from your branch that does not exist here; only the nine lines belonging to this commit were taken.
• toolchain-verification.ts — took sha256:a0882b44…. The old value describes the unpatched toolchain and is simply no longer true.

That last one is the only thing needing a decision before the run. The pinned constant now names your deepseek-harness-0.1.0-rc.6-bullseye-v3 build, and native modules do not have to come out byte-identical elsewhere, so the head-to-head has to either use that build or be rebuilt with --write to re-pin. Verification fails closed on the mismatch, so it will not silently run against the wrong tree — but it is worth deciding rather than discovering.

Verified here: Eval Node 34/34, test_relay_lifecycle.py 17 (3 skipped), Eval typecheck, lint, format.

Blocker status after this: #2960 merged, so #2962 and #2963 are what remain. #2962's CI is green now and only the merge gate is left. #2963 is still at eeec9529a with the admittedRequests question open — that one is the last thing pointing at you.

---

两个修复已移植为 f7f972349，从你的 28ebe0949 rebase 过来。感谢真机排查，dpkg/tzdata 这条链本地环境不可能暴露出来。

四处冲突都不是机械冲突，解法说明如下：

• harbor-external-subject.ts — 你那边还是 if/else 链，这边已改写成 PROFILE_PREPARERS 表，三个 env 进了表里的 deepseek-harness 项，而不是 else 分支。
• relay_agent.py — _preserve_process_group_on_exit 与 fix(eval): make the subject isolation contract hold #2960 带来的 _require_constrained_subject 相邻，两者独立，都保留。
• README.md — 3-way 把你分支上这边不存在的一整段也拉了进来，只取了属于这个 commit 的九行。
• toolchain-verification.ts — 取 sha256:a0882b44…，旧值描述的是未打补丁的 toolchain，已不成立。

最后一条是开跑前唯一需要拍板的：pin 的常量现在指向你构建的 deepseek-harness-0.1.0-rc.6-bullseye-v3，而原生模块换机器不保证 byte-identical，所以头对头要么用你那份构建，要么 --write 重新 pin。不匹配时验证会 fail closed，不会静默跑错树，但值得先定而不是撞上。

本地验证：Eval Node 34/34，test_relay_lifecycle.py 17 项（3 skip），Eval typecheck、lint、format 通过。

阻塞项现状：#2960 已合并，剩 #2962 和 #2963。#2962 CI 已绿，只差合并门。#2963 仍停在 eeec9529a，admittedRequests 那条还开着 —— 这是唯一还指向你的事项。

Drafted with Claude Code; the port, the conflict resolutions, and the verification above were reviewed by me.

[Astro-Han]

Requesting review. The PR body carries the full account; this is a reading order for it.

@hqhq1025 — your five findings are answered under "Review response", and your two real-machine fixes are ported in f7f972349. Three of them needed a second and third pass, which is noted there. @M4n5ter — this touches the relay lifecycle boundary you reviewed in #2960 and #2928; the change there is a deletion, described below.

What changed since the last round (3256b97e2..4852209e5, 7 commits). A fourth adversarial review round found that three of the previous round's own fixes were incomplete, each in the same shape:

• 3256b97e2 — the relay's cancellation path still tore down the scope of a subject that had already exited. 055327076 removed this from the ordinary path; cancellation kept doing it. A subject that has exited has nothing to bring to a stop, and what the verifier scores is the environment the task was left in.
• f8ee825bd — the metering proxy latched settled after awaiting a snapshot of in-flight requests, so a request admitted during the drain was never waited for. The commit above makes such arrivals ordinary rather than hypothetical. Admission now closes first; a later request gets 503 and is never forwarded.
• 97f117765 — the Maka adapter preferred the subject's self-reported cost, so the claim in 2ab2f4d87 that both arms bill from one table was false. Eval now prices every arm; what the subject billed itself is kept as a subject-reported-cost artifact.

Cost accounting is more accurate than it was (4852209e5, 0b44f7508). The price table is now checked against DeepSeek's published figures — 0.14 / 0.28 / 0.0028 per million input / output / cache-read tokens, which is what packages/runtime/src/telemetry/model-pricing.generated.ts already held, so the two tables in this repository agree. The usage decoder also reads prompt_cache_hit_tokens, which is where DeepSeek's chat-completions responses put cache reads.

Not in scope here, but timely: DeepSeek moves this model to peak / off-peak billing at 16:00 UTC on 16 August, at 3–5× the current rates. A static table stops being able to express cost then.

Two things still blocking the head-to-head cohort: #2962 and #2963. Neither is a request for changes here.

Local: 35/35 Node across the three affected files, 14 Python (2 skipped), lint, format, typecheck. CI green except the two Windows jobs still running.

[Astro-Han]

@hqhq1025 @M4n5ter Could you take a look at the latest head? Appreciated!

[M4n5ter]

English

LGTM — this looks ready to approve for its stated purpose.

Three non-blocking follow-ups are worth tracking:

• Update the static DeepSeek pricing, or explicitly describe it as normalized cost, before publishing cohort cost results.
• Harden the metering checkpoint boundary: authenticate or isolate it from the subject, bound reads, and chmod the temporary file before rename.
• Define and test a consistent framework-timeout policy for background processes across all arms.

Since #2962 and #2963 still need to land first, please also verify that the final conflict resolution preserves DeepSeek Harness recovery and admission handling.

中文

LGTM——按照这个 PR 的既定目的，我认为可以 approve。

有三个非阻塞 follow-up 值得继续跟进：

• 在发布 cohort 成本结果前，更新 DeepSeek 静态价格，或明确说明它是 normalized cost。
• 加固 metering checkpoint 边界：进行认证或与 subject 隔离、限制读取大小，并在 rename 前完成临时文件的 chmod。
• 为所有 arm 明确定义并测试一致的 framework-timeout 后台进程策略。

由于 #2962 和 #2963 仍需先合入，也请在最终解决冲突后确认 DeepSeek Harness 的 recovery 和 admission 逻辑没有丢失。