fix(eval): make the subject isolation contract hold

[Astro-Han]

Summary

packages/eval/README.md states that the subject receives only what it needs to trust the per-cell CA, that all its traffic is forced through the audited proxy, and that the proxy blocks benchmark and public-solution contamination URLs. Five places from #2947 let that contract silently not hold.

The CA private key was readable by the subject. mitmproxy's confdir was the shared state volume, so the volume the subject mounts to trust the CA also carried mitmproxy-ca.pem and hits.jsonl, which records which of the subject's own requests were blocked. The proxy now creates the CA before it serves, keeps confdir and the audit log container-private, and publishes only mitmproxy-ca-cert.pem into a certificate-only volume, by rename so a restart cannot expose a truncated certificate. Publishing before the port opens means the health gate cannot release main against a missing or left-over certificate. The subject-visible path and containerCaPath are unchanged.

The namespace policy accepted ICMP unconditionally while rejecting every other non-TCP protocol. Harbor needs no ICMP here, so both accepts are gone and no narrowed exception replaces them.

The subject could forge the sidecar's packet mark and bypass the proxy entirely. The policy returned and accepted on that mark as a trusted identity — but the sidecar shares the subject's network namespace, so a mark the sidecar can set is one the subject can set too. Verified in a real cell before the fix: setsockopt(SOL_SOCKET, SO_MARK, 114514) followed by a direct connection to 1.1.1.1:443 succeeded, skipping both the NAT redirect and the filter chain (Docker grants NET_RAW by default, and on Linux 5.17+ NET_RAW alone suffices for SO_MARK).

The exemption also protected nothing: proxy-only mode empties gost's allowlist, so gost forwards no traffic that would need it. Both rules are therefore deleted rather than defended — removing them cannot widen the policy, since traffic gost would have carried is refused in this mode either way.

The subject could also send below the policy entirely. The rules hook the IP output path, so a socket that writes beneath it is invisible to them. With Docker's default capabilities the subject can open an AF_PACKET socket and address a frame to the gateway's hardware address: reproduced in a cell with the full policy applied, where a DNS query reached 8.8.8.8 and its answer came back while ordinary UDP to the same address was rejected. Reported by @M4n5ter, who found it independently. NET_ADMIN is equivalent in effect for a different reason — it can delete the ruleset outright.

The overlay drops NET_RAW, which closes the socket. That alone is not enough: this overlay is merged onto the task's own Compose file, and a cap_add there wins over a cap_drop here, so any task declaring either capability would silently restore the bypass — confirmed with docker compose config and docker run --cap-drop NET_RAW --cap-add NET_ADMIN. So the relay also reads the subject's effective capability set inside the subject container, after the policy is live and before the subject exists, and fails the attempt when it holds NET_RAW or NET_ADMIN; an unreadable capability set fails the same way. Both layers are needed: without the drop every task fails, and without the check the drop is advisory.

Contamination rules never saw the hostname. The search subject was path and query only, so only the exact tbench.ai host matched and https://terminal-bench.io/tasks/answers was allowed through. The host is now searched alongside the path, as a separate field so no rule can match across their boundary.

pier could declare an egress proxy it never enforces. decodeOptions accepted egressProxy for both frameworks, but only Harbor's branch of run_trial.py applies the namespace policy. Decoding a pier spec that declares it now fails with that reason.

Fixes #2958, which covers the first two. The rest come from the approving review on #2947 and from adversarial review of this branch; the remaining follow-ups there are not about isolation and are out of scope.

Verification

test_run_trial_policy.py only asserts that two Python attributes are assigned, which nftables rules failing outright would not disturb — the coverage gap the review named. harbor/test_cell_egress_namespace.py brings up the checked-in Compose overlay and network-policy against a minimal Harbor-shaped base, with a sidecar sharing the subject namespace, and asserts the contract in a real cell namespace. It is opt-in (MAKA_EVAL_EGRESS_NAMESPACE_TEST=1).

Every negative assertion is vacuous on a host that cannot reach the target anyway, so each target's reachability is asserted before the policy exists. Failing there is deliberate: an earlier revision skipped instead, and that skip silently retired one of the assertions for an unrelated reason (see below).

MAKA_EVAL_EGRESS_NAMESPACE_TEST=1 python3 harbor/test_cell_egress_namespace.py — passes on Docker 29.7.2, all eight assertions executing. With the policy applied: ls -A /opt/maka-egress shows exactly mitmproxy-ca-cert.pem; explicit-proxy HTTPS succeeds; curl --noproxy '*', direct-IP TCP, a forged SO_MARK, a link-layer AF_PACKET frame, external UDP and external ICMP all fail; the loopback provider proxy and Docker DNS still work.

Every assertion was checked against the mutation it exists to catch, and fails on it:

• restore the accept lines in network-policy → [external ICMP] AssertionError: 0 == 0
• restore the meta mark rules → [forged sidecar packet mark] AssertionError: 'reachable' != 'blocked'
• remove cap_drop: [NET_RAW] → [link-layer AF_PACKET] AssertionError: 'reachable' != 'denied'
• drop NET_ADMIN from the relay's bypass set → the capability contract test fails
• remove the TCP redirect rule → [curl --noproxy] AssertionError: 0 == 0 and [direct IP TCP] AssertionError: 'reachable' != 'blocked'
• point confdir back at the shared volume → the listing becomes ['hits.jsonl', 'mitmproxy-ca-cert.cer', 'mitmproxy-ca-cert.p12', 'mitmproxy-ca-cert.pem', 'mitmproxy-ca.p12', 'mitmproxy-ca.pem', 'mitmproxy-dhparam.pem']
• search the path alone → the new host-only URLs return None instead of a rule
• accept egressProxy for any framework → ✖ pier cannot declare an egress proxy it never enforces
• report only CapEff from the probe → the bounded-capability service is admitted, live and in the contract test
• make the namespace evidence a constant → the out-of-namespace service is admitted, live
• move cap_drop or the read-only flag onto the proxy service → the compose assertions fail
• answer deny-all with the proxy-only ruleset → [no-network] AssertionError: 0 == 0
• keep meta l4proto != tcp reject → a connection opened before the policy keeps serving requests after it

Also run: root npm run build, npm run test:dist in packages/eval (31 node tests, all Python suites), npm run format:check at the root. No full-repository test run.

Two things worth confirming outside this change. The policy's priority dstnat on a nat output chain needs nftables 1.1 or newer; Debian bookworm's 1.0.6 rejects it outright, which is why the test sidecar is built on trixie — Harbor's own sidecar image is outside this PR, but on 1.0.x the policy would never apply at all. And test:dist runs in CI without MAKA_EVAL_EGRESS_NAMESPACE_TEST=1, so this test currently skips there; wiring a Docker-capable job for it is a separate infrastructure decision.

Review record

Reviewed adversarially in two rounds by Codex, kimi-k3-256k, and a Claude subagent, run independently against this branch; the second round was given no knowledge of the first. Their findings were verified against source and by running the cell before being accepted; severities here are mine, not theirs.

Round one accepted: the forged-mark bypass (Codex only, reproduced in a live cell), the cross-field false positive introduced by the host-matching commit, the all-or-nothing skip in the test baseline, a vacuous compose assertion, the undiagnosable pier error, and several deletions.

Round two accepted, each reproduced first: the cap_drop layer being overridable by the task's own compose (two reviewers), which replaced that fix with deleting the mark exemption; a UDP probe sending a malformed one-byte datagram, whose reply never comes, so that assertion had never once executed; a --noproxy assertion neutered by its own CURL_CA_BUNDLE; the per-capability skip machinery itself, whose failure mode is reporting OK (skipped) for a contract the run never exercised; a README over-claim about which rules match the host; a missing mkdir -p; and a class-cleanup leak.

Rejected with reasons: replacing the capability check with an nftables cgroup match (unnecessary once nothing trusts the mark, and unreliable across cgroup namespaces); an execution-side pier guard for a state the decoder makes unreachable; deleting audit rules that are subsumed for the block decision but distinct in the audit label; and pinning the throwaway test images.

Round five closes the one case @M4n5ter left open when approving: the namespace check read the sidecar proxy's listening socket, which is visible only inside its own namespace but is still a proxy for the claim — a task that declares its own networking and happens to listen on the same port in it satisfied the evidence without sharing the namespace. The claim refers to the namespace Harbor installs the policy in, and the environment interface the relay already holds reaches that service, so the gate now reads /proc/self/ns/net on both sides and requires one identity. An inode has no collision space, so the case is removed rather than made unlikely. Net deletion: the port constant and the cross-language pin that kept it in step with the policy script are gone, and the pin now covers the sidecar service name, which is the coupling that remains. New mutations, each verified to fail: dropping the comparison; reading the policy side from the subject instead of the sidecar; removing the check that an answer is the kernel's own link form, without which two sides that both failed to answer compare equal; and renaming the service on either side of the pin.

Round four came from @M4n5ter's second review plus three fresh adversarial model reviews run against the branch. Accepted, each reproduced first: capability sets other than the effective one, where a non-root subject reports nothing while a cap_net_raw+ep executable takes NET_RAW back from the bounding set; a subject left outside the namespace the policy was applied to, because Harbor respects a task's own networking on the subject service and applies the policy in the sidecar regardless; a connection opened in an earlier phase surviving the policy switch in full, since the redirect is a NAT rule and NAT sees only a connection's first packet; deny-all being answered with proxy access; compose assertions that matched anywhere in the file; and an isolation gate whose refusals had no live coverage at all, because the contract test's environment fabricates the probe output rather than running it.

Deleted rather than fixed: the packet-mark probe. With NET_RAW dropped the subject cannot set a mark, so it had stopped measuring the rule it named and only restated that the capability was gone.

Rejected with reasons: rejecting a task's own networking in the decoder rather than observing the outcome, which would reimplement Harbor's service-selection rule here and let the two drift; replacing the probe with behavioural socket tests, which no task image is guaranteed to be able to run — the gate has sh, sed and grep and nothing more; and calling the gate forgeable, which it is, by a task image that already controls everything else in the cell. What the gate holds against is the subject, which starts only after it has passed, and a task that loses the isolation by accident. The README says so now.

Not fixed here and reported separately: Docker's embedded resolver at 127.0.0.11 forwards names it does not own to the host's upstream resolvers, and the namespace-local exemption that keeps the loopback provider proxies reachable keeps that reachable too. Reproduced by three reviewers and again here — a TXT query for an external zone returned real records while every other path was blocked. Closing it means letting the subject reach the proxy without resolving its name, which changes how the cell is addressed rather than what this ruleset says. A CONNECT tunnel carrying neither TLS nor HTTP likewise reaches no contamination rule and no audit record. Both are stated in the README instead of being left implied.

Round three came from @M4n5ter's review on this PR: the AF_PACKET link-layer bypass, which none of the three model reviewers found and which my own removal of cap_drop had left open — I had weighed that capability only against SO_MARK. Reproduced before fixing, and both the capability drop and the relay's fail-closed check are covered by mutation-verified tests.

A more fundamental fix exists and is deliberately not in this PR: give the subject its own network namespace whose only route out is the proxy, so isolation comes from topology rather than from packet filtering, and no capability can reach past it. That conflicts with Harbor's phase model, which switches network capability between task download, Agent.run(), and verification — topology is a container-lifetime property. It belongs in a separate architecture decision.

This is an AI-assisted review; it does not substitute for independent human review.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — described under Summary above
• No

Generated-by: Claude Code

[M4n5ter]

English

Blocking: main still has Docker’s default NET_RAW capability, so the subject can use an AF_PACKET socket to send traffic below these nftables output hooks.

I reproduced this after applying the new policy: normal UDP was rejected, but an AF_PACKET DNS request reached 8.8.8.8 and received a response without going through the proxy. Dropping NET_RAW prevented the socket from being created.

Please remove NET_RAW from main, fail closed if a task re-adds equivalent privileges, and cover this path in the namespace test. Otherwise the isolation contract this PR aims to establish still does not hold.

中文

阻塞问题： main 仍保留 Docker 默认的 NET_RAW capability，因此 subject 可以通过 AF_PACKET socket，在当前 nftables output hooks 之下直接发送流量。

我在应用新 policy 后复现了该问题：普通 UDP 会被拒绝，但 AF_PACKET DNS 请求仍能直接访问 8.8.8.8 并收到响应，完全不经过代理；移除 NET_RAW 后该 socket 无法创建。

请从 main 移除 NET_RAW，在任务重新添加等效权限时 fail closed，并在 namespace 测试中覆盖该路径。否则本 PR 要建立的隔离契约仍然不成立。

[Astro-Han]

Confirmed, and thank you — this was a real hole and the reproduction was exact.

I reproduced it in a cell with the full policy applied: an AF_PACKET frame addressed to the gateway's hardware address carried a DNS query to 8.8.8.8 and the answer came back, while ordinary UDP to the same address was rejected. The policy hooks the IP output path, so it cannot see that socket at all.

This one is mine. An earlier revision of this PR did drop NET_RAW; I removed it after finding that a task's own compose can declare cap_add and win over an overlay's cap_drop, and I had only weighed the capability against SO_MARK. AF_PACKET is the use I missed.

Fixed in 696ead2 with both layers, since neither is sufficient alone:

• The overlay drops NET_RAW again, which closes the socket.
• The relay reads the subject's effective capability set from inside the subject container — after the policy is live and before the subject exists — and fails the attempt when it holds NET_RAW or NET_ADMIN (which is equivalent in effect, since it can delete the ruleset outright). An unreadable capability set fails the same way. Without the drop every task would fail; without the check the drop is only advisory.

Coverage: the namespace test gains a link-layer probe that builds the IP and UDP headers itself. It asserts denied rather than blocked, so a probe failing for any other reason cannot read as the contract holding, and unlike the other negative assertions it carries no pre-policy reachability assertion — this path is shut by the missing capability rather than by the policy, so it is already closed before the policy exists. The relay check has its own contract tests. Both were mutation-verified: removing the capability drop turns the probe reachable, and removing NET_ADMIN from the relay's set fails the capability test.

One thing worth raising separately: all of this is patching around the fact that the subject shares a network namespace with the sidecar enforcing the policy, which makes them peers. Giving the subject its own namespace whose only route out is the proxy would make isolation a property of topology rather than of remembering every layer a capability can reach past. That conflicts with Harbor's phase model — network capability changes between task download, Agent.run(), and verification, and topology is a container-lifetime property — so I have not done it here. Happy to open an issue for it if you agree it is the right end state.

[M4n5ter]

English

Blocking: three isolation paths remain:

1. The relay only quiesces the recorded process group. A subject child can call setsid(), outlive that group, then regain public network access when Harbor restores the verifier policy. I reproduced the original scope becoming inactive while the detached child remained alive. Cleanup needs a container/cgroup-owned boundary, or the restricted policy must remain until main is stopped.

2. The capability check reads only CapEff. With a non-root agent, the probe can report zero while a file-capability executable reacquires NET_RAW from CapBnd and opens AF_PACKET. Please fail closed on all capability sets that can retain or reacquire NET_RAW/NET_ADMIN.

3. Harbor respects task-authored main.networks/network_mode and excludes main from the generated sidecar namespace overlay, while still applying the policy inside the sidecar. The relay then admits an unrestricted main. Please reject this configuration or attest that main and the policy sidecar actually share the same network namespace before starting the subject.

These paths directly violate the isolation contract this PR aims to establish, so I cannot approve it yet.

中文

阻塞问题： 目前仍有三条隔离绕过路径：

1. Relay 只清理记录的进程组。Subject 子进程可以调用 setsid() 脱离该进程组，并在 Harbor 为 verifier 恢复公网策略后继续运行。我已复现原 scope 被判定为 inactive、但 detached child 仍存活。这里需要由容器/cgroup 持有完整的清理边界，或者在停止 main 前始终保留限制策略。

2. Capability 检查只读取 CapEff。非 root agent 的探测结果可以为零，但带 file capability 的可执行文件仍能从 CapBnd 重新获得 NET_RAW 并打开 AF_PACKET。请对所有可能保留或重新获得 NET_RAW/NET_ADMIN 的 capability 集合 fail closed。

3. Harbor 会尊重 task 自带的 main.networks/network_mode，并将 main 排除在生成的 sidecar namespace overlay 之外，但仍在 sidecar 内应用策略，之后 Relay 会放行不受限制的 main。请拒绝这种配置，或在启动 subject 前确认 main 与策略 sidecar 实际共享同一个 network namespace。

这些路径都会直接破坏本 PR 要建立的隔离契约，因此目前还不能 approve。

[Astro-Han]

Three findings, three different outcomes.

Capability sets other than the effective one — fixed, at the gate. Confirmed before fixing: a non-root container reports CapEff=0000000000000000 while CapBnd keeps NET_RAW, and a cap_net_raw+ep copy of busybox then pings 8.8.8.8 successfully; with the capability bounded away the same executable will not execute at all. The probe now reports every Cap line and the gate refuses on their union. 36066b00c.

A subject outside the policy's namespace — fixed at the gate, not in the decoder. _egress_controlled_service_names in Harbor 0.20 excludes any service declaring network_mode or networks, main included, and the sidecar applies the policy and reports success regardless. I did not take the "reject this configuration" half of your suggestion: that means reimplementing Harbor's own service-selection rule in our decoder, where the two can drift apart silently. The gate instead requires the subject to see the sidecar proxy's listening socket, which is visible only from inside that namespace. Same commit.

Detached children surviving into the verifier phase — not fixed, and here is why. The obvious local patch is to sweep and kill what the process group missed. That would also kill a service a task deliberately left running for its verifier to find, trading a leak for wrong scores. Your two sound remedies — a container- or cgroup-owned cleanup boundary, or keeping the restricted policy until main stops — both need Harbor to own the boundary and both change benchmark-visible behaviour. That is the same topology decision this PR already defers, and I would rather make it once, deliberately, than approximate it here.

Your review then prompted three fresh adversarial reviews of the branch, which found more:

• A connection opened in an earlier phase kept working through the whole agent phase. The redirect is a NAT rule and NAT is evaluated on a connection's first packet. Reproduced: a keep-alive TLS connection opened under allow-all served another request after proxy-only was applied. The filter chain now rejects everything it has not already accepted, which is also one rule where there were two. 3c62205d9.
• deny-all was answered with proxy access, which Harbor's own deny-all does not grant. a89cee62a.
• The gate's refusals had no live coverage: the contract test's environment fabricates the probe output instead of running it, so a probe returning one constant answer passed. The namespace test now has two services the gate must refuse — one outside the policy's namespace, one whose empty effective set hides a bounding set that still carries NET_RAW — and both mutations fail.
• The compose assertions matched anywhere in the file, so moving cap_drop onto the proxy service left them green.
• The packet-mark probe is deleted. Without NET_RAW the subject cannot set a mark at all, so it had stopped measuring the rule it named and only restated that the capability was gone.

Two gaps are stated rather than closed, both reproduced: Docker's embedded resolver at 127.0.0.11 forwards external names, which the namespace-local exemption cannot tell apart from the loopback provider proxies (#2976), and a CONNECT tunnel carrying neither TLS nor HTTP reaches no contamination rule and no audit record (#2977). The README says so now instead of claiming otherwise.

Ready for another look.

[M4n5ter]

English

The previous blockers are addressed:

• The relay now checks every capability set that can retain or reacquire NET_RAW or NET_ADMIN, including the bounding set.
• It fails closed when the subject is outside the policy namespace.
• The nftables rules now reject connections established before the restricted phase, and deny-all no longer permits proxy access.

The namespace check can still be confused if a task-owned namespace already has a listener on the internal port, but that requires a task with custom networking and an exact port collision. It is not reachable in the current pinned Terminal-Bench cohort and can be handled as a follow-up with direct namespace identity comparison.

Approved.

中文

之前的阻塞问题已经解决：

• Relay 现在会检查所有能够保留或重新获得 NET_RAW、NET_ADMIN 的 capability set，包括 bounding set。
• Subject 不在策略 namespace 内时会 fail closed。
• nftables 现在会拒绝限制阶段之前建立的连接，deny-all 也不会再允许代理访问。

如果 task 自带网络且恰好已有进程监听内部端口，当前 namespace 检查仍可能误判；但这需要自定义网络与精确端口碰撞同时发生，在当前固定的 Terminal-Bench cohort 中不可达，可以后续改为直接比较 namespace identity。

批准合并。