RFC: headless agent experiment lab (@maka/lab)

[Astro-Han]

Motivation

Today we tune agents by feel — swap a model, tweak a prompt, eyeball the result. As agents get more complex (multi-tool, multi-step), that hits a ceiling fast. We want a way to treat an agent configuration as something testable, measurable, and comparable — run a config against a task, record exactly what it did, score it, and compare configs head-to-head.

Most of the foundation already exists. @maka/runtime is Electron-free and headless-capable (RuntimeRunner takes an InvocationRequest → InvocationResult), and runtime-events.jsonl is already a canonical execution trajectory. What's missing is (1) a headless process entry point and (2) the experiment layer on top.

Goal & non-goals

MVP = the shared engine: Config × Task → run in sandbox → trajectory → score → compare. First task type: coding tasks scored by running tests (SWE-bench style). This engine is the model-comparison feature for free.

Explicitly out of MVP scope (all pure additions later, no debt): self-evolution / optimization loop · LLM / rule evaluators (MVP = command/test evaluator only) · Docker backend & real benchmark task packs · Inspect AI adapter · network allowlist · any Electron UI · a general-purpose third-party headless entry point.

Design: three orthogonal contracts

The core idea is experiment = Config × Task, so the two must be separable. Three contracts in @maka/core:

• Task — instruction + initial workspace + verification (test command + FAIL_TO_PASS/PASS_TO_PASS semantics). Fields mirror the SWE-bench instance schema so a SWE-bench instance maps in losslessly. Verification lives on the Task, not the Config — so a config under test cannot grade itself (essential for honest benchmarking).
• Config (the variable under test) — does not invent a new format; references Maka's existing model/connection selection, toolset, and params, plus a systemPrompt override. The "toolset" is a capability set, not an interactive permission policy.
• Execution — in the MVP this is not a config field, it's a single non-overridable safe default. It graduates into a contract later (a pure addition) when a Docker backend lands.

Plus ResultRecord (one per run: pass/fail + cost / duration / steps).

Safety model

The lab is a separate trust context, physically isolated from the interactive permission model. Maka's interactive ask/permission flow does not enter the lab — so "non-interactive batch" and Maka's "never auto-approve" stance don't conflict; they're two different trust models, not a relaxation of the existing one.

Inside the lab, tools auto-approve, and safety comes from isolation, not asking. The non-overridable safe default per run:

• a throwaway workspace copy (never reuse an agent-mutated dir)
• network off by default (MVP fixtures pre-vendor deps; a network allowlist is a named later addition)
• env allowlist (no host secrets)
• timeout + process-tree kill + output cap + temp cleanup
• evaluator runs separately after the agent finishes, using hidden / re-injected test assets (so the agent can't game the tests)

Packaging & output

• New package @maka/lab (depends on core + runtime + storage) for orchestration + scoring + compare.
• A thin CLI bin with experiment-only verbs (run, compare). Peer to desktop, no mutual dependency. (@maka/lab ≠ "headless" — headless is a property of @maka/runtime; the lab is its first consumer. A general third-party headless entry point would be a separate layer.)
• Reuses the existing run dir (sessions/<id>/runs/<runId>/) and runtime-events.jsonl.
• ResultRecord JSONL is canonical truth; MVP derives a markdown comparison table (git-diffable, shareable) + a terminal table. HTML/UI are later derived views that never touch the data.

End-to-end: maka-lab run → read Task + a set of Configs → each combo runs once in the safe sandbox (via RuntimeRunner) → trajectory to runtime-events.jsonl → evaluator produces pass/fail + metrics → write ResultRecord → derive markdown table.

Proposed first step

A walking skeleton: single Config × Task end-to-end on one hand-built fixture — the three contracts (minimal), the safe sandbox default, one RuntimeRunner run, the command evaluator, one ResultRecord. No matrix / compare / CLI polish yet. The contracts are easy; the integration (sandbox + headless RuntimeRunner + evaluator composing into one real run) is where the surprises hide, so de-risk that first.

Open questions

1. Comparison axes for the first experiments — model × systemPrompt enough, or also vary toolset/params early?
2. Network-off-by-default for MVP fixtures — acceptable, or is a minimal allowlist needed sooner?
3. Does mirroring the SWE-bench instance schema on Task match how we'd want to ingest real benchmark packs later?

---

Prior art surveyed: jeju · SWE-bench · Inspect AI

[likun666661]

I agree with the direction, but after the Terminal-Bench / Harbor spike I think this RFC needs one important correction before we treat it as the final shape.

The lab is not just a headless Config x Task -> one RuntimeRunner invocation -> evaluator batch runner. For real benchmarks it is a second Maka product shape: an autonomous task agent. The interactive product is human-driven: the user sends the next message when the agent gets stuck. In a benchmark, there is no human continuation, so the agent loop has to drive itself until submit/pass/budget-exhausted.

Concretely, the current difficulties we hit are:

1. Single invocation is not enough.
   The current walking-skeleton shape can run one instruction and then score it, but Terminal-Bench exposed the missing loop: attempt -> observe/self-check -> continue -> final submit. AiSdkBackend.maxSteps only covers tool calls inside one model invocation; it is not the benchmark continuation loop.

2. Self-check and final verifier are different contracts.
   The final evaluator must stay hidden / re-injected after the agent exits. But an autonomous task agent also needs a visible or task-derived self-check signal that can be fed back into the same session. If we only run the hidden verifier after exit, every repair opportunity is lost.

3. Execution/isolation probably has to become first-class earlier.
   The RFC says Execution can stay a non-overridable default in MVP and graduate later. The Harbor spike suggests the execution boundary is not optional plumbing: permission policy, cwd, network/env, process lifecycle, and tool execution all depend on whether we are in an isolated benchmark container. In the local spike I made real backends fail closed unless the caller supplies explicit external isolation.

4. Interactive permission policy must not leak into the lab.
   The RFC is correct that lab safety should come from isolation, not interactive approval. We saw the inverse problem in practice: interactive-style denials block normal benchmark operations such as rewriting files, removing stale outputs, chmod, or starting/killing task processes inside the isolated container.

5. Result taxonomy matters.
   A failed/incomplete agent run should usually be benchmark data with score 0, not an infrastructure exception. Infra failures should still be exceptions. We need this distinction in ResultRecord/metadata, otherwise benchmark means get polluted by harness errors or, worse, false completed passes.

6. Placeholder verification is dangerous.
   In the Harbor adapter the inner headless verification had to be a placeholder so the generic runner could execute, while Harbor owns the true final score. That placeholder must never be reported as task success. The result schema should make the scorer identity explicit.

So I would revise the product boundary like this:

• @maka/runtime: interactive/session runtime and event ledger.
• @maka/lab or equivalent: autonomous task product surface.
• Task: instruction, workspace, final scorer, optional visible self-check contract.
• Config: model, prompt, tool/capability set, params.
• Execution/Isolation: first-class non-user-overridable policy for the run, including executor, env/network policy, timeout, permission mode, and cleanup.
• AutonomousTaskController: owns the loop: run agent turn, collect observations/self-check output, continue in the same session until pass/submission/budget/failure.
• ResultRecord: final score plus attempts, self-checks, trajectory refs, token/cost stats, and failure kind (scored_failure / agent_incomplete / infra_failure).

This keeps the RFC's core idea, but avoids building a lab that only measures one-shot task attempts. Terminal-Bench is a useful first consumer because it forces the separation between agent-owned loop, visible self-check, hidden final verifier, isolation policy, and benchmark accounting.

[Astro-Han]

Superseded by #45 — the Terminal-Bench/Harbor spike showed the headless lab RFC's 'one-shot batch runner' framing was too narrow. #45 reframes this as a separate Autonomous Task Maka product shape and carries the full design. Closing this in favor of that thread.