feat(headless): RSI prompt-optimization loop, validated end-to-end (#64) - #149
Merged
Conversation
…r-consistent The prompt-optimization controller round-trips two values it reads back from the Harbor cell output, but the ai-sdk cell path broke both, so every task would fail plumbing classification once wired to the real controller (unit tests passed only because they used self-consistent fake runners): - Prompt hash: harborCellSystemPrompt() prepended a fixed preamble to config.systemPrompt before the runtime hashed it, so the emitted systemPromptHash was stableHash(preamble+candidate) while the controller expects hashSystemPrompt(candidate) -> prompt_hash_mismatch every task. An explicit prompt is now passed through byte-for-byte (the preamble remains only as the default for prompt-less ad-hoc cell runs). This also makes "the prompt being optimized" equal "the prompt that runs". - Cost: builtin pricing has no deepseek-v4-flash entry, so computeCost(null) returned 0 -> costUsd=0 with tokens>0 -> zero_cost_with_tokens plumbing failure, and the budget ceiling would never trip. The cell now honors the same MAKA_TRIAL_*_USD_PER_1M env the Python adapter (trial_pricing.py) already reads, so one pricing source feeds both runtime cell cost and Harbor trial cost. Tests cover explicit-prompt pass-through and the pricing override. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
…-sdk cell Two more prerequisites for running the prompt-optimization loop on the real ai-sdk + DeepSeek path: - Key files: the ai-sdk cell now resolves an API key from `<NAME>_API_KEY_FILE` (e.g. DEEPSEEK_API_KEY_FILE) in addition to the raw env var. The Harbor adapter already forwards the *_API_KEY_FILE path, so the secret can stay in a mounted file and only its path travels through the Harbor CLI / job config — never on a command line. A raw key still takes precedence. (Resolves the deferred AI-SDK _API_KEY_FILE support noted in PR #85.) - Trial pricing reaches the cell: maka_agent.py now forwards MAKA_TRIAL_*_USD_PER_1M (+ MAKA_TRIAL_PRICING_SOURCE) into the in-container cell env, so the cell's pricing override actually activates and emits real costUsd for models absent from builtin pricing (deepseek-v4-flash). Without this the override was unreachable and cost stayed 0. Tests: cell key-file resolution (file + raw-precedence), adapter forwards trial pricing env into _cell_env. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
…e prompt loop Implements the production HarborTaskRunner the fixed-prompt controller injects (previously only test fakes existed). Per task it generates an isolated single-task Harbor JobConfig (agent maka_agent:MakaAgent, MAKA_BACKEND=ai-sdk), runs `harbor run --config`, and parses results: - reward from the per-trial verifier/reward.txt (the per-trial result.json has no reward field), parsed to a finite number; - status/errorClass/promptHash/cost from agent/maka-cell-output.json; - runtimeEventsPath rewritten to the host trial path (agent/runtime-events.jsonl) so the controller's reward-hack scan and structural smoke can read raw events instead of the container-local path the cell records. Infra vs benchmark split: harbor non-zero exit, missing cell output, or missing/empty/non-numeric reward throw HarborInfraError (controller -> infra_failed, excluded from scoring); a present-but-failed cell (e.g. runtime_error) is returned so the controller marks it unscored+eligible. The system prompt is passed verbatim (trailing newline preserved) for the hash round-trip, the API key travels only as a mounted *_API_KEY_FILE path, and each task uses its own run/round/task job dir to avoid cross-reads under the controller's concurrency. Process invocation is injectable (HarborProcessRunner); tests drive the full parse/classify/JobConfig path with a fake that writes real trial-shaped output. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
The prompt-candidate loop only had a scripted meta-agent for tests. Add the real one: - runtime: runOneShotCompletion — a single-turn, tool-less text completion over the same getAIModel + `ai` SDK stack the agent backend uses. Keeps the AI SDK dependency in runtime. - headless: createAiSdkMetaAgent / createAiSdkMetaAgentCompletion wrap it as the loop's MetaAgentCompletion. A JSON-only system instruction plus extractJsonObject (strips ```json fences / surrounding prose) make the model output survive the strict parseMetaAgentResult. The underlying generator is injectable so the loop is testable without a network call. Default model is deepseek-v4-flash (deepseek-chat is a deprecated alias). Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
Compose the existing baseline-calibration, candidate-round, fixed-prompt controller, acceptance-policy, and structural-smoke layers into one unattended driver: calibrate noise bands, then per round ask the meta-agent for a candidate prompt, sweep held-in + held-out via Harbor, scan held-in trajectories for reward-hacking, and KEEP (advance the lineage) or DISCARD (roll the candidate commit back). Harbor and the meta-agent are injected, so the full composition is tested with fakes and no network or containers. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
… default Hard-coded timeouts turn slow-but-healthy tasks into infra failures, and the deprecated deepseek-chat alias is being retired. Make both the cell wall-clock budget (MAKA_CELL_TIMEOUT_SEC) and the in-container Bash per-command floor (MAKA_CELL_COMMAND_TIMEOUT_MS) operator-configurable with safe defaults, and default the model to deepseek-v4-flash in both the adapter and the cell. The adapter forwards the command-timeout floor to the cell; malformed or non-positive timeout values fall back to the default. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
Compose a real RSI run: deterministic id-sorted held-in/held-out partition of cached Terminal-Bench tasks, DeepSeek v4-flash/v4-pro pricing constants, canary-GUID reward-hack verifier patterns (the canary lives only in a task's test files, so seeing it in model output is the reward-hack signal), cached-task discovery, and runPromptOptimizationRun gluing the real Harbor runner + DeepSeek meta-agent + CLI git before the loop. Expensive components stay injectable so the wiring is tested without Docker or the network. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
Adversarial review surfaced several real defects before the first run: - Native DeepSeek would 400: the runner sent a provider-qualified model id (deepseek/deepseek-v4-flash) while also setting MAKA_PROVIDER, and the cell preserves the slash when a provider is set. Strip the provider- matching prefix for both MAKA_MODEL and model_name; a gateway model whose prefix differs (anthropic/... via openai-compatible) keeps its slash. - Secret exfiltration: the cell passed the full env (incl. *_API_KEY / *_API_KEY_FILE) to every task tool subprocess, so a candidate prompt could cat the key. Scrub provider secrets from the tool child env; the LLM backend already captured the key. - Held-out leak: the driver relied on the caller to hide the held-out TSV. Always pass it as a controller-only artifact so the candidate round rejects if it is visible to the meta-agent. - Duplicate cached task ids would silently collide in the controller's event keying; discovery now fails loud. - Unattended hang: harbor run had no wall-clock ceiling. Add a configurable timeout (default 45m) so a stuck Docker/Harbor becomes an infra failure. - Crash-safety: revert a discarded candidate before persisting the discard decision, so the WAL and git HEAD cannot disagree. - Budget: check the cost/infra stop guard before each round so an over-budget baseline cannot start a new candidate. Documents that the canary reward-hack scan is a best-effort backstop. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
A thin Harbor entry that resolves config, builds the seed prompt repo (program.md + system_prompt.md), discovers and partitions cached Terminal-Bench tasks, derives canary verifier patterns, and calls runPromptOptimizationRun, then persists the result and structural smoke report. The DeepSeek key travels as a file path only. Defaults to a 60/20 partition over 10 rounds; cheap-smoke overrides documented in the header. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
…agent The first real run failed immediately: harbor (a uv-installed tool) could not resolve the bare `maka_agent:MakaAgent` import path because the adapter directory was not on its PYTHONPATH, and the runner invoked harbor with an unmodified env. Prepend <repo>/packages/headless/harbor to PYTHONPATH for the harbor process (preserving any inherited value). Verified: harbor's python imports maka_agent (name = maka) only with this set. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
The stream idle watchdog (120s default) was paused only for subagent tools. A regular tool that ran longer than the idle timeout — an apt-get install, a build, an ML training step — left the watchdog armed, so at 120s it fired and aborted the whole invocation with "Model stream idle timeout" (redacted to "Request timed out"), even though the model stream is legitimately silent between steps while any tool runs. In the ai-sdk step loop a tool always executes between model requests, so provider silence during a tool is expected for every tool, not just subagents. Pause the watchdog for all tool executions; the tool carries its own timeout (Bash timeout_ms, the cell command timeout) and the trial/run layer is the outer backstop. Surfaced by the Harbor RSI smoke: a held-in trial whose first command was a ~120s `apt-get install r-base` was killed at exactly +120000ms with 0 tokens and no prompt hash, while a sibling trial that never ran a single >120s command completed normally. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
MAKA_PROMPT_HELD_IN_IDS / MAKA_PROMPT_HELD_OUT_IDS (comma-separated) override the count-based id-sorted partition so a controlled smoke can target known tasks instead of whatever sorts first. Both must be set together, must be disjoint, and every id must exist or the run aborts before spending Docker time. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
The driver calibrated directly on every configured task, so a single baseline trial that did not complete (scored + eligible) across all sweeps tripped the strict completeness check and aborted the whole run — fatal for an unattended 60/20 run over hard Terminal-Bench tasks where occasional non-completion is expected. Wire the existing selectStablePromptTasks into the baseline step: drop tasks that fail to complete across every baseline sweep (completion-only filter — a flaky-pass task's variance is honest noise the band already absorbs), then calibrate and run all candidate rounds on the stable subset. Dropped tasks are excluded from candidate sweeps too, so they neither cost more nor skew a decision. The run aborts only when a whole partition has no stable task left. Candidate-round non-completion was already handled gracefully (lower coverage -> coverage_regressed -> discard); only baseline calibration was fragile. Result gains droppedHeldInTaskIds / droppedHeldOutTaskIds for transparency; the runner logs them. Structural smoke is unaffected: an unscored task_completed still counts as completed and trips no failure (only plumbing failures do). Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
Now that the stream idle watchdog is paused during all tool executions, it no longer incidentally caps a tool's wall-clock time, so every process-spawning tool must carry its own bound. Bash already does (spawn timeout + abortSignal); Grep ran ripgrep via execAsync with neither, so a pathological tree (network mount, /proc, a FIFO) could hang it and — with the watchdog paused — hang the whole turn. Add a generous self-timeout and forward the turn's abort signal so the rg process is killed, not leaked. Found by codex challenge review of the watchdog change. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
…task ids Two robustness gaps from the codex challenge review of the baseline stability filter: - The driver only aborted when a partition had zero stable tasks, so a harness/cache regression that silently dropped most tasks would still calibrate and decide on a tiny, unrepresentative subset and could promote an overfit prompt. Add minStableHeldInTasks / minStableHeldOutTasks (default 1, threaded through runPromptOptimizationRun and exposed as MAKA_PROMPT_MIN_STABLE_HELD_IN / _HELD_OUT) so a run can fail loud when too few tasks survive baseline. - Explicit MAKA_PROMPT_HELD_IN_IDS / _HELD_OUT_IDS accepted duplicates, which would double-weight a task in the pass-rate summary. Reject duplicates up front with a clear error. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
…script The loop already supports an infra-failure-rate stop guard; surface it as an env knob so a long unattended run aborts early if Docker/Harbor breaks mid-run (e.g., the VM dies) instead of burning hours emitting infra_failed events. Left unset by default. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
Some Terminal-Bench tasks are very slow for a small model (many pdflatex / compile / training iterations). With Docker capped at ~8GB the run must stay at low concurrency, so a handful of multi-minute tasks dominate wall-clock and can push a full 10-round run past 12 hours. Add maxStableTaskDurationMs (threaded through runPromptOptimizationRun, env MAKA_PROMPT_MAX_STABLE_TASK_MS): a task whose baseline trial exceeded the cap in any sweep is dropped from the calibrated set and every candidate round via the existing selectStablePromptTasks 'too_slow' path. Unset = no cap (prior behaviour). Calibration stays honest — the cap only trims which tasks the loop iterates on. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
A held-in task whose tests carry no `terminal-bench-canary GUID` string cannot be scanned for reward-hacking, so the controller quarantines every round it completes in — which fails the structural smoke. headless-terminal is such a task and sorts into the default 60-task held-in partition. Drop no-pattern tasks from held-in (an unverifiable task does not belong in the set the meta-agent optimizes); held-out keeps them since it is not reward-hack-scanned. When held-in ids were pinned explicitly, fail loud instead of silently changing the caller's set. Claude-Session: https://claude.ai/code/session_015yqs8KqJo3DB5DtkGeiNTA
Addresses review findings on the prompt-optimization runner: - P1: default the cost ceiling to $30 and validate every guard knob via fail-loud parsers (envFinitePositiveNumber/envRatio/envNonNegativeInt), so a malformed env value throws instead of yielding NaN that passes a `!== undefined` check and silently disables the guard. - P1: exit non-zero when the structural smoke does not pass, so CI and shell callers don't treat a structurally-broken run as success. - P2: scale the min-stable floors with the actual post-drop partition sizes (ceil(size * ratio), at least 1) instead of a flat default of 1; an explicit MAKA_PROMPT_MIN_STABLE_* still wins for cheap smokes. - P2: scan a task's tests/ tree recursively for canary GUIDs so verifier material nested in fixtures (tests/data/…) is found, not silently treated as "no pattern" and dropped from held-in. Extracts the parsers into a pure, unit-tested prompt-optimization-env module and adds behavior tests for the validators and the recursive canary scan. Full headless + runtime suites green.
Resolve the builtin-tools.ts conflict: drop the now-dead BASH_MAX_OUTPUT_BYTES (main's #92 P0 moved output bounding into runShellWithBoundedTail) and keep this branch's GREP_TIMEOUT_MS, which its self-bounding Grep tool still uses. Full runtime + headless suites green after the merge.
… min-stable P1 (blocking): the prompt-optimization-env.ts source was left untracked in the prior commit, so index.ts/runner/tests imported a module absent from the tree — a clean checkout's build would fail. Commit the source. P2: resolveMinStable now rejects an explicit floor of 0. The loop's guard is `selectedTaskIds.length < floor`, so a 0 floor can never trip and would silently disable the stable-sample protection; an explicit value must be a positive integer (cheap smokes already pass 1). The loop-level "0 stable tasks aborts calibration" behavior is already covered in prompt-optimization-loop.test.ts.
DeepSeek pricing was exported from @maka/headless's public index, leaking vendor-specific run config into the package's generic surface. - Delete DEEPSEEK_V4_PRO_PRICING outright — it was defined, exported and tested but never consumed (dead code). - Move DEEPSEEK_V4_FLASH_PRICING into the runner (run-prompt-optimization.mjs), its only consumer; drop the definition from prompt-optimization-run.ts and the export from index.ts. The harbor-adapter wiring test still asserts the constant is present in the runner source. @maka/headless's public API now exposes only the generic loop / Harbor runner / discovery helpers; the genuinely generic, injectable runPromptOptimizationRun and its task-discovery helpers stay (they carry no vendor specifics). The pricing-value unit test is dropped: the figures are operational config, not correctness logic, and the wiring test still pins the constant name.
The DeepSeek pricing object now lives in the plain-JS runner, so it lost both the HarborTaskPricing type-check and the value unit test. A mistyped rate field would leave it undefined and the runner would silently emit wrong/zero costUsd. Assert each canonical rate field is a finite, non-negative number at startup — before any Docker time. The field-name -> MAKA_TRIAL_* forwarding contract remains covered in harbor-task-runner.test.ts.
MAKA_PROMPT_ROUNDS=0 produced a baseline-only run that trivially passed the structural smoke (minimumRounds 0), contradicting the unattended >=1-round validation the runner exists to perform. Add a fail-loud envPositiveInt parser (rejects 0/negative/non-integer) and use it for rounds. Unit-tested.
…the task #64 LOOP step 6: a thrown Harbor/Docker error is an infra failure and is often transient. Retry the harborRunner once; only record task_infra_failed when the retry also throws. Keeps a single Docker hiccup from poisoning a candidate's pass-eligible rate.
…heck budget at sweep boundaries #64 LOOP steps 7-10: split acceptanceReason into a held-in gate and a held-out gate sharing one decision source. The loop now runs the held-in sweep, evaluates heldInGateReason, and only spends the held-out sweep when the candidate clears held-in (delta beyond the noise band). A candidate that regresses or sits within noise is discarded without paying for held-out. Also enforce the cost ceiling at sweep boundaries (step 14): guard before each baseline sweep and before the held-out sweep, reverting the candidate prompt and breaking out cleanly when the budget is hit, so the loop never overshoots into an extra sweep. Concurrency stays at 4 — the ceiling is handled at the loop's stopGuard, never passed into the controller.
…ge entry The env-parsing helpers (envFinitePositiveNumber, envNonNegativeInt, envPositiveInt, envRatio, resolveMinStable, smokeExitCode) are consumed only by the prompt-optimization runner, not by package API consumers. Expose them through a dedicated "@maka/headless/prompt-optimization-env" subpath export instead of the main entry, and repoint the Harbor runner's import. Keeps the package's public surface to genuine contracts; tests still import the helpers via their relative module path.
…t misconfig Three runner-config fixes from review: - Pin the model to deepseek/deepseek-v4-flash instead of reading MAKA_PROMPT_MODEL. DEEPSEEK_V4_FLASH_PRICING is tied to that model, so an override would let cost and smoke accounting silently use the wrong rates. The loop is contractually a deepseek-v4-flash run; provider/baseUrl stay overridable since they don't change the pricing. - Parse MAKA_PROMPT_MAX_CONCURRENCY with the positive-int parser instead of the finite-positive one, so a fractional value (e.g. 1.5) fails loud at startup rather than being silently floored later in the controller. Widen envPositiveInt to accept an optional fallback (mirroring envFinitePositiveNumber) so the undefined-default knob stays type-honest. - Clarify the cost-ceiling comment: it is a round/sweep-boundary ceiling (checked before each baseline sweep and the held-out sweep), not a hard mid-task cap, so a single in-flight sweep can complete past it.
…s cannot pass the smoke The runner already rejects MAKA_PROMPT_ROUNDS=0 via envPositiveInt, but runPromptOptimizationLoop is callable directly and only rejected negative rounds. A 0-round run is baseline-only and would trivially pass the structural smoke (minimumRounds 0), so it must not share the normal structural-pass semantics. Enforce rounds >= 1 at the public API too.
…controller The CLI env parsers validated env strings, but runPromptOptimizationLoop and runFixedPromptController are public and callable directly, where a NaN, fraction, or 0 would slip past a `value < 1` / `cost >= ceiling` comparison and silently disable a guard or change semantics (rounds 1.5 runs two rounds; a NaN ceiling never trips; minStable 0 disables the stable-task floor). Add a single source of truth for the invariants — numeric-guards (assertPositiveInt / assertNonNegativeInt / assertFinitePositive / assertRatio) — and enforce it at both public APIs: - loop: validate rounds, baselineRuns, zScore, minStableHeldIn/Out, maxStableTaskDurationMs, costCeilingUsd, maxInfraFailureRate, maxConcurrency. - controller: validate costCeilingUsd and maxInfraFailureRate up front; normalizeMaxConcurrency now rejects a fractional value instead of flooring it, and always runs (even when a stop guard forces concurrency to 1) so the value is never left unchecked. - env parsers become string->number then delegate to the same guards, so the CLI and the core enforce one identical contract.
…t a public subpath The runner-only env helpers were reachable as the public subpath export "@maka/headless/prompt-optimization-env", widening the package's contract for internals that only the Harbor runner uses. Move them behind a Node package "imports" entry (#prompt-optimization-env) so the runner can still import them but external consumers cannot — the public surface is just the main entry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Implements the v1 RSI prompt-optimization loop (Issue #64): an unattended loop that lets a meta-agent iteratively rewrite a benchmark system prompt, evaluating each candidate on cached Terminal-Bench tasks via Harbor + a real DeepSeek v4-flash backend in Docker, and accepting/discarding candidates under a rigorous policy.
Pieces
HarborTaskRunnerthat drivesharbor run, with secret-safe API-key files (never argv/env values), per-trial pricing, and configurable cell timeouts.Validation
Full unattended run (59 held-in / 20 held-out, 3 baseline sweeps, 10 rounds, real DeepSeek v4-flash + Docker, ~20.5h):
keep=0is the expected pass criterion for v1 structural validation — it proves the whole pipeline runs end-to-end (generate → commit → evaluate → scan → decide → discard/reset → guards → report) with zero plumbing failures and zero reward-hack leaks. Most discards werecoverage_regressed: a blind prompt edit occasionally makes one task stop producing a scorable artifact, which the policy treats as a hard "do no harm" veto.Build green;
runtime592 tests andheadless286 tests all pass.Follow-ups (not in this PR)
tool_failedrun-trace signal into the meta-agent input so it can avoid coverage regressions (the dominant discard cause). See the enhancement comment on RFC: Harness RSI loop — autonomous system prompt optimization #64.Part of #64.