[STG-2457] fix(evals): update browse eval harness to CLI v0.9.1 contract

[shrey150]

Summary

Scope note (2026-07-06): this PR has been slimmed down to only the browse CLI contract fix. The V3Evaluator verifier wiring that previously lived here is superseded by #2138, which consolidates verifier grading for both the claude_code and codex harnesses (shared verifierAdapter helpers + Braintrust spans). The two PRs are independent and merge-order agnostic: this one fixes whether the agent can drive the browser at all; #2138 fixes how finished runs are graded.

Scope note (2026-07-07): per Miguel's review, this PR is now scoped to the bench path only (claudeCodeToolAdapter.ts, shared by the claude_code/codex external harnesses). The core/tools/browse_cli.ts change is reverted — that file is a separate, independently-hardcoded consumer of the browse CLI used only by the deterministic core tool-vs-tool comparison suite, and its contract drift runs deeper than this PR's scope (newpage vs tab new, click_xy vs mouse click, -t/-s vs --timeout/--state, etc.). That gets a full sweep in a separate follow-up PR rather than a partial fix here.

Contract fix — browse CLI v0.9.1 (bench harnesses)

The harness drove browse with a stale contract (browse --json … env local). browse CLI v0.9.1 dropped the env subcommand and the global --json flag, so every browse invocation in the harness was broken. Fixed to:

• Per-command mode selection: --local / --remote instead of the removed env subcommand.
• --session <name> on every command to pin the eval's daemon.
• Rely on the CLI's JSON-by-default output (no --json).
• The mode flag is only appended to commands that accept it (skipped for stop / status) and is explicit so a set BROWSERBASE_API_KEY cannot silently auto-select remote when we asked for local.
• runBrowseSetup (which ran browse env … at adapter-prepare time) is deleted — the daemon now starts lazily on the first driver command, so adapter preparation no longer launches a browser at all.

Because codexToolAdapter delegates to the same prepareBrowseCliHarnessAdapter, the fix propagates to both external harnesses.

Files: packages/evals/framework/claudeCodeToolAdapter.ts.

No changeset: @browserbasehq/stagehand-evals is private: true (never published), so it does not trigger a release.

E2E Test Matrix

Command / flow	Observed output	Confidence / sufficiency
pnpm --dir packages/evals build + typecheck	build Done (esm + cli), tsc --noEmit exit 0	Proves the changed file compiles and typechecks against current core.
pnpm --dir packages/evals test:unit (vitest run)	345 passed across 46 test files, 0 failed	Full unit suite green on this branch, including after the core/tools/browse_cli.ts revert.
Adapter E2E smoke — real prepareBrowseCliHarnessAdapter (LOCAL) against <local build> of packages/cli, driving its generated wrapper script	open https://example.com → JSON {"mode":"managed-local","pages":[…"url":"https://example.com/"…]}; snapshot → parsed JSON tree; status (modeless) → JSON OK; cleanup() → {"stopped":true} on the pinned session	Proves the exact wrapper the agent receives works end-to-end on the current CLI contract: explicit --local, pinned --session, JSON-by-default, and the modeless skip for stop/status.
Negative control — browse status --local invoked directly against <local build>	Error: Nonexistent flag: --local (non-zero exit)	Proves the wrapper's modeless skip is load-bearing: passing the mode flag to status/stop would hard-fail.
Compatibility with #2138 — throwaway local merge of miguelgonzalez/verifier-10-benchmark-instrumentation into this branch	Merge clean (0 conflicts — the PRs share no files), typecheck exit 0, vitest 355 passed across 48 files (including #2138's new suites)	Proves the two PRs land independently, in either order, with no interaction.

Linear

STG-2457 — https://linear.app/browserbase/issue/STG-2457/repair-claude-code-browse-eval-harness-rubric-score-via-v3evaluator

Follow-up (full core/tools/browse_cli.ts contract sweep) to be filed as a separate ticket/PR per Miguel's review.

[changeset-bot]

⚠️ No Changeset found

Latest commit: e63c335

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

3 issues found across 5 files

Confidence score: 3/5

• In packages/evals/framework/claudeCodeRunner.ts, V3Evaluator can silently fall back to legacy self-report when judgeModel is overridden without matching judgeClientOptions, which can invalidate verifier-based results while appearing successful — require explicit client-option/model validation (or fail fast) before merging.
• In packages/evals/framework/benchHarness.ts, deriving verifier task IDs from raw instruction text can create unstable or unsafe trajectory paths when instructions include path-like characters, risking misplaced outputs or run failures — sanitize and normalize the fallback ID segment before assigning taskSpec.id.
• In packages/evals/framework/claudeCodeToolAdapter.ts, the new browse wrapper has untested command-specific --local/--remote flag assembly, so a regression here could silently break harness startup/dispatch for affected subcommands — add targeted tests for each browse subcommand/flag combination before merging.

Architecture diagram

sequenceDiagram
    participant H as benchHarness
    participant P as externalHarnessPlan
    participant TA as claudeCodeToolAdapter
    participant BCLI as browse CLI v0.9.1
    participant CR as claudeCodeRunner
    participant VE as V3Evaluator
    participant LLM as Judge Model (Gemini)
    participant RC as RubricCache

    Note over H,RC: Claude Code Eval Flow with Verifier

    H->>P: buildExternalHarnessTaskPlan(input)
    P-->>H: plan (includes precomputedRubric, expectedAnswer)

    H->>H: buildClaudeCodeVerifierConfig(plan, logger)
    Note over H: Check EVAL_CLAUDE_CODE_VERIFIER env
    alt Verifier enabled (default)
        H->>H: Resolve judge model & API key
        H->>H: Create V3 carrier (disableAPI: true)
        H->>H: Build taskSpec with rubric/answer
        H-->>CR: verifier config (v3, taskSpec, judgeModel, judgeClientOptions)
    else Verifier disabled
        H-->>CR: verifier undefined
    end

    H->>CR: runClaudeCodeAgent(plan, model, verifier, ...)

    CR->>TA: prepareBrowseCliHarnessAdapter(input)
    Note over TA: Build wrapper script with new CLI contract
    TA->>TA: Write wrapper bash script
    Note over TA: Uses --local/--remote, --session, no --json
    TA-->>CR: tool surface, adapter cleanup

    CR->>CR: Spawn agent process (Claude Code)
    Note over CR: Agent uses browse via wrapper script

    CR->>BCLI: browse (via wrapper) with new args
    Note over BCLI: e.g., browse --local --session eval-sess-123 goto url

    alt Verifier config present
        CR->>VE: V3Evaluator(v3, { backend:"verifier", modelName, modelClientOptions })
        VE->>VE: Determine rubric
        alt precomputedRubric provided
            VE->>VE: Use taskSpec.precomputedRubric
        else generate rubric
            VE->>RC: getOrGenerateRubric(dataset, instruction, judgeModel)
            RC->>LLM: Generate rubric
            LLM-->>RC: rubric
            RC-->>VE: rubric
        end
        VE->>LLM: Judge trajectory using rubric
        LLM-->>VE: EvaluationResult (outcome, process, evidence)
        VE-->>CR: result
        CR-->>H: TaskResult with verifier score
    else No verifier (legacy)
        CR->>CR: Parse agent self-reported EVAL_RESULT
        CR-->>H: TaskResult with self-report score
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

2 issues found across 1 file (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

2 issues found across 1 file (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[shrey150]

No changeset needed: this change is scoped entirely to @browserbasehq/stagehand-evals, which is a private package ("private": true) and not published, so there is no release/version bump to trigger. Both cubic nits addressed in 821d497 (gateway judge credential resolution + focused verifier-config regression tests; unit suite 345 → 349).

[shrey150]

@miguelg719 slimmed this down per our thread — the branch is now a single commit on latest main containing only the browse CLI v0.9.1 contract fix (claudeCodeToolAdapter.ts + core/tools/browse_cli.ts). All the V3Evaluator wiring, judge plumbing, and verifierConfig.test.ts are dropped in favor of #2138.

A few things verified while slimming (details in the updated PR body):

• Independent, either merge order works: the contract fix un-breaks browse invocations for the harnesses that already exist on main — it is not a no-op without feat(evals): add verifier benchmark instrumentation #2138 (runs just stay self-report-graded until feat(evals): add verifier benchmark instrumentation #2138 lands).
• Compat-tested against feat(evals): add verifier benchmark instrumentation #2138: throwaway local merge of your branch into this one → 0 conflicts, typecheck clean, 355/355 evals unit tests (including your new suites).
• Bonus for feat(evals): add verifier benchmark instrumentation #2138: this PR deletes runBrowseSetup (browse env … no longer exists in the CLI), so adapter prepare no longer invokes browse at all — which also shrinks cubic's "failed adapter prep leaves a browser session behind" finding on your PR to a temp-dir-only concern once this merges.
• One heads-up on judge config: with the wiring dropped here, feat(evals): add verifier benchmark instrumentation #2138's judge is fixed at V3Evaluator's default (google/gemini-2.5-flash via GEMINI_API_KEY/GOOGLE_GENERATIVE_AI_API_KEY). The judge-model override + gateway key routing we had here doesn't exist there — fine for now (the mis-keying bug can't fire without an override path), just flagging in case the A/B wants a non-Gemini judge later.