[STG-2510] fix: source eval browser skill from packages/cli browse SKILL.md

[shrey150]

Why

The Claude Code eval harness installs a browser skill for the agent it spawns from packages/evals/skills/browser/SKILL.md — a 68-line eval-local copy created once in "Evals v2" (#2011) and never touched again. The real, maintained browse skill lives at packages/cli/skills/browse/SKILL.md (355 lines, ships with the CLI). Checking git history:

• packages/evals/skills/browser/SKILL.md: 1 commit ever (2026-05-01, Evals v2 #2011), never edited since.
• packages/cli/skills/browse/SKILL.md: 10 commits since 2026-06-05, most recently 2 days ago — roughly one edit every 3 days.

That gap already produced a concrete factual error: the most recent CLI-skill commit (#2296, STG-2450) made browse snapshot lean-by-default with --full needed for ref maps, but the frozen eval skill still documented the old always-full behavior. It also never mentioned --verified/--proxies/--auto-connect, browse doctor, retry discipline, tab/network/cdp/mouse/viewport commands, or Browse.sh skill discovery. The harness was testing an outdated mental model of browse.

Design

Single source of truth with an install-time eval addendum:

1. BROWSER_SKILL_SOURCE → BROWSE_SKILL_SOURCE, now pointing at packages/cli/skills/browse/SKILL.md (built from getRepoRootDir(), matching the existing BROWSE_CLI_ENTRYPOINT-style constants).

2. installBrowserSkill → installBrowseSkill now reads the CLI skill and inserts a code-level EVAL_HARNESS_ADDENDUM template literal immediately after the YAML frontmatter (not appended at the end) before writing the combined file. The addendum:

   • States browse is preinstalled/pinned by the harness (no npm install, no --local/--remote/--session — the wrapper injects those).
   • Requires exactly one browse ... command per Bash call (shell operators rejected by the harness).
   • Tells the model to ignore the CLI skill's install/Browse.sh-discovery/cloud/Functions/templates sections — out of scope during evals.
   • Reiterates no repo edits, no non-browse network tools, and the EVAL_RESULT reporting format.

   Why prepend and not append: isAllowedBrowseCommand only checks that a Bash command starts with browse and has no shell metacharacters — it does not restrict which browse subcommand runs. So the addendum's "ignore cloud/functions/skills" instruction is the actual scope-enforcement mechanism, not just a courtesy note, and it needs to be read before the model encounters the CLI skill's concrete (and tempting) examples of those commands, not after. A live smoke run (see below) shows the model reaching for browse cloud fetch and browse skills find once it got stuck on a bot-protected page — evidence this ordering concern is real, not theoretical.

3. Skill name consistency: installed skill dir renamed .claude/skills/browser/ → .claude/skills/browse/ to match the CLI skill's own name: browse frontmatter; all harness prompt/log references to "a project skill named browser" updated to "browse". The stagehand_browser MCP server name (used by the unrelated playwright_code/cdp_code tool surfaces) is untouched.

4. Deleted packages/evals/skills/browser/SKILL.md and the now-empty packages/evals/skills/ directory.

5. Updated packages/evals/tests/framework/claudeCodeToolAdapter.test.ts for the renamed export/path/skill-name, plus new assertions that the installed file contains both the CLI-skill content and the addendum, with the addendum's string index before ## Cloud APIs's index (regression guard against the addendum silently drifting back to append-at-end).

Follow-up (review comment from ajmcquilkin): the hand-rolled regex in insertAfterFrontmatter had already needed a CRLF patch and still failed silently on BOM-prefixed files or a --- line embedded in a YAML multiline string. Swapped it for gray-matter (new packages/evals devDependency, private package, no changeset), but only for boundary detection — matter(markdown) locates where the frontmatter block ends; reassembly still uses the original raw string (markdown.slice(0, markdown.length - parsed.content.length) for the frontmatter, parsed.content for the body) rather than matter.stringify(), since that would re-serialize the YAML through js-yaml and reformat the shipped skill's frontmatter (e.g. its folded description: > block). insertAfterFrontmatter is now exported and directly unit-tested.

No changeset — packages/evals is private, eval-infra only.

Overlap note: this touches claudeCodeToolAdapter.ts; open PR #2299 also touches that file but in a different region (contract fix, not the skill-install path). Trivial rebase for whichever lands second.

Linear: STG-2510

E2E Test Matrix

Command / flow	Observed output	Confidence / sufficiency
pnpm turbo run build --filter=@browserbasehq/stagehand --filter=browse then pnpm --dir packages/evals build	All 4 turbo tasks + evals build:esm/build:cli completed successfully in <worktree>	Confirms the changed adapter compiles against the real CLI/core build artifacts it now depends on (packages/cli/skills/browse/SKILL.md, packages/cli/dist/...).
pnpm --dir packages/evals exec vitest run tests/framework/claudeCodeToolAdapter.test.ts	Test Files 1 passed (1), Tests 17 passed (17)	Covers the renamed export, new install path/skill name, the addendum-ordering assertion, and (as of the gray-matter follow-up below) the frontmatter boundary-detection cases. Narrow to this file.
pnpm --dir packages/evals run test:unit (full evals suite)	Test Files 48 passed (48), Tests 362 passed (362)	Confirms no other test in the package depends on the old browser skill name/path, installBrowserSkill export, or the removed regex helper.
node -e script importing installBrowseSkill from built packages/evals/dist/esm/framework/claudeCodeToolAdapter.js, run against a temp dir	Installed at <temp dir>/.claude/skills/browse/SKILL.md; head matches CLI skill frontmatter (name: browse, full description); tail/body contains ## Eval Harness Addendum positioned before ## Cloud APIs (indices 1054 vs 9018); frontmatter YAML block intact as the first bytes of the file	Direct artifact proof of the exact shape described above — installed path, single-source content, and addendum placement — independent of the eval harness runtime.
Live harness run: EVAL_CLAUDE_CODE_ALLOW_UNSANDBOXED_LOCAL=true EVAL_CLAUDE_CODE_MAX_TURNS=45 node packages/evals/dist/cli/cli.js run b:webtailbench -l 1 -t 1 -c 1 --harness claude_code -e local -m anthropic/claude-haiku-4-5-20251001 (verbose logging on)	Log line Installed browse skill at <temp dir>/.claude/skills/browse/SKILL.md; agent then issued browse open, browse snapshot, browse doctor, browse status, browse stop --force, browse cloud fetch, browse skills find — every one of them accepted by the harness with no "Only browse commands are allowed" / "Only Skill and Bash are allowed" contract denial anywhere in the log. Task itself ended error_max_turns fighting a bot-protected United.com page (unrelated to this change)	Proves the real install → load → drive pipeline works end-to-end with zero contract errors. Task pass/fail is expected to be noisy on this benchmark target and is not the bar here; the pipeline mechanics are what this row proves.
Follow-up (review comment): pnpm --dir packages/evals build (esm + cli) after adding gray-matter as a devDependency and rewriting insertAfterFrontmatter to use it for boundary detection only	Both build:esm/build:cli completed; dist/esm/framework/claudeCodeToolAdapter.js shows import matter from "gray-matter" and the new exported insertAfterFrontmatter	Confirms the new dependency resolves and the adapter still builds against packages/cli/skills/browse/SKILL.md.
node script against the built installBrowseSkill/insertAfterFrontmatter (not source), run in a temp dir	frontmatter byte-identical to source? true; text immediately after the frontmatter is only whitespace before ## Eval Harness Addendum (no leftover body content); ## Cloud APIs still present after it; direct insertAfterFrontmatter calls for the no-frontmatter fallback and an embedded --- inside a YAML multiline string both produced correctly-bounded output	Real artifact proof (not unit-test mocks) that swapping in gray-matter didn't change the installed file's byte layout — the specific regression the reviewer's suggested library could introduce via matter.stringify(), which this implementation deliberately avoids.
New unit tests in claudeCodeToolAdapter.test.ts: LF/CRLF/BOM-prefixed frontmatter, a --- line inside an indented YAML > block, no-frontmatter fallback, unterminated/invalid-YAML fallback (gray-matter throws; now caught), and a byte-identical-to-source frontmatter assertion via installBrowseSkill	All pass as part of the 17/17 and 362/362 runs above	Locks in the exact boundary-detection contract the reviewer flagged as fragile; the byte-identical assertion is a standing regression guard against ever switching to matter.stringify().
pnpm --dir packages/evals run lint (prettier + eslint + tsc)	All matched files use Prettier code style!; eslint clean; tsc --noEmit clean	Confirms the gray-matter typings (matter.GrayMatterFile<string>) satisfy the package's strict TS config and formatting rules.

Future work

A separate in-flight PR (shrey/cli-skills-show, not yet on main) adds browse skills show (prints the bundled skill to stdout) plus a "Start here (for AI agents)" pointer in browse --help, so a real sandboxed agent with no eval scaffolding could self-discover the skill instead of having it handed to it.

This PR's shape — inject the real skill + eval addendum into .claude/skills/browse/ at prepare time — stays the right default: it's the conventional eval pattern (benchmarks inject tool docs deterministically, the agent can't skip it), and it mirrors the actual supported CLI workflow (a user who already ran browse skills install). It also can't reference browse skills show today since that command doesn't exist on main yet.

Once browse skills show ships, "agent discovers the skill itself via browse --help" becomes a good second, more sandbox-realistic eval arm (skill-injected vs. self-discovered A/B), not a replacement — worth noting for whoever builds it: isAllowedBrowseCommand in this file only checks for a browse prefix and absence of shell metacharacters, so browse skills show already passes that gate today with zero adapter changes needed for permissions. The one thing that arm would still need is its own way to deliver the eval-specific overrides (session/environment pinning, one-command-per-call, out-of-scope sections) — browse skills show would print the bundled skill verbatim, so that arm likely wants the same EVAL_HARNESS_ADDENDUM content delivered via the top-level task prompt instead of a pre-installed skill file.

🤖 Generated with Claude Code

---

Summary by cubic

Switches the eval harness to install the real browse skill from packages/cli/skills/browse/SKILL.md, injecting a small eval-only addendum right after the frontmatter. Fixes stale guidance and keeps eval behavior aligned with the CLI (Linear STG-2510).

• Bug Fixes

  • Source browse skill from packages/cli/skills/browse/SKILL.md to stop drift and correct outdated docs/behavior.
  • Insert EVAL_HARNESS_ADDENDUM after frontmatter to pin env/session, require one browse command per Bash call, and de-scope cloud/functions/templates/skills install.
  • Parse frontmatter with gray-matter to handle BOM/CRLF/embedded ---, ensuring correct insertion across platforms.

• Refactors

  • Rename installBrowserSkill → installBrowseSkill; install to .claude/skills/browse/; update prompts/logs to "browse". Keep stagehand_browser MCP name unchanged.
  • Remove packages/evals/skills/browser/SKILL.md.
  • Export insertAfterFrontmatter, add unit tests for boundary cases and addendum-before-"Cloud APIs"; add gray-matter as a devDependency.

^{Written for commit 486cab8. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 486cab8

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 3 files

Confidence score: 5/5

• Safe to merge after the addressed issues were fixed.

Architecture diagram

sequenceDiagram
    participant Harness as Eval Harness
    participant Adapter as claudeCodeToolAdapter
    participant Fs as File System
    participant Skill as CLI Browse Skill (packages/cli/skills/browse/SKILL.md)
    participant Agent as Claude Code Agent (Eval)
    participant Prompt as Harness Prompt
    
    Note over Harness,Prompt: Eval Setup: Skill Installation
    
    Harness->>Adapter: prepareBrowseCliHarnessAdapter()
    Adapter->>Adapter: installBrowseSkill(wrapperDir)
    Adapter->>Fs: readFile(BROWSE_SKILL_SOURCE)
    Fs-->>Adapter: CLI skill markdown (355 lines, single source of truth)
    Adapter->>Adapter: insertAfterFrontmatter(cliSkill, EVAL_HARNESS_ADDENDUM)
    Note over Adapter: Injects addendum between YAML frontmatter and CLI skill body
    Adapter->>Fs: writeFile(.claude/skills/browse/SKILL.md)
    Fs-->>Adapter: written
    Adapter-->>Harness: Installed browse skill at ...
    
    Note over Agent,Skill: Eval Runtime: Agent Loads & Uses Skill
    
    Agent->>Skill: Skill tool: load browse skill
    Skill-->>Agent: Combined skill content (CLI skill + addendum)
    Note over Agent: Addendum overrides CLI cloud/functions/skills examples
    Agent->>Agent: Follows addendum rules (one browse command per Bash call)
    Agent->>Adapter: Bash: browse open <url>
    Adapter->>Adapter: isAllowedBrowseCommand() - checks starts with "browse " + no shell metacharacters
    
    alt Valid browse command
        Adapter-->>Agent: Executes browse command
        Agent->>Adapter: Bash: browse snapshot
        Adapter->>Adapter: isAllowedBrowseCommand()
        Adapter-->>Agent: Executes browse snapshot
        Agent->>Adapter: Bash: browse doctor
        Adapter->>Adapter: isAllowedBrowseCommand()
        Adapter-->>Agent: Executes browse doctor
    else Command violates rules
        Agent->>Adapter: Bash: browse cloud fetch
        Adapter->>Adapter: isAllowedBrowseCommand()
        Note over Adapter: Allowed syntactically, but addendum says ignore
        Agent->>Agent: Addendum prohibits cloud commands, agent should skip
    end
    
    Note over Agent,Harness: Eval Runtime: Result Reporting
    
    Agent->>Prompt: EVAL_RESULT format
    Prompt-->>Harness: Benchmark result
    
    alt Agent attempts disallowed pattern
        Agent->>Adapter: Bash: browse snapshot | grep title
        Adapter->>Adapter: isAllowedBrowseCommand() - detects shell operators
        Adapter-->>Agent: Rejected: only single browse command allowed
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>