fix multi observe resolution, move dom settling to a method, eval passes - #2

jeremypress · 2024-05-20T21:06:02Z

Why

we need evals that pass. in this example the demo button would resolve to multiple selectors on the page which you can't verify (you need one). We take the same naive approach we take with act that we'll take the first valid resolution of a selector. This can be improved later

What changed

take the first locator when observing
waitForDomSettled has to be called in user space for now. the callback we were using was not await-able. more on this later as this becomes a problem we need to squash.

Test plan

The eval passes now!

Samarth ap patch 1

# why fix 1/2 ai sdk google thought signature buts The version of google provider + ai sdk we currently use has a bug resulting in errors with google thought signatures They are in the process of releasing a fix for bug #2 but it is not yet released - w ewill need to bump again when that is live # what changed bumped ai sdk + google providers versions # test plan  --- ## Summary by cubic Bumped AI SDK and @ai-sdk/google to pick up the fix for Google “thought signature” errors. A second upstream fix is pending and will require another bump later. - **Dependencies** - ai: ^5.0.0 -> ^5.0.133 (core, evals) - @ai-sdk/google: ^2.0.23 -> ^2.0.53 (core) - Updated lockfile and added changeset for @browserbasehq/stagehand (patch) Written for commit b6c94cb. Summary will update on new commits. <a href="https://cubic.dev/pr/browserbase/stagehand/pull/1685">Review in cubic</a>

@pirate

…che replay (#1983) Mirrored from external contributor PR #1978 after approval by @pirate. Original author: @a7med3liamin Original PR: #1978 Approved source head SHA: `2149aa265a04dc37154d5a84411f3ab4d1045897` @a7med3liamin, please continue any follow-up discussion on this mirrored PR. When the external PR gets new commits, this same internal PR will be marked stale until the latest external commit is approved and refreshed here. ## Original description - [x] Check the [documentation](https://docs.stagehand.dev/) for relevant information - [x] Search existing [issues](https://github.com/browserbase/stagehand/issues) to avoid duplicates Fixes #1776 ## Problem The `keys` tool has no variable substitution in either the live execution or cache replay paths. When the agent uses `%variableName%` tokens with the keys tool, the literal token string gets typed instead of the resolved value. ## Fix This PR combines two fixes into one: ### 1. Live execution (original fix by @trillville from #1777) - Accept `variables` parameter in `keysTool` (matching `typeTool`) - Call `substituteVariables()` before `page.type()` in the `method === "type"` branch - Pass `variables` to `keysTool` in `createAgentTools` - Update schema description to advertise available variables to the LLM - Return original token in result to avoid exposing sensitive values to LLM ### 2. Cache replay (new fix) - Import `substituteVariables` in `AgentCache.ts` - Pass `variables` through to `replayAgentKeysStep` - Call `substituteVariables(text, variables)` before `page.type()` in the replay path Without fix #2, cached `keys` steps with `method="type"` replay by typing literal `%variableName%` tokens even when variables are provided, since `replayAgentKeysStep` had no access to the variables map. ## Credit The live execution fix (part 1) is from @trillville's work in #1777/#1813. We merged it here with the cache replay fix per @pirate's request to consolidate into a single PR.   --- ## Summary by cubic Add variable substitution to the `keys` tool for both live execution and cache replay so `%variableName%` tokens are resolved before typing. This fixes cases where literal tokens were typed and brings parity with the `type` tool. - **Bug Fixes** - Pass `variables` into `keys` and call `substituteVariables()` before `page.type()`; update the input schema to list available variables. - In cache replay, forward `variables` to `replayAgentKeysStep` and substitute before typing to avoid replaying literal tokens. - Record and return the original tokenized value (not the resolved value) to avoid leaking sensitive data. Written for commit abb3905. Summary will update on new commits. <a href="https://cubic.dev/pr/browserbase/stagehand/pull/1983">Review in cubic</a>  --------- Co-authored-by: Ahmed Ali <a7med3liamin@gmail.com> Co-authored-by: trillville <trillville@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Nick Sweeting <git@sweeting.me>

…-failure backoff (#2248) ## Summary Makes browse driver (browser session) failures actionable, classified, and self-correcting. Today an invalid `BROWSERBASE_API_KEY` surfaces a bare `Error: 401 Unauthorized` with no remediation, the 5s init-failure cache makes every retry instant and identical, and most driver failures reach telemetry as `unexpected`. Linear: [STG-2277](https://linear.app/browserbase/issue/STG-2277/make-browse-driver-errors-actionable-with-result-codes-and-init) ## Impact if merged This targets the browse CLI's largest failure mode by volume and by user pain. 71 installs stuck in get/screenshot retry loops generate 92.3% of ALL CLI telemetry (~5.5M events/30d); ~375k of those events come from tagged claude-code and codex agents on current versions — exactly the target ICP (coding agents driving browsers). Root cause (smoke-tested): any `BROWSERBASE_API_KEY` forces remote mode; an invalid key surfaces a bare `Error: 401 Unauthorized` with no remediation, and the 5s init-failure cache makes every retry instant, so agents can't self-correct and loop forever. Separately, 2,337 distinct users hit missing_api_key/auth_401 in 30d, and `open` — whose failures are 94% unclassifiable today (result_code `unexpected`) — gates activation: only 28.5% of real users reach an activated session, and a failed first command cuts 7-day retention 12.4x. This PR makes auth/driver failures actionable (agents recover in one turn) and classified (we can finally measure why open fails). ## Implementation notes - **Remote init classification** (`remote.ts`): new `classifyRemoteInitError()` duck-types the SDK error's `status` — 401 → `remote_auth_401` (invalid-key message with settings link, `--local`, `browse doctor`), 403 → `remote_auth_403` (permissions/plan wording, same escape hatches), other → `remote_session_create_failed` (original message preserved + `browse doctor` pointer). Wired through the `RemoteCapability` interface so local-only builds compile. - **Chrome-not-found** (`session-manager.ts`): chrome-launcher's `ERR_LAUNCHER_NOT_INSTALLED` / `ERR_LAUNCHER_PATH_NOT_SET` failures in managed-local mode get install/`--cdp`/remote guidance instead of leaking launcher internals. - **Init-failure backoff**: cached init failures now back off exponentially — `min(5s * 2^(n-1), 5min)` — reset on success and `close()`. After ≥3 consecutive failures the cached message gains a `(failing repeatedly — fix BROWSERBASE_API_KEY, use --local, or run browse doctor)` suffix (deduped on rethrow). - **Result codes over the daemon protocol**: `ErrorResponseSchema` gains optional `code`/`httpStatus` (backward compatible — old daemons omit them); the daemon's `formatError` surfaces them from typed `DriverError`s; the client rethrows as `CommandFailure` with `resultCode`/`httpStatus` so the existing #2210 telemetry plumbing records them. Client-side fail sites tagged: `daemon_lock_timeout`, `daemon_unresponsive`, `daemon_socket_timeout`, `daemon_spawn_failed`. Already-authored driver errors tagged: `stale_ref` (unknown ref), `no_active_page`. - **Local-only build contract preserved**: remediation strings that mention `BROWSERBASE_API_KEY` live behind the remote capability (`driverInitHints()`), so the `build:local-only` artifact stays key-free (guarded by the existing `local-only-build.test.ts`, which caught the first draft). ## E2E Test Matrix | Command / flow | Observed output | Confidence / sufficiency | | --- | --- | --- | | `BROWSERBASE_API_KEY=bb_invalid_test <local build> get url` | `Browserbase rejected your BROWSERBASE_API_KEY (401 Unauthorized). A set key makes browse default to remote mode. Check the key at https://browserbase.com/settings, run without one using --local (browse open <url> --local), or diagnose with browse doctor.` exit=1 | Proves the new 401 classification flows daemon → protocol → client → stderr end-to-end against the real Browserbase API. | | Same command 4x rapidly (cached failure window) | Identical actionable message each time, ~400ms per run (no remote round-trip) | Proves cached failures keep the actionable message and stay instant; does not by itself prove backoff growth. | | Same command after 6s, then after 11s more (real failures #2, #3) | Message gains ` (failing repeatedly — fix BROWSERBASE_API_KEY, use --local, or run browse doctor)` suffix, exactly once, exit=1 | Proves the ≥3-consecutive-failures hint and suffix dedupe on the live failure path. | | Valid key: `open https://example.com` → `get title` → `stop` | `"mode": "remote" ... "title": "Example Domain"`, then `{"title": "Example Domain"}`, then `{"stopped": true}` | Proves the remote happy path is unchanged (no regression in outputs or exit codes). | | `env -u BROWSERBASE_API_KEY <local build> open https://example.com --local` → `get url` | `"mode": "managed-local" ... "url": "https://example.com/"`, then `{"url": "https://example.com/"}` exit=0 | Proves keyless managed-local mode is unaffected. | | `get text @9-99` on the local session | `Unknown ref "9-99" - run browse snapshot first to populate refs (have 0 refs).` exit=1 | Proves the stale-ref message is unchanged while now carrying `stale_ref` through the protocol (round-trip unit-tested). | | `browse doctor` with and without key | `Status: ok` in both; `target remote` with key, `target managed-local` without | Proves doctor behavior unchanged. | | `pnpm build` + `pnpm lint` (prettier, eslint, tsc) | All pass | Supporting only. | | `pnpm test:cli` | 16 files / 228 tests pass, incl. new `driver-errors.test.ts` (classification, backoff schedule, chrome-not-found detection, protocol round-trip, key-free local-only hints) and the `local-only-build` artifact guard | Supporting; covers mappings and the local-only security contract not exercised by live smokes. | 🤖 Generated with [Claude Code](https://claude.com/claude-code)  --- ## Summary by cubic Makes browse driver failures actionable and self-correcting with classified result codes and exponential init backoff. Addresses Linear STG-2277 by giving clear fixes for bad `BROWSERBASE_API_KEY`, missing Chrome/Chromium, and daemon issues, while improving telemetry. - **New Features** - Classify remote init errors into actionable messages with codes: `remote_auth_401`, `remote_auth_403`, `remote_session_create_failed` (with links to settings, `--local`, and `browse doctor`). - Add error result codes to the daemon protocol (`code`, `httpStatus`) and propagate to the client for telemetry. - Exponential backoff for cached init failures (5s doubling, capped at 1 minute) with a “failing repeatedly” hint after 3 failures. - Tag common failures with stable codes: `daemon_lock_timeout`, `daemon_unresponsive`, `daemon_socket_timeout`, `daemon_spawn_failed`, `stale_ref`, `no_active_page`, `no_chrome_found`. - Use `http-status-codes` for status mapping and extract chrome-launcher error codes to a constant (no behavior change). - **Bug Fixes** - Chrome-not-found now gives Chromium-first guidance: Linux `apt install chromium`; macOS `brew install --cask google-chrome` or set `CHROME_PATH` for Chromium, plus `--cdp` or remote as options. - Keep the local-only build key-free by moving `BROWSERBASE_API_KEY` remediation strings behind the remote capability. Written for commit b7a3f7e. Summary will update on new commits. <a href="https://cubic.dev/pr/browserbase/stagehand/pull/2248?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>  --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* vite plus scaffold * copy v3 runtime sources * align public api zod parity * fix compile * add service worker bridge smoke * remove zod compatibility shim * vp fmt * improve client options public schema * use chat completion options schema * clean public zod surface --------- Co-authored-by: samfinton <samfinton@samfintons-MacBook-Pro.local>

Part of STG-2671 (deterministic suite). Completes the deterministic a/e/o suite on the v4 API: **act 40/40, extract 25/25, observe 12/12** under `tasks/bench-v4/`, all on `stagehand.act/extract/observe` with async page accessors and the new config shape. ## Porting rules applied uniformly - Behavior preserved byte-for-byte where the API allows — same URLs, instructions, success criteria, timings - Schemaless v3 extracts → explicit single-key schemas (`z.object({ extraction: z.string() })`, V4_API_LOGS #2); snake_case schemas ported verbatim so wire-casing stays comparable (#14) - `framework/observeReplay.ts`: shared observe→act replay standing in for the missing `act(observeResult)` (#1) — delete when the SDK ships `ReplayActionSchema` - `heal_*` tasks fail loudly (`V4 GAP`) rather than fake self-healing v4 doesn't have - `combination/` removed — both tasks depended on the deprecated agent-era surface ## Verification - Full-suite sweeps on LOCAL and Browserbase vs v4-spike `93af925`: 53/74 pass; every failure classified (frames/OOPIF, observe ranking #15, targeting gaps) in V4_API_LOGS - v3 baseline on identical tasks: 65/74 — regression set is concentrated and nameable - Schema audit: 59 tasks byte-equivalent to v3, 13 documented wrapper conversions, 0 drift - Typecheck, eslint, prettier clean  --- ## Summary by cubic Ports the full deterministic bench suite to the v4 SDK for STG-2671, completing act 40/40, extract 25/25, and observe 12/12 under `tasks/bench-v4`. Preserves v3 behavior where possible and documents v4 gaps; adds an observe→act replay helper and minor stability updates. - **New Features** - Complete v4 bench on `stagehand.act/extract/observe`, with async page accessors and the new config shape. - Added `framework/observeReplay.ts` to replay observe→act until v4 ships native replay. - **Refactors** - Schemaless extracts now use explicit schemas (e.g., `z.object({ extraction: z.string() })`); snake_case schemas kept to compare wire casing. - Handled v4 gaps: `heal_*` tasks fail with clear V4 GAP messages; iframe/node-id checks re-expressed in-page; removed `combination/` tasks tied to the deprecated agent surface; renamed `act/nonsense_action` to assert failure handling. - Verification: local and Browserbase sweeps pass 53/74; failures classified; typecheck/lint clean. Written for commit a2db76f. Summary will update on new commits. <a href="https://cubic.dev/pr/browserbase/stagehand/pull/2387?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>  --------- Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

…preflight validation, dedup Task error contract: every bench-v4 catch block now returns the canonical TaskResult `error` key with a sanitized string (message only — raw Errors serialize as {} and `reason:`/`message:` keys are invisible to the runner's progress reporting). extract_jfk_links additionally logs structured failure data before returning. Eval validity: - extract_public_notices gated on compareStrings().meetsThreshold in BOTH the v3 and v4 copies (the bare object is always truthy, so both suites reported success regardless of extracted content); fixed on both sides to keep cross-SDK scoring comparable - google_flights converted to an explicit V4 GAP stub: v4 locator.click has no force option and no act(observeResult) replay, so the force-click benchmark cannot run on v4 - extract_recipe error messages say "expected" where they compared against the expected constants - stale "no { page } option" comments in tab_handling/multi_tab corrected (the option exists; active-page tracking is exercised deliberately) v4 init parity and diagnostics: initV4 sets selfHeal: true (matching initV3) and bridges SDK logs into the EvalLogger via the nested logging.onLog config, feature-detected against the loaded SDK schema. Preflight validation: explicit --sdk now requires --harness stagehand; --sdk v4 with --api or agent modes fails once at command validation instead of per-row; explicit --sdk with core-tier tasks is rejected before runs are misfiled into SDK-comparison projects. Dedup: shared framework/textScoring.ts (normalizeString/compareStrings, byte-identical to the 7 inlined copies) and framework/observeSelectors.ts (in-page selector resolution, 6 identical copies unified); observe_simple_google_search uses the shared observeReplay; observeReplay.click now forwards the observed mouse button like v3. Tests: focused coverage for observeReplay dispatch (14), the v4 bench harness startup/guards (7), planner sdk propagation (5), and initV4 param building (13). V4_API_LOGS #1/#2/#7/#8 examples updated to the stagehand.* surface. Suite: 55 files / 442 tests green; typecheck clean. Scoring-semantics findings intentionally not "fixed" where the v4 task is a byte-for-byte port of the same weakness in v3 (staff_members, area_codes, jstor ordering, next_chunk modal fallback, observe_taxes innerText gate) — strengthening only the v4 side would break cross-SDK comparability.

…preflight validation, dedup Task error contract: every bench-v4 catch block now returns the canonical TaskResult `error` key with a sanitized string (message only — raw Errors serialize as {} and `reason:`/`message:` keys are invisible to the runner's progress reporting). extract_jfk_links additionally logs structured failure data before returning. Eval validity: - extract_public_notices gated on compareStrings().meetsThreshold in BOTH the v3 and v4 copies (the bare object is always truthy, so both suites reported success regardless of extracted content); fixed on both sides to keep cross-SDK scoring comparable - google_flights converted to an explicit V4 GAP stub: v4 locator.click has no force option and no act(observeResult) replay, so the force-click benchmark cannot run on v4 - extract_recipe error messages say "expected" where they compared against the expected constants - stale "no { page } option" comments in tab_handling/multi_tab corrected (the option exists; active-page tracking is exercised deliberately) v4 init parity and diagnostics: initV4 sets selfHeal: true (matching initV3) and bridges SDK logs into the EvalLogger via the nested logging.onLog config, feature-detected against the loaded SDK schema. Preflight validation: explicit --sdk now requires --harness stagehand; --sdk v4 with --api or agent modes fails once at command validation instead of per-row; explicit --sdk with core-tier tasks is rejected before runs are misfiled into SDK-comparison projects. Dedup: shared framework/textScoring.ts (normalizeString/compareStrings, byte-identical to the 7 inlined copies) and framework/observeSelectors.ts (in-page selector resolution, 6 identical copies unified); observe_simple_google_search uses the shared observeReplay; observeReplay.click now forwards the observed mouse button like v3. V4_API_LOGS #1/#2/#7/#8 examples updated to the stagehand.* surface. Suite green; typecheck clean. Scoring-semantics findings intentionally not "fixed" where the v4 task is a byte-for-byte port of the same weakness in v3 (staff_members, area_codes, jstor ordering, next_chunk modal fallback, observe_taxes innerText gate) — strengthening only the v4 side would break cross-SDK comparability.

fix multi observe resolution, move dom settling to a method, eval passes

# why fix 1/2 ai sdk google thought signature buts The version of google provider + ai sdk we currently use has a bug resulting in errors with google thought signatures They are in the process of releasing a fix for bug browserbase#2 but it is not yet released - w ewill need to bump again when that is live # what changed bumped ai sdk + google providers versions # test plan  --- ## Summary by cubic Bumped AI SDK and @ai-sdk/google to pick up the fix for Google “thought signature” errors. A second upstream fix is pending and will require another bump later. - **Dependencies** - ai: ^5.0.0 -> ^5.0.133 (core, evals) - @ai-sdk/google: ^2.0.23 -> ^2.0.53 (core) - Updated lockfile and added changeset for @browserbasehq/stagehand (patch) Written for commit b6c94cb. Summary will update on new commits. <a href="https://cubic.dev/pr/browserbase/stagehand/pull/1685">Review in cubic</a>

@pirate

…p during cache replay (browserbase#1983) Mirrored from external contributor PR browserbase#1978 after approval by @pirate. Original author: @a7med3liamin Original PR: browserbase#1978 Approved source head SHA: `2149aa265a04dc37154d5a84411f3ab4d1045897` @a7med3liamin, please continue any follow-up discussion on this mirrored PR. When the external PR gets new commits, this same internal PR will be marked stale until the latest external commit is approved and refreshed here. ## Original description - [x] Check the [documentation](https://docs.stagehand.dev/) for relevant information - [x] Search existing [issues](https://github.com/browserbase/stagehand/issues) to avoid duplicates Fixes browserbase#1776 ## Problem The `keys` tool has no variable substitution in either the live execution or cache replay paths. When the agent uses `%variableName%` tokens with the keys tool, the literal token string gets typed instead of the resolved value. ## Fix This PR combines two fixes into one: ### 1. Live execution (original fix by @trillville from browserbase#1777) - Accept `variables` parameter in `keysTool` (matching `typeTool`) - Call `substituteVariables()` before `page.type()` in the `method === "type"` branch - Pass `variables` to `keysTool` in `createAgentTools` - Update schema description to advertise available variables to the LLM - Return original token in result to avoid exposing sensitive values to LLM ### 2. Cache replay (new fix) - Import `substituteVariables` in `AgentCache.ts` - Pass `variables` through to `replayAgentKeysStep` - Call `substituteVariables(text, variables)` before `page.type()` in the replay path Without fix browserbase#2, cached `keys` steps with `method="type"` replay by typing literal `%variableName%` tokens even when variables are provided, since `replayAgentKeysStep` had no access to the variables map. ## Credit The live execution fix (part 1) is from @trillville's work in browserbase#1777/browserbase#1813. We merged it here with the cache replay fix per @pirate's request to consolidate into a single PR.   --- ## Summary by cubic Add variable substitution to the `keys` tool for both live execution and cache replay so `%variableName%` tokens are resolved before typing. This fixes cases where literal tokens were typed and brings parity with the `type` tool. - **Bug Fixes** - Pass `variables` into `keys` and call `substituteVariables()` before `page.type()`; update the input schema to list available variables. - In cache replay, forward `variables` to `replayAgentKeysStep` and substitute before typing to avoid replaying literal tokens. - Record and return the original tokenized value (not the resolved value) to avoid leaking sensitive data. Written for commit abb3905. Summary will update on new commits. <a href="https://cubic.dev/pr/browserbase/stagehand/pull/1983">Review in cubic</a>  ---------

…-failure backoff (browserbase#2248) ## Summary Makes browse driver (browser session) failures actionable, classified, and self-correcting. Today an invalid `BROWSERBASE_API_KEY` surfaces a bare `Error: 401 Unauthorized` with no remediation, the 5s init-failure cache makes every retry instant and identical, and most driver failures reach telemetry as `unexpected`. Linear: [STG-2277](https://linear.app/browserbase/issue/STG-2277/make-browse-driver-errors-actionable-with-result-codes-and-init) ## Impact if merged This targets the browse CLI's largest failure mode by volume and by user pain. 71 installs stuck in get/screenshot retry loops generate 92.3% of ALL CLI telemetry (~5.5M events/30d); ~375k of those events come from tagged claude-code and codex agents on current versions — exactly the target ICP (coding agents driving browsers). Root cause (smoke-tested): any `BROWSERBASE_API_KEY` forces remote mode; an invalid key surfaces a bare `Error: 401 Unauthorized` with no remediation, and the 5s init-failure cache makes every retry instant, so agents can't self-correct and loop forever. Separately, 2,337 distinct users hit missing_api_key/auth_401 in 30d, and `open` — whose failures are 94% unclassifiable today (result_code `unexpected`) — gates activation: only 28.5% of real users reach an activated session, and a failed first command cuts 7-day retention 12.4x. This PR makes auth/driver failures actionable (agents recover in one turn) and classified (we can finally measure why open fails). ## Implementation notes - **Remote init classification** (`remote.ts`): new `classifyRemoteInitError()` duck-types the SDK error's `status` — 401 → `remote_auth_401` (invalid-key message with settings link, `--local`, `browse doctor`), 403 → `remote_auth_403` (permissions/plan wording, same escape hatches), other → `remote_session_create_failed` (original message preserved + `browse doctor` pointer). Wired through the `RemoteCapability` interface so local-only builds compile. - **Chrome-not-found** (`session-manager.ts`): chrome-launcher's `ERR_LAUNCHER_NOT_INSTALLED` / `ERR_LAUNCHER_PATH_NOT_SET` failures in managed-local mode get install/`--cdp`/remote guidance instead of leaking launcher internals. - **Init-failure backoff**: cached init failures now back off exponentially — `min(5s * 2^(n-1), 5min)` — reset on success and `close()`. After ≥3 consecutive failures the cached message gains a `(failing repeatedly — fix BROWSERBASE_API_KEY, use --local, or run browse doctor)` suffix (deduped on rethrow). - **Result codes over the daemon protocol**: `ErrorResponseSchema` gains optional `code`/`httpStatus` (backward compatible — old daemons omit them); the daemon's `formatError` surfaces them from typed `DriverError`s; the client rethrows as `CommandFailure` with `resultCode`/`httpStatus` so the existing browserbase#2210 telemetry plumbing records them. Client-side fail sites tagged: `daemon_lock_timeout`, `daemon_unresponsive`, `daemon_socket_timeout`, `daemon_spawn_failed`. Already-authored driver errors tagged: `stale_ref` (unknown ref), `no_active_page`. - **Local-only build contract preserved**: remediation strings that mention `BROWSERBASE_API_KEY` live behind the remote capability (`driverInitHints()`), so the `build:local-only` artifact stays key-free (guarded by the existing `local-only-build.test.ts`, which caught the first draft). ## E2E Test Matrix | Command / flow | Observed output | Confidence / sufficiency | | --- | --- | --- | | `BROWSERBASE_API_KEY=bb_invalid_test <local build> get url` | `Browserbase rejected your BROWSERBASE_API_KEY (401 Unauthorized). A set key makes browse default to remote mode. Check the key at https://browserbase.com/settings, run without one using --local (browse open <url> --local), or diagnose with browse doctor.` exit=1 | Proves the new 401 classification flows daemon → protocol → client → stderr end-to-end against the real Browserbase API. | | Same command 4x rapidly (cached failure window) | Identical actionable message each time, ~400ms per run (no remote round-trip) | Proves cached failures keep the actionable message and stay instant; does not by itself prove backoff growth. | | Same command after 6s, then after 11s more (real failures browserbase#2, browserbase#3) | Message gains ` (failing repeatedly — fix BROWSERBASE_API_KEY, use --local, or run browse doctor)` suffix, exactly once, exit=1 | Proves the ≥3-consecutive-failures hint and suffix dedupe on the live failure path. | | Valid key: `open https://example.com` → `get title` → `stop` | `"mode": "remote" ... "title": "Example Domain"`, then `{"title": "Example Domain"}`, then `{"stopped": true}` | Proves the remote happy path is unchanged (no regression in outputs or exit codes). | | `env -u BROWSERBASE_API_KEY <local build> open https://example.com --local` → `get url` | `"mode": "managed-local" ... "url": "https://example.com/"`, then `{"url": "https://example.com/"}` exit=0 | Proves keyless managed-local mode is unaffected. | | `get text @9-99` on the local session | `Unknown ref "9-99" - run browse snapshot first to populate refs (have 0 refs).` exit=1 | Proves the stale-ref message is unchanged while now carrying `stale_ref` through the protocol (round-trip unit-tested). | | `browse doctor` with and without key | `Status: ok` in both; `target remote` with key, `target managed-local` without | Proves doctor behavior unchanged. | | `pnpm build` + `pnpm lint` (prettier, eslint, tsc) | All pass | Supporting only. | | `pnpm test:cli` | 16 files / 228 tests pass, incl. new `driver-errors.test.ts` (classification, backoff schedule, chrome-not-found detection, protocol round-trip, key-free local-only hints) and the `local-only-build` artifact guard | Supporting; covers mappings and the local-only security contract not exercised by live smokes. | 🤖 Generated with [Claude Code](https://claude.com/claude-code)  --- ## Summary by cubic Makes browse driver failures actionable and self-correcting with classified result codes and exponential init backoff. Addresses Linear STG-2277 by giving clear fixes for bad `BROWSERBASE_API_KEY`, missing Chrome/Chromium, and daemon issues, while improving telemetry. - **New Features** - Classify remote init errors into actionable messages with codes: `remote_auth_401`, `remote_auth_403`, `remote_session_create_failed` (with links to settings, `--local`, and `browse doctor`). - Add error result codes to the daemon protocol (`code`, `httpStatus`) and propagate to the client for telemetry. - Exponential backoff for cached init failures (5s doubling, capped at 1 minute) with a “failing repeatedly” hint after 3 failures. - Tag common failures with stable codes: `daemon_lock_timeout`, `daemon_unresponsive`, `daemon_socket_timeout`, `daemon_spawn_failed`, `stale_ref`, `no_active_page`, `no_chrome_found`. - Use `http-status-codes` for status mapping and extract chrome-launcher error codes to a constant (no behavior change). - **Bug Fixes** - Chrome-not-found now gives Chromium-first guidance: Linux `apt install chromium`; macOS `brew install --cask google-chrome` or set `CHROME_PATH` for Chromium, plus `--cdp` or remote as options. - Keep the local-only build key-free by moving `BROWSERBASE_API_KEY` remediation strings behind the remote capability. Written for commit b7a3f7e. Summary will update on new commits. <a href="https://cubic.dev/pr/browserbase/stagehand/pull/2248?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>  ---------

jeremypress added 2 commits May 20, 2024 14:03

fix multi observe resolution, move dom settling to a method, eval passes

6939ba3

cleanup

700c833

jeremypress merged commit 42ddf2d into main May 20, 2024

pkiv deleted the jp-eval branch October 29, 2024 11:23

filip-michalsky mentioned this pull request Jul 25, 2025

Better agent logging #914

Open

seanmcguire12 mentioned this pull request Aug 12, 2025

enable scrolling inside of iframes #919

Merged

into-the-night referenced this pull request in kairos-computer/stagehand Nov 20, 2025

Merge pull request #2 from kairos-computer/SamarthAP-patch-1

23f91ca

Samarth ap patch 1

tkattkat mentioned this pull request Feb 14, 2026

bump ai sdk + google providers version #1685

Merged

greptile-apps Bot mentioned this pull request Feb 25, 2026

fix(cli): daemon reliability improvements #1756

Closed

5 tasks

a7med3liamin mentioned this pull request Apr 8, 2026

fix: add variable substitution to keys step during cache replay #1978

Closed

2 tasks

github-actions Bot mentioned this pull request Apr 8, 2026

[Claimed #1978] fix: add variable substitution to keys step during cache replay #1983

Merged

2 tasks

shrey150 mentioned this pull request Jun 12, 2026

[STG-2277] fix(cli): actionable driver errors, result codes, and init-failure backoff #2248

Merged

daric93 mentioned this pull request Jun 17, 2026

feat(cache): add Valkey backend for act/agent cache edlng/stagehand#1

Open

shrey150 mentioned this pull request Jun 27, 2026

[STG-2422] feat(cli): named contexts (local name→ID map) #2284

Merged

shriyatheunicorn mentioned this pull request Jul 23, 2026

evals: port full act/extract/observe bench suite to the v4 SDK #2387

Merged

cubic-dev-ai Bot mentioned this pull request Jul 28, 2026

evals: migrate the canonical benchmark suite to the current Stagehand API #2472

Closed

felipeofdev-ai pushed a commit to felipeofdev-ai/stagehand that referenced this pull request Aug 4, 2026

Merge pull request browserbase#2 from browserbase/jp-eval

77cf8a6

fix multi observe resolution, move dom settling to a method, eval passes

miguelg719 mentioned this pull request Aug 10, 2026

feat(integrations): Eve (native defineTool) #2666

Merged

Navigation Menu

Uh oh!

fix multi observe resolution, move dom settling to a method, eval passes - #2