Non interactive leaf elements are now included in the LLMs context - #8
Merged
Conversation
miguelg719
pushed a commit
that referenced
this pull request
Jul 23, 2026
* add loopback runtime protocol methods * configure service worker for loopback cdp * add browser.get_version assertion in loopback test
shriyatheunicorn
added a commit
that referenced
this pull request
Jul 28, 2026
…preflight validation, dedup
Task error contract: every bench-v4 catch block now returns the canonical
TaskResult `error` key with a sanitized string (message only — raw Errors
serialize as {} and `reason:`/`message:` keys are invisible to the runner's
progress reporting). extract_jfk_links additionally logs structured failure
data before returning.
Eval validity:
- extract_public_notices gated on compareStrings().meetsThreshold in BOTH
the v3 and v4 copies (the bare object is always truthy, so both suites
reported success regardless of extracted content); fixed on both sides to
keep cross-SDK scoring comparable
- google_flights converted to an explicit V4 GAP stub: v4 locator.click has
no force option and no act(observeResult) replay, so the force-click
benchmark cannot run on v4
- extract_recipe error messages say "expected" where they compared against
the expected constants
- stale "no { page } option" comments in tab_handling/multi_tab corrected
(the option exists; active-page tracking is exercised deliberately)
v4 init parity and diagnostics: initV4 sets selfHeal: true (matching
initV3) and bridges SDK logs into the EvalLogger via the nested
logging.onLog config, feature-detected against the loaded SDK schema.
Preflight validation: explicit --sdk now requires --harness stagehand;
--sdk v4 with --api or agent modes fails once at command validation instead
of per-row; explicit --sdk with core-tier tasks is rejected before runs are
misfiled into SDK-comparison projects.
Dedup: shared framework/textScoring.ts (normalizeString/compareStrings,
byte-identical to the 7 inlined copies) and framework/observeSelectors.ts
(in-page selector resolution, 6 identical copies unified);
observe_simple_google_search uses the shared observeReplay;
observeReplay.click now forwards the observed mouse button like v3.
Tests: focused coverage for observeReplay dispatch (14), the v4 bench
harness startup/guards (7), planner sdk propagation (5), and initV4 param
building (13). V4_API_LOGS #1/#2/#7/#8 examples updated to the stagehand.*
surface. Suite: 55 files / 442 tests green; typecheck clean.
Scoring-semantics findings intentionally not "fixed" where the v4 task is a
byte-for-byte port of the same weakness in v3 (staff_members, area_codes,
jstor ordering, next_chunk modal fallback, observe_taxes innerText gate) —
strengthening only the v4 side would break cross-SDK comparability.
shriyatheunicorn
added a commit
that referenced
this pull request
Jul 28, 2026
…preflight validation, dedup
Task error contract: every bench-v4 catch block now returns the canonical
TaskResult `error` key with a sanitized string (message only — raw Errors
serialize as {} and `reason:`/`message:` keys are invisible to the runner's
progress reporting). extract_jfk_links additionally logs structured failure
data before returning.
Eval validity:
- extract_public_notices gated on compareStrings().meetsThreshold in BOTH
the v3 and v4 copies (the bare object is always truthy, so both suites
reported success regardless of extracted content); fixed on both sides to
keep cross-SDK scoring comparable
- google_flights converted to an explicit V4 GAP stub: v4 locator.click has
no force option and no act(observeResult) replay, so the force-click
benchmark cannot run on v4
- extract_recipe error messages say "expected" where they compared against
the expected constants
- stale "no { page } option" comments in tab_handling/multi_tab corrected
(the option exists; active-page tracking is exercised deliberately)
v4 init parity and diagnostics: initV4 sets selfHeal: true (matching
initV3) and bridges SDK logs into the EvalLogger via the nested
logging.onLog config, feature-detected against the loaded SDK schema.
Preflight validation: explicit --sdk now requires --harness stagehand;
--sdk v4 with --api or agent modes fails once at command validation instead
of per-row; explicit --sdk with core-tier tasks is rejected before runs are
misfiled into SDK-comparison projects.
Dedup: shared framework/textScoring.ts (normalizeString/compareStrings,
byte-identical to the 7 inlined copies) and framework/observeSelectors.ts
(in-page selector resolution, 6 identical copies unified);
observe_simple_google_search uses the shared observeReplay;
observeReplay.click now forwards the observed mouse button like v3.
V4_API_LOGS #1/#2/#7/#8 examples updated to the stagehand.* surface.
Suite green; typecheck clean.
Scoring-semantics findings intentionally not "fixed" where the v4 task is a
byte-for-byte port of the same weakness in v3 (staff_members, area_codes,
jstor ordering, next_chunk modal fallback, observe_taxes innerText gate) —
strengthening only the v4 side would break cross-SDK comparability.
felipeofdev-ai
pushed a commit
to felipeofdev-ai/stagehand
that referenced
this pull request
Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
why
Let's say you're trying to automate buying peelers (a common occurance) and so you tell your trusty LLM to navigate to the peeler page and buy one. But on this peeler website, their call to action to add to cart is actually disjoint from the copy describing the peeler, so it doesn't know which check out button adds the peeler!
Previously, stagehand only looked at interactive elements but this breaks down pretty easily. The new approach is to include interactive and leaf elements (things that hold text). While this brings way more content to the LLM, we still have plenty of tools in our tool box to slim and chunk the dom down more as needed
what changed
test plan
Run evals, i'll run them more once we upgrade on braintrust