Skip to content

Non interactive leaf elements are now included in the LLMs context - #8

Merged
jeremypress merged 2 commits into
mainfrom
jp-peeler4
May 22, 2024
Merged

Non interactive leaf elements are now included in the LLMs context#8
jeremypress merged 2 commits into
mainfrom
jp-peeler4

Conversation

@jeremypress

Copy link
Copy Markdown
Contributor

why

Let's say you're trying to automate buying peelers (a common occurance) and so you tell your trusty LLM to navigate to the peeler page and buy one. But on this peeler website, their call to action to add to cart is actually disjoint from the copy describing the peeler, so it doesn't know which check out button adds the peeler!

Previously, stagehand only looked at interactive elements but this breaks down pretty easily. The new approach is to include interactive and leaf elements (things that hold text). While this brings way more content to the LLM, we still have plenty of tools in our tool box to slim and chunk the dom down more as needed

what changed

  1. moved dom cleaning to it's own file
  2. added some html to peeler.html to make the case more complicated and easier to debug parsing issues
  3. the cleaner logic now uses js dom which was orders of magnitude faster than doing js eval in playwright. We'll use chunking and filtering strategies if /when memory usage becomes a problem
  4. small prompt tweaks, as we now include a comma separated list of DOM elements to the LLM

test plan

Run evals, i'll run them more once we upgrade on braintrust

@jeremypress
jeremypress merged commit bfaf985 into main May 22, 2024
@pkiv
pkiv deleted the jp-peeler4 branch October 29, 2024 11:23
miguelg719 pushed a commit that referenced this pull request Jul 23, 2026
* add loopback runtime protocol methods

* configure service worker for loopback cdp

* add browser.get_version assertion in loopback test
shriyatheunicorn added a commit that referenced this pull request Jul 28, 2026
…preflight validation, dedup

Task error contract: every bench-v4 catch block now returns the canonical
TaskResult `error` key with a sanitized string (message only — raw Errors
serialize as {} and `reason:`/`message:` keys are invisible to the runner's
progress reporting). extract_jfk_links additionally logs structured failure
data before returning.

Eval validity:
- extract_public_notices gated on compareStrings().meetsThreshold in BOTH
  the v3 and v4 copies (the bare object is always truthy, so both suites
  reported success regardless of extracted content); fixed on both sides to
  keep cross-SDK scoring comparable
- google_flights converted to an explicit V4 GAP stub: v4 locator.click has
  no force option and no act(observeResult) replay, so the force-click
  benchmark cannot run on v4
- extract_recipe error messages say "expected" where they compared against
  the expected constants
- stale "no { page } option" comments in tab_handling/multi_tab corrected
  (the option exists; active-page tracking is exercised deliberately)

v4 init parity and diagnostics: initV4 sets selfHeal: true (matching
initV3) and bridges SDK logs into the EvalLogger via the nested
logging.onLog config, feature-detected against the loaded SDK schema.

Preflight validation: explicit --sdk now requires --harness stagehand;
--sdk v4 with --api or agent modes fails once at command validation instead
of per-row; explicit --sdk with core-tier tasks is rejected before runs are
misfiled into SDK-comparison projects.

Dedup: shared framework/textScoring.ts (normalizeString/compareStrings,
byte-identical to the 7 inlined copies) and framework/observeSelectors.ts
(in-page selector resolution, 6 identical copies unified);
observe_simple_google_search uses the shared observeReplay;
observeReplay.click now forwards the observed mouse button like v3.

Tests: focused coverage for observeReplay dispatch (14), the v4 bench
harness startup/guards (7), planner sdk propagation (5), and initV4 param
building (13). V4_API_LOGS #1/#2/#7/#8 examples updated to the stagehand.*
surface. Suite: 55 files / 442 tests green; typecheck clean.

Scoring-semantics findings intentionally not "fixed" where the v4 task is a
byte-for-byte port of the same weakness in v3 (staff_members, area_codes,
jstor ordering, next_chunk modal fallback, observe_taxes innerText gate) —
strengthening only the v4 side would break cross-SDK comparability.
shriyatheunicorn added a commit that referenced this pull request Jul 28, 2026
…preflight validation, dedup

Task error contract: every bench-v4 catch block now returns the canonical
TaskResult `error` key with a sanitized string (message only — raw Errors
serialize as {} and `reason:`/`message:` keys are invisible to the runner's
progress reporting). extract_jfk_links additionally logs structured failure
data before returning.

Eval validity:
- extract_public_notices gated on compareStrings().meetsThreshold in BOTH
  the v3 and v4 copies (the bare object is always truthy, so both suites
  reported success regardless of extracted content); fixed on both sides to
  keep cross-SDK scoring comparable
- google_flights converted to an explicit V4 GAP stub: v4 locator.click has
  no force option and no act(observeResult) replay, so the force-click
  benchmark cannot run on v4
- extract_recipe error messages say "expected" where they compared against
  the expected constants
- stale "no { page } option" comments in tab_handling/multi_tab corrected
  (the option exists; active-page tracking is exercised deliberately)

v4 init parity and diagnostics: initV4 sets selfHeal: true (matching
initV3) and bridges SDK logs into the EvalLogger via the nested
logging.onLog config, feature-detected against the loaded SDK schema.

Preflight validation: explicit --sdk now requires --harness stagehand;
--sdk v4 with --api or agent modes fails once at command validation instead
of per-row; explicit --sdk with core-tier tasks is rejected before runs are
misfiled into SDK-comparison projects.

Dedup: shared framework/textScoring.ts (normalizeString/compareStrings,
byte-identical to the 7 inlined copies) and framework/observeSelectors.ts
(in-page selector resolution, 6 identical copies unified);
observe_simple_google_search uses the shared observeReplay;
observeReplay.click now forwards the observed mouse button like v3.

V4_API_LOGS #1/#2/#7/#8 examples updated to the stagehand.* surface.
Suite green; typecheck clean.

Scoring-semantics findings intentionally not "fixed" where the v4 task is a
byte-for-byte port of the same weakness in v3 (staff_members, area_codes,
jstor ordering, next_chunk modal fallback, observe_taxes innerText gate) —
strengthening only the v4 side would break cross-SDK comparability.
felipeofdev-ai pushed a commit to felipeofdev-ai/stagehand that referenced this pull request Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant