Watchtower | Devpost

Banner
Website Navigation Tree
AI Agents Navigating a Browser

Inspiration

As agents take on longer, multi-step tasks, a simple success or failure label stops being useful. When an agent gets stuck, the underlying cause could be a broken page, an unexpected website state, a bad action, or information that was lost between stages of the workflow. The more agents involved, the harder it becomes to reconstruct what actually happened from disconnected logs.

We built Watchtower because we wanted observability that matched the way these systems really work. Developers should be able to watch agents as they navigate, see the paths they were assigned, and trace a failure back to the evidence that supports a diagnosis. Better visibility makes it easier to debug both the agents and the interfaces they use, and, over time, creates the kind of structured run data that could help evaluate and improve future agents.

What it does

Watchtower provides real-time observability into agent workflows, making it easy to see what agents are doing, where they fail, and why. For easy visualization of this observability, we applied the Watchtower framework to a browser-agent workflow. To implement this use of watchtower, we’ve outlined the following steps for our website:

A user enters a target URL, a task, and a worker count.
A discovery agent explores the locally hosted version of the site and builds a map of states, interactions, and transitions that are relevant to that task.
An orchestrator selects a varied set of valid paths through the map, favoring different strategies such as direct navigation, category browsing, and search.
Workers execute those paths in separate Browserbase sessions. Each worker receives one validated node instruction at a time and verifies the resulting page state before continuing.
The Observation Hub shows the live browser sessions, plans, events, outcomes, and investigation reports.
When a failure signal appears, a separate read-only observer examines same-run evidence and produces a report that separates observed facts from likely causes.

This makes Watchtower useful in two related ways. It is an observability tool for studying where the browser agents fail, and in this application it additionally becomes a practical UI testing tool: several isolated workers can exercise different relevant routes through the same product and expose edge cases that ordinary scripted tests may miss. Rather than trusting an agent's claim that it finished, Watchtower checks whether its actions reached the expected states and preserves the successful and failed traces for later review.

How we built it

Web Application What we built: An API, event stream, and monitoring dashboard for starting and stopping runs and viewing live activity. Technology: TypeScript, Node.js, Express, HTML, CSS, JavaScript

Discovery What we built: A local crawler that renders the site, extracts compact DOM snapshots, follows only goal-relevant interactions, records coverage limits, and builds a reusable flow map. Technology: Playwright, Chromium, OpenAI Responses API, Zod

Planning What we built: A path orchestrator that validates root-to-leaf candidates and selects behaviorally varied routes within a configurable path limit. Technology: OpenAI, TypeScript

Execution What we built: Independent cloud browser sessions whose workers execute one validated transition at a time and verify each destination state. Technology: Browserbase, Playwright over CDP

Evidence What we built: A shared instrumentation SDK that correlates runs, agent executions, browser sessions, model calls, tool calls, errors, links, and artifacts. Technology: PostgreSQL, TypeScript

Investigation What we built: A durable job worker and read-only observer that retrieves bounded, run-scoped evidence and submits a structured report with citations. Technology: OpenAI, PostgreSQL

Telemetry What we built: Correlated spans and external trace references alongside the authoritative PostgreSQL evidence store. Technology: Sentry

The crawler agents and Watchtower agent that analyses the crawlers use separately configurable, low-reasoning OpenAI calls with structured outputs. Worker and observer calls are also schema-validated, and website content is always treated as untrusted data rather than as instructions. PostgreSQL is the authoritative evidence store, while Sentry supplies correlated operational telemetry without becoming a dependency for persistence

Challenges we ran into

Keeping the Crawler From Cycling Early versions could rediscover the same rendered page through several routes and keep expanding it, which left the crawler stuck in repeated states. This mattered because a website graph naturally contains shared destinations and cycles, while a worker needs a finite, valid path to execute. We now fingerprint rendered states, reuse an existing state when a fingerprint repeats, track visited state IDs, represent repeated destinations as references, and reject looping worker paths during validation.

Making Local Discovery Match Remote Execution We found that a correctly discovered state could still fail worker verification because the DOM contained localhost URLs while the worker saw the public tunnel origin. The pages looked equivalent to a person, but their fingerprints differed, so workers could report an unexpected state and halt prematurely. We now rewrite embedded local origins throughout the snapshot, recompute the worker-facing fingerprint, and retain a separate local fingerprint for replay validation.

Keeping Execution and Diagnosis Honest A worker completing its assigned clicks did not necessarily mean it had completed the user's task, and an observer's plausible explanation was not necessarily a proven cause. Both problems matter because confident but unsupported success and diagnosis are exactly the data-quality failures Watchtower is meant to reveal. Workers now verify every destination and the final goal, while the observer can only query bounded same-run evidence and must keep cited facts, hypotheses, alternatives, and missing evidence separate.

Accomplishments that we're proud of

We coordinate a configurable number of workers while giving every worker its own isolated Browserbase session and the same starting point.
We built node-by-node execution: workers receive only the current validated transition, verify the resulting state, and do not improvise unrelated exploration.
We preserve correlation across run IDs, agent execution IDs, logical agent IDs, Browserbase session IDs, events, artifacts, and external traces.
We built a durable investigation pipeline in PostgreSQL with deduplicated incident jobs and structured reports that cite real events from the same run.
We retain explicit evidence about failed actions, dead ends, unsupported interactions, state and depth limits, and unexplored branches instead of presenting partial coverage as complete.
We made the browser activity approachable through a monitoring-room interface where users can watch parallel sessions and connect what they see to plans, events, results, and observer findings.
One of our biggest technical breakthroughs was making discovery and execution share the same state model even though discovery happens on localhost and execution happens in isolated cloud browsers through a public tunnel.

What we learned

Browser agents fail in smaller and stranger ways than a final status suggests. A stale fingerprint, a blank default tab, a model that treats an earlier selection as proof of success, or a path that ends one page before checkout can all produce similar-looking failures for completely different reasons.

We also learned that more exploration is not automatically better. A useful system needs a clear task boundary, explicit coverage limits, and a way to preserve unexplored branches without sending workers through every control on the site. For Watchtower, “stop when the requested evidence is visible” became as important as “find a route that works.” Reproducibility required us to treat state as evidence rather than as an assumption. We validate the source state before an action, validate the destination afterward, record replay divergence, and preserve the exact relationships between model calls, tool calls, browser events, and outcomes. The observer follows those records, but it is not allowed to rewrite them or turn a hypothesis into an observed fact.

Finally, the frontend, orchestration, execution, and observability layers could not be designed independently. The interface is only useful if IDs remain stable through the entire pipeline; the observer is only useful if execution emits meaningful events; and parallel browser panes are only understandable if each one has a clear assigned path. Building the vertical flow exposed integration mistakes much earlier than polishing any one layer in isolation would have.

What's next for Watchtower

The current project is a working prototype, and the following are future directions rather than shipped features:

Add searchable run history, side-by-side comparisons, and regression views across repeated tests.
Build richer replay and evidence navigation so a report citation can open the exact browser moment, DOM snapshot, model call, or external trace behind it.
Improve classification of agent mistakes, website defects, environmental failures, and inconclusive cases using reviewed outcomes rather than unsupported certainty.
Expand beyond unauthenticated, disposable demo sites with carefully designed authentication, data reset, tenant isolation, and sensitive-data controls.
Support more complex interactions and application states while keeping exploration limits explicit.
Turn reviewed and redacted run records into curated evaluation and training datasets for multi-agent systems.

Our longer-term goal is for Watchtower to become more than a wall of browser sessions. We want it to be the place where developers can reconstruct a multi-agent workflow, understand the first meaningful failure, and turn that evidence into a better agent or a better product.