Skip to main content
agent-qa on GitHub,904stars
agent-qa

Quickstart

Copy page
View as MarkdownOpen the raw Markdown source.Open in ChatGPTAsk questions about this page.Open in ClaudeAsk questions about this page.Open in MistralAsk questions about this page.Open in VS CodeAsk Copilot Chat about this page.Open in CursorAsk Cursor about this page.

Install agent-qa, connect a model and run your first tests in about ten minutes.

This page takes you from an empty folder to a first run. The examples test Komazon, a small shop at https://playground.vostride.dev/shop/ that exists for this purpose. One of its pages has a deliberate bug, and one of the examples catches it.

Working with a coding agent? The agent quickstart hands this whole page to it with one prompt.

Before you start

  • Node.js 24 or newer.
  • A model. A Claude or Codex subscription works, and so does an API key for an OpenAI-compatible, Anthropic-compatible or Gemini endpoint. Local models work through the compatible endpoints.
  • Network access to playground.vostride.dev and to your model provider.
  • Docker, but only for tests that use hooks.

1. Install

Run these from the folder that will hold your tests. In an existing app, that is usually the repository root.

Install agent-qa
npm install -D agent-qa

With a Claude or Codex subscription, also install the sign-in plugin:

Install subscription sign-in
npm install -D @vostride/agent-qa-subscription-auth

2. Install a browser

The examples run in Chromium. agent-qa installs its own copy, which leaves your everyday browsers alone:

Install Chromium
npx agent-qa install-browsers --chromium

On Linux, add --with-deps to install the system libraries Chromium needs.

3. Create the project

Initialize agent-qa
npx agent-qa init

In a terminal, init asks which platform you test and how agent-qa reaches a model. Every question has a default, and each one has a flag if you would rather skip it:

--llmModel entry it writesDefault modelSigns in with
claude-code (default)claude-codeclaude-sonnet-5-5Claude subscription
codexcodexgpt-6.1-solCodex subscription
openai-compatibledefaultgpt-5.5API key
anthropic-compatibledefaultclaude-sonnet-4-6API key or bearer token
geminidefaultgemini-3-flash-previewAPI key

--model <id> replaces the default model. --base-url <url> points the two compatible providers at another endpoint, such as a local model server. For example:

npx agent-qa init --platform web --llm openai-compatible --model my-model --base-url http://127.0.0.1:1234/v1

init writes a config, three example tests, a suite that runs two of them, a hook example and the knowledge folder. Select a file to see what it contains:

  • agent-qa-knowledge
  • policies
  • global.md
  • .gitignore
  • AGENTS.md
  • README.md
  • scripts
  • examples
  • make-shipping-address.mjs
  • suites
  • examples.suite.yaml
  • tests
  • examples
  • komazon-cart-total-bug.yaml
  • komazon-checkout.yaml
  • komazon-shipping-hook.yaml
  • .env.secrets.local
  • .gitignore
  • agent-qa.config.yaml
  • agent-qa.local.yaml
  • hooks.yaml
  • package.json
agent-qa.config.yaml
# agent-qa Configuration# File discovery and project settingsworkspace:  testMatch:    - tests/**/*.yaml  suiteMatch:    - suites/**/*.suite.yaml  hooksFile: hooks.yaml  envFile: .env  secretsFile: .env.secrets.local  # Its values are shown as [secret:<name>] and never restored; nothing else is masked.# Optional: ignore archived or generated tests.  # testPathIgnore:  #   - tests/archive/**/*.yaml# Infrastructure services (dashboard, cache, auth state, logging)services:  dashboard:    port: 3100    artifactsDir: .agent-qa/artifacts    # Optional: persist dashboard state to a custom SQLite path.    # dbPath: .agent-qa/dashboard.sqlite  cache:    dir: .agent-qa/cache    ttl: 7d  authState:    dir: .agent-qa/auth-states  vision:    screenshotSize: 256k    effectiveResolution: 768  # Accessibility checks run once at the end of each test; results show in the dashboard's run detail.  accessibility:    enabled: true    standard: wcag2aa    runAfter: test-end    failOnViolation: false  recording:    enabled: true  logging:    level: warn# Committed knowledge root (optional block; these defaults apply when absent)# knowledge:#   dir: agent-qa-knowledge#   commitArtifacts: none# Named resource definitions (LLM configs, app targets)registry:  # Optional: local and cloud device/provider profiles.  # devices:  #   android-emu:  #     platform: android  #     transport: local  # providers:  #   browserstack:  #     username: ${BROWSERSTACK_USERNAME}  #     accessKey: ${BROWSERSTACK_ACCESS_KEY}  llms:    - name: claude-code      provider: anthropic-subscription      model: claude-sonnet-5-5  targets:    # Optional: add more web or mobile targets here.    # my-mobile:    #   platform: android    #   appPackage: com.example.app    #   appActivity: .MainActivity    #   app:    #     path: apps/example.apk    komazon:      platform: web      url: https://playground.vostride.dev/shop/    komazon-cart-total-bug:      platform: web      url: https://playground.vostride.dev/shop/?bug=cart-total# Optional subscription auth plugin declarationsplugins:  auth:    - package: "@vostride/agent-qa-subscription-auth"# Execution settings (cascades: global -> suite -> test -> CLI flags)use:  browser:    name: chromium    headless: true    viewport:      width: 1280      height: 720  mobile:    appState: preserve  timeout:    step: 20m    test: 90m    navigation: 2m  # Per-step limits: maxActions (actions) and maxPlanningTurns (planner calls); a step that needs more stops with the limit reason  planner:    maxActions: 30    maxPlanningTurns: 30    previousStepCount: 5  logCapture:    console: true    network: true  parallel: false  llm: claude-code  # Optional: bind mobile runs to a configured device profile.  # device: android-emu

If the folder is not a git repository yet, make it one and commit. Later you will read what agent-qa learned as a git diff against this commit.

git init
git add .
git commit -m "Add agent-qa"

The .gitignore that init wrote keeps .agent-qa/, .env, .env.secrets.local and agent-qa.local.yaml out of the commit.

4. Connect the model

With a subscription, sign in. Name the model entry init wrote:

npx agent-qa auth login --config claude-code

It prints a sign-in URL and opens it in your browser. For Claude, paste the code the page shows back into the terminal. Use --config codex for a Codex subscription.

With an API key, save it for the default entry:

npx agent-qa auth set --config default --type api-key

auth set asks for the key at a hidden prompt, so it never lands in your shell history. agent-qa keeps credentials in ~/.agent-qa/auth.json, outside the project. Models and credentials covers every provider.

5. Check the setup

npx agent-qa validate

validate reads the config and every test and suite, and ends with All files valid when they are. For a wider check, npx agent-qa doctor also looks at Node.js, the browser, Docker and your credentials, and sends one short request to your model.

6. Run the examples

Run the example suite
npx agent-qa run suites/examples.suite.yaml

Expect one pass, one failure and exit code 1:

  • The checkout journey signs in, buys a mug and checks the order. It passes.
  • The cart-total test catches the playground's deliberate bug. Two mugs should cost $53.00, and the bag shows $29.00.

The summary ends with Tests: 1 of 2 passed. After the verdict, agent-qa spends up to a minute analyzing the run and recording what it learned. That work never changes the exit code.

7. Look around

Open the dashboard

Open the dashboard
npx agent-qa dashboard --open

It prints Dashboard running at and a URL. Each run shows its tests, steps and evidence. Accessibility results appear here too: open a test's last step and its A11y tab. Stop the dashboard with Ctrl+C.

Read what agent-qa learned

Runs write what they learn into agent-qa-knowledge/. After the first runs every record is a new file, so ask git to include new files in the diff:

git add --intent-to-add agent-qa-knowledge
git diff -- agent-qa-knowledge

Commit the records you want to keep. memidex explains what each file means.

See the action cache

agent-qa remembers the action that completed a step and replays it next time without asking the model, as long as the screen looks the same. On your default branch it only saves when asked, so the first of these two runs passes --save-cache:

npx agent-qa run tests/examples/komazon-checkout.yaml --save-cache
npx agent-qa run tests/examples/komazon-checkout.yaml

Steps that check something, or that type a secret, are never cached, so in this journey only the address step can replay. The Cache: line in the summary counts hits and misses. Caching has the details.

Connect your coding agent

Give Claude Code, Codex or another MCP client the agent-qa skills and tools: Coding agents.

When something is missing

SituationWhat agent-qa printsFix
Node.js older than 24agent-qa needs Node.js 24 or newer.Install Node.js 24 or newer from nodejs.org.
No browseragent-qa browser support is not installed for Chromium.npx agent-qa install-browsers --chromium
Subscription not signed inSign in to "Anthropic subscription" for the "claude-code" confignpx agent-qa auth login --config claude-code
Subscription plugin missingneeds the subscription-auth pluginnpm install -D @vostride/agent-qa-subscription-auth
No API keySave an API key for the "default" confignpx agent-qa auth set --config default --type api-key
A config errorConfig validation failed, then the keyFix or remove that key, then run npx agent-qa validate.
A config from v0.1This config was written for agent-qa before v2.Follow v0.1.x to v0.2.x.

What is agent-qa

Test web and mobile apps in plain English. agent-qa builds persistent memory from validated run evidence, reuses what works, and adapts as your app changes.

Agent quickstart

Hand the whole setup to your coding agent with one prompt, from install to a verified first test.

On this page

Before you start1. Install2. Install a browser3. Create the project4. Connect the model5. Check the setup6. Run the examples7. Look aroundOpen the dashboardRead what agent-qa learnedSee the action cacheConnect your coding agentWhen something is missing