Development

Everything you need to build, test, lint, and contribute to WaveHouse — from first-clone to hot-reload dev server to full end-to-end SDK tests. If you’re only trying the product, start with the Getting Started guide instead.

Prerequisites

You need these on your PATH before any make recipe will work end-to-end:

Tool	Required version	Why	Install
Go	Any Go ≥ 1.21; make runs the version go.mod pins, its toolchain line if it has one, else its go line (currently 1.26.10), downloading it once if yours differs (needs network). make GOTOOLCHAIN=local … uses yours instead (an environment value is ignored): it needs a Go ≥ that version, and make lint-go needs go.mod’s minor	Compiles cmd/wavehouse; also runs the pinned tool deps (gotestsum, gofumpt, goimports, govulncheck, go-test-coverage, gocover-cobertura, deadcode, gsa, goda) via go tool	go.dev/dl
GNU Make	4.0+	The Makefile uses --output-sync=target (Make 4 only) and bash-pinned recipes. macOS ships with GNU Make 3.81, which is not supported: it silently ignores --output-sync, so the parallel output of make verify / make ci interleaves line by line and a failure can scroll away behind other jobs’ output	macOS: brew install make then use gmake or put $(brew --prefix make)/libexec/gnubin on your PATH. Linux®: usually already installed
bash	4+ recommended	Recipes are pinned to bash; the helper scripts under scripts/ use set -euo pipefail and bash arrays	macOS default is bash 3.2 (works for current recipes, but brew install bash is safer); Linux distros ship 4+
Docker® (or Podman)	Engine 20.10+ with the Compose v2 plugin (docker compose, no hyphen)	Compose stacks under deployments/compose/; the E2E and integration suites boot ClickHouse® and a Redis® via testcontainers (no compose file), the integration suite also dynamodb-local, and the integration suite also runs the shared cache backend against Redis, Valkey, Dragonfly (pulled from docker.dragonflydb.io) and a one-node Redis Cluster	Docker Desktop, colima, or Podman with podman-compose / the podman compose plugin. The testcontainers Go library also honors DOCKER_HOST for rootless Podman setups
Node.js®	22 LTS — pinned via .nvmrc at the repo root	Runtime for pnpm and the Vitest suites. Pinned to match CI (setup-node uses 22) and to avoid Node-major surprises; older Vitest versions in this repo were known to crash on Node 26 with a V8 heap-allocation abort	nodejs.org or nvm use / fnm use / volta (all read .nvmrc)
pnpm	11.21+ (pinned via packageManager in the root package.json)	Package manager for the TypeScript® SDK, E2E test harness, and docs site (managed as a single pnpm workspace from the repo root); make tools, make verify, make lint, make fmt, make fix, make ci, make build-ts, make dev-ts, make test-ts, make test-e2e, make test-all, make check-docs, make build-docs, make build-all, make dev-docs, make preview-docs all shell out to pnpm (make tools itself runs pnpm install, and make verify is what the pre-commit hook runs, so a Go-only contributor needs pnpm too)	corepack enable && corepack prepare pnpm@11.21.0 --activate (recommended), or npm i -g pnpm
git + curl + jq	any recent	git for source + version metadata in builds; curl is used by the Makefile to fetch the pinned golangci-lint and shellcheck binaries into .bin/ and, for make build-ts, the oldest Node that engines.node admits; jq is needed by make verify (the hook tests) and by the Claude Code™ hooks	usually preinstalled (jq ships with macOS 15+); otherwise apt install jq / brew install jq

Auto-installed by make tools

Run make tools once after cloning to populate everything that doesn’t have to be on your PATH:

• golangci-lint v2.11.4 → installed to .bin/<os>_<arch>/ (version-pinned in the Makefile; bumping the version triggers a reinstall). Not in go.mod because its dependency tree conflicts with the main module.
• misspell v0.8.0, shellcheck v0.11.0, actionlint v1.7.12 → installed to .bin/<os>_<arch>/; they back make lint-prose, make lint-sh, and make lint-gha. make tools also points core.hooksPath at .githooks/, which is what installs the pre-commit and pre-push gates (git config --unset core.hooksPath turns them off for every worktree of the clone, and the next make tools turns them back on; use --no-verify for a one-off).
• air v1.65.1 → installed to .bin/<os>_<arch>/ via go install; used by make dev for hot-reload. Same exclusion principle as golangci-lint — air’s transitive deps (Hugo, Sass libs) would bloat go.sum.
• Go tool deps (gotestsum, gofumpt, goimports, govulncheck, go-test-coverage, gocover-cobertura, deadcode, gsa, goda) — pinned in go.mod via native tool directives (Go 1.24+), invoked with go tool <name>. make tools runs go mod download so they’re cached; they compile lazily on first invocation.
• pnpm deps for the repo root plus clients/ts/, tests/e2e/sdk/, and docs/ (via pnpm install --frozen-lockfile). The root workspace is where the repo-wide linters live — Biome (JS/TS/JSON) and markdownlint-cli2 (Markdown/MDX, including the repo-local WH001/WH002 rules). Neither is ever run from a global install: make lint / make fix shell out to pnpm -w run, and every target that needs them declares the pnpm-install prerequisite, so a clean clone is linting correctly after make tools with nothing else on your PATH. make tools runs only the pnpm install; make build-ts also downloads the oldest Node that engines.node admits (a 45–51 MB tarball, about 175 MB unpacked, once, into .bin/, verified against a pinned sha256) to smoke-load the built SDK entry points with make smoke-ts-dist; the Playwright Chromium binary (~130 MB) is fetched on-demand by make build-docs / make dev-docs via the internal install-playwright-docs target, so Go-only contributors don’t pay the download cost. When you do hit build-docs / dev-docs, Chromium is required by two parts of the docs build: rehype-mermaid (SVG diagram rendering) and the diagramPng integration from @wave-rf/astro-themed-mermaid/png (wired in docs/astro.config.mjs), which rasterizes each diagram to light/dark PNGs (a solid surface-card variant plus a transparent-background variant for slide decks) at astro:build:done for the Copy/Download buttons in the diagramZoom lightbox. Both reuse the same Playwright Chromium, as does the manual docs/scripts/screenshot.mjs QA helper. starlight-links-validator runs under build-docs / CI only — the dev-docs watch loop skips it so a mid-edit dangling link doesn’t fail every rebuild (CI still enforces link validity before merge; run MD_ROUTER_DEV_STRICT=1 make dev-docs to keep the validator on locally). The --with-deps flag (which apt-installs Chromium’s system libraries: libnspr4, libnss3, etc.) is only added when $CI is set, so contributor laptops don’t get an unexpected sudo prompt. On Linux dev machines without those libs already present, run pnpm --filter wavehouse-docs exec playwright install-deps chromium once manually. The docs site is a pnpm workspace package (wavehouse-docs); the root Makefile drives it directly via pnpm --filter (no sub-Makefile) — the *-docs targets show up in make help. It is also a real @wavehouse/sdk consumer, but of the published package: the landing page’s live demo takes the SDK from the registry, pinned in docs/package.json, not from clients/ts/. That is deliberate — the demo streams against a separately-deployed backend, so binding it to the workspace would ship an unreleased wire format to the live site the moment a change merged (#568). The pin is raised on purpose when a new SDK release is tagged. Nothing in the docs build needs build-ts, including when you drive Astro directly through pnpm (e.g. pnpm --filter wavehouse-docs run start). pnpm’s hoisted fallback is off; see Updating Dependencies.

Verify your setup

go version          # go1.21+ (make runs go.mod's version itself)
make --version      # GNU Make 4.x
docker compose version
node --version      # v22.x (matches .nvmrc and CI)
pnpm --version      # 11.21+
jq --version

If any of those are wrong/missing, the Makefile recipes will fail or misbehave confusingly (e.g. a failure buried in interleaved parallel output on Make 3.81; pnpm: command not found on make tools).

Optional but recommended

Tool	Why	Install
Claude Code	The repo ships team-wide configuration in .claude/ — slash commands, subagents, hooks, status line. See Claude Code & AI agents for setup.	brew install --cask claude-code (macOS) or follow official install
worktrunk	Wraps git worktree for parallel-agent workflows. Project hooks live in .config/wt.toml (auto-runs make tools on new worktrees, make verify on pre-merge).	brew install worktrunk && wt config shell install

Quick Start

This is the fastest way to get a fully functional local environment:

# 1. Clone and bootstrap (Go modules + golangci-lint + pnpm deps)
git clone https://github.com/Wave-RF/WaveHouse.git
cd WaveHouse
make tools

# 2. Start ClickHouse (the only external dependency)
docker compose -f deployments/compose/dependencies.yaml up -d clickhouse

# 3. Create a table in ClickHouse
docker compose -f deployments/compose/dependencies.yaml exec clickhouse \
  clickhouse-client --query "
    CREATE TABLE IF NOT EXISTS clicks (
      page String,
      button String,
      score Float64,
      received_timestamp DateTime64(3, 'UTC') DEFAULT now64(3, 'UTC')
    ) ENGINE = MergeTree()
    ORDER BY (page)
  "

# 4. Run with hot-reload (recompiles on every .go file save)
make dev

WaveHouse is now running at http://localhost:8080 in standalone mode with:

• Embedded NATS™ (JetStream) — no external MQ needed
• In-process cache (Ristretto, cache.backend: local) — no external cache needed; to try the shared one, start Redis with docker compose -f deployments/compose/dependencies.yaml --profile redis up -d and set WH_CACHE_BACKEND=redis WH_CACHE_REDIS_ADDRS=localhost:6379
• Trial policy — the dev settings directory ./settings is seeded on first run with the compose stack’s permissive public policy, so tokenless requests to the demo tables work (see Test the API)
• Dedup disabled by default — no Pebble needed
• Schema discovery — automatically finds your ClickHouse tables

Test the API

On first run make dev seeds the gitignored ./settings directory with wavehouse bootstrap and copies in the compose stack’s trial policy (deployments/compose/settings/policies.json + roles.json — the public role: read/write clicks/events, no token). WaveHouse is otherwise fail-closed (the bootstrap seed ships no policy), so a ./settings you emptied by hand denies every request until you put a policy back. Edits to any file in ./settings hot-reload without a restart.

The tokenless data-plane calls work (create a clicks table first — see the Getting Started walkthrough):

# Ingest an event
curl -s -X POST "http://localhost:8080/v1/ingest?table=clicks" \
  -H "Content-Type: application/json" \
  -d '{"page": "/home", "button": "signup", "score": 42.5}'
# → {"ok":true}

# Query it back (wait for the batch flush: ingest.max_batch_wait_ms, 5s in the seed)
curl -s -X POST "http://localhost:8080/v1/query?table=clicks" \
  -H "Content-Type: application/json" \
  -d '{"columns": ["page", "button", "score"], "limit": 10}'

# Open an SSE stream for a specific table (Ctrl+C to stop)
curl -N "http://localhost:8080/v1/stream?table=clicks"

# With gap-fill (replays events since the given timestamp, then switches to live)
curl -N "http://localhost:8080/v1/stream?table=clicks&since=2026-03-24T11:00:00Z"

# Liveness / readiness (no auth required)
curl http://localhost:8080/livez   # → {"status":"ok"}
curl http://localhost:8080/readyz  # → {"status":"ready"}

The admin surface — /v1/ops/schema, /v1/ops/query (raw SQL), /v1/ops/dlq/stats — needs the admin role, which the public trial role doesn’t have. Mint an admin JWT (see Validating tokens below) and pass it:

curl -s http://localhost:8080/v1/ops/schema -H "Authorization: Bearer $TOKEN" | jq
curl -s -X POST http://localhost:8080/v1/ops/query -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" -d '{"sql": "SELECT * FROM clicks LIMIT 10"}'
curl -s http://localhost:8080/v1/ops/dlq/stats -H "Authorization: Bearer $TOKEN"

How make dev works

make dev is a one-stop convenience target for backend and frontend development. The recipe is essentially:

dev: deps-up $(AIR)
    air -c .air.toml

deps-up runs docker compose ... up -d --wait clickhouse, which blocks until the ClickHouse container’s /ping healthcheck flips to healthy. $(AIR) lazily installs air to .bin/<os>_<arch>/ if missing. Then air takes over: it watches cmd/, internal/ and pkg/ (the .go and .yaml files within them), rebuilds tmp/wavehouse on change, and restarts the binary. Boot config is not hot-reloaded (the settings directory it points at is — see Enable Dedup): make dev runs the binary with WH_CONFIG=.config.local.yaml — a gitignored personal copy seeded once from config.yaml on first run (it won’t re-copy if it already exists). So to change dev config, edit .config.local.yaml (not config.yaml) and restart make dev; air watches neither root file. The flip side: when config.yaml gains a key your copy lacks, your copy is stale — make dev warns when config.yaml is newer than .config.local.yaml, and the fix is to port the change over or delete .config.local.yaml to re-seed it. settings.dir is the one key that will stop the server outright if it’s missing (it’s required with no default), so a checkout whose local copy predates it fails at config validation until re-seeded.

air is pinned to a specific version and installed via go install rather than a go.mod tool directive — its transitive deps (Hugo, godartsass, Sass libs) would bloat go.sum for everyone. Same exclusion principle as golangci-lint.

While make dev is running you get:

• WaveHouse on http://localhost:8080 with the default allow-all CORS posture, so a browser-based app on any localhost port can hit the API directly.
• The trial public policy ships in ./settings (see Test the API); its auth.jwt_secret is empty, so no token validates until you set one there (see Validating tokens).
• ClickHouse on http://localhost:8123 (HTTP) and localhost:9000 (native protocol), Compose project name wavehouse-dev so containers/volumes are namespaced.
• Go’s pprof endpoints on http://localhost:6060/debug/pprof/ (go tool pprof http://localhost:6060/debug/pprof/heap): make dev sets WH_PROFILING_ENABLED=true unless it is already set in your environment, so WH_PROFILING_ENABLED=false make dev turns them off. The listener’s address is profiling.addr in .config.local.yaml (see Configuration → Profiling).
• Hot reload: editing any .go file under cmd/, internal/ or pkg/ triggers a debounced rebuild + restart. Boot config isn’t hot-reloaded (settings-directory files are) — make dev loads .config.local.yaml (a gitignored copy seeded once from config.yaml), so edit .config.local.yaml and restart to apply config changes. Air’s stdout/stderr stream live so you see compile errors and server logs in the same terminal.

Dev convenience targets

These are the small targets behind make dev — useful directly when you want to run WaveHouse outside of air (e.g. make settings/config.json && make build && ./bin/wavehouse), or when you need to poke at ClickHouse:

Target	What it does
make deps-up	Start ClickHouse and block until healthy. Idempotent.
make deps-down	Stop ClickHouse. Data volume is preserved.
make deps-logs	docker compose logs -f clickhouse (Ctrl+C detaches; container keeps running).
make deps-shell	Drop into a clickhouse-client REPL on the running container.
make deps-wipe	Stop ClickHouse and destroy its data volume. Use when you want a clean schema.
make clean-all	Nuclear option — every make artifact + dev/E2E containers + volumes + data/.

Stopping make dev: Ctrl+C stops air, which propagates SIGINT to WaveHouse for a graceful shutdown (NATS JetStream flush, etc.). ClickHouse stays up — re-running make dev is fast because the volume is preserved. Use make deps-down or make deps-wipe to stop ClickHouse explicitly.

Running with observability

With otel.enabled on, WaveHouse exports standard OpenTelemetry™ (OTLP) data to the endpoint in OTEL_EXPORTER_OTLP_ENDPOINT. Rather than coupling a heavy observability database stack to the dev server, we provide three lightweight, single-container dashboard options.

You run these in a separate terminal tab alongside make dev or your test suites (make test-e2e).

They block the terminal and stream logs; simply press Ctrl+C to instantly tear them down and clean up the container.

Target	What it does	UI URL
make obs-aspire	Boots the Aspire dashboard. Extremely fast, in-memory only, and requires no login. Ideal for quick trace and log debugging.	http://localhost:18888
make obs-grafana	Boots Grafana® LGTM (Loki™, Grafana, Tempo™, Prometheus®). Pre-configured to bypass login. Best for advanced UI charting and trace-to-log correlation.	http://localhost:3000
make obs-front	Boots OTel-Front for a basic, alternative trace viewer.	http://localhost:8000

Typical Workflow:

1. Open Tab 1: run make obs-aspire (UI opens automatically)
2. Open Tab 2: run make test-e2e, which enables OTel and points it at http://127.0.0.1:4317 for you, or WH_OTEL_ENABLED=true OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 make dev (make dev leaves OTel off, and the SDK’s unset default is TLS, which these local receivers reject)
3. View traces, metrics, and logs flowing into the UI instantly. No accounts or auth tokens required.

Using the SDK against make dev

There’s no bundled playground — point an @wavehouse/sdk client at your local server (baseURL: "http://localhost:8080"); the trial policy make dev seeds into ./settings authorizes the demo-table requests:

For streaming, use @wavehouse/sdk 0.2.0 or later against a local server

A make dev server speaks the v2 SSE wire — an event: schema frame plus positional rows — which @wavehouse/sdk speaks from 0.2.0; the 0.1.x line speaks the pre-v2 one. Point a 0.1.x client at /v1/stream and every event arrives with data: undefined, with no error callback and nothing to catch, while REST and structured queries keep working — which makes it easy to misread as a server bug. Use 0.2.0 or later, the in-tree SDK (make build-ts) or @wavehouse/sdk@dev for streaming work. See Transport Behavior, which carries the full danger note on upgrading the SDK and the server together.

make dev

See the SDK guide for the client API and examples.

Frontend devs running their own dev server (Vite, Next.js, etc.) can import { createClient } from '@wavehouse/sdk' and point baseURL: 'http://localhost:8080'; CORS is permissive so cross-origin browser requests just work.

Validating tokens

There is no auth on/off switch — the JWT middleware always runs, but authorization is the policy’s job (an empty policies.json adopts no policy and denies every token-based caller, admins included — only the operator key below still reaches the admin surface). To exercise token auth in dev, set a known secret in ./settings/config.json (a settings key, so it reloads live under a running make dev) — the trial policy’s admin_role defaults to admin, so a JWT with role: admin unlocks the admin surface:

"auth": { "jwks_url": "", "jwt_secret": "my-secret", "role_claim": "role" }

The operator key is a non-JWT alternative: set one and send it in an Authorization: Operator <key> header (or the X-Operator-Key alias). Even with no policy adopted it reaches the admin surface — enough to trigger a settings reload over HTTP after fixing policies.json (the break-glass path). Once a policy is adopted, the key’s role resolves to admin, so it then has full data-plane access too (pipes, queries, streaming, ingest) — handy for trialing without minting a JWT:

WH_AUTH_OPERATOR_KEY=dev-operator-key make dev
# ...then, in another shell — the admin surface works even with no policy adopted:
curl -X POST -H "Authorization: Operator dev-operator-key" http://localhost:8080/v1/ops/settings/reload
# the X-Operator-Key alias works too:
curl -H "X-Operator-Key: dev-operator-key" http://localhost:8080/v1/ops/pipes

The exception is a pipe whose SQL binds a {{ jwt.<path> }} claim: the key carries no claims, so such a pipe answers it [] (a write pipe, 403). Trial those with a token that carries the claim (Binding token claims).

Then mint a token (role == the policy admin_role) and call an admin endpoint:

# Using jwt-cli (https://github.com/mike-engel/jwt-cli)
export TOKEN=$(jwt encode --secret "my-secret" '{"role": "admin", "exp": 9999999999}')

curl -s -X POST http://localhost:8080/v1/ops/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sql": "SELECT * FROM clicks LIMIT 10"}'

Enable Dedup (Optional)

make dev seeds ./settings (gitignored) from the embedded seed on first run and points settings.dir at it; the seed ships with dedupe.enabled: false. Flip it there — never in the checked-in pkg/settings/seed, which is what wavehouse bootstrap writes for everyone:

# ./settings already exists after the first `make dev` (or: make settings/config.json)
# set "enabled": true under "dedupe" in ./settings/config.json

The key hot-reloads, so once the server is running you can toggle it by editing config.json — no restart. Records dedupe on their event_id field by default; the same file overrides the field globally or per table (see Settings Directory — Deduplication).

Then include the dedup field in your ingest body:

curl -s -X POST "http://localhost:8080/v1/ingest?table=clicks" \
  -H "Content-Type: application/json" \
  -d '{"event_id": "550e8400-e29b-41d4-a716-446655440001", "page": "/home"}'
# → {"ok":true}

# Same event_id again → deduplicated
curl -s -X POST "http://localhost:8080/v1/ingest?table=clicks" \
  -H "Content-Type: application/json" \
  -d '{"event_id": "550e8400-e29b-41d4-a716-446655440001", "page": "/home"}'
# → {"duplicate":true}

Using an .env File

# .env — boot config only (the operator key, sizing, exporters); the
# ClickHouse wiring and the token verifier are settings-directory keys
export WH_AUTH_OPERATOR_KEY=

Then:

source .env
make settings/config.json   # seeds the gitignored ./settings that config.yaml points at (no-op if it exists)
go run ./cmd/wavehouse

Building

# Build the binary to bin/
make build

# Build individual binaries
go build -o bin/wavehouse ./cmd/wavehouse

Running Modes at a Glance

What you want	Command
Hot-reload standalone dev server	make dev
Standalone binary (default config)	make settings/config.json && make build && ./bin/wavehouse
Standalone via Docker Compose	docker compose -f deployments/compose/standalone.yaml up -d
Infrastructure deps only (ClickHouse)	docker compose -f deployments/compose/dependencies.yaml up -d clickhouse

Testing

How It Works

All Go test commands use gotestsum for pytest-style colored output with pass/fail icons, durations, and a summary. Tool versions are pinned in go.mod via tool directives — the Makefile invokes them as go tool <name>, so no global installation is needed.

All tests run with Go’s race detector (-race) enabled by default. WaveHouse is highly concurrent (NATS consumers, singleflight caching, SSE hubs) — the race detector catches data races that would panic in production.

Quick Reference

# Prefix any test target with V=1 for verbose output, e.g. `V=1 make test`

# Unit tests (compact output) — alias for `test-unit`
make test

# Run specific test(s)
make test ARGS="-run TestValidate"

# Go integration tests (requires Docker)
make test-integration

# SDK vitest unit tests + coverage + gate against suites.ts-unit
# (`make cov` auto-merges ts-unit + ts-e2e — no separate command)
make test-ts

# E2E SDK suite against bin/wavehouse-cov
make test-e2e

# All four suites sequentially + merged coverage
make test-all

# Full CI: parallel verify + builds (Go + SDK + docs) + test + test-ts,
# then test-integration + test-e2e + cov
make ci

# Merge available covdata + gate against total threshold
make cov

Each test target writes covdata to tmp/coverage/<suite>/data/, renders a textfmt + HTML report, and gates against the per-suite threshold in .testcoverage.yml. make cov merges whichever suites have run and gates against the total.

Verbose output: Use V=1 to switch from compact testdox format to full verbose output. This is a standard Makefile convention (make test -v can’t work because -v is a make flag).

Extra flags: All test targets accept ARGS="..." for additional go test flags (e.g., -run, -count, -timeout).

Time limits: make test-unit gives each unit package 15s (UNIT_TIMEOUT), the budget it has to fit when the suite runs on its own, as it does in the unit job in CI. make ci gives it 60s (CI_UNIT_TIMEOUT) in its parallel phase, where the suite shares every core with the lint and build jobs, so a package over the budget can pass make ci and still fail in CI. tests/integration gets 900s (INTEGRATION_TIMEOUT). Each is a Makefile variable, so a slower machine can raise one for a run: make ci CI_UNIT_TIMEOUT=90s.

Note on timing: gotestsum’s DONE ... in X.XXXs reports pure test execution time. The total wall time includes Go compiling all packages — the first run compiles everything (~15s), subsequent runs use the build cache (~1s).

Build cache across worktrees: make exports GOFLAGS=-trimpath (your own GOFLAGS are kept), so compiled packages don’t embed the checkout’s path and every worktree of the same source shares one build cache entry per package and variant, instead of building its own. Opt out with GOFLAGS=-trimpath=false make …. make build is the exception: its debug binary keeps absolute source paths, so a debugger finds the files. Release and coverage binaries and test binaries built through make are trimmed; their frames read github.com/Wave-RF/WaveHouse/internal/… (standard-library frames are plain import paths such as runtime/proc.go), so map the module path to your checkout, for example, at the (dlv) prompt, config substitute-path github.com/Wave-RF/WaveHouse /absolute/path/to/your/checkout (or a substitute-path entry with from and to in ~/.config/dlv/config.yml, or your editor’s substitutePath). Code that needs a file from the repository finds it from the working directory, never from runtime.Caller, whose paths are module-relative under -trimpath.

Running make ci on another machine

make ci-remote HOST=<host> runs make ci for your HEAD commit on another machine and streams its output back, so the heavy suites can run off your laptop, or on another OS or architecture than yours (CI runs on Linux amd64).

make ci-remote HOST=build-box                  # any ssh destination: user@host, a ~/.ssh/config alias
make ci-remote HOST=docker:wavehouse-dev       # a running local container or devcontainer
make ci-remote HOST=build-box KEEP=1 REMOTE_DIR=/scratch/wavehouse-ci

Only the commit travels, so the worktree must be clean: HEAD is streamed as a git bundle over the ssh or docker exec connection, and nothing is pushed. The host checks it out in a fresh directory under REMOTE_DIR (default .cache/wavehouse-ci; a relative path is under the host’s home), runs make tools && make ci there, and removes the directory afterwards unless KEEP=1. After the local ==> make ci on … line, the first line the host prints is its uname -sm, and each run keeps a copy of its log in tmp/ci-remote-<tree>-<run>.log, so two runs of one tree, say on two architectures at once, don’t overwrite each other. If make ci-remote is stopped, by Ctrl-C or a signal, the host stops its make ci within a second or two: the host’s end of the connection stays open only while the local command runs, and the host stops the run when it closes.

The host needs what a local make ci needs: bash, git, curl, jq, GNU Make 4+, Go, Node and pnpm from Prerequisites, network access for make tools, and a Docker daemon for the integration and e2e suites. Over ssh the commands run in a non-interactive login shell, so the tools must be on the PATH the host’s profile sets; in a container they must be on the container’s own PATH, and the container must reach a Docker daemon (a mounted socket or Docker-in-Docker). If make ci runs inside a container on the host’s network, testcontainers may try to reach the suites’ containers through the Docker bridge gateway and time out; setting TESTCONTAINERS_HOST_OVERRIDE=localhost in the host’s environment points it at the published ports instead. A Linux host without Chromium’s system libraries needs them installed once for the docs build, as Prerequisites describes (pnpm --filter wavehouse-docs exec playwright install-deps chromium).

How the pre-push gate treats it. The gate’s meaning doesn’t change: a code push still needs tmp/ci-passed-tree-<tree>, which says make ci passed for exactly that tree. make ci-remote writes it (and the verify marker) only if the host exits 0, the tree the host reports having checked out equals your HEAD^{tree}, make ci left the host’s checkout clean, and your worktree is still clean and on that tree when the run ends; otherwise it writes nothing. Beside the marker, tmp/ci-passed-tree-<tree>.provenance records the host, its OS and architecture, the start and end times and the log path, and the pre-push hook prints make ci passed on <host> (<os>/<arch>) from it. A marker from a local make ci has no provenance file and works exactly as before.

Test Structure

Category	Location	Docker?	Command
Unit tests	internal/**/*_test.go, pkg/**/*_test.go	No	make test
SDK unit tests	clients/ts/src/**/*.test.ts	No	make test-ts (always includes coverage + gate)
Integration tests (Go)	tests/integration/*_test.go, internal/cache/*_integration_test.go, plus internal/mq/natsspike and internal/mq’s integration-tagged tests	Yes	make test-integration
E2E tests (SDK)	tests/e2e/sdk/*.test.ts	Yes	make test-e2e

• Unit tests live beside the code they test (e.g., internal/discovery/discovery_test.go). They use mocks or embedded NATS (in-process, no Docker needed).
• Integration tests use the //go:build integration build tag. In tests/integration, TestMain starts one ClickHouse testcontainer and a dynamodb-local one (for the DynamoDB dedupe backend’s tests), and boots the production wiring against it through app.New (embedded NATS, ingest worker, sweeper, hub, the API server on a random loopback port); tests reach it via env(t) and create their own tables. TestNATSBackend_EndToEnd also starts a NATS container configured from deployments/nats/values.yaml, applies deployments/nats/jetstream.yaml to it through internal/mq/natstest, and boots two processes on mq.backend: nats against it. TestMain also builds the wavehouse binary while the containers start, so the build is not charged to the -timeout: the TestRoles_* tests run it as separate OS processes (two API, two then three ingest, one killed with SIGKILL) over NATS, Redis and dynamodb-local, and read each ingest process’s shard ownership from its /metrics. DLQ tests use assert.Eventually with a 30-second timeout for the 5-second ingest worker batch window. internal/cache’s integration tests start their own containers instead — Redis, Valkey, Dragonfly and a one-node Redis Cluster — for the shared backend. shared_cache_test.go starts its own Redis testcontainer per test (startRedis) and boots extra, independent cache.backend: redis instances over that same ClickHouse (bootRedisApp), to exercise the cache shared across processes rather than one package in isolation. make test-integration runs the suite in two parts at once, and CI runs each part as a job of its own: make test-integration-app runs tests/integration, and make test-integration-backends runs the backends’ own suites. The backends part also runs internal/mq/natsspike, which pins the nats-server behavior the external-NATS topology depends on, against an in-process server with no Docker. It lives under internal/mq because only that tree may import NATS, and it runs here rather than in the unit suite because each test takes seconds and the unit suite has a 15-second limit per package. For the same reason the external NATS broker’s tests (internal/mq/external*_test.go, including its run of the mqtest conformance suite) and the NATS KV lease tests (internal/mq/lease_test.go, including their run of the coordtest conformance suite) carry the integration tag inside internal/mq, and the backends part runs the tests those tagged files declare (scripts/ci/tagged-tests.sh), whatever their names, so the package’s untagged tests stay in the unit suite alone. In tests/integration, most of the slow tests that bring up their own servers call t.Parallel() (never one that touches process-wide state, such as t.Setenv or logtest.Capture), and the app part runs with -parallel 8, since those tests mostly wait.
• Images come from Docker Hub, except Dragonfly’s, and Docker Hub limits anonymous pulls per IP address. CI points each container job’s Docker daemon at mirror.gcr.io, a cache of Docker Hub, before the first pull, with Docker Hub as the fallback (invariant 8 of .github/workflows/README.md). A local run that fails with toomanyrequests can do the same: add "registry-mirrors": ["https://mirror.gcr.io"] to the Docker daemon’s configuration, merging it into the existing JSON object, and restart the daemon (Docker Desktop: Settings → Docker Engine, then Apply & restart; Linux: /etc/docker/daemon.json, then sudo systemctl restart docker; rootless Docker: ~/.config/docker/daemon.json, then systemctl --user restart docker), or run docker login.

Shared test utilities live in internal/testutil/. The packages log through slog.Default(), so tests reach log output through internal/testutil/logtest: logtest.Silence() in a package’s TestMain discards it, and logtest.Capture(t, level) routes it to a buffer for a test that asserts on log lines — such a test must not call t.Parallel(), because the default logger is process-wide. A test that starts the embedded broker keeps its store in internal/testutil/storedir’s storedir.New(t) rather than a bare t.TempDir() (testutil.NewEmbeddedMQ does): the NATS server can finish writing a consumer’s state after Close returns, which fails t.TempDir’s one-shot removal, and storedir removes the store again until those writes have landed (#442).

Adding New Tests

• Unit test for internal/foo/ → create internal/foo/foo_test.go (same package).
• Integration test needing Docker → add a subtest under tests/integration/ (e.g. a new file with //go:build integration). A test of one package against its own external server — the shared cache backend against Redis, Valkey and Dragonfly containers — lives beside the package instead (internal/cache/redis_integration_test.go, same build tag), and the package joins a part of the suite in the Makefile (INTEGRATION_APP_PKGS or INTEGRATION_BACKEND_PKGS): every part fails while a package with integration-tagged tests is in none of the listed parts, while a test-integration-<part> target is missing from INTEGRATION_PARTS, or while a listed part has no target. A package listed there runs all its tests, untagged ones too (internal/cache does); one whose untagged tests belong to the unit suite alone instead gets a run of its own, selected by scripts/ci/tagged-tests.sh run, as internal/mq has in test-integration-backends.
• E2E test via SDK → add a tests/e2e/sdk/*.test.ts file. These tests exercise the full pipeline (ingest → ClickHouse → query) through the TypeScript SDK. Run with make test-e2e.
• Test helpers → add to internal/testutil/ (Go) or tests/e2e/sdk/helpers.ts (E2E).

E2E Tests via SDK

The primary E2E integration test suite lives in tests/e2e/sdk/. It uses the TypeScript SDK as the test harness — every ingest→query test simultaneously validates the full Go backend pipeline and confirms SDK compatibility.

Architecture:

• scripts/orchestrator — the E2E entrypoint behind make test-e2e: it starts a clean ClickHouse testcontainer and a Redis one (the fixture’s shared cache, cache.backend: redis) per run, launches the wavehouse-cov binary on a random free port, runs the SDK suite against it, then SIGINTs the binary to flush coverage. No Compose file is involved. CI runs the exact same path.
• tests/e2e/sdk/setup.ts — globalSetup. Probes the CLICKHOUSE_URL / WAVEHOUSE_URL the orchestrator injects, creates the per-suite tables, refreshes the schema, and writes the baseline policy into the run’s settings directory (adopted via POST /v1/ops/settings/reload — files are the only write path). It starts nothing itself and fails fast if either URL isn’t up. It also prints the active Node/undici version, warning when the local Node major differs from .nvmrc — a runtime-specific transport bug is otherwise indistinguishable from a code failure (see #440).
• tests/e2e/sdk/helpers.ts — JWT factories, typed client constructors, async wait helpers, direct ClickHouse query helper.

Running E2E tests:

# Build the cover binary, install deps, run all E2E tests
make test-e2e

make test-e2e builds bin/wavehouse-cov (coverage-instrumented) and runs the orchestrator under scripts/orchestrator/ to wire ClickHouse + the cover binary into the suite. covdata flushes on SIGINT into tmp/coverage/e2e/data/.

The orchestrator always provisions its own stack — fresh ClickHouse and Redis testcontainers plus wavehouse-cov on a random free port — so a running make dev on :8080 is neither detected nor reused, and the two don’t collide. To run vitest against a stack you manage yourself, start a Redis for the fixture’s cache.backend: redis, then the server from the repo root with the E2E fixture config:

docker compose -f deployments/compose/dependencies.yaml --profile redis up -d
WH_CONFIG=tests/e2e/fixtures/config.yaml WH_CACHE_REDIS_ADDRS=localhost:6379 go run ./cmd/wavehouse

The fixture matters: the suite signs its tokens with its sdk-dev-secret and depends on its dedupe, DLQ, and 5s schema-refresh settings. Point the suite at a default make dev server (whose ./settings/config.json has an empty auth.jwt_secret, so the suite’s tokens do not validate) and setup’s schema calls are rejected, then global setup dies 30s later on a misleading schema not refreshed within 30s. The repo root matters too — the fixture’s settings.dir is relative to the working directory. The fixture’s settings directory (policy, pipes, and tunables) points at ClickHouse on localhost:9000; if yours isn’t there, edit clickhouse.addr / http_port in tests/e2e/fixtures/settings/config.json (the orchestrator patches them itself for its testcontainer).

Prefixing the variable to make dev does not work: that recipe pins WH_CONFIG=.config.local.yaml inline, which overrides anything inherited from the environment.

Then set CLICKHOUSE_URL / WAVEHOUSE_URL and run pnpm test from tests/e2e/sdk/; teardown is a no-op on that path, so your stack survives between iterations.

If a previous run was killed (harness timeout, stop button, SIGKILL), it can leave a wavehouse-cov behind. That process shares tmp/data and tmp/wavehouse-cov.log with the next run and will corrupt it, so the orchestrator kills any leftover before starting and says so.

Environment knobs:

Variable	Effect
V=1	Stream the WaveHouse subprocess log live in addition to capturing it to tmp/wavehouse-cov.log. The on-failure log excerpt is then skipped — you have already seen it
E2E_CH_QUERY_TIMEOUT_MS	Per-request ceiling for the suite’s direct ClickHouse queries (default 10000)
E2E_NO_COVERAGE=1	Drop --coverage from the vitest run (skips v8 instrumentation and report generation) while chasing a flake. Local debugging only — no report is written. Ignored (with a log line) under make ci / make test-all, so an exported-and-forgotten var can’t produce a green coverage gate with the TS e2e report missing

Test files (tests/e2e/sdk/*.test.ts): admin, auth, batching, cache, dlq, ingest, ndjson, query, streaming, stress, plus helpers — a stack-free unit test of the harness’s own waitForCondition poll helper rather than a pipeline test.

Linting

make lint

golangci-lint is kept out of go.mod (its dependency tree conflicts with the main module), so the Makefile pins the version instead and downloads it into .bin/ on first use — make lint and make tools both handle it, and no global install is needed.

A global copy is never what make lint runs, so install one only if you want to invoke golangci-lint outside make:

• macOS: brew install golangci-lint
• Binary: See golangci-lint.run/welcome/install/

The Makefile exports GOTOOLCHAIN pinned to go.mod’s toolchain line if it has one, else its go directive, so local runs and CI use the same Go even when your installed Go is newer (the first run downloads that toolchain once). The source-build deployments/Dockerfile pins the matching golang image tag instead (the image sets GOTOOLCHAIN=local), and make check-dockerfile-go, part of make verify, fails if the tag and go.mod’s Go version differ. A bare go test outside make is not pinned. Raising the go directive to a new minor version (1.27, not a 1.26 patch) also needs a golangci-lint release built with that minor or newer: otherwise make lint-go refuses with can't load config: the Go language version (go1.26) used to build golangci-lint is lower than the targeted Go version (1.27.1). Bump the directive with go get go@1.N.P, because go mod edit -go=1.27 writes a version make cannot pin.

The configuration is in .golangci.yml (v2 format with default: none for explicit control) — that file is the authoritative list of enabled linters. Highlights:

• errcheck — Unchecked error returns
• govet — Suspicious constructs
• staticcheck — Static analysis
• unused — Unused code
• gosec — Security issues
• gocritic — Opinionated style checks
• revive — Extensible linter (replaces golint)
• ineffassign — Ineffective assignments
• misspell — Spelling errors in comments/strings
• bodyclose — Unclosed HTTP response bodies
• noctx — HTTP requests without context
• errorlint — Proper error wrapping checks (%w, errors.Is/As)
• tparallel — Missing t.Parallel() in test subtests
• depguard — Import boundaries: only internal/mq may import NATS/JetStream (github.com/nats-io/…); everything else goes through an mq-owned type. And the pkg/ packages import only the standard library, each other, go-humanize and yaml.v3 (plus testify in tests): an allow list, so a new dependency there fails lint until it is listed

Formatting (gofumpt — strict superset of gofmt — and goimports import grouping) is enforced through the v2 formatters: section rather than as linters.

Markdown and MDX

make lint also covers documentation: markdownlint owns Markdown and MDX style (rules in .markdownlint.json; file selection in .markdownlint-cli2.jsonc for .md, plus the **/*.mdx glob on lint:md in package.json) and misspell owns spelling. Two rules are repo-local, in scripts/markdownlint-rules/:

• WH001 / no-hard-wrapped-prose — a paragraph must be one line. Hard-wrapping at 72/80 columns turns a one-word edit into a five-line diff. Autofixes. Tables (leading pipes optional), code, headings, setext underlines, blockquotes, JSX, $$ display math, and multi-line MDX import/export are left alone; a list item is joined as a unit; an aside’s body is joined but its ::: delimiters are not. Note the four-space rule cuts both ways: any line indented four or more spaces is read as an indented code block and skipped, so a nested list item is never joined and must be unwrapped by hand.
• WH002 / mdx-fence-needs-blank-line — an MDX code fence directly against a JSX tag (<TabItem label="YAML"> immediately followed by a fence). MDX renders that fine; the problem is that markdownlint parses CommonMark, where the tag opens an HTML block that runs to the next blank line — so the fence is not a code block to any generic rule, and markdownlint --fix will happily reformat the code inside it. The blank line is what keeps the two parsers agreeing.

Run make fix to apply them. The generic markdownlint fixers run over .md only. markdownlint parses CommonMark and MDX does not, and where the two disagree an autofix rewrites the inside of a code block — de-indenting YAML comments it reads as headings, autolinking bare URLs. Reporting that disagreement is useful, so make lint still checks .mdx; acting on it is not. MDX therefore gets exactly one structural fixer, scripts/fix-mdx-fences.mjs, which only ever inserts a blank line beside a JSX tag. (misspell still auto-corrects spelling in .mdx — its corrections come from a curated list and don’t depend on parsing the document.)

The practical consequence: a problem make lint reports in an .mdx file may need fixing by hand, WH001’s hard-wrapped prose included. A bare markdownlint-cli2 --fix can’t reach MDX either: .markdownlint-cli2.jsonc globs .md only and the .mdx glob sits on lint:md, so the guard is structural rather than a rule you have to remember. Improving this — a formatter that understands MDX rather than one that guesses — is tracked in #499.

The markdownlint extension reads .markdownlint-cli2.jsonc, customRules and all, so no editor setting is needed — but it only activates on Markdown, so in-editor squiggles cover WH001 in .md and never WH002 in .mdx. Agent-written files get the same treatment at write time from .claude/hooks/markdown-on-save.sh — markdownlint in .md, the fence pass in .mdx — and everyone else gets it from make fix.

Project Structure

WaveHouse/
├── cmd/                    # Binary entry points
│   └── wavehouse/          # Standalone all-in-one binary
├── internal/               # Private application packages
│   ├── api/                # HTTP handlers, router, middleware
│   ├── app/                # Process wiring (build every component, run under one errgroup, release in reverse)
│   ├── auth/               # JWT/JWKS authentication middleware
│   ├── cache/              # Query cache: Ristretto L1 + the tenant-led version index; the Redis-compatible shared backend
│   ├── chconn/             # ClickHouse pools, one per connection tuple (reconciled on settings reload)
│   ├── config/             # YAML + env var configuration
│   ├── coord/              # Leases with fencing tokens (in-process Local, RunElected, coordtest suite)
│   ├── dedupe/             # Optional deduplication (Reserve/Commit/Release; Pebble or DynamoDB)
│   ├── discovery/          # ClickHouse schema introspection + validation
│   ├── ingest/             # Batch buffering + DLQ + Active Sweeper + shard claims
│   ├── keyenc/             # One escaping for composite keys (NATS subject tokens, cache keys, dedupe keys)
│   ├── mq/                 # MQ boundary: the only NATS/JetStream importer
│   ├── observability/      # OpenTelemetry pipeline (traces/metrics/logs + Prometheus)
│   ├── query/              # Structured query AST + SQL builder
│   ├── settings/           # Settings directory runtime: adopted snapshot, reload, watcher, WriteSeed
│   ├── stream/             # SSE fan-out: Hub, Subscriber queue, Bucket, keepalive wheel
│   └── testutil/           # Shared test helpers and mocks (cachetest suite)
├── pkg/                    # Public packages: importable at a release tag, no internal/ or runtime dependency
│   ├── chsql/              # Shared ClickHouse SQL helpers (quoting, bind-safety, SQL scanning, write classifier)
│   ├── pipes/              # Named query pipes (types + parameter and claim binding)
│   ├── policy/             # Access control policies (types + evaluation + claim resolution)
│   ├── settings/           # Settings model: document types, Finding, the validator (on disk and over bytes), embedded seed
│   ├── shard/              # Placement hashing: Jump consistent hash, Mix finalizer
│   └── tenant/             # Tenant id: type, grammar, reserved default, request header name
├── tests/                  # Integration & E2E tests
│   ├── integration/        # Go integration tests (//go:build integration)
│   └── e2e/                # E2E suite (SDK specs; `scripts/orchestrator` starts ClickHouse and Redis testcontainers)
│       ├── fixtures/       # E2E server config + settings-directory fixtures
│       └── sdk/            # E2E specs driven through the TypeScript SDK (Vitest)
├── clients/                # Client SDKs
│   └── ts/                 # TypeScript SDK (@wavehouse/sdk, pnpm workspace)
├── deployments/
│   ├── compose/            # Docker Compose files (standalone.yaml, dependencies.yaml)
│   ├── nats/               # Helm values and JetStream manifests for an external NATS (`mq.backend: nats`)
│   ├── Dockerfile          # Runtime image
│   └── Dockerfile.goreleaser  # Release image (built by GoReleaser)
├── scripts/                # E2E orchestrator, cov tool, CI/hook helpers
├── docs/                   # Documentation
├── config.yaml             # Default configuration file
├── Makefile                # Build, test, lint, release targets
├── .golangci.yml           # Linter configuration
├── .goreleaser.yaml        # Release build configuration
└── .air.toml               # Hot-reload configuration

Code Conventions

• Strict Go formatting: Use gofumpt (a stricter superset of gofmt, enforced by CI). make fmt checks it, as does make verify (what CI and the pre-commit hook run); make fix applies it.
• Interface-first design: Core behaviors (Cache, Deduplicator, Publisher, Subscriber) are defined as interfaces so implementations can be swapped behind a stable contract.
• Package boundaries: The internal/ directory ensures packages are private to this module; pkg/ is the importable surface — what a settings writer outside this repository validates with, and what a tool computing the same placements hashes with — so a package there imports nothing under internal/ and none of the server’s runtime dependencies (the public-packages depguard rule).
• Error handling: Return errors to callers. Use slog for structured logging, through the default logger (slog.InfoContext(ctx, …) and its siblings) — constructors don’t take a *slog.Logger; tests silence or capture it with internal/testutil/logtest.
• Schema-driven: ClickHouse is the schema source of truth. WaveHouse discovers and validates against real table schemas.

Makefile Targets

Run make help to see all targets. Key ones:

Target	Description
make help	Show all targets with descriptions (always the source of truth)
make tools	Bootstrap: install pinned tools (golangci-lint, air, misspell, shellcheck, actionlint), Go modules, pnpm deps, and git hooks (core.hooksPath → .githooks/)
Dev
make dev	Hot-reload dev server: ClickHouse via Compose + WaveHouse under air on :8080
make deps-up	Start ClickHouse alone (idempotent; blocks until healthy)
make deps-down	Stop ClickHouse (preserves data volume)
make deps-logs	Tail ClickHouse logs
make deps-shell	clickhouse-client REPL on the running container
make deps-wipe	Stop ClickHouse AND destroy its data volume (DESTRUCTIVE)
Observability
make obs-aspire	Prebuilt 0-config o11y UI to show WaveHouse metrics, logs, and traces locally
make obs-grafana	Grafana alternative to aspire, more advanced and complicated
make obs-front	Custom graphs like grafana, but is simpler and easier to configure like aspire
Static checks
make fmt	Check formatting across Go (gofumpt) + TS (Biome). Run make fix to apply.
make tidy	Verify go.mod/go.sum are tidy (run make fix to apply)
make lint	Run linters across Go (golangci-lint) + TS (Biome) + Markdown/MDX (markdownlint) + prose (misspell)
make vulncheck	Run govulncheck -scan package (V=1: the default symbol-level scan, with example traces)
make verify	Repo-wide static checks: Go (tidy + fmt + vulncheck + lint) + TS (Biome + tsc typecheck) + Markdown/MDX (markdownlint + rule fixtures) + prose (misspell) + shell (shellcheck) + workflows (actionlint) + path-classifier fixtures + release-channel fixtures + go-toolchain fixtures + Dockerfile Go-version check + NATS server-line check (go.mod, the external-NATS test image, the Helm tag, recommendedNATSMinor and deployment.md agree) + tagged-test selector and integration-parts fixtures + PR build-cache and main cache prune fixtures + test-image prefetch fixtures (scripts/ci/prefetch-test-images.sh against the real sources) + review-gate hook tests + git-hook tests (.githooks/) + Claude Code hook-command tests (.claude/settings.json) + make ci-remote tests + docs type-check (astro check — not a full build, so link validation is left to make build-docs, which make ci runs); a bare make verify runs these in parallel
make fix	Auto-fixes across Go (tidy + gofumpt + goimports + lint --fix), TS (Biome --write), Markdown (markdownlint --fix), MDX (fix-mdx-fences only — the generic fixers never run over .mdx), and docs-prose spelling (misspell, both)
Build
make build	Compile wavehouse → bin/wavehouse (debug symbols kept)
make build-release	Stripped release-style build → bin/wavehouse-release
make build-cover	Coverage-instrumented build → bin/wavehouse-cov (used by E2E)
make build-ts	Build TypeScript SDK → clients/ts/dist/, then smoke-load its ESM, CJS and IIFE entry points at the oldest Node engines.node admits, and type-check a CommonJS and an ESM consumer against the built declarations with tsc --module node16
Test
make test	Alias for test-unit
make test-unit	Go unit tests + render coverage + gate suite threshold
make test-integration	Go integration tests (requires Docker) + coverage gate: the parts test-integration-app and test-integration-backends, at once
make test-ts	SDK vitest unit tests + v8 coverage + gate against suites.ts-unit (matches Go’s “always coverage” pattern)
make cov	Merge Go + TS coverage and gate against thresholds. Auto-runs after make test-all and make ci; standalone make cov is “show me the merged numbers without re-running.” Each side skips silently if its data is missing, but make cov fails if both are empty (you ran it before any test target).
make test-e2e	E2E SDK suite against bin/wavehouse-cov + coverage gate
make test-all	All four suites sequentially + merged coverage gate
make ci	Full pipeline: parallel verify + builds + unit/SDK tests, then integration + E2E + cov
make ci-remote HOST=<host>	make ci for HEAD on an ssh host or docker:<container>; writes the same marker if it passes there for this exact tree (details)
Release (see Cutting a release)
make release-server VERSION=X.Y.Z	Tag a server release — binaries + container image
make release-sdk-ts VERSION=X.Y.Z	Tag a TypeScript SDK release — npm
make release-sdk-go VERSION=X.Y.Z	Tag a Go SDK release — go get (pending #434)
Analysis (informational, not in CI)
make size	Binary size analysis → tmp/analysis/ (text + SVG + interactive HTML)
make audit-cgo	Audit dependency tree for C files (builds use CGO_ENABLED=0)
make deadcode	Find unreachable functions
make dep-cut	Top cuttable deps by transitive weight (LIMIT=N to override)
make binary-analysis	Combined: size + audit-cgo + deadcode
Cleanup (tiered — compose explicitly for partial resets)
make clean	Build outputs only (bin/, dist/, clients/ts/dist/, docs/dist/, docs/.dev-dist/)
make clean-test	Test outputs only (tmp/ — coverage data, logs, NATS state)
make clean-tools	Installed tools and pnpm deps (.bin/, node_modules/)
make clean-all	Full reset: above + data/ + Docker volumes

All test targets accept ARGS="..." for pass-through go test flags. Build targets accept TAGS="..." for Go build tags. V=1 switches to verbose gotestsum output.

Dependency Management

Updating Dependencies

pnpm’s hoisted fallback is off (hoist: false in pnpm-workspace.yaml), so after adding or bumping an npm dependency, one that imports something it doesn’t declare fails with Cannot find package 'X' imported from …/node_modules/.pnpm/… (ESM) or Cannot find module 'X' plus a require stack (CJS) when the importing module loads (in a build, test or dev server), not at install time; an undeclared type-only import isn’t caught at all, since with skipLibCheck on it silently degrades to any. Fix a runtime one with a packageExtensions entry in pnpm-workspace.yaml: a dependency entry for a plain missing package, a peer entry when the package must share the app’s instance (astro, for an Astro integration). Report the missing declaration upstream too, so the entry can go: starlight-image-zoom needed an astro peer entry until 0.16.1 declared it.

For Go modules:

go get -u ./...                         # Update the main module's non-test deps (direct and indirect); add -t for test-only deps, and run go get -u tool for the tool block
go mod tidy                             # Remove unused, add missing

Vulnerability Scanning

make vulncheck runs govulncheck -scan package: it reports every known vulnerability in a package your code imports, without checking whether the vulnerable function is reachable. That is stricter than govulncheck’s default call-graph scan, so a hit does not on its own mean the code path is used.

make vulncheck       # package-level scan (what make verify and CI run)
V=1 make vulncheck   # default symbol-level (call-graph) scan, with example call traces

For a combined security scan, run make verify — it runs vulncheck alongside lint, and gosec is one of the linters enabled in .golangci.yml. This is also what CI runs on every push and pull request.

Dependabot

Dependabot is configured in .github/dependabot.yml to open weekly grouped PRs for three update configs:

• Go modules (root) — outdated or vulnerable Go dependencies, commit prefix deps:.
• GitHub® Actions (root and /.github/actions/setup-env) — outdated action versions tracked against the SHA pins across .github/workflows/* and the setup-env composite action, commit prefix ci:
• npm — pnpm workspace (root) — covers all three TypeScript packages (the docs site, the SDK, and the E2E tests) in one grouped PR, commit prefix deps:

PRs are grouped per config to reduce noise. The npm config is pointed at the workspace root (directory: /), not the individual member directories. The repo has a single root pnpm-lock.yaml, and Dependabot only updates a lockfile co-located with the manifest it targets — so a per-member config (the previous setup) bumped a member’s package.json without regenerating the root lockfile, and every such PR then failed CI’s pnpm install --frozen-lockfile with ERR_PNPM_OUTDATED_LOCKFILE. Pointing at the root lets Dependabot read pnpm-workspace.yaml, walk every member, and update the one lockfile.

The GitHub Actions config names two directories. directory: / reaches .github/workflows/ but does not descend into .github/actions/*/action.yml, so the setup-env composite action — which owns every cache in ci.yml — was invisible to Dependabot, and its pins went stale against upstream and diverged from publish-npm.yml, which Dependabot does track and which doesn’t call setup-env. Listing its directory under directories: brings it into the same weekly group; adding a composite action means adding its directory there, because nothing else catches the drift.

The npm config sets a 7-day cooldown to match pnpm’s minimumReleaseAge, with the same @wave-rf/* and @wavehouse/* exclusions as minimumReleaseAgeExclude. It covers only the package Dependabot bumps, so transitive packages can still fail pnpm’s check (#441), and security updates bypass it.

That remainder fails loudly rather than silently: pnpm 11.21 checks every lockfile entry against minimumReleaseAge even under --frozen-lockfile, so a Dependabot lockfile that pulled in a too-young transitive package fails the pnpm install inside make verify with ERR_PNPM_MINIMUM_RELEASE_AGE_VIOLATION and nothing merges. Dependabot cannot repair it itself, so a human does. Commit the fix to the Dependabot PR’s own dependabot/npm_and_yarn/** branch, the same branch an automated write-back would target. Set the manifest range of the bumped package to a release that resolves under the policy, typically the newest release at least 7 days old, which may be newer or older than the version the PR proposed (the PR’s own version can keep the young transitive pins). Then run plain pnpm install so pnpm resolves only the new subtree under the policy and leaves every untouched pin alone, and run pnpm install --frozen-lockfile to confirm the result is policy-clean. On PR #571 the working fix moved astro from ^7.1.1 to ^7.2.10, not to the PR’s 7.2.8. pnpm clean --lockfile && pnpm install also works but re-resolves the whole tree and sweeps in unrelated bumps. Automating this with a workflow that regenerates the lockfile and pushes it back to the dependabot/npm_and_yarn/** branch needs a token that can push and trigger CI, which is not set up; that remains the open part of #441.

typescript majors are held back (ignore: version-update:semver-major) because tsup vendors a rollup-plugin-dts that crashes on TypeScript 7 during clients/ts’s prepare script — i.e. inside pnpm install, which takes every Node job down at once. See the comment in .github/dependabot.yml for the condition that lets it be removed.

No auto-merge. Dependabot PRs go through the same merge gate as any other PR — an approval from the @Wave-RF/wavehouse-admins team (the ruleset’s required_reviewers rule) plus the required checks. (The former dependabot-automerge.yml, which auto-approved and merged patch/minor bumps hands-off, was removed — every bump now gets a human admin review.)

Cutting a release

Cutting a release is one tag — no version bump in code, no release branch:

make release-server VERSION=0.1.0   # tag v0.1.0            → binaries + container image
make release-sdk-ts VERSION=0.1.0   # tag clients/ts/v0.1.0 → @wavehouse/sdk on npm

make release-sdk-go exists too, wired ahead of the Go SDK landing (#434); it refuses to run until clients/go/ is in the repo.

The one thing to do before tagging is promote the changelog: AGENTS.md requires every PR to add its entry under ## Unreleased, so open a PR renaming that heading to ## [X.Y.Z] - YYYY-MM-DD and adding the matching link reference at the foot of the file. When the release needs upgrade steps, add a ## From X to Y section to Upgrading in the same PR. Nothing in the release pipeline reads CHANGELOG.md — this is for the file’s own readers.

After tagging an SDK release, raise docs/package.json’s @wavehouse/sdk range to the new line. A caret range on a 0.x version floats only within that minor (^0.1.1 stops at 0.2.0), so a new minor does not reach the docs site — including its live hero demo — until someone bumps it, which is the point: the bump is where you decide that the deployed demo backend and the site’s SDK move together. minimumReleaseAgeExclude already exempts @wavehouse/* from pnpm’s 7-day minimumReleaseAge (and the Dependabot cooldown excludes it too), so the freshly published version installs immediately. Dependabot will not propose this one for you — @wavehouse/sdk is in the npm ignore list in .github/dependabot.yml for exactly this reason, since a grouped Monday deps: PR is reviewed by someone thinking about dependency hygiene, not about which wire the demo backend speaks.

Each runs scripts/release.sh, which preflights (on main, clean tree, in sync with origin/main, the tag free both locally and on the remote, the required CI check green on this exact commit), prints exactly what will be published, and asks before pushing. DRY_RUN=1 make release-… stops after the plan. Tag creation is admin-only via the release tag protection ruleset.

The components are independent — releasing the server does not publish the SDK, and their version numbers need not match.

Why there is no version to bump

The tag is the version, everywhere:

Component	Where the version comes from
Server	GoReleaser’s -ldflags at build time, from the tag
Go SDK	The tag itself — a Go module has no version file
TypeScript SDK	publish-npm.yml stamps clients/ts/package.json from the tag before publishing

The npm case is the one that usually forces a bump commit, and it is deliberately not the source of truth here: the main ruleset forbids pushing to main directly, so bumping package.json would mean a reviewed PR merged before every release. The committed value is documentation — if it drifts from the tag, the workflow logs a notice and the tag wins.

Tag naming

v0.1.0              server (root Go module)
clients/ts/v0.1.0   @wavehouse/sdk
clients/go/v0.1.0   Go SDK

The clients/<lang>/ prefix isn’t a style choice. Go resolves a module in a subdirectory only against a tag carrying that subdirectory as a prefix (the module reference is explicit), so go get github.com/Wave-RF/WaveHouse/clients/go@v0.1.0 requires precisely clients/go/v0.1.0. Every client follows that shape rather than leaving Go as the exception, and a new language is one more clients/<lang>/v* family.

Tag globs are anchored at the start of the ref name, so v* never matches a clients/... tag — which is what keeps a client release from triggering a server release.

What a release publishes

• Server — a GitHub Release with the Linux archives (amd64 and arm64, .tar.gz) and checksums.txt. A tag carrying a prerelease suffix (v0.1.0-alpha.1) is marked as a GitHub pre-release, so it never takes the “Latest release” badge from a shipped stable version.
• Both — release notes generated by GitHub from the PRs merged since the previous tag in the same family — one line per PR, since main is squash-merged, grouped into the categories defined in .github/release.yml. Grouping is by PR label: github_actions / documentation are applied automatically by actions/labeler, but breaking-change, security, bug, and enhancement are applied by hand — an unlabelled PR lands in “Other changes”. Dependabot is split out by author rather than by label, because the labels actions/labeler applies by path — github_actions, documentation — mark our own PRs too; our CI work gets its own “CI & build” section — ordered above Documentation, since a CI PR here nearly always updates docs too — and Dependencies is pure Dependabot residue. Any category keyed on a label a Dependabot PR can carry needs that author exclude — labeler’s path labels and the ecosystem labels Dependabot applies itself (dependencies, javascript, go, github_actions; javascript is in neither labeler.yml nor our categories) — or that category intercepts bumps before they reach the 📦 Dependencies catch-all. CHANGELOG.md is not the source of the release body; it is the longer-form record of why each change was made.
• Server — a GHCR image at ghcr.io/wave-rf/wavehouse, with two tags: the immutable :vX.Y.Z, and one moving channel pointer. A stable release moves :latest; a prerelease moves :alpha / :beta / :rc / :next instead, matching the npm dist-tag it would get. The channel comes from the first prerelease identifier, matched exactly: v0.2.0-rc.1 → :rc, while -alpha1, -preview.1, or any other form → :next. scripts/ci/release-channel.sh is the single rule every publisher uses, so ghcr.io/wave-rf/wavehouse:rc and @wavehouse/sdk@rc can’t drift apart. A prerelease-only project therefore has no :latest tag — that is deliberate; :latest starts existing when the first stable release ships.
• TypeScript SDK — an npm publish of @wavehouse/sdk under latest (stable) or alpha/beta/rc/next (prerelease), plus its own GitHub Release.
• Server — build-provenance attestations (Sigstore, free for public repos) over every archive and over the image’s multi-arch manifest digest; the image attestation is stored alongside the image in GHCR. The release job verifies its own attestations before finishing, so a release that publishes unverifiable provenance goes red.
• TypeScript SDK — an npm provenance attestation via npm publish --provenance, surfaced as the provenance badge on the package page and checkable with npm audit signatures. The npm job does not re-verify it the way the server job does.

Binaries installed with go install github.com/Wave-RF/WaveHouse/cmd/wavehouse@vX.Y.Z get no -ldflags, so they read their version from the module metadata the Go toolchain embeds; /version reports the tag without its leading v (0.1.0, not v0.1.0), with git_commit/build_time as unknown. Release archives and container images carry all three.

Consumers can check provenance for themselves:

gh attestation verify wavehouse_linux_amd64.tar.gz \
  --repo Wave-RF/WaveHouse \
  --signer-workflow Wave-RF/WaveHouse/.github/workflows/release.yml

gh attestation verify oci://ghcr.io/wave-rf/wavehouse:v0.1.0 \
  --repo Wave-RF/WaveHouse \
  --signer-workflow Wave-RF/WaveHouse/.github/workflows/release.yml

--signer-workflow is not optional garnish: --repo alone accepts an attestation produced by any workflow in the repo. Same point, and the :dev equivalent, in Deployment → Registry and SECURITY.md.

Releasing from the GitHub UI instead

Publishing a release from Releases → Draft a new release creates the tag, which fires the same workflow — so it works, and GoReleaser’s default mode: keep-existing (the key is not set in .goreleaser.yaml) means it will not overwrite notes you wrote. Two things the make targets do for you and the UI does not: none of the preflight checks run, and you must click Generate release notes yourself, because a body you publish empty stays empty.

The dev channel

Between releases, every push to main republishes both artifacts so @dev always means “current main”:

• ghcr.io/wave-rf/wavehouse:dev — a rolling pointer, plus an immutable :dev-<sha> (pruned after 30 days by cleanup-ghcr.yml, newest 5 always kept). Built by the same GoReleaser pipeline with WAVEHOUSE_DEV=1, which suppresses the GitHub Release. Note a Docker tag is only a pointer: docker run …:dev reuses a stale local image unless you docker pull first or pass --pull=always.

• @wavehouse/sdk@dev — 0.0.1-dev.<utc-stamp>.h<build-hash>, published only when the published package actually changes. The trailing hash covers every file npm pack would ship — the built dist/, package.json minus its version, and the bundled README/LICENSE — so a push whose package would be byte-identical to the current dev publish is skipped, while a change to exports, files, bin, or engines republishes even though dist/ is untouched.

  The <utc-stamp> is load-bearing, not decoration. npm install …@dev records a range in your package.json, not the dist-tag, so what you get on the next install is the highest version matching that range. Under the old 0.0.0-dev.h<hash> scheme, semver’s lexical ordering of alphanumeric prerelease identifiers meant the newest publish routinely wasn’t the highest one, and a range could resolve backwards — an @dev install landed on a two-month-old build (#475). A numeric identifier compares numerically, so the channel now orders by publish time. The 0.0.1 base keeps the channel below every real release (so a dev build can never satisfy ^0.1.0) and above the legacy 0.0.0-dev.* publishes, which npm’s 72-hour unpublish window makes permanent.

Docker needs no equivalent fix: a tag is a mutable pointer resolved fresh on every pull, with no version ranges and no ordering, so the failure mode #475 describes cannot arise there.

CI & review automation

AI automation sits alongside the normal CI checks — advisory PR review from CodeRabbit and Copilot, plus the local Claude Code tooling in .claude/. Full detail lives in AGENTS.md §Repository Automation; this section covers the contributor-facing behavior.

PR title and Conventional Commits

PR titles must match Conventional Commits format and stay ≤ 72 characters — the title becomes the squash-merge commit subject. Both rules are enforced by the PR title job under the required CI check (.github/workflows/ci.yml); validate locally with scripts/lint-pr-title.sh "<title>":

<type>(optional-scope)(optional-!): <lowercase subject, no trailing period>

Allowed types: feat, fix, docs, refactor, test, chore, ci, deps, build, perf, revert, style.

The ! before : marks a breaking change per Conventional Commits 1.0.0 (e.g., feat!: remove deprecated endpoint, refactor(api)!: rename handlers). Titles are also capped at 72 characters — they become squash-merge commit subjects (Dependabot PRs are exempt from the cap).

If the title doesn’t match, a sticky comment posts on the PR explaining the format (from the PR housekeeping workflow, which mirrors the same script); it auto-removes once the title is fixed. Fixing the title needs no new push — the edit triggers housekeeping, which re-runs the failed PR title job (the job re-reads the title from the API, not the stale event payload).

Required status checks

The main branch protection ruleset requires one status check to pass before any PR can merge:

• CI — the aggregator job of .github/workflows/ci.yml. The workflow is a job DAG over the same Makefile targets local make ci runs: lint (make verify), unit (make test-unit test-ts), integration (one job per part of make test-integration: make test-integration-app, make test-integration-backends), e2e (make -j test-e2e — builds its own SDK dist + cover binary on a warm cache, runs the suite exactly like a local run), coverage (make cov over every suite’s uploaded coverage fragment + threshold gates, like local make ci’s final step), docs-build (make build-docs when docs-affecting files changed, uploading the docs dist artifact), PR title (Conventional Commits), and the docs preview/deploy jobs. The aggregator fails if any job failed or was canceled and treats skipped jobs as passing — docs-only PRs skip the Go test suites by design, and fork PRs run everything except the (secret-bearing) docs deploys. Every run’s Summary page gets a per-job wall-clock table from the non-gating Timing summary job. The full architecture — DAG diagram, design invariants, cache policy, how to add a job — lives in .github/workflows/README.md.

The PR housekeeping workflow still runs on every PR (labels + the title explainer comment) but is no longer a required check. A separate Cache cleanup workflow deletes a closed PR’s Actions caches, after each PR run deletes the Go build cache entries no run will read, and after each main run deletes main’s superseded cache generations.

The ruleset also requires an approval from the @Wave-RF/wavehouse-admins team (the required_reviewers rule — this is what mandates an admin sign-off, replacing the old Admin approval status-check workflow), plus 1 approving review, approval of the most recent push by someone other than its author, resolution of all review threads, linear history, no branch deletion, no force-push, and squash-merge only. Repository admins may bypass these requirements when merging their own PR (e.g. a trivial .github change) but still cannot push directly to main.

Approved, green PRs land through a merge queue (“Merge when ready”): the queue re-runs the required CI check against the PR merged with current main (a merge_group event — the CI workflow runs the full test suite for these) and fast-forwards only on green. That integration re-test replaces the old “branch is out-of-date with the base branch” requirement — queued PRs don’t need manual branch updates, and the queue never pushes to the PR branch.

Dependabot PRs go through the same admin review as any other PR — there is no auto-merge (see the Dependabot section above).

Merge behavior

Squash-only merges. The PR title becomes the commit subject (with (#NN) appended automatically), the PR body becomes the commit message. Keep PR bodies tight — they land in git log on main. The PR template gives the right shape (Summary / Test plan / Related Issues).

Include Closes #NN in the PR body to auto-close the related issue on merge. Alternatively, link the issue in the sidebar’s Development section — that triggers auto-close even without the keyword.

Auto-merge is enabled repo-wide: click “Enable auto-merge (squash)” on a PR and it merges once checks + approvals land.

AI reviewers

Advisory PR review comes from marketplace apps configured at the org/repo level:

• CodeRabbit — automated PR review; auto-reviews on open + push, re-trigger with @coderabbitai review.
• Copilot — tied to individual reviewer subscriptions; shows up on PRs where a maintainer with Copilot Pro is listed as a reviewer.

Both are advisory — the ruleset’s required_reviewers rule (an @Wave-RF/wavehouse-admins approval) plus its thread-resolution / linear-history / required-check rules are the actual merge-gate.

Reviewer assignment and the Task Board

Reviewer assignment is GitHub-native, and so is the board:

• Reviewer assignment: the main branch protection ruleset’s required_reviewers rule requests the @Wave-RF/wavehouse-admins team on every PR, and the team’s code-review assignment (configured on the team) auto-assigns and load-balances a specific member. No workflow is involved; dismiss_stale_reviews_on_push clears approvals on new commits, and GitHub re-requests per its own rules.
• Merge gate: the ruleset’s required_reviewers rule requires an APPROVED review from the @Wave-RF/wavehouse-admins team; it also adds review-thread resolution, linear history, and squash-only. Auto-merge (squash) takes over once checks and approvals land.
• Task Board (Projects v2, project #7): card placement and status are handled by GitHub-native Projects v2 automation configured in the project UI — there is no workflow-driven board state machine. Priority lives on the board’s Priority field (set during issue triage, below).

Dependabot PRs go through the same admin review as any other PR — there is no auto-merge (see the Dependabot section above).

Invoking bots manually

• CodeRabbit: comment @coderabbitai review to re-trigger a review, or @coderabbitai <question> to ask it something. Works in both top-level and inline review comments.
• Copilot: the re-request-review button on the PR page sends a fresh request.

Review-response expectations

Every review comment (human or AI) must get a substantive reply before merge — not “fixed” alone. The ruleset’s required_review_thread_resolution: true means unresolved conversations literally block merge. Agents working on PRs follow the pattern documented in AGENTS.md §“Review Response”: accept / push back / defer, reply with detail, resolve when settled.

When pushing back on a bot’s suggestion, end the reply with the bot’s mention (e.g. @coderabbitai) to invite a counter-reply so the dialog actually loops.

Issue triage

Issue triage is manual. There is no triage workflow: .github/workflows/triage.yml classified new and edited issues with GitHub Models, which GitHub retired on 2026-07-30 — the inference endpoint returns 410 for every request, so the workflow failed on every issue open/edit until it was removed (#431).

What that means in practice:

• area/*, security, and breaking-change labels are applied by hand when an issue is triaged. PR labels are unaffected — those come from actions/labeler (below), which is path-based and needs no model.
• Priority on the Task Board project #7 is set by hand on the board’s Priority field. .github/board-config.env existed only for that workflow and is gone with it. The PROJECT_BOARD_TOKEN repo secret is now unused — it has no consumers left, so it is worth deleting and revoking the PAT behind it.
• Maintainers running Claude Code can use the /pm-triage skill, which does the same classification locally against the live board.

Auto-labeling PRs

The PR housekeeping workflow (.github/workflows/housekeeping.yml) runs actions/labeler with .github/labeler.yml to apply area/*, dependencies, github_actions, go, and documentation labels to PRs based on the files they change. Sync-mode: labels follow the current changed-file set.

When adding a new internal/<pkg>/ package

Follow the checklist in AGENTS.md §“Common Tasks / Adding a new internal package” — the automation-relevant steps are:

1. Create a matching area/<pkg> repo label with a meaningful description.
2. Add the path → label mapping to .github/labeler.yml so PRs touching the new package get auto-labeled.

Step 2 is the one that automates anything — PR labeling is path-based. Step 1’s label is applied by hand during issue triage.