Configuration

WaveHouse is configured via a YAML file with environment variable overrides. All environment variables use the WH_ prefix.

Loading Order

1. If a config file exists at the specified path (default: config.yaml), it is loaded first.
2. Environment variables override any values from the YAML file. A key the file sets always wins over its default, including an explicit false, 0 or "": otel.traces.enabled: false turns traces off, and otel.traces.sample_rate: 0 exports no traces. Only a key the file leaves out takes the default listed below.
3. If no config file exists, all values are read from environment variables. Every key has a default except settings.dir (WH_SETTINGS_DIR), which must be set either way.
4. Both sources are strict. A YAML key this page doesn’t list — a typo, or a tunable that has moved to the settings directory (dlq.enabled, clickhouse.addr, stream.*, ingest.*, a leftover policy: or pipes: block, …) — refuses to boot and names every offending key, so nothing is read, ignored, and believed. A WH_* environment variable that binds to no key on this page (WH_DEDUPE_ENABLED, WH_CH_ADDR, a misspelling) refuses to boot the same way. Two variables have no YAML key and are exempt because they are not config keys at all but process-level settings main reads directly: WH_CONFIG (below), which locates the file, and WH_LOG_LEVEL. Only the WH_ prefix is checked, since the environment always carries names that aren’t WaveHouse’s. One outside source does share the prefix. Kubernetes® injects {SERVICE}_SERVICE_HOST, {SERVICE}_PORT, and similar link variables into every pod in a Service’s own namespace, for each Service with a cluster IP that existed before the pod started (a headless Service injects nothing, and a Service in another namespace is harmless). The name is uppercased with - mapped to _, so a Service named wh produces WH_SERVICE_HOST and WH_PORT, one named wh-foo produces WH_FOO_SERVICE_HOST and WH_FOO_PORT, and either way the pod refuses to boot on its next restart. Set enableServiceLinks: false on the pod spec, or name the Service something else. The error says so.
5. Before anything dials out, data_dir is probed — when a selected backend keeps state there, as the in-process mq backend does, and the in-process dedupe backend does in a process running the api role — and boot refuses on any of these: the value is empty; the path exists but is not a directory; the path, or any component above it, is a dangling symlink (a mount that never came up); the directory exists but the process cannot write to it; the directory is absent and its nearest existing ancestor is not writable, so it could not be created. The probe runs before ClickHouse® discovery, so the refusal lands at the top of the log, and a permission denial — on the write probe, or on reaching the path at all through a parent without search permission — carries the UID-65532 remediation, since a bind mount owned by root is the typical cause.

Boot is the validator for this half of configuration: there is no dry run, and a refused boot with the offending key, variable, or path named in the error is the loud signal. The hot-reloadable half has a dry run — wavehouse validate — because it is edited under a running server; boot config only ever takes effect through a restart, so the restart is where it is checked.

Set WH_CONFIG to change the config file path:

export WH_CONFIG=/etc/wavehouse/config.yaml

This page is boot config only — what the platform operator owns (wiring, lifecycle, secrets), read once at startup. Tenant-owned behavioral tunables, the access-control policy, and the named pipes are not here: they live in the hot-reloadable Settings Directory.

Full Reference

State

YAML Key	Env Var	Default	Description
data_dir	WH_DATA_DIR	./data	Root directory for embedded state. NATS™ JetStream lives at <data_dir>/nats; Pebble (with dedupe.backend: pebble), holding every tenant’s dedupe store while any tenant has dedupe enabled, at <data_dir>/pebble. Subdirectory names are conventions, not config — one knob, one mount. In a container this MUST resolve to a host-backed volume; the relative default is for local binary use. WaveHouse logs a startup WARN when the directory is missing or empty (no prior state). See Persistent Storage.

Backends

Each layer’s implementation is chosen once, at boot. Every layer’s default is its in-process backend, so a config that sets none of these keys runs as it always has. The message queue and the coordination layer also have a shared backend, nats; so do the cache, redis, and dedupe, dynamodb. A value this build has no backend for refuses boot and names the valid ones.

YAML Key	Env Var	Default	Description
mq.backend	WH_MQ_BACKEND	embedded	The message queue. embedded: NATS JetStream inside this process, under <data_dir>/nats. It listens on no port, so no other process can reach its queue. nats: a NATS JetStream cluster you run, shared by every WaveHouse process that names it, configured by mq.nats; nothing is kept under data_dir/nats.
cache.backend	WH_CACHE_BACKEND	local	The query-result cache. local: in this process, sized by cache.l1_max_cost. redis: one Redis®-compatible server shared by every process, configured by cache.redis, so an insert one process makes invalidates what every process has cached.
dedupe.backend	WH_DEDUPE_BACKEND	pebble	Where ingest dedupe keeps the event ids it has seen. pebble: in this process, under <data_dir>/pebble, open while any tenant has dedupe on; two processes do not share seen ids. dynamodb: one DynamoDB table that every tenant and every process shares, configured by dedupe.dynamodb.
coord.backend	WH_COORD_BACKEND	local	Where leases are held: the sweeper’s on mq.backend=embedded, and the ingest processes’ on mq.backend=nats. local: in this process, so the one process always holds them. It shares nothing with another process, so it serves one process on mq.backend=embedded, or a process without the ingest role. nats: a KV bucket you create on the mq.nats cluster, reached over the same connection and credentials, so every process sees the same leases; configured by coord.nats. It needs mq.backend=nats.

Settings for one backend go in a sub-block named after it, <layer>.<backend>, read only when that backend is selected. mq.nats, coord.nats, cache.redis and dedupe.dynamodb are the only ones so far; any other, mq.embedded included, is an unknown key and refuses boot. A sub-block written while its layer runs another backend is not read. Boot warns about an ignored mq.nats, coord.nats or cache.redis.addrs; a dedupe.dynamodb block under pebble is ignored silently. mq and dedupe also appear in the settings directory’s config.json, with different keys (mq.max_bytes_gb, dedupe.enabled, …): those are per-tenant tunables and stay there, and one written in config.yaml refuses boot as an unknown key.

External NATS (mq.nats)

Read only with mq.backend: nats. WaveHouse connects to NATS you run and uses streams and durable consumers you create: it never creates, changes, purges or deletes one. Deployment → External NATS covers the cluster, the streams, the user’s permissions and the gauges. Secrets are file paths only, such as a mounted Kubernetes Secret; no key takes a secret inline, and mq.nats.password or WH_MQ_NATS_PASSWORD refuses boot as an unknown key.

YAML Key	Env Var	Default	Description
mq.nats.urls	WH_MQ_NATS_URLS	(none)	Required. The servers to dial: a YAML list, or a comma-separated variable. A URL carrying credentials (user:password@ or token@) refuses boot.
mq.nats.name	WH_MQ_NATS_NAME	wavehouse-<hostname>	The connection name the server reports.
mq.nats.creds_file	WH_MQ_NATS_CREDS_FILE	(empty)	A .creds file (user JWT and nkey seed), for decentralized auth.
mq.nats.nkey_seed_file	WH_MQ_NATS_NKEY_SEED_FILE	(empty)	An nkey seed file.
mq.nats.user	WH_MQ_NATS_USER	(empty)	A user name, with password_file.
mq.nats.password_file	WH_MQ_NATS_PASSWORD_FILE	(empty)	The file holding user’s password; a trailing newline is dropped. Refused without user.
mq.nats.tls.ca_file	WH_MQ_NATS_TLS_CA_FILE	(empty)	CA bundle for the servers’ certificates.
mq.nats.tls.cert_file	WH_MQ_NATS_TLS_CERT_FILE	(empty)	A client certificate, for mutual TLS. It comes as a pair with key_file.
mq.nats.tls.key_file	WH_MQ_NATS_TLS_KEY_FILE	(empty)	The client certificate’s private key. It comes as a pair with cert_file.
mq.nats.tls.server_name	WH_MQ_NATS_TLS_SERVER_NAME	(empty)	The name to verify the servers’ certificates against, when it is not the host dialed.
mq.nats.tls.handshake_first	WH_MQ_NATS_TLS_HANDSHAKE_FIRST	false	Start TLS before the NATS protocol, for servers that require it.
mq.nats.js_domain	WH_MQ_NATS_JS_DOMAIN	(empty)	The JetStream domain, for a leafnode or hub-and-spoke deployment. WaveHouse then sends its JetStream API requests and lease writes under $JS.<domain>.API.; generate the user’s permissions with wavehouse mq permissions --js-domain <domain> (Permissions).
mq.nats.subject_prefix	WH_MQ_NATS_SUBJECT_PREFIX	wh	Leads every subject WaveHouse uses (<prefix>.ingest.…, <prefix>.hist.…, <prefix>.dlq.…). One token of [a-z0-9_-].
mq.nats.partitions	WH_MQ_NATS_PARTITIONS	1	How many ingest partition streams there are. It must equal the number you created; boot finds each one by its subject.
mq.nats.shards	WH_MQ_NATS_SHARDS	32	How many shards each partition has: a table’s events all go to one shard, by a consistent hash of the tenant and table, and each shard has a durable of its own that one ingest process at a time consumes. It must equal the number you created (wavehouse mq manifests --shards); from 1 to 256. It caps the ingest processes that get work at partitions × shards, and at 64.
mq.nats.ingest_consumer	WH_MQ_NATS_INGEST_CONSUMER	wh-ingest	Names the shard durables: shard s of every partition is <ingest_consumer>-<s>, unpadded. Generate them with wavehouse mq manifests --ingest-consumer.
mq.nats.history_stream	WH_MQ_NATS_HISTORY_STREAM	<PREFIX>_HISTORY	The history stream, which SSE replay and the live hub read. WaveHouse looks it up by this name; the partition and dead-letter streams are found by subject and can have any name. The default is the upper-cased prefix, WH_HISTORY for wh, as wavehouse mq manifests names it; --history-stream names another.
mq.nats.connect_timeout	WH_MQ_NATS_CONNECT_TIMEOUT	5s	Bounds one dial.
mq.nats.publish_timeout	WH_MQ_NATS_PUBLISH_TIMEOUT	5s	Bounds one publish attempt. A publish is tried at most three times; a partition’s duplicate_window must cover all three, or boot refuses. wavehouse mq manifests --publish-timeout sizes the window for it.
mq.nats.topology_wait	WH_MQ_NATS_TOPOLOGY_WAIT	60s	How long boot waits for the cluster and for your streams and consumers to be right. Boot then refuses with every finding at once.

Boot refuses a nats block with no URLs, a URL with credentials in it, more than one of creds_file, nkey_seed_file and user, half a certificate pair, a prefix outside the grammar, fewer than one partition, a shard count outside 1 to 256, an ingest_consumer or history_stream outside [a-zA-Z0-9_-], or a timeout that is not positive. Durations take Go syntax (5s, 2m).

A tenant’s mq.max_bytes_gb is not applied under nats: its events share a partition stream with other tenants, and that stream’s limits, which you set, bound them. Boot logs a warning saying so.

NATS leases (coord.nats)

Read only with coord.backend: nats. It has no connection settings: the leases ride the mq.nats connection, which is why coord.backend=nats refuses boot without mq.backend=nats. WaveHouse never creates the bucket. Boot waits for it with the rest of your topology (mq.nats.topology_wait) and then refuses, naming it; Deployment → External NATS shows how to create it.

YAML Key	Env Var	Default	Description
coord.nats.bucket	WH_COORD_NATS_BUCKET	<prefix>_coord	The KV bucket the leases live in, one key per lease. The default follows mq.nats.subject_prefix (wh_coord for wh), as wavehouse mq manifests names it, so two deployments sharing one NATS account under different prefixes never contend for one lease. To use another name, generate the bucket with wavehouse mq manifests --coord-bucket <name> and change the wavehouse user’s permissions in values.yaml to match. A name outside [a-zA-Z0-9_-] refuses boot.

A lease is taken over only after its holder has stopped renewing it: a process that wants it must see the same revision unchanged for 15 seconds on its own clock, so no two servers’ clocks are compared. The holder renews every 2 seconds and steps down after 10 seconds without a successful renewal, before anyone can take over. A process that stops cleanly deletes its lease, so the next holder takes over at its next attempt (within 2 seconds). The bucket must not expire keys (ttl unset), because a lease that expires on the server’s clock can end under a live holder.

Dedupe

Whether a tenant dedupes, and on which field, are settings-directory keys (Deduplication). What is boot config is where the seen ids live and how a claim behaves.

YAML Key	Env Var	Default	Description
dedupe.lease	WH_DEDUPE_LEASE	30s	How long a record’s id stays claimed while the record is published. Another request carrying the same id meanwhile gets 503 with this as Retry-After, in whole seconds; a claim that is neither committed nor released, because its process died mid-publish, lapses after it. With mq.backend: embedded, the lease plus its own ceiling to the next whole second plus one more second must fit the embedded queue’s 2-minute duplicate window, so the lease is at most 59s: a client that obeys Retry-After after a publish whose outcome it never learned can republish as late as the lease plus that ceiling plus a second after the claim, since DynamoDB rounds a claim’s expiry up to the second. A longer lease refuses boot. Under mq.backend: nats there is no cap here; instead every partition’s duplicate_window must cover the same span, or boot refuses with a topology finding (Deployment). A Go duration (30s, 45s); 0 refuses boot.
dedupe.reserve_concurrency	WH_DEDUPE_RESERVE_CONCURRENCY	64	The most parallel calls one Reserve, Commit or Release makes to a remote dedupe backend, and the idle connections per host the DynamoDB client keeps to match, never fewer than the SDK’s own default (10). Ingest reserves and commits a window of up to 256 ids per call; pebble ignores it. 0 refuses boot.

DynamoDB dedupe

Read only when dedupe.backend is dynamodb. Credentials come from the AWS™ SDK’s default chain (EKS Pod Identity or IRSA in a pod; AWS_* variables or a profile locally), never from this file; the table and its IAM policy are described in Deployment. At boot WaveHouse checks the table: its key schema must be pk and sk (both String: partition and sort key) or pk (String) alone, the only layout 0.2.0 accepts, and the backend writes to the table in whichever layout it finds; TTL off on ex is logged as a warning. A misconfigured table — missing, with the wrong key schema, or denied to the process’s credentials — refuses boot only with a flat settings directory whose tenant has dedupe on, and is logged at ERROR otherwise. In every other case — a transient failure (a throttle, a timeout, the network), a nested directory, or no tenant with dedupe on yet — the process boots, every tenant with dedupe on answers ingest 503 (dedupe store unavailable, Retry-After: 5) until the check passes, and the check is retried in the background, backing off from one second to thirty, and at once after every reload, so a table that comes good is picked up without a restart. A reload makes no table call, and does not wait on a tenant whose dedupe setting is unchanged: it applies each tenant’s switch against the last check’s result, so a tenant it switches on fails closed until the retry passes. A reload waits only for a tenant whose store it closes — dedupe switched off, or the tenant removed or rejected — and then only for that tenant’s in-flight Reserve/Commit/Release calls, before the store itself closes. The check runs in every process running the api role, the one that opens the dedupe stores, whether or not any tenant has dedupe on.

YAML Key	Env Var	Default	Description
dedupe.dynamodb.table	WH_DEDUPE_DYNAMODB_TABLE	(required)	The shared table.
dedupe.dynamodb.region	WH_DEDUPE_DYNAMODB_REGION	(empty)	The table’s region. Empty uses the SDK chain’s (AWS_REGION); no region from either refuses boot.
dedupe.dynamodb.endpoint	WH_DEDUPE_DYNAMODB_ENDPOINT	(empty)	A custom endpoint, for dynamodb-local in development and tests. Leave it empty against AWS.
dedupe.dynamodb.timeout	WH_DEDUPE_DYNAMODB_TIMEOUT	250ms	Deadline for each DynamoDB call, the SDK’s retries included. The retries back off with full jitter, each wait capped at timeout / (2 × (max_attempts − 1)), so together they wait at most half of it and a throttled call fails on its last attempt’s answer rather than on the deadline. The boot and background table check (verifying the key schema and TTL) is not one of these calls: it runs under its own deadline of 10 × timeout (2.5s by default). 0 refuses boot.
dedupe.dynamodb.max_attempts	WH_DEDUPE_DYNAMODB_MAX_ATTEMPTS	3	Attempts per call, the first included. More attempts share the same half of timeout for their waits, so each retry waits less rather than the call running longer. 0 refuses boot.
dedupe.dynamodb.retry_mode	WH_DEDUPE_DYNAMODB_RETRY_MODE	standard	standard, or adaptive, which also slows the client down after throttling. Anything else, empty included, refuses boot.
dedupe.dynamodb.create_table	WH_DEDUPE_DYNAMODB_CREATE_TABLE	false	Development only: create the table at boot if it is missing, keyed by pk and sk, with TTL on ex; an existing table is left as it is, whichever layout it has. Refused unless endpoint is set, so it never creates a table in AWS; the production table belongs to your infrastructure code.

Boot warnings

Boot logs each of these at WARN rather than refusing. The first two are right for a single replica only, which one process cannot tell from many; the last depends on a window only your streams know:

• mq.backend=nats with cache.backend=local, in a process running api: an event ingested on another replica never invalidates this one’s cache, so its reads stay stale until the cached entry expires.
• mq.backend=nats with dedupe.backend=pebble, in a process running api: an id seen by another replica is not seen by this one.
• mq.backend=nats: mq.max_bytes_gb is not applied (above).
• mq.nats set with mq.backend=embedded, or coord.nats set with coord.backend=local: the block is ignored.
• cache.redis.addrs set with cache.backend=local, in a process running api: the block is ignored.
• cache.redis.tls.insecure_skip_verify on, in a process running api: the cache accepts any certificate (see Cache).
• server.trusted_proxies with a /0 entry (0.0.0.0/0, ::/0), in every process, one per entry: every peer of that entry’s address family is trusted (0.0.0.0/0 every IPv4 peer, ::/0 every IPv6 one), so a client of that family that reaches server.port directly picks its own client.address and joins its traceparent to your traces (see Reverse proxy).
• profiling.enabled with a profiling.addr that is not loopback (an empty host, 0.0.0.0, a pod address), in every process: the unauthenticated pprof endpoints are reachable beyond the host (see Profiling).
• mq.backend=nats and a tenant with dedupe on whose finite dedupe.retention is under the partitions’ duplicate_window, in a process running api, at boot and after every reload: see Deployment → Create the topology.

Process roles

By default one process does all the work. roles splits it, so that the API and the background workers can run in separate processes, for example one Kubernetes Deployment per role (see Deployment). The binary and its entry point are the same for every role; only this key differs.

YAML Key	Env Var	Default	Description
roles	WH_ROLES	api,ingest,sweeper	The roles this process runs: a YAML list, or a comma-separated variable. Order does not matter. An empty list, an empty entry, an unknown role, or a role named twice refuses boot.
instance_id	WH_INSTANCE_ID	(empty)	Names this process: every log line after the boot config loads carries it as instance_id, its spans, metric series and OTLP log records carry it as the service.instance.id resource attribute (see OTel), and under coord.backend=nats it is the holder a lease’s value names, so you can see which process holds a lease. Empty resolves at boot to <hostname>-<8 hex>, with a fresh random suffix every time, so a restarted process is a new instance.

Role	Runs
api	The HTTP API, and what answers it: schema discovery, the token verifiers and their JWKS refresh, the dedupe stores, and the SSE hub with its bridge off the queue and its keepalive wheels. Every API process runs its own set of these, and each API process receives every event for its own SSE clients.
ingest	The ingest worker, which writes the queue to ClickHouse. Under mq.backend=nats the ingest processes share the queue’s shards out through membership leases in coord.nats.bucket, one process per shard at a time, so a table’s rows are written by one process.
sweeper	The sweeper, which purges messages that are written and older than their tenant’s gap window. It runs under the sweeper lease, on mq.backend=embedded only. Under mq.backend=nats your streams’ retention does its work, so it is not wired there, and a process whose only role is sweeper is refused (below). A tenant whose gap window is longer than the history stream keeps is checked by the API processes instead, at boot and after each reload, with one warning per tenant and window.

Every process, whatever its roles, reads the settings directory and reloads it (SIGHUP, the directory watcher, and the reload route), and serves server.port. A process without the api role serves only an ops listener there: /livez, /readyz and their /healthz, /health, /ready aliases, /version, the metrics path when prometheus.port is 0, and POST /v1/ops/settings/reload. Every other route answers 404; under /v1/ops, only once the operator-key check has passed (403 without a credential; a bearer token is refused with 401, since no token verifier runs without the api role). The reload route on that listener accepts only the operator key, because no token verifier runs without the api role. /readyz checks the NATS connection and topology (mq) in every such process — a process without the api role always runs mq.backend: nats — and ClickHouse (clickhouse) in one running ingest.

Boot refuses a role set the selected backends cannot serve:

• Any split with mq.backend=embedded. The embedded queue lives inside its process and listens on no port, so a process without every role could not reach it. Choose mq.backend=nats to split.
• ingest with mq.backend=nats and coord.backend=local. The ingest processes share the queue through leases, so each needs the shared coordinator. Set coord.backend=nats. A process without the ingest role holds no lease, so it may keep local.
• Only sweeper with mq.backend=nats. The sweeper is not wired there, so the process would run nothing. Remove it.
• coord.backend=nats without mq.backend=nats. The leases ride the NATS connection.
• api without ingest, or ingest without api, with cache.backend=local. The ingest worker invalidates the cache the API reads, and a local cache in another process never sees that invalidation. Run api and ingest together, or set cache.backend: redis, one cache every process shares.

Server

YAML Key	Env Var	Default	Description
server.port	WH_SERVER_PORT	8080	HTTP server listen port.
server.shutdown_timeout	WH_SERVER_SHUTDOWN_TIMEOUT	10	Seconds the drain phase of a stop may take: in-flight requests and ingest batches finish (open SSE streams end immediately). The release of the stores (5s) and the telemetry flush (3s) have fixed budgets on top, so a stop takes at most this plus 8s; see Stopping.
server.trusted_proxies	WH_SERVER_TRUSTED_PROXIES	(empty)	The reverse proxies whose X-Forwarded-For names the client and whose traceparent a request’s trace joins: CIDRs or bare addresses, a YAML list or a comma-separated variable. Empty trusts no peer, so the client is the connection’s own address and an incoming trace is linked, not joined. An entry that is neither, or an IPv4-mapped prefix (::ffff:…) shorter than /96, refuses boot; a mapped prefix of /96 or longer is read as the IPv4 CIDR it maps. See Reverse proxy → Client address and trace context.

The server speaks plain HTTP — there is no inbound-TLS setting. Terminate TLS at a reverse proxy for internet-facing deployments. (The settings directory’s clickhouse.http_scheme and clickhouse.tls are the outbound WaveHouse → ClickHouse hops, unrelated to your clients’ TLS.)

ClickHouse

Only the connection ceiling is boot config. The wiring — native address, HTTP port and scheme, database, username, password, query and insert timeouts, TLS, headers, pool sizes — is the settings directory’s clickhouse block, where a change re-dials without a restart; the password lives there with the user it pairs with, so rotating it is a reload.

YAML Key	Env Var	Default	Description
clickhouse.max_total_conns	WH_CH_MAX_TOTAL_CONNS	0	Ceiling on the native ClickHouse connections the process holds open: the max_open_conns of the open pools — one per distinct connection tuple among the served tenants, see the settings directory — must not add up to more. On a reload a pool resized above it is refused and keeps its size, and a new pool that would cross it is not opened — its tenants keep the pool they had, or have none — both logged (the reload itself still reports adopted) and retried by the next reload. At boot it depends on the settings directory’s shape: over a flat directory pools above it refuse to start, naming the sum and the ceiling; over a nested directory the tenants are placed in id order, and each whose pool would cross it at its turn is served on no pool, logged and counted, while the rest boot on their pools — a refused value stays in the folder, so fix it before the next restart, which would refuse to start, or serve on none every tenant that no longer fits in id order, other folders’ included. 0 is no ceiling. Capacity is sized once per process, which is why it is boot config rather than a settings key.

Server-side resource limits

WaveHouse enforces a role’s per-role resource caps (from the access-control policy) by attaching them to each query as ClickHouse settings. Server-wide limits — the backstop that applies to every query regardless of role, including raw admin SQL — are configured in ClickHouse itself, via its settings profiles and quotas. This keeps one authoritative place for global governance, has ClickHouse enforce it natively (defense-in-depth, even against a WaveHouse bug), and lets you use standard ClickHouse operations.

Set the backstop on the profile of the ClickHouse user WaveHouse connects as (the settings directory’s clickhouse.username). For example, in users.xml:

<clickhouse>
  <profiles>
    <default>
      <!-- Per-query ceilings: a query that exceeds these is rejected. -->
      <max_memory_usage>4000000000</max_memory_usage>     <!-- ~4 GiB -->
      <max_execution_time>30</max_execution_time>         <!-- seconds -->
      <max_rows_to_read>1000000000</max_rows_to_read>

      <!-- Optionally make the memory ceiling a hard cap WaveHouse's per-role
           settings cannot raise above (but may still tighten). -->
      <constraints>
        <max_memory_usage><max>8000000000</max></max_memory_usage>
      </constraints>
    </default>
  </profiles>

  <!-- Quotas add time-windowed limits (queries/sec, rows read per hour, …)
       that per-query settings can't express. -->
  <quotas>
    <default>
      <interval>
        <duration>3600</duration>
        <read_rows>10000000000</read_rows>
      </interval>
    </default>
  </quotas>
</clickhouse>

What a caller sees when one of these trips: a row or byte limit is 400 clickhouse.limit_exceeded; a server-wide time, memory or quota limit is 503 clickhouse.unavailable (retryable, so the SDK retries it — except on a pipe that writes, which is never retryable), since WaveHouse cannot tell it from server pressure — except on /v1/query for a role that sets its own max_memory_usage, where every memory-limit error is read as that cap and answered 400 clickhouse.limit_exceeded. See ClickHouse errors on the query paths.

How the two layers compose

WaveHouse’s per-role caps are sent as per-query SETTINGS on its connection, so they compose with the ClickHouse profile — a per-role cap tightens within the profile’s ceiling, and a <constraints> block bounds how far any setting can move. But if the profile marks a setting readonly (or <constraints> disallows changing it), ClickHouse will reject WaveHouse’s per-query override and the query fails. So keep the settings WaveHouse manages (max_memory_usage, max_execution_time, max_rows_to_read, max_result_rows) changeable for its user — use a <max> constraint, not readonly, if you want a hard ceiling.

Message Queue (NATS)

Three keys tune the ingest worker’s durable consumer of the queue, on either backend: under embedded they shape the consumer on each tenant’s queue; under nats they are what the operator’s shard durables must agree with, checked at boot like the shard count (Create the topology), and wavehouse mq manifests --ack-wait --max-ack-pending generates durables that do. They are sized once per process, which is why they are boot config; the batch triggers, which a tenant’s row width decides, are the settings directory’s.

YAML Key	Env Var	Default	Description
mq.ack_wait	WH_MQ_ACK_WAIT	60s	The redelivery timeout: a row the worker has not acked within it is delivered again. It must cover a table’s batch wait plus its insert: each tenant’s ingest.max_batch_wait_ms plus clickhouse.insert_timeout must stay under it, which the ingest processes check when a tenant’s settings are adopted (boot refuses over a flat directory, and over a nested one serves the tenant with none, its rows handed back to the queue; a reload keeps the tenant’s previous values, or hands its rows back when it has none), or a batch still inserting is redelivered and inserted twice. The rule bounds a batch’s own wait plus one INSERT; a batch queued behind an in-flight flush, several column lists or row-by-row isolation add deadlines on top, so keep the sum well under this, as the seeds do (35 s). Before lowering it, check every tenant’s sum: one at or above the new value refuses a flat directory’s boot and leaves a nested directory’s tenant with none. A Go duration (60s, 2m); 0 refuses boot. Under nats every shard durable’s ack_wait must be at least this, or boot refuses with a finding.
mq.max_ack_pending	WH_MQ_MAX_ACK_PENDING	100000	How many rows a delivery unit may have delivered and not yet acked before the queue pauses delivery from it (backpressure), and no other unit’s: a tenant’s queue under embedded; under nats a shard durable’s max_ack_pending, shared by the tenants in that shard, and the budget of unsettled rows an ingest process shares out over its shards (Scaling ingest processes). A tenant’s tables share it, so the batches of several busy tables can hold it together before any reaches ingest.max_batch_bytes, and flush on the timer: at the defaults past about 38 tables of 100-byte rows, or 19 of 50-byte rows, busy at once. The worker holds every delivered row until its batch is acked, so its memory grows with this, per unit, and the budget is a count of rows, not bytes: a unit whose ClickHouse has stalled holds this many rows at whatever width they came in, so wide rows during an incident cost far more memory than the same count of ordinary ones (#957). At least mq.prefetch. A durable with a lower value is a recommended finding at boot, not a refusal: the durable’s own cap is what the server enforces, so that shard holds fewer rows than this until it is regenerated (wavehouse mq manifests --max-ack-pending).
mq.prefetch	WH_MQ_PREFETCH	500	How many rows the worker’s consumer fetches ahead of its batches, split across the units it consumes (a tenant’s queue, or a shard; at least one each). At least 1. Twice this is also the floor under a shard’s share of mq.max_ack_pending (or mq.max_ack_pending itself, if that is smaller). Each tenant table the worker has seen keeps a channel of this many parsed rows (about 120 bytes each) for the life of the process, so raising it costs memory per table served. Under nats a shard durable’s max_request_batch must be unset (as generated) or at least this, or boot refuses.

The rest of this section describes the embedded backend. mq.backend: nats takes its other settings from mq.nats instead.

Each tenant’s queue has its own disk budget, mq.max_bytes_gb, a hot-reloadable key in the Settings Directory — there is no boot-config knob for it.

Durability. The embedded server runs with JetStream SyncAlways, so every event is fsync’d to disk before POST /v1/ingest returns 200. This makes your storage’s fsync latency your ingest latency floor — see Durability & Storage to check whether your substrate can sustain it. There is no knob to relax this today (#139 tracks a configurable group-commit interval).

Cache

YAML Key	Env Var	Default	Description
cache.l1_max_cost	WH_CACHE_L1_MAX_COST	67108864	Maximum size in bytes (~64 MB) of the local backend’s in-process cache. The time-range bucket structured queries normalize to is query.timestamp_bucket_seconds in the Settings Directory.

The redis backend’s settings, read only when cache.backend is redis. It is tested on Redis, Valkey, Dragonfly and a Redis Cluster node, and its data commands are only GET, SET and MGET (no scripts, no client tracking), which ElastiCache and MemoryDB also serve. An ACL user also needs the connection commands the client sends when it dials: HELLO, CLIENT, SELECT, and CLUSTER in cluster mode; without them it is refused (NOPERM) and the cache stays bypassed. Whoever can write to the server can replace cached query results, which are served after the access policy has already been applied, so treat the server as part of WaveHouse’s trust boundary (see Deployment). Deployment covers sizing, maxmemory-policy and what a reader on another instance can see.

YAML Key	Env Var	Default	Description
cache.redis.addrs	WH_CACHE_REDIS_ADDRS	(empty)	Required with backend: redis. host:port of the server: exactly one in standalone mode, where a second would be ignored and so refuses boot; several are a cluster’s seed nodes. Comma-separated in the env var. Not a URL: redis://user:password@host:port refuses boot, without repeating the value, so set the credentials through username and password, and rediss:// through tls.enabled.
cache.redis.mode	WH_CACHE_REDIS_MODE	standalone	standalone or cluster. sentinel refuses boot until #656: the cache does not yet authenticate to the sentinels or refresh their topology, so it could not be trusted to follow a failover.
cache.redis.username	WH_CACHE_REDIS_USERNAME	(empty)	ACL user. Empty uses the server’s default user.
cache.redis.password	WH_CACHE_REDIS_PASSWORD	(empty)	A secret: set it through the environment (or your secret store’s env injection), not in a tracked config.yaml.
cache.redis.db	WH_CACHE_REDIS_DB	0	Database number (SELECT). Standalone only: a cluster has only database 0, and any other value refuses boot.
cache.redis.tls.enabled	WH_CACHE_REDIS_TLS_ENABLED	false	Connect over TLS, verifying the server against the system roots or ca_file. Any other tls key set while this is off refuses boot, rather than connecting in plaintext.
cache.redis.tls.ca_file	WH_CACHE_REDIS_TLS_CA_FILE	(empty)	PEM file of the authorities to trust instead of the system roots.
cache.redis.tls.cert_file	WH_CACHE_REDIS_TLS_CERT_FILE	(empty)	Client certificate (PEM) for mutual TLS. Set together with key_file.
cache.redis.tls.key_file	WH_CACHE_REDIS_TLS_KEY_FILE	(empty)	The client certificate’s private key (PEM).
cache.redis.tls.server_name	WH_CACHE_REDIS_TLS_SERVER_NAME	(empty)	Name to verify the server’s certificate against, when it differs from the address.
cache.redis.tls.insecure_skip_verify	WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY	false	Accept any server certificate. Logged at WARN at boot: whoever can intercept the connection can read and replace cached results.
cache.redis.key_prefix	WH_CACHE_REDIS_KEY_PREFIX	wh	Leads every key, so several deployments can share one server, provided you trust each as much as the others: any of them can overwrite what the rest serve. No { or }.
cache.redis.timeout	WH_CACHE_REDIS_TIMEOUT	100ms	Per operation, at most 1s. A lookup or fill that takes longer is a miss or a skipped fill, never a failed query.
cache.redis.dial_timeout	WH_CACHE_REDIS_DIAL_TIMEOUT	1s	Per connection attempt, at most 1s. An attempt is a connect and a handshake, each bounded by this, and in cluster mode a topology read bounded by the larger of this and timeout. Boot and shutdown each wait out one in flight, so the two caps keep that under 3s, inside a stop’s fixed 5s release budget (see Stopping).
cache.redis.max_value_bytes	WH_CACHE_REDIS_MAX_VALUE_BYTES	1048576	Largest result stored, after compression (1 MiB). A larger one is returned to the caller but not cached.
cache.redis.compress_min_bytes	WH_CACHE_REDIS_COMPRESS_MIN_BYTES	1024	Results at least this large are zstd-compressed when that makes them smaller. 0 never compresses.
cache.redis.version_ttl	WH_CACHE_REDIS_VERSION_TTL	168h	How long a table’s or tenant’s version token outlives its last write, so the tokens of dropped tables and removed tenants eventually expire. At least 2s. An expired token only causes misses.

When the server is unreachable or misbehaves, the cache is bypassed; queries are not. A failure or a timeout makes the lookup a miss and the fill a no-op. Five in a row, one reply refusing writes (READONLY from a demoted primary, OOM when full under noeviction, and the like), or one refusing the credentials (WRONGPASS, NOAUTH — what an already-open connection meets once the password is rotated) open a circuit breaker that skips the server entirely until a probe write, every 5 s, succeeds within timeout. A closed breaker opening is logged once, at ERROR for the credentials and WARN otherwise; a failed probe opens it again and logs at DEBUG, unless it failed for another cause than the one last logged (rejected credentials after a restart, say), which is logged at its own level. A server that stays down is one line for the outage. NOPERM does not open it: it names one key or command an ACL user cannot use, not every operation, so it should not bypass the cache for every tenant. Queries then go straight to ClickHouse, still coalesced per instance by singleflight. An invalidation the server did not take is kept and retried until it lands, and until then the instance that owes it bypasses the lookups it would orphan. /readyz does not depend on the cache.

Boot does not wait for the server. A malformed block (an address without a port, mode: cluster with db other than 0, an unreadable or unparsable TLS file) refuses boot. A server that cannot be reached, or that refuses the credentials, does not: the process boots with the cache bypassed and keeps reconnecting, with backoff up to 30 s. A rejected credential (WRONGPASS, NOAUTH, or NOPERM for an ACL user missing a connection command) is logged at ERROR on every attempt; any other failure at WARN. This is deliberate: a rotated Redis password must not crash-loop every instance at once. Watch wavehouse_cache_breaker_open, which reads 1 while the cache is bypassed, including before the first connection.

Authentication

Only the operator key is boot config, shared by every tenant. The verifier wiring — jwks_url, jwt_secret and role_claim — is the settings directory’s auth block, one verifier per tenant, where a change rebuilds that tenant’s verifier without a restart; the HMAC secret is a tenant’s own, so rotating it is a reload.

YAML Key	Env Var	Default	Description
auth.operator_key	WH_AUTH_OPERATOR_KEY	(empty)	Non-JWT operator credential. A request presenting it — in an Authorization: Operator <key> header, or the X-Operator-Key alias — is authorized as a full-access platform operator (the whole data plane and the /v1/ops/* management surface, except that a pipe binding a {{ jwt.<path> }} claim answers it [] (a write pipe, 403), since it carries no claims), independent of the JWT verifier, and it is honored even when no policy is adopted (break-glass). Empty disables it. Treat it as an admin secret. See below and Access Control.

WaveHouse accepts only the signing algorithms matching the active verifier — HS256/HS384/HS512 for the HMAC secret, or the asymmetric family (RS*/ES*/PS*/EdDSA) for JWKS — and validates the token’s alg before any key is used, so your IdP must sign with one of these and alg: none is always rejected.

There is no auth on/off switch. The JWT middleware always runs. A request with no token, or an invalid/expired one, falls back to the policy default_role — except a token sent while its tenant’s JWKS has not been fetched yet, which is refused with a retryable 503 rather than downgraded (see the auth block); elevated access needs a valid token whose role is granted (or equals the policy admin_role). The privileged role and public access are policy settings, not config flags:

• admin_role (policy field, "admin" by default, exact case-sensitive match): the role granted full access and the /v1/ops/* gate. There is no separate service role, though the non-JWT operator_key (below) reaches the same surface without a token.
• default_role (policy field): set it to open public (no-token) access — roleless requests are evaluated as that role; remove it to close public access. Setting it equal to admin_role is allowed and makes every roleless request admin (including /v1/ops/*) — handy for local/dev, logged loudly on every node that loads such a policy, and not for production use. /v1/ops/* — raw SQL, pipe inspection, settings reload, schema, DLQ — is admin-only, and a pipe with no allowed_roles authorizes nobody but the admin role.

Operator key (break-glass). auth.operator_key, when set, is a non-JWT credential for the person running the deployment. A request presenting it — in the standard Authorization: Operator <key> header (forwarded verbatim by proxies, no collision with Bearer JWTs), or the X-Operator-Key alias — is authorized as a full-access platform operator without minting a JWT — the /v1/ops/* surface always, and the whole data plane while a policy is adopted (with none, data-plane requests evaluate the nil policy and are denied like any other; and carrying no token claims, it gets [] from a pipe whose SQL binds a {{ jwt.<path> }} claim, 403 from a write pipe that does — see Binding token claims). It is honored even when no policy is adopted, so an operator locked out by a wiped or broken policies.json can still trigger POST /v1/ops/settings/reload after fixing the file. It is matched with a constant-time comparison and is independent of jwt_secret/jwks_url; it takes precedence over any Bearer token on the same request. Because it is effectively an admin secret, load it from a secret store, serve it only over TLS, and leave it empty on deployments that do not need it.

See API — Authentication.

Access Control Policy and Named Pipes

There are no boot-config keys for either. The policy is the settings directory’s policies.json (with its role registry in roles.json) and the named pipes are its pipes.json; both are validated at boot and re-adopted on file change, SIGHUP, or POST /v1/ops/settings/reload. A leftover policy: or pipes: block in the YAML refuses boot by name.

Settings Directory

The one boot-config key for the Settings Directory is where it lives:

YAML Key	Env Var	Default	Description
settings.dir	WH_SETTINGS_DIR	(required)	Path to the settings directory. The server loads it at boot; the wavehouse validate and wavehouse bootstrap CLIs — like wavehouse health — resolve their input directly from the [dir] argument or WH_SETTINGS_DIR without loading this file, so a value set only in the YAML key validates on boot but not via the bare CLI. No default: a baked-in path would turn a missing mount into silent misconfiguration instead of an explicit operator choice. The container images preset it to /app/settings but ship no files there — bind-mount a directory you wrote with bootstrap.

OTel

The master switch is otel.enabled. When true, each signal (traces/metrics/logs) is then individually gated by its own enabled flag — you can run traces-only, logs-only, etc. Prometheus® exposition is configured in its own top-level prometheus block; it operates independently and works in any combination with OTel (OTLP push only, Prometheus only, both, or neither). Stdout is always active for logs (the logger fans out to both stdout and the OTLP exporter), so logs never disappear regardless of collector state. gRPC exporters are lazy, so an unreachable collector does not block startup; transient export errors are surfaced via the OTel SDK’s error handler. The if err != nil fallback in internal/app only fires for genuine init errors (malformed exporter options, or the meter provider failing to register an instrument such as wavehouse_build_info); a malformed OTEL_RESOURCE_ATTRIBUTES is only logged (see Which process sent it below).

Sampling rates apply only to the OTLP push path. Stdout always emits 100% of records — operators using a scraping-style pipeline (Promtail™/Grafana Alloy → Loki™, Vector, Fluent Bit™, etc.) set the collection rate at the scraper, not the application. WaveHouse pushes telemetry to an OTel collector; the scraper world owns its own ingest policy. If you want to throttle OTLP volume for cost, lower the rates below. If you want to throttle Loki/Datadog Logs/etc., do it at that pipeline.

Direct-to-cloud OTLP. The OTLP destination is configured through the standard OTEL_EXPORTER_OTLP_* environment variables read by the OpenTelemetry™ SDK, not WaveHouse config: OTEL_EXPORTER_OTLP_ENDPOINT (always include a scheme — https:// selects TLS via system root CAs, http:// selects plaintext), OTEL_EXPORTER_OTLP_HEADERS for per-RPC auth, OTEL_EXPORTER_OTLP_CERTIFICATE to trust a custom/private CA, and OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE / OTEL_EXPORTER_OTLP_CLIENT_KEY for mutual TLS. Without a scheme the SDK misreads the endpoint and each signal handles it differently (the logs exporter, for one, can be left with no destination at all), so always write the scheme. An unset endpoint means TLS to localhost:4317, so a plaintext local collector needs http://localhost:4317, or OTEL_EXPORTER_OTLP_INSECURE=true with the endpoint left unset. The custom/private CA and mutual TLS variables apply to all three signals (traces, metrics and logs). With these set, WaveHouse ships telemetry straight to a TLS-protected cloud gateway (Grafana® Cloud’s OTLP gateway, Honeycomb, etc.) — no sidecar required (a sidecar is still useful for egress queuing, batching, and tail-based sampling; it’s just no longer mandatory). A malformed entry in OTEL_EXPORTER_OTLP_HEADERS never fails startup, but the signals treat it differently: traces and metrics log and skip just that entry, while the logs exporter logs it and discards the whole variable, so logs are sent with no headers at all and an authenticating gateway rejects them. Datadog® has no public direct-to-cloud OTLP endpoint — use the local DDOT Collector path in the deployment guide. See Deployment → Observability for worked Honeycomb / Grafana Cloud examples.

Export cadence. Traces are exported in batches every 5 s and metrics every 15 s: the fallbacks WaveHouse passes to the SDK only while the standard OTEL_BSP_SCHEDULE_DELAY and OTEL_METRIC_EXPORT_INTERVAL variables (whole milliseconds) are unset, since an option passed to the SDK would override the variable. Set either and the SDK’s value applies; a value it cannot read falls back to its own default (5 s for traces, 60 s for metrics); the metrics SDK prints a line to stderr about it, the trace SDK nothing. The Go runtime metrics are read at most every 15 s, the SDK’s default. Logs are batched by the SDK’s OTEL_BLRP_* variables, which WaveHouse never overrode.

Which process sent it. Every span, metric series and OTLP log record carries the process’s identity as OpenTelemetry resource attributes: service.name (wavehouse), service.version, service.instance.id (the instance_id), wavehouse.roles (the roles, sorted and comma-separated, such as api,ingest), wavehouse.git_commit and wavehouse.build_time. Set the standard OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES environment variables to override any of them or add your own (OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod); they win on a shared key. A malformed OTEL_RESOURCE_ATTRIBUTES never fails startup: the entries that can be read apply, and the error is logged. The Prometheus endpoint reports the resource as target_info, and every metrics pipeline exports wavehouse_build_info (see Deployment → Telling processes apart).

YAML Key	Env Var	Default	Description
otel.enabled	WH_OTEL_ENABLED	false	Master switch. When false, no signals are initialized regardless of the sub-toggles below. The OTLP endpoint, TLS, custom CA, mutual TLS, and auth headers are configured via the standard OTEL_EXPORTER_OTLP_* env vars (see the note above), not a WaveHouse key.
otel.traces.enabled	WH_OTEL_TRACES_ENABLED	true	Export traces via OTLP gRPC.
otel.traces.sample_rate	WH_OTEL_TRACES_SAMPLE_RATE	1.0	Head-based trace sampling rate in [0.0, 1.0]. 1.0 exports every trace; 0.0 exports none. Defaults to 100% (matches the OpenTelemetry SDK default); lower it for high-QPS production services where collector or backend cost is a concern. Best practice is “100% at the source, downsample at the collector” via tail-based sampling. Validated at config load.
otel.metrics.enabled	WH_OTEL_METRICS_ENABLED	true	Export metrics + Go runtime metrics via OTLP gRPC, every 15 s unless OTEL_METRIC_EXPORT_INTERVAL says otherwise (see Export cadence above). Metrics are pre-aggregated so there is no sampling knob. Whether they carry a table or tenant label is prometheus.per_table_metrics or prometheus.per_tenant_metrics, and whether they include the HTTP body size histograms is prometheus.http_body_size_metrics: all three apply here too.
otel.logs.enabled	WH_OTEL_LOGS_ENABLED	true	Export logs via OTLP gRPC. Disabling it leaves stdout with 100% of records, but from the boot logger: lines keep trace_id/span_id, request_id, tenant and instance_id, and lose component, source and the build fields.
otel.logs.sample_rate	WH_OTEL_LOGS_SAMPLE_RATE	1.0	OTLP export rate for the DEBUG/INFO records WH_LOG_LEVEL admits, in [0.0, 1.0]. Validated at config load. WARN and ERROR records always export at 100% — dropping them silently during incidents is too dangerous to expose as a knob. Stdout always receives 100% of records regardless of this rate (see the scraper note above).

Prometheus

Prometheus exposition is its own top-level config block, independent of otel.*, except for prometheus.per_table_metrics, prometheus.per_tenant_metrics and prometheus.http_body_size_metrics, which shape every metrics output, OTLP push included. Operators using a scrape-based pipeline (Grafana Alloy™, Mimir™, the standalone Prometheus server) can leave the entire [otel] block at its enabled: false default and turn on only prometheus.enabled — no OTLP collector required. Conversely, OTLP push and Prometheus can both be on at once (the underlying OTel MeterProvider drives both readers, same Meter() API). Disabled by default since enabling adds an unauthenticated endpoint, so opt-in is explicit.

YAML Key	Env Var	Default	Description
prometheus.enabled	WH_PROMETHEUS_ENABLED	false	Expose a Prometheus-format /metrics endpoint. Works standalone (no OTel push) or alongside otel.metrics.enabled.
prometheus.path	WH_PROMETHEUS_PATH	/metrics	URL path. Must start with /, and may not collide with a reserved probe path (/livez, /readyz, /healthz, /health, /ready). When port is 0 (mounted on the main server) it also may not be /v1 or sit under /v1/, which would shadow the authenticated API. An invalid path fails validation at startup.
prometheus.port	WH_PROMETHEUS_PORT	0	Listener port. 0 mounts the endpoint on the main API server (server.port) — simplest, no extra port to expose. Non-zero spins up a dedicated HTTP listener, which lets you firewall metrics off the public API surface (common production posture). Must not equal server.port when non-zero.
prometheus.per_table_metrics	WH_METRICS_PER_TABLE	true	Keep the table label on the metrics that carry one. false drops it, so each such metric reports one series summed over every table. Applies to every metric output, OTLP push included, since both share one meter provider. See Metric labels.
prometheus.per_tenant_metrics	WH_METRICS_PER_TENANT	true	Keep the tenant label on the metrics tied to a tenant, each process labeling its first 100 tenants and reporting the rest as tenant="_other". false drops it, so each such metric reports one series summed over every tenant. Applies to every metric output. See Metric labels.
prometheus.http_body_size_metrics	WH_METRICS_HTTP_BODY_SIZE	false	Export the HTTP server’s http.server.request.body.size and http.server.response.body.size histograms, the largest families a process exports. Applies to every metric output.

Metric labels

Every series is a cost in the time-series database that stores it, and the labels a metric carries multiply its series. WaveHouse discovers whatever tables you have and sets no limit on how many, so a label holding a table name grows with your schema: wavehouse_sse_rows_withheld_total{table, role} at 2,000 tables and 10 roles is up to 20,000 series. The metrics labeled by table are wavehouse_sse_rows_withheld_total, wavehouse_sse_dropped_frames_total, wavehouse_ingest_retries_total, wavehouse_ingest_poison_total, wavehouse_ingest_rows_total, wavehouse_ingest_bytes_total, wavehouse_ingest_flushes_total, wavehouse_ingest_isolations_total, wavehouse_ingest_records_total, wavehouse_ingest_dedupe_missing_id_total, wavehouse_ingest_dedupe_commit_failed_total, wavehouse_ingest_dedupe_disabled_total and wavehouse_dedupe_hashed_id_total.

prometheus.per_table_metrics: false drops the table label from all of them at once, and each keeps its other labels, summed over every table. It stays true by default, because per-table detail is what you want while one table misbehaves. Once the metrics have carried more than 500 distinct table names, the process logs one WARN that names this setting; the label stays on until you change it. Turning it off needs a restart.

Tenant labels. The metrics tied to a tenant carry a tenant label (#891): wavehouse_http_request_duration_seconds, wavehouse_clickhouse_query_duration_seconds, wavehouse_ingest_records_total, wavehouse_ingest_dedupe_missing_id_total, wavehouse_ingest_dedupe_disabled_total, wavehouse_ingest_dedupe_commit_failed_total, wavehouse_dedupe_hashed_id_total, wavehouse_mq_publish_duration_seconds, wavehouse_ingest_rows_total, wavehouse_ingest_bytes_total, wavehouse_ingest_flushes_total, wavehouse_ingest_insert_duration_seconds, wavehouse_ingest_batch_rows, wavehouse_ingest_batch_bytes, wavehouse_ingest_isolations_total, wavehouse_ingest_retries_total, wavehouse_ingest_poison_total, the wavehouse_sse_* metrics but wavehouse_sse_queue_peak_frames, the wavehouse_cache_* metrics but wavehouse_cache_breaker_open and wavehouse_cache_invalidations_pending (and a cache invalidation round trip, which can carry several tenants), and wavehouse_schema_refresh_failures_total. A request carries its tenant once it is one the process serves: a request for a tenant it does not serve (a 404 or 503 from tenant resolution), a probe, or a reload of every tenant carries none. The ingest worker’s measurements carry the tenant only while that tenant has a ClickHouse connection: a message whose topic names a removed or unknown tenant, or one no pool could be opened for, at boot over a nested directory or at a reload (such as by the connection ceiling), carries none. Process and shared metrics (Go runtime, ingest shard and lease state, readiness, settings reloads, the MQ connection) carry none.

The label is on by default. Each process labels the first 100 distinct tenants it records a metric for, and reports every tenant after them as tenant="_other", logging one WARN that names prometheus.per_tenant_metrics the first time it does. A tenant keeps the label it was first given until the process restarts, and a removed tenant’s series simply stop growing. Each process fills its own 100, so in a deployment with more tenants than that a tenant can be labeled on one replica and in _other on another. _other is also a valid tenant id, and a tenant with that id shares the bucket’s series. With prometheus.per_tenant_metrics: false there is no label, so no cap and no warning.

There is no separate set of tenant-free metrics, since two sets would be easy to double-count. Read the deployment-wide figure with sum without (tenant) (...), leave tenants out with tenant!~"...", and add recording rules where a dashboard reads one often. prometheus.per_tenant_metrics: false drops the label from every one of them, for deployments with many tenants. Turning it on or off needs a restart.

HTTP server metrics. The OpenTelemetry HTTP server instruments (http.server.request.duration, and with prometheus.http_body_size_metrics on, http.server.request.body.size and http.server.response.body.size) do not carry server.address or server.port. Both come from the request’s Host header, so with a hostname per tenant they would be an uncapped tenant label. The two body size histograms are off by default: on v0.2.1 they were 38% of a replica’s series before any ingest traffic.

Label sets per metric. The OpenTelemetry SDK keeps at most 2,000 label sets for each metric, the overflow set among them. Once a metric has 1,999 label sets of its own, every measurement with a new one goes to a single set that carries otel_metric_overflow="true" (otel.metric.overflow=true over OTLP push) and none of the metric’s own labels, so a dashboard filtering on tenant, table or result misses those measurements. A counter or histogram adds them up in that set. A gauge keeps only the last value observed there, so overflowed gauge readings are lost rather than combined. Counters and histograms are cumulative on the Prometheus endpoint, and on OTLP push by default, so their label sets, once recorded, hold their place until the process restarts; setting the SDK’s OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta makes OTLP push start each export interval with none, while the Prometheus endpoint stays cumulative. The limit counts label sets, not the time series your database stores: a histogram exports one series per bucket plus _sum and _count for each label set, so wavehouse_http_request_duration_seconds, with 12 bucket boundaries, writes 15 series per label set, up to 30,000 at the limit. The tenant and table labels multiply a metric’s other labels: wavehouse_ingest_records_total at 100 tenants writing 20 tables each, for example, fills the limit with a single result per tenant and table. Alert on any series with that overflow label. When one appears, set prometheus.per_table_metrics: false or prometheus.per_tenant_metrics: false, since the limit counts label sets after those settings drop their label. The SDK’s own experimental variable OTEL_GO_X_CARDINALITY_LIMIT raises the limit (0 or less removes it), at the cost of those series in your time-series database.

Counters that start at 0. rate() and increase() work from a series’ second sample, so a counter series whose first sample is already 1 hides the event that created it, and an alert on increase(...) > 0 misses a first failure. WaveHouse therefore exports these counters at 0 for every value of their fixed labels once the component that counts them starts (#905):

• Always: wavehouse_ingest_shard_events_total for every event in a process that claims ingest shards, wavehouse_coord_lease_renewal_failures_total for each lease from when the process first takes it, wavehouse_auth_operator_key_requests_total and wavehouse_auth_operator_key_failures_total, wavehouse_dedupe_dynamodb_requests_total for every op and outcome along with wavehouse_dedupe_dynamodb_unprocessed_items_total and wavehouse_dedupe_dynamodb_short_circuits_total under the DynamoDB dedupe backend, wavehouse_dedupe_swept_keys_total for both reasons under the embedded one, and wavehouse_http_panics_recovered_total, which has no labels.
• With prometheus.per_tenant_metrics: false: the wavehouse_cache_* counters for each result or reason, wavehouse_sse_frames_sent_total and wavehouse_sse_bytes_sent_total for each kind, and wavehouse_schema_refresh_failures_total.
• With prometheus.per_table_metrics: false and prometheus.per_tenant_metrics: false both: wavehouse_sse_dropped_frames_total, wavehouse_ingest_records_total, wavehouse_ingest_rows_total, wavehouse_ingest_bytes_total, wavehouse_ingest_flushes_total, wavehouse_ingest_retries_total, wavehouse_ingest_poison_total and wavehouse_ingest_isolations_total for every value of their other labels, and wavehouse_ingest_dedupe_missing_id_total, wavehouse_ingest_dedupe_commit_failed_total, wavehouse_ingest_dedupe_disabled_total and wavehouse_dedupe_hashed_id_total.

While a counter keeps its table or tenant label, its series still appear at their first event, since WaveHouse cannot know the values in advance. wavehouse_sse_rows_withheld_total is labeled by role, whose values come from your policy, so it never starts at 0. To catch a first event on such a series, alert on the series appearing as well as on its increase:

increase(wavehouse_ingest_poison_total[10m]) > 0
  or ((wavehouse_ingest_poison_total > 0) unless wavehouse_ingest_poison_total offset 10m)

New metrics follow one rule for their labels:

Label	Use
A closed set: kind, role, reason, a status or an error class	Always fine. A counter whose values are fixed in the code exports each at 0 when its component starts, through observability.ZeroCounter.
table	Fine on a counter, an up-down counter (observable included) or a histogram; prometheus.per_table_metrics drops it with the rest, and the series sum. Not on a gauge: with the label dropped, a gauge reports one table’s value, not the total.
column	Avoid: it multiplies against table.
tenant	On a metric tied to one tenant’s traffic or data (requests, queries, ingest, SSE, cache, schema refresh), recorded through observability.MetricAttrs or TenantLabel so it is capped and prometheus.per_tenant_metrics drops it. Not on a process or shared metric (Go runtime, shard and lease state, readiness, settings reloads), and not on a gauge, for the same reason as table.
A value from a row or a request payload	Never.

Profiling

Go’s net/http/pprof endpoints (heap, CPU, goroutine, allocation, thread-creation, block and mutex profiles, the execution trace, the command line and symbol lookup) are served under /debug/pprof/ on a listener of their own, never on server.port: a profile exposes the process’s internals and the endpoints take no credential. Off by default. With it on, every process serves it, whatever its roles. The default address is the loopback interface, so turning it on exposes nothing beyond the host; kubectl port-forward still reaches a loopback listener in a pod. An address that is not loopback is logged at WARN at boot. Block and mutex profiles stay empty, since WaveHouse sets no block or mutex sampling rate. make dev turns profiling on unless WH_PROFILING_ENABLED is set in its environment. See Deployment → Profiling for the commands.

YAML Key	Env Var	Default	Description
profiling.enabled	WH_PROFILING_ENABLED	false	Serve /debug/pprof/ on profiling.addr. A failure to bind is logged at ERROR and the process serves on without it.
profiling.addr	WH_PROFILING_ADDR	127.0.0.1:6060	host:port to listen on. The port must be 1–65535 and must not be server.port or, with prometheus.enabled, prometheus.port. An empty host (:6060) listens on every interface. Read only when profiling.enabled is true, and an invalid value fails validation at startup.

Logging

Env Var	Default	Description
WH_LOG_LEVEL	INFO	Minimum log level. One of DEBUG, INFO, WARN, ERROR (case-insensitive). Applies to both stdout and (when OTel is enabled) the OTLP log exporter: a record below it is neither written nor exported, WARN included when the level is ERROR. See otel.logs.sample_rate above for the OTLP export rate.

Example Config File

Every key, with its default. Save the YAML as config.yaml next to the binary (or point WH_CONFIG at it), or supply the same settings as environment variables — e.g. an env_file in Compose:

YAML

data_dir: ./data         # nats → ./data/nats, pebble → ./data/pebble

roles: [api, ingest, sweeper]   # the work this process runs; a split needs shared backends
instance_id: ""          # on every log line and span; empty = <hostname>-<8 hex>, fresh at every boot

server:
  port: 8080
  shutdown_timeout: 10
  trusted_proxies: []    # CIDRs or addresses whose X-Forwarded-For and traceparent are believed; empty = none

clickhouse:
  # addr, ports, database, username, password, query_timeout, insert_timeout,
  # tls, headers and pool sizes are settings (config.json)
  max_total_conns: 0     # ceiling on open native connections; 0 = none

mq:
  backend: embedded      # in-process NATS JetStream under <data_dir>/nats
  ack_wait: 60s          # the worker's redelivery timeout, on either backend
  max_ack_pending: 100000 # rows delivered and unacked per tenant queue, or per shard
  prefetch: 500          # rows the worker fetches ahead of its batches
  # With backend: nats, the operator's cluster instead (read only then):
  # nats:
  #   urls: ["nats://nats.nats.svc:4222"]
  #   user: wavehouse
  #   password_file: /var/run/secrets/nats/password
  #   partitions: 4        # the number of ingest partition streams
  #   shards: 8            # shards per partition, as generated (default 32)

cache:
  backend: local         # local | redis
  l1_max_cost: 67108864  # the local backend's size
  redis:                 # read only with backend: redis
    addrs: []            # required with backend: redis, e.g. ["redis:6379"]
    mode: standalone     # standalone | cluster
    username: ""
    password: ""         # a secret: set WH_CACHE_REDIS_PASSWORD instead
    db: 0
    tls:
      enabled: false
      ca_file: ""
      cert_file: ""
      key_file: ""
      server_name: ""
      insecure_skip_verify: false
    key_prefix: wh
    timeout: 100ms
    dial_timeout: 1s
    max_value_bytes: 1048576
    compress_min_bytes: 1024   # 0 = never
    version_ttl: 168h

dedupe:
  backend: pebble        # in-process Pebble under <data_dir>/pebble; or dynamodb
  lease: 30s             # at most 59s with the embedded mq
  reserve_concurrency: 64
  # dynamodb:            # read only when backend is dynamodb
  #   table: wavehouse-dedupe-prod
  #   region: ""         # empty = AWS_REGION
  #   endpoint: ""       # dynamodb-local only
  #   timeout: 250ms
  #   max_attempts: 3
  #   retry_mode: standard
  #   create_table: false  # dynamodb-local only

coord:
  backend: local         # in-process leases (the sweeper's)
  # With backend: nats (needs mq.backend: nats; read only then):
  # nats:
  #   bucket: ""         # empty = <mq.nats.subject_prefix>_coord

auth:
  # jwt_secret, jwks_url and role_claim are settings (config.json)
  operator_key: "" # non-JWT full-access operator credential (Authorization: Operator <key>, or X-Operator-Key); empty disables

settings:
  dir: ./settings        # REQUIRED — no default: settings directory (roles/
                         # policies/pipes/config .json), validated at boot,
                         # reloaded on file change, SIGHUP, or
                         # POST /v1/ops/settings/reload; create one with
                         # `wavehouse bootstrap ./settings` (the access-control
                         # policy, the named pipes, and the dedupe, dlq, ingest,
                         # query, schema, stream, cors keys live there)

otel:
  enabled: false         # master switch — set true to export via OTLP gRPC
  # Endpoint, TLS, custom CA, mTLS, and auth headers come from the standard
  # OTEL_EXPORTER_OTLP_* env vars (OTEL_EXPORTER_OTLP_ENDPOINT=https://host:port,
  # OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=KEY), read by the OTel SDK;
  # so does the export cadence (OTEL_BSP_SCHEDULE_DELAY, OTEL_METRIC_EXPORT_INTERVAL).
  traces:
    enabled: true
    sample_rate: 1.0     # head-based, [0.0, 1.0]; tune down for high QPS
  metrics:
    enabled: true        # OTLP push for metrics
  logs:
    enabled: true
    sample_rate: 1.0     # DEBUG/INFO OTLP rate; WARN+ always 100%, stdout always 100%

prometheus:
  enabled: false         # independent of otel — works standalone for scrape
  path: /metrics
  port: 0                # 0 = mount on server.port; non-zero = sidecar listener
  per_table_metrics: true # false = no table label, one series over all tables (OTLP too)
  per_tenant_metrics: true # false = no tenant label, one series over all tenants (OTLP too)
  http_body_size_metrics: false # true = keep otelhttp's request/response body size histograms

profiling:
  enabled: false         # net/http/pprof under /debug/pprof/, never on server.port
  addr: 127.0.0.1:6060   # loopback by default; the endpoints are unauthenticated

Environment

WH_DATA_DIR=./data

WH_ROLES=api,ingest,sweeper
# Empty = <hostname>-<8 hex>, fresh at every boot.
WH_INSTANCE_ID=

WH_SERVER_PORT=8080
WH_SERVER_SHUTDOWN_TIMEOUT=10
# Comma-separated CIDRs or addresses; empty = trust no peer.
WH_SERVER_TRUSTED_PROXIES=

WH_CH_MAX_TOTAL_CONNS=0

WH_MQ_BACKEND=embedded
WH_MQ_ACK_WAIT=60s
WH_MQ_MAX_ACK_PENDING=100000
WH_MQ_PREFETCH=500
# With WH_MQ_BACKEND=nats (read only then):
# WH_MQ_NATS_URLS=nats://nats.nats.svc:4222
# WH_MQ_NATS_USER=wavehouse
# WH_MQ_NATS_PASSWORD_FILE=/var/run/secrets/nats/password
# WH_MQ_NATS_PARTITIONS=4
# WH_MQ_NATS_SHARDS=8
WH_CACHE_BACKEND=local
WH_CACHE_L1_MAX_COST=67108864
# Read only with WH_CACHE_BACKEND=redis; WH_CACHE_REDIS_ADDRS is then required.
WH_CACHE_REDIS_ADDRS=
WH_CACHE_REDIS_MODE=standalone
WH_CACHE_REDIS_USERNAME=
WH_CACHE_REDIS_PASSWORD=
WH_CACHE_REDIS_DB=0
WH_CACHE_REDIS_TLS_ENABLED=false
WH_CACHE_REDIS_TLS_CA_FILE=
WH_CACHE_REDIS_TLS_CERT_FILE=
WH_CACHE_REDIS_TLS_KEY_FILE=
WH_CACHE_REDIS_TLS_SERVER_NAME=
WH_CACHE_REDIS_TLS_INSECURE_SKIP_VERIFY=false
WH_CACHE_REDIS_KEY_PREFIX=wh
WH_CACHE_REDIS_TIMEOUT=100ms
WH_CACHE_REDIS_DIAL_TIMEOUT=1s
WH_CACHE_REDIS_MAX_VALUE_BYTES=1048576
WH_CACHE_REDIS_COMPRESS_MIN_BYTES=1024
WH_CACHE_REDIS_VERSION_TTL=168h
WH_DEDUPE_BACKEND=pebble
WH_DEDUPE_LEASE=30s
WH_DEDUPE_RESERVE_CONCURRENCY=64
# Read only with WH_DEDUPE_BACKEND=dynamodb:
# WH_DEDUPE_DYNAMODB_TABLE=wavehouse-dedupe-prod
# WH_DEDUPE_DYNAMODB_REGION=
# WH_DEDUPE_DYNAMODB_ENDPOINT=
# WH_DEDUPE_DYNAMODB_TIMEOUT=250ms
# WH_DEDUPE_DYNAMODB_MAX_ATTEMPTS=3
# WH_DEDUPE_DYNAMODB_RETRY_MODE=standard
# WH_DEDUPE_DYNAMODB_CREATE_TABLE=false
WH_COORD_BACKEND=local
# With WH_COORD_BACKEND=nats (read only then; empty = <prefix>_coord):
# WH_COORD_NATS_BUCKET=

WH_AUTH_OPERATOR_KEY=

# Required, no default: `wavehouse bootstrap ./settings` writes a starter
# directory (edit clickhouse.addr in its config.json if ClickHouse is not on
# localhost:9000; its policies.json is empty, so add a policy before
# expecting any request through). The container images preset WH_SETTINGS_DIR=/app/settings
# and ship no files there; you mount a directory at that path instead.
# (Comments stay on their own line — an env_file does not strip inline ones.)
WH_SETTINGS_DIR=./settings

WH_OTEL_ENABLED=false
WH_OTEL_TRACES_ENABLED=true
WH_OTEL_TRACES_SAMPLE_RATE=1.0
WH_OTEL_METRICS_ENABLED=true
WH_OTEL_LOGS_ENABLED=true
WH_OTEL_LOGS_SAMPLE_RATE=1.0

WH_PROMETHEUS_ENABLED=false
WH_PROMETHEUS_PATH=/metrics
WH_PROMETHEUS_PORT=0
WH_METRICS_PER_TABLE=true
WH_METRICS_PER_TENANT=true
WH_METRICS_HTTP_BODY_SIZE=false

WH_PROFILING_ENABLED=false
WH_PROFILING_ADDR=127.0.0.1:6060