Settings Directory

Boot config — the YAML file and WH_* environment variables on the Configuration page — is what only a restart can change: resource sizing, listeners, exporters, and the platform’s own secrets. This page covers everything else: the settings directory, the behavioral configuration a running instance re-reads without restarting — the access-control policy and its role registry, the named pipes, and the tunables including the ClickHouse® and auth wiring.

The settings directory holds WaveHouse’s file-based settings as exactly four JSON documents: roles.json, policies.json, pipes.json, and config.json. The files are the only write path — standalone, you edit them on the host; on WaveHouse Cloud the control plane writes them — and there is no API that writes back to them. Every file must exist (an empty document is {} — a missing file always means deletion or a wrong path, never “defaults”), and any other entry in the directory is an error, so a typoed filename or a stray backup fails loudly instead of being silently ignored. Dot-prefixed entries are the one exception: editor swap files and the ..data machinery Kubernetes® ConfigMap mounts publish through are ignored.

Create one with wavehouse bootstrap [dir]: it writes all four files with every key at its default and refuses a non-empty directory, so an existing settings directory is never overwritten. The binary carries no compiled defaults but one — a missing dedupe.retention means "0", forever — so the seed is where the defaults live, and what the server adopts is exactly what the files say. The seed ships no policy (policies.json is {}, roles.json and pipes.json are empty lists), so a freshly bootstrapped directory boots fail-closed — every request is denied until you write a policy. The container images ship no settings directory: they preset WH_SETTINGS_DIR=/app/settings and expect a bind mount there — a host directory you wrote with bootstrap (the reference compose file mounts the checked-in deployments/compose/settings/, the seed with clickhouse.addr pointed at the clickhouse service and a permissive public trial policy in policies.json / roles.json). A bind mount, not a named volume: the images are distroless, with no shell to edit files inside a volume. A missing mount refuses to boot rather than running on defaults nobody chose.

Check a directory with wavehouse validate [dir]. Both commands resolve the directory the same way — the argument, falling back to WH_SETTINGS_DIR, and a usage error (exit 2) with neither — so the path you seed is the path you validate, and inside the container images (which preset WH_SETTINGS_DIR=/app/settings) both work with no argument at all. validate validates without starting the server (JSON syntax including unknown fields and duplicate keys, per-file shape rules including the required keys, and cross-file role references), prints every finding in one pass, then the token claims each pipe of a valid directory or tenant folder binds (pipes.json: pipes[0] "daily_totals" binds token claims jwt.app_metadata.tenant_ids, one line per claim-scoped pipe — Binding token claims), and exits 0 for valid (warnings allowed), 1 for invalid, 2 for usage — so operators and CI can gate a settings change before it reaches a running instance.

Where the directory lives is the one boot-config key involved: settings.dir / WH_SETTINGS_DIR, documented in Configuration — Settings Directory. It’s required with no default; the container images preset it to /app/settings.

Loading and hot reload

The server validates and adopts the directory at boot — a missing or invalid directory refuses to start, so a typo, a missing mount, or an invalid policy surfaces immediately instead of silently denying every request (run wavehouse validate to reproduce the findings, wavehouse bootstrap to create a starter directory). A nested directory that would serve no tenant — every folder rejected, or none named by a tenant id — refuses to start the same way. Boot config has no equivalent command and needs none: it only takes effect through a restart, and the restart refuses on an undeclared YAML key, an unbound WH_* variable, or an unusable data_dir — see Configuration — Loading Order.

A running instance re-validates and re-adopts on any of three triggers, all funneling through one serialized reload path:

File change — the directory is watched; a burst of writes (an editor save, a Kubernetes Secret republish — a Secret rather than a ConfigMap, since config.json carries the tenant’s secrets) coalesces into one reload, and a directory that is deleted and recreated (or renamed over) is picked up again.
SIGHUP — kill -HUP <pid>.
POST /v1/ops/settings/reload (admin-only) — returns {"adopted": bool, "findings": [...]}; 200 when adopted, 422 when rejected.

A reload that fails validation is logged (and reported by the endpoint) and the previous good settings stay in effect — an operator mid-edit, a deleted file, even a vanished directory degrades to a log line, never to a broken server. Warnings don’t block adoption, matching wavehouse validate. Each process also counts its reloads and their results in metrics, so a reload rejected by one pod can be alerted on.

“Previous good settings” is the in-memory snapshot of the running process, nothing more: there is no persisted copy of the files. A restart re-validates the directory from scratch and refuses to start on the same findings the reload rejected, so bad files never survive a restart silently — fix them (or run wavehouse validate) before bouncing the server.

A directory that holds one folder per tenant instead of the four files is a nested settings directory: each folder is everything this page describes, but it is not watched, a rejected folder stops its tenant being served rather than keeping the previous settings, and the keys the whole process shares are not read from the tenant’s own folder: they come from tenant 0’s — that section lists which keys.

Every adoption — boot and every reload — goes through the same Validate, so the policy, the roles, and the pipes are checked with the current rules each time they are read; there is no stored copy that can skip validation. All four files are adopted as one snapshot: a request is evaluated against the policy, pipes, and tunables of a single adoption, never a mix.

Every file is decoded strictly: an unknown key, a duplicate key, a UTF-8 byte order mark, an all-whitespace file, or a top-level null is an error, so a misspelled rule can never load as “no rule”.

Validating from Go

A program that writes settings directories — a control plane, a provisioning tool, a GitOps generator — can check a directory with the server’s own rules before it hands it over: no running server, nothing on disk. The settings model and its validator are public packages of the server module, versioned by its release tags, so validate with the package from the same tag as the server: go get github.com/Wave-RF/WaveHouse@vX.Y.Z checks with exactly the rules a vX.Y.Z server applies, and a writer that re-implemented them would drift the first time they changed. The packages are public from v0.2.1; v0.2.0 has none to import.

import (
    "encoding/json"
    "errors"
    "fmt"

    "github.com/Wave-RF/WaveHouse/pkg/settings"
)

files, err := settings.Seed() // the starter directory: every file, every key at its default
if err != nil {
    return err
}
var config settings.TenantConfig
if err := json.Unmarshal(files[settings.FileConfig], &config); err != nil {
    return err
}
*config.ClickHouse.Addr = "clickhouse:9000"
if files[settings.FileConfig], err = json.Marshal(config); err != nil {
    return err
}

doc, findings := settings.ValidateFiles(files)
for _, f := range findings {
    // The seed prints one finding, "warning: policies.json: empty document — no policy; ...";
    // an addr without a port would add "error: config.json: clickhouse.addr: must be host:port, got ...".
    fmt.Println(f)
}
if doc == nil {
    return errors.New("the server would refuse this directory") // an error finding; warnings alone leave doc set
}

settings.ValidateFiles takes the four files by name (settings.Files()) as bytes and runs the pass wavehouse validate, boot and a reload run — there is one validator, and every path goes through it — returning every finding at once, the same {"severity", "file", "path", "message"} objects POST /v1/ops/settings/reload returns. It skips only what needs a directory: that it exists, that each file is a regular file (not a subdirectory or a FIFO), and config.json’s permission bits; settings.ValidateDir runs those too, over a directory on disk, and settings.Validate over a root of either shape. Names are read as directory entries are: a dot-prefixed name is skipped, any other name outside the four is an unexpected file, and an absent or nil file is missing. For a nested directory, check a folder name with tenant.Parse (github.com/Wave-RF/WaveHouse/pkg/tenant) before creating the folder. The document types — settings.TenantConfig, settings.RolesFile, settings.PipesFile, and policy.Policy from github.com/Wave-RF/WaveHouse/pkg/policy — are what the validator decodes into, and the seed round-trips through them to a document the validator reads identically, so a writer that starts from settings.Seed() and edits the structs emits documents in the shape the server reads. Some checks need the table schema (a policy check naming a column the table lacks, say) and run only against ClickHouse; they are not part of this package. Nor are the rules that need boot config: a folder’s ingest.max_batch_wait_ms plus clickhouse.insert_timeout must stay under the deployment’s mq.ack_wait (Ingest), and the open pools’ max_open_conns together must not exceed its clickhouse.max_total_conns, when one is set (ClickHouse), which only the processes check when they adopt the folder — so a writer that knows the deployment’s boot config checks them itself before writing the folder.

`roles.json`

The role registry — every role a policy grant or a pipe allowlist may name:

{ "roles": ["admin", "analyst", "public"] }

Rules: each name must be non-empty, without surrounding whitespace, and unique. Cross-file, every role referenced elsewhere must be declared here: policies.json’s default_role and admin_role, every role keyed under a table’s grants, and every entry in a pipe’s allowed_roles. An undeclared reference is an error, so a typo in a grant fails validation instead of silently matching nobody.

Does the admin role belong in this list? Only if policies.json names it: setting admin_role (or using the admin role as default_role) requires declaring that role here, like any other reference. Leaving admin_role unset — the built-in default admin — requires nothing; the reference compose seed’s roles.json is just ["public"] and validates clean for exactly this reason. Naming the admin role inside a table grant or a pipe’s allowed_roles never requires a declaration either — admin is an unconditional bypass, so such an entry is dead config and only warns.

`policies.json`

The access-control policy — one policy document (default_role, admin_role, tables):

{
  "default_role": "public",
  "admin_role": "admin",
  "tables": {
    "events": {
      "public": {
        "select": { "allow_columns": ["*"] },
        "insert": { "allow_columns": ["*"] }
      }
    }
  }
}

An empty document ({}) means no policy: it validates with a warning, and the server adopts it fail-closed — every request is denied, including one carrying the admin role; only the operator key gets through. A non-empty document must pass the full policy validation (column and row rules, the claim-template grammar, resource limits), and its roles must be declared in roles.json. The layout is role-first — tables.<table>.<role>.select — and a document still using the pre-v2 operation-first nesting is rejected, with the message linking straight to the migration note. A role named after an operation is the shape the two layouts most visibly collide in, and it gets its own message rather than the migration one: tables.<table>.select.select grants the same access under either reading and is accepted, but tables.<table>.select.insert means a read-only role named insert under the old layout and a write-only role named select under the new one — so validation refuses to guess and asks you to rename the role. Three states warn rather than fail: default_role equal to the admin role (every roleless request is admin — dev only), a grant keyed by the admin role (admin is an unconditional bypass, so the grant has no effect), and a grant that sets neither select nor insert (the role gets no access to the table — most often a half-finished migration). A leftover pre-v2 "select": {} block decodes into a grant keyed by a role literally named select. Whether you see this warning or an error then depends on roles.json: if select is not declared there — the usual case — the undeclared-role error fires first and you never reach the warning; if it is declared, you get the warning and the document adopts.

A reload applies to the very next request: the policy is read per request off the adopted snapshot, and a live GET /v1/stream connection’s role is re-evaluated against the new policy on the next event.

`pipes.json`

The named pipes:

{
  "pipes": [
    {
      "name": "top_pages",
      "description": "Most-visited pages over a window",
      "sql": "SELECT page, count() AS hits FROM events WHERE ts > now() - INTERVAL {{hours:24}} HOUR GROUP BY page ORDER BY hits DESC LIMIT {{limit}}",
      "parameters": [
        { "name": "hours", "type": "number", "default": 24 },
        { "name": "limit", "type": "number", "required": true }
      ],
      "allowed_roles": ["analyst"]
    }
  ]
}

Each pipe carries name, sql, optional description, optional parameters (name, type, required, default), and allowed_roles. Rules: names unique and non-empty; sql non-empty; parameter names non-empty and unique within the pipe; type one of string, number, boolean, array (or omitted); no parameter named jwt; every {{ in sql that starts like a placeholder (optional spaces, then a letter, digit, _ or .) is a {{param}} placeholder or a well-formed {{ jwt.<path> }} claim reference (Binding token claims), any other {{ is left to ClickHouse, and no placeholder sits inside a comment or inside quotes (a string literal, a quoted identifier, a heredoc), where ClickHouse reads text but the binder would still bind; every allowed_roles entry non-empty and declared in roles.json. A default on a required parameter warns (it is never used), and listing the admin role in allowed_roles warns (admin can always execute). The token claims a pipe binds are not a finding: wavehouse validate lists them under its findings (for the daily_totals pipe, listed first: pipes.json: pipes[0] "daily_totals" binds token claims jwt.app_metadata.tenant_ids). A pipe with no allowed_roles authorizes nobody but the admin role. Table and column existence are not checked — that is schema discovery’s job at execution time.

Pipes are read per request, so a reload changes what the next GET /v1/pipes/{name} executes and what GET /v1/ops/pipes lists.

`config.json` keys

The tenant tunables. Every key is required (a missing one is a validation error) except dedupe.retention and the per-table overrides; the “Seed” column is what wavehouse bootstrap writes:

Key	Seed	Description
`clickhouse.addr`	`localhost:9000`	Native-protocol `host:port` — schema discovery, structured queries, pipes, `/readyz`. See ClickHouse.
`clickhouse.http_port`	`8123`	HTTP interface port on the same host (ingest `INSERT`s and the raw-SQL proxy), `1–65535`.
`clickhouse.http_scheme`	`http`	`http` or `https` for that HTTP hop — one of the two outbound TLS switches, with `tls.enabled` for the native hop; unrelated to your clients’ TLS.
`clickhouse.database`	`default`	Database tables are discovered from.
`clickhouse.username`	`default`	Connection user.
`clickhouse.password`	`""`	The user’s password; empty for a passwordless user. A tenant’s own secret, kept here with the user it pairs with: a change is a new connection tuple, so a reload moves the tenant to a pool on the new password (see ClickHouse).
`clickhouse.query_timeout`	`30`	Seconds (`>= 1`) a ClickHouse call may take on the query paths — structured queries, pipes (a write pipe included) and `/v1/ops/query`. On `/v1/query` under a role’s `max_execution_time`, the smaller of the two is sent to ClickHouse as `max_execution_time`; otherwise, on the native paths, it bounds the client deadline, from which the driver derives a server-side `max_execution_time`; on `/v1/ops/query` it is the HTTP request’s deadline.
`clickhouse.insert_timeout`	`30`	Seconds (`>= 1`) each ingest `INSERT` may take, the deadline of every request the worker sends for this tenant’s batches, row-by-row isolation included; a timed-out insert is retried with backoff, never parked. With `ingest.max_batch_wait_ms` it must stay under the boot config’s `mq.ack_wait`, which the ingest processes check when they adopt the tenant’s settings: see Ingest.
`clickhouse.tls.enabled`	`false`	Switches the native-protocol hop (`addr`) to TLS. The HTTP hop’s switch stays `http_scheme`; the rest of the `tls` block applies to whichever hop uses TLS. See ClickHouse.
`clickhouse.tls.ca_file`	`""`	PEM bundle the server certificate is verified against; empty uses the system roots. A path, read when the connection is built and re-read when the `tls` block changes — validation does not open it.
`clickhouse.tls.cert_file`	`""`	Client certificate for mutual TLS, PEM; set together with `key_file` or not at all.
`clickhouse.tls.key_file`	`""`	The client certificate’s private key, PEM.
`clickhouse.tls.insecure_skip_verify`	`false`	Skips server certificate verification. `true` validates with a warning: both hops then accept any certificate.
`clickhouse.tls.server_name`	`""`	Name the server certificate is verified against; empty derives it from the host in `addr` (native) or the URL (HTTP).
`clickhouse.headers`	`{}`	Extra request headers for the HTTP interface (ingest `INSERT`s, `POST /v1/ops/query`). WaveHouse’s own `Content-Type` and credential headers, and the `traceparent` and `tracestate` of a request WaveHouse traces, are set after them and win; naming `X-ClickHouse-User`, `X-ClickHouse-Key` or `Authorization` is a validation error, and so are two spellings of one name (names are case-insensitive). See ClickHouse.
`clickhouse.max_open_conns`	`10`	Native connection pool size (`>= max_idle_conns`); tenants sharing a pool size it to the largest ask among them. The open pools together must not exceed the boot config’s `clickhouse.max_total_conns` when one is set. See ClickHouse.
`clickhouse.max_idle_conns`	`5`	Idle native connections kept open (`>= 1`).
`auth.jwks_url`	`""`	JWKS endpoint (absolute `http(s)` URL). When set, JWKS is the sole verifier and `jwt_secret` is ignored (validation warns when both are set). See Authentication.
`auth.jwt_secret`	`""`	HMAC secret (`HS256`/`HS384`/`HS512`) tokens are verified with while `jwks_url` is empty. A tenant’s own secret, kept here with the verifier wiring: a change rebuilds the verifier on reload, and tokens signed with the previous secret stop validating at once. Empty with `jwks_url` empty validates no token — every request is the policy `default_role` — which boot warns about per tenant. See Authentication.
`auth.role_claim`	`role`	Dot-separated JWT claim path the role is read from (e.g. `app_metadata.role`).
`dedupe.enabled`	`false`	Turn deduplication on; a reload opens or closes this tenant’s store — see Deduplication.
`dedupe.id_field`	`event_id`	Dedup key field — see Deduplication.
`dedupe.require_id`	`false`	Reject rows missing the id field — see Deduplication.
`dedupe.retention`	`"0"`	How long a committed id stays a duplicate, as a duration (`"720h"`); `"0"`, or leaving the key out, keeps it forever — see Deduplication.
`dedupe.tables.<table>.{id_field, require_id, retention}`	`{}`	Optional per-table overrides; each entry overrides only the fields it names and inherits the rest.
`dlq.enabled`	`true`	Park poison rows — those ClickHouse still rejects after row-by-row isolation, and every row of a batch whose tenant has no ClickHouse connection — on the tenant’s dead-letter stream (`DLQ_{tenant}`) (`false`: leave them unacked for redelivery — except an envelope the worker cannot read, which is dropped and counted) — see Dead Letter Queue.
`dlq.tables.<table>.enabled`	`{}`	Optional per-table override of the switch.
`ingest.max_batch_bytes`	`262144`	Bytes of row JSON (`>= 1`) at which a table’s batch is flushed to ClickHouse: the sum of its rows’ `JSONCompactEachRow` lines, an estimate of the uncompressed `INSERT`. Read at each batch’s first row. See Ingest.
`ingest.max_batch_wait_ms`	`5000`	Milliseconds (`>= 1`) after a batch’s first row at which the batch is flushed, whatever its size; a flush asked for while the table’s previous insert is running starts when that insert ends. Read at each batch’s first row. See Ingest.
`ingest.retry_backoff_max`	`30`	Seconds (`>= 1`) the retry backoff of a ClickHouse that cannot take this tenant’s inserts doubles up to, from one second. See Ingest.
`query.timestamp_bucket_seconds`	`60`	Bucket (seconds, `>= 0`) that a structured query’s relative time range is truncated to, so near-identical queries share a cache entry; `0` disables bucketing. Read per query.
`query.default_max_rows`	`10000`	Fallback result `LIMIT` (`>= 1`) applied to a structured query when the caller and policy specify none. A result-shaping default, not a resource limit — server-wide limits (memory, rows scanned, execution time) belong in ClickHouse, see Server-side resource limits.
`schema.refresh_interval`	`60`	Seconds between ClickHouse table-schema re-discoveries (`>= 1`), per tenant; a reloaded value takes effect from the next refresh cycle, and the first periodic refresh lands at a random point within the interval. Schemas are also refreshable on demand via `POST /v1/ops/schema/refresh` (admin-only).
`stream.keepalive_interval`	`30`	Seconds (`>= 1`) a quiet `GET /v1/stream` connection may go without a write before the server sends a `:` keepalive comment — keep it under your proxy’s idle timeout; see Streaming.
`stream.keepalive_buckets`	`3`	Load-spreading (`1` to `100`): connections are spread across N buckets so each tick nudges ~1/N of live streams. Most deployments leave it.
`stream.gap_window_minutes`	`15`	Minutes (`>= 0`) of written-to-ClickHouse history the Active Sweeper keeps in NATS™ for `Last-Event-ID` gap-fill; applies from the next sweep. Under `mq.backend: nats` the history stream’s `max_age` decides instead, and the API processes check at boot and after each reload, warning once per tenant and window about a tenant whose window is longer.
`stream.subscriber_queue`	`64`	Frames (`1` to `10000`) each `GET /v1/stream` connection may have waiting for a slow client before the ones that do not fit are dropped; applies to connections opened after a reload. See Streaming.
`mq.max_bytes_gb`	`50`	Disk budget (GB, `>= 1`) for the tenant’s embedded NATS ingest stream (`INGEST_{tenant}`); its dead-letter stream (`DLQ_{tenant}`) gets a tenth of it. A reload updates the live streams in place. See Message Queue.
`cors.allowed_origins`	`["*"]`	Allowed CORS origins, applied per request. `""` allows any browser origin. WaveHouse is a Bearer-token API — `Access-Control-Allow-Credentials` is intentionally never sent, so this allowlist controls which origins can read responses, not cookie scope. Tighten to your frontend’s exact origin(s) in production (e.g. `["https://dashboard.example.com", "http://localhost:3000"]`). An empty list `[]` denies every browser origin (no `Access-Control-Allow-Origin` is ever sent); `""` is the only allow-all spelling. Over a nested settings directory each tenant’s list decorates its own responses, the preflight included; which list answers a preflight, the tenant-exempt routes, and a refused request is spelled out there.

{
  "clickhouse": {
    "addr": "clickhouse.internal:9440",
    "http_port": 8443,
    "http_scheme": "https",
    "database": "analytics",
    "username": "wavehouse",
    "password": "s3cret",
    "query_timeout": 30,
    "insert_timeout": 30,
    "tls": {
      "enabled": true,
      "ca_file": "/etc/wavehouse/clickhouse-ca.pem",
      "cert_file": "",
      "key_file": "",
      "insecure_skip_verify": false,
      "server_name": ""
    },
    "headers": {},
    "max_open_conns": 10,
    "max_idle_conns": 5
  },
  "auth": { "jwks_url": "https://auth.example.com/.well-known/jwks.json", "jwt_secret": "", "role_claim": "app_metadata.role" },
  "dedupe": {
    "enabled": false,
    "id_field": "event_id",
    "require_id": false,
    "retention": "720h",
    "tables": {
      "clicks": { "id_field": "click_id", "retention": "24h" }
    }
  },
  "dlq": {
    "enabled": true,
    "tables": {
      "audit_log": { "enabled": false }
    }
  },
  "ingest": { "max_batch_bytes": 262144, "max_batch_wait_ms": 5000, "retry_backoff_max": 30 },
  "query": { "default_max_rows": 10000, "timestamp_bucket_seconds": 60 },
  "schema": { "refresh_interval": 60 },
  "stream": { "keepalive_interval": 30, "keepalive_buckets": 3, "gap_window_minutes": 15, "subscriber_queue": 64 },
  "mq": { "max_bytes_gb": 50 },
  "cors": { "allowed_origins": ["*"] }
}

What stays in boot config is only what cannot change under a running process — the implementation each layer runs on (mq.backend, cache.backend, dedupe.backend, coord.backend) and a shared backend’s connection (mq.nats, coord.nats, cache.redis, dedupe.dynamodb), how a dedupe claim behaves (dedupe.lease, dedupe.reserve_concurrency), the process’s roles, resource sizing (data_dir, cache.l1_max_cost, clickhouse.max_total_conns, and the ingest worker’s consumer: mq.ack_wait, mq.max_ack_pending, mq.prefetch), the listeners, the observability exporters — and the platform’s own secrets: auth.operator_key and cache.redis.password, which stay in the environment and need a restart to rotate. A tenant’s secrets — clickhouse.password, auth.jwt_secret — are keys here, beside the user and the verifier they belong to, so they reload with them; validation warns, at boot and on every reload, when config.json carries one of them but is world-readable (see Deployment — Persistent Storage). See Configuration. Everything else lives here and reloads.

Deduplication

Every per-tenant dedupe knob lives here. Where the seen ids are kept (dedupe.backend) and how long a claim is held (dedupe.lease) are boot config, the same for every tenant. The switch and its fields are resolved per record from one snapshot (table override → global value):

dedupe.enabled (seed default false) — turns deduplication on. Hot-reloadable: a reload that flips it opens or closes this tenant’s store (in the embedded Pebble instance at <data_dir>/pebble, or its share of the DynamoDB table under dedupe.backend: dynamodb), so no restart is needed; seen ids persist across an off/on cycle. If the store fails to open on a reload, the failure is logged and ingest fails closed (503 dedupe store unavailable, Retry-After: 5) until it opens — the files asked for dedupe, so publishing un-deduped is not a fallback. With dedupe.backend: pebble that is the next reload or restart, and at boot a failed open over a flat directory refuses to start, like every other store; with dynamodb it is the background retry described below. A record that lands in the instant of the flip itself is published un-deduped: if the settings already say on but the store is not yet open, it’s counted by wavehouse_ingest_dedupe_disabled_total; in the reverse case (settings already say off, store still open) the handler skips dedupe like any other disabled record and nothing is counted. That counter should only ever tick during a reload, so a steadily climbing rate means the store and the settings have come apart. Over a nested directory with dedupe.backend: pebble, every tenant’s seen ids live in that one instance, each key led by its tenant and table, and it is open while any tenant’s switch is on: each tenant’s store follows its own folder’s dedupe.enabled the same way; a tenant’s seen ids are never another’s; a rejected or removed folder closes its tenant’s store and keeps its seen ids for the folder that restores it; and if that instance fails to open, at boot or on reload, every tenant with dedupe on fails closed — its ingest answers 503 dedupe store unavailable (Retry-After: 5) until a reload opens it — while the tenants with dedupe off carry on. Under dedupe.backend: dynamodb the table check plays the instance’s part, in either shape: the table is checked whether or not any tenant’s switch is on, and a table that fails it fails every tenant with dedupe on closed until the check, retried in the background and at once after every reload, passes. Only a misconfigured table (missing, the wrong key schema, access denied) over a flat directory whose tenant has dedupe on refuses boot instead (Configuration).
dedupe.id_field (seed default event_id) — JSON field name in the ingest body used as the dedup key. An id is a duplicate only within its own tenant and table: the same value in two tables is two ids. An id longer than 1,024 bytes once escaped (every byte but an ASCII letter, digit, _ or - takes three) is stored as its SHA-256, counted by wavehouse_dedupe_hashed_id_total. While its record is being published, an id is held for its lease (dedupe.lease, 30 seconds by default): another request carrying the same id meanwhile gets 503 (a request with the same dedupe id is in flight) with the lease, in whole seconds, as Retry-After — see the ingest errors. An id is committed only after its record is published; if that commit fails (counted by wavehouse_ingest_dedupe_commit_failed_total, which should stay at zero), the record is still answered ok and the id lapses with its lease: a retry of it before then answers in-flight, one inside the ingest queue’s two-minute duplicate window is dropped there by its idempotency key, and one after that is stored again.
dedupe.require_id (seed default false) — controls what happens to a row missing id_field, or carrying it as null (which can’t be deduped, so idempotency wouldn’t apply to it). Such a row is always logged at WARN and counted by wavehouse_ingest_dedupe_missing_id_total, in both modes. false: it is then published un-deduped. true rejects it instead (400 for a single insert; a per-record failure in a batch) — a server-side tripwire for producers that must guarantee the id.
dedupe.retention (optional; seed default "0") — how long a committed id stays a duplicate, as a Go duration string: "24h", "720h" (30 days), "90m". There is no day unit. "0" keeps every id forever, which was the only behavior before this key existed, and a config.json without the key means the same. Once an id’s retention has ended, the next record carrying it is published as new. With dedupe.backend: pebble, a background sweep over the shared Pebble instance deletes the expired id: first about a minute after the instance opens (when the first tenant switches dedupe on), then hourly while any tenant keeps it on, counted by wavehouse_dedupe_swept_keys_total{reason="expired"}. With dynamodb, no sweep runs: the table’s TTL on ex deletes the item (Deployment). A finite retention must be at least "2m", the embedded ingest queue’s duplicate window (under mq.backend: nats, keep it at least the partitions’ duplicate_window too, which boot warns about but this file cannot check; see External NATS): every deduped record is published under an idempotency key derived from its id, so an id re-sent after a shorter retention would be claimed again and then dropped by the queue as a copy, while the client was told it was accepted. A retention below that is refused, not raised to the minimum; so are a negative value and anything that is not a duration, such as "30d", a number with no unit ("300" needs one: "300s"; "0" is the one exception), or a JSON number rather than a string. Hot-reloadable: a change applies to ids committed after the reload, and an id already committed keeps the expiry it was stored with.
dedupe.tables.<table>.{id_field, require_id, retention} — per-table overrides; each entry overrides only the fields it names and inherits the rest, so a table with no retention keeps the tenant’s (forever when the tenant sets none). A table can keep ids for a shorter time than its tenant, or for longer, or forever ("retention": "0") under a finite tenant retention.

ClickHouse

The clickhouse block is the connection wiring, password included. A reload applies it to every consumer (schema discovery, structured queries, pipes, /readyz, the ingest worker’s HTTP INSERTs, and the raw-SQL proxy — the HTTP-side ones re-read the target per request). A change to addr, database, username, password or the tls block moves the tenant to the pool of its new tuple, opened for it when no served tenant has that tuple; a change to max_open_conns or max_idle_conns resizes its pool; a change to http_port, http_scheme, headers, query_timeout or insert_timeout needs no new connection (the worker reads insert_timeout with the batch’s ingest limits at its first row, so a batch’s wait and its insert deadline always come from one document, and a reload applies to the next batch). A replaced connection stays open for the longest query_timeout among the tenants that were on it, so in-flight queries finish. The change is unconditional: the adopted settings are the authority, so an address that isn’t reachable is applied all the same and shows up where reachability already does — schema discovery retries and logs, /readyz fails, queries return errors — until the next reload fixes it. Two things are refused instead: a tls block whose certificate files cannot be loaded (unreadable, not PEM, or a cert_file and key_file that do not pair), and a pool that would put the process over the boot config’s clickhouse.max_total_conns — a max_open_conns raised above it keeps the pool at its size, and a new connection tuple that would cross it is not opened. An option the ClickHouse driver refuses to open is refused the same way, though validation already excludes every one it knows of. A refused tenant keeps the pool and wiring it had until the next reload, or, newly served with no pool to keep, stays on none and its queries answer 503. The reload itself still reports adopted — the refusal is an ERROR log line, not a finding — and the next reload retries it. At boot it depends on the directory’s shape, as for the redelivery timeout: over a flat directory either refuses to start; over a nested directory the refused tenant is served on no pool, logged at ERROR and counted, and every other tenant boots — so a refused value left in the folder stops no restart, but costs tenants their pools until a reload fixes it: the folder’s own for tls files, and under the ceiling each, in id order, whose pool no longer fits, which can be other folders’ (a raise refused for acme can leave globex on none after a restart). Validation checks shape only (host:port, port range, scheme, non-empty database and user, timeouts >= 1, the tls block’s shape without opening its paths, header names and values, pool sizes); reachability is a runtime concern, so wavehouse validate needs no ClickHouse and no certificate files. At boot the address is dialed lazily, as before: an unreachable ClickHouse degrades /livez and retries rather than refusing to start.

TLS. There are two hops and two switches: tls.enabled puts the native-protocol connection (addr) on TLS, and http_scheme: "https" does the same for the HTTP interface (http_port). The rest of the tls block — the authority bundle, a client certificate, insecure_skip_verify, server_name — applies to whichever hop uses TLS, so a ClickHouse behind a private authority needs ca_file once for both. The certificate files are read when a pool opens: at boot, and on a reload that names a tuple no open pool has — a changed tls block, addr, database, username or password. A reload that changes only http_port, http_scheme, headers, query_timeout, insert_timeout or the pool sizes reuses the material already loaded, so a file replaced in place is picked up the next time a pool for its tuple opens, or by a restart. Both switches move the hop to ClickHouse’s TLS listeners, so the ports move too: addr to the secure native port (9440 by default) and http_port to the HTTPS one (8443), as the example above does. Validation warns when only one hop is on TLS: a plaintext HTTP hop carries the credentials on every insert and raw-SQL query, and a plaintext native hop sends the password in its handshake. In the container images these are container paths: bind-mount the bundle (-v /srv/clickhouse-ca.pem:/etc/wavehouse/clickhouse-ca.pem:ro) and make it readable by UID 65532, like the settings directory.

Headers. clickhouse.headers rides on every HTTP-interface request: routing or identification metadata for a proxy or gateway in front of ClickHouse. Values are stored in the file as written, so a credential placed there is only as protected as the settings directory itself — which already carries the tenant’s secrets (clickhouse.password, auth.jwt_secret), so protect the directory accordingly (see Deployment — Persistent Storage). WaveHouse’s own headers — Content-Type and the X-ClickHouse-User / X-ClickHouse-Key credentials, and on a request WaveHouse traces, traceparent and tracestate (see Following one request or one tenant) — are set after them and win. Naming X-ClickHouse-User or X-ClickHouse-Key is a validation error, and so is Authorization: ClickHouse’s HTTP interface reads it as Basic credentials, a second and conflicting credential path, so a gateway that authenticates that way needs a header name of its own. The native protocol carries no headers.

Per tenant. Over a nested settings directory each folder’s clickhouse block is that tenant’s own. The process opens one native connection pool per distinct addr, database, username, password and tls tuple among the tenants being served — ClickHouse authenticates per connection, so tenants with different credentials never share one — and tenants naming the same tuple share one pool, sized to the largest max_open_conns and the largest max_idle_conns among them. http_port, http_scheme, headers, query_timeout and insert_timeout stay each tenant’s own whatever it shares: the HTTP hop is addressed per request and the deadlines are per query and per insert. A reload reconciles the pools: a tenant whose tuple changed moves to the pool that tuple names, opened if it is new; a pool no tenant names any more closes once the longest query_timeout among the tenants it had has passed; a pool whose largest ask changed is resized the same way. The boot config’s clickhouse.max_total_conns bounds the open pools together: a resize above it is refused and the pool keeps its size, and a tuple that cannot be opened — the ceiling, a tls block whose files cannot be loaded, or options the driver refuses — leaves its tenants on the pool they had, or on none for a tenant that had none, at boot and on a reload alike (at boot the tenants are placed in id order, so under the ceiling it is a later folder that is refused, never an earlier one that fit; on a reload the pools already open count first, so the folder refused is the one whose new pool, move or raise would cross the ceiling, wherever it sorts), and every other tenant boots. A tenant on no pool answers 503 with Retry-After on every route that reaches its ClickHouse until a reload opens one. The refusals are ERROR log lines, not findings, and the next reload retries; only a flat directory refuses to start on one, as above. Schema discovery is per tenant too: each tenant’s tables are discovered from its own database over its own pool, on its own schema.refresh_interval, with the first periodic refresh at a random point within the interval so tenants adopted together do not refresh together; until a tenant’s first discovery, its table lookups answer 503 with Retry-After rather than 404 — the table may well exist.

Moving to a different ClickHouse address or database starts the tenant’s schema discovery over with the reload, not at its next schema.refresh_interval: its table lookups answer 503 with Retry-After until the new database’s tables are discovered, retried with backoff while it cannot be reached, and are never answered from the previous database’s schema. A tenant whose address and database did not change keeps its schema. Events already accepted stay in the queue, and the worker’s next flush uses the new target.

Authentication

The auth block is the verifier wiring, HMAC secret included; only the operator key is boot config. jwks_url (absolute http(s) URL, or ""), jwt_secret and role_claim reload live: a reload that changes any of them (the jwt_secret only while jwks_url is empty — a JWKS tenant ignores it, so editing it there keeps the verifier it has) builds a new verifier and swaps it in atomically — a rotated jwt_secret takes effect on the next request, and tokens signed with the previous one stop validating — so a request never sees a JWKS key source paired with the HMAC algorithm allowlist; one that changes none of them keeps the verifier it has. The rebuild does not depend on the JWKS being fetchable (the adopted wiring wins), and neither boot nor a reload waits on the endpoint: the key set is fetched in the background, and until a fetch has succeeded the verifier is pending — a request carrying a token answers 503 {"error": "token verifier not ready: the tenant's JWKS has not been fetched yet"} with Retry-After: 30, never an evaluation under the policy default_role that could accept its data under a lesser role, while a request carrying no token is unaffected and served as the default_role. Each failed fetch logs jwks refresh failed; no token validates until it succeeds with the URL and the cause; once a set has been fetched, a later failed refresh keeps that set in use, tokens go on validating against it, and the line reads jwks refresh failed; the previously fetched key set stays in use instead. Until the first fetch succeeds it is retried with backoff — one second, doubling to a minute — so an endpoint that comes back is picked up within a minute, without a reload; a fetched set is then refreshed hourly and, rate-limited to one per five minutes, on an unknown key id. While that limiter is closed, a token naming a key id the set does not hold is refused as invalid at once (the request falls back to the policy default_role like any other unverifiable token), unless the next refetch is under a minute away, in which case it waits for it (at most a minute) and is then checked against the newly fetched set. A response over 1 MiB (1048576 bytes) is refused as if unreachable, with jwks response exceeds 1048576 bytes as the cause — a real key set is a few hundred bytes per key, so if a provider’s set ever grows past that, this log line is the symptom to look for. Switching between HMAC (jwt_secret set, jwks_url empty) and JWKS is a file edit. Over a nested directory each tenant’s folder wires its own verifier, HMAC secret included; the operator key is boot config, shared by every tenant.

Dead Letter Queue

A batch insert ClickHouse rejects is retried row by row; a row ClickHouse rejects again on its own is a poison row. A ClickHouse that cannot take the insert at all — down, unreachable, overloaded, read-only, refusing the credentials — makes no poison rows, whatever this switch says: the batch stays in the ingest queue and is retried with backoff (see Ingest Pipeline). A batch whose tenant has no ClickHouse connection — one no longer served, or one no pool could be opened for (such as by the connection ceiling) — skips the row-by-row retry, which no row of it could pass, and every row of it is a poison row. dlq.enabled (seed default true) decides what happens to it, resolved per table (dlq.tables.<table>.enabled → global) at the moment of the failure, so a reload applies to the next poison row:

true — the row is published to the tenant’s dead-letter stream (DLQ_{tenant}) under dlq.{tenant}.{table} (0 for a directory that holds the four files) with the failure in its headers, and its original is acked. Inspect it with GET /v1/ops/dlq/stats (admin-only; ?tenant= names the tenant).
false — the row is left unacked, so NATS redelivers it and it retries until it inserts or the switch is flipped back. For every row the worker can read, nothing is ever dropped either way — the choice is park it versus keep retrying. One exception: an envelope the worker cannot read at all — malformed JSON, an unknown format, or columns and row that do not pair — can never insert, so redelivering it forever would wedge the consumer. With the DLQ off for the table it is acked and dropped, logged at ERROR and counted by wavehouse_ingest_poison_total with disposition="dropped" (also labeled by table and reason; an envelope parked on the DLQ carries disposition="parked"). See Ingest Pipeline.

For a tenant no longer served — its folder removed or rejected — there is no switch to read: its rows are always parked, so none of them sits unacked in its ingest queue, redelivered for as long as the tenant is away and stopping the Active Sweeper purging that queue.

A tenant’s dead-letter stream is opened when the tenant is first served (an empty stream costs nothing) and the stats endpoint is always registered — the switch is purely behavioral, which is what makes it safe to reload.

Ingest

The ingest block is how the worker batches this tenant’s tables on their way to ClickHouse: each table’s batch is flushed once its rows’ JSON reaches ingest.max_batch_bytes, or ingest.max_batch_wait_ms after the batch’s first row, whichever comes first, a flush asked for while the table’s previous one runs starting when it ends (see Ingest Pipeline). Both are read at each batch’s first row, so a reload applies to the next batch of every table, and a batch is never sized by one document and timed by another. The right values depend on the tenant’s row width and rate, which is why they are the tenant’s; the consumer the batches come through — the redelivery timeout, the rows delivered and unacked, the prefetch — is boot config, sized once per process (Message Queue).

ingest.max_batch_bytes (seed default 262144, 256 KiB) — the byte target: the sum of the batch’s rows’ JSON (each queued envelope’s row, one JSONCompactEachRow line), an estimate of the uncompressed INSERT body, and so of the part ClickHouse writes for it. A “try to flush” threshold rather than a cap: rows that arrive while an insert is in flight coalesce into the next batch, which can pass it. A batch can never hold more rows than the queue has delivered and not yet acked: under embedded the tenant’s mq.max_ack_pending (100,000 by default); under mq.backend: nats the shard’s share of it: an even share over the shards the process holds (about 3,000 rows each at the defaults for a process holding 32 shards), never under twice mq.prefetch (or mq.max_ack_pending, if that is smaller), so 1,000 rows by default once a process holds more than a hundred shards. A target above that many rows of the table’s typical width is only ever reached by the timer, so raise the bound with the target: mq.max_ack_pending (under nats with the durables regenerated, wavehouse mq manifests --max-ack-pending), and under nats also mq.prefetch once its floor is what binds. The bound is shared — under embedded by all the tenant’s tables, under nats by the tables on the shard — so one busy table reaches the seed well before it (by a few hundred rows of an ordinary table, or a few thousand narrow ones, around 50 bytes each), but tables busy together each hold a batch short of the target, and once their batches together hold the bound (under embedded at the defaults, about 19 tables of 50-byte rows, 38 of 100 or 76 of 200) the queue delivers nothing more until a timer flushes one of them: the tenant is written at most about mq.max_ack_pending rows per ingest.max_batch_wait_ms, 20,000 rows a second at the defaults, however fast rows arrive. Under nats the share binds even one table at a time: at the defaults a process holding 32 shards gives each about 3,000 rows, so rows under about 85 bytes of JSON never reach the seed and flush on the timer, and past a hundred shards the 1,000-row floor leaves rows under about 260 bytes there; a shard at its share fetches almost nothing more while its batches wait (one row per 5 s pull renewal), so its tables are written at most about one share per ingest.max_batch_wait_ms, some 600 rows a second at 32 shards and 200 at the floor. To write faster, lower ingest.max_batch_bytes under each busy table’s part of the bound times its row width, or raise the bound as above; the bound is a count of rows, so the memory a stalled tenant can hold in the worker grows with it, at its rows’ width (#957).
ingest.max_batch_wait_ms (seed default 5000) — the time after the first row of a table’s batch at which the batch is flushed, in milliseconds (>= 1); a flush asked for while the table’s previous insert is still running starts when that insert ends and then runs its own, a wait the redelivery-timeout rule below does not count. The timer is the latency floor of a quiet table, and what a /v1/query right after an ingest waits for. A burst on one table that stops short of ingest.max_batch_bytes waits this out before it is written, the trade of sizing batches by bytes with no row trigger beside it; the retune of the target, and the budget-aware trigger that would flush a tenant’s batches as its held rows near mq.max_ack_pending, are #779.
ingest.retry_backoff_max (seed default 30) — the cap, in seconds (>= 1), of the backoff under which a ClickHouse that cannot take this tenant’s inserts — down, unreachable, overloaded, read-only, refusing the credentials, or past clickhouse.insert_timeout — is retried: the delay doubles from one second up to it, each jittered down to half, and rows handed back to the queue wait it out there (Ingest Pipeline). The backoff is per ClickHouse pool and per failing table; on a pool tenants share, each window opens under the cap of the tenant whose flush met the failure and is cut to a smaller cap by any tenant whose rows meet it meanwhile, so no tenant waits longer than its own cap, one with a larger cap retries sooner than it asked, and a reload that raises a cap applies at the next failure. Read with the batch’s limits at its first row, so a reload applies to the next batch.

The redelivery timeout. A batch’s wait plus its insert — ingest.max_batch_wait_ms plus clickhouse.insert_timeout — must stay under the queue’s redelivery timeout, the boot config’s mq.ack_wait (a minute by default), or the queue redelivers the rows of a batch that is still inserting and they insert twice. The rule bounds a batch’s own wait plus one INSERT, the common case: a batch queued behind its table’s in-flight flush, a flush of several column lists (a schema change mid-batch) and the row-by-row isolation of a rejected batch each add INSERT deadlines on top, so keep the sum well under mq.ack_wait rather than just under it, as the seeds do (35 s against 60 s); a row the queue redelivers mid-flush inserts twice, as a retried insert can (at-least-once). Boot config is what the two must fit, so wavehouse validate cannot check it, the way it cannot check dedupe.retention against the partitions’ duplicate_window: the ingest processes check it when they adopt the tenant’s settings. At boot it depends on the directory’s shape, as for a queue or dedupe store that cannot open: over a flat directory it refuses to start, naming both values; over a nested directory the folder’s tenant is served with no batch limits, logged at ERROR and counted, and every other tenant boots. Either way a reload that refuses them leaves them in the files, so fix them before the next restart, which would refuse to start or serve that tenant with none. On a reload the tenant’s new ingest and insert_timeout values are refused and the ones it had stay in force, logged at ERROR and counted by wavehouse_settings_reconcile_failures_total{component="ingest"} (Settings reloads), and the next reload retries; POST /v1/ops/settings/reload still answers "adopted": true, since the refusal is not a finding, but wavehouse_settings_reloads_total counts the reload partial. A tenant adopted with values that fail the rule, and none to keep (a folder refused at boot over a nested directory or new to one, or one served again after it was rejected or removed, whose earlier limits were dropped when it stopped being served), has no batch limits until a reload adopts valid ones. Its ingest keeps answering 200, so the worker parks nothing of its: it hands each of its rows back to the queue for one mq.ack_wait and takes it again, counted under wavehouse_ingest_retries_total{reason="unconfigured"} (Ingest Pipeline), and the rows insert on the first batch after the reload that fixes the folder; meanwhile its queue fills toward its mq.max_bytes_gb (under nats, its partition’s limits), as during an outage; under nats its rows, handed back with a delay, keep their slots in the shard durable’s max_ack_pending as a refused insert’s do, so once they fill it every table in that shard stops, other tenants’ included (Limits that differ from the embedded queue).

Message Queue

mq.max_bytes_gb (seed default 50) — not applied under mq.backend: nats, where the partition streams’ limits are the operator’s; everything below is the embedded queue. The disk budget for the tenant’s embedded JetStream ingest stream (INGEST_{tenant}), which buffers its ingested events until the worker writes them to ClickHouse; its dead-letter stream (DLQ_{tenant}) gets a tenth of it. Each tenant’s pair of streams is its own, opened when the tenant is first served and kept, at the budget it last had, when its folder is rejected or removed. The ingest stream runs DiscardNew, so when it’s full new publishes are rejected and POST /v1/ingest returns 503 for that tenant alone — backpressure by construction. A reload updates both streams’ limits in place without touching what’s buffered: growing takes effect immediately; shrinking below what’s currently on disk makes the ingest stream refuse new publishes until the Active Sweeper purges it back under the limit — what it purges is what is both written to ClickHouse and past the tenant’s stream.gap_window_minutes, and nothing already accepted is dropped — and a dead-letter stream holding more than a tenth of the new budget is kept at what it holds rather than shrunk, since shrinking it would delete its oldest parked rows; that is logged, and the stream then makes room for each new row by dropping its oldest, as a full one always does. If NATS rejects the update, the rest of the reload is still adopted, the failure is logged, and the next reload retries it. A queue NATS will not open at all refuses boot, like every other store; over a nested directory it costs that tenant alone, at boot or on reload — its ingest answers 503, each reload trying the queue again, and so does a publish, at most once every five seconds — while every other tenant carries on. A queue that opens while the server runs but that a consumer — the ingest worker’s or the stream hub’s — cannot join is treated the same way: it costs that tenant alone, its ingest answering 503 until a reload or a publish joins the consumer, so no row is accepted that the worker would not write or the tenant’s GET /v1/stream connections would not see. The two streams are resized as a pair: a failed DLQ resize undoes the ingest one so both stay on the previous budget, but if that undo fails too the ingest stream keeps the new limit and the DLQ the previous one until a later reload succeeds — the log line says which happened.

Sizing the volume. Nothing checks the budget against the disk — neither one tenant’s nor what the tenants’ add up to (#138) — so keep within the free space of the <data_dir>/nats volume every tenant’s budget plus its dead-letter stream’s cap: a tenth of the budget, or what the stream held when a smaller budget arrived, if that is more. Count every tenant ever served on the volume, not only those served now: a rejected or removed tenant’s queue is kept and nothing deletes it, so what it holds goes on holding disk — a rejected tenant’s replay history, and the rows parked on either one’s dead-letter stream, up to that cap. A disk that fills before a budget does fails every tenant’s writes, not one: the failed write is logged, the publish goes unanswered until it times out, and ingest answers 500 (publish failed) for every tenant on that volume, not the 503 with Retry-After of a full budget. It does not clear on its own: a stream that failed a write refuses every later one until WaveHouse restarts, so free the space and then restart.

Streaming

GET /v1/stream keepalives, gap-fill and the outbound queue are tuned here:

stream.keepalive_interval (seed default 30) — seconds a quiet connection may go without a write before the server sends a : keepalive comment. It exists to stay under whatever idle timeout sits between WaveHouse and the client; the default clears the common 55–60s proxy windows with margin, and a tighter edge (Azure Application Gateway 20s, CloudFront 30s) wants a lower value — see Behind a reverse proxy → Idle timeouts. A reload rebuilds the keepalive wheel in place: live connections stay open and are redistributed across the new ring, each getting at most one full new period before its next keepalive.
stream.keepalive_buckets (seed default 3) — spreads the keepalive writes across the interval (one bucket fires every keepalive_interval ÷ keepalive_buckets) so the server nudges ~1/N of connections per tick instead of all at once. It changes only how the writes are spread in time, never the period. At most 100: the wheel ticks once per bucket per interval and holds a set per bucket, so the cap bounds each tenant’s tick rate and ring, and a hundred buckets already spreads the writes to 1% of the tenant’s streams per tick.
stream.gap_window_minutes (seed default 15) — minutes of already-written-to-ClickHouse history the Active Sweeper keeps in NATS so a reconnecting client’s Last-Event-ID replay can bridge the gap; a drop longer than this resumes with a hole. Bounded by the tenant’s disk budget, mq.max_bytes_gb. A reload applies from the next sweep (every minute). Under mq.backend: nats the history stream’s max_age decides instead, and the API processes warn once per tenant and window about a tenant whose window is longer.
stream.subscriber_queue (seed default 64) — each connection’s outbound queue, in frames: how many event frames may wait for a client that reads slower than its table’s events arrive before the frames that do not fit are dropped, counted and logged as dropped SSE frames. Each connection allocates its queue when the stream opens, about 40 bytes a slot; a slot references its frame, serialized once per table and role and shared by that role’s connections, but a client that falls behind keeps up to stream.subscriber_queue event frames in memory until it reads them, so a large queue raises the worst case by that many frames per lagging connection. At most 10000, so no one connection can make the process allocate more than that; a client that far behind is better served by the reconnect below than by a deeper queue. A client that falls behind by more than the queue loses the frames that did not fit: its Last-Event-ID moves past the gap, so only a reconnect with ?since= recovers them (dropped SSE frames; #152 tracks measuring the right size). Read as each stream opens, so a reload applies to connections opened after it.