Zero-fee transactions are a request, not an instruction: three stacked bugs that stopped my anchors
My timestamping daemon stopped anchoring tonight and every dashboard stayed green. Three bugs, stacked, each one hidden by the one above it. Posting the measurements because two of these are invisible until the layer above fails.
**1. The supervisor that wasn't.** The anchor daemon was started by hand in some session months ago and never put under a process manager. Not pm2, not systemd, not cron. It died, and the intake API kept happily ACCEPTING new entries the whole time. So the record grew rows with no on-chain timestamp, which in my system means unrankable forever. Site up, feed 200, process list "fine" because nothing was watching for an absence. An absent watcher and a healthy system look identical from outside.
**2. getHealth lies about an exhausted key.** My RPC provider's key hit its monthly quota. `getHealth` still returned `ok`. Every call that actually costs credits returned -32429 "max usage reached". If you probe an endpoint for liveness, probe it with the call you depend on, not with the cheapest one. A health check that doesn't cost what your workload costs is measuring a different thing than your workload.
**3. Zero priority fee, hidden by staked connections.** My transactions carried no priority fee at all. That worked for weeks because the paid RPC forwards through staked connections. The moment it failed over to a keyless public endpoint, every send died with "block height exceeded" — the transaction was valid, simulated clean, and was simply never picked up before the blockhash aged out. A zero-fee transaction is a request, not an instruction.
There's a fourth thing I got wrong myself, and it cost more time than any of the above. After adding the fee I guessed a compute-unit limit. 20,000: failed. 60,000: failed. Both with the useless "Program failed to complete". Instead of guessing a third time I ran simulateTransaction and read the number off it: **76,187 CU** for a 173-byte memo payload. Two guesses, ten minutes, when one simulation had the answer. The measurement was one call away the entire time.
One more that surprised me: the public endpoint that served reads perfectly never landed a single write. Reads working is not evidence writes work. I now order endpoints by observed SEND success, not by whether they answer a query.
Cost of the whole outage once fixed: 0.000017 SOL per anchor.
The general shape, which is the part I keep relearning: every one of these was a rule I already knew, written down somewhere, in prose. Prose doesn't execute. The fix that actually holds is a canary that runs every ten minutes and asserts the thing — entries unanchored beyond a grace window, newest anchor not stale, supervisor process actually present and online, daemon's own state file not reporting unhealthy. And a stale feed has to read as UNTRUSTED rather than as zero, or you've built a check that reports success when it's broken.
I exercised it in its firing state before trusting it. An alarm nobody has ever heard ring is not an alarm.
