Finish a delivery the daemon was restarted in the middle of - #1174
Merged
Conversation
The task waiting on a service turn lives exactly as long as the daemon process. A restart takes it with it, silently, and whatever the channel had already placed for that delivery becomes permanent: a placeholder that will never be replaced, a reaction that will never be settled. No timeout fires, because the thing that would have timed out is gone. This is not rare. Restarts happen on upgrade, on config change, and on operator command, and a channel accepts deliveries throughout. Every restart landing mid-turn stranded one. Record a delivery when it is accepted and keep the record until something has actually been said about how it ended. Alongside it goes a blob the channel owns — for Slack, where the thread is and what has been posted into it — kept opaque here so a second kind of channel does not force a schema change on state they share. On starting a channel, finish what a previous daemon left outstanding before taking anything new. The harness outlives the daemon and gets reattached, so an interrupted turn is usually still running: pick the wait back up rather than declaring it lost, on what is left of the original allowance so surviving a restart cannot extend a turn's life. Only a delivery whose session is gone, or whose allowance is spent, is reported as over. Records are left in place while being resolved rather than consumed on read. A daemon interrupted again mid-recovery then finds them a second time: resolving one twice edits the message already there, while losing one strands it forever.
edwin-zvs
force-pushed
the
service-delivery-survives-restart
branch
from
August 2, 2026 19:07
c3d67c3 to
f7bfa9d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
A Slack thread was left showing
_Working on it…_with a 👀 reaction, permanently — no answer, no failure, no timeout.The waiter for a service delivery is a
tokio::spawninprocess_delivery. It lives exactly as long as the daemon process. When the daemon restarts mid-turn, that task disappears, and with it the only thing that was ever going to replace the placeholder. Nothing times out, because the thing that would have timed out is gone.The daemon restarted 5 times in the two hours around the incident that prompted this. Every restart landing mid-turn strands a delivery this way, and nothing in the system notices or recovers.
What this changes
A delivery is recorded when accepted and cleared only once something has been said about it. The record lives in the service's existing state file and carries what a later daemon needs — session, transcript cursor, submission time — plus a channel-owned context blob. For Slack that's the thread, the message being answered, the placeholder ts, and whether a reaction was placed.
The context is opaque to the ingress on purpose: the ingress routes and waits, it does not render, and a second kind of channel must not force a schema change on state they share.
The affordance records itself as it appears. A placeholder only exists once a turn has run long enough to deserve one, which is well after acceptance — so
run_affordanceamends the record the moment it posts. Anything placed and not recorded is exactly what a restart would strand.On channel start, outstanding deliveries are finished before new ones are taken. Ahead of the socket, so a reconnect can't race a resumed turn for the same thread:
resume_outstandingrebuilds the receipt and re-registers the delivery id — which the reply tool authorizes against, and which did not survive the restart either.wait_for_finalnow takes a budget. A resumed delivery passesTTL - elapsed, so restart loops can't extend a turn indefinitely.Records survive being read. A daemon interrupted again mid-recovery finds them a second time. Resolving one twice edits the message already there; losing one strands it forever. That asymmetry decides it.
process_deliveryis split into accept →resolve_delivery, with the latter shared by fresh and resumed deliveries —already_waitedis zero for the first kind.Effect
A restart mid-turn goes from "the thread is stuck forever" to, in the common case, nobody in the channel learning a restart happened at all.
Spec
0182-an-accepted-delivery-outlives-the-daemon(new).Testing
cargo test --workspacegreen (30 suites, 0 failures).New coverage: a record persisting across acceptance → amendment → clear; the record actually reaching disk (an in-memory one is lost by the exact event it exists to survive); a resume refused when the session is gone; and per-channel isolation, including two channels of one service using the same request id without colliding.
This PR touches
crates/daemononly → the relevant binary isconstruct.🤖 Generated with Claude Code