fix(service): reopen a routing key whose session was deleted - #1160
Merged
Conversation
A service's routing table maps a routing key to the session serving it, but nothing prunes that table when a session is deleted — the operator can delete a routed session from any client, and the service is never told. The delivery path then resolved the key to a session id that no longer existed and propagated the resulting "session not found" error, leaving the stale entry in place. That bricked the key permanently: every later delivery failed the same way, and the only repair was hand-editing the service's state file. Resolve the key through a liveness probe instead. A routing entry whose session is gone is dropped, and the delivery falls through to the normal creation path, so the key opens a fresh session and keeps working. Both the canonical entry and any legacy bare-key duplicate are cleared, along with the ownership record, so a deleted id cannot later expose an unrelated session through the channel. This covers `single` routing as well as `session-key`: both resolve through the same keyed path, `single` under a fixed key. `per-event` never routed by key and was unaffected. Membership in the session registry is probed rather than a full session read, so a transient transcript or storage error is not mistaken for a deletion and does not abandon a live conversation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Delete a session that a service created under
session-keyrouting, and that routing key is bricked. Every later message from the channel with the same key fails instead of starting a new conversation.ServiceIngress::submit_trackedresolved the key to a persisted session id and went straight tomanager.detail(&id).await?. For a deleted session that returnsErr("session not found: <id>"), and the?propagated it out of the delivery. Nothing anywhere prunes the routing map —ingress.rsonly everinserts, and session deletion has no hook into the service supervisor — so the dangling entry survived, and so did the failure. The only repair was hand-editing<data_dir>/services/<service>.json.singlerouting had the identical bug: it resolves through the same keyed path under the fixed key__single__, so deleting the shared session broke the channel for every caller.per-eventnever routes by key and was unaffected.The fix
Resolve the key through a liveness probe, in a new
live_session_for_key. An entry whose session is gone is dropped and the delivery falls through to the normal creation path, so the key opens a fresh session and keeps working. Both the canonical entry and any legacy bare-key duplicate are cleared, along with the ownership record, so a deleted id cannot later expose an unrelated session through the channel. Pruning happens under the state lock the creation path already holds, so it stays atomic against a concurrent delivery on the same key.Liveness is
get_entry— registry membership — not the fulldetailread. A transient transcript or storage error is not evidence of deletion and must not abandon a live conversation, so that case still fails the delivery and keeps the entry.Tests
Four unit tests in
crates/daemon/src/service/ingress.rs, covering the keyed case, thesinglefixed key, the legacy bare-key entry, and an unrouted key. Verified non-vacuous: with the probe forced to the old always-reuse behavior, the three deletion tests fail and the rest pass.Full workspace suite is green.
pty_render::tests::smith_tool_expand_collapse_rebuilds_only_retained_suffixfailed on two loaded runs and passed on four unloaded ones — it assertsexpand_us < 80_000in wall-clock microseconds, so it is load-sensitive and unrelated to this diff (it also fails/passes the same way with the diff stashed).Spec
specs/0177-service-routing-keys-outlive-their-sessions.mdrecords the rule: the routing table is a cache, not a guarantee, and a dangling entry means "unrouted", not "broken".