Skip to content

Read Playbook run progress from the session that runs it - #1147

Merged
edwin-zvs merged 1 commit into
mainfrom
fix-playbook-run-owner-idle-race
Aug 2, 2026
Merged

Read Playbook run progress from the session that runs it#1147
edwin-zvs merged 1 commit into
mainfrom
fix-playbook-run-owner-idle-race

Conversation

@edwin-zvs

Copy link
Copy Markdown
Contributor

The flake

Two Playbook e2e tests fail intermittently and take main red with them (6 of the last 12 main runs):

  • playbook_view.rs::playbook_selection_fork_run_delivers_and_submits_promptassert!(annotated_block.shimmer)
  • playbook_shimmer.rs::playbook_stops_shimmering_when_the_owning_session_dies — the precondition poll, "Run never lit the executed region"

Both assert a block should be shimmering and find it is not, and both have the same cause.

Root cause

A run adopts session state transitions that have nothing to do with it.

A PTY harness announces Running when it spawns and idles again at its first prompt. A Run is armed before its prompt goes out (#1122), so when that boot pair is still in flight the run reads it as its own turn starting and ending — and settles its own shimmer milliseconds after arming it.

Traced against a real daemon (the failing e2e fixture, instrumented):

44.665248  ARM run   session=s984… refs=["b1:0"]
44.666149  NOTE state Running        present=(seen_running=true, …)  ← owner's boot Status(Running)
44.667221  NOTE state AwaitingInput  clear=true                      ← owner's first pgrp tick
44.695417  NARROW run decls=2 → "no run present"                     ← the annotation edit, 28ms later

The fork case is worse than a timing window. A run is keyed by the Playbook's session, but a fork Run's turn happens in the fork — so the owner, a bystander doing no work for this run, settled the fork's blocks just by finishing something of its own. That is a live bug, not only a test artifact: Run a selection into a fork, then keep working in the Playbook session, and the fork's shimmer disappears while the fork is still going.

The shimmer test failed the same way; it passed locally only because the client's first playbook.get beat the daemon's processing of that pair by ~1ms. On a loaded runner it loses.

The fix

A run now carries the session its turn actually happens in and the instant its prompt reached that session, and reads lifecycle signals only from that session, only after that instant.

The daemon's view of a session's state is advanced by the same in-order event drain that reports these transitions — so "the session was idle when we dispatched" is a statement about the event stream: everything emitted before that idle has already been consumed, and a Running after it can only be a new turn. When the session was not idle at dispatch, the turn already in flight isn't ours; the run's turn starts at the next Running after an intervening idle. That also fixes a Run queued behind a live turn settling when that turn ends.

The idle-without-a-turn backstop now runs from delivery rather than from arming, which matters for a fork whose prompt waits on harness readiness (spec 0149).

Rule recorded in specs/0176-playbook-run-tracks-its-execution-session.md.

Flake rates (this machine, 12 spinner processes of CPU contention)

fork Run test shimmer test
before (origin/main) 19/20 failed 5/20 failed
after 0/25 failed 0/25 failed

Without load, the fork test failed 10/10 on origin/main and 0/25 after.

Tests

  • cargo test -p construct-daemon --lib — 536 passed (3 new: pre-dispatch transitions ignored, fork run reads the fork's lifecycle, dead Playbook session still clears its run). Two existing tests were updated to encode the new rule rather than the old one.
  • cargo test -p construct-cli — 1321 passed
  • cargo test -p construct-protocol — passed
  • full cargo test --workspace — see comment below

No recording: this is daemon lifecycle logic with no new user-visible surface (the visible effect is a shimmer that stops vanishing).

Binaries

The PR touches crates/daemon, crates/protocol, and a CLI test fixture; the daemon is what runs this code, and it lives in the single construct binary:

/Users/moon/agentd/.claude/worktrees/fix-playbook-run-owner-idle-race/target/debug/construct

Both Playbook e2e regressions that keep `main` red assert that a block
should be shimmering and find that it is not. One cause: a run adopts
session state transitions that have nothing to do with it.

A PTY harness announces `Running` when it spawns and idles again at its
first prompt. A Run is armed before its prompt goes out (#1122), so when
that boot pair is still in flight the run reads it as its own turn
starting and ending, and settles its shimmer milliseconds after arming
it. Daemon traces from the failing test show exactly that: ARM at
+0.0ms, `Running` at +0.9ms (latches `seen_running`), `AwaitingInput` at
+2.0ms (clears the run), and the annotation edit 28ms later finds no run
to declare against.

A fork Run was worse. The run is keyed by the Playbook's session but its
turn happens in the fork, so the *owner* — a bystander doing no work for
this run — settled the fork's blocks just by finishing something of its
own, or by finishing booting.

A run now carries the session its turn actually happens in and the
instant its prompt reached that session, and only reads lifecycle
signals from that session after that instant. Since the daemon's view of
a session's state is advanced by the same in-order event drain that
reports these transitions, an idle observed at dispatch also means every
event emitted before it has been consumed — so a `Running` after it can
only be a new turn. When the session was not idle at dispatch, the turn
already in flight is not this run's: its turn starts at the next
`Running` after an intervening idle, which also stops a Run queued
behind a live turn from settling when *that* turn ends. The
idle-without-a-turn backstop now runs from delivery rather than from
arming, which matters for a fork whose prompt waits on harness
readiness (spec 0149).

Rule recorded in specs/0176. Local flake rate under CPU contention,
before → after: fork Run 19/20 → 0/25 failures, shimmer 5/20 → 0/25.
@edwin-zvs

Copy link
Copy Markdown
Contributor Author

Full cargo test --workspace on this branch (macOS, debug): 2332 passed, 0 failed across 48 test binaries, including both e2e suites that flake today.

Loop detail behind the table in the description — each iteration is a fresh cargo test invocation of the single test, with 12 busy-loop processes pinning the CPU:

fork Run test shimmer test
origin/main, 12x load, 20 runs 19 failed 5 failed
origin/main, no load, 10 runs 10 failed 0 failed
this branch, 12x load, 25 runs 0 failed 0 failed
this branch, no load, 25 runs 0 failed 0 failed

The fork test failing 10/10 locally with no load is the same race as CI's ~50%, just with the ordering pinned the other way: on this machine the owner's boot Running/AwaitingInput pair lands ~1ms after the run is armed, every time.

@edwin-zvs
edwin-zvs merged commit 43520f3 into main Aug 2, 2026
1 check passed
@edwin-zvs
edwin-zvs deleted the fix-playbook-run-owner-idle-race branch August 2, 2026 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant