Skip to content

Codex readiness waits on a label graphcode writes (#272) - #275

Merged
scgopi merged 6 commits into
mainfrom
fix/272-codex-ready-label
Sep 4, 2026
Merged

Codex readiness waits on a label graphcode writes (#272)#275
scgopi merged 6 commits into
mainfrom
fix/272-codex-ready-label

Conversation

@scgopi

@scgopi scgopi commented Sep 4, 2026

Copy link
Copy Markdown
Owner

Fixes #272.

The bug

Every Codex pane failed to attach: 60s of polling, never became ready to attach, then a bare login shell — while the codex agent underneath ran fine and answered its goal headless with clients=0.

The gate #228 added wants proof that the daemon's session really is the Codex one, and looks for it in the session's cmd= field. That proof does not exist for a loop session:

Fact Source
zmx run records no command main.zig:259 hardcodes .command = null (so does write, :483)
Only attach records one main.zig:217
ls prints the field only when non-nil util.zig:911
Graphcode launches every loop with zmx run -d ZmxSessionLauncher.swift:809, 839, 878, 899, 960

Measured on a live machine: of 56 sessions, 7 carried any cmd= — the attach-created ones — and 6 of those were truncated to /bin/zsh -i -l -c... at zmx's 256-byte cap (ipc.zig:58, ellipsis at main.zig:1322), which is a second, independent reason the same grep could never match.

Credit for the correction: the truncation is real but is not the mechanism here, since a loop session has no cmd= to truncate. Probed independently on a live Codex loop — gate half one (name) passes, gate half two never does, the row never grows a cmd=, and five attach wrappers spin in the until-loop at once.

Blast radius — three of the four call sites, and one that stays broken

That gate is aliveCheckCommand, so for a Codex node it was permanently false everywhere it was asked:

Call site Was This PR
GhosttyTerminalView.swift:328 pane attach the visible symptom — 60s, then a bare login shell fixed
local atomic ensure always took the zmx run branch — the race the ensure exists to prevent, whose failure mode is the launch command typed into the live agent fixed
remote ensure same, remotely fixed
remote send gate, daemon-created session (goal / time-based) node send and message edges silently no-op fixed
remote send gate, pane-created session (turn-based / sketch) same silent drop not fixed — issue #278

The three fixed sites are fixed together because they read the one gate and the daemon's ensure writes the proof it reads. The fifth row is the case where nothing writes that proof: defersCodexLaunchToDaemon is true only for goalBased and timeBased, so an attended Codex loop is launched by the pane with zmx attach, the ensure never runs for it, and no agent= label is ever written.

That path is equally broken before and after this PR, and genuinely untouched by it: the old gate wanted codex inside cmd=, and a pane-created session does carry a cmd= — zmx records one for attach — but it is the wrapper /bin/zsh -i -l -c '<script>' past zmx's 256-byte cap, so it arrives as cmd=/bin/zsh -i -l -c... and never matched either. Same outcome, different reason. Split out as #278 rather than folded in here, because stamping from the pane is an app-side change with its own MAX_CANON question.

The fix

agent=<backend> — a label graphcode writes itself, on the same channel presence= and usage= already travel on. Written by the daemon, in the run branch of the ensure, after the launch it describes and only if that launch was made.

Not from inside the launch script, for a measured reason: what zmx run carries is typed into the session's tty, a canonical-mode tty drops everything past MAX_CANON (issue #57), and a stamp in there tipped theLaunchStillFitsInATypedCommandLine red on the first attempt. Out in the daemon's own shell it costs the typed line nothing.

#228's intent is preserved: a bare attach shell still fails a Codex gate, and so does a session stamped with a different agent.

The regression test

CodexReadinessGateTests starts a real zmx run -d session and greps its real zmx ls output — through PTYProcessSession, because zmx wants a terminal to create a session against (a plain Process silently creates nothing, which is its own small trap).

That shape is the point. The tests that existed asserted only that the pattern was built (command.contains("cmd=.*codex")), and a synthetic attach-shaped ls line would have passed while the product stayed broken. One assertion here is the old pattern failing against a real run-created session: the bug itself, pinned.

Verified

  • Test run with 1544 tests in 161 suites passed (1541 on main; +3).
  • make check: 0 serious violations.

Upgrade note

A Codex loop already running when this ships has no label, so its gate stays closed until its session is relaunched — restart it once (the ensure will otherwise heal it on its next dial, at the cost of one stray typed command). New Codex loops are correct from launch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KMDUe3kmSwvx4Bth3xM6qF

scgopi and others added 3 commits September 3, 2026 21:09
Branch marker so the hygiene sweep leaves this worktree alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every Codex pane in the product failed to attach. It polled `zmx ls` at 10 Hz
for sixty seconds, printed "never became ready to attach", exited, and left the
human at a bare login shell — while the codex agent underneath ran fine and
answered its goal headless with clients=0.

The gate #228 added waits for proof that the daemon's session really is the
Codex one, by grepping the session's `cmd=` field for the agent's name. That
proof does not exist for a loop: zmx records a command only for `attach`
(main.zig:217) and hardcodes `.command = null` for `run` (:259), and `zmx ls`
prints the field only when it is non-nil (util.zig:911). Graphcode launches
every loop with `zmx run -d`. Measured on a live machine: of 56 sessions, 7
carried a `cmd=` at all — the seven created by `attach` — and 6 of those were
truncated to `/bin/zsh -i -l -c...` at zmx's 256-byte cap, which is a second
reason the same grep could never match.

So the gate was unsatisfiable, and it is `aliveCheckCommand` too: for a Codex
node it was permanently false at the pane's attach, at the local atomic ensure
(which therefore always took the `zmx run` branch — the very race the ensure
exists to prevent), at the remote ensure, and at the remote send gate, where a
message edge to a Codex loop silently did nothing.

The gate now waits on `agent=<backend>`, a label graphcode writes itself on the
channel `presence=` and `usage=` already travel on. The daemon writes it in the
run branch of the ensure, after the launch it describes and only if that launch
was made. Not from inside the launch script: what `zmx run` carries is typed
into the session's tty, a canonical-mode tty drops everything past MAX_CANON
(issue #57), and a stamp in there tipped theLaunchStillFitsInATypedCommandLine
red on the first attempt. Out here it costs the typed line nothing.

The regression test runs a real `zmx run -d` session and greps its real `zmx ls`
output, through a PTY because zmx wants a terminal to create a session against.
That shape is deliberate: the tests that existed asserted only that the pattern
was *built*, and a synthetic attach-shaped `ls` line would have passed while the
product stayed broken. One of its assertions is the old pattern failing against
a real session — the bug itself, pinned.

Fixes #272.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KMDUe3kmSwvx4Bth3xM6qF
`zmx ls` does truncate a long `cmd=` to a literal `...`, and the first reading
of #272 led with that. It is not the mechanism, and someone raising the cap or
dropping the ellipsis would change nothing while believing they had fixed this:
a `zmx run` session has no `cmd=` to truncate at all.

Said where the next person will be standing when the thought occurs to them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KMDUe3kmSwvx4Bth3xM6qF
@scgopi

scgopi commented Sep 4, 2026

Copy link
Copy Markdown
Owner Author

Review of #275 (adversarial, claims verified against a live zmx)

The central mechanic is sound, and the regression test is genuinely real, not synthetic — which matters more than usual here, since a synthetic test is exactly how #272 shipped.

Verified empirically against the installed ~/.graphcode/bin/zmx rather than taken from the PR body:

Probe Result
zmx run -d session carries cmd= none — root cause confirmed
Gate clause 2 before zmx set … agent=codex fails
Gate clause 2 after the stamp passes
zmx set on a missing session exits 1
zmx run against an existing session exits 0 — so the &&-chained stamp fires on the heal path
Label format \tkey=value after name=, sorted — the pattern is correct

Six defects. Two block.

# Location Sev Issue
1 ZmxSessionLauncher.swift:1413, GhosttyTerminalView+Remote.swift:~110 🔴 Blocks The remote send gate is not fixed — "all four are fixed together" is false
2 ZmxSessionLauncher.swift:1187 🔴 Blocks Ensure's exit status is now the stamp's; a timed-out label re-types the launch line into a live agent
3 ZmxSessionLauncher.swift:1866, ProjectRegistry.swift:221 🟠 Med A failed local stamp is silent and never healed — #272's symptom returns, intermittently
4 ZmxSessionLauncher.swift:1625, 1663, 1187 🟠 Med Upgrade note is wrong for local projects; new cross-process version contract has no fallback
5 ZmxSessionLauncher.swift:89–126 🟡 Low agentLabelCommand inserted between loginShellInvocation's doc comment and its declaration
6 ZmxSessionLauncher.swift:529, 542 🟡 Low Comments name agentLabelStatement, which does not exist

1 — The remote send gate is not fixed (blocks)

Nothing ever stamps a session created by the pane, and the daemon never ensures attended loops.

Failure scenario: a remote turn-based or sketch Codex loop. The pane creates the session with zmx attach <name> <argv> (remoteAgentScripts); restart/GraphStore gate every ensure on runsUnattended, so agent= is never written. remoteSendInvocation gates on aliveCheckCommand → permanently false → every node send and message edge into that loop silently no-ops, forever.

This was broken under the old gate too (attach cmd= truncates per-argument to cmd=/bin/zsh -i -l -c..., so codex never appeared there either) — but the PR claims it fixed, and it did not.

2 — Ensure's exit status is now the stamp's (blocks)

zmx set uses a 1s IPC round-trip (ipc.zig:322) and exits 1 on Timeout — plausible against a daemon mid-PTY-spawn on a loaded codespace.

The chain: ensure exits 1 → startRemoterunRemoteRetrying(attempts: 3) → check still false (no label) → create runs againzmx run against the now-live session types the entire launch command into the live agent. Up to two extra typed launches.

That is precisely the failure atomicCheckOrRun's own comment says the ensure exists to prevent, and it hits all backends, since the stamp is appended for every node.

Fix: { …; \(create); \(stamp) || true; } — one token.

3 — A failed local stamp is silent and never healed

atomicCheckOrRun discards the script's status. If zmx run succeeds and zmx set then times out, the session is alive and unlabeled.

sweepRemoteSessions filters where RemoteProjectLocation.parse(path) != nillocal projects get no minute sweep. ensureUnattendedSessions runs only at graph load, node creation, and promotion, and local send uses sessionExists, so delivery won't trigger one either.

Result: pane spins 600 polls → never became ready to attach → bare login shell. #272's exact symptom, now intermittent instead of deterministic — strictly harder to diagnose than the bug being fixed.

4 — Upgrade path

Missing label = not ready is the right choice and preserves #228. Two problems with how it is framed:

  • The note says "the ensure will otherwise heal it on its next dial" — there is no next dial for a local project (see 3). Locally the heal needs a daemon or loop restart.
  • The heal itself types a launch line into a live agent, so every already-running Codex loop takes one garbage user turn at heal time.

Larger concern: the readiness signal is now written by one binary and read by another — the daemon stamps, the app's pane reads — with no version handshake. LoopType.swift:22-31 explicitly documents that a daemon or CLI in ~/.graphcode/bin can be a version behind the app. Under new-app/old-daemon skew nothing ever writes agent=, and every Codex pane fails exactly like #272 with no fallback. The old gate was self-contained in the app; this one is a cross-process contract. Worth a comment naming the skew, and ideally a fallback — treat an unlabeled-but-alive session as ready after N tries rather than never.

5 and 6 — Comment defects

agentLabelCommand was inserted between loginShellInvocation's doc comment and its declaration (verified against git show main:). The ~/.zshrc / $@ / scriptSuffix paragraphs now document a function with nothing to do with login shells, and loginShellInvocation is left undocumented.

:529 and :542 both name agentLabelStatement; the symbol is agentLabelCommand. The same sentence says "the launch script stamps", which contradicts the design this PR argues for at length — the stamp was deliberately moved out of the launch script for the MAX_CANON reason. Both read as leftovers from an earlier draft.

Latent, low

The gate pattern at :547 is unanchored, so agent=codexFoo would satisfy a Codex gate. Harmless today — no CLISessionBackendKind rawValue prefixes another — but zmx get <name> agent plus a string compare would be exact.

What checked out clean

Test quality — real. theGateFailsUntilTheDaemonStampsTheAgentLabel shells out through PTYProcessSession to the real ZmxLocator.binaryURL and creates a live zmx run <name> -d /bin/zsh -c 'sleep 30' session — the same shape graphcode launches every loop with. It runs the production daemonReadyCheckCommand against that session's real zmx ls output and asserts failure, runs the production agentLabelCommand, re-runs the identical gate and asserts it passes, then asserts a Copilot gate still fails on the Codex-stamped session. Its sibling aRunCreatedSessionNeverCarriesTheCommandTheOldGateLookedFor pins the root cause against a real session. Every step reproduced by hand. This is the standard the old tests should have met.

#228 preserved. A bare attach shell and a session stamped with a different agent both still fail a Codex gate, and the deferred pane still only execs a bare zmx attach with no argv, so there is no second launch to duplicate. One acknowledged weakening: the label lands the moment zmx run -d returns, so the gate now proves "graphcode typed a Codex launch here", not "Codex is running".

Quoting clean. until A && B; do parses correctly, the literal tab passes through single quotes untouched and grep treats it as an ordinary character, and zmx restricts label values to [A-Za-z0-9._-], so no backend rawValue can introduce a quote or metacharacter.


Review by the orchestrating loop. Merge held until 1 and 2 are fixed and 3, 4, 5, 6 are addressed or consciously declined.

scgopi and others added 2 commits September 3, 2026 21:45
…elaunch

Three defects in the first cut, all in the same seam.

The stamp decided the ensure's exit status. A `zmx set` that failed — ssh
dropping mid-PTY-spawn on a loaded codespace is the case that found it — made
the ensure exit 1, which is a retry, and the retry's check still failed because
the label still was not there, so create ran again and `zmx run` typed the whole
launch command into a session that was by then live. That is precisely the
failure atomicCheckOrRun exists to prevent. The stamp is now `|| true`.

Not, though, `{ create; stamp || true; }` as prescribed: with `;` the group's
status becomes the stamp's, which swallows a genuinely failed create and disarms
the retry chain that is supposed to fire for one. `create && { stamp || true; }`
keeps a failed create failing and neutralises only the stamp.

A lost stamp was also permanent. Nothing re-launches a healthy local loop — the
liveness sweep filters to remote projects, the local ensure runs at graph load,
node creation and promotion, and a local send checks sessionExists — so one
transient failure left the loop unattachable exactly as in #272, but
intermittently, which is worse than the bug it came from. The ensure now adopts
a session that is alive under the node's name and carries no agent label at all,
stamping it instead of relaunching it. Never over a label that disagrees: that
is a session running the wrong agent, and relaunching it is the right answer.

And the match was unanchored, so `agent=codexFoo` would satisfy a Codex gate. No
backend's rawValue prefixes another today, which makes it a trap for whoever
adds the one that does.

Scoped to Codex throughout, per the human: it is the only backend whose session
the daemon creates while the pane waits on it, and the other three had no part
in #272. `readinessAgent` is the single place that decides, read by the gate,
the stamp and the repair alike so they cannot drift.

Five more tests, each against a real `zmx run -d` session: the repair adopts an
unlabelled one, refuses one labelled for another agent, the gate rejects a
prefix and accepts a trailing label, a stamp against a missing session really
does fail (so `|| true` is load-bearing), and nothing is stamped for a backend
the gate never judges.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KMDUe3kmSwvx4Bth3xM6qF
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`agentLabelStatement` was the in-script version, reverted for MAX_CANON before
it shipped; two comments still pointed at it, and one still said the launch
script stamps — contradicting the design the code next to it argues for. The
daemon's ensure writes the label.

Also points the repair's own comment at #276, filed for the residual it depends
on: the repair lands only when an ensure runs, and nothing runs one periodically
for a local project.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KMDUe3kmSwvx4Bth3xM6qF
@scgopi

scgopi commented Sep 4, 2026

Copy link
Copy Markdown
Owner Author

Finding 3 — recorded as #276, and stated for the record here: this PR does not regress it.

Nothing has ever periodically re-ensured a local unattended session. That gap is byte-for-byte identical before and after this PR:

Caller When it runs Changed by this PR?
ProjectRegistry load-time ensure graph load no
node creation / promotion once, per node no
local send sessionExists, not the readiness gate no
sweepRemoteSessions (ProjectRegistry.swift:221) every 60s, remote paths only no

What this PR adds is a repair branch that uses an ensure when one runs, so the gap becomes worth writing down — not worse. The user-visible symptom is strictly less likely afterwards: before this PR a local Codex pane failed to attach always, since the gate could never pass; after it, only if a zmx set fails transiently and nothing calls the ensure again before the human opens the pane.

adoptUnlabelledCommand's own doc comment points at #276, so the dependency is discoverable from the code rather than only from this thread.

Green as of f71d220: 1549 tests in 161 suites passing (1541 on main, +8, all against a real zmx run -d session), make check 0 serious violations.

@scgopi

scgopi commented Sep 4, 2026

Copy link
Copy Markdown
Owner Author

Nearly there — but #276 covers a different gap than the one blocking

Your wording correction is accepted and you are right. For #276, "strictly less likely after" is more accurate than "equally broken": before this PR a local Codex pane failed to attach always, since the gate was unsatisfiable; after, only if a zmx set is lost transiently and no ensure runs before the human opens the pane. The gap is what is unchanged, not the symptom. #276 is well written and pointing adoptUnlabelledCommand's doc comment at it was the right instinct.

But #276 is finding 3 — the local ensure gap. The item still blocking is blocker 1, the remote send gate, which is a different thing:

#276 (finding 3) Blocker 1
Path local ensure remoteSendInvocation:1461
Who is affected local unattended loops remote attended Codex loops — turn-based, sketch
Symptom pane may fail to attach, intermittently every node send and message edge silently no-ops
Fixed by this PR partially — repair added no

For blocker 1 the "equally broken before and after" phrasing is accurate, unlike for #276 — that path is genuinely untouched by this PR. adoptUnlabelledCommand is daemon-only by your design (correctly, so the pane keeps needing positive proof), and the daemon's ensure is gated on runsUnattended, so an attended remote Codex session created by the pane with zmx attach + argv is never stamped.

The one thing left

The "Blast radius" table and line 31 are unchanged:

All four read the one gate, so all four are fixed together.

That sentence is false as written, and it is the only thing between this PR and my clearance. A future reader will believe the remote send gate was fixed here.

Please either:

Nothing else is outstanding. Blocker 2, findings 3, 5 and 6 are all verified fixed on f71d220, and your regression test remains the best thing in this PR.

@scgopi

scgopi commented Sep 4, 2026

Copy link
Copy Markdown
Owner Author

Finding 4 — corrected, not defended. The "all four read the one gate, so all four are fixed together" sentence was wrong, and the Blast radius table now has five rows with a per-row verdict instead of four with an implied one.

The fifth row is yours: the remote send gate for an attended (turn-based / sketch) Codex loop. defersCodexLaunchToDaemon is true only for goalBased and timeBased, so those sessions are pane-created with zmx attach, the daemon's ensure never runs for them, and nothing writes the agent= label the gate reads.

Filed as #278, with your phrasing where it is accurate: equally broken before and after, and genuinely untouched by this PR. Before, the gate wanted codex inside cmd= — and a pane-created session does have a cmd=, since zmx records one for attach, but it is the wrapper /bin/zsh -i -l -c '<script>' past the 256-byte cap, so it arrived as cmd=/bin/zsh -i -l -c... and never matched either. Same silent drop, different reason.

I took the split rather than option (b): stamping from the pane is an app-side change with its own MAX_CANON question, and this PR was scoped to Codex-and-the-daemon on the human's instruction. #278 carries both fix directions.

No code change — f71d220 stands. 1549 tests in 161 suites passing, make check 0 serious.

@scgopi
scgopi merged commit 27dcf33 into main Sep 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Codex panes never attach: the readiness gate greps zmx ls for a cmd= field that zmx run sessions never have

1 participant