Skip to content

Stop a busy port from stranding every session a home has - #1175

Merged
edwin-zvs merged 1 commit into
mainfrom
router-port-strands-sessions
Aug 2, 2026
Merged

Stop a busy port from stranding every session a home has#1175
edwin-zvs merged 1 commit into
mainfrom
router-port-strands-sessions

Conversation

@edwin-zvs

Copy link
Copy Markdown
Contributor

What broke

A live fleet, right now:

$ for pid in $(pgrep -f "construct __adapter"); do ps eww -p $pid | grep -o 'HTTPS_PROXY=.*127.0.0.1:[0-9]*'; done | sort | uniq -c
   4 64205
  67 8917

67 of 71 running adapters are dialing 8917, where nothing has listened since:

17:32:04  daemon restart requested; exec self
17:32:31  WARN router port busy; binding an ephemeral free port  preferred=8917
17:32:31  router listening port=64205

Every one of those sessions fails its next model call. What the user sees is the harness's error, three hours and one restart removed from the cause:

■ stream disconnected before completion: error sending request for url
(https://chatgpt.com/backend-api/codex/responses)

Route transport is injected into a session's environment once, at spawn (PROXY_ENV, router.rs). There is no renegotiation and no way to tell a running harness the router moved — the port it was handed is the only one it will ever try. The module header already says reclaiming the port is mandatory for exactly this reason; the fallback path quietly violated it.

And it made it permanent. The fallback was written to router.port, so the next start preferred 64205 and the home never went back. 8917 has been free for a while now. Nothing will ever reclaim it.

What this changes

Wait for the port instead of giving up on first refusal. During a restart the likeliest holder is the daemon that just exec'd away, which lets go in milliseconds. One second of retries at startup, versus every live session losing its route.

Never record a stand-in as the home's port. Binding elsewhere so the daemon still boots is right; recording it is what turned one busy moment into a permanent move. A home that already has a port keeps preferring it. A home with no port yet has nothing dialing it, so it adopts what it gets — that's how a second home on one machine gets a stable identity, and it's preserved.

The rule is extracted as records_port(pinned, bound, claimed_preferred, established) and tested directly, because the interesting case (a first boot whose default port is taken) can't be staged without binding the machine's real default port.

Report a fallback as the outage it is. tracing::error!, not warn, naming what was lost — the daemon knows at that instant that every session it spawned is unroutable.

Take the port back when it frees, and serve on it alongside the stand-in, so neither the sessions predating the fallback nor those spawned during it are cut off. New sessions get the reclaimed port, so the home converges back to one. Attribution is by proxy credential, not by arrival port, so serving two ports changes nothing about routing.

A pinned [router] port = N is exempt throughout — it never falls back and is never rewritten.

Behavior change to call out

auto_port_falls_back_and_persists_when_preferred_is_busy asserted the old rule and is replaced by a_stand_in_port_does_not_replace_this_homes_own, which asserts the opposite. That inversion is the fix, not a casualty of it: its own doc comment claimed "reclaim-on-restart is covered", which persisting the fallback is precisely what defeated.

Not fixed here

Why 8917 and the webui's 5746 were both busy at 17:32 — almost certainly a second daemon instance. This PR makes that survivable rather than silently destructive.

Spec

0183-a-home-does-not-give-up-its-router-port (new). Module header updated to match.

Testing

cargo test --workspace green. New: the persist rule across all six cases, a stand-in leaving the recorded port alone, and the restart race — a holder that releases mid-window has its port taken rather than abandoned.

This PR touches crates/daemon only → the relevant binary is construct.

🤖 Generated with Claude Code

Route transport is injected into a session's environment once, at spawn,
and never revisited. There is no renegotiation and no way to tell a
running harness the router moved: the port it was handed is the only one
it will ever dial.

So when the preferred port was busy, the router bound an ephemeral one,
logged a warning about a port, and recorded the fallback as this home's
port. Nothing failed at that moment. The failures arrived later, one per
session, as a connection error naming a URL — attributable to the
network, the provider, or the harness, but not to the daemon that had
moved out from under them. Recording the fallback made it permanent: the
next start preferred the stand-in, so the home never went back, and every
session spawned before the collision was unroutable for the rest of its
life. Observed on a live fleet: 67 of 71 running adapters dialing a port
nothing had listened on for half an hour.

Four changes, all following from the port being part of a home's identity:

Wait for it rather than give up on the first refusal — during a restart
the likeliest holder is the daemon that just left, and it lets go in
moments.

Never record a stand-in as the home's port. A home that already has one
keeps preferring it; only a home with nothing dialing it yet adopts what
it can get, which is how a second home on one machine gets an identity.

Report a fallback as the outage it is. The daemon knows at that moment
that every session it spawned has lost its route, which is an error about
sessions, not a warning about a port.

Take the port back when it frees, and serve on it alongside the stand-in
so neither generation of sessions is cut off.

A pinned port is exempt throughout: it never falls back and is never
rewritten.
@edwin-zvs
edwin-zvs force-pushed the router-port-strands-sessions branch from 6f93230 to 04e7dfb Compare August 2, 2026 19:14
@edwin-zvs
edwin-zvs merged commit 0047234 into main Aug 2, 2026
1 check passed
@edwin-zvs
edwin-zvs deleted the router-port-strands-sessions branch August 2, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant