Skip to content

fix(orchestrator): Prevent redundant delegated completion turns - #5311

Merged
juliusmarminge merged 5 commits into
pingdotgg:t3code/codex-turn-mappingfrom
mwolson:fix/delegated-completion-delivery
Aug 5, 2026
Merged

fix(orchestrator): Prevent redundant delegated completion turns#5311
juliusmarminge merged 5 commits into
pingdotgg:t3code/codex-turn-mappingfrom
mwolson:fix/delegated-completion-delivery

Conversation

@mwolson

@mwolson mwolson commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Decision brief: evidence, product
    tradeoffs, and the final durable-delivery design.
  • Coalesce app-owned delegated child terminals into one durable, queued
    completion delivery per parent-run cohort, with at most one successor for
    results that arrive after the first delivery starts.
  • Let a parent acknowledge a terminal result through task_status or an
    untruncated direct-child read, then cancel its unstarted stale delivery
    without deleting the result.
  • Hide automatic completion deliveries from web and mobile queue controls while
    preserving their scheduler priority and allowing their prompt to appear if
    the delivery enters the conversation.
  • Add durable Stop, task-cancel, Queue Remove, archive, delete, and recovery
    barriers around server-owned completion delivery.

Problem and Fix

Problem and Why it Happened Fix
#4499 delegated-task-parent-wake deliberately created one real queued parent continuation for every async child terminal. Sibling completions therefore fanned out into repeated parent turns and timeline output. A parent-run cohort reserves one delivery that terminal siblings join before it starts. A late result gets at most one bounded successor instead of recursively re-arming the parent.
A parent could read a result through task_status or its direct child timeline, but the scheduler had no durable observation signal, so its queued completion still ran later. Terminal task_status reads and untruncated direct-child terminal-result reads durably acknowledge the task and remove it from the queued delivery. Status-only waits, prompt-only reads, and truncated results do not acknowledge. When no owned task remains, the queued run is cancelled.
Stop, Queue Remove, archive, and delete cancelled transient work but did not own or dispose future completion delivery. Durable cohort and task-delivery states make these actions replay-safe barriers. Results remain inspectable even when their automatic delivery is disposed.
Server-created completion messages looked like normal queue entries and exposed Edit, Reorder, and Steer controls. In particular, a Grok wake could be converted into interrupt-and-restart steering. Server commands reject those mutations, while web and mobile omit automatic completion entries from visible queue controls. The scheduler still gives every surviving hidden delivery priority over ordinary queued messages, and the prompt remains visible once it enters the conversation.

Defensive Fixes

Problem and Why it Happened Fix
Provider lifecycle updates can persist a full run or subagent row captured before a later acknowledgement or Stop barrier. Projection storage preserves newer delivery state when a stale provider row omits it, and run finalization intentionally excludes its old cohort snapshot.
A continuation offer can survive in memory while a cohort is acknowledged, disposed, archived, deleted, or replaced by a later generation. The worker rereads durable projection ownership immediately before dispatch and drops an offer that is no longer current.

Validation

  • Focused MCP, continuation-worker, projection, Claude allowlist,
    scheduler-priority, and hidden queue presentation tests pass.
  • Direct-child read coverage proves that only the complete terminal result,
    rather than t3_thread_wait, a prompt-only page, or a truncated result,
    acknowledges delivery.
  • Isolated real-provider verification proved one durable delivery for one child,
    coalescing for two async Grok children, acknowledgement persistence, and
    durable acknowledgement and retraction before a queued delivery starts.
  • vp check passes with no warnings or lint errors, all 15 typecheck tasks pass,
    and the desktop production build succeeds.
  • The current source tip was integrated into the v2.1 trial, where focused
    queue and delegated-completion coverage and guarded Nightly AppImage package
    verification pass.

Note

High Risk
Changes orchestration scheduling, MCP tool semantics, and durable completion-delivery state across server projection, continuation workers, and clients—incorrect acknowledgement or disposal could drop or duplicate parent work.

Overview
Stops one parent continuation per async child terminal by coalescing app-owned delegated results into a single durable queued delivery per parent-run cohort, with at most one bounded successor when a terminal arrives after the first delivery has already started.

MCP observation now owns delivery lifecycle: task_status, delegate/wait paths, and an untruncated direct-child t3_thread_read of the terminal assistant result dispatch completion-delivery.acknowledge and cancel the stale queued run when no tasks remain. t3_thread_wait, prompt-only reads, and truncated pages do not acknowledge. task_cancel interrupts the child when possible and disposes automatic delivery (including on already-terminal tasks) without hiding the result. Tool metadata marks task_status and t3_thread_read as non-read-only; Claude’s read-only MCP allowlist drops those tools accordingly.

Race-safety: checkpoint capture and related run finalization omit delegatedCompletion from stale run snapshots so newer cohort state is not overwritten; tests cover acknowledge/dispose idempotency and wake-policy upgrades after delivery settled.

Mobile queue rows gain a Remove control wired to queued-run.cancel, with shared presentation helpers for enablement and command building. Integration tests expand heavily around coalescing, Stop barriers, server-owned queue mutation rejects, late completions, and queue remove.

Reviewed by Cursor Bugbot for commit a60f38e. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Prevent redundant delegated completion turns by tracking delivery state across runs

  • Adds a delegatedCompletion cohort to runs and completionDelivery to subagents in the contract schema, enabling generation-tracked, durable ownership of delegated task completion delivery.
  • ProviderContinuationService now dispatches delegated completions as server-owned queued messages with capped exponential backoff retries and revalidation before re-offer; drops on archivedAt/deletedAt.
  • ProjectionStore preserves delegatedCompletion on runs and completionDelivery on subagents when newer events omit those fields, both in-memory and in SQL upserts.
  • CheckpointCaptureService and RunExecutionService strip delegatedCompletion from finalized run snapshots to avoid overwriting a newer cohort.
  • MCP tool handlers (readTask, waitForTask, taskStatus, cancelTask, readThread) now dispatch delegated_task.completion-delivery.acknowledge or dispose commands at the appropriate points in the task lifecycle.
  • Automatic completion delivery runs are hidden from the visible queue in both web and mobile clients via deriveThreadQueueWorkflowState and queuedRunsInDeliveryOrder.
  • Risk: ProjectionStore SQL upsert behavior changes for any row that previously stored delegatedCompletion/completionDelivery; stale events no longer overwrite those fields.

Macroscope summarized a60f38e.

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1b7060e3-68e2-4610-bd0a-74c7849c81cf

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added size:XXL 1,000+ changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list. labels Aug 4, 2026
@mwolson mwolson changed the title Fix delegated completion delivery fan-out fix(orchestrator): Prevent redundant delegated completion turns Aug 4, 2026
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch 2 times, most recently from c5e53c8 to 922f352 Compare August 4, 2026 13:48
@mwolson
mwolson force-pushed the fix/delegated-completion-delivery branch from db285db to cfb5bac Compare August 4, 2026 15:43
@mwolson
mwolson marked this pull request as ready for review August 4, 2026 15:44
Comment thread apps/mobile/src/features/threads/ThreadQueueControl.tsx
Comment thread apps/mobile/src/features/threads/thread-work-log.tsx
Comment thread apps/server/scripts/acp-replay-agent.ts
Comment thread apps/server/scripts/record-codex-app-server-replay-fixture.ts
Comment thread apps/mobile/src/features/threads/ThreadFeed.tsx
Comment thread apps/mobile/src/state/use-thread-selection.ts
Comment thread apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts
Comment thread apps/server/src/orchestration-v2/Adapters/CursorAdapterV2.ts
Comment thread apps/mobile/src/test-fixtures.ts
Comment thread apps/server/src/orchestration-v2/Adapters/GrokAdapterV2.testkit.ts
@macroscopeapp

macroscopeapp Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs human review

1 blocking correctness issue found. Diff is too large for automated approval analysis. A human reviewer should evaluate this PR.

You can customize Macroscope's approvability policy. Learn more.

Comment thread apps/mobile/src/features/threads/ThreadQueueControl.tsx Outdated
@mwolson
mwolson force-pushed the fix/delegated-completion-delivery branch from cf960a2 to d137da8 Compare August 4, 2026 18:10
Comment thread apps/server/src/orchestration-v2/Orchestrator.ts
Comment thread apps/server/src/orchestration-v2/Orchestrator.ts
Comment thread apps/server/src/orchestration-v2/RunExecutionService.ts
Comment thread apps/server/src/orchestration-v2/Orchestrator.ts
@mwolson
mwolson force-pushed the fix/delegated-completion-delivery branch 2 times, most recently from 9aeb22b to a999ce7 Compare August 5, 2026 07:28
@mwolson
mwolson force-pushed the fix/delegated-completion-delivery branch from a999ce7 to d1025b8 Compare August 5, 2026 07:50
Comment thread apps/server/src/mcp/OrchestratorMcpService.ts
Comment thread apps/server/src/orchestration-v2/ProviderContinuationService.ts

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d1025b8. Configure here.

Comment thread apps/server/src/mcp/OrchestratorMcpService.ts
@mwolson
mwolson force-pushed the fix/delegated-completion-delivery branch from d1025b8 to a60f38e Compare August 5, 2026 08:24
@juliusmarminge
juliusmarminge merged commit 99c957d into pingdotgg:t3code/codex-turn-mapping Aug 5, 2026
16 of 17 checks passed
@mwolson
mwolson deleted the fix/delegated-completion-delivery branch August 5, 2026 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants