[TRTLLM-12721][fix] Add gated C++ NIXL in-flight cancellation and safe cleanup

[chienchunhung]

Summary

This PR adds experimental, default-off deadline enforcement and in-flight KV-transfer cancellation for the PyExecutor C++ NIXL cache transceiver using the NIXL UCX plugin.

The change is self-contained at the request and transfer-resource boundary. When cancellation is enabled, a timed-out or user-cancelled request is not cleaned up merely because cancellation was requested. C++ transfer bookkeeping retains the request lifetime, and TransferSession retains any reserved staging-buffer slots until the distributed transfer outcome is terminal. Slots are released only when reuse is known to be safe; otherwise the transfer-buffer pool is poisoned fail-closed and the executor requires a process restart.

Set TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL=1 to opt in. The flag remains disabled by default.

Activation and compatibility

Opt-in is accepted only with:

• the PyExecutor C++ cache transceiver (transceiver_runtime='CPP' or the C++ default);
• backend='NIXL' with the NIXL UCX plugin;
• a configured kv_transfer_timeout_ms;
• asynchronous, non-layer-wise KV transfer; and
• zero-copy KV transfer disabled.

Python/V2, direct UCX, MPI, Mooncake, NIXL Libfabric, the legacy C++ executor, and ambiguous legacy backend selection reject opt-in before an unsupported cancellation path can run. With the flag unset, existing timeout handling, backend selection, and transceiver behavior remain unchanged.

All participating ranks and both disaggregated services must use the same feature setting and compatible transport configuration. This PR does not negotiate cancellation capability between CTX and GEN.

Cancellation and cleanup flow

1. The executor observes that a context or generation KV transfer exceeded its configured deadline, or receives a user cancellation.
2. Local completed, failed, and timed-out observations enter the existing distributed transfer-state reduction. Timeout or failure wins across participating ranks.
3. Every nonterminal rank requests cancellation for the agreed request set. NIXL status polling stays bounded so the executor can continue scheduling and transfer progress between polls.
4. C++ transfer bookkeeping retains the request lifetime, while TransferSession retains the receive-buffer reservations used by KV, MLA, and RNN formatters as cancellation races with terminal status.
5. The request remains in AsyncTransferManager or the active generation set until C++ reports a terminal distributed outcome. Attention-DP error handling votes on request IDs before entering response collectives.
6. On a safely terminal transfer, the NIXL transfer handle is released and staging slots return to their pools. A late success after the deadline still fails the request, but cleanup occurs only after the transfer is quiesced.
7. If quiescence or buffer reuse cannot be established, the affected slot or pool is poisoned. Poison is reduced across the executor world and treated as fatal rather than permitting possible buffer reuse.

The opt-in path rejects blockAll / nullopt wait-all status calls. Supporting wait-all semantics together with finite cancellation deadlines remains follow-up work.

Scope boundaries

Area	Ownership
Deadline and cancellation decision	Finite deadline detection, distributed timeout/failure reduction, and cancellation of queued or active C++ NIXL transfers are included here.
Request and buffer lifetime	Deferred request cleanup, transfer-session staging-slot ownership, NIXL handle release, and fail-closed buffer poisoning are included here because they are required for safe opt-in.
Immediate PyExecutor integration	Rank-aligned error handling, retained transfer owners, user-cancellation retry, and fatal poison escalation are included here.
Full IPC/worker lifecycle	Process-level fatal propagation, proxy shutdown, and disposable-worker restart behavior remain in #15795.
Cross-service protocol	CTX/GEN capability and effective-mode negotiation remain in #15798. Peer cancellation, terminal evidence, and bounded metadata retirement remain in #15799.
Other transports	Python/V2, direct UCX, MPI, Mooncake, and NIXL Libfabric require backend-specific cancellation and quiescence contracts.
Default-on rollout	Remains in #15738 after the required protocol and lifecycle gates are complete.

Because this PR now owns the minimum native session and buffer-safety envelope, #15794 should be rebased and narrowed to its remaining C++ protocol, capability, and teardown hardening instead of duplicating this implementation. #15795 continues to own the broader PyExecutor IPC and worker-lifecycle layer.

Dependency graph

Arrows point from prerequisite to dependent. The bounded-polling work originally proposed in #15181 was subsumed by merged #15356; #15356 and #15737 are inherited from main.

flowchart LR
    MAIN["main<br/>prerequisites merged"]
    PR15238["#15238 current<br/>gated cancellation + safe cleanup"]
    PR15794["#15794 draft<br/>remaining C++ protocol hardening"]
    PR15795["#15795 draft<br/>IPC and worker lifecycle"]
    PR15798["#15798 follow-up<br/>CTX/GEN mode negotiation"]
    PR15799["#15799 follow-up<br/>peer terminal protocol"]
    PR15738["#15738 draft<br/>qualified default-on policy"]

    MAIN --> PR15238
    PR15238 --> PR15794
    PR15794 --> PR15795
    PR15794 --> PR15798
    PR15798 --> PR15799
    PR15795 --> PR15738
    PR15798 --> PR15738
    PR15799 -.->|or approved bounded fail/restart policy| PR15738

    classDef merged fill:#d8f3dc,stroke:#2d6a4f,color:#1b4332
    classDef current fill:#cfe8ff,stroke:#0969da,color:#0a3069,stroke-width:3px
    classDef draft fill:#fff3bf,stroke:#e67700,color:#7c2d12
    classDef followup fill:#eeeeee,stroke:#6e7781,color:#24292f,stroke-dasharray: 5 4

    class MAIN merged
    class PR15238 current
    class PR15794,PR15795,PR15738 draft
    class PR15798,PR15799 followup

Loading

Validation

• Rebased onto upstream main at 1225445ceb; merged prerequisites are inherited from main.
• Full PR-diff formatting, lint, YAML, test-list, merge-marker, and static pre-commit checks passed locally.
• git diff --check and Python syntax compilation passed.
• Added focused coverage for default-off compatibility, opt-in qualification, distributed timeout/error handling, deferred cleanup, user cancellation, buffer ownership and poisoning, and NIXL transfer-handle release.
• Added C++/Python NIXL integration coverage to the applicable CI test lists.
• Focused pytest execution is unavailable in the local Darwin checkout because the Python environment lacks transformers and the changed C++ bindings are not built. Exact-head CI remains required for compiled and distributed runtime validation.

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #53406 [ run ] triggered by Bot. Commit: d16f28c Link to invocation

[tensorrt-cicd]

PR_Github #53406 [ run ] completed with state SUCCESS. Commit: d16f28c
/LLM/main/L0_MergeRequest_PR pipeline #42578 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #53628 [ run ] triggered by Bot. Commit: 631cd67 Link to invocation

[tensorrt-cicd]

PR_Github #53628 [ run ] completed with state SUCCESS. Commit: 631cd67
/LLM/main/L0_MergeRequest_PR pipeline #42773 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #53788 [ run ] triggered by Bot. Commit: 631cd67 Link to invocation

[tensorrt-cicd]

PR_Github #53788 [ run ] completed with state FAILURE. Commit: 631cd67
/LLM/main/L0_MergeRequest_PR pipeline #42906 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #53928 [ run ] triggered by Bot. Commit: 36f9ee5 Link to invocation

[tensorrt-cicd]

PR_Github #53928 [ run ] completed with state SUCCESS. Commit: 36f9ee5
/LLM/main/L0_MergeRequest_PR pipeline #43021 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #54148 [ run ] triggered by Bot. Commit: ff622ff Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #54155 [ run ] triggered by Bot. Commit: 6954acf Link to invocation

[tensorrt-cicd]

PR_Github #54148 [ run ] completed with state ABORTED. Commit: ff622ff

Link to invocation

[tensorrt-cicd]

PR_Github #54155 [ run ] completed with state SUCCESS. Commit: 6954acf
/LLM/main/L0_MergeRequest_PR pipeline #43238 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #54336 [ run ] triggered by Bot. Commit: 24eb554 Link to invocation

[chienchunhung]

/bot run

[tensorrt-cicd]

PR_Github #54356 [ run ] triggered by Bot. Commit: 24eb554 Link to invocation

[tensorrt-cicd]

PR_Github #54336 [ run ] completed with state ABORTED. Commit: 24eb554

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #59016 [ run ] triggered by Bot. Commit: 4b9db25 Link to invocation

[tensorrt-cicd]

PR_Github #59016 [ run ] completed with state SUCCESS. Commit: 4b9db25
/LLM/main/L0_MergeRequest_PR pipeline #47543 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #59042 [ run ] triggered by Bot. Commit: 4b9db25 Link to invocation

[tensorrt-cicd]

PR_Github #59042 [ run ] completed with state SUCCESS. Commit: 4b9db25
/LLM/main/L0_MergeRequest_PR pipeline #47567 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast --stage-list "DGX_B200-4_GPUs-PyTorch-3,DGX_B200-4_GPUs-PyTorch-Ray-1"

[tensorrt-cicd]

PR_Github #59126 [ run ] triggered by Bot. Commit: 4b9db25 Link to invocation

[tensorrt-cicd]

PR_Github #59126 [ run ] completed with state SUCCESS. Commit: 4b9db25
/LLM/main/L0_MergeRequest_PR pipeline #47636 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #59232 [ run ] triggered by Bot. Commit: 4b9db25 Link to invocation

[tensorrt-cicd]

PR_Github #59232 [ run ] completed with state SUCCESS. Commit: 4b9db25
/LLM/main/L0_MergeRequest_PR pipeline #47726 completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --post-merge --disable-fail-fast --disable-reuse-test --stage-list "DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-2,DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-3,DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-6"

[tensorrt-cicd]

PR_Github #59245 [ run ] triggered by Bot. Commit: 4b9db25 Link to invocation

[tensorrt-cicd]

PR_Github #59245 [ run ] completed with state SUCCESS. Commit: 4b9db25
/LLM/main/L0_MergeRequest_PR pipeline #47735 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --post-merge --disable-fail-fast --disable-reuse-test --stage-list "DGX_B200-16_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU8-Post-Merge-4,GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge-1,GB200-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX2-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1,GB200-24_GPUs-6_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE4-GPU16-Post-Merge-2"

[tensorrt-cicd]

PR_Github #59257 [ run ] triggered by Bot. Commit: 4b9db25 Link to invocation

[tensorrt-cicd]

PR_Github #59257 [ run ] completed with state SUCCESS. Commit: 4b9db25
/LLM/main/L0_MergeRequest_PR pipeline #47746 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #59280 [ run ] triggered by Bot. Commit: 4b9db25 Link to invocation

[tensorrt-cicd]

PR_Github #59280 [ run ] completed with state SUCCESS. Commit: 4b9db25
/LLM/main/L0_MergeRequest_PR pipeline #47767 completed with status: 'SUCCESS'

CI Report

Link to invocation