Skip to content

[https://nvbugs/6114610][test] unwaive disagg tests fixed by UCX_TLS setter#14440

Merged
pcastonguay merged 1 commit into
NVIDIA:mainfrom
xwang233:nvbugs/6114610-unwaive
May 26, 2026
Merged

[https://nvbugs/6114610][test] unwaive disagg tests fixed by UCX_TLS setter#14440
pcastonguay merged 1 commit into
NVIDIA:mainfrom
xwang233:nvbugs/6114610-unwaive

Conversation

@xwang233

@xwang233 xwang233 commented May 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

Test plan

  • Built HEAD from source and validated test_disaggregated_ctxpp4_gentp4[TinyLlama-1.1B-Chat-v1.0] passes on the B200 cluster where QA previously reproduced the hang (146s, zero KV-cache-transfer-timeout signals)

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Tests
    • Updated test waiver configurations for disaggregated integration tests, adjusting which test variants are marked as waived.

Review Change Stack

@coderabbitai

coderabbitai Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: cb793803-501f-4e49-ad7e-9d49d2c12981

📥 Commits

Reviewing files that changed from the base of the PR and between 25e5a99 and 34864e0.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
  • tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

Updated the test waiver list in waives.txt by removing waivers for specific TinyLlama parameterization variants and the GPT-OSS-120B harmony test, while keeping the ctxtp2_genpp2 TinyLlama waiver. Net change: removed 4 test waiver entries.

Changes

Test Waiver Management

Layer / File(s) Summary
Disaggregated test waiver adjustments
tests/integration/test_lists/waives.txt
Removed waivers for TinyLlama ctxpp4_* and ctxtp2pp2_* parameterizations while preserving ctxtp2_genpp2, and removed the GPT-OSS-120B harmony test waiver.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: unwaiving disaggregated tests that were fixed by a UCX_TLS environment variable setter, directly matching the changeset modifications.
Description check ✅ Passed The PR description includes a clear summary explaining the issue (hanging tests due to missing UCX_TLS), the solution (relying on previous fix), and validation results, covering all essential aspects.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@xwang233
xwang233 force-pushed the nvbugs/6114610-unwaive branch from 34864e0 to 8b46a02 Compare May 22, 2026 05:11
@xwang233

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Ray-1, DGX_B200-4_GPUs-PyTorch-1, DGX_B200-4_GPUs-PyTorch-2, DGX_B200-4_GPUs-PyTorch-3"

@xwang233
xwang233 requested a review from chuangz0 May 22, 2026 05:23
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #49854 [ run ] triggered by Bot. Commit: 8b46a02 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #49854 [ run ] completed with state SUCCESS. Commit: 8b46a02
/LLM/main/L0_MergeRequest_PR pipeline #39438 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

@pcastonguay

Copy link
Copy Markdown
Collaborator

/bot skip --comment "Unwaive tests which were run individually in CI"

@pcastonguay
pcastonguay enabled auto-merge (squash) May 22, 2026 16:01
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #49960 [ skip ] triggered by Bot. Commit: 8b46a02 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #49960 [ skip ] completed with state SUCCESS. Commit: 8b46a02
Skipping testing for commit 8b46a02

Link to invocation

@xwang233
xwang233 requested a review from a team May 23, 2026 17:12
…setter

The underlying cause of nvbugs/6114610 (and duplicates 6114612,
6011317) was that `run_disaggregated_test` did not set `UCX_TLS`,
leaving IB transports enabled by default on nsc-svg-slurm-1 where
a cluster-specific IB endpoint timeout (nvbugs/6044893) caused NIXL
notifications to be silently dropped.

Commit 1109e1c (PR NVIDIA#14168, Chuang Zhu, 2026-05-18) added the
missing `run_env["UCX_TLS"] = get_ucx_tls()` setter to
`run_disaggregated_test`, which now correctly excludes IB on B200
(SM>=90) via `^ib,gdr_copy`.

Validated on nsc-svg-slurm-1-gpu-107 (the failing cluster) with
HEAD built from source: `test_disaggregated_ctxpp4_gentp4` passes
in 146s with zero KV-cache-transfer-timeout signals.

Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@xwang233
xwang233 force-pushed the nvbugs/6114610-unwaive branch from 8b46a02 to 38ebbe4 Compare May 25, 2026 22:45
@xwang233

Copy link
Copy Markdown
Collaborator Author

/bot skip --comment "Unwaive tests which were run individually in CI"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #50240 [ skip ] triggered by Bot. Commit: 38ebbe4 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #50240 [ skip ] completed with state SUCCESS. Commit: 38ebbe4
Skipping testing for commit 38ebbe4

Link to invocation

@pcastonguay
pcastonguay merged commit e2497f0 into NVIDIA:main May 26, 2026
7 checks passed
bmarimuthu-nv pushed a commit to nv-auto-deploy/TensorRT-LLM that referenced this pull request May 28, 2026
…setter (NVIDIA#14440)

Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants