[https://nvbugs/6094068][fix] Fix Qwen3-Next bf16 4gpu test#15206

JadoTu · 2026-06-10T07:21:51Z

Summary by CodeRabbit

Tests

Updated CUDA graph and LLM configuration parameters for test scenarios
Adjusted test waiver list to modify skip conditions for specific test cases

Description

Fix the OOM issue via max_batch_size setting.
Unwaive related tests.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Signed-off-by: jiant <107457950+JadoTu@users.noreply.github.com>

coderabbitai · 2026-06-10T07:25:19Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e8b180f6-cfdf-491a-acd1-44f86375353a

📥 Commits

Reviewing files that changed from the base of the PR and between 31e730a and 3fe1f2a.

📒 Files selected for processing (2)

tests/integration/defs/accuracy/test_llm_api_pytorch.py
tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

This PR updates LLM API PyTorch test configurations and adjusts corresponding test waivers. Specifically, it modifies CUDA graph and batch size parameters in test_bf16_4gpu and test_nvfp4, then removes skip waivers for configurations that are no longer expected to fail.

Changes

Test Configuration and Waivers Update

Layer / File(s)	Summary
CUDA graph and LLM parameter configuration `tests/integration/defs/accuracy/test_llm_api_pytorch.py`	`test_bf16_4gpu` now configures CUDA graphs with fixed `max_batch_size=128` and padding enabled, and sets LLM `max_num_tokens=8192` and `max_batch_size=128`. `test_nvfp4` explicitly sets `max_batch_size=512` in LLM instantiation.
Test waivers alignment `tests/integration/test_lists/waives.txt`	Removed skip waivers for `test_bf16_4gpu` with `tp4ep4_cudagraph_overlap_adp_on` and `test_nvfp4` with `tp4ep4_adp_on-trtllm`, while retaining `test_nvfp4[tp1-cutlass]` skip waiver.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Suggested reviewers

kaiyux

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly matches the main change: fixing a Qwen3-Next bf16 4gpu test issue as described by the NVBugs ticket reference.
Description check	✅ Passed	The description includes required sections (Description and PR Checklist) with relevant details about fixing an OOM issue and unwaiving tests, though Test Coverage section is missing.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

⚔️ Resolve merge conflicts

Resolve merge conflict in branch bugfix_6094068

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Signed-off-by: Jian Tu <107457950+JadoTu@users.noreply.github.com>

JadoTu · 2026-06-10T09:38:17Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-10T09:44:44Z

PR_Github #53294 [ run ] triggered by Bot. Commit: 72fb348 Link to invocation

tensorrt-cicd · 2026-06-10T19:04:07Z

PR_Github #53294 [ run ] completed with state SUCCESS. Commit: 72fb348
/LLM/main/L0_MergeRequest_PR pipeline #42483 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JadoTu · 2026-06-11T03:16:35Z

/bot run

tensorrt-cicd · 2026-06-11T03:22:28Z

PR_Github #53458 [ run ] triggered by Bot. Commit: 72fb348 Link to invocation

tensorrt-cicd · 2026-06-11T07:56:01Z

PR_Github #53458 [ run ] completed with state SUCCESS. Commit: 72fb348
/LLM/main/L0_MergeRequest_PR pipeline #42623 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JadoTu · 2026-06-12T09:25:11Z

/bot run

tensorrt-cicd · 2026-06-12T09:31:41Z

PR_Github #53864 [ run ] triggered by Bot. Commit: 72fb348 Link to invocation

tensorrt-cicd · 2026-06-12T14:19:33Z

PR_Github #53864 [ run ] completed with state SUCCESS. Commit: 72fb348
/LLM/main/L0_MergeRequest_PR pipeline #42971 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JadoTu · 2026-06-13T04:52:55Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-13T04:59:02Z

PR_Github #54011 [ run ] triggered by Bot. Commit: a66ffcd Link to invocation

tensorrt-cicd · 2026-06-13T13:47:22Z

PR_Github #54011 [ run ] completed with state SUCCESS. Commit: a66ffcd
/LLM/main/L0_MergeRequest_PR pipeline #43095 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JadoTu · 2026-06-15T07:41:39Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-15T07:50:45Z

PR_Github #54256 [ run ] triggered by Bot. Commit: 2c08dad Link to invocation

tensorrt-cicd · 2026-06-15T16:42:12Z

PR_Github #54256 [ run ] completed with state SUCCESS. Commit: 2c08dad
/LLM/main/L0_MergeRequest_PR pipeline #43331 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Jian Tu <107457950+JadoTu@users.noreply.github.com>

JadoTu · 2026-06-16T05:18:20Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-16T05:24:04Z

PR_Github #54472 [ run ] triggered by Bot. Commit: 40dd603 Link to invocation

tensorrt-cicd · 2026-06-16T10:22:58Z

PR_Github #54472 [ run ] completed with state SUCCESS. Commit: 40dd603
/LLM/main/L0_MergeRequest_PR pipeline #43538 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JadoTu · 2026-06-17T05:35:24Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-17T05:40:50Z

PR_Github #54770 [ run ] triggered by Bot. Commit: ed6a78c Link to invocation

tensorrt-cicd · 2026-06-17T10:53:55Z

PR_Github #54770 [ run ] completed with state SUCCESS. Commit: ed6a78c
/LLM/main/L0_MergeRequest_PR pipeline #43789 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

JadoTu · 2026-06-22T04:27:35Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-22T04:33:05Z

PR_Github #55016 [ run ] triggered by Bot. Commit: d0f775b Link to invocation

tensorrt-cicd · 2026-06-22T09:21:34Z

PR_Github #55016 [ run ] completed with state SUCCESS. Commit: d0f775b
/LLM/main/L0_MergeRequest_PR pipeline #44008 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JadoTu · 2026-06-24T03:28:34Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-24T03:35:04Z

PR_Github #55391 [ run ] triggered by Bot. Commit: 93de504 Link to invocation

tensorrt-cicd · 2026-06-24T08:29:02Z

PR_Github #55391 [ run ] completed with state SUCCESS. Commit: 93de504
/LLM/main/L0_MergeRequest_PR pipeline #44338 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Jian Tu <107457950+JadoTu@users.noreply.github.com>

JadoTu · 2026-06-25T05:06:59Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-25T05:14:11Z

PR_Github #55704 [ run ] triggered by Bot. Commit: 7d0ef42 Link to invocation

tensorrt-cicd · 2026-06-25T05:57:32Z

PR_Github #55704 [ run ] completed with state SUCCESS. Commit: 7d0ef42
/LLM/main/L0_MergeRequest_PR pipeline #44608 completed with status: 'SUCCESS'

CI Report

Link to invocation

…5206) Signed-off-by: jiant <107457950+JadoTu@users.noreply.github.com> Signed-off-by: Jian Tu <107457950+JadoTu@users.noreply.github.com>

fix NVBUG#6094068

3fe1f2a

Signed-off-by: jiant <107457950+JadoTu@users.noreply.github.com>

JadoTu requested a review from a team as a code owner June 10, 2026 07:21

github-actions Bot assigned JadoTu Jun 10, 2026

Merge branch 'main' into bugfix_6094068

72fb348

Signed-off-by: Jian Tu <107457950+JadoTu@users.noreply.github.com>

JadoTu mentioned this pull request Jun 10, 2026

[https://nvbugs/6094068][fix] Cap Mamba cache max_batch_size for memory and add CUDA P2P check for DeepEP #13474

Closed

2 tasks

jieli-matrix approved these changes Jun 12, 2026

View reviewed changes

Merge branch 'main' into bugfix_6094068

a66ffcd

Merge branch 'main' into bugfix_6094068

2c08dad

Merge branch 'main' into bugfix_6094068

40dd603

Signed-off-by: Jian Tu <107457950+JadoTu@users.noreply.github.com>

JadoTu enabled auto-merge (squash) June 16, 2026 05:18

Merge branch 'main' into bugfix_6094068

ed6a78c

Merge branch 'main' into bugfix_6094068

d0f775b

Merge branch 'main' into bugfix_6094068

93de504

Merge branch 'main' into bugfix_6094068

7d0ef42

Signed-off-by: Jian Tu <107457950+JadoTu@users.noreply.github.com>

JadoTu merged commit 4c4c74a into NVIDIA:main Jun 25, 2026
7 checks passed

Uh oh!

Conversation

JadoTu commented Jun 10, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Tests

Description

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented Jun 10, 2026

Walkthrough

Changes

Estimated code review effort

Suggested reviewers

❌ Failed checks (1 warning)

Uh oh!

JadoTu commented Jun 10, 2026

Uh oh!

tensorrt-cicd commented Jun 10, 2026

Uh oh!

tensorrt-cicd commented Jun 10, 2026

Uh oh!

JadoTu commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

JadoTu commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

JadoTu commented Jun 13, 2026

Uh oh!

tensorrt-cicd commented Jun 13, 2026

Uh oh!

tensorrt-cicd commented Jun 13, 2026

Uh oh!

JadoTu commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

tensorrt-cicd commented Jun 15, 2026

Uh oh!

JadoTu commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

JadoTu commented Jun 17, 2026

Uh oh!

tensorrt-cicd commented Jun 17, 2026

Uh oh!

tensorrt-cicd commented Jun 17, 2026

Uh oh!

JadoTu commented Jun 22, 2026

Uh oh!

tensorrt-cicd commented Jun 22, 2026

Uh oh!

tensorrt-cicd commented Jun 22, 2026

Uh oh!

JadoTu commented Jun 24, 2026

Uh oh!

tensorrt-cicd commented Jun 24, 2026

Uh oh!

tensorrt-cicd commented Jun 24, 2026

Uh oh!

JadoTu commented Jun 25, 2026

Uh oh!

tensorrt-cicd commented Jun 25, 2026

Uh oh!

tensorrt-cicd commented Jun 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

JadoTu commented Jun 10, 2026 •

edited by coderabbitai Bot

Loading