Skip to content

[https://nvbugs/6074943][fix] Disable new aiperf server metrics to stop hang.#13314

Merged
dominicshanshan merged 1 commit into
NVIDIA:mainfrom
dominicshanshan:user/shanshan/nvbug_6074943
Apr 23, 2026
Merged

[https://nvbugs/6074943][fix] Disable new aiperf server metrics to stop hang.#13314
dominicshanshan merged 1 commit into
NVIDIA:mainfrom
dominicshanshan:user/shanshan/nvbug_6074943

Conversation

@dominicshanshan

@dominicshanshan dominicshanshan commented Apr 22, 2026

Copy link
Copy Markdown
Collaborator

Summary by CodeRabbit

  • Tests
    • Improved stress test stability by disabling server metrics collection during profile operations.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Apr 22, 2026

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

Added the --no-server-metrics CLI flag to the create_aiperf_command() function in the stress test to disable server metrics collection during aiperf profiling, preventing hangs caused by incompatible metrics endpoint format.

Changes

Cohort / File(s) Summary
Stress Test Configuration
tests/integration/defs/stress_test/stress_test.py
Added --no-server-metrics flag to the aiperf profile command generation to prevent server metrics manager from hanging on incompatible metrics endpoint formats.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR description contains only the template with no actual content filled in; the Description and Test Coverage sections are empty, and only the checklist box is marked without explanatory details. Fill in the Description section explaining the issue and solution, and the Test Coverage section listing relevant tests that validate the changes.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly indicates the main change: disabling aiperf server metrics to resolve a hang issue referenced in the NVBugs ticket.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
tests/integration/defs/stress_test/stress_test.py (1)

1-1: ⚠️ Potential issue | 🟡 Minor

Update the NVIDIA copyright year in this modified file.

This header still ends at 2024, but this file has meaningful 2026 changes. Please update the year range accordingly.

As per coding guidelines "Add NVIDIA copyright header on ALL new files and update year on modified files" and "All TensorRT-LLM source files must contain an NVIDIA copyright header with the year of latest meaningful modification".

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/integration/defs/stress_test/stress_test.py` at line 1, Update the SPDX
header year range in the file's top comment from "2022-2024" to include the
latest meaningful modification year (change "2022-2024" to "2022-2026") so the
NVIDIA copyright header reflects the 2026 changes.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@tests/integration/defs/stress_test/stress_test.py`:
- Line 1: Update the SPDX header year range in the file's top comment from
"2022-2024" to include the latest meaningful modification year (change
"2022-2024" to "2022-2026") so the NVIDIA copyright header reflects the 2026
changes.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: e7152f01-b48d-4c0a-bdb2-718b05b1237a

📥 Commits

Reviewing files that changed from the base of the PR and between 55419a9 and 634b3a2.

📒 Files selected for processing (1)
  • tests/integration/defs/stress_test/stress_test.py

@dominicshanshan

Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "A10-PyTorch-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #44879 [ run ] triggered by Bot. Commit: 634b3a2 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #44879 [ run ] completed with state SUCCESS. Commit: 634b3a2
/LLM/main/L0_MergeRequest_PR pipeline #35215 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

@dominicshanshan

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #44929 [ run ] triggered by Bot. Commit: 634b3a2 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #44929 [ run ] completed with state SUCCESS. Commit: 634b3a2
/LLM/main/L0_MergeRequest_PR pipeline #35257 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

@dominicshanshan

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #45078 [ run ] triggered by Bot. Commit: 634b3a2 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #45078 [ run ] completed with state SUCCESS. Commit: 634b3a2
/LLM/main/L0_MergeRequest_PR pipeline #35375 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

@dominicshanshan

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #45125 [ run ] triggered by Bot. Commit: 634b3a2 Link to invocation

@crazydemo crazydemo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR fixes the aiperf issue.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #45125 [ run ] completed with state SUCCESS. Commit: 634b3a2
/LLM/main/L0_MergeRequest_PR pipeline #35418 completed with status: 'SUCCESS'

CI Report

Link to invocation

@dominicshanshan
dominicshanshan merged commit b3a3b94 into NVIDIA:main Apr 23, 2026
9 of 10 checks passed
ziyixiong-nv pushed a commit to ziyixiong-nv/TensorRT-LLM that referenced this pull request Apr 24, 2026
…op hang. (NVIDIA#13314)

Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants