[None][infra] Waive 9 failed cases for main in post-merge by xinhe-nv · Pull Request #14515 · NVIDIA/TensorRT-LLM

xinhe-nv · 2026-05-25T05:52:35Z

Auto-generated Waive PR

Created by: TensorRT LLM CI Report (requested by [email protected])
Target branch: main
Bug(s): 6185146, 6198785, 6207678, 6211185, 6211189, 6211693, 6215689, 6215736

Waive entries added

accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_fp8_blockscale[throughput_mtp] SKIP (https://nvbugs/6215736)
accuracy/test_llm_api_pytorch.py::TestNemotronV3Super::test_nvfp4_8gpus_mtp SKIP (https://nvbugs/6211693)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[tp4-mtp_nextn=0-attention_dp=False-cuda_graph=True-overlap_scheduler=False-torch_compile=False] SKIP (https://nvbugs/6198785)
accuracy/test_llm_api_pytorch.py::TestLagunaXS::test_fp8 SKIP (https://nvbugs/6211185)
accuracy/test_llm_api_pytorch.py::TestLagunaXS::test_bf16 SKIP (https://nvbugs/6211185)
accuracy/test_llm_api_pytorch.py::TestLagunaXS::test_nvfp4 SKIP (https://nvbugs/6211185)
accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_cute_dsl_nvfp4_4gpus[tp4-fp8kv=False-attention_dp=False-cuda_graph=False-overlap_scheduler=False-torch_compile=True] SKIP (https://nvbugs/6185146)
stress_test/stress_test.py::stress_test::test_run_stress_test[DeepSeek-R1-0528-FP4_tp4-stress_time_3600s_timeout_10800s-GUARANTEED_NO_EVICT-pytorch-stress-test-with-accuracy] SKIP (https://nvbugs/6207678)
stress_test/stress_test.py::stress_test::test_run_stress_test[DeepSeek-R1-0528-FP4_tp4-stress_time_3600s_timeout_10800s-MAX_UTILIZATION-pytorch-stress-test-with-accuracy] SKIP (https://nvbugs/6207678)

Already waived (skipped)

accuracy/test_llm_api_pytorch_multimodal.py::TestExaone4_5_33B::test_auto_dtype[full_budget]
accuracy/test_llm_api_pytorch_multimodal.py::TestGemma3_27BInstruct::test_fp8_prequantized
accuracy/test_llm_api_pytorch_multimodal.py::TestExaone4_5_33B::test_auto_dtype[forced_chunked_prefill]

This PR was auto-generated by TensorRT LLM CI Report. Please review the waive entries before merging.

Summary by CodeRabbit

Tests
- Updated test waiver configuration to account for additional test cases across multiple model configurations and test environments.

jieli-matrix · 2026-05-25T05:53:28Z

/bot run --stage-list ""

coderabbitai · 2026-05-25T05:54:15Z

📝 Walkthrough

Walkthrough

Updated tests/integration/test_lists/waives.txt by adding nine new SKIP/waiver entries for failing test cases across multiple models and test suites, including accuracy tests for DeepSeekR1, DeepSeekV3Lite, LagunaXS, and NemotronV3Super, plus stress tests for DeepSeek-R1.

Changes

Test Waivers for Known Issues

Layer / File(s)	Summary
Adding test waivers for failing accuracy and stress tests `tests/integration/test_lists/waives.txt`	Nine new SKIP waiver entries added for DeepSeekR1 fp8 blockscale throughput, DeepSeekV3Lite bfloat16 and nvfp4 variants, LagunaXS mixed-precision tests, NemotronV3Super nvfp4 testing, and DeepSeek-R1 stress test modes with GUARANTEED_NO_EVICT and MAX_UTILIZATION.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#14455: Both PRs are data-only updates to tests/integration/test_lists/waives.txt, adding new SKIP/waiver entries for different failing test cases in the same accuracy test suite.
NVIDIA/TensorRT-LLM#14503: Both PRs add new entries to tests/integration/test_lists/waives.txt to skip failing tests in accuracy/test_llm_api_pytorch.py for different specific test cases.
NVIDIA/TensorRT-LLM#13986: Both PRs directly modify tests/integration/test_lists/waives.txt with overlapping waiver entries for accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite including bfloat16_4gpus and nvfp4 configurations.

Suggested reviewers

jieli-matrix
StanleySun639

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Description check	❓ Inconclusive	The PR description is auto-generated and provides necessary context: waived test cases with their associated bug references, but it lacks most required template sections including explicit Description, Test Coverage, and PR Checklist items.	Although auto-generated, consider adding explicit 'Description' and 'Test Coverage' sections, and marking the PR Checklist items as applicable (even if checking 'N/A' for infra changes).

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: waiving 9 failed test cases for the main branch in post-merge.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/test_lists/waives.txt`:
- Around line 346-347: The two waiver lines include an incorrect class segment
"::stress_test::" in the pytest node IDs; update both entries so they use the
exact function-level node IDs (remove the "::stress_test::" segment) e.g. change
"stress_test/stress_test.py::stress_test::test_run_stress_test[...]" to
"stress_test/stress_test.py::test_run_stress_test[...]" for both occurrences so
validate-test-lists will match the collected tests.

🪄 Autofix (Beta)

❌ Autofix failed (check again to retry)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f7a9c658-5fad-4273-a89b-ee4605b8e10d

📥 Commits

Reviewing files that changed from the base of the PR and between ce788e0 and c5733cf.

📒 Files selected for processing (1)

tests/integration/test_lists/waives.txt

tensorrt-cicd · 2026-05-25T05:59:30Z

PR_Github #50153 [ run ] triggered by Bot. Commit: c5733cf Link to invocation

tensorrt-cicd · 2026-05-25T06:14:04Z

PR_Github #50153 [ run ] completed with state FAILURE. Commit: c5733cf
/LLM/main/L0_MergeRequest_PR pipeline #39699 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

coderabbitai · 2026-05-25T06:40:07Z

Note

Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it.

An unexpected error occurred while generating fixes: Not Found - https://docs.github.com/rest/git/refs#get-a-reference

jieli-matrix · 2026-05-25T06:46:45Z

/bot run --stage-list ""

tensorrt-cicd · 2026-05-25T06:53:16Z

PR_Github #50164 [ run ] triggered by Bot. Commit: 6255b97 Link to invocation

tensorrt-cicd · 2026-05-25T08:17:30Z

PR_Github #50164 [ run ] completed with state SUCCESS. Commit: 6255b97
/LLM/main/L0_MergeRequest_PR pipeline #39708 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

jieli-matrix · 2026-05-25T08:24:00Z

/bot reuse-pipeline

tensorrt-cicd · 2026-05-25T08:29:40Z

PR_Github #50183 [ reuse-pipeline ] triggered by Bot. Commit: f5458e7 Link to invocation

tensorrt-cicd · 2026-05-25T08:35:12Z

PR_Github #50183 [ reuse-pipeline ] completed with state SUCCESS. Commit: f5458e7
Reusing PR_Github #50164 (Partly Tested) for commit f5458e7

Link to invocation

jieli-matrix · 2026-05-25T09:21:33Z

/bot reuse-pipeline

tensorrt-cicd · 2026-05-25T09:27:50Z

PR_Github #50196 [ reuse-pipeline ] triggered by Bot. Commit: 2153196 Link to invocation

tensorrt-cicd · 2026-05-25T09:35:32Z

PR_Github #50196 [ reuse-pipeline ] completed with state SUCCESS. Commit: 2153196
Reusing PR_Github #50164 (Partly Tested) for commit 2153196

Link to invocation

Bug(s): 6185146, 6198785, 6207678, 6211185, 6211189, 6211693, 6215689, 6215736 Requested by: [email protected] Signed-off-by: xinhe-nv <[email protected]>

Signed-off-by: Jie Li <[email protected]>

jieli-matrix · 2026-05-25T10:37:14Z

/bot reuse-pipeline

tensorrt-cicd · 2026-05-25T10:42:47Z

PR_Github #50209 [ reuse-pipeline ] triggered by Bot. Commit: 34f7a91 Link to invocation

tensorrt-cicd · 2026-05-25T10:48:37Z

PR_Github #50209 [ reuse-pipeline ] completed with state SUCCESS. Commit: 34f7a91
Reusing PR_Github #50164 (Partly Tested) for commit 34f7a91

Link to invocation

Signed-off-by: xinhe-nv <[email protected]>

github-actions Bot assigned xinhe-nv May 25, 2026

jieli-matrix assigned jieli-matrix and unassigned xinhe-nv May 25, 2026

jieli-matrix self-requested a review May 25, 2026 05:53

jieli-matrix approved these changes May 25, 2026

View reviewed changes

jieli-matrix force-pushed the trtllm-ci-report/waive-20260525-055232 branch from 881ea5c to c5733cf Compare May 25, 2026 05:53

jieli-matrix enabled auto-merge (squash) May 25, 2026 05:53

jieli-matrix mentioned this pull request May 25, 2026

[None][test] Waive 9 failed cases for main in QA CI #14501

Closed

coderabbitai Bot reviewed May 25, 2026

View reviewed changes

Comment thread tests/integration/test_lists/waives.txt Outdated

jieli-matrix force-pushed the trtllm-ci-report/waive-20260525-055232 branch from 94c9000 to 6255b97 Compare May 25, 2026 06:45

jieli-matrix mentioned this pull request May 25, 2026

[None][test] Waive 7 failed cases for main in QA CI #14498

Merged

jieli-matrix force-pushed the trtllm-ci-report/waive-20260525-055232 branch from 6255b97 to f5458e7 Compare May 25, 2026 08:21

jieli-matrix force-pushed the trtllm-ci-report/waive-20260525-055232 branch from f5458e7 to 2153196 Compare May 25, 2026 09:17

xinhe-nv and others added 2 commits May 25, 2026 18:35

[None][infra] Waive 9 failed cases for main in post-merge

b6ed168

Bug(s): 6185146, 6198785, 6207678, 6211185, 6211189, 6211693, 6215689, 6215736 Requested by: [email protected] Signed-off-by: xinhe-nv <[email protected]>

Update waives.txt

34f7a91

Signed-off-by: Jie Li <[email protected]>

jieli-matrix force-pushed the trtllm-ci-report/waive-20260525-055232 branch from 2153196 to 34f7a91 Compare May 25, 2026 10:35

jieli-matrix merged commit 4517988 into NVIDIA:main May 25, 2026
7 checks passed

KleinBlueC pushed a commit to KleinBlueC/TensorRT-LLM that referenced this pull request May 26, 2026

[None][infra] Waive 9 failed cases for main in post-merge (NVIDIA#14515)

e60fad0

Signed-off-by: xinhe-nv <[email protected]>

This was referenced May 26, 2026

[https://nvbugs/6211185][fix] Fix failed GSM8K accuracy tests for LagunaXS on B200/GB200/B300 #14580

Merged

[https://nvbugs/6165866][infra] Waive 1 failed cases for main in pre-merge 40081 #14653

Merged

bmarimuthu-nv pushed a commit to nv-auto-deploy/TensorRT-LLM that referenced this pull request May 28, 2026

[None][infra] Waive 9 failed cases for main in post-merge (NVIDIA#14515)

062d921

Signed-off-by: xinhe-nv <[email protected]>

coderabbitai Bot mentioned this pull request May 30, 2026

[None][chore] Waive failing multi-gpu test #14788

Merged

1 task

Conversation

xinhe-nv commented May 25, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Auto-generated Waive PR

Waive entries added

Already waived (skipped)

Summary by CodeRabbit

Uh oh!

jieli-matrix commented May 25, 2026

Uh oh!

coderabbitai Bot commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

❌ Failed checks (1 inconclusive)

Uh oh!

coderabbitai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tensorrt-cicd commented May 25, 2026

Uh oh!

tensorrt-cicd commented May 25, 2026

Uh oh!

coderabbitai Bot commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jieli-matrix commented May 25, 2026

Uh oh!

tensorrt-cicd commented May 25, 2026

Uh oh!

tensorrt-cicd commented May 25, 2026

Uh oh!

jieli-matrix commented May 25, 2026

Uh oh!

tensorrt-cicd commented May 25, 2026

Uh oh!

tensorrt-cicd commented May 25, 2026

Uh oh!

jieli-matrix commented May 25, 2026

Uh oh!

tensorrt-cicd commented May 25, 2026

Uh oh!

tensorrt-cicd commented May 25, 2026

Uh oh!

jieli-matrix commented May 25, 2026

Uh oh!

tensorrt-cicd commented May 25, 2026

Uh oh!

tensorrt-cicd commented May 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xinhe-nv commented May 25, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 25, 2026 •

edited

Loading

coderabbitai Bot left a comment •

edited

Loading

coderabbitai Bot commented May 25, 2026 •

edited

Loading