[https://nvbugs/6226933][fix] canonicalize multimodal cache-key serialization to prevent hash collisions#14800

venkywonka · 2026-05-31T19:10:13Z

Problem

Multimodal KV-cache block reuse (enable_block_reuse=True by default) keys cache blocks on a BLAKE3 content hash of each multimodal item (apply_mm_hashes). The serialization feeding that hash omitted semantic structure, so semantically different inputs produced identical bytes and therefore identical cache keys, which can cause incorrect cross-request KV-cache reuse / cache poisoning for multimodal serving. Reported in #14603.

On main, the following distinct inputs collided to the same hash:

PIL images with transposed dimensions (e.g. 30×100 vs 100×30)
ndarrays/tensors sharing raw bytes but differing in shape/dtype (e.g. (3,224,224) vs (1,3,224,224))
VideoData differing only in sampling metadata (fps / duration / frames_indices / total_num_frames)
a VideoData (no audio) vs a plain list of the same frames

Fix

Make multimodal serialization canonical and self-describing — every value is encoded as [type tag][typed metadata][length-prefixed payload]:

images encode mode + width + height
tensors and ndarrays encode dtype + shape
lists, tuples and dicts get distinct, length-framed encodings (dict keys sorted); ints are width/sign-unambiguous and bool is distinct from int
VideoData carries a modality tag and includes its sampling metadata in the hash; AudioData binds samples to sample_rate
the per-item hash is prefixed with a versioned scheme tag, and the two previously-divergent list-framing paths are unified

Tests

Adds CPU-only regression tests in tests/unittest/llmapi/test_llm_kv_cache_events.py covering each collision class (transposed image dims, reshaped ndarray, reshaped tensor, video-metadata sensitivity, and VideoData-vs-frame-list), plus a positive determinism test (identical inputs still match). Updates one existing assertion that had encoded the VideoData-vs-list collision as expected behavior.

Summary by CodeRabbit

Release Notes

Refactor
- Enhanced multimodal content caching with improved, collision-resistant hashing for better consistency across cache key evolution.
Tests
- Expanded test coverage for multimodal hashing to validate correct behavior across various data types and metadata scenarios.

venkywonka · 2026-06-01T00:29:38Z

/bot run

tensorrt-cicd · 2026-06-01T00:35:15Z

PR_Github #51272 [ run ] triggered by Bot. Commit: fdf0786 Link to invocation

tensorrt-cicd · 2026-06-01T03:26:45Z

PR_Github #51272 [ run ] completed with state SUCCESS. Commit: fdf0786
/LLM/main/L0_MergeRequest_PR pipeline #40691 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

venkywonka · 2026-06-01T06:10:27Z

/bot run

tensorrt-cicd · 2026-06-01T06:16:01Z

PR_Github #51319 [ run ] triggered by Bot. Commit: fdf0786 Link to invocation

tensorrt-cicd · 2026-06-01T11:06:17Z

PR_Github #51319 [ run ] completed with state SUCCESS. Commit: fdf0786
/LLM/main/L0_MergeRequest_PR pipeline #40733 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

venkywonka · 2026-06-01T11:21:51Z

/bot run

venkywonka · 2026-06-01T11:34:19Z

/bot run

…lization to prevent hash collisions Multimodal cache keys (apply_mm_hashes) fed a non-canonical serialization that omitted shape/dtype/dimensions and modality identity, so semantically different multimodal inputs could serialize to identical bytes and incorrectly share KV-cache blocks (enable_block_reuse is on by default). Encode every multimodal value canonically and self-describingly as [type tag][typed metadata][length-prefixed payload]: images carry mode+width+height, tensors/ndarrays carry dtype+shape, containers are length-framed (dict keys sorted, ints width/sign-unambiguous, bool distinct from int), VideoData carries a modality tag and its sampling metadata, and AudioData binds samples to sample_rate. The per-item hash is prefixed with a versioned scheme tag and the two divergent list-framing paths are unified. Adds regression tests for each collision class. Fixes NVIDIA#14603. Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com>

… serialization serialize_item raised ValueError on numpy scalar values (np.int64, np.float32, np.bool_, ...) reachable through VideoData metadata, because in numpy 2.x these are not subclasses of Python int/float/bool and bypassed every type check. Normalize numpy scalars to their equivalent Python scalar via .item() and recurse, so numpy-typed metadata hashes identically to its Python counterpart. Adds a regression test. Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com>

venkywonka · 2026-06-01T15:04:54Z

/bot run

tensorrt-cicd · 2026-06-01T15:10:55Z

PR_Github #51366 [ run ] triggered by Bot. Commit: d6e7d0d Link to invocation

coderabbitai · 2026-06-01T16:21:58Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6d07d412-3286-4eb4-935d-63ae8fb17137

📥 Commits

Reviewing files that changed from the base of the PR and between 06456e1 and d6e7d0d.

📒 Files selected for processing (3)

tensorrt_llm/inputs/multimodal.py
tensorrt_llm/inputs/multimodal_data.py
tests/unittest/llmapi/test_llm_kv_cache_events.py

📝 Walkthrough

Walkthrough

This PR overhauls multimodal KV-cache hashing with versioned scheme identification and canonical serialization. A versioned tag is prepended to all hashes. The new serialize_item() function implements typed, length-prefixed encoding for primitives, images, tensors, arrays, and containers. Audio and video hashing integrate with the new scheme and metadata now participates in cache identity. Comprehensive tests validate the new behavior.

Changes

Multimodal Hash Scheme Versioning and Canonical Serialization

Layer / File(s)	Summary
Hash scheme versioning and integration `tensorrt_llm/inputs/multimodal.py`	Introduces `_HASH_SCHEME_TAG` (v1) prefix for self-describing cache-key evolution. Integrates tag into `_update_hash()` entry point and simplifies hashing to route non-`BaseModalityData` items exclusively through `serialize_item()`. Removes obsolete hash collision TODO comment.
Canonical serialization encoding `tensorrt_llm/inputs/multimodal_data.py`	Rewrites `serialize_item()` with typed, big-endian, length-prefixed encoding. Adds helper encoders (`_u8`, `_u32`, `_u64`, `_len_prefixed`) for structured bytes. Handles primitives with type tags, images with mode/dimensions/RGBA, tensors/arrays with dtype/shape/contiguous data, and recursive containers with sorted dict keys. Normalizes NumPy scalars via `np.generic`.
Modality-specific hash updates `tensorrt_llm/inputs/multimodal_data.py`	`AudioData.update_hash()` serializes `(samples, sample_rate)` tuple via `serialize_item()`. `VideoData.update_hash()` includes filtered sampling metadata (`_VIDEO_HASH_METADATA_FIELDS`) before hashing frames using the updated serialization logic.
Test coverage and validation `tests/unittest/llmapi/test_llm_kv_cache_events.py`	Adds `_make_video()` helper for consistent `VideoData` construction. Refactors existing audio/video hash test to use helper and assert `VideoData` differs from bare frame list. Adds tests validating image dimension sensitivity, tensor/ndarray shape distinction, video metadata participation, NumPy scalar determinism, modality-type structural differences, and deterministic hashing across identical inputs.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main fix: canonicalizing multimodal cache-key serialization to prevent hash collisions, which aligns perfectly with the core problem and solution described in the PR.
Description check	✅ Passed	The PR description comprehensively covers the problem, fix, and tests. However, it does not explicitly address the PR Checklist items required by the description template (API compatibility labels, dependency scanning, CODEOWNERS updates, documentation updates, etc.).
Docstring Coverage	✅ Passed	Docstring coverage is 84.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

…ainer-agnostic Address PR review feedback: - Merge the torch.Tensor and np.ndarray serialization paths so the container type is not part of the cache-key identity; only dtype, shape and raw bytes are. A tensor and an ndarray with identical content now hash identically. - Collapse the tuple/list tags into a single ordered-sequence tag (the container type is not part of the identity). The VideoData-vs-frame-list distinction is unaffected; it comes from the modality tag in _update_hash. - Single backticks in the serialize_item docstring; concise test helper docstring; drop redundant per-test docstrings. - Add serialize_item container-agnostic regression tests. Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com>

tensorrt-cicd · 2026-06-01T19:11:45Z

PR_Github #51366 [ run ] completed with state SUCCESS. Commit: d6e7d0d
/LLM/main/L0_MergeRequest_PR pipeline #40776 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

venkywonka · 2026-06-01T19:23:42Z

/bot run

tensorrt-cicd · 2026-06-01T19:29:22Z

PR_Github #51410 [ run ] triggered by Bot. Commit: 0725e25 Link to invocation

tensorrt-cicd · 2026-06-01T19:57:05Z

PR_Github #51410 [ run ] completed with state FAILURE. Commit: 0725e25
/LLM/main/L0_MergeRequest_PR pipeline #40820 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

venkywonka · 2026-06-01T20:00:26Z

/bot run

tensorrt-cicd · 2026-06-01T20:07:11Z

PR_Github #51411 [ run ] triggered by Bot. Commit: 0725e25 Link to invocation

tensorrt-cicd · 2026-06-01T20:21:38Z

PR_Github #51411 [ run ] completed with state FAILURE. Commit: 0725e25
/LLM/main/L0_MergeRequest_PR pipeline #40821 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

venkywonka · 2026-06-01T20:32:40Z

/bot run

tensorrt-cicd · 2026-06-01T20:39:02Z

PR_Github #51415 [ run ] triggered by Bot. Commit: 0725e25 Link to invocation

tensorrt-cicd · 2026-06-01T21:20:17Z

PR_Github #51415 [ run ] completed with state FAILURE. Commit: 0725e25
/LLM/main/L0_MergeRequest_PR pipeline #40825 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

venkywonka · 2026-06-01T23:05:47Z

/bot run

tensorrt-cicd · 2026-06-01T23:11:43Z

PR_Github #51438 [ run ] triggered by Bot. Commit: 0725e25 Link to invocation

tensorrt-cicd · 2026-06-02T01:22:57Z

PR_Github #51438 [ run ] completed with state SUCCESS. Commit: 0725e25
/LLM/main/L0_MergeRequest_PR pipeline #40847 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

venkywonka · 2026-06-02T01:38:55Z

/bot run

tensorrt-cicd · 2026-06-02T01:45:23Z

PR_Github #51468 [ run ] triggered by Bot. Commit: 0725e25 Link to invocation

tensorrt-cicd · 2026-06-02T04:40:14Z

PR_Github #51468 [ run ] completed with state SUCCESS. Commit: 0725e25
/LLM/main/L0_MergeRequest_PR pipeline #40873 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

venkywonka · 2026-06-02T04:42:39Z

/bot run

tensorrt-cicd · 2026-06-02T04:49:30Z

PR_Github #51502 [ run ] triggered by Bot. Commit: 0725e25 Link to invocation

tensorrt-cicd · 2026-06-02T07:49:16Z

PR_Github #51502 [ run ] completed with state SUCCESS. Commit: 0725e25
/LLM/main/L0_MergeRequest_PR pipeline #40908 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

github-actions Bot assigned venkywonka May 31, 2026

venkywonka added 2 commits June 1, 2026 08:03

venkywonka force-pushed the venky/mm-cache-key-collision branch from fdf0786 to d6e7d0d Compare June 1, 2026 15:04

venkywonka marked this pull request as ready for review June 1, 2026 16:15

venkywonka requested a review from a team as a code owner June 1, 2026 16:15

venkywonka requested review from moraxu and yechank-nvidia June 1, 2026 16:15

venkywonka commented Jun 1, 2026

View reviewed changes

Comment thread tests/unittest/llmapi/test_llm_kv_cache_events.py Outdated

2ez4bz approved these changes Jun 1, 2026

View reviewed changes

moraxu approved these changes Jun 1, 2026

View reviewed changes

venkywonka merged commit cd38dfb into NVIDIA:main Jun 2, 2026
7 checks passed

taylor-yb-lee mentioned this pull request Jun 5, 2026

[https://nvbugs/6272668][infra] Unwaive DSR1 and Qwen3.5 again #15010

Merged

1 task

Uh oh!

Conversation

venkywonka commented May 31, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Tests

Summary by CodeRabbit

Release Notes

Uh oh!

venkywonka commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

venkywonka commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

venkywonka commented Jun 1, 2026

Uh oh!

venkywonka commented Jun 1, 2026

Uh oh!

venkywonka commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

Uh oh!

coderabbitai Bot commented Jun 1, 2026

Walkthrough

Changes

Estimated code review effort

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

venkywonka commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

venkywonka commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

venkywonka commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

venkywonka commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 1, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

venkywonka commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

venkywonka commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

Uh oh!

venkywonka commented May 31, 2026 •

edited by coderabbitai Bot

Loading