Add Support for RangeIndex Diskbased Replication#1807
Merged
Conversation
a5d1834 to
8c40b4c
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds disk-based replication support for RangeIndex (BfTree) files so replicas can restore RangeIndex state after checkpoint recovery.
Changes:
- Add replication plumbing for RangeIndex flush/snapshot
.bftreefiles (enumeration, reader, transmit source, receiver sink, handler integration). - Update server gating: allow RangeIndex preview in cluster mode except for diskless replication.
- Add a new cluster test project and plumb
enableRangeIndexPreviewthrough test helpers/context.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| test/standalone/Garnet.test/TestUtils.cs | Adds enableRangeIndexPreview parameter forwarding for test server options. |
| test/cluster/Garnet.test.cluster/ClusterTestContext.cs | Adds enableRangeIndexPreview option to cluster test instance creation paths. |
| test/cluster/Garnet.test.cluster.replication.rangeindex/Garnet.test.cluster.replication.rangeindex.csproj | Introduces a new test project for RangeIndex replication scenarios. |
| test/cluster/Garnet.test.cluster.replication.rangeindex/ClusterRangeIndexReplicationTests.cs | Adds end-to-end cluster replication tests for RangeIndex behavior. |
| libs/server/StoreWrapper.cs | Exposes rangeIndexManager for cross-assembly replication usage. |
| libs/server/Resp/RangeIndex/RangeIndexManager.cs | Adds RI file enumeration helpers and replication file listing API. |
| libs/host/GarnetServer.cs | Updates validation to disallow RI preview only in diskless replication mode. |
| libs/cluster/Server/Replication/ReplicaOps/DiskbasedReplication/ReceiveCheckpointHandler.cs | Adds handling to create RI sinks from metadata headers. |
| libs/cluster/Server/Replication/ReplicaOps/DiskbasedReplication/RangeIndexFileDataSink.cs | Implements replica-side sink that writes received RI file bytes to disk. |
| libs/cluster/Server/Replication/PrimaryOps/DiskbasedReplication/ReplicaSyncSession.cs | Adds RangeIndexSnapshotReader to checkpoint transmission when RI preview enabled. |
| libs/cluster/Server/Replication/PrimaryOps/DiskbasedReplication/RangeIndexSnapshotReader.cs | Enumerates RI files for a checkpoint and yields transmit sources. |
| libs/cluster/Server/Replication/PrimaryOps/DiskbasedReplication/RangeIndexFileTransmitSource.cs | Sends RI metadata header + chunked file bytes + end marker over snapshot protocol. |
| libs/cluster/Server/Replication/PrimaryOps/DiskbasedReplication/RangeIndexFileDataSource.cs | Implements FileStream-backed chunk reader plus metadata generation for RI files. |
| libs/cluster/Server/Replication/CheckpointFileType.cs | Adds two new CheckpointFileType values for RI flush/snapshot files. |
| Garnet.slnx | Adds the new RangeIndex replication test project to the solution. |
986b212 to
81861f8
Compare
81861f8 to
ec87222
Compare
badrishc
reviewed
May 22, 2026
ec87222 to
8d9e5a4
Compare
tiagonapoli
reviewed
May 25, 2026
tiagonapoli
reviewed
May 25, 2026
tiagonapoli
reviewed
May 25, 2026
tiagonapoli
reviewed
May 25, 2026
33733bb to
e05249e
Compare
tiagonapoli
reviewed
May 27, 2026
tiagonapoli
approved these changes
May 27, 2026
badrishc
approved these changes
May 28, 2026
Add EnumerateFilesForReplication() to RangeIndexManager that lists all flush.bftree and checkpoint snapshot files needed for a given checkpoint token and hlog address range. Refactor flush file parsing into a shared EnumerateFlushFiles() helper used by both OnTruncateImpl and the new replication enumeration method. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nsmit Implement the primary-side pipeline for shipping RangeIndex files during disk-based replication: - Add STORE_RANGEINDEX_FLUSH (7) and STORE_RANGEINDEX_SNAPSHOT (8) to CheckpointFileType enum - RangeIndexFileDataSource: FileStream-based ISnapshotDataSource for chunked reading of .bftree files - RangeIndexFileTransmitSource: sends metadata header (keyHash + address as binary, startAddress=-1) then streams file content in chunks - RangeIndexSnapshotReader: ISnapshotReader that enumerates RI files via EnumerateFilesForReplication and yields transmit sources - Make RangeIndexFileEntry, EnumerateFilesForReplication, and RiLogRoot public for cross-project access Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add RangeIndexFileDataSink with FromMetadata factory for zero-copy deserialization of metadata header directly from ReadOnlySpan - Integrate RangeIndexSnapshotReader into ReplicaSyncSession.SendCheckpointAsync conditioned on serverOptions.EnableRangeIndexPreview - Move GetMetadata serialization into RangeIndexFileDataSource (not interface) - Make LogFlushPath/CheckpointSnapshotPath public, rangeIndexManager public - Remove unused Garnet.server/System.Text usings from handler Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
e05249e to
17cbde7
Compare
tiagonapoli
approved these changes
May 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for shipping RangeIndex (BfTree) files during disk-based cluster replication, so replicas can fully recreate range index state after checkpoint recovery.
Work Items
Changes
Primary Side (Sender)
RangeIndexManager.EnumerateFilesForReplication()yields all flush and snapshot .bftree files for a given checkpointRangeIndexFileDataSource— FileStream-based chunked reader withGetMetadata()for binary serialization of keyHash + addressRangeIndexFileTransmitSource— sends metadata header (startAddress=-1) then streams file content in 128KB chunksRangeIndexSnapshotReader— ISnapshotReader that enumerates RI files and yields transmit sourcesReplicaSyncSession.SendCheckpointAsyncaddsRangeIndexSnapshotReaderto the transmission driver whenEnableRangeIndexPreviewis setReplica Side (Receiver)
RangeIndexFileDataSinkwithFromMetadata()factory — zero-copy deserialization from metadata span, derives target path viaRangeIndexManagerhelpers, writes chunks via FileStreamReceiveCheckpointHandler.ProcessSnapshotDatahandles RI metadata headers and file segmentsShared Infrastructure
STORE_RANGEINDEX_FLUSH=7,STORE_RANGEINDEX_SNAPSHOT=8LogFlushPath,CheckpointSnapshotPath,rangeIndexManagerpublic for cross-assembly accessWire Protocol
Each RI file is transmitted as: