Skip to content

Conversation

@levkk
Copy link
Collaborator

@levkk levkk commented Jul 19, 2025

Description

Implement logical replication and COPY fan-out between N origin shards and M destination shards. This reshards an existing cluster, one table at a time and streams table updates to the right shard(s) by parsing the replication stream.

Implementation

  1. Using binary COPY, stream tables to destination cluster while sharding between nodes.
  2. Using logical replication, stream changes to each table and convert logical repl. messages into upserts using prepared statements.

Upserts allow us to rewind the stream back and replay without duplicates. All tables have to have a primary key, which will be used as the replica identity. DO NOT use the whole row as a replica identity, that doesn't work for upserts.

Missing pieces

  1. Foreign keys not supported (easy fix, see below)
  2. Schemas not automatically synchronized, requires manual pg_dump/pg_restore (see below)
  3. Updates and deletes (easy fix, see below)

To solve the FK problem, we need to sync the schema ourselves without the constraints (or additional indexes). Once the replication is caught up, re-create the constraints & add back the indexes.

Updates and deletes just need us to hook into the query parser a bit to get shard on the where clause. This is still a PoC, this will be done in the next PR.

Misc

  1. Connections from connection pools that have replication_mode enabled will be closed instead of re-used. This ensures that connections left in partial state after logical replication are not re-used by other clients.
  2. Add Cluster::execute method to execute a query on all primaries in the cluster. Just a helper, useful for testing,
  3. Add Pool::standalone to create a connection untracked by the pool. Can be used for whatever reason, incl. replication.

@levkk levkk force-pushed the levkk-logical-stream-2 branch from 4b48abc to d6439af Compare July 21, 2025 22:08
@levkk levkk changed the title reshard with loglcal Reshard with logical replication, part I Jul 22, 2025
@levkk levkk force-pushed the levkk-logical-stream-2 branch from b26f3ee to a18aa9f Compare July 23, 2025 17:00
@levkk levkk marked this pull request as ready for review July 23, 2025 17:33
@levkk levkk merged commit 45168fa into main Jul 23, 2025
5 checks passed
@levkk levkk deleted the levkk-logical-stream-2 branch July 23, 2025 17:45
@levkk levkk mentioned this pull request Aug 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant