Advise around a sync engine with thousands of docs

@Hackinet Piyush Jha
Sun, 28 Dec 2025 09:19:36 +0000

I am building an app where 30K-40K pages can exist on UI. Each page is a JSON object, and the content property is a Yjs doc.

I am using Tiptap, and only the editors in viewport are rendered. For realtime sync and awareness am using Hocuspocus (single socket, multiplexed by document name).

Right now on every doc.on('update',...), I persist the whole state to IndexedDB. I do not store individual updates.

I also have an events system. A local per device queue + a server events table with a monotonically increasing eventIndex. Each device tracks a event “checkpoint” and advances it after consuming an event, like UPDATE_PAGE and CREATE_PAGE etc, but these are strictly for JSON properties/objects. These events are realtime with a socket but also can be replayed once a device comes online, and is behind.

I have some problems around sync for pages that are not rendered / do not have an active provider:

A device is offline, it edits Page 1, creates 20 more pages, and the page 1 is never rendered again so provider based sync never runs.

A device is online and updates Page X, another device is online too but Page X isn’t open / rendered, so no it never receives an update.

I considered two approaches:

Utilize my events table. But update would produce an event and would bloat the events table. Even with debounce, it would be too much.

On each Yjs update, I can increment a “unsynced” counter for that page in IndexedDB (I have had race issues with boolean flags before). A background worker can unsynced pages to server. Other device’s worker can periodically poll for all “updatedAt” and pull/merge changes.

But even with these I am worried about some edge cases:

If I mark a document as unsynced when offline, and then we come online, the provider might sync it anyway and my worker could still push (the whole state), causing duplicate writes on server?

Should I mark a document unsynced even if it’s currently rendered and has a active provider. If I don’t. What if the socket drops and the tab is refreshed?

I don’t see a way in Hocuspocus to know if a update has been “acknowledged by the server” so it’s hard to decide when to mark an update as “synced”, or if it even sends individual updates at all or maybe it batches them?

Is there a better way to handle sync? Maybe I need to write my own provider and server implementation?