Repository navigation
Start big shards indexing with delay and load distribution #5811
Description
Summary
I have a large shard which needs to be indexed using a new view. When the indexing operation started, the same shard was indexed on several nodes.
This can be optimised using this logic.
- If the shard has X changes, then the required delay is indexing for Y milliseconds. The delay may be larger on a node with a high CPU load and smaller on a node with less CPU load;
- check the other cluster node started indexing for the same shard;
- if the shard is already indexed on another node, then skip this shard for indexing now.
The same logic applied to all shards. At the end, all nodes are randomly distributed shards for indexing.
On the next view request, the same shard will be indexed on the other node.
Desired Behaviour
Nodes distribute the shards indexing task and do not index the same shard on several nodes at the same time.
Additional context
This allows for distributing CPU load on nodes and makes CPU spikes smoother.
Activity
rnewson commentedon Dec 5, 2025
I don't see an optimisation here. Each node has (or should have) independent CPU's etc, there's no reason not to index all N copies at once in parallel. As long as you didn't query the new view the index building process runs at a background level, the
ioqsystem vastly preferring interactive requests over the view build I/O.sergey-safarov commentedon Dec 5, 2025
Let me explain.
Please imagine that processing of 3,000,000 documents requires 24 hours. Like in my screenshot.
When you have a database
q=12, a 3-node cluster, and each node has 4 core CPU.In this case, on each node, processing will start for 3,000,000 docs for 12 shards.
When the logic described in the initial description is applied, then each node will be taken for indexing 1,000,000 docs, and 4 shards.
This will reduce the time to get a valid response from 24 hours to 8 hours.
Do you agree?
rnewson commentedon Dec 5, 2025
I see what you are getting at, you'd ideally like couchdb to build just one copy of each of the
qindex shards first (and evenly divided across the nodes), so that you can query it sooner, and the other 2 copies should wait until afterward to start.One suggestion for today is to configure the background indexing concurrency downward. See https://docs.couchdb.org/en/stable/config/indexbuilds.html for details. Ken unfortunately won't be aware of your preferred order.
sergey-safarov commentedon Dec 5, 2025
Yes, when the number of changes is larger than
max_incremental_updates, then it will be better to index the first shard copies and process the next copy when the previous is completed.This also decreases the requirements for memory. A smaller number of documents processed simultaneously requires less memory, and this reduces the probability of a message
The proposed mechanism looks simple to understand.
rnewson commentedon Dec 5, 2025
We'd accept a pull request that implements this behaviour (as long as it's optional, as this mode will not work for everybody). Thank you for clarifying your idea.