Improve chain priority logic - #690

DZakh · 2025-08-18T13:00:05Z

Prioritize chains with a buffer containing a whole batch of events. This way, it'll hit the preload optimization more effectively.

Summary by CodeRabbit

New Features
- Unordered batch processing now prioritizes full batches based on the configured max batch size.
- New public helper to construct/modify fetch-state objects for testing and setup.
Documentation
- Added a Testing section with guidance on using public APIs and running/filtering the test suite.
- Updated Key Features throughput claim to "over 10,000+ events per second."
Tests
- Refactored tests to use public APIs and added cases validating full-batch prioritization and unordered-batch selection.
Chores
- Updated network identifier mapping for Hyperliquid.

coderabbitai · 2025-08-18T13:02:11Z

Walkthrough

Adds batch-fullness prioritization to unordered-batch sorting (new hasFullBatch + ~maxBatchSize), updates ChainManager call site, refactors and extends FetchState tests to use public APIs, changes Hyperliquid network discriminant, and appends a Testing note to navigation docs.

Changes

Cohort / File(s)	Summary
Docs: Testing note `.cursor/rules/navigation.mdc`	Appends a "## Testing" section recommending use of public module API for tests and commands (`pnpm rescript`, `pnpm mocha`) with `_only` to filter.
Fetch state: batch selection logic `codegenerator/cli/npm/envio/src/FetchState.res`	Adds internal `hasFullBatch` that checks whether a queue contains a full batch given `maxBatchSize`; updates `filterAndSortForUnorderedBatch` to accept `~maxBatchSize: int` and to prefer full batches over partial ones, falling back to timestamp tie-breaker.
ChainManager invocation `codegenerator/cli/templates/static/codegen/src/eventFetching/ChainManager.res`	Updates call site to pass `~maxBatchSize` into `FetchState.filterAndSortForUnorderedBatch`.
Tests: refactor and new scenarios `scenarios/test_codegen/test/lib_tests/FetchState_test.res`	Reworks tests to construct states via public APIs (e.g., `handleQueryResult`, new `FetchState.updateInternal`), adds cases asserting inclusion/exclusion and prioritization of full batches, and reduces duplicated scaffolding.
Docs: README feature blurb `codegenerator/cli/README.md`	Updates "High performance" metric from “over 5,000+ events per second” to “over 10,000+ events per second.”
Config: network discriminant `codegenerator/cli/src/config_parsing/chain_helpers.rs`	Changes `Network::Hyperliquid` discriminant from `645749` to `999`, updating the numeric ID round-trip for that variant.

Sequence Diagram(s)

sequenceDiagram
  participant CM as ChainManager
  participant FS as FetchState
  participant Q as Queues

  CM->>Q: collect unordered fetch queues
  CM->>FS: filterAndSortForUnorderedBatch(~maxBatchSize, queues)
  FS->>FS: hasFullBatch(queue, latestFullyFetchedBlock, maxBatchSize)
  FS-->>CM: sorted queues (full batches first, then timestamps)
  CM->>Q: select queues for unordered batch

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

Improve chain selection for unordered batch creation #683 — Alters FetchState.filterAndSortForUnorderedBatch and unordered-batch selection logic; strongly related to the hasFullBatch / comparator changes.
Optimise batch creation to include items from the same chain as much as possible #547 — Related changes to unordered-batch creation and maxBatchSize-aware ordering affecting ChainManager and FetchState.
Indexing performance metrics #545 — Modifies network enum entries; related to the Hyperliquid discriminant update.

Suggested reviewers

JonoPrest
JasoonS

Poem

I hop through queues with whiskers twitching bright,
Full baskets first in the soft morning light.
Timestamps trail like breadcrumbs in snow,
Max size guides where the carrots should go.
I celebrate tests passing — a rabbit’s delight! 🥕🐇

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

✨ Finishing Touches

📝 Generate Docstrings

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch dz/improve-chain-priority

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

Visit our Status Page to check the current availability of CodeRabbit.
Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

coderabbitai

Actionable comments posted: 2

🧹 Nitpick comments (2)

.cursor/rules/navigation.mdc (1)

35-39: Good addition: lightweight testing guidance.

Clear and useful. One nit: consider warning contributors not to leave focused tests committed.

You could append a short note:
 ## Testing

 Prefer Public module API for testing.

-Verify that tests pass by running a compiler `pnpm rescript` and tests `pnpm mocha`. Use `_only` to specify which tests to run.
+Verify that tests pass by running a compiler `pnpm rescript` and tests `pnpm mocha`. Use `_only` to specify which tests to run.
+Note: avoid committing focused tests (e.g. `describe_only` / `_only`) to main.

scenarios/test_codegen/test/lib_tests/FetchState_test.res (1)

2660-2675: Public-API-based construction looks solid; add an “exactly full” case.

Given the off-by-one risk, add a case where queue.length === maxBatchSize to pin expected behavior.

You can extend this block with:

       Assert.deepEqual(
         prepared->Array.map(fs => (fs.queue->Utils.Array.last->Option.getUnsafe).blockNumber),
         [1, 5],
       )
     },
   )
+
+  it("Treats exactly maxBatchSize items as a full batch (public API)", () => {
+    let mk = () => makeInitial()
+    let mkQuery = (fetchState: FetchState.t) => {
+      {
+        FetchState.partitionId: "0",
+        target: Head,
+        selection: fetchState.normalSelection,
+        addressesByContractName: Js.Dict.empty(),
+        fromBlock: 0,
+        indexingContracts: fetchState.indexingContracts,
+      }
+    }
+    let makeFsWith = (~latestBlock: int, ~queueBlocks: array<int>): FetchState.t => {
+      let fs0 = mk()
+      let query = mkQuery(fs0)
+      fs0
+      ->FetchState.handleQueryResult(
+        ~query,
+        ~latestFetchedBlock={blockNumber: latestBlock, blockTimestamp: latestBlock},
+        ~reversedNewItems=queueBlocks->Array.map(b => mockEvent(~blockNumber=b)),
+        ~currentBlockHeight=latestBlock,
+      )
+      ->Result.getExn
+    }
+    let fsExactlyFull = makeFsWith(~latestBlock=10, ~queueBlocks=[3, 2])
+    let fsHalf = makeFsWith(~latestBlock=10, ~queueBlocks=[1])
+    let prepared =
+      [fsHalf, fsExactlyFull]->FetchState.filterAndSortForUnorderedBatch(~maxBatchSize=2)
+    Assert.deepEqual(
+      prepared->Array.map(fs => (fs.queue->Utils.Array.last->Option.getUnsafe).blockNumber),
+      [2, 1],
+    )
+  })

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

MCP integration is disabled by default for public repositories
Jira integration is disabled by default for public repositories
Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between cef9a67 and 0544b2c.

📒 Files selected for processing (4)

.cursor/rules/navigation.mdc (1 hunks)
codegenerator/cli/npm/envio/src/FetchState.res (1 hunks)
codegenerator/cli/templates/static/codegen/src/eventFetching/ChainManager.res (1 hunks)
scenarios/test_codegen/test/lib_tests/FetchState_test.res (6 hunks)

🧰 Additional context used

📓 Path-based instructions (4)

**/*.{res,resi}

📄 CodeRabbit Inference Engine (.cursor/rules/rescript.mdc)

**/*.{res,resi}: Never use [| item |] to create an array. Use [ item ] instead.
Must always use = for setting value to a field. Use := only for ref values created using ref function.
ReScript has record types which require a type definition before hand. You can access record fields by dot like foo.myField.
It's also possible to define an inline object, it'll have quoted fields in this case.
Use records when working with structured data, and objects to conveniently pass payload data between functions.
Never use %raw to access object fields if you know the type.

Files:

codegenerator/cli/npm/envio/src/FetchState.res
codegenerator/cli/templates/static/codegen/src/eventFetching/ChainManager.res
scenarios/test_codegen/test/lib_tests/FetchState_test.res

codegenerator/cli/npm/envio/**

📄 CodeRabbit Inference Engine (.cursor/rules/navigation.mdc)

Library-fied runtime shared across indexers lives in codegenerator/cli/npm/envio.

Files:

codegenerator/cli/npm/envio/src/FetchState.res

codegenerator/cli/templates/{dynamic/**/*.hbs,static/**}

📄 CodeRabbit Inference Engine (.cursor/rules/navigation.mdc)

Templates live under codegenerator/cli/templates: dynamic/ for Handlebars (.hbs), static/ for raw Rescript files copied verbatim.

Files:

codegenerator/cli/templates/static/codegen/src/eventFetching/ChainManager.res

{**/generated/src/**/*.res,codegenerator/cli/templates/static/codegen/src/**/*.res,codegenerator/cli/templates/dynamic/codegen/src/**/*.res}

📄 CodeRabbit Inference Engine (.cursor/rules/navigation.mdc)

Runtime code lives in each project’s generated/src, but template versions (good for editing) are under codegenerator/cli/templates/static/codegen/src or codegenerator/cli/templates/dynamic/codegen/src.

Files:

codegenerator/cli/templates/static/codegen/src/eventFetching/ChainManager.res

🧠 Learnings (5)