Skip to content

Refactor LoadLayer - #99

Merged
DZakh merged 17 commits into
mainfrom
dz/refactor-load-layer
Aug 6, 2024
Merged

Refactor LoadLayer#99
DZakh merged 17 commits into
mainfrom
dz/refactor-load-layer

Conversation

@DZakh

@DZakh DZakh commented Jul 31, 2024

Copy link
Copy Markdown
Member

Refactoring the Loader Layer in the Indexer

Overview

  • Refactored the loader layer in the indexer to fix a bug related to asynchronous operations stacking infinitely.
  • Improved code cleanup and performance.
  • Added batching for getting entities in handlers

Link to Loom

PR presentation:
https://www.loom.com/share/b6e4e83cdecc4bc0a4f9b12b6d628037

Explaining the changes:
https://www.loom.com/share/a0cb74cc2c4d4e4ab92a76c990fa3384

@DZakh
DZakh marked this pull request as draft July 31, 2024 16:43
Base automatically changed from dz/perf-optimisations to main August 1, 2024 09:27
@DZakh
DZakh force-pushed the dz/refactor-load-layer branch from 7adfe21 to 73c3b0c Compare August 1, 2024 09:29
Comment thread codegenerator/cli/templates/static/codegen/src/globalState/GlobalState.res Outdated
@DZakh
DZakh requested a review from JonoPrest August 1, 2024 12:35
@DZakh
DZakh marked this pull request as ready for review August 1, 2024 12:35
singleEntities: dict<array<loadSingle<Entities.internalEntity>>>,
lookupByIndex: dict<loadIndex<Entities.internalEntity>>,
entityMod: module(Entities.InternalEntity),
logger: Pino.t,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with hoisting the logger up is that logger actually contains a context for the logging. So in other words if we hand down the logger from a given handler/loader it will contain info about which event was being handled on the actual log. So it's probably better to put it on the action

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I'll think about it

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I'll think about it

It's pretty important for debugging that when you get a log/error it prints out info of which event it came from on which chain

Comment thread codegenerator/cli/templates/static/codegen/src/LoadLayer.res Outdated
@DZakh

DZakh commented Aug 1, 2024

Copy link
Copy Markdown
Member Author

In the two PRs above:

  • Run load by index and load by entityId for the same entity in parallel
  • Reject pending load promise on execution error, so the error propagates to the GlobalState and terminates the indexer
  • Reuse existing promise instead of storing an array of resolve/reject functions for the same load request
  • Renamed LoadActionMap to BatchQueue, since it makes more sense to me
  • Started passing logger to the loadByEntites function

Incoming:

  • Pass inMemTable via args
  • Decide what to do with logger and do a fix
  • Unit tests

@DZakh
DZakh force-pushed the dz/refactor-load-layer branch from a886d0c to ce0b0b4 Compare August 1, 2024 18:15
Comment on lines +54 to +63
let promise = Promise.make((resolve, reject) => {
let loadRecord: loadSingle = {
resolve,
reject,
promise: %raw(`null`)
}
map.byId->Js.Dict.set(entityId, loadRecord)
})
// Don't use ref, since it'll allocate an object to store .contents
(map.byId->Js.Dict.unsafeGet(entityId)).promise = promise

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made something very similar in utils FYI. (Called promiseWithHandles I think)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll take a look

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've checked it out, but decided to remove promiseWithHandles instead. I found it less performant, and since it's not used anywhere, I thought that it's better not to have the abstraction.

Comment thread codegenerator/cli/templates/static/codegen/src/LoadLayer.res Outdated
Comment thread codegenerator/cli/templates/static/codegen/src/LoadLayer.res
Comment thread codegenerator/cli/templates/static/codegen/src/LoadLayer.res Outdated
@DZakh

DZakh commented Aug 2, 2024

Copy link
Copy Markdown
Member Author

I fixed all your comments besides logger. The problem I can see, that unless we pass the logger on every user's .get call, it won't contain relevant context, since we group actions by entities and ids/indexes, so they'll have mixed contexts.

Also, for the case when we load by ids, that's a single operation while every id can have their own logger context.

Hence, I think having a more general context which doesn't contain a specific event is fine for now.

) => promise<array<Entities.internalEntity>>,
makeLoadEntitiesByField: (
~entityMod: module(Entities.InternalEntity),
. ~entityMod: module(Entities.InternalEntity),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the . needed?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, but for some reason vscode extention started to formatting the file using the curried mode setting on. So to prevent breaking it, I've decided to add . here.

})

Async.it(
"Correctly gets entity from inMemoryStore by index if the entity set after the index creation",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#105 This should change the outcome of this test right?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

@JonoPrest JonoPrest left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome stuff Dmitry! The tests are really clear and nice. And I'm super stoked about the refactor.

Really amazing addition to the indexer 👏🏼 👏🏼

@DZakh
DZakh merged commit 912e4c1 into main Aug 6, 2024
@DZakh
DZakh deleted the dz/refactor-load-layer branch August 6, 2024 07:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants