Skip to content

Initial implementation of whereEq queries#79

Merged
JonoPrest merged 6 commits into
mainfrom
jp/where-queries
Jul 30, 2024
Merged

Initial implementation of whereEq queries#79
JonoPrest merged 6 commits into
mainfrom
jp/where-queries

Conversation

@JonoPrest

@JonoPrest JonoPrest commented Jul 17, 2024

Copy link
Copy Markdown
Collaborator

Implements "where equals" query for loaders on all indexed fields.

Flow:

  • loader calls the query
  • during loader execution query creates an "index" in the entity tables (if it doesn't already exist)
  • if query is unique it gets executed and values added to the in memory table.
  • The index set on the in memory table means that every add will evaluate and index its ID if it matches
  • At the end the loader will look up the index via in memory store to get matching ids
  • set and delete functions on an entity will manage current indices on the table

Note this can only be done in a loader because... In the scenario where you add a new entity that exists only in memory. The index needs to exist on the in memory table before the add in order for it to appear on lookup in the index.

Deleting an entity will search through all indices on that entity and remove its id from any index it exists in.

The mock db in test helpers will currently prune all indexes of an entity and then add indexes for all fields of that entity every time an operation is called to prevent stale values existing in indices.

@JonoPrest JonoPrest force-pushed the jp/where-queries branch 2 times, most recently from b3593b9 to cde08c7 Compare July 19, 2024 12:39
@JonoPrest JonoPrest marked this pull request as ready for review July 19, 2024 16:04
@JonoPrest JonoPrest requested review from DZakh and JasoonS July 19, 2024 16:04
Comment on lines +4 to +7
dict: dict<'val>,
dict: Js.Dict.t<'val>,

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to revert back to dict

@DZakh DZakh left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed everything besides codegenerator/cli/templates/static/codegen/src/InMemoryTable.res
I'll continue tomorrow morning. Looks good so far

switch (a, b) {
//For big decimal use custom equals operator otherwise let Caml_obj.equal do its magic
| (Some(BigDecimal(bdA)), Some(BigDecimal(bdB))) => BigDecimal.equals(bdA, bdB)
| (a, b) => a == b

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might not work for array of BigDecimal

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we handle them in a custom way, then we can use === instead of Caml_obj.equal, which should be a little bit faster and clener output

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we handle them in a custom way, then we can use === instead of Caml_obj.equal, which should be a little bit faster and clener output

This comment is optional

{{#each entities as | entity |}}
{{entity.name.uncapitalized}}: {
get: makeLoader(loadLayer.{{entity.name.uncapitalized}}, ~entityName={{entity.name.capitalized}}, ~entityIdsMap),
get: makeLoader(loadLayer.{{entity.name.uncapitalized}}, ~entityName={{entity.name.capitalized}}, ~entityIdsMap, ...),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove curring from here

Comment on lines 196 to 206
{{#each entities as | entity |}}
loadLayer.{{entity.name.uncapitalized}}->executeLoadActionMap(
~inMemTable=inMemoryStore.{{entity.name.uncapitalized}},
~batchLoadIds=Entities.batchRead(~entityMod=module(Entities.{{entity.name.capitalized}}))(DbFunctions.sql, _),
~batchLoadIds=DbFunctionsEntities.batchRead(~entityMod=module(Entities.{{entity.name.capitalized}}))(DbFunctions.sql, _),
~whereFnComposer=DbFunctionsEntities.makeWhereEq(
DbFunctions.sql,
~entityMod=module(Entities.{{entity.name.capitalized}}),
),
),
{{/each}}
]->Promise.all

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A comment for future. If there's an external await in between of .load method calls, then hasLoadActions check won't be able to track it exit. It'll cause the .load method to stack as an unresolved pending promise.

loadCallbacks: [loadCallback],
},
)
| Some({loadCallbacks}) => loadCallbacks->Js.Array2.push(loadCallback)->ignore

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a random idea, but theoretically, it might be faster. But the number of allocated function contexts is a concern

Suggested change
| Some({loadCallbacks}) => loadCallbacks->Js.Array2.push(loadCallback)->ignore
| Some(lookup) => {
lookup.callback = (v) => {
lookup.callback(v)
loadCallback(v)
}
}

@DZakh DZakh Jul 22, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another fun idea, which will probably be even more performant is to create a pending promise and expose the resolve function:

let resolveRef = ref(%raw(`null`))
let resultPromise = Promise.make((resolve, _reject) => {
  resolveRef := resolve
})

// Store both resultPromise and resolve in the LoadLayer

// Then when we add a new item to the LoadLayer we can do:
let _ = resultPromise->Promise.thenResolve(callback)

// And to actually resolve everything simply run the resolve function
resolve(result)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: I don't have benchmarks and since it's not a bottleneck, you don't need to change anything

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's chat through some of these thoughts on a call! Definitely keen to squeeze more performance.

@JonoPrest JonoPrest force-pushed the jp/where-queries branch 2 times, most recently from 1917fca to 4e922e8 Compare July 29, 2024 15:31

@DZakh DZakh left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't read some places in much details, but in general I don't have any comments 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants