Implement start saving history the near the head - #235
Conversation
2033d7d to
7b47a75
Compare
| Set(entity)->Types.mkEntityUpdate(~eventIdentifier, ~entityId, ~shouldSaveHistory), | ||
| ~shouldSaveHistory, |
There was a problem hiding this comment.
I realise that there is logic duplication here for "shouldSaveHistory" and I want to refactor that code.
| let executeBatch = async (sql, ~inMemoryStore: InMemoryStore.t, ~isInReorgThreshold) => { | ||
| let entityDbExecutionComposer = | ||
| RegisterHandlers.getConfig()->Config.shouldRollbackOnReorg | ||
| RegisterHandlers.getConfig()->Config.shouldSaveHistory(~isInReorgThreshold) |
There was a problem hiding this comment.
We now only start saving history when we are in a threshold for reorgs
| let shouldSaveHistory = (config, ~isInReorgThreshold) => | ||
| switch config.historyConfig { | ||
| | {rollbackFlag: RollbackOnReorg} if isInReorgThreshold => true | ||
| | {historyFlag: FullHistory} => true | ||
| | _ => false | ||
| } | ||
|
|
There was a problem hiding this comment.
Previously was using the shouldRollbackOnReorg function. But this is more specific. If we are saving full history it should always be true, if it's just standard rollback on reorg config it should only save if the indexer is in the reorg threshold
| inMemTable: t<'entity>, | ||
| entityUpdate: Types.entityUpdate<'entity>, | ||
| ~shouldRollbackOnReorg, | ||
| ~shouldSaveHistory, |
There was a problem hiding this comment.
Renamed this, since it's more appropriate for the usecase
| let mk = cmp => { | ||
| { | ||
| val: Some(cmp), | ||
| isInReorgThreshold: isInReorgThreshold || | ||
| cmp.earliestEvent->FetchState.queueItemIsInReorgThreshold(~heighestBlockBelowThreshold), | ||
| } | ||
| } |
There was a problem hiding this comment.
We now calculate if an event is in reorg threshold as we create a batch
| //If there is item on the arbitray events queue, and it is earlier than | ||
| //than the earlist event, take the item off from there | ||
| | Some(itemWithPopFn) | ||
| | Some({val: itemWithPopFn, isInReorgThreshold}) |
There was a problem hiding this comment.
Can we have the isInReorgThreshold as part of the eventBatchQueueItem instead of calculating it in the chainManager? I think it'd simplify quite a lot of code. Honestly, I don't like the increased complexity, and I hope it can be simplified to a single bool field with primitive logic.
There was a problem hiding this comment.
It can't just be on the eventBatchQueue item because it's not a static value, it changes as the head moves on and it's only relevant at the point in time when we create a batch.
- We would still have to reduce through the created batch to see whether any item was within the threshold
- A queue with no item that was fetching in the threshold would also be a valid point to start saving items.
| fetchStatesMap: ChainMap.t<PartitionedFetchState.t>, | ||
| arbitraryEventQueue: array<Types.eventBatchQueueItem>, | ||
| let hasChainItemsOnArbQueue = (self: t, ~chain): bool => { | ||
| self.arbitraryEventQueue->Js.Array2.find(item => item.chain == chain)->Option.isSome |
There was a problem hiding this comment.
Just for you to know, there's https://rescript-lang.org/docs/manual/latest/api/js/array2#value-some
JonoPrest
changed the title
2 participants
I have just run a number of manual tests so far to see that it's working as expected.
I still need to add some tests and thinking about the best way for this.