[5] Remove old entity_history code and add history pruning - #338
Conversation
| let head = array->Js.Array2.slice(~start=0, ~end_=index) | ||
| let tail = array->Belt.Array.sliceToEnd(index + 1) | ||
| [...head, ...tail] | ||
| Belt.Array.concat(head, tail) |
There was a problem hiding this comment.
No reason other than my tree sitter plugin wasn't parsing spreads and annoying me 😅
| let shouldSaveHistory = | ||
| RegisterHandlers.getConfig()->Config.shouldSaveHistory(~isInReorgThreshold) |
There was a problem hiding this comment.
Another place to remove the config lookup at the file level to ensure tests behave accordingly. "shouldSaveHistory" is now calculated and handed down.
| const tableName = makeHistoryTableName(entityName); | ||
| return sql` | ||
| WITH first_change AS ( | ||
| SELECT | ||
| MIN(serial) AS first_change_serial | ||
| FROM | ||
| public.${sql(tableName)} | ||
| WHERE | ||
| ${Utils.$$Array.interleave( | ||
| safeChainIdAndBlockNumberArray.map( | ||
| ({ chainId, blockNumber }) => | ||
| sql`(entity_history_chain_id = ${chainId} AND entity_history_block_number > ${blockNumber})`, | ||
| ), | ||
| sql` OR `, | ||
| )} | ||
| ), |
There was a problem hiding this comment.
Since this doesn't relate to a given reorged chain, we need to check each chain for changes in their respective reorg thresholds and selected the earliest change out of those.
There was a problem hiding this comment.
Now safeChainIdAndBlockNumberArray started make sense :)
| WHERE | ||
| -- Delete all entity history of entities that are not in the reorg threshold | ||
| eh.id NOT IN (SELECT id FROM items_in_reorg_threshold) | ||
| -- Delete all rows where id matches a row in previous_items but has a lower serial | ||
| OR | ||
| eh.serial < (SELECT serial FROM previous_items WHERE previous_items.id = eh.id); |
There was a problem hiding this comment.
The amazing advantage with this, is we only need to store history items for changes in the current reorg threshold + 1 historical change for each of those entities. Everything else can be cleared.
On new inserts, a defaulted zero value historical item is copied in from the current entity table so we can always keep the history table quite slim.
There was a problem hiding this comment.
The only disadvantage is that this now has to be done as part of the batch transaction so we don't accidentally remove needed items due to async conditions. The main reason this query was running heavy before though is that there had to be at least one copy of every entity in the history table so it was quite large to do these expensive queries. Now it should be small based on changes within the reorg threshold of a chain.
| //History pruning needs to happen last in the transaction | ||
| //It deletes all unneeded history rows outside of the reorg threshold | ||
| pruneEntityHistory, |
There was a problem hiding this comment.
History now gets pruned in the transaction after new history has been added and entities have been updated.
| ~logger, | ||
| ~loadLayer, | ||
| ~isInReorgThreshold, | ||
| ~shouldSaveHistory=config->Config.shouldSaveHistory(~isInReorgThreshold), |
There was a problem hiding this comment.
Like that you started using `shouldSaveHistory
| ~inMemoryStore, | ||
| ~isInReorgThreshold=false, | ||
| ~config, | ||
| ~safeChainIdAndBlockNumberArray=[], //No need to prune history for dynamic contract pre registration |
There was a problem hiding this comment.
The naming is confusing. Maybe I'll understand what it means after reviewing further, but for now, I can only guess.
| serializer: Utils.magic, | ||
| }), | ||
| ])->S.setName("GqlDbCustomTypes.Float") | ||
| let schema = S.float->S.setName("GqlDbCustomTypes.Float") |
| const tableName = makeHistoryTableName(entityName); | ||
| return sql` | ||
| WITH first_change AS ( | ||
| SELECT | ||
| MIN(serial) AS first_change_serial | ||
| FROM | ||
| public.${sql(tableName)} | ||
| WHERE | ||
| ${Utils.$$Array.interleave( | ||
| safeChainIdAndBlockNumberArray.map( | ||
| ({ chainId, blockNumber }) => | ||
| sql`(entity_history_chain_id = ${chainId} AND entity_history_block_number > ${blockNumber})`, | ||
| ), | ||
| sql` OR `, | ||
| )} | ||
| ), |
There was a problem hiding this comment.
Now safeChainIdAndBlockNumberArray started make sense :)
| //On the first time we enter the reorg threshold, copy all entities to entity history | ||
| //And set the isInReorgThreshold isInReorgThreshold state to true | ||
| dispatchAction(SetIsInReorgThreshold(true)) | ||
| await DbFunctions.sql->DbFunctions.EntityHistory.copyAllEntitiesToEntityHistory |
Co-authored-by: Dmitry Zakharov <dzakh.dev@gmail.com>
64e94b6 to
fe8a277
Compare
2 participants
Uh oh!
There was an error while loading. Please reload this page.