Add more benchmarks and improve framework#299
Conversation
JonoPrest
commented
Oct 25, 2024
JonoPrest
commented
- Adds a benchmark for every event handler to help identify handler hotspots
- Improves the framework where SummaryData is not an array of floats as a data point but a rolling accumulation. This helps with not having continuous memory increasing as pushing to these arrays. But it's not yet safe to leave running on an indexer since there's no safety added to the continuous float multiplication/summing. So at some point it could overflow the js number. We could look at converting to bigint but then we loose floating points.
- Improve the json file write scheduler to only write every 500ms rather than on every benchmark update. It's helpful so you can fire off many more smaller benchmarks to the data set like the per handler one I added without slowing things down too much.
| sum: self.sum +. val, | ||
| sumOfSquares: self.sumOfSquares +. val *. val, |
There was a problem hiding this comment.
These are the two places where a long running benchmark could eventually overflow.
There was a problem hiding this comment.
This is no longer relevant I just changed this to using BigDecimal type.
| @as("std-dev") stdDev: float, | ||
| min: float, | ||
| max: float, | ||
| last: float, |
There was a problem hiding this comment.
I actually never meant to keep last in before. I don't think it's very useful. And it won't work with this model of accumulated data rather than saving all data points in the data set.
| let _ = Js.Global.setTimeout(() => { | ||
| start()->ignore | ||
| }, intervalMillis - timeSinceLastRun->Belt.Float.toInt) |
There was a problem hiding this comment.
This can be scheduled multiple times. Nothing should break, just a risk of unexpected execution.
There was a problem hiding this comment.
It can but it overwrites the scheduled function on each schedule. So more like it just calls start multiple times.
Maybe it's best to just add the "isWriting" if check here or in the start function again.
There was a problem hiding this comment.
Added the check in "start" 👍🏼
| @@ -187,8 +261,7 @@ module Summary = { | |||
| @as("std-dev") stdDev: float, | |||
There was a problem hiding this comment.
Never worked with std-dev before. I usually look at mean and avg
There was a problem hiding this comment.
I use it quite a lot. It will essentially give you an idea of how much variance your data has.
For eg. the circle in green has low variance and the mean should be an accurate representation.
The red circle has high variance so something makes it inconsistent. Could speak to something that is cached vs not-cached on different runs etc.
| let eventsPerSecond = | ||
| batchSizesSum | ||
| ->BigDecimal.div(BigDecimal.fromFloat(totalRuntimeSeconds)) | ||
| ->BigDecimal.decimalPlaces(2) |
There was a problem hiding this comment.
It doesn't really matter I suppose. I just kept it at 2.
2 participants
