perf(core): skip unchanged buckets on O3 incremental refresh - #7112
Merged
Conversation
When a historical (O3) write into a mat view's base table lands far
behind the current commit position, the cached refresh intervals get
combined into a single [getQuick(0), getQuick(size-1)] envelope inside
findRefreshIntervals. The resulting SAMPLE BY cursor then scans every
non-empty bucket between the O3 timestamp and "now" -- including
previously-refreshed buckets that no WAL transaction has touched.
This change adds a cost-aware clustering pass between WAL interval
loading and iterator construction:
* MatViewState tracks two exponential moving averages: average commit
latency (avgCommitNanos) and average per-timestamp-unit scan rate
(avgScanNanosPerTsUnit). Samples come from the actual commitWithParams,
commitMatView, and factory.getCursor calls inside insertAsSelect.
Samples larger than 5x the current average are capped before folding
(single-GC-pause protection); EMA smoothing factor is alpha = 1/8.
* MatViewRefreshJob.clusterIntervals merges two adjacent cached
intervals when the gap between them is smaller than the EMA-derived
threshold (commitNanos / scanNanosPerTsUnit). A safety cap on the
number of clusters per refresh prevents pathological scattered-O3
workloads from emitting hundreds of tiny commits; configurable via
cairo.mat.view.refresh.max.clusters (default 32).
* MatViewRefreshJob.capStepByNarrowestInterval reduces the iterator
step to the width of the narrowest cluster so that one step-group
can never straddle two clusters. Without this cap, clustering alone
is ineffective: even with disjoint intervals, a huge step-group
intersects both and the cursor's range filter widens to the full
span.
Three new columns on materialized_views() expose the live cost-model
state for debugging: refresh_avg_commit_nanos,
refresh_avg_scan_nanos_per_ts_unit, refresh_gap_threshold_ts_units.
REFRESH MATERIALIZED VIEW <name> STATS resets the EMA values to cold-
start, useful after a workload-shape change makes the historical
averages unrepresentative.
Benchmark (MatViewO3RefreshBenchmark, 512 symbols, 1440-min seed):
o3LagMin=720, rowsPerQuery=1M: 159.6 ms -> 1.97 ms (81x)
o3LagMin=120, rowsPerQuery=1M: 22.1 ms -> 1.87 ms (12x)
o3LagMin=30, rowsPerQuery=1M: 6.4 ms -> 2.10 ms (3.0x)
o3LagMin=0, rowsPerQuery=1M: 0.45 ms -> 0.40 ms (baseline)
The 1M curve is flat across O3 lags after the change -- refresh cost
scales with dirty work, not envelope width.
Tests: 390 existing MatView* tests pass unchanged. New tests:
* MatViewRefreshJobClusterTest: 13 direct unit tests for the cost
helpers (empty, single, adjacent merge, distant split, maxClusters
cap, overflow guard, partitionCount edge cases).
* MatViewTest.testRefreshIntervalsO3SplitsWideEnvelope: regression
test for the original O3-envelope-widening scenario.
* MatViewTest.testRefreshIntervalsO3MergesNarrowGap: companion test
that exercises the auto-tune's merge branch.
* MatViewTest.testRefreshIntervalsO3SingleIntervalUnaffected: guard
against the step-cap over-shrinking with a lone cached interval.
* MatViewTest.testRefreshIntervalsO3PeriodMatView: period mat view
path with clustering active.
* MatViewTest.testRefreshMaterializedViewStatsResetsEma: SQL-level
smoke test for the new STATS form.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Edge-case review surfaced three places where the cost-model arithmetic
could misbehave on pathological inputs:
* MatViewState.recordCommitNanos / recordScanMetrics fold samples into
an EMA via prev * (EMA_ALPHA_INV - 1). For prev values near
Long.MAX_VALUE / 7 this multiplication overflows to a negative
value, contaminating the EMA. Both fold paths now route through a
shared foldEma helper that uses Math.multiplyExact / Math.addExact
and falls back to a 50/50 blend on overflow -- keeps the average
monotone in sign and lets subsequent samples pull it back to a
realistic regime.
* MatViewRefreshJob.clusterIntervals computed gap as (lo - prevHi).
For intervals at the extreme ends of the long range the subtraction
wraps and the gap appears negative, which would have triggered an
incorrect merge. Now uses Math.subtractExact and treats overflow as
"definitely large, do not merge".
* MatViewRefreshJob.capStepByNarrowestInterval similarly computed
width as (hi - lo) without overflow handling, and would have
overflowed to negative for malformed (hi < lo) entries. Now uses
subtractExact and skips malformed entries.
Adds two focused test classes:
* MatViewRefreshJobClusterTest gains overflow + maxClusters + many-
point-intervals scenarios.
* MatViewStateEmaTest (new) covers EMA arithmetic at Long.MAX_VALUE,
cold-start lock-in protection, outlier capping, zero-range scan
samples, and refreshStats() reset behaviour.
407 mat view tests pass (was 390). Benchmark unchanged at o3LagMin=720
rowsPerQuery=1M: 2.4 ms median (vs 2.0 ms before -- within JIT noise).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
nwoolmer
added
Core
nwoolmer
changed the title
Review fixes:
* Move the gap-threshold javadoc to the correct method
(getCommitGapThresholdTsUnits), fix the "microseconds of timestamp
range" wording to "ts-units (us or ns matching the base table)".
* Fix stale {@link MatViewState#getCommitGapThresholdMicros} reference
in MatViewRefreshJob.findRefreshIntervals to the renamed method.
* Rename setRefreshMetricsForTesting parameter scanNanosPerMicro to
scanNanosPerTsUnit, recordScanMetrics parameter rangeMicros to
rangeTsUnits; update both javadocs to match the unit-agnostic API.
* Add @testonly annotation to setRefreshMetricsForTesting.
* Reorder MatViewState members: COLD_START_GAP_THRESHOLD_TS_UNITS goes
first among public static finals (C < M); setRefreshMetricsForTesting
moves into the set* block; foldEma moves up next to the other static
helpers; refreshFail comes before refreshStats.
* Reorder MatViewRefreshJob.capStepByNarrowestInterval before
clusterIntervals (Cap < Clu) to match the alphabetical convention.
* Reorder PropServerConfiguration.matViewRefreshIntervalsUpdatePeriod
before matViewRefreshMaxClusters (Intervals < Max).
* Rename local booleans mergeable -> isMergeable in MatViewRefreshJob,
statsReset -> isStatsReset in SqlCompilerImpl.
CI fix:
* ViewsFunctionTest.testViewsConsistentWithMatViewsAndTablesCommands
snapshotted the full materialized_views() schema. Adding three new
EMA columns broke the assertion; the new columns also carry
timing-dependent values that can't be hardcoded. Project only the
deterministic columns in the query.
New tests:
* MatViewTest.testRefreshIntervalsMaxClustersConfigCapsClusterCount
proves the cairo.mat.view.refresh.max.clusters property flows from
PropertyKey through to clusterIntervals.
* MatViewTest.testRefreshMaterializedViewStatsRejectsGarbageTail
covers `REFRESH ... STATS FROM ...` and `STATS INCREMENTAL` being
rejected at parse time.
* MatViewTest.testRefreshMaterializedViewUnknownActionMentionsStats
asserts the unknown-action error mentions 'stats' so operators can
discover the new form.
* MatViewTest.testRefreshMaterializedViewStatsBusyView covers the
tryLock-fails path returning a retryable SqlException.
* MatViewTest.testRefreshMaterializedViewStatsResetsEma now also
asserts the catalogue function surfaces the EMA values via SQL
after a real refresh, catching column-order regressions in
MatViewsFunctionFactory.
* MatViewStateEmaTest.testEmaOuterMultiplierOverflowFallback exercises
the multiplyExact catch branch in the outlier cap; the existing
testEmaArithmeticDoesNotOverflowOnMaxValues now asserts the exact
50/50 blend value rather than just non-zero.
* MatViewRefreshJobClusterTest.testCapStepSkipsMalformedHiLessThanLo
covers the widthTsUnits < 0 defensive continue.
Hardening:
* MatViewStateEmaTest now constructs a real (empty) MatViewDefinition
instead of passing null. Future maintainers adding a
viewDefinition.getMatViewToken() call to any EMA method will fail
loudly with a definition rather than NPE silently.
* MatViewO3RefreshBenchmark throws if rowsDelta <= 0 so a future
refresh-path regression that silently emits no rows reports as a
failure rather than a phantom speedup.
423 mat view tests pass (was 407 before this commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two CI snapshot tests lagged behind the mat view refresh changes on this branch and failed across all griffin/other platforms: - SqlParserTest#testRefreshMatView5 asserted the old parser error message. The STATS sentinel added in 63125db widened the message to mention 'stats' as a valid refresh keyword. - ServerMainTest#testShowParameters compares the full set of configured properties. The new cairo.mat.view.refresh.max.clusters property (default 32, non-reloadable) was missing from the expected set. Both tests now pass locally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
clusterIntervals used to mutate the LongList it received in place, and
the caller passed in viewState.refreshIntervals -- the live reference
to the cached union of unprocessed WAL ranges. On a refresh failure
(or the TableReferenceOutOfDateException re-enqueue path that does
not invoke refreshFailState), the clustered list survived in memory
and was persisted on the next cache write or restart. Once the gap
information between merged intervals was lost it never came back: the
next refresh's union started from the already-clustered cache, and
subsequent retries kept widening the envelope.
Make the cache stay loss-free:
* clusterIntervals takes an explicit destination LongList; it clears
the destination, copies the source into it, then runs the existing
merge algorithm on the destination. The source is never written.
* MatViewRefreshJob holds a clusteredIntervals scratch field. The
refresh callsite now rebinds the local refreshIntervals variable
to that scratch after clustering, so the period and refresh-limit
intersect/union helpers downstream also operate on the working
copy. The viewState reference is observable but read-only from the
clustering callsite onward.
Tests:
* Update every MatViewRefreshJobClusterTest call to the new
signature; assertions now check the destination list.
* Add testClusterSourceIsNeverMutated to lock the source-read-only
contract.
* Add testClusterReusesDestinationListAcrossCalls to lock the
clear-on-entry contract -- the production scratch is reused across
refreshes and must not leak stale entries.
Verification: 22/22 MatViewRefreshJobClusterTest, 184/184 MatViewTest,
89/89 across the other mat-view test classes, 22/22 MatViewFuzzTest.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The cost-model accessors hid two regimes behind a Math.max(1, ...) clamp that made dormant clustering look active in materialized_views(): recordScanMetrics: A sub-resolution sample (scan finishes faster than one ts-unit of range) was floored to 0 by integer division, then pinned to 1 by the clamp. Subsequent EMA folding locked the rate orders of magnitude above the true scan cost, which in turn collapsed getCommitGapThresholdTsUnits below the cold-start default. Reject these samples up front so the cold-start guard keeps firing instead. getCommitGapThresholdTsUnits: When commit < scanPerTsUnit, the cost model says a fresh REPLACE_RANGE commit is cheaper than scanning a single ts-unit of gap, so gap-based merging should be off. The previous Math.max(1, 0) = 1 looked like "merge any gap below 1 ts unit" in the catalogue but was functionally indistinguishable from disabled, because IntervalUtils.unionInPlace guarantees adjacent intervals are at least one ts-unit apart and the condition gap < 1 is unreachable. Return the raw quotient (0 in that regime) so operators reading materialized_views() can tell clustering is dormant; clusterIntervals already treats a non-positive threshold as gap-merge disabled, only the maxClusters safety cap fires. testRefreshMaterializedViewStatsResetsEma seeded its EMA values via a single-row refresh, which is now correctly rejected as sub-resolution. Acquire the refresh latch and seed scan metrics explicitly so the STATS reset assertions still have something non-trivial to clear, and drop the post-reset scan-EMA recovery check that depended on the previous lenient recording path. EMA edge-case tests updated for the new contract: 0 is the gap-merge disabled sentinel, not 1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb into mv-o3-refresh-clustering
The per-sample-ratio storage (sampleNanos / rangeTsUnits) rejected every scan sample whose wall-clock duration was shorter than the timestamp range it covered. On TIMESTAMP_NS bases the two quantities share the same magnitude so the guard fired on every realistic refresh, leaving avgScanNanosPerTsUnit at 0 and pinning the gap-merge threshold to the cold-start sentinel for the lifetime of the view. The auto-tune the clustering work introduced was dormant on ns workloads. Switch to a two-EMA design: store wall-clock sample and ts-range width as separate rolling averages, derive the per-ts-unit rate at read time via commit * range / sample (saturating to MAX/2 on overflow). Every positive sample folds; the catalogue exposes two natural-unit columns (refresh_avg_scan_sample_nanos, refresh_avg_scan_range_ts_units) instead of one dimensionally-weird scaled column. Make the cold-start sentinel unit-aware. The previous constant was interpreted as 2 s on us bases but 2 ms on ns bases -- a 1000x asymmetry that created an abrupt threshold jump after the first refresh on ns views. COLD_START_GAP_THRESHOLD_MICROS is now converted via the base table's TimestampDriver, so both base types start with the same wall-clock-equivalent merge budget. Round half-up in foldEma. With prev = 1 the floor-divide form created a fixed point: cap = 5 made (7 + capped)/8 always 0 or 1 and the EMA could not grow regardless of sample size. The trap is narrow under the two-EMA design but the one-line fix is strictly better, with a sub-permille bias against typical EMA magnitudes. Tighten testRefreshIntervalsMaxClustersConfigCapsClusterCount. The existing assertions (intervals-count before / after) passed regardless of the cap value -- both raw and capped runs cleared the cache. The test now reads the mat-view writer txn before and after the refresh and asserts the delta is 2 (matches the cap of 2). Without the cap the same workload emits 5 commits, so a regression that ignores the cap would fail loudly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
recordScanMetrics ran at the top of each interval-iterator entry, before the rollback-and-retry check that fires when insertedRows > batchSize. The retry path rescans the same data with a smaller step, so the same ts-range gets sampled twice -- once on the wasted big-step pass and again on the finer-step retry. The OOM and commit-failure paths in the same retry loop have the same shape: iterations completed before the throw had their samples folded into the EMA, then walWriter.rollback() discarded the inserts. Accumulate scan metrics into a pending (sampleNanos, rangeTsUnits) pair across iterations of a single attempt. Flush into the EMA only after a REPLACE_RANGE commit succeeds. Rollback paths (retry, exception) restart the OUTER for-loop, which re-declares the locals at zero -- no explicit clear needed. The change makes the two scan EMAs dimensionally consistent with avgCommitNanos, which has always folded one sample per commit. The sample/range ratio is preserved between per-iteration and per-commit aggregation, so steady-state threshold magnitude is unchanged. Cold start lingers one fold longer (until the first successful commit), which is the safer direction -- the cost model only sees data that actually committed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kafka1991
reviewed
May 19, 2026
Address kafka1991 PR review: capStepByNarrowestInterval pinned the
iterator step globally to the narrowest cluster's width, so a wide
recent cluster paired with a far-back narrow O3 cluster paid one
cursor open per capped step-group across the entire wide cluster --
e.g. a 1000-bucket wide cluster + a point-interval far-back cluster
walked the wide cluster in 1000 step-groups.
Switch to per-cluster stepping:
* computePerClusterSteps(intervals, approxBucketSize, naturalStep,
out) replaces capStepByNarrowestInterval. It fills out with one
step per cluster: min(naturalStep, clusterWidthBuckets). A narrow
cluster no longer constrains the step on wider siblings, and vice
versa.
* SampleByIntervalIterator carries a parallel LongList stepPerInterval
on top of the existing intervals list. The new toTop(LongList)
arms the per-cluster path; toTop(long) keeps the legacy single-step
behaviour for unaffected tests. On cluster transition the inner
next() loop swaps `step` to the new cluster's step.
* snapToCluster(clusterLo) is a new abstract hook implemented by the
two iterator subclasses. When the current step-group is entirely
before the next cluster, the base class snaps the iterator's hi to
the cluster boundary before re-entering next0(), so a small
per-cluster step never walks the gap one bucket at a time. The
FixedOffsetIntervalIterator snap is a sampler.round() on the
cluster lo; the TimeZoneIntervalIterator snap converts to local
time, applies adjustLoBoundary, then re-runs the shift-tracking
state machine up to the snapped position.
* The retry path now scales naturalStep down and recomputes the
per-cluster step list rather than carrying a single intervalStep
variable. RefreshContext gains naturalStep / stepPerInterval /
refreshIntervals / approxBucketSize fields so the OOM and
too-many-rows retry branches can recompute without re-running
findRefreshIntervals.
Benchmark on (rowsPerQuery=1M, o3LagMin=720, NUM_SYMBOLS=512):
| wideBuckets | before | after | delta |
|-------------|--------|--------|-------|
| 1 | 1.79ms | 1.73ms | flat (no cap to remove) |
| 10 | 1.76ms | 1.58ms | -10% |
| 100 | 1.03ms | 0.36ms | -65% |
| 1000 | 5.33ms | 0.76ms | -86% |
Wide-only (o3LagMin=0) baselines are unchanged within noise across
all wideBuckets values, confirming the change is targeted at the
mixed-width path rather than affecting single-cluster refreshes.
Tests:
* MatViewRefreshJobClusterTest: 11 cap tests migrated to the new
computePerClusterSteps function. 10 clusterIntervals tests
unchanged.
* FixedOffsetIntervalIteratorTest: 2 new tests
(testPerClusterStepSnapsAcrossGap, testPerClusterStepThreeClusters)
pin down the cluster-transition + snap behaviour.
* 397-test mat-view suite (MatViewTest 184, CreateMatViewTest 100,
MatViewFuzzTest 22, etc.) passes unchanged.
MatViewO3RefreshBenchmark extended with a WIDE_RECENT_BUCKETS_SWEEP
dimension so future changes can be measured against the same
mixed-width scenario kafka1991 raised.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
testRefreshIntervalsO3RandomFuzz drives 8-30 random batches of inserts
into a 1-minute mat view's base table. Each batch picks:
* A signed lag in [-maxLagMinutes, +maxLagMinutes] from the current
watermark; the sign is biased 70:30 toward negative so most batches
land out-of-order. maxLagMinutes itself is random in [60, 2060]
so individual runs cover both short and long-haul O3.
* A width in [1, maxBatchWidth] consecutive minute rows in a single
INSERT (one WAL txn = one cached refresh interval, so the width
directly drives cluster widths -- a 1-row batch is a point cluster,
a 30-row batch is a wide cluster).
Between batches it randomly issues an incremental refresh with
probability midDrainProb (also seeded), so the cached refresh interval
list accumulates different shapes across runs. A final incremental
refresh flushes any remaining intervals.
The test then asserts two invariants:
* The view is not invalidated by the O3 traffic.
* The view content matches a fresh "select ts, last(price), min(price),
max(price) from base_price sample by 1m" of the base table. This
is the strongest invariant on the per-cluster step path: if the
cluster transition swap or the snapToCluster hook has any
off-by-one or boundary bug, the mat view drifts from the ground
truth and assertSqlCursors reports the bucket-level diff.
10 runs at varying seeds covered batches=8..28, maxBatchWidth=2..23,
maxLagMinutes=232..1700, midDrainProb=0.22..0.66 -- the failing seed
is logged by generateRandom so any future regression is reproducible.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add two TimeZoneIntervalIteratorTest cases that exercise the post-DST cluster boundary on the Europe/Berlin fall-back and spring-forward transitions. Each test sets up two single-minute intervals straddling the transition with a 1h sampler and per-cluster step of 1. The post-transition cluster forces snapToCluster() to walk shiftLoIndex / shiftOffset past the fold or gap and re-derive the UTC bucket boundaries from the new offset. A regression where the shift state is not advanced would emit buckets one hour off from the expected [03:00, 04:00) UTC. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pull in the latest client commits on the mv-o3-refresh-clustering branch, including the QWP sender batch-cap clamp, IPv4 / GEOHASH column support, binary payloads, zstd defaults, and a few config fixes that landed in the client repo. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bluestreak01
previously approved these changes
May 20, 2026
bluestreak01
approved these changes
May 23, 2026
Contributor
[PR Coverage check]😍 pass : 268 / 275 (97.45%) file detail
|
RaphDal
pushed a commit
that referenced
this pull request
May 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When an out-of-order historical write into a mat view's base table lands
far behind the current commit position, the incremental refresh on
master scans every non-empty bucket between the O3 timestamp and "now",
including pre-existing ones that no WAL transaction touched. The cursor
filter is set to the full
[minTs, maxTs]envelope and the iterator'sstep-group spans the gap. For a 512-symbol mat view with a 12-hour O3
lag the refresh takes 160 ms instead of the ~2 ms the actual work
requires.
This PR adds a cost-aware clustering pass in
MatViewRefreshJob. Tworolling EMAs on
MatViewState(avg commit latency in ns, avg scanlatency per timestamp-unit) drive a gap-width threshold below which two
adjacent cached intervals get merged.
computePerClusterStepsthenproduces one iterator step per cluster (
min(naturalStep, clusterWidthBuckets)) so a step-group never straddles two clusters --the existing gap-skip excises the gap cheaply, and each cluster gets
the step that best fits its own width.
The iterator (
SampleByIntervalIterator+FixedOffsetIntervalIteratorTimeZoneIntervalIterator) carries a parallelLongList stepPerIntervaland swaps the active step on cluster transition. A newsnapToCluster()hook aligns the iterator's hi to the next cluster'sbucket-aligned lo, so a small per-cluster step doesn't walk the gap
one bucket at a time.
materialized_views()gains three columns (refresh_avg_commit_nanos,refresh_avg_scan_nanos_per_ts_unit,refresh_gap_threshold_ts_units)for operator visibility into the cost model.
REFRESH MATERIALIZED VIEW <name> STATSresets the EMAs to cold-start when the workload shapechanges.
One new configuration key:
cairo.mat.view.refresh.max.clusters(default 32) caps the number of clusters per refresh to prevent
pathological many-disjoint-intervals workloads from emitting hundreds
of tiny commits.
Benchmark numbers
MatViewO3RefreshBenchmark, 512 symbols, 1440-minute seed.Original sweep (single-tick recent writes,
wideBuckets=1):Mixed-width sweep (
rowsPerQuery=1M, o3LagMin=720, varying thenumber of consecutive recent ticks per iteration). This exposes the
scenario where a narrow far-back O3 cluster co-exists with a wide
recent cluster:
The global step-cap variant pins the iterator step to the narrowest
cluster's width, so a 1000-bucket wide cluster paired with a point
narrow cluster pays 1000
factory.getCursor()calls. The per-clusterstep recovers that gap at K=1000 (-86%) and K=100 (-65%) while leaving
the K=1 case unchanged.
Tradeoffs
0.45 ms to 0.40 ms; the difference is at the level of bench noise.
cluster); the K=1 mixed case is ~1.7 ms regardless of step strategy
because it is two point-interval commits.
RefreshContextgains four fields (naturalStep,stepPerInterval,refreshIntervals,approxBucketSize) so the OOM / too-many-rowsretry paths can recompute the per-cluster step list without re-running
findRefreshIntervals.Test plan
MatViewRefreshJobClusterTest-- 23 tests covering empty/singleintervals, adjacent merges, distant splits, maxClusters cap,
overflow handling at
Long.MAX_VALUE, 200-point-interval cappressure, point intervals, malformed inputs, and the new
computePerClusterStepsper-cluster cap semanticsMatViewStateEmaTest-- 16 tests covering cold-start, EMAconvergence, zero/negative samples, outlier capping, pathological
cold-start scan rates,
STATSreset, overflow atLong.MAX_VALUEFixedOffsetIntervalIteratorTest.testPerClusterStepSnapsAcrossGap--verifies the snap on cluster transition produces a fresh step-group
anchored at the cluster boundary
FixedOffsetIntervalIteratorTest.testPerClusterStepThreeClusters--multi-cluster transition with mixed step widths
MatViewTest:testRefreshIntervalsO3SplitsWideEnvelope-- regression test forthe original wasted-recompute scenario
testRefreshIntervalsO3MergesNarrowGap-- auto-tune merge branchtestRefreshIntervalsO3SingleIntervalUnaffected-- guards againststep-cap over-shrinking on a single dirty interval
testRefreshIntervalsO3PeriodMatView-- period mat view withclustering active
testRefreshMaterializedViewStatsResetsEma-- SQL-level smoke testfor the new
STATSformMatViewTest,MatViewStateTest,MatViewTelemetryTest,MatViewReloadOnRestartTest,CreateMatViewTest,MatViewFuzzTesttests pass unchanged.