Skip to content

fix(sql): add cursor self-consistency checks and fault injection to query fuzzer and harden query engine - #7217

Merged
bluestreak01 merged 63 commits into
masterfrom
puzpuzpuz_query_fuzzer_mk3
Jun 20, 2026
Merged

fix(sql): add cursor self-consistency checks and fault injection to query fuzzer and harden query engine#7217
bluestreak01 merged 63 commits into
masterfrom
puzpuzpuz_query_fuzzer_mk3

Conversation

@puzpuzpuz

@puzpuzpuz puzpuzpuz commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

What this does

Three rounds of enhancements to the seeded query fuzzer (QueryFuzzTest).

Cursor self-consistency checks (on by default). After materializing each result, the runner verifies invariants that hold for any RecordCursor regardless of access path:

  • re-iterating after toTop() reproduces the result set;
  • toTop() leaves preComputedStateSize() unchanged;
  • size() and calculateSize() agree with the materialized row count.

A divergent row set, a changed preComputedStateSize(), or a count mismatch is always a real defect and fails the query. The one skip valve is narrow (see bug fix #17): when the clean first pass is followed by a re-pass that throws a per-row cast or numeric overflow the first pass did not, that is accepted parallel-filter non-determinism, not a cursor-reuse defect, and the query is skipped.

Fault injection (on by default, -Dquestdb.fuzz.faults). On a fraction of queries (-Dquestdb.fuzz.fault.pct, default 15) one fault is armed and the runner applies a crash-and-recover oracle instead of the differential one:

  • FILE — one filesystem op fails once (FailureFileFacade);
  • MALLOC — a native allocation trips the RSS memory limit (the real out-of-memory path);
  • FUNCTION — a dev-mode test_fault() woven into the query throws once mid-evaluation.

For each, it asserts the factory frees its resources on the error path (native memory and file descriptors return to baseline) and that the same query runs cleanly once the fault is removed. Fault decisions are drawn from the seeded rnd so failures replay deterministically. By default the fault query runs under parallel SQL execution (-Dquestdb.fuzz.fault.parallel, default true), so the parallel filter / GROUP BY / top-K reduce error paths are exercised; the writer pool is halted for the duration of the query loop so no background job competes with an armed FILE / MALLOC fault (FILE is scoped to the test thread plus the query-pool workers, MALLOC's process-global RSS ceiling is tripped only by the query's own allocations). Pass -Dquestdb.fuzz.fault.parallel=false to run faults serially.

Window-function shapes (on by default, -Dquestdb.fuzz.window). A new WindowClause generates fn(...) OVER (PARTITION BY ... ORDER BY ts [frame]) projections across the ranking family (row_number, rank, dense_rank, cume_dist, percent_rank, ntile), the windowed aggregates (avg, sum, min, max, count) and the navigation functions (first_value, last_value, lag, lead, nth_value), with ROWS and time-based RANGE frames. Every window orders by the strictly-increasing designated timestamp, so the window values are reproducible across runs and the differential oracle compares full result sets rather than just row counts. The WINDOW band is carved from the upper half of the SIMPLE range, so the other shapes' frequencies are unchanged; pass -Dquestdb.fuzz.window=false to drop window shapes.

Status

Work in progress. Fault injection and the window-function shapes are doing their job: they surface real defects, so the fuzzer goes red on the seeds that hit them. Fixes land on this branch, each in its own commit with a regression test.

Bug fixes

  1. SAMPLE BY FILL(LINEAR) cleanup on out-of-memory. SampleByInterpolateRecordCursorFactory allocates its record-key map, data map and group-by allocator in the cursor constructor. When one of those native allocations failed, the constructor's error path called close() before the remaining fields were assigned, and close() dereferenced them directly, throwing NullPointerException that masked the real out-of-memory error. close() now frees the fields through the null-safe Misc.free().

  2. Parquet page-frame buffer leak on out-of-memory (production fix landed upstream in feat(sql): add a memory-budgeted cache for Parquet random access #7230). PageFrameMemoryPool removed a per-frame buffer from its free list and then called reopen(), which allocates the buffer's page-address lists one by one. If that allocation tripped the RSS limit, the buffer was already off the free list but not yet recorded in the cache, so the pool could no longer reach it on close and the lists it had allocated leaked. The fuzzer's malloc fault injection surfaced this. The fix landed independently in master through feat(sql): add a memory-budgeted cache for Parquet random access #7230, which rewrote PageFrameMemoryPool around a memory-budgeted decode-buffer cache and guards a failed reopen() by closing the buffer on the error path; after merging master this branch carries no PageFrameMemoryPool change of its own. The regression test stays: ParquetScanOomTest sweeps the RSS ceiling across the buffer reopen and asserts the scan leaves no native memory behind, now exercising the upstream guard through the same fault-injection path that first surfaced the leak.

  3. Covering posting index crash on a sidecar I/O error. AbstractPostingIndexReader swallowed any error raised while opening a covering index's sidecar files, cleared the sidecar memories, and carried on. Because the covering plan is fixed at compile time and the covered read path has no base-table fallback, a covered read then walked an empty sidecar and threw index out of bounds, 0 >= 0 (an out-of-bounds read with assertions disabled), masking the real I/O error. The reader now propagates the failure so the query fails with the underlying error and the reader is closed.

  4. Posting index reader buffer leak on out-of-memory. PostingGenLookup allocated its two native cache buffers as inline field initializers. If the second allocation tripped the RSS limit, the first leaked, because the half-built lookup was never assigned to the reader that was constructing it, so the reader's close() could not reach it. The allocations now live in an exception-safe constructor that frees the partial result on failure.

  5. Wrong calculateSize() for a backward interval scan. Surfaced by the cursor self-consistency check rather than fault injection. For a descending scan over a half-open timestamp range (e.g. WHERE ts < X ORDER BY ts DESC), IntervalBwdPartitionFrameCursor.calculateSize() diverged from the cursor's own next(). With an open lower bound it shortcut the start index to zero instead of the -1 sentinel its whole-partition check relied on, which both undercounted the newest partition by one and stopped the walk after that partition, ignoring every earlier one in range; a two-partition table that iterates to 96 rows reported a size of 47. calculateSize() now mirrors next(), so the counted size matches the iterated row count. Closed and open-ended intervals, and the forward cursor (which never had the open-bound shortcut), were already correct. The existing interval frame-cursor tests only ran the forward direction over closed-lower-bound intervals, so they missed it; the battery now runs every shape in both directions and adds open-start and open-end cases. Upstream fix(core): fix Parquet crashes on malformed files and data corruption on partial reads #7223 independently fixed the same bug with a minimal one-line patch that keeps the old structure, changing the open-bound start index from 0 to the -1 sentinel in place. The two fixes produce identical row counts; merging master resolves the overlap by keeping this branch's mirror-next() rewrite, which subsumes the upstream patch by removing the structural divergence between calculateSize() and next() that caused the bug in the first place.

  6. DATE-argument window value functions threw UnsupportedOperationException on read. max / min / first_value / last_value / nth_value / lag / lead over a DATE argument were backed by the timestamp window function (WindowTimestampFunction), whose getType() returned the argument type (DATE) but which supplied no getDate() accessor, so reading the result hit the default-throwing WindowFunction.getDate; the value is stored in timestamp ticks while DATE is milliseconds, so a naive passthrough accessor would also have read ~1000x off. Each value function now splits into a type-agnostic base holding the framing logic plus thin DATE and TIMESTAMP subclasses: the DATE subclass stores and returns the value in milliseconds directly through getDate(), and the TIMESTAMP subclass caches the timestamp driver once and converts ticks to milliseconds (lag / lead already followed this shape). With every timestamp subclass now supplying its own getDate(), the type-dispatching getDate() default on WindowTimestampFunction is removed and the shared empty-frame null function returns its stored NULL directly. WindowDateFunctionTest covers all seven functions over DATE across running, partition, range, rows and whole-result-set frames, plus casting a TIMESTAMP window result to DATE. Repro (now passing): SELECT max(d) OVER (ORDER BY ts) FROM t with d DATE.

  7. rank() / dense_rank() crash at compile time when a pass-through column of an unserializable type is also projected. In the streaming WindowRecordCursorFactory path (the window order matches the designated timestamp), RankOverPartitionFunction remembered each partition's previous row by copying the whole projected row into a MapValue, building the copier with RecordValueSinkFactory over every projected column. Any pass-through column the MapValue cannot hold (UUID, STRING, VARCHAR, BINARY, LONG256, array, INTERVAL) threw UnsupportedOperationException at compile time, so a plain SELECT u, dense_rank() OVER (PARTITION BY g ORDER BY ts) FROM t failed even though the rank itself was fine; projecting only the window function took the cached path and worked, so the trigger was the pass-through column. Only the ORDER BY columns decide whether two consecutive rows are peers, so the MapValue now stores just those (compacted) plus the running rank and count, and the comparator and rank maps are rebuilt over a matching compacted metadata so they read back only those columns. computeNext caches the previous row's ORDER BY values, overwrites the slots with the current row, then compares, so the comparator never touches the live record. WindowFunctionTest.testRankWithUnserializablePassThroughColumn covers UUID / STRING / VARCHAR / LONG256 / DOUBLE[] pass-through columns ascending and descending, plus a BINARY drain, and pins the streaming plan. Repro (now passing): SELECT u, dense_rank() OVER (PARTITION BY g ORDER BY ts) FROM t with u UUID.

  8. lead() over a high-precision DECIMAL crashed with UnsupportedOperationException on read. lead over a Decimal128 or Decimal256 argument reported ZERO_PASS, unlike every other lead variant (Double, Long, and the narrower decimals all inherit ONE_PASS). When a query's only window functions were wide-decimal leads ordered by the designated timestamp, that hint routed it through the streaming WindowRecordCursorFactory, where the cursor reads each result back through the function's own accessor. But lead reads ahead, so it cannot run in the forward streaming pass (it is computed in the cached executor with a backward pass1 scan), and the wide-decimal lead exposed no getDecimal128 / getDecimal256 accessor there, so the read fell through to the default-throwing WindowFunction accessor. Earlier all-types tests masked this because a narrow-decimal column in the same projection forced the cached path. Removing the erroneous getPassCount() override lets both over-partition functions inherit ONE_PASS like the rest, keeping the planner on the cached path where pass1 already does the work and the record chain reads the value back. WindowDecimalFunctionTest.testLeadWideDecimalOverPartitionStreamingShape covers Decimal128 and Decimal256 lead over a partition, alone and together, and pins the cached plan; because this suite randomizes cairo.sql.window.cached.light.enabled per run, the pinned plan label is config-aware (CachedWindow or CachedWindowLight), which a hardcoded CachedWindow label had got wrong on the cache-light half of the seeds. Repro (now passing): SELECT lead(d, 5) OVER (PARTITION BY g ORDER BY ts) FROM t with d DECIMAL(76,3).

  9. sum / avg over a high-precision DECIMAL raising an overflow is a genuine arithmetic limit, not an engine defect (fuzzer over-reported it). A sum or avg over a Decimal256-backed DECIMAL (precision > 38) raises CairoException: ... aggregation failed: an overflow occurred once the running total of values near the DECIMAL(76,3) maximum exceeds Decimal256's 256-bit capacity. Triage: the sum is genuinely unrepresentable (Decimal256 is the widest decimal type, so there is no wider accumulator), the plain group-by sum / avg overflows identically on the same data, and WindowDecimalFunctionTest already pins this overflow as the expected behaviour. The fuzzer's crash-only oracle was flagging it because it treats any leaked CairoException as an internal error. QueryRunner.isAcceptedSkip now accepts a CairoException whose message contains both "aggregation failed" and the word "overflow" (case-insensitive), classifying it like a NumericException overflow. Requiring "overflow" -- not just "aggregation failed" -- keeps the skip narrow: the decimal sum / avg factories report the overflow in two message shapes, the explicit "an overflow occurred" flag message and a wrapped Decimal256 NumericException beginning "Overflow in ...", and both carry the word; a future non-overflow numeric error wrapped into the same "aggregation failed: ..." text is reported as a failure instead of being silently skipped. WindowDecimalFunctionTest.testAvgDecimal256OverPartitionGenuineOverflow / testSumDecimal256OverPartitionGenuineOverflow pin the window-over-partition and group-by overflow, and QueryFuzzTest.testDecimalAggregationOverflowToleratedByOracle drives the repros through the oracle and asserts the skip while a non-overflowing query still completes. Repro (now tolerated): SELECT avg(d) OVER (PARTITION BY g ORDER BY ts) FROM t with d DECIMAL(76,3).

  10. rank() / dense_rank() over a partition built its streaming map from the wrong key types. In the streaming WindowRecordCursorFactory path the code generator builds each window function's partition map lazily, in initRecordComparator(), after the whole projection has been walked. It reuses one key-types buffer that it clears and rebuilds for every window column's PARTITION BY, but RankOverPartitionFunction kept a live reference to that buffer instead of a copy, so with two or more streaming window functions partitioning by differently typed keys it built its map from the last window column's key types. That either crashed -- the partition sink wrote a type the wrongly picked map could not hold, e.g. a BYTE partition writing into the Unordered4Map chosen for a SYMBOL key threw UnsupportedOperationException -- or silently mis-keyed the partitions: a Decimal256 partition map built for a two-column key fed by a single-column sink reported rank 1 for every row and could diverge across a toTop() re-read (the previously open bug below). The over-partition functions now snapshot the key types in their constructor through a shared AbstractWindowFunctionFactory.copyKeyTypes() helper; cume_dist / percent_rank carry the same idiom and get the same snapshot, though being two-pass they always take the cached path and were not yet exploitable. WindowFunctionTest.testRankOverPartitionMixedPartitionKeyTypes covers rank/dense_rank with mixed BYTE/SYMBOL and wide-DECIMAL partition keys on the streaming path, and testCumeDistMixedPartitionKeyTypes covers cume_dist/percent_rank with mixed keys on the cached path. Repro (now passing): SELECT b, rank() OVER (PARTITION BY b ORDER BY ts), dense_rank() OVER (PARTITION BY g ORDER BY ts) FROM t with b BYTE.

  11. Window value / navigation / aggregate functions over an untyped null literal behaved non-deterministically across platforms -- a compile-time crash, an inconsistent error, or silent acceptance. A window call whose argument is the bare null literal (e.g. lead(null, 2), min(null)) first surfaced as an internal java.lang.IllegalArgumentException: Unexpected column type: NULL: the cached window path asked RecordSinkFactory to build a sink over the NULL output column and leaked it through the query API, while the streaming path tolerated it -- so the same function crashed on one frame and silently returned nulls on another. The deeper cause is in overload resolution: an untyped null ties against every typed variant of the function (NULL has zero overload distance to any type), so the five lead factories (Double, Long, Decimal, Date, Timestamp) all match with the same score and the winner is decided by Java's stable sort preserving the classpath scan order, which differs by filesystem and architecture. Each winner behaves differently -- the type-adaptive Timestamp / Date variants yield a NULL output type, the Decimal variant throws its own lead is not yet implemented for NULL, and the Double / Long variants yield a concrete type and silently accept the query. An initial guard in the code generator rejected only the NULL-output case, so it held on x86-64 (where lead resolved to the Timestamp factory) but testNullLiteralArgumentRejectedWithCleanError failed on linux-arm64, where the same query resolved to the Decimal factory and threw the other message. FunctionParser now rejects an untyped null value argument to a window function with more than one overload before any factory runs, so every platform and frame gives the same SqlException at the function position suggesting a concrete cast (null::double compiles and computes over NULL); the two code-generator output-type guards are removed. This makes the rejection uniform: lead, lag, min, max, sum, avg, first_value, last_value, nth_value and count over an untyped null are now all rejected the same way -- a behaviour change for the variants that previously resolved to a concrete type and computed over NULL (lag / max / first_value / last_value / avg / sum / count) and for nth_value, which previously reported its own message. Window functions with a single overload keep their own argument validation, so ntile(null) still reports bucket count cannot be NULL, and the no-argument ranking functions (row_number, rank, dense_rank) are unaffected. The clean SqlException is an accepted skip in QueryRunner.isAcceptedSkip, so the seeds that previously crashed on lead(null) / min(null) now run green. WindowFunctionTest.testNullLiteralArgumentRejectedWithCleanError pins the rejection across frames and a partition, the uniform nth_value message, and the null::double cast workaround. Repro (now rejected cleanly): SELECT lead(null, 2) OVER (ORDER BY ts) FROM t.

  12. Window RANGE-frame functions returned a wrong, run-varying value when the designated timestamp was not in the projection. A query that mixes window functions ordering differently (e.g. count(...) OVER (... ORDER BY ts DESC) plus min(...) OVER (... ORDER BY ts ASC RANGE ...)) takes the cached CachedWindowRecordCursorFactory path, which materialises the base into a record chain whose columns are reordered -- the window output columns occupy the leading slots and the remaining base columns are appended afterwards. A RANGE-framed function reads the designated timestamp by index, and the code generator passed it the base cursor's timestamp index. When the timestamp was not part of the SELECT projection it landed at a different index in the chain than in the base, so the function read the wrong chain column -- frequently its own, as-yet-unwritten output slot -- as the timestamp and computed the frame over uninitialised memory. The value came out wrong and varied between consecutive executions of the same SQL because the slot reuses freed native memory; this was the open bug previously listed below. The code generator now tracks where the designated timestamp lands in the chain and passes that index to the cached window functions, leaving the streaming path unchanged. WindowFunctionTest.testRangeFrameTimestampIndexInCachedWindow pins a count ordered by ts DESC plus a min over a RANGE frame ordered by ts ASC with an unprojected timestamp; before the fix the min framed on a zeroed slot and collapsed to the whole-partition prefix minimum. Repro (now correct): seeds 553854821671111/1781174964170, 553832024235320/1781174941373 and 553862616269146/1781174971965.

  13. sum() / avg() over a high-precision DECIMAL produced a wrong, run-varying result over a sliding frame. sum or avg over a Decimal256-backed argument running over a sliding frame accumulated values with the raw, scale-agnostic Decimal256.uncheckedAdd but evicted the oldest value as it left the frame through the scale-aware Decimal256.subtract. The running accumulator keeps its scale field at 0 (it holds raw limbs), while a value arriving through a cast such as (c2)::DECIMAL(76, 6) carries scale 6. The first non-null eviction therefore rescaled the accumulator from scale 0 up to the value scale, multiplying the running total by 10^scale and baking a large constant offset into every later row of that frame. The accumulator instance is shared across the cursor's forward passes, so the rescale happened only once: the first pass (scale 0) was wrong and the post-toTop() re-read came out correct, which is how the fuzzer's cursor self-consistency check flagged it. A new Decimal256.uncheckedSubtract, the scale-agnostic counterpart of uncheckedAdd, now performs the frame eviction in the Decimal256 sum, avg and rescaled-avg window functions across the rows, partition-rows, range and partition-range frames, so a removed value cancels exactly the limbs that uncheckedAdd contributed regardless of either operand's scale field; the Decimal128 paths already subtracted raw limbs with an explicit scale of 0 and were unaffected. Decimal256Test covers uncheckedSubtract (raw subtraction, scale independence, add/subtract round-trip and cross-limb borrow) and WindowDecimalFunctionTest.testSumAvgDecimal256SlidingFrameEviction materializes the raw first pass and the toTop() pass separately and pins the hand-computed sums and averages. Repro (now correct): seed 625297067085445/1781246406416, SELECT sum((c2)::DECIMAL(76, 6)) OVER (PARTITION BY sym ORDER BY ts ASC ROWS BETWEEN 7 PRECEDING AND CURRENT ROW) FROM fuzz_t2.

  14. TopK / ORDER BY ... LIMIT cleanup crashed with NullPointerException when a tree-chain allocation failed mid-construction. LimitedSizeLongTreeChain allocates two DirectIntList freelists and a native value heap in its constructor, and on any allocation failure the constructor's error path calls close(). close() routed through clear(), which dereferenced chainFreeList.clear() directly, so a malloc fault on the second freelist (or the value-heap allocation after it) left chainFreeList null and the cleanup threw NullPointerException, masking the real out-of-memory error. close() now resets the tree state inline and frees the freelists through the null-safe Misc.free(). The inline state reset is load-bearing rather than cosmetic: the chain is reused across cursor passes via reopen(), and toTop() reads chain.size(), so a first attempt that simply dropped the clear() call and reset nothing regressed OrderByLimitTest.testAsOfJoinWithNestedTopK, whose second cursor pass then read a stale row count and returned an empty result. A sweep for the same partial-construction shape fixed two more sites, neither reachable through the fuzzer today because fault queries run with parallel execution disabled and the first is a parallel-only path: AsyncHorizonJoinAtom / AsyncMultiHorizonJoinAtom.closeAggregationState() dereferenced shardingCtx.close() although the base-class constructor calls close() on its own error path before the subclass assigns shardingCtx, and RecordChain.clear() dereferenced mem.close() when Vm.getCARWInstance() itself faulted; both now use Misc.free(). Repro (now passing): seed 633879473312317/1781254988822, SELECT 'CDZWD'::VARCHAR, ... FROM long_sequence(35) WHERE (x IS NULL OR ...) ORDER BY 4 LIMIT 34.

  15. JIT filter wrapped a nested INT product that the Java filter widened to 64 bits, giving JIT-only wrong results. Surfaced by the cursor self-consistency check. A predicate like c9 <= ((c0 - c2) * (c8 * -776782)) (with c9 FLOAT, c0 LONG, c2 SHORT, c8 INT) computes an inner INT * INT product that overflows int32 and then feeds a LONG-width multiply. The Java filter reads the inner subtree through MulInt / AddInt / SubInt#getLong, which widen every operand to 64 bits and never wrap; the JIT IR serializer normally matches this by pre-widening narrow operands to i64, but a FLOAT/DOUBLE operand in the predicate suppresses that widening, so the JIT computed the inner product at int32 width (wrapping mod 2^32) and only sign-extended the already-wrapped result. The two paths disagreed whenever the inner product overflowed int32: the JIT-on cursor returned 55 rows where the Java filter returned 54, and under the parallel filter this surfaced as a toTop() re-read changing the row set (51 then 58). CompiledFilterIRSerializer now walks each float-suppressed predicate and tags the narrow leaves that live under a LONG-width arithmetic subtree, emitting a 64-bit sign-extend (columns / bind variables) or an i64 immediate (constants) for exactly those, so the JIT computes the inner product at 64 bits and stays on rather than falling back to the slower Java filter. The tagging mirrors the C++ convert() rule: only the operands of a narrow-width op computed at 64 bits are sign-extended, because a wide (i64) op already promotes its narrow operand -- so anint * 9000000000 (whose i64 constant already lifts the multiply to 64 bits) is left untouched while anint * 100000 under a long multiply is widened. The sign-extend is scalar-only, so a tagged predicate runs scalar instead of vectorized; that is a small cost against keeping these uncommon mixed-width nested-arithmetic-with-float predicates on the JIT at all. CompiledFilterRegressionTest.testNarrowIntArithUnderLongWithFloat compares the JIT and Java cursors for the diverging shapes and pins that they stay JIT-compiled, and CompiledFilterIRSerializerTest adds two IR-level tests: one pins the sign-extend on the inner INT product, the other pins that an i64 constant operand suppresses the widening. Repro (now correct): seed 643056465770421/1781264165814, SELECT t0.* FROM fuzz_t1 t0 WHERE t0.c9 <= ((t0.c0 - t0.c2) * (t0.c8 * -776782)).

  16. Keyed ASOF JOIN sink-heap leak on out-of-memory during cursor open. The keyed ASOF JOIN cursors backed by AsOfJoinFastRecordCursorFactory and FilteredAsOfJoinFastRecordCursorFactory reopen two SingleRecordSink heaps in of(). reopen() allocates each heap lazily, so when the second reopen() tripped the RSS memory limit the first sink's heap was already allocated. of() then threw out of getCursor(), whose catch block frees the master and slave cursors but not the reusable cursor, and the factory's _close() does not free that cursor either, so the half-opened cursor was orphaned and its 8-byte heap, tagged NATIVE_RECORD_CHAIN, leaked. Unlike the constructor-cleanup leaks above (Event Appender #1, Parameterize TxListener.onCommit() #4, Messaging Patterns #14) the fault hits the cursor-open path, so the fix lives in of() rather than close(): of() now reopens both sinks inside a try/catch that closes them on any throwable before rethrowing, so a failed cursor open leaves no native memory behind. Closing a sink that never allocated is a no-op, so the cleanup is correct regardless of which reopen() failed. The query fuzzer's malloc fault injection surfaced this. AsOfJoinFastOomTest sweeps the RSS ceiling across the cursor-open allocation points for both the plain and filtered keyed cursors and asserts the join leaves no native memory behind. Repro (now passing): seed 654768710483172/1781275878059.

  17. A LIMIT query whose filter overflows on a row past the cutoff surfaces the cast error non-deterministically across cursor passes -- accepted parallel-filter non-determinism, not a cursor-reuse defect (fuzzer over-reported it). Surfaced by the cursor self-consistency check. The parallel filter reduces whole page frames eagerly on worker threads, so a row whose filter expression overflows -- e.g. (c6 - c2)::DECIMAL(9, 3) for a value outside DECIMAL(9,3) -- can sit in a frame past the LIMIT cutoff. Whether that frame's stored ImplicitCastException surfaces depends on which frames the reducer evaluated before the LIMIT cancelled the sequence, so the same cursor returns its 18 rows on the clean first pass and throws the cast error on a re-pass. The failing seed tripped this through the toTop() re-iteration; replaying it tripped the identical query through the calculateSize() recompute; other replays did not trip it at all -- the mark of a reduction-order race, not a deterministic toTop() defect. It is the same non-determinism the differential axes already tolerate (an out-of-range cast surfaces on a full scan but not on an index probe). Both self-checks now treat a re-pass throw of a per-row cast or numeric overflow (ImplicitCastException / NumericException) after a clean first pass as a tolerated skip, carried on a new CursorCheckException.tolerated flag that run() converts to a skip; a divergent row set, a changed preComputedStateSize(), a count mismatch, or a re-pass throw of any other type still fails the query. The valve is deliberately narrower than the oracle's isAcceptedSkip: a SqlException is raised at compile time and cannot fire mid-iteration, and a decimal-aggregation overflow rides the aggregate accumulator rather than the filter, where a per-pass divergence is a real defect -- it is how bug fix Export tool #13 was caught -- not reduction-order noise. It keys on exactly the ImplicitCastException / NumericException pair the differential axes' index/parquet asymmetry rules (bug fix Asynchronous append() #9's neighbours) already use, narrowing the self-checks' "no skip valves" guarantee to this one data-dependent-error case. Repro (now skipped or clean across reruns): seed 659553157043430/1781280662506, SELECT * FROM fuzz_t0 WHERE NOT ('2024-12-27'::DATE = ((c6)::LONG)::DATE OR (c6 - c2)::DECIMAL(9, 3) < (c2)::DECIMAL(76, 2)) ORDER BY ts DESC LIMIT 18.

  18. A FILE-fault query reported a native-memory leak that an engine background job, not the query, caused (fuzzer over-reported it). The fault injection's FailureFileFacade armed a process-global op counter, so it counted down and fired on file ops from any thread rather than just the query thread. While a FILE fault was armed for one query, the engine's background posting-seal purge job -- running on a shared worker thread and routing its own file I/O through the same facade -- tripped the fault on a writer append, distressing that writer and leaving ~512 KiB mapped; the per-query leak oracle, which samples process-global native memory, then charged that to the query under test even though the query referenced a nonexistent table and opened no data files. The global counter also made the "fail after N ops" trigger point depend on background-thread timing, so the fire point was not reproducible from the seed alone. The facade gains an opt-in thread-scoped arming method (setToFailAfterOnCurrentThread) that the query fuzzer uses for FILE faults: only ops on the arming thread count down and fire. Fault queries already run with parallel execution disabled, so the query under test issues all its file ops on that thread and fault coverage is unchanged -- FILE still fires on ~67% of armed queries at -Dquestdb.fuzz.fault.pct=100 -Dquestdb.fuzz.queries=2000. The default unscoped setToFailAfter is left as is for the WAL/recovery fuzz harness, whose injected failures legitimately surface on background apply/purge threads. Repro (now green): seed 672609384240043/1781347718070.

  19. Vectorized GROUP BY leaked a page-frame address buffer on out-of-memory during cursor open. The keyed count() aggregate under a LIMIT from the open bug below runs through the vectorized rosti GROUP BY (GroupByRecordCursorFactory), not the map-based AsyncGroupByRecordCursorFactory the bug was first attributed to. RostiRecordCursor.of() reopens the factory's shared PageFrameAddressCache, whose of() reopens four off-heap DirectLongLists one by one; each reopen() reallocates native memory and can trip the RSS limit. When a later reopen() failed after an earlier one had already allocated, of() propagated the exception leaving the cache partially open, and because getCursor() then never returned the cursor, the caller could not close it and the factory's _close() does not free the cache either -- so the already-reopened 512-byte DirectLongList (NATIVE_DEFAULT) was orphaned. This made the leak parallel-only only incidentally: the rosti path is reached the same way under serial execution, but the seeds that hit it happened to run the aggregate in the parallel phase. PageFrameAddressCache.of() now reopens the lists atomically: on any reopen() failure it closes the ones already reopened, returning the cache to its fully-closed (zero native memory) state, so a caller that propagates the failure without reaching close() leaks nothing. The fix covers every of() caller, not just the rosti GROUP BY (the async filter / group-by page-frame sequences, SAMPLE BY first/last, the time-frame cursor and the QWP egress path). GroupByVectorizedOomTest sweeps the RSS ceiling across the cursor-open allocation points for a single-key sum() aggregate and asserts the query leaves no native memory behind. Repro (now green): seed 709239330479655/1781513480332 with -Dquestdb.fuzz.queries=400 -Dquestdb.fuzz.fault.pct=70; minimal query WITH cte0 AS (SELECT c0 AS k, count() AS cnt FROM fuzz_t0) SELECT * FROM cte0 LIMIT 30.

  20. SELECT max(ts) FROM t WHERE ... and its min / first / last siblings over the designated timestamp swallowed a fired fault under parallel execution -- legitimate early termination from an implicit LIMIT, not a swallowed error (fuzzer over-reported it). SqlOptimiser.rewriteSingleFirstLastGroupBy rewrites a lone min / max / first / last over the designated timestamp into an ORDER BY ts [DESC] LIMIT 1 page-frame scan: the newest (for max / last, a backward scan) or oldest (for min / first, a forward scan) matching row answers the aggregate, so the aggregation step disappears. Under the eager parallel reduce a fired test_fault() can land on a frame whose rows fall past that single-row cutoff -- the faulting frame's result is discarded and the query completes with a correct answer. That is the same legitimate early termination an explicit LIMIT gives, and which frame the fault lands on depends on worker timing, so the verdict must not flip on it. The swallow oracle already tolerated an explicit LIMIT by matching the limit keyword in the SQL text, but the rewrite leaves no keyword there. QueryRunner now also consults the rewritten plan: runRaw / runRawMallocFault render the factory plan (heap-only, before the cursor opens / before the RSS limit is armed, so it stays outside the fault window) and carry a hasPushedLimit flag on the Outcome. planHasPushedLimit keys off the limit: attribute that AsyncFilteredRecordCursorFactory / AsyncJitFilteredRecordCursorFactory emit only for a pushed-down limit, so the relaxation spans both the explicit and the implicit cases. Aggregates that keep a real aggregation step carry no pushed limit and so are unaffected -- max(c0) over a non-timestamp column, count(), and max(ts) projected alongside another column all run through the async GROUP BY, which surfaces a fired fault as before. This is a test-oracle change only; no engine behaviour changes. QueryFuzzTest.testImplicitTimestampLimitTolerated pins the pushed-limit plan marker for the four timestamp aggregates and its absence for the non-rewritten aggregates. Repro (now tolerated): seed 718495134431076/1781522736136, query SELECT max(ts) AS a0 FROM fuzz_t2 WHERE test_fault() ORDER BY a0.

  21. The vectorized rosti GROUP BY over-freed NATIVE_ROSTI native memory when an aggregate's wrapUp() resized the map (previously listed as known open bug Event Appender #1). On the single-worker build path the factory called each aggregate function's wrapUp() without bracketing it with updateMemoryUsage(). wrapUp() inserts the null-key slot that carries the aggregate value for column-top rows; when the live keys have already filled the map to its growth threshold, that insert resizes the rosti. The single-worker path did not record the resize, so the cursor's close() reset later subtracted the full grown size from the NATIVE_ROSTI tag and the tag ended the run with a negative net delta -- the over-free reported as NATIVE_ROSTI, difference: -<n> (one resize step, tens of KiB). The multi-worker merge path already bracketed wrapUp() with updateMemoryUsage() for exactly this reason ("some wrapUp() methods can increase rosti size"); the single-worker path now does the same, and the multi-worker path is unchanged. This is a pure accounting defect -- query results are unaffected -- and it reproduces deterministically without any fault; the fault injection surfaced it only as an end-of-run NATIVE_ROSTI imbalance. GroupByVectorizedRostiAccountingTest sweeps the live-key count across the growth boundary with column-top null keys so one iteration lands the wrapUp resize, and asserts the NATIVE_ROSTI accounting stays balanced on both the single-worker and multi-worker build paths.

  22. The vectorized rosti GROUP BY left a stale aggregate task in the shared queue when a fault aborted the cursor's drain, crashing a later query that work-stole it (previously listed as known open bug Event Appender #1). GroupByRecordCursorFactory publishes per-frame VectorAggregateEntry tasks to the engine-global vector-aggregate queue, then in buildRosti's finally block drains the queue through runWhatsLeft -- work-stealing and running each task -- until its own doneLatch is satisfied, before freeing the per-build frameMemoryPools. runWhatsLeft let a stolen entry.run() throw straight out, so when a MALLOC fault tripped a parquet decode inside navigateTo during the drain, the throw aborted the drain with published entries still in the queue. Because that throw originates in the finally, it bypasses the catch that cancels the shared circuit breaker, so the survivors kept a live (non-cancelled) circuit breaker and a zero oomCounter -- the guards that would otherwise make a stolen entry skip its work. The factory then closed and freed its frameMemoryPools, nulling each pool's addressCache, so a later vectorized GROUP BY -- including the fault oracle's recovery re-run -- work-stole a survivor and dereferenced the freed pool, throwing NullPointerException: Cannot invoke "PageFrameAddressCache.getFrameFormat(int)" because "this.addressCache" is null (the same NPE the GroupByVectorAggregateJob worker threads were already swallowing and logging). runWhatsLeft now wraps each stolen entry.run(): a throwing entry no longer aborts the drain, so the build's published entries are all consumed and none survives in the shared queue referencing the about-to-be-freed pools; the first error surfaces once the queue is fully drained, preserving the query's failure semantics. GroupByVectorizedOomTest.testWorkStolenEntryDoesNotOutliveFreedPoolsOverParquet converts a multi-partition table to parquet and sweeps the RSS ceiling across buildRosti (armed after compile so the trip lands in the drain), running a clean recovery query after each ceiling; before the fix the recovery work-stole a survivor and hit the NPE. Repro (now green): seed 718495134431076/1781522736136 with -Dquestdb.fuzz.fault.pct=60 -Dquestdb.fuzz.queries=2000 -Dquestdb.fuzz.window=false.

  23. A GROUP BY key that references the alias of a trivial arithmetic expression failed to compile, while the same query with a bind variable in place of the constant compiled (literal/bind divergence). SqlOptimiser.rewriteTrivialGroupByExpressions lifts a trivial key such as 859371 + (cnt * -237288) out of the inner GROUP BY when its base column (cnt) is also a key, recomputing the offset in an outer VIRTUAL and removing the lifted key column from the group-by model (the ClickBench Q35 optimisation). It removed the column but left the key's alias reference behind in the model's GROUP BY list, so GroupByUtils.validateGroupByColumns looked the alias up in the model's now-smaller column set, did not find it, and rejected the query with "group by column does not match any key column". The bind-variable form (:b0::INT + (cnt * -237288)) hid the bug: a bind-variable cast is a FUNCTION node, which the trivial-expression visitor does not treat as trivial, so the lift never ran and the key stayed in the group-by model. The literal form was therefore rejected at compile time while the bind form compiled and returned rows -- the divergence the bind-variant axis flagged. This is an engine bug, not a fuzzer-oracle issue: the query is valid SQL (grouping by cnt and a function of cnt produces the same groups as grouping by cnt alone) and must compile either way. rewriteTrivialGroupByExpressions now drops the stale alias reference from the nested GROUP BY list whenever it removes the lifted key column; expression keys spelled out in GROUP BY (e.g. ClientIP - 1) are untouched because validateGroupByColumns matches those against the surviving base column rather than by alias. GroupByTest.testGroupByTrivialExpressionKeyReferencedByAlias pins both the alias-reference and spelled-out forms against the hand-computed result. Repro (now correct): seed 738911385687093/1781543152387, query SELECT t0.cnt AS e0, (859371 + (t0.cnt * -237288)) AS e1, count() AS a0 FROM (SELECT c4 AS k, count() AS cnt FROM fuzz_t1) t0 WHERE ... GROUP BY t0.cnt, e1 ORDER BY 1 ASC, e1.

  24. A multi-key index scan (sym = ... OR sym = ..., sym != ...) leaked posting-index block buffers when an out-of-memory error hit while building its per-key row cursors on the first page frame. HeapRowCursorFactory.getCursor() builds one RowCursor per symbol key into the factory's own cursors list, then hands that list to the HeapRowCursor via cursor.of(). HeapRowCursorFactory.close() relied solely on Misc.free(cursor) -> HeapRowCursor.close() to drain the list, but the HeapRowCursor only frees the list reference it received in of(); when a MALLOC fault tripped the RSS limit mid-build on the very first page frame -- before of() ran -- that reference was still null, so the per-key posting cursors already built into the factory's list were never closed and each retained its block buffer (NATIVE_INDEX_READER), which the owning PostingIndexFwdReader / PostingIndexBwdReader's close() never reclaimed because the cursor was never returned to the reader's free pool. HeapRowCursorFactory.close() now drains its own cursors list directly; once of() has run the two references point at the same list, so the HeapRowCursor's close() empties it and the extra drain is a no-op. Separately, PostingIndexFwdReader.getCursor() / PostingIndexBwdReader.getCursor() now release a cursor's native buffers if its of() throws -- a cursor popped from the free pool (or freshly created) is not yet owned by the caller, so a throw while growing the block buffer would otherwise strand it with neither the caller nor the reader able to reclaim it. SequentialRowCursorFactory shares the build-into-a-list shape but is unaffected: its SequentialRowCursor is an inner class that frees the factory's cursors field directly. Surfaced by malloc fault injection as an end-of-run NATIVE_INDEX_READER, difference: 144 imbalance. Repro (now green): seed 747545817585554/1781551786819.

  25. A non-keyed count_distinct over a constant (or a fully-enumerated symbol) swallowed a fired fault under parallel execution: legitimate early termination from the group-by's early exit, not a swallowed error (fuzzer over-reported it). When a non-keyed aggregate's value is final after the first matching row (a count_distinct over a constant always lands on 1, and over a symbol it stops once every distinct symbol has been seen), the optimiser routes the query to GroupByNotKeyedRecordCursorFactory's early-exit cursor, which stops scanning the base cursor after that row. Beneath it the async filter keeps reducing later page frames on worker threads, so a fired test_fault() can land on a frame the early-exit consumer never reads; that frame's result and its throw are discarded and the query completes with a correct answer. This is the same legitimate early termination an explicit or pushed LIMIT gives (bug fix Generic data access #20), and which frame the fault lands on depends on worker timing, so the verdict must not flip on it. The swallow oracle already tolerated a row-limit cutoff (the limit keyword in the SQL text or the limit: plan attribute); it now also tolerates an early-exit non-keyed group by. QueryRunner.runRaw / runRawMallocFault carry a hasEarlyExitGroupBy flag on the Outcome, computed by factoryHasEarlyExitGroupBy, which walks the factory's base-factory chain (the top-level consumption spine, not anywhere in the plan as the limit check does) for a GroupByNotKeyedRecordCursorFactory whose isEarlyExitSupported() is true. The relaxation is narrow and cannot hide an error on a frame the query actually reads: a fault on a consumed frame surfaces when the group by pulls it, so outcome.failure is non-null and the swallow branch never runs; and the fuzzer injects test_fault() only into the single scan a non-keyed group by consumes. Aggregates that keep scanning every row carry no early-exit cursor and surface a fired fault as before: count_distinct over a long constant and count(*) run through the async GROUP BY, and a plain count_distinct(symbol) is rewritten to a keyed group by. The only production change is a read-only isEarlyExitSupported() predicate on GroupByNotKeyedRecordCursorFactory; no engine behaviour changes. EarlyExitGroupByDetectionTest pins the detection for count_distinct over a constant (with and without a filter) and its absence for the non-early-exit shapes; the swallow reproduces about 1 in 50 runs at -Dquestdb.fuzz.workers=12 with the tolerance forced off. Repro (now tolerated): seed 808495966924054/1781612736968, query SELECT count_distinct('B') AS a0 FROM fuzz_t2 WHERE test_fault().

Window-function fuzzing sweep (no known open bugs remain)

The window cursor toTop() self-consistency failure previously listed here -- re-iterating returned a different row set for a query partitioning a window by a Decimal256 column -- shares its root cause with bug fix #10 above and is fixed by it. The seed that surfaced it now runs clean:

mvn -pl core -Dtest=QueryFuzzTest -Dquestdb.fuzz.s0=394723763672806 -Dquestdb.fuzz.s1=1781015833112 -Dquestdb.fuzz.queries=2000 test

A 10-seed sweep at -Dquestdb.fuzz.queries=3000 (window and fault shapes on by default) went red on 5 of 10 seeds, every failure falling into one of two window-function defects. Both are now fixed: the lead(null) crash (bug fix #11 -- seeds 553825244550718/1781174934593 and 553881716081004/1781174991065) and the RANGE-frame timestamp-index bug (bug fix #12 -- seeds 553854821671111/1781174964170, 553832024235320/1781174941373 and 553862616269146/1781174971965). No fault-injection leak or recovery failures and no other crashes remained, so the serial cleanup paths the earlier fixes cover are holding. The sweep now runs green.

Bind-variable differential: bug surfaced, fixed in #7225 (merged)

The bind-variable variant (each bindable literal also runs as a :bN::TYPE placeholder, with the two forms cross-checked) surfaced a wrong-results bug in SPLICE / RIGHT OUTER / FULL OUTER joins: a WHERE predicate referencing only the NULL-extended master (left) table was pushed below the join, so the null-master output rows bypassed it; for a literal constant the optimiser could additionally propagate the predicate to the slave through the join key, which a bind variable cannot, so the literal and bind forms disagreed on the row count. This is an engine optimiser bug independent of the fuzzer; the fix landed in its own dedicated PR, #7225, not on this branch. #7225 has since merged to master and is now included here through the master merge, so the repro runs green on this branch. Repro seed: -Dquestdb.fuzz.s0=574598955642074 -Dquestdb.fuzz.s1=1781195708304. The bind-variable axis also surfaced bug fix #23 above, an unrelated GROUP BY compile-time divergence (a trivial-expression key referenced by alias), which is fixed on this branch.

Known open bugs

None remain. A high fault rate surfaced three MALLOC-fault-recovery defects in the page-frame allocation paths, all now fixed: the 512-byte NATIVE_DEFAULT leak by a keyed GROUP BY under a LIMIT (bug fix #19, traced to the vectorized rosti GROUP BY's page-frame address cache, not the map-based parallel GROUP BY it was first filed against), the NATIVE_ROSTI over-free (bug fix #21, the single-worker wrapUp() resize was not bracketed by updateMemoryUsage()), and the recovery crash from a stale aggregate task left in the shared vector-aggregate queue (bug fix #22). The seeds that surfaced them now run clean, including an 8000-query sweep at -Dquestdb.fuzz.fault.pct=60 (about 4x the default) with 905 MALLOC faults fired.

Tradeoffs

  • With faults and window shapes enabled by default, CI can go red until the surfaced bugs are fixed. The fault rate is modest (15%), and disabling faults was previously enough for a green run; the window shapes hit remaining open defects on many seeds, so until those are fixed a clean run needs -Dquestdb.fuzz.window=false.
  • Fault queries now run under parallel SQL execution by default. Query (SQL reduce) jobs and writer (O3 / purge / index) jobs run on two separate worker pools; the writer pool is halted once the tables are built, so during the query loop the only file ops and native allocations come from the test thread and the query-pool workers -- the query under test. That quiesced query phase lets FILE and MALLOC faults exercise the parallel error-path cleanup that was previously uncovered, at the cost of fire-point determinism (under parallel reduce, "fail after N ops" fires at a non-deterministic point across worker threads, so a seed reproduces the pass/skip/fail verdict but not the exact fault site). This surfaced a parallel-only leak in the vectorized GROUP BY cursor open, now fixed (bug fix Benchmarks #19); pass -Dquestdb.fuzz.fault.parallel=false to fall back to serial fault injection.
  • The per-query leak check drains pools and tolerates a small slack, so it catches gross leaks; the end-of-run assertMemoryLeak remains the authoritative net check.
  • User-visible behaviour change (bug fix Persist to Existing DB #11): a window aggregate / navigation function over an untyped null literal -- sum(null) OVER (...), lag(null) OVER (...), etc. -- now raises a clean SqlException asking for a concrete cast, where some of these variants previously resolved to a concrete type and silently computed over NULL. The previous behaviour was platform-dependent, so this trades a small loss of a degenerate, non-portable form for a uniform, deterministic rejection; null::double (or any concrete cast) keeps working.

Test plan

  • mvn -pl core -Dtest=QueryFuzzTest test (faults and window shapes on by default)
  • High-rate fault stress: -Dquestdb.fuzz.fault.pct=70 -Dquestdb.fuzz.queries=500
  • Replay a failure: -Dquestdb.fuzz.s0=L -Dquestdb.fuzz.s1=L
  • Opt out of faults: -Dquestdb.fuzz.faults=false
  • Opt out of window shapes: -Dquestdb.fuzz.window=false

Summary by CodeRabbit

  • Bug Fixes

    • Improved resource cleanup and memory leak prevention across query execution paths
    • Fixed error propagation when sidecar files fail to load
    • Resolved GROUP BY alias handling in nested queries
    • Enhanced null-safety in resource deallocation
  • New Features

    • Added DATE type support for window functions (first_value, last_value, nth_value, max, min, lead, lag)
    • Expanded window function framework with improved timestamp tracking in cached windows
  • Performance

    • Optimized decimal arithmetic operations in window aggregations
    • Improved query planning for window function execution

puzpuzpuz and others added 3 commits June 9, 2026 12:29
QueryRunner now re-verifies every materialized cursor against four
invariants that hold for any RecordCursor regardless of access path:

- re-iterating after toTop() reproduces the first pass (same row
  multiset for a deterministic query, same row count otherwise);
- toTop() leaves preComputedStateSize() unchanged;
- size(), when known, equals the materialized row count;
- calculateSize() from the top counts that same number of rows and
  leaves hasNext() false.

These mirror the calculateSize()/preComputedStateSize() cross-checks
the assertQuery battery runs. Because no benign access-path choice can
make a cursor diverge from itself, a violation always fails the query
with no skip valves, unlike the JIT/storage/bind diff axes. checkToTop
and checkCalculateSize run on every pass (primary, shadow, JIT-on/off,
bind) via a CursorCheckException that short-circuits the per-axis
reconciliation, so each distinct factory shape gets exercised.

A new questdb.fuzz.verify.cursor flag (default true) gates the checks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The query fuzzer can now inject a fault into a query run and apply a
crash-and-recover oracle in place of the differential one (a faulting
query is not comparable). A per-query coin flip, drawn from the seeded
rnd so replay reproduces it, selects a fault kind:

- FILE: one filesystem op fails once via FailureFileFacade, installed
  as the engine's files facade for the run;
- MALLOC: a native allocation fails once via the RSS memory limit, the
  same out-of-memory path production hits.

FaultType also reserves FUNCTION for a follow-up.

For each injected query QueryRunner.runFault arms the fault at a random
trigger point, runs the query once, and then asserts: a fired fault is
not swallowed; native memory and open file descriptors return to their
pre-fault baseline after the cursor closes, proving the factory freed
its resources on the error path; and the same query runs cleanly once
the fault is removed.

Fault queries run with parallel execution disabled. A fault thrown
mid-parallel-execution can leave a pooled page-frame reduce task
holding an un-released buffer, which then corrupts a later query that
reuses the task; serial execution exercises the cleanup path without
that shared-state hazard.

Two new knobs control the feature: questdb.fuzz.faults (default true)
and questdb.fuzz.fault.pct (default 15).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the fuzzer's fault-injection set with a FUNCTION fault. A new
dev-mode test function, test_fault(), returns true until armed and then
throws once on its Nth per-row evaluation; QueryGenerator weaves it into
a generated query's WHERE clause (via the shared PredicateGenerator
helper) so the throw happens mid-cursor, and QueryRunner.runFault arms it
like the FILE and MALLOC faults. The emission draws the same rnd ops
whether or not it fires, so non-fault generation stays byte-identical and
the bind-variant determinism invariant holds; function-fault queries skip
the posting path so test_fault() always lands in an emitting clause.

QueryFuzzTest enables dev mode so the function is active. The run summary
now reports how many injected faults actually fired (vs armed but not
reached).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@puzpuzpuz puzpuzpuz added the SQL Issues or changes relating to SQL execution label Jun 9, 2026
@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: a3cfe671-3176-4d30-9a92-c47920018cf2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Walkthrough

This PR hardens several engine failure paths, rewrites backward interval size calculation, adds shared and DATE-specific window-function implementations, updates Decimal256 and compiled-filter behavior, and expands fuzz, fault-injection, OOM, and regression test coverage.

Changes

Engine runtime correctness and cleanup

Layer / File(s) Summary
Backward interval sizing and cached timestamp remap
core/src/main/java/io/questdb/cairo/IntervalBwdPartitionFrameCursor.java, core/src/main/java/io/questdb/griffin/SqlCodeGenerator.java, core/src/test/java/io/questdb/test/griffin/engine/table/PageFrameRecordCursorImplFactoryTest.java
Backward calculateSize() now mirrors backward interval traversal semantics, cached window context uses the timestamp slot in chain metadata, and interval tests cover open-ended bounds and both scan directions.
Constructor, reopen, and cursor cleanup paths
core/src/main/java/io/questdb/cairo/RecordChain.java, core/src/main/java/io/questdb/cairo/idx/*, core/src/main/java/io/questdb/cairo/sql/PageFrameAddressCache.java, core/src/main/java/io/questdb/griffin/engine/groupby/..., core/src/main/java/io/questdb/griffin/engine/join/..., core/src/main/java/io/questdb/griffin/engine/table/..., core/src/test/java/io/questdb/test/cairo/*, core/src/test/java/io/questdb/test/griffin/engine/groupby/*, core/src/test/java/io/questdb/test/griffin/engine/join/*, core/src/test/java/io/questdb/test/griffin/engine/table/parquet/*
Multiple readers, joins, group-by cursors, and table helpers now free partial allocations or rethrow after cleanup, with OOM and file-fault regressions validating recovery and leak-free teardown.
Compiled filter widening fix
core/src/main/java/io/questdb/jit/CompiledFilterIRSerializer.java, core/src/test/java/io/questdb/test/griffin/CompiledFilterRegressionTest.java, core/src/test/java/io/questdb/test/jit/CompiledFilterIRSerializerTest.java
Compiled filter serialization now tracks specific narrow integer leaves that require i64 widening under float expressions, and tests cover emitted IR and JIT-versus-Java result parity.

Window execution and SQL correctness

Layer / File(s) Summary
SQL resolution and validation fixes
core/src/main/java/io/questdb/griffin/FunctionParser.java, core/src/main/java/io/questdb/griffin/SqlOptimiser.java, core/src/test/java/io/questdb/test/griffin/GroupByTest.java, core/src/test/java/io/questdb/test/griffin/engine/window/WindowFunctionTest.java
Window resolution now rejects ambiguous untyped NULL arguments, lifted trivial GROUP BY aliases remove stale nested literals, and regressions cover both behaviors plus cached RANGE timestamp remapping.
Shared max/min helper and DATE-capable base handling
core/src/main/java/io/questdb/griffin/engine/functions/window/AbstractWindowFunctionFactory.java, core/src/main/java/io/questdb/griffin/engine/functions/window/MaxMinWindowFunctionFactoryHelper.java, core/src/main/java/io/questdb/griffin/engine/functions/window/MinTimestampWindowFunctionFactory.java, core/src/main/java/io/questdb/griffin/engine/functions/window/MaxDateWindowFunctionFactory.java, core/src/main/java/io/questdb/griffin/engine/functions/window/MinDateWindowFunctionFactory.java
Window base helpers now snapshot key types and read DATE values, min timestamp creation delegates through a new shared max/min helper, and DATE max/min factories are built on the shared dispatcher and frame bases.
DATE first, last, and nth window factories
core/src/main/java/io/questdb/griffin/engine/functions/window/FirstValueDateWindowFunctionFactory.java, core/src/main/java/io/questdb/griffin/engine/functions/window/LastValueDateWindowFunctionFactory.java, core/src/main/java/io/questdb/griffin/engine/functions/window/NthValueDateWindowFunctionFactory.java, core/src/main/java/io/questdb/griffin/engine/functions/window/NthValueTimestampWindowFunctionFactory.java, core/src/test/java/io/questdb/test/griffin/engine/window/WindowDateFunctionTest.java, core/src/test/java/io/questdb/test/griffin/engine/window/WindowFunctionTest.java
New DATE-specific first_value, last_value, and nth_value factories are added, timestamp nth-value wrappers move onto shared helpers, and DATE window behavior is covered across cached, running, partitioned, range, rows, and cast cases.
Nth helper, rank state, and Decimal256 frame eviction
core/src/main/java/io/questdb/griffin/engine/functions/window/NthValueWindowFunctionFactoryHelper.java, core/src/main/java/io/questdb/griffin/engine/functions/window/RankFunctionFactory.java, core/src/main/java/io/questdb/griffin/engine/functions/window/CumeDistFunctionFactory.java, core/src/main/java/io/questdb/griffin/engine/functions/window/PercentRankFunctionFactory.java, core/src/main/java/io/questdb/std/Decimal256.java, core/src/main/java/io/questdb/griffin/engine/functions/window/*Decimal*WindowFunctionFactory.java, core/src/test/java/io/questdb/test/std/Decimal256Test.java, core/src/test/java/io/questdb/test/griffin/engine/window/WindowDecimalFunctionTest.java, core/src/test/java/io/questdb/test/griffin/engine/window/WindowFunctionTest.java
nth_value execution is centralized in a shared helper, rank and percentile paths snapshot key types and add a streaming ordered-partition path, Decimal256 adds unchecked subtraction for sliding frame eviction, and regressions cover rank, cume/percent rank, decimal overflow, and frame re-iteration behavior.

Fuzz fault injection and query oracle updates

Layer / File(s) Summary
Fault primitives and configuration
core/src/main/java/io/questdb/griffin/engine/functions/test/TestFaultFunctionFactory.java, core/src/test/java/io/questdb/test/cairo/fuzz/FailureFileFacade.java, core/src/test/java/io/questdb/test/griffin/fuzz/FaultType.java, core/src/test/java/io/questdb/test/griffin/fuzz/FuzzConfig.java
A dev-only test_fault() function, scoped file-fault arming modes, a fault-type enum, and fuzz configuration flags for faults, cursor verification, and window generation are added.
Fault-aware query generation
core/src/test/java/io/questdb/test/griffin/fuzz/PredicateGenerator.java, core/src/test/java/io/questdb/test/griffin/fuzz/QueryGenerator.java, core/src/test/java/io/questdb/test/griffin/fuzz/clauses/*
Predicate and clause generation now thread fault injection through WHERE construction, query generation adds a window-query band and suppresses posting-index reads during injected faults, and a new window clause builds randomized OVER (...) queries.
Runner verification and fault oracles
core/src/main/java/io/questdb/griffin/engine/groupby/GroupByNotKeyedRecordCursorFactory.java, core/src/test/java/io/questdb/test/griffin/fuzz/QueryFuzzTest.java, core/src/test/java/io/questdb/test/griffin/fuzz/QueryRunner.java, core/src/test/java/io/questdb/test/griffin/fuzz/EarlyExitGroupByDetectionTest.java
Fuzz execution now uses named worker pools, cursor self-checks, early-termination-aware fault swallowing, per-fault accounting, and early-exit group-by detection, with targeted regressions for accepted overflow and pushed-limit behavior.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Possibly related PRs

  • questdb/questdb#6130: Both changes touch window null-handling infrastructure around DATE/TIMESTAMP result behavior.
  • questdb/questdb#6504: Both changes modify IntervalBwdPartitionFrameCursor sizing and backward interval/frame boundary handling.
  • questdb/questdb#6712: Both changes affect percent_rank() window execution and related comparator/key-type handling.

Poem

🐇 I hopped through frames of date and rank,
and patched each leaky memory bank.
With faulty burrows, tests now play,
then spring right back the normal way.
Carrots for every cursor tight—
the warren hums through day and night.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch puzpuzpuz_query_fuzzer_mk3

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

puzpuzpuz and others added 7 commits June 9, 2026 14:30
SampleByInterpolateRecordCursorFactory builds its record-key map, data
map and group-by allocator in the cursor constructor. When one of those
native allocations fails (for example the RSS memory limit trips), the
constructor's error path calls close() before the later fields are
assigned. close() called recordKeyMap.close() / dataMap.close() /
allocator.close() directly, so a null field threw NullPointerException
and masked the real out-of-memory error.

Free the three fields through Misc.free(), which null-guards each one,
so the original CairoException propagates to the caller. The downstream
Misc.clear()/Misc.free() calls on the function fields are already
null-safe.

The query fuzzer's malloc fault injection surfaced this. The added test
sweeps the RSS ceiling across the cursor-construction allocation points
and asserts the out-of-memory error surfaces instead of an NPE, then
that the query recovers once the ceiling is removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PageFrameMemoryPool.nextFreeBuffers() pulls a ParquetBuffers off the free
list, removes it from that list, and only then calls reopen() on it.
reopen() allocates the per-frame page-address lists one by one. If that
allocation trips the RSS memory limit, the buffer has already been removed
from the free list but not yet recorded in the cache, so the pool can no
longer reach it on close and the page-address lists it managed to allocate
leak.

On a failed reopen(), close the buffer to free whatever it allocated and
return it to the free list, so the pool keeps ownership and frees it on
close. The happy path is unchanged.

The query fuzzer's malloc fault injection surfaced this as a small
NATIVE_DEFAULT leak when scanning a parquet partition. The added test
sweeps the RSS ceiling finely across the buffer reopen and asserts the
scan leaves no native memory behind, then that the query recovers once
the ceiling is removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AbstractPostingIndexReader.openSidecarFilesIfPresent() swallowed any error
raised while opening a covering index's sidecar info file: it logged the
failure, cleared the sidecar memories, and returned normally. The covering
plan, however, is fixed at compile time and the covered read path has no
base-table fallback, so a covered read then walked an empty sidecarMems and
threw "index out of bounds, 0 >= 0" (an out-of-bounds read once assertions
are disabled), masking the real I/O error.

Propagate the failure instead. The caller already wraps the open in
catch (Throwable) { close(); throw; }, so the reader is closed and the query
fails with the underlying error rather than reading an absent sidecar.

The query fuzzer's file fault injection surfaced this on a covered VARCHAR
column. The added test builds a posting covering index over a VARCHAR column
and sweeps a failing filesystem op across the reader's open path, asserting
the covered read never throws an AssertionError and recovers once the fault
is removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PostingGenLookup allocated its cache buffers as inline field initializers:
a DirectLongList followed by a DirectIntLongHashMap, both tagged
NATIVE_INDEX_READER. If the second allocation tripped the RSS memory limit,
the first one leaked: the half-built PostingGenLookup was never assigned to
the AbstractPostingIndexReader that was constructing it, so the reader's
close() could not reach the buffer to free it.

Move both allocations into an exception-safe constructor that frees whichever
buffer was already allocated before rethrowing, so a failed construction
leaves nothing behind.

The query fuzzer's malloc fault injection surfaced this as a small
NATIVE_INDEX_READER leak when reading through a posting index. The added test
sweeps the RSS ceiling across posting index reader construction and relies on
assertMemoryLeak to confirm the native buffers return to baseline, then that
the query recovers once the ceiling is removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
IntervalBwdPartitionFrameCursor.calculateSize() diverged from next()
and returned a wrong row count for a backward scan over a half-open
interval such as `ts < X ORDER BY ts DESC`.

When the lower interval bound was open (Long.MIN_VALUE), the method
shortcut `lo = 0` instead of letting findTimestamp() return the -1
sentinel its whole-partition check relied on. That single wrong value
caused two faults at once: it undercounted the partition by one (it
used `hi = rowCount - 1` against `lo = 0`), and, because `lo == -1`
never held, it always took the fragment branch and stopped after the
first (newest) partition, ignoring every earlier partition in range.
A two-partition table that materialized 96 rows reported a size of 47.

calculateSize() now mirrors next() exactly, the same way the forward
cursor's calculateSize() mirrors forward next(): it derives limitHi
from partitionLimit, reads the high bound with timestampAt(limitHi),
applies the +1 boundary convention for lo/hi, and detects a whole
partition via `lo == 0`. Closed and open-ended intervals were already
counted correctly and are unaffected; the forward cursor never had the
open-bound shortcut and is correct.

The bug surfaced through the query fuzzer's cursor self-consistency
check (calculateSize() vs the materialized row count). The existing
interval frame-cursor tests only ran the forward direction and only
closed-lower-bound intervals, so they missed it. The test battery now
runs every interval shape in both directions and adds open-start and
open-end cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
assertCalculateSize() only compared calculateSize() to itself across
toTop() and a cursor reopen. Those checks confirm the value is stable,
but not that it is correct: a cursor whose calculateSize() is
consistently wrong -- for instance a frame cursor that stops after the
first partition and returns the same undercount every time -- satisfied
all three comparisons.

The method now first materializes the cursor with a full hasNext()
iteration and asserts calculateSize() equals that count, which is the
property calculateSize() actually promises (how many rows the cursor
would yield). The full assertQuery battery already made an equivalent
comparison in assertCursor() behind a random skip; pulling a
deterministic, unconditional version into assertCalculateSize() removes
the dependence on that random draw and makes the dedicated check
self-sufficient.

Verified across a broad cross-section of query shapes (limit, order by,
distinct, latest by, sample by, interval filters, parquet, hash/asof
joins, union, group by) with no regressions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reverts "Anchor assertCalculateSize to the materialized row count".

The check it added was redundant: assertCursor() already compares
calculateSize() to the materialized hasNext() count on every non-empty
.returns() assertion, and runs before assertCalculateSize() on the same
factory, so the row-count anchor was already enforced. The only gaps the
revert reopens are minor (empty cursors and making the from-top check
deterministic rather than behind a random skip), and they do not justify
a second full cursor iteration on every assertQuery in the suite.

The genuine coverage that catches the backward interval-scan bug stays:
the bidirectional interval frame-cursor tests added alongside the fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@puzpuzpuz puzpuzpuz added the Bug Incorrect or unexpected behavior label Jun 9, 2026
Add a WindowClause that emits SELECT projections with window functions
over OVER (PARTITION BY ... ORDER BY ts [frame]) clauses. It spans the
ranking family (row_number, rank, dense_rank, cume_dist, percent_rank,
ntile), the windowed aggregates (avg, sum, min, max, count) and the
navigation functions (first_value, last_value, lag, lead, nth_value),
with ROWS and time-based RANGE frames. Every window orders by the
designated timestamp, which the fuzz tables generate strictly
increasing, so a total order inside each partition makes the window
values reproducible across runs and the differential oracle can compare
full result sets rather than just row counts.

As with the other clauses, every main-rnd draw is independent of the
BindContext, so the literal and bind passes build the same tree shape;
only the constant leaves rendered through the context differ.

The QueryGenerator carves the WINDOW band out of the upper half of the
SIMPLE range, leaving the SAMPLE BY and GROUP BY bands untouched, so
toggling window off restores the exact pre-window shape distribution.
The questdb.fuzz.window knob is on by default, like fault injection;
it immediately surfaces several known-open window-function defects, so
the run goes red on the seeds that hit them until those are fixed. Pass
-Dquestdb.fuzz.window=false to drop window shapes and exercise the rest.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@puzpuzpuz
puzpuzpuz force-pushed the puzpuzpuz_query_fuzzer_mk3 branch from afae2e2 to f32849a Compare June 9, 2026 14:40
@puzpuzpuz puzpuzpuz changed the title test(sql): add cursor self-consistency checks and fault injection to query fuzzer fix(sql): add cursor self-consistency checks and fault injection to query fuzzer and harden query engine Jun 9, 2026
max/min/first_value/last_value/nth_value over a DATE argument
previously fell through to the timestamp window factories. The
resulting WindowTimestampFunction reported getType() = DATE but read
and stored the value in timestamp microseconds, so the streaming read
path threw UnsupportedOperationException from WindowFunction.getDate
and the cached read path returned dates roughly 1000x in the future
(microseconds read back as milliseconds).

Add dedicated DATE window factories (max(M), min(M), first_value(M),
last_value(M), nth_value(ML)) backed by WindowDateFunction, mirroring
the existing lag/lead DATE factories. They read arg.getDate() and
store, write, and emit milliseconds, so both the streaming and cached
paths return correct DATE values. MinDate reuses MaxDate's inner
classes; the classpath scanner registers the new factories
automatically. Add DateNullFunction for empty-frame results.

Cover the change with WindowDateFunctionTest (all five functions on
the streaming and cached paths, with NULLs and partitions) and four
DATE methods in WindowFunctionTest.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@puzpuzpuz puzpuzpuz added the DO NOT MERGE These changes should not be merged to main branch label Jun 10, 2026
puzpuzpuz and others added 12 commits June 10, 2026 10:35
The DATE max/min/first_value/last_value/nth_value window factories
duplicated the entire timestamp implementation. Collapse them into thin
subclasses of the timestamp factories that only override getSignature().

The timestamp value functions now read the argument through
readArgValue(), which returns the native long for every argument type
the factories accept: getDate() for a DATE argument (milliseconds), and
getTimestamp() otherwise (timestamp ticks, including a SYMBOL, STRING or
VARCHAR parsed to a timestamp). A type-aware getDate() default on
WindowTimestampFunction converts the stored value back to DATE
milliseconds for the streaming read path.

readArgValue routes TIMESTAMP arguments, and SYMBOL/STRING/VARCHAR
arguments, through getTimestamp() exactly as before, so the change
preserves behavior for the timestamp factories while removing about
10,800 lines of duplicated code.

Add a nanosecond designated-timestamp case to WindowDateFunctionTest.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Refactor the max/min window value functions into the lag/lead pattern: a
type-agnostic abstract base per window shape plus thin DATE and TIMESTAMP
subclasses that each carry a clean per-type accessor.

New MaxMinWindowFunctionFactoryHelper holds the nine abstract bases (the
framing logic, the native-long value buffers, the comparator and name), the
seven constructor functional interfaces, the TimestampComparator and the
MAX_*_COLUMN_TYPES constants. Its single newInstance dispatcher replaces the
two near-duplicate per-shape dispatchers that previously lived in the Max and
Min timestamp factories; the caller passes a comparator, a name and the
per-type constructor references, so the same control flow serves both DATE and
TIMESTAMP arguments and both max and min.

The TIMESTAMP subclasses (MaxTimestampWindowFunctionFactory) cache the
timestamp driver once in the constructor and expose getTimestamp() as the
stored value plus a getDate() that converts ticks to DATE milliseconds. This
removes the per-row type re-derivation that the previous polymorphic getDate()
performed. The DATE subclasses (MaxDateWindowFunctionFactory) store and emit
the value in DATE milliseconds and only expose getDate(); WindowDateFunction
derives the rest, including getTimestamp() = value * 1000. Two-pass shapes
(over partition, over whole result set) materialize their result and never read
back through the accessor, so they only override getType().

MaxDate and MinDate now extend AbstractWindowFunctionFactory directly and build
the DATE subclasses, rather than inheriting the TIMESTAMP factory's
newInstance. Min reuses Max's subclasses with a LESS_THAN comparator.

The heavy WindowTimestampFunction.getDate() default is left in place: the
first_value/last_value/nth_value timestamp shapes still rely on it and have not
been split yet.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply the lag/lead pattern to the first_value window value functions, as
done for max/min in the previous commit. Each window shape now has one
type-agnostic abstract base in the new FirstValueWindowFunctionFactoryHelper
(holding all framing logic and native-long value buffers), plus a thin DATE
or TIMESTAMP subclass that adds only the per-type accessor: the TIMESTAMP
subclass caches the timestamp driver once and exposes getTimestamp() as the
stored value plus a getDate() that converts ticks to milliseconds, while the
DATE subclass exposes the stored milliseconds directly through getDate().
Two-pass shapes expose only getType().

The helper holds the 15 shape bases (the RESPECT NULLS bases plus the IGNORE
NULLS bases that extend them), the seven constructor functional interfaces,
the shared FIRST_VALUE_COLUMN_TYPES, and one consolidated newInstance
dispatcher parameterized by per-type constructor references so the same
control flow serves both DATE and TIMESTAMP arguments. The dispatcher now
frees the map and circular-buffer memory on the constructor error path via
try/catch, closing a leak the query fuzzer's fault injection surfaces.

FirstValueDateWindowFunctionFactory stops inheriting the timestamp factory's
newInstance and becomes a real factory that builds the DATE subclasses.

Verified green: WindowFunctionTest 606, WindowDateFunctionTest 22,
WindowDecimalFunctionTest 431, WindowFunctionUnitTest 40, and the PR fuzzer
seed s0=394723763672806 s1=1781015833112 with window and faults enabled.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply the same lag/lead pattern to the last_value window value functions.
Each window shape now has one type-agnostic abstract base in the new
LastValueWindowFunctionFactoryHelper (holding all framing logic and
native-long value buffers), plus a thin DATE or TIMESTAMP subclass that adds
only the per-type accessor: the TIMESTAMP subclass caches the timestamp
driver once and exposes getTimestamp() as the stored value plus a getDate()
that converts ticks to milliseconds, while the DATE subclass exposes the
stored milliseconds directly through getDate(). Shapes that write the result
column directly during a backward pass1 scan expose only getType().

The helper holds the 17 shape bases (two IGNORE NULLS bases extend their
RESPECT NULLS sibling; the rest are independent), the eight constructor
functional interfaces, the four shared column-type templates, and one
consolidated newInstance dispatcher parameterized by per-type constructor
references so the same control flow serves both DATE and TIMESTAMP arguments.
The dispatcher now frees the map and circular-buffer memory on the
constructor error path via try/catch, closing a leak the query fuzzer's
fault injection surfaces.

LastValueDateWindowFunctionFactory stops inheriting the timestamp factory's
newInstance and becomes a real factory that builds the DATE subclasses.

Verified green: WindowFunctionTest 606, WindowDateFunctionTest 22,
WindowFunctionUnitTest 40, and the PR fuzzer seed s0=394723763672806
s1=1781015833112 with window and faults enabled.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply the same lag/lead pattern to the nth_value window value functions.
Each window shape now has one type-agnostic abstract base in the new
NthValueWindowFunctionFactoryHelper (holding all framing logic and the
native-long value buffers), plus a thin DATE or TIMESTAMP subclass that adds
only the per-type accessor: the TIMESTAMP subclass caches the timestamp
driver once and exposes getTimestamp() as the stored value plus a getDate()
that converts ticks to milliseconds, while the DATE subclass exposes the
stored milliseconds directly through getDate(). Two-pass shapes expose only
getType().

The helper holds the 11 independent shape bases, the nine constructor
functional interfaces (the n index threads through each), and the single
newInstance dispatcher parameterized by per-type constructor references so
the same control flow serves both DATE and TIMESTAMP arguments. The
dispatcher keeps the existing try/catch map and circular-buffer cleanup on
the constructor error path.

NthValueDateWindowFunctionFactory stops inheriting the timestamp factory's
newInstance and becomes a real factory that builds the DATE subclasses.

Verified green: WindowFunctionTest 606, WindowDateFunctionTest 22,
WindowFunctionUnitTest 40, and the PR fuzzer seed s0=394723763672806
s1=1781015833112 with window and faults enabled.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
With max/min/first_value/last_value/nth_value now split into per-type
subclasses, every streaming TIMESTAMP window value function overrides
getDate() with its cached timestamp driver, so the type-dispatching getDate()
default on WindowTimestampFunction is no longer reached on that path. Drop it
so each timestamp window function states its own getDate() (or inherits the
throwing default when it never serves the accessor).

The only accessor-read function that relied on the default was the shared
empty-frame TimestampNullFunction, so it now provides getDate() directly,
returning its stored LONG_NULL (the NULL sentinel for both DATE and TIMESTAMP
results). Two-pass and backward-scan shapes (whole-result-set, partition,
lead/lead-over-partition) write their result straight into the window column
and are read back from the record chain, never through getDate(); their
inherited throwing default is unreachable, exactly as before this change
(the old default would itself have thrown via the throwing getTimestamp()).

The change is behavior-preserving: a streaming timestamp subclass returned
driver.toDate(value) before and after. Verified green: WindowFunctionTest 606,
WindowDateFunctionTest 22, WindowDecimalFunctionTest 431, WindowFunctionUnitTest
40, and the PR fuzzer seed s0=394723763672806 s1=1781015833112.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rank() and dense_rank() over (partition by ... order by ts) take the
streaming WindowRecordCursorFactory path when the order matches the
designated timestamp. That path remembered each partition's previous row by
copying the whole projected row into a MapValue and reading the ORDER BY
columns back from it. RankOverPartitionFunction built the copier with
RecordValueSinkFactory over every projected column, so any pass-through
column the MapValue cannot hold (UUID, STRING, VARCHAR, BINARY, LONG256,
arrays, INTERVAL) threw UnsupportedOperationException at compile time. A
plain SELECT then failed even though the rank itself was fine; projecting
only the window function avoided it.

Only the ORDER BY columns decide whether two consecutive rows are peers, so
the MapValue now holds just those (compacted into slots 0..k-1) plus the
running rank and count. The value sink reads the ORDER BY columns from the
live record at their base indices and writes them compacted; the comparator
and rank maps are built over a matching compacted metadata so they read
back only those columns.

computeNext no longer compares against the live record. It caches the
previous row's ORDER BY values via setLeft (which the comparator stores by
value for every type the MapValue can hold), overwrites the slots with the
current row, then compares the cached previous row against the MapValue.
Keeping the comparator off the live record is what lets the compacted
layout work.

Rank maps are indexed by compacted ORDER BY position while the symbol
tables live at base column indices, so init() builds each map through a
small position-to-base-index mapping. This only matters for the rare
advice-driven symbol ORDER BY; the common designated-timestamp ORDER BY has
no symbol maps. The no-partition path was already safe (RecordSinkFactory
over the ORDER BY columns plus memeq), and the cached path is unchanged.

WindowFunctionTest.testRankWithUnserializablePassThroughColumn asserts
rank/dense_rank over UUID, STRING, VARCHAR, LONG256 and DOUBLE[]
pass-through columns (ascending and descending), drains a BINARY
pass-through, and pins the streaming Window plan so the fix keeps
exercising the right path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lead() over a Decimal128 or Decimal256 argument reported ZERO_PASS,
unlike every other lead variant (Double, Long, and the narrower
decimals all inherit ONE_PASS). When a query's only window functions
were wide-decimal leads and the window ordered by the designated
timestamp, that ZERO_PASS hint routed the query through the streaming
window executor.

lead reads ahead, so it cannot run in the forward streaming pass; it
is computed in the cached executor with a backward pass1 scan. In the
streaming path the wide-decimal lead also exposed no Decimal128 or
Decimal256 read accessor, so reading the result fell through to the
default-throwing WindowFunction accessor and raised
UnsupportedOperationException. Earlier all-types tests masked this
because a narrow-decimal column in the same projection forced the
cached path.

Removing the erroneous getPassCount() override lets these two
over-partition functions inherit ONE_PASS like the rest, so the planner
keeps them on the cached path where pass1 already does the work and the
record chain reads the value back. Repro (now passing):
SELECT lead(d, 5) OVER (PARTITION BY g ORDER BY ts) FROM t with
d DECIMAL(76, 3).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Triage of open bug #1 from the window-function fuzzing: a sum() or avg()
over a high-precision DECIMAL (Decimal256-backed, precision > 38) raises
"CairoException: ... aggregation failed: an overflow occurred". The repro
is avg(d) OVER (PARTITION BY g ORDER BY ts) with d DECIMAL(76,3).

This is a genuine, data-dependent arithmetic limit, not an engine defect.
The running total of two values near the DECIMAL(76,3) maximum exceeds
Decimal256's 256-bit capacity, so the sum is genuinely unrepresentable.
The plain group-by sum/avg overflows identically on the same data, and
WindowDecimalFunctionTest already pins this overflow as the expected
behaviour. Decimal256 is the widest decimal type, so there is no wider
accumulator to promote to.

The fuzzer oracle was reporting it as a failure because it treats any
leaked CairoException as an internal error. Teach isAcceptedSkip to also
accept a CairoException whose message contains "aggregation failed",
classifying it like a NumericException overflow. The match is narrow:
only the decimal sum/avg functions emit "aggregation failed", and every
such message is an overflow.

Tests:
- WindowDecimalFunctionTest.testAvgDecimal256OverPartitionGenuineOverflow
  and testSumDecimal256OverPartitionGenuineOverflow pin that both the
  window-over-partition form and the plain group-by form overflow on
  near-max DECIMAL(76,3) data.
- QueryFuzzTest.testDecimalAggregationOverflowToleratedByOracle drives
  the repro queries through QueryRunner.run() and asserts the oracle
  skips them, while a non-overflowing query still runs to completion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In the streaming WindowRecordCursorFactory path the code generator
builds each window function's partition map lazily, in
initRecordComparator(), after the whole projection has been walked. The
generator reuses a single key-types buffer and clears and rebuilds it
for every window column's PARTITION BY, but rank()/dense_rank() over a
partition kept a live reference to that buffer instead of a copy. So
when a query had two or more streaming window functions partitioning by
differently typed keys, the rank function built its map from the last
window column's key types rather than its own.

That either crashed - when the partition sink wrote a type the wrongly
picked map could not hold, e.g. a BYTE partition writing into the
Unordered4Map selected for a SYMBOL key threw
UnsupportedOperationException - or silently mis-keyed the partitions,
e.g. a Decimal256 partition map built for a two-column key fed by a
single-column sink reported rank 1 for every row and could even diverge
across a toTop() re-read. A single rank function, or several sharing one
partition key, masked it.

RankOverPartitionFunction now snapshots the key types in its
constructor, while the buffer still describes its own PARTITION BY. The
two sibling over-partition functions that store the same reference,
cume_dist and percent_rank, get the same snapshot for consistency; they
are two-pass and always take the cached path where initRecordComparator
runs in-loop, so they were not yet exploitable, but the shared idiom is
now uniformly safe. The copy lives in a shared
AbstractWindowFunctionFactory.copyKeyTypes() helper.

Adds WindowFunctionTest regression coverage for rank/dense_rank with
mixed BYTE/SYMBOL and wide-DECIMAL partition keys on the streaming path,
plus cume_dist/percent_rank with mixed keys on the cached path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A window value or navigation function whose argument is the untyped
null literal (e.g. lead(null, 2), min(null)) resolves to a NULL output
column type. The cached window path then asks RecordSinkFactory to
build a sink over that NULL column and leaks an internal
"IllegalArgumentException: Unexpected column type: NULL" through the
query API. Whether the leak surfaced depended on the access path: the
streaming path tolerated the NULL column, while the cached path
(read-ahead lead, or min/max over a DESC ROWS frame) crashed, so the
same function failed on one frame and silently returned nulls on
another.

The code generator now rejects a window function whose output type is
NULL with a user-facing SqlException at the function position,
suggesting a concrete cast (e.g. null::double, which compiles and
computes over NULL). The guard sits in both window parse loops, so the
rejection is consistent across the streaming and cached paths. Window
functions that resolve to a concrete output type for a null literal
(lag, max, first_value, last_value, avg, sum, count) keep computing
over NULL as before; nth_value already had its own NULL guard.

This is open bug #1 surfaced by the query fuzzer's window shapes. The
clean SqlException is an accepted skip in QueryRunner.isAcceptedSkip,
so the fuzzer seeds that hit lead(null)/min(null) now run green
instead of crashing on the internal exception.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
puzpuzpuz and others added 2 commits June 17, 2026 14:07
testNullLiteralArgumentRejectedWithCleanError failed on linux-arm64 (a
clean rejection but with the wrong message: "lead is not yet implemented
for NULL" instead of "does not support an untyped NULL argument").

Root cause: a window value function over the untyped null literal, e.g.
lead(null, 2), ties across every typed variant during overload
resolution -- NULL has zero overload distance to any type. The five
lead factories (Double, Long, Decimal, Date, Timestamp) all match with
the same score, so the winner is decided by Java's stable sort
preserving the classpath scan order, which differs by filesystem and
architecture. Each winner behaves differently:

  - Timestamp/Date variants are type-adaptive and yield a NULL output
    type, which the SqlCodeGenerator guard rejected cleanly (x64);
  - the Decimal variant throws "lead is not yet implemented for NULL"
    from its own factory (arm64);
  - the Double/Long variants yield a concrete output type and silently
    accept the query, returning NULLs.

So the same SQL gave three different results across platforms, and the
SqlCodeGenerator guard only covered the first.

FunctionParser now rejects an untyped NULL value argument to a
polymorphic window function before any factory runs, so every platform
gives the same clean SqlException at the same position. The check is
scoped to window functions with more than one overload, so ntile -
whose single argument is a bucket count with its own NULL validation -
keeps its specific "bucket count cannot be NULL" error. The two now
redundant SqlCodeGenerator guards are removed; the resolution-time
check covers both the streaming and cached paths at once.

The test's nth_value assertion is updated: nth_value over an untyped
NULL is now rejected with the same uniform message as every other
window function rather than its old factory-specific one.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 12

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
core/src/test/java/io/questdb/test/cairo/fuzz/FailureFileFacade.java (1)

519-557: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Reset all scoping fields when changing arming modes.

setToFailAfter(...) and setToFailAfterOnCurrentThread(...) leave mode-3 state (queryWorkerNamePrefix/armingThread) behind. After a mode switch, checkForFailure() can keep filtering calls by stale query-thread rules, so the intended fault never fires.

Suggested fix
 public void setToFailAfter(int ioFailureCallCount) {
         // Default (unscoped): any thread's op counts and fires. Arm only when not
         // already armed so a pending fault is not reset by a concurrent caller.
         armedThread = null;
+        armingThread = null;
+        queryWorkerNamePrefix = null;
+        contaminatingThreadName = null;
         int osCalls;
         while (true) {
             osCalls = osCallsCount.get();
             if (osCalls > 0 || osCallsCount.compareAndSet(osCalls, ioFailureCallCount)) {
                 return;
@@
 public void setToFailAfterOnCurrentThread(int ioFailureCallCount) {
+        armingThread = null;
+        queryWorkerNamePrefix = null;
+        contaminatingThreadName = null;
         // Only the arming thread counts down and fires (see checkForFailure), so
         // the same thread both writes and reads osCallsCount and no CAS guard is
         // needed against concurrent decrements.
         osCallsCount.set(ioFailureCallCount);
         armedThread = Thread.currentThread();
 }

Also applies to: 627-642

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@core/src/test/java/io/questdb/test/cairo/fuzz/FailureFileFacade.java` around
lines 519 - 557, The arming mode methods do not properly reset scoping state
when switching between modes, leaving stale filtering rules from previous modes.
In setToFailAfter(), in addition to setting armedThread to null, also reset
queryWorkerNamePrefix to null and armingThread to null. In
setToFailAfterOnCurrentThread(), before setting the new thread-scoped state,
also reset queryWorkerNamePrefix to null and armingThread to null. These resets
ensure that checkForFailure() uses only the current mode's filtering rules and
not stale state from a previous mode.
🧹 Nitpick comments (7)
core/src/test/java/io/questdb/test/griffin/engine/table/PageFrameRecordCursorImplFactoryTest.java (1)

471-517: ⚡ Quick win

Use underscore separators in new 5+ digit numeric literals.

The new tests introduce several 5+ digit literals (10000, 1000000L, etc.) without underscore grouping. Please update these to match repository convention.

Suggested diff
-        final int numOfRows = 10000;
-        final long increment = 1000000L * 60 * 60;
-        final long intervalHi = 1000000L * 60 * 60 * 24 * 3;
+        final int numOfRows = 10_000;
+        final long increment = 1_000_000L * 60 * 60;
+        final long intervalHi = 1_000_000L * 60 * 60 * 24 * 3;

As per coding guidelines, “Use underscore to separate thousands in numbers with 5 digits or more in test SQL and code” and “Always use them in numbers of 5 digits or more in both test code and implementation.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@core/src/test/java/io/questdb/test/griffin/engine/table/PageFrameRecordCursorImplFactoryTest.java`
around lines 471 - 517, Add underscore separators to all numeric literals with 5
or more digits in the test methods
testFactory_IntervalPartitionFrameCursorFactory_openStart,
testFactory_IntervalPartitionFrameCursorFactory_openStart_skip,
testFactory_IntervalPartitionFrameCursorFactory_openEnd, and
testFactory_IntervalPartitionFrameCursorFactory_openEnd_skip. Specifically,
update 10000 to 10_000 and 1000000L to 1_000_000L throughout these test methods
to match the repository's coding convention for numeric literal formatting.

Source: Coding guidelines

core/src/test/java/io/questdb/test/griffin/engine/groupby/vect/GroupByVectorizedRostiAccountingTest.java (1)

123-127: ⚡ Quick win

Use underscore separators for 5+ digit numeric literals in test SQL.

Please format the timestamp literals with separators to match the repo rule and improve readability.

Proposed fix
-        execute("INSERT INTO tab SELECT (x * 1000000L)::timestamp, x FROM long_sequence(8)", context);
+        execute("INSERT INTO tab SELECT (x * 1_000_000L)::timestamp, x FROM long_sequence(8)", context);
...
-        execute("INSERT INTO tab SELECT (86400000000L + x * 1000000L)::timestamp, x, x::int " +
+        execute("INSERT INTO tab SELECT (86_400_000_000L + x * 1_000_000L)::timestamp, x, x::int " +
                 "FROM long_sequence(" + liveKeys + ")", context);

As per coding guidelines, “Use underscore to separate thousands in numbers with 5 digits or more in test SQL and code” and “Always use them in numbers of 5 digits or more in both test code and implementation.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@core/src/test/java/io/questdb/test/griffin/engine/groupby/vect/GroupByVectorizedRostiAccountingTest.java`
around lines 123 - 127, The numeric literals in the SQL execute statements
within the test method are missing underscore separators for readability. Locate
the two execute method calls that contain the timestamp literals: the first one
with `1000000L` and the second one with `86400000000L`. Add underscore
separators to both numeric literals following the convention of separating every
three digits from the right (for example, `1000000L` becomes `1_000_000L` and
`86400000000L` becomes `86_400_000_000L`) to comply with the coding guidelines
for numeric literals with 5 or more digits.

Source: Coding guidelines

core/src/test/java/io/questdb/test/griffin/fuzz/PredicateGenerator.java (1)

68-76: ⚡ Quick win

Rename boolean flag to is... / has... form.

injectFaultFn should follow the repo boolean naming rule (for example, isFaultFnInjected or hasFaultFnInjection).

As per coding guidelines: "When choosing a name for a boolean variable, field or method, always use the is... or has... prefix, as appropriate."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@core/src/test/java/io/questdb/test/griffin/fuzz/PredicateGenerator.java`
around lines 68 - 76, Rename the boolean parameter `injectFaultFn` in the
`appendWhere` method to follow the repository's boolean naming convention.
Change it to use the `is...` or `has...` prefix form (such as
`isFaultFnInjected` or `hasFaultFnInjection`). Update all references to this
parameter throughout the method body to use the new name consistently.

Source: Coding guidelines

core/src/test/java/io/questdb/test/griffin/fuzz/QueryGenerator.java (1)

70-70: ⚡ Quick win

Use is... boolean names in the public generator signature.

Please rename injectFaultFn / windowEnabled to is.../has... forms to keep API naming consistent with project conventions.

As per coding guidelines: "When choosing a name for a boolean variable, field or method, always use the is... or has... prefix, as appropriate."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@core/src/test/java/io/questdb/test/griffin/fuzz/QueryGenerator.java` at line
70, The boolean parameters in the generate method signature do not follow the
project's naming convention for boolean variables. Rename injectFaultFn to
isInjectFaultFn and windowEnabled to isWindowEnabled (or use hasWindowEnabled if
more semantically appropriate) in the method signature and update all references
to these parameters throughout the method body to maintain consistency with the
coding guidelines that require is... or has... prefixes for boolean names.

Source: Coding guidelines

core/src/test/java/io/questdb/test/griffin/fuzz/FuzzConfig.java (1)

39-44: ⚡ Quick win

Keep new static constants alphabetically ordered in the static-public member group.

FAULTS_PROP is placed before FAULT_PARALLEL_PROP and FAULT_PCT_PROP, which breaks the Java member-ordering rule used in this repo.

Suggested reorder
-    public static final String FAULTS_PROP = "questdb.fuzz.faults";
     public static final String FAULT_PARALLEL_PROP = "questdb.fuzz.fault.parallel";
     public static final String FAULT_PCT_PROP = "questdb.fuzz.fault.pct";
+    public static final String FAULTS_PROP = "questdb.fuzz.faults";

As per coding guidelines: "Group Java class members by kind (static vs. instance) and visibility, and sort them alphabetically. Insert new methods and fields in the correct alphabetical position among existing members of the same kind."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@core/src/test/java/io/questdb/test/griffin/fuzz/FuzzConfig.java` around lines
39 - 44, The static constants in the FuzzConfig class are not in alphabetical
order. Reorder the constants FAULTS_PROP, FAULT_PARALLEL_PROP, and
FAULT_PCT_PROP so they appear in correct alphabetical sequence.
FAULT_PARALLEL_PROP and FAULT_PCT_PROP (which start with underscore after
FAULT_) should come before FAULTS_PROP (which ends with S after FAULT_), then
follow with QUERIES_PROP, VERIFY_CURSOR_PROP, and WINDOW_PROP.

Source: Coding guidelines

core/src/test/java/io/questdb/test/griffin/GroupByTest.java (1)

1420-1421: ⚡ Quick win

Use underscores in 5+ digit numeric literals for test readability and guideline compliance.

Please format 859371 and 237288 as 859_371 and 237_288 in comments and SQL literals.

Suggested patch
-        // 859371 + (cnt * -237288) out of the inner GROUP BY when its base
+        // 859_371 + (cnt * -237_288) out of the inner GROUP BY when its base
@@
-            assertQuery("SELECT t0.cnt AS e0, (859371 + (t0.cnt * -237288)) AS e1, count() AS a0\n"
+            assertQuery("SELECT t0.cnt AS e0, (859_371 + (t0.cnt * -237_288)) AS e1, count() AS a0\n"
@@
-            assertQuery("SELECT t0.cnt AS e0, (859371 + (t0.cnt * -237288)) AS e1, count() AS a0\n"
-                    + body + "GROUP BY t0.cnt, (859371 + (t0.cnt * -237288)) ORDER BY 1 ASC, e1")
+            assertQuery("SELECT t0.cnt AS e0, (859_371 + (t0.cnt * -237_288)) AS e1, count() AS a0\n"
+                    + body + "GROUP BY t0.cnt, (859_371 + (t0.cnt * -237_288)) ORDER BY 1 ASC, e1")

As per coding guidelines, "Use underscore to separate thousands in numbers with 5 digits or more in test SQL and code."

Also applies to: 1438-1439, 1444-1445

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@core/src/test/java/io/questdb/test/griffin/GroupByTest.java` around lines
1420 - 1421, Update numeric literals in the GroupByTest.java file to comply with
coding guidelines by adding underscores to separate thousands in all numbers
with 5 or more digits. Specifically, format 859371 as 859_371 and 237288 as
237_288 in the comments and any SQL literals at the specified locations.
Additionally, review and update any other 5+ digit numeric literals at lines
1438-1439 and 1444-1445 with the same underscore formatting pattern.

Source: Coding guidelines

core/src/test/java/io/questdb/test/griffin/engine/window/WindowFunctionTest.java (1)

2448-2453: ⚡ Quick win

Use text blocks for multi-row INSERT SQL in tests.

These long multi-row INSERTs are still assembled via concatenated strings. Please switch them to Java multiline string literals ("""...""") for readability and guideline consistency.

As per coding guidelines, "Use multiline strings for longer SQL statements (multiple INSERT rows, complex queries), as well as to assert multiline query results in tests."

Also applies to: 7313-7318, 8702-8707, 11062-11067, 17351-17357, 17647-17653, 18045-18054

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@core/src/test/java/io/questdb/test/griffin/engine/window/WindowFunctionTest.java`
around lines 2448 - 2453, Replace the multi-row INSERT statement in the
execute() method call that uses concatenated strings (with + operators) with a
Java text block using triple quotes ("""). Convert the entire SQL statement from
the string concatenation format into a single readable multiline text block
enclosed in triple quotes, ensuring the SQL syntax remains unchanged. This
change should also be applied to all other similar multi-row INSERT statements
mentioned in the additional locations (lines 7313-7318, 8702-8707, 11062-11067,
17351-17357, 17647-17653, 18045-18054) in the same test file.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@core/src/main/java/io/questdb/griffin/engine/functions/window/MaxMinWindowFunctionFactoryHelper.java`:
- Around line 959-964: The close() method in the
MaxMinWindowFunctionFactoryHelper class closes the memory field but fails to
close the dequeMemory field, which is allocated for bounded-row frames and
causes native buffer leaks. Add a call to close dequeMemory in the close()
method after the super.close() call to ensure both memory buffers are properly
released.
- Around line 1095-1114: The toPlan() method in the
MaxMinWindowFunctionFactoryHelper class incorrectly calculates the upper bound
for ROWS frames ending in PRECEDING using the formula bufferSize - frameSize on
lines 1113 and 1570 (and in the similar block at lines 1555-1573). This
calculation underreports the actual preceding row offset. Replace the expression
bufferSize - frameSize with just frameSize to correctly represent the number of
rows preceding the current row at which the frame upper bound occurs. This
applies to all locations where frameIncludesCurrentValue is false and we are
rendering the "AND <n> PRECEDING" portion of the window frame specification.

In `@core/src/main/java/io/questdb/griffin/FunctionParser.java`:
- Around line 1236-1241: The error position in the SqlException thrown for
untyped NULL arguments is using node.position which marks the function name
instead of the actual problematic argument. Replace node.position with
argPositions.getQuick(0) in the SqlException constructor call to point the error
diagnostic at the first argument position where the NULL argument actually
appears, providing more precise error location for users.

In `@core/src/main/java/io/questdb/jit/CompiledFilterIRSerializer.java`:
- Around line 1032-1085: The boolean variable names in the
CompiledFilterIRSerializer class do not follow the repository's naming
convention which requires boolean variables to use is... or has... prefixes.
Rename the following boolean parameters and variables to comply with this
convention: the underLong parameter in markI64Widen method, the thisLong
variable in markI64Widen method, the parentLong parameter in markI64WidenOperand
method, the widenToI64 variable around line 1212, and the keepI4 variable around
lines 1770-1780. Choose appropriate is... or has... prefixes based on the
semantic meaning of each boolean (for example, underLong could become
isUnderLong, thisLong could become isThisLong, parentLong could become
isParentLong, etc.). Update all references to these renamed variables throughout
the affected methods to maintain consistency.

In
`@core/src/test/java/io/questdb/test/cairo/covering/CoveringIndexSidecarFaultTest.java`:
- Around line 77-87: The test loop iterating through failAfter values (1 to 80)
and calling drain(query, scratch) does not verify that the fault injection
actually triggers at least once. Add a tracking variable (boolean flag or
counter) that records when an exception is caught in the catch block within the
loop, and after the loop completes, add an assertion to verify that at least one
fault was triggered during the iterations. This ensures the test validates that
the covering-sidecar open path is actually being exercised and prevents the test
from passing when no faults occur.

In
`@core/src/test/java/io/questdb/test/griffin/CompiledFilterRegressionTest.java`:
- Around line 1559-1573: The boolean parameter expectJit in the
assertJitMatchesJava method does not follow the repository naming convention for
boolean variables. Rename the parameter expectJit to isJitExpected (or similar
is... prefix format) and update all references to this parameter throughout the
method body, specifically in the Assert.assertEquals call where expectJit is
currently being used as an argument.
- Around line 1073-1087: Add underscore separators to all numeric literals with
five or more digits in this test method. Specifically, update the literals in
the execute INSERT statement (1000000 and 1800000000L) and in all the
assertJitMatchesJava SQL queries (776782 and 2000000000) to use underscore
grouping for thousands to match the coding guidelines for this file.

In
`@core/src/test/java/io/questdb/test/griffin/engine/groupby/SampleByInterpolateOomTest.java`:
- Around line 79-84: The test currently only creates and closes a
RecordCursorFactory without actually opening the cursor, which means it does not
exercise the cursor-construction cleanup path that the test is meant to verify.
After creating the RecordCursorFactory with select(query), you need to open the
cursor from the factory (typically by calling getRecordCursor() on the factory)
and then iterate through or access the cursor data to actually trigger the
construction and cleanup logic. Make sure the opened cursor is properly closed
or managed within a try-with-resources block to ensure the regression is
properly caught.

In
`@core/src/test/java/io/questdb/test/griffin/engine/table/parquet/ParquetScanOomTest.java`:
- Line 59: Fix two coding guideline violations in the ParquetScanOomTest method:
First, apply underscore separators to the numeric literal 1000000L on line 59 by
converting it to 1_000_000L to improve readability for numbers with more than 6
digits. Second, rename the boolean variable sawOom (appearing on lines 73, 82,
and 87) to use the has prefix convention, changing it to hasOom throughout the
method to follow the naming guidelines for boolean variables.

In
`@core/src/test/java/io/questdb/test/griffin/fuzz/EarlyExitGroupByDetectionTest.java`:
- Around line 49-50: In the execute method call within
EarlyExitGroupByDetectionTest, the numeric literal `1000000L` has 6 digits and
should use underscore separators for improved readability and consistency with
project coding guidelines. Replace `1000000L` with the underscore-separated
version following the thousands separator pattern to make the number more
readable.

In `@core/src/test/java/io/questdb/test/griffin/fuzz/QueryRunner.java`:
- Around line 418-421: Remove the broad SQL-text heuristic from the
discardedEarly condition in the QueryRunner class. The
Chars.containsLowerCase(sql, "limit") check incorrectly treats any query
containing the word LIMIT as early-terminating, which can mask swallowed faults
on aggregate-with-limit queries. Replace the parallel condition so it only gates
discardedEarly on the planner-derived signals: outcome.hasPushedLimit and
outcome.hasEarlyExitGroupBy. Keep the parallel check but remove the
Chars.containsLowerCase call entirely, allowing only the actual planner outputs
to determine whether a query is early-terminating.

In `@core/src/test/java/io/questdb/test/jit/CompiledFilterIRSerializerTest.java`:
- Around line 415-416: The numeric literals in the test SQL strings lack
underscore separators for readability. In the serialize method call on line 415,
replace the numeric literal 100000 with underscore separators (5+ digit numbers
require this formatting). Also apply the same formatting fix to the numeric
literal 9000000000 referenced in the comment for lines 423-424. Update both the
serialize() call strings and their corresponding assertIR() assertion strings to
use properly formatted numeric literals with underscores for thousands
separation.

---

Outside diff comments:
In `@core/src/test/java/io/questdb/test/cairo/fuzz/FailureFileFacade.java`:
- Around line 519-557: The arming mode methods do not properly reset scoping
state when switching between modes, leaving stale filtering rules from previous
modes. In setToFailAfter(), in addition to setting armedThread to null, also
reset queryWorkerNamePrefix to null and armingThread to null. In
setToFailAfterOnCurrentThread(), before setting the new thread-scoped state,
also reset queryWorkerNamePrefix to null and armingThread to null. These resets
ensure that checkForFailure() uses only the current mode's filtering rules and
not stale state from a previous mode.

---

Nitpick comments:
In
`@core/src/test/java/io/questdb/test/griffin/engine/groupby/vect/GroupByVectorizedRostiAccountingTest.java`:
- Around line 123-127: The numeric literals in the SQL execute statements within
the test method are missing underscore separators for readability. Locate the
two execute method calls that contain the timestamp literals: the first one with
`1000000L` and the second one with `86400000000L`. Add underscore separators to
both numeric literals following the convention of separating every three digits
from the right (for example, `1000000L` becomes `1_000_000L` and `86400000000L`
becomes `86_400_000_000L`) to comply with the coding guidelines for numeric
literals with 5 or more digits.

In
`@core/src/test/java/io/questdb/test/griffin/engine/table/PageFrameRecordCursorImplFactoryTest.java`:
- Around line 471-517: Add underscore separators to all numeric literals with 5
or more digits in the test methods
testFactory_IntervalPartitionFrameCursorFactory_openStart,
testFactory_IntervalPartitionFrameCursorFactory_openStart_skip,
testFactory_IntervalPartitionFrameCursorFactory_openEnd, and
testFactory_IntervalPartitionFrameCursorFactory_openEnd_skip. Specifically,
update 10000 to 10_000 and 1000000L to 1_000_000L throughout these test methods
to match the repository's coding convention for numeric literal formatting.

In
`@core/src/test/java/io/questdb/test/griffin/engine/window/WindowFunctionTest.java`:
- Around line 2448-2453: Replace the multi-row INSERT statement in the execute()
method call that uses concatenated strings (with + operators) with a Java text
block using triple quotes ("""). Convert the entire SQL statement from the
string concatenation format into a single readable multiline text block enclosed
in triple quotes, ensuring the SQL syntax remains unchanged. This change should
also be applied to all other similar multi-row INSERT statements mentioned in
the additional locations (lines 7313-7318, 8702-8707, 11062-11067, 17351-17357,
17647-17653, 18045-18054) in the same test file.

In `@core/src/test/java/io/questdb/test/griffin/fuzz/FuzzConfig.java`:
- Around line 39-44: The static constants in the FuzzConfig class are not in
alphabetical order. Reorder the constants FAULTS_PROP, FAULT_PARALLEL_PROP, and
FAULT_PCT_PROP so they appear in correct alphabetical sequence.
FAULT_PARALLEL_PROP and FAULT_PCT_PROP (which start with underscore after
FAULT_) should come before FAULTS_PROP (which ends with S after FAULT_), then
follow with QUERIES_PROP, VERIFY_CURSOR_PROP, and WINDOW_PROP.

In `@core/src/test/java/io/questdb/test/griffin/fuzz/PredicateGenerator.java`:
- Around line 68-76: Rename the boolean parameter `injectFaultFn` in the
`appendWhere` method to follow the repository's boolean naming convention.
Change it to use the `is...` or `has...` prefix form (such as
`isFaultFnInjected` or `hasFaultFnInjection`). Update all references to this
parameter throughout the method body to use the new name consistently.

In `@core/src/test/java/io/questdb/test/griffin/fuzz/QueryGenerator.java`:
- Line 70: The boolean parameters in the generate method signature do not follow
the project's naming convention for boolean variables. Rename injectFaultFn to
isInjectFaultFn and windowEnabled to isWindowEnabled (or use hasWindowEnabled if
more semantically appropriate) in the method signature and update all references
to these parameters throughout the method body to maintain consistency with the
coding guidelines that require is... or has... prefixes for boolean names.

In `@core/src/test/java/io/questdb/test/griffin/GroupByTest.java`:
- Around line 1420-1421: Update numeric literals in the GroupByTest.java file to
comply with coding guidelines by adding underscores to separate thousands in all
numbers with 5 or more digits. Specifically, format 859371 as 859_371 and 237288
as 237_288 in the comments and any SQL literals at the specified locations.
Additionally, review and update any other 5+ digit numeric literals at lines
1438-1439 and 1444-1445 with the same underscore formatting pattern.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: d8d4b7bf-0ab4-4b69-b490-ebcbfa3704b1

📥 Commits

Reviewing files that changed from the base of the PR and between 97e8a6a and 42cbf5e.

📒 Files selected for processing (72)
  • core/src/main/java/io/questdb/cairo/IntervalBwdPartitionFrameCursor.java
  • core/src/main/java/io/questdb/cairo/RecordChain.java
  • core/src/main/java/io/questdb/cairo/idx/AbstractPostingIndexReader.java
  • core/src/main/java/io/questdb/cairo/idx/PostingGenLookup.java
  • core/src/main/java/io/questdb/cairo/idx/PostingIndexBwdReader.java
  • core/src/main/java/io/questdb/cairo/idx/PostingIndexFwdReader.java
  • core/src/main/java/io/questdb/cairo/sql/PageFrameAddressCache.java
  • core/src/main/java/io/questdb/griffin/FunctionParser.java
  • core/src/main/java/io/questdb/griffin/SqlCodeGenerator.java
  • core/src/main/java/io/questdb/griffin/SqlOptimiser.java
  • core/src/main/java/io/questdb/griffin/engine/functions/test/TestFaultFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/AbstractWindowFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/AvgDecimalRescaleWindowFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/AvgDecimalWindowFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/CumeDistFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/FirstValueDateWindowFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/FirstValueTimestampWindowFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/FirstValueWindowFunctionFactoryHelper.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/LastValueDateWindowFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/LastValueTimestampWindowFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/LastValueWindowFunctionFactoryHelper.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/LeadDecimalFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/MaxDateWindowFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/MaxMinWindowFunctionFactoryHelper.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/MaxTimestampWindowFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/MinDateWindowFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/MinTimestampWindowFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/NthValueDateWindowFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/NthValueTimestampWindowFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/NthValueWindowFunctionFactoryHelper.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/PercentRankFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/RankFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/functions/window/SumDecimalWindowFunctionFactory.java
  • core/src/main/java/io/questdb/griffin/engine/groupby/GroupByNotKeyedRecordCursorFactory.java
  • core/src/main/java/io/questdb/griffin/engine/groupby/SampleByInterpolateRecordCursorFactory.java
  • core/src/main/java/io/questdb/griffin/engine/groupby/vect/GroupByRecordCursorFactory.java
  • core/src/main/java/io/questdb/griffin/engine/join/AsOfJoinFastRecordCursorFactory.java
  • core/src/main/java/io/questdb/griffin/engine/join/FilteredAsOfJoinFastRecordCursorFactory.java
  • core/src/main/java/io/questdb/griffin/engine/orderby/LimitedSizeLongTreeChain.java
  • core/src/main/java/io/questdb/griffin/engine/table/AsyncHorizonJoinAtom.java
  • core/src/main/java/io/questdb/griffin/engine/table/AsyncMultiHorizonJoinAtom.java
  • core/src/main/java/io/questdb/griffin/engine/table/HeapRowCursorFactory.java
  • core/src/main/java/io/questdb/jit/CompiledFilterIRSerializer.java
  • core/src/main/java/io/questdb/std/Decimal256.java
  • core/src/test/java/io/questdb/test/cairo/PostingIndexReaderOomTest.java
  • core/src/test/java/io/questdb/test/cairo/covering/CoveringIndexSidecarFaultTest.java
  • core/src/test/java/io/questdb/test/cairo/fuzz/FailureFileFacade.java
  • core/src/test/java/io/questdb/test/griffin/CompiledFilterRegressionTest.java
  • core/src/test/java/io/questdb/test/griffin/GroupByTest.java
  • core/src/test/java/io/questdb/test/griffin/engine/groupby/SampleByInterpolateOomTest.java
  • core/src/test/java/io/questdb/test/griffin/engine/groupby/vect/GroupByVectorizedOomTest.java
  • core/src/test/java/io/questdb/test/griffin/engine/groupby/vect/GroupByVectorizedRostiAccountingTest.java
  • core/src/test/java/io/questdb/test/griffin/engine/join/AsOfJoinFastOomTest.java
  • core/src/test/java/io/questdb/test/griffin/engine/table/PageFrameRecordCursorImplFactoryTest.java
  • core/src/test/java/io/questdb/test/griffin/engine/table/parquet/ParquetScanOomTest.java
  • core/src/test/java/io/questdb/test/griffin/engine/window/WindowDateFunctionTest.java
  • core/src/test/java/io/questdb/test/griffin/engine/window/WindowDecimalFunctionTest.java
  • core/src/test/java/io/questdb/test/griffin/engine/window/WindowFunctionTest.java
  • core/src/test/java/io/questdb/test/griffin/fuzz/EarlyExitGroupByDetectionTest.java
  • core/src/test/java/io/questdb/test/griffin/fuzz/FaultType.java
  • core/src/test/java/io/questdb/test/griffin/fuzz/FuzzConfig.java
  • core/src/test/java/io/questdb/test/griffin/fuzz/PredicateGenerator.java
  • core/src/test/java/io/questdb/test/griffin/fuzz/QueryFuzzTest.java
  • core/src/test/java/io/questdb/test/griffin/fuzz/QueryGenerator.java
  • core/src/test/java/io/questdb/test/griffin/fuzz/QueryRunner.java
  • core/src/test/java/io/questdb/test/griffin/fuzz/clauses/GroupByClause.java
  • core/src/test/java/io/questdb/test/griffin/fuzz/clauses/SampleByClause.java
  • core/src/test/java/io/questdb/test/griffin/fuzz/clauses/SimpleClause.java
  • core/src/test/java/io/questdb/test/griffin/fuzz/clauses/TemporalJoinClause.java
  • core/src/test/java/io/questdb/test/griffin/fuzz/clauses/WindowClause.java
  • core/src/test/java/io/questdb/test/jit/CompiledFilterIRSerializerTest.java
  • core/src/test/java/io/questdb/test/std/Decimal256Test.java
💤 Files with no reviewable changes (1)
  • core/src/main/java/io/questdb/griffin/engine/functions/window/LeadDecimalFunctionFactory.java

Comment thread core/src/main/java/io/questdb/griffin/FunctionParser.java
Comment thread core/src/main/java/io/questdb/jit/CompiledFilterIRSerializer.java
Comment thread core/src/test/java/io/questdb/test/griffin/fuzz/QueryRunner.java Outdated
Apply four cleanups raised in the PR #7217 automated review.

Hoist the window value-function argument type check out of the per-row
path. readArgValue() re-derived ColumnType.tagOf(arg.getType()) for
every row at ~60 call sites, though the DATE vs TIMESTAMP choice is
invariant per instance. BaseWindowFunction now caches it as argIsDate
in its constructor and exposes an instance readArgValue(record).

Replace Decimal256.uncheckedSubtract's negate-then-add with a direct
subtract-with-borrow that mirrors uncheckedAdd's carry chain, halving
the limb arithmetic. Decimal256Test and the window sliding-frame
eviction tests cover it.

Drop the stale FuzzConfig comment claiming FILE / MALLOC faults stay
serial: all three fault types now run under parallel SQL execution
(commit "Run FILE and MALLOC fuzzer faults in parallel").

Fold the per-test DDL into .ddl(CREATE, INSERT) across the 22
WindowDateFunctionTest cases and drop the redundant assertMemoryLeak
wrapper plus .noLeakCheck(); the assertQuery builder leak-checks by
default.

The review's one Moderate code finding (drop the swallow-oracle SQL
text "limit" arm) was rejected: an offset-limit filtered query
(LIMIT lo,hi) terminates early through an outer LimitRecordCursorFactory
with no pushed-limit plan marker, so dropping the arm would make the
fuzzer report a false-positive swallowed fault.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@puzpuzpuz puzpuzpuz self-assigned this Jun 17, 2026
@puzpuzpuz puzpuzpuz removed the DO NOT MERGE These changes should not be merged to main branch label Jun 17, 2026
puzpuzpuz and others added 2 commits June 17, 2026 18:01
openSidecarFilesIfPresent() now returns early when metadata is null.
A rowid-only reader (built via the null-metadata convenience
constructor) cannot map writer indices to dense columns, so it serves
no covered reads and has no sidecar to set up.

Previously the method swallowed the resulting NullPointerException from
denseIndexFromWriter() and degraded to coverCount = 0. After this branch
made the catch propagate (correct for production, where a covered read
has no base-table fallback), that latent NPE began failing the constructor
for rowid-only readers. The early return restores the coverCount = 0 state
for those readers without re-swallowing genuine sidecar-open failures.

In production metadata is never null: IndexFactory and TableReader always
pass real metadata. The null-metadata path is test-only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@puzpuzpuz

Copy link
Copy Markdown
Contributor Author

Code review (level 3, independent re-pass)

Full mission-critical pass: change-surface map, 9 scoped review agents plus the two mandatory adversarial agents (fresh-context, adversarial-performance), and per-finding source verification against this branch. Scoped against upstream/master (73 files, +16,359/-10,064). The branch is in strong shape. No Critical or correctness-affecting bugs. ~25 leak/crash/wrong-result hypotheses were raised across the agents and all but the test-infra ones verified false — and several changes fix latent bugs already on master.

Critical

None.

Moderate

M1 — Fuzzer swallow-oracle over-tolerates faults on aggregate-with-LIMIT queriescore/src/test/java/io/questdb/test/griffin/fuzz/QueryRunner.java:419 (in-diff)

boolean discardedEarly = parallel
        && (Chars.containsLowerCase(sql, "limit")   // <-- crude SQL-text match
        || outcome.hasPushedLimit
        || outcome.hasEarlyExitGroupBy);

Independently re-confirmed. The containsLowerCase(sql,"limit") arm tolerates a swallowed fault for any query whose text contains LIMIT. But a LIMIT over a row-count-changing aggregate (SELECT count()/sum()/avg() ... LIMIT N) does not push a scan limit — the async filter scans every frame, so a fault fired on a worker must surface. GroupByClause emits keyless agg(...) ... LIMIT N ~25% of the time. Running the plans: for count/sum/avg/count_distinct ... LIMIT 5, hasPushedLimit=false and hasEarlyExitGroupBy=false, but the text arm flips discardedEarly=true — so a genuine swallow in exactly the bug class this PR exists to catch would be recorded as a tolerated skip. The robust signals already classify every legitimate early-termination case (SELECT * ... LIMIT 5 -> hasPushedLimit=true; the implicit-LIMIT 1 rewrite is covered too), and testImplicitTimestampLimitTolerated already pins count() ... WHERE test_fault() as not pushed.

Fix: drop the Chars.containsLowerCase(sql,"limit") disjunct (and the now-unused Chars import); keep hasPushedLimit || hasEarlyExitGroupBy. Removes the false negative, adds no false positives.

Minor

  • toPlan() mis-renders preceding-ending ROWS boundsMaxMinWindowFunctionFactoryHelper.java:1113, 1570 (in-diff but pre-existing & codebase-wide). bufferSize - frameSize underreports the upper bound for ROWS BETWEEN N PRECEDING AND M PRECEDING (5 PRECEDING AND 3 PRECEDING -> "...and 2 preceding"). The PR's own plan tests bless the wrong text (WindowDecimalFunctionTest.java:2814, :2648). The same idiom is copy-pasted across ~16 window factories (old MaxTimestamp had it too). EXPLAIN-text only; cursor results are correct. A real fix should be a separate codebase-wide change with all plan tests updated — out of scope here, but flagging so it's tracked.
  • Error position points at the function tokenFunctionParser.java:1240. The untyped-NULL rejection throws SqlException.$(node.position, ...); per QuestDB convention it should point at the offending arg: argPositions.getQuick(0). Cosmetic. (Matches CodeRabbit.)
  • Boolean namingCompiledFilterIRSerializer.java:1054/1060/1079/1212/1778. New locals/params underLong, thisLong, parentLong, widenToI64, keepI4 don't use the is.../has... prefix. (Matches CodeRabbit.)
  • JIT widen marking is O(N^2) at compile timeCompiledFilterIRSerializer.java (markI64Widen/arithExprType/isI64WidenLeaf). Subtree types recomputed per ancestor; isI64WidenLeaf linear-scans. Gated on hasFloatInPredicate, runs once per compile on tiny ASTs — immaterial. A single memoized post-order pass would be O(N). Note only.
  • Test coverage gaps (all in-diff):
    • CoveringIndexSidecarFaultTest can pass vacuously — unlike the RSS-OOM tests it has no sawFault-style assertion that the swept fault actually hit the sidecar-open path. Add one.
    • The copyKeyTypes fix's cume_dist/percent_rank path has no true regression testtestCumeDistMixedPartitionKeyTypes is defensive-only by its own comment (the rank/dense_rank portion is regression-grade).
    • Untyped-NULL rejection only asserts lead/min/nth_value; the body claims uniform rejection across lag/max/sum/avg/first_value/last_value/count — not individually exercised. (ntile single-overload negative control is guarded separately.)
    • WindowDateFunctionTest is thin on frame modes: first_value ROWS, last_value/nth_value RANGE, min over ROWS/RANGE/bounded, bounded N PRECEDING, current-row-only, and whole-result-set OVER () (only max has it) are DATE-untested.
    • The negative-constant i64-widen path is covered behaviorally (CompiledFilterRegressionTest) but not pinned as a locked IR-shape assertIR case.
    • Trivial: underscore separators omitted in a few constants (e.g. EarlyExitGroupByDetectionTest.java:56 1000000L).
  • Defensive (optional, not a leak): adding close(){ super.close(); memory.close(); } to the window partition-ROWS bases and dequeMemory.close() to MaxMinOverPartitionRowsFrameBase would restore self-containment/symmetry with the RANGE classes. Verified not an active leak (see Downgraded).

PR metadata

  • Title: fix(sql) undersells the change and runs ~100 chars. The largest user-facing change is new DATE-typed window function support (a new capability). Consider feat(sql): support DATE-typed window functions and harden query engine, or at minimum drop the fuzzer clause and lead with the DATE support (the fuzzer tooling is test-only).
  • Labels: [Bug, SQL] -> add New feature (DATE windows), Enhancement, Performance (Decimal256 uncheckedSubtract), Core (index/page-frame/OOM hardening). Not WAL.
  • Fixes #NNN: absent — add if it closes a tracked issue. Body tone is level-headed; the DATE refactor is documented (bug-fix Query language #6).

Downgraded (verified false positives / resolved)

  • Window memory ring-buffer leak (drafted HIGH) — FALSE POSITIVE. The premise (refactor "lost" a close() free) is wrong: the old partition-ROWS class also freed memory only in reset(); the old close(){...memory.close();} belonged to the RANGE-frame class (which allocates eagerly). The factory _close() closes the cursor (-> resetFunctions() -> reset() -> frees memory) at WindowRecordCursorFactory.java:135 / CachedWindowRecordCursorFactory.java:295 before freeing the function list — on every discard path (incl. exception-during-iteration and caller-abandons-cursor). No reachable leak.
  • readArgValue per-row regression — REFUTED. argIsDate is a final field set in the BaseWindowFunction constructor (:42); zero per-row getType()/tagOf calls in any helper. No per-row cost.
  • assertMemoryLeak + lone-assertQuery anti-pattern — REFUTED. 0 occurrences; WindowDateFunctionTest uses bare assertQuery(...).ddl(...).returns(...); every assertMemoryLeak block is genuinely multi-statement.
  • dequeMemory leak (CodeRabbit) — NOT a leak (freed in reset(), which precedes close()); faithful copy of old MaxTimestamp. Defensive nit only.
  • TestFault double-throw race — only one thread observes getAndDecrement()==0; at most one throw.
  • vect GROUP BY drain hangdoneLatch.countDown() is in aggregate()'s finally; loop always terminates. (The drain-rethrow runs in buildRosti's finally and skips the eager freeObjListAndKeepObjects/rosti-minimize, but _close() frees both and the skip is pre-existing — informational, not a finding.)
  • HeapRowCursor double-freefreeObjListAndClear empties the shared list; RowCursor.close() is contractually idempotent. Reclaims a real mid-build-OOM leak.
  • FunctionParser args double-freeargs is the caller's transient; the caller's try/catch doesn't wrap createFunction; every existing throw already freeObjList(args). Safe.
  • AsOfJoin sink reopen, posting-reader of()-throw, PageFrameAddressCache, LimitedSizeLongTreeChain, RecordChain, Horizon atoms, SampleByInterpolate, AbstractPostingIndexReader swallow->throw, Decimal256 borrow chain — all traced and cleared (no double-free/UAF/leak; the swallow->throw's sole caller wraps catch{close();throw}).

Verified correct (highlights)

The production correctness work checks out, and several items fix latent master bugs:

  • IntervalBwd.calculateSize rewrite — a faithful branch-for-branch mirror of next(); fixes the old empty-partition stale-partitionLimit and unbounded-low undercount.
  • JIT float-suppressed i64 widening — node-identity threading (incl. negative constants), marking lifecycle, arithExprType promotion, and forceScalarMode precedence all correct; SX_I64 is genuinely SIMD-unsupported so forcing scalar is necessary.
  • SqlCodeGenerator.chainTimestampIndex — fixes RANGE-framed cached-window functions reading the wrong chain column (wrong results); both passes compute it correctly.
  • Decimal256.uncheckedSubtract — borrow chain is the exact inverse of uncheckedAdd's carry chain (5M-case BigInteger differential, 0 failures); evicts only previously-added in-range values.
  • RankFunctionFactory.copyKeyTypes — fixes a real multi-window-column PARTITION-BY-key-types aliasing bug (transient shared ArrayColumnTypes buffer).
  • LeadDecimal getPassCount removal — restores ONE_PASS so pass1() actually runs; fixes a latent streaming-path crash.
  • DATE window factories — registered via classpath scan, routed by arg type, share the Long.MIN_VALUE NULL sentinel; cast(max(ts) AS date) works (latent UnsupportedOperationException on master fixed).
  • All ~10 resource-cleanup fixes — null-safe, idempotent, no double-free/UAF; verified on the error paths.

Out-of-scope follow-up (pre-existing, not this PR)

Rank/CumeDist reset() frees map without nulling it — harmless under the current lifecycle but worth a separate look.

Summary

  • Verdict: approve. One Moderate fix worth making (M1 oracle valve, ~2 lines); the rest are coverage/metadata/style. No blocking issues.
  • ~25 draft findings verified, ~22 dropped as false positives (incl. the drafted HIGH leak). All surviving findings are in-diff — the cross-context pass (window factory registration/routing, the swallow->throw reader's sole caller, the shared HeapRowCursor list, base-ctor polymorphic close, AsyncHorizon close ownership, vect doneLatch per-cursor) found no out-of-diff bugs; zero out-of-diff is genuine here because the changes are self-contained refactors plus localized error-path hardening.
  • Net positive beyond the stated scope: the refactor fixes latent master bugs (chainTimestampIndex wrong results, IntervalBwd undercount, LeadDecimal crash, DATE-window UnsupportedOperation, JIT float-suppressed widening ~1,810 wrong-result cases, rank copyKeyTypes aliasing).

🤖 Generated with Claude Code

puzpuzpuz and others added 4 commits June 18, 2026 12:54
The CI "List of required changes" step runs the IntelliJ formatter over
the checkout and fails when "git diff --exit-code" finds deviations. The
gate was red, which skipped every downstream Rust and test step.

Bring continuation-line indentation into line with the formatter in the
files it flagged:

- SqlException.parserErr(): indent the chained .put(...) calls
- ConcurrentHashMap / ConcurrentIntHashMap / ConcurrentLongHashMap:
  indent the chained .newInstance(...) call in toArray()
- WhereClauseParserTest.replaceTimestampSuffix(): de-indent the chained
  .replaceAll(...) calls

No behavior change; formatting only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first_value/last_value DATE and TIMESTAMP window factories delegate
into FirstValueWindowFunctionFactoryHelper and
LastValueWindowFunctionFactoryHelper, which hold one base class per frame
mode (whole result set, current row, bounded/unbounded ROWS, bounded
RANGE, partitioned variants, and the IGNORE NULLS counterparts). The
existing tests only reached the whole-set, current-row and unbounded
paths, leaving the bounded ROWS/RANGE and IGNORE NULLS base classes
unexercised. The TIMESTAMP factories had almost no coverage at all.

Extend WindowDateFunctionTest from 22 to 45 cases, adding bounded ROWS
and RANGE frames (partitioned and not, RESPECT and IGNORE NULLS), the
CURRENT ROW frame, and whole-set/whole-partition IGNORE NULLS. Add
WindowTimestampFunctionTest as the TIMESTAMP mirror; the logical results
match the DATE cases and only the rendered precision differs.

Each expected result is computed from window-frame semantics on small,
hand-traceable datasets (5-6 rows). Measured against the helper and
factory sources, this raises covered changed-lines on these files by
about 1,055 (first/last value helpers from ~11% to ~60%, the date and
timestamp factories from the single digits to 55-90%).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review item M1 on PR #7217. The fault-injection swallow
oracle relaxed its "fault must surface" check for any parallel query
whose SQL text contained "limit". That over-tolerated a swallowed
fault for a LIMIT over a row-count-changing aggregate
(count()/sum()/avg()/GROUP BY ... LIMIT N): such a query pushes no
scan limit, the async filter still scans every frame, and the
aggregate drains all of them, so a fired fault must surface. A real
swallow in exactly that bug class would have been recorded as a
tolerated skip.

Simply dropping the SQL-text arm (as the review suggested) trades
the false negative for false positives. A streaming consumer under a
LIMIT -- a window function or a bare projection -- stops pulling once
the limit is met, so under the eager parallel reduce a fault on a
frame past the cutoff is legitimately discarded, the same early
termination a pushed LIMIT gives. But a window/projection between the
scan and the LIMIT stops the optimiser rendering the "limit: N" scan
marker, so planHasPushedLimit does not fire. Running the fuzzer with
the text arm removed surfaced two such window-over-async-filter
queries as spurious swallow failures.

Keep the text-LIMIT tolerance but gate it with a new
factoryHasBlockingAggregation signal that walks the consumption spine
for a count(*), keyed or non-keyed GROUP BY (serial or async). Those
drain every frame regardless of a downstream LIMIT, so a swallow
there is always a real defect; a streaming consumer carries no such
operator and stays tolerated. The check errs toward not blocking: an
aggregation factory it fails to name leaves the original (over-
tolerant) behaviour, a missed swallow rather than a false failure.

Add EarlyExitGroupByDetectionTest.testBlockingAggregationDetection to
pin the factory classification, and extend the pushed-limit pins in
QueryFuzzTest with the aggregate-with-LIMIT shapes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@puzpuzpuz

Copy link
Copy Markdown
Contributor Author

/azp run macwin

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

bluestreak01
bluestreak01 previously approved these changes Jun 20, 2026

@bluestreak01 bluestreak01 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after a level-3 mission-critical review (full change-surface map + parallel correctness/resource/test/adversarial/performance passes + per-finding source verification).

All 17 core engine fixes traced to source and confirmed correct:

  • #5 IntervalBwd calculateSize() is now a faithful mirror of next() (no extra lookups, cursor state untouched).
  • #12 cached-window RANGE timestamp index correctly scoped — streaming path (configureWindowContext @9519) left unchanged, no regression.
  • #13 Decimal256.uncheckedSubtract is an exact limb-order mirror of uncheckedAdd (carry↔borrow, two-step borrow cannot double-count).
  • #14 null-safe close()s verified complete (root via super.close(), value heap via reopen(), cursor is inline-final/never-null).
  • #15 JIT narrow-int-under-long-with-float widening correctly mirrors C++ convert(); shouldWiden() && !hasFloat() prevents double-widening.
  • #7/#10 rank streaming compacted-ORDER-BY map sound under the assert-guarded cache-by-value invariant; key-types snapshot fixes the reused-buffer aliasing.
  • #11 untyped-NULL rejection follows the existing freeObjList-before-throw pattern (no double-free).
  • Resource fixes #1/#3/#4/#16/#19/#21/#22/#24 — all cleanup paths idempotent and double-free-free; runWhatsLeft cannot deadlock (latch counted down in finally even on throw).
  • #6/#8 DATE window functions (ms vs ticks scale) correct; ZERO_PASS removal aligns wide-decimal lead with siblings.

Tests: 0 assertion-API violations (0 .returnsOnce, 0 banned assertSql), OOM sweeps net-leak-check the whole RSS sweep and assert the trip fired, window tests cover NULL/DATE-scale/mixed-keys and pin streaming-vs-cached plans.

No Critical or production-correctness defects. Non-blocking follow-ups:

  • Moderate (test-oracle only): the re-pass ImplicitCastException/NumericException tolerance in QueryRunner.checkToTop/checkCalculateSize keys on exception class alone — consider gating it on the existing hasPushedLimit flag so the relaxation matches its 'parallel-filter LIMIT non-determinism' justification and doesn't blunt the new self-consistency checks.
  • Minor: thousands separators in new SQL literals (2000000000, 859371, -237288); optional memoization of the compile-time arithExprType/markI64Widen walk.

LGTM.

Brings in 10 upstream commits, including PR #7184 (per-query memory
limits), and advances the java-questdb-client submodule to a02732c3d3.

Conflict resolutions:

- RecordChain.java: kept both the MemoryTracker import (from #7184) and
  the Misc import (from this branch); both types are used.

- AsOfJoinFastRecordCursorFactory / FilteredAsOfJoinFastRecordCursorFactory:
  both branches independently fixed the same bug - a sink heap allocation
  that trips the RSS limit during reopen() must free already-open sinks
  exactly once. Took upstream's canonical fix (reopen the sinks before
  super.of() adopts the cursors, letting the normal close() path free them)
  and dropped this branch's now-redundant post-of() try/catch, which would
  otherwise have reopened the already-open sinks a second time.

- First/Last/Max/Nth Value Timestamp window factories: this branch had
  refactored their implementations into shared helper classes, while #7184
  added per-query memory tracking to the old monolithic factories. Kept the
  refactored thin factories and ported #7184's tracking into the four shared
  helpers (FirstValue/LastValue/Nth/MaxMin): each range-frame base now binds
  its value ring buffer (and MaxMin's deque buffer) to the query's
  MemoryTracker via setMemoryTracker(), and defers the ring-buffer allocation
  from the constructor to reopen() so it is charged under the bound tracker.
  Mirrors #7184 exactly; the per-partition map binding already comes from the
  merged BasePartitionedWindowFunction. min() inherits the treatment through
  the shared MaxMin helper. ROWS-frame ring buffers remain global-only, the
  same documented coverage limitation as upstream.

Validation: core main + test sources compile; WindowMemoryTrackerTest (21),
WindowFunctionTest (633), AsOfJoinFastOomTest (2) and
RecordChainMemoryTrackerTest (12) all pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@bluestreak01

Copy link
Copy Markdown
Member

/azp run macwin

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@mtopolnik

Copy link
Copy Markdown
Contributor

[PR Coverage check]

😍 pass : 3029 / 4134 (73.27%)

file detail

path covered line new line coverage
🔵 io/questdb/griffin/engine/functions/window/NthValueDateWindowFunctionFactory.java 11 35 31.43%
🔵 io/questdb/cairo/idx/PostingIndexBwdReader.java 4 10 40.00%
🔵 io/questdb/cairo/idx/PostingIndexFwdReader.java 4 10 40.00%
🔵 io/questdb/griffin/engine/functions/window/LastValueWindowFunctionFactoryHelper.java 544 972 55.97%
🔵 io/questdb/griffin/engine/functions/window/LastValueTimestampWindowFunctionFactory.java 29 51 56.86%
🔵 io/questdb/griffin/engine/functions/window/FirstValueWindowFunctionFactoryHelper.java 633 936 67.63%
🔵 io/questdb/griffin/engine/functions/window/MaxDateWindowFunctionFactory.java 20 29 68.97%
🔵 io/questdb/griffin/engine/functions/window/NthValueTimestampWindowFunctionFactory.java 23 32 71.88%
🔵 io/questdb/griffin/engine/functions/window/FirstValueTimestampWindowFunctionFactory.java 40 55 72.73%
🔵 io/questdb/griffin/engine/functions/window/MaxTimestampWindowFunctionFactory.java 19 26 73.08%
🔵 io/questdb/griffin/engine/functions/window/MaxMinWindowFunctionFactoryHelper.java 616 820 75.12%
🔵 io/questdb/griffin/engine/functions/window/LastValueDateWindowFunctionFactory.java 40 52 76.92%
🔵 io/questdb/griffin/engine/groupby/vect/GroupByRecordCursorFactory.java 11 14 78.57%
🔵 io/questdb/griffin/engine/functions/window/AbstractWindowFunctionFactory.java 4 5 80.00%
🔵 io/questdb/griffin/engine/functions/window/RankFunctionFactory.java 30 37 81.08%
🔵 io/questdb/cairo/IntervalBwdPartitionFrameCursor.java 32 36 88.89%
🔵 io/questdb/jit/CompiledFilterIRSerializer.java 84 91 92.31%
🔵 io/questdb/griffin/engine/functions/window/NthValueWindowFunctionFactoryHelper.java 718 755 95.10%
🔵 io/questdb/griffin/engine/functions/window/FirstValueDateWindowFunctionFactory.java 48 49 97.96%
🔵 io/questdb/griffin/engine/table/AsyncMultiHorizonJoinAtom.java 1 1 100.00%
🔵 io/questdb/griffin/engine/functions/window/MinTimestampWindowFunctionFactory.java 3 3 100.00%
🔵 io/questdb/griffin/engine/functions/window/MinDateWindowFunctionFactory.java 4 4 100.00%
🔵 io/questdb/std/Decimal256.java 15 15 100.00%
🔵 io/questdb/cairo/RecordChain.java 1 1 100.00%
🔵 io/questdb/griffin/engine/table/AsyncHorizonJoinAtom.java 1 1 100.00%
🔵 io/questdb/griffin/engine/functions/window/BaseWindowFunction.java 2 2 100.00%
🔵 io/questdb/griffin/SqlCodeGenerator.java 4 4 100.00%
🔵 io/questdb/griffin/SqlOptimiser.java 8 8 100.00%
🔵 io/questdb/griffin/engine/functions/window/AvgDecimalRescaleWindowFunctionFactory.java 4 4 100.00%
🔵 io/questdb/griffin/engine/functions/window/PercentRankFunctionFactory.java 1 1 100.00%
🔵 io/questdb/griffin/engine/table/HeapRowCursorFactory.java 1 1 100.00%
🔵 io/questdb/cairo/idx/AbstractPostingIndexReader.java 3 3 100.00%
🔵 io/questdb/cairo/idx/PostingGenLookup.java 13 13 100.00%
🔵 io/questdb/griffin/engine/groupby/SampleByInterpolateRecordCursorFactory.java 3 3 100.00%
🔵 io/questdb/griffin/engine/orderby/LimitedSizeLongTreeChain.java 5 5 100.00%
🔵 io/questdb/griffin/engine/functions/window/CumeDistFunctionFactory.java 1 1 100.00%
🔵 io/questdb/griffin/FunctionParser.java 10 10 100.00%
🔵 io/questdb/griffin/engine/functions/window/SumDecimalWindowFunctionFactory.java 4 4 100.00%
🔵 io/questdb/griffin/engine/functions/test/TestFaultFunctionFactory.java 22 22 100.00%
🔵 io/questdb/griffin/engine/functions/window/AvgDecimalWindowFunctionFactory.java 4 4 100.00%
🔵 io/questdb/cairo/sql/PageFrameAddressCache.java 8 8 100.00%
🔵 io/questdb/griffin/engine/groupby/GroupByNotKeyedRecordCursorFactory.java 1 1 100.00%

@bluestreak01
bluestreak01 merged commit 536760c into master Jun 20, 2026
53 checks passed
@bluestreak01
bluestreak01 deleted the puzpuzpuz_query_fuzzer_mk3 branch June 20, 2026 19:44
puzpuzpuz added a commit that referenced this pull request Jun 22, 2026
Brings in 11 master commits, including #7184 (per-query memory limits /
MemoryTracker on window functions) and #7217 (query-engine hardening that
refactored the TIMESTAMP window functions - first_value/last_value/max/min/
nth_value - into shared helper bases and added new DATE window functions).

Conflict resolution:

- Mechanical (MemoryTag, WindowFunction, SqlExecutionContext[+Impl], the
  Double/Long/Decimal window factories): kept this branch's live-view
  snapshot/restore work and layered master's setMemoryTracker overrides into
  their alphabetical slot.

- RankFunctionFactory: adopted master's compacted-ORDER-BY MapValue layout
  (which also fixes a latent compile crash on var-size ORDER BY columns) and
  grafted the live-view snapshot/tombstone/eviction machinery on top.
  computeNext keeps master's setLeft(mapValue)+compare(mapValue) flow plus this
  branch's count==0 "uninitialized" reset signal for the ANCHOR path.

- The five timestamp window-function factories now use master's thin
  helper-delegating versions; master's new DATE factories and the four helper
  classes merge in cleanly.

Known pending (tracked as a follow-up on this branch): snapshot support has not
yet been re-ported onto the four window-function helpers, so timestamp and date
window functions are currently rejected at live-view CREATE (safe, not wrong
output). core main sources compile; this commit is a restore point before the
re-port lands.
puzpuzpuz added a commit that referenced this pull request Jun 22, 2026
The origin/master merge (#7184 lazy per-partition maps + per-query
MemoryTracker, #7217 window-fn helper refactor) silently broke the entire
window-function live-view refresh and restore surface, not just the
timestamp/date CREATE-reject the handoff described.

Fixes:
- WindowRecordCursorFactory.ofIncremental self-bootstraps when !isOpen.
  The LV refresh only calls getIncrementalCursor(); with the lazy-map
  isOpen=false it tripped "incremental cursor requires a prior bootstrap"
  and produced empty output for every window-function live view.
- BasePartitionedWindowFunction.onSnapshotRestoreBegin reopens the lazy
  map before clearing. On a fresh restart the restore ran before any
  cursor reopened the map, so createValue() crashed the JVM (SIGSEGV).
- New WindowRecordCursorFactory.openForLiveViewRestore binds the tracker,
  reopens the maps and marks the cursor open before restore writes state,
  so the first post-restore refresh preserves it (fixes row_number()
  off-by-one after restart restore).

Re-port: max/min over TIMESTAMP and DATE work in live views again.
Migrated the three partitioned ZERO_PASS bases of the shared MaxMin
helper to the snapshot contract; added an UnboundedPartitionConstructor
so the unbounded shape no longer shares the whole-partition (TWO_PASS)
constructor. The unbounded shape re-anchors via the null sentinel, so the
non-LV layout is unchanged. Min reuses Max's subclasses.

Also fixes a deque-buffer leak in the MaxMin ROWS close() (the LV close
path runs close() without reset()), and removes a duplicate
testReplaceRangeAddsPartitionsAboveLastThenRebuilds in
WalWriterReplaceRangeTest (the branch's finding-2 test was upstreamed as
master's #7289; the merge kept both copies, breaking test compilation).

Validated: full LiveViewSmokeTest 307 run / 0 failures / 4 errors (the 4
are the still-un-migrated NthValue timestamp tests that reject at CREATE);
LiveViewTest/Checkpoint/InMemoryTier and WindowTimestamp/DateFunctionTest
all green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
puzpuzpuz added a commit that referenced this pull request Jun 22, 2026
Re-add the live-view snapshot/restore contract to the shared
NthValueWindowFunctionFactoryHelper bases so nth_value() over TIMESTAMP
and DATE arguments works in live views again after the origin/master
#7217 refactor moved these functions onto shared helper bases. This
clears the four red testNthValueTimestampOver*SnapshotRoundTrip tests;
LiveViewSmokeTest is now fully green.

Migrate the four partitioned ZERO_PASS bases to the snapshot contract:

- NthValueOverPartitionRangeFrameBase (RANGE, tombstone slot 5)
- NthValueOverPartitionRowsFrameBase (bounded ROWS, tombstone slot 3)
- NthValueOverPartitionRowsFrameUnboundedBase (ROWS UNBOUNDED PRECEDING
  AND K PRECEDING, tombstone slot 2, state [count, lockedValue])
- NthValueOverUnboundedPartitionFrameBase (anchored UNBOUNDED PRECEDING
  AND CURRENT ROW, tombstone slot 2, state [value, count]; the only one
  that participates in frontier compaction via newCompactionScratch and
  re-anchors through a count==0 sentinel in computeNext)

Each base gains liveView/keyColumnTypes/mapValueTypes fields, sets
tombstoneValueIndex in its constructor, writes the tombstone slot in
computeNext, and adds getPartitionMap, getSnapshotKey{ColumnTypes,
StartIndex}, resetPartition, restorePartitionState, snapshotFormat/
MinSupportedVersion, snapshotPartitionState and supportsSnapshot. The
ring-buffer bases also override onSnapshotRestoreBegin to truncate the
native arena. Add six new static map layouts and thread
partitionByKeyTypes + liveView (+ configuration for the unbounded
shape) through the four functional-interface constructors, the
newInstance call sites, and the thin TIMESTAMP and DATE subclasses.

There is no decimal nth_value to crib from, so the reference is the
pre-merge per-factory diff plus the in-tree, passing
NthValueDoubleWindowFunctionFactory twin, mapped onto the helper's
readArgValue/Numbers.LONG_NULL long domain (TIMESTAMP and DATE both
store an 8-byte long, so serialization is identical for the pair). The
non-LV path stays pristine: tombstoneValueIndex is -1 and the tombstone
branches are inert.

Validation: LiveViewSmokeTest 307/0/0, LiveViewTest 25,
LiveViewCheckpointTest 12, LiveViewInMemoryTierTest 9,
WindowTimestampFunctionTest 43, WindowDateFunctionTest 45,
WindowFunctionTest 633 (non-LV nth_value paths), LiveViewFuzzTest 5,
all green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
puzpuzpuz added a commit that referenced this pull request Jun 22, 2026
Re-add the live-view snapshot/restore contract to the shared
FirstValueWindowFunctionFactoryHelper bases so first_value() over TIMESTAMP
and DATE arguments works in live views again after the origin/master #7217
refactor moved these functions onto shared helper bases.

Migrate the three partitioned ZERO_PASS RESPECT bases and their IGNORE-NULLS
(FirstNotNull) subclasses:

- FirstValueOverPartitionRangeFrameBase (RANGE, tombstone slot 5) and
  FirstNotNullValueOverPartitionRangeFrameBase
- FirstValueOverPartitionRowsFrameBase (bounded ROWS, tombstone slot 3) and
  FirstNotNullValueOverPartitionRowsFrameBase (tombstone slot 4)
- FirstValueOverUnboundedPartitionRowsFrameBase (anchored UNBOUNDED PRECEDING
  AND CURRENT ROW, tombstone slot 2, [value, initialized] state, the only one
  with newCompactionScratch) and FirstNotNullValueOverUnboundedPartitionRows-
  FrameBase

Each base gains liveView/keyColumnTypes/mapValueTypes fields, sets
tombstoneValueIndex in its constructor, writes the tombstone slot in
computeNext, and adds getPartitionMap, getSnapshotKey{ColumnTypes,StartIndex},
resetPartition, restorePartitionState, snapshotFormat/MinSupportedVersion,
snapshotPartitionState and supportsSnapshot. The ring-buffer bases override
onSnapshotRestoreBegin to truncate the native arena. Add eight new static map
layouts and thread partitionByKeyTypes + liveView (+ configuration for the
unbounded shape) through the helper's PartitionRange/PartitionRows functional
interfaces, a new UnboundedPartitionConstructor (so the unbounded shape no
longer shares PartitionConstructor with the un-migrated whole-partition
TWO_PASS shape), the newInstance call sites, and the thin TIMESTAMP and DATE
subclasses in both factories.

The in-tree FirstValueDecimalWindowFunctionFactory decimal64 bases were the
exact structural twin (decimal64 is a raw long), translated arg.getDecimal64
to readArgValue and Decimals.DECIMAL64_NULL to Numbers.LONG_NULL. Two
serialization decisions follow that twin rather than the pre-merge per-factory
timestamp diff:

- The IGNORE NULLS RANGE base serializes the full physical ring and preserves
  firstIdx. Its unbounded-lo fast path uses firstIdx as a 0/1 capture flag with
  the value pinned at physical index 0, so the pre-merge timestamp factory's
  logical-from-firstIdx serialization was a latent bug for that shape. The
  RESPECT RANGE base keeps the logical serialization (rebases firstIdx to 0).
- The IGNORE NULLS unbounded path keeps the initialized byte at 0 until the
  first non-null and recaptures after each anchor reset, fixing the recapture
  gap the double/long FirstNotNull reference still has.

The non-LV map layouts stay pristine; computeNext branches on liveView
(decimal-family convention) instead of mutating the base layout. The RESPECT
ROWS base gained a close() override (super.close() + memory.close()): the
helper base lacked one and the live-view close path runs close() without
reset(), so the ring memory leaked.

Tests: eight new LiveViewSmokeTest cases - six byte-exact snapshot round-trips
covering all three shapes for both RESPECT and IGNORE NULLS, plus two
anchor-reset correctness tests (RESPECT and IGNORE NULLS, the latter with a
leading null per day). LiveViewSmokeTest 315/0/0; LiveViewTest,
LiveViewCheckpointTest, LiveViewInMemoryTierTest, WindowTimestampFunctionTest,
WindowDateFunctionTest, WindowFunctionTest and LiveViewFuzzTest all green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
puzpuzpuz added a commit that referenced this pull request Jun 22, 2026
Re-add the live-view snapshot/restore contract to the shared
LastValueWindowFunctionFactoryHelper bases so last_value() over TIMESTAMP
and DATE arguments works in live views again after the origin/master #7217
refactor moved these functions onto shared helper bases. This is the last
of the four #7217 helper families to be re-ported.

Unlike first_value, last_value has only two migratable partitioned
ZERO_PASS shapes (RANGE and ROWS), each in a RESPECT and an IGNORE-NULLS
flavour:

- LastValueOverPartitionRangeFrameBase (RESPECT, tombstone slot 4) and
  LastNotNullValueOverPartitionRangeFrameBase (IGNORE) which extends the
  RESPECT base and inherits its whole snapshot surface, only threading
  liveView to super and writing the tombstone in its own computeNext.
  last_value RANGE has no unbounded-lo capture-flag fast path, so the
  IGNORE variant serializes logically via the inherited parent method -
  no firstIdx-flag trap, unlike first_value's IGNORE RANGE.
- LastValueOverPartitionRowsFrameBase (RESPECT) and
  LastNotNullValueOverPartitionRowsFrameBase (IGNORE), both tombstone
  slot 3.

last_value has no migratable partitioned unbounded-preceding shape: the
UNBOUNDED PRECEDING AND CURRENT ROW frame's last element is the current
row, so RESPECT routes to the lightweight (un-migrated) IncludeCurrent
shape and the IGNORE unbounded base was never migrated pre-merge either.
An ANCHORed last_value therefore routes to IncludeCurrent, so these frame
bases are not reachable via an ANCHOR - matching the pre-merge behaviour.

Each base gains liveView/keyColumnTypes/mapValueTypes fields, sets
tombstoneValueIndex in its constructor, writes the tombstone slot in
computeNext, and adds getPartitionMap, getSnapshotKey{ColumnTypes,
StartIndex}, resetPartition, restorePartitionState, snapshotFormat/
MinSupportedVersion, snapshotPartitionState, supportsSnapshot and
onSnapshotRestoreBegin (truncate the native ring). Add four static _LV
map layouts (existing slots plus a trailing tombstone BYTE) and thread
partitionByKeyTypes + liveView through the helper's PartitionRange and
PartitionRows functional interfaces, the newInstance call sites, and the
thin TIMESTAMP and DATE subclasses in both factories.

Both ROWS bases gained a close() override (super.close() + memory.close()):
the helper base lacked one and the live-view close path runs close()
without reset(), so the ring memory leaked - the same leak the MaxMin and
FirstValue ROWS bases hit. The non-LV map layouts stay pristine and the
new branches gate on liveView (the decimal-family convention). The leaner
helper style drops the pre-merge factory's frontier-compaction freeList
(no newCompactionScratch).

The reference was the pre-merge per-factory diff
(git diff 3b6d3ed..9b1011e -- LastValueTimestampWindowFunctionFactory.java:
three migrated classes - RANGE RESPECT, ROWS RESPECT, ROWS IGNORE - with
RANGE IGNORE inheriting) plus the in-tree FirstValue helper twin for the
helper mechanics.

Tests: six new LiveViewSmokeTest cases via the new
assertLastValueTimestampDateFrameRoundTrip helper, covering both shapes for
both RESPECT and IGNORE NULLS over TIMESTAMP and DATE. RESPECT tests use a
frame ending strictly before the current row to reach the partition bases;
IGNORE tests use ... AND CURRENT ROW and skip a NULL. Each asserts the
windowed value is correct (proving CREATE-accept) then a byte-exact
write/toTop/restore/write round-trip of the partition state.

LiveViewSmokeTest 321/0/0; LiveViewTest, LiveViewCheckpointTest,
LiveViewInMemoryTierTest, WindowTimestampFunctionTest,
WindowDateFunctionTest, WindowFunctionTest and LiveViewFuzzTest all green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bug Incorrect or unexpected behavior SQL Issues or changes relating to SQL execution

Projects

None yet

3 participants