fix(sql): add cursor self-consistency checks and fault injection to query fuzzer and harden query engine - #7217
Conversation
QueryRunner now re-verifies every materialized cursor against four invariants that hold for any RecordCursor regardless of access path: - re-iterating after toTop() reproduces the first pass (same row multiset for a deterministic query, same row count otherwise); - toTop() leaves preComputedStateSize() unchanged; - size(), when known, equals the materialized row count; - calculateSize() from the top counts that same number of rows and leaves hasNext() false. These mirror the calculateSize()/preComputedStateSize() cross-checks the assertQuery battery runs. Because no benign access-path choice can make a cursor diverge from itself, a violation always fails the query with no skip valves, unlike the JIT/storage/bind diff axes. checkToTop and checkCalculateSize run on every pass (primary, shadow, JIT-on/off, bind) via a CursorCheckException that short-circuits the per-axis reconciliation, so each distinct factory shape gets exercised. A new questdb.fuzz.verify.cursor flag (default true) gates the checks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The query fuzzer can now inject a fault into a query run and apply a crash-and-recover oracle in place of the differential one (a faulting query is not comparable). A per-query coin flip, drawn from the seeded rnd so replay reproduces it, selects a fault kind: - FILE: one filesystem op fails once via FailureFileFacade, installed as the engine's files facade for the run; - MALLOC: a native allocation fails once via the RSS memory limit, the same out-of-memory path production hits. FaultType also reserves FUNCTION for a follow-up. For each injected query QueryRunner.runFault arms the fault at a random trigger point, runs the query once, and then asserts: a fired fault is not swallowed; native memory and open file descriptors return to their pre-fault baseline after the cursor closes, proving the factory freed its resources on the error path; and the same query runs cleanly once the fault is removed. Fault queries run with parallel execution disabled. A fault thrown mid-parallel-execution can leave a pooled page-frame reduce task holding an un-released buffer, which then corrupts a later query that reuses the task; serial execution exercises the cleanup path without that shared-state hazard. Two new knobs control the feature: questdb.fuzz.faults (default true) and questdb.fuzz.fault.pct (default 15). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the fuzzer's fault-injection set with a FUNCTION fault. A new dev-mode test function, test_fault(), returns true until armed and then throws once on its Nth per-row evaluation; QueryGenerator weaves it into a generated query's WHERE clause (via the shared PredicateGenerator helper) so the throw happens mid-cursor, and QueryRunner.runFault arms it like the FILE and MALLOC faults. The emission draws the same rnd ops whether or not it fires, so non-fault generation stays byte-identical and the bind-variant determinism invariant holds; function-fault queries skip the posting path so test_fault() always lands in an emitting clause. QueryFuzzTest enables dev mode so the function is active. The run summary now reports how many injected faults actually fired (vs armed but not reached). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughThis PR hardens several engine failure paths, rewrites backward interval size calculation, adds shared and DATE-specific window-function implementations, updates Decimal256 and compiled-filter behavior, and expands fuzz, fault-injection, OOM, and regression test coverage. ChangesEngine runtime correctness and cleanup
Window execution and SQL correctness
Fuzz fault injection and query oracle updates
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Poem
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
SampleByInterpolateRecordCursorFactory builds its record-key map, data map and group-by allocator in the cursor constructor. When one of those native allocations fails (for example the RSS memory limit trips), the constructor's error path calls close() before the later fields are assigned. close() called recordKeyMap.close() / dataMap.close() / allocator.close() directly, so a null field threw NullPointerException and masked the real out-of-memory error. Free the three fields through Misc.free(), which null-guards each one, so the original CairoException propagates to the caller. The downstream Misc.clear()/Misc.free() calls on the function fields are already null-safe. The query fuzzer's malloc fault injection surfaced this. The added test sweeps the RSS ceiling across the cursor-construction allocation points and asserts the out-of-memory error surfaces instead of an NPE, then that the query recovers once the ceiling is removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PageFrameMemoryPool.nextFreeBuffers() pulls a ParquetBuffers off the free list, removes it from that list, and only then calls reopen() on it. reopen() allocates the per-frame page-address lists one by one. If that allocation trips the RSS memory limit, the buffer has already been removed from the free list but not yet recorded in the cache, so the pool can no longer reach it on close and the page-address lists it managed to allocate leak. On a failed reopen(), close the buffer to free whatever it allocated and return it to the free list, so the pool keeps ownership and frees it on close. The happy path is unchanged. The query fuzzer's malloc fault injection surfaced this as a small NATIVE_DEFAULT leak when scanning a parquet partition. The added test sweeps the RSS ceiling finely across the buffer reopen and asserts the scan leaves no native memory behind, then that the query recovers once the ceiling is removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AbstractPostingIndexReader.openSidecarFilesIfPresent() swallowed any error
raised while opening a covering index's sidecar info file: it logged the
failure, cleared the sidecar memories, and returned normally. The covering
plan, however, is fixed at compile time and the covered read path has no
base-table fallback, so a covered read then walked an empty sidecarMems and
threw "index out of bounds, 0 >= 0" (an out-of-bounds read once assertions
are disabled), masking the real I/O error.
Propagate the failure instead. The caller already wraps the open in
catch (Throwable) { close(); throw; }, so the reader is closed and the query
fails with the underlying error rather than reading an absent sidecar.
The query fuzzer's file fault injection surfaced this on a covered VARCHAR
column. The added test builds a posting covering index over a VARCHAR column
and sweeps a failing filesystem op across the reader's open path, asserting
the covered read never throws an AssertionError and recovers once the fault
is removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PostingGenLookup allocated its cache buffers as inline field initializers: a DirectLongList followed by a DirectIntLongHashMap, both tagged NATIVE_INDEX_READER. If the second allocation tripped the RSS memory limit, the first one leaked: the half-built PostingGenLookup was never assigned to the AbstractPostingIndexReader that was constructing it, so the reader's close() could not reach the buffer to free it. Move both allocations into an exception-safe constructor that frees whichever buffer was already allocated before rethrowing, so a failed construction leaves nothing behind. The query fuzzer's malloc fault injection surfaced this as a small NATIVE_INDEX_READER leak when reading through a posting index. The added test sweeps the RSS ceiling across posting index reader construction and relies on assertMemoryLeak to confirm the native buffers return to baseline, then that the query recovers once the ceiling is removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
IntervalBwdPartitionFrameCursor.calculateSize() diverged from next() and returned a wrong row count for a backward scan over a half-open interval such as `ts < X ORDER BY ts DESC`. When the lower interval bound was open (Long.MIN_VALUE), the method shortcut `lo = 0` instead of letting findTimestamp() return the -1 sentinel its whole-partition check relied on. That single wrong value caused two faults at once: it undercounted the partition by one (it used `hi = rowCount - 1` against `lo = 0`), and, because `lo == -1` never held, it always took the fragment branch and stopped after the first (newest) partition, ignoring every earlier partition in range. A two-partition table that materialized 96 rows reported a size of 47. calculateSize() now mirrors next() exactly, the same way the forward cursor's calculateSize() mirrors forward next(): it derives limitHi from partitionLimit, reads the high bound with timestampAt(limitHi), applies the +1 boundary convention for lo/hi, and detects a whole partition via `lo == 0`. Closed and open-ended intervals were already counted correctly and are unaffected; the forward cursor never had the open-bound shortcut and is correct. The bug surfaced through the query fuzzer's cursor self-consistency check (calculateSize() vs the materialized row count). The existing interval frame-cursor tests only ran the forward direction and only closed-lower-bound intervals, so they missed it. The test battery now runs every interval shape in both directions and adds open-start and open-end cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
assertCalculateSize() only compared calculateSize() to itself across toTop() and a cursor reopen. Those checks confirm the value is stable, but not that it is correct: a cursor whose calculateSize() is consistently wrong -- for instance a frame cursor that stops after the first partition and returns the same undercount every time -- satisfied all three comparisons. The method now first materializes the cursor with a full hasNext() iteration and asserts calculateSize() equals that count, which is the property calculateSize() actually promises (how many rows the cursor would yield). The full assertQuery battery already made an equivalent comparison in assertCursor() behind a random skip; pulling a deterministic, unconditional version into assertCalculateSize() removes the dependence on that random draw and makes the dedicated check self-sufficient. Verified across a broad cross-section of query shapes (limit, order by, distinct, latest by, sample by, interval filters, parquet, hash/asof joins, union, group by) with no regressions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reverts "Anchor assertCalculateSize to the materialized row count". The check it added was redundant: assertCursor() already compares calculateSize() to the materialized hasNext() count on every non-empty .returns() assertion, and runs before assertCalculateSize() on the same factory, so the row-count anchor was already enforced. The only gaps the revert reopens are minor (empty cursors and making the from-top check deterministic rather than behind a random skip), and they do not justify a second full cursor iteration on every assertQuery in the suite. The genuine coverage that catches the backward interval-scan bug stays: the bidirectional interval frame-cursor tests added alongside the fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a WindowClause that emits SELECT projections with window functions over OVER (PARTITION BY ... ORDER BY ts [frame]) clauses. It spans the ranking family (row_number, rank, dense_rank, cume_dist, percent_rank, ntile), the windowed aggregates (avg, sum, min, max, count) and the navigation functions (first_value, last_value, lag, lead, nth_value), with ROWS and time-based RANGE frames. Every window orders by the designated timestamp, which the fuzz tables generate strictly increasing, so a total order inside each partition makes the window values reproducible across runs and the differential oracle can compare full result sets rather than just row counts. As with the other clauses, every main-rnd draw is independent of the BindContext, so the literal and bind passes build the same tree shape; only the constant leaves rendered through the context differ. The QueryGenerator carves the WINDOW band out of the upper half of the SIMPLE range, leaving the SAMPLE BY and GROUP BY bands untouched, so toggling window off restores the exact pre-window shape distribution. The questdb.fuzz.window knob is on by default, like fault injection; it immediately surfaces several known-open window-function defects, so the run goes red on the seeds that hit them until those are fixed. Pass -Dquestdb.fuzz.window=false to drop window shapes and exercise the rest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
afae2e2 to
f32849a
Compare
max/min/first_value/last_value/nth_value over a DATE argument previously fell through to the timestamp window factories. The resulting WindowTimestampFunction reported getType() = DATE but read and stored the value in timestamp microseconds, so the streaming read path threw UnsupportedOperationException from WindowFunction.getDate and the cached read path returned dates roughly 1000x in the future (microseconds read back as milliseconds). Add dedicated DATE window factories (max(M), min(M), first_value(M), last_value(M), nth_value(ML)) backed by WindowDateFunction, mirroring the existing lag/lead DATE factories. They read arg.getDate() and store, write, and emit milliseconds, so both the streaming and cached paths return correct DATE values. MinDate reuses MaxDate's inner classes; the classpath scanner registers the new factories automatically. Add DateNullFunction for empty-frame results. Cover the change with WindowDateFunctionTest (all five functions on the streaming and cached paths, with NULLs and partitions) and four DATE methods in WindowFunctionTest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The DATE max/min/first_value/last_value/nth_value window factories duplicated the entire timestamp implementation. Collapse them into thin subclasses of the timestamp factories that only override getSignature(). The timestamp value functions now read the argument through readArgValue(), which returns the native long for every argument type the factories accept: getDate() for a DATE argument (milliseconds), and getTimestamp() otherwise (timestamp ticks, including a SYMBOL, STRING or VARCHAR parsed to a timestamp). A type-aware getDate() default on WindowTimestampFunction converts the stored value back to DATE milliseconds for the streaming read path. readArgValue routes TIMESTAMP arguments, and SYMBOL/STRING/VARCHAR arguments, through getTimestamp() exactly as before, so the change preserves behavior for the timestamp factories while removing about 10,800 lines of duplicated code. Add a nanosecond designated-timestamp case to WindowDateFunctionTest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Refactor the max/min window value functions into the lag/lead pattern: a type-agnostic abstract base per window shape plus thin DATE and TIMESTAMP subclasses that each carry a clean per-type accessor. New MaxMinWindowFunctionFactoryHelper holds the nine abstract bases (the framing logic, the native-long value buffers, the comparator and name), the seven constructor functional interfaces, the TimestampComparator and the MAX_*_COLUMN_TYPES constants. Its single newInstance dispatcher replaces the two near-duplicate per-shape dispatchers that previously lived in the Max and Min timestamp factories; the caller passes a comparator, a name and the per-type constructor references, so the same control flow serves both DATE and TIMESTAMP arguments and both max and min. The TIMESTAMP subclasses (MaxTimestampWindowFunctionFactory) cache the timestamp driver once in the constructor and expose getTimestamp() as the stored value plus a getDate() that converts ticks to DATE milliseconds. This removes the per-row type re-derivation that the previous polymorphic getDate() performed. The DATE subclasses (MaxDateWindowFunctionFactory) store and emit the value in DATE milliseconds and only expose getDate(); WindowDateFunction derives the rest, including getTimestamp() = value * 1000. Two-pass shapes (over partition, over whole result set) materialize their result and never read back through the accessor, so they only override getType(). MaxDate and MinDate now extend AbstractWindowFunctionFactory directly and build the DATE subclasses, rather than inheriting the TIMESTAMP factory's newInstance. Min reuses Max's subclasses with a LESS_THAN comparator. The heavy WindowTimestampFunction.getDate() default is left in place: the first_value/last_value/nth_value timestamp shapes still rely on it and have not been split yet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply the lag/lead pattern to the first_value window value functions, as done for max/min in the previous commit. Each window shape now has one type-agnostic abstract base in the new FirstValueWindowFunctionFactoryHelper (holding all framing logic and native-long value buffers), plus a thin DATE or TIMESTAMP subclass that adds only the per-type accessor: the TIMESTAMP subclass caches the timestamp driver once and exposes getTimestamp() as the stored value plus a getDate() that converts ticks to milliseconds, while the DATE subclass exposes the stored milliseconds directly through getDate(). Two-pass shapes expose only getType(). The helper holds the 15 shape bases (the RESPECT NULLS bases plus the IGNORE NULLS bases that extend them), the seven constructor functional interfaces, the shared FIRST_VALUE_COLUMN_TYPES, and one consolidated newInstance dispatcher parameterized by per-type constructor references so the same control flow serves both DATE and TIMESTAMP arguments. The dispatcher now frees the map and circular-buffer memory on the constructor error path via try/catch, closing a leak the query fuzzer's fault injection surfaces. FirstValueDateWindowFunctionFactory stops inheriting the timestamp factory's newInstance and becomes a real factory that builds the DATE subclasses. Verified green: WindowFunctionTest 606, WindowDateFunctionTest 22, WindowDecimalFunctionTest 431, WindowFunctionUnitTest 40, and the PR fuzzer seed s0=394723763672806 s1=1781015833112 with window and faults enabled. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply the same lag/lead pattern to the last_value window value functions. Each window shape now has one type-agnostic abstract base in the new LastValueWindowFunctionFactoryHelper (holding all framing logic and native-long value buffers), plus a thin DATE or TIMESTAMP subclass that adds only the per-type accessor: the TIMESTAMP subclass caches the timestamp driver once and exposes getTimestamp() as the stored value plus a getDate() that converts ticks to milliseconds, while the DATE subclass exposes the stored milliseconds directly through getDate(). Shapes that write the result column directly during a backward pass1 scan expose only getType(). The helper holds the 17 shape bases (two IGNORE NULLS bases extend their RESPECT NULLS sibling; the rest are independent), the eight constructor functional interfaces, the four shared column-type templates, and one consolidated newInstance dispatcher parameterized by per-type constructor references so the same control flow serves both DATE and TIMESTAMP arguments. The dispatcher now frees the map and circular-buffer memory on the constructor error path via try/catch, closing a leak the query fuzzer's fault injection surfaces. LastValueDateWindowFunctionFactory stops inheriting the timestamp factory's newInstance and becomes a real factory that builds the DATE subclasses. Verified green: WindowFunctionTest 606, WindowDateFunctionTest 22, WindowFunctionUnitTest 40, and the PR fuzzer seed s0=394723763672806 s1=1781015833112 with window and faults enabled. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply the same lag/lead pattern to the nth_value window value functions. Each window shape now has one type-agnostic abstract base in the new NthValueWindowFunctionFactoryHelper (holding all framing logic and the native-long value buffers), plus a thin DATE or TIMESTAMP subclass that adds only the per-type accessor: the TIMESTAMP subclass caches the timestamp driver once and exposes getTimestamp() as the stored value plus a getDate() that converts ticks to milliseconds, while the DATE subclass exposes the stored milliseconds directly through getDate(). Two-pass shapes expose only getType(). The helper holds the 11 independent shape bases, the nine constructor functional interfaces (the n index threads through each), and the single newInstance dispatcher parameterized by per-type constructor references so the same control flow serves both DATE and TIMESTAMP arguments. The dispatcher keeps the existing try/catch map and circular-buffer cleanup on the constructor error path. NthValueDateWindowFunctionFactory stops inheriting the timestamp factory's newInstance and becomes a real factory that builds the DATE subclasses. Verified green: WindowFunctionTest 606, WindowDateFunctionTest 22, WindowFunctionUnitTest 40, and the PR fuzzer seed s0=394723763672806 s1=1781015833112 with window and faults enabled. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
With max/min/first_value/last_value/nth_value now split into per-type subclasses, every streaming TIMESTAMP window value function overrides getDate() with its cached timestamp driver, so the type-dispatching getDate() default on WindowTimestampFunction is no longer reached on that path. Drop it so each timestamp window function states its own getDate() (or inherits the throwing default when it never serves the accessor). The only accessor-read function that relied on the default was the shared empty-frame TimestampNullFunction, so it now provides getDate() directly, returning its stored LONG_NULL (the NULL sentinel for both DATE and TIMESTAMP results). Two-pass and backward-scan shapes (whole-result-set, partition, lead/lead-over-partition) write their result straight into the window column and are read back from the record chain, never through getDate(); their inherited throwing default is unreachable, exactly as before this change (the old default would itself have thrown via the throwing getTimestamp()). The change is behavior-preserving: a streaming timestamp subclass returned driver.toDate(value) before and after. Verified green: WindowFunctionTest 606, WindowDateFunctionTest 22, WindowDecimalFunctionTest 431, WindowFunctionUnitTest 40, and the PR fuzzer seed s0=394723763672806 s1=1781015833112. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rank() and dense_rank() over (partition by ... order by ts) take the streaming WindowRecordCursorFactory path when the order matches the designated timestamp. That path remembered each partition's previous row by copying the whole projected row into a MapValue and reading the ORDER BY columns back from it. RankOverPartitionFunction built the copier with RecordValueSinkFactory over every projected column, so any pass-through column the MapValue cannot hold (UUID, STRING, VARCHAR, BINARY, LONG256, arrays, INTERVAL) threw UnsupportedOperationException at compile time. A plain SELECT then failed even though the rank itself was fine; projecting only the window function avoided it. Only the ORDER BY columns decide whether two consecutive rows are peers, so the MapValue now holds just those (compacted into slots 0..k-1) plus the running rank and count. The value sink reads the ORDER BY columns from the live record at their base indices and writes them compacted; the comparator and rank maps are built over a matching compacted metadata so they read back only those columns. computeNext no longer compares against the live record. It caches the previous row's ORDER BY values via setLeft (which the comparator stores by value for every type the MapValue can hold), overwrites the slots with the current row, then compares the cached previous row against the MapValue. Keeping the comparator off the live record is what lets the compacted layout work. Rank maps are indexed by compacted ORDER BY position while the symbol tables live at base column indices, so init() builds each map through a small position-to-base-index mapping. This only matters for the rare advice-driven symbol ORDER BY; the common designated-timestamp ORDER BY has no symbol maps. The no-partition path was already safe (RecordSinkFactory over the ORDER BY columns plus memeq), and the cached path is unchanged. WindowFunctionTest.testRankWithUnserializablePassThroughColumn asserts rank/dense_rank over UUID, STRING, VARCHAR, LONG256 and DOUBLE[] pass-through columns (ascending and descending), drains a BINARY pass-through, and pins the streaming Window plan so the fix keeps exercising the right path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lead() over a Decimal128 or Decimal256 argument reported ZERO_PASS, unlike every other lead variant (Double, Long, and the narrower decimals all inherit ONE_PASS). When a query's only window functions were wide-decimal leads and the window ordered by the designated timestamp, that ZERO_PASS hint routed the query through the streaming window executor. lead reads ahead, so it cannot run in the forward streaming pass; it is computed in the cached executor with a backward pass1 scan. In the streaming path the wide-decimal lead also exposed no Decimal128 or Decimal256 read accessor, so reading the result fell through to the default-throwing WindowFunction accessor and raised UnsupportedOperationException. Earlier all-types tests masked this because a narrow-decimal column in the same projection forced the cached path. Removing the erroneous getPassCount() override lets these two over-partition functions inherit ONE_PASS like the rest, so the planner keeps them on the cached path where pass1 already does the work and the record chain reads the value back. Repro (now passing): SELECT lead(d, 5) OVER (PARTITION BY g ORDER BY ts) FROM t with d DECIMAL(76, 3). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Triage of open bug #1 from the window-function fuzzing: a sum() or avg() over a high-precision DECIMAL (Decimal256-backed, precision > 38) raises "CairoException: ... aggregation failed: an overflow occurred". The repro is avg(d) OVER (PARTITION BY g ORDER BY ts) with d DECIMAL(76,3). This is a genuine, data-dependent arithmetic limit, not an engine defect. The running total of two values near the DECIMAL(76,3) maximum exceeds Decimal256's 256-bit capacity, so the sum is genuinely unrepresentable. The plain group-by sum/avg overflows identically on the same data, and WindowDecimalFunctionTest already pins this overflow as the expected behaviour. Decimal256 is the widest decimal type, so there is no wider accumulator to promote to. The fuzzer oracle was reporting it as a failure because it treats any leaked CairoException as an internal error. Teach isAcceptedSkip to also accept a CairoException whose message contains "aggregation failed", classifying it like a NumericException overflow. The match is narrow: only the decimal sum/avg functions emit "aggregation failed", and every such message is an overflow. Tests: - WindowDecimalFunctionTest.testAvgDecimal256OverPartitionGenuineOverflow and testSumDecimal256OverPartitionGenuineOverflow pin that both the window-over-partition form and the plain group-by form overflow on near-max DECIMAL(76,3) data. - QueryFuzzTest.testDecimalAggregationOverflowToleratedByOracle drives the repro queries through QueryRunner.run() and asserts the oracle skips them, while a non-overflowing query still runs to completion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In the streaming WindowRecordCursorFactory path the code generator builds each window function's partition map lazily, in initRecordComparator(), after the whole projection has been walked. The generator reuses a single key-types buffer and clears and rebuilds it for every window column's PARTITION BY, but rank()/dense_rank() over a partition kept a live reference to that buffer instead of a copy. So when a query had two or more streaming window functions partitioning by differently typed keys, the rank function built its map from the last window column's key types rather than its own. That either crashed - when the partition sink wrote a type the wrongly picked map could not hold, e.g. a BYTE partition writing into the Unordered4Map selected for a SYMBOL key threw UnsupportedOperationException - or silently mis-keyed the partitions, e.g. a Decimal256 partition map built for a two-column key fed by a single-column sink reported rank 1 for every row and could even diverge across a toTop() re-read. A single rank function, or several sharing one partition key, masked it. RankOverPartitionFunction now snapshots the key types in its constructor, while the buffer still describes its own PARTITION BY. The two sibling over-partition functions that store the same reference, cume_dist and percent_rank, get the same snapshot for consistency; they are two-pass and always take the cached path where initRecordComparator runs in-loop, so they were not yet exploitable, but the shared idiom is now uniformly safe. The copy lives in a shared AbstractWindowFunctionFactory.copyKeyTypes() helper. Adds WindowFunctionTest regression coverage for rank/dense_rank with mixed BYTE/SYMBOL and wide-DECIMAL partition keys on the streaming path, plus cume_dist/percent_rank with mixed keys on the cached path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A window value or navigation function whose argument is the untyped null literal (e.g. lead(null, 2), min(null)) resolves to a NULL output column type. The cached window path then asks RecordSinkFactory to build a sink over that NULL column and leaks an internal "IllegalArgumentException: Unexpected column type: NULL" through the query API. Whether the leak surfaced depended on the access path: the streaming path tolerated the NULL column, while the cached path (read-ahead lead, or min/max over a DESC ROWS frame) crashed, so the same function failed on one frame and silently returned nulls on another. The code generator now rejects a window function whose output type is NULL with a user-facing SqlException at the function position, suggesting a concrete cast (e.g. null::double, which compiles and computes over NULL). The guard sits in both window parse loops, so the rejection is consistent across the streaming and cached paths. Window functions that resolve to a concrete output type for a null literal (lag, max, first_value, last_value, avg, sum, count) keep computing over NULL as before; nth_value already had its own NULL guard. This is open bug #1 surfaced by the query fuzzer's window shapes. The clean SqlException is an accepted skip in QueryRunner.isAcceptedSkip, so the fuzzer seeds that hit lead(null)/min(null) now run green instead of crashing on the internal exception. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
testNullLiteralArgumentRejectedWithCleanError failed on linux-arm64 (a
clean rejection but with the wrong message: "lead is not yet implemented
for NULL" instead of "does not support an untyped NULL argument").
Root cause: a window value function over the untyped null literal, e.g.
lead(null, 2), ties across every typed variant during overload
resolution -- NULL has zero overload distance to any type. The five
lead factories (Double, Long, Decimal, Date, Timestamp) all match with
the same score, so the winner is decided by Java's stable sort
preserving the classpath scan order, which differs by filesystem and
architecture. Each winner behaves differently:
- Timestamp/Date variants are type-adaptive and yield a NULL output
type, which the SqlCodeGenerator guard rejected cleanly (x64);
- the Decimal variant throws "lead is not yet implemented for NULL"
from its own factory (arm64);
- the Double/Long variants yield a concrete output type and silently
accept the query, returning NULLs.
So the same SQL gave three different results across platforms, and the
SqlCodeGenerator guard only covered the first.
FunctionParser now rejects an untyped NULL value argument to a
polymorphic window function before any factory runs, so every platform
gives the same clean SqlException at the same position. The check is
scoped to window functions with more than one overload, so ntile -
whose single argument is a bucket count with its own NULL validation -
keeps its specific "bucket count cannot be NULL" error. The two now
redundant SqlCodeGenerator guards are removed; the resolution-time
check covers both the streaming and cached paths at once.
The test's nth_value assertion is updated: nth_value over an untyped
NULL is now rejected with the same uniform message as every other
window function rather than its old factory-specific one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 12
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
core/src/test/java/io/questdb/test/cairo/fuzz/FailureFileFacade.java (1)
519-557:⚠️ Potential issue | 🟠 Major | ⚡ Quick winReset all scoping fields when changing arming modes.
setToFailAfter(...)andsetToFailAfterOnCurrentThread(...)leave mode-3 state (queryWorkerNamePrefix/armingThread) behind. After a mode switch,checkForFailure()can keep filtering calls by stale query-thread rules, so the intended fault never fires.Suggested fix
public void setToFailAfter(int ioFailureCallCount) { // Default (unscoped): any thread's op counts and fires. Arm only when not // already armed so a pending fault is not reset by a concurrent caller. armedThread = null; + armingThread = null; + queryWorkerNamePrefix = null; + contaminatingThreadName = null; int osCalls; while (true) { osCalls = osCallsCount.get(); if (osCalls > 0 || osCallsCount.compareAndSet(osCalls, ioFailureCallCount)) { return; @@ public void setToFailAfterOnCurrentThread(int ioFailureCallCount) { + armingThread = null; + queryWorkerNamePrefix = null; + contaminatingThreadName = null; // Only the arming thread counts down and fires (see checkForFailure), so // the same thread both writes and reads osCallsCount and no CAS guard is // needed against concurrent decrements. osCallsCount.set(ioFailureCallCount); armedThread = Thread.currentThread(); }Also applies to: 627-642
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/src/test/java/io/questdb/test/cairo/fuzz/FailureFileFacade.java` around lines 519 - 557, The arming mode methods do not properly reset scoping state when switching between modes, leaving stale filtering rules from previous modes. In setToFailAfter(), in addition to setting armedThread to null, also reset queryWorkerNamePrefix to null and armingThread to null. In setToFailAfterOnCurrentThread(), before setting the new thread-scoped state, also reset queryWorkerNamePrefix to null and armingThread to null. These resets ensure that checkForFailure() uses only the current mode's filtering rules and not stale state from a previous mode.
🧹 Nitpick comments (7)
core/src/test/java/io/questdb/test/griffin/engine/table/PageFrameRecordCursorImplFactoryTest.java (1)
471-517: ⚡ Quick winUse underscore separators in new 5+ digit numeric literals.
The new tests introduce several 5+ digit literals (
10000,1000000L, etc.) without underscore grouping. Please update these to match repository convention.Suggested diff
- final int numOfRows = 10000; - final long increment = 1000000L * 60 * 60; - final long intervalHi = 1000000L * 60 * 60 * 24 * 3; + final int numOfRows = 10_000; + final long increment = 1_000_000L * 60 * 60; + final long intervalHi = 1_000_000L * 60 * 60 * 24 * 3;As per coding guidelines, “Use underscore to separate thousands in numbers with 5 digits or more in test SQL and code” and “Always use them in numbers of 5 digits or more in both test code and implementation.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/src/test/java/io/questdb/test/griffin/engine/table/PageFrameRecordCursorImplFactoryTest.java` around lines 471 - 517, Add underscore separators to all numeric literals with 5 or more digits in the test methods testFactory_IntervalPartitionFrameCursorFactory_openStart, testFactory_IntervalPartitionFrameCursorFactory_openStart_skip, testFactory_IntervalPartitionFrameCursorFactory_openEnd, and testFactory_IntervalPartitionFrameCursorFactory_openEnd_skip. Specifically, update 10000 to 10_000 and 1000000L to 1_000_000L throughout these test methods to match the repository's coding convention for numeric literal formatting.Source: Coding guidelines
core/src/test/java/io/questdb/test/griffin/engine/groupby/vect/GroupByVectorizedRostiAccountingTest.java (1)
123-127: ⚡ Quick winUse underscore separators for 5+ digit numeric literals in test SQL.
Please format the timestamp literals with separators to match the repo rule and improve readability.
Proposed fix
- execute("INSERT INTO tab SELECT (x * 1000000L)::timestamp, x FROM long_sequence(8)", context); + execute("INSERT INTO tab SELECT (x * 1_000_000L)::timestamp, x FROM long_sequence(8)", context); ... - execute("INSERT INTO tab SELECT (86400000000L + x * 1000000L)::timestamp, x, x::int " + + execute("INSERT INTO tab SELECT (86_400_000_000L + x * 1_000_000L)::timestamp, x, x::int " + "FROM long_sequence(" + liveKeys + ")", context);As per coding guidelines, “Use underscore to separate thousands in numbers with 5 digits or more in test SQL and code” and “Always use them in numbers of 5 digits or more in both test code and implementation.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/src/test/java/io/questdb/test/griffin/engine/groupby/vect/GroupByVectorizedRostiAccountingTest.java` around lines 123 - 127, The numeric literals in the SQL execute statements within the test method are missing underscore separators for readability. Locate the two execute method calls that contain the timestamp literals: the first one with `1000000L` and the second one with `86400000000L`. Add underscore separators to both numeric literals following the convention of separating every three digits from the right (for example, `1000000L` becomes `1_000_000L` and `86400000000L` becomes `86_400_000_000L`) to comply with the coding guidelines for numeric literals with 5 or more digits.Source: Coding guidelines
core/src/test/java/io/questdb/test/griffin/fuzz/PredicateGenerator.java (1)
68-76: ⚡ Quick winRename boolean flag to
is.../has...form.
injectFaultFnshould follow the repo boolean naming rule (for example,isFaultFnInjectedorhasFaultFnInjection).As per coding guidelines: "When choosing a name for a boolean variable, field or method, always use the
is...orhas...prefix, as appropriate."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/src/test/java/io/questdb/test/griffin/fuzz/PredicateGenerator.java` around lines 68 - 76, Rename the boolean parameter `injectFaultFn` in the `appendWhere` method to follow the repository's boolean naming convention. Change it to use the `is...` or `has...` prefix form (such as `isFaultFnInjected` or `hasFaultFnInjection`). Update all references to this parameter throughout the method body to use the new name consistently.Source: Coding guidelines
core/src/test/java/io/questdb/test/griffin/fuzz/QueryGenerator.java (1)
70-70: ⚡ Quick winUse
is...boolean names in the public generator signature.Please rename
injectFaultFn/windowEnabledtois.../has...forms to keep API naming consistent with project conventions.As per coding guidelines: "When choosing a name for a boolean variable, field or method, always use the
is...orhas...prefix, as appropriate."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/src/test/java/io/questdb/test/griffin/fuzz/QueryGenerator.java` at line 70, The boolean parameters in the generate method signature do not follow the project's naming convention for boolean variables. Rename injectFaultFn to isInjectFaultFn and windowEnabled to isWindowEnabled (or use hasWindowEnabled if more semantically appropriate) in the method signature and update all references to these parameters throughout the method body to maintain consistency with the coding guidelines that require is... or has... prefixes for boolean names.Source: Coding guidelines
core/src/test/java/io/questdb/test/griffin/fuzz/FuzzConfig.java (1)
39-44: ⚡ Quick winKeep new static constants alphabetically ordered in the static-public member group.
FAULTS_PROPis placed beforeFAULT_PARALLEL_PROPandFAULT_PCT_PROP, which breaks the Java member-ordering rule used in this repo.Suggested reorder
- public static final String FAULTS_PROP = "questdb.fuzz.faults"; public static final String FAULT_PARALLEL_PROP = "questdb.fuzz.fault.parallel"; public static final String FAULT_PCT_PROP = "questdb.fuzz.fault.pct"; + public static final String FAULTS_PROP = "questdb.fuzz.faults";As per coding guidelines: "Group Java class members by kind (static vs. instance) and visibility, and sort them alphabetically. Insert new methods and fields in the correct alphabetical position among existing members of the same kind."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/src/test/java/io/questdb/test/griffin/fuzz/FuzzConfig.java` around lines 39 - 44, The static constants in the FuzzConfig class are not in alphabetical order. Reorder the constants FAULTS_PROP, FAULT_PARALLEL_PROP, and FAULT_PCT_PROP so they appear in correct alphabetical sequence. FAULT_PARALLEL_PROP and FAULT_PCT_PROP (which start with underscore after FAULT_) should come before FAULTS_PROP (which ends with S after FAULT_), then follow with QUERIES_PROP, VERIFY_CURSOR_PROP, and WINDOW_PROP.Source: Coding guidelines
core/src/test/java/io/questdb/test/griffin/GroupByTest.java (1)
1420-1421: ⚡ Quick winUse underscores in 5+ digit numeric literals for test readability and guideline compliance.
Please format
859371and237288as859_371and237_288in comments and SQL literals.Suggested patch
- // 859371 + (cnt * -237288) out of the inner GROUP BY when its base + // 859_371 + (cnt * -237_288) out of the inner GROUP BY when its base @@ - assertQuery("SELECT t0.cnt AS e0, (859371 + (t0.cnt * -237288)) AS e1, count() AS a0\n" + assertQuery("SELECT t0.cnt AS e0, (859_371 + (t0.cnt * -237_288)) AS e1, count() AS a0\n" @@ - assertQuery("SELECT t0.cnt AS e0, (859371 + (t0.cnt * -237288)) AS e1, count() AS a0\n" - + body + "GROUP BY t0.cnt, (859371 + (t0.cnt * -237288)) ORDER BY 1 ASC, e1") + assertQuery("SELECT t0.cnt AS e0, (859_371 + (t0.cnt * -237_288)) AS e1, count() AS a0\n" + + body + "GROUP BY t0.cnt, (859_371 + (t0.cnt * -237_288)) ORDER BY 1 ASC, e1")As per coding guidelines, "Use underscore to separate thousands in numbers with 5 digits or more in test SQL and code."
Also applies to: 1438-1439, 1444-1445
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/src/test/java/io/questdb/test/griffin/GroupByTest.java` around lines 1420 - 1421, Update numeric literals in the GroupByTest.java file to comply with coding guidelines by adding underscores to separate thousands in all numbers with 5 or more digits. Specifically, format 859371 as 859_371 and 237288 as 237_288 in the comments and any SQL literals at the specified locations. Additionally, review and update any other 5+ digit numeric literals at lines 1438-1439 and 1444-1445 with the same underscore formatting pattern.Source: Coding guidelines
core/src/test/java/io/questdb/test/griffin/engine/window/WindowFunctionTest.java (1)
2448-2453: ⚡ Quick winUse text blocks for multi-row INSERT SQL in tests.
These long multi-row INSERTs are still assembled via concatenated strings. Please switch them to Java multiline string literals (
"""...""") for readability and guideline consistency.As per coding guidelines, "Use multiline strings for longer SQL statements (multiple INSERT rows, complex queries), as well as to assert multiline query results in tests."
Also applies to: 7313-7318, 8702-8707, 11062-11067, 17351-17357, 17647-17653, 18045-18054
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/src/test/java/io/questdb/test/griffin/engine/window/WindowFunctionTest.java` around lines 2448 - 2453, Replace the multi-row INSERT statement in the execute() method call that uses concatenated strings (with + operators) with a Java text block using triple quotes ("""). Convert the entire SQL statement from the string concatenation format into a single readable multiline text block enclosed in triple quotes, ensuring the SQL syntax remains unchanged. This change should also be applied to all other similar multi-row INSERT statements mentioned in the additional locations (lines 7313-7318, 8702-8707, 11062-11067, 17351-17357, 17647-17653, 18045-18054) in the same test file.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@core/src/main/java/io/questdb/griffin/engine/functions/window/MaxMinWindowFunctionFactoryHelper.java`:
- Around line 959-964: The close() method in the
MaxMinWindowFunctionFactoryHelper class closes the memory field but fails to
close the dequeMemory field, which is allocated for bounded-row frames and
causes native buffer leaks. Add a call to close dequeMemory in the close()
method after the super.close() call to ensure both memory buffers are properly
released.
- Around line 1095-1114: The toPlan() method in the
MaxMinWindowFunctionFactoryHelper class incorrectly calculates the upper bound
for ROWS frames ending in PRECEDING using the formula bufferSize - frameSize on
lines 1113 and 1570 (and in the similar block at lines 1555-1573). This
calculation underreports the actual preceding row offset. Replace the expression
bufferSize - frameSize with just frameSize to correctly represent the number of
rows preceding the current row at which the frame upper bound occurs. This
applies to all locations where frameIncludesCurrentValue is false and we are
rendering the "AND <n> PRECEDING" portion of the window frame specification.
In `@core/src/main/java/io/questdb/griffin/FunctionParser.java`:
- Around line 1236-1241: The error position in the SqlException thrown for
untyped NULL arguments is using node.position which marks the function name
instead of the actual problematic argument. Replace node.position with
argPositions.getQuick(0) in the SqlException constructor call to point the error
diagnostic at the first argument position where the NULL argument actually
appears, providing more precise error location for users.
In `@core/src/main/java/io/questdb/jit/CompiledFilterIRSerializer.java`:
- Around line 1032-1085: The boolean variable names in the
CompiledFilterIRSerializer class do not follow the repository's naming
convention which requires boolean variables to use is... or has... prefixes.
Rename the following boolean parameters and variables to comply with this
convention: the underLong parameter in markI64Widen method, the thisLong
variable in markI64Widen method, the parentLong parameter in markI64WidenOperand
method, the widenToI64 variable around line 1212, and the keepI4 variable around
lines 1770-1780. Choose appropriate is... or has... prefixes based on the
semantic meaning of each boolean (for example, underLong could become
isUnderLong, thisLong could become isThisLong, parentLong could become
isParentLong, etc.). Update all references to these renamed variables throughout
the affected methods to maintain consistency.
In
`@core/src/test/java/io/questdb/test/cairo/covering/CoveringIndexSidecarFaultTest.java`:
- Around line 77-87: The test loop iterating through failAfter values (1 to 80)
and calling drain(query, scratch) does not verify that the fault injection
actually triggers at least once. Add a tracking variable (boolean flag or
counter) that records when an exception is caught in the catch block within the
loop, and after the loop completes, add an assertion to verify that at least one
fault was triggered during the iterations. This ensures the test validates that
the covering-sidecar open path is actually being exercised and prevents the test
from passing when no faults occur.
In
`@core/src/test/java/io/questdb/test/griffin/CompiledFilterRegressionTest.java`:
- Around line 1559-1573: The boolean parameter expectJit in the
assertJitMatchesJava method does not follow the repository naming convention for
boolean variables. Rename the parameter expectJit to isJitExpected (or similar
is... prefix format) and update all references to this parameter throughout the
method body, specifically in the Assert.assertEquals call where expectJit is
currently being used as an argument.
- Around line 1073-1087: Add underscore separators to all numeric literals with
five or more digits in this test method. Specifically, update the literals in
the execute INSERT statement (1000000 and 1800000000L) and in all the
assertJitMatchesJava SQL queries (776782 and 2000000000) to use underscore
grouping for thousands to match the coding guidelines for this file.
In
`@core/src/test/java/io/questdb/test/griffin/engine/groupby/SampleByInterpolateOomTest.java`:
- Around line 79-84: The test currently only creates and closes a
RecordCursorFactory without actually opening the cursor, which means it does not
exercise the cursor-construction cleanup path that the test is meant to verify.
After creating the RecordCursorFactory with select(query), you need to open the
cursor from the factory (typically by calling getRecordCursor() on the factory)
and then iterate through or access the cursor data to actually trigger the
construction and cleanup logic. Make sure the opened cursor is properly closed
or managed within a try-with-resources block to ensure the regression is
properly caught.
In
`@core/src/test/java/io/questdb/test/griffin/engine/table/parquet/ParquetScanOomTest.java`:
- Line 59: Fix two coding guideline violations in the ParquetScanOomTest method:
First, apply underscore separators to the numeric literal 1000000L on line 59 by
converting it to 1_000_000L to improve readability for numbers with more than 6
digits. Second, rename the boolean variable sawOom (appearing on lines 73, 82,
and 87) to use the has prefix convention, changing it to hasOom throughout the
method to follow the naming guidelines for boolean variables.
In
`@core/src/test/java/io/questdb/test/griffin/fuzz/EarlyExitGroupByDetectionTest.java`:
- Around line 49-50: In the execute method call within
EarlyExitGroupByDetectionTest, the numeric literal `1000000L` has 6 digits and
should use underscore separators for improved readability and consistency with
project coding guidelines. Replace `1000000L` with the underscore-separated
version following the thousands separator pattern to make the number more
readable.
In `@core/src/test/java/io/questdb/test/griffin/fuzz/QueryRunner.java`:
- Around line 418-421: Remove the broad SQL-text heuristic from the
discardedEarly condition in the QueryRunner class. The
Chars.containsLowerCase(sql, "limit") check incorrectly treats any query
containing the word LIMIT as early-terminating, which can mask swallowed faults
on aggregate-with-limit queries. Replace the parallel condition so it only gates
discardedEarly on the planner-derived signals: outcome.hasPushedLimit and
outcome.hasEarlyExitGroupBy. Keep the parallel check but remove the
Chars.containsLowerCase call entirely, allowing only the actual planner outputs
to determine whether a query is early-terminating.
In `@core/src/test/java/io/questdb/test/jit/CompiledFilterIRSerializerTest.java`:
- Around line 415-416: The numeric literals in the test SQL strings lack
underscore separators for readability. In the serialize method call on line 415,
replace the numeric literal 100000 with underscore separators (5+ digit numbers
require this formatting). Also apply the same formatting fix to the numeric
literal 9000000000 referenced in the comment for lines 423-424. Update both the
serialize() call strings and their corresponding assertIR() assertion strings to
use properly formatted numeric literals with underscores for thousands
separation.
---
Outside diff comments:
In `@core/src/test/java/io/questdb/test/cairo/fuzz/FailureFileFacade.java`:
- Around line 519-557: The arming mode methods do not properly reset scoping
state when switching between modes, leaving stale filtering rules from previous
modes. In setToFailAfter(), in addition to setting armedThread to null, also
reset queryWorkerNamePrefix to null and armingThread to null. In
setToFailAfterOnCurrentThread(), before setting the new thread-scoped state,
also reset queryWorkerNamePrefix to null and armingThread to null. These resets
ensure that checkForFailure() uses only the current mode's filtering rules and
not stale state from a previous mode.
---
Nitpick comments:
In
`@core/src/test/java/io/questdb/test/griffin/engine/groupby/vect/GroupByVectorizedRostiAccountingTest.java`:
- Around line 123-127: The numeric literals in the SQL execute statements within
the test method are missing underscore separators for readability. Locate the
two execute method calls that contain the timestamp literals: the first one with
`1000000L` and the second one with `86400000000L`. Add underscore separators to
both numeric literals following the convention of separating every three digits
from the right (for example, `1000000L` becomes `1_000_000L` and `86400000000L`
becomes `86_400_000_000L`) to comply with the coding guidelines for numeric
literals with 5 or more digits.
In
`@core/src/test/java/io/questdb/test/griffin/engine/table/PageFrameRecordCursorImplFactoryTest.java`:
- Around line 471-517: Add underscore separators to all numeric literals with 5
or more digits in the test methods
testFactory_IntervalPartitionFrameCursorFactory_openStart,
testFactory_IntervalPartitionFrameCursorFactory_openStart_skip,
testFactory_IntervalPartitionFrameCursorFactory_openEnd, and
testFactory_IntervalPartitionFrameCursorFactory_openEnd_skip. Specifically,
update 10000 to 10_000 and 1000000L to 1_000_000L throughout these test methods
to match the repository's coding convention for numeric literal formatting.
In
`@core/src/test/java/io/questdb/test/griffin/engine/window/WindowFunctionTest.java`:
- Around line 2448-2453: Replace the multi-row INSERT statement in the execute()
method call that uses concatenated strings (with + operators) with a Java text
block using triple quotes ("""). Convert the entire SQL statement from the
string concatenation format into a single readable multiline text block enclosed
in triple quotes, ensuring the SQL syntax remains unchanged. This change should
also be applied to all other similar multi-row INSERT statements mentioned in
the additional locations (lines 7313-7318, 8702-8707, 11062-11067, 17351-17357,
17647-17653, 18045-18054) in the same test file.
In `@core/src/test/java/io/questdb/test/griffin/fuzz/FuzzConfig.java`:
- Around line 39-44: The static constants in the FuzzConfig class are not in
alphabetical order. Reorder the constants FAULTS_PROP, FAULT_PARALLEL_PROP, and
FAULT_PCT_PROP so they appear in correct alphabetical sequence.
FAULT_PARALLEL_PROP and FAULT_PCT_PROP (which start with underscore after
FAULT_) should come before FAULTS_PROP (which ends with S after FAULT_), then
follow with QUERIES_PROP, VERIFY_CURSOR_PROP, and WINDOW_PROP.
In `@core/src/test/java/io/questdb/test/griffin/fuzz/PredicateGenerator.java`:
- Around line 68-76: Rename the boolean parameter `injectFaultFn` in the
`appendWhere` method to follow the repository's boolean naming convention.
Change it to use the `is...` or `has...` prefix form (such as
`isFaultFnInjected` or `hasFaultFnInjection`). Update all references to this
parameter throughout the method body to use the new name consistently.
In `@core/src/test/java/io/questdb/test/griffin/fuzz/QueryGenerator.java`:
- Line 70: The boolean parameters in the generate method signature do not follow
the project's naming convention for boolean variables. Rename injectFaultFn to
isInjectFaultFn and windowEnabled to isWindowEnabled (or use hasWindowEnabled if
more semantically appropriate) in the method signature and update all references
to these parameters throughout the method body to maintain consistency with the
coding guidelines that require is... or has... prefixes for boolean names.
In `@core/src/test/java/io/questdb/test/griffin/GroupByTest.java`:
- Around line 1420-1421: Update numeric literals in the GroupByTest.java file to
comply with coding guidelines by adding underscores to separate thousands in all
numbers with 5 or more digits. Specifically, format 859371 as 859_371 and 237288
as 237_288 in the comments and any SQL literals at the specified locations.
Additionally, review and update any other 5+ digit numeric literals at lines
1438-1439 and 1444-1445 with the same underscore formatting pattern.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: d8d4b7bf-0ab4-4b69-b490-ebcbfa3704b1
📒 Files selected for processing (72)
core/src/main/java/io/questdb/cairo/IntervalBwdPartitionFrameCursor.javacore/src/main/java/io/questdb/cairo/RecordChain.javacore/src/main/java/io/questdb/cairo/idx/AbstractPostingIndexReader.javacore/src/main/java/io/questdb/cairo/idx/PostingGenLookup.javacore/src/main/java/io/questdb/cairo/idx/PostingIndexBwdReader.javacore/src/main/java/io/questdb/cairo/idx/PostingIndexFwdReader.javacore/src/main/java/io/questdb/cairo/sql/PageFrameAddressCache.javacore/src/main/java/io/questdb/griffin/FunctionParser.javacore/src/main/java/io/questdb/griffin/SqlCodeGenerator.javacore/src/main/java/io/questdb/griffin/SqlOptimiser.javacore/src/main/java/io/questdb/griffin/engine/functions/test/TestFaultFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/AbstractWindowFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/AvgDecimalRescaleWindowFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/AvgDecimalWindowFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/CumeDistFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/FirstValueDateWindowFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/FirstValueTimestampWindowFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/FirstValueWindowFunctionFactoryHelper.javacore/src/main/java/io/questdb/griffin/engine/functions/window/LastValueDateWindowFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/LastValueTimestampWindowFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/LastValueWindowFunctionFactoryHelper.javacore/src/main/java/io/questdb/griffin/engine/functions/window/LeadDecimalFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/MaxDateWindowFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/MaxMinWindowFunctionFactoryHelper.javacore/src/main/java/io/questdb/griffin/engine/functions/window/MaxTimestampWindowFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/MinDateWindowFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/MinTimestampWindowFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/NthValueDateWindowFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/NthValueTimestampWindowFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/NthValueWindowFunctionFactoryHelper.javacore/src/main/java/io/questdb/griffin/engine/functions/window/PercentRankFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/RankFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/functions/window/SumDecimalWindowFunctionFactory.javacore/src/main/java/io/questdb/griffin/engine/groupby/GroupByNotKeyedRecordCursorFactory.javacore/src/main/java/io/questdb/griffin/engine/groupby/SampleByInterpolateRecordCursorFactory.javacore/src/main/java/io/questdb/griffin/engine/groupby/vect/GroupByRecordCursorFactory.javacore/src/main/java/io/questdb/griffin/engine/join/AsOfJoinFastRecordCursorFactory.javacore/src/main/java/io/questdb/griffin/engine/join/FilteredAsOfJoinFastRecordCursorFactory.javacore/src/main/java/io/questdb/griffin/engine/orderby/LimitedSizeLongTreeChain.javacore/src/main/java/io/questdb/griffin/engine/table/AsyncHorizonJoinAtom.javacore/src/main/java/io/questdb/griffin/engine/table/AsyncMultiHorizonJoinAtom.javacore/src/main/java/io/questdb/griffin/engine/table/HeapRowCursorFactory.javacore/src/main/java/io/questdb/jit/CompiledFilterIRSerializer.javacore/src/main/java/io/questdb/std/Decimal256.javacore/src/test/java/io/questdb/test/cairo/PostingIndexReaderOomTest.javacore/src/test/java/io/questdb/test/cairo/covering/CoveringIndexSidecarFaultTest.javacore/src/test/java/io/questdb/test/cairo/fuzz/FailureFileFacade.javacore/src/test/java/io/questdb/test/griffin/CompiledFilterRegressionTest.javacore/src/test/java/io/questdb/test/griffin/GroupByTest.javacore/src/test/java/io/questdb/test/griffin/engine/groupby/SampleByInterpolateOomTest.javacore/src/test/java/io/questdb/test/griffin/engine/groupby/vect/GroupByVectorizedOomTest.javacore/src/test/java/io/questdb/test/griffin/engine/groupby/vect/GroupByVectorizedRostiAccountingTest.javacore/src/test/java/io/questdb/test/griffin/engine/join/AsOfJoinFastOomTest.javacore/src/test/java/io/questdb/test/griffin/engine/table/PageFrameRecordCursorImplFactoryTest.javacore/src/test/java/io/questdb/test/griffin/engine/table/parquet/ParquetScanOomTest.javacore/src/test/java/io/questdb/test/griffin/engine/window/WindowDateFunctionTest.javacore/src/test/java/io/questdb/test/griffin/engine/window/WindowDecimalFunctionTest.javacore/src/test/java/io/questdb/test/griffin/engine/window/WindowFunctionTest.javacore/src/test/java/io/questdb/test/griffin/fuzz/EarlyExitGroupByDetectionTest.javacore/src/test/java/io/questdb/test/griffin/fuzz/FaultType.javacore/src/test/java/io/questdb/test/griffin/fuzz/FuzzConfig.javacore/src/test/java/io/questdb/test/griffin/fuzz/PredicateGenerator.javacore/src/test/java/io/questdb/test/griffin/fuzz/QueryFuzzTest.javacore/src/test/java/io/questdb/test/griffin/fuzz/QueryGenerator.javacore/src/test/java/io/questdb/test/griffin/fuzz/QueryRunner.javacore/src/test/java/io/questdb/test/griffin/fuzz/clauses/GroupByClause.javacore/src/test/java/io/questdb/test/griffin/fuzz/clauses/SampleByClause.javacore/src/test/java/io/questdb/test/griffin/fuzz/clauses/SimpleClause.javacore/src/test/java/io/questdb/test/griffin/fuzz/clauses/TemporalJoinClause.javacore/src/test/java/io/questdb/test/griffin/fuzz/clauses/WindowClause.javacore/src/test/java/io/questdb/test/jit/CompiledFilterIRSerializerTest.javacore/src/test/java/io/questdb/test/std/Decimal256Test.java
💤 Files with no reviewable changes (1)
- core/src/main/java/io/questdb/griffin/engine/functions/window/LeadDecimalFunctionFactory.java
Apply four cleanups raised in the PR #7217 automated review. Hoist the window value-function argument type check out of the per-row path. readArgValue() re-derived ColumnType.tagOf(arg.getType()) for every row at ~60 call sites, though the DATE vs TIMESTAMP choice is invariant per instance. BaseWindowFunction now caches it as argIsDate in its constructor and exposes an instance readArgValue(record). Replace Decimal256.uncheckedSubtract's negate-then-add with a direct subtract-with-borrow that mirrors uncheckedAdd's carry chain, halving the limb arithmetic. Decimal256Test and the window sliding-frame eviction tests cover it. Drop the stale FuzzConfig comment claiming FILE / MALLOC faults stay serial: all three fault types now run under parallel SQL execution (commit "Run FILE and MALLOC fuzzer faults in parallel"). Fold the per-test DDL into .ddl(CREATE, INSERT) across the 22 WindowDateFunctionTest cases and drop the redundant assertMemoryLeak wrapper plus .noLeakCheck(); the assertQuery builder leak-checks by default. The review's one Moderate code finding (drop the swallow-oracle SQL text "limit" arm) was rejected: an offset-limit filtered query (LIMIT lo,hi) terminates early through an outer LimitRecordCursorFactory with no pushed-limit plan marker, so dropping the arm would make the fuzzer report a false-positive swallowed fault. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
openSidecarFilesIfPresent() now returns early when metadata is null. A rowid-only reader (built via the null-metadata convenience constructor) cannot map writer indices to dense columns, so it serves no covered reads and has no sidecar to set up. Previously the method swallowed the resulting NullPointerException from denseIndexFromWriter() and degraded to coverCount = 0. After this branch made the catch propagate (correct for production, where a covered read has no base-table fallback), that latent NPE began failing the constructor for rowid-only readers. The early return restores the coverCount = 0 state for those readers without re-swallowing genuine sidecar-open failures. In production metadata is never null: IndexFactory and TableReader always pass real metadata. The null-metadata path is test-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Code review (level 3, independent re-pass)Full mission-critical pass: change-surface map, 9 scoped review agents plus the two mandatory adversarial agents (fresh-context, adversarial-performance), and per-finding source verification against this branch. Scoped against CriticalNone. ModerateM1 — Fuzzer swallow-oracle over-tolerates faults on aggregate-with- boolean discardedEarly = parallel
&& (Chars.containsLowerCase(sql, "limit") // <-- crude SQL-text match
|| outcome.hasPushedLimit
|| outcome.hasEarlyExitGroupBy);Independently re-confirmed. The Fix: drop the Minor
PR metadata
Downgraded (verified false positives / resolved)
Verified correct (highlights)The production correctness work checks out, and several items fix latent
Out-of-scope follow-up (pre-existing, not this PR)
Summary
🤖 Generated with Claude Code |
The CI "List of required changes" step runs the IntelliJ formatter over the checkout and fails when "git diff --exit-code" finds deviations. The gate was red, which skipped every downstream Rust and test step. Bring continuation-line indentation into line with the formatter in the files it flagged: - SqlException.parserErr(): indent the chained .put(...) calls - ConcurrentHashMap / ConcurrentIntHashMap / ConcurrentLongHashMap: indent the chained .newInstance(...) call in toArray() - WhereClauseParserTest.replaceTimestampSuffix(): de-indent the chained .replaceAll(...) calls No behavior change; formatting only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first_value/last_value DATE and TIMESTAMP window factories delegate into FirstValueWindowFunctionFactoryHelper and LastValueWindowFunctionFactoryHelper, which hold one base class per frame mode (whole result set, current row, bounded/unbounded ROWS, bounded RANGE, partitioned variants, and the IGNORE NULLS counterparts). The existing tests only reached the whole-set, current-row and unbounded paths, leaving the bounded ROWS/RANGE and IGNORE NULLS base classes unexercised. The TIMESTAMP factories had almost no coverage at all. Extend WindowDateFunctionTest from 22 to 45 cases, adding bounded ROWS and RANGE frames (partitioned and not, RESPECT and IGNORE NULLS), the CURRENT ROW frame, and whole-set/whole-partition IGNORE NULLS. Add WindowTimestampFunctionTest as the TIMESTAMP mirror; the logical results match the DATE cases and only the rendered precision differs. Each expected result is computed from window-frame semantics on small, hand-traceable datasets (5-6 rows). Measured against the helper and factory sources, this raises covered changed-lines on these files by about 1,055 (first/last value helpers from ~11% to ~60%, the date and timestamp factories from the single digits to 55-90%). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses review item M1 on PR #7217. The fault-injection swallow oracle relaxed its "fault must surface" check for any parallel query whose SQL text contained "limit". That over-tolerated a swallowed fault for a LIMIT over a row-count-changing aggregate (count()/sum()/avg()/GROUP BY ... LIMIT N): such a query pushes no scan limit, the async filter still scans every frame, and the aggregate drains all of them, so a fired fault must surface. A real swallow in exactly that bug class would have been recorded as a tolerated skip. Simply dropping the SQL-text arm (as the review suggested) trades the false negative for false positives. A streaming consumer under a LIMIT -- a window function or a bare projection -- stops pulling once the limit is met, so under the eager parallel reduce a fault on a frame past the cutoff is legitimately discarded, the same early termination a pushed LIMIT gives. But a window/projection between the scan and the LIMIT stops the optimiser rendering the "limit: N" scan marker, so planHasPushedLimit does not fire. Running the fuzzer with the text arm removed surfaced two such window-over-async-filter queries as spurious swallow failures. Keep the text-LIMIT tolerance but gate it with a new factoryHasBlockingAggregation signal that walks the consumption spine for a count(*), keyed or non-keyed GROUP BY (serial or async). Those drain every frame regardless of a downstream LIMIT, so a swallow there is always a real defect; a streaming consumer carries no such operator and stays tolerated. The check errs toward not blocking: an aggregation factory it fails to name leaves the original (over- tolerant) behaviour, a missed swallow rather than a false failure. Add EarlyExitGroupByDetectionTest.testBlockingAggregationDetection to pin the factory classification, and extend the pushed-limit pins in QueryFuzzTest with the aggregate-with-LIMIT shapes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
/azp run macwin |
|
Azure Pipelines successfully started running 1 pipeline(s). |
bluestreak01
left a comment
There was a problem hiding this comment.
Approved after a level-3 mission-critical review (full change-surface map + parallel correctness/resource/test/adversarial/performance passes + per-finding source verification).
All 17 core engine fixes traced to source and confirmed correct:
- #5 IntervalBwd calculateSize() is now a faithful mirror of next() (no extra lookups, cursor state untouched).
- #12 cached-window RANGE timestamp index correctly scoped — streaming path (configureWindowContext @9519) left unchanged, no regression.
- #13 Decimal256.uncheckedSubtract is an exact limb-order mirror of uncheckedAdd (carry↔borrow, two-step borrow cannot double-count).
- #14 null-safe close()s verified complete (root via super.close(), value heap via reopen(), cursor is inline-final/never-null).
- #15 JIT narrow-int-under-long-with-float widening correctly mirrors C++ convert(); shouldWiden() && !hasFloat() prevents double-widening.
- #7/#10 rank streaming compacted-ORDER-BY map sound under the assert-guarded cache-by-value invariant; key-types snapshot fixes the reused-buffer aliasing.
- #11 untyped-NULL rejection follows the existing freeObjList-before-throw pattern (no double-free).
- Resource fixes #1/#3/#4/#16/#19/#21/#22/#24 — all cleanup paths idempotent and double-free-free; runWhatsLeft cannot deadlock (latch counted down in finally even on throw).
- #6/#8 DATE window functions (ms vs ticks scale) correct; ZERO_PASS removal aligns wide-decimal lead with siblings.
Tests: 0 assertion-API violations (0 .returnsOnce, 0 banned assertSql), OOM sweeps net-leak-check the whole RSS sweep and assert the trip fired, window tests cover NULL/DATE-scale/mixed-keys and pin streaming-vs-cached plans.
No Critical or production-correctness defects. Non-blocking follow-ups:
- Moderate (test-oracle only): the re-pass ImplicitCastException/NumericException tolerance in QueryRunner.checkToTop/checkCalculateSize keys on exception class alone — consider gating it on the existing hasPushedLimit flag so the relaxation matches its 'parallel-filter LIMIT non-determinism' justification and doesn't blunt the new self-consistency checks.
- Minor: thousands separators in new SQL literals (2000000000, 859371, -237288); optional memoization of the compile-time arithExprType/markI64Widen walk.
LGTM.
Brings in 10 upstream commits, including PR #7184 (per-query memory limits), and advances the java-questdb-client submodule to a02732c3d3. Conflict resolutions: - RecordChain.java: kept both the MemoryTracker import (from #7184) and the Misc import (from this branch); both types are used. - AsOfJoinFastRecordCursorFactory / FilteredAsOfJoinFastRecordCursorFactory: both branches independently fixed the same bug - a sink heap allocation that trips the RSS limit during reopen() must free already-open sinks exactly once. Took upstream's canonical fix (reopen the sinks before super.of() adopts the cursors, letting the normal close() path free them) and dropped this branch's now-redundant post-of() try/catch, which would otherwise have reopened the already-open sinks a second time. - First/Last/Max/Nth Value Timestamp window factories: this branch had refactored their implementations into shared helper classes, while #7184 added per-query memory tracking to the old monolithic factories. Kept the refactored thin factories and ported #7184's tracking into the four shared helpers (FirstValue/LastValue/Nth/MaxMin): each range-frame base now binds its value ring buffer (and MaxMin's deque buffer) to the query's MemoryTracker via setMemoryTracker(), and defers the ring-buffer allocation from the constructor to reopen() so it is charged under the bound tracker. Mirrors #7184 exactly; the per-partition map binding already comes from the merged BasePartitionedWindowFunction. min() inherits the treatment through the shared MaxMin helper. ROWS-frame ring buffers remain global-only, the same documented coverage limitation as upstream. Validation: core main + test sources compile; WindowMemoryTrackerTest (21), WindowFunctionTest (633), AsOfJoinFastOomTest (2) and RecordChainMemoryTrackerTest (12) all pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
/azp run macwin |
|
Azure Pipelines successfully started running 1 pipeline(s). |
[PR Coverage check]😍 pass : 3029 / 4134 (73.27%) file detail
|
Brings in 11 master commits, including #7184 (per-query memory limits / MemoryTracker on window functions) and #7217 (query-engine hardening that refactored the TIMESTAMP window functions - first_value/last_value/max/min/ nth_value - into shared helper bases and added new DATE window functions). Conflict resolution: - Mechanical (MemoryTag, WindowFunction, SqlExecutionContext[+Impl], the Double/Long/Decimal window factories): kept this branch's live-view snapshot/restore work and layered master's setMemoryTracker overrides into their alphabetical slot. - RankFunctionFactory: adopted master's compacted-ORDER-BY MapValue layout (which also fixes a latent compile crash on var-size ORDER BY columns) and grafted the live-view snapshot/tombstone/eviction machinery on top. computeNext keeps master's setLeft(mapValue)+compare(mapValue) flow plus this branch's count==0 "uninitialized" reset signal for the ANCHOR path. - The five timestamp window-function factories now use master's thin helper-delegating versions; master's new DATE factories and the four helper classes merge in cleanly. Known pending (tracked as a follow-up on this branch): snapshot support has not yet been re-ported onto the four window-function helpers, so timestamp and date window functions are currently rejected at live-view CREATE (safe, not wrong output). core main sources compile; this commit is a restore point before the re-port lands.
The origin/master merge (#7184 lazy per-partition maps + per-query MemoryTracker, #7217 window-fn helper refactor) silently broke the entire window-function live-view refresh and restore surface, not just the timestamp/date CREATE-reject the handoff described. Fixes: - WindowRecordCursorFactory.ofIncremental self-bootstraps when !isOpen. The LV refresh only calls getIncrementalCursor(); with the lazy-map isOpen=false it tripped "incremental cursor requires a prior bootstrap" and produced empty output for every window-function live view. - BasePartitionedWindowFunction.onSnapshotRestoreBegin reopens the lazy map before clearing. On a fresh restart the restore ran before any cursor reopened the map, so createValue() crashed the JVM (SIGSEGV). - New WindowRecordCursorFactory.openForLiveViewRestore binds the tracker, reopens the maps and marks the cursor open before restore writes state, so the first post-restore refresh preserves it (fixes row_number() off-by-one after restart restore). Re-port: max/min over TIMESTAMP and DATE work in live views again. Migrated the three partitioned ZERO_PASS bases of the shared MaxMin helper to the snapshot contract; added an UnboundedPartitionConstructor so the unbounded shape no longer shares the whole-partition (TWO_PASS) constructor. The unbounded shape re-anchors via the null sentinel, so the non-LV layout is unchanged. Min reuses Max's subclasses. Also fixes a deque-buffer leak in the MaxMin ROWS close() (the LV close path runs close() without reset()), and removes a duplicate testReplaceRangeAddsPartitionsAboveLastThenRebuilds in WalWriterReplaceRangeTest (the branch's finding-2 test was upstreamed as master's #7289; the merge kept both copies, breaking test compilation). Validated: full LiveViewSmokeTest 307 run / 0 failures / 4 errors (the 4 are the still-un-migrated NthValue timestamp tests that reject at CREATE); LiveViewTest/Checkpoint/InMemoryTier and WindowTimestamp/DateFunctionTest all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-add the live-view snapshot/restore contract to the shared NthValueWindowFunctionFactoryHelper bases so nth_value() over TIMESTAMP and DATE arguments works in live views again after the origin/master #7217 refactor moved these functions onto shared helper bases. This clears the four red testNthValueTimestampOver*SnapshotRoundTrip tests; LiveViewSmokeTest is now fully green. Migrate the four partitioned ZERO_PASS bases to the snapshot contract: - NthValueOverPartitionRangeFrameBase (RANGE, tombstone slot 5) - NthValueOverPartitionRowsFrameBase (bounded ROWS, tombstone slot 3) - NthValueOverPartitionRowsFrameUnboundedBase (ROWS UNBOUNDED PRECEDING AND K PRECEDING, tombstone slot 2, state [count, lockedValue]) - NthValueOverUnboundedPartitionFrameBase (anchored UNBOUNDED PRECEDING AND CURRENT ROW, tombstone slot 2, state [value, count]; the only one that participates in frontier compaction via newCompactionScratch and re-anchors through a count==0 sentinel in computeNext) Each base gains liveView/keyColumnTypes/mapValueTypes fields, sets tombstoneValueIndex in its constructor, writes the tombstone slot in computeNext, and adds getPartitionMap, getSnapshotKey{ColumnTypes, StartIndex}, resetPartition, restorePartitionState, snapshotFormat/ MinSupportedVersion, snapshotPartitionState and supportsSnapshot. The ring-buffer bases also override onSnapshotRestoreBegin to truncate the native arena. Add six new static map layouts and thread partitionByKeyTypes + liveView (+ configuration for the unbounded shape) through the four functional-interface constructors, the newInstance call sites, and the thin TIMESTAMP and DATE subclasses. There is no decimal nth_value to crib from, so the reference is the pre-merge per-factory diff plus the in-tree, passing NthValueDoubleWindowFunctionFactory twin, mapped onto the helper's readArgValue/Numbers.LONG_NULL long domain (TIMESTAMP and DATE both store an 8-byte long, so serialization is identical for the pair). The non-LV path stays pristine: tombstoneValueIndex is -1 and the tombstone branches are inert. Validation: LiveViewSmokeTest 307/0/0, LiveViewTest 25, LiveViewCheckpointTest 12, LiveViewInMemoryTierTest 9, WindowTimestampFunctionTest 43, WindowDateFunctionTest 45, WindowFunctionTest 633 (non-LV nth_value paths), LiveViewFuzzTest 5, all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-add the live-view snapshot/restore contract to the shared FirstValueWindowFunctionFactoryHelper bases so first_value() over TIMESTAMP and DATE arguments works in live views again after the origin/master #7217 refactor moved these functions onto shared helper bases. Migrate the three partitioned ZERO_PASS RESPECT bases and their IGNORE-NULLS (FirstNotNull) subclasses: - FirstValueOverPartitionRangeFrameBase (RANGE, tombstone slot 5) and FirstNotNullValueOverPartitionRangeFrameBase - FirstValueOverPartitionRowsFrameBase (bounded ROWS, tombstone slot 3) and FirstNotNullValueOverPartitionRowsFrameBase (tombstone slot 4) - FirstValueOverUnboundedPartitionRowsFrameBase (anchored UNBOUNDED PRECEDING AND CURRENT ROW, tombstone slot 2, [value, initialized] state, the only one with newCompactionScratch) and FirstNotNullValueOverUnboundedPartitionRows- FrameBase Each base gains liveView/keyColumnTypes/mapValueTypes fields, sets tombstoneValueIndex in its constructor, writes the tombstone slot in computeNext, and adds getPartitionMap, getSnapshotKey{ColumnTypes,StartIndex}, resetPartition, restorePartitionState, snapshotFormat/MinSupportedVersion, snapshotPartitionState and supportsSnapshot. The ring-buffer bases override onSnapshotRestoreBegin to truncate the native arena. Add eight new static map layouts and thread partitionByKeyTypes + liveView (+ configuration for the unbounded shape) through the helper's PartitionRange/PartitionRows functional interfaces, a new UnboundedPartitionConstructor (so the unbounded shape no longer shares PartitionConstructor with the un-migrated whole-partition TWO_PASS shape), the newInstance call sites, and the thin TIMESTAMP and DATE subclasses in both factories. The in-tree FirstValueDecimalWindowFunctionFactory decimal64 bases were the exact structural twin (decimal64 is a raw long), translated arg.getDecimal64 to readArgValue and Decimals.DECIMAL64_NULL to Numbers.LONG_NULL. Two serialization decisions follow that twin rather than the pre-merge per-factory timestamp diff: - The IGNORE NULLS RANGE base serializes the full physical ring and preserves firstIdx. Its unbounded-lo fast path uses firstIdx as a 0/1 capture flag with the value pinned at physical index 0, so the pre-merge timestamp factory's logical-from-firstIdx serialization was a latent bug for that shape. The RESPECT RANGE base keeps the logical serialization (rebases firstIdx to 0). - The IGNORE NULLS unbounded path keeps the initialized byte at 0 until the first non-null and recaptures after each anchor reset, fixing the recapture gap the double/long FirstNotNull reference still has. The non-LV map layouts stay pristine; computeNext branches on liveView (decimal-family convention) instead of mutating the base layout. The RESPECT ROWS base gained a close() override (super.close() + memory.close()): the helper base lacked one and the live-view close path runs close() without reset(), so the ring memory leaked. Tests: eight new LiveViewSmokeTest cases - six byte-exact snapshot round-trips covering all three shapes for both RESPECT and IGNORE NULLS, plus two anchor-reset correctness tests (RESPECT and IGNORE NULLS, the latter with a leading null per day). LiveViewSmokeTest 315/0/0; LiveViewTest, LiveViewCheckpointTest, LiveViewInMemoryTierTest, WindowTimestampFunctionTest, WindowDateFunctionTest, WindowFunctionTest and LiveViewFuzzTest all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-add the live-view snapshot/restore contract to the shared LastValueWindowFunctionFactoryHelper bases so last_value() over TIMESTAMP and DATE arguments works in live views again after the origin/master #7217 refactor moved these functions onto shared helper bases. This is the last of the four #7217 helper families to be re-ported. Unlike first_value, last_value has only two migratable partitioned ZERO_PASS shapes (RANGE and ROWS), each in a RESPECT and an IGNORE-NULLS flavour: - LastValueOverPartitionRangeFrameBase (RESPECT, tombstone slot 4) and LastNotNullValueOverPartitionRangeFrameBase (IGNORE) which extends the RESPECT base and inherits its whole snapshot surface, only threading liveView to super and writing the tombstone in its own computeNext. last_value RANGE has no unbounded-lo capture-flag fast path, so the IGNORE variant serializes logically via the inherited parent method - no firstIdx-flag trap, unlike first_value's IGNORE RANGE. - LastValueOverPartitionRowsFrameBase (RESPECT) and LastNotNullValueOverPartitionRowsFrameBase (IGNORE), both tombstone slot 3. last_value has no migratable partitioned unbounded-preceding shape: the UNBOUNDED PRECEDING AND CURRENT ROW frame's last element is the current row, so RESPECT routes to the lightweight (un-migrated) IncludeCurrent shape and the IGNORE unbounded base was never migrated pre-merge either. An ANCHORed last_value therefore routes to IncludeCurrent, so these frame bases are not reachable via an ANCHOR - matching the pre-merge behaviour. Each base gains liveView/keyColumnTypes/mapValueTypes fields, sets tombstoneValueIndex in its constructor, writes the tombstone slot in computeNext, and adds getPartitionMap, getSnapshotKey{ColumnTypes, StartIndex}, resetPartition, restorePartitionState, snapshotFormat/ MinSupportedVersion, snapshotPartitionState, supportsSnapshot and onSnapshotRestoreBegin (truncate the native ring). Add four static _LV map layouts (existing slots plus a trailing tombstone BYTE) and thread partitionByKeyTypes + liveView through the helper's PartitionRange and PartitionRows functional interfaces, the newInstance call sites, and the thin TIMESTAMP and DATE subclasses in both factories. Both ROWS bases gained a close() override (super.close() + memory.close()): the helper base lacked one and the live-view close path runs close() without reset(), so the ring memory leaked - the same leak the MaxMin and FirstValue ROWS bases hit. The non-LV map layouts stay pristine and the new branches gate on liveView (the decimal-family convention). The leaner helper style drops the pre-merge factory's frontier-compaction freeList (no newCompactionScratch). The reference was the pre-merge per-factory diff (git diff 3b6d3ed..9b1011e -- LastValueTimestampWindowFunctionFactory.java: three migrated classes - RANGE RESPECT, ROWS RESPECT, ROWS IGNORE - with RANGE IGNORE inheriting) plus the in-tree FirstValue helper twin for the helper mechanics. Tests: six new LiveViewSmokeTest cases via the new assertLastValueTimestampDateFrameRoundTrip helper, covering both shapes for both RESPECT and IGNORE NULLS over TIMESTAMP and DATE. RESPECT tests use a frame ending strictly before the current row to reach the partition bases; IGNORE tests use ... AND CURRENT ROW and skip a NULL. Each asserts the windowed value is correct (proving CREATE-accept) then a byte-exact write/toTop/restore/write round-trip of the partition state. LiveViewSmokeTest 321/0/0; LiveViewTest, LiveViewCheckpointTest, LiveViewInMemoryTierTest, WindowTimestampFunctionTest, WindowDateFunctionTest, WindowFunctionTest and LiveViewFuzzTest all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
What this does
Three rounds of enhancements to the seeded query fuzzer (
QueryFuzzTest).Cursor self-consistency checks (on by default). After materializing each result, the runner verifies invariants that hold for any
RecordCursorregardless of access path:toTop()reproduces the result set;toTop()leavespreComputedStateSize()unchanged;size()andcalculateSize()agree with the materialized row count.A divergent row set, a changed
preComputedStateSize(), or a count mismatch is always a real defect and fails the query. The one skip valve is narrow (see bug fix #17): when the clean first pass is followed by a re-pass that throws a per-row cast or numeric overflow the first pass did not, that is accepted parallel-filter non-determinism, not a cursor-reuse defect, and the query is skipped.Fault injection (on by default,
-Dquestdb.fuzz.faults). On a fraction of queries (-Dquestdb.fuzz.fault.pct, default 15) one fault is armed and the runner applies a crash-and-recover oracle instead of the differential one:FILE— one filesystem op fails once (FailureFileFacade);MALLOC— a native allocation trips the RSS memory limit (the real out-of-memory path);FUNCTION— a dev-modetest_fault()woven into the query throws once mid-evaluation.For each, it asserts the factory frees its resources on the error path (native memory and file descriptors return to baseline) and that the same query runs cleanly once the fault is removed. Fault decisions are drawn from the seeded rnd so failures replay deterministically. By default the fault query runs under parallel SQL execution (
-Dquestdb.fuzz.fault.parallel, default true), so the parallel filter / GROUP BY / top-K reduce error paths are exercised; the writer pool is halted for the duration of the query loop so no background job competes with an armedFILE/MALLOCfault (FILEis scoped to the test thread plus the query-pool workers,MALLOC's process-global RSS ceiling is tripped only by the query's own allocations). Pass-Dquestdb.fuzz.fault.parallel=falseto run faults serially.Window-function shapes (on by default,
-Dquestdb.fuzz.window). A newWindowClausegeneratesfn(...) OVER (PARTITION BY ... ORDER BY ts [frame])projections across the ranking family (row_number,rank,dense_rank,cume_dist,percent_rank,ntile), the windowed aggregates (avg,sum,min,max,count) and the navigation functions (first_value,last_value,lag,lead,nth_value), withROWSand time-basedRANGEframes. Every window orders by the strictly-increasing designated timestamp, so the window values are reproducible across runs and the differential oracle compares full result sets rather than just row counts. TheWINDOWband is carved from the upper half of theSIMPLErange, so the other shapes' frequencies are unchanged; pass-Dquestdb.fuzz.window=falseto drop window shapes.Status
Work in progress. Fault injection and the window-function shapes are doing their job: they surface real defects, so the fuzzer goes red on the seeds that hit them. Fixes land on this branch, each in its own commit with a regression test.
Bug fixes
SAMPLE BY FILL(LINEAR) cleanup on out-of-memory.
SampleByInterpolateRecordCursorFactoryallocates its record-key map, data map and group-by allocator in the cursor constructor. When one of those native allocations failed, the constructor's error path calledclose()before the remaining fields were assigned, andclose()dereferenced them directly, throwingNullPointerExceptionthat masked the real out-of-memory error.close()now frees the fields through the null-safeMisc.free().Parquet page-frame buffer leak on out-of-memory (production fix landed upstream in feat(sql): add a memory-budgeted cache for Parquet random access #7230).
PageFrameMemoryPoolremoved a per-frame buffer from its free list and then calledreopen(), which allocates the buffer's page-address lists one by one. If that allocation tripped the RSS limit, the buffer was already off the free list but not yet recorded in the cache, so the pool could no longer reach it on close and the lists it had allocated leaked. The fuzzer's malloc fault injection surfaced this. The fix landed independently in master through feat(sql): add a memory-budgeted cache for Parquet random access #7230, which rewrotePageFrameMemoryPoolaround a memory-budgeted decode-buffer cache and guards a failedreopen()by closing the buffer on the error path; after merging master this branch carries noPageFrameMemoryPoolchange of its own. The regression test stays:ParquetScanOomTestsweeps the RSS ceiling across the buffer reopen and asserts the scan leaves no native memory behind, now exercising the upstream guard through the same fault-injection path that first surfaced the leak.Covering posting index crash on a sidecar I/O error.
AbstractPostingIndexReaderswallowed any error raised while opening a covering index's sidecar files, cleared the sidecar memories, and carried on. Because the covering plan is fixed at compile time and the covered read path has no base-table fallback, a covered read then walked an empty sidecar and threwindex out of bounds, 0 >= 0(an out-of-bounds read with assertions disabled), masking the real I/O error. The reader now propagates the failure so the query fails with the underlying error and the reader is closed.Posting index reader buffer leak on out-of-memory.
PostingGenLookupallocated its two native cache buffers as inline field initializers. If the second allocation tripped the RSS limit, the first leaked, because the half-built lookup was never assigned to the reader that was constructing it, so the reader'sclose()could not reach it. The allocations now live in an exception-safe constructor that frees the partial result on failure.Wrong
calculateSize()for a backward interval scan. Surfaced by the cursor self-consistency check rather than fault injection. For a descending scan over a half-open timestamp range (e.g.WHERE ts < X ORDER BY ts DESC),IntervalBwdPartitionFrameCursor.calculateSize()diverged from the cursor's ownnext(). With an open lower bound it shortcut the start index to zero instead of the-1sentinel its whole-partition check relied on, which both undercounted the newest partition by one and stopped the walk after that partition, ignoring every earlier one in range; a two-partition table that iterates to 96 rows reported a size of 47.calculateSize()now mirrorsnext(), so the counted size matches the iterated row count. Closed and open-ended intervals, and the forward cursor (which never had the open-bound shortcut), were already correct. The existing interval frame-cursor tests only ran the forward direction over closed-lower-bound intervals, so they missed it; the battery now runs every shape in both directions and adds open-start and open-end cases. Upstream fix(core): fix Parquet crashes on malformed files and data corruption on partial reads #7223 independently fixed the same bug with a minimal one-line patch that keeps the old structure, changing the open-bound start index from0to the-1sentinel in place. The two fixes produce identical row counts; merging master resolves the overlap by keeping this branch's mirror-next()rewrite, which subsumes the upstream patch by removing the structural divergence betweencalculateSize()andnext()that caused the bug in the first place.DATE-argument window value functions threw
UnsupportedOperationExceptionon read.max/min/first_value/last_value/nth_value/lag/leadover a DATE argument were backed by the timestamp window function (WindowTimestampFunction), whosegetType()returned the argument type (DATE) but which supplied nogetDate()accessor, so reading the result hit the default-throwingWindowFunction.getDate; the value is stored in timestamp ticks while DATE is milliseconds, so a naive passthrough accessor would also have read ~1000x off. Each value function now splits into a type-agnostic base holding the framing logic plus thin DATE and TIMESTAMP subclasses: the DATE subclass stores and returns the value in milliseconds directly throughgetDate(), and the TIMESTAMP subclass caches the timestamp driver once and converts ticks to milliseconds (lag/leadalready followed this shape). With every timestamp subclass now supplying its owngetDate(), the type-dispatchinggetDate()default onWindowTimestampFunctionis removed and the shared empty-frame null function returns its stored NULL directly.WindowDateFunctionTestcovers all seven functions over DATE across running, partition, range, rows and whole-result-set frames, plus casting a TIMESTAMP window result to DATE. Repro (now passing):SELECT max(d) OVER (ORDER BY ts) FROM twithd DATE.rank()/dense_rank()crash at compile time when a pass-through column of an unserializable type is also projected. In the streamingWindowRecordCursorFactorypath (the window order matches the designated timestamp),RankOverPartitionFunctionremembered each partition's previous row by copying the whole projected row into aMapValue, building the copier withRecordValueSinkFactoryover every projected column. Any pass-through column theMapValuecannot hold (UUID, STRING, VARCHAR, BINARY, LONG256, array, INTERVAL) threwUnsupportedOperationExceptionat compile time, so a plainSELECT u, dense_rank() OVER (PARTITION BY g ORDER BY ts) FROM tfailed even though the rank itself was fine; projecting only the window function took the cached path and worked, so the trigger was the pass-through column. Only the ORDER BY columns decide whether two consecutive rows are peers, so theMapValuenow stores just those (compacted) plus the running rank and count, and the comparator and rank maps are rebuilt over a matching compacted metadata so they read back only those columns.computeNextcaches the previous row's ORDER BY values, overwrites the slots with the current row, then compares, so the comparator never touches the live record.WindowFunctionTest.testRankWithUnserializablePassThroughColumncovers UUID / STRING / VARCHAR / LONG256 / DOUBLE[] pass-through columns ascending and descending, plus a BINARY drain, and pins the streaming plan. Repro (now passing):SELECT u, dense_rank() OVER (PARTITION BY g ORDER BY ts) FROM twithu UUID.lead()over a high-precision DECIMAL crashed withUnsupportedOperationExceptionon read.leadover a Decimal128 or Decimal256 argument reportedZERO_PASS, unlike every otherleadvariant (Double, Long, and the narrower decimals all inheritONE_PASS). When a query's only window functions were wide-decimalleads ordered by the designated timestamp, that hint routed it through the streamingWindowRecordCursorFactory, where the cursor reads each result back through the function's own accessor. Butleadreads ahead, so it cannot run in the forward streaming pass (it is computed in the cached executor with a backwardpass1scan), and the wide-decimalleadexposed nogetDecimal128/getDecimal256accessor there, so the read fell through to the default-throwingWindowFunctionaccessor. Earlier all-types tests masked this because a narrow-decimal column in the same projection forced the cached path. Removing the erroneousgetPassCount()override lets both over-partition functions inheritONE_PASSlike the rest, keeping the planner on the cached path wherepass1already does the work and the record chain reads the value back.WindowDecimalFunctionTest.testLeadWideDecimalOverPartitionStreamingShapecovers Decimal128 and Decimal256leadover a partition, alone and together, and pins the cached plan; because this suite randomizescairo.sql.window.cached.light.enabledper run, the pinned plan label is config-aware (CachedWindoworCachedWindowLight), which a hardcodedCachedWindowlabel had got wrong on the cache-light half of the seeds. Repro (now passing):SELECT lead(d, 5) OVER (PARTITION BY g ORDER BY ts) FROM twithd DECIMAL(76,3).sum/avgover a high-precision DECIMAL raising an overflow is a genuine arithmetic limit, not an engine defect (fuzzer over-reported it). Asumoravgover a Decimal256-backed DECIMAL (precision > 38) raisesCairoException: ... aggregation failed: an overflow occurredonce the running total of values near the DECIMAL(76,3) maximum exceeds Decimal256's 256-bit capacity. Triage: the sum is genuinely unrepresentable (Decimal256 is the widest decimal type, so there is no wider accumulator), the plain group-bysum/avgoverflows identically on the same data, andWindowDecimalFunctionTestalready pins this overflow as the expected behaviour. The fuzzer's crash-only oracle was flagging it because it treats any leakedCairoExceptionas an internal error.QueryRunner.isAcceptedSkipnow accepts aCairoExceptionwhose message contains both "aggregation failed" and the word "overflow" (case-insensitive), classifying it like aNumericExceptionoverflow. Requiring "overflow" -- not just "aggregation failed" -- keeps the skip narrow: the decimalsum/avgfactories report the overflow in two message shapes, the explicit "an overflow occurred" flag message and a wrapped Decimal256NumericExceptionbeginning "Overflow in ...", and both carry the word; a future non-overflow numeric error wrapped into the same "aggregation failed: ..." text is reported as a failure instead of being silently skipped.WindowDecimalFunctionTest.testAvgDecimal256OverPartitionGenuineOverflow/testSumDecimal256OverPartitionGenuineOverflowpin the window-over-partition and group-by overflow, andQueryFuzzTest.testDecimalAggregationOverflowToleratedByOracledrives the repros through the oracle and asserts the skip while a non-overflowing query still completes. Repro (now tolerated):SELECT avg(d) OVER (PARTITION BY g ORDER BY ts) FROM twithd DECIMAL(76,3).rank()/dense_rank()over a partition built its streaming map from the wrong key types. In the streamingWindowRecordCursorFactorypath the code generator builds each window function's partition map lazily, ininitRecordComparator(), after the whole projection has been walked. It reuses one key-types buffer that it clears and rebuilds for every window column'sPARTITION BY, butRankOverPartitionFunctionkept a live reference to that buffer instead of a copy, so with two or more streaming window functions partitioning by differently typed keys it built its map from the last window column's key types. That either crashed -- the partition sink wrote a type the wrongly picked map could not hold, e.g. a BYTE partition writing into theUnordered4Mapchosen for a SYMBOL key threwUnsupportedOperationException-- or silently mis-keyed the partitions: a Decimal256 partition map built for a two-column key fed by a single-column sink reported rank 1 for every row and could diverge across atoTop()re-read (the previously open bug below). The over-partition functions now snapshot the key types in their constructor through a sharedAbstractWindowFunctionFactory.copyKeyTypes()helper;cume_dist/percent_rankcarry the same idiom and get the same snapshot, though being two-pass they always take the cached path and were not yet exploitable.WindowFunctionTest.testRankOverPartitionMixedPartitionKeyTypescovers rank/dense_rank with mixed BYTE/SYMBOL and wide-DECIMAL partition keys on the streaming path, andtestCumeDistMixedPartitionKeyTypescovers cume_dist/percent_rank with mixed keys on the cached path. Repro (now passing):SELECT b, rank() OVER (PARTITION BY b ORDER BY ts), dense_rank() OVER (PARTITION BY g ORDER BY ts) FROM twithb BYTE.Window value / navigation / aggregate functions over an untyped
nullliteral behaved non-deterministically across platforms -- a compile-time crash, an inconsistent error, or silent acceptance. A window call whose argument is the barenullliteral (e.g.lead(null, 2),min(null)) first surfaced as an internaljava.lang.IllegalArgumentException: Unexpected column type: NULL: the cached window path askedRecordSinkFactoryto build a sink over the NULL output column and leaked it through the query API, while the streaming path tolerated it -- so the same function crashed on one frame and silently returned nulls on another. The deeper cause is in overload resolution: an untypednullties against every typed variant of the function (NULL has zero overload distance to any type), so the fiveleadfactories (Double, Long, Decimal, Date, Timestamp) all match with the same score and the winner is decided by Java's stable sort preserving the classpath scan order, which differs by filesystem and architecture. Each winner behaves differently -- the type-adaptive Timestamp / Date variants yield a NULL output type, the Decimal variant throws its ownlead is not yet implemented for NULL, and the Double / Long variants yield a concrete type and silently accept the query. An initial guard in the code generator rejected only the NULL-output case, so it held on x86-64 (whereleadresolved to the Timestamp factory) buttestNullLiteralArgumentRejectedWithCleanErrorfailed on linux-arm64, where the same query resolved to the Decimal factory and threw the other message.FunctionParsernow rejects an untypednullvalue argument to a window function with more than one overload before any factory runs, so every platform and frame gives the sameSqlExceptionat the function position suggesting a concrete cast (null::doublecompiles and computes over NULL); the two code-generator output-type guards are removed. This makes the rejection uniform:lead,lag,min,max,sum,avg,first_value,last_value,nth_valueandcountover an untypednullare now all rejected the same way -- a behaviour change for the variants that previously resolved to a concrete type and computed over NULL (lag/max/first_value/last_value/avg/sum/count) and fornth_value, which previously reported its own message. Window functions with a single overload keep their own argument validation, sontile(null)still reportsbucket count cannot be NULL, and the no-argument ranking functions (row_number,rank,dense_rank) are unaffected. The cleanSqlExceptionis an accepted skip inQueryRunner.isAcceptedSkip, so the seeds that previously crashed onlead(null)/min(null)now run green.WindowFunctionTest.testNullLiteralArgumentRejectedWithCleanErrorpins the rejection across frames and a partition, the uniformnth_valuemessage, and thenull::doublecast workaround. Repro (now rejected cleanly):SELECT lead(null, 2) OVER (ORDER BY ts) FROM t.Window
RANGE-frame functions returned a wrong, run-varying value when the designated timestamp was not in the projection. A query that mixes window functions ordering differently (e.g.count(...) OVER (... ORDER BY ts DESC)plusmin(...) OVER (... ORDER BY ts ASC RANGE ...)) takes the cachedCachedWindowRecordCursorFactorypath, which materialises the base into a record chain whose columns are reordered -- the window output columns occupy the leading slots and the remaining base columns are appended afterwards. ARANGE-framed function reads the designated timestamp by index, and the code generator passed it the base cursor's timestamp index. When the timestamp was not part of theSELECTprojection it landed at a different index in the chain than in the base, so the function read the wrong chain column -- frequently its own, as-yet-unwritten output slot -- as the timestamp and computed the frame over uninitialised memory. The value came out wrong and varied between consecutive executions of the same SQL because the slot reuses freed native memory; this was the open bug previously listed below. The code generator now tracks where the designated timestamp lands in the chain and passes that index to the cached window functions, leaving the streaming path unchanged.WindowFunctionTest.testRangeFrameTimestampIndexInCachedWindowpins acountordered byts DESCplus aminover aRANGEframe ordered byts ASCwith an unprojected timestamp; before the fix theminframed on a zeroed slot and collapsed to the whole-partition prefix minimum. Repro (now correct): seeds553854821671111/1781174964170,553832024235320/1781174941373and553862616269146/1781174971965.sum()/avg()over a high-precision DECIMAL produced a wrong, run-varying result over a sliding frame.sumoravgover a Decimal256-backed argument running over a sliding frame accumulated values with the raw, scale-agnosticDecimal256.uncheckedAddbut evicted the oldest value as it left the frame through the scale-awareDecimal256.subtract. The running accumulator keeps its scale field at 0 (it holds raw limbs), while a value arriving through a cast such as(c2)::DECIMAL(76, 6)carries scale 6. The first non-null eviction therefore rescaled the accumulator from scale 0 up to the value scale, multiplying the running total by10^scaleand baking a large constant offset into every later row of that frame. The accumulator instance is shared across the cursor's forward passes, so the rescale happened only once: the first pass (scale 0) was wrong and the post-toTop()re-read came out correct, which is how the fuzzer's cursor self-consistency check flagged it. A newDecimal256.uncheckedSubtract, the scale-agnostic counterpart ofuncheckedAdd, now performs the frame eviction in the Decimal256sum,avgand rescaled-avgwindow functions across the rows, partition-rows, range and partition-range frames, so a removed value cancels exactly the limbs thatuncheckedAddcontributed regardless of either operand's scale field; the Decimal128 paths already subtracted raw limbs with an explicit scale of 0 and were unaffected.Decimal256TestcoversuncheckedSubtract(raw subtraction, scale independence, add/subtract round-trip and cross-limb borrow) andWindowDecimalFunctionTest.testSumAvgDecimal256SlidingFrameEvictionmaterializes the raw first pass and thetoTop()pass separately and pins the hand-computed sums and averages. Repro (now correct): seed625297067085445/1781246406416,SELECT sum((c2)::DECIMAL(76, 6)) OVER (PARTITION BY sym ORDER BY ts ASC ROWS BETWEEN 7 PRECEDING AND CURRENT ROW) FROM fuzz_t2.TopK /
ORDER BY ... LIMITcleanup crashed withNullPointerExceptionwhen a tree-chain allocation failed mid-construction.LimitedSizeLongTreeChainallocates twoDirectIntListfreelists and a native value heap in its constructor, and on any allocation failure the constructor's error path callsclose().close()routed throughclear(), which dereferencedchainFreeList.clear()directly, so a malloc fault on the second freelist (or the value-heap allocation after it) leftchainFreeListnull and the cleanup threwNullPointerException, masking the real out-of-memory error.close()now resets the tree state inline and frees the freelists through the null-safeMisc.free(). The inline state reset is load-bearing rather than cosmetic: the chain is reused across cursor passes viareopen(), andtoTop()readschain.size(), so a first attempt that simply dropped theclear()call and reset nothing regressedOrderByLimitTest.testAsOfJoinWithNestedTopK, whose second cursor pass then read a stale row count and returned an empty result. A sweep for the same partial-construction shape fixed two more sites, neither reachable through the fuzzer today because fault queries run with parallel execution disabled and the first is a parallel-only path:AsyncHorizonJoinAtom/AsyncMultiHorizonJoinAtom.closeAggregationState()dereferencedshardingCtx.close()although the base-class constructor callsclose()on its own error path before the subclass assignsshardingCtx, andRecordChain.clear()dereferencedmem.close()whenVm.getCARWInstance()itself faulted; both now useMisc.free(). Repro (now passing): seed633879473312317/1781254988822,SELECT 'CDZWD'::VARCHAR, ... FROM long_sequence(35) WHERE (x IS NULL OR ...) ORDER BY 4 LIMIT 34.JIT filter wrapped a nested INT product that the Java filter widened to 64 bits, giving JIT-only wrong results. Surfaced by the cursor self-consistency check. A predicate like
c9 <= ((c0 - c2) * (c8 * -776782))(withc9 FLOAT,c0 LONG,c2 SHORT,c8 INT) computes an innerINT * INTproduct that overflows int32 and then feeds a LONG-width multiply. The Java filter reads the inner subtree throughMulInt/AddInt/SubInt#getLong, which widen every operand to 64 bits and never wrap; the JIT IR serializer normally matches this by pre-widening narrow operands to i64, but a FLOAT/DOUBLE operand in the predicate suppresses that widening, so the JIT computed the inner product at int32 width (wrapping mod 2^32) and only sign-extended the already-wrapped result. The two paths disagreed whenever the inner product overflowed int32: the JIT-on cursor returned 55 rows where the Java filter returned 54, and under the parallel filter this surfaced as atoTop()re-read changing the row set (51 then 58).CompiledFilterIRSerializernow walks each float-suppressed predicate and tags the narrow leaves that live under a LONG-width arithmetic subtree, emitting a 64-bit sign-extend (columns / bind variables) or an i64 immediate (constants) for exactly those, so the JIT computes the inner product at 64 bits and stays on rather than falling back to the slower Java filter. The tagging mirrors the C++convert()rule: only the operands of a narrow-width op computed at 64 bits are sign-extended, because a wide (i64) op already promotes its narrow operand -- soanint * 9000000000(whose i64 constant already lifts the multiply to 64 bits) is left untouched whileanint * 100000under a long multiply is widened. The sign-extend is scalar-only, so a tagged predicate runs scalar instead of vectorized; that is a small cost against keeping these uncommon mixed-width nested-arithmetic-with-float predicates on the JIT at all.CompiledFilterRegressionTest.testNarrowIntArithUnderLongWithFloatcompares the JIT and Java cursors for the diverging shapes and pins that they stay JIT-compiled, andCompiledFilterIRSerializerTestadds two IR-level tests: one pins the sign-extend on the inner INT product, the other pins that an i64 constant operand suppresses the widening. Repro (now correct): seed643056465770421/1781264165814,SELECT t0.* FROM fuzz_t1 t0 WHERE t0.c9 <= ((t0.c0 - t0.c2) * (t0.c8 * -776782)).Keyed ASOF JOIN sink-heap leak on out-of-memory during cursor open. The keyed ASOF JOIN cursors backed by
AsOfJoinFastRecordCursorFactoryandFilteredAsOfJoinFastRecordCursorFactoryreopen twoSingleRecordSinkheaps inof().reopen()allocates each heap lazily, so when the secondreopen()tripped the RSS memory limit the first sink's heap was already allocated.of()then threw out ofgetCursor(), whose catch block frees the master and slave cursors but not the reusable cursor, and the factory's_close()does not free that cursor either, so the half-opened cursor was orphaned and its 8-byte heap, taggedNATIVE_RECORD_CHAIN, leaked. Unlike the constructor-cleanup leaks above (Event Appender #1, Parameterize TxListener.onCommit() #4, Messaging Patterns #14) the fault hits the cursor-open path, so the fix lives inof()rather thanclose():of()now reopens both sinks inside a try/catch that closes them on any throwable before rethrowing, so a failed cursor open leaves no native memory behind. Closing a sink that never allocated is a no-op, so the cleanup is correct regardless of whichreopen()failed. The query fuzzer's malloc fault injection surfaced this.AsOfJoinFastOomTestsweeps the RSS ceiling across the cursor-open allocation points for both the plain and filtered keyed cursors and asserts the join leaves no native memory behind. Repro (now passing): seed654768710483172/1781275878059.A
LIMITquery whose filter overflows on a row past the cutoff surfaces the cast error non-deterministically across cursor passes -- accepted parallel-filter non-determinism, not a cursor-reuse defect (fuzzer over-reported it). Surfaced by the cursor self-consistency check. The parallel filter reduces whole page frames eagerly on worker threads, so a row whose filter expression overflows -- e.g.(c6 - c2)::DECIMAL(9, 3)for a value outside DECIMAL(9,3) -- can sit in a frame past theLIMITcutoff. Whether that frame's storedImplicitCastExceptionsurfaces depends on which frames the reducer evaluated before theLIMITcancelled the sequence, so the same cursor returns its 18 rows on the clean first pass and throws the cast error on a re-pass. The failing seed tripped this through thetoTop()re-iteration; replaying it tripped the identical query through thecalculateSize()recompute; other replays did not trip it at all -- the mark of a reduction-order race, not a deterministictoTop()defect. It is the same non-determinism the differential axes already tolerate (an out-of-range cast surfaces on a full scan but not on an index probe). Both self-checks now treat a re-pass throw of a per-row cast or numeric overflow (ImplicitCastException/NumericException) after a clean first pass as a tolerated skip, carried on a newCursorCheckException.toleratedflag thatrun()converts to a skip; a divergent row set, a changedpreComputedStateSize(), a count mismatch, or a re-pass throw of any other type still fails the query. The valve is deliberately narrower than the oracle'sisAcceptedSkip: aSqlExceptionis raised at compile time and cannot fire mid-iteration, and a decimal-aggregation overflow rides the aggregate accumulator rather than the filter, where a per-pass divergence is a real defect -- it is how bug fix Export tool #13 was caught -- not reduction-order noise. It keys on exactly theImplicitCastException/NumericExceptionpair the differential axes' index/parquet asymmetry rules (bug fix Asynchronous append() #9's neighbours) already use, narrowing the self-checks' "no skip valves" guarantee to this one data-dependent-error case. Repro (now skipped or clean across reruns): seed659553157043430/1781280662506,SELECT * FROM fuzz_t0 WHERE NOT ('2024-12-27'::DATE = ((c6)::LONG)::DATE OR (c6 - c2)::DECIMAL(9, 3) < (c2)::DECIMAL(76, 2)) ORDER BY ts DESC LIMIT 18.A
FILE-fault query reported a native-memory leak that an engine background job, not the query, caused (fuzzer over-reported it). The fault injection'sFailureFileFacadearmed a process-global op counter, so it counted down and fired on file ops from any thread rather than just the query thread. While aFILEfault was armed for one query, the engine's background posting-seal purge job -- running on a shared worker thread and routing its own file I/O through the same facade -- tripped the fault on a writer append, distressing that writer and leaving ~512 KiB mapped; the per-query leak oracle, which samples process-global native memory, then charged that to the query under test even though the query referenced a nonexistent table and opened no data files. The global counter also made the "fail after N ops" trigger point depend on background-thread timing, so the fire point was not reproducible from the seed alone. The facade gains an opt-in thread-scoped arming method (setToFailAfterOnCurrentThread) that the query fuzzer uses forFILEfaults: only ops on the arming thread count down and fire. Fault queries already run with parallel execution disabled, so the query under test issues all its file ops on that thread and fault coverage is unchanged --FILEstill fires on ~67% of armed queries at-Dquestdb.fuzz.fault.pct=100 -Dquestdb.fuzz.queries=2000. The default unscopedsetToFailAfteris left as is for the WAL/recovery fuzz harness, whose injected failures legitimately surface on background apply/purge threads. Repro (now green): seed672609384240043/1781347718070.Vectorized GROUP BY leaked a page-frame address buffer on out-of-memory during cursor open. The keyed
count()aggregate under aLIMITfrom the open bug below runs through the vectorized rosti GROUP BY (GroupByRecordCursorFactory), not the map-basedAsyncGroupByRecordCursorFactorythe bug was first attributed to.RostiRecordCursor.of()reopens the factory's sharedPageFrameAddressCache, whoseof()reopens four off-heapDirectLongLists one by one; eachreopen()reallocates native memory and can trip the RSS limit. When a laterreopen()failed after an earlier one had already allocated,of()propagated the exception leaving the cache partially open, and becausegetCursor()then never returned the cursor, the caller could not close it and the factory's_close()does not free the cache either -- so the already-reopened 512-byteDirectLongList(NATIVE_DEFAULT) was orphaned. This made the leak parallel-only only incidentally: the rosti path is reached the same way under serial execution, but the seeds that hit it happened to run the aggregate in the parallel phase.PageFrameAddressCache.of()now reopens the lists atomically: on anyreopen()failure it closes the ones already reopened, returning the cache to its fully-closed (zero native memory) state, so a caller that propagates the failure without reachingclose()leaks nothing. The fix covers everyof()caller, not just the rosti GROUP BY (the async filter / group-by page-frame sequences,SAMPLE BYfirst/last, the time-frame cursor and the QWP egress path).GroupByVectorizedOomTestsweeps the RSS ceiling across the cursor-open allocation points for a single-keysum()aggregate and asserts the query leaves no native memory behind. Repro (now green): seed709239330479655/1781513480332with-Dquestdb.fuzz.queries=400 -Dquestdb.fuzz.fault.pct=70; minimal queryWITH cte0 AS (SELECT c0 AS k, count() AS cnt FROM fuzz_t0) SELECT * FROM cte0 LIMIT 30.SELECT max(ts) FROM t WHERE ...and itsmin/first/lastsiblings over the designated timestamp swallowed a fired fault under parallel execution -- legitimate early termination from an implicitLIMIT, not a swallowed error (fuzzer over-reported it).SqlOptimiser.rewriteSingleFirstLastGroupByrewrites a lonemin/max/first/lastover the designated timestamp into anORDER BY ts [DESC] LIMIT 1page-frame scan: the newest (formax/last, a backward scan) or oldest (formin/first, a forward scan) matching row answers the aggregate, so the aggregation step disappears. Under the eager parallel reduce a firedtest_fault()can land on a frame whose rows fall past that single-row cutoff -- the faulting frame's result is discarded and the query completes with a correct answer. That is the same legitimate early termination an explicitLIMITgives, and which frame the fault lands on depends on worker timing, so the verdict must not flip on it. The swallow oracle already tolerated an explicitLIMITby matching thelimitkeyword in the SQL text, but the rewrite leaves no keyword there.QueryRunnernow also consults the rewritten plan:runRaw/runRawMallocFaultrender the factory plan (heap-only, before the cursor opens / before the RSS limit is armed, so it stays outside the fault window) and carry ahasPushedLimitflag on theOutcome.planHasPushedLimitkeys off thelimit:attribute thatAsyncFilteredRecordCursorFactory/AsyncJitFilteredRecordCursorFactoryemit only for a pushed-down limit, so the relaxation spans both the explicit and the implicit cases. Aggregates that keep a real aggregation step carry no pushed limit and so are unaffected --max(c0)over a non-timestamp column,count(), andmax(ts)projected alongside another column all run through the async GROUP BY, which surfaces a fired fault as before. This is a test-oracle change only; no engine behaviour changes.QueryFuzzTest.testImplicitTimestampLimitToleratedpins the pushed-limit plan marker for the four timestamp aggregates and its absence for the non-rewritten aggregates. Repro (now tolerated): seed718495134431076/1781522736136, querySELECT max(ts) AS a0 FROM fuzz_t2 WHERE test_fault() ORDER BY a0.The vectorized rosti GROUP BY over-freed
NATIVE_ROSTInative memory when an aggregate'swrapUp()resized the map (previously listed as known open bug Event Appender #1). On the single-worker build path the factory called each aggregate function'swrapUp()without bracketing it withupdateMemoryUsage().wrapUp()inserts the null-key slot that carries the aggregate value for column-top rows; when the live keys have already filled the map to its growth threshold, that insert resizes the rosti. The single-worker path did not record the resize, so the cursor'sclose()reset later subtracted the full grown size from theNATIVE_ROSTItag and the tag ended the run with a negative net delta -- the over-free reported asNATIVE_ROSTI, difference: -<n>(one resize step, tens of KiB). The multi-worker merge path already bracketedwrapUp()withupdateMemoryUsage()for exactly this reason ("some wrapUp() methods can increase rosti size"); the single-worker path now does the same, and the multi-worker path is unchanged. This is a pure accounting defect -- query results are unaffected -- and it reproduces deterministically without any fault; the fault injection surfaced it only as an end-of-runNATIVE_ROSTIimbalance.GroupByVectorizedRostiAccountingTestsweeps the live-key count across the growth boundary with column-top null keys so one iteration lands thewrapUpresize, and asserts theNATIVE_ROSTIaccounting stays balanced on both the single-worker and multi-worker build paths.The vectorized rosti GROUP BY left a stale aggregate task in the shared queue when a fault aborted the cursor's drain, crashing a later query that work-stole it (previously listed as known open bug Event Appender #1).
GroupByRecordCursorFactorypublishes per-frameVectorAggregateEntrytasks to the engine-global vector-aggregate queue, then inbuildRosti'sfinallyblock drains the queue throughrunWhatsLeft-- work-stealing and running each task -- until its owndoneLatchis satisfied, before freeing the per-buildframeMemoryPools.runWhatsLeftlet a stolenentry.run()throw straight out, so when aMALLOCfault tripped a parquet decode insidenavigateToduring the drain, the throw aborted the drain with published entries still in the queue. Because that throw originates in thefinally, it bypasses the catch that cancels the shared circuit breaker, so the survivors kept a live (non-cancelled) circuit breaker and a zerooomCounter-- the guards that would otherwise make a stolen entry skip its work. The factory then closed and freed itsframeMemoryPools, nulling each pool'saddressCache, so a later vectorized GROUP BY -- including the fault oracle's recovery re-run -- work-stole a survivor and dereferenced the freed pool, throwingNullPointerException: Cannot invoke "PageFrameAddressCache.getFrameFormat(int)" because "this.addressCache" is null(the same NPE theGroupByVectorAggregateJobworker threads were already swallowing and logging).runWhatsLeftnow wraps each stolenentry.run(): a throwing entry no longer aborts the drain, so the build's published entries are all consumed and none survives in the shared queue referencing the about-to-be-freed pools; the first error surfaces once the queue is fully drained, preserving the query's failure semantics.GroupByVectorizedOomTest.testWorkStolenEntryDoesNotOutliveFreedPoolsOverParquetconverts a multi-partition table to parquet and sweeps the RSS ceiling acrossbuildRosti(armed after compile so the trip lands in the drain), running a clean recovery query after each ceiling; before the fix the recovery work-stole a survivor and hit the NPE. Repro (now green): seed718495134431076/1781522736136with-Dquestdb.fuzz.fault.pct=60 -Dquestdb.fuzz.queries=2000 -Dquestdb.fuzz.window=false.A GROUP BY key that references the alias of a trivial arithmetic expression failed to compile, while the same query with a bind variable in place of the constant compiled (literal/bind divergence).
SqlOptimiser.rewriteTrivialGroupByExpressionslifts a trivial key such as859371 + (cnt * -237288)out of the inner GROUP BY when its base column (cnt) is also a key, recomputing the offset in an outerVIRTUALand removing the lifted key column from the group-by model (the ClickBench Q35 optimisation). It removed the column but left the key's alias reference behind in the model's GROUP BY list, soGroupByUtils.validateGroupByColumnslooked the alias up in the model's now-smaller column set, did not find it, and rejected the query with "group by column does not match any key column". The bind-variable form (:b0::INT + (cnt * -237288)) hid the bug: a bind-variable cast is aFUNCTIONnode, which the trivial-expression visitor does not treat as trivial, so the lift never ran and the key stayed in the group-by model. The literal form was therefore rejected at compile time while the bind form compiled and returned rows -- the divergence the bind-variant axis flagged. This is an engine bug, not a fuzzer-oracle issue: the query is valid SQL (grouping bycntand a function ofcntproduces the same groups as grouping bycntalone) and must compile either way.rewriteTrivialGroupByExpressionsnow drops the stale alias reference from the nested GROUP BY list whenever it removes the lifted key column; expression keys spelled out in GROUP BY (e.g.ClientIP - 1) are untouched becausevalidateGroupByColumnsmatches those against the surviving base column rather than by alias.GroupByTest.testGroupByTrivialExpressionKeyReferencedByAliaspins both the alias-reference and spelled-out forms against the hand-computed result. Repro (now correct): seed738911385687093/1781543152387, querySELECT t0.cnt AS e0, (859371 + (t0.cnt * -237288)) AS e1, count() AS a0 FROM (SELECT c4 AS k, count() AS cnt FROM fuzz_t1) t0 WHERE ... GROUP BY t0.cnt, e1 ORDER BY 1 ASC, e1.A multi-key index scan (
sym = ... OR sym = ...,sym != ...) leaked posting-index block buffers when an out-of-memory error hit while building its per-key row cursors on the first page frame.HeapRowCursorFactory.getCursor()builds oneRowCursorper symbol key into the factory's owncursorslist, then hands that list to theHeapRowCursorviacursor.of().HeapRowCursorFactory.close()relied solely onMisc.free(cursor)->HeapRowCursor.close()to drain the list, but theHeapRowCursoronly frees the list reference it received inof(); when aMALLOCfault tripped the RSS limit mid-build on the very first page frame -- beforeof()ran -- that reference was still null, so the per-key posting cursors already built into the factory's list were never closed and each retained its block buffer (NATIVE_INDEX_READER), which the owningPostingIndexFwdReader/PostingIndexBwdReader'sclose()never reclaimed because the cursor was never returned to the reader's free pool.HeapRowCursorFactory.close()now drains its owncursorslist directly; onceof()has run the two references point at the same list, so theHeapRowCursor'sclose()empties it and the extra drain is a no-op. Separately,PostingIndexFwdReader.getCursor()/PostingIndexBwdReader.getCursor()now release a cursor's native buffers if itsof()throws -- a cursor popped from the free pool (or freshly created) is not yet owned by the caller, so a throw while growing the block buffer would otherwise strand it with neither the caller nor the reader able to reclaim it.SequentialRowCursorFactoryshares the build-into-a-list shape but is unaffected: itsSequentialRowCursoris an inner class that frees the factory'scursorsfield directly. Surfaced by malloc fault injection as an end-of-runNATIVE_INDEX_READER, difference: 144imbalance. Repro (now green): seed747545817585554/1781551786819.A non-keyed
count_distinctover a constant (or a fully-enumerated symbol) swallowed a fired fault under parallel execution: legitimate early termination from the group-by's early exit, not a swallowed error (fuzzer over-reported it). When a non-keyed aggregate's value is final after the first matching row (acount_distinctover a constant always lands on 1, and over a symbol it stops once every distinct symbol has been seen), the optimiser routes the query toGroupByNotKeyedRecordCursorFactory's early-exit cursor, which stops scanning the base cursor after that row. Beneath it the async filter keeps reducing later page frames on worker threads, so a firedtest_fault()can land on a frame the early-exit consumer never reads; that frame's result and its throw are discarded and the query completes with a correct answer. This is the same legitimate early termination an explicit or pushedLIMITgives (bug fix Generic data access #20), and which frame the fault lands on depends on worker timing, so the verdict must not flip on it. The swallow oracle already tolerated a row-limit cutoff (thelimitkeyword in the SQL text or thelimit:plan attribute); it now also tolerates an early-exit non-keyed group by.QueryRunner.runRaw/runRawMallocFaultcarry ahasEarlyExitGroupByflag on theOutcome, computed byfactoryHasEarlyExitGroupBy, which walks the factory's base-factory chain (the top-level consumption spine, not anywhere in the plan as the limit check does) for aGroupByNotKeyedRecordCursorFactorywhoseisEarlyExitSupported()is true. The relaxation is narrow and cannot hide an error on a frame the query actually reads: a fault on a consumed frame surfaces when the group by pulls it, sooutcome.failureis non-null and the swallow branch never runs; and the fuzzer injectstest_fault()only into the single scan a non-keyed group by consumes. Aggregates that keep scanning every row carry no early-exit cursor and surface a fired fault as before:count_distinctover a long constant andcount(*)run through the async GROUP BY, and a plaincount_distinct(symbol)is rewritten to a keyed group by. The only production change is a read-onlyisEarlyExitSupported()predicate onGroupByNotKeyedRecordCursorFactory; no engine behaviour changes.EarlyExitGroupByDetectionTestpins the detection forcount_distinctover a constant (with and without a filter) and its absence for the non-early-exit shapes; the swallow reproduces about 1 in 50 runs at-Dquestdb.fuzz.workers=12with the tolerance forced off. Repro (now tolerated): seed808495966924054/1781612736968, querySELECT count_distinct('B') AS a0 FROM fuzz_t2 WHERE test_fault().Window-function fuzzing sweep (no known open bugs remain)
The window cursor
toTop()self-consistency failure previously listed here -- re-iterating returned a different row set for a query partitioning a window by a Decimal256 column -- shares its root cause with bug fix #10 above and is fixed by it. The seed that surfaced it now runs clean:A 10-seed sweep at
-Dquestdb.fuzz.queries=3000(window and fault shapes on by default) went red on 5 of 10 seeds, every failure falling into one of two window-function defects. Both are now fixed: thelead(null)crash (bug fix #11 -- seeds553825244550718/1781174934593and553881716081004/1781174991065) and theRANGE-frame timestamp-index bug (bug fix #12 -- seeds553854821671111/1781174964170,553832024235320/1781174941373and553862616269146/1781174971965). No fault-injection leak or recovery failures and no other crashes remained, so the serial cleanup paths the earlier fixes cover are holding. The sweep now runs green.Bind-variable differential: bug surfaced, fixed in #7225 (merged)
The bind-variable variant (each bindable literal also runs as a
:bN::TYPEplaceholder, with the two forms cross-checked) surfaced a wrong-results bug inSPLICE/RIGHT OUTER/FULL OUTERjoins: aWHEREpredicate referencing only the NULL-extended master (left) table was pushed below the join, so the null-master output rows bypassed it; for a literal constant the optimiser could additionally propagate the predicate to the slave through the join key, which a bind variable cannot, so the literal and bind forms disagreed on the row count. This is an engine optimiser bug independent of the fuzzer; the fix landed in its own dedicated PR, #7225, not on this branch. #7225 has since merged to master and is now included here through the master merge, so the repro runs green on this branch. Repro seed:-Dquestdb.fuzz.s0=574598955642074 -Dquestdb.fuzz.s1=1781195708304. The bind-variable axis also surfaced bug fix #23 above, an unrelated GROUP BY compile-time divergence (a trivial-expression key referenced by alias), which is fixed on this branch.Known open bugs
None remain. A high fault rate surfaced three MALLOC-fault-recovery defects in the page-frame allocation paths, all now fixed: the 512-byte
NATIVE_DEFAULTleak by a keyed GROUP BY under aLIMIT(bug fix #19, traced to the vectorized rosti GROUP BY's page-frame address cache, not the map-based parallel GROUP BY it was first filed against), theNATIVE_ROSTIover-free (bug fix #21, the single-workerwrapUp()resize was not bracketed byupdateMemoryUsage()), and the recovery crash from a stale aggregate task left in the shared vector-aggregate queue (bug fix #22). The seeds that surfaced them now run clean, including an 8000-query sweep at-Dquestdb.fuzz.fault.pct=60(about 4x the default) with 905 MALLOC faults fired.Tradeoffs
-Dquestdb.fuzz.window=false.FILEandMALLOCfaults exercise the parallel error-path cleanup that was previously uncovered, at the cost of fire-point determinism (under parallel reduce, "fail after N ops" fires at a non-deterministic point across worker threads, so a seed reproduces the pass/skip/fail verdict but not the exact fault site). This surfaced a parallel-only leak in the vectorized GROUP BY cursor open, now fixed (bug fix Benchmarks #19); pass-Dquestdb.fuzz.fault.parallel=falseto fall back to serial fault injection.assertMemoryLeakremains the authoritative net check.nullliteral --sum(null) OVER (...),lag(null) OVER (...), etc. -- now raises a cleanSqlExceptionasking for a concrete cast, where some of these variants previously resolved to a concrete type and silently computed over NULL. The previous behaviour was platform-dependent, so this trades a small loss of a degenerate, non-portable form for a uniform, deterministic rejection;null::double(or any concrete cast) keeps working.Test plan
mvn -pl core -Dtest=QueryFuzzTest test(faults and window shapes on by default)-Dquestdb.fuzz.fault.pct=70 -Dquestdb.fuzz.queries=500-Dquestdb.fuzz.s0=L -Dquestdb.fuzz.s1=L-Dquestdb.fuzz.faults=false-Dquestdb.fuzz.window=falseSummary by CodeRabbit
Bug Fixes
New Features
Performance