[SPARK-59890][SQL][TESTS] Replace ResolveAsOfJoinSuite with end-to-end ASOF JOIN tests - #59158
Closed
LukaZdravic wants to merge 1 commit into
Closed
LukaZdravic wants to merge 1 commit into
LukaZdravic wants to merge 1 commit into
Conversation
…d ASOF JOIN tests Remove ResolveAsOfJoinSuite. It tests only the fixed-point ResolveAsOfJoin rule and pins internal expression shapes. End-to-end tests already cover 19 of its 20 cases, and in tests they also run the single-pass resolver. Add an AsOfJoinSQLSuite test for the last case: INNER ASOF JOIN keeps a NOT NULL right column NOT NULL, and LEFT ASOF JOIN makes it nullable. Co-authored-by: Isaac <no-reply@databricks.com>
uros-b
approved these changes
Oct 1, 2026
uros-b
pushed a commit
that referenced
this pull request
Oct 1, 2026
… ASOF JOIN tests ### What changes were proposed in this pull request? Remove `ResolveAsOfJoinSuite`. It unit-tests only the fixed-point analyzer rule `ResolveAsOfJoin`: it builds plans by hand and checks internal expression shapes, for example that the `MATCH_CONDITION` distance is a `Subtract` or a `ZipWith`. Add one end-to-end test to `AsOfJoinSQLSuite` for the only behavior no other test checked: `INNER ASOF JOIN` keeps a NOT NULL right column NOT NULL, and `LEFT ASOF JOIN` makes it nullable. ### Why are the changes needed? The suite does not test the single-pass resolver (`AsOfJoinResolver`), which calls the same `MATCH_CONDITION` code. It also pins internal shapes, so an internal change must edit it even when no query result changes. SPARK-59749 had to. End-to-end tests already cover the other 19 of its 20 cases: the operators and the match per operand type (`join-asof-grammar.sql`, `join-asof-datatypes.sql`, `AsOfJoinSortMergeSQLSuite`), `USING` and `ON` (`join-asof-grammar.sql`, `join-asof-semantics.sql`), and the rejected operands (`join-asof-errors.sql`, `AsOfJoinSQLSuite`). In tests, these queries run both analyzers where the single-pass resolver supports them, and fail if the plans differ (`spark.sql.analyzer.singlePassResolver.dualRunWithLegacy` is on while testing). ### Does this PR introduce _any_ user-facing change? No. Test-only change. ### How was this patch tested? Ran the ASOF JOIN suites (`AsOfJoinSQLSuite`, `AsOfJoinSortMergeSQLSuite`, `AsOfJoinMatchConditionTypesSuite`, `RewriteAsOfJoinSuite`, `SortMergeAsOfJoinSuite`, `DataFrameAsOfJoinSuite`, `StreamingAsOfJoinSuite`) and `SQLQueryTestSuite`, which includes the `join-asof-*.sql` golden files. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code This pull request and its description were written by Isaac. Closes #59158 from LukaZdravic/asof-remove-resolve-suite. Authored-by: Luka Zdravić <lukazdravic10@gmail.com> Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com> (cherry picked from commit f74ba34) Signed-off-by: Uros Bojanic <221401595+uros-b@users.noreply.github.com>
Member
Member
|
Thank you @LukaZdravic! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Remove
ResolveAsOfJoinSuite. It unit-tests only the fixed-point analyzer ruleResolveAsOfJoin: it builds plans by hand and checks internal expression shapes, for example that theMATCH_CONDITIONdistance is aSubtractor aZipWith.Add one end-to-end test to
AsOfJoinSQLSuitefor the only behavior no other test checked:INNER ASOF JOINkeeps a NOT NULL right column NOT NULL, andLEFT ASOF JOINmakes it nullable.Why are the changes needed?
The suite does not test the single-pass resolver (
AsOfJoinResolver), which calls the sameMATCH_CONDITIONcode. It also pins internal shapes, so an internal change must edit it even when no query result changes. SPARK-59749 had to.End-to-end tests already cover the other 19 of its 20 cases: the operators and the match per operand type (
join-asof-grammar.sql,join-asof-datatypes.sql,AsOfJoinSortMergeSQLSuite),USINGandON(join-asof-grammar.sql,join-asof-semantics.sql), and the rejected operands (join-asof-errors.sql,AsOfJoinSQLSuite). In tests, these queries run both analyzers where the single-pass resolver supports them, and fail if the plans differ (spark.sql.analyzer.singlePassResolver.dualRunWithLegacyis on while testing).Does this PR introduce any user-facing change?
No. Test-only change.
How was this patch tested?
Ran the ASOF JOIN suites (
AsOfJoinSQLSuite,AsOfJoinSortMergeSQLSuite,AsOfJoinMatchConditionTypesSuite,RewriteAsOfJoinSuite,SortMergeAsOfJoinSuite,DataFrameAsOfJoinSuite,StreamingAsOfJoinSuite) andSQLQueryTestSuite, which includes thejoin-asof-*.sqlgolden files.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
This pull request and its description were written by Isaac.