Release DataFusion `45.0.0`

[alamb]

Is your feature request related to a problem or challenge?

Tracking ticket for next release, also a place to track desired inclusions

Last release was https://crates.io/crates/datafusion/440.0 December 31, 2024 so next major release would be around Feb 1, 2025

Steps:

• Update version and changelog: Prepare for 45.0.0 release: Version and Changelog #14397
• Voting Thread: https://lists.apache.org/thread/g20ywc9yto8xp07lcllmvgyn8g5z4420
• Release to crates.io: https://dist.apache.org/repos/dist/release/datafusion/datafusion-45.0.0
• Create ticket for next release: Release DataFusion 46.0.0 #14123
• Port changelog to main: Minor: Update changelog for 45.0.0 and tweak CHANGELOG docs #14545

Pre-relese testing

• Test with Comet: Test DataFusion 45.0.0 with Comet #14274
• Test with delta.rs: chore: update to DataFusion 45.0.0 / arrow/parquet 54.0.0 delta-io/delta-rs#3175
• Test with SailHQ: Test DataFusion 45.0.0 with Sail #14408
• Test with InfluxData
• Test with DataFusion Python: Test DataFusion 45 with datafusion-python #14410

Prior release tickets:

• 44.0.0 Release DataFusion 44.0.0 #13334

Please let me know if you would like to add any items on this list or move the categorization

Items to fix before release

• Upgrade to arrow-rs/parquet 54.0.0 #14114
• Regression: DataFrame::schema returns incorrect schema for NATURAL JOIN #14058
• Regression in CASE expression since DF 44 #14277
• string_numeric_coercion in binary.rs does not support Utf8View #13359
• DataFusion Regression (Starting in v43): Type Coercion for UDF Arguments (X --> String) for Specified UDFs #14230
• Regression: Invalid comparison operation: Utf8 == Utf8View error during LEFT ANTI JOIN #13510
• Type Coercion fails for List with inner type struct which has large/view types #14154
• [Epic] A Collection of Additional UTF8View support tickets #13504
• encode(..., "hex") errors on non-UTF-8 binaries since Datafusion v43 #14055
• Improve deprecation message for MemoryExec #14322
• LimitPushdown rule uncorrect remove some GlobalLimitExec #14204

Items maybe to complete (not sure if they are blockers)

• Can no longer easily join duplicate schemas as of version 43 #14112
• Query produces different results after physical plan round trip to bytes #14253
• Potential performance regression with comparisions to scalar values #14291
• Error with type coercion with CREATE TABLE AS SELECT ... inserting VALUES  #13124
• Bug: applying multiple times EnforceDistribution generates invalid plan #14150

Nice to Have (but non blockers -- e.g. bugs but not regressions)

• feat: metadata columns #14057
• Exponential planning time (100s of seconds) with UNION and ORDER BY queries #13748
• External Error prefix is repeated multiple times #14080
• FFI support for versions and alternate tokio runtimes #13937
• Improve speed of median by implementing special GroupsAccumulator  #13681
• FULL OUTER JOIN and LIMIT produces wrong results #14335

[alamb]

@andygrove would you like to coordinate this release or would you like me to? (or does anyone else want to do so?)

[alamb]

I also added some issues to the description above that I think would be worth fixing

[andygrove]

@andygrove would you like to coordinate this release or would you like me to? (or does anyone else want to do so?)

I don't have a preference. I will traveling around this time though, so perhaps it would make sense for someone else to be release manager for this one.

[alamb]

I don't have a preference. I will traveling around this time though, so perhaps it would make sense for someone else to be release manager for this one.

I am happy to do it again for 45 if no one else would like the opportunity (see what I did there 😆 )

[xudong963]

I don't have a preference. I will traveling around this time though, so perhaps it would make sense for someone else to be release manager for this one.

I am happy to do it again for 45 if no one else would like the opportunity (see what I did there 😆 )

Thanks, alamb, I booked 46 in advance!

[alamb]

I am happy to do it again for 45 if no one else would like the opportunity (see what I did there 😆 )

Thanks, alamb, I booked 46 in advance!

Awesome -- I filed #14123 to track 46

[alamb]

I plan to start assembing the release candidate and test on the week of Jan 27 (in about 2 weeks time()

[shehabgamin]

As promised, Sail is working on porting relevant tests into DataFusion.

A good starting point is a regression our tests caught in DataFusion 43, which still seems to persist in DataFusion 44. A regression was introduced in DataFusion 43.0.0 related to casting to UTF8 in various places. Upgrading to DataFusion 43.0.0 required adding explicit casting in several areas as a workaround. This PR (lakehq/sail#355) comments out those changes to expose the regression through the 12 additional failed tests compared to the main branch.

Once I’ve pinpointed the root cause(s) of the regression, I’ll create an issue in DataFusion to track the work. I want to ensure the issue accurately reflects the problem before filing it. I’m happy to address these regressions and port over the tests that cover them in the same PR. Hopefully, we can get this resolved in time for the DataFusion 45 release!

[alamb]

Once I’ve pinpointed the root cause(s) of the regression, I’ll create an issue in DataFusion to track the work. I want to ensure the issue accurately reflects the problem before filing it. I’m happy to address these regressions and port over the tests that cover them in the same PR. Hopefully, we can get this resolved in time for the DataFusion 45 release!

Thank you very much @shehabgamin 🙏

I strongly suspect this is related to switching to Utf8View by default in Parquet; You can validate this theory by disabling the following config setting:

https://datafusion.apache.org/user-guide/configs.html

datafusion.execution.parquet.schema_force_view_types

I think we are pretty close to closing out the Utf8View epic (now that we have upgraded to the latest arrow):

• [Epic] A Collection of Additional UTF8View support tickets #13504

I'll add that to the list for 45 too

[alamb]

I plan to start preparing / testing / pushing this release the week of Jan 27, aiming to get an release candidate early the next week

[shehabgamin]

I strongly suspect this is related to switching to Utf8View by default in Parquet; You can validate this theory by disabling the following config setting:

https://datafusion.apache.org/user-guide/configs.html

datafusion.execution.parquet.schema_force_view_types

I think we are pretty close to closing out the Utf8View epic (now that we have upgraded to the latest arrow):

* [[Epic] A Collection of Additional UTF8View support tickets #13504](https://github.com/apache/datafusion/issues/13504)

Thanks for the pointer @alamb!

I tried setting datafusion.execution.parquet.schema_force_view_types to false, but unfortunately, none of the 12 failed tests passed.

I'll take a deeper look into the issue after the weekend. Hope you have a great rest of your weekend!

[shehabgamin]

I'll take a deeper look into the issue after the weekend. Hope you have a great rest of your weekend!

Most of the regressions are related to this issue: #14230. I should be able to resolve them well before the 45 release.

While testing my local Sail code with the latest commit on DataFusion's main branch, I encountered several breaking changes that may make DataFusion 45 a jarring upgrade for some users. Given the previous discussion about wanting to make releases less jarring (#13334 (comment)), I wanted to bring this to your attention, @alamb.

Aside from that, there is one remaining regression I haven't investigated yet, which seems to be related to Parquet.

[alamb]

While testing my local Sail code with the latest commit on DataFusion's main branch, I encountered several breaking changes that may make DataFusion 45 a jarring upgrade for some users

Thanks @shehabgamin -- Can you enumerate these changes (or point me at a PR) so we can see if there is some way to make jarring

[shehabgamin]

Thanks @shehabgamin -- Can you enumerate these changes (or point me at a PR) so we can see if there is some way to make jarring

Yeah I'll work on that right now!