Release DataFusion `46.0.0`

[alamb]

Is your feature request related to a problem or challenge?

Tracking ticket for next release, also a place to track desired inclusions

Previous release will be https://crates.io/crates/datafusion/45.0.0 (likely Feb 1, 2025) December 31, 2024 so next major release would be around March 1, 2025

Steps:

• Update version and changelog
• Test with DataFusion Python:
• Test with DataFusion Comet: feat: Upgrade to DataFusion 46.0.0-rc2 datafusion-comet#1423
• Test with delta.rs: chore: upgrade to DataFusion 46.0.0 delta-io/delta-rs#3261
• Test with SailHQ:
• Test with InfluxData
• Voting Thread 1: https://lists.apache.org/thread/zh7s3n2h089fbhrh64y9pcx7bdw7ptv4
• Create ticket for next release: Release DataFusion 47.0.0 (April 2025) #15072

Prior release tickets:

• 44.0.0 Release DataFusion 45.0.0 #14008 (comment)
• 45.0.0 Release DataFusion 45.0.0 #14008 (comment)

Bugs that would be good to fix

• datafusion-cli regression: explain plan output looks bad (error rendering multi-lines) #14947
• index out of bounds: the len is 2 but the index is 2 in some data sources #14957
• Add Upgrade Guide for DataFusion 46.0.0 #14891
• bug: Physical plan round trip fails in some cases after datasource refactor #14679
• Incorrect backslash treatment in string literals in DataFusion CLI #13286
• Error projecting statistics in DataSourceExec #14905
• fix: fetch is missed during EnforceDistribution #14207
• fix: EnforceSorting should not remove a needed coalesces #14637
• New Datafusion-cli streaming printing way should handle corner case for only one small batch which lines are less than max_rows #14920
• Fix failing UNION [ALL] BY NAME tests due to sanity checker #14806
• Regression since 45.0.0: select count(), count(*) does not work #14855
• Extended sqllite tests are failing on main #14853
• DataSink::write_all given invalid RecordBatchStream #14394
• Querying Parquet file specifically with a predicate returns invalid data error but works in other situations #14281
• Fix: limit is missing after removing SPM #14569
• bug: improve schema checking for insert into cases #14572

Nice to have but non blockers

• Allow FileSource-specific repartitioning #14754
• Change in 46: count_all() expr_fn function now displayed as count(1) rather than count(*) #14894

Needs some traige (should it be a blocker?)

TODO

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

I think @xudong963 said they might be interested in running this one:

• Release DataFusion 45.0.0 #14008 (comment)

Thanks, alamb, I booked 46 in advance!

[xudong963]

@alamb I'll also do some updates in the issue summary.

Considering that this is the first time I've been involved in this process, could you please remind me if some critical time comes up and I don't do anything about it?

[alamb]

@alamb I'll also do some updates in the issue summary.

Considering that this is the first time I've been involved in this process, could you please remind me if some critical time comes up and I don't do anything about it?

Yes of course. Thank you @xudong963

I am planning to be involved in this one too -- perhaps I can help by testing the release e.g. with delta-rs and influxdb_iox

[alamb]

@xudong963 when would you like to start making the release? Maybe we should targe the week of Feb 24 🤔

[xudong963]

@xudong963 when would you like to start making the release? Maybe we should targe the week of Feb 24 🤔

yes, the week is suitable.

[andygrove]

I've created a draft PR to upgrade Comet to use latest DataFusion:
apache/datafusion-comet#1423

[andygrove]

I noticed that older releases have not been removed from https://dist.apache.org/repos/dist/release/datafusion/. Perhaps this can be cleaned up as part of this release.

[alamb]

@xudong963 are we still thinking of trying to get the release ready this upcoming week?

I will try and focus my efforts starting tomorrow on ensuring the bugs listed in "Bugs that would be good to fix" are in

[xudong963]

@alamb In Bugs that would be good to fix, four issues already have PRs, and one does not four issues and all have PRs. I'll focus on reviewing the four in the next two days and plan to update version and changelog on Wed. What do you think?

From experience, It seems that we should leave one week to test with Comet, delta, SailHQ, and InfluxData, so the final release may be delayed to next Wed, about 3.5.

[alamb]

@alamb In Bugs that would be good to fix, four issues already have PRs, and one does not four issues and all have PRs. I'll focus on reviewing the four in the next two days and plan to update version and changelog on Wed. What do you think?

I think it is a good idea. I'll try and make a PR to delta-rs as well

Maybe @shehabgamin can help with testing SailHQ

Looks like @andygrove has made a PR for testing with comet (linked above): apache/datafusion-comet#1423

[alamb]

I made a PR for testing in delta here:

• chore: upgrade to DataFusion 46.0.0 delta-io/delta-rs#3261

Still has some issues to work out

[shehabgamin]

I will test with Sail by Wednesday!

[andygrove]

I am running into one error in Comet: Function sha256 does not implement invoke but called. I will investigate this today.