Release DataFusion `49.0.0` (July 2025)

[alamb]

Is your feature request related to a problem or challenge?

Tracking ticket for next release, also a place to track desired inclusions

Previous release will be https://crates.io/crates/datafusion/47.0.0 (June, 2025) so next major release would be late June or Early July 2025

Steps:

• Update version and changelog: [branch-49] Update version to 49.0.0, add changelog #16822
• Create a release branch: https://github.com/apache/datafusion/tree/branch-49
• Test with DataFusion Python:
• Test with DataFusion Comet: chore: migrate to DF 49.0.0 datafusion-comet#2040
• Test with delta.rs: chore: update to DataFusion 49.0.0 delta-io/delta-rs#3603
• Test with iceberg-rust:
• Test with LakeSail: chore: Upgrade to DataFusion 49.0.0 lakehq/sail#653
• Write upgrade guide:
• Test with parquet viewer
• Test with datafusion-materialized-views: Upgrade to DF49 datafusion-contrib/datafusion-materialized-views#75
• Voting Thread:
• Create ticket for next release: Release DataFusion 50.0.0 (Aug/Sep 2025) #16799
• Forward port version and changelog to main: [main] Update version to 49.0.0, add 49.0.0 changelog #16855

TODOs

• Update upgrade md for new unified config for sql string mapping to utf8view when we release datafusion 49.0.0 #16428
• Blog post for the DataFusion 49 release #16758

Prior release tickets:

• 48.0.0: Release DataFusion 48.0.0 (June 2025) #15771

Changes to add to upgrade guide

These PRs made changes that deserve a mention in the upgrade guide

• [MAJOR] Equivalence System Overhaul #16217
• Add compression option to SpillManager #16268

Features to mention in the blog (if they make it)

• Introduce Async User Defined Functions  #14837
• [EPIC] A collection of tickets for improving sorting larger than memory datasets / spilling sorts #15271
• TopK dynamic filter pushdown attempt 2 #15770
• Support integration with Parquet modular encryption #15216
• [MAJOR] Equivalence System Overhaul #16217
• Use Tokio's task budget consistently, better APIs to support task cancellation #16398
• feat: Parquet modular encryption #16351

Bugs that need to be fixed

• Panic in datafusion_expr::window_state::WindowAggState::update #16308
• SortQueryFuzzer found a failing case on main #16452
• Wrong results when pushdown_filters is enabled (starting in 48.0.0) #16588
• restore topk pre-filtering of batches and make sort query fuzzer less sensitive to expected non determinism  #16501
• Extended tests are failing #16595

Bugs that would be good to fix / investigate

• 1000x slowdown opening parquet file due to partitions #16676
• How to write csv file to disk from a empty dataframe? #16240
• Release DataFusion 49.0.0 (July 2025) #16235 (comment)

Community Wishlist

• Format Date32 to string given timestamp specifiers #15361
• [Epic] Pipeline breaking cancellation support and improvement #16353

[alamb]

I am thinking we should begin the 49.0.0 release next week. I'll publicize to the lists / etc

[xudong963]

@alamb Yeah, let's do it (time flies

[adriangb]

We introduced new APIs in #16461 and more importantly removed the existing SchemaAdapter. I am not sure how upgrading other systems has gone but if they relied on custom SchemaAdapters they'll be broken. I think we should restore the default SchemaAdapter and make it possible to choose which path you use. To get there I think we need at least part of #15057 and to revert the change of removing SchemaAdapter in the original PR. I'm sorry if this causes a delay, I'll get to it ASAP on Tuesday.

[xudong963]

We introduced new APIs in #16461 and more importantly removed the existing SchemaAdapter. I am not sure how upgrading other systems has gone but if they relied on custom SchemaAdapters they'll be broken. I think we should restore the default SchemaAdapter and make it possible to choose which path you use. To get there I think we need at least part of #15057 and to revert the change of removing SchemaAdapter in the original PR. I'm sorry if this causes a delay, I'll get to it ASAP on Tuesday.

I'm okay to wait for these changes. It's always good to not break users.

[alamb]

I'm okay to wait for these changes. It's always good to not break users.

Sounds good -- I added it to the checklist on the description of this ticket

[adriangb]

I'm okay to wait for these changes. It's always good to not break users.

We're trying to minimize breakage! Thanks to you and everyone else for your your patience as we make these exciting but difficult improvements.

[adriangb]

xpost: #15057 (comment)

[adriangb]

I've opened #16791 to address backwards compatibility

[alamb]

I am hoping to start with some of the various release tasks tomorrow (like ensuring hte upgrade guide is in a good place) but I have been very busy with other things recently. Hopefully we can sort out #16791 soon and get the relese process started

[shehabgamin]

Almost missed this thread. Will start testing when there is a 49 branch!

[alamb]

I am still behind -- i will likely start in earnest on this task tomorrow (Friday) and over the weekend and hopefully we'll be reado to cut a Release Candidate (RC) early next week