Dec 13, 2024: This week(s) in DataFusion

[alamb]

Introduction

This ticket is a weekly-ish summary of interesting things happening in DataFusion. Note this is not a complete list (it is what I remember / can find). Please feel free to leave comments on this ticket about things that I may have missed or you think should get wider attention by the community. Follow on to #13630

Loosely inspired by https://this-week-in-rust.org/

Reminder, find new content (and please post some!) to

• Concepts, Readings, Events page

Community Highlights

• @Rachelint became a committer!

Theme: DataFusion Fever is Spreading

I think DataFusion is reaching an inflection point: It is now good enough that more than early adopters can and are building real, production systems using DataFusion. This is a great milestone 🎉 and I think the project is adjusting to this new reality.

One major theme we have been discussing in the last week or two is making upgrades easier. The recent pushes in 43.0.0 and 44.0.0 to clean up / complete projects such as StringView, window function migration, improved APIs, etc have caused significant downstream complications upgrading. Going forward as a community, we are discussing ways to improve this process.

I hope to write more on this topic

You can read more about this here

• [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) #13648
• [DISCUSSION] Making it easier to use DataFusion (lessons from GlareDB) #13525

New Blog Website

• @timsaucer has reworked the blog https://datafusion.apache.org/blog/ so that it now autopublishes when changes are made to https://github.com/apache/datafusion-site 🎉

Releases

• DataFusion python 43.0.0 was released: https://crates.io/crates/datafusion-python/43.1.0 🎉
• sqlparser release is in voting phase: Release sqlparser-rs version 0.53.0 / sqlparser_derive 0.3.0 datafusion-sqlparser-rs#1517
• arrow-rs major release Release arrow-rs / parquet major version 54.0.0 (December 2024) arrow-rs#6342
• We are discussing DataFusion 44.0.0: Release DataFusion 44.0.0 #13334
• DataFusion python release: chore: Upgrade to DataFusion 44 datafusion-python#972

Performance

The community loves a good benchmark challenge. We are off to a great start making h20 benchmark even faster, see

• [EPIC] Improved performance in H20.ai benchmarks #13548
• @2010YOUY01 / @Dandandan and @jayzhan211 made corr more than 3x faster 🚀
• @Rachellint is also working on median doing Improve performance of median function #13550

I also also made a change with example to allow array reuse in functions, which adds to the 🚀 🔨 🧰

• Performance: enable array allocation reuse (ScalarFunctionArgs gets owned ColumnReference) #13637

Thanks @dhegberg for a CSV loading benchmark

• Add csv loading benchmarks. #13544

Also thanks to @richox, @Zhangli20, @tlm365 @jayzhan211 @Weijun-H, @comphead and @Dandandan for improving the speed of other functions

• Optimize performance of initcap function (~2x faster) #13691
• Optimize performance of character_length function #13696
• Improve substr() performance by avoiding using owned string #13688
• refactor: replace Vec with IndexMap for expression mappings in ProjectionMapping and EquivalenceGroup #13675

🐛 fixes, and improvements

@onursatici has been on a tear along with @korowa @haohuaijin

• fix union serialisation order in proto #13709

• fix: repartitioned reads of CSV with custom line terminator #13677

• Fix hash join with sort push down #13560

• Thanks to @Eason0729 for Handle alias when parsing sql(parse_sql_expr) #12939

@findepi @jonahgao @comphead and others have been cleaning up the code 🧹

• Reveal implementing type and return type in simple UDF implementations #13730
• Report current operation when coercion fails #13628
• refactor: replace OnceLock with LazyLock #13641
• chore: macros crate cleanup #13685
• refactor: simplify the make_udf_function macro #13712
• chore: clean up dependencies #13728

Unparser

We have been cranking away filling out plan --> SQL feature, thanks to @goldmedal

• Support unparsing UNNEST plan to UNNEST table factor SQL #13660
• Improve unparsing after optimize_projections optimization #13599

Hashbrown

@crepererum has been working to migrate our use of hashbrown to higher level APIs

• Upgrade to hashbrown 0.15.1: migrate from hashbrown::raw::RawTable to hashbrown::hash_table::HashTable #13433
• refactor: change some hashbrown RawTable uses to HashTable (round 3) #13658

Looking to get more involved? Try code review!

(can you see what I did there 🎣 )

DataFusion has a long history of community members contributing in all aspects of the project. Reviewing PRs is an especially great way to get introduced to the project, help the community and grow your own knowledge -- researching and understanding the code enough to review PRs also often inspires additional ideas for improvements.

We have docs about reviews. TLDR is: look for test coverage, if the change is understandable and well documented, and if the code can be improved. When you think the PR looks good to merge, try @ mentioning one of the committers.

Help wanted

• I would love to see the community offer additional help testing, triaging bugs helping to make DataFusion a more stable foundation for building systems

Please feel leave your own comments on this ticket if you are looking for help

Community

• Weekly Call
• Slack/Discord: info links

Upcoming meetups:

• 2024 Dec 18 Chicago: https://lu.ma/eq5myc5i @adriangb @timsaucer
• DISCUSSION: January 2025 DataFusion Meetup in Amsterdam / CIDR 2025 #12988
• 2025 Jan 15 Boston
• 2025 Jan 24 Amsterdam

[alamb]

@Omega359 made some signficant progress towards running the sqllogictest suite from sqlite against datafusion -- see this ticket for more:

• [EPIC] Run full sqllogic / sqlite test suite against DataFusion #13811

[alamb]

I have also started collecting a list of things to improve dev time / experience

• [EPIC] A collection of items to improve developer / CI speed  #13813
• Datafusion binary size has been getting bigger #13816
• Building project takes a *long* time (esp compilation time for datafusion core crate) #13814

[alamb]

Fun project: dbfiddle for DataFusion in browser:

• Browser-accessible official DataFusion playground #13818

[alamb]

Maybe time to ressurect discussion about stable / long term releases

• [Discuss] Release cadence / patch releases / Long Term Supported (lts) minor releases #5269

Is anyone else interested in this? We would likely need help with the additional maintenance burden

[alamb]

@suremarc is cranking along with a materialized view implementation 🚀

• Incremental view maintenance datafusion-contrib/datafusion-materialized-views#3