Update DataFusion to 43

[Dandandan]

Which issue does this PR close?

Closes #1124

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

[andygrove]

Thanks @Dandandan

[Dandandan]

I can succesfully run the tpc-h queries, let's wait until build is green.

[milenkovicm]

I have mixed feelings about keeping that test which involves ballista python, should we remove it and risk breaking python from time to time? wdyt @Dandandan @andygrove

[Epicism]

PoV: Shouldn't the Python code be in its own repo? Ballista is being redesigned as a library, and Python is an application of the library...

…

On Wed, Nov 20, 2024 at 4:05 PM Marko Milenković ***@***.***> wrote: I have mixed feelings about keeping that test which involves ballista python, should we remove it and risk breaking python from time to time? wdyt @Dandandan <https://github.com/Dandandan> @andygrove <https://github.com/andygrove> — Reply to this email directly, view it on GitHub <#1125 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACNEIPJOQLOCMX5DNZEFQGL2BT2P7AVCNFSM6AAAAABSFBQZ3GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBZGUZTSMRTGI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[milenkovicm]

once we merge this patch, ballista will read deltalake (custom codecs needed)

  let config = SessionConfig::new_with_ballista()
      .with_target_partitions(1)
      .with_ballista_standalone_parallelism(2)
      .with_ballista_logical_extension_codec(Arc::new(BallistaDeltaLogicalCodec::default()))
      .with_ballista_physical_extension_codec(Arc::new(BallistaDeltaPhysicalCodec::default()));

  let state = SessionStateBuilder::new()
      .with_config(config)
      .with_default_features()
      .build();

  let table = deltalake::open_table("/Users/ballissta/TMP/delta_table/")
      .await
      .unwrap();

  let ctx = SessionContext::standalone_with_state(state).await?;

  ctx.register_table("demo", Arc::new(table)).unwrap();

  let df = ctx.sql("select * from demo").await?;

  df.show().await?;

relates to #457

[tbar4]

PoV: Shouldn't the Python code be in its own repo? Ballista is being redesigned as a library, and Python is an application of the library...
…
On Wed, Nov 20, 2024 at 4:05 PM Marko Milenković @.> wrote: I have mixed feelings about keeping that test which involves ballista python, should we remove it and risk breaking python from time to time? wdyt @Dandandan https://github.com/Dandandan @andygrove https://github.com/andygrove — Reply to this email directly, view it on GitHub <#1125 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNEIPJOQLOCMX5DNZEFQGL2BT2P7AVCNFSM6AAAAABSFBQZ3GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBZGUZTSMRTGI . You are receiving this because you are subscribed to this thread.Message ID: @.>

We could split it into its own repo, but right now it seems like it is just Marko, Andy, Daniel, and I that have been making updates. If Ballista Python gets more support than me there could be an argument for it, but at this time it is better to manage it all in one place.

[tbar4]

https://github.com/apache/datafusion-ballista/actions/runs/11938679849/job/33278199638?pr=1125#step:8:1494

pyballista will be changing to ballista

[milenkovicm]

With (py)ballista issues outlined in #1142 would it make sense to release this version without it? That would give us some time to figure out python support?

Wdyt @andygrove ?

[milenkovicm]

@Dandandan could you please rebase this PR now when we have datafusion python released ?
(don't worry if you have time to fix py part, i can follow up with PR later )

[Dandandan]

@milenkovicm This is now up to date with main

[milenkovicm]

Thanks @Dandandan!

Would you be able to merge this and other two open PR, looks like @andygrove is busy and I have no merge rights

[Dandandan]

Would you be able to merge this and other two open PR, looks like @andygrove is busy and I have no merge rights

Yes, done!