Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/ffi enter tokio runtime #13937

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

timsaucer
Copy link
Contributor

Which issue does this PR close?

Closes #13827
Closes #13851

Rationale for this change

During testing with delta-kernel-rs and datafusion-table-providers it was discovered that there are multiple cases where table providers are spawning processes using tokio. Since these modules may be running in a different thread than the main executable, they do not have the tokio runtime. Here we are optionally adding in a tokio runtime reference so we an enter the runtime during calls to the record batch stream.

In a recent change to DataFusion core, there has been a required change in the API for operating over the FFI boundary. With this change, it will break when someone attempts to load a module from DF43 into DF44. This PR addresses this by adding a version() function that will return the major version for which the module was compiled against. This will allow users to check for compatibility before attempting to use modules across the boundary.

What changes are included in this PR?

We add a reference to the tokio runtime that is entered during calls to the record batch stream so we can spawn async processes during execution without panicing.

We add a single version() function at the root of the FFI API.

We add an integration test that builds a module and loads it from another executable to test the FFI boundary. This has both a synchronous and an asynchronous table provider.

Are these changes tested?

These are tested against datafusion-python with a python wrapper around delta-kernel-rs. Additionally an integration test is added to this repository.

Are there any user-facing changes?

This exposes a new version() function in the FFI module and adds an new method for creating FFI_TableProvider with the runtime. The existing From<> implementation still remains if the user is not spawning via tokio.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

FFI Execution Plans that spawn threads panic Add version checking to FFI crate
1 participant