Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Port cdf tests from delta-spark to kernel #611

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

OussamaSaoudi-db
Copy link
Collaborator

@OussamaSaoudi-db OussamaSaoudi-db commented Dec 20, 2024

What changes are proposed in this pull request?

This PR adds several CDF tests from delta-spark. We check the following:

  • CDF over various version ranges
  • Update operations are read correctly from cdc files
  • data_change=false means the action is skipped
  • A range with start > end is an error.
  • Start version greater than latest table version is an error
  • CDF works on partition tables
  • CDF works on tables with backticks in the column names
  • CDF is correct in deletion cases with unconditional deletes, conditional deletes that remove all rows, and selective conditional deletes.

Table-changes construction is also changed so that CDF version error is checked before snapshots are created. This makes the error message clearer in the case that the start version is beyond the end of the table.

@OussamaSaoudi-db OussamaSaoudi-db changed the title Cdf delta spark tests test: Cdf delta spark tests Dec 20, 2024
Copy link

codecov bot commented Dec 20, 2024

Codecov Report

Attention: Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review.

Project coverage is 83.43%. Comparing base (c3a868f) to head (8be1910).

Files with missing lines Patch % Lines
kernel/src/table_changes/mod.rs 85.71% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #611      +/-   ##
==========================================
- Coverage   83.45%   83.43%   -0.02%     
==========================================
  Files          74       74              
  Lines       16877    16878       +1     
  Branches    16877    16878       +1     
==========================================
- Hits        14084    14083       -1     
  Misses       2135     2135              
- Partials      658      660       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@OussamaSaoudi-db OussamaSaoudi-db changed the title test: Cdf delta spark tests test: Port cdf tests from delta-spark to kernel Dec 20, 2024
@OussamaSaoudi-db OussamaSaoudi-db marked this pull request as ready for review December 20, 2024 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants