Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing snapshot creation for earlier versions than the latest checkpoint #322

Closed
wants to merge 16 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 106 additions & 2 deletions kernel/src/snapshot.rs
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,8 @@ impl Snapshot {
_ => list_log_files(fs_client.as_ref(), &log_url)?,
};

println!("Commit files: {:?}", commit_files.iter().map(|f| f.location.clone()).collect::<Vec<_>>());
println!("Checkpoint files: {:?}", checkpoint_files.iter().map(|f| f.location.clone()).collect::<Vec<_>>());
scovich marked this conversation as resolved.
Show resolved Hide resolved
// remove all files above requested version
if let Some(version) = version {
commit_files.retain(|meta| {
Expand Down Expand Up @@ -301,13 +303,18 @@ fn read_last_checkpoint(
log_root: &Url,
) -> DeltaResult<Option<CheckpointMetadata>> {
let file_path = LogPath::new(log_root).child(LAST_CHECKPOINT_FILE_NAME)?;
println!("Reading last checkpoint from: {}", file_path);
match fs_client
.read_files(vec![(file_path, None)])
.and_then(|mut data| data.next().expect("read_files should return one file"))
{
Ok(data) => Ok(serde_json::from_slice(&data)
Ok(data) => {
// print the data in bytes as a string
println!("Data: {:?}", std::str::from_utf8(&data).unwrap());
Ok(serde_json::from_slice(&data)
.inspect_err(|e| warn!("invalid _last_checkpoint JSON: {e}"))
.ok()),
.ok())
},
Err(Error::FileNotFound(_)) => Ok(None),
Err(err) => Err(err),
}
Expand Down Expand Up @@ -546,4 +553,101 @@ mod tests {
Some(3)
);
}

#[test]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aside: @nicklan -- This file is getting pretty large. I wonder what the rust best practice is for where unit tests should live?

My experience from non-rust projects is that bulky unit tests should generally live in separate (test-only) source files -- especially tests that only use public interfaces. In-file testing would make more sense for exercising private/internal code that isn't accessible outside the source file.

fn test_snapshot_version_0_with_checkpoint_at_version_1() {
let path =
std::fs::canonicalize(PathBuf::from("./tests/data/app-txn-checkpoint/")).unwrap();
let url = url::Url::from_directory_path(path).unwrap();

let engine = SyncEngine::new();

// First, let's verify the content of the _last_checkpoint file
let fs_client = engine.get_file_system_client();
let log_url = LogPath::new(&url).child("_delta_log/").unwrap();
println!("Log root: {}", log_url);
let last_checkpoint = read_last_checkpoint(fs_client.as_ref(), &log_url).unwrap();

assert!(
last_checkpoint.is_some(),
"_last_checkpoint file should exist"
);
let checkpoint_meta = last_checkpoint.unwrap();
println!("Checkpoint metadata: {:#?}", checkpoint_meta);
assert_eq!(
checkpoint_meta.version, 1,
"Last checkpoint should be at version 1"
);
assert_eq!(checkpoint_meta.size, 8, "Checkpoint size should be 8");
// assert_eq!(
// checkpoint_meta.size_in_bytes,
// Some(21857),
// "Checkpoint size in bytes should be 21857"
// );
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why commented out, out of curiosity? Is the byte count wrong and needs to be updated?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting a None result for checkpoint_meta.size_in_bytes; hence, the test is failing. I commented the code so that I first focus on the snapshot versioning bug! I'm looking into this one, too, now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The root cause of the issue is a mismatch between the casing conventions used in the JSON data and the struct definition. The struct is expecting camelCase (e.g., sizeInBytes) due to the #[serde(rename_all = "camelCase")] attribute, but the actual JSON data is using snake_case (size_in_bytes).

Your short-term fix of adding an alias is a good temporary solution:

#[serde(alias = "size_in_bytes")]
pub(crate) size_in_bytes: Option<i64>,

However, for a long-term solution, we need to address the inconsistency between the Delta protocol specification and the implementation in delta-rs (I built my test data from delta-rs, assuming that the other test data was also created using the same, including the last_checkpoint file).

According to the Delta protocol documentation, the last checkpoint file schema should indeed use camelCase for field names. The fact that delta-rs is writing the metadata in snake_case suggests a deviation from the protocol specification.

The long-term solution should involve:

  1. Align the delta-rs implementation with the Delta protocol specification by ensuring the fields in the last checkpoint file are written using camelCase field names.
  2. Updating the CheckpointMetadata struct to expect camelCase without needing aliases.
  3. If any backward compatibility is required, consider implementing a more robust deserialization to handle camelCase and snake_case variants.

It would be worth investigating why delta-rs is writing the metadata in snake_case contrary to the protocol specification.

What are your thoughts @scovich ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created a new issue to move this conversation #326 . Would love to know your thoughts on the issue link.


// Now, request snapshot at version 0
let snapshot = Snapshot::try_new(url.clone(), &engine, Some(0));

match snapshot {
Ok(snap) => {
assert_eq!(snap.version(), 0, "Snapshot version should be 0");

// Verify that the snapshot contains the correct files
assert_eq!(
snap.log_segment.commit_files.len(),
1,
"There should be one commit file"
);
assert_eq!(
LogPath::new(&snap.log_segment.commit_files[0].location).version,
Some(0),
"The commit file should be version 0"
);

assert!(
snap.log_segment.checkpoint_files.is_empty(),
"Snapshot for version 0 should not contain checkpoint files"
);

// You might want to add more assertions here about the content of version 0
}
Err(e) => {
panic!("Failed to create snapshot for version 0: {:?}", e);
}
}

// Verify the snapshot at version 1 (the checkpoint version)
let snapshot_1 = Snapshot::try_new(url, &engine, Some(1)).unwrap();
assert_eq!(snapshot_1.version(), 1, "Snapshot version should be 1");
assert_eq!(
snapshot_1.log_segment.checkpoint_files.len(),
1,
"There should be one checkpoint file for version 1"
);
assert_eq!(
LogPath::new(&snapshot_1.log_segment.checkpoint_files[0].location).version,
Some(1),
"The checkpoint file should be version 1"
);
}

#[test]
fn test_snapshot_with_version_less_than_latest_checkpoint() {
let path = std::fs::canonicalize(PathBuf::from("./tests/data/multiple-checkpoint/")).unwrap();
let url = url::Url::from_directory_path(path).unwrap();

let engine = SyncEngine::new();

// Attempt to create a snapshot at version 10
let result = Snapshot::try_new(url, &engine, Some(10));

// Check if the operation succeeded
assert!(result.is_ok(), "Expected snapshot creation to succeed for version 10");

let snapshot = result.unwrap();

// Verify the snapshot properties
assert_eq!(snapshot.version(), 10, "Snapshot version should be 10");

}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}
{"metaData":{"id":"7f63b305-b7d1-4151-ac85-770932b6582d","name":null,"description":null,"format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"id\",\"type\":\"integer\",\"nullable\":false,\"metadata\":{}},{\"name\":\"value\",\"type\":\"string\",\"nullable\":false,\"metadata\":{}}]}","partitionColumns":[],"createdTime":1725530488008,"configuration":{}}}
{"add":{"path":"part-00001-c7fed910-a531-4273-9f8c-80b935c60baf-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488019,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"value\":\"F7jKTLb\",\"id\":14},\"maxValues\":{\"id\":60,\"value\":\"x21RiYO\"},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488020,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-4e6bc849-0634-49f9-98cc-6239fcaadec5-c000.snappy.parquet","partitionValues":{},"size":858,"modificationTime":1725530488034,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"id\":33,\"value\":\"5FSXDXs\"},\"maxValues\":{\"id\":85,\"value\":\"YGU3NZT\"},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488034,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-c2a2c39f-d19a-41bd-b6b7-d7ca99cac849-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488043,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"value\":\"3IZIUyp\",\"id\":11},\"maxValues\":{\"value\":\"uRjG8dk\",\"id\":69},\"nullCount\":{\"value\":0,\"id\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488043,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-e984043f-f3ee-4104-b60f-faceb1f04126-c000.snappy.parquet","partitionValues":{},"size":858,"modificationTime":1725530488051,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"id\":22,\"value\":\"Gn9NBv0\"},\"maxValues\":{\"id\":92,\"value\":\"UfzjcgL\"},\"nullCount\":{\"value\":0,\"id\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488051,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-1092d017-2263-4d47-a73d-7b8903f79ac5-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488059,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"value\":\"AGffh52\",\"id\":35},\"maxValues\":{\"value\":\"o3kV5mT\",\"id\":97},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488059,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-bad64600-079d-48c4-b6f7-f6a256427182-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488079,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"id\":7,\"value\":\"6ouIqI8\"},\"maxValues\":{\"id\":81,\"value\":\"lzN28es\"},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488079,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-3b66979d-47ab-4fbd-9f16-2a00037eb098-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488097,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"value\":\"30kBets\",\"id\":10},\"maxValues\":{\"id\":78,\"value\":\"enH9RLJ\"},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488098,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-ffabbcd0-c86d-4c96-a08e-8c20fcb05b56-c000.snappy.parquet","partitionValues":{},"size":865,"modificationTime":1725530488110,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"value\":\"7NRr55w\",\"id\":51},\"maxValues\":{\"value\":\"akAH8Pj\",\"id\":85},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488111,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-c0519cba-e509-4ca7-8e58-e70f5a286e0a-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488124,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"id\":37,\"value\":\"3gPueyH\"},\"maxValues\":{\"id\":98,\"value\":\"vQVIbcI\"},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488124,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-3816edee-8d6c-4cd3-8645-d757b5d17d41-c000.snappy.parquet","partitionValues":{},"size":865,"modificationTime":1725530488137,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"id\":40,\"value\":\"3GVAdf4\"},\"maxValues\":{\"value\":\"ud06xMt\",\"id\":95},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488137,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-db982975-da7b-4fcd-a613-a3494678c23e-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488159,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"value\":\"FKOg7MV\",\"id\":7},\"maxValues\":{\"id\":54,\"value\":\"sdIVoKi\"},\"nullCount\":{\"value\":0,\"id\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488159,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-bb0db258-0bc6-4b5c-8c81-3df5d2245111-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488172,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"value\":\"5NjpmVx\",\"id\":4},\"maxValues\":{\"id\":87,\"value\":\"pCpzDPj\"},\"nullCount\":{\"value\":0,\"id\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488172,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-a723f04e-f49e-4881-aab4-bc7cd4677532-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488186,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"value\":\"15SMv3m\",\"id\":21},\"maxValues\":{\"id\":76,\"value\":\"acZXfvx\"},\"nullCount\":{\"value\":0,\"id\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488186,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-e999f2d7-4989-43e4-8e7c-76aa339cd42e-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488201,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"id\":31,\"value\":\"Zirdxyk\"},\"maxValues\":{\"id\":71,\"value\":\"njG7WMg\"},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488201,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-39fd00ae-0a83-4e5b-b555-c1ca23b47339-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488216,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"value\":\"ACgqyBV\",\"id\":29},\"maxValues\":{\"value\":\"pK4bDUM\",\"id\":71},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488216,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-52dc2232-f68a-4e2e-9545-1eee3ee3f1e3-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488241,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"value\":\"37WlqED\",\"id\":4},\"maxValues\":{\"id\":77,\"value\":\"T9Hdxe1\"},\"nullCount\":{\"value\":0,\"id\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488241,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-5e9ae675-eb91-4612-ae5a-035a0f734b75-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488256,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"id\":6,\"value\":\"6CEq9q7\"},\"maxValues\":{\"value\":\"h5QUhgd\",\"id\":76},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488256,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-163a714c-5d25-40a0-82d7-7e8751b28118-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488271,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"value\":\"8ERf8LC\",\"id\":4},\"maxValues\":{\"id\":77,\"value\":\"NYaGoaY\"},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488271,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-e34c711f-c5c6-4e79-ac62-021ebbd3c5c3-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488285,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"id\":53,\"value\":\"66dZ0uI\"},\"maxValues\":{\"value\":\"yxSk13R\",\"id\":90},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488285,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-e659a621-cba2-4f7e-b130-ece5ec644e2e-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488300,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"value\":\"OP5jNUt\",\"id\":49},\"maxValues\":{\"id\":71,\"value\":\"rkdUHdl\"},\"nullCount\":{\"value\":0,\"id\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488300,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-5714f047-47c8-4d1b-af9b-c84e0e65d19e-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488325,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"value\":\"2kT4Ljr\",\"id\":4},\"maxValues\":{\"id\":87,\"value\":\"IFtkbKc\"},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488325,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-19ca3da4-53af-4324-a641-b35c23e08a08-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488340,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"id\":24,\"value\":\"1r7ne1k\"},\"maxValues\":{\"id\":94,\"value\":\"eLWH0ms\"},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488341,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"add":{"path":"part-00001-651054c3-2a1c-4758-9a2b-1dcb29002e99-c000.snappy.parquet","partitionValues":{},"size":864,"modificationTime":1725530488356,"dataChange":true,"stats":"{\"numRecords\":4,\"minValues\":{\"id\":25,\"value\":\"2JkqfIN\"},\"maxValues\":{\"id\":76,\"value\":\"bOjhfiD\"},\"nullCount\":{\"id\":0,\"value\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}}
{"commitInfo":{"timestamp":1725530488356,"operation":"WRITE","operationParameters":{"mode":"Append"},"clientVersion":"delta-rs.0.19.0"}}
Loading
Loading