Inconsistency in the last checkpoint metadata field naming: snake_case used instead of camelCase

[hackintoshrao]

While investigating snapshot creation for earlier versions (PR #322), I uncovered an inconsistency in how checkpoint metadata fields are named in the _last_checkpoint file.

Current Behavior:
The _last_checkpoint file contains fields in snake_case format. For example:

{"size":8,"size_in_bytes":21857,"version":1}

Expected Behavior:
According to the Delta Protocol specification, these fields should be in camelCase. The expected format should be:

{"size":8,"sizeInBytes":21857,"version":1}

Impact:
This inconsistency causes issues with deserialization when the CheckpointMetadata struct is annotated with #[serde(rename_all = "camelCase")], leading to some fields (like size_in_bytes) being incorrectly set to None.

Current Workaround:
We've temporarily addressed this in PR #322 by adding an alias to the affected field:

#[serde(alias = "size_in_bytes")]
pub(crate) size_in_bytes: Option<i64>,

Proposed Long-term Solution:

1. Investigate why delta-rs is writing checkpoint metadata in snake_case instead of camelCase.
2. Update the checkpoint writing logic to use camelCase field names, aligning with the Delta protocol specification.
3. Update the CheckpointMetadata struct to expect camelCase without needing aliases.
4. If backwards compatibility is required, implement a more robust deserialization that can handle both camelCase and snake_case variants.

Questions to Address:

1. Was the use of snake_case a deliberate choice? If so, what was the reasoning?
2. Are there any backward compatibility concerns we need to consider when fixing this?
3. Are there any other parts of the codebase affected by this inconsistency?

[scovich]

Investigate why delta-rs is writing checkpoint metadata in snake_case instead of camelCase.

My guess: rust clippy enforces snake_case for identifiers, and the json serde will automatically follow that convention unless specifically instructed to convert it to camelCase.

[scovich]

Confirmed: https://github.com/delta-io/delta-rs/blob/main/crates/core/src/table/mod.rs#L42-L46

/// Metadata for a checkpoint file
#[derive(Serialize, Deserialize, Debug, Default, Clone, Copy)]
pub struct CheckPoint {
      ...
    #[serde(skip_serializing_if = "Option::is_none")]
    /// The number of bytes of the checkpoint. This field is optional.
    pub(crate) size_in_bytes: Option<i64>,
    #[serde(skip_serializing_if = "Option::is_none")]
    /// The number of AddFile actions in the checkpoint. This field is optional.
    pub(crate) num_of_add_files: Option<i64>,
}

[feniljain]

Corresponding PR on the delta-rs is merged :)

[zachschuermann]

Given this is fixed in delta-rs now, and the alias never landed in kernel, seems we can close this issue and only read the field as camelCase. Given _last_checkpoint files are rather ephemeral I wouldn't think we need to keep around the ability to read the incorrectly-written field?