feat: write manifests in background tasks

[evenyag]

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

• Writing manifest files outside of the write thread #3456

What's changed and what's your intention?

This PR spawns a background task to write the manifest file.

It replaces the writable flag with RegionState to represent the state of the region.

/// State of the region.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum RegionState {
    /// The region is opened but is still read-only.
    ReadOnly,
    /// The region is opened and is writable.
    Writable,
    /// The region is altering.
    Altering,
    /// The region is dropping.
    Dropping,
    /// The region is truncating.
    Truncating,
    /// The region is handling a region edit.
    Editing,
}

The region rejects write requests while it is not in the writable state. This avoids the region handling other requests while updating the manifest in the background.

It wraps the manifest manager and the region state into a new struct ManifestContext. Before updating the manifest, the context will also check the region state.

struct ManifestContext {
    /// Manager to maintain manifest for this region.
    manifest_manager: tokio::sync::RwLock<RegionManifestManager>,
    /// The state of the region. The region checks the state before updating
    /// manifest.
    state: AtomicCell<RegionState>,
}

We need to ensure the region version is updated while holding the write lock of the manifest manager. So the context provides a method update_manifest() to update the manifest and apply the update to the region version.

Flush and Compaction

Since the region flushes and compacts files in the background job, it can update the manifest in the background directly.

Truncate

It rewrites the implementation of truncate to spawn a background job to write the manifest (See handle_manifest_truncate_action()). After the manifest is updated, the background job sends the TruncateResult back to the worker loop.

As the manifest action created by the flush/compaction job before truncation should be ignored, the method ManifestContext::update_manifest() needs to handle it correctly.

Alter

It sets the state and then spawns a background job to write the manifest.

Regoin Edit

Similar to alteration, the worker spawns a background job to do this.

Drop

Drop doesn't update the manifest so we still write the marker file in the worker loop to avoid this PR getting to large. But it still sets the region state.

Checklist

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.
• This PR does not require documentation updates.

[codecov]

Codecov Report

Attention: Patch coverage is 79.88827% with 108 lines in your changes are missing coverage. Please review.

Project coverage is 85.23%. Comparing base (3e9bda3) to head (acbd3c5).
Report is 30 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3709      +/-   ##
==========================================
- Coverage   85.43%   85.23%   -0.20%     
==========================================
  Files         966      947      -19     
  Lines      162088   160217    -1871     
==========================================
- Hits       138482   136567    -1915     
- Misses      23606    23650      +44     

[waynexia]

The region rejects write requests while it is not in the writable state. This avoids the region handling other requests while updating the manifest in the background.

Haven't gone through the code. It sounds like the write will be rejected if the region is flushing, right? We may need to distinguish different non-writable statuses. Read-only can reject write requests, but holding and retrying later for flush/compact looks more reasonable.

[evenyag]

The region rejects write requests while it is not in the writable state. This avoids the region handling other requests while updating the manifest in the background.

Haven't gone through the code. It sounds like the write will be rejected if the region is flushing, right? We may need to distinguish different non-writable statuses. Read-only can reject write requests, but holding and retrying later for flush/compact looks more reasonable.

No, flush/compaction won't change the region state. The region is still in the writable state.

[v0y4g3r]

LGTM