Writing manifest files outside of the write thread

[evenyag]

What type of enhancement is this?

Performance

What does the enhancement do?

As mentioned in #3447 (comment), we write manifest files inside the region worker thread (the write thread) and is likely to block the thread for a long time if there are many flush jobs.

We should move this operation to the background thread if possible and this should reduce the block time of the worker.

Implementation challenges

Once we write manifests outside of the worker thread, we must find a way to serialize the following operations:

• checks whether a region is active (not been closed, dropped)
• writes the manifest file
• gets current region version
• apply edits to the region version

Steps

• refactor: Move manifest manager lock to MitoRegion #3689
• Spawns a task to write the manifest feat: write manifests in background tasks #3709

[evenyag]

If we update the manifest in the background, we should ensure the background flush and compaction don't corrupt the manifest. For example, a flush job updates the manifest after a truncate action is written to the manifest. Then we might see data still exist after truncation.

Fortunately, flush and compaction don't break most operations:

• ALTER
  • We always flush all memtables before altering the flush job is already finished
  • Compaction doesn't interrupt existing alteration as they only modify existing files
• DROP
  • We write the drop marker so changing the manifest doesn't affect it
• TRUNCATE
  • The only operation we need to consider is truncating the region
  • We can detect whether a region is truncated and whether a RegionEdit is created before truncation. Then we can ignore these edits.
• SET READONLY
  • We can set and check the state of the region inside the manifest lock
  • We don't update the manifest if the region is read-only
• EDIT
  • This is protected by manifest lock
  • We only expect the Edit action add files

To alter, drop, edit, or truncate a region, we set the state in the worker loop. After that, the region won't be able to process other requests until the background manifest job is done.

[evenyag]

There is a remaining issue: If a region is editing the manifest, then sending another RegionEdit to the region will get a RegionState error. A more user-friendly way would be queueing the request and executing it after the current edit request is done.