Metadata scanning performance

[sergiimk]

Options:

• Read-through local FS cache
  • Implemented
• Skip-lists
  • RFC: Metadata Chain Skip Lists
• Global in-memory cache (in-process)
• Extended summaries

Tasks:

• Rank options based on impact
• See how we will measure improvements (logs / metrics?)
• Implement top improvement
• Persistent storage of dataset dependencies graph kamu-cli#854
• Refactoring: Separate planning and execution phases in key dataset manipulation services kamu-cli#984
• Isolate low-level dataset storage repositories in a new crate and name key traits to emphasize STORAGE aspect kamu-cli#979
• Refactoring: avoid outbox-based integration of basic dataset lifecycle events and datasets domain kamu-cli#980
• Use SQLite database in single-tenant Kamu-CLI case too kamu-cli#981
• Store HEAD reference in persistent storage kamu-cli#978
• Store key dataset blocks in database for fast visiting in typical scenarios kamu-cli#986
• SHOW TABLES query times out after a minute kamu-cli#857
• Refactoring: get rid of DatasetSummary as a repository file, store these key properties in the database kamu-cli#983
• Refactoring: RemoteAliasesRegistry, RemoteAliasResolver, and RemoteRepositoryRegistry should exist only in CLI layers kamu-cli#985
• Improve telemetry for repository access kamu-cli#855
• Ineffeciencies when listing dataset flows via GraphQL query kamu-cli#856