Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactoring: avoid outbox-based integration of basic dataset lifecycle events and datasets domain #980

Open
zaychenko-sergei opened this issue Dec 9, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request rust Pull requests that update Rust code

Comments

@zaychenko-sergei
Copy link
Contributor

Blocked with #979

Currently we have a misalignment of responsibilities between core and datasets domains, when it comes to server basic dataset use cases, such as:

  • create dataset
  • crate dataset via snapshot
  • rename dataset
  • delete dataset

Right now the implementation of use cases is fully concentrated in infra/core crate, and the datasets domain handlers, such as dataset entry maintainance, dependency graph updates, and future planned actions like #978, are integrated via Outbox events of immediate delivery type.

This feels misaligned, as basically we are talking about the very same domain, and it should not be that difficult to synchronously update both the storage and the datasets domain records.

These key datasetuse cases already abstract away from storage system via DatasetRepositoryWriter trait, and currently DatasetRepositoryLocalFs and DatasetRepositoryS3 implement it directly.

Proposal: provide an alternative implementation for DatasetRepositoryWriter in datasets domain, which would:

  • make CRUD operations on dataset entries directly, instead of expecting immediate events
  • make pre-validations against the database vision first (storage versions should be simplified, i.e. naming collision detection)
  • update dependency graph directly
  • set initial HEAD reference in database when creating dataset in the same transaction as creating a dataset entry
  • invoke storage repositories to do their part within the same transaction
@zaychenko-sergei zaychenko-sergei added enhancement New feature or request rust Pull requests that update Rust code labels Dec 9, 2024
@zaychenko-sergei zaychenko-sergei self-assigned this Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request rust Pull requests that update Rust code
Projects
None yet
Development

No branches or pull requests

1 participant