Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cold start performance #367

Open
jchris opened this issue Nov 22, 2024 · 0 comments
Open

cold start performance #367

jchris opened this issue Nov 22, 2024 · 0 comments

Comments

@jchris
Copy link
Contributor

jchris commented Nov 22, 2024

related to #125

this issue is to track the idea of segmenting the dataset so the reader can pull just the current read only set, the log, or even the hinted result of 1st paint, as a single GET.

From discord regarding v0.17 in March:

Performance seems suitable up to about 100k encrypted events in a database, at which point cold start time becomes the next optimization frontier.

If we output the Pail and the event log in different streams we can snapshot the Pail root for quick cold loading. We can also split the document bodies into their own stream. We'll want to do level-based compaction so we aren't compacting the whole thing as frequently, and we can add policy so that events outside of the finality window are left for archive but not synced.

With these improvements, we should expect the cold start performance of the database to be a function of the number of records, not the number of events.

The frontier will then become lazy loading of the CRDT root Pail, so big datasets can query before full load.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant