Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimise segment ref scans #459

Closed
kjnilsson opened this issue Jul 22, 2024 · 1 comment
Closed

Optimise segment ref scans #459

kjnilsson opened this issue Jul 22, 2024 · 1 comment
Milestone

Comments

@kjnilsson
Copy link
Contributor

kjnilsson commented Jul 22, 2024

Is your feature request related to a problem? Please describe.

Segment refs are kept in the ra_log_reader in a list. To find the first index in the log the entire list needs to be scanned. For some uses, such as quorum queues, messages that are consumed are always near the beginning of the list of segments thus it makes sense to try to optimise this part.

Describe the solution you'd like

Use a skiplist. Care needs to be taken such that we don't accidently read overwritten entries that have the same index but a lower term.

Skiplists in erlang are very memory efficient due to structural sharing of lists and are relatively simple to implement.

a new module: ra_log_segrefs should be introduced to hold the logic for the list of segment references.

@kjnilsson kjnilsson added this to the 2.16.0 milestone Dec 2, 2024
@kjnilsson
Copy link
Contributor Author

Done in #487

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant