Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sqld: disable checkpoint on primary conn create #1898

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

LucioFranco
Copy link
Contributor

This commit changes our primary connection initialization code in two ways to achieve the ability to disable checkpointing the wal.

  1. We ignore the initial checkpoint that we call directly into sqlite3
    before we restore from bottomless. There is a fixme above that
    explains why we need this but to me right now its not totally clear
    why without digging deeper into the internals of bottomless. We
    should do this but for the moment this unblocks us and from the fixme
    comment it does not sound unsafe rather doing extra work potentially.

  2. When bottomless needs to get the local change counter is creates a
    sqlite connection. When this connection drops it seems like it
    checkpoints the wal. I took a brief look at the sqlite3_close code
    and did not find anything obvious, I have been told in the past that
    sqlite3 likes to checkpoint at weird points so this could be one of
    those. For the moment, the temporary fix like above is to
    std::mem::forget the connection so that Drop never gets called
    and thus sqlite3_close never gets called.

With both of these changes we now don't checkpoint the wal unless we hit the max size or interval (which for testing I have set very high).

@LucioFranco LucioFranco force-pushed the lucio/disable-checkpointing-bottomless branch from 28c5ef6 to c8673f4 Compare December 31, 2024 21:00
This commit changes our primary connection initialization code in two
ways to achieve the ability to disable checkpointing the wal.

1) We ignore the initial checkpoint that we call directly into sqlite3
   before we restore from bottomless. There is a fixme above that
   explains why we need this but to me right now its not totally clear
   why without digging deeper into the internals of bottomless. We
   should do this but for the moment this unblocks us and from the fixme
   comment it does not sound unsafe rather doing extra work potentially.

2) When bottomless needs to get the local change counter is creates a
   sqlite connection. When this connection drops it seems like it
   checkpoints the wal. I took a brief look at the `sqlite3_close` code
   and did not find anything obvious, I have been told in the past that
   sqlite3 likes to checkpoint at weird points so this could be one of
   those. For the moment, the temporary fix like above is to
   `std::mem::forget` the connection so that `Drop` never gets called
   and thus `sqlite3_close` never gets called.

With both of these changes we now don't checkpoint the wal unless we hit
the max size or interval (which for testing I have set very high). This
changes are not enabled by default but must be enabled by setting the
following env var:

```
LIBSQL_DISABLE_INIT_CHECKPOINTING=1
```
@LucioFranco LucioFranco force-pushed the lucio/disable-checkpointing-bottomless branch from c8673f4 to a807731 Compare December 31, 2024 21:00
@LucioFranco LucioFranco marked this pull request as ready for review December 31, 2024 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant