-
Notifications
You must be signed in to change notification settings - Fork 56
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor!: store inscription data in postgres instead of sqlite #375
base: develop
Are you sure you want to change the base?
Conversation
.await | ||
.map_err(|e| format!("unable to begin pg transaction: {e}"))?; | ||
{ | ||
let transaction_ref: &'a Transaction<'a> = unsafe { std::mem::transmute(&transaction) }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks spooky, I'll just assume it's all good
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, good catch, I remember doing that to get over some compiler thing. I'll take another look to see if there's a better way to handle the Transaction
lifetime
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a few pieces of code implemented here that I'm a bit surprised isn't available/usable from existing postgres related crates (e.g. u64<->numeric handling).
But from a quick review, this looks great!
This PR is a very extensive refactor that migrates the underlying inscription data storage engine from a local SQLite database to a PostgreSQL database. This is a breaking change that requires a full reindex of the entire Bitcoin inscription history. Some of the most significant changes are:
chainhook-postgres
A new component called
chainhook-postgres
is created which contains tools for working with pg databases such as:u64
,u128
) into PGnumeric
columnswith_pg_transaction
)This is meant to be a shared component which will be moved to the chainhook repo in the future once all indexers are moved to postgres.
DB schema upgrades
This refactor creates a full DB schema that stores all inscription data, including cache tables for counts, ownership pointers, etc. Previously, SQLite only stored a partial data set that needed to be complemented with a bitcoind RPC call in order to serve the inscription data to predicates. This will no longer be necessary.
More importantly, since we now have a full DB schema the Ordinals API will now be able to read directly from this DB without a re-ingestion in order to serve its endpoints. This also applies to the BRC-20 index.
To configure the new database connections, these (example) entries will need to be added to the
.toml
config file:This also renders the
ordhook-sdk-js
component obsolete.Forced linear indexing
Starting with this update, ordhook will only be able to index in sequence, without skipping blocks, and starting at the current chain tip. This means that per-block scanning, per-transaction scanning, etc. is now disabled.
This change was done to simplify the codebase, to make a global index state easier to manage, and to make DevOps tasks more predictable.
Update to Rust 1.80.1
Ordhook is now upgraded to Rust 1.80.1.
CI upgrades
Since we now use Postgres, the CI was upgraded to include an integration environment with Docker+Postgres.
Closes #374