Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor!: store inscription data in postgres instead of sqlite #375

Open
wants to merge 59 commits into
base: develop
Choose a base branch
from

Conversation

rafaelcr
Copy link
Collaborator

@rafaelcr rafaelcr commented Nov 26, 2024

This PR is a very extensive refactor that migrates the underlying inscription data storage engine from a local SQLite database to a PostgreSQL database. This is a breaking change that requires a full reindex of the entire Bitcoin inscription history. Some of the most significant changes are:

chainhook-postgres

A new component called chainhook-postgres is created which contains tools for working with pg databases such as:

  • Automatic connection pool management and reuse
  • Support for large integer number encoding (u64, u128) into PG numeric columns
  • Easy to use closures for automatic transaction commits/rollbacks (with_pg_transaction)
  • Easy to use methods for connecting to a DB with optional retry

This is meant to be a shared component which will be moved to the chainhook repo in the future once all indexers are moved to postgres.

DB schema upgrades

This refactor creates a full DB schema that stores all inscription data, including cache tables for counts, ownership pointers, etc. Previously, SQLite only stored a partial data set that needed to be complemented with a bitcoind RPC call in order to serve the inscription data to predicates. This will no longer be necessary.

More importantly, since we now have a full DB schema the Ordinals API will now be able to read directly from this DB without a re-ingestion in order to serve its endpoints. This also applies to the BRC-20 index.

To configure the new database connections, these (example) entries will need to be added to the .toml config file:

[ordinals_db]
database = "ordinals"
host = "localhost"
port = 5432
username = "postgres"
password = "postgres"
search_path = "public"

[brc20_db]
database = "brc20"
host = "localhost"
port = 5432
username = "postgres"
password = "postgres"
search_path = "public"

This also renders the ordhook-sdk-js component obsolete.

Forced linear indexing

Starting with this update, ordhook will only be able to index in sequence, without skipping blocks, and starting at the current chain tip. This means that per-block scanning, per-transaction scanning, etc. is now disabled.

This change was done to simplify the codebase, to make a global index state easier to manage, and to make DevOps tasks more predictable.

Update to Rust 1.80.1

Ordhook is now upgraded to Rust 1.80.1.

CI upgrades

Since we now use Postgres, the CI was upgraded to include an integration environment with Docker+Postgres.

Closes #374

@rafaelcr rafaelcr linked an issue Dec 4, 2024 that may be closed by this pull request
@rafaelcr rafaelcr marked this pull request as ready for review December 4, 2024 03:57
@rafaelcr rafaelcr had a problem deploying to Development-mainnet-blue December 4, 2024 04:16 — with GitHub Actions Error
@rafaelcr rafaelcr had a problem deploying to Development-mainnet-blue December 4, 2024 16:57 — with GitHub Actions Error
@rafaelcr rafaelcr had a problem deploying to Development-mainnet-blue December 4, 2024 20:39 — with GitHub Actions Error
@rafaelcr rafaelcr had a problem deploying to Development-mainnet-blue December 5, 2024 16:05 — with GitHub Actions Error
@rafaelcr rafaelcr had a problem deploying to Development-mainnet-blue December 5, 2024 21:00 — with GitHub Actions Error
@rafaelcr rafaelcr had a problem deploying to Development-mainnet-blue December 5, 2024 23:25 — with GitHub Actions Error
.await
.map_err(|e| format!("unable to begin pg transaction: {e}"))?;
{
let transaction_ref: &'a Transaction<'a> = unsafe { std::mem::transmute(&transaction) };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks spooky, I'll just assume it's all good

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, good catch, I remember doing that to get over some compiler thing. I'll take another look to see if there's a better way to handle the Transaction lifetime

Copy link
Member

@zone117x zone117x left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a few pieces of code implemented here that I'm a bit surprised isn't available/usable from existing postgres related crates (e.g. u64<->numeric handling).

But from a quick review, this looks great!

@rafaelcr rafaelcr requested a deployment to Development-mainnet-blue December 21, 2024 17:47 — with GitHub Actions Waiting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: 👀 In Review
Development

Successfully merging this pull request may close these issues.

Migrate ordhook's SQLite database to postgres
2 participants