Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add new getcfilters message for utreexo nodes #192

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Add new getcfilters message for utreexo nodes #192

wants to merge 5 commits into from

Conversation

alainjr10
Copy link

This is meant to be just a draft PR

Add support for new getcfilters message called UtreexoCFilter
When a new block is added, a new UtreexoCFilter filter is created and persisted on the node.
This filter is created by serializing the contents of the roots at that height.
When a node receives a getcfilters message, the logic doesn't change much, it handles it almost identical to how it deals with basic filters.

@alainjr10 alainjr10 marked this pull request as draft June 28, 2024 15:11
kcalvinalvin
kcalvinalvin reviewed Jul 2, 2024
View reviewed changes
Copy link
Contributor

@kcalvinalvin kcalvinalvin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think maybe creating a new index is a better way of approaching this. With the new index you'd only store the filter header.

blockchain/chainio.go Show resolved Hide resolved
blockchain/indexers/cfindex.go Show resolved Hide resolved
wire/msgcfilter.go Show resolved Hide resolved
btcutil/gcs/builder/builder.go Show resolved Hide resolved
blockchain/indexers/utreexoproofindex.go Outdated Show resolved Hide resolved
@kcalvinalvin
Copy link
Contributor

Could you make the commits atomic?

Ideally all the changes to each of the packages should be committed together with each commit achieving one single thing.

The change to btcutil/gcs/builder/builder.go could be a single commit. Maybe even the first commit. The change to wire/msgcfilter.go as well.

Then all the changes to the package indexers can be committed. Then the changes to server.go. Then config.go and utreexod.go. Then finally the rpc related files.

@kcalvinalvin
Copy link
Contributor 

[N] calvin@nixos ~/b/u/utreexod ((296c58f6))> ./utreexod --datadir=. --logdir=. --signet --prune=0 --utreexocfilters
2024-07-19 15:40:47.600 [INF] BTCD: Version 0.3.1-beta
2024-07-19 15:40:47.600 [INF] BTCD: Loading block database from 'signet/blocks_ffldb'
2024-07-19 15:40:47.638 [INF] BTCD: Block database loaded
2024-07-19 15:40:47.640 [INF] INDX: Utreexo C filter index enabled
2024-07-19 15:40:47.641 [INF] INDX: Catching up indexes from height -1 to 0
2024-07-19 15:40:47.641 [INF] BTCD: Gracefully shutting down the database...
2024-07-19 15:40:47.693 [INF] BTCD: Shutdown complete
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6f1d73]
        
goroutine 1 [running]:
github.com/utreexo/utreexod/database/ffldb.rollbackOnPanic(0xc000189450)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/database/ffldb/db.go:2221 +0x56
panic({0x9898e0?, 0xf379b0?})
        /nix/store/zg65r8ys8y5865lcwmmybrq5gn30n1az-go-1.21.6/share/go/src/runtime/panic.go:914 +0x21f
github.com/utreexo/utreexod/blockchain.(*BlockChain).IsUtreexoViewActive(0x0)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/utreexoviewpoint.go:849 +0x13
github.com/utreexo/utreexod/blockchain/indexers.(*UtreexoCFIndex).fetchUtreexoRoots(0xc00030a060, {0xbeebf0, 0xc000189450}, 0xc0000bec00?)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/utreexocfindex.go:178 +0x32
github.com/utreexo/utreexod/blockchain/indexers.(*UtreexoCFIndex).ConnectBlock(0xbeebf0?, {0xbeebf0, 0xc000189450}, 0x1a?, {0x1a?, 0x10?, 0xc0000bec00?})
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/utreexocfindex.go:155 +0x45
github.com/utreexo/utreexod/blockchain/indexers.dbIndexConnectBlock({0xbeebf0, 0xc000189450}, {0xbed2a0, 0xc00030a060}, 0xc000210480, {0x0, 0x0, 0x0})
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/manager.go:123 +0x11a
github.com/utreexo/utreexod/blockchain/indexers.(*Manager).Init.func6({0xbeebf0?, 0xc000189450?})
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/manager.go:509 +0x3b
github.com/utreexo/utreexod/database/ffldb.(*db).Update(0xc000318000?, 0xc0001d2600)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/database/ffldb/db.go:2278 +0x7e
github.com/utreexo/utreexod/blockchain/indexers.(*Manager).Init(0xc00030a090, 0xbe77a0?, 0xc000318000?)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/manager.go:508 +0xdf7
github.com/utreexo/utreexod/blockchain.New(0xc0002236e0)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/chain.go:2525 +0x8c2
main.newServer({0xc00005dd40, 0x1, 0x1}, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0xbec1a0, ...}, ...)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/server.go:3474 +0x1945
main.btcdMain(0x0)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/utreexod.go:406 +0xea7
main.main()
        /home/calvin/bitcoin-projects/utreexod1/utreexod/utreexod.go:592 +0x96

The index panics out on start

@alainjr10
Copy link
Author 

[N] calvin@nixos ~/b/u/utreexod ((296c58f6))> ./utreexod --datadir=. --logdir=. --signet --prune=0 --utreexocfilters
2024-07-19 15:40:47.600 [INF] BTCD: Version 0.3.1-beta
2024-07-19 15:40:47.600 [INF] BTCD: Loading block database from 'signet/blocks_ffldb'
2024-07-19 15:40:47.638 [INF] BTCD: Block database loaded
2024-07-19 15:40:47.640 [INF] INDX: Utreexo C filter index enabled
2024-07-19 15:40:47.641 [INF] INDX: Catching up indexes from height -1 to 0
2024-07-19 15:40:47.641 [INF] BTCD: Gracefully shutting down the database...
2024-07-19 15:40:47.693 [INF] BTCD: Shutdown complete
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6f1d73]
        
goroutine 1 [running]:
github.com/utreexo/utreexod/database/ffldb.rollbackOnPanic(0xc000189450)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/database/ffldb/db.go:2221 +0x56
panic({0x9898e0?, 0xf379b0?})
        /nix/store/zg65r8ys8y5865lcwmmybrq5gn30n1az-go-1.21.6/share/go/src/runtime/panic.go:914 +0x21f
github.com/utreexo/utreexod/blockchain.(*BlockChain).IsUtreexoViewActive(0x0)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/utreexoviewpoint.go:849 +0x13
github.com/utreexo/utreexod/blockchain/indexers.(*UtreexoCFIndex).fetchUtreexoRoots(0xc00030a060, {0xbeebf0, 0xc000189450}, 0xc0000bec00?)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/utreexocfindex.go:178 +0x32
github.com/utreexo/utreexod/blockchain/indexers.(*UtreexoCFIndex).ConnectBlock(0xbeebf0?, {0xbeebf0, 0xc000189450}, 0x1a?, {0x1a?, 0x10?, 0xc0000bec00?})
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/utreexocfindex.go:155 +0x45
github.com/utreexo/utreexod/blockchain/indexers.dbIndexConnectBlock({0xbeebf0, 0xc000189450}, {0xbed2a0, 0xc00030a060}, 0xc000210480, {0x0, 0x0, 0x0})
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/manager.go:123 +0x11a
github.com/utreexo/utreexod/blockchain/indexers.(*Manager).Init.func6({0xbeebf0?, 0xc000189450?})
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/manager.go:509 +0x3b
github.com/utreexo/utreexod/database/ffldb.(*db).Update(0xc000318000?, 0xc0001d2600)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/database/ffldb/db.go:2278 +0x7e
github.com/utreexo/utreexod/blockchain/indexers.(*Manager).Init(0xc00030a090, 0xbe77a0?, 0xc000318000?)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/indexers/manager.go:508 +0xdf7
github.com/utreexo/utreexod/blockchain.New(0xc0002236e0)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/blockchain/chain.go:2525 +0x8c2
main.newServer({0xc00005dd40, 0x1, 0x1}, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0xbec1a0, ...}, ...)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/server.go:3474 +0x1945
main.btcdMain(0x0)
        /home/calvin/bitcoin-projects/utreexod1/utreexod/utreexod.go:406 +0xea7
main.main()
        /home/calvin/bitcoin-projects/utreexod1/utreexod/utreexod.go:592 +0x96

The index panics out on start

Let me look into it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Davidson-Souza Davidson-Souza Davidson-Souza left review comments

@kcalvinalvin kcalvinalvin kcalvinalvin left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@alainjr10 @kcalvinalvin @Davidson-Souza