Floresta v0.7.0

Notable changes and new features

• Async-std To Tokio (#172): This ends a major milestone of replacing async-std with tokio as the async runtime, given that the former isn't well maintained and lacks features like work stealing and doesn't have as much tooling as the latter. This project was part of 2024 Summer Of Bitcoin intership.
• florestad: expose assumeutreexo in lib mode (#211): If you are using florestad as a lib, you can pass the assumeutreexo values yourself, so your application is the source of truth for this. Another useful use-case would be to start a mobile florestad from an already-running instance somewhere else, by scanning a QR code with the assumeutreexo values, so the process is still trustless, you're just skipping redundant work.
• Feature: daemonize (#225): On *NIX enviremonets, you may want florestad to run in the background as a daemon. This PR adds a clean way of doing this, by adding a --daemon option that daemonizes florestad right after we parse configuration related data.
• Wire: handle block filters out-of-order (#226): Before that, we would assume that filters would arrive in order, but this may not be true due to network conditions or even multiprocess request handling by our peers. After #226 filters can arrive in any order, and we'll sort them and make sure our storage is consistent
• ssl init (#223): Adds support for SSL to our Electrum Server. Now you can connect with Floresta's Electrum Server using a secure and private connection using SSL encryption. This might be useful for e.g. public networks where people may sniff your traffic.
• Add fuzz to floresta(#227, #228): Adds fuzz to floresta using cargo fuzz with a few targets (LocalAddress and UtreexoBlock deserialization)
• wallet: make it work without the Electrum Server (#233): This is part of a project that tries to make the inners of floresta work independently. Before #233, the wallet needed a running Electrum Server to update it, but now you can run the watch-only wallet without an Electrum Server, if you don't want one. In the future, florestad will be able to run without an Electrum Serve, the wallet and a json-rpc, making all optional.
• chain/wire: remove the explicit rescan code (#234): This was a feature that allowed users to rescan by downloading all blocks in a range, but this is no longer useful since we have compact block filters, that are way more efficient.
• Improve connection (#235): Many small changes to how we open new connections with the goal of improving our resilience and making sure we won't stall by not having utreexo peers.
• Update getutxo (#237): Now the gettxout rpc only returns UTXOS that are cached by the wallet (and only if they are unspent). If you want to use compact block filters to find a UTXO, use the findtxout rpc.
• deps: remove unused deps (#239): Remove dependencies that are no longer used. Those were found by using cargo-udep
• cli: replace reqwest with jsonrpc (#240): Changes the http lib used for requests in floresta_cli, because reqwest is too much for this simple code. Now we use jsonrpc which has a small code footprint and almost no cross-dependencies.
• Improve CI caching (#247): Now CI build artifacts are cached. Unless they are changed by the PR, we don't need to build it again, making CI more efficient and faster.
• No_std support for floresta-common (#254): now floresta-common (a subcrate that defines some useful primitives to all other crates) can be built with no-std. We also test it to make sure we won't break the no-std build in the future.
• Add criterion benches and restructure testdata (#259): This adds benchmarks to floresta using criterion. Currently, we mostrly have benchmark for block validation times.
• rpc: cache addresses on loaddescriptor (#268): loaddescriptors didn't derive addresses when called, only saved the descriptor to our database. After #268 it'll actually derive and cache addresses as expected.
• CI: Test all feature combinations (#277): Tests all feature combinations on CI for a PR. Making sure that a change won't silently break non-standard builds.
• Add test-features recipe to justfile (#276): Similar to #276, but add a just command to easily test feature combinations locally.
• feat: allow hostnames where we take addresses (#286): Now, every option that takes in an address (like --connect or --rpc-address) will also take a hostname (e.g. localhost or floresta-container) and resolve with the system resolver.
• Add floresta-cli to Docker image (#281): Now you can use floresta-cli inside a docker container.

Notable bugfixes and documentation updates

• minor improvement when printing height (#218)
• readme: fix mention of --assume-utreexo option (#219)
• Added prerequisite to README (#220)
• Only keep extra peers if they give us a new block (#221)
• Update copyright notices and use the copyright ident on every public interface file (#224)
• docs: Update the repo url (#232)
• rpc: don't try to run some rpcs if in ibd (#229)
• Fix local seeds (#244)
• Correct Cargo categories (#243)
• Fix our dockerfile (once again) (#245)
• Remove unused maybe_reindex (#246)
• fix:including requests on requirements.txt (#249)
• Fixing path to config.toml.sample (#256)
• Feat: refactor nix files. (#255)
• Update read me (#264)
• changed from error to warn message on logging the ssl cert load (#265)
• Fix clippy warnings (#267)
• Add full block data for testing and benchmarking (#263)
• Improve README formatting (#269)
• Fix electrum tests (#271)
• Fix cli tests and florestad panic messages (#272)
• update rust bitcoin to upstream 0.32.4 (#270)
• Organize ignore files and add IDE configurations (#273)
• Actually run CI linting with nightly (#274)
• Fix compile errors for all feature combinations (#275)
• README: Add badges and fix headers (#278)
• fix: asserting acc length only one time on load_acc (#279)
• feat: Specifying rustfmt and clippy to match github actions (#283)
• Remove an externally reachable unwrap (#285)
• docs: correct typos in docstrings within florestad/src (#288)
• Fix on allow-hostnames (#289)
• Fix shutdown save peers utreexo (#280)
• flush more often on ibd (#291)
• Group NodeCommon fields (#292)
• Update chain_state.rs (ChainStateInner struct changes) (#297)
• fix: specify version on just commands to avoid incorrect versions being used (#299)

Full Changelog: 0.6.0...0.7.0

v0.6.0

Notable changes

New features

• expose some unexposed values for jsonrpc(#103): Lets the user select between getting a serialized block or a json. The json should mimic core's output for the equivalent getblock rpc.
• Bring our functional tests to life (#112): attempts to build some basis for the Python-based tests. Now we have utils to spawn an Utreexod node, build blocks, transactions and send/receive network messages. Those utils aren't used yet in any serious test.
• feat: async add new address to electrum(#113): Now, if you subscribe to an address that we don't follow yet, we'll start following it. Note that this currently only works for the experimental blockchain.scriptpubkye.subscribe endpoint.
• Implement tagged hashes for the leafhashes (#118): This futures-proof the leaf commitment scheme from future modifications of the committed data, and it's now part of the utreexo protocol
• feat: -connect cli option (#119): This option lets you connect exclusively to one specific node, given its IP address
• [WIP]: Create a json-rpc lib (#121): Now the json-rpc has a Rust lib that can be used by other projects to call florestad's json-rpc interface
• Rework internal node structure(#128): Use our actors model to build optmized nodes for each phase of the startup process: Headers sync, downloading blocks and normally running node.
• Add AssumeValidArg enum and correct verify_script (#131): This adds a new AssumeValidArg to make communicating the desired assume-valid more ergonomic. It can either be disabled, using the hard-coded value or user-provided.
• feature: batch request for electrum (#134): Now electrum lets users perform multiple requests at once, rather than sending one at the time. This has some small bandwidth and CPU savings.
• feat: PoW fraud proof(#133, #147): Implement pow fraud proofs for our node and enable it on signet for testing. This is not enabled on mainnet yet.
• feature: log to file (#136): Now you can write the logs to a file. This is specially useful if running florestad as a background process
• florestad: Add lib.rs (#124, #149): florestad is also a lib now, it can be used on other applications and. To see an example of how to use it in other languages that is not Rust, check out this repo.
• download and store filters from the network (#161, #190, #207): Now we can download BIP-158 Compact Block Filters and use them to recover historical transactions without downloading the whole blockchain.
• Cache headers on ibd (#169): Hold headers in memory and save all-at-once, because the database can optimize some writing operations
• electrum: new experimental electrum endpoints (#173): This adds some experimental (and not used by any wallet) endpoints to the electrum server blockchain.scriptpubkey. They are an exact copy of blockchain.scripthash but uses scripts insted of script hashes. The reson why those exists is to make it possible to rescan for wallet addresses without having the user to manually inform their xpub or descriptor
• Nixify (#176): Adds a nix-based build system and some developer tools to floresta
• Unit tests for all modules (#168, #202, #200): An ongoing project to unit-test all modules
• Assume utreexo + bugfixes (#181): make assumeutreexo work on mainnet
• actually use the -debug option (#184): The option actually always existed, but there wasn't any handling for it. Now using -d or --debug will make the node print debug information to the terminal. This doesn't affect the log file though.
• Transaction Consensus Rules (#165): Expands floresta's consensus rules implementation, implementing some missing rules for transactions, specially regarding scripts sizes.
• Test utils for floresta-wire/p2p-wire
• Improve just file (#204): Adds some useful dev-related comands like lint
• florestad: rework cli by (#209, #216): Remove some unused options, document all options and improve how we handle versions

Bug fixes and documentation

• Download blocks if lagging behind (#163)
• Bugfix: recover from utreexo nodes hangup (#100)
• If we stay long without recieving any message, send out a ping (#101)
• small refactor + comments (#105)
• add our discord to readme (#109)
• style: add .rustfmt.toml and justfile (#111)
• bump all rust-bitcoin crates (#108)
• don't ban on the first violation (#114)
• Update readme.md (#122)
• Improve reindex_chain readability and small fixes (#125)
• Fix the CI error with clippy (#129)
• Return when there's a tx validation error (#130)
• wire: hunt down some minor bugs on running_node (#132)
• fixed adding florestad to path while building (#137)
• CI:fixes the linting error (#140)
• chore: update deps (#141)
• Document how to test (#154)
• chore: remove duplicated CI checkout (#152)
• Declared version normalization (#159)
• Fix/create datadir if missing (#164)
• fix an incompatibility between our implementation of is_unpendable and utreexo's (#170)
• bump rustreexo (#171)
• Improve ibd stability (#174)
• Potential bug fix in floresta-wire/chain_selector.rs (#179)
• Bug fix: chain_selector (#182)
• Node.rs minor fixes (#183)
• Fix/creating too much conn (#185)
• fix a small incompatibility with electrum implementation (#189)
• Broken Link on README (#192)
• Fix warnings (#193)
• floresta-electrum: Increase test timeout (#196)
• add witness and network to fixed peers (#195)
• fix: force utreexo peers (#201)
• bug fix: remove inflights of banned peers (#203)
• fix a crash with node.rs (#208)
• improve readme (#210)
• chain-select-node: ban peers on invalid tips (#213)

Full Changelog: Davidson-Souza/Floresta@v0.5.1...0.6.0

Floresta v0.5.1

This is a new release with a few bugfixes, as well some minor features. We invite everyone to test, report bugs and give feedbacks.

What's changed

New features

Allow to add manual connections to the node (#77 and #78): Now you can use the addnode RPC to manually connect with a given peer

Socks5 Support (#76): Now we can connect with peers over multiple transports, including socks5. This allows running florestad behind a socks proxy like Tor.

Update a few RPCs to show more useful data (#82): Some RPCs like getblockchaininfo and getrawtransaction got updated to return more useful data. They are now close to what Core returns in their respective RPC, but you shouldn't expect a 1-1 equivalence. getblockchaininfo now returns Utreexo-related data.

Add a ZMQ block notification (#93): Now you can subscribe over ZMQ to receive new blocks as they are accepted by our chainstate

Build and store BIP-158 Compact Block Filters (#89): We now create and store Compact Block Filters for easy wallet rescan and finding UTXOs inside the chain. Currently, we only build the filters and expose them over a new rpc getfilter. This implementation doesn't download the filters from p2p, we build them locally and let users pick what is stored, reducing disk usage. A multiphase approach where we use the p2p-provided ones to sync-up user's wallet during IBD and only store the locally computed ones should bring the best of both worlds.

Refactor and Bugfixes:

Don't die if we lose all Utreexo peers (#79): losing all utreexo peers may put our node in an undefined state, since we need those peers to get proofs. This PR hardens our node against it, and improves our recovery logic.

Don't invalidate block if proof is invalid (#81): Since proofs aren't committed by PoW, a misbehaving peer might send an invalid proof without changing the block hash. Therefore, if we get an invalid proof, just ban this peer and try to get the proof from someone else.

Refactor the node struct and rework trait bounds (#83): UtreexoNode is the biggest struct inside this codebase, and it's generic over many attributes. This PR is an attempt to simplify those bounds. Furthermore, making internal methods private.

improve Error enum names so we can remove clippy::enum_variant_name (#85): Some enum members ended with the enum's name (e.g. SomethingError), this would trigger a clippy warning. This pr reworks invariant naming, so now we satisfy that linting.

Rework block notification (#92): Once chainstate validates a new block, it sends that block to some subscribers. Before this PR, a subscriber would pass the Sender end of a channel, and we would use this channel to send new blocks. The problem here is that some modules aren't services (i.e. they don't have an always running loop) and can't work with this channel model. Now, anything implementing the BlockConsumer trait may subscribe by passing an Arc-ed instance of self to chainstate. For modules using the old model, this pr implements a simple Single Producer, Single Consumer channel that implements BlockConsumer.

Make database more robust (#95): Our default chainstore database had many issues with unclean shutdowns, leading to a broken state. This pr is another attempt to make our DB more robust against unexpected scenarios.

Make dockerfile build florestad (#96): After #58 our Dockerfile wasn't working. Now we keep official docker images on docker hub.

v0.5.0: "I promise I'll use semversioning more often"

This is the first release after the crate reorg, which. For now on, the GitHub releases will reflect florestad, and every subcrate has its version.

What's Changed

• Break this project into a workspace by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/58
• Create LICENSE by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/60
• Use env var to pass xpubs by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/61
• Parallel sync 1/n: Refactor: Move stateless code to a new module by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/62
• Rate limit peers by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/66
• Parallel sync 2/n: Partial chainstate by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/63
• Readme.md typo by @bitkarrot in https://github.com/Davidson-Souza/Floresta/pull/68
• Send addresses by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/67
• Improve assumeutreexo by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/70
• Improve support to wasm by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/71
• Feature/refactor manifests by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/72
• Fix: return leaf_hashes by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/73
• Docs improvements by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/74

Full Changelog: Davidson-Souza/Floresta@v0.4.0...v0.5.0

v0.4.0

New major version of Floresta, with some bugfixes and new features like a json-rpc, rescan, output descriptors env var support and more...

What's Changed

• Fix config file parse error and docs ortography by @Guilospanck in https://github.com/Davidson-Souza/Floresta/pull/27
• Added support for output descriptors by @douglaz in https://github.com/Davidson-Souza/Floresta/pull/30
• Fix sparrow incompatibility by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/31
• Add missing rules by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/29
• Feature/dos protection by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/32
• Clippy by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/33
• Minor p2p improvements by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/34
• Add more tests to ci by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/22
• Fix/small docs fixups by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/26
• Derive more addresses as we get transactions by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/35
• Fix a bug with prot 70016 by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/37
• Improve address selection by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/36
• Fix/remove unreachable by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/38
• Improve address cache by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/39
• Ask for headers only once by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/40
• read xpub from env var by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/41
• Fix a bug of hanging forever while on ibd by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/42
• Fix some retargeting bugs by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/43
• Revert "Fix some retargeting bugs (#43)" by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/44
• Fix some retargeting bugs by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/45
• Improve versioning and folder structure by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/46
• Reindex by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/48
• feat: Rescan by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/49
• create a json-rpc server by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/50
• Fix a merge conflict with json-rpc by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/51
• Update transaction view on broadcast by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/47
• Refactor node by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/52
• Node request by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/53
• Node request by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/54
• Add a rpc to stop the node by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/55
• Move context by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/56
• Bump version 0.3.0 -> 0.4.0 by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/57

Full Changelog: Davidson-Souza/Floresta@0.3.0...v0.4.0

p2p module

This release is the first version with p2p code merged in. There's still numerous things to change, but should be able to run on its own by now.

What's Changed

• Create rust.yml by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/7
• Update readme.md by @kcalvinalvin in https://github.com/Davidson-Souza/Floresta/pull/4
• Handle reorgs by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/8
• Fix a bug with descriptor caching by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/9
• Feature/test framework by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/10
• Cache individual addresses by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/11
• P2p chain by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/12
• implement all electrum protocol by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/13
• A few bugfixes by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/14
• Remove debug print by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/17
• Optmize Dockerfile by @jaonoctus in https://github.com/Davidson-Souza/Floresta/pull/18
• Minor fixups by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/19
• Persist peers db by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/20
• Add transaction broadcast logic by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/21
• A few teaks on electrum implementation by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/23
• Docs by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/24
• Bump version by @Davidson-Souza in https://github.com/Davidson-Souza/Floresta/pull/25

Full Changelog: Davidson-Souza/Floresta@0.1.0...0.3.0

v0.2.1

Some minor bugfixes from previous release.

v0.2.0-alpha

New release with a more stable code, also some cleanups. Some major improvements over the last release:

• Update upstream to use reqwest for json-rpc requests, add support to TLS and keep-alive connections
• Add headers-first sync, so we can know almost everything about a potential tip before we validate. This will be specially useful when the p2p code is merged
• Clean-up command line args and introduce config file.
• Add a asumevalid option for faster IBD
• Various bug-fixes

What's Changed

• Create rust.yml by @Davidson-Souza in #7
• Update readme.md by @kcalvinalvin in #4
• Handle reorgs by @Davidson-Souza in #8

Full Changelog: 0.1.0...0.2.0_alpha

v0.1.0

First release with compiled binaries for people to test out.