Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IF: Populated fork db ASAP; Vote processing off the main thread #2385

Merged
merged 32 commits into from
Apr 10, 2024
Merged

IF: Populated fork db ASAP; Vote processing off the main thread #2385

merged 32 commits into from
Apr 10, 2024

Conversation

heifner
Copy link
Member

@heifner heifner commented Apr 8, 2024

  • Add blocks to the fork db as soon as header validation is complete. They are added off the main thread (net threads).
    • Move logging of received blocks to the controller since many are processed in maybe_switch_forks and would not be logged by the producer_plugin after a call to push_block otherwise.
  • Call log_irreversible even when head is not updated as processed qcs can move irreversible at any time.
  • Integrate QC and consider voting as soon as header validation is complete off the main thread (net threads).
  • Add logging to app().quit() to be able to determine if nodeos is stopped via a SIGTERM.

Does not fully resolve #2102. Still need to move the processing into a voting thread pool. That will come as a separate PR as the changes here already are rather extensive.

heifner added 30 commits April 2, 2024 13:05
@heifner
GH-2102 Use vector of atomic bools to avoid mutex lock for duplicate …
ad64fc7 
…check
@heifner
GH-2102 Add block to fork db as soon as header is validated so it is …
3887b2c 
…available for vote processing
@heifner
GH-2102 May already be in forkdb since we are adding as soon as heade…
bd43d61 
…r validation complete
@heifner
GH-2102 Add logging on app().quit()
c84bab1
@heifner
GH-2102 Integrate qc and vote when switching forks if first time bloc…
cf50336 
…ks are validated
@heifner
GH-2102 Log irreversible even when head not updated
053fe72
@heifner
GH-2102 Check for validated block to avoid extra processing
4066109
@heifner
GH-2102 Init processed on reflection
9411c05
@heifner
GH-2102 Handle corner case of trx locally applied but not in a block yet
0c82e7a
@heifner
GH-2102 Only report fork switch on actual fork switch. maybe_switch_f…
b9deb58 
…orks now also can apply staged blocks in the fork db.
@heifner
GH-2102 Check for terminate_at_block during apply of fork db blocks
8a7b973
@heifner
GH-2102 If forked out again then could still be in local state
274e44a
@heifner
GH-2102 Allow ctrl-c shutdown during sync of a large number of blocks…
2e49794 
…. Improve switch logging.
@heifner
GH-2102 Fix sync issue with receiving a current block while syncing
4c3535f
@heifner
GH-2102 On startup pending_head forkdb blocks are processed. If asked…
112bb69 
… to terminate at a block don't put any blocks above that in the fork database.
@heifner
GH-2102 Fix comparison in waitForBlock
914f218
@heifner
GH-2102 Better error reporting and a bit more tolerance for trxs in b…
5e9af0c 
…lock
@heifner
GH-2102 Add additional logging for applied blocks
2449830
@heifner
Revert "GH-2102 Fix comparison in waitForBlock"
e464aee 
This reverts commit 914f218.
@heifner
GH-2102 Improve logging during sync
a74450c
@heifner
GH-2102 Add a large_atomic wrapper around mutex
cafd0b6
@heifner
GH-2102 Use large_atomic for if_irreversible_block_id
302b957
@heifner
GH-2102 No need to recalculate finality digest
be77ca6
@heifner
GH-2102 Move valid_qc into pending_qc and make thread safe
23945ff
@heifner
GH-2102 Improve test conditions
ba5006e
@heifner
GH-2102 Move produced/received block logging into controller so it lo…
9de11ff 
…gs all processed blocks
@heifner
GH-2102 Fix log_applied to not be called during replay
8e98c18
@heifner
GH-2102 Add a copyable atomic type
a9058a1
@heifner
GH-2102 Integrate qc and vote if possible off the main thread.
6d64c27
@heifner
Merge remote-tracking branch 'origin/hotstuff_integration' into GH-2102…
6bd8128 
…-vote-processing
@heifner heifner requested review from greg7mdp and linh2931 April 8, 2024 14:57
@heifner heifner added the OCI Work exclusive to OCI team label Apr 8, 2024
@heifner heifner linked an issue Apr 8, 2024 that may be closed by this pull request
IF: Move vote processing off the net/main thread #2102
Closed
linh2931
linh2931 reviewed Apr 9, 2024
View reviewed changes
libraries/chain/block_state.cpp Show resolved Hide resolved
libraries/chain/hotstuff/hotstuff.cpp Outdated Show resolved Hide resolved
libraries/chain/hotstuff/hotstuff.cpp Show resolved Hide resolved
libraries/chain/hotstuff/hotstuff.cpp Show resolved Hide resolved
libraries/chain/controller.cpp Show resolved Hide resolved
libraries/chain/controller.cpp Outdated Show resolved Hide resolved
libraries/chain/controller.cpp Show resolved Hide resolved
greg7mdp
greg7mdp reviewed Apr 9, 2024
View reviewed changes
libraries/chain/controller.cpp Outdated Show resolved Hide resolved
libraries/chain/controller.cpp Outdated Show resolved Hide resolved
libraries/chain/controller.cpp Show resolved Hide resolved
libraries/chain/controller.cpp Outdated Show resolved Hide resolved
@heifner heifner merged commit f9786e1 into hotstuff_integration Apr 10, 2024
36 checks passed
@heifner heifner deleted the GH-2102-vote-processing branch April 10, 2024 00:17
This was referenced Jun 13, 2024
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@linh2931 linh2931 linh2931 approved these changes

@greg7mdp greg7mdp greg7mdp approved these changes

Assignees
No one assigned
Labels
OCI Work exclusive to OCI team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

IF: Move vote processing off the net/main thread
3 participants
@heifner @greg7mdp @linh2931