Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blocking command port when node is not caught up #2039

Merged
merged 22 commits into from
Dec 26, 2024

Conversation

danieldoglas
Copy link
Contributor

@danieldoglas danieldoglas commented Dec 23, 2024

Details

This PR blocks the command port if a node is more than 12.5K commits behind the leader, and reopens when we're 1K behind.

Fixed Issues

Fixes https://github.com/Expensify/Expensify/issues/437855

Tests


Internal Testing Reminder: when changing bedrock, please compile auth against your new changes

@danieldoglas danieldoglas self-assigned this Dec 23, 2024
BedrockServer.h Outdated Show resolved Hide resolved
sqlitecluster/SQLiteNode.cpp Outdated Show resolved Hide resolved
sqlitecluster/SQLiteNode.cpp Outdated Show resolved Hide resolved
sqlitecluster/SQLiteNode.cpp Outdated Show resolved Hide resolved
@danieldoglas danieldoglas requested a review from cead22 December 24, 2024 15:49
sqlitecluster/SQLiteNode.cpp Outdated Show resolved Hide resolved
sqlitecluster/SQLiteNode.cpp Outdated Show resolved Hide resolved
atomic<SQLiteNodeState> _state;

// Keeps track if we have closed the command port for commits fallen behind.
bool _blockedCommandPort{false};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we block the command port for other reasons, we should put the fact that this is being set based on the node being behind on the variable name, for clarity

Suggested change
bool _blockedCommandPort{false};
bool _blockedCommandPortForBeingBehind{false};

But do we really need this variable? Is there an easy way to check the reason the command port is blocked so we can do something like

_blockedCommandPortForBeingBehind = getReason() == "COMMITS_LAGGING_BEHIND";
if (peer == _leadPeer && currentCommitDifference >= 12'500 && !_blockedCommandPort) {
    ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But do we really need this variable? Is there an easy way to check the reason the command port is blocked so we can do something like

tl;dr: Adding this as a separate variable so we don't need to add complexity to the current methods related to block reasons. We had an implementation like you're suggesting before, but that would require adding new locks/changing current lock types.

This was discussed in this thread: https://expensify.slack.com/archives/CC7NECV4L/p1734981555759429?thread_ts=1733270912.693829&cid=CC7NECV4L

sqlitecluster/SQLiteNode.cpp Outdated Show resolved Hide resolved
@tylerkaraszewski tylerkaraszewski merged commit e218e46 into main Dec 26, 2024
1 check passed
@tylerkaraszewski tylerkaraszewski deleted the dsilva_blockingCommandPortWhenNotCaughtUp branch December 26, 2024 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants