Decrement thread count on exception.

[tylerkaraszewski]

Details

So, we increment the thread count here, before starting a replication thread:

Bedrock/sqlitecluster/SQLiteNode.cpp

Line 1644 in a30c7e1

auto threadID = _replicationThreadCount.fetch_add(1);

Normally, we will decrement this inside the replication thread when it exits, which is handled here:

Bedrock/sqlitecluster/SQLiteNode.cpp

Line 199 in a30c7e1

ScopedDecrement<decltype(_replicationThreadCount)> decrementer(_replicationThreadCount);

However, if the thread fails to start and throws system_error, we will get to this point:

Bedrock/sqlitecluster/SQLiteNode.cpp

Line 1655 in a30c7e1

_changeState(SQLiteNodeState::SEARCHING, message.calcU64("NewCount") - 1);

Without ever having run the thread, and thus, without the thread count ever being decremented. Further, change state will wait for the thread count to be 0 before it completes, which means the above line blocks and we never log the warning or throw the error.

Here's where _changeState blocks on this:

Bedrock/sqlitecluster/SQLiteNode.cpp

Lines 1874 to 1880 in a30c7e1

	while (_replicationThreadCount) {
	if (infoCount % 100 == 0) {
	SINFO("Waiting for " << _replicationThreadCount << " remaining replication threads.");
	}
	infoCount++;
	usleep(10'000);
	}

The fix is to decrement the counter if we hit the exception case. I've also moved logging the warning to above the _changeState call so that it will be visible sooner that we've hit this exception.

Fixed Issues

Fixes https://github.com/Expensify/Expensify/issues/440475

Tests

---

Internal Testing Reminder: when changing bedrock, please compile auth against your new changes

[flodnv]

cc @danieldoglas since IIRC you looked at this code this year - #1767

[flodnv]

Thanks for the great investigation and explanation! 👍

[flodnv]

What happens after we throw here?

[tylerkaraszewski]

We should go synchronizing as if the node was just turned on.

[danieldoglas]

I didn't know that counter existed when I changed this, thanks for finding and fixing!