Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow multiple scripthashes' subscription in parallel #600

Merged
merged 1 commit into from
Oct 30, 2021
Merged

Allow multiple scripthashes' subscription in parallel #600

merged 1 commit into from
Oct 30, 2021

Conversation

romanz
Copy link
Owner

@romanz romanz commented Oct 28, 2021 •
edited
Loading

Tested locally (with a warm cache) using netcat 127.0.0.1 50001 < all12_subs.txt:

$ RUST_LOG=electrs=info,electrs::server=debug ./server.sh bitcoin 2>&1 | cut -c -80
...
[2021-10-29T07:51:44.068Z DEBUG electrs::server] 0: connected
[2021-10-29T07:51:44.112Z DEBUG electrs::server] 0: recv [{"jsonrpc":"2.0","meth
[2021-10-29T07:51:44.112Z DEBUG electrs::server] 0: recv [{"jsonrpc":"2.0","meth
[2021-10-29T07:51:51.768Z DEBUG electrs::server] 0: send [{"id":"m/0/0","jsonrpc
[2021-10-29T07:51:52.370Z DEBUG electrs::server] 0: send [{"id":"m/1/0","jsonrpc
[2021-10-29T07:51:58.962Z DEBUG electrs::server] 0: disconnected

all12_subs.txt

$ curl -s localhost:4224 | grep blockchain.scripthash
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.000001"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.000002"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.000005"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.00001"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.00002"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.00005"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.0001"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.0002"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.0005"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.001"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.002"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.005"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.01"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.02"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.05"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.1"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.2"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.5"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="1"} 1
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="2"} 1
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="5"} 1
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="10"} 2
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="20"} 2
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="50"} 2
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="100"} 2
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="+Inf"} 2
rpc_duration_sum{method="blockchain.scripthash.subscribe:multi"} 8.225756585
rpc_duration_count{method="blockchain.scripthash.subscribe:multi"} 2

@romanz romanz force-pushed the rpc-batch branch 2 times, most recently from 14c9e4e to 6dfe16d Compare October 29, 2021 07:26
@romanz
Copy link
Owner Author

romanz commented Oct 29, 2021

With cold cache (using echo 3 | sudo tee /proc/sys/vm/drop_caches):

[2021-10-29T07:53:55.593Z DEBUG electrs::server] 0: connected
[2021-10-29T07:53:55.660Z DEBUG electrs::server] 0: recv [{"jsonrpc":"2.0","meth
[2021-10-29T07:53:55.660Z DEBUG electrs::server] 0: recv [{"jsonrpc":"2.0","meth
[2021-10-29T07:54:14.080Z DEBUG electrs::server] 0: send [{"id":"m/0/0","jsonrpc
[2021-10-29T07:54:16.222Z DEBUG electrs::server] 0: send [{"id":"m/1/0","jsonrpc
[2021-10-29T07:54:23.174Z DEBUG electrs::server] 0: disconnected

$ curl -s localhost:4224 | grep blockchain.scripthash
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.000001"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.000002"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.000005"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.00001"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.00002"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.00005"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.0001"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.0002"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.0005"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.001"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.002"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.005"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.01"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.02"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.05"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.1"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.2"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="0.5"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="1"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="2"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="5"} 1
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="10"} 1
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="20"} 2
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="50"} 2
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="100"} 2
rpc_duration_bucket{method="blockchain.scripthash.subscribe:multi",le="+Inf"} 2
rpc_duration_sum{method="blockchain.scripthash.subscribe:multi"} 20.521422041
rpc_duration_count{method="blockchain.scripthash.subscribe:multi"} 2

@romanz
Copy link
Owner Author

romanz commented Oct 29, 2021

Comparing to 563d8b7 (without parallel subscription):

Warm cache:

$ curl -s localhost:4224 | grep blockchain.scripthash
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.000001"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.000002"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.000005"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.00001"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.00002"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.00005"} 24
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.0001"} 37
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.0002"} 38
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.0005"} 38
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.001"} 38
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.002"} 38
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.005"} 38
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.01"} 74
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.02"} 103
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.05"} 339
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.1"} 347
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.2"} 347
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.5"} 348
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="1"} 348
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="2"} 348
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="5"} 348
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="10"} 348
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="20"} 348
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="50"} 348
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="100"} 348
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="+Inf"} 348
rpc_duration_sum{method="blockchain.scripthash.subscribe"} 8.736816384999996
rpc_duration_count{method="blockchain.scripthash.subscribe"} 348

Cold cache:

$ curl -s localhost:4224 | grep blockchain.scripthash
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.000001"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.000002"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.000005"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.00001"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.00002"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.00005"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.0001"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.0002"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.0005"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.001"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.002"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.005"} 0
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.01"} 1
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.02"} 6
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.05"} 67
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.1"} 269
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.2"} 321
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="0.5"} 347
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="1"} 348
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="2"} 348
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="5"} 348
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="10"} 348
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="20"} 348
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="50"} 348
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="100"} 348
rpc_duration_bucket{method="blockchain.scripthash.subscribe",le="+Inf"} 348
rpc_duration_sum{method="blockchain.scripthash.subscribe"} 30.658128465000015
rpc_duration_count{method="blockchain.scripthash.subscribe"} 348

@romanz
Copy link
Owner Author

romanz commented Oct 29, 2021 •
edited
Loading

TL;DR: this PR seems to help on cold cache subscriptions :)
I guess it helps when RocksDB is being I/O-bound, since we can do multiple lookups concurrently.

@romanz romanz mentioned this pull request Oct 29, 2021
Closed
@romanz romanz linked an issue Oct 29, 2021 that may be closed by this pull request
Optimize batched RPC lookups in P2P branch #464
Closed
@romanz romanz mentioned this pull request Oct 29, 2021
Closed
@romanz romanz force-pushed the rpc-batch branch 2 times, most recently from 865f49d to 539a0b0 Compare October 29, 2021 08:46
Kixunil
Kixunil approved these changes Oct 29, 2021
View reviewed changes
Copy link
Contributor

@Kixunil Kixunil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks cool! Couldn't review very deeply but I don't see any issue.

src/electrum.rs Outdated Show resolved Hide resolved
src/electrum.rs Outdated
scripthashes
.iter()
.map(|scripthash| values.remove(scripthash).unwrap())
.collect()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand the code above correctly it should be possible to do it in a single parallelized pass. I will probably try to do it.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplified the code a bit, please take a look :)

src/electrum.rs
_ => None, // exit if any of the calls is not supported
}
})
.collect::<Option<Vec<ScriptHash>>>()?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good first step if wallets send subscriptions like this but would be nice to just process subscriptions separately from non-subscriptions.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, currently tested with Sparrow :)

@romanz romanz merged commit fabc634 into master Oct 30, 2021
@romanz romanz deleted the rpc-batch branch October 30, 2021 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Kixunil Kixunil Kixunil approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Optimize batched RPC lookups in P2P branch
2 participants
@romanz @Kixunil