Show: PubSub simulator & benchmark, with demos (some interactive)

[pancsta]

Hi everyone,

This is a cross-post from discuss.libp2p.io regarding pancsta/go-libp2p-pubsub-benchmark.

Hi, I wrote a simple pubsub simulator with join/leave topic, add/remove peer and peers friending each other and joining each others topics. The discovery is set to MockDiscovery (theres code for DHT and mDNS too, but had a huge lag with DHT). All the params can be adjusted (amount of peers, topic, etc), so maybe someone will find it useful.

The motivation behind it was to stress test a state machine library I’ve been working on, for which Ive ported ~half of pubsub to asyncmachine-go, including MockDiscovery. Theoretically any implementation could be used, but the state-based one exports bulks of data, which can be consumed by grafana/jaeger.

Theres also a benchmark with charts showing the overhead for TestSimpleDiscovery. This one is very open to work with other implementations and/or running any other test from the suite.

Since the frequency on the open maintainers call was low, I've recorded a screencast with some silent-movie-style captions, so everyone can take a look. For anyone interested in investigating the details of PubSub, Discovery and their simulator nodes, there are interactive debugger instances available, with all the mentioned machines having a synchronized timeline. They are accessible through either web or ssh, thanks to am-dbg-ssh and wetty. Unfortunately, public instances of Jaeger and Grafana are not available.

• am-dbg over web
• am-dbg over ssh - ssh 188.166.101.108 -p 4444
• jaeger traces

Thanks, hope you'll like it.

PS. Sorry for all the typos :) No time Toulouse...

@dhuseby

[MarcoPolo]

Very cool @pancsta. Thanks for sharing!

A random idea, but if you've gotten rid of the channels and async, can you compile this with TinyGo to WASM and run it from another libp2p host? (e.g. rust-libp2p?)

How many messages/second can this parse relative to the standard go-libp2p-pubsub? I'm curious from a fuzzing-feasibility perspective.

[pancsta]

A random idea, but if you've gotten rid of the channels and async, can you compile this with TinyGo to WASM and run it from another libp2p host? (e.g. rust-libp2p?)

Not sure if I understand the question. You can probably compile the whole simulator+pubsub+libp2p to WASM even now using GOARCH=wasm (I did it with am-dbg via tcell wasm), but the problem with WASM right now is that its slow, as it lacks multi threading. It'd obviously be sandboxed, but it's a simulator, so that's not a problem.

If you would like to add a rust-based peer to the simulator (outside of wasm and the browser), then that would require implementing the sim-peer API over the network (probably via grpc), but is theoretically feasible.

How many messages/second can this parse relative to the standard go-libp2p-pubsub? I'm curious from a fuzzing-feasibility perspective.

Here are some charts from the benchmark based on TestSimpleDiscovery and it shows 10-20% overhead for both CPU and memory (for the state machine version).

The problem with TestSimpleDiscovery is that is uses time.Sleep() (like most of the test suite), thus making the results less accurate. Im planning to subtract these times from the results somehow. I also like your idea of measuring TPS, for which I will create a dedicated test with multiple simultaneous senders (currently it's one sender flooding the network at the time) and plot the results.