Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

relay-related issues connecting many client Endpoints to one server Endpoint #2971

Open
PaulOlteanu opened this issue Nov 28, 2024 · 1 comment
Open

relay-related issues connecting many client Endpoints to one server Endpoint #2971

PaulOlteanu opened this issue Nov 28, 2024 · 1 comment
Assignees
@flub
Labels
bug Something isn't working c-iroh c-iroh-relay
Milestone
v0.30.0

Comments

@PaulOlteanu
Copy link
Contributor

Not 100% sure what to title this, I don't know if the issue was on the Endpoint side or the relay server side.

I had about 10,000 clients trying to connect to a server at a rate of ~300/min, and then sending <1kb every 2 sec. On my server Endpoint side I was seeing a lot of try_send: iroh_net::magicsock: send relay: message dropped, channel to actor is full. When I restarted the server, which would result in all the clients trying to connect at once, which resulted in a wall of that log:

2024-11-25T22:05:57.709301Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=z4wj5ubvyixfn6jw relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.709688Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=z4wj5ubvyixfn6jw relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.709717Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: relay channel full, dropping call-me-maybe dstkey=z4wj5ubvyixfn6jw relayurl=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.709760Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=unzs4fhhvzsxvaq5 relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.710196Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=unzs4fhhvzsxvaq5 relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.710228Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: relay channel full, dropping call-me-maybe dstkey=unzs4fhhvzsxvaq5 relayurl=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.710266Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=p6yde6zwtrpwerqw relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.710839Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=p6yde6zwtrpwerqw relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.710867Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: relay channel full, dropping call-me-maybe dstkey=p6yde6zwtrpwerqw relayurl=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.710903Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=fstlzagjalkbnymo relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.711368Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=fstlzagjalkbnymo relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.711420Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: relay channel full, dropping call-me-maybe dstkey=fstlzagjalkbnymo relayurl=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.711457Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=4poqvd4kppki6dao relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.711959Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=4poqvd4kppki6dao relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.711990Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: relay channel full, dropping call-me-maybe dstkey=4poqvd4kppki6dao relayurl=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.712028Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=yzewz7orsbjmwlpx relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.712434Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=yzewz7orsbjmwlpx relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.712463Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: relay channel full, dropping call-me-maybe dstkey=yzewz7orsbjmwlpx relayurl=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.712499Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=5t44ivrezsrhw4es relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.713075Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=5t44ivrezsrhw4es relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.713104Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: relay channel full, dropping call-me-maybe dstkey=5t44ivrezsrhw4es relayurl=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.713247Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=ikm6sqdfrrxlrntl relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.713541Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=ikm6sqdfrrxlrntl relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.713596Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: relay channel full, dropping call-me-maybe dstkey=ikm6sqdfrrxlrntl relayurl=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.713633Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=2g6jdbviav6t3vxv relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.713976Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=2g6jdbviav6t3vxv relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.714006Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: relay channel full, dropping call-me-maybe dstkey=2g6jdbviav6t3vxv relayurl=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.714041Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=sygycyckomhlhyvo relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.714280Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=sygycyckomhlhyvo relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.714307Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: relay channel full, dropping call-me-maybe dstkey=sygycyckomhlhyvo relayurl=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.714344Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=6lb527ubwgqk6ykn relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.714676Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=6lb527ubwgqk6ykn relay_url=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.714704Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: relay channel full, dropping call-me-maybe dstkey=6lb527ubwgqk6ykn relayurl=xxx me=vr6nx5o36gts5kqw
2024-11-25T22:05:57.714730Z  WARN ep:magicsock:actor:handle_ping_actions: iroh_net::magicsock: send relay: message dropped, channel to actor is full node=bv7ti723wdm7bgck relay_url=xxx me=vr6nx5o36gts5kqw
etc...

on the relay server side, there were a lot of logs of

2024-11-25T21:40:49.528804Z  WARN iroh_relay::server::actor: no way to reach client, dropped packet dst=PublicKey(ysbnyw5olxxtxuzt)
2024-11-25T21:40:49.529038Z  WARN iroh_relay::server::actor: no way to reach client, dropped packet dst=PublicKey(ysbnyw5olxxtxuzt)
2024-11-25T21:40:49.529549Z  WARN iroh_relay::server::actor: no way to reach client, dropped packet dst=PublicKey(ysbnyw5olxxtxuzt)
2024-11-25T21:40:49.529561Z  WARN iroh_relay::server::actor: no way to reach client, dropped packet dst=PublicKey(ysbnyw5olxxtxuzt)
2024-11-25T21:40:49.529565Z  WARN iroh_relay::server::actor: no way to reach client, dropped packet dst=PublicKey(ysbnyw5olxxtxuzt)
2024-11-25T21:40:49.529570Z  WARN iroh_relay::server::actor: no way to reach client, dropped packet dst=PublicKey(ysbnyw5olxxtxuzt)
2024-11-25T21:40:49.529573Z  WARN iroh_relay::server::actor: no way to reach client, dropped packet dst=PublicKey(ysbnyw5olxxtxuzt)

with the PublicKey being that of my server-Endpoint at the time.

There were some clients able to connect and get requests through successfully, but most failed trying to connect
@flub flub added bug Something isn't working c-iroh-relay c-iroh labels Nov 28, 2024
@flub
Copy link
Contributor

flub commented Nov 28, 2024

Several actor channels are filling up and we are dropping packets because of this. The dropped packets result in everything being a pretty bad experience since there are DISCO packets dropped which coordinate the direct connection establishment. Since all communication starts via the relay server you end up with the clients not being able to connect as they never learn about the direct addresses of the server.

We need to improve the actor channels and backpressure probably. This should be relatively straightforward to reproduce by getting many endpoints to connect to a single one.

@flub flub mentioned this issue Dec 5, 2024
Open
@flub flub self-assigned this Dec 5, 2024
@flub flub added this to the v0.30.0 milestone Dec 9, 2024
@ramfox ramfox moved this to 👍 Ready in iroh Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@flub flub

Labels
bug Something isn't working c-iroh c-iroh-relay
Projects
iroh
Status: 👍 Ready
Milestone
v0.30.0
Development

No branches or pull requests
2 participants
@flub @PaulOlteanu