The relay send path of an Endpoint is too easily congested.

[PaulOlteanu]

I'm occasionally seeing Endpoints get into a state where they're unable to connect to any other nodes, and the logs start showing this:

2024-12-04 04:31:18 -0500 EST debug "send relay: message dropped, channel to actor is full"
2024-12-04 04:31:18.002 -0500 EST debug "relay channel full, dropping call-me-maybe"
2024-12-04 04:31:19.926 -0500 EST debug "send relay: message dropped, channel to actor is full"
2024-12-04 04:31:19.928 -0500 EST debug "send relay: message dropped, channel to actor is full"
2024-12-04 04:31:19.929 -0500 EST debug "relay channel full, dropping call-me-maybe"
2024-12-04 04:31:19.931 -0500 EST debug "send relay: message dropped, channel to actor is full"
2024-12-04 04:31:20.926 -0500 EST debug "send relay: message dropped, channel to actor is full"
2024-12-04 04:31:20.929 -0500 EST debug "no UDP or relay paths available for node"
2024-12-04 04:31:20.93 -0500 EST debug "send relay: message dropped, channel to actor is full"
2024-12-04 04:31:20.932 -0500 EST debug "no UDP or relay paths available for node"
2024-12-04 04:31:22.924 -0500 EST debug "send relay: message dropped, channel to actor is full"
2024-12-04 04:31:22.925 -0500 EST debug "no UDP or relay paths available for node"
2024-12-04 04:31:22.926 -0500 EST debug "send relay: message dropped, channel to actor is full"
2024-12-04 04:31:22.927 -0500 EST debug "no UDP or relay paths available for node"
2024-12-04 04:31:22.999 -0500 EST debug "send relay: message dropped, channel to actor is full"
2024-12-04 04:31:23.001 -0500 EST debug "send relay: message dropped, channel to actor is full"
2024-12-04 04:31:23.003 -0500 EST debug "relay channel full, dropping call-me-maybe"
2024-12-04 04:31:23.004 -0500 EST debug "send relay: message dropped, channel to actor is full"
2024-12-04 04:31:23.006 -0500 EST debug "send relay: message dropped, channel to actor is full"
2024-12-04 04:31:23.007 -0500 EST debug "relay channel full, dropping call-me-maybe"

This continues for a few minutes and never seems to recover until the Endpoint is dropped and a new one is created, at which point things work fine again.

In this particular case the client was in India, connecting to a relay in the US (because we're only running one relay). Not sure if latency to the relay could affect this. I have seen this happen with a client from Canada so I don't think it would be exclusively due to high latency

[flub]

Thanks for filing this. It is a kind of a dup to #2951 (I appreciate there's no way to tell this!) but I hadn't split the sender-side relay path into it's own issue yet like the others so will use this one to track it.

[flub]

#2971 is also very likely a dup of this. Though that also talks about the relay server not finding clients, which is likely either a red herring or a different issue.