Automated interventions for performance issues: DDoS Protection and Performance Notifications

[dunkOnIT]

After the poor performance of WCA Live during 3x3 R1, which appears that it may be the result of a DDoS (not clear if malicious), some thoughts come to mind in terms of improving such situations in the future. I'm looking to understand whether any of these would be viable to implement. If so, I'll go ahead and open issues for them.

DDoS Protection

At a naive level, we should be able to programmatically identify an unreasonable amount of requests from the same IP and take some kind of action against them. I'd expect there are already services and technologies in place that enable this.

What is usually used to provide DDoS protection, and why don't we/can't we use it?

Performance Notifications

It would be great if we were already aware of live performance issues and making proactive contact with our users, instead of needing to be informed by users/via Slack.

Can we enable notifications for when is significant load on our servers? (eg, CPU hitting 100% for several minutes). Even better if we can put them in a Slack channel to be discussed in threads.

[dunkOnIT]

@FinnIckler @gregorbg you may have input here.

[FinnIckler]

I was thinking about this last night as well. There are two options for blocking these.
a) we have some logic on the nginx/the software that is part of the liv setup
b) we switch to the AWS load balancer and attach a WAF (web application firewall) to it. There we can define rules like if X amount of traffic comes from IP Y for Z amount of time block it.

As we are planning to switch to a load balanced approach anyway, b) seems the most reasonable. Each rule check does come with a price though, which we need to look into. As it depends on traffic we might save some money from not serving the Malicious requests and it will even itself out.

For Monitoring, I can set up some kind of slack integration

[dunkOnIT]

Slack integration would be awesome! Would you be happy for me to create/assign an issue to you for that?

[dunkOnIT]

Load balancer + WAF definitely sounds preferable to my ears, especially because I'm imagining those rules can be configured via AWS, and so easily propagated to other servers, as opposed to nginx config/logic which would be a bit harder to ship around?

How much effort would load balancer + WAF be to set up? I would suggest we try set it up, run it for a week, and use that to generate a cost estimate - or would we rather do some napkin math beforehand?

[dunkOnIT]

(if we get WAF working, we might well want to use that as a basis to re-introduce /results)

[FinnIckler]

The issue for /results was that even a very small amount of requests could lead to a DB overload. We don't want to block any legitimate users so I think the caching idea is better.

[FinnIckler]

It would not be much effort to set up the infrastructure (maybe an hour), I think the biggest change we need to think about how the load balancer integrates with the web sockets. ALB does support Web sockets, but we need to change the SSL Termination to the ALB and think about correct values for idle connection timeout settings.