Cloudflare (continuously) rate limiting API calls

[arbrebinaire]

Hi,

It seems the Cloudflare API is over solicited by the bouncer. One of the multiple log lines:

level=error msg="you have been ratelimited please wait and try again (10040)"

That would be fine if it would be occasional; but on the contrary each and every single call made is rate limited without a single call being successful, for multiple succeeding days. That is rather not sustainable, especially since data is transferred continuously (1 Mb/s, all day, every day -- please see below a screen grab of my Linode IPv6 transfer)

The only way to give the API a rest:

systemctl stop crowdsec-cloudflare-bouncer

After restarting it, an hour or so of peace and quiet. Then, back to the 10040 error code.

I'd suggest some sort of exponential backoff. I would try to tackle the issue myself but I'm barely a script kiddie in Go.

Thanks.

[LtSich]

Hi, what is your update_frequency for cloudflare on your bouncer ?
Try to increase it. I have 1min on my bouncer.
If you have really a lot of request to do, try even more if needed.

[kafkaah]

Hi, what is your update_frequency for cloudflare on your bouncer ? Try to increase it. I have 1min on my bouncer. If you have really a lot of request to do, try even more if needed.

It's at 10 seconds... The default.

I'll try 120s and see what happens.

Thanks.

[arbrebinaire]

Happy to report that changing the update_frequency to 3600 seconds finally solved the problem. 3 hourly calls were rate limited and the 4th was successful. Please note that the first 3 failed hourly calls only peaked at a few Kb/s instead of a perpetual 1+ Mb/s: don't understand why.

Since I use both my box local firewall and Cloudflare (Cloudflare alone can only block port 80 and 443 and I use it mainly to reduce the stress on the local firewall), the (very) low frequency is not much of a problem.

Thank you for a very simple fix.

[marcogiorgio]

Hi,
I am experiencing the same problem but I don't understand if the issue is fixed or not..
Based on this comment it seems that GET requests shouldn't count against rate limitations.
However, I've disabled the CAPI functionality in my Crowdsec LAPI server, but when I try to ban a single IP, the Cloudflare bouncer gets rate limited

[LtSich]

You will need to use that for now if you want to ban on Cloudflare side : https://docs.crowdsec.net/u/bouncers/cloudflare-workers/
Using list is totally broken as Cloudflare rate limit you even if you only make 1 request / hour...

[marcogiorgio]

Thanks for letting me know. Unfortunately, I am on the free Cloudflare plan, so based on the first banner in the page you linked I guess I'll need to wait..

[blotus]

Hello @marcogiorgio,

You can technically use the new cloudflare bouncer on the free plan, but with some limitations (including a very big one, hence the warning in the documentation):

• You are allowed 100k worker requests per day. If your website needs more than that, the bouncer will stop working for the rest of the day
• You are allowed 100k read requests to the KV store: same thing as above.
• You are allowed 1k write requests to the KV store: This is the biggest one. Using the community blocklist (or 3rd party lists) can easily make you make over 50k writes (and only the first 1k will be actually written to the KV), this means that you won't be able to use the community blocklist content in a free plan and have to set crowdsec_config.only_include_decisions_from to ["crowdsec", "cscli"] in the bouncer configuration.

[marcogiorgio]

Hello @marcogiorgio,

You can technically use the new cloudflare bouncer on the free plan, but with some limitations (including a very big one, hence the warning in the documentation):

* You are allowed 100k worker requests per day. If your website needs more than that, the bouncer will stop working for the rest of the day

* You are allowed 100k read requests to the KV store: same thing as above.

* You are allowed 1k write requests to the KV store: This is the biggest one. Using the community blocklist (or 3rd party lists) can easily make you make over 50k writes (and only the first 1k will be actually written to the KV), this means that you won't be able to use the community blocklist content in a free plan and have to set `crowdsec_config.only_include_decisions_from` to `["crowdsec", "cscli"]` in the bouncer configuration.

Hi @blotus,
thanks for the info! I am trying to use this bouncer but it doesn't seem to work (I can't see any IP added in the KV store when a ban is triggered).
I can see from the LAPI logs that the bouncer periodically performs GET /v1/decisions/, but the bouncer logs doesn't show anything (the last line of its logs is "Starting turnstile rotator" for the zone I am interested in.
This is the config I am using:

Bouncer config

# Config generated by using /etc/crowdsec/bouncers/crowdsec-cloudflare-worker-bouncer.yaml as base
cloudflare_config:
  accounts:
      - id: <id>
        ban_template: "" # template to use for ban action, set empty to use default
        zones:
          - zone_id: <id>
            actions: # supported actions for this zone. eg value ["ban", "captcha"]
              - ban
            default_action: captcha
            routes_to_protect:
              - '*<my_domain>/*'
            turnstile: # Turnstile must be enabled if captcha action is used.
              enabled: true
              rotate_secret_key: true
              rotate_secret_key_every: 168h0m0s
              mode: managed
          - zone_id: <id>
            actions: # supported actions for this zone. eg value ["ban", "captcha"]
              - ban
            default_action: ban
            routes_to_protect:
              - '*<my_domain>/*'
            turnstile: # Turnstile must be enabled if captcha action is used.
              enabled: true
              rotate_secret_key: true
              rotate_secret_key_every: 168h0m0s
              mode: managed
          - zone_id: <id>
            actions: # supported actions for this zone. eg value ["ban", "captcha"]
              - ban
            default_action: ban
            routes_to_protect:
              - '*<my_domain>/*'
            turnstile: # Turnstile must be enabled if captcha action is used.
              enabled: true
              rotate_secret_key: true
              rotate_secret_key_every: 168h0m0s
              mode: managed
          - zone_id: <id>
            actions: # supported actions for this zone. eg value ["ban", "captcha"]
              - ban
            default_action: ban
            routes_to_protect:
              - '*<my_domain>/*'
            turnstile: # Turnstile must be enabled if captcha action is used.
              enabled: true
              rotate_secret_key: true
              rotate_secret_key_every: 168h0m0s
              mode: managed
        token: <my_token>
        account_name: <my_account_name>
crowdsec_config:
  lapi_url: http://crowdsec-engine:8080/
  lapi_key: <my_lapi_key>
  update_frequency: 10s
  include_scenarios_containing: [] # ignore IPs banned for triggering scenarios not containing either of provided word
  exclude_scenarios_containing: [] # ignore IPs banned for triggering scenarios containing either of provided word
  only_include_decisions_from: ["cscli"] # only include IPs banned due to decisions orginating from provided sources. eg value ["cscli", "crowdsec"]
  key_path: ""
  cert_path: ""
  ca_cert_path: ""
daemon: false
log_level: info
log_mode: ""
log_dir: /var/log/
prometheus:
  enabled: true
  listen_addr: 127.0.0.1
  listen_port: "2112"

Am I missing something obvious?

EDIT: Yes, I missed "crowdsec" in crowdsec_config.only_include_decisions_from. For some reason I though I only needed to set cscli to exclude the community lists (which I thought they were the IPs taken from the Central API). But it's not like that. My bad, thanks again 😄