Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable multi-node support #20

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

cfmack
Copy link

@cfmack cfmack commented Dec 31, 2024

My understanding of the implementation struggled in multi-node instances of celery running. It relied on the server sending the task to the celery queue to be the same server processing the queue. That implementation assumption does not work with a cluster of celery nodes (or dedicated worker machines).

The changes in this PR boil down to a few things:

  1. If the event does not have a name at the time of creation, it will give it a name based on a new configuration called task-prefix and appends the task guid. The name would be mytasks-1231-111-1111-1111 as an example
  2. Allow the "Starting" event to create a task with a name based on the above. I could not find a way to extract the name from the Start event (unsure where that is sourced from). Name appears to be critical for downstream consumption when using patterns
  3. Move pattern from fnmatch to re. I simply could not get fnmath to work with non-alpha numeric characters. re is much more common
  4. Add ability to whitelist events up to Cloudwatch. In a multi-node deployment, if there is a dedicated worker node, then you do not want to send up Sent events, as that is misleading telemetry. One can also limit all the other metrics for a machine that only sends celery tasks to the queue but does not work them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant