Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alertmanager grouping is not working on Calert 2.0.2 #44

Open
marceloavilaoliveira opened this issue May 17, 2022 · 6 comments
Open

Alertmanager grouping is not working on Calert 2.0.2 #44

marceloavilaoliveira opened this issue May 17, 2022 · 6 comments
Assignees
Labels
bug Something isn't working

Comments

@marceloavilaoliveira
Copy link

Prometheus 2.35.0
Alertmanager 0.24.0
Calert 2.0.2

After I upgraded the Calert from 1.2.4 to 2.0.2, Alertmanager grouped alerts are no longer being sent to a unique Google chat thread.

On Calert 1.2.4

Calert-1 2 4

On Calert 2.0.2

Calert-2 0 2

Note1: I noted that if I removed the "fingerprint" field of the Alertmanager JSON test payload or use the same "fingerprint" value for the grouped alerts, then the alerts go to the same thread. Actually, this is not possible because Alertmanager generates different fingerprints for different alerts in the same group.

Note2: The Calert threading support seems to be working correctly, I mean, the "Firing" and "Resolved" alerts are being sent to the same Google chat thread (one thread for each alert in the group, unfortunately).

@mr-karan
Copy link
Owner

Can you show your Alertmanager config? Wanted to see what’s the grouping you’ve defined there.

@marceloavilaoliveira
Copy link
Author

route:
  group_by: [group, alertname]
  group_wait: 45s
  group_interval: 5m
  repeat_interval: 8h
...

The "group" label is the team responsible for that alert and defines which chat room will receive it. If you need more info please let me know.

@mr-karan mr-karan self-assigned this May 18, 2022
@mr-karan mr-karan added the bug Something isn't working label May 18, 2022
@mr-karan
Copy link
Owner

mr-karan commented Sep 5, 2022

@marceloavilaoliveira I think the current behavior is OK. Different thread will help in seeing that a Resolved alert came to the same thread. The issue with first approach is that you don't clearly see how many alerts are pending to be resolved.

@mr-karan mr-karan closed this as completed Sep 5, 2022
@marceloavilaoliveira
Copy link
Author

marceloavilaoliveira commented Sep 5, 2022

I don't know if I understood correctly... are you saying that we can't group alerts anymore? This is an essential Alertmanager feature because, sometimes, an issue rises many alerts and it's needed to group them in the same message.

@marceloavilaoliveira
Copy link
Author

There's no sense in do not group the alerts in the same chat thread, because this is the way they're grouped when using e-mail instead of chat.

@mr-karan mr-karan reopened this Dec 30, 2022
@mr-karan
Copy link
Owner

@marceloavilaoliveira Will make a provision to disable grouping in a future release as an optional configuration. Can't promise a timeline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants