Add discarded samples per labelset metric #6464
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Ben Ye <[email protected]>
alanprot
reviewed
Dec 27, 2024
pkg/ingester/ingester.go
Outdated
Comment on lines
1156
to
1170
| updateLabelSetCounter := func(matchedLabelSetLimits []validation.LimitsPerLabelSet, reason string) { | ||
| for _, l := range matchedLabelSetLimits { | ||
| if c, exists := labelSetCounters[l.Hash]; exists { | ||
| c.reasonCounter[reason]++ | ||
| } else { | ||
| labelSetCounters[l.Hash] = &labelsCounter{ | ||
| reasonCounter: map[string]int{ | ||
| reason: 1, | ||
| }, | ||
| lbls: l.LabelSet, | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
Could this be a function of the (l *Limits) struct?
There was a problem hiding this comment.
Made it a function of the struct
|
ok.. this is ok i guess as the new metric will only be available if the "per labelset limit" is configure. |
alanprot
approved these changes
Dec 28, 2024
Signed-off-by: Ben Ye <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Add
cortex_discarded_samples_per_labelset_totalmetric to record samples discarded for each configured label set limit.The
labelSetTrackeradded in #6443 was part of the distributor package. I refactored that and use the same data structure to track label set used by validate metrics.Note that not all discarded samples reason supports this per labelset metric. Certain reasons such as too many HA groups, ingestion rate exceeded just drops the whole push request instead of an individual series. So in order to get per labelset discarded samples we have to iterate over all series labels, which is possible to have performance impact so I didn't do that in this PR.
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]