I would configure your Prometheus jobs to add the following labels:

• cluster: name of the K8S cluster
• job: "/<deployment/statefulset>" (eg. if Cortex is deployed in the "cortex-01" namespaces then ingesters would have the job label "cortex-01/ingester)

Could you precise the job part ?

Currently my prometheus is sending an external_label for cluster and jobs are automatically name after the deployment like cortex-compactor

Could you precise the job part ?

The job label is a label that we expect to be added by Prometheus (configured in the Prometheus scraping config) whose value is <namespace>/<deployment|statefulset|daemonset> where <namespace> is the namespace of where the pod is running and <deployment|statefulset|daemonset> is the name of the pod's deployment/statefulset/daemonset.

There is a job label in the metrics, it's just not in the value format Grafana expects:

Prometheus records the label as: job="cortex-distributor"
Grafana regex looks to match the label as: job=~"($namespace)/(distributor|cortex$)"

There is no way these two would match. I'm still hanging on the though that these label values should match between what Prometheus collects vs what Grafana expects and I'm not sure how comes they are so far apart.

Job label value is just one example, there are other labels in Grafana queries that would prevent the results from showing in the dashboards.

Anyway, the way I intent to deal with this is to fork cortex-mixin and fix queries to match Prometheus labels as much as possible, then adjust dashboards manually for the remaining.

The job label is a label that we expect to be added by Prometheus (configured in the Prometheus scraping config) whose value is <namespace>/<deployment|statefulset|daemonset> where <namespace> is the namespace of where the pod is running and <deployment|statefulset|daemonset> is the name of the pod's deployment/statefulset/daemonset.

Are there relabeling rules documented somewhere ? I'm trying to eliminate the guessing factor from the efforts. I deployed Cortex from your helm chart and the metrics come straight out of the Service Monitor without any relabeling. I've added the recording rules from cortex-mixin next to that, but I didn't come across any scrape config requirements in my readings.

To follow-up on this issue, I opened cortexproject/cortex-helm-chart#233 to ensure container names produced by the Cortex Helm chart align with cortex-mixin container label values.
Other than that, I had to significantly modify cortex-mixin's to get the dashboards to work after deploying Cortex from the official Helm chart. Waiting for a small change in the helm chart before I can PR this change.