Forwarding pod labels as Prometheus metric labels

[clouedoc]

Hi there,

I am migrating to the Flow mode, and I am struggling to replicate a functionality: keeping Pod labels as Prometheus metric labels.

Here is an example pod:

It exposes metrics, and those are scrapped well in my current configuration.

However, I also want my metrics to have the following labels:

app_kubernetes_io_name=ns
noselenium_tagrecords_dev_account_owner=gen_account
etc.

I have tried using the re-labeling component to no success:

// Discover Kubernetes pods to collect metrics from.
discovery.kubernetes "pods" {
  role = "pod"
}

// Scrape metrics from Kubernetes pods and send to a prometheus.remote_write
// component.
prometheus.scrape "default" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [prometheus.relabel.add_k8s_labels.receiver]
  honor_labels = true
}

prometheus.relabel "add_k8s_labels" {
  forward_to = [prometheus.remote_write.default.receiver]

  // Expose Pod labels as metric labels
  rule {
    action = "labelmap"
    regex = "__meta_kubernetes_pod_label_(.+)"
  }
}

// Collect and send metrics to a Prometheus remote_write endpoint.
prometheus.remote_write "default" {
  endpoint {
    url = "http://mimir-nginx.grafana.svc:80/api/v1/push"
  }
}

Here's how my graph looks:

Here are the labels that are available on my metric:

---

Thanks for your help!