Adding kubernetes labels to Kepler metrics

[AdrianHammond]

Does Kepler, or can Kepler, include K8s label with the data it exports. The use case is that labels are used to tag namespaces / project with organisation data to make it easier to attribute ownership.

Thanks Adrian

[rootfs]

kepler metrics currently have pod name and pod namespace. It wouldn't be hard to group/aggregate/filter by kube_pod_labels

[AdrianHammond]

Would it help for me to get more info on what other labels the customer uses for you? I assume this would be an enhancement to Kepler?

[rootfs]

we can start with prometheus query, the join metrics example could be a good start.

[AdrianHammond]

@rootfs - I spoke to the customer who raised this with me today, they use n/s labels to associate all applications with a business unit. They want to be able to visualise data at the BU level in addition to individual n/s. Thanks A

[AdrianHammond]

This JIRA has been opened - https://issues.redhat.com/browse/OBSDA-496

[marceloamaral]

First draft implementation of this: https://github.com/marceloamaral/label-exporter

[marceloamaral]

Does the use need more than one label?
If only one label is needed, you can just change the prefix --label-prefix="application.name".
If the user need more labels, I change update it to the label-prefix parameter be a list.

[AdrianHammond]

Yes, likely they would use multiples. e.g. application.name/retail and application.process/payment

[marceloamaral]

@AdrianHammond I will update for the list then!
Also, I updated the README so you can see the example of using only one label for now.

[marceloamaral]

@AdrianHammond, updated the pod label exporter. The user can configure it with a list of labels now.
You can see the example in the github README.

[AdrianHammond]

@marceloamaral LGTM - thank you

[sthaha]

@rootfs @marceloamaral , Could we consider allowing the same functionality as a feature flag in kepler? e.g.
kepler --collector.labels=<label-key> --collector.labels=<label-key> this would allows kepler to export labels only if required and the overhead should be minimal. The additional benefit is that it allows for the deployment management (daemonsets, rbac, configuration, ) and project management (release and maintenance etc) to be minimal.

[marceloamaral]

The label exporter should be a single deployment in the cluster, and then we can also support other kind of objects such as Job. Don't need to be only pods.

Note that we cannot watch for jobs in Kepler because it will introduce too much overhead in the API server as Jobs cannot be filtered per node as we can do for pods.

Also, Kepler metrics already have many labels and we should not increase too much the cardinality because of Prometheus overhead constraints.

@rootfs can we move the label-exporter under the https://github.com/sustainable-computing-io/?
Once we have it there, we could support the deployment via operator.

[sthaha]

I agree on the cardinality in general but in my humble opinion that choice can be left to users? By default, kepler collects no additional labels. Also, I believe k-s-m already has the feature to collect pod labels see: https://github.com/kubernetes/kube-state-metrics/blob/main/docs/pod-metrics.md. But again, you don't need an additional deployment/ overhead in cluster if kepler provides this (optional) feature.

[rootfs]

thank @marceloamaral the label-exporter is now here https://github.com/sustainable-computing-io/label-exporter