notebook controller, pipelines components additional pod labels

[shalberd]

/kind feature

Why you need this feature:
Corporate-level admission controllers like https://kyverno.io that do not allow any pod to be created unless it has certain corporate policy-based labels.

Is there a way in kubeflow notebooks to set common labels for all workbench pods created?
We have this requirement that all pods created dynamically and in general need to have certain corporate label key-value combinations set.
At notebook / statefulSet spec.template.metadata.labels.

I am currently patching derived statefulsets' podSpec with our labels by means of a cronjob so our label enforcement tool / admission controller Kyverno allows pods to be created. That is just a workaround, though.

Apache Airflow has thought of that topic ( common labels) for all its KubernetesExecutor related resources and even for the static Helm Deployment. However, I think this additional / custom pod labels requirement is currently not implemented in notebooks, or even kubeflow pipelines, for that matter, globally.

At kubeflow upstream https://www.kubeflow.org/docs/components/notebooks/quickstart-guide/, I see mention of PodDefault Resource and profiles that could well do the job ... what do you think? Is something like this on the radar for ODH in terms of kind:PodDefault i.e. in ODH kubeflow notebook 2.0 ?

BUT I am not sure though if that concept applies to setting additional labels in a pod metadata as well.
Cause the similar concept of podPreset says: "Note: A Pod Preset is capable of modifying the .spec.containers field in a Pod spec when appropriate. No resource definition from the Pod Preset will be applied to the initContainers field."
https://cjyabraham.gitlab.io/docs/concepts/workloads/pods/podpreset/
In any case, that requirement for additional labels is often implemented in software not just Airflow ... so I'd be surprised if there were no way to achieve this by means of kubeflow.

Describe the solution you'd like:
I need to have a way to centrally define and configure additional pod (not notebook) labels for pods derived from a notebook spec / statefulset.
Then, notebook controller needs to add those labels to the notebook podSpec and to the statefulSet podSpec on reconciliation.

Anything else you would like to add:
Same issue applied to spawned objects / pods in kubeflow pipelines as well.
Is there a way somewhere to specifiy additional pod labels including their values that are set when pods are spawned dynamically.

[jiridanek]

Currently we don't reconcile labels and annotations, except when doing the initial creation. So setting annotations and labels for the podspec in Notebook CR will propagate it to statefulset and then to pod, but if you later change it or add new ones on the Notebook CR, that will not be applied then.

Is there a way in kubeflow notebooks to set common labels for all workbench pods created?

Sounds to me like the Notebooks 2.0 feature about WorkspaceKinds, if you already had a look at that. In OpenDataHub we have a Template that Dashboard uses when spawning, so that would be also a way. The 1.x notebook-controller does not have a feature like that, as far as I know.

[jiridanek]

At kubeflow upstream https://www.kubeflow.org/docs/components/notebooks/quickstart-guide/, I see mention of PodDefault Resource and profiles that could well do the job ... what do you think?

That's implemented through the UI spawner, right? So same thing as the OpenDataHub Templates, then.

[jiridanek]

I am currently patching derived statefulsets' podSpec with our labels by means of a cronjob so our label enforcement tool / admission controller Kyverno allows pods to be created. That is just a workaround, though.

Kyverno itself could do the patching in a mutating webhook, I think. It's not any better than what you describe, that's true.

[shalberd]

Yeah, the idea behind this ticket is to get upstream awareness, this is an important issue across kubeflow

[shalberd]

Hi @jiridanek regarding notebook CR assembly and your statement on templates:

In OpenDataHub we have a Template that Dashboard uses when spawning, so that would be also a way.

Where does not happen currently with regard to pod labels? I do not see that, need some guidance. https://github.com/opendatahub-io/odh-dashboard/tree/main/manifests/common/crd

I do not see anything custom during assemby for spec.template.metadata.labels https://github.com/opendatahub-io/odh-dashboard/blob/main/frontend/src/api/k8s/notebooks.ts#L125

[jiridanek]

Dashboard has the "template" hardcoded, so you're right, what I said is happening is not actually happening

https://github.com/opendatahub-io/odh-dashboard/blob/fee50c52cdac09421dae92a925a328d4f3eaad26/frontend/src/api/k8s/notebooks.ts#L96

I must've confused that with modelservers, they are indeed created using OpenShift template

https://github.com/skodjob/odh-e2e/blob/215368e00a8784b6a6d89e4cbfddff203cc34c9e/src/test/java/io/odh/test/e2e/standard/ModelServingST.java#L224

[shalberd]

I just realized I did post this at ODH instead of upstream ... wondering whether making a feature request upstream, be it for kubeflow notebooks 2.0 or kubeflow pipelines, anything that spawns a pod, would be a good idea ... the topic is so generic, it should be on the radar upstream.

[jiridanek]

wondering whether making a feature request upstream, be it for kubeflow notebooks 2.0 or kubeflow pipelines, anything that spawns a pod, would be a good idea

YES!

[jiridanek]

posted in kubeflow at

• Kubeflow Notebooks 2.0 (Kubeflow Workspaces) kubeflow/notebooks#85 (comment)
• feature: allow custom pod labels in notebooks, pipeline components, serving pods kubeflow/community#800