A CUE Based Approach to Kubernetes Manifest Abstractions

[slewiskelly]

The below discusses a CUE based approach to generate Kubernetes manifests from one or more abstractions provided to users.

The intention of this is to receive feedback and suggestions from the CUE community.

Goals

The goals of the abstraction framework are:

• To generate an artifact, a set of Kubernetes manifests, based on one or more higher level abstractions
• Provide users with a means to add certain resources which are not covered by a high-level abstraction (ConfigMap, etc)
• Provide users with certain options, such as, but not limited to:
  • Common configuration settings that may require frequent tuning (scaling, resources, etc)
  • Enabling and customization of features (Istio, rollout types, etc)
• The ability to version, users should be able to make informed decisions about when to receive updates as a result of version changes
  • Automation can help in ensuring that version updates are received, but it out of the scope of this discussion

Future goals of adopting CUE, but are not immediate, may be:

• Providing users with constrained access to Kubernetes resources, as an alternative to YAML
  • The benefit of this would be to allow users to configure their manifests using CUE, without having to copy lots of configuration which is either considered boilerplate, default, or required
• Providing users with unconstrained access to Kubernetes resources, as an alternative to YAML
  • The benefits of this are less clear than constrained access, but use cases for choosing CUE over YAML for such manifests may exist

Non-Goals

The abstraction framework does not, at least yet:

• Attempt to deploy/prune resources to/from a cluster
  • This is a concern for the delivery system
  • As there are multiple paths to deploying resources (e.g. kubectl, Spinnaker), this concern that may be addressed by the relevant component

Design

Abstractions

Note

• Examples below are from a work in progress
• Examples below may contain incomplete or duplicate information for the sake of illustration purposes
  • A more complete example can be found in this PR

Application

The reason this is expressed as an application and not a Deployment is because the result is a set of Kubernetes resources, rather than just a single Deployment. The comment in the below example shows which resources are generated.

// Application is an abstraction of multiple Kubernetes resources.
//
// Depending on the configuration of the application it may generate the
// following resources:
//  - Deployment
//  - DestinationRule (if Istio is enabled)
//  - HorizontalPodAutoscaler
//  - Pipeline
//  - PodDisruptionBudget
//  - Service (if ports are exposed)
//  - VirtualService (if Istio is enabled)
application: [Name=_]: base & {
	// Name of the application.
	name: Name

	// Spinnaker specific configuration.
	delivery: #Delivery

	// Docker image name, excluding the image registry prefix.
	// The image registry is assumed to be "gcr.io/$SERVICE-prod/".
	// Defaults to the application's name.
	image: string | *name

	// Minimum number of replicas, as a percentage, that must be available.
	// Replicas cannot be rescheduled until at least the number of repliacs
	// are available.
	// Defaults to "50%".
	minAvailable: string & =~"^([1-9]%$|^[1-9][0-9]%$|^100%)$" | *"50%"

	// Network configuration.
	network: {
		// Whether the application should use the Istio service mesh.
		// Defaults to false.
		istio: bool | *false
	}

	// Scaling configuration.
	scaling: {
		// Minimum number of replicas of the application.
		// This must always be greater than one, and less than or equal to, the
		// maximum number of replicas.
		// Defaults to 2.
		minReplicas: int

		// Maximum number of replicas of the application.
		// This must always be greater than, or equal to, the minimum number of
		// replicas.
		// Defaults to 3.
		maxReplicas: int

		// Target percentage CPU utilization before additional replicas are
		// created.
		// Defaults to "80%".
		targetCPU: string & =~"^([1-9]%$|^[1-9][0-9]%$|^100%)$" | *"80%"
	}

	additionalContainers: [...core_v1.#Container]

	// Environment variables required by the application.
	// These environment variables are simple key/value pairs.
	env: [string]: string

	// Environment variables that are sourced from ConfigMaps or Secrets.
	envFrom: [...core_v1.#EnvFromSource]

	// Environment variables required by the application.
	// These environment variables are more complex structures than key/value
	// pairs, such as those that reference values from fields or secrets.
	envSpec: [string]: {}

	// Ports exposed by the application.
	// Ports specified here will be exposed by a corresponding service.
	expose: [Name=_]: #Port & {name: Name}

	// Docker image name, excluding the image registry prefix.
	// The image registry is assumed to be "gcr.io/$SERVICE-prod/".
	// Defaults to the application's name.
	image: string

	// Ports exposed by the application.
	// Ports specified here will _not_ be exposed by a corresponding service.
	port: [Name=_]: #Port & {name: Name}

	// Minimum and maximum resources required by the application.
	resources: #Resources

	// CloudSQL configuration.
	sql: {
		// Set of Cloud SQL instances the proxy is to connect.
		instances: [...string]
	}

	// Volumes to be mounted by the application.
	volume: [Name=_]: #Volume & {name: Name}
}

See the appendix for complete examples of how a user would declare an application, and what the result would be.

It is noted that some of the syntax here may be intimidating to users who are used to configuring Kubernetes in pure YAML. For example, how would a user, just by reading this definition, know how to declare environment variables (especially when using envSpec), ports, or volumes.

In practice, the configuration a user would write is actually quite simple, but users will rely on good documentation and examples to get started.

The same concerns can also be applied to Jobs and CronJobs, mentioned below.

Job

Similarly to an application, more than just a Job resource is generated, but the naming is general enough that this distinction does not matter.

// Job is an abstraction of a Kubernetes Job.
job: [Name=_]: base & {
	// Name of the job.
	name: Name

	// Docker image name, excluding the image registry prefix.
	// The image registry is assumed to be "gcr.io/$SERVICE-prod/".
	// Defaults to the application's name.
	image: string | *Name

	// Network configuration.
	network: {
		// Whether the application should use the Istio service mesh.
		// Defaults to false.
		istio: bool | *false
	}

	// Number of completions of the job before signalling overall success.
	// A null completion signals success after a single completion, and allows
	// any number of instances to run in parallel.
	completions: *null | int

	// Maximum number of instances of the job that can be executed in parallel.
	// Number of executing instances may be lower if:
	// (completions - successes) < parallelism
	// Setting parallelism to zero blocks any instances from executing until
	// it is increased.
	// Defaults to 1.
	parallelism: int | *1

	// Number of seconds before the job is automatically deleted, after is has
	// completed. A null TTL signals that the job should not be automatically
	// deleted.
	ttl: *null | int

	additionalContainers: [...core_v1.#Container]

	// Environment variables required by the application.
	// These environment variables are simple key/value pairs.
	env: [string]: string

	// Environment variables that are sourced from ConfigMaps or Secrets.
	envFrom: [...core_v1.#EnvFromSource]

	// Environment variables required by the application.
	// These environment variables are more complex structures than key/value
	// pairs, such as those that reference values from fields or secrets.
	envSpec: [string]: {}

	// Ports exposed by the application.
	// Ports specified here will be exposed by a corresponding service.
	expose: [Name=_]: #Port & {name: Name}

	// Docker image name, excluding the image registry prefix.
	// The image registry is assumed to be "gcr.io/$SERVICE-prod/".
	// Defaults to the application's name.
	image: string

	// Ports exposed by the application.
	// Ports specified here will _not_ be exposed by a corresponding service.
	port: [Name=_]: #Port & {name: Name}

	// Minimum and maximum resources required by the application.
	resources: #Resources

	// CloudSQL configuration.
	sql: {
		// Set of Cloud SQL instances the proxy is to connect.
		instances: [...string]
	}

	// Volumes to be mounted by the application.
	volume: [Name=_]: #Volume & {name: Name}
}

See the appendix for complete examples of how a user would declare a job, and what the result would be.

Cron

Similarly to a Job, more than just a CronJob resource is generated. Again, the naming is general enough that this distinction does not matter.

A CronJob is a superset of a Job, adding only two additional fields:

// Cron is an abstraction of a Kubernetes CronJob.
cron: [Name=_]: base & {
        // Specifies a CronJob's policy for concurrent execution:
        //  - allow (default): concurrent executions are allowed
        //  - forbid: concurrent executions are not allowed
        //  - replace: existing executions will be cancelled
        policy:   *"allow" | "forbid" | "replace"

        // Schedule in which the job should execute, in Cron format.
        // See https://en.wikipedia.org/wiki/Cron.
        schedule: string
}

See the appendix for complete examples of how a user would declare a cronJob, and what the result would be.

Config Map

The abstraction of a ConfigMap exposed to the user is as simple as:

// ConfigMap is a minor abstraction of a Kubernetes ConfigMap.
configMap: [Name=_]: base & {
        // Name of the config map.
        name: Name

        // Key-value pairs representing the configuration data.
        data: [string]: string

        // Patches to be applied to individual resources.
        // Patches will fail to apply if they have been set by the abstraction.
        patch: configMap: {}
}

A user would then express a configMap like so:

// config_map.cue
configMap: foo: data: {
        hello: "world"
        lorem: "ipsum"
}

As a ConfigMap can be used by multiple applications within a namespace, there’s no simple way of binding it to a particular application, unless it is bound to all of them. There are also different ways in which a ConfigMap can be used (e.g. as environment variables or a volume mount)

As such, the user will have to reference it by the application(s) that require it, how it is intended to be used:

// ConfigMap as environment variables
application: foo: {
        // ...

        envFrom: bar: 
        envFrom: [
                {configMapRef: {name: "foo", optional: true}},
                {secretRef: {name:    "bar", optional: false}, prefix: "baz"},
        ]
}

configMap: bar: data: {
        hello: "world"
        lorem: "ipsum"
}
---
// Single value from ConfigMap as environment variables
application: foo: {
        // ...

        envSpec: HELLO: valueFrom: configMapKeyRef: {
                key: hello
                name: bar
        }
}

configMap: bar: data: {
        hello: "world"
        lorem: "ipsum"
}
---
// ConfigMap as a volume mount
application: foo: {
        // ...

        volume: bar: {
                mountPath: "/mnt/bar"
                source: configMap: name: "bar"
        }
}

configMap: bar: data: {
        hello: "world"
        lorem: "ipsum"
}

Service

Services are not an abstraction by themself. They are generated according to the ports listed under an application’s expose field. This way, the user does not need to bind the service to the Pod, and assign ports in the container.

A user would express which ports they want to expose the application on:

application: foo: {
        // ...

        expose: {
                grpc: port: 5000
                http: port: 8080
        }
}

Which will generate the following:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: foo
spec:
  template:
    spec:
      containers:
      - name: foo
        ports:
        - name: grpc
          protocol: tcp
          containerPort: 5000
        - name: http
          protocol: tcp
          containerPort: 8080
---
kind: Service
apiVersion: v1
metadata:
  name: foo
spec:
  selector:
    app: foo
  ports:
  - name: grpc
    port: 5000
    protocol: tcp
    targetPort: 5000
  - name: http
    port: 8080
    protocol: tcp
    targetPort: 8080

Beyond Abstractions

While the immediate goal is to provide abstractions, in the future it would be possible to provide CUE as the configuration language for all Kubernetes manifests.

Users would be able to choose from three ways to create their manifests:

• Raw Kubernetes schemas
• Kubernetes schemas with platform constraints
• Abstractions

While services can access raw Kubernetes directly. Some resource types should be protected, encouraging users to use the resources which have constraints applied, or more preferably, the provided abstractions. The reason for this is for convenience (sensible defaults can be set), and to ensure platform requirements are satisfied.

Providing access to raw Kubernetes manifests would be particularly useful when deploying third-party applications, where not all platform constraints can be applied.

Users may even, in the future, be able to create their own abstraction on top of the provided schemas.

Hierarchy

Our environment is multi-cluster, multi-environment, multi-tenant and our configuration repository reflects this.

It is important to developers to not have to copy all configuration that can be applied across environments and/or clusters. This is currently achieved by patching via Kustomize, where a general base is defined and more specific patches are applied following a standard directory structure.

CUE is able to accommodate the same, leading to a structure similar to the following:

microservices/
├── details
│   ├── dev
│   │   ├── kube.cue
│   │   └── osaka
│   │       └── kube.cue
│   ├── kube.cue
│   └── prod
│       ├── kube.cue
│       └── tokyo
│           └── kube.cue
├── productpage
│   └── prod
│       ├── kube.cue
│       ├── osaka
│       │   └── kube.cue
│       └── tokyo
│           └── kube.cue
├── ratings
│   └── dev
│       └── osaka
│           └── kube.cue
└── reviews
    ├── dev
    │   ├── kube.cue        # Configuration that applies to defined resources across all clusters in the "dev" environment
    │   └── tokyo
    │       └── kube.cue    # Configuration that applies to defined resources in the Tokyo cluster in the "dev" environment
    ├── kube.cue            # Configuration that applies to defined resources across all clusters in all environments
    └── prod
        ├── kube.cue        # Configuration that applies to defined resources across all clusters in the "prod" environment
        ├── osaka
        │   └── kube.cue    # Configuration that applies to defined resources in the Osaka cluster in the "prod" environment
        └── tokyo
            └── kube.cue    # Configuration that applies to defined resources in the Tokyo cluster in the "prod" environment

Appendix

Complete Example

Note

• Examples below are from a work in progress
  • A more complete example can be found in this PR

Minimal

Consider the following CUE configuration which declares an application, job and cronJob.

package kube

base: {
        environment: "prod"
        region:     "tokyo"
        service:     "minimal"
}

application: minimal: {}

job: minimal: {}

cron: minimal: schedule: "1 0 * * *"

This is the absolute minimum configuration required for each resource, and would generate this set of resources.

Customized

Now consider the following CUE configuration which now declares an application, job and a configMap, the cronJob is a superset of a job, so no need to show a customized version of it.

package kube

base: {
        environment: "prod"
        region:      "tokyo"
        service:     "maximal"
}

application: maximal: {
        labels: version: "v8"

        delivery: {
                type:           "canary"
                maxUnavailable: "0%"
                maxSurge:       "50%"
                trigger: enabled: true
        }

        minReplicas: 2
        maxReplicas: 3
        targetCPU:   "60%"

        minAvailable: "50%"

        expose: {
                grpc: port: 5000
                http: port: 8080
        }

        sql: instances: [
                "acme-maximal-prod:asia-northeast1:sql=tcp:3306",
                "acme-maximal-prod:asia-northeast2:sql=tcp:3306",
        ]

        resources: {
                requests: {cpu: 1000, memory: 512}
                limits: {cpu: 2000, memory: 1024}
        }

        env: {
                FOO:   "bar"
                HELLO: "world"
        }

        volume: foo: {
                mountPath: "/mnt/foo"
                source: configMap: name: "foo"
        }

        volume: bar: mountPath: "/mnt/bar"

        patch: deployment: spec: {
                minReadySeconds:         10
                progressDeadlineSeconds: 120
        }
}

cron: maximal: {
        completions: 1
        schedule:    "1 0 * * *"
        ttl:         36400
}

configMap: maximal: data: {
        hello: "world"
        lorem: "ipsum"
}

This is not exhaustive of all configuration options, but would generate this set of resources.

[from-nibly]

I think helper functions would be a lot more useful than abstractions. Sometimes it seems like kubernetes resources allow you to specify way too many properties that you don't care about, but you always end up needing to break out and configure one thing or another. I've been doing kubernetes stuff since 1.4 and you definitely want full access to the resource properties. Using helper functions instead of full blown abstractions can help you reduce boilerplate even when you are customizing one thing or another. if abstractions are wanted I think that would be something that should be stored in the repo for each orgs kubernetes setup, that way they can customize those abstractions to be what they need.

[mpvl]

A problem with abstractions is that they need to be updated as new features become available. This results in toil.

One technique one could use is to have a field that allows the user to specify raw configuration that is then unified with the root in which it is declared. The kubernetes tutorial has an example of that.

Regarding the foo: [string]: string syntax, the idea was to make it somewhat like Typescript in syntax, having at least some kind of familiarity.

In general, {} creates new values, () evaluates and substitutes, and [] is used for pattern matching. Hopefully explaining that regularity can help get users get up to speed quickly.

[from-nibly]

If you are going to make standard abstractions and then just make an escape hatch to go back to the raw resource why not just use helper functions? Then it's an opt in rather than an opt out. Any time the helper functions are missing or get wrong just don't use helper functions.

Unfortunately I'm a pretty big newb with CUElang but essentially you would do something like:

const defaultTCPHealth = (containerPorts: ContainerPort[]) => {
  return {
    tcpSocket: {
      port: containerPorts[0].containerPort,
    },
    initialDelaySeconds: 5,
    periodSeconds: 10,
  };
};

const containerPorts = [{ containerPort: 8000 }];

const deployment = {
  //....
  containers: [
    {
      name: 'foo',
      containerPorts: containerPorts,
      readinessProbe: defaultTCPHealth(containerPorts),
    },
  ],
};

Having a plethora of little helper functions provides a lot more utility and flexibility to defining resources than large abstractions that have to be escaped from when you need something more custom.

It's really a composition vs inheritance issue. Composition allows you to use or not use whatever things you need. Having to maintain abstractions that account for different network setups is going to be difficult with an abstraction. Calling out istio in the config is pretty prescriptive. If that's what this is going for then I just didn't do a good job of reading the brief and I apologize.

[slewiskelly]

A problem with abstractions is that they need to be updated as new features become available. This results in toil.

Right, but in the case where a group manages the platform (and any abstractions that come with it) for many users, it may allow them to rollout new features/patches/updates with less involvement from users. The same can be said for enforcing constraints.

Overall, I think we consider this a worthy investment in time.

One technique one could use is to have a field that allows the user to specify raw configuration that is then unified with the root in which it is declared. The kubernetes tutorial has an example of that.

I didn't call this out explicitly, but I did include a mechanism for this, as can be seen here. The customized example in the appendix shows this being used.

If you are going to make standard abstractions and then just make an escape hatch to go back to the raw resource why not just use helper functions?

Thanks for the suggestion, but exclusively calling out helper functions doesn't seem to offer complete alternative. From the example you've provided, I assume you're proposing that users express the configuration in code similar to cdk8s; and provide helper functions that assist with the build-up of configuration?

I see downsides to this approach as well; but as I don't have enough information, I wouldn't want to comment.

Also note that I've mentioned in the goals that, while not an immediate, we'd like to offer constrained an unconstrained CUE representations of Kubernetes resources.

Calling out istio in the config is pretty prescriptive. If that's what this is going for then I just didn't do a good job of reading the brief and I apologize.

The idea here is not to provide abstractions that can be used universally. The specific implementation that I have presented is specific to my environment, though the approach could be practiced for other environments by changing specific implementation details.

I don't think useful abstractions can be provided for general usage, or environments that lack consistency, for the reasons you've mentioned.

I think that would be something that should be stored in the repo for each orgs kubernetes setup, that way they can customize those abstractions to be what they need.

Letting users create their own abstractions is not explicitly called out, but may be the side-effect of also allowing them to use constrained or unconstrained CUE representations of Kubernetes resources (as mentioned above).

I've been doing kubernetes stuff since 1.4 and you definitely want full access to the resource properties.

I think this is a matter of perspective. Some developers don't know, or care to know, the full Kubernetes API. It is considered that many of our users fall into this category, so we want to offer an abstraction to such users so they can spend less time thinking about configuring Kubernetes.

I decided not to include it, but I think it may have been beneficial to explain a little more about the environment, the users, and how we got to considering such abstractions.

[cueckoo]

This discussion has been migrated to cue-lang/cue#806.

For more details about CUE's migration to a new home, please see cue-lang/cue#1078.