Auto import images for containerd image store

[vitorsavian]

Proposed Changes

• ADR
• Auto import images form .txt files and tar files

Types of Changes

• New Feature

Verification

• run a k3s server

k3s server

• create the images folder

mkdir /var/lib/rancher/k3s/agent/images

• create a .txt file and add a docker.io image path to the .txt file

docker.io/library/mysql:latest

• move the file to /var/lib/rancher/k3s/agent/images

cp example.txt /var/lib/rancher/k3s/agent/images

• then see if the file was imported in the containerd image store

k3s ctr images list | grep "mysql"

• you can also change the file inside var/lib/rancher/k3s/agent/images

nano /var/lib/rancher/k3s/agent/images/example.txt

• then you can add

docker.io/library/redis:latest

• the controller will see the change and then import the redis image

k3s ctr images list | grep "redis"

Testing

Linked Issues

• Auto-import images into the embedded registry while k3s is running #9759
• E2E: Add preload images subtest #8084

User-Facing Change

Users can now auto import images to containerd by just throwing the image in the agent/images folder while k3s is running

Further Comments

[brandond]

Users can now add images in the embedded registry auto import folder while k3s is running

To avoid confusion, remove any references to importing things into the embedded registry - the embedded registry does not have a discrete image store to import into. We import images into the containerd image store, and images in the containerd image store can be used by Kubernetes pods without needing to be pulled from a remote registry, and they can also be shared between nodes by the embedded registry.

[codecov]

Codecov Report

Attention: Patch coverage is 45.20548% with 120 lines in your changes missing coverage. Please review.

Project coverage is 45.37%. Comparing base (4ec2617) to head (e5e2332).

Files with missing lines	Patch %	Lines
pkg/agent/containerd/watcher.go	46.63%	79 Missing and 24 partials ⚠️
pkg/agent/containerd/containerd.go	34.61%	11 Missing and 6 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10973      +/-   ##
==========================================
- Coverage   46.98%   45.37%   -1.61%     
==========================================
  Files         179      180       +1     
  Lines       18610    18812     +202     
==========================================
- Hits         8743     8536     -207     
- Misses       8505     8972     +467     
+ Partials     1362     1304      -58     

Flag	Coverage Δ
e2etests	39.67% <45.20%> (-1.74%)	⬇️
inttests	34.63% <19.17%> (-0.13%)	⬇️
unittests	13.67% <0.00%> (-0.15%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[brandond]

We probably shouldn't do it in this PR, but it would be nice if whatever we used for watching image file changes was generic, and could be reused by the deploy controller - right now it just re-scans everything once a minute:

k3s/pkg/deploy/controller.go

Lines 85 to 100 in 8ce04d3

	// start calls listFiles at regular intervals to trigger application of manifests that have changed on disk.
	func (w *watcher) start(ctx context.Context, client kubernetes.Interface) {
	w.recorder = pkgutil.BuildControllerEventRecorder(client, ControllerName, metav1.NamespaceSystem)
	force := true
	for {
	if err := w.listFiles(force); err == nil {
	force = false
	} else {
	logrus.Errorf("Failed to process config: %v", err)
	}
	select {
	case <-ctx.Done():
	return
	case <-time.After(15 * time.Second):
	}
	}

[brandond]

Why are we creating this now? Must the folder exist in order to watch it? Is there any other way to handle this?

[brandond]

What happens if the user makes the images dir a symlink to another path? What happens if the image dir is deleted and recreated while the watcher is going? These are all things that users are going to try at some point.

[vitorsavian]

The behavior after the changes now are that the image watcher will watch the k3s/agent and only work with the events that contains the agent/images, if the image folder is already created, the preloadImages will load the images, so the watcher will not need to load, but if the image folder is created or symlink with a folder with images inside, the watcher will send to the workqueue for the images to be loaded.

If the folder is deleted, fsnotify can auto delete and because we are watching the /k3s/agent, if the user creates the folder again, we will watch it.

[brandond]

minor nit, lgtm otherwise!

[manuelbuil]

Sorry for the late review. I'm a bit confused about the use case. My guess is: we have a cluster where we would like only one (or more) node(s) to pull the images for the whole cluster. The rest of the nodes would pull the images from these selected nodes by using the embedded registry mirror. We can't easily do this today because containerd will only pull the images if there is a local pod requiring it. This feature would "force" containerd to pull the image, even if no local pod requires it. Is my understanding correct?

[manuelbuil]

Why would users have to manually import images? Aren't they automatically pulled by containerd once the pod is scheduled in the node?

[manuelbuil]

What does the pinned tag mean? That is was manually pulled?

[manuelbuil]

Why do we need to restart the cluster?