User Workspaces in Kubernetes

[ConnorNelson]

This is still a work in progress.

This PR migrates user workspaces from Docker to Kubernetes in order to support multi-node deployments of the dojo. The goal is to still support single-node (for easy development, and smaller deployments), while making it possible for user workspaces to appear on other nodes in larger deployments.

In multi-node deployments, there will be a primary node that hosts all of the core infrastructure (site, db, redis, nginx, discord bot, etc). That primary node will also spin up a container that acts as the kubernetes "server" (in control). Other nodes will only spin up a kubernetes "agent" (worker) to house the user workspaces.

For example on the primary node:

$ docker run -it --rm --privileged -e DOJO_HOST=pwn.college -p 22:22 -p 80:80 -p 443:443 -v $PWD/data:/opt/pwn.college/data pwncollege/dojo

And on any worker

$ docker run -it --rm --privileged -e DOJO_SERVER=pwn.college -e DOJO_SECRET=112233445566778899aabbccddeeff -v $PWD/data:/opt/pwn.college/data pwncollege/dojo

Once a worker completes its setup and connect-in process to the primary node, user workspaces may now be scheduled to run there.

The goal of this PR is not to run other services (the site, db, etc) in kubernetes. However, we might later decide to migrate those services there if it makes sense.

---

I am going to be storing TODOs, thoughts, brainstorming, etc. here as I continue to build this out.

Registry

• Now build the challenge image as registry:5000/challenge:latest, and then push it to the registry, and kubernetes pulls it down.
  • This allows us to get the challenge image into kubernetes.
  • Feels slow; have to not only build it, but then also push it and pull it.
  • Workspace starts are now delayed waiting for that pull (on updates); this takes minutes (multi-GB pull down).
  • Example: small change to vscode/desktop service script after previous cached build
    • 8 minutes to build (mostly image layer COPY nonsense)
    • 9 minutes to push a single 5GB layer
    • 14 minutes to pull that layer (triggered by starting a challenge)
    • Total of 30 minutes to make a small challenge image change
  • Why do we have these super long COPYs? Is it the COPY / /; why is this not fast?

Networking

• Make it so nginx can talk to the workspace pods.
  • Right now, manually run ip route add 10.42.0.0/16 via $(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}} dev br-{{slice .NetworkID 0 12}}{{end}}' kube-server) on the infrastructure container.
• network isolate workspace pods from each other
• network isolate workspace pods from the internet
• INTERNET award policy

Security

• Seccomp filter

Services

• Changed workspace services to bind on 0.0.0.0, since /etc/hosts is hard to update with dynamic ip
  • Could instead use $(hostname -I | awk '{print $1}') (just $(hostname -I) in single-interface setup) to get the ip of the container, and bind to that
  • Theoretical concern is that service ports can no longer be used by hypothetical challenges on localhost

SSH

• Support it

NFS

• NFS feels sketchy, run ps aux | grep ' D' to see if there are any (NFS) processes in D state.
• What happens under more agressive shutdown conditions (e.g. docker kill dojo)?

ZFS

• How to get kube to support ZFS?
• OverlayFS is supported in ZFS 2.2
  • Ubuntu 24.04 ships with >= 2.2 (https://launchpad.net/ubuntu/noble/+source/zfs-linux)
  • Unclear if storage-opt size is possible with OverlayFS over ZFS
• Other nodes don't have additional SSDs with ZFS setup; do we care?
• Possibly we don't need to put root storage of pods on ZFS
  • Kube might support limits on simple overlayfs storage via monitoring and eviction
  • https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-emphemeralstorage-consumption

Flag

• Insert flag
  • Could apply: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/

Misc

• Workspace pod start is now async from the request context
  • Pods seem to start SUPER fast (except when a pull is required)
  • Need to make sure the user knows when it's actually ready
• There's a whole bunch of KUBE/NFS environment variables hanging out; should we get rid of them somehow?
• How do people use kube namespaces? Should we put all our workspace pods in a namespace namde "workspaces"?

[ConnorNelson]

Support for SSH was added, but the user enters as root. This is a minor limitation of kubectl exec that this cannot be modified.

The solution is going to live in /opt/pwn.college/ssh-entrypoint. I need to rebase to get this entrypoint. Rather than just being a symlink, it will be a small wrapper script that will:

• Wipe out DOJO_FLAG environment variable
• Wipe out KUBERNETES_* environment varaibles
• Become hacker user

[ConnorNelson]

As a part of this merge commit, I decided to bring down my test container:

$ docker kill dojo-dev
Error response from daemon: Cannot kill container: dojo-dev: tried to kill container, but did not receive an exit event

$ docker exec -it 57dd7f951256 bash
OCI runtime exec failed: exec failed: unable to start container process: error executing setns process: exit status 1: unknow

$ docker ps
CONTAINER ID   IMAGE      COMMAND        CREATED       STATUS       PORTS                     NAMES
57dd7f951256   dojo-dev   "dojo start"   2 weeks ago   Up 2 weeks   22/tcp, 80/tcp, 443/tcp   dojo-dev

$ ps aux | grep ' D'
_apt     1391366  0.0  0.0      0     0 ?        D    Apr01   0:02 [rpcbind]
root     1391940  0.0  0.0      0     0 ?        D    Apr01   0:00 [nfsd]

After some time (<1 minute):

$ docker kill dojo-dev
dojo-dev

$ ps aux | grep ' D'
kanak      99218  0.0  0.0   6480  2256 pts/13   S+   23:54   0:00 grep --color=auto  D

In other words, the deadlock does seem to sort itself out reasonably quickly.