Spike: Dockerfiles POC

[natalieparellano]

Description

To help verify the feasibility of buildpacks/rfcs#173 and work out some of the thorny details that would go in a spec PR, we should spike out a POC lifecycle that supports Dockerfiles. This would eventually also allow potential end users to tinker with the feature and provide their feedback.

Proposed solution

Let's create a dockerfiles-poc branch on the lifecycle with the following functionality. See linked issues for individual units of work that should be more-or-less parallelizable.

Note: "gross" code is okay. Out of scope:

• Windows
• creator support

[natalieparellano]

dockerfiles-poc branch is created. Let's make PRs into that branch (vs the main branch).

[cmoulliard]

Questions:

• Do we have a machine, environment where we could test ?
• Where the test cases should be created (folder, ...) ?
• Is there a template defined to capture the test scenario and validate the POC ?
  @natalieparellano

[natalieparellano]

@cmoulliard since support for this feature in pack will need to come after the lifecycle, we could use Tekton to test it. I'm envisioning something similar to #681

[cmoulliard]

we could use Tekton to test it

This is an option but we will perhaps faced to an issue as openshift will create a pod where the user is not allow to execute root privileged commands. I will check.

[phracek]

I would like to help with coding/writing tests in Go, But I am a newbie in Go. Do you have some parts that can be done from my side?

[natalieparellano]

@phracek we would love to have your help! Do any issues in the epic look interesting to you? I think that #716 and #717 would require the least intricate knowledge of the buildpacks spec, if you would like to play around with https://github.com/GoogleContainerTools/kaniko.

[cmoulliard]

We need also to create a sample project containing the different files such as Dockerfile, Hook TOML, ... to help the guys who will develop or test to know what the foundation should be.

• Could you create a ticket for that purpose ? @natalieparellano
• Who could provide such a dummy sample project (Stephen Levine, ....) ? @sclevine

[natalieparellano]

Please see https://github.com/buildpacks/samples/compare/dockerfiles-poc?expand=1 . It is by no means complete, but I am hoping that we could expand upon it.

Some initial validation steps:

• Create a builder that has:
  • samples/extensions/curl at /cnb/ext/samples_curl/0.0.1
  • samples/extensions/rebasable at /cnb/ext/samples_rebasable/0.0.1
• Note the updated builder.toml (under builders/bionic) that includes curl & rebasable extensions in the same group as samples/java-maven
• pack build test-app -B test-builder ~/workspace/samples/apps/java-maven
• some-arg-build-value should be present in the build logs (this indicates the curl Dockerfile was applied to the build image)
• docker run --rm --entrypoint /bin/bash test-app cat /opt/arg.txt should output some-arg-run-value (this indicates the curl Dockerfile was applied to the run image)
• io.buildpacks.sbom should be populated on the test-app image (this shows /cnb/image/genpkgs was run)
• io.buildpacks.rebasable should be false on the test-app image

• When the order.toml is updated to remove the curl extension,
  • and pack build test-app -B test-builder ~/workspace/samples/apps/java-maven is re-run
  • then io.buildpacks.rebasable should be true on the test-app image

Edit: notable features of the RFC that are not exercised: build plan interactions, "single stage" Dockerfiles e.g., build.Dockerfile or run.Dockerfile

[cmoulliard]

Please see https://github.com/buildpacks/samples/compare/dockerfiles-poc?expand=1 . It is by no means complete, but I am hoping that we could expand upon it.

Can you create a PR to allow us to review/comment it please ? @natalieparellano

[natalieparellano]

Sure, done! I made it a draft as I'm pretty sure it's not mergeable yet :)

[cmoulliard]

To be sure that we are on the page, can you review the following description please ?

The current lifecycle workflow executes the following steps: Detect -> Analyze -> Restore -> Build -> Export
To support the Dockerfiles it will be needed that a new step is included within the workflow in order to augment the existing build and/or run images to package additional tools, libs, executables using root privileged user or to execute root commands.

Such a step should take place after the detect phase to determine based on the [[entries.requires]] if some extensions could be executed to resolve the requirements.

Example A. During the detect phase execution, it appears that the project to be build is using Maven as compilation/packaging tool and Java is the language framework. Then detect will generate within the plan.toml the following entries

  [[entries.requires]]
    name = "Maven"
    [entries.requires.metadata]
      version = "3.8.1"

  [[entries.requires]]
    name = "JDK"
    [entries.requires.metadata]
      version = "11"

REMARK: Ideally the name (OR ID) of the required entry should map the name of the package listed part of the SBoM AND PURL (https://cyclonedx.org/use-cases/).

Next, the new step that we could call extend will check if there is a matching between the id + version of the extensions and the entries = requirements. If this is the case, then the extension(s) will be executed to perform a container build using either the Dockerfile provided within the extension folder or to execute the build/bin able to produce a Dockerfile on the fly

REMARK: If a build/detect is part of the extension, then it could be used to collect the ENV or Parameters to be used to execute the container build command, to check the OS (amd, arm, ...), ....

WDYT: @natalieparellano @jkutner @aemengo @sclevine

[aemengo]

Minor comment: as of PLATFORM: 0.7, Analyze and Detect are switched. It's now: Analyze -> Detect -> Restore -> Build -> Export. Otherwise, that was my understanding.

[cmoulliard]

Otherwise, that was my understanding.

So the new step should be inserted here:

lifecycle/cmd/lifecycle/creator.go

Line 242 in 8c2edb2

? @aemengo

[aemengo]

@cmoulliard Yes, that seems appropriate.

Especially because this is still a POC, it would probably be expedient to make a discrete step there and then we could revise after.

It would be a big help, if you're able to get this moving for the community 🙂

[cmoulliard]

FYI: We need also a new acceptance test case validating some of the scenario we will test using the extend phase

[cmoulliard]

Question: Should we use the docker client part of the lifecycle Dockerfiles change to perform an container build or to execute a command using the container client (docker, podman, ....) @aemengo @sclevine @natalieparellano @jabrown85

[jabrown85]

@cmoulliard the RFC says that is up to the platform - not exactly sure how that is intended to work. I would think that lifecycle ships with a kaniko based implementation.

[cmoulliard]

with a kaniko based implementation.

AFAIK, kaniko is shipped with their own image and cannot be used as go lib within the lifecycle application to execute a build - see: https://github.com/GoogleContainerTools/kaniko#running-kaniko

The lib to be used and which could help us is buildah (already used by podman): https://github.com/containers/podman/blob/main/cmd/podman/images/build.go

WDYT ? @jabrown85 @aemengo @natalieparellano

[cmoulliard]

FYI: I create a simple buildah go app able to build a dockerfile - https://github.com/redhat-buildpacks/poc/blob/main/bud/bud.go#L85

[cmoulliard]

Questions:

• Will the new feature build image from Dockerfiles of lifecycle publish the build or run image(s) published on a registry ? If yes, should we pass as parameters (to the lifecycle) the creds to be authenticated with the registry ?
• How will the building tool such as buildah, ... be able to perform a build/push against a private registry where it is needed to mount a self signed certificate ?
• As it could be difficult to perform a build on non-linux machines (Macos, Windows), will the Dockerfiles feature be documented as non available on non Linux machines and where Root privileged is needed when you use pack client ?

@natalieparellano @jabrown85 @aemengo @sclevine

[jabrown85]

• Will the new feature build image from Dockerfiles of lifecycle publish the build or run image(s) published on a registry ?

I don't think so, no. The idea, as I understand it, is to execute the Dockerfile commands live on the build image that lifecycle is already executing on (inside of builder) right before executing buildpacks.

For the run image, the new layers created during the execution of the Dockerfile would carry over in a volume so the exporter would take run image + extension layers + buildpack layers.

Does that make sense?

[cmoulliard]

I don't think so, no. The idea, as I understand it, is to execute the Dockerfile commands live on the build image that lifecycle is already executing on (inside of builder) right before executing buildpacks.

Make sense and that will simplify our life ;-) if we dont have to push it somewhere

[cmoulliard]

For the run image, the new layers created during the execution of the Dockerfile would carry over in a volume so the exporter would take run image + extension layers + buildpack layers.

Is the exporter able to publish the newly image created then ?

[jabrown85]

Is the exporter able to publish the newly image created then ?

Yes, the exporter gets registry credentials mounted from the platform to push the resulting app image. It creates a new manifest using the manifest of the run image + buildpack layers and exports that manifest and layers to the destination (docker, registry).

[cmoulliard]

is to execute the Dockerfile commands live on the build image

We should also define a convention to name the image newly created as it could be cached to be reused for a next build and by consequence first searched before to execute the command to apply the docker file ? WDYT

[jabrown85]

Caching is not identified in the RFC today. I think caching may start off as a platform-specific feature.

[cmoulliard]

Caching is not identified in the RFC today. I think caching may start off as a platform-specific feature.

Ok but then it will be needed when pack build or kpack build will take place to execute every time the build Dockerfiles ? @jabrown85

[jabrown85]

Ok but then it will be needed when pack build or kpack build will take place to execute every time the build Dockerfiles ? @jabrown85

That is how I understand it. Each build.Dockerfile, may need to re-executed on each build. Any optimizations to that have yet to be identified AFAIK. A platform could execute the build.Dockerfile and then run builder on top of that image and use whatever caching/registry they want, but it isn't defined. A platform could execute the build.Dockerfile instructions in the same container that builder runs on. If a platform used kaniko to apply the instructions, the platform may be able store the snapshots in a platform specific cache volume for subsequent rebuilds.

[cmoulliard]

I was able to create a simple POC which supports the concept to apply a Dockerfile. It uses a go buildah lib which can create an image from a Dockerfiles, next extract the layer(s) content. The code used is equivalent to the buildah bud command.

https://github.com/redhat-buildpacks/poc/blob/main/k8s/manifest.yml#L72-L135

Remark: As it is needed to modify the path to access the layer(s) content extracted within a volume (e.g. /layers --> export PATH="/layers/usr/bin:$PATH", ...), then the RFC perhaps should include an additional ARG or ENV VAR (CNB_EXTENSION_LAYERS) to let to specify where the layers will be extracted in order to change the $PATH during the build step

WDYT: @natalieparellano @sclevine @aemengo @jabrown85

[cmoulliard]

I talked a lot with Giuseppe Scrivano today (podman project) and I think that I found the right tools/projects (skopeo, umoci) to manage the images after the dockerfiles have been applied. As umoci supports to unpack/repack an image, we could imagine to merge the content generated by the execution of the Dockerfiles into a snapshot builder image that next lifecycle will use to perform the build. Same thing could also take place for the runtime image if additional stuffs should be added.

1. Buildah bud

cat <<EOF > Dockerfile
FROM registry.access.redhat.com/ubi8:8.4-211

RUN yum install -y --setopt=tsflags=nodocs nodejs && \
	rpm -V nodejs && \
	yum -y clean all
EOF
REPO="buildpack-poc"
sudo buildah bud -f Dockerfile -t $REPO .

2. Skopeo
   We can extract locally the content of an image

GRAPH_DRIVER="overlay"
TAG=$(sudo buildah --storage-driver $GRAPH_DRIVER images | awk -v r="$REPO" '$0 ~ r {print $2;}')
IMAGE_ID=$(sudo buildah --storage-driver $GRAPH_DRIVER images | awk -v r="$REPO" '$0 ~ r {print $3;}')
sudo skopeo copy containers-storage:$IMAGE_ID oci:$(pwd)/$IMAGE_ID:$TAG

2. Umoci
   We extract the blob and next we can merge them into the image using repack

sudo umoci unpack --image $IMAGE_ID:$TAG bundle

Remark: If we use Tekton, then no need to develop something else as we could apply some pre-steps (= initcontainer) to perform the execution of steps 1-2 and 3. For local development using pack then, that will be a different story !!

End to end script tested

sudo rm -f _temp && mkdir _temp
pushd _temp  

cat <<EOF > Dockerfile
FROM registry.access.redhat.com/ubi8:8.4-211

RUN yum install -y --setopt=tsflags=nodocs nodejs && \
	rpm -V nodejs && \
	yum -y clean all
EOF

REPO="buildpack-poc"
sudo buildah bud -f Dockerfile -t $REPO .

GRAPH_DRIVER="overlay"
TAG=$(sudo buildah --storage-driver $GRAPH_DRIVER images | awk -v r="$REPO" '$0 ~ r {print $2;}')
IMAGE_ID=$(sudo buildah --storage-driver $GRAPH_DRIVER images | awk -v r="$REPO" '$0 ~ r {print $3;}')
sudo skopeo copy containers-storage:$IMAGE_ID oci:$(pwd)/$IMAGE_ID:$TAG

sudo ls -la $IMAGE_ID
sudo ls -la $IMAGE_ID/blobs/sha256/

sudo ../umoci unpack --image $IMAGE_ID:$TAG bundle

sudo ls -la bundle
sudo ls -la bundle/rootfs
total 4
dr-xr-xr-x. 18 root root  242 Sep 14 16:20 .
drwx------.  3 root root  142 Oct 29 14:50 ..
lrwxrwxrwx.  1 root root    7 Apr 23  2020 bin -> usr/bin
dr-xr-xr-x.  2 root root    6 Apr 23  2020 boot
drwxr-xr-x.  2 root root    6 Sep 14 16:19 dev
drwxr-xr-x. 50 root root 4096 Oct 29 14:46 etc
drwxr-xr-x.  2 root root    6 Apr 23  2020 home
lrwxrwxrwx.  1 root root    7 Apr 23  2020 lib -> usr/lib
lrwxrwxrwx.  1 root root    9 Apr 23  2020 lib64 -> usr/lib64
drwx------.  2 root root    6 Sep 14 16:19 lost+found
drwxr-xr-x.  2 root root    6 Apr 23  2020 media
drwxr-xr-x.  2 root root    6 Apr 23  2020 mnt
drwxr-xr-x.  2 root root    6 Apr 23  2020 opt
drwxr-xr-x.  2 root root    6 Sep 14 16:19 proc
dr-xr-x---.  3 root root  213 Sep 14 16:38 root
drwxr-xr-x.  5 root root   66 Oct 29 14:46 run
lrwxrwxrwx.  1 root root    8 Apr 23  2020 sbin -> usr/sbin
drwxr-xr-x.  2 root root    6 Apr 23  2020 srv
drwxr-xr-x.  2 root root    6 Sep 14 16:19 sys
drwxrwxrwt.  2 root root   58 Oct 29 14:46 tmp
drwxr-xr-x. 13 root root  155 Oct 29 14:46 usr
drwxr-xr-x. 19 root root  249 Sep 14 16:20 var



popd

Remark: buildah, skopeo and umoci should be installed to play with the technology before we integrate them within the lifecycle

[jabrown85]

As @sclevine mentioned in slack, I think we should concentrate on one implementation at a time if possible. Stephen listed two distinct implementations that we could concentrate on. I, personally, would like to concentrate on the pure userspace version first.

Here is what I imagined could happen at a high level.

For build extensions, a new phase executor build-extender would exist between detector and builder that would parse the Dockerfile(s) and execute the steps directly - not involving any docker daemon or images. It would assume the platform executing the dockerfiles is ensuring the same builder image is used throughout a build. After each dockerfile is executed, a diff of the changes could be stored into a volume. Then, builder would run as root and restore these changes prior to buildpack execution. It would drop permissions prior to executing buildpacks. If the diffing and storing is too expensive, we could start out by running the dockerfile steps in builder and then dropping permissions before executing the buildpacks. This has fewer moving parts and is likely a good place to start a PoC, but things like caching are more difficult.

For run extensions, a new phase run-extender would exist before exporter and after detector. A container would run against the run image, with the run-extender mounted in (or it could already be there). The run-extender like the build-extender would execute the dockerfile steps. Before each new buildpack, the resulting diff of changes would be stored into a volume. After all extensions have processed, the new phase exits. The exporter now has access to both extension layers and buildpack layers and would be responsible for stitching them up against the run image directly against the registry. I'm not sure I understand how run-extender could happen in a single container flow (e.g. creator), but maybe @sclevine had ideas around that.

[cmoulliard]

Then, builder would run as root and restore these changes prior to buildpack execution. It would drop permissions prior to executing buildpacks.

If I understand correctly what you say here, the idea is to execute the lifecycle builder step as 'root and end of the step, the permissions should be restored to what it is defined according to the images CNB_** parameters - correct ?

If the answer is yes, then I suppose that it will be needed that part of Tekton, kpack builds(= openshift or kubernetes builds executed using a pod), that the privileged option is enabled (https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) - correct ?

@jabrown85

[natalieparellano]

Please see #786 for a lifecycle that goes part of the way toward meeting the acceptance criteria outlined in this comment by:

• Running /bin/detect for extensions as part of regular detect
• Running /bin/build for extensions as a new invocation of build

This could be paired with the "extender" POC that @cmoulliard is working on.

[natalieparellano]

I think this spike has more or less served its purpose. Through #802 we were able to identify many of the changes that would be necessary, which are outlined in buildpacks/spec#298.

buildpacks/spec#308 and buildpacks/spec#307 break this down further into changes that would be required for "phase 1" of the implementation (using Dockerfiles to switch the runtime base image, only). #849 can be used to track the work for this.

With all that said I think we can close this issue, and related spike issues.