Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiple steps in a job #123

Open
Krakinou opened this issue May 6, 2023 · 7 comments
Open

Multiple steps in a job #123

Krakinou opened this issue May 6, 2023 · 7 comments

Comments

@Krakinou
Copy link

Krakinou commented May 6, 2023 •
edited
Loading

Hello,
I was wondering if it was possible to use the same containers across multiple steps in the same jobs?
Something along the line of:

    - name : Build system (qemu)
      uses: uraimo/[email protected]
      with:
        arch: ${{ matrix.qemu_arch }}
        distro: ubuntu20.04
        install: |
          echo "Install packages"
          apt-get update && apt-get install -y pkg1 pkg2
        run: |
          echo "System successfully built"

    - name : Build project (qemu)
      uses: uraimo/[email protected]
      with:
        arch: ${{ matrix.qemu_arch }}
        distro: ubuntu20.04
        run: |
          ... build project...

    - name: functional-tests (qemu)
      uses: uraimo/[email protected]
      with:
        arch: ${{ matrix.qemu_arch }}
        distro: ubuntu20.04
        run: |
          ...do some tests
      continue-on-error: false
...

The idea is to split the building chain into multiple steps so it's easier to monitor.
It seems that at the moment, each steps is recreating the container at each run...
@felfert
Copy link

felfert commented May 6, 2023 •
edited
Loading

This should be possible right now, provided the steps exist in the same workflow.
If it does not happen and the images get rebuilt on every run, perhaps your github.token does not have the necessary permission to push the image. After running your workflow, simply look at your project's published "packages". There should be at least one containing the string run-on-arch-action like here. If there are none, then your token does not have these permissions. In that case, in the github GUI, go the Project settings, select Actions->General in the left pane, then scroll down to Workflow permissions and select Read and write permissions there, and hit the Save button.

For possible future releases:

Oh, and looking at your example, you also should have the install: parameter present exactly the same in all invocations. After all, the install step gets saved to the image and if the 2nd and 3rd invocation do not have exactly the same install parameters, this would result in different images.

@Krakinou
Copy link
Author

Krakinou commented May 6, 2023 •
edited
Loading

I do have the package in my repo, so this is not the issue

Oh, and looking at your example, you also should have the install: parameter present exactly the same in all invocations. After all, the install step gets saved to the image and if the 2nd and 3rd invocation do not have exactly the same install parameters, this would result in different images.

I will try to add the exact same install parameter and check if it works, thanks for the tip!

@uraimo
Copy link
Owner

uraimo commented May 6, 2023

Hi, thanks @felfert, duplicating the install is not necessary and until the workflow, arch and distro are still the same the next steps should pickup the original image created with the install step, if you set and configured correctly the token.

@Krakinou
Copy link
Author

Krakinou commented May 7, 2023

Hi,
So I ran a bunch of test and I confirm what @felfert said : install parameter have to be the same accross multiple steps so that the docker image is not reinstalled again.
If I skip the install parameter, the image is rebuilt at each step (which obviously does not work).

@felfert
Copy link

felfert commented May 7, 2023 •
edited
Loading

Well, I guess that @uraimo is right. Problem is, that you did just post an example and not the real script that you actually use. I believe you use this one. In there, ALL invocations include the parameter githubToken which enables the docker push in every step. This is wrong. If you use that parameter in the Build system step ONLY, then it should work without the duplicated install as well.

@Krakinou
Copy link
Author

Krakinou commented May 9, 2023

Yep, you're correct this is the action I'm working on. Sorry for not giving the full script (as it was quite long with a lot of stuff, I did not want to put too many unrelated info).
However, I tried removing the github token parameter and it does not seems to work: I have 2 versions of the same script, one with token+install parameter, and the second one without:
The first script is running correctly, with each step re-using the container as installed (.
The second one is failling on the first step using run-on-arch without install parameter as it does not find git (which was installed in the container).

Or am I missing something obvious?

@Krakinou
Copy link
Author

So, I made a lighter script to test. Both github token and install part are required so that the steps reused the cached package:

  • With only the install, the script will rerun the install (as expected)
  • With only the token, it will not reuse the cached packages.

Here is the script:

name: Check run on arch

on:
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-22.04

    steps:

    - name : Build system
      uses: uraimo/[email protected]
      with:
        arch: armv7
        distro: ubuntu22.04
        githubToken: ${{ github.token }}
        install: |
            echo "Install packages"
            apt-get update && apt-get -y install python3
        run: |
          echo "system installed"
      
    - name : Check system 1 - Token & install
      uses: uraimo/[email protected]
      with:
        arch: armv7
        distro: ubuntu22.04
        githubToken: ${{ github.token }}
        install: |
            echo "Install packages"
            apt-get update && apt-get -y install python3
        run: |
          echo "this step will run"
          python3 --version

    - name : Check system 2 - Install only
      uses: uraimo/[email protected]
      with:
        arch: armv7
        distro: ubuntu22.04
        install: |
            echo "Install packages"
            apt-get update && apt-get -y install python3
        run: |
          echo "this step will also run but will reinstall everything"
          python3 --version

    - name : Check system 3 - Token only
      uses: uraimo/[email protected]
      with:
        arch: armv7
        distro: ubuntu22.04
        githubToken: ${{ github.token }}
        run: |
          echo "this step will fail"
          python3 --version
Here are the logs for step 1 
Run uraimo/[email protected]
Configuring Docker for multi-architecture support
/home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/src/run-on-arch.sh /home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/Dockerfiles/Dockerfile.armv7.ubuntu22.04 run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04
Build container
  GitHub token provided, caching to ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04
  WARNING! Your password will be stored unencrypted in /home/runner/.docker/config.json.
  Configure a credential helper to remove this warning. See
  https://docs.docker.com/engine/reference/commandline/login/#credentials-store
  Login Succeeded
  latest: Pulling from krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04
  Digest: sha256:4ef06c3de1eac1fa18976a946d985e793bdd4bc95e7d6332754b0f218b17ac18
  Status: Image is up to date for ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04:latest
  ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04:latest
  Sending build context to Docker daemon  69.63kB
  Step 1/3 : FROM arm32v7/ubuntu:22.04
   ---> eb792700b748
  Step 2/3 : COPY ./run-on-arch-install.sh /root/run-on-arch-install.sh
   ---> Using cache
   ---> 1aad52f28564
  Step 3/3 : RUN chmod +x /root/run-on-arch-install.sh && /root/run-on-arch-install.sh
   ---> Using cache
   ---> c094c8af338e
  Successfully built c094c8af338e
  Successfully tagged run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04:latest
  Using default tag: latest
  The push refers to repository [ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04]
  479eec1abe5f: Preparing
  f71ce63d2bea: Preparing
  8d5d171f01dd: Preparing
  f71ce63d2bea: Layer already exists
  8d5d171f01dd: Layer already exists
  479eec1abe5f: Layer already exists
  latest: digest: sha256:4ef06c3de1eac1fa18976a946d985e793bdd4bc95e7d6332754b0f218b17ac18 size: 948
Run container
  WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested
  this step will run
  Python 3.10.6
Here are the logs for step 2 
Run uraimo/[email protected]
Configuring Docker for multi-architecture support
/home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/src/run-on-arch.sh /home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/Dockerfiles/Dockerfile.armv7.ubuntu22.04 run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04
Build container
  Sending build context to Docker daemon  69.63kB
  Step 1/3 : FROM arm32v7/ubuntu:22.04
   ---> eb792700b748
  Step 2/3 : COPY ./run-on-arch-install.sh /root/run-on-arch-install.sh
   ---> 8d7c50bc4535
  Step 3/3 : RUN chmod +x /root/run-on-arch-install.sh && /root/run-on-arch-install.sh
  Warning: rning] The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested
   ---> Running in 9c1df93cbda8
  Install packages
  Get:1 http://ports.ubuntu.com/ubuntu-ports jammy InRelease [270 kB]
  Get:2 http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease [119 kB]
  Get:3 http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease [108 kB]
  Get:4 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease [110 kB]
  Get:5 http://ports.ubuntu.com/ubuntu-ports jammy/multiverse armhf Packages [199 kB]
  Get:6 http://ports.ubuntu.com/ubuntu-ports jammy/universe armhf Packages [16.8 MB]
  Get:7 http://ports.ubuntu.com/ubuntu-ports jammy/main armhf Packages [1701 kB]
  Get:8 http://ports.ubuntu.com/ubuntu-ports jammy/restricted armhf Packages [13.3 kB]
  Get:9 http://ports.ubuntu.com/ubuntu-ports jammy-updates/multiverse armhf Packages [3270 B]
  Get:10 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf Packages [633 kB]
  Get:11 http://ports.ubuntu.com/ubuntu-ports jammy-updates/restricted armhf Packages [11.2 kB]
  Get:12 http://ports.ubuntu.com/ubuntu-ports jammy-updates/universe armhf Packages [838 kB]
  Get:13 http://ports.ubuntu.com/ubuntu-ports jammy-backports/main armhf Packages [48.9 kB]
  Get:14 http://ports.ubuntu.com/ubuntu-ports jammy-backports/universe armhf Packages [23.6 kB]
  Get:15 http://ports.ubuntu.com/ubuntu-ports jammy-security/restricted armhf Packages [10.3 kB]
  Get:16 http://ports.ubuntu.com/ubuntu-ports jammy-security/universe armhf Packages [612 kB]
  Get:17 http://ports.ubuntu.com/ubuntu-ports jammy-security/main armhf Packages [359 kB]
  Get:18 http://ports.ubuntu.com/ubuntu-ports jammy-security/multiverse armhf Packages [594 B]
  Fetched 21.8 MB in 12s (1768 kB/s)
  Reading package lists...
  Reading package lists...
  Building dependency tree...
  Reading state information...
  The following additional packages will be installed:
    libexpat1 libmpdec3 libpython3-stdlib libpython3.10-minimal
    libpython3.10-stdlib libreadline8 libsqlite3-0 media-types python3-minimal
    python3.10 python3.10-minimal readline-common
  Suggested packages:
    python3-doc python3-tk python3-venv python3.10-venv python3.10-doc binutils
    binfmt-support readline-doc
  The following NEW packages will be installed:
    libexpat1 libmpdec3 libpython3-stdlib libpython3.10-minimal
    libpython3.10-stdlib libreadline8 libsqlite3-0 media-types python3
    python3-minimal python3.10 python3.10-minimal readline-common
  0 upgraded, 13 newly installed, 0 to remove and 4 not upgraded.
  Need to get 5933 kB of archives.
  After this operation, 19.6 MB of additional disk space will be used.
  Get:1 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf libpython3.10-minimal armhf 3.10.6-1~22.04.2ubuntu1 [794 kB]
  Get:2 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf libexpat1 armhf 2.4.7-1ubuntu0.2 [66.9 kB]
  Get:3 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf python3.10-minimal armhf 3.10.6-1~22.04.2ubuntu1 [1941 kB]
  Get:4 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf python3-minimal armhf 3.10.6-1~22.04 [24.3 kB]
  Get:5 http://ports.ubuntu.com/ubuntu-ports jammy/main armhf media-types all 7.0.0 [25.5 kB]
  Get:6 http://ports.ubuntu.com/ubuntu-ports jammy/main armhf libmpdec3 armhf 2.5.1-2build2 [76.5 kB]
  Get:7 http://ports.ubuntu.com/ubuntu-ports jammy/main armhf readline-common all 8.1.2-1 [53.5 kB]
  Get:8 http://ports.ubuntu.com/ubuntu-ports jammy/main armhf libreadline8 armhf 8.1.2-1 [128 kB]
  Get:9 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf libsqlite3-0 armhf 3.37.2-2ubuntu0.1 [548 kB]
  Get:10 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf libpython3.10-stdlib armhf 3.10.6-1~22.04.2ubuntu1 [1750 kB]
  Get:11 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf python3.10 armhf 3.10.6-1~22.04.2ubuntu1 [497 kB]
  Get:12 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf libpython3-stdlib armhf 3.10.6-1~22.04 [6910 B]
  Get:13 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf python3 armhf 3.10.6-1~22.04 [22.8 kB]
  debconf: delaying package configuration, since apt-utils is not installed
  Fetched 5933 kB in 3s (2323 kB/s)
  Selecting previously unselected package libpython3.10-minimal:armhf.
  (Reading database ... 
  (Reading database ... 5%
  (Reading database ... 10%
  (Reading database ... 15%
  (Reading database ... 20%
  (Reading database ... 25%
  (Reading database ... 30%
  (Reading database ... 35%
  (Reading database ... 40%
  (Reading database ... 45%
  (Reading database ... 50%
  (Reading database ... 55%
  (Reading database ... 60%
  (Reading database ... 65%
  (Reading database ... 70%
  (Reading database ... 75%
  (Reading database ... 80%
  (Reading database ... 85%
  (Reading database ... 90%
  (Reading database ... 95%
  (Reading database ... 100%
  (Reading database ... 4390 files and directories currently installed.)
  Preparing to unpack .../libpython3.10-minimal_3.10.6-1~22.04.2ubuntu1_armhf.deb ...
  Unpacking libpython3.10-minimal:armhf (3.10.6-1~22.04.2ubuntu1) ...
  Selecting previously unselected package libexpat1:armhf.
  Preparing to unpack .../libexpat1_2.4.7-1ubuntu0.2_armhf.deb ...
  Unpacking libexpat1:armhf (2.4.7-1ubuntu0.2) ...
  Selecting previously unselected package python3.10-minimal.
  Preparing to unpack .../python3.10-minimal_3.10.6-1~22.04.2ubuntu1_armhf.deb ...
  Unpacking python3.10-minimal (3.10.6-1~22.04.2ubuntu1) ...
  Setting up libpython3.10-minimal:armhf (3.10.6-1~22.04.2ubuntu1) ...
  Setting up libexpat1:armhf (2.4.7-1ubuntu0.2) ...
  Setting up python3.10-minimal (3.10.6-1~22.04.2ubuntu1) ...
  Selecting previously unselected package python3-minimal.
  (Reading database ... 
  (Reading database ... 5%
  (Reading database ... 10%
  (Reading database ... 15%
  (Reading database ... 20%
  (Reading database ... 25%
  (Reading database ... 30%
  (Reading database ... 35%
  (Reading database ... 40%
  (Reading database ... 45%
  (Reading database ... 50%
  (Reading database ... 55%
  (Reading database ... 60%
  (Reading database ... 65%
  (Reading database ... 70%
  (Reading database ... 75%
  (Reading database ... 80%
  (Reading database ... 85%
  (Reading database ... 90%
  (Reading database ... 95%
  (Reading database ... 100%
  (Reading database ... 4692 files and directories currently installed.)
  Preparing to unpack .../0-python3-minimal_3.10.6-1~22.04_armhf.deb ...
  Unpacking python3-minimal (3.10.6-1~22.04) ...
  Selecting previously unselected package media-types.
  Preparing to unpack .../1-media-types_7.0.0_all.deb ...
  Unpacking media-types (7.0.0) ...
  Selecting previously unselected package libmpdec3:armhf.
  Preparing to unpack .../2-libmpdec3_2.5.1-2build2_armhf.deb ...
  Unpacking libmpdec3:armhf (2.5.1-2build2) ...
  Selecting previously unselected package readline-common.
  Preparing to unpack .../3-readline-common_8.1.2-1_all.deb ...
  Unpacking readline-common (8.1.2-1) ...
  Selecting previously unselected package libreadline8:armhf.
  Preparing to unpack .../4-libreadline8_8.1.2-1_armhf.deb ...
  Unpacking libreadline8:armhf (8.1.2-1) ...
  Selecting previously unselected package libsqlite3-0:armhf.
  Preparing to unpack .../5-libsqlite3-0_3.37.2-2ubuntu0.1_armhf.deb ...
  Unpacking libsqlite3-0:armhf (3.37.2-2ubuntu0.1) ...
  Selecting previously unselected package libpython3.10-stdlib:armhf.
  Preparing to unpack .../6-libpython3.10-stdlib_3.10.6-1~22.04.2ubuntu1_armhf.deb ...
  Unpacking libpython3.10-stdlib:armhf (3.10.6-1~22.04.2ubuntu1) ...
  Selecting previously unselected package python3.10.
  Preparing to unpack .../7-python3.10_3.10.6-1~22.04.2ubuntu1_armhf.deb ...
  Unpacking python3.10 (3.10.6-1~22.04.2ubuntu1) ...
  Selecting previously unselected package libpython3-stdlib:armhf.
  Preparing to unpack .../8-libpython3-stdlib_3.10.6-1~22.04_armhf.deb ...
  Unpacking libpython3-stdlib:armhf (3.10.6-1~22.04) ...
  Setting up python3-minimal (3.10.6-1~22.04) ...
  Selecting previously unselected package python3.
  (Reading database ... 
  (Reading database ... 5%
  (Reading database ... 10%
  (Reading database ... 15%
  (Reading database ... 20%
  (Reading database ... 25%
  (Reading database ... 30%
  (Reading database ... 35%
  (Reading database ... 40%
  (Reading database ... 45%
  (Reading database ... 50%
  (Reading database ... 55%
  (Reading database ... 60%
  (Reading database ... 65%
  (Reading database ... 70%
  (Reading database ... 75%
  (Reading database ... 80%
  (Reading database ... 85%
  (Reading database ... 90%
  (Reading database ... 95%
  (Reading database ... 100%
  (Reading database ... 5121 files and directories currently installed.)
  Preparing to unpack .../python3_3.10.6-1~22.04_armhf.deb ...
  Unpacking python3 (3.10.6-1~22.04) ...
  Setting up media-types (7.0.0) ...
  Setting up libsqlite3-0:armhf (3.37.2-2ubuntu0.1) ...
  Setting up libmpdec3:armhf (2.5.1-2build2) ...
  Setting up readline-common (8.1.2-1) ...
  Setting up libreadline8:armhf (8.1.2-1) ...
  Setting up libpython3.10-stdlib:armhf (3.10.6-1~22.04.2ubuntu1) ...
  Setting up libpython3-stdlib:armhf (3.10.6-1~22.04) ...
  Setting up python3.10 (3.10.6-1~22.04.2ubuntu1) ...
  Setting up python3 (3.10.6-1~22.04) ...
  Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
  Removing intermediate container 9c1df93cbda8
   ---> b976105ef36a
  Successfully built b976105ef36a
  Successfully tagged run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04:latest
Run container
  WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested
  this step will also run but will reinstall everything
  Python 3.10.6
Here are the logs for step 3 
Run uraimo/[email protected]
Configuring Docker for multi-architecture support
/home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/src/run-on-arch.sh /home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/Dockerfiles/Dockerfile.armv7.ubuntu22.04 run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04
Build container
  GitHub token provided, caching to ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04
  WARNING! Your password will be stored unencrypted in /home/runner/.docker/config.json.
  Configure a credential helper to remove this warning. See
  https://docs.docker.com/engine/reference/commandline/login/#credentials-store
  
  Login Succeeded
  latest: Pulling from krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04
  Digest: sha256:4ef06c3de1eac1fa18976a946d985e793bdd4bc95e7d6332754b0f218b17ac18
  Status: Image is up to date for ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04:latest
  ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04:latest
  Sending build context to Docker daemon  69.63kB
  
  Step 1/3 : FROM arm32v7/ubuntu:22.04
   ---> eb792700b748
  Step 2/3 : COPY ./run-on-arch-install.sh /root/run-on-arch-install.sh
   ---> 7ff62b887b06
  Step 3/3 : RUN chmod +x /root/run-on-arch-install.sh && /root/run-on-arch-install.sh
  Warning: rning] The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested
   ---> Running in 68cc33632e91
  Removing intermediate container 68cc33632e91
   ---> a4521b503491
  Successfully built a4521b503491
  Successfully tagged run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04:latest
  Using default tag: latest
  The push refers to repository [ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04]
  9d76813aa08e: Preparing
  2bd3b0e1ef93: Preparing
  8d5d171f01dd: Preparing
  8d5d171f01dd: Layer already exists
  2bd3b0e1ef93: Pushed
  9d76813aa08e: Pushed
  latest: digest: sha256:af074a21d4696411f029dcaca470ddcfc9fb124f1dfcdd31f37814b1e303fd0b size: 943
Run container
  WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested
  this step will fail
  /home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/src/run-on-arch-commands.sh: line 4: python3: command not found
  Error: The process '/home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/src/run-on-arch.sh' failed with exit code 127````
</details>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@uraimo @felfert @Krakinou