From c64ffea86dae39b27a7fe5101c17042df3cf263f Mon Sep 17 00:00:00 2001 From: Alex Chan Date: Fri, 22 Nov 2024 11:08:50 +0000 Subject: [PATCH 1/3] Add a GitHub Action to check my Caddyfiles are formatted --- .github/workflows/caddy_fmt.yml | 53 +++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 .github/workflows/caddy_fmt.yml diff --git a/.github/workflows/caddy_fmt.yml b/.github/workflows/caddy_fmt.yml new file mode 100644 index 000000000..12cd3e38a --- /dev/null +++ b/.github/workflows/caddy_fmt.yml @@ -0,0 +1,53 @@ +name: "Lint my Caddyfiles" + +on: + push: + branches: + - main + + paths: + - '.github/workflows/caddy_fmt.yml' + - 'Caddyfile' + - '**/Caddyfile' + - '**/*.Caddyfile' + + pull_request: + branches: + - main + + paths: + - '.github/workflows/caddy_fmt.yml' + - 'Caddyfile' + - '**/Caddyfile' + - '**/*.Caddyfile' +jobs: + lint: + runs-on: ubuntu-latest + steps: + - name: "Check out the repo" + uses: actions/checkout@v4 + + - name: Install Caddy + env: + GH_TOKEN: ${{ github.token }} + run: | + gh release download \ + --repo caddyserver/caddy \ + --pattern 'caddy_*_linux_amd64.tar.gz' \ + --output caddy.tar.gz + tar -xzf caddy.tar.gz --directory /usr/local/bin + chmod +x /usr/local/bin/caddy + which caddy + + - name: Format my Caddyfiles + run: | + find . \ + -name Caddyfile \ + -o -name '*.Caddyfile' ! -name 'redirects.Caddyfile' | \ + xargs -I '{}' --verbose caddy fmt --overwrite '{}' + + - name: Print changes + run: git diff + + - name: Check for changes + run: git diff --exit-code From 432a5f32d738568e726837ec535bdf17d93d8aba Mon Sep 17 00:00:00 2001 From: Alex Chan Date: Fri, 22 Nov 2024 16:55:10 +0000 Subject: [PATCH 2/3] Add a TIL about debugging `find` and `xargs` --- .../2024-11-22-debugging-find-and-xargs.md | 71 +++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 src/_til/2024/2024-11-22-debugging-find-and-xargs.md diff --git a/src/_til/2024/2024-11-22-debugging-find-and-xargs.md b/src/_til/2024/2024-11-22-debugging-find-and-xargs.md new file mode 100644 index 000000000..f57cb02e1 --- /dev/null +++ b/src/_til/2024/2024-11-22-debugging-find-and-xargs.md @@ -0,0 +1,71 @@ +--- +layout: til +title: Debugging some confusing behaviour with `find` and `xargs` +date: 2024-11-22 16:35:11 +0000 +tags: + - shell scripting +summary: + Use the `--verbose` flag to see what `xargs` is doing; don't rely on `find` to return files in a consistent order. +--- +I was trying to write a GitHub Actions workflow to format my Caddyfiles. +I'd written a command to find all the Caddyfiles in my repo, and pass them to `caddy fmt`: + +``` +find . -name Caddyfile | xargs caddy fmt --overwrite +``` + +Thsi command seemed to work when I ran it locally on my Mac, but it didn't do what I expected when it ran on Ubuntu in GitHub Actions. +It took me a while to work out what was going on! + +I learnt two useful debugging techniques along the way: + +1. **Use the `--verbose` flag to see what `xargs` is doing.** + This will print the exact command which is run by `xargs`. + + ```console + $ ls *.jpg | xargs echo + 26704189108_9e71d865d8_b.jpg 51119071159_905212c30f_b.jpg 8076936667_c5605863a9_o.jpg + + $ ls *.jpg | xargs --verbose echo + echo 26704189108_9e71d865d8_b.jpg 51119071159_905212c30f_b.jpg 8076936667_c5605863a9_o.jpg + 26704189108_9e71d865d8_b.jpg 51119071159_905212c30f_b.jpg 8076936667_c5605863a9_o.jpg + + $ ls *.jpg | xargs --verbose -I '{}' echo '{}' + echo 26704189108_9e71d865d8_b.jpg + 26704189108_9e71d865d8_b.jpg + echo 51119071159_905212c30f_b.jpg + 51119071159_905212c30f_b.jpg + echo 8076936667_c5605863a9_o.jpg + 8076936667_c5605863a9_o.jpg + ``` + + In particular, I could see whether the command I was running with `xargs` was being called with all the arguments at once, or passed them individually. + +2. **Files discovered by `find` may be in a different order on different OSes.** + + This was the heart of my issue -- I was passing all my filenames in one big list, and `caddy fmt` will [only format the first path it gets](https://github.com/caddyserver/caddy/issues/6702). + Because the order of files is different on different OSes, I was getting different behaviour. + + Compare: + + ```console + $ # macOS + $ find . -name Caddyfile -o -name '*.Caddyfile' + ./Caddyfile + ./caddy/gone.Caddyfile + ./caddy/redirects.Caddyfile + + $ # Ubuntu + $ find . -name Caddyfile -o -name '*.Caddyfile' + ./caddy/gone.Caddyfile + ./caddy/redirects.Caddyfile + ./Caddyfile + ``` + + In hindsight, this makes sense -- there are multiple ways you might search a directory tree, and one isn't necessarily better than the other. + + Don't rely on the order to be consistent! + + I've had a brief glance at the man page, and `find` does have some options for controlling the ordering -- for example, you can use `-d`/`--depth` to do a depth-first search (process each directory's contents before the directory itself). + + This reminds me of when I discovered `WalkDir::contents_first()` and `os.walk(topdown: bool)` while [writing `emptydir`](/2024/emptydir/#how-does-it-work). From 40c35b15f3fec236758a2a2d565595822886d1d7 Mon Sep 17 00:00:00 2001 From: Alex Chan Date: Fri, 22 Nov 2024 16:55:24 +0000 Subject: [PATCH 3/3] Add a TIL about downloading assets from a GitHub release --- ...2024-11-22-install-from-github-releases.md | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 src/_til/2024/2024-11-22-install-from-github-releases.md diff --git a/src/_til/2024/2024-11-22-install-from-github-releases.md b/src/_til/2024/2024-11-22-install-from-github-releases.md new file mode 100644 index 000000000..eb676c4ca --- /dev/null +++ b/src/_til/2024/2024-11-22-install-from-github-releases.md @@ -0,0 +1,47 @@ +--- +layout: til +title: How to install an asset from a GitHub release +date: 2024-11-22 11:35:50 +0000 +tags: + - github +summary: + Use `gh release download`, which includes a pattern matcher if you want to pick a specific asset. +--- +I wanted to install Caddy in GitHub Actions, and one of the ways [to install it](https://caddyserver.com/docs/install#static-binaries) is by downloading static binaries from their GitHub releases. + +I wasn't sure how to download release assets in a sensible way -- I've written shell scripts for it before, but it feels messy and fragile. +Surely there must be a better way! + +I stumbled upon [a GitHub Action](https://github.com/php/php-src/blob/master/.github/actions/setup-caddy/action.yml) in the PHP repository (!) which pointed me in the right direction: + +```shell +gh release -R caddyserver/caddy download --pattern 'caddy_*_linux_amd64.tar.gz' -O - | sudo tar -xz -C /usr/bin caddy +sudo chmod +x /usr/bin/caddy +``` + +This is using [the GitHub CLI](https://cli.github.com/), which I haven't used before -- but it's pre-installed in GitHub Actions, which is very convenient for my purposes! + +## In a GitHub Action + +Here's how I packaged this as a step in a GitHub Action: + +```yml +- name: Install Caddy + env: + GH_TOKEN: ${{ github.token }} + run: | + gh release download \ + --repo caddyserver/caddy \ + --pattern 'caddy_*_linux_amd64.tar.gz' \ + --output caddy.tar.gz + tar -xzf caddy.tar.gz --directory /usr/local/bin + chmod +x /usr/local/bin/caddy + which caddy +``` + +It's a bit more verbose and leaves a `caddy.tar.gz` file lying around, but I find this version easier to understand and debug. + +## Notes + +* You can specific a release to `gh release download`, or if not it will pick the latest release. + If you let it pick automatically, it skips pre-releases.