How to tell JSON source it's OK for a key not to be found

[jsok]

Hi there,

My build tool emits a JSON file containing the name and image of the release container images, and I'm trying to hook this up to updatecli to then propaget the image to a set of Dockerfiles.

My issue is that the JSON file won't always contain the specific key(s) I'm interested in, e.g. if the image wasn't built as part of that release.

I can't work out how to tell updatecli that it's fine and it should just skip the target.

For context, the JSON file is a skaffold build output file, e.g.:

{
  "builds": [
    {
      "imageName": "example",
      "tag": "example:latest@sha256:f3df2f00b9bbc305120f2153411bbc149ce7d9708f429c64fb072d34f2810142"
    }
  ]
}

My updatecli YAML looks like:

sources:
  example:
    name: Example image
    kind: json
    spec:
      file: artifacts.json
      key: "builds.(imageName=example).tag"
targets:
  dockerfile:
    name: Dockerfile to update
    kind: dockerfile
    sourceid: example
    spec:
      file: Dockerfile
      instruction:
        keyword: "ARG"
        matcher: "EXAMPLE"

When the example key is present it all works as expected, but when it's not it gives the obvious error:

SOURCES
=======

example
-------
ERROR: ✗ ✗ cannot find value for path "builds.(imageName=example).tag" from file "artifacts.json"
ERROR: ✗ ✗ cannot find value for path "builds.(imageName=example).tag" from file "artifacts.json"
Pipeline "EXAMPLE.YAML" failed
Skipping due to:
        sources stage:  "✗ cannot find value for path \"builds.(imageName=example).tag\" from file \"artifacts.json\""

How do I just turn this into a no-op. Having updatecli fail means any other configs or targets won't get updated and makes the CI config complicated.

[olblak]

Mmmmh that's a good question and I don't find a good answer today.

I tried multiple time in the past to allow different exit code per resource and globally for the pipeline.
But, none of those experimentations were successful, as each of them introduced regression...

I keep this issue #233 to remind myself that I would like to go back to that area.
I also have this pull request blocked #1964 because of the same reason, some source fails which abort the pipeline...

Another option would be to improve the Updatecli json plugin to not fail on missing keys, I am not a big fan of solving this problem at the JSON level as I think it's a global issue. Since the json plugin relies on https://github.com/TomWright/dasel maybe there is a way to craft a query that wouldn't fail.
The Updatecli JSON plugin is not jsonquery compliant so maybe doing something similar to the YAML plugin by introducing another engine that would be jsonquery compliant.

I think the short answer would be to move your pipeline to a dedicated manifest so it doesn't affect your other manifest, but I understand that it wouldn't be an ideal solution.

Today Updatecli executes for each pipeline in sequence, all sources, then all conditions, then all targets. Once all pipelines have been processed, it runs all actions.
Something I am more and more considering would be to build a graph of operation, so a source could depend on a condition which is not possible today. Today a resource can only depend on another resource of the same type as either a source, a condition, or a target.