Support `aqua update` command updating packages and registries #1657

suzuki-shunsuke · 2023-02-22T04:38:14Z

Feature Overview

Add a command to update packages and registries.

Why is the feature needed?

Please explain the problem you want to solve.

Upgrade an aqua config without renovate. I'm a huge fan of renovate but it's a huge ask for folks to all use it especially if they are just setting up a global tools config.

Does the feature include Breaking Changes?

No.

Example Code

command and configuration

Prepare aqua.yaml which contains old registries and packages

registries:
- type: standard
  ref: v3.90.0 # Old
packages:
- name: cli/[email protected] # Old
- name: junegunn/[email protected] # Old

Run aqua update.

$ aqua update

Then all registries and packages are updated.

registries:
- type: standard
  ref: v3.166.1 # Updated
packages:
- name: cli/[email protected] # Updated
- name: junegunn/[email protected] # Updated

Support -tag. https://aquaproj.github.io/docs/tutorial-extras/package-tag/

aqua update -t dev

Ideas

Support version constraint like ^1.0.0
Add a field to exclude update

packages:
- name: cli/[email protected]
  exclude_update: true

I prefer 3 than 2.

Exclude packages using long syntax

packages:
- name: cli/[email protected] # Update
- name: cli/cli
  version: v2.0.0 # Don't update

Conform to policy

This is nice to have but isn't mandatory.
I guess it is a bit difficult to implement this feature.

Reference

The text was updated successfully, but these errors were encountered:

CrystalMethod · 2023-02-22T22:17:37Z

This exact question came up today in my project team. A must have is that update respects pinned versions and leaves them unchanged. And maybe a --dry-run option.

suzuki-shunsuke · 2023-02-23T00:48:45Z

A must have is that update respects pinned versions and leaves them unchanged.

What do you mean?
aqua forces to pin all registries and packages versions, so I can't understand what you want.
Do you expect version constraint like ^1.0.0?

suzuki-shunsuke · 2023-02-23T01:09:35Z

I like and strongly recommend Renovate, so I'd like to know why users who want this feature don't use Renovate.

especially if they are just setting up a global tools config

Some users manage aqua.yaml in their dotfiles repositories, which means you can update global configuration by Renovate as well.

CrystalMethod · 2023-03-02T05:42:59Z

I like and strongly recommend Renovate, so I'd like to know why users who want this feature don't use Renovate.

In my case: because the company policies do not allow it. It is not allowed - or requires a separate and lengthy review/audit - to use such managed services. We are experimenting with running Renovate selfhosted, but it is not the same.

suzuki-shunsuke · 2023-03-02T08:11:57Z

Thank you for your explanation.

CrystalMethod · 2023-03-02T09:00:08Z

Currently I try to use updatecli for this - maybe this is a good companion, because it supports other systems as well. This could be a suitable replacement for Renovate; at least in my case.

suzuki-shunsuke · 2023-03-02T09:16:02Z

I don't know updatecli well, but I wonder you can update aqua.yaml by updatecli.
If so, it can be used instead of aqua update command.

And I think Self hosted Renovate by GitHub Actions is also useful if it is allowed in your company.
This is just a GitHub Actions, so the hosting isn't difficult.

CrystalMethod · 2023-03-02T10:05:20Z

I don't know updatecli well, but I wonder you can update aqua.yaml by updatecli. If so, it can be used instead of aqua update command.

I am working on a PoC for this and will share my findings.

And I think Self hosted Renovate by GitHub Actions is also useful if it is allowed in your company. This is just a GitHub Actions, so the hosting isn't difficult.

This is a completely different story: we are using the self-hosted Bitbucket (formerly known as Stash) and finishing the migration to GitHub Enterpise will take several more months.

suzuki-shunsuke · 2023-03-02T10:10:11Z

I am working on a PoC for this and will share my findings.

Thanks. I'm looking forward to it. I'll also take a look updatecli when I have a time.

This is a completely different story: we are using the self-hosted Bitbucket (formerly known as Stash) and finishing the migration to GitHub Enterpise will take several more months.

Oh, I see.

CrystalMethod · 2023-03-02T11:36:22Z

aqua.yaml

---
# aqua - Declarative CLI Version Manager
# https://aquaproj.github.io/
registries:
  - type: standard
    ref: v3.138.0
packages:
- name: argoproj/[email protected]
- name: dagger/[email protected] # latest is v0.3.13
- name: kubernetes/kubectl
  version: v1.24.10

updatecli.dagger.yaml

name: Update Dagger GitHub Release

sources:
  dagger:
    name: Get latest Dagger version from GitHub releases
    kind: githubrelease
    spec:
      owner: dagger
      repository: dagger
      token: '{{ requiredEnv "GITHUB_TOKEN" }}'
      username: '{{ requiredEnv "GITHUB_ACTOR" }}'
      versionfilter:
        kind: regex
        pattern: ^v(\d+)\.(\d+)\.(\d+)$

conditions:
  dagger:
    name: Check latest Dagger release
    kind: githubrelease
    sourceid: dagger
    spec:
      owner: dagger
      repository: dagger
      token: '{{ requiredEnv "GITHUB_TOKEN" }}'
      username: '{{ requiredEnv "GITHUB_ACTOR" }}'
      typefilter:
        release: true

targets:
  dagger:
    name: Update Dagger version in aqua.yaml
    kind: yaml
    transformers:
      - addprefix: "dagger/dagger@"
    spec:
      file: "aqua.yaml"
      key: "packages[1].name" # array access is limited to indices

The targets.<yaml>.spec.key syntax is currently limited to named keys and array indices. In addition, the challenge is not to modify pinned versions. (But this would probably have to be ensured by an additional condition)

updatecli diff --config updatecli.dagger.yaml

+++++++++++
+ PREPARE +
+++++++++++

Loading Pipeline "updatecli.dagger.yaml"

SCM repository retrieved: 0


++++++++++++++++++
+ AUTO DISCOVERY +
++++++++++++++++++



++++++++++++
+ PIPELINE +
++++++++++++



#############
# EXAMPLE 1 #
#############


SOURCES
=======

dagger
------
Searching for version matching pattern "^v(\\d+)\\.(\\d+)\\.(\\d+)$"
✔ Github Release version "v0.3.13" found matching pattern "^v(\\d+)\\.(\\d+)\\.(\\d+)$"


CHANGELOG:
----------

Release published on the 2023-02-24 18:06:22 +0000 UTC at the url https://github.com/dagger/dagger/releases/tag/v0.3.13

## Changelog

* :sparkles: Services, a.k.a. container-to-container networking by @vito in https://github.com/dagger/dagger/pull/4505
* :lock: Scrub secrets by @dolanor in https://github.com/dagger/dagger/pull/4518
* Initial support for remote cache config. by @sipsma in https://github.com/dagger/dagger/pull/4543
* Download runc binary instead of building it.  by @sipsma in https://github.com/dagger/dagger/pull/4559
* fix: remove workdir & configPath default values. by @TomChv in https://github.com/dagger/dagger/pull/4465
* Remove import dependency of CLI on containerd packages.  by @sipsma in https://github.com/dagger/dagger/pull/4562
* Use comma instead of semicolon for cache config separator. by @sipsma in https://github.com/dagger/dagger/pull/4563
* Disable logrus output only in CLI main. by @sipsma in https://github.com/dagger/dagger/pull/4572
* fix Directory.WithTimestamps with subdir by @vito in https://github.com/dagger/dagger/pull/4606
* docs: improve GraphQL with example and better format by @TomChv in https://github.com/dagger/dagger/pull/4367
* Get sha of the image after Container.From by @LokeshN in https://github.com/dagger/dagger/pull/4600
* Update buildkit to commit w/ cache exporter fix + nil panic fix by @sipsma in https://github.com/dagger/dagger/pull/4630
* Additive caching support for S3 backend by @sipsma in https://github.com/dagger/dagger/pull/4577

**Full Changelog**: https://github.com/dagger/dagger/compare/v0.3.12...v0.3.13

## What to do next?
- Read the [documentation](https://docs.dagger.io)
- Join our [Discord server](https://discord.gg/dagger-io)
- Follow us on [Twitter](https://twitter.com/dagger_io)



CONDITIONS:
===========

dagger
------
✔ Github Release version "v0.3.13" found


TARGETS
========

dagger
------

**Dry Run enabled**

⚠ Key 'packages[1].name', from file 'aqua.yaml', was updated from 'dagger/[email protected]' to 'dagger/[email protected]'

=============================

REPORTS:


⚠ Example 1:
        Source:
                ✔ [dagger] Get latest Dagger version from GitHub releases (kind: githubrelease)
        Condition:
                ✔ [dagger] Check latest Dagger release (kind: githubrelease)
        Target:
                ⚠ [dagger] Update Dagger version in aqua.yaml (kind: yaml)


Run Summary
===========
Pipeline(s) run:
  * Changed:    1
  * Failed:     0
  * Skipped:    0
  * Succeeded:  0
  * Total:      1

CrystalMethod · 2023-03-02T11:47:47Z

aqua forces to pin all registries and packages versions, so I can't understand what you want.
Do you expect version constraint like ^1.0.0?

Originally I only referred to "3. Exclude packages using long syntax".

But in the meantime I also believe that version ranges - especially based on semver - can be an interesting pattern.

suzuki-shunsuke · 2023-03-02T11:52:00Z

Oh, thank you for sharing! It looks interesting.
Should we prepare configuration per tool?
If so, it is hard to maintain when the number of tools increases.

And the restriction The targets.<yaml>.spec.key syntax is currently limited to named keys and array indices. is also cumbersome.

suzuki-shunsuke · 2023-03-02T11:55:43Z

aqua forces to pin all registries and packages versions, so I can't understand what you want.
Do you expect version constraint like ^1.0.0?

Originally I only referred to "3. Exclude packages using long syntax".

But in the meantime I also believe that version ranges - especially based on semver - can be an interesting pattern.

I see. Thanks. I think version range is interesting too, but it can make aqua complicated and hard to maintain.

CrystalMethod · 2023-03-02T12:24:59Z

I see. Thanks. I think version range is interesting too, but it can make aqua complicated and hard to maintain.

This is the reason why I am evaluating Updatecli. I think that tools like Renovate or Updatecli are better suited for this kind of tasks.

CrystalMethod · 2023-03-02T12:41:49Z

Should we prepare configuration per tool? If so, it is hard to maintain when the number of tools increases.

I'm not sure what the best layout would be. It was just a first shot, to be more used as a prototype.

Check out this exhaustive update definition, where they are using one file (pipeline) per tool: https://github.com/jenkins-infra/packer-images/tree/main/updatecli/updatecli.d

And the restriction The targets.<yaml>.spec.key syntax is currently limited to named keys and array indices. is also cumbersome.

Looking at the source code, I come to the conclusion that this is a self-made implementation of Yamlpath.

I just noticed that there is a target type file defining targets.<file>.spec.matchpattern and targets.<file>.spec.replacepattern . That might fit better.

Otherwise you will propaply need a special type "aqua" whose implementation encapsulates some of the necessary but verbose yaml configuration.

CrystalMethod · 2023-03-02T14:07:56Z

I think one configuration (pipeline) per tool has the advantage to have individual commits/pr per tool to be updated. If you dont need or want this then you are fine to add all pairs of source-target (one per tool) to the same configuration.

Update: Kind file works much smoother. Also the "do not touch pinned versions" issue is solved, because you only have to define a proper regexp. It's still a bunch of Yaml source code. But in the first step I'm somewhat happy with this solution. I'll push my configuration as a separate "showcase" repository to GitHub.

CrystalMethod · 2023-03-03T11:57:14Z

https://github.com/CrystalMethod/aqua-meets-updatecli

TODO:

document the motivation
document more findings
~~add GHA Workflow~~

olblak · 2023-03-10T17:58:27Z

sorry to jump in your discussion, accidentally found it and it was too much interesting to ignore it.
Getting feedback on opensource project can be challenging and yet super valuable.

Looking at the source code, I come to the conclusion that this is a self-made implementation of Yamlpath.

You are right, at the beginning of the project I didn't find a library that suited my need and went with the custom one. The situation changed and I am evaluating different library. updatecli/updatecli#903 (comment)

when those situation happen do not hesitate to add a +1 on an issue, it helps to prioritize.

suzuki-shunsuke · 2023-03-11T01:38:16Z

We also use goccy/go-yaml to patch YAML.
https://github.com/aquaproj/aqua/blob/3f31cf223d8968ee662fe2a06bc5a577958b7b10/pkg/controller/generate/output/insert.go

goccy/go-yaml is awesome, but let me share some known issues.

Can't update YAML keeping newlines goccy/go-yaml#285 : goccy/go-yaml removes newlines
- aqua g -i removes new lines #584
Exec performance regression #582 : When we parsed a large YAML, the performance was degraded. Maybe this issue has already been solved, but I'm not sure. If YAML is not so large, this issue wouldn't occur

suzuki-shunsuke · 2023-05-09T22:34:23Z

Update only the specific package

Change the version of cli/cli to v2.2.0. If cli/cli isn't found in aqua.yaml, the command fails.

$ aqua update cli/[email protected]

Select the version of cli/cli interactively and change the version. If cli/cli isn't found in aqua.yaml, the command fails.

$ aqua update -s cli/cli

Update cli/cli to the latest.

$ aqua update cli/cli

suzuki-shunsuke · 2023-05-09T22:38:25Z

Idea: `-major` option: Major update

By default, aqua update avoids major updates to keep the compatibility.
aqua update -major allows major updates.

tmeijn · 2023-07-26T10:51:55Z

Hey @suzuki-shunsuke I was wondering if you might have plans to add this command to the CLI.

suzuki-shunsuke · 2023-07-26T12:25:42Z

Yeah, we have a plan to work on this, but we're working on other tasks so we can't work on this for a while.
You need this feature because you can't use Renovate in your projects, right?

suzuki-shunsuke · 2023-10-09T05:43:35Z

We're working on it.
We'd like to handle this issue by this year.

suzuki-shunsuke · 2023-10-15T02:22:38Z

I'm working on it at #2329 .
I published a prerelease version v2.14.0-1.

For the detail, please see the description of the pull request.

You can try the prerelease version by downloading the prebuilt binary from GitHub Release or running aqua update-aqua.

aqua update-aqua v2.14.0-1

Note that this is still work in progress so there may be bugs and the specification may be changed.
I'm adding tests.
Your feedback is welcome.

suzuki-shunsuke · 2023-10-16T11:48:43Z

aqua v2.14.0 is out 🎉
https://github.com/aquaproj/aqua/releases/tag/v2.14.0

sheldonhull · 2023-10-16T18:25:49Z

You are amazing as usual. Thank you so much for this. A great quality of life improvement even if renovate is possible. Great for adhoc updates. Cheers!

suzuki-shunsuke added the enhancement New feature or request label Feb 22, 2023

suzuki-shunsuke added this to main Feb 22, 2023

suzuki-shunsuke moved this to Todo in main Feb 22, 2023

suzuki-shunsuke mentioned this issue Feb 24, 2023

feat: add update command #577

Closed

suzuki-shunsuke moved this from Todo to Backlog in main Mar 1, 2023

suzuki-shunsuke mentioned this issue Apr 24, 2023

Reinserting an existing package in the aqua.yaml should update not duplicate itself #1903

Open

suzuki-shunsuke moved this from Backlog to Todo in main May 8, 2023

suzuki-shunsuke moved this from Todo to Backlog in main Jul 26, 2023

suzuki-shunsuke moved this from Backlog to Todo in main Jul 26, 2023

suzuki-shunsuke pinned this issue Aug 29, 2023

This was referenced Oct 9, 2023

feat: add update command #2329

Merged

add the document about update command aquaproj/aquaproj.github.io#600

Merged

suzuki-shunsuke added the command:update label Oct 16, 2023

suzuki-shunsuke moved this from Todo to In Progress in main Oct 16, 2023

suzuki-shunsuke closed this as completed Oct 16, 2023

github-project-automation bot moved this from In Progress to Done in main Oct 16, 2023

suzuki-shunsuke added this to the v2.14.0 milestone Oct 16, 2023

suzuki-shunsuke unpinned this issue Oct 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support `aqua update` command updating packages and registries #1657

Support `aqua update` command updating packages and registries #1657

suzuki-shunsuke commented Feb 22, 2023 •

edited

Loading

CrystalMethod commented Feb 22, 2023 •

edited

Loading

suzuki-shunsuke commented Feb 23, 2023

suzuki-shunsuke commented Feb 23, 2023 •

edited

Loading

CrystalMethod commented Mar 2, 2023

suzuki-shunsuke commented Mar 2, 2023

CrystalMethod commented Mar 2, 2023

suzuki-shunsuke commented Mar 2, 2023

CrystalMethod commented Mar 2, 2023

suzuki-shunsuke commented Mar 2, 2023

CrystalMethod commented Mar 2, 2023

CrystalMethod commented Mar 2, 2023

suzuki-shunsuke commented Mar 2, 2023

suzuki-shunsuke commented Mar 2, 2023

CrystalMethod commented Mar 2, 2023

CrystalMethod commented Mar 2, 2023 •

edited

Loading

CrystalMethod commented Mar 2, 2023 •

edited

Loading

CrystalMethod commented Mar 3, 2023 •

edited

Loading

olblak commented Mar 10, 2023 •

edited

Loading

suzuki-shunsuke commented Mar 11, 2023 •

edited

Loading

suzuki-shunsuke commented May 9, 2023

suzuki-shunsuke commented May 9, 2023

tmeijn commented Jul 26, 2023

suzuki-shunsuke commented Jul 26, 2023 •

edited

Loading

suzuki-shunsuke commented Oct 9, 2023

suzuki-shunsuke commented Oct 15, 2023

suzuki-shunsuke commented Oct 16, 2023

sheldonhull commented Oct 16, 2023

Support aqua update command updating packages and registries #1657

Support aqua update command updating packages and registries #1657

Comments

suzuki-shunsuke commented Feb 22, 2023 • edited Loading

Feature Overview

Why is the feature needed?

Does the feature include Breaking Changes?

Example Code

Ideas

Reference

CrystalMethod commented Feb 22, 2023 • edited Loading

suzuki-shunsuke commented Feb 23, 2023

suzuki-shunsuke commented Feb 23, 2023 • edited Loading

CrystalMethod commented Mar 2, 2023

suzuki-shunsuke commented Mar 2, 2023

CrystalMethod commented Mar 2, 2023

suzuki-shunsuke commented Mar 2, 2023

CrystalMethod commented Mar 2, 2023

suzuki-shunsuke commented Mar 2, 2023

CrystalMethod commented Mar 2, 2023

aqua.yaml

updatecli.dagger.yaml

updatecli diff --config updatecli.dagger.yaml

CrystalMethod commented Mar 2, 2023

suzuki-shunsuke commented Mar 2, 2023

suzuki-shunsuke commented Mar 2, 2023

CrystalMethod commented Mar 2, 2023

CrystalMethod commented Mar 2, 2023 • edited Loading

CrystalMethod commented Mar 2, 2023 • edited Loading

CrystalMethod commented Mar 3, 2023 • edited Loading

olblak commented Mar 10, 2023 • edited Loading

suzuki-shunsuke commented Mar 11, 2023 • edited Loading

suzuki-shunsuke commented May 9, 2023

Update only the specific package

suzuki-shunsuke commented May 9, 2023

Idea: -major option: Major update

tmeijn commented Jul 26, 2023

suzuki-shunsuke commented Jul 26, 2023 • edited Loading

suzuki-shunsuke commented Oct 9, 2023

suzuki-shunsuke commented Oct 15, 2023

suzuki-shunsuke commented Oct 16, 2023

sheldonhull commented Oct 16, 2023

Support `aqua update` command updating packages and registries #1657

Support `aqua update` command updating packages and registries #1657

suzuki-shunsuke commented Feb 22, 2023 •

edited

Loading

CrystalMethod commented Feb 22, 2023 •

edited

Loading

suzuki-shunsuke commented Feb 23, 2023 •

edited

Loading

CrystalMethod commented Mar 2, 2023 •

edited

Loading

CrystalMethod commented Mar 2, 2023 •

edited

Loading

CrystalMethod commented Mar 3, 2023 •

edited

Loading

olblak commented Mar 10, 2023 •

edited

Loading

suzuki-shunsuke commented Mar 11, 2023 •

edited

Loading

Idea: `-major` option: Major update

suzuki-shunsuke commented Jul 26, 2023 •

edited

Loading