Add a manifest introspection utility

[dmitry-potepalov]

Add a command-line tool that can be used to list/dump backup manifests in object storage without interacting with the server. Useful when debugging manifest compatibility issues/restoration bugs.

List example:

# astacus manifest -c astacus.conf list
backup-2024-10-15T13:54:19+00:00
backup-2024-10-16T13:54:20+00:00

Describe example:

# astacus manifest -c astacus.conf describe backup-2024-10-16T13:54:20+00:00
clickhouse manifest backup-2024-10-16T13:54:20+00:00
===============================
Started at: 2024-10-16T13:54:20.802652+00:00
Finished at: 2024-10-16T13:54:21.873076+00:00
Attempts: 1
Snapshot #0 created at 2024-10-16T13:54:20.875002+00:00 contains 11 files (25145965 bytes)
Node #0 uploaded 13605207 bytes out of 25145547
References 0 objects in tiered storage totalling 0 bytes

Dump example:

# astacus manifest -c /run/aiven/astacus.conf dump --hashes backup-2024-10-16T13:54:20+00:00
{
  "start": "2024-10-16T13:54:20.802652Z",
  "end": "2024-10-16T13:54:21.873076Z",
  "attempt": 1,
  "snapshot_results": [
    {
      "start": "2024-10-16T13:54:20.875002Z",
      "end": "2024-10-16T13:54:21.008648Z",
      "state": {
        "root_globs": [
          "disks/remote_storage/shadow/astacus/store/**/*",
          "shadow/astacus/store/**/*"
        ],
        "files": "<redacted, has 11 entries>"
      },
      "files": 11,
      "total_size": 25145965,
      "hashes": [
        {
          "hexdigest": "c5414b9c932f41fcfa3ef9051aff2bcde4520f9bcec4f171321cff019491ebf1",
          "size": 356
        },
...

[joelynch]

How is performance in general here?

if indent is 0 and we are not redacting, we can just copy the contents of the file to stdout. Maybe for very large manifests this would be a good performance optimization.

[dmitry-potepalov]

good point, changed to "dump to stdout without parsing" by default, set all the formatting to be an optional extra