"Saving cache config" is always pretty long (with "TAM-verified archive" for every archive)

[szpak]

I have ~10GB local repository with ~1,5K archives (backups) of multiple sets of data (multiple distinct source directories) done over a few years (kept in one repository to allow for deduplication on data hierarchy changes). All the directories are backed up together (one by one) on a regular basis. As number of changes in a particular source directories is usually small, the single "create" command takes <1s to process it, but after that there is ~30s delay on "Saving files cache". As that happens for every directory, the total backup time is significantly increased (in comparison to the actual "backup time" - without the post processing).

Sample command: create --progress --compression auto,zstd --list --filter=AME --stats $BACKUP_PATH::$RESOURCE_NAME-'{hostname}-{now}' $PATH_TO_BACKUP.

When executed with --debug, I clearly see that, during the aforementioned time, Borg (1.2.6 on Linux) always performs verification of the TAM authentication on every historical backup (~1,5K):

Archive authentication DISABLED.
TAM-verified archive
Archive authentication DISABLED.
TAM-verified archive
...

I performed the update procedure for 1.2.6 (although there might be some backups with it created with 1.2.6 due to the Borg update in the operating system) and there were no "tam:none", so according to the procedure, I haven't performed "borg upgrade". I'm not sure, if I did that on 1.0.9.

My related questions:

1. Should I see (as debug) "Archive authentication DISABLED" every time for my repository (somehow related to that thread)? Should that time confusing procedure be performed every time?
2. If yes, is it possible to explicitly disable that behavior on demand? I could just do it for the first directory from the set (and maybe for the last).
3. If yes, maybe there could some extra logging informing about "TAM authentication verification" or "Saving files cache" could be extended to mention also other postprocessing (I was initially thinking why saving cache for non changes takes so long)?

[ThomasWaldmann]

If you only have few changed source files, it is expected that the overhead of loading and saving the files cache, the chunks and repo index is more than the actual backup.

The archive TAM related stuff is relatively inexpensive as that is not much data to process, but of course the time needed for that scales with the archive count (btw, other borg operations, like borg check are impacted way more by that).

You can ignore the TAM related debug log messages, they are normal and only for debugging.

If your files cache is rather big and takes a lot of time to load/save and you are backing up different sets of source files using different borg create commands, you can split the files cache, see the environment variables documentation.

[szpak]

It's definitely better. No extra delay. Everything within 2 seconds. Thanks for your suggestion!

However, I like --stats :-/. It allows me to generate a summary about increments in particular backups (by parsing the partial output logs). Do they need to always go together?

Update. Maybe I can generate stats for previous "create" execution(s)?

[ThomasWaldmann]

The best way to observe increments is to use borg create --stats.

Due to dedup, you can not compute that after the next backup was done.

But guess we can close this. "Everything comes at a price."

[szpak]

But guess we can close this. "Everything comes at a price."

Good to have it explained :). Thanks.

[szpak]

As clarified by @ThomasWaldmann, it is not a problem with cache, but --stats generates extra time to go through all the existing archives in the repository which may take some time, if you have many of them (here 1,5K+).

[jdchristensen]

@ThomasWaldmann Is there any chance that something a bit inefficient is going on here? For me, the part that reads the archives to generate the stats only takes about 4s (for 278 archives), which isn't terrible, but seems a bit long compared to how long other operations take.

[jdchristensen]

Or maybe there could be a variant of --stats which shows the This archive: line but not the All archives: line? I'm not sure, but I'm guessing that it's only for the All archives line that all of the other archives need to be examined.

[ThomasWaldmann]

@jdchristensen your timings are quite similar to @szpak's timings, he has 6x as many archives as you and also about 6x the time needed for the stats.

Speed depends a bit on the borg version and the version of the archives - I remember doing some tuning / caching there in borg 1.2 (but first info/stats will be slow).

Stats related stuff will already be a bit different in borg2 (e.g. because I had to remove the csize values, because there we will always have some infos in the archives, etc.).

But some all archives or repo-wide stuff might be still slow - guess that would be an interesting experiment.