fix: GUI and documentation of Auto-remove

[buhtz]

See meta issue #1945

Problem

The Auto-remove (and its Smart-remove part) feature is not well documented, nor is its presentation in the Auto-remove TAB of the Manage profiles dialog; hence, it is unclear and sometimes misleading for users.

Goals

1. Modify the GUI accordingly. See mockup.
2. Improve and create the user manual with description and examples.
3. Don't modify the behavior. This will be done in a later release. But maybe add a warning-like message to the GUI and the manual that this behavior will be modified in the foreseen future.

Details about modifications planed

• Remove "smart remove" checkbox to disable/enable all "smart" rules at once are removed.
• Remove term "smart".
• Inode/space rules: Disable them by default in new created profiles.
• Link to the user manual
• Info Symbol instead of warning/attention

EDIT:

[DerekVeit]

I like the re-ordering and the clarifications.

With the current GUI default, "Smart remove" is unchecked, and the effect of that is that nothing will be deleted (until it is 10 years old if that other option is left checked). Choosing to have backups automatically deleted is done at the same time as choosing what to keep. With this new design, everything is deleted unless some "keep" options are checked and set, which seems a little strange. With this update, an existing configuration that doesn't have "Smart remove" enabled would have to have some arbitrary number of days-to-keep imposed.

Since removing backups is a positive action by the application, and not necessarily wanted by some users, I think I would rather keep the option to not automatically delete backups at all.

Maybe the words "Smart remove" could be changed to "Remove previous snapshots, but keep...".

Disabling the inode/space rules by default seems good to me.

[DerekVeit]

Or if each "keep" option allowed a "forever" value, that could give even better control over it.

[buhtz]

Thank you for the feedback. You are right and I totally missed that aspect. A modified the mockup and added it to my initial post.

[noyannus]

Below is a more general approach, looking at the competitors (re-inventing the wheel will not make it rounder. :-) ). Retention management is a solved problem, and BiT can copy from the best.

Let's have a look at the most popular backup applications listed by LinuxLinks:, skipping over the non-GUI, no retention management, and industry-scale applications.

BackRest interface for Restic
By time

By count

"Retention policy" could be labelled more intuitively. "Keep...."

Vorta interface for Borg

Vorta does it right, with a good explanation on top.

Duplicity (source)

A rather crude approach, or I didn't find the other settings.

Timeshift (source

An "on boot" option is good for experts who frequently modify their system. Not necessary for BiT, or maybe under "Advanced"?

Duplicati (source

Listing all policies at once with radio buttons (they are alternatives) would be easier to grok.
I didn't find a good screenshot for Duplicati Advanced on the quick; this explanation must do:

The retention can be set in 5 ways:

Keep all backups:
Backups will never be deleted. This is the most safe option, but remote storage capacity will keep increasing.
Delete backups that are older than:
Backups older than a specified number of days, weeks, months or years will be deleted.
Keep a specific number of backups:
The specified number of backup versions will be kept, all older backups will be deleted.
Custom backup retention: Enter your own settings in the text box that appears.
Smart backup retention: retains one backup for each of the last 7 days, each of the last 4 weeks, and each of the last 12 months.

Consider the case of Smart Backup Retention. Using Duplicati notation it can be written as '1W:1D,4W:1W,12M:1M'. This results in this behaviour:

Active files: Files that exist in the source will continue to be included in at least on this explanatione backup version. For example, a file that exists in the source but which is 3 years old will still be in at least one backup version, regardless of the rentention policy.

Deleted files: When a file is deleted from the source it will not be included in any new backups. Old backup versions will retain a copy of the deleted file for at least 12 months. After 12 months the last backup version to include it will eventually be removed.

Modified files: When a file's content changes, previous versions will follow the backup retention policy. In other words, there will be a maximum of one version per day for the last week, one version per week for the last 4 weeks, and one version per month for the last 12 months. A daily version that ages to over a week old will either be retained as that week's version, or deleted.

---

Listing the options from the above:
Restic/Backrest -- keep N of X-ly period; count
Borg/Vorta -- keep N of X-ly period; keep all of last X-ly period
Duplicity -- delete all after
Timeshift -- N X-ly; has an additional "backup on boot" option
Duplicati -- Has an advanced option and five default settings: Keep all; keep until ; keep N (regardless of age); keep 1 per last 7 days + 4 of the last 4 weeks + all of the last 12 months (the "smart backup").

Notes for Duplicati:

• It has rolling periods, for 7 days and 4 weeks. The words "week" or "month" are not used, this prevents confusion with calendar-set periods.
• Period lengths can be customized: "The retention policy is a "bucket" based strategy, where you define how many backups to keep in each "bucket" and what a "bucket" covers." So "Of each 17-week period keep [the last, I think] 5 days" is valid.

---

So, "keep" is the most popular wording and to be preferred where possible. It is more intuitive than "delete except", because it a) settings do not vary, and b) it is a smaller cognitive load if the switch from original intention ("What do I want to be accessible if needed") to its negative ("What do I have to delete to only have what I want to be accessible if needed") is avoided. A similar "Delete older than" requires an explicit notion that the "keep" settings of all longer periods apply regardless.

"Keep N of X-ly period" ist common and easily understood. Don't change it.

The period can be made rolling, but that would require an up-front alert for the user, i.e., a brief text on the settings pane (even a tooltip only is not enough, it requires action to reach). Rolling periods could be a possible future enhancement for BiT, unless it is very quick and easy to implement, I'd say.

Need to define "week" - ie. Sunday to Saturday, or Monday to Sunday and not any 7 consecutive days. Also should state that "month" means a calendar month.

Seconded, except that "7 days" could be a small enhancement in the future.

Or if each "keep" option allowed a "forever" value, that could give even better control over it.

I think I would rather keep the option to not automatically delete backups at all.

Duplicati uses the "U" (unlimited) in the settings for that. A "0" or empty field could also work. In any case this needs an indication right on the settings pane; e.g., a small-text footnote; "keep*...": "*To keep all snapshots, [leave the fields blank|enter'[U|0]']". Also state that "forever" overrides all longer-period settings.

Disabling the inode/space rules by default seems good to me.

I'm for keeping both but moving inodes to the "Advanced" tab. It's understood only by experts anyway.

All defaults should not delete any data.

IMO the most useful extra would be a trail run option (like the rsync -n option) - show what would happen without actually doing anything.

+1. A button at the bottom "Show dry run (not executing)" that opens a dialog with the terminal output, including the prompt, could do that.

I'm not sure that all the "hand-holding" that's been suggested is needed. Good documentation (I keep repeating!) solves most issues.

Not needed, sure. But if you create a GUI, do it right. KISS, intuitiveness, and discoverability have been found to be very good leading principles. Make things as easy to find out as possible.

[buhtz]

A lot to think about. Thank you. I linked your comment also in #1977 , which is about modifying the auto-remove behavior in a release after the upcoming release.

I like some of the ideas. I don't understand everything. For example the Vorta GUI does not help me to understand. For me it is not self explaining. What does "Hourly 48" means? Keep 48 backups in each hour? Keep all backups for the last 48 hours? If it means "Keep one backup for each 48 hours" I would say the GUI does not say that.

The same goes for BackRest. What is the meaning of the numbers?

Also TimeShift. "Hourly keep 6". Does it mean to keep 6 backups each hour? So if there are backups for the last 4 hours there would be 24 backups (6x4) kept?

[DerekVeit]

Thanks, @noyannus, That's a great survey and all worth looking at for ideas. I had in the back of my mind to do the same thing. Good analysis too.

I chose Back In Time largely because it's configuration was much better, in my opinion, than all the alternatives that I had looked at, including Timeshift, which comes standard with Linux Mint and which I deliberately moved away from in favor of Back In Time. I think Back In Time's configuration interface is still better than the rest even while it can be improved.

Most of those are more ambiguous than Back In Time about retention, e.g. "2 Monthly". On the face of it that could even be mistaken as 2 per month. Oh, I see Christian pointed this out too.

Overall, what I like about Back In Time's configuration GUI is that it is very articulate and well organized. I find it easy to understand, navigate, and change. It feels simple enough and yet allows a surprisingly deep amount of customization and control without needing to refer to a manual.

I like the "dry run" button idea. Issue #248 is open for that.

And I very much agree with those GUI design principles. And contrasted with those other UIs, maybe Back In Time is already clear enough now with its clarifications about retention.

[buhtz]

Here is a new version of the mockup. I like the term "Retention policy".

I would say it is the less worst solution without modifying the behavior. But I will take all your comments into account when it comes to #1977 in a later release.

Does anyone see big problems in this mockup?

I will start with the user manual.

[noyannus]

This is a good milestone, IMO. There's 'nuff road ahead if one wants to travel further, but for now it is a major improvement.

Small random things that came to mind, without particular priority or order:

"user manual" in the label could be a direct link to the help in BiT or to a web page.

"Later rules override earlier ones..." is correct for the top-down reading sequence, but usually "later" and "earlier" denote the position of points in (or periods of) time on a timeline. Here we have a logical topic, so something like "Rules of longer periods override..." would IMO be more suitable. Also, a more concrete corollary helps understanding, so maybe add "...to always keep the maximum number of snapshots of every rule" or similar. Native speakers, your opinion, please!

The "Attention:" feels strange to me, it sounds too much like the stereotypical German in post-world-war movies. Again a point for native speakers to refine. Or simply use an icon ("🛈" or "ℹ️" for example) instead, everybody will understand that, it saves space, and the eye is drawn to it more than to text.

"Retention policy" is good; the "keep" in all relevant places makes its meaning clear. Ditto the absence of the confusing "x-ly" wording. (@buthz: I see the point of "hourly" in Vorta etc.. Apparently this is a tricky word for us non-native speakers.)

"Remove snapshots older than NN years" conflicts with "Keep the last snapshot of each year for all years" when both "Remove..." and "Retention policy" are active, because "Keep... years" cannot be switched off.

How about this:

• Make Retention Policy not optional, i.e., the checkbox has to go and the line now serves as a headline. Rules will then always be active, with sensible defaults as in the example.
• Make "Keep ... for .. years" into a numeric setting like the others above it, with a default of a spelled-out "all", or a note "For all years, enter 'A' [or 'all']", or "For forever, enter 'U' ", or " '0' (or '-1', or so) to keep all snapshots" ... you get my drift.
• Then you can safely omit the "Remove snapshots older..." line, but you must add a warning line below the "keep ..years" line, like "⚠ All snapshots will be removed that have aged beyond these periods, unless a longer 'keep...' rule below still keeps them", or so. This is redundant with the "Attention" text and could be merged with it.

And now I'll try to get a life. :-)

[buhtz]

Thank you for the detailed feedback. I will take this into account, e.g. the link to user manual and the info symbol. The other things currently "to big" to change. I will consider this for a later release in context of #1977 . I also referenced your comment in #1977 to not forget it.