Cludes

Bashtic’s cludes feature takes all the hard work out of using complex include/exclude rules with restic.

First we’ll descibe the current way of achieving this with restic alone:

  • Explicitly listing every file for backup

  • Mixing include and exclude patterns

Then we’ll describe the bashtic approach.

Explicitly listing every file for backup

One approach with restic is to use one of the --files-from flags to provide an explicit list of what to backup with an external command such as find. This will however pollute your restic snapshot’s ‘paths’ metadata with every entry from the given file.

Apart from the visual clutter when listing snapshots this also defeats restic’s logic during backup when searching for a parent: parent-snapshot detection fails with changing –files-from · issue #2246. Without a parent the backup will scan every file, slowly the backup process considerably.

You can workaround this by telling restic exactly which snapshot to use as a parent. However this can also be complex depending on how you manage your backups with different hosts, tags and paths.

Some have suggested further fixes for this, e.g. add an option to restic backup to override the snapshot’s “paths” field · issue #2714. However these suggestions have not moved forward.

Mixing include and exclude patterns

In March 2022 a new feature was added to restic to allow mixing include and exclude patterns: Allow mixing include and exclude patterns for backup and restore · issue #233.

This meant users could add gitignore-style negative patterns with their include patterns. This helps a lot but crafting and maintaining the right mix of rules is still complex for all but the simplest cases:

  • Intermediate directories between a deeper and a shallower rule must have their own rules or the result will be wrong.

  • To backup just one directory in an otherwise ignored directory you must first exclude everything than add a negative include.

  • Suffixing your rules with /* is sometimes required and sometimes not depending on the presence of any rules applying to deeper paths and whether paths actually exist on disk or not.

Despite these drawbacks it does not suffer from the issues with --files-from as noted above i.e. verbose snapshot metadata and defeating snapshot parent logic.

The bashtic approach

It’s better to let the user describe their backup set in a simple, maintable form and have a program translate that into the right include/exclude rules for restic’s needs.

Bashtic offers this through the cludes setting of a location configuration. Simply list the paths you want to include/exclude and bashtic will work out what the exclude file should look like for restic’s --exclude-file option.

For example, we can exclude the user’s .var dir but still include a much deeper dir with these two lines:

cludes=(
"$HOME/.var#x"
"$HOME/.var/app/net.minetest.Minetest/.minetest/worlds#i"
)

The special suffixes #x and #i mean eXclude and Include respectively.

This simple configuation actually results in a complex set of exclude rules for restic as we have to deal with every intermediate directory between the two paths to get the right result:

/home/you/.var/*
!/home/you/.var/app
/home/you/.var/app/*
!/home/you/.var/app/net.minetest.Minetest
/home/you/.var/app/net.minetest.Minetest/*
!/home/you/.var/app/net.minetest.Minetest/.minetest
/home/you/.var/app/net.minetest.Minetest/.minetest/*
!/home/you/.var/app/net.minetest.Minetest/.minetest/worlds

Nobody wants to maintain these rules themselves. Bashtic creates this file for you and feeds it to restic so you don’t have to.

The other good news is that using --exclude-file does not suffer from the same drawbacks as --files-from i.e. snapshot metadata is not complicated and parent logic is unaffected.