Cludes¶
Bashtic’s cludes feature takes all the hard work out of using complex include/exclude rules with restic.
First we’ll descibe the current way of achieving this with restic alone:
Explicitly listing every file for backup
Mixing include and exclude patterns
Then we’ll describe the bashtic approach.
Explicitly listing every file for backup¶
One approach with restic is to use one of the --files-from
flags
to provide an explicit list of what to backup with an external command such as
find
. This will however pollute your restic snapshot’s ‘paths’ metadata with
every entry from the given file.
Apart from the visual clutter when listing snapshots this also defeats restic’s logic during backup when searching for a parent: parent-snapshot detection fails with changing –files-from · issue #2246. Without a parent the backup will scan every file, slowly the backup process considerably.
You can workaround this by telling restic exactly which snapshot to use as a parent. However this can also be complex depending on how you manage your backups with different hosts, tags and paths.
Some have suggested further fixes for this, e.g. add an option to restic backup to override the snapshot’s “paths” field · issue #2714. However these suggestions have not moved forward.
Mixing include and exclude patterns¶
In March 2022 a new feature was added to restic to allow mixing include and exclude patterns: Allow mixing include and exclude patterns for backup and restore · issue #233.
This meant users could add gitignore-style negative patterns with their include patterns. This helps a lot but crafting and maintaining the right mix of rules is still complex for all but the simplest cases:
Intermediate directories between a deeper and a shallower rule must have their own rules or the result will be wrong.
To backup just one directory in an otherwise ignored directory you must first exclude everything than add a negative include.
Suffixing your rules with
/*
is sometimes required and sometimes not depending on the presence of any rules applying to deeper paths and whether paths actually exist on disk or not.
Despite these drawbacks it does not suffer from the issues with --files-from
as noted above i.e. verbose snapshot metadata and defeating snapshot parent
logic.
The bashtic approach¶
It’s better to let the user describe their backup set in a simple, maintable form and have a program translate that into the right include/exclude rules for restic’s needs.
Bashtic offers this through the cludes
setting of a
location configuration. Simply list the paths you
want to include/exclude and bashtic will work out what the exclude file should
look like for restic’s --exclude-file
option.
For example, we can exclude the user’s .var
dir but still include a much
deeper dir with these two lines:
cludes=(
"$HOME/.var#x"
"$HOME/.var/app/net.minetest.Minetest/.minetest/worlds#i"
)
The special suffixes #x
and #i
mean eXclude and Include respectively.
This simple configuation actually results in a complex set of exclude rules for restic as we have to deal with every intermediate directory between the two paths to get the right result:
/home/you/.var/*
!/home/you/.var/app
/home/you/.var/app/*
!/home/you/.var/app/net.minetest.Minetest
/home/you/.var/app/net.minetest.Minetest/*
!/home/you/.var/app/net.minetest.Minetest/.minetest
/home/you/.var/app/net.minetest.Minetest/.minetest/*
!/home/you/.var/app/net.minetest.Minetest/.minetest/worlds
Nobody wants to maintain these rules themselves. Bashtic creates this file for you and feeds it to restic so you don’t have to.
The other good news is that using --exclude-file
does not suffer from the same
drawbacks as --files-from
i.e. snapshot metadata is not complicated and parent
logic is unaffected.