Monitoring restic and borg backups: alert on failed or missing snapshots

restic and BorgBackup are excellent, but they only run when something runs them. A cron line lost in a server rebuild, an expired repository password, or a full disk all stop your backups without anything on screen. The fix is a heartbeat that expects a successful backup every night and tells you when one doesn't arrive.

restic

#!/bin/sh
# /usr/local/bin/backup-restic.sh, run nightly from cron or a systemd timer
export RESTIC_REPOSITORY=s3:s3.amazonaws.com/my-backups
export RESTIC_PASSWORD_FILE=/root/.restic-pass
restic backup /etc /srv /home --exclude-caches \
  && restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune \
  && restic check --read-data-subset=1%

Run it through the cronpup-run wrapper, so the exit code and the tail of the output are reported:

0 2 * * *  cronpup-run https://cronpup.com/p/UUID /usr/local/bin/backup-restic.sh

borg

#!/bin/sh
export BORG_REPO=ssh://backup@host/./repo
export BORG_PASSCOMMAND="cat /root/.borg-pass"
borg create --stats ::'{hostname}-{now}' /etc /srv /home \
  && borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 \
  && borg compact

borg exits with 1 for warnings (e.g. a file changed while being read) and 2 for errors. If warnings are normal for you, map them to success in the script (rc=$?; [ $rc -le 1 ] && rc=0; exit $rc), so you're only alerted on real failures.

Also check that you can restore

A heartbeat proves the backup job ran. Add a second monthly job that restores a sample file and compares checksums, with its own monitor. That proves the backups are usable.

Monitor your backups. 10 monitors are free.