Everything in this series ends up in InfluxDB: years of grid readings, the PV yield, the prices and every number the posts are built on. The home server backs it up every week, and Duplicati uploads the backups to the cloud. I was sure that this works. Then I tried to restore it, and found three problems in one week. None of them showed up in any log I usually look at.
Restoring into a throwaway InfluxDB
A restore test should not touch the real database. So a small script starts a second InfluxDB in a Docker container, the same version as the one on the host, without network and with the backup folder mounted read-only. It restores the newest backup set bucket by bucket and then counts the points of the last full day before the backup, in the restored copy and in the live database:
docker run -d --name influx-restore-test --network none -v "$backup_dir:/backup:ro" \
-e DOCKER_INFLUXDB_INIT_MODE=setup ... "influxdb:$version"
for bucket in $buckets; do
restored restore "${auth[@]}" --bucket "$bucket" /backup
live=$(count influx query ...)
backup=$(count restored query "${auth[@]}")
done
The whole script has about 80 lines and works for any InfluxDB 2 with influx backup sets: influx-restore-test.sh.
The first run looked good:
openhab_db live 695890 restored 695890 ok openhab_db_calc live 631 restored 631 ok speedtest-tracker live 224 restored 224 ok restore test ok
The backup itself was fine. What came after it was not.
Problem 1: Duplicati could not read the backup
Duplicati runs as its own user and gets read access to the backup folder through an ACL, an extra permission entry for just that user. New files inherit it from the folder. Except they did not.
influx backup writes its files with mode 0600, readable for the owner only. With ACLs that has a side effect: the group bits of the mode become the ACL mask, and the mask limits every extra entry. Mode 0600 means mask ---, so the inherited read permission for Duplicati was still there, but cancelled. The Duplicati log had the answer, if I had looked:
PermissionDenied: ... BackUp/influx/...
The job ran, finished, and skipped exactly the files it was there for. The fix is one line after every backup run:
ExecStartPost=/usr/bin/find /path/to/BackUp/influx -maxdepth 1 -type f -perm 0600 -exec setfacl -m g::-,m::r {} +
The mask becomes r and the group gets nothing, so only the Duplicati user can read the set. That matters, because a backup set contains the token database.
Problem 2: “encrypted” but not encrypted
The Duplicati UI showed the job as an encrypted backup. Duplicati keeps a small database per job with every file it uploaded, and there the volumes ended in .zip. Encrypted volumes end in .zip.aes.
The job was created years ago in the old UI and carried an advanced option --no-encryption. The new UI neither shows nor removes it. So the database backups went to the cloud as plain ZIP files, while the UI said the opposite. I created the job again in the new UI, and now the volumes end in .zip.aes.
The lesson for me: check what is on the remote side, not what the UI says.
Problem 3: a backup that was 20 bytes
This blog runs on a separate server, and it needed a backup before an upgrade. The plan was simple. Every night at 03:30 the blog server writes a database dump, /etc and the WordPress files into one folder. At 04:00 the home server pulls that folder with rsync into a dated folder on the RAID. Unchanged files are hard links to the day before, so 14 days of snapshots take little more space than one.
The home server gets its own SSH key, and on the blog server that key can do exactly one thing:
restrict,command="/usr/bin/rrsync -ro /var/backups/webserver" ssh-ed25519 AAAA...
rrsync only allows rsync, read-only, below that one folder. No shell, no forwarding.
The first run finished without an error. The snapshot was 812 MB. And the database dump was 20 bytes.
mariadb-dump: Got error: 1045: "Access denied for user 'root'@'localhost' (using password: NO)"
The MariaDB root user on that server needs a password. mariadb-dump failed, but it ran in a pipe:
mariadb-dump --all-databases | gzip > db.sql.gz
In a shell pipe only the exit code of the last command counts. gzip compressed nothing very successfully, the script carried on and wrote its “completed” marker. Without the restore check I would have had a backup with every file of the blog and none of its posts.
Three changes fixed it. The script runs with set -o pipefail, so a failing dump stops it. After the dump it checks that the last line says Dump completed. And the dump runs as a read-only database user that logs in through the Unix socket, so there is no password on the server:
CREATE USER 'backup'@'localhost' IDENTIFIED VIA unix_socket; GRANT SELECT, SHOW VIEW, TRIGGER, LOCK TABLES, EVENT, PROCESS ON *.* TO 'backup'@'localhost';
runuser -u backup -- mariadb-dump --user=backup --single-transaction --routines --events --all-databases \
| gzip > db.sql.gz.tmp
zcat db.sql.gz.tmp | tail -1 | grep -q "Dump completed" || exit 1
The pull on the home server only accepts a day when the marker is there. The next dump was 3.4 MB and ended with -- Dump completed.
And the ACL trick from problem 1 came back on the first pull. The backup folder passes default:user::rw- to new folders, without x, so the pull user could not even enter the folder it had just created. Two default ACL entries on the target folder fixed that.
What I check now
- The restore test runs after every InfluxDB upgrade. It takes two minutes and ends with “restore test ok” or tells me which bucket differs.
- A database dump only counts if it ends with
Dump completed. - Encryption is checked on the remote side, by the file names, not in the UI.
- Read access is checked as the user who reads, for example with
docker exec -u <user> duplicati head -c 16 <file>.
All three problems had one thing in common. Every job reported success, and the backups looked fine from the outside. Only reading them back showed what was missing.