Borgmatic backup issue
Written by skeirss
Problem
When running borgmatic create prune via the systemd service (borgmatic.service), the backup process hung indefinitely during the create phase after the log line Processing files …
and the warning
/var/lib/docker/containers/...-json.log: file changed while we backed it up.
The hang occurred only when the database hooks were active (mariadb_databases and sqlite_databases). These hooks automatically enable --read-special and cause borgmatic to include temporary dump files, which in turn triggered extensive special-file exclusions under /var/lib/docker.
A broad exclusion
- /var/lib/docker/**
prevented the hang but also omitted all Docker volume data (persistent application files located under /var/lib/docker/volumes/).
Manual execution of borgmatic create from an interactive root shell completed successfully, confirming the issue was specific to the non-interactive systemd execution environment combined with Docker’s overlay2 layers, container logs, and runtime files.
Root Cause
-
Database hooks force --read-special and interact with rapidly changing or special files inside /var/lib/docker.
-
Borg’s file processing becomes blocked when encountering Docker overlay files, container log files that change during backup, and certain pseudo-files in the overlay2 directory.
-
The broad Docker exclusion resolved the hang but removed the desired volume backups.
Solution
Retain the database hooks (they provide consistent, point-in-time dumps) and replace the broad Docker exclusion with granular exclusions that target only the ephemeral/problematic paths. Update the exclude_patterns section in /etc/borgmatic/config.yaml as follows (add/replace the Docker-related lines):
exclude_patterns:
...
- /var/lib/docker/overlay2/**
- /var/lib/docker/containers/**
- /var/lib/docker/image/**
- /var/lib/docker/tmp/**
- /var/lib/docker/volumes/*/_data/*.log
More info about this issue in the docs: https://torsion.org/borgmatic/how-to/backup-your-databases/