I've organized my first home automated backup system and tried my best to describe it (the system seems a bit complicated to me, so in case something goes wrong I'll definitely need a reference). Since this is my first attempt at setting up a backup system, it may have some flaws.

Backup System Documentation

TL;DR

• All backups are stored on NAS (192.168.88.40)
• Storage path: /srv/backups, /mnt/media/backups (for media)
• Tools: borg + borgmatic + bash scripts
• Clients: arch-pc, vps-nl, NAS itself
• Check status: systemctl status borgmatic.timer
• Logs: journalctl -u borgmatic.service
• Restore: see "Restore" section

Architecture

[arch-pc] ──(borgmatic.timer -> daily at 22:00)──borgmatic {/etc/borgmatic.d/arch-pc.yaml} ──▶ [NAS]──/srv/backups/arch-pc (Samsung SSD)

[vps-nl] ──(borgmatic.timer -> daily at 16:00 Server time (21:00 EKT))──borgmatic {/etc/borgmatic/config.yaml} ──▶ [NAS]──/srv/backups/vps-nl (Samsung SSD)

[arch-pc-media] ──media-backup.sh (running manually)──borgmatic {/etc/borgmatic.d/arch-pc-media.yaml} ──▶ [NAS]──/mnt/media/backups/arch-pc-media (External HDD)

[nas] ──(borgmatic.timer)──borgmatic {/etc/borgmatic/config.yaml}──▶ /srv/backups/nas (Samsung SSD)

Networking

arch-pc → NAS

• Direct SSH connection
• SSH Alias: home-borg (described in /root/.ssh/config)
• Path: ssh://home-borg/srv/backups/arch-pc

arch-pc-media → NAS

• Direct SSH connection
• SSH Alias: home-borg (described in /root/.ssh/config)
• Path: ssh://home-borg/srv/backups/arch-pc-media

VPS → NAS

• Reverse SSH tunnel (described in /etc/systemd/system/ssh-tunnel-persistent_amsterdam.service)
• Mapping: VPS localhost:2233 → NAS:3315
• SSH Alias: home-borg (described in /root/.ssh/config)
• Path: ssh://home-borg/srv/backups/vps-nl

NAS → NAS

• No network, direct data transfer

Storage Layout

/srv/backups/

arch-pc/

vps-nl/

nas/

/run/media/bernd/backups/

arch-pc-media/

Archive format: {hostname}-{timestamp}

Monitoring

Notifications:

• Telegram bot

Triggers:

• success
• failure

Script:

• /usr/local/bin/send_message_tg.sh

Restore

List archives: borg list /path/to/repo or outside the NAS: borgmatic repo-list --config /etc/borgmatic.d/arch-pc.yaml

Mount the archive to /mnt/borg (*): sudo borgmatic mount --config /etc/borgmatic.d/arch-pc.yaml --archive archpc-2025-10-03T21:25:12 --mount-point /mnt/borg

Unmount: sudo borgmatic umount --mount-point /mnt/borg

Extract a file from the archive: sudo borgmatic extract --config /etc/borgmatic.d/arch-pc.yaml --archive archpc-2025-10-03T21:25:12 --path /path/to/file/to/extact --destination /tmp

Restore all files from the archive: borgmatic extract --config /etc/borgmatic.d/arch-pc.yaml --archive archpc-2025-10-03T21:25:12

(*) fuse2 and python-llfuse packages should be installed

Troubleshooting

Case 1. borg.crypto.key.ArchiveTAMInvalid: Data integrity error: Archive authentication did not verify error when trying to open an archive

Solution: This problem is related to Trusted Archive Metadata (TAM). The solution is to repair the archives:

su - 

BORG_WORKAROUNDS=ignore_invalid_archive_tam borg upgrade --archives-tam /path/to/repo

Check: borg check /path/to/repo

To-Do

• Docker containers configs + DB backup and restore
• There are two weak points in the current setup: the NAS backing up to itself (so this is not really a backup, just versioning), and the lack of an offsite backup. Both issues could be addressed by periodically copying backup archives to an encrypted external drive stored elsewhere.

So today last.fm stopped working for Russia:

❯ curl -I https://www.last.fm/ | head -n1
HTTP/2 403

So here's the problem: now my dockerized Navidrome server won't scrobble tracks to last.fm, and since Navidrome doesn't support proxy configuration, I have to route all network traffic from the container through a proxy server.

Sometimes I feel like I'm between a rock and a hard place, because the restrictions are coming from both sides... But something good always comes out of every bad situation, and from this one I'll learn a bit more about proxying docker containers. Anyways.

What's my goal: make Navidrome scrobble tracks to last.fm again by routing its traffic though my host machine's Xray proxy. What's my steps:

• Install and configure xray on my server
• Update docker compose, add proxy variables

So first I need to update my Xray configuration. Basically I need to tell Xray to listen on all network interfaces (0.0.0.0) so the Docker container can reach it:

  "inbounds": [
    {
      "listen": "0.0.0.0",
      "port": 10808,
      "protocol": "socks"
    },
    {
      "listen": "0.0.0.0",
      "port": 10809,
      "protocol": "http"
    }
  ],

And since my server is local and working behind a router and cg-nat, it is a safe approach.

Now I need to update my docker compose file, so Navidrome can find the host and use the proxy vars:

services:
  navidrome:
    extra_hosts:
      - "host.docker.internal:host-gateway"
    environment:
      - HTTP_PROXY=http://host.docker.internal:10809
      - HTTPS_PROXY=http://host.docker.internal:10809
      - ALL_PROXY=socks5://host.docker.internal:10808
      - NO_PROXY=localhost,127.0.0.1

Why extra_hosts? It creates a "map" so the container knows host.docker.internal = your host machine's IP. Checking that variables described in the docker file are read within the container:

 root@thinkcentre:~# docker exec navidrome-navidrome-1 env | grep -i proxy
no_proxy=localhost,127.0.0.1
http_proxy=http://host.docker.internal:10809
https_proxy=http://host.docker.internal:10809
all_proxy=socks5://host.docker.internal:10809

Looks like it's fine. Too bad we can't test connection inside the container using curl, because there's no such utility in the container. Let's just play a song and see if it is scrobbling or not: Yes, it is working as intended, yay!

Only what he finds conducive is to his taste; his pleasure, his enjoyment stops when the mark of what is conducive is overstepped. He guesses correctly what will heal harm, he exploits strokes of bad luck to his advantage; what does not kill him makes him stronger.