Passa al contenuto principale

Backup Cross-Region Storage Box Mirror

Akira mirrors the existing ADR-0012 PostgreSQL backup layout from the primary Hetzner Storage Box in fsn1 to a second Storage Box in nbg1.

This runbook follows the accepted backup strategy:

  • nightly pg_dump --format=custom files in /backups/pg;
  • continuous WAL archive files in /backups/wal;
  • restore validation through the existing DR drill flow.

Provisioning​

Create the second Storage Box manually in Hetzner Cloud Console:

  • product: Storage Box BX11, 1 TB;
  • location: nbg1;
  • subaccount: akira-backup-mirror;
  • authentication: ASCII password plus SSH public key.

Populate and encrypt the vault values:

vault_storagebox_mirror_host: "u123456.your-storagebox.de"
vault_storagebox_mirror_user: "akira-backup-mirror"
vault_storagebox_mirror_password: "<password>"
vault_storagebox_mirror_ssh_key: "<encrypted private key block>"
vault_storagebox_mirror_host_pubkey: "u123456.your-storagebox.de ssh-ed25519 ..."

After provisioning, set backup_mirror_enabled: true for the DB primary host and run the timescaledb role.

Schedule​

akira-backup-mirror.timer runs every 6 hours at 00:30, 06:30, 12:30, and 18:30 with up to 10 minutes of randomized delay.

The job copies missing files from the primary Storage Box to the mirror:

  • /backups/pg/akira-*.dump;
  • /backups/wal/*.

The mirror keeps longer retention than the primary operational window:

  • dumps: 45 days;
  • WAL: 21 days.

Verification​

Weekly verification runs Mondays at 07:00:

systemctl status akira-backup-mirror-verify.timer
journalctl -u akira-backup-mirror-verify.service -n 100

Manual check:

sudo BACKUP_MIRROR_ENV_FILE=/etc/akira/backup-mirror.env /usr/local/bin/verify-backup-mirror.sh

The check fails if the newest mirrored dump lags the primary dump by more than 8 hours.

Alerting​

backup-mirror-sync.sh writes node_exporter textfile metrics to:

/var/lib/node_exporter/textfile_collector/akira_backup_mirror.prom

Prometheus rule BackupMirrorLagHigh fires when the last successful mirror sync is older than 8 hours for 30 minutes.

Restore From Mirror​

Use the mirror if the primary fsn1 Storage Box or region is unavailable:

sudo BACKUP_MIRROR_ENV_FILE=/etc/akira/backup-mirror.env \
  /usr/local/bin/dr-restore-from-mirror.sh mirror akira-dr-scratch

The script downloads the latest mirrored dump, validates it with pg_restore --list, uploads it to the scratch host, recreates the target database and runs pg_restore.

For the primary Storage Box path:

sudo /usr/local/bin/dr-restore-from-mirror.sh primary akira-dr-scratch

Failure Handling​

If nbg1 is unavailable, the primary backup path continues to work. The mirror job is idempotent and resumes at the next timer run, copying only missing files.

If the mirror lags:

  1. Check journalctl -u akira-backup-mirror.service -n 200.
  2. Verify both SSH keys and Storage Box host keys.
  3. Check free space and quota on the mirror Storage Box.
  4. Run /usr/local/bin/backup-mirror-sync.sh manually after fixing access.

Cost​

The pilot cost increase is one additional BX11 Storage Box in nbg1, about EUR 4/month. Total Storage Box backup cost is about EUR 8/month.