truenas-stacks/docs/operations/PD_BACKUP_DEPLOYMENT.md

PD Backup Deployment

Deploy the PD → Serenity backup runner from the real compose path on PD:

/mnt/docker-ssd/docker/compose/automation

Why this model

• PD is the compose/data host, so backups should originate there.
• TrueNAS SCALE is happier with plain cron + shell than extra appliance-fighting service glue.
• Root cron on PD avoids the common sudo/Docker socket permission mess.
• Backups should still run even if n8n is down.

Prerequisites

1. Repo content present on PD at /mnt/docker-ssd/docker/compose/automation
2. Writable dump directory:

   mkdir -p /mnt/tank/docker/backups/db-dumps
   

3. SSH trust from PD to Serenity for the selected account
4. Real .env created from .env.example
5. Preferred: install the cron job in root's crontab on PD
6. If using a non-root PD account instead, sudo -n /usr/bin/docker must work

Suggested .env values to review

SERENITY_BACKUP_HOST=root@10.5.1.5
SERENITY_BACKUP_ROOT=/mnt/user/backups/plausible-deniability
PD_APPDATA_ROOT=/mnt/docker-ssd/docker/appdata
PD_DATABASES_ROOT=/mnt/docker-ssd/docker/databases
PD_TANK_DOCKER_ROOT=/mnt/tank/docker
PD_DB_DUMP_ROOT=/mnt/tank/docker/backups/db-dumps
POSTGRES_CONTAINER=shared-postgres
POSTGRES_DB_LIST="n8n paperless"
BACKUP_SSH_KEY=/home/truenas_admin/.ssh/serenity_backup_ed25519
DOCKER_BIN=/usr/bin/docker
SUDO_BIN=/usr/bin/sudo
USE_SUDO_FOR_DOCKER=true

Manual first run

cd /mnt/docker-ssd/docker/compose/automation
cp .env.example .env
chmod 600 .env
mkdir -p /mnt/tank/docker/backups/db-dumps
/usr/bin/bash bin/run_pd_backups.sh

Confirm:

• fresh *.sql.gz files appear in /mnt/tank/docker/backups/db-dumps
• appdata/database/tank-docker content lands under the chosen Serenity backup root
• no sudo prompt appeared during docker exec

Cron install

Preferred: install in root's crontab on PD.

# BEGIN PD BACKUPS
15 2 * * * cd /mnt/docker-ssd/docker/compose/automation && /usr/bin/bash bin/run_pd_backups.sh >> /mnt/tank/docker/backups/pd-backups.log 2>&1
# END PD BACKUPS

Quarterly restore verification:

30 3 1 */3 * cd /mnt/docker-ssd/docker/compose/automation && /usr/bin/bash bin/run_pd_restore_verification.sh >> /mnt/tank/docker/backups/pd-restore-verify.log 2>&1

Post-deploy checks

tail -100 /mnt/tank/docker/backups/pd-backups.log
ls -lh /mnt/tank/docker/backups/db-dumps
ssh root@10.5.1.5 'find /mnt/user/backups/plausible-deniability -maxdepth 2 -type d | sort | head -40'

Restore verification

After the backup path is stable, run the staged restore verifier from PD:

cd /mnt/docker-ssd/docker/compose/automation
/usr/bin/bash bin/run_pd_restore_verification.sh

What it proves:

• the latest configured Postgres dumps can restore into a temporary container
• a real appdata/config sample can be staged back from Serenity
• expected files still exist inside the restored sample
• the latest run state can be exported to Prometheus via node-exporter's textfile collector

Important limitations

• Do not use systemctl restart docker on PD.
• Keep source/destination paths under /mnt/....
• rsync --delete is intentional; use a dedicated destination root, not a shared miscellaneous folder.
• This scaffolding does not yet back up .env files here because those are already handled by the separate encrypted/private-repo process.
• If you insist on a non-root cron user on PD, you will need passwordless Docker access (sudo -n /usr/bin/docker ...) and write access to the chosen dump directory.