chore: sync homelab ops, identity, and monitoring docs

This commit is contained in:
Fizzlepoof
2026-05-21 02:01:09 +00:00
parent 4c41b19e13
commit 2e99873634
44 changed files with 6295 additions and 346 deletions

View File

@@ -33,6 +33,10 @@ automation/bin/karakeep_api.py /api/health --raw-path --pretty
- `bin/pd_backup_postgres.sh` — creates gzipped `pg_dump` exports for the configured Postgres DB list (quote space-separated DB names in `.env`, e.g. `POSTGRES_DB_LIST="n8n paperless"`)
- `bin/pd_backup_sync_to_serenity.sh``rsync`s configured backup/config roots to Serenity
- `bin/run_pd_backups.sh` — wrapper that runs both steps in order
- `bin/pd_restore_verify_postgres.sh` — restores the latest dump for each configured database into a throwaway Postgres container and fails on SQL errors
- `bin/pd_restore_verify_appdata.sh` — stages one appdata/config tree back from Serenity and verifies required files exist
- `bin/run_pd_restore_verification.sh` — wrapper that runs both restore checks in order
- writes Prometheus textfile metrics for run status, duration, and last successful verification timestamp
These scripts are the live PD backup runner model.
@@ -49,6 +53,7 @@ Current documented live state:
- the PD → Serenity backup flow is the intended deployed model
- root cron on PD is the preferred scheduler
- first-run verification and the deployed status are tracked in `docs/operations/PD_BACKUP_DEPLOYMENT.md` and `docs/planning/TODO.md`
- quarterly restore verification is now expected via `bin/run_pd_restore_verification.sh`
## PD / TrueNAS deployment recommendation
@@ -93,6 +98,34 @@ Install in **root's crontab on PD**:
# END PD BACKUPS
```
## Recommended quarterly restore verification
Run from PD after a backup has completed:
```bash
cd /mnt/docker-ssd/docker/compose/automation
/usr/bin/bash bin/run_pd_restore_verification.sh
```
This validates:
- latest configured Postgres dumps can be restored cleanly
- a real config/appdata sample can be pulled back from Serenity
- expected files still exist in the restored copy
If node-exporter is configured with the textfile collector, the wrapper also publishes:
- `pd_restore_verification_success`
- `pd_restore_verification_last_run_timestamp_seconds`
- `pd_restore_verification_last_success_timestamp_seconds`
- `pd_restore_verification_duration_seconds`
Recommended root cron entry on PD:
```cron
30 3 1 */3 * cd /mnt/docker-ssd/docker/compose/automation && /usr/bin/bash bin/run_pd_restore_verification.sh >> /mnt/tank/docker/backups/pd-restore-verify.log 2>&1
```
Recommended first-run checklist:
```bash