147 lines
5.4 KiB
Markdown
147 lines
5.4 KiB
Markdown
# Automation helpers
|
|
|
|
This directory includes repo-side helpers for homelab operational tasks.
|
|
|
|
## KitchenOwl recipe import helper
|
|
|
|
- `bin/kitchenowl_recipe_import.py` — Doris-managed KitchenOwl recipe importer
|
|
- tries KitchenOwl's own scrape endpoint first
|
|
- falls back to direct page fetch + schema.org `Recipe` JSON-LD parsing
|
|
- normalizes ingredients/tags/timings into KitchenOwl's create-recipe payload
|
|
- supports dry-run by default and live create with `--create`
|
|
|
|
See `docs/operations/KITCHENOWL_RECIPE_IMPORT.md` for setup and usage.
|
|
|
|
## Karakeep API helper
|
|
|
|
- `bin/karakeep_api.py` — tiny Karakeep REST helper backed by the local encrypted vault entry
|
|
- reads `.secrets/karakeep_secrets.json.enc`
|
|
- uses `baseUrl` + `apiKey`
|
|
- targets `/api/v1/...` routes by default
|
|
- supports ad hoc GET/POST/PATCH/DELETE calls with query/body args
|
|
|
|
Examples:
|
|
|
|
```bash
|
|
automation/bin/karakeep_api.py bookmarks --query limit=5 --pretty
|
|
automation/bin/karakeep_api.py tags --pretty
|
|
automation/bin/karakeep_api.py /api/health --raw-path --pretty
|
|
```
|
|
|
|
## Backup scripts
|
|
|
|
- `bin/pd_backup_postgres.sh` — creates gzipped `pg_dump` exports for the configured Postgres DB list (quote space-separated DB names in `.env`, e.g. `POSTGRES_DB_LIST="gitea n8n paperless"`)
|
|
- `bin/pd_backup_sync_to_serenity.sh` — `rsync`s configured backup/config roots to Serenity
|
|
- `bin/run_pd_backups.sh` — wrapper that runs both steps in order
|
|
- `bin/pd_restore_verify_postgres.sh` — restores the latest dump for each configured database into a throwaway Postgres container and fails on SQL errors
|
|
- `bin/pd_restore_verify_appdata.sh` — stages one appdata/config tree back from Serenity and verifies required files exist
|
|
- `bin/run_pd_restore_verification.sh` — wrapper that runs both restore checks in order
|
|
- writes Prometheus textfile metrics for run status, duration, and last successful verification timestamp
|
|
|
|
These scripts are the live PD backup runner model.
|
|
|
|
Repo-side deployment prerequisites remain:
|
|
|
|
1. real values in `.env`
|
|
2. a writable dump destination on PD
|
|
3. SSH trust / key path for Serenity (`root@10.5.30.5` by default)
|
|
- recommended known_hosts path on PD: `/home/truenas_admin/.ssh/known_hosts`
|
|
4. a real scheduler on the live host
|
|
|
|
Current documented live state:
|
|
|
|
- the PD → Serenity backup flow is the intended deployed model
|
|
- root cron on PD is the preferred scheduler
|
|
- first-run verification and the deployed status are tracked in `docs/operations/PD_BACKUP_DEPLOYMENT.md` and `docs/planning/TODO.md`
|
|
- quarterly restore verification is now expected via `bin/run_pd_restore_verification.sh`
|
|
|
|
## PD / TrueNAS deployment recommendation
|
|
|
|
Use a plain cron job on PD, not n8n and not a custom systemd timer.
|
|
|
|
Reasons:
|
|
|
|
- survives n8n outages
|
|
- keeps the backup runner close to the compose/data host
|
|
- avoids extra appliance fights on TrueNAS SCALE
|
|
- matches the existing shell-first operational style on PD
|
|
|
|
Preferred mode on PD: **root cron**.
|
|
|
|
Why: the dump path under `/mnt/tank/...` and Docker access are typically cleaner from root on TrueNAS than from a limited operator account.
|
|
|
|
TrueNAS-specific notes:
|
|
|
|
- scripts use `/usr/bin/bash` explicitly
|
|
- default Docker path is `/usr/bin/docker`
|
|
- if run as root, scripts call Docker directly
|
|
- if run as a non-root user, default behavior is `sudo -n /usr/bin/docker ...`
|
|
- keep all live paths under `/mnt/...`; never rely on rootfs write locations
|
|
|
|
## Example live flow
|
|
|
|
```bash
|
|
cd /mnt/docker-ssd/docker/compose/automation
|
|
cp .env.example .env
|
|
chmod 600 .env
|
|
mkdir -p /mnt/tank/docker/backups/db-dumps
|
|
bin/run_pd_backups.sh
|
|
```
|
|
|
|
## Recommended PD cron block
|
|
|
|
Install in **root's crontab on PD**:
|
|
|
|
```cron
|
|
# BEGIN PD BACKUPS
|
|
15 2 * * * cd /mnt/docker-ssd/docker/compose/automation && /usr/bin/bash bin/run_pd_backups.sh >> /mnt/tank/docker/backups/pd-backups.log 2>&1
|
|
# END PD BACKUPS
|
|
```
|
|
|
|
## Recommended quarterly restore verification
|
|
|
|
Run from PD after a backup has completed:
|
|
|
|
```bash
|
|
cd /mnt/docker-ssd/docker/compose/automation
|
|
/usr/bin/bash bin/run_pd_restore_verification.sh
|
|
```
|
|
|
|
This validates:
|
|
|
|
- latest configured Postgres dumps can be restored cleanly
|
|
- a real config/appdata sample can be pulled back from Serenity
|
|
- expected files still exist in the restored copy
|
|
|
|
If node-exporter is configured with the textfile collector, the wrapper also publishes:
|
|
|
|
- `pd_restore_verification_success`
|
|
- `pd_restore_verification_last_run_timestamp_seconds`
|
|
- `pd_restore_verification_last_success_timestamp_seconds`
|
|
- `pd_restore_verification_duration_seconds`
|
|
|
|
Recommended root cron entry on PD:
|
|
|
|
```cron
|
|
30 3 1 */3 * cd /mnt/docker-ssd/docker/compose/automation && /usr/bin/bash bin/run_pd_restore_verification.sh >> /mnt/tank/docker/backups/pd-restore-verify.log 2>&1
|
|
```
|
|
|
|
Recommended first-run checklist:
|
|
|
|
```bash
|
|
cd /mnt/docker-ssd/docker/compose/automation
|
|
cp .env.example .env # if not already present
|
|
chmod 600 .env
|
|
mkdir -p /mnt/tank/docker/backups/db-dumps
|
|
/usr/bin/bash bin/run_pd_backups.sh
|
|
tail -100 /mnt/tank/docker/backups/pd-backups.log
|
|
```
|
|
|
|
## Safety notes
|
|
|
|
- DB dumps are the priority restore artifacts; do not rely only on raw volume copies.
|
|
- For Gitea specifically, keep the logical Postgres dump in `POSTGRES_DB_LIST`; a live rsynced Postgres volume alone is not the preferred recovery artifact.
|
|
- The sync script uses `rsync --delete` inside the destination backup root, so point it at a dedicated backup path.
|
|
- Keep `.env` and SSH material out of git.
|
|
- If cron runs under a non-root PD account, `sudo -n /usr/bin/docker` must work or the DB dump step will fail.
|