bin/kitchenowl_recipe_import.py — Doris-managed KitchenOwl recipe importer
- tries KitchenOwl's own scrape endpoint first
- falls back to direct page fetch + schema.org Recipe JSON-LD parsing
- normalizes ingredients/tags/timings into KitchenOwl's create-recipe payload
- supports dry-run by default and live create with --create

See docs/operations/KITCHENOWL_RECIPE_IMPORT.md for setup and usage.

Karakeep API helper

bin/karakeep_api.py — tiny Karakeep REST helper backed by the local encrypted vault entry
- reads .secrets/karakeep_secrets.json.enc
- uses baseUrl + apiKey
- targets /api/v1/... routes by default
- supports ad hoc GET/POST/PATCH/DELETE calls with query/body args

Examples:

automation/bin/karakeep_api.py bookmarks --query limit=5 --pretty
automation/bin/karakeep_api.py tags --pretty
automation/bin/karakeep_api.py /api/health --raw-path --pretty

Backup scripts

bin/pd_backup_postgres.sh — creates gzipped pg_dump exports for the configured Postgres DB list (quote space-separated DB names in .env, e.g. POSTGRES_DB_LIST="gitea n8n paperless")
bin/pd_backup_sync_to_serenity.sh — rsyncs configured backup/config roots to Serenity
bin/run_pd_backups.sh — wrapper that runs both steps in order
bin/pd_restore_verify_postgres.sh — restores the latest dump for each configured database into a throwaway Postgres container and fails on SQL errors
bin/pd_restore_verify_appdata.sh — stages one appdata/config tree back from Serenity and verifies required files exist
bin/run_pd_restore_verification.sh — wrapper that runs both restore checks in order
- writes Prometheus textfile metrics for run status, duration, and last successful verification timestamp

These scripts are the live PD backup runner model.

Repo-side deployment prerequisites remain:

real values in .env
a writable dump destination on PD
SSH trust / key path for Serenity (root@10.5.30.5 by default)
- recommended known_hosts path on PD: /home/truenas_admin/.ssh/known_hosts
a real scheduler on the live host

Current documented live state:

the PD → Serenity backup flow is the intended deployed model
root cron on PD is the preferred scheduler
first-run verification and the deployed status are tracked in docs/operations/PD_BACKUP_DEPLOYMENT.md and docs/planning/TODO.md
quarterly restore verification is now expected via bin/run_pd_restore_verification.sh

PD / TrueNAS deployment recommendation

Use a plain cron job on PD, not n8n and not a custom systemd timer.

Reasons:

survives n8n outages
keeps the backup runner close to the compose/data host
avoids extra appliance fights on TrueNAS SCALE
matches the existing shell-first operational style on PD

Preferred mode on PD: root cron.

Why: the dump path under /mnt/tank/... and Docker access are typically cleaner from root on TrueNAS than from a limited operator account.

TrueNAS-specific notes:

scripts use /usr/bin/bash explicitly
default Docker path is /usr/bin/docker
if run as root, scripts call Docker directly
if run as a non-root user, default behavior is sudo -n /usr/bin/docker ...
keep all live paths under /mnt/...; never rely on rootfs write locations

Example live flow

cd /mnt/docker-ssd/docker/compose/automation
cp .env.example .env
chmod 600 .env
mkdir -p /mnt/tank/docker/backups/db-dumps
bin/run_pd_backups.sh

Recommended PD cron block

Install in root's crontab on PD:

# BEGIN PD BACKUPS
15 2 * * * cd /mnt/docker-ssd/docker/compose/automation && /usr/bin/bash bin/run_pd_backups.sh >> /mnt/tank/docker/backups/pd-backups.log 2>&1
# END PD BACKUPS

Recommended quarterly restore verification

Run from PD after a backup has completed:

cd /mnt/docker-ssd/docker/compose/automation
/usr/bin/bash bin/run_pd_restore_verification.sh

This validates:

latest configured Postgres dumps can be restored cleanly
a real config/appdata sample can be pulled back from Serenity
expected files still exist in the restored copy

If node-exporter is configured with the textfile collector, the wrapper also publishes:

pd_restore_verification_success
pd_restore_verification_last_run_timestamp_seconds
pd_restore_verification_last_success_timestamp_seconds
pd_restore_verification_duration_seconds

Recommended root cron entry on PD:

30 3 1 */3 * cd /mnt/docker-ssd/docker/compose/automation && /usr/bin/bash bin/run_pd_restore_verification.sh >> /mnt/tank/docker/backups/pd-restore-verify.log 2>&1

Recommended first-run checklist:

cd /mnt/docker-ssd/docker/compose/automation
cp .env.example .env   # if not already present
chmod 600 .env
mkdir -p /mnt/tank/docker/backups/db-dumps
/usr/bin/bash bin/run_pd_backups.sh
tail -100 /mnt/tank/docker/backups/pd-backups.log

Safety notes

DB dumps are the priority restore artifacts; do not rely only on raw volume copies.
For Gitea specifically, keep the logical Postgres dump in POSTGRES_DB_LIST; a live rsynced Postgres volume alone is not the preferred recovery artifact.
The sync script uses rsync --delete inside the destination backup root, so point it at a dedicated backup path.
Keep .env and SSH material out of git.
If cron runs under a non-root PD account, sudo -n /usr/bin/docker must work or the DB dump step will fail.