# Automation helpers

This directory includes repo-side helpers for homelab operational tasks.

## KitchenOwl recipe import helper

- `bin/kitchenowl_recipe_import.py` — Doris-managed KitchenOwl recipe importer
  - tries KitchenOwl's own scrape endpoint first
  - falls back to direct page fetch + schema.org `Recipe` JSON-LD parsing
  - normalizes ingredients/tags/timings into KitchenOwl's create-recipe payload
  - supports dry-run by default and live create with `--create`

See `docs/operations/KITCHENOWL_RECIPE_IMPORT.md` for setup and usage.

## Karakeep API helper

- `bin/karakeep_api.py` — tiny Karakeep REST helper backed by the local encrypted vault entry
  - reads `.secrets/karakeep_secrets.json.enc`
  - uses `baseUrl` + `apiKey`
  - targets `/api/v1/...` routes by default
  - supports ad hoc GET/POST/PATCH/DELETE calls with query/body args

Examples:

```bash
automation/bin/karakeep_api.py bookmarks --query limit=5 --pretty
automation/bin/karakeep_api.py tags --pretty
automation/bin/karakeep_api.py /api/health --raw-path --pretty
```

## Backup scripts

- `bin/pd_backup_postgres.sh` — creates gzipped `pg_dump` exports for the configured Postgres DB list (quote space-separated DB names in `.env`, e.g. `POSTGRES_DB_LIST="n8n paperless"`)
- `bin/pd_backup_sync_to_serenity.sh` — `rsync`s configured backup/config roots to Serenity
- `bin/run_pd_backups.sh` — wrapper that runs both steps in order
- `bin/pd_restore_verify_postgres.sh` — restores the latest dump for each configured database into a throwaway Postgres container and fails on SQL errors
- `bin/pd_restore_verify_appdata.sh` — stages one appdata/config tree back from Serenity and verifies required files exist
- `bin/run_pd_restore_verification.sh` — wrapper that runs both restore checks in order
  - writes Prometheus textfile metrics for run status, duration, and last successful verification timestamp

These scripts are the live PD backup runner model.

Repo-side deployment prerequisites remain:

1. real values in `.env`
2. a writable dump destination on PD
3. SSH trust / key path for Serenity (`root@10.5.30.5` by default)
   - recommended known_hosts path on PD: `/home/truenas_admin/.ssh/known_hosts`
4. a real scheduler on the live host

Current documented live state:

- the PD → Serenity backup flow is the intended deployed model
- root cron on PD is the preferred scheduler
- first-run verification and the deployed status are tracked in `docs/operations/PD_BACKUP_DEPLOYMENT.md` and `docs/planning/TODO.md`
- quarterly restore verification is now expected via `bin/run_pd_restore_verification.sh`

## PD / TrueNAS deployment recommendation

Use a plain cron job on PD, not n8n and not a custom systemd timer.

Reasons:

- survives n8n outages
- keeps the backup runner close to the compose/data host
- avoids extra appliance fights on TrueNAS SCALE
- matches the existing shell-first operational style on PD

Preferred mode on PD: **root cron**.

Why: the dump path under `/mnt/tank/...` and Docker access are typically cleaner from root on TrueNAS than from a limited operator account.

TrueNAS-specific notes:

- scripts use `/usr/bin/bash` explicitly
- default Docker path is `/usr/bin/docker`
- if run as root, scripts call Docker directly
- if run as a non-root user, default behavior is `sudo -n /usr/bin/docker ...`
- keep all live paths under `/mnt/...`; never rely on rootfs write locations

## Example live flow

```bash
cd /mnt/docker-ssd/docker/compose/automation
cp .env.example .env
chmod 600 .env
mkdir -p /mnt/tank/docker/backups/db-dumps
bin/run_pd_backups.sh
```

## Recommended PD cron block

Install in **root's crontab on PD**:

```cron
# BEGIN PD BACKUPS
15 2 * * * cd /mnt/docker-ssd/docker/compose/automation && /usr/bin/bash bin/run_pd_backups.sh >> /mnt/tank/docker/backups/pd-backups.log 2>&1
# END PD BACKUPS
```

## Recommended quarterly restore verification

Run from PD after a backup has completed:

```bash
cd /mnt/docker-ssd/docker/compose/automation
/usr/bin/bash bin/run_pd_restore_verification.sh
```

This validates:

- latest configured Postgres dumps can be restored cleanly
- a real config/appdata sample can be pulled back from Serenity
- expected files still exist in the restored copy

If node-exporter is configured with the textfile collector, the wrapper also publishes:

- `pd_restore_verification_success`
- `pd_restore_verification_last_run_timestamp_seconds`
- `pd_restore_verification_last_success_timestamp_seconds`
- `pd_restore_verification_duration_seconds`

Recommended root cron entry on PD:

```cron
30 3 1 */3 * cd /mnt/docker-ssd/docker/compose/automation && /usr/bin/bash bin/run_pd_restore_verification.sh >> /mnt/tank/docker/backups/pd-restore-verify.log 2>&1
```

Recommended first-run checklist:

```bash
cd /mnt/docker-ssd/docker/compose/automation
cp .env.example .env   # if not already present
chmod 600 .env
mkdir -p /mnt/tank/docker/backups/db-dumps
/usr/bin/bash bin/run_pd_backups.sh
tail -100 /mnt/tank/docker/backups/pd-backups.log
```

## Safety notes

- DB dumps are the priority restore artifacts; do not rely only on raw volume copies.
- The sync script uses `rsync --delete` inside the destination backup root, so point it at a dedicated backup path.
- Keep `.env` and SSH material out of git.
- If cron runs under a non-root PD account, `sudo -n /usr/bin/docker` must work or the DB dump step will fail.