Add n8n workflows 01-08 (with upgrades), 14-16; planning docs; .gitignore

2026-05-09 13:20:11 -05:00
parent 9f6144e28e
commit 7ec40ef437
22 changed files with 7859 additions and 10 deletions
--- a/docs/planning/DEPLOY_KIMA_HUB.md
+++ b/docs/planning/DEPLOY_KIMA_HUB.md
@@ -0,0 +1,91 @@
+# Kima-Hub Deployment Plan
+
+**Host:** PlausibleDeniability (PD)  
+**Port:** 3333  
+**Stack dir:** `/mnt/docker-ssd/docker/compose/media-extra/`  
+**Status:** Pending
+
+## What It Is
+
+Kima-Hub is a self-hosted music streaming platform — think personal Spotify. Point it at your music library and it handles cataloging, artist discovery, AI-powered playlists, podcast subscriptions, and seamless integration with Lidarr and Audiobookshelf.
+
+## Pre-Deploy Checklist
+
+- [ ] Confirm music library path on Unraid NFS (`/mnt/unraid/data/media/music` or similar)
+- [ ] Create appdata directory: `sudo mkdir -p /mnt/tank/docker/appdata/kima-hub`
+- [ ] Create stack directory: `sudo mkdir -p /mnt/docker-ssd/docker/compose/media-extra`
+
+## docker-compose.yaml
+
+```yaml
+services:
+  kima-hub:
+    image: chevron7locked/kima:latest
+    container_name: kima-hub
+    restart: unless-stopped
+    ports:
+      - "3333:3030"
+    volumes:
+      - /mnt/tank/docker/appdata/kima-hub:/data
+      - /mnt/unraid/data/media/music:/music:ro
+    environment:
+      - TZ=${TZ}
+    healthcheck:
+      test: ["CMD-SHELL", "cat /proc/net/tcp6 | grep -q 0BD6 || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+    networks:
+      - pangolin
+
+networks:
+  pangolin:
+    external: true
+```
+
+> **Note:** Internal port is `3030`, mapped to `3333` on the host. Hex `0BD6` = port 3030 for healthcheck.  
+> **Music path:** Adjust `/mnt/unraid/data/media/music` to match your actual Unraid music library path.  
+> **Read-only mount:** Music library is mounted `:ro` — Kima-Hub should not need write access to source files.
+
+## .env.example
+
+```env
+TZ=America/New_York
+```
+
+## Deploy
+
+```bash
+cd /mnt/docker-ssd/docker/compose/media-extra
+sudo docker compose --env-file .env config    # validate
+sudo docker compose --env-file .env up -d
+sudo docker logs kima-hub --tail 30
+```
+
+## Post-Deploy
+
+1. Open `http://10.5.1.6:3333` and create your account
+2. Point Kima at your music directory (`/music`)
+3. In Lidarr: Settings → Connect → add Kima-Hub webhook
+4. In Audiobookshelf: verify Kima can reach it at `http://10.5.1.6:13358`
+5. Add Pangolin resource: `kima.paccoco.com` → port 3333
+
+## Pangolin Resource
+
+Add in Pangolin dashboard:
+- **Subdomain:** `kima.paccoco.com`
+- **Target:** `http://10.5.1.6:3333`
+- **Auth:** Enable (personal use only)
+
+## Storage Decision
+
+| Path | Tier | Reason |
+|------|------|--------|
+| `/mnt/tank/docker/appdata/kima-hub` | tank | Config, DB, playlists — not write-heavy |
+| `/mnt/unraid/data/media/music` | Unraid NFS (read-only) | Source library stays on Serenity |
+
+## Notes
+
+- NFS mount must be up before starting (`sudo bash /mnt/tank/docker/scripts/mount-unraid-nfs.sh`)
+- If you transcode on the fly, CPU load on PD will spike; Kima-Hub is CPU-bound for transcoding
+- Kima-Hub has no GPU acceleration — pure CPU transcoding
--- a/docs/planning/DEPLOY_PIHOLE.md
+++ b/docs/planning/DEPLOY_PIHOLE.md
@@ -0,0 +1,527 @@
+# Pi-hole Deployment Plan
+
+**Status:** Pending  
+**Architecture:** 3-node HA cluster with Keepalived VIP, Unbound recursive DNS, Nebula Sync
+
+---
+
+## Architecture Overview
+
+```
+[All LAN clients]
+       ↓  DNS: 10.5.1.53  (Virtual IP — floats to whichever node is MASTER)
+       ↓
+ ┌─────┴──────────────────────────────────┐
+ │           Keepalived VIP               │
+ │           10.5.1.53/24                 │
+ └──────┬──────────────┬──────────────────┘
+        │              │                  │
+   PD (MASTER)   Serenity (BACKUP1)  RPi4 (BACKUP2)
+   10.5.1.6       10.5.1.5           10.5.1.X
+   priority 100   priority 90        priority 80
+   Pi-hole :53    Pi-hole :53        Pi-hole :53
+   Unbound :5335  Unbound :5335      Unbound :5335
+        │              │                  │
+        └──────────────┴──────────────────┘
+                       ↓
+              [Root DNS servers]
+              (recursive — no third-party upstream)
+```
+
+- **Keepalived** floats the VIP `10.5.1.53` to whichever node is alive with the highest priority
+- **Unbound** on each node acts as the local recursive resolver (replaces Cloudflare/Quad9)
+- **Pi-hole** on each node forwards upstream queries to its local Unbound (`127.0.0.1#5335`)
+- **Nebula Sync** (running on PD) syncs all Pi-hole configs from primary → replicas daily
+
+All nodes use `network_mode: host` for Pi-hole, Unbound, and Keepalived — this is required for Keepalived to manipulate the VIP and for Pi-hole to bind cleanly to port 53.
+
+---
+
+## Pre-Deploy: Network Planning
+
+1. **Pick a free static IP for the VIP** — `10.5.1.53` is recommended (memorable for DNS). Reserve it in UniFi so DHCP never assigns it.
+2. **Find the LAN interface name** on each node:
+   ```bash
+   # PD / RPi4:
+   ip link show
+   # Look for the interface with 10.5.1.x address — likely eth0, eno1, enp3s0, etc.
+   ```
+3. **Note RPi4 IP** — fill in `10.5.1.X` throughout this doc once known. Reserve it as static in UniFi.
+4. **Verify port 53 is free on PD:**
+   ```bash
+   sudo ss -tulnp | grep ':53'
+   # Should show only 127.0.0.1:53 (TrueNAS loopback resolver). 
+   # If it shows 0.0.0.0:53, see Gotchas section.
+   ```
+
+---
+
+## Unbound Config (shared across all nodes)
+
+Each node runs Unbound as a Docker container on the host network, listening on `127.0.0.1:5335`.
+
+**Save this as `unbound.conf` in each node's appdata path** (see per-node sections below):
+
+```ini
+server:
+    verbosity: 1
+    interface: 127.0.0.1
+    port: 5335
+    do-ip4: yes
+    do-udp: yes
+    do-tcp: yes
+    do-ip6: no
+
+    # DNSSEC
+    auto-trust-anchor-file: "/opt/unbound/etc/unbound/var/root.key"
+    val-permissive-mode: no
+
+    # Performance
+    num-threads: 2
+    cache-min-ttl: 300
+    cache-max-ttl: 86400
+    prefetch: yes
+    prefetch-key: yes
+    minimal-responses: yes
+
+    # Privacy
+    qname-minimisation: yes
+    hide-identity: yes
+    hide-version: yes
+
+    # Access control
+    access-control: 127.0.0.1/32 allow
+    access-control: 0.0.0.0/0 refuse
+
+    # Root hints — download updated copy periodically
+    root-hints: "/opt/unbound/etc/unbound/var/root.hints"
+```
+
+**Download root hints** (run once on each node, into the appdata path):
+```bash
+curl -o /path/to/appdata/unbound/var/root.hints https://www.internic.net/domain/named.cache
+mkdir -p /path/to/appdata/unbound/var
+# Unbound will auto-manage root.key for DNSSEC
+```
+
+---
+
+## Node 1: PlausibleDeniability (MASTER)
+
+**Stack dir:** `/mnt/docker-ssd/docker/compose/pihole/`  
+**Web UI port:** `8953`
+
+### Pre-Deploy
+
+```bash
+sudo mkdir -p /mnt/tank/docker/appdata/pihole-primary/etc-pihole
+sudo mkdir -p /mnt/tank/docker/appdata/unbound-primary/var
+sudo mkdir -p /mnt/docker-ssd/docker/compose/pihole
+# Download root hints
+sudo curl -o /mnt/tank/docker/appdata/unbound-primary/var/root.hints https://www.internic.net/domain/named.cache
+```
+
+### docker-compose.yaml
+
+```yaml
+services:
+  unbound:
+    image: mvance/unbound:latest
+    container_name: unbound
+    network_mode: host
+    restart: unless-stopped
+    volumes:
+      - /mnt/tank/docker/appdata/unbound-primary:/opt/unbound/etc/unbound/
+    healthcheck:
+      test: ["CMD-SHELL", "drill @127.0.0.1 -p 5335 cloudflare.com || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+
+  pihole:
+    image: pihole/pihole:latest
+    container_name: pihole
+    network_mode: host
+    restart: unless-stopped
+    depends_on:
+      unbound:
+        condition: service_healthy
+    environment:
+      TZ: ${TZ}
+      FTLCONF_webserver_api_password: ${PIHOLE_PASSWORD}
+      FTLCONF_dns_listeningMode: "LOCAL"
+      FTLCONF_dns_upstreams: "127.0.0.1#5335"
+      FTLCONF_webserver_port: "8953"
+    volumes:
+      - /mnt/tank/docker/appdata/pihole-primary/etc-pihole:/etc/pihole
+    cap_add:
+      - SYS_NICE
+
+  keepalived:
+    image: osixia/keepalived:2.0.20
+    container_name: keepalived
+    network_mode: host
+    restart: unless-stopped
+    cap_add:
+      - NET_ADMIN
+      - NET_BROADCAST
+      - NET_RAW
+    volumes:
+      - ./keepalived.conf:/container/environment/01-custom/keepalived.conf
+    environment:
+      KEEPALIVED_VIRTUAL_IPS: "#PYTHON2BASH:['10.5.1.53/24']"
+```
+
+### keepalived.conf (PD — MASTER)
+
+```
+global_defs {
+    router_id pihole_pd
+}
+
+vrrp_instance VI_DNS {
+    state MASTER
+    interface ${LAN_INTERFACE}
+    virtual_router_id 53
+    priority 100
+    advert_int 1
+    nopreempt
+
+    authentication {
+        auth_type PASS
+        auth_pass ${KEEPALIVED_PASSWORD}
+    }
+
+    virtual_ipaddress {
+        10.5.1.53/24
+    }
+}
+```
+> Replace `${LAN_INTERFACE}` with your actual interface (e.g. `eno1`). Replace `${KEEPALIVED_PASSWORD}` with a shared secret used on all three nodes.
+
+### .env.example
+
+```env
+TZ=America/New_York
+PIHOLE_PASSWORD=CHANGE_ME
+KEEPALIVED_PASSWORD=CHANGE_ME
+LAN_INTERFACE=eno1
+```
+
+### Deploy
+
+```bash
+cd /mnt/docker-ssd/docker/compose/pihole
+# Copy unbound.conf to appdata
+sudo cp unbound.conf /mnt/tank/docker/appdata/unbound-primary/
+sudo docker compose --env-file .env config
+sudo docker compose --env-file .env up -d
+sudo docker logs pihole --tail 20
+sudo docker logs unbound --tail 20
+```
+
+---
+
+## Node 2: Serenity (BACKUP1)
+
+Serenity runs Unraid. Deploy via Docker Compose in Unraid's **User Scripts** or the **Compose Manager** plugin.
+
+**Appdata:** `/mnt/user/appdata/`  
+**Web UI port:** `8954`
+
+### docker-compose.yaml
+
+Same structure as PD but with different ports/paths and priority:
+
+```yaml
+services:
+  unbound:
+    image: mvance/unbound:latest
+    container_name: unbound-pihole
+    network_mode: host
+    restart: unless-stopped
+    volumes:
+      - /mnt/user/appdata/unbound-pihole:/opt/unbound/etc/unbound/
+    healthcheck:
+      test: ["CMD-SHELL", "drill @127.0.0.1 -p 5335 cloudflare.com || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+
+  pihole:
+    image: pihole/pihole:latest
+    container_name: pihole-secondary
+    network_mode: host
+    restart: unless-stopped
+    depends_on:
+      unbound-pihole:
+        condition: service_healthy
+    environment:
+      TZ: ${TZ}
+      FTLCONF_webserver_api_password: ${PIHOLE_PASSWORD}
+      FTLCONF_dns_listeningMode: "LOCAL"
+      FTLCONF_dns_upstreams: "127.0.0.1#5335"
+      FTLCONF_webserver_port: "8954"
+    volumes:
+      - /mnt/user/appdata/pihole-secondary/etc-pihole:/etc/pihole
+    cap_add:
+      - SYS_NICE
+
+  keepalived:
+    image: osixia/keepalived:2.0.20
+    container_name: keepalived-pihole
+    network_mode: host
+    restart: unless-stopped
+    cap_add:
+      - NET_ADMIN
+      - NET_BROADCAST
+      - NET_RAW
+    volumes:
+      - ./keepalived.conf:/container/environment/01-custom/keepalived.conf
+```
+
+### keepalived.conf (Serenity — BACKUP1)
+
+```
+global_defs {
+    router_id pihole_serenity
+}
+
+vrrp_instance VI_DNS {
+    state BACKUP
+    interface eth0
+    virtual_router_id 53
+    priority 90
+    advert_int 1
+
+    authentication {
+        auth_type PASS
+        auth_pass SAME_PASSWORD_AS_PRIMARY
+    }
+
+    virtual_ipaddress {
+        10.5.1.53/24
+    }
+}
+```
+
+---
+
+## Node 3: Raspberry Pi 4 (BACKUP2)
+
+### Install Docker (if not already installed)
+
+```bash
+curl -fsSL https://get.docker.com | sh
+sudo usermod -aG docker $USER
+```
+
+### Pre-Deploy
+
+```bash
+mkdir -p ~/pihole/etc-pihole
+mkdir -p ~/pihole/unbound/var
+curl -o ~/pihole/unbound/var/root.hints https://www.internic.net/domain/named.cache
+sudo apt install keepalived -y
+```
+
+### docker-compose.yaml
+
+```yaml
+services:
+  unbound:
+    image: mvance/unbound:latest
+    container_name: unbound
+    network_mode: host
+    restart: unless-stopped
+    volumes:
+      - ~/pihole/unbound:/opt/unbound/etc/unbound/
+    healthcheck:
+      test: ["CMD-SHELL", "drill @127.0.0.1 -p 5335 cloudflare.com || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+
+  pihole:
+    image: pihole/pihole:latest
+    container_name: pihole
+    network_mode: host
+    restart: unless-stopped
+    depends_on:
+      unbound:
+        condition: service_healthy
+    environment:
+      TZ: ${TZ}
+      FTLCONF_webserver_api_password: ${PIHOLE_PASSWORD}
+      FTLCONF_dns_listeningMode: "LOCAL"
+      FTLCONF_dns_upstreams: "127.0.0.1#5335"
+      FTLCONF_webserver_port: "8955"
+    volumes:
+      - ~/pihole/etc-pihole:/etc/pihole
+    cap_add:
+      - SYS_NICE
+```
+
+### keepalived (native on RPi4)
+
+```bash
+sudo nano /etc/keepalived/keepalived.conf
+```
+
+```
+global_defs {
+    router_id pihole_rpi4
+}
+
+vrrp_instance VI_DNS {
+    state BACKUP
+    interface eth0
+    virtual_router_id 53
+    priority 80
+    advert_int 1
+
+    authentication {
+        auth_type PASS
+        auth_pass SAME_PASSWORD_AS_PRIMARY
+    }
+
+    virtual_ipaddress {
+        10.5.1.53/24
+    }
+}
+```
+
+```bash
+sudo systemctl enable keepalived
+sudo systemctl start keepalived
+```
+
+---
+
+## Nebula Sync (on PD)
+
+[Nebula Sync](https://github.com/lovelaze/nebula-sync) syncs Pi-hole v6 configuration (gravity, blocklists, allowlists, custom DNS) from the primary to both replicas.
+
+**Add to PD's pihole docker-compose.yaml:**
+
+```yaml
+  nebula-sync:
+    image: lovelaze/nebula-sync:latest
+    container_name: nebula-sync
+    restart: unless-stopped
+    environment:
+      PRIMARY: "http://127.0.0.1:8953|${PIHOLE_PASSWORD}"
+      REPLICAS: "http://10.5.1.5:8954|${PIHOLE_PASSWORD_SERENITY},http://10.5.1.X:8955|${PIHOLE_PASSWORD_RPI}"
+      RUN_GRAVITY: "true"
+      FULL_SYNC: "false"
+      CRON: "0 3 * * *"
+    networks:
+      - default
+```
+
+> - `PRIMARY` — your PD Pi-hole (note: uses `127.0.0.1` since nebula-sync is on the same host; port 8953 is the web UI, also the API port in Pi-hole v6)
+> - `REPLICAS` — comma-separated list of secondary instances
+> - `CRON` — syncs daily at 3 AM; force a manual sync with `docker exec nebula-sync nebula-sync run`
+> - Each Pi-hole instance can (and should) have a **different** password — nebula-sync handles auth per-instance
+> - Add `PIHOLE_PASSWORD_SERENITY` and `PIHOLE_PASSWORD_RPI` to your `.env` file
+
+### What Nebula Sync copies
+- Gravity database (blocklists, allowlists, group assignments)
+- Custom DNS/CNAME records
+- Client groups
+- Does **not** sync per-instance settings (web password, interface binding — these stay local)
+
+---
+
+## Router Configuration (UniFi)
+
+Point **only the VIP** as DNS:
+
+1. UniFi Network → Settings → Networks → **LAN** → DHCP Name Server: **Manual**
+2. **DNS Server 1:** `10.5.1.53` (VIP — floats between nodes)
+3. **DNS Server 2:** *(leave blank or set a public fallback like `1.1.1.1` only if you want internet DNS as last resort)*
+4. Save and renew DHCP leases
+
+> **Why single VIP instead of listing all three IPs?**  
+> Clients round-robin across all listed DNS servers — so some queries bypass Pi-hole entirely. With Keepalived the VIP always points to exactly one live Pi-hole node, ensuring 100% of queries are filtered.
+
+---
+
+## Post-Deploy
+
+### Verify VIP is working
+
+```bash
+# Check which node holds the VIP
+ip addr show | grep 10.5.1.53   # Run on each node — only MASTER shows it
+
+# Test DNS through VIP
+nslookup google.com 10.5.1.53
+nslookup doubleclick.net 10.5.1.53   # Should return 0.0.0.0 (blocked)
+```
+
+### Test failover
+
+```bash
+# On PD, bring down keepalived
+sudo docker stop keepalived
+# Wait ~3 seconds, then check from another machine:
+nslookup google.com 10.5.1.53   # Should still work — Serenity took over VIP
+# Restore
+sudo docker start keepalived
+```
+
+### Add blocklists (on primary — Nebula Sync propagates to replicas)
+
+| List | URL |
+|------|-----|
+| Steven Black (unified) | `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts` |
+| OISD Basic | `https://basic.oisd.nl/` |
+| Hagezi Pro | `https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/pro.txt` |
+
+After adding: **Tools → Update Gravity** on PD, then run Nebula Sync to push to replicas.
+
+### Local DNS records
+
+Add in Pi-hole (primary) → Local DNS → DNS Records:
+
+| Domain | IP |
+|--------|----|
+| pd.lan | 10.5.1.6 |
+| serenity.lan | 10.5.1.5 |
+| nomad.lan | 10.5.1.16 |
+| rocinante.lan | 10.5.1.112 |
+| rpi4.lan | 10.5.1.X |
+
+Nebula Sync will propagate these to all replicas.
+
+### Pangolin for admin UIs (optional)
+
+| Subdomain | Target |
+|-----------|--------|
+| `pihole.paccoco.com` | `http://10.5.1.6:8953` (primary) |
+| `pihole2.paccoco.com` | `http://10.5.1.5:8954` (Serenity) |
+
+Enable Pangolin auth — Pi-hole admin should not be publicly accessible.
+
+---
+
+## Storage Decisions
+
+| Node | Path | Tier | Reason |
+|------|------|------|--------|
+| PD | `/mnt/tank/docker/appdata/pihole-primary` | tank | Low write frequency |
+| PD | `/mnt/tank/docker/appdata/unbound-primary` | tank | Config + root hints only |
+| Serenity | `/mnt/user/appdata/pihole-secondary` | Unraid array | Low write frequency |
+| RPi4 | `~/pihole/` | SD card / SSD | Dedicated Pi, no other concerns |
+
+---
+
+## Known Gotchas
+
+- **Port 53 conflict on PD:** TrueNAS binds to `127.0.0.1:53`. With `network_mode: host`, Pi-hole will try to bind to all interfaces. Set `FTLCONF_dns_listeningMode: LOCAL` so it only listens on the actual LAN interface, not loopback.
+- **keepalived interface name:** Must match the actual NIC name on each host. Get it with `ip link show`.
+- **`virtual_router_id` must be unique** per VRRP group on your LAN. `53` is a good mnemonic. Confirm no other keepalived instance on your LAN uses the same ID.
+- **Pi-hole v6 API port:** In v6, the web UI and API both run on the `FTLCONF_webserver_port`. Nebula Sync and the web browser use the same port.
+- **nopreempt on MASTER:** The `nopreempt` option in PD's keepalived.conf prevents PD from immediately reclaiming the VIP when it comes back up after a failure. Remove it if you want PD to always retake MASTER on recovery.
+- **Unbound cold start:** Unbound takes a few seconds to start and validate DNSSEC. The `depends_on: condition: service_healthy` in the Pi-hole compose ensures Pi-hole waits for Unbound before starting.
--- a/docs/planning/N8N_WORKFLOW_IDEAS.md
+++ b/docs/planning/N8N_WORKFLOW_IDEAS.md
@@ -0,0 +1,186 @@
+# n8n Workflow Ideas & Upgrade Proposals
+
+Generated: 2026-05-09
+
+---
+
+## Upgrades to Existing Workflows
+
+### 01 — Grafana Alert → AI → Gotify
+**Current state:** Webhook → parse → LiteLLM summarize → Gotify  
+**Suggested upgrades:**
+- **Alert deduplication:** Store last-seen alert fingerprint in Postgres. If the same alert fires again within 30 min, suppress the Gotify push (you're already awake, you've seen it).
+- **Recovery notifications:** Grafana sends both `firing` and `resolved` states. Add a branch so resolved alerts get a `✅ RESOLVED` message with a distinct priority (1 = low).
+- **Auto-acknowledge low-priority alerts:** If priority < 3 and it's between midnight and 7am, skip Gotify and log to Postgres only.
+
+---
+
+### 04 — Whisper Audio Transcription
+**Current state:** Returns transcript inline, no Gotify notification  
+**Suggested upgrades:**
+- **Add completion notification:** When a long transcription finishes, push to Gotify (Whisper Transcriptions app) so you know it's done without polling.
+- **Save transcript to Paperless:** After transcription, POST the text to Paperless-NGX as a new document with auto-tagging (`transcription`, `audio`, date).
+- **Save to Qdrant:** After transcription + summary, auto-ingest into RAG so all your audio content is searchable.
+
+---
+
+### 05 — Paperless → RAG Ingest
+**Current state:** Webhook → fetch → chunk → embed → Qdrant → Gotify  
+**Suggested upgrades:**
+- **Retry on failure:** If Qdrant upsert fails, retry up to 3 times with exponential backoff before erroring out.
+- **Collection routing:** Add metadata tagging so documents from different Paperless tags go into different Qdrant collections (e.g., `knowledge_base` vs `receipts` vs `medical`).
+
+---
+
+### 08 — Class Recording → Transcribe → RAG
+**Current state:** Webhook → Whisper → AI notes → RAG → Gotify  
+**Suggested upgrades:**
+- **Export notes to Paperless:** After extracting key concepts, create a Paperless document for the class notes (title = class + lecture, tags = class name, `lecture-notes`).
+- **Weekly class digest:** Scheduled workflow that queries RAG for all chunks from the past week, grouped by class, and sends a consolidated study summary to Gotify every Sunday night.
+- **Quiz generation:** Add an optional webhook endpoint that takes a class name + RAG query and generates 5 practice questions from the lecture material via LiteLLM.
+
+---
+
+## New Workflow Proposals
+
+### 09 — Immich → AI Event Tagger
+**Trigger:** Immich post-upload webhook (or scheduled poll via Immich API)  
+**What it does:**
+1. Fetch newly uploaded photos from Immich API
+2. Send image to a vision-capable model (gemma3:27b on Rocinante via LiteLLM)
+3. AI generates descriptive tags, detects events (birthday, trip, holiday), extracts visible text
+4. Write tags back to Immich via API
+5. Notify via Gotify (Immich app)
+
+**Why:** Immich's built-in ML does faces, not semantic tagging. This adds searchable event context automatically.
+
+---
+
+### 10 — Home Assistant → Morning Briefing
+**Trigger:** Schedule (e.g. 7:00 AM daily)  
+**What it does:**
+1. Fetch current weather from Home Assistant or weather API
+2. Query Grafana/Prometheus for overnight anomalies (high CPU, disk warnings)
+3. Check Pi-hole stats (queries blocked, top blocked domain)
+4. Check Paperless for any unread/untagged documents
+5. Summarize via LiteLLM into a 5-bullet morning briefing
+6. Push to Gotify (Morning Briefing app) and optionally to Home Assistant notification
+
+**Why:** Single daily digest so you know what happened overnight without checking 4 different dashboards.
+
+---
+
+### 11 — Uptime Kuma → Smart Alert Escalation
+**Trigger:** Uptime Kuma webhook on status change  
+**What it does:**
+1. Receive down/recovery event from Uptime Kuma
+2. Check Prometheus to confirm actual service health (avoids false positives)
+3. If confirmed down: check how long it's been down
+   - < 2 min: log only (transient blip)
+   - 2–10 min: Gotify push (medium priority)
+   - > 10 min: Gotify push (high priority) + attempt auto-restart via `docker restart` API call to PD
+4. On recovery: send resolution notification with total downtime
+
+**Why:** Uptime Kuma already sends webhooks but has no intelligence. This adds confirmation, suppression of transient blips, and auto-remediation for stuck containers.
+
+---
+
+### 12 — Pi-hole Stats Digest
+**Trigger:** Schedule (daily at 8 AM)  
+**What it does:**
+1. Query Pi-hole API on PD (`http://10.5.1.6:8953/api/stats/summary`)
+2. Query secondary Pi-hole on Serenity
+3. LiteLLM summarizes notable blocked domains and compares primary vs secondary query load
+4. Push digest to Gotify (Pi-hole app) with 24h block count, top blocked domains, and any unusual spikes
+
+**Why:** Passive DNS visibility — know if something on your network is phoning home unexpectedly.
+
+---
+
+### 13 — Lidarr → Kima-Hub New Music Notifier
+**Trigger:** Lidarr webhook on album download complete  
+**What it does:**
+1. Receive download event from Lidarr
+2. Fetch album art and metadata from MusicBrainz or Lidarr API
+3. Format a rich notification: artist, album, year, genre
+4. Push to Gotify (Music app) with album art link
+5. Optionally trigger Kima-Hub library rescan via API
+
+**Why:** Know when new music lands without watching Lidarr activity logs.
+
+---
+
+### 14 — Paperless Inbox Reminder
+**Trigger:** Schedule (every Monday 9 AM)  
+**What it does:**
+1. Query Paperless API for documents with no tags or in the inbox
+2. If count > 0: push reminder to Gotify listing the oldest unprocessed documents
+3. If inbox is clear: send a ✅ confirmation
+
+**Why:** It's easy to forget to process scanned documents. This ensures nothing sits in the inbox untagged for weeks.
+
+---
+
+### 15 — n8n Self-Health Monitor
+**Trigger:** Schedule (every 15 min)  
+**What it does:**
+1. Check n8n's own execution history via API — look for failed executions in the past hour
+2. If failures found: fetch the error message from n8n API
+3. Push a "Workflow Failed" notification to Gotify (n8n Health app) with the workflow name and error summary
+4. Skip if the failure is the same as the last-notified one (dedup via Postgres)
+
+**Why:** n8n doesn't natively alert you when a workflow silently fails. This closes that blind spot.
+
+---
+
+### 16 — NFS Mount Watchdog + Auto-Heal
+**Trigger:** Schedule (every 5 min)  
+**What it does:**
+1. SSH into PD (10.5.1.6) and probe each expected NFS mount path with `mountpoint -q`:
+   - `/mnt/unraid/data/media` (Plex, Audiobookshelf)
+   - `/mnt/unraid/data/photos` (Immich)
+   - Any other mounts defined in the mount script
+2. If all mounts are healthy: silent exit
+3. If any mount is missing:
+   a. Run the NFS mount script: `sudo bash /mnt/tank/docker/scripts/mount-unraid-nfs.sh`
+   b. Wait 10 seconds, re-probe to confirm mounts came up
+   c. If mounts confirmed: restart only the affected containers:
+      - Media mount missing → `sudo docker restart plex audiobookshelf`
+      - Photos mount missing → `sudo docker restart immich-server immich-microservices immich-machine-learning`
+   d. If mounts still missing after script: escalate to Gotify (high priority — NFS down, manual intervention needed)
+4. Push result to Gotify (Infra app):
+   - Auto-healed: priority 5, list which mounts were remounted and which containers restarted
+   - Escalation: priority 9, list which paths are still unavailable
+
+**Implementation notes:**
+- Uses n8n SSH node connecting to `10.5.1.6` with an SSH credential (key-based auth from n8n container)
+- SSH node runs a single compound command: `mountpoint -q /mnt/unraid/data/media && echo MEDIA_OK || echo MEDIA_MISSING` etc.
+- Parse output in a Code node to determine which mounts failed
+- Use separate SSH nodes for: (1) running the mount script, (2) restarting containers — keeps the flow readable
+- `continueOnFail: true` on the mount script SSH node so a TrueNAS-unreachable scenario still escalates instead of dying silently
+- Add a Postgres dedup table (`nfs_heal_log`) so repeated failures within 30 min only send one Gotify push, but every successful auto-heal always notifies
+
+**Why:** TrueNAS reboots (updates, power blips) drop NFS mounts and crash-loop dependent containers. This closes the gap between "Serenity rebooted at 3 AM" and "you notice Immich is broken at noon."
+
+---
+
+## Suggested Gotify App / Channel Structure
+
+Based on all workflows above, here's the full recommended channel layout:
+
+| App Name | Workflows |
+|----------|-----------|
+| Grafana Alerts | 01 |
+| Paperless | 03 |
+| Paperless RAG | 05 |
+| RSS Feed | 06 |
+| Git Commits | 07 |
+| Class Recordings | 08 |
+| Immich | 09 (new) |
+| Morning Briefing | 10 (new) |
+| Uptime / Infra | 11, 16 (new) |
+| Pi-hole | 12 (new) |
+| Music | 13 (new) |
+| Reminders | 14 (new) |
+| n8n Health | 15 (new) |
+| Whisper | 04 (after upgrade) |
--- a/docs/planning/TODO.md
+++ b/docs/planning/TODO.md
@@ -5,7 +5,9 @@
 - [ ] Regenerate N.O.M.A.D. Newt secret (was exposed in chat)

 ## Infrastructure
- [ ] Deploy Pi-hole ×2 (primary on PD + backup on Serenity or Pi4)
+- [ ] Deploy Pi-hole 3-node HA cluster (PD + Serenity + RPi4) — see docs/planning/DEPLOY_PIHOLE.md
+  - RPi4 IP still TBD — fill in before deploying
+- [ ] Deploy Kima-Hub — see docs/planning/DEPLOY_KIMA_HUB.md
 - [ ] Set up off-site backup for vital data (appdata, databases, config repo, photos)
 - [ ] Remove dead Unraid-Cloudflared-Tunnel container from Serenity
 - [ ] Serenity malcolm pool capacity planning (heavily utilized)
@@ -32,6 +34,14 @@
 - [ ] Document meshtastic stack fully (see docs/reference/MESHTASTIC.md)
 - [ ] Document N.O.M.A.D. containers (start_nomad.sh services)

+## n8n Workflows
+- [ ] Create Gotify apps and n8n credentials: Reminders, n8n Health, Uptime / Infra
+- [ ] Add env vars to n8n: `PAPERLESS_API_TOKEN`, `PAPERLESS_TAG_TRANSCRIPTION`, `PAPERLESS_TAG_LECTURE_NOTES`, `N8N_API_KEY`
+- [ ] Set up `PD SSH` credential in n8n for workflow 16 (see n8n-workflows/README.md)
+- [ ] Add sudoers NOPASSWD rule for n8n watchdog (see n8n-workflows/README.md)
+- [ ] Import and activate upgraded workflows: 01, 04, 08
+- [ ] Import and activate new workflows: 14, 15 (16 already active)
+
 ## Completed
 - [x] Deploy Gotify (Phase 1) — 2026-05-05
 - [x] Deploy AI stack — LiteLLM, OpenWebUI, Qdrant, Whisper, SearXNG (Phase 2) — 2026-05-05
@@ -43,3 +53,7 @@
 - [x] Fill in port/URL columns in SERVICES_DIRECTORY.md — 2026-05-09
 - [x] Create ROCINANTE.md server doc — 2026-05-09
 - [x] Update all docs to reflect deployed state — 2026-05-09
+- [x] Write n8n workflow upgrade + new workflow plans (01, 04, 08, 14, 15, 16) — 2026-05-09
+- [x] Write Pi-hole 3-node HA deployment plan — 2026-05-09
+- [x] Write Kima-Hub deployment plan — 2026-05-09
+- [x] Create Postgres dedup tables (grafana_alert_dedup, n8n_health_dedup) on PD — 2026-05-09
--- a/docs/troubleshooting/KNOWN_QUIRKS.md
+++ b/docs/troubleshooting/KNOWN_QUIRKS.md
@@ -4,6 +4,15 @@ Per-service gotchas that aren't bugs but will bite you if you forget them.

 ## PlausibleDeniability

+### n8n workflow 16 (NFS Watchdog)
+- Runs `sudo docker restart` and `sudo bash .../mount-unraid-nfs.sh` over SSH from n8n container
+- Requires a NOPASSWD sudoers rule or SSH commands will hang waiting for a password:
+  ```
+  truenas_admin ALL=(ALL) NOPASSWD: /usr/bin/docker, /bin/bash /mnt/tank/docker/scripts/mount-unraid-nfs.sh
+  ```
+  Add via `sudo visudo -f /etc/sudoers.d/n8n-watchdog`
+- SSH credential in n8n must use key-based auth (see n8n-workflows/README.md for keygen steps)
+
 ### immich-ml
 - Healthcheck has no curl/wget — uses `/proc/net/tcp6` pattern
 - Cache goes to `/mnt/docker-ssd/docker/appdata/immich-ml` (separate from immich-server's appdata)