     1|# Homelab Expansion — Full Buildout Plan
     2|
     3|> Generated: 2026-05-04
     4|> Based on: HOMELAB_EXPANSION_PLAN.md, ARCHITECTURE_OVERVIEW.md, STACK_STANDARDS.md
     5|
     6|This document contains everything needed to deploy all six expansion phases — docker-compose files, .env templates, directory scaffolding commands, database init SQL, Pangolin configuration, and validation steps. Each phase follows the established stack standards and deployment checklist.
     7|
     8|---
     9|
    10|## Table of Contents
    11|
    12|1. [Prerequisites & Conventions](#prerequisites--conventions)
    13|2. [Phase 1 — Gotify (Notifications)](#phase-1--gotify-notifications)
    14|3. [Phase 2 — Qdrant (Vector Database)](#phase-2--qdrant-vector-database)
    15|4. [Phase 3 — n8n (Workflow Automation)](#phase-3--n8n-workflow-automation)
    16|5. [Phase 4 — Paperless-NGX (Document Intelligence)](#phase-4--paperless-ngx-document-intelligence)
    17|6. [Phase 5 — Home Assistant (Home Automation)](#phase-5--home-assistant-home-automation)
    18|7. [Phase 6 — Grafana + Prometheus (Observability)](#phase-6--grafana--prometheus-observability)
    19|8. [Phase 7 — LiteLLM (AI Gateway)](#phase-7--litellm-ai-gateway)
    20|9. [Phase 8 — Reranker (RAG Quality)](#phase-8--reranker-rag-quality)
    21|10. [Phase 9 — faster-whisper (Speech-to-Text)](#phase-9--faster-whisper-speech-to-text)
    22|11. [Additional Tools Setup](#additional-tools-setup)
    23|12. [Deployment Order Summary](#deployment-order-summary)
    24|13. [Post-Deployment Validation Master Checklist](#post-deployment-validation-master-checklist)
    25|
    26|---
    27|
    28|## Prerequisites & Conventions
    29|
    30|### Deployment Standards (recap)
    31|
    32|All stacks on PlausibleDeniability follow the same pattern:
    33|
    34|- **Repo root:** `/mnt/docker-ssd/docker/compose`
    35|- **Validate before deploy:** `docker compose --env-file .env config`
    36|- **Deploy:** `docker compose --env-file .env up -d`
    37|- **Teardown:** `docker compose --env-file .env down`
    38|
    39|### Storage Tiers
    40|
    41|| Tier | Mount | Use For |
    42||------|-------|---------|
    43|| SSD | `/mnt/docker-ssd/docker/appdata/<service>` | Write-heavy, SQLite, GPU/model, databases |
    44|| Tank | `/mnt/tank/docker/appdata/<service>` | General appdata, configs, uploads |
    45|| Unraid | `/mnt/unraid/data/media/` | Media libraries only |
    46|
    47|### Networks
    48|
    49|| Network | Created By | Purpose |
    50||---------|-----------|---------|
    51|| `ix-databases_shared-databases` | databases stack | Access to shared-postgres, shared-mariadb, shared-redis |
    52|| `pangolin` | newt (infrastructure stack) | Reverse proxy / external exposure |
    53|
    54|### Secret Generation Commands
    55|
    56|```bash
    57|# Database passwords
    58|openssl rand -hex 24
    59|
    60|# JWT / encryption keys
    61|openssl rand -hex 32
    62|
    63|# Paperless secret key
    64|python3 -c "import secrets; print(secrets.token_urlsafe(50))"
    65|```
    66|
    67|### Deployment Order
    68|
    69|```
    70|databases (already running) → infrastructure (already running)
    71|  → Phase 1: Gotify
    72|  → Phase 2: Qdrant  (creates the ai-services network)
    73|  → Phase 3: n8n     (depends on Gotify + Qdrant — joins ai-services as external)
    74|  → Phase 4: Paperless-NGX (depends on n8n for automation hooks)
    75|  → Phase 5: Home Assistant (depends on n8n for heavy automation)
    76|  → Phase 6: Grafana + Prometheus (on N.O.M.A.D., independent but benefits from all above)
    77|```
    78|
    79|> **Important:** The `ai` stack (Phase 2) must be deployed before the `automation` stack (Phase 3) because n8n declares `ai-services` as an external network. If the `ai` stack isn't up, `docker compose up` for `automation` will fail with a missing network error.
    80|
    81|---
    82|
    83|## Phase 1 — Gotify (Notifications)
    84|
    85|**Host:** PlausibleDeniability
    86|**Stack directory:** `/mnt/docker-ssd/docker/compose/automation/`
    87|**Why first:** Every subsequent phase sends notifications through Gotify. It's the output bus.
    88|
    89|### 1.1 — Scaffold Directories
    90|
    91|```bash
    92|# Appdata on SSD (SQLite backend — must not be on NFS)
    93|sudo mkdir -p /mnt/docker-ssd/docker/appdata/gotify
    94|
    95|# Stack directory (may already exist if automation/ is planned)
    96|sudo mkdir -p /mnt/docker-ssd/docker/compose/automation
    97|```
    98|
    99|### 1.2 — Database Init
   100|
   101|None required — Gotify uses an embedded SQLite database stored in its data volume.
   102|
   103|### 1.3 — docker-compose.yaml
   104|
   105|Add to `/mnt/docker-ssd/docker/compose/automation/docker-compose.yaml`:
   106|
   107|```yaml
   108|name: automation
   109|
   110|services:
   111|  gotify:
   112|    image: gotify/server:2.6.1
   113|    container_name: gotify
   114|    restart: unless-stopped
   115|    ports:
   116|      - "8484:80"
   117|    environment:
   118|      TZ: ${TZ}
   119|      GOTIFY_DEFAULTUSER_PASS: ${GOTIFY_ADMIN_PASS}
   120|      GOTIFY_SERVER_PORT: 80
   121|      GOTIFY_DATABASE_DIALECT: sqlite3
   122|      GOTIFY_DATABASE_CONNECTION: data/gotify.db
   123|    volumes:
   124|      - /mnt/docker-ssd/docker/appdata/gotify:/app/data
   125|    networks:
   126|      - pangolin
   127|      - default
   128|
   129|networks:
   130|  pangolin:
   131|    external: true
   132|```
   133|
   134|> **Image note:** `gotify/server:2.6.1` is the latest stable as of May 2026. Check [Docker Hub](https://hub.docker.com/r/gotify/server/tags) before deploying — pin to the exact version.
   135|
   136|### 1.4 — .env.example
   137|
   138|```bash
   139|# /mnt/docker-ssd/docker/compose/automation/.env.example
   140|
   141|TZ=America/New_York
   142|GOTIFY_ADMIN_PASS=CHANGE_ME
   143|```
   144|
   145|### 1.5 — .env (create from example)
   146|
   147|```bash
   148|cd /mnt/docker-ssd/docker/compose/automation
   149|cp .env.example .env
   150|# Edit .env with real values:
   151|# GOTIFY_ADMIN_PASS=$(openssl rand -hex 16)
   152|nano .env
   153|```
   154|
   155|### 1.6 — Pangolin Configuration
   156|
   157|In your Pangolin dashboard, create a new resource:
   158|
   159|| Field | Value |
   160||-------|-------|
   161|| Domain | `gotify.paccoco.com` |
   162|| Scheme | `http` |
   163|| Host | `gotify` |
   164|| Port | `80` |
   165|| Network | Docker service name resolution via `pangolin` network |
   166|
   167|### 1.7 — Validate & Deploy
   168|
   169|```bash
   170|cd /mnt/docker-ssd/docker/compose/automation
   171|docker compose --env-file .env config
   172|docker compose --env-file .env up -d
   173|```
   174|
   175|### 1.8 — Post-Deploy Verification
   176|
   177|```bash
   178|# Container running?
   179|docker ps --filter name=gotify
   180|
   181|# Clean startup logs?
   182|docker logs gotify --tail 20
   183|
   184|# Mounts correct?
   185|docker inspect gotify --format '{{json .Mounts}}' | python3 -m json.tool
   186|
   187|# Quick health check
   188|curl -s http://localhost:8484/health
   189|
   190|# Test notification via API
   191|curl -s "http://localhost:8484/message?token=*** \
   192|  -F "title=Homelab" \
   193|  -F "message=Gotify is online" \
   194|  -F "priority=5"
   195|```
   196|
   197|### 1.9 — First-Run Setup
   198|
   199|1. Navigate to `https://gotify.paccoco.com`
   200|2. Log in with the admin password from `.env`
   201|3. **Change the default admin password** in the UI
   202|4. Create application tokens for each notification source:
   203|   - `n8n-workflows` — for all n8n automations
   204|   - `infrastructure-alerts` — for Uptime Kuma, Grafana, etc.
   205|   - `media-notifications` — for Sonarr/Radarr/Tautulli hooks
   206|   - `home-assistant` — for HA automations
   207|5. Create client tokens for each receiving device (phone, desktop)
   208|6. Install the Gotify Android app and configure with your client token
   209|
   210|---
   211|
   212|## Phase 2 — AI Stack (Vector DB, LLM, Embeddings, Reranker, STT)
   213|
   214|**Host:** PlausibleDeniability
   215|**Stack directory:** `/mnt/docker-ssd/docker/compose/ai/`
   216|**Status:** DEPLOYED AND VERIFIED (2026-05-05)
   217|
   218|> **DEPLOYED NOTE:** This phase was deployed as a complete AI stack with 6 services: Ollama (11434), OpenWebUI (8282), Qdrant (6333/6334), LiteLLM (4000), Reranker/TEI (8787), and faster-whisper (8786). The compose below shows the original Qdrant-only plan — see the actual running compose on PD at `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml` and the `project_ai_stack_deployed.md` memory for port mappings and fixes. Key lesson: always use `10.5.30.6` not `localhost` for health checks on TrueNAS Scale.
   219|
   220|### 2.1 — Scaffold Directories
   221|
   222|```bash
   223|# Appdata on SSD (write-heavy vector storage)
   224|sudo mkdir -p /mnt/docker-ssd/docker/appdata/qdrant/storage
   225|sudo mkdir -p /mnt/docker-ssd/docker/appdata/qdrant/snapshots
   226|
   227|# Stack directory
   228|sudo mkdir -p /mnt/docker-ssd/docker/compose/ai
   229|```
   230|
   231|### 2.2 — docker-compose.yaml
   232|
   233|Create or update `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml`:
   234|
   235|```yaml
   236|name: ai
   237|
   238|services:
   239|  qdrant:
   240|    image: qdrant/qdrant:v1.14.0
   241|    container_name: qdrant
   242|    restart: unless-stopped
   243|    ports:
   244|      - "6333:6333"   # REST API
   245|      - "6334:6334"   # gRPC
   246|    environment:
   247|      TZ: ${TZ}
   248|      QDRANT__SERVICE__GRPC_PORT: 6334
   249|      QDRANT__STORAGE__STORAGE_PATH: /qdrant/storage
   250|      QDRANT__STORAGE__SNAPSHOTS_PATH: /qdrant/snapshots
   251|    volumes:
   252|      - /mnt/docker-ssd/docker/appdata/qdrant/storage:/qdrant/storage
   253|      - /mnt/docker-ssd/docker/appdata/qdrant/snapshots:/qdrant/snapshots
   254|    networks:
   255|      - ai-services
   256|      - default
   257|
   258|  # -------------------------------------------------------
   259|  # Ollama and OpenWebUI go here when deployed.
   260|  # They share this stack and the default + ai-services networks so
   261|  # OpenWebUI can reach Qdrant at http://qdrant:6333
   262|  # -------------------------------------------------------
   263|
   264|  # ollama:
   265|  #   image: ollama/ollama:latest
   266|  #   container_name: ollama
   267|  #   ...
   268|
   269|  # openwebui:
   270|  #   image: ghcr.io/open-webui/open-webui:main
   271|  #   container_name: openwebui
   272|  #   ...
   273|
   274|networks:
   275|  ai-services:
   276|    name: ai-services
   277|```
   278|
   279|> **Image note:** `qdrant/qdrant:v1.14.0` is the latest stable as of May 2026. Check [GitHub releases](https://github.com/qdrant/qdrant/releases) before deploying.
   280|
   281|> **Cross-stack connectivity:** The `ai-services` network is defined here with an explicit `name:` so other stacks (like `automation/n8n`) can declare it as `external: true` and reach Qdrant by service name (`http://qdrant:6333`). This follows the same pattern as `ix-databases_shared-databases`.
   282|
   283|### 2.3 — .env.example
   284|
   285|```bash
   286|# /mnt/docker-ssd/docker/compose/ai/.env.example
   287|
   288|TZ=America/New_York
   289|```
   290|
   291|### 2.4 — Validate & Deploy
   292|
   293|```bash
   294|cd /mnt/docker-ssd/docker/compose/ai
   295|cp .env.example .env
   296|nano .env  # Set timezone
   297|
   298|docker compose --env-file .env config
   299|docker compose --env-file .env up -d
   300|```
   301|
   302|### 2.5 — Post-Deploy Verification
   303|
   304|```bash
   305|# Container running?
   306|docker ps --filter name=qdrant
   307|
   308|# Clean startup?
   309|docker logs qdrant --tail 20
   310|
   311|# REST API responding?
   312|curl -s http://localhost:6333/healthz
   313|# Expected: {"title":"qdrant - vectorass engine","version":"1.14.0","commit":"..."}
   314|
   315|# Create a test collection
   316|curl -X PUT http://localhost:6333/collections/test_collection \
   317|  -H "Content-Type: application/json" \
   318|  -d '{
   319|    "vectors": {
   320|      "size": 384,
   321|      "distance": "Cosine"
   322|    }
   323|  }'
   324|
   325|# Verify it exists
   326|curl -s http://localhost:6333/collections | python3 -m json.tool
   327|
   328|# Clean up test
   329|curl -X DELETE http://localhost:6333/collections/test_collection
   330|```
   331|
   332|### 2.6 — OpenWebUI Integration (when deployed)
   333|
   334|When Ollama and OpenWebUI are brought online in this same stack, configure OpenWebUI's RAG settings:
   335|
   336|1. Go to OpenWebUI → Admin → Settings → Documents
   337|2. Set the vector database to Qdrant
   338|3. Endpoint: `http://qdrant:6333` (Docker DNS within the `ai` stack network)
   339|4. Collection name: `openwebui_docs` (or your preference)
   340|
   341|### 2.7 — Collections to Create (for n8n in Phase 3)
   342|
   343|These collections will be created programmatically by n8n workflows, but for reference:
   344|
   345|| Collection | Vector Size | Content |
   346||-----------|-------------|---------|
   347|| `homelab_docs` | 384 (nomic-embed-text) | Homelab markdown documentation |
   348|| `gitea_commits` | 384 | Gitea commit messages + diffs |
   349|| `media_metadata` | 384 | Plex/Tautulli metadata |
   350|| `obsidian_notes` | 384 | Personal notes from Obsidian vault |
   351|
   352|> **Vector size note:** 384 is the dimension for `nomic-embed-text` via Ollama. If you use a different embedding model, adjust accordingly.
   353|
   354|---
   355|
   356|## Phase 3 — n8n (Workflow Automation)
   357|
   358|**Host:** PlausibleDeniability
   359|**Stack directory:** `/mnt/docker-ssd/docker/compose/automation/` (same stack as Gotify)
   360|**Why third:** n8n is the orchestration backbone — it ties Gotify, Qdrant, Ollama, and all triggers together.
   361|
   362|### 3.1 — Scaffold Directories
   363|
   364|```bash
   365|# Config on tank (workflow definitions, credentials store)
   366|# NOTE: n8n also writes execution logs here. If execution logging becomes
   367|# heavy (many workflows running frequently), consider moving to SSD.
   368|# For typical homelab usage (~20 workflows), tank is fine.
   369|sudo mkdir -p /mnt/tank/docker/appdata/n8n
   370|
   371|# Set ownership — n8n runs as UID 1000 (node user)
   372|sudo chown 1000:1000 /mnt/tank/docker/appdata/n8n
   373|```
   374|
   375|### 3.2 — Database Init
   376|
   377|n8n uses the existing shared-postgres. Create its database and user:
   378|
   379|```bash
   380|docker exec -i shared-postgres psql -U postgres <<'SQL'
   381|CREATE USER n8n WITH PASSWORD 'REPLACE_WITH_GENERATED_PASSWORD';
   382|CREATE DATABASE n8n OWNER n8n;
   383|GRANT ALL PRIVILEGES ON DATABASE n8n TO n8n;
   384|SQL
   385|```
   386|
   387|Generate the password first:
   388|```bash
   389|openssl rand -hex 24
   390|```
   391|
   392|### 3.3 — docker-compose.yaml
   393|
   394|Update `/mnt/docker-ssd/docker/compose/automation/docker-compose.yaml` to add n8n alongside Gotify:
   395|
   396|```yaml
   397|name: automation
   398|
   399|services:
   400|  gotify:
   401|    image: gotify/server:2.6.1
   402|    container_name: gotify
   403|    restart: unless-stopped
   404|    ports:
   405|      - "8484:80"
   406|    environment:
   407|      TZ: ${TZ}
   408|      GOTIFY_DEFAULTUSER_PASS: ${GOTIFY_ADMIN_PASS}
   409|      GOTIFY_SERVER_PORT: 80
   410|      GOTIFY_DATABASE_DIALECT: sqlite3
   411|      GOTIFY_DATABASE_CONNECTION: data/gotify.db
   412|    volumes:
   413|      - /mnt/docker-ssd/docker/appdata/gotify:/app/data
   414|    networks:
   415|      - pangolin
   416|      - default
   417|
   418|  n8n:
   419|    image: docker.n8n.io/n8nio/n8n:1.88.0
   420|    container_name: n8n
   421|    restart: unless-stopped
   422|    ports:
   423|      - "5678:5678"
   424|    environment:
   425|      TZ: ${TZ}
   426|      # Database
   427|      DB_TYPE: postgresdb
   428|      DB_POSTGRESDB_HOST: shared-postgres
   429|      DB_POSTGRESDB_PORT: 5432
   430|      DB_POSTGRESDB_DATABASE: ${N8N_DB_NAME}
   431|      DB_POSTGRESDB_USER: ${N8N_DB_USER}
   432|      DB_POSTGRESDB_PASSWORD: ${N8N_DB_PASS}
   433|      # Security
   434|      N8N_ENCRYPTION_KEY: ${N8N_ENCRYPTION_KEY}
   435|      # Webhook / Reverse Proxy
   436|      N8N_HOST: ${N8N_HOST}
   437|      N8N_PROTOCOL: https
   438|      N8N_PORT: 5678
   439|      WEBHOOK_URL: https://${N8N_HOST}/
   440|      N8N_PROXY_HOPS: 1
   441|      # General
   442|      GENERIC_TIMEZONE: ${TZ}
   443|      N8N_DIAGNOSTICS_ENABLED: false
   444|      N8N_PERSONALIZATION_ENABLED: false
   445|    volumes:
   446|      - /mnt/tank/docker/appdata/n8n:/home/node/.n8n
   447|    networks:
   448|      - pangolin
   449|      - ix-databases_shared-databases
   450|      - ai-services
   451|      - default
   452|    depends_on:
   453|      - gotify
   454|
   455|networks:
   456|  pangolin:
   457|    external: true
   458|  ix-databases_shared-databases:
   459|    external: true
   460|  ai-services:
   461|    external: true
   462|```
   463|
   464|> **Image note:** `docker.n8n.io/n8nio/n8n:1.88.0` — n8n uses calendar-ish versioning. Check [n8n releases](https://github.com/n8n-io/n8n/releases) for the latest stable. The `docker.n8n.io` registry is preferred for production.
   465|
   466|> **Cross-stack access:** n8n joins `ai-services` (created by the `ai` stack) so it can reach Qdrant at `http://qdrant:6333` by Docker DNS. It also joins `ix-databases_shared-databases` for Postgres access — same pattern.
   467|
   468|### 3.4 — .env.example (updated for both services)
   469|
   470|```bash
   471|# /mnt/docker-ssd/docker/compose/automation/.env.example
   472|
   473|TZ=America/New_York
   474|
   475|# Gotify
   476|GOTIFY_ADMIN_PASS=CHANGE_ME
   477|
   478|# n8n — Database
   479|N8N_DB_NAME=n8n
   480|N8N_DB_USER=n8n
   481|N8N_DB_PASS=CHANGE_ME
   482|
   483|# n8n — Security
   484|N8N_ENCRYPTION_KEY=CHANGE_ME
   485|
   486|# n8n — Hostname
   487|N8N_HOST=n8n.paccoco.com
   488|```
   489|
   490|### 3.5 — Pangolin Configuration
   491|
   492|| Field | Value |
   493||-------|-------|
   494|| Domain | `n8n.paccoco.com` |
   495|| Scheme | `http` |
   496|| Host | `n8n` |
   497|| Port | `5678` |
   498|| Headers | Forward `X-Forwarded-Proto: https` |
   499|
   500|### 3.6 — Validate & Deploy
   501|
   502|```bash
   503|cd /mnt/docker-ssd/docker/compose/automation
   504|
   505|# Update .env with real passwords
   506|nano .env
   507|
   508|docker compose --env-file .env config
   509|docker compose --env-file .env up -d
   510|```
   511|
   512|### 3.7 — Post-Deploy Verification
   513|
   514|```bash
   515|# Both containers running?
   516|docker ps --filter name=gotify --filter name=n8n
   517|
   518|# n8n logs clean?
   519|docker logs n8n --tail 30
   520|
   521|# n8n on correct networks?
   522|docker inspect n8n --format '{{json .NetworkSettings.Networks}}' | python3 -m json.tool
   523|
   524|# n8n UI accessible?
   525|curl -s -o /dev/null -w "%{http_code}" http://localhost:5678/
   526|# Expected: 200
   527|
   528|# Postgres connection working? (check logs for DB migration messages)
   529|docker logs n8n 2>&1 | grep -i "migrat"
   530|```
   531|
   532|### 3.8 — First-Run Setup
   533|
   534|1. Navigate to `https://n8n.paccoco.com`
   535|2. Create your admin account
   536|3. Install community nodes you'll need:
   537|   - `n8n-nodes-gotify` (if available) or use HTTP Request node
   538|   - Ollama nodes (built-in as of n8n 1.x)
   539|
   540|### 3.9 — Workflow Blueprints
   541|
   542|Below are starter workflow descriptions for each planned automation. These are meant to be built in the n8n UI — the structure and node types are described so you can wire them up.
   543|
   544|#### Media Pipeline Workflows
   545|
   546|**Workflow: Sonarr/Radarr Download Notification**
   547|```
   548|Trigger:  Webhook node (POST from Sonarr/Radarr on download/import)
   549|Step 1:   Extract series/movie name, quality, file path from webhook body
   550|Step 2:   HTTP Request → TMDB API to fetch poster image URL
   551|Step 3:   HTTP Request → Gotify REST API (POST /message)
   552|            - Title: "New Download: {title}"
   553|            - Message: "{quality} — {episodeTitle or year}"
   554|            - Priority: 5
   555|            - Extras: attach poster URL as markdown image
   556|Also:     HTTP Request → Discord webhook (formatted embed with poster)
   557|```
   558|
   559|Configure Sonarr/Radarr webhooks:
   560|- Sonarr: Settings → Connect → Webhook → URL: `https://n8n.paccoco.com/webhook/sonarr`
   561|- Radarr: Settings → Connect → Webhook → URL: `https://n8n.paccoco.com/webhook/radarr`
   562|
   563|**Workflow: Tautulli Play Logging**
   564|```
   565|Trigger:  Webhook node (POST from Tautulli on playback start)
   566|Step 1:   Extract user, media title, player, quality from payload
   567|Step 2:   Postgres node → INSERT into watch_history table
   568|Step 3:   (Optional) Gotify notification for specific users/media
   569|```
   570|
   571|Tautulli config: Settings → Notification Agents → Webhook → URL: `https://n8n.paccoco.com/webhook/tautulli-play`
   572|
   573|**Workflow: Weekly Watch Digest**
   574|```
   575|Trigger:  Cron node (every Sunday at 10:00 AM)
   576|Step 1:   Postgres node → SELECT watch history for past 7 days
   577|Step 2:   Format data as structured text
   578|Step 3:   HTTP Request → Ollama API (POST to PD's qwen2.5:14b)
   579|            - Prompt: "Summarize this week's viewing in a fun digest: {data}"
   580|Step 4:   HTTP Request → Gotify (send digest)
   581|```
   582|
   583|#### Infrastructure Monitoring Workflows
   584|
   585|**Workflow: Uptime Kuma Enhanced Alerts**
   586|```
   587|Trigger:  Webhook node (from Uptime Kuma notification)
   588|Step 1:   Extract monitor name, status, response time
   589|Step 2:   HTTP Request → Netdata API for related metrics context
   590|Step 3:   HTTP Request → Gotify
   591|            - Title: "🔴 {monitor} DOWN" or "🟢 {monitor} UP"
   592|            - Message: include Netdata context (CPU, mem, disk at time of alert)
   593|            - Priority: 8 (high for down, 3 for recovery)
   594|```
   595|
   596|**Workflow: ZFS Pool Utilization Alert**
   597|```
   598|Trigger:  Cron node (every 6 hours)
   599|Step 1:   SSH node → Serenity: `zpool list -Hp malcolm`
   600|Step 2:   Parse capacity percentage
   601|Step 3:   IF capacity > 85% → Gotify alert (priority 8)
   602|Step 4:   IF capacity > 90% → Gotify alert (priority 10) + Discord webhook
   603|```
   604|
   605|**Workflow: Grafana Alert Remediation**
   606|```
   607|Trigger:  Webhook node (from Grafana alerting)
   608|Step 1:   Parse alert labels (container, host, metric)
   609|Step 2:   Switch node → route by alert type:
   610|            - High CPU container → SSH → docker restart {container}
   611|            - Disk full → SSH → pause qBittorrent, notify via Gotify
   612|            - Memory pressure → Gotify alert only (manual intervention)
   613|Step 3:   Log action taken to Postgres
   614|```
   615|
   616|#### Homelab Ops Workflows
   617|
   618|**Workflow: Gitea Commit Summary**
   619|```
   620|Trigger:  Webhook node (Gitea webhook on push to truenas-stacks)
   621|Step 1:   Extract commit messages, author, files changed
   622|Step 2:   HTTP Request → Ollama API
   623|            - Prompt: "Summarize this commit in one sentence: {commit_message}"
   624|Step 3:   HTTP Request → Gotify
   625|            - Title: "Commit to truenas-stacks"
   626|            - Message: Ollama-generated summary
   627|```
   628|
   629|Gitea config: Repository → Settings → Webhooks → URL: `https://n8n.paccoco.com/webhook/gitea-push`
   630|
   631|**Workflow: qBittorrent Auto-Rescan**
   632|```
   633|Trigger:  Webhook or polling (qBittorrent API for completed+moved torrents)
   634|Step 1:   Determine if file is in Sonarr or Radarr path
   635|Step 2:   HTTP Request → Sonarr API (POST /command → RescanSeries)
   636|           OR HTTP Request → Radarr API (POST /command → RescanMovie)
   637|Step 3:   Gotify notification confirming rescan triggered
   638|```
   639|
   640|#### AI Pipeline Workflows
   641|
   642|**Workflow: URL Digest Pipeline**
   643|```
   644|Trigger:  Webhook node (POST with URL in body)
   645|Step 1:   HTTP Request → fetch page content
   646|Step 2:   Code node → extract text, chunk into ~500 token segments
   647|Step 3:   HTTP Request → Ollama API → summarize each chunk
   648|Step 4:   Code node → combine summaries into digest
   649|Step 5:   Postgres node → store digest with metadata
   650|Step 6:   Return digest in webhook response
   651|```
   652|
   653|**Workflow: Multi-Model Query Router (simplified by LiteLLM — see Phase 7)**
   654|```
   655|Trigger:  Webhook node (POST with query + complexity hint)
   656|Step 1:   HTTP Request → LiteLLM (POST http://litellm:4000/v1/chat/completions)
   657|            - model: complexity parameter ("light", "medium", or "heavy")
   658|            - LiteLLM handles routing to the correct Ollama instance
   659|Step 2:   Return response via webhook
   660|```
   661|> With LiteLLM deployed, the Switch node and three separate Ollama endpoints
   662|> are replaced by a single HTTP Request node. The routing logic lives in
   663|> LiteLLM's config.yaml instead of n8n workflow logic.
   664|
   665|**Workflow: Qdrant Index Updater**
   666|```
   667|Trigger:  Webhook node (from Gitea push to any watched repo)
   668|Step 1:   HTTP Request → Gitea API → fetch changed files content
   669|Step 2:   Code node → chunk text into embedding-sized segments
   670|Step 3:   HTTP Request → Ollama API (embed endpoint with nomic-embed-text)
   671|Step 4:   HTTP Request → Qdrant API (PUT /collections/homelab_docs/points)
   672|Step 5:   Gotify notification: "{n} documents re-indexed"
   673|```
   674|
   675|#### Home / Business Workflows
   676|
   677|**Workflow: KitchenOwl Grocery Notification**
   678|```
   679|Trigger:  Polling (KitchenOwl API) or webhook if supported
   680|Step 1:   Fetch current shopping list items
   681|Step 2:   Format as clean text list
   682|Step 3:   HTTP Request → Gotify → phone push notification
   683|```
   684|
   685|**Workflow: Donetick Task Reminder**
   686|```
   687|Trigger:  Cron node (daily at 9:00 AM)
   688|Step 1:   HTTP Request → Donetick API → fetch tasks due today/overdue
   689|Step 2:   Format task list
   690|Step 3:   HTTP Request → Gotify (priority 5)
   691|```
   692|
   693|**Workflow: Long and Low Crafts Order Pipeline**
   694|```
   695|Trigger:  Webhook (Etsy webhook or email trigger via IMAP node)
   696|Step 1:   Parse order details (item, quantity, customer, shipping)
   697|Step 2:   HTTP Request → Donetick API → create fulfillment task
   698|Step 3:   HTTP Request → Gotify DM notification
   699|            - Title: "New L&L Order"
   700|            - Message: "{item} x{qty} — ship by {date}"
   701|            - Priority: 7
   702|```
   703|
   704|---
   705|
   706|## Phase 4 — Paperless-NGX (Document Intelligence)
   707|
   708|**Host:** PlausibleDeniability
   709|**Stack directory:** `/mnt/docker-ssd/docker/compose/documents/` (new stack)
   710|**Why fourth:** Depends on n8n for automation hooks and Gotify for notifications.
   711|
   712|### 4.1 — Scaffold Directories
   713|
   714|```bash
   715|# Index/data on SSD (search index is write-heavy)
   716|sudo mkdir -p /mnt/docker-ssd/docker/appdata/paperless/data
   717|
   718|# Documents (media) on tank (bulk storage, read-heavy)
   719|sudo mkdir -p /mnt/tank/docker/appdata/paperless/media
   720|
   721|# Consume folder on tank (drop zone for new documents)
   722|sudo mkdir -p /mnt/tank/docker/appdata/paperless/consume
   723|
   724|# Export folder on tank
   725|sudo mkdir -p /mnt/tank/docker/appdata/paperless/export
   726|
   727|# Stack directory
   728|sudo mkdir -p /mnt/docker-ssd/docker/compose/documents
   729|```
   730|
   731|### 4.2 — Database Init
   732|
   733|Paperless uses the existing shared-postgres and shared-redis:
   734|
   735|```bash
   736|docker exec -i shared-postgres psql -U postgres <<'SQL'
   737|CREATE USER paperless WITH PASSWORD 'REPLACE_WITH_GENERATED_PASSWORD';
   738|CREATE DATABASE paperless OWNER paperless;
   739|GRANT ALL PRIVILEGES ON DATABASE paperless TO paperless;
   740|SQL
   741|```
   742|
   743|### 4.3 — docker-compose.yaml
   744|
   745|Create `/mnt/docker-ssd/docker/compose/documents/docker-compose.yaml`:
   746|
   747|```yaml
   748|name: documents
   749|
   750|services:
   751|  paperless:
   752|    image: ghcr.io/paperless-ngx/paperless-ngx:2.16
   753|    container_name: paperless
   754|    restart: unless-stopped
   755|    ports:
   756|      - "8000:8000"
   757|    environment:
   758|      TZ: ${TZ}
   759|      # Database
   760|      PAPERLESS_DBENGINE: postgresql
   761|      PAPERLESS_DBHOST: shared-postgres
   762|      PAPERLESS_DBPORT: 5432
   763|      PAPERLESS_DBNAME: ${PAPERLESS_DB_NAME}
   764|      PAPERLESS_DBUSER: ${PAPERLESS_DB_USER}
   765|      PAPERLESS_DBPASS: ${PAPERLESS_DB_PASS}
   766|      # Redis (using shared-redis)
   767|      PAPERLESS_REDIS: redis://shared-redis:***@10.5.30.7
  1063|
  1064|# Stack directory
  1065|sudo mkdir -p /opt/monitoring
  1066|
  1067|# Prometheus data on hdd-2 (has more headroom)
  1068|sudo mkdir -p /mnt/hdd-2/prometheus-data
  1069|sudo chown 65534:65534 /mnt/hdd-2/prometheus-data  # nobody user (Prometheus default)
  1070|
  1071|# Grafana data on hdd-2
  1072|sudo mkdir -p /mnt/hdd-2/grafana-data
  1073|sudo chown 472:472 /mnt/hdd-2/grafana-data  # grafana user
  1074|
  1075|# Config directories
  1076|sudo mkdir -p /opt/monitoring/provisioning/datasources
  1077|sudo mkdir -p /opt/monitoring/provisioning/dashboards
  1078|```
  1079|
  1080|### 6.2 — prometheus.yml
  1081|
  1082|Create `/opt/monitoring/prometheus.yml`:
  1083|
  1084|```yaml
  1085|global:
  1086|  scrape_interval: 15s
  1087|  evaluation_interval: 15s
  1088|
  1089|scrape_configs:
  1090|  # ------- Local (N.O.M.A.D.) -------
  1091|  - job_name: 'nomad-node'
  1092|    static_configs:
  1093|      - targets: ['node-exporter:9100']
  1094|        labels:
  1095|          host: 'nomad'
  1096|
  1097|  # ------- PlausibleDeniability -------
  1098|  - job_name: 'pd-netdata'
  1099|    metrics_path: /api/v1/allmetrics
  1100|    params:
  1101|      format: [prometheus]
  1102|    static_configs:
  1103|      - targets: ['10.5.1.X:19999']    # Replace with PD's IP
  1104|        labels:
  1105|          host: 'plausible-deniability'
  1106|
  1107|  # ------- Serenity -------
  1108|  - job_name: 'serenity-netdata'
  1109|    metrics_path: /api/v1/allmetrics
  1110|    params:
  1111|      format: [prometheus]
  1112|    static_configs:
  1113|      - targets: ['10.5.30.5:19999']
  1114|        labels:
  1115|          host: 'serenity'
  1116|
  1117|  # ------- Gotify -------
  1118|  # Gotify exposes /health but no Prometheus endpoint natively.
  1119|  # Use blackbox exporter or just rely on Uptime Kuma.
  1120|
  1121|  # ------- n8n -------
  1122|  # n8n doesn't expose Prometheus metrics by default.
  1123|  # Monitor via container resource metrics from Netdata.
  1124|
  1125|  # -------- Add more targets as needed --------
  1126|  # When node-exporter is installed on PD and Serenity:
  1127|  # - job_name: 'pd-node'
  1128|  #   static_configs:
  1129|  #     - targets: ['10.5.1.X:9100']
  1130|  #       labels:
  1131|  #         host: 'plausible-deniability'
  1132|  #
  1133|  # - job_name: 'serenity-node'
  1134|  #   static_configs:
  1135|  #     - targets: ['10.5.30.5:9100']
  1136|  #       labels:
  1137|  #         host: 'serenity'
  1138|```
  1139|
  1140|> **Netdata as a Prometheus target:** Both PD and Serenity already run Netdata. Netdata has a built-in Prometheus exporter at `/api/v1/allmetrics?format=prometheus`. This gives you CPU, memory, disk, network, and ZFS metrics without installing node-exporter on those hosts.
  1141|
  1142|### 6.3 — Grafana Provisioning
  1143|
  1144|Create `/opt/monitoring/provisioning/datasources/prometheus.yml`:
  1145|
  1146|```yaml
  1147|apiVersion: 1
  1148|
  1149|datasources:
  1150|  - name: Prometheus
  1151|    type: prometheus
  1152|    access: proxy
  1153|    url: http://prometheus:9090
  1154|    isDefault: true
  1155|    editable: true
  1156|```
  1157|
  1158|Create `/opt/monitoring/provisioning/dashboards/dashboards.yml`:
  1159|
  1160|```yaml
  1161|apiVersion: 1
  1162|
  1163|providers:
  1164|  - name: 'default'
  1165|    orgId: 1
  1166|    folder: ''
  1167|    type: file
  1168|    disableDeletion: false
  1169|    editable: true
  1170|    options:
  1171|      path: /var/lib/grafana/dashboards
  1172|      foldersFromFilesStructure: false
  1173|```
  1174|
  1175|### 6.4 — docker-compose.yaml
  1176|
  1177|Create `/opt/monitoring/docker-compose.yaml`:
  1178|
  1179|```yaml
  1180|name: monitoring
  1181|
  1182|services:
  1183|  prometheus:
  1184|    image: prom/prometheus:v3.4.0
  1185|    container_name: prometheus
  1186|    restart: unless-stopped
  1187|    ports:
  1188|      - "9090:9090"
  1189|    command:
  1190|      - '--config.file=/etc/prometheus/prometheus.yml'
  1191|      - '--storage.tsdb.path=/prometheus'
  1192|      - '--storage.tsdb.retention.time=90d'
  1193|      - '--web.enable-lifecycle'
  1194|    volumes:
  1195|      - /opt/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
  1196|      - /mnt/hdd-2/prometheus-data:/prometheus
  1197|    networks:
  1198|      - monitoring
  1199|
  1200|  grafana:
  1201|    image: grafana/grafana:13.0.1
  1202|    container_name: grafana
  1203|    restart: unless-stopped
  1204|    ports:
  1205|      - "3000:3000"
  1206|    environment:
  1207|      TZ: ${TZ}
  1208|      GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER}
  1209|      GF_SECURITY_ADMIN_PASSWORD: ${GF_ADMIN_PASS}
  1210|      GF_SERVER_ROOT_URL: https://${GF_HOST}/
  1211|    volumes:
  1212|      - /mnt/hdd-2/grafana-data:/var/lib/grafana
  1213|      - /opt/monitoring/provisioning:/etc/grafana/provisioning:ro
  1214|    networks:
  1215|      - monitoring
  1216|
  1217|  node-exporter:
  1218|    image: prom/node-exporter:v1.9.0
  1219|    container_name: node-exporter
  1220|    restart: unless-stopped
  1221|    ports:
  1222|      - "9100:9100"
  1223|    command:
  1224|      - '--path.procfs=/host/proc'
  1225|      - '--path.sysfs=/host/sys'
  1226|      - '--path.rootfs=/rootfs'
  1227|      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
  1228|    volumes:
  1229|      - /proc:/host/proc:ro
  1230|      - /sys:/host/sys:ro
  1231|      - /:/rootfs:ro
  1232|    networks:
  1233|      - monitoring
  1234|
  1235|networks:
  1236|  monitoring:
  1237|    driver: bridge
  1238|```
  1239|
  1240|> **Image notes:**
  1241|> - `prom/prometheus:v3.4.0` — Prometheus v3 is the current major line. **Do not use `:latest`** — it still resolves to 2.x due to a known tagging issue. Always use an explicit `v3.x.y` tag.
  1242|> - `grafana/grafana:latest` — use the moving stable image by default here; only pin Grafana if a breakage or compatibility reason is documented.
  1243|> - `prom/node-exporter:v1.9.0` — check [releases](https://github.com/prometheus/node_exporter/releases).
  1244|
  1245|### 6.5 — .env.example
  1246|
  1247|```bash
  1248|# /opt/monitoring/.env.example
  1249|
  1250|TZ=America/New_York
  1251|
  1252|# Grafana
  1253|GF_ADMIN_USER=admin
  1254|GF_ADMIN_PASS=CHANGE_ME
  1255|GF_HOST=grafana.paccoco.com
  1256|```
  1257|
  1258|### 6.6 — Pangolin Configuration
  1259|
  1260|Grafana runs on N.O.M.A.D., not PD where the main Newt agent lives. You have two options:
  1261|
  1262|**Option A — Route via N.O.M.A.D.'s existing Newt (recommended if already connected)**
  1263|
  1264|N.O.M.A.D. already has a Newt container from the Project N.O.M.A.D. setup. If it's connected to your Pangolin VPS, just add a new resource in the Pangolin dashboard pointing to `http://grafana:3000` or `http://10.5.30.7:3000`.
  1265|
  1266|**Option B — Add a dedicated Newt to the monitoring stack**
  1267|
  1268|If N.O.M.A.D.'s existing Newt is not connected to Pangolin (or is a separate Pangolin instance), add Newt to the monitoring compose:
  1269|
  1270|```yaml
  1271|  newt:
  1272|    image: ghcr.io/fosrl/newt:latest
  1273|    container_name: monitoring-newt
  1274|    restart: unless-stopped
  1275|    environment:
  1276|      PANGOLIN_ENDPOINT: ${PANGOLIN_ENDPOINT}
  1277|      NEWT_ID: ${NEWT_ID}
  1278|      NEWT_SECRET: ${NEWT_SECRET}
  1279|    networks:
  1280|      - monitoring
  1281|```
  1282|
  1283|Then in Pangolin dashboard:
  1284|
  1285|| Field | Value |
  1286||-------|-------|
  1287|| Domain | `grafana.paccoco.com` |
  1288|| Scheme | `http` |
  1289|| Host | `grafana` (Docker DNS via shared network) |
  1290|| Port | `3000` |
  1291|
  1292|**Option C — Direct IP routing (simplest, no Newt needed)**
  1293|
  1294|If Pangolin's Newt on PD can reach N.O.M.A.D. by LAN IP (they're on the same subnet):
  1295|
  1296|| Field | Value |
  1297||-------|-------|
  1298|| Domain | `grafana.paccoco.com` |
  1299|| Scheme | `http` |
  1300|| Host | `10.5.30.7` |
  1301|| Port | `3000` |
  1302|
  1303|> **Decision point:** Check if N.O.M.A.D.'s existing Newt is connected to your Pangolin instance before deploying. Run `docker ps --filter name=newt` on N.O.M.A.D. to verify. If it's running and connected, Option A is zero-effort. If not, Option C is the simplest fallback.
  1304|
  1305|### 6.7 — Validate & Deploy
  1306|
  1307|```bash
  1308|ssh nomad@10.5.30.7
  1309|
  1310|cd /opt/monitoring
  1311|cp .env.example .env
  1312|nano .env  # Fill in real values
  1313|
  1314|docker compose --env-file .env config
  1315|docker compose --env-file .env up -d
  1316|```
  1317|
  1318|### 6.8 — Post-Deploy Verification
  1319|
  1320|```bash
  1321|# All three containers running?
  1322|docker ps --filter name=prometheus --filter name=grafana --filter name=node-exporter
  1323|
  1324|# Prometheus scraping targets?
  1325|curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | head -40
  1326|
  1327|# Grafana UI accessible?
  1328|curl -s -o /dev/null -w "%{http_code}" http://localhost:3000/
  1329|# Expected: 200 or 302
  1330|
  1331|# Node exporter metrics flowing?
  1332|curl -s http://localhost:9100/metrics | head -10
  1333|```
  1334|
  1335|### 6.9 — Recommended Dashboards
  1336|
  1337|Import these from [Grafana Dashboard Library](https://grafana.com/grafana/dashboards/):
  1338|
  1339|| Dashboard | ID | Purpose |
  1340||-----------|----|---------|
  1341|| Node Exporter Full | 1860 | System metrics for N.O.M.A.D. |
  1342|| Docker Container Stats | 893 | Container resource usage |
  1343|| Netdata via Prometheus | (search) | PD and Serenity system metrics |
  1344|
  1345|To import: Grafana → Dashboards → New → Import → Enter dashboard ID.
  1346|
  1347|### 6.10 — Metrics Targets Summary
  1348|
  1349|| Target | Host | Method | Endpoint |
  1350||--------|------|--------|----------|
  1351|| N.O.M.A.D. system | localhost | node-exporter | node-exporter:9100 |
  1352|| PD system | 10.5.1.X | Netdata Prometheus | 10.5.1.X:19999/api/v1/allmetrics |
  1353|| Serenity system | 10.5.30.5 | Netdata Prometheus | 10.5.30.5:19999/api/v1/allmetrics |
  1354|| ZFS pools | via Netdata | Netdata exports ZFS metrics | Included in Netdata scrape |
  1355|| Container stats | via Netdata | Netdata exports cgroup metrics | Included in Netdata scrape |
  1356|| Plex streams | Tautulli | n8n polling → Postgres | Via n8n workflow (Phase 3) |
  1357|| qBit stats | qBittorrent API | n8n polling → Postgres | Via n8n workflow (Phase 3) |
  1358|| Tailscale latency | Tailscale API | n8n polling → Postgres | Via n8n workflow (Phase 3) |
  1359|
  1360|### 6.11 — n8n Integration (Grafana → n8n alert webhook)
  1361|
  1362|In Grafana → Alerting → Contact Points, create a webhook contact point:
  1363|
  1364|| Field | Value |
  1365||-------|-------|
  1366|| Name | `n8n-alerts` |
  1367|| Type | Webhook |
  1368|| URL | `https://n8n.paccoco.com/webhook/grafana-alert` |
  1369|| HTTP Method | POST |
  1370|
  1371|Then create alert rules for:
  1372|- ZFS pool utilization > 85%
  1373|- Container memory > 90% of limit
  1374|- Host CPU sustained > 90% for 5 minutes
  1375|- Disk I/O latency spikes
  1376|
  1377|These fire into the "Grafana Alert Remediation" n8n workflow (see Phase 3).
  1378|
  1379|---
  1380|
  1381|## Phase 7 — LiteLLM (AI Gateway)
  1382|
  1383|**Host:** PlausibleDeniability
  1384|**Stack directory:** `/mnt/docker-ssd/docker/compose/ai/` (same stack as Qdrant)
  1385|**Why:** Replaces manual multi-model routing with a unified OpenAI-compatible API. Every app (OpenWebUI, Continue.dev, n8n, Paperless summarization) points at one endpoint instead of juggling three Ollama IPs.
  1386|
  1387|### 7.1 — What LiteLLM Does
  1388|
  1389|LiteLLM sits in front of your three Ollama instances and presents a single OpenAI-compatible API at `http://litellm:4000`. It handles:
  1390|
  1391|- **Model routing** — requests for `qwen3:32b` go to ROCINANTE, `qwen2.5:14b` goes to PD, `phi4` goes to N.O.M.A.D.
  1392|- **Failover** — if ROCINANTE is offline, LiteLLM can fall back to PD automatically
  1393|- **Load balancing** — distribute requests across instances running the same model
  1394|- **Usage tracking** — logs token counts, latency, and costs per model/user via its built-in database
  1395|
  1396|### 7.2 — Scaffold Directories
  1397|
  1398|```bash
  1399|# Config on SSD (SQLite DB for usage tracking)
  1400|sudo mkdir -p /mnt/docker-ssd/docker/appdata/litellm
  1401|
  1402|# LiteLLM config file
  1403|sudo mkdir -p /mnt/docker-ssd/docker/compose/ai/litellm
  1404|```
  1405|
  1406|### 7.3 — LiteLLM Config
  1407|
  1408|Create `/mnt/docker-ssd/docker/compose/ai/litellm/config.yaml`:
  1409|
  1410|```yaml
  1411|model_list:
  1412|  # ---- ROCINANTE (RTX 4090, 24GB) — heavy reasoning ----
  1413|  - model_name: "heavy"
  1414|    litellm_params:
  1415|      model: "ollama/qwen3:32b"
  1416|      api_base: "http://10.5.1.ROCINANTE:11434"
  1417|      timeout: 300
  1418|      stream_timeout: 300
  1419|    model_info:
  1420|      description: "Heavy reasoning, long context, complex code"
  1421|
  1422|  - model_name: "heavy"
  1423|    litellm_params:
  1424|      model: "ollama/deepseek-r1:32b"
  1425|      api_base: "http://10.5.1.ROCINANTE:11434"
  1426|      timeout: 300
  1427|      stream_timeout: 300
  1428|    model_info:
  1429|      description: "Deep reasoning fallback on ROCINANTE"
  1430|
  1431|  # ---- PlausibleDeniability (RTX 2080 Ti, 11GB) — general ----
  1432|  - model_name: "medium"
  1433|    litellm_params:
  1434|      model: "ollama/qwen2.5:14b"
  1435|      api_base: "http://host.docker.internal:11434"
  1436|      timeout: 120
  1437|      stream_timeout: 120
  1438|    model_info:
  1439|      description: "General homelab assistant, RAG queries"
  1440|
  1441|  # ---- N.O.M.A.D. (GTX 1080, 8GB) — lightweight ----
  1442|  - model_name: "light"
  1443|    litellm_params:
  1444|      model: "ollama/phi4"
  1445|      api_base: "http://10.5.30.7:11434"
  1446|      timeout: 60
  1447|      stream_timeout: 60
  1448|    model_info:
  1449|      description: "Fast local inference, lightweight tasks"
  1450|
  1451|  - model_name: "light"
  1452|    litellm_params:
  1453|      model: "ollama/llama3.2:3b"
  1454|      api_base: "http://10.5.30.7:11434"
  1455|      timeout: 60
  1456|      stream_timeout: 60
  1457|    model_info:
  1458|      description: "Ultra-light fallback on N.O.M.A.D."
  1459|
  1460|  # ---- Embeddings ----
  1461|  - model_name: "embed"
  1462|    litellm_params:
  1463|      model: "ollama/nomic-embed-text"
  1464|      api_base: "http://host.docker.internal:11434"
  1465|    model_info:
  1466|      description: "Text embeddings for RAG pipeline"
  1467|
  1468|  # ---- Direct model access (bypass routing) ----
  1469|  # These let you request a specific model by its full name
  1470|  - model_name: "ollama/qwen3:32b"
  1471|    litellm_params:
  1472|      model: "ollama/qwen3:32b"
  1473|      api_base: "http://10.5.1.ROCINANTE:11434"
  1474|
  1475|  - model_name: "ollama/qwen2.5:14b"
  1476|    litellm_params:
  1477|      model: "ollama/qwen2.5:14b"
  1478|      api_base: "http://host.docker.internal:11434"
  1479|
  1480|  - model_name: "ollama/phi4"
  1481|    litellm_params:
  1482|      model: "ollama/phi4"
  1483|      api_base: "http://10.5.30.7:11434"
  1484|
  1485|litellm_settings:
  1486|  drop_params: true
  1487|  set_verbose: false
  1488|  request_timeout: 300
  1489|  num_retries: 2
  1490|  retry_after: 5
  1491|  allowed_fails: 3
  1492|  cooldown_time: 60
  1493|
  1494|general_settings:
  1495|  master_key: "os.environ/LITELLM_MASTER_KEY"
  1496|```
  1497|
  1498|> **DEPLOYED NOTE (2026-05-05):** Do NOT add `database_url` to general_settings — newer LiteLLM versions include Prisma ORM that requires PostgreSQL, and adding any database_url causes a crash loop. Without it, LiteLLM runs in config-only mode which is fine for homelab use. The `master_key` uses `os.environ/LITELLM_MASTER_KEY` syntax to read from the container's environment variable (set via .env → docker compose).
  1499|
  1500|> **Routing explained:** Multiple entries with the same `model_name` (like "heavy") enable load balancing and failover within that tier. When you request model `heavy`, LiteLLM picks the healthiest deployment. The direct-access entries (like `ollama/qwen3:32b`) let you bypass the tier system and target a specific model when needed.
  1501|
  1502|> **Replace IPs:** Update `10.5.1.ROCINANTE` with ROCINANTE's actual LAN or Tailscale IP. PD uses `host.docker.internal` since LiteLLM runs on PD alongside Ollama.
  1503|
  1504|### 7.4 — docker-compose.yaml (updated ai stack)
  1505|
  1506|Update `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml` to add LiteLLM:
  1507|
  1508|```yaml
  1509|name: ai
  1510|
  1511|services:
  1512|  qdrant:
  1513|    image: qdrant/qdrant:v1.14.0
  1514|    container_name: qdrant
  1515|    restart: unless-stopped
  1516|    ports:
  1517|      - "6333:6333"   # REST API
  1518|      - "6334:6334"   # gRPC
  1519|    environment:
  1520|      TZ: ${TZ}
  1521|      QDRANT__SERVICE__GRPC_PORT: 6334
  1522|      QDRANT__STORAGE__STORAGE_PATH: /qdrant/storage
  1523|      QDRANT__STORAGE__SNAPSHOTS_PATH: /qdrant/snapshots
  1524|    volumes:
  1525|      - /mnt/docker-ssd/docker/appdata/qdrant/storage:/qdrant/storage
  1526|      - /mnt/docker-ssd/docker/appdata/qdrant/snapshots:/qdrant/snapshots
  1527|    networks:
  1528|      - ai-services
  1529|      - default
  1530|
  1531|  litellm:
  1532|    image: ghcr.io/berriai/litellm:main-latest
  1533|    container_name: litellm
  1534|    restart: unless-stopped
  1535|    ports:
  1536|      - "4000:4000"
  1537|    environment:
  1538|      TZ: ${TZ}
  1539|      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
  1540|    volumes:
  1541|      - /mnt/docker-ssd/docker/compose/ai/litellm/config.yaml:/app/config.yaml:ro
  1542|      - /mnt/docker-ssd/docker/appdata/litellm:/app/data
  1543|    command: ["--config", "/app/config.yaml", "--port", "4000"]
  1544|    extra_hosts:
  1545|      - "host.docker.internal:host-gateway"
  1546|    networks:
  1547|      - ai-services
  1548|      - default
  1549|
  1550|  reranker:
  1551|    image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
  1552|    container_name: reranker
  1553|    restart: unless-stopped
  1554|    ports:
  1555|      - "8787:80"
  1556|    environment:
  1557|      MODEL_ID: ${RERANKER_MODEL}
  1558|    volumes:
  1559|      - /mnt/docker-ssd/docker/appdata/reranker:/data
  1560|    networks:
  1561|      - ai-services
  1562|      - default
  1563|
  1564|  whisper:
  1565|    image: fedirz/faster-whisper-server:latest-cuda
  1566|    container_name: whisper
  1567|    restart: unless-stopped
  1568|    ports:
  1569|      - "8786:8000"
  1570|    environment:
  1571|      TZ: ${TZ}
  1572|      WHISPER__MODEL: ${WHISPER_MODEL}
  1573|      WHISPER__DEVICE: cuda
  1574|      WHISPER__COMPUTE_TYPE: float16
  1575|    volumes:
  1576|      - /mnt/docker-ssd/docker/appdata/whisper:/root/.cache/huggingface
  1577|    deploy:
  1578|      resources:
  1579|        reservations:
  1580|          devices:
  1581|            - driver: nvidia
  1582|              count: 1
  1583|              capabilities: [gpu]
  1584|    networks:
  1585|      - ai-services
  1586|      - default
  1587|
  1588|  # -------------------------------------------------------
  1589|  # Ollama and OpenWebUI go here when deployed.
  1590|  # They share this stack and the default + ai-services networks.
  1591|  # OpenWebUI → Qdrant at http://qdrant:6333
  1592|  # OpenWebUI → LiteLLM at http://litellm:4000 (or direct Ollama)
  1593|  # -------------------------------------------------------
  1594|
  1595|  # ollama:
  1596|  #   image: ollama/ollama:latest
  1597|  #   container_name: ollama
  1598|  #   ...
  1599|
  1600|  # openwebui:
  1601|  #   image: ghcr.io/open-webui/open-webui:main
  1602|  #   container_name: openwebui
  1603|  #   ...
  1604|
  1605|networks:
  1606|  ai-services:
  1607|    name: ai-services
  1608|```
  1609|
  1610|> **GPU sharing note:** On PD, the RTX 2080 Ti is shared between Plex (hardware transcoding), Immich ML, and Ollama. faster-whisper will also need the GPU when processing audio. These workloads are bursty (not constant), so time-sharing the GPU is viable — but be aware that a large Whisper transcription job will temporarily impact Ollama inference latency. If this becomes an issue, consider running Whisper on N.O.M.A.D.'s GTX 1080 instead and change `whisper`'s Ollama-style routing accordingly.
  1611|
  1612|> **Reranker on CPU:** The reranker uses the `cpu-latest` image variant intentionally — do NOT pin to `cpu-1.5` as it has an hf-hub compatibility bug. Reranking is a lightweight operation (scoring ~50 chunks takes <1s on CPU) and doesn't justify competing for GPU VRAM. The model loads via safetensors fallback since no ONNX files exist for bge-reranker-v2-m3. Max batch size is 4 on CPU. If you find latency is an issue, a GPU variant exists (`ghcr.io/huggingface/text-embeddings-inference:turing-latest` for your 2080 Ti).
  1613|
  1614|### 7.5 — .env.example (updated ai stack)
  1615|
  1616|```bash
  1617|# /mnt/docker-ssd/docker/compose/ai/.env.example
  1618|
  1619|TZ=America/New_York
  1620|
  1621|# LiteLLM
  1622|LITELLM_MASTER_KEY=sk-CHANGE_ME
  1623|
  1624|# Reranker
  1625|RERANKER_MODEL=BAAI/bge-reranker-v2-m3
  1626|
  1627|# Whisper
  1628|WHISPER_MODEL=Systran/faster-distil-whisper-large-v3
  1629|```
  1630|
  1631|### 7.6 — Validate & Deploy
  1632|
  1633|```bash
  1634|cd /mnt/docker-ssd/docker/compose/ai
  1635|cp .env.example .env
  1636|nano .env  # Set LITELLM_MASTER_KEY=$(openssl rand -hex 32)
  1637|
  1638|docker compose --env-file .env config
  1639|docker compose --env-file .env up -d
  1640|```
  1641|
  1642|### 7.7 — Post-Deploy Verification
  1643|
  1644|```bash
  1645|# All containers running?
  1646|docker ps --filter name=qdrant --filter name=litellm --filter name=reranker --filter name=whisper
  1647|
  1648|# LiteLLM health?
  1649|curl -s http://localhost:4000/health
  1650|
  1651|# LiteLLM can see all models?
  1652|curl -s http://localhost:4000/v1/models \
  1653|  -H "Authorization: Bearer *** | python3 -m json.tool
  1654|
  1655|# Test a chat completion through LiteLLM
  1656|curl -s http://localhost:4000/v1/chat/completions \
  1657|  -H "Authorization: Bearer *** \
  1658|  -H "Content-Type: application/json" \
  1659|  -d '{
  1660|    "model": "medium",
  1661|    "messages": [{"role": "user", "content": "Hello, which model are you?"}]
  1662|  }' | python3 -m json.tool
  1663|
  1664|# Test embeddings through LiteLLM
  1665|curl -s http://localhost:4000/v1/embeddings \
  1666|  -H "Authorization: Bearer *** \
  1667|  -H "Content-Type: application/json" \
  1668|  -d '{
  1669|    "model": "embed",
  1670|    "input": "test embedding"
  1671|  }' | python3 -m json.tool
  1672|```
  1673|
  1674|### 7.8 — Integration Updates
  1675|
  1676|With LiteLLM deployed, all apps should point at `http://litellm:4000` (within the `ai-services` network) or `http://PD_IP:4000` (from external hosts) instead of direct Ollama endpoints.
  1677|
  1678|**OpenWebUI:** Settings → Connections → add OpenAI-compatible endpoint:
  1679|- URL: `http://litellm:4000/v1`
  1680|- API Key: your `LITELLM_MASTER_KEY`
  1681|- This gives OpenWebUI access to all models across all three machines through one connection
  1682|
  1683|**Continue.dev:** Update `~/.continue/config.json`:
  1684|```json
  1685|{
  1686|  "models": [
  1687|    {
  1688|      "title": "Heavy (ROCINANTE)",
  1689|      "provider": "openai",
  1690|      "model": "heavy",
  1691|      "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
  1692|      "apiKey": "YOUR_MASTER_KEY"
  1693|    },
  1694|    {
  1695|      "title": "Medium (PD)",
  1696|      "provider": "openai",
  1697|      "model": "medium",
  1698|      "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
  1699|      "apiKey": "YOUR_MASTER_KEY"
  1700|    }
  1701|  ],
  1702|  "tabAutocompleteModel": {
  1703|    "title": "Light (N.O.M.A.D.)",
  1704|    "provider": "openai",
  1705|    "model": "light",
  1706|    "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
  1707|    "apiKey": "YOUR_MASTER_KEY"
  1708|  }
  1709|}
  1710|```
  1711|
  1712|> **Key change:** Continue.dev now uses `provider: "openai"` instead of `provider: "ollama"` and points at LiteLLM. One IP to remember, and if you add or move models between machines, you only update `config.yaml` — not every app.
  1713|
  1714|**n8n workflows:** All HTTP Request nodes that call Ollama directly should be updated:
  1715|- Old: `http://10.5.30.7:11434/api/generate` (N.O.M.A.D.)
  1716|- New: `http://litellm:4000/v1/chat/completions` with `model: "light"`
  1717|- n8n is on the `ai-services` network, so it reaches LiteLLM by Docker DNS
  1718|
  1719|**n8n Multi-Model Query Router workflow:** This workflow is now **simplified dramatically** — instead of a Switch node routing to three different Ollama IPs, it becomes a single HTTP Request node that passes the model tier name (`light`, `medium`, `heavy`) to LiteLLM and lets the gateway handle routing. The Switch node logic can be removed entirely.
  1720|
  1721|---
  1722|
  1723|## Phase 8 — Reranker (RAG Quality)
  1724|
  1725|**Host:** PlausibleDeniability (deployed as part of the `ai` stack in Phase 7)
  1726|**Service:** `reranker` container (already in the compose above)
  1727|**Why:** Dramatically improves RAG answer quality by filtering out noisy retrieval results before they reach the LLM.
  1728|
  1729|### 8.1 — How Reranking Works
  1730|
  1731|Without a reranker, your RAG pipeline does:
  1732|```
  1733|Query → embed → Qdrant top-10 by vector similarity → all 10 chunks go to LLM
  1734|```
  1735|
  1736|The problem: vector similarity often returns "close but irrelevant" chunks. The LLM gets noisy context and hallucinates.
  1737|
  1738|With a reranker:
  1739|```
  1740|Query → embed → Qdrant top-50 by vector similarity → reranker scores each (query, chunk) pair → top-5 by relevance go to LLM
  1741|```
  1742|
  1743|The reranker is a cross-encoder that reads the full query AND each chunk together, producing a much more accurate relevance score than vector distance alone. It over-retrieves cheaply from Qdrant, then precisely filters.
  1744|
  1745|### 8.2 — Scaffold Directories
  1746|
  1747|```bash
  1748|# Model cache on SSD (reranker model is ~1.1GB, downloaded on first start)
  1749|sudo mkdir -p /mnt/docker-ssd/docker/appdata/reranker
  1750|```
  1751|
  1752|### 8.3 — Post-Deploy Verification
  1753|
  1754|```bash
  1755|# Container running?
  1756|docker ps --filter name=reranker
  1757|
  1758|# Health check?
  1759|curl -s http://localhost:8787/health
  1760|
  1761|# Test reranking
  1762|curl -s http://localhost:8787/rerank \
  1763|  -H "Content-Type: application/json" \
  1764|  -d '{
  1765|    "query": "How do I restart a Docker container?",
  1766|    "texts": [
  1767|      "Use docker restart <container_name> to restart a running container.",
  1768|      "Docker was founded in 2013 by Solomon Hykes.",
  1769|      "The docker compose down command stops and removes containers.",
  1770|      "Kubernetes pods can be restarted by deleting them."
  1771|    ]
  1772|  }' | python3 -m json.tool
  1773|# Expected: the first text scores highest, second scores lowest
  1774|```
  1775|
  1776|### 8.4 — n8n RAG Pipeline Integration
  1777|
  1778|Update the "Qdrant Index Updater" and any RAG query workflows to include a reranking step.
  1779|
  1780|**Updated RAG Query Workflow (for OpenWebUI or any n8n-based query):**
  1781|```
  1782|Trigger:  Webhook node (POST with query)
  1783|Step 1:   HTTP Request → LiteLLM /v1/embeddings
  1784|            - model: "embed"
  1785|            - input: query text
  1786|Step 2:   HTTP Request → Qdrant API (POST /collections/{name}/points/search)
  1787|            - vector: embedding from step 1
  1788|            - limit: 50  (over-retrieve)
  1789|Step 3:   HTTP Request → Reranker (POST http://reranker:80/rerank)
  1790|            - query: original query text
  1791|            - texts: array of 50 chunk texts from Qdrant results
  1792|Step 4:   Code node → take top 5 by reranker score, format as context
  1793|Step 5:   HTTP Request → LiteLLM /v1/chat/completions
  1794|            - model: "medium" (or "heavy" for complex queries)
  1795|            - messages: system prompt with top-5 context + user query
  1796|Step 6:   Return response via webhook
  1797|```
  1798|
  1799|> **Key difference from the original plan:** Step 2 now retrieves 50 results instead of 10, and Step 3 (reranking) filters down to the best 5. This "over-retrieve then rerank" pattern is the standard approach for production RAG systems.
  1800|
  1801|---
  1802|
  1803|## Phase 9 — faster-whisper (Speech-to-Text)
  1804|
  1805|**Host:** PlausibleDeniability (deployed as part of the `ai` stack in Phase 7)
  1806|**Service:** `whisper` container (already in the compose above)
  1807|**Why:** Replaces shelved Scriberr with an OpenAI-compatible STT API. No SQLite dependency, no ZFS/ACL issues.
  1808|
  1809|### 9.1 — Scaffold Directories
  1810|
  1811|```bash
  1812|# Model cache on SSD (Whisper models are 1-3GB)
  1813|sudo mkdir -p /mnt/docker-ssd/docker/appdata/whisper
  1814|```
  1815|
  1816|### 9.2 — Post-Deploy Verification
  1817|
  1818|```bash
  1819|# Container running?
  1820|docker ps --filter name=whisper
  1821|
  1822|# Health check?
  1823|curl -s http://localhost:8786/health
  1824|
  1825|# Test transcription with a sample audio file
  1826|curl -s http://localhost:8786/v1/audio/transcriptions \
  1827|  -F "file=@/path/to/test-audio.wav" \
  1828|  -F "model=Systran/faster-distil-whisper-large-v3" \
  1829|  | python3 -m json.tool
  1830|```
  1831|
  1832|### 9.3 — n8n Integration Workflows
  1833|
  1834|**Workflow: Voice Note → Text → Summary**
  1835|```
  1836|Trigger:  Webhook node (POST with audio file in body)
  1837|Step 1:   HTTP Request → Whisper (POST http://whisper:8000/v1/audio/transcriptions)
  1838|            - Multipart form with audio file
  1839|            - response_format: "json"
  1840|Step 2:   HTTP Request → LiteLLM /v1/chat/completions
  1841|            - model: "medium"
  1842|            - Prompt: "Summarize this voice note concisely: {transcript}"
  1843|Step 3:   HTTP Request → Gotify → push summary to phone
  1844|Step 4:   (Optional) Postgres node → log transcript and summary
  1845|```
  1846|
  1847|**Workflow: Audio File → Paperless Document**
  1848|```
  1849|Trigger:  Webhook or filesystem watcher
  1850|Step 1:   HTTP Request → Whisper → get transcript
  1851|Step 2:   Code node → format transcript as text document
  1852|Step 3:   HTTP Request → Paperless API (POST /api/documents/post_document/)
  1853|            - Upload transcript as .txt
  1854|            - Tag: "transcription"
  1855|Step 4:   Gotify notification: "Audio transcribed and filed: {title}"
  1856|```
  1857|
  1858|**Home Assistant Voice Integration:**
  1859|If you want HA voice commands, Whisper can serve as the STT backend:
  1860|1. In Home Assistant → Settings → Voice Assistants
  1861|2. Add speech-to-text provider: "Whisper" at `http://PD_IP:8786`
  1862|3. Pair with Piper TTS (future addition) for full voice assistant loop
  1863|
  1864|### 9.4 — GPU vs CPU Considerations
  1865|
  1866|The compose file above uses the GPU variant with CUDA. If GPU contention with Ollama becomes an issue:
  1867|
  1868|**Option A — CPU fallback on PD:**
  1869|Change the image from `latest-cuda` to `fedirz/faster-whisper-server:latest-cpu` and remove the `deploy.resources` block and the CUDA environment variables (`WHISPER__DEVICE`, `WHISPER__COMPUTE_TYPE`). Transcription will be slower (~4x real-time instead of ~20x) but won't compete for VRAM.
  1870|
  1871|**Option B — Move to N.O.M.A.D.:**
  1872|Run Whisper on N.O.M.A.D.'s GTX 1080 which is less contested. Add it to N.O.M.A.D.'s compose or run standalone. N.O.M.A.D.'s 8GB VRAM handles Whisper models comfortably since phi4 doesn't fill it.
  1873|
  1874|---
  1875|
  1876|## Additional Tools Setup
  1877|
  1878|### Continue.dev (Local AI Code Completion)
  1879|
  1880|1. Install the Continue extension in VS Code
  1881|2. Create/edit `~/.continue/config.json`:
  1882|
  1883|**With LiteLLM (recommended — see Phase 7):**
  1884|```json
  1885|{
  1886|  "models": [
  1887|    {
  1888|      "title": "Heavy (ROCINANTE)",
  1889|      "provider": "openai",
  1890|      "model": "heavy",
  1891|      "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
  1892|      "apiKey": "YOUR_LITELLM_MASTER_KEY"
  1893|    },
  1894|    {
  1895|      "title": "Medium (PD)",
  1896|      "provider": "openai",
  1897|      "model": "medium",
  1898|      "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
  1899|      "apiKey": "YOUR_LITELLM_MASTER_KEY"
  1900|    }
  1901|  ],
  1902|  "tabAutocompleteModel": {
  1903|    "title": "Light (N.O.M.A.D.)",
  1904|    "provider": "openai",
  1905|    "model": "light",
  1906|    "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
  1907|    "apiKey": "YOUR_LITELLM_MASTER_KEY"
  1908|  }
  1909|}
  1910|```
  1911|
  1912|> One IP, one API key, all three machines. If you add or move models, update LiteLLM's `config.yaml` — not every app.
  1913|
  1914|**Without LiteLLM (direct Ollama, if Phase 7 is not yet deployed):**
  1915|```json
  1916|{
  1917|  "models": [
  1918|    {
  1919|      "title": "PD - qwen2.5:14b",
  1920|      "provider": "ollama",
  1921|      "model": "qwen2.5:14b",
  1922|      "apiBase": "http://PD_TAILSCALE_IP:11434"
  1923|    },
  1924|    {
  1925|      "title": "ROCINANTE - qwen3:32b",
  1926|      "provider": "ollama",
  1927|      "model": "qwen3:32b",
  1928|      "apiBase": "http://ROCINANTE_TAILSCALE_IP:11434"
  1929|    }
  1930|  ],
  1931|  "tabAutocompleteModel": {
  1932|    "title": "N.O.M.A.D. - phi4",
  1933|    "provider": "ollama",
  1934|    "model": "phi4",
  1935|    "apiBase": "http://NOMAD_TAILSCALE_IP:11434"
  1936|  }
  1937|}
  1938|```
  1939|
  1940|### Obsidian + Gitea Sync
  1941|
  1942|1. In Obsidian, install the "Obsidian Git" community plugin
  1943|2. Initialize your vault as a git repo:
  1944|   ```bash
  1945|   cd /path/to/obsidian/vault
  1946|   git init
  1947|   git remote add origin http://PD_IP:3000/fizzlepoof/obsidian-vault.git
  1948|   ```
  1949|3. In Obsidian Git settings:
  1950|   - Auto backup interval: 10 minutes
  1951|   - Pull on startup: enabled
  1952|4. Create the repo in Gitea first: `http://PD_IP:3000` → New Repository → `obsidian-vault`
  1953|5. Add a Gitea webhook to trigger the n8n "Qdrant Index Updater" workflow (see Phase 3)
  1954|
  1955|### Homepage Ollama Widget
  1956|
  1957|Add to your Homepage configuration (`/mnt/tank/docker/appdata/homepage/services.yaml`):
  1958|
  1959|```yaml
  1960|- AI:
  1961|    - Ollama (PD):
  1962|        icon: ollama.svg
  1963|        href: http://PD_IP:11434
  1964|        widget:
  1965|          type: ollama
  1966|          url: http://PD_IP:11434
  1967|    - Ollama (ROCINANTE):
  1968|        icon: ollama.svg
  1969|        href: http://ROCINANTE_IP:11434
  1970|        widget:
  1971|          type: ollama
  1972|          url: http://ROCINANTE_IP:11434
  1973|```
  1974|
  1975|---
  1976|
  1977|## Deployment Order Summary
  1978|
  1979|```
  1980|Week 1:  Phase 1 — Gotify
  1981|           └─ Deploy, create app tokens, install phone app
  1982|           └─ Test: send manual notification via API
  1983|
  1984|Week 1:  Phase 2 — Qdrant
  1985|           └─ Deploy, verify REST API
  1986|           └─ Create initial collections (empty, ready for n8n)
  1987|
  1988|Week 2:  Phase 3 — n8n
  1989|           └─ Deploy, create admin account
  1990|           └─ Build workflows incrementally:
  1991|              Day 1: Gitea commit → Gotify (simplest, proves the pipeline)
  1992|              Day 2: Sonarr/Radarr → TMDB → Gotify + Discord
  1993|              Day 3: Tautulli play logging + weekly digest
  1994|              Day 4: Uptime Kuma enhanced alerts
  1995|              Day 5: ZFS pool monitoring
  1996|              Day 6: Multi-model query router
  1997|              Day 7: Qdrant index updater
  1998|
  1999|Week 3:  Phase 4 — Paperless-NGX
  2000|           └─ Deploy, ingest test documents
  2001|