diff --git a/HOMELAB_BUILDOUT_PLAN.md b/HOMELAB_BUILDOUT_PLAN.md index f80c202..37b6bd5 100644 --- a/HOMELAB_BUILDOUT_PLAN.md +++ b/HOMELAB_BUILDOUT_PLAN.md @@ -1,2078 +1,1706 @@ -# Homelab Expansion — Full Buildout Plan - -> Generated: 2026-05-04 -> Based on: HOMELAB_EXPANSION_PLAN.md, ARCHITECTURE_OVERVIEW.md, STACK_STANDARDS.md - -This document contains everything needed to deploy all six expansion phases — docker-compose files, .env templates, directory scaffolding commands, database init SQL, Pangolin configuration, and validation steps. Each phase follows the established stack standards and deployment checklist. - ---- - -## Table of Contents - -1. [Prerequisites & Conventions](#prerequisites--conventions) -2. [Phase 1 — Gotify (Notifications)](#phase-1--gotify-notifications) -3. [Phase 2 — Qdrant (Vector Database)](#phase-2--qdrant-vector-database) -4. [Phase 3 — n8n (Workflow Automation)](#phase-3--n8n-workflow-automation) -5. [Phase 4 — Paperless-NGX (Document Intelligence)](#phase-4--paperless-ngx-document-intelligence) -6. [Phase 5 — Home Assistant (Home Automation)](#phase-5--home-assistant-home-automation) -7. [Phase 6 — Grafana + Prometheus (Observability)](#phase-6--grafana--prometheus-observability) -8. [Phase 7 — LiteLLM (AI Gateway)](#phase-7--litellm-ai-gateway) -9. [Phase 8 — Reranker (RAG Quality)](#phase-8--reranker-rag-quality) -10. [Phase 9 — faster-whisper (Speech-to-Text)](#phase-9--faster-whisper-speech-to-text) -11. [Additional Tools Setup](#additional-tools-setup) -12. [Deployment Order Summary](#deployment-order-summary) -13. [Post-Deployment Validation Master Checklist](#post-deployment-validation-master-checklist) - ---- - -## Prerequisites & Conventions - -### Deployment Standards (recap) - -All stacks on PlausibleDeniability follow the same pattern: - -- **Repo root:** `/mnt/docker-ssd/docker/compose` -- **Validate before deploy:** `docker compose --env-file .env config` -- **Deploy:** `docker compose --env-file .env up -d` -- **Teardown:** `docker compose --env-file .env down` - -### Storage Tiers - -| Tier | Mount | Use For | -|------|-------|---------| -| SSD | `/mnt/docker-ssd/docker/appdata/` | Write-heavy, SQLite, GPU/model, databases | -| Tank | `/mnt/tank/docker/appdata/` | General appdata, configs, uploads | -| Unraid | `/mnt/unraid/data/media/` | Media libraries only | - -### Networks - -| Network | Created By | Purpose | -|---------|-----------|---------| -| `ix-databases_shared-databases` | databases stack | Access to shared-postgres, shared-mariadb, shared-redis | -| `pangolin` | newt (infrastructure stack) | Reverse proxy / external exposure | - -### Secret Generation Commands - -```bash -# Database passwords -openssl rand -hex 24 - -# JWT / encryption keys -openssl rand -hex 32 - -# Paperless secret key -python3 -c "import secrets; print(secrets.token_urlsafe(50))" -``` - -### Deployment Order - -``` -databases (already running) → infrastructure (already running) - → Phase 1: Gotify - → Phase 2: Qdrant (creates the ai-services network) - → Phase 3: n8n (depends on Gotify + Qdrant — joins ai-services as external) - → Phase 4: Paperless-NGX (depends on n8n for automation hooks) - → Phase 5: Home Assistant (depends on n8n for heavy automation) - → Phase 6: Grafana + Prometheus (on N.O.M.A.D., independent but benefits from all above) -``` - -> **Important:** The `ai` stack (Phase 2) must be deployed before the `automation` stack (Phase 3) because n8n declares `ai-services` as an external network. If the `ai` stack isn't up, `docker compose up` for `automation` will fail with a missing network error. - ---- - -## Phase 1 — Gotify (Notifications) - -**Host:** PlausibleDeniability -**Stack directory:** `/mnt/docker-ssd/docker/compose/automation/` -**Why first:** Every subsequent phase sends notifications through Gotify. It's the output bus. - -### 1.1 — Scaffold Directories - -```bash -# Appdata on SSD (SQLite backend — must not be on NFS) -sudo mkdir -p /mnt/docker-ssd/docker/appdata/gotify - -# Stack directory (may already exist if automation/ is planned) -sudo mkdir -p /mnt/docker-ssd/docker/compose/automation -``` - -### 1.2 — Database Init - -None required — Gotify uses an embedded SQLite database stored in its data volume. - -### 1.3 — docker-compose.yaml - -Add to `/mnt/docker-ssd/docker/compose/automation/docker-compose.yaml`: - -```yaml -name: automation - -services: - gotify: - image: gotify/server:2.6.1 - container_name: gotify - restart: unless-stopped - ports: - - "8484:80" - environment: - TZ: ${TZ} - GOTIFY_DEFAULTUSER_PASS: ${GOTIFY_ADMIN_PASS} - GOTIFY_SERVER_PORT: 80 - GOTIFY_DATABASE_DIALECT: sqlite3 - GOTIFY_DATABASE_CONNECTION: data/gotify.db - volumes: - - /mnt/docker-ssd/docker/appdata/gotify:/app/data - networks: - - pangolin - - default - -networks: - pangolin: - external: true -``` - -> **Image note:** `gotify/server:2.6.1` is the latest stable as of May 2026. Check [Docker Hub](https://hub.docker.com/r/gotify/server/tags) before deploying — pin to the exact version. - -### 1.4 — .env.example - -```bash -# /mnt/docker-ssd/docker/compose/automation/.env.example - -TZ=America/New_York -GOTIFY_ADMIN_PASS=CHANGE_ME -``` - -### 1.5 — .env (create from example) - -```bash -cd /mnt/docker-ssd/docker/compose/automation -cp .env.example .env -# Edit .env with real values: -# GOTIFY_ADMIN_PASS=$(openssl rand -hex 16) -nano .env -``` - -### 1.6 — Pangolin Configuration - -In your Pangolin dashboard, create a new resource: - -| Field | Value | -|-------|-------| -| Domain | `gotify.paccoco.com` | -| Scheme | `http` | -| Host | `gotify` | -| Port | `80` | -| Network | Docker service name resolution via `pangolin` network | - -### 1.7 — Validate & Deploy - -```bash -cd /mnt/docker-ssd/docker/compose/automation -docker compose --env-file .env config -docker compose --env-file .env up -d -``` - -### 1.8 — Post-Deploy Verification - -```bash -# Container running? -docker ps --filter name=gotify - -# Clean startup logs? -docker logs gotify --tail 20 - -# Mounts correct? -docker inspect gotify --format '{{json .Mounts}}' | python3 -m json.tool - -# Quick health check -curl -s http://localhost:8484/health - -# Test notification via API -curl -s "http://localhost:8484/message?token=YOUR_APP_TOKEN" \ - -F "title=Homelab" \ - -F "message=Gotify is online" \ - -F "priority=5" -``` - -### 1.9 — First-Run Setup - -1. Navigate to `https://gotify.paccoco.com` -2. Log in with the admin password from `.env` -3. **Change the default admin password** in the UI -4. Create application tokens for each notification source: - - `n8n-workflows` — for all n8n automations - - `infrastructure-alerts` — for Uptime Kuma, Grafana, etc. - - `media-notifications` — for Sonarr/Radarr/Tautulli hooks - - `home-assistant` — for HA automations -5. Create client tokens for each receiving device (phone, desktop) -6. Install the Gotify Android app and configure with your client token - ---- - -## Phase 2 — AI Stack (Vector DB, LLM, Embeddings, Reranker, STT) - -**Host:** PlausibleDeniability -**Stack directory:** `/mnt/docker-ssd/docker/compose/ai/` -**Status:** DEPLOYED AND VERIFIED (2026-05-05) - -> **DEPLOYED NOTE:** This phase was deployed as a complete AI stack with 6 services: Ollama (11434), OpenWebUI (8282), Qdrant (6333/6334), LiteLLM (4000), Reranker/TEI (8787), and faster-whisper (8786). The compose below shows the original Qdrant-only plan — see the actual running compose on PD at `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml` and the `project_ai_stack_deployed.md` memory for port mappings and fixes. Key lesson: always use `10.5.1.6` not `localhost` for health checks on TrueNAS Scale. - -### 2.1 — Scaffold Directories - -```bash -# Appdata on SSD (write-heavy vector storage) -sudo mkdir -p /mnt/docker-ssd/docker/appdata/qdrant/storage -sudo mkdir -p /mnt/docker-ssd/docker/appdata/qdrant/snapshots - -# Stack directory -sudo mkdir -p /mnt/docker-ssd/docker/compose/ai -``` - -### 2.2 — docker-compose.yaml - -Create or update `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml`: - -```yaml -name: ai - -services: - qdrant: - image: qdrant/qdrant:v1.14.0 - container_name: qdrant - restart: unless-stopped - ports: - - "6333:6333" # REST API - - "6334:6334" # gRPC - environment: - TZ: ${TZ} - QDRANT__SERVICE__GRPC_PORT: 6334 - QDRANT__STORAGE__STORAGE_PATH: /qdrant/storage - QDRANT__STORAGE__SNAPSHOTS_PATH: /qdrant/snapshots - volumes: - - /mnt/docker-ssd/docker/appdata/qdrant/storage:/qdrant/storage - - /mnt/docker-ssd/docker/appdata/qdrant/snapshots:/qdrant/snapshots - networks: - - ai-services - - default - - # ------------------------------------------------------- - # Ollama and OpenWebUI go here when deployed. - # They share this stack and the default + ai-services networks so - # OpenWebUI can reach Qdrant at http://qdrant:6333 - # ------------------------------------------------------- - - # ollama: - # image: ollama/ollama:latest - # container_name: ollama - # ... - - # openwebui: - # image: ghcr.io/open-webui/open-webui:main - # container_name: openwebui - # ... - -networks: - ai-services: - name: ai-services -``` - -> **Image note:** `qdrant/qdrant:v1.14.0` is the latest stable as of May 2026. Check [GitHub releases](https://github.com/qdrant/qdrant/releases) before deploying. - -> **Cross-stack connectivity:** The `ai-services` network is defined here with an explicit `name:` so other stacks (like `automation/n8n`) can declare it as `external: true` and reach Qdrant by service name (`http://qdrant:6333`). This follows the same pattern as `ix-databases_shared-databases`. - -### 2.3 — .env.example - -```bash -# /mnt/docker-ssd/docker/compose/ai/.env.example - -TZ=America/New_York -``` - -### 2.4 — Validate & Deploy - -```bash -cd /mnt/docker-ssd/docker/compose/ai -cp .env.example .env -nano .env # Set timezone - -docker compose --env-file .env config -docker compose --env-file .env up -d -``` - -### 2.5 — Post-Deploy Verification - -```bash -# Container running? -docker ps --filter name=qdrant - -# Clean startup? -docker logs qdrant --tail 20 - -# REST API responding? -curl -s http://localhost:6333/healthz -# Expected: {"title":"qdrant - vectorass engine","version":"1.14.0","commit":"..."} - -# Create a test collection -curl -X PUT http://localhost:6333/collections/test_collection \ - -H "Content-Type: application/json" \ - -d '{ - "vectors": { - "size": 384, - "distance": "Cosine" - } - }' - -# Verify it exists -curl -s http://localhost:6333/collections | python3 -m json.tool - -# Clean up test -curl -X DELETE http://localhost:6333/collections/test_collection -``` - -### 2.6 — OpenWebUI Integration (when deployed) - -When Ollama and OpenWebUI are brought online in this same stack, configure OpenWebUI's RAG settings: - -1. Go to OpenWebUI → Admin → Settings → Documents -2. Set the vector database to Qdrant -3. Endpoint: `http://qdrant:6333` (Docker DNS within the `ai` stack network) -4. Collection name: `openwebui_docs` (or your preference) - -### 2.7 — Collections to Create (for n8n in Phase 3) - -These collections will be created programmatically by n8n workflows, but for reference: - -| Collection | Vector Size | Content | -|-----------|-------------|---------| -| `homelab_docs` | 384 (nomic-embed-text) | Homelab markdown documentation | -| `gitea_commits` | 384 | Gitea commit messages + diffs | -| `media_metadata` | 384 | Plex/Tautulli metadata | -| `obsidian_notes` | 384 | Personal notes from Obsidian vault | - -> **Vector size note:** 384 is the dimension for `nomic-embed-text` via Ollama. If you use a different embedding model, adjust accordingly. - ---- - -## Phase 3 — n8n (Workflow Automation) - -**Host:** PlausibleDeniability -**Stack directory:** `/mnt/docker-ssd/docker/compose/automation/` (same stack as Gotify) -**Why third:** n8n is the orchestration backbone — it ties Gotify, Qdrant, Ollama, and all triggers together. - -### 3.1 — Scaffold Directories - -```bash -# Config on tank (workflow definitions, credentials store) -# NOTE: n8n also writes execution logs here. If execution logging becomes -# heavy (many workflows running frequently), consider moving to SSD. -# For typical homelab usage (~20 workflows), tank is fine. -sudo mkdir -p /mnt/tank/docker/appdata/n8n - -# Set ownership — n8n runs as UID 1000 (node user) -sudo chown 1000:1000 /mnt/tank/docker/appdata/n8n -``` - -### 3.2 — Database Init - -n8n uses the existing shared-postgres. Create its database and user: - -```bash -docker exec -i shared-postgres psql -U postgres <<'SQL' -CREATE USER n8n WITH PASSWORD 'REPLACE_WITH_GENERATED_PASSWORD'; -CREATE DATABASE n8n OWNER n8n; -GRANT ALL PRIVILEGES ON DATABASE n8n TO n8n; -SQL -``` - -Generate the password first: -```bash -openssl rand -hex 24 -``` - -### 3.3 — docker-compose.yaml - -Update `/mnt/docker-ssd/docker/compose/automation/docker-compose.yaml` to add n8n alongside Gotify: - -```yaml -name: automation - -services: - gotify: - image: gotify/server:2.6.1 - container_name: gotify - restart: unless-stopped - ports: - - "8484:80" - environment: - TZ: ${TZ} - GOTIFY_DEFAULTUSER_PASS: ${GOTIFY_ADMIN_PASS} - GOTIFY_SERVER_PORT: 80 - GOTIFY_DATABASE_DIALECT: sqlite3 - GOTIFY_DATABASE_CONNECTION: data/gotify.db - volumes: - - /mnt/docker-ssd/docker/appdata/gotify:/app/data - networks: - - pangolin - - default - - n8n: - image: docker.n8n.io/n8nio/n8n:1.88.0 - container_name: n8n - restart: unless-stopped - ports: - - "5678:5678" - environment: - TZ: ${TZ} - # Database - DB_TYPE: postgresdb - DB_POSTGRESDB_HOST: shared-postgres - DB_POSTGRESDB_PORT: 5432 - DB_POSTGRESDB_DATABASE: ${N8N_DB_NAME} - DB_POSTGRESDB_USER: ${N8N_DB_USER} - DB_POSTGRESDB_PASSWORD: ${N8N_DB_PASS} - # Security - N8N_ENCRYPTION_KEY: ${N8N_ENCRYPTION_KEY} - # Webhook / Reverse Proxy - N8N_HOST: ${N8N_HOST} - N8N_PROTOCOL: https - N8N_PORT: 5678 - WEBHOOK_URL: https://${N8N_HOST}/ - N8N_PROXY_HOPS: 1 - # General - GENERIC_TIMEZONE: ${TZ} - N8N_DIAGNOSTICS_ENABLED: false - N8N_PERSONALIZATION_ENABLED: false - volumes: - - /mnt/tank/docker/appdata/n8n:/home/node/.n8n - networks: - - pangolin - - ix-databases_shared-databases - - ai-services - - default - depends_on: - - gotify - -networks: - pangolin: - external: true - ix-databases_shared-databases: - external: true - ai-services: - external: true -``` - -> **Image note:** `docker.n8n.io/n8nio/n8n:1.88.0` — n8n uses calendar-ish versioning. Check [n8n releases](https://github.com/n8n-io/n8n/releases) for the latest stable. The `docker.n8n.io` registry is preferred for production. - -> **Cross-stack access:** n8n joins `ai-services` (created by the `ai` stack) so it can reach Qdrant at `http://qdrant:6333` by Docker DNS. It also joins `ix-databases_shared-databases` for Postgres access — same pattern. - -### 3.4 — .env.example (updated for both services) - -```bash -# /mnt/docker-ssd/docker/compose/automation/.env.example - -TZ=America/New_York - -# Gotify -GOTIFY_ADMIN_PASS=CHANGE_ME - -# n8n — Database -N8N_DB_NAME=n8n -N8N_DB_USER=n8n -N8N_DB_PASS=CHANGE_ME - -# n8n — Security -N8N_ENCRYPTION_KEY=CHANGE_ME - -# n8n — Hostname -N8N_HOST=n8n.paccoco.com -``` - -### 3.5 — Pangolin Configuration - -| Field | Value | -|-------|-------| -| Domain | `n8n.paccoco.com` | -| Scheme | `http` | -| Host | `n8n` | -| Port | `5678` | -| Headers | Forward `X-Forwarded-Proto: https` | - -### 3.6 — Validate & Deploy - -```bash -cd /mnt/docker-ssd/docker/compose/automation - -# Update .env with real passwords -nano .env - -docker compose --env-file .env config -docker compose --env-file .env up -d -``` - -### 3.7 — Post-Deploy Verification - -```bash -# Both containers running? -docker ps --filter name=gotify --filter name=n8n - -# n8n logs clean? -docker logs n8n --tail 30 - -# n8n on correct networks? -docker inspect n8n --format '{{json .NetworkSettings.Networks}}' | python3 -m json.tool - -# n8n UI accessible? -curl -s -o /dev/null -w "%{http_code}" http://localhost:5678/ -# Expected: 200 - -# Postgres connection working? (check logs for DB migration messages) -docker logs n8n 2>&1 | grep -i "migrat" -``` - -### 3.8 — First-Run Setup - -1. Navigate to `https://n8n.paccoco.com` -2. Create your admin account -3. Install community nodes you'll need: - - `n8n-nodes-gotify` (if available) or use HTTP Request node - - Ollama nodes (built-in as of n8n 1.x) - -### 3.9 — Workflow Blueprints - -Below are starter workflow descriptions for each planned automation. These are meant to be built in the n8n UI — the structure and node types are described so you can wire them up. - -#### Media Pipeline Workflows - -**Workflow: Sonarr/Radarr Download Notification** -``` -Trigger: Webhook node (POST from Sonarr/Radarr on download/import) -Step 1: Extract series/movie name, quality, file path from webhook body -Step 2: HTTP Request → TMDB API to fetch poster image URL -Step 3: HTTP Request → Gotify REST API (POST /message) - - Title: "New Download: {title}" - - Message: "{quality} — {episodeTitle or year}" - - Priority: 5 - - Extras: attach poster URL as markdown image -Also: HTTP Request → Discord webhook (formatted embed with poster) -``` - -Configure Sonarr/Radarr webhooks: -- Sonarr: Settings → Connect → Webhook → URL: `https://n8n.paccoco.com/webhook/sonarr` -- Radarr: Settings → Connect → Webhook → URL: `https://n8n.paccoco.com/webhook/radarr` - -**Workflow: Tautulli Play Logging** -``` -Trigger: Webhook node (POST from Tautulli on playback start) -Step 1: Extract user, media title, player, quality from payload -Step 2: Postgres node → INSERT into watch_history table -Step 3: (Optional) Gotify notification for specific users/media -``` - -Tautulli config: Settings → Notification Agents → Webhook → URL: `https://n8n.paccoco.com/webhook/tautulli-play` - -**Workflow: Weekly Watch Digest** -``` -Trigger: Cron node (every Sunday at 10:00 AM) -Step 1: Postgres node → SELECT watch history for past 7 days -Step 2: Format data as structured text -Step 3: HTTP Request → Ollama API (POST to PD's qwen2.5:14b) - - Prompt: "Summarize this week's viewing in a fun digest: {data}" -Step 4: HTTP Request → Gotify (send digest) -``` - -#### Infrastructure Monitoring Workflows - -**Workflow: Uptime Kuma Enhanced Alerts** -``` -Trigger: Webhook node (from Uptime Kuma notification) -Step 1: Extract monitor name, status, response time -Step 2: HTTP Request → Netdata API for related metrics context -Step 3: HTTP Request → Gotify - - Title: "🔴 {monitor} DOWN" or "🟢 {monitor} UP" - - Message: include Netdata context (CPU, mem, disk at time of alert) - - Priority: 8 (high for down, 3 for recovery) -``` - -**Workflow: ZFS Pool Utilization Alert** -``` -Trigger: Cron node (every 6 hours) -Step 1: SSH node → Serenity: `zpool list -Hp malcolm` -Step 2: Parse capacity percentage -Step 3: IF capacity > 85% → Gotify alert (priority 8) -Step 4: IF capacity > 90% → Gotify alert (priority 10) + Discord webhook -``` - -**Workflow: Grafana Alert Remediation** -``` -Trigger: Webhook node (from Grafana alerting) -Step 1: Parse alert labels (container, host, metric) -Step 2: Switch node → route by alert type: - - High CPU container → SSH → docker restart {container} - - Disk full → SSH → pause qBittorrent, notify via Gotify - - Memory pressure → Gotify alert only (manual intervention) -Step 3: Log action taken to Postgres -``` - -#### Homelab Ops Workflows - -**Workflow: Gitea Commit Summary** -``` -Trigger: Webhook node (Gitea webhook on push to truenas-stacks) -Step 1: Extract commit messages, author, files changed -Step 2: HTTP Request → Ollama API - - Prompt: "Summarize this commit in one sentence: {commit_message}" -Step 3: HTTP Request → Gotify - - Title: "Commit to truenas-stacks" - - Message: Ollama-generated summary -``` - -Gitea config: Repository → Settings → Webhooks → URL: `https://n8n.paccoco.com/webhook/gitea-push` - -**Workflow: qBittorrent Auto-Rescan** -``` -Trigger: Webhook or polling (qBittorrent API for completed+moved torrents) -Step 1: Determine if file is in Sonarr or Radarr path -Step 2: HTTP Request → Sonarr API (POST /command → RescanSeries) - OR HTTP Request → Radarr API (POST /command → RescanMovie) -Step 3: Gotify notification confirming rescan triggered -``` - -#### AI Pipeline Workflows - -**Workflow: URL Digest Pipeline** -``` -Trigger: Webhook node (POST with URL in body) -Step 1: HTTP Request → fetch page content -Step 2: Code node → extract text, chunk into ~500 token segments -Step 3: HTTP Request → Ollama API → summarize each chunk -Step 4: Code node → combine summaries into digest -Step 5: Postgres node → store digest with metadata -Step 6: Return digest in webhook response -``` - -**Workflow: Multi-Model Query Router (simplified by LiteLLM — see Phase 7)** -``` -Trigger: Webhook node (POST with query + complexity hint) -Step 1: HTTP Request → LiteLLM (POST http://litellm:4000/v1/chat/completions) - - model: complexity parameter ("light", "medium", or "heavy") - - LiteLLM handles routing to the correct Ollama instance -Step 2: Return response via webhook -``` -> With LiteLLM deployed, the Switch node and three separate Ollama endpoints -> are replaced by a single HTTP Request node. The routing logic lives in -> LiteLLM's config.yaml instead of n8n workflow logic. - -**Workflow: Qdrant Index Updater** -``` -Trigger: Webhook node (from Gitea push to any watched repo) -Step 1: HTTP Request → Gitea API → fetch changed files content -Step 2: Code node → chunk text into embedding-sized segments -Step 3: HTTP Request → Ollama API (embed endpoint with nomic-embed-text) -Step 4: HTTP Request → Qdrant API (PUT /collections/homelab_docs/points) -Step 5: Gotify notification: "{n} documents re-indexed" -``` - -#### Home / Business Workflows - -**Workflow: KitchenOwl Grocery Notification** -``` -Trigger: Polling (KitchenOwl API) or webhook if supported -Step 1: Fetch current shopping list items -Step 2: Format as clean text list -Step 3: HTTP Request → Gotify → phone push notification -``` - -**Workflow: Donetick Task Reminder** -``` -Trigger: Cron node (daily at 9:00 AM) -Step 1: HTTP Request → Donetick API → fetch tasks due today/overdue -Step 2: Format task list -Step 3: HTTP Request → Gotify (priority 5) -``` - -**Workflow: Long and Low Crafts Order Pipeline** -``` -Trigger: Webhook (Etsy webhook or email trigger via IMAP node) -Step 1: Parse order details (item, quantity, customer, shipping) -Step 2: HTTP Request → Donetick API → create fulfillment task -Step 3: HTTP Request → Gotify DM notification - - Title: "New L&L Order" - - Message: "{item} x{qty} — ship by {date}" - - Priority: 7 -``` - ---- - -## Phase 4 — Paperless-NGX (Document Intelligence) - -**Host:** PlausibleDeniability -**Stack directory:** `/mnt/docker-ssd/docker/compose/documents/` (new stack) -**Why fourth:** Depends on n8n for automation hooks and Gotify for notifications. - -### 4.1 — Scaffold Directories - -```bash -# Index/data on SSD (search index is write-heavy) -sudo mkdir -p /mnt/docker-ssd/docker/appdata/paperless/data - -# Documents (media) on tank (bulk storage, read-heavy) -sudo mkdir -p /mnt/tank/docker/appdata/paperless/media - -# Consume folder on tank (drop zone for new documents) -sudo mkdir -p /mnt/tank/docker/appdata/paperless/consume - -# Export folder on tank -sudo mkdir -p /mnt/tank/docker/appdata/paperless/export - -# Stack directory -sudo mkdir -p /mnt/docker-ssd/docker/compose/documents -``` - -### 4.2 — Database Init - -Paperless uses the existing shared-postgres and shared-redis: - -```bash -docker exec -i shared-postgres psql -U postgres <<'SQL' -CREATE USER paperless WITH PASSWORD 'REPLACE_WITH_GENERATED_PASSWORD'; -CREATE DATABASE paperless OWNER paperless; -GRANT ALL PRIVILEGES ON DATABASE paperless TO paperless; -SQL -``` - -### 4.3 — docker-compose.yaml - -Create `/mnt/docker-ssd/docker/compose/documents/docker-compose.yaml`: - -```yaml -name: documents - -services: - paperless: - image: ghcr.io/paperless-ngx/paperless-ngx:2.16 - container_name: paperless - restart: unless-stopped - ports: - - "8000:8000" - environment: - TZ: ${TZ} - # Database - PAPERLESS_DBENGINE: postgresql - PAPERLESS_DBHOST: shared-postgres - PAPERLESS_DBPORT: 5432 - PAPERLESS_DBNAME: ${PAPERLESS_DB_NAME} - PAPERLESS_DBUSER: ${PAPERLESS_DB_USER} - PAPERLESS_DBPASS: ${PAPERLESS_DB_PASS} - # Redis (using shared-redis) - PAPERLESS_REDIS: redis://shared-redis:6379 - # Security - PAPERLESS_SECRET_KEY: ${PAPERLESS_SECRET_KEY} - PAPERLESS_URL: https://${PAPERLESS_HOST} - PAPERLESS_ADMIN_USER: ${PAPERLESS_ADMIN_USER} - PAPERLESS_ADMIN_PASSWORD: ${PAPERLESS_ADMIN_PASS} - # OCR - PAPERLESS_OCR_LANGUAGE: eng - PAPERLESS_OCR_MODE: skip - # Tika/Gotenberg for Office documents - PAPERLESS_TIKA_ENABLED: 1 - PAPERLESS_TIKA_ENDPOINT: http://tika:9998 - PAPERLESS_TIKA_GOTENBERG_ENDPOINT: http://gotenberg:3000 - # Performance - PAPERLESS_TASK_WORKERS: 2 - PAPERLESS_THREADS_PER_WORKER: 2 - volumes: - - /mnt/docker-ssd/docker/appdata/paperless/data:/usr/src/paperless/data - - /mnt/tank/docker/appdata/paperless/media:/usr/src/paperless/media - - /mnt/tank/docker/appdata/paperless/consume:/usr/src/paperless/consume - - /mnt/tank/docker/appdata/paperless/export:/usr/src/paperless/export - networks: - - pangolin - - ix-databases_shared-databases - - default - depends_on: - - gotenberg - - tika - - gotenberg: - image: gotenberg/gotenberg:8.17 - container_name: paperless-gotenberg - restart: unless-stopped - command: - - "gotenberg" - - "--chromium-disable-javascript=true" - - "--chromium-allow-list=file:///tmp/.*" - networks: - - default - - tika: - image: apache/tika:3.1 - container_name: paperless-tika - restart: unless-stopped - networks: - - default - -networks: - pangolin: - external: true - ix-databases_shared-databases: - external: true -``` - -> **Image notes:** -> - `ghcr.io/paperless-ngx/paperless-ngx:2.16` — check [releases](https://github.com/paperless-ngx/paperless-ngx/releases) for latest. -> - `gotenberg/gotenberg:8.17` — pin to a specific 8.x tag. -> - `apache/tika:3.1` — needed for Office doc support (.docx, .xlsx, .odt). -> - Gotenberg and Tika are only needed if you ingest Office documents. For PDF-only, you can omit them and set `PAPERLESS_TIKA_ENABLED: 0`. - -### 4.4 — .env.example - -```bash -# /mnt/docker-ssd/docker/compose/documents/.env.example - -TZ=America/New_York - -# Paperless — Database -PAPERLESS_DB_NAME=paperless -PAPERLESS_DB_USER=paperless -PAPERLESS_DB_PASS=CHANGE_ME - -# Paperless — Security -PAPERLESS_SECRET_KEY=CHANGE_ME -PAPERLESS_ADMIN_USER=admin -PAPERLESS_ADMIN_PASS=CHANGE_ME - -# Paperless — Hostname -PAPERLESS_HOST=paperless.paccoco.com -``` - -### 4.5 — Pangolin Configuration - -| Field | Value | -|-------|-------| -| Domain | `paperless.paccoco.com` | -| Scheme | `http` | -| Host | `paperless` | -| Port | `8000` | - -### 4.6 — Validate & Deploy - -```bash -cd /mnt/docker-ssd/docker/compose/documents -cp .env.example .env -nano .env # Fill in real values - -docker compose --env-file .env config -docker compose --env-file .env up -d -``` - -### 4.7 — Post-Deploy Verification - -```bash -# All three containers running? -docker ps --filter name=paperless --filter name=paperless-gotenberg --filter name=paperless-tika - -# Paperless logs clean? (watch for DB migration output) -docker logs paperless --tail 30 - -# Network connectivity to shared-postgres and shared-redis? -docker exec paperless python3 -c "import psycopg2; print('postgres OK')" 2>/dev/null || echo "Check DB connection" - -# UI accessible? -curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/ -# Expected: 200 or 302 (redirect to login) -``` - -### 4.8 — n8n Integration Workflows - -**Workflow: Email Attachment → Paperless** -``` -Trigger: IMAP node (poll email inbox every 5 minutes) -Step 1: Filter for emails with PDF/document attachments -Step 2: Download attachment binary -Step 3: HTTP Request → Paperless API (POST /api/documents/post_document/) - - Multipart form with document file - - Optional: set correspondent, document type, tags -Step 4: Wait node (30 seconds for OCR processing) -Step 5: HTTP Request → Paperless API (GET document details) -Step 6: IF tagged as "action-required" → - HTTP Request → Donetick API → create task -Step 7: HTTP Request → Gotify → "New document ingested: {title}" -``` - -**Workflow: Ollama Document Summarizer** -``` -Trigger: Paperless webhook (on document consumed) or n8n polling -Step 1: HTTP Request → Paperless API → fetch document text -Step 2: HTTP Request → Ollama API (PD's qwen2.5:14b) - - Prompt: "Extract key data from this document: dates, amounts, parties, action items. {text}" -Step 3: HTTP Request → Paperless API → update document notes with summary -Step 4: HTTP Request → Gotify → "Document summarized: {title}" -``` - ---- - -## Phase 5 — Home Assistant (Home Automation) - -**Host:** PlausibleDeniability (preferred — same network as most services) -**Stack directory:** `/mnt/docker-ssd/docker/compose/homeassistant/` (standalone stack — host networking isolates it) -**Why fifth:** Benefits from n8n being operational. n8n handles complex automation logic while HA handles device control. - -### 5.1 — Scaffold Directories - -```bash -# Config on SSD (SQLite database, write-heavy) -sudo mkdir -p /mnt/docker-ssd/docker/appdata/homeassistant -``` - -### 5.2 — docker-compose.yaml - -Create `/mnt/docker-ssd/docker/compose/homeassistant/docker-compose.yaml`: - -```yaml -name: homeassistant - -services: - homeassistant: - image: ghcr.io/home-assistant/home-assistant:2026.5 - container_name: homeassistant - restart: unless-stopped - privileged: true - network_mode: host - environment: - TZ: ${TZ} - volumes: - - /mnt/docker-ssd/docker/appdata/homeassistant:/config - - /etc/localtime:/etc/localtime:ro - - /run/dbus:/run/dbus:ro - # devices: - # - /dev/ttyUSB0:/dev/ttyUSB0 # Uncomment for USB Zigbee/Z-Wave sticks -``` - -> **Image note:** `ghcr.io/home-assistant/home-assistant:2026.5` — use the latest stable monthly release. Check [HA releases](https://www.home-assistant.io/blog/categories/release-notes/). Pin to a specific minor like `2026.5.1` once you confirm it's stable. - -> **Why host networking:** Home Assistant requires `network_mode: host` for mDNS/SSDP device discovery. This means it does NOT join `pangolin` — you'll access it directly by IP and port, or configure Pangolin to proxy to `http://10.5.1.X:8123`. - -> **Why standalone stack:** Host networking is incompatible with other services in the same compose that use bridge networking. HA must be in its own compose file. - -### 5.3 — .env.example - -```bash -# /mnt/docker-ssd/docker/compose/homeassistant/.env.example - -TZ=America/New_York -``` - -### 5.4 — Pangolin Configuration - -Since HA uses host networking, Pangolin needs to reach it by the host's IP: - -| Field | Value | -|-------|-------| -| Domain | `ha.paccoco.com` | -| Scheme | `http` | -| Host | `10.5.1.X` (PlausibleDeniability's LAN IP) | -| Port | `8123` | - -> Find PD's IP with: `hostname -I | awk '{print $1}'` - -### 5.5 — Validate & Deploy - -```bash -cd /mnt/docker-ssd/docker/compose/homeassistant -cp .env.example .env -nano .env - -docker compose --env-file .env config -docker compose --env-file .env up -d -``` - -### 5.6 — Post-Deploy Verification - -```bash -# Container running? -docker ps --filter name=homeassistant - -# Clean startup? -docker logs homeassistant --tail 30 - -# Web UI accessible? -curl -s -o /dev/null -w "%{http_code}" http://localhost:8123/ -# Expected: 200 -``` - -### 5.7 — First-Run Setup & Integrations - -1. Navigate to `http://PD_IP:8123` (or `https://ha.paccoco.com`) -2. Complete the onboarding wizard -3. Add integrations: - -**Smart Plug Power Monitoring:** -- Install TP-Link Kasa / Tapo / Shelly integration (depends on your plug brand) -- Add plugs for Serenity, PD, N.O.M.A.D. -- Create energy dashboard for power consumption tracking - -**UPS Monitoring:** -- Install NUT (Network UPS Tools) integration if you add a UPS -- Note: PD and N.O.M.A.D. currently have no UPS — this is a known gap -- When UPS is added, create automations for graceful shutdown via n8n - -**Presence Detection:** -- Mobile app integration for phone-based presence -- Or use router/UniFi integration for network-based detection - -### 5.8 — n8n ↔ Home Assistant Integration - -In n8n, use the Home Assistant nodes (built-in): - -1. In HA: Profile → Long-Lived Access Tokens → Create Token -2. In n8n: Settings → Credentials → Home Assistant API - - Host: `http://10.5.1.X:8123` - - Access Token: (paste from step 1) - -**Workflow: Presence → Server Sleep/Wake** -``` -Trigger: HA event node (person.fizzlepoof state change) -Step 1: IF state = "not_home" for > 30 min AND no active Plex streams: - SSH → N.O.M.A.D.: systemctl suspend -Step 2: IF state = "home": - Wake-on-LAN → N.O.M.A.D. MAC address -Step 3: Gotify notification: "Server {action}: N.O.M.A.D." -``` - -**Workflow: Physical Device → n8n Trigger** -``` -Trigger: HA webhook or event node (smart plug power draw spike) -Step 1: IF washing machine plug power < 5W for 5 min after being > 100W: - Gotify notification: "Laundry is done!" -``` - ---- - -## Phase 6 — Grafana + Prometheus (Observability) - -**Host:** N.O.M.A.D. (Ubuntu 25.10 — dedicated HDDs have headroom) -**Stack directory:** `/opt/monitoring/` (outside the N.O.M.A.D. project directory) -**Why last:** Observability benefits from all other services being online — more to monitor. - -### 6.1 — Scaffold Directories - -SSH into N.O.M.A.D.: - -```bash -ssh nomad@10.5.1.16 - -# Stack directory -sudo mkdir -p /opt/monitoring - -# Prometheus data on hdd-2 (has more headroom) -sudo mkdir -p /mnt/hdd-2/prometheus-data -sudo chown 65534:65534 /mnt/hdd-2/prometheus-data # nobody user (Prometheus default) - -# Grafana data on hdd-2 -sudo mkdir -p /mnt/hdd-2/grafana-data -sudo chown 472:472 /mnt/hdd-2/grafana-data # grafana user - -# Config directories -sudo mkdir -p /opt/monitoring/provisioning/datasources -sudo mkdir -p /opt/monitoring/provisioning/dashboards -``` - -### 6.2 — prometheus.yml - -Create `/opt/monitoring/prometheus.yml`: - -```yaml -global: - scrape_interval: 15s - evaluation_interval: 15s - -scrape_configs: - # ------- Local (N.O.M.A.D.) ------- - - job_name: 'nomad-node' - static_configs: - - targets: ['node-exporter:9100'] - labels: - host: 'nomad' - - # ------- PlausibleDeniability ------- - - job_name: 'pd-netdata' - metrics_path: /api/v1/allmetrics - params: - format: [prometheus] - static_configs: - - targets: ['10.5.1.X:19999'] # Replace with PD's IP - labels: - host: 'plausible-deniability' - - # ------- Serenity ------- - - job_name: 'serenity-netdata' - metrics_path: /api/v1/allmetrics - params: - format: [prometheus] - static_configs: - - targets: ['10.5.1.5:19999'] - labels: - host: 'serenity' - - # ------- Gotify ------- - # Gotify exposes /health but no Prometheus endpoint natively. - # Use blackbox exporter or just rely on Uptime Kuma. - - # ------- n8n ------- - # n8n doesn't expose Prometheus metrics by default. - # Monitor via container resource metrics from Netdata. - - # -------- Add more targets as needed -------- - # When node-exporter is installed on PD and Serenity: - # - job_name: 'pd-node' - # static_configs: - # - targets: ['10.5.1.X:9100'] - # labels: - # host: 'plausible-deniability' - # - # - job_name: 'serenity-node' - # static_configs: - # - targets: ['10.5.1.5:9100'] - # labels: - # host: 'serenity' -``` - -> **Netdata as a Prometheus target:** Both PD and Serenity already run Netdata. Netdata has a built-in Prometheus exporter at `/api/v1/allmetrics?format=prometheus`. This gives you CPU, memory, disk, network, and ZFS metrics without installing node-exporter on those hosts. - -### 6.3 — Grafana Provisioning - -Create `/opt/monitoring/provisioning/datasources/prometheus.yml`: - -```yaml -apiVersion: 1 - -datasources: - - name: Prometheus - type: prometheus - access: proxy - url: http://prometheus:9090 - isDefault: true - editable: true -``` - -Create `/opt/monitoring/provisioning/dashboards/dashboards.yml`: - -```yaml -apiVersion: 1 - -providers: - - name: 'default' - orgId: 1 - folder: '' - type: file - disableDeletion: false - editable: true - options: - path: /var/lib/grafana/dashboards - foldersFromFilesStructure: false -``` - -### 6.4 — docker-compose.yaml - -Create `/opt/monitoring/docker-compose.yaml`: - -```yaml -name: monitoring - -services: - prometheus: - image: prom/prometheus:v3.4.0 - container_name: prometheus - restart: unless-stopped - ports: - - "9090:9090" - command: - - '--config.file=/etc/prometheus/prometheus.yml' - - '--storage.tsdb.path=/prometheus' - - '--storage.tsdb.retention.time=90d' - - '--web.enable-lifecycle' - volumes: - - /opt/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro - - /mnt/hdd-2/prometheus-data:/prometheus - networks: - - monitoring - - grafana: - image: grafana/grafana:13.0.1 - container_name: grafana - restart: unless-stopped - ports: - - "3000:3000" - environment: - TZ: ${TZ} - GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER} - GF_SECURITY_ADMIN_PASSWORD: ${GF_ADMIN_PASS} - GF_SERVER_ROOT_URL: https://${GF_HOST}/ - volumes: - - /mnt/hdd-2/grafana-data:/var/lib/grafana - - /opt/monitoring/provisioning:/etc/grafana/provisioning:ro - networks: - - monitoring - - node-exporter: - image: prom/node-exporter:v1.9.0 - container_name: node-exporter - restart: unless-stopped - ports: - - "9100:9100" - command: - - '--path.procfs=/host/proc' - - '--path.sysfs=/host/sys' - - '--path.rootfs=/rootfs' - - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)' - volumes: - - /proc:/host/proc:ro - - /sys:/host/sys:ro - - /:/rootfs:ro - networks: - - monitoring - -networks: - monitoring: - driver: bridge -``` - -> **Image notes:** -> - `prom/prometheus:v3.4.0` — Prometheus v3 is the current major line. **Do not use `:latest`** — it still resolves to 2.x due to a known tagging issue. Always use an explicit `v3.x.y` tag. -> - `grafana/grafana:latest` — use the moving stable image by default here; only pin Grafana if a breakage or compatibility reason is documented. -> - `prom/node-exporter:v1.9.0` — check [releases](https://github.com/prometheus/node_exporter/releases). - -### 6.5 — .env.example - -```bash -# /opt/monitoring/.env.example - -TZ=America/New_York - -# Grafana -GF_ADMIN_USER=admin -GF_ADMIN_PASS=CHANGE_ME -GF_HOST=grafana.paccoco.com -``` - -### 6.6 — Pangolin Configuration - -Grafana runs on N.O.M.A.D., not PD where the main Newt agent lives. You have two options: - -**Option A — Route via N.O.M.A.D.'s existing Newt (recommended if already connected)** - -N.O.M.A.D. already has a Newt container from the Project N.O.M.A.D. setup. If it's connected to your Pangolin VPS, just add a new resource in the Pangolin dashboard pointing to `http://grafana:3000` or `http://10.5.1.16:3000`. - -**Option B — Add a dedicated Newt to the monitoring stack** - -If N.O.M.A.D.'s existing Newt is not connected to Pangolin (or is a separate Pangolin instance), add Newt to the monitoring compose: - -```yaml - newt: - image: ghcr.io/fosrl/newt:latest - container_name: monitoring-newt - restart: unless-stopped - environment: - PANGOLIN_ENDPOINT: ${PANGOLIN_ENDPOINT} - NEWT_ID: ${NEWT_ID} - NEWT_SECRET: ${NEWT_SECRET} - networks: - - monitoring -``` - -Then in Pangolin dashboard: - -| Field | Value | -|-------|-------| -| Domain | `grafana.paccoco.com` | -| Scheme | `http` | -| Host | `grafana` (Docker DNS via shared network) | -| Port | `3000` | - -**Option C — Direct IP routing (simplest, no Newt needed)** - -If Pangolin's Newt on PD can reach N.O.M.A.D. by LAN IP (they're on the same subnet): - -| Field | Value | -|-------|-------| -| Domain | `grafana.paccoco.com` | -| Scheme | `http` | -| Host | `10.5.1.16` | -| Port | `3000` | - -> **Decision point:** Check if N.O.M.A.D.'s existing Newt is connected to your Pangolin instance before deploying. Run `docker ps --filter name=newt` on N.O.M.A.D. to verify. If it's running and connected, Option A is zero-effort. If not, Option C is the simplest fallback. - -### 6.7 — Validate & Deploy - -```bash -ssh nomad@10.5.1.16 - -cd /opt/monitoring -cp .env.example .env -nano .env # Fill in real values - -docker compose --env-file .env config -docker compose --env-file .env up -d -``` - -### 6.8 — Post-Deploy Verification - -```bash -# All three containers running? -docker ps --filter name=prometheus --filter name=grafana --filter name=node-exporter - -# Prometheus scraping targets? -curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | head -40 - -# Grafana UI accessible? -curl -s -o /dev/null -w "%{http_code}" http://localhost:3000/ -# Expected: 200 or 302 - -# Node exporter metrics flowing? -curl -s http://localhost:9100/metrics | head -10 -``` - -### 6.9 — Recommended Dashboards - -Import these from [Grafana Dashboard Library](https://grafana.com/grafana/dashboards/): - -| Dashboard | ID | Purpose | -|-----------|----|---------| -| Node Exporter Full | 1860 | System metrics for N.O.M.A.D. | -| Docker Container Stats | 893 | Container resource usage | -| Netdata via Prometheus | (search) | PD and Serenity system metrics | - -To import: Grafana → Dashboards → New → Import → Enter dashboard ID. - -### 6.10 — Metrics Targets Summary - -| Target | Host | Method | Endpoint | -|--------|------|--------|----------| -| N.O.M.A.D. system | localhost | node-exporter | node-exporter:9100 | -| PD system | 10.5.1.X | Netdata Prometheus | 10.5.1.X:19999/api/v1/allmetrics | -| Serenity system | 10.5.1.5 | Netdata Prometheus | 10.5.1.5:19999/api/v1/allmetrics | -| ZFS pools | via Netdata | Netdata exports ZFS metrics | Included in Netdata scrape | -| Container stats | via Netdata | Netdata exports cgroup metrics | Included in Netdata scrape | -| Plex streams | Tautulli | n8n polling → Postgres | Via n8n workflow (Phase 3) | -| qBit stats | qBittorrent API | n8n polling → Postgres | Via n8n workflow (Phase 3) | -| Tailscale latency | Tailscale API | n8n polling → Postgres | Via n8n workflow (Phase 3) | - -### 6.11 — n8n Integration (Grafana → n8n alert webhook) - -In Grafana → Alerting → Contact Points, create a webhook contact point: - -| Field | Value | -|-------|-------| -| Name | `n8n-alerts` | -| Type | Webhook | -| URL | `https://n8n.paccoco.com/webhook/grafana-alert` | -| HTTP Method | POST | - -Then create alert rules for: -- ZFS pool utilization > 85% -- Container memory > 90% of limit -- Host CPU sustained > 90% for 5 minutes -- Disk I/O latency spikes - -These fire into the "Grafana Alert Remediation" n8n workflow (see Phase 3). - ---- - -## Phase 7 — LiteLLM (AI Gateway) - -**Host:** PlausibleDeniability -**Stack directory:** `/mnt/docker-ssd/docker/compose/ai/` (same stack as Qdrant) -**Why:** Replaces manual multi-model routing with a unified OpenAI-compatible API. Every app (OpenWebUI, Continue.dev, n8n, Paperless summarization) points at one endpoint instead of juggling three Ollama IPs. - -### 7.1 — What LiteLLM Does - -LiteLLM sits in front of your three Ollama instances and presents a single OpenAI-compatible API at `http://litellm:4000`. It handles: - -- **Model routing** — requests for `qwen3:32b` go to ROCINANTE, `qwen2.5:14b` goes to PD, `phi4` goes to N.O.M.A.D. -- **Failover** — if ROCINANTE is offline, LiteLLM can fall back to PD automatically -- **Load balancing** — distribute requests across instances running the same model -- **Usage tracking** — logs token counts, latency, and costs per model/user via its built-in database - -### 7.2 — Scaffold Directories - -```bash -# Config on SSD (SQLite DB for usage tracking) -sudo mkdir -p /mnt/docker-ssd/docker/appdata/litellm - -# LiteLLM config file -sudo mkdir -p /mnt/docker-ssd/docker/compose/ai/litellm -``` - -### 7.3 — LiteLLM Config - -Create `/mnt/docker-ssd/docker/compose/ai/litellm/config.yaml`: - -```yaml -model_list: - # ---- ROCINANTE (RTX 4090, 24GB) — heavy reasoning ---- - - model_name: "heavy" - litellm_params: - model: "ollama/qwen3:32b" - api_base: "http://10.5.1.ROCINANTE:11434" - timeout: 300 - stream_timeout: 300 - model_info: - description: "Heavy reasoning, long context, complex code" - - - model_name: "heavy" - litellm_params: - model: "ollama/deepseek-r1:32b" - api_base: "http://10.5.1.ROCINANTE:11434" - timeout: 300 - stream_timeout: 300 - model_info: - description: "Deep reasoning fallback on ROCINANTE" - - # ---- PlausibleDeniability (RTX 2080 Ti, 11GB) — general ---- - - model_name: "medium" - litellm_params: - model: "ollama/qwen2.5:14b" - api_base: "http://host.docker.internal:11434" - timeout: 120 - stream_timeout: 120 - model_info: - description: "General homelab assistant, RAG queries" - - # ---- N.O.M.A.D. (GTX 1080, 8GB) — lightweight ---- - - model_name: "light" - litellm_params: - model: "ollama/phi4" - api_base: "http://10.5.1.16:11434" - timeout: 60 - stream_timeout: 60 - model_info: - description: "Fast local inference, lightweight tasks" - - - model_name: "light" - litellm_params: - model: "ollama/llama3.2:3b" - api_base: "http://10.5.1.16:11434" - timeout: 60 - stream_timeout: 60 - model_info: - description: "Ultra-light fallback on N.O.M.A.D." - - # ---- Embeddings ---- - - model_name: "embed" - litellm_params: - model: "ollama/nomic-embed-text" - api_base: "http://host.docker.internal:11434" - model_info: - description: "Text embeddings for RAG pipeline" - - # ---- Direct model access (bypass routing) ---- - # These let you request a specific model by its full name - - model_name: "ollama/qwen3:32b" - litellm_params: - model: "ollama/qwen3:32b" - api_base: "http://10.5.1.ROCINANTE:11434" - - - model_name: "ollama/qwen2.5:14b" - litellm_params: - model: "ollama/qwen2.5:14b" - api_base: "http://host.docker.internal:11434" - - - model_name: "ollama/phi4" - litellm_params: - model: "ollama/phi4" - api_base: "http://10.5.1.16:11434" - -litellm_settings: - drop_params: true - set_verbose: false - request_timeout: 300 - num_retries: 2 - retry_after: 5 - allowed_fails: 3 - cooldown_time: 60 - -general_settings: - master_key: "os.environ/LITELLM_MASTER_KEY" -``` - -> **DEPLOYED NOTE (2026-05-05):** Do NOT add `database_url` to general_settings — newer LiteLLM versions include Prisma ORM that requires PostgreSQL, and adding any database_url causes a crash loop. Without it, LiteLLM runs in config-only mode which is fine for homelab use. The `master_key` uses `os.environ/LITELLM_MASTER_KEY` syntax to read from the container's environment variable (set via .env → docker compose). - -> **Routing explained:** Multiple entries with the same `model_name` (like "heavy") enable load balancing and failover within that tier. When you request model `heavy`, LiteLLM picks the healthiest deployment. The direct-access entries (like `ollama/qwen3:32b`) let you bypass the tier system and target a specific model when needed. - -> **Replace IPs:** Update `10.5.1.ROCINANTE` with ROCINANTE's actual LAN or Tailscale IP. PD uses `host.docker.internal` since LiteLLM runs on PD alongside Ollama. - -### 7.4 — docker-compose.yaml (updated ai stack) - -Update `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml` to add LiteLLM: - -```yaml -name: ai - -services: - qdrant: - image: qdrant/qdrant:v1.14.0 - container_name: qdrant - restart: unless-stopped - ports: - - "6333:6333" # REST API - - "6334:6334" # gRPC - environment: - TZ: ${TZ} - QDRANT__SERVICE__GRPC_PORT: 6334 - QDRANT__STORAGE__STORAGE_PATH: /qdrant/storage - QDRANT__STORAGE__SNAPSHOTS_PATH: /qdrant/snapshots - volumes: - - /mnt/docker-ssd/docker/appdata/qdrant/storage:/qdrant/storage - - /mnt/docker-ssd/docker/appdata/qdrant/snapshots:/qdrant/snapshots - networks: - - ai-services - - default - - litellm: - image: ghcr.io/berriai/litellm:main-latest - container_name: litellm - restart: unless-stopped - ports: - - "4000:4000" - environment: - TZ: ${TZ} - LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY} - volumes: - - /mnt/docker-ssd/docker/compose/ai/litellm/config.yaml:/app/config.yaml:ro - - /mnt/docker-ssd/docker/appdata/litellm:/app/data - command: ["--config", "/app/config.yaml", "--port", "4000"] - extra_hosts: - - "host.docker.internal:host-gateway" - networks: - - ai-services - - default - - reranker: - image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest - container_name: reranker - restart: unless-stopped - ports: - - "8787:80" - environment: - MODEL_ID: ${RERANKER_MODEL} - volumes: - - /mnt/docker-ssd/docker/appdata/reranker:/data - networks: - - ai-services - - default - - whisper: - image: fedirz/faster-whisper-server:latest-cuda - container_name: whisper - restart: unless-stopped - ports: - - "8786:8000" - environment: - TZ: ${TZ} - WHISPER__MODEL: ${WHISPER_MODEL} - WHISPER__DEVICE: cuda - WHISPER__COMPUTE_TYPE: float16 - volumes: - - /mnt/docker-ssd/docker/appdata/whisper:/root/.cache/huggingface - deploy: - resources: - reservations: - devices: - - driver: nvidia - count: 1 - capabilities: [gpu] - networks: - - ai-services - - default - - # ------------------------------------------------------- - # Ollama and OpenWebUI go here when deployed. - # They share this stack and the default + ai-services networks. - # OpenWebUI → Qdrant at http://qdrant:6333 - # OpenWebUI → LiteLLM at http://litellm:4000 (or direct Ollama) - # ------------------------------------------------------- - - # ollama: - # image: ollama/ollama:latest - # container_name: ollama - # ... - - # openwebui: - # image: ghcr.io/open-webui/open-webui:main - # container_name: openwebui - # ... - -networks: - ai-services: - name: ai-services -``` - -> **GPU sharing note:** On PD, the RTX 2080 Ti is shared between Plex (hardware transcoding), Immich ML, and Ollama. faster-whisper will also need the GPU when processing audio. These workloads are bursty (not constant), so time-sharing the GPU is viable — but be aware that a large Whisper transcription job will temporarily impact Ollama inference latency. If this becomes an issue, consider running Whisper on N.O.M.A.D.'s GTX 1080 instead and change `whisper`'s Ollama-style routing accordingly. - -> **Reranker on CPU:** The reranker uses the `cpu-latest` image variant intentionally — do NOT pin to `cpu-1.5` as it has an hf-hub compatibility bug. Reranking is a lightweight operation (scoring ~50 chunks takes <1s on CPU) and doesn't justify competing for GPU VRAM. The model loads via safetensors fallback since no ONNX files exist for bge-reranker-v2-m3. Max batch size is 4 on CPU. If you find latency is an issue, a GPU variant exists (`ghcr.io/huggingface/text-embeddings-inference:turing-latest` for your 2080 Ti). - -### 7.5 — .env.example (updated ai stack) - -```bash -# /mnt/docker-ssd/docker/compose/ai/.env.example - -TZ=America/New_York - -# LiteLLM -LITELLM_MASTER_KEY=sk-CHANGE_ME - -# Reranker -RERANKER_MODEL=BAAI/bge-reranker-v2-m3 - -# Whisper -WHISPER_MODEL=Systran/faster-distil-whisper-large-v3 -``` - -### 7.6 — Validate & Deploy - -```bash -cd /mnt/docker-ssd/docker/compose/ai -cp .env.example .env -nano .env # Set LITELLM_MASTER_KEY=$(openssl rand -hex 32) - -docker compose --env-file .env config -docker compose --env-file .env up -d -``` - -### 7.7 — Post-Deploy Verification - -```bash -# All containers running? -docker ps --filter name=qdrant --filter name=litellm --filter name=reranker --filter name=whisper - -# LiteLLM health? -curl -s http://localhost:4000/health - -# LiteLLM can see all models? -curl -s http://localhost:4000/v1/models \ - -H "Authorization: Bearer YOUR_MASTER_KEY" | python3 -m json.tool - -# Test a chat completion through LiteLLM -curl -s http://localhost:4000/v1/chat/completions \ - -H "Authorization: Bearer YOUR_MASTER_KEY" \ - -H "Content-Type: application/json" \ - -d '{ - "model": "medium", - "messages": [{"role": "user", "content": "Hello, which model are you?"}] - }' | python3 -m json.tool - -# Test embeddings through LiteLLM -curl -s http://localhost:4000/v1/embeddings \ - -H "Authorization: Bearer YOUR_MASTER_KEY" \ - -H "Content-Type: application/json" \ - -d '{ - "model": "embed", - "input": "test embedding" - }' | python3 -m json.tool -``` - -### 7.8 — Integration Updates - -With LiteLLM deployed, all apps should point at `http://litellm:4000` (within the `ai-services` network) or `http://PD_IP:4000` (from external hosts) instead of direct Ollama endpoints. - -**OpenWebUI:** Settings → Connections → add OpenAI-compatible endpoint: -- URL: `http://litellm:4000/v1` -- API Key: your `LITELLM_MASTER_KEY` -- This gives OpenWebUI access to all models across all three machines through one connection - -**Continue.dev:** Update `~/.continue/config.json`: -```json -{ - "models": [ - { - "title": "Heavy (ROCINANTE)", - "provider": "openai", - "model": "heavy", - "apiBase": "http://PD_TAILSCALE_IP:4000/v1", - "apiKey": "YOUR_MASTER_KEY" - }, - { - "title": "Medium (PD)", - "provider": "openai", - "model": "medium", - "apiBase": "http://PD_TAILSCALE_IP:4000/v1", - "apiKey": "YOUR_MASTER_KEY" - } - ], - "tabAutocompleteModel": { - "title": "Light (N.O.M.A.D.)", - "provider": "openai", - "model": "light", - "apiBase": "http://PD_TAILSCALE_IP:4000/v1", - "apiKey": "YOUR_MASTER_KEY" - } -} -``` - -> **Key change:** Continue.dev now uses `provider: "openai"` instead of `provider: "ollama"` and points at LiteLLM. One IP to remember, and if you add or move models between machines, you only update `config.yaml` — not every app. - -**n8n workflows:** All HTTP Request nodes that call Ollama directly should be updated: -- Old: `http://10.5.1.16:11434/api/generate` (N.O.M.A.D.) -- New: `http://litellm:4000/v1/chat/completions` with `model: "light"` -- n8n is on the `ai-services` network, so it reaches LiteLLM by Docker DNS - -**n8n Multi-Model Query Router workflow:** This workflow is now **simplified dramatically** — instead of a Switch node routing to three different Ollama IPs, it becomes a single HTTP Request node that passes the model tier name (`light`, `medium`, `heavy`) to LiteLLM and lets the gateway handle routing. The Switch node logic can be removed entirely. - ---- - -## Phase 8 — Reranker (RAG Quality) - -**Host:** PlausibleDeniability (deployed as part of the `ai` stack in Phase 7) -**Service:** `reranker` container (already in the compose above) -**Why:** Dramatically improves RAG answer quality by filtering out noisy retrieval results before they reach the LLM. - -### 8.1 — How Reranking Works - -Without a reranker, your RAG pipeline does: -``` -Query → embed → Qdrant top-10 by vector similarity → all 10 chunks go to LLM -``` - -The problem: vector similarity often returns "close but irrelevant" chunks. The LLM gets noisy context and hallucinates. - -With a reranker: -``` -Query → embed → Qdrant top-50 by vector similarity → reranker scores each (query, chunk) pair → top-5 by relevance go to LLM -``` - -The reranker is a cross-encoder that reads the full query AND each chunk together, producing a much more accurate relevance score than vector distance alone. It over-retrieves cheaply from Qdrant, then precisely filters. - -### 8.2 — Scaffold Directories - -```bash -# Model cache on SSD (reranker model is ~1.1GB, downloaded on first start) -sudo mkdir -p /mnt/docker-ssd/docker/appdata/reranker -``` - -### 8.3 — Post-Deploy Verification - -```bash -# Container running? -docker ps --filter name=reranker - -# Health check? -curl -s http://localhost:8787/health - -# Test reranking -curl -s http://localhost:8787/rerank \ - -H "Content-Type: application/json" \ - -d '{ - "query": "How do I restart a Docker container?", - "texts": [ - "Use docker restart to restart a running container.", - "Docker was founded in 2013 by Solomon Hykes.", - "The docker compose down command stops and removes containers.", - "Kubernetes pods can be restarted by deleting them." - ] - }' | python3 -m json.tool -# Expected: the first text scores highest, second scores lowest -``` - -### 8.4 — n8n RAG Pipeline Integration - -Update the "Qdrant Index Updater" and any RAG query workflows to include a reranking step. - -**Updated RAG Query Workflow (for OpenWebUI or any n8n-based query):** -``` -Trigger: Webhook node (POST with query) -Step 1: HTTP Request → LiteLLM /v1/embeddings - - model: "embed" - - input: query text -Step 2: HTTP Request → Qdrant API (POST /collections/{name}/points/search) - - vector: embedding from step 1 - - limit: 50 (over-retrieve) -Step 3: HTTP Request → Reranker (POST http://reranker:80/rerank) - - query: original query text - - texts: array of 50 chunk texts from Qdrant results -Step 4: Code node → take top 5 by reranker score, format as context -Step 5: HTTP Request → LiteLLM /v1/chat/completions - - model: "medium" (or "heavy" for complex queries) - - messages: system prompt with top-5 context + user query -Step 6: Return response via webhook -``` - -> **Key difference from the original plan:** Step 2 now retrieves 50 results instead of 10, and Step 3 (reranking) filters down to the best 5. This "over-retrieve then rerank" pattern is the standard approach for production RAG systems. - ---- - -## Phase 9 — faster-whisper (Speech-to-Text) - -**Host:** PlausibleDeniability (deployed as part of the `ai` stack in Phase 7) -**Service:** `whisper` container (already in the compose above) -**Why:** Replaces shelved Scriberr with an OpenAI-compatible STT API. No SQLite dependency, no ZFS/ACL issues. - -### 9.1 — Scaffold Directories - -```bash -# Model cache on SSD (Whisper models are 1-3GB) -sudo mkdir -p /mnt/docker-ssd/docker/appdata/whisper -``` - -### 9.2 — Post-Deploy Verification - -```bash -# Container running? -docker ps --filter name=whisper - -# Health check? -curl -s http://localhost:8786/health - -# Test transcription with a sample audio file -curl -s http://localhost:8786/v1/audio/transcriptions \ - -F "file=@/path/to/test-audio.wav" \ - -F "model=Systran/faster-distil-whisper-large-v3" \ - | python3 -m json.tool -``` - -### 9.3 — n8n Integration Workflows - -**Workflow: Voice Note → Text → Summary** -``` -Trigger: Webhook node (POST with audio file in body) -Step 1: HTTP Request → Whisper (POST http://whisper:8000/v1/audio/transcriptions) - - Multipart form with audio file - - response_format: "json" -Step 2: HTTP Request → LiteLLM /v1/chat/completions - - model: "medium" - - Prompt: "Summarize this voice note concisely: {transcript}" -Step 3: HTTP Request → Gotify → push summary to phone -Step 4: (Optional) Postgres node → log transcript and summary -``` - -**Workflow: Audio File → Paperless Document** -``` -Trigger: Webhook or filesystem watcher -Step 1: HTTP Request → Whisper → get transcript -Step 2: Code node → format transcript as text document -Step 3: HTTP Request → Paperless API (POST /api/documents/post_document/) - - Upload transcript as .txt - - Tag: "transcription" -Step 4: Gotify notification: "Audio transcribed and filed: {title}" -``` - -**Home Assistant Voice Integration:** -If you want HA voice commands, Whisper can serve as the STT backend: -1. In Home Assistant → Settings → Voice Assistants -2. Add speech-to-text provider: "Whisper" at `http://PD_IP:8786` -3. Pair with Piper TTS (future addition) for full voice assistant loop - -### 9.4 — GPU vs CPU Considerations - -The compose file above uses the GPU variant with CUDA. If GPU contention with Ollama becomes an issue: - -**Option A — CPU fallback on PD:** -Change the image from `latest-cuda` to `fedirz/faster-whisper-server:latest-cpu` and remove the `deploy.resources` block and the CUDA environment variables (`WHISPER__DEVICE`, `WHISPER__COMPUTE_TYPE`). Transcription will be slower (~4x real-time instead of ~20x) but won't compete for VRAM. - -**Option B — Move to N.O.M.A.D.:** -Run Whisper on N.O.M.A.D.'s GTX 1080 which is less contested. Add it to N.O.M.A.D.'s compose or run standalone. N.O.M.A.D.'s 8GB VRAM handles Whisper models comfortably since phi4 doesn't fill it. - ---- - -## Additional Tools Setup - -### Continue.dev (Local AI Code Completion) - -1. Install the Continue extension in VS Code -2. Create/edit `~/.continue/config.json`: - -**With LiteLLM (recommended — see Phase 7):** -```json -{ - "models": [ - { - "title": "Heavy (ROCINANTE)", - "provider": "openai", - "model": "heavy", - "apiBase": "http://PD_TAILSCALE_IP:4000/v1", - "apiKey": "YOUR_LITELLM_MASTER_KEY" - }, - { - "title": "Medium (PD)", - "provider": "openai", - "model": "medium", - "apiBase": "http://PD_TAILSCALE_IP:4000/v1", - "apiKey": "YOUR_LITELLM_MASTER_KEY" - } - ], - "tabAutocompleteModel": { - "title": "Light (N.O.M.A.D.)", - "provider": "openai", - "model": "light", - "apiBase": "http://PD_TAILSCALE_IP:4000/v1", - "apiKey": "YOUR_LITELLM_MASTER_KEY" - } -} -``` - -> One IP, one API key, all three machines. If you add or move models, update LiteLLM's `config.yaml` — not every app. - -**Without LiteLLM (direct Ollama, if Phase 7 is not yet deployed):** -```json -{ - "models": [ - { - "title": "PD - qwen2.5:14b", - "provider": "ollama", - "model": "qwen2.5:14b", - "apiBase": "http://PD_TAILSCALE_IP:11434" - }, - { - "title": "ROCINANTE - qwen3:32b", - "provider": "ollama", - "model": "qwen3:32b", - "apiBase": "http://ROCINANTE_TAILSCALE_IP:11434" - } - ], - "tabAutocompleteModel": { - "title": "N.O.M.A.D. - phi4", - "provider": "ollama", - "model": "phi4", - "apiBase": "http://NOMAD_TAILSCALE_IP:11434" - } -} -``` - -### Obsidian + Gitea Sync - -1. In Obsidian, install the "Obsidian Git" community plugin -2. Initialize your vault as a git repo: - ```bash - cd /path/to/obsidian/vault - git init - git remote add origin http://PD_IP:3000/fizzlepoof/obsidian-vault.git - ``` -3. In Obsidian Git settings: - - Auto backup interval: 10 minutes - - Pull on startup: enabled -4. Create the repo in Gitea first: `http://PD_IP:3000` → New Repository → `obsidian-vault` -5. Add a Gitea webhook to trigger the n8n "Qdrant Index Updater" workflow (see Phase 3) - -### Homepage Ollama Widget - -Add to your Homepage configuration (`/mnt/tank/docker/appdata/homepage/services.yaml`): - -```yaml -- AI: - - Ollama (PD): - icon: ollama.svg - href: http://PD_IP:11434 - widget: - type: ollama - url: http://PD_IP:11434 - - Ollama (ROCINANTE): - icon: ollama.svg - href: http://ROCINANTE_IP:11434 - widget: - type: ollama - url: http://ROCINANTE_IP:11434 -``` - ---- - -## Deployment Order Summary - -``` -Week 1: Phase 1 — Gotify - └─ Deploy, create app tokens, install phone app - └─ Test: send manual notification via API - -Week 1: Phase 2 — Qdrant - └─ Deploy, verify REST API - └─ Create initial collections (empty, ready for n8n) - -Week 2: Phase 3 — n8n - └─ Deploy, create admin account - └─ Build workflows incrementally: - Day 1: Gitea commit → Gotify (simplest, proves the pipeline) - Day 2: Sonarr/Radarr → TMDB → Gotify + Discord - Day 3: Tautulli play logging + weekly digest - Day 4: Uptime Kuma enhanced alerts - Day 5: ZFS pool monitoring - Day 6: Multi-model query router - Day 7: Qdrant index updater - -Week 3: Phase 4 — Paperless-NGX - └─ Deploy, ingest test documents - └─ Build n8n workflow: email → Paperless → Ollama summary - └─ Set up document tags and correspondents for L&L Crafts - -Week 4: Phase 5 — Home Assistant - └─ Deploy, onboard, add integrations - └─ Connect to n8n via long-lived access token - └─ Set up smart plug monitoring - -Week 4: Phase 6 — Grafana + Prometheus (on N.O.M.A.D.) - └─ Deploy, verify scrape targets - └─ Import dashboards - └─ Set up Grafana → n8n alert webhook - └─ Build alert rules - -Week 5: Phase 7/8/9 — AI Stack Expansion (all deploy together in ai stack) - └─ LiteLLM: deploy, verify model routing across all 3 machines - └─ Reranker: deploy, test scoring with sample chunks - └─ faster-whisper: deploy, test transcription - └─ Update OpenWebUI, Continue.dev, and n8n to use LiteLLM endpoint - └─ Update RAG workflows to include reranking step - └─ Build voice note → transcription → summary workflow -``` - ---- - -## Post-Deployment Validation Master Checklist - -Run through this after all six phases are deployed: - -### Infrastructure Health - -- [ ] `docker ps` on PD shows: gotify, n8n, qdrant, litellm, reranker, whisper, paperless, paperless-gotenberg, paperless-tika, homeassistant — all healthy -- [ ] `docker ps` on N.O.M.A.D. shows: prometheus, grafana, node-exporter — all healthy -- [ ] All services accessible via Pangolin subdomains (gotify, n8n, paperless, ha, grafana) - -### Network Connectivity - -- [ ] n8n can reach shared-postgres (test: check n8n logs for successful DB migration) -- [ ] n8n can reach Gotify (test: trigger a workflow that sends a notification) -- [ ] n8n can reach Qdrant at `http://qdrant:6333` via `ai-services` network (test: query collections from n8n HTTP node) -- [ ] n8n can reach LiteLLM at `http://litellm:4000` via `ai-services` network (test: send a chat completion) -- [ ] LiteLLM can route to all 3 Ollama instances (test: request model "light", "medium", "heavy" and verify each responds) -- [ ] Reranker responds at `http://reranker:80/rerank` (test: POST sample texts) -- [ ] Whisper responds at `http://whisper:8000/v1/audio/transcriptions` (test: transcribe a sample .wav) -- [ ] Paperless can reach shared-postgres and shared-redis -- [ ] Prometheus can scrape PD and Serenity Netdata endpoints -- [ ] Grafana alert webhook reaches n8n - -### Data Flow End-to-End - -- [ ] Sonarr webhook → n8n → Gotify push on phone -- [ ] Gitea commit → n8n → Qdrant indexing → Gotify notification -- [ ] RAG query → Qdrant top-50 → reranker top-5 → LiteLLM → accurate answer -- [ ] Audio file → Whisper transcription → n8n → Gotify summary -- [ ] Document dropped in consume folder → Paperless OCR → n8n summary → Gotify -- [ ] Grafana alert fires → n8n webhook → remediation action or Gotify alert -- [ ] HA presence change → n8n → server wake/sleep - -### Backup Considerations - -- [ ] Gotify SQLite DB is on SSD — include `/mnt/docker-ssd/docker/appdata/gotify` in backup plan -- [ ] Qdrant storage on SSD — include `/mnt/docker-ssd/docker/appdata/qdrant` in backup plan -- [ ] n8n config/workflows — include `/mnt/tank/docker/appdata/n8n` in backup plan -- [ ] Paperless media — include `/mnt/tank/docker/appdata/paperless/media` in backup plan -- [ ] HA config — include `/mnt/docker-ssd/docker/appdata/homeassistant` in backup plan -- [ ] LiteLLM DB + config — include `/mnt/docker-ssd/docker/appdata/litellm` and `compose/ai/litellm/config.yaml` in backup plan -- [ ] Grafana/Prometheus data — include `/mnt/hdd-2/grafana-data` and `/mnt/hdd-2/prometheus-data` in N.O.M.A.D. backup plan -- [ ] All new databases (n8n, paperless) are in shared-postgres — ensure pg_dump covers them - -### Storage Capacity Check - -- [ ] `docker-ssd` (500GB Crucial MX500) still has headroom after adding Gotify, Qdrant, HA, Paperless data -- [ ] `tank` mirror has room for n8n config + Paperless media + consume/export -- [ ] N.O.M.A.D. hdd-2 has room for Prometheus TSDB (90-day retention) + Grafana data - ---- - -> **Remember:** Everything self-hosted. Nothing touches the cloud. + 1|# Homelab Expansion — Full Buildout Plan + 2| + 3|> Generated: 2026-05-04 + 4|> Based on: HOMELAB_EXPANSION_PLAN.md, ARCHITECTURE_OVERVIEW.md, STACK_STANDARDS.md + 5| + 6|This document contains everything needed to deploy all six expansion phases — docker-compose files, .env templates, directory scaffolding commands, database init SQL, Pangolin configuration, and validation steps. Each phase follows the established stack standards and deployment checklist. + 7| + 8|--- + 9| + 10|## Table of Contents + 11| + 12|1. [Prerequisites & Conventions](#prerequisites--conventions) + 13|2. [Phase 1 — Gotify (Notifications)](#phase-1--gotify-notifications) + 14|3. [Phase 2 — Qdrant (Vector Database)](#phase-2--qdrant-vector-database) + 15|4. [Phase 3 — n8n (Workflow Automation)](#phase-3--n8n-workflow-automation) + 16|5. [Phase 4 — Paperless-NGX (Document Intelligence)](#phase-4--paperless-ngx-document-intelligence) + 17|6. [Phase 5 — Home Assistant (Home Automation)](#phase-5--home-assistant-home-automation) + 18|7. [Phase 6 — Grafana + Prometheus (Observability)](#phase-6--grafana--prometheus-observability) + 19|8. [Phase 7 — LiteLLM (AI Gateway)](#phase-7--litellm-ai-gateway) + 20|9. [Phase 8 — Reranker (RAG Quality)](#phase-8--reranker-rag-quality) + 21|10. [Phase 9 — faster-whisper (Speech-to-Text)](#phase-9--faster-whisper-speech-to-text) + 22|11. [Additional Tools Setup](#additional-tools-setup) + 23|12. [Deployment Order Summary](#deployment-order-summary) + 24|13. [Post-Deployment Validation Master Checklist](#post-deployment-validation-master-checklist) + 25| + 26|--- + 27| + 28|## Prerequisites & Conventions + 29| + 30|### Deployment Standards (recap) + 31| + 32|All stacks on PlausibleDeniability follow the same pattern: + 33| + 34|- **Repo root:** `/mnt/docker-ssd/docker/compose` + 35|- **Validate before deploy:** `docker compose --env-file .env config` + 36|- **Deploy:** `docker compose --env-file .env up -d` + 37|- **Teardown:** `docker compose --env-file .env down` + 38| + 39|### Storage Tiers + 40| + 41|| Tier | Mount | Use For | + 42||------|-------|---------| + 43|| SSD | `/mnt/docker-ssd/docker/appdata/` | Write-heavy, SQLite, GPU/model, databases | + 44|| Tank | `/mnt/tank/docker/appdata/` | General appdata, configs, uploads | + 45|| Unraid | `/mnt/unraid/data/media/` | Media libraries only | + 46| + 47|### Networks + 48| + 49|| Network | Created By | Purpose | + 50||---------|-----------|---------| + 51|| `ix-databases_shared-databases` | databases stack | Access to shared-postgres, shared-mariadb, shared-redis | + 52|| `pangolin` | newt (infrastructure stack) | Reverse proxy / external exposure | + 53| + 54|### Secret Generation Commands + 55| + 56|```bash + 57|# Database passwords + 58|openssl rand -hex 24 + 59| + 60|# JWT / encryption keys + 61|openssl rand -hex 32 + 62| + 63|# Paperless secret key + 64|python3 -c "import secrets; print(secrets.token_urlsafe(50))" + 65|``` + 66| + 67|### Deployment Order + 68| + 69|``` + 70|databases (already running) → infrastructure (already running) + 71| → Phase 1: Gotify + 72| → Phase 2: Qdrant (creates the ai-services network) + 73| → Phase 3: n8n (depends on Gotify + Qdrant — joins ai-services as external) + 74| → Phase 4: Paperless-NGX (depends on n8n for automation hooks) + 75| → Phase 5: Home Assistant (depends on n8n for heavy automation) + 76| → Phase 6: Grafana + Prometheus (on N.O.M.A.D., independent but benefits from all above) + 77|``` + 78| + 79|> **Important:** The `ai` stack (Phase 2) must be deployed before the `automation` stack (Phase 3) because n8n declares `ai-services` as an external network. If the `ai` stack isn't up, `docker compose up` for `automation` will fail with a missing network error. + 80| + 81|--- + 82| + 83|## Phase 1 — Gotify (Notifications) + 84| + 85|**Host:** PlausibleDeniability + 86|**Stack directory:** `/mnt/docker-ssd/docker/compose/automation/` + 87|**Why first:** Every subsequent phase sends notifications through Gotify. It's the output bus. + 88| + 89|### 1.1 — Scaffold Directories + 90| + 91|```bash + 92|# Appdata on SSD (SQLite backend — must not be on NFS) + 93|sudo mkdir -p /mnt/docker-ssd/docker/appdata/gotify + 94| + 95|# Stack directory (may already exist if automation/ is planned) + 96|sudo mkdir -p /mnt/docker-ssd/docker/compose/automation + 97|``` + 98| + 99|### 1.2 — Database Init + 100| + 101|None required — Gotify uses an embedded SQLite database stored in its data volume. + 102| + 103|### 1.3 — docker-compose.yaml + 104| + 105|Add to `/mnt/docker-ssd/docker/compose/automation/docker-compose.yaml`: + 106| + 107|```yaml + 108|name: automation + 109| + 110|services: + 111| gotify: + 112| image: gotify/server:2.6.1 + 113| container_name: gotify + 114| restart: unless-stopped + 115| ports: + 116| - "8484:80" + 117| environment: + 118| TZ: ${TZ} + 119| GOTIFY_DEFAULTUSER_PASS: ${GOTIFY_ADMIN_PASS} + 120| GOTIFY_SERVER_PORT: 80 + 121| GOTIFY_DATABASE_DIALECT: sqlite3 + 122| GOTIFY_DATABASE_CONNECTION: data/gotify.db + 123| volumes: + 124| - /mnt/docker-ssd/docker/appdata/gotify:/app/data + 125| networks: + 126| - pangolin + 127| - default + 128| + 129|networks: + 130| pangolin: + 131| external: true + 132|``` + 133| + 134|> **Image note:** `gotify/server:2.6.1` is the latest stable as of May 2026. Check [Docker Hub](https://hub.docker.com/r/gotify/server/tags) before deploying — pin to the exact version. + 135| + 136|### 1.4 — .env.example + 137| + 138|```bash + 139|# /mnt/docker-ssd/docker/compose/automation/.env.example + 140| + 141|TZ=America/New_York + 142|GOTIFY_ADMIN_PASS=CHANGE_ME + 143|``` + 144| + 145|### 1.5 — .env (create from example) + 146| + 147|```bash + 148|cd /mnt/docker-ssd/docker/compose/automation + 149|cp .env.example .env + 150|# Edit .env with real values: + 151|# GOTIFY_ADMIN_PASS=$(openssl rand -hex 16) + 152|nano .env + 153|``` + 154| + 155|### 1.6 — Pangolin Configuration + 156| + 157|In your Pangolin dashboard, create a new resource: + 158| + 159|| Field | Value | + 160||-------|-------| + 161|| Domain | `gotify.paccoco.com` | + 162|| Scheme | `http` | + 163|| Host | `gotify` | + 164|| Port | `80` | + 165|| Network | Docker service name resolution via `pangolin` network | + 166| + 167|### 1.7 — Validate & Deploy + 168| + 169|```bash + 170|cd /mnt/docker-ssd/docker/compose/automation + 171|docker compose --env-file .env config + 172|docker compose --env-file .env up -d + 173|``` + 174| + 175|### 1.8 — Post-Deploy Verification + 176| + 177|```bash + 178|# Container running? + 179|docker ps --filter name=gotify + 180| + 181|# Clean startup logs? + 182|docker logs gotify --tail 20 + 183| + 184|# Mounts correct? + 185|docker inspect gotify --format '{{json .Mounts}}' | python3 -m json.tool + 186| + 187|# Quick health check + 188|curl -s http://localhost:8484/health + 189| + 190|# Test notification via API + 191|curl -s "http://localhost:8484/message?token=*** \ + 192| -F "title=Homelab" \ + 193| -F "message=Gotify is online" \ + 194| -F "priority=5" + 195|``` + 196| + 197|### 1.9 — First-Run Setup + 198| + 199|1. Navigate to `https://gotify.paccoco.com` + 200|2. Log in with the admin password from `.env` + 201|3. **Change the default admin password** in the UI + 202|4. Create application tokens for each notification source: + 203| - `n8n-workflows` — for all n8n automations + 204| - `infrastructure-alerts` — for Uptime Kuma, Grafana, etc. + 205| - `media-notifications` — for Sonarr/Radarr/Tautulli hooks + 206| - `home-assistant` — for HA automations + 207|5. Create client tokens for each receiving device (phone, desktop) + 208|6. Install the Gotify Android app and configure with your client token + 209| + 210|--- + 211| + 212|## Phase 2 — AI Stack (Vector DB, LLM, Embeddings, Reranker, STT) + 213| + 214|**Host:** PlausibleDeniability + 215|**Stack directory:** `/mnt/docker-ssd/docker/compose/ai/` + 216|**Status:** DEPLOYED AND VERIFIED (2026-05-05) + 217| + 218|> **DEPLOYED NOTE:** This phase was deployed as a complete AI stack with 6 services: Ollama (11434), OpenWebUI (8282), Qdrant (6333/6334), LiteLLM (4000), Reranker/TEI (8787), and faster-whisper (8786). The compose below shows the original Qdrant-only plan — see the actual running compose on PD at `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml` and the `project_ai_stack_deployed.md` memory for port mappings and fixes. Key lesson: always use `10.5.30.6` not `localhost` for health checks on TrueNAS Scale. + 219| + 220|### 2.1 — Scaffold Directories + 221| + 222|```bash + 223|# Appdata on SSD (write-heavy vector storage) + 224|sudo mkdir -p /mnt/docker-ssd/docker/appdata/qdrant/storage + 225|sudo mkdir -p /mnt/docker-ssd/docker/appdata/qdrant/snapshots + 226| + 227|# Stack directory + 228|sudo mkdir -p /mnt/docker-ssd/docker/compose/ai + 229|``` + 230| + 231|### 2.2 — docker-compose.yaml + 232| + 233|Create or update `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml`: + 234| + 235|```yaml + 236|name: ai + 237| + 238|services: + 239| qdrant: + 240| image: qdrant/qdrant:v1.14.0 + 241| container_name: qdrant + 242| restart: unless-stopped + 243| ports: + 244| - "6333:6333" # REST API + 245| - "6334:6334" # gRPC + 246| environment: + 247| TZ: ${TZ} + 248| QDRANT__SERVICE__GRPC_PORT: 6334 + 249| QDRANT__STORAGE__STORAGE_PATH: /qdrant/storage + 250| QDRANT__STORAGE__SNAPSHOTS_PATH: /qdrant/snapshots + 251| volumes: + 252| - /mnt/docker-ssd/docker/appdata/qdrant/storage:/qdrant/storage + 253| - /mnt/docker-ssd/docker/appdata/qdrant/snapshots:/qdrant/snapshots + 254| networks: + 255| - ai-services + 256| - default + 257| + 258| # ------------------------------------------------------- + 259| # Ollama and OpenWebUI go here when deployed. + 260| # They share this stack and the default + ai-services networks so + 261| # OpenWebUI can reach Qdrant at http://qdrant:6333 + 262| # ------------------------------------------------------- + 263| + 264| # ollama: + 265| # image: ollama/ollama:latest + 266| # container_name: ollama + 267| # ... + 268| + 269| # openwebui: + 270| # image: ghcr.io/open-webui/open-webui:main + 271| # container_name: openwebui + 272| # ... + 273| + 274|networks: + 275| ai-services: + 276| name: ai-services + 277|``` + 278| + 279|> **Image note:** `qdrant/qdrant:v1.14.0` is the latest stable as of May 2026. Check [GitHub releases](https://github.com/qdrant/qdrant/releases) before deploying. + 280| + 281|> **Cross-stack connectivity:** The `ai-services` network is defined here with an explicit `name:` so other stacks (like `automation/n8n`) can declare it as `external: true` and reach Qdrant by service name (`http://qdrant:6333`). This follows the same pattern as `ix-databases_shared-databases`. + 282| + 283|### 2.3 — .env.example + 284| + 285|```bash + 286|# /mnt/docker-ssd/docker/compose/ai/.env.example + 287| + 288|TZ=America/New_York + 289|``` + 290| + 291|### 2.4 — Validate & Deploy + 292| + 293|```bash + 294|cd /mnt/docker-ssd/docker/compose/ai + 295|cp .env.example .env + 296|nano .env # Set timezone + 297| + 298|docker compose --env-file .env config + 299|docker compose --env-file .env up -d + 300|``` + 301| + 302|### 2.5 — Post-Deploy Verification + 303| + 304|```bash + 305|# Container running? + 306|docker ps --filter name=qdrant + 307| + 308|# Clean startup? + 309|docker logs qdrant --tail 20 + 310| + 311|# REST API responding? + 312|curl -s http://localhost:6333/healthz + 313|# Expected: {"title":"qdrant - vectorass engine","version":"1.14.0","commit":"..."} + 314| + 315|# Create a test collection + 316|curl -X PUT http://localhost:6333/collections/test_collection \ + 317| -H "Content-Type: application/json" \ + 318| -d '{ + 319| "vectors": { + 320| "size": 384, + 321| "distance": "Cosine" + 322| } + 323| }' + 324| + 325|# Verify it exists + 326|curl -s http://localhost:6333/collections | python3 -m json.tool + 327| + 328|# Clean up test + 329|curl -X DELETE http://localhost:6333/collections/test_collection + 330|``` + 331| + 332|### 2.6 — OpenWebUI Integration (when deployed) + 333| + 334|When Ollama and OpenWebUI are brought online in this same stack, configure OpenWebUI's RAG settings: + 335| + 336|1. Go to OpenWebUI → Admin → Settings → Documents + 337|2. Set the vector database to Qdrant + 338|3. Endpoint: `http://qdrant:6333` (Docker DNS within the `ai` stack network) + 339|4. Collection name: `openwebui_docs` (or your preference) + 340| + 341|### 2.7 — Collections to Create (for n8n in Phase 3) + 342| + 343|These collections will be created programmatically by n8n workflows, but for reference: + 344| + 345|| Collection | Vector Size | Content | + 346||-----------|-------------|---------| + 347|| `homelab_docs` | 384 (nomic-embed-text) | Homelab markdown documentation | + 348|| `gitea_commits` | 384 | Gitea commit messages + diffs | + 349|| `media_metadata` | 384 | Plex/Tautulli metadata | + 350|| `obsidian_notes` | 384 | Personal notes from Obsidian vault | + 351| + 352|> **Vector size note:** 384 is the dimension for `nomic-embed-text` via Ollama. If you use a different embedding model, adjust accordingly. + 353| + 354|--- + 355| + 356|## Phase 3 — n8n (Workflow Automation) + 357| + 358|**Host:** PlausibleDeniability + 359|**Stack directory:** `/mnt/docker-ssd/docker/compose/automation/` (same stack as Gotify) + 360|**Why third:** n8n is the orchestration backbone — it ties Gotify, Qdrant, Ollama, and all triggers together. + 361| + 362|### 3.1 — Scaffold Directories + 363| + 364|```bash + 365|# Config on tank (workflow definitions, credentials store) + 366|# NOTE: n8n also writes execution logs here. If execution logging becomes + 367|# heavy (many workflows running frequently), consider moving to SSD. + 368|# For typical homelab usage (~20 workflows), tank is fine. + 369|sudo mkdir -p /mnt/tank/docker/appdata/n8n + 370| + 371|# Set ownership — n8n runs as UID 1000 (node user) + 372|sudo chown 1000:1000 /mnt/tank/docker/appdata/n8n + 373|``` + 374| + 375|### 3.2 — Database Init + 376| + 377|n8n uses the existing shared-postgres. Create its database and user: + 378| + 379|```bash + 380|docker exec -i shared-postgres psql -U postgres <<'SQL' + 381|CREATE USER n8n WITH PASSWORD 'REPLACE_WITH_GENERATED_PASSWORD'; + 382|CREATE DATABASE n8n OWNER n8n; + 383|GRANT ALL PRIVILEGES ON DATABASE n8n TO n8n; + 384|SQL + 385|``` + 386| + 387|Generate the password first: + 388|```bash + 389|openssl rand -hex 24 + 390|``` + 391| + 392|### 3.3 — docker-compose.yaml + 393| + 394|Update `/mnt/docker-ssd/docker/compose/automation/docker-compose.yaml` to add n8n alongside Gotify: + 395| + 396|```yaml + 397|name: automation + 398| + 399|services: + 400| gotify: + 401| image: gotify/server:2.6.1 + 402| container_name: gotify + 403| restart: unless-stopped + 404| ports: + 405| - "8484:80" + 406| environment: + 407| TZ: ${TZ} + 408| GOTIFY_DEFAULTUSER_PASS: ${GOTIFY_ADMIN_PASS} + 409| GOTIFY_SERVER_PORT: 80 + 410| GOTIFY_DATABASE_DIALECT: sqlite3 + 411| GOTIFY_DATABASE_CONNECTION: data/gotify.db + 412| volumes: + 413| - /mnt/docker-ssd/docker/appdata/gotify:/app/data + 414| networks: + 415| - pangolin + 416| - default + 417| + 418| n8n: + 419| image: docker.n8n.io/n8nio/n8n:1.88.0 + 420| container_name: n8n + 421| restart: unless-stopped + 422| ports: + 423| - "5678:5678" + 424| environment: + 425| TZ: ${TZ} + 426| # Database + 427| DB_TYPE: postgresdb + 428| DB_POSTGRESDB_HOST: shared-postgres + 429| DB_POSTGRESDB_PORT: 5432 + 430| DB_POSTGRESDB_DATABASE: ${N8N_DB_NAME} + 431| DB_POSTGRESDB_USER: ${N8N_DB_USER} + 432| DB_POSTGRESDB_PASSWORD: ${N8N_DB_PASS} + 433| # Security + 434| N8N_ENCRYPTION_KEY: ${N8N_ENCRYPTION_KEY} + 435| # Webhook / Reverse Proxy + 436| N8N_HOST: ${N8N_HOST} + 437| N8N_PROTOCOL: https + 438| N8N_PORT: 5678 + 439| WEBHOOK_URL: https://${N8N_HOST}/ + 440| N8N_PROXY_HOPS: 1 + 441| # General + 442| GENERIC_TIMEZONE: ${TZ} + 443| N8N_DIAGNOSTICS_ENABLED: false + 444| N8N_PERSONALIZATION_ENABLED: false + 445| volumes: + 446| - /mnt/tank/docker/appdata/n8n:/home/node/.n8n + 447| networks: + 448| - pangolin + 449| - ix-databases_shared-databases + 450| - ai-services + 451| - default + 452| depends_on: + 453| - gotify + 454| + 455|networks: + 456| pangolin: + 457| external: true + 458| ix-databases_shared-databases: + 459| external: true + 460| ai-services: + 461| external: true + 462|``` + 463| + 464|> **Image note:** `docker.n8n.io/n8nio/n8n:1.88.0` — n8n uses calendar-ish versioning. Check [n8n releases](https://github.com/n8n-io/n8n/releases) for the latest stable. The `docker.n8n.io` registry is preferred for production. + 465| + 466|> **Cross-stack access:** n8n joins `ai-services` (created by the `ai` stack) so it can reach Qdrant at `http://qdrant:6333` by Docker DNS. It also joins `ix-databases_shared-databases` for Postgres access — same pattern. + 467| + 468|### 3.4 — .env.example (updated for both services) + 469| + 470|```bash + 471|# /mnt/docker-ssd/docker/compose/automation/.env.example + 472| + 473|TZ=America/New_York + 474| + 475|# Gotify + 476|GOTIFY_ADMIN_PASS=CHANGE_ME + 477| + 478|# n8n — Database + 479|N8N_DB_NAME=n8n + 480|N8N_DB_USER=n8n + 481|N8N_DB_PASS=CHANGE_ME + 482| + 483|# n8n — Security + 484|N8N_ENCRYPTION_KEY=CHANGE_ME + 485| + 486|# n8n — Hostname + 487|N8N_HOST=n8n.paccoco.com + 488|``` + 489| + 490|### 3.5 — Pangolin Configuration + 491| + 492|| Field | Value | + 493||-------|-------| + 494|| Domain | `n8n.paccoco.com` | + 495|| Scheme | `http` | + 496|| Host | `n8n` | + 497|| Port | `5678` | + 498|| Headers | Forward `X-Forwarded-Proto: https` | + 499| + 500|### 3.6 — Validate & Deploy + 501| + 502|```bash + 503|cd /mnt/docker-ssd/docker/compose/automation + 504| + 505|# Update .env with real passwords + 506|nano .env + 507| + 508|docker compose --env-file .env config + 509|docker compose --env-file .env up -d + 510|``` + 511| + 512|### 3.7 — Post-Deploy Verification + 513| + 514|```bash + 515|# Both containers running? + 516|docker ps --filter name=gotify --filter name=n8n + 517| + 518|# n8n logs clean? + 519|docker logs n8n --tail 30 + 520| + 521|# n8n on correct networks? + 522|docker inspect n8n --format '{{json .NetworkSettings.Networks}}' | python3 -m json.tool + 523| + 524|# n8n UI accessible? + 525|curl -s -o /dev/null -w "%{http_code}" http://localhost:5678/ + 526|# Expected: 200 + 527| + 528|# Postgres connection working? (check logs for DB migration messages) + 529|docker logs n8n 2>&1 | grep -i "migrat" + 530|``` + 531| + 532|### 3.8 — First-Run Setup + 533| + 534|1. Navigate to `https://n8n.paccoco.com` + 535|2. Create your admin account + 536|3. Install community nodes you'll need: + 537| - `n8n-nodes-gotify` (if available) or use HTTP Request node + 538| - Ollama nodes (built-in as of n8n 1.x) + 539| + 540|### 3.9 — Workflow Blueprints + 541| + 542|Below are starter workflow descriptions for each planned automation. These are meant to be built in the n8n UI — the structure and node types are described so you can wire them up. + 543| + 544|#### Media Pipeline Workflows + 545| + 546|**Workflow: Sonarr/Radarr Download Notification** + 547|``` + 548|Trigger: Webhook node (POST from Sonarr/Radarr on download/import) + 549|Step 1: Extract series/movie name, quality, file path from webhook body + 550|Step 2: HTTP Request → TMDB API to fetch poster image URL + 551|Step 3: HTTP Request → Gotify REST API (POST /message) + 552| - Title: "New Download: {title}" + 553| - Message: "{quality} — {episodeTitle or year}" + 554| - Priority: 5 + 555| - Extras: attach poster URL as markdown image + 556|Also: HTTP Request → Discord webhook (formatted embed with poster) + 557|``` + 558| + 559|Configure Sonarr/Radarr webhooks: + 560|- Sonarr: Settings → Connect → Webhook → URL: `https://n8n.paccoco.com/webhook/sonarr` + 561|- Radarr: Settings → Connect → Webhook → URL: `https://n8n.paccoco.com/webhook/radarr` + 562| + 563|**Workflow: Tautulli Play Logging** + 564|``` + 565|Trigger: Webhook node (POST from Tautulli on playback start) + 566|Step 1: Extract user, media title, player, quality from payload + 567|Step 2: Postgres node → INSERT into watch_history table + 568|Step 3: (Optional) Gotify notification for specific users/media + 569|``` + 570| + 571|Tautulli config: Settings → Notification Agents → Webhook → URL: `https://n8n.paccoco.com/webhook/tautulli-play` + 572| + 573|**Workflow: Weekly Watch Digest** + 574|``` + 575|Trigger: Cron node (every Sunday at 10:00 AM) + 576|Step 1: Postgres node → SELECT watch history for past 7 days + 577|Step 2: Format data as structured text + 578|Step 3: HTTP Request → Ollama API (POST to PD's qwen2.5:14b) + 579| - Prompt: "Summarize this week's viewing in a fun digest: {data}" + 580|Step 4: HTTP Request → Gotify (send digest) + 581|``` + 582| + 583|#### Infrastructure Monitoring Workflows + 584| + 585|**Workflow: Uptime Kuma Enhanced Alerts** + 586|``` + 587|Trigger: Webhook node (from Uptime Kuma notification) + 588|Step 1: Extract monitor name, status, response time + 589|Step 2: HTTP Request → Netdata API for related metrics context + 590|Step 3: HTTP Request → Gotify + 591| - Title: "🔴 {monitor} DOWN" or "🟢 {monitor} UP" + 592| - Message: include Netdata context (CPU, mem, disk at time of alert) + 593| - Priority: 8 (high for down, 3 for recovery) + 594|``` + 595| + 596|**Workflow: ZFS Pool Utilization Alert** + 597|``` + 598|Trigger: Cron node (every 6 hours) + 599|Step 1: SSH node → Serenity: `zpool list -Hp malcolm` + 600|Step 2: Parse capacity percentage + 601|Step 3: IF capacity > 85% → Gotify alert (priority 8) + 602|Step 4: IF capacity > 90% → Gotify alert (priority 10) + Discord webhook + 603|``` + 604| + 605|**Workflow: Grafana Alert Remediation** + 606|``` + 607|Trigger: Webhook node (from Grafana alerting) + 608|Step 1: Parse alert labels (container, host, metric) + 609|Step 2: Switch node → route by alert type: + 610| - High CPU container → SSH → docker restart {container} + 611| - Disk full → SSH → pause qBittorrent, notify via Gotify + 612| - Memory pressure → Gotify alert only (manual intervention) + 613|Step 3: Log action taken to Postgres + 614|``` + 615| + 616|#### Homelab Ops Workflows + 617| + 618|**Workflow: Gitea Commit Summary** + 619|``` + 620|Trigger: Webhook node (Gitea webhook on push to truenas-stacks) + 621|Step 1: Extract commit messages, author, files changed + 622|Step 2: HTTP Request → Ollama API + 623| - Prompt: "Summarize this commit in one sentence: {commit_message}" + 624|Step 3: HTTP Request → Gotify + 625| - Title: "Commit to truenas-stacks" + 626| - Message: Ollama-generated summary + 627|``` + 628| + 629|Gitea config: Repository → Settings → Webhooks → URL: `https://n8n.paccoco.com/webhook/gitea-push` + 630| + 631|**Workflow: qBittorrent Auto-Rescan** + 632|``` + 633|Trigger: Webhook or polling (qBittorrent API for completed+moved torrents) + 634|Step 1: Determine if file is in Sonarr or Radarr path + 635|Step 2: HTTP Request → Sonarr API (POST /command → RescanSeries) + 636| OR HTTP Request → Radarr API (POST /command → RescanMovie) + 637|Step 3: Gotify notification confirming rescan triggered + 638|``` + 639| + 640|#### AI Pipeline Workflows + 641| + 642|**Workflow: URL Digest Pipeline** + 643|``` + 644|Trigger: Webhook node (POST with URL in body) + 645|Step 1: HTTP Request → fetch page content + 646|Step 2: Code node → extract text, chunk into ~500 token segments + 647|Step 3: HTTP Request → Ollama API → summarize each chunk + 648|Step 4: Code node → combine summaries into digest + 649|Step 5: Postgres node → store digest with metadata + 650|Step 6: Return digest in webhook response + 651|``` + 652| + 653|**Workflow: Multi-Model Query Router (simplified by LiteLLM — see Phase 7)** + 654|``` + 655|Trigger: Webhook node (POST with query + complexity hint) + 656|Step 1: HTTP Request → LiteLLM (POST http://litellm:4000/v1/chat/completions) + 657| - model: complexity parameter ("light", "medium", or "heavy") + 658| - LiteLLM handles routing to the correct Ollama instance + 659|Step 2: Return response via webhook + 660|``` + 661|> With LiteLLM deployed, the Switch node and three separate Ollama endpoints + 662|> are replaced by a single HTTP Request node. The routing logic lives in + 663|> LiteLLM's config.yaml instead of n8n workflow logic. + 664| + 665|**Workflow: Qdrant Index Updater** + 666|``` + 667|Trigger: Webhook node (from Gitea push to any watched repo) + 668|Step 1: HTTP Request → Gitea API → fetch changed files content + 669|Step 2: Code node → chunk text into embedding-sized segments + 670|Step 3: HTTP Request → Ollama API (embed endpoint with nomic-embed-text) + 671|Step 4: HTTP Request → Qdrant API (PUT /collections/homelab_docs/points) + 672|Step 5: Gotify notification: "{n} documents re-indexed" + 673|``` + 674| + 675|#### Home / Business Workflows + 676| + 677|**Workflow: KitchenOwl Grocery Notification** + 678|``` + 679|Trigger: Polling (KitchenOwl API) or webhook if supported + 680|Step 1: Fetch current shopping list items + 681|Step 2: Format as clean text list + 682|Step 3: HTTP Request → Gotify → phone push notification + 683|``` + 684| + 685|**Workflow: Donetick Task Reminder** + 686|``` + 687|Trigger: Cron node (daily at 9:00 AM) + 688|Step 1: HTTP Request → Donetick API → fetch tasks due today/overdue + 689|Step 2: Format task list + 690|Step 3: HTTP Request → Gotify (priority 5) + 691|``` + 692| + 693|**Workflow: Long and Low Crafts Order Pipeline** + 694|``` + 695|Trigger: Webhook (Etsy webhook or email trigger via IMAP node) + 696|Step 1: Parse order details (item, quantity, customer, shipping) + 697|Step 2: HTTP Request → Donetick API → create fulfillment task + 698|Step 3: HTTP Request → Gotify DM notification + 699| - Title: "New L&L Order" + 700| - Message: "{item} x{qty} — ship by {date}" + 701| - Priority: 7 + 702|``` + 703| + 704|--- + 705| + 706|## Phase 4 — Paperless-NGX (Document Intelligence) + 707| + 708|**Host:** PlausibleDeniability + 709|**Stack directory:** `/mnt/docker-ssd/docker/compose/documents/` (new stack) + 710|**Why fourth:** Depends on n8n for automation hooks and Gotify for notifications. + 711| + 712|### 4.1 — Scaffold Directories + 713| + 714|```bash + 715|# Index/data on SSD (search index is write-heavy) + 716|sudo mkdir -p /mnt/docker-ssd/docker/appdata/paperless/data + 717| + 718|# Documents (media) on tank (bulk storage, read-heavy) + 719|sudo mkdir -p /mnt/tank/docker/appdata/paperless/media + 720| + 721|# Consume folder on tank (drop zone for new documents) + 722|sudo mkdir -p /mnt/tank/docker/appdata/paperless/consume + 723| + 724|# Export folder on tank + 725|sudo mkdir -p /mnt/tank/docker/appdata/paperless/export + 726| + 727|# Stack directory + 728|sudo mkdir -p /mnt/docker-ssd/docker/compose/documents + 729|``` + 730| + 731|### 4.2 — Database Init + 732| + 733|Paperless uses the existing shared-postgres and shared-redis: + 734| + 735|```bash + 736|docker exec -i shared-postgres psql -U postgres <<'SQL' + 737|CREATE USER paperless WITH PASSWORD 'REPLACE_WITH_GENERATED_PASSWORD'; + 738|CREATE DATABASE paperless OWNER paperless; + 739|GRANT ALL PRIVILEGES ON DATABASE paperless TO paperless; + 740|SQL + 741|``` + 742| + 743|### 4.3 — docker-compose.yaml + 744| + 745|Create `/mnt/docker-ssd/docker/compose/documents/docker-compose.yaml`: + 746| + 747|```yaml + 748|name: documents + 749| + 750|services: + 751| paperless: + 752| image: ghcr.io/paperless-ngx/paperless-ngx:2.16 + 753| container_name: paperless + 754| restart: unless-stopped + 755| ports: + 756| - "8000:8000" + 757| environment: + 758| TZ: ${TZ} + 759| # Database + 760| PAPERLESS_DBENGINE: postgresql + 761| PAPERLESS_DBHOST: shared-postgres + 762| PAPERLESS_DBPORT: 5432 + 763| PAPERLESS_DBNAME: ${PAPERLESS_DB_NAME} + 764| PAPERLESS_DBUSER: ${PAPERLESS_DB_USER} + 765| PAPERLESS_DBPASS: ${PAPERLESS_DB_PASS} + 766| # Redis (using shared-redis) + 767| PAPERLESS_REDIS: redis://shared-redis:***@10.5.30.7 + 1063| + 1064|# Stack directory + 1065|sudo mkdir -p /opt/monitoring + 1066| + 1067|# Prometheus data on hdd-2 (has more headroom) + 1068|sudo mkdir -p /mnt/hdd-2/prometheus-data + 1069|sudo chown 65534:65534 /mnt/hdd-2/prometheus-data # nobody user (Prometheus default) + 1070| + 1071|# Grafana data on hdd-2 + 1072|sudo mkdir -p /mnt/hdd-2/grafana-data + 1073|sudo chown 472:472 /mnt/hdd-2/grafana-data # grafana user + 1074| + 1075|# Config directories + 1076|sudo mkdir -p /opt/monitoring/provisioning/datasources + 1077|sudo mkdir -p /opt/monitoring/provisioning/dashboards + 1078|``` + 1079| + 1080|### 6.2 — prometheus.yml + 1081| + 1082|Create `/opt/monitoring/prometheus.yml`: + 1083| + 1084|```yaml + 1085|global: + 1086| scrape_interval: 15s + 1087| evaluation_interval: 15s + 1088| + 1089|scrape_configs: + 1090| # ------- Local (N.O.M.A.D.) ------- + 1091| - job_name: 'nomad-node' + 1092| static_configs: + 1093| - targets: ['node-exporter:9100'] + 1094| labels: + 1095| host: 'nomad' + 1096| + 1097| # ------- PlausibleDeniability ------- + 1098| - job_name: 'pd-netdata' + 1099| metrics_path: /api/v1/allmetrics + 1100| params: + 1101| format: [prometheus] + 1102| static_configs: + 1103| - targets: ['10.5.1.X:19999'] # Replace with PD's IP + 1104| labels: + 1105| host: 'plausible-deniability' + 1106| + 1107| # ------- Serenity ------- + 1108| - job_name: 'serenity-netdata' + 1109| metrics_path: /api/v1/allmetrics + 1110| params: + 1111| format: [prometheus] + 1112| static_configs: + 1113| - targets: ['10.5.30.5:19999'] + 1114| labels: + 1115| host: 'serenity' + 1116| + 1117| # ------- Gotify ------- + 1118| # Gotify exposes /health but no Prometheus endpoint natively. + 1119| # Use blackbox exporter or just rely on Uptime Kuma. + 1120| + 1121| # ------- n8n ------- + 1122| # n8n doesn't expose Prometheus metrics by default. + 1123| # Monitor via container resource metrics from Netdata. + 1124| + 1125| # -------- Add more targets as needed -------- + 1126| # When node-exporter is installed on PD and Serenity: + 1127| # - job_name: 'pd-node' + 1128| # static_configs: + 1129| # - targets: ['10.5.1.X:9100'] + 1130| # labels: + 1131| # host: 'plausible-deniability' + 1132| # + 1133| # - job_name: 'serenity-node' + 1134| # static_configs: + 1135| # - targets: ['10.5.30.5:9100'] + 1136| # labels: + 1137| # host: 'serenity' + 1138|``` + 1139| + 1140|> **Netdata as a Prometheus target:** Both PD and Serenity already run Netdata. Netdata has a built-in Prometheus exporter at `/api/v1/allmetrics?format=prometheus`. This gives you CPU, memory, disk, network, and ZFS metrics without installing node-exporter on those hosts. + 1141| + 1142|### 6.3 — Grafana Provisioning + 1143| + 1144|Create `/opt/monitoring/provisioning/datasources/prometheus.yml`: + 1145| + 1146|```yaml + 1147|apiVersion: 1 + 1148| + 1149|datasources: + 1150| - name: Prometheus + 1151| type: prometheus + 1152| access: proxy + 1153| url: http://prometheus:9090 + 1154| isDefault: true + 1155| editable: true + 1156|``` + 1157| + 1158|Create `/opt/monitoring/provisioning/dashboards/dashboards.yml`: + 1159| + 1160|```yaml + 1161|apiVersion: 1 + 1162| + 1163|providers: + 1164| - name: 'default' + 1165| orgId: 1 + 1166| folder: '' + 1167| type: file + 1168| disableDeletion: false + 1169| editable: true + 1170| options: + 1171| path: /var/lib/grafana/dashboards + 1172| foldersFromFilesStructure: false + 1173|``` + 1174| + 1175|### 6.4 — docker-compose.yaml + 1176| + 1177|Create `/opt/monitoring/docker-compose.yaml`: + 1178| + 1179|```yaml + 1180|name: monitoring + 1181| + 1182|services: + 1183| prometheus: + 1184| image: prom/prometheus:v3.4.0 + 1185| container_name: prometheus + 1186| restart: unless-stopped + 1187| ports: + 1188| - "9090:9090" + 1189| command: + 1190| - '--config.file=/etc/prometheus/prometheus.yml' + 1191| - '--storage.tsdb.path=/prometheus' + 1192| - '--storage.tsdb.retention.time=90d' + 1193| - '--web.enable-lifecycle' + 1194| volumes: + 1195| - /opt/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro + 1196| - /mnt/hdd-2/prometheus-data:/prometheus + 1197| networks: + 1198| - monitoring + 1199| + 1200| grafana: + 1201| image: grafana/grafana:13.0.1 + 1202| container_name: grafana + 1203| restart: unless-stopped + 1204| ports: + 1205| - "3000:3000" + 1206| environment: + 1207| TZ: ${TZ} + 1208| GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER} + 1209| GF_SECURITY_ADMIN_PASSWORD: ${GF_ADMIN_PASS} + 1210| GF_SERVER_ROOT_URL: https://${GF_HOST}/ + 1211| volumes: + 1212| - /mnt/hdd-2/grafana-data:/var/lib/grafana + 1213| - /opt/monitoring/provisioning:/etc/grafana/provisioning:ro + 1214| networks: + 1215| - monitoring + 1216| + 1217| node-exporter: + 1218| image: prom/node-exporter:v1.9.0 + 1219| container_name: node-exporter + 1220| restart: unless-stopped + 1221| ports: + 1222| - "9100:9100" + 1223| command: + 1224| - '--path.procfs=/host/proc' + 1225| - '--path.sysfs=/host/sys' + 1226| - '--path.rootfs=/rootfs' + 1227| - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)' + 1228| volumes: + 1229| - /proc:/host/proc:ro + 1230| - /sys:/host/sys:ro + 1231| - /:/rootfs:ro + 1232| networks: + 1233| - monitoring + 1234| + 1235|networks: + 1236| monitoring: + 1237| driver: bridge + 1238|``` + 1239| + 1240|> **Image notes:** + 1241|> - `prom/prometheus:v3.4.0` — Prometheus v3 is the current major line. **Do not use `:latest`** — it still resolves to 2.x due to a known tagging issue. Always use an explicit `v3.x.y` tag. + 1242|> - `grafana/grafana:latest` — use the moving stable image by default here; only pin Grafana if a breakage or compatibility reason is documented. + 1243|> - `prom/node-exporter:v1.9.0` — check [releases](https://github.com/prometheus/node_exporter/releases). + 1244| + 1245|### 6.5 — .env.example + 1246| + 1247|```bash + 1248|# /opt/monitoring/.env.example + 1249| + 1250|TZ=America/New_York + 1251| + 1252|# Grafana + 1253|GF_ADMIN_USER=admin + 1254|GF_ADMIN_PASS=CHANGE_ME + 1255|GF_HOST=grafana.paccoco.com + 1256|``` + 1257| + 1258|### 6.6 — Pangolin Configuration + 1259| + 1260|Grafana runs on N.O.M.A.D., not PD where the main Newt agent lives. You have two options: + 1261| + 1262|**Option A — Route via N.O.M.A.D.'s existing Newt (recommended if already connected)** + 1263| + 1264|N.O.M.A.D. already has a Newt container from the Project N.O.M.A.D. setup. If it's connected to your Pangolin VPS, just add a new resource in the Pangolin dashboard pointing to `http://grafana:3000` or `http://10.5.30.7:3000`. + 1265| + 1266|**Option B — Add a dedicated Newt to the monitoring stack** + 1267| + 1268|If N.O.M.A.D.'s existing Newt is not connected to Pangolin (or is a separate Pangolin instance), add Newt to the monitoring compose: + 1269| + 1270|```yaml + 1271| newt: + 1272| image: ghcr.io/fosrl/newt:latest + 1273| container_name: monitoring-newt + 1274| restart: unless-stopped + 1275| environment: + 1276| PANGOLIN_ENDPOINT: ${PANGOLIN_ENDPOINT} + 1277| NEWT_ID: ${NEWT_ID} + 1278| NEWT_SECRET: ${NEWT_SECRET} + 1279| networks: + 1280| - monitoring + 1281|``` + 1282| + 1283|Then in Pangolin dashboard: + 1284| + 1285|| Field | Value | + 1286||-------|-------| + 1287|| Domain | `grafana.paccoco.com` | + 1288|| Scheme | `http` | + 1289|| Host | `grafana` (Docker DNS via shared network) | + 1290|| Port | `3000` | + 1291| + 1292|**Option C — Direct IP routing (simplest, no Newt needed)** + 1293| + 1294|If Pangolin's Newt on PD can reach N.O.M.A.D. by LAN IP (they're on the same subnet): + 1295| + 1296|| Field | Value | + 1297||-------|-------| + 1298|| Domain | `grafana.paccoco.com` | + 1299|| Scheme | `http` | + 1300|| Host | `10.5.30.7` | + 1301|| Port | `3000` | + 1302| + 1303|> **Decision point:** Check if N.O.M.A.D.'s existing Newt is connected to your Pangolin instance before deploying. Run `docker ps --filter name=newt` on N.O.M.A.D. to verify. If it's running and connected, Option A is zero-effort. If not, Option C is the simplest fallback. + 1304| + 1305|### 6.7 — Validate & Deploy + 1306| + 1307|```bash + 1308|ssh nomad@10.5.30.7 + 1309| + 1310|cd /opt/monitoring + 1311|cp .env.example .env + 1312|nano .env # Fill in real values + 1313| + 1314|docker compose --env-file .env config + 1315|docker compose --env-file .env up -d + 1316|``` + 1317| + 1318|### 6.8 — Post-Deploy Verification + 1319| + 1320|```bash + 1321|# All three containers running? + 1322|docker ps --filter name=prometheus --filter name=grafana --filter name=node-exporter + 1323| + 1324|# Prometheus scraping targets? + 1325|curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | head -40 + 1326| + 1327|# Grafana UI accessible? + 1328|curl -s -o /dev/null -w "%{http_code}" http://localhost:3000/ + 1329|# Expected: 200 or 302 + 1330| + 1331|# Node exporter metrics flowing? + 1332|curl -s http://localhost:9100/metrics | head -10 + 1333|``` + 1334| + 1335|### 6.9 — Recommended Dashboards + 1336| + 1337|Import these from [Grafana Dashboard Library](https://grafana.com/grafana/dashboards/): + 1338| + 1339|| Dashboard | ID | Purpose | + 1340||-----------|----|---------| + 1341|| Node Exporter Full | 1860 | System metrics for N.O.M.A.D. | + 1342|| Docker Container Stats | 893 | Container resource usage | + 1343|| Netdata via Prometheus | (search) | PD and Serenity system metrics | + 1344| + 1345|To import: Grafana → Dashboards → New → Import → Enter dashboard ID. + 1346| + 1347|### 6.10 — Metrics Targets Summary + 1348| + 1349|| Target | Host | Method | Endpoint | + 1350||--------|------|--------|----------| + 1351|| N.O.M.A.D. system | localhost | node-exporter | node-exporter:9100 | + 1352|| PD system | 10.5.1.X | Netdata Prometheus | 10.5.1.X:19999/api/v1/allmetrics | + 1353|| Serenity system | 10.5.30.5 | Netdata Prometheus | 10.5.30.5:19999/api/v1/allmetrics | + 1354|| ZFS pools | via Netdata | Netdata exports ZFS metrics | Included in Netdata scrape | + 1355|| Container stats | via Netdata | Netdata exports cgroup metrics | Included in Netdata scrape | + 1356|| Plex streams | Tautulli | n8n polling → Postgres | Via n8n workflow (Phase 3) | + 1357|| qBit stats | qBittorrent API | n8n polling → Postgres | Via n8n workflow (Phase 3) | + 1358|| Tailscale latency | Tailscale API | n8n polling → Postgres | Via n8n workflow (Phase 3) | + 1359| + 1360|### 6.11 — n8n Integration (Grafana → n8n alert webhook) + 1361| + 1362|In Grafana → Alerting → Contact Points, create a webhook contact point: + 1363| + 1364|| Field | Value | + 1365||-------|-------| + 1366|| Name | `n8n-alerts` | + 1367|| Type | Webhook | + 1368|| URL | `https://n8n.paccoco.com/webhook/grafana-alert` | + 1369|| HTTP Method | POST | + 1370| + 1371|Then create alert rules for: + 1372|- ZFS pool utilization > 85% + 1373|- Container memory > 90% of limit + 1374|- Host CPU sustained > 90% for 5 minutes + 1375|- Disk I/O latency spikes + 1376| + 1377|These fire into the "Grafana Alert Remediation" n8n workflow (see Phase 3). + 1378| + 1379|--- + 1380| + 1381|## Phase 7 — LiteLLM (AI Gateway) + 1382| + 1383|**Host:** PlausibleDeniability + 1384|**Stack directory:** `/mnt/docker-ssd/docker/compose/ai/` (same stack as Qdrant) + 1385|**Why:** Replaces manual multi-model routing with a unified OpenAI-compatible API. Every app (OpenWebUI, Continue.dev, n8n, Paperless summarization) points at one endpoint instead of juggling three Ollama IPs. + 1386| + 1387|### 7.1 — What LiteLLM Does + 1388| + 1389|LiteLLM sits in front of your three Ollama instances and presents a single OpenAI-compatible API at `http://litellm:4000`. It handles: + 1390| + 1391|- **Model routing** — requests for `qwen3:32b` go to ROCINANTE, `qwen2.5:14b` goes to PD, `phi4` goes to N.O.M.A.D. + 1392|- **Failover** — if ROCINANTE is offline, LiteLLM can fall back to PD automatically + 1393|- **Load balancing** — distribute requests across instances running the same model + 1394|- **Usage tracking** — logs token counts, latency, and costs per model/user via its built-in database + 1395| + 1396|### 7.2 — Scaffold Directories + 1397| + 1398|```bash + 1399|# Config on SSD (SQLite DB for usage tracking) + 1400|sudo mkdir -p /mnt/docker-ssd/docker/appdata/litellm + 1401| + 1402|# LiteLLM config file + 1403|sudo mkdir -p /mnt/docker-ssd/docker/compose/ai/litellm + 1404|``` + 1405| + 1406|### 7.3 — LiteLLM Config + 1407| + 1408|Create `/mnt/docker-ssd/docker/compose/ai/litellm/config.yaml`: + 1409| + 1410|```yaml + 1411|model_list: + 1412| # ---- ROCINANTE (RTX 4090, 24GB) — heavy reasoning ---- + 1413| - model_name: "heavy" + 1414| litellm_params: + 1415| model: "ollama/qwen3:32b" + 1416| api_base: "http://10.5.1.ROCINANTE:11434" + 1417| timeout: 300 + 1418| stream_timeout: 300 + 1419| model_info: + 1420| description: "Heavy reasoning, long context, complex code" + 1421| + 1422| - model_name: "heavy" + 1423| litellm_params: + 1424| model: "ollama/deepseek-r1:32b" + 1425| api_base: "http://10.5.1.ROCINANTE:11434" + 1426| timeout: 300 + 1427| stream_timeout: 300 + 1428| model_info: + 1429| description: "Deep reasoning fallback on ROCINANTE" + 1430| + 1431| # ---- PlausibleDeniability (RTX 2080 Ti, 11GB) — general ---- + 1432| - model_name: "medium" + 1433| litellm_params: + 1434| model: "ollama/qwen2.5:14b" + 1435| api_base: "http://host.docker.internal:11434" + 1436| timeout: 120 + 1437| stream_timeout: 120 + 1438| model_info: + 1439| description: "General homelab assistant, RAG queries" + 1440| + 1441| # ---- N.O.M.A.D. (GTX 1080, 8GB) — lightweight ---- + 1442| - model_name: "light" + 1443| litellm_params: + 1444| model: "ollama/phi4" + 1445| api_base: "http://10.5.30.7:11434" + 1446| timeout: 60 + 1447| stream_timeout: 60 + 1448| model_info: + 1449| description: "Fast local inference, lightweight tasks" + 1450| + 1451| - model_name: "light" + 1452| litellm_params: + 1453| model: "ollama/llama3.2:3b" + 1454| api_base: "http://10.5.30.7:11434" + 1455| timeout: 60 + 1456| stream_timeout: 60 + 1457| model_info: + 1458| description: "Ultra-light fallback on N.O.M.A.D." + 1459| + 1460| # ---- Embeddings ---- + 1461| - model_name: "embed" + 1462| litellm_params: + 1463| model: "ollama/nomic-embed-text" + 1464| api_base: "http://host.docker.internal:11434" + 1465| model_info: + 1466| description: "Text embeddings for RAG pipeline" + 1467| + 1468| # ---- Direct model access (bypass routing) ---- + 1469| # These let you request a specific model by its full name + 1470| - model_name: "ollama/qwen3:32b" + 1471| litellm_params: + 1472| model: "ollama/qwen3:32b" + 1473| api_base: "http://10.5.1.ROCINANTE:11434" + 1474| + 1475| - model_name: "ollama/qwen2.5:14b" + 1476| litellm_params: + 1477| model: "ollama/qwen2.5:14b" + 1478| api_base: "http://host.docker.internal:11434" + 1479| + 1480| - model_name: "ollama/phi4" + 1481| litellm_params: + 1482| model: "ollama/phi4" + 1483| api_base: "http://10.5.30.7:11434" + 1484| + 1485|litellm_settings: + 1486| drop_params: true + 1487| set_verbose: false + 1488| request_timeout: 300 + 1489| num_retries: 2 + 1490| retry_after: 5 + 1491| allowed_fails: 3 + 1492| cooldown_time: 60 + 1493| + 1494|general_settings: + 1495| master_key: "os.environ/LITELLM_MASTER_KEY" + 1496|``` + 1497| + 1498|> **DEPLOYED NOTE (2026-05-05):** Do NOT add `database_url` to general_settings — newer LiteLLM versions include Prisma ORM that requires PostgreSQL, and adding any database_url causes a crash loop. Without it, LiteLLM runs in config-only mode which is fine for homelab use. The `master_key` uses `os.environ/LITELLM_MASTER_KEY` syntax to read from the container's environment variable (set via .env → docker compose). + 1499| + 1500|> **Routing explained:** Multiple entries with the same `model_name` (like "heavy") enable load balancing and failover within that tier. When you request model `heavy`, LiteLLM picks the healthiest deployment. The direct-access entries (like `ollama/qwen3:32b`) let you bypass the tier system and target a specific model when needed. + 1501| + 1502|> **Replace IPs:** Update `10.5.1.ROCINANTE` with ROCINANTE's actual LAN or Tailscale IP. PD uses `host.docker.internal` since LiteLLM runs on PD alongside Ollama. + 1503| + 1504|### 7.4 — docker-compose.yaml (updated ai stack) + 1505| + 1506|Update `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml` to add LiteLLM: + 1507| + 1508|```yaml + 1509|name: ai + 1510| + 1511|services: + 1512| qdrant: + 1513| image: qdrant/qdrant:v1.14.0 + 1514| container_name: qdrant + 1515| restart: unless-stopped + 1516| ports: + 1517| - "6333:6333" # REST API + 1518| - "6334:6334" # gRPC + 1519| environment: + 1520| TZ: ${TZ} + 1521| QDRANT__SERVICE__GRPC_PORT: 6334 + 1522| QDRANT__STORAGE__STORAGE_PATH: /qdrant/storage + 1523| QDRANT__STORAGE__SNAPSHOTS_PATH: /qdrant/snapshots + 1524| volumes: + 1525| - /mnt/docker-ssd/docker/appdata/qdrant/storage:/qdrant/storage + 1526| - /mnt/docker-ssd/docker/appdata/qdrant/snapshots:/qdrant/snapshots + 1527| networks: + 1528| - ai-services + 1529| - default + 1530| + 1531| litellm: + 1532| image: ghcr.io/berriai/litellm:main-latest + 1533| container_name: litellm + 1534| restart: unless-stopped + 1535| ports: + 1536| - "4000:4000" + 1537| environment: + 1538| TZ: ${TZ} + 1539| LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY} + 1540| volumes: + 1541| - /mnt/docker-ssd/docker/compose/ai/litellm/config.yaml:/app/config.yaml:ro + 1542| - /mnt/docker-ssd/docker/appdata/litellm:/app/data + 1543| command: ["--config", "/app/config.yaml", "--port", "4000"] + 1544| extra_hosts: + 1545| - "host.docker.internal:host-gateway" + 1546| networks: + 1547| - ai-services + 1548| - default + 1549| + 1550| reranker: + 1551| image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest + 1552| container_name: reranker + 1553| restart: unless-stopped + 1554| ports: + 1555| - "8787:80" + 1556| environment: + 1557| MODEL_ID: ${RERANKER_MODEL} + 1558| volumes: + 1559| - /mnt/docker-ssd/docker/appdata/reranker:/data + 1560| networks: + 1561| - ai-services + 1562| - default + 1563| + 1564| whisper: + 1565| image: fedirz/faster-whisper-server:latest-cuda + 1566| container_name: whisper + 1567| restart: unless-stopped + 1568| ports: + 1569| - "8786:8000" + 1570| environment: + 1571| TZ: ${TZ} + 1572| WHISPER__MODEL: ${WHISPER_MODEL} + 1573| WHISPER__DEVICE: cuda + 1574| WHISPER__COMPUTE_TYPE: float16 + 1575| volumes: + 1576| - /mnt/docker-ssd/docker/appdata/whisper:/root/.cache/huggingface + 1577| deploy: + 1578| resources: + 1579| reservations: + 1580| devices: + 1581| - driver: nvidia + 1582| count: 1 + 1583| capabilities: [gpu] + 1584| networks: + 1585| - ai-services + 1586| - default + 1587| + 1588| # ------------------------------------------------------- + 1589| # Ollama and OpenWebUI go here when deployed. + 1590| # They share this stack and the default + ai-services networks. + 1591| # OpenWebUI → Qdrant at http://qdrant:6333 + 1592| # OpenWebUI → LiteLLM at http://litellm:4000 (or direct Ollama) + 1593| # ------------------------------------------------------- + 1594| + 1595| # ollama: + 1596| # image: ollama/ollama:latest + 1597| # container_name: ollama + 1598| # ... + 1599| + 1600| # openwebui: + 1601| # image: ghcr.io/open-webui/open-webui:main + 1602| # container_name: openwebui + 1603| # ... + 1604| + 1605|networks: + 1606| ai-services: + 1607| name: ai-services + 1608|``` + 1609| + 1610|> **GPU sharing note:** On PD, the RTX 2080 Ti is shared between Plex (hardware transcoding), Immich ML, and Ollama. faster-whisper will also need the GPU when processing audio. These workloads are bursty (not constant), so time-sharing the GPU is viable — but be aware that a large Whisper transcription job will temporarily impact Ollama inference latency. If this becomes an issue, consider running Whisper on N.O.M.A.D.'s GTX 1080 instead and change `whisper`'s Ollama-style routing accordingly. + 1611| + 1612|> **Reranker on CPU:** The reranker uses the `cpu-latest` image variant intentionally — do NOT pin to `cpu-1.5` as it has an hf-hub compatibility bug. Reranking is a lightweight operation (scoring ~50 chunks takes <1s on CPU) and doesn't justify competing for GPU VRAM. The model loads via safetensors fallback since no ONNX files exist for bge-reranker-v2-m3. Max batch size is 4 on CPU. If you find latency is an issue, a GPU variant exists (`ghcr.io/huggingface/text-embeddings-inference:turing-latest` for your 2080 Ti). + 1613| + 1614|### 7.5 — .env.example (updated ai stack) + 1615| + 1616|```bash + 1617|# /mnt/docker-ssd/docker/compose/ai/.env.example + 1618| + 1619|TZ=America/New_York + 1620| + 1621|# LiteLLM + 1622|LITELLM_MASTER_KEY=sk-CHANGE_ME + 1623| + 1624|# Reranker + 1625|RERANKER_MODEL=BAAI/bge-reranker-v2-m3 + 1626| + 1627|# Whisper + 1628|WHISPER_MODEL=Systran/faster-distil-whisper-large-v3 + 1629|``` + 1630| + 1631|### 7.6 — Validate & Deploy + 1632| + 1633|```bash + 1634|cd /mnt/docker-ssd/docker/compose/ai + 1635|cp .env.example .env + 1636|nano .env # Set LITELLM_MASTER_KEY=$(openssl rand -hex 32) + 1637| + 1638|docker compose --env-file .env config + 1639|docker compose --env-file .env up -d + 1640|``` + 1641| + 1642|### 7.7 — Post-Deploy Verification + 1643| + 1644|```bash + 1645|# All containers running? + 1646|docker ps --filter name=qdrant --filter name=litellm --filter name=reranker --filter name=whisper + 1647| + 1648|# LiteLLM health? + 1649|curl -s http://localhost:4000/health + 1650| + 1651|# LiteLLM can see all models? + 1652|curl -s http://localhost:4000/v1/models \ + 1653| -H "Authorization: Bearer *** | python3 -m json.tool + 1654| + 1655|# Test a chat completion through LiteLLM + 1656|curl -s http://localhost:4000/v1/chat/completions \ + 1657| -H "Authorization: Bearer *** \ + 1658| -H "Content-Type: application/json" \ + 1659| -d '{ + 1660| "model": "medium", + 1661| "messages": [{"role": "user", "content": "Hello, which model are you?"}] + 1662| }' | python3 -m json.tool + 1663| + 1664|# Test embeddings through LiteLLM + 1665|curl -s http://localhost:4000/v1/embeddings \ + 1666| -H "Authorization: Bearer *** \ + 1667| -H "Content-Type: application/json" \ + 1668| -d '{ + 1669| "model": "embed", + 1670| "input": "test embedding" + 1671| }' | python3 -m json.tool + 1672|``` + 1673| + 1674|### 7.8 — Integration Updates + 1675| + 1676|With LiteLLM deployed, all apps should point at `http://litellm:4000` (within the `ai-services` network) or `http://PD_IP:4000` (from external hosts) instead of direct Ollama endpoints. + 1677| + 1678|**OpenWebUI:** Settings → Connections → add OpenAI-compatible endpoint: + 1679|- URL: `http://litellm:4000/v1` + 1680|- API Key: your `LITELLM_MASTER_KEY` + 1681|- This gives OpenWebUI access to all models across all three machines through one connection + 1682| + 1683|**Continue.dev:** Update `~/.continue/config.json`: + 1684|```json + 1685|{ + 1686| "models": [ + 1687| { + 1688| "title": "Heavy (ROCINANTE)", + 1689| "provider": "openai", + 1690| "model": "heavy", + 1691| "apiBase": "http://PD_TAILSCALE_IP:4000/v1", + 1692| "apiKey": "YOUR_MASTER_KEY" + 1693| }, + 1694| { + 1695| "title": "Medium (PD)", + 1696| "provider": "openai", + 1697| "model": "medium", + 1698| "apiBase": "http://PD_TAILSCALE_IP:4000/v1", + 1699| "apiKey": "YOUR_MASTER_KEY" + 1700| } + 1701| ], + 1702| "tabAutocompleteModel": { + 1703| "title": "Light (N.O.M.A.D.)", + 1704| "provider": "openai", + 1705| "model": "light", + 1706| "apiBase": "http://PD_TAILSCALE_IP:4000/v1", + 1707| "apiKey": "YOUR_MASTER_KEY" + 1708| } + 1709|} + 1710|``` + 1711| + 1712|> **Key change:** Continue.dev now uses `provider: "openai"` instead of `provider: "ollama"` and points at LiteLLM. One IP to remember, and if you add or move models between machines, you only update `config.yaml` — not every app. + 1713| + 1714|**n8n workflows:** All HTTP Request nodes that call Ollama directly should be updated: + 1715|- Old: `http://10.5.30.7:11434/api/generate` (N.O.M.A.D.) + 1716|- New: `http://litellm:4000/v1/chat/completions` with `model: "light"` + 1717|- n8n is on the `ai-services` network, so it reaches LiteLLM by Docker DNS + 1718| + 1719|**n8n Multi-Model Query Router workflow:** This workflow is now **simplified dramatically** — instead of a Switch node routing to three different Ollama IPs, it becomes a single HTTP Request node that passes the model tier name (`light`, `medium`, `heavy`) to LiteLLM and lets the gateway handle routing. The Switch node logic can be removed entirely. + 1720| + 1721|--- + 1722| + 1723|## Phase 8 — Reranker (RAG Quality) + 1724| + 1725|**Host:** PlausibleDeniability (deployed as part of the `ai` stack in Phase 7) + 1726|**Service:** `reranker` container (already in the compose above) + 1727|**Why:** Dramatically improves RAG answer quality by filtering out noisy retrieval results before they reach the LLM. + 1728| + 1729|### 8.1 — How Reranking Works + 1730| + 1731|Without a reranker, your RAG pipeline does: + 1732|``` + 1733|Query → embed → Qdrant top-10 by vector similarity → all 10 chunks go to LLM + 1734|``` + 1735| + 1736|The problem: vector similarity often returns "close but irrelevant" chunks. The LLM gets noisy context and hallucinates. + 1737| + 1738|With a reranker: + 1739|``` + 1740|Query → embed → Qdrant top-50 by vector similarity → reranker scores each (query, chunk) pair → top-5 by relevance go to LLM + 1741|``` + 1742| + 1743|The reranker is a cross-encoder that reads the full query AND each chunk together, producing a much more accurate relevance score than vector distance alone. It over-retrieves cheaply from Qdrant, then precisely filters. + 1744| + 1745|### 8.2 — Scaffold Directories + 1746| + 1747|```bash + 1748|# Model cache on SSD (reranker model is ~1.1GB, downloaded on first start) + 1749|sudo mkdir -p /mnt/docker-ssd/docker/appdata/reranker + 1750|``` + 1751| + 1752|### 8.3 — Post-Deploy Verification + 1753| + 1754|```bash + 1755|# Container running? + 1756|docker ps --filter name=reranker + 1757| + 1758|# Health check? + 1759|curl -s http://localhost:8787/health + 1760| + 1761|# Test reranking + 1762|curl -s http://localhost:8787/rerank \ + 1763| -H "Content-Type: application/json" \ + 1764| -d '{ + 1765| "query": "How do I restart a Docker container?", + 1766| "texts": [ + 1767| "Use docker restart to restart a running container.", + 1768| "Docker was founded in 2013 by Solomon Hykes.", + 1769| "The docker compose down command stops and removes containers.", + 1770| "Kubernetes pods can be restarted by deleting them." + 1771| ] + 1772| }' | python3 -m json.tool + 1773|# Expected: the first text scores highest, second scores lowest + 1774|``` + 1775| + 1776|### 8.4 — n8n RAG Pipeline Integration + 1777| + 1778|Update the "Qdrant Index Updater" and any RAG query workflows to include a reranking step. + 1779| + 1780|**Updated RAG Query Workflow (for OpenWebUI or any n8n-based query):** + 1781|``` + 1782|Trigger: Webhook node (POST with query) + 1783|Step 1: HTTP Request → LiteLLM /v1/embeddings + 1784| - model: "embed" + 1785| - input: query text + 1786|Step 2: HTTP Request → Qdrant API (POST /collections/{name}/points/search) + 1787| - vector: embedding from step 1 + 1788| - limit: 50 (over-retrieve) + 1789|Step 3: HTTP Request → Reranker (POST http://reranker:80/rerank) + 1790| - query: original query text + 1791| - texts: array of 50 chunk texts from Qdrant results + 1792|Step 4: Code node → take top 5 by reranker score, format as context + 1793|Step 5: HTTP Request → LiteLLM /v1/chat/completions + 1794| - model: "medium" (or "heavy" for complex queries) + 1795| - messages: system prompt with top-5 context + user query + 1796|Step 6: Return response via webhook + 1797|``` + 1798| + 1799|> **Key difference from the original plan:** Step 2 now retrieves 50 results instead of 10, and Step 3 (reranking) filters down to the best 5. This "over-retrieve then rerank" pattern is the standard approach for production RAG systems. + 1800| + 1801|--- + 1802| + 1803|## Phase 9 — faster-whisper (Speech-to-Text) + 1804| + 1805|**Host:** PlausibleDeniability (deployed as part of the `ai` stack in Phase 7) + 1806|**Service:** `whisper` container (already in the compose above) + 1807|**Why:** Replaces shelved Scriberr with an OpenAI-compatible STT API. No SQLite dependency, no ZFS/ACL issues. + 1808| + 1809|### 9.1 — Scaffold Directories + 1810| + 1811|```bash + 1812|# Model cache on SSD (Whisper models are 1-3GB) + 1813|sudo mkdir -p /mnt/docker-ssd/docker/appdata/whisper + 1814|``` + 1815| + 1816|### 9.2 — Post-Deploy Verification + 1817| + 1818|```bash + 1819|# Container running? + 1820|docker ps --filter name=whisper + 1821| + 1822|# Health check? + 1823|curl -s http://localhost:8786/health + 1824| + 1825|# Test transcription with a sample audio file + 1826|curl -s http://localhost:8786/v1/audio/transcriptions \ + 1827| -F "file=@/path/to/test-audio.wav" \ + 1828| -F "model=Systran/faster-distil-whisper-large-v3" \ + 1829| | python3 -m json.tool + 1830|``` + 1831| + 1832|### 9.3 — n8n Integration Workflows + 1833| + 1834|**Workflow: Voice Note → Text → Summary** + 1835|``` + 1836|Trigger: Webhook node (POST with audio file in body) + 1837|Step 1: HTTP Request → Whisper (POST http://whisper:8000/v1/audio/transcriptions) + 1838| - Multipart form with audio file + 1839| - response_format: "json" + 1840|Step 2: HTTP Request → LiteLLM /v1/chat/completions + 1841| - model: "medium" + 1842| - Prompt: "Summarize this voice note concisely: {transcript}" + 1843|Step 3: HTTP Request → Gotify → push summary to phone + 1844|Step 4: (Optional) Postgres node → log transcript and summary + 1845|``` + 1846| + 1847|**Workflow: Audio File → Paperless Document** + 1848|``` + 1849|Trigger: Webhook or filesystem watcher + 1850|Step 1: HTTP Request → Whisper → get transcript + 1851|Step 2: Code node → format transcript as text document + 1852|Step 3: HTTP Request → Paperless API (POST /api/documents/post_document/) + 1853| - Upload transcript as .txt + 1854| - Tag: "transcription" + 1855|Step 4: Gotify notification: "Audio transcribed and filed: {title}" + 1856|``` + 1857| + 1858|**Home Assistant Voice Integration:** + 1859|If you want HA voice commands, Whisper can serve as the STT backend: + 1860|1. In Home Assistant → Settings → Voice Assistants + 1861|2. Add speech-to-text provider: "Whisper" at `http://PD_IP:8786` + 1862|3. Pair with Piper TTS (future addition) for full voice assistant loop + 1863| + 1864|### 9.4 — GPU vs CPU Considerations + 1865| + 1866|The compose file above uses the GPU variant with CUDA. If GPU contention with Ollama becomes an issue: + 1867| + 1868|**Option A — CPU fallback on PD:** + 1869|Change the image from `latest-cuda` to `fedirz/faster-whisper-server:latest-cpu` and remove the `deploy.resources` block and the CUDA environment variables (`WHISPER__DEVICE`, `WHISPER__COMPUTE_TYPE`). Transcription will be slower (~4x real-time instead of ~20x) but won't compete for VRAM. + 1870| + 1871|**Option B — Move to N.O.M.A.D.:** + 1872|Run Whisper on N.O.M.A.D.'s GTX 1080 which is less contested. Add it to N.O.M.A.D.'s compose or run standalone. N.O.M.A.D.'s 8GB VRAM handles Whisper models comfortably since phi4 doesn't fill it. + 1873| + 1874|--- + 1875| + 1876|## Additional Tools Setup + 1877| + 1878|### Continue.dev (Local AI Code Completion) + 1879| + 1880|1. Install the Continue extension in VS Code + 1881|2. Create/edit `~/.continue/config.json`: + 1882| + 1883|**With LiteLLM (recommended — see Phase 7):** + 1884|```json + 1885|{ + 1886| "models": [ + 1887| { + 1888| "title": "Heavy (ROCINANTE)", + 1889| "provider": "openai", + 1890| "model": "heavy", + 1891| "apiBase": "http://PD_TAILSCALE_IP:4000/v1", + 1892| "apiKey": "YOUR_LITELLM_MASTER_KEY" + 1893| }, + 1894| { + 1895| "title": "Medium (PD)", + 1896| "provider": "openai", + 1897| "model": "medium", + 1898| "apiBase": "http://PD_TAILSCALE_IP:4000/v1", + 1899| "apiKey": "YOUR_LITELLM_MASTER_KEY" + 1900| } + 1901| ], + 1902| "tabAutocompleteModel": { + 1903| "title": "Light (N.O.M.A.D.)", + 1904| "provider": "openai", + 1905| "model": "light", + 1906| "apiBase": "http://PD_TAILSCALE_IP:4000/v1", + 1907| "apiKey": "YOUR_LITELLM_MASTER_KEY" + 1908| } + 1909|} + 1910|``` + 1911| + 1912|> One IP, one API key, all three machines. If you add or move models, update LiteLLM's `config.yaml` — not every app. + 1913| + 1914|**Without LiteLLM (direct Ollama, if Phase 7 is not yet deployed):** + 1915|```json + 1916|{ + 1917| "models": [ + 1918| { + 1919| "title": "PD - qwen2.5:14b", + 1920| "provider": "ollama", + 1921| "model": "qwen2.5:14b", + 1922| "apiBase": "http://PD_TAILSCALE_IP:11434" + 1923| }, + 1924| { + 1925| "title": "ROCINANTE - qwen3:32b", + 1926| "provider": "ollama", + 1927| "model": "qwen3:32b", + 1928| "apiBase": "http://ROCINANTE_TAILSCALE_IP:11434" + 1929| } + 1930| ], + 1931| "tabAutocompleteModel": { + 1932| "title": "N.O.M.A.D. - phi4", + 1933| "provider": "ollama", + 1934| "model": "phi4", + 1935| "apiBase": "http://NOMAD_TAILSCALE_IP:11434" + 1936| } + 1937|} + 1938|``` + 1939| + 1940|### Obsidian + Gitea Sync + 1941| + 1942|1. In Obsidian, install the "Obsidian Git" community plugin + 1943|2. Initialize your vault as a git repo: + 1944| ```bash + 1945| cd /path/to/obsidian/vault + 1946| git init + 1947| git remote add origin http://PD_IP:3000/fizzlepoof/obsidian-vault.git + 1948| ``` + 1949|3. In Obsidian Git settings: + 1950| - Auto backup interval: 10 minutes + 1951| - Pull on startup: enabled + 1952|4. Create the repo in Gitea first: `http://PD_IP:3000` → New Repository → `obsidian-vault` + 1953|5. Add a Gitea webhook to trigger the n8n "Qdrant Index Updater" workflow (see Phase 3) + 1954| + 1955|### Homepage Ollama Widget + 1956| + 1957|Add to your Homepage configuration (`/mnt/tank/docker/appdata/homepage/services.yaml`): + 1958| + 1959|```yaml + 1960|- AI: + 1961| - Ollama (PD): + 1962| icon: ollama.svg + 1963| href: http://PD_IP:11434 + 1964| widget: + 1965| type: ollama + 1966| url: http://PD_IP:11434 + 1967| - Ollama (ROCINANTE): + 1968| icon: ollama.svg + 1969| href: http://ROCINANTE_IP:11434 + 1970| widget: + 1971| type: ollama + 1972| url: http://ROCINANTE_IP:11434 + 1973|``` + 1974| + 1975|--- + 1976| + 1977|## Deployment Order Summary + 1978| + 1979|``` + 1980|Week 1: Phase 1 — Gotify + 1981| └─ Deploy, create app tokens, install phone app + 1982| └─ Test: send manual notification via API + 1983| + 1984|Week 1: Phase 2 — Qdrant + 1985| └─ Deploy, verify REST API + 1986| └─ Create initial collections (empty, ready for n8n) + 1987| + 1988|Week 2: Phase 3 — n8n + 1989| └─ Deploy, create admin account + 1990| └─ Build workflows incrementally: + 1991| Day 1: Gitea commit → Gotify (simplest, proves the pipeline) + 1992| Day 2: Sonarr/Radarr → TMDB → Gotify + Discord + 1993| Day 3: Tautulli play logging + weekly digest + 1994| Day 4: Uptime Kuma enhanced alerts + 1995| Day 5: ZFS pool monitoring + 1996| Day 6: Multi-model query router + 1997| Day 7: Qdrant index updater + 1998| + 1999|Week 3: Phase 4 — Paperless-NGX + 2000| └─ Deploy, ingest test documents + 2001| \ No newline at end of file diff --git a/home/doris-barbell/app/templates/base.html b/home/doris-barbell/app/templates/base.html index f7251b0..1a11019 100644 --- a/home/doris-barbell/app/templates/base.html +++ b/home/doris-barbell/app/templates/base.html @@ -17,8 +17,8 @@ Doris Constabulary diff --git a/home/doris-kitchen/app/templates/base.html b/home/doris-kitchen/app/templates/base.html index 8be83c3..2406777 100644 --- a/home/doris-kitchen/app/templates/base.html +++ b/home/doris-kitchen/app/templates/base.html @@ -17,7 +17,7 @@ Doris Constabulary