Files

Fizzlepoof 2e99873634 chore: sync homelab ops, identity, and monitoring docs

2026-05-21 02:01:09 +00:00

67 KiB

Raw Blame History

Homelab Expansion — Full Buildout Plan

Generated: 2026-05-04 Based on: HOMELAB_EXPANSION_PLAN.md, ARCHITECTURE_OVERVIEW.md, STACK_STANDARDS.md

This document contains everything needed to deploy all six expansion phases — docker-compose files, .env templates, directory scaffolding commands, database init SQL, Pangolin configuration, and validation steps. Each phase follows the established stack standards and deployment checklist.

Prerequisites & Conventions
Phase 1 — Gotify (Notifications)
Phase 2 — Qdrant (Vector Database)
Phase 3 — n8n (Workflow Automation)
Phase 4 — Paperless-NGX (Document Intelligence)
Phase 5 — Home Assistant (Home Automation)
Phase 6 — Grafana + Prometheus (Observability)
Phase 7 — LiteLLM (AI Gateway)
Phase 8 — Reranker (RAG Quality)
Phase 9 — faster-whisper (Speech-to-Text)
Additional Tools Setup
Deployment Order Summary
Post-Deployment Validation Master Checklist

Prerequisites & Conventions

Deployment Standards (recap)

All stacks on PlausibleDeniability follow the same pattern:

Repo root: /mnt/docker-ssd/docker/compose
Validate before deploy: docker compose --env-file .env config
Deploy: docker compose --env-file .env up -d
Teardown: docker compose --env-file .env down

Storage Tiers

Tier	Mount	Use For
SSD	`/mnt/docker-ssd/docker/appdata/<service>`	Write-heavy, SQLite, GPU/model, databases
Tank	`/mnt/tank/docker/appdata/<service>`	General appdata, configs, uploads
Unraid	`/mnt/unraid/data/media/`	Media libraries only

Networks

Network	Created By	Purpose
`ix-databases_shared-databases`	databases stack	Access to shared-postgres, shared-mariadb, shared-redis
`pangolin`	newt (infrastructure stack)	Reverse proxy / external exposure

Secret Generation Commands

# Database passwords
openssl rand -hex 24

# JWT / encryption keys
openssl rand -hex 32

# Paperless secret key
python3 -c "import secrets; print(secrets.token_urlsafe(50))"

Deployment Order

databases (already running) → infrastructure (already running)
  → Phase 1: Gotify
  → Phase 2: Qdrant  (creates the ai-services network)
  → Phase 3: n8n     (depends on Gotify + Qdrant — joins ai-services as external)
  → Phase 4: Paperless-NGX (depends on n8n for automation hooks)
  → Phase 5: Home Assistant (depends on n8n for heavy automation)
  → Phase 6: Grafana + Prometheus (on N.O.M.A.D., independent but benefits from all above)

Important: The ai stack (Phase 2) must be deployed before the automation stack (Phase 3) because n8n declares ai-services as an external network. If the ai stack isn't up, docker compose up for automation will fail with a missing network error.

Phase 1 — Gotify (Notifications)

Host: PlausibleDeniability Stack directory: /mnt/docker-ssd/docker/compose/automation/ Why first: Every subsequent phase sends notifications through Gotify. It's the output bus.

1.1 — Scaffold Directories

# Appdata on SSD (SQLite backend — must not be on NFS)
sudo mkdir -p /mnt/docker-ssd/docker/appdata/gotify

# Stack directory (may already exist if automation/ is planned)
sudo mkdir -p /mnt/docker-ssd/docker/compose/automation

1.2 — Database Init

None required — Gotify uses an embedded SQLite database stored in its data volume.

1.3 — docker-compose.yaml

Add to /mnt/docker-ssd/docker/compose/automation/docker-compose.yaml:

name: automation

services:
  gotify:
    image: gotify/server:2.6.1
    container_name: gotify
    restart: unless-stopped
    ports:
      - "8484:80"
    environment:
      TZ: ${TZ}
      GOTIFY_DEFAULTUSER_PASS: ${GOTIFY_ADMIN_PASS}
      GOTIFY_SERVER_PORT: 80
      GOTIFY_DATABASE_DIALECT: sqlite3
      GOTIFY_DATABASE_CONNECTION: data/gotify.db
    volumes:
      - /mnt/docker-ssd/docker/appdata/gotify:/app/data
    networks:
      - pangolin
      - default

networks:
  pangolin:
    external: true

Image note: gotify/server:2.6.1 is the latest stable as of May 2026. Check Docker Hub before deploying — pin to the exact version.

1.4 — .env.example

# /mnt/docker-ssd/docker/compose/automation/.env.example

TZ=America/New_York
GOTIFY_ADMIN_PASS=CHANGE_ME

1.5 — .env (create from example)

cd /mnt/docker-ssd/docker/compose/automation
cp .env.example .env
# Edit .env with real values:
# GOTIFY_ADMIN_PASS=$(openssl rand -hex 16)
nano .env

1.6 — Pangolin Configuration

In your Pangolin dashboard, create a new resource:

Field	Value
Domain	`gotify.paccoco.com`
Scheme	`http`
Host	`gotify`
Port	`80`
Network	Docker service name resolution via `pangolin` network

1.7 — Validate & Deploy

cd /mnt/docker-ssd/docker/compose/automation
docker compose --env-file .env config
docker compose --env-file .env up -d

1.8 — Post-Deploy Verification

# Container running?
docker ps --filter name=gotify

# Clean startup logs?
docker logs gotify --tail 20

# Mounts correct?
docker inspect gotify --format '{{json .Mounts}}' | python3 -m json.tool

# Quick health check
curl -s http://localhost:8484/health

# Test notification via API
curl -s "http://localhost:8484/message?token=YOUR_APP_TOKEN" \
  -F "title=Homelab" \
  -F "message=Gotify is online" \
  -F "priority=5"

1.9 — First-Run Setup

Navigate to https://gotify.paccoco.com
Log in with the admin password from .env
Change the default admin password in the UI
Create application tokens for each notification source:
- n8n-workflows — for all n8n automations
- infrastructure-alerts — for Uptime Kuma, Grafana, etc.
- media-notifications — for Sonarr/Radarr/Tautulli hooks
- home-assistant — for HA automations
Create client tokens for each receiving device (phone, desktop)
Install the Gotify Android app and configure with your client token

Phase 2 — AI Stack (Vector DB, LLM, Embeddings, Reranker, STT)

Host: PlausibleDeniability Stack directory: /mnt/docker-ssd/docker/compose/ai/ Status: DEPLOYED AND VERIFIED (2026-05-05)

DEPLOYED NOTE: This phase was deployed as a complete AI stack with 6 services: Ollama (11434), OpenWebUI (8282), Qdrant (6333/6334), LiteLLM (4000), Reranker/TEI (8787), and faster-whisper (8786). The compose below shows the original Qdrant-only plan — see the actual running compose on PD at /mnt/docker-ssd/docker/compose/ai/docker-compose.yaml and the project_ai_stack_deployed.md memory for port mappings and fixes. Key lesson: always use 10.5.1.6 not localhost for health checks on TrueNAS Scale.

2.1 — Scaffold Directories

# Appdata on SSD (write-heavy vector storage)
sudo mkdir -p /mnt/docker-ssd/docker/appdata/qdrant/storage
sudo mkdir -p /mnt/docker-ssd/docker/appdata/qdrant/snapshots

# Stack directory
sudo mkdir -p /mnt/docker-ssd/docker/compose/ai

2.2 — docker-compose.yaml

Create or update /mnt/docker-ssd/docker/compose/ai/docker-compose.yaml:

name: ai

services:
  qdrant:
    image: qdrant/qdrant:v1.14.0
    container_name: qdrant
    restart: unless-stopped
    ports:
      - "6333:6333"   # REST API
      - "6334:6334"   # gRPC
    environment:
      TZ: ${TZ}
      QDRANT__SERVICE__GRPC_PORT: 6334
      QDRANT__STORAGE__STORAGE_PATH: /qdrant/storage
      QDRANT__STORAGE__SNAPSHOTS_PATH: /qdrant/snapshots
    volumes:
      - /mnt/docker-ssd/docker/appdata/qdrant/storage:/qdrant/storage
      - /mnt/docker-ssd/docker/appdata/qdrant/snapshots:/qdrant/snapshots
    networks:
      - ai-services
      - default

  # -------------------------------------------------------
  # Ollama and OpenWebUI go here when deployed.
  # They share this stack and the default + ai-services networks so
  # OpenWebUI can reach Qdrant at http://qdrant:6333
  # -------------------------------------------------------

  # ollama:
  #   image: ollama/ollama:latest
  #   container_name: ollama
  #   ...

  # openwebui:
  #   image: ghcr.io/open-webui/open-webui:main
  #   container_name: openwebui
  #   ...

networks:
  ai-services:
    name: ai-services

Image note: qdrant/qdrant:v1.14.0 is the latest stable as of May 2026. Check GitHub releases before deploying.

Cross-stack connectivity: The ai-services network is defined here with an explicit name: so other stacks (like automation/n8n) can declare it as external: true and reach Qdrant by service name (http://qdrant:6333). This follows the same pattern as ix-databases_shared-databases.

2.3 — .env.example

# /mnt/docker-ssd/docker/compose/ai/.env.example

TZ=America/New_York

2.4 — Validate & Deploy

cd /mnt/docker-ssd/docker/compose/ai
cp .env.example .env
nano .env  # Set timezone

docker compose --env-file .env config
docker compose --env-file .env up -d

2.5 — Post-Deploy Verification

# Container running?
docker ps --filter name=qdrant

# Clean startup?
docker logs qdrant --tail 20

# REST API responding?
curl -s http://localhost:6333/healthz
# Expected: {"title":"qdrant - vectorass engine","version":"1.14.0","commit":"..."}

# Create a test collection
curl -X PUT http://localhost:6333/collections/test_collection \
  -H "Content-Type: application/json" \
  -d '{
    "vectors": {
      "size": 384,
      "distance": "Cosine"
    }
  }'

# Verify it exists
curl -s http://localhost:6333/collections | python3 -m json.tool

# Clean up test
curl -X DELETE http://localhost:6333/collections/test_collection

2.6 — OpenWebUI Integration (when deployed)

When Ollama and OpenWebUI are brought online in this same stack, configure OpenWebUI's RAG settings:

Go to OpenWebUI → Admin → Settings → Documents
Set the vector database to Qdrant
Endpoint: http://qdrant:6333 (Docker DNS within the ai stack network)
Collection name: openwebui_docs (or your preference)

2.7 — Collections to Create (for n8n in Phase 3)

These collections will be created programmatically by n8n workflows, but for reference:

Collection	Vector Size	Content
`homelab_docs`	384 (nomic-embed-text)	Homelab markdown documentation
`gitea_commits`	384	Gitea commit messages + diffs
`media_metadata`	384	Plex/Tautulli metadata
`obsidian_notes`	384	Personal notes from Obsidian vault

Vector size note: 384 is the dimension for nomic-embed-text via Ollama. If you use a different embedding model, adjust accordingly.

Phase 3 — n8n (Workflow Automation)

Host: PlausibleDeniability Stack directory: /mnt/docker-ssd/docker/compose/automation/ (same stack as Gotify) Why third: n8n is the orchestration backbone — it ties Gotify, Qdrant, Ollama, and all triggers together.

3.1 — Scaffold Directories

# Config on tank (workflow definitions, credentials store)
# NOTE: n8n also writes execution logs here. If execution logging becomes
# heavy (many workflows running frequently), consider moving to SSD.
# For typical homelab usage (~20 workflows), tank is fine.
sudo mkdir -p /mnt/tank/docker/appdata/n8n

# Set ownership — n8n runs as UID 1000 (node user)
sudo chown 1000:1000 /mnt/tank/docker/appdata/n8n

3.2 — Database Init

n8n uses the existing shared-postgres. Create its database and user:

docker exec -i shared-postgres psql -U postgres <<'SQL'
CREATE USER n8n WITH PASSWORD 'REPLACE_WITH_GENERATED_PASSWORD';
CREATE DATABASE n8n OWNER n8n;
GRANT ALL PRIVILEGES ON DATABASE n8n TO n8n;
SQL

Generate the password first:

openssl rand -hex 24

3.3 — docker-compose.yaml

Update /mnt/docker-ssd/docker/compose/automation/docker-compose.yaml to add n8n alongside Gotify:

name: automation

services:
  gotify:
    image: gotify/server:2.6.1
    container_name: gotify
    restart: unless-stopped
    ports:
      - "8484:80"
    environment:
      TZ: ${TZ}
      GOTIFY_DEFAULTUSER_PASS: ${GOTIFY_ADMIN_PASS}
      GOTIFY_SERVER_PORT: 80
      GOTIFY_DATABASE_DIALECT: sqlite3
      GOTIFY_DATABASE_CONNECTION: data/gotify.db
    volumes:
      - /mnt/docker-ssd/docker/appdata/gotify:/app/data
    networks:
      - pangolin
      - default

  n8n:
    image: docker.n8n.io/n8nio/n8n:1.88.0
    container_name: n8n
    restart: unless-stopped
    ports:
      - "5678:5678"
    environment:
      TZ: ${TZ}
      # Database
      DB_TYPE: postgresdb
      DB_POSTGRESDB_HOST: shared-postgres
      DB_POSTGRESDB_PORT: 5432
      DB_POSTGRESDB_DATABASE: ${N8N_DB_NAME}
      DB_POSTGRESDB_USER: ${N8N_DB_USER}
      DB_POSTGRESDB_PASSWORD: ${N8N_DB_PASS}
      # Security
      N8N_ENCRYPTION_KEY: ${N8N_ENCRYPTION_KEY}
      # Webhook / Reverse Proxy
      N8N_HOST: ${N8N_HOST}
      N8N_PROTOCOL: https
      N8N_PORT: 5678
      WEBHOOK_URL: https://${N8N_HOST}/
      N8N_PROXY_HOPS: 1
      # General
      GENERIC_TIMEZONE: ${TZ}
      N8N_DIAGNOSTICS_ENABLED: false
      N8N_PERSONALIZATION_ENABLED: false
    volumes:
      - /mnt/tank/docker/appdata/n8n:/home/node/.n8n
    networks:
      - pangolin
      - ix-databases_shared-databases
      - ai-services
      - default
    depends_on:
      - gotify

networks:
  pangolin:
    external: true
  ix-databases_shared-databases:
    external: true
  ai-services:
    external: true

Image note: docker.n8n.io/n8nio/n8n:1.88.0 — n8n uses calendar-ish versioning. Check n8n releases for the latest stable. The docker.n8n.io registry is preferred for production.

Cross-stack access: n8n joins ai-services (created by the ai stack) so it can reach Qdrant at http://qdrant:6333 by Docker DNS. It also joins ix-databases_shared-databases for Postgres access — same pattern.

3.4 — .env.example (updated for both services)

# /mnt/docker-ssd/docker/compose/automation/.env.example

TZ=America/New_York

# Gotify
GOTIFY_ADMIN_PASS=CHANGE_ME

# n8n — Database
N8N_DB_NAME=n8n
N8N_DB_USER=n8n
N8N_DB_PASS=CHANGE_ME

# n8n — Security
N8N_ENCRYPTION_KEY=CHANGE_ME

# n8n — Hostname
N8N_HOST=n8n.paccoco.com

3.5 — Pangolin Configuration

Field	Value
Domain	`n8n.paccoco.com`
Scheme	`http`
Host	`n8n`
Port	`5678`
Headers	Forward `X-Forwarded-Proto: https`

3.6 — Validate & Deploy

cd /mnt/docker-ssd/docker/compose/automation

# Update .env with real passwords
nano .env

docker compose --env-file .env config
docker compose --env-file .env up -d

3.7 — Post-Deploy Verification

# Both containers running?
docker ps --filter name=gotify --filter name=n8n

# n8n logs clean?
docker logs n8n --tail 30

# n8n on correct networks?
docker inspect n8n --format '{{json .NetworkSettings.Networks}}' | python3 -m json.tool

# n8n UI accessible?
curl -s -o /dev/null -w "%{http_code}" http://localhost:5678/
# Expected: 200

# Postgres connection working? (check logs for DB migration messages)
docker logs n8n 2>&1 | grep -i "migrat"

3.8 — First-Run Setup

Navigate to https://n8n.paccoco.com
Create your admin account
Install community nodes you'll need:
- n8n-nodes-gotify (if available) or use HTTP Request node
- Ollama nodes (built-in as of n8n 1.x)

3.9 — Workflow Blueprints

Below are starter workflow descriptions for each planned automation. These are meant to be built in the n8n UI — the structure and node types are described so you can wire them up.

Media Pipeline Workflows

Workflow: Sonarr/Radarr Download Notification

Trigger:  Webhook node (POST from Sonarr/Radarr on download/import)
Step 1:   Extract series/movie name, quality, file path from webhook body
Step 2:   HTTP Request → TMDB API to fetch poster image URL
Step 3:   HTTP Request → Gotify REST API (POST /message)
            - Title: "New Download: {title}"
            - Message: "{quality} — {episodeTitle or year}"
            - Priority: 5
            - Extras: attach poster URL as markdown image
Also:     HTTP Request → Discord webhook (formatted embed with poster)

Configure Sonarr/Radarr webhooks:

Sonarr: Settings → Connect → Webhook → URL: https://n8n.paccoco.com/webhook/sonarr
Radarr: Settings → Connect → Webhook → URL: https://n8n.paccoco.com/webhook/radarr

Workflow: Tautulli Play Logging

Trigger:  Webhook node (POST from Tautulli on playback start)
Step 1:   Extract user, media title, player, quality from payload
Step 2:   Postgres node → INSERT into watch_history table
Step 3:   (Optional) Gotify notification for specific users/media

Tautulli config: Settings → Notification Agents → Webhook → URL: https://n8n.paccoco.com/webhook/tautulli-play

Workflow: Weekly Watch Digest

Trigger:  Cron node (every Sunday at 10:00 AM)
Step 1:   Postgres node → SELECT watch history for past 7 days
Step 2:   Format data as structured text
Step 3:   HTTP Request → Ollama API (POST to PD's qwen2.5:14b)
            - Prompt: "Summarize this week's viewing in a fun digest: {data}"
Step 4:   HTTP Request → Gotify (send digest)

Infrastructure Monitoring Workflows

Workflow: Uptime Kuma Enhanced Alerts

Trigger:  Webhook node (from Uptime Kuma notification)
Step 1:   Extract monitor name, status, response time
Step 2:   HTTP Request → Netdata API for related metrics context
Step 3:   HTTP Request → Gotify
            - Title: "🔴 {monitor} DOWN" or "🟢 {monitor} UP"
            - Message: include Netdata context (CPU, mem, disk at time of alert)
            - Priority: 8 (high for down, 3 for recovery)

Workflow: ZFS Pool Utilization Alert

Trigger:  Cron node (every 6 hours)
Step 1:   SSH node → Serenity: `zpool list -Hp malcolm`
Step 2:   Parse capacity percentage
Step 3:   IF capacity > 85% → Gotify alert (priority 8)
Step 4:   IF capacity > 90% → Gotify alert (priority 10) + Discord webhook

Workflow: Grafana Alert Remediation

Trigger:  Webhook node (from Grafana alerting)
Step 1:   Parse alert labels (container, host, metric)
Step 2:   Switch node → route by alert type:
            - High CPU container → SSH → docker restart {container}
            - Disk full → SSH → pause qBittorrent, notify via Gotify
            - Memory pressure → Gotify alert only (manual intervention)
Step 3:   Log action taken to Postgres

Homelab Ops Workflows

Workflow: Gitea Commit Summary

Trigger:  Webhook node (Gitea webhook on push to truenas-stacks)
Step 1:   Extract commit messages, author, files changed
Step 2:   HTTP Request → Ollama API
            - Prompt: "Summarize this commit in one sentence: {commit_message}"
Step 3:   HTTP Request → Gotify
            - Title: "Commit to truenas-stacks"
            - Message: Ollama-generated summary

Gitea config: Repository → Settings → Webhooks → URL: https://n8n.paccoco.com/webhook/gitea-push

Workflow: qBittorrent Auto-Rescan

Trigger:  Webhook or polling (qBittorrent API for completed+moved torrents)
Step 1:   Determine if file is in Sonarr or Radarr path
Step 2:   HTTP Request → Sonarr API (POST /command → RescanSeries)
           OR HTTP Request → Radarr API (POST /command → RescanMovie)
Step 3:   Gotify notification confirming rescan triggered

AI Pipeline Workflows

Workflow: URL Digest Pipeline

Trigger:  Webhook node (POST with URL in body)
Step 1:   HTTP Request → fetch page content
Step 2:   Code node → extract text, chunk into ~500 token segments
Step 3:   HTTP Request → Ollama API → summarize each chunk
Step 4:   Code node → combine summaries into digest
Step 5:   Postgres node → store digest with metadata
Step 6:   Return digest in webhook response

Workflow: Multi-Model Query Router (simplified by LiteLLM — see Phase 7)

Trigger:  Webhook node (POST with query + complexity hint)
Step 1:   HTTP Request → LiteLLM (POST http://litellm:4000/v1/chat/completions)
            - model: complexity parameter ("light", "medium", or "heavy")
            - LiteLLM handles routing to the correct Ollama instance
Step 2:   Return response via webhook

With LiteLLM deployed, the Switch node and three separate Ollama endpoints are replaced by a single HTTP Request node. The routing logic lives in LiteLLM's config.yaml instead of n8n workflow logic.

Workflow: Qdrant Index Updater

Trigger:  Webhook node (from Gitea push to any watched repo)
Step 1:   HTTP Request → Gitea API → fetch changed files content
Step 2:   Code node → chunk text into embedding-sized segments
Step 3:   HTTP Request → Ollama API (embed endpoint with nomic-embed-text)
Step 4:   HTTP Request → Qdrant API (PUT /collections/homelab_docs/points)
Step 5:   Gotify notification: "{n} documents re-indexed"

Home / Business Workflows

Workflow: KitchenOwl Grocery Notification

Trigger:  Polling (KitchenOwl API) or webhook if supported
Step 1:   Fetch current shopping list items
Step 2:   Format as clean text list
Step 3:   HTTP Request → Gotify → phone push notification

Workflow: Donetick Task Reminder

Trigger:  Cron node (daily at 9:00 AM)
Step 1:   HTTP Request → Donetick API → fetch tasks due today/overdue
Step 2:   Format task list
Step 3:   HTTP Request → Gotify (priority 5)

Workflow: Long and Low Crafts Order Pipeline

Trigger:  Webhook (Etsy webhook or email trigger via IMAP node)
Step 1:   Parse order details (item, quantity, customer, shipping)
Step 2:   HTTP Request → Donetick API → create fulfillment task
Step 3:   HTTP Request → Gotify DM notification
            - Title: "New L&L Order"
            - Message: "{item} x{qty} — ship by {date}"
            - Priority: 7

Phase 4 — Paperless-NGX (Document Intelligence)

Host: PlausibleDeniability Stack directory: /mnt/docker-ssd/docker/compose/documents/ (new stack) Why fourth: Depends on n8n for automation hooks and Gotify for notifications.

4.1 — Scaffold Directories

# Index/data on SSD (search index is write-heavy)
sudo mkdir -p /mnt/docker-ssd/docker/appdata/paperless/data

# Documents (media) on tank (bulk storage, read-heavy)
sudo mkdir -p /mnt/tank/docker/appdata/paperless/media

# Consume folder on tank (drop zone for new documents)
sudo mkdir -p /mnt/tank/docker/appdata/paperless/consume

# Export folder on tank
sudo mkdir -p /mnt/tank/docker/appdata/paperless/export

# Stack directory
sudo mkdir -p /mnt/docker-ssd/docker/compose/documents

4.2 — Database Init

Paperless uses the existing shared-postgres and shared-redis:

docker exec -i shared-postgres psql -U postgres <<'SQL'
CREATE USER paperless WITH PASSWORD 'REPLACE_WITH_GENERATED_PASSWORD';
CREATE DATABASE paperless OWNER paperless;
GRANT ALL PRIVILEGES ON DATABASE paperless TO paperless;
SQL

4.3 — docker-compose.yaml

Create /mnt/docker-ssd/docker/compose/documents/docker-compose.yaml:

name: documents

services:
  paperless:
    image: ghcr.io/paperless-ngx/paperless-ngx:2.16
    container_name: paperless
    restart: unless-stopped
    ports:
      - "8000:8000"
    environment:
      TZ: ${TZ}
      # Database
      PAPERLESS_DBENGINE: postgresql
      PAPERLESS_DBHOST: shared-postgres
      PAPERLESS_DBPORT: 5432
      PAPERLESS_DBNAME: ${PAPERLESS_DB_NAME}
      PAPERLESS_DBUSER: ${PAPERLESS_DB_USER}
      PAPERLESS_DBPASS: ${PAPERLESS_DB_PASS}
      # Redis (using shared-redis)
      PAPERLESS_REDIS: redis://shared-redis:6379
      # Security
      PAPERLESS_SECRET_KEY: ${PAPERLESS_SECRET_KEY}
      PAPERLESS_URL: https://${PAPERLESS_HOST}
      PAPERLESS_ADMIN_USER: ${PAPERLESS_ADMIN_USER}
      PAPERLESS_ADMIN_PASSWORD: ${PAPERLESS_ADMIN_PASS}
      # OCR
      PAPERLESS_OCR_LANGUAGE: eng
      PAPERLESS_OCR_MODE: skip
      # Tika/Gotenberg for Office documents
      PAPERLESS_TIKA_ENABLED: 1
      PAPERLESS_TIKA_ENDPOINT: http://tika:9998
      PAPERLESS_TIKA_GOTENBERG_ENDPOINT: http://gotenberg:3000
      # Performance
      PAPERLESS_TASK_WORKERS: 2
      PAPERLESS_THREADS_PER_WORKER: 2
    volumes:
      - /mnt/docker-ssd/docker/appdata/paperless/data:/usr/src/paperless/data
      - /mnt/tank/docker/appdata/paperless/media:/usr/src/paperless/media
      - /mnt/tank/docker/appdata/paperless/consume:/usr/src/paperless/consume
      - /mnt/tank/docker/appdata/paperless/export:/usr/src/paperless/export
    networks:
      - pangolin
      - ix-databases_shared-databases
      - default
    depends_on:
      - gotenberg
      - tika

  gotenberg:
    image: gotenberg/gotenberg:8.17
    container_name: paperless-gotenberg
    restart: unless-stopped
    command:
      - "gotenberg"
      - "--chromium-disable-javascript=true"
      - "--chromium-allow-list=file:///tmp/.*"
    networks:
      - default

  tika:
    image: apache/tika:3.1
    container_name: paperless-tika
    restart: unless-stopped
    networks:
      - default

networks:
  pangolin:
    external: true
  ix-databases_shared-databases:
    external: true

Image notes:

ghcr.io/paperless-ngx/paperless-ngx:2.16 — check releases for latest.

gotenberg/gotenberg:8.17 — pin to a specific 8.x tag.

apache/tika:3.1 — needed for Office doc support (.docx, .xlsx, .odt).

Gotenberg and Tika are only needed if you ingest Office documents. For PDF-only, you can omit them and set PAPERLESS_TIKA_ENABLED: 0.

4.4 — .env.example

# /mnt/docker-ssd/docker/compose/documents/.env.example

TZ=America/New_York

# Paperless — Database
PAPERLESS_DB_NAME=paperless
PAPERLESS_DB_USER=paperless
PAPERLESS_DB_PASS=CHANGE_ME

# Paperless — Security
PAPERLESS_SECRET_KEY=CHANGE_ME
PAPERLESS_ADMIN_USER=admin
PAPERLESS_ADMIN_PASS=CHANGE_ME

# Paperless — Hostname
PAPERLESS_HOST=paperless.paccoco.com

4.5 — Pangolin Configuration

Field	Value
Domain	`paperless.paccoco.com`
Scheme	`http`
Host	`paperless`
Port	`8000`

4.6 — Validate & Deploy

cd /mnt/docker-ssd/docker/compose/documents
cp .env.example .env
nano .env  # Fill in real values

docker compose --env-file .env config
docker compose --env-file .env up -d

4.7 — Post-Deploy Verification

# All three containers running?
docker ps --filter name=paperless --filter name=paperless-gotenberg --filter name=paperless-tika

# Paperless logs clean? (watch for DB migration output)
docker logs paperless --tail 30

# Network connectivity to shared-postgres and shared-redis?
docker exec paperless python3 -c "import psycopg2; print('postgres OK')" 2>/dev/null || echo "Check DB connection"

# UI accessible?
curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/
# Expected: 200 or 302 (redirect to login)

4.8 — n8n Integration Workflows

Workflow: Email Attachment → Paperless

Trigger:  IMAP node (poll email inbox every 5 minutes)
Step 1:   Filter for emails with PDF/document attachments
Step 2:   Download attachment binary
Step 3:   HTTP Request → Paperless API (POST /api/documents/post_document/)
            - Multipart form with document file
            - Optional: set correspondent, document type, tags
Step 4:   Wait node (30 seconds for OCR processing)
Step 5:   HTTP Request → Paperless API (GET document details)
Step 6:   IF tagged as "action-required" →
            HTTP Request → Donetick API → create task
Step 7:   HTTP Request → Gotify → "New document ingested: {title}"

Workflow: Ollama Document Summarizer

Trigger:  Paperless webhook (on document consumed) or n8n polling
Step 1:   HTTP Request → Paperless API → fetch document text
Step 2:   HTTP Request → Ollama API (PD's qwen2.5:14b)
            - Prompt: "Extract key data from this document: dates, amounts, parties, action items. {text}"
Step 3:   HTTP Request → Paperless API → update document notes with summary
Step 4:   HTTP Request → Gotify → "Document summarized: {title}"

Phase 5 — Home Assistant (Home Automation)

Host: PlausibleDeniability (preferred — same network as most services) Stack directory: /mnt/docker-ssd/docker/compose/homeassistant/ (standalone stack — host networking isolates it) Why fifth: Benefits from n8n being operational. n8n handles complex automation logic while HA handles device control.

5.1 — Scaffold Directories

# Config on SSD (SQLite database, write-heavy)
sudo mkdir -p /mnt/docker-ssd/docker/appdata/homeassistant

5.2 — docker-compose.yaml

Create /mnt/docker-ssd/docker/compose/homeassistant/docker-compose.yaml:

name: homeassistant

services:
  homeassistant:
    image: ghcr.io/home-assistant/home-assistant:2026.5
    container_name: homeassistant
    restart: unless-stopped
    privileged: true
    network_mode: host
    environment:
      TZ: ${TZ}
    volumes:
      - /mnt/docker-ssd/docker/appdata/homeassistant:/config
      - /etc/localtime:/etc/localtime:ro
      - /run/dbus:/run/dbus:ro
    # devices:
    #   - /dev/ttyUSB0:/dev/ttyUSB0   # Uncomment for USB Zigbee/Z-Wave sticks

Image note: ghcr.io/home-assistant/home-assistant:2026.5 — use the latest stable monthly release. Check HA releases. Pin to a specific minor like 2026.5.1 once you confirm it's stable.

Why host networking: Home Assistant requires network_mode: host for mDNS/SSDP device discovery. This means it does NOT join pangolin — you'll access it directly by IP and port, or configure Pangolin to proxy to http://10.5.1.X:8123.

Why standalone stack: Host networking is incompatible with other services in the same compose that use bridge networking. HA must be in its own compose file.

5.3 — .env.example

# /mnt/docker-ssd/docker/compose/homeassistant/.env.example

TZ=America/New_York

5.4 — Pangolin Configuration

Since HA uses host networking, Pangolin needs to reach it by the host's IP:

Field	Value
Domain	`ha.paccoco.com`
Scheme	`http`
Host	`10.5.1.X` (PlausibleDeniability's LAN IP)
Port	`8123`

Find PD's IP with: hostname -I | awk '{print $1}'

5.5 — Validate & Deploy

cd /mnt/docker-ssd/docker/compose/homeassistant
cp .env.example .env
nano .env

docker compose --env-file .env config
docker compose --env-file .env up -d

5.6 — Post-Deploy Verification

# Container running?
docker ps --filter name=homeassistant

# Clean startup?
docker logs homeassistant --tail 30

# Web UI accessible?
curl -s -o /dev/null -w "%{http_code}" http://localhost:8123/
# Expected: 200

5.7 — First-Run Setup & Integrations

Navigate to http://PD_IP:8123 (or https://ha.paccoco.com)
Complete the onboarding wizard
Add integrations:

Smart Plug Power Monitoring:

Install TP-Link Kasa / Tapo / Shelly integration (depends on your plug brand)
Add plugs for Serenity, PD, N.O.M.A.D.
Create energy dashboard for power consumption tracking

UPS Monitoring:

Install NUT (Network UPS Tools) integration if you add a UPS
Note: PD and N.O.M.A.D. currently have no UPS — this is a known gap
When UPS is added, create automations for graceful shutdown via n8n

Presence Detection:

Mobile app integration for phone-based presence
Or use router/UniFi integration for network-based detection

5.8 — n8n ↔ Home Assistant Integration

In n8n, use the Home Assistant nodes (built-in):

In HA: Profile → Long-Lived Access Tokens → Create Token
In n8n: Settings → Credentials → Home Assistant API
- Host: http://10.5.1.X:8123
- Access Token: (paste from step 1)

Workflow: Presence → Server Sleep/Wake

Trigger:  HA event node (person.fizzlepoof state change)
Step 1:   IF state = "not_home" for > 30 min AND no active Plex streams:
            SSH → N.O.M.A.D.: systemctl suspend
Step 2:   IF state = "home":
            Wake-on-LAN → N.O.M.A.D. MAC address
Step 3:   Gotify notification: "Server {action}: N.O.M.A.D."

Workflow: Physical Device → n8n Trigger

Trigger:  HA webhook or event node (smart plug power draw spike)
Step 1:   IF washing machine plug power < 5W for 5 min after being > 100W:
            Gotify notification: "Laundry is done!"

Phase 6 — Grafana + Prometheus (Observability)

Host: N.O.M.A.D. (Ubuntu 25.10 — dedicated HDDs have headroom) Stack directory: /opt/monitoring/ (outside the N.O.M.A.D. project directory) Why last: Observability benefits from all other services being online — more to monitor.

6.1 — Scaffold Directories

SSH into N.O.M.A.D.:

ssh nomad@10.5.1.16

# Stack directory
sudo mkdir -p /opt/monitoring

# Prometheus data on hdd-2 (has more headroom)
sudo mkdir -p /mnt/hdd-2/prometheus-data
sudo chown 65534:65534 /mnt/hdd-2/prometheus-data  # nobody user (Prometheus default)

# Grafana data on hdd-2
sudo mkdir -p /mnt/hdd-2/grafana-data
sudo chown 472:472 /mnt/hdd-2/grafana-data  # grafana user

# Config directories
sudo mkdir -p /opt/monitoring/provisioning/datasources
sudo mkdir -p /opt/monitoring/provisioning/dashboards

6.2 — prometheus.yml

Create /opt/monitoring/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # ------- Local (N.O.M.A.D.) -------
  - job_name: 'nomad-node'
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          host: 'nomad'

  # ------- PlausibleDeniability -------
  - job_name: 'pd-netdata'
    metrics_path: /api/v1/allmetrics
    params:
      format: [prometheus]
    static_configs:
      - targets: ['10.5.1.X:19999']    # Replace with PD's IP
        labels:
          host: 'plausible-deniability'

  # ------- Serenity -------
  - job_name: 'serenity-netdata'
    metrics_path: /api/v1/allmetrics
    params:
      format: [prometheus]
    static_configs:
      - targets: ['10.5.1.5:19999']
        labels:
          host: 'serenity'

  # ------- Gotify -------
  # Gotify exposes /health but no Prometheus endpoint natively.
  # Use blackbox exporter or just rely on Uptime Kuma.

  # ------- n8n -------
  # n8n doesn't expose Prometheus metrics by default.
  # Monitor via container resource metrics from Netdata.

  # -------- Add more targets as needed --------
  # When node-exporter is installed on PD and Serenity:
  # - job_name: 'pd-node'
  #   static_configs:
  #     - targets: ['10.5.1.X:9100']
  #       labels:
  #         host: 'plausible-deniability'
  #
  # - job_name: 'serenity-node'
  #   static_configs:
  #     - targets: ['10.5.1.5:9100']
  #       labels:
  #         host: 'serenity'

Netdata as a Prometheus target: Both PD and Serenity already run Netdata. Netdata has a built-in Prometheus exporter at /api/v1/allmetrics?format=prometheus. This gives you CPU, memory, disk, network, and ZFS metrics without installing node-exporter on those hosts.

6.3 — Grafana Provisioning

Create /opt/monitoring/provisioning/datasources/prometheus.yml:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true

Create /opt/monitoring/provisioning/dashboards/dashboards.yml:

apiVersion: 1

providers:
  - name: 'default'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    editable: true
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: false

6.4 — docker-compose.yaml

Create /opt/monitoring/docker-compose.yaml:

name: monitoring

services:
  prometheus:
    image: prom/prometheus:v3.4.0
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=90d'
      - '--web.enable-lifecycle'
    volumes:
      - /opt/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - /mnt/hdd-2/prometheus-data:/prometheus
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:13.0.1
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      TZ: ${TZ}
      GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER}
      GF_SECURITY_ADMIN_PASSWORD: ${GF_ADMIN_PASS}
      GF_SERVER_ROOT_URL: https://${GF_HOST}/
    volumes:
      - /mnt/hdd-2/grafana-data:/var/lib/grafana
      - /opt/monitoring/provisioning:/etc/grafana/provisioning:ro
    networks:
      - monitoring

  node-exporter:
    image: prom/node-exporter:v1.9.0
    container_name: node-exporter
    restart: unless-stopped
    ports:
      - "9100:9100"
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

Image notes:

prom/prometheus:v3.4.0 — Prometheus v3 is the current major line. Do not use :latest — it still resolves to 2.x due to a known tagging issue. Always use an explicit v3.x.y tag.

grafana/grafana:latest — use the moving stable image by default here; only pin Grafana if a breakage or compatibility reason is documented.

prom/node-exporter:v1.9.0 — check releases.

6.5 — .env.example

# /opt/monitoring/.env.example

TZ=America/New_York

# Grafana
GF_ADMIN_USER=admin
GF_ADMIN_PASS=CHANGE_ME
GF_HOST=grafana.paccoco.com

6.6 — Pangolin Configuration

Grafana runs on N.O.M.A.D., not PD where the main Newt agent lives. You have two options:

Option A — Route via N.O.M.A.D.'s existing Newt (recommended if already connected)

N.O.M.A.D. already has a Newt container from the Project N.O.M.A.D. setup. If it's connected to your Pangolin VPS, just add a new resource in the Pangolin dashboard pointing to http://grafana:3000 or http://10.5.1.16:3000.

Option B — Add a dedicated Newt to the monitoring stack

If N.O.M.A.D.'s existing Newt is not connected to Pangolin (or is a separate Pangolin instance), add Newt to the monitoring compose:

  newt:
    image: ghcr.io/fosrl/newt:latest
    container_name: monitoring-newt
    restart: unless-stopped
    environment:
      PANGOLIN_ENDPOINT: ${PANGOLIN_ENDPOINT}
      NEWT_ID: ${NEWT_ID}
      NEWT_SECRET: ${NEWT_SECRET}
    networks:
      - monitoring

Then in Pangolin dashboard:

Field	Value
Domain	`grafana.paccoco.com`
Scheme	`http`
Host	`grafana` (Docker DNS via shared network)
Port	`3000`

Option C — Direct IP routing (simplest, no Newt needed)

If Pangolin's Newt on PD can reach N.O.M.A.D. by LAN IP (they're on the same subnet):

Field	Value
Domain	`grafana.paccoco.com`
Scheme	`http`
Host	`10.5.1.16`
Port	`3000`

Decision point: Check if N.O.M.A.D.'s existing Newt is connected to your Pangolin instance before deploying. Run docker ps --filter name=newt on N.O.M.A.D. to verify. If it's running and connected, Option A is zero-effort. If not, Option C is the simplest fallback.

6.7 — Validate & Deploy

ssh nomad@10.5.1.16

cd /opt/monitoring
cp .env.example .env
nano .env  # Fill in real values

docker compose --env-file .env config
docker compose --env-file .env up -d

6.8 — Post-Deploy Verification

# All three containers running?
docker ps --filter name=prometheus --filter name=grafana --filter name=node-exporter

# Prometheus scraping targets?
curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | head -40

# Grafana UI accessible?
curl -s -o /dev/null -w "%{http_code}" http://localhost:3000/
# Expected: 200 or 302

# Node exporter metrics flowing?
curl -s http://localhost:9100/metrics | head -10

6.9 — Recommended Dashboards

Import these from Grafana Dashboard Library:

Dashboard	ID	Purpose
Node Exporter Full	1860	System metrics for N.O.M.A.D.
Docker Container Stats	893	Container resource usage
Netdata via Prometheus	(search)	PD and Serenity system metrics

To import: Grafana → Dashboards → New → Import → Enter dashboard ID.

6.10 — Metrics Targets Summary

Target	Host	Method	Endpoint
N.O.M.A.D. system	localhost	node-exporter	node-exporter:9100
PD system	10.5.1.X	Netdata Prometheus	10.5.1.X:19999/api/v1/allmetrics
Serenity system	10.5.1.5	Netdata Prometheus	10.5.1.5:19999/api/v1/allmetrics
ZFS pools	via Netdata	Netdata exports ZFS metrics	Included in Netdata scrape
Container stats	via Netdata	Netdata exports cgroup metrics	Included in Netdata scrape
Plex streams	Tautulli	n8n polling → Postgres	Via n8n workflow (Phase 3)
qBit stats	qBittorrent API	n8n polling → Postgres	Via n8n workflow (Phase 3)
Tailscale latency	Tailscale API	n8n polling → Postgres	Via n8n workflow (Phase 3)

6.11 — n8n Integration (Grafana → n8n alert webhook)

In Grafana → Alerting → Contact Points, create a webhook contact point:

Field	Value
Name	`n8n-alerts`
Type	Webhook
URL	`https://n8n.paccoco.com/webhook/grafana-alert`
HTTP Method	POST

Then create alert rules for:

ZFS pool utilization > 85%
Container memory > 90% of limit
Host CPU sustained > 90% for 5 minutes
Disk I/O latency spikes

These fire into the "Grafana Alert Remediation" n8n workflow (see Phase 3).

Phase 7 — LiteLLM (AI Gateway)

Host: PlausibleDeniability Stack directory: /mnt/docker-ssd/docker/compose/ai/ (same stack as Qdrant) Why: Replaces manual multi-model routing with a unified OpenAI-compatible API. Every app (OpenWebUI, Continue.dev, n8n, Paperless summarization) points at one endpoint instead of juggling three Ollama IPs.

7.1 — What LiteLLM Does

LiteLLM sits in front of your three Ollama instances and presents a single OpenAI-compatible API at http://litellm:4000. It handles:

Model routing — requests for qwen3:32b go to ROCINANTE, qwen2.5:14b goes to PD, phi4 goes to N.O.M.A.D.
Failover — if ROCINANTE is offline, LiteLLM can fall back to PD automatically
Load balancing — distribute requests across instances running the same model
Usage tracking — logs token counts, latency, and costs per model/user via its built-in database

7.2 — Scaffold Directories

# Config on SSD (SQLite DB for usage tracking)
sudo mkdir -p /mnt/docker-ssd/docker/appdata/litellm

# LiteLLM config file
sudo mkdir -p /mnt/docker-ssd/docker/compose/ai/litellm

7.3 — LiteLLM Config

Create /mnt/docker-ssd/docker/compose/ai/litellm/config.yaml:

model_list:
  # ---- ROCINANTE (RTX 4090, 24GB) — heavy reasoning ----
  - model_name: "heavy"
    litellm_params:
      model: "ollama/qwen3:32b"
      api_base: "http://10.5.1.ROCINANTE:11434"
      timeout: 300
      stream_timeout: 300
    model_info:
      description: "Heavy reasoning, long context, complex code"

  - model_name: "heavy"
    litellm_params:
      model: "ollama/deepseek-r1:32b"
      api_base: "http://10.5.1.ROCINANTE:11434"
      timeout: 300
      stream_timeout: 300
    model_info:
      description: "Deep reasoning fallback on ROCINANTE"

  # ---- PlausibleDeniability (RTX 2080 Ti, 11GB) — general ----
  - model_name: "medium"
    litellm_params:
      model: "ollama/qwen2.5:14b"
      api_base: "http://host.docker.internal:11434"
      timeout: 120
      stream_timeout: 120
    model_info:
      description: "General homelab assistant, RAG queries"

  # ---- N.O.M.A.D. (GTX 1080, 8GB) — lightweight ----
  - model_name: "light"
    litellm_params:
      model: "ollama/phi4"
      api_base: "http://10.5.1.16:11434"
      timeout: 60
      stream_timeout: 60
    model_info:
      description: "Fast local inference, lightweight tasks"

  - model_name: "light"
    litellm_params:
      model: "ollama/llama3.2:3b"
      api_base: "http://10.5.1.16:11434"
      timeout: 60
      stream_timeout: 60
    model_info:
      description: "Ultra-light fallback on N.O.M.A.D."

  # ---- Embeddings ----
  - model_name: "embed"
    litellm_params:
      model: "ollama/nomic-embed-text"
      api_base: "http://host.docker.internal:11434"
    model_info:
      description: "Text embeddings for RAG pipeline"

  # ---- Direct model access (bypass routing) ----
  # These let you request a specific model by its full name
  - model_name: "ollama/qwen3:32b"
    litellm_params:
      model: "ollama/qwen3:32b"
      api_base: "http://10.5.1.ROCINANTE:11434"

  - model_name: "ollama/qwen2.5:14b"
    litellm_params:
      model: "ollama/qwen2.5:14b"
      api_base: "http://host.docker.internal:11434"

  - model_name: "ollama/phi4"
    litellm_params:
      model: "ollama/phi4"
      api_base: "http://10.5.1.16:11434"

litellm_settings:
  drop_params: true
  set_verbose: false
  request_timeout: 300
  num_retries: 2
  retry_after: 5
  allowed_fails: 3
  cooldown_time: 60

general_settings:
  master_key: "os.environ/LITELLM_MASTER_KEY"

DEPLOYED NOTE (2026-05-05): Do NOT add database_url to general_settings — newer LiteLLM versions include Prisma ORM that requires PostgreSQL, and adding any database_url causes a crash loop. Without it, LiteLLM runs in config-only mode which is fine for homelab use. The master_key uses os.environ/LITELLM_MASTER_KEY syntax to read from the container's environment variable (set via .env → docker compose).

Routing explained: Multiple entries with the same model_name (like "heavy") enable load balancing and failover within that tier. When you request model heavy, LiteLLM picks the healthiest deployment. The direct-access entries (like ollama/qwen3:32b) let you bypass the tier system and target a specific model when needed.

Replace IPs: Update 10.5.1.ROCINANTE with ROCINANTE's actual LAN or Tailscale IP. PD uses host.docker.internal since LiteLLM runs on PD alongside Ollama.

7.4 — docker-compose.yaml (updated ai stack)

Update /mnt/docker-ssd/docker/compose/ai/docker-compose.yaml to add LiteLLM:

name: ai

services:
  qdrant:
    image: qdrant/qdrant:v1.14.0
    container_name: qdrant
    restart: unless-stopped
    ports:
      - "6333:6333"   # REST API
      - "6334:6334"   # gRPC
    environment:
      TZ: ${TZ}
      QDRANT__SERVICE__GRPC_PORT: 6334
      QDRANT__STORAGE__STORAGE_PATH: /qdrant/storage
      QDRANT__STORAGE__SNAPSHOTS_PATH: /qdrant/snapshots
    volumes:
      - /mnt/docker-ssd/docker/appdata/qdrant/storage:/qdrant/storage
      - /mnt/docker-ssd/docker/appdata/qdrant/snapshots:/qdrant/snapshots
    networks:
      - ai-services
      - default

  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    restart: unless-stopped
    ports:
      - "4000:4000"
    environment:
      TZ: ${TZ}
      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
    volumes:
      - /mnt/docker-ssd/docker/compose/ai/litellm/config.yaml:/app/config.yaml:ro
      - /mnt/docker-ssd/docker/appdata/litellm:/app/data
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - ai-services
      - default

  reranker:
    image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
    container_name: reranker
    restart: unless-stopped
    ports:
      - "8787:80"
    environment:
      MODEL_ID: ${RERANKER_MODEL}
    volumes:
      - /mnt/docker-ssd/docker/appdata/reranker:/data
    networks:
      - ai-services
      - default

  whisper:
    image: fedirz/faster-whisper-server:latest-cuda
    container_name: whisper
    restart: unless-stopped
    ports:
      - "8786:8000"
    environment:
      TZ: ${TZ}
      WHISPER__MODEL: ${WHISPER_MODEL}
      WHISPER__DEVICE: cuda
      WHISPER__COMPUTE_TYPE: float16
    volumes:
      - /mnt/docker-ssd/docker/appdata/whisper:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    networks:
      - ai-services
      - default

  # -------------------------------------------------------
  # Ollama and OpenWebUI go here when deployed.
  # They share this stack and the default + ai-services networks.
  # OpenWebUI → Qdrant at http://qdrant:6333
  # OpenWebUI → LiteLLM at http://litellm:4000 (or direct Ollama)
  # -------------------------------------------------------

  # ollama:
  #   image: ollama/ollama:latest
  #   container_name: ollama
  #   ...

  # openwebui:
  #   image: ghcr.io/open-webui/open-webui:main
  #   container_name: openwebui
  #   ...

networks:
  ai-services:
    name: ai-services

GPU sharing note: On PD, the RTX 2080 Ti is shared between Plex (hardware transcoding), Immich ML, and Ollama. faster-whisper will also need the GPU when processing audio. These workloads are bursty (not constant), so time-sharing the GPU is viable — but be aware that a large Whisper transcription job will temporarily impact Ollama inference latency. If this becomes an issue, consider running Whisper on N.O.M.A.D.'s GTX 1080 instead and change whisper's Ollama-style routing accordingly.

Reranker on CPU: The reranker uses the cpu-latest image variant intentionally — do NOT pin to cpu-1.5 as it has an hf-hub compatibility bug. Reranking is a lightweight operation (scoring ~50 chunks takes <1s on CPU) and doesn't justify competing for GPU VRAM. The model loads via safetensors fallback since no ONNX files exist for bge-reranker-v2-m3. Max batch size is 4 on CPU. If you find latency is an issue, a GPU variant exists (ghcr.io/huggingface/text-embeddings-inference:turing-latest for your 2080 Ti).

7.5 — .env.example (updated ai stack)

# /mnt/docker-ssd/docker/compose/ai/.env.example

TZ=America/New_York

# LiteLLM
LITELLM_MASTER_KEY=sk-CHANGE_ME

# Reranker
RERANKER_MODEL=BAAI/bge-reranker-v2-m3

# Whisper
WHISPER_MODEL=Systran/faster-distil-whisper-large-v3

7.6 — Validate & Deploy

cd /mnt/docker-ssd/docker/compose/ai
cp .env.example .env
nano .env  # Set LITELLM_MASTER_KEY=$(openssl rand -hex 32)

docker compose --env-file .env config
docker compose --env-file .env up -d

7.7 — Post-Deploy Verification

# All containers running?
docker ps --filter name=qdrant --filter name=litellm --filter name=reranker --filter name=whisper

# LiteLLM health?
curl -s http://localhost:4000/health

# LiteLLM can see all models?
curl -s http://localhost:4000/v1/models \
  -H "Authorization: Bearer YOUR_MASTER_KEY" | python3 -m json.tool

# Test a chat completion through LiteLLM
curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer YOUR_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "medium",
    "messages": [{"role": "user", "content": "Hello, which model are you?"}]
  }' | python3 -m json.tool

# Test embeddings through LiteLLM
curl -s http://localhost:4000/v1/embeddings \
  -H "Authorization: Bearer YOUR_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "embed",
    "input": "test embedding"
  }' | python3 -m json.tool

7.8 — Integration Updates

With LiteLLM deployed, all apps should point at http://litellm:4000 (within the ai-services network) or http://PD_IP:4000 (from external hosts) instead of direct Ollama endpoints.

OpenWebUI: Settings → Connections → add OpenAI-compatible endpoint:

URL: http://litellm:4000/v1
API Key: your LITELLM_MASTER_KEY
This gives OpenWebUI access to all models across all three machines through one connection

Continue.dev: Update ~/.continue/config.json:

{
  "models": [
    {
      "title": "Heavy (ROCINANTE)",
      "provider": "openai",
      "model": "heavy",
      "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
      "apiKey": "YOUR_MASTER_KEY"
    },
    {
      "title": "Medium (PD)",
      "provider": "openai",
      "model": "medium",
      "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
      "apiKey": "YOUR_MASTER_KEY"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Light (N.O.M.A.D.)",
    "provider": "openai",
    "model": "light",
    "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
    "apiKey": "YOUR_MASTER_KEY"
  }
}

Key change: Continue.dev now uses provider: "openai" instead of provider: "ollama" and points at LiteLLM. One IP to remember, and if you add or move models between machines, you only update config.yaml — not every app.

n8n workflows: All HTTP Request nodes that call Ollama directly should be updated:

Old: http://10.5.1.16:11434/api/generate (N.O.M.A.D.)
New: http://litellm:4000/v1/chat/completions with model: "light"
n8n is on the ai-services network, so it reaches LiteLLM by Docker DNS

n8n Multi-Model Query Router workflow: This workflow is now simplified dramatically — instead of a Switch node routing to three different Ollama IPs, it becomes a single HTTP Request node that passes the model tier name (light, medium, heavy) to LiteLLM and lets the gateway handle routing. The Switch node logic can be removed entirely.

Phase 8 — Reranker (RAG Quality)

Host: PlausibleDeniability (deployed as part of the ai stack in Phase 7) Service: reranker container (already in the compose above) Why: Dramatically improves RAG answer quality by filtering out noisy retrieval results before they reach the LLM.

8.1 — How Reranking Works

Without a reranker, your RAG pipeline does:

Query → embed → Qdrant top-10 by vector similarity → all 10 chunks go to LLM

The problem: vector similarity often returns "close but irrelevant" chunks. The LLM gets noisy context and hallucinates.

With a reranker:

Query → embed → Qdrant top-50 by vector similarity → reranker scores each (query, chunk) pair → top-5 by relevance go to LLM

The reranker is a cross-encoder that reads the full query AND each chunk together, producing a much more accurate relevance score than vector distance alone. It over-retrieves cheaply from Qdrant, then precisely filters.

8.2 — Scaffold Directories

# Model cache on SSD (reranker model is ~1.1GB, downloaded on first start)
sudo mkdir -p /mnt/docker-ssd/docker/appdata/reranker

8.3 — Post-Deploy Verification

# Container running?
docker ps --filter name=reranker

# Health check?
curl -s http://localhost:8787/health

# Test reranking
curl -s http://localhost:8787/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "query": "How do I restart a Docker container?",
    "texts": [
      "Use docker restart <container_name> to restart a running container.",
      "Docker was founded in 2013 by Solomon Hykes.",
      "The docker compose down command stops and removes containers.",
      "Kubernetes pods can be restarted by deleting them."
    ]
  }' | python3 -m json.tool
# Expected: the first text scores highest, second scores lowest

8.4 — n8n RAG Pipeline Integration

Update the "Qdrant Index Updater" and any RAG query workflows to include a reranking step.

Updated RAG Query Workflow (for OpenWebUI or any n8n-based query):

Trigger:  Webhook node (POST with query)
Step 1:   HTTP Request → LiteLLM /v1/embeddings
            - model: "embed"
            - input: query text
Step 2:   HTTP Request → Qdrant API (POST /collections/{name}/points/search)
            - vector: embedding from step 1
            - limit: 50  (over-retrieve)
Step 3:   HTTP Request → Reranker (POST http://reranker:80/rerank)
            - query: original query text
            - texts: array of 50 chunk texts from Qdrant results
Step 4:   Code node → take top 5 by reranker score, format as context
Step 5:   HTTP Request → LiteLLM /v1/chat/completions
            - model: "medium" (or "heavy" for complex queries)
            - messages: system prompt with top-5 context + user query
Step 6:   Return response via webhook

Key difference from the original plan: Step 2 now retrieves 50 results instead of 10, and Step 3 (reranking) filters down to the best 5. This "over-retrieve then rerank" pattern is the standard approach for production RAG systems.

Phase 9 — faster-whisper (Speech-to-Text)

Host: PlausibleDeniability (deployed as part of the ai stack in Phase 7) Service: whisper container (already in the compose above) Why: Replaces shelved Scriberr with an OpenAI-compatible STT API. No SQLite dependency, no ZFS/ACL issues.

9.1 — Scaffold Directories

# Model cache on SSD (Whisper models are 1-3GB)
sudo mkdir -p /mnt/docker-ssd/docker/appdata/whisper

9.2 — Post-Deploy Verification

# Container running?
docker ps --filter name=whisper

# Health check?
curl -s http://localhost:8786/health

# Test transcription with a sample audio file
curl -s http://localhost:8786/v1/audio/transcriptions \
  -F "file=@/path/to/test-audio.wav" \
  -F "model=Systran/faster-distil-whisper-large-v3" \
  | python3 -m json.tool

9.3 — n8n Integration Workflows

Workflow: Voice Note → Text → Summary

Trigger:  Webhook node (POST with audio file in body)
Step 1:   HTTP Request → Whisper (POST http://whisper:8000/v1/audio/transcriptions)
            - Multipart form with audio file
            - response_format: "json"
Step 2:   HTTP Request → LiteLLM /v1/chat/completions
            - model: "medium"
            - Prompt: "Summarize this voice note concisely: {transcript}"
Step 3:   HTTP Request → Gotify → push summary to phone
Step 4:   (Optional) Postgres node → log transcript and summary

Workflow: Audio File → Paperless Document

Trigger:  Webhook or filesystem watcher
Step 1:   HTTP Request → Whisper → get transcript
Step 2:   Code node → format transcript as text document
Step 3:   HTTP Request → Paperless API (POST /api/documents/post_document/)
            - Upload transcript as .txt
            - Tag: "transcription"
Step 4:   Gotify notification: "Audio transcribed and filed: {title}"

Home Assistant Voice Integration: If you want HA voice commands, Whisper can serve as the STT backend:

In Home Assistant → Settings → Voice Assistants
Add speech-to-text provider: "Whisper" at http://PD_IP:8786
Pair with Piper TTS (future addition) for full voice assistant loop

9.4 — GPU vs CPU Considerations

The compose file above uses the GPU variant with CUDA. If GPU contention with Ollama becomes an issue:

Option A — CPU fallback on PD: Change the image from latest-cuda to fedirz/faster-whisper-server:latest-cpu and remove the deploy.resources block and the CUDA environment variables (WHISPER__DEVICE, WHISPER__COMPUTE_TYPE). Transcription will be slower (~4x real-time instead of ~20x) but won't compete for VRAM.

Option B — Move to N.O.M.A.D.: Run Whisper on N.O.M.A.D.'s GTX 1080 which is less contested. Add it to N.O.M.A.D.'s compose or run standalone. N.O.M.A.D.'s 8GB VRAM handles Whisper models comfortably since phi4 doesn't fill it.

Additional Tools Setup

Continue.dev (Local AI Code Completion)

Install the Continue extension in VS Code
Create/edit ~/.continue/config.json:

With LiteLLM (recommended — see Phase 7):

{
  "models": [
    {
      "title": "Heavy (ROCINANTE)",
      "provider": "openai",
      "model": "heavy",
      "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
      "apiKey": "YOUR_LITELLM_MASTER_KEY"
    },
    {
      "title": "Medium (PD)",
      "provider": "openai",
      "model": "medium",
      "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
      "apiKey": "YOUR_LITELLM_MASTER_KEY"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Light (N.O.M.A.D.)",
    "provider": "openai",
    "model": "light",
    "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
    "apiKey": "YOUR_LITELLM_MASTER_KEY"
  }
}

One IP, one API key, all three machines. If you add or move models, update LiteLLM's config.yaml — not every app.

Without LiteLLM (direct Ollama, if Phase 7 is not yet deployed):

{
  "models": [
    {
      "title": "PD - qwen2.5:14b",
      "provider": "ollama",
      "model": "qwen2.5:14b",
      "apiBase": "http://PD_TAILSCALE_IP:11434"
    },
    {
      "title": "ROCINANTE - qwen3:32b",
      "provider": "ollama",
      "model": "qwen3:32b",
      "apiBase": "http://ROCINANTE_TAILSCALE_IP:11434"
    }
  ],
  "tabAutocompleteModel": {
    "title": "N.O.M.A.D. - phi4",
    "provider": "ollama",
    "model": "phi4",
    "apiBase": "http://NOMAD_TAILSCALE_IP:11434"
  }
}

Obsidian + Gitea Sync

In Obsidian, install the "Obsidian Git" community plugin

Initialize your vault as a git repo:

cd /path/to/obsidian/vault
git init
git remote add origin http://PD_IP:3000/fizzlepoof/obsidian-vault.git

In Obsidian Git settings:
- Auto backup interval: 10 minutes
- Pull on startup: enabled
Create the repo in Gitea first: http://PD_IP:3000 → New Repository → obsidian-vault
Add a Gitea webhook to trigger the n8n "Qdrant Index Updater" workflow (see Phase 3)

Add to your Homepage configuration (/mnt/tank/docker/appdata/homepage/services.yaml):

- AI:
    - Ollama (PD):
        icon: ollama.svg
        href: http://PD_IP:11434
        widget:
          type: ollama
          url: http://PD_IP:11434
    - Ollama (ROCINANTE):
        icon: ollama.svg
        href: http://ROCINANTE_IP:11434
        widget:
          type: ollama
          url: http://ROCINANTE_IP:11434

Deployment Order Summary

Week 1:  Phase 1 — Gotify
           └─ Deploy, create app tokens, install phone app
           └─ Test: send manual notification via API

Week 1:  Phase 2 — Qdrant
           └─ Deploy, verify REST API
           └─ Create initial collections (empty, ready for n8n)

Week 2:  Phase 3 — n8n
           └─ Deploy, create admin account
           └─ Build workflows incrementally:
              Day 1: Gitea commit → Gotify (simplest, proves the pipeline)
              Day 2: Sonarr/Radarr → TMDB → Gotify + Discord
              Day 3: Tautulli play logging + weekly digest
              Day 4: Uptime Kuma enhanced alerts
              Day 5: ZFS pool monitoring
              Day 6: Multi-model query router
              Day 7: Qdrant index updater

Week 3:  Phase 4 — Paperless-NGX
           └─ Deploy, ingest test documents
           └─ Build n8n workflow: email → Paperless → Ollama summary
           └─ Set up document tags and correspondents for L&L Crafts

Week 4:  Phase 5 — Home Assistant
           └─ Deploy, onboard, add integrations
           └─ Connect to n8n via long-lived access token
           └─ Set up smart plug monitoring

Week 4:  Phase 6 — Grafana + Prometheus (on N.O.M.A.D.)
           └─ Deploy, verify scrape targets
           └─ Import dashboards
           └─ Set up Grafana → n8n alert webhook
           └─ Build alert rules

Week 5:  Phase 7/8/9 — AI Stack Expansion (all deploy together in ai stack)
           └─ LiteLLM: deploy, verify model routing across all 3 machines
           └─ Reranker: deploy, test scoring with sample chunks
           └─ faster-whisper: deploy, test transcription
           └─ Update OpenWebUI, Continue.dev, and n8n to use LiteLLM endpoint
           └─ Update RAG workflows to include reranking step
           └─ Build voice note → transcription → summary workflow

Post-Deployment Validation Master Checklist

Run through this after all six phases are deployed:

Infrastructure Health

docker ps on PD shows: gotify, n8n, qdrant, litellm, reranker, whisper, paperless, paperless-gotenberg, paperless-tika, homeassistant — all healthy
docker ps on N.O.M.A.D. shows: prometheus, grafana, node-exporter — all healthy
All services accessible via Pangolin subdomains (gotify, n8n, paperless, ha, grafana)

Network Connectivity

n8n can reach shared-postgres (test: check n8n logs for successful DB migration)
n8n can reach Gotify (test: trigger a workflow that sends a notification)
n8n can reach Qdrant at http://qdrant:6333 via ai-services network (test: query collections from n8n HTTP node)
n8n can reach LiteLLM at http://litellm:4000 via ai-services network (test: send a chat completion)
LiteLLM can route to all 3 Ollama instances (test: request model "light", "medium", "heavy" and verify each responds)
Reranker responds at http://reranker:80/rerank (test: POST sample texts)
Whisper responds at http://whisper:8000/v1/audio/transcriptions (test: transcribe a sample .wav)
Paperless can reach shared-postgres and shared-redis
Prometheus can scrape PD and Serenity Netdata endpoints
Grafana alert webhook reaches n8n

Data Flow End-to-End

Sonarr webhook → n8n → Gotify push on phone
Gitea commit → n8n → Qdrant indexing → Gotify notification
RAG query → Qdrant top-50 → reranker top-5 → LiteLLM → accurate answer
Audio file → Whisper transcription → n8n → Gotify summary
Document dropped in consume folder → Paperless OCR → n8n summary → Gotify
Grafana alert fires → n8n webhook → remediation action or Gotify alert
HA presence change → n8n → server wake/sleep

Backup Considerations

Gotify SQLite DB is on SSD — include /mnt/docker-ssd/docker/appdata/gotify in backup plan
Qdrant storage on SSD — include /mnt/docker-ssd/docker/appdata/qdrant in backup plan
n8n config/workflows — include /mnt/tank/docker/appdata/n8n in backup plan
Paperless media — include /mnt/tank/docker/appdata/paperless/media in backup plan
HA config — include /mnt/docker-ssd/docker/appdata/homeassistant in backup plan
LiteLLM DB + config — include /mnt/docker-ssd/docker/appdata/litellm and compose/ai/litellm/config.yaml in backup plan
Grafana/Prometheus data — include /mnt/hdd-2/grafana-data and /mnt/hdd-2/prometheus-data in N.O.M.A.D. backup plan
All new databases (n8n, paperless) are in shared-postgres — ensure pg_dump covers them

Storage Capacity Check

docker-ssd (500GB Crucial MX500) still has headroom after adding Gotify, Qdrant, HA, Paperless data
tank mirror has room for n8n config + Paperless media + consume/export
N.O.M.A.D. hdd-2 has room for Prometheus TSDB (90-day retention) + Grafana data

Remember: Everything self-hosted. Nothing touches the cloud.

67 KiB Raw Blame History