truenas-stacks/HOMELAB_BUILDOUT_PLAN.md

# Homelab Expansion — Full Buildout Plan

> Generated: 2026-05-04
> Based on: HOMELAB_EXPANSION_PLAN.md, ARCHITECTURE_OVERVIEW.md, STACK_STANDARDS.md

This document contains everything needed to deploy all six expansion phases — docker-compose files, .env templates, directory scaffolding commands, database init SQL, Pangolin configuration, and validation steps. Each phase follows the established stack standards and deployment checklist.

---

## Table of Contents

1. [Prerequisites & Conventions](#prerequisites--conventions)
2. [Phase 1 — Gotify (Notifications)](#phase-1--gotify-notifications)
3. [Phase 2 — Qdrant (Vector Database)](#phase-2--qdrant-vector-database)
4. [Phase 3 — n8n (Workflow Automation)](#phase-3--n8n-workflow-automation)
5. [Phase 4 — Paperless-NGX (Document Intelligence)](#phase-4--paperless-ngx-document-intelligence)
6. [Phase 5 — Home Assistant (Home Automation)](#phase-5--home-assistant-home-automation)
7. [Phase 6 — Grafana + Prometheus (Observability)](#phase-6--grafana--prometheus-observability)
8. [Phase 7 — LiteLLM (AI Gateway)](#phase-7--litellm-ai-gateway)
9. [Phase 8 — Reranker (RAG Quality)](#phase-8--reranker-rag-quality)
10. [Phase 9 — faster-whisper (Speech-to-Text)](#phase-9--faster-whisper-speech-to-text)
11. [Additional Tools Setup](#additional-tools-setup)
12. [Deployment Order Summary](#deployment-order-summary)
13. [Post-Deployment Validation Master Checklist](#post-deployment-validation-master-checklist)

---

## Prerequisites & Conventions

### Deployment Standards (recap)

All stacks on PlausibleDeniability follow the same pattern:

- **Repo root:** `/mnt/docker-ssd/docker/compose`
- **Validate before deploy:** `docker compose --env-file .env config`
- **Deploy:** `docker compose --env-file .env up -d`
- **Teardown:** `docker compose --env-file .env down`

### Storage Tiers

| Tier | Mount | Use For |
|------|-------|---------|
| SSD | `/mnt/docker-ssd/docker/appdata/<service>` | Write-heavy, SQLite, GPU/model, databases |
| Tank | `/mnt/tank/docker/appdata/<service>` | General appdata, configs, uploads |
| Unraid | `/mnt/unraid/data/media/` | Media libraries only |

### Networks

| Network | Created By | Purpose |
|---------|-----------|---------|
| `ix-databases_shared-databases` | databases stack | Access to shared-postgres, shared-mariadb, shared-redis |
| `pangolin` | newt (infrastructure stack) | Reverse proxy / external exposure |

### Secret Generation Commands

```bash
# Database passwords
openssl rand -hex 24

# JWT / encryption keys
openssl rand -hex 32

# Paperless secret key
python3 -c "import secrets; print(secrets.token_urlsafe(50))"
```

### Deployment Order

```
databases (already running) → infrastructure (already running)
  → Phase 1: Gotify
  → Phase 2: Qdrant  (creates the ai-services network)
  → Phase 3: n8n     (depends on Gotify + Qdrant — joins ai-services as external)
  → Phase 4: Paperless-NGX (depends on n8n for automation hooks)
  → Phase 5: Home Assistant (depends on n8n for heavy automation)
  → Phase 6: Grafana + Prometheus (on N.O.M.A.D., independent but benefits from all above)
```

> **Important:** The `ai` stack (Phase 2) must be deployed before the `automation` stack (Phase 3) because n8n declares `ai-services` as an external network. If the `ai` stack isn't up, `docker compose up` for `automation` will fail with a missing network error.

---

## Phase 1 — Gotify (Notifications)

**Host:** PlausibleDeniability
**Stack directory:** `/mnt/docker-ssd/docker/compose/automation/`
**Why first:** Every subsequent phase sends notifications through Gotify. It's the output bus.

### 1.1 — Scaffold Directories

```bash
# Appdata on SSD (SQLite backend — must not be on NFS)
sudo mkdir -p /mnt/docker-ssd/docker/appdata/gotify

# Stack directory (may already exist if automation/ is planned)
sudo mkdir -p /mnt/docker-ssd/docker/compose/automation
```

### 1.2 — Database Init

None required — Gotify uses an embedded SQLite database stored in its data volume.

### 1.3 — docker-compose.yaml

Add to `/mnt/docker-ssd/docker/compose/automation/docker-compose.yaml`:

```yaml
name: automation

services:
  gotify:
    image: gotify/server:2.6.1
    container_name: gotify
    restart: unless-stopped
    ports:
      - "8484:80"
    environment:
      TZ: ${TZ}
      GOTIFY_DEFAULTUSER_PASS: ${GOTIFY_ADMIN_PASS}
      GOTIFY_SERVER_PORT: 80
      GOTIFY_DATABASE_DIALECT: sqlite3
      GOTIFY_DATABASE_CONNECTION: data/gotify.db
    volumes:
      - /mnt/docker-ssd/docker/appdata/gotify:/app/data
    networks:
      - pangolin
      - default

networks:
  pangolin:
    external: true
```

> **Image note:** `gotify/server:2.6.1` is the latest stable as of May 2026. Check [Docker Hub](https://hub.docker.com/r/gotify/server/tags) before deploying — pin to the exact version.

### 1.4 — .env.example

```bash
# /mnt/docker-ssd/docker/compose/automation/.env.example

TZ=America/New_York
GOTIFY_ADMIN_PASS=CHANGE_ME
```

### 1.5 — .env (create from example)

```bash
cd /mnt/docker-ssd/docker/compose/automation
cp .env.example .env
# Edit .env with real values:
# GOTIFY_ADMIN_PASS=$(openssl rand -hex 16)
nano .env
```

### 1.6 — Pangolin Configuration

In your Pangolin dashboard, create a new resource:

| Field | Value |
|-------|-------|
| Domain | `gotify.paccoco.com` |
| Scheme | `http` |
| Host | `gotify` |
| Port | `80` |
| Network | Docker service name resolution via `pangolin` network |

### 1.7 — Validate & Deploy

```bash
cd /mnt/docker-ssd/docker/compose/automation
docker compose --env-file .env config
docker compose --env-file .env up -d
```

### 1.8 — Post-Deploy Verification

```bash
# Container running?
docker ps --filter name=gotify

# Clean startup logs?
docker logs gotify --tail 20

# Mounts correct?
docker inspect gotify --format '{{json .Mounts}}' | python3 -m json.tool

# Quick health check
curl -s http://localhost:8484/health

# Test notification via API
curl -s "http://localhost:8484/message?token=YOUR_APP_TOKEN" \
  -F "title=Homelab" \
  -F "message=Gotify is online" \
  -F "priority=5"
```

### 1.9 — First-Run Setup

1. Navigate to `https://gotify.paccoco.com`
2. Log in with the admin password from `.env`
3. **Change the default admin password** in the UI
4. Create application tokens for each notification source:
   - `n8n-workflows` — for all n8n automations
   - `infrastructure-alerts` — for Uptime Kuma, Grafana, etc.
   - `media-notifications` — for Sonarr/Radarr/Tautulli hooks
   - `home-assistant` — for HA automations
5. Create client tokens for each receiving device (phone, desktop)
6. Install the Gotify Android app and configure with your client token

---

## Phase 2 — AI Stack (Vector DB, LLM, Embeddings, Reranker, STT)

**Host:** PlausibleDeniability
**Stack directory:** `/mnt/docker-ssd/docker/compose/ai/`
**Status:** DEPLOYED AND VERIFIED (2026-05-05)

> **DEPLOYED NOTE:** This phase was deployed as a complete AI stack with 6 services: Ollama (11434), OpenWebUI (8282), Qdrant (6333/6334), LiteLLM (4000), Reranker/TEI (8787), and faster-whisper (8786). The compose below shows the original Qdrant-only plan — see the actual running compose on PD at `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml` and the `project_ai_stack_deployed.md` memory for port mappings and fixes. Key lesson: always use `10.5.1.6` not `localhost` for health checks on TrueNAS Scale.

### 2.1 — Scaffold Directories

```bash
# Appdata on SSD (write-heavy vector storage)
sudo mkdir -p /mnt/docker-ssd/docker/appdata/qdrant/storage
sudo mkdir -p /mnt/docker-ssd/docker/appdata/qdrant/snapshots

# Stack directory
sudo mkdir -p /mnt/docker-ssd/docker/compose/ai
```

### 2.2 — docker-compose.yaml

Create or update `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml`:

```yaml
name: ai

services:
  qdrant:
    image: qdrant/qdrant:v1.14.0
    container_name: qdrant
    restart: unless-stopped
    ports:
      - "6333:6333"   # REST API
      - "6334:6334"   # gRPC
    environment:
      TZ: ${TZ}
      QDRANT__SERVICE__GRPC_PORT: 6334
      QDRANT__STORAGE__STORAGE_PATH: /qdrant/storage
      QDRANT__STORAGE__SNAPSHOTS_PATH: /qdrant/snapshots
    volumes:
      - /mnt/docker-ssd/docker/appdata/qdrant/storage:/qdrant/storage
      - /mnt/docker-ssd/docker/appdata/qdrant/snapshots:/qdrant/snapshots
    networks:
      - ai-services
      - default

  # -------------------------------------------------------
  # Ollama and OpenWebUI go here when deployed.
  # They share this stack and the default + ai-services networks so
  # OpenWebUI can reach Qdrant at http://qdrant:6333
  # -------------------------------------------------------

  # ollama:
  #   image: ollama/ollama:latest
  #   container_name: ollama
  #   ...

  # openwebui:
  #   image: ghcr.io/open-webui/open-webui:main
  #   container_name: openwebui
  #   ...

networks:
  ai-services:
    name: ai-services
```

> **Image note:** `qdrant/qdrant:v1.14.0` is the latest stable as of May 2026. Check [GitHub releases](https://github.com/qdrant/qdrant/releases) before deploying.

> **Cross-stack connectivity:** The `ai-services` network is defined here with an explicit `name:` so other stacks (like `automation/n8n`) can declare it as `external: true` and reach Qdrant by service name (`http://qdrant:6333`). This follows the same pattern as `ix-databases_shared-databases`.

### 2.3 — .env.example

```bash
# /mnt/docker-ssd/docker/compose/ai/.env.example

TZ=America/New_York
```

### 2.4 — Validate & Deploy

```bash
cd /mnt/docker-ssd/docker/compose/ai
cp .env.example .env
nano .env  # Set timezone

docker compose --env-file .env config
docker compose --env-file .env up -d
```

### 2.5 — Post-Deploy Verification

```bash
# Container running?
docker ps --filter name=qdrant

# Clean startup?
docker logs qdrant --tail 20

# REST API responding?
curl -s http://localhost:6333/healthz
# Expected: {"title":"qdrant - vectorass engine","version":"1.14.0","commit":"..."}

# Create a test collection
curl -X PUT http://localhost:6333/collections/test_collection \
  -H "Content-Type: application/json" \
  -d '{
    "vectors": {
      "size": 384,
      "distance": "Cosine"
    }
  }'

# Verify it exists
curl -s http://localhost:6333/collections | python3 -m json.tool

# Clean up test
curl -X DELETE http://localhost:6333/collections/test_collection
```

### 2.6 — OpenWebUI Integration (when deployed)

When Ollama and OpenWebUI are brought online in this same stack, configure OpenWebUI's RAG settings:

1. Go to OpenWebUI → Admin → Settings → Documents
2. Set the vector database to Qdrant
3. Endpoint: `http://qdrant:6333` (Docker DNS within the `ai` stack network)
4. Collection name: `openwebui_docs` (or your preference)

### 2.7 — Collections to Create (for n8n in Phase 3)

These collections will be created programmatically by n8n workflows, but for reference:

| Collection | Vector Size | Content |
|-----------|-------------|---------|
| `homelab_docs` | 384 (nomic-embed-text) | Homelab markdown documentation |
| `gitea_commits` | 384 | Gitea commit messages + diffs |
| `media_metadata` | 384 | Plex/Tautulli metadata |
| `obsidian_notes` | 384 | Personal notes from Obsidian vault |

> **Vector size note:** 384 is the dimension for `nomic-embed-text` via Ollama. If you use a different embedding model, adjust accordingly.

---

## Phase 3 — n8n (Workflow Automation)

**Host:** PlausibleDeniability
**Stack directory:** `/mnt/docker-ssd/docker/compose/automation/` (same stack as Gotify)
**Why third:** n8n is the orchestration backbone — it ties Gotify, Qdrant, Ollama, and all triggers together.

### 3.1 — Scaffold Directories

```bash
# Config on tank (workflow definitions, credentials store)
# NOTE: n8n also writes execution logs here. If execution logging becomes
# heavy (many workflows running frequently), consider moving to SSD.
# For typical homelab usage (~20 workflows), tank is fine.
sudo mkdir -p /mnt/tank/docker/appdata/n8n

# Set ownership — n8n runs as UID 1000 (node user)
sudo chown 1000:1000 /mnt/tank/docker/appdata/n8n
```

### 3.2 — Database Init

n8n uses the existing shared-postgres. Create its database and user:

```bash
docker exec -i shared-postgres psql -U postgres <<'SQL'
CREATE USER n8n WITH PASSWORD 'REPLACE_WITH_GENERATED_PASSWORD';
CREATE DATABASE n8n OWNER n8n;
GRANT ALL PRIVILEGES ON DATABASE n8n TO n8n;
SQL
```

Generate the password first:
```bash
openssl rand -hex 24
```

### 3.3 — docker-compose.yaml

Update `/mnt/docker-ssd/docker/compose/automation/docker-compose.yaml` to add n8n alongside Gotify:

```yaml
name: automation

services:
  gotify:
    image: gotify/server:2.6.1
    container_name: gotify
    restart: unless-stopped
    ports:
      - "8484:80"
    environment:
      TZ: ${TZ}
      GOTIFY_DEFAULTUSER_PASS: ${GOTIFY_ADMIN_PASS}
      GOTIFY_SERVER_PORT: 80
      GOTIFY_DATABASE_DIALECT: sqlite3
      GOTIFY_DATABASE_CONNECTION: data/gotify.db
    volumes:
      - /mnt/docker-ssd/docker/appdata/gotify:/app/data
    networks:
      - pangolin
      - default

  n8n:
    image: docker.n8n.io/n8nio/n8n:1.88.0
    container_name: n8n
    restart: unless-stopped
    ports:
      - "5678:5678"
    environment:
      TZ: ${TZ}
      # Database
      DB_TYPE: postgresdb
      DB_POSTGRESDB_HOST: shared-postgres
      DB_POSTGRESDB_PORT: 5432
      DB_POSTGRESDB_DATABASE: ${N8N_DB_NAME}
      DB_POSTGRESDB_USER: ${N8N_DB_USER}
      DB_POSTGRESDB_PASSWORD: ${N8N_DB_PASS}
      # Security
      N8N_ENCRYPTION_KEY: ${N8N_ENCRYPTION_KEY}
      # Webhook / Reverse Proxy
      N8N_HOST: ${N8N_HOST}
      N8N_PROTOCOL: https
      N8N_PORT: 5678
      WEBHOOK_URL: https://${N8N_HOST}/
      N8N_PROXY_HOPS: 1
      # General
      GENERIC_TIMEZONE: ${TZ}
      N8N_DIAGNOSTICS_ENABLED: false
      N8N_PERSONALIZATION_ENABLED: false
    volumes:
      - /mnt/tank/docker/appdata/n8n:/home/node/.n8n
    networks:
      - pangolin
      - ix-databases_shared-databases
      - ai-services
      - default
    depends_on:
      - gotify

networks:
  pangolin:
    external: true
  ix-databases_shared-databases:
    external: true
  ai-services:
    external: true
```

> **Image note:** `docker.n8n.io/n8nio/n8n:1.88.0` — n8n uses calendar-ish versioning. Check [n8n releases](https://github.com/n8n-io/n8n/releases) for the latest stable. The `docker.n8n.io` registry is preferred for production.

> **Cross-stack access:** n8n joins `ai-services` (created by the `ai` stack) so it can reach Qdrant at `http://qdrant:6333` by Docker DNS. It also joins `ix-databases_shared-databases` for Postgres access — same pattern.

### 3.4 — .env.example (updated for both services)

```bash
# /mnt/docker-ssd/docker/compose/automation/.env.example

TZ=America/New_York

# Gotify
GOTIFY_ADMIN_PASS=CHANGE_ME

# n8n — Database
N8N_DB_NAME=n8n
N8N_DB_USER=n8n
N8N_DB_PASS=CHANGE_ME

# n8n — Security
N8N_ENCRYPTION_KEY=CHANGE_ME

# n8n — Hostname
N8N_HOST=n8n.paccoco.com
```

### 3.5 — Pangolin Configuration

| Field | Value |
|-------|-------|
| Domain | `n8n.paccoco.com` |
| Scheme | `http` |
| Host | `n8n` |
| Port | `5678` |
| Headers | Forward `X-Forwarded-Proto: https` |

### 3.6 — Validate & Deploy

```bash
cd /mnt/docker-ssd/docker/compose/automation

# Update .env with real passwords
nano .env

docker compose --env-file .env config
docker compose --env-file .env up -d
```

### 3.7 — Post-Deploy Verification

```bash
# Both containers running?
docker ps --filter name=gotify --filter name=n8n

# n8n logs clean?
docker logs n8n --tail 30

# n8n on correct networks?
docker inspect n8n --format '{{json .NetworkSettings.Networks}}' | python3 -m json.tool

# n8n UI accessible?
curl -s -o /dev/null -w "%{http_code}" http://localhost:5678/
# Expected: 200

# Postgres connection working? (check logs for DB migration messages)
docker logs n8n 2>&1 | grep -i "migrat"
```

### 3.8 — First-Run Setup

1. Navigate to `https://n8n.paccoco.com`
2. Create your admin account
3. Install community nodes you'll need:
   - `n8n-nodes-gotify` (if available) or use HTTP Request node
   - Ollama nodes (built-in as of n8n 1.x)

### 3.9 — Workflow Blueprints

Below are starter workflow descriptions for each planned automation. These are meant to be built in the n8n UI — the structure and node types are described so you can wire them up.

#### Media Pipeline Workflows

**Workflow: Sonarr/Radarr Download Notification**
```
Trigger:  Webhook node (POST from Sonarr/Radarr on download/import)
Step 1:   Extract series/movie name, quality, file path from webhook body
Step 2:   HTTP Request → TMDB API to fetch poster image URL
Step 3:   HTTP Request → Gotify REST API (POST /message)
            - Title: "New Download: {title}"
            - Message: "{quality} — {episodeTitle or year}"
            - Priority: 5
            - Extras: attach poster URL as markdown image
Also:     HTTP Request → Discord webhook (formatted embed with poster)
```

Configure Sonarr/Radarr webhooks:
- Sonarr: Settings → Connect → Webhook → URL: `https://n8n.paccoco.com/webhook/sonarr`
- Radarr: Settings → Connect → Webhook → URL: `https://n8n.paccoco.com/webhook/radarr`

**Workflow: Tautulli Play Logging**
```
Trigger:  Webhook node (POST from Tautulli on playback start)
Step 1:   Extract user, media title, player, quality from payload
Step 2:   Postgres node → INSERT into watch_history table
Step 3:   (Optional) Gotify notification for specific users/media
```

Tautulli config: Settings → Notification Agents → Webhook → URL: `https://n8n.paccoco.com/webhook/tautulli-play`

**Workflow: Weekly Watch Digest**
```
Trigger:  Cron node (every Sunday at 10:00 AM)
Step 1:   Postgres node → SELECT watch history for past 7 days
Step 2:   Format data as structured text
Step 3:   HTTP Request → Ollama API (POST to PD's qwen2.5:14b)
            - Prompt: "Summarize this week's viewing in a fun digest: {data}"
Step 4:   HTTP Request → Gotify (send digest)
```

#### Infrastructure Monitoring Workflows

**Workflow: Uptime Kuma Enhanced Alerts**
```
Trigger:  Webhook node (from Uptime Kuma notification)
Step 1:   Extract monitor name, status, response time
Step 2:   HTTP Request → Netdata API for related metrics context
Step 3:   HTTP Request → Gotify
            - Title: "🔴 {monitor} DOWN" or "🟢 {monitor} UP"
            - Message: include Netdata context (CPU, mem, disk at time of alert)
            - Priority: 8 (high for down, 3 for recovery)
```

**Workflow: ZFS Pool Utilization Alert**
```
Trigger:  Cron node (every 6 hours)
Step 1:   SSH node → Serenity: `zpool list -Hp malcolm`
Step 2:   Parse capacity percentage
Step 3:   IF capacity > 85% → Gotify alert (priority 8)
Step 4:   IF capacity > 90% → Gotify alert (priority 10) + Discord webhook
```

**Workflow: Grafana Alert Remediation**
```
Trigger:  Webhook node (from Grafana alerting)
Step 1:   Parse alert labels (container, host, metric)
Step 2:   Switch node → route by alert type:
            - High CPU container → SSH → docker restart {container}
            - Disk full → SSH → pause qBittorrent, notify via Gotify
            - Memory pressure → Gotify alert only (manual intervention)
Step 3:   Log action taken to Postgres
```

#### Homelab Ops Workflows

**Workflow: Gitea Commit Summary**
```
Trigger:  Webhook node (Gitea webhook on push to truenas-stacks)
Step 1:   Extract commit messages, author, files changed
Step 2:   HTTP Request → Ollama API
            - Prompt: "Summarize this commit in one sentence: {commit_message}"
Step 3:   HTTP Request → Gotify
            - Title: "Commit to truenas-stacks"
            - Message: Ollama-generated summary
```

Gitea config: Repository → Settings → Webhooks → URL: `https://n8n.paccoco.com/webhook/gitea-push`

**Workflow: qBittorrent Auto-Rescan**
```
Trigger:  Webhook or polling (qBittorrent API for completed+moved torrents)
Step 1:   Determine if file is in Sonarr or Radarr path
Step 2:   HTTP Request → Sonarr API (POST /command → RescanSeries)
           OR HTTP Request → Radarr API (POST /command → RescanMovie)
Step 3:   Gotify notification confirming rescan triggered
```

#### AI Pipeline Workflows

**Workflow: URL Digest Pipeline**
```
Trigger:  Webhook node (POST with URL in body)
Step 1:   HTTP Request → fetch page content
Step 2:   Code node → extract text, chunk into ~500 token segments
Step 3:   HTTP Request → Ollama API → summarize each chunk
Step 4:   Code node → combine summaries into digest
Step 5:   Postgres node → store digest with metadata
Step 6:   Return digest in webhook response
```

**Workflow: Multi-Model Query Router (simplified by LiteLLM — see Phase 7)**
```
Trigger:  Webhook node (POST with query + complexity hint)
Step 1:   HTTP Request → LiteLLM (POST http://litellm:4000/v1/chat/completions)
            - model: complexity parameter ("light", "medium", or "heavy")
            - LiteLLM handles routing to the correct Ollama instance
Step 2:   Return response via webhook
```
> With LiteLLM deployed, the Switch node and three separate Ollama endpoints
> are replaced by a single HTTP Request node. The routing logic lives in
> LiteLLM's config.yaml instead of n8n workflow logic.

**Workflow: Qdrant Index Updater**
```
Trigger:  Webhook node (from Gitea push to any watched repo)
Step 1:   HTTP Request → Gitea API → fetch changed files content
Step 2:   Code node → chunk text into embedding-sized segments
Step 3:   HTTP Request → Ollama API (embed endpoint with nomic-embed-text)
Step 4:   HTTP Request → Qdrant API (PUT /collections/homelab_docs/points)
Step 5:   Gotify notification: "{n} documents re-indexed"
```

#### Home / Business Workflows

**Workflow: KitchenOwl Grocery Notification**
```
Trigger:  Polling (KitchenOwl API) or webhook if supported
Step 1:   Fetch current shopping list items
Step 2:   Format as clean text list
Step 3:   HTTP Request → Gotify → phone push notification
```

**Workflow: Donetick Task Reminder**
```
Trigger:  Cron node (daily at 9:00 AM)
Step 1:   HTTP Request → Donetick API → fetch tasks due today/overdue
Step 2:   Format task list
Step 3:   HTTP Request → Gotify (priority 5)
```

**Workflow: Long and Low Crafts Order Pipeline**
```
Trigger:  Webhook (Etsy webhook or email trigger via IMAP node)
Step 1:   Parse order details (item, quantity, customer, shipping)
Step 2:   HTTP Request → Donetick API → create fulfillment task
Step 3:   HTTP Request → Gotify DM notification
            - Title: "New L&L Order"
            - Message: "{item} x{qty} — ship by {date}"
            - Priority: 7
```

---

## Phase 4 — Paperless-NGX (Document Intelligence)

**Host:** PlausibleDeniability
**Stack directory:** `/mnt/docker-ssd/docker/compose/documents/` (new stack)
**Why fourth:** Depends on n8n for automation hooks and Gotify for notifications.

### 4.1 — Scaffold Directories

```bash
# Index/data on SSD (search index is write-heavy)
sudo mkdir -p /mnt/docker-ssd/docker/appdata/paperless/data

# Documents (media) on tank (bulk storage, read-heavy)
sudo mkdir -p /mnt/tank/docker/appdata/paperless/media

# Consume folder on tank (drop zone for new documents)
sudo mkdir -p /mnt/tank/docker/appdata/paperless/consume

# Export folder on tank
sudo mkdir -p /mnt/tank/docker/appdata/paperless/export

# Stack directory
sudo mkdir -p /mnt/docker-ssd/docker/compose/documents
```

### 4.2 — Database Init

Paperless uses the existing shared-postgres and shared-redis:

```bash
docker exec -i shared-postgres psql -U postgres <<'SQL'
CREATE USER paperless WITH PASSWORD 'REPLACE_WITH_GENERATED_PASSWORD';
CREATE DATABASE paperless OWNER paperless;
GRANT ALL PRIVILEGES ON DATABASE paperless TO paperless;
SQL
```

### 4.3 — docker-compose.yaml

Create `/mnt/docker-ssd/docker/compose/documents/docker-compose.yaml`:

```yaml
name: documents

services:
  paperless:
    image: ghcr.io/paperless-ngx/paperless-ngx:2.16
    container_name: paperless
    restart: unless-stopped
    ports:
      - "8000:8000"
    environment:
      TZ: ${TZ}
      # Database
      PAPERLESS_DBENGINE: postgresql
      PAPERLESS_DBHOST: shared-postgres
      PAPERLESS_DBPORT: 5432
      PAPERLESS_DBNAME: ${PAPERLESS_DB_NAME}
      PAPERLESS_DBUSER: ${PAPERLESS_DB_USER}
      PAPERLESS_DBPASS: ${PAPERLESS_DB_PASS}
      # Redis (using shared-redis)
      PAPERLESS_REDIS: redis://shared-redis:6379
      # Security
      PAPERLESS_SECRET_KEY: ${PAPERLESS_SECRET_KEY}
      PAPERLESS_URL: https://${PAPERLESS_HOST}
      PAPERLESS_ADMIN_USER: ${PAPERLESS_ADMIN_USER}
      PAPERLESS_ADMIN_PASSWORD: ${PAPERLESS_ADMIN_PASS}
      # OCR
      PAPERLESS_OCR_LANGUAGE: eng
      PAPERLESS_OCR_MODE: skip
      # Tika/Gotenberg for Office documents
      PAPERLESS_TIKA_ENABLED: 1
      PAPERLESS_TIKA_ENDPOINT: http://tika:9998
      PAPERLESS_TIKA_GOTENBERG_ENDPOINT: http://gotenberg:3000
      # Performance
      PAPERLESS_TASK_WORKERS: 2
      PAPERLESS_THREADS_PER_WORKER: 2
    volumes:
      - /mnt/docker-ssd/docker/appdata/paperless/data:/usr/src/paperless/data
      - /mnt/tank/docker/appdata/paperless/media:/usr/src/paperless/media
      - /mnt/tank/docker/appdata/paperless/consume:/usr/src/paperless/consume
      - /mnt/tank/docker/appdata/paperless/export:/usr/src/paperless/export
    networks:
      - pangolin
      - ix-databases_shared-databases
      - default
    depends_on:
      - gotenberg
      - tika

  gotenberg:
    image: gotenberg/gotenberg:8.17
    container_name: paperless-gotenberg
    restart: unless-stopped
    command:
      - "gotenberg"
      - "--chromium-disable-javascript=true"
      - "--chromium-allow-list=file:///tmp/.*"
    networks:
      - default

  tika:
    image: apache/tika:3.1
    container_name: paperless-tika
    restart: unless-stopped
    networks:
      - default

networks:
  pangolin:
    external: true
  ix-databases_shared-databases:
    external: true
```

> **Image notes:**
> - `ghcr.io/paperless-ngx/paperless-ngx:2.16` — check [releases](https://github.com/paperless-ngx/paperless-ngx/releases) for latest.
> - `gotenberg/gotenberg:8.17` — pin to a specific 8.x tag.
> - `apache/tika:3.1` — needed for Office doc support (.docx, .xlsx, .odt).
> - Gotenberg and Tika are only needed if you ingest Office documents. For PDF-only, you can omit them and set `PAPERLESS_TIKA_ENABLED: 0`.

### 4.4 — .env.example

```bash
# /mnt/docker-ssd/docker/compose/documents/.env.example

TZ=America/New_York

# Paperless — Database
PAPERLESS_DB_NAME=paperless
PAPERLESS_DB_USER=paperless
PAPERLESS_DB_PASS=CHANGE_ME

# Paperless — Security
PAPERLESS_SECRET_KEY=CHANGE_ME
PAPERLESS_ADMIN_USER=admin
PAPERLESS_ADMIN_PASS=CHANGE_ME

# Paperless — Hostname
PAPERLESS_HOST=paperless.paccoco.com
```

### 4.5 — Pangolin Configuration

| Field | Value |
|-------|-------|
| Domain | `paperless.paccoco.com` |
| Scheme | `http` |
| Host | `paperless` |
| Port | `8000` |

### 4.6 — Validate & Deploy

```bash
cd /mnt/docker-ssd/docker/compose/documents
cp .env.example .env
nano .env  # Fill in real values

docker compose --env-file .env config
docker compose --env-file .env up -d
```

### 4.7 — Post-Deploy Verification

```bash
# All three containers running?
docker ps --filter name=paperless --filter name=paperless-gotenberg --filter name=paperless-tika

# Paperless logs clean? (watch for DB migration output)
docker logs paperless --tail 30

# Network connectivity to shared-postgres and shared-redis?
docker exec paperless python3 -c "import psycopg2; print('postgres OK')" 2>/dev/null || echo "Check DB connection"

# UI accessible?
curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/
# Expected: 200 or 302 (redirect to login)
```

### 4.8 — n8n Integration Workflows

**Workflow: Email Attachment → Paperless**
```
Trigger:  IMAP node (poll email inbox every 5 minutes)
Step 1:   Filter for emails with PDF/document attachments
Step 2:   Download attachment binary
Step 3:   HTTP Request → Paperless API (POST /api/documents/post_document/)
            - Multipart form with document file
            - Optional: set correspondent, document type, tags
Step 4:   Wait node (30 seconds for OCR processing)
Step 5:   HTTP Request → Paperless API (GET document details)
Step 6:   IF tagged as "action-required" →
            HTTP Request → Donetick API → create task
Step 7:   HTTP Request → Gotify → "New document ingested: {title}"
```

**Workflow: Ollama Document Summarizer**
```
Trigger:  Paperless webhook (on document consumed) or n8n polling
Step 1:   HTTP Request → Paperless API → fetch document text
Step 2:   HTTP Request → Ollama API (PD's qwen2.5:14b)
            - Prompt: "Extract key data from this document: dates, amounts, parties, action items. {text}"
Step 3:   HTTP Request → Paperless API → update document notes with summary
Step 4:   HTTP Request → Gotify → "Document summarized: {title}"
```

---

## Phase 5 — Home Assistant (Home Automation)

**Host:** PlausibleDeniability (preferred — same network as most services)
**Stack directory:** `/mnt/docker-ssd/docker/compose/homeassistant/` (standalone stack — host networking isolates it)
**Why fifth:** Benefits from n8n being operational. n8n handles complex automation logic while HA handles device control.

### 5.1 — Scaffold Directories

```bash
# Config on SSD (SQLite database, write-heavy)
sudo mkdir -p /mnt/docker-ssd/docker/appdata/homeassistant
```

### 5.2 — docker-compose.yaml

Create `/mnt/docker-ssd/docker/compose/homeassistant/docker-compose.yaml`:

```yaml
name: homeassistant

services:
  homeassistant:
    image: ghcr.io/home-assistant/home-assistant:2026.5
    container_name: homeassistant
    restart: unless-stopped
    privileged: true
    network_mode: host
    environment:
      TZ: ${TZ}
    volumes:
      - /mnt/docker-ssd/docker/appdata/homeassistant:/config
      - /etc/localtime:/etc/localtime:ro
      - /run/dbus:/run/dbus:ro
    # devices:
    #   - /dev/ttyUSB0:/dev/ttyUSB0   # Uncomment for USB Zigbee/Z-Wave sticks
```

> **Image note:** `ghcr.io/home-assistant/home-assistant:2026.5` — use the latest stable monthly release. Check [HA releases](https://www.home-assistant.io/blog/categories/release-notes/). Pin to a specific minor like `2026.5.1` once you confirm it's stable.

> **Why host networking:** Home Assistant requires `network_mode: host` for mDNS/SSDP device discovery. This means it does NOT join `pangolin` — you'll access it directly by IP and port, or configure Pangolin to proxy to `http://10.5.1.X:8123`.

> **Why standalone stack:** Host networking is incompatible with other services in the same compose that use bridge networking. HA must be in its own compose file.

### 5.3 — .env.example

```bash
# /mnt/docker-ssd/docker/compose/homeassistant/.env.example

TZ=America/New_York
```

### 5.4 — Pangolin Configuration

Since HA uses host networking, Pangolin needs to reach it by the host's IP:

| Field | Value |
|-------|-------|
| Domain | `ha.paccoco.com` |
| Scheme | `http` |
| Host | `10.5.1.X` (PlausibleDeniability's LAN IP) |
| Port | `8123` |

> Find PD's IP with: `hostname -I | awk '{print $1}'`

### 5.5 — Validate & Deploy

```bash
cd /mnt/docker-ssd/docker/compose/homeassistant
cp .env.example .env
nano .env

docker compose --env-file .env config
docker compose --env-file .env up -d
```

### 5.6 — Post-Deploy Verification

```bash
# Container running?
docker ps --filter name=homeassistant

# Clean startup?
docker logs homeassistant --tail 30

# Web UI accessible?
curl -s -o /dev/null -w "%{http_code}" http://localhost:8123/
# Expected: 200
```

### 5.7 — First-Run Setup & Integrations

1. Navigate to `http://PD_IP:8123` (or `https://ha.paccoco.com`)
2. Complete the onboarding wizard
3. Add integrations:

**Smart Plug Power Monitoring:**
- Install TP-Link Kasa / Tapo / Shelly integration (depends on your plug brand)
- Add plugs for Serenity, PD, N.O.M.A.D.
- Create energy dashboard for power consumption tracking

**UPS Monitoring:**
- Install NUT (Network UPS Tools) integration if you add a UPS
- Note: PD and N.O.M.A.D. currently have no UPS — this is a known gap
- When UPS is added, create automations for graceful shutdown via n8n

**Presence Detection:**
- Mobile app integration for phone-based presence
- Or use router/UniFi integration for network-based detection

### 5.8 — n8n ↔ Home Assistant Integration

In n8n, use the Home Assistant nodes (built-in):

1. In HA: Profile → Long-Lived Access Tokens → Create Token
2. In n8n: Settings → Credentials → Home Assistant API
   - Host: `http://10.5.1.X:8123`
   - Access Token: (paste from step 1)

**Workflow: Presence → Server Sleep/Wake**
```
Trigger:  HA event node (person.fizzlepoof state change)
Step 1:   IF state = "not_home" for > 30 min AND no active Plex streams:
            SSH → N.O.M.A.D.: systemctl suspend
Step 2:   IF state = "home":
            Wake-on-LAN → N.O.M.A.D. MAC address
Step 3:   Gotify notification: "Server {action}: N.O.M.A.D."
```

**Workflow: Physical Device → n8n Trigger**
```
Trigger:  HA webhook or event node (smart plug power draw spike)
Step 1:   IF washing machine plug power < 5W for 5 min after being > 100W:
            Gotify notification: "Laundry is done!"
```

---

## Phase 6 — Grafana + Prometheus (Observability)

**Host:** N.O.M.A.D. (Ubuntu 25.10 — dedicated HDDs have headroom)
**Stack directory:** `/opt/monitoring/` (outside the N.O.M.A.D. project directory)
**Why last:** Observability benefits from all other services being online — more to monitor.

### 6.1 — Scaffold Directories

SSH into N.O.M.A.D.:

```bash
ssh nomad@10.5.1.16

# Stack directory
sudo mkdir -p /opt/monitoring

# Prometheus data on hdd-2 (has more headroom)
sudo mkdir -p /mnt/hdd-2/prometheus-data
sudo chown 65534:65534 /mnt/hdd-2/prometheus-data  # nobody user (Prometheus default)

# Grafana data on hdd-2
sudo mkdir -p /mnt/hdd-2/grafana-data
sudo chown 472:472 /mnt/hdd-2/grafana-data  # grafana user

# Config directories
sudo mkdir -p /opt/monitoring/provisioning/datasources
sudo mkdir -p /opt/monitoring/provisioning/dashboards
```

### 6.2 — prometheus.yml

Create `/opt/monitoring/prometheus.yml`:

```yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # ------- Local (N.O.M.A.D.) -------
  - job_name: 'nomad-node'
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          host: 'nomad'

  # ------- PlausibleDeniability -------
  - job_name: 'pd-netdata'
    metrics_path: /api/v1/allmetrics
    params:
      format: [prometheus]
    static_configs:
      - targets: ['10.5.1.X:19999']    # Replace with PD's IP
        labels:
          host: 'plausible-deniability'

  # ------- Serenity -------
  - job_name: 'serenity-netdata'
    metrics_path: /api/v1/allmetrics
    params:
      format: [prometheus]
    static_configs:
      - targets: ['10.5.1.5:19999']
        labels:
          host: 'serenity'

  # ------- Gotify -------
  # Gotify exposes /health but no Prometheus endpoint natively.
  # Use blackbox exporter or just rely on Uptime Kuma.

  # ------- n8n -------
  # n8n doesn't expose Prometheus metrics by default.
  # Monitor via container resource metrics from Netdata.

  # -------- Add more targets as needed --------
  # When node-exporter is installed on PD and Serenity:
  # - job_name: 'pd-node'
  #   static_configs:
  #     - targets: ['10.5.1.X:9100']
  #       labels:
  #         host: 'plausible-deniability'
  #
  # - job_name: 'serenity-node'
  #   static_configs:
  #     - targets: ['10.5.1.5:9100']
  #       labels:
  #         host: 'serenity'
```

> **Netdata as a Prometheus target:** Both PD and Serenity already run Netdata. Netdata has a built-in Prometheus exporter at `/api/v1/allmetrics?format=prometheus`. This gives you CPU, memory, disk, network, and ZFS metrics without installing node-exporter on those hosts.

### 6.3 — Grafana Provisioning

Create `/opt/monitoring/provisioning/datasources/prometheus.yml`:

```yaml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true
```

Create `/opt/monitoring/provisioning/dashboards/dashboards.yml`:

```yaml
apiVersion: 1

providers:
  - name: 'default'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    editable: true
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: false
```

### 6.4 — docker-compose.yaml

Create `/opt/monitoring/docker-compose.yaml`:

```yaml
name: monitoring

services:
  prometheus:
    image: prom/prometheus:v3.4.0
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=90d'
      - '--web.enable-lifecycle'
    volumes:
      - /opt/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - /mnt/hdd-2/prometheus-data:/prometheus
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:13.0.1
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      TZ: ${TZ}
      GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER}
      GF_SECURITY_ADMIN_PASSWORD: ${GF_ADMIN_PASS}
      GF_SERVER_ROOT_URL: https://${GF_HOST}/
    volumes:
      - /mnt/hdd-2/grafana-data:/var/lib/grafana
      - /opt/monitoring/provisioning:/etc/grafana/provisioning:ro
    networks:
      - monitoring

  node-exporter:
    image: prom/node-exporter:v1.9.0
    container_name: node-exporter
    restart: unless-stopped
    ports:
      - "9100:9100"
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge
```

> **Image notes:**
> - `prom/prometheus:v3.4.0` — Prometheus v3 is the current major line. **Do not use `:latest`** — it still resolves to 2.x due to a known tagging issue. Always use an explicit `v3.x.y` tag.
> - `grafana/grafana:13.0.1` — Grafana 13 is the current stable.
> - `prom/node-exporter:v1.9.0` — check [releases](https://github.com/prometheus/node_exporter/releases).

### 6.5 — .env.example

```bash
# /opt/monitoring/.env.example

TZ=America/New_York

# Grafana
GF_ADMIN_USER=admin
GF_ADMIN_PASS=CHANGE_ME
GF_HOST=grafana.paccoco.com
```

### 6.6 — Pangolin Configuration

Grafana runs on N.O.M.A.D., not PD where the main Newt agent lives. You have two options:

**Option A — Route via N.O.M.A.D.'s existing Newt (recommended if already connected)**

N.O.M.A.D. already has a Newt container from the Project N.O.M.A.D. setup. If it's connected to your Pangolin VPS, just add a new resource in the Pangolin dashboard pointing to `http://grafana:3000` or `http://10.5.1.16:3000`.

**Option B — Add a dedicated Newt to the monitoring stack**

If N.O.M.A.D.'s existing Newt is not connected to Pangolin (or is a separate Pangolin instance), add Newt to the monitoring compose:

```yaml
  newt:
    image: ghcr.io/fosrl/newt:latest
    container_name: monitoring-newt
    restart: unless-stopped
    environment:
      PANGOLIN_ENDPOINT: ${PANGOLIN_ENDPOINT}
      NEWT_ID: ${NEWT_ID}
      NEWT_SECRET: ${NEWT_SECRET}
    networks:
      - monitoring
```

Then in Pangolin dashboard:

| Field | Value |
|-------|-------|
| Domain | `grafana.paccoco.com` |
| Scheme | `http` |
| Host | `grafana` (Docker DNS via shared network) |
| Port | `3000` |

**Option C — Direct IP routing (simplest, no Newt needed)**

If Pangolin's Newt on PD can reach N.O.M.A.D. by LAN IP (they're on the same subnet):

| Field | Value |
|-------|-------|
| Domain | `grafana.paccoco.com` |
| Scheme | `http` |
| Host | `10.5.1.16` |
| Port | `3000` |

> **Decision point:** Check if N.O.M.A.D.'s existing Newt is connected to your Pangolin instance before deploying. Run `docker ps --filter name=newt` on N.O.M.A.D. to verify. If it's running and connected, Option A is zero-effort. If not, Option C is the simplest fallback.

### 6.7 — Validate & Deploy

```bash
ssh nomad@10.5.1.16

cd /opt/monitoring
cp .env.example .env
nano .env  # Fill in real values

docker compose --env-file .env config
docker compose --env-file .env up -d
```

### 6.8 — Post-Deploy Verification

```bash
# All three containers running?
docker ps --filter name=prometheus --filter name=grafana --filter name=node-exporter

# Prometheus scraping targets?
curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | head -40

# Grafana UI accessible?
curl -s -o /dev/null -w "%{http_code}" http://localhost:3000/
# Expected: 200 or 302

# Node exporter metrics flowing?
curl -s http://localhost:9100/metrics | head -10
```

### 6.9 — Recommended Dashboards

Import these from [Grafana Dashboard Library](https://grafana.com/grafana/dashboards/):

| Dashboard | ID | Purpose |
|-----------|----|---------|
| Node Exporter Full | 1860 | System metrics for N.O.M.A.D. |
| Docker Container Stats | 893 | Container resource usage |
| Netdata via Prometheus | (search) | PD and Serenity system metrics |

To import: Grafana → Dashboards → New → Import → Enter dashboard ID.

### 6.10 — Metrics Targets Summary

| Target | Host | Method | Endpoint |
|--------|------|--------|----------|
| N.O.M.A.D. system | localhost | node-exporter | node-exporter:9100 |
| PD system | 10.5.1.X | Netdata Prometheus | 10.5.1.X:19999/api/v1/allmetrics |
| Serenity system | 10.5.1.5 | Netdata Prometheus | 10.5.1.5:19999/api/v1/allmetrics |
| ZFS pools | via Netdata | Netdata exports ZFS metrics | Included in Netdata scrape |
| Container stats | via Netdata | Netdata exports cgroup metrics | Included in Netdata scrape |
| Plex streams | Tautulli | n8n polling → Postgres | Via n8n workflow (Phase 3) |
| qBit stats | qBittorrent API | n8n polling → Postgres | Via n8n workflow (Phase 3) |
| Tailscale latency | Tailscale API | n8n polling → Postgres | Via n8n workflow (Phase 3) |

### 6.11 — n8n Integration (Grafana → n8n alert webhook)

In Grafana → Alerting → Contact Points, create a webhook contact point:

| Field | Value |
|-------|-------|
| Name | `n8n-alerts` |
| Type | Webhook |
| URL | `https://n8n.paccoco.com/webhook/grafana-alert` |
| HTTP Method | POST |

Then create alert rules for:
- ZFS pool utilization > 85%
- Container memory > 90% of limit
- Host CPU sustained > 90% for 5 minutes
- Disk I/O latency spikes

These fire into the "Grafana Alert Remediation" n8n workflow (see Phase 3).

---

## Phase 7 — LiteLLM (AI Gateway)

**Host:** PlausibleDeniability
**Stack directory:** `/mnt/docker-ssd/docker/compose/ai/` (same stack as Qdrant)
**Why:** Replaces manual multi-model routing with a unified OpenAI-compatible API. Every app (OpenWebUI, Continue.dev, n8n, Paperless summarization) points at one endpoint instead of juggling three Ollama IPs.

### 7.1 — What LiteLLM Does

LiteLLM sits in front of your three Ollama instances and presents a single OpenAI-compatible API at `http://litellm:4000`. It handles:

- **Model routing** — requests for `qwen3:32b` go to ROCINANTE, `qwen2.5:14b` goes to PD, `phi4` goes to N.O.M.A.D.
- **Failover** — if ROCINANTE is offline, LiteLLM can fall back to PD automatically
- **Load balancing** — distribute requests across instances running the same model
- **Usage tracking** — logs token counts, latency, and costs per model/user via its built-in database

### 7.2 — Scaffold Directories

```bash
# Config on SSD (SQLite DB for usage tracking)
sudo mkdir -p /mnt/docker-ssd/docker/appdata/litellm

# LiteLLM config file
sudo mkdir -p /mnt/docker-ssd/docker/compose/ai/litellm
```

### 7.3 — LiteLLM Config

Create `/mnt/docker-ssd/docker/compose/ai/litellm/config.yaml`:

```yaml
model_list:
  # ---- ROCINANTE (RTX 4090, 24GB) — heavy reasoning ----
  - model_name: "heavy"
    litellm_params:
      model: "ollama/qwen3:32b"
      api_base: "http://10.5.1.ROCINANTE:11434"
      timeout: 300
      stream_timeout: 300
    model_info:
      description: "Heavy reasoning, long context, complex code"

  - model_name: "heavy"
    litellm_params:
      model: "ollama/deepseek-r1:32b"
      api_base: "http://10.5.1.ROCINANTE:11434"
      timeout: 300
      stream_timeout: 300
    model_info:
      description: "Deep reasoning fallback on ROCINANTE"

  # ---- PlausibleDeniability (RTX 2080 Ti, 11GB) — general ----
  - model_name: "medium"
    litellm_params:
      model: "ollama/qwen2.5:14b"
      api_base: "http://host.docker.internal:11434"
      timeout: 120
      stream_timeout: 120
    model_info:
      description: "General homelab assistant, RAG queries"

  # ---- N.O.M.A.D. (GTX 1080, 8GB) — lightweight ----
  - model_name: "light"
    litellm_params:
      model: "ollama/phi4"
      api_base: "http://10.5.1.16:11434"
      timeout: 60
      stream_timeout: 60
    model_info:
      description: "Fast local inference, lightweight tasks"

  - model_name: "light"
    litellm_params:
      model: "ollama/llama3.2:3b"
      api_base: "http://10.5.1.16:11434"
      timeout: 60
      stream_timeout: 60
    model_info:
      description: "Ultra-light fallback on N.O.M.A.D."

  # ---- Embeddings ----
  - model_name: "embed"
    litellm_params:
      model: "ollama/nomic-embed-text"
      api_base: "http://host.docker.internal:11434"
    model_info:
      description: "Text embeddings for RAG pipeline"

  # ---- Direct model access (bypass routing) ----
  # These let you request a specific model by its full name
  - model_name: "ollama/qwen3:32b"
    litellm_params:
      model: "ollama/qwen3:32b"
      api_base: "http://10.5.1.ROCINANTE:11434"

  - model_name: "ollama/qwen2.5:14b"
    litellm_params:
      model: "ollama/qwen2.5:14b"
      api_base: "http://host.docker.internal:11434"

  - model_name: "ollama/phi4"
    litellm_params:
      model: "ollama/phi4"
      api_base: "http://10.5.1.16:11434"

litellm_settings:
  drop_params: true
  set_verbose: false
  request_timeout: 300
  num_retries: 2
  retry_after: 5
  allowed_fails: 3
  cooldown_time: 60

general_settings:
  master_key: "os.environ/LITELLM_MASTER_KEY"
```

> **DEPLOYED NOTE (2026-05-05):** Do NOT add `database_url` to general_settings — newer LiteLLM versions include Prisma ORM that requires PostgreSQL, and adding any database_url causes a crash loop. Without it, LiteLLM runs in config-only mode which is fine for homelab use. The `master_key` uses `os.environ/LITELLM_MASTER_KEY` syntax to read from the container's environment variable (set via .env → docker compose).

> **Routing explained:** Multiple entries with the same `model_name` (like "heavy") enable load balancing and failover within that tier. When you request model `heavy`, LiteLLM picks the healthiest deployment. The direct-access entries (like `ollama/qwen3:32b`) let you bypass the tier system and target a specific model when needed.

> **Replace IPs:** Update `10.5.1.ROCINANTE` with ROCINANTE's actual LAN or Tailscale IP. PD uses `host.docker.internal` since LiteLLM runs on PD alongside Ollama.

### 7.4 — docker-compose.yaml (updated ai stack)

Update `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml` to add LiteLLM:

```yaml
name: ai

services:
  qdrant:
    image: qdrant/qdrant:v1.14.0
    container_name: qdrant
    restart: unless-stopped
    ports:
      - "6333:6333"   # REST API
      - "6334:6334"   # gRPC
    environment:
      TZ: ${TZ}
      QDRANT__SERVICE__GRPC_PORT: 6334
      QDRANT__STORAGE__STORAGE_PATH: /qdrant/storage
      QDRANT__STORAGE__SNAPSHOTS_PATH: /qdrant/snapshots
    volumes:
      - /mnt/docker-ssd/docker/appdata/qdrant/storage:/qdrant/storage
      - /mnt/docker-ssd/docker/appdata/qdrant/snapshots:/qdrant/snapshots
    networks:
      - ai-services
      - default

  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    restart: unless-stopped
    ports:
      - "4000:4000"
    environment:
      TZ: ${TZ}
      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
    volumes:
      - /mnt/docker-ssd/docker/compose/ai/litellm/config.yaml:/app/config.yaml:ro
      - /mnt/docker-ssd/docker/appdata/litellm:/app/data
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - ai-services
      - default

  reranker:
    image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
    container_name: reranker
    restart: unless-stopped
    ports:
      - "8787:80"
    environment:
      MODEL_ID: ${RERANKER_MODEL}
    volumes:
      - /mnt/docker-ssd/docker/appdata/reranker:/data
    networks:
      - ai-services
      - default

  whisper:
    image: fedirz/faster-whisper-server:latest-cuda
    container_name: whisper
    restart: unless-stopped
    ports:
      - "8786:8000"
    environment:
      TZ: ${TZ}
      WHISPER__MODEL: ${WHISPER_MODEL}
      WHISPER__DEVICE: cuda
      WHISPER__COMPUTE_TYPE: float16
    volumes:
      - /mnt/docker-ssd/docker/appdata/whisper:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    networks:
      - ai-services
      - default

  # -------------------------------------------------------
  # Ollama and OpenWebUI go here when deployed.
  # They share this stack and the default + ai-services networks.
  # OpenWebUI → Qdrant at http://qdrant:6333
  # OpenWebUI → LiteLLM at http://litellm:4000 (or direct Ollama)
  # -------------------------------------------------------

  # ollama:
  #   image: ollama/ollama:latest
  #   container_name: ollama
  #   ...

  # openwebui:
  #   image: ghcr.io/open-webui/open-webui:main
  #   container_name: openwebui
  #   ...

networks:
  ai-services:
    name: ai-services
```

> **GPU sharing note:** On PD, the RTX 2080 Ti is shared between Plex (hardware transcoding), Immich ML, and Ollama. faster-whisper will also need the GPU when processing audio. These workloads are bursty (not constant), so time-sharing the GPU is viable — but be aware that a large Whisper transcription job will temporarily impact Ollama inference latency. If this becomes an issue, consider running Whisper on N.O.M.A.D.'s GTX 1080 instead and change `whisper`'s Ollama-style routing accordingly.

> **Reranker on CPU:** The reranker uses the `cpu-latest` image variant intentionally — do NOT pin to `cpu-1.5` as it has an hf-hub compatibility bug. Reranking is a lightweight operation (scoring ~50 chunks takes <1s on CPU) and doesn't justify competing for GPU VRAM. The model loads via safetensors fallback since no ONNX files exist for bge-reranker-v2-m3. Max batch size is 4 on CPU. If you find latency is an issue, a GPU variant exists (`ghcr.io/huggingface/text-embeddings-inference:turing-latest` for your 2080 Ti).

### 7.5 — .env.example (updated ai stack)

```bash
# /mnt/docker-ssd/docker/compose/ai/.env.example

TZ=America/New_York

# LiteLLM
LITELLM_MASTER_KEY=sk-CHANGE_ME

# Reranker
RERANKER_MODEL=BAAI/bge-reranker-v2-m3

# Whisper
WHISPER_MODEL=Systran/faster-distil-whisper-large-v3
```

### 7.6 — Validate & Deploy

```bash
cd /mnt/docker-ssd/docker/compose/ai
cp .env.example .env
nano .env  # Set LITELLM_MASTER_KEY=$(openssl rand -hex 32)

docker compose --env-file .env config
docker compose --env-file .env up -d
```

### 7.7 — Post-Deploy Verification

```bash
# All containers running?
docker ps --filter name=qdrant --filter name=litellm --filter name=reranker --filter name=whisper

# LiteLLM health?
curl -s http://localhost:4000/health

# LiteLLM can see all models?
curl -s http://localhost:4000/v1/models \
  -H "Authorization: Bearer YOUR_MASTER_KEY" | python3 -m json.tool

# Test a chat completion through LiteLLM
curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer YOUR_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "medium",
    "messages": [{"role": "user", "content": "Hello, which model are you?"}]
  }' | python3 -m json.tool

# Test embeddings through LiteLLM
curl -s http://localhost:4000/v1/embeddings \
  -H "Authorization: Bearer YOUR_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "embed",
    "input": "test embedding"
  }' | python3 -m json.tool
```

### 7.8 — Integration Updates

With LiteLLM deployed, all apps should point at `http://litellm:4000` (within the `ai-services` network) or `http://PD_IP:4000` (from external hosts) instead of direct Ollama endpoints.

**OpenWebUI:** Settings → Connections → add OpenAI-compatible endpoint:
- URL: `http://litellm:4000/v1`
- API Key: your `LITELLM_MASTER_KEY`
- This gives OpenWebUI access to all models across all three machines through one connection

**Continue.dev:** Update `~/.continue/config.json`:
```json
{
  "models": [
    {
      "title": "Heavy (ROCINANTE)",
      "provider": "openai",
      "model": "heavy",
      "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
      "apiKey": "YOUR_MASTER_KEY"
    },
    {
      "title": "Medium (PD)",
      "provider": "openai",
      "model": "medium",
      "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
      "apiKey": "YOUR_MASTER_KEY"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Light (N.O.M.A.D.)",
    "provider": "openai",
    "model": "light",
    "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
    "apiKey": "YOUR_MASTER_KEY"
  }
}
```

> **Key change:** Continue.dev now uses `provider: "openai"` instead of `provider: "ollama"` and points at LiteLLM. One IP to remember, and if you add or move models between machines, you only update `config.yaml` — not every app.

**n8n workflows:** All HTTP Request nodes that call Ollama directly should be updated:
- Old: `http://10.5.1.16:11434/api/generate` (N.O.M.A.D.)
- New: `http://litellm:4000/v1/chat/completions` with `model: "light"`
- n8n is on the `ai-services` network, so it reaches LiteLLM by Docker DNS

**n8n Multi-Model Query Router workflow:** This workflow is now **simplified dramatically** — instead of a Switch node routing to three different Ollama IPs, it becomes a single HTTP Request node that passes the model tier name (`light`, `medium`, `heavy`) to LiteLLM and lets the gateway handle routing. The Switch node logic can be removed entirely.

---

## Phase 8 — Reranker (RAG Quality)

**Host:** PlausibleDeniability (deployed as part of the `ai` stack in Phase 7)
**Service:** `reranker` container (already in the compose above)
**Why:** Dramatically improves RAG answer quality by filtering out noisy retrieval results before they reach the LLM.

### 8.1 — How Reranking Works

Without a reranker, your RAG pipeline does:
```
Query → embed → Qdrant top-10 by vector similarity → all 10 chunks go to LLM
```

The problem: vector similarity often returns "close but irrelevant" chunks. The LLM gets noisy context and hallucinates.

With a reranker:
```
Query → embed → Qdrant top-50 by vector similarity → reranker scores each (query, chunk) pair → top-5 by relevance go to LLM
```

The reranker is a cross-encoder that reads the full query AND each chunk together, producing a much more accurate relevance score than vector distance alone. It over-retrieves cheaply from Qdrant, then precisely filters.

### 8.2 — Scaffold Directories

```bash
# Model cache on SSD (reranker model is ~1.1GB, downloaded on first start)
sudo mkdir -p /mnt/docker-ssd/docker/appdata/reranker
```

### 8.3 — Post-Deploy Verification

```bash
# Container running?
docker ps --filter name=reranker

# Health check?
curl -s http://localhost:8787/health

# Test reranking
curl -s http://localhost:8787/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "query": "How do I restart a Docker container?",
    "texts": [
      "Use docker restart <container_name> to restart a running container.",
      "Docker was founded in 2013 by Solomon Hykes.",
      "The docker compose down command stops and removes containers.",
      "Kubernetes pods can be restarted by deleting them."
    ]
  }' | python3 -m json.tool
# Expected: the first text scores highest, second scores lowest
```

### 8.4 — n8n RAG Pipeline Integration

Update the "Qdrant Index Updater" and any RAG query workflows to include a reranking step.

**Updated RAG Query Workflow (for OpenWebUI or any n8n-based query):**
```
Trigger:  Webhook node (POST with query)
Step 1:   HTTP Request → LiteLLM /v1/embeddings
            - model: "embed"
            - input: query text
Step 2:   HTTP Request → Qdrant API (POST /collections/{name}/points/search)
            - vector: embedding from step 1
            - limit: 50  (over-retrieve)
Step 3:   HTTP Request → Reranker (POST http://reranker:80/rerank)
            - query: original query text
            - texts: array of 50 chunk texts from Qdrant results
Step 4:   Code node → take top 5 by reranker score, format as context
Step 5:   HTTP Request → LiteLLM /v1/chat/completions
            - model: "medium" (or "heavy" for complex queries)
            - messages: system prompt with top-5 context + user query
Step 6:   Return response via webhook
```

> **Key difference from the original plan:** Step 2 now retrieves 50 results instead of 10, and Step 3 (reranking) filters down to the best 5. This "over-retrieve then rerank" pattern is the standard approach for production RAG systems.

---

## Phase 9 — faster-whisper (Speech-to-Text)

**Host:** PlausibleDeniability (deployed as part of the `ai` stack in Phase 7)
**Service:** `whisper` container (already in the compose above)
**Why:** Replaces shelved Scriberr with an OpenAI-compatible STT API. No SQLite dependency, no ZFS/ACL issues.

### 9.1 — Scaffold Directories

```bash
# Model cache on SSD (Whisper models are 1-3GB)
sudo mkdir -p /mnt/docker-ssd/docker/appdata/whisper
```

### 9.2 — Post-Deploy Verification

```bash
# Container running?
docker ps --filter name=whisper

# Health check?
curl -s http://localhost:8786/health

# Test transcription with a sample audio file
curl -s http://localhost:8786/v1/audio/transcriptions \
  -F "file=@/path/to/test-audio.wav" \
  -F "model=Systran/faster-distil-whisper-large-v3" \
  | python3 -m json.tool
```

### 9.3 — n8n Integration Workflows

**Workflow: Voice Note → Text → Summary**
```
Trigger:  Webhook node (POST with audio file in body)
Step 1:   HTTP Request → Whisper (POST http://whisper:8000/v1/audio/transcriptions)
            - Multipart form with audio file
            - response_format: "json"
Step 2:   HTTP Request → LiteLLM /v1/chat/completions
            - model: "medium"
            - Prompt: "Summarize this voice note concisely: {transcript}"
Step 3:   HTTP Request → Gotify → push summary to phone
Step 4:   (Optional) Postgres node → log transcript and summary
```

**Workflow: Audio File → Paperless Document**
```
Trigger:  Webhook or filesystem watcher
Step 1:   HTTP Request → Whisper → get transcript
Step 2:   Code node → format transcript as text document
Step 3:   HTTP Request → Paperless API (POST /api/documents/post_document/)
            - Upload transcript as .txt
            - Tag: "transcription"
Step 4:   Gotify notification: "Audio transcribed and filed: {title}"
```

**Home Assistant Voice Integration:**
If you want HA voice commands, Whisper can serve as the STT backend:
1. In Home Assistant → Settings → Voice Assistants
2. Add speech-to-text provider: "Whisper" at `http://PD_IP:8786`
3. Pair with Piper TTS (future addition) for full voice assistant loop

### 9.4 — GPU vs CPU Considerations

The compose file above uses the GPU variant with CUDA. If GPU contention with Ollama becomes an issue:

**Option A — CPU fallback on PD:**
Change the image from `latest-cuda` to `fedirz/faster-whisper-server:latest-cpu` and remove the `deploy.resources` block and the CUDA environment variables (`WHISPER__DEVICE`, `WHISPER__COMPUTE_TYPE`). Transcription will be slower (~4x real-time instead of ~20x) but won't compete for VRAM.

**Option B — Move to N.O.M.A.D.:**
Run Whisper on N.O.M.A.D.'s GTX 1080 which is less contested. Add it to N.O.M.A.D.'s compose or run standalone. N.O.M.A.D.'s 8GB VRAM handles Whisper models comfortably since phi4 doesn't fill it.

---

## Additional Tools Setup

### Continue.dev (Local AI Code Completion)

1. Install the Continue extension in VS Code
2. Create/edit `~/.continue/config.json`:

**With LiteLLM (recommended — see Phase 7):**
```json
{
  "models": [
    {
      "title": "Heavy (ROCINANTE)",
      "provider": "openai",
      "model": "heavy",
      "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
      "apiKey": "YOUR_LITELLM_MASTER_KEY"
    },
    {
      "title": "Medium (PD)",
      "provider": "openai",
      "model": "medium",
      "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
      "apiKey": "YOUR_LITELLM_MASTER_KEY"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Light (N.O.M.A.D.)",
    "provider": "openai",
    "model": "light",
    "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
    "apiKey": "YOUR_LITELLM_MASTER_KEY"
  }
}
```

> One IP, one API key, all three machines. If you add or move models, update LiteLLM's `config.yaml` — not every app.

**Without LiteLLM (direct Ollama, if Phase 7 is not yet deployed):**
```json
{
  "models": [
    {
      "title": "PD - qwen2.5:14b",
      "provider": "ollama",
      "model": "qwen2.5:14b",
      "apiBase": "http://PD_TAILSCALE_IP:11434"
    },
    {
      "title": "ROCINANTE - qwen3:32b",
      "provider": "ollama",
      "model": "qwen3:32b",
      "apiBase": "http://ROCINANTE_TAILSCALE_IP:11434"
    }
  ],
  "tabAutocompleteModel": {
    "title": "N.O.M.A.D. - phi4",
    "provider": "ollama",
    "model": "phi4",
    "apiBase": "http://NOMAD_TAILSCALE_IP:11434"
  }
}
```

### Obsidian + Gitea Sync

1. In Obsidian, install the "Obsidian Git" community plugin
2. Initialize your vault as a git repo:
   ```bash
   cd /path/to/obsidian/vault
   git init
   git remote add origin http://PD_IP:3000/fizzlepoof/obsidian-vault.git
   ```
3. In Obsidian Git settings:
   - Auto backup interval: 10 minutes
   - Pull on startup: enabled
4. Create the repo in Gitea first: `http://PD_IP:3000` → New Repository → `obsidian-vault`
5. Add a Gitea webhook to trigger the n8n "Qdrant Index Updater" workflow (see Phase 3)

### Homepage Ollama Widget

Add to your Homepage configuration (`/mnt/tank/docker/appdata/homepage/services.yaml`):

```yaml
- AI:
    - Ollama (PD):
        icon: ollama.svg
        href: http://PD_IP:11434
        widget:
          type: ollama
          url: http://PD_IP:11434
    - Ollama (ROCINANTE):
        icon: ollama.svg
        href: http://ROCINANTE_IP:11434
        widget:
          type: ollama
          url: http://ROCINANTE_IP:11434
```

---

## Deployment Order Summary

```
Week 1:  Phase 1 — Gotify
           └─ Deploy, create app tokens, install phone app
           └─ Test: send manual notification via API

Week 1:  Phase 2 — Qdrant
           └─ Deploy, verify REST API
           └─ Create initial collections (empty, ready for n8n)

Week 2:  Phase 3 — n8n
           └─ Deploy, create admin account
           └─ Build workflows incrementally:
              Day 1: Gitea commit → Gotify (simplest, proves the pipeline)
              Day 2: Sonarr/Radarr → TMDB → Gotify + Discord
              Day 3: Tautulli play logging + weekly digest
              Day 4: Uptime Kuma enhanced alerts
              Day 5: ZFS pool monitoring
              Day 6: Multi-model query router
              Day 7: Qdrant index updater

Week 3:  Phase 4 — Paperless-NGX
           └─ Deploy, ingest test documents
           └─ Build n8n workflow: email → Paperless → Ollama summary
           └─ Set up document tags and correspondents for L&L Crafts

Week 4:  Phase 5 — Home Assistant
           └─ Deploy, onboard, add integrations
           └─ Connect to n8n via long-lived access token
           └─ Set up smart plug monitoring

Week 4:  Phase 6 — Grafana + Prometheus (on N.O.M.A.D.)
           └─ Deploy, verify scrape targets
           └─ Import dashboards
           └─ Set up Grafana → n8n alert webhook
           └─ Build alert rules

Week 5:  Phase 7/8/9 — AI Stack Expansion (all deploy together in ai stack)
           └─ LiteLLM: deploy, verify model routing across all 3 machines
           └─ Reranker: deploy, test scoring with sample chunks
           └─ faster-whisper: deploy, test transcription
           └─ Update OpenWebUI, Continue.dev, and n8n to use LiteLLM endpoint
           └─ Update RAG workflows to include reranking step
           └─ Build voice note → transcription → summary workflow
```

---

## Post-Deployment Validation Master Checklist

Run through this after all six phases are deployed:

### Infrastructure Health

- [ ] `docker ps` on PD shows: gotify, n8n, qdrant, litellm, reranker, whisper, paperless, paperless-gotenberg, paperless-tika, homeassistant — all healthy
- [ ] `docker ps` on N.O.M.A.D. shows: prometheus, grafana, node-exporter — all healthy
- [ ] All services accessible via Pangolin subdomains (gotify, n8n, paperless, ha, grafana)

### Network Connectivity

- [ ] n8n can reach shared-postgres (test: check n8n logs for successful DB migration)
- [ ] n8n can reach Gotify (test: trigger a workflow that sends a notification)
- [ ] n8n can reach Qdrant at `http://qdrant:6333` via `ai-services` network (test: query collections from n8n HTTP node)
- [ ] n8n can reach LiteLLM at `http://litellm:4000` via `ai-services` network (test: send a chat completion)
- [ ] LiteLLM can route to all 3 Ollama instances (test: request model "light", "medium", "heavy" and verify each responds)
- [ ] Reranker responds at `http://reranker:80/rerank` (test: POST sample texts)
- [ ] Whisper responds at `http://whisper:8000/v1/audio/transcriptions` (test: transcribe a sample .wav)
- [ ] Paperless can reach shared-postgres and shared-redis
- [ ] Prometheus can scrape PD and Serenity Netdata endpoints
- [ ] Grafana alert webhook reaches n8n

### Data Flow End-to-End

- [ ] Sonarr webhook → n8n → Gotify push on phone
- [ ] Gitea commit → n8n → Qdrant indexing → Gotify notification
- [ ] RAG query → Qdrant top-50 → reranker top-5 → LiteLLM → accurate answer
- [ ] Audio file → Whisper transcription → n8n → Gotify summary
- [ ] Document dropped in consume folder → Paperless OCR → n8n summary → Gotify
- [ ] Grafana alert fires → n8n webhook → remediation action or Gotify alert
- [ ] HA presence change → n8n → server wake/sleep

### Backup Considerations

- [ ] Gotify SQLite DB is on SSD — include `/mnt/docker-ssd/docker/appdata/gotify` in backup plan
- [ ] Qdrant storage on SSD — include `/mnt/docker-ssd/docker/appdata/qdrant` in backup plan
- [ ] n8n config/workflows — include `/mnt/tank/docker/appdata/n8n` in backup plan
- [ ] Paperless media — include `/mnt/tank/docker/appdata/paperless/media` in backup plan
- [ ] HA config — include `/mnt/docker-ssd/docker/appdata/homeassistant` in backup plan
- [ ] LiteLLM DB + config — include `/mnt/docker-ssd/docker/appdata/litellm` and `compose/ai/litellm/config.yaml` in backup plan
- [ ] Grafana/Prometheus data — include `/mnt/hdd-2/grafana-data` and `/mnt/hdd-2/prometheus-data` in N.O.M.A.D. backup plan
- [ ] All new databases (n8n, paperless) are in shared-postgres — ensure pg_dump covers them

### Storage Capacity Check

- [ ] `docker-ssd` (500GB Crucial MX500) still has headroom after adding Gotify, Qdrant, HA, Paperless data
- [ ] `tank` mirror has room for n8n config + Paperless media + consume/export
- [ ] N.O.M.A.D. hdd-2 has room for Prometheus TSDB (90-day retention) + Grafana data

---

> **Remember:** Everything self-hosted. Nothing touches the cloud.