fizzlepoof/truenas-stacks
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ab348932ff8ccffdc80d2fb2a47e1873aa6778ac
truenas-stacks/HOMELAB_BUILDOUT_PLAN.md

1706 lines
65 KiB
Markdown
Raw Blame History

1|# Homelab Expansion — Full Buildout Plan
2|
3|> Generated: 2026-05-04
4|> Based on: HOMELAB_EXPANSION_PLAN.md, ARCHITECTURE_OVERVIEW.md, STACK_STANDARDS.md
5|
6|This document contains everything needed to deploy all six expansion phases — docker-compose files, .env templates, directory scaffolding commands, database init SQL, Pangolin configuration, and validation steps. Each phase follows the established stack standards and deployment checklist.
7|
8|---
9|
10|## Table of Contents
11|
12|1. [Prerequisites & Conventions](#prerequisites--conventions)
13|2. [Phase 1 — Gotify (Notifications)](#phase-1--gotify-notifications)
14|3. [Phase 2 — Qdrant (Vector Database)](#phase-2--qdrant-vector-database)
15|4. [Phase 3 — n8n (Workflow Automation)](#phase-3--n8n-workflow-automation)
16|5. [Phase 4 — Paperless-NGX (Document Intelligence)](#phase-4--paperless-ngx-document-intelligence)
17|6. [Phase 5 — Home Assistant (Home Automation)](#phase-5--home-assistant-home-automation)
18|7. [Phase 6 — Grafana + Prometheus (Observability)](#phase-6--grafana--prometheus-observability)
19|8. [Phase 7 — LiteLLM (AI Gateway)](#phase-7--litellm-ai-gateway)
20|9. [Phase 8 — Reranker (RAG Quality)](#phase-8--reranker-rag-quality)
21|10. [Phase 9 — faster-whisper (Speech-to-Text)](#phase-9--faster-whisper-speech-to-text)
22|11. [Additional Tools Setup](#additional-tools-setup)
23|12. [Deployment Order Summary](#deployment-order-summary)
24|13. [Post-Deployment Validation Master Checklist](#post-deployment-validation-master-checklist)
25|
26|---
27|
28|## Prerequisites & Conventions
29|
30|### Deployment Standards (recap)
31|
32|All stacks on PlausibleDeniability follow the same pattern:
33|
34|- **Repo root:** `/mnt/docker-ssd/docker/compose`
35|- **Validate before deploy:** `docker compose --env-file .env config`
36|- **Deploy:** `docker compose --env-file .env up -d`
37|- **Teardown:** `docker compose --env-file .env down`
38|
39|### Storage Tiers
40|
41|| Tier | Mount | Use For |
42||------|-------|---------|
43|| SSD | `/mnt/docker-ssd/docker/appdata/<service>` | Write-heavy, SQLite, GPU/model, databases |
44|| Tank | `/mnt/tank/docker/appdata/<service>` | General appdata, configs, uploads |
45|| Unraid | `/mnt/unraid/data/media/` | Media libraries only |
46|
47|### Networks
48|
49|| Network | Created By | Purpose |
50||---------|-----------|---------|
51|| `ix-databases_shared-databases` | databases stack | Access to shared-postgres, shared-mariadb, shared-redis |
52|| `pangolin` | newt (infrastructure stack) | Reverse proxy / external exposure |
53|
54|### Secret Generation Commands
55|
56|```bash
57|# Database passwords
58|openssl rand -hex 24
59|
60|# JWT / encryption keys
61|openssl rand -hex 32
62|
63|# Paperless secret key
64|python3 -c "import secrets; print(secrets.token_urlsafe(50))"
65|```
66|
67|### Deployment Order
68|
69|```
70|databases (already running) → infrastructure (already running)
71| → Phase 1: Gotify
72| → Phase 2: Qdrant (creates the ai-services network)
73| → Phase 3: n8n (depends on Gotify + Qdrant — joins ai-services as external)
74| → Phase 4: Paperless-NGX (depends on n8n for automation hooks)
75| → Phase 5: Home Assistant (depends on n8n for heavy automation)
76| → Phase 6: Grafana + Prometheus (on N.O.M.A.D., independent but benefits from all above)
77|```
78|
79|> **Important:** The `ai` stack (Phase 2) must be deployed before the `automation` stack (Phase 3) because n8n declares `ai-services` as an external network. If the `ai` stack isn't up, `docker compose up` for `automation` will fail with a missing network error.
80|
81|---
82|
83|## Phase 1 — Gotify (Notifications)
84|
85|**Host:** PlausibleDeniability
86|**Stack directory:** `/mnt/docker-ssd/docker/compose/automation/`
87|**Why first:** Every subsequent phase sends notifications through Gotify. It's the output bus.
88|
89|### 1.1 — Scaffold Directories
90|
91|```bash
92|# Appdata on SSD (SQLite backend — must not be on NFS)
93|sudo mkdir -p /mnt/docker-ssd/docker/appdata/gotify
94|
95|# Stack directory (may already exist if automation/ is planned)
96|sudo mkdir -p /mnt/docker-ssd/docker/compose/automation
97|```
98|
99|### 1.2 — Database Init
100|
101|None required — Gotify uses an embedded SQLite database stored in its data volume.
102|
103|### 1.3 — docker-compose.yaml
104|
105|Add to `/mnt/docker-ssd/docker/compose/automation/docker-compose.yaml`:
106|
107|```yaml
108|name: automation
109|
110|services:
111| gotify:
112| image: gotify/server:2.6.1
113| container_name: gotify
114| restart: unless-stopped
115| ports:
116| - "8484:80"
117| environment:
118| TZ: ${TZ}
119| GOTIFY_DEFAULTUSER_PASS: ${GOTIFY_ADMIN_PASS}
120| GOTIFY_SERVER_PORT: 80
121| GOTIFY_DATABASE_DIALECT: sqlite3
122| GOTIFY_DATABASE_CONNECTION: data/gotify.db
123| volumes:
124| - /mnt/docker-ssd/docker/appdata/gotify:/app/data
125| networks:
126| - pangolin
127| - default
128|
129|networks:
130| pangolin:
131| external: true
132|```
133|
134|> **Image note:** `gotify/server:2.6.1` is the latest stable as of May 2026. Check [Docker Hub](https://hub.docker.com/r/gotify/server/tags) before deploying — pin to the exact version.
135|
136|### 1.4 — .env.example
137|
138|```bash
139|# /mnt/docker-ssd/docker/compose/automation/.env.example
140|
141|TZ=America/New_York
142|GOTIFY_ADMIN_PASS=CHANGE_ME
143|```
144|
145|### 1.5 — .env (create from example)
146|
147|```bash
148|cd /mnt/docker-ssd/docker/compose/automation
149|cp .env.example .env
150|# Edit .env with real values:
151|# GOTIFY_ADMIN_PASS=$(openssl rand -hex 16)
152|nano .env
153|```
154|
155|### 1.6 — Pangolin Configuration
156|
157|In your Pangolin dashboard, create a new resource:
158|
159|| Field | Value |
160||-------|-------|
161|| Domain | `gotify.paccoco.com` |
162|| Scheme | `http` |
163|| Host | `gotify` |
164|| Port | `80` |
165|| Network | Docker service name resolution via `pangolin` network |
166|
167|### 1.7 — Validate & Deploy
168|
169|```bash
170|cd /mnt/docker-ssd/docker/compose/automation
171|docker compose --env-file .env config
172|docker compose --env-file .env up -d
173|```
174|
175|### 1.8 — Post-Deploy Verification
176|
177|```bash
178|# Container running?
179|docker ps --filter name=gotify
180|
181|# Clean startup logs?
182|docker logs gotify --tail 20
183|
184|# Mounts correct?
185|docker inspect gotify --format '{{json .Mounts}}' | python3 -m json.tool
186|
187|# Quick health check
188|curl -s http://localhost:8484/health
189|
190|# Test notification via API
191|curl -s "http://localhost:8484/message?token=*** \
192| -F "title=Homelab" \
193| -F "message=Gotify is online" \
194| -F "priority=5"
195|```
196|
197|### 1.9 — First-Run Setup
198|
199|1. Navigate to `https://gotify.paccoco.com`
200|2. Log in with the admin password from `.env`
201|3. **Change the default admin password** in the UI
202|4. Create application tokens for each notification source:
203| - `n8n-workflows` — for all n8n automations
204| - `infrastructure-alerts` — for Uptime Kuma, Grafana, etc.
205| - `media-notifications` — for Sonarr/Radarr/Tautulli hooks
206| - `home-assistant` — for HA automations
207|5. Create client tokens for each receiving device (phone, desktop)
208|6. Install the Gotify Android app and configure with your client token
209|
210|---
211|
212|## Phase 2 — AI Stack (Vector DB, LLM, Embeddings, Reranker, STT)
213|
214|**Host:** PlausibleDeniability
215|**Stack directory:** `/mnt/docker-ssd/docker/compose/ai/`
216|**Status:** DEPLOYED AND VERIFIED (2026-05-05)
217|
218|> **DEPLOYED NOTE:** This phase was deployed as a complete AI stack with 6 services: Ollama (11434), OpenWebUI (8282), Qdrant (6333/6334), LiteLLM (4000), Reranker/TEI (8787), and faster-whisper (8786). The compose below shows the original Qdrant-only plan — see the actual running compose on PD at `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml` and the `project_ai_stack_deployed.md` memory for port mappings and fixes. Key lesson: always use `10.5.30.6` not `localhost` for health checks on TrueNAS Scale.
219|
220|### 2.1 — Scaffold Directories
221|
222|```bash
223|# Appdata on SSD (write-heavy vector storage)
224|sudo mkdir -p /mnt/docker-ssd/docker/appdata/qdrant/storage
225|sudo mkdir -p /mnt/docker-ssd/docker/appdata/qdrant/snapshots
226|
227|# Stack directory
228|sudo mkdir -p /mnt/docker-ssd/docker/compose/ai
229|```
230|
231|### 2.2 — docker-compose.yaml
232|
233|Create or update `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml`:
234|
235|```yaml
236|name: ai
237|
238|services:
239| qdrant:
240| image: qdrant/qdrant:v1.14.0
241| container_name: qdrant
242| restart: unless-stopped
243| ports:
244| - "6333:6333" # REST API
245| - "6334:6334" # gRPC
246| environment:
247| TZ: ${TZ}
248| QDRANT__SERVICE__GRPC_PORT: 6334
249| QDRANT__STORAGE__STORAGE_PATH: /qdrant/storage
250| QDRANT__STORAGE__SNAPSHOTS_PATH: /qdrant/snapshots
251| volumes:
252| - /mnt/docker-ssd/docker/appdata/qdrant/storage:/qdrant/storage
253| - /mnt/docker-ssd/docker/appdata/qdrant/snapshots:/qdrant/snapshots
254| networks:
255| - ai-services
256| - default
257|
258| # -------------------------------------------------------
259| # Ollama and OpenWebUI go here when deployed.
260| # They share this stack and the default + ai-services networks so
261| # OpenWebUI can reach Qdrant at http://qdrant:6333
262| # -------------------------------------------------------
263|
264| # ollama:
265| # image: ollama/ollama:latest
266| # container_name: ollama
267| # ...
268|
269| # openwebui:
270| # image: ghcr.io/open-webui/open-webui:main
271| # container_name: openwebui
272| # ...
273|
274|networks:
275| ai-services:
276| name: ai-services
277|```
278|
279|> **Image note:** `qdrant/qdrant:v1.14.0` is the latest stable as of May 2026. Check [GitHub releases](https://github.com/qdrant/qdrant/releases) before deploying.
280|
281|> **Cross-stack connectivity:** The `ai-services` network is defined here with an explicit `name:` so other stacks (like `automation/n8n`) can declare it as `external: true` and reach Qdrant by service name (`http://qdrant:6333`). This follows the same pattern as `ix-databases_shared-databases`.
282|
283|### 2.3 — .env.example
284|
285|```bash
286|# /mnt/docker-ssd/docker/compose/ai/.env.example
287|
288|TZ=America/New_York
289|```
290|
291|### 2.4 — Validate & Deploy
292|
293|```bash
294|cd /mnt/docker-ssd/docker/compose/ai
295|cp .env.example .env
296|nano .env # Set timezone
297|
298|docker compose --env-file .env config
299|docker compose --env-file .env up -d
300|```
301|
302|### 2.5 — Post-Deploy Verification
303|
304|```bash
305|# Container running?
306|docker ps --filter name=qdrant
307|
308|# Clean startup?
309|docker logs qdrant --tail 20
310|
311|# REST API responding?
312|curl -s http://localhost:6333/healthz
313|# Expected: {"title":"qdrant - vectorass engine","version":"1.14.0","commit":"..."}
314|
315|# Create a test collection
316|curl -X PUT http://localhost:6333/collections/test_collection \
317| -H "Content-Type: application/json" \
318| -d '{
319| "vectors": {
320| "size": 384,
321| "distance": "Cosine"
322| }
323| }'
324|
325|# Verify it exists
326|curl -s http://localhost:6333/collections | python3 -m json.tool
327|
328|# Clean up test
329|curl -X DELETE http://localhost:6333/collections/test_collection
330|```
331|
332|### 2.6 — OpenWebUI Integration (when deployed)
333|
334|When Ollama and OpenWebUI are brought online in this same stack, configure OpenWebUI's RAG settings:
335|
336|1. Go to OpenWebUI → Admin → Settings → Documents
337|2. Set the vector database to Qdrant
338|3. Endpoint: `http://qdrant:6333` (Docker DNS within the `ai` stack network)
339|4. Collection name: `openwebui_docs` (or your preference)
340|
341|### 2.7 — Collections to Create (for n8n in Phase 3)
342|
343|These collections will be created programmatically by n8n workflows, but for reference:
344|
345|| Collection | Vector Size | Content |
346||-----------|-------------|---------|
347|| `homelab_docs` | 384 (nomic-embed-text) | Homelab markdown documentation |
348|| `gitea_commits` | 384 | Gitea commit messages + diffs |
349|| `media_metadata` | 384 | Plex/Tautulli metadata |
350|| `obsidian_notes` | 384 | Personal notes from Obsidian vault |
351|
352|> **Vector size note:** 384 is the dimension for `nomic-embed-text` via Ollama. If you use a different embedding model, adjust accordingly.
353|
354|---
355|
356|## Phase 3 — n8n (Workflow Automation)
357|
358|**Host:** PlausibleDeniability
359|**Stack directory:** `/mnt/docker-ssd/docker/compose/automation/` (same stack as Gotify)
360|**Why third:** n8n is the orchestration backbone — it ties Gotify, Qdrant, Ollama, and all triggers together.
361|
362|### 3.1 — Scaffold Directories
363|
364|```bash
365|# Config on tank (workflow definitions, credentials store)
366|# NOTE: n8n also writes execution logs here. If execution logging becomes
367|# heavy (many workflows running frequently), consider moving to SSD.
368|# For typical homelab usage (~20 workflows), tank is fine.
369|sudo mkdir -p /mnt/tank/docker/appdata/n8n
370|
371|# Set ownership — n8n runs as UID 1000 (node user)
372|sudo chown 1000:1000 /mnt/tank/docker/appdata/n8n
373|```
374|
375|### 3.2 — Database Init
376|
377|n8n uses the existing shared-postgres. Create its database and user:
378|
379|```bash
380|docker exec -i shared-postgres psql -U postgres <<'SQL'
381|CREATE USER n8n WITH PASSWORD 'REPLACE_WITH_GENERATED_PASSWORD';
382|CREATE DATABASE n8n OWNER n8n;
383|GRANT ALL PRIVILEGES ON DATABASE n8n TO n8n;
384|SQL
385|```
386|
387|Generate the password first:
388|```bash
389|openssl rand -hex 24
390|```
391|
392|### 3.3 — docker-compose.yaml
393|
394|Update `/mnt/docker-ssd/docker/compose/automation/docker-compose.yaml` to add n8n alongside Gotify:
395|
396|```yaml
397|name: automation
398|
399|services:
400| gotify:
401| image: gotify/server:2.6.1
402| container_name: gotify
403| restart: unless-stopped
404| ports:
405| - "8484:80"
406| environment:
407| TZ: ${TZ}
408| GOTIFY_DEFAULTUSER_PASS: ${GOTIFY_ADMIN_PASS}
409| GOTIFY_SERVER_PORT: 80
410| GOTIFY_DATABASE_DIALECT: sqlite3
411| GOTIFY_DATABASE_CONNECTION: data/gotify.db
412| volumes:
413| - /mnt/docker-ssd/docker/appdata/gotify:/app/data
414| networks:
415| - pangolin
416| - default
417|
418| n8n:
419| image: docker.n8n.io/n8nio/n8n:1.88.0
420| container_name: n8n
421| restart: unless-stopped
422| ports:
423| - "5678:5678"
424| environment:
425| TZ: ${TZ}
426| # Database
427| DB_TYPE: postgresdb
428| DB_POSTGRESDB_HOST: shared-postgres
429| DB_POSTGRESDB_PORT: 5432
430| DB_POSTGRESDB_DATABASE: ${N8N_DB_NAME}
431| DB_POSTGRESDB_USER: ${N8N_DB_USER}
432| DB_POSTGRESDB_PASSWORD: ${N8N_DB_PASS}
433| # Security
434| N8N_ENCRYPTION_KEY: ${N8N_ENCRYPTION_KEY}
435| # Webhook / Reverse Proxy
436| N8N_HOST: ${N8N_HOST}
437| N8N_PROTOCOL: https
438| N8N_PORT: 5678
439| WEBHOOK_URL: https://${N8N_HOST}/
440| N8N_PROXY_HOPS: 1
441| # General
442| GENERIC_TIMEZONE: ${TZ}
443| N8N_DIAGNOSTICS_ENABLED: false
444| N8N_PERSONALIZATION_ENABLED: false
445| volumes:
446| - /mnt/tank/docker/appdata/n8n:/home/node/.n8n
447| networks:
448| - pangolin
449| - ix-databases_shared-databases
450| - ai-services
451| - default
452| depends_on:
453| - gotify
454|
455|networks:
456| pangolin:
457| external: true
458| ix-databases_shared-databases:
459| external: true
460| ai-services:
461| external: true
462|```
463|
464|> **Image note:** `docker.n8n.io/n8nio/n8n:1.88.0` — n8n uses calendar-ish versioning. Check [n8n releases](https://github.com/n8n-io/n8n/releases) for the latest stable. The `docker.n8n.io` registry is preferred for production.
465|
466|> **Cross-stack access:** n8n joins `ai-services` (created by the `ai` stack) so it can reach Qdrant at `http://qdrant:6333` by Docker DNS. It also joins `ix-databases_shared-databases` for Postgres access — same pattern.
467|
468|### 3.4 — .env.example (updated for both services)
469|
470|```bash
471|# /mnt/docker-ssd/docker/compose/automation/.env.example
472|
473|TZ=America/New_York
474|
475|# Gotify
476|GOTIFY_ADMIN_PASS=CHANGE_ME
477|
478|# n8n — Database
479|N8N_DB_NAME=n8n
480|N8N_DB_USER=n8n
481|N8N_DB_PASS=CHANGE_ME
482|
483|# n8n — Security
484|N8N_ENCRYPTION_KEY=CHANGE_ME
485|
486|# n8n — Hostname
487|N8N_HOST=n8n.paccoco.com
488|```
489|
490|### 3.5 — Pangolin Configuration
491|
492|| Field | Value |
493||-------|-------|
494|| Domain | `n8n.paccoco.com` |
495|| Scheme | `http` |
496|| Host | `n8n` |
497|| Port | `5678` |
498|| Headers | Forward `X-Forwarded-Proto: https` |
499|
500|### 3.6 — Validate & Deploy
501|
502|```bash
503|cd /mnt/docker-ssd/docker/compose/automation
504|
505|# Update .env with real passwords
506|nano .env
507|
508|docker compose --env-file .env config
509|docker compose --env-file .env up -d
510|```
511|
512|### 3.7 — Post-Deploy Verification
513|
514|```bash
515|# Both containers running?
516|docker ps --filter name=gotify --filter name=n8n
517|
518|# n8n logs clean?
519|docker logs n8n --tail 30
520|
521|# n8n on correct networks?
522|docker inspect n8n --format '{{json .NetworkSettings.Networks}}' | python3 -m json.tool
523|
524|# n8n UI accessible?
525|curl -s -o /dev/null -w "%{http_code}" http://localhost:5678/
526|# Expected: 200
527|
528|# Postgres connection working? (check logs for DB migration messages)
529|docker logs n8n 2>&1 | grep -i "migrat"
530|```
531|
532|### 3.8 — First-Run Setup
533|
534|1. Navigate to `https://n8n.paccoco.com`
535|2. Create your admin account
536|3. Install community nodes you'll need:
537| - `n8n-nodes-gotify` (if available) or use HTTP Request node
538| - Ollama nodes (built-in as of n8n 1.x)
539|
540|### 3.9 — Workflow Blueprints
541|
542|Below are starter workflow descriptions for each planned automation. These are meant to be built in the n8n UI — the structure and node types are described so you can wire them up.
543|
544|#### Media Pipeline Workflows
545|
546|**Workflow: Sonarr/Radarr Download Notification**
547|```
548|Trigger: Webhook node (POST from Sonarr/Radarr on download/import)
549|Step 1: Extract series/movie name, quality, file path from webhook body
550|Step 2: HTTP Request → TMDB API to fetch poster image URL
551|Step 3: HTTP Request → Gotify REST API (POST /message)
552| - Title: "New Download: {title}"
553| - Message: "{quality} — {episodeTitle or year}"
554| - Priority: 5
555| - Extras: attach poster URL as markdown image
556|Also: HTTP Request → Discord webhook (formatted embed with poster)
557|```
558|
559|Configure Sonarr/Radarr webhooks:
560|- Sonarr: Settings → Connect → Webhook → URL: `https://n8n.paccoco.com/webhook/sonarr`
561|- Radarr: Settings → Connect → Webhook → URL: `https://n8n.paccoco.com/webhook/radarr`
562|
563|**Workflow: Tautulli Play Logging**
564|```
565|Trigger: Webhook node (POST from Tautulli on playback start)
566|Step 1: Extract user, media title, player, quality from payload
567|Step 2: Postgres node → INSERT into watch_history table
568|Step 3: (Optional) Gotify notification for specific users/media
569|```
570|
571|Tautulli config: Settings → Notification Agents → Webhook → URL: `https://n8n.paccoco.com/webhook/tautulli-play`
572|
573|**Workflow: Weekly Watch Digest**
574|```
575|Trigger: Cron node (every Sunday at 10:00 AM)
576|Step 1: Postgres node → SELECT watch history for past 7 days
577|Step 2: Format data as structured text
578|Step 3: HTTP Request → Ollama API (POST to PD's qwen2.5:14b)
579| - Prompt: "Summarize this week's viewing in a fun digest: {data}"
580|Step 4: HTTP Request → Gotify (send digest)
581|```
582|
583|#### Infrastructure Monitoring Workflows
584|
585|**Workflow: Uptime Kuma Enhanced Alerts**
586|```
587|Trigger: Webhook node (from Uptime Kuma notification)
588|Step 1: Extract monitor name, status, response time
589|Step 2: HTTP Request → Netdata API for related metrics context
590|Step 3: HTTP Request → Gotify
591| - Title: "🔴 {monitor} DOWN" or "🟢 {monitor} UP"
592| - Message: include Netdata context (CPU, mem, disk at time of alert)
593| - Priority: 8 (high for down, 3 for recovery)
594|```
595|
596|**Workflow: ZFS Pool Utilization Alert**
597|```
598|Trigger: Cron node (every 6 hours)
599|Step 1: SSH node → Serenity: `zpool list -Hp malcolm`
600|Step 2: Parse capacity percentage
601|Step 3: IF capacity > 85% → Gotify alert (priority 8)
602|Step 4: IF capacity > 90% → Gotify alert (priority 10) + Discord webhook
603|```
604|
605|**Workflow: Grafana Alert Remediation**
606|```
607|Trigger: Webhook node (from Grafana alerting)
608|Step 1: Parse alert labels (container, host, metric)
609|Step 2: Switch node → route by alert type:
610| - High CPU container → SSH → docker restart {container}
611| - Disk full → SSH → pause qBittorrent, notify via Gotify
612| - Memory pressure → Gotify alert only (manual intervention)
613|Step 3: Log action taken to Postgres
614|```
615|
616|#### Homelab Ops Workflows
617|
618|**Workflow: Gitea Commit Summary**
619|```
620|Trigger: Webhook node (Gitea webhook on push to truenas-stacks)
621|Step 1: Extract commit messages, author, files changed
622|Step 2: HTTP Request → Ollama API
623| - Prompt: "Summarize this commit in one sentence: {commit_message}"
624|Step 3: HTTP Request → Gotify
625| - Title: "Commit to truenas-stacks"
626| - Message: Ollama-generated summary
627|```
628|
629|Gitea config: Repository → Settings → Webhooks → URL: `https://n8n.paccoco.com/webhook/gitea-push`
630|
631|**Workflow: qBittorrent Auto-Rescan**
632|```
633|Trigger: Webhook or polling (qBittorrent API for completed+moved torrents)
634|Step 1: Determine if file is in Sonarr or Radarr path
635|Step 2: HTTP Request → Sonarr API (POST /command → RescanSeries)
636| OR HTTP Request → Radarr API (POST /command → RescanMovie)
637|Step 3: Gotify notification confirming rescan triggered
638|```
639|
640|#### AI Pipeline Workflows
641|
642|**Workflow: URL Digest Pipeline**
643|```
644|Trigger: Webhook node (POST with URL in body)
645|Step 1: HTTP Request → fetch page content
646|Step 2: Code node → extract text, chunk into ~500 token segments
647|Step 3: HTTP Request → Ollama API → summarize each chunk
648|Step 4: Code node → combine summaries into digest
649|Step 5: Postgres node → store digest with metadata
650|Step 6: Return digest in webhook response
651|```
652|
653|**Workflow: Multi-Model Query Router (simplified by LiteLLM — see Phase 7)**
654|```
655|Trigger: Webhook node (POST with query + complexity hint)
656|Step 1: HTTP Request → LiteLLM (POST http://litellm:4000/v1/chat/completions)
657| - model: complexity parameter ("light", "medium", or "heavy")
658| - LiteLLM handles routing to the correct Ollama instance
659|Step 2: Return response via webhook
660|```
661|> With LiteLLM deployed, the Switch node and three separate Ollama endpoints
662|> are replaced by a single HTTP Request node. The routing logic lives in
663|> LiteLLM's config.yaml instead of n8n workflow logic.
664|
665|**Workflow: Qdrant Index Updater**
666|```
667|Trigger: Webhook node (from Gitea push to any watched repo)
668|Step 1: HTTP Request → Gitea API → fetch changed files content
669|Step 2: Code node → chunk text into embedding-sized segments
670|Step 3: HTTP Request → Ollama API (embed endpoint with nomic-embed-text)
671|Step 4: HTTP Request → Qdrant API (PUT /collections/homelab_docs/points)
672|Step 5: Gotify notification: "{n} documents re-indexed"
673|```
674|
675|#### Home / Business Workflows
676|
677|**Workflow: KitchenOwl Grocery Notification**
678|```
679|Trigger: Polling (KitchenOwl API) or webhook if supported
680|Step 1: Fetch current shopping list items
681|Step 2: Format as clean text list
682|Step 3: HTTP Request → Gotify → phone push notification
683|```
684|
685|**Workflow: Donetick Task Reminder**
686|```
687|Trigger: Cron node (daily at 9:00 AM)
688|Step 1: HTTP Request → Donetick API → fetch tasks due today/overdue
689|Step 2: Format task list
690|Step 3: HTTP Request → Gotify (priority 5)
691|```
692|
693|**Workflow: Long and Low Crafts Order Pipeline**
694|```
695|Trigger: Webhook (Etsy webhook or email trigger via IMAP node)
696|Step 1: Parse order details (item, quantity, customer, shipping)
697|Step 2: HTTP Request → Donetick API → create fulfillment task
698|Step 3: HTTP Request → Gotify DM notification
699| - Title: "New L&L Order"
700| - Message: "{item} x{qty} — ship by {date}"
701| - Priority: 7
702|```
703|
704|---
705|
706|## Phase 4 — Paperless-NGX (Document Intelligence)
707|
708|**Host:** PlausibleDeniability
709|**Stack directory:** `/mnt/docker-ssd/docker/compose/documents/` (new stack)
710|**Why fourth:** Depends on n8n for automation hooks and Gotify for notifications.
711|
712|### 4.1 — Scaffold Directories
713|
714|```bash
715|# Index/data on SSD (search index is write-heavy)
716|sudo mkdir -p /mnt/docker-ssd/docker/appdata/paperless/data
717|
718|# Documents (media) on tank (bulk storage, read-heavy)
719|sudo mkdir -p /mnt/tank/docker/appdata/paperless/media
720|
721|# Consume folder on tank (drop zone for new documents)
722|sudo mkdir -p /mnt/tank/docker/appdata/paperless/consume
723|
724|# Export folder on tank
725|sudo mkdir -p /mnt/tank/docker/appdata/paperless/export
726|
727|# Stack directory
728|sudo mkdir -p /mnt/docker-ssd/docker/compose/documents
729|```
730|
731|### 4.2 — Database Init
732|
733|Paperless uses the existing shared-postgres and shared-redis:
734|
735|```bash
736|docker exec -i shared-postgres psql -U postgres <<'SQL'
737|CREATE USER paperless WITH PASSWORD 'REPLACE_WITH_GENERATED_PASSWORD';
738|CREATE DATABASE paperless OWNER paperless;
739|GRANT ALL PRIVILEGES ON DATABASE paperless TO paperless;
740|SQL
741|```
742|
743|### 4.3 — docker-compose.yaml
744|
745|Create `/mnt/docker-ssd/docker/compose/documents/docker-compose.yaml`:
746|
747|```yaml
748|name: documents
749|
750|services:
751| paperless:
752| image: ghcr.io/paperless-ngx/paperless-ngx:2.16
753| container_name: paperless
754| restart: unless-stopped
755| ports:
756| - "8000:8000"
757| environment:
758| TZ: ${TZ}
759| # Database
760| PAPERLESS_DBENGINE: postgresql
761| PAPERLESS_DBHOST: shared-postgres
762| PAPERLESS_DBPORT: 5432
763| PAPERLESS_DBNAME: ${PAPERLESS_DB_NAME}
764| PAPERLESS_DBUSER: ${PAPERLESS_DB_USER}
765| PAPERLESS_DBPASS: ${PAPERLESS_DB_PASS}
766| # Redis (using shared-redis)
767| PAPERLESS_REDIS: redis://shared-redis:***@10.5.30.7
1063|
1064|# Stack directory
1065|sudo mkdir -p /opt/monitoring
1066|
1067|# Prometheus data on hdd-2 (has more headroom)
1068|sudo mkdir -p /mnt/hdd-2/prometheus-data
1069|sudo chown 65534:65534 /mnt/hdd-2/prometheus-data # nobody user (Prometheus default)
1070|
1071|# Grafana data on hdd-2
1072|sudo mkdir -p /mnt/hdd-2/grafana-data
1073|sudo chown 472:472 /mnt/hdd-2/grafana-data # grafana user
1074|
1075|# Config directories
1076|sudo mkdir -p /opt/monitoring/provisioning/datasources
1077|sudo mkdir -p /opt/monitoring/provisioning/dashboards
1078|```
1079|
1080|### 6.2 — prometheus.yml
1081|
1082|Create `/opt/monitoring/prometheus.yml`:
1083|
1084|```yaml
1085|global:
1086| scrape_interval: 15s
1087| evaluation_interval: 15s
1088|
1089|scrape_configs:
1090| # ------- Local (N.O.M.A.D.) -------
1091| - job_name: 'nomad-node'
1092| static_configs:
1093| - targets: ['node-exporter:9100']
1094| labels:
1095| host: 'nomad'
1096|
1097| # ------- PlausibleDeniability -------
1098| - job_name: 'pd-netdata'
1099| metrics_path: /api/v1/allmetrics
1100| params:
1101| format: [prometheus]
1102| static_configs:
1103| - targets: ['10.5.1.X:19999'] # Replace with PD's IP
1104| labels:
1105| host: 'plausible-deniability'
1106|
1107| # ------- Serenity -------
1108| - job_name: 'serenity-netdata'
1109| metrics_path: /api/v1/allmetrics
1110| params:
1111| format: [prometheus]
1112| static_configs:
1113| - targets: ['10.5.30.5:19999']
1114| labels:
1115| host: 'serenity'
1116|
1117| # ------- Gotify -------
1118| # Gotify exposes /health but no Prometheus endpoint natively.
1119| # Use blackbox exporter or just rely on Uptime Kuma.
1120|
1121| # ------- n8n -------
1122| # n8n doesn't expose Prometheus metrics by default.
1123| # Monitor via container resource metrics from Netdata.
1124|
1125| # -------- Add more targets as needed --------
1126| # When node-exporter is installed on PD and Serenity:
1127| # - job_name: 'pd-node'
1128| # static_configs:
1129| # - targets: ['10.5.1.X:9100']
1130| # labels:
1131| # host: 'plausible-deniability'
1132| #
1133| # - job_name: 'serenity-node'
1134| # static_configs:
1135| # - targets: ['10.5.30.5:9100']
1136| # labels:
1137| # host: 'serenity'
1138|```
1139|
1140|> **Netdata as a Prometheus target:** Both PD and Serenity already run Netdata. Netdata has a built-in Prometheus exporter at `/api/v1/allmetrics?format=prometheus`. This gives you CPU, memory, disk, network, and ZFS metrics without installing node-exporter on those hosts.
1141|
1142|### 6.3 — Grafana Provisioning
1143|
1144|Create `/opt/monitoring/provisioning/datasources/prometheus.yml`:
1145|
1146|```yaml
1147|apiVersion: 1
1148|
1149|datasources:
1150| - name: Prometheus
1151| type: prometheus
1152| access: proxy
1153| url: http://prometheus:9090
1154| isDefault: true
1155| editable: true
1156|```
1157|
1158|Create `/opt/monitoring/provisioning/dashboards/dashboards.yml`:
1159|
1160|```yaml
1161|apiVersion: 1
1162|
1163|providers:
1164| - name: 'default'
1165| orgId: 1
1166| folder: ''
1167| type: file
1168| disableDeletion: false
1169| editable: true
1170| options:
1171| path: /var/lib/grafana/dashboards
1172| foldersFromFilesStructure: false
1173|```
1174|
1175|### 6.4 — docker-compose.yaml
1176|
1177|Create `/opt/monitoring/docker-compose.yaml`:
1178|
1179|```yaml
1180|name: monitoring
1181|
1182|services:
1183| prometheus:
1184| image: prom/prometheus:v3.4.0
1185| container_name: prometheus
1186| restart: unless-stopped
1187| ports:
1188| - "9090:9090"
1189| command:
1190| - '--config.file=/etc/prometheus/prometheus.yml'
1191| - '--storage.tsdb.path=/prometheus'
1192| - '--storage.tsdb.retention.time=90d'
1193| - '--web.enable-lifecycle'
1194| volumes:
1195| - /opt/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
1196| - /mnt/hdd-2/prometheus-data:/prometheus
1197| networks:
1198| - monitoring
1199|
1200| grafana:
1201| image: grafana/grafana:13.0.1
1202| container_name: grafana
1203| restart: unless-stopped
1204| ports:
1205| - "3000:3000"
1206| environment:
1207| TZ: ${TZ}
1208| GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER}
1209| GF_SECURITY_ADMIN_PASSWORD: ${GF_ADMIN_PASS}
1210| GF_SERVER_ROOT_URL: https://${GF_HOST}/
1211| volumes:
1212| - /mnt/hdd-2/grafana-data:/var/lib/grafana
1213| - /opt/monitoring/provisioning:/etc/grafana/provisioning:ro
1214| networks:
1215| - monitoring
1216|
1217| node-exporter:
1218| image: prom/node-exporter:v1.9.0
1219| container_name: node-exporter
1220| restart: unless-stopped
1221| ports:
1222| - "9100:9100"
1223| command:
1224| - '--path.procfs=/host/proc'
1225| - '--path.sysfs=/host/sys'
1226| - '--path.rootfs=/rootfs'
1227| - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
1228| volumes:
1229| - /proc:/host/proc:ro
1230| - /sys:/host/sys:ro
1231| - /:/rootfs:ro
1232| networks:
1233| - monitoring
1234|
1235|networks:
1236| monitoring:
1237| driver: bridge
1238|```
1239|
1240|> **Image notes:**
1241|> - `prom/prometheus:v3.4.0` — Prometheus v3 is the current major line. **Do not use `:latest`** — it still resolves to 2.x due to a known tagging issue. Always use an explicit `v3.x.y` tag.
1242|> - `grafana/grafana:latest` — use the moving stable image by default here; only pin Grafana if a breakage or compatibility reason is documented.
1243|> - `prom/node-exporter:v1.9.0` — check [releases](https://github.com/prometheus/node_exporter/releases).
1244|
1245|### 6.5 — .env.example
1246|
1247|```bash
1248|# /opt/monitoring/.env.example
1249|
1250|TZ=America/New_York
1251|
1252|# Grafana
1253|GF_ADMIN_USER=admin
1254|GF_ADMIN_PASS=CHANGE_ME
1255|GF_HOST=grafana.paccoco.com
1256|```
1257|
1258|### 6.6 — Pangolin Configuration
1259|
1260|Grafana runs on N.O.M.A.D., not PD where the main Newt agent lives. You have two options:
1261|
1262|**Option A — Route via N.O.M.A.D.'s existing Newt (recommended if already connected)**
1263|
1264|N.O.M.A.D. already has a Newt container from the Project N.O.M.A.D. setup. If it's connected to your Pangolin VPS, just add a new resource in the Pangolin dashboard pointing to `http://grafana:3000` or `http://10.5.30.7:3000`.
1265|
1266|**Option B — Add a dedicated Newt to the monitoring stack**
1267|
1268|If N.O.M.A.D.'s existing Newt is not connected to Pangolin (or is a separate Pangolin instance), add Newt to the monitoring compose:
1269|
1270|```yaml
1271| newt:
1272| image: ghcr.io/fosrl/newt:latest
1273| container_name: monitoring-newt
1274| restart: unless-stopped
1275| environment:
1276| PANGOLIN_ENDPOINT: ${PANGOLIN_ENDPOINT}
1277| NEWT_ID: ${NEWT_ID}
1278| NEWT_SECRET: ${NEWT_SECRET}
1279| networks:
1280| - monitoring
1281|```
1282|
1283|Then in Pangolin dashboard:
1284|
1285|| Field | Value |
1286||-------|-------|
1287|| Domain | `grafana.paccoco.com` |
1288|| Scheme | `http` |
1289|| Host | `grafana` (Docker DNS via shared network) |
1290|| Port | `3000` |
1291|
1292|**Option C — Direct IP routing (simplest, no Newt needed)**
1293|
1294|If Pangolin's Newt on PD can reach N.O.M.A.D. by LAN IP (they're on the same subnet):
1295|
1296|| Field | Value |
1297||-------|-------|
1298|| Domain | `grafana.paccoco.com` |
1299|| Scheme | `http` |
1300|| Host | `10.5.30.7` |
1301|| Port | `3000` |
1302|
1303|> **Decision point:** Check if N.O.M.A.D.'s existing Newt is connected to your Pangolin instance before deploying. Run `docker ps --filter name=newt` on N.O.M.A.D. to verify. If it's running and connected, Option A is zero-effort. If not, Option C is the simplest fallback.
1304|
1305|### 6.7 — Validate & Deploy
1306|
1307|```bash
1308|ssh nomad@10.5.30.7
1309|
1310|cd /opt/monitoring
1311|cp .env.example .env
1312|nano .env # Fill in real values
1313|
1314|docker compose --env-file .env config
1315|docker compose --env-file .env up -d
1316|```
1317|
1318|### 6.8 — Post-Deploy Verification
1319|
1320|```bash
1321|# All three containers running?
1322|docker ps --filter name=prometheus --filter name=grafana --filter name=node-exporter
1323|
1324|# Prometheus scraping targets?
1325|curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | head -40
1326|
1327|# Grafana UI accessible?
1328|curl -s -o /dev/null -w "%{http_code}" http://localhost:3000/
1329|# Expected: 200 or 302
1330|
1331|# Node exporter metrics flowing?
1332|curl -s http://localhost:9100/metrics | head -10
1333|```
1334|
1335|### 6.9 — Recommended Dashboards
1336|
1337|Import these from [Grafana Dashboard Library](https://grafana.com/grafana/dashboards/):
1338|
1339|| Dashboard | ID | Purpose |
1340||-----------|----|---------|
1341|| Node Exporter Full | 1860 | System metrics for N.O.M.A.D. |
1342|| Docker Container Stats | 893 | Container resource usage |
1343|| Netdata via Prometheus | (search) | PD and Serenity system metrics |
1344|
1345|To import: Grafana → Dashboards → New → Import → Enter dashboard ID.
1346|
1347|### 6.10 — Metrics Targets Summary
1348|
1349|| Target | Host | Method | Endpoint |
1350||--------|------|--------|----------|
1351|| N.O.M.A.D. system | localhost | node-exporter | node-exporter:9100 |
1352|| PD system | 10.5.1.X | Netdata Prometheus | 10.5.1.X:19999/api/v1/allmetrics |
1353|| Serenity system | 10.5.30.5 | Netdata Prometheus | 10.5.30.5:19999/api/v1/allmetrics |
1354|| ZFS pools | via Netdata | Netdata exports ZFS metrics | Included in Netdata scrape |
1355|| Container stats | via Netdata | Netdata exports cgroup metrics | Included in Netdata scrape |
1356|| Plex streams | Tautulli | n8n polling → Postgres | Via n8n workflow (Phase 3) |
1357|| qBit stats | qBittorrent API | n8n polling → Postgres | Via n8n workflow (Phase 3) |
1358|| Tailscale latency | Tailscale API | n8n polling → Postgres | Via n8n workflow (Phase 3) |
1359|
1360|### 6.11 — n8n Integration (Grafana → n8n alert webhook)
1361|
1362|In Grafana → Alerting → Contact Points, create a webhook contact point:
1363|
1364|| Field | Value |
1365||-------|-------|
1366|| Name | `n8n-alerts` |
1367|| Type | Webhook |
1368|| URL | `https://n8n.paccoco.com/webhook/grafana-alert` |
1369|| HTTP Method | POST |
1370|
1371|Then create alert rules for:
1372|- ZFS pool utilization > 85%
1373|- Container memory > 90% of limit
1374|- Host CPU sustained > 90% for 5 minutes
1375|- Disk I/O latency spikes
1376|
1377|These fire into the "Grafana Alert Remediation" n8n workflow (see Phase 3).
1378|
1379|---
1380|
1381|## Phase 7 — LiteLLM (AI Gateway)
1382|
1383|**Host:** PlausibleDeniability
1384|**Stack directory:** `/mnt/docker-ssd/docker/compose/ai/` (same stack as Qdrant)
1385|**Why:** Replaces manual multi-model routing with a unified OpenAI-compatible API. Every app (OpenWebUI, Continue.dev, n8n, Paperless summarization) points at one endpoint instead of juggling three Ollama IPs.
1386|
1387|### 7.1 — What LiteLLM Does
1388|
1389|LiteLLM sits in front of your three Ollama instances and presents a single OpenAI-compatible API at `http://litellm:4000`. It handles:
1390|
1391|- **Model routing** — requests for `qwen3:32b` go to ROCINANTE, `qwen2.5:14b` goes to PD, `phi4` goes to N.O.M.A.D.
1392|- **Failover** — if ROCINANTE is offline, LiteLLM can fall back to PD automatically
1393|- **Load balancing** — distribute requests across instances running the same model
1394|- **Usage tracking** — logs token counts, latency, and costs per model/user via its built-in database
1395|
1396|### 7.2 — Scaffold Directories
1397|
1398|```bash
1399|# Config on SSD (SQLite DB for usage tracking)
1400|sudo mkdir -p /mnt/docker-ssd/docker/appdata/litellm
1401|
1402|# LiteLLM config file
1403|sudo mkdir -p /mnt/docker-ssd/docker/compose/ai/litellm
1404|```
1405|
1406|### 7.3 — LiteLLM Config
1407|
1408|Create `/mnt/docker-ssd/docker/compose/ai/litellm/config.yaml`:
1409|
1410|```yaml
1411|model_list:
1412| # ---- ROCINANTE (RTX 4090, 24GB) — heavy reasoning ----
1413| - model_name: "heavy"
1414| litellm_params:
1415| model: "ollama/qwen3:32b"
1416| api_base: "http://10.5.1.ROCINANTE:11434"
1417| timeout: 300
1418| stream_timeout: 300
1419| model_info:
1420| description: "Heavy reasoning, long context, complex code"
1421|
1422| - model_name: "heavy"
1423| litellm_params:
1424| model: "ollama/deepseek-r1:32b"
1425| api_base: "http://10.5.1.ROCINANTE:11434"
1426| timeout: 300
1427| stream_timeout: 300
1428| model_info:
1429| description: "Deep reasoning fallback on ROCINANTE"
1430|
1431| # ---- PlausibleDeniability (RTX 2080 Ti, 11GB) — general ----
1432| - model_name: "medium"
1433| litellm_params:
1434| model: "ollama/qwen2.5:14b"
1435| api_base: "http://host.docker.internal:11434"
1436| timeout: 120
1437| stream_timeout: 120
1438| model_info:
1439| description: "General homelab assistant, RAG queries"
1440|
1441| # ---- N.O.M.A.D. (GTX 1080, 8GB) — lightweight ----
1442| - model_name: "light"
1443| litellm_params:
1444| model: "ollama/phi4"
1445| api_base: "http://10.5.30.7:11434"
1446| timeout: 60
1447| stream_timeout: 60
1448| model_info:
1449| description: "Fast local inference, lightweight tasks"
1450|
1451| - model_name: "light"
1452| litellm_params:
1453| model: "ollama/llama3.2:3b"
1454| api_base: "http://10.5.30.7:11434"
1455| timeout: 60
1456| stream_timeout: 60
1457| model_info:
1458| description: "Ultra-light fallback on N.O.M.A.D."
1459|
1460| # ---- Embeddings ----
1461| - model_name: "embed"
1462| litellm_params:
1463| model: "ollama/nomic-embed-text"
1464| api_base: "http://host.docker.internal:11434"
1465| model_info:
1466| description: "Text embeddings for RAG pipeline"
1467|
1468| # ---- Direct model access (bypass routing) ----
1469| # These let you request a specific model by its full name
1470| - model_name: "ollama/qwen3:32b"
1471| litellm_params:
1472| model: "ollama/qwen3:32b"
1473| api_base: "http://10.5.1.ROCINANTE:11434"
1474|
1475| - model_name: "ollama/qwen2.5:14b"
1476| litellm_params:
1477| model: "ollama/qwen2.5:14b"
1478| api_base: "http://host.docker.internal:11434"
1479|
1480| - model_name: "ollama/phi4"
1481| litellm_params:
1482| model: "ollama/phi4"
1483| api_base: "http://10.5.30.7:11434"
1484|
1485|litellm_settings:
1486| drop_params: true
1487| set_verbose: false
1488| request_timeout: 300
1489| num_retries: 2
1490| retry_after: 5
1491| allowed_fails: 3
1492| cooldown_time: 60
1493|
1494|general_settings:
1495| master_key: "os.environ/LITELLM_MASTER_KEY"
1496|```
1497|
1498|> **DEPLOYED NOTE (2026-05-05):** Do NOT add `database_url` to general_settings — newer LiteLLM versions include Prisma ORM that requires PostgreSQL, and adding any database_url causes a crash loop. Without it, LiteLLM runs in config-only mode which is fine for homelab use. The `master_key` uses `os.environ/LITELLM_MASTER_KEY` syntax to read from the container's environment variable (set via .env → docker compose).
1499|
1500|> **Routing explained:** Multiple entries with the same `model_name` (like "heavy") enable load balancing and failover within that tier. When you request model `heavy`, LiteLLM picks the healthiest deployment. The direct-access entries (like `ollama/qwen3:32b`) let you bypass the tier system and target a specific model when needed.
1501|
1502|> **Replace IPs:** Update `10.5.1.ROCINANTE` with ROCINANTE's actual LAN or Tailscale IP. PD uses `host.docker.internal` since LiteLLM runs on PD alongside Ollama.
1503|
1504|### 7.4 — docker-compose.yaml (updated ai stack)
1505|
1506|Update `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml` to add LiteLLM:
1507|
1508|```yaml
1509|name: ai
1510|
1511|services:
1512| qdrant:
1513| image: qdrant/qdrant:v1.14.0
1514| container_name: qdrant
1515| restart: unless-stopped
1516| ports:
1517| - "6333:6333" # REST API
1518| - "6334:6334" # gRPC
1519| environment:
1520| TZ: ${TZ}
1521| QDRANT__SERVICE__GRPC_PORT: 6334
1522| QDRANT__STORAGE__STORAGE_PATH: /qdrant/storage
1523| QDRANT__STORAGE__SNAPSHOTS_PATH: /qdrant/snapshots
1524| volumes:
1525| - /mnt/docker-ssd/docker/appdata/qdrant/storage:/qdrant/storage
1526| - /mnt/docker-ssd/docker/appdata/qdrant/snapshots:/qdrant/snapshots
1527| networks:
1528| - ai-services
1529| - default
1530|
1531| litellm:
1532| image: ghcr.io/berriai/litellm:main-latest
1533| container_name: litellm
1534| restart: unless-stopped
1535| ports:
1536| - "4000:4000"
1537| environment:
1538| TZ: ${TZ}
1539| LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
1540| volumes:
1541| - /mnt/docker-ssd/docker/compose/ai/litellm/config.yaml:/app/config.yaml:ro
1542| - /mnt/docker-ssd/docker/appdata/litellm:/app/data
1543| command: ["--config", "/app/config.yaml", "--port", "4000"]
1544| extra_hosts:
1545| - "host.docker.internal:host-gateway"
1546| networks:
1547| - ai-services
1548| - default
1549|
1550| reranker:
1551| image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
1552| container_name: reranker
1553| restart: unless-stopped
1554| ports:
1555| - "8787:80"
1556| environment:
1557| MODEL_ID: ${RERANKER_MODEL}
1558| volumes:
1559| - /mnt/docker-ssd/docker/appdata/reranker:/data
1560| networks:
1561| - ai-services
1562| - default
1563|
1564| whisper:
1565| image: fedirz/faster-whisper-server:latest-cuda
1566| container_name: whisper
1567| restart: unless-stopped
1568| ports:
1569| - "8786:8000"
1570| environment:
1571| TZ: ${TZ}
1572| WHISPER__MODEL: ${WHISPER_MODEL}
1573| WHISPER__DEVICE: cuda
1574| WHISPER__COMPUTE_TYPE: float16
1575| volumes:
1576| - /mnt/docker-ssd/docker/appdata/whisper:/root/.cache/huggingface
1577| deploy:
1578| resources:
1579| reservations:
1580| devices:
1581| - driver: nvidia
1582| count: 1
1583| capabilities: [gpu]
1584| networks:
1585| - ai-services
1586| - default
1587|
1588| # -------------------------------------------------------
1589| # Ollama and OpenWebUI go here when deployed.
1590| # They share this stack and the default + ai-services networks.
1591| # OpenWebUI → Qdrant at http://qdrant:6333
1592| # OpenWebUI → LiteLLM at http://litellm:4000 (or direct Ollama)
1593| # -------------------------------------------------------
1594|
1595| # ollama:
1596| # image: ollama/ollama:latest
1597| # container_name: ollama
1598| # ...
1599|
1600| # openwebui:
1601| # image: ghcr.io/open-webui/open-webui:main
1602| # container_name: openwebui
1603| # ...
1604|
1605|networks:
1606| ai-services:
1607| name: ai-services
1608|```
1609|
1610|> **GPU sharing note:** On PD, the RTX 2080 Ti is shared between Plex (hardware transcoding), Immich ML, and Ollama. faster-whisper will also need the GPU when processing audio. These workloads are bursty (not constant), so time-sharing the GPU is viable — but be aware that a large Whisper transcription job will temporarily impact Ollama inference latency. If this becomes an issue, consider running Whisper on N.O.M.A.D.'s GTX 1080 instead and change `whisper`'s Ollama-style routing accordingly.
1611|
1612|> **Reranker on CPU:** The reranker uses the `cpu-latest` image variant intentionally — do NOT pin to `cpu-1.5` as it has an hf-hub compatibility bug. Reranking is a lightweight operation (scoring ~50 chunks takes <1s on CPU) and doesn't justify competing for GPU VRAM. The model loads via safetensors fallback since no ONNX files exist for bge-reranker-v2-m3. Max batch size is 4 on CPU. If you find latency is an issue, a GPU variant exists (`ghcr.io/huggingface/text-embeddings-inference:turing-latest` for your 2080 Ti).
1613|
1614|### 7.5 — .env.example (updated ai stack)
1615|
1616|```bash
1617|# /mnt/docker-ssd/docker/compose/ai/.env.example
1618|
1619|TZ=America/New_York
1620|
1621|# LiteLLM
1622|LITELLM_MASTER_KEY=sk-CHANGE_ME
1623|
1624|# Reranker
1625|RERANKER_MODEL=BAAI/bge-reranker-v2-m3
1626|
1627|# Whisper
1628|WHISPER_MODEL=Systran/faster-distil-whisper-large-v3
1629|```
1630|
1631|### 7.6 — Validate & Deploy
1632|
1633|```bash
1634|cd /mnt/docker-ssd/docker/compose/ai
1635|cp .env.example .env
1636|nano .env # Set LITELLM_MASTER_KEY=$(openssl rand -hex 32)
1637|
1638|docker compose --env-file .env config
1639|docker compose --env-file .env up -d
1640|```
1641|
1642|### 7.7 — Post-Deploy Verification
1643|
1644|```bash
1645|# All containers running?
1646|docker ps --filter name=qdrant --filter name=litellm --filter name=reranker --filter name=whisper
1647|
1648|# LiteLLM health?
1649|curl -s http://localhost:4000/health
1650|
1651|# LiteLLM can see all models?
1652|curl -s http://localhost:4000/v1/models \
1653| -H "Authorization: Bearer *** | python3 -m json.tool
1654|
1655|# Test a chat completion through LiteLLM
1656|curl -s http://localhost:4000/v1/chat/completions \
1657| -H "Authorization: Bearer *** \
1658| -H "Content-Type: application/json" \
1659| -d '{
1660| "model": "medium",
1661| "messages": [{"role": "user", "content": "Hello, which model are you?"}]
1662| }' | python3 -m json.tool
1663|
1664|# Test embeddings through LiteLLM
1665|curl -s http://localhost:4000/v1/embeddings \
1666| -H "Authorization: Bearer *** \
1667| -H "Content-Type: application/json" \
1668| -d '{
1669| "model": "embed",
1670| "input": "test embedding"
1671| }' | python3 -m json.tool
1672|```
1673|
1674|### 7.8 — Integration Updates
1675|
1676|With LiteLLM deployed, all apps should point at `http://litellm:4000` (within the `ai-services` network) or `http://PD_IP:4000` (from external hosts) instead of direct Ollama endpoints.
1677|
1678|**OpenWebUI:** Settings → Connections → add OpenAI-compatible endpoint:
1679|- URL: `http://litellm:4000/v1`
1680|- API Key: your `LITELLM_MASTER_KEY`
1681|- This gives OpenWebUI access to all models across all three machines through one connection
1682|
1683|**Continue.dev:** Update `~/.continue/config.json`:
1684|```json
1685|{
1686| "models": [
1687| {
1688| "title": "Heavy (ROCINANTE)",
1689| "provider": "openai",
1690| "model": "heavy",
1691| "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
1692| "apiKey": "YOUR_MASTER_KEY"
1693| },
1694| {
1695| "title": "Medium (PD)",
1696| "provider": "openai",
1697| "model": "medium",
1698| "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
1699| "apiKey": "YOUR_MASTER_KEY"
1700| }
1701| ],
1702| "tabAutocompleteModel": {
1703| "title": "Light (N.O.M.A.D.)",
1704| "provider": "openai",
1705| "model": "light",
1706| "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
1707| "apiKey": "YOUR_MASTER_KEY"
1708| }
1709|}
1710|```
1711|
1712|> **Key change:** Continue.dev now uses `provider: "openai"` instead of `provider: "ollama"` and points at LiteLLM. One IP to remember, and if you add or move models between machines, you only update `config.yaml` — not every app.
1713|
1714|**n8n workflows:** All HTTP Request nodes that call Ollama directly should be updated:
1715|- Old: `http://10.5.30.7:11434/api/generate` (N.O.M.A.D.)
1716|- New: `http://litellm:4000/v1/chat/completions` with `model: "light"`
1717|- n8n is on the `ai-services` network, so it reaches LiteLLM by Docker DNS
1718|
1719|**n8n Multi-Model Query Router workflow:** This workflow is now **simplified dramatically** — instead of a Switch node routing to three different Ollama IPs, it becomes a single HTTP Request node that passes the model tier name (`light`, `medium`, `heavy`) to LiteLLM and lets the gateway handle routing. The Switch node logic can be removed entirely.
1720|
1721|---
1722|
1723|## Phase 8 — Reranker (RAG Quality)
1724|
1725|**Host:** PlausibleDeniability (deployed as part of the `ai` stack in Phase 7)
1726|**Service:** `reranker` container (already in the compose above)
1727|**Why:** Dramatically improves RAG answer quality by filtering out noisy retrieval results before they reach the LLM.
1728|
1729|### 8.1 — How Reranking Works
1730|
1731|Without a reranker, your RAG pipeline does:
1732|```
1733|Query → embed → Qdrant top-10 by vector similarity → all 10 chunks go to LLM
1734|```
1735|
1736|The problem: vector similarity often returns "close but irrelevant" chunks. The LLM gets noisy context and hallucinates.
1737|
1738|With a reranker:
1739|```
1740|Query → embed → Qdrant top-50 by vector similarity → reranker scores each (query, chunk) pair → top-5 by relevance go to LLM
1741|```
1742|
1743|The reranker is a cross-encoder that reads the full query AND each chunk together, producing a much more accurate relevance score than vector distance alone. It over-retrieves cheaply from Qdrant, then precisely filters.
1744|
1745|### 8.2 — Scaffold Directories
1746|
1747|```bash
1748|# Model cache on SSD (reranker model is ~1.1GB, downloaded on first start)
1749|sudo mkdir -p /mnt/docker-ssd/docker/appdata/reranker
1750|```
1751|
1752|### 8.3 — Post-Deploy Verification
1753|
1754|```bash
1755|# Container running?
1756|docker ps --filter name=reranker
1757|
1758|# Health check?
1759|curl -s http://localhost:8787/health
1760|
1761|# Test reranking
1762|curl -s http://localhost:8787/rerank \
1763| -H "Content-Type: application/json" \
1764| -d '{
1765| "query": "How do I restart a Docker container?",
1766| "texts": [
1767| "Use docker restart <container_name> to restart a running container.",
1768| "Docker was founded in 2013 by Solomon Hykes.",
1769| "The docker compose down command stops and removes containers.",
1770| "Kubernetes pods can be restarted by deleting them."
1771| ]
1772| }' | python3 -m json.tool
1773|# Expected: the first text scores highest, second scores lowest
1774|```
1775|
1776|### 8.4 — n8n RAG Pipeline Integration
1777|
1778|Update the "Qdrant Index Updater" and any RAG query workflows to include a reranking step.
1779|
1780|**Updated RAG Query Workflow (for OpenWebUI or any n8n-based query):**
1781|```
1782|Trigger: Webhook node (POST with query)
1783|Step 1: HTTP Request → LiteLLM /v1/embeddings
1784| - model: "embed"
1785| - input: query text
1786|Step 2: HTTP Request → Qdrant API (POST /collections/{name}/points/search)
1787| - vector: embedding from step 1
1788| - limit: 50 (over-retrieve)
1789|Step 3: HTTP Request → Reranker (POST http://reranker:80/rerank)
1790| - query: original query text
1791| - texts: array of 50 chunk texts from Qdrant results
1792|Step 4: Code node → take top 5 by reranker score, format as context
1793|Step 5: HTTP Request → LiteLLM /v1/chat/completions
1794| - model: "medium" (or "heavy" for complex queries)
1795| - messages: system prompt with top-5 context + user query
1796|Step 6: Return response via webhook
1797|```
1798|
1799|> **Key difference from the original plan:** Step 2 now retrieves 50 results instead of 10, and Step 3 (reranking) filters down to the best 5. This "over-retrieve then rerank" pattern is the standard approach for production RAG systems.
1800|
1801|---
1802|
1803|## Phase 9 — faster-whisper (Speech-to-Text)
1804|
1805|**Host:** PlausibleDeniability (deployed as part of the `ai` stack in Phase 7)
1806|**Service:** `whisper` container (already in the compose above)
1807|**Why:** Replaces shelved Scriberr with an OpenAI-compatible STT API. No SQLite dependency, no ZFS/ACL issues.
1808|
1809|### 9.1 — Scaffold Directories
1810|
1811|```bash
1812|# Model cache on SSD (Whisper models are 1-3GB)
1813|sudo mkdir -p /mnt/docker-ssd/docker/appdata/whisper
1814|```
1815|
1816|### 9.2 — Post-Deploy Verification
1817|
1818|```bash
1819|# Container running?
1820|docker ps --filter name=whisper
1821|
1822|# Health check?
1823|curl -s http://localhost:8786/health
1824|
1825|# Test transcription with a sample audio file
1826|curl -s http://localhost:8786/v1/audio/transcriptions \
1827| -F "file=@/path/to/test-audio.wav" \
1828| -F "model=Systran/faster-distil-whisper-large-v3" \
1829| | python3 -m json.tool
1830|```
1831|
1832|### 9.3 — n8n Integration Workflows
1833|
1834|**Workflow: Voice Note → Text → Summary**
1835|```
1836|Trigger: Webhook node (POST with audio file in body)
1837|Step 1: HTTP Request → Whisper (POST http://whisper:8000/v1/audio/transcriptions)
1838| - Multipart form with audio file
1839| - response_format: "json"
1840|Step 2: HTTP Request → LiteLLM /v1/chat/completions
1841| - model: "medium"
1842| - Prompt: "Summarize this voice note concisely: {transcript}"
1843|Step 3: HTTP Request → Gotify → push summary to phone
1844|Step 4: (Optional) Postgres node → log transcript and summary
1845|```
1846|
1847|**Workflow: Audio File → Paperless Document**
1848|```
1849|Trigger: Webhook or filesystem watcher
1850|Step 1: HTTP Request → Whisper → get transcript
1851|Step 2: Code node → format transcript as text document
1852|Step 3: HTTP Request → Paperless API (POST /api/documents/post_document/)
1853| - Upload transcript as .txt
1854| - Tag: "transcription"
1855|Step 4: Gotify notification: "Audio transcribed and filed: {title}"
1856|```
1857|
1858|**Home Assistant Voice Integration:**
1859|If you want HA voice commands, Whisper can serve as the STT backend:
1860|1. In Home Assistant → Settings → Voice Assistants
1861|2. Add speech-to-text provider: "Whisper" at `http://PD_IP:8786`
1862|3. Pair with Piper TTS (future addition) for full voice assistant loop
1863|
1864|### 9.4 — GPU vs CPU Considerations
1865|
1866|The compose file above uses the GPU variant with CUDA. If GPU contention with Ollama becomes an issue:
1867|
1868|**Option A — CPU fallback on PD:**
1869|Change the image from `latest-cuda` to `fedirz/faster-whisper-server:latest-cpu` and remove the `deploy.resources` block and the CUDA environment variables (`WHISPER__DEVICE`, `WHISPER__COMPUTE_TYPE`). Transcription will be slower (~4x real-time instead of ~20x) but won't compete for VRAM.
1870|
1871|**Option B — Move to N.O.M.A.D.:**
1872|Run Whisper on N.O.M.A.D.'s GTX 1080 which is less contested. Add it to N.O.M.A.D.'s compose or run standalone. N.O.M.A.D.'s 8GB VRAM handles Whisper models comfortably since phi4 doesn't fill it.
1873|
1874|---
1875|
1876|## Additional Tools Setup
1877|
1878|### Continue.dev (Local AI Code Completion)
1879|
1880|1. Install the Continue extension in VS Code
1881|2. Create/edit `~/.continue/config.json`:
1882|
1883|**With LiteLLM (recommended — see Phase 7):**
1884|```json
1885|{
1886| "models": [
1887| {
1888| "title": "Heavy (ROCINANTE)",
1889| "provider": "openai",
1890| "model": "heavy",
1891| "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
1892| "apiKey": "YOUR_LITELLM_MASTER_KEY"
1893| },
1894| {
1895| "title": "Medium (PD)",
1896| "provider": "openai",
1897| "model": "medium",
1898| "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
1899| "apiKey": "YOUR_LITELLM_MASTER_KEY"
1900| }
1901| ],
1902| "tabAutocompleteModel": {
1903| "title": "Light (N.O.M.A.D.)",
1904| "provider": "openai",
1905| "model": "light",
1906| "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
1907| "apiKey": "YOUR_LITELLM_MASTER_KEY"
1908| }
1909|}
1910|```
1911|
1912|> One IP, one API key, all three machines. If you add or move models, update LiteLLM's `config.yaml` — not every app.
1913|
1914|**Without LiteLLM (direct Ollama, if Phase 7 is not yet deployed):**
1915|```json
1916|{
1917| "models": [
1918| {
1919| "title": "PD - qwen2.5:14b",
1920| "provider": "ollama",
1921| "model": "qwen2.5:14b",
1922| "apiBase": "http://PD_TAILSCALE_IP:11434"
1923| },
1924| {
1925| "title": "ROCINANTE - qwen3:32b",
1926| "provider": "ollama",
1927| "model": "qwen3:32b",
1928| "apiBase": "http://ROCINANTE_TAILSCALE_IP:11434"
1929| }
1930| ],
1931| "tabAutocompleteModel": {
1932| "title": "N.O.M.A.D. - phi4",
1933| "provider": "ollama",
1934| "model": "phi4",
1935| "apiBase": "http://NOMAD_TAILSCALE_IP:11434"
1936| }
1937|}
1938|```
1939|
1940|### Obsidian + Gitea Sync
1941|
1942|1. In Obsidian, install the "Obsidian Git" community plugin
1943|2. Initialize your vault as a git repo:
1944| ```bash
1945| cd /path/to/obsidian/vault
1946| git init
1947| git remote add origin http://PD_IP:3000/fizzlepoof/obsidian-vault.git
1948| ```
1949|3. In Obsidian Git settings:
1950| - Auto backup interval: 10 minutes
1951| - Pull on startup: enabled
1952|4. Create the repo in Gitea first: `http://PD_IP:3000` → New Repository → `obsidian-vault`
1953|5. Add a Gitea webhook to trigger the n8n "Qdrant Index Updater" workflow (see Phase 3)
1954|
1955|### Homepage Ollama Widget
1956|
1957|Add to your Homepage configuration (`/mnt/tank/docker/appdata/homepage/services.yaml`):
1958|
1959|```yaml
1960|- AI:
1961| - Ollama (PD):
1962| icon: ollama.svg
1963| href: http://PD_IP:11434
1964| widget:
1965| type: ollama
1966| url: http://PD_IP:11434
1967| - Ollama (ROCINANTE):
1968| icon: ollama.svg
1969| href: http://ROCINANTE_IP:11434
1970| widget:
1971| type: ollama
1972| url: http://ROCINANTE_IP:11434
1973|```
1974|
1975|---
1976|
1977|## Deployment Order Summary
1978|
1979|```
1980|Week 1: Phase 1 — Gotify
1981| └─ Deploy, create app tokens, install phone app
1982| └─ Test: send manual notification via API
1983|
1984|Week 1: Phase 2 — Qdrant
1985| └─ Deploy, verify REST API
1986| └─ Create initial collections (empty, ready for n8n)
1987|
1988|Week 2: Phase 3 — n8n
1989| └─ Deploy, create admin account
1990| └─ Build workflows incrementally:
1991| Day 1: Gitea commit → Gotify (simplest, proves the pipeline)
1992| Day 2: Sonarr/Radarr → TMDB → Gotify + Discord
1993| Day 3: Tautulli play logging + weekly digest
1994| Day 4: Uptime Kuma enhanced alerts
1995| Day 5: ZFS pool monitoring
1996| Day 6: Multi-model query router
1997| Day 7: Qdrant index updater
1998|
1999|Week 3: Phase 4 — Paperless-NGX
2000| └─ Deploy, ingest test documents
2001|
Reference in New Issue View Git Blame Copy Permalink