1712 lines
66 KiB
Markdown
1712 lines
66 KiB
Markdown
1|# Homelab Expansion — Full Buildout Plan
|
|
2|
|
|
3|> Generated: 2026-05-04
|
|
4|> Based on: HOMELAB_EXPANSION_PLAN.md, ARCHITECTURE_OVERVIEW.md, STACK_STANDARDS.md
|
|
5|
|
|
6|This document contains everything needed to deploy all six expansion phases — docker-compose files, .env templates, directory scaffolding commands, database init SQL, Pangolin configuration, and validation steps. Each phase follows the established stack standards and deployment checklist.
|
|
7|
|
|
8|---
|
|
9|
|
|
10|## Table of Contents
|
|
11|
|
|
12|1. [Prerequisites & Conventions](#prerequisites--conventions)
|
|
13|2. [Phase 1 — Gotify (Notifications)](#phase-1--gotify-notifications)
|
|
14|3. [Phase 2 — Qdrant (Vector Database)](#phase-2--qdrant-vector-database)
|
|
15|4. [Phase 3 — n8n (Workflow Automation)](#phase-3--n8n-workflow-automation)
|
|
16|5. [Phase 4 — Paperless-NGX (Document Intelligence)](#phase-4--paperless-ngx-document-intelligence)
|
|
17|6. [Phase 5 — Home Assistant (Home Automation)](#phase-5--home-assistant-home-automation)
|
|
18|7. [Phase 6 — Grafana + Prometheus (Observability)](#phase-6--grafana--prometheus-observability)
|
|
19|8. [Phase 7 — LiteLLM (AI Gateway)](#phase-7--litellm-ai-gateway)
|
|
20|9. [Phase 8 — Reranker (RAG Quality)](#phase-8--reranker-rag-quality)
|
|
21|10. [Phase 9 — faster-whisper (Speech-to-Text)](#phase-9--faster-whisper-speech-to-text)
|
|
22|11. [Additional Tools Setup](#additional-tools-setup)
|
|
23|12. [Deployment Order Summary](#deployment-order-summary)
|
|
24|13. [Post-Deployment Validation Master Checklist](#post-deployment-validation-master-checklist)
|
|
25|
|
|
26|---
|
|
27|
|
|
28|## Prerequisites & Conventions
|
|
29|
|
|
30|### Deployment Standards (recap)
|
|
31|
|
|
32|All stacks on PlausibleDeniability follow the same pattern:
|
|
33|
|
|
34|- **Repo root:** `/mnt/docker-ssd/docker/compose`
|
|
35|- **Validate before deploy:** `docker compose --env-file .env config`
|
|
36|- **Deploy:** `docker compose --env-file .env up -d`
|
|
37|- **Teardown:** `docker compose --env-file .env down`
|
|
38|
|
|
39|### Storage Tiers
|
|
40|
|
|
41|| Tier | Mount | Use For |
|
|
42||------|-------|---------|
|
|
43|| SSD | `/mnt/docker-ssd/docker/appdata/<service>` | Write-heavy, SQLite, GPU/model, databases |
|
|
44|| Tank | `/mnt/tank/docker/appdata/<service>` | General appdata, configs, uploads |
|
|
45|| Unraid | `/mnt/unraid/data/media/` | Media libraries only |
|
|
46|
|
|
47|### Networks
|
|
48|
|
|
49|| Network | Created By | Purpose |
|
|
50||---------|-----------|---------|
|
|
51|| `ix-databases_shared-databases` | databases stack | Access to shared-postgres, shared-mariadb, shared-redis |
|
|
52|| `pangolin` | newt (infrastructure stack) | Reverse proxy / external exposure |
|
|
53|
|
|
54|### Secret Generation Commands
|
|
55|
|
|
56|```bash
|
|
57|# Database passwords
|
|
58|openssl rand -hex 24
|
|
59|
|
|
60|# JWT / encryption keys
|
|
61|openssl rand -hex 32
|
|
62|
|
|
63|# Paperless secret key
|
|
64|python3 -c "import secrets; print(secrets.token_urlsafe(50))"
|
|
65|```
|
|
66|
|
|
67|### Deployment Order
|
|
68|
|
|
69|```
|
|
70|databases (already running) → infrastructure (already running)
|
|
71| → Phase 1: Gotify
|
|
72| → Phase 2: Qdrant (creates the ai-services network)
|
|
73| → Phase 3: n8n (depends on Gotify + Qdrant — joins ai-services as external)
|
|
74| → Phase 4: Paperless-NGX (depends on n8n for automation hooks)
|
|
75| → Phase 5: Home Assistant (depends on n8n for heavy automation)
|
|
76| → Phase 6: Grafana + Prometheus (on N.O.M.A.D., independent but benefits from all above)
|
|
77|```
|
|
78|
|
|
79|> **Important:** The `ai` stack (Phase 2) must be deployed before the `automation` stack (Phase 3) because n8n declares `ai-services` as an external network. If the `ai` stack isn't up, `docker compose up` for `automation` will fail with a missing network error.
|
|
80|
|
|
81|---
|
|
82|
|
|
83|## Phase 1 — Gotify (Notifications)
|
|
84|
|
|
85|**Host:** PlausibleDeniability
|
|
86|**Stack directory:** `/mnt/docker-ssd/docker/compose/automation/`
|
|
87|**Why first:** Every subsequent phase sends notifications through Gotify. It's the output bus.
|
|
88|
|
|
89|### 1.1 — Scaffold Directories
|
|
90|
|
|
91|```bash
|
|
92|# Appdata on SSD (SQLite backend — must not be on NFS)
|
|
93|sudo mkdir -p /mnt/docker-ssd/docker/appdata/gotify
|
|
94|
|
|
95|# Stack directory (may already exist if automation/ is planned)
|
|
96|sudo mkdir -p /mnt/docker-ssd/docker/compose/automation
|
|
97|```
|
|
98|
|
|
99|### 1.2 — Database Init
|
|
100|
|
|
101|None required — Gotify uses an embedded SQLite database stored in its data volume.
|
|
102|
|
|
103|### 1.3 — docker-compose.yaml
|
|
104|
|
|
105|Add to `/mnt/docker-ssd/docker/compose/automation/docker-compose.yaml`:
|
|
106|
|
|
107|```yaml
|
|
108|name: automation
|
|
109|
|
|
110|services:
|
|
111| gotify:
|
|
112| image: gotify/server:2.6.1
|
|
113| container_name: gotify
|
|
114| restart: unless-stopped
|
|
115| ports:
|
|
116| - "8484:80"
|
|
117| environment:
|
|
118| TZ: ${TZ}
|
|
119| GOTIFY_DEFAULTUSER_PASS: ${GOTIFY_ADMIN_PASS}
|
|
120| GOTIFY_SERVER_PORT: 80
|
|
121| GOTIFY_DATABASE_DIALECT: sqlite3
|
|
122| GOTIFY_DATABASE_CONNECTION: data/gotify.db
|
|
123| volumes:
|
|
124| - /mnt/docker-ssd/docker/appdata/gotify:/app/data
|
|
125| networks:
|
|
126| - pangolin
|
|
127| - default
|
|
128|
|
|
129|networks:
|
|
130| pangolin:
|
|
131| external: true
|
|
132|```
|
|
133|
|
|
134|> **Image note:** `gotify/server:2.6.1` is the latest stable as of May 2026. Check [Docker Hub](https://hub.docker.com/r/gotify/server/tags) before deploying — pin to the exact version.
|
|
135|
|
|
136|### 1.4 — .env.example
|
|
137|
|
|
138|```bash
|
|
139|# /mnt/docker-ssd/docker/compose/automation/.env.example
|
|
140|
|
|
141|TZ=America/New_York
|
|
142|GOTIFY_ADMIN_PASS=CHANGE_ME
|
|
143|```
|
|
144|
|
|
145|### 1.5 — .env (create from example)
|
|
146|
|
|
147|```bash
|
|
148|cd /mnt/docker-ssd/docker/compose/automation
|
|
149|cp .env.example .env
|
|
150|# Edit .env with real values:
|
|
151|# GOTIFY_ADMIN_PASS=$(openssl rand -hex 16)
|
|
152|nano .env
|
|
153|```
|
|
154|
|
|
155|### 1.6 — Pangolin Configuration
|
|
156|
|
|
157|In your Pangolin dashboard, create a new resource:
|
|
158|
|
|
159|| Field | Value |
|
|
160||-------|-------|
|
|
161|| Domain | `gotify.paccoco.com` |
|
|
162|| Scheme | `http` |
|
|
163|| Host | `gotify` |
|
|
164|| Port | `80` |
|
|
165|| Network | Docker service name resolution via `pangolin` network |
|
|
166|
|
|
167|### 1.7 — Validate & Deploy
|
|
168|
|
|
169|```bash
|
|
170|cd /mnt/docker-ssd/docker/compose/automation
|
|
171|docker compose --env-file .env config
|
|
172|docker compose --env-file .env up -d
|
|
173|```
|
|
174|
|
|
175|### 1.8 — Post-Deploy Verification
|
|
176|
|
|
177|```bash
|
|
178|# Container running?
|
|
179|docker ps --filter name=gotify
|
|
180|
|
|
181|# Clean startup logs?
|
|
182|docker logs gotify --tail 20
|
|
183|
|
|
184|# Mounts correct?
|
|
185|docker inspect gotify --format '{{json .Mounts}}' | python3 -m json.tool
|
|
186|
|
|
187|# Quick health check
|
|
188|curl -s http://localhost:8484/health
|
|
189|
|
|
190|# Test notification via API
|
|
191|curl -s "http://localhost:8484/message?token=*** \
|
|
192| -F "title=Homelab" \
|
|
193| -F "message=Gotify is online" \
|
|
194| -F "priority=5"
|
|
195|```
|
|
196|
|
|
197|### 1.9 — First-Run Setup
|
|
198|
|
|
199|1. Navigate to `https://gotify.paccoco.com`
|
|
200|2. Log in with the admin password from `.env`
|
|
201|3. **Change the default admin password** in the UI
|
|
202|4. Create application tokens for each notification source:
|
|
203| - `n8n-workflows` — for all n8n automations
|
|
204| - `infrastructure-alerts` — for Uptime Kuma, Grafana, etc.
|
|
205| - `media-notifications` — for Sonarr/Radarr/Tautulli hooks
|
|
206| - `home-assistant` — for HA automations
|
|
207|5. Create client tokens for each receiving device (phone, desktop)
|
|
208|6. Install the Gotify Android app and configure with your client token
|
|
209|
|
|
210|---
|
|
211|
|
|
212|## Phase 2 — AI Stack (Vector DB, LLM, Embeddings, Reranker, STT)
|
|
213|
|
|
214|**Host:** PlausibleDeniability
|
|
215|**Stack directory:** `/mnt/docker-ssd/docker/compose/ai/`
|
|
216|**Status:** DEPLOYED AND VERIFIED (2026-05-05)
|
|
217|
|
|
218|> **DEPLOYED NOTE:** This phase was deployed as a complete AI stack with 6 services: Ollama (11434), OpenWebUI (8282), Qdrant (6333/6334), LiteLLM (4000), Reranker/TEI (8787), and faster-whisper (8786). The compose below shows the original Qdrant-only plan — see the actual running compose on PD at `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml` and the `project_ai_stack_deployed.md` memory for port mappings and fixes. Key lesson: always use `10.5.30.6` not `localhost` for health checks on TrueNAS Scale.
|
|
219|
|
|
220|### 2.1 — Scaffold Directories
|
|
221|
|
|
222|```bash
|
|
223|# Appdata on SSD (write-heavy vector storage)
|
|
224|sudo mkdir -p /mnt/docker-ssd/docker/appdata/qdrant/storage
|
|
225|sudo mkdir -p /mnt/docker-ssd/docker/appdata/qdrant/snapshots
|
|
226|
|
|
227|# Stack directory
|
|
228|sudo mkdir -p /mnt/docker-ssd/docker/compose/ai
|
|
229|```
|
|
230|
|
|
231|### 2.2 — docker-compose.yaml
|
|
232|
|
|
233|Create or update `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml`:
|
|
234|
|
|
235|```yaml
|
|
236|name: ai
|
|
237|
|
|
238|services:
|
|
239| qdrant:
|
|
240| image: qdrant/qdrant:v1.14.0
|
|
241| container_name: qdrant
|
|
242| restart: unless-stopped
|
|
243| ports:
|
|
244| - "6333:6333" # REST API
|
|
245| - "6334:6334" # gRPC
|
|
246| environment:
|
|
247| TZ: ${TZ}
|
|
248| QDRANT__SERVICE__GRPC_PORT: 6334
|
|
249| QDRANT__STORAGE__STORAGE_PATH: /qdrant/storage
|
|
250| QDRANT__STORAGE__SNAPSHOTS_PATH: /qdrant/snapshots
|
|
251| volumes:
|
|
252| - /mnt/docker-ssd/docker/appdata/qdrant/storage:/qdrant/storage
|
|
253| - /mnt/docker-ssd/docker/appdata/qdrant/snapshots:/qdrant/snapshots
|
|
254| networks:
|
|
255| - ai-services
|
|
256| - default
|
|
257|
|
|
258| # -------------------------------------------------------
|
|
259| # Ollama and OpenWebUI go here when deployed.
|
|
260| # They share this stack and the default + ai-services networks so
|
|
261| # OpenWebUI can reach Qdrant at http://qdrant:6333
|
|
262| # -------------------------------------------------------
|
|
263|
|
|
264| # ollama:
|
|
265| # image: ollama/ollama:latest
|
|
266| # container_name: ollama
|
|
267| # ...
|
|
268|
|
|
269| # openwebui:
|
|
270| # image: ghcr.io/open-webui/open-webui:main
|
|
271| # container_name: openwebui
|
|
272| # ...
|
|
273|
|
|
274|networks:
|
|
275| ai-services:
|
|
276| name: ai-services
|
|
277|```
|
|
278|
|
|
279|> **Image note:** `qdrant/qdrant:v1.14.0` is the latest stable as of May 2026. Check [GitHub releases](https://github.com/qdrant/qdrant/releases) before deploying.
|
|
280|
|
|
281|> **Cross-stack connectivity:** The `ai-services` network is defined here with an explicit `name:` so other stacks (like `automation/n8n`) can declare it as `external: true` and reach Qdrant by service name (`http://qdrant:6333`). This follows the same pattern as `ix-databases_shared-databases`.
|
|
282|
|
|
283|### 2.3 — .env.example
|
|
284|
|
|
285|```bash
|
|
286|# /mnt/docker-ssd/docker/compose/ai/.env.example
|
|
287|
|
|
288|TZ=America/New_York
|
|
289|```
|
|
290|
|
|
291|### 2.4 — Validate & Deploy
|
|
292|
|
|
293|```bash
|
|
294|cd /mnt/docker-ssd/docker/compose/ai
|
|
295|cp .env.example .env
|
|
296|nano .env # Set timezone
|
|
297|
|
|
298|docker compose --env-file .env config
|
|
299|docker compose --env-file .env up -d
|
|
300|```
|
|
301|
|
|
302|### 2.5 — Post-Deploy Verification
|
|
303|
|
|
304|```bash
|
|
305|# Container running?
|
|
306|docker ps --filter name=qdrant
|
|
307|
|
|
308|# Clean startup?
|
|
309|docker logs qdrant --tail 20
|
|
310|
|
|
311|# REST API responding?
|
|
312|curl -s http://localhost:6333/healthz
|
|
313|# Expected: {"title":"qdrant - vectorass engine","version":"1.14.0","commit":"..."}
|
|
314|
|
|
315|# Create a test collection
|
|
316|curl -X PUT http://localhost:6333/collections/test_collection \
|
|
317| -H "Content-Type: application/json" \
|
|
318| -d '{
|
|
319| "vectors": {
|
|
320| "size": 384,
|
|
321| "distance": "Cosine"
|
|
322| }
|
|
323| }'
|
|
324|
|
|
325|# Verify it exists
|
|
326|curl -s http://localhost:6333/collections | python3 -m json.tool
|
|
327|
|
|
328|# Clean up test
|
|
329|curl -X DELETE http://localhost:6333/collections/test_collection
|
|
330|```
|
|
331|
|
|
332|### 2.6 — OpenWebUI Integration (when deployed)
|
|
333|
|
|
334|When Ollama and OpenWebUI are brought online in this same stack, configure OpenWebUI's RAG settings:
|
|
335|
|
|
336|1. Go to OpenWebUI → Admin → Settings → Documents
|
|
337|2. Set the vector database to Qdrant
|
|
338|3. Endpoint: `http://qdrant:6333` (Docker DNS within the `ai` stack network)
|
|
339|4. Collection name: `openwebui_docs` (or your preference)
|
|
340|
|
|
341|### 2.7 — Collections to Create (for n8n in Phase 3)
|
|
342|
|
|
343|These collections will be created programmatically by n8n workflows, but for reference:
|
|
344|
|
|
345|| Collection | Vector Size | Content |
|
|
346||-----------|-------------|---------|
|
|
347|| `homelab_docs` | 384 (nomic-embed-text) | Homelab markdown documentation |
|
|
348|| `gitea_commits` | 384 | Gitea commit messages + diffs |
|
|
349|| `media_metadata` | 384 | Plex/Tautulli metadata |
|
|
350|| `obsidian_notes` | 384 | Personal notes from Obsidian vault |
|
|
351|
|
|
352|> **Vector size note:** 384 is the dimension for `nomic-embed-text` via Ollama. If you use a different embedding model, adjust accordingly.
|
|
353|
|
|
354|---
|
|
355|
|
|
356|## Phase 3 — n8n (Workflow Automation)
|
|
357|
|
|
358|**Host:** PlausibleDeniability
|
|
359|**Stack directory:** `/mnt/docker-ssd/docker/compose/automation/` (same stack as Gotify)
|
|
360|**Why third:** n8n is the orchestration backbone — it ties Gotify, Qdrant, Ollama, and all triggers together.
|
|
361|
|
|
362|### 3.1 — Scaffold Directories
|
|
363|
|
|
364|```bash
|
|
365|# Config on tank (workflow definitions, credentials store)
|
|
366|# NOTE: n8n also writes execution logs here. If execution logging becomes
|
|
367|# heavy (many workflows running frequently), consider moving to SSD.
|
|
368|# For typical homelab usage (~20 workflows), tank is fine.
|
|
369|sudo mkdir -p /mnt/tank/docker/appdata/n8n
|
|
370|
|
|
371|# Set ownership — n8n runs as UID 1000 (node user)
|
|
372|sudo chown 1000:1000 /mnt/tank/docker/appdata/n8n
|
|
373|```
|
|
374|
|
|
375|### 3.2 — Database Init
|
|
376|
|
|
377|n8n uses the existing shared-postgres. Create its database and user:
|
|
378|
|
|
379|```bash
|
|
380|docker exec -i shared-postgres psql -U postgres <<'SQL'
|
|
381|CREATE USER n8n WITH PASSWORD 'REPLACE_WITH_GENERATED_PASSWORD';
|
|
382|CREATE DATABASE n8n OWNER n8n;
|
|
383|GRANT ALL PRIVILEGES ON DATABASE n8n TO n8n;
|
|
384|SQL
|
|
385|```
|
|
386|
|
|
387|Generate the password first:
|
|
388|```bash
|
|
389|openssl rand -hex 24
|
|
390|```
|
|
391|
|
|
392|### 3.3 — docker-compose.yaml
|
|
393|
|
|
394|Update `/mnt/docker-ssd/docker/compose/automation/docker-compose.yaml` to add n8n alongside Gotify:
|
|
395|
|
|
396|```yaml
|
|
397|name: automation
|
|
398|
|
|
399|services:
|
|
400| gotify:
|
|
401| image: gotify/server:2.6.1
|
|
402| container_name: gotify
|
|
403| restart: unless-stopped
|
|
404| ports:
|
|
405| - "8484:80"
|
|
406| environment:
|
|
407| TZ: ${TZ}
|
|
408| GOTIFY_DEFAULTUSER_PASS: ${GOTIFY_ADMIN_PASS}
|
|
409| GOTIFY_SERVER_PORT: 80
|
|
410| GOTIFY_DATABASE_DIALECT: sqlite3
|
|
411| GOTIFY_DATABASE_CONNECTION: data/gotify.db
|
|
412| volumes:
|
|
413| - /mnt/docker-ssd/docker/appdata/gotify:/app/data
|
|
414| networks:
|
|
415| - pangolin
|
|
416| - default
|
|
417|
|
|
418| n8n:
|
|
419| image: docker.n8n.io/n8nio/n8n:1.88.0
|
|
420| container_name: n8n
|
|
421| restart: unless-stopped
|
|
422| ports:
|
|
423| - "5678:5678"
|
|
424| environment:
|
|
425| TZ: ${TZ}
|
|
426| # Database
|
|
427| DB_TYPE: postgresdb
|
|
428| DB_POSTGRESDB_HOST: shared-postgres
|
|
429| DB_POSTGRESDB_PORT: 5432
|
|
430| DB_POSTGRESDB_DATABASE: ${N8N_DB_NAME}
|
|
431| DB_POSTGRESDB_USER: ${N8N_DB_USER}
|
|
432| DB_POSTGRESDB_PASSWORD: ${N8N_DB_PASS}
|
|
433| # Security
|
|
434| N8N_ENCRYPTION_KEY: ${N8N_ENCRYPTION_KEY}
|
|
435| # Webhook / Reverse Proxy
|
|
436| N8N_HOST: ${N8N_HOST}
|
|
437| N8N_PROTOCOL: https
|
|
438| N8N_PORT: 5678
|
|
439| WEBHOOK_URL: https://${N8N_HOST}/
|
|
440| N8N_PROXY_HOPS: 1
|
|
441| # General
|
|
442| GENERIC_TIMEZONE: ${TZ}
|
|
443| N8N_DIAGNOSTICS_ENABLED: false
|
|
444| N8N_PERSONALIZATION_ENABLED: false
|
|
445| volumes:
|
|
446| - /mnt/tank/docker/appdata/n8n:/home/node/.n8n
|
|
447| networks:
|
|
448| - pangolin
|
|
449| - ix-databases_shared-databases
|
|
450| - ai-services
|
|
451| - default
|
|
452| depends_on:
|
|
453| - gotify
|
|
454|
|
|
455|networks:
|
|
456| pangolin:
|
|
457| external: true
|
|
458| ix-databases_shared-databases:
|
|
459| external: true
|
|
460| ai-services:
|
|
461| external: true
|
|
462|```
|
|
463|
|
|
464|> **Image note:** `docker.n8n.io/n8nio/n8n:1.88.0` — n8n uses calendar-ish versioning. Check [n8n releases](https://github.com/n8n-io/n8n/releases) for the latest stable. The `docker.n8n.io` registry is preferred for production.
|
|
465|
|
|
466|> **Cross-stack access:** n8n joins `ai-services` (created by the `ai` stack) so it can reach Qdrant at `http://qdrant:6333` by Docker DNS. It also joins `ix-databases_shared-databases` for Postgres access — same pattern.
|
|
467|
|
|
468|### 3.4 — .env.example (updated for both services)
|
|
469|
|
|
470|```bash
|
|
471|# /mnt/docker-ssd/docker/compose/automation/.env.example
|
|
472|
|
|
473|TZ=America/New_York
|
|
474|
|
|
475|# Gotify
|
|
476|GOTIFY_ADMIN_PASS=CHANGE_ME
|
|
477|
|
|
478|# n8n — Database
|
|
479|N8N_DB_NAME=n8n
|
|
480|N8N_DB_USER=n8n
|
|
481|N8N_DB_PASS=CHANGE_ME
|
|
482|
|
|
483|# n8n — Security
|
|
484|N8N_ENCRYPTION_KEY=CHANGE_ME
|
|
485|
|
|
486|# n8n — Hostname
|
|
487|N8N_HOST=n8n.paccoco.com
|
|
488|```
|
|
489|
|
|
490|### 3.5 — Pangolin Configuration
|
|
491|
|
|
492|| Field | Value |
|
|
493||-------|-------|
|
|
494|| Domain | `n8n.paccoco.com` |
|
|
495|| Scheme | `http` |
|
|
496|| Host | `n8n` |
|
|
497|| Port | `5678` |
|
|
498|| Headers | Forward `X-Forwarded-Proto: https` |
|
|
499|
|
|
500|### 3.6 — Validate & Deploy
|
|
501|
|
|
502|```bash
|
|
503|cd /mnt/docker-ssd/docker/compose/automation
|
|
504|
|
|
505|# Update .env with real passwords
|
|
506|nano .env
|
|
507|
|
|
508|docker compose --env-file .env config
|
|
509|docker compose --env-file .env up -d
|
|
510|```
|
|
511|
|
|
512|### 3.7 — Post-Deploy Verification
|
|
513|
|
|
514|```bash
|
|
515|# Both containers running?
|
|
516|docker ps --filter name=gotify --filter name=n8n
|
|
517|
|
|
518|# n8n logs clean?
|
|
519|docker logs n8n --tail 30
|
|
520|
|
|
521|# n8n on correct networks?
|
|
522|docker inspect n8n --format '{{json .NetworkSettings.Networks}}' | python3 -m json.tool
|
|
523|
|
|
524|# n8n UI accessible?
|
|
525|curl -s -o /dev/null -w "%{http_code}" http://localhost:5678/
|
|
526|# Expected: 200
|
|
527|
|
|
528|# Postgres connection working? (check logs for DB migration messages)
|
|
529|docker logs n8n 2>&1 | grep -i "migrat"
|
|
530|```
|
|
531|
|
|
532|### 3.8 — First-Run Setup
|
|
533|
|
|
534|1. Navigate to `https://n8n.paccoco.com`
|
|
535|2. Create your admin account
|
|
536|3. Install community nodes you'll need:
|
|
537| - `n8n-nodes-gotify` (if available) or use HTTP Request node
|
|
538| - Ollama nodes (built-in as of n8n 1.x)
|
|
539|
|
|
540|### 3.9 — Workflow Blueprints
|
|
541|
|
|
542|Below are starter workflow descriptions for each planned automation. These are meant to be built in the n8n UI — the structure and node types are described so you can wire them up.
|
|
543|
|
|
544|#### Media Pipeline Workflows
|
|
545|
|
|
546|**Workflow: Sonarr/Radarr Download Notification**
|
|
547|```
|
|
548|Trigger: Webhook node (POST from Sonarr/Radarr on download/import)
|
|
549|Step 1: Extract series/movie name, quality, file path from webhook body
|
|
550|Step 2: HTTP Request → TMDB API to fetch poster image URL
|
|
551|Step 3: HTTP Request → Gotify REST API (POST /message)
|
|
552| - Title: "New Download: {title}"
|
|
553| - Message: "{quality} — {episodeTitle or year}"
|
|
554| - Priority: 5
|
|
555| - Extras: attach poster URL as markdown image
|
|
556|Also: HTTP Request → Discord webhook (formatted embed with poster)
|
|
557|```
|
|
558|
|
|
559|Configure Sonarr/Radarr webhooks:
|
|
560|- Sonarr: Settings → Connect → Webhook → URL: `https://n8n.paccoco.com/webhook/sonarr`
|
|
561|- Radarr: Settings → Connect → Webhook → URL: `https://n8n.paccoco.com/webhook/radarr`
|
|
562|
|
|
563|**Workflow: Tautulli Play Logging**
|
|
564|```
|
|
565|Trigger: Webhook node (POST from Tautulli on playback start)
|
|
566|Step 1: Extract user, media title, player, quality from payload
|
|
567|Step 2: Postgres node → INSERT into watch_history table
|
|
568|Step 3: (Optional) Gotify notification for specific users/media
|
|
569|```
|
|
570|
|
|
571|Tautulli config: Settings → Notification Agents → Webhook → URL: `https://n8n.paccoco.com/webhook/tautulli-play`
|
|
572|
|
|
573|**Workflow: Weekly Watch Digest**
|
|
574|```
|
|
575|Trigger: Cron node (every Sunday at 10:00 AM)
|
|
576|Step 1: Postgres node → SELECT watch history for past 7 days
|
|
577|Step 2: Format data as structured text
|
|
578|Step 3: HTTP Request → Ollama API (POST to the local/default automation model host, preferably N.O.M.A.D.-local rather than PD)
|
|
579| - Prompt: "Summarize this week's viewing in a fun digest: {data}"
|
|
580|Step 4: HTTP Request → Gotify (send digest)
|
|
581|```
|
|
582|
|
|
583|#### Infrastructure Monitoring Workflows
|
|
584|
|
|
585|**Workflow: Uptime Kuma Enhanced Alerts**
|
|
586|```
|
|
587|Trigger: Webhook node (from Uptime Kuma notification)
|
|
588|Step 1: Extract monitor name, status, response time
|
|
589|Step 2: HTTP Request → Netdata API for related metrics context
|
|
590|Step 3: HTTP Request → Gotify
|
|
591| - Title: "🔴 {monitor} DOWN" or "🟢 {monitor} UP"
|
|
592| - Message: include Netdata context (CPU, mem, disk at time of alert)
|
|
593| - Priority: 8 (high for down, 3 for recovery)
|
|
594|```
|
|
595|
|
|
596|**Workflow: ZFS Pool Utilization Alert**
|
|
597|```
|
|
598|Trigger: Cron node (every 6 hours)
|
|
599|Step 1: SSH node → Serenity: `zpool list -Hp malcolm`
|
|
600|Step 2: Parse capacity percentage
|
|
601|Step 3: IF capacity > 85% → Gotify alert (priority 8)
|
|
602|Step 4: IF capacity > 90% → Gotify alert (priority 10) + Discord webhook
|
|
603|```
|
|
604|
|
|
605|**Workflow: Grafana Alert Remediation**
|
|
606|```
|
|
607|Trigger: Webhook node (from Grafana alerting)
|
|
608|Step 1: Parse alert labels (container, host, metric)
|
|
609|Step 2: Switch node → route by alert type:
|
|
610| - High CPU container → SSH → docker restart {container}
|
|
611| - Disk full → SSH → pause qBittorrent, notify via Gotify
|
|
612| - Memory pressure → Gotify alert only (manual intervention)
|
|
613|Step 3: Log action taken to Postgres
|
|
614|```
|
|
615|
|
|
616|#### Homelab Ops Workflows
|
|
617|
|
|
618|**Workflow: Gitea Commit Summary**
|
|
619|```
|
|
620|Trigger: Webhook node (Gitea webhook on push to truenas-stacks)
|
|
621|Step 1: Extract commit messages, author, files changed
|
|
622|Step 2: HTTP Request → Ollama API
|
|
623| - Prompt: "Summarize this commit in one sentence: {commit_message}"
|
|
624|Step 3: HTTP Request → Gotify
|
|
625| - Title: "Commit to truenas-stacks"
|
|
626| - Message: Ollama-generated summary
|
|
627|```
|
|
628|
|
|
629|Gitea config: Repository → Settings → Webhooks → URL: `https://n8n.paccoco.com/webhook/gitea-push`
|
|
630|
|
|
631|**Workflow: qBittorrent Auto-Rescan**
|
|
632|```
|
|
633|Trigger: Webhook or polling (qBittorrent API for completed+moved torrents)
|
|
634|Step 1: Determine if file is in Sonarr or Radarr path
|
|
635|Step 2: HTTP Request → Sonarr API (POST /command → RescanSeries)
|
|
636| OR HTTP Request → Radarr API (POST /command → RescanMovie)
|
|
637|Step 3: Gotify notification confirming rescan triggered
|
|
638|```
|
|
639|
|
|
640|#### AI Pipeline Workflows
|
|
641|
|
|
642|**Workflow: URL Digest Pipeline**
|
|
643|```
|
|
644|Trigger: Webhook node (POST with URL in body)
|
|
645|Step 1: HTTP Request → fetch page content
|
|
646|Step 2: Code node → extract text, chunk into ~500 token segments
|
|
647|Step 3: HTTP Request → Ollama API → summarize each chunk
|
|
648|Step 4: Code node → combine summaries into digest
|
|
649|Step 5: Postgres node → store digest with metadata
|
|
650|Step 6: Return digest in webhook response
|
|
651|```
|
|
652|
|
|
653|**Workflow: Multi-Model Query Router (simplified by LiteLLM — see Phase 7)**
|
|
654|```
|
|
655|Trigger: Webhook node (POST with query + complexity hint)
|
|
656|Step 1: HTTP Request → LiteLLM (POST http://litellm:4000/v1/chat/completions)
|
|
657| - model: complexity parameter ("light", "medium", or "heavy")
|
|
658| - LiteLLM handles routing to the correct Ollama instance
|
|
659|Step 2: Return response via webhook
|
|
660|```
|
|
661|> With LiteLLM deployed, the Switch node and three separate Ollama endpoints
|
|
662|> are replaced by a single HTTP Request node. The routing logic lives in
|
|
663|> LiteLLM's config.yaml instead of n8n workflow logic.
|
|
664|
|
|
665|**Workflow: Qdrant Index Updater**
|
|
666|```
|
|
667|Trigger: Webhook node (from Gitea push to any watched repo)
|
|
668|Step 1: HTTP Request → Gitea API → fetch changed files content
|
|
669|Step 2: Code node → chunk text into embedding-sized segments
|
|
670|Step 3: HTTP Request → Ollama API (embed endpoint with nomic-embed-text)
|
|
671|Step 4: HTTP Request → Qdrant API (PUT /collections/homelab_docs/points)
|
|
672|Step 5: Gotify notification: "{n} documents re-indexed"
|
|
673|```
|
|
674|
|
|
675|#### Home / Business Workflows
|
|
676|
|
|
677|**Workflow: KitchenOwl Grocery Notification**
|
|
678|```
|
|
679|Trigger: Polling (KitchenOwl API) or webhook if supported
|
|
680|Step 1: Fetch current shopping list items
|
|
681|Step 2: Format as clean text list
|
|
682|Step 3: HTTP Request → Gotify → phone push notification
|
|
683|```
|
|
684|
|
|
685|**Workflow: Donetick Task Reminder**
|
|
686|```
|
|
687|Trigger: Cron node (daily at 9:00 AM)
|
|
688|Step 1: HTTP Request → Donetick API → fetch tasks due today/overdue
|
|
689|Step 2: Format task list
|
|
690|Step 3: HTTP Request → Gotify (priority 5)
|
|
691|```
|
|
692|
|
|
693|**Workflow: Long and Low Crafts Order Pipeline**
|
|
694|```
|
|
695|Trigger: Webhook (Etsy webhook or email trigger via IMAP node)
|
|
696|Step 1: Parse order details (item, quantity, customer, shipping)
|
|
697|Step 2: HTTP Request → Donetick API → create fulfillment task
|
|
698|Step 3: HTTP Request → Gotify DM notification
|
|
699| - Title: "New L&L Order"
|
|
700| - Message: "{item} x{qty} — ship by {date}"
|
|
701| - Priority: 7
|
|
702|```
|
|
703|
|
|
704|---
|
|
705|
|
|
706|## Phase 4 — Paperless-NGX (Document Intelligence)
|
|
707|
|
|
708|**Host:** PlausibleDeniability
|
|
709|**Stack directory:** `/mnt/docker-ssd/docker/compose/documents/` (new stack)
|
|
710|**Why fourth:** Depends on n8n for automation hooks and Gotify for notifications.
|
|
711|
|
|
712|### 4.1 — Scaffold Directories
|
|
713|
|
|
714|```bash
|
|
715|# Index/data on SSD (search index is write-heavy)
|
|
716|sudo mkdir -p /mnt/docker-ssd/docker/appdata/paperless/data
|
|
717|
|
|
718|# Documents (media) on tank (bulk storage, read-heavy)
|
|
719|sudo mkdir -p /mnt/tank/docker/appdata/paperless/media
|
|
720|
|
|
721|# Consume folder on tank (drop zone for new documents)
|
|
722|sudo mkdir -p /mnt/tank/docker/appdata/paperless/consume
|
|
723|
|
|
724|# Export folder on tank
|
|
725|sudo mkdir -p /mnt/tank/docker/appdata/paperless/export
|
|
726|
|
|
727|# Stack directory
|
|
728|sudo mkdir -p /mnt/docker-ssd/docker/compose/documents
|
|
729|```
|
|
730|
|
|
731|### 4.2 — Database Init
|
|
732|
|
|
733|Paperless uses the existing shared-postgres and shared-redis:
|
|
734|
|
|
735|```bash
|
|
736|docker exec -i shared-postgres psql -U postgres <<'SQL'
|
|
737|CREATE USER paperless WITH PASSWORD 'REPLACE_WITH_GENERATED_PASSWORD';
|
|
738|CREATE DATABASE paperless OWNER paperless;
|
|
739|GRANT ALL PRIVILEGES ON DATABASE paperless TO paperless;
|
|
740|SQL
|
|
741|```
|
|
742|
|
|
743|### 4.3 — docker-compose.yaml
|
|
744|
|
|
745|Create `/mnt/docker-ssd/docker/compose/documents/docker-compose.yaml`:
|
|
746|
|
|
747|```yaml
|
|
748|name: documents
|
|
749|
|
|
750|services:
|
|
751| paperless:
|
|
752| image: ghcr.io/paperless-ngx/paperless-ngx:2.16
|
|
753| container_name: paperless
|
|
754| restart: unless-stopped
|
|
755| ports:
|
|
756| - "8000:8000"
|
|
757| environment:
|
|
758| TZ: ${TZ}
|
|
759| # Database
|
|
760| PAPERLESS_DBENGINE: postgresql
|
|
761| PAPERLESS_DBHOST: shared-postgres
|
|
762| PAPERLESS_DBPORT: 5432
|
|
763| PAPERLESS_DBNAME: ${PAPERLESS_DB_NAME}
|
|
764| PAPERLESS_DBUSER: ${PAPERLESS_DB_USER}
|
|
765| PAPERLESS_DBPASS: ${PAPERLESS_DB_PASS}
|
|
766| # Redis (using shared-redis)
|
|
767| PAPERLESS_REDIS: redis://shared-redis:***@10.5.30.7
|
|
1063|
|
|
1064|# Stack directory
|
|
1065|sudo mkdir -p /opt/monitoring
|
|
1066|
|
|
1067|# Prometheus data on hdd-2 (has more headroom)
|
|
1068|sudo mkdir -p /mnt/hdd-2/prometheus-data
|
|
1069|sudo chown 65534:65534 /mnt/hdd-2/prometheus-data # nobody user (Prometheus default)
|
|
1070|
|
|
1071|# Grafana data on hdd-2
|
|
1072|sudo mkdir -p /mnt/hdd-2/grafana-data
|
|
1073|sudo chown 472:472 /mnt/hdd-2/grafana-data # grafana user
|
|
1074|
|
|
1075|# Config directories
|
|
1076|sudo mkdir -p /opt/monitoring/provisioning/datasources
|
|
1077|sudo mkdir -p /opt/monitoring/provisioning/dashboards
|
|
1078|```
|
|
1079|
|
|
1080|### 6.2 — prometheus.yml
|
|
1081|
|
|
1082|Create `/opt/monitoring/prometheus.yml`:
|
|
1083|
|
|
1084|```yaml
|
|
1085|global:
|
|
1086| scrape_interval: 15s
|
|
1087| evaluation_interval: 15s
|
|
1088|
|
|
1089|scrape_configs:
|
|
1090| # ------- Local (N.O.M.A.D.) -------
|
|
1091| - job_name: 'nomad-node'
|
|
1092| static_configs:
|
|
1093| - targets: ['node-exporter:9100']
|
|
1094| labels:
|
|
1095| host: 'nomad'
|
|
1096|
|
|
1097| # ------- PlausibleDeniability -------
|
|
1098| - job_name: 'pd-netdata'
|
|
1099| metrics_path: /api/v1/allmetrics
|
|
1100| params:
|
|
1101| format: [prometheus]
|
|
1102| static_configs:
|
|
1103| - targets: ['10.5.1.X:19999'] # Replace with PD's IP
|
|
1104| labels:
|
|
1105| host: 'plausible-deniability'
|
|
1106|
|
|
1107| # ------- Serenity -------
|
|
1108| - job_name: 'serenity-netdata'
|
|
1109| metrics_path: /api/v1/allmetrics
|
|
1110| params:
|
|
1111| format: [prometheus]
|
|
1112| static_configs:
|
|
1113| - targets: ['10.5.30.5:19999']
|
|
1114| labels:
|
|
1115| host: 'serenity'
|
|
1116|
|
|
1117| # ------- Gotify -------
|
|
1118| # Gotify exposes /health but no Prometheus endpoint natively.
|
|
1119| # Use blackbox exporter or just rely on Uptime Kuma.
|
|
1120|
|
|
1121| # ------- n8n -------
|
|
1122| # n8n doesn't expose Prometheus metrics by default.
|
|
1123| # Monitor via container resource metrics from Netdata.
|
|
1124|
|
|
1125| # -------- Add more targets as needed --------
|
|
1126| # When node-exporter is installed on PD and Serenity:
|
|
1127| # - job_name: 'pd-node'
|
|
1128| # static_configs:
|
|
1129| # - targets: ['10.5.1.X:9100']
|
|
1130| # labels:
|
|
1131| # host: 'plausible-deniability'
|
|
1132| #
|
|
1133| # - job_name: 'serenity-node'
|
|
1134| # static_configs:
|
|
1135| # - targets: ['10.5.30.5:9100']
|
|
1136| # labels:
|
|
1137| # host: 'serenity'
|
|
1138|```
|
|
1139|
|
|
1140|> **Netdata as a Prometheus target:** Both PD and Serenity already run Netdata. Netdata has a built-in Prometheus exporter at `/api/v1/allmetrics?format=prometheus`. This gives you CPU, memory, disk, network, and ZFS metrics without installing node-exporter on those hosts.
|
|
1141|
|
|
1142|### 6.3 — Grafana Provisioning
|
|
1143|
|
|
1144|Create `/opt/monitoring/provisioning/datasources/prometheus.yml`:
|
|
1145|
|
|
1146|```yaml
|
|
1147|apiVersion: 1
|
|
1148|
|
|
1149|datasources:
|
|
1150| - name: Prometheus
|
|
1151| type: prometheus
|
|
1152| access: proxy
|
|
1153| url: http://prometheus:9090
|
|
1154| isDefault: true
|
|
1155| editable: true
|
|
1156|```
|
|
1157|
|
|
1158|Create `/opt/monitoring/provisioning/dashboards/dashboards.yml`:
|
|
1159|
|
|
1160|```yaml
|
|
1161|apiVersion: 1
|
|
1162|
|
|
1163|providers:
|
|
1164| - name: 'default'
|
|
1165| orgId: 1
|
|
1166| folder: ''
|
|
1167| type: file
|
|
1168| disableDeletion: false
|
|
1169| editable: true
|
|
1170| options:
|
|
1171| path: /var/lib/grafana/dashboards
|
|
1172| foldersFromFilesStructure: false
|
|
1173|```
|
|
1174|
|
|
1175|### 6.4 — docker-compose.yaml
|
|
1176|
|
|
1177|Create `/opt/monitoring/docker-compose.yaml`:
|
|
1178|
|
|
1179|```yaml
|
|
1180|name: monitoring
|
|
1181|
|
|
1182|services:
|
|
1183| prometheus:
|
|
1184| image: prom/prometheus:v3.4.0
|
|
1185| container_name: prometheus
|
|
1186| restart: unless-stopped
|
|
1187| ports:
|
|
1188| - "9090:9090"
|
|
1189| command:
|
|
1190| - '--config.file=/etc/prometheus/prometheus.yml'
|
|
1191| - '--storage.tsdb.path=/prometheus'
|
|
1192| - '--storage.tsdb.retention.time=90d'
|
|
1193| - '--web.enable-lifecycle'
|
|
1194| volumes:
|
|
1195| - /opt/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
|
1196| - /mnt/hdd-2/prometheus-data:/prometheus
|
|
1197| networks:
|
|
1198| - monitoring
|
|
1199|
|
|
1200| grafana:
|
|
1201| image: grafana/grafana:13.0.1
|
|
1202| container_name: grafana
|
|
1203| restart: unless-stopped
|
|
1204| ports:
|
|
1205| - "3000:3000"
|
|
1206| environment:
|
|
1207| TZ: ${TZ}
|
|
1208| GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER}
|
|
1209| GF_SECURITY_ADMIN_PASSWORD: ${GF_ADMIN_PASS}
|
|
1210| GF_SERVER_ROOT_URL: https://${GF_HOST}/
|
|
1211| volumes:
|
|
1212| - /mnt/hdd-2/grafana-data:/var/lib/grafana
|
|
1213| - /opt/monitoring/provisioning:/etc/grafana/provisioning:ro
|
|
1214| networks:
|
|
1215| - monitoring
|
|
1216|
|
|
1217| node-exporter:
|
|
1218| image: prom/node-exporter:v1.9.0
|
|
1219| container_name: node-exporter
|
|
1220| restart: unless-stopped
|
|
1221| ports:
|
|
1222| - "9100:9100"
|
|
1223| command:
|
|
1224| - '--path.procfs=/host/proc'
|
|
1225| - '--path.sysfs=/host/sys'
|
|
1226| - '--path.rootfs=/rootfs'
|
|
1227| - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
|
|
1228| volumes:
|
|
1229| - /proc:/host/proc:ro
|
|
1230| - /sys:/host/sys:ro
|
|
1231| - /:/rootfs:ro
|
|
1232| networks:
|
|
1233| - monitoring
|
|
1234|
|
|
1235|networks:
|
|
1236| monitoring:
|
|
1237| driver: bridge
|
|
1238|```
|
|
1239|
|
|
1240|> **Image notes:**
|
|
1241|> - `prom/prometheus:v3.4.0` — Prometheus v3 is the current major line. **Do not use `:latest`** — it still resolves to 2.x due to a known tagging issue. Always use an explicit `v3.x.y` tag.
|
|
1242|> - `grafana/grafana:latest` — use the moving stable image by default here; only pin Grafana if a breakage or compatibility reason is documented.
|
|
1243|> - `prom/node-exporter:v1.9.0` — check [releases](https://github.com/prometheus/node_exporter/releases).
|
|
1244|
|
|
1245|### 6.5 — .env.example
|
|
1246|
|
|
1247|```bash
|
|
1248|# /opt/monitoring/.env.example
|
|
1249|
|
|
1250|TZ=America/New_York
|
|
1251|
|
|
1252|# Grafana
|
|
1253|GF_ADMIN_USER=admin
|
|
1254|GF_ADMIN_PASS=CHANGE_ME
|
|
1255|GF_HOST=grafana.paccoco.com
|
|
1256|```
|
|
1257|
|
|
1258|### 6.6 — Pangolin Configuration
|
|
1259|
|
|
1260|Grafana runs on N.O.M.A.D., not PD where the main Newt agent lives. You have two options:
|
|
1261|
|
|
1262|**Option A — Route via N.O.M.A.D.'s existing Newt (recommended if already connected)**
|
|
1263|
|
|
1264|N.O.M.A.D. already has a Newt container from the Project N.O.M.A.D. setup. If it's connected to your Pangolin VPS, just add a new resource in the Pangolin dashboard pointing to `http://grafana:3000` or `http://10.5.30.7:3000`.
|
|
1265|
|
|
1266|**Option B — Add a dedicated Newt to the monitoring stack**
|
|
1267|
|
|
1268|If N.O.M.A.D.'s existing Newt is not connected to Pangolin (or is a separate Pangolin instance), add Newt to the monitoring compose:
|
|
1269|
|
|
1270|```yaml
|
|
1271| newt:
|
|
1272| image: ghcr.io/fosrl/newt:latest
|
|
1273| container_name: monitoring-newt
|
|
1274| restart: unless-stopped
|
|
1275| environment:
|
|
1276| PANGOLIN_ENDPOINT: ${PANGOLIN_ENDPOINT}
|
|
1277| NEWT_ID: ${NEWT_ID}
|
|
1278| NEWT_SECRET: ${NEWT_SECRET}
|
|
1279| networks:
|
|
1280| - monitoring
|
|
1281|```
|
|
1282|
|
|
1283|Then in Pangolin dashboard:
|
|
1284|
|
|
1285|| Field | Value |
|
|
1286||-------|-------|
|
|
1287|| Domain | `grafana.paccoco.com` |
|
|
1288|| Scheme | `http` |
|
|
1289|| Host | `grafana` (Docker DNS via shared network) |
|
|
1290|| Port | `3000` |
|
|
1291|
|
|
1292|**Option C — Direct IP routing (simplest, no Newt needed)**
|
|
1293|
|
|
1294|If Pangolin's Newt on PD can reach N.O.M.A.D. by LAN IP (they're on the same subnet):
|
|
1295|
|
|
1296|| Field | Value |
|
|
1297||-------|-------|
|
|
1298|| Domain | `grafana.paccoco.com` |
|
|
1299|| Scheme | `http` |
|
|
1300|| Host | `10.5.30.7` |
|
|
1301|| Port | `3000` |
|
|
1302|
|
|
1303|> **Decision point:** Check if N.O.M.A.D.'s existing Newt is connected to your Pangolin instance before deploying. Run `docker ps --filter name=newt` on N.O.M.A.D. to verify. If it's running and connected, Option A is zero-effort. If not, Option C is the simplest fallback.
|
|
1304|
|
|
1305|### 6.7 — Validate & Deploy
|
|
1306|
|
|
1307|```bash
|
|
1308|ssh nomad@10.5.30.7
|
|
1309|
|
|
1310|cd /opt/monitoring
|
|
1311|cp .env.example .env
|
|
1312|nano .env # Fill in real values
|
|
1313|
|
|
1314|docker compose --env-file .env config
|
|
1315|docker compose --env-file .env up -d
|
|
1316|```
|
|
1317|
|
|
1318|### 6.8 — Post-Deploy Verification
|
|
1319|
|
|
1320|```bash
|
|
1321|# All three containers running?
|
|
1322|docker ps --filter name=prometheus --filter name=grafana --filter name=node-exporter
|
|
1323|
|
|
1324|# Prometheus scraping targets?
|
|
1325|curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | head -40
|
|
1326|
|
|
1327|# Grafana UI accessible?
|
|
1328|curl -s -o /dev/null -w "%{http_code}" http://localhost:3000/
|
|
1329|# Expected: 200 or 302
|
|
1330|
|
|
1331|# Node exporter metrics flowing?
|
|
1332|curl -s http://localhost:9100/metrics | head -10
|
|
1333|```
|
|
1334|
|
|
1335|### 6.9 — Recommended Dashboards
|
|
1336|
|
|
1337|Import these from [Grafana Dashboard Library](https://grafana.com/grafana/dashboards/):
|
|
1338|
|
|
1339|| Dashboard | ID | Purpose |
|
|
1340||-----------|----|---------|
|
|
1341|| Node Exporter Full | 1860 | System metrics for N.O.M.A.D. |
|
|
1342|| Docker Container Stats | 893 | Container resource usage |
|
|
1343|| Netdata via Prometheus | (search) | PD and Serenity system metrics |
|
|
1344|
|
|
1345|To import: Grafana → Dashboards → New → Import → Enter dashboard ID.
|
|
1346|
|
|
1347|### 6.10 — Metrics Targets Summary
|
|
1348|
|
|
1349|| Target | Host | Method | Endpoint |
|
|
1350||--------|------|--------|----------|
|
|
1351|| N.O.M.A.D. system | localhost | node-exporter | node-exporter:9100 |
|
|
1352|| PD system | 10.5.1.X | Netdata Prometheus | 10.5.1.X:19999/api/v1/allmetrics |
|
|
1353|| Serenity system | 10.5.30.5 | Netdata Prometheus | 10.5.30.5:19999/api/v1/allmetrics |
|
|
1354|| ZFS pools | via Netdata | Netdata exports ZFS metrics | Included in Netdata scrape |
|
|
1355|| Container stats | via Netdata | Netdata exports cgroup metrics | Included in Netdata scrape |
|
|
1356|| Plex streams | Tautulli | n8n polling → Postgres | Via n8n workflow (Phase 3) |
|
|
1357|| qBit stats | qBittorrent API | n8n polling → Postgres | Via n8n workflow (Phase 3) |
|
|
1358|| Tailscale latency | Tailscale API | n8n polling → Postgres | Via n8n workflow (Phase 3) |
|
|
1359|
|
|
1360|### 6.11 — n8n Integration (Grafana → n8n alert webhook)
|
|
1361|
|
|
1362|In Grafana → Alerting → Contact Points, create a webhook contact point:
|
|
1363|
|
|
1364|| Field | Value |
|
|
1365||-------|-------|
|
|
1366|| Name | `n8n-alerts` |
|
|
1367|| Type | Webhook |
|
|
1368|| URL | `https://n8n.paccoco.com/webhook/grafana-alert` |
|
|
1369|| HTTP Method | POST |
|
|
1370|
|
|
1371|Then create alert rules for:
|
|
1372|- ZFS pool utilization > 85%
|
|
1373|- Container memory > 90% of limit
|
|
1374|- Host CPU sustained > 90% for 5 minutes
|
|
1375|- Disk I/O latency spikes
|
|
1376|
|
|
1377|These fire into the "Grafana Alert Remediation" n8n workflow (see Phase 3).
|
|
1378|
|
|
1379|---
|
|
1380|
|
|
1381|## Phase 7 — LiteLLM (AI Gateway)
|
|
1382|
|
|
1383|**Host:** PlausibleDeniability
|
|
1384|**Stack directory:** `/mnt/docker-ssd/docker/compose/ai/` (same stack as Qdrant)
|
|
1385|**Why:** Replaces manual multi-model routing with a unified OpenAI-compatible API. Every app (OpenWebUI, Continue.dev, n8n, Paperless summarization) points at one endpoint instead of juggling three Ollama IPs.
|
|
1386|
|
|
1387|### 7.1 — What LiteLLM Does
|
|
1388|
|
|
1389|LiteLLM sits in front of your three Ollama instances and presents a single OpenAI-compatible API at `http://litellm:4000`. It handles:
|
|
1390|
|
|
- **Model routing** — current reliable default is N.O.M.A.D.-local Ollama for background/automation paths, with PD kept available for shared light-tier use and Rocinante treated as optional/manual heavy capacity.
|
|
- **Failover** — avoid designs that require Rocinante to be online; degrade to N.O.M.A.D.-local or explicitly operator-invoked PD paths instead of automatic dependence on a personal PC.
|
|
- **Load balancing** — distribute requests across instances running the same model when you have multiple dependable backends, but do not assume Rocinante qualifies as one.
|
|
1394|- **Usage tracking** — logs token counts, latency, and costs per model/user via its built-in database
|
|
1395|
|
|
1396|### 7.2 — Scaffold Directories
|
|
1397|
|
|
1398|```bash
|
|
1399|# Config on SSD (SQLite DB for usage tracking)
|
|
1400|sudo mkdir -p /mnt/docker-ssd/docker/appdata/litellm
|
|
1401|
|
|
1402|# LiteLLM config file
|
|
1403|sudo mkdir -p /mnt/docker-ssd/docker/compose/ai/litellm
|
|
1404|```
|
|
1405|
|
|
1406|### 7.3 — LiteLLM Config
|
|
1407|
|
|
1408|Create `/mnt/docker-ssd/docker/compose/ai/litellm/config.yaml`:
|
|
1409|
|
|
1410|```yaml
|
|
1411|model_list:
|
|
1412| # ---- ROCINANTE (RTX 4090, 24GB) — heavy reasoning ----
|
|
1413| - model_name: "heavy"
|
|
1414| litellm_params:
|
|
1415| model: "ollama/qwen3:32b"
|
|
1416| api_base: "http://10.5.1.ROCINANTE:11434"
|
|
1417| timeout: 300
|
|
1418| stream_timeout: 300
|
|
1419| model_info:
|
|
1420| description: "Heavy reasoning, long context, complex code"
|
|
1421|
|
|
1422| - model_name: "heavy"
|
|
1423| litellm_params:
|
|
1424| model: "ollama/deepseek-r1:32b"
|
|
1425| api_base: "http://10.5.1.ROCINANTE:11434"
|
|
1426| timeout: 300
|
|
1427| stream_timeout: 300
|
|
1428| model_info:
|
|
1429| description: "Deep reasoning fallback on ROCINANTE"
|
|
1430|
|
|
1431| # ---- PlausibleDeniability (RTX 2080 Ti, 11GB) — general ----
|
|
1432| - model_name: "medium"
|
|
1433| litellm_params:
|
|
1434| model: "ollama/qwen2.5:14b"
|
|
1435| api_base: "http://host.docker.internal:11434"
|
|
1436| timeout: 120
|
|
1437| stream_timeout: 120
|
|
1438| model_info:
|
|
1439| description: "General homelab assistant, RAG queries"
|
|
1440|
|
|
1441| # ---- N.O.M.A.D. (GTX 1080, 8GB) — lightweight ----
|
|
1442| - model_name: "light"
|
|
1443| litellm_params:
|
|
1444| model: "ollama/phi4"
|
|
1445| api_base: "http://10.5.30.7:11434"
|
|
1446| timeout: 60
|
|
1447| stream_timeout: 60
|
|
1448| model_info:
|
|
1449| description: "Fast local inference, lightweight tasks"
|
|
1450|
|
|
1451| - model_name: "light"
|
|
1452| litellm_params:
|
|
1453| model: "ollama/llama3.2:3b"
|
|
1454| api_base: "http://10.5.30.7:11434"
|
|
1455| timeout: 60
|
|
1456| stream_timeout: 60
|
|
1457| model_info:
|
|
1458| description: "Ultra-light fallback on N.O.M.A.D."
|
|
1459|
|
|
1460| # ---- Embeddings ----
|
|
1461| - model_name: "embed"
|
|
1462| litellm_params:
|
|
1463| model: "ollama/nomic-embed-text"
|
|
1464| api_base: "http://host.docker.internal:11434"
|
|
1465| model_info:
|
|
1466| description: "Text embeddings for RAG pipeline"
|
|
1467|
|
|
1468| # ---- Direct model access (bypass routing) ----
|
|
1469| # These let you request a specific model by its full name
|
|
1470| - model_name: "ollama/qwen3:32b"
|
|
1471| litellm_params:
|
|
1472| model: "ollama/qwen3:32b"
|
|
1473| api_base: "http://10.5.1.ROCINANTE:11434"
|
|
1474|
|
|
1475| - model_name: "ollama/qwen2.5:14b"
|
|
1476| litellm_params:
|
|
1477| model: "ollama/qwen2.5:14b"
|
|
1478| api_base: "http://host.docker.internal:11434"
|
|
1479|
|
|
1480| - model_name: "ollama/phi4"
|
|
1481| litellm_params:
|
|
1482| model: "ollama/phi4"
|
|
1483| api_base: "http://10.5.30.7:11434"
|
|
1484|
|
|
1485|litellm_settings:
|
|
1486| drop_params: true
|
|
1487| set_verbose: false
|
|
1488| request_timeout: 300
|
|
1489| num_retries: 2
|
|
1490| retry_after: 5
|
|
1491| allowed_fails: 3
|
|
1492| cooldown_time: 60
|
|
1493|
|
|
1494|general_settings:
|
|
1495| master_key: "os.environ/LITELLM_MASTER_KEY"
|
|
1496|```
|
|
1497|
|
|
1498|> **DEPLOYED NOTE (2026-05-05):** Do NOT add `database_url` to general_settings — newer LiteLLM versions include Prisma ORM that requires PostgreSQL, and adding any database_url causes a crash loop. Without it, LiteLLM runs in config-only mode which is fine for homelab use. The `master_key` uses `os.environ/LITELLM_MASTER_KEY` syntax to read from the container's environment variable (set via .env → docker compose).
|
|
1499|
|
|
1500|> **Routing explained:** Multiple entries with the same `model_name` (like "heavy") enable load balancing and failover within that tier. When you request model `heavy`, LiteLLM picks the healthiest deployment. The direct-access entries (like `ollama/qwen3:32b`) let you bypass the tier system and target a specific model when needed.
|
|
1501|
|
|
1502|> **Replace IPs:** Update `10.5.1.ROCINANTE` with ROCINANTE's actual LAN or Tailscale IP. PD uses `host.docker.internal` since LiteLLM runs on PD alongside Ollama.
|
|
1503|
|
|
1504|### 7.4 — docker-compose.yaml (updated ai stack)
|
|
1505|
|
|
1506|Update `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml` to add LiteLLM:
|
|
1507|
|
|
1508|```yaml
|
|
1509|name: ai
|
|
1510|
|
|
1511|services:
|
|
1512| qdrant:
|
|
1513| image: qdrant/qdrant:v1.14.0
|
|
1514| container_name: qdrant
|
|
1515| restart: unless-stopped
|
|
1516| ports:
|
|
1517| - "6333:6333" # REST API
|
|
1518| - "6334:6334" # gRPC
|
|
1519| environment:
|
|
1520| TZ: ${TZ}
|
|
1521| QDRANT__SERVICE__GRPC_PORT: 6334
|
|
1522| QDRANT__STORAGE__STORAGE_PATH: /qdrant/storage
|
|
1523| QDRANT__STORAGE__SNAPSHOTS_PATH: /qdrant/snapshots
|
|
1524| volumes:
|
|
1525| - /mnt/docker-ssd/docker/appdata/qdrant/storage:/qdrant/storage
|
|
1526| - /mnt/docker-ssd/docker/appdata/qdrant/snapshots:/qdrant/snapshots
|
|
1527| networks:
|
|
1528| - ai-services
|
|
1529| - default
|
|
1530|
|
|
1531| litellm:
|
|
1532| image: ghcr.io/berriai/litellm:main-latest
|
|
1533| container_name: litellm
|
|
1534| restart: unless-stopped
|
|
1535| ports:
|
|
1536| - "4000:4000"
|
|
1537| environment:
|
|
1538| TZ: ${TZ}
|
|
1539| LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
|
|
1540| volumes:
|
|
1541| - /mnt/docker-ssd/docker/compose/ai/litellm/config.yaml:/app/config.yaml:ro
|
|
1542| - /mnt/docker-ssd/docker/appdata/litellm:/app/data
|
|
1543| command: ["--config", "/app/config.yaml", "--port", "4000"]
|
|
1544| extra_hosts:
|
|
1545| - "host.docker.internal:host-gateway"
|
|
1546| networks:
|
|
1547| - ai-services
|
|
1548| - default
|
|
1549|
|
|
1550| reranker:
|
|
1551| image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
|
|
1552| container_name: reranker
|
|
1553| restart: unless-stopped
|
|
1554| ports:
|
|
1555| - "8787:80"
|
|
1556| environment:
|
|
1557| MODEL_ID: ${RERANKER_MODEL}
|
|
1558| volumes:
|
|
1559| - /mnt/docker-ssd/docker/appdata/reranker:/data
|
|
1560| networks:
|
|
1561| - ai-services
|
|
1562| - default
|
|
1563|
|
|
1564| whisper:
|
|
1565| image: fedirz/faster-whisper-server:latest-cuda
|
|
1566| container_name: whisper
|
|
1567| restart: unless-stopped
|
|
1568| ports:
|
|
1569| - "8786:8000"
|
|
1570| environment:
|
|
1571| TZ: ${TZ}
|
|
1572| WHISPER__MODEL: ${WHISPER_MODEL}
|
|
1573| WHISPER__DEVICE: cuda
|
|
1574| WHISPER__COMPUTE_TYPE: float16
|
|
1575| volumes:
|
|
1576| - /mnt/docker-ssd/docker/appdata/whisper:/root/.cache/huggingface
|
|
1577| deploy:
|
|
1578| resources:
|
|
1579| reservations:
|
|
1580| devices:
|
|
1581| - driver: nvidia
|
|
1582| count: 1
|
|
1583| capabilities: [gpu]
|
|
1584| networks:
|
|
1585| - ai-services
|
|
1586| - default
|
|
1587|
|
|
1588| # -------------------------------------------------------
|
|
1589| # Ollama and OpenWebUI go here when deployed.
|
|
1590| # They share this stack and the default + ai-services networks.
|
|
1591| # OpenWebUI → Qdrant at http://qdrant:6333
|
|
1592| # OpenWebUI → LiteLLM at http://litellm:4000 (or direct Ollama)
|
|
1593| # -------------------------------------------------------
|
|
1594|
|
|
1595| # ollama:
|
|
1596| # image: ollama/ollama:latest
|
|
1597| # container_name: ollama
|
|
1598| # ...
|
|
1599|
|
|
1600| # openwebui:
|
|
1601| # image: ghcr.io/open-webui/open-webui:main
|
|
1602| # container_name: openwebui
|
|
1603| # ...
|
|
1604|
|
|
1605|networks:
|
|
1606| ai-services:
|
|
1607| name: ai-services
|
|
1608|```
|
|
1609|
|
|
1610|> **GPU sharing note:** On PD, the RTX 2080 Ti is shared between Plex (hardware transcoding), Immich ML, and Ollama. faster-whisper will also need the GPU when processing audio. These workloads are bursty (not constant), so time-sharing the GPU is viable — but be aware that a large Whisper transcription job will temporarily impact Ollama inference latency. If this becomes an issue, consider running Whisper on N.O.M.A.D.'s GTX 1080 instead and change `whisper`'s Ollama-style routing accordingly.
|
|
1611|
|
|
1612|> **Reranker on CPU:** The reranker uses the `cpu-latest` image variant intentionally — do NOT pin to `cpu-1.5` as it has an hf-hub compatibility bug. Reranking is a lightweight operation (scoring ~50 chunks takes <1s on CPU) and doesn't justify competing for GPU VRAM. The model loads via safetensors fallback since no ONNX files exist for bge-reranker-v2-m3. Max batch size is 4 on CPU. If you find latency is an issue, a GPU variant exists (`ghcr.io/huggingface/text-embeddings-inference:turing-latest` for your 2080 Ti).
|
|
1613|
|
|
1614|### 7.5 — .env.example (updated ai stack)
|
|
1615|
|
|
1616|```bash
|
|
1617|# /mnt/docker-ssd/docker/compose/ai/.env.example
|
|
1618|
|
|
1619|TZ=America/New_York
|
|
1620|
|
|
1621|# LiteLLM
|
|
1622|LITELLM_MASTER_KEY=sk-CHANGE_ME
|
|
1623|
|
|
1624|# Reranker
|
|
1625|RERANKER_MODEL=BAAI/bge-reranker-v2-m3
|
|
1626|
|
|
1627|# Whisper
|
|
1628|WHISPER_MODEL=Systran/faster-distil-whisper-large-v3
|
|
1629|```
|
|
1630|
|
|
1631|### 7.6 — Validate & Deploy
|
|
1632|
|
|
1633|```bash
|
|
1634|cd /mnt/docker-ssd/docker/compose/ai
|
|
1635|cp .env.example .env
|
|
1636|nano .env # Set LITELLM_MASTER_KEY=$(openssl rand -hex 32)
|
|
1637|
|
|
1638|docker compose --env-file .env config
|
|
1639|docker compose --env-file .env up -d
|
|
1640|```
|
|
1641|
|
|
1642|### 7.7 — Post-Deploy Verification
|
|
1643|
|
|
1644|```bash
|
|
1645|# All containers running?
|
|
1646|docker ps --filter name=qdrant --filter name=litellm --filter name=reranker --filter name=whisper
|
|
1647|
|
|
1648|# LiteLLM health?
|
|
1649|curl -s http://localhost:4000/health
|
|
1650|
|
|
1651|# LiteLLM can see all models?
|
|
1652|curl -s http://localhost:4000/v1/models \
|
|
1653| -H "Authorization: Bearer *** | python3 -m json.tool
|
|
1654|
|
|
1655|# Test a chat completion through LiteLLM
|
|
1656|curl -s http://localhost:4000/v1/chat/completions \
|
|
1657| -H "Authorization: Bearer *** \
|
|
1658| -H "Content-Type: application/json" \
|
|
1659| -d '{
|
|
1660| "model": "medium",
|
|
1661| "messages": [{"role": "user", "content": "Hello, which model are you?"}]
|
|
1662| }' | python3 -m json.tool
|
|
1663|
|
|
1664|# Test embeddings through LiteLLM
|
|
1665|curl -s http://localhost:4000/v1/embeddings \
|
|
1666| -H "Authorization: Bearer *** \
|
|
1667| -H "Content-Type: application/json" \
|
|
1668| -d '{
|
|
1669| "model": "embed",
|
|
1670| "input": "test embedding"
|
|
1671| }' | python3 -m json.tool
|
|
1672|```
|
|
1673|
|
|
1674|### 7.8 — Integration Updates
|
|
1675|
|
|
1676|With LiteLLM deployed, all apps should point at `http://litellm:4000` (within the `ai-services` network) or `http://PD_IP:4000` (from external hosts) instead of direct Ollama endpoints.
|
|
1677|
|
|
1678|**OpenWebUI:** Settings → Connections → add OpenAI-compatible endpoint:
|
|
1679|- URL: `http://litellm:4000/v1`
|
|
1680|- API Key: your `LITELLM_MASTER_KEY`
|
|
1681|- This gives OpenWebUI access to all models across all three machines through one connection
|
|
1682|
|
|
1683|**Continue.dev:** Update `~/.continue/config.json`:
|
|
1684|```json
|
|
1685|{
|
|
1686| "models": [
|
|
1687| {
|
|
1688| "title": "Heavy (ROCINANTE)",
|
|
1689| "provider": "openai",
|
|
1690| "model": "heavy",
|
|
1691| "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
|
|
1692| "apiKey": "YOUR_MASTER_KEY"
|
|
1693| },
|
|
1694| {
|
|
1695| "title": "Medium (PD)",
|
|
1696| "provider": "openai",
|
|
1697| "model": "medium",
|
|
1698| "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
|
|
1699| "apiKey": "YOUR_MASTER_KEY"
|
|
1700| }
|
|
1701| ],
|
|
1702| "tabAutocompleteModel": {
|
|
1703| "title": "Light (N.O.M.A.D.)",
|
|
1704| "provider": "openai",
|
|
1705| "model": "light",
|
|
1706| "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
|
|
1707| "apiKey": "YOUR_MASTER_KEY"
|
|
1708| }
|
|
1709|}
|
|
1710|```
|
|
1711|
|
|
1712|> **Key change:** Continue.dev now uses `provider: "openai"` instead of `provider: "ollama"` and points at LiteLLM. One IP to remember, and if you add or move models between machines, you only update `config.yaml` — not every app.
|
|
1713|
|
|
1714|**n8n workflows:** All HTTP Request nodes that call Ollama directly should be updated:
|
|
1715|- Old: `http://10.5.30.7:11434/api/generate` (N.O.M.A.D.)
|
|
1716|- New: `http://litellm:4000/v1/chat/completions` with `model: "light"`
|
|
1717|- n8n is on the `ai-services` network, so it reaches LiteLLM by Docker DNS
|
|
1718|
|
|
1719|**n8n Multi-Model Query Router workflow:** This workflow is now **simplified dramatically** — instead of a Switch node routing to three different Ollama IPs, it becomes a single HTTP Request node that passes the model tier name (`light`, `medium`, `heavy`) to LiteLLM and lets the gateway handle routing. The Switch node logic can be removed entirely.
|
|
1720|
|
|
1721|---
|
|
1722|
|
|
1723|## Phase 8 — Reranker (RAG Quality)
|
|
1724|
|
|
1725|**Host:** PlausibleDeniability (deployed as part of the `ai` stack in Phase 7)
|
|
1726|**Service:** `reranker` container (already in the compose above)
|
|
1727|**Why:** Dramatically improves RAG answer quality by filtering out noisy retrieval results before they reach the LLM.
|
|
1728|
|
|
1729|### 8.1 — How Reranking Works
|
|
1730|
|
|
1731|Without a reranker, your RAG pipeline does:
|
|
1732|```
|
|
1733|Query → embed → Qdrant top-10 by vector similarity → all 10 chunks go to LLM
|
|
1734|```
|
|
1735|
|
|
1736|The problem: vector similarity often returns "close but irrelevant" chunks. The LLM gets noisy context and hallucinates.
|
|
1737|
|
|
1738|With a reranker:
|
|
1739|```
|
|
1740|Query → embed → Qdrant top-50 by vector similarity → reranker scores each (query, chunk) pair → top-5 by relevance go to LLM
|
|
1741|```
|
|
1742|
|
|
1743|The reranker is a cross-encoder that reads the full query AND each chunk together, producing a much more accurate relevance score than vector distance alone. It over-retrieves cheaply from Qdrant, then precisely filters.
|
|
1744|
|
|
1745|### 8.2 — Scaffold Directories
|
|
1746|
|
|
1747|```bash
|
|
1748|# Model cache on SSD (reranker model is ~1.1GB, downloaded on first start)
|
|
1749|sudo mkdir -p /mnt/docker-ssd/docker/appdata/reranker
|
|
1750|```
|
|
1751|
|
|
1752|### 8.3 — Post-Deploy Verification
|
|
1753|
|
|
1754|```bash
|
|
1755|# Container running?
|
|
1756|docker ps --filter name=reranker
|
|
1757|
|
|
1758|# Health check?
|
|
1759|curl -s http://localhost:8787/health
|
|
1760|
|
|
1761|# Test reranking
|
|
1762|curl -s http://localhost:8787/rerank \
|
|
1763| -H "Content-Type: application/json" \
|
|
1764| -d '{
|
|
1765| "query": "How do I restart a Docker container?",
|
|
1766| "texts": [
|
|
1767| "Use docker restart <container_name> to restart a running container.",
|
|
1768| "Docker was founded in 2013 by Solomon Hykes.",
|
|
1769| "The docker compose down command stops and removes containers.",
|
|
1770| "Kubernetes pods can be restarted by deleting them."
|
|
1771| ]
|
|
1772| }' | python3 -m json.tool
|
|
1773|# Expected: the first text scores highest, second scores lowest
|
|
1774|```
|
|
1775|
|
|
1776|### 8.4 — n8n RAG Pipeline Integration
|
|
1777|
|
|
1778|Update the "Qdrant Index Updater" and any RAG query workflows to include a reranking step.
|
|
1779|
|
|
1780|**Updated RAG Query Workflow (for OpenWebUI or any n8n-based query):**
|
|
1781|```
|
|
1782|Trigger: Webhook node (POST with query)
|
|
1783|Step 1: HTTP Request → LiteLLM /v1/embeddings
|
|
1784| - model: "embed"
|
|
1785| - input: query text
|
|
1786|Step 2: HTTP Request → Qdrant API (POST /collections/{name}/points/search)
|
|
1787| - vector: embedding from step 1
|
|
1788| - limit: 50 (over-retrieve)
|
|
1789|Step 3: HTTP Request → Reranker (POST http://reranker:80/rerank)
|
|
1790| - query: original query text
|
|
1791| - texts: array of 50 chunk texts from Qdrant results
|
|
1792|Step 4: Code node → take top 5 by reranker score, format as context
|
|
1793|Step 5: HTTP Request → LiteLLM /v1/chat/completions
|
|
1794| - model: "medium" (or "heavy" for complex queries)
|
|
1795| - messages: system prompt with top-5 context + user query
|
|
1796|Step 6: Return response via webhook
|
|
1797|```
|
|
1798|
|
|
1799|> **Key difference from the original plan:** Step 2 now retrieves 50 results instead of 10, and Step 3 (reranking) filters down to the best 5. This "over-retrieve then rerank" pattern is the standard approach for production RAG systems.
|
|
1800|
|
|
1801|---
|
|
1802|
|
|
1803|## Phase 9 — faster-whisper (Speech-to-Text)
|
|
1804|
|
|
1805|**Host:** PlausibleDeniability (deployed as part of the `ai` stack in Phase 7)
|
|
1806|**Service:** `whisper` container (already in the compose above)
|
|
1807|**Why:** Replaces shelved Scriberr with an OpenAI-compatible STT API. No SQLite dependency, no ZFS/ACL issues.
|
|
1808|
|
|
1809|### 9.1 — Scaffold Directories
|
|
1810|
|
|
1811|```bash
|
|
1812|# Model cache on SSD (Whisper models are 1-3GB)
|
|
1813|sudo mkdir -p /mnt/docker-ssd/docker/appdata/whisper
|
|
1814|```
|
|
1815|
|
|
1816|### 9.2 — Post-Deploy Verification
|
|
1817|
|
|
1818|```bash
|
|
1819|# Container running?
|
|
1820|docker ps --filter name=whisper
|
|
1821|
|
|
1822|# Health check?
|
|
1823|curl -s http://localhost:8786/health
|
|
1824|
|
|
1825|# Test transcription with a sample audio file
|
|
1826|curl -s http://localhost:8786/v1/audio/transcriptions \
|
|
1827| -F "file=@/path/to/test-audio.wav" \
|
|
1828| -F "model=Systran/faster-distil-whisper-large-v3" \
|
|
1829| | python3 -m json.tool
|
|
1830|```
|
|
1831|
|
|
1832|### 9.3 — n8n Integration Workflows
|
|
1833|
|
|
1834|**Workflow: Voice Note → Text → Summary**
|
|
1835|```
|
|
1836|Trigger: Webhook node (POST with audio file in body)
|
|
1837|Step 1: HTTP Request → Whisper (POST http://whisper:8000/v1/audio/transcriptions)
|
|
1838| - Multipart form with audio file
|
|
1839| - response_format: "json"
|
|
1840|Step 2: HTTP Request → LiteLLM /v1/chat/completions
|
|
1841| - model: "medium"
|
|
1842| - Prompt: "Summarize this voice note concisely: {transcript}"
|
|
1843|Step 3: HTTP Request → Gotify → push summary to phone
|
|
1844|Step 4: (Optional) Postgres node → log transcript and summary
|
|
1845|```
|
|
1846|
|
|
1847|**Workflow: Audio File → Paperless Document**
|
|
1848|```
|
|
1849|Trigger: Webhook or filesystem watcher
|
|
1850|Step 1: HTTP Request → Whisper → get transcript
|
|
1851|Step 2: Code node → format transcript as text document
|
|
1852|Step 3: HTTP Request → Paperless API (POST /api/documents/post_document/)
|
|
1853| - Upload transcript as .txt
|
|
1854| - Tag: "transcription"
|
|
1855|Step 4: Gotify notification: "Audio transcribed and filed: {title}"
|
|
1856|```
|
|
1857|
|
|
1858|**Home Assistant Voice Integration:**
|
|
1859|If you want HA voice commands, Whisper can serve as the STT backend:
|
|
1860|1. In Home Assistant → Settings → Voice Assistants
|
|
1861|2. Add speech-to-text provider: "Whisper" at `http://PD_IP:8786`
|
|
1862|3. Pair with Piper TTS (future addition) for full voice assistant loop
|
|
1863|
|
|
1864|### 9.4 — GPU vs CPU Considerations
|
|
1865|
|
|
1866|The compose file above uses the GPU variant with CUDA. If GPU contention with Ollama becomes an issue:
|
|
1867|
|
|
1868|**Option A — CPU fallback on PD:**
|
|
1869|Change the image from `latest-cuda` to `fedirz/faster-whisper-server:latest-cpu` and remove the `deploy.resources` block and the CUDA environment variables (`WHISPER__DEVICE`, `WHISPER__COMPUTE_TYPE`). Transcription will be slower (~4x real-time instead of ~20x) but won't compete for VRAM.
|
|
1870|
|
|
1871|**Option B — Move to N.O.M.A.D.:**
|
|
1872|Run Whisper on N.O.M.A.D.'s GTX 1080 which is less contested. Add it to N.O.M.A.D.'s compose or run standalone. N.O.M.A.D.'s 8GB VRAM handles Whisper models comfortably since phi4 doesn't fill it.
|
|
1873|
|
|
1874|---
|
|
1875|
|
|
1876|## Additional Tools Setup
|
|
1877|
|
|
1878|### Continue.dev (Local AI Code Completion)
|
|
1879|
|
|
1880|1. Install the Continue extension in VS Code
|
|
1881|2. Create/edit `~/.continue/config.json`:
|
|
1882|
|
|
1883|**With LiteLLM (recommended — see Phase 7):**
|
|
1884|```json
|
|
1885|{
|
|
1886| "models": [
|
|
1887| {
|
|
1888| "title": "Heavy (ROCINANTE)",
|
|
1889| "provider": "openai",
|
|
1890| "model": "heavy",
|
|
1891| "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
|
|
1892| "apiKey": "YOUR_LITELLM_MASTER_KEY"
|
|
1893| },
|
|
1894| {
|
|
1895| "title": "Medium (PD)",
|
|
1896| "provider": "openai",
|
|
1897| "model": "medium",
|
|
1898| "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
|
|
1899| "apiKey": "YOUR_LITELLM_MASTER_KEY"
|
|
1900| }
|
|
1901| ],
|
|
1902| "tabAutocompleteModel": {
|
|
1903| "title": "Light (N.O.M.A.D.)",
|
|
1904| "provider": "openai",
|
|
1905| "model": "light",
|
|
1906| "apiBase": "http://PD_TAILSCALE_IP:4000/v1",
|
|
1907| "apiKey": "YOUR_LITELLM_MASTER_KEY"
|
|
1908| }
|
|
1909|}
|
|
1910|```
|
|
1911|
|
|
1912|> One IP, one API key, all three machines. If you add or move models, update LiteLLM's `config.yaml` — not every app.
|
|
1913|
|
|
1914|**Without LiteLLM (direct Ollama, if Phase 7 is not yet deployed):**
|
|
1915|```json
|
|
1916|{
|
|
1917| "models": [
|
|
1918| {
|
|
1919| "title": "N.O.M.A.D. - qwen2.5:3b",
|
|
1920| "provider": "ollama",
|
|
1921| "model": "qwen2.5:3b",
|
|
1922| "apiBase": "http://NOMAD_LOCAL_OR_TAILSCALE_IP:11434"
|
|
1923| },
|
|
1924| {
|
|
1925| "title": "PD - shared light tier (manual / fallback)",
|
|
1926| "provider": "ollama",
|
|
1927| "model": "qwen2.5:14b",
|
|
1928| "apiBase": "http://PD_TAILSCALE_IP:11434"
|
|
1929| },
|
|
1930| {
|
|
1931| "title": "ROCINANTE - opportunistic heavy",
|
|
1932| "provider": "ollama",
|
|
1933| "model": "qwen3:32b",
|
|
1934| "apiBase": "http://ROCINANTE_TAILSCALE_IP:11434"
|
|
1935| }
|
|
1930| ],
|
|
1931| "tabAutocompleteModel": {
|
|
1932| "title": "N.O.M.A.D. - phi4",
|
|
1933| "provider": "ollama",
|
|
1934| "model": "phi4",
|
|
1935| "apiBase": "http://NOMAD_TAILSCALE_IP:11434"
|
|
1936| }
|
|
1937|}
|
|
1938|```
|
|
1939|
|
|
1940|### Obsidian + Gitea Sync
|
|
1941|
|
|
1942|1. In Obsidian, install the "Obsidian Git" community plugin
|
|
1943|2. Initialize your vault as a git repo:
|
|
1944| ```bash
|
|
1945| cd /path/to/obsidian/vault
|
|
1946| git init
|
|
1947| git remote add origin http://PD_IP:3000/fizzlepoof/obsidian-vault.git
|
|
1948| ```
|
|
1949|3. In Obsidian Git settings:
|
|
1950| - Auto backup interval: 10 minutes
|
|
1951| - Pull on startup: enabled
|
|
1952|4. Create the repo in Gitea first: `http://PD_IP:3000` → New Repository → `obsidian-vault`
|
|
1953|5. Add a Gitea webhook to trigger the n8n "Qdrant Index Updater" workflow (see Phase 3)
|
|
1954|
|
|
1955|### Homepage Ollama Widget
|
|
1956|
|
|
1957|Add to your Homepage configuration (`/mnt/tank/docker/appdata/homepage/services.yaml`):
|
|
1958|
|
|
1959|```yaml
|
|
1960|- AI:
|
|
1961| - Ollama (PD):
|
|
1962| icon: ollama.svg
|
|
1963| href: http://PD_IP:11434
|
|
1964| widget:
|
|
1965| type: ollama
|
|
1966| url: http://PD_IP:11434
|
|
1967| - Ollama (ROCINANTE):
|
|
1968| icon: ollama.svg
|
|
1969| href: http://ROCINANTE_IP:11434
|
|
1970| widget:
|
|
1971| type: ollama
|
|
1972| url: http://ROCINANTE_IP:11434
|
|
1973|```
|
|
1974|
|
|
1975|---
|
|
1976|
|
|
1977|## Deployment Order Summary
|
|
1978|
|
|
1979|```
|
|
1980|Week 1: Phase 1 — Gotify
|
|
1981| └─ Deploy, create app tokens, install phone app
|
|
1982| └─ Test: send manual notification via API
|
|
1983|
|
|
1984|Week 1: Phase 2 — Qdrant
|
|
1985| └─ Deploy, verify REST API
|
|
1986| └─ Create initial collections (empty, ready for n8n)
|
|
1987|
|
|
1988|Week 2: Phase 3 — n8n
|
|
1989| └─ Deploy, create admin account
|
|
1990| └─ Build workflows incrementally:
|
|
1991| Day 1: Gitea commit → Gotify (simplest, proves the pipeline)
|
|
1992| Day 2: Sonarr/Radarr → TMDB → Gotify + Discord
|
|
1993| Day 3: Tautulli play logging + weekly digest
|
|
1994| Day 4: Uptime Kuma enhanced alerts
|
|
1995| Day 5: ZFS pool monitoring
|
|
1996| Day 6: Multi-model query router
|
|
1997| Day 7: Qdrant index updater
|
|
1998|
|
|
1999|Week 3: Phase 4 — Paperless-NGX
|
|
2000| └─ Deploy, ingest test documents
|
|
2001| |