docs: add full homelab documentation structure
This commit is contained in:
Paccoco
0
docs/README.md
Normal file
0
docs/README.md
Normal file
107
docs/architecture/ARCHITECTURE_OVERVIEW.md
Normal file
107
docs/architecture/ARCHITECTURE_OVERVIEW.md
Normal file
@@ -0,0 +1,107 @@
|
||||
# Homelab Architecture Overview
|
||||
|
||||
## Primary Docker Host
|
||||
Host: `PlausibleDeniability` (TrueNAS Scale)
|
||||
|
||||
Production Docker workloads run on PlausibleDeniability via standalone Docker Compose stacks.
|
||||
TrueNAS Apps (ix-app format) have been fully migrated to compose stacks.
|
||||
|
||||
## Repository Layout
|
||||
Repo root: `/mnt/docker-ssd/docker/compose`
|
||||
Remote: `git@github.com:Paccoco/truenas-stacks.git`
|
||||
Gitea mirror: `https://gitea.paccoco.com/fizzlepoof/truenas-stacks.git`
|
||||
|
||||
Stack directories:
|
||||
- `databases/` = shared-postgres, shared-mariadb, shared-redis — deploy FIRST
|
||||
- `infrastructure/` = newt, homepage, uptime-kuma, dockhand, netdata, tailscale — deploy SECOND
|
||||
- `media/` = plex, tautulli, audiobookshelf, seerr, calibre-web-automated
|
||||
- `photos/` = immich server + machine learning
|
||||
- `home/` = kitchenowl, donetick, dakboard bridge, shlink, shlink-web-client, qui
|
||||
- `dev/` = gitea, rackpeek
|
||||
- `ai/` = ollama, openwebui
|
||||
- `meshtastic/` = meshmonitor, mqtt-proxy, tileserver, meshtastic-map
|
||||
|
||||
## Storage Model
|
||||
|
||||
### Local SSD
|
||||
Mounted at: `/mnt/docker-ssd`
|
||||
- Docker data root: `/mnt/docker-ssd/docker/docker-data`
|
||||
- Databases: `/mnt/docker-ssd/docker/databases/`
|
||||
- Write-heavy appdata: `/mnt/docker-ssd/docker/appdata/`
|
||||
- GPU/model storage
|
||||
- Apps that perform chown or aggressive startup writes
|
||||
- SQLite-backed apps
|
||||
|
||||
Notable appdata on SSD:
|
||||
- plex, tautulli, audiobookshelf, seerr, dockhand, rackpeek
|
||||
- immich, immich-ml, ollama, openwebui
|
||||
- shlink, gitea, netdata, tailscale, uptime-kuma
|
||||
|
||||
### TrueNAS Pool (tank)
|
||||
Mounted at: `/mnt/tank`
|
||||
- General appdata: `/mnt/tank/docker/appdata/`
|
||||
- Backups: `/mnt/tank/docker/backups/`
|
||||
- Downloads: `/mnt/tank/docker/downloads/`
|
||||
|
||||
Notable appdata on tank:
|
||||
- homepage, newt, dockhand (config only)
|
||||
- kitchenowl, donetick, qui
|
||||
- calibre-web-automated config
|
||||
|
||||
### Unraid NFS
|
||||
Mounted at: `/mnt/unraid`
|
||||
- Media libraries: `/mnt/unraid/data/media/`
|
||||
- Immich media: `/mnt/unraid/immich`
|
||||
|
||||
## Networks
|
||||
- `ix-databases_shared-databases` — created by databases stack; project name `ix-databases` set in compose
|
||||
- `pangolin` — created by newt at runtime; used by all externally exposed services
|
||||
|
||||
## Active Stacks (as of 2026-05-05)
|
||||
|
||||
### databases/
|
||||
- shared-postgres (immich-optimized image with vectorchord + pgvector)
|
||||
- shared-mariadb
|
||||
- shared-redis
|
||||
|
||||
### infrastructure/
|
||||
- newt, homepage, dockhand, uptime-kuma, netdata, tailscale
|
||||
|
||||
### media/
|
||||
- ix-plex-plex-1 (GPU passthrough, RTX 2080 Ti)
|
||||
- ix-tautulli-tautulli-1
|
||||
- ix-audiobookshelf-audiobookshelf-1
|
||||
- seerr (Postgres backend)
|
||||
- calibre-web-automated
|
||||
|
||||
### photos/
|
||||
- ix-immich-immich-server-1
|
||||
- ix-immich-immich-machine-learning-1 (GPU, CUDA)
|
||||
|
||||
### home/
|
||||
- kitchenowl, donetick, donetick-dakboard-bridge, qui, shlink, shlink-web-client
|
||||
|
||||
### dev/
|
||||
- gitea, ix-rackpeek-rackpeek-1
|
||||
|
||||
### ai/
|
||||
- ollama (GPU, RTX 2080 Ti, port 11434)
|
||||
- openwebui (Postgres backend, port 8282)
|
||||
|
||||
### meshtastic/
|
||||
- meshmonitor, meshmonitor-mqtt-proxy, meshmonitor-tileserver, meshtastic-map
|
||||
|
||||
## Planned / Not Yet Deployed
|
||||
- dispatcharr, gotify, bytestash, kima-hub, n8n, qdrant, paperless-ngx, home-assistant
|
||||
|
||||
## Shelved
|
||||
- scriberr — SQLite incompatible with ZFS nfsv4 ACLs
|
||||
|
||||
## Design Principles
|
||||
- deterministic storage paths
|
||||
- no anonymous volumes
|
||||
- stack-local .env for secrets; .env.example committed with placeholders
|
||||
- SSD for write-heavy / DB / SQLite / GPU workloads
|
||||
- tank NFS for general appdata and configs
|
||||
- Unraid NFS for media only
|
||||
- databases stack uses `name: ix-databases` to maintain network name compatibility
|
||||
77
docs/architecture/HARDWARE_INVENTORY.md
Normal file
77
docs/architecture/HARDWARE_INVENTORY.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# Hardware Inventory
|
||||
|
||||
## Serenity (NAS / Media Ingestion)
|
||||
- **OS:** Unraid 7.2.4
|
||||
- **IP:** 10.5.1.5 | Tailscale: 100.94.87.79
|
||||
- **CPU:** Xeon X5690
|
||||
- **RAM:** 102GB
|
||||
- **GPU:** None (Matrox IPMI only)
|
||||
- **UPS:** None
|
||||
|
||||
### Storage
|
||||
| Pool | Type | Drives | Usable | Notes |
|
||||
|------|------|--------|--------|-------|
|
||||
| malcolm | RAIDZ1 | 6× 10TB HDD | ~40TB | Heavily utilized |
|
||||
| cache | ZFS single | 1× Kingston SNV2S 1TB NVMe | ~1TB | Appdata/Docker |
|
||||
|
||||
### Key Shares
|
||||
| Share | Size | Purpose |
|
||||
|-------|------|---------|
|
||||
| data | 35TB | Media library |
|
||||
| immich | 62GB | Immich photo storage |
|
||||
| nextcloud | 7.4GB | Nextcloud data |
|
||||
| Tesla/dashcam | 1.6TB | Dashcam footage |
|
||||
| fizzlebang | 17GB | Personal files |
|
||||
| isos | 22GB | ISO images |
|
||||
| Syncthing | 4.6GB | Syncthing data |
|
||||
| manndra | — | Partner's storage |
|
||||
| Long and Low Crafts | — | Crafts business |
|
||||
| zfs_backups_array | — | ZFS backup target |
|
||||
|
||||
---
|
||||
|
||||
## PlausibleDeniability (Primary Docker Host)
|
||||
- **OS:** TrueNAS Scale 25.10.2.1
|
||||
- **CPU:** Ryzen 9 3950X
|
||||
- **RAM:** 32GB
|
||||
- **GPU:** Nvidia RTX 2080 Ti (11GB VRAM) — Plex, Immich ML, Ollama
|
||||
- **UPS:** None
|
||||
|
||||
### Storage
|
||||
| Pool | Type | Mount | Notes |
|
||||
|------|------|-------|-------|
|
||||
| tank | ZFS mirror (2 drives) | `/mnt/tank` | ~3.5TB usable |
|
||||
| docker-ssd | Single Crucial MX500 500GB | `/mnt/docker-ssd` | No redundancy |
|
||||
| boot-pool | Single sdc3 | — | TrueNAS boot |
|
||||
|
||||
### NFS Mounts
|
||||
- Serenity data array → `/mnt/unraid/data`
|
||||
|
||||
---
|
||||
|
||||
## N.O.M.A.D. (Offline Knowledge + Game Servers)
|
||||
- **OS:** Ubuntu 25.10 (bare metal)
|
||||
- **IP:** 10.5.1.16
|
||||
- **CPU:** Intel Core i7-4770K @ 3.50GHz (4c/8t, up to 3.9GHz)
|
||||
- **RAM:** 32GB DDR3 1600MHz (4×8GB)
|
||||
- **GPU:** NVIDIA GeForce GTX 1080 (8GB VRAM)
|
||||
- **NIC:** Intel I218-V (1GbE) + Aquantia AQC113 (10GbE)
|
||||
- **Motherboard:** ASRock H97M Pro4
|
||||
- **UPS:** None
|
||||
|
||||
### Storage
|
||||
| Device | Size | Mount | Purpose |
|
||||
|--------|------|-------|---------|
|
||||
| Samsung SSD 840 (sdb) | 232.9GB | `/` (LVM 100GB) | OS boot |
|
||||
| WD Red HDD (sda) | 3.6TB | `/mnt/hdd-1` | Primary data, project-nomad symlink |
|
||||
| WD Red HDD (sdc) | 3.6TB | `/mnt/hdd-2` | Docker data-root |
|
||||
| Seagate SSD (sdd) | 223.6GB | `/mnt/ssd-1` | Fast scratch |
|
||||
|
||||
---
|
||||
|
||||
## Desktop: ROCINANTE
|
||||
- **OS:** Windows 11 Pro
|
||||
- **CPU:** Ryzen 7 9850X3D
|
||||
- **RAM:** 64GB
|
||||
- **GPU:** RTX 4090 (24GB VRAM)
|
||||
- **Role:** Heavy AI inference, local Ollama (expose via `OLLAMA_HOST=0.0.0.0`)
|
||||
56
docs/architecture/NETWORKING_MODEL.md
Normal file
56
docs/architecture/NETWORKING_MODEL.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# Homelab Networking Model
|
||||
|
||||
## Physical Network
|
||||
- **Gear:** Unifi throughout
|
||||
- **Main LAN:** 10.5.1.x (Trusted VLAN)
|
||||
- **Other VLANs:** Cameras, Guest, IoT
|
||||
- **Serenity IP:** 10.5.1.5
|
||||
- **N.O.M.A.D. IP:** 10.5.1.16
|
||||
- **PlausibleDeniability:** DHCP on LAN
|
||||
|
||||
## Remote Access
|
||||
- **Pangolin VPS + Newt:** Services exposed as subdomains via reverse proxy. Newt agents run on PD and N.O.M.A.D.
|
||||
- **Tailscale:** Personal remote access, runs on Serenity (100.94.87.79)
|
||||
- **Cloudflared:** No longer used. Dead container on Serenity to be removed.
|
||||
|
||||
## DNS
|
||||
- **Provider:** Cloudflare
|
||||
- All public subdomains point at Pangolin VPS IP
|
||||
|
||||
## Known Public Subdomains
|
||||
- `paccoco.com` → Pangolin VPS
|
||||
- `panel.paccoco.com` → Pelican Panel (N.O.M.A.D.)
|
||||
- `node1.paccoco.com` → Wings daemon (N.O.M.A.D., no auth)
|
||||
- `meshmonitor.paccoco.com` → Meshmonitor (PD)
|
||||
- `gitea.paccoco.com` → Gitea (PD)
|
||||
- `ai.paccoco.com` → OpenWebUI (PD) — DNS/Pangolin resource pending
|
||||
|
||||
## Docker Networking (PlausibleDeniability)
|
||||
|
||||
### ix-databases_shared-databases
|
||||
Created by the `databases` stack. The databases compose file uses `name: ix-databases` so Docker names the network `ix-databases_shared-databases`, matching what all other stacks declare as external.
|
||||
|
||||
Any stack connecting to shared-postgres, shared-mariadb, or shared-redis must declare:
|
||||
```yaml
|
||||
networks:
|
||||
ix-databases_shared-databases:
|
||||
external: true
|
||||
```
|
||||
|
||||
### pangolin
|
||||
Created at runtime by the `newt` container in the infrastructure stack. Any service needing reverse-proxy exposure must declare:
|
||||
```yaml
|
||||
networks:
|
||||
pangolin:
|
||||
external: true
|
||||
```
|
||||
|
||||
### Internal Service Model
|
||||
Internal services run HTTP only behind the reverse proxy. Compose service names used for internal DNS between services in the same stack/network.
|
||||
|
||||
## N.O.M.A.D. Networking Notes
|
||||
- `panel.paccoco.com` added to `/etc/hosts` → `127.0.0.1` for NAT loopback
|
||||
- nginx listens on 80 and 443 (self-signed cert) for Wings → Panel communication
|
||||
- Wings uses `--ignore-certificate-errors` for local loopback
|
||||
- Pangolin resource `node1.paccoco.com` has auth disabled (Wings handles its own auth)
|
||||
- Minecraft port 25565 TCP/UDP goes directly via home router port forward, not through Pangolin
|
||||
103
docs/architecture/SERVICES_DIRECTORY.md
Normal file
103
docs/architecture/SERVICES_DIRECTORY.md
Normal file
@@ -0,0 +1,103 @@
|
||||
# Services Directory
|
||||
|
||||
All running services across the homelab with ports, URLs, and backend info.
|
||||
|
||||
## PlausibleDeniability
|
||||
|
||||
### Databases (internal only)
|
||||
| Service | Port | Backend | Notes |
|
||||
|---------|------|---------|-------|
|
||||
| shared-postgres | 5432 | — | immich-optimized, vectorchord + pgvector |
|
||||
| shared-mariadb | 3306 | — | |
|
||||
| shared-redis | 6379 | — | |
|
||||
|
||||
### Infrastructure
|
||||
| Service | Internal Port | Public URL | Notes |
|
||||
|---------|--------------|------------|-------|
|
||||
| newt | — | — | Pangolin reverse proxy agent |
|
||||
| homepage | — | — | Dashboard |
|
||||
| dockhand | — | — | Container management |
|
||||
| uptime-kuma | — | — | MariaDB backend |
|
||||
| netdata | — | — | Metrics |
|
||||
| tailscale | — | — | VPN |
|
||||
|
||||
### Media
|
||||
| Service | Internal Port | Public URL | Notes |
|
||||
|---------|--------------|------------|-------|
|
||||
| plex | — | — | GPU passthrough, RTX 2080 Ti |
|
||||
| tautulli | — | — | Plex analytics |
|
||||
| audiobookshelf | — | — | Audiobooks |
|
||||
| seerr | — | — | Postgres backend |
|
||||
| calibre-web-automated | — | — | Ebooks |
|
||||
|
||||
### Photos
|
||||
| Service | Internal Port | Public URL | Notes |
|
||||
|---------|--------------|------------|-------|
|
||||
| immich-server | — | — | Postgres + Redis, GPU |
|
||||
| immich-ml | — | — | GPU, CUDA |
|
||||
|
||||
### Home
|
||||
| Service | Internal Port | Public URL | Notes |
|
||||
|---------|--------------|------------|-------|
|
||||
| kitchenowl | — | — | Grocery/tasks |
|
||||
| donetick | — | — | Postgres backend, selfhosted.yaml required |
|
||||
| shlink | — | — | URL shortener, Postgres |
|
||||
| shlink-web-client | — | — | |
|
||||
| qui | — | — | qBittorrent UI, manages Serenity remotely |
|
||||
|
||||
### Dev
|
||||
| Service | Internal Port | Public URL | Notes |
|
||||
|---------|--------------|------------|-------|
|
||||
| gitea | — | gitea.paccoco.com | Postgres backend |
|
||||
| rackpeek | — | — | |
|
||||
|
||||
### AI
|
||||
| Service | Internal Port | Public URL | Notes |
|
||||
|---------|--------------|------------|-------|
|
||||
| ollama | 11434 | — | GPU, RTX 2080 Ti |
|
||||
| openwebui | 8282→8080 | ai.paccoco.com (pending) | Postgres backend |
|
||||
|
||||
### Meshtastic
|
||||
| Service | Internal Port | Public URL | Notes |
|
||||
|---------|--------------|------------|-------|
|
||||
| meshmonitor | 8081→3001 | meshmonitor.paccoco.com | Postgres backend |
|
||||
| meshmonitor virtual node | 4404 | — | TCP virtual node server |
|
||||
| meshmonitor-tileserver | 8082→8080 | — | Map tiles |
|
||||
| meshmonitor-mqtt-proxy | — | — | MQTT→TCP bridge |
|
||||
|
||||
---
|
||||
|
||||
## Serenity
|
||||
|
||||
### Media Ingestion
|
||||
| Service | Notes |
|
||||
|---------|-------|
|
||||
| sonarr | TV |
|
||||
| sonarr-anime | Anime |
|
||||
| radarr | Movies |
|
||||
| lidarr | Music |
|
||||
| readarr | Books |
|
||||
| readarr-epub | EPUBs |
|
||||
| prowlarr | Indexer manager |
|
||||
| bazarr | Subtitles |
|
||||
| unpackerr | Archive extraction |
|
||||
| qBittorrent | Download client |
|
||||
| GluetunVPN | VPN tunnel for qbit |
|
||||
| autobrr | Auto-grab |
|
||||
| qbit_manage | qBit maintenance |
|
||||
| Notifiarr | Notifications |
|
||||
| Wizarr | User invites |
|
||||
| romm | ROM manager |
|
||||
| GameVault | Game library |
|
||||
| shelfmark | Book/audiobook search, port 8084 |
|
||||
|
||||
---
|
||||
|
||||
## N.O.M.A.D.
|
||||
|
||||
| Service | Port | Notes |
|
||||
|---------|------|-------|
|
||||
| Pelican Panel | 80/443 | panel.paccoco.com, PHP 8.4 + nginx + MariaDB |
|
||||
| Wings | 8443 | node1.paccoco.com, game server daemon |
|
||||
| Minecraft | 25565 | Purpur, Java 21, direct port forward |
|
||||
| N.O.M.A.D. containers | — | 10 containers, start_nomad.sh / stop_nomad.sh |
|
||||
42
docs/operations/BACKUP_POLICY.md
Normal file
42
docs/operations/BACKUP_POLICY.md
Normal file
@@ -0,0 +1,42 @@
|
||||
# Backup Policy
|
||||
|
||||
## Current State (Known Gaps)
|
||||
- **No off-site backup** — this is a known gap, improvement pending
|
||||
- Serenity has local ZFS snapshots and replication
|
||||
- PlausibleDeniability has no automated backup of appdata or databases
|
||||
|
||||
## Serenity Backup Strategy
|
||||
- Sanoid snapshots cache/appdata: 7 daily, 4 weekly, 6 monthly
|
||||
- Syncoid replicates cache/appdata → malcolm/zfs_backups_array (strict-mirror)
|
||||
- qBit Mover script: pauses torrents 0–4 days old then runs Unraid mover
|
||||
- Mover Status script: sends Discord webhook notifications
|
||||
|
||||
## PlausibleDeniability Backup Scope
|
||||
Back up at minimum:
|
||||
- Compose repo: `/mnt/docker-ssd/docker/compose` (in git — GitHub + Gitea)
|
||||
- Local SSD appdata: `/mnt/docker-ssd/docker/appdata/`
|
||||
- Databases: `/mnt/docker-ssd/docker/databases/`
|
||||
- Selected appdata/backups on tank: `/mnt/tank/docker/`
|
||||
- .env files (separately — gitignored, handle via encrypted backup or private repo)
|
||||
|
||||
## Database Backups
|
||||
- Keep app-consistent dumps for major databases
|
||||
- Do not rely only on raw volume copies for restores
|
||||
- Document restore procedure per major service
|
||||
|
||||
Recommended dump locations: `/mnt/tank/docker/backups/db-dumps/`
|
||||
|
||||
## .env File Backup Options
|
||||
1. **Private Gitea repo** — short term, push all .env files to a private repo on your local Gitea
|
||||
2. **Encrypted restic** — include .env files in restic backup with encryption, push offsite (Backblaze B2)
|
||||
3. **Vaultwarden** — store each .env as a secure note (best long-term once deployed)
|
||||
|
||||
## Restore Practice
|
||||
- Test restores quarterly
|
||||
- Test at least one app DB restore and one full-config restore
|
||||
- Verify that restored stacks can start with current compose files
|
||||
|
||||
## Operational Notes
|
||||
- Media libraries on Unraid are large and may not need the same backup frequency as configs/databases
|
||||
- SSD-local DB and SQLite workloads are highest priority for backup validation
|
||||
- Serenity malcolm pool is heavily utilized — capacity planning needed before it fills
|
||||
63
docs/operations/DEPLOYMENT_CHECKLIST.md
Normal file
63
docs/operations/DEPLOYMENT_CHECKLIST.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# New Service Deployment Checklist
|
||||
|
||||
## Before Adding a Service
|
||||
|
||||
- [ ] Determine DB requirements (postgres, mariadb, sqlite, none)
|
||||
- [ ] Decide storage tier: SSD (`/mnt/docker-ssd/docker/appdata/`) vs tank (`/mnt/tank/docker/appdata/`) vs Unraid
|
||||
- [ ] Check if service needs `ix-databases_shared-databases` network
|
||||
- [ ] Check if service needs `pangolin` network for reverse proxy exposure
|
||||
|
||||
## Scaffold
|
||||
|
||||
- [ ] Create host directories for appdata
|
||||
- [ ] Set correct ownership if needed (`chown` or `chmod 777` for permission-sensitive apps)
|
||||
- [ ] Create stack folder under `/mnt/docker-ssd/docker/compose/<stack>/`
|
||||
- [ ] Create `docker-compose.yaml`
|
||||
- [ ] Create `.env` and `.env.example`
|
||||
- [ ] If using postgres: create DB and user, or add init SQL to `initdb/`
|
||||
- [ ] If using mariadb: add init script to `initdb-mariadb/`
|
||||
|
||||
## Validate Before Deploying
|
||||
|
||||
```bash
|
||||
cd /mnt/docker-ssd/docker/compose/<stack>
|
||||
docker compose --env-file .env config
|
||||
```
|
||||
|
||||
No errors = safe to proceed.
|
||||
|
||||
## Deploy
|
||||
|
||||
```bash
|
||||
docker compose --env-file .env up -d
|
||||
```
|
||||
|
||||
## After Starting
|
||||
|
||||
- [ ] `docker ps` shows container running
|
||||
- [ ] `docker logs <container> --tail 20` shows clean startup
|
||||
- [ ] `docker inspect <container>` confirms mounts are correct
|
||||
- [ ] Data persists after `docker compose --env-file .env down && up -d`
|
||||
|
||||
## Network Checklist
|
||||
|
||||
If service needs shared databases:
|
||||
- Top-level networks block declares `ix-databases_shared-databases` as external
|
||||
- Service networks list includes `ix-databases_shared-databases`
|
||||
- Verify: `docker inspect <container> --format '{{json .NetworkSettings.Networks}}'`
|
||||
|
||||
If service needs reverse proxy:
|
||||
- Top-level networks block declares `pangolin` as external
|
||||
- Service networks list includes `pangolin`
|
||||
- Infrastructure stack (newt) must be running first
|
||||
|
||||
## Common Failure Modes
|
||||
|
||||
| Symptom | Likely Cause | Fix |
|
||||
|---|---|---|
|
||||
| `hostname resolving error` for shared-postgres | Not on `ix-databases_shared-databases` network | `down && up` not just restart |
|
||||
| `not writable` errors on startup | NFS ownership mismatch | Move to SSD or chmod 777 |
|
||||
| `unsupported database type` | App requires config file, not just env vars | Create `selfhosted.yaml` or equivalent |
|
||||
| `JWT is invalid` after restart | JWT secret changed | Clear browser cookies/localStorage |
|
||||
| Port conflict on startup | Port already bound | Remove conflicting port mapping |
|
||||
| Auto-upgrade crashes on TrueNAS | Bind mount source path missing, Docker tries to create at root | Pre-create the directory manually |
|
||||
61
docs/operations/DOCKERSTACK_STORAGE_NOTES.md
Normal file
61
docs/operations/DOCKERSTACK_STORAGE_NOTES.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# Dockerstack Storage Notes
|
||||
|
||||
## Local SSD
|
||||
Use `/docker/...` for:
|
||||
- Docker root (`/docker/docker-data`)
|
||||
- GPU/model storage
|
||||
- SQLite-backed apps
|
||||
- Apps that chown on startup
|
||||
- Write-heavy DB/search storage
|
||||
|
||||
Examples:
|
||||
- `/mnt/docker-ssd/docker/docker-data`
|
||||
- `/mnt/docker-ssd/docker/appdata/ollama`
|
||||
- `/mnt/docker-ssd/docker/appdata/openwebui`
|
||||
- `/mnt/docker-ssd/docker/appdata/immich`
|
||||
- `/mnt/docker-ssd/docker/appdata/immich-ml`
|
||||
- `/mnt/docker-ssd/docker/appdata/plex`
|
||||
- `/mnt/docker-ssd/docker/appdata/tautulli`
|
||||
- `/mnt/docker-ssd/docker/appdata/audiobookshelf`
|
||||
- `/mnt/docker-ssd/docker/appdata/seerr`
|
||||
- `/mnt/docker-ssd/docker/appdata/dockhand`
|
||||
- `/mnt/docker-ssd/docker/appdata/rackpeek`
|
||||
- `/mnt/docker-ssd/docker/appdata/shlink`
|
||||
- `/mnt/docker-ssd/docker/appdata/gitea`
|
||||
- `/mnt/docker-ssd/docker/appdata/netdata`
|
||||
- `/mnt/docker-ssd/docker/appdata/tailscale`
|
||||
- `/mnt/docker-ssd/docker/appdata/uptime-kuma`
|
||||
- `/mnt/docker-ssd/docker/databases/postgres`
|
||||
- `/mnt/docker-ssd/docker/databases/mariadb`
|
||||
- `/mnt/docker-ssd/docker/databases/redis`
|
||||
|
||||
## TrueNAS NFS
|
||||
Use `/mnt/tank/docker/...` for:
|
||||
- Normal appdata
|
||||
- Uploads
|
||||
- Non-fussy config and data
|
||||
- Backups
|
||||
- Downloads
|
||||
|
||||
Examples:
|
||||
- `/mnt/tank/docker/appdata/homepage`
|
||||
- `/mnt/tank/docker/appdata/newt`
|
||||
- `/mnt/tank/docker/appdata/kitchenowl`
|
||||
- `/mnt/tank/docker/appdata/donetick`
|
||||
- `/mnt/tank/docker/appdata/qui`
|
||||
- `/mnt/tank/docker/appdata/calibre-web-automated`
|
||||
- `/mnt/tank/docker/backups`
|
||||
- `/mnt/tank/docker/downloads`
|
||||
|
||||
## Unraid NFS
|
||||
Use `/mnt/unraid/...` for media and shared bulk content only:
|
||||
- `/mnt/unraid/data/media/` — Plex libraries, books, etc.
|
||||
- `/mnt/unraid/immich` — Immich photo/video storage
|
||||
|
||||
## Rules Learned
|
||||
- SQLite apps and apps that write logs aggressively should prefer local SSD.
|
||||
- Database/search/model-heavy workloads should prefer local SSD.
|
||||
- General config and less write-sensitive appdata can live on TrueNAS NFS.
|
||||
- Media should stay on Unraid mounts.
|
||||
- SQLite on ZFS with nfsv4 ACLs will always fail — move to SSD or use Postgres.
|
||||
- Apps with auto-upgrade features that use bind mounts: pre-create all source directories manually on TrueNAS to prevent Docker from trying to create them at the read-only root.
|
||||
64
docs/operations/SECRETS_MANAGEMENT.md
Normal file
64
docs/operations/SECRETS_MANAGEMENT.md
Normal file
@@ -0,0 +1,64 @@
|
||||
# Secrets Management
|
||||
|
||||
## Principles
|
||||
- `.env` files are gitignored and never committed to any public repo
|
||||
- `.env.example` files are committed with placeholder values
|
||||
- Real `.env` files live only on the host at the stack directory
|
||||
- Secrets are generated with standard tools, never reused across services
|
||||
|
||||
## Generating Secrets
|
||||
|
||||
```bash
|
||||
# Database passwords
|
||||
openssl rand -hex 24
|
||||
|
||||
# JWT / session secrets
|
||||
openssl rand -hex 32
|
||||
|
||||
# API keys (where format allows)
|
||||
openssl rand -base64 32
|
||||
```
|
||||
|
||||
## .env File Locations
|
||||
All `.env` files live alongside their `docker-compose.yaml`:
|
||||
```
|
||||
/mnt/docker-ssd/docker/compose/<stack>/.env
|
||||
```
|
||||
|
||||
## Backing Up .env Files
|
||||
|
||||
### Short term: Private Gitea repo
|
||||
Create a private repo on Gitea (not GitHub) and push all .env files there:
|
||||
```bash
|
||||
# In a separate directory, not the main stacks repo
|
||||
git init homelab-secrets
|
||||
cd homelab-secrets
|
||||
# Copy and add .env files
|
||||
git remote add origin https://gitea.paccoco.com/fizzlepoof/homelab-secrets.git
|
||||
git push -u origin main
|
||||
```
|
||||
|
||||
### Long term: Restic + offsite
|
||||
Include the compose directory (with .env files) in a restic backup:
|
||||
```bash
|
||||
restic -r <repo> backup /mnt/docker-ssd/docker/compose --exclude="*.example"
|
||||
```
|
||||
|
||||
### Best long term: Vaultwarden
|
||||
Once deployed, store each service's .env as a secure note in Vaultwarden. Single source of truth, accessible from anywhere.
|
||||
|
||||
## Known Credentials Locations
|
||||
|
||||
| Service | Where stored |
|
||||
|---------|-------------|
|
||||
| shared-postgres | `databases/.env` |
|
||||
| shared-mariadb | `databases/.env` |
|
||||
| OpenWebUI DB | `ai/.env` — user: openwebui, db: openwebui |
|
||||
| Donetick | `home/.env` + `selfhosted.yaml` |
|
||||
| Meshmonitor | `meshtastic/.env` |
|
||||
| Gitea | `dev/.env` |
|
||||
|
||||
## Security Reminders
|
||||
- Regenerate Wings token on N.O.M.A.D. (was exposed in chat — TODO)
|
||||
- Regenerate N.O.M.A.D. Newt secret (was exposed in chat — TODO)
|
||||
- Disable signups in OpenWebUI after creating initial account
|
||||
122
docs/operations/STACK_STANDARDS.md
Normal file
122
docs/operations/STACK_STANDARDS.md
Normal file
@@ -0,0 +1,122 @@
|
||||
# Homelab Stack Standards
|
||||
|
||||
## Repository Root
|
||||
`/mnt/docker-ssd/docker/compose`
|
||||
|
||||
Remotes:
|
||||
- GitHub: `git@github.com:Paccoco/truenas-stacks.git`
|
||||
- Gitea: `https://gitea.paccoco.com/fizzlepoof/truenas-stacks.git`
|
||||
|
||||
## Stack Layout
|
||||
Each production stack lives in a subdirectory of the repo root.
|
||||
|
||||
Each stack contains:
|
||||
- `docker-compose.yaml` — required
|
||||
- `.env` — required (gitignored)
|
||||
- `.env.example` — committed with placeholder values
|
||||
- optional init scripts (e.g. `initdb/`, `initdb-mariadb/`)
|
||||
|
||||
## Deployment Order
|
||||
1. `databases/` — must be up before anything else
|
||||
2. `infrastructure/` — newt creates the pangolin network
|
||||
3. All other stacks in any order
|
||||
|
||||
## Deployment Workflow
|
||||
|
||||
Validate before deploying:
|
||||
```bash
|
||||
cd /mnt/docker-ssd/docker/compose/<stack>
|
||||
docker compose --env-file .env config
|
||||
```
|
||||
|
||||
Deploy:
|
||||
```bash
|
||||
docker compose --env-file .env up -d
|
||||
```
|
||||
|
||||
Teardown a single service:
|
||||
```bash
|
||||
docker compose --env-file .env down <service>
|
||||
```
|
||||
|
||||
## Networks
|
||||
|
||||
### ix-databases_shared-databases
|
||||
Created by the `databases` stack with `name: ix-databases` at top level. Any stack connecting to shared databases must declare:
|
||||
```yaml
|
||||
networks:
|
||||
ix-databases_shared-databases:
|
||||
external: true
|
||||
```
|
||||
|
||||
### pangolin
|
||||
Created by `newt` at runtime. Any externally exposed service must declare:
|
||||
```yaml
|
||||
networks:
|
||||
pangolin:
|
||||
external: true
|
||||
```
|
||||
|
||||
## Storage Standards
|
||||
|
||||
### Local SSD: `/mnt/docker-ssd`
|
||||
Use `/mnt/docker-ssd/docker/appdata/<service>` for:
|
||||
- Postgres / MariaDB / Redis data
|
||||
- SQLite-backed apps
|
||||
- Apps that chown aggressively on startup
|
||||
- GPU/model caches
|
||||
- Write-heavy logs, cache, state
|
||||
|
||||
### TrueNAS Pool (tank): `/mnt/tank`
|
||||
Use `/mnt/tank/docker/appdata/<service>` for:
|
||||
- General appdata
|
||||
- Configs
|
||||
- Uploads
|
||||
- Non-permission-sensitive state
|
||||
|
||||
### Unraid NFS: `/mnt/unraid`
|
||||
- `/mnt/unraid/data/media/` — media libraries
|
||||
- `/mnt/unraid/immich` — Immich photo/video storage
|
||||
|
||||
## Volume Rules
|
||||
- no anonymous volumes
|
||||
- no ad-hoc home directory paths
|
||||
- all binds use standardized production paths
|
||||
- evaluate SSD vs tank vs Unraid before deploying any new service
|
||||
- apps known to have permission issues on NFS go on SSD
|
||||
|
||||
## Known Permission Quirks
|
||||
- `shlink` data directory must be chmod 777 — runs as non-root user
|
||||
- `immich-ml` cache goes to `/mnt/docker-ssd/docker/appdata/immich-ml` (separate from immich-server)
|
||||
- `donetick` requires `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml` — env vars alone insufficient
|
||||
|
||||
## Secrets Management
|
||||
- `.env` files are gitignored
|
||||
- `.env.example` files are committed with placeholder values
|
||||
- Database passwords: `openssl rand -hex 24`
|
||||
- JWT secrets: `openssl rand -hex 32`
|
||||
|
||||
## Healthcheck Patterns
|
||||
Always check available tools before writing healthchecks:
|
||||
```bash
|
||||
sudo docker exec <container> sh -c "which curl; which wget" 2>&1
|
||||
```
|
||||
|
||||
- Has curl: `["CMD-SHELL", "curl -sf http://127.0.0.1:<port>/ || exit 1"]`
|
||||
- Has wget only: `["CMD-SHELL", "wget -qO- http://127.0.0.1:<port>/ >/dev/null 2>&1 || exit 1"]`
|
||||
- Has neither: `["CMD-SHELL", "cat /proc/net/tcp6 | grep -q <HEX_PORT> || exit 1"]`
|
||||
- Redirecting endpoint: `["CMD-SHELL", "curl -sfL http://127.0.0.1:<port>/ || exit 1"]`
|
||||
|
||||
Common hex port values: 8080=1F90, 3003=BBB, 3000=BB8, 8181=1FF5, 5055=13BF, 2021=7E5
|
||||
|
||||
After changing a healthcheck, must `down` then `up` (not just restart).
|
||||
|
||||
## Git Workflow
|
||||
```bash
|
||||
cd /mnt/docker-ssd/docker/compose
|
||||
git add .
|
||||
git commit -m "description"
|
||||
git push # pushes to both GitHub and Gitea
|
||||
```
|
||||
|
||||
All `.env` files are gitignored. Only compose files, `.env.example`, and init scripts are committed.
|
||||
106
docs/planning/HOMELAB_EXPANSION_PLAN.md
Normal file
106
docs/planning/HOMELAB_EXPANSION_PLAN.md
Normal file
@@ -0,0 +1,106 @@
|
||||
# Homelab Expansion Plan
|
||||
|
||||
## Deployment Order
|
||||
1. **Gotify** — unified notification bus, unblocks all n8n workflows
|
||||
2. **Qdrant** — vector DB, unblocks proper RAG pipeline
|
||||
3. **n8n** — automation backbone, ties everything together
|
||||
4. **Paperless-NGX** — document intelligence
|
||||
5. **Home Assistant** — home automation layer
|
||||
6. **Grafana + Prometheus** — unified observability
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Notifications (Gotify)
|
||||
**Host:** PlausibleDeniability
|
||||
**Stack:** `/mnt/docker-ssd/docker/compose/automation/`
|
||||
**Storage:** appdata on SSD (SQLite backend)
|
||||
**Network:** pangolin (expose as `gotify.paccoco.com`)
|
||||
|
||||
Single output target for all n8n workflows and homelab alerts.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Vector Database (Qdrant)
|
||||
**Host:** PlausibleDeniability
|
||||
**Stack:** `/mnt/docker-ssd/docker/compose/ai/`
|
||||
**Storage:** appdata on SSD
|
||||
|
||||
Feed it: homelab markdown docs, Gitea commit history, Plex/Tautulli metadata, Obsidian notes.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Workflow Automation (n8n)
|
||||
**Host:** PlausibleDeniability
|
||||
**Stack:** `/mnt/docker-ssd/docker/compose/automation/`
|
||||
**Storage:** config on tank, Postgres backend
|
||||
**Network:** pangolin (`n8n.paccoco.com`) + ix-databases_shared-databases
|
||||
|
||||
### Planned Workflows
|
||||
- Sonarr/Radarr webhook → Discord + Gotify notification with poster art
|
||||
- Tautulli play start → log to Postgres
|
||||
- Nightly watch history → Ollama → weekly digest via Gotify
|
||||
- Poll Uptime Kuma → cross-reference Netdata → rich alert via Gotify
|
||||
- Gitea commit to truenas-stacks → Gotify notification with Ollama summary
|
||||
- qBittorrent torrent complete → trigger Sonarr/Radarr rescan
|
||||
- KitchenOwl grocery list change → shopping list via Gotify
|
||||
- Donetick task due → Gotify reminder
|
||||
- Long and Low Crafts order → Donetick task + Gotify DM
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Document Intelligence (Paperless-NGX)
|
||||
**Host:** PlausibleDeniability
|
||||
**Network:** pangolin (`paperless.paccoco.com`) + ix-databases_shared-databases
|
||||
|
||||
OCR and full-text search. Key for Long and Low Crafts invoices, receipts, and manuals.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Home Automation (Home Assistant)
|
||||
**Host:** PlausibleDeniability or N.O.M.A.D.
|
||||
**Storage:** config on SSD (SQLite, write-heavy)
|
||||
**Network:** host networking recommended for device discovery
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Observability (Grafana + Prometheus)
|
||||
**Host:** N.O.M.A.D.
|
||||
**Storage:** Prometheus data on hdd-1 or hdd-2
|
||||
|
||||
Metrics to scrape: ZFS pool health, container CPU/RAM, Plex streams, qBittorrent, Tailscale latency, Netdata endpoints.
|
||||
|
||||
---
|
||||
|
||||
## Multi-Model Routing Strategy
|
||||
|
||||
| Machine | GPU | Model | Use Case |
|
||||
|---|---|---|---|
|
||||
| ROCINANTE | RTX 4090 (24GB) | qwen3:32b / deepseek-r1:32b | Heavy reasoning, long context |
|
||||
| PlausibleDeniability | RTX 2080 Ti (11GB) | qwen2.5:14b | General homelab assistant, RAG |
|
||||
| N.O.M.A.D. | GTX 1080 (8GB) | phi4 / llama3.2:3b | Fast local inference |
|
||||
|
||||
---
|
||||
|
||||
## Additional Tools
|
||||
- **Continue.dev** — VS Code plugin, local Copilot alternative
|
||||
- **Obsidian + Gitea Sync** — personal notes → RAG pipeline
|
||||
- **Homepage Ollama Widget** — VRAM usage dashboard widget
|
||||
- **kima-hub** — self-hosted Spotify alternative
|
||||
- **dispatcharr** — planned
|
||||
- **bytestash** — planned
|
||||
|
||||
---
|
||||
|
||||
## Target Architecture
|
||||
|
||||
```
|
||||
Triggers (Sonarr, Radarr, Tautulli, Gitea, Home Assistant, schedules, email)
|
||||
↓
|
||||
n8n (routing, logic, orchestration)
|
||||
↓
|
||||
Ollama (intelligence — model selected by complexity)
|
||||
↓ ↑
|
||||
Qdrant (semantic context retrieval)
|
||||
↓
|
||||
Outputs (Gotify → phone, Donetick tasks, Postgres logging, Discord)
|
||||
```
|
||||
43
docs/planning/TODO.md
Normal file
43
docs/planning/TODO.md
Normal file
@@ -0,0 +1,43 @@
|
||||
# Homelab TODO
|
||||
|
||||
## Security (do ASAP)
|
||||
- [ ] Regenerate Wings token on N.O.M.A.D. (was exposed in chat)
|
||||
- [ ] Regenerate N.O.M.A.D. Newt secret (was exposed in chat)
|
||||
|
||||
## Infrastructure
|
||||
- [ ] Deploy Pi-hole ×2 (primary on PD + backup on Serenity or Pi4)
|
||||
- [ ] Set up off-site backup for vital data (appdata, databases, config repo, photos)
|
||||
- [ ] Remove dead Unraid-Cloudflared-Tunnel container from Serenity
|
||||
- [ ] Serenity malcolm pool capacity planning (heavily utilized)
|
||||
- [ ] Add Pangolin resource for `ai.paccoco.com` → openwebui (port 8282)
|
||||
- [ ] Add Cloudflare DNS A record for `ai.paccoco.com`
|
||||
- [ ] Install fresh editor on PD
|
||||
- [ ] Set up private Gitea repo for .env file backups
|
||||
|
||||
## AI Stack
|
||||
- [ ] Pull `qwen2.5:14b` on PD ollama: `sudo docker exec -it ollama ollama pull qwen2.5:14b`
|
||||
- [ ] Create OpenWebUI account and set system prompt from HOMELAB_AI_CONTEXT.md
|
||||
- [ ] Disable signups in OpenWebUI after creating account
|
||||
- [ ] Connect desktop Ollama to OpenWebUI (Admin → Settings → Connections)
|
||||
- [ ] Set `OLLAMA_HOST=0.0.0.0` on desktop and restart Ollama service
|
||||
|
||||
## Planned Deployments (PD)
|
||||
- [ ] dispatcharr
|
||||
- [ ] gotify
|
||||
- [ ] bytestash
|
||||
- [ ] kima-hub
|
||||
- [ ] n8n
|
||||
- [ ] qdrant
|
||||
- [ ] paperless-ngx
|
||||
|
||||
## Healthchecks
|
||||
- [ ] Add healthcheck to ix-plex-plex-1
|
||||
- [ ] Add healthcheck to ix-tailscale-tailscale-1
|
||||
- [ ] Add healthcheck to ix-newt-newt-1
|
||||
- [ ] Add healthcheck to ollama (no curl/wget — needs /proc/net/tcp6 or custom solution)
|
||||
- [ ] Document and add healthchecks for meshtastic stack
|
||||
|
||||
## Documentation
|
||||
- [ ] Document meshtastic stack fully (see docs/reference/MESHTASTIC.md)
|
||||
- [ ] Fill in port/URL columns in SERVICES_DIRECTORY.md
|
||||
- [ ] Document N.O.M.A.D. containers (start_nomad.sh services)
|
||||
56
docs/reference/HOMELAB_AI_CONTEXT.md
Normal file
56
docs/reference/HOMELAB_AI_CONTEXT.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# Homelab AI Context Document
|
||||
|
||||
This document gives an AI assistant full context of this homelab for infrastructure questions, deployments, debugging, and planning.
|
||||
|
||||
See the individual docs in this repo for detailed information:
|
||||
|
||||
- [Architecture Overview](../architecture/ARCHITECTURE_OVERVIEW.md)
|
||||
- [Networking Model](../architecture/NETWORKING_MODEL.md)
|
||||
- [Hardware Inventory](../architecture/HARDWARE_INVENTORY.md)
|
||||
- [Services Directory](../architecture/SERVICES_DIRECTORY.md)
|
||||
- [Stack Standards](../operations/STACK_STANDARDS.md)
|
||||
- [Deployment Checklist](../operations/DEPLOYMENT_CHECKLIST.md)
|
||||
- [Storage Notes](../operations/DOCKERSTACK_STORAGE_NOTES.md)
|
||||
- [Backup Policy](../operations/BACKUP_POLICY.md)
|
||||
- [Secrets Management](../operations/SECRETS_MANAGEMENT.md)
|
||||
- [Troubleshooting](../troubleshooting/TROUBLESHOOTING.md)
|
||||
- [Known Quirks](../troubleshooting/KNOWN_QUIRKS.md)
|
||||
- [Serenity](../servers/SERENITY.md)
|
||||
- [PlausibleDeniability](../servers/PLAUSIBLEDENABILITY.md)
|
||||
- [N.O.M.A.D.](../servers/NOMAD.md)
|
||||
- [Meshtastic Stack](MESHTASTIC.md)
|
||||
- [Expansion Plan](../planning/HOMELAB_EXPANSION_PLAN.md)
|
||||
- [TODO](../planning/TODO.md)
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Communication Preferences
|
||||
- One command block at a time
|
||||
- Prefer scripted/automated solutions over interactive multi-step processes
|
||||
- Uses `fresh` editor (not nano) on Linux
|
||||
- Manages remote servers via SSH from Windows 11
|
||||
- Prefers MariaDB over PostgreSQL for standalone services
|
||||
|
||||
### Three Servers
|
||||
| Server | Role | OS | IP |
|
||||
|--------|------|----|----|
|
||||
| Serenity | NAS / Media ingestion | Unraid 7.2.4 | 10.5.1.5 |
|
||||
| PlausibleDeniability | Primary Docker host | TrueNAS Scale 25.10.2.1 | LAN DHCP |
|
||||
| N.O.M.A.D. | Offline knowledge + game servers | Ubuntu 25.10 | 10.5.1.16 |
|
||||
|
||||
### Deployment Pattern (PD)
|
||||
```bash
|
||||
cd /mnt/docker-ssd/docker/compose/<stack>
|
||||
docker compose --env-file .env config # validate first
|
||||
docker compose --env-file .env up -d # deploy
|
||||
docker logs <container> --tail 20 # verify
|
||||
```
|
||||
|
||||
### Network names
|
||||
- Shared DB network: `ix-databases_shared-databases`
|
||||
- Reverse proxy network: `pangolin`
|
||||
|
||||
### Storage decision
|
||||
- Write-heavy / DB / SQLite / GPU → `/mnt/docker-ssd/docker/appdata/<service>`
|
||||
- General config / non-fussy → `/mnt/tank/docker/appdata/<service>`
|
||||
- Media only → `/mnt/unraid/data/media/`
|
||||
70
docs/reference/MESHTASTIC.md
Normal file
70
docs/reference/MESHTASTIC.md
Normal file
@@ -0,0 +1,70 @@
|
||||
# Meshtastic Stack
|
||||
|
||||
## Overview
|
||||
Self-hosted Meshtastic monitoring and management stack running on PlausibleDeniability.
|
||||
|
||||
## Stack Location
|
||||
`/mnt/docker-ssd/docker/compose/meshtastic/`
|
||||
|
||||
## Containers
|
||||
|
||||
| Container | Image | Port | Purpose |
|
||||
|-----------|-------|------|---------|
|
||||
| meshmonitor | ghcr.io/yeraze/meshmonitor:4.2.0 | 8081→3001 | Main UI + backend |
|
||||
| meshmonitor (virtual node) | — | 4404 | TCP virtual node server |
|
||||
| meshmonitor-tileserver | maptiler/tileserver-gl-light | 8082→8080 | Offline map tiles |
|
||||
| meshmonitor-mqtt-proxy | ghcr.io/ln4cy/mqtt-proxy:master | — | MQTT→TCP bridge |
|
||||
| meshmonitor-upgrader | docker:latest | — | Auto-upgrade watchdog |
|
||||
|
||||
## Architecture
|
||||
```
|
||||
Meshtastic Node (10.5.1.172)
|
||||
↓ MQTT
|
||||
meshmonitor-mqtt-proxy
|
||||
↓ TCP (port 4404)
|
||||
meshmonitor virtual node server
|
||||
↓
|
||||
meshmonitor backend
|
||||
↓
|
||||
PostgreSQL (shared-postgres, db: meshmonitor)
|
||||
```
|
||||
|
||||
## Node Connection
|
||||
- Node IP: `10.5.1.172`
|
||||
- Connection type: MQTT via mqtt-proxy (not direct TCP)
|
||||
- mqtt-proxy connects to meshmonitor's virtual node server on port 4404
|
||||
|
||||
## Storage
|
||||
- Appdata: `/mnt/tank/docker/appdata/meshmonitor/data` → `/data`
|
||||
- Scripts: `./scripts` → `/data/scripts` (must be pre-created on host)
|
||||
- Tiles: `./tiles` (gitignored — large .mbtiles files)
|
||||
|
||||
## Auto-Upgrade
|
||||
`AUTO_UPGRADE_ENABLED=true` — meshmonitor checks for new images and triggers self-upgrade.
|
||||
|
||||
**Important:** The upgrader mounts the compose directory as `/compose` inside the container. All relative bind mount paths (e.g. `./scripts`) resolve to `/compose/scripts` on the host when the upgrader re-runs compose. Since TrueNAS root filesystem is read-only, these directories must be pre-created:
|
||||
|
||||
```bash
|
||||
mkdir -p /mnt/docker-ssd/docker/compose/meshtastic/scripts
|
||||
```
|
||||
|
||||
## Networks
|
||||
- `default` (internal stack network)
|
||||
- `ix-databases_shared-databases` (meshmonitor only)
|
||||
- `pangolin` (meshmonitor + tileserver)
|
||||
|
||||
## Public URL
|
||||
`https://meshmonitor.paccoco.com`
|
||||
|
||||
## Known Warnings
|
||||
- Node `!dd972536` has a low-entropy public key — this is a device configuration issue, not a meshmonitor bug
|
||||
- Frequent connect/disconnect cycles in logs are normal when the MQTT proxy reconnects
|
||||
|
||||
## Tiles
|
||||
Map tiles stored in `./tiles/`. Currently includes `zurich_switzerland.mbtiles` (34MB).
|
||||
Tiles directory is gitignored. To add new tiles, download `.mbtiles` files and place in the `tiles/` directory.
|
||||
|
||||
## TODO
|
||||
- [ ] Document all running meshtastic-map container config
|
||||
- [ ] Document meshtastic-map-mosquitto and meshtastic-mqtt containers
|
||||
- [ ] Add healthcheck to meshmonitor container
|
||||
73
docs/servers/NOMAD.md
Normal file
73
docs/servers/NOMAD.md
Normal file
@@ -0,0 +1,73 @@
|
||||
# N.O.M.A.D. Server
|
||||
|
||||
Offline survival/knowledge server and game server host.
|
||||
|
||||
## Specs
|
||||
- **OS:** Ubuntu 25.10 (bare metal)
|
||||
- **IP:** 10.5.1.16
|
||||
- **CPU:** Intel Core i7-4770K @ 3.50GHz (4c/8t)
|
||||
- **RAM:** 32GB DDR3 1600MHz
|
||||
- **GPU:** NVIDIA GeForce GTX 1080 (8GB VRAM)
|
||||
- **NIC:** Intel I218-V (1GbE) + Aquantia AQC113 (10GbE)
|
||||
- **Motherboard:** ASRock H97M Pro4
|
||||
- **UPS:** None
|
||||
|
||||
## Storage
|
||||
| Device | Size | Mount | Purpose |
|
||||
|--------|------|-------|---------|
|
||||
| Samsung SSD 840 (sdb) | 232.9GB | `/` (LVM 100GB) | OS boot |
|
||||
| WD Red HDD (sda) | 3.6TB | `/mnt/hdd-1` | Primary data |
|
||||
| WD Red HDD (sdc) | 3.6TB | `/mnt/hdd-2` | Docker data-root |
|
||||
| Seagate SSD (sdd) | 223.6GB | `/mnt/ssd-1` | Fast scratch |
|
||||
|
||||
## Project N.O.M.A.D.
|
||||
- 10 containers running via `start_nomad.sh` / `stop_nomad.sh`
|
||||
- `/opt/project-nomad` → symlinked from `/mnt/hdd-1/project-nomad`
|
||||
- Docker data-root: `/mnt/hdd-2/docker`
|
||||
- Based on Crosstalk Solutions offline survival/knowledge server
|
||||
|
||||
## Pelican Panel (Game Server Management)
|
||||
- **URL:** `https://panel.paccoco.com`
|
||||
- **Stack:** PHP 8.4 + nginx + MariaDB (native, not Docker)
|
||||
- **Panel files:** `/var/www/pelican`
|
||||
- **DB:** MariaDB, database `pelican`, user `pelican`@`127.0.0.1`
|
||||
- **APP_URL:** `https://panel.paccoco.com`
|
||||
- **TRUSTED_PROXIES:** `*`
|
||||
- **Cron:** `* * * * * php /var/www/pelican/artisan schedule:run` (www-data)
|
||||
- **Admin user:** `fizzlepoof`
|
||||
- **Reverse proxy:** Pangolin via Newt container at `/opt/newt`
|
||||
|
||||
## Wings (Game Server Daemon)
|
||||
- **Binary:** `/usr/local/bin/wings`
|
||||
- **Config:** `/etc/pelican/config.yml`
|
||||
- **Systemd:** `wings.service`
|
||||
- **ExecStart:** `/usr/local/bin/wings --ignore-certificate-errors`
|
||||
- **API port:** 8443
|
||||
- **SFTP port:** 2022
|
||||
- **Node domain:** `node1.paccoco.com` (Pangolin, no auth, connection port 443)
|
||||
- **Game server volumes:** `/var/lib/pelican/volumes`
|
||||
- **Docker network:** `pelican_nw` (172.20.0.0/16)
|
||||
|
||||
## Active Game Servers
|
||||
- **Minecraft:** Purpur, Java 21, port 25565 TCP/UDP (direct host port forward)
|
||||
|
||||
## Networking Notes
|
||||
- `panel.paccoco.com` → `127.0.0.1` in `/etc/hosts` for NAT loopback
|
||||
- nginx on 80 and 443 (self-signed cert) for local Wings → Panel
|
||||
- Wings uses `--ignore-certificate-errors` for self-signed cert
|
||||
- Pangolin resource `node1.paccoco.com` has auth disabled
|
||||
|
||||
## Known Quirks
|
||||
- NVIDIA container toolkit repo has invalid GPG key — remove `/etc/apt/sources.list.d/*nvidia*` if apt errors
|
||||
- Port 8080 occupied by `nomad_admin` container — Wings uses 8443
|
||||
- Interactive artisan commands break over SSH — always use inline flags
|
||||
- Wings CORS derived from `remote:` URL — keep remote as `https://` and use `--ignore-certificate-errors`
|
||||
|
||||
## Newt Tunnel
|
||||
- Container: `newt` at `/opt/newt/docker-compose.yaml`
|
||||
- `network_mode: host`
|
||||
- Connects to Pangolin VPS at `https://paccoco.com`
|
||||
|
||||
## Pending
|
||||
- Regenerate Wings token (was exposed in chat)
|
||||
- Regenerate Newt secret (was exposed in chat)
|
||||
60
docs/servers/PLAUSIBLEDENABILITY.md
Normal file
60
docs/servers/PLAUSIBLEDENABILITY.md
Normal file
@@ -0,0 +1,60 @@
|
||||
# PlausibleDeniability
|
||||
|
||||
Primary Docker host running all production workloads.
|
||||
|
||||
## Specs
|
||||
- **OS:** TrueNAS Scale 25.10.2.1
|
||||
- **CPU:** Ryzen 9 3950X
|
||||
- **RAM:** 32GB
|
||||
- **GPU:** Nvidia RTX 2080 Ti (11GB VRAM)
|
||||
- **UPS:** None
|
||||
|
||||
## Storage Tiers
|
||||
|
||||
| Tier | Mount | Use for |
|
||||
|------|-------|---------|
|
||||
| Local SSD (Crucial MX500 500GB) | `/mnt/docker-ssd` | Docker root, databases, write-heavy appdata, GPU/model caches |
|
||||
| TrueNAS pool (tank, 2-drive ZFS mirror ~3.5TB) | `/mnt/tank` | General appdata, configs, backups, downloads |
|
||||
| Unraid NFS | `/mnt/unraid` | Media libraries, Immich media |
|
||||
|
||||
## Docker Root
|
||||
`/mnt/docker-ssd/docker/docker-data`
|
||||
|
||||
## Compose Repo
|
||||
`/mnt/docker-ssd/docker/compose`
|
||||
|
||||
## Key Docker Networks
|
||||
- `ix-databases_shared-databases` — shared DB access network (created by databases stack)
|
||||
- `pangolin` — reverse proxy exposure network (created by newt)
|
||||
|
||||
## GPU Usage
|
||||
RTX 2080 Ti used by:
|
||||
- Plex (hardware transcoding)
|
||||
- immich-ml (CUDA face recognition)
|
||||
- ollama (LLM inference)
|
||||
|
||||
Recommended models for 11GB VRAM:
|
||||
| Model | VRAM | Best for |
|
||||
|-------|------|---------|
|
||||
| `qwen2.5:14b` | ~8GB | General use, coding, reasoning |
|
||||
| `qwen2.5-coder:7b` | ~5GB | Code tasks |
|
||||
| `deepseek-r1:8b` | ~5GB | Math, logic |
|
||||
| `phi4` | ~8GB | Analytical tasks |
|
||||
|
||||
## Remote Access
|
||||
- Pangolin VPS + Newt for external services
|
||||
- SSH from Windows 11
|
||||
|
||||
## NFS Exports to Other Servers
|
||||
- `data` share → PD at `/mnt/unraid/data`
|
||||
- `immich` share → PD at `/mnt/unraid/immich`
|
||||
|
||||
## Known Issues
|
||||
- immich-ml intermittently unhealthy — investigate GPU/CUDA health
|
||||
- docker-ssd has no redundancy — single point of failure for all appdata
|
||||
- No off-site backup
|
||||
|
||||
## Pending
|
||||
- Deploy: dispatcharr, gotify, bytestash, kima-hub, n8n, qdrant, paperless-ngx
|
||||
- Add Pangolin resource + DNS for `ai.paccoco.com`
|
||||
- Install fresh editor
|
||||
55
docs/servers/SERENITY.md
Normal file
55
docs/servers/SERENITY.md
Normal file
@@ -0,0 +1,55 @@
|
||||
# Serenity
|
||||
|
||||
Unraid NAS and media ingestion server.
|
||||
|
||||
## Specs
|
||||
- **OS:** Unraid 7.2.4
|
||||
- **IP:** 10.5.1.5 | Tailscale: 100.94.87.79
|
||||
- **CPU:** Xeon X5690
|
||||
- **RAM:** 102GB
|
||||
- **GPU:** None (Matrox IPMI only)
|
||||
- **UPS:** None
|
||||
|
||||
## Storage
|
||||
- `malcolm` pool: RAIDZ1 of 6×10TB HDDs (~40TB usable) — heavily utilized
|
||||
- Cache pool: single Kingston SNV2S 1TB NVMe (ZFS)
|
||||
|
||||
## Docker Networks
|
||||
`arrs`, `hotio`, `br0` (macvlan), `bridge`
|
||||
|
||||
## Active Containers
|
||||
- **Media:** sonarr, sonarr-anime, radarr, lidarr, readarr, readarr-epub, prowlarr, bazarr, unpackerr
|
||||
- **Download:** qBittorrent, GluetunVPN, autobrr, qbit_manage
|
||||
- **Notifications:** Notifiarr
|
||||
- **Access:** Wizarr, Newt, dockersocket, Hawser
|
||||
- **Databases:** postgresql15, MariaDB-Official
|
||||
- **Other:** romm, GameVault, shelfmark (port 8084), netdata
|
||||
- **Dead/remove:** Unraid-Cloudflared-Tunnel
|
||||
|
||||
## Backup Strategy
|
||||
- Sanoid snapshots cache/appdata: 7 daily, 4 weekly, 6 monthly
|
||||
- Syncoid replicates cache/appdata → malcolm/zfs_backups_array (strict-mirror)
|
||||
- No off-site backup currently
|
||||
|
||||
## Custom Scripts
|
||||
- **qBit Mover script:** pauses torrents 0–4 days old, then runs Unraid mover
|
||||
- **MAM-init script:** updates MyAnonamouse IP inside qbit container
|
||||
- **Mover Status script:** sends Discord webhook notifications
|
||||
|
||||
## Key Shares
|
||||
| Share | Size | Purpose |
|
||||
|-------|------|---------|
|
||||
| data | 35TB | Media library (NFS-mounted by PD at `/mnt/unraid/data`) |
|
||||
| immich | 62GB | Immich photo storage (NFS-mounted by PD at `/mnt/unraid/immich`) |
|
||||
| nextcloud | 7.4GB | Nextcloud data |
|
||||
| Tesla/dashcam | 1.6TB | Dashcam footage |
|
||||
| fizzlebang | 17GB | Personal files |
|
||||
| isos | 22GB | ISO images |
|
||||
| Syncthing | 4.6GB | Syncthing data |
|
||||
| manndra | — | Partner's storage |
|
||||
| Long and Low Crafts | — | Crafts business files |
|
||||
| zfs_backups_array | — | ZFS backup target for cache/appdata |
|
||||
|
||||
## Pending
|
||||
- Remove dead Unraid-Cloudflared-Tunnel container
|
||||
- malcolm pool capacity planning (heavily utilized)
|
||||
69
docs/troubleshooting/KNOWN_QUIRKS.md
Normal file
69
docs/troubleshooting/KNOWN_QUIRKS.md
Normal file
@@ -0,0 +1,69 @@
|
||||
# Known Quirks
|
||||
|
||||
Per-service gotchas that aren't bugs but will bite you if you forget them.
|
||||
|
||||
## PlausibleDeniability
|
||||
|
||||
### immich-ml
|
||||
- Healthcheck has no curl/wget — uses `/proc/net/tcp6` pattern
|
||||
- Cache goes to `/mnt/docker-ssd/docker/appdata/immich-ml` (separate from immich-server's appdata)
|
||||
|
||||
### rackpeek
|
||||
- No curl/wget in image — healthcheck uses `/proc/net/tcp6`
|
||||
|
||||
### donetick
|
||||
- Env vars alone are insufficient for DB type selection
|
||||
- Requires `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml` with full DB config
|
||||
- Uses viper config loader — `DT_ENV` controls config file path
|
||||
|
||||
### shlink
|
||||
- Data directory must be `chmod 777` — runs as non-root user that doesn't match host default ownership
|
||||
|
||||
### openwebui
|
||||
- DB: user=`openwebui`, db=`openwebui` on shared-postgres
|
||||
- If restart-looping with auth errors: container wasn't on `ix-databases_shared-databases` at creation time — must `down && up`
|
||||
|
||||
### plex
|
||||
- Port `5353/udp` conflicts with system avahi/mDNS — remove from port mappings
|
||||
|
||||
### scriberr (shelved)
|
||||
- SQLite incompatible with ZFS nfsv4 ACLs (SQLITE_CANTOPEN error 14)
|
||||
- Revisit when Postgres support available in CUDA image
|
||||
|
||||
### meshmonitor (auto-upgrade)
|
||||
- `AUTO_UPGRADE_ENABLED=true` causes crashes on TrueNAS if bind mount source directories don't exist
|
||||
- The upgrader container mounts compose dir as `/compose` — relative paths resolve differently
|
||||
- Fix: pre-create all bind mount source directories before deploying
|
||||
|
||||
### Any container using `down && up` requirement
|
||||
- Network attachments happen at container creation, not restart
|
||||
- `docker restart` or `docker compose restart` will NOT fix missing network attachments
|
||||
- Must use `docker compose --env-file .env down <service> && docker compose --env-file .env up -d <service>`
|
||||
|
||||
---
|
||||
|
||||
## Serenity
|
||||
|
||||
### qBit Mover Script
|
||||
- Pauses torrents 0–4 days old before running mover to prevent partial file moves
|
||||
- MAM-init script updates MyAnonamouse IP inside qbit container on startup
|
||||
|
||||
### Unraid-Cloudflared-Tunnel
|
||||
- Dead container, should be removed
|
||||
|
||||
---
|
||||
|
||||
## N.O.M.A.D.
|
||||
|
||||
### NVIDIA Container Toolkit
|
||||
- Repo has invalid GPG key — remove `/etc/apt/sources.list.d/*nvidia*` if apt errors
|
||||
|
||||
### Pelican / Wings
|
||||
- Interactive artisan commands break over SSH (stty issue) — always use inline flags:
|
||||
`--email=x --username=y --password=z --admin=1`
|
||||
- Wings CORS derived from `remote:` URL, not `allowed_origins` — keep remote as `https://`
|
||||
- `panel.paccoco.com` in `/etc/hosts` → `127.0.0.1` for NAT loopback
|
||||
- Wings uses `--ignore-certificate-errors` for local self-signed cert
|
||||
|
||||
### Port 8080
|
||||
- Occupied by `nomad_admin` container — Wings uses 8443 instead
|
||||
129
docs/troubleshooting/TROUBLESHOOTING.md
Normal file
129
docs/troubleshooting/TROUBLESHOOTING.md
Normal file
@@ -0,0 +1,129 @@
|
||||
# Troubleshooting Guide
|
||||
|
||||
## Permission Denied / Readonly Database / Not Writable
|
||||
|
||||
Common causes:
|
||||
- App placed on NFS (tank) when it needs local SSD
|
||||
- NFS ownership mismatch between host user and container user
|
||||
|
||||
Checks:
|
||||
1. `docker logs <container> --tail 30`
|
||||
2. `ls -la <host-appdata-path>` — check ownership and permissions
|
||||
3. `docker run --rm --entrypoint sh <image> -lc 'id'` — check container UID/GID
|
||||
|
||||
Fix options:
|
||||
- Move appdata to `/mnt/docker-ssd/docker/appdata/<service>` if write-heavy
|
||||
- `sudo chmod -R 777 <appdata-path>` for apps with fixed internal user (e.g. shlink)
|
||||
- `sudo chown -R <uid>:<gid> <appdata-path>` if UID/GID is known
|
||||
|
||||
## SQLite Fails on ZFS with nfsv4 ACLs (SQLITE_CANTOPEN error 14)
|
||||
|
||||
Symptom: `unable to open database file: out of memory (14)`
|
||||
|
||||
Cause: SQLite cannot acquire POSIX file locks on ZFS datasets configured with `acltype=nfsv4`.
|
||||
|
||||
Diagnosis:
|
||||
```bash
|
||||
sudo zfs get acltype,aclmode,xattr <dataset>
|
||||
```
|
||||
|
||||
Fix options:
|
||||
1. Change acltype: `sudo zfs set acltype=posix <dataset>`
|
||||
2. Move the SQLite database to local SSD
|
||||
3. Use an app version that supports Postgres instead of SQLite
|
||||
|
||||
Note: scriberr CUDA image is shelved for this reason.
|
||||
|
||||
## Container Can't Resolve Shared Database Hostname
|
||||
|
||||
Symptom: `lookup shared-postgres on 127.0.0.11:53: no such host`
|
||||
|
||||
Fix:
|
||||
```bash
|
||||
docker compose --env-file .env down <service>
|
||||
docker compose --env-file .env up -d <service>
|
||||
```
|
||||
|
||||
Verify network attachment:
|
||||
```bash
|
||||
docker inspect <container> --format '{{json .NetworkSettings.Networks}}' | tr ',' '\n' | grep -o '"[^"]*"'
|
||||
```
|
||||
|
||||
## Postgres Auth Fails Over TCP but Works via Socket
|
||||
|
||||
Diagnosis:
|
||||
```bash
|
||||
sudo docker exec shared-postgres psql -h 127.0.0.1 -U <user> -d <db> -W -c "\conninfo"
|
||||
```
|
||||
|
||||
Fix:
|
||||
- Verify pg_hba.conf has `host all all all scram-sha-256`
|
||||
- Check DATABASE_URL password matches: `ALTER USER <user> WITH PASSWORD '<pass>';`
|
||||
- Avoid special characters in passwords in DATABASE_URLs
|
||||
|
||||
## Healthcheck Failing Despite Container Being Healthy
|
||||
|
||||
Diagnosis:
|
||||
```bash
|
||||
sudo docker exec <container> sh -c "which curl; which wget" 2>&1
|
||||
sudo docker inspect <container> --format '{{json .State.Health.Log}}' | python3 -m json.tool | tail -20
|
||||
```
|
||||
|
||||
After fixing the compose file, must do `down && up` (not just restart).
|
||||
|
||||
## App Ignores Environment Variables for DB Type
|
||||
|
||||
Some apps (e.g. donetick) require a YAML config file for certain settings even when env vars are set.
|
||||
|
||||
For donetick, create `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml`:
|
||||
```yaml
|
||||
database:
|
||||
type: postgres
|
||||
host: shared-postgres
|
||||
port: 5432
|
||||
user: donetick
|
||||
password: <password>
|
||||
name: donetick
|
||||
migration: true
|
||||
jwt:
|
||||
secret: <32-byte-hex>
|
||||
session_time: 168h
|
||||
max_refresh: 168h
|
||||
server:
|
||||
port: 2021
|
||||
serve_frontend: true
|
||||
```
|
||||
|
||||
## JWT Invalid / Login Fails After Restart
|
||||
|
||||
Cause: JWT secret changed. Fix: open app in incognito or clear cookies/localStorage.
|
||||
|
||||
## Port Already In Use
|
||||
|
||||
Common offenders:
|
||||
- `5353/udp` — held by system avahi/mDNS. Remove from Plex ports.
|
||||
- `3306`, `5432`, `6379` — only one DB stack can bind these.
|
||||
|
||||
## Auto-Upgrade Crashes on TrueNAS (exit code 128)
|
||||
|
||||
Symptom: `error while creating mount source path '/compose/scripts': mkdir /compose: read-only file system`
|
||||
|
||||
Cause: Auto-upgrade container runs docker compose from inside a container where the project dir is mounted as `/compose`. Relative bind mount paths like `./scripts` resolve to `/compose/scripts` on the host. TrueNAS root filesystem is read-only so Docker can't auto-create the directory.
|
||||
|
||||
Fix: Pre-create the directory manually on the host before starting the stack:
|
||||
```bash
|
||||
mkdir -p /mnt/docker-ssd/docker/compose/<stack>/scripts
|
||||
```
|
||||
|
||||
## Databases Stack Network Name Wrong
|
||||
|
||||
Must have `name: ix-databases` at top of databases compose. Verify:
|
||||
```bash
|
||||
docker network ls | grep databases
|
||||
# Should show: ix-databases_shared-databases
|
||||
```
|
||||
|
||||
## Checking Container Runtime UID
|
||||
```bash
|
||||
docker run --rm --entrypoint sh <image> -lc 'id'
|
||||
```
|
||||
Reference in New Issue
Block a user