docs: add full homelab documentation structure
This commit is contained in:
Paccoco
73
README.md
73
README.md
@@ -1,35 +1,60 @@
|
|||||||
# truenas-stacks
|
# truenas-stacks
|
||||||
|
|
||||||
Docker Compose stacks for PlausibleDeniability (TrueNAS Scale).
|
Docker Compose stacks for PlausibleDeniability (TrueNAS Scale), plus documentation for the full homelab.
|
||||||
|
|
||||||
## Stacks
|
## Servers
|
||||||
|
|
||||||
- databases/ - shared-postgres, shared-mariadb, shared-redis - deploy FIRST
|
| Server | Role | OS | IP |
|
||||||
- infrastructure/ - newt, homepage, uptime-kuma, dockhand, netdata, tailscale - deploy SECOND
|
|--------|------|----|----|
|
||||||
- media/ - plex, tautulli, audiobookshelf, seerr, calibre-web
|
| Serenity | NAS / Media ingestion | Unraid 7.2.4 | 10.5.1.5 |
|
||||||
- photos/ - immich server + machine learning
|
| PlausibleDeniability | Primary Docker host | TrueNAS Scale 25.10.2.1 | LAN DHCP |
|
||||||
- home/ - kitchenowl, donetick, dakboard bridge, shlink, qui
|
| N.O.M.A.D. | Offline knowledge + game servers | Ubuntu 25.10 | 10.5.1.16 |
|
||||||
- dev/ - gitea, rackpeek
|
|
||||||
- ai/ - ollama, openwebui, scriberr
|
|
||||||
|
|
||||||
## Deployment order
|
## Repo Layout
|
||||||
|
|
||||||
1. databases
|
```
|
||||||
2. infrastructure
|
databases/ # shared-postgres, shared-mariadb, shared-redis — deploy FIRST
|
||||||
3. All other stacks in any order
|
infrastructure/ # newt, homepage, uptime-kuma, dockhand, netdata, tailscale — deploy SECOND
|
||||||
|
media/ # plex, tautulli, audiobookshelf, seerr, calibre-web-automated
|
||||||
|
photos/ # immich server + machine learning
|
||||||
|
home/ # kitchenowl, donetick, shlink, qui, dakboard bridges
|
||||||
|
dev/ # gitea, rackpeek
|
||||||
|
ai/ # ollama, openwebui
|
||||||
|
meshtastic/ # meshmonitor, mqtt-proxy, tileserver, meshtastic-map
|
||||||
|
automation/ # planned: gotify, n8n
|
||||||
|
search/ # planned: qdrant
|
||||||
|
docs/ # all homelab documentation
|
||||||
|
```
|
||||||
|
|
||||||
## Setup on a new machine
|
## Quick Start
|
||||||
|
|
||||||
git clone git@github.com:Paccoco/truenas-stacks.git /mnt/docker-ssd/docker/compose
|
1. Deploy `databases/` first
|
||||||
cp databases/.env.example databases/.env && nano databases/.env
|
2. Deploy `infrastructure/` second (creates pangolin network via newt)
|
||||||
# repeat for each stack, then import each truenas-app.yaml via TrueNAS Apps UI
|
3. Deploy remaining stacks in any order
|
||||||
|
|
||||||
## Networks
|
```bash
|
||||||
|
cd /mnt/docker-ssd/docker/compose/<stack>
|
||||||
|
docker compose --env-file .env config # validate
|
||||||
|
docker compose --env-file .env up -d # deploy
|
||||||
|
```
|
||||||
|
|
||||||
- ix-databases_shared-databases: created by databases stack, used by media/photos/home/dev/ai
|
## Documentation
|
||||||
- pangolin: created by newt at runtime, used by any service exposed via reverse proxy
|
|
||||||
|
|
||||||
## Secrets
|
- [Architecture Overview](docs/architecture/ARCHITECTURE_OVERVIEW.md)
|
||||||
|
- [Networking Model](docs/architecture/NETWORKING_MODEL.md)
|
||||||
.env files are gitignored. .env.example files are committed with placeholder values.
|
- [Hardware Inventory](docs/architecture/HARDWARE_INVENTORY.md)
|
||||||
Real .env files live only on TrueNAS at /mnt/docker-ssd/docker/compose/<stack>/.env
|
- [Services Directory](docs/architecture/SERVICES_DIRECTORY.md)
|
||||||
|
- [Stack Standards](docs/operations/STACK_STANDARDS.md)
|
||||||
|
- [Deployment Checklist](docs/operations/DEPLOYMENT_CHECKLIST.md)
|
||||||
|
- [Storage Notes](docs/operations/DOCKERSTACK_STORAGE_NOTES.md)
|
||||||
|
- [Backup Policy](docs/operations/BACKUP_POLICY.md)
|
||||||
|
- [Secrets Management](docs/operations/SECRETS_MANAGEMENT.md)
|
||||||
|
- [Troubleshooting](docs/troubleshooting/TROUBLESHOOTING.md)
|
||||||
|
- [Known Quirks](docs/troubleshooting/KNOWN_QUIRKS.md)
|
||||||
|
- [Expansion Plan](docs/planning/HOMELAB_EXPANSION_PLAN.md)
|
||||||
|
- [TODO](docs/planning/TODO.md)
|
||||||
|
- [Serenity](docs/servers/SERENITY.md)
|
||||||
|
- [PlausibleDeniability](docs/servers/PLAUSIBLEDENABILITY.md)
|
||||||
|
- [N.O.M.A.D.](docs/servers/NOMAD.md)
|
||||||
|
- [Meshtastic Stack](docs/reference/MESHTASTIC.md)
|
||||||
|
- [AI Context](docs/reference/HOMELAB_AI_CONTEXT.md)
|
||||||
|
|||||||
0
docs/README.md
Normal file
0
docs/README.md
Normal file
107
docs/architecture/ARCHITECTURE_OVERVIEW.md
Normal file
107
docs/architecture/ARCHITECTURE_OVERVIEW.md
Normal file
@@ -0,0 +1,107 @@
|
|||||||
|
# Homelab Architecture Overview
|
||||||
|
|
||||||
|
## Primary Docker Host
|
||||||
|
Host: `PlausibleDeniability` (TrueNAS Scale)
|
||||||
|
|
||||||
|
Production Docker workloads run on PlausibleDeniability via standalone Docker Compose stacks.
|
||||||
|
TrueNAS Apps (ix-app format) have been fully migrated to compose stacks.
|
||||||
|
|
||||||
|
## Repository Layout
|
||||||
|
Repo root: `/mnt/docker-ssd/docker/compose`
|
||||||
|
Remote: `git@github.com:Paccoco/truenas-stacks.git`
|
||||||
|
Gitea mirror: `https://gitea.paccoco.com/fizzlepoof/truenas-stacks.git`
|
||||||
|
|
||||||
|
Stack directories:
|
||||||
|
- `databases/` = shared-postgres, shared-mariadb, shared-redis — deploy FIRST
|
||||||
|
- `infrastructure/` = newt, homepage, uptime-kuma, dockhand, netdata, tailscale — deploy SECOND
|
||||||
|
- `media/` = plex, tautulli, audiobookshelf, seerr, calibre-web-automated
|
||||||
|
- `photos/` = immich server + machine learning
|
||||||
|
- `home/` = kitchenowl, donetick, dakboard bridge, shlink, shlink-web-client, qui
|
||||||
|
- `dev/` = gitea, rackpeek
|
||||||
|
- `ai/` = ollama, openwebui
|
||||||
|
- `meshtastic/` = meshmonitor, mqtt-proxy, tileserver, meshtastic-map
|
||||||
|
|
||||||
|
## Storage Model
|
||||||
|
|
||||||
|
### Local SSD
|
||||||
|
Mounted at: `/mnt/docker-ssd`
|
||||||
|
- Docker data root: `/mnt/docker-ssd/docker/docker-data`
|
||||||
|
- Databases: `/mnt/docker-ssd/docker/databases/`
|
||||||
|
- Write-heavy appdata: `/mnt/docker-ssd/docker/appdata/`
|
||||||
|
- GPU/model storage
|
||||||
|
- Apps that perform chown or aggressive startup writes
|
||||||
|
- SQLite-backed apps
|
||||||
|
|
||||||
|
Notable appdata on SSD:
|
||||||
|
- plex, tautulli, audiobookshelf, seerr, dockhand, rackpeek
|
||||||
|
- immich, immich-ml, ollama, openwebui
|
||||||
|
- shlink, gitea, netdata, tailscale, uptime-kuma
|
||||||
|
|
||||||
|
### TrueNAS Pool (tank)
|
||||||
|
Mounted at: `/mnt/tank`
|
||||||
|
- General appdata: `/mnt/tank/docker/appdata/`
|
||||||
|
- Backups: `/mnt/tank/docker/backups/`
|
||||||
|
- Downloads: `/mnt/tank/docker/downloads/`
|
||||||
|
|
||||||
|
Notable appdata on tank:
|
||||||
|
- homepage, newt, dockhand (config only)
|
||||||
|
- kitchenowl, donetick, qui
|
||||||
|
- calibre-web-automated config
|
||||||
|
|
||||||
|
### Unraid NFS
|
||||||
|
Mounted at: `/mnt/unraid`
|
||||||
|
- Media libraries: `/mnt/unraid/data/media/`
|
||||||
|
- Immich media: `/mnt/unraid/immich`
|
||||||
|
|
||||||
|
## Networks
|
||||||
|
- `ix-databases_shared-databases` — created by databases stack; project name `ix-databases` set in compose
|
||||||
|
- `pangolin` — created by newt at runtime; used by all externally exposed services
|
||||||
|
|
||||||
|
## Active Stacks (as of 2026-05-05)
|
||||||
|
|
||||||
|
### databases/
|
||||||
|
- shared-postgres (immich-optimized image with vectorchord + pgvector)
|
||||||
|
- shared-mariadb
|
||||||
|
- shared-redis
|
||||||
|
|
||||||
|
### infrastructure/
|
||||||
|
- newt, homepage, dockhand, uptime-kuma, netdata, tailscale
|
||||||
|
|
||||||
|
### media/
|
||||||
|
- ix-plex-plex-1 (GPU passthrough, RTX 2080 Ti)
|
||||||
|
- ix-tautulli-tautulli-1
|
||||||
|
- ix-audiobookshelf-audiobookshelf-1
|
||||||
|
- seerr (Postgres backend)
|
||||||
|
- calibre-web-automated
|
||||||
|
|
||||||
|
### photos/
|
||||||
|
- ix-immich-immich-server-1
|
||||||
|
- ix-immich-immich-machine-learning-1 (GPU, CUDA)
|
||||||
|
|
||||||
|
### home/
|
||||||
|
- kitchenowl, donetick, donetick-dakboard-bridge, qui, shlink, shlink-web-client
|
||||||
|
|
||||||
|
### dev/
|
||||||
|
- gitea, ix-rackpeek-rackpeek-1
|
||||||
|
|
||||||
|
### ai/
|
||||||
|
- ollama (GPU, RTX 2080 Ti, port 11434)
|
||||||
|
- openwebui (Postgres backend, port 8282)
|
||||||
|
|
||||||
|
### meshtastic/
|
||||||
|
- meshmonitor, meshmonitor-mqtt-proxy, meshmonitor-tileserver, meshtastic-map
|
||||||
|
|
||||||
|
## Planned / Not Yet Deployed
|
||||||
|
- dispatcharr, gotify, bytestash, kima-hub, n8n, qdrant, paperless-ngx, home-assistant
|
||||||
|
|
||||||
|
## Shelved
|
||||||
|
- scriberr — SQLite incompatible with ZFS nfsv4 ACLs
|
||||||
|
|
||||||
|
## Design Principles
|
||||||
|
- deterministic storage paths
|
||||||
|
- no anonymous volumes
|
||||||
|
- stack-local .env for secrets; .env.example committed with placeholders
|
||||||
|
- SSD for write-heavy / DB / SQLite / GPU workloads
|
||||||
|
- tank NFS for general appdata and configs
|
||||||
|
- Unraid NFS for media only
|
||||||
|
- databases stack uses `name: ix-databases` to maintain network name compatibility
|
||||||
77
docs/architecture/HARDWARE_INVENTORY.md
Normal file
77
docs/architecture/HARDWARE_INVENTORY.md
Normal file
@@ -0,0 +1,77 @@
|
|||||||
|
# Hardware Inventory
|
||||||
|
|
||||||
|
## Serenity (NAS / Media Ingestion)
|
||||||
|
- **OS:** Unraid 7.2.4
|
||||||
|
- **IP:** 10.5.1.5 | Tailscale: 100.94.87.79
|
||||||
|
- **CPU:** Xeon X5690
|
||||||
|
- **RAM:** 102GB
|
||||||
|
- **GPU:** None (Matrox IPMI only)
|
||||||
|
- **UPS:** None
|
||||||
|
|
||||||
|
### Storage
|
||||||
|
| Pool | Type | Drives | Usable | Notes |
|
||||||
|
|------|------|--------|--------|-------|
|
||||||
|
| malcolm | RAIDZ1 | 6× 10TB HDD | ~40TB | Heavily utilized |
|
||||||
|
| cache | ZFS single | 1× Kingston SNV2S 1TB NVMe | ~1TB | Appdata/Docker |
|
||||||
|
|
||||||
|
### Key Shares
|
||||||
|
| Share | Size | Purpose |
|
||||||
|
|-------|------|---------|
|
||||||
|
| data | 35TB | Media library |
|
||||||
|
| immich | 62GB | Immich photo storage |
|
||||||
|
| nextcloud | 7.4GB | Nextcloud data |
|
||||||
|
| Tesla/dashcam | 1.6TB | Dashcam footage |
|
||||||
|
| fizzlebang | 17GB | Personal files |
|
||||||
|
| isos | 22GB | ISO images |
|
||||||
|
| Syncthing | 4.6GB | Syncthing data |
|
||||||
|
| manndra | — | Partner's storage |
|
||||||
|
| Long and Low Crafts | — | Crafts business |
|
||||||
|
| zfs_backups_array | — | ZFS backup target |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PlausibleDeniability (Primary Docker Host)
|
||||||
|
- **OS:** TrueNAS Scale 25.10.2.1
|
||||||
|
- **CPU:** Ryzen 9 3950X
|
||||||
|
- **RAM:** 32GB
|
||||||
|
- **GPU:** Nvidia RTX 2080 Ti (11GB VRAM) — Plex, Immich ML, Ollama
|
||||||
|
- **UPS:** None
|
||||||
|
|
||||||
|
### Storage
|
||||||
|
| Pool | Type | Mount | Notes |
|
||||||
|
|------|------|-------|-------|
|
||||||
|
| tank | ZFS mirror (2 drives) | `/mnt/tank` | ~3.5TB usable |
|
||||||
|
| docker-ssd | Single Crucial MX500 500GB | `/mnt/docker-ssd` | No redundancy |
|
||||||
|
| boot-pool | Single sdc3 | — | TrueNAS boot |
|
||||||
|
|
||||||
|
### NFS Mounts
|
||||||
|
- Serenity data array → `/mnt/unraid/data`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## N.O.M.A.D. (Offline Knowledge + Game Servers)
|
||||||
|
- **OS:** Ubuntu 25.10 (bare metal)
|
||||||
|
- **IP:** 10.5.1.16
|
||||||
|
- **CPU:** Intel Core i7-4770K @ 3.50GHz (4c/8t, up to 3.9GHz)
|
||||||
|
- **RAM:** 32GB DDR3 1600MHz (4×8GB)
|
||||||
|
- **GPU:** NVIDIA GeForce GTX 1080 (8GB VRAM)
|
||||||
|
- **NIC:** Intel I218-V (1GbE) + Aquantia AQC113 (10GbE)
|
||||||
|
- **Motherboard:** ASRock H97M Pro4
|
||||||
|
- **UPS:** None
|
||||||
|
|
||||||
|
### Storage
|
||||||
|
| Device | Size | Mount | Purpose |
|
||||||
|
|--------|------|-------|---------|
|
||||||
|
| Samsung SSD 840 (sdb) | 232.9GB | `/` (LVM 100GB) | OS boot |
|
||||||
|
| WD Red HDD (sda) | 3.6TB | `/mnt/hdd-1` | Primary data, project-nomad symlink |
|
||||||
|
| WD Red HDD (sdc) | 3.6TB | `/mnt/hdd-2` | Docker data-root |
|
||||||
|
| Seagate SSD (sdd) | 223.6GB | `/mnt/ssd-1` | Fast scratch |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Desktop: ROCINANTE
|
||||||
|
- **OS:** Windows 11 Pro
|
||||||
|
- **CPU:** Ryzen 7 9850X3D
|
||||||
|
- **RAM:** 64GB
|
||||||
|
- **GPU:** RTX 4090 (24GB VRAM)
|
||||||
|
- **Role:** Heavy AI inference, local Ollama (expose via `OLLAMA_HOST=0.0.0.0`)
|
||||||
56
docs/architecture/NETWORKING_MODEL.md
Normal file
56
docs/architecture/NETWORKING_MODEL.md
Normal file
@@ -0,0 +1,56 @@
|
|||||||
|
# Homelab Networking Model
|
||||||
|
|
||||||
|
## Physical Network
|
||||||
|
- **Gear:** Unifi throughout
|
||||||
|
- **Main LAN:** 10.5.1.x (Trusted VLAN)
|
||||||
|
- **Other VLANs:** Cameras, Guest, IoT
|
||||||
|
- **Serenity IP:** 10.5.1.5
|
||||||
|
- **N.O.M.A.D. IP:** 10.5.1.16
|
||||||
|
- **PlausibleDeniability:** DHCP on LAN
|
||||||
|
|
||||||
|
## Remote Access
|
||||||
|
- **Pangolin VPS + Newt:** Services exposed as subdomains via reverse proxy. Newt agents run on PD and N.O.M.A.D.
|
||||||
|
- **Tailscale:** Personal remote access, runs on Serenity (100.94.87.79)
|
||||||
|
- **Cloudflared:** No longer used. Dead container on Serenity to be removed.
|
||||||
|
|
||||||
|
## DNS
|
||||||
|
- **Provider:** Cloudflare
|
||||||
|
- All public subdomains point at Pangolin VPS IP
|
||||||
|
|
||||||
|
## Known Public Subdomains
|
||||||
|
- `paccoco.com` → Pangolin VPS
|
||||||
|
- `panel.paccoco.com` → Pelican Panel (N.O.M.A.D.)
|
||||||
|
- `node1.paccoco.com` → Wings daemon (N.O.M.A.D., no auth)
|
||||||
|
- `meshmonitor.paccoco.com` → Meshmonitor (PD)
|
||||||
|
- `gitea.paccoco.com` → Gitea (PD)
|
||||||
|
- `ai.paccoco.com` → OpenWebUI (PD) — DNS/Pangolin resource pending
|
||||||
|
|
||||||
|
## Docker Networking (PlausibleDeniability)
|
||||||
|
|
||||||
|
### ix-databases_shared-databases
|
||||||
|
Created by the `databases` stack. The databases compose file uses `name: ix-databases` so Docker names the network `ix-databases_shared-databases`, matching what all other stacks declare as external.
|
||||||
|
|
||||||
|
Any stack connecting to shared-postgres, shared-mariadb, or shared-redis must declare:
|
||||||
|
```yaml
|
||||||
|
networks:
|
||||||
|
ix-databases_shared-databases:
|
||||||
|
external: true
|
||||||
|
```
|
||||||
|
|
||||||
|
### pangolin
|
||||||
|
Created at runtime by the `newt` container in the infrastructure stack. Any service needing reverse-proxy exposure must declare:
|
||||||
|
```yaml
|
||||||
|
networks:
|
||||||
|
pangolin:
|
||||||
|
external: true
|
||||||
|
```
|
||||||
|
|
||||||
|
### Internal Service Model
|
||||||
|
Internal services run HTTP only behind the reverse proxy. Compose service names used for internal DNS between services in the same stack/network.
|
||||||
|
|
||||||
|
## N.O.M.A.D. Networking Notes
|
||||||
|
- `panel.paccoco.com` added to `/etc/hosts` → `127.0.0.1` for NAT loopback
|
||||||
|
- nginx listens on 80 and 443 (self-signed cert) for Wings → Panel communication
|
||||||
|
- Wings uses `--ignore-certificate-errors` for local loopback
|
||||||
|
- Pangolin resource `node1.paccoco.com` has auth disabled (Wings handles its own auth)
|
||||||
|
- Minecraft port 25565 TCP/UDP goes directly via home router port forward, not through Pangolin
|
||||||
103
docs/architecture/SERVICES_DIRECTORY.md
Normal file
103
docs/architecture/SERVICES_DIRECTORY.md
Normal file
@@ -0,0 +1,103 @@
|
|||||||
|
# Services Directory
|
||||||
|
|
||||||
|
All running services across the homelab with ports, URLs, and backend info.
|
||||||
|
|
||||||
|
## PlausibleDeniability
|
||||||
|
|
||||||
|
### Databases (internal only)
|
||||||
|
| Service | Port | Backend | Notes |
|
||||||
|
|---------|------|---------|-------|
|
||||||
|
| shared-postgres | 5432 | — | immich-optimized, vectorchord + pgvector |
|
||||||
|
| shared-mariadb | 3306 | — | |
|
||||||
|
| shared-redis | 6379 | — | |
|
||||||
|
|
||||||
|
### Infrastructure
|
||||||
|
| Service | Internal Port | Public URL | Notes |
|
||||||
|
|---------|--------------|------------|-------|
|
||||||
|
| newt | — | — | Pangolin reverse proxy agent |
|
||||||
|
| homepage | — | — | Dashboard |
|
||||||
|
| dockhand | — | — | Container management |
|
||||||
|
| uptime-kuma | — | — | MariaDB backend |
|
||||||
|
| netdata | — | — | Metrics |
|
||||||
|
| tailscale | — | — | VPN |
|
||||||
|
|
||||||
|
### Media
|
||||||
|
| Service | Internal Port | Public URL | Notes |
|
||||||
|
|---------|--------------|------------|-------|
|
||||||
|
| plex | — | — | GPU passthrough, RTX 2080 Ti |
|
||||||
|
| tautulli | — | — | Plex analytics |
|
||||||
|
| audiobookshelf | — | — | Audiobooks |
|
||||||
|
| seerr | — | — | Postgres backend |
|
||||||
|
| calibre-web-automated | — | — | Ebooks |
|
||||||
|
|
||||||
|
### Photos
|
||||||
|
| Service | Internal Port | Public URL | Notes |
|
||||||
|
|---------|--------------|------------|-------|
|
||||||
|
| immich-server | — | — | Postgres + Redis, GPU |
|
||||||
|
| immich-ml | — | — | GPU, CUDA |
|
||||||
|
|
||||||
|
### Home
|
||||||
|
| Service | Internal Port | Public URL | Notes |
|
||||||
|
|---------|--------------|------------|-------|
|
||||||
|
| kitchenowl | — | — | Grocery/tasks |
|
||||||
|
| donetick | — | — | Postgres backend, selfhosted.yaml required |
|
||||||
|
| shlink | — | — | URL shortener, Postgres |
|
||||||
|
| shlink-web-client | — | — | |
|
||||||
|
| qui | — | — | qBittorrent UI, manages Serenity remotely |
|
||||||
|
|
||||||
|
### Dev
|
||||||
|
| Service | Internal Port | Public URL | Notes |
|
||||||
|
|---------|--------------|------------|-------|
|
||||||
|
| gitea | — | gitea.paccoco.com | Postgres backend |
|
||||||
|
| rackpeek | — | — | |
|
||||||
|
|
||||||
|
### AI
|
||||||
|
| Service | Internal Port | Public URL | Notes |
|
||||||
|
|---------|--------------|------------|-------|
|
||||||
|
| ollama | 11434 | — | GPU, RTX 2080 Ti |
|
||||||
|
| openwebui | 8282→8080 | ai.paccoco.com (pending) | Postgres backend |
|
||||||
|
|
||||||
|
### Meshtastic
|
||||||
|
| Service | Internal Port | Public URL | Notes |
|
||||||
|
|---------|--------------|------------|-------|
|
||||||
|
| meshmonitor | 8081→3001 | meshmonitor.paccoco.com | Postgres backend |
|
||||||
|
| meshmonitor virtual node | 4404 | — | TCP virtual node server |
|
||||||
|
| meshmonitor-tileserver | 8082→8080 | — | Map tiles |
|
||||||
|
| meshmonitor-mqtt-proxy | — | — | MQTT→TCP bridge |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Serenity
|
||||||
|
|
||||||
|
### Media Ingestion
|
||||||
|
| Service | Notes |
|
||||||
|
|---------|-------|
|
||||||
|
| sonarr | TV |
|
||||||
|
| sonarr-anime | Anime |
|
||||||
|
| radarr | Movies |
|
||||||
|
| lidarr | Music |
|
||||||
|
| readarr | Books |
|
||||||
|
| readarr-epub | EPUBs |
|
||||||
|
| prowlarr | Indexer manager |
|
||||||
|
| bazarr | Subtitles |
|
||||||
|
| unpackerr | Archive extraction |
|
||||||
|
| qBittorrent | Download client |
|
||||||
|
| GluetunVPN | VPN tunnel for qbit |
|
||||||
|
| autobrr | Auto-grab |
|
||||||
|
| qbit_manage | qBit maintenance |
|
||||||
|
| Notifiarr | Notifications |
|
||||||
|
| Wizarr | User invites |
|
||||||
|
| romm | ROM manager |
|
||||||
|
| GameVault | Game library |
|
||||||
|
| shelfmark | Book/audiobook search, port 8084 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## N.O.M.A.D.
|
||||||
|
|
||||||
|
| Service | Port | Notes |
|
||||||
|
|---------|------|-------|
|
||||||
|
| Pelican Panel | 80/443 | panel.paccoco.com, PHP 8.4 + nginx + MariaDB |
|
||||||
|
| Wings | 8443 | node1.paccoco.com, game server daemon |
|
||||||
|
| Minecraft | 25565 | Purpur, Java 21, direct port forward |
|
||||||
|
| N.O.M.A.D. containers | — | 10 containers, start_nomad.sh / stop_nomad.sh |
|
||||||
42
docs/operations/BACKUP_POLICY.md
Normal file
42
docs/operations/BACKUP_POLICY.md
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
# Backup Policy
|
||||||
|
|
||||||
|
## Current State (Known Gaps)
|
||||||
|
- **No off-site backup** — this is a known gap, improvement pending
|
||||||
|
- Serenity has local ZFS snapshots and replication
|
||||||
|
- PlausibleDeniability has no automated backup of appdata or databases
|
||||||
|
|
||||||
|
## Serenity Backup Strategy
|
||||||
|
- Sanoid snapshots cache/appdata: 7 daily, 4 weekly, 6 monthly
|
||||||
|
- Syncoid replicates cache/appdata → malcolm/zfs_backups_array (strict-mirror)
|
||||||
|
- qBit Mover script: pauses torrents 0–4 days old then runs Unraid mover
|
||||||
|
- Mover Status script: sends Discord webhook notifications
|
||||||
|
|
||||||
|
## PlausibleDeniability Backup Scope
|
||||||
|
Back up at minimum:
|
||||||
|
- Compose repo: `/mnt/docker-ssd/docker/compose` (in git — GitHub + Gitea)
|
||||||
|
- Local SSD appdata: `/mnt/docker-ssd/docker/appdata/`
|
||||||
|
- Databases: `/mnt/docker-ssd/docker/databases/`
|
||||||
|
- Selected appdata/backups on tank: `/mnt/tank/docker/`
|
||||||
|
- .env files (separately — gitignored, handle via encrypted backup or private repo)
|
||||||
|
|
||||||
|
## Database Backups
|
||||||
|
- Keep app-consistent dumps for major databases
|
||||||
|
- Do not rely only on raw volume copies for restores
|
||||||
|
- Document restore procedure per major service
|
||||||
|
|
||||||
|
Recommended dump locations: `/mnt/tank/docker/backups/db-dumps/`
|
||||||
|
|
||||||
|
## .env File Backup Options
|
||||||
|
1. **Private Gitea repo** — short term, push all .env files to a private repo on your local Gitea
|
||||||
|
2. **Encrypted restic** — include .env files in restic backup with encryption, push offsite (Backblaze B2)
|
||||||
|
3. **Vaultwarden** — store each .env as a secure note (best long-term once deployed)
|
||||||
|
|
||||||
|
## Restore Practice
|
||||||
|
- Test restores quarterly
|
||||||
|
- Test at least one app DB restore and one full-config restore
|
||||||
|
- Verify that restored stacks can start with current compose files
|
||||||
|
|
||||||
|
## Operational Notes
|
||||||
|
- Media libraries on Unraid are large and may not need the same backup frequency as configs/databases
|
||||||
|
- SSD-local DB and SQLite workloads are highest priority for backup validation
|
||||||
|
- Serenity malcolm pool is heavily utilized — capacity planning needed before it fills
|
||||||
63
docs/operations/DEPLOYMENT_CHECKLIST.md
Normal file
63
docs/operations/DEPLOYMENT_CHECKLIST.md
Normal file
@@ -0,0 +1,63 @@
|
|||||||
|
# New Service Deployment Checklist
|
||||||
|
|
||||||
|
## Before Adding a Service
|
||||||
|
|
||||||
|
- [ ] Determine DB requirements (postgres, mariadb, sqlite, none)
|
||||||
|
- [ ] Decide storage tier: SSD (`/mnt/docker-ssd/docker/appdata/`) vs tank (`/mnt/tank/docker/appdata/`) vs Unraid
|
||||||
|
- [ ] Check if service needs `ix-databases_shared-databases` network
|
||||||
|
- [ ] Check if service needs `pangolin` network for reverse proxy exposure
|
||||||
|
|
||||||
|
## Scaffold
|
||||||
|
|
||||||
|
- [ ] Create host directories for appdata
|
||||||
|
- [ ] Set correct ownership if needed (`chown` or `chmod 777` for permission-sensitive apps)
|
||||||
|
- [ ] Create stack folder under `/mnt/docker-ssd/docker/compose/<stack>/`
|
||||||
|
- [ ] Create `docker-compose.yaml`
|
||||||
|
- [ ] Create `.env` and `.env.example`
|
||||||
|
- [ ] If using postgres: create DB and user, or add init SQL to `initdb/`
|
||||||
|
- [ ] If using mariadb: add init script to `initdb-mariadb/`
|
||||||
|
|
||||||
|
## Validate Before Deploying
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /mnt/docker-ssd/docker/compose/<stack>
|
||||||
|
docker compose --env-file .env config
|
||||||
|
```
|
||||||
|
|
||||||
|
No errors = safe to proceed.
|
||||||
|
|
||||||
|
## Deploy
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose --env-file .env up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
## After Starting
|
||||||
|
|
||||||
|
- [ ] `docker ps` shows container running
|
||||||
|
- [ ] `docker logs <container> --tail 20` shows clean startup
|
||||||
|
- [ ] `docker inspect <container>` confirms mounts are correct
|
||||||
|
- [ ] Data persists after `docker compose --env-file .env down && up -d`
|
||||||
|
|
||||||
|
## Network Checklist
|
||||||
|
|
||||||
|
If service needs shared databases:
|
||||||
|
- Top-level networks block declares `ix-databases_shared-databases` as external
|
||||||
|
- Service networks list includes `ix-databases_shared-databases`
|
||||||
|
- Verify: `docker inspect <container> --format '{{json .NetworkSettings.Networks}}'`
|
||||||
|
|
||||||
|
If service needs reverse proxy:
|
||||||
|
- Top-level networks block declares `pangolin` as external
|
||||||
|
- Service networks list includes `pangolin`
|
||||||
|
- Infrastructure stack (newt) must be running first
|
||||||
|
|
||||||
|
## Common Failure Modes
|
||||||
|
|
||||||
|
| Symptom | Likely Cause | Fix |
|
||||||
|
|---|---|---|
|
||||||
|
| `hostname resolving error` for shared-postgres | Not on `ix-databases_shared-databases` network | `down && up` not just restart |
|
||||||
|
| `not writable` errors on startup | NFS ownership mismatch | Move to SSD or chmod 777 |
|
||||||
|
| `unsupported database type` | App requires config file, not just env vars | Create `selfhosted.yaml` or equivalent |
|
||||||
|
| `JWT is invalid` after restart | JWT secret changed | Clear browser cookies/localStorage |
|
||||||
|
| Port conflict on startup | Port already bound | Remove conflicting port mapping |
|
||||||
|
| Auto-upgrade crashes on TrueNAS | Bind mount source path missing, Docker tries to create at root | Pre-create the directory manually |
|
||||||
61
docs/operations/DOCKERSTACK_STORAGE_NOTES.md
Normal file
61
docs/operations/DOCKERSTACK_STORAGE_NOTES.md
Normal file
@@ -0,0 +1,61 @@
|
|||||||
|
# Dockerstack Storage Notes
|
||||||
|
|
||||||
|
## Local SSD
|
||||||
|
Use `/docker/...` for:
|
||||||
|
- Docker root (`/docker/docker-data`)
|
||||||
|
- GPU/model storage
|
||||||
|
- SQLite-backed apps
|
||||||
|
- Apps that chown on startup
|
||||||
|
- Write-heavy DB/search storage
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
- `/mnt/docker-ssd/docker/docker-data`
|
||||||
|
- `/mnt/docker-ssd/docker/appdata/ollama`
|
||||||
|
- `/mnt/docker-ssd/docker/appdata/openwebui`
|
||||||
|
- `/mnt/docker-ssd/docker/appdata/immich`
|
||||||
|
- `/mnt/docker-ssd/docker/appdata/immich-ml`
|
||||||
|
- `/mnt/docker-ssd/docker/appdata/plex`
|
||||||
|
- `/mnt/docker-ssd/docker/appdata/tautulli`
|
||||||
|
- `/mnt/docker-ssd/docker/appdata/audiobookshelf`
|
||||||
|
- `/mnt/docker-ssd/docker/appdata/seerr`
|
||||||
|
- `/mnt/docker-ssd/docker/appdata/dockhand`
|
||||||
|
- `/mnt/docker-ssd/docker/appdata/rackpeek`
|
||||||
|
- `/mnt/docker-ssd/docker/appdata/shlink`
|
||||||
|
- `/mnt/docker-ssd/docker/appdata/gitea`
|
||||||
|
- `/mnt/docker-ssd/docker/appdata/netdata`
|
||||||
|
- `/mnt/docker-ssd/docker/appdata/tailscale`
|
||||||
|
- `/mnt/docker-ssd/docker/appdata/uptime-kuma`
|
||||||
|
- `/mnt/docker-ssd/docker/databases/postgres`
|
||||||
|
- `/mnt/docker-ssd/docker/databases/mariadb`
|
||||||
|
- `/mnt/docker-ssd/docker/databases/redis`
|
||||||
|
|
||||||
|
## TrueNAS NFS
|
||||||
|
Use `/mnt/tank/docker/...` for:
|
||||||
|
- Normal appdata
|
||||||
|
- Uploads
|
||||||
|
- Non-fussy config and data
|
||||||
|
- Backups
|
||||||
|
- Downloads
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
- `/mnt/tank/docker/appdata/homepage`
|
||||||
|
- `/mnt/tank/docker/appdata/newt`
|
||||||
|
- `/mnt/tank/docker/appdata/kitchenowl`
|
||||||
|
- `/mnt/tank/docker/appdata/donetick`
|
||||||
|
- `/mnt/tank/docker/appdata/qui`
|
||||||
|
- `/mnt/tank/docker/appdata/calibre-web-automated`
|
||||||
|
- `/mnt/tank/docker/backups`
|
||||||
|
- `/mnt/tank/docker/downloads`
|
||||||
|
|
||||||
|
## Unraid NFS
|
||||||
|
Use `/mnt/unraid/...` for media and shared bulk content only:
|
||||||
|
- `/mnt/unraid/data/media/` — Plex libraries, books, etc.
|
||||||
|
- `/mnt/unraid/immich` — Immich photo/video storage
|
||||||
|
|
||||||
|
## Rules Learned
|
||||||
|
- SQLite apps and apps that write logs aggressively should prefer local SSD.
|
||||||
|
- Database/search/model-heavy workloads should prefer local SSD.
|
||||||
|
- General config and less write-sensitive appdata can live on TrueNAS NFS.
|
||||||
|
- Media should stay on Unraid mounts.
|
||||||
|
- SQLite on ZFS with nfsv4 ACLs will always fail — move to SSD or use Postgres.
|
||||||
|
- Apps with auto-upgrade features that use bind mounts: pre-create all source directories manually on TrueNAS to prevent Docker from trying to create them at the read-only root.
|
||||||
64
docs/operations/SECRETS_MANAGEMENT.md
Normal file
64
docs/operations/SECRETS_MANAGEMENT.md
Normal file
@@ -0,0 +1,64 @@
|
|||||||
|
# Secrets Management
|
||||||
|
|
||||||
|
## Principles
|
||||||
|
- `.env` files are gitignored and never committed to any public repo
|
||||||
|
- `.env.example` files are committed with placeholder values
|
||||||
|
- Real `.env` files live only on the host at the stack directory
|
||||||
|
- Secrets are generated with standard tools, never reused across services
|
||||||
|
|
||||||
|
## Generating Secrets
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Database passwords
|
||||||
|
openssl rand -hex 24
|
||||||
|
|
||||||
|
# JWT / session secrets
|
||||||
|
openssl rand -hex 32
|
||||||
|
|
||||||
|
# API keys (where format allows)
|
||||||
|
openssl rand -base64 32
|
||||||
|
```
|
||||||
|
|
||||||
|
## .env File Locations
|
||||||
|
All `.env` files live alongside their `docker-compose.yaml`:
|
||||||
|
```
|
||||||
|
/mnt/docker-ssd/docker/compose/<stack>/.env
|
||||||
|
```
|
||||||
|
|
||||||
|
## Backing Up .env Files
|
||||||
|
|
||||||
|
### Short term: Private Gitea repo
|
||||||
|
Create a private repo on Gitea (not GitHub) and push all .env files there:
|
||||||
|
```bash
|
||||||
|
# In a separate directory, not the main stacks repo
|
||||||
|
git init homelab-secrets
|
||||||
|
cd homelab-secrets
|
||||||
|
# Copy and add .env files
|
||||||
|
git remote add origin https://gitea.paccoco.com/fizzlepoof/homelab-secrets.git
|
||||||
|
git push -u origin main
|
||||||
|
```
|
||||||
|
|
||||||
|
### Long term: Restic + offsite
|
||||||
|
Include the compose directory (with .env files) in a restic backup:
|
||||||
|
```bash
|
||||||
|
restic -r <repo> backup /mnt/docker-ssd/docker/compose --exclude="*.example"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Best long term: Vaultwarden
|
||||||
|
Once deployed, store each service's .env as a secure note in Vaultwarden. Single source of truth, accessible from anywhere.
|
||||||
|
|
||||||
|
## Known Credentials Locations
|
||||||
|
|
||||||
|
| Service | Where stored |
|
||||||
|
|---------|-------------|
|
||||||
|
| shared-postgres | `databases/.env` |
|
||||||
|
| shared-mariadb | `databases/.env` |
|
||||||
|
| OpenWebUI DB | `ai/.env` — user: openwebui, db: openwebui |
|
||||||
|
| Donetick | `home/.env` + `selfhosted.yaml` |
|
||||||
|
| Meshmonitor | `meshtastic/.env` |
|
||||||
|
| Gitea | `dev/.env` |
|
||||||
|
|
||||||
|
## Security Reminders
|
||||||
|
- Regenerate Wings token on N.O.M.A.D. (was exposed in chat — TODO)
|
||||||
|
- Regenerate N.O.M.A.D. Newt secret (was exposed in chat — TODO)
|
||||||
|
- Disable signups in OpenWebUI after creating initial account
|
||||||
122
docs/operations/STACK_STANDARDS.md
Normal file
122
docs/operations/STACK_STANDARDS.md
Normal file
@@ -0,0 +1,122 @@
|
|||||||
|
# Homelab Stack Standards
|
||||||
|
|
||||||
|
## Repository Root
|
||||||
|
`/mnt/docker-ssd/docker/compose`
|
||||||
|
|
||||||
|
Remotes:
|
||||||
|
- GitHub: `git@github.com:Paccoco/truenas-stacks.git`
|
||||||
|
- Gitea: `https://gitea.paccoco.com/fizzlepoof/truenas-stacks.git`
|
||||||
|
|
||||||
|
## Stack Layout
|
||||||
|
Each production stack lives in a subdirectory of the repo root.
|
||||||
|
|
||||||
|
Each stack contains:
|
||||||
|
- `docker-compose.yaml` — required
|
||||||
|
- `.env` — required (gitignored)
|
||||||
|
- `.env.example` — committed with placeholder values
|
||||||
|
- optional init scripts (e.g. `initdb/`, `initdb-mariadb/`)
|
||||||
|
|
||||||
|
## Deployment Order
|
||||||
|
1. `databases/` — must be up before anything else
|
||||||
|
2. `infrastructure/` — newt creates the pangolin network
|
||||||
|
3. All other stacks in any order
|
||||||
|
|
||||||
|
## Deployment Workflow
|
||||||
|
|
||||||
|
Validate before deploying:
|
||||||
|
```bash
|
||||||
|
cd /mnt/docker-ssd/docker/compose/<stack>
|
||||||
|
docker compose --env-file .env config
|
||||||
|
```
|
||||||
|
|
||||||
|
Deploy:
|
||||||
|
```bash
|
||||||
|
docker compose --env-file .env up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
Teardown a single service:
|
||||||
|
```bash
|
||||||
|
docker compose --env-file .env down <service>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Networks
|
||||||
|
|
||||||
|
### ix-databases_shared-databases
|
||||||
|
Created by the `databases` stack with `name: ix-databases` at top level. Any stack connecting to shared databases must declare:
|
||||||
|
```yaml
|
||||||
|
networks:
|
||||||
|
ix-databases_shared-databases:
|
||||||
|
external: true
|
||||||
|
```
|
||||||
|
|
||||||
|
### pangolin
|
||||||
|
Created by `newt` at runtime. Any externally exposed service must declare:
|
||||||
|
```yaml
|
||||||
|
networks:
|
||||||
|
pangolin:
|
||||||
|
external: true
|
||||||
|
```
|
||||||
|
|
||||||
|
## Storage Standards
|
||||||
|
|
||||||
|
### Local SSD: `/mnt/docker-ssd`
|
||||||
|
Use `/mnt/docker-ssd/docker/appdata/<service>` for:
|
||||||
|
- Postgres / MariaDB / Redis data
|
||||||
|
- SQLite-backed apps
|
||||||
|
- Apps that chown aggressively on startup
|
||||||
|
- GPU/model caches
|
||||||
|
- Write-heavy logs, cache, state
|
||||||
|
|
||||||
|
### TrueNAS Pool (tank): `/mnt/tank`
|
||||||
|
Use `/mnt/tank/docker/appdata/<service>` for:
|
||||||
|
- General appdata
|
||||||
|
- Configs
|
||||||
|
- Uploads
|
||||||
|
- Non-permission-sensitive state
|
||||||
|
|
||||||
|
### Unraid NFS: `/mnt/unraid`
|
||||||
|
- `/mnt/unraid/data/media/` — media libraries
|
||||||
|
- `/mnt/unraid/immich` — Immich photo/video storage
|
||||||
|
|
||||||
|
## Volume Rules
|
||||||
|
- no anonymous volumes
|
||||||
|
- no ad-hoc home directory paths
|
||||||
|
- all binds use standardized production paths
|
||||||
|
- evaluate SSD vs tank vs Unraid before deploying any new service
|
||||||
|
- apps known to have permission issues on NFS go on SSD
|
||||||
|
|
||||||
|
## Known Permission Quirks
|
||||||
|
- `shlink` data directory must be chmod 777 — runs as non-root user
|
||||||
|
- `immich-ml` cache goes to `/mnt/docker-ssd/docker/appdata/immich-ml` (separate from immich-server)
|
||||||
|
- `donetick` requires `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml` — env vars alone insufficient
|
||||||
|
|
||||||
|
## Secrets Management
|
||||||
|
- `.env` files are gitignored
|
||||||
|
- `.env.example` files are committed with placeholder values
|
||||||
|
- Database passwords: `openssl rand -hex 24`
|
||||||
|
- JWT secrets: `openssl rand -hex 32`
|
||||||
|
|
||||||
|
## Healthcheck Patterns
|
||||||
|
Always check available tools before writing healthchecks:
|
||||||
|
```bash
|
||||||
|
sudo docker exec <container> sh -c "which curl; which wget" 2>&1
|
||||||
|
```
|
||||||
|
|
||||||
|
- Has curl: `["CMD-SHELL", "curl -sf http://127.0.0.1:<port>/ || exit 1"]`
|
||||||
|
- Has wget only: `["CMD-SHELL", "wget -qO- http://127.0.0.1:<port>/ >/dev/null 2>&1 || exit 1"]`
|
||||||
|
- Has neither: `["CMD-SHELL", "cat /proc/net/tcp6 | grep -q <HEX_PORT> || exit 1"]`
|
||||||
|
- Redirecting endpoint: `["CMD-SHELL", "curl -sfL http://127.0.0.1:<port>/ || exit 1"]`
|
||||||
|
|
||||||
|
Common hex port values: 8080=1F90, 3003=BBB, 3000=BB8, 8181=1FF5, 5055=13BF, 2021=7E5
|
||||||
|
|
||||||
|
After changing a healthcheck, must `down` then `up` (not just restart).
|
||||||
|
|
||||||
|
## Git Workflow
|
||||||
|
```bash
|
||||||
|
cd /mnt/docker-ssd/docker/compose
|
||||||
|
git add .
|
||||||
|
git commit -m "description"
|
||||||
|
git push # pushes to both GitHub and Gitea
|
||||||
|
```
|
||||||
|
|
||||||
|
All `.env` files are gitignored. Only compose files, `.env.example`, and init scripts are committed.
|
||||||
106
docs/planning/HOMELAB_EXPANSION_PLAN.md
Normal file
106
docs/planning/HOMELAB_EXPANSION_PLAN.md
Normal file
@@ -0,0 +1,106 @@
|
|||||||
|
# Homelab Expansion Plan
|
||||||
|
|
||||||
|
## Deployment Order
|
||||||
|
1. **Gotify** — unified notification bus, unblocks all n8n workflows
|
||||||
|
2. **Qdrant** — vector DB, unblocks proper RAG pipeline
|
||||||
|
3. **n8n** — automation backbone, ties everything together
|
||||||
|
4. **Paperless-NGX** — document intelligence
|
||||||
|
5. **Home Assistant** — home automation layer
|
||||||
|
6. **Grafana + Prometheus** — unified observability
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 1 — Notifications (Gotify)
|
||||||
|
**Host:** PlausibleDeniability
|
||||||
|
**Stack:** `/mnt/docker-ssd/docker/compose/automation/`
|
||||||
|
**Storage:** appdata on SSD (SQLite backend)
|
||||||
|
**Network:** pangolin (expose as `gotify.paccoco.com`)
|
||||||
|
|
||||||
|
Single output target for all n8n workflows and homelab alerts.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 2 — Vector Database (Qdrant)
|
||||||
|
**Host:** PlausibleDeniability
|
||||||
|
**Stack:** `/mnt/docker-ssd/docker/compose/ai/`
|
||||||
|
**Storage:** appdata on SSD
|
||||||
|
|
||||||
|
Feed it: homelab markdown docs, Gitea commit history, Plex/Tautulli metadata, Obsidian notes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 3 — Workflow Automation (n8n)
|
||||||
|
**Host:** PlausibleDeniability
|
||||||
|
**Stack:** `/mnt/docker-ssd/docker/compose/automation/`
|
||||||
|
**Storage:** config on tank, Postgres backend
|
||||||
|
**Network:** pangolin (`n8n.paccoco.com`) + ix-databases_shared-databases
|
||||||
|
|
||||||
|
### Planned Workflows
|
||||||
|
- Sonarr/Radarr webhook → Discord + Gotify notification with poster art
|
||||||
|
- Tautulli play start → log to Postgres
|
||||||
|
- Nightly watch history → Ollama → weekly digest via Gotify
|
||||||
|
- Poll Uptime Kuma → cross-reference Netdata → rich alert via Gotify
|
||||||
|
- Gitea commit to truenas-stacks → Gotify notification with Ollama summary
|
||||||
|
- qBittorrent torrent complete → trigger Sonarr/Radarr rescan
|
||||||
|
- KitchenOwl grocery list change → shopping list via Gotify
|
||||||
|
- Donetick task due → Gotify reminder
|
||||||
|
- Long and Low Crafts order → Donetick task + Gotify DM
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 4 — Document Intelligence (Paperless-NGX)
|
||||||
|
**Host:** PlausibleDeniability
|
||||||
|
**Network:** pangolin (`paperless.paccoco.com`) + ix-databases_shared-databases
|
||||||
|
|
||||||
|
OCR and full-text search. Key for Long and Low Crafts invoices, receipts, and manuals.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 5 — Home Automation (Home Assistant)
|
||||||
|
**Host:** PlausibleDeniability or N.O.M.A.D.
|
||||||
|
**Storage:** config on SSD (SQLite, write-heavy)
|
||||||
|
**Network:** host networking recommended for device discovery
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 6 — Observability (Grafana + Prometheus)
|
||||||
|
**Host:** N.O.M.A.D.
|
||||||
|
**Storage:** Prometheus data on hdd-1 or hdd-2
|
||||||
|
|
||||||
|
Metrics to scrape: ZFS pool health, container CPU/RAM, Plex streams, qBittorrent, Tailscale latency, Netdata endpoints.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Multi-Model Routing Strategy
|
||||||
|
|
||||||
|
| Machine | GPU | Model | Use Case |
|
||||||
|
|---|---|---|---|
|
||||||
|
| ROCINANTE | RTX 4090 (24GB) | qwen3:32b / deepseek-r1:32b | Heavy reasoning, long context |
|
||||||
|
| PlausibleDeniability | RTX 2080 Ti (11GB) | qwen2.5:14b | General homelab assistant, RAG |
|
||||||
|
| N.O.M.A.D. | GTX 1080 (8GB) | phi4 / llama3.2:3b | Fast local inference |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Additional Tools
|
||||||
|
- **Continue.dev** — VS Code plugin, local Copilot alternative
|
||||||
|
- **Obsidian + Gitea Sync** — personal notes → RAG pipeline
|
||||||
|
- **Homepage Ollama Widget** — VRAM usage dashboard widget
|
||||||
|
- **kima-hub** — self-hosted Spotify alternative
|
||||||
|
- **dispatcharr** — planned
|
||||||
|
- **bytestash** — planned
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Target Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Triggers (Sonarr, Radarr, Tautulli, Gitea, Home Assistant, schedules, email)
|
||||||
|
↓
|
||||||
|
n8n (routing, logic, orchestration)
|
||||||
|
↓
|
||||||
|
Ollama (intelligence — model selected by complexity)
|
||||||
|
↓ ↑
|
||||||
|
Qdrant (semantic context retrieval)
|
||||||
|
↓
|
||||||
|
Outputs (Gotify → phone, Donetick tasks, Postgres logging, Discord)
|
||||||
|
```
|
||||||
43
docs/planning/TODO.md
Normal file
43
docs/planning/TODO.md
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
# Homelab TODO
|
||||||
|
|
||||||
|
## Security (do ASAP)
|
||||||
|
- [ ] Regenerate Wings token on N.O.M.A.D. (was exposed in chat)
|
||||||
|
- [ ] Regenerate N.O.M.A.D. Newt secret (was exposed in chat)
|
||||||
|
|
||||||
|
## Infrastructure
|
||||||
|
- [ ] Deploy Pi-hole ×2 (primary on PD + backup on Serenity or Pi4)
|
||||||
|
- [ ] Set up off-site backup for vital data (appdata, databases, config repo, photos)
|
||||||
|
- [ ] Remove dead Unraid-Cloudflared-Tunnel container from Serenity
|
||||||
|
- [ ] Serenity malcolm pool capacity planning (heavily utilized)
|
||||||
|
- [ ] Add Pangolin resource for `ai.paccoco.com` → openwebui (port 8282)
|
||||||
|
- [ ] Add Cloudflare DNS A record for `ai.paccoco.com`
|
||||||
|
- [ ] Install fresh editor on PD
|
||||||
|
- [ ] Set up private Gitea repo for .env file backups
|
||||||
|
|
||||||
|
## AI Stack
|
||||||
|
- [ ] Pull `qwen2.5:14b` on PD ollama: `sudo docker exec -it ollama ollama pull qwen2.5:14b`
|
||||||
|
- [ ] Create OpenWebUI account and set system prompt from HOMELAB_AI_CONTEXT.md
|
||||||
|
- [ ] Disable signups in OpenWebUI after creating account
|
||||||
|
- [ ] Connect desktop Ollama to OpenWebUI (Admin → Settings → Connections)
|
||||||
|
- [ ] Set `OLLAMA_HOST=0.0.0.0` on desktop and restart Ollama service
|
||||||
|
|
||||||
|
## Planned Deployments (PD)
|
||||||
|
- [ ] dispatcharr
|
||||||
|
- [ ] gotify
|
||||||
|
- [ ] bytestash
|
||||||
|
- [ ] kima-hub
|
||||||
|
- [ ] n8n
|
||||||
|
- [ ] qdrant
|
||||||
|
- [ ] paperless-ngx
|
||||||
|
|
||||||
|
## Healthchecks
|
||||||
|
- [ ] Add healthcheck to ix-plex-plex-1
|
||||||
|
- [ ] Add healthcheck to ix-tailscale-tailscale-1
|
||||||
|
- [ ] Add healthcheck to ix-newt-newt-1
|
||||||
|
- [ ] Add healthcheck to ollama (no curl/wget — needs /proc/net/tcp6 or custom solution)
|
||||||
|
- [ ] Document and add healthchecks for meshtastic stack
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
- [ ] Document meshtastic stack fully (see docs/reference/MESHTASTIC.md)
|
||||||
|
- [ ] Fill in port/URL columns in SERVICES_DIRECTORY.md
|
||||||
|
- [ ] Document N.O.M.A.D. containers (start_nomad.sh services)
|
||||||
56
docs/reference/HOMELAB_AI_CONTEXT.md
Normal file
56
docs/reference/HOMELAB_AI_CONTEXT.md
Normal file
@@ -0,0 +1,56 @@
|
|||||||
|
# Homelab AI Context Document
|
||||||
|
|
||||||
|
This document gives an AI assistant full context of this homelab for infrastructure questions, deployments, debugging, and planning.
|
||||||
|
|
||||||
|
See the individual docs in this repo for detailed information:
|
||||||
|
|
||||||
|
- [Architecture Overview](../architecture/ARCHITECTURE_OVERVIEW.md)
|
||||||
|
- [Networking Model](../architecture/NETWORKING_MODEL.md)
|
||||||
|
- [Hardware Inventory](../architecture/HARDWARE_INVENTORY.md)
|
||||||
|
- [Services Directory](../architecture/SERVICES_DIRECTORY.md)
|
||||||
|
- [Stack Standards](../operations/STACK_STANDARDS.md)
|
||||||
|
- [Deployment Checklist](../operations/DEPLOYMENT_CHECKLIST.md)
|
||||||
|
- [Storage Notes](../operations/DOCKERSTACK_STORAGE_NOTES.md)
|
||||||
|
- [Backup Policy](../operations/BACKUP_POLICY.md)
|
||||||
|
- [Secrets Management](../operations/SECRETS_MANAGEMENT.md)
|
||||||
|
- [Troubleshooting](../troubleshooting/TROUBLESHOOTING.md)
|
||||||
|
- [Known Quirks](../troubleshooting/KNOWN_QUIRKS.md)
|
||||||
|
- [Serenity](../servers/SERENITY.md)
|
||||||
|
- [PlausibleDeniability](../servers/PLAUSIBLEDENABILITY.md)
|
||||||
|
- [N.O.M.A.D.](../servers/NOMAD.md)
|
||||||
|
- [Meshtastic Stack](MESHTASTIC.md)
|
||||||
|
- [Expansion Plan](../planning/HOMELAB_EXPANSION_PLAN.md)
|
||||||
|
- [TODO](../planning/TODO.md)
|
||||||
|
|
||||||
|
## Quick Reference
|
||||||
|
|
||||||
|
### Communication Preferences
|
||||||
|
- One command block at a time
|
||||||
|
- Prefer scripted/automated solutions over interactive multi-step processes
|
||||||
|
- Uses `fresh` editor (not nano) on Linux
|
||||||
|
- Manages remote servers via SSH from Windows 11
|
||||||
|
- Prefers MariaDB over PostgreSQL for standalone services
|
||||||
|
|
||||||
|
### Three Servers
|
||||||
|
| Server | Role | OS | IP |
|
||||||
|
|--------|------|----|----|
|
||||||
|
| Serenity | NAS / Media ingestion | Unraid 7.2.4 | 10.5.1.5 |
|
||||||
|
| PlausibleDeniability | Primary Docker host | TrueNAS Scale 25.10.2.1 | LAN DHCP |
|
||||||
|
| N.O.M.A.D. | Offline knowledge + game servers | Ubuntu 25.10 | 10.5.1.16 |
|
||||||
|
|
||||||
|
### Deployment Pattern (PD)
|
||||||
|
```bash
|
||||||
|
cd /mnt/docker-ssd/docker/compose/<stack>
|
||||||
|
docker compose --env-file .env config # validate first
|
||||||
|
docker compose --env-file .env up -d # deploy
|
||||||
|
docker logs <container> --tail 20 # verify
|
||||||
|
```
|
||||||
|
|
||||||
|
### Network names
|
||||||
|
- Shared DB network: `ix-databases_shared-databases`
|
||||||
|
- Reverse proxy network: `pangolin`
|
||||||
|
|
||||||
|
### Storage decision
|
||||||
|
- Write-heavy / DB / SQLite / GPU → `/mnt/docker-ssd/docker/appdata/<service>`
|
||||||
|
- General config / non-fussy → `/mnt/tank/docker/appdata/<service>`
|
||||||
|
- Media only → `/mnt/unraid/data/media/`
|
||||||
70
docs/reference/MESHTASTIC.md
Normal file
70
docs/reference/MESHTASTIC.md
Normal file
@@ -0,0 +1,70 @@
|
|||||||
|
# Meshtastic Stack
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
Self-hosted Meshtastic monitoring and management stack running on PlausibleDeniability.
|
||||||
|
|
||||||
|
## Stack Location
|
||||||
|
`/mnt/docker-ssd/docker/compose/meshtastic/`
|
||||||
|
|
||||||
|
## Containers
|
||||||
|
|
||||||
|
| Container | Image | Port | Purpose |
|
||||||
|
|-----------|-------|------|---------|
|
||||||
|
| meshmonitor | ghcr.io/yeraze/meshmonitor:4.2.0 | 8081→3001 | Main UI + backend |
|
||||||
|
| meshmonitor (virtual node) | — | 4404 | TCP virtual node server |
|
||||||
|
| meshmonitor-tileserver | maptiler/tileserver-gl-light | 8082→8080 | Offline map tiles |
|
||||||
|
| meshmonitor-mqtt-proxy | ghcr.io/ln4cy/mqtt-proxy:master | — | MQTT→TCP bridge |
|
||||||
|
| meshmonitor-upgrader | docker:latest | — | Auto-upgrade watchdog |
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
```
|
||||||
|
Meshtastic Node (10.5.1.172)
|
||||||
|
↓ MQTT
|
||||||
|
meshmonitor-mqtt-proxy
|
||||||
|
↓ TCP (port 4404)
|
||||||
|
meshmonitor virtual node server
|
||||||
|
↓
|
||||||
|
meshmonitor backend
|
||||||
|
↓
|
||||||
|
PostgreSQL (shared-postgres, db: meshmonitor)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Node Connection
|
||||||
|
- Node IP: `10.5.1.172`
|
||||||
|
- Connection type: MQTT via mqtt-proxy (not direct TCP)
|
||||||
|
- mqtt-proxy connects to meshmonitor's virtual node server on port 4404
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
- Appdata: `/mnt/tank/docker/appdata/meshmonitor/data` → `/data`
|
||||||
|
- Scripts: `./scripts` → `/data/scripts` (must be pre-created on host)
|
||||||
|
- Tiles: `./tiles` (gitignored — large .mbtiles files)
|
||||||
|
|
||||||
|
## Auto-Upgrade
|
||||||
|
`AUTO_UPGRADE_ENABLED=true` — meshmonitor checks for new images and triggers self-upgrade.
|
||||||
|
|
||||||
|
**Important:** The upgrader mounts the compose directory as `/compose` inside the container. All relative bind mount paths (e.g. `./scripts`) resolve to `/compose/scripts` on the host when the upgrader re-runs compose. Since TrueNAS root filesystem is read-only, these directories must be pre-created:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mkdir -p /mnt/docker-ssd/docker/compose/meshtastic/scripts
|
||||||
|
```
|
||||||
|
|
||||||
|
## Networks
|
||||||
|
- `default` (internal stack network)
|
||||||
|
- `ix-databases_shared-databases` (meshmonitor only)
|
||||||
|
- `pangolin` (meshmonitor + tileserver)
|
||||||
|
|
||||||
|
## Public URL
|
||||||
|
`https://meshmonitor.paccoco.com`
|
||||||
|
|
||||||
|
## Known Warnings
|
||||||
|
- Node `!dd972536` has a low-entropy public key — this is a device configuration issue, not a meshmonitor bug
|
||||||
|
- Frequent connect/disconnect cycles in logs are normal when the MQTT proxy reconnects
|
||||||
|
|
||||||
|
## Tiles
|
||||||
|
Map tiles stored in `./tiles/`. Currently includes `zurich_switzerland.mbtiles` (34MB).
|
||||||
|
Tiles directory is gitignored. To add new tiles, download `.mbtiles` files and place in the `tiles/` directory.
|
||||||
|
|
||||||
|
## TODO
|
||||||
|
- [ ] Document all running meshtastic-map container config
|
||||||
|
- [ ] Document meshtastic-map-mosquitto and meshtastic-mqtt containers
|
||||||
|
- [ ] Add healthcheck to meshmonitor container
|
||||||
73
docs/servers/NOMAD.md
Normal file
73
docs/servers/NOMAD.md
Normal file
@@ -0,0 +1,73 @@
|
|||||||
|
# N.O.M.A.D. Server
|
||||||
|
|
||||||
|
Offline survival/knowledge server and game server host.
|
||||||
|
|
||||||
|
## Specs
|
||||||
|
- **OS:** Ubuntu 25.10 (bare metal)
|
||||||
|
- **IP:** 10.5.1.16
|
||||||
|
- **CPU:** Intel Core i7-4770K @ 3.50GHz (4c/8t)
|
||||||
|
- **RAM:** 32GB DDR3 1600MHz
|
||||||
|
- **GPU:** NVIDIA GeForce GTX 1080 (8GB VRAM)
|
||||||
|
- **NIC:** Intel I218-V (1GbE) + Aquantia AQC113 (10GbE)
|
||||||
|
- **Motherboard:** ASRock H97M Pro4
|
||||||
|
- **UPS:** None
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
| Device | Size | Mount | Purpose |
|
||||||
|
|--------|------|-------|---------|
|
||||||
|
| Samsung SSD 840 (sdb) | 232.9GB | `/` (LVM 100GB) | OS boot |
|
||||||
|
| WD Red HDD (sda) | 3.6TB | `/mnt/hdd-1` | Primary data |
|
||||||
|
| WD Red HDD (sdc) | 3.6TB | `/mnt/hdd-2` | Docker data-root |
|
||||||
|
| Seagate SSD (sdd) | 223.6GB | `/mnt/ssd-1` | Fast scratch |
|
||||||
|
|
||||||
|
## Project N.O.M.A.D.
|
||||||
|
- 10 containers running via `start_nomad.sh` / `stop_nomad.sh`
|
||||||
|
- `/opt/project-nomad` → symlinked from `/mnt/hdd-1/project-nomad`
|
||||||
|
- Docker data-root: `/mnt/hdd-2/docker`
|
||||||
|
- Based on Crosstalk Solutions offline survival/knowledge server
|
||||||
|
|
||||||
|
## Pelican Panel (Game Server Management)
|
||||||
|
- **URL:** `https://panel.paccoco.com`
|
||||||
|
- **Stack:** PHP 8.4 + nginx + MariaDB (native, not Docker)
|
||||||
|
- **Panel files:** `/var/www/pelican`
|
||||||
|
- **DB:** MariaDB, database `pelican`, user `pelican`@`127.0.0.1`
|
||||||
|
- **APP_URL:** `https://panel.paccoco.com`
|
||||||
|
- **TRUSTED_PROXIES:** `*`
|
||||||
|
- **Cron:** `* * * * * php /var/www/pelican/artisan schedule:run` (www-data)
|
||||||
|
- **Admin user:** `fizzlepoof`
|
||||||
|
- **Reverse proxy:** Pangolin via Newt container at `/opt/newt`
|
||||||
|
|
||||||
|
## Wings (Game Server Daemon)
|
||||||
|
- **Binary:** `/usr/local/bin/wings`
|
||||||
|
- **Config:** `/etc/pelican/config.yml`
|
||||||
|
- **Systemd:** `wings.service`
|
||||||
|
- **ExecStart:** `/usr/local/bin/wings --ignore-certificate-errors`
|
||||||
|
- **API port:** 8443
|
||||||
|
- **SFTP port:** 2022
|
||||||
|
- **Node domain:** `node1.paccoco.com` (Pangolin, no auth, connection port 443)
|
||||||
|
- **Game server volumes:** `/var/lib/pelican/volumes`
|
||||||
|
- **Docker network:** `pelican_nw` (172.20.0.0/16)
|
||||||
|
|
||||||
|
## Active Game Servers
|
||||||
|
- **Minecraft:** Purpur, Java 21, port 25565 TCP/UDP (direct host port forward)
|
||||||
|
|
||||||
|
## Networking Notes
|
||||||
|
- `panel.paccoco.com` → `127.0.0.1` in `/etc/hosts` for NAT loopback
|
||||||
|
- nginx on 80 and 443 (self-signed cert) for local Wings → Panel
|
||||||
|
- Wings uses `--ignore-certificate-errors` for self-signed cert
|
||||||
|
- Pangolin resource `node1.paccoco.com` has auth disabled
|
||||||
|
|
||||||
|
## Known Quirks
|
||||||
|
- NVIDIA container toolkit repo has invalid GPG key — remove `/etc/apt/sources.list.d/*nvidia*` if apt errors
|
||||||
|
- Port 8080 occupied by `nomad_admin` container — Wings uses 8443
|
||||||
|
- Interactive artisan commands break over SSH — always use inline flags
|
||||||
|
- Wings CORS derived from `remote:` URL — keep remote as `https://` and use `--ignore-certificate-errors`
|
||||||
|
|
||||||
|
## Newt Tunnel
|
||||||
|
- Container: `newt` at `/opt/newt/docker-compose.yaml`
|
||||||
|
- `network_mode: host`
|
||||||
|
- Connects to Pangolin VPS at `https://paccoco.com`
|
||||||
|
|
||||||
|
## Pending
|
||||||
|
- Regenerate Wings token (was exposed in chat)
|
||||||
|
- Regenerate Newt secret (was exposed in chat)
|
||||||
60
docs/servers/PLAUSIBLEDENABILITY.md
Normal file
60
docs/servers/PLAUSIBLEDENABILITY.md
Normal file
@@ -0,0 +1,60 @@
|
|||||||
|
# PlausibleDeniability
|
||||||
|
|
||||||
|
Primary Docker host running all production workloads.
|
||||||
|
|
||||||
|
## Specs
|
||||||
|
- **OS:** TrueNAS Scale 25.10.2.1
|
||||||
|
- **CPU:** Ryzen 9 3950X
|
||||||
|
- **RAM:** 32GB
|
||||||
|
- **GPU:** Nvidia RTX 2080 Ti (11GB VRAM)
|
||||||
|
- **UPS:** None
|
||||||
|
|
||||||
|
## Storage Tiers
|
||||||
|
|
||||||
|
| Tier | Mount | Use for |
|
||||||
|
|------|-------|---------|
|
||||||
|
| Local SSD (Crucial MX500 500GB) | `/mnt/docker-ssd` | Docker root, databases, write-heavy appdata, GPU/model caches |
|
||||||
|
| TrueNAS pool (tank, 2-drive ZFS mirror ~3.5TB) | `/mnt/tank` | General appdata, configs, backups, downloads |
|
||||||
|
| Unraid NFS | `/mnt/unraid` | Media libraries, Immich media |
|
||||||
|
|
||||||
|
## Docker Root
|
||||||
|
`/mnt/docker-ssd/docker/docker-data`
|
||||||
|
|
||||||
|
## Compose Repo
|
||||||
|
`/mnt/docker-ssd/docker/compose`
|
||||||
|
|
||||||
|
## Key Docker Networks
|
||||||
|
- `ix-databases_shared-databases` — shared DB access network (created by databases stack)
|
||||||
|
- `pangolin` — reverse proxy exposure network (created by newt)
|
||||||
|
|
||||||
|
## GPU Usage
|
||||||
|
RTX 2080 Ti used by:
|
||||||
|
- Plex (hardware transcoding)
|
||||||
|
- immich-ml (CUDA face recognition)
|
||||||
|
- ollama (LLM inference)
|
||||||
|
|
||||||
|
Recommended models for 11GB VRAM:
|
||||||
|
| Model | VRAM | Best for |
|
||||||
|
|-------|------|---------|
|
||||||
|
| `qwen2.5:14b` | ~8GB | General use, coding, reasoning |
|
||||||
|
| `qwen2.5-coder:7b` | ~5GB | Code tasks |
|
||||||
|
| `deepseek-r1:8b` | ~5GB | Math, logic |
|
||||||
|
| `phi4` | ~8GB | Analytical tasks |
|
||||||
|
|
||||||
|
## Remote Access
|
||||||
|
- Pangolin VPS + Newt for external services
|
||||||
|
- SSH from Windows 11
|
||||||
|
|
||||||
|
## NFS Exports to Other Servers
|
||||||
|
- `data` share → PD at `/mnt/unraid/data`
|
||||||
|
- `immich` share → PD at `/mnt/unraid/immich`
|
||||||
|
|
||||||
|
## Known Issues
|
||||||
|
- immich-ml intermittently unhealthy — investigate GPU/CUDA health
|
||||||
|
- docker-ssd has no redundancy — single point of failure for all appdata
|
||||||
|
- No off-site backup
|
||||||
|
|
||||||
|
## Pending
|
||||||
|
- Deploy: dispatcharr, gotify, bytestash, kima-hub, n8n, qdrant, paperless-ngx
|
||||||
|
- Add Pangolin resource + DNS for `ai.paccoco.com`
|
||||||
|
- Install fresh editor
|
||||||
55
docs/servers/SERENITY.md
Normal file
55
docs/servers/SERENITY.md
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
# Serenity
|
||||||
|
|
||||||
|
Unraid NAS and media ingestion server.
|
||||||
|
|
||||||
|
## Specs
|
||||||
|
- **OS:** Unraid 7.2.4
|
||||||
|
- **IP:** 10.5.1.5 | Tailscale: 100.94.87.79
|
||||||
|
- **CPU:** Xeon X5690
|
||||||
|
- **RAM:** 102GB
|
||||||
|
- **GPU:** None (Matrox IPMI only)
|
||||||
|
- **UPS:** None
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
- `malcolm` pool: RAIDZ1 of 6×10TB HDDs (~40TB usable) — heavily utilized
|
||||||
|
- Cache pool: single Kingston SNV2S 1TB NVMe (ZFS)
|
||||||
|
|
||||||
|
## Docker Networks
|
||||||
|
`arrs`, `hotio`, `br0` (macvlan), `bridge`
|
||||||
|
|
||||||
|
## Active Containers
|
||||||
|
- **Media:** sonarr, sonarr-anime, radarr, lidarr, readarr, readarr-epub, prowlarr, bazarr, unpackerr
|
||||||
|
- **Download:** qBittorrent, GluetunVPN, autobrr, qbit_manage
|
||||||
|
- **Notifications:** Notifiarr
|
||||||
|
- **Access:** Wizarr, Newt, dockersocket, Hawser
|
||||||
|
- **Databases:** postgresql15, MariaDB-Official
|
||||||
|
- **Other:** romm, GameVault, shelfmark (port 8084), netdata
|
||||||
|
- **Dead/remove:** Unraid-Cloudflared-Tunnel
|
||||||
|
|
||||||
|
## Backup Strategy
|
||||||
|
- Sanoid snapshots cache/appdata: 7 daily, 4 weekly, 6 monthly
|
||||||
|
- Syncoid replicates cache/appdata → malcolm/zfs_backups_array (strict-mirror)
|
||||||
|
- No off-site backup currently
|
||||||
|
|
||||||
|
## Custom Scripts
|
||||||
|
- **qBit Mover script:** pauses torrents 0–4 days old, then runs Unraid mover
|
||||||
|
- **MAM-init script:** updates MyAnonamouse IP inside qbit container
|
||||||
|
- **Mover Status script:** sends Discord webhook notifications
|
||||||
|
|
||||||
|
## Key Shares
|
||||||
|
| Share | Size | Purpose |
|
||||||
|
|-------|------|---------|
|
||||||
|
| data | 35TB | Media library (NFS-mounted by PD at `/mnt/unraid/data`) |
|
||||||
|
| immich | 62GB | Immich photo storage (NFS-mounted by PD at `/mnt/unraid/immich`) |
|
||||||
|
| nextcloud | 7.4GB | Nextcloud data |
|
||||||
|
| Tesla/dashcam | 1.6TB | Dashcam footage |
|
||||||
|
| fizzlebang | 17GB | Personal files |
|
||||||
|
| isos | 22GB | ISO images |
|
||||||
|
| Syncthing | 4.6GB | Syncthing data |
|
||||||
|
| manndra | — | Partner's storage |
|
||||||
|
| Long and Low Crafts | — | Crafts business files |
|
||||||
|
| zfs_backups_array | — | ZFS backup target for cache/appdata |
|
||||||
|
|
||||||
|
## Pending
|
||||||
|
- Remove dead Unraid-Cloudflared-Tunnel container
|
||||||
|
- malcolm pool capacity planning (heavily utilized)
|
||||||
69
docs/troubleshooting/KNOWN_QUIRKS.md
Normal file
69
docs/troubleshooting/KNOWN_QUIRKS.md
Normal file
@@ -0,0 +1,69 @@
|
|||||||
|
# Known Quirks
|
||||||
|
|
||||||
|
Per-service gotchas that aren't bugs but will bite you if you forget them.
|
||||||
|
|
||||||
|
## PlausibleDeniability
|
||||||
|
|
||||||
|
### immich-ml
|
||||||
|
- Healthcheck has no curl/wget — uses `/proc/net/tcp6` pattern
|
||||||
|
- Cache goes to `/mnt/docker-ssd/docker/appdata/immich-ml` (separate from immich-server's appdata)
|
||||||
|
|
||||||
|
### rackpeek
|
||||||
|
- No curl/wget in image — healthcheck uses `/proc/net/tcp6`
|
||||||
|
|
||||||
|
### donetick
|
||||||
|
- Env vars alone are insufficient for DB type selection
|
||||||
|
- Requires `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml` with full DB config
|
||||||
|
- Uses viper config loader — `DT_ENV` controls config file path
|
||||||
|
|
||||||
|
### shlink
|
||||||
|
- Data directory must be `chmod 777` — runs as non-root user that doesn't match host default ownership
|
||||||
|
|
||||||
|
### openwebui
|
||||||
|
- DB: user=`openwebui`, db=`openwebui` on shared-postgres
|
||||||
|
- If restart-looping with auth errors: container wasn't on `ix-databases_shared-databases` at creation time — must `down && up`
|
||||||
|
|
||||||
|
### plex
|
||||||
|
- Port `5353/udp` conflicts with system avahi/mDNS — remove from port mappings
|
||||||
|
|
||||||
|
### scriberr (shelved)
|
||||||
|
- SQLite incompatible with ZFS nfsv4 ACLs (SQLITE_CANTOPEN error 14)
|
||||||
|
- Revisit when Postgres support available in CUDA image
|
||||||
|
|
||||||
|
### meshmonitor (auto-upgrade)
|
||||||
|
- `AUTO_UPGRADE_ENABLED=true` causes crashes on TrueNAS if bind mount source directories don't exist
|
||||||
|
- The upgrader container mounts compose dir as `/compose` — relative paths resolve differently
|
||||||
|
- Fix: pre-create all bind mount source directories before deploying
|
||||||
|
|
||||||
|
### Any container using `down && up` requirement
|
||||||
|
- Network attachments happen at container creation, not restart
|
||||||
|
- `docker restart` or `docker compose restart` will NOT fix missing network attachments
|
||||||
|
- Must use `docker compose --env-file .env down <service> && docker compose --env-file .env up -d <service>`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Serenity
|
||||||
|
|
||||||
|
### qBit Mover Script
|
||||||
|
- Pauses torrents 0–4 days old before running mover to prevent partial file moves
|
||||||
|
- MAM-init script updates MyAnonamouse IP inside qbit container on startup
|
||||||
|
|
||||||
|
### Unraid-Cloudflared-Tunnel
|
||||||
|
- Dead container, should be removed
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## N.O.M.A.D.
|
||||||
|
|
||||||
|
### NVIDIA Container Toolkit
|
||||||
|
- Repo has invalid GPG key — remove `/etc/apt/sources.list.d/*nvidia*` if apt errors
|
||||||
|
|
||||||
|
### Pelican / Wings
|
||||||
|
- Interactive artisan commands break over SSH (stty issue) — always use inline flags:
|
||||||
|
`--email=x --username=y --password=z --admin=1`
|
||||||
|
- Wings CORS derived from `remote:` URL, not `allowed_origins` — keep remote as `https://`
|
||||||
|
- `panel.paccoco.com` in `/etc/hosts` → `127.0.0.1` for NAT loopback
|
||||||
|
- Wings uses `--ignore-certificate-errors` for local self-signed cert
|
||||||
|
|
||||||
|
### Port 8080
|
||||||
|
- Occupied by `nomad_admin` container — Wings uses 8443 instead
|
||||||
129
docs/troubleshooting/TROUBLESHOOTING.md
Normal file
129
docs/troubleshooting/TROUBLESHOOTING.md
Normal file
@@ -0,0 +1,129 @@
|
|||||||
|
# Troubleshooting Guide
|
||||||
|
|
||||||
|
## Permission Denied / Readonly Database / Not Writable
|
||||||
|
|
||||||
|
Common causes:
|
||||||
|
- App placed on NFS (tank) when it needs local SSD
|
||||||
|
- NFS ownership mismatch between host user and container user
|
||||||
|
|
||||||
|
Checks:
|
||||||
|
1. `docker logs <container> --tail 30`
|
||||||
|
2. `ls -la <host-appdata-path>` — check ownership and permissions
|
||||||
|
3. `docker run --rm --entrypoint sh <image> -lc 'id'` — check container UID/GID
|
||||||
|
|
||||||
|
Fix options:
|
||||||
|
- Move appdata to `/mnt/docker-ssd/docker/appdata/<service>` if write-heavy
|
||||||
|
- `sudo chmod -R 777 <appdata-path>` for apps with fixed internal user (e.g. shlink)
|
||||||
|
- `sudo chown -R <uid>:<gid> <appdata-path>` if UID/GID is known
|
||||||
|
|
||||||
|
## SQLite Fails on ZFS with nfsv4 ACLs (SQLITE_CANTOPEN error 14)
|
||||||
|
|
||||||
|
Symptom: `unable to open database file: out of memory (14)`
|
||||||
|
|
||||||
|
Cause: SQLite cannot acquire POSIX file locks on ZFS datasets configured with `acltype=nfsv4`.
|
||||||
|
|
||||||
|
Diagnosis:
|
||||||
|
```bash
|
||||||
|
sudo zfs get acltype,aclmode,xattr <dataset>
|
||||||
|
```
|
||||||
|
|
||||||
|
Fix options:
|
||||||
|
1. Change acltype: `sudo zfs set acltype=posix <dataset>`
|
||||||
|
2. Move the SQLite database to local SSD
|
||||||
|
3. Use an app version that supports Postgres instead of SQLite
|
||||||
|
|
||||||
|
Note: scriberr CUDA image is shelved for this reason.
|
||||||
|
|
||||||
|
## Container Can't Resolve Shared Database Hostname
|
||||||
|
|
||||||
|
Symptom: `lookup shared-postgres on 127.0.0.11:53: no such host`
|
||||||
|
|
||||||
|
Fix:
|
||||||
|
```bash
|
||||||
|
docker compose --env-file .env down <service>
|
||||||
|
docker compose --env-file .env up -d <service>
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify network attachment:
|
||||||
|
```bash
|
||||||
|
docker inspect <container> --format '{{json .NetworkSettings.Networks}}' | tr ',' '\n' | grep -o '"[^"]*"'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Postgres Auth Fails Over TCP but Works via Socket
|
||||||
|
|
||||||
|
Diagnosis:
|
||||||
|
```bash
|
||||||
|
sudo docker exec shared-postgres psql -h 127.0.0.1 -U <user> -d <db> -W -c "\conninfo"
|
||||||
|
```
|
||||||
|
|
||||||
|
Fix:
|
||||||
|
- Verify pg_hba.conf has `host all all all scram-sha-256`
|
||||||
|
- Check DATABASE_URL password matches: `ALTER USER <user> WITH PASSWORD '<pass>';`
|
||||||
|
- Avoid special characters in passwords in DATABASE_URLs
|
||||||
|
|
||||||
|
## Healthcheck Failing Despite Container Being Healthy
|
||||||
|
|
||||||
|
Diagnosis:
|
||||||
|
```bash
|
||||||
|
sudo docker exec <container> sh -c "which curl; which wget" 2>&1
|
||||||
|
sudo docker inspect <container> --format '{{json .State.Health.Log}}' | python3 -m json.tool | tail -20
|
||||||
|
```
|
||||||
|
|
||||||
|
After fixing the compose file, must do `down && up` (not just restart).
|
||||||
|
|
||||||
|
## App Ignores Environment Variables for DB Type
|
||||||
|
|
||||||
|
Some apps (e.g. donetick) require a YAML config file for certain settings even when env vars are set.
|
||||||
|
|
||||||
|
For donetick, create `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml`:
|
||||||
|
```yaml
|
||||||
|
database:
|
||||||
|
type: postgres
|
||||||
|
host: shared-postgres
|
||||||
|
port: 5432
|
||||||
|
user: donetick
|
||||||
|
password: <password>
|
||||||
|
name: donetick
|
||||||
|
migration: true
|
||||||
|
jwt:
|
||||||
|
secret: <32-byte-hex>
|
||||||
|
session_time: 168h
|
||||||
|
max_refresh: 168h
|
||||||
|
server:
|
||||||
|
port: 2021
|
||||||
|
serve_frontend: true
|
||||||
|
```
|
||||||
|
|
||||||
|
## JWT Invalid / Login Fails After Restart
|
||||||
|
|
||||||
|
Cause: JWT secret changed. Fix: open app in incognito or clear cookies/localStorage.
|
||||||
|
|
||||||
|
## Port Already In Use
|
||||||
|
|
||||||
|
Common offenders:
|
||||||
|
- `5353/udp` — held by system avahi/mDNS. Remove from Plex ports.
|
||||||
|
- `3306`, `5432`, `6379` — only one DB stack can bind these.
|
||||||
|
|
||||||
|
## Auto-Upgrade Crashes on TrueNAS (exit code 128)
|
||||||
|
|
||||||
|
Symptom: `error while creating mount source path '/compose/scripts': mkdir /compose: read-only file system`
|
||||||
|
|
||||||
|
Cause: Auto-upgrade container runs docker compose from inside a container where the project dir is mounted as `/compose`. Relative bind mount paths like `./scripts` resolve to `/compose/scripts` on the host. TrueNAS root filesystem is read-only so Docker can't auto-create the directory.
|
||||||
|
|
||||||
|
Fix: Pre-create the directory manually on the host before starting the stack:
|
||||||
|
```bash
|
||||||
|
mkdir -p /mnt/docker-ssd/docker/compose/<stack>/scripts
|
||||||
|
```
|
||||||
|
|
||||||
|
## Databases Stack Network Name Wrong
|
||||||
|
|
||||||
|
Must have `name: ix-databases` at top of databases compose. Verify:
|
||||||
|
```bash
|
||||||
|
docker network ls | grep databases
|
||||||
|
# Should show: ix-databases_shared-databases
|
||||||
|
```
|
||||||
|
|
||||||
|
## Checking Container Runtime UID
|
||||||
|
```bash
|
||||||
|
docker run --rm --entrypoint sh <image> -lc 'id'
|
||||||
|
```
|
||||||
Reference in New Issue
Block a user