docs: add full homelab documentation structure

2026-05-05 14:43:09 -05:00
parent ee8a412a69
commit 370998b8d5
20 changed files with 1405 additions and 24 deletions
--- a/README.md
+++ b/README.md
@@ -1,35 +1,60 @@
 # truenas-stacks

-Docker Compose stacks for PlausibleDeniability (TrueNAS Scale).
+Docker Compose stacks for PlausibleDeniability (TrueNAS Scale), plus documentation for the full homelab.

-## Stacks
+## Servers

- databases/ - shared-postgres, shared-mariadb, shared-redis - deploy FIRST
- infrastructure/ - newt, homepage, uptime-kuma, dockhand, netdata, tailscale - deploy SECOND
- media/ - plex, tautulli, audiobookshelf, seerr, calibre-web
- photos/ - immich server + machine learning
- home/ - kitchenowl, donetick, dakboard bridge, shlink, qui
- dev/ - gitea, rackpeek
- ai/ - ollama, openwebui, scriberr
+| Server | Role | OS | IP |
+|--------|------|----|----|
+| Serenity | NAS / Media ingestion | Unraid 7.2.4 | 10.5.1.5 |
+| PlausibleDeniability | Primary Docker host | TrueNAS Scale 25.10.2.1 | LAN DHCP |
+| N.O.M.A.D. | Offline knowledge + game servers | Ubuntu 25.10 | 10.5.1.16 |

-## Deployment order
+## Repo Layout

-1. databases
-2. infrastructure
-3. All other stacks in any order
+```
+databases/       # shared-postgres, shared-mariadb, shared-redis — deploy FIRST
+infrastructure/  # newt, homepage, uptime-kuma, dockhand, netdata, tailscale — deploy SECOND
+media/           # plex, tautulli, audiobookshelf, seerr, calibre-web-automated
+photos/          # immich server + machine learning
+home/            # kitchenowl, donetick, shlink, qui, dakboard bridges
+dev/             # gitea, rackpeek
+ai/              # ollama, openwebui
+meshtastic/      # meshmonitor, mqtt-proxy, tileserver, meshtastic-map
+automation/      # planned: gotify, n8n
+search/          # planned: qdrant
+docs/            # all homelab documentation
+```

-## Setup on a new machine
+## Quick Start

-    git clone git@github.com:Paccoco/truenas-stacks.git /mnt/docker-ssd/docker/compose
-    cp databases/.env.example databases/.env && nano databases/.env
-    # repeat for each stack, then import each truenas-app.yaml via TrueNAS Apps UI
+1. Deploy `databases/` first
+2. Deploy `infrastructure/` second (creates pangolin network via newt)
+3. Deploy remaining stacks in any order

-## Networks
+```bash
+cd /mnt/docker-ssd/docker/compose/<stack>
+docker compose --env-file .env config   # validate
+docker compose --env-file .env up -d    # deploy
+```

- ix-databases_shared-databases: created by databases stack, used by media/photos/home/dev/ai
- pangolin: created by newt at runtime, used by any service exposed via reverse proxy
+## Documentation

-## Secrets
-
-.env files are gitignored. .env.example files are committed with placeholder values.
-Real .env files live only on TrueNAS at /mnt/docker-ssd/docker/compose/<stack>/.env
+- [Architecture Overview](docs/architecture/ARCHITECTURE_OVERVIEW.md)
+- [Networking Model](docs/architecture/NETWORKING_MODEL.md)
+- [Hardware Inventory](docs/architecture/HARDWARE_INVENTORY.md)
+- [Services Directory](docs/architecture/SERVICES_DIRECTORY.md)
+- [Stack Standards](docs/operations/STACK_STANDARDS.md)
+- [Deployment Checklist](docs/operations/DEPLOYMENT_CHECKLIST.md)
+- [Storage Notes](docs/operations/DOCKERSTACK_STORAGE_NOTES.md)
+- [Backup Policy](docs/operations/BACKUP_POLICY.md)
+- [Secrets Management](docs/operations/SECRETS_MANAGEMENT.md)
+- [Troubleshooting](docs/troubleshooting/TROUBLESHOOTING.md)
+- [Known Quirks](docs/troubleshooting/KNOWN_QUIRKS.md)
+- [Expansion Plan](docs/planning/HOMELAB_EXPANSION_PLAN.md)
+- [TODO](docs/planning/TODO.md)
+- [Serenity](docs/servers/SERENITY.md)
+- [PlausibleDeniability](docs/servers/PLAUSIBLEDENABILITY.md)
+- [N.O.M.A.D.](docs/servers/NOMAD.md)
+- [Meshtastic Stack](docs/reference/MESHTASTIC.md)
+- [AI Context](docs/reference/HOMELAB_AI_CONTEXT.md)
--- a/docs/README.md
+++ b/docs/README.md
--- a/docs/architecture/ARCHITECTURE_OVERVIEW.md
+++ b/docs/architecture/ARCHITECTURE_OVERVIEW.md
@@ -0,0 +1,107 @@
+# Homelab Architecture Overview
+
+## Primary Docker Host
+Host: `PlausibleDeniability` (TrueNAS Scale)
+
+Production Docker workloads run on PlausibleDeniability via standalone Docker Compose stacks.
+TrueNAS Apps (ix-app format) have been fully migrated to compose stacks.
+
+## Repository Layout
+Repo root: `/mnt/docker-ssd/docker/compose`
+Remote: `git@github.com:Paccoco/truenas-stacks.git`
+Gitea mirror: `https://gitea.paccoco.com/fizzlepoof/truenas-stacks.git`
+
+Stack directories:
+- `databases/` = shared-postgres, shared-mariadb, shared-redis — deploy FIRST
+- `infrastructure/` = newt, homepage, uptime-kuma, dockhand, netdata, tailscale — deploy SECOND
+- `media/` = plex, tautulli, audiobookshelf, seerr, calibre-web-automated
+- `photos/` = immich server + machine learning
+- `home/` = kitchenowl, donetick, dakboard bridge, shlink, shlink-web-client, qui
+- `dev/` = gitea, rackpeek
+- `ai/` = ollama, openwebui
+- `meshtastic/` = meshmonitor, mqtt-proxy, tileserver, meshtastic-map
+
+## Storage Model
+
+### Local SSD
+Mounted at: `/mnt/docker-ssd`
+- Docker data root: `/mnt/docker-ssd/docker/docker-data`
+- Databases: `/mnt/docker-ssd/docker/databases/`
+- Write-heavy appdata: `/mnt/docker-ssd/docker/appdata/`
+- GPU/model storage
+- Apps that perform chown or aggressive startup writes
+- SQLite-backed apps
+
+Notable appdata on SSD:
+- plex, tautulli, audiobookshelf, seerr, dockhand, rackpeek
+- immich, immich-ml, ollama, openwebui
+- shlink, gitea, netdata, tailscale, uptime-kuma
+
+### TrueNAS Pool (tank)
+Mounted at: `/mnt/tank`
+- General appdata: `/mnt/tank/docker/appdata/`
+- Backups: `/mnt/tank/docker/backups/`
+- Downloads: `/mnt/tank/docker/downloads/`
+
+Notable appdata on tank:
+- homepage, newt, dockhand (config only)
+- kitchenowl, donetick, qui
+- calibre-web-automated config
+
+### Unraid NFS
+Mounted at: `/mnt/unraid`
+- Media libraries: `/mnt/unraid/data/media/`
+- Immich media: `/mnt/unraid/immich`
+
+## Networks
+- `ix-databases_shared-databases` — created by databases stack; project name `ix-databases` set in compose
+- `pangolin` — created by newt at runtime; used by all externally exposed services
+
+## Active Stacks (as of 2026-05-05)
+
+### databases/
+- shared-postgres (immich-optimized image with vectorchord + pgvector)
+- shared-mariadb
+- shared-redis
+
+### infrastructure/
+- newt, homepage, dockhand, uptime-kuma, netdata, tailscale
+
+### media/
+- ix-plex-plex-1 (GPU passthrough, RTX 2080 Ti)
+- ix-tautulli-tautulli-1
+- ix-audiobookshelf-audiobookshelf-1
+- seerr (Postgres backend)
+- calibre-web-automated
+
+### photos/
+- ix-immich-immich-server-1
+- ix-immich-immich-machine-learning-1 (GPU, CUDA)
+
+### home/
+- kitchenowl, donetick, donetick-dakboard-bridge, qui, shlink, shlink-web-client
+
+### dev/
+- gitea, ix-rackpeek-rackpeek-1
+
+### ai/
+- ollama (GPU, RTX 2080 Ti, port 11434)
+- openwebui (Postgres backend, port 8282)
+
+### meshtastic/
+- meshmonitor, meshmonitor-mqtt-proxy, meshmonitor-tileserver, meshtastic-map
+
+## Planned / Not Yet Deployed
+- dispatcharr, gotify, bytestash, kima-hub, n8n, qdrant, paperless-ngx, home-assistant
+
+## Shelved
+- scriberr — SQLite incompatible with ZFS nfsv4 ACLs
+
+## Design Principles
+- deterministic storage paths
+- no anonymous volumes
+- stack-local .env for secrets; .env.example committed with placeholders
+- SSD for write-heavy / DB / SQLite / GPU workloads
+- tank NFS for general appdata and configs
+- Unraid NFS for media only
+- databases stack uses `name: ix-databases` to maintain network name compatibility
--- a/docs/architecture/HARDWARE_INVENTORY.md
+++ b/docs/architecture/HARDWARE_INVENTORY.md
@@ -0,0 +1,77 @@
+# Hardware Inventory
+
+## Serenity (NAS / Media Ingestion)
+- **OS:** Unraid 7.2.4
+- **IP:** 10.5.1.5 | Tailscale: 100.94.87.79
+- **CPU:** Xeon X5690
+- **RAM:** 102GB
+- **GPU:** None (Matrox IPMI only)
+- **UPS:** None
+
+### Storage
+| Pool | Type | Drives | Usable | Notes |
+|------|------|--------|--------|-------|
+| malcolm | RAIDZ1 | 6× 10TB HDD | ~40TB | Heavily utilized |
+| cache | ZFS single | 1× Kingston SNV2S 1TB NVMe | ~1TB | Appdata/Docker |
+
+### Key Shares
+| Share | Size | Purpose |
+|-------|------|---------|
+| data | 35TB | Media library |
+| immich | 62GB | Immich photo storage |
+| nextcloud | 7.4GB | Nextcloud data |
+| Tesla/dashcam | 1.6TB | Dashcam footage |
+| fizzlebang | 17GB | Personal files |
+| isos | 22GB | ISO images |
+| Syncthing | 4.6GB | Syncthing data |
+| manndra | — | Partner's storage |
+| Long and Low Crafts | — | Crafts business |
+| zfs_backups_array | — | ZFS backup target |
+
+---
+
+## PlausibleDeniability (Primary Docker Host)
+- **OS:** TrueNAS Scale 25.10.2.1
+- **CPU:** Ryzen 9 3950X
+- **RAM:** 32GB
+- **GPU:** Nvidia RTX 2080 Ti (11GB VRAM) — Plex, Immich ML, Ollama
+- **UPS:** None
+
+### Storage
+| Pool | Type | Mount | Notes |
+|------|------|-------|-------|
+| tank | ZFS mirror (2 drives) | `/mnt/tank` | ~3.5TB usable |
+| docker-ssd | Single Crucial MX500 500GB | `/mnt/docker-ssd` | No redundancy |
+| boot-pool | Single sdc3 | — | TrueNAS boot |
+
+### NFS Mounts
+- Serenity data array → `/mnt/unraid/data`
+
+---
+
+## N.O.M.A.D. (Offline Knowledge + Game Servers)
+- **OS:** Ubuntu 25.10 (bare metal)
+- **IP:** 10.5.1.16
+- **CPU:** Intel Core i7-4770K @ 3.50GHz (4c/8t, up to 3.9GHz)
+- **RAM:** 32GB DDR3 1600MHz (4×8GB)
+- **GPU:** NVIDIA GeForce GTX 1080 (8GB VRAM)
+- **NIC:** Intel I218-V (1GbE) + Aquantia AQC113 (10GbE)
+- **Motherboard:** ASRock H97M Pro4
+- **UPS:** None
+
+### Storage
+| Device | Size | Mount | Purpose |
+|--------|------|-------|---------|
+| Samsung SSD 840 (sdb) | 232.9GB | `/` (LVM 100GB) | OS boot |
+| WD Red HDD (sda) | 3.6TB | `/mnt/hdd-1` | Primary data, project-nomad symlink |
+| WD Red HDD (sdc) | 3.6TB | `/mnt/hdd-2` | Docker data-root |
+| Seagate SSD (sdd) | 223.6GB | `/mnt/ssd-1` | Fast scratch |
+
+---
+
+## Desktop: ROCINANTE
+- **OS:** Windows 11 Pro
+- **CPU:** Ryzen 7 9850X3D
+- **RAM:** 64GB
+- **GPU:** RTX 4090 (24GB VRAM)
+- **Role:** Heavy AI inference, local Ollama (expose via `OLLAMA_HOST=0.0.0.0`)
--- a/docs/architecture/NETWORKING_MODEL.md
+++ b/docs/architecture/NETWORKING_MODEL.md
@@ -0,0 +1,56 @@
+# Homelab Networking Model
+
+## Physical Network
+- **Gear:** Unifi throughout
+- **Main LAN:** 10.5.1.x (Trusted VLAN)
+- **Other VLANs:** Cameras, Guest, IoT
+- **Serenity IP:** 10.5.1.5
+- **N.O.M.A.D. IP:** 10.5.1.16
+- **PlausibleDeniability:** DHCP on LAN
+
+## Remote Access
+- **Pangolin VPS + Newt:** Services exposed as subdomains via reverse proxy. Newt agents run on PD and N.O.M.A.D.
+- **Tailscale:** Personal remote access, runs on Serenity (100.94.87.79)
+- **Cloudflared:** No longer used. Dead container on Serenity to be removed.
+
+## DNS
+- **Provider:** Cloudflare
+- All public subdomains point at Pangolin VPS IP
+
+## Known Public Subdomains
+- `paccoco.com` → Pangolin VPS
+- `panel.paccoco.com` → Pelican Panel (N.O.M.A.D.)
+- `node1.paccoco.com` → Wings daemon (N.O.M.A.D., no auth)
+- `meshmonitor.paccoco.com` → Meshmonitor (PD)
+- `gitea.paccoco.com` → Gitea (PD)
+- `ai.paccoco.com` → OpenWebUI (PD) — DNS/Pangolin resource pending
+
+## Docker Networking (PlausibleDeniability)
+
+### ix-databases_shared-databases
+Created by the `databases` stack. The databases compose file uses `name: ix-databases` so Docker names the network `ix-databases_shared-databases`, matching what all other stacks declare as external.
+
+Any stack connecting to shared-postgres, shared-mariadb, or shared-redis must declare:
+```yaml
+networks:
+  ix-databases_shared-databases:
+    external: true
+```
+
+### pangolin
+Created at runtime by the `newt` container in the infrastructure stack. Any service needing reverse-proxy exposure must declare:
+```yaml
+networks:
+  pangolin:
+    external: true
+```
+
+### Internal Service Model
+Internal services run HTTP only behind the reverse proxy. Compose service names used for internal DNS between services in the same stack/network.
+
+## N.O.M.A.D. Networking Notes
+- `panel.paccoco.com` added to `/etc/hosts` → `127.0.0.1` for NAT loopback
+- nginx listens on 80 and 443 (self-signed cert) for Wings → Panel communication
+- Wings uses `--ignore-certificate-errors` for local loopback
+- Pangolin resource `node1.paccoco.com` has auth disabled (Wings handles its own auth)
+- Minecraft port 25565 TCP/UDP goes directly via home router port forward, not through Pangolin
--- a/docs/architecture/SERVICES_DIRECTORY.md
+++ b/docs/architecture/SERVICES_DIRECTORY.md
@@ -0,0 +1,103 @@
+# Services Directory
+
+All running services across the homelab with ports, URLs, and backend info.
+
+## PlausibleDeniability
+
+### Databases (internal only)
+| Service | Port | Backend | Notes |
+|---------|------|---------|-------|
+| shared-postgres | 5432 | — | immich-optimized, vectorchord + pgvector |
+| shared-mariadb | 3306 | — | |
+| shared-redis | 6379 | — | |
+
+### Infrastructure
+| Service | Internal Port | Public URL | Notes |
+|---------|--------------|------------|-------|
+| newt | — | — | Pangolin reverse proxy agent |
+| homepage | — | — | Dashboard |
+| dockhand | — | — | Container management |
+| uptime-kuma | — | — | MariaDB backend |
+| netdata | — | — | Metrics |
+| tailscale | — | — | VPN |
+
+### Media
+| Service | Internal Port | Public URL | Notes |
+|---------|--------------|------------|-------|
+| plex | — | — | GPU passthrough, RTX 2080 Ti |
+| tautulli | — | — | Plex analytics |
+| audiobookshelf | — | — | Audiobooks |
+| seerr | — | — | Postgres backend |
+| calibre-web-automated | — | — | Ebooks |
+
+### Photos
+| Service | Internal Port | Public URL | Notes |
+|---------|--------------|------------|-------|
+| immich-server | — | — | Postgres + Redis, GPU |
+| immich-ml | — | — | GPU, CUDA |
+
+### Home
+| Service | Internal Port | Public URL | Notes |
+|---------|--------------|------------|-------|
+| kitchenowl | — | — | Grocery/tasks |
+| donetick | — | — | Postgres backend, selfhosted.yaml required |
+| shlink | — | — | URL shortener, Postgres |
+| shlink-web-client | — | — | |
+| qui | — | — | qBittorrent UI, manages Serenity remotely |
+
+### Dev
+| Service | Internal Port | Public URL | Notes |
+|---------|--------------|------------|-------|
+| gitea | — | gitea.paccoco.com | Postgres backend |
+| rackpeek | — | — | |
+
+### AI
+| Service | Internal Port | Public URL | Notes |
+|---------|--------------|------------|-------|
+| ollama | 11434 | — | GPU, RTX 2080 Ti |
+| openwebui | 8282→8080 | ai.paccoco.com (pending) | Postgres backend |
+
+### Meshtastic
+| Service | Internal Port | Public URL | Notes |
+|---------|--------------|------------|-------|
+| meshmonitor | 8081→3001 | meshmonitor.paccoco.com | Postgres backend |
+| meshmonitor virtual node | 4404 | — | TCP virtual node server |
+| meshmonitor-tileserver | 8082→8080 | — | Map tiles |
+| meshmonitor-mqtt-proxy | — | — | MQTT→TCP bridge |
+
+---
+
+## Serenity
+
+### Media Ingestion
+| Service | Notes |
+|---------|-------|
+| sonarr | TV |
+| sonarr-anime | Anime |
+| radarr | Movies |
+| lidarr | Music |
+| readarr | Books |
+| readarr-epub | EPUBs |
+| prowlarr | Indexer manager |
+| bazarr | Subtitles |
+| unpackerr | Archive extraction |
+| qBittorrent | Download client |
+| GluetunVPN | VPN tunnel for qbit |
+| autobrr | Auto-grab |
+| qbit_manage | qBit maintenance |
+| Notifiarr | Notifications |
+| Wizarr | User invites |
+| romm | ROM manager |
+| GameVault | Game library |
+| shelfmark | Book/audiobook search, port 8084 |
+
+---
+
+## N.O.M.A.D.
+
+| Service | Port | Notes |
+|---------|------|-------|
+| Pelican Panel | 80/443 | panel.paccoco.com, PHP 8.4 + nginx + MariaDB |
+| Wings | 8443 | node1.paccoco.com, game server daemon |
+| Minecraft | 25565 | Purpur, Java 21, direct port forward |
+| N.O.M.A.D. containers | — | 10 containers, start_nomad.sh / stop_nomad.sh |
--- a/docs/operations/BACKUP_POLICY.md
+++ b/docs/operations/BACKUP_POLICY.md
@@ -0,0 +1,42 @@
+# Backup Policy
+
+## Current State (Known Gaps)
+- **No off-site backup** — this is a known gap, improvement pending
+- Serenity has local ZFS snapshots and replication
+- PlausibleDeniability has no automated backup of appdata or databases
+
+## Serenity Backup Strategy
+- Sanoid snapshots cache/appdata: 7 daily, 4 weekly, 6 monthly
+- Syncoid replicates cache/appdata → malcolm/zfs_backups_array (strict-mirror)
+- qBit Mover script: pauses torrents 0–4 days old then runs Unraid mover
+- Mover Status script: sends Discord webhook notifications
+
+## PlausibleDeniability Backup Scope
+Back up at minimum:
+- Compose repo: `/mnt/docker-ssd/docker/compose` (in git — GitHub + Gitea)
+- Local SSD appdata: `/mnt/docker-ssd/docker/appdata/`
+- Databases: `/mnt/docker-ssd/docker/databases/`
+- Selected appdata/backups on tank: `/mnt/tank/docker/`
+- .env files (separately — gitignored, handle via encrypted backup or private repo)
+
+## Database Backups
+- Keep app-consistent dumps for major databases
+- Do not rely only on raw volume copies for restores
+- Document restore procedure per major service
+
+Recommended dump locations: `/mnt/tank/docker/backups/db-dumps/`
+
+## .env File Backup Options
+1. **Private Gitea repo** — short term, push all .env files to a private repo on your local Gitea
+2. **Encrypted restic** — include .env files in restic backup with encryption, push offsite (Backblaze B2)
+3. **Vaultwarden** — store each .env as a secure note (best long-term once deployed)
+
+## Restore Practice
+- Test restores quarterly
+- Test at least one app DB restore and one full-config restore
+- Verify that restored stacks can start with current compose files
+
+## Operational Notes
+- Media libraries on Unraid are large and may not need the same backup frequency as configs/databases
+- SSD-local DB and SQLite workloads are highest priority for backup validation
+- Serenity malcolm pool is heavily utilized — capacity planning needed before it fills
--- a/docs/operations/DEPLOYMENT_CHECKLIST.md
+++ b/docs/operations/DEPLOYMENT_CHECKLIST.md
@@ -0,0 +1,63 @@
+# New Service Deployment Checklist
+
+## Before Adding a Service
+
+- [ ] Determine DB requirements (postgres, mariadb, sqlite, none)
+- [ ] Decide storage tier: SSD (`/mnt/docker-ssd/docker/appdata/`) vs tank (`/mnt/tank/docker/appdata/`) vs Unraid
+- [ ] Check if service needs `ix-databases_shared-databases` network
+- [ ] Check if service needs `pangolin` network for reverse proxy exposure
+
+## Scaffold
+
+- [ ] Create host directories for appdata
+- [ ] Set correct ownership if needed (`chown` or `chmod 777` for permission-sensitive apps)
+- [ ] Create stack folder under `/mnt/docker-ssd/docker/compose/<stack>/`
+- [ ] Create `docker-compose.yaml`
+- [ ] Create `.env` and `.env.example`
+- [ ] If using postgres: create DB and user, or add init SQL to `initdb/`
+- [ ] If using mariadb: add init script to `initdb-mariadb/`
+
+## Validate Before Deploying
+
+```bash
+cd /mnt/docker-ssd/docker/compose/<stack>
+docker compose --env-file .env config
+```
+
+No errors = safe to proceed.
+
+## Deploy
+
+```bash
+docker compose --env-file .env up -d
+```
+
+## After Starting
+
+- [ ] `docker ps` shows container running
+- [ ] `docker logs <container> --tail 20` shows clean startup
+- [ ] `docker inspect <container>` confirms mounts are correct
+- [ ] Data persists after `docker compose --env-file .env down && up -d`
+
+## Network Checklist
+
+If service needs shared databases:
+- Top-level networks block declares `ix-databases_shared-databases` as external
+- Service networks list includes `ix-databases_shared-databases`
+- Verify: `docker inspect <container> --format '{{json .NetworkSettings.Networks}}'`
+
+If service needs reverse proxy:
+- Top-level networks block declares `pangolin` as external
+- Service networks list includes `pangolin`
+- Infrastructure stack (newt) must be running first
+
+## Common Failure Modes
+
+| Symptom | Likely Cause | Fix |
+|---|---|---|
+| `hostname resolving error` for shared-postgres | Not on `ix-databases_shared-databases` network | `down && up` not just restart |
+| `not writable` errors on startup | NFS ownership mismatch | Move to SSD or chmod 777 |
+| `unsupported database type` | App requires config file, not just env vars | Create `selfhosted.yaml` or equivalent |
+| `JWT is invalid` after restart | JWT secret changed | Clear browser cookies/localStorage |
+| Port conflict on startup | Port already bound | Remove conflicting port mapping |
+| Auto-upgrade crashes on TrueNAS | Bind mount source path missing, Docker tries to create at root | Pre-create the directory manually |
--- a/docs/operations/DOCKERSTACK_STORAGE_NOTES.md
+++ b/docs/operations/DOCKERSTACK_STORAGE_NOTES.md
@@ -0,0 +1,61 @@
+# Dockerstack Storage Notes
+
+## Local SSD
+Use `/docker/...` for:
+- Docker root (`/docker/docker-data`)
+- GPU/model storage
+- SQLite-backed apps
+- Apps that chown on startup
+- Write-heavy DB/search storage
+
+Examples:
+- `/mnt/docker-ssd/docker/docker-data`
+- `/mnt/docker-ssd/docker/appdata/ollama`
+- `/mnt/docker-ssd/docker/appdata/openwebui`
+- `/mnt/docker-ssd/docker/appdata/immich`
+- `/mnt/docker-ssd/docker/appdata/immich-ml`
+- `/mnt/docker-ssd/docker/appdata/plex`
+- `/mnt/docker-ssd/docker/appdata/tautulli`
+- `/mnt/docker-ssd/docker/appdata/audiobookshelf`
+- `/mnt/docker-ssd/docker/appdata/seerr`
+- `/mnt/docker-ssd/docker/appdata/dockhand`
+- `/mnt/docker-ssd/docker/appdata/rackpeek`
+- `/mnt/docker-ssd/docker/appdata/shlink`
+- `/mnt/docker-ssd/docker/appdata/gitea`
+- `/mnt/docker-ssd/docker/appdata/netdata`
+- `/mnt/docker-ssd/docker/appdata/tailscale`
+- `/mnt/docker-ssd/docker/appdata/uptime-kuma`
+- `/mnt/docker-ssd/docker/databases/postgres`
+- `/mnt/docker-ssd/docker/databases/mariadb`
+- `/mnt/docker-ssd/docker/databases/redis`
+
+## TrueNAS NFS
+Use `/mnt/tank/docker/...` for:
+- Normal appdata
+- Uploads
+- Non-fussy config and data
+- Backups
+- Downloads
+
+Examples:
+- `/mnt/tank/docker/appdata/homepage`
+- `/mnt/tank/docker/appdata/newt`
+- `/mnt/tank/docker/appdata/kitchenowl`
+- `/mnt/tank/docker/appdata/donetick`
+- `/mnt/tank/docker/appdata/qui`
+- `/mnt/tank/docker/appdata/calibre-web-automated`
+- `/mnt/tank/docker/backups`
+- `/mnt/tank/docker/downloads`
+
+## Unraid NFS
+Use `/mnt/unraid/...` for media and shared bulk content only:
+- `/mnt/unraid/data/media/` — Plex libraries, books, etc.
+- `/mnt/unraid/immich` — Immich photo/video storage
+
+## Rules Learned
+- SQLite apps and apps that write logs aggressively should prefer local SSD.
+- Database/search/model-heavy workloads should prefer local SSD.
+- General config and less write-sensitive appdata can live on TrueNAS NFS.
+- Media should stay on Unraid mounts.
+- SQLite on ZFS with nfsv4 ACLs will always fail — move to SSD or use Postgres.
+- Apps with auto-upgrade features that use bind mounts: pre-create all source directories manually on TrueNAS to prevent Docker from trying to create them at the read-only root.
--- a/docs/operations/SECRETS_MANAGEMENT.md
+++ b/docs/operations/SECRETS_MANAGEMENT.md
@@ -0,0 +1,64 @@
+# Secrets Management
+
+## Principles
+- `.env` files are gitignored and never committed to any public repo
+- `.env.example` files are committed with placeholder values
+- Real `.env` files live only on the host at the stack directory
+- Secrets are generated with standard tools, never reused across services
+
+## Generating Secrets
+
+```bash
+# Database passwords
+openssl rand -hex 24
+
+# JWT / session secrets
+openssl rand -hex 32
+
+# API keys (where format allows)
+openssl rand -base64 32
+```
+
+## .env File Locations
+All `.env` files live alongside their `docker-compose.yaml`:
+```
+/mnt/docker-ssd/docker/compose/<stack>/.env
+```
+
+## Backing Up .env Files
+
+### Short term: Private Gitea repo
+Create a private repo on Gitea (not GitHub) and push all .env files there:
+```bash
+# In a separate directory, not the main stacks repo
+git init homelab-secrets
+cd homelab-secrets
+# Copy and add .env files
+git remote add origin https://gitea.paccoco.com/fizzlepoof/homelab-secrets.git
+git push -u origin main
+```
+
+### Long term: Restic + offsite
+Include the compose directory (with .env files) in a restic backup:
+```bash
+restic -r <repo> backup /mnt/docker-ssd/docker/compose --exclude="*.example"
+```
+
+### Best long term: Vaultwarden
+Once deployed, store each service's .env as a secure note in Vaultwarden. Single source of truth, accessible from anywhere.
+
+## Known Credentials Locations
+
+| Service | Where stored |
+|---------|-------------|
+| shared-postgres | `databases/.env` |
+| shared-mariadb | `databases/.env` |
+| OpenWebUI DB | `ai/.env` — user: openwebui, db: openwebui |
+| Donetick | `home/.env` + `selfhosted.yaml` |
+| Meshmonitor | `meshtastic/.env` |
+| Gitea | `dev/.env` |
+
+## Security Reminders
+- Regenerate Wings token on N.O.M.A.D. (was exposed in chat — TODO)
+- Regenerate N.O.M.A.D. Newt secret (was exposed in chat — TODO)
+- Disable signups in OpenWebUI after creating initial account
--- a/docs/operations/STACK_STANDARDS.md
+++ b/docs/operations/STACK_STANDARDS.md
@@ -0,0 +1,122 @@
+# Homelab Stack Standards
+
+## Repository Root
+`/mnt/docker-ssd/docker/compose`
+
+Remotes:
+- GitHub: `git@github.com:Paccoco/truenas-stacks.git`
+- Gitea: `https://gitea.paccoco.com/fizzlepoof/truenas-stacks.git`
+
+## Stack Layout
+Each production stack lives in a subdirectory of the repo root.
+
+Each stack contains:
+- `docker-compose.yaml` — required
+- `.env` — required (gitignored)
+- `.env.example` — committed with placeholder values
+- optional init scripts (e.g. `initdb/`, `initdb-mariadb/`)
+
+## Deployment Order
+1. `databases/` — must be up before anything else
+2. `infrastructure/` — newt creates the pangolin network
+3. All other stacks in any order
+
+## Deployment Workflow
+
+Validate before deploying:
+```bash
+cd /mnt/docker-ssd/docker/compose/<stack>
+docker compose --env-file .env config
+```
+
+Deploy:
+```bash
+docker compose --env-file .env up -d
+```
+
+Teardown a single service:
+```bash
+docker compose --env-file .env down <service>
+```
+
+## Networks
+
+### ix-databases_shared-databases
+Created by the `databases` stack with `name: ix-databases` at top level. Any stack connecting to shared databases must declare:
+```yaml
+networks:
+  ix-databases_shared-databases:
+    external: true
+```
+
+### pangolin
+Created by `newt` at runtime. Any externally exposed service must declare:
+```yaml
+networks:
+  pangolin:
+    external: true
+```
+
+## Storage Standards
+
+### Local SSD: `/mnt/docker-ssd`
+Use `/mnt/docker-ssd/docker/appdata/<service>` for:
+- Postgres / MariaDB / Redis data
+- SQLite-backed apps
+- Apps that chown aggressively on startup
+- GPU/model caches
+- Write-heavy logs, cache, state
+
+### TrueNAS Pool (tank): `/mnt/tank`
+Use `/mnt/tank/docker/appdata/<service>` for:
+- General appdata
+- Configs
+- Uploads
+- Non-permission-sensitive state
+
+### Unraid NFS: `/mnt/unraid`
+- `/mnt/unraid/data/media/` — media libraries
+- `/mnt/unraid/immich` — Immich photo/video storage
+
+## Volume Rules
+- no anonymous volumes
+- no ad-hoc home directory paths
+- all binds use standardized production paths
+- evaluate SSD vs tank vs Unraid before deploying any new service
+- apps known to have permission issues on NFS go on SSD
+
+## Known Permission Quirks
+- `shlink` data directory must be chmod 777 — runs as non-root user
+- `immich-ml` cache goes to `/mnt/docker-ssd/docker/appdata/immich-ml` (separate from immich-server)
+- `donetick` requires `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml` — env vars alone insufficient
+
+## Secrets Management
+- `.env` files are gitignored
+- `.env.example` files are committed with placeholder values
+- Database passwords: `openssl rand -hex 24`
+- JWT secrets: `openssl rand -hex 32`
+
+## Healthcheck Patterns
+Always check available tools before writing healthchecks:
+```bash
+sudo docker exec <container> sh -c "which curl; which wget" 2>&1
+```
+
+- Has curl: `["CMD-SHELL", "curl -sf http://127.0.0.1:<port>/ || exit 1"]`
+- Has wget only: `["CMD-SHELL", "wget -qO- http://127.0.0.1:<port>/ >/dev/null 2>&1 || exit 1"]`
+- Has neither: `["CMD-SHELL", "cat /proc/net/tcp6 | grep -q <HEX_PORT> || exit 1"]`
+- Redirecting endpoint: `["CMD-SHELL", "curl -sfL http://127.0.0.1:<port>/ || exit 1"]`
+
+Common hex port values: 8080=1F90, 3003=BBB, 3000=BB8, 8181=1FF5, 5055=13BF, 2021=7E5
+
+After changing a healthcheck, must `down` then `up` (not just restart).
+
+## Git Workflow
+```bash
+cd /mnt/docker-ssd/docker/compose
+git add .
+git commit -m "description"
+git push   # pushes to both GitHub and Gitea
+```
+
+All `.env` files are gitignored. Only compose files, `.env.example`, and init scripts are committed.
--- a/docs/planning/HOMELAB_EXPANSION_PLAN.md
+++ b/docs/planning/HOMELAB_EXPANSION_PLAN.md
@@ -0,0 +1,106 @@
+# Homelab Expansion Plan
+
+## Deployment Order
+1. **Gotify** — unified notification bus, unblocks all n8n workflows
+2. **Qdrant** — vector DB, unblocks proper RAG pipeline
+3. **n8n** — automation backbone, ties everything together
+4. **Paperless-NGX** — document intelligence
+5. **Home Assistant** — home automation layer
+6. **Grafana + Prometheus** — unified observability
+
+---
+
+## Phase 1 — Notifications (Gotify)
+**Host:** PlausibleDeniability
+**Stack:** `/mnt/docker-ssd/docker/compose/automation/`
+**Storage:** appdata on SSD (SQLite backend)
+**Network:** pangolin (expose as `gotify.paccoco.com`)
+
+Single output target for all n8n workflows and homelab alerts.
+
+---
+
+## Phase 2 — Vector Database (Qdrant)
+**Host:** PlausibleDeniability
+**Stack:** `/mnt/docker-ssd/docker/compose/ai/`
+**Storage:** appdata on SSD
+
+Feed it: homelab markdown docs, Gitea commit history, Plex/Tautulli metadata, Obsidian notes.
+
+---
+
+## Phase 3 — Workflow Automation (n8n)
+**Host:** PlausibleDeniability
+**Stack:** `/mnt/docker-ssd/docker/compose/automation/`
+**Storage:** config on tank, Postgres backend
+**Network:** pangolin (`n8n.paccoco.com`) + ix-databases_shared-databases
+
+### Planned Workflows
+- Sonarr/Radarr webhook → Discord + Gotify notification with poster art
+- Tautulli play start → log to Postgres
+- Nightly watch history → Ollama → weekly digest via Gotify
+- Poll Uptime Kuma → cross-reference Netdata → rich alert via Gotify
+- Gitea commit to truenas-stacks → Gotify notification with Ollama summary
+- qBittorrent torrent complete → trigger Sonarr/Radarr rescan
+- KitchenOwl grocery list change → shopping list via Gotify
+- Donetick task due → Gotify reminder
+- Long and Low Crafts order → Donetick task + Gotify DM
+
+---
+
+## Phase 4 — Document Intelligence (Paperless-NGX)
+**Host:** PlausibleDeniability
+**Network:** pangolin (`paperless.paccoco.com`) + ix-databases_shared-databases
+
+OCR and full-text search. Key for Long and Low Crafts invoices, receipts, and manuals.
+
+---
+
+## Phase 5 — Home Automation (Home Assistant)
+**Host:** PlausibleDeniability or N.O.M.A.D.
+**Storage:** config on SSD (SQLite, write-heavy)
+**Network:** host networking recommended for device discovery
+
+---
+
+## Phase 6 — Observability (Grafana + Prometheus)
+**Host:** N.O.M.A.D.
+**Storage:** Prometheus data on hdd-1 or hdd-2
+
+Metrics to scrape: ZFS pool health, container CPU/RAM, Plex streams, qBittorrent, Tailscale latency, Netdata endpoints.
+
+---
+
+## Multi-Model Routing Strategy
+
+| Machine | GPU | Model | Use Case |
+|---|---|---|---|
+| ROCINANTE | RTX 4090 (24GB) | qwen3:32b / deepseek-r1:32b | Heavy reasoning, long context |
+| PlausibleDeniability | RTX 2080 Ti (11GB) | qwen2.5:14b | General homelab assistant, RAG |
+| N.O.M.A.D. | GTX 1080 (8GB) | phi4 / llama3.2:3b | Fast local inference |
+
+---
+
+## Additional Tools
+- **Continue.dev** — VS Code plugin, local Copilot alternative
+- **Obsidian + Gitea Sync** — personal notes → RAG pipeline
+- **Homepage Ollama Widget** — VRAM usage dashboard widget
+- **kima-hub** — self-hosted Spotify alternative
+- **dispatcharr** — planned
+- **bytestash** — planned
+
+---
+
+## Target Architecture
+
+```
+Triggers (Sonarr, Radarr, Tautulli, Gitea, Home Assistant, schedules, email)
+        ↓
+      n8n (routing, logic, orchestration)
+        ↓
+    Ollama (intelligence — model selected by complexity)
+      ↓  ↑
+    Qdrant (semantic context retrieval)
+        ↓
+     Outputs (Gotify → phone, Donetick tasks, Postgres logging, Discord)
+```
--- a/docs/planning/TODO.md
+++ b/docs/planning/TODO.md
@@ -0,0 +1,43 @@
+# Homelab TODO
+
+## Security (do ASAP)
+- [ ] Regenerate Wings token on N.O.M.A.D. (was exposed in chat)
+- [ ] Regenerate N.O.M.A.D. Newt secret (was exposed in chat)
+
+## Infrastructure
+- [ ] Deploy Pi-hole ×2 (primary on PD + backup on Serenity or Pi4)
+- [ ] Set up off-site backup for vital data (appdata, databases, config repo, photos)
+- [ ] Remove dead Unraid-Cloudflared-Tunnel container from Serenity
+- [ ] Serenity malcolm pool capacity planning (heavily utilized)
+- [ ] Add Pangolin resource for `ai.paccoco.com` → openwebui (port 8282)
+- [ ] Add Cloudflare DNS A record for `ai.paccoco.com`
+- [ ] Install fresh editor on PD
+- [ ] Set up private Gitea repo for .env file backups
+
+## AI Stack
+- [ ] Pull `qwen2.5:14b` on PD ollama: `sudo docker exec -it ollama ollama pull qwen2.5:14b`
+- [ ] Create OpenWebUI account and set system prompt from HOMELAB_AI_CONTEXT.md
+- [ ] Disable signups in OpenWebUI after creating account
+- [ ] Connect desktop Ollama to OpenWebUI (Admin → Settings → Connections)
+- [ ] Set `OLLAMA_HOST=0.0.0.0` on desktop and restart Ollama service
+
+## Planned Deployments (PD)
+- [ ] dispatcharr
+- [ ] gotify
+- [ ] bytestash
+- [ ] kima-hub
+- [ ] n8n
+- [ ] qdrant
+- [ ] paperless-ngx
+
+## Healthchecks
+- [ ] Add healthcheck to ix-plex-plex-1
+- [ ] Add healthcheck to ix-tailscale-tailscale-1
+- [ ] Add healthcheck to ix-newt-newt-1
+- [ ] Add healthcheck to ollama (no curl/wget — needs /proc/net/tcp6 or custom solution)
+- [ ] Document and add healthchecks for meshtastic stack
+
+## Documentation
+- [ ] Document meshtastic stack fully (see docs/reference/MESHTASTIC.md)
+- [ ] Fill in port/URL columns in SERVICES_DIRECTORY.md
+- [ ] Document N.O.M.A.D. containers (start_nomad.sh services)
--- a/docs/reference/HOMELAB_AI_CONTEXT.md
+++ b/docs/reference/HOMELAB_AI_CONTEXT.md
@@ -0,0 +1,56 @@
+# Homelab AI Context Document
+
+This document gives an AI assistant full context of this homelab for infrastructure questions, deployments, debugging, and planning.
+
+See the individual docs in this repo for detailed information:
+
+- [Architecture Overview](../architecture/ARCHITECTURE_OVERVIEW.md)
+- [Networking Model](../architecture/NETWORKING_MODEL.md)
+- [Hardware Inventory](../architecture/HARDWARE_INVENTORY.md)
+- [Services Directory](../architecture/SERVICES_DIRECTORY.md)
+- [Stack Standards](../operations/STACK_STANDARDS.md)
+- [Deployment Checklist](../operations/DEPLOYMENT_CHECKLIST.md)
+- [Storage Notes](../operations/DOCKERSTACK_STORAGE_NOTES.md)
+- [Backup Policy](../operations/BACKUP_POLICY.md)
+- [Secrets Management](../operations/SECRETS_MANAGEMENT.md)
+- [Troubleshooting](../troubleshooting/TROUBLESHOOTING.md)
+- [Known Quirks](../troubleshooting/KNOWN_QUIRKS.md)
+- [Serenity](../servers/SERENITY.md)
+- [PlausibleDeniability](../servers/PLAUSIBLEDENABILITY.md)
+- [N.O.M.A.D.](../servers/NOMAD.md)
+- [Meshtastic Stack](MESHTASTIC.md)
+- [Expansion Plan](../planning/HOMELAB_EXPANSION_PLAN.md)
+- [TODO](../planning/TODO.md)
+
+## Quick Reference
+
+### Communication Preferences
+- One command block at a time
+- Prefer scripted/automated solutions over interactive multi-step processes
+- Uses `fresh` editor (not nano) on Linux
+- Manages remote servers via SSH from Windows 11
+- Prefers MariaDB over PostgreSQL for standalone services
+
+### Three Servers
+| Server | Role | OS | IP |
+|--------|------|----|----|
+| Serenity | NAS / Media ingestion | Unraid 7.2.4 | 10.5.1.5 |
+| PlausibleDeniability | Primary Docker host | TrueNAS Scale 25.10.2.1 | LAN DHCP |
+| N.O.M.A.D. | Offline knowledge + game servers | Ubuntu 25.10 | 10.5.1.16 |
+
+### Deployment Pattern (PD)
+```bash
+cd /mnt/docker-ssd/docker/compose/<stack>
+docker compose --env-file .env config   # validate first
+docker compose --env-file .env up -d    # deploy
+docker logs <container> --tail 20       # verify
+```
+
+### Network names
+- Shared DB network: `ix-databases_shared-databases`
+- Reverse proxy network: `pangolin`
+
+### Storage decision
+- Write-heavy / DB / SQLite / GPU → `/mnt/docker-ssd/docker/appdata/<service>`
+- General config / non-fussy → `/mnt/tank/docker/appdata/<service>`
+- Media only → `/mnt/unraid/data/media/`
--- a/docs/reference/MESHTASTIC.md
+++ b/docs/reference/MESHTASTIC.md
@@ -0,0 +1,70 @@
+# Meshtastic Stack
+
+## Overview
+Self-hosted Meshtastic monitoring and management stack running on PlausibleDeniability.
+
+## Stack Location
+`/mnt/docker-ssd/docker/compose/meshtastic/`
+
+## Containers
+
+| Container | Image | Port | Purpose |
+|-----------|-------|------|---------|
+| meshmonitor | ghcr.io/yeraze/meshmonitor:4.2.0 | 8081→3001 | Main UI + backend |
+| meshmonitor (virtual node) | — | 4404 | TCP virtual node server |
+| meshmonitor-tileserver | maptiler/tileserver-gl-light | 8082→8080 | Offline map tiles |
+| meshmonitor-mqtt-proxy | ghcr.io/ln4cy/mqtt-proxy:master | — | MQTT→TCP bridge |
+| meshmonitor-upgrader | docker:latest | — | Auto-upgrade watchdog |
+
+## Architecture
+```
+Meshtastic Node (10.5.1.172)
+        ↓ MQTT
+meshmonitor-mqtt-proxy
+        ↓ TCP (port 4404)
+meshmonitor virtual node server
+        ↓
+meshmonitor backend
+        ↓
+PostgreSQL (shared-postgres, db: meshmonitor)
+```
+
+## Node Connection
+- Node IP: `10.5.1.172`
+- Connection type: MQTT via mqtt-proxy (not direct TCP)
+- mqtt-proxy connects to meshmonitor's virtual node server on port 4404
+
+## Storage
+- Appdata: `/mnt/tank/docker/appdata/meshmonitor/data` → `/data`
+- Scripts: `./scripts` → `/data/scripts` (must be pre-created on host)
+- Tiles: `./tiles` (gitignored — large .mbtiles files)
+
+## Auto-Upgrade
+`AUTO_UPGRADE_ENABLED=true` — meshmonitor checks for new images and triggers self-upgrade.
+
+**Important:** The upgrader mounts the compose directory as `/compose` inside the container. All relative bind mount paths (e.g. `./scripts`) resolve to `/compose/scripts` on the host when the upgrader re-runs compose. Since TrueNAS root filesystem is read-only, these directories must be pre-created:
+
+```bash
+mkdir -p /mnt/docker-ssd/docker/compose/meshtastic/scripts
+```
+
+## Networks
+- `default` (internal stack network)
+- `ix-databases_shared-databases` (meshmonitor only)
+- `pangolin` (meshmonitor + tileserver)
+
+## Public URL
+`https://meshmonitor.paccoco.com`
+
+## Known Warnings
+- Node `!dd972536` has a low-entropy public key — this is a device configuration issue, not a meshmonitor bug
+- Frequent connect/disconnect cycles in logs are normal when the MQTT proxy reconnects
+
+## Tiles
+Map tiles stored in `./tiles/`. Currently includes `zurich_switzerland.mbtiles` (34MB).
+Tiles directory is gitignored. To add new tiles, download `.mbtiles` files and place in the `tiles/` directory.
+
+## TODO
+- [ ] Document all running meshtastic-map container config
+- [ ] Document meshtastic-map-mosquitto and meshtastic-mqtt containers
+- [ ] Add healthcheck to meshmonitor container
--- a/docs/servers/NOMAD.md
+++ b/docs/servers/NOMAD.md
@@ -0,0 +1,73 @@
+# N.O.M.A.D. Server
+
+Offline survival/knowledge server and game server host.
+
+## Specs
+- **OS:** Ubuntu 25.10 (bare metal)
+- **IP:** 10.5.1.16
+- **CPU:** Intel Core i7-4770K @ 3.50GHz (4c/8t)
+- **RAM:** 32GB DDR3 1600MHz
+- **GPU:** NVIDIA GeForce GTX 1080 (8GB VRAM)
+- **NIC:** Intel I218-V (1GbE) + Aquantia AQC113 (10GbE)
+- **Motherboard:** ASRock H97M Pro4
+- **UPS:** None
+
+## Storage
+| Device | Size | Mount | Purpose |
+|--------|------|-------|---------|
+| Samsung SSD 840 (sdb) | 232.9GB | `/` (LVM 100GB) | OS boot |
+| WD Red HDD (sda) | 3.6TB | `/mnt/hdd-1` | Primary data |
+| WD Red HDD (sdc) | 3.6TB | `/mnt/hdd-2` | Docker data-root |
+| Seagate SSD (sdd) | 223.6GB | `/mnt/ssd-1` | Fast scratch |
+
+## Project N.O.M.A.D.
+- 10 containers running via `start_nomad.sh` / `stop_nomad.sh`
+- `/opt/project-nomad` → symlinked from `/mnt/hdd-1/project-nomad`
+- Docker data-root: `/mnt/hdd-2/docker`
+- Based on Crosstalk Solutions offline survival/knowledge server
+
+## Pelican Panel (Game Server Management)
+- **URL:** `https://panel.paccoco.com`
+- **Stack:** PHP 8.4 + nginx + MariaDB (native, not Docker)
+- **Panel files:** `/var/www/pelican`
+- **DB:** MariaDB, database `pelican`, user `pelican`@`127.0.0.1`
+- **APP_URL:** `https://panel.paccoco.com`
+- **TRUSTED_PROXIES:** `*`
+- **Cron:** `* * * * * php /var/www/pelican/artisan schedule:run` (www-data)
+- **Admin user:** `fizzlepoof`
+- **Reverse proxy:** Pangolin via Newt container at `/opt/newt`
+
+## Wings (Game Server Daemon)
+- **Binary:** `/usr/local/bin/wings`
+- **Config:** `/etc/pelican/config.yml`
+- **Systemd:** `wings.service`
+- **ExecStart:** `/usr/local/bin/wings --ignore-certificate-errors`
+- **API port:** 8443
+- **SFTP port:** 2022
+- **Node domain:** `node1.paccoco.com` (Pangolin, no auth, connection port 443)
+- **Game server volumes:** `/var/lib/pelican/volumes`
+- **Docker network:** `pelican_nw` (172.20.0.0/16)
+
+## Active Game Servers
+- **Minecraft:** Purpur, Java 21, port 25565 TCP/UDP (direct host port forward)
+
+## Networking Notes
+- `panel.paccoco.com` → `127.0.0.1` in `/etc/hosts` for NAT loopback
+- nginx on 80 and 443 (self-signed cert) for local Wings → Panel
+- Wings uses `--ignore-certificate-errors` for self-signed cert
+- Pangolin resource `node1.paccoco.com` has auth disabled
+
+## Known Quirks
+- NVIDIA container toolkit repo has invalid GPG key — remove `/etc/apt/sources.list.d/*nvidia*` if apt errors
+- Port 8080 occupied by `nomad_admin` container — Wings uses 8443
+- Interactive artisan commands break over SSH — always use inline flags
+- Wings CORS derived from `remote:` URL — keep remote as `https://` and use `--ignore-certificate-errors`
+
+## Newt Tunnel
+- Container: `newt` at `/opt/newt/docker-compose.yaml`
+- `network_mode: host`
+- Connects to Pangolin VPS at `https://paccoco.com`
+
+## Pending
+- Regenerate Wings token (was exposed in chat)
+- Regenerate Newt secret (was exposed in chat)
--- a/docs/servers/PLAUSIBLEDENABILITY.md
+++ b/docs/servers/PLAUSIBLEDENABILITY.md
@@ -0,0 +1,60 @@
+# PlausibleDeniability
+
+Primary Docker host running all production workloads.
+
+## Specs
+- **OS:** TrueNAS Scale 25.10.2.1
+- **CPU:** Ryzen 9 3950X
+- **RAM:** 32GB
+- **GPU:** Nvidia RTX 2080 Ti (11GB VRAM)
+- **UPS:** None
+
+## Storage Tiers
+
+| Tier | Mount | Use for |
+|------|-------|---------|
+| Local SSD (Crucial MX500 500GB) | `/mnt/docker-ssd` | Docker root, databases, write-heavy appdata, GPU/model caches |
+| TrueNAS pool (tank, 2-drive ZFS mirror ~3.5TB) | `/mnt/tank` | General appdata, configs, backups, downloads |
+| Unraid NFS | `/mnt/unraid` | Media libraries, Immich media |
+
+## Docker Root
+`/mnt/docker-ssd/docker/docker-data`
+
+## Compose Repo
+`/mnt/docker-ssd/docker/compose`
+
+## Key Docker Networks
+- `ix-databases_shared-databases` — shared DB access network (created by databases stack)
+- `pangolin` — reverse proxy exposure network (created by newt)
+
+## GPU Usage
+RTX 2080 Ti used by:
+- Plex (hardware transcoding)
+- immich-ml (CUDA face recognition)
+- ollama (LLM inference)
+
+Recommended models for 11GB VRAM:
+| Model | VRAM | Best for |
+|-------|------|---------|
+| `qwen2.5:14b` | ~8GB | General use, coding, reasoning |
+| `qwen2.5-coder:7b` | ~5GB | Code tasks |
+| `deepseek-r1:8b` | ~5GB | Math, logic |
+| `phi4` | ~8GB | Analytical tasks |
+
+## Remote Access
+- Pangolin VPS + Newt for external services
+- SSH from Windows 11
+
+## NFS Exports to Other Servers
+- `data` share → PD at `/mnt/unraid/data`
+- `immich` share → PD at `/mnt/unraid/immich`
+
+## Known Issues
+- immich-ml intermittently unhealthy — investigate GPU/CUDA health
+- docker-ssd has no redundancy — single point of failure for all appdata
+- No off-site backup
+
+## Pending
+- Deploy: dispatcharr, gotify, bytestash, kima-hub, n8n, qdrant, paperless-ngx
+- Add Pangolin resource + DNS for `ai.paccoco.com`
+- Install fresh editor
--- a/docs/servers/SERENITY.md
+++ b/docs/servers/SERENITY.md
@@ -0,0 +1,55 @@
+# Serenity
+
+Unraid NAS and media ingestion server.
+
+## Specs
+- **OS:** Unraid 7.2.4
+- **IP:** 10.5.1.5 | Tailscale: 100.94.87.79
+- **CPU:** Xeon X5690
+- **RAM:** 102GB
+- **GPU:** None (Matrox IPMI only)
+- **UPS:** None
+
+## Storage
+- `malcolm` pool: RAIDZ1 of 6×10TB HDDs (~40TB usable) — heavily utilized
+- Cache pool: single Kingston SNV2S 1TB NVMe (ZFS)
+
+## Docker Networks
+`arrs`, `hotio`, `br0` (macvlan), `bridge`
+
+## Active Containers
+- **Media:** sonarr, sonarr-anime, radarr, lidarr, readarr, readarr-epub, prowlarr, bazarr, unpackerr
+- **Download:** qBittorrent, GluetunVPN, autobrr, qbit_manage
+- **Notifications:** Notifiarr
+- **Access:** Wizarr, Newt, dockersocket, Hawser
+- **Databases:** postgresql15, MariaDB-Official
+- **Other:** romm, GameVault, shelfmark (port 8084), netdata
+- **Dead/remove:** Unraid-Cloudflared-Tunnel
+
+## Backup Strategy
+- Sanoid snapshots cache/appdata: 7 daily, 4 weekly, 6 monthly
+- Syncoid replicates cache/appdata → malcolm/zfs_backups_array (strict-mirror)
+- No off-site backup currently
+
+## Custom Scripts
+- **qBit Mover script:** pauses torrents 0–4 days old, then runs Unraid mover
+- **MAM-init script:** updates MyAnonamouse IP inside qbit container
+- **Mover Status script:** sends Discord webhook notifications
+
+## Key Shares
+| Share | Size | Purpose |
+|-------|------|---------|
+| data | 35TB | Media library (NFS-mounted by PD at `/mnt/unraid/data`) |
+| immich | 62GB | Immich photo storage (NFS-mounted by PD at `/mnt/unraid/immich`) |
+| nextcloud | 7.4GB | Nextcloud data |
+| Tesla/dashcam | 1.6TB | Dashcam footage |
+| fizzlebang | 17GB | Personal files |
+| isos | 22GB | ISO images |
+| Syncthing | 4.6GB | Syncthing data |
+| manndra | — | Partner's storage |
+| Long and Low Crafts | — | Crafts business files |
+| zfs_backups_array | — | ZFS backup target for cache/appdata |
+
+## Pending
+- Remove dead Unraid-Cloudflared-Tunnel container
+- malcolm pool capacity planning (heavily utilized)
--- a/docs/troubleshooting/KNOWN_QUIRKS.md
+++ b/docs/troubleshooting/KNOWN_QUIRKS.md
@@ -0,0 +1,69 @@
+# Known Quirks
+
+Per-service gotchas that aren't bugs but will bite you if you forget them.
+
+## PlausibleDeniability
+
+### immich-ml
+- Healthcheck has no curl/wget — uses `/proc/net/tcp6` pattern
+- Cache goes to `/mnt/docker-ssd/docker/appdata/immich-ml` (separate from immich-server's appdata)
+
+### rackpeek
+- No curl/wget in image — healthcheck uses `/proc/net/tcp6`
+
+### donetick
+- Env vars alone are insufficient for DB type selection
+- Requires `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml` with full DB config
+- Uses viper config loader — `DT_ENV` controls config file path
+
+### shlink
+- Data directory must be `chmod 777` — runs as non-root user that doesn't match host default ownership
+
+### openwebui
+- DB: user=`openwebui`, db=`openwebui` on shared-postgres
+- If restart-looping with auth errors: container wasn't on `ix-databases_shared-databases` at creation time — must `down && up`
+
+### plex
+- Port `5353/udp` conflicts with system avahi/mDNS — remove from port mappings
+
+### scriberr (shelved)
+- SQLite incompatible with ZFS nfsv4 ACLs (SQLITE_CANTOPEN error 14)
+- Revisit when Postgres support available in CUDA image
+
+### meshmonitor (auto-upgrade)
+- `AUTO_UPGRADE_ENABLED=true` causes crashes on TrueNAS if bind mount source directories don't exist
+- The upgrader container mounts compose dir as `/compose` — relative paths resolve differently
+- Fix: pre-create all bind mount source directories before deploying
+
+### Any container using `down && up` requirement
+- Network attachments happen at container creation, not restart
+- `docker restart` or `docker compose restart` will NOT fix missing network attachments
+- Must use `docker compose --env-file .env down <service> && docker compose --env-file .env up -d <service>`
+
+---
+
+## Serenity
+
+### qBit Mover Script
+- Pauses torrents 0–4 days old before running mover to prevent partial file moves
+- MAM-init script updates MyAnonamouse IP inside qbit container on startup
+
+### Unraid-Cloudflared-Tunnel
+- Dead container, should be removed
+
+---
+
+## N.O.M.A.D.
+
+### NVIDIA Container Toolkit
+- Repo has invalid GPG key — remove `/etc/apt/sources.list.d/*nvidia*` if apt errors
+
+### Pelican / Wings
+- Interactive artisan commands break over SSH (stty issue) — always use inline flags:
+  `--email=x --username=y --password=z --admin=1`
+- Wings CORS derived from `remote:` URL, not `allowed_origins` — keep remote as `https://`
+- `panel.paccoco.com` in `/etc/hosts` → `127.0.0.1` for NAT loopback
+- Wings uses `--ignore-certificate-errors` for local self-signed cert
+
+### Port 8080
+- Occupied by `nomad_admin` container — Wings uses 8443 instead
--- a/docs/troubleshooting/TROUBLESHOOTING.md
+++ b/docs/troubleshooting/TROUBLESHOOTING.md
@@ -0,0 +1,129 @@
+# Troubleshooting Guide
+
+## Permission Denied / Readonly Database / Not Writable
+
+Common causes:
+- App placed on NFS (tank) when it needs local SSD
+- NFS ownership mismatch between host user and container user
+
+Checks:
+1. `docker logs <container> --tail 30`
+2. `ls -la <host-appdata-path>` — check ownership and permissions
+3. `docker run --rm --entrypoint sh <image> -lc 'id'` — check container UID/GID
+
+Fix options:
+- Move appdata to `/mnt/docker-ssd/docker/appdata/<service>` if write-heavy
+- `sudo chmod -R 777 <appdata-path>` for apps with fixed internal user (e.g. shlink)
+- `sudo chown -R <uid>:<gid> <appdata-path>` if UID/GID is known
+
+## SQLite Fails on ZFS with nfsv4 ACLs (SQLITE_CANTOPEN error 14)
+
+Symptom: `unable to open database file: out of memory (14)`
+
+Cause: SQLite cannot acquire POSIX file locks on ZFS datasets configured with `acltype=nfsv4`.
+
+Diagnosis:
+```bash
+sudo zfs get acltype,aclmode,xattr <dataset>
+```
+
+Fix options:
+1. Change acltype: `sudo zfs set acltype=posix <dataset>`
+2. Move the SQLite database to local SSD
+3. Use an app version that supports Postgres instead of SQLite
+
+Note: scriberr CUDA image is shelved for this reason.
+
+## Container Can't Resolve Shared Database Hostname
+
+Symptom: `lookup shared-postgres on 127.0.0.11:53: no such host`
+
+Fix:
+```bash
+docker compose --env-file .env down <service>
+docker compose --env-file .env up -d <service>
+```
+
+Verify network attachment:
+```bash
+docker inspect <container> --format '{{json .NetworkSettings.Networks}}' | tr ',' '\n' | grep -o '"[^"]*"'
+```
+
+## Postgres Auth Fails Over TCP but Works via Socket
+
+Diagnosis:
+```bash
+sudo docker exec shared-postgres psql -h 127.0.0.1 -U <user> -d <db> -W -c "\conninfo"
+```
+
+Fix:
+- Verify pg_hba.conf has `host all all all scram-sha-256`
+- Check DATABASE_URL password matches: `ALTER USER <user> WITH PASSWORD '<pass>';`
+- Avoid special characters in passwords in DATABASE_URLs
+
+## Healthcheck Failing Despite Container Being Healthy
+
+Diagnosis:
+```bash
+sudo docker exec <container> sh -c "which curl; which wget" 2>&1
+sudo docker inspect <container> --format '{{json .State.Health.Log}}' | python3 -m json.tool | tail -20
+```
+
+After fixing the compose file, must do `down && up` (not just restart).
+
+## App Ignores Environment Variables for DB Type
+
+Some apps (e.g. donetick) require a YAML config file for certain settings even when env vars are set.
+
+For donetick, create `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml`:
+```yaml
+database:
+  type: postgres
+  host: shared-postgres
+  port: 5432
+  user: donetick
+  password: <password>
+  name: donetick
+  migration: true
+jwt:
+  secret: <32-byte-hex>
+  session_time: 168h
+  max_refresh: 168h
+server:
+  port: 2021
+  serve_frontend: true
+```
+
+## JWT Invalid / Login Fails After Restart
+
+Cause: JWT secret changed. Fix: open app in incognito or clear cookies/localStorage.
+
+## Port Already In Use
+
+Common offenders:
+- `5353/udp` — held by system avahi/mDNS. Remove from Plex ports.
+- `3306`, `5432`, `6379` — only one DB stack can bind these.
+
+## Auto-Upgrade Crashes on TrueNAS (exit code 128)
+
+Symptom: `error while creating mount source path '/compose/scripts': mkdir /compose: read-only file system`
+
+Cause: Auto-upgrade container runs docker compose from inside a container where the project dir is mounted as `/compose`. Relative bind mount paths like `./scripts` resolve to `/compose/scripts` on the host. TrueNAS root filesystem is read-only so Docker can't auto-create the directory.
+
+Fix: Pre-create the directory manually on the host before starting the stack:
+```bash
+mkdir -p /mnt/docker-ssd/docker/compose/<stack>/scripts
+```
+
+## Databases Stack Network Name Wrong
+
+Must have `name: ix-databases` at top of databases compose. Verify:
+```bash
+docker network ls | grep databases
+# Should show: ix-databases_shared-databases
+```
+
+## Checking Container Runtime UID
+```bash
+docker run --rm --entrypoint sh <image> -lc 'id'
+```