From 370998b8d5eba132e65bee74b33a78fc89148967 Mon Sep 17 00:00:00 2001 From: Paccoco <99041736+Paccoco@users.noreply.github.com> Date: Tue, 5 May 2026 14:43:09 -0500 Subject: [PATCH] docs: add full homelab documentation structure --- README.md | 73 +++++++---- docs/README.md | 0 docs/architecture/ARCHITECTURE_OVERVIEW.md | 107 +++++++++++++++ docs/architecture/HARDWARE_INVENTORY.md | 77 +++++++++++ docs/architecture/NETWORKING_MODEL.md | 56 ++++++++ docs/architecture/SERVICES_DIRECTORY.md | 103 +++++++++++++++ docs/operations/BACKUP_POLICY.md | 42 ++++++ docs/operations/DEPLOYMENT_CHECKLIST.md | 63 +++++++++ docs/operations/DOCKERSTACK_STORAGE_NOTES.md | 61 +++++++++ docs/operations/SECRETS_MANAGEMENT.md | 64 +++++++++ docs/operations/STACK_STANDARDS.md | 122 ++++++++++++++++++ docs/planning/HOMELAB_EXPANSION_PLAN.md | 106 +++++++++++++++ docs/planning/TODO.md | 43 +++++++ docs/reference/HOMELAB_AI_CONTEXT.md | 56 ++++++++ docs/reference/MESHTASTIC.md | 70 ++++++++++ docs/servers/NOMAD.md | 73 +++++++++++ docs/servers/PLAUSIBLEDENABILITY.md | 60 +++++++++ docs/servers/SERENITY.md | 55 ++++++++ docs/troubleshooting/KNOWN_QUIRKS.md | 69 ++++++++++ docs/troubleshooting/TROUBLESHOOTING.md | 129 +++++++++++++++++++ 20 files changed, 1405 insertions(+), 24 deletions(-) create mode 100644 docs/README.md create mode 100644 docs/architecture/ARCHITECTURE_OVERVIEW.md create mode 100644 docs/architecture/HARDWARE_INVENTORY.md create mode 100644 docs/architecture/NETWORKING_MODEL.md create mode 100644 docs/architecture/SERVICES_DIRECTORY.md create mode 100644 docs/operations/BACKUP_POLICY.md create mode 100644 docs/operations/DEPLOYMENT_CHECKLIST.md create mode 100644 docs/operations/DOCKERSTACK_STORAGE_NOTES.md create mode 100644 docs/operations/SECRETS_MANAGEMENT.md create mode 100644 docs/operations/STACK_STANDARDS.md create mode 100644 docs/planning/HOMELAB_EXPANSION_PLAN.md create mode 100644 docs/planning/TODO.md create mode 100644 docs/reference/HOMELAB_AI_CONTEXT.md create mode 100644 docs/reference/MESHTASTIC.md create mode 100644 docs/servers/NOMAD.md create mode 100644 docs/servers/PLAUSIBLEDENABILITY.md create mode 100644 docs/servers/SERENITY.md create mode 100644 docs/troubleshooting/KNOWN_QUIRKS.md create mode 100644 docs/troubleshooting/TROUBLESHOOTING.md diff --git a/README.md b/README.md index ec2612a..f042488 100644 --- a/README.md +++ b/README.md @@ -1,35 +1,60 @@ # truenas-stacks -Docker Compose stacks for PlausibleDeniability (TrueNAS Scale). +Docker Compose stacks for PlausibleDeniability (TrueNAS Scale), plus documentation for the full homelab. -## Stacks +## Servers -- databases/ - shared-postgres, shared-mariadb, shared-redis - deploy FIRST -- infrastructure/ - newt, homepage, uptime-kuma, dockhand, netdata, tailscale - deploy SECOND -- media/ - plex, tautulli, audiobookshelf, seerr, calibre-web -- photos/ - immich server + machine learning -- home/ - kitchenowl, donetick, dakboard bridge, shlink, qui -- dev/ - gitea, rackpeek -- ai/ - ollama, openwebui, scriberr +| Server | Role | OS | IP | +|--------|------|----|----| +| Serenity | NAS / Media ingestion | Unraid 7.2.4 | 10.5.1.5 | +| PlausibleDeniability | Primary Docker host | TrueNAS Scale 25.10.2.1 | LAN DHCP | +| N.O.M.A.D. | Offline knowledge + game servers | Ubuntu 25.10 | 10.5.1.16 | -## Deployment order +## Repo Layout -1. databases -2. infrastructure -3. All other stacks in any order +``` +databases/ # shared-postgres, shared-mariadb, shared-redis — deploy FIRST +infrastructure/ # newt, homepage, uptime-kuma, dockhand, netdata, tailscale — deploy SECOND +media/ # plex, tautulli, audiobookshelf, seerr, calibre-web-automated +photos/ # immich server + machine learning +home/ # kitchenowl, donetick, shlink, qui, dakboard bridges +dev/ # gitea, rackpeek +ai/ # ollama, openwebui +meshtastic/ # meshmonitor, mqtt-proxy, tileserver, meshtastic-map +automation/ # planned: gotify, n8n +search/ # planned: qdrant +docs/ # all homelab documentation +``` -## Setup on a new machine +## Quick Start - git clone git@github.com:Paccoco/truenas-stacks.git /mnt/docker-ssd/docker/compose - cp databases/.env.example databases/.env && nano databases/.env - # repeat for each stack, then import each truenas-app.yaml via TrueNAS Apps UI +1. Deploy `databases/` first +2. Deploy `infrastructure/` second (creates pangolin network via newt) +3. Deploy remaining stacks in any order -## Networks +```bash +cd /mnt/docker-ssd/docker/compose/ +docker compose --env-file .env config # validate +docker compose --env-file .env up -d # deploy +``` -- ix-databases_shared-databases: created by databases stack, used by media/photos/home/dev/ai -- pangolin: created by newt at runtime, used by any service exposed via reverse proxy +## Documentation -## Secrets - -.env files are gitignored. .env.example files are committed with placeholder values. -Real .env files live only on TrueNAS at /mnt/docker-ssd/docker/compose//.env +- [Architecture Overview](docs/architecture/ARCHITECTURE_OVERVIEW.md) +- [Networking Model](docs/architecture/NETWORKING_MODEL.md) +- [Hardware Inventory](docs/architecture/HARDWARE_INVENTORY.md) +- [Services Directory](docs/architecture/SERVICES_DIRECTORY.md) +- [Stack Standards](docs/operations/STACK_STANDARDS.md) +- [Deployment Checklist](docs/operations/DEPLOYMENT_CHECKLIST.md) +- [Storage Notes](docs/operations/DOCKERSTACK_STORAGE_NOTES.md) +- [Backup Policy](docs/operations/BACKUP_POLICY.md) +- [Secrets Management](docs/operations/SECRETS_MANAGEMENT.md) +- [Troubleshooting](docs/troubleshooting/TROUBLESHOOTING.md) +- [Known Quirks](docs/troubleshooting/KNOWN_QUIRKS.md) +- [Expansion Plan](docs/planning/HOMELAB_EXPANSION_PLAN.md) +- [TODO](docs/planning/TODO.md) +- [Serenity](docs/servers/SERENITY.md) +- [PlausibleDeniability](docs/servers/PLAUSIBLEDENABILITY.md) +- [N.O.M.A.D.](docs/servers/NOMAD.md) +- [Meshtastic Stack](docs/reference/MESHTASTIC.md) +- [AI Context](docs/reference/HOMELAB_AI_CONTEXT.md) diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/architecture/ARCHITECTURE_OVERVIEW.md b/docs/architecture/ARCHITECTURE_OVERVIEW.md new file mode 100644 index 0000000..4847746 --- /dev/null +++ b/docs/architecture/ARCHITECTURE_OVERVIEW.md @@ -0,0 +1,107 @@ +# Homelab Architecture Overview + +## Primary Docker Host +Host: `PlausibleDeniability` (TrueNAS Scale) + +Production Docker workloads run on PlausibleDeniability via standalone Docker Compose stacks. +TrueNAS Apps (ix-app format) have been fully migrated to compose stacks. + +## Repository Layout +Repo root: `/mnt/docker-ssd/docker/compose` +Remote: `git@github.com:Paccoco/truenas-stacks.git` +Gitea mirror: `https://gitea.paccoco.com/fizzlepoof/truenas-stacks.git` + +Stack directories: +- `databases/` = shared-postgres, shared-mariadb, shared-redis — deploy FIRST +- `infrastructure/` = newt, homepage, uptime-kuma, dockhand, netdata, tailscale — deploy SECOND +- `media/` = plex, tautulli, audiobookshelf, seerr, calibre-web-automated +- `photos/` = immich server + machine learning +- `home/` = kitchenowl, donetick, dakboard bridge, shlink, shlink-web-client, qui +- `dev/` = gitea, rackpeek +- `ai/` = ollama, openwebui +- `meshtastic/` = meshmonitor, mqtt-proxy, tileserver, meshtastic-map + +## Storage Model + +### Local SSD +Mounted at: `/mnt/docker-ssd` +- Docker data root: `/mnt/docker-ssd/docker/docker-data` +- Databases: `/mnt/docker-ssd/docker/databases/` +- Write-heavy appdata: `/mnt/docker-ssd/docker/appdata/` +- GPU/model storage +- Apps that perform chown or aggressive startup writes +- SQLite-backed apps + +Notable appdata on SSD: +- plex, tautulli, audiobookshelf, seerr, dockhand, rackpeek +- immich, immich-ml, ollama, openwebui +- shlink, gitea, netdata, tailscale, uptime-kuma + +### TrueNAS Pool (tank) +Mounted at: `/mnt/tank` +- General appdata: `/mnt/tank/docker/appdata/` +- Backups: `/mnt/tank/docker/backups/` +- Downloads: `/mnt/tank/docker/downloads/` + +Notable appdata on tank: +- homepage, newt, dockhand (config only) +- kitchenowl, donetick, qui +- calibre-web-automated config + +### Unraid NFS +Mounted at: `/mnt/unraid` +- Media libraries: `/mnt/unraid/data/media/` +- Immich media: `/mnt/unraid/immich` + +## Networks +- `ix-databases_shared-databases` — created by databases stack; project name `ix-databases` set in compose +- `pangolin` — created by newt at runtime; used by all externally exposed services + +## Active Stacks (as of 2026-05-05) + +### databases/ +- shared-postgres (immich-optimized image with vectorchord + pgvector) +- shared-mariadb +- shared-redis + +### infrastructure/ +- newt, homepage, dockhand, uptime-kuma, netdata, tailscale + +### media/ +- ix-plex-plex-1 (GPU passthrough, RTX 2080 Ti) +- ix-tautulli-tautulli-1 +- ix-audiobookshelf-audiobookshelf-1 +- seerr (Postgres backend) +- calibre-web-automated + +### photos/ +- ix-immich-immich-server-1 +- ix-immich-immich-machine-learning-1 (GPU, CUDA) + +### home/ +- kitchenowl, donetick, donetick-dakboard-bridge, qui, shlink, shlink-web-client + +### dev/ +- gitea, ix-rackpeek-rackpeek-1 + +### ai/ +- ollama (GPU, RTX 2080 Ti, port 11434) +- openwebui (Postgres backend, port 8282) + +### meshtastic/ +- meshmonitor, meshmonitor-mqtt-proxy, meshmonitor-tileserver, meshtastic-map + +## Planned / Not Yet Deployed +- dispatcharr, gotify, bytestash, kima-hub, n8n, qdrant, paperless-ngx, home-assistant + +## Shelved +- scriberr — SQLite incompatible with ZFS nfsv4 ACLs + +## Design Principles +- deterministic storage paths +- no anonymous volumes +- stack-local .env for secrets; .env.example committed with placeholders +- SSD for write-heavy / DB / SQLite / GPU workloads +- tank NFS for general appdata and configs +- Unraid NFS for media only +- databases stack uses `name: ix-databases` to maintain network name compatibility diff --git a/docs/architecture/HARDWARE_INVENTORY.md b/docs/architecture/HARDWARE_INVENTORY.md new file mode 100644 index 0000000..d377dea --- /dev/null +++ b/docs/architecture/HARDWARE_INVENTORY.md @@ -0,0 +1,77 @@ +# Hardware Inventory + +## Serenity (NAS / Media Ingestion) +- **OS:** Unraid 7.2.4 +- **IP:** 10.5.1.5 | Tailscale: 100.94.87.79 +- **CPU:** Xeon X5690 +- **RAM:** 102GB +- **GPU:** None (Matrox IPMI only) +- **UPS:** None + +### Storage +| Pool | Type | Drives | Usable | Notes | +|------|------|--------|--------|-------| +| malcolm | RAIDZ1 | 6× 10TB HDD | ~40TB | Heavily utilized | +| cache | ZFS single | 1× Kingston SNV2S 1TB NVMe | ~1TB | Appdata/Docker | + +### Key Shares +| Share | Size | Purpose | +|-------|------|---------| +| data | 35TB | Media library | +| immich | 62GB | Immich photo storage | +| nextcloud | 7.4GB | Nextcloud data | +| Tesla/dashcam | 1.6TB | Dashcam footage | +| fizzlebang | 17GB | Personal files | +| isos | 22GB | ISO images | +| Syncthing | 4.6GB | Syncthing data | +| manndra | — | Partner's storage | +| Long and Low Crafts | — | Crafts business | +| zfs_backups_array | — | ZFS backup target | + +--- + +## PlausibleDeniability (Primary Docker Host) +- **OS:** TrueNAS Scale 25.10.2.1 +- **CPU:** Ryzen 9 3950X +- **RAM:** 32GB +- **GPU:** Nvidia RTX 2080 Ti (11GB VRAM) — Plex, Immich ML, Ollama +- **UPS:** None + +### Storage +| Pool | Type | Mount | Notes | +|------|------|-------|-------| +| tank | ZFS mirror (2 drives) | `/mnt/tank` | ~3.5TB usable | +| docker-ssd | Single Crucial MX500 500GB | `/mnt/docker-ssd` | No redundancy | +| boot-pool | Single sdc3 | — | TrueNAS boot | + +### NFS Mounts +- Serenity data array → `/mnt/unraid/data` + +--- + +## N.O.M.A.D. (Offline Knowledge + Game Servers) +- **OS:** Ubuntu 25.10 (bare metal) +- **IP:** 10.5.1.16 +- **CPU:** Intel Core i7-4770K @ 3.50GHz (4c/8t, up to 3.9GHz) +- **RAM:** 32GB DDR3 1600MHz (4×8GB) +- **GPU:** NVIDIA GeForce GTX 1080 (8GB VRAM) +- **NIC:** Intel I218-V (1GbE) + Aquantia AQC113 (10GbE) +- **Motherboard:** ASRock H97M Pro4 +- **UPS:** None + +### Storage +| Device | Size | Mount | Purpose | +|--------|------|-------|---------| +| Samsung SSD 840 (sdb) | 232.9GB | `/` (LVM 100GB) | OS boot | +| WD Red HDD (sda) | 3.6TB | `/mnt/hdd-1` | Primary data, project-nomad symlink | +| WD Red HDD (sdc) | 3.6TB | `/mnt/hdd-2` | Docker data-root | +| Seagate SSD (sdd) | 223.6GB | `/mnt/ssd-1` | Fast scratch | + +--- + +## Desktop: ROCINANTE +- **OS:** Windows 11 Pro +- **CPU:** Ryzen 7 9850X3D +- **RAM:** 64GB +- **GPU:** RTX 4090 (24GB VRAM) +- **Role:** Heavy AI inference, local Ollama (expose via `OLLAMA_HOST=0.0.0.0`) diff --git a/docs/architecture/NETWORKING_MODEL.md b/docs/architecture/NETWORKING_MODEL.md new file mode 100644 index 0000000..6219ee7 --- /dev/null +++ b/docs/architecture/NETWORKING_MODEL.md @@ -0,0 +1,56 @@ +# Homelab Networking Model + +## Physical Network +- **Gear:** Unifi throughout +- **Main LAN:** 10.5.1.x (Trusted VLAN) +- **Other VLANs:** Cameras, Guest, IoT +- **Serenity IP:** 10.5.1.5 +- **N.O.M.A.D. IP:** 10.5.1.16 +- **PlausibleDeniability:** DHCP on LAN + +## Remote Access +- **Pangolin VPS + Newt:** Services exposed as subdomains via reverse proxy. Newt agents run on PD and N.O.M.A.D. +- **Tailscale:** Personal remote access, runs on Serenity (100.94.87.79) +- **Cloudflared:** No longer used. Dead container on Serenity to be removed. + +## DNS +- **Provider:** Cloudflare +- All public subdomains point at Pangolin VPS IP + +## Known Public Subdomains +- `paccoco.com` → Pangolin VPS +- `panel.paccoco.com` → Pelican Panel (N.O.M.A.D.) +- `node1.paccoco.com` → Wings daemon (N.O.M.A.D., no auth) +- `meshmonitor.paccoco.com` → Meshmonitor (PD) +- `gitea.paccoco.com` → Gitea (PD) +- `ai.paccoco.com` → OpenWebUI (PD) — DNS/Pangolin resource pending + +## Docker Networking (PlausibleDeniability) + +### ix-databases_shared-databases +Created by the `databases` stack. The databases compose file uses `name: ix-databases` so Docker names the network `ix-databases_shared-databases`, matching what all other stacks declare as external. + +Any stack connecting to shared-postgres, shared-mariadb, or shared-redis must declare: +```yaml +networks: + ix-databases_shared-databases: + external: true +``` + +### pangolin +Created at runtime by the `newt` container in the infrastructure stack. Any service needing reverse-proxy exposure must declare: +```yaml +networks: + pangolin: + external: true +``` + +### Internal Service Model +Internal services run HTTP only behind the reverse proxy. Compose service names used for internal DNS between services in the same stack/network. + +## N.O.M.A.D. Networking Notes +- `panel.paccoco.com` added to `/etc/hosts` → `127.0.0.1` for NAT loopback +- nginx listens on 80 and 443 (self-signed cert) for Wings → Panel communication +- Wings uses `--ignore-certificate-errors` for local loopback +- Pangolin resource `node1.paccoco.com` has auth disabled (Wings handles its own auth) +- Minecraft port 25565 TCP/UDP goes directly via home router port forward, not through Pangolin diff --git a/docs/architecture/SERVICES_DIRECTORY.md b/docs/architecture/SERVICES_DIRECTORY.md new file mode 100644 index 0000000..56929c0 --- /dev/null +++ b/docs/architecture/SERVICES_DIRECTORY.md @@ -0,0 +1,103 @@ +# Services Directory + +All running services across the homelab with ports, URLs, and backend info. + +## PlausibleDeniability + +### Databases (internal only) +| Service | Port | Backend | Notes | +|---------|------|---------|-------| +| shared-postgres | 5432 | — | immich-optimized, vectorchord + pgvector | +| shared-mariadb | 3306 | — | | +| shared-redis | 6379 | — | | + +### Infrastructure +| Service | Internal Port | Public URL | Notes | +|---------|--------------|------------|-------| +| newt | — | — | Pangolin reverse proxy agent | +| homepage | — | — | Dashboard | +| dockhand | — | — | Container management | +| uptime-kuma | — | — | MariaDB backend | +| netdata | — | — | Metrics | +| tailscale | — | — | VPN | + +### Media +| Service | Internal Port | Public URL | Notes | +|---------|--------------|------------|-------| +| plex | — | — | GPU passthrough, RTX 2080 Ti | +| tautulli | — | — | Plex analytics | +| audiobookshelf | — | — | Audiobooks | +| seerr | — | — | Postgres backend | +| calibre-web-automated | — | — | Ebooks | + +### Photos +| Service | Internal Port | Public URL | Notes | +|---------|--------------|------------|-------| +| immich-server | — | — | Postgres + Redis, GPU | +| immich-ml | — | — | GPU, CUDA | + +### Home +| Service | Internal Port | Public URL | Notes | +|---------|--------------|------------|-------| +| kitchenowl | — | — | Grocery/tasks | +| donetick | — | — | Postgres backend, selfhosted.yaml required | +| shlink | — | — | URL shortener, Postgres | +| shlink-web-client | — | — | | +| qui | — | — | qBittorrent UI, manages Serenity remotely | + +### Dev +| Service | Internal Port | Public URL | Notes | +|---------|--------------|------------|-------| +| gitea | — | gitea.paccoco.com | Postgres backend | +| rackpeek | — | — | | + +### AI +| Service | Internal Port | Public URL | Notes | +|---------|--------------|------------|-------| +| ollama | 11434 | — | GPU, RTX 2080 Ti | +| openwebui | 8282→8080 | ai.paccoco.com (pending) | Postgres backend | + +### Meshtastic +| Service | Internal Port | Public URL | Notes | +|---------|--------------|------------|-------| +| meshmonitor | 8081→3001 | meshmonitor.paccoco.com | Postgres backend | +| meshmonitor virtual node | 4404 | — | TCP virtual node server | +| meshmonitor-tileserver | 8082→8080 | — | Map tiles | +| meshmonitor-mqtt-proxy | — | — | MQTT→TCP bridge | + +--- + +## Serenity + +### Media Ingestion +| Service | Notes | +|---------|-------| +| sonarr | TV | +| sonarr-anime | Anime | +| radarr | Movies | +| lidarr | Music | +| readarr | Books | +| readarr-epub | EPUBs | +| prowlarr | Indexer manager | +| bazarr | Subtitles | +| unpackerr | Archive extraction | +| qBittorrent | Download client | +| GluetunVPN | VPN tunnel for qbit | +| autobrr | Auto-grab | +| qbit_manage | qBit maintenance | +| Notifiarr | Notifications | +| Wizarr | User invites | +| romm | ROM manager | +| GameVault | Game library | +| shelfmark | Book/audiobook search, port 8084 | + +--- + +## N.O.M.A.D. + +| Service | Port | Notes | +|---------|------|-------| +| Pelican Panel | 80/443 | panel.paccoco.com, PHP 8.4 + nginx + MariaDB | +| Wings | 8443 | node1.paccoco.com, game server daemon | +| Minecraft | 25565 | Purpur, Java 21, direct port forward | +| N.O.M.A.D. containers | — | 10 containers, start_nomad.sh / stop_nomad.sh | diff --git a/docs/operations/BACKUP_POLICY.md b/docs/operations/BACKUP_POLICY.md new file mode 100644 index 0000000..801256e --- /dev/null +++ b/docs/operations/BACKUP_POLICY.md @@ -0,0 +1,42 @@ +# Backup Policy + +## Current State (Known Gaps) +- **No off-site backup** — this is a known gap, improvement pending +- Serenity has local ZFS snapshots and replication +- PlausibleDeniability has no automated backup of appdata or databases + +## Serenity Backup Strategy +- Sanoid snapshots cache/appdata: 7 daily, 4 weekly, 6 monthly +- Syncoid replicates cache/appdata → malcolm/zfs_backups_array (strict-mirror) +- qBit Mover script: pauses torrents 0–4 days old then runs Unraid mover +- Mover Status script: sends Discord webhook notifications + +## PlausibleDeniability Backup Scope +Back up at minimum: +- Compose repo: `/mnt/docker-ssd/docker/compose` (in git — GitHub + Gitea) +- Local SSD appdata: `/mnt/docker-ssd/docker/appdata/` +- Databases: `/mnt/docker-ssd/docker/databases/` +- Selected appdata/backups on tank: `/mnt/tank/docker/` +- .env files (separately — gitignored, handle via encrypted backup or private repo) + +## Database Backups +- Keep app-consistent dumps for major databases +- Do not rely only on raw volume copies for restores +- Document restore procedure per major service + +Recommended dump locations: `/mnt/tank/docker/backups/db-dumps/` + +## .env File Backup Options +1. **Private Gitea repo** — short term, push all .env files to a private repo on your local Gitea +2. **Encrypted restic** — include .env files in restic backup with encryption, push offsite (Backblaze B2) +3. **Vaultwarden** — store each .env as a secure note (best long-term once deployed) + +## Restore Practice +- Test restores quarterly +- Test at least one app DB restore and one full-config restore +- Verify that restored stacks can start with current compose files + +## Operational Notes +- Media libraries on Unraid are large and may not need the same backup frequency as configs/databases +- SSD-local DB and SQLite workloads are highest priority for backup validation +- Serenity malcolm pool is heavily utilized — capacity planning needed before it fills diff --git a/docs/operations/DEPLOYMENT_CHECKLIST.md b/docs/operations/DEPLOYMENT_CHECKLIST.md new file mode 100644 index 0000000..74657d3 --- /dev/null +++ b/docs/operations/DEPLOYMENT_CHECKLIST.md @@ -0,0 +1,63 @@ +# New Service Deployment Checklist + +## Before Adding a Service + +- [ ] Determine DB requirements (postgres, mariadb, sqlite, none) +- [ ] Decide storage tier: SSD (`/mnt/docker-ssd/docker/appdata/`) vs tank (`/mnt/tank/docker/appdata/`) vs Unraid +- [ ] Check if service needs `ix-databases_shared-databases` network +- [ ] Check if service needs `pangolin` network for reverse proxy exposure + +## Scaffold + +- [ ] Create host directories for appdata +- [ ] Set correct ownership if needed (`chown` or `chmod 777` for permission-sensitive apps) +- [ ] Create stack folder under `/mnt/docker-ssd/docker/compose//` +- [ ] Create `docker-compose.yaml` +- [ ] Create `.env` and `.env.example` +- [ ] If using postgres: create DB and user, or add init SQL to `initdb/` +- [ ] If using mariadb: add init script to `initdb-mariadb/` + +## Validate Before Deploying + +```bash +cd /mnt/docker-ssd/docker/compose/ +docker compose --env-file .env config +``` + +No errors = safe to proceed. + +## Deploy + +```bash +docker compose --env-file .env up -d +``` + +## After Starting + +- [ ] `docker ps` shows container running +- [ ] `docker logs --tail 20` shows clean startup +- [ ] `docker inspect ` confirms mounts are correct +- [ ] Data persists after `docker compose --env-file .env down && up -d` + +## Network Checklist + +If service needs shared databases: +- Top-level networks block declares `ix-databases_shared-databases` as external +- Service networks list includes `ix-databases_shared-databases` +- Verify: `docker inspect --format '{{json .NetworkSettings.Networks}}'` + +If service needs reverse proxy: +- Top-level networks block declares `pangolin` as external +- Service networks list includes `pangolin` +- Infrastructure stack (newt) must be running first + +## Common Failure Modes + +| Symptom | Likely Cause | Fix | +|---|---|---| +| `hostname resolving error` for shared-postgres | Not on `ix-databases_shared-databases` network | `down && up` not just restart | +| `not writable` errors on startup | NFS ownership mismatch | Move to SSD or chmod 777 | +| `unsupported database type` | App requires config file, not just env vars | Create `selfhosted.yaml` or equivalent | +| `JWT is invalid` after restart | JWT secret changed | Clear browser cookies/localStorage | +| Port conflict on startup | Port already bound | Remove conflicting port mapping | +| Auto-upgrade crashes on TrueNAS | Bind mount source path missing, Docker tries to create at root | Pre-create the directory manually | diff --git a/docs/operations/DOCKERSTACK_STORAGE_NOTES.md b/docs/operations/DOCKERSTACK_STORAGE_NOTES.md new file mode 100644 index 0000000..3113eb8 --- /dev/null +++ b/docs/operations/DOCKERSTACK_STORAGE_NOTES.md @@ -0,0 +1,61 @@ +# Dockerstack Storage Notes + +## Local SSD +Use `/docker/...` for: +- Docker root (`/docker/docker-data`) +- GPU/model storage +- SQLite-backed apps +- Apps that chown on startup +- Write-heavy DB/search storage + +Examples: +- `/mnt/docker-ssd/docker/docker-data` +- `/mnt/docker-ssd/docker/appdata/ollama` +- `/mnt/docker-ssd/docker/appdata/openwebui` +- `/mnt/docker-ssd/docker/appdata/immich` +- `/mnt/docker-ssd/docker/appdata/immich-ml` +- `/mnt/docker-ssd/docker/appdata/plex` +- `/mnt/docker-ssd/docker/appdata/tautulli` +- `/mnt/docker-ssd/docker/appdata/audiobookshelf` +- `/mnt/docker-ssd/docker/appdata/seerr` +- `/mnt/docker-ssd/docker/appdata/dockhand` +- `/mnt/docker-ssd/docker/appdata/rackpeek` +- `/mnt/docker-ssd/docker/appdata/shlink` +- `/mnt/docker-ssd/docker/appdata/gitea` +- `/mnt/docker-ssd/docker/appdata/netdata` +- `/mnt/docker-ssd/docker/appdata/tailscale` +- `/mnt/docker-ssd/docker/appdata/uptime-kuma` +- `/mnt/docker-ssd/docker/databases/postgres` +- `/mnt/docker-ssd/docker/databases/mariadb` +- `/mnt/docker-ssd/docker/databases/redis` + +## TrueNAS NFS +Use `/mnt/tank/docker/...` for: +- Normal appdata +- Uploads +- Non-fussy config and data +- Backups +- Downloads + +Examples: +- `/mnt/tank/docker/appdata/homepage` +- `/mnt/tank/docker/appdata/newt` +- `/mnt/tank/docker/appdata/kitchenowl` +- `/mnt/tank/docker/appdata/donetick` +- `/mnt/tank/docker/appdata/qui` +- `/mnt/tank/docker/appdata/calibre-web-automated` +- `/mnt/tank/docker/backups` +- `/mnt/tank/docker/downloads` + +## Unraid NFS +Use `/mnt/unraid/...` for media and shared bulk content only: +- `/mnt/unraid/data/media/` — Plex libraries, books, etc. +- `/mnt/unraid/immich` — Immich photo/video storage + +## Rules Learned +- SQLite apps and apps that write logs aggressively should prefer local SSD. +- Database/search/model-heavy workloads should prefer local SSD. +- General config and less write-sensitive appdata can live on TrueNAS NFS. +- Media should stay on Unraid mounts. +- SQLite on ZFS with nfsv4 ACLs will always fail — move to SSD or use Postgres. +- Apps with auto-upgrade features that use bind mounts: pre-create all source directories manually on TrueNAS to prevent Docker from trying to create them at the read-only root. diff --git a/docs/operations/SECRETS_MANAGEMENT.md b/docs/operations/SECRETS_MANAGEMENT.md new file mode 100644 index 0000000..f85512c --- /dev/null +++ b/docs/operations/SECRETS_MANAGEMENT.md @@ -0,0 +1,64 @@ +# Secrets Management + +## Principles +- `.env` files are gitignored and never committed to any public repo +- `.env.example` files are committed with placeholder values +- Real `.env` files live only on the host at the stack directory +- Secrets are generated with standard tools, never reused across services + +## Generating Secrets + +```bash +# Database passwords +openssl rand -hex 24 + +# JWT / session secrets +openssl rand -hex 32 + +# API keys (where format allows) +openssl rand -base64 32 +``` + +## .env File Locations +All `.env` files live alongside their `docker-compose.yaml`: +``` +/mnt/docker-ssd/docker/compose//.env +``` + +## Backing Up .env Files + +### Short term: Private Gitea repo +Create a private repo on Gitea (not GitHub) and push all .env files there: +```bash +# In a separate directory, not the main stacks repo +git init homelab-secrets +cd homelab-secrets +# Copy and add .env files +git remote add origin https://gitea.paccoco.com/fizzlepoof/homelab-secrets.git +git push -u origin main +``` + +### Long term: Restic + offsite +Include the compose directory (with .env files) in a restic backup: +```bash +restic -r backup /mnt/docker-ssd/docker/compose --exclude="*.example" +``` + +### Best long term: Vaultwarden +Once deployed, store each service's .env as a secure note in Vaultwarden. Single source of truth, accessible from anywhere. + +## Known Credentials Locations + +| Service | Where stored | +|---------|-------------| +| shared-postgres | `databases/.env` | +| shared-mariadb | `databases/.env` | +| OpenWebUI DB | `ai/.env` — user: openwebui, db: openwebui | +| Donetick | `home/.env` + `selfhosted.yaml` | +| Meshmonitor | `meshtastic/.env` | +| Gitea | `dev/.env` | + +## Security Reminders +- Regenerate Wings token on N.O.M.A.D. (was exposed in chat — TODO) +- Regenerate N.O.M.A.D. Newt secret (was exposed in chat — TODO) +- Disable signups in OpenWebUI after creating initial account diff --git a/docs/operations/STACK_STANDARDS.md b/docs/operations/STACK_STANDARDS.md new file mode 100644 index 0000000..5a76ba1 --- /dev/null +++ b/docs/operations/STACK_STANDARDS.md @@ -0,0 +1,122 @@ +# Homelab Stack Standards + +## Repository Root +`/mnt/docker-ssd/docker/compose` + +Remotes: +- GitHub: `git@github.com:Paccoco/truenas-stacks.git` +- Gitea: `https://gitea.paccoco.com/fizzlepoof/truenas-stacks.git` + +## Stack Layout +Each production stack lives in a subdirectory of the repo root. + +Each stack contains: +- `docker-compose.yaml` — required +- `.env` — required (gitignored) +- `.env.example` — committed with placeholder values +- optional init scripts (e.g. `initdb/`, `initdb-mariadb/`) + +## Deployment Order +1. `databases/` — must be up before anything else +2. `infrastructure/` — newt creates the pangolin network +3. All other stacks in any order + +## Deployment Workflow + +Validate before deploying: +```bash +cd /mnt/docker-ssd/docker/compose/ +docker compose --env-file .env config +``` + +Deploy: +```bash +docker compose --env-file .env up -d +``` + +Teardown a single service: +```bash +docker compose --env-file .env down +``` + +## Networks + +### ix-databases_shared-databases +Created by the `databases` stack with `name: ix-databases` at top level. Any stack connecting to shared databases must declare: +```yaml +networks: + ix-databases_shared-databases: + external: true +``` + +### pangolin +Created by `newt` at runtime. Any externally exposed service must declare: +```yaml +networks: + pangolin: + external: true +``` + +## Storage Standards + +### Local SSD: `/mnt/docker-ssd` +Use `/mnt/docker-ssd/docker/appdata/` for: +- Postgres / MariaDB / Redis data +- SQLite-backed apps +- Apps that chown aggressively on startup +- GPU/model caches +- Write-heavy logs, cache, state + +### TrueNAS Pool (tank): `/mnt/tank` +Use `/mnt/tank/docker/appdata/` for: +- General appdata +- Configs +- Uploads +- Non-permission-sensitive state + +### Unraid NFS: `/mnt/unraid` +- `/mnt/unraid/data/media/` — media libraries +- `/mnt/unraid/immich` — Immich photo/video storage + +## Volume Rules +- no anonymous volumes +- no ad-hoc home directory paths +- all binds use standardized production paths +- evaluate SSD vs tank vs Unraid before deploying any new service +- apps known to have permission issues on NFS go on SSD + +## Known Permission Quirks +- `shlink` data directory must be chmod 777 — runs as non-root user +- `immich-ml` cache goes to `/mnt/docker-ssd/docker/appdata/immich-ml` (separate from immich-server) +- `donetick` requires `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml` — env vars alone insufficient + +## Secrets Management +- `.env` files are gitignored +- `.env.example` files are committed with placeholder values +- Database passwords: `openssl rand -hex 24` +- JWT secrets: `openssl rand -hex 32` + +## Healthcheck Patterns +Always check available tools before writing healthchecks: +```bash +sudo docker exec sh -c "which curl; which wget" 2>&1 +``` + +- Has curl: `["CMD-SHELL", "curl -sf http://127.0.0.1:/ || exit 1"]` +- Has wget only: `["CMD-SHELL", "wget -qO- http://127.0.0.1:/ >/dev/null 2>&1 || exit 1"]` +- Has neither: `["CMD-SHELL", "cat /proc/net/tcp6 | grep -q || exit 1"]` +- Redirecting endpoint: `["CMD-SHELL", "curl -sfL http://127.0.0.1:/ || exit 1"]` + +Common hex port values: 8080=1F90, 3003=BBB, 3000=BB8, 8181=1FF5, 5055=13BF, 2021=7E5 + +After changing a healthcheck, must `down` then `up` (not just restart). + +## Git Workflow +```bash +cd /mnt/docker-ssd/docker/compose +git add . +git commit -m "description" +git push # pushes to both GitHub and Gitea +``` + +All `.env` files are gitignored. Only compose files, `.env.example`, and init scripts are committed. diff --git a/docs/planning/HOMELAB_EXPANSION_PLAN.md b/docs/planning/HOMELAB_EXPANSION_PLAN.md new file mode 100644 index 0000000..ed3bea7 --- /dev/null +++ b/docs/planning/HOMELAB_EXPANSION_PLAN.md @@ -0,0 +1,106 @@ +# Homelab Expansion Plan + +## Deployment Order +1. **Gotify** — unified notification bus, unblocks all n8n workflows +2. **Qdrant** — vector DB, unblocks proper RAG pipeline +3. **n8n** — automation backbone, ties everything together +4. **Paperless-NGX** — document intelligence +5. **Home Assistant** — home automation layer +6. **Grafana + Prometheus** — unified observability + +--- + +## Phase 1 — Notifications (Gotify) +**Host:** PlausibleDeniability +**Stack:** `/mnt/docker-ssd/docker/compose/automation/` +**Storage:** appdata on SSD (SQLite backend) +**Network:** pangolin (expose as `gotify.paccoco.com`) + +Single output target for all n8n workflows and homelab alerts. + +--- + +## Phase 2 — Vector Database (Qdrant) +**Host:** PlausibleDeniability +**Stack:** `/mnt/docker-ssd/docker/compose/ai/` +**Storage:** appdata on SSD + +Feed it: homelab markdown docs, Gitea commit history, Plex/Tautulli metadata, Obsidian notes. + +--- + +## Phase 3 — Workflow Automation (n8n) +**Host:** PlausibleDeniability +**Stack:** `/mnt/docker-ssd/docker/compose/automation/` +**Storage:** config on tank, Postgres backend +**Network:** pangolin (`n8n.paccoco.com`) + ix-databases_shared-databases + +### Planned Workflows +- Sonarr/Radarr webhook → Discord + Gotify notification with poster art +- Tautulli play start → log to Postgres +- Nightly watch history → Ollama → weekly digest via Gotify +- Poll Uptime Kuma → cross-reference Netdata → rich alert via Gotify +- Gitea commit to truenas-stacks → Gotify notification with Ollama summary +- qBittorrent torrent complete → trigger Sonarr/Radarr rescan +- KitchenOwl grocery list change → shopping list via Gotify +- Donetick task due → Gotify reminder +- Long and Low Crafts order → Donetick task + Gotify DM + +--- + +## Phase 4 — Document Intelligence (Paperless-NGX) +**Host:** PlausibleDeniability +**Network:** pangolin (`paperless.paccoco.com`) + ix-databases_shared-databases + +OCR and full-text search. Key for Long and Low Crafts invoices, receipts, and manuals. + +--- + +## Phase 5 — Home Automation (Home Assistant) +**Host:** PlausibleDeniability or N.O.M.A.D. +**Storage:** config on SSD (SQLite, write-heavy) +**Network:** host networking recommended for device discovery + +--- + +## Phase 6 — Observability (Grafana + Prometheus) +**Host:** N.O.M.A.D. +**Storage:** Prometheus data on hdd-1 or hdd-2 + +Metrics to scrape: ZFS pool health, container CPU/RAM, Plex streams, qBittorrent, Tailscale latency, Netdata endpoints. + +--- + +## Multi-Model Routing Strategy + +| Machine | GPU | Model | Use Case | +|---|---|---|---| +| ROCINANTE | RTX 4090 (24GB) | qwen3:32b / deepseek-r1:32b | Heavy reasoning, long context | +| PlausibleDeniability | RTX 2080 Ti (11GB) | qwen2.5:14b | General homelab assistant, RAG | +| N.O.M.A.D. | GTX 1080 (8GB) | phi4 / llama3.2:3b | Fast local inference | + +--- + +## Additional Tools +- **Continue.dev** — VS Code plugin, local Copilot alternative +- **Obsidian + Gitea Sync** — personal notes → RAG pipeline +- **Homepage Ollama Widget** — VRAM usage dashboard widget +- **kima-hub** — self-hosted Spotify alternative +- **dispatcharr** — planned +- **bytestash** — planned + +--- + +## Target Architecture + +``` +Triggers (Sonarr, Radarr, Tautulli, Gitea, Home Assistant, schedules, email) + ↓ + n8n (routing, logic, orchestration) + ↓ + Ollama (intelligence — model selected by complexity) + ↓ ↑ + Qdrant (semantic context retrieval) + ↓ + Outputs (Gotify → phone, Donetick tasks, Postgres logging, Discord) +``` diff --git a/docs/planning/TODO.md b/docs/planning/TODO.md new file mode 100644 index 0000000..aa19c60 --- /dev/null +++ b/docs/planning/TODO.md @@ -0,0 +1,43 @@ +# Homelab TODO + +## Security (do ASAP) +- [ ] Regenerate Wings token on N.O.M.A.D. (was exposed in chat) +- [ ] Regenerate N.O.M.A.D. Newt secret (was exposed in chat) + +## Infrastructure +- [ ] Deploy Pi-hole ×2 (primary on PD + backup on Serenity or Pi4) +- [ ] Set up off-site backup for vital data (appdata, databases, config repo, photos) +- [ ] Remove dead Unraid-Cloudflared-Tunnel container from Serenity +- [ ] Serenity malcolm pool capacity planning (heavily utilized) +- [ ] Add Pangolin resource for `ai.paccoco.com` → openwebui (port 8282) +- [ ] Add Cloudflare DNS A record for `ai.paccoco.com` +- [ ] Install fresh editor on PD +- [ ] Set up private Gitea repo for .env file backups + +## AI Stack +- [ ] Pull `qwen2.5:14b` on PD ollama: `sudo docker exec -it ollama ollama pull qwen2.5:14b` +- [ ] Create OpenWebUI account and set system prompt from HOMELAB_AI_CONTEXT.md +- [ ] Disable signups in OpenWebUI after creating account +- [ ] Connect desktop Ollama to OpenWebUI (Admin → Settings → Connections) +- [ ] Set `OLLAMA_HOST=0.0.0.0` on desktop and restart Ollama service + +## Planned Deployments (PD) +- [ ] dispatcharr +- [ ] gotify +- [ ] bytestash +- [ ] kima-hub +- [ ] n8n +- [ ] qdrant +- [ ] paperless-ngx + +## Healthchecks +- [ ] Add healthcheck to ix-plex-plex-1 +- [ ] Add healthcheck to ix-tailscale-tailscale-1 +- [ ] Add healthcheck to ix-newt-newt-1 +- [ ] Add healthcheck to ollama (no curl/wget — needs /proc/net/tcp6 or custom solution) +- [ ] Document and add healthchecks for meshtastic stack + +## Documentation +- [ ] Document meshtastic stack fully (see docs/reference/MESHTASTIC.md) +- [ ] Fill in port/URL columns in SERVICES_DIRECTORY.md +- [ ] Document N.O.M.A.D. containers (start_nomad.sh services) diff --git a/docs/reference/HOMELAB_AI_CONTEXT.md b/docs/reference/HOMELAB_AI_CONTEXT.md new file mode 100644 index 0000000..396bacb --- /dev/null +++ b/docs/reference/HOMELAB_AI_CONTEXT.md @@ -0,0 +1,56 @@ +# Homelab AI Context Document + +This document gives an AI assistant full context of this homelab for infrastructure questions, deployments, debugging, and planning. + +See the individual docs in this repo for detailed information: + +- [Architecture Overview](../architecture/ARCHITECTURE_OVERVIEW.md) +- [Networking Model](../architecture/NETWORKING_MODEL.md) +- [Hardware Inventory](../architecture/HARDWARE_INVENTORY.md) +- [Services Directory](../architecture/SERVICES_DIRECTORY.md) +- [Stack Standards](../operations/STACK_STANDARDS.md) +- [Deployment Checklist](../operations/DEPLOYMENT_CHECKLIST.md) +- [Storage Notes](../operations/DOCKERSTACK_STORAGE_NOTES.md) +- [Backup Policy](../operations/BACKUP_POLICY.md) +- [Secrets Management](../operations/SECRETS_MANAGEMENT.md) +- [Troubleshooting](../troubleshooting/TROUBLESHOOTING.md) +- [Known Quirks](../troubleshooting/KNOWN_QUIRKS.md) +- [Serenity](../servers/SERENITY.md) +- [PlausibleDeniability](../servers/PLAUSIBLEDENABILITY.md) +- [N.O.M.A.D.](../servers/NOMAD.md) +- [Meshtastic Stack](MESHTASTIC.md) +- [Expansion Plan](../planning/HOMELAB_EXPANSION_PLAN.md) +- [TODO](../planning/TODO.md) + +## Quick Reference + +### Communication Preferences +- One command block at a time +- Prefer scripted/automated solutions over interactive multi-step processes +- Uses `fresh` editor (not nano) on Linux +- Manages remote servers via SSH from Windows 11 +- Prefers MariaDB over PostgreSQL for standalone services + +### Three Servers +| Server | Role | OS | IP | +|--------|------|----|----| +| Serenity | NAS / Media ingestion | Unraid 7.2.4 | 10.5.1.5 | +| PlausibleDeniability | Primary Docker host | TrueNAS Scale 25.10.2.1 | LAN DHCP | +| N.O.M.A.D. | Offline knowledge + game servers | Ubuntu 25.10 | 10.5.1.16 | + +### Deployment Pattern (PD) +```bash +cd /mnt/docker-ssd/docker/compose/ +docker compose --env-file .env config # validate first +docker compose --env-file .env up -d # deploy +docker logs --tail 20 # verify +``` + +### Network names +- Shared DB network: `ix-databases_shared-databases` +- Reverse proxy network: `pangolin` + +### Storage decision +- Write-heavy / DB / SQLite / GPU → `/mnt/docker-ssd/docker/appdata/` +- General config / non-fussy → `/mnt/tank/docker/appdata/` +- Media only → `/mnt/unraid/data/media/` diff --git a/docs/reference/MESHTASTIC.md b/docs/reference/MESHTASTIC.md new file mode 100644 index 0000000..8cc783e --- /dev/null +++ b/docs/reference/MESHTASTIC.md @@ -0,0 +1,70 @@ +# Meshtastic Stack + +## Overview +Self-hosted Meshtastic monitoring and management stack running on PlausibleDeniability. + +## Stack Location +`/mnt/docker-ssd/docker/compose/meshtastic/` + +## Containers + +| Container | Image | Port | Purpose | +|-----------|-------|------|---------| +| meshmonitor | ghcr.io/yeraze/meshmonitor:4.2.0 | 8081→3001 | Main UI + backend | +| meshmonitor (virtual node) | — | 4404 | TCP virtual node server | +| meshmonitor-tileserver | maptiler/tileserver-gl-light | 8082→8080 | Offline map tiles | +| meshmonitor-mqtt-proxy | ghcr.io/ln4cy/mqtt-proxy:master | — | MQTT→TCP bridge | +| meshmonitor-upgrader | docker:latest | — | Auto-upgrade watchdog | + +## Architecture +``` +Meshtastic Node (10.5.1.172) + ↓ MQTT +meshmonitor-mqtt-proxy + ↓ TCP (port 4404) +meshmonitor virtual node server + ↓ +meshmonitor backend + ↓ +PostgreSQL (shared-postgres, db: meshmonitor) +``` + +## Node Connection +- Node IP: `10.5.1.172` +- Connection type: MQTT via mqtt-proxy (not direct TCP) +- mqtt-proxy connects to meshmonitor's virtual node server on port 4404 + +## Storage +- Appdata: `/mnt/tank/docker/appdata/meshmonitor/data` → `/data` +- Scripts: `./scripts` → `/data/scripts` (must be pre-created on host) +- Tiles: `./tiles` (gitignored — large .mbtiles files) + +## Auto-Upgrade +`AUTO_UPGRADE_ENABLED=true` — meshmonitor checks for new images and triggers self-upgrade. + +**Important:** The upgrader mounts the compose directory as `/compose` inside the container. All relative bind mount paths (e.g. `./scripts`) resolve to `/compose/scripts` on the host when the upgrader re-runs compose. Since TrueNAS root filesystem is read-only, these directories must be pre-created: + +```bash +mkdir -p /mnt/docker-ssd/docker/compose/meshtastic/scripts +``` + +## Networks +- `default` (internal stack network) +- `ix-databases_shared-databases` (meshmonitor only) +- `pangolin` (meshmonitor + tileserver) + +## Public URL +`https://meshmonitor.paccoco.com` + +## Known Warnings +- Node `!dd972536` has a low-entropy public key — this is a device configuration issue, not a meshmonitor bug +- Frequent connect/disconnect cycles in logs are normal when the MQTT proxy reconnects + +## Tiles +Map tiles stored in `./tiles/`. Currently includes `zurich_switzerland.mbtiles` (34MB). +Tiles directory is gitignored. To add new tiles, download `.mbtiles` files and place in the `tiles/` directory. + +## TODO +- [ ] Document all running meshtastic-map container config +- [ ] Document meshtastic-map-mosquitto and meshtastic-mqtt containers +- [ ] Add healthcheck to meshmonitor container diff --git a/docs/servers/NOMAD.md b/docs/servers/NOMAD.md new file mode 100644 index 0000000..cac5f49 --- /dev/null +++ b/docs/servers/NOMAD.md @@ -0,0 +1,73 @@ +# N.O.M.A.D. Server + +Offline survival/knowledge server and game server host. + +## Specs +- **OS:** Ubuntu 25.10 (bare metal) +- **IP:** 10.5.1.16 +- **CPU:** Intel Core i7-4770K @ 3.50GHz (4c/8t) +- **RAM:** 32GB DDR3 1600MHz +- **GPU:** NVIDIA GeForce GTX 1080 (8GB VRAM) +- **NIC:** Intel I218-V (1GbE) + Aquantia AQC113 (10GbE) +- **Motherboard:** ASRock H97M Pro4 +- **UPS:** None + +## Storage +| Device | Size | Mount | Purpose | +|--------|------|-------|---------| +| Samsung SSD 840 (sdb) | 232.9GB | `/` (LVM 100GB) | OS boot | +| WD Red HDD (sda) | 3.6TB | `/mnt/hdd-1` | Primary data | +| WD Red HDD (sdc) | 3.6TB | `/mnt/hdd-2` | Docker data-root | +| Seagate SSD (sdd) | 223.6GB | `/mnt/ssd-1` | Fast scratch | + +## Project N.O.M.A.D. +- 10 containers running via `start_nomad.sh` / `stop_nomad.sh` +- `/opt/project-nomad` → symlinked from `/mnt/hdd-1/project-nomad` +- Docker data-root: `/mnt/hdd-2/docker` +- Based on Crosstalk Solutions offline survival/knowledge server + +## Pelican Panel (Game Server Management) +- **URL:** `https://panel.paccoco.com` +- **Stack:** PHP 8.4 + nginx + MariaDB (native, not Docker) +- **Panel files:** `/var/www/pelican` +- **DB:** MariaDB, database `pelican`, user `pelican`@`127.0.0.1` +- **APP_URL:** `https://panel.paccoco.com` +- **TRUSTED_PROXIES:** `*` +- **Cron:** `* * * * * php /var/www/pelican/artisan schedule:run` (www-data) +- **Admin user:** `fizzlepoof` +- **Reverse proxy:** Pangolin via Newt container at `/opt/newt` + +## Wings (Game Server Daemon) +- **Binary:** `/usr/local/bin/wings` +- **Config:** `/etc/pelican/config.yml` +- **Systemd:** `wings.service` +- **ExecStart:** `/usr/local/bin/wings --ignore-certificate-errors` +- **API port:** 8443 +- **SFTP port:** 2022 +- **Node domain:** `node1.paccoco.com` (Pangolin, no auth, connection port 443) +- **Game server volumes:** `/var/lib/pelican/volumes` +- **Docker network:** `pelican_nw` (172.20.0.0/16) + +## Active Game Servers +- **Minecraft:** Purpur, Java 21, port 25565 TCP/UDP (direct host port forward) + +## Networking Notes +- `panel.paccoco.com` → `127.0.0.1` in `/etc/hosts` for NAT loopback +- nginx on 80 and 443 (self-signed cert) for local Wings → Panel +- Wings uses `--ignore-certificate-errors` for self-signed cert +- Pangolin resource `node1.paccoco.com` has auth disabled + +## Known Quirks +- NVIDIA container toolkit repo has invalid GPG key — remove `/etc/apt/sources.list.d/*nvidia*` if apt errors +- Port 8080 occupied by `nomad_admin` container — Wings uses 8443 +- Interactive artisan commands break over SSH — always use inline flags +- Wings CORS derived from `remote:` URL — keep remote as `https://` and use `--ignore-certificate-errors` + +## Newt Tunnel +- Container: `newt` at `/opt/newt/docker-compose.yaml` +- `network_mode: host` +- Connects to Pangolin VPS at `https://paccoco.com` + +## Pending +- Regenerate Wings token (was exposed in chat) +- Regenerate Newt secret (was exposed in chat) diff --git a/docs/servers/PLAUSIBLEDENABILITY.md b/docs/servers/PLAUSIBLEDENABILITY.md new file mode 100644 index 0000000..f6f84d6 --- /dev/null +++ b/docs/servers/PLAUSIBLEDENABILITY.md @@ -0,0 +1,60 @@ +# PlausibleDeniability + +Primary Docker host running all production workloads. + +## Specs +- **OS:** TrueNAS Scale 25.10.2.1 +- **CPU:** Ryzen 9 3950X +- **RAM:** 32GB +- **GPU:** Nvidia RTX 2080 Ti (11GB VRAM) +- **UPS:** None + +## Storage Tiers + +| Tier | Mount | Use for | +|------|-------|---------| +| Local SSD (Crucial MX500 500GB) | `/mnt/docker-ssd` | Docker root, databases, write-heavy appdata, GPU/model caches | +| TrueNAS pool (tank, 2-drive ZFS mirror ~3.5TB) | `/mnt/tank` | General appdata, configs, backups, downloads | +| Unraid NFS | `/mnt/unraid` | Media libraries, Immich media | + +## Docker Root +`/mnt/docker-ssd/docker/docker-data` + +## Compose Repo +`/mnt/docker-ssd/docker/compose` + +## Key Docker Networks +- `ix-databases_shared-databases` — shared DB access network (created by databases stack) +- `pangolin` — reverse proxy exposure network (created by newt) + +## GPU Usage +RTX 2080 Ti used by: +- Plex (hardware transcoding) +- immich-ml (CUDA face recognition) +- ollama (LLM inference) + +Recommended models for 11GB VRAM: +| Model | VRAM | Best for | +|-------|------|---------| +| `qwen2.5:14b` | ~8GB | General use, coding, reasoning | +| `qwen2.5-coder:7b` | ~5GB | Code tasks | +| `deepseek-r1:8b` | ~5GB | Math, logic | +| `phi4` | ~8GB | Analytical tasks | + +## Remote Access +- Pangolin VPS + Newt for external services +- SSH from Windows 11 + +## NFS Exports to Other Servers +- `data` share → PD at `/mnt/unraid/data` +- `immich` share → PD at `/mnt/unraid/immich` + +## Known Issues +- immich-ml intermittently unhealthy — investigate GPU/CUDA health +- docker-ssd has no redundancy — single point of failure for all appdata +- No off-site backup + +## Pending +- Deploy: dispatcharr, gotify, bytestash, kima-hub, n8n, qdrant, paperless-ngx +- Add Pangolin resource + DNS for `ai.paccoco.com` +- Install fresh editor diff --git a/docs/servers/SERENITY.md b/docs/servers/SERENITY.md new file mode 100644 index 0000000..b2399c7 --- /dev/null +++ b/docs/servers/SERENITY.md @@ -0,0 +1,55 @@ +# Serenity + +Unraid NAS and media ingestion server. + +## Specs +- **OS:** Unraid 7.2.4 +- **IP:** 10.5.1.5 | Tailscale: 100.94.87.79 +- **CPU:** Xeon X5690 +- **RAM:** 102GB +- **GPU:** None (Matrox IPMI only) +- **UPS:** None + +## Storage +- `malcolm` pool: RAIDZ1 of 6×10TB HDDs (~40TB usable) — heavily utilized +- Cache pool: single Kingston SNV2S 1TB NVMe (ZFS) + +## Docker Networks +`arrs`, `hotio`, `br0` (macvlan), `bridge` + +## Active Containers +- **Media:** sonarr, sonarr-anime, radarr, lidarr, readarr, readarr-epub, prowlarr, bazarr, unpackerr +- **Download:** qBittorrent, GluetunVPN, autobrr, qbit_manage +- **Notifications:** Notifiarr +- **Access:** Wizarr, Newt, dockersocket, Hawser +- **Databases:** postgresql15, MariaDB-Official +- **Other:** romm, GameVault, shelfmark (port 8084), netdata +- **Dead/remove:** Unraid-Cloudflared-Tunnel + +## Backup Strategy +- Sanoid snapshots cache/appdata: 7 daily, 4 weekly, 6 monthly +- Syncoid replicates cache/appdata → malcolm/zfs_backups_array (strict-mirror) +- No off-site backup currently + +## Custom Scripts +- **qBit Mover script:** pauses torrents 0–4 days old, then runs Unraid mover +- **MAM-init script:** updates MyAnonamouse IP inside qbit container +- **Mover Status script:** sends Discord webhook notifications + +## Key Shares +| Share | Size | Purpose | +|-------|------|---------| +| data | 35TB | Media library (NFS-mounted by PD at `/mnt/unraid/data`) | +| immich | 62GB | Immich photo storage (NFS-mounted by PD at `/mnt/unraid/immich`) | +| nextcloud | 7.4GB | Nextcloud data | +| Tesla/dashcam | 1.6TB | Dashcam footage | +| fizzlebang | 17GB | Personal files | +| isos | 22GB | ISO images | +| Syncthing | 4.6GB | Syncthing data | +| manndra | — | Partner's storage | +| Long and Low Crafts | — | Crafts business files | +| zfs_backups_array | — | ZFS backup target for cache/appdata | + +## Pending +- Remove dead Unraid-Cloudflared-Tunnel container +- malcolm pool capacity planning (heavily utilized) diff --git a/docs/troubleshooting/KNOWN_QUIRKS.md b/docs/troubleshooting/KNOWN_QUIRKS.md new file mode 100644 index 0000000..c305788 --- /dev/null +++ b/docs/troubleshooting/KNOWN_QUIRKS.md @@ -0,0 +1,69 @@ +# Known Quirks + +Per-service gotchas that aren't bugs but will bite you if you forget them. + +## PlausibleDeniability + +### immich-ml +- Healthcheck has no curl/wget — uses `/proc/net/tcp6` pattern +- Cache goes to `/mnt/docker-ssd/docker/appdata/immich-ml` (separate from immich-server's appdata) + +### rackpeek +- No curl/wget in image — healthcheck uses `/proc/net/tcp6` + +### donetick +- Env vars alone are insufficient for DB type selection +- Requires `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml` with full DB config +- Uses viper config loader — `DT_ENV` controls config file path + +### shlink +- Data directory must be `chmod 777` — runs as non-root user that doesn't match host default ownership + +### openwebui +- DB: user=`openwebui`, db=`openwebui` on shared-postgres +- If restart-looping with auth errors: container wasn't on `ix-databases_shared-databases` at creation time — must `down && up` + +### plex +- Port `5353/udp` conflicts with system avahi/mDNS — remove from port mappings + +### scriberr (shelved) +- SQLite incompatible with ZFS nfsv4 ACLs (SQLITE_CANTOPEN error 14) +- Revisit when Postgres support available in CUDA image + +### meshmonitor (auto-upgrade) +- `AUTO_UPGRADE_ENABLED=true` causes crashes on TrueNAS if bind mount source directories don't exist +- The upgrader container mounts compose dir as `/compose` — relative paths resolve differently +- Fix: pre-create all bind mount source directories before deploying + +### Any container using `down && up` requirement +- Network attachments happen at container creation, not restart +- `docker restart` or `docker compose restart` will NOT fix missing network attachments +- Must use `docker compose --env-file .env down && docker compose --env-file .env up -d ` + +--- + +## Serenity + +### qBit Mover Script +- Pauses torrents 0–4 days old before running mover to prevent partial file moves +- MAM-init script updates MyAnonamouse IP inside qbit container on startup + +### Unraid-Cloudflared-Tunnel +- Dead container, should be removed + +--- + +## N.O.M.A.D. + +### NVIDIA Container Toolkit +- Repo has invalid GPG key — remove `/etc/apt/sources.list.d/*nvidia*` if apt errors + +### Pelican / Wings +- Interactive artisan commands break over SSH (stty issue) — always use inline flags: + `--email=x --username=y --password=z --admin=1` +- Wings CORS derived from `remote:` URL, not `allowed_origins` — keep remote as `https://` +- `panel.paccoco.com` in `/etc/hosts` → `127.0.0.1` for NAT loopback +- Wings uses `--ignore-certificate-errors` for local self-signed cert + +### Port 8080 +- Occupied by `nomad_admin` container — Wings uses 8443 instead diff --git a/docs/troubleshooting/TROUBLESHOOTING.md b/docs/troubleshooting/TROUBLESHOOTING.md new file mode 100644 index 0000000..646c326 --- /dev/null +++ b/docs/troubleshooting/TROUBLESHOOTING.md @@ -0,0 +1,129 @@ +# Troubleshooting Guide + +## Permission Denied / Readonly Database / Not Writable + +Common causes: +- App placed on NFS (tank) when it needs local SSD +- NFS ownership mismatch between host user and container user + +Checks: +1. `docker logs --tail 30` +2. `ls -la ` — check ownership and permissions +3. `docker run --rm --entrypoint sh -lc 'id'` — check container UID/GID + +Fix options: +- Move appdata to `/mnt/docker-ssd/docker/appdata/` if write-heavy +- `sudo chmod -R 777 ` for apps with fixed internal user (e.g. shlink) +- `sudo chown -R : ` if UID/GID is known + +## SQLite Fails on ZFS with nfsv4 ACLs (SQLITE_CANTOPEN error 14) + +Symptom: `unable to open database file: out of memory (14)` + +Cause: SQLite cannot acquire POSIX file locks on ZFS datasets configured with `acltype=nfsv4`. + +Diagnosis: +```bash +sudo zfs get acltype,aclmode,xattr +``` + +Fix options: +1. Change acltype: `sudo zfs set acltype=posix ` +2. Move the SQLite database to local SSD +3. Use an app version that supports Postgres instead of SQLite + +Note: scriberr CUDA image is shelved for this reason. + +## Container Can't Resolve Shared Database Hostname + +Symptom: `lookup shared-postgres on 127.0.0.11:53: no such host` + +Fix: +```bash +docker compose --env-file .env down +docker compose --env-file .env up -d +``` + +Verify network attachment: +```bash +docker inspect --format '{{json .NetworkSettings.Networks}}' | tr ',' '\n' | grep -o '"[^"]*"' +``` + +## Postgres Auth Fails Over TCP but Works via Socket + +Diagnosis: +```bash +sudo docker exec shared-postgres psql -h 127.0.0.1 -U -d -W -c "\conninfo" +``` + +Fix: +- Verify pg_hba.conf has `host all all all scram-sha-256` +- Check DATABASE_URL password matches: `ALTER USER WITH PASSWORD '';` +- Avoid special characters in passwords in DATABASE_URLs + +## Healthcheck Failing Despite Container Being Healthy + +Diagnosis: +```bash +sudo docker exec sh -c "which curl; which wget" 2>&1 +sudo docker inspect --format '{{json .State.Health.Log}}' | python3 -m json.tool | tail -20 +``` + +After fixing the compose file, must do `down && up` (not just restart). + +## App Ignores Environment Variables for DB Type + +Some apps (e.g. donetick) require a YAML config file for certain settings even when env vars are set. + +For donetick, create `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml`: +```yaml +database: + type: postgres + host: shared-postgres + port: 5432 + user: donetick + password: + name: donetick + migration: true +jwt: + secret: <32-byte-hex> + session_time: 168h + max_refresh: 168h +server: + port: 2021 + serve_frontend: true +``` + +## JWT Invalid / Login Fails After Restart + +Cause: JWT secret changed. Fix: open app in incognito or clear cookies/localStorage. + +## Port Already In Use + +Common offenders: +- `5353/udp` — held by system avahi/mDNS. Remove from Plex ports. +- `3306`, `5432`, `6379` — only one DB stack can bind these. + +## Auto-Upgrade Crashes on TrueNAS (exit code 128) + +Symptom: `error while creating mount source path '/compose/scripts': mkdir /compose: read-only file system` + +Cause: Auto-upgrade container runs docker compose from inside a container where the project dir is mounted as `/compose`. Relative bind mount paths like `./scripts` resolve to `/compose/scripts` on the host. TrueNAS root filesystem is read-only so Docker can't auto-create the directory. + +Fix: Pre-create the directory manually on the host before starting the stack: +```bash +mkdir -p /mnt/docker-ssd/docker/compose//scripts +``` + +## Databases Stack Network Name Wrong + +Must have `name: ix-databases` at top of databases compose. Verify: +```bash +docker network ls | grep databases +# Should show: ix-databases_shared-databases +``` + +## Checking Container Runtime UID +```bash +docker run --rm --entrypoint sh -lc 'id' +```