docs: add full homelab documentation structure

2026-05-05 14:43:09 -05:00
parent ee8a412a69
commit 370998b8d5
20 changed files with 1405 additions and 24 deletions
--- a/docs/operations/BACKUP_POLICY.md
+++ b/docs/operations/BACKUP_POLICY.md
@@ -0,0 +1,42 @@
+# Backup Policy
+
+## Current State (Known Gaps)
+- **No off-site backup** — this is a known gap, improvement pending
+- Serenity has local ZFS snapshots and replication
+- PlausibleDeniability has no automated backup of appdata or databases
+
+## Serenity Backup Strategy
+- Sanoid snapshots cache/appdata: 7 daily, 4 weekly, 6 monthly
+- Syncoid replicates cache/appdata → malcolm/zfs_backups_array (strict-mirror)
+- qBit Mover script: pauses torrents 0–4 days old then runs Unraid mover
+- Mover Status script: sends Discord webhook notifications
+
+## PlausibleDeniability Backup Scope
+Back up at minimum:
+- Compose repo: `/mnt/docker-ssd/docker/compose` (in git — GitHub + Gitea)
+- Local SSD appdata: `/mnt/docker-ssd/docker/appdata/`
+- Databases: `/mnt/docker-ssd/docker/databases/`
+- Selected appdata/backups on tank: `/mnt/tank/docker/`
+- .env files (separately — gitignored, handle via encrypted backup or private repo)
+
+## Database Backups
+- Keep app-consistent dumps for major databases
+- Do not rely only on raw volume copies for restores
+- Document restore procedure per major service
+
+Recommended dump locations: `/mnt/tank/docker/backups/db-dumps/`
+
+## .env File Backup Options
+1. **Private Gitea repo** — short term, push all .env files to a private repo on your local Gitea
+2. **Encrypted restic** — include .env files in restic backup with encryption, push offsite (Backblaze B2)
+3. **Vaultwarden** — store each .env as a secure note (best long-term once deployed)
+
+## Restore Practice
+- Test restores quarterly
+- Test at least one app DB restore and one full-config restore
+- Verify that restored stacks can start with current compose files
+
+## Operational Notes
+- Media libraries on Unraid are large and may not need the same backup frequency as configs/databases
+- SSD-local DB and SQLite workloads are highest priority for backup validation
+- Serenity malcolm pool is heavily utilized — capacity planning needed before it fills
--- a/docs/operations/DEPLOYMENT_CHECKLIST.md
+++ b/docs/operations/DEPLOYMENT_CHECKLIST.md
@@ -0,0 +1,63 @@
+# New Service Deployment Checklist
+
+## Before Adding a Service
+
+- [ ] Determine DB requirements (postgres, mariadb, sqlite, none)
+- [ ] Decide storage tier: SSD (`/mnt/docker-ssd/docker/appdata/`) vs tank (`/mnt/tank/docker/appdata/`) vs Unraid
+- [ ] Check if service needs `ix-databases_shared-databases` network
+- [ ] Check if service needs `pangolin` network for reverse proxy exposure
+
+## Scaffold
+
+- [ ] Create host directories for appdata
+- [ ] Set correct ownership if needed (`chown` or `chmod 777` for permission-sensitive apps)
+- [ ] Create stack folder under `/mnt/docker-ssd/docker/compose/<stack>/`
+- [ ] Create `docker-compose.yaml`
+- [ ] Create `.env` and `.env.example`
+- [ ] If using postgres: create DB and user, or add init SQL to `initdb/`
+- [ ] If using mariadb: add init script to `initdb-mariadb/`
+
+## Validate Before Deploying
+
+```bash
+cd /mnt/docker-ssd/docker/compose/<stack>
+docker compose --env-file .env config
+```
+
+No errors = safe to proceed.
+
+## Deploy
+
+```bash
+docker compose --env-file .env up -d
+```
+
+## After Starting
+
+- [ ] `docker ps` shows container running
+- [ ] `docker logs <container> --tail 20` shows clean startup
+- [ ] `docker inspect <container>` confirms mounts are correct
+- [ ] Data persists after `docker compose --env-file .env down && up -d`
+
+## Network Checklist
+
+If service needs shared databases:
+- Top-level networks block declares `ix-databases_shared-databases` as external
+- Service networks list includes `ix-databases_shared-databases`
+- Verify: `docker inspect <container> --format '{{json .NetworkSettings.Networks}}'`
+
+If service needs reverse proxy:
+- Top-level networks block declares `pangolin` as external
+- Service networks list includes `pangolin`
+- Infrastructure stack (newt) must be running first
+
+## Common Failure Modes
+
+| Symptom | Likely Cause | Fix |
+|---|---|---|
+| `hostname resolving error` for shared-postgres | Not on `ix-databases_shared-databases` network | `down && up` not just restart |
+| `not writable` errors on startup | NFS ownership mismatch | Move to SSD or chmod 777 |
+| `unsupported database type` | App requires config file, not just env vars | Create `selfhosted.yaml` or equivalent |
+| `JWT is invalid` after restart | JWT secret changed | Clear browser cookies/localStorage |
+| Port conflict on startup | Port already bound | Remove conflicting port mapping |
+| Auto-upgrade crashes on TrueNAS | Bind mount source path missing, Docker tries to create at root | Pre-create the directory manually |
--- a/docs/operations/DOCKERSTACK_STORAGE_NOTES.md
+++ b/docs/operations/DOCKERSTACK_STORAGE_NOTES.md
@@ -0,0 +1,61 @@
+# Dockerstack Storage Notes
+
+## Local SSD
+Use `/docker/...` for:
+- Docker root (`/docker/docker-data`)
+- GPU/model storage
+- SQLite-backed apps
+- Apps that chown on startup
+- Write-heavy DB/search storage
+
+Examples:
+- `/mnt/docker-ssd/docker/docker-data`
+- `/mnt/docker-ssd/docker/appdata/ollama`
+- `/mnt/docker-ssd/docker/appdata/openwebui`
+- `/mnt/docker-ssd/docker/appdata/immich`
+- `/mnt/docker-ssd/docker/appdata/immich-ml`
+- `/mnt/docker-ssd/docker/appdata/plex`
+- `/mnt/docker-ssd/docker/appdata/tautulli`
+- `/mnt/docker-ssd/docker/appdata/audiobookshelf`
+- `/mnt/docker-ssd/docker/appdata/seerr`
+- `/mnt/docker-ssd/docker/appdata/dockhand`
+- `/mnt/docker-ssd/docker/appdata/rackpeek`
+- `/mnt/docker-ssd/docker/appdata/shlink`
+- `/mnt/docker-ssd/docker/appdata/gitea`
+- `/mnt/docker-ssd/docker/appdata/netdata`
+- `/mnt/docker-ssd/docker/appdata/tailscale`
+- `/mnt/docker-ssd/docker/appdata/uptime-kuma`
+- `/mnt/docker-ssd/docker/databases/postgres`
+- `/mnt/docker-ssd/docker/databases/mariadb`
+- `/mnt/docker-ssd/docker/databases/redis`
+
+## TrueNAS NFS
+Use `/mnt/tank/docker/...` for:
+- Normal appdata
+- Uploads
+- Non-fussy config and data
+- Backups
+- Downloads
+
+Examples:
+- `/mnt/tank/docker/appdata/homepage`
+- `/mnt/tank/docker/appdata/newt`
+- `/mnt/tank/docker/appdata/kitchenowl`
+- `/mnt/tank/docker/appdata/donetick`
+- `/mnt/tank/docker/appdata/qui`
+- `/mnt/tank/docker/appdata/calibre-web-automated`
+- `/mnt/tank/docker/backups`
+- `/mnt/tank/docker/downloads`
+
+## Unraid NFS
+Use `/mnt/unraid/...` for media and shared bulk content only:
+- `/mnt/unraid/data/media/` — Plex libraries, books, etc.
+- `/mnt/unraid/immich` — Immich photo/video storage
+
+## Rules Learned
+- SQLite apps and apps that write logs aggressively should prefer local SSD.
+- Database/search/model-heavy workloads should prefer local SSD.
+- General config and less write-sensitive appdata can live on TrueNAS NFS.
+- Media should stay on Unraid mounts.
+- SQLite on ZFS with nfsv4 ACLs will always fail — move to SSD or use Postgres.
+- Apps with auto-upgrade features that use bind mounts: pre-create all source directories manually on TrueNAS to prevent Docker from trying to create them at the read-only root.
--- a/docs/operations/SECRETS_MANAGEMENT.md
+++ b/docs/operations/SECRETS_MANAGEMENT.md
@@ -0,0 +1,64 @@
+# Secrets Management
+
+## Principles
+- `.env` files are gitignored and never committed to any public repo
+- `.env.example` files are committed with placeholder values
+- Real `.env` files live only on the host at the stack directory
+- Secrets are generated with standard tools, never reused across services
+
+## Generating Secrets
+
+```bash
+# Database passwords
+openssl rand -hex 24
+
+# JWT / session secrets
+openssl rand -hex 32
+
+# API keys (where format allows)
+openssl rand -base64 32
+```
+
+## .env File Locations
+All `.env` files live alongside their `docker-compose.yaml`:
+```
+/mnt/docker-ssd/docker/compose/<stack>/.env
+```
+
+## Backing Up .env Files
+
+### Short term: Private Gitea repo
+Create a private repo on Gitea (not GitHub) and push all .env files there:
+```bash
+# In a separate directory, not the main stacks repo
+git init homelab-secrets
+cd homelab-secrets
+# Copy and add .env files
+git remote add origin https://gitea.paccoco.com/fizzlepoof/homelab-secrets.git
+git push -u origin main
+```
+
+### Long term: Restic + offsite
+Include the compose directory (with .env files) in a restic backup:
+```bash
+restic -r <repo> backup /mnt/docker-ssd/docker/compose --exclude="*.example"
+```
+
+### Best long term: Vaultwarden
+Once deployed, store each service's .env as a secure note in Vaultwarden. Single source of truth, accessible from anywhere.
+
+## Known Credentials Locations
+
+| Service | Where stored |
+|---------|-------------|
+| shared-postgres | `databases/.env` |
+| shared-mariadb | `databases/.env` |
+| OpenWebUI DB | `ai/.env` — user: openwebui, db: openwebui |
+| Donetick | `home/.env` + `selfhosted.yaml` |
+| Meshmonitor | `meshtastic/.env` |
+| Gitea | `dev/.env` |
+
+## Security Reminders
+- Regenerate Wings token on N.O.M.A.D. (was exposed in chat — TODO)
+- Regenerate N.O.M.A.D. Newt secret (was exposed in chat — TODO)
+- Disable signups in OpenWebUI after creating initial account
--- a/docs/operations/STACK_STANDARDS.md
+++ b/docs/operations/STACK_STANDARDS.md
@@ -0,0 +1,122 @@
+# Homelab Stack Standards
+
+## Repository Root
+`/mnt/docker-ssd/docker/compose`
+
+Remotes:
+- GitHub: `git@github.com:Paccoco/truenas-stacks.git`
+- Gitea: `https://gitea.paccoco.com/fizzlepoof/truenas-stacks.git`
+
+## Stack Layout
+Each production stack lives in a subdirectory of the repo root.
+
+Each stack contains:
+- `docker-compose.yaml` — required
+- `.env` — required (gitignored)
+- `.env.example` — committed with placeholder values
+- optional init scripts (e.g. `initdb/`, `initdb-mariadb/`)
+
+## Deployment Order
+1. `databases/` — must be up before anything else
+2. `infrastructure/` — newt creates the pangolin network
+3. All other stacks in any order
+
+## Deployment Workflow
+
+Validate before deploying:
+```bash
+cd /mnt/docker-ssd/docker/compose/<stack>
+docker compose --env-file .env config
+```
+
+Deploy:
+```bash
+docker compose --env-file .env up -d
+```
+
+Teardown a single service:
+```bash
+docker compose --env-file .env down <service>
+```
+
+## Networks
+
+### ix-databases_shared-databases
+Created by the `databases` stack with `name: ix-databases` at top level. Any stack connecting to shared databases must declare:
+```yaml
+networks:
+  ix-databases_shared-databases:
+    external: true
+```
+
+### pangolin
+Created by `newt` at runtime. Any externally exposed service must declare:
+```yaml
+networks:
+  pangolin:
+    external: true
+```
+
+## Storage Standards
+
+### Local SSD: `/mnt/docker-ssd`
+Use `/mnt/docker-ssd/docker/appdata/<service>` for:
+- Postgres / MariaDB / Redis data
+- SQLite-backed apps
+- Apps that chown aggressively on startup
+- GPU/model caches
+- Write-heavy logs, cache, state
+
+### TrueNAS Pool (tank): `/mnt/tank`
+Use `/mnt/tank/docker/appdata/<service>` for:
+- General appdata
+- Configs
+- Uploads
+- Non-permission-sensitive state
+
+### Unraid NFS: `/mnt/unraid`
+- `/mnt/unraid/data/media/` — media libraries
+- `/mnt/unraid/immich` — Immich photo/video storage
+
+## Volume Rules
+- no anonymous volumes
+- no ad-hoc home directory paths
+- all binds use standardized production paths
+- evaluate SSD vs tank vs Unraid before deploying any new service
+- apps known to have permission issues on NFS go on SSD
+
+## Known Permission Quirks
+- `shlink` data directory must be chmod 777 — runs as non-root user
+- `immich-ml` cache goes to `/mnt/docker-ssd/docker/appdata/immich-ml` (separate from immich-server)
+- `donetick` requires `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml` — env vars alone insufficient
+
+## Secrets Management
+- `.env` files are gitignored
+- `.env.example` files are committed with placeholder values
+- Database passwords: `openssl rand -hex 24`
+- JWT secrets: `openssl rand -hex 32`
+
+## Healthcheck Patterns
+Always check available tools before writing healthchecks:
+```bash
+sudo docker exec <container> sh -c "which curl; which wget" 2>&1
+```
+
+- Has curl: `["CMD-SHELL", "curl -sf http://127.0.0.1:<port>/ || exit 1"]`
+- Has wget only: `["CMD-SHELL", "wget -qO- http://127.0.0.1:<port>/ >/dev/null 2>&1 || exit 1"]`
+- Has neither: `["CMD-SHELL", "cat /proc/net/tcp6 | grep -q <HEX_PORT> || exit 1"]`
+- Redirecting endpoint: `["CMD-SHELL", "curl -sfL http://127.0.0.1:<port>/ || exit 1"]`
+
+Common hex port values: 8080=1F90, 3003=BBB, 3000=BB8, 8181=1FF5, 5055=13BF, 2021=7E5
+
+After changing a healthcheck, must `down` then `up` (not just restart).
+
+## Git Workflow
+```bash
+cd /mnt/docker-ssd/docker/compose
+git add .
+git commit -m "description"
+git push   # pushes to both GitHub and Gitea
+```
+
+All `.env` files are gitignored. Only compose files, `.env.example`, and init scripts are committed.