docs: add full homelab documentation structure

2026-05-05 14:43:09 -05:00
parent ee8a412a69
commit 370998b8d5
20 changed files with 1405 additions and 24 deletions
--- a/docs/troubleshooting/KNOWN_QUIRKS.md
+++ b/docs/troubleshooting/KNOWN_QUIRKS.md
@@ -0,0 +1,69 @@
+# Known Quirks
+
+Per-service gotchas that aren't bugs but will bite you if you forget them.
+
+## PlausibleDeniability
+
+### immich-ml
+- Healthcheck has no curl/wget — uses `/proc/net/tcp6` pattern
+- Cache goes to `/mnt/docker-ssd/docker/appdata/immich-ml` (separate from immich-server's appdata)
+
+### rackpeek
+- No curl/wget in image — healthcheck uses `/proc/net/tcp6`
+
+### donetick
+- Env vars alone are insufficient for DB type selection
+- Requires `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml` with full DB config
+- Uses viper config loader — `DT_ENV` controls config file path
+
+### shlink
+- Data directory must be `chmod 777` — runs as non-root user that doesn't match host default ownership
+
+### openwebui
+- DB: user=`openwebui`, db=`openwebui` on shared-postgres
+- If restart-looping with auth errors: container wasn't on `ix-databases_shared-databases` at creation time — must `down && up`
+
+### plex
+- Port `5353/udp` conflicts with system avahi/mDNS — remove from port mappings
+
+### scriberr (shelved)
+- SQLite incompatible with ZFS nfsv4 ACLs (SQLITE_CANTOPEN error 14)
+- Revisit when Postgres support available in CUDA image
+
+### meshmonitor (auto-upgrade)
+- `AUTO_UPGRADE_ENABLED=true` causes crashes on TrueNAS if bind mount source directories don't exist
+- The upgrader container mounts compose dir as `/compose` — relative paths resolve differently
+- Fix: pre-create all bind mount source directories before deploying
+
+### Any container using `down && up` requirement
+- Network attachments happen at container creation, not restart
+- `docker restart` or `docker compose restart` will NOT fix missing network attachments
+- Must use `docker compose --env-file .env down <service> && docker compose --env-file .env up -d <service>`
+
+---
+
+## Serenity
+
+### qBit Mover Script
+- Pauses torrents 0–4 days old before running mover to prevent partial file moves
+- MAM-init script updates MyAnonamouse IP inside qbit container on startup
+
+### Unraid-Cloudflared-Tunnel
+- Dead container, should be removed
+
+---
+
+## N.O.M.A.D.
+
+### NVIDIA Container Toolkit
+- Repo has invalid GPG key — remove `/etc/apt/sources.list.d/*nvidia*` if apt errors
+
+### Pelican / Wings
+- Interactive artisan commands break over SSH (stty issue) — always use inline flags:
+  `--email=x --username=y --password=z --admin=1`
+- Wings CORS derived from `remote:` URL, not `allowed_origins` — keep remote as `https://`
+- `panel.paccoco.com` in `/etc/hosts` → `127.0.0.1` for NAT loopback
+- Wings uses `--ignore-certificate-errors` for local self-signed cert
+
+### Port 8080
+- Occupied by `nomad_admin` container — Wings uses 8443 instead
--- a/docs/troubleshooting/TROUBLESHOOTING.md
+++ b/docs/troubleshooting/TROUBLESHOOTING.md
@@ -0,0 +1,129 @@
+# Troubleshooting Guide
+
+## Permission Denied / Readonly Database / Not Writable
+
+Common causes:
+- App placed on NFS (tank) when it needs local SSD
+- NFS ownership mismatch between host user and container user
+
+Checks:
+1. `docker logs <container> --tail 30`
+2. `ls -la <host-appdata-path>` — check ownership and permissions
+3. `docker run --rm --entrypoint sh <image> -lc 'id'` — check container UID/GID
+
+Fix options:
+- Move appdata to `/mnt/docker-ssd/docker/appdata/<service>` if write-heavy
+- `sudo chmod -R 777 <appdata-path>` for apps with fixed internal user (e.g. shlink)
+- `sudo chown -R <uid>:<gid> <appdata-path>` if UID/GID is known
+
+## SQLite Fails on ZFS with nfsv4 ACLs (SQLITE_CANTOPEN error 14)
+
+Symptom: `unable to open database file: out of memory (14)`
+
+Cause: SQLite cannot acquire POSIX file locks on ZFS datasets configured with `acltype=nfsv4`.
+
+Diagnosis:
+```bash
+sudo zfs get acltype,aclmode,xattr <dataset>
+```
+
+Fix options:
+1. Change acltype: `sudo zfs set acltype=posix <dataset>`
+2. Move the SQLite database to local SSD
+3. Use an app version that supports Postgres instead of SQLite
+
+Note: scriberr CUDA image is shelved for this reason.
+
+## Container Can't Resolve Shared Database Hostname
+
+Symptom: `lookup shared-postgres on 127.0.0.11:53: no such host`
+
+Fix:
+```bash
+docker compose --env-file .env down <service>
+docker compose --env-file .env up -d <service>
+```
+
+Verify network attachment:
+```bash
+docker inspect <container> --format '{{json .NetworkSettings.Networks}}' | tr ',' '\n' | grep -o '"[^"]*"'
+```
+
+## Postgres Auth Fails Over TCP but Works via Socket
+
+Diagnosis:
+```bash
+sudo docker exec shared-postgres psql -h 127.0.0.1 -U <user> -d <db> -W -c "\conninfo"
+```
+
+Fix:
+- Verify pg_hba.conf has `host all all all scram-sha-256`
+- Check DATABASE_URL password matches: `ALTER USER <user> WITH PASSWORD '<pass>';`
+- Avoid special characters in passwords in DATABASE_URLs
+
+## Healthcheck Failing Despite Container Being Healthy
+
+Diagnosis:
+```bash
+sudo docker exec <container> sh -c "which curl; which wget" 2>&1
+sudo docker inspect <container> --format '{{json .State.Health.Log}}' | python3 -m json.tool | tail -20
+```
+
+After fixing the compose file, must do `down && up` (not just restart).
+
+## App Ignores Environment Variables for DB Type
+
+Some apps (e.g. donetick) require a YAML config file for certain settings even when env vars are set.
+
+For donetick, create `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml`:
+```yaml
+database:
+  type: postgres
+  host: shared-postgres
+  port: 5432
+  user: donetick
+  password: <password>
+  name: donetick
+  migration: true
+jwt:
+  secret: <32-byte-hex>
+  session_time: 168h
+  max_refresh: 168h
+server:
+  port: 2021
+  serve_frontend: true
+```
+
+## JWT Invalid / Login Fails After Restart
+
+Cause: JWT secret changed. Fix: open app in incognito or clear cookies/localStorage.
+
+## Port Already In Use
+
+Common offenders:
+- `5353/udp` — held by system avahi/mDNS. Remove from Plex ports.
+- `3306`, `5432`, `6379` — only one DB stack can bind these.
+
+## Auto-Upgrade Crashes on TrueNAS (exit code 128)
+
+Symptom: `error while creating mount source path '/compose/scripts': mkdir /compose: read-only file system`
+
+Cause: Auto-upgrade container runs docker compose from inside a container where the project dir is mounted as `/compose`. Relative bind mount paths like `./scripts` resolve to `/compose/scripts` on the host. TrueNAS root filesystem is read-only so Docker can't auto-create the directory.
+
+Fix: Pre-create the directory manually on the host before starting the stack:
+```bash
+mkdir -p /mnt/docker-ssd/docker/compose/<stack>/scripts
+```
+
+## Databases Stack Network Name Wrong
+
+Must have `name: ix-databases` at top of databases compose. Verify:
+```bash
+docker network ls | grep databases
+# Should show: ix-databases_shared-databases
+```
+
+## Checking Container Runtime UID
+```bash
+docker run --rm --entrypoint sh <image> -lc 'id'
+```