docs: add full homelab documentation structure

2026-05-05 14:43:09 -05:00
parent ee8a412a69
commit 370998b8d5
20 changed files with 1405 additions and 24 deletions
--- a/docs/troubleshooting/TROUBLESHOOTING.md
+++ b/docs/troubleshooting/TROUBLESHOOTING.md
@@ -0,0 +1,129 @@
+# Troubleshooting Guide
+
+## Permission Denied / Readonly Database / Not Writable
+
+Common causes:
+- App placed on NFS (tank) when it needs local SSD
+- NFS ownership mismatch between host user and container user
+
+Checks:
+1. `docker logs <container> --tail 30`
+2. `ls -la <host-appdata-path>` — check ownership and permissions
+3. `docker run --rm --entrypoint sh <image> -lc 'id'` — check container UID/GID
+
+Fix options:
+- Move appdata to `/mnt/docker-ssd/docker/appdata/<service>` if write-heavy
+- `sudo chmod -R 777 <appdata-path>` for apps with fixed internal user (e.g. shlink)
+- `sudo chown -R <uid>:<gid> <appdata-path>` if UID/GID is known
+
+## SQLite Fails on ZFS with nfsv4 ACLs (SQLITE_CANTOPEN error 14)
+
+Symptom: `unable to open database file: out of memory (14)`
+
+Cause: SQLite cannot acquire POSIX file locks on ZFS datasets configured with `acltype=nfsv4`.
+
+Diagnosis:
+```bash
+sudo zfs get acltype,aclmode,xattr <dataset>
+```
+
+Fix options:
+1. Change acltype: `sudo zfs set acltype=posix <dataset>`
+2. Move the SQLite database to local SSD
+3. Use an app version that supports Postgres instead of SQLite
+
+Note: scriberr CUDA image is shelved for this reason.
+
+## Container Can't Resolve Shared Database Hostname
+
+Symptom: `lookup shared-postgres on 127.0.0.11:53: no such host`
+
+Fix:
+```bash
+docker compose --env-file .env down <service>
+docker compose --env-file .env up -d <service>
+```
+
+Verify network attachment:
+```bash
+docker inspect <container> --format '{{json .NetworkSettings.Networks}}' | tr ',' '\n' | grep -o '"[^"]*"'
+```
+
+## Postgres Auth Fails Over TCP but Works via Socket
+
+Diagnosis:
+```bash
+sudo docker exec shared-postgres psql -h 127.0.0.1 -U <user> -d <db> -W -c "\conninfo"
+```
+
+Fix:
+- Verify pg_hba.conf has `host all all all scram-sha-256`
+- Check DATABASE_URL password matches: `ALTER USER <user> WITH PASSWORD '<pass>';`
+- Avoid special characters in passwords in DATABASE_URLs
+
+## Healthcheck Failing Despite Container Being Healthy
+
+Diagnosis:
+```bash
+sudo docker exec <container> sh -c "which curl; which wget" 2>&1
+sudo docker inspect <container> --format '{{json .State.Health.Log}}' | python3 -m json.tool | tail -20
+```
+
+After fixing the compose file, must do `down && up` (not just restart).
+
+## App Ignores Environment Variables for DB Type
+
+Some apps (e.g. donetick) require a YAML config file for certain settings even when env vars are set.
+
+For donetick, create `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml`:
+```yaml
+database:
+  type: postgres
+  host: shared-postgres
+  port: 5432
+  user: donetick
+  password: <password>
+  name: donetick
+  migration: true
+jwt:
+  secret: <32-byte-hex>
+  session_time: 168h
+  max_refresh: 168h
+server:
+  port: 2021
+  serve_frontend: true
+```
+
+## JWT Invalid / Login Fails After Restart
+
+Cause: JWT secret changed. Fix: open app in incognito or clear cookies/localStorage.
+
+## Port Already In Use
+
+Common offenders:
+- `5353/udp` — held by system avahi/mDNS. Remove from Plex ports.
+- `3306`, `5432`, `6379` — only one DB stack can bind these.
+
+## Auto-Upgrade Crashes on TrueNAS (exit code 128)
+
+Symptom: `error while creating mount source path '/compose/scripts': mkdir /compose: read-only file system`
+
+Cause: Auto-upgrade container runs docker compose from inside a container where the project dir is mounted as `/compose`. Relative bind mount paths like `./scripts` resolve to `/compose/scripts` on the host. TrueNAS root filesystem is read-only so Docker can't auto-create the directory.
+
+Fix: Pre-create the directory manually on the host before starting the stack:
+```bash
+mkdir -p /mnt/docker-ssd/docker/compose/<stack>/scripts
+```
+
+## Databases Stack Network Name Wrong
+
+Must have `name: ix-databases` at top of databases compose. Verify:
+```bash
+docker network ls | grep databases
+# Should show: ix-databases_shared-databases
+```
+
+## Checking Container Runtime UID
+```bash
+docker run --rm --entrypoint sh <image> -lc 'id'
+```