truenas-stacks/docs/troubleshooting/TROUBLESHOOTING.md

Troubleshooting Guide

Permission Denied / Readonly Database / Not Writable

Common causes:

• App placed on NFS (tank) when it needs local SSD
• NFS ownership mismatch between host user and container user

Checks:

1. docker logs <container> --tail 30
2. ls -la <host-appdata-path> — check ownership and permissions
3. docker run --rm --entrypoint sh <image> -lc 'id' — check container UID/GID

Fix options:

• Move appdata to /mnt/docker-ssd/docker/appdata/<service> if write-heavy
• sudo chmod -R 777 <appdata-path> for apps with fixed internal user (e.g. shlink)
• sudo chown -R <uid>:<gid> <appdata-path> if UID/GID is known

SQLite Fails on ZFS with nfsv4 ACLs (SQLITE_CANTOPEN error 14)

Symptom: unable to open database file: out of memory (14)

Cause: SQLite cannot acquire POSIX file locks on ZFS datasets configured with acltype=nfsv4.

Diagnosis:

sudo zfs get acltype,aclmode,xattr <dataset>

Fix options:

1. Change acltype: sudo zfs set acltype=posix <dataset>
2. Move the SQLite database to local SSD
3. Use an app version that supports Postgres instead of SQLite

Note: scriberr CUDA image is shelved for this reason.

Container Can't Resolve Shared Database Hostname

Symptom: lookup shared-postgres on 127.0.0.11:53: no such host

Fix:

docker compose --env-file .env down <service>
docker compose --env-file .env up -d <service>

Verify network attachment:

docker inspect <container> --format '{{json .NetworkSettings.Networks}}' | tr ',' '\n' | grep -o '"[^"]*"'

Postgres Auth Fails Over TCP but Works via Socket

Diagnosis:

sudo docker exec shared-postgres psql -h 127.0.0.1 -U <user> -d <db> -W -c "\conninfo"

Fix:

• Verify pg_hba.conf has host all all all scram-sha-256
• Check DATABASE_URL password matches: ALTER USER <user> WITH PASSWORD '<pass>';
• Avoid special characters in passwords in DATABASE_URLs

Healthcheck Failing Despite Container Being Healthy

Diagnosis:

sudo docker exec <container> sh -c "which curl; which wget" 2>&1
sudo docker inspect <container> --format '{{json .State.Health.Log}}' | python3 -m json.tool | tail -20

After fixing the compose file, must do down && up (not just restart).

App Ignores Environment Variables for DB Type

Some apps (e.g. donetick) require a YAML config file for certain settings even when env vars are set.

For donetick, create /mnt/tank/docker/appdata/donetick/config/selfhosted.yaml:

database:
  type: postgres
  host: shared-postgres
  port: 5432
  user: donetick
  password: <password>
  name: donetick
  migration: true
jwt:
  secret: <32-byte-hex>
  session_time: 168h
  max_refresh: 168h
server:
  port: 2021
  serve_frontend: true

JWT Invalid / Login Fails After Restart

Cause: JWT secret changed. Fix: open app in incognito or clear cookies/localStorage.

Port Already In Use

Common offenders:

• 5353/udp — held by system avahi/mDNS. Remove from Plex ports.
• 3306, 5432, 6379 — only one DB stack can bind these.

Auto-Upgrade Crashes on TrueNAS (exit code 128)

Symptom: error while creating mount source path '/compose/scripts': mkdir /compose: read-only file system

Cause: Auto-upgrade container runs docker compose from inside a container where the project dir is mounted as /compose. Relative bind mount paths like ./scripts resolve to /compose/scripts on the host. TrueNAS root filesystem is read-only so Docker can't auto-create the directory.

Fix: Pre-create the directory manually on the host before starting the stack:

mkdir -p /mnt/docker-ssd/docker/compose/<stack>/scripts

Databases Stack Network Name Wrong

Must have name: ix-databases at top of databases compose. Verify:

docker network ls | grep databases
# Should show: ix-databases_shared-databases

Checking Container Runtime UID

docker run --rm --entrypoint sh <image> -lc 'id'