# Troubleshooting Guide

## Permission Denied / Readonly Database / Not Writable

Common causes:
- App placed on NFS (tank) when it needs local SSD
- NFS ownership mismatch between host user and container user

Checks:
1. `docker logs <container> --tail 30`
2. `ls -la <host-appdata-path>` — check ownership and permissions
3. `docker run --rm --entrypoint sh <image> -lc 'id'` — check container UID/GID

Fix options:
- Move appdata to `/mnt/docker-ssd/docker/appdata/<service>` if write-heavy
- `sudo chmod -R 777 <appdata-path>` for apps with fixed internal user (e.g. shlink)
- `sudo chown -R <uid>:<gid> <appdata-path>` if UID/GID is known

## SQLite Fails on ZFS with nfsv4 ACLs (SQLITE_CANTOPEN error 14)

Symptom: `unable to open database file: out of memory (14)`

Cause: SQLite cannot acquire POSIX file locks on ZFS datasets configured with `acltype=nfsv4`.

Diagnosis:
```bash
sudo zfs get acltype,aclmode,xattr <dataset>
```

Fix options:
1. Change acltype: `sudo zfs set acltype=posix <dataset>`
2. Move the SQLite database to local SSD
3. Use an app version that supports Postgres instead of SQLite

Note: scriberr CUDA image is shelved for this reason.

## Container Can't Resolve Shared Database Hostname

Symptom: `lookup shared-postgres on 127.0.0.11:53: no such host`

Fix:
```bash
docker compose --env-file .env down <service>
docker compose --env-file .env up -d <service>
```

Verify network attachment:
```bash
docker inspect <container> --format '{{json .NetworkSettings.Networks}}' | tr ',' '\n' | grep -o '"[^"]*"'
```

## Postgres Auth Fails Over TCP but Works via Socket

Diagnosis:
```bash
sudo docker exec shared-postgres psql -h 127.0.0.1 -U <user> -d <db> -W -c "\conninfo"
```

Fix:
- Verify pg_hba.conf has `host all all all scram-sha-256`
- Check DATABASE_URL password matches: `ALTER USER <user> WITH PASSWORD '<pass>';`
- Avoid special characters in passwords in DATABASE_URLs

## Healthcheck Failing Despite Container Being Healthy

Diagnosis:
```bash
sudo docker exec <container> sh -c "which curl; which wget" 2>&1
sudo docker inspect <container> --format '{{json .State.Health.Log}}' | python3 -m json.tool | tail -20
```

After fixing the compose file, must do `down && up` (not just restart).

## App Ignores Environment Variables for DB Type

Some apps (e.g. donetick) require a YAML config file for certain settings even when env vars are set.

For donetick, create `/mnt/tank/docker/appdata/donetick/config/selfhosted.yaml`:
```yaml
database:
  type: postgres
  host: shared-postgres
  port: 5432
  user: donetick
  password: <password>
  name: donetick
  migration: true
jwt:
  secret: <32-byte-hex>
  session_time: 168h
  max_refresh: 168h
server:
  port: 2021
  serve_frontend: true
```

## JWT Invalid / Login Fails After Restart

Cause: JWT secret changed. Fix: open app in incognito or clear cookies/localStorage.

## Port Already In Use

Common offenders:
- `5353/udp` — held by system avahi/mDNS. Remove from Plex ports.
- `3306`, `5432`, `6379` — only one DB stack can bind these.

## Auto-Upgrade Crashes on TrueNAS (exit code 128)

Symptom: `error while creating mount source path '/compose/scripts': mkdir /compose: read-only file system`

Cause: Auto-upgrade container runs docker compose from inside a container where the project dir is mounted as `/compose`. Relative bind mount paths like `./scripts` resolve to `/compose/scripts` on the host. TrueNAS root filesystem is read-only so Docker can't auto-create the directory.

Fix: Pre-create the directory manually on the host before starting the stack:
```bash
mkdir -p /mnt/docker-ssd/docker/compose/<stack>/scripts
```

## Databases Stack Network Name Wrong

Must have `name: ix-databases` at top of databases compose. Verify:
```bash
docker network ls | grep databases
# Should show: ix-databases_shared-databases
```

## Checking Container Runtime UID
```bash
docker run --rm --entrypoint sh <image> -lc 'id'
```