truenas-stacks/docs/planning/SERENITY_DOCKER_AUDIT.md

# Serenity Docker Audit

Status: live-audited baseline for cleanup and migration planning.

Last live verification: 2026-05-25

## Access path used

Direct SSH from this session to `root@10.5.30.5` failed with `Permission denied`.
Live audit succeeded by hopping through PD:

- `ssh pd`
- then `ssh -i /home/truenas_admin/.ssh/serenity_backup_ed25519 root@10.5.30.5`

That means the inventory below is grounded in the current live runtime, but through PD's trusted Serenity backup key rather than direct local SSH.

## What is currently running on Serenity

### Keep for now until PD storage ownership changes
These are the containers whose current placement still makes operational sense because Serenity owns the active media/torrent locality today.

- `qbit`
- `GluetunVPN`
- `qbit_manage`
- `prowlarr`
- `sonarr`
- `sonarr-anime`
- `radarr`
- `lidarr`
- `readarr`
- `readarr-epub`
- `bazarr`
- `autobrr`
- `unpackerr`
- `Notifiarr`
- `shelfmark`

Common pattern observed from live mounts:
- these stacks are bound heavily to `/mnt/user/data`
- torrent state and backup artifacts live under Serenity-owned paths like `/mnt/user/data/torrents`, `/mnt/user/data/BT_backup`, and appdata under `/mnt/user/appdata/*`
- current path locality still argues for keeping them on Serenity until PD directly owns the disks and final media paths

### Keep temporarily, but plan to move or collapse later
These are live today, but they are not good long-term reasons to keep Serenity alive after PD is rebuilt.

- `reranker`
  - currently on Serenity because CPU-only TEI was moved off PD
  - planned long-term home: PD if PD remains the AI control-plane/core host
- `technitium-dns-pilot`
  - current backup Technitium node on Serenity (`10.5.30.10`)
  - long-term: keep at least one off-PD backup resolver, but that does not have to remain on Serenity forever
- `Newt`
  - live on Serenity now
  - operator confirmed it is still needed, so it should be preserved during cleanup and only rehomed deliberately later
- `Hawser`
  - helper/observability tooling, not a reason to preserve Serenity as a host role
- `netdata`
  - useful while Serenity remains live, not a reason to keep Serenity permanently
- `GameVault`
- `romm`
- `Wizarr`
  - these are normal app workloads, not storage-appliance-only workloads
  - long-term candidates for PD once the storage and app-host migration is ready

### Retire candidates
These should be treated as cleanup targets unless a specific live dependency is rediscovered.

- `Unraid-Cloudflared-Tunnel`
  - live audit on 2026-05-25 showed a remote-managed Cloudflare Tunnel with stale legacy `192.168.1.x` origins and zero observed proxied requests on the current run
  - stopped and removed on 2026-05-25 after post-stop checks showed sampled public hostnames remained healthy without it
- `binhex-official-pihole`
- `pihole-serenity`
- `unbound-pihole-serenity`
- `keepalived-pihole-serenity`
  - these were legacy DNS/HA remnants once Technitium became the intended resolver strategy
  - mixed-host DNS verification passed during retirement on 2026-05-25, and the Serenity Pi-hole HA stack was then stopped and removed without immediate DNS regression
- `postgresql15`
- `MariaDB-Official`
  - both are live, but they are suspicious because current docs position PD as the shared database home
  - do not delete blindly; first identify whether any Serenity-only apps still depend on them
  - treat them as consolidation candidates, not permanent Serenity residents

## Not-running / stale containers seen in `docker ps -a`
These did not appear live and should be reviewed for deletion after confirming their data is not needed.

Created only (retired on 2026-05-25 after metadata verification):
- `calibre-web`
- `SuggestArr`
- `Cleanuparr`
- `calibre`
- `agregarr`

Exited:
- `Huntarr` — exited 6 weeks ago
- `omegabrr` — exited 4 months ago

These are strong clutter candidates.

## Live project roots observed

Compose Manager project roots found on Serenity:
- `/boot/config/plugins/compose.manager/projects/pihole-ha`
- `/boot/config/plugins/compose.manager/projects/re-ranker`

Direct compose file under appdata:
- `/mnt/user/appdata/technitium-serenity/docker-compose.yaml`

Operational implication:
- much of Serenity appears to be managed through Unraid Docker templates or ad-hoc container definitions, not a clean compose-per-stack layout
- cleanup work should expect drift between repo planning docs, backup stack snapshots, and the actual Unraid runtime inventory

## Important live mount observations

Examples from the live container inspection:
- `qbit` binds `/mnt/user/data -> /data`
- `qbit_manage` binds `/mnt/user/data`, `/mnt/user/data/BT_backup`, and `/mnt/user/appdata/qbit_manage`
- `shelfmark` binds `/mnt/user/data`, `/mnt/user/data/media/books/Audiobooks`, `/mnt/user/data/media/books/ingest`, and `/mnt/user/data/torrents`
- most ARR-family services bind `/mnt/user/data`
- `reranker` binds `/mnt/user/appdate/reranker -> /data`

Notable typo/risk:
- `reranker` is mounted from `/mnt/user/appdate/reranker` (note `appdate`, not `appdata`)
- verify whether that path is intentional or an unnoticed typo before migration work touches it

## Recommended cleanup order

### Phase 1: remove obvious deadwood
1. verify no dependency remains on `Unraid-Cloudflared-Tunnel`
2. inspect whether the legacy Pi-hole stack still serves any traffic or admin purpose
3. remove stale created/exited containers that have no active role:
   - `calibre-web`
   - `calibre`
   - `SuggestArr`
   - `Cleanuparr`
   - `agregarr`
   - `Huntarr`
   - `omegabrr`

### Phase 2: identify silent database dependencies
Before touching `postgresql15` or `MariaDB-Official`, map which containers point at them.

Questions to answer:
- which apps on Serenity still use local Postgres?
- which apps on Serenity still use local MariaDB?
- are GameVault, RomM, Shelfmark, or Wizarr still backed by these local DBs?
- can any of those apps be moved to PD's shared DB stack cleanly?

### Phase 3: preserve the only things that currently justify Serenity
Until PD owns the storage locally, keep the torrent/media-ingest locality group together:
- qBittorrent/VPN path
- ARR family
- qbit_manage
- autobrr
- unpackerr
- related helpers like Notifiarr and Shelfmark

### Phase 4: cut over after PD storage migration
Once PD directly owns the relevant media/torrent datasets:
- move qbit + ARR locality to PD
- move reranker to PD if desired
- keep at least one off-PD Technitium node somewhere, preferably NOMAD if Serenity is retiring
- move or retire the miscellaneous app workloads still left on Serenity
- shut Serenity down as a permanent app host

## Kanban-ready workstreams

### Epic A — Serenity live inventory normalization
- capture `docker ps`, `docker ps -a`, mounts, ports, and database dependencies
- map every live container to one of: keep-now, move-to-PD, retire-now, or remove-as-stale
- verify whether any runtime definitions exist only in Unraid templates and not in repo-managed compose

### Epic B — obvious dead container cleanup
- remove dead Cloudflared tunnel
- remove stale created/exited clutter containers
- verify and retire legacy Pi-hole/Keepalived/Unbound stack if no hidden dependency remains

### Epic C — database dependency audit
- identify which Serenity apps use `postgresql15`
- identify which Serenity apps use `MariaDB-Official`
- decide whether those DBs move to PD shared databases or are retired with the apps

### Epic D — torrent/media-locality preservation until PD cutover
- keep qbit/ARR stack stable on Serenity for now
- document exact media/torrent paths that must exist on PD before migration
- prevent premature moves that would recreate NFS-path weirdness

### Epic E — final Serenity retirement
- move remaining wanted apps to PD or NOMAD
- preserve only the intended backup-DNS failure-domain role off PD
- decommission Serenity once no production path depends on it

## Resolved operator decisions

Resolved on 2026-05-25:

1. No app should deliberately remain on Serenity once PD owns the disks locally.
2. Technitium covers the desired DNS role; the legacy Serenity Pi-hole stack should be treated as removable.
3. `Newt` on Serenity is still needed and should not be treated as cleanup.
4. `GameVault` and `RomM` should migrate rather than be pruned.
5. End-state remains:
   - move qbit + ARR family to PD after storage cutover
   - leave no intentional production app role on Serenity
   - retire Serenity entirely

## Remaining verification questions

- Confirm whether anything beyond `GameVault` still depends on `postgresql15`.
- Confirm whether anything beyond `RomM` still depends on `MariaDB-Official`.
- Is `/mnt/user/appdate/reranker` intentional, or a typo that should be corrected before migration?
- When Pangolin API write auth is available again, rewrite Serenity target health-check hostnames away from stale `10.5.1.5` so the runtime alias can be removed cleanly.

## Pangolin / Newt live remediation status

Live verification on 2026-05-25 showed Serenity Newt still probing stale pre-renumbering health-check URLs like `http://10.5.1.5:8787/` even though the Pangolin target objects already showed `ip=localhost` or `10.5.30.5` for those resources.

Affected target IDs observed live:
- `15` (`autobrr`)
- `20` (`notifiarr`)
- `25` (`readarr`)
- `29` (`wizarr`)
- `56` (`readarr-epub`)
- `57` (`sonarr-anime`)
- `58` (`romm`)
- `69` (`gamevault`)

Diagnostic probe performed live on Serenity:
- temporarily added `10.5.1.5/32` to `br0`
- this immediately restored health for several targets, proving the stale `hcHostname` diagnosis

However, the old `10.5.1.5` address is no longer allowed on that VLAN, so the alias was removed again.

Verification after removal:
- `ip addr del 10.5.1.5/32 dev br0`
- Newt immediately resumed failures such as:
  - target `56` -> `http://10.5.1.5:8788/`
  - target `57` -> `http://10.5.1.5:8990/`
  - target `15` -> `http://10.5.1.5:7474/`
  - target `25` -> `http://10.5.1.5:8787/`

Interpretation:
- the alias was useful as a proof-of-cause test only
- it is not an acceptable steady-state fix here
- the real remaining task is to authoritatively rewrite the stale Pangolin `hcHostname` values away from `10.5.1.5`

## Pangolin / Newt authoritative fix completed

Follow-up live mutation on 2026-05-25 rewrote the remaining Serenity site targets that were still drifting on stale `10.5.1.5` health checks:
- target `25` (`readarr`) -> `ip=10.5.30.5`, `hcHostname=10.5.30.5`, `hcHealth=healthy`
- target `56` (`readarr-epub`) -> `ip=10.5.30.5`, `hcHostname=10.5.30.5`, `hcHealth=healthy`
- target `57` (`sonarr-anime`) -> `ip=10.5.30.5`, `hcHostname=10.5.30.5`, `hcHealth=healthy`

Post-fix verification:
- `docker exec Newt wget http://10.5.30.5:8787/`, `:8788/`, and `:8990/` all succeeded from inside Serenity's `Newt` container
- public probes for `readarr.paccoco.com`, `readarr-epub.paccoco.com`, and `sonarr-anime.paccoco.com` all returned the expected Pangolin-auth redirect flow
- live Pangolin API inventory for site `serenity` no longer contains any target with `ip=10.5.1.5` or `hcHostname=10.5.1.5`

Current steady state for the audited Serenity-hosted Pangolin targets (`15`, `20`, `25`, `29`, `56`, `57`, `58`, `69`):
- all now show `ip=10.5.30.5`
- all now show `hcHostname=10.5.30.5`
- all now report `hcHealth=healthy`

Rollback / evidence artifacts:
- pre-change backup for targets `56` and `57`: `/home/fizzlepoof/pangolin-target-backup-serenity-hc-authoritative-20260525T204548Z.json`
- post-fix snapshot for audited targets: `/home/fizzlepoof/pangolin-target-snapshot-serenity-post-hc-fix-20260525T204741Z.json`

## Database dependency findings

Live inspection of container environments currently points to:

- `GameVault` -> local `postgresql15`
  - `DB_HOST=10.5.30.5`
  - `DB_PORT=5432`
- `RomM` -> local `MariaDB-Official`
  - `DB_HOST=10.5.30.5`
  - `DB_PORT=3306`

No immediate database dependency was surfaced from the quick live environment check for:

- `Wizarr`
- `Shelfmark`
- `Notifiarr`

Operational implication:

- `postgresql15` should currently be treated as a `GameVault` dependency until proven otherwise.
- `MariaDB-Official` should currently be treated as a `RomM` dependency until proven otherwise.
- those databases can likely retire once their dependent apps are migrated to PD and verified there.
- the legacy Pi-hole containers can be scheduled for removal at the next cleanup window.