truenas-stacks/docs/planning/SERENITY_DOCKER_AUDIT.md

Serenity Docker Audit

Status: live-audited baseline for cleanup and migration planning, refreshed after the GameVault/RomM cutover cleanup and the broader keep/move/retire placement review.

Last live verification: 2026-05-26

Access path used

Live audit now succeeds directly from NOMAD using the configured host alias:

• ssh serenity

Older discovery in this document used the PD pivot path before direct local SSH was wired up.

What is currently running on Serenity

Keep for now until PD storage ownership changes

These are the containers whose current placement still makes operational sense because Serenity owns the active media/torrent locality today.

• qbit
• GluetunVPN
• qbit_manage
• prowlarr
• sonarr
• sonarr-anime
• radarr
• lidarr
• readarr
• readarr-epub
• bazarr
• autobrr
• unpackerr
• Notifiarr
• shelfmark

Common pattern observed from live mounts:

• these stacks are bound heavily to /mnt/user/data
• torrent state and backup artifacts live under Serenity-owned paths like /mnt/user/data/torrents, /mnt/user/data/BT_backup, and appdata under /mnt/user/appdata/*
• current path locality still argues for keeping them on Serenity until PD directly owns the disks and final media paths
• shelfmark is not a generic "random app" here; its live mounts also tie it to Serenity-owned /mnt/user/data, audiobook ingest paths, and torrent/library paths

Keep temporarily, but plan to move or collapse later

These are live today, but they are not good long-term reasons to keep Serenity alive after PD is rebuilt.

• reranker
  • currently on Serenity because CPU-only TEI was moved off PD
  • planned long-term home: PD if PD remains the AI control-plane/core host
  • live runtime uses /mnt/user/appdate/reranker, not /mnt/user/appdata/reranker; treat that as a migration hazard that must be handled deliberately
• technitium-dns-pilot
  • current backup Technitium node on Serenity (10.5.30.10)
  • long-term: keep at least one off-PD backup resolver, but that does not have to remain on Serenity forever
• Newt
  • live on Serenity now
  • operator confirmed it is still needed, so it should be preserved during cleanup and only rehomed deliberately later
• Hawser
  • helper/observability tooling, not a reason to preserve Serenity as a host role
• dockersocket
  • Hawser helper sidecar only; keep or move with Hawser rather than treating it as a standalone host role
• netdata
  • useful while Serenity remains live, not a reason to keep Serenity permanently
• Wizarr
  • this is a normal app workload, not a storage-appliance-only workload
  • long-term candidate for PD well before any final Serenity storage retirement if you want to peel off low-risk miscellaneous apps early

Placement verdict by service group

Keep on Serenity until the storage-locality cutover is designed

• torrent / download lane:
  • qbit
  • GluetunVPN
  • qbit_manage
  • autobrr
  • unpackerr
• media automation lane:
  • prowlarr
  • sonarr
  • sonarr-anime
  • radarr
  • lidarr
  • readarr
  • readarr-epub
  • bazarr
• locality-adjacent helpers:
  • Notifiarr
  • shelfmark

Reason:

• all of these either mount Serenity-owned /mnt/user/data directly or depend on the downloader/media-path locality that still lives on Serenity today
• moving them before PD owns the storage locally would just trade one cleanup project for a fragile NFS-path rewrite project

Move off Serenity whenever the destination design is ready

• Wizarr
• Hawser
• dockersocket
• netdata

Reason:

• these are not durable storage-owner roles
• they can move independently of the qbit/ARR locality lane once you decide whether PD should absorb them or whether one of them is no longer worth keeping

Keep as explicit temporary exceptions, then redesign last

• reranker
• technitium-dns-pilot
• Newt

Reason:

• these are the remaining special cases that still justify Serenity for non-media reasons
• each one has a real redesign question attached:
  • reranker: whether PD should re-absorb AI helper services later
  • Technitium backup node: which off-PD host should own the durable backup resolver role
  • Newt: when Serenity-hosted Pangolin resources are deliberately rehomed so the local tunnel is no longer required

Retire candidates

These should be treated as cleanup targets unless a specific live dependency is rediscovered.

• Unraid-Cloudflared-Tunnel
  • live audit on 2026-05-25 showed a remote-managed Cloudflare Tunnel with stale legacy 192.168.1.x origins and zero observed proxied requests on the current run
  • stopped and removed on 2026-05-25 after post-stop checks showed sampled public hostnames remained healthy without it
• binhex-official-pihole
• pihole-serenity
• unbound-pihole-serenity
• keepalived-pihole-serenity
  • these were legacy DNS/HA remnants once Technitium became the intended resolver strategy
  • mixed-host DNS verification passed during retirement on 2026-05-25, and the Serenity Pi-hole HA stack was then stopped and removed without immediate DNS regression
• the old local DB pair (postgresql15 and MariaDB-Official) was retired on 2026-05-25 after PD validation and rollback bundle creation
• the stale stopped source app containers (romm and GameVault) were removed on 2026-05-26
• retained rollback paths still exist on Serenity:
  • /mnt/user/backups/doris/serenity-d7-db-retire-20260525-213531
  • /mnt/user/appdata/romm
  • /mnt/user/appdata/gamevault
  • /mnt/user/appdata/mariadb-official
  • /mnt/cache/appdata/postgresql15

Not-running / stale containers seen in docker ps -a

These did not appear live and should be reviewed for deletion after confirming their data is not needed.

Created only (retired on 2026-05-25 after metadata verification):

• calibre-web
• SuggestArr
• Cleanuparr
• calibre
• agregarr

Exited:

• none remaining from the previously audited stale set; Huntarr, omegabrr, romm, and GameVault have now been removed

Current result:

• no stale created/exited containers remain in docker ps -a

Live project roots observed

Compose Manager project roots found on Serenity:

• /boot/config/plugins/compose.manager/projects/pihole-ha
• /boot/config/plugins/compose.manager/projects/re-ranker

Direct compose file under appdata:

• /mnt/user/appdata/technitium-serenity/docker-compose.yaml

Operational implication:

• much of Serenity appears to be managed through Unraid Docker templates or ad-hoc container definitions, not a clean compose-per-stack layout
• cleanup work should expect drift between repo planning docs, backup stack snapshots, and the actual Unraid runtime inventory

Important live mount observations

Examples from the live container inspection:

• qbit binds /mnt/user/data -> /data
• qbit_manage binds /mnt/user/data, /mnt/user/data/BT_backup, and /mnt/user/appdata/qbit_manage
• shelfmark binds /mnt/user/data, /mnt/user/data/media/books/Audiobooks, /mnt/user/data/media/books/ingest, and /mnt/user/data/torrents
• most ARR-family services bind /mnt/user/data
• reranker binds /mnt/user/appdate/reranker -> /data
• Wizarr only binds its own appdata/database paths and has no meaningful storage-locality reason to stay on Serenity
• Hawser + dockersocket are local helper tooling, not media-path owners

Notable typo/risk:

• reranker is mounted from /mnt/user/appdate/reranker (note appdate, not appdata)
• /mnt/user/appdate/reranker exists live; /mnt/user/appdata/reranker does not currently exist
• treat that as an intentional-on-disk reality for now, but fix the naming deliberately during a future migration rather than by surprise during unrelated cleanup

Recommended next work order

Phase 1: completed cleanup baseline

• dead Cloudflared / Pi-hole / stale-container cleanup is done
• the old local DB pair and source GameVault/RomM containers are already retired
• current live work should start from the narrower remaining set, not re-open that cleanup unless new evidence appears

Phase 2: peel off the low-risk non-locality apps

• move Wizarr to PD when convenient
• move or retire Hawser + dockersocket
• decide whether netdata still provides unique value once the broader monitoring stack is considered

Phase 3: preserve the only things that currently justify Serenity

Until PD owns the storage locally, keep the torrent/media-ingest locality group together:

• qBittorrent/VPN path
• ARR family
• qbit_manage
• autobrr
• unpackerr
• related helpers like Notifiarr and Shelfmark

Phase 4: cut over after PD storage migration

Once PD directly owns the relevant media/torrent datasets:

• move qbit + ARR locality to PD
• keep path locality on the box that owns the disks
• move or retire the locality-adjacent helpers that were intentionally left with that lane

Phase 5: redesign the special cases last

• move reranker to PD if desired, while normalizing the appdate/appdata path naming intentionally
• keep at least one off-PD Technitium node somewhere, preferably NOMAD if Serenity is retiring
• re-home or retire Serenity-local Newt only after Pangolin dependencies are deliberately redesigned

Kanban-ready workstreams

Epic A — Serenity live inventory normalization

• capture docker ps, mounts, ports, and remaining host-role assumptions
• keep the repo docs aligned with the actual Unraid runtime inventory
• treat ad-hoc Unraid container definitions as a drift risk until they are intentionally replaced

Epic B — low-risk miscellaneous app peel-off

• move Wizarr to PD or retire it
• decide whether Hawser + dockersocket should move as a pair or just disappear
• decide whether netdata still earns its keep alongside the broader monitoring stack

Epic C — torrent/media-locality preservation until PD cutover

• keep qbit/ARR stack stable on Serenity for now
• document exact media/torrent paths that must exist on PD before migration
• prevent premature moves that would recreate NFS-path weirdness

Epic D — special-case redesign

• normalize the reranker path oddity during a planned move rather than an incidental cleanup
• decide where the off-PD backup Technitium role should live after Serenity
• re-home Pangolin/Newt dependencies last, not during the media cutover

Epic E — final Serenity retirement

• move remaining wanted apps to PD or NOMAD
• preserve only the intended backup-DNS failure-domain role off PD
• decommission Serenity once no production path depends on it

Resolved operator decisions

Resolved on 2026-05-25:

1. No app should deliberately remain on Serenity once PD owns the disks locally.
2. Technitium covers the desired DNS role; the legacy Serenity Pi-hole stack should be treated as removable.
3. Newt on Serenity is still needed and should not be treated as cleanup.
4. GameVault and RomM should migrate rather than be pruned.
5. End-state remains:
   • move qbit + ARR family to PD after storage cutover
   • leave no intentional production app role on Serenity
   • retire Serenity entirely
6. Wizarr, Hawser/dockersocket, and probably netdata are explicitly low-risk move/retire candidates that do not need to wait for the storage cutover.

Remaining verification questions

• No additional live dependency beyond the retired GameVault/RomM pair was surfaced during the quick DB audit; if that changes later, treat it as a rediscovery against rollback artifacts rather than a live-stack blocker.
• Decide when the retained DB/appdata rollback artifacts are old enough to archive more aggressively or finally delete.
• Normalize the reranker path naming during the eventual move: preserve the live /mnt/user/appdate/reranker data now, but decide whether the destination should standardize back to appdata.
• Serenity's Pangolin/Newt health-check drift is fixed now; any future Newt rehome is an architecture task, not a stale-health-check incident response.

Pangolin / Newt live remediation status

Live verification on 2026-05-25 showed Serenity Newt still probing stale pre-renumbering health-check URLs like http://10.5.1.5:8787/ even though the Pangolin target objects already showed ip=localhost or 10.5.30.5 for those resources.

Affected target IDs observed live:

• 15 (autobrr)
• 20 (notifiarr)
• 25 (readarr)
• 29 (wizarr)
• 56 (readarr-epub)
• 57 (sonarr-anime)
• 58 (romm)
• 69 (gamevault)

Diagnostic probe performed live on Serenity:

• temporarily added 10.5.1.5/32 to br0
• this immediately restored health for several targets, proving the stale hcHostname diagnosis

However, the old 10.5.1.5 address is no longer allowed on that VLAN, so the alias was removed again.

Verification after removal:

• ip addr del 10.5.1.5/32 dev br0
• Newt immediately resumed failures such as:
  • target 56 -> http://10.5.1.5:8788/
  • target 57 -> http://10.5.1.5:8990/
  • target 15 -> http://10.5.1.5:7474/
  • target 25 -> http://10.5.1.5:8787/

Interpretation:

• the alias was useful as a proof-of-cause test only
• it is not an acceptable steady-state fix here
• the real remaining task is to authoritatively rewrite the stale Pangolin hcHostname values away from 10.5.1.5

Pangolin / Newt authoritative fix completed

Follow-up live mutation on 2026-05-25 rewrote the remaining Serenity site targets that were still drifting on stale 10.5.1.5 health checks:

• target 25 (readarr) -> ip=10.5.30.5, hcHostname=10.5.30.5, hcHealth=healthy
• target 56 (readarr-epub) -> ip=10.5.30.5, hcHostname=10.5.30.5, hcHealth=healthy
• target 57 (sonarr-anime) -> ip=10.5.30.5, hcHostname=10.5.30.5, hcHealth=healthy

Post-fix verification:

• docker exec Newt wget http://10.5.30.5:8787/, :8788/, and :8990/ all succeeded from inside Serenity's Newt container
• public probes for readarr.paccoco.com, readarr-epub.paccoco.com, and sonarr-anime.paccoco.com all returned the expected Pangolin-auth redirect flow
• live Pangolin API inventory for site serenity no longer contains any target with ip=10.5.1.5 or hcHostname=10.5.1.5

Current steady state for the audited Serenity-hosted Pangolin targets (15, 20, 25, 29, 56, 57, 58, 69):

• all now show ip=10.5.30.5
• all now show hcHostname=10.5.30.5
• all now report hcHealth=healthy

Rollback / evidence artifacts:

• pre-change backup for targets 56 and 57: /home/fizzlepoof/pangolin-target-backup-serenity-hc-authoritative-20260525T204548Z.json
• post-fix snapshot for audited targets: /home/fizzlepoof/pangolin-target-snapshot-serenity-post-hc-fix-20260525T204741Z.json

Database dependency findings

Live inspection before the GameVault/RomM retirement pointed to:

• GameVault -> local postgresql15
  • DB_HOST=10.5.30.5
  • DB_PORT=5432
• RomM -> local MariaDB-Official
  • DB_HOST=10.5.30.5
  • DB_PORT=3306

No immediate database dependency was surfaced from the quick live environment check for:

• Wizarr
• Shelfmark
• Notifiarr

Operational implication:

• postgresql15 should currently be treated as a GameVault dependency until proven otherwise.
• MariaDB-Official should currently be treated as a RomM dependency until proven otherwise.
• those databases can likely retire once their dependent apps are migrated to PD and verified there.
• the legacy Pi-hole containers can be scheduled for removal at the next cleanup window.