truenas-stacks/docs/planning/SERENITY_CLEANUP_WAVE_1.md

Serenity Cleanup Wave 1 Plan

Status: approved planning baseline for the first safe cleanup pass on Serenity.

Goal

Reduce obvious legacy clutter on Serenity without breaking the still-needed torrent/media-locality group or the still-needed Serenity Newt path.

This wave is intentionally conservative. It does not move qBittorrent/ARR off Serenity yet. It does not retire Serenity yet. It does not delete databases that still back live apps.

Operator decisions already resolved

• Nothing should intentionally remain on Serenity after PD owns the disks locally.
• Technitium already covers the DNS role John wants.
• Serenity Pi-hole remnants should be treated as removable.
• Serenity Newt is still needed and must be preserved.
• GameVault and RomM should migrate, not be pruned.
• Final end-state remains:
  • move qbit + ARR family to PD after storage cutover
  • leave no intentional production app role on Serenity
  • retire Serenity entirely

Scope of cleanup wave 1

Wave 1 includes only these categories:

1. Remove legacy DNS clutter that should no longer be serving production traffic.
2. Remove obviously stale created/exited containers.
3. Document migration order for the two database-backed apps that should move later.

Wave 1 explicitly excludes:

• qbit
• GluetunVPN
• qbit_manage
• prowlarr
• sonarr
• sonarr-anime
• radarr
• lidarr
• readarr
• readarr-epub
• bazarr
• autobrr
• unpackerr
• Notifiarr
• shelfmark
• Newt
• technitium-dns-pilot
• GameVault
• romm
• reranker

Live facts this plan is based on

From the live Serenity audit:

• ARR/torrent locality is still tied to /mnt/user/data
• GameVault points at local Postgres on 10.5.30.5:5432
• RomM points at local MariaDB on 10.5.30.5:3306
• live container env inspection still shows those explicit DB_HOST/DB_PORT bindings on 2026-05-25
• quick live checks did not surface immediate DB dependencies for Wizarr, Shelfmark, or Notifiarr
• Newt is still needed
• legacy Pi-hole containers are still running even though Technitium is now the intended DNS path
• Serenity Newt-backed Pangolin routes still had stale health-check hostnames pointing at old 10.5.1.5 even though their target IPs had already been rewritten to localhost/10.5.30.5
• a temporary 10.5.1.5/32 alias on br0 validated the diagnosis, but it was removed because the old IP is no longer allowed on that VLAN
• authoritative Pangolin cleanup was then completed by rewriting the audited Serenity target set to 10.5.30.5 for both routing and health checks, removing the need for any legacy-IP workaround

Wave 1 current execution status

Live execution on 2026-05-25 established:

• Huntarr and omegabrr were low-risk stale stopped containers and were removed from Serenity
• the recent non-running Created containers (calibre-web, SuggestArr, Cleanuparr, calibre, agregarr) were metadata-verified and then removed after confirming they were still only Created, had restart=no, and were not live workloads
• Unraid-Cloudflared-Tunnel was audited, stopped, and removed after verification showed it was only transport-alive and no current public traffic depended on it
• the Serenity Pi-hole HA stack (binhex-official-pihole, pihole-serenity, unbound-pihole-serenity, keepalived-pihole-serenity) was then stopped and removed after live mixed-host DNS checks confirmed the Technitium path remained healthy without it

Wave 1-A: legacy Pi-hole removal

Target containers

• binhex-official-pihole
• pihole-serenity
• unbound-pihole-serenity
• keepalived-pihole-serenity

Why they are in scope

• They are legacy DNS/HA remnants.
• Current homelab docs describe the active internal DNS path as the Technitium trio.
• Operator confirmed Technitium covers the intended DNS role.
• Keeping old DNS stacks around increases confusion and future troubleshooting blast radius.

Preconditions

Before removal, verify only these read-only checks:

1. Serenity Technitium backup node is healthy.
2. DHCP-advertised resolver set is still PD/NOMAD/Serenity Technitium, not Pi-hole.
3. No Pangolin route, bookmark, or admin workflow still intentionally points at a Pi-hole UI.
4. No host on the LAN still relies on the old Pi-hole admin port out of habit.

Removal order

1. stop keepalived-pihole-serenity
2. stop pihole-serenity
3. stop unbound-pihole-serenity
4. stop binhex-official-pihole
5. verify Technitium-only DNS behavior still looks normal
6. remove the stopped containers
7. archive or delete their stale appdata only after a short observation window

Verification after removal

• Serenity Technitium container remained healthy
• mixed-host DNS checks stayed good after removal:
  • from NOMAD, 10.5.30.8 and 10.5.30.10 still resolved both public and homelab names
  • from PD, 10.5.30.9 and 10.5.30.10 still resolved both public and homelab names
• 10.5.30.53 continued answering DNS even after Serenity Pi-hole removal, confirming it is no longer tied to the removed Serenity Pi-hole containers
• no immediate client-facing DNS regression was observed during the removal window
• no public regression was observed on sampled hostnames such as panel.paccoco.com and audiobookshelf.paccoco.com

Wave 1-B: stale container pruning

Created-only clutter to remove

• calibre-web
• SuggestArr
• Cleanuparr
• calibre
• agregarr

Exited clutter to remove

• Huntarr
• omegabrr

Why they are in scope

• They are not live workloads.
• They add noise to docker ps -a and make host intent harder to understand.
• There is no current architecture reason to preserve them as active Serenity residents.

Safe pruning rules

Before deleting each one:

1. confirm container status is still Created or Exited
2. confirm it is not referenced by a live reverse-proxy route
3. confirm it is not the only source of some needed config/data you still care about
4. if uncertain, export one final metadata snapshot first:
   • image name
   • mounts
   • env file path if obvious

Practical order

1. remove Created containers first
2. remove long-dead exited containers second
3. leave appdata in place initially
4. only delete appdata later after a short cooling-off window

Wave 1-C: cloudflared deadwood removal

Target container

• Unraid-Cloudflared-Tunnel

Why it is in scope

• It is already documented in repo docs as dead/stale.
• Pangolin/Newt is the active exposure pattern now.
• Live audit on 2026-05-25 showed the container is a remote-managed Cloudflare Tunnel carrying a stale legacy config, not an active Serenity service path.

Live audit findings (2026-05-25)

• Unraid-Cloudflared-Tunnel is healthy at the transport layer (/ready reported 4 ready connections), but metrics showed cloudflared_tunnel_total_requests 0 and cloudflared_tunnel_request_errors 0 for the current run.
• Startup logs exposed the remote-managed ingress set as legacy hostnames pointed at old 192.168.1.x origins:
  • wazuh.paccoco.com -> https://192.168.1.102
  • remotely.paccoco.com -> http://192.168.1.180:5001
  • hp.paccoco.com -> http://192.168.1.180:3000
  • octoprint.paccoco.com -> http://192.168.1.52
  • audiobookshelf.paccoco.com -> http://192.168.1.5:13378
  • spoolman.paccoco.com -> http://192.168.1.202:7912
  • node1.paccoco.com -> https://192.168.1.189:8080
  • hp2.paccoco.com -> http://192.168.1.224:3000
  • panel.paccoco.com -> https://192.168.1.189
  • nextcloud.paccoco.com -> http://192.168.1.5:11000
• The tunnel has no local config files under /mnt/user/appdata/cloudflared; it is driven entirely by Cloudflare-side config plus a token.
• Public checks for those hostnames currently resolve to 172.245.79.139 and return either live responses or front-door 404s without producing any new tunnel traffic, indicating the present public path is elsewhere and not traversing this Serenity container.
• Repo/docs no longer identify this tunnel as an intended live exposure path; the only repeated modern exposure requirement in Serenity docs is the local Pangolin/Newt lane.

Preconditions

1. verify no current DNS/public route still expects this tunnel
2. verify no local notes still treat it as the active exposure path
3. verify Newt-based routes are the real live path

Status after live audit:

• (1) satisfied enough for container retirement: no observed current public traffic is reaching this tunnel
• (2) satisfied: local docs treat it as stale legacy deadwood, not the intended active path
• (3) satisfied for Serenity-hosted apps: Pangolin/Newt remains the intentionally preserved exposure path

Recommended retirement path

1. stop the container and watch briefly for any unexpected complaint or new public breakage
2. remove the container
3. keep /mnt/user/appdata/cloudflared during a cooling-off window even though it appears empty/unneeded
4. later, from the Cloudflare side, delete or repoint the stale tunnel config/hostnames if they still exist there

Status:

• completed on 2026-05-25: Unraid-Cloudflared-Tunnel was stopped and removed
• sampled public checks (panel.paccoco.com, audiobookshelf.paccoco.com) remained healthy after removal

Wave 1-D: DB-backed migration ordering

These apps should not be deleted in wave 1. They need planned migration.

Pair 1: GameVault + local Postgres

Live dependency:

• GameVault -> postgresql15

Recommended sequence:

1. create PD-side target appdata path
2. create PD-side Postgres DB/user on shared-postgres, or a deliberate dedicated PD Postgres if there is a reason not to use shared-postgres
3. export GameVault DB from Serenity
4. import into PD target database
5. migrate GameVault appdata/config
6. recreate GameVault on PD attached to the shared database network if using shared-postgres
7. verify login, library visibility, and metadata path behavior
8. only then retire Serenity postgresql15

Default recommendation:

• prefer PD shared-postgres unless GameVault has a proven reason to stay isolated
• note: live psql inspection on 2026-05-25 showed a Postgres collation-version mismatch warning on Serenity (gamevault created with glibc collation 2.36 while host now provides 2.41), so include a post-migration refresh/reindex plan rather than carrying that debt forward silently

Pair 2: RomM + local MariaDB

Live dependency:

• RomM -> MariaDB-Official

Recommended sequence:

1. create PD-side target appdata path
2. create PD-side MariaDB DB/user on shared-mariadb, or a deliberate dedicated PD MariaDB only if needed
3. export RomM DB from Serenity
4. import into PD target MariaDB
5. migrate RomM appdata/config/assets/resources
6. recreate RomM on PD attached to the shared database network if using shared-mariadb
7. verify UI, library, metadata, and asset behavior
8. only then retire Serenity MariaDB-Official

Default recommendation:

• prefer PD shared-mariadb unless RomM proves awkward on the shared stack

Recommended order across all wave 1 work

1. verify Technitium is the only intended active DNS path
2. remove legacy Pi-hole stack
3. remove dead Cloudflared tunnel
4. remove stale created/exited containers
5. leave GameVault/Postgres and RomM/MariaDB in place until their PD migration is prepared
6. keep qbit/ARR locality untouched until PD storage cutover is real

Risks and guardrails

Do not touch yet

Do not touch in this wave:

• qbit
• ARR family
• GluetunVPN
• qbit_manage
• Newt
• technitium-dns-pilot
• GameVault
• romm
• postgresql15
• MariaDB-Official

Specific guardrails

• Do not delete any appdata directory in the same step as container removal unless the dependency is unquestionably dead.
• Do not remove postgresql15 until GameVault is verified on PD.
• Do not remove MariaDB-Official until RomM is verified on PD.
• Do not move qbit/ARR until PD directly owns the relevant media/torrent paths.
• Do not break Serenity Newt while cleanup is happening.

Suggested Kanban decomposition

Card A1 — verify legacy Pi-hole is truly unused

Definition of done:

• current DNS path confirmed as Technitium-only
• no intentional admin dependency on Serenity Pi-hole remains

Card A2 — remove Serenity legacy Pi-hole containers

Definition of done:

• all four legacy Pi-hole containers stopped and removed
• no DNS regression observed

Card B1 — remove stale created containers

Definition of done:

• created-only clutter removed
• appdata retained for cooling-off period

Card B2 — remove stale exited containers

Definition of done:

• exited clutter removed
• appdata retained for cooling-off period

Card C1 — remove dead Unraid Cloudflared tunnel

Definition of done:

• no public path depends on it
• container removed

Card D1 — choose PD target layout for GameVault and RomM wave

Definition of done:

• PD target appdata paths chosen for both apps
• decision recorded to use PD shared-postgres for GameVault unless blocked
• decision recorded to use PD shared-mariadb for RomM unless blocked
• required PD mount paths for libraries/assets/resources identified

Chosen target layout (2026-05-25):

• place both services in the PD media stack so they follow the same steady-state placement already documented for PD media/library apps
• attach both services to:
  • media-net for local app adjacency
  • ix-databases_shared-databases for shared DB access
  • pangolin for internal ingress / public exposure
• preserve the current host ports on PD because live checks showed them free there:
  • GameVault: 8785:8080
  • RomM: 8457:8080
• preserve the public hostnames already in use during cutover:
  • gamevault.paccoco.com
  • romm.paccoco.com
• use the PD internal ingress model (Pangolin -> Traefik -> app) when the public route is re-homed, rather than introducing another direct-to-container edge pattern

Chosen storage layout:

• GameVault appdata on SSD because it is small and write-active:
  • /mnt/docker-ssd/docker/appdata/gamevault/media -> /media
  • /mnt/docker-ssd/docker/appdata/gamevault/logs -> /logs
• GameVault canonical content mounts should be re-used, not copied:
  • /mnt/unraid/data/media/Games-Apps Isos -> /files
  • /mnt/unraid/data/media/Saved Games -> /savefiles
• RomM split layout:
  • write-active cache on SSD:
    • /mnt/docker-ssd/docker/appdata/romm/redis-data -> /redis-data
  • config/assets/resources on tank appdata:
    • /mnt/tank/docker/appdata/romm/config -> /romm/config
    • /mnt/tank/docker/appdata/romm/assets -> /romm/assets
    • /mnt/tank/docker/appdata/romm/resources -> /romm/resources
• RomM canonical library mount should be re-used, not copied:
  • /mnt/unraid/data/media/RomM -> /romm/library

Chosen database layout:

• GameVault -> PD shared-postgres on ix-databases_shared-databases
  • target DB: gamevault
  • target DB user: gamevault
• RomM -> PD shared-mariadb on ix-databases_shared-databases
  • target DB: romm
  • target DB user: romm

Routing/proxy notes to preserve during implementation:

• RomM upstream docs call out reverse-proxy sensitivity; keep websocket support intact when the route is moved behind PD Traefik
• existing Traefik file-provider patterns in ingress/traefik/dynamic/routes.yml should be extended rather than inventing a new routing mechanism for these two apps

Card D2 — stage PD database targets for the wave

Definition of done:

• GameVault target DB/user created on PD shared-postgres or explicit exception documented
• RomM target DB/user created on PD shared-mariadb or explicit exception documented
• connection details verified from the future PD app network context
• migration rollback notes captured before any source export

Execution notes (2026-05-25):

• staged PD targets on the live shared DB services:
  • Postgres DB/user: gamevault / gamevault
  • MariaDB DB/user: romm / romm
• stored live credentials in the PD media stack env file and synced that env into the encrypted secrets repo; credentials were not written into the main repo
• verified future app-network connectivity with ephemeral clients on ix-databases_shared-databases:
  • Postgres check returned gamevault|gamevault
  • MariaDB check returned romm / romm@%
• rollback notes:
  • pre-change backup of PD media env: /mnt/docker-ssd/docker/compose/media/.env.pre-serenity-wave2-d2-20260525-182938
  • if D3/D4 later uncover an issue, keep the staged DBs unused, restore the prior media .env if needed, and rotate/drop the staged users before re-attempting
  • no Serenity source data was touched yet in D2; this card only prepared empty PD landing zones

Card D3 — export and validate Serenity source databases

Definition of done:

• pre-cutover exports from Serenity are taken for both apps so restore/import flow can be tested before downtime
• gamevault Postgres dump exported from Serenity and integrity-checked
• romm MariaDB dump exported from Serenity and integrity-checked
• source app versions and DB container versions recorded alongside dumps
• Postgres collation-version warning captured as a post-import remediation item
• final cutover note recorded: these pre-cutover dumps are not the authoritative final state; a last quiesced export must be taken after the Serenity app is stopped during cutover and imported to PD before PD goes live

Execution notes (2026-05-25):

• successful pre-cutover exports were captured and staged under PD backup storage:
  • latest verified dump set: /mnt/tank/docker/backups/db-dumps/serenity-wave2/20260525-184917/
  • 20260525-184917-serenity-gamevault.pgcustom — 175290 bytes, sha256 1ebd0286483e7f34dddc21076a7540c966a812d8891a8ed7a24247fe972d927d
  • 20260525-184917-serenity-romm.sql — 8526126 bytes, sha256 c74a2d9d715f6c0982d707ba9f41174e19ba681a1debebde865e94e63271302b
• source runtime facts recorded during export validation:
  • Serenity postgresql15: PostgreSQL 15.18, gamevault has 18 public tables and ~12 MB logical size
  • Serenity MariaDB-Official: MariaDB 12.2.2, romm has 20 tables and ~12.41 MB logical size
  • the existing Serenity Postgres collation-version mismatch warning is still present and should be remediated after final import on PD
• GameVault restore-path validation succeeded:
  • the pre-cutover Postgres dump restored cleanly into PD shared-postgres
  • PD gamevault target currently shows the expected 18 public tables
• RomM restore-path validation exposed a blocker on the planned PD shared-mariadb target:
  • importing the Serenity RomM dump into PD shared-mariadb failed at line 167 while creating device_save_sync
  • MariaDB returned ERROR 1005 (HY000): Can't create table romm.device_save_sync (errno: 121 "Duplicate key on write or update")
  • PD shared-mariadb is currently MariaDB 11.4.11 while the Serenity RomM source dump was produced from MariaDB 12.2.2
  • treat this as a compatibility / target-selection issue, not a dump-corruption issue; the dump itself is populated and preserved in backup storage
• net result:
  • D3 is complete for GameVault
  • D3 is only partially complete for RomM until the MariaDB target strategy is revised and the restore path is proven

Card D4 — sync appdata for GameVault and RomM to PD staging paths

Definition of done:

• GameVault config/media/log paths copied to PD staging
• RomM config/assets/resources paths copied to PD staging
• large library mounts intentionally re-used from canonical storage instead of blindly duplicating data
• ownership/permissions on PD staging paths verified

Note:

• do not advance RomM cutover assumptions until the MariaDB target mismatch found in D3 is resolved; RomM may need a different PD DB target than the original shared-mariadb assumption

Card D5 — cut over GameVault to PD

Definition of done:

• Serenity GameVault stopped only for the final cutover window
• after GameVault is stopped on Serenity, a final quiesced gamevault Postgres export is taken and imported to the PD target DB so PD does not come up on stale data
• PD GameVault starts against the PD target Postgres DB
• login, library visibility, metadata behavior, and save/upload paths verified
• public/LAN route updated if needed and verified
• Serenity postgresql15 kept in place but clearly marked retirement-ready if cutover succeeds

Card D6 — cut over RomM to PD

Definition of done:

• Serenity RomM stopped only for the final cutover window
• after RomM is stopped on Serenity, a final quiesced romm MariaDB export is taken and imported to the PD target DB so PD does not come up on stale data
• PD RomM starts against the PD target MariaDB DB
• UI, library visibility, assets/resources, background jobs, and metadata behavior verified
• public/LAN route updated if needed and verified
• Serenity MariaDB-Official kept in place but clearly marked retirement-ready if cutover succeeds

Card D7 — retire Serenity DB remnants after cooldown

Definition of done:

• GameVault and RomM remain healthy on PD through an observation window
• Serenity postgresql15 and MariaDB-Official are stopped and removed only after successful PD validation
• Serenity-side DB appdata is retained for cooldown/rollback, not deleted immediately
• docs and host inventory updated to show Serenity no longer carries those DB pairs

Open item that still needs verification

• reranker mounts /mnt/user/appdate/reranker
• verify whether appdate is intentional before any future reranker move or cleanup

Expected result after wave 1

After wave 1, Serenity should still be alive for the workloads that currently justify it, but with much less misleading baggage:

• torrent/media-locality group still intact
• Newt still intact
• Technitium backup node still intact
• GameVault and RomM still live until their migration is prepared
• legacy Pi-hole gone
• dead Cloudflared gone
• stale created/exited clutter gone

That leaves a cleaner host and a safer runway for the later PD storage cutover and full Serenity retirement.

Wave 1 verification results (2026-05-25)

Verified during this planning pass:

• Serenity still has these live containers relevant to wave 1:
  • technitium-dns-pilot
  • binhex-official-pihole
  • pihole-serenity
  • unbound-pihole-serenity
  • keepalived-pihole-serenity
  • Newt
  • Unraid-Cloudflared-Tunnel
• PD still runs the primary Technitium stack plus its own Pi-hole and Newt lane.
• From NOMAD, /etc/resolv.conf currently lists 10.5.30.8, 10.5.30.9, and 10.5.30.10 ahead of external fallback 9.9.9.9.
• From NOMAD, dig to 10.5.30.8 and 10.5.30.10 succeeded for public DNS resolution; same-host checks to 10.5.30.9 were unreliable, matching the existing macvlan caveat in the docs.
• Unraid-Cloudflared-Tunnel is still running, but repo docs already classify it as dead/stale and its container has Restart=no.
• Serenity Newt must not be treated as deadwood: operator confirmed Pangolin tunnels Serenity resources through Serenity's local Newt instead of routing from PD or NOMAD to Serenity resources over the Serenity LAN IP.
• Live Serenity Newt logs still show repeated Pangolin health checks against stale 10.5.1.5 targets (7474, 8785, 8787, 8788, 8990, 8457, 5690, 5454).

Operational implication:

• removing Cloudflared remains low-risk after one final dependency check
• removing the legacy Pi-hole stack remains appropriate
• removing Serenity Newt is not appropriate during wave 1
• Pangolin target drift for Serenity-hosted resources should be repaired later by rehoming those resources to the correct Serenity site/local-path model instead of stale literal pre-migration IPs