# Serenity Cleanup Wave 1 Plan Status: approved planning baseline for the first safe cleanup pass on Serenity. ## Goal Reduce obvious legacy clutter on Serenity without breaking the still-needed torrent/media-locality group or the still-needed Serenity Newt path. This wave is intentionally conservative. It does not move qBittorrent/ARR off Serenity yet. It does not retire Serenity yet. It does not delete databases that still back live apps. ## Operator decisions already resolved - Nothing should intentionally remain on Serenity after PD owns the disks locally. - Technitium already covers the DNS role John wants. - Serenity Pi-hole remnants should be treated as removable. - Serenity Newt is still needed and must be preserved. - GameVault and RomM should migrate, not be pruned. - Final end-state remains: - move qbit + ARR family to PD after storage cutover - leave no intentional production app role on Serenity - retire Serenity entirely ## Scope of cleanup wave 1 Wave 1 includes only these categories: 1. Remove legacy DNS clutter that should no longer be serving production traffic. 2. Remove obviously stale created/exited containers. 3. Document migration order for the two database-backed apps that should move later. Wave 1 explicitly excludes: - qbit - GluetunVPN - qbit_manage - prowlarr - sonarr - sonarr-anime - radarr - lidarr - readarr - readarr-epub - bazarr - autobrr - unpackerr - Notifiarr - shelfmark - Newt - technitium-dns-pilot - GameVault - romm - reranker ## Live facts this plan is based on From the live Serenity audit: - ARR/torrent locality is still tied to `/mnt/user/data` - `GameVault` points at local Postgres on `10.5.30.5:5432` - `RomM` points at local MariaDB on `10.5.30.5:3306` - live container env inspection still shows those explicit DB_HOST/DB_PORT bindings on 2026-05-25 - quick live checks did not surface immediate DB dependencies for `Wizarr`, `Shelfmark`, or `Notifiarr` - `Newt` is still needed - legacy Pi-hole containers are still running even though Technitium is now the intended DNS path - Serenity Newt-backed Pangolin routes still had stale health-check hostnames pointing at old `10.5.1.5` even though their target IPs had already been rewritten to `localhost`/`10.5.30.5` - a temporary `10.5.1.5/32` alias on `br0` validated the diagnosis, but it was removed because the old IP is no longer allowed on that VLAN - authoritative Pangolin cleanup was then completed by rewriting the audited Serenity target set to `10.5.30.5` for both routing and health checks, removing the need for any legacy-IP workaround ## Wave 1 current execution status Live execution on 2026-05-25 established: - `Huntarr` and `omegabrr` were low-risk stale stopped containers and were removed from Serenity - the recent non-running `Created` containers (`calibre-web`, `SuggestArr`, `Cleanuparr`, `calibre`, `agregarr`) were metadata-verified and then removed after confirming they were still only `Created`, had `restart=no`, and were not live workloads - `Unraid-Cloudflared-Tunnel` was audited, stopped, and removed after verification showed it was only transport-alive and no current public traffic depended on it - the Serenity Pi-hole HA stack (`binhex-official-pihole`, `pihole-serenity`, `unbound-pihole-serenity`, `keepalived-pihole-serenity`) was then stopped and removed after live mixed-host DNS checks confirmed the Technitium path remained healthy without it ## Wave 1-A: legacy Pi-hole removal ### Target containers - `binhex-official-pihole` - `pihole-serenity` - `unbound-pihole-serenity` - `keepalived-pihole-serenity` ### Why they are in scope - They are legacy DNS/HA remnants. - Current homelab docs describe the active internal DNS path as the Technitium trio. - Operator confirmed Technitium covers the intended DNS role. - Keeping old DNS stacks around increases confusion and future troubleshooting blast radius. ### Preconditions Before removal, verify only these read-only checks: 1. Serenity Technitium backup node is healthy. 2. DHCP-advertised resolver set is still PD/NOMAD/Serenity Technitium, not Pi-hole. 3. No Pangolin route, bookmark, or admin workflow still intentionally points at a Pi-hole UI. 4. No host on the LAN still relies on the old Pi-hole admin port out of habit. ### Removal order 1. stop `keepalived-pihole-serenity` 2. stop `pihole-serenity` 3. stop `unbound-pihole-serenity` 4. stop `binhex-official-pihole` 5. verify Technitium-only DNS behavior still looks normal 6. remove the stopped containers 7. archive or delete their stale appdata only after a short observation window ### Verification after removal - Serenity Technitium container remained healthy - mixed-host DNS checks stayed good after removal: - from NOMAD, `10.5.30.8` and `10.5.30.10` still resolved both public and homelab names - from PD, `10.5.30.9` and `10.5.30.10` still resolved both public and homelab names - `10.5.30.53` continued answering DNS even after Serenity Pi-hole removal, confirming it is no longer tied to the removed Serenity Pi-hole containers - no immediate client-facing DNS regression was observed during the removal window - no public regression was observed on sampled hostnames such as `panel.paccoco.com` and `audiobookshelf.paccoco.com` ## Wave 1-B: stale container pruning ### Created-only clutter to remove - `calibre-web` - `SuggestArr` - `Cleanuparr` - `calibre` - `agregarr` ### Exited clutter to remove - `Huntarr` - `omegabrr` ### Why they are in scope - They are not live workloads. - They add noise to `docker ps -a` and make host intent harder to understand. - There is no current architecture reason to preserve them as active Serenity residents. ### Safe pruning rules Before deleting each one: 1. confirm container status is still `Created` or `Exited` 2. confirm it is not referenced by a live reverse-proxy route 3. confirm it is not the only source of some needed config/data you still care about 4. if uncertain, export one final metadata snapshot first: - image name - mounts - env file path if obvious ### Practical order 1. remove `Created` containers first 2. remove long-dead exited containers second 3. leave appdata in place initially 4. only delete appdata later after a short cooling-off window ## Wave 1-C: cloudflared deadwood removal ### Target container - `Unraid-Cloudflared-Tunnel` ### Why it is in scope - It is already documented in repo docs as dead/stale. - Pangolin/Newt is the active exposure pattern now. - Live audit on 2026-05-25 showed the container is a remote-managed Cloudflare Tunnel carrying a stale legacy config, not an active Serenity service path. ### Live audit findings (2026-05-25) - `Unraid-Cloudflared-Tunnel` is healthy at the transport layer (`/ready` reported 4 ready connections), but metrics showed `cloudflared_tunnel_total_requests 0` and `cloudflared_tunnel_request_errors 0` for the current run. - Startup logs exposed the remote-managed ingress set as legacy hostnames pointed at old `192.168.1.x` origins: - `wazuh.paccoco.com` -> `https://192.168.1.102` - `remotely.paccoco.com` -> `http://192.168.1.180:5001` - `hp.paccoco.com` -> `http://192.168.1.180:3000` - `octoprint.paccoco.com` -> `http://192.168.1.52` - `audiobookshelf.paccoco.com` -> `http://192.168.1.5:13378` - `spoolman.paccoco.com` -> `http://192.168.1.202:7912` - `node1.paccoco.com` -> `https://192.168.1.189:8080` - `hp2.paccoco.com` -> `http://192.168.1.224:3000` - `panel.paccoco.com` -> `https://192.168.1.189` - `nextcloud.paccoco.com` -> `http://192.168.1.5:11000` - The tunnel has no local config files under `/mnt/user/appdata/cloudflared`; it is driven entirely by Cloudflare-side config plus a token. - Public checks for those hostnames currently resolve to `172.245.79.139` and return either live responses or front-door 404s without producing any new tunnel traffic, indicating the present public path is elsewhere and not traversing this Serenity container. - Repo/docs no longer identify this tunnel as an intended live exposure path; the only repeated modern exposure requirement in Serenity docs is the local Pangolin/Newt lane. ### Preconditions 1. verify no current DNS/public route still expects this tunnel 2. verify no local notes still treat it as the active exposure path 3. verify Newt-based routes are the real live path Status after live audit: - (1) satisfied enough for container retirement: no observed current public traffic is reaching this tunnel - (2) satisfied: local docs treat it as stale legacy deadwood, not the intended active path - (3) satisfied for Serenity-hosted apps: Pangolin/Newt remains the intentionally preserved exposure path ### Recommended retirement path 1. stop the container and watch briefly for any unexpected complaint or new public breakage 2. remove the container 3. keep `/mnt/user/appdata/cloudflared` during a cooling-off window even though it appears empty/unneeded 4. later, from the Cloudflare side, delete or repoint the stale tunnel config/hostnames if they still exist there Status: - completed on 2026-05-25: `Unraid-Cloudflared-Tunnel` was stopped and removed - sampled public checks (`panel.paccoco.com`, `audiobookshelf.paccoco.com`) remained healthy after removal ## Wave 1-D: DB-backed migration ordering These apps should not be deleted in wave 1. They need planned migration. ### Pair 1: GameVault + local Postgres Live dependency: - `GameVault` -> `postgresql15` Recommended sequence: 1. create PD-side target appdata path 2. create PD-side Postgres DB/user on shared-postgres, or a deliberate dedicated PD Postgres if there is a reason not to use shared-postgres 3. export GameVault DB from Serenity 4. import into PD target database 5. migrate GameVault appdata/config 6. recreate GameVault on PD attached to the shared database network if using shared-postgres 7. verify login, library visibility, and metadata path behavior 8. only then retire Serenity `postgresql15` Default recommendation: - prefer PD shared-postgres unless GameVault has a proven reason to stay isolated - note: live `psql` inspection on 2026-05-25 showed a Postgres collation-version mismatch warning on Serenity (`gamevault` created with glibc collation 2.36 while host now provides 2.41), so include a post-migration refresh/reindex plan rather than carrying that debt forward silently ### Pair 2: RomM + local MariaDB Live dependency: - `RomM` -> `MariaDB-Official` Recommended sequence: 1. create PD-side target appdata path 2. create PD-side MariaDB DB/user on shared-mariadb, or a deliberate dedicated PD MariaDB only if needed 3. export RomM DB from Serenity 4. import into PD target MariaDB 5. migrate RomM appdata/config/assets/resources 6. recreate RomM on PD attached to the shared database network if using shared-mariadb 7. verify UI, library, metadata, and asset behavior 8. only then retire Serenity `MariaDB-Official` Default recommendation: - prefer PD shared-mariadb unless RomM proves awkward on the shared stack ## Recommended order across all wave 1 work 1. verify Technitium is the only intended active DNS path 2. remove legacy Pi-hole stack 3. remove dead Cloudflared tunnel 4. remove stale created/exited containers 5. leave GameVault/Postgres and RomM/MariaDB in place until their PD migration is prepared 6. keep qbit/ARR locality untouched until PD storage cutover is real ## Risks and guardrails ### Do not touch yet Do not touch in this wave: - qbit - ARR family - GluetunVPN - qbit_manage - Newt - technitium-dns-pilot - GameVault - romm - postgresql15 - MariaDB-Official ### Specific guardrails - Do not delete any appdata directory in the same step as container removal unless the dependency is unquestionably dead. - Do not remove `postgresql15` until GameVault is verified on PD. - Do not remove `MariaDB-Official` until RomM is verified on PD. - Do not move qbit/ARR until PD directly owns the relevant media/torrent paths. - Do not break Serenity Newt while cleanup is happening. ## Suggested Kanban decomposition ### Card A1 — verify legacy Pi-hole is truly unused Definition of done: - current DNS path confirmed as Technitium-only - no intentional admin dependency on Serenity Pi-hole remains ### Card A2 — remove Serenity legacy Pi-hole containers Definition of done: - all four legacy Pi-hole containers stopped and removed - no DNS regression observed ### Card B1 — remove stale created containers Definition of done: - created-only clutter removed - appdata retained for cooling-off period ### Card B2 — remove stale exited containers Definition of done: - exited clutter removed - appdata retained for cooling-off period ### Card C1 — remove dead Unraid Cloudflared tunnel Definition of done: - no public path depends on it - container removed ### Card D1 — choose PD target layout for GameVault and RomM wave Definition of done: - PD target appdata paths chosen for both apps - decision recorded to use PD shared-postgres for GameVault unless blocked - decision recorded to use PD shared-mariadb for RomM unless blocked - required PD mount paths for libraries/assets/resources identified Chosen target layout (2026-05-25): - place both services in the PD `media` stack so they follow the same steady-state placement already documented for PD media/library apps - attach both services to: - `media-net` for local app adjacency - `pangolin` for internal ingress / public exposure - attach only GameVault to `ix-databases_shared-databases` because it keeps using PD `shared-postgres` - keep RomM on `media-net` + `pangolin`; its database will be a dedicated sibling service on `media-net`, not a shared DB consumer - preserve the current host ports on PD because live checks showed them free there: - GameVault: `8785:8080` - RomM: `8457:8080` - preserve the public hostnames already in use during cutover: - `gamevault.paccoco.com` - `romm.paccoco.com` - use the PD internal ingress model (`Pangolin -> Traefik -> app`) when the public route is re-homed, rather than introducing another direct-to-container edge pattern Chosen storage layout: - GameVault appdata on SSD because it is small and write-active: - `/mnt/docker-ssd/docker/appdata/gamevault/media` -> `/media` - `/mnt/docker-ssd/docker/appdata/gamevault/logs` -> `/logs` - GameVault canonical content mounts should be re-used, not copied: - `/mnt/unraid/data/media/Games-Apps Isos` -> `/files` - `/mnt/unraid/data/media/Saved Games` -> `/savefiles` - RomM split layout: - write-active cache on SSD: - `/mnt/docker-ssd/docker/appdata/romm/redis-data` -> `/redis-data` - config/assets/resources on tank appdata: - `/mnt/tank/docker/appdata/romm/config` -> `/romm/config` - `/mnt/tank/docker/appdata/romm/assets` -> `/romm/assets` - `/mnt/tank/docker/appdata/romm/resources` -> `/romm/resources` - RomM canonical library mount should be re-used, not copied: - `/mnt/unraid/data/media/RomM` -> `/romm/library` Chosen database layout: - GameVault -> PD `shared-postgres` on `ix-databases_shared-databases` - target DB: `gamevault` - target DB user: `gamevault` - RomM -> dedicated PD `romm-db` service on `media-net` - target image pin: `mariadb:12.2` - target DB: `romm` - target DB user: `romm` - target data path: `/mnt/docker-ssd/docker/appdata/romm-db/mysql` Routing/proxy notes to preserve during implementation: - RomM upstream docs call out reverse-proxy sensitivity; keep websocket support intact when the route is moved behind PD Traefik - existing Traefik file-provider patterns in `ingress/traefik/dynamic/routes.yml` should be extended rather than inventing a new routing mechanism for these two apps ### Card D2 — stage PD database targets for the wave Definition of done: - GameVault target DB/user created on PD shared-postgres or explicit exception documented - RomM target DB/user created on the chosen PD MariaDB target or explicit exception documented - connection details verified from the future PD app network context - migration rollback notes captured before any source export Execution notes (2026-05-25): - staged PD targets on the live shared DB services: - Postgres DB/user: `gamevault` / `gamevault` - MariaDB DB/user: `romm` / `romm` - stored live credentials in the PD media stack env file and synced that env into the encrypted secrets repo; credentials were not written into the main repo - verified future app-network connectivity with ephemeral clients on `ix-databases_shared-databases`: - Postgres check returned `gamevault|gamevault` - MariaDB check returned `romm` / `romm@%` - rollback notes: - pre-change backup of PD media env: `/mnt/docker-ssd/docker/compose/media/.env.pre-serenity-wave2-d2-20260525-182938` - if D3/D4 later uncover an issue, keep the staged DBs unused, restore the prior media `.env` if needed, and rotate/drop the staged users before re-attempting - no Serenity source data was touched yet in D2; this card only prepared empty PD landing zones - note after D3b: - the staged shared-mariadb `romm` target is now superseded by the dedicated `romm-db` plan and should not be used for RomM cutover ### Card D3 — export and validate Serenity source databases Definition of done: - pre-cutover exports from Serenity are taken for both apps so restore/import flow can be tested before downtime - `gamevault` Postgres dump exported from Serenity and integrity-checked - `romm` MariaDB dump exported from Serenity and integrity-checked - source app versions and DB container versions recorded alongside dumps - Postgres collation-version warning captured as a post-import remediation item - final cutover note recorded: these pre-cutover dumps are not the authoritative final state; a last quiesced export must be taken after the Serenity app is stopped during cutover and imported to PD before PD goes live Execution notes (2026-05-25): - successful pre-cutover exports were captured and staged under PD backup storage: - latest verified dump set: `/mnt/tank/docker/backups/db-dumps/serenity-wave2/20260525-184917/` - `20260525-184917-serenity-gamevault.pgcustom` — 175290 bytes, sha256 `1ebd0286483e7f34dddc21076a7540c966a812d8891a8ed7a24247fe972d927d` - `20260525-184917-serenity-romm.sql` — 8526126 bytes, sha256 `c74a2d9d715f6c0982d707ba9f41174e19ba681a1debebde865e94e63271302b` - source runtime facts recorded during export validation: - Serenity `postgresql15`: PostgreSQL 15.18, `gamevault` has 18 public tables and ~12 MB logical size - Serenity `MariaDB-Official`: MariaDB 12.2.2, `romm` has 20 tables and ~12.41 MB logical size - the existing Serenity Postgres collation-version mismatch warning is still present and should be remediated after final import on PD - GameVault restore-path validation succeeded: - the pre-cutover Postgres dump restored cleanly into PD shared-postgres - PD `gamevault` target currently shows the expected 18 public tables - RomM restore-path validation exposed a blocker on the planned PD shared-mariadb target: - importing the Serenity RomM dump into PD shared-mariadb failed at line 167 while creating `device_save_sync` - MariaDB returned `ERROR 1005 (HY000): Can't create table romm.device_save_sync (errno: 121 "Duplicate key on write or update")` - PD shared-mariadb is currently MariaDB 11.4.11 while the Serenity RomM source dump was produced from MariaDB 12.2.2 - treat this as a compatibility / target-selection issue, not a dump-corruption issue; the dump itself is populated and preserved in backup storage - net result: - D3 is complete for GameVault - D3 export validation is complete for RomM, but the original PD shared-mariadb landing target is rejected ### Card D3b — resolve RomM PD database target compatibility Decision: - do not upgrade PD `shared-mariadb` just to fit RomM - keep PD `shared-mariadb` on its current conservative shared-infra track unless a separate approved maintenance change is planned for all consumers - provision a dedicated PD MariaDB target for RomM instead Evidence behind the decision: - the RomM source dump repeatedly uses reused foreign-key names such as `CONSTRAINT \`1\`` and `CONSTRAINT \`2\`` across multiple tables - MariaDB docs state that foreign-key names must be unique per database before MariaDB 12.1, and only become reusable across tables in MariaDB 12.1+ - Serenity `MariaDB-Official` is MariaDB 12.2.2, which explains why the source schema can exist there - PD `shared-mariadb` is MariaDB 11.4.11, where the same dump fails with `errno: 121` while creating `device_save_sync` - the PD databases stack currently provisions MariaDB specifically for Uptime Kuma via `databases/initdb-mariadb/01-create-uptime-kuma-db.sh`; there is no evidence in repo or live stack layout that shared-mariadb was intentionally upgraded or validated as a broad multi-app MariaDB landing zone - RomM's own setup docs commonly show a dedicated MariaDB container and even call out compatibility-conscious MariaDB version choices in example guides Operational recommendation: - create a dedicated PD MariaDB service for RomM, isolated from `shared-mariadb` - pin that dedicated service to a MariaDB 12.x line compatible with the Serenity source behavior unless RomM docs or release notes later justify a different pin - attach PD RomM only to its own DB service plus the networks it already needs for ingress/library access - leave `shared-mariadb` untouched for now to avoid coupling the RomM migration to a shared-database major-version change and validation cycle Definition of done for D3b: - dedicated PD MariaDB target for RomM is chosen and documented - target image/version pin recorded - restore path is re-tested successfully against that dedicated target before RomM cutover proceeds Execution notes (2026-05-26): - restore-path proof was executed on PD using an isolated test container pinned to `mariadb:12.2` - test target details: - container: `romm-db-d3b-test` - image/runtime version: `mariadb:12.2` / `12.2.2-MariaDB` - data path: `/mnt/docker-ssd/docker/appdata/romm-db-d3b-test/mysql` - the preserved Serenity RomM dump `/home/fizzlepoof/serenity-wave2-d3-work/20260525-184736-serenity-romm.sql` imported cleanly into the 12.2 target on PD - post-import verification on the PD test target returned: - `20` tables in schema `romm` - approximately `10.42 MB` logical table+index footprint - result: - the compatibility blocker is resolved for a dedicated MariaDB 12.2 target - RomM should proceed against dedicated `romm-db`, not `shared-mariadb` ### Card D4 — sync appdata for GameVault and RomM to PD staging paths Definition of done: - GameVault config/media/log paths copied to PD staging - RomM config/assets/resources paths copied to PD staging - large library mounts intentionally re-used from canonical storage instead of blindly duplicating data - ownership/permissions on PD staging paths verified Note: - do not advance RomM cutover assumptions until the MariaDB target mismatch found in D3 is resolved; RomM may need a different PD DB target than the original shared-mariadb assumption ### Card D5 — cut over GameVault to PD Definition of done: - Serenity GameVault stopped only for the final cutover window - after GameVault is stopped on Serenity, a final quiesced `gamevault` Postgres export is taken and imported to the PD target DB so PD does not come up on stale data - PD GameVault starts against the PD target Postgres DB - login, library visibility, metadata behavior, and save/upload paths verified - public/LAN route updated if needed and verified - Serenity `postgresql15` kept in place but clearly marked retirement-ready if cutover succeeds ### Card D6 — cut over RomM to PD Definition of done: - Serenity RomM stopped only for the final cutover window - after RomM is stopped on Serenity, a final quiesced `romm` MariaDB export is taken and imported to the PD target DB so PD does not come up on stale data - PD RomM starts against the PD target MariaDB DB - UI, library visibility, assets/resources, background jobs, and metadata behavior verified - public/LAN route updated if needed and verified - Serenity `MariaDB-Official` kept in place but clearly marked retirement-ready if cutover succeeds ### Card D7 — retire Serenity DB remnants after cooldown Definition of done: - GameVault and RomM remain healthy on PD through an observation window - Serenity `postgresql15` and `MariaDB-Official` are stopped and removed only after successful PD validation - Serenity-side DB appdata is retained for cooldown/rollback, not deleted immediately - docs and host inventory updated to show Serenity no longer carries those DB pairs ## Open item that still needs verification - `reranker` mounts `/mnt/user/appdate/reranker` - verify whether `appdate` is intentional before any future reranker move or cleanup ## Expected result after wave 1 After wave 1, Serenity should still be alive for the workloads that currently justify it, but with much less misleading baggage: - torrent/media-locality group still intact - Newt still intact - Technitium backup node still intact - GameVault and RomM still live until their migration is prepared - legacy Pi-hole gone - dead Cloudflared gone - stale created/exited clutter gone That leaves a cleaner host and a safer runway for the later PD storage cutover and full Serenity retirement. ## Wave 1 verification results (2026-05-25) Verified during this planning pass: - Serenity still has these live containers relevant to wave 1: - `technitium-dns-pilot` - `binhex-official-pihole` - `pihole-serenity` - `unbound-pihole-serenity` - `keepalived-pihole-serenity` - `Newt` - `Unraid-Cloudflared-Tunnel` - PD still runs the primary Technitium stack plus its own Pi-hole and Newt lane. - From NOMAD, `/etc/resolv.conf` currently lists `10.5.30.8`, `10.5.30.9`, and `10.5.30.10` ahead of external fallback `9.9.9.9`. - From NOMAD, `dig` to `10.5.30.8` and `10.5.30.10` succeeded for public DNS resolution; same-host checks to `10.5.30.9` were unreliable, matching the existing macvlan caveat in the docs. - `Unraid-Cloudflared-Tunnel` is still running, but repo docs already classify it as dead/stale and its container has `Restart=no`. - Serenity `Newt` must not be treated as deadwood: operator confirmed Pangolin tunnels Serenity resources through Serenity's local Newt instead of routing from PD or NOMAD to Serenity resources over the Serenity LAN IP. - Live Serenity `Newt` logs still show repeated Pangolin health checks against stale `10.5.1.5` targets (`7474`, `8785`, `8787`, `8788`, `8990`, `8457`, `5690`, `5454`). Operational implication: - removing Cloudflared remains low-risk after one final dependency check - removing the legacy Pi-hole stack remains appropriate - removing Serenity `Newt` is not appropriate during wave 1 - Pangolin target drift for Serenity-hosted resources should be repaired later by rehoming those resources to the correct Serenity site/local-path model instead of stale literal pre-migration IPs