# Serenity Cleanup Wave 1 Plan

Status: approved planning baseline for the first safe cleanup pass on Serenity.

## Goal

Reduce obvious legacy clutter on Serenity without breaking the still-needed torrent/media-locality group or the still-needed Serenity Newt path.

This wave is intentionally conservative.
It does not move qBittorrent/ARR off Serenity yet.
It does not retire Serenity yet.
It does not delete databases that still back live apps.

## Operator decisions already resolved

- Nothing should intentionally remain on Serenity after PD owns the disks locally.
- Technitium already covers the DNS role John wants.
- Serenity Pi-hole remnants should be treated as removable.
- Serenity Newt is still needed and must be preserved.
- GameVault and RomM should migrate, not be pruned.
- Final end-state remains:
  - move qbit + ARR family to PD after storage cutover
  - leave no intentional production app role on Serenity
  - retire Serenity entirely

## Scope of cleanup wave 1

Wave 1 includes only these categories:

1. Remove legacy DNS clutter that should no longer be serving production traffic.
2. Remove obviously stale created/exited containers.
3. Document migration order for the two database-backed apps that should move later.

Wave 1 explicitly excludes:

- qbit
- GluetunVPN
- qbit_manage
- prowlarr
- sonarr
- sonarr-anime
- radarr
- lidarr
- readarr
- readarr-epub
- bazarr
- autobrr
- unpackerr
- Notifiarr
- shelfmark
- Newt
- technitium-dns-pilot
- GameVault
- romm
- reranker

## Live facts this plan is based on

From the live Serenity audit:

- ARR/torrent locality is still tied to `/mnt/user/data`
- `GameVault` points at local Postgres on `10.5.30.5:5432`
- `RomM` points at local MariaDB on `10.5.30.5:3306`
- quick live checks did not surface immediate DB dependencies for `Wizarr`, `Shelfmark`, or `Notifiarr`
- `Newt` is still needed
- legacy Pi-hole containers are still running even though Technitium is now the intended DNS path
- Serenity Newt-backed Pangolin routes still had stale health-check hostnames pointing at old `10.5.1.5` even though their target IPs had already been rewritten to `localhost`/`10.5.30.5`
- a temporary `10.5.1.5/32` alias on `br0` validated the diagnosis, but it was removed because the old IP is no longer allowed on that VLAN
- authoritative Pangolin cleanup was then completed by rewriting the audited Serenity target set to `10.5.30.5` for both routing and health checks, removing the need for any legacy-IP workaround

## Wave 1 current execution status

Live execution on 2026-05-25 established:
- `Huntarr` and `omegabrr` were low-risk stale stopped containers and were removed from Serenity
- the recent non-running `Created` containers (`calibre-web`, `SuggestArr`, `Cleanuparr`, `calibre`, `agregarr`) were metadata-verified and then removed after confirming they were still only `Created`, had `restart=no`, and were not live workloads
- `Unraid-Cloudflared-Tunnel` was audited, stopped, and removed after verification showed it was only transport-alive and no current public traffic depended on it
- the Serenity Pi-hole HA stack (`binhex-official-pihole`, `pihole-serenity`, `unbound-pihole-serenity`, `keepalived-pihole-serenity`) was then stopped and removed after live mixed-host DNS checks confirmed the Technitium path remained healthy without it

## Wave 1-A: legacy Pi-hole removal

### Target containers

- `binhex-official-pihole`
- `pihole-serenity`
- `unbound-pihole-serenity`
- `keepalived-pihole-serenity`

### Why they are in scope

- They are legacy DNS/HA remnants.
- Current homelab docs describe the active internal DNS path as the Technitium trio.
- Operator confirmed Technitium covers the intended DNS role.
- Keeping old DNS stacks around increases confusion and future troubleshooting blast radius.

### Preconditions

Before removal, verify only these read-only checks:

1. Serenity Technitium backup node is healthy.
2. DHCP-advertised resolver set is still PD/NOMAD/Serenity Technitium, not Pi-hole.
3. No Pangolin route, bookmark, or admin workflow still intentionally points at a Pi-hole UI.
4. No host on the LAN still relies on the old Pi-hole admin port out of habit.

### Removal order

1. stop `keepalived-pihole-serenity`
2. stop `pihole-serenity`
3. stop `unbound-pihole-serenity`
4. stop `binhex-official-pihole`
5. verify Technitium-only DNS behavior still looks normal
6. remove the stopped containers
7. archive or delete their stale appdata only after a short observation window

### Verification after removal

- Serenity Technitium container remained healthy
- mixed-host DNS checks stayed good after removal:
  - from NOMAD, `10.5.30.8` and `10.5.30.10` still resolved both public and homelab names
  - from PD, `10.5.30.9` and `10.5.30.10` still resolved both public and homelab names
- `10.5.30.53` continued answering DNS even after Serenity Pi-hole removal, confirming it is no longer tied to the removed Serenity Pi-hole containers
- no immediate client-facing DNS regression was observed during the removal window
- no public regression was observed on sampled hostnames such as `panel.paccoco.com` and `audiobookshelf.paccoco.com`

## Wave 1-B: stale container pruning

### Created-only clutter to remove

- `calibre-web`
- `SuggestArr`
- `Cleanuparr`
- `calibre`
- `agregarr`

### Exited clutter to remove

- `Huntarr`
- `omegabrr`

### Why they are in scope

- They are not live workloads.
- They add noise to `docker ps -a` and make host intent harder to understand.
- There is no current architecture reason to preserve them as active Serenity residents.

### Safe pruning rules

Before deleting each one:

1. confirm container status is still `Created` or `Exited`
2. confirm it is not referenced by a live reverse-proxy route
3. confirm it is not the only source of some needed config/data you still care about
4. if uncertain, export one final metadata snapshot first:
   - image name
   - mounts
   - env file path if obvious

### Practical order

1. remove `Created` containers first
2. remove long-dead exited containers second
3. leave appdata in place initially
4. only delete appdata later after a short cooling-off window

## Wave 1-C: cloudflared deadwood removal

### Target container

- `Unraid-Cloudflared-Tunnel`

### Why it is in scope

- It is already documented in repo docs as dead/stale.
- Pangolin/Newt is the active exposure pattern now.
- Live audit on 2026-05-25 showed the container is a remote-managed Cloudflare Tunnel carrying a stale legacy config, not an active Serenity service path.

### Live audit findings (2026-05-25)

- `Unraid-Cloudflared-Tunnel` is healthy at the transport layer (`/ready` reported 4 ready connections), but metrics showed `cloudflared_tunnel_total_requests 0` and `cloudflared_tunnel_request_errors 0` for the current run.
- Startup logs exposed the remote-managed ingress set as legacy hostnames pointed at old `192.168.1.x` origins:
  - `wazuh.paccoco.com` -> `https://192.168.1.102`
  - `remotely.paccoco.com` -> `http://192.168.1.180:5001`
  - `hp.paccoco.com` -> `http://192.168.1.180:3000`
  - `octoprint.paccoco.com` -> `http://192.168.1.52`
  - `audiobookshelf.paccoco.com` -> `http://192.168.1.5:13378`
  - `spoolman.paccoco.com` -> `http://192.168.1.202:7912`
  - `node1.paccoco.com` -> `https://192.168.1.189:8080`
  - `hp2.paccoco.com` -> `http://192.168.1.224:3000`
  - `panel.paccoco.com` -> `https://192.168.1.189`
  - `nextcloud.paccoco.com` -> `http://192.168.1.5:11000`
- The tunnel has no local config files under `/mnt/user/appdata/cloudflared`; it is driven entirely by Cloudflare-side config plus a token.
- Public checks for those hostnames currently resolve to `172.245.79.139` and return either live responses or front-door 404s without producing any new tunnel traffic, indicating the present public path is elsewhere and not traversing this Serenity container.
- Repo/docs no longer identify this tunnel as an intended live exposure path; the only repeated modern exposure requirement in Serenity docs is the local Pangolin/Newt lane.

### Preconditions

1. verify no current DNS/public route still expects this tunnel
2. verify no local notes still treat it as the active exposure path
3. verify Newt-based routes are the real live path

Status after live audit:
- (1) satisfied enough for container retirement: no observed current public traffic is reaching this tunnel
- (2) satisfied: local docs treat it as stale legacy deadwood, not the intended active path
- (3) satisfied for Serenity-hosted apps: Pangolin/Newt remains the intentionally preserved exposure path

### Recommended retirement path

1. stop the container and watch briefly for any unexpected complaint or new public breakage
2. remove the container
3. keep `/mnt/user/appdata/cloudflared` during a cooling-off window even though it appears empty/unneeded
4. later, from the Cloudflare side, delete or repoint the stale tunnel config/hostnames if they still exist there

Status:
- completed on 2026-05-25: `Unraid-Cloudflared-Tunnel` was stopped and removed
- sampled public checks (`panel.paccoco.com`, `audiobookshelf.paccoco.com`) remained healthy after removal

## Wave 1-D: DB-backed migration ordering

These apps should not be deleted in wave 1.
They need planned migration.

### Pair 1: GameVault + local Postgres

Live dependency:
- `GameVault` -> `postgresql15`

Recommended sequence:
1. create PD-side target appdata path
2. create PD-side Postgres DB/user on shared-postgres, or a deliberate dedicated PD Postgres if there is a reason not to use shared-postgres
3. export GameVault DB from Serenity
4. import into PD target database
5. migrate GameVault appdata/config
6. recreate GameVault on PD attached to the shared database network if using shared-postgres
7. verify login, library visibility, and metadata path behavior
8. only then retire Serenity `postgresql15`

Default recommendation:
- prefer PD shared-postgres unless GameVault has a proven reason to stay isolated

### Pair 2: RomM + local MariaDB

Live dependency:
- `RomM` -> `MariaDB-Official`

Recommended sequence:
1. create PD-side target appdata path
2. create PD-side MariaDB DB/user on shared-mariadb, or a deliberate dedicated PD MariaDB only if needed
3. export RomM DB from Serenity
4. import into PD target MariaDB
5. migrate RomM appdata/config/assets/resources
6. recreate RomM on PD attached to the shared database network if using shared-mariadb
7. verify UI, library, metadata, and asset behavior
8. only then retire Serenity `MariaDB-Official`

Default recommendation:
- prefer PD shared-mariadb unless RomM proves awkward on the shared stack

## Recommended order across all wave 1 work

1. verify Technitium is the only intended active DNS path
2. remove legacy Pi-hole stack
3. remove dead Cloudflared tunnel
4. remove stale created/exited containers
5. leave GameVault/Postgres and RomM/MariaDB in place until their PD migration is prepared
6. keep qbit/ARR locality untouched until PD storage cutover is real

## Risks and guardrails

### Do not touch yet

Do not touch in this wave:
- qbit
- ARR family
- GluetunVPN
- qbit_manage
- Newt
- technitium-dns-pilot
- GameVault
- romm
- postgresql15
- MariaDB-Official

### Specific guardrails

- Do not delete any appdata directory in the same step as container removal unless the dependency is unquestionably dead.
- Do not remove `postgresql15` until GameVault is verified on PD.
- Do not remove `MariaDB-Official` until RomM is verified on PD.
- Do not move qbit/ARR until PD directly owns the relevant media/torrent paths.
- Do not break Serenity Newt while cleanup is happening.

## Suggested Kanban decomposition

### Card A1 — verify legacy Pi-hole is truly unused
Definition of done:
- current DNS path confirmed as Technitium-only
- no intentional admin dependency on Serenity Pi-hole remains

### Card A2 — remove Serenity legacy Pi-hole containers
Definition of done:
- all four legacy Pi-hole containers stopped and removed
- no DNS regression observed

### Card B1 — remove stale created containers
Definition of done:
- created-only clutter removed
- appdata retained for cooling-off period

### Card B2 — remove stale exited containers
Definition of done:
- exited clutter removed
- appdata retained for cooling-off period

### Card C1 — remove dead Unraid Cloudflared tunnel
Definition of done:
- no public path depends on it
- container removed

### Card D1 — prepare GameVault migration to PD
Definition of done:
- target DB/appdata path chosen
- export/import path documented
- cutover checklist ready

### Card D2 — prepare RomM migration to PD
Definition of done:
- target DB/appdata path chosen
- export/import path documented
- cutover checklist ready

## Open item that still needs verification

- `reranker` mounts `/mnt/user/appdate/reranker`
- verify whether `appdate` is intentional before any future reranker move or cleanup

## Expected result after wave 1

After wave 1, Serenity should still be alive for the workloads that currently justify it, but with much less misleading baggage:

- torrent/media-locality group still intact
- Newt still intact
- Technitium backup node still intact
- GameVault and RomM still live until their migration is prepared
- legacy Pi-hole gone
- dead Cloudflared gone
- stale created/exited clutter gone

That leaves a cleaner host and a safer runway for the later PD storage cutover and full Serenity retirement.

## Wave 1 verification results (2026-05-25)

Verified during this planning pass:

- Serenity still has these live containers relevant to wave 1:
  - `technitium-dns-pilot`
  - `binhex-official-pihole`
  - `pihole-serenity`
  - `unbound-pihole-serenity`
  - `keepalived-pihole-serenity`
  - `Newt`
  - `Unraid-Cloudflared-Tunnel`
- PD still runs the primary Technitium stack plus its own Pi-hole and Newt lane.
- From NOMAD, `/etc/resolv.conf` currently lists `10.5.30.8`, `10.5.30.9`, and `10.5.30.10` ahead of external fallback `9.9.9.9`.
- From NOMAD, `dig` to `10.5.30.8` and `10.5.30.10` succeeded for public DNS resolution; same-host checks to `10.5.30.9` were unreliable, matching the existing macvlan caveat in the docs.
- `Unraid-Cloudflared-Tunnel` is still running, but repo docs already classify it as dead/stale and its container has `Restart=no`.
- Serenity `Newt` must not be treated as deadwood: operator confirmed Pangolin tunnels Serenity resources through Serenity's local Newt instead of routing from PD or NOMAD to Serenity resources over the Serenity LAN IP.
- Live Serenity `Newt` logs still show repeated Pangolin health checks against stale `10.5.1.5` targets (`7474`, `8785`, `8787`, `8788`, `8990`, `8457`, `5690`, `5454`).

Operational implication:

- removing Cloudflared remains low-risk after one final dependency check
- removing the legacy Pi-hole stack remains appropriate
- removing Serenity `Newt` is not appropriate during wave 1
- Pangolin target drift for Serenity-hosted resources should be repaired later by rehoming those resources to the correct Serenity site/local-path model instead of stale literal pre-migration IPs