diff --git a/docs/planning/SERENITY_CLEANUP_WAVE_1.md b/docs/planning/SERENITY_CLEANUP_WAVE_1.md index 956000f..65b56c8 100644 --- a/docs/planning/SERENITY_CLEANUP_WAVE_1.md +++ b/docs/planning/SERENITY_CLEANUP_WAVE_1.md @@ -64,8 +64,8 @@ From the live Serenity audit: - quick live checks did not surface immediate DB dependencies for `Wizarr`, `Shelfmark`, or `Notifiarr` - `Newt` is still needed - legacy Pi-hole containers are still running even though Technitium is now the intended DNS path -- Serenity Newt-backed Pangolin routes still had stale health-check hostnames pointing at old `10.5.1.5` even though their target IPs had already been rewritten to `localhost` -- live workaround applied on 2026-05-25: Serenity now carries runtime alias `10.5.1.5/32` on `br0` so those stale health checks resolve locally again +- Serenity Newt-backed Pangolin routes still had stale health-check hostnames pointing at old `10.5.1.5` even though their target IPs had already been rewritten to `localhost`/`10.5.30.5` +- a temporary `10.5.1.5/32` alias on `br0` validated the diagnosis, but it was removed because the old IP is no longer allowed on that VLAN; authoritative Pangolin health-check cleanup is still required ## Wave 1-A: legacy Pi-hole removal diff --git a/docs/planning/SERENITY_DOCKER_AUDIT.md b/docs/planning/SERENITY_DOCKER_AUDIT.md index 6597812..5c0b5df 100644 --- a/docs/planning/SERENITY_DOCKER_AUDIT.md +++ b/docs/planning/SERENITY_DOCKER_AUDIT.md @@ -210,9 +210,9 @@ Resolved on 2026-05-25: - Is `/mnt/user/appdate/reranker` intentional, or a typo that should be corrected before migration? - When Pangolin API write auth is available again, rewrite Serenity target health-check hostnames away from stale `10.5.1.5` so the runtime alias can be removed cleanly. -## Pangolin / Newt live remediation applied +## Pangolin / Newt live remediation status -Live verification on 2026-05-25 showed Serenity Newt still probing stale pre-renumbering health-check URLs like `http://10.5.1.5:8787/` even though the Pangolin target objects already showed `ip=localhost` for those resources. +Live verification on 2026-05-25 showed Serenity Newt still probing stale pre-renumbering health-check URLs like `http://10.5.1.5:8787/` even though the Pangolin target objects already showed `ip=localhost` or `10.5.30.5` for those resources. Affected target IDs observed live: - `15` (`autobrr`) @@ -224,17 +224,24 @@ Affected target IDs observed live: - `58` (`romm`) - `69` (`gamevault`) -Fast remediation applied live on Serenity: -- `ip addr add 10.5.1.5/32 dev br0` +Diagnostic probe performed live on Serenity: +- temporarily added `10.5.1.5/32` to `br0` +- this immediately restored health for several targets, proving the stale `hcHostname` diagnosis -Post-fix verification: -- Newt logs recorded recovery to healthy for targets `15`, `20`, `25`, `29`, `56`, and `57` -- direct Serenity HTTP probes to `http://10.5.1.5:8785/` and `http://10.5.1.5:8457/` returned `200` -- inside the `Newt` container, `wget` to `10.5.1.5:8785` and `10.5.1.5:8457` succeeded +However, the old `10.5.1.5` address is no longer allowed on that VLAN, so the alias was removed again. + +Verification after removal: +- `ip addr del 10.5.1.5/32 dev br0` +- Newt immediately resumed failures such as: + - target `56` -> `http://10.5.1.5:8788/` + - target `57` -> `http://10.5.1.5:8990/` + - target `15` -> `http://10.5.1.5:7474/` + - target `25` -> `http://10.5.1.5:8787/` Interpretation: -- the immediate production symptom was repaired without waiting for Pangolin API write access -- this is only a runtime compatibility patch; it must be treated as temporary until the stale Pangolin `hcHostname` fields are authoritatively rewritten +- the alias was useful as a proof-of-cause test only +- it is not an acceptable steady-state fix here +- the real remaining task is to authoritatively rewrite the stale Pangolin `hcHostname` values away from `10.5.1.5` ## Database dependency findings diff --git a/docs/troubleshooting/KNOWN_QUIRKS.md b/docs/troubleshooting/KNOWN_QUIRKS.md index e04cb99..ee08cb9 100644 --- a/docs/troubleshooting/KNOWN_QUIRKS.md +++ b/docs/troubleshooting/KNOWN_QUIRKS.md @@ -71,15 +71,12 @@ Per-service gotchas that aren't bugs but will bite you if you forget them. - Dead container, should be removed ### Serenity Newt / Pangolin stale health-check IP drift -- Several Serenity Newt-backed Pangolin targets were corrected to `ip=localhost` but still retained stale `hcHostname=10.5.1.5`, so Newt health checks kept failing with `no route to host` +- Several Serenity Newt-backed Pangolin targets were corrected to `ip=localhost`/`10.5.30.5` but still retained stale `hcHostname=10.5.1.5`, so Newt health checks failed with `no route to host` - Verified affected target IDs on 2026-05-25: `15`, `20`, `25`, `29`, `56`, `57`, `58`, `69` -- Fast live recovery on Serenity: add runtime compatibility alias `10.5.1.5/32` on `br0` - - command used: `ip addr add 10.5.1.5/32 dev br0` -- Verified results after alias: - - Newt logged recovery to healthy for targets `15`, `20`, `25`, `29`, `56`, `57` - - direct HTTP probes to `http://10.5.1.5:8785/` and `http://10.5.1.5:8457/` returned `200` - - `docker exec Newt` confirmed `wget` success to ports `8785` and `8457` -- This is a runtime workaround, not an authoritative fix. It will disappear on reboot unless deliberately persisted, and the real cleanup is still to rewrite Pangolin health-check hostnames away from the stale pre-renumbering IP. +- A temporary local `/32` alias on Serenity (`ip addr add 10.5.1.5/32 dev br0`) proved the diagnosis by flipping multiple targets back to healthy +- That alias was then removed because the old `10.5.1.5` address is no longer allowed on the VLAN +- After removal, targets immediately fell back to unhealthy on the same stale health-check URLs, confirming the real problem is authoritative Pangolin metadata drift, not local app failure +- Treat the alias only as a diagnostic probe; do not persist it. The correct fix is to rewrite Pangolin health-check hostnames away from the stale pre-renumbering IP. ---