From db8dce8c8b409da7953abe6ea60f4f4bed710a38 Mon Sep 17 00:00:00 2001 From: Fizzlepoof Date: Fri, 29 May 2026 00:59:39 +0000 Subject: [PATCH] docs: record PD qdrant/seerr drift cleanup incident --- docs/README.md | 1 + ...CIDENT_2026-05-28_PD_QDRANT_SEERR_DRIFT.md | 126 ++++++++++++++++++ docs/troubleshooting/KNOWN_QUIRKS.md | 9 ++ 3 files changed, 136 insertions(+) create mode 100644 docs/operations/INCIDENT_2026-05-28_PD_QDRANT_SEERR_DRIFT.md diff --git a/docs/README.md b/docs/README.md index 94ecb83..247884a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -49,3 +49,4 @@ Central index for all homelab infrastructure documentation. | [Backup Policy](operations/BACKUP_POLICY.md) | Backup scope, database dumps, and restore cadence | | [PD Backup Deployment](operations/PD_BACKUP_DEPLOYMENT.md) | Live PD backup runner deployment details | | [Secrets Management](operations/SECRETS_MANAGEMENT.md) | `.env` handling, encrypted backups, and incident guardrails | +| [PD qdrant / seerr drift cleanup incident](operations/INCIDENT_2026-05-28_PD_QDRANT_SEERR_DRIFT.md) | Backup-first reconciliation after manual qdrant recovery and media/AI compose drift cleanup | diff --git a/docs/operations/INCIDENT_2026-05-28_PD_QDRANT_SEERR_DRIFT.md b/docs/operations/INCIDENT_2026-05-28_PD_QDRANT_SEERR_DRIFT.md new file mode 100644 index 0000000..a214d93 --- /dev/null +++ b/docs/operations/INCIDENT_2026-05-28_PD_QDRANT_SEERR_DRIFT.md @@ -0,0 +1,126 @@ +# Incident: PD qdrant / seerr compose drift cleanup (2026-05-28) + +## Summary + +An earlier recovery on PlausibleDeniability left the runtime state healthier than the operator metadata around it. + +Public and LAN access were restored, but `qdrant` had been manually recreated from `docker inspect` data because its prior compose-path labeling was stale or absent. That fixed service availability, but it also created the risk that live state, repo state, and future operator assumptions could drift apart. + +On 2026-05-28, the stack was squared away by taking fresh backups, re-validating the live compose roots, re-syncing repo-managed compose content into the live directories, pruning a stale backup artifact from the live media compose directory, and re-verifying health. + +## What broke + +The incident had two layers: + +1. **Service health / accessibility breakage** + - `qdrant` and related AI/media surface area needed recovery work earlier in the day. + - `seerr` public/local availability also had to be checked as part of the same cleanup pass. + +2. **Post-recovery metadata drift risk** + - `qdrant` no longer had a trustworthy compose lifecycle trail from the original broken state. + - Manual recreation from `docker inspect` restored the container, but that is not the same as a clean repo-driven compose reconciliation. + - A stale file (`docker-compose.yaml.pre-d3b-20260526-003732`) was still sitting in the live media compose directory, which was not dangerous by itself but increased operator ambiguity. + +## What was manually recreated + +During the earlier recovery, `qdrant` was recreated manually from live container metadata when the prior compose-path labels were stale or missing. + +By the end of the cleanup, the running `qdrant` container correctly reported: + +- working dir: `/mnt/docker-ssd/docker/compose/ai` +- compose file: `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml` +- service: `qdrant` + +`seerr` correctly reported: + +- working dir: `/mnt/docker-ssd/docker/compose/media` +- compose file: `/mnt/docker-ssd/docker/compose/media/docker-compose.yaml` +- service: `seerr` + +## Backups taken before cleanup + +Fresh rollback-friendly backups were created on PD at: + +- directory: `/mnt/docker-ssd/docker/backups/post-incident-sync-20260528-192945` +- tarball: `/mnt/docker-ssd/docker/backups/post-incident-sync-20260528-192945.tgz` + +Backup contents include: + +- live `ai` compose tree +- live `media` compose tree +- `qdrant` appdata +- `docker inspect` output for `qdrant`, `openwebui`, and `seerr` +- rendered `docker compose config` output for both stacks + +## Cleanup actions performed + +1. verified the live container labels for `qdrant`, `openwebui`, and `seerr` +2. compared live compose files against the repo-managed copies under `truenas-stacks` +3. confirmed the active tracked compose content already matched repo +4. re-synced repo-managed `ai/` and `media/` compose trees into the live PD compose directories +5. pruned the stale live file `docker-compose.yaml.pre-d3b-20260526-003732` from the media compose directory **after** backup +6. re-ran `docker compose config` validation in both live directories +7. re-checked HTTP health for `qdrant`, `openwebui`, and `seerr` + +## What was verified at the end + +### Live compose / label alignment + +The live runtime and repo are back in agreement for the affected services: + +- `qdrant` labels point to `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml` +- `seerr` labels point to `/mnt/docker-ssd/docker/compose/media/docker-compose.yaml` +- repo-managed tracked compose content matches the live active files + +### Health + +The following checks passed after cleanup: + +- `http://10.5.30.6:6333/healthz` → `200` +- `http://10.5.30.6:8282/` → `200` +- `http://10.5.30.6:5055/` → `200` + +### Compose validation + +These both rendered successfully: + +- `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml` +- `/mnt/docker-ssd/docker/compose/media/docker-compose.yaml` + +## Important non-actions + +- The Docker daemon was **not** restarted. +- The stacks were **not** redeployed during this cleanup pass. +- No repo-unrelated working tree changes were touched. + +That was intentional: the active tracked compose files already matched repo, and the services were healthy, so a no-redeploy reconciliation was the safer path. + +## Why this matters + +The main value of this cleanup was not changing application behavior. It was restoring operator confidence that: + +- the live compose roots are the authoritative ones +- the running containers point back to those roots +- repo state and live state are aligned again +- future maintenance can start from the compose directories instead of from ad hoc inspect output + +## Next time: safer recovery procedure + +If a PD stack service must be rebuilt under pressure: + +1. **Identify the authoritative compose directory first.** + - Prefer `/mnt/docker-ssd/docker/compose//docker-compose.yaml` over ad hoc recreation. +2. **Take a backup before cleanup, even if the service is already back up.** + - Capture compose trees, relevant appdata, and `docker inspect` output. +3. **If manual recreation is unavoidable, treat it as temporary stabilization only.** + - Reconcile labels, compose roots, and repo state immediately afterward. +4. **Compare repo ↔ live compose content before redeploying.** + - If tracked content already matches and health is good, avoid unnecessary churn. +5. **Prune stale backup/override files from live compose dirs only after backup.** + - Leaving old compose fragments nearby makes future incident work harder. +6. **Verify both health and metadata.** + - A container being up is not enough; confirm compose labels, working dir, and health endpoints. + +## Follow-up status + +No immediate further action was required after the cleanup pass. The remaining lesson is procedural: manual container recreation is acceptable as a short-term rescue, but it should always be followed by a backup-first repo/live reconciliation so drift does not become the new baseline. diff --git a/docs/troubleshooting/KNOWN_QUIRKS.md b/docs/troubleshooting/KNOWN_QUIRKS.md index 7cc8f03..c73e5a0 100644 --- a/docs/troubleshooting/KNOWN_QUIRKS.md +++ b/docs/troubleshooting/KNOWN_QUIRKS.md @@ -42,6 +42,15 @@ Per-service gotchas that aren't bugs but will bite you if you forget them. - DB: user=`openwebui`, db=`openwebui` on shared-postgres - If restart-looping with auth errors: container wasn't on `ix-databases_shared-databases` at creation time — must `down && up` +### qdrant +- If the container has to be manually recreated from `docker inspect` because compose labels are stale or missing, treat that as emergency stabilization only, not final cleanup +- After emergency recovery, immediately reconcile the runtime back to `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml`, back up the live state, and verify the container labels again before calling the incident closed +- The container being `Up` is not enough; also confirm `com.docker.compose.project.working_dir`, `com.docker.compose.project.config_files`, and `/healthz` + +### seerr +- After adjacent media-stack incident work, verify both LAN/public availability and that the container still points at `/mnt/docker-ssd/docker/compose/media/docker-compose.yaml` +- Stale `*.pre-*` compose backups do not belong in the live compose directory long-term; back them up elsewhere, then prune them to reduce operator ambiguity + ### plex - Port `5353/udp` conflicts with system avahi/mDNS — remove from port mappings