docs: record PD qdrant/seerr drift cleanup incident

docs/operations/INCIDENT_2026-05-28_PD_QDRANT_SEERR_DRIFT.md

@@ -0,0 +1,126 @@
+# Incident: PD qdrant / seerr compose drift cleanup (2026-05-28)
+
+## Summary
+
+An earlier recovery on PlausibleDeniability left the runtime state healthier than the operator metadata around it.
+
+Public and LAN access were restored, but `qdrant` had been manually recreated from `docker inspect` data because its prior compose-path labeling was stale or absent. That fixed service availability, but it also created the risk that live state, repo state, and future operator assumptions could drift apart.
+
+On 2026-05-28, the stack was squared away by taking fresh backups, re-validating the live compose roots, re-syncing repo-managed compose content into the live directories, pruning a stale backup artifact from the live media compose directory, and re-verifying health.
+
+## What broke
+
+The incident had two layers:
+
+1. **Service health / accessibility breakage**
+   - `qdrant` and related AI/media surface area needed recovery work earlier in the day.
+   - `seerr` public/local availability also had to be checked as part of the same cleanup pass.
+
+2. **Post-recovery metadata drift risk**
+   - `qdrant` no longer had a trustworthy compose lifecycle trail from the original broken state.
+   - Manual recreation from `docker inspect` restored the container, but that is not the same as a clean repo-driven compose reconciliation.
+   - A stale file (`docker-compose.yaml.pre-d3b-20260526-003732`) was still sitting in the live media compose directory, which was not dangerous by itself but increased operator ambiguity.
+
+## What was manually recreated
+
+During the earlier recovery, `qdrant` was recreated manually from live container metadata when the prior compose-path labels were stale or missing.
+
+By the end of the cleanup, the running `qdrant` container correctly reported:
+
+- working dir: `/mnt/docker-ssd/docker/compose/ai`
+- compose file: `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml`
+- service: `qdrant`
+
+`seerr` correctly reported:
+
+- working dir: `/mnt/docker-ssd/docker/compose/media`
+- compose file: `/mnt/docker-ssd/docker/compose/media/docker-compose.yaml`
+- service: `seerr`
+
+## Backups taken before cleanup
+
+Fresh rollback-friendly backups were created on PD at:
+
+- directory: `/mnt/docker-ssd/docker/backups/post-incident-sync-20260528-192945`
+- tarball: `/mnt/docker-ssd/docker/backups/post-incident-sync-20260528-192945.tgz`
+
+Backup contents include:
+
+- live `ai` compose tree
+- live `media` compose tree
+- `qdrant` appdata
+- `docker inspect` output for `qdrant`, `openwebui`, and `seerr`
+- rendered `docker compose config` output for both stacks
+
+## Cleanup actions performed
+
+1. verified the live container labels for `qdrant`, `openwebui`, and `seerr`
+2. compared live compose files against the repo-managed copies under `truenas-stacks`
+3. confirmed the active tracked compose content already matched repo
+4. re-synced repo-managed `ai/` and `media/` compose trees into the live PD compose directories
+5. pruned the stale live file `docker-compose.yaml.pre-d3b-20260526-003732` from the media compose directory **after** backup
+6. re-ran `docker compose config` validation in both live directories
+7. re-checked HTTP health for `qdrant`, `openwebui`, and `seerr`
+
+## What was verified at the end
+
+### Live compose / label alignment
+
+The live runtime and repo are back in agreement for the affected services:
+
+- `qdrant` labels point to `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml`
+- `seerr` labels point to `/mnt/docker-ssd/docker/compose/media/docker-compose.yaml`
+- repo-managed tracked compose content matches the live active files
+
+### Health
+
+The following checks passed after cleanup:
+
+- `http://10.5.30.6:6333/healthz` → `200`
+- `http://10.5.30.6:8282/` → `200`
+- `http://10.5.30.6:5055/` → `200`
+
+### Compose validation
+
+These both rendered successfully:
+
+- `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml`
+- `/mnt/docker-ssd/docker/compose/media/docker-compose.yaml`
+
+## Important non-actions
+
+- The Docker daemon was **not** restarted.
+- The stacks were **not** redeployed during this cleanup pass.
+- No repo-unrelated working tree changes were touched.
+
+That was intentional: the active tracked compose files already matched repo, and the services were healthy, so a no-redeploy reconciliation was the safer path.
+
+## Why this matters
+
+The main value of this cleanup was not changing application behavior. It was restoring operator confidence that:
+
+- the live compose roots are the authoritative ones
+- the running containers point back to those roots
+- repo state and live state are aligned again
+- future maintenance can start from the compose directories instead of from ad hoc inspect output
+
+## Next time: safer recovery procedure
+
+If a PD stack service must be rebuilt under pressure:
+
+1. **Identify the authoritative compose directory first.**
+   - Prefer `/mnt/docker-ssd/docker/compose/<stack>/docker-compose.yaml` over ad hoc recreation.
+2. **Take a backup before cleanup, even if the service is already back up.**
+   - Capture compose trees, relevant appdata, and `docker inspect` output.
+3. **If manual recreation is unavoidable, treat it as temporary stabilization only.**
+   - Reconcile labels, compose roots, and repo state immediately afterward.
+4. **Compare repo ↔ live compose content before redeploying.**
+   - If tracked content already matches and health is good, avoid unnecessary churn.
+5. **Prune stale backup/override files from live compose dirs only after backup.**
+   - Leaving old compose fragments nearby makes future incident work harder.
+6. **Verify both health and metadata.**
+   - A container being up is not enough; confirm compose labels, working dir, and health endpoints.
+
+## Follow-up status
+
+No immediate further action was required after the cleanup pass. The remaining lesson is procedural: manual container recreation is acceptable as a short-term rescue, but it should always be followed by a backup-first repo/live reconciliation so drift does not become the new baseline.