# Incident: PD qdrant / seerr compose drift cleanup (2026-05-28)

## Summary

An earlier recovery on PlausibleDeniability left the runtime state healthier than the operator metadata around it.

Public and LAN access were restored, but `qdrant` had been manually recreated from `docker inspect` data because its prior compose-path labeling was stale or absent. That fixed service availability, but it also created the risk that live state, repo state, and future operator assumptions could drift apart.

On 2026-05-28, the stack was squared away by taking fresh backups, re-validating the live compose roots, re-syncing repo-managed compose content into the live directories, pruning a stale backup artifact from the live media compose directory, and re-verifying health.

## What broke

The incident had two layers:

1. **Service health / accessibility breakage**
   - `qdrant` and related AI/media surface area needed recovery work earlier in the day.
   - `seerr` public/local availability also had to be checked as part of the same cleanup pass.

2. **Post-recovery metadata drift risk**
   - `qdrant` no longer had a trustworthy compose lifecycle trail from the original broken state.
   - Manual recreation from `docker inspect` restored the container, but that is not the same as a clean repo-driven compose reconciliation.
   - A stale file (`docker-compose.yaml.pre-d3b-20260526-003732`) was still sitting in the live media compose directory, which was not dangerous by itself but increased operator ambiguity.

## What was manually recreated

During the earlier recovery, `qdrant` was recreated manually from live container metadata when the prior compose-path labels were stale or missing.

By the end of the cleanup, the running `qdrant` container correctly reported:

- working dir: `/mnt/docker-ssd/docker/compose/ai`
- compose file: `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml`
- service: `qdrant`

`seerr` correctly reported:

- working dir: `/mnt/docker-ssd/docker/compose/media`
- compose file: `/mnt/docker-ssd/docker/compose/media/docker-compose.yaml`
- service: `seerr`

## Backups taken before cleanup

Fresh rollback-friendly backups were created on PD at:

- directory: `/mnt/docker-ssd/docker/backups/post-incident-sync-20260528-192945`
- tarball: `/mnt/docker-ssd/docker/backups/post-incident-sync-20260528-192945.tgz`

Backup contents include:

- live `ai` compose tree
- live `media` compose tree
- `qdrant` appdata
- `docker inspect` output for `qdrant`, `openwebui`, and `seerr`
- rendered `docker compose config` output for both stacks

## Cleanup actions performed

1. verified the live container labels for `qdrant`, `openwebui`, and `seerr`
2. compared live compose files against the repo-managed copies under `truenas-stacks`
3. confirmed the active tracked compose content already matched repo
4. re-synced repo-managed `ai/` and `media/` compose trees into the live PD compose directories
5. pruned the stale live file `docker-compose.yaml.pre-d3b-20260526-003732` from the media compose directory **after** backup
6. re-ran `docker compose config` validation in both live directories
7. re-checked HTTP health for `qdrant`, `openwebui`, and `seerr`

## What was verified at the end

### Live compose / label alignment

The live runtime and repo are back in agreement for the affected services:

- `qdrant` labels point to `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml`
- `seerr` labels point to `/mnt/docker-ssd/docker/compose/media/docker-compose.yaml`
- repo-managed tracked compose content matches the live active files

### Health

The following checks passed after cleanup:

- `http://10.5.30.6:6333/healthz` → `200`
- `http://10.5.30.6:8282/` → `200`
- `http://10.5.30.6:5055/` → `200`

### Compose validation

These both rendered successfully:

- `/mnt/docker-ssd/docker/compose/ai/docker-compose.yaml`
- `/mnt/docker-ssd/docker/compose/media/docker-compose.yaml`

## Important non-actions

- The Docker daemon was **not** restarted.
- The stacks were **not** redeployed during this cleanup pass.
- No repo-unrelated working tree changes were touched.

That was intentional: the active tracked compose files already matched repo, and the services were healthy, so a no-redeploy reconciliation was the safer path.

## Why this matters

The main value of this cleanup was not changing application behavior. It was restoring operator confidence that:

- the live compose roots are the authoritative ones
- the running containers point back to those roots
- repo state and live state are aligned again
- future maintenance can start from the compose directories instead of from ad hoc inspect output

## Next time: safer recovery procedure

If a PD stack service must be rebuilt under pressure:

1. **Identify the authoritative compose directory first.**
   - Prefer `/mnt/docker-ssd/docker/compose/<stack>/docker-compose.yaml` over ad hoc recreation.
2. **Take a backup before cleanup, even if the service is already back up.**
   - Capture compose trees, relevant appdata, and `docker inspect` output.
3. **If manual recreation is unavoidable, treat it as temporary stabilization only.**
   - Reconcile labels, compose roots, and repo state immediately afterward.
4. **Compare repo ↔ live compose content before redeploying.**
   - If tracked content already matches and health is good, avoid unnecessary churn.
5. **Prune stale backup/override files from live compose dirs only after backup.**
   - Leaving old compose fragments nearby makes future incident work harder.
6. **Verify both health and metadata.**
   - A container being up is not enough; confirm compose labels, working dir, and health endpoints.

## Follow-up status

No immediate further action was required after the cleanup pass. The remaining lesson is procedural: manual container recreation is acceptable as a short-term rescue, but it should always be followed by a backup-first repo/live reconciliation so drift does not become the new baseline.