truenas-stacks/docs/operations/INCIDENT_2026-05-28_PD_QDRANT_SEERR_DRIFT.md

Incident: PD qdrant / seerr compose drift cleanup (2026-05-28)

Summary

An earlier recovery on PlausibleDeniability left the runtime state healthier than the operator metadata around it.

Public and LAN access were restored, but qdrant had been manually recreated from docker inspect data because its prior compose-path labeling was stale or absent. That fixed service availability, but it also created the risk that live state, repo state, and future operator assumptions could drift apart.

On 2026-05-28, the stack was squared away by taking fresh backups, re-validating the live compose roots, re-syncing repo-managed compose content into the live directories, pruning a stale backup artifact from the live media compose directory, and re-verifying health.

What broke

The incident had two layers:

1. Service health / accessibility breakage

   • qdrant and related AI/media surface area needed recovery work earlier in the day.
   • seerr public/local availability also had to be checked as part of the same cleanup pass.

2. Post-recovery metadata drift risk

   • qdrant no longer had a trustworthy compose lifecycle trail from the original broken state.
   • Manual recreation from docker inspect restored the container, but that is not the same as a clean repo-driven compose reconciliation.
   • A stale file (docker-compose.yaml.pre-d3b-20260526-003732) was still sitting in the live media compose directory, which was not dangerous by itself but increased operator ambiguity.

What was manually recreated

During the earlier recovery, qdrant was recreated manually from live container metadata when the prior compose-path labels were stale or missing.

By the end of the cleanup, the running qdrant container correctly reported:

• working dir: /mnt/docker-ssd/docker/compose/ai
• compose file: /mnt/docker-ssd/docker/compose/ai/docker-compose.yaml
• service: qdrant

seerr correctly reported:

• working dir: /mnt/docker-ssd/docker/compose/media
• compose file: /mnt/docker-ssd/docker/compose/media/docker-compose.yaml
• service: seerr

Backups taken before cleanup

Fresh rollback-friendly backups were created on PD at:

• directory: /mnt/docker-ssd/docker/backups/post-incident-sync-20260528-192945
• tarball: /mnt/docker-ssd/docker/backups/post-incident-sync-20260528-192945.tgz

Backup contents include:

• live ai compose tree
• live media compose tree
• qdrant appdata
• docker inspect output for qdrant, openwebui, and seerr
• rendered docker compose config output for both stacks

Cleanup actions performed

1. verified the live container labels for qdrant, openwebui, and seerr
2. compared live compose files against the repo-managed copies under truenas-stacks
3. confirmed the active tracked compose content already matched repo
4. re-synced repo-managed ai/ and media/ compose trees into the live PD compose directories
5. pruned the stale live file docker-compose.yaml.pre-d3b-20260526-003732 from the media compose directory after backup
6. re-ran docker compose config validation in both live directories
7. re-checked HTTP health for qdrant, openwebui, and seerr

What was verified at the end

Live compose / label alignment

The live runtime and repo are back in agreement for the affected services:

• qdrant labels point to /mnt/docker-ssd/docker/compose/ai/docker-compose.yaml
• seerr labels point to /mnt/docker-ssd/docker/compose/media/docker-compose.yaml
• repo-managed tracked compose content matches the live active files

Health

The following checks passed after cleanup:

• http://10.5.30.6:6333/healthz → 200
• http://10.5.30.6:8282/ → 200
• http://10.5.30.6:5055/ → 200

Compose validation

These both rendered successfully:

• /mnt/docker-ssd/docker/compose/ai/docker-compose.yaml
• /mnt/docker-ssd/docker/compose/media/docker-compose.yaml

Important non-actions

• The Docker daemon was not restarted.
• The stacks were not redeployed during this cleanup pass.
• No repo-unrelated working tree changes were touched.

That was intentional: the active tracked compose files already matched repo, and the services were healthy, so a no-redeploy reconciliation was the safer path.

Why this matters

The main value of this cleanup was not changing application behavior. It was restoring operator confidence that:

• the live compose roots are the authoritative ones
• the running containers point back to those roots
• repo state and live state are aligned again
• future maintenance can start from the compose directories instead of from ad hoc inspect output

Next time: safer recovery procedure

If a PD stack service must be rebuilt under pressure:

1. Identify the authoritative compose directory first.
   • Prefer /mnt/docker-ssd/docker/compose/<stack>/docker-compose.yaml over ad hoc recreation.
2. Take a backup before cleanup, even if the service is already back up.
   • Capture compose trees, relevant appdata, and docker inspect output.
3. If manual recreation is unavoidable, treat it as temporary stabilization only.
   • Reconcile labels, compose roots, and repo state immediately afterward.
4. Compare repo ↔ live compose content before redeploying.
   • If tracked content already matches and health is good, avoid unnecessary churn.
5. Prune stale backup/override files from live compose dirs only after backup.
   • Leaving old compose fragments nearby makes future incident work harder.
6. Verify both health and metadata.
   • A container being up is not enough; confirm compose labels, working dir, and health endpoints.

Follow-up status

No immediate further action was required after the cleanup pass. The remaining lesson is procedural: manual container recreation is acceptable as a short-term rescue, but it should always be followed by a backup-first repo/live reconciliation so drift does not become the new baseline.