docs(nomad): capture container ownership and recovery notes

2026-05-14 17:35:24 +00:00
parent 991ed7a6ae
commit e4e48394b8
2 changed files with 204 additions and 5 deletions
--- a/docs/planning/TODO.md
+++ b/docs/planning/TODO.md
@@ -32,7 +32,11 @@

 ## Documentation
 - [ ] Document meshtastic stack fully (see docs/reference/MESHTASTIC.md)
- [ ] Document N.O.M.A.D. containers (start_nomad.sh services)
+- [x] Document N.O.M.A.D. Docker/container audit (compose-managed, Project N.O.M.A.D.-managed, unmanaged, and Pelican/Wings-managed containers) — 2026-05-14
+- [x] Decide which services should run on NOMAD, which should not, and whether each should live under raw compose, Project N.O.M.A.D.-managed state, systemd, or another more explicit deployment model — 2026-05-14
+  - Project N.O.M.A.D.-managed app containers are explicitly hands-off and stay exactly as-is.
+  - `whisper` and `node-exporter` may be removed; exact runtime was documented in `docs/servers/NOMAD.md` for emergency rebuild.
+  - Pelican/Wings-managed game containers stay and are documented as permanent game-server infrastructure on NOMAD.

 ## n8n Workflows
 - [ ] Create Gotify apps and n8n credentials: Reminders, n8n Health, Uptime / Infra
@@ -41,8 +45,8 @@
 - [ ] Add sudoers NOPASSWD rule for n8n watchdog (see n8n-workflows/README.md)
 - [ ] Import and activate upgraded workflows: 04, 08 (01 done — v1.4 active and tested)
 - [ ] Import and activate new workflows: 14, 15 (16 already active)
- [ ] Build Telegram chat-driven school-work intake for Paperless using shared Postgres, deterministic `intake_id`, and class/version metadata
 - [ ] Configure class profiles + Paperless metadata mapping for current courses (ENGL-1010, HIST-2020, future classes)
+- [x] Delete legacy duplicate Paperless notes for school docs 42–49 after approval / with proper token context — completed 2026-05-14; docs now each have one canonical note
 - [ ] Set up DB and Docker backups to Serenity following homelab SOP

 ## Completed
@@ -62,3 +66,4 @@
 - [x] Create Postgres dedup tables (grafana_alert_dedup, n8n_health_dedup) in n8n_rss DB on PD — 2026-05-09
 - [x] Workflow 01 fully tested end-to-end (dedup, LiteLLM, Gotify) — v1.4 active — 2026-05-09
 - [x] Add LITELLM_API_KEY + N8N_BLOCK_ENV_ACCESS_IN_NODE to n8n docker-compose — 2026-05-09
+- [x] Build Telegram chat-driven school-work intake for Paperless using shared Postgres, deterministic `intake_id`, and class/version metadata — 2026-05-14
--- a/docs/servers/NOMAD.md
+++ b/docs/servers/NOMAD.md
@@ -21,10 +21,195 @@ Offline survival/knowledge server and game server host.
 | Seagate SSD (sdd) | 223.6GB | `/mnt/ssd-1` | Fast scratch |

 ## Project N.O.M.A.D.
- 10 containers running via `start_nomad.sh` / `stop_nomad.sh`
- `/opt/project-nomad` → symlinked from `/mnt/hdd-1/project-nomad`
+- Core compose project lives at `/opt/project-nomad` → symlinked from `/mnt/hdd-1/project-nomad`
+- Main compose file: `/mnt/hdd-1/project-nomad/compose.yml`
 - Docker data-root: `/mnt/hdd-2/docker`
 - Based on Crosstalk Solutions offline survival/knowledge server
+- Current runtime is split between:
+  - the base `project-nomad` compose stack
+  - `project-nomad-managed` app containers launched/managed by the N.O.M.A.D. admin layer
+  - a few extra non-compose / non-NOMAD containers on the host
+
+## Docker Audit — 2026-05-14
+
+### Compose files found on NOMAD
+- `/mnt/hdd-1/project-nomad/compose.yml` (same project as `/opt/project-nomad/compose.yml` via symlink)
+- `/opt/newt/docker-compose.yaml`
+
+### Running containers tied directly to compose files
+**`project-nomad` compose (`/mnt/hdd-1/project-nomad/compose.yml`)**
+- `nomad_admin`
+- `nomad_mysql`
+- `nomad_redis`
+- `nomad_dozzle`
+- `nomad_disk_collector`
+- `nomad_updater`
+
+**`newt` compose (`/opt/newt/docker-compose.yaml`)**
+- `newt`
+
+### Running containers managed by Project N.O.M.A.D. itself
+These are labeled `com.docker.compose.project=project-nomad-managed` and `io.project-nomad.managed=true`, but no standalone compose files for them were found on disk during this audit.
+- `nomad_cyberchef`
+- `nomad_flatnotes`
+- `nomad_kiwix_server`
+- `nomad_kolibri`
+- `nomad_ollama`
+- `nomad_qdrant`
+
+**Operator decision (John, 2026-05-14):** this layer stays exactly as-is. It is self-maintained by Project N.O.M.A.D. Document it, but **never manually touch, rebuild, migrate, or “clean up” these containers** unless John explicitly overrides that rule.
+
+Known storage paths discovered from mounts / project storage:
+- Flatnotes data: `/opt/project-nomad/storage/flatnotes`
+- Kiwix ZIM library: `/opt/project-nomad/storage/zim`
+- Kolibri data: `/opt/project-nomad/storage/kolibri`
+- Ollama data/models: `/opt/project-nomad/storage/ollama`
+- Qdrant storage: `/opt/project-nomad/storage/qdrant`
+
+### Running containers not tied to discovered compose files
+**Likely host-managed / manually launched / app-managed elsewhere**
+
+#### `whisper` — `fedirz/faster-whisper-server:latest-cpu`
+**Operator decision (John, 2026-05-14):** this can go away. Do not preserve it as a required service, but keep enough detail here to rebuild it if removing it causes issues.
+- Purpose: ad-hoc Faster Whisper API endpoint
+- Network mode: `bridge`
+- Container IP during audit: `172.17.0.2`
+- Port publish: host `8786/tcp` → container `8000/tcp` (IPv4 and IPv6)
+- Restart policy: `unless-stopped`
+- Entrypoint: image default
+- Command:
+  - `uv run uvicorn --factory faster_whisper_server.main:create_app`
+- Environment:
+  - `WHISPER__MODEL=Systran/faster-distil-whisper-small.en`
+  - `WHISPER__INFERENCE_DEVICE=cpu`
+  - `UVICORN_HOST=0.0.0.0`
+  - `UVICORN_PORT=8000`
+- Mounts: none
+- Compose labels: none found
+- Rebuild-equivalent `docker run`:
+  ```bash
+  docker run -d \
+    --name whisper \
+    --restart unless-stopped \
+    -p 8786:8000 \
+    -e WHISPER__MODEL=Systran/faster-distil-whisper-small.en \
+    -e WHISPER__INFERENCE_DEVICE=cpu \
+    -e UVICORN_HOST=0.0.0.0 \
+    -e UVICORN_PORT=8000 \
+    fedirz/faster-whisper-server:latest-cpu \
+    uv run uvicorn --factory faster_whisper_server.main:create_app
+  ```
+
+#### `node-exporter` — `prom/node-exporter:v1.9.0`
+**Operator decision (John, 2026-05-14):** this can go away. Do not preserve it as a required service, but keep enough detail here to rebuild it if removing it causes issues.
+- Purpose: host Prometheus metrics exporter
+- Network mode: `host`
+- Restart policy: `unless-stopped`
+- Entrypoint: `/bin/node_exporter`
+- Command:
+  - `--path.procfs=/host/proc`
+  - `--path.sysfs=/host/sys`
+  - `--path.rootfs=/rootfs`
+  - `--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)`
+- Read-only bind mounts:
+  - `/proc` → `/host/proc`
+  - `/sys` → `/host/sys`
+  - `/` → `/rootfs`
+- Compose labels: none found
+- Rebuild-equivalent `docker run`:
+  ```bash
+  docker run -d \
+    --name node-exporter \
+    --restart unless-stopped \
+    --network host \
+    -v /proc:/host/proc:ro \
+    -v /sys:/host/sys:ro \
+    -v /:/rootfs:ro \
+    prom/node-exporter:v1.9.0 \
+    --path.procfs=/host/proc \
+    --path.sysfs=/host/sys \
+    --path.rootfs=/rootfs \
+    '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
+  ```
+
+**Pelican/Wings-managed game containers**
+**Operator decision (John, 2026-05-14):** this is the game server stack. It stays. Document it; do not remove or migrate it unless John explicitly says so.
+
+- `11d799cb-487a-416a-b6f9-9bf28f0e57e7`
+  - Image: `ghcr.io/pelican-eggs/yolks:java_25`
+  - Network: `pelican_nw` (`172.20.0.2` during audit)
+  - Ports: `25565/tcp` and `25565/udp` exposed on host `0.0.0.0`
+  - Restart policy: `no` (Wings owns lifecycle)
+  - Entrypoint: `/usr/bin/tini -g --`
+  - Command: `/entrypoint.sh`
+  - Important env:
+    - `SERVER_IP=0.0.0.0`
+    - `SERVER_PORT=25565`
+    - `SERVER_JARFILE=server.jar`
+    - `STARTUP=java --add-modules=jdk.incubator.vector -Xms128M -XX:MaxRAMPercentage=95.0 -Dterminal.jline=false -Dterminal.ansi=true -jar server.jar`
+    - `P_SERVER_UUID=11d799cb-487a-416a-b6f9-9bf28f0e57e7`
+  - Mounts:
+    - `/var/lib/pelican/volumes/11d799cb-487a-416a-b6f9-9bf28f0e57e7` → `/home/container` (rw)
+    - `/etc/pelican/passwd` → `/etc/passwd` (ro)
+    - `/etc/pelican/group` → `/etc/group` (ro)
+    - `/etc/pelican/machine-id/11d799cb-487a-416a-b6f9-9bf28f0e57e7` → `/etc/machine-id` (ro)
+
+- `d2aca31e-0d4d-433a-8295-cfad397dab0c`
+  - Image: `ghcr.io/pelican-eggs/yolks:java_21`
+  - Network: `pelican_nw` (`172.20.0.3` during audit)
+  - Ports: `25566/tcp` and `25566/udp` exposed on host `0.0.0.0`
+  - Restart policy: `no` (Wings owns lifecycle)
+  - Entrypoint: `/usr/bin/tini -g --`
+  - Command: `/entrypoint.sh`
+  - Important env:
+    - `SERVER_IP=0.0.0.0`
+    - `SERVER_PORT=25566`
+    - `SERVER_MEMORY=8192`
+    - `MC_VERSION=1.21.1`
+    - `NEOFORGE_VERSION=21.1.229`
+    - `STARTUP=java -Xms128M -XX:MaxRAMPercentage=95.0 -Dterminal.jline=false -Dterminal.ansi=true @unix_args.txt`
+    - `P_SERVER_UUID=d2aca31e-0d4d-433a-8295-cfad397dab0c`
+  - Mounts:
+    - `/var/lib/pelican/volumes/d2aca31e-0d4d-433a-8295-cfad397dab0c` → `/home/container` (rw)
+    - `/etc/pelican/passwd` → `/etc/passwd` (ro)
+    - `/etc/pelican/group` → `/etc/group` (ro)
+    - `/etc/pelican/machine-id/d2aca31e-0d4d-433a-8295-cfad397dab0c` → `/etc/machine-id` (ro)
+
+These are expected to be managed by Wings/Pelican rather than a local compose file.
+
+### Dormant compose files found but not currently represented in running containers
+- None found during this audit in searched paths.
+- Search scope: `/opt`, `/srv`, `/home/fizzlepoof`, and `/mnt` for `docker-compose.yml`, `docker-compose.yaml`, `compose.yml`, and `compose.yaml`.
+
+### Decisions captured from this audit
+- Project N.O.M.A.D.-managed app containers stay exactly as-is and are **hands-off** unless John explicitly says otherwise.
+- `whisper` and `node-exporter` are not protected services; they can be removed, but their exact runtime has been documented above for rebuild/recovery.
+- Pelican/Wings-managed game containers are part of the permanent game-server stack on NOMAD and stay in place.
+
+### Safe to remove / how to restore
+#### Safe to remove now
+- `whisper`
+- `node-exporter`
+
+Removing them should not be treated as a Project N.O.M.A.D. change or a Wings/Pelican change. They are standalone host-level containers from this audit’s point of view.
+
+#### Restore checklist if removal causes trouble
+1. Confirm what broke:
+   - speech/transcription endpoint dependency → restore `whisper`
+   - Prometheus scraping / host metrics gap → restore `node-exporter`
+2. Recreate the container using the documented `docker run` command in this file.
+3. Verify it came back cleanly:
+   - `docker ps --filter name=whisper`
+   - `docker ps --filter name=node-exporter`
+   - `docker logs --tail 50 whisper`
+   - `docker logs --tail 50 node-exporter`
+4. Re-check any dependent dashboards, scrapers, or automations.
+5. If the recreated container needs to persist long-term again, document *why* before leaving it in place.
+
+#### Restore intent
+- Restore `whisper` only if something still expects a local Faster Whisper HTTP endpoint on port `8786`.
+- Restore `node-exporter` only if something still depends on host Prometheus metrics from this box.
+- Do **not** fold either one into Project N.O.M.A.D. or Wings/Pelican casually; if they come back, they come back as deliberate standalone services unless John says otherwise.

 ## Pelican Panel (Game Server Management)
 - **URL:** `https://panel.paccoco.com`
@@ -49,7 +234,16 @@ Offline survival/knowledge server and game server host.
 - **Docker network:** `pelican_nw` (172.20.0.0/16)

 ## Active Game Servers
- **Minecraft:** Purpur, Java 21, port 25565 TCP/UDP (direct host port forward)
+- **Container `11d799cb-487a-416a-b6f9-9bf28f0e57e7`**
+  - Workload: Minecraft / Purpur-style Java server
+  - Java runtime image: `ghcr.io/pelican-eggs/yolks:java_25`
+  - Public port: `25565` TCP/UDP
+  - Data path: `/var/lib/pelican/volumes/11d799cb-487a-416a-b6f9-9bf28f0e57e7`
+- **Container `d2aca31e-0d4d-433a-8295-cfad397dab0c`**
+  - Workload: modded Minecraft-style server (NeoForge-related env present)
+  - Java runtime image: `ghcr.io/pelican-eggs/yolks:java_21`
+  - Public port: `25566` TCP/UDP
+  - Data path: `/var/lib/pelican/volumes/d2aca31e-0d4d-433a-8295-cfad397dab0c`

 ## Networking Notes
 - `panel.paccoco.com` → `127.0.0.1` in `/etc/hosts` for NAT loopback