diff --git a/docs/planning/TODO.md b/docs/planning/TODO.md index 581cd30..4b964e5 100644 --- a/docs/planning/TODO.md +++ b/docs/planning/TODO.md @@ -32,7 +32,11 @@ ## Documentation - [ ] Document meshtastic stack fully (see docs/reference/MESHTASTIC.md) -- [ ] Document N.O.M.A.D. containers (start_nomad.sh services) +- [x] Document N.O.M.A.D. Docker/container audit (compose-managed, Project N.O.M.A.D.-managed, unmanaged, and Pelican/Wings-managed containers) — 2026-05-14 +- [x] Decide which services should run on NOMAD, which should not, and whether each should live under raw compose, Project N.O.M.A.D.-managed state, systemd, or another more explicit deployment model — 2026-05-14 + - Project N.O.M.A.D.-managed app containers are explicitly hands-off and stay exactly as-is. + - `whisper` and `node-exporter` may be removed; exact runtime was documented in `docs/servers/NOMAD.md` for emergency rebuild. + - Pelican/Wings-managed game containers stay and are documented as permanent game-server infrastructure on NOMAD. ## n8n Workflows - [ ] Create Gotify apps and n8n credentials: Reminders, n8n Health, Uptime / Infra @@ -41,8 +45,8 @@ - [ ] Add sudoers NOPASSWD rule for n8n watchdog (see n8n-workflows/README.md) - [ ] Import and activate upgraded workflows: 04, 08 (01 done — v1.4 active and tested) - [ ] Import and activate new workflows: 14, 15 (16 already active) -- [ ] Build Telegram chat-driven school-work intake for Paperless using shared Postgres, deterministic `intake_id`, and class/version metadata - [ ] Configure class profiles + Paperless metadata mapping for current courses (ENGL-1010, HIST-2020, future classes) +- [x] Delete legacy duplicate Paperless notes for school docs 42–49 after approval / with proper token context — completed 2026-05-14; docs now each have one canonical note - [ ] Set up DB and Docker backups to Serenity following homelab SOP ## Completed @@ -62,3 +66,4 @@ - [x] Create Postgres dedup tables (grafana_alert_dedup, n8n_health_dedup) in n8n_rss DB on PD — 2026-05-09 - [x] Workflow 01 fully tested end-to-end (dedup, LiteLLM, Gotify) — v1.4 active — 2026-05-09 - [x] Add LITELLM_API_KEY + N8N_BLOCK_ENV_ACCESS_IN_NODE to n8n docker-compose — 2026-05-09 +- [x] Build Telegram chat-driven school-work intake for Paperless using shared Postgres, deterministic `intake_id`, and class/version metadata — 2026-05-14 diff --git a/docs/servers/NOMAD.md b/docs/servers/NOMAD.md index cac5f49..98d2bee 100644 --- a/docs/servers/NOMAD.md +++ b/docs/servers/NOMAD.md @@ -21,10 +21,195 @@ Offline survival/knowledge server and game server host. | Seagate SSD (sdd) | 223.6GB | `/mnt/ssd-1` | Fast scratch | ## Project N.O.M.A.D. -- 10 containers running via `start_nomad.sh` / `stop_nomad.sh` -- `/opt/project-nomad` → symlinked from `/mnt/hdd-1/project-nomad` +- Core compose project lives at `/opt/project-nomad` → symlinked from `/mnt/hdd-1/project-nomad` +- Main compose file: `/mnt/hdd-1/project-nomad/compose.yml` - Docker data-root: `/mnt/hdd-2/docker` - Based on Crosstalk Solutions offline survival/knowledge server +- Current runtime is split between: + - the base `project-nomad` compose stack + - `project-nomad-managed` app containers launched/managed by the N.O.M.A.D. admin layer + - a few extra non-compose / non-NOMAD containers on the host + +## Docker Audit — 2026-05-14 + +### Compose files found on NOMAD +- `/mnt/hdd-1/project-nomad/compose.yml` (same project as `/opt/project-nomad/compose.yml` via symlink) +- `/opt/newt/docker-compose.yaml` + +### Running containers tied directly to compose files +**`project-nomad` compose (`/mnt/hdd-1/project-nomad/compose.yml`)** +- `nomad_admin` +- `nomad_mysql` +- `nomad_redis` +- `nomad_dozzle` +- `nomad_disk_collector` +- `nomad_updater` + +**`newt` compose (`/opt/newt/docker-compose.yaml`)** +- `newt` + +### Running containers managed by Project N.O.M.A.D. itself +These are labeled `com.docker.compose.project=project-nomad-managed` and `io.project-nomad.managed=true`, but no standalone compose files for them were found on disk during this audit. +- `nomad_cyberchef` +- `nomad_flatnotes` +- `nomad_kiwix_server` +- `nomad_kolibri` +- `nomad_ollama` +- `nomad_qdrant` + +**Operator decision (John, 2026-05-14):** this layer stays exactly as-is. It is self-maintained by Project N.O.M.A.D. Document it, but **never manually touch, rebuild, migrate, or “clean up” these containers** unless John explicitly overrides that rule. + +Known storage paths discovered from mounts / project storage: +- Flatnotes data: `/opt/project-nomad/storage/flatnotes` +- Kiwix ZIM library: `/opt/project-nomad/storage/zim` +- Kolibri data: `/opt/project-nomad/storage/kolibri` +- Ollama data/models: `/opt/project-nomad/storage/ollama` +- Qdrant storage: `/opt/project-nomad/storage/qdrant` + +### Running containers not tied to discovered compose files +**Likely host-managed / manually launched / app-managed elsewhere** + +#### `whisper` — `fedirz/faster-whisper-server:latest-cpu` +**Operator decision (John, 2026-05-14):** this can go away. Do not preserve it as a required service, but keep enough detail here to rebuild it if removing it causes issues. +- Purpose: ad-hoc Faster Whisper API endpoint +- Network mode: `bridge` +- Container IP during audit: `172.17.0.2` +- Port publish: host `8786/tcp` → container `8000/tcp` (IPv4 and IPv6) +- Restart policy: `unless-stopped` +- Entrypoint: image default +- Command: + - `uv run uvicorn --factory faster_whisper_server.main:create_app` +- Environment: + - `WHISPER__MODEL=Systran/faster-distil-whisper-small.en` + - `WHISPER__INFERENCE_DEVICE=cpu` + - `UVICORN_HOST=0.0.0.0` + - `UVICORN_PORT=8000` +- Mounts: none +- Compose labels: none found +- Rebuild-equivalent `docker run`: + ```bash + docker run -d \ + --name whisper \ + --restart unless-stopped \ + -p 8786:8000 \ + -e WHISPER__MODEL=Systran/faster-distil-whisper-small.en \ + -e WHISPER__INFERENCE_DEVICE=cpu \ + -e UVICORN_HOST=0.0.0.0 \ + -e UVICORN_PORT=8000 \ + fedirz/faster-whisper-server:latest-cpu \ + uv run uvicorn --factory faster_whisper_server.main:create_app + ``` + +#### `node-exporter` — `prom/node-exporter:v1.9.0` +**Operator decision (John, 2026-05-14):** this can go away. Do not preserve it as a required service, but keep enough detail here to rebuild it if removing it causes issues. +- Purpose: host Prometheus metrics exporter +- Network mode: `host` +- Restart policy: `unless-stopped` +- Entrypoint: `/bin/node_exporter` +- Command: + - `--path.procfs=/host/proc` + - `--path.sysfs=/host/sys` + - `--path.rootfs=/rootfs` + - `--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)` +- Read-only bind mounts: + - `/proc` → `/host/proc` + - `/sys` → `/host/sys` + - `/` → `/rootfs` +- Compose labels: none found +- Rebuild-equivalent `docker run`: + ```bash + docker run -d \ + --name node-exporter \ + --restart unless-stopped \ + --network host \ + -v /proc:/host/proc:ro \ + -v /sys:/host/sys:ro \ + -v /:/rootfs:ro \ + prom/node-exporter:v1.9.0 \ + --path.procfs=/host/proc \ + --path.sysfs=/host/sys \ + --path.rootfs=/rootfs \ + '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)' + ``` + +**Pelican/Wings-managed game containers** +**Operator decision (John, 2026-05-14):** this is the game server stack. It stays. Document it; do not remove or migrate it unless John explicitly says so. + +- `11d799cb-487a-416a-b6f9-9bf28f0e57e7` + - Image: `ghcr.io/pelican-eggs/yolks:java_25` + - Network: `pelican_nw` (`172.20.0.2` during audit) + - Ports: `25565/tcp` and `25565/udp` exposed on host `0.0.0.0` + - Restart policy: `no` (Wings owns lifecycle) + - Entrypoint: `/usr/bin/tini -g --` + - Command: `/entrypoint.sh` + - Important env: + - `SERVER_IP=0.0.0.0` + - `SERVER_PORT=25565` + - `SERVER_JARFILE=server.jar` + - `STARTUP=java --add-modules=jdk.incubator.vector -Xms128M -XX:MaxRAMPercentage=95.0 -Dterminal.jline=false -Dterminal.ansi=true -jar server.jar` + - `P_SERVER_UUID=11d799cb-487a-416a-b6f9-9bf28f0e57e7` + - Mounts: + - `/var/lib/pelican/volumes/11d799cb-487a-416a-b6f9-9bf28f0e57e7` → `/home/container` (rw) + - `/etc/pelican/passwd` → `/etc/passwd` (ro) + - `/etc/pelican/group` → `/etc/group` (ro) + - `/etc/pelican/machine-id/11d799cb-487a-416a-b6f9-9bf28f0e57e7` → `/etc/machine-id` (ro) + +- `d2aca31e-0d4d-433a-8295-cfad397dab0c` + - Image: `ghcr.io/pelican-eggs/yolks:java_21` + - Network: `pelican_nw` (`172.20.0.3` during audit) + - Ports: `25566/tcp` and `25566/udp` exposed on host `0.0.0.0` + - Restart policy: `no` (Wings owns lifecycle) + - Entrypoint: `/usr/bin/tini -g --` + - Command: `/entrypoint.sh` + - Important env: + - `SERVER_IP=0.0.0.0` + - `SERVER_PORT=25566` + - `SERVER_MEMORY=8192` + - `MC_VERSION=1.21.1` + - `NEOFORGE_VERSION=21.1.229` + - `STARTUP=java -Xms128M -XX:MaxRAMPercentage=95.0 -Dterminal.jline=false -Dterminal.ansi=true @unix_args.txt` + - `P_SERVER_UUID=d2aca31e-0d4d-433a-8295-cfad397dab0c` + - Mounts: + - `/var/lib/pelican/volumes/d2aca31e-0d4d-433a-8295-cfad397dab0c` → `/home/container` (rw) + - `/etc/pelican/passwd` → `/etc/passwd` (ro) + - `/etc/pelican/group` → `/etc/group` (ro) + - `/etc/pelican/machine-id/d2aca31e-0d4d-433a-8295-cfad397dab0c` → `/etc/machine-id` (ro) + +These are expected to be managed by Wings/Pelican rather than a local compose file. + +### Dormant compose files found but not currently represented in running containers +- None found during this audit in searched paths. +- Search scope: `/opt`, `/srv`, `/home/fizzlepoof`, and `/mnt` for `docker-compose.yml`, `docker-compose.yaml`, `compose.yml`, and `compose.yaml`. + +### Decisions captured from this audit +- Project N.O.M.A.D.-managed app containers stay exactly as-is and are **hands-off** unless John explicitly says otherwise. +- `whisper` and `node-exporter` are not protected services; they can be removed, but their exact runtime has been documented above for rebuild/recovery. +- Pelican/Wings-managed game containers are part of the permanent game-server stack on NOMAD and stay in place. + +### Safe to remove / how to restore +#### Safe to remove now +- `whisper` +- `node-exporter` + +Removing them should not be treated as a Project N.O.M.A.D. change or a Wings/Pelican change. They are standalone host-level containers from this audit’s point of view. + +#### Restore checklist if removal causes trouble +1. Confirm what broke: + - speech/transcription endpoint dependency → restore `whisper` + - Prometheus scraping / host metrics gap → restore `node-exporter` +2. Recreate the container using the documented `docker run` command in this file. +3. Verify it came back cleanly: + - `docker ps --filter name=whisper` + - `docker ps --filter name=node-exporter` + - `docker logs --tail 50 whisper` + - `docker logs --tail 50 node-exporter` +4. Re-check any dependent dashboards, scrapers, or automations. +5. If the recreated container needs to persist long-term again, document *why* before leaving it in place. + +#### Restore intent +- Restore `whisper` only if something still expects a local Faster Whisper HTTP endpoint on port `8786`. +- Restore `node-exporter` only if something still depends on host Prometheus metrics from this box. +- Do **not** fold either one into Project N.O.M.A.D. or Wings/Pelican casually; if they come back, they come back as deliberate standalone services unless John says otherwise. ## Pelican Panel (Game Server Management) - **URL:** `https://panel.paccoco.com` @@ -49,7 +234,16 @@ Offline survival/knowledge server and game server host. - **Docker network:** `pelican_nw` (172.20.0.0/16) ## Active Game Servers -- **Minecraft:** Purpur, Java 21, port 25565 TCP/UDP (direct host port forward) +- **Container `11d799cb-487a-416a-b6f9-9bf28f0e57e7`** + - Workload: Minecraft / Purpur-style Java server + - Java runtime image: `ghcr.io/pelican-eggs/yolks:java_25` + - Public port: `25565` TCP/UDP + - Data path: `/var/lib/pelican/volumes/11d799cb-487a-416a-b6f9-9bf28f0e57e7` +- **Container `d2aca31e-0d4d-433a-8295-cfad397dab0c`** + - Workload: modded Minecraft-style server (NeoForge-related env present) + - Java runtime image: `ghcr.io/pelican-eggs/yolks:java_21` + - Public port: `25566` TCP/UDP + - Data path: `/var/lib/pelican/volumes/d2aca31e-0d4d-433a-8295-cfad397dab0c` ## Networking Notes - `panel.paccoco.com` → `127.0.0.1` in `/etc/hosts` for NAT loopback