truenas-stacks/docs/servers/NOMAD.md

# N.O.M.A.D. Server

Offline survival/knowledge server and game server host.

## Specs
- **OS:** Ubuntu 25.10 (bare metal)
- **IP:** 10.5.30.7
- **CPU:** Intel Core i7-4770K @ 3.50GHz (4c/8t)
- **RAM:** 32GB DDR3 1600MHz
- **GPU:** NVIDIA GeForce GTX 1080 (8GB VRAM)
- **NIC:** Intel I218-V (1GbE) + Aquantia AQC113 (10GbE)
- **Motherboard:** ASRock H97M Pro4
- **UPS:** None

## Planned long-term role
- Keep NOMAD intentionally separate from PD's production stack.
- Continue to host Project NOMAD, game services, and the backup Technitium resolver.
- Accept only small bounded helper workloads; do not let it drift into being a second general-purpose production host.

## Storage
| Device | Size | Mount | Purpose |
|--------|------|-------|---------|
| Samsung SSD 840 (sdb) | 232.9GB | `/` (LVM 100GB) | OS boot |
| WD Red HDD (sda) | 3.6TB | `/mnt/hdd-1` | Primary data |
| WD Red HDD (sdc) | 3.6TB | `/mnt/hdd-2` | Docker data-root |
| Seagate SSD (sdd) | 223.6GB | `/mnt/ssd-1` | Fast scratch |

## Project N.O.M.A.D.
- Core compose project lives at `/opt/project-nomad` → symlinked from `/mnt/hdd-1/project-nomad`
- Main compose file: `/mnt/hdd-1/project-nomad/compose.yml`
- Docker data-root: `/mnt/hdd-2/docker`
- Based on Crosstalk Solutions offline survival/knowledge server
- Current runtime is split between:
  - the base `project-nomad` compose stack
  - `project-nomad-managed` app containers launched/managed by the N.O.M.A.D. admin layer
  - a few extra non-compose / non-NOMAD containers on the host

## Docker Audit — 2026-05-14

### Compose files found on NOMAD
- `/mnt/hdd-1/project-nomad/compose.yml` (same project as `/opt/project-nomad/compose.yml` via symlink)
- `/opt/newt/docker-compose.yaml`

### Running containers tied directly to compose files
**`project-nomad` compose (`/mnt/hdd-1/project-nomad/compose.yml`)**
- `nomad_admin`
- `nomad_mysql`
- `nomad_redis`
- `nomad_dozzle`
- `nomad_disk_collector`
- `nomad_updater`

**`newt` compose (`/opt/newt/docker-compose.yaml`)**
- `newt`

### Running containers managed by Project N.O.M.A.D. itself
These are labeled `com.docker.compose.project=project-nomad-managed` and `io.project-nomad.managed=true`, but no standalone compose files for them were found on disk during this audit.
- `nomad_cyberchef`
- `nomad_flatnotes`
- `nomad_kiwix_server`
- `nomad_kolibri`
- `nomad_ollama`
- `nomad_qdrant`

**Operator decision (John, 2026-05-14):** this layer stays exactly as-is. It is self-maintained by Project N.O.M.A.D. Document it, but **never manually touch, rebuild, migrate, or “clean up” these containers** unless John explicitly overrides that rule.

Known storage paths discovered from mounts / project storage:
- Flatnotes data: `/opt/project-nomad/storage/flatnotes`
- Kiwix ZIM library: `/opt/project-nomad/storage/zim`
- Kolibri data: `/opt/project-nomad/storage/kolibri`
- Ollama data/models: `/opt/project-nomad/storage/ollama`
- Qdrant storage: `/opt/project-nomad/storage/qdrant`

### Running containers not tied to discovered compose files
**Likely host-managed / manually launched / app-managed elsewhere**

#### `whisper` — `fedirz/faster-whisper-server:latest-cpu`
**Operator decision (John, 2026-05-14):** this can go away. Do not preserve it as a required service, but keep enough detail here to rebuild it if removing it causes issues.
- Purpose: ad-hoc Faster Whisper API endpoint
- Network mode: `bridge`
- Container IP during audit: `172.17.0.2`
- Port publish: host `8786/tcp` → container `8000/tcp` (IPv4 and IPv6)
- Restart policy: `unless-stopped`
- Entrypoint: image default
- Command:
  - `uv run uvicorn --factory faster_whisper_server.main:create_app`
- Environment:
  - `WHISPER__MODEL=Systran/faster-distil-whisper-small.en`
  - `WHISPER__INFERENCE_DEVICE=cpu`
  - `UVICORN_HOST=0.0.0.0`
  - `UVICORN_PORT=8000`
- Mounts: none
- Compose labels: none found
- Rebuild-equivalent `docker run`:
  ```bash
  docker run -d \
    --name whisper \
    --restart unless-stopped \
    -p 8786:8000 \
    -e WHISPER__MODEL=Systran/faster-distil-whisper-small.en \
    -e WHISPER__INFERENCE_DEVICE=cpu \
    -e UVICORN_HOST=0.0.0.0 \
    -e UVICORN_PORT=8000 \
    fedirz/faster-whisper-server:latest-cpu \
    uv run uvicorn --factory faster_whisper_server.main:create_app
  ```

#### `node-exporter` — `prom/node-exporter:v1.9.0`
**Operator decision (John, 2026-05-14):** this can go away. Do not preserve it as a required service, but keep enough detail here to rebuild it if removing it causes issues.
- Purpose: host Prometheus metrics exporter
- Network mode: `host`
- Restart policy: `unless-stopped`
- Entrypoint: `/bin/node_exporter`
- Command:
  - `--path.procfs=/host/proc`
  - `--path.sysfs=/host/sys`
  - `--path.rootfs=/rootfs`
  - `--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)`
- Read-only bind mounts:
  - `/proc` → `/host/proc`
  - `/sys` → `/host/sys`
  - `/` → `/rootfs`
- Compose labels: none found
- Rebuild-equivalent `docker run`:
  ```bash
  docker run -d \
    --name node-exporter \
    --restart unless-stopped \
    --network host \
    -v /proc:/host/proc:ro \
    -v /sys:/host/sys:ro \
    -v /:/rootfs:ro \
    prom/node-exporter:v1.9.0 \
    --path.procfs=/host/proc \
    --path.sysfs=/host/sys \
    --path.rootfs=/rootfs \
    '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
  ```

**Pelican/Wings-managed game containers**
**Operator decision (John, 2026-05-14):** this is the game server stack. It stays. Document it; do not remove or migrate it unless John explicitly says so.

- `11d799cb-487a-416a-b6f9-9bf28f0e57e7`
  - Image: `ghcr.io/pelican-eggs/yolks:java_25`
  - Network: `pelican_nw` (`172.20.0.2` during audit)
  - Ports: `25565/tcp` and `25565/udp` exposed on host `0.0.0.0`
  - Restart policy: `no` (Wings owns lifecycle)
  - Entrypoint: `/usr/bin/tini -g --`
  - Command: `/entrypoint.sh`
  - Important env:
    - `SERVER_IP=0.0.0.0`
    - `SERVER_PORT=25565`
    - `SERVER_JARFILE=server.jar`
    - `STARTUP=java --add-modules=jdk.incubator.vector -Xms128M -XX:MaxRAMPercentage=95.0 -Dterminal.jline=false -Dterminal.ansi=true -jar server.jar`
    - `P_SERVER_UUID=11d799cb-487a-416a-b6f9-9bf28f0e57e7`
  - Mounts:
    - `/var/lib/pelican/volumes/11d799cb-487a-416a-b6f9-9bf28f0e57e7` → `/home/container` (rw)
    - `/etc/pelican/passwd` → `/etc/passwd` (ro)
    - `/etc/pelican/group` → `/etc/group` (ro)
    - `/etc/pelican/machine-id/11d799cb-487a-416a-b6f9-9bf28f0e57e7` → `/etc/machine-id` (ro)

- `d2aca31e-0d4d-433a-8295-cfad397dab0c`
  - Image: `ghcr.io/pelican-eggs/yolks:java_21`
  - Network: `pelican_nw` (`172.20.0.3` during audit)
  - Ports: `25566/tcp` and `25566/udp` exposed on host `0.0.0.0`
  - Restart policy: `no` (Wings owns lifecycle)
  - Entrypoint: `/usr/bin/tini -g --`
  - Command: `/entrypoint.sh`
  - Important env:
    - `SERVER_IP=0.0.0.0`
    - `SERVER_PORT=25566`
    - `SERVER_MEMORY=8192`
    - `MC_VERSION=1.21.1`
    - `NEOFORGE_VERSION=21.1.229`
    - `STARTUP=java -Xms128M -XX:MaxRAMPercentage=95.0 -Dterminal.jline=false -Dterminal.ansi=true @unix_args.txt`
    - `P_SERVER_UUID=d2aca31e-0d4d-433a-8295-cfad397dab0c`
  - Mounts:
    - `/var/lib/pelican/volumes/d2aca31e-0d4d-433a-8295-cfad397dab0c` → `/home/container` (rw)
    - `/etc/pelican/passwd` → `/etc/passwd` (ro)
    - `/etc/pelican/group` → `/etc/group` (ro)
    - `/etc/pelican/machine-id/d2aca31e-0d4d-433a-8295-cfad397dab0c` → `/etc/machine-id` (ro)

These are expected to be managed by Wings/Pelican rather than a local compose file.

### Dormant compose files found but not currently represented in running containers
- None found during this audit in searched paths.
- Search scope: `/opt`, `/srv`, `/home/fizzlepoof`, and `/mnt` for `docker-compose.yml`, `docker-compose.yaml`, `compose.yml`, and `compose.yaml`.

### Decisions captured from this audit
- Project N.O.M.A.D.-managed app containers stay exactly as-is and are **hands-off** unless John explicitly says otherwise.
- `whisper` and `node-exporter` are not protected services; they can be removed, but their exact runtime has been documented above for rebuild/recovery.
- Pelican/Wings-managed game containers are part of the permanent game-server stack on NOMAD and stay in place.
- Doris Dashboard source belongs in the homelab repo, and the live NOMAD runtime belongs at `/opt/doris-dashboard`.

### Doris Dashboard (local NOMAD homepage)
- Live runtime path: `/opt/doris-dashboard`
- Repo source path: `/home/fizzlepoof/repos/truenas-stacks/home/doris-dashboard`
- Purpose: local-only dashboard for briefing/news/weather/homelab pulse/Paperless review on NOMAD
- Generator: `home/doris-dashboard/bin/generate_dashboard.py`
- Interactive LAN server: `home/doris-dashboard/bin/dashboard_server.py`
- Service file source: `home/doris-dashboard/doris-dashboard.service`
- Runtime notes:
  - Generated output and queue snapshots are runtime data; source/config lives in git, generated JSON/HTML does not need to be treated as canonical.
  - Topic thumbs write back to `doris-digest/data/feedback.json`.
  - Paperless acknowledge/dismiss state writes to `/opt/doris-dashboard/data/paperless_review_state.json` on the live host.

### Technitium Backup Resolver
- Live stack path: `/opt/technitium-nomad`
- Live secrets file: `/opt/technitium-nomad/.env`
- Resolver bind IP: `10.5.30.9`
- Role: active DHCP-advertised backup Technitium resolver for the homelab
- Config is refreshed from PD's live Technitium directory by the PD-side sync runner every 15 minutes
- The live `.env` is also backed up into the encrypted PD secrets repo as `env/technitium-nomad.env`
- Same-host checks from NOMAD to `10.5.30.9` are unreliable because the resolver sits behind macvlan networking; verify from another LAN peer instead

### LocalSend Trusted inbox
- Live stack path: `/opt/localsend-nomad`
- Repo source path: `/home/fizzlepoof/repos/truenas-stacks/localsend-nomad`
- Trusted bind IP: `10.5.1.16`
- Advertised device name: `Doris Trusted Inbox`
- Role: headless LocalSend receiver for Trusted-LAN handoff into `/home/fizzlepoof/private/inbox-secrets`
- Runtime model: standalone Docker Compose stack on a Trusted-LAN macvlan (`enp5s0.51`, VLAN 51) while the host stays on Servers (`10.5.30.7`)
- LocalSend listener ports: `53317/tcp`, `53317/udp`
- Same-host checks from NOMAD to `10.5.1.16` are unreliable because the receiver sits behind macvlan networking; verify from another Trusted-LAN peer instead
- Inbox watcher for Minerva staging: `minerva-localsend-autosort.path`
- Sorter service: `minerva-localsend-autosort.service`
- Sorter script: `/home/fizzlepoof/.local/bin/minerva-localsend-autosort.py`

### MeshCore companion observer
- Current runtime: `meshcore-capture.service` (systemd)
- Install root: `/home/fizzlepoof/.meshcore-packet-capture`
- Service unit: `/etc/systemd/system/meshcore-capture.service`
- Runtime command: `/home/fizzlepoof/.meshcore-packet-capture/venv/bin/python3 /home/fizzlepoof/.meshcore-packet-capture/packet_capture.py`
- Connection mode: direct serial on the Heltec via `/dev/serial/by-id/usb-Espressif_Systems_heltec_wifi_lora_32_v4__16_MB_FLASH__2_MB_PSRAM__441BF670B684-if00`
- Observer role:
  - NOMAD now uses the LetsMesh companion observer flow instead of the legacy `mctomqtt` relay
  - The companion process connects to the Heltec locally, signs JWTs on-device, and publishes packet capture data upstream to MQTT brokers
- Service health observed on 2026-07-03:
  - `systemctl status meshcore-capture` shows the service active/running
  - Logs confirm successful serial connection to device name `Peachtree Mesh Monitor`
  - Logs confirm MQTT connectivity to LetsMesh US, LetsMesh EU, and NashMe
  - After removing the temporary PD embedded-broker test target, startup logs now show `Connected to 3 MQTT broker(s)`
  - Packet capture mode is enabled and waiting for packets
- Initial-start caveat:
  - One installer-start attempt failed with `No such file or directory` for the `/dev/serial/by-id/...` path
  - A subsequent manual `systemctl start meshcore-capture` succeeded once the device path was present again
- Legacy relay retirement:
  - The old standalone Docker container `mctomqtt` was retired on 2026-06-13
  - Backup of the retired relay files/config was saved under `/home/fizzlepoof/private/backups/mctomqtt-retire/20260613-174536`
- Documentation linkage:
  - Related mesh stack notes live in [MESHTASTIC.md](../reference/MESHTASTIC.md)

### Live deployment path rule
- On NOMAD, live service roots belong under `/opt/<service>`.
- That includes host-run services **and** standalone Docker Compose deployments that are not part of the core Project N.O.M.A.D. stack.
- Keep service-local runtime data with the service under `/opt/<service>/data` unless there is a clear reason to place heavy state elsewhere.
- Examples: `/opt/newt`, `/opt/doris-dashboard`, `/opt/pihole-nomad`.
- Do not run production services from any agent workspace or ad-hoc checkout under `/home/fizzlepoof/.openclaw/workspace`.

### Safe to remove / how to restore
#### Safe to remove now
- `whisper`
- `node-exporter`

Removing them should not be treated as a Project N.O.M.A.D. change or a Wings/Pelican change. They are standalone host-level containers from this audit’s point of view.

#### Restore checklist if removal causes trouble
1. Confirm what broke:
   - speech/transcription endpoint dependency → restore `whisper`
   - Prometheus scraping / host metrics gap → restore `node-exporter`
2. Recreate the container using the documented `docker run` command in this file.
3. Verify it came back cleanly:
   - `docker ps --filter name=whisper`
   - `docker ps --filter name=node-exporter`
   - `docker logs --tail 50 whisper`
   - `docker logs --tail 50 node-exporter`
4. Re-check any dependent dashboards, scrapers, or automations.
5. If the recreated container needs to persist long-term again, document *why* before leaving it in place.

#### Restore intent
- Restore `whisper` only if something still expects a local Faster Whisper HTTP endpoint on port `8786`.
- Restore `node-exporter` only if something still depends on host Prometheus metrics from this box.
- Do **not** fold either one into Project N.O.M.A.D. or Wings/Pelican casually; if they come back, they come back as deliberate standalone services unless John says otherwise.

## Pelican Panel (Game Server Management)
- **URL:** `https://panel.paccoco.com`
- **Stack:** PHP 8.4 + nginx + MariaDB (native, not Docker)
- **Panel files:** `/var/www/pelican`
- **DB:** MariaDB, database `pelican`, user `pelican`@`127.0.0.1`
- **APP_URL:** `https://panel.paccoco.com`
- **TRUSTED_PROXIES:** `*`
- **Cron:** `* * * * * php /var/www/pelican/artisan schedule:run` (www-data)
- **Admin user:** `fizzlepoof`
- **Reverse proxy:** Pangolin via Newt container at `/opt/newt`

## Wings (Game Server Daemon)
- **Binary:** `/usr/local/bin/wings`
- **Config:** `/etc/pelican/config.yml`
- **Systemd:** `wings.service`
- **ExecStart:** `/usr/local/bin/wings --ignore-certificate-errors`
- **API port:** 8443
- **SFTP port:** 2022
- **Node domain:** `node1.paccoco.com` (Pangolin, no auth, connection port 443)
- **Game server volumes:** `/var/lib/pelican/volumes`
- **Docker network:** `pelican_nw` (172.20.0.0/16)

## Active Game Servers
- **Container `11d799cb-487a-416a-b6f9-9bf28f0e57e7`**
  - Workload: Minecraft / Purpur-style Java server
  - Java runtime image: `ghcr.io/pelican-eggs/yolks:java_25`
  - Public port: `25565` TCP/UDP
  - Data path: `/var/lib/pelican/volumes/11d799cb-487a-416a-b6f9-9bf28f0e57e7`
- **Container `d2aca31e-0d4d-433a-8295-cfad397dab0c`**
  - Workload: modded Minecraft-style server (NeoForge-related env present)
  - Java runtime image: `ghcr.io/pelican-eggs/yolks:java_21`
  - Public port: `25566` TCP/UDP
  - Data path: `/var/lib/pelican/volumes/d2aca31e-0d4d-433a-8295-cfad397dab0c`

## Networking Notes
- `panel.paccoco.com` → `127.0.0.1` in `/etc/hosts` for NAT loopback
- nginx on 80 and 443 (self-signed cert) for local Wings → Panel
- Wings uses `--ignore-certificate-errors` for self-signed cert
- For local operator-safe Pelican health checks, use `http://10.5.30.7:8080` instead of the self-signed TLS path; it redirects cleanly to `/home` without tripping certificate validation.
- Pangolin resource `node1.paccoco.com` has auth disabled

## Known Quirks
- NVIDIA container toolkit repo has invalid GPG key — remove `/etc/apt/sources.list.d/*nvidia*` if apt errors
- Port 8080 occupied by `nomad_admin` container — Wings uses 8443
- Interactive artisan commands break over SSH — always use inline flags
- Wings CORS derived from `remote:` URL — keep remote as `https://` and use `--ignore-certificate-errors`

## Newt Tunnel
- Container: `newt` at `/opt/newt/docker-compose.yaml`
- `network_mode: host`
- Connects to Pangolin VPS at `https://paccoco.com`

## Pending
- Regenerate Wings token (was exposed in chat)
- Regenerate Newt secret (was exposed in chat)