328 lines
16 KiB
Markdown
328 lines
16 KiB
Markdown
# N.O.M.A.D. Server
|
||
|
||
Offline survival/knowledge server and game server host.
|
||
|
||
## Specs
|
||
- **OS:** Ubuntu 25.10 (bare metal)
|
||
- **IP:** 10.5.1.16
|
||
- **CPU:** Intel Core i7-4770K @ 3.50GHz (4c/8t)
|
||
- **RAM:** 32GB DDR3 1600MHz
|
||
- **GPU:** NVIDIA GeForce GTX 1080 (8GB VRAM)
|
||
- **NIC:** Intel I218-V (1GbE) + Aquantia AQC113 (10GbE)
|
||
- **Motherboard:** ASRock H97M Pro4
|
||
- **UPS:** None
|
||
|
||
## Storage
|
||
| Device | Size | Mount | Purpose |
|
||
|--------|------|-------|---------|
|
||
| Samsung SSD 840 (sdb) | 232.9GB | `/` (LVM 100GB) | OS boot |
|
||
| WD Red HDD (sda) | 3.6TB | `/mnt/hdd-1` | Primary data |
|
||
| WD Red HDD (sdc) | 3.6TB | `/mnt/hdd-2` | Docker data-root |
|
||
| Seagate SSD (sdd) | 223.6GB | `/mnt/ssd-1` | Fast scratch |
|
||
|
||
## Project N.O.M.A.D.
|
||
- Core compose project lives at `/opt/project-nomad` → symlinked from `/mnt/hdd-1/project-nomad`
|
||
- Main compose file: `/mnt/hdd-1/project-nomad/compose.yml`
|
||
- Docker data-root: `/mnt/hdd-2/docker`
|
||
- Based on Crosstalk Solutions offline survival/knowledge server
|
||
- Current runtime is split between:
|
||
- the base `project-nomad` compose stack
|
||
- `project-nomad-managed` app containers launched/managed by the N.O.M.A.D. admin layer
|
||
- a few extra non-compose / non-NOMAD containers on the host
|
||
|
||
## Docker Audit — 2026-05-14
|
||
|
||
### Compose files found on NOMAD
|
||
- `/mnt/hdd-1/project-nomad/compose.yml` (same project as `/opt/project-nomad/compose.yml` via symlink)
|
||
- `/opt/newt/docker-compose.yaml`
|
||
|
||
### Running containers tied directly to compose files
|
||
**`project-nomad` compose (`/mnt/hdd-1/project-nomad/compose.yml`)**
|
||
- `nomad_admin`
|
||
- `nomad_mysql`
|
||
- `nomad_redis`
|
||
- `nomad_dozzle`
|
||
- `nomad_disk_collector`
|
||
- `nomad_updater`
|
||
|
||
**`newt` compose (`/opt/newt/docker-compose.yaml`)**
|
||
- `newt`
|
||
|
||
### Running containers managed by Project N.O.M.A.D. itself
|
||
These are labeled `com.docker.compose.project=project-nomad-managed` and `io.project-nomad.managed=true`, but no standalone compose files for them were found on disk during this audit.
|
||
- `nomad_cyberchef`
|
||
- `nomad_flatnotes`
|
||
- `nomad_kiwix_server`
|
||
- `nomad_kolibri`
|
||
- `nomad_ollama`
|
||
- `nomad_qdrant`
|
||
|
||
**Operator decision (John, 2026-05-14):** this layer stays exactly as-is. It is self-maintained by Project N.O.M.A.D. Document it, but **never manually touch, rebuild, migrate, or “clean up” these containers** unless John explicitly overrides that rule.
|
||
|
||
Known storage paths discovered from mounts / project storage:
|
||
- Flatnotes data: `/opt/project-nomad/storage/flatnotes`
|
||
- Kiwix ZIM library: `/opt/project-nomad/storage/zim`
|
||
- Kolibri data: `/opt/project-nomad/storage/kolibri`
|
||
- Ollama data/models: `/opt/project-nomad/storage/ollama`
|
||
- Qdrant storage: `/opt/project-nomad/storage/qdrant`
|
||
|
||
### Running containers not tied to discovered compose files
|
||
**Likely host-managed / manually launched / app-managed elsewhere**
|
||
|
||
#### `whisper` — `fedirz/faster-whisper-server:latest-cpu`
|
||
**Operator decision (John, 2026-05-14):** this can go away. Do not preserve it as a required service, but keep enough detail here to rebuild it if removing it causes issues.
|
||
- Purpose: ad-hoc Faster Whisper API endpoint
|
||
- Network mode: `bridge`
|
||
- Container IP during audit: `172.17.0.2`
|
||
- Port publish: host `8786/tcp` → container `8000/tcp` (IPv4 and IPv6)
|
||
- Restart policy: `unless-stopped`
|
||
- Entrypoint: image default
|
||
- Command:
|
||
- `uv run uvicorn --factory faster_whisper_server.main:create_app`
|
||
- Environment:
|
||
- `WHISPER__MODEL=Systran/faster-distil-whisper-small.en`
|
||
- `WHISPER__INFERENCE_DEVICE=cpu`
|
||
- `UVICORN_HOST=0.0.0.0`
|
||
- `UVICORN_PORT=8000`
|
||
- Mounts: none
|
||
- Compose labels: none found
|
||
- Rebuild-equivalent `docker run`:
|
||
```bash
|
||
docker run -d \
|
||
--name whisper \
|
||
--restart unless-stopped \
|
||
-p 8786:8000 \
|
||
-e WHISPER__MODEL=Systran/faster-distil-whisper-small.en \
|
||
-e WHISPER__INFERENCE_DEVICE=cpu \
|
||
-e UVICORN_HOST=0.0.0.0 \
|
||
-e UVICORN_PORT=8000 \
|
||
fedirz/faster-whisper-server:latest-cpu \
|
||
uv run uvicorn --factory faster_whisper_server.main:create_app
|
||
```
|
||
|
||
#### `node-exporter` — `prom/node-exporter:v1.9.0`
|
||
**Operator decision (John, 2026-05-14):** this can go away. Do not preserve it as a required service, but keep enough detail here to rebuild it if removing it causes issues.
|
||
- Purpose: host Prometheus metrics exporter
|
||
- Network mode: `host`
|
||
- Restart policy: `unless-stopped`
|
||
- Entrypoint: `/bin/node_exporter`
|
||
- Command:
|
||
- `--path.procfs=/host/proc`
|
||
- `--path.sysfs=/host/sys`
|
||
- `--path.rootfs=/rootfs`
|
||
- `--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)`
|
||
- Read-only bind mounts:
|
||
- `/proc` → `/host/proc`
|
||
- `/sys` → `/host/sys`
|
||
- `/` → `/rootfs`
|
||
- Compose labels: none found
|
||
- Rebuild-equivalent `docker run`:
|
||
```bash
|
||
docker run -d \
|
||
--name node-exporter \
|
||
--restart unless-stopped \
|
||
--network host \
|
||
-v /proc:/host/proc:ro \
|
||
-v /sys:/host/sys:ro \
|
||
-v /:/rootfs:ro \
|
||
prom/node-exporter:v1.9.0 \
|
||
--path.procfs=/host/proc \
|
||
--path.sysfs=/host/sys \
|
||
--path.rootfs=/rootfs \
|
||
'--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
|
||
```
|
||
|
||
**Pelican/Wings-managed game containers**
|
||
**Operator decision (John, 2026-05-14):** this is the game server stack. It stays. Document it; do not remove or migrate it unless John explicitly says so.
|
||
|
||
- `11d799cb-487a-416a-b6f9-9bf28f0e57e7`
|
||
- Image: `ghcr.io/pelican-eggs/yolks:java_25`
|
||
- Network: `pelican_nw` (`172.20.0.2` during audit)
|
||
- Ports: `25565/tcp` and `25565/udp` exposed on host `0.0.0.0`
|
||
- Restart policy: `no` (Wings owns lifecycle)
|
||
- Entrypoint: `/usr/bin/tini -g --`
|
||
- Command: `/entrypoint.sh`
|
||
- Important env:
|
||
- `SERVER_IP=0.0.0.0`
|
||
- `SERVER_PORT=25565`
|
||
- `SERVER_JARFILE=server.jar`
|
||
- `STARTUP=java --add-modules=jdk.incubator.vector -Xms128M -XX:MaxRAMPercentage=95.0 -Dterminal.jline=false -Dterminal.ansi=true -jar server.jar`
|
||
- `P_SERVER_UUID=11d799cb-487a-416a-b6f9-9bf28f0e57e7`
|
||
- Mounts:
|
||
- `/var/lib/pelican/volumes/11d799cb-487a-416a-b6f9-9bf28f0e57e7` → `/home/container` (rw)
|
||
- `/etc/pelican/passwd` → `/etc/passwd` (ro)
|
||
- `/etc/pelican/group` → `/etc/group` (ro)
|
||
- `/etc/pelican/machine-id/11d799cb-487a-416a-b6f9-9bf28f0e57e7` → `/etc/machine-id` (ro)
|
||
|
||
- `d2aca31e-0d4d-433a-8295-cfad397dab0c`
|
||
- Image: `ghcr.io/pelican-eggs/yolks:java_21`
|
||
- Network: `pelican_nw` (`172.20.0.3` during audit)
|
||
- Ports: `25566/tcp` and `25566/udp` exposed on host `0.0.0.0`
|
||
- Restart policy: `no` (Wings owns lifecycle)
|
||
- Entrypoint: `/usr/bin/tini -g --`
|
||
- Command: `/entrypoint.sh`
|
||
- Important env:
|
||
- `SERVER_IP=0.0.0.0`
|
||
- `SERVER_PORT=25566`
|
||
- `SERVER_MEMORY=8192`
|
||
- `MC_VERSION=1.21.1`
|
||
- `NEOFORGE_VERSION=21.1.229`
|
||
- `STARTUP=java -Xms128M -XX:MaxRAMPercentage=95.0 -Dterminal.jline=false -Dterminal.ansi=true @unix_args.txt`
|
||
- `P_SERVER_UUID=d2aca31e-0d4d-433a-8295-cfad397dab0c`
|
||
- Mounts:
|
||
- `/var/lib/pelican/volumes/d2aca31e-0d4d-433a-8295-cfad397dab0c` → `/home/container` (rw)
|
||
- `/etc/pelican/passwd` → `/etc/passwd` (ro)
|
||
- `/etc/pelican/group` → `/etc/group` (ro)
|
||
- `/etc/pelican/machine-id/d2aca31e-0d4d-433a-8295-cfad397dab0c` → `/etc/machine-id` (ro)
|
||
|
||
These are expected to be managed by Wings/Pelican rather than a local compose file.
|
||
|
||
### Dormant compose files found but not currently represented in running containers
|
||
- None found during this audit in searched paths.
|
||
- Search scope: `/opt`, `/srv`, `/home/fizzlepoof`, and `/mnt` for `docker-compose.yml`, `docker-compose.yaml`, `compose.yml`, and `compose.yaml`.
|
||
|
||
### Decisions captured from this audit
|
||
- Project N.O.M.A.D.-managed app containers stay exactly as-is and are **hands-off** unless John explicitly says otherwise.
|
||
- `whisper` and `node-exporter` are not protected services; they can be removed, but their exact runtime has been documented above for rebuild/recovery.
|
||
- Pelican/Wings-managed game containers are part of the permanent game-server stack on NOMAD and stay in place.
|
||
- Doris Dashboard source belongs in the homelab repo, and the live NOMAD runtime belongs at `/opt/doris-dashboard`.
|
||
|
||
### Doris Dashboard (local NOMAD homepage)
|
||
- Live runtime path: `/opt/doris-dashboard`
|
||
- Repo source path: `/home/fizzlepoof/repos/truenas-stacks/home/doris-dashboard`
|
||
- Purpose: local-only dashboard for briefing/news/weather/homelab pulse/Paperless review on NOMAD
|
||
- Generator: `home/doris-dashboard/bin/generate_dashboard.py`
|
||
- Interactive LAN server: `home/doris-dashboard/bin/dashboard_server.py`
|
||
- Service file source: `home/doris-dashboard/doris-dashboard.service`
|
||
- Runtime notes:
|
||
- Generated output and queue snapshots are runtime data; source/config lives in git, generated JSON/HTML does not need to be treated as canonical.
|
||
- Topic thumbs write back to `doris-digest/data/feedback.json`.
|
||
- Paperless acknowledge/dismiss state writes to `/opt/doris-dashboard/data/paperless_review_state.json` on the live host.
|
||
|
||
### MeshCore to MQTT relay
|
||
- Install date: 2026-05-18
|
||
- Install root: `/opt/mctomqtt`
|
||
- Config root: `/etc/mctomqtt`
|
||
- Service account: `mctomqtt:mctomqtt`
|
||
- Installer metadata: `.version_info` reports `installer_version: 1.3.0.0-preview`, repo `Cisien/meshcoretomqtt`, branch `main`, commit `10f7c0e`
|
||
- Detected install type marker: `/opt/mctomqtt/.install_type` contains `docker`
|
||
- Runtime model:
|
||
- Active runtime is a standalone Docker container named `mctomqtt`
|
||
- Image: `mctomqtt:latest`
|
||
- Command: `python3 /opt/mctomqtt/mctomqtt.py`
|
||
- User: `mctomqtt`
|
||
- Restart policy: `unless-stopped`
|
||
- Network mode: `bridge`
|
||
- Config bind mount: `/etc/mctomqtt:/etc/mctomqtt:ro`
|
||
- Device passthrough: the Heltec serial device is mapped directly into the container at the same `/dev/serial/by-id/... ` path
|
||
- Systemd note:
|
||
- The packaged repo also includes `/opt/mctomqtt/mctomqtt.service`, but it is not the active runtime on NOMAD
|
||
- No installed unit was present in `/etc/systemd/system` or `/lib/systemd/system`
|
||
- The bundled unit expects `/opt/mctomqtt/venv/bin/python3`, but no `venv/` directory exists under `/opt/mctomqtt`
|
||
- Treat the systemd unit as an unused package artifact for this host unless the deployment model changes later
|
||
- Config layering:
|
||
- Base reference: `/etc/mctomqtt/config.toml`
|
||
- Preset overlay: `/etc/mctomqtt/config.d/10-meshmapper.toml`
|
||
- Local overrides: `/etc/mctomqtt/config.d/99-user.toml`
|
||
- Current local overrides observed:
|
||
- IATA region code set to `BNA`
|
||
- Serial device pinned to `/dev/serial/by-id/usb-Espressif_Systems_heltec_wifi_lora_32_v4__16_MB_FLASH__2_MB_PSRAM__441BF670B684-if00`
|
||
- Enabled broker targets include MeshMapper, LetsMesh US, and NashMe
|
||
- Credentials/secrets are stored inline in `99-user.toml`; do not mirror them into git-backed docs
|
||
- Container health observed during inspection:
|
||
- Logs show the relay loading all three config layers, opening the Heltec serial device, and connecting successfully to MeshMapper, NashMe, and LetsMesh US after restart
|
||
- The current container had been up about 5 minutes during inspection with `MQTT: 3/3`, zero failures, and live RX air-time reported
|
||
- An earlier start logged one DNS resolution failure for a broker label shown as `custom-2`, but the current running container recovered and all three named brokers connected cleanly
|
||
- Container logs / inspection:
|
||
- Use `docker logs mctomqtt` for runtime output
|
||
- Use `docker inspect mctomqtt` for bind mounts, device mappings, and restart policy
|
||
- Documentation linkage:
|
||
- Related mesh stack notes live in [MESHTASTIC.md](../reference/MESHTASTIC.md)
|
||
|
||
### Live deployment path rule
|
||
- On NOMAD, live service roots belong under `/opt/<service>`.
|
||
- That includes host-run services **and** standalone Docker Compose deployments that are not part of the core Project N.O.M.A.D. stack.
|
||
- Keep service-local runtime data with the service under `/opt/<service>/data` unless there is a clear reason to place heavy state elsewhere.
|
||
- Examples: `/opt/newt`, `/opt/doris-dashboard`, `/opt/pihole-nomad`.
|
||
- Do not run production services from any agent workspace or ad-hoc checkout under `/home/fizzlepoof/.openclaw/workspace`.
|
||
|
||
### Safe to remove / how to restore
|
||
#### Safe to remove now
|
||
- `whisper`
|
||
- `node-exporter`
|
||
|
||
Removing them should not be treated as a Project N.O.M.A.D. change or a Wings/Pelican change. They are standalone host-level containers from this audit’s point of view.
|
||
|
||
#### Restore checklist if removal causes trouble
|
||
1. Confirm what broke:
|
||
- speech/transcription endpoint dependency → restore `whisper`
|
||
- Prometheus scraping / host metrics gap → restore `node-exporter`
|
||
2. Recreate the container using the documented `docker run` command in this file.
|
||
3. Verify it came back cleanly:
|
||
- `docker ps --filter name=whisper`
|
||
- `docker ps --filter name=node-exporter`
|
||
- `docker logs --tail 50 whisper`
|
||
- `docker logs --tail 50 node-exporter`
|
||
4. Re-check any dependent dashboards, scrapers, or automations.
|
||
5. If the recreated container needs to persist long-term again, document *why* before leaving it in place.
|
||
|
||
#### Restore intent
|
||
- Restore `whisper` only if something still expects a local Faster Whisper HTTP endpoint on port `8786`.
|
||
- Restore `node-exporter` only if something still depends on host Prometheus metrics from this box.
|
||
- Do **not** fold either one into Project N.O.M.A.D. or Wings/Pelican casually; if they come back, they come back as deliberate standalone services unless John says otherwise.
|
||
|
||
## Pelican Panel (Game Server Management)
|
||
- **URL:** `https://panel.paccoco.com`
|
||
- **Stack:** PHP 8.4 + nginx + MariaDB (native, not Docker)
|
||
- **Panel files:** `/var/www/pelican`
|
||
- **DB:** MariaDB, database `pelican`, user `pelican`@`127.0.0.1`
|
||
- **APP_URL:** `https://panel.paccoco.com`
|
||
- **TRUSTED_PROXIES:** `*`
|
||
- **Cron:** `* * * * * php /var/www/pelican/artisan schedule:run` (www-data)
|
||
- **Admin user:** `fizzlepoof`
|
||
- **Reverse proxy:** Pangolin via Newt container at `/opt/newt`
|
||
|
||
## Wings (Game Server Daemon)
|
||
- **Binary:** `/usr/local/bin/wings`
|
||
- **Config:** `/etc/pelican/config.yml`
|
||
- **Systemd:** `wings.service`
|
||
- **ExecStart:** `/usr/local/bin/wings --ignore-certificate-errors`
|
||
- **API port:** 8443
|
||
- **SFTP port:** 2022
|
||
- **Node domain:** `node1.paccoco.com` (Pangolin, no auth, connection port 443)
|
||
- **Game server volumes:** `/var/lib/pelican/volumes`
|
||
- **Docker network:** `pelican_nw` (172.20.0.0/16)
|
||
|
||
## Active Game Servers
|
||
- **Container `11d799cb-487a-416a-b6f9-9bf28f0e57e7`**
|
||
- Workload: Minecraft / Purpur-style Java server
|
||
- Java runtime image: `ghcr.io/pelican-eggs/yolks:java_25`
|
||
- Public port: `25565` TCP/UDP
|
||
- Data path: `/var/lib/pelican/volumes/11d799cb-487a-416a-b6f9-9bf28f0e57e7`
|
||
- **Container `d2aca31e-0d4d-433a-8295-cfad397dab0c`**
|
||
- Workload: modded Minecraft-style server (NeoForge-related env present)
|
||
- Java runtime image: `ghcr.io/pelican-eggs/yolks:java_21`
|
||
- Public port: `25566` TCP/UDP
|
||
- Data path: `/var/lib/pelican/volumes/d2aca31e-0d4d-433a-8295-cfad397dab0c`
|
||
|
||
## Networking Notes
|
||
- `panel.paccoco.com` → `127.0.0.1` in `/etc/hosts` for NAT loopback
|
||
- nginx on 80 and 443 (self-signed cert) for local Wings → Panel
|
||
- Wings uses `--ignore-certificate-errors` for self-signed cert
|
||
- Pangolin resource `node1.paccoco.com` has auth disabled
|
||
|
||
## Known Quirks
|
||
- NVIDIA container toolkit repo has invalid GPG key — remove `/etc/apt/sources.list.d/*nvidia*` if apt errors
|
||
- Port 8080 occupied by `nomad_admin` container — Wings uses 8443
|
||
- Interactive artisan commands break over SSH — always use inline flags
|
||
- Wings CORS derived from `remote:` URL — keep remote as `https://` and use `--ignore-certificate-errors`
|
||
|
||
## Newt Tunnel
|
||
- Container: `newt` at `/opt/newt/docker-compose.yaml`
|
||
- `network_mode: host`
|
||
- Connects to Pangolin VPS at `https://paccoco.com`
|
||
|
||
## Pending
|
||
- Regenerate Wings token (was exposed in chat)
|
||
- Regenerate Newt secret (was exposed in chat)
|