fizzlepoof/truenas-stacks
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
482a490d3d256510a929cd085cbb105319c85e4e
truenas-stacks/docs/servers/NOMAD.md
Fizzlepoof 482a490d3d
Some checks failed
secret-guardrails / artifact-secret-scan (push) Has been cancelled
Details
secret-guardrails / gitleaks (push) Has been cancelled
Details
meshtastic: map usb meshcore companion into meshmonitor
2026-07-03 03:10:07 +00:00

17 KiB
Raw Blame History

N.O.M.A.D. Server

Offline survival/knowledge server and game server host.

Specs

  • OS: Ubuntu 25.10 (bare metal)
  • IP: 10.5.30.7
  • CPU: Intel Core i7-4770K @ 3.50GHz (4c/8t)
  • RAM: 32GB DDR3 1600MHz
  • GPU: NVIDIA GeForce GTX 1080 (8GB VRAM)
  • NIC: Intel I218-V (1GbE) + Aquantia AQC113 (10GbE)
  • Motherboard: ASRock H97M Pro4
  • UPS: None

Planned long-term role

  • Keep NOMAD intentionally separate from PD's production stack.
  • Continue to host Project NOMAD, game services, and the backup Technitium resolver.
  • Accept only small bounded helper workloads; do not let it drift into being a second general-purpose production host.

Storage

Device Size Mount Purpose
Samsung SSD 840 (sdb) 232.9GB / (LVM 100GB) OS boot
WD Red HDD (sda) 3.6TB /mnt/hdd-1 Primary data
WD Red HDD (sdc) 3.6TB /mnt/hdd-2 Docker data-root
Seagate SSD (sdd) 223.6GB /mnt/ssd-1 Fast scratch

Project N.O.M.A.D.

  • Core compose project lives at /opt/project-nomad → symlinked from /mnt/hdd-1/project-nomad
  • Main compose file: /mnt/hdd-1/project-nomad/compose.yml
  • Docker data-root: /mnt/hdd-2/docker
  • Based on Crosstalk Solutions offline survival/knowledge server
  • Current runtime is split between:
    • the base project-nomad compose stack
    • project-nomad-managed app containers launched/managed by the N.O.M.A.D. admin layer
    • a few extra non-compose / non-NOMAD containers on the host

Docker Audit — 2026-05-14

Compose files found on NOMAD

  • /mnt/hdd-1/project-nomad/compose.yml (same project as /opt/project-nomad/compose.yml via symlink)
  • /opt/newt/docker-compose.yaml

Running containers tied directly to compose files

project-nomad compose (/mnt/hdd-1/project-nomad/compose.yml)

  • nomad_admin
  • nomad_mysql
  • nomad_redis
  • nomad_dozzle
  • nomad_disk_collector
  • nomad_updater

newt compose (/opt/newt/docker-compose.yaml)

  • newt

Running containers managed by Project N.O.M.A.D. itself

These are labeled com.docker.compose.project=project-nomad-managed and io.project-nomad.managed=true, but no standalone compose files for them were found on disk during this audit.

  • nomad_cyberchef
  • nomad_flatnotes
  • nomad_kiwix_server
  • nomad_kolibri
  • nomad_ollama
  • nomad_qdrant

Operator decision (John, 2026-05-14): this layer stays exactly as-is. It is self-maintained by Project N.O.M.A.D. Document it, but never manually touch, rebuild, migrate, or “clean up” these containers unless John explicitly overrides that rule.

Known storage paths discovered from mounts / project storage:

  • Flatnotes data: /opt/project-nomad/storage/flatnotes
  • Kiwix ZIM library: /opt/project-nomad/storage/zim
  • Kolibri data: /opt/project-nomad/storage/kolibri
  • Ollama data/models: /opt/project-nomad/storage/ollama
  • Qdrant storage: /opt/project-nomad/storage/qdrant

Running containers not tied to discovered compose files

Likely host-managed / manually launched / app-managed elsewhere

whisper — fedirz/faster-whisper-server:latest-cpu

Operator decision (John, 2026-05-14): this can go away. Do not preserve it as a required service, but keep enough detail here to rebuild it if removing it causes issues.

  • Purpose: ad-hoc Faster Whisper API endpoint
  • Network mode: bridge
  • Container IP during audit: 172.17.0.2
  • Port publish: host 8786/tcp → container 8000/tcp (IPv4 and IPv6)
  • Restart policy: unless-stopped
  • Entrypoint: image default
  • Command:
    • uv run uvicorn --factory faster_whisper_server.main:create_app
  • Environment:
    • WHISPER__MODEL=Systran/faster-distil-whisper-small.en
    • WHISPER__INFERENCE_DEVICE=cpu
    • UVICORN_HOST=0.0.0.0
    • UVICORN_PORT=8000
  • Mounts: none
  • Compose labels: none found
  • Rebuild-equivalent docker run: 
    docker run -d \
  --name whisper \
  --restart unless-stopped \
  -p 8786:8000 \
  -e WHISPER__MODEL=Systran/faster-distil-whisper-small.en \
  -e WHISPER__INFERENCE_DEVICE=cpu \
  -e UVICORN_HOST=0.0.0.0 \
  -e UVICORN_PORT=8000 \
  fedirz/faster-whisper-server:latest-cpu \
  uv run uvicorn --factory faster_whisper_server.main:create_app

node-exporter — prom/node-exporter:v1.9.0

Operator decision (John, 2026-05-14): this can go away. Do not preserve it as a required service, but keep enough detail here to rebuild it if removing it causes issues.

  • Purpose: host Prometheus metrics exporter
  • Network mode: host
  • Restart policy: unless-stopped
  • Entrypoint: /bin/node_exporter
  • Command:
    • --path.procfs=/host/proc
    • --path.sysfs=/host/sys
    • --path.rootfs=/rootfs
    • --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)
  • Read-only bind mounts:
    • /proc → /host/proc
    • /sys → /host/sys
    • / → /rootfs
  • Compose labels: none found
  • Rebuild-equivalent docker run: 
    docker run -d \
  --name node-exporter \
  --restart unless-stopped \
  --network host \
  -v /proc:/host/proc:ro \
  -v /sys:/host/sys:ro \
  -v /:/rootfs:ro \
  prom/node-exporter:v1.9.0 \
  --path.procfs=/host/proc \
  --path.sysfs=/host/sys \
  --path.rootfs=/rootfs \
  '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'

Pelican/Wings-managed game containers Operator decision (John, 2026-05-14): this is the game server stack. It stays. Document it; do not remove or migrate it unless John explicitly says so.

  • 11d799cb-487a-416a-b6f9-9bf28f0e57e7

    • Image: ghcr.io/pelican-eggs/yolks:java_25
    • Network: pelican_nw (172.20.0.2 during audit)
    • Ports: 25565/tcp and 25565/udp exposed on host 0.0.0.0
    • Restart policy: no (Wings owns lifecycle)
    • Entrypoint: /usr/bin/tini -g --
    • Command: /entrypoint.sh
    • Important env:
      • SERVER_IP=0.0.0.0
      • SERVER_PORT=25565
      • SERVER_JARFILE=server.jar
      • STARTUP=java --add-modules=jdk.incubator.vector -Xms128M -XX:MaxRAMPercentage=95.0 -Dterminal.jline=false -Dterminal.ansi=true -jar server.jar
      • P_SERVER_UUID=11d799cb-487a-416a-b6f9-9bf28f0e57e7
    • Mounts:
      • /var/lib/pelican/volumes/11d799cb-487a-416a-b6f9-9bf28f0e57e7 → /home/container (rw)
      • /etc/pelican/passwd → /etc/passwd (ro)
      • /etc/pelican/group → /etc/group (ro)
      • /etc/pelican/machine-id/11d799cb-487a-416a-b6f9-9bf28f0e57e7 → /etc/machine-id (ro)

  • d2aca31e-0d4d-433a-8295-cfad397dab0c

    • Image: ghcr.io/pelican-eggs/yolks:java_21
    • Network: pelican_nw (172.20.0.3 during audit)
    • Ports: 25566/tcp and 25566/udp exposed on host 0.0.0.0
    • Restart policy: no (Wings owns lifecycle)
    • Entrypoint: /usr/bin/tini -g --
    • Command: /entrypoint.sh
    • Important env:
      • SERVER_IP=0.0.0.0
      • SERVER_PORT=25566
      • SERVER_MEMORY=8192
      • MC_VERSION=1.21.1
      • NEOFORGE_VERSION=21.1.229
      • STARTUP=java -Xms128M -XX:MaxRAMPercentage=95.0 -Dterminal.jline=false -Dterminal.ansi=true @unix_args.txt
      • P_SERVER_UUID=d2aca31e-0d4d-433a-8295-cfad397dab0c
    • Mounts:
      • /var/lib/pelican/volumes/d2aca31e-0d4d-433a-8295-cfad397dab0c → /home/container (rw)
      • /etc/pelican/passwd → /etc/passwd (ro)
      • /etc/pelican/group → /etc/group (ro)
      • /etc/pelican/machine-id/d2aca31e-0d4d-433a-8295-cfad397dab0c → /etc/machine-id (ro)

These are expected to be managed by Wings/Pelican rather than a local compose file.

Dormant compose files found but not currently represented in running containers

  • None found during this audit in searched paths.
  • Search scope: /opt, /srv, /home/fizzlepoof, and /mnt for docker-compose.yml, docker-compose.yaml, compose.yml, and compose.yaml.

Decisions captured from this audit

  • Project N.O.M.A.D.-managed app containers stay exactly as-is and are hands-off unless John explicitly says otherwise.
  • whisper and node-exporter are not protected services; they can be removed, but their exact runtime has been documented above for rebuild/recovery.
  • Pelican/Wings-managed game containers are part of the permanent game-server stack on NOMAD and stay in place.
  • Doris Dashboard source belongs in the homelab repo, and the live NOMAD runtime belongs at /opt/doris-dashboard.

Doris Dashboard (local NOMAD homepage)

  • Live runtime path: /opt/doris-dashboard
  • Repo source path: /home/fizzlepoof/repos/truenas-stacks/home/doris-dashboard
  • Purpose: local-only dashboard for briefing/news/weather/homelab pulse/Paperless review on NOMAD
  • Generator: home/doris-dashboard/bin/generate_dashboard.py
  • Interactive LAN server: home/doris-dashboard/bin/dashboard_server.py
  • Service file source: home/doris-dashboard/doris-dashboard.service
  • Runtime notes:
    • Generated output and queue snapshots are runtime data; source/config lives in git, generated JSON/HTML does not need to be treated as canonical.
    • Topic thumbs write back to doris-digest/data/feedback.json.
    • Paperless acknowledge/dismiss state writes to /opt/doris-dashboard/data/paperless_review_state.json on the live host.

Technitium Backup Resolver

  • Live stack path: /opt/technitium-nomad
  • Live secrets file: /opt/technitium-nomad/.env
  • Resolver bind IP: 10.5.30.9
  • Role: active DHCP-advertised backup Technitium resolver for the homelab
  • Config is refreshed from PD's live Technitium directory by the PD-side sync runner every 15 minutes
  • The live .env is also backed up into the encrypted PD secrets repo as env/technitium-nomad.env
  • Same-host checks from NOMAD to 10.5.30.9 are unreliable because the resolver sits behind macvlan networking; verify from another LAN peer instead

LocalSend Trusted inbox

  • Live stack path: /opt/localsend-nomad
  • Repo source path: /home/fizzlepoof/repos/truenas-stacks/localsend-nomad
  • Trusted bind IP: 10.5.1.16
  • Advertised device name: Doris Trusted Inbox
  • Role: headless LocalSend receiver for Trusted-LAN handoff into /home/fizzlepoof/private/inbox-secrets
  • Runtime model: standalone Docker Compose stack on a Trusted-LAN macvlan (enp5s0.51, VLAN 51) while the host stays on Servers (10.5.30.7)
  • LocalSend listener ports: 53317/tcp, 53317/udp
  • Same-host checks from NOMAD to 10.5.1.16 are unreliable because the receiver sits behind macvlan networking; verify from another Trusted-LAN peer instead
  • Inbox watcher for Minerva staging: minerva-localsend-autosort.path
  • Sorter service: minerva-localsend-autosort.service
  • Sorter script: /home/fizzlepoof/.local/bin/minerva-localsend-autosort.py

MeshCore companion observer

  • Current runtime: meshcore-capture.service (systemd)
  • Install root: /home/fizzlepoof/.meshcore-packet-capture
  • Service unit: /etc/systemd/system/meshcore-capture.service
  • Runtime command: /home/fizzlepoof/.meshcore-packet-capture/venv/bin/python3 /home/fizzlepoof/.meshcore-packet-capture/packet_capture.py
  • Connection mode: direct serial on the Heltec via /dev/serial/by-id/usb-Espressif_Systems_heltec_wifi_lora_32_v4__16_MB_FLASH__2_MB_PSRAM__441BF670B684-if00
  • Observer role:
    • NOMAD now uses the LetsMesh companion observer flow instead of the legacy mctomqtt relay
    • The companion process connects to the Heltec locally, signs JWTs on-device, and publishes packet capture data upstream to MQTT brokers
  • Service health observed on 2026-07-03:
    • systemctl status meshcore-capture shows the service active/running
    • Logs confirm successful serial connection to device name Peachtree Mesh Monitor
    • Logs confirm MQTT connectivity to LetsMesh US, LetsMesh EU, and NashMe
    • After removing the temporary PD embedded-broker test target, startup logs now show Connected to 3 MQTT broker(s)
    • Packet capture mode is enabled and waiting for packets
  • Initial-start caveat:
    • One installer-start attempt failed with No such file or directory for the /dev/serial/by-id/... path
    • A subsequent manual systemctl start meshcore-capture succeeded once the device path was present again
  • Legacy relay retirement:
    • The old standalone Docker container mctomqtt was retired on 2026-06-13
    • Backup of the retired relay files/config was saved under /home/fizzlepoof/private/backups/mctomqtt-retire/20260613-174536
  • Documentation linkage:

Live deployment path rule

  • On NOMAD, live service roots belong under /opt/<service>.
  • That includes host-run services and standalone Docker Compose deployments that are not part of the core Project N.O.M.A.D. stack.
  • Keep service-local runtime data with the service under /opt/<service>/data unless there is a clear reason to place heavy state elsewhere.
  • Examples: /opt/newt, /opt/doris-dashboard, /opt/pihole-nomad.
  • Do not run production services from any agent workspace or ad-hoc checkout under /home/fizzlepoof/.openclaw/workspace.

Safe to remove / how to restore

Safe to remove now

  • whisper
  • node-exporter

Removing them should not be treated as a Project N.O.M.A.D. change or a Wings/Pelican change. They are standalone host-level containers from this audit’s point of view.

Restore checklist if removal causes trouble

  1. Confirm what broke:
    • speech/transcription endpoint dependency → restore whisper
    • Prometheus scraping / host metrics gap → restore node-exporter
  2. Recreate the container using the documented docker run command in this file.
  3. Verify it came back cleanly:
    • docker ps --filter name=whisper
    • docker ps --filter name=node-exporter
    • docker logs --tail 50 whisper
    • docker logs --tail 50 node-exporter
  4. Re-check any dependent dashboards, scrapers, or automations.
  5. If the recreated container needs to persist long-term again, document why before leaving it in place.

Restore intent

  • Restore whisper only if something still expects a local Faster Whisper HTTP endpoint on port 8786.
  • Restore node-exporter only if something still depends on host Prometheus metrics from this box.
  • Do not fold either one into Project N.O.M.A.D. or Wings/Pelican casually; if they come back, they come back as deliberate standalone services unless John says otherwise.

Pelican Panel (Game Server Management)

  • URL: https://panel.paccoco.com
  • Stack: PHP 8.4 + nginx + MariaDB (native, not Docker)
  • Panel files: /var/www/pelican
  • DB: MariaDB, database pelican, user pelican@127.0.0.1
  • APP_URL: https://panel.paccoco.com
  • TRUSTED_PROXIES: *
  • Cron: * * * * * php /var/www/pelican/artisan schedule:run (www-data)
  • Admin user: fizzlepoof
  • Reverse proxy: Pangolin via Newt container at /opt/newt

Wings (Game Server Daemon)

  • Binary: /usr/local/bin/wings
  • Config: /etc/pelican/config.yml
  • Systemd: wings.service
  • ExecStart: /usr/local/bin/wings --ignore-certificate-errors
  • API port: 8443
  • SFTP port: 2022
  • Node domain: node1.paccoco.com (Pangolin, no auth, connection port 443)
  • Game server volumes: /var/lib/pelican/volumes
  • Docker network: pelican_nw (172.20.0.0/16)

Active Game Servers

  • Container 11d799cb-487a-416a-b6f9-9bf28f0e57e7
    • Workload: Minecraft / Purpur-style Java server
    • Java runtime image: ghcr.io/pelican-eggs/yolks:java_25
    • Public port: 25565 TCP/UDP
    • Data path: /var/lib/pelican/volumes/11d799cb-487a-416a-b6f9-9bf28f0e57e7
  • Container d2aca31e-0d4d-433a-8295-cfad397dab0c
    • Workload: modded Minecraft-style server (NeoForge-related env present)
    • Java runtime image: ghcr.io/pelican-eggs/yolks:java_21
    • Public port: 25566 TCP/UDP
    • Data path: /var/lib/pelican/volumes/d2aca31e-0d4d-433a-8295-cfad397dab0c

Networking Notes

  • panel.paccoco.com → 127.0.0.1 in /etc/hosts for NAT loopback
  • nginx on 80 and 443 (self-signed cert) for local Wings → Panel
  • Wings uses --ignore-certificate-errors for self-signed cert
  • For local operator-safe Pelican health checks, use http://10.5.30.7:8080 instead of the self-signed TLS path; it redirects cleanly to /home without tripping certificate validation.
  • Pangolin resource node1.paccoco.com has auth disabled

Known Quirks

  • NVIDIA container toolkit repo has invalid GPG key — remove /etc/apt/sources.list.d/*nvidia* if apt errors
  • Port 8080 occupied by nomad_admin container — Wings uses 8443
  • Interactive artisan commands break over SSH — always use inline flags
  • Wings CORS derived from remote: URL — keep remote as https:// and use --ignore-certificate-errors

Newt Tunnel

  • Container: newt at /opt/newt/docker-compose.yaml
  • network_mode: host
  • Connects to Pangolin VPS at https://paccoco.com

Pending

  • Regenerate Wings token (was exposed in chat)
  • Regenerate Newt secret (was exposed in chat)
Reference in New Issue View Git Blame Copy Permalink