meshtastic: map usb meshcore companion into meshmonitor

DEPLOYMENT_HANDOFF.md

@@ -7,6 +7,7 @@ Phase 2 (AI stack) is fully deployed and verified on PlausibleDeniability (10.5.
 ## What's Done
 
 ### Phase 2 — AI Stack (COMPLETE)
+Historical note: this handoff reflects the original PD-first AI deployment snapshot, not the current Honcho routing policy.
 All 6 containers running and verified on PD (10.5.30.6):
 - **Ollama:** port 11434 — qwen2.5:14b loaded
 - **OpenWebUI:** port 8282 (→8080)

HOMELAB_BUILDOUT_PLAN.md

@@ -575,7 +575,7 @@
    575|Trigger:  Cron node (every Sunday at 10:00 AM)
    576|Step 1:   Postgres node → SELECT watch history for past 7 days
    577|Step 2:   Format data as structured text
-   578|Step 3:   HTTP Request → Ollama API (POST to PD's qwen2.5:14b)
+   578|Step 3:   HTTP Request → Ollama API (POST to the local/default automation model host, preferably N.O.M.A.D.-local rather than PD)
    579|            - Prompt: "Summarize this week's viewing in a fun digest: {data}"
    580|Step 4:   HTTP Request → Gotify (send digest)
    581|```
@@ -1093,9 +1093,9 @@
   1388|
   1389|LiteLLM sits in front of your three Ollama instances and presents a single OpenAI-compatible API at `http://litellm:4000`. It handles:
   1390|
-  1391|- **Model routing** — requests for `qwen3:32b` go to ROCINANTE, `qwen2.5:14b` goes to PD, `phi4` goes to N.O.M.A.D.
+  - **Model routing** — current reliable default is N.O.M.A.D.-local Ollama for background/automation paths, with PD kept available for shared light-tier use and Rocinante treated as optional/manual heavy capacity.
-  1392|- **Failover** — if ROCINANTE is offline, LiteLLM can fall back to PD automatically
+  - **Failover** — avoid designs that require Rocinante to be online; degrade to N.O.M.A.D.-local or explicitly operator-invoked PD paths instead of automatic dependence on a personal PC.
-  1393|- **Load balancing** — distribute requests across instances running the same model
+  - **Load balancing** — distribute requests across instances running the same model when you have multiple dependable backends, but do not assume Rocinante qualifies as one.
   1394|- **Usage tracking** — logs token counts, latency, and costs per model/user via its built-in database
   1395|
   1396|### 7.2 — Scaffold Directories
@@ -1621,17 +1621,23 @@
   1916|{
   1917|  "models": [
   1918|    {
-  1919|      "title": "PD - qwen2.5:14b",
+  1919|      "title": "N.O.M.A.D. - qwen2.5:3b",
   1920|      "provider": "ollama",
-  1921|      "model": "qwen2.5:14b",
+  1921|      "model": "qwen2.5:3b",
-  1922|      "apiBase": "http://PD_TAILSCALE_IP:11434"
+  1922|      "apiBase": "http://NOMAD_LOCAL_OR_TAILSCALE_IP:11434"
   1923|    },
   1924|    {
-  1925|      "title": "ROCINANTE - qwen3:32b",
+  1925|      "title": "PD - shared light tier (manual / fallback)",
   1926|      "provider": "ollama",
-  1927|      "model": "qwen3:32b",
+  1927|      "model": "qwen2.5:14b",
-  1928|      "apiBase": "http://ROCINANTE_TAILSCALE_IP:11434"
+  1928|      "apiBase": "http://PD_TAILSCALE_IP:11434"
-  1929|    }
+  1929|    },
+  1930|    {
+  1931|      "title": "ROCINANTE - opportunistic heavy",
+  1932|      "provider": "ollama",
+  1933|      "model": "qwen3:32b",
+  1934|      "apiBase": "http://ROCINANTE_TAILSCALE_IP:11434"
+  1935|    }
   1930|  ],
   1931|  "tabAutocompleteModel": {
   1932|    "title": "N.O.M.A.D. - phi4",

README.md

@@ -21,7 +21,7 @@ home/            # kitchenowl, donetick, shlink, qui, dakboard bridges, doris-da
 dev/             # gitea, rackpeek
 ai/              # ollama, openwebui
 headscale/       # pilot self-hosted tailnet control plane + Headplane UI on PD
-meshtastic/      # meshmonitor, mqtt-proxy, tileserver, meshtastic-map
+meshtastic/      # meshmonitor, tileserver, meshtastic-map, and MeshCore/MeshMonitor integration notes
 automation/      # planned: gotify, n8n
 search/          # planned: qdrant
 docs/            # all homelab documentation

docs/architecture/ARCHITECTURE_OVERVIEW.md

@@ -40,7 +40,8 @@ Current live homelab stack plus the planned direction for the upgraded PD platfo
 - **Dockhand** (3230) — Docker management UI
 
 ### AI Layer
-- **LiteLLM** (4000) — unified LLM proxy; routes to Ollama on PD (light) and Rocinante (heavy)
+- **LiteLLM** (4000) — unified LLM proxy; PD remains the shared light tier, while Rocinante is an opportunistic heavy tier when the PC is online
+- **Honcho** — memory/deriver stack on N.O.M.A.D. using local Ollama first so PD outages do not cascade into MeshMonitor/database incidents
 - **OpenWebUI** (8282) — chat UI with RAG, STT, and web search
 - **Qdrant** (6333/6334) — vector database for RAG
 - **Whisper** (8786 / 8787) — CPU speech-to-text on N.O.M.A.D. plus CUDA speech-to-text on Rocinante

docs/architecture/SERVICES_DIRECTORY.md

@@ -2,7 +2,7 @@
 
 All homelab services, their hosts, ports, and current status.
 
-*Last updated: 2026-05-27*
+*Last updated: 2026-06-13*
 
 ## Shared Databases (PD)
 
@@ -40,8 +40,8 @@ All homelab services, their hosts, ports, and current status.
 | Whisper (faster-whisper, CPU) | N.O.M.A.D. (10.5.30.7) | 8786 | ✅ Active |
 | Whisper (faster-whisper, CUDA large-v3) | Rocinante (10.5.30.112) | 8787 | ⚠️ Re-check from N.O.M.A.D. if Whisper-specific traffic is needed; Ollama-only recovery was validated 2026-05-27 |
 | SearXNG | PD | 8888 | ✅ Active |
-| Ollama — light tier | PD (10.5.30.6) | 11434 | ✅ Active |
+| Ollama — light tier | PD (10.5.30.6) | 11434 | ✅ Active; keep routine Honcho load off this host to protect MeshMonitor/shared Postgres |
-| Ollama — heavy tier | Rocinante (10.5.30.112) | 11434 | ✅ Reachable from N.O.M.A.D.; re-validated 2026-05-27 |
+| Ollama — heavy tier | Rocinante (10.5.30.112) | 11434 | ⚠️ Personal PC / opportunistic only; do not depend on it for routine background routing |
 | Reranker (TEI) | Serenity (10.5.30.5) | 9787 | ✅ Active |
 
 OpenClaw is no longer treated as an active PD service in the operator inventory. Only helper-artifact references remain under the school-intake planning docs.
@@ -100,7 +100,7 @@ OpenClaw is no longer treated as an active PD service in the operator inventory.
 
 | Service | Host | Port | Status |
 |---------|------|------|--------|
-| MeshCore to MQTT relay | N.O.M.A.D. (10.5.30.7) | outbound MQTT only | ✅ Running as Docker container `mctomqtt` (`mctomqtt:latest`); config bind-mounted from `/etc/mctomqtt` |
+| MeshCore companion observer | N.O.M.A.D. (10.5.30.7) | outbound MQTT only | ✅ Running as systemd service `meshcore-capture`; install root `/home/fizzlepoof/.meshcore-packet-capture`; direct serial connection to the Heltec companion node |
 
 ## Local file intake
 

docs/reference/MESHTASTIC.md

@@ -1,9 +1,14 @@
 # Meshtastic Stack
 
 ## Overview
-Self-hosted Meshtastic monitoring and management stack running on PlausibleDeniability.
+Self-hosted MeshMonitor stack running on PlausibleDeniability, plus the current operator note for the separate NOMAD-side MeshCore companion observer.
 
-Related NOMAD-side bridge note: a separate MeshCore-to-MQTT relay is running on N.O.M.A.D. as Docker container `mctomqtt` (`mctomqtt:latest`), with config bind-mounted from `/etc/mctomqtt`. The package also includes a systemd unit under `/opt/mctomqtt`, but that is not the active runtime on this host.
+Related NOMAD-side bridge note: the legacy `mctomqtt` Docker relay was retired on 2026-06-13. N.O.M.A.D. now uses the LetsMesh companion observer flow via `meshcore-capture.service`, running from `/home/fizzlepoof/.meshcore-packet-capture` and connecting directly to the Heltec over `/dev/serial/by-id/...`.
+
+Important split:
+- **PD** hosts the MeshMonitor web/UI stack.
+- **NOMAD** owns the USB-attached Heltec companion observer via `meshcore-capture.service`.
+- Do not assume MeshMonitor and the companion observer are the same runtime path.
 
 ## Stack Location
 `/mnt/docker-ssd/docker/compose/meshtastic/`
@@ -12,29 +17,43 @@ Related NOMAD-side bridge note: a separate MeshCore-to-MQTT relay is running on
 
 | Container | Image | Port | Purpose |
 |-----------|-------|------|---------|
-| meshmonitor | ghcr.io/yeraze/meshmonitor:4.2.0 | 8081→3001 | Main UI + backend |
+| meshmonitor | ghcr.io/yeraze/meshmonitor:latest | 8081→3001, 4404→4404, 1883→1883 | Main UI + backend; live compose publishes the virtual-node port and embedded MQTT broker directly from this container, and now maps `/dev/ttyACM0` into the container for native MeshCore USB sources |
-| meshmonitor (virtual node) | — | 4404 | TCP virtual node server |
+| meshmonitor (virtual node) | — | 4404 | TCP virtual node server exposed directly by the meshmonitor container |
-| meshmonitor-tileserver | maptiler/tileserver-gl-light | 8082→8080 | Offline map tiles |
+| meshmonitor (embedded MQTT broker) | — | 1883 | Embedded Aedes-based MQTT broker exposed directly by the meshmonitor container |
-| meshmonitor-mqtt-proxy | ghcr.io/ln4cy/mqtt-proxy:master | — | MQTT→TCP bridge |
+| meshmonitor-tileserver | maptiler/tileserver-gl-light:latest | 8082→8080 | Offline map tiles |
 | meshmonitor-upgrader | docker:latest | — | Auto-upgrade watchdog |
 
+Live runtime verified on 2026-07-03:
+- `docker compose ps` on PD showed `meshmonitor`, `meshmonitor-tileserver`, and `meshmonitor-upgrader` running.
+- Repo compose and live runtime now agree that there is no separate `meshmonitor-mqtt-proxy` sidecar in the active stack.
+- The embedded MQTT broker source `Meshcore Peachtree Observer` is configured in MeshMonitor and logs `MQTT broker listening on 0.0.0.0:1883`.
+- A native MeshCore USB source `PT Meshcore` is now auto-connected in MeshMonitor via `/dev/ttyACM0` after the compose file was updated to map the device into the container.
+- NOMAD `meshcore-capture.service` was later returned to its three upstream brokers after the embedded-broker test path proved unnecessary for real MeshCore ingestion.
+
 ## Architecture
 ```
-Meshtastic Node (10.5.1.172)
+Trusted-LAN mesh node (10.5.1.120)
-        ↓ MQTT
+        ↓ direct node connection configured by MESHTASTIC_NODE_IP
-meshmonitor-mqtt-proxy
+meshmonitor backend on PD
-        ↓ TCP (port 4404)
-meshmonitor virtual node server
-        ↓
-meshmonitor backend
         ↓
 PostgreSQL (shared-postgres, db: meshmonitor)
+
+Separate path:
+NOMAD Heltec companion observer (`meshcore-capture.service`)
+        ↓ upstream MQTT brokers (LetsMesh US / EU + NashMe)
+
+PD native MeshCore USB source (`PT Meshcore`)
+        ↓ `/dev/ttyACM0` mapped into the meshmonitor container
+        ↓ meshcore.js native companion backend
 ```
 
+Note: the embedded MeshMonitor broker now accepts the NOMAD connection on `10.5.30.6:1883`, but current MeshMonitor logs still show protobuf decode warnings for MeshCore packet payloads. Treat this as confirmed broker fanout connectivity, not yet confirmed first-class MeshCore ingestion.
+
 ## Node Connection
-- Node IP: `10.5.1.172`
+- PD MeshMonitor compose is currently configured with `MESHTASTIC_NODE_IP=10.5.1.120`
-- Connection type: MQTT via mqtt-proxy (not direct TCP)
+- MeshMonitor itself publishes port `4404` for the virtual-node path and `1883` for the embedded MQTT broker; there is no separate `meshmonitor-mqtt-proxy` service in the active stack
-- mqtt-proxy connects to meshmonitor's virtual node server on port 4404
+- For native MeshCore USB sources on PD, the live compose now maps `/dev/ttyACM0` directly into the `meshmonitor` container
+- NOMAD's companion observer is a different ingest path and should not be documented as a MeshMonitor sidecar
 
 ## Storage
 - Appdata: `/mnt/tank/docker/appdata/meshmonitor/data` → `/data`
@@ -60,7 +79,7 @@ mkdir -p /mnt/docker-ssd/docker/compose/meshtastic/scripts
 
 ## Known Warnings
 - Node `!dd972536` has a low-entropy public key — this is a device configuration issue, not a meshmonitor bug
-- Frequent connect/disconnect cycles in logs are normal when the MQTT proxy reconnects
+- MeshMonitor logs can still show noisy reconnect/admin-session churn during node or radio instability; do not confuse that with the retired NOMAD `mctomqtt` path
 
 ## Tiles
 Map tiles stored in `./tiles/`. Currently includes `zurich_switzerland.mbtiles` (34MB).
@@ -68,5 +87,4 @@ Tiles directory is gitignored. To add new tiles, download `.mbtiles` files and p
 
 ## TODO
 - [ ] Document all running meshtastic-map container config
-- [ ] Document meshtastic-map-mosquitto and meshtastic-mqtt containers
 - [ ] Add healthcheck to meshmonitor container

docs/servers/NOMAD.md

@@ -226,43 +226,27 @@ These are expected to be managed by Wings/Pelican rather than a local compose fi
 - Sorter service: `minerva-localsend-autosort.service`
 - Sorter script: `/home/fizzlepoof/.local/bin/minerva-localsend-autosort.py`
 
-### MeshCore to MQTT relay
+### MeshCore companion observer
-- Install date: 2026-05-18
+- Current runtime: `meshcore-capture.service` (systemd)
-- Install root: `/opt/mctomqtt`
+- Install root: `/home/fizzlepoof/.meshcore-packet-capture`
-- Config root: `/etc/mctomqtt`
+- Service unit: `/etc/systemd/system/meshcore-capture.service`
-- Service account: `mctomqtt:mctomqtt`
+- Runtime command: `/home/fizzlepoof/.meshcore-packet-capture/venv/bin/python3 /home/fizzlepoof/.meshcore-packet-capture/packet_capture.py`
-- Installer metadata: `.version_info` reports `installer_version: 1.3.0.0-preview`, repo `Cisien/meshcoretomqtt`, branch `main`, commit `10f7c0e`
+- Connection mode: direct serial on the Heltec via `/dev/serial/by-id/usb-Espressif_Systems_heltec_wifi_lora_32_v4__16_MB_FLASH__2_MB_PSRAM__441BF670B684-if00`
-- Detected install type marker: `/opt/mctomqtt/.install_type` contains `docker`
+- Observer role:
-- Runtime model:
+  - NOMAD now uses the LetsMesh companion observer flow instead of the legacy `mctomqtt` relay
-  - Active runtime is a standalone Docker container named `mctomqtt`
+  - The companion process connects to the Heltec locally, signs JWTs on-device, and publishes packet capture data upstream to MQTT brokers
-  - Image: `mctomqtt:latest`
+- Service health observed on 2026-07-03:
-  - Command: `python3 /opt/mctomqtt/mctomqtt.py`
+  - `systemctl status meshcore-capture` shows the service active/running
-  - User: `mctomqtt`
+  - Logs confirm successful serial connection to device name `Peachtree Mesh Monitor`
-  - Restart policy: `unless-stopped`
+  - Logs confirm MQTT connectivity to LetsMesh US, LetsMesh EU, and NashMe
-  - Network mode: `bridge`
+  - After removing the temporary PD embedded-broker test target, startup logs now show `Connected to 3 MQTT broker(s)`
-  - Config bind mount: `/etc/mctomqtt:/etc/mctomqtt:ro`
+  - Packet capture mode is enabled and waiting for packets
-  - Device passthrough: the Heltec serial device is mapped directly into the container at the same `/dev/serial/by-id/... ` path
+- Initial-start caveat:
-- Systemd note:
+  - One installer-start attempt failed with `No such file or directory` for the `/dev/serial/by-id/...` path
-  - The packaged repo also includes `/opt/mctomqtt/mctomqtt.service`, but it is not the active runtime on NOMAD
+  - A subsequent manual `systemctl start meshcore-capture` succeeded once the device path was present again
-  - No installed unit was present in `/etc/systemd/system` or `/lib/systemd/system`
+- Legacy relay retirement:
-  - The bundled unit expects `/opt/mctomqtt/venv/bin/python3`, but no `venv/` directory exists under `/opt/mctomqtt`
+  - The old standalone Docker container `mctomqtt` was retired on 2026-06-13
-  - Treat the systemd unit as an unused package artifact for this host unless the deployment model changes later
+  - Backup of the retired relay files/config was saved under `/home/fizzlepoof/private/backups/mctomqtt-retire/20260613-174536`
-- Config layering:
-  - Base reference: `/etc/mctomqtt/config.toml`
-  - Preset overlay: `/etc/mctomqtt/config.d/10-meshmapper.toml`
-  - Local overrides: `/etc/mctomqtt/config.d/99-user.toml`
-- Current local overrides observed:
-  - IATA region code set to `BNA`
-  - Serial device pinned to `/dev/serial/by-id/usb-Espressif_Systems_heltec_wifi_lora_32_v4__16_MB_FLASH__2_MB_PSRAM__441BF670B684-if00`
-  - Enabled broker targets include MeshMapper, LetsMesh US, and NashMe
-  - Credentials/secrets are stored inline in `99-user.toml`; do not mirror them into git-backed docs
-- Container health observed during inspection:
-  - Logs show the relay loading all three config layers, opening the Heltec serial device, and connecting successfully to MeshMapper, NashMe, and LetsMesh US after restart
-  - The current container had been up about 5 minutes during inspection with `MQTT: 3/3`, zero failures, and live RX air-time reported
-  - An earlier start logged one DNS resolution failure for a broker label shown as `custom-2`, but the current running container recovered and all three named brokers connected cleanly
-- Container logs / inspection:
-  - Use `docker logs mctomqtt` for runtime output
-  - Use `docker inspect mctomqtt` for bind mounts, device mappings, and restart policy
 - Documentation linkage:
   - Related mesh stack notes live in [MESHTASTIC.md](../reference/MESHTASTIC.md)
 

docs/servers/PLAUSIBLEDENABILITY.md

@@ -47,9 +47,10 @@ Recommended models for 11GB VRAM:
 | `phi4` | ~8GB | Analytical tasks |
 
 Current light-tier notes:
-- PD Ollama is the active light-tier endpoint for LiteLLM on `http://10.5.30.6:11434`
+- PD Ollama remains the shared light-tier endpoint for the broader AI stack on `http://10.5.30.6:11434`, but it should be protected from routine Honcho background load.
-- Local Honcho currently points at PD Ollama over LAN (`http://10.5.30.6:11434/v1`) instead of using a separate local model host
+- Local Honcho now uses N.O.M.A.D.'s own Ollama on `http://127.0.0.1:11434/v1` so PD pressure does not take down MeshMonitor or shared Postgres.
-- Current Honcho routing split keeps embeddings on `nomic-embed-text:latest`, cheap/default paths on `qwen2.5:7b`, and deeper high/max paths on `qwen2.5:14b`
+- Current Honcho routing keeps embeddings on `nomic-embed-text:v1.5`, cheap/default paths on `qwen2.5:3b`, and high/max paths on `llama3.1:latest`.
+- Rocinante is John's PC and should be treated as an opportunistic/manual heavy tier, not a dependable always-on backend.
 
 ## Remote Access
 - Pangolin VPS + Newt for external services

docs/servers/ROCINANTE.md

@@ -24,15 +24,15 @@ Ollama runs on port `11434` and hosts the large-model ("heavy") tier.
 | `qwen2.5:32b` | Large-context tasks |
 | `gemma3:27b` | Multimodal / vision |
 
-> **LiteLLM tier:** Rocinante is the **"heavy"** tier in LiteLLM `config.yaml`.
+> **LiteLLM tier:** Rocinante may serve as an **opportunistic heavy/manual tier** when the PC is online.
-> Heavy-tier requests from OpenWebUI and n8n route to `http://10.5.30.112:11434`.
+> Do not depend on `http://10.5.30.112:11434` for routine background routing; verify live reachability first and prefer N.O.M.A.D.-local paths for guaranteed availability.
 
 ## Current operational note
 
-- VLAN access to Rocinante from N.O.M.A.D. was re-validated on 2026-05-27.
+- Rocinante is John's personal PC and is not a dependable always-on homelab tier.
-- Rocinante Ollama now answers from N.O.M.A.D. at `http://10.5.30.112:11434`; `/api/tags`, `/api/generate`, and OpenAI-compatible `/v1/chat/completions` all succeeded.
+- Treat Ollama/Whisper on Rocinante as opportunistic capacity when the machine is online, not as a required background dependency.
-- Hermes configs that still pointed at the legacy `10.5.1.112` endpoint were corrected to `10.5.30.112`.
+- If N.O.M.A.D. or PD need guaranteed inference, prefer local N.O.M.A.D. routing first and keep PD protected from routine background load.
-- Re-check CUDA Whisper on `http://10.5.30.112:8787/health` separately if Whisper-specific reachability is needed; this pass validated Ollama end-to-end.
+- Re-check live reachability from N.O.M.A.D. before any Rocinante-dependent maintenance or cutover.
 
 ## Notes
 

meshtastic/docker-compose.yaml

@@ -5,10 +5,13 @@ services:
     ports:
       - "8081:3001"
       - "4404:4404"
+      - "1883:1883"
     restart: unless-stopped
     volumes:
       - /mnt/tank/docker/appdata/meshmonitor/data:/data
       - /mnt/docker-ssd/docker/compose/meshtastic/scripts:/data/scripts
+    devices:
+      - /dev/ttyACM0:/dev/ttyACM0
     env_file: .env
     environment:
       - NODE_ENV=production
@@ -68,28 +71,6 @@ services:
     networks:
       - default
       - pangolin
-  mqtt-proxy:
-    image: ghcr.io/ln4cy/mqtt-proxy:master
-    container_name: meshmonitor-mqtt-proxy
-    restart: unless-stopped
-    environment:
-      - INTERFACE_TYPE=tcp
-      - TCP_NODE_HOST=meshmonitor
-      - TCP_NODE_PORT=4404
-      - LOG_LEVEL=INFO
-      - TCP_TIMEOUT=300
-      - CONFIG_WAIT_TIMEOUT=60
-      - HEALTH_CHECK_ACTIVITY_TIMEOUT=300
-    depends_on:
-      - meshmonitor
-    healthcheck:
-      test: ["CMD-SHELL", "test -f /tmp/healthy && find /tmp/healthy -mmin -1 | grep -q healthy"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-      start_period: 60s
-    networks:
-      - default
 networks:
   ix-databases_shared-databases:
     external: true

n8n-workflows/README.md

@@ -171,10 +171,16 @@ curl -X PUT http://10.5.30.6:6333/collections/knowledge_base \
 
 (768 dimensions matches `nomic-embed-text`. If you use a different embedding model, adjust the size.)
 
-### Pull nomic-embed-text into Ollama (required for workflows 02, 05, 08)
+### Pull `nomic-embed-text` into the Ollama host backing these workflows (required for workflows 02, 05, 08)
 
+For N.O.M.A.D.-local workflows / Honcho-style routing:
 ```bash
-curl http://10.5.30.6:11434/api/pull -d '{"name": "nomic-embed-text"}'
+curl http://127.0.0.1:11434/api/pull -d '{"name": "nomic-embed-text:v1.5"}'
+```
+
+For PD-hosted shared workflows, use the PD endpoint explicitly instead:
+```bash
+curl http://10.5.30.6:11434/api/pull -d '{"name": "nomic-embed-text:latest"}'
 ```
 
 ### Paperless Webhook (required for workflows 03, 05, 17, and 19)