fizzlepoof/truenas-stacks
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: sync homelab ops, identity, and monitoring docs

Browse Source
This commit is contained in:
Fizzlepoof
2026-05-21 02:01:09 +00:00
parent 4c41b19e13
commit 2e99873634
44 changed files with 6295 additions and 346 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/architecture/ARCHITECTURE_OVERVIEW.md
View File

@@ -8,7 +8,7 @@ Full homelab stack as of 2026-05-09. All 6 expansion phases are complete and dep
|--------|----|----|------|
| **PlausibleDeniability (PD)** | TrueNAS Scale 25.10.2.1 | 10.5.1.6 | Primary Docker host — all compose stacks |
| **Serenity** | Unraid 7.2.4 | 10.5.1.5 | NAS, ARR stack, CPU reranker |
| **N.O.M.A.D.** | Ubuntu 25.10 | 10.5.1.16 | Light Ollama models, game servers |
| **N.O.M.A.D.** | Ubuntu 25.10 | 10.5.1.16 | Offline knowledge, game servers, and standalone local services |
| **Rocinante** | (bare metal) | 10.5.1.112 | Heavy Ollama models (RTX 4090) |
## Network
@@ -28,10 +28,10 @@ Full homelab stack as of 2026-05-09. All 6 expansion phases are complete and dep
- **Dockhand** (3230) — Docker management UI
### AI Layer
- **LiteLLM** (4000) — unified LLM proxy; routes to Ollama on N.O.M.A.D. (light) and Rocinante (heavy)
- **LiteLLM** (4000) — unified LLM proxy; routes to Ollama on PD (light) and Rocinante (heavy)
- **OpenWebUI** (8282) — chat UI with RAG, STT, and web search
- **Qdrant** (6333/6334) — vector database for RAG
- **Whisper** (8786) — CUDA speech-to-text on PD's RTX 2080 Ti
- **Whisper** (8786 / 8787) — CPU speech-to-text on N.O.M.A.D. plus CUDA speech-to-text on Rocinante
- **SearXNG** (8888) — private web search backend
- **Reranker** (Serenity:9787) — TEI bge-reranker-v2-m3, CPU-only
@@ -57,7 +57,7 @@ Full homelab stack as of 2026-05-09. All 6 expansion phases are complete and dep
## Key Architectural Decisions
1. **LiteLLM as unified proxy** — all AI clients target LiteLLM; model routing is transparent to clients
2. **Reranker on Serenity (CPU)** — frees RTX 2080 Ti on PD exclusively for Whisper CUDA; CPU latency acceptable for RAG reranking
2. **Reranker on Serenity (CPU)** — keeps reranking off the primary inference GPUs; CPU latency is acceptable for RAG reranking
3. **External Docker networks** — `ai-services` bridges all AI containers across compose files; `ix-databases_shared-databases` provides shared DB access
4. **NFS for Immich** — photo library lives on Serenity's large array; PD mounts on demand (not at boot)
5. **TrueNAS manages Docker** — never use `systemctl restart docker` on PD; use `docker compose` or `docker restart` for individual containers

14
docs/architecture/SERVICES_DIRECTORY.md
View File

@@ -2,7 +2,7 @@
All homelab services, their hosts, ports, and current status.
*Last updated: 2026-05-10*
*Last updated: 2026-05-20*
## Shared Databases (PD)
@@ -24,6 +24,7 @@ All homelab services, their hosts, ports, and current status.
| Shlink | 8087 | http://pd:8087 | ✅ Active |
| Shlink Web Client | 8088 | http://pd:8088 | ✅ Active |
| RackPeek | 8283 | http://pd:8283 | ✅ Active |
| Authelia | 9091 | http://pd:9091 / https://auth.paccoco.com | ✅ Active internally on PD; public Pangolin route still pending |
## AI Stack
@@ -36,7 +37,7 @@ All homelab services, their hosts, ports, and current status.
| Whisper (faster-whisper, CUDA large-v3) | Rocinante (10.5.1.112) | 8787 | ✅ Active |
| SearXNG | PD | 8888 | ✅ Active |
| OpenClaw | PD | 18789 | ✅ Active |
| Ollama — light tier | N.O.M.A.D. (10.5.1.16) | 11434 | ✅ Active |
| Ollama — light tier | PD (10.5.1.6) | 11434 | ✅ Active |
| Ollama — heavy tier | Rocinante (10.5.1.112) | 11434 | ✅ Active |
| Reranker (TEI) | Serenity (10.5.1.5) | 9787 | ✅ Active |
@@ -76,8 +77,17 @@ All homelab services, their hosts, ports, and current status.
|---------|------|--------|
| Grafana | 3000 | ✅ Active |
| Prometheus | 9090 | ✅ Active |
| Loki | internal | ✅ Active |
| Promtail | internal | ✅ Active |
| cAdvisor | internal | ✅ Active |
| MeshMonitor | 8081 | ✅ Active |
## Radio / Mesh
| Service | Host | Port | Status |
|---------|------|------|--------|
| MeshCore to MQTT relay | N.O.M.A.D. (10.5.1.16) | outbound MQTT only | ✅ Running as Docker container `mctomqtt` (`mctomqtt:latest`); config bind-mounted from `/etc/mctomqtt` |
## Mapping / Geo (PD)
| Service | Port | Status |

4
docs/operations/BACKUP_POLICY.md
View File

@@ -45,6 +45,10 @@ Recommended live deployment model on PD: a plain **root cron** job invoking `/us
- Test restores quarterly
- Test at least one app DB restore and one full-config restore
- Verify that restored stacks can start with current compose files
- Repo-staged verification runner: automation/bin/run_pd_restore_verification.sh
- Current default verification targets:
- latest dumps from POSTGRES_DB_LIST
- Serenity-backed tank-docker/appdata/grafana sample restore with required files present
## Operational Notes
- Media libraries on Unraid are large and may not need the same backup frequency as configs/databases

64
docs/operations/GITEA_SSO_PREP.md Normal file
View File

@@ -0,0 +1,64 @@
# Gitea SSO Prep
This is the staged cutover plan for moving the live PD Gitea instance to Authelia-backed sign-in without flipping it live yet.
## Current live shape
- stack path: `dev/`
- public URL: `https://gitea.paccoco.com`
- internal HTTP: `http://gitea:3000` on the `pangolin` network
- shared Postgres backend already in use
- Gitea version observed live on PD: `1.25.3`
## Safe hardening before SSO
- disable public self-registration
- keep normal local admin login working until Authelia sign-in is proven
- do not change `ROOT_URL` or SSH settings during the auth cutover
## Authelia side
Add a dedicated OIDC client for Gitea in the live Authelia `configuration.yml` when ready to test.
Recommended values:
- `client_id`: `gitea`
- `client_name`: `Gitea`
- `authorization_policy`: `one_factor`
- `scopes`: `openid`, `profile`, `email`, `groups`
- `grant_types`: `authorization_code`
- `response_types`: `code`
- `token_endpoint_auth_method`: `client_secret_basic`
Redirect URI guidance:
- use the callback URL shown by the Gitea authentication-source form
- for standard Gitea OIDC sources this is typically under `/user/oauth2/<source-name>/callback`
- validate the exact callback path against the live Gitea UI before committing the Authelia client entry
## Gitea side
In Gitea:
1. Sign in as a local admin.
2. Go to `Site Administration` -> `Authentication Sources`.
3. Add a new source of type `OpenID Connect`.
4. Use Authelia discovery if the form supports it, otherwise enter the manual endpoints from `https://auth.paccoco.com/.well-known/openid-configuration`.
Recommended source values:
- name: `Authelia`
- scopes: `openid profile email groups`
- required claim mapping priority: email first, then username/login
- keep the source enabled only after confirming the callback URL matches the Authelia client entry
## Rollout notes
- keep local admin auth available for the first pass
- test with one admin account first
- after successful sign-in, decide whether to keep local auth as break-glass only
- do not disable Pangolin auth in front of Gitea until Authelia login is actually working end-to-end
## Related weak spot
The live Gitea app.ini observed on PD had public self-registration enabled. That should stay off for this homelab unless there is a deliberate reason to reopen it.

60
docs/operations/IDENTITY_AND_SSO.md Normal file
View File

@@ -0,0 +1,60 @@
# Identity and SSO
Authelia is the staged identity provider for the homelab.
## Why Authelia
- centralizes auth for the growing pile of web apps
- works with shared Postgres/Redis already in the lab
- supports OIDC for apps that speak it properly
- avoids inventing separate local auth for every exposed service
## Current scope
- stack path: identity/
- host target: PlausibleDeniability
- public entrypoint: auth.paccoco.com
- status: repo-staged, not assumed live until deployed on PD
## Initial bootstrap flow
1. Copy identity/.env.example to identity/.env and generate fresh secrets.
2. Copy identity/authelia/configuration.yml.example to identity/authelia/configuration.yml.
3. Render concrete secrets and passwords into identity/authelia/configuration.yml on PD; do not rely on runtime env interpolation inside the YAML.
4. Copy identity/authelia/users_database.example.yml to identity/authelia/users_database.yml.
5. Generate an Argon2 password hash with the Authelia image and replace the sample hash.
6. Create the authelia Postgres database and user in the shared database stack.
7. Set `AUTHELIA_SESSION_REDIS_PASSWORD` from the live shared Redis stack on PD.
8. Add a Pangolin resource for auth.paccoco.com pointing at container authelia:9091.
9. Validate with docker compose --env-file .env config and docker exec authelia authelia config validate --config /config/configuration.yml.
## Rollout order
This repo stage is the control-plane bootstrap, not the full client cutover.
After Authelia is healthy on PD, wire OIDC or forward-auth clients one at a time.
Recommended first targets:
- Grafana
- Gitea
- any new public app before it gains yet another standalone login
## Grafana first-cutover notes
- Use Authelia OIDC as a Generic OAuth provider in Grafana.
- Keep Grafana's local admin login enabled for the first rollout.
- After OIDC is verified, disable Pangolin auth in front of Grafana to avoid double-auth.
## Gitea prep notes
- The live Gitea stack is the `dev/` compose on PD.
- Disable public self-registration before or during the SSO prep pass.
- Keep local admin login working for the first OIDC test.
- Stage the exact Gitea auth-source values in `docs/operations/GITEA_SSO_PREP.md` before creating the live source.
## Notes
- File-backed users are intentional for the initial rollout; this is enough for the current household scale.
- If identity sprawl grows later, migrate the authentication backend to LDAP instead of bolting extra auth stores onto apps ad hoc.
- OIDC signing keys and per-app client secrets should be generated during deployment on PD, not committed to the repo.

22
docs/operations/PD_BACKUP_DEPLOYMENT.md
View File

@@ -68,6 +68,12 @@ Preferred: install in **root's crontab** on PD.
# END PD BACKUPS
```
Quarterly restore verification:
```cron
30 3 1 */3 * cd /mnt/docker-ssd/docker/compose/automation && /usr/bin/bash bin/run_pd_restore_verification.sh >> /mnt/tank/docker/backups/pd-restore-verify.log 2>&1
```
## Post-deploy checks
```bash
@@ -76,6 +82,22 @@ ls -lh /mnt/tank/docker/backups/db-dumps
ssh root@10.5.1.5 'find /mnt/user/backups/plausible-deniability -maxdepth 2 -type d | sort | head -40'
```
## Restore verification
After the backup path is stable, run the staged restore verifier from PD:
```bash
cd /mnt/docker-ssd/docker/compose/automation
/usr/bin/bash bin/run_pd_restore_verification.sh
```
What it proves:
- the latest configured Postgres dumps can restore into a temporary container
- a real appdata/config sample can be staged back from Serenity
- expected files still exist inside the restored sample
- the latest run state can be exported to Prometheus via node-exporter's textfile collector
## Important limitations
- Do **not** use `systemctl restart docker` on PD.

2
docs/reference/MESHTASTIC.md
View File

@@ -3,6 +3,8 @@
## Overview
Self-hosted Meshtastic monitoring and management stack running on PlausibleDeniability.
Related NOMAD-side bridge note: a separate MeshCore-to-MQTT relay is running on N.O.M.A.D. as Docker container `mctomqtt` (`mctomqtt:latest`), with config bind-mounted from `/etc/mctomqtt`. The package also includes a systemd unit under `/opt/mctomqtt`, but that is not the active runtime on this host.
## Stack Location
`/mnt/docker-ssd/docker/compose/meshtastic/`

40
docs/servers/NOMAD.md
View File

@@ -199,6 +199,46 @@ These are expected to be managed by Wings/Pelican rather than a local compose fi
- Topic thumbs write back to `doris-digest/data/feedback.json`.
- Paperless acknowledge/dismiss state writes to `/opt/doris-dashboard/data/paperless_review_state.json` on the live host.
### MeshCore to MQTT relay
- Install date: 2026-05-18
- Install root: `/opt/mctomqtt`
- Config root: `/etc/mctomqtt`
- Service account: `mctomqtt:mctomqtt`
- Installer metadata: `.version_info` reports `installer_version: 1.3.0.0-preview`, repo `Cisien/meshcoretomqtt`, branch `main`, commit `10f7c0e`
- Detected install type marker: `/opt/mctomqtt/.install_type` contains `docker`
- Runtime model:
- Active runtime is a standalone Docker container named `mctomqtt`
- Image: `mctomqtt:latest`
- Command: `python3 /opt/mctomqtt/mctomqtt.py`
- User: `mctomqtt`
- Restart policy: `unless-stopped`
- Network mode: `bridge`
- Config bind mount: `/etc/mctomqtt:/etc/mctomqtt:ro`
- Device passthrough: the Heltec serial device is mapped directly into the container at the same `/dev/serial/by-id/... ` path
- Systemd note:
- The packaged repo also includes `/opt/mctomqtt/mctomqtt.service`, but it is not the active runtime on NOMAD
- No installed unit was present in `/etc/systemd/system` or `/lib/systemd/system`
- The bundled unit expects `/opt/mctomqtt/venv/bin/python3`, but no `venv/` directory exists under `/opt/mctomqtt`
- Treat the systemd unit as an unused package artifact for this host unless the deployment model changes later
- Config layering:
- Base reference: `/etc/mctomqtt/config.toml`
- Preset overlay: `/etc/mctomqtt/config.d/10-meshmapper.toml`
- Local overrides: `/etc/mctomqtt/config.d/99-user.toml`
- Current local overrides observed:
- IATA region code set to `BNA`
- Serial device pinned to `/dev/serial/by-id/usb-Espressif_Systems_heltec_wifi_lora_32_v4__16_MB_FLASH__2_MB_PSRAM__441BF670B684-if00`
- Enabled broker targets include MeshMapper, LetsMesh US, and NashMe
- Credentials/secrets are stored inline in `99-user.toml`; do not mirror them into git-backed docs
- Container health observed during inspection:
- Logs show the relay loading all three config layers, opening the Heltec serial device, and connecting successfully to MeshMapper, NashMe, and LetsMesh US after restart
- The current container had been up about 5 minutes during inspection with `MQTT: 3/3`, zero failures, and live RX air-time reported
- An earlier start logged one DNS resolution failure for a broker label shown as `custom-2`, but the current running container recovered and all three named brokers connected cleanly
- Container logs / inspection:
- Use `docker logs mctomqtt` for runtime output
- Use `docker inspect mctomqtt` for bind mounts, device mappings, and restart policy
- Documentation linkage:
- Related mesh stack notes live in [MESHTASTIC.md](../reference/MESHTASTIC.md)
### Live deployment path rule
- On NOMAD, live service roots belong under `/opt/<service>`.
- That includes host-run services **and** standalone Docker Compose deployments that are not part of the core Project N.O.M.A.D. stack.

10
docs/servers/PLAUSIBLEDENABILITY.md
View File

@@ -31,18 +31,24 @@ Primary Docker host running all production workloads.
RTX 2080 Ti used by:
- Plex (hardware transcoding)
- immich-ml (CUDA face recognition)
- Ollama light-tier inference (qwen2.5:14b and smaller)
- Ollama light-tier inference for the main AI stack and local Honcho backend
> **Note:** Whisper moved to N.O.M.A.D. (10.5.1.16:8786) as CPU inference — avoids VRAM conflict with Ollama. See [NOMAD.md](NOMAD.md).
Recommended models for 11GB VRAM:
| Model | VRAM | Best for |
|-------|------|---------|
| `qwen2.5:14b` | ~8GB | General use, coding, reasoning |
| `qwen2.5:7b` | ~5GB | Cheap/default reasoning, summaries, derivation |
| `qwen2.5:14b` | ~8GB | Heavier reasoning / high-max paths |
| `qwen2.5-coder:7b` | ~5GB | Code tasks |
| `deepseek-r1:8b` | ~5GB | Math, logic |
| `phi4` | ~8GB | Analytical tasks |
Current light-tier notes:
- PD Ollama is the active light-tier endpoint for LiteLLM on `http://10.5.1.6:11434`
- Local Honcho currently points at PD Ollama over LAN (`http://10.5.1.6:11434/v1`) instead of using a separate local model host
- Current Honcho routing split keeps embeddings on `nomic-embed-text:latest`, cheap/default paths on `qwen2.5:7b`, and deeper high/max paths on `qwen2.5:14b`
## Remote Access
- Pangolin VPS + Newt for external services
- SSH from Windows 11