fizzlepoof/truenas-stacks
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: sync homelab ops, identity, and monitoring docs

Browse Source
This commit is contained in:
Fizzlepoof
2026-05-21 02:01:09 +00:00
parent 4c41b19e13
commit 2e99873634
44 changed files with 6295 additions and 346 deletions
Download Patch File Download Diff File Expand all files Collapse all files

151
monitoring/DEPLOY.md
View File

@@ -1,118 +1,105 @@
# Phase 6 — Grafana + Prometheus Deployment on PD
# Monitoring Deployment on PD
> Adapted from HOMELAB_BUILDOUT_PLAN.md Phase 6, changed host from N.O.M.A.D. to PlausibleDeniability.
This stack now includes:
## 1. Scaffold Directories
- Prometheus
- Grafana
- Loki
- Promtail
- node-exporter
- cAdvisor
SSH into PD (or run locally):
Source-of-truth path:
- /mnt/docker-ssd/docker/compose/monitoring
## 1. Stage repo files on PD
Make sure the monitoring directory from this repo is present on PD at:
- /mnt/docker-ssd/docker/compose/monitoring
## 2. Prepare runtime directories
Create the required appdata paths on PD:
```bash
# Stack directory
sudo mkdir -p /opt/monitoring/provisioning/datasources
sudo mkdir -p /opt/monitoring/provisioning/dashboards
mkdir -p /mnt/tank/docker/appdata/prometheus
mkdir -p /mnt/tank/docker/appdata/grafana/dashboards
mkdir -p /mnt/tank/docker/appdata/loki
mkdir -p /mnt/tank/docker/appdata/promtail
# Prometheus TSDB on tank (sequential writes, grows with retention)
sudo mkdir -p /mnt/tank/docker/appdata/prometheus
sudo chown 65534:65534 /mnt/tank/docker/appdata/prometheus
# Grafana data on tank
sudo mkdir -p /mnt/tank/docker/appdata/grafana
sudo chown 472:472 /mnt/tank/docker/appdata/grafana
chown 65534:65534 /mnt/tank/docker/appdata/prometheus
chown 472:472 /mnt/tank/docker/appdata/grafana
chown 472:472 /mnt/tank/docker/appdata/grafana/dashboards
```
## 2. Copy Files to PD
Copy everything from this `monitoring/` folder to `/opt/monitoring/` on PD:
## 3. Create the live env file
```bash
# From your local machine (adjust source path as needed)
scp -r monitoring/* pd:/opt/monitoring/
# Or if working directly on PD, copy files into place:
# docker-compose.yaml → /opt/monitoring/docker-compose.yaml
# prometheus.yml → /opt/monitoring/prometheus.yml
# provisioning/ → /opt/monitoring/provisioning/
# .env.example → /opt/monitoring/.env.example
```
## 3. Create .env
```bash
cd /opt/monitoring
cd /mnt/docker-ssd/docker/compose/monitoring
cp .env.example .env
nano .env
# Set GF_ADMIN_PASS=$(openssl rand -hex 16)
chmod 600 .env
```
## 4. Pangolin Configuration
Set a real Grafana password before deploy.
Set a real Grafana OIDC client secret before enabling Authelia login.
In Pangolin dashboard, create a new resource using PD's existing Newt:
## 4. Pangolin exposure
| Field | Value |
|--------|--------------------------|
| Domain | `grafana.paccoco.com` |
| Scheme | `http` |
| Host | `grafana` |
| Port | `3000` |
Create a Pangolin resource for Grafana:
Since Grafana joins the `pangolin` network, PD's Newt can reach it by container name.
- Domain: grafana.paccoco.com
- Scheme: http
- Host: grafana
- Port: 3000
## 5. Validate & Deploy
If Grafana is moving to Authelia OIDC, Pangolin auth in front of the Grafana resource should be disabled after OIDC is verified. Otherwise users get double-auth prompts for no good reason.
Loki stays internal-only on PD. Do not publish its host port unless you intentionally move or retire any service already using `3100`.
## 5. Validate and deploy
```bash
cd /opt/monitoring
cd /mnt/docker-ssd/docker/compose/monitoring
docker compose --env-file .env config
docker compose --env-file .env up -d
```
## 6. Post-Deploy Verification
Or run the helper:
```bash
# All three containers running?
docker ps --filter name=prometheus --filter name=grafana --filter name=node-exporter
# Prometheus scraping targets?
curl -s http://10.5.1.6:9090/api/v1/targets | python3 -m json.tool | head -40
# Grafana UI accessible?
curl -s -o /dev/null -w "%{http_code}" http://10.5.1.6:3000/
# Expected: 200 or 302
# Node exporter metrics flowing?
curl -s http://10.5.1.6:9100/metrics | head -10
cd /mnt/docker-ssd/docker/compose/monitoring
./deploy.sh
```
> Remember: use `10.5.1.6` not `localhost` for health checks on TrueNAS Scale.
## 6. Post-deploy verification
## 7. Recommended Dashboard Imports
```bash
docker ps --filter name=prometheus --filter name=grafana --filter name=loki --filter name=promtail --filter name=node-exporter --filter name=cadvisor
In Grafana → Dashboards → New → Import → Enter ID:
curl -fsS http://10.5.1.6:9090/-/ready
curl -fsS http://10.5.1.6:9100/metrics | head -10
curl -fsS http://10.5.1.6:9090/api/v1/targets | python3 -m json.tool | head -60
sudo docker inspect -f '{{.State.Status}}' loki
```
| Dashboard | ID | Purpose |
|------------------------|------|-----------------------------------|
| Node Exporter Full | 1860 | PD system metrics |
| Docker Container Stats | 893 | Container resource usage |
| Netdata via Prometheus | search | PD and Serenity system metrics |
In Grafana, confirm:
## 8. Grafana → n8n Alert Webhook
- Prometheus datasource is healthy
- Loki datasource is healthy
- Explore can query labels from Loki
- login page shows the `Authelia` OAuth option
In Grafana → Alerting → Contact Points, create:
## 7. What this adds
| Field | Value |
|-------------|--------------------------------------------------|
| Name | `n8n-alerts` |
| Type | Webhook |
| URL | `https://n8n.paccoco.com/webhook/grafana-alert` |
| HTTP Method | POST |
Suggested alert rules:
- ZFS pool utilization > 85%
- Container memory > 90% of limit
- Host CPU sustained > 90% for 5 minutes
- Disk I/O latency spikes
- host metrics from node-exporter
- PD container metrics from cAdvisor
- PD Docker logs into Loki via Promtail
- Grafana Explore for metrics and logs in one place
## Notes
- **Storage:** Prometheus TSDB and Grafana data are on tank (plenty of headroom). 90-day retention is configured. Prometheus writes sequentially so tank performs fine here.
- **N.O.M.A.D. scraping:** The N.O.M.A.D. target in prometheus.yml is commented out. Uncomment once Netdata or node-exporter is running on N.O.M.A.D. at 10.5.1.16.
- **Image versions:** Prometheus v3.4.0, Grafana 13.0.1, node-exporter v1.9.0 — verify these are still current before deploying.
- Promtail is intentionally used instead of Alloy here because it is simpler to validate and recover on this host.
- cAdvisor currently covers PD containers only. If you want NOMAD container metrics too, deploy a separate exporter on NOMAD instead of pretending PD can see across hosts.
- Prometheus remains explicitly pinned to the v3 line because latest has been sloppy about major-version intent.