119 lines
3.9 KiB
Markdown
119 lines
3.9 KiB
Markdown
# Phase 6 — Grafana + Prometheus Deployment on PD
|
|
|
|
> Adapted from HOMELAB_BUILDOUT_PLAN.md Phase 6, changed host from N.O.M.A.D. to PlausibleDeniability.
|
|
|
|
## 1. Scaffold Directories
|
|
|
|
SSH into PD (or run locally):
|
|
|
|
```bash
|
|
# Stack directory
|
|
sudo mkdir -p /opt/monitoring/provisioning/datasources
|
|
sudo mkdir -p /opt/monitoring/provisioning/dashboards
|
|
|
|
# Prometheus TSDB on tank (sequential writes, grows with retention)
|
|
sudo mkdir -p /mnt/tank/docker/appdata/prometheus
|
|
sudo chown 65534:65534 /mnt/tank/docker/appdata/prometheus
|
|
|
|
# Grafana data on tank
|
|
sudo mkdir -p /mnt/tank/docker/appdata/grafana
|
|
sudo chown 472:472 /mnt/tank/docker/appdata/grafana
|
|
```
|
|
|
|
## 2. Copy Files to PD
|
|
|
|
Copy everything from this `monitoring/` folder to `/opt/monitoring/` on PD:
|
|
|
|
```bash
|
|
# From your local machine (adjust source path as needed)
|
|
scp -r monitoring/* pd:/opt/monitoring/
|
|
|
|
# Or if working directly on PD, copy files into place:
|
|
# docker-compose.yaml → /opt/monitoring/docker-compose.yaml
|
|
# prometheus.yml → /opt/monitoring/prometheus.yml
|
|
# provisioning/ → /opt/monitoring/provisioning/
|
|
# .env.example → /opt/monitoring/.env.example
|
|
```
|
|
|
|
## 3. Create .env
|
|
|
|
```bash
|
|
cd /opt/monitoring
|
|
cp .env.example .env
|
|
nano .env
|
|
# Set GF_ADMIN_PASS=$(openssl rand -hex 16)
|
|
```
|
|
|
|
## 4. Pangolin Configuration
|
|
|
|
In Pangolin dashboard, create a new resource using PD's existing Newt:
|
|
|
|
| Field | Value |
|
|
|--------|--------------------------|
|
|
| Domain | `grafana.paccoco.com` |
|
|
| Scheme | `http` |
|
|
| Host | `grafana` |
|
|
| Port | `3000` |
|
|
|
|
Since Grafana joins the `pangolin` network, PD's Newt can reach it by container name.
|
|
|
|
## 5. Validate & Deploy
|
|
|
|
```bash
|
|
cd /opt/monitoring
|
|
docker compose --env-file .env config
|
|
docker compose --env-file .env up -d
|
|
```
|
|
|
|
## 6. Post-Deploy Verification
|
|
|
|
```bash
|
|
# All three containers running?
|
|
docker ps --filter name=prometheus --filter name=grafana --filter name=node-exporter
|
|
|
|
# Prometheus scraping targets?
|
|
curl -s http://10.5.1.6:9090/api/v1/targets | python3 -m json.tool | head -40
|
|
|
|
# Grafana UI accessible?
|
|
curl -s -o /dev/null -w "%{http_code}" http://10.5.1.6:3000/
|
|
# Expected: 200 or 302
|
|
|
|
# Node exporter metrics flowing?
|
|
curl -s http://10.5.1.6:9100/metrics | head -10
|
|
```
|
|
|
|
> Remember: use `10.5.1.6` not `localhost` for health checks on TrueNAS Scale.
|
|
|
|
## 7. Recommended Dashboard Imports
|
|
|
|
In Grafana → Dashboards → New → Import → Enter ID:
|
|
|
|
| Dashboard | ID | Purpose |
|
|
|------------------------|------|-----------------------------------|
|
|
| Node Exporter Full | 1860 | PD system metrics |
|
|
| Docker Container Stats | 893 | Container resource usage |
|
|
| Netdata via Prometheus | search | PD and Serenity system metrics |
|
|
|
|
## 8. Grafana → n8n Alert Webhook
|
|
|
|
In Grafana → Alerting → Contact Points, create:
|
|
|
|
| Field | Value |
|
|
|-------------|--------------------------------------------------|
|
|
| Name | `n8n-alerts` |
|
|
| Type | Webhook |
|
|
| URL | `https://n8n.paccoco.com/webhook/grafana-alert` |
|
|
| HTTP Method | POST |
|
|
|
|
Suggested alert rules:
|
|
- ZFS pool utilization > 85%
|
|
- Container memory > 90% of limit
|
|
- Host CPU sustained > 90% for 5 minutes
|
|
- Disk I/O latency spikes
|
|
|
|
## Notes
|
|
|
|
- **Storage:** Prometheus TSDB and Grafana data are on tank (plenty of headroom). 90-day retention is configured. Prometheus writes sequentially so tank performs fine here.
|
|
- **N.O.M.A.D. scraping:** The N.O.M.A.D. target in prometheus.yml is commented out. Uncomment once Netdata or node-exporter is running on N.O.M.A.D. at 10.5.1.16.
|
|
- **Image versions:** Prometheus v3.4.0, Grafana 13.0.1, node-exporter v1.9.0 — verify these are still current before deploying.
|