Local homelab changes

2026-05-10 09:56:29 -05:00
parent 2048a07abf
commit 22d2d75f6a
24 changed files with 4669 additions and 0 deletions
--- a/monitoring/DEPLOY.md
+++ b/monitoring/DEPLOY.md
@@ -0,0 +1,118 @@
+# Phase 6 — Grafana + Prometheus Deployment on PD
+
+> Adapted from HOMELAB_BUILDOUT_PLAN.md Phase 6, changed host from N.O.M.A.D. to PlausibleDeniability.
+
+## 1. Scaffold Directories
+
+SSH into PD (or run locally):
+
+```bash
+# Stack directory
+sudo mkdir -p /opt/monitoring/provisioning/datasources
+sudo mkdir -p /opt/monitoring/provisioning/dashboards
+
+# Prometheus TSDB on tank (sequential writes, grows with retention)
+sudo mkdir -p /mnt/tank/docker/appdata/prometheus
+sudo chown 65534:65534 /mnt/tank/docker/appdata/prometheus
+
+# Grafana data on tank
+sudo mkdir -p /mnt/tank/docker/appdata/grafana
+sudo chown 472:472 /mnt/tank/docker/appdata/grafana
+```
+
+## 2. Copy Files to PD
+
+Copy everything from this `monitoring/` folder to `/opt/monitoring/` on PD:
+
+```bash
+# From your local machine (adjust source path as needed)
+scp -r monitoring/* pd:/opt/monitoring/
+
+# Or if working directly on PD, copy files into place:
+# docker-compose.yaml   → /opt/monitoring/docker-compose.yaml
+# prometheus.yml        → /opt/monitoring/prometheus.yml
+# provisioning/         → /opt/monitoring/provisioning/
+# .env.example          → /opt/monitoring/.env.example
+```
+
+## 3. Create .env
+
+```bash
+cd /opt/monitoring
+cp .env.example .env
+nano .env
+# Set GF_ADMIN_PASS=$(openssl rand -hex 16)
+```
+
+## 4. Pangolin Configuration
+
+In Pangolin dashboard, create a new resource using PD's existing Newt:
+
+| Field  | Value                    |
+|--------|--------------------------|
+| Domain | `grafana.paccoco.com`    |
+| Scheme | `http`                   |
+| Host   | `grafana`                |
+| Port   | `3000`                   |
+
+Since Grafana joins the `pangolin` network, PD's Newt can reach it by container name.
+
+## 5. Validate & Deploy
+
+```bash
+cd /opt/monitoring
+docker compose --env-file .env config
+docker compose --env-file .env up -d
+```
+
+## 6. Post-Deploy Verification
+
+```bash
+# All three containers running?
+docker ps --filter name=prometheus --filter name=grafana --filter name=node-exporter
+
+# Prometheus scraping targets?
+curl -s http://10.5.1.6:9090/api/v1/targets | python3 -m json.tool | head -40
+
+# Grafana UI accessible?
+curl -s -o /dev/null -w "%{http_code}" http://10.5.1.6:3000/
+# Expected: 200 or 302
+
+# Node exporter metrics flowing?
+curl -s http://10.5.1.6:9100/metrics | head -10
+```
+
+> Remember: use `10.5.1.6` not `localhost` for health checks on TrueNAS Scale.
+
+## 7. Recommended Dashboard Imports
+
+In Grafana → Dashboards → New → Import → Enter ID:
+
+| Dashboard              | ID   | Purpose                           |
+|------------------------|------|-----------------------------------|
+| Node Exporter Full     | 1860 | PD system metrics                 |
+| Docker Container Stats | 893  | Container resource usage          |
+| Netdata via Prometheus | search | PD and Serenity system metrics  |
+
+## 8. Grafana → n8n Alert Webhook
+
+In Grafana → Alerting → Contact Points, create:
+
+| Field       | Value                                            |
+|-------------|--------------------------------------------------|
+| Name        | `n8n-alerts`                                     |
+| Type        | Webhook                                          |
+| URL         | `https://n8n.paccoco.com/webhook/grafana-alert`  |
+| HTTP Method | POST                                             |
+
+Suggested alert rules:
+- ZFS pool utilization > 85%
+- Container memory > 90% of limit
+- Host CPU sustained > 90% for 5 minutes
+- Disk I/O latency spikes
+
+## Notes
+
+- **Storage:** Prometheus TSDB and Grafana data are on tank (plenty of headroom). 90-day retention is configured. Prometheus writes sequentially so tank performs fine here.
+- **N.O.M.A.D. scraping:** The N.O.M.A.D. target in prometheus.yml is commented out. Uncomment once Netdata or node-exporter is running on N.O.M.A.D. at 10.5.1.16.
+- **Image versions:** Prometheus v3.4.0, Grafana 13.0.1, node-exporter v1.9.0 — verify these are still current before deploying.