Phase 6 — Grafana + Prometheus Deployment on PD

Adapted from HOMELAB_BUILDOUT_PLAN.md Phase 6, changed host from N.O.M.A.D. to PlausibleDeniability.

1. Scaffold Directories

SSH into PD (or run locally):

# Stack directory
sudo mkdir -p /opt/monitoring/provisioning/datasources
sudo mkdir -p /opt/monitoring/provisioning/dashboards

# Prometheus TSDB on tank (sequential writes, grows with retention)
sudo mkdir -p /mnt/tank/docker/appdata/prometheus
sudo chown 65534:65534 /mnt/tank/docker/appdata/prometheus

# Grafana data on tank
sudo mkdir -p /mnt/tank/docker/appdata/grafana
sudo chown 472:472 /mnt/tank/docker/appdata/grafana

2. Copy Files to PD

Copy everything from this monitoring/ folder to /opt/monitoring/ on PD:

# From your local machine (adjust source path as needed)
scp -r monitoring/* pd:/opt/monitoring/

# Or if working directly on PD, copy files into place:
# docker-compose.yaml   → /opt/monitoring/docker-compose.yaml
# prometheus.yml        → /opt/monitoring/prometheus.yml
# provisioning/         → /opt/monitoring/provisioning/
# .env.example          → /opt/monitoring/.env.example

3. Create .env

cd /opt/monitoring
cp .env.example .env
nano .env
# Set GF_ADMIN_PASS=$(openssl rand -hex 16)

4. Pangolin Configuration

In Pangolin dashboard, create a new resource using PD's existing Newt:

Field	Value
Domain	`grafana.paccoco.com`
Scheme	`http`
Host	`grafana`
Port	`3000`

Since Grafana joins the pangolin network, PD's Newt can reach it by container name.

5. Validate & Deploy

cd /opt/monitoring
docker compose --env-file .env config
docker compose --env-file .env up -d

6. Post-Deploy Verification

# All three containers running?
docker ps --filter name=prometheus --filter name=grafana --filter name=node-exporter

# Prometheus scraping targets?
curl -s http://10.5.1.6:9090/api/v1/targets | python3 -m json.tool | head -40

# Grafana UI accessible?
curl -s -o /dev/null -w "%{http_code}" http://10.5.1.6:3000/
# Expected: 200 or 302

# Node exporter metrics flowing?
curl -s http://10.5.1.6:9100/metrics | head -10

Remember: use 10.5.1.6 not localhost for health checks on TrueNAS Scale.

7. Recommended Dashboard Imports

In Grafana → Dashboards → New → Import → Enter ID:

Dashboard	ID	Purpose
Node Exporter Full	1860	PD system metrics
Docker Container Stats	893	Container resource usage
Netdata via Prometheus	search	PD and Serenity system metrics

8. Grafana → n8n Alert Webhook

In Grafana → Alerting → Contact Points, create:

Field	Value
Name	`n8n-alerts`
Type	Webhook
URL	`https://n8n.paccoco.com/webhook/grafana-alert`
HTTP Method	POST

Suggested alert rules:

ZFS pool utilization > 85%
Container memory > 90% of limit
Host CPU sustained > 90% for 5 minutes
Disk I/O latency spikes

Notes

Storage: Prometheus TSDB and Grafana data are on tank (plenty of headroom). 90-day retention is configured. Prometheus writes sequentially so tank performs fine here.
N.O.M.A.D. scraping: The N.O.M.A.D. target in prometheus.yml is commented out. Uncomment once Netdata or node-exporter is running on N.O.M.A.D. at 10.5.1.16.
Image versions: Prometheus v3.4.0, Grafana 13.0.1, node-exporter v1.9.0 — verify these are still current before deploying.

3.9 KiB Raw Blame History