# Phase 6 — Grafana + Prometheus Deployment on PD

> Adapted from HOMELAB_BUILDOUT_PLAN.md Phase 6, changed host from N.O.M.A.D. to PlausibleDeniability.

## 1. Scaffold Directories

SSH into PD (or run locally):

```bash
# Stack directory
sudo mkdir -p /opt/monitoring/provisioning/datasources
sudo mkdir -p /opt/monitoring/provisioning/dashboards

# Prometheus TSDB on tank (sequential writes, grows with retention)
sudo mkdir -p /mnt/tank/docker/appdata/prometheus
sudo chown 65534:65534 /mnt/tank/docker/appdata/prometheus

# Grafana data on tank
sudo mkdir -p /mnt/tank/docker/appdata/grafana
sudo chown 472:472 /mnt/tank/docker/appdata/grafana
```

## 2. Copy Files to PD

Copy everything from this `monitoring/` folder to `/opt/monitoring/` on PD:

```bash
# From your local machine (adjust source path as needed)
scp -r monitoring/* pd:/opt/monitoring/

# Or if working directly on PD, copy files into place:
# docker-compose.yaml   → /opt/monitoring/docker-compose.yaml
# prometheus.yml        → /opt/monitoring/prometheus.yml
# provisioning/         → /opt/monitoring/provisioning/
# .env.example          → /opt/monitoring/.env.example
```

## 3. Create .env

```bash
cd /opt/monitoring
cp .env.example .env
nano .env
# Set GF_ADMIN_PASS=$(openssl rand -hex 16)
```

## 4. Pangolin Configuration

In Pangolin dashboard, create a new resource using PD's existing Newt:

| Field  | Value                    |
|--------|--------------------------|
| Domain | `grafana.paccoco.com`    |
| Scheme | `http`                   |
| Host   | `grafana`                |
| Port   | `3000`                   |

Since Grafana joins the `pangolin` network, PD's Newt can reach it by container name.

## 5. Validate & Deploy

```bash
cd /opt/monitoring
docker compose --env-file .env config
docker compose --env-file .env up -d
```

## 6. Post-Deploy Verification

```bash
# All three containers running?
docker ps --filter name=prometheus --filter name=grafana --filter name=node-exporter

# Prometheus scraping targets?
curl -s http://10.5.1.6:9090/api/v1/targets | python3 -m json.tool | head -40

# Grafana UI accessible?
curl -s -o /dev/null -w "%{http_code}" http://10.5.1.6:3000/
# Expected: 200 or 302

# Node exporter metrics flowing?
curl -s http://10.5.1.6:9100/metrics | head -10
```

> Remember: use `10.5.1.6` not `localhost` for health checks on TrueNAS Scale.

## 7. Recommended Dashboard Imports

In Grafana → Dashboards → New → Import → Enter ID:

| Dashboard              | ID   | Purpose                           |
|------------------------|------|-----------------------------------|
| Node Exporter Full     | 1860 | PD system metrics                 |
| Docker Container Stats | 893  | Container resource usage          |
| Netdata via Prometheus | search | PD and Serenity system metrics  |

## 8. Grafana → n8n Alert Webhook

In Grafana → Alerting → Contact Points, create:

| Field       | Value                                            |
|-------------|--------------------------------------------------|
| Name        | `n8n-alerts`                                     |
| Type        | Webhook                                          |
| URL         | `https://n8n.paccoco.com/webhook/grafana-alert`  |
| HTTP Method | POST                                             |

Suggested alert rules:
- ZFS pool utilization > 85%
- Container memory > 90% of limit
- Host CPU sustained > 90% for 5 minutes
- Disk I/O latency spikes

## Notes

- **Storage:** Prometheus TSDB and Grafana data are on tank (plenty of headroom). 90-day retention is configured. Prometheus writes sequentially so tank performs fine here.
- **N.O.M.A.D. scraping:** The N.O.M.A.D. target in prometheus.yml is commented out. Uncomment once Netdata or node-exporter is running on N.O.M.A.D. at 10.5.1.16.
- **Image versions:** Prometheus v3.4.0, Grafana 13.0.1, node-exporter v1.9.0 — verify these are still current before deploying.