# Monitoring Deployment on PD

This stack now includes:

- Prometheus
- Grafana
- Loki
- Promtail
- node-exporter
- cAdvisor

Source-of-truth path:

- /mnt/docker-ssd/docker/compose/monitoring

## 1. Stage repo files on PD

Make sure the monitoring directory from this repo is present on PD at:

- /mnt/docker-ssd/docker/compose/monitoring

## 2. Prepare runtime directories

Create the required appdata paths on PD:

```bash
mkdir -p /mnt/tank/docker/appdata/prometheus
mkdir -p /mnt/tank/docker/appdata/grafana/dashboards
mkdir -p /mnt/tank/docker/appdata/loki
mkdir -p /mnt/tank/docker/appdata/promtail

chown 65534:65534 /mnt/tank/docker/appdata/prometheus
chown 472:472 /mnt/tank/docker/appdata/grafana
chown 472:472 /mnt/tank/docker/appdata/grafana/dashboards
```

## 3. Create the live env file

```bash
cd /mnt/docker-ssd/docker/compose/monitoring
cp .env.example .env
chmod 600 .env
```

Set a real Grafana password before deploy.
Set a real Grafana OIDC client secret before enabling Authelia login.

## 4. Pangolin exposure

Create a Pangolin resource for Grafana:

- Domain: grafana.paccoco.com
- Scheme: http
- Host: grafana
- Port: 3000

If Grafana is moving to Authelia OIDC, Pangolin auth in front of the Grafana resource should be disabled after OIDC is verified. Otherwise users get double-auth prompts for no good reason.

Loki stays internal-only on PD. Do not publish its host port unless you intentionally move or retire any service already using `3100`.

## 5. Validate and deploy

```bash
cd /mnt/docker-ssd/docker/compose/monitoring
docker compose --env-file .env config
docker compose --env-file .env up -d
```

Or run the helper:

```bash
cd /mnt/docker-ssd/docker/compose/monitoring
./deploy.sh
```

## 6. Post-deploy verification

```bash
docker ps --filter name=prometheus --filter name=grafana --filter name=loki --filter name=promtail --filter name=node-exporter --filter name=cadvisor

curl -fsS http://10.5.30.6:9090/-/ready
curl -fsS http://10.5.30.6:9100/metrics | head -10
curl -fsS http://10.5.30.6:9090/api/v1/targets | python3 -m json.tool | head -60
sudo docker inspect -f '{{.State.Status}}' loki
```

In Grafana, confirm:

- Prometheus datasource is healthy
- Loki datasource is healthy
- Explore can query labels from Loki
- login page shows the `Authelia` OAuth option

## 7. What this adds

- host metrics from node-exporter
- PD container metrics from cAdvisor
- PD Docker logs into Loki via Promtail
- Grafana Explore for metrics and logs in one place

## Notes

- Promtail is intentionally used instead of Alloy here because it is simpler to validate and recover on this host.
- cAdvisor currently covers PD containers only. If you want NOMAD container metrics too, deploy a separate exporter on NOMAD instead of pretending PD can see across hosts.
- Prometheus remains explicitly pinned to the v3 line because latest has been sloppy about major-version intent.