Monitoring Deployment on PD

This stack now includes:

Prometheus
Grafana
Loki
Promtail
node-exporter
cAdvisor

Source-of-truth path:

/mnt/docker-ssd/docker/compose/monitoring

1. Stage repo files on PD

Make sure the monitoring directory from this repo is present on PD at:

/mnt/docker-ssd/docker/compose/monitoring

2. Prepare runtime directories

Create the required appdata paths on PD:

mkdir -p /mnt/tank/docker/appdata/prometheus
mkdir -p /mnt/tank/docker/appdata/grafana/dashboards
mkdir -p /mnt/tank/docker/appdata/loki
mkdir -p /mnt/tank/docker/appdata/promtail

chown 65534:65534 /mnt/tank/docker/appdata/prometheus
chown 472:472 /mnt/tank/docker/appdata/grafana
chown 472:472 /mnt/tank/docker/appdata/grafana/dashboards

3. Create the live env file

cd /mnt/docker-ssd/docker/compose/monitoring
cp .env.example .env
chmod 600 .env

Set a real Grafana password before deploy. Set a real Grafana OIDC client secret before enabling Authelia login.

4. Pangolin exposure

Create a Pangolin resource for Grafana:

Domain: grafana.paccoco.com
Scheme: http
Host: grafana
Port: 3000

If Grafana is moving to Authelia OIDC, Pangolin auth in front of the Grafana resource should be disabled after OIDC is verified. Otherwise users get double-auth prompts for no good reason.

Loki stays internal-only on PD. Do not publish its host port unless you intentionally move or retire any service already using 3100.

5. Validate and deploy

cd /mnt/docker-ssd/docker/compose/monitoring
docker compose --env-file .env config
docker compose --env-file .env up -d

Or run the helper:

cd /mnt/docker-ssd/docker/compose/monitoring
./deploy.sh

6. Post-deploy verification

docker ps --filter name=prometheus --filter name=grafana --filter name=loki --filter name=promtail --filter name=node-exporter --filter name=cadvisor

curl -fsS http://10.5.30.6:9090/-/ready
curl -fsS http://10.5.30.6:9100/metrics | head -10
curl -fsS http://10.5.30.6:9090/api/v1/targets | python3 -m json.tool | head -60
sudo docker inspect -f '{{.State.Status}}' loki

In Grafana, confirm:

Prometheus datasource is healthy
Loki datasource is healthy
Explore can query labels from Loki
login page shows the Authelia OAuth option

7. What this adds

host metrics from node-exporter
PD container metrics from cAdvisor
PD Docker logs into Loki via Promtail
Grafana Explore for metrics and logs in one place

Notes

Promtail is intentionally used instead of Alloy here because it is simpler to validate and recover on this host.
cAdvisor currently covers PD containers only. If you want NOMAD container metrics too, deploy a separate exporter on NOMAD instead of pretending PD can see across hosts.
Prometheus remains explicitly pinned to the v3 line because latest has been sloppy about major-version intent.

2.8 KiB Raw Blame History