106 lines
2.8 KiB
Markdown
106 lines
2.8 KiB
Markdown
# Monitoring Deployment on PD
|
|
|
|
This stack now includes:
|
|
|
|
- Prometheus
|
|
- Grafana
|
|
- Loki
|
|
- Promtail
|
|
- node-exporter
|
|
- cAdvisor
|
|
|
|
Source-of-truth path:
|
|
|
|
- /mnt/docker-ssd/docker/compose/monitoring
|
|
|
|
## 1. Stage repo files on PD
|
|
|
|
Make sure the monitoring directory from this repo is present on PD at:
|
|
|
|
- /mnt/docker-ssd/docker/compose/monitoring
|
|
|
|
## 2. Prepare runtime directories
|
|
|
|
Create the required appdata paths on PD:
|
|
|
|
```bash
|
|
mkdir -p /mnt/tank/docker/appdata/prometheus
|
|
mkdir -p /mnt/tank/docker/appdata/grafana/dashboards
|
|
mkdir -p /mnt/tank/docker/appdata/loki
|
|
mkdir -p /mnt/tank/docker/appdata/promtail
|
|
|
|
chown 65534:65534 /mnt/tank/docker/appdata/prometheus
|
|
chown 472:472 /mnt/tank/docker/appdata/grafana
|
|
chown 472:472 /mnt/tank/docker/appdata/grafana/dashboards
|
|
```
|
|
|
|
## 3. Create the live env file
|
|
|
|
```bash
|
|
cd /mnt/docker-ssd/docker/compose/monitoring
|
|
cp .env.example .env
|
|
chmod 600 .env
|
|
```
|
|
|
|
Set a real Grafana password before deploy.
|
|
Set a real Grafana OIDC client secret before enabling Authelia login.
|
|
|
|
## 4. Pangolin exposure
|
|
|
|
Create a Pangolin resource for Grafana:
|
|
|
|
- Domain: grafana.paccoco.com
|
|
- Scheme: http
|
|
- Host: grafana
|
|
- Port: 3000
|
|
|
|
If Grafana is moving to Authelia OIDC, Pangolin auth in front of the Grafana resource should be disabled after OIDC is verified. Otherwise users get double-auth prompts for no good reason.
|
|
|
|
Loki stays internal-only on PD. Do not publish its host port unless you intentionally move or retire any service already using `3100`.
|
|
|
|
## 5. Validate and deploy
|
|
|
|
```bash
|
|
cd /mnt/docker-ssd/docker/compose/monitoring
|
|
docker compose --env-file .env config
|
|
docker compose --env-file .env up -d
|
|
```
|
|
|
|
Or run the helper:
|
|
|
|
```bash
|
|
cd /mnt/docker-ssd/docker/compose/monitoring
|
|
./deploy.sh
|
|
```
|
|
|
|
## 6. Post-deploy verification
|
|
|
|
```bash
|
|
docker ps --filter name=prometheus --filter name=grafana --filter name=loki --filter name=promtail --filter name=node-exporter --filter name=cadvisor
|
|
|
|
curl -fsS http://10.5.30.6:9090/-/ready
|
|
curl -fsS http://10.5.30.6:9100/metrics | head -10
|
|
curl -fsS http://10.5.30.6:9090/api/v1/targets | python3 -m json.tool | head -60
|
|
sudo docker inspect -f '{{.State.Status}}' loki
|
|
```
|
|
|
|
In Grafana, confirm:
|
|
|
|
- Prometheus datasource is healthy
|
|
- Loki datasource is healthy
|
|
- Explore can query labels from Loki
|
|
- login page shows the `Authelia` OAuth option
|
|
|
|
## 7. What this adds
|
|
|
|
- host metrics from node-exporter
|
|
- PD container metrics from cAdvisor
|
|
- PD Docker logs into Loki via Promtail
|
|
- Grafana Explore for metrics and logs in one place
|
|
|
|
## Notes
|
|
|
|
- Promtail is intentionally used instead of Alloy here because it is simpler to validate and recover on this host.
|
|
- cAdvisor currently covers PD containers only. If you want NOMAD container metrics too, deploy a separate exporter on NOMAD instead of pretending PD can see across hosts.
|
|
- Prometheus remains explicitly pinned to the v3 line because latest has been sloppy about major-version intent.
|