chore: sync homelab ops, identity, and monitoring docs
This commit is contained in:
Fizzlepoof
@@ -4,3 +4,21 @@ TZ=America/New_York
|
||||
GF_ADMIN_USER=admin
|
||||
GF_ADMIN_PASS=CHANGE_ME
|
||||
GF_HOST=grafana.paccoco.com
|
||||
LOKI_HOST=loki.paccoco.com
|
||||
AUTHELIA_HOST=auth.paccoco.com
|
||||
GF_AUTH_GENERIC_OAUTH_ENABLED=true
|
||||
GF_AUTH_GENERIC_OAUTH_NAME=Authelia
|
||||
GF_AUTH_GENERIC_OAUTH_ICON=signin
|
||||
GF_AUTH_GENERIC_OAUTH_CLIENT_ID=grafana
|
||||
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=CHANGE_ME
|
||||
GF_AUTH_GENERIC_OAUTH_SCOPES=openid profile email groups
|
||||
GF_AUTH_GENERIC_OAUTH_EMPTY_SCOPES=false
|
||||
GF_AUTH_GENERIC_OAUTH_USE_PKCE=true
|
||||
GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
|
||||
GF_AUTH_GENERIC_OAUTH_AUTO_LOGIN=false
|
||||
GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH=preferred_username
|
||||
GF_AUTH_GENERIC_OAUTH_NAME_ATTRIBUTE_PATH=name
|
||||
GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH=email
|
||||
GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=contains(groups[*], 'admins') && 'Admin' || 'Viewer'
|
||||
GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN=true
|
||||
GF_AUTH_DISABLE_LOGIN_FORM=false
|
||||
|
||||
@@ -1,118 +1,105 @@
|
||||
# Phase 6 — Grafana + Prometheus Deployment on PD
|
||||
# Monitoring Deployment on PD
|
||||
|
||||
> Adapted from HOMELAB_BUILDOUT_PLAN.md Phase 6, changed host from N.O.M.A.D. to PlausibleDeniability.
|
||||
This stack now includes:
|
||||
|
||||
## 1. Scaffold Directories
|
||||
- Prometheus
|
||||
- Grafana
|
||||
- Loki
|
||||
- Promtail
|
||||
- node-exporter
|
||||
- cAdvisor
|
||||
|
||||
SSH into PD (or run locally):
|
||||
Source-of-truth path:
|
||||
|
||||
- /mnt/docker-ssd/docker/compose/monitoring
|
||||
|
||||
## 1. Stage repo files on PD
|
||||
|
||||
Make sure the monitoring directory from this repo is present on PD at:
|
||||
|
||||
- /mnt/docker-ssd/docker/compose/monitoring
|
||||
|
||||
## 2. Prepare runtime directories
|
||||
|
||||
Create the required appdata paths on PD:
|
||||
|
||||
```bash
|
||||
# Stack directory
|
||||
sudo mkdir -p /opt/monitoring/provisioning/datasources
|
||||
sudo mkdir -p /opt/monitoring/provisioning/dashboards
|
||||
mkdir -p /mnt/tank/docker/appdata/prometheus
|
||||
mkdir -p /mnt/tank/docker/appdata/grafana/dashboards
|
||||
mkdir -p /mnt/tank/docker/appdata/loki
|
||||
mkdir -p /mnt/tank/docker/appdata/promtail
|
||||
|
||||
# Prometheus TSDB on tank (sequential writes, grows with retention)
|
||||
sudo mkdir -p /mnt/tank/docker/appdata/prometheus
|
||||
sudo chown 65534:65534 /mnt/tank/docker/appdata/prometheus
|
||||
|
||||
# Grafana data on tank
|
||||
sudo mkdir -p /mnt/tank/docker/appdata/grafana
|
||||
sudo chown 472:472 /mnt/tank/docker/appdata/grafana
|
||||
chown 65534:65534 /mnt/tank/docker/appdata/prometheus
|
||||
chown 472:472 /mnt/tank/docker/appdata/grafana
|
||||
chown 472:472 /mnt/tank/docker/appdata/grafana/dashboards
|
||||
```
|
||||
|
||||
## 2. Copy Files to PD
|
||||
|
||||
Copy everything from this `monitoring/` folder to `/opt/monitoring/` on PD:
|
||||
## 3. Create the live env file
|
||||
|
||||
```bash
|
||||
# From your local machine (adjust source path as needed)
|
||||
scp -r monitoring/* pd:/opt/monitoring/
|
||||
|
||||
# Or if working directly on PD, copy files into place:
|
||||
# docker-compose.yaml → /opt/monitoring/docker-compose.yaml
|
||||
# prometheus.yml → /opt/monitoring/prometheus.yml
|
||||
# provisioning/ → /opt/monitoring/provisioning/
|
||||
# .env.example → /opt/monitoring/.env.example
|
||||
```
|
||||
|
||||
## 3. Create .env
|
||||
|
||||
```bash
|
||||
cd /opt/monitoring
|
||||
cd /mnt/docker-ssd/docker/compose/monitoring
|
||||
cp .env.example .env
|
||||
nano .env
|
||||
# Set GF_ADMIN_PASS=$(openssl rand -hex 16)
|
||||
chmod 600 .env
|
||||
```
|
||||
|
||||
## 4. Pangolin Configuration
|
||||
Set a real Grafana password before deploy.
|
||||
Set a real Grafana OIDC client secret before enabling Authelia login.
|
||||
|
||||
In Pangolin dashboard, create a new resource using PD's existing Newt:
|
||||
## 4. Pangolin exposure
|
||||
|
||||
| Field | Value |
|
||||
|--------|--------------------------|
|
||||
| Domain | `grafana.paccoco.com` |
|
||||
| Scheme | `http` |
|
||||
| Host | `grafana` |
|
||||
| Port | `3000` |
|
||||
Create a Pangolin resource for Grafana:
|
||||
|
||||
Since Grafana joins the `pangolin` network, PD's Newt can reach it by container name.
|
||||
- Domain: grafana.paccoco.com
|
||||
- Scheme: http
|
||||
- Host: grafana
|
||||
- Port: 3000
|
||||
|
||||
## 5. Validate & Deploy
|
||||
If Grafana is moving to Authelia OIDC, Pangolin auth in front of the Grafana resource should be disabled after OIDC is verified. Otherwise users get double-auth prompts for no good reason.
|
||||
|
||||
Loki stays internal-only on PD. Do not publish its host port unless you intentionally move or retire any service already using `3100`.
|
||||
|
||||
## 5. Validate and deploy
|
||||
|
||||
```bash
|
||||
cd /opt/monitoring
|
||||
cd /mnt/docker-ssd/docker/compose/monitoring
|
||||
docker compose --env-file .env config
|
||||
docker compose --env-file .env up -d
|
||||
```
|
||||
|
||||
## 6. Post-Deploy Verification
|
||||
Or run the helper:
|
||||
|
||||
```bash
|
||||
# All three containers running?
|
||||
docker ps --filter name=prometheus --filter name=grafana --filter name=node-exporter
|
||||
|
||||
# Prometheus scraping targets?
|
||||
curl -s http://10.5.1.6:9090/api/v1/targets | python3 -m json.tool | head -40
|
||||
|
||||
# Grafana UI accessible?
|
||||
curl -s -o /dev/null -w "%{http_code}" http://10.5.1.6:3000/
|
||||
# Expected: 200 or 302
|
||||
|
||||
# Node exporter metrics flowing?
|
||||
curl -s http://10.5.1.6:9100/metrics | head -10
|
||||
cd /mnt/docker-ssd/docker/compose/monitoring
|
||||
./deploy.sh
|
||||
```
|
||||
|
||||
> Remember: use `10.5.1.6` not `localhost` for health checks on TrueNAS Scale.
|
||||
## 6. Post-deploy verification
|
||||
|
||||
## 7. Recommended Dashboard Imports
|
||||
```bash
|
||||
docker ps --filter name=prometheus --filter name=grafana --filter name=loki --filter name=promtail --filter name=node-exporter --filter name=cadvisor
|
||||
|
||||
In Grafana → Dashboards → New → Import → Enter ID:
|
||||
curl -fsS http://10.5.1.6:9090/-/ready
|
||||
curl -fsS http://10.5.1.6:9100/metrics | head -10
|
||||
curl -fsS http://10.5.1.6:9090/api/v1/targets | python3 -m json.tool | head -60
|
||||
sudo docker inspect -f '{{.State.Status}}' loki
|
||||
```
|
||||
|
||||
| Dashboard | ID | Purpose |
|
||||
|------------------------|------|-----------------------------------|
|
||||
| Node Exporter Full | 1860 | PD system metrics |
|
||||
| Docker Container Stats | 893 | Container resource usage |
|
||||
| Netdata via Prometheus | search | PD and Serenity system metrics |
|
||||
In Grafana, confirm:
|
||||
|
||||
## 8. Grafana → n8n Alert Webhook
|
||||
- Prometheus datasource is healthy
|
||||
- Loki datasource is healthy
|
||||
- Explore can query labels from Loki
|
||||
- login page shows the `Authelia` OAuth option
|
||||
|
||||
In Grafana → Alerting → Contact Points, create:
|
||||
## 7. What this adds
|
||||
|
||||
| Field | Value |
|
||||
|-------------|--------------------------------------------------|
|
||||
| Name | `n8n-alerts` |
|
||||
| Type | Webhook |
|
||||
| URL | `https://n8n.paccoco.com/webhook/grafana-alert` |
|
||||
| HTTP Method | POST |
|
||||
|
||||
Suggested alert rules:
|
||||
- ZFS pool utilization > 85%
|
||||
- Container memory > 90% of limit
|
||||
- Host CPU sustained > 90% for 5 minutes
|
||||
- Disk I/O latency spikes
|
||||
- host metrics from node-exporter
|
||||
- PD container metrics from cAdvisor
|
||||
- PD Docker logs into Loki via Promtail
|
||||
- Grafana Explore for metrics and logs in one place
|
||||
|
||||
## Notes
|
||||
|
||||
- **Storage:** Prometheus TSDB and Grafana data are on tank (plenty of headroom). 90-day retention is configured. Prometheus writes sequentially so tank performs fine here.
|
||||
- **N.O.M.A.D. scraping:** The N.O.M.A.D. target in prometheus.yml is commented out. Uncomment once Netdata or node-exporter is running on N.O.M.A.D. at 10.5.1.16.
|
||||
- **Image versions:** Prometheus v3.4.0, Grafana 13.0.1, node-exporter v1.9.0 — verify these are still current before deploying.
|
||||
- Promtail is intentionally used instead of Alloy here because it is simpler to validate and recover on this host.
|
||||
- cAdvisor currently covers PD containers only. If you want NOMAD container metrics too, deploy a separate exporter on NOMAD instead of pretending PD can see across hosts.
|
||||
- Prometheus remains explicitly pinned to the v3 line because latest has been sloppy about major-version intent.
|
||||
|
||||
56
monitoring/alerts.yml
Normal file
56
monitoring/alerts.yml
Normal file
@@ -0,0 +1,56 @@
|
||||
groups:
|
||||
- name: homelab-core
|
||||
rules:
|
||||
- alert: ScrapeTargetDown
|
||||
expr: up == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Prometheus target down: {{ $labels.job }}"
|
||||
description: "Target {{ $labels.instance }} for job {{ $labels.job }} has been down for more than 5 minutes."
|
||||
|
||||
- alert: PDMemoryPressureHigh
|
||||
expr: 100 * (1 - (node_memory_MemAvailable_bytes{job="pd-node"} / node_memory_MemTotal_bytes{job="pd-node"})) > 90
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "PD memory usage is high"
|
||||
description: "PlausibleDeniability memory usage has been above 90% for more than 10 minutes."
|
||||
|
||||
- alert: PDFilesystemUsageHigh
|
||||
expr: 100 * (1 - (node_filesystem_avail_bytes{job="pd-node", mountpoint="/", fstype!~"tmpfs|ramfs|overlay|squashfs|nfs4"} / node_filesystem_size_bytes{job="pd-node", mountpoint="/", fstype!~"tmpfs|ramfs|overlay|squashfs|nfs4"})) > 85
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "PD root filesystem is filling up"
|
||||
description: "PlausibleDeniability root filesystem has been above 85% for more than 15 minutes."
|
||||
|
||||
- alert: PDContainerRecentlyRestarted
|
||||
expr: changes(container_start_time_seconds{job="pd-cadvisor", name!=""}[30m]) > 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: info
|
||||
annotations:
|
||||
summary: "PD container restarted recently"
|
||||
description: "Container {{ $labels.name }} restarted within the last 30 minutes."
|
||||
|
||||
- alert: PDRestoreVerificationFailed
|
||||
expr: pd_restore_verification_success == 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "PD restore verification failed"
|
||||
description: "The last scheduled restore verification on PlausibleDeniability failed."
|
||||
|
||||
- alert: PDRestoreVerificationStale
|
||||
expr: time() - pd_restore_verification_last_success_timestamp_seconds > 8640000
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "PD restore verification is stale"
|
||||
description: "No successful PD restore verification has been recorded in the last 100 days."
|
||||
714
monitoring/dashboards/homelab-overview.json
Normal file
714
monitoring/dashboards/homelab-overview.json
Normal file
@@ -0,0 +1,714 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"pluginVersion": "13.0.1",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "max(up{host=\"plausible-deniability\"})",
|
||||
"instant": true,
|
||||
"legendFormat": "PD",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "PD Up",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 4,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"pluginVersion": "13.0.1",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "max(up{host=\"nomad\"})",
|
||||
"instant": true,
|
||||
"legendFormat": "NOMAD",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "NOMAD Up",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 8,
|
||||
"y": 0
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"pluginVersion": "13.0.1",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "max(up{host=\"serenity\"})",
|
||||
"instant": true,
|
||||
"legendFormat": "Serenity",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Serenity Up",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 1
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 2
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"pluginVersion": "13.0.1",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "count(up) - sum(up)",
|
||||
"instant": true,
|
||||
"legendFormat": "Down",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Targets Down",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 16,
|
||||
"y": 0
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"pluginVersion": "13.0.1",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "rate(prometheus_tsdb_head_samples_appended_total[5m])",
|
||||
"instant": true,
|
||||
"legendFormat": "samples/s",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Samples / Sec",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 20,
|
||||
"y": 0
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"pluginVersion": "13.0.1",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "prometheus_tsdb_head_series",
|
||||
"instant": true,
|
||||
"legendFormat": "series",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Head Series",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 15,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"showPoints": "never"
|
||||
},
|
||||
"unit": "percent"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 4
|
||||
},
|
||||
"id": 7,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "13.0.1",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "100 - (avg by (host) (rate(node_cpu_seconds_total{host=~\"plausible-deniability|nomad\", mode=\"idle\"}[5m])) * 100)",
|
||||
"legendFormat": "{{host}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "100 - avg by (host) (netdata_system_cpu_percentage_average{host=\"serenity\", dimension=\"idle\"})",
|
||||
"legendFormat": "{{host}}",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "CPU Usage %",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisPlacement": "auto",
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 15,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"showPoints": "never"
|
||||
},
|
||||
"unit": "percent"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 4
|
||||
},
|
||||
"id": 8,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "13.0.1",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "100 * (1 - (node_memory_MemAvailable_bytes{host=~\"plausible-deniability|nomad\"} / node_memory_MemTotal_bytes{host=~\"plausible-deniability|nomad\"}))",
|
||||
"legendFormat": "{{host}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "100 * (netdata_system_ram_MiB_average{host=\"serenity\", dimension=\"used\"} / (netdata_system_ram_MiB_average{host=\"serenity\", dimension=\"used\"} + netdata_system_ram_MiB_average{host=\"serenity\", dimension=\"free\"} + netdata_system_ram_MiB_average{host=\"serenity\", dimension=\"cached\"} + netdata_system_ram_MiB_average{host=\"serenity\", dimension=\"buffers\"}))",
|
||||
"legendFormat": "serenity",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Memory Used %",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisPlacement": "auto",
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 15,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"showPoints": "never"
|
||||
},
|
||||
"unit": "percent"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 12
|
||||
},
|
||||
"id": 9,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "13.0.1",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "100 * (1 - (node_filesystem_avail_bytes{host=~\"plausible-deniability|nomad\", mountpoint=\"/\", fstype!~\"tmpfs|ramfs|overlay|squashfs|nfs4\"} / node_filesystem_size_bytes{host=~\"plausible-deniability|nomad\", mountpoint=\"/\", fstype!~\"tmpfs|ramfs|overlay|squashfs|nfs4\"}))",
|
||||
"legendFormat": "{{host}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "100 * (sum by (host) (netdata_disk_space_GiB_average{host=\"serenity\", mount_point=\"/\", dimension=\"used\"}) / (sum by (host) (netdata_disk_space_GiB_average{host=\"serenity\", mount_point=\"/\", dimension=\"used\"}) + sum by (host) (netdata_disk_space_GiB_average{host=\"serenity\", mount_point=\"/\", dimension=\"avail\"})))",
|
||||
"legendFormat": "{{host}}",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Root Filesystem Used %",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisPlacement": "auto",
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"showPoints": "never"
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 12
|
||||
},
|
||||
"id": 10,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "13.0.1",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "node_load1{host=~\"plausible-deniability|nomad\"}",
|
||||
"legendFormat": "{{host}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "netdata_system_load_load_average{host=\"serenity\", dimension=\"load1\"}",
|
||||
"legendFormat": "{{host}}",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Load 1m",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisPlacement": "auto",
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"showPoints": "never"
|
||||
},
|
||||
"unit": "Bps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 20
|
||||
},
|
||||
"id": 11,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "13.0.1",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (host) (rate(node_network_receive_bytes_total{host=~\"plausible-deniability|nomad\", device!~\"lo|veth.*|br-.*|docker.*\"}[5m]))",
|
||||
"legendFormat": "{{host}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (host) (netdata_net_net_kilobits_persec_average{host=\"serenity\", interface_type=\"real\", dimension=\"received\"}) * 125",
|
||||
"legendFormat": "{{host}}",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Network Receive",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"max": 100,
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 75
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 90
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "percent"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 20
|
||||
},
|
||||
"id": 12,
|
||||
"options": {
|
||||
"displayMode": "gradient",
|
||||
"minVizHeight": 16,
|
||||
"minVizWidth": 8,
|
||||
"orientation": "horizontal",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showUnfilled": true
|
||||
},
|
||||
"pluginVersion": "13.0.1",
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "topk(12, 100 * (1 - node_filesystem_avail_bytes{host=~\"plausible-deniability|nomad\", fstype!~\"tmpfs|ramfs|overlay|squashfs|nfs4\", mountpoint!~\"/var/lib/docker/.+\"} / node_filesystem_size_bytes{host=~\"plausible-deniability|nomad\", fstype!~\"tmpfs|ramfs|overlay|squashfs|nfs4\", mountpoint!~\"/var/lib/docker/.+\"}))",
|
||||
"instant": true,
|
||||
"legendFormat": "{{host}} {{mountpoint}}",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Top Filesystem Usage (PD + NOMAD)",
|
||||
"type": "bargauge"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 41,
|
||||
"tags": [
|
||||
"homelab",
|
||||
"prometheus",
|
||||
"overview"
|
||||
],
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"time": {
|
||||
"from": "now-24h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "",
|
||||
"title": "Homelab Overview",
|
||||
"uid": "homelab-overview",
|
||||
"version": 1,
|
||||
"weekStart": ""
|
||||
}
|
||||
1087
monitoring/dashboards/linux-host-drilldown.json
Normal file
1087
monitoring/dashboards/linux-host-drilldown.json
Normal file
File diff suppressed because it is too large
Load Diff
562
monitoring/dashboards/monitoring-stack.json
Normal file
562
monitoring/dashboards/monitoring-stack.json
Normal file
@@ -0,0 +1,562 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "sum(up)",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Targets Up",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 1
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 2
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 4,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "count(up) - sum(up)",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Targets Down",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 8,
|
||||
"y": 0
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "sum(prometheus_target_scrape_pool_targets)",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Scrape Targets",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "rate(prometheus_tsdb_head_samples_appended_total[5m])",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Samples / Sec",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 16,
|
||||
"y": 0
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "prometheus_tsdb_head_series",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Head Series",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 20,
|
||||
"y": 0
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "prometheus_tsdb_storage_blocks_bytes",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "TSDB Blocks",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisPlacement": "auto",
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"showPoints": "never"
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 4
|
||||
},
|
||||
"id": 7,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "up",
|
||||
"legendFormat": "{{job}} {{host}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Target Health",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisPlacement": "auto",
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"showPoints": "never"
|
||||
},
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 4
|
||||
},
|
||||
"id": 8,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "scrape_duration_seconds{job=~\"pd-node|nomad-node|serenity-netdata\"}",
|
||||
"legendFormat": "{{job}} {{host}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Scrape Duration",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisPlacement": "auto",
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"showPoints": "never"
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 12
|
||||
},
|
||||
"id": 9,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "scrape_samples_post_metric_relabeling{job=~\"pd-node|nomad-node|serenity-netdata\"}",
|
||||
"legendFormat": "{{job}} {{host}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Samples Per Scrape",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisPlacement": "auto",
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"showPoints": "never"
|
||||
},
|
||||
"unit": "ops"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 12
|
||||
},
|
||||
"id": 10,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "rate(prometheus_tsdb_head_samples_appended_total[5m])",
|
||||
"legendFormat": "samples/s",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "rate(prometheus_http_requests_total[5m])",
|
||||
"legendFormat": "http req/s",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Prometheus Throughput",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"align": "auto",
|
||||
"cellOptions": {
|
||||
"type": "auto"
|
||||
},
|
||||
"inspect": false
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 20
|
||||
},
|
||||
"id": 11,
|
||||
"options": {
|
||||
"cellHeight": "sm",
|
||||
"footer": {
|
||||
"show": false
|
||||
},
|
||||
"showHeader": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "up",
|
||||
"format": "table",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Current Target States",
|
||||
"transformations": [
|
||||
{
|
||||
"id": "organize",
|
||||
"options": {
|
||||
"excludeByName": {
|
||||
"Time": true
|
||||
},
|
||||
"indexByName": {
|
||||
"Value": 0,
|
||||
"host": 1,
|
||||
"instance": 2,
|
||||
"job": 3
|
||||
},
|
||||
"renameByName": {
|
||||
"Value": "up"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"type": "table"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 41,
|
||||
"tags": [
|
||||
"homelab",
|
||||
"prometheus",
|
||||
"monitoring"
|
||||
],
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"time": {
|
||||
"from": "now-24h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "",
|
||||
"title": "Monitoring Stack",
|
||||
"uid": "monitoring-stack",
|
||||
"version": 1,
|
||||
"weekStart": ""
|
||||
}
|
||||
512
monitoring/dashboards/pd-containers.json
Normal file
512
monitoring/dashboards/pd-containers.json
Normal file
@@ -0,0 +1,512 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "count(container_start_time_seconds{job=\"pd-cadvisor\",name!=\"\"})",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Tracked Containers",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 1
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 5
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 6,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "sum(changes(container_start_time_seconds{job=\"pd-cadvisor\",name!=\"\"}[24h]) > 0)",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Containers Restarted (24h)",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes",
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 1073741824
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 2147483648
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "max(container_memory_working_set_bytes{job=\"pd-cadvisor\",name!=\"\"})",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Top Container RAM",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"decimals": 2,
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 0.7
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1.5
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 18,
|
||||
"y": 0
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "max(rate(container_cpu_usage_seconds_total{job=\"pd-cadvisor\",name!=\"\"}[5m]))",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Top Container CPU/sec",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes",
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 4
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"displayMode": "gradient",
|
||||
"orientation": "horizontal",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showUnfilled": true,
|
||||
"sizing": "auto"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "topk(12, container_memory_working_set_bytes{job=\"pd-cadvisor\",name!=\"\"})",
|
||||
"legendFormat": "{{name}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Top Container RAM (PD)",
|
||||
"type": "bargauge"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"decimals": 3,
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 4
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"displayMode": "gradient",
|
||||
"orientation": "horizontal",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showUnfilled": true,
|
||||
"sizing": "auto"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "topk(12, rate(container_cpu_usage_seconds_total{job=\"pd-cadvisor\",name!=\"\"}[5m]))",
|
||||
"legendFormat": "{{name}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Top Container CPU/sec (PD)",
|
||||
"type": "bargauge"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 15,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 2,
|
||||
"pointSize": 4,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 13
|
||||
},
|
||||
"id": 7,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "topk(8, container_memory_working_set_bytes{job=\"pd-cadvisor\",name!=\"\"})",
|
||||
"legendFormat": "{{name}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Top RAM Consumers Over Time (PD)",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 13
|
||||
},
|
||||
"id": 8,
|
||||
"options": {
|
||||
"showHeader": true,
|
||||
"sortBy": [
|
||||
{
|
||||
"desc": true,
|
||||
"displayName": "Value"
|
||||
}
|
||||
]
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "sort_desc(container_start_time_seconds{job=\"pd-cadvisor\",name!=\"\"})",
|
||||
"format": "table",
|
||||
"instant": true,
|
||||
"legendFormat": "{{name}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Container Start Times (PD)",
|
||||
"transformations": [
|
||||
{
|
||||
"id": "organize",
|
||||
"options": {
|
||||
"excludeByName": {
|
||||
"Time": true,
|
||||
"__name__": true,
|
||||
"container_label_com_docker_compose_config_hash": true,
|
||||
"container_label_com_docker_compose_container_number": true,
|
||||
"container_label_com_docker_compose_image": true,
|
||||
"container_label_com_docker_compose_oneoff": true,
|
||||
"container_label_com_docker_compose_project_config_files": true,
|
||||
"container_label_com_docker_compose_project_environment_file": true,
|
||||
"container_label_com_docker_compose_project_working_dir": true,
|
||||
"container_label_com_docker_compose_replace": true,
|
||||
"container_label_com_docker_compose_version": true,
|
||||
"id": true,
|
||||
"image": true,
|
||||
"instance": true,
|
||||
"job": true
|
||||
},
|
||||
"indexByName": {
|
||||
"name": 0,
|
||||
"container_label_com_docker_compose_project": 1,
|
||||
"container_label_com_docker_compose_service": 2,
|
||||
"Value": 3
|
||||
},
|
||||
"renameByName": {
|
||||
"name": "container",
|
||||
"container_label_com_docker_compose_project": "compose_project",
|
||||
"container_label_com_docker_compose_service": "compose_service",
|
||||
"Value": "started_at_epoch"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"type": "table"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 39,
|
||||
"style": "dark",
|
||||
"tags": [
|
||||
"monitoring",
|
||||
"docker",
|
||||
"pd"
|
||||
],
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"time": {
|
||||
"from": "now-24h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "",
|
||||
"title": "PD Containers",
|
||||
"uid": "pd-containers",
|
||||
"version": 1,
|
||||
"weekStart": ""
|
||||
}
|
||||
550
monitoring/dashboards/serenity-containers.json
Normal file
550
monitoring/dashboards/serenity-containers.json
Normal file
@@ -0,0 +1,550 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "count(netdata_docker_container_state_state_average{host=\"serenity\", dimension=\"running\"} == 1)",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Running Containers",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 6,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "count(netdata_docker_container_state_state_average{host=\"serenity\", dimension=\"exited\"} == 1)",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Exited Containers",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "count(netdata_docker_container_state_state_average{host=\"serenity\", dimension=\"created\"} == 1)",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Created Containers",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 18,
|
||||
"y": 0
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "count(netdata_cgroup_mem_usage_MiB_average{host=\"serenity\", dimension=\"ram\"})",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Tracked Cgroups",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 7,
|
||||
"w": 8,
|
||||
"x": 0,
|
||||
"y": 4
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"displayLabels": [
|
||||
"name",
|
||||
"value"
|
||||
],
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"pieType": "pie",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "single"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "count by (dimension) (netdata_docker_container_state_state_average{host=\"serenity\"} == 1)",
|
||||
"legendFormat": "{{dimension}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Container States",
|
||||
"type": "piechart"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 16384,
|
||||
"min": 0,
|
||||
"unit": "decmbytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 7,
|
||||
"w": 8,
|
||||
"x": 8,
|
||||
"y": 4
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"displayMode": "gradient",
|
||||
"minVizHeight": 16,
|
||||
"minVizWidth": 8,
|
||||
"orientation": "horizontal",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showUnfilled": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "topk(12, max by (cgroup_name) (netdata_cgroup_mem_usage_MiB_average{host=\"serenity\", dimension=\"ram\"}))",
|
||||
"instant": true,
|
||||
"legendFormat": "{{cgroup_name}}",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Top Container RAM (MiB)",
|
||||
"type": "bargauge"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 100,
|
||||
"min": 0,
|
||||
"unit": "percent"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 7,
|
||||
"w": 8,
|
||||
"x": 16,
|
||||
"y": 4
|
||||
},
|
||||
"id": 7,
|
||||
"options": {
|
||||
"displayMode": "gradient",
|
||||
"minVizHeight": 16,
|
||||
"minVizWidth": 8,
|
||||
"orientation": "horizontal",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showUnfilled": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "topk(12, max by (cgroup_name) (netdata_cgroup_cpu_percentage_average{host=\"serenity\"}))",
|
||||
"instant": true,
|
||||
"legendFormat": "{{cgroup_name}}",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Top Container CPU %",
|
||||
"type": "bargauge"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"align": "auto",
|
||||
"cellOptions": {
|
||||
"type": "auto"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 11
|
||||
},
|
||||
"id": 8,
|
||||
"options": {
|
||||
"cellHeight": "sm",
|
||||
"footer": {
|
||||
"show": false
|
||||
},
|
||||
"showHeader": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "max by (container_name, image, dimension) (netdata_docker_container_state_state_average{host=\"serenity\", dimension!=\"running\"} == 1)",
|
||||
"format": "table",
|
||||
"instant": true,
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Non-Running Containers",
|
||||
"transformations": [
|
||||
{
|
||||
"id": "organize",
|
||||
"options": {
|
||||
"excludeByName": {
|
||||
"Time": true
|
||||
},
|
||||
"indexByName": {
|
||||
"Value": 0,
|
||||
"container_name": 1,
|
||||
"dimension": 2,
|
||||
"image": 3
|
||||
},
|
||||
"renameByName": {
|
||||
"Value": "state_active"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"type": "table"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisPlacement": "auto",
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"showPoints": "never"
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 20
|
||||
},
|
||||
"id": 9,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "count(netdata_docker_container_state_state_average{host=\"serenity\", dimension=\"running\"} == 1)",
|
||||
"legendFormat": "running",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "count(netdata_docker_container_state_state_average{host=\"serenity\", dimension!=\"running\"} == 1)",
|
||||
"legendFormat": "not running",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Container Count Over Time",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisPlacement": "auto",
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"showPoints": "never"
|
||||
},
|
||||
"unit": "decmbytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 20
|
||||
},
|
||||
"id": 10,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "topk(8, max by (cgroup_name) (netdata_cgroup_mem_usage_MiB_average{host=\"serenity\", dimension=\"ram\"}))",
|
||||
"legendFormat": "{{cgroup_name}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Top RAM Consumers Over Time",
|
||||
"type": "timeseries"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 41,
|
||||
"tags": [
|
||||
"serenity",
|
||||
"containers",
|
||||
"netdata"
|
||||
],
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"time": {
|
||||
"from": "now-24h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "",
|
||||
"title": "Serenity Containers",
|
||||
"uid": "serenity-containers",
|
||||
"version": 1,
|
||||
"weekStart": ""
|
||||
}
|
||||
997
monitoring/dashboards/serenity-host-health.json
Normal file
997
monitoring/dashboards/serenity-host-health.json
Normal file
@@ -0,0 +1,997 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 0,
|
||||
"links": [],
|
||||
"liveNow": false,
|
||||
"panels": [
|
||||
{
|
||||
"type": "stat",
|
||||
"title": "Serenity Up",
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "max(up{host=\"serenity\"})",
|
||||
"instant": true,
|
||||
"legendFormat": "serenity",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "stat",
|
||||
"title": "CPU Busy %",
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 4,
|
||||
"y": 0
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 70
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 90
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "sum(netdata_system_cpu_percentage_average{host=\"serenity\",dimension!~\"idle|iowait|steal|guest|guest_nice\"})",
|
||||
"instant": true,
|
||||
"legendFormat": "CPU Busy",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "stat",
|
||||
"title": "Memory Used %",
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 8,
|
||||
"y": 0
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 75
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 90
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "100 * (1 - (netdata_mem_available_MiB_average{host=\"serenity\",dimension=\"avail\"} / scalar(sum(netdata_system_ram_MiB_average{host=\"serenity\",dimension=~\"free|used|cached|buffers\"}))))",
|
||||
"instant": true,
|
||||
"legendFormat": "Memory",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "stat",
|
||||
"title": "Load 1m",
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 4
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 8
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "netdata_system_load_load_average{host=\"serenity\",dimension=\"load1\"}",
|
||||
"instant": true,
|
||||
"legendFormat": "load1",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "stat",
|
||||
"title": "Root Used %",
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 16,
|
||||
"y": 0
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 75
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 90
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "100 * sum(netdata_disk_space_GiB_average{host=\"serenity\",mount_point=\"/\",dimension=\"used\"}) / sum(netdata_disk_space_GiB_average{host=\"serenity\",mount_point=\"/\",dimension=~\"used|avail\"})",
|
||||
"instant": true,
|
||||
"legendFormat": "/",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "stat",
|
||||
"title": "Uptime",
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 20,
|
||||
"y": 0
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "value_and_name"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "netdata_system_uptime_seconds_average{host=\"serenity\",dimension=\"uptime\"}",
|
||||
"instant": true,
|
||||
"legendFormat": "uptime",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "CPU Busy Over Time",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 4
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 12,
|
||||
"gradientMode": "opacity",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 2,
|
||||
"pointSize": 3,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "sum(netdata_system_cpu_percentage_average{host=\"serenity\",dimension!~\"idle|iowait|steal|guest|guest_nice\"})",
|
||||
"legendFormat": "CPU Busy",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "sum(netdata_system_cpu_percentage_average{host=\"serenity\",dimension=\"iowait\"})",
|
||||
"legendFormat": "IO Wait",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Load Averages",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 4
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 12,
|
||||
"gradientMode": "opacity",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 2,
|
||||
"pointSize": 3,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "netdata_system_load_load_average{host=\"serenity\",dimension=\"load1\"}",
|
||||
"legendFormat": "load1",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "netdata_system_load_load_average{host=\"serenity\",dimension=\"load5\"}",
|
||||
"legendFormat": "load5",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "netdata_system_load_load_average{host=\"serenity\",dimension=\"load15\"}",
|
||||
"legendFormat": "load15",
|
||||
"range": true,
|
||||
"refId": "C"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Memory Breakdown",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 12
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "decgbytes",
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 12,
|
||||
"gradientMode": "opacity",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 2,
|
||||
"pointSize": 3,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "netdata_system_ram_MiB_average{host=\"serenity\",dimension=\"used\"} * 1048576",
|
||||
"legendFormat": "used",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "netdata_system_ram_MiB_average{host=\"serenity\",dimension=\"cached\"} * 1048576",
|
||||
"legendFormat": "cached",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "netdata_system_ram_MiB_average{host=\"serenity\",dimension=\"free\"} * 1048576",
|
||||
"legendFormat": "free",
|
||||
"range": true,
|
||||
"refId": "C"
|
||||
},
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "netdata_mem_available_MiB_average{host=\"serenity\",dimension=\"avail\"} * 1048576",
|
||||
"legendFormat": "available",
|
||||
"range": true,
|
||||
"refId": "D"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Network Throughput",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 12
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "binbps",
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 12,
|
||||
"gradientMode": "opacity",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 2,
|
||||
"pointSize": 3,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (device) (netdata_net_net_kilobits_persec_average{host=\"serenity\",interface_type=\"real\",dimension=\"received\"} * 1024)",
|
||||
"legendFormat": "{{device}} rx",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (device) (-1 * netdata_net_net_kilobits_persec_average{host=\"serenity\",interface_type=\"real\",dimension=\"sent\"} * 1024)",
|
||||
"legendFormat": "{{device}} tx",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "bargauge",
|
||||
"title": "Filesystem Used %",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 20
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"max": 100,
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 75
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 90
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"displayMode": "gradient",
|
||||
"orientation": "horizontal",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showUnfilled": true,
|
||||
"text": {}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "100 * (netdata_disk_space_GiB_average{host=\"serenity\",dimension=\"used\",filesystem!~\"tmpfs|proc|sysfs|overlay|squashfs|fusectl|cgroup2?|shm\"} / (netdata_disk_space_GiB_average{host=\"serenity\",dimension=\"used\",filesystem!~\"tmpfs|proc|sysfs|overlay|squashfs|fusectl|cgroup2?|shm\"} + netdata_disk_space_GiB_average{host=\"serenity\",dimension=\"avail\",filesystem!~\"tmpfs|proc|sysfs|overlay|squashfs|fusectl|cgroup2?|shm\"}))",
|
||||
"legendFormat": "{{mount_point}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Disk Utilization",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 20
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 12,
|
||||
"gradientMode": "opacity",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 2,
|
||||
"pointSize": 3,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "netdata_disk_util___of_time_working_average{host=\"serenity\",device_type=\"physical\"}",
|
||||
"legendFormat": "{{device}}",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "table",
|
||||
"title": "Top Disk Busy Devices",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 28
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 75
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 90
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"cellHeight": "sm",
|
||||
"footer": {
|
||||
"enablePagination": true,
|
||||
"fields": "",
|
||||
"reducer": [
|
||||
"sum"
|
||||
],
|
||||
"show": false
|
||||
},
|
||||
"showHeader": true,
|
||||
"sortBy": [
|
||||
{
|
||||
"desc": true,
|
||||
"displayName": "Value"
|
||||
}
|
||||
]
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "topk(10, netdata_disk_util___of_time_working_average{host=\"serenity\",device_type=\"physical\"})",
|
||||
"format": "table",
|
||||
"instant": true,
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Disk IO Throughput",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 28
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "Bps",
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 12,
|
||||
"gradientMode": "opacity",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 2,
|
||||
"pointSize": 3,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (device) (netdata_disk_io_KiB_persec_average{host=\"serenity\",dimension=\"reads\",device_type=\"physical\"} * 1024)",
|
||||
"legendFormat": "{{device}} read",
|
||||
"range": true,
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (device) (-1 * netdata_disk_io_KiB_persec_average{host=\"serenity\",dimension=\"writes\",device_type=\"physical\"} * 1024)",
|
||||
"legendFormat": "{{device}} write",
|
||||
"range": true,
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 41,
|
||||
"tags": [
|
||||
"homelab",
|
||||
"serenity",
|
||||
"netdata"
|
||||
],
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"time": {
|
||||
"from": "now-24h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "browser",
|
||||
"title": "Serenity Host Health",
|
||||
"uid": "serenity-host-health",
|
||||
"version": 1,
|
||||
"weekStart": ""
|
||||
}
|
||||
251
monitoring/deploy.sh
Normal file → Executable file
251
monitoring/deploy.sh
Normal file → Executable file
@@ -1,236 +1,41 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# ============================================================
|
||||
# Phase 6 — Grafana + Prometheus Deployment on PD
|
||||
# Run this script on PlausibleDeniability as root or with sudo
|
||||
# ============================================================
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$SCRIPT_DIR"
|
||||
|
||||
echo "=== Phase 6: Grafana + Prometheus ==="
|
||||
echo ""
|
||||
|
||||
# ---- 1. Scaffold directories ----
|
||||
echo "[1/6] Scaffolding directories..."
|
||||
|
||||
mkdir -p /mnt/docker-ssd/docker/compose/monitoring/provisioning/datasources
|
||||
mkdir -p /mnt/docker-ssd/docker/compose/monitoring/provisioning/dashboards
|
||||
echo "=== Monitoring stack deploy ==="
|
||||
|
||||
mkdir -p /mnt/tank/docker/appdata/prometheus
|
||||
chown 65534:65534 /mnt/tank/docker/appdata/prometheus
|
||||
|
||||
mkdir -p /mnt/tank/docker/appdata/grafana
|
||||
mkdir -p /mnt/tank/docker/appdata/grafana/dashboards
|
||||
mkdir -p /mnt/tank/docker/appdata/loki
|
||||
mkdir -p /mnt/tank/docker/appdata/promtail
|
||||
|
||||
chown 65534:65534 /mnt/tank/docker/appdata/prometheus
|
||||
chown 472:472 /mnt/tank/docker/appdata/grafana
|
||||
chown 472:472 /mnt/tank/docker/appdata/grafana/dashboards
|
||||
|
||||
echo " Done."
|
||||
if [[ ! -f .env ]]; then
|
||||
cp .env.example .env
|
||||
gf_pass="$(openssl rand -hex 16)"
|
||||
sed -i "s/^GF_ADMIN_PASS=CHANGE_ME$/GF_ADMIN_PASS=$gf_pass/" .env
|
||||
chmod 600 .env
|
||||
echo "Created .env with a random Grafana admin password."
|
||||
fi
|
||||
|
||||
# ---- 2. Write docker-compose.yaml ----
|
||||
echo "[2/6] Writing docker-compose.yaml..."
|
||||
|
||||
cat > /mnt/docker-ssd/docker/compose/monitoring/docker-compose.yaml << 'COMPOSE'
|
||||
name: monitoring
|
||||
|
||||
services:
|
||||
prometheus:
|
||||
image: prom/prometheus:v3.4.0
|
||||
container_name: prometheus
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "9090:9090"
|
||||
command:
|
||||
- '--config.file=/etc/prometheus/prometheus.yml'
|
||||
- '--storage.tsdb.path=/prometheus'
|
||||
- '--storage.tsdb.retention.time=90d'
|
||||
- '--web.enable-lifecycle'
|
||||
volumes:
|
||||
- /mnt/tank/docker/appdata/prometheus:/prometheus
|
||||
- /mnt/docker-ssd/docker/compose/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
||||
networks:
|
||||
- default
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:13.0.1
|
||||
container_name: grafana
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "3000:3000"
|
||||
environment:
|
||||
TZ: ${TZ}
|
||||
GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER}
|
||||
GF_SECURITY_ADMIN_PASSWORD: ${GF_ADMIN_PASS}
|
||||
GF_SERVER_ROOT_URL: https://${GF_HOST}/
|
||||
volumes:
|
||||
- /mnt/tank/docker/appdata/grafana:/var/lib/grafana
|
||||
- /mnt/docker-ssd/docker/compose/monitoring/provisioning:/etc/grafana/provisioning:ro
|
||||
networks:
|
||||
- pangolin
|
||||
- default
|
||||
|
||||
node-exporter:
|
||||
image: prom/node-exporter:v1.9.0
|
||||
container_name: node-exporter
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "9100:9100"
|
||||
command:
|
||||
- '--path.procfs=/host/proc'
|
||||
- '--path.sysfs=/host/sys'
|
||||
- '--path.rootfs=/rootfs'
|
||||
- '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
|
||||
volumes:
|
||||
- /proc:/host/proc:ro
|
||||
- /sys:/host/sys:ro
|
||||
- /:/rootfs:ro
|
||||
networks:
|
||||
- default
|
||||
|
||||
networks:
|
||||
pangolin:
|
||||
external: true
|
||||
COMPOSE
|
||||
|
||||
echo " Done."
|
||||
|
||||
# ---- 3. Write prometheus.yml ----
|
||||
echo "[3/6] Writing prometheus.yml..."
|
||||
|
||||
cat > /mnt/docker-ssd/docker/compose/monitoring/prometheus.yml << 'PROMCFG'
|
||||
global:
|
||||
scrape_interval: 15s
|
||||
evaluation_interval: 15s
|
||||
|
||||
scrape_configs:
|
||||
# ------- PlausibleDeniability (local) -------
|
||||
- job_name: 'pd-node'
|
||||
static_configs:
|
||||
- targets: ['node-exporter:9100']
|
||||
labels:
|
||||
host: 'plausible-deniability'
|
||||
|
||||
- job_name: 'pd-netdata'
|
||||
metrics_path: /api/v1/allmetrics
|
||||
params:
|
||||
format: [prometheus]
|
||||
static_configs:
|
||||
- targets: ['10.5.1.6:19999']
|
||||
labels:
|
||||
host: 'plausible-deniability'
|
||||
|
||||
# ------- Serenity -------
|
||||
- job_name: 'serenity-netdata'
|
||||
metrics_path: /api/v1/allmetrics
|
||||
params:
|
||||
format: [prometheus]
|
||||
static_configs:
|
||||
- targets: ['10.5.1.5:19999']
|
||||
labels:
|
||||
host: 'serenity'
|
||||
|
||||
# ------- N.O.M.A.D. -------
|
||||
# Uncomment when Netdata or node-exporter is available on N.O.M.A.D.
|
||||
# - job_name: 'nomad-netdata'
|
||||
# metrics_path: /api/v1/allmetrics
|
||||
# params:
|
||||
# format: [prometheus]
|
||||
# static_configs:
|
||||
# - targets: ['10.5.1.16:19999']
|
||||
# labels:
|
||||
# host: 'nomad'
|
||||
PROMCFG
|
||||
|
||||
echo " Done."
|
||||
|
||||
# ---- 4. Write Grafana provisioning ----
|
||||
echo "[4/6] Writing Grafana provisioning configs..."
|
||||
|
||||
cat > /mnt/docker-ssd/docker/compose/monitoring/provisioning/datasources/prometheus.yml << 'DATASRC'
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
editable: true
|
||||
DATASRC
|
||||
|
||||
cat > /mnt/docker-ssd/docker/compose/monitoring/provisioning/dashboards/dashboards.yml << 'DASHCFG'
|
||||
apiVersion: 1
|
||||
|
||||
providers:
|
||||
- name: 'default'
|
||||
orgId: 1
|
||||
folder: ''
|
||||
type: file
|
||||
disableDeletion: false
|
||||
editable: true
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
foldersFromFilesStructure: false
|
||||
DASHCFG
|
||||
|
||||
echo " Done."
|
||||
|
||||
# ---- 5. Generate .env ----
|
||||
echo "[5/6] Generating .env..."
|
||||
|
||||
GF_PASS=$(openssl rand -hex 16)
|
||||
|
||||
cat > /mnt/docker-ssd/docker/compose/monitoring/.env << ENV
|
||||
TZ=America/New_York
|
||||
|
||||
# Grafana
|
||||
GF_ADMIN_USER=admin
|
||||
GF_ADMIN_PASS=${GF_PASS}
|
||||
GF_HOST=grafana.paccoco.com
|
||||
ENV
|
||||
|
||||
echo " Done."
|
||||
echo ""
|
||||
echo " Grafana admin password: ${GF_PASS}"
|
||||
echo " (saved in /mnt/docker-ssd/docker/compose/monitoring/.env)"
|
||||
echo ""
|
||||
|
||||
# ---- 6. Validate and deploy ----
|
||||
echo "[6/6] Validating and deploying..."
|
||||
|
||||
cd /mnt/docker-ssd/docker/compose/monitoring
|
||||
docker compose --env-file .env config > /dev/null 2>&1
|
||||
echo " Compose config valid."
|
||||
if compgen -G "$SCRIPT_DIR/dashboards/*.json" >/dev/null; then
|
||||
cp "$SCRIPT_DIR"/dashboards/*.json /mnt/tank/docker/appdata/grafana/dashboards/
|
||||
chown 472:472 /mnt/tank/docker/appdata/grafana/dashboards/*.json
|
||||
fi
|
||||
|
||||
docker compose --env-file .env config >/dev/null
|
||||
docker compose --env-file .env up -d
|
||||
echo ""
|
||||
echo "=== Deployment complete ==="
|
||||
echo ""
|
||||
echo "Waiting 10 seconds for containers to start..."
|
||||
sleep 10
|
||||
|
||||
# ---- Post-deploy checks ----
|
||||
echo ""
|
||||
echo "=== Post-Deploy Verification ==="
|
||||
echo ""
|
||||
|
||||
echo "--- Container status ---"
|
||||
docker ps --filter name=prometheus --filter name=grafana --filter name=node-exporter --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
|
||||
echo ""
|
||||
|
||||
echo "--- Prometheus targets ---"
|
||||
curl -s http://10.5.1.6:9090/api/v1/targets 2>/dev/null | python3 -m json.tool 2>/dev/null | head -30 || echo "Prometheus not yet responding (may need a moment)"
|
||||
echo ""
|
||||
|
||||
echo "--- Grafana health ---"
|
||||
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" http://10.5.1.6:3000/ 2>/dev/null || echo "000")
|
||||
echo "Grafana HTTP status: ${HTTP_CODE}"
|
||||
echo ""
|
||||
|
||||
echo "--- Node exporter ---"
|
||||
curl -s http://10.5.1.6:9100/metrics 2>/dev/null | head -5 || echo "Node exporter not yet responding"
|
||||
echo ""
|
||||
|
||||
echo "=== All done! ==="
|
||||
echo ""
|
||||
echo "Next steps:"
|
||||
echo " 1. Add Pangolin resource: grafana.paccoco.com -> http://grafana:3000"
|
||||
echo " 2. Log in at https://grafana.paccoco.com with admin / ${GF_PASS}"
|
||||
echo " 3. Import dashboards: Node Exporter Full (ID 1860), Docker Stats (ID 893)"
|
||||
echo " 4. Set up Grafana -> n8n alert webhook at https://n8n.paccoco.com/webhook/grafana-alert"
|
||||
echo
|
||||
docker ps --filter name=prometheus --filter name=grafana --filter name=loki --filter name=promtail --filter name=cadvisor --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
|
||||
echo
|
||||
curl -fsS http://127.0.0.1:9090/-/ready >/dev/null && echo "Prometheus ready"
|
||||
[[ "$(docker inspect -f '{{.State.Running}}' loki 2>/dev/null)" == "true" ]] && echo "Loki running"
|
||||
curl -fsS http://127.0.0.1:9100/metrics >/dev/null && echo "node-exporter ready"
|
||||
docker exec cadvisor wget -qO- http://127.0.0.1:8080/metrics >/dev/null 2>&1 && echo "cAdvisor ready" || true
|
||||
|
||||
@@ -15,11 +15,24 @@ services:
|
||||
volumes:
|
||||
- /mnt/tank/docker/appdata/prometheus:/prometheus
|
||||
- /mnt/docker-ssd/docker/compose/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
||||
- /mnt/docker-ssd/docker/compose/monitoring/alerts.yml:/etc/prometheus/alerts.yml:ro
|
||||
networks:
|
||||
- default
|
||||
|
||||
loki:
|
||||
image: grafana/loki:latest
|
||||
container_name: loki
|
||||
restart: unless-stopped
|
||||
command:
|
||||
- '-config.file=/etc/loki/config.yml'
|
||||
volumes:
|
||||
- /mnt/tank/docker/appdata/loki:/loki
|
||||
- /mnt/docker-ssd/docker/compose/monitoring/loki-config.yml:/etc/loki/config.yml:ro
|
||||
networks:
|
||||
- default
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:13.0.1
|
||||
image: grafana/grafana:latest
|
||||
container_name: grafana
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
@@ -29,13 +42,46 @@ services:
|
||||
GF_SECURITY_ADMIN_USER: ${GF_ADMIN_USER}
|
||||
GF_SECURITY_ADMIN_PASSWORD: ${GF_ADMIN_PASS}
|
||||
GF_SERVER_ROOT_URL: https://${GF_HOST}/
|
||||
GF_AUTH_GENERIC_OAUTH_ENABLED: ${GF_AUTH_GENERIC_OAUTH_ENABLED}
|
||||
GF_AUTH_GENERIC_OAUTH_NAME: ${GF_AUTH_GENERIC_OAUTH_NAME}
|
||||
GF_AUTH_GENERIC_OAUTH_ICON: ${GF_AUTH_GENERIC_OAUTH_ICON}
|
||||
GF_AUTH_GENERIC_OAUTH_CLIENT_ID: ${GF_AUTH_GENERIC_OAUTH_CLIENT_ID}
|
||||
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: ${GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET}
|
||||
GF_AUTH_GENERIC_OAUTH_SCOPES: ${GF_AUTH_GENERIC_OAUTH_SCOPES}
|
||||
GF_AUTH_GENERIC_OAUTH_EMPTY_SCOPES: ${GF_AUTH_GENERIC_OAUTH_EMPTY_SCOPES}
|
||||
GF_AUTH_GENERIC_OAUTH_AUTH_URL: https://${AUTHELIA_HOST}/api/oidc/authorization
|
||||
GF_AUTH_GENERIC_OAUTH_TOKEN_URL: https://${AUTHELIA_HOST}/api/oidc/token
|
||||
GF_AUTH_GENERIC_OAUTH_API_URL: https://${AUTHELIA_HOST}/api/oidc/userinfo
|
||||
GF_AUTH_GENERIC_OAUTH_USE_PKCE: ${GF_AUTH_GENERIC_OAUTH_USE_PKCE}
|
||||
GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP: ${GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP}
|
||||
GF_AUTH_GENERIC_OAUTH_AUTO_LOGIN: ${GF_AUTH_GENERIC_OAUTH_AUTO_LOGIN}
|
||||
GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH: ${GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH}
|
||||
GF_AUTH_GENERIC_OAUTH_NAME_ATTRIBUTE_PATH: ${GF_AUTH_GENERIC_OAUTH_NAME_ATTRIBUTE_PATH}
|
||||
GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH: ${GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH}
|
||||
GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH: ${GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH}
|
||||
GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN: ${GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN}
|
||||
GF_AUTH_DISABLE_LOGIN_FORM: ${GF_AUTH_DISABLE_LOGIN_FORM}
|
||||
volumes:
|
||||
- /mnt/tank/docker/appdata/grafana:/var/lib/grafana
|
||||
- /mnt/docker-ssd/docker/compose/monitoring/dashboards:/var/lib/grafana/dashboards:ro
|
||||
- /mnt/docker-ssd/docker/compose/monitoring/provisioning:/etc/grafana/provisioning:ro
|
||||
networks:
|
||||
- pangolin
|
||||
- default
|
||||
|
||||
promtail:
|
||||
image: grafana/promtail:latest
|
||||
container_name: promtail
|
||||
restart: unless-stopped
|
||||
command:
|
||||
- '-config.file=/etc/promtail/config.yml'
|
||||
volumes:
|
||||
- /mnt/tank/docker/appdata/promtail:/var/lib/promtail
|
||||
- /mnt/docker-ssd/docker/compose/monitoring/promtail-config.yml:/etc/promtail/config.yml:ro
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
networks:
|
||||
- default
|
||||
|
||||
node-exporter:
|
||||
image: prom/node-exporter:v1.9.0
|
||||
container_name: node-exporter
|
||||
@@ -47,10 +93,29 @@ services:
|
||||
- '--path.sysfs=/host/sys'
|
||||
- '--path.rootfs=/rootfs'
|
||||
- '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
|
||||
- '--collector.textfile.directory=/var/lib/node_exporter/textfile_collector'
|
||||
volumes:
|
||||
- /proc:/host/proc:ro
|
||||
- /sys:/host/sys:ro
|
||||
- /:/rootfs:ro
|
||||
- /mnt/tank/docker/metrics/node-exporter:/var/lib/node_exporter/textfile_collector
|
||||
networks:
|
||||
- default
|
||||
|
||||
cadvisor:
|
||||
image: gcr.io/cadvisor/cadvisor:latest
|
||||
container_name: cadvisor
|
||||
restart: unless-stopped
|
||||
privileged: true
|
||||
command:
|
||||
- '--docker_only=true'
|
||||
- '--housekeeping_interval=30s'
|
||||
volumes:
|
||||
- /:/rootfs:ro
|
||||
- /var/run:/var/run:rw
|
||||
- /sys:/sys:ro
|
||||
- /var/lib/docker:/var/lib/docker:ro
|
||||
- /dev/disk:/dev/disk:ro
|
||||
networks:
|
||||
- default
|
||||
|
||||
|
||||
38
monitoring/loki-config.yml
Normal file
38
monitoring/loki-config.yml
Normal file
@@ -0,0 +1,38 @@
|
||||
auth_enabled: false
|
||||
|
||||
server:
|
||||
http_listen_port: 3100
|
||||
|
||||
common:
|
||||
path_prefix: /loki
|
||||
replication_factor: 1
|
||||
ring:
|
||||
kvstore:
|
||||
store: inmemory
|
||||
storage:
|
||||
filesystem:
|
||||
chunks_directory: /loki/chunks
|
||||
rules_directory: /loki/rules
|
||||
|
||||
schema_config:
|
||||
configs:
|
||||
- from: 2024-01-01
|
||||
store: tsdb
|
||||
object_store: filesystem
|
||||
schema: v13
|
||||
index:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
|
||||
limits_config:
|
||||
retention_period: 30d
|
||||
allow_structured_metadata: true
|
||||
volume_enabled: true
|
||||
|
||||
compactor:
|
||||
working_directory: /loki/compactor
|
||||
retention_enabled: true
|
||||
delete_request_store: filesystem
|
||||
|
||||
analytics:
|
||||
reporting_enabled: false
|
||||
@@ -2,6 +2,9 @@ global:
|
||||
scrape_interval: 15s
|
||||
evaluation_interval: 15s
|
||||
|
||||
rule_files:
|
||||
- /etc/prometheus/alerts.yml
|
||||
|
||||
scrape_configs:
|
||||
# ------- PlausibleDeniability (local via node-exporter) -------
|
||||
- job_name: 'pd-node'
|
||||
@@ -10,6 +13,12 @@ scrape_configs:
|
||||
labels:
|
||||
host: 'plausible-deniability'
|
||||
|
||||
- job_name: 'pd-cadvisor'
|
||||
static_configs:
|
||||
- targets: ['cadvisor:8080']
|
||||
labels:
|
||||
host: 'plausible-deniability'
|
||||
|
||||
# ------- Serenity (via Netdata Prometheus exporter) -------
|
||||
- job_name: 'serenity-netdata'
|
||||
metrics_path: /api/v1/allmetrics
|
||||
|
||||
27
monitoring/promtail-config.yml
Normal file
27
monitoring/promtail-config.yml
Normal file
@@ -0,0 +1,27 @@
|
||||
server:
|
||||
http_listen_port: 9080
|
||||
grpc_listen_port: 0
|
||||
|
||||
positions:
|
||||
filename: /var/lib/promtail/positions.yml
|
||||
|
||||
clients:
|
||||
- url: http://loki:3100/loki/api/v1/push
|
||||
|
||||
scrape_configs:
|
||||
- job_name: docker
|
||||
docker_sd_configs:
|
||||
- host: unix:///var/run/docker.sock
|
||||
refresh_interval: 15s
|
||||
relabel_configs:
|
||||
- source_labels: ['__meta_docker_container_name']
|
||||
regex: '/(.*)'
|
||||
target_label: container
|
||||
- source_labels: ['__meta_docker_container_label_com_docker_compose_project']
|
||||
target_label: compose_project
|
||||
- source_labels: ['__meta_docker_container_label_com_docker_compose_service']
|
||||
target_label: compose_service
|
||||
- source_labels: ['__meta_docker_container_log_stream']
|
||||
target_label: stream
|
||||
- replacement: plausible-deniability
|
||||
target_label: host
|
||||
@@ -7,3 +7,8 @@ datasources:
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
editable: true
|
||||
- name: Loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki:3100
|
||||
editable: true
|
||||
|
||||
Reference in New Issue
Block a user