Sync network redesign repo state and cutover artifacts

This commit is contained in:
Fizzlepoof
2026-05-22 20:16:21 +00:00
parent 4b1e3d4061
commit 8c72a0c287
59 changed files with 5389 additions and 244 deletions

View File

@@ -94,7 +94,7 @@ sudo docker logs --tail 60 kima
## Post-deploy checks
1. Open `http://10.5.1.6:3333`
1. Open `http://10.5.30.6:3333`
2. Confirm the app starts and initial setup loads
3. Confirm `/music` is visible to Kima
4. Confirm Kima creates and uses `/data` persistence correctly
@@ -103,7 +103,7 @@ sudo docker logs --tail 60 kima
Verified on 2026-05-15:
- container `kima` running and healthy
- `http://10.5.1.6:3333/api/health` returned status OK
- `http://10.5.30.6:3333/api/health` returned status OK
- live mount narrowed to `/mnt/unraid/data/media/music:/music`
## Optional integrations / secrets

View File

@@ -1,23 +1,25 @@
# Pi-hole Deployment Plan
**Status:** PD primary and NOMAD backup are live and working; DNS clients are pointed at VIP `10.5.1.53`. RPi4 remains optional future BACKUP2 work.
**Status:** Historical Trusted-VLAN design. PD primary is still valid, but after NOMAD moved to Servers (`10.5.30.7`) its participation in the old Trusted-side VIP `10.5.30.53` was disabled to restore correct routing. Do not reuse this document for NOMAD as-is until a same-subnet HA redesign exists.
**Architecture:** 3-node HA cluster with Keepalived VIP, Unbound recursive DNS, Nebula Sync
> **Warning:** The examples below describe the original Trusted-side plan where NOMAD was `10.5.30.7` and shared VIP `10.5.30.53/24`. They are retained as historical deployment notes, not the current NOMAD runtime truth.
---
## Architecture Overview
```
[All LAN clients]
↓ DNS: 10.5.1.53 (Virtual IP — floats to whichever node is MASTER)
↓ DNS: 10.5.30.53 (Virtual IP — floats to whichever node is MASTER)
┌─────┴──────────────────────────────────┐
│ Keepalived VIP │
│ 10.5.1.53/24 │
│ 10.5.30.53/24 │
└──────┬──────────────┬──────────────────┘
│ │ │
PD (MASTER) N.O.M.A.D. (BACKUP1) RPi4 (BACKUP2)
10.5.1.6 10.5.1.16 10.5.1.X
10.5.30.6 10.5.30.7 10.5.1.X
priority 100 priority 90 priority 80
Pi-hole :53 Pi-hole :53 Pi-hole :53
Unbound :5335 Unbound :5335 Unbound :5335
@@ -28,11 +30,11 @@
(recursive — no third-party upstream)
```
- **Keepalived** floats the VIP `10.5.1.53` to whichever node is alive with the highest priority
- **Keepalived** floats the VIP `10.5.30.53` to whichever node is alive with the highest priority
- **Unbound** on each node acts as the local recursive resolver (replaces Cloudflare/Quad9)
- **Pi-hole** on each node forwards upstream queries to its local Unbound (`127.0.0.1#5335`)
- **Nebula Sync** (running on PD) syncs all Pi-hole configs from primary → replicas daily
- Current live replica target is NOMAD at `10.5.1.16:8954`
- Current live replica target is NOMAD at `10.5.30.7:8954`
All nodes use `network_mode: host` for Pi-hole, Unbound, and Keepalived — this is required for Keepalived to manipulate the VIP and for Pi-hole to bind cleanly to port 53.
@@ -40,7 +42,7 @@ All nodes use `network_mode: host` for Pi-hole, Unbound, and Keepalived — this
## Pre-Deploy: Network Planning
1. **Pick a free static IP for the VIP**`10.5.1.53` is recommended (memorable for DNS). Reserve it in UniFi so DHCP never assigns it.
1. **Pick a free static IP for the VIP**`10.5.30.53` is recommended (memorable for DNS). Reserve it in UniFi so DHCP never assigns it.
2. **Find the LAN interface name** on each node:
```bash
# PD / RPi4:
@@ -171,7 +173,7 @@ services:
volumes:
- ./keepalived.conf:/container/environment/01-custom/keepalived.conf
environment:
KEEPALIVED_VIRTUAL_IPS: "#PYTHON2BASH:['10.5.1.53/24']"
KEEPALIVED_VIRTUAL_IPS: "#PYTHON2BASH:['10.5.30.53/24']"
```
### keepalived.conf (PD — MASTER)
@@ -195,7 +197,7 @@ vrrp_instance VI_DNS {
}
virtual_ipaddress {
10.5.1.53/24
10.5.30.53/24
}
}
```
@@ -323,7 +325,7 @@ vrrp_instance VI_DNS {
}
virtual_ipaddress {
10.5.1.53/24
10.5.30.53/24
}
}
```
@@ -409,7 +411,7 @@ vrrp_instance VI_DNS {
}
virtual_ipaddress {
10.5.1.53/24
10.5.30.53/24
}
}
```
@@ -434,7 +436,7 @@ sudo systemctl start keepalived
restart: unless-stopped
environment:
PRIMARY: "http://127.0.0.1:8953|${PIHOLE_PASSWORD}"
REPLICAS: "http://10.5.1.16:8954|${PIHOLE_PASSWORD_NOMAD},http://10.5.1.X:8955|${PIHOLE_PASSWORD_RPI}"
REPLICAS: "http://10.5.30.7:8954|${PIHOLE_PASSWORD_NOMAD},http://10.5.1.X:8955|${PIHOLE_PASSWORD_RPI}"
RUN_GRAVITY: "true"
FULL_SYNC: "false"
CRON: "0 3 * * *"
@@ -461,7 +463,7 @@ sudo systemctl start keepalived
Point **only the VIP** as DNS:
1. UniFi Network → Settings → Networks → **LAN** → DHCP Name Server: **Manual**
2. **DNS Server 1:** `10.5.1.53` (VIP — floats between nodes)
2. **DNS Server 1:** `10.5.30.53` (VIP — floats between nodes)
3. **DNS Server 2:** *(leave blank or set a public fallback like `1.1.1.1` only if you want internet DNS as last resort)*
4. Save and renew DHCP leases
@@ -472,9 +474,9 @@ Point **only the VIP** as DNS:
## Current live endpoints
- **PD admin UI:** `http://10.5.1.6:8953/admin/`
- **NOMAD admin UI:** `http://10.5.1.16:8954/admin/`
- **VIP for DNS clients:** `10.5.1.53`
- **PD admin UI:** `http://10.5.30.6:8953/admin/`
- **NOMAD admin UI:** `http://10.5.30.7:8954/admin/`
- **VIP for DNS clients:** `10.5.30.53`
## Post-Deploy
@@ -482,11 +484,11 @@ Point **only the VIP** as DNS:
```bash
# Check which node holds the VIP
ip addr show | grep 10.5.1.53 # Run on each node — only MASTER shows it
ip addr show | grep 10.5.30.53 # Run on each node — only MASTER shows it
# Test DNS through VIP
nslookup google.com 10.5.1.53
nslookup doubleclick.net 10.5.1.53 # Should return 0.0.0.0 (blocked)
nslookup google.com 10.5.30.53
nslookup doubleclick.net 10.5.30.53 # Should return 0.0.0.0 (blocked)
```
### Test failover
@@ -495,7 +497,7 @@ nslookup doubleclick.net 10.5.1.53 # Should return 0.0.0.0 (blocked)
# On PD, bring down keepalived
sudo docker stop keepalived
# Wait ~3 seconds, then check from another machine:
nslookup google.com 10.5.1.53 # Should still work — N.O.M.A.D. took over VIP
nslookup google.com 10.5.30.53 # Should still work — N.O.M.A.D. took over VIP
# Restore
sudo docker start keepalived
```
@@ -516,8 +518,8 @@ Add in Pi-hole (primary) → Local DNS → DNS Records:
| Domain | IP |
|--------|----|
| pd.lan | 10.5.1.6 |
| nomad.lan | 10.5.1.16 |
| pd.lan | 10.5.30.6 |
| nomad.lan | 10.5.30.7 |
| rocinante.lan | 10.5.1.112 |
| rpi4.lan | 10.5.1.X |
@@ -527,8 +529,8 @@ Nebula Sync will propagate these to all replicas.
| Subdomain | Target |
|-----------|--------|
| `pihole.paccoco.com` | `http://10.5.1.6:8953` (primary) |
| `pihole2.paccoco.com` | `http://10.5.1.16:8954` (N.O.M.A.D.) |
| `pihole.paccoco.com` | `http://10.5.30.6:8953` (primary) |
| `pihole2.paccoco.com` | `http://10.5.30.7:8954` (N.O.M.A.D.) |
Use `/admin/` on the public hostname for the Pi-hole admin UI path.

View File

@@ -88,7 +88,7 @@ Generated: 2026-05-09
### 12 — Pi-hole Stats Digest
**Trigger:** Schedule (daily at 8 AM)
**What it does:**
1. Query Pi-hole API on PD (`http://10.5.1.6:8953/api/stats/summary`)
1. Query Pi-hole API on PD (`http://10.5.30.6:8953/api/stats/summary`)
2. Query secondary Pi-hole on Serenity
3. LiteLLM summarizes notable blocked domains and compares primary vs secondary query load
4. Push digest to Gotify (Pi-hole app) with 24h block count, top blocked domains, and any unusual spikes
@@ -136,7 +136,7 @@ Generated: 2026-05-09
### 16 — NFS Mount Watchdog + Auto-Heal
**Trigger:** Schedule (every 5 min)
**What it does:**
1. SSH into PD (10.5.1.6) and probe each expected NFS mount path with `mountpoint -q`:
1. SSH into PD (10.5.30.6) and probe each expected NFS mount path with `mountpoint -q`:
- `/mnt/unraid/data/media` (Plex, Audiobookshelf)
- `/mnt/unraid/data/photos` (Immich)
- Any other mounts defined in the mount script
@@ -153,7 +153,7 @@ Generated: 2026-05-09
- Escalation: priority 9, list which paths are still unavailable
**Implementation notes:**
- Uses n8n SSH node connecting to `10.5.1.6` with an SSH credential (key-based auth from n8n container)
- Uses n8n SSH node connecting to `10.5.30.6` with an SSH credential (key-based auth from n8n container)
- SSH node runs a single compound command: `mountpoint -q /mnt/unraid/data/media && echo MEDIA_OK || echo MEDIA_MISSING` etc.
- Parse output in a Code node to determine which mounts failed
- Use separate SSH nodes for: (1) running the mount script, (2) restarting containers — keeps the flow readable

View File

@@ -5,7 +5,7 @@
- [x] Regenerate N.O.M.A.D. Newt secret (was exposed in chat) — completed 2026-05-15
## Infrastructure
- [x] Deploy Pi-hole HA on PD + N.O.M.A.D. with VIP `10.5.1.53` — completed 2026-05-15; see docs/planning/DEPLOY_PIHOLE.md
- [x] Deploy Pi-hole HA on PD + N.O.M.A.D. with VIP `10.5.30.53` — completed 2026-05-15; historical Trusted-VLAN design, later superseded after NOMAD moved to Servers (`10.5.30.7`) and its Trusted-side Keepalived role was disabled pending a same-subnet HA redesign; see docs/planning/DEPLOY_PIHOLE.md
- PD primary live at `/mnt/docker-ssd/docker/compose/pihole`
- NOMAD backup live at `/opt/pihole-nomad`
- RPi4 remains optional future BACKUP2 work if you want a third node later