docs: capture DNS resilience and Technitium backup sync
This commit is contained in:
Fizzlepoof
@@ -17,7 +17,7 @@ Central index for all homelab infrastructure documentation.
|
||||
|------|-------------|
|
||||
| [Architecture Overview](architecture/ARCHITECTURE_OVERVIEW.md) | Full stack diagram and service map |
|
||||
| [Services Directory](architecture/SERVICES_DIRECTORY.md) | All services with hosts, ports, and status |
|
||||
| [Network Map](architecture/NETWORK.md) | VLANs, Tailscale, DNS |
|
||||
| [Networking Model](architecture/NETWORKING_MODEL.md) | VLANs, firewall policy, DHCP DNS, and remote access |
|
||||
| [AI Services Network](architecture/AI_SERVICES_NETWORK.md) | Cross-stack Docker networking for AI services |
|
||||
|
||||
## Stacks
|
||||
@@ -35,3 +35,12 @@ Central index for all homelab infrastructure documentation.
|
||||
|------|-------------|
|
||||
| [Expansion Plan](planning/HOMELAB_EXPANSION_PLAN.md) | 6-phase buildout — all phases complete as of 2026-05-06 |
|
||||
| [TODO](planning/TODO.md) | Active and backlog tasks |
|
||||
|
||||
## Operations
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| [DNS Resilience](operations/DNS_RESILIENCE.md) | Technitium backup-node topology, PD sync automation, fallback semantics, and validation SOP |
|
||||
| [Backup Policy](operations/BACKUP_POLICY.md) | Backup scope, database dumps, and restore cadence |
|
||||
| [PD Backup Deployment](operations/PD_BACKUP_DEPLOYMENT.md) | Live PD backup runner deployment details |
|
||||
| [Secrets Management](operations/SECRETS_MANAGEMENT.md) | `.env` handling, encrypted backups, and incident guardrails |
|
||||
|
||||
@@ -23,8 +23,9 @@ Full homelab stack as of 2026-05-09. All 6 expansion phases are complete and dep
|
||||
- **UniFi policy zones:** `Internal` = Management + Trusted + Servers, `Untrusted` = IoT + Cameras + Old IoT, `Hotspot` = Guest
|
||||
- **Current custom firewall hardening:** `Allow Internal to Untrusted`, `Block Untrusted to Gateway Admin Surfaces`, `Allow Untrusted to DNS Resolver`, `Block Untrusted to Gateway Default Services`, `Allow Untrusted to Gateway DHCP`, `Allow Untrusted to Gateway mDNS`, plus explicit public-DNS allows used for fallback
|
||||
- **Current DHCP DNS policy:** `Management`, `Trusted`, `Servers`, `IoT`, `Camera`, and `Old IoT` advertise the Technitium trio `10.5.30.8`, `10.5.30.9`, and `10.5.30.10`, followed by external fallback `9.9.9.9`; `Guest` advertises `9.9.9.9` and `1.1.1.1`
|
||||
- **Net effect:** every non-guest lane now has three independent internal Technitium resolvers for both public DNS and the private `home.paccoco.com` zone, plus external fallback if all homelab resolvers are down; untrusted lanes still retain DNS, DHCP, and mDNS but no longer have general gateway access or UDM Pro admin-surface access
|
||||
- **Current DNS caveat:** the backup Technitium nodes on N.O.M.A.D. (`10.5.30.9`) and Serenity (`10.5.30.10`) start from a cloned PD Technitium config, so future authoritative-zone changes on PD must be resynced or automated to keep all three nodes aligned
|
||||
- **Net effect:** every non-guest lane now has three independent internal Technitium resolvers for both public DNS and the private `home.paccoco.com` zone, plus external fallback for general internet name resolution if all homelab resolvers are down; untrusted lanes still retain DNS, DHCP, and mDNS but no longer have general gateway access or UDM Pro admin-surface access
|
||||
- **Current sync model:** PD (`10.5.30.8`) is the authoritative Technitium source, and root cron on PD runs `/mnt/docker-ssd/docker/compose/technitium-pilot/bin/sync_backup_nodes.sh` every 15 minutes to rsync the live config to N.O.M.A.D. (`10.5.30.9`) and Serenity (`10.5.30.10`), restart those backup stacks, and verify both private and public DNS answers
|
||||
- **Current DNS caveats:** `9.9.9.9` preserves public DNS continuity but is not an authoritative substitute for the internal `home.paccoco.com` zone; same-host macvlan tests are also misleading, so resolver verification must come from an off-host client or peer host
|
||||
- **Tailscale:** Serenity reachable at 100.94.87.79
|
||||
- **NFS:** Serenity exports `/mnt/user/data` and `/mnt/user/immich` → mounted on PD at `/mnt/unraid/` via post-init script
|
||||
- **Docker networking:** Cross-stack communication via named external networks (`ai-services`, `ix-databases_shared-databases`)
|
||||
|
||||
@@ -31,8 +31,9 @@
|
||||
- **Current DHCP DNS policy:**
|
||||
- `Management`, `Trusted`, `Servers`, `IoT`, `Camera`, and `Old IoT` advertise `10.5.30.8`, `10.5.30.9`, `10.5.30.10`, and `9.9.9.9` via DHCP
|
||||
- `Guest` advertises `9.9.9.9` and `1.1.1.1`
|
||||
- **Current hardening result:** every non-guest lane now has three internal Technitium resolvers for both public DNS and the private `home.paccoco.com` zone, plus an external fallback; IoT/Camera/Old IoT keep DNS, DHCP, and mDNS, but general `Untrusted -> Gateway` access and UDM Pro admin-surface access are blocked
|
||||
- **Current DNS caveat:** the N.O.M.A.D. (`10.5.30.9`) and Serenity (`10.5.30.10`) Technitium backups are seeded from a cloned PD config. Any future authoritative-zone changes on PD must be resynced or automated so all three resolvers stay aligned
|
||||
- **Current hardening result:** every non-guest lane now has three internal Technitium resolvers for both public DNS and the private `home.paccoco.com` zone, plus an external fallback for general internet name resolution; IoT/Camera/Old IoT keep DNS, DHCP, and mDNS, but general `Untrusted -> Gateway` access and UDM Pro admin-surface access are blocked
|
||||
- **Current sync model:** PD (`10.5.30.8`) is the Technitium source of truth, and root cron on PD runs `/mnt/docker-ssd/docker/compose/technitium-pilot/bin/sync_backup_nodes.sh` every 15 minutes to push the live config to N.O.M.A.D. (`10.5.30.9`) and Serenity (`10.5.30.10`), restart those backup nodes, and verify both `dns.home.paccoco.com` and a public recursive lookup
|
||||
- **Current DNS caveats:** `9.9.9.9` is only a public-DNS fallback and does not preserve authoritative internal-zone behavior by itself; same-host checks against macvlan IPs are not reliable, so health verification must come from another host or client on the LAN/VLAN
|
||||
- **Serenity IP:** 10.5.30.5
|
||||
- **N.O.M.A.D. IP:** 10.5.30.7
|
||||
- **PlausibleDeniability:** 10.5.30.6 (Servers VLAN)
|
||||
|
||||
129
docs/operations/DNS_RESILIENCE.md
Normal file
129
docs/operations/DNS_RESILIENCE.md
Normal file
@@ -0,0 +1,129 @@
|
||||
# DNS Resilience
|
||||
|
||||
Current homelab DNS resilience model after the single-node outage lesson.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
When Technitium on PD died, clients that depended on that single internal resolver lost DNS. Public internet connectivity did not matter because name resolution was gone first.
|
||||
|
||||
The fix was to stop treating DNS as a single-host service and advertise multiple internal resolvers across the important LANs.
|
||||
|
||||
## Current Resolver Topology
|
||||
|
||||
| Resolver | Host | Service role | Resolver IP |
|
||||
|----------|------|--------------|-------------|
|
||||
| PD Technitium | PlausibleDeniability | primary live source of truth | `10.5.30.8` |
|
||||
| Nomad Technitium | N.O.M.A.D. | backup internal resolver | `10.5.30.9` |
|
||||
| Serenity Technitium | Serenity | backup internal resolver | `10.5.30.10` |
|
||||
| Quad9 | external | public-DNS fallback only | `9.9.9.9` |
|
||||
|
||||
All three Technitium nodes serve:
|
||||
- recursive public DNS
|
||||
- the private authoritative zone `home.paccoco.com`
|
||||
|
||||
## DHCP Advertisement Policy
|
||||
|
||||
UniFi DHCP currently advertises the Technitium trio plus an external fallback on:
|
||||
- `Management`
|
||||
- `Trusted`
|
||||
- `Servers`
|
||||
- `IoT`
|
||||
- `Camera`
|
||||
- `Old IoT`
|
||||
|
||||
Advertised order:
|
||||
1. `10.5.30.8`
|
||||
2. `10.5.30.9`
|
||||
3. `10.5.30.10`
|
||||
4. `9.9.9.9`
|
||||
|
||||
`Guest` remains public-only:
|
||||
- `9.9.9.9`
|
||||
- `1.1.1.1`
|
||||
|
||||
## What This Fixes
|
||||
|
||||
- A single Technitium host failure no longer wipes DNS for Trusted or Servers clients.
|
||||
- Internal-zone continuity now survives the loss of any one Technitium node, as long as at least one of the remaining Technitium nodes is healthy.
|
||||
- Public DNS continuity still exists even if the whole internal Technitium trio is down, because clients also have `9.9.9.9`.
|
||||
|
||||
## Important Limits
|
||||
|
||||
### External fallback is not full internal-DNS continuity
|
||||
|
||||
`9.9.9.9` is only a public fallback.
|
||||
|
||||
It can preserve general internet resolution if all internal resolvers die, but it does not replace the authoritative internal `home.paccoco.com` zone behavior.
|
||||
|
||||
Operationally: if all three Technitium nodes are down, clients may still resolve public domains, but private homelab names may be missing or semantically wrong.
|
||||
|
||||
### Macvlan health checks are deceptive
|
||||
|
||||
These Technitium nodes use dedicated macvlan IPs.
|
||||
|
||||
That means a host often cannot reliably query its own resolver IP:
|
||||
- PD may fail to query `10.5.30.8` from the PD host itself
|
||||
- NOMAD may fail to query `10.5.30.9` from NOMAD itself
|
||||
- Serenity may fail to query `10.5.30.10` from Serenity itself
|
||||
|
||||
Do not treat same-host DNS failures as proof that the resolver is down.
|
||||
|
||||
Use an off-host probe from another LAN host or client.
|
||||
|
||||
## Current Sync Model
|
||||
|
||||
PD is the authoritative Technitium source of truth.
|
||||
|
||||
Live sync runner:
|
||||
- path: `/mnt/docker-ssd/docker/compose/technitium-pilot/bin/sync_backup_nodes.sh`
|
||||
|
||||
Live schedule:
|
||||
- root cron on PD every 15 minutes
|
||||
|
||||
Log path:
|
||||
- `/mnt/tank/docker/appdata/technitium-pilot/logs/sync-backup-nodes.log`
|
||||
|
||||
The runner does all of the following:
|
||||
1. rsync the live PD Technitium config to NOMAD and Serenity
|
||||
2. exclude rebuildable cache/stats/blocklist artifacts
|
||||
3. restart the backup stacks
|
||||
4. verify a private authoritative answer for `dns.home.paccoco.com`
|
||||
5. verify a public recursive answer for `cloudflare.com`
|
||||
6. stamp each backup with `last_pd_sync_utc.txt`
|
||||
|
||||
## Validation SOP
|
||||
|
||||
When verifying DNS resilience, check from the correct network context.
|
||||
|
||||
Minimum proof:
|
||||
1. confirm UniFi DHCP advertises `10.5.30.8`, `10.5.30.9`, `10.5.30.10`, and `9.9.9.9` on the intended VLANs
|
||||
2. from an off-host client or peer host, query each internal resolver directly for:
|
||||
- `dns.home.paccoco.com A`
|
||||
- `cloudflare.com A`
|
||||
3. confirm the PD sync cron exists
|
||||
4. inspect the latest sync log for successful Nomad and Serenity verification
|
||||
|
||||
Recommended direct checks:
|
||||
- `dig @10.5.30.8 dns.home.paccoco.com A`
|
||||
- `dig @10.5.30.9 dns.home.paccoco.com A`
|
||||
- `dig @10.5.30.10 dns.home.paccoco.com A`
|
||||
- `dig @10.5.30.8 cloudflare.com A`
|
||||
- `dig @10.5.30.9 cloudflare.com A`
|
||||
- `dig @10.5.30.10 cloudflare.com A`
|
||||
|
||||
## Operator Guidance
|
||||
|
||||
- If internal-zone records change on PD, the backups should pick them up automatically within 15 minutes.
|
||||
- If the sync log shows failures, treat that as an internal-zone resilience regression even if public DNS still works.
|
||||
- If testing from the host that owns the macvlan IP fails, repeat the test from another host before escalating.
|
||||
- Keep at least one external fallback in DHCP for non-guest lanes.
|
||||
|
||||
## Source Paths
|
||||
|
||||
- Repo stack docs:
|
||||
- `technitium-pilot/README.md`
|
||||
- `technitium-nomad/README.md`
|
||||
- `technitium-serenity/README.md`
|
||||
- Architecture docs:
|
||||
- `docs/architecture/ARCHITECTURE_OVERVIEW.md`
|
||||
- `docs/architecture/NETWORKING_MODEL.md`
|
||||
@@ -199,6 +199,13 @@ These are expected to be managed by Wings/Pelican rather than a local compose fi
|
||||
- Topic thumbs write back to `doris-digest/data/feedback.json`.
|
||||
- Paperless acknowledge/dismiss state writes to `/opt/doris-dashboard/data/paperless_review_state.json` on the live host.
|
||||
|
||||
### Technitium Backup Resolver
|
||||
- Live stack path: `/opt/technitium-nomad`
|
||||
- Resolver bind IP: `10.5.30.9`
|
||||
- Role: active DHCP-advertised backup Technitium resolver for the homelab
|
||||
- Config is refreshed from PD's live Technitium directory by the PD-side sync runner every 15 minutes
|
||||
- Same-host checks from NOMAD to `10.5.30.9` are unreliable because the resolver sits behind macvlan networking; verify from another LAN peer instead
|
||||
|
||||
### MeshCore to MQTT relay
|
||||
- Install date: 2026-05-18
|
||||
- Install root: `/opt/mctomqtt`
|
||||
|
||||
@@ -53,6 +53,12 @@ Current light-tier notes:
|
||||
- Pangolin VPS + Newt for external services
|
||||
- SSH from Windows 11
|
||||
|
||||
## DNS Role
|
||||
- Primary Technitium source-of-truth node binds `10.5.30.8` via the `technitium-pilot` stack
|
||||
- Root cron runs `/mnt/docker-ssd/docker/compose/technitium-pilot/bin/sync_backup_nodes.sh` every 15 minutes to push the live Technitium config to the backup nodes on N.O.M.A.D. (`10.5.30.9`) and Serenity (`10.5.30.10`)
|
||||
- Sync logs live at `/mnt/tank/docker/appdata/technitium-pilot/logs/sync-backup-nodes.log`
|
||||
- External fallback `9.9.9.9` protects public DNS continuity, but authoritative `home.paccoco.com` answers still depend on at least one Technitium node being up
|
||||
|
||||
## NFS Mounts (from Serenity)
|
||||
- Serenity `/mnt/user/data` → `/mnt/unraid/data` (media libraries)
|
||||
- Serenity `/mnt/user/immich` → `/mnt/unraid/immich` (Immich photo library)
|
||||
|
||||
@@ -53,6 +53,15 @@ Text Embeddings Inference reranker for OpenWebUI RAG pipeline.
|
||||
|
||||
Managed via Unraid Community Apps. See Unraid dashboard for individual service ports and status.
|
||||
|
||||
### Technitium Backup Resolver
|
||||
|
||||
Serenity hosts the secondary backup Technitium node on `10.5.30.10`.
|
||||
|
||||
- Live stack path: `/mnt/user/appdata/technitium-serenity`
|
||||
- Resolver bind IP: `10.5.30.10`
|
||||
- Config is refreshed from PD's live Technitium directory by the PD-side sync runner every 15 minutes
|
||||
- Same-host checks from Serenity to `10.5.30.10` are unreliable because the resolver sits behind macvlan networking; verify from another LAN peer instead
|
||||
|
||||
## Notes
|
||||
|
||||
- No GPU; reranker uses CPU-only TEI image
|
||||
|
||||
@@ -9,7 +9,7 @@ Backup Technitium DNS node cloned from the PD pilot for Nomad.
|
||||
- Web console: `http://10.5.30.9/`
|
||||
- Bootstrap domain if reset from empty config: `technitium-nomad.home.paccoco.com`
|
||||
|
||||
This backup node is intended to replace Pi-hole fallback in DHCP resolver lists once validated. It reuses the PD Technitium config as the initial authoritative baseline.
|
||||
This backup node is an active DHCP-advertised backup resolver at `10.5.30.9`. It is kept aligned from PD's live Technitium config and preserves both recursive DNS service and the private `home.paccoco.com` zone when PD is unavailable.
|
||||
|
||||
Current mirrored baseline from Pi-hole:
|
||||
- blocklists:
|
||||
@@ -36,4 +36,4 @@ Notes:
|
||||
- Clients will resolve `dns.home.paccoco.com` only when they query Technitium directly, or after DHCP/cutover points them at Technitium.
|
||||
|
||||
- Host interface parent for macvlan: `enp5s0`
|
||||
- Operational note: this node starts from a cloned Technitium config copied from PD; future record changes on PD must be resynced here until replication automation exists.
|
||||
- Operational note: this node is refreshed from PD by `/mnt/docker-ssd/docker/compose/technitium-pilot/bin/sync_backup_nodes.sh`, which root cron runs on PD every 15 minutes. Same-host checks from Nomad to `10.5.30.9` are unreliable because of macvlan isolation; verify from another host when testing health.
|
||||
|
||||
@@ -34,5 +34,7 @@ Notes:
|
||||
- For Authentik native OIDC, the Technitium OAuth2 provider must have the default OpenID scope mappings attached: `openid`, `email`, and `profile`. Without them, Authentik will mint a token with no OpenID scopes and Technitium fails the `/application/o/userinfo/` call with `Scope mismatch` / `SSO authentication failed`.
|
||||
- The authoritative zone `home.paccoco.com` is now hosted in Technitium with `dns.home.paccoco.com -> 10.5.30.8`.
|
||||
- Clients will resolve `dns.home.paccoco.com` only when they query Technitium directly, or after DHCP/cutover points them at Technitium.
|
||||
- `bin/sync_backup_nodes.sh` is the PD-side replication runner that pushes the live Technitium config to the Nomad (`10.5.30.9`) and Serenity (`10.5.30.10`) backup nodes, then recycles those backup stacks and verifies DNS answers.
|
||||
- Preferred automation model: root cron on PD every 15 minutes, logging to `/mnt/tank/docker/appdata/technitium-pilot/logs/sync-backup-nodes.log`.
|
||||
- `bin/sync_backup_nodes.sh` is the PD-side replication runner that pushes the live Technitium config to the Nomad (`10.5.30.9`) and Serenity (`10.5.30.10`) backup nodes, then recycles those backup stacks and verifies both a private authoritative answer and a public recursive answer.
|
||||
- Active automation model: root cron on PD runs the sync every 15 minutes from `/mnt/docker-ssd/docker/compose/technitium-pilot`, logging to `/mnt/tank/docker/appdata/technitium-pilot/logs/sync-backup-nodes.log`.
|
||||
- External fallback `9.9.9.9` keeps general internet DNS available if the Technitium trio is down, but it is not a substitute for the private `home.paccoco.com` zone.
|
||||
- Because the pilot and backup nodes use macvlan IPs, same-host DNS probes can fail even when the resolver is healthy for real clients; verify from another LAN host when checking node health.
|
||||
|
||||
@@ -9,7 +9,7 @@ Backup Technitium DNS node cloned from the PD pilot for Serenity.
|
||||
- Web console: `http://10.5.30.10/`
|
||||
- Bootstrap domain if reset from empty config: `technitium-serenity.home.paccoco.com`
|
||||
|
||||
This backup node is intended to replace Pi-hole fallback in DHCP resolver lists once validated. It reuses the PD Technitium config as the initial authoritative baseline.
|
||||
This backup node is an active DHCP-advertised backup resolver at `10.5.30.10`. It is kept aligned from PD's live Technitium config and preserves both recursive DNS service and the private `home.paccoco.com` zone when PD is unavailable.
|
||||
|
||||
Current mirrored baseline from Pi-hole:
|
||||
- blocklists:
|
||||
@@ -36,4 +36,4 @@ Notes:
|
||||
- Clients will resolve `dns.home.paccoco.com` only when they query Technitium directly, or after DHCP/cutover points them at Technitium.
|
||||
|
||||
- Host interface parent for macvlan: `br0`
|
||||
- Operational note: this node starts from a cloned Technitium config copied from PD; future record changes on PD must be resynced here until replication automation exists.
|
||||
- Operational note: this node is refreshed from PD by `/mnt/docker-ssd/docker/compose/technitium-pilot/bin/sync_backup_nodes.sh`, which root cron runs on PD every 15 minutes. Same-host checks from Serenity to `10.5.30.10` are unreliable because of macvlan isolation; verify from another host when testing health.
|
||||
|
||||
Reference in New Issue
Block a user