docs: capture DNS resilience and Technitium backup sync
This commit is contained in:
Fizzlepoof
129
docs/operations/DNS_RESILIENCE.md
Normal file
129
docs/operations/DNS_RESILIENCE.md
Normal file
@@ -0,0 +1,129 @@
|
||||
# DNS Resilience
|
||||
|
||||
Current homelab DNS resilience model after the single-node outage lesson.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
When Technitium on PD died, clients that depended on that single internal resolver lost DNS. Public internet connectivity did not matter because name resolution was gone first.
|
||||
|
||||
The fix was to stop treating DNS as a single-host service and advertise multiple internal resolvers across the important LANs.
|
||||
|
||||
## Current Resolver Topology
|
||||
|
||||
| Resolver | Host | Service role | Resolver IP |
|
||||
|----------|------|--------------|-------------|
|
||||
| PD Technitium | PlausibleDeniability | primary live source of truth | `10.5.30.8` |
|
||||
| Nomad Technitium | N.O.M.A.D. | backup internal resolver | `10.5.30.9` |
|
||||
| Serenity Technitium | Serenity | backup internal resolver | `10.5.30.10` |
|
||||
| Quad9 | external | public-DNS fallback only | `9.9.9.9` |
|
||||
|
||||
All three Technitium nodes serve:
|
||||
- recursive public DNS
|
||||
- the private authoritative zone `home.paccoco.com`
|
||||
|
||||
## DHCP Advertisement Policy
|
||||
|
||||
UniFi DHCP currently advertises the Technitium trio plus an external fallback on:
|
||||
- `Management`
|
||||
- `Trusted`
|
||||
- `Servers`
|
||||
- `IoT`
|
||||
- `Camera`
|
||||
- `Old IoT`
|
||||
|
||||
Advertised order:
|
||||
1. `10.5.30.8`
|
||||
2. `10.5.30.9`
|
||||
3. `10.5.30.10`
|
||||
4. `9.9.9.9`
|
||||
|
||||
`Guest` remains public-only:
|
||||
- `9.9.9.9`
|
||||
- `1.1.1.1`
|
||||
|
||||
## What This Fixes
|
||||
|
||||
- A single Technitium host failure no longer wipes DNS for Trusted or Servers clients.
|
||||
- Internal-zone continuity now survives the loss of any one Technitium node, as long as at least one of the remaining Technitium nodes is healthy.
|
||||
- Public DNS continuity still exists even if the whole internal Technitium trio is down, because clients also have `9.9.9.9`.
|
||||
|
||||
## Important Limits
|
||||
|
||||
### External fallback is not full internal-DNS continuity
|
||||
|
||||
`9.9.9.9` is only a public fallback.
|
||||
|
||||
It can preserve general internet resolution if all internal resolvers die, but it does not replace the authoritative internal `home.paccoco.com` zone behavior.
|
||||
|
||||
Operationally: if all three Technitium nodes are down, clients may still resolve public domains, but private homelab names may be missing or semantically wrong.
|
||||
|
||||
### Macvlan health checks are deceptive
|
||||
|
||||
These Technitium nodes use dedicated macvlan IPs.
|
||||
|
||||
That means a host often cannot reliably query its own resolver IP:
|
||||
- PD may fail to query `10.5.30.8` from the PD host itself
|
||||
- NOMAD may fail to query `10.5.30.9` from NOMAD itself
|
||||
- Serenity may fail to query `10.5.30.10` from Serenity itself
|
||||
|
||||
Do not treat same-host DNS failures as proof that the resolver is down.
|
||||
|
||||
Use an off-host probe from another LAN host or client.
|
||||
|
||||
## Current Sync Model
|
||||
|
||||
PD is the authoritative Technitium source of truth.
|
||||
|
||||
Live sync runner:
|
||||
- path: `/mnt/docker-ssd/docker/compose/technitium-pilot/bin/sync_backup_nodes.sh`
|
||||
|
||||
Live schedule:
|
||||
- root cron on PD every 15 minutes
|
||||
|
||||
Log path:
|
||||
- `/mnt/tank/docker/appdata/technitium-pilot/logs/sync-backup-nodes.log`
|
||||
|
||||
The runner does all of the following:
|
||||
1. rsync the live PD Technitium config to NOMAD and Serenity
|
||||
2. exclude rebuildable cache/stats/blocklist artifacts
|
||||
3. restart the backup stacks
|
||||
4. verify a private authoritative answer for `dns.home.paccoco.com`
|
||||
5. verify a public recursive answer for `cloudflare.com`
|
||||
6. stamp each backup with `last_pd_sync_utc.txt`
|
||||
|
||||
## Validation SOP
|
||||
|
||||
When verifying DNS resilience, check from the correct network context.
|
||||
|
||||
Minimum proof:
|
||||
1. confirm UniFi DHCP advertises `10.5.30.8`, `10.5.30.9`, `10.5.30.10`, and `9.9.9.9` on the intended VLANs
|
||||
2. from an off-host client or peer host, query each internal resolver directly for:
|
||||
- `dns.home.paccoco.com A`
|
||||
- `cloudflare.com A`
|
||||
3. confirm the PD sync cron exists
|
||||
4. inspect the latest sync log for successful Nomad and Serenity verification
|
||||
|
||||
Recommended direct checks:
|
||||
- `dig @10.5.30.8 dns.home.paccoco.com A`
|
||||
- `dig @10.5.30.9 dns.home.paccoco.com A`
|
||||
- `dig @10.5.30.10 dns.home.paccoco.com A`
|
||||
- `dig @10.5.30.8 cloudflare.com A`
|
||||
- `dig @10.5.30.9 cloudflare.com A`
|
||||
- `dig @10.5.30.10 cloudflare.com A`
|
||||
|
||||
## Operator Guidance
|
||||
|
||||
- If internal-zone records change on PD, the backups should pick them up automatically within 15 minutes.
|
||||
- If the sync log shows failures, treat that as an internal-zone resilience regression even if public DNS still works.
|
||||
- If testing from the host that owns the macvlan IP fails, repeat the test from another host before escalating.
|
||||
- Keep at least one external fallback in DHCP for non-guest lanes.
|
||||
|
||||
## Source Paths
|
||||
|
||||
- Repo stack docs:
|
||||
- `technitium-pilot/README.md`
|
||||
- `technitium-nomad/README.md`
|
||||
- `technitium-serenity/README.md`
|
||||
- Architecture docs:
|
||||
- `docs/architecture/ARCHITECTURE_OVERVIEW.md`
|
||||
- `docs/architecture/NETWORKING_MODEL.md`
|
||||
Reference in New Issue
Block a user