# DNS Resilience

Current homelab DNS resilience model after the single-node outage lesson.

## Problem Statement

When Technitium on PD died, clients that depended on that single internal resolver lost DNS. Public internet connectivity did not matter because name resolution was gone first.

The fix was to stop treating DNS as a single-host service and advertise multiple internal resolvers across the important LANs.

## Current Resolver Topology

| Resolver | Host | Service role | Resolver IP |
|----------|------|--------------|-------------|
| PD Technitium | PlausibleDeniability | primary live source of truth | `10.5.30.8` |
| Nomad Technitium | N.O.M.A.D. | backup internal resolver | `10.5.30.9` |
| Serenity Technitium | Serenity | backup internal resolver | `10.5.30.10` |
| Quad9 | external | public-DNS fallback only | `9.9.9.9` |

All three Technitium nodes serve:
- recursive public DNS
- the private authoritative zone `home.paccoco.com`

## DHCP Advertisement Policy

UniFi DHCP currently advertises the Technitium trio plus an external fallback on:
- `Management`
- `Trusted`
- `Servers`
- `IoT`
- `Camera`
- `Old IoT`

Advertised order:
1. `10.5.30.8`
2. `10.5.30.9`
3. `10.5.30.10`
4. `9.9.9.9`

`Guest` remains public-only:
- `9.9.9.9`
- `1.1.1.1`

## What This Fixes

- A single Technitium host failure no longer wipes DNS for Trusted or Servers clients.
- Internal-zone continuity now survives the loss of any one Technitium node, as long as at least one of the remaining Technitium nodes is healthy.
- Public DNS continuity still exists even if the whole internal Technitium trio is down, because clients also have `9.9.9.9`.

## Important Limits

### External fallback is not full internal-DNS continuity

`9.9.9.9` is only a public fallback.

It can preserve general internet resolution if all internal resolvers die, but it does not replace the authoritative internal `home.paccoco.com` zone behavior.

Operationally: if all three Technitium nodes are down, clients may still resolve public domains, but private homelab names may be missing or semantically wrong.

### Macvlan health checks are deceptive

These Technitium nodes use dedicated macvlan IPs.

That means a host often cannot reliably query its own resolver IP:
- PD may fail to query `10.5.30.8` from the PD host itself
- NOMAD may fail to query `10.5.30.9` from NOMAD itself
- Serenity may fail to query `10.5.30.10` from Serenity itself

Do not treat same-host DNS failures as proof that the resolver is down.

Use an off-host probe from another LAN host or client.

## Current Sync Model

PD is the authoritative Technitium source of truth.

Live sync runner:
- path: `/mnt/docker-ssd/docker/compose/technitium-pilot/bin/sync_backup_nodes.sh`

Live schedule:
- root cron on PD every 15 minutes

Log path:
- `/mnt/tank/docker/appdata/technitium-pilot/logs/sync-backup-nodes.log`

The runner does all of the following:
1. rsync the live PD Technitium config to NOMAD and Serenity
2. exclude rebuildable cache/stats/blocklist artifacts
3. restart the backup stacks
4. verify a private authoritative answer for `dns.home.paccoco.com`
5. verify a public recursive answer for `cloudflare.com`
6. stamp each backup with `last_pd_sync_utc.txt`

## Validation SOP

When verifying DNS resilience, check from the correct network context.

Minimum proof:
1. confirm UniFi DHCP advertises `10.5.30.8`, `10.5.30.9`, `10.5.30.10`, and `9.9.9.9` on the intended VLANs
2. from an off-host client or peer host, query each internal resolver directly for:
   - `dns.home.paccoco.com A`
   - `cloudflare.com A`
3. confirm the PD sync cron exists
4. inspect the latest sync log for successful Nomad and Serenity verification

Recommended direct checks:
- `dig @10.5.30.8 dns.home.paccoco.com A`
- `dig @10.5.30.9 dns.home.paccoco.com A`
- `dig @10.5.30.10 dns.home.paccoco.com A`
- `dig @10.5.30.8 cloudflare.com A`
- `dig @10.5.30.9 cloudflare.com A`
- `dig @10.5.30.10 cloudflare.com A`

## Operator Guidance

- If internal-zone records change on PD, the backups should pick them up automatically within 15 minutes.
- If the sync log shows failures, treat that as an internal-zone resilience regression even if public DNS still works.
- If testing from the host that owns the macvlan IP fails, repeat the test from another host before escalating.
- Keep at least one external fallback in DHCP for non-guest lanes.

## Source Paths

- Repo stack docs:
  - `technitium-pilot/README.md`
  - `technitium-nomad/README.md`
  - `technitium-serenity/README.md`
- Architecture docs:
  - `docs/architecture/ARCHITECTURE_OVERVIEW.md`
  - `docs/architecture/NETWORKING_MODEL.md`