truenas-stacks/docs/operations/DNS_RESILIENCE.md

DNS Resilience

Current homelab DNS resilience model after the single-node outage lesson.

Problem Statement

When Technitium on PD died, clients that depended on that single internal resolver lost DNS. Public internet connectivity did not matter because name resolution was gone first.

The fix was to stop treating DNS as a single-host service and advertise multiple internal resolvers across the important LANs.

Current Resolver Topology

Resolver	Host	Service role	Resolver IP
PD Technitium	PlausibleDeniability	primary live source of truth	10.5.30.8
Nomad Technitium	N.O.M.A.D.	backup internal resolver	10.5.30.9
Serenity Technitium	Serenity	backup internal resolver	10.5.30.10
Quad9	external	public-DNS fallback only	9.9.9.9

All three Technitium nodes serve:

• recursive public DNS
• the private authoritative zone home.paccoco.com

DHCP Advertisement Policy

UniFi DHCP currently advertises the Technitium trio plus an external fallback on:

• Management
• Trusted
• Servers
• IoT
• Camera
• Old IoT

Advertised order:

1. 10.5.30.8
2. 10.5.30.9
3. 10.5.30.10
4. 9.9.9.9

Guest remains public-only:

• 9.9.9.9
• 1.1.1.1

What This Fixes

• A single Technitium host failure no longer wipes DNS for Trusted or Servers clients.
• Internal-zone continuity now survives the loss of any one Technitium node, as long as at least one of the remaining Technitium nodes is healthy.
• Public DNS continuity still exists even if the whole internal Technitium trio is down, because clients also have 9.9.9.9.

Important Limits

External fallback is not full internal-DNS continuity

9.9.9.9 is only a public fallback.

It can preserve general internet resolution if all internal resolvers die, but it does not replace the authoritative internal home.paccoco.com zone behavior.

Operationally: if all three Technitium nodes are down, clients may still resolve public domains, but private homelab names may be missing or semantically wrong.

Macvlan health checks are deceptive

These Technitium nodes use dedicated macvlan IPs.

That means a host often cannot reliably query its own resolver IP:

• PD may fail to query 10.5.30.8 from the PD host itself
• NOMAD may fail to query 10.5.30.9 from NOMAD itself
• Serenity may fail to query 10.5.30.10 from Serenity itself

Do not treat same-host DNS failures as proof that the resolver is down.

Use an off-host probe from another LAN host or client.

Current Sync Model

PD is the authoritative Technitium source of truth.

Live sync runner:

• path: /mnt/docker-ssd/docker/compose/technitium-pilot/bin/sync_backup_nodes.sh

Live schedule:

• root cron on PD every 15 minutes

Log path:

• /mnt/tank/docker/appdata/technitium-pilot/logs/sync-backup-nodes.log

The runner does all of the following:

1. rsync the live PD Technitium config to NOMAD and Serenity
2. exclude rebuildable cache/stats/blocklist artifacts
3. restart the backup stacks
4. verify a private authoritative answer for dns.home.paccoco.com
5. verify a public recursive answer for cloudflare.com
6. stamp each backup with last_pd_sync_utc.txt

Validation SOP

When verifying DNS resilience, check from the correct network context.

Minimum proof:

1. confirm UniFi DHCP advertises 10.5.30.8, 10.5.30.9, 10.5.30.10, and 9.9.9.9 on the intended VLANs
2. from an off-host client or peer host, query each internal resolver directly for:
   • dns.home.paccoco.com A
   • cloudflare.com A
3. confirm the PD sync cron exists
4. inspect the latest sync log for successful Nomad and Serenity verification

Recommended direct checks:

• dig @10.5.30.8 dns.home.paccoco.com A
• dig @10.5.30.9 dns.home.paccoco.com A
• dig @10.5.30.10 dns.home.paccoco.com A
• dig @10.5.30.8 cloudflare.com A
• dig @10.5.30.9 cloudflare.com A
• dig @10.5.30.10 cloudflare.com A

Operator Guidance

• If internal-zone records change on PD, the backups should pick them up automatically within 15 minutes.
• If the sync log shows failures, treat that as an internal-zone resilience regression even if public DNS still works.
• If testing from the host that owns the macvlan IP fails, repeat the test from another host before escalating.
• Keep at least one external fallback in DHCP for non-guest lanes.

Source Paths

• Repo stack docs:
  • technitium-pilot/README.md
  • technitium-nomad/README.md
  • technitium-serenity/README.md
• Architecture docs:
  • docs/architecture/ARCHITECTURE_OVERVIEW.md
  • docs/architecture/NETWORKING_MODEL.md