diff --git a/docs/README.md b/docs/README.md index 16899d8..ac2c287 100644 --- a/docs/README.md +++ b/docs/README.md @@ -17,7 +17,7 @@ Central index for all homelab infrastructure documentation. |------|-------------| | [Architecture Overview](architecture/ARCHITECTURE_OVERVIEW.md) | Full stack diagram and service map | | [Services Directory](architecture/SERVICES_DIRECTORY.md) | All services with hosts, ports, and status | -| [Network Map](architecture/NETWORK.md) | VLANs, Tailscale, DNS | +| [Networking Model](architecture/NETWORKING_MODEL.md) | VLANs, firewall policy, DHCP DNS, and remote access | | [AI Services Network](architecture/AI_SERVICES_NETWORK.md) | Cross-stack Docker networking for AI services | ## Stacks @@ -35,3 +35,12 @@ Central index for all homelab infrastructure documentation. |------|-------------| | [Expansion Plan](planning/HOMELAB_EXPANSION_PLAN.md) | 6-phase buildout — all phases complete as of 2026-05-06 | | [TODO](planning/TODO.md) | Active and backlog tasks | + +## Operations + +| File | Description | +|------|-------------| +| [DNS Resilience](operations/DNS_RESILIENCE.md) | Technitium backup-node topology, PD sync automation, fallback semantics, and validation SOP | +| [Backup Policy](operations/BACKUP_POLICY.md) | Backup scope, database dumps, and restore cadence | +| [PD Backup Deployment](operations/PD_BACKUP_DEPLOYMENT.md) | Live PD backup runner deployment details | +| [Secrets Management](operations/SECRETS_MANAGEMENT.md) | `.env` handling, encrypted backups, and incident guardrails | diff --git a/docs/architecture/ARCHITECTURE_OVERVIEW.md b/docs/architecture/ARCHITECTURE_OVERVIEW.md index 5cb51c6..69e8e3a 100644 --- a/docs/architecture/ARCHITECTURE_OVERVIEW.md +++ b/docs/architecture/ARCHITECTURE_OVERVIEW.md @@ -23,8 +23,9 @@ Full homelab stack as of 2026-05-09. All 6 expansion phases are complete and dep - **UniFi policy zones:** `Internal` = Management + Trusted + Servers, `Untrusted` = IoT + Cameras + Old IoT, `Hotspot` = Guest - **Current custom firewall hardening:** `Allow Internal to Untrusted`, `Block Untrusted to Gateway Admin Surfaces`, `Allow Untrusted to DNS Resolver`, `Block Untrusted to Gateway Default Services`, `Allow Untrusted to Gateway DHCP`, `Allow Untrusted to Gateway mDNS`, plus explicit public-DNS allows used for fallback - **Current DHCP DNS policy:** `Management`, `Trusted`, `Servers`, `IoT`, `Camera`, and `Old IoT` advertise the Technitium trio `10.5.30.8`, `10.5.30.9`, and `10.5.30.10`, followed by external fallback `9.9.9.9`; `Guest` advertises `9.9.9.9` and `1.1.1.1` -- **Net effect:** every non-guest lane now has three independent internal Technitium resolvers for both public DNS and the private `home.paccoco.com` zone, plus external fallback if all homelab resolvers are down; untrusted lanes still retain DNS, DHCP, and mDNS but no longer have general gateway access or UDM Pro admin-surface access -- **Current DNS caveat:** the backup Technitium nodes on N.O.M.A.D. (`10.5.30.9`) and Serenity (`10.5.30.10`) start from a cloned PD Technitium config, so future authoritative-zone changes on PD must be resynced or automated to keep all three nodes aligned +- **Net effect:** every non-guest lane now has three independent internal Technitium resolvers for both public DNS and the private `home.paccoco.com` zone, plus external fallback for general internet name resolution if all homelab resolvers are down; untrusted lanes still retain DNS, DHCP, and mDNS but no longer have general gateway access or UDM Pro admin-surface access +- **Current sync model:** PD (`10.5.30.8`) is the authoritative Technitium source, and root cron on PD runs `/mnt/docker-ssd/docker/compose/technitium-pilot/bin/sync_backup_nodes.sh` every 15 minutes to rsync the live config to N.O.M.A.D. (`10.5.30.9`) and Serenity (`10.5.30.10`), restart those backup stacks, and verify both private and public DNS answers +- **Current DNS caveats:** `9.9.9.9` preserves public DNS continuity but is not an authoritative substitute for the internal `home.paccoco.com` zone; same-host macvlan tests are also misleading, so resolver verification must come from an off-host client or peer host - **Tailscale:** Serenity reachable at 100.94.87.79 - **NFS:** Serenity exports `/mnt/user/data` and `/mnt/user/immich` → mounted on PD at `/mnt/unraid/` via post-init script - **Docker networking:** Cross-stack communication via named external networks (`ai-services`, `ix-databases_shared-databases`) diff --git a/docs/architecture/NETWORKING_MODEL.md b/docs/architecture/NETWORKING_MODEL.md index de682f1..f848245 100644 --- a/docs/architecture/NETWORKING_MODEL.md +++ b/docs/architecture/NETWORKING_MODEL.md @@ -31,8 +31,9 @@ - **Current DHCP DNS policy:** - `Management`, `Trusted`, `Servers`, `IoT`, `Camera`, and `Old IoT` advertise `10.5.30.8`, `10.5.30.9`, `10.5.30.10`, and `9.9.9.9` via DHCP - `Guest` advertises `9.9.9.9` and `1.1.1.1` -- **Current hardening result:** every non-guest lane now has three internal Technitium resolvers for both public DNS and the private `home.paccoco.com` zone, plus an external fallback; IoT/Camera/Old IoT keep DNS, DHCP, and mDNS, but general `Untrusted -> Gateway` access and UDM Pro admin-surface access are blocked -- **Current DNS caveat:** the N.O.M.A.D. (`10.5.30.9`) and Serenity (`10.5.30.10`) Technitium backups are seeded from a cloned PD config. Any future authoritative-zone changes on PD must be resynced or automated so all three resolvers stay aligned +- **Current hardening result:** every non-guest lane now has three internal Technitium resolvers for both public DNS and the private `home.paccoco.com` zone, plus an external fallback for general internet name resolution; IoT/Camera/Old IoT keep DNS, DHCP, and mDNS, but general `Untrusted -> Gateway` access and UDM Pro admin-surface access are blocked +- **Current sync model:** PD (`10.5.30.8`) is the Technitium source of truth, and root cron on PD runs `/mnt/docker-ssd/docker/compose/technitium-pilot/bin/sync_backup_nodes.sh` every 15 minutes to push the live config to N.O.M.A.D. (`10.5.30.9`) and Serenity (`10.5.30.10`), restart those backup nodes, and verify both `dns.home.paccoco.com` and a public recursive lookup +- **Current DNS caveats:** `9.9.9.9` is only a public-DNS fallback and does not preserve authoritative internal-zone behavior by itself; same-host checks against macvlan IPs are not reliable, so health verification must come from another host or client on the LAN/VLAN - **Serenity IP:** 10.5.30.5 - **N.O.M.A.D. IP:** 10.5.30.7 - **PlausibleDeniability:** 10.5.30.6 (Servers VLAN) diff --git a/docs/operations/DNS_RESILIENCE.md b/docs/operations/DNS_RESILIENCE.md new file mode 100644 index 0000000..67fb82d --- /dev/null +++ b/docs/operations/DNS_RESILIENCE.md @@ -0,0 +1,129 @@ +# DNS Resilience + +Current homelab DNS resilience model after the single-node outage lesson. + +## Problem Statement + +When Technitium on PD died, clients that depended on that single internal resolver lost DNS. Public internet connectivity did not matter because name resolution was gone first. + +The fix was to stop treating DNS as a single-host service and advertise multiple internal resolvers across the important LANs. + +## Current Resolver Topology + +| Resolver | Host | Service role | Resolver IP | +|----------|------|--------------|-------------| +| PD Technitium | PlausibleDeniability | primary live source of truth | `10.5.30.8` | +| Nomad Technitium | N.O.M.A.D. | backup internal resolver | `10.5.30.9` | +| Serenity Technitium | Serenity | backup internal resolver | `10.5.30.10` | +| Quad9 | external | public-DNS fallback only | `9.9.9.9` | + +All three Technitium nodes serve: +- recursive public DNS +- the private authoritative zone `home.paccoco.com` + +## DHCP Advertisement Policy + +UniFi DHCP currently advertises the Technitium trio plus an external fallback on: +- `Management` +- `Trusted` +- `Servers` +- `IoT` +- `Camera` +- `Old IoT` + +Advertised order: +1. `10.5.30.8` +2. `10.5.30.9` +3. `10.5.30.10` +4. `9.9.9.9` + +`Guest` remains public-only: +- `9.9.9.9` +- `1.1.1.1` + +## What This Fixes + +- A single Technitium host failure no longer wipes DNS for Trusted or Servers clients. +- Internal-zone continuity now survives the loss of any one Technitium node, as long as at least one of the remaining Technitium nodes is healthy. +- Public DNS continuity still exists even if the whole internal Technitium trio is down, because clients also have `9.9.9.9`. + +## Important Limits + +### External fallback is not full internal-DNS continuity + +`9.9.9.9` is only a public fallback. + +It can preserve general internet resolution if all internal resolvers die, but it does not replace the authoritative internal `home.paccoco.com` zone behavior. + +Operationally: if all three Technitium nodes are down, clients may still resolve public domains, but private homelab names may be missing or semantically wrong. + +### Macvlan health checks are deceptive + +These Technitium nodes use dedicated macvlan IPs. + +That means a host often cannot reliably query its own resolver IP: +- PD may fail to query `10.5.30.8` from the PD host itself +- NOMAD may fail to query `10.5.30.9` from NOMAD itself +- Serenity may fail to query `10.5.30.10` from Serenity itself + +Do not treat same-host DNS failures as proof that the resolver is down. + +Use an off-host probe from another LAN host or client. + +## Current Sync Model + +PD is the authoritative Technitium source of truth. + +Live sync runner: +- path: `/mnt/docker-ssd/docker/compose/technitium-pilot/bin/sync_backup_nodes.sh` + +Live schedule: +- root cron on PD every 15 minutes + +Log path: +- `/mnt/tank/docker/appdata/technitium-pilot/logs/sync-backup-nodes.log` + +The runner does all of the following: +1. rsync the live PD Technitium config to NOMAD and Serenity +2. exclude rebuildable cache/stats/blocklist artifacts +3. restart the backup stacks +4. verify a private authoritative answer for `dns.home.paccoco.com` +5. verify a public recursive answer for `cloudflare.com` +6. stamp each backup with `last_pd_sync_utc.txt` + +## Validation SOP + +When verifying DNS resilience, check from the correct network context. + +Minimum proof: +1. confirm UniFi DHCP advertises `10.5.30.8`, `10.5.30.9`, `10.5.30.10`, and `9.9.9.9` on the intended VLANs +2. from an off-host client or peer host, query each internal resolver directly for: + - `dns.home.paccoco.com A` + - `cloudflare.com A` +3. confirm the PD sync cron exists +4. inspect the latest sync log for successful Nomad and Serenity verification + +Recommended direct checks: +- `dig @10.5.30.8 dns.home.paccoco.com A` +- `dig @10.5.30.9 dns.home.paccoco.com A` +- `dig @10.5.30.10 dns.home.paccoco.com A` +- `dig @10.5.30.8 cloudflare.com A` +- `dig @10.5.30.9 cloudflare.com A` +- `dig @10.5.30.10 cloudflare.com A` + +## Operator Guidance + +- If internal-zone records change on PD, the backups should pick them up automatically within 15 minutes. +- If the sync log shows failures, treat that as an internal-zone resilience regression even if public DNS still works. +- If testing from the host that owns the macvlan IP fails, repeat the test from another host before escalating. +- Keep at least one external fallback in DHCP for non-guest lanes. + +## Source Paths + +- Repo stack docs: + - `technitium-pilot/README.md` + - `technitium-nomad/README.md` + - `technitium-serenity/README.md` +- Architecture docs: + - `docs/architecture/ARCHITECTURE_OVERVIEW.md` + - `docs/architecture/NETWORKING_MODEL.md` \ No newline at end of file diff --git a/docs/servers/NOMAD.md b/docs/servers/NOMAD.md index a9ba8da..f03489c 100644 --- a/docs/servers/NOMAD.md +++ b/docs/servers/NOMAD.md @@ -199,6 +199,13 @@ These are expected to be managed by Wings/Pelican rather than a local compose fi - Topic thumbs write back to `doris-digest/data/feedback.json`. - Paperless acknowledge/dismiss state writes to `/opt/doris-dashboard/data/paperless_review_state.json` on the live host. +### Technitium Backup Resolver +- Live stack path: `/opt/technitium-nomad` +- Resolver bind IP: `10.5.30.9` +- Role: active DHCP-advertised backup Technitium resolver for the homelab +- Config is refreshed from PD's live Technitium directory by the PD-side sync runner every 15 minutes +- Same-host checks from NOMAD to `10.5.30.9` are unreliable because the resolver sits behind macvlan networking; verify from another LAN peer instead + ### MeshCore to MQTT relay - Install date: 2026-05-18 - Install root: `/opt/mctomqtt` diff --git a/docs/servers/PLAUSIBLEDENABILITY.md b/docs/servers/PLAUSIBLEDENABILITY.md index eddc2f9..a478d9e 100644 --- a/docs/servers/PLAUSIBLEDENABILITY.md +++ b/docs/servers/PLAUSIBLEDENABILITY.md @@ -53,6 +53,12 @@ Current light-tier notes: - Pangolin VPS + Newt for external services - SSH from Windows 11 +## DNS Role +- Primary Technitium source-of-truth node binds `10.5.30.8` via the `technitium-pilot` stack +- Root cron runs `/mnt/docker-ssd/docker/compose/technitium-pilot/bin/sync_backup_nodes.sh` every 15 minutes to push the live Technitium config to the backup nodes on N.O.M.A.D. (`10.5.30.9`) and Serenity (`10.5.30.10`) +- Sync logs live at `/mnt/tank/docker/appdata/technitium-pilot/logs/sync-backup-nodes.log` +- External fallback `9.9.9.9` protects public DNS continuity, but authoritative `home.paccoco.com` answers still depend on at least one Technitium node being up + ## NFS Mounts (from Serenity) - Serenity `/mnt/user/data` → `/mnt/unraid/data` (media libraries) - Serenity `/mnt/user/immich` → `/mnt/unraid/immich` (Immich photo library) diff --git a/docs/servers/SERENITY.md b/docs/servers/SERENITY.md index 940dfb1..2ba1933 100644 --- a/docs/servers/SERENITY.md +++ b/docs/servers/SERENITY.md @@ -53,6 +53,15 @@ Text Embeddings Inference reranker for OpenWebUI RAG pipeline. Managed via Unraid Community Apps. See Unraid dashboard for individual service ports and status. +### Technitium Backup Resolver + +Serenity hosts the secondary backup Technitium node on `10.5.30.10`. + +- Live stack path: `/mnt/user/appdata/technitium-serenity` +- Resolver bind IP: `10.5.30.10` +- Config is refreshed from PD's live Technitium directory by the PD-side sync runner every 15 minutes +- Same-host checks from Serenity to `10.5.30.10` are unreliable because the resolver sits behind macvlan networking; verify from another LAN peer instead + ## Notes - No GPU; reranker uses CPU-only TEI image diff --git a/technitium-nomad/README.md b/technitium-nomad/README.md index c541b6a..6bfd0a7 100644 --- a/technitium-nomad/README.md +++ b/technitium-nomad/README.md @@ -9,7 +9,7 @@ Backup Technitium DNS node cloned from the PD pilot for Nomad. - Web console: `http://10.5.30.9/` - Bootstrap domain if reset from empty config: `technitium-nomad.home.paccoco.com` -This backup node is intended to replace Pi-hole fallback in DHCP resolver lists once validated. It reuses the PD Technitium config as the initial authoritative baseline. +This backup node is an active DHCP-advertised backup resolver at `10.5.30.9`. It is kept aligned from PD's live Technitium config and preserves both recursive DNS service and the private `home.paccoco.com` zone when PD is unavailable. Current mirrored baseline from Pi-hole: - blocklists: @@ -36,4 +36,4 @@ Notes: - Clients will resolve `dns.home.paccoco.com` only when they query Technitium directly, or after DHCP/cutover points them at Technitium. - Host interface parent for macvlan: `enp5s0` -- Operational note: this node starts from a cloned Technitium config copied from PD; future record changes on PD must be resynced here until replication automation exists. +- Operational note: this node is refreshed from PD by `/mnt/docker-ssd/docker/compose/technitium-pilot/bin/sync_backup_nodes.sh`, which root cron runs on PD every 15 minutes. Same-host checks from Nomad to `10.5.30.9` are unreliable because of macvlan isolation; verify from another host when testing health. diff --git a/technitium-pilot/README.md b/technitium-pilot/README.md index 5ac7f62..91dcf54 100644 --- a/technitium-pilot/README.md +++ b/technitium-pilot/README.md @@ -34,5 +34,7 @@ Notes: - For Authentik native OIDC, the Technitium OAuth2 provider must have the default OpenID scope mappings attached: `openid`, `email`, and `profile`. Without them, Authentik will mint a token with no OpenID scopes and Technitium fails the `/application/o/userinfo/` call with `Scope mismatch` / `SSO authentication failed`. - The authoritative zone `home.paccoco.com` is now hosted in Technitium with `dns.home.paccoco.com -> 10.5.30.8`. - Clients will resolve `dns.home.paccoco.com` only when they query Technitium directly, or after DHCP/cutover points them at Technitium. -- `bin/sync_backup_nodes.sh` is the PD-side replication runner that pushes the live Technitium config to the Nomad (`10.5.30.9`) and Serenity (`10.5.30.10`) backup nodes, then recycles those backup stacks and verifies DNS answers. -- Preferred automation model: root cron on PD every 15 minutes, logging to `/mnt/tank/docker/appdata/technitium-pilot/logs/sync-backup-nodes.log`. +- `bin/sync_backup_nodes.sh` is the PD-side replication runner that pushes the live Technitium config to the Nomad (`10.5.30.9`) and Serenity (`10.5.30.10`) backup nodes, then recycles those backup stacks and verifies both a private authoritative answer and a public recursive answer. +- Active automation model: root cron on PD runs the sync every 15 minutes from `/mnt/docker-ssd/docker/compose/technitium-pilot`, logging to `/mnt/tank/docker/appdata/technitium-pilot/logs/sync-backup-nodes.log`. +- External fallback `9.9.9.9` keeps general internet DNS available if the Technitium trio is down, but it is not a substitute for the private `home.paccoco.com` zone. +- Because the pilot and backup nodes use macvlan IPs, same-host DNS probes can fail even when the resolver is healthy for real clients; verify from another LAN host when checking node health. diff --git a/technitium-serenity/README.md b/technitium-serenity/README.md index 5a7a4cd..2ffb649 100644 --- a/technitium-serenity/README.md +++ b/technitium-serenity/README.md @@ -9,7 +9,7 @@ Backup Technitium DNS node cloned from the PD pilot for Serenity. - Web console: `http://10.5.30.10/` - Bootstrap domain if reset from empty config: `technitium-serenity.home.paccoco.com` -This backup node is intended to replace Pi-hole fallback in DHCP resolver lists once validated. It reuses the PD Technitium config as the initial authoritative baseline. +This backup node is an active DHCP-advertised backup resolver at `10.5.30.10`. It is kept aligned from PD's live Technitium config and preserves both recursive DNS service and the private `home.paccoco.com` zone when PD is unavailable. Current mirrored baseline from Pi-hole: - blocklists: @@ -36,4 +36,4 @@ Notes: - Clients will resolve `dns.home.paccoco.com` only when they query Technitium directly, or after DHCP/cutover points them at Technitium. - Host interface parent for macvlan: `br0` -- Operational note: this node starts from a cloned Technitium config copied from PD; future record changes on PD must be resynced here until replication automation exists. +- Operational note: this node is refreshed from PD by `/mnt/docker-ssd/docker/compose/technitium-pilot/bin/sync_backup_nodes.sh`, which root cron runs on PD every 15 minutes. Same-host checks from Serenity to `10.5.30.10` are unreliable because of macvlan isolation; verify from another host when testing health.