Add Technitium backup sync automation

docs/architecture/ARCHITECTURE_OVERVIEW.md

@@ -21,7 +21,10 @@ Full homelab stack as of 2026-05-09. All 6 expansion phases are complete and dep
 - **Guest:** 10.5.90.0/24
 - **Legacy quarantine:** 192.168.1.0/24 (`Old IoT`)
 - **UniFi policy zones:** `Internal` = Management + Trusted + Servers, `Untrusted` = IoT + Cameras + Old IoT, `Hotspot` = Guest
-- **Current custom firewall hardening:** `Allow Internal to Untrusted` plus an explicit `Block Untrusted to Gateway Admin Surfaces` rule to keep IoT/Camera/Old IoT clients off the UDM Pro admin TCP ports while leaving the existing zone defaults in place
+- **Current custom firewall hardening:** `Allow Internal to Untrusted`, `Block Untrusted to Gateway Admin Surfaces`, `Allow Untrusted to DNS Resolver`, `Block Untrusted to Gateway Default Services`, `Allow Untrusted to Gateway DHCP`, `Allow Untrusted to Gateway mDNS`, plus explicit public-DNS allows used for fallback
+- **Current DHCP DNS policy:** `Management`, `Trusted`, `Servers`, `IoT`, `Camera`, and `Old IoT` advertise the Technitium trio `10.5.30.8`, `10.5.30.9`, and `10.5.30.10`, followed by external fallback `9.9.9.9`; `Guest` advertises `9.9.9.9` and `1.1.1.1`
+- **Net effect:** every non-guest lane now has three independent internal Technitium resolvers for both public DNS and the private `home.paccoco.com` zone, plus external fallback if all homelab resolvers are down; untrusted lanes still retain DNS, DHCP, and mDNS but no longer have general gateway access or UDM Pro admin-surface access
+- **Current DNS caveat:** the backup Technitium nodes on N.O.M.A.D. (`10.5.30.9`) and Serenity (`10.5.30.10`) start from a cloned PD Technitium config, so future authoritative-zone changes on PD must be resynced or automated to keep all three nodes aligned
 - **Tailscale:** Serenity reachable at 100.94.87.79
 - **NFS:** Serenity exports `/mnt/user/data` and `/mnt/user/immich` → mounted on PD at `/mnt/unraid/` via post-init script
 - **Docker networking:** Cross-stack communication via named external networks (`ai-services`, `ix-databases_shared-databases`)

docs/architecture/NETWORKING_MODEL.md

@@ -22,7 +22,17 @@
   - `Hotspot` = Guest
 - **Current custom policy additions on top of UniFi defaults:**
   - `Allow Internal to Untrusted`
-  - `Block Untrusted to Gateway Admin Surfaces` (blocks IoT/Camera/Old IoT access to the UDM Pro admin TCP ports while leaving DHCP/DNS/mDNS/internet behavior to the existing zone defaults)
+  - `Block Untrusted to Gateway Admin Surfaces`
+  - `Allow Untrusted to DNS Resolver`
+  - `Block Untrusted to Gateway Default Services`
+  - `Allow Untrusted to Gateway DHCP`
+  - `Allow Untrusted to Gateway mDNS`
+  - explicit `Allow Public DNS` fallbacks
+- **Current DHCP DNS policy:**
+  - `Management`, `Trusted`, `Servers`, `IoT`, `Camera`, and `Old IoT` advertise `10.5.30.8`, `10.5.30.9`, `10.5.30.10`, and `9.9.9.9` via DHCP
+  - `Guest` advertises `9.9.9.9` and `1.1.1.1`
+- **Current hardening result:** every non-guest lane now has three internal Technitium resolvers for both public DNS and the private `home.paccoco.com` zone, plus an external fallback; IoT/Camera/Old IoT keep DNS, DHCP, and mDNS, but general `Untrusted -> Gateway` access and UDM Pro admin-surface access are blocked
+- **Current DNS caveat:** the N.O.M.A.D. (`10.5.30.9`) and Serenity (`10.5.30.10`) Technitium backups are seeded from a cloned PD config. Any future authoritative-zone changes on PD must be resynced or automated so all three resolvers stay aligned
 - **Serenity IP:** 10.5.30.5
 - **N.O.M.A.D. IP:** 10.5.30.7
 - **PlausibleDeniability:** 10.5.30.6 (Servers VLAN)

technitium-nomad/.env.example

@@ -0,0 +1,18 @@
+TZ=America/Chicago
+LAN_INTERFACE=enp5s0
+LAN_SUBNET=10.5.30.0/24
+LAN_GATEWAY=10.5.30.1
+TECHNITIUM_BIND_IP=10.5.30.9
+APPDATA_TECHNITIUM_ROOT=/opt/technitium-nomad/data
+DNS_SERVER_DOMAIN=technitium-nomad.home.paccoco.com
+DNS_SERVER_ADMIN_PASSWORD=CHANGE_ME
+DNS_SERVER_SSO_ENABLED=false
+DNS_SERVER_SSO_AUTHORITY=
+DNS_SERVER_SSO_CLIENT_ID=
+DNS_SERVER_SSO_CLIENT_SECRET=
+DNS_SERVER_SSO_METADATA_ADDRESS=
+DNS_SERVER_SSO_SCOPES=openid profile email groups
+DNS_SERVER_SSO_ALLOW_SIGNUP=true
+DNS_SERVER_SSO_ALLOW_SIGNUP_ONLY_FOR_MAPPED_USERS=true
+DNS_SERVER_SSO_GROUP_MAP=authentik Admins=Administrators
+DNS_SERVER_BLOCK_LIST_URLS=https://big.oisd.nl/

technitium-nomad/README.md

@@ -0,0 +1,39 @@
+# Technitium backup on N.O.M.A.D.
+
+Backup Technitium DNS node cloned from the PD pilot for Nomad.
+
+- Repo stack path: `/home/fizzlepoof/repos/truenas-stacks/technitium-nomad`
+- Live stack path: `/opt/technitium-nomad`
+- Bind IP: `10.5.30.9`
+- DNS ports: `53/tcp`, `53/udp`
+- Web console: `http://10.5.30.9/`
+- Bootstrap domain if reset from empty config: `technitium-nomad.home.paccoco.com`
+
+This backup node is intended to replace Pi-hole fallback in DHCP resolver lists once validated. It reuses the PD Technitium config as the initial authoritative baseline.
+
+Current mirrored baseline from Pi-hole:
+- blocklists:
+  - `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts`
+  - `https://big.oisd.nl`
+- no custom local DNS entries detected
+- no conditional forwarding detected
+- no extra dnsmasq fragments detected
+
+Pilot-specific adaptation:
+- Production Pi-hole forwards to PD-local Unbound on `127.0.0.1:5335`.
+- Because the Technitium pilot uses a dedicated macvlan IP, that loopback-only upstream is not reachable from the pilot without changing production Unbound.
+- For pilot safety, Technitium is seeded as a standalone recursive resolver instead of reconfiguring production Unbound.
+
+Notes:
+- The stack binds only to `TECHNITIUM_BIND_IP`, so PD must have that secondary address present on `LAN_INTERFACE` before `docker compose up`.
+- The helper script adds the pilot IP alias read/write on the live host for the current boot only. That is deliberate for pilot safety.
+- Technitium reads the environment initialization only on first boot when `/etc/dns` is empty. If you need to re-seed initial settings, stop the container and clear the config directory first.
+- SSO-capable deployments should set the `DNS_SERVER_SSO_*` variables. Keep a known local admin account as rollback because Technitium may only apply initial auth/env seeding cleanly on first boot or with an empty `/etc/dns` config dir.
+- For native Authentik OIDC on this pilot, `DNS_SERVER_SSO_ALLOW_SIGNUP` must be `true` at least for the initial SSO user provisioning. Keeping `DNS_SERVER_SSO_ALLOW_SIGNUP_ONLY_FOR_MAPPED_USERS=true` preserves the gate so only users in mapped Authentik groups can auto-provision.
+- Authentik native OIDC callback for Technitium is `https://dns.paccoco.com/sso/callback`.
+- For Authentik native OIDC, the Technitium OAuth2 provider must have the default OpenID scope mappings attached: `openid`, `email`, and `profile`. Without them, Authentik will mint a token with no OpenID scopes and Technitium fails the `/application/o/userinfo/` call with `Scope mismatch` / `SSO authentication failed`.
+- The authoritative zone `home.paccoco.com` is now hosted in Technitium with `dns.home.paccoco.com -> 10.5.30.8`.
+- Clients will resolve `dns.home.paccoco.com` only when they query Technitium directly, or after DHCP/cutover points them at Technitium.
+
+- Host interface parent for macvlan: `enp5s0`
+- Operational note: this node starts from a cloned Technitium config copied from PD; future record changes on PD must be resynced here until replication automation exists.

technitium-nomad/bin/configure_technitium.py

@@ -0,0 +1,160 @@
+#!/usr/bin/env python3
+import argparse
+import json
+import os
+import sys
+from typing import Any
+from urllib import parse, request, error
+
+
+def api_request(base_url: str, method: str, path: str, *, token: str | None = None, query: dict | None = None) -> tuple[int, Any]:
+    url = base_url.rstrip('/') + path
+    if query:
+        url += '?' + parse.urlencode(query, doseq=True)
+    headers = {'Accept': 'application/json'}
+    if token:
+        headers['Authorization'] = f'Bearer {token}'
+    req = request.Request(url, headers=headers, method=method)
+    try:
+        with request.urlopen(req, timeout=30) as resp:
+            return resp.status, json.load(resp)
+    except error.HTTPError as exc:
+        body = exc.read().decode('utf-8', 'replace')
+        try:
+            payload = json.loads(body)
+        except Exception:
+            payload = body
+        return exc.code, payload
+
+
+def expect_ok(status, payload, context):
+    if status != 200 or payload.get('status') != 'ok':
+        print(f'{context} failed: http={status}', file=sys.stderr)
+        print(json.dumps(payload, indent=2, sort_keys=True), file=sys.stderr)
+        raise SystemExit(1)
+
+
+def bool_s(v: bool) -> str:
+    return 'true' if v else 'false'
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description='Idempotent Technitium baseline config for the homelab pilot')
+    ap.add_argument('--base-url', default='http://127.0.0.1:5380')
+    ap.add_argument('--admin-user', default='admin')
+    ap.add_argument('--admin-pass', default=os.environ.get('DNS_SERVER_ADMIN_PASSWORD'))
+    ap.add_argument('--server-domain', default='dns.home.paccoco.com')
+    ap.add_argument('--zone', default='home.paccoco.com')
+    ap.add_argument('--record-host', default='dns.home.paccoco.com')
+    ap.add_argument('--record-ip', default='10.5.30.8')
+    ap.add_argument('--web-port', type=int, default=80)
+    ap.add_argument('--blocklist', action='append', default=None)
+    args = ap.parse_args()
+
+    if not args.admin_pass:
+        print('Missing admin password: pass --admin-pass or set DNS_SERVER_ADMIN_PASSWORD', file=sys.stderr)
+        return 1
+
+    desired_blocklists = args.blocklist or ['https://big.oisd.nl/']
+
+    st, login = api_request(args.base_url, 'GET', '/api/user/login', query={
+        'user': args.admin_user,
+        'pass': args.admin_pass,
+        'includeInfo': 'true',
+    })
+    expect_ok(st, login, 'login')
+    token = login['token']
+
+    st, settings = api_request(args.base_url, 'GET', '/api/settings/get', token=token)
+    expect_ok(st, settings, 'settings/get')
+    resp = settings['response']
+
+    query = {
+        'dnsServerDomain': args.server_domain,
+        'webServiceLocalAddresses': ','.join(resp.get('webServiceLocalAddresses') or ['[::]']),
+        'webServiceHttpPort': str(args.web_port),
+        'webServiceEnableTls': bool_s(False),
+        'webServiceTlsPort': str(resp.get('webServiceTlsPort', 53443)),
+        'webServiceHttpToTlsRedirect': bool_s(False),
+        'webServiceUseSelfSignedTlsCertificate': bool_s(False),
+        'preferIPv6': bool_s(bool(resp.get('preferIPv6', False))),
+        'dnssecValidation': bool_s(True),
+        'enableBlocking': bool_s(True),
+        'allowTxtBlockingReport': bool_s(True),
+        'blockListUrls': ','.join(desired_blocklists),
+        'blockListUpdateIntervalHours': str(resp.get('blockListUpdateIntervalHours', 24)),
+        'logQueries': bool_s(False),
+        'allowRecursion': bool_s(True),
+        'allowRecursionOnlyForPrivateNetworks': bool_s(True),
+        'qnameMinimization': bool_s(True),
+        'serveStale': bool_s(True),
+    }
+
+    st, updated = api_request(args.base_url, 'GET', '/api/settings/set', token=token, query=query)
+    expect_ok(st, updated, 'settings/set')
+
+    verify_base_url = args.base_url
+    parsed_base = parse.urlparse(args.base_url)
+    if parsed_base.port and parsed_base.port != args.web_port:
+        host = parsed_base.hostname or '127.0.0.1'
+        verify_base_url = f'{parsed_base.scheme}://{host}:{args.web_port}'
+
+    st, zones = api_request(verify_base_url, 'GET', '/api/zones/list', token=token)
+    expect_ok(st, zones, 'zones/list')
+    zone_names = {z['name'] for z in zones['response']['zones']}
+    if args.zone not in zone_names:
+        st, created = api_request(verify_base_url, 'GET', '/api/zones/create', token=token, query={
+            'zone': args.zone,
+            'type': 'Primary',
+        })
+        expect_ok(st, created, 'zones/create')
+
+    st, current_records = api_request(verify_base_url, 'GET', '/api/zones/records/get', token=token, query={
+        'domain': args.record_host,
+        'zone': args.zone,
+        'listZone': 'false',
+    })
+    expect_ok(st, current_records, 'zones/records/get')
+    for record in current_records['response'].get('records', []):
+        if record.get('type') == 'A':
+            ip = (record.get('rData') or {}).get('ipAddress')
+            if ip and ip != args.record_ip:
+                st, deleted = api_request(verify_base_url, 'GET', '/api/zones/records/delete', token=token, query={
+                    'domain': args.record_host,
+                    'zone': args.zone,
+                    'type': 'A',
+                    'ipAddress': ip,
+                })
+                expect_ok(st, deleted, 'zones/records/delete')
+
+    st, added = api_request(verify_base_url, 'GET', '/api/zones/records/add', token=token, query={
+        'domain': args.record_host,
+        'zone': args.zone,
+        'type': 'A',
+        'ipAddress': args.record_ip,
+        'overwrite': 'true',
+        'ttl': '300',
+    })
+    expect_ok(st, added, 'zones/records/add')
+
+    st, verify = api_request(verify_base_url, 'GET', '/api/settings/get', token=token)
+    expect_ok(st, verify, 'settings/get verify')
+    v = verify['response']
+    summary = {
+        'dnsServerDomain': v.get('dnsServerDomain'),
+        'webServiceHttpPort': v.get('webServiceHttpPort'),
+        'webServiceEnableTls': v.get('webServiceEnableTls'),
+        'recursion': v.get('recursion'),
+        'dnssecValidation': v.get('dnssecValidation'),
+        'enableBlocking': v.get('enableBlocking'),
+        'allowTxtBlockingReport': v.get('allowTxtBlockingReport'),
+        'blockListUrls': v.get('blockListUrls'),
+        'zone': args.zone,
+        'record': {args.record_host: args.record_ip},
+    }
+    print(json.dumps(summary, indent=2, sort_keys=True))
+    return 0
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())

technitium-nomad/bin/prepare_pd.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+STACK_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$STACK_DIR"
+
+if [[ ! -f .env ]]; then
+  echo "Missing .env in $STACK_DIR"
+  exit 1
+fi
+
+set -a
+source .env
+set +a
+
+if [[ -z "${LAN_INTERFACE:-}" || -z "${LAN_SUBNET:-}" || -z "${LAN_GATEWAY:-}" || -z "${TECHNITIUM_BIND_IP:-}" || -z "${APPDATA_TECHNITIUM_ROOT:-}" ]]; then
+  echo "LAN_INTERFACE, LAN_SUBNET, LAN_GATEWAY, TECHNITIUM_BIND_IP, and APPDATA_TECHNITIUM_ROOT must be set in .env"
+  exit 1
+fi
+
+sudo mkdir -p "${APPDATA_TECHNITIUM_ROOT}/config" "${APPDATA_TECHNITIUM_ROOT}/logs"
+
+echo "Validating compose config"
+sudo docker compose --env-file .env config >/tmp/technitium-pilot-compose.rendered.yaml
+
+echo "Rendered compose saved to /tmp/technitium-pilot-compose.rendered.yaml"
+
+if [[ "${1:-}" == "--up" ]]; then
+  echo "Starting Technitium pilot"
+  sudo docker compose --env-file .env up -d
+fi

technitium-nomad/docker-compose.yaml

@@ -0,0 +1,48 @@
+services:
+  technitium:
+    image: docker.io/technitium/dns-server@sha256:99c250f0ee494f2f31e09a76d83f8213bc4c5d6f106b0895cd5f327518469c48
+    container_name: technitium-dns-pilot
+    hostname: technitium-dns-pilot
+    restart: unless-stopped
+    networks:
+      lan_macvlan:
+        ipv4_address: ${TECHNITIUM_BIND_IP}
+    environment:
+      DNS_SERVER_DOMAIN: ${DNS_SERVER_DOMAIN}
+      DNS_SERVER_ADMIN_PASSWORD: ${DNS_SERVER_ADMIN_PASSWORD}
+      DNS_SERVER_SSO_ENABLED: ${DNS_SERVER_SSO_ENABLED:-false}
+      DNS_SERVER_SSO_AUTHORITY: ${DNS_SERVER_SSO_AUTHORITY:-}
+      DNS_SERVER_SSO_CLIENT_ID: ${DNS_SERVER_SSO_CLIENT_ID:-}
+      DNS_SERVER_SSO_CLIENT_SECRET: ${DNS_SERVER_SSO_CLIENT_SECRET:-}
+      DNS_SERVER_SSO_METADATA_ADDRESS: ${DNS_SERVER_SSO_METADATA_ADDRESS:-}
+      DNS_SERVER_SSO_SCOPES: ${DNS_SERVER_SSO_SCOPES:-openid profile email groups}
+      DNS_SERVER_SSO_ALLOW_SIGNUP: ${DNS_SERVER_SSO_ALLOW_SIGNUP:-false}
+      DNS_SERVER_SSO_ALLOW_SIGNUP_ONLY_FOR_MAPPED_USERS: ${DNS_SERVER_SSO_ALLOW_SIGNUP_ONLY_FOR_MAPPED_USERS:-true}
+      DNS_SERVER_SSO_GROUP_MAP: ${DNS_SERVER_SSO_GROUP_MAP:-authentik Admins=Administrators}
+      DNS_SERVER_WEB_SERVICE_HTTP_PORT: "80"
+      DNS_SERVER_WEB_SERVICE_HTTPS_PORT: "443"
+      DNS_SERVER_WEB_SERVICE_ENABLE_HTTPS: "true"
+      DNS_SERVER_WEB_SERVICE_USE_SELF_SIGNED_CERT: "true"
+      DNS_SERVER_WEB_SERVICE_HTTP_TO_TLS_REDIRECT: "true"
+      DNS_SERVER_OPTIONAL_PROTOCOL_DNS_OVER_HTTP: "false"
+      DNS_SERVER_RECURSION: "AllowOnlyForPrivateNetworks"
+      DNS_SERVER_ENABLE_BLOCKING: "true"
+      DNS_SERVER_ALLOW_TXT_BLOCKING_REPORT: "true"
+      DNS_SERVER_BLOCK_LIST_URLS: ${DNS_SERVER_BLOCK_LIST_URLS}
+      DNS_SERVER_LOG_USING_LOCAL_TIME: "true"
+      DNS_SERVER_LOG_FOLDER_PATH: /var/log/technitium/dns
+      DNS_SERVER_LOG_MAX_LOG_FILE_DAYS: "30"
+      DNS_SERVER_STATS_MAX_STAT_FILE_DAYS: "30"
+    volumes:
+      - ${APPDATA_TECHNITIUM_ROOT}/config:/etc/dns
+      - ${APPDATA_TECHNITIUM_ROOT}/logs:/var/log/technitium/dns
+
+networks:
+  lan_macvlan:
+    driver: macvlan
+    driver_opts:
+      parent: ${LAN_INTERFACE}
+    ipam:
+      config:
+        - subnet: ${LAN_SUBNET}
+          gateway: ${LAN_GATEWAY}

technitium-pilot/.env.example

@@ -12,7 +12,7 @@ DNS_SERVER_SSO_CLIENT_ID=
 DNS_SERVER_SSO_CLIENT_SECRET=
 DNS_SERVER_SSO_METADATA_ADDRESS=
 DNS_SERVER_SSO_SCOPES=openid profile email groups
-DNS_SERVER_SSO_ALLOW_SIGNUP=false
+DNS_SERVER_SSO_ALLOW_SIGNUP=true
 DNS_SERVER_SSO_ALLOW_SIGNUP_ONLY_FOR_MAPPED_USERS=true
 DNS_SERVER_SSO_GROUP_MAP=authentik Admins=Administrators
 DNS_SERVER_BLOCK_LIST_URLS=https://big.oisd.nl/

technitium-pilot/README.md

@@ -29,6 +29,10 @@ Notes:
 - The helper script adds the pilot IP alias read/write on the live host for the current boot only. That is deliberate for pilot safety.
 - Technitium reads the environment initialization only on first boot when `/etc/dns` is empty. If you need to re-seed initial settings, stop the container and clear the config directory first.
 - SSO-capable deployments should set the `DNS_SERVER_SSO_*` variables. Keep a known local admin account as rollback because Technitium may only apply initial auth/env seeding cleanly on first boot or with an empty `/etc/dns` config dir.
+- For native Authentik OIDC on this pilot, `DNS_SERVER_SSO_ALLOW_SIGNUP` must be `true` at least for the initial SSO user provisioning. Keeping `DNS_SERVER_SSO_ALLOW_SIGNUP_ONLY_FOR_MAPPED_USERS=true` preserves the gate so only users in mapped Authentik groups can auto-provision.
 - Authentik native OIDC callback for Technitium is `https://dns.paccoco.com/sso/callback`.
+- For Authentik native OIDC, the Technitium OAuth2 provider must have the default OpenID scope mappings attached: `openid`, `email`, and `profile`. Without them, Authentik will mint a token with no OpenID scopes and Technitium fails the `/application/o/userinfo/` call with `Scope mismatch` / `SSO authentication failed`.
 - The authoritative zone `home.paccoco.com` is now hosted in Technitium with `dns.home.paccoco.com -> 10.5.30.8`.
 - Clients will resolve `dns.home.paccoco.com` only when they query Technitium directly, or after DHCP/cutover points them at Technitium.
+- `bin/sync_backup_nodes.sh` is the PD-side replication runner that pushes the live Technitium config to the Nomad (`10.5.30.9`) and Serenity (`10.5.30.10`) backup nodes, then recycles those backup stacks and verifies DNS answers.
+- Preferred automation model: root cron on PD every 15 minutes, logging to `/mnt/tank/docker/appdata/technitium-pilot/logs/sync-backup-nodes.log`.

technitium-pilot/bin/sync_backup_nodes.sh

@@ -0,0 +1,192 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+STACK_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
+
+PD_CONFIG_ROOT="${PD_CONFIG_ROOT:-/mnt/tank/docker/appdata/technitium-pilot/config}"
+LOG_ROOT="${LOG_ROOT:-/mnt/tank/docker/appdata/technitium-pilot/logs}"
+DIG_BIN="${DIG_BIN:-/usr/bin/dig}"
+RSYNC_BIN="${RSYNC_BIN:-/usr/bin/rsync}"
+SSH_BIN="${SSH_BIN:-/usr/bin/ssh}"
+DATE_BIN="${DATE_BIN:-/usr/bin/date}"
+MKDIR_BIN="${MKDIR_BIN:-/usr/bin/mkdir}"
+
+SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new)
+RSYNC_EXCLUDES=(--exclude cache.bin --exclude 'stats/' --exclude 'blocklists/')
+
+NOMAD_HOST="${NOMAD_HOST:-fizzlepoof@10.5.30.7}"
+NOMAD_KEY="${NOMAD_KEY:-/home/truenas_admin/.ssh/nomad_technitium_sync_ed25519}"
+NOMAD_STACK_ROOT="${NOMAD_STACK_ROOT:-/opt/technitium-nomad}"
+NOMAD_DNS_IP="${NOMAD_DNS_IP:-10.5.30.9}"
+
+SERENITY_HOST="${SERENITY_HOST:-root@10.5.30.5}"
+SERENITY_KEY="${SERENITY_KEY:-/home/truenas_admin/.ssh/serenity_backup_ed25519}"
+SERENITY_STACK_ROOT="${SERENITY_STACK_ROOT:-/mnt/user/appdata/technitium-serenity}"
+SERENITY_DNS_IP="${SERENITY_DNS_IP:-10.5.30.10}"
+
+DRY_RUN=false
+ONLY_TARGET="all"
+SKIP_RESTART=false
+
+usage() {
+  cat <<'EOF'
+Usage: sync_backup_nodes.sh [--dry-run] [--target nomad|serenity|all] [--skip-restart]
+
+Push the live PD Technitium config to the Nomad and Serenity backup Technitium nodes,
+then recycle their compose stacks and verify both private and public DNS answers.
+EOF
+}
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --dry-run)
+      DRY_RUN=true
+      ;;
+    --target)
+      ONLY_TARGET="${2:-}"
+      shift
+      ;;
+    --skip-restart)
+      SKIP_RESTART=true
+      ;;
+    -h|--help)
+      usage
+      exit 0
+      ;;
+    *)
+      echo "Unknown argument: $1" >&2
+      usage >&2
+      exit 1
+      ;;
+  esac
+  shift
+done
+
+if [[ ! -d "$PD_CONFIG_ROOT" ]]; then
+  echo "PD config root missing: $PD_CONFIG_ROOT" >&2
+  exit 1
+fi
+
+if [[ ! -x "$RSYNC_BIN" ]]; then
+  echo "rsync missing at $RSYNC_BIN" >&2
+  exit 1
+fi
+
+if [[ ! -x "$SSH_BIN" ]]; then
+  echo "ssh missing at $SSH_BIN" >&2
+  exit 1
+fi
+
+if [[ ! -x "$DIG_BIN" ]]; then
+  echo "dig missing at $DIG_BIN" >&2
+  exit 1
+fi
+
+$MKDIR_BIN -p "$LOG_ROOT"
+STAMP="$($DATE_BIN -u +%Y-%m-%dT%H:%M:%SZ)"
+
+log() {
+  printf '[%s] %s\n' "$($DATE_BIN -u +%Y-%m-%dT%H:%M:%SZ)" "$*"
+}
+
+remote_prepare_nomad() {
+  "$SSH_BIN" -i "$NOMAD_KEY" "${SSH_OPTS[@]}" "$NOMAD_HOST" \
+    "sudo -n mkdir -p '$NOMAD_STACK_ROOT/data/config' '$NOMAD_STACK_ROOT/data/logs'"
+}
+
+remote_prepare_serenity() {
+  "$SSH_BIN" -i "$SERENITY_KEY" "${SSH_OPTS[@]}" "$SERENITY_HOST" \
+    "mkdir -p '$SERENITY_STACK_ROOT/data/config' '$SERENITY_STACK_ROOT/data/logs'"
+}
+
+remote_restart_nomad() {
+  "$SSH_BIN" -i "$NOMAD_KEY" "${SSH_OPTS[@]}" "$NOMAD_HOST" \
+    "printf '%s\n' '$STAMP' | sudo -n tee '$NOMAD_STACK_ROOT/data/last_pd_sync_utc.txt' >/dev/null && \
+     cd '$NOMAD_STACK_ROOT' && \
+     sudo -n docker compose --env-file .env down && \
+     sudo -n docker compose --env-file .env up -d"
+}
+
+remote_restart_serenity() {
+  "$SSH_BIN" -i "$SERENITY_KEY" "${SSH_OPTS[@]}" "$SERENITY_HOST" \
+    "printf '%s\n' '$STAMP' > '$SERENITY_STACK_ROOT/data/last_pd_sync_utc.txt' && \
+     cd '$SERENITY_STACK_ROOT' && \
+     docker compose --env-file .env down && \
+     docker compose --env-file .env up -d"
+}
+
+verify_dns() {
+  local label="$1"
+  local dns_ip="$2"
+  local private_tmp="/tmp/technitium-sync-${label}-private.txt"
+  local public_tmp="/tmp/technitium-sync-${label}-public.txt"
+  local attempt
+  log "Verifying $label at $dns_ip"
+  for attempt in {1..10}; do
+    if "$DIG_BIN" +time=2 +tries=1 @"$dns_ip" dns.home.paccoco.com A >"$private_tmp" 2>/dev/null && \
+       "$DIG_BIN" +time=2 +tries=1 @"$dns_ip" cloudflare.com A >"$public_tmp" 2>/dev/null && \
+       grep -q 'status: NOERROR' "$private_tmp" && \
+       grep -q 'status: NOERROR' "$public_tmp" && \
+       grep -q '10.5.30.8' "$private_tmp"; then
+      return 0
+    fi
+    sleep 2
+  done
+  echo "Verification failed for $label ($dns_ip) after retries" >&2
+  cat "$private_tmp" 2>/dev/null >&2 || true
+  echo '---' >&2
+  cat "$public_tmp" 2>/dev/null >&2 || true
+  return 1
+}
+
+sync_nomad() {
+  log "Preparing Nomad target"
+  remote_prepare_nomad
+  local rsync_cmd=("$RSYNC_BIN" -a --delete "${RSYNC_EXCLUDES[@]}" -e "$SSH_BIN -i $NOMAD_KEY ${SSH_OPTS[*]}" --rsync-path="sudo -n $RSYNC_BIN" "$PD_CONFIG_ROOT/" "$NOMAD_HOST:$NOMAD_STACK_ROOT/data/config/")
+  if $DRY_RUN; then
+    rsync_cmd=("$RSYNC_BIN" -an --delete "${RSYNC_EXCLUDES[@]}" -e "$SSH_BIN -i $NOMAD_KEY ${SSH_OPTS[*]}" --rsync-path="sudo -n $RSYNC_BIN" "$PD_CONFIG_ROOT/" "$NOMAD_HOST:$NOMAD_STACK_ROOT/data/config/")
+  fi
+  log "Syncing Nomad config"
+  "${rsync_cmd[@]}"
+  if ! $DRY_RUN && ! $SKIP_RESTART; then
+    log "Recycling Nomad Technitium stack"
+    remote_restart_nomad
+    verify_dns nomad "$NOMAD_DNS_IP"
+  fi
+}
+
+sync_serenity() {
+  log "Preparing Serenity target"
+  remote_prepare_serenity
+  local rsync_cmd=("$RSYNC_BIN" -a --delete "${RSYNC_EXCLUDES[@]}" -e "$SSH_BIN -i $SERENITY_KEY ${SSH_OPTS[*]}" "$PD_CONFIG_ROOT/" "$SERENITY_HOST:$SERENITY_STACK_ROOT/data/config/")
+  if $DRY_RUN; then
+    rsync_cmd=("$RSYNC_BIN" -an --delete "${RSYNC_EXCLUDES[@]}" -e "$SSH_BIN -i $SERENITY_KEY ${SSH_OPTS[*]}" "$PD_CONFIG_ROOT/" "$SERENITY_HOST:$SERENITY_STACK_ROOT/data/config/")
+  fi
+  log "Syncing Serenity config"
+  "${rsync_cmd[@]}"
+  if ! $DRY_RUN && ! $SKIP_RESTART; then
+    log "Recycling Serenity Technitium stack"
+    remote_restart_serenity
+    verify_dns serenity "$SERENITY_DNS_IP"
+  fi
+}
+
+case "$ONLY_TARGET" in
+  all)
+    sync_nomad
+    sync_serenity
+    ;;
+  nomad)
+    sync_nomad
+    ;;
+  serenity)
+    sync_serenity
+    ;;
+  *)
+    echo "Invalid --target value: $ONLY_TARGET" >&2
+    exit 1
+    ;;
+esac
+
+log "Technitium backup sync finished successfully"

technitium-serenity/.env.example

@@ -0,0 +1,18 @@
+TZ=America/Chicago
+LAN_INTERFACE=br0
+LAN_SUBNET=10.5.30.0/24
+LAN_GATEWAY=10.5.30.1
+TECHNITIUM_BIND_IP=10.5.30.10
+APPDATA_TECHNITIUM_ROOT=/mnt/user/appdata/technitium-serenity/data
+DNS_SERVER_DOMAIN=technitium-serenity.home.paccoco.com
+DNS_SERVER_ADMIN_PASSWORD=CHANGE_ME
+DNS_SERVER_SSO_ENABLED=false
+DNS_SERVER_SSO_AUTHORITY=
+DNS_SERVER_SSO_CLIENT_ID=
+DNS_SERVER_SSO_CLIENT_SECRET=
+DNS_SERVER_SSO_METADATA_ADDRESS=
+DNS_SERVER_SSO_SCOPES=openid profile email groups
+DNS_SERVER_SSO_ALLOW_SIGNUP=true
+DNS_SERVER_SSO_ALLOW_SIGNUP_ONLY_FOR_MAPPED_USERS=true
+DNS_SERVER_SSO_GROUP_MAP=authentik Admins=Administrators
+DNS_SERVER_BLOCK_LIST_URLS=https://big.oisd.nl/

technitium-serenity/README.md

@@ -0,0 +1,39 @@
+# Technitium backup on Serenity
+
+Backup Technitium DNS node cloned from the PD pilot for Serenity.
+
+- Repo stack path: `/home/fizzlepoof/repos/truenas-stacks/technitium-serenity`
+- Live stack path: `/mnt/user/appdata/technitium-serenity`
+- Bind IP: `10.5.30.10`
+- DNS ports: `53/tcp`, `53/udp`
+- Web console: `http://10.5.30.10/`
+- Bootstrap domain if reset from empty config: `technitium-serenity.home.paccoco.com`
+
+This backup node is intended to replace Pi-hole fallback in DHCP resolver lists once validated. It reuses the PD Technitium config as the initial authoritative baseline.
+
+Current mirrored baseline from Pi-hole:
+- blocklists:
+  - `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts`
+  - `https://big.oisd.nl`
+- no custom local DNS entries detected
+- no conditional forwarding detected
+- no extra dnsmasq fragments detected
+
+Pilot-specific adaptation:
+- Production Pi-hole forwards to PD-local Unbound on `127.0.0.1:5335`.
+- Because the Technitium pilot uses a dedicated macvlan IP, that loopback-only upstream is not reachable from the pilot without changing production Unbound.
+- For pilot safety, Technitium is seeded as a standalone recursive resolver instead of reconfiguring production Unbound.
+
+Notes:
+- The stack binds only to `TECHNITIUM_BIND_IP`, so PD must have that secondary address present on `LAN_INTERFACE` before `docker compose up`.
+- The helper script adds the pilot IP alias read/write on the live host for the current boot only. That is deliberate for pilot safety.
+- Technitium reads the environment initialization only on first boot when `/etc/dns` is empty. If you need to re-seed initial settings, stop the container and clear the config directory first.
+- SSO-capable deployments should set the `DNS_SERVER_SSO_*` variables. Keep a known local admin account as rollback because Technitium may only apply initial auth/env seeding cleanly on first boot or with an empty `/etc/dns` config dir.
+- For native Authentik OIDC on this pilot, `DNS_SERVER_SSO_ALLOW_SIGNUP` must be `true` at least for the initial SSO user provisioning. Keeping `DNS_SERVER_SSO_ALLOW_SIGNUP_ONLY_FOR_MAPPED_USERS=true` preserves the gate so only users in mapped Authentik groups can auto-provision.
+- Authentik native OIDC callback for Technitium is `https://dns.paccoco.com/sso/callback`.
+- For Authentik native OIDC, the Technitium OAuth2 provider must have the default OpenID scope mappings attached: `openid`, `email`, and `profile`. Without them, Authentik will mint a token with no OpenID scopes and Technitium fails the `/application/o/userinfo/` call with `Scope mismatch` / `SSO authentication failed`.
+- The authoritative zone `home.paccoco.com` is now hosted in Technitium with `dns.home.paccoco.com -> 10.5.30.8`.
+- Clients will resolve `dns.home.paccoco.com` only when they query Technitium directly, or after DHCP/cutover points them at Technitium.
+
+- Host interface parent for macvlan: `br0`
+- Operational note: this node starts from a cloned Technitium config copied from PD; future record changes on PD must be resynced here until replication automation exists.

technitium-serenity/bin/configure_technitium.py

@@ -0,0 +1,160 @@
+#!/usr/bin/env python3
+import argparse
+import json
+import os
+import sys
+from typing import Any
+from urllib import parse, request, error
+
+
+def api_request(base_url: str, method: str, path: str, *, token: str | None = None, query: dict | None = None) -> tuple[int, Any]:
+    url = base_url.rstrip('/') + path
+    if query:
+        url += '?' + parse.urlencode(query, doseq=True)
+    headers = {'Accept': 'application/json'}
+    if token:
+        headers['Authorization'] = f'Bearer {token}'
+    req = request.Request(url, headers=headers, method=method)
+    try:
+        with request.urlopen(req, timeout=30) as resp:
+            return resp.status, json.load(resp)
+    except error.HTTPError as exc:
+        body = exc.read().decode('utf-8', 'replace')
+        try:
+            payload = json.loads(body)
+        except Exception:
+            payload = body
+        return exc.code, payload
+
+
+def expect_ok(status, payload, context):
+    if status != 200 or payload.get('status') != 'ok':
+        print(f'{context} failed: http={status}', file=sys.stderr)
+        print(json.dumps(payload, indent=2, sort_keys=True), file=sys.stderr)
+        raise SystemExit(1)
+
+
+def bool_s(v: bool) -> str:
+    return 'true' if v else 'false'
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description='Idempotent Technitium baseline config for the homelab pilot')
+    ap.add_argument('--base-url', default='http://127.0.0.1:5380')
+    ap.add_argument('--admin-user', default='admin')
+    ap.add_argument('--admin-pass', default=os.environ.get('DNS_SERVER_ADMIN_PASSWORD'))
+    ap.add_argument('--server-domain', default='dns.home.paccoco.com')
+    ap.add_argument('--zone', default='home.paccoco.com')
+    ap.add_argument('--record-host', default='dns.home.paccoco.com')
+    ap.add_argument('--record-ip', default='10.5.30.8')
+    ap.add_argument('--web-port', type=int, default=80)
+    ap.add_argument('--blocklist', action='append', default=None)
+    args = ap.parse_args()
+
+    if not args.admin_pass:
+        print('Missing admin password: pass --admin-pass or set DNS_SERVER_ADMIN_PASSWORD', file=sys.stderr)
+        return 1
+
+    desired_blocklists = args.blocklist or ['https://big.oisd.nl/']
+
+    st, login = api_request(args.base_url, 'GET', '/api/user/login', query={
+        'user': args.admin_user,
+        'pass': args.admin_pass,
+        'includeInfo': 'true',
+    })
+    expect_ok(st, login, 'login')
+    token = login['token']
+
+    st, settings = api_request(args.base_url, 'GET', '/api/settings/get', token=token)
+    expect_ok(st, settings, 'settings/get')
+    resp = settings['response']
+
+    query = {
+        'dnsServerDomain': args.server_domain,
+        'webServiceLocalAddresses': ','.join(resp.get('webServiceLocalAddresses') or ['[::]']),
+        'webServiceHttpPort': str(args.web_port),
+        'webServiceEnableTls': bool_s(False),
+        'webServiceTlsPort': str(resp.get('webServiceTlsPort', 53443)),
+        'webServiceHttpToTlsRedirect': bool_s(False),
+        'webServiceUseSelfSignedTlsCertificate': bool_s(False),
+        'preferIPv6': bool_s(bool(resp.get('preferIPv6', False))),
+        'dnssecValidation': bool_s(True),
+        'enableBlocking': bool_s(True),
+        'allowTxtBlockingReport': bool_s(True),
+        'blockListUrls': ','.join(desired_blocklists),
+        'blockListUpdateIntervalHours': str(resp.get('blockListUpdateIntervalHours', 24)),
+        'logQueries': bool_s(False),
+        'allowRecursion': bool_s(True),
+        'allowRecursionOnlyForPrivateNetworks': bool_s(True),
+        'qnameMinimization': bool_s(True),
+        'serveStale': bool_s(True),
+    }
+
+    st, updated = api_request(args.base_url, 'GET', '/api/settings/set', token=token, query=query)
+    expect_ok(st, updated, 'settings/set')
+
+    verify_base_url = args.base_url
+    parsed_base = parse.urlparse(args.base_url)
+    if parsed_base.port and parsed_base.port != args.web_port:
+        host = parsed_base.hostname or '127.0.0.1'
+        verify_base_url = f'{parsed_base.scheme}://{host}:{args.web_port}'
+
+    st, zones = api_request(verify_base_url, 'GET', '/api/zones/list', token=token)
+    expect_ok(st, zones, 'zones/list')
+    zone_names = {z['name'] for z in zones['response']['zones']}
+    if args.zone not in zone_names:
+        st, created = api_request(verify_base_url, 'GET', '/api/zones/create', token=token, query={
+            'zone': args.zone,
+            'type': 'Primary',
+        })
+        expect_ok(st, created, 'zones/create')
+
+    st, current_records = api_request(verify_base_url, 'GET', '/api/zones/records/get', token=token, query={
+        'domain': args.record_host,
+        'zone': args.zone,
+        'listZone': 'false',
+    })
+    expect_ok(st, current_records, 'zones/records/get')
+    for record in current_records['response'].get('records', []):
+        if record.get('type') == 'A':
+            ip = (record.get('rData') or {}).get('ipAddress')
+            if ip and ip != args.record_ip:
+                st, deleted = api_request(verify_base_url, 'GET', '/api/zones/records/delete', token=token, query={
+                    'domain': args.record_host,
+                    'zone': args.zone,
+                    'type': 'A',
+                    'ipAddress': ip,
+                })
+                expect_ok(st, deleted, 'zones/records/delete')
+
+    st, added = api_request(verify_base_url, 'GET', '/api/zones/records/add', token=token, query={
+        'domain': args.record_host,
+        'zone': args.zone,
+        'type': 'A',
+        'ipAddress': args.record_ip,
+        'overwrite': 'true',
+        'ttl': '300',
+    })
+    expect_ok(st, added, 'zones/records/add')
+
+    st, verify = api_request(verify_base_url, 'GET', '/api/settings/get', token=token)
+    expect_ok(st, verify, 'settings/get verify')
+    v = verify['response']
+    summary = {
+        'dnsServerDomain': v.get('dnsServerDomain'),
+        'webServiceHttpPort': v.get('webServiceHttpPort'),
+        'webServiceEnableTls': v.get('webServiceEnableTls'),
+        'recursion': v.get('recursion'),
+        'dnssecValidation': v.get('dnssecValidation'),
+        'enableBlocking': v.get('enableBlocking'),
+        'allowTxtBlockingReport': v.get('allowTxtBlockingReport'),
+        'blockListUrls': v.get('blockListUrls'),
+        'zone': args.zone,
+        'record': {args.record_host: args.record_ip},
+    }
+    print(json.dumps(summary, indent=2, sort_keys=True))
+    return 0
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())

technitium-serenity/bin/prepare_pd.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+STACK_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$STACK_DIR"
+
+if [[ ! -f .env ]]; then
+  echo "Missing .env in $STACK_DIR"
+  exit 1
+fi
+
+set -a
+source .env
+set +a
+
+if [[ -z "${LAN_INTERFACE:-}" || -z "${LAN_SUBNET:-}" || -z "${LAN_GATEWAY:-}" || -z "${TECHNITIUM_BIND_IP:-}" || -z "${APPDATA_TECHNITIUM_ROOT:-}" ]]; then
+  echo "LAN_INTERFACE, LAN_SUBNET, LAN_GATEWAY, TECHNITIUM_BIND_IP, and APPDATA_TECHNITIUM_ROOT must be set in .env"
+  exit 1
+fi
+
+sudo mkdir -p "${APPDATA_TECHNITIUM_ROOT}/config" "${APPDATA_TECHNITIUM_ROOT}/logs"
+
+echo "Validating compose config"
+sudo docker compose --env-file .env config >/tmp/technitium-pilot-compose.rendered.yaml
+
+echo "Rendered compose saved to /tmp/technitium-pilot-compose.rendered.yaml"
+
+if [[ "${1:-}" == "--up" ]]; then
+  echo "Starting Technitium pilot"
+  sudo docker compose --env-file .env up -d
+fi

technitium-serenity/docker-compose.yaml

@@ -0,0 +1,43 @@
+services:
+  technitium:
+    image: docker.io/technitium/dns-server@sha256:99c250f0ee494f2f31e09a76d83f8213bc4c5d6f106b0895cd5f327518469c48
+    container_name: technitium-dns-pilot
+    hostname: technitium-dns-pilot
+    restart: unless-stopped
+    networks:
+      lan_macvlan:
+        ipv4_address: ${TECHNITIUM_BIND_IP}
+    environment:
+      DNS_SERVER_DOMAIN: ${DNS_SERVER_DOMAIN}
+      DNS_SERVER_ADMIN_PASSWORD: ${DNS_SERVER_ADMIN_PASSWORD}
+      DNS_SERVER_SSO_ENABLED: ${DNS_SERVER_SSO_ENABLED:-false}
+      DNS_SERVER_SSO_AUTHORITY: ${DNS_SERVER_SSO_AUTHORITY:-}
+      DNS_SERVER_SSO_CLIENT_ID: ${DNS_SERVER_SSO_CLIENT_ID:-}
+      DNS_SERVER_SSO_CLIENT_SECRET: ${DNS_SERVER_SSO_CLIENT_SECRET:-}
+      DNS_SERVER_SSO_METADATA_ADDRESS: ${DNS_SERVER_SSO_METADATA_ADDRESS:-}
+      DNS_SERVER_SSO_SCOPES: ${DNS_SERVER_SSO_SCOPES:-openid profile email groups}
+      DNS_SERVER_SSO_ALLOW_SIGNUP: ${DNS_SERVER_SSO_ALLOW_SIGNUP:-false}
+      DNS_SERVER_SSO_ALLOW_SIGNUP_ONLY_FOR_MAPPED_USERS: ${DNS_SERVER_SSO_ALLOW_SIGNUP_ONLY_FOR_MAPPED_USERS:-true}
+      DNS_SERVER_SSO_GROUP_MAP: ${DNS_SERVER_SSO_GROUP_MAP:-authentik Admins=Administrators}
+      DNS_SERVER_WEB_SERVICE_HTTP_PORT: "80"
+      DNS_SERVER_WEB_SERVICE_HTTPS_PORT: "443"
+      DNS_SERVER_WEB_SERVICE_ENABLE_HTTPS: "true"
+      DNS_SERVER_WEB_SERVICE_USE_SELF_SIGNED_CERT: "true"
+      DNS_SERVER_WEB_SERVICE_HTTP_TO_TLS_REDIRECT: "true"
+      DNS_SERVER_OPTIONAL_PROTOCOL_DNS_OVER_HTTP: "false"
+      DNS_SERVER_RECURSION: "AllowOnlyForPrivateNetworks"
+      DNS_SERVER_ENABLE_BLOCKING: "true"
+      DNS_SERVER_ALLOW_TXT_BLOCKING_REPORT: "true"
+      DNS_SERVER_BLOCK_LIST_URLS: ${DNS_SERVER_BLOCK_LIST_URLS}
+      DNS_SERVER_LOG_USING_LOCAL_TIME: "true"
+      DNS_SERVER_LOG_FOLDER_PATH: /var/log/technitium/dns
+      DNS_SERVER_LOG_MAX_LOG_FILE_DAYS: "30"
+      DNS_SERVER_STATS_MAX_STAT_FILE_DAYS: "30"
+    volumes:
+      - ${APPDATA_TECHNITIUM_ROOT}/config:/etc/dns
+      - ${APPDATA_TECHNITIUM_ROOT}/logs:/var/log/technitium/dns
+
+networks:
+  lan_macvlan:
+    external: true
+    name: br0