truenas-stacks/docs/planning/DEPLOY_PIHOLE.md

# Pi-hole Deployment Plan

**Status:** PD primary stack scaffolded repo-side at `pihole/`; live 3-node rollout still pending replica details and PD-side deployment verification  
**Architecture:** 3-node HA cluster with Keepalived VIP, Unbound recursive DNS, Nebula Sync

---

## Architecture Overview

```
[All LAN clients]
       ↓  DNS: 10.5.1.53  (Virtual IP — floats to whichever node is MASTER)
       ↓
 ┌─────┴──────────────────────────────────┐
 │           Keepalived VIP               │
 │           10.5.1.53/24                 │
 └──────┬──────────────┬──────────────────┘
        │              │                  │
   PD (MASTER)   N.O.M.A.D. (BACKUP1) RPi4 (BACKUP2)
   10.5.1.6       10.5.1.16           10.5.1.X
   priority 100   priority 90        priority 80
   Pi-hole :53    Pi-hole :53        Pi-hole :53
   Unbound :5335  Unbound :5335      Unbound :5335
        │              │                  │
        └──────────────┴──────────────────┘
                       ↓
              [Root DNS servers]
              (recursive — no third-party upstream)
```

- **Keepalived** floats the VIP `10.5.1.53` to whichever node is alive with the highest priority
- **Unbound** on each node acts as the local recursive resolver (replaces Cloudflare/Quad9)
- **Pi-hole** on each node forwards upstream queries to its local Unbound (`127.0.0.1#5335`)
- **Nebula Sync** (running on PD) syncs all Pi-hole configs from primary → replicas daily

All nodes use `network_mode: host` for Pi-hole, Unbound, and Keepalived — this is required for Keepalived to manipulate the VIP and for Pi-hole to bind cleanly to port 53.

---

## Pre-Deploy: Network Planning

1. **Pick a free static IP for the VIP** — `10.5.1.53` is recommended (memorable for DNS). Reserve it in UniFi so DHCP never assigns it.
2. **Find the LAN interface name** on each node:
   ```bash
   # PD / RPi4:
   ip link show
   # Look for the interface with 10.5.1.x address — likely eth0, eno1, enp3s0, etc.
   ```
3. **Note RPi4 IP** — fill in `10.5.1.X` throughout this doc once known. Reserve it as static in UniFi.
4. **Verify port 53 is free on PD:**
   ```bash
   sudo ss -tulnp | grep ':53'
   # Should show only 127.0.0.1:53 (TrueNAS loopback resolver). 
   # If it shows 0.0.0.0:53, see Gotchas section.
   ```

---

## Unbound Config (shared across all nodes)

Each node runs Unbound as a Docker container on the host network, listening on `127.0.0.1:5335`.

**Save this as `unbound.conf` in each node's appdata path** (see per-node sections below):

```ini
server:
    verbosity: 1
    interface: 127.0.0.1
    port: 5335
    do-ip4: yes
    do-udp: yes
    do-tcp: yes
    do-ip6: no

    # DNSSEC
    auto-trust-anchor-file: "/opt/unbound/etc/unbound/var/root.key"
    val-permissive-mode: no

    # Performance
    num-threads: 2
    cache-min-ttl: 300
    cache-max-ttl: 86400
    prefetch: yes
    prefetch-key: yes
    minimal-responses: yes

    # Privacy
    qname-minimisation: yes
    hide-identity: yes
    hide-version: yes

    # Access control
    access-control: 127.0.0.1/32 allow
    access-control: 0.0.0.0/0 refuse

    # Root hints — download updated copy periodically
    root-hints: "/opt/unbound/etc/unbound/var/root.hints"
```

**Download root hints** (run once on each node, into the appdata path):
```bash
curl -o /path/to/appdata/unbound/var/root.hints https://www.internic.net/domain/named.cache
mkdir -p /path/to/appdata/unbound/var
# Unbound will auto-manage root.key for DNSSEC
```

---

## Node 1: PlausibleDeniability (MASTER)

**Repo stack dir:** `/home/fizzlepoof/repos/truenas-stacks/pihole/`  
**Live stack dir:** `/mnt/docker-ssd/docker/compose/pihole/`  
**Web UI port:** `8953`

### Pre-Deploy

```bash
sudo mkdir -p /mnt/tank/docker/appdata/pihole-primary/etc-pihole
sudo mkdir -p /mnt/tank/docker/appdata/unbound-primary/var
sudo mkdir -p /mnt/docker-ssd/docker/compose/pihole
# Download root hints
sudo curl -o /mnt/tank/docker/appdata/unbound-primary/var/root.hints https://www.internic.net/domain/named.cache
```

### docker-compose.yaml

```yaml
services:
  unbound:
    image: mvance/unbound:latest
    container_name: unbound
    network_mode: host
    restart: unless-stopped
    volumes:
      - /mnt/tank/docker/appdata/unbound-primary:/opt/unbound/etc/unbound/
    healthcheck:
      test: ["CMD-SHELL", "drill @127.0.0.1 -p 5335 cloudflare.com || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3

  pihole:
    image: pihole/pihole:latest
    container_name: pihole
    network_mode: host
    restart: unless-stopped
    depends_on:
      unbound:
        condition: service_healthy
    environment:
      TZ: ${TZ}
      FTLCONF_webserver_api_password: ${PIHOLE_PASSWORD}
      FTLCONF_dns_listeningMode: "LOCAL"
      FTLCONF_dns_upstreams: "127.0.0.1#5335"
      FTLCONF_webserver_port: "8953"
    volumes:
      - /mnt/tank/docker/appdata/pihole-primary/etc-pihole:/etc/pihole
    cap_add:
      - SYS_NICE

  keepalived:
    image: osixia/keepalived:2.0.20
    container_name: keepalived
    network_mode: host
    restart: unless-stopped
    cap_add:
      - NET_ADMIN
      - NET_BROADCAST
      - NET_RAW
    volumes:
      - ./keepalived.conf:/container/environment/01-custom/keepalived.conf
    environment:
      KEEPALIVED_VIRTUAL_IPS: "#PYTHON2BASH:['10.5.1.53/24']"
```

### keepalived.conf (PD — MASTER)

```
global_defs {
    router_id pihole_pd
}

vrrp_instance VI_DNS {
    state MASTER
    interface ${LAN_INTERFACE}
    virtual_router_id 53
    priority 100
    advert_int 1
    nopreempt

    authentication {
        auth_type PASS
        auth_pass ${KEEPALIVED_PASSWORD}
    }

    virtual_ipaddress {
        10.5.1.53/24
    }
}
```
> Replace `${LAN_INTERFACE}` with your actual interface (e.g. `eno1`). Replace `${KEEPALIVED_PASSWORD}` with a shared secret used on all three nodes.

### .env.example

```env
TZ=America/New_York
PIHOLE_PASSWORD=CHANGE_ME
KEEPALIVED_PASSWORD=CHANGE_ME
LAN_INTERFACE=eno1
```

### Deploy

Preferred PD-first flow now that the repo-side stack exists:

```bash
cd /mnt/docker-ssd/docker/compose/pihole
cp .env.example .env
nano .env
/usr/bin/bash bin/prepare_pd.sh
/usr/bin/bash bin/prepare_pd.sh --up
sudo docker logs pihole-primary --tail 20
sudo docker logs unbound-pihole-primary --tail 20
```

---

## Node 2: N.O.M.A.D. (BACKUP1)

N.O.M.A.D. runs Ubuntu bare metal. Deploy via Docker Compose on the host.

**Appdata:** `/mnt/hdd-2/docker/appdata/`  
**Web UI port:** `8954`

### docker-compose.yaml

Same structure as PD but with different ports/paths and priority:

```yaml
services:
  unbound:
    image: mvance/unbound:latest
    container_name: unbound-pihole
    network_mode: host
    restart: unless-stopped
    volumes:
      - /mnt/hdd-2/docker/appdata/unbound-nomad:/opt/unbound/etc/unbound/
    healthcheck:
      test: ["CMD-SHELL", "drill @127.0.0.1 -p 5335 cloudflare.com || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3

  pihole:
    image: pihole/pihole:latest
    container_name: pihole-nomad
    network_mode: host
    restart: unless-stopped
    depends_on:
      unbound-pihole:
        condition: service_healthy
    environment:
      TZ: ${TZ}
      FTLCONF_webserver_api_password: ${PIHOLE_PASSWORD}
      FTLCONF_dns_listeningMode: "LOCAL"
      FTLCONF_dns_upstreams: "127.0.0.1#5335"
      FTLCONF_webserver_port: "8954"
      FTLCONF_ntp_ipv4_active: "false"
      FTLCONF_ntp_ipv6_active: "false"
      FTLCONF_ntp_sync_active: "false"
    volumes:
      - /mnt/hdd-2/docker/appdata/pihole-nomad/etc-pihole:/etc/pihole
    cap_add:
      - SYS_NICE

  keepalived:
    image: osixia/keepalived:2.0.20
    container_name: keepalived-pihole
    network_mode: host
    restart: unless-stopped
    cap_add:
      - NET_ADMIN
      - NET_BROADCAST
      - NET_RAW
    volumes:
      - ./keepalived.conf:/container/environment/01-custom/keepalived.conf
```

### keepalived.conf (N.O.M.A.D. — BACKUP1)

```
global_defs {
    router_id pihole_nomad
}

vrrp_instance VI_DNS {
    state BACKUP
    interface enp5s0
    virtual_router_id 53
    priority 90
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass SAME_PASSWORD_AS_PRIMARY
    }

    virtual_ipaddress {
        10.5.1.53/24
    }
}
```

---

## Node 3: Raspberry Pi 4 (BACKUP2)

### Install Docker (if not already installed)

```bash
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
```

### Pre-Deploy

```bash
mkdir -p ~/pihole/etc-pihole
mkdir -p ~/pihole/unbound/var
curl -o ~/pihole/unbound/var/root.hints https://www.internic.net/domain/named.cache
sudo apt install keepalived -y
```

### docker-compose.yaml

```yaml
services:
  unbound:
    image: mvance/unbound:latest
    container_name: unbound
    network_mode: host
    restart: unless-stopped
    volumes:
      - ~/pihole/unbound:/opt/unbound/etc/unbound/
    healthcheck:
      test: ["CMD-SHELL", "drill @127.0.0.1 -p 5335 cloudflare.com || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3

  pihole:
    image: pihole/pihole:latest
    container_name: pihole
    network_mode: host
    restart: unless-stopped
    depends_on:
      unbound:
        condition: service_healthy
    environment:
      TZ: ${TZ}
      FTLCONF_webserver_api_password: ${PIHOLE_PASSWORD}
      FTLCONF_dns_listeningMode: "LOCAL"
      FTLCONF_dns_upstreams: "127.0.0.1#5335"
      FTLCONF_webserver_port: "8955"
    volumes:
      - ~/pihole/etc-pihole:/etc/pihole
    cap_add:
      - SYS_NICE
```

### keepalived (native on RPi4)

```bash
sudo nano /etc/keepalived/keepalived.conf
```

```
global_defs {
    router_id pihole_rpi4
}

vrrp_instance VI_DNS {
    state BACKUP
    interface eth0
    virtual_router_id 53
    priority 80
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass SAME_PASSWORD_AS_PRIMARY
    }

    virtual_ipaddress {
        10.5.1.53/24
    }
}
```

```bash
sudo systemctl enable keepalived
sudo systemctl start keepalived
```

---

## Nebula Sync (on PD)

[Nebula Sync](https://github.com/lovelaze/nebula-sync) syncs Pi-hole v6 configuration (gravity, blocklists, allowlists, custom DNS) from the primary to both replicas.

**Add to PD's pihole docker-compose.yaml:**

```yaml
  nebula-sync:
    image: lovelaze/nebula-sync:latest
    container_name: nebula-sync
    restart: unless-stopped
    environment:
      PRIMARY: "http://127.0.0.1:8953|${PIHOLE_PASSWORD}"
      REPLICAS: "http://10.5.1.16:8954|${PIHOLE_PASSWORD_NOMAD},http://10.5.1.X:8955|${PIHOLE_PASSWORD_RPI}"
      RUN_GRAVITY: "true"
      FULL_SYNC: "false"
      CRON: "0 3 * * *"
    networks:
      - default
```

> - `PRIMARY` — your PD Pi-hole (note: uses `127.0.0.1` since nebula-sync is on the same host; port 8953 is the web UI, also the API port in Pi-hole v6)
> - `REPLICAS` — comma-separated list of secondary instances
> - `CRON` — syncs daily at 3 AM; force a manual sync with `docker exec nebula-sync nebula-sync run`
> - Each Pi-hole instance can (and should) have a **different** password — nebula-sync handles auth per-instance
> - Add `PIHOLE_PASSWORD_NOMAD` and `PIHOLE_PASSWORD_RPI` to your `.env` file

### What Nebula Sync copies
- Gravity database (blocklists, allowlists, group assignments)
- Custom DNS/CNAME records
- Client groups
- Does **not** sync per-instance settings (web password, interface binding — these stay local)

---

## Router Configuration (UniFi)

Point **only the VIP** as DNS:

1. UniFi Network → Settings → Networks → **LAN** → DHCP Name Server: **Manual**
2. **DNS Server 1:** `10.5.1.53` (VIP — floats between nodes)
3. **DNS Server 2:** *(leave blank or set a public fallback like `1.1.1.1` only if you want internet DNS as last resort)*
4. Save and renew DHCP leases

> **Why single VIP instead of listing all three IPs?**  
> Clients round-robin across all listed DNS servers — so some queries bypass Pi-hole entirely. With Keepalived the VIP always points to exactly one live Pi-hole node, ensuring 100% of queries are filtered.

---

## Post-Deploy

### Verify VIP is working

```bash
# Check which node holds the VIP
ip addr show | grep 10.5.1.53   # Run on each node — only MASTER shows it

# Test DNS through VIP
nslookup google.com 10.5.1.53
nslookup doubleclick.net 10.5.1.53   # Should return 0.0.0.0 (blocked)
```

### Test failover

```bash
# On PD, bring down keepalived
sudo docker stop keepalived
# Wait ~3 seconds, then check from another machine:
nslookup google.com 10.5.1.53   # Should still work — N.O.M.A.D. took over VIP
# Restore
sudo docker start keepalived
```

### Add blocklists (on primary — Nebula Sync propagates to replicas)

| List | URL |
|------|-----|
| Steven Black (unified) | `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts` |
| OISD Basic | `https://basic.oisd.nl/` |
| Hagezi Pro | `https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/pro.txt` |

After adding: **Tools → Update Gravity** on PD, then run Nebula Sync to push to replicas.

### Local DNS records

Add in Pi-hole (primary) → Local DNS → DNS Records:

| Domain | IP |
|--------|----|
| pd.lan | 10.5.1.6 |
| nomad.lan | 10.5.1.16 |
| rocinante.lan | 10.5.1.112 |
| rpi4.lan | 10.5.1.X |

Nebula Sync will propagate these to all replicas.

### Pangolin for admin UIs (optional)

| Subdomain | Target |
|-----------|--------|
| `pihole.paccoco.com` | `http://10.5.1.6:8953` (primary) |
| `pihole2.paccoco.com` | `http://10.5.1.16:8954` (N.O.M.A.D.) |

Enable Pangolin auth — Pi-hole admin should not be publicly accessible.

---

## Storage Decisions

| Node | Path | Tier | Reason |
|------|------|------|--------|
| PD | `/mnt/tank/docker/appdata/pihole-primary` | tank | Low write frequency |
| PD | `/mnt/tank/docker/appdata/unbound-primary` | tank | Config + root hints only |
| N.O.M.A.D. | `/mnt/hdd-2/docker/appdata/pihole-nomad` | Local HDD | Docker-managed local appdata |
| RPi4 | `~/pihole/` | SD card / SSD | Dedicated Pi, no other concerns |

---

## Known Gotchas

- **Port 53 conflict on PD:** TrueNAS binds to `127.0.0.1:53`. With `network_mode: host`, Pi-hole will try to bind to all interfaces. Set `FTLCONF_dns_listeningMode: LOCAL` so it only listens on the actual LAN interface, not loopback.
- **keepalived interface name:** Must match the actual NIC name on each host. Get it with `ip link show`.
- **`virtual_router_id` must be unique** per VRRP group on your LAN. `53` is a good mnemonic. Confirm no other keepalived instance on your LAN uses the same ID.
- **Pi-hole v6 API port:** In v6, the web UI and API both run on the `FTLCONF_webserver_port`. Nebula Sync and the web browser use the same port.
- **nopreempt on MASTER:** The `nopreempt` option in PD's keepalived.conf prevents PD from immediately reclaiming the VIP when it comes back up after a failure. Remove it if you want PD to always retake MASTER on recovery.
- **Unbound cold start:** Unbound takes a few seconds to start and validate DNSSEC. The `depends_on: condition: service_healthy` in the Pi-hole compose ensures Pi-hole waits for Unbound before starting.