Add internal Traefik ingress for Pangolin and Authelia

docs/architecture/NETWORKING_MODEL.md

@@ -64,6 +64,14 @@ networks:
     external: true
 ```
 
+### Internal Ingress Pattern
+- Pangolin remains the public edge on the VPS.
+- PD now runs an internal-only Traefik container on the `pangolin` network.
+- Secure routed apps should move toward: `Pangolin -> Traefik -> app`.
+- Traefik does not mount `docker.sock`; routes are explicit file-provider config.
+- Authelia is the policy decision point for per-domain and per-group access.
+- Internal services can stay HTTP-only behind Traefik unless a backend specifically needs HTTPS.
+
 ### Internal Service Model
 Internal services run HTTP only behind the reverse proxy. Compose service names used for internal DNS between services in the same stack/network.
 

docs/architecture/SERVICES_DIRECTORY.md

@@ -19,12 +19,13 @@ All homelab services, their hosts, ports, and current status.
 | Homepage | 3300 | http://pd:3300 | ✅ Active |
 | Dockhand | 3230 | http://pd:3230 | ✅ Active |
 | Uptime Kuma | 3001 | http://pd:3001 | ✅ Active |
+| Traefik | internal only | Pangolin-targeted internal ingress on `pangolin` network | ✅ Active |
 | Gotify | 8443 | https://gotify.paccoco.com | ✅ Active |
 | Gitea | 3002 / 2222 | http://pd:3002 / http://10.5.30.6:3002 (SSH: `ssh://git@10.5.30.6:2222/<owner>/<repo>.git`) | ✅ Active |
 | Shlink | 8087 | http://pd:8087 | ✅ Active |
 | Shlink Web Client | 8088 | http://pd:8088 | ✅ Active |
 | RackPeek | 8283 | http://pd:8283 | ✅ Active |
-| Authelia | 9091 | http://pd:9091 / https://auth.paccoco.com | ✅ Active internally on PD; public Pangolin route still pending |
+| Authelia | 9091 | http://pd:9091 / https://auth.paccoco.com | ✅ Active internally on PD; Traefik is ready to enforce app auth once Pangolin targets are cut over |
 
 ## AI Stack
 

docs/operations/IDENTITY_AND_SSO.md

@@ -14,7 +14,25 @@ Authelia is the staged identity provider for the homelab.
 - stack path: identity/
 - host target: PlausibleDeniability
 - public entrypoint: auth.paccoco.com
-- status: repo-staged, not assumed live until deployed on PD
+- status: Authelia is live on PD; Traefik-based protected app routing is the next cutover layer
+
+## Access-control direction
+
+Traefik is the internal router and Authelia is the policy engine.
+
+Current intended protected domains:
+- `gitea.paccoco.com`
+- `grafana.paccoco.com`
+- `dns.paccoco.com`
+- `traefik.paccoco.com`
+
+Current baseline group policy:
+- `admins` -> full access to the domains above
+
+Planned expansion pattern:
+- create additional Authelia groups per audience or app class
+- grant access in `access_control.rules` by domain + `group:<name>` subject
+- keep Pangolin SSO disabled for routes that Traefik/Authelia protect, to avoid double auth
 
 ## Initial bootstrap flow
 

identity/authelia/configuration.yml.example

@@ -22,9 +22,24 @@ authentication_backend:
 access_control:
   default_policy: deny
   rules:
-    - domain:
-        - '*.paccoco.com'
+    - domain: auth.paccoco.com
+      policy: bypass
+    - domain: gitea.paccoco.com
       policy: one_factor
+      subject:
+        - group:admins
+    - domain: grafana.paccoco.com
+      policy: one_factor
+      subject:
+        - group:admins
+    - domain: dns.paccoco.com
+      policy: one_factor
+      subject:
+        - group:admins
+    - domain: traefik.paccoco.com
+      policy: one_factor
+      subject:
+        - group:admins
 
 session:
   secret: 'CHANGE_ME_SESSION_SECRET'

identity/authelia/users_database.example.yml

@@ -2,7 +2,11 @@ users:
   john:
     disabled: false
     displayname: John
-    email: john@paccoco.com
-    password: "$argon2id$v=19$m=65536,t=3,p=4$REPLACE$ME"
+    email: john@example.com
+    password: '$argon2id$v=19$m=65536,t=3,p=4$REPLACE$REPLACE'
     groups:
       - admins
+      # future examples:
+      # - infra
+      # - media
+      # - family

ingress/README.md

@@ -0,0 +1,30 @@
+# Ingress / Traefik
+
+This stack adds an internal Traefik layer on PD.
+
+Why this exists:
+- keep Pangolin as the public tunnel/edge from the VPS
+- centralize app routing behind one internal reverse proxy
+- put Authelia in front of selected apps with domain-based policy
+- fix browser cert pain for apps like Technitium by terminating the public HTTPS edge before the self-signed backend
+- avoid giving Traefik Docker socket access; routes are explicit file-provider config
+
+Design choices:
+- Traefik is internal-only on the `pangolin` Docker network
+- no host port publishing
+- no ACME on PD
+- no Docker provider / no docker.sock mount
+- Pangolin resources should target `traefik:80` on site 4 when an app is cut over to this layer
+- Authelia remains the identity decision point for domain/group access
+
+Current routed hostnames in dynamic config:
+- `auth.paccoco.com` -> Authelia
+- `gitea.paccoco.com` -> Gitea (Authelia-protected)
+- `grafana.paccoco.com` -> Grafana (Authelia-protected)
+- `dns.paccoco.com` -> Technitium (Authelia-protected)
+- `traefik.paccoco.com` -> Traefik dashboard (Authelia-protected)
+
+Cutover note:
+- Deploying this stack does not automatically rewrite Pangolin resources.
+- The secure end state is: Pangolin tunnel -> Traefik -> app.
+- If an existing Pangolin resource still points directly at an app, that app will keep behaving the old way until its target is changed.

ingress/docker-compose.yaml

@@ -0,0 +1,32 @@
+name: ingress
+
+networks:
+  pangolin:
+    external: true
+
+services:
+  traefik:
+    image: traefik:v3.1
+    container_name: traefik
+    restart: unless-stopped
+    security_opt:
+      - no-new-privileges:true
+    cap_drop:
+      - ALL
+    cap_add:
+      - NET_BIND_SERVICE
+    read_only: true
+    tmpfs:
+      - /tmp
+      - /run
+    networks:
+      - pangolin
+    volumes:
+      - /mnt/docker-ssd/docker/compose/ingress/traefik/traefik.yml:/etc/traefik/traefik.yml:ro
+      - /mnt/docker-ssd/docker/compose/ingress/traefik/dynamic:/etc/traefik/dynamic:ro
+    healthcheck:
+      test: ["CMD", "traefik", "healthcheck", "--ping"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 20s

ingress/traefik/dynamic/routes.yml

@@ -0,0 +1,89 @@
+http:
+  routers:
+    auth:
+      rule: Host(`auth.paccoco.com`)
+      entryPoints: [web]
+      service: authelia
+      middlewares:
+        - security-headers
+
+    gitea:
+      rule: Host(`gitea.paccoco.com`)
+      entryPoints: [web]
+      service: gitea
+      middlewares:
+        - security-headers
+        - authelia-forwardauth
+
+    grafana:
+      rule: Host(`grafana.paccoco.com`)
+      entryPoints: [web]
+      service: grafana
+      middlewares:
+        - security-headers
+        - authelia-forwardauth
+
+    technitium:
+      rule: Host(`dns.paccoco.com`)
+      entryPoints: [web]
+      service: technitium
+      middlewares:
+        - security-headers
+        - authelia-forwardauth
+
+    traefik-dashboard:
+      rule: Host(`traefik.paccoco.com`)
+      entryPoints: [web]
+      service: api@internal
+      middlewares:
+        - security-headers
+        - authelia-forwardauth
+
+  services:
+    authelia:
+      loadBalancer:
+        servers:
+          - url: http://authelia:9091
+
+    gitea:
+      loadBalancer:
+        servers:
+          - url: http://gitea:3000
+
+    grafana:
+      loadBalancer:
+        servers:
+          - url: http://grafana:3000
+
+    technitium:
+      loadBalancer:
+        passHostHeader: true
+        serversTransport: technitium-selfsigned
+        servers:
+          - url: https://technitium-dns-pilot:443
+
+  middlewares:
+    authelia-forwardauth:
+      forwardAuth:
+        address: http://authelia:9091/api/authz/forward-auth
+        trustForwardHeader: true
+        authResponseHeaders:
+          - Remote-User
+          - Remote-Groups
+          - Remote-Name
+          - Remote-Email
+
+    security-headers:
+      headers:
+        frameDeny: true
+        browserXssFilter: true
+        contentTypeNosniff: true
+        referrerPolicy: same-origin
+        customRequestHeaders:
+          X-Forwarded-Proto: https
+        customResponseHeaders:
+          X-Robots-Tag: noindex, nofollow, noarchive, nosnippet
+
+  serversTransports:
+    technitium-selfsigned:
+      insecureSkipVerify: true

ingress/traefik/traefik.yml

@@ -0,0 +1,22 @@
+global:
+  checkNewVersion: false
+  sendAnonymousUsage: false
+
+api:
+  dashboard: true
+
+ping: {}
+
+entryPoints:
+  web:
+    address: ":80"
+
+providers:
+  file:
+    directory: /etc/traefik/dynamic
+    watch: true
+
+log:
+  level: INFO
+
+accessLog: {}

technitium-pilot/.env.example

@@ -0,0 +1,9 @@
+TZ=America/Chicago
+LAN_INTERFACE=enp3s0f0
+LAN_SUBNET=10.5.30.0/24
+LAN_GATEWAY=10.5.30.1
+TECHNITIUM_BIND_IP=10.5.30.8
+APPDATA_TECHNITIUM_ROOT=/mnt/tank/docker/appdata/technitium-pilot
+DNS_SERVER_DOMAIN=dns.home.paccoco.com
+DNS_SERVER_ADMIN_PASSWORD=CHANGE_ME
+DNS_SERVER_BLOCK_LIST_URLS=https://big.oisd.nl/

technitium-pilot/README.md

@@ -0,0 +1,32 @@
+# Technitium DNS pilot on PD
+
+Non-disruptive Technitium DNS Server pilot for PlausibleDeniability.
+
+- Repo stack path: `/home/fizzlepoof/repos/truenas-stacks/technitium-pilot`
+- Live PD stack path: `/mnt/docker-ssd/docker/compose/technitium-pilot`
+- Pilot bind IP: `10.5.30.8`
+- DNS ports: `53/tcp`, `53/udp`
+- Web console: `http://10.5.30.8/`
+- Friendly hostname inside Technitium: `http://dns.home.paccoco.com/`
+
+This pilot intentionally does not touch the live Pi-hole service on `10.5.30.6` or the client VIP on `10.5.30.53`.
+
+Current mirrored baseline from Pi-hole:
+- blocklists:
+  - `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts`
+  - `https://big.oisd.nl`
+- no custom local DNS entries detected
+- no conditional forwarding detected
+- no extra dnsmasq fragments detected
+
+Pilot-specific adaptation:
+- Production Pi-hole forwards to PD-local Unbound on `127.0.0.1:5335`.
+- Because the Technitium pilot uses a dedicated macvlan IP, that loopback-only upstream is not reachable from the pilot without changing production Unbound.
+- For pilot safety, Technitium is seeded as a standalone recursive resolver instead of reconfiguring production Unbound.
+
+Notes:
+- The stack binds only to `TECHNITIUM_BIND_IP`, so PD must have that secondary address present on `LAN_INTERFACE` before `docker compose up`.
+- The helper script adds the pilot IP alias read/write on the live host for the current boot only. That is deliberate for pilot safety.
+- Technitium reads the environment initialization only on first boot when `/etc/dns` is empty. If you need to re-seed initial settings, stop the container and clear the config directory first.
+- The authoritative zone `home.paccoco.com` is now hosted in Technitium with `dns.home.paccoco.com -> 10.5.30.8`.
+- Clients will resolve `dns.home.paccoco.com` only when they query Technitium directly, or after DHCP/cutover points them at Technitium.

technitium-pilot/bin/configure_technitium.py

@@ -0,0 +1,160 @@
+#!/usr/bin/env python3
+import argparse
+import json
+import os
+import sys
+from typing import Any
+from urllib import parse, request, error
+
+
+def api_request(base_url: str, method: str, path: str, *, token: str | None = None, query: dict | None = None) -> tuple[int, Any]:
+    url = base_url.rstrip('/') + path
+    if query:
+        url += '?' + parse.urlencode(query, doseq=True)
+    headers = {'Accept': 'application/json'}
+    if token:
+        headers['Authorization'] = f'Bearer {token}'
+    req = request.Request(url, headers=headers, method=method)
+    try:
+        with request.urlopen(req, timeout=30) as resp:
+            return resp.status, json.load(resp)
+    except error.HTTPError as exc:
+        body = exc.read().decode('utf-8', 'replace')
+        try:
+            payload = json.loads(body)
+        except Exception:
+            payload = body
+        return exc.code, payload
+
+
+def expect_ok(status, payload, context):
+    if status != 200 or payload.get('status') != 'ok':
+        print(f'{context} failed: http={status}', file=sys.stderr)
+        print(json.dumps(payload, indent=2, sort_keys=True), file=sys.stderr)
+        raise SystemExit(1)
+
+
+def bool_s(v: bool) -> str:
+    return 'true' if v else 'false'
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description='Idempotent Technitium baseline config for the homelab pilot')
+    ap.add_argument('--base-url', default='http://127.0.0.1:5380')
+    ap.add_argument('--admin-user', default='admin')
+    ap.add_argument('--admin-pass', default=os.environ.get('DNS_SERVER_ADMIN_PASSWORD'))
+    ap.add_argument('--server-domain', default='dns.home.paccoco.com')
+    ap.add_argument('--zone', default='home.paccoco.com')
+    ap.add_argument('--record-host', default='dns.home.paccoco.com')
+    ap.add_argument('--record-ip', default='10.5.30.8')
+    ap.add_argument('--web-port', type=int, default=80)
+    ap.add_argument('--blocklist', action='append', default=None)
+    args = ap.parse_args()
+
+    if not args.admin_pass:
+        print('Missing admin password: pass --admin-pass or set DNS_SERVER_ADMIN_PASSWORD', file=sys.stderr)
+        return 1
+
+    desired_blocklists = args.blocklist or ['https://big.oisd.nl/']
+
+    st, login = api_request(args.base_url, 'GET', '/api/user/login', query={
+        'user': args.admin_user,
+        'pass': args.admin_pass,
+        'includeInfo': 'true',
+    })
+    expect_ok(st, login, 'login')
+    token = login['token']
+
+    st, settings = api_request(args.base_url, 'GET', '/api/settings/get', token=token)
+    expect_ok(st, settings, 'settings/get')
+    resp = settings['response']
+
+    query = {
+        'dnsServerDomain': args.server_domain,
+        'webServiceLocalAddresses': ','.join(resp.get('webServiceLocalAddresses') or ['[::]']),
+        'webServiceHttpPort': str(args.web_port),
+        'webServiceEnableTls': bool_s(False),
+        'webServiceTlsPort': str(resp.get('webServiceTlsPort', 53443)),
+        'webServiceHttpToTlsRedirect': bool_s(False),
+        'webServiceUseSelfSignedTlsCertificate': bool_s(False),
+        'preferIPv6': bool_s(bool(resp.get('preferIPv6', False))),
+        'dnssecValidation': bool_s(True),
+        'enableBlocking': bool_s(True),
+        'allowTxtBlockingReport': bool_s(True),
+        'blockListUrls': ','.join(desired_blocklists),
+        'blockListUpdateIntervalHours': str(resp.get('blockListUpdateIntervalHours', 24)),
+        'logQueries': bool_s(False),
+        'allowRecursion': bool_s(True),
+        'allowRecursionOnlyForPrivateNetworks': bool_s(True),
+        'qnameMinimization': bool_s(True),
+        'serveStale': bool_s(True),
+    }
+
+    st, updated = api_request(args.base_url, 'GET', '/api/settings/set', token=token, query=query)
+    expect_ok(st, updated, 'settings/set')
+
+    verify_base_url = args.base_url
+    parsed_base = parse.urlparse(args.base_url)
+    if parsed_base.port and parsed_base.port != args.web_port:
+        host = parsed_base.hostname or '127.0.0.1'
+        verify_base_url = f'{parsed_base.scheme}://{host}:{args.web_port}'
+
+    st, zones = api_request(verify_base_url, 'GET', '/api/zones/list', token=token)
+    expect_ok(st, zones, 'zones/list')
+    zone_names = {z['name'] for z in zones['response']['zones']}
+    if args.zone not in zone_names:
+        st, created = api_request(verify_base_url, 'GET', '/api/zones/create', token=token, query={
+            'zone': args.zone,
+            'type': 'Primary',
+        })
+        expect_ok(st, created, 'zones/create')
+
+    st, current_records = api_request(verify_base_url, 'GET', '/api/zones/records/get', token=token, query={
+        'domain': args.record_host,
+        'zone': args.zone,
+        'listZone': 'false',
+    })
+    expect_ok(st, current_records, 'zones/records/get')
+    for record in current_records['response'].get('records', []):
+        if record.get('type') == 'A':
+            ip = (record.get('rData') or {}).get('ipAddress')
+            if ip and ip != args.record_ip:
+                st, deleted = api_request(verify_base_url, 'GET', '/api/zones/records/delete', token=token, query={
+                    'domain': args.record_host,
+                    'zone': args.zone,
+                    'type': 'A',
+                    'ipAddress': ip,
+                })
+                expect_ok(st, deleted, 'zones/records/delete')
+
+    st, added = api_request(verify_base_url, 'GET', '/api/zones/records/add', token=token, query={
+        'domain': args.record_host,
+        'zone': args.zone,
+        'type': 'A',
+        'ipAddress': args.record_ip,
+        'overwrite': 'true',
+        'ttl': '300',
+    })
+    expect_ok(st, added, 'zones/records/add')
+
+    st, verify = api_request(verify_base_url, 'GET', '/api/settings/get', token=token)
+    expect_ok(st, verify, 'settings/get verify')
+    v = verify['response']
+    summary = {
+        'dnsServerDomain': v.get('dnsServerDomain'),
+        'webServiceHttpPort': v.get('webServiceHttpPort'),
+        'webServiceEnableTls': v.get('webServiceEnableTls'),
+        'recursion': v.get('recursion'),
+        'dnssecValidation': v.get('dnssecValidation'),
+        'enableBlocking': v.get('enableBlocking'),
+        'allowTxtBlockingReport': v.get('allowTxtBlockingReport'),
+        'blockListUrls': v.get('blockListUrls'),
+        'zone': args.zone,
+        'record': {args.record_host: args.record_ip},
+    }
+    print(json.dumps(summary, indent=2, sort_keys=True))
+    return 0
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())

technitium-pilot/bin/prepare_pd.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+STACK_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$STACK_DIR"
+
+if [[ ! -f .env ]]; then
+  echo "Missing .env in $STACK_DIR"
+  exit 1
+fi
+
+set -a
+source .env
+set +a
+
+if [[ -z "${LAN_INTERFACE:-}" || -z "${LAN_SUBNET:-}" || -z "${LAN_GATEWAY:-}" || -z "${TECHNITIUM_BIND_IP:-}" || -z "${APPDATA_TECHNITIUM_ROOT:-}" ]]; then
+  echo "LAN_INTERFACE, LAN_SUBNET, LAN_GATEWAY, TECHNITIUM_BIND_IP, and APPDATA_TECHNITIUM_ROOT must be set in .env"
+  exit 1
+fi
+
+sudo mkdir -p "${APPDATA_TECHNITIUM_ROOT}/config" "${APPDATA_TECHNITIUM_ROOT}/logs"
+
+echo "Validating compose config"
+sudo docker compose --env-file .env config >/tmp/technitium-pilot-compose.rendered.yaml
+
+echo "Rendered compose saved to /tmp/technitium-pilot-compose.rendered.yaml"
+
+if [[ "${1:-}" == "--up" ]]; then
+  echo "Starting Technitium pilot"
+  sudo docker compose --env-file .env up -d
+fi

technitium-pilot/docker-compose.yaml

@@ -0,0 +1,42 @@
+services:
+  technitium:
+    image: docker.io/technitium/dns-server@sha256:99c250f0ee494f2f31e09a76d83f8213bc4c5d6f106b0895cd5f327518469c48
+    container_name: technitium-dns-pilot
+    hostname: technitium-dns-pilot
+    restart: unless-stopped
+    networks:
+      lan_macvlan:
+        ipv4_address: ${TECHNITIUM_BIND_IP}
+      pangolin: {}
+    environment:
+      DNS_SERVER_DOMAIN: ${DNS_SERVER_DOMAIN}
+      DNS_SERVER_ADMIN_PASSWORD: ${DNS_SERVER_ADMIN_PASSWORD}
+      DNS_SERVER_WEB_SERVICE_HTTP_PORT: "80"
+      DNS_SERVER_WEB_SERVICE_HTTPS_PORT: "443"
+      DNS_SERVER_WEB_SERVICE_ENABLE_HTTPS: "true"
+      DNS_SERVER_WEB_SERVICE_USE_SELF_SIGNED_CERT: "true"
+      DNS_SERVER_WEB_SERVICE_HTTP_TO_TLS_REDIRECT: "true"
+      DNS_SERVER_OPTIONAL_PROTOCOL_DNS_OVER_HTTP: "false"
+      DNS_SERVER_RECURSION: "AllowOnlyForPrivateNetworks"
+      DNS_SERVER_ENABLE_BLOCKING: "true"
+      DNS_SERVER_ALLOW_TXT_BLOCKING_REPORT: "true"
+      DNS_SERVER_BLOCK_LIST_URLS: ${DNS_SERVER_BLOCK_LIST_URLS}
+      DNS_SERVER_LOG_USING_LOCAL_TIME: "true"
+      DNS_SERVER_LOG_FOLDER_PATH: /var/log/technitium/dns
+      DNS_SERVER_LOG_MAX_LOG_FILE_DAYS: "30"
+      DNS_SERVER_STATS_MAX_STAT_FILE_DAYS: "30"
+    volumes:
+      - ${APPDATA_TECHNITIUM_ROOT}/config:/etc/dns
+      - ${APPDATA_TECHNITIUM_ROOT}/logs:/var/log/technitium/dns
+
+networks:
+  pangolin:
+    external: true
+  lan_macvlan:
+    driver: macvlan
+    driver_opts:
+      parent: ${LAN_INTERFACE}
+    ipam:
+      config:
+        - subnet: ${LAN_SUBNET}
+          gateway: ${LAN_GATEWAY}