fizzlepoof/truenas-stacks
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ba958f7ef624c7e32343e6e8a00da37d9b604e7b
truenas-stacks/docs/operations/SECRETS_MANAGEMENT.md
Fizzlepoof ffc1280f7c
Some checks failed
secret-guardrails / artifact-secret-scan (push) Has been cancelled
Details
secret-guardrails / gitleaks (push) Has been cancelled
Details
docs: update secrets backup coverage
2026-05-24 03:20:07 +00:00

11 KiB
Raw Blame History

Secrets Management

Overview

Secrets are managed in two layers:

  1. Live secrets — .env files in the active deployment trees on PD, N.O.M.A.D., and Serenity; gitignored and never committed to the main repo
  2. Encrypted backup — those live .env files are backed up into the private git-crypt encrypted Gitea repo at /mnt/docker-ssd/docker/secrets/ on PD

PD is the secrets-backup hub. PD-origin .env files sync locally into the encrypted repo, and standalone .env files from N.O.M.A.D. or Serenity must also be copied into that same encrypted repo so the backup set is not PD-only.

Principles

  • .env files are gitignored in the main repo — never committed there
  • .env.example files with placeholder values are committed instead
  • The secrets repo encrypts everything at rest using git-crypt (symmetric key)
  • Secrets are generated with standard tools, never reused across services

Generating Secrets

# Database passwords
openssl rand -hex 24

# JWT / session secrets
openssl rand -hex 32

# API keys (where format allows)
openssl rand -base64 32

Live .env File Locations

PD compose tree

PD stack .env files live alongside their docker-compose.yaml:

/mnt/docker-ssd/docker/compose/
  ai/.env
  automation/.env
  databases/.env
  dev/.env
  documents/.env
  home/.env
  infrastructure/.env
  media/.env
  media/kima-hub/.env
  mesh-mqtt-observer/.env
  meshtastic/.env
  monitoring/.env
  photos/.env

N.O.M.A.D. standalone service roots

N.O.M.A.D. keeps standalone service deployments under /opt/<service>. Any service-local .env there is also part of the secrets backup scope.

Backed-up examples currently include:

/opt/doris-kitchen/.env
/opt/doris-schoolhouse/.env
/opt/hawser-nomad/.env
/opt/honcho/.env
/opt/pihole-nomad/.env
/opt/technitium-nomad/.env

Serenity appdata roots

Serenity service-local .env files under /mnt/user/appdata/<service>/.env are also in scope for encrypted backup.

Current verified example:

/mnt/user/appdata/technitium-serenity/.env

Secrets Repo (git-crypt)
Gitea repo https://gitea.paccoco.com/fizzlepoof/homelab-secrets (private)
SSH remote ssh://git@10.5.30.6:2222/fizzlepoof/homelab-secrets.git
Local path on PD /mnt/docker-ssd/docker/secrets/
Symmetric key location /mnt/docker-ssd/docker/secrets/.git-crypt-secrets.key (if copied there) or retrieve from password manager
Key backup Password manager + C:\Users\Fizzlepoof\Downloads\.git-crypt-secrets.key

Repo structure

Directory Contents
env/ .env files for PD stacks and standalone service roots, named with stable service-oriented filenames such as <stack>.env, <service>-nomad.env, or <service>-serenity.env
keys/ API keys and tokens
certs/ TLS certificates
tokens/ Service tokens (Paperless, LiteLLM, etc.)
ssh/ SSH private keys

All files except README.md and .gitattributes are encrypted by git-crypt on commit.

Syncing .env Files to the Secrets Repo

PD-native compose env sync

Use the sync script from PD after any live PD .env change. Run it in a root-capable context and make sure git-crypt is on PATH:

export PATH="/mnt/docker-ssd/bin:/root/bin:$PATH"
bash /mnt/docker-ssd/docker/compose/scripts/sync-envs-to-secrets.sh

What it does:

  • Scans all stack directories under /mnt/docker-ssd/docker/compose/ for .env files
  • Copies changed/new files to /mnt/docker-ssd/docker/secrets/env/ as <stack>.env
  • Skips unchanged files
  • Commits and pushes to Gitea automatically

Run this any time you create or update a PD stack .env file.

Operational rule: after any live .env change on PD, Doris or the operator must run the sync script and confirm whether it committed/pushed changes or reported the secrets repo already up to date. Do not treat live .env edits as complete until this sync step is checked.

N.O.M.A.D. and Serenity standalone env backup rule

Not every important .env lives under PD's compose tree.

For N.O.M.A.D. /opt/<service>/.env files and Serenity /mnt/user/appdata/<service>/.env files:

  1. pull or copy the live file contents to PD through an approved host path
  2. store them in /mnt/docker-ssd/docker/secrets/env/ using a stable name like <service>-nomad.env or <service>-serenity.env
  3. git add, git commit, and git push from the encrypted secrets repo on PD
  4. verify the backed-up file hash matches the live source before declaring the backup complete

Current naming examples:

env/doris-kitchen.env
env/doris-schoolhouse.env
env/hawser-nomad.env
env/honcho.env
env/pihole-nomad.env
env/technitium-nomad.env
env/technitium-serenity.env

Operationally, treat these non-PD env files as first-class backup scope. The homelab secrets backup is not complete if only PD compose env files are protected.

git-crypt on TrueNAS

apt is blocked on TrueNAS Scale. git-crypt is installed via Docker.

Preferred installation location

/mnt/docker-ssd/bin/git-crypt

Compatibility note

The original bootstrap script installs to /root/bin/git-crypt, while the newer runbook expects /mnt/docker-ssd/bin/git-crypt. Standardize on /mnt/docker-ssd/bin/git-crypt and make sure PATH includes that directory for both root and truenas_admin.

Recommended PATH entry:

export PATH="/mnt/docker-ssd/bin:$PATH"

Re-installing after a full rebuild

Option 1 — one-liner:

sudo -i
docker run --rm -v /tmp:/out debian:bookworm-slim \
  bash -c "apt-get update -qq && apt-get install -y -qq git-crypt && cp /usr/bin/git-crypt /out/git-crypt"
mkdir -p /mnt/docker-ssd/bin
cp /tmp/git-crypt /mnt/docker-ssd/bin/git-crypt
chmod +x /mnt/docker-ssd/bin/git-crypt
echo 'export PATH="/mnt/docker-ssd/bin:$PATH"' >> /root/.bashrc
echo 'export PATH="/mnt/docker-ssd/bin:$PATH"' >> /home/truenas_admin/.bashrc

Option 2 — via dev stack compose:

cd /mnt/docker-ssd/docker/compose/dev
sudo docker compose --profile setup up git-crypt-init
# copies git-crypt to /root/bin — then move to persistent location:
sudo cp /root/bin/git-crypt /mnt/docker-ssd/bin/git-crypt

Unlocking the Secrets Repo on a New Machine

# Install git-crypt first (see above), then:
git clone ssh://git@10.5.30.6:2222/fizzlepoof/homelab-secrets.git /mnt/docker-ssd/docker/secrets
cd /mnt/docker-ssd/docker/secrets
git-crypt unlock /path/to/.git-crypt-secrets.key

After unlock, the env/ directory contains plaintext .env files ready to copy back into the stack directories.

Full Rebuild Runbook

On a fresh PD after reinstalling TrueNAS:

  1. Install git-crypt (Option 1 above — docker is available immediately after TrueNAS install)
  2. Restore secrets repo: 
    git clone ssh://git@10.5.30.6:2222/fizzlepoof/homelab-secrets.git /mnt/docker-ssd/docker/secrets
cd /mnt/docker-ssd/docker/secrets
git-crypt unlock /path/to/.git-crypt-secrets.key
  3. Restore .env files from secrets/env/ back to their stack directories: 
    cp /mnt/docker-ssd/docker/secrets/env/ai.env /mnt/docker-ssd/docker/compose/ai/.env
cp /mnt/docker-ssd/docker/secrets/env/databases.env /mnt/docker-ssd/docker/compose/databases/.env
# ... etc for each stack
  4. Clone the main stacks repo: 
    git clone ssh://git@10.5.30.6:2222/fizzlepoof/truenas-stacks.git /mnt/docker-ssd/docker/compose
  5. Redeploy stacks per DEPLOYMENT_CHECKLIST.md
  6. After any future live .env edit, immediately run: 
    export PATH="/mnt/docker-ssd/bin:$PATH"
bash /mnt/docker-ssd/docker/compose/scripts/sync-envs-to-secrets.sh
    Then verify the secrets repo is either updated and pushed or already unchanged.

Known Credentials Locations

Service File
shared-postgres databases/.env
shared-mariadb databases/.env
OpenWebUI DB ai/.env — user: openwebui, db: openwebui
Donetick home/.env + selfhosted.yaml
Meshmonitor meshtastic/.env
Gitea dev/.env
LiteLLM ai/.env
Paperless documents/.env
UniFi Doris operator automation/.env

Controller Export / Baseline Artifact Rules

Raw infrastructure exports must be treated as potentially secret-bearing even when they are not .env files.

Examples:

  • UniFi networkconf / portconf dumps
  • firewall/router/controller JSON exports
  • VPN client or server config exports
  • diagnostic snapshots taken from appliances or operator APIs

Permanent rules

  • Do not commit raw controller or appliance exports to the main repo.
  • Commit only sanitized markdown summaries or explicitly redacted artifacts.
  • If raw retention is needed, store the export in the encrypted secrets repo or another operator-only location outside the main repo.
  • Before committing infra artifacts, scan for obvious secret-bearing keys such as:
    • private_key
    • wireguard
    • token
    • secret
    • password
    • Authorization
    • cookie
    • client_secret

Incident reference

A real leak occurred on 2026-05-22 when a raw UniFi baseline export under home/doris-dashboard/docs/baselines/ was committed with embedded WireGuard secrets.

See:

  • docs/operations/INCIDENT_2026-05-22_UNIFI_BASELINE_SECRET_LEAK.md
  • home/doris-dashboard/docs/baselines/README.md

Git hook + CI guardrails for infra artifacts

The repo now includes layered guardrails aimed specifically at catching secret-bearing controller/export artifacts before they leave a workstation.

Files:

  • scripts/scan-secret-bearing-artifacts.sh
  • .githooks/pre-commit
  • .githooks/pre-push
  • .github/workflows/secret-guardrails.yml

Enable the hooks once per clone:

git config core.hooksPath .githooks
chmod +x .githooks/pre-commit .githooks/pre-push scripts/scan-secret-bearing-artifacts.sh

Manual runs:

scripts/scan-secret-bearing-artifacts.sh --staged
scripts/scan-secret-bearing-artifacts.sh --tracked
scripts/scan-secret-bearing-artifacts.sh --git-range origin/main..HEAD

What each layer does:

  • pre-commit: scans staged files before a local commit is created
  • pre-push: scans the commits about to be pushed, including unpublished branch history
  • GitHub Actions: runs the narrow artifact scan across tracked files and also runs Gitleaks on pushes and pull requests

What it is for:

  • raw baselines
  • exports
  • snapshots
  • dumps
  • similar machine-generated infra artifacts

What it is not:

  • a full replacement for operator judgment about what belongs in the repo
  • proof that a raw export is safe just because the scanner stayed quiet
  • a reason to skip rotating live secrets if one ever does leak

Security Reminders

  • Key backup: C:\Users\Fizzlepoof\Downloads\.git-crypt-secrets.key — also store in password manager
  • Regenerate Wings token on N.O.M.A.D. (was exposed in chat — TODO)
  • Regenerate N.O.M.A.D. Newt secret (was exposed in chat — TODO)
  • Disable signups in OpenWebUI after creating initial account
  • The Gitea token in setup-secrets-repo.sh should be rotated after initial setup
Reference in New Issue View Git Blame Copy Permalink