From 33d0666a3e244ba83f7ef225cea3a44f0c254fb5 Mon Sep 17 00:00:00 2001 From: Fizzlepoof Date: Tue, 12 May 2026 08:30:12 -0500 Subject: [PATCH] docs: expand SECRETS_MANAGEMENT with full runbook and sync workflow --- docs/operations/SECRETS_MANAGEMENT.md | 177 ++++++++++++++++++-------- 1 file changed, 124 insertions(+), 53 deletions(-) diff --git a/docs/operations/SECRETS_MANAGEMENT.md b/docs/operations/SECRETS_MANAGEMENT.md index 52c6f00..443306b 100644 --- a/docs/operations/SECRETS_MANAGEMENT.md +++ b/docs/operations/SECRETS_MANAGEMENT.md @@ -1,11 +1,25 @@ # Secrets Management +## Overview + +Secrets are managed in two layers: + +1. **Live secrets** — `.env` files on PD at `/mnt/docker-ssd/docker/compose//.env`, gitignored and never committed to the main repo +2. **Encrypted backup** — all `.env` files are synced to a private git-crypt encrypted Gitea repo at `/mnt/docker-ssd/docker/secrets/` + +The sync is a one-command operation and is safe to re-run at any time. + +--- + ## Principles -- `.env` files are gitignored and never committed to any public repo -- Real `.env` files live only on the host at the stack directory -- `.env.example` files are committed with placeholder values + +- `.env` files are gitignored in the main repo — never committed there +- `.env.example` files with placeholder values are committed instead +- The secrets repo encrypts everything at rest using git-crypt (symmetric key) - Secrets are generated with standard tools, never reused across services +--- + ## Generating Secrets ```bash @@ -19,113 +33,170 @@ openssl rand -hex 32 openssl rand -base64 32 ``` -## .env File Locations +--- + +## Live .env File Locations + All `.env` files live alongside their `docker-compose.yaml`: + ``` -/mnt/docker-ssd/docker/compose//.env +/mnt/docker-ssd/docker/compose/ + ai/.env + automation/.env + databases/.env + dev/.env + documents/.env + home/.env + infrastructure/.env + media/.env + media/kima-hub/.env + mesh-mqtt-observer/.env + meshtastic/.env + monitoring/.env + photos/.env ``` --- ## Secrets Repo (git-crypt) -All secrets are backed up to a **private, git-crypt encrypted** Gitea repo: - | | | |---|---| -| **Repo** | `https://gitea.paccoco.com/fizzlepoof/homelab-secrets` (private) | -| **Local path** | `/mnt/docker-ssd/docker/secrets/` | -| **Symmetric key** | `/root/.git-crypt-secrets.key` on PD | -| **Key backup** | Password manager / USB — **do not lose this** | +| **Gitea repo** | `https://gitea.paccoco.com/fizzlepoof/homelab-secrets` (private) | +| **SSH remote** | `ssh://git@10.5.1.6:2222/fizzlepoof/homelab-secrets.git` | +| **Local path on PD** | `/mnt/docker-ssd/docker/secrets/` | +| **Symmetric key location** | `/mnt/docker-ssd/docker/secrets/.git-crypt-secrets.key` (if copied there) or retrieve from password manager | +| **Key backup** | Password manager + `C:\Users\Fizzlepoof\Downloads\.git-crypt-secrets.key` | -### Directory structure +### Repo structure | Directory | Contents | |-----------|----------| -| `env/` | `.env` files for each stack | +| `env/` | `.env` files for each stack, named `.env` | | `keys/` | API keys and tokens | | `certs/` | TLS certificates | | `tokens/` | Service tokens (Paperless, LiteLLM, etc.) | | `ssh/` | SSH private keys | -### Adding a secret +All files except `README.md` and `.gitattributes` are encrypted by git-crypt on commit. + +--- + +## Syncing .env Files to the Secrets Repo + +Use the sync script — no root required, safe to re-run: ```bash -# As root on PD: -cp /mnt/docker-ssd/docker/compose/ai/.env /mnt/docker-ssd/docker/secrets/env/ai.env -cd /mnt/docker-ssd/docker/secrets -git add -A -git -c user.name="Fizzlepoof" -c user.email="admin@paccoco.com" commit -m "chore: add ai stack env" -git push +bash /mnt/docker-ssd/docker/compose/scripts/sync-envs-to-secrets.sh ``` -Files are encrypted automatically by git-crypt on push — no extra steps needed. +What it does: +- Scans all stack directories under `/mnt/docker-ssd/docker/compose/` for `.env` files +- Copies changed/new files to `/mnt/docker-ssd/docker/secrets/env/` as `.env` +- Skips unchanged files +- Commits and pushes to Gitea automatically -### Unlocking on a new machine +Run this any time you create or update a stack's `.env` file. + +--- + +## git-crypt on TrueNAS + +apt is blocked on TrueNAS Scale. git-crypt is installed via Docker and stored on the persistent dataset at `/mnt/docker-ssd/bin/git-crypt`. + +### Current installation (already done on PD) + +``` +/mnt/docker-ssd/bin/git-crypt +``` + +PATH is set in `/home/truenas_admin/.bashrc` and `/root/.bashrc`: ```bash -git clone https://gitea.paccoco.com/fizzlepoof/homelab-secrets -cd homelab-secrets -git-crypt unlock /path/to/.git-crypt-secrets.key +export PATH="/mnt/docker-ssd/bin:$PATH" +``` + +### Re-installing after a full rebuild + +**Option 1 — one-liner:** +```bash +sudo -i +docker run --rm -v /tmp:/out debian:bookworm-slim \ + bash -c "apt-get update -qq && apt-get install -y -qq git-crypt && cp /usr/bin/git-crypt /out/git-crypt" +mkdir -p /mnt/docker-ssd/bin +cp /tmp/git-crypt /mnt/docker-ssd/bin/git-crypt +chmod +x /mnt/docker-ssd/bin/git-crypt +echo 'export PATH="/mnt/docker-ssd/bin:$PATH"' >> /root/.bashrc +echo 'export PATH="/mnt/docker-ssd/bin:$PATH"' >> /home/truenas_admin/.bashrc +``` + +**Option 2 — via dev stack compose:** +```bash +cd /mnt/docker-ssd/docker/compose/dev +sudo docker compose --profile setup up git-crypt-init +# copies git-crypt to /root/bin — then move to persistent location: +sudo cp /root/bin/git-crypt /mnt/docker-ssd/bin/git-crypt ``` --- -## Installing git-crypt on TrueNAS (apt is blocked) +## Unlocking the Secrets Repo on a New Machine -### Option 1 — one-liner (as root) ```bash -docker run --rm -v /tmp:/out debian:bookworm-slim \ - bash -c "apt-get update -qq && apt-get install -y -qq git-crypt && cp /usr/bin/git-crypt /out/git-crypt" -mkdir -p /root/bin && cp /tmp/git-crypt /root/bin/git-crypt && chmod +x /root/bin/git-crypt -export PATH="/root/bin:$PATH" +# Install git-crypt first (see above), then: +git clone ssh://git@10.5.1.6:2222/fizzlepoof/homelab-secrets.git /mnt/docker-ssd/docker/secrets +cd /mnt/docker-ssd/docker/secrets +git-crypt unlock /path/to/.git-crypt-secrets.key ``` -### Option 2 — via Gitea stack compose (recommended for rebuilds) -```bash -cd /mnt/docker-ssd/docker/compose/dev -docker compose --profile setup up git-crypt-init -export PATH="/root/bin:$PATH" -``` - -The `git-crypt-init` service in `dev/docker-compose.yaml` handles the install automatically. - -### Make PATH persistent (add to /root/.bashrc) -```bash -echo 'export PATH="/root/bin:$PATH"' >> /root/.bashrc -``` +After unlock, the `env/` directory contains plaintext `.env` files ready to copy back into the stack directories. --- ## Full Rebuild Runbook -On a fresh PD: +On a fresh PD after reinstalling TrueNAS: -1. Install git-crypt (Option 1 or 2 above) -2. Clone and unlock the secrets repo: +1. **Install git-crypt** (Option 1 above — docker is available immediately after TrueNAS install) +2. **Restore secrets repo:** ```bash - git clone https://gitea.paccoco.com/fizzlepoof/homelab-secrets /mnt/docker-ssd/docker/secrets + git clone ssh://git@10.5.1.6:2222/fizzlepoof/homelab-secrets.git /mnt/docker-ssd/docker/secrets cd /mnt/docker-ssd/docker/secrets git-crypt unlock /path/to/.git-crypt-secrets.key ``` -3. Copy `.env` files back into their stack directories from `secrets/env/` -4. Redeploy stacks +3. **Restore .env files** from `secrets/env/` back to their stack directories: + ```bash + cp /mnt/docker-ssd/docker/secrets/env/ai.env /mnt/docker-ssd/docker/compose/ai/.env + cp /mnt/docker-ssd/docker/secrets/env/databases.env /mnt/docker-ssd/docker/compose/databases/.env + # ... etc for each stack + ``` +4. **Clone the main stacks repo:** + ```bash + git clone git@gitea.paccoco.com:fizzlepoof/truenas-stacks.git /mnt/docker-ssd/docker/compose + ``` +5. **Redeploy stacks** per `DEPLOYMENT_CHECKLIST.md` --- ## Known Credentials Locations -| Service | Where stored | -|---------|-------------| +| Service | File | +|---------|------| | shared-postgres | `databases/.env` | | shared-mariadb | `databases/.env` | | OpenWebUI DB | `ai/.env` — user: openwebui, db: openwebui | | Donetick | `home/.env` + `selfhosted.yaml` | | Meshmonitor | `meshtastic/.env` | | Gitea | `dev/.env` | +| LiteLLM | `ai/.env` | +| Paperless | `documents/.env` | + +--- ## Security Reminders + +- **Key backup**: `C:\Users\Fizzlepoof\Downloads\.git-crypt-secrets.key` — also store in password manager - Regenerate Wings token on N.O.M.A.D. (was exposed in chat — TODO) - Regenerate N.O.M.A.D. Newt secret (was exposed in chat — TODO) - Disable signups in OpenWebUI after creating initial account -- Back up `/root/.git-crypt-secrets.key` to password manager +- The Gitea token in `setup-secrets-repo.sh` should be rotated after initial setup