280 lines
8.4 KiB
Markdown
280 lines
8.4 KiB
Markdown
# Secrets Management
|
|
|
|
## Overview
|
|
|
|
Secrets are managed in two layers:
|
|
|
|
1. **Live secrets** — `.env` files on PD at `/mnt/docker-ssd/docker/compose/<stack>/.env`, gitignored and never committed to the main repo
|
|
2. **Encrypted backup** — all `.env` files are synced to a private git-crypt encrypted Gitea repo at `/mnt/docker-ssd/docker/secrets/`
|
|
|
|
The sync is a one-command operation and is safe to re-run at any time.
|
|
|
|
---
|
|
|
|
## Principles
|
|
|
|
- `.env` files are gitignored in the main repo — never committed there
|
|
- `.env.example` files with placeholder values are committed instead
|
|
- The secrets repo encrypts everything at rest using git-crypt (symmetric key)
|
|
- Secrets are generated with standard tools, never reused across services
|
|
|
|
---
|
|
|
|
## Generating Secrets
|
|
|
|
```bash
|
|
# Database passwords
|
|
openssl rand -hex 24
|
|
|
|
# JWT / session secrets
|
|
openssl rand -hex 32
|
|
|
|
# API keys (where format allows)
|
|
openssl rand -base64 32
|
|
```
|
|
|
|
---
|
|
|
|
## Live .env File Locations
|
|
|
|
All `.env` files live alongside their `docker-compose.yaml`:
|
|
|
|
```
|
|
/mnt/docker-ssd/docker/compose/
|
|
ai/.env
|
|
automation/.env
|
|
databases/.env
|
|
dev/.env
|
|
documents/.env
|
|
home/.env
|
|
infrastructure/.env
|
|
media/.env
|
|
media/kima-hub/.env
|
|
mesh-mqtt-observer/.env
|
|
meshtastic/.env
|
|
monitoring/.env
|
|
photos/.env
|
|
```
|
|
|
|
---
|
|
|
|
## Secrets Repo (git-crypt)
|
|
|
|
| | |
|
|
|---|---|
|
|
| **Gitea repo** | `https://gitea.paccoco.com/fizzlepoof/homelab-secrets` (private) |
|
|
| **SSH remote** | `ssh://git@10.5.30.6:2222/fizzlepoof/homelab-secrets.git` |
|
|
| **Local path on PD** | `/mnt/docker-ssd/docker/secrets/` |
|
|
| **Symmetric key location** | `/mnt/docker-ssd/docker/secrets/.git-crypt-secrets.key` (if copied there) or retrieve from password manager |
|
|
| **Key backup** | Password manager + `C:\Users\Fizzlepoof\Downloads\.git-crypt-secrets.key` |
|
|
|
|
### Repo structure
|
|
|
|
| Directory | Contents |
|
|
|-----------|----------|
|
|
| `env/` | `.env` files for each stack, named `<stack>.env` |
|
|
| `keys/` | API keys and tokens |
|
|
| `certs/` | TLS certificates |
|
|
| `tokens/` | Service tokens (Paperless, LiteLLM, etc.) |
|
|
| `ssh/` | SSH private keys |
|
|
|
|
All files except `README.md` and `.gitattributes` are encrypted by git-crypt on commit.
|
|
|
|
---
|
|
|
|
## Syncing .env Files to the Secrets Repo
|
|
|
|
Use the sync script — no root required, safe to re-run:
|
|
|
|
```bash
|
|
bash /mnt/docker-ssd/docker/compose/scripts/sync-envs-to-secrets.sh
|
|
```
|
|
|
|
What it does:
|
|
- Scans all stack directories under `/mnt/docker-ssd/docker/compose/` for `.env` files
|
|
- Copies changed/new files to `/mnt/docker-ssd/docker/secrets/env/` as `<stack>.env`
|
|
- Skips unchanged files
|
|
- Commits and pushes to Gitea automatically
|
|
|
|
Run this any time you create or update a stack's `.env` file.
|
|
|
|
**Operational rule:** after any live `.env` change on PD, Doris or the operator must run the sync script and confirm whether it committed/pushed changes or reported the secrets repo already up to date. Do not treat live `.env` edits as complete until this sync step is checked.
|
|
|
|
---
|
|
|
|
## git-crypt on TrueNAS
|
|
|
|
apt is blocked on TrueNAS Scale. git-crypt is installed via Docker.
|
|
|
|
### Preferred installation location
|
|
|
|
```
|
|
/mnt/docker-ssd/bin/git-crypt
|
|
```
|
|
|
|
### Compatibility note
|
|
|
|
The original bootstrap script installs to `/root/bin/git-crypt`, while the newer runbook expects `/mnt/docker-ssd/bin/git-crypt`. Standardize on `/mnt/docker-ssd/bin/git-crypt` and make sure PATH includes that directory for both `root` and `truenas_admin`.
|
|
|
|
Recommended PATH entry:
|
|
|
|
```bash
|
|
export PATH="/mnt/docker-ssd/bin:$PATH"
|
|
```
|
|
|
|
### Re-installing after a full rebuild
|
|
|
|
**Option 1 — one-liner:**
|
|
```bash
|
|
sudo -i
|
|
docker run --rm -v /tmp:/out debian:bookworm-slim \
|
|
bash -c "apt-get update -qq && apt-get install -y -qq git-crypt && cp /usr/bin/git-crypt /out/git-crypt"
|
|
mkdir -p /mnt/docker-ssd/bin
|
|
cp /tmp/git-crypt /mnt/docker-ssd/bin/git-crypt
|
|
chmod +x /mnt/docker-ssd/bin/git-crypt
|
|
echo 'export PATH="/mnt/docker-ssd/bin:$PATH"' >> /root/.bashrc
|
|
echo 'export PATH="/mnt/docker-ssd/bin:$PATH"' >> /home/truenas_admin/.bashrc
|
|
```
|
|
|
|
**Option 2 — via dev stack compose:**
|
|
```bash
|
|
cd /mnt/docker-ssd/docker/compose/dev
|
|
sudo docker compose --profile setup up git-crypt-init
|
|
# copies git-crypt to /root/bin — then move to persistent location:
|
|
sudo cp /root/bin/git-crypt /mnt/docker-ssd/bin/git-crypt
|
|
```
|
|
|
|
---
|
|
|
|
## Unlocking the Secrets Repo on a New Machine
|
|
|
|
```bash
|
|
# Install git-crypt first (see above), then:
|
|
git clone ssh://git@10.5.30.6:2222/fizzlepoof/homelab-secrets.git /mnt/docker-ssd/docker/secrets
|
|
cd /mnt/docker-ssd/docker/secrets
|
|
git-crypt unlock /path/to/.git-crypt-secrets.key
|
|
```
|
|
|
|
After unlock, the `env/` directory contains plaintext `.env` files ready to copy back into the stack directories.
|
|
|
|
---
|
|
|
|
## Full Rebuild Runbook
|
|
|
|
On a fresh PD after reinstalling TrueNAS:
|
|
|
|
1. **Install git-crypt** (Option 1 above — docker is available immediately after TrueNAS install)
|
|
2. **Restore secrets repo:**
|
|
```bash
|
|
git clone ssh://git@10.5.30.6:2222/fizzlepoof/homelab-secrets.git /mnt/docker-ssd/docker/secrets
|
|
cd /mnt/docker-ssd/docker/secrets
|
|
git-crypt unlock /path/to/.git-crypt-secrets.key
|
|
```
|
|
3. **Restore .env files** from `secrets/env/` back to their stack directories:
|
|
```bash
|
|
cp /mnt/docker-ssd/docker/secrets/env/ai.env /mnt/docker-ssd/docker/compose/ai/.env
|
|
cp /mnt/docker-ssd/docker/secrets/env/databases.env /mnt/docker-ssd/docker/compose/databases/.env
|
|
# ... etc for each stack
|
|
```
|
|
4. **Clone the main stacks repo:**
|
|
```bash
|
|
git clone ssh://git@10.5.30.6:2222/fizzlepoof/truenas-stacks.git /mnt/docker-ssd/docker/compose
|
|
```
|
|
5. **Redeploy stacks** per `DEPLOYMENT_CHECKLIST.md`
|
|
6. **After any future live `.env` edit**, immediately run:
|
|
```bash
|
|
export PATH="/mnt/docker-ssd/bin:$PATH"
|
|
bash /mnt/docker-ssd/docker/compose/scripts/sync-envs-to-secrets.sh
|
|
```
|
|
Then verify the secrets repo is either updated and pushed or already unchanged.
|
|
|
|
---
|
|
|
|
## Known Credentials Locations
|
|
|
|
| Service | File |
|
|
|---------|------|
|
|
| shared-postgres | `databases/.env` |
|
|
| shared-mariadb | `databases/.env` |
|
|
| OpenWebUI DB | `ai/.env` — user: openwebui, db: openwebui |
|
|
| Donetick | `home/.env` + `selfhosted.yaml` |
|
|
| Meshmonitor | `meshtastic/.env` |
|
|
| Gitea | `dev/.env` |
|
|
| LiteLLM | `ai/.env` |
|
|
| Paperless | `documents/.env` |
|
|
| UniFi Doris operator | `automation/.env` |
|
|
|
|
---
|
|
|
|
## Controller Export / Baseline Artifact Rules
|
|
|
|
Raw infrastructure exports must be treated as potentially secret-bearing even when they are not `.env` files.
|
|
|
|
Examples:
|
|
- UniFi `networkconf` / `portconf` dumps
|
|
- firewall/router/controller JSON exports
|
|
- VPN client or server config exports
|
|
- diagnostic snapshots taken from appliances or operator APIs
|
|
|
|
### Permanent rules
|
|
|
|
- Do **not** commit raw controller or appliance exports to the main repo.
|
|
- Commit only sanitized markdown summaries or explicitly redacted artifacts.
|
|
- If raw retention is needed, store the export in the encrypted secrets repo or another operator-only location outside the main repo.
|
|
- Before committing infra artifacts, scan for obvious secret-bearing keys such as:
|
|
- `private_key`
|
|
- `wireguard`
|
|
- `token`
|
|
- `secret`
|
|
- `password`
|
|
- `Authorization`
|
|
- `cookie`
|
|
- `client_secret`
|
|
|
|
### Incident reference
|
|
|
|
A real leak occurred on 2026-05-22 when a raw UniFi baseline export under `home/doris-dashboard/docs/baselines/` was committed with embedded WireGuard secrets.
|
|
|
|
See:
|
|
- `docs/operations/INCIDENT_2026-05-22_UNIFI_BASELINE_SECRET_LEAK.md`
|
|
- `home/doris-dashboard/docs/baselines/README.md`
|
|
|
|
### Pre-commit guardrail for infra artifacts
|
|
|
|
The repo now includes a narrow pre-commit scanner aimed specifically at catching secret-bearing controller/export artifacts before push.
|
|
|
|
Files:
|
|
- `scripts/scan-secret-bearing-artifacts.sh`
|
|
- `.githooks/pre-commit`
|
|
|
|
Enable it once per clone:
|
|
|
|
```bash
|
|
git config core.hooksPath .githooks
|
|
chmod +x .githooks/pre-commit scripts/scan-secret-bearing-artifacts.sh
|
|
```
|
|
|
|
Run it manually on staged changes:
|
|
|
|
```bash
|
|
scripts/scan-secret-bearing-artifacts.sh --staged
|
|
```
|
|
|
|
What it is for:
|
|
- raw baselines
|
|
- exports
|
|
- snapshots
|
|
- dumps
|
|
- similar machine-generated infra artifacts
|
|
|
|
What it is not:
|
|
- a full replacement for GitHub/GitGuardian or a generic secret scanner for every file type
|
|
|
|
## Security Reminders
|
|
|
|
- **Key backup**: `C:\Users\Fizzlepoof\Downloads\.git-crypt-secrets.key` — also store in password manager
|
|
- Regenerate Wings token on N.O.M.A.D. (was exposed in chat — TODO)
|
|
- Regenerate N.O.M.A.D. Newt secret (was exposed in chat — TODO)
|
|
- Disable signups in OpenWebUI after creating initial account
|
|
- The Gitea token in `setup-secrets-repo.sh` should be rotated after initial setup
|