fizzlepoof/truenas-stacks
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: expand SECRETS_MANAGEMENT with full runbook and sync workflow

Browse Source
This commit is contained in:
Fizzlepoof
2026-05-12 08:30:12 -05:00
parent e40fd48547
commit 30e01a9a09
1 changed files with 124 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files

177
docs/operations/SECRETS_MANAGEMENT.md
View File

@@ -1,11 +1,25 @@
# Secrets Management # Secrets Management
## Overview
Secrets are managed in two layers:
1. **Live secrets** — `.env` files on PD at `/mnt/docker-ssd/docker/compose/<stack>/.env`, gitignored and never committed to the main repo
2. **Encrypted backup** — all `.env` files are synced to a private git-crypt encrypted Gitea repo at `/mnt/docker-ssd/docker/secrets/`
The sync is a one-command operation and is safe to re-run at any time.
---
## Principles ## Principles
- `.env` files are gitignored and never committed to any public repo
- Real `.env` files live only on the host at the stack directory - `.env` files are gitignored in the main repo — never committed there
- `.env.example` files are committed with placeholder values - `.env.example` files with placeholder values are committed instead
- The secrets repo encrypts everything at rest using git-crypt (symmetric key)
- Secrets are generated with standard tools, never reused across services - Secrets are generated with standard tools, never reused across services
---
## Generating Secrets ## Generating Secrets
```bash ```bash
@@ -19,113 +33,170 @@ openssl rand -hex 32
openssl rand -base64 32 openssl rand -base64 32
``` ```
## .env File Locations ---
## Live .env File Locations
All `.env` files live alongside their `docker-compose.yaml`: All `.env` files live alongside their `docker-compose.yaml`:
``` ```
/mnt/docker-ssd/docker/compose/<stack>/.env /mnt/docker-ssd/docker/compose/
ai/.env
automation/.env
databases/.env
dev/.env
documents/.env
home/.env
infrastructure/.env
media/.env
media/kima-hub/.env
mesh-mqtt-observer/.env
meshtastic/.env
monitoring/.env
photos/.env
``` ```
--- ---
## Secrets Repo (git-crypt) ## Secrets Repo (git-crypt)
All secrets are backed up to a **private, git-crypt encrypted** Gitea repo:
| | | | | |
|---|---| |---|---|
| **Repo** | `https://gitea.paccoco.com/fizzlepoof/homelab-secrets` (private) | | **Gitea repo** | `https://gitea.paccoco.com/fizzlepoof/homelab-secrets` (private) |
| **Local path** | `/mnt/docker-ssd/docker/secrets/` | | **SSH remote** | `ssh://git@10.5.1.6:2222/fizzlepoof/homelab-secrets.git` |
| **Symmetric key** | `/root/.git-crypt-secrets.key` on PD | | **Local path on PD** | `/mnt/docker-ssd/docker/secrets/` |
| **Key backup** | Password manager / USB — **do not lose this** | | **Symmetric key location** | `/mnt/docker-ssd/docker/secrets/.git-crypt-secrets.key` (if copied there) or retrieve from password manager |
| **Key backup** | Password manager + `C:\Users\Fizzlepoof\Downloads\.git-crypt-secrets.key` |
### Directory structure ### Repo structure
| Directory | Contents | | Directory | Contents |
|-----------|----------| |-----------|----------|
| `env/` | `.env` files for each stack | | `env/` | `.env` files for each stack, named `<stack>.env` |
| `keys/` | API keys and tokens | | `keys/` | API keys and tokens |
| `certs/` | TLS certificates | | `certs/` | TLS certificates |
| `tokens/` | Service tokens (Paperless, LiteLLM, etc.) | | `tokens/` | Service tokens (Paperless, LiteLLM, etc.) |
| `ssh/` | SSH private keys | | `ssh/` | SSH private keys |
### Adding a secret All files except `README.md` and `.gitattributes` are encrypted by git-crypt on commit.
---
## Syncing .env Files to the Secrets Repo
Use the sync script — no root required, safe to re-run:
```bash ```bash
# As root on PD: bash /mnt/docker-ssd/docker/compose/scripts/sync-envs-to-secrets.sh
cp /mnt/docker-ssd/docker/compose/ai/.env /mnt/docker-ssd/docker/secrets/env/ai.env
cd /mnt/docker-ssd/docker/secrets
git add -A
git -c user.name="Fizzlepoof" -c user.email="admin@paccoco.com" commit -m "chore: add ai stack env"
git push
``` ```
Files are encrypted automatically by git-crypt on push — no extra steps needed. What it does:
- Scans all stack directories under `/mnt/docker-ssd/docker/compose/` for `.env` files
- Copies changed/new files to `/mnt/docker-ssd/docker/secrets/env/` as `<stack>.env`
- Skips unchanged files
- Commits and pushes to Gitea automatically
### Unlocking on a new machine Run this any time you create or update a stack's `.env` file.
---
## git-crypt on TrueNAS
apt is blocked on TrueNAS Scale. git-crypt is installed via Docker and stored on the persistent dataset at `/mnt/docker-ssd/bin/git-crypt`.
### Current installation (already done on PD)
```
/mnt/docker-ssd/bin/git-crypt
```
PATH is set in `/home/truenas_admin/.bashrc` and `/root/.bashrc`:
```bash ```bash
git clone https://gitea.paccoco.com/fizzlepoof/homelab-secrets export PATH="/mnt/docker-ssd/bin:$PATH"
cd homelab-secrets ```
git-crypt unlock /path/to/.git-crypt-secrets.key
### Re-installing after a full rebuild
**Option 1 — one-liner:**
```bash
sudo -i
docker run --rm -v /tmp:/out debian:bookworm-slim \
bash -c "apt-get update -qq && apt-get install -y -qq git-crypt && cp /usr/bin/git-crypt /out/git-crypt"
mkdir -p /mnt/docker-ssd/bin
cp /tmp/git-crypt /mnt/docker-ssd/bin/git-crypt
chmod +x /mnt/docker-ssd/bin/git-crypt
echo 'export PATH="/mnt/docker-ssd/bin:$PATH"' >> /root/.bashrc
echo 'export PATH="/mnt/docker-ssd/bin:$PATH"' >> /home/truenas_admin/.bashrc
```
**Option 2 — via dev stack compose:**
```bash
cd /mnt/docker-ssd/docker/compose/dev
sudo docker compose --profile setup up git-crypt-init
# copies git-crypt to /root/bin — then move to persistent location:
sudo cp /root/bin/git-crypt /mnt/docker-ssd/bin/git-crypt
``` ```
--- ---
## Installing git-crypt on TrueNAS (apt is blocked) ## Unlocking the Secrets Repo on a New Machine
### Option 1 — one-liner (as root)
```bash ```bash
docker run --rm -v /tmp:/out debian:bookworm-slim \ # Install git-crypt first (see above), then:
bash -c "apt-get update -qq && apt-get install -y -qq git-crypt && cp /usr/bin/git-crypt /out/git-crypt" git clone ssh://git@10.5.1.6:2222/fizzlepoof/homelab-secrets.git /mnt/docker-ssd/docker/secrets
mkdir -p /root/bin && cp /tmp/git-crypt /root/bin/git-crypt && chmod +x /root/bin/git-crypt cd /mnt/docker-ssd/docker/secrets
export PATH="/root/bin:$PATH" git-crypt unlock /path/to/.git-crypt-secrets.key
``` ```
### Option 2 — via Gitea stack compose (recommended for rebuilds) After unlock, the `env/` directory contains plaintext `.env` files ready to copy back into the stack directories.
```bash
cd /mnt/docker-ssd/docker/compose/dev
docker compose --profile setup up git-crypt-init
export PATH="/root/bin:$PATH"
```
The `git-crypt-init` service in `dev/docker-compose.yaml` handles the install automatically.
### Make PATH persistent (add to /root/.bashrc)
```bash
echo 'export PATH="/root/bin:$PATH"' >> /root/.bashrc
```
--- ---
## Full Rebuild Runbook ## Full Rebuild Runbook
On a fresh PD: On a fresh PD after reinstalling TrueNAS:
1. Install git-crypt (Option 1 or 2 above) 1. **Install git-crypt** (Option 1 above — docker is available immediately after TrueNAS install)
2. Clone and unlock the secrets repo: 2. **Restore secrets repo:**
```bash ```bash
git clone https://gitea.paccoco.com/fizzlepoof/homelab-secrets /mnt/docker-ssd/docker/secrets git clone ssh://git@10.5.1.6:2222/fizzlepoof/homelab-secrets.git /mnt/docker-ssd/docker/secrets
cd /mnt/docker-ssd/docker/secrets cd /mnt/docker-ssd/docker/secrets
git-crypt unlock /path/to/.git-crypt-secrets.key git-crypt unlock /path/to/.git-crypt-secrets.key
``` ```
3. Copy `.env` files back into their stack directories from `secrets/env/` 3. **Restore .env files** from `secrets/env/` back to their stack directories:
4. Redeploy stacks ```bash
cp /mnt/docker-ssd/docker/secrets/env/ai.env /mnt/docker-ssd/docker/compose/ai/.env
cp /mnt/docker-ssd/docker/secrets/env/databases.env /mnt/docker-ssd/docker/compose/databases/.env
# ... etc for each stack
```
4. **Clone the main stacks repo:**
```bash
git clone git@gitea.paccoco.com:fizzlepoof/truenas-stacks.git /mnt/docker-ssd/docker/compose
```
5. **Redeploy stacks** per `DEPLOYMENT_CHECKLIST.md`
--- ---
## Known Credentials Locations ## Known Credentials Locations
| Service | Where stored | | Service | File |
|---------|-------------| |---------|------|
| shared-postgres | `databases/.env` | | shared-postgres | `databases/.env` |
| shared-mariadb | `databases/.env` | | shared-mariadb | `databases/.env` |
| OpenWebUI DB | `ai/.env` — user: openwebui, db: openwebui | | OpenWebUI DB | `ai/.env` — user: openwebui, db: openwebui |
| Donetick | `home/.env` + `selfhosted.yaml` | | Donetick | `home/.env` + `selfhosted.yaml` |
| Meshmonitor | `meshtastic/.env` | | Meshmonitor | `meshtastic/.env` |
| Gitea | `dev/.env` | | Gitea | `dev/.env` |
| LiteLLM | `ai/.env` |
| Paperless | `documents/.env` |
---
## Security Reminders ## Security Reminders
- **Key backup**: `C:\Users\Fizzlepoof\Downloads\.git-crypt-secrets.key` — also store in password manager
- Regenerate Wings token on N.O.M.A.D. (was exposed in chat — TODO) - Regenerate Wings token on N.O.M.A.D. (was exposed in chat — TODO)
- Regenerate N.O.M.A.D. Newt secret (was exposed in chat — TODO) - Regenerate N.O.M.A.D. Newt secret (was exposed in chat — TODO)
- Disable signups in OpenWebUI after creating initial account - Disable signups in OpenWebUI after creating initial account
- Back up `/root/.git-crypt-secrets.key` to password manager - The Gitea token in `setup-secrets-repo.sh` should be rotated after initial setup