docs: expand SECRETS_MANAGEMENT with full runbook and sync workflow
This commit is contained in:
Fizzlepoof
@@ -1,11 +1,25 @@
|
||||
# Secrets Management
|
||||
|
||||
## Overview
|
||||
|
||||
Secrets are managed in two layers:
|
||||
|
||||
1. **Live secrets** — `.env` files on PD at `/mnt/docker-ssd/docker/compose/<stack>/.env`, gitignored and never committed to the main repo
|
||||
2. **Encrypted backup** — all `.env` files are synced to a private git-crypt encrypted Gitea repo at `/mnt/docker-ssd/docker/secrets/`
|
||||
|
||||
The sync is a one-command operation and is safe to re-run at any time.
|
||||
|
||||
---
|
||||
|
||||
## Principles
|
||||
- `.env` files are gitignored and never committed to any public repo
|
||||
- Real `.env` files live only on the host at the stack directory
|
||||
- `.env.example` files are committed with placeholder values
|
||||
|
||||
- `.env` files are gitignored in the main repo — never committed there
|
||||
- `.env.example` files with placeholder values are committed instead
|
||||
- The secrets repo encrypts everything at rest using git-crypt (symmetric key)
|
||||
- Secrets are generated with standard tools, never reused across services
|
||||
|
||||
---
|
||||
|
||||
## Generating Secrets
|
||||
|
||||
```bash
|
||||
@@ -19,113 +33,170 @@ openssl rand -hex 32
|
||||
openssl rand -base64 32
|
||||
```
|
||||
|
||||
## .env File Locations
|
||||
---
|
||||
|
||||
## Live .env File Locations
|
||||
|
||||
All `.env` files live alongside their `docker-compose.yaml`:
|
||||
|
||||
```
|
||||
/mnt/docker-ssd/docker/compose/<stack>/.env
|
||||
/mnt/docker-ssd/docker/compose/
|
||||
ai/.env
|
||||
automation/.env
|
||||
databases/.env
|
||||
dev/.env
|
||||
documents/.env
|
||||
home/.env
|
||||
infrastructure/.env
|
||||
media/.env
|
||||
media/kima-hub/.env
|
||||
mesh-mqtt-observer/.env
|
||||
meshtastic/.env
|
||||
monitoring/.env
|
||||
photos/.env
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Secrets Repo (git-crypt)
|
||||
|
||||
All secrets are backed up to a **private, git-crypt encrypted** Gitea repo:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Repo** | `https://gitea.paccoco.com/fizzlepoof/homelab-secrets` (private) |
|
||||
| **Local path** | `/mnt/docker-ssd/docker/secrets/` |
|
||||
| **Symmetric key** | `/root/.git-crypt-secrets.key` on PD |
|
||||
| **Key backup** | Password manager / USB — **do not lose this** |
|
||||
| **Gitea repo** | `https://gitea.paccoco.com/fizzlepoof/homelab-secrets` (private) |
|
||||
| **SSH remote** | `ssh://git@10.5.1.6:2222/fizzlepoof/homelab-secrets.git` |
|
||||
| **Local path on PD** | `/mnt/docker-ssd/docker/secrets/` |
|
||||
| **Symmetric key location** | `/mnt/docker-ssd/docker/secrets/.git-crypt-secrets.key` (if copied there) or retrieve from password manager |
|
||||
| **Key backup** | Password manager + `C:\Users\Fizzlepoof\Downloads\.git-crypt-secrets.key` |
|
||||
|
||||
### Directory structure
|
||||
### Repo structure
|
||||
|
||||
| Directory | Contents |
|
||||
|-----------|----------|
|
||||
| `env/` | `.env` files for each stack |
|
||||
| `env/` | `.env` files for each stack, named `<stack>.env` |
|
||||
| `keys/` | API keys and tokens |
|
||||
| `certs/` | TLS certificates |
|
||||
| `tokens/` | Service tokens (Paperless, LiteLLM, etc.) |
|
||||
| `ssh/` | SSH private keys |
|
||||
|
||||
### Adding a secret
|
||||
All files except `README.md` and `.gitattributes` are encrypted by git-crypt on commit.
|
||||
|
||||
---
|
||||
|
||||
## Syncing .env Files to the Secrets Repo
|
||||
|
||||
Use the sync script — no root required, safe to re-run:
|
||||
|
||||
```bash
|
||||
# As root on PD:
|
||||
cp /mnt/docker-ssd/docker/compose/ai/.env /mnt/docker-ssd/docker/secrets/env/ai.env
|
||||
cd /mnt/docker-ssd/docker/secrets
|
||||
git add -A
|
||||
git -c user.name="Fizzlepoof" -c user.email="admin@paccoco.com" commit -m "chore: add ai stack env"
|
||||
git push
|
||||
bash /mnt/docker-ssd/docker/compose/scripts/sync-envs-to-secrets.sh
|
||||
```
|
||||
|
||||
Files are encrypted automatically by git-crypt on push — no extra steps needed.
|
||||
What it does:
|
||||
- Scans all stack directories under `/mnt/docker-ssd/docker/compose/` for `.env` files
|
||||
- Copies changed/new files to `/mnt/docker-ssd/docker/secrets/env/` as `<stack>.env`
|
||||
- Skips unchanged files
|
||||
- Commits and pushes to Gitea automatically
|
||||
|
||||
### Unlocking on a new machine
|
||||
Run this any time you create or update a stack's `.env` file.
|
||||
|
||||
---
|
||||
|
||||
## git-crypt on TrueNAS
|
||||
|
||||
apt is blocked on TrueNAS Scale. git-crypt is installed via Docker and stored on the persistent dataset at `/mnt/docker-ssd/bin/git-crypt`.
|
||||
|
||||
### Current installation (already done on PD)
|
||||
|
||||
```
|
||||
/mnt/docker-ssd/bin/git-crypt
|
||||
```
|
||||
|
||||
PATH is set in `/home/truenas_admin/.bashrc` and `/root/.bashrc`:
|
||||
|
||||
```bash
|
||||
git clone https://gitea.paccoco.com/fizzlepoof/homelab-secrets
|
||||
cd homelab-secrets
|
||||
git-crypt unlock /path/to/.git-crypt-secrets.key
|
||||
export PATH="/mnt/docker-ssd/bin:$PATH"
|
||||
```
|
||||
|
||||
### Re-installing after a full rebuild
|
||||
|
||||
**Option 1 — one-liner:**
|
||||
```bash
|
||||
sudo -i
|
||||
docker run --rm -v /tmp:/out debian:bookworm-slim \
|
||||
bash -c "apt-get update -qq && apt-get install -y -qq git-crypt && cp /usr/bin/git-crypt /out/git-crypt"
|
||||
mkdir -p /mnt/docker-ssd/bin
|
||||
cp /tmp/git-crypt /mnt/docker-ssd/bin/git-crypt
|
||||
chmod +x /mnt/docker-ssd/bin/git-crypt
|
||||
echo 'export PATH="/mnt/docker-ssd/bin:$PATH"' >> /root/.bashrc
|
||||
echo 'export PATH="/mnt/docker-ssd/bin:$PATH"' >> /home/truenas_admin/.bashrc
|
||||
```
|
||||
|
||||
**Option 2 — via dev stack compose:**
|
||||
```bash
|
||||
cd /mnt/docker-ssd/docker/compose/dev
|
||||
sudo docker compose --profile setup up git-crypt-init
|
||||
# copies git-crypt to /root/bin — then move to persistent location:
|
||||
sudo cp /root/bin/git-crypt /mnt/docker-ssd/bin/git-crypt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Installing git-crypt on TrueNAS (apt is blocked)
|
||||
## Unlocking the Secrets Repo on a New Machine
|
||||
|
||||
### Option 1 — one-liner (as root)
|
||||
```bash
|
||||
docker run --rm -v /tmp:/out debian:bookworm-slim \
|
||||
bash -c "apt-get update -qq && apt-get install -y -qq git-crypt && cp /usr/bin/git-crypt /out/git-crypt"
|
||||
mkdir -p /root/bin && cp /tmp/git-crypt /root/bin/git-crypt && chmod +x /root/bin/git-crypt
|
||||
export PATH="/root/bin:$PATH"
|
||||
# Install git-crypt first (see above), then:
|
||||
git clone ssh://git@10.5.1.6:2222/fizzlepoof/homelab-secrets.git /mnt/docker-ssd/docker/secrets
|
||||
cd /mnt/docker-ssd/docker/secrets
|
||||
git-crypt unlock /path/to/.git-crypt-secrets.key
|
||||
```
|
||||
|
||||
### Option 2 — via Gitea stack compose (recommended for rebuilds)
|
||||
```bash
|
||||
cd /mnt/docker-ssd/docker/compose/dev
|
||||
docker compose --profile setup up git-crypt-init
|
||||
export PATH="/root/bin:$PATH"
|
||||
```
|
||||
|
||||
The `git-crypt-init` service in `dev/docker-compose.yaml` handles the install automatically.
|
||||
|
||||
### Make PATH persistent (add to /root/.bashrc)
|
||||
```bash
|
||||
echo 'export PATH="/root/bin:$PATH"' >> /root/.bashrc
|
||||
```
|
||||
After unlock, the `env/` directory contains plaintext `.env` files ready to copy back into the stack directories.
|
||||
|
||||
---
|
||||
|
||||
## Full Rebuild Runbook
|
||||
|
||||
On a fresh PD:
|
||||
On a fresh PD after reinstalling TrueNAS:
|
||||
|
||||
1. Install git-crypt (Option 1 or 2 above)
|
||||
2. Clone and unlock the secrets repo:
|
||||
1. **Install git-crypt** (Option 1 above — docker is available immediately after TrueNAS install)
|
||||
2. **Restore secrets repo:**
|
||||
```bash
|
||||
git clone https://gitea.paccoco.com/fizzlepoof/homelab-secrets /mnt/docker-ssd/docker/secrets
|
||||
git clone ssh://git@10.5.1.6:2222/fizzlepoof/homelab-secrets.git /mnt/docker-ssd/docker/secrets
|
||||
cd /mnt/docker-ssd/docker/secrets
|
||||
git-crypt unlock /path/to/.git-crypt-secrets.key
|
||||
```
|
||||
3. Copy `.env` files back into their stack directories from `secrets/env/`
|
||||
4. Redeploy stacks
|
||||
3. **Restore .env files** from `secrets/env/` back to their stack directories:
|
||||
```bash
|
||||
cp /mnt/docker-ssd/docker/secrets/env/ai.env /mnt/docker-ssd/docker/compose/ai/.env
|
||||
cp /mnt/docker-ssd/docker/secrets/env/databases.env /mnt/docker-ssd/docker/compose/databases/.env
|
||||
# ... etc for each stack
|
||||
```
|
||||
4. **Clone the main stacks repo:**
|
||||
```bash
|
||||
git clone git@gitea.paccoco.com:fizzlepoof/truenas-stacks.git /mnt/docker-ssd/docker/compose
|
||||
```
|
||||
5. **Redeploy stacks** per `DEPLOYMENT_CHECKLIST.md`
|
||||
|
||||
---
|
||||
|
||||
## Known Credentials Locations
|
||||
|
||||
| Service | Where stored |
|
||||
|---------|-------------|
|
||||
| Service | File |
|
||||
|---------|------|
|
||||
| shared-postgres | `databases/.env` |
|
||||
| shared-mariadb | `databases/.env` |
|
||||
| OpenWebUI DB | `ai/.env` — user: openwebui, db: openwebui |
|
||||
| Donetick | `home/.env` + `selfhosted.yaml` |
|
||||
| Meshmonitor | `meshtastic/.env` |
|
||||
| Gitea | `dev/.env` |
|
||||
| LiteLLM | `ai/.env` |
|
||||
| Paperless | `documents/.env` |
|
||||
|
||||
---
|
||||
|
||||
## Security Reminders
|
||||
|
||||
- **Key backup**: `C:\Users\Fizzlepoof\Downloads\.git-crypt-secrets.key` — also store in password manager
|
||||
- Regenerate Wings token on N.O.M.A.D. (was exposed in chat — TODO)
|
||||
- Regenerate N.O.M.A.D. Newt secret (was exposed in chat — TODO)
|
||||
- Disable signups in OpenWebUI after creating initial account
|
||||
- Back up `/root/.git-crypt-secrets.key` to password manager
|
||||
- The Gitea token in `setup-secrets-repo.sh` should be rotated after initial setup
|
||||
|
||||
Reference in New Issue
Block a user