truenas-stacks/docs/operations/SECRETS_MANAGEMENT.md

# Secrets Management

## Overview

Secrets are managed in two layers:

1. **Live secrets** — `.env` files on PD at `/mnt/docker-ssd/docker/compose/<stack>/.env`, gitignored and never committed to the main repo
2. **Encrypted backup** — all `.env` files are synced to a private git-crypt encrypted Gitea repo at `/mnt/docker-ssd/docker/secrets/`

The sync is a one-command operation and is safe to re-run at any time.

---

## Principles

- `.env` files are gitignored in the main repo — never committed there
- `.env.example` files with placeholder values are committed instead
- The secrets repo encrypts everything at rest using git-crypt (symmetric key)
- Secrets are generated with standard tools, never reused across services

---

## Generating Secrets

```bash
# Database passwords
openssl rand -hex 24

# JWT / session secrets
openssl rand -hex 32

# API keys (where format allows)
openssl rand -base64 32
```

---

## Live .env File Locations

All `.env` files live alongside their `docker-compose.yaml`:

```
/mnt/docker-ssd/docker/compose/
  ai/.env
  automation/.env
  databases/.env
  dev/.env
  documents/.env
  home/.env
  infrastructure/.env
  media/.env
  media/kima-hub/.env
  mesh-mqtt-observer/.env
  meshtastic/.env
  monitoring/.env
  photos/.env
```

---

## Secrets Repo (git-crypt)

| | |
|---|---|
| **Gitea repo** | `https://gitea.paccoco.com/fizzlepoof/homelab-secrets` (private) |
| **SSH remote** | `ssh://git@10.5.1.6:2222/fizzlepoof/homelab-secrets.git` |
| **Local path on PD** | `/mnt/docker-ssd/docker/secrets/` |
| **Symmetric key location** | `/mnt/docker-ssd/docker/secrets/.git-crypt-secrets.key` (if copied there) or retrieve from password manager |
| **Key backup** | Password manager + `C:\Users\Fizzlepoof\Downloads\.git-crypt-secrets.key` |

### Repo structure

| Directory | Contents |
|-----------|----------|
| `env/` | `.env` files for each stack, named `<stack>.env` |
| `keys/` | API keys and tokens |
| `certs/` | TLS certificates |
| `tokens/` | Service tokens (Paperless, LiteLLM, etc.) |
| `ssh/` | SSH private keys |

All files except `README.md` and `.gitattributes` are encrypted by git-crypt on commit.

---

## Syncing .env Files to the Secrets Repo

Use the sync script — no root required, safe to re-run:

```bash
bash /mnt/docker-ssd/docker/compose/scripts/sync-envs-to-secrets.sh
```

What it does:
- Scans all stack directories under `/mnt/docker-ssd/docker/compose/` for `.env` files
- Copies changed/new files to `/mnt/docker-ssd/docker/secrets/env/` as `<stack>.env`
- Skips unchanged files
- Commits and pushes to Gitea automatically

Run this any time you create or update a stack's `.env` file.

---

## git-crypt on TrueNAS

apt is blocked on TrueNAS Scale. git-crypt is installed via Docker and stored on the persistent dataset at `/mnt/docker-ssd/bin/git-crypt`.

### Current installation (already done on PD)

```
/mnt/docker-ssd/bin/git-crypt
```

PATH is set in `/home/truenas_admin/.bashrc` and `/root/.bashrc`:

```bash
export PATH="/mnt/docker-ssd/bin:$PATH"
```

### Re-installing after a full rebuild

**Option 1 — one-liner:**
```bash
sudo -i
docker run --rm -v /tmp:/out debian:bookworm-slim \
  bash -c "apt-get update -qq && apt-get install -y -qq git-crypt && cp /usr/bin/git-crypt /out/git-crypt"
mkdir -p /mnt/docker-ssd/bin
cp /tmp/git-crypt /mnt/docker-ssd/bin/git-crypt
chmod +x /mnt/docker-ssd/bin/git-crypt
echo 'export PATH="/mnt/docker-ssd/bin:$PATH"' >> /root/.bashrc
echo 'export PATH="/mnt/docker-ssd/bin:$PATH"' >> /home/truenas_admin/.bashrc
```

**Option 2 — via dev stack compose:**
```bash
cd /mnt/docker-ssd/docker/compose/dev
sudo docker compose --profile setup up git-crypt-init
# copies git-crypt to /root/bin — then move to persistent location:
sudo cp /root/bin/git-crypt /mnt/docker-ssd/bin/git-crypt
```

---

## Unlocking the Secrets Repo on a New Machine

```bash
# Install git-crypt first (see above), then:
git clone ssh://git@10.5.1.6:2222/fizzlepoof/homelab-secrets.git /mnt/docker-ssd/docker/secrets
cd /mnt/docker-ssd/docker/secrets
git-crypt unlock /path/to/.git-crypt-secrets.key
```

After unlock, the `env/` directory contains plaintext `.env` files ready to copy back into the stack directories.

---

## Full Rebuild Runbook

On a fresh PD after reinstalling TrueNAS:

1. **Install git-crypt** (Option 1 above — docker is available immediately after TrueNAS install)
2. **Restore secrets repo:**
   ```bash
   git clone ssh://git@10.5.1.6:2222/fizzlepoof/homelab-secrets.git /mnt/docker-ssd/docker/secrets
   cd /mnt/docker-ssd/docker/secrets
   git-crypt unlock /path/to/.git-crypt-secrets.key
   ```
3. **Restore .env files** from `secrets/env/` back to their stack directories:
   ```bash
   cp /mnt/docker-ssd/docker/secrets/env/ai.env /mnt/docker-ssd/docker/compose/ai/.env
   cp /mnt/docker-ssd/docker/secrets/env/databases.env /mnt/docker-ssd/docker/compose/databases/.env
   # ... etc for each stack
   ```
4. **Clone the main stacks repo:**
   ```bash
   git clone git@gitea.paccoco.com:fizzlepoof/truenas-stacks.git /mnt/docker-ssd/docker/compose
   ```
5. **Redeploy stacks** per `DEPLOYMENT_CHECKLIST.md`

---

## Known Credentials Locations

| Service | File |
|---------|------|
| shared-postgres | `databases/.env` |
| shared-mariadb | `databases/.env` |
| OpenWebUI DB | `ai/.env` — user: openwebui, db: openwebui |
| Donetick | `home/.env` + `selfhosted.yaml` |
| Meshmonitor | `meshtastic/.env` |
| Gitea | `dev/.env` |
| LiteLLM | `ai/.env` |
| Paperless | `documents/.env` |

---

## Security Reminders

- **Key backup**: `C:\Users\Fizzlepoof\Downloads\.git-crypt-secrets.key` — also store in password manager
- Regenerate Wings token on N.O.M.A.D. (was exposed in chat — TODO)
- Regenerate N.O.M.A.D. Newt secret (was exposed in chat — TODO)
- Disable signups in OpenWebUI after creating initial account
- The Gitea token in `setup-secrets-repo.sh` should be rotated after initial setup