truenas-stacks/docs/operations/SECRETS_MANAGEMENT.md

# Secrets Management

## Overview

Secrets are managed in two layers:

1. **Live secrets** — `.env` files on PD at `/mnt/docker-ssd/docker/compose/<stack>/.env`, gitignored and never committed to the main repo
2. **Encrypted backup** — all `.env` files are synced to a private git-crypt encrypted Gitea repo at `/mnt/docker-ssd/docker/secrets/`

The sync is a one-command operation and is safe to re-run at any time.

---

## Principles

- `.env` files are gitignored in the main repo — never committed there
- `.env.example` files with placeholder values are committed instead
- The secrets repo encrypts everything at rest using git-crypt (symmetric key)
- Secrets are generated with standard tools, never reused across services

---

## Generating Secrets

```bash
# Database passwords
openssl rand -hex 24

# JWT / session secrets
openssl rand -hex 32

# API keys (where format allows)
openssl rand -base64 32
```

---

## Live .env File Locations

All `.env` files live alongside their `docker-compose.yaml`:

```
/mnt/docker-ssd/docker/compose/
  ai/.env
  automation/.env
  databases/.env
  dev/.env
  documents/.env
  home/.env
  infrastructure/.env
  media/.env
  media/kima-hub/.env
  mesh-mqtt-observer/.env
  meshtastic/.env
  monitoring/.env
  photos/.env
```

---

## Secrets Repo (git-crypt)

| | |
|---|---|
| **Gitea repo** | `https://gitea.paccoco.com/fizzlepoof/homelab-secrets` (private) |
| **SSH remote** | `ssh://git@10.5.30.6:2222/fizzlepoof/homelab-secrets.git` |
| **Local path on PD** | `/mnt/docker-ssd/docker/secrets/` |
| **Symmetric key location** | `/mnt/docker-ssd/docker/secrets/.git-crypt-secrets.key` (if copied there) or retrieve from password manager |
| **Key backup** | Password manager + `C:\Users\Fizzlepoof\Downloads\.git-crypt-secrets.key` |

### Repo structure

| Directory | Contents |
|-----------|----------|
| `env/` | `.env` files for each stack, named `<stack>.env` |
| `keys/` | API keys and tokens |
| `certs/` | TLS certificates |
| `tokens/` | Service tokens (Paperless, LiteLLM, etc.) |
| `ssh/` | SSH private keys |

All files except `README.md` and `.gitattributes` are encrypted by git-crypt on commit.

---

## Syncing .env Files to the Secrets Repo

Use the sync script — no root required, safe to re-run:

```bash
bash /mnt/docker-ssd/docker/compose/scripts/sync-envs-to-secrets.sh
```

What it does:
- Scans all stack directories under `/mnt/docker-ssd/docker/compose/` for `.env` files
- Copies changed/new files to `/mnt/docker-ssd/docker/secrets/env/` as `<stack>.env`
- Skips unchanged files
- Commits and pushes to Gitea automatically

Run this any time you create or update a stack's `.env` file.

**Operational rule:** after any live `.env` change on PD, Doris or the operator must run the sync script and confirm whether it committed/pushed changes or reported the secrets repo already up to date. Do not treat live `.env` edits as complete until this sync step is checked.

---

## git-crypt on TrueNAS

apt is blocked on TrueNAS Scale. git-crypt is installed via Docker.

### Preferred installation location

```
/mnt/docker-ssd/bin/git-crypt
```

### Compatibility note

The original bootstrap script installs to `/root/bin/git-crypt`, while the newer runbook expects `/mnt/docker-ssd/bin/git-crypt`. Standardize on `/mnt/docker-ssd/bin/git-crypt` and make sure PATH includes that directory for both `root` and `truenas_admin`.

Recommended PATH entry:

```bash
export PATH="/mnt/docker-ssd/bin:$PATH"
```

### Re-installing after a full rebuild

**Option 1 — one-liner:**
```bash
sudo -i
docker run --rm -v /tmp:/out debian:bookworm-slim \
  bash -c "apt-get update -qq && apt-get install -y -qq git-crypt && cp /usr/bin/git-crypt /out/git-crypt"
mkdir -p /mnt/docker-ssd/bin
cp /tmp/git-crypt /mnt/docker-ssd/bin/git-crypt
chmod +x /mnt/docker-ssd/bin/git-crypt
echo 'export PATH="/mnt/docker-ssd/bin:$PATH"' >> /root/.bashrc
echo 'export PATH="/mnt/docker-ssd/bin:$PATH"' >> /home/truenas_admin/.bashrc
```

**Option 2 — via dev stack compose:**
```bash
cd /mnt/docker-ssd/docker/compose/dev
sudo docker compose --profile setup up git-crypt-init
# copies git-crypt to /root/bin — then move to persistent location:
sudo cp /root/bin/git-crypt /mnt/docker-ssd/bin/git-crypt
```

---

## Unlocking the Secrets Repo on a New Machine

```bash
# Install git-crypt first (see above), then:
git clone ssh://git@10.5.30.6:2222/fizzlepoof/homelab-secrets.git /mnt/docker-ssd/docker/secrets
cd /mnt/docker-ssd/docker/secrets
git-crypt unlock /path/to/.git-crypt-secrets.key
```

After unlock, the `env/` directory contains plaintext `.env` files ready to copy back into the stack directories.

---

## Full Rebuild Runbook

On a fresh PD after reinstalling TrueNAS:

1. **Install git-crypt** (Option 1 above — docker is available immediately after TrueNAS install)
2. **Restore secrets repo:**
   ```bash
   git clone ssh://git@10.5.30.6:2222/fizzlepoof/homelab-secrets.git /mnt/docker-ssd/docker/secrets
   cd /mnt/docker-ssd/docker/secrets
   git-crypt unlock /path/to/.git-crypt-secrets.key
   ```
3. **Restore .env files** from `secrets/env/` back to their stack directories:
   ```bash
   cp /mnt/docker-ssd/docker/secrets/env/ai.env /mnt/docker-ssd/docker/compose/ai/.env
   cp /mnt/docker-ssd/docker/secrets/env/databases.env /mnt/docker-ssd/docker/compose/databases/.env
   # ... etc for each stack
   ```
4. **Clone the main stacks repo:**
   ```bash
   git clone ssh://git@10.5.30.6:2222/fizzlepoof/truenas-stacks.git /mnt/docker-ssd/docker/compose
   ```
5. **Redeploy stacks** per `DEPLOYMENT_CHECKLIST.md`
6. **After any future live `.env` edit**, immediately run:
   ```bash
   export PATH="/mnt/docker-ssd/bin:$PATH"
   bash /mnt/docker-ssd/docker/compose/scripts/sync-envs-to-secrets.sh
   ```
   Then verify the secrets repo is either updated and pushed or already unchanged.

---

## Known Credentials Locations

| Service | File |
|---------|------|
| shared-postgres | `databases/.env` |
| shared-mariadb | `databases/.env` |
| OpenWebUI DB | `ai/.env` — user: openwebui, db: openwebui |
| Donetick | `home/.env` + `selfhosted.yaml` |
| Meshmonitor | `meshtastic/.env` |
| Gitea | `dev/.env` |
| LiteLLM | `ai/.env` |
| Paperless | `documents/.env` |
| UniFi Doris operator | `automation/.env` |

---

## Controller Export / Baseline Artifact Rules

Raw infrastructure exports must be treated as potentially secret-bearing even when they are not `.env` files.

Examples:
- UniFi `networkconf` / `portconf` dumps
- firewall/router/controller JSON exports
- VPN client or server config exports
- diagnostic snapshots taken from appliances or operator APIs

### Permanent rules

- Do **not** commit raw controller or appliance exports to the main repo.
- Commit only sanitized markdown summaries or explicitly redacted artifacts.
- If raw retention is needed, store the export in the encrypted secrets repo or another operator-only location outside the main repo.
- Before committing infra artifacts, scan for obvious secret-bearing keys such as:
  - `private_key`
  - `wireguard`
  - `token`
  - `secret`
  - `password`
  - `Authorization`
  - `cookie`
  - `client_secret`

### Incident reference

A real leak occurred on 2026-05-22 when a raw UniFi baseline export under `home/doris-dashboard/docs/baselines/` was committed with embedded WireGuard secrets.

See:
- `docs/operations/INCIDENT_2026-05-22_UNIFI_BASELINE_SECRET_LEAK.md`
- `home/doris-dashboard/docs/baselines/README.md`

### Git hook + CI guardrails for infra artifacts

The repo now includes layered guardrails aimed specifically at catching secret-bearing controller/export artifacts before they leave a workstation.

Files:
- `scripts/scan-secret-bearing-artifacts.sh`
- `.githooks/pre-commit`
- `.githooks/pre-push`
- `.github/workflows/secret-guardrails.yml`

Enable the hooks once per clone:

```bash
git config core.hooksPath .githooks
chmod +x .githooks/pre-commit .githooks/pre-push scripts/scan-secret-bearing-artifacts.sh
```

Manual runs:

```bash
scripts/scan-secret-bearing-artifacts.sh --staged
scripts/scan-secret-bearing-artifacts.sh --tracked
scripts/scan-secret-bearing-artifacts.sh --git-range origin/main..HEAD
```

What each layer does:
- `pre-commit`: scans staged files before a local commit is created
- `pre-push`: scans the commits about to be pushed, including unpublished branch history
- GitHub Actions: runs the narrow artifact scan across tracked files and also runs Gitleaks on pushes and pull requests

What it is for:
- raw baselines
- exports
- snapshots
- dumps
- similar machine-generated infra artifacts

What it is not:
- a full replacement for operator judgment about what belongs in the repo
- proof that a raw export is safe just because the scanner stayed quiet
- a reason to skip rotating live secrets if one ever does leak

## Security Reminders

- **Key backup**: `C:\Users\Fizzlepoof\Downloads\.git-crypt-secrets.key` — also store in password manager
- Regenerate Wings token on N.O.M.A.D. (was exposed in chat — TODO)
- Regenerate N.O.M.A.D. Newt secret (was exposed in chat — TODO)
- Disable signups in OpenWebUI after creating initial account
- The Gitea token in `setup-secrets-repo.sh` should be rotated after initial setup