Publish sanitized current homelab architecture

docs/operations.md

@@ -0,0 +1,78 @@
+# Operations
+
+## Deployment order
+
+A predictable dependency order reduces partial deployments:
+
+1. storage mounts and host prerequisites
+2. shared databases and caches
+3. ingress, identity, DNS, and management services
+4. observability and notifications
+5. application stacks
+6. optional AI, media, and edge integrations
+
+## Change workflow
+
+1. **Read first.** Capture current host, service, network, and storage state.
+2. **Define scope.** Name the exact stack, files, and expected result.
+3. **Create rollback.** Back up configuration and state before mutation.
+4. **Validate configuration.** Render Compose, parse structured files, and run syntax checks.
+5. **Apply narrowly.** Restart or recreate only the service that consumes the change.
+6. **Verify from the consumer side.** A process being `running` is not proof that the workflow works.
+7. **Read back external state.** Confirm the controller, API, repository, or database actually retained the change.
+8. **Document the result.** Record current truth, rollback location, and deferred work.
+
+## Compose validation
+
+A sanitized example stack is provided under `examples/compose/`.
+
+```bash
+cp examples/compose/.env.example examples/compose/.env
+# Replace placeholders only in the untracked .env file.
+docker compose --env-file examples/compose/.env \
+  -f examples/compose/docker-compose.yml config
+```
+
+Never commit the generated `.env` file.
+
+## Service verification
+
+Use multiple layers of evidence:
+
+- container or service state
+- health endpoint
+- recent logs
+- dependency connectivity
+- application-level request
+- persistence after restart
+- monitoring visibility
+
+For stateful applications, also check a representative database object or file after restart.
+
+## Backup model
+
+- filesystem snapshots protect bulk datasets
+- application-aware exports protect databases and stateful services
+- encrypted Git stores recoverable configuration secrets
+- normal Git stores source, sanitized templates, and operator documentation
+- synchronized user files use versioning in addition to storage snapshots
+- restore tests are scheduled; backup creation alone is not treated as recovery proof
+
+## Storage and data locality
+
+Large data stays on the storage host. Applications consume it over authenticated mounts where practical. Ingestion workloads remain close to the data when moving them would create avoidable network traffic or fragile cross-host dependencies.
+
+## Monitoring and notifications
+
+Metrics, logs, uptime checks, and push notifications are separate components so one dashboard failure does not erase the underlying evidence. Alerts should be low-volume, actionable, and routed to an operator channel with enough context to diagnose.
+
+## Updating documentation
+
+Documentation is divided into:
+
+- **current state:** what is running and where
+- **operating procedure:** how to change and verify it
+- **historical evidence:** dated incident or migration records
+- **future plan:** proposed work that is not yet live
+
+Dated historical notes should never read like the active backlog. Public documentation describes roles and patterns; exact operational coordinates remain private.