Update Schoolhouse and backup docs to live state
This commit is contained in:
Fizzlepoof
@@ -3,7 +3,7 @@
|
||||
## Current State (Known Gaps)
|
||||
- **No off-site backup** — this is a known gap, improvement pending
|
||||
- Serenity has local ZFS snapshots and replication
|
||||
- PlausibleDeniability has no automated backup of appdata or databases
|
||||
- PlausibleDeniability now has repo-documented automated backup scaffolding deployed to PD for Postgres dumps plus rsync replication to Serenity
|
||||
|
||||
## Serenity Backup Strategy
|
||||
- Sanoid snapshots cache/appdata: 7 daily, 4 weekly, 6 monthly
|
||||
@@ -32,7 +32,7 @@ Repo-side scaffolding now lives under `automation/`:
|
||||
- `automation/bin/pd_backup_sync_to_serenity.sh`
|
||||
- `automation/bin/run_pd_backups.sh`
|
||||
|
||||
These are staged helper scripts only until they are deployed with a real `.env`, SSH trust to Serenity, and a live scheduler on PD.
|
||||
These helper scripts are now the live PD backup runner model, with deployment details tracked in `docs/operations/PD_BACKUP_DEPLOYMENT.md`.
|
||||
|
||||
Recommended live deployment model on PD: a plain **root cron** job invoking `/usr/bin/bash bin/run_pd_backups.sh` from `/mnt/docker-ssd/docker/compose/automation`.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ Current workflow versions:
|
||||
- `n8n-workflows/18-school-paperless-intake-v1.2.json`
|
||||
- `n8n-workflows/19-paperless-school-metadata-enrichment-v1.1.json`
|
||||
|
||||
This workflow set is meant for a Telegram chat-driven upload pipeline that sends schoolwork into Paperless-NGX while recording intake metadata in shared Postgres.
|
||||
This workflow set is the Telegram chat-driven upload pipeline that sends schoolwork into Paperless-NGX with deterministic intake IDs and follow-up metadata enrichment.
|
||||
|
||||
## Versioning rule
|
||||
|
||||
@@ -31,16 +31,16 @@ When these workflow artifacts change, the version must change too:
|
||||
- `telegram_chat_id`
|
||||
- `telegram_message_id`
|
||||
4. Generates a deterministic `intake_id` from date + class + assignment + file checksum
|
||||
5. Stores/updates the intake row in Postgres first
|
||||
5. Builds a deterministic intake envelope from the request metadata and upload details
|
||||
6. Uploads the file to Paperless using the deterministic intake ID in the filename
|
||||
7. Marks the row as `uploaded` after Paperless accepts the upload
|
||||
|
||||
## Shared Postgres expectations
|
||||
## Current correlation model
|
||||
|
||||
- Use the shared Postgres network/credentials pattern already used by the automation stack.
|
||||
- Earlier drafts used shared Postgres tracking, but the current live-tested v1.2/v1.1 path works without Postgres dependency in the webhook chain.
|
||||
- Deterministic filename/title are now the source of correlation between upload and enrichment.
|
||||
- Tracked schema: `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql` (kept for future DB-backed intake tracking if reinstated)
|
||||
- Earlier drafts stored intake rows in shared Postgres.
|
||||
- The current live-tested `v1.2` + `v1.1` chain does **not** require Postgres in the webhook path.
|
||||
- Deterministic filename/title are the current source of correlation between upload and enrichment.
|
||||
- `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql` is retained for a future DB-backed intake ledger if that gets reinstated.
|
||||
|
||||
## Required environment variables
|
||||
|
||||
@@ -104,7 +104,7 @@ The deterministic filename/title preserve the `intake_id` for later correlation.
|
||||
Workflow 19 handles the follow-up metadata pass after Paperless finishes processing:
|
||||
- fetch Paperless document by webhook document id
|
||||
- resolve `intake_id` from the stored filename
|
||||
- look up the intake record in shared Postgres
|
||||
- recover class/assignment/submission metadata from the deterministic filename and title
|
||||
- re-apply deterministic title
|
||||
- merge generic school tags plus optional class/submission-kind mappings
|
||||
- optionally set document type / correspondent from JSON env mappings
|
||||
@@ -116,7 +116,7 @@ Use the example curl request in:
|
||||
|
||||
Minimum test checklist:
|
||||
1. Import the workflow into n8n.
|
||||
2. Attach the `Shared Postgres` credential.
|
||||
2. Import workflow 19 alongside workflow 18.
|
||||
3. Send the sample multipart request.
|
||||
4. Verify the uploaded Paperless document title and stored filename both include the deterministic intake ID.
|
||||
5. Trigger workflow 19 with the Paperless document id and verify title/tags enrichment succeeds.
|
||||
@@ -124,7 +124,7 @@ Minimum test checklist:
|
||||
|
||||
## Known assumptions / caveats
|
||||
|
||||
- The workflow is repo-only and was not live-tested against the running n8n/Paperless stack.
|
||||
- The Postgres node uses SQL expressions inline rather than parameter binding because exported n8n node JSON can differ by version.
|
||||
- Workflow 18 v1.0 needed `require('crypto')`; workflow 18 v1.1 removed that dependency for easier live rollout.
|
||||
- Workflow 18 `v1.2` assignment intake was live smoke-tested successfully on 2026-05-15.
|
||||
- Workflow 19 `v1.1` is designed around the deterministic filename/title model rather than a Postgres lookup.
|
||||
- Workflow 18 v1.0 needed `require('crypto')`; workflow 18 v1.1 removed that dependency for easier live rollout; workflow 18 v1.2 replaced it with an internal stable hash.
|
||||
- If Paperless needs tags as repeated `tags[]` fields instead of a comma-separated `tags` field, adjust the HTTP Request node after import.
|
||||
|
||||
@@ -31,11 +31,9 @@ Do not run production from an agent workspace.
|
||||
|
||||
## Current state
|
||||
|
||||
This is an implementation scaffold with:
|
||||
The app is live on NOMAD at `/opt/doris-schoolhouse` and uses shared Postgres in production.
|
||||
|
||||
## Current local persistence
|
||||
|
||||
Until the full shared-Postgres integration is wired, the scaffold uses JSON state files for fast local iteration:
|
||||
For local iteration, the source tree still supports a JSON-backed mode:
|
||||
|
||||
- default state dir: `/data/state`
|
||||
- default upload temp dir: `/data/uploads`
|
||||
@@ -47,18 +45,15 @@ export SCHOOLHOUSE_STATE_DIR=/tmp/doris-schoolhouse-state
|
||||
export UPLOAD_TMP_DIR=/tmp/doris-schoolhouse-uploads
|
||||
```
|
||||
|
||||
Production target remains shared Postgres plus Paperless/n8n integration.
|
||||
Production runtime uses shared Postgres plus Paperless/n8n integration.
|
||||
|
||||
`SCHOOLHOUSE_STORAGE_BACKEND=auto` prefers Postgres when psycopg and a PostgreSQL DSN are available, and falls back to JSON state for local iteration.
|
||||
|
||||
|
||||
- route stubs
|
||||
- templates
|
||||
- schema draft
|
||||
Repo contents still include:
|
||||
- route handlers and templates
|
||||
- schema/bootstrap helpers
|
||||
- compose/env examples
|
||||
- D2L / Paperless integration placeholders
|
||||
|
||||
It is not deployed yet.
|
||||
- D2L / Paperless / n8n integration code
|
||||
|
||||
## Layout
|
||||
|
||||
@@ -159,7 +154,7 @@ That image includes:
|
||||
|
||||
So even if the host is missing `ffmpeg`, the containerized runtime can still do WAV → MP3 conversion.
|
||||
|
||||
## Future deploy shape
|
||||
## Deploy shape
|
||||
|
||||
Expected live deploy flow on NOMAD:
|
||||
|
||||
@@ -172,7 +167,7 @@ docker compose --env-file .env config
|
||||
docker compose --env-file .env up -d --build
|
||||
```
|
||||
|
||||
Not doing that yet. No live deployment should happen without explicit rollout work.
|
||||
That flow is the reference for refreshing the live runtime from repo source.
|
||||
|
||||
|
||||
## Live integration notes
|
||||
|
||||
Reference in New Issue
Block a user