Update Schoolhouse and backup docs to live state

This commit is contained in:
Fizzlepoof
2026-05-16 15:54:20 +00:00
parent 383cd41ae7
commit ecf44fe65f
3 changed files with 23 additions and 28 deletions

View File

@@ -3,7 +3,7 @@
## Current State (Known Gaps) ## Current State (Known Gaps)
- **No off-site backup** — this is a known gap, improvement pending - **No off-site backup** — this is a known gap, improvement pending
- Serenity has local ZFS snapshots and replication - Serenity has local ZFS snapshots and replication
- PlausibleDeniability has no automated backup of appdata or databases - PlausibleDeniability now has repo-documented automated backup scaffolding deployed to PD for Postgres dumps plus rsync replication to Serenity
## Serenity Backup Strategy ## Serenity Backup Strategy
- Sanoid snapshots cache/appdata: 7 daily, 4 weekly, 6 monthly - Sanoid snapshots cache/appdata: 7 daily, 4 weekly, 6 monthly
@@ -32,7 +32,7 @@ Repo-side scaffolding now lives under `automation/`:
- `automation/bin/pd_backup_sync_to_serenity.sh` - `automation/bin/pd_backup_sync_to_serenity.sh`
- `automation/bin/run_pd_backups.sh` - `automation/bin/run_pd_backups.sh`
These are staged helper scripts only until they are deployed with a real `.env`, SSH trust to Serenity, and a live scheduler on PD. These helper scripts are now the live PD backup runner model, with deployment details tracked in `docs/operations/PD_BACKUP_DEPLOYMENT.md`.
Recommended live deployment model on PD: a plain **root cron** job invoking `/usr/bin/bash bin/run_pd_backups.sh` from `/mnt/docker-ssd/docker/compose/automation`. Recommended live deployment model on PD: a plain **root cron** job invoking `/usr/bin/bash bin/run_pd_backups.sh` from `/mnt/docker-ssd/docker/compose/automation`.

View File

@@ -4,7 +4,7 @@ Current workflow versions:
- `n8n-workflows/18-school-paperless-intake-v1.2.json` - `n8n-workflows/18-school-paperless-intake-v1.2.json`
- `n8n-workflows/19-paperless-school-metadata-enrichment-v1.1.json` - `n8n-workflows/19-paperless-school-metadata-enrichment-v1.1.json`
This workflow set is meant for a Telegram chat-driven upload pipeline that sends schoolwork into Paperless-NGX while recording intake metadata in shared Postgres. This workflow set is the Telegram chat-driven upload pipeline that sends schoolwork into Paperless-NGX with deterministic intake IDs and follow-up metadata enrichment.
## Versioning rule ## Versioning rule
@@ -31,16 +31,16 @@ When these workflow artifacts change, the version must change too:
- `telegram_chat_id` - `telegram_chat_id`
- `telegram_message_id` - `telegram_message_id`
4. Generates a deterministic `intake_id` from date + class + assignment + file checksum 4. Generates a deterministic `intake_id` from date + class + assignment + file checksum
5. Stores/updates the intake row in Postgres first 5. Builds a deterministic intake envelope from the request metadata and upload details
6. Uploads the file to Paperless using the deterministic intake ID in the filename 6. Uploads the file to Paperless using the deterministic intake ID in the filename
7. Marks the row as `uploaded` after Paperless accepts the upload 7. Marks the row as `uploaded` after Paperless accepts the upload
## Shared Postgres expectations ## Current correlation model
- Use the shared Postgres network/credentials pattern already used by the automation stack. - Earlier drafts stored intake rows in shared Postgres.
- Earlier drafts used shared Postgres tracking, but the current live-tested v1.2/v1.1 path works without Postgres dependency in the webhook chain. - The current live-tested `v1.2` + `v1.1` chain does **not** require Postgres in the webhook path.
- Deterministic filename/title are now the source of correlation between upload and enrichment. - Deterministic filename/title are the current source of correlation between upload and enrichment.
- Tracked schema: `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql` (kept for future DB-backed intake tracking if reinstated) - `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql` is retained for a future DB-backed intake ledger if that gets reinstated.
## Required environment variables ## Required environment variables
@@ -104,7 +104,7 @@ The deterministic filename/title preserve the `intake_id` for later correlation.
Workflow 19 handles the follow-up metadata pass after Paperless finishes processing: Workflow 19 handles the follow-up metadata pass after Paperless finishes processing:
- fetch Paperless document by webhook document id - fetch Paperless document by webhook document id
- resolve `intake_id` from the stored filename - resolve `intake_id` from the stored filename
- look up the intake record in shared Postgres - recover class/assignment/submission metadata from the deterministic filename and title
- re-apply deterministic title - re-apply deterministic title
- merge generic school tags plus optional class/submission-kind mappings - merge generic school tags plus optional class/submission-kind mappings
- optionally set document type / correspondent from JSON env mappings - optionally set document type / correspondent from JSON env mappings
@@ -116,7 +116,7 @@ Use the example curl request in:
Minimum test checklist: Minimum test checklist:
1. Import the workflow into n8n. 1. Import the workflow into n8n.
2. Attach the `Shared Postgres` credential. 2. Import workflow 19 alongside workflow 18.
3. Send the sample multipart request. 3. Send the sample multipart request.
4. Verify the uploaded Paperless document title and stored filename both include the deterministic intake ID. 4. Verify the uploaded Paperless document title and stored filename both include the deterministic intake ID.
5. Trigger workflow 19 with the Paperless document id and verify title/tags enrichment succeeds. 5. Trigger workflow 19 with the Paperless document id and verify title/tags enrichment succeeds.
@@ -124,7 +124,7 @@ Minimum test checklist:
## Known assumptions / caveats ## Known assumptions / caveats
- The workflow is repo-only and was not live-tested against the running n8n/Paperless stack. - Workflow 18 `v1.2` assignment intake was live smoke-tested successfully on 2026-05-15.
- The Postgres node uses SQL expressions inline rather than parameter binding because exported n8n node JSON can differ by version. - Workflow 19 `v1.1` is designed around the deterministic filename/title model rather than a Postgres lookup.
- Workflow 18 v1.0 needed `require('crypto')`; workflow 18 v1.1 removed that dependency for easier live rollout. - Workflow 18 v1.0 needed `require('crypto')`; workflow 18 v1.1 removed that dependency for easier live rollout; workflow 18 v1.2 replaced it with an internal stable hash.
- If Paperless needs tags as repeated `tags[]` fields instead of a comma-separated `tags` field, adjust the HTTP Request node after import. - If Paperless needs tags as repeated `tags[]` fields instead of a comma-separated `tags` field, adjust the HTTP Request node after import.

View File

@@ -31,11 +31,9 @@ Do not run production from an agent workspace.
## Current state ## Current state
This is an implementation scaffold with: The app is live on NOMAD at `/opt/doris-schoolhouse` and uses shared Postgres in production.
## Current local persistence For local iteration, the source tree still supports a JSON-backed mode:
Until the full shared-Postgres integration is wired, the scaffold uses JSON state files for fast local iteration:
- default state dir: `/data/state` - default state dir: `/data/state`
- default upload temp dir: `/data/uploads` - default upload temp dir: `/data/uploads`
@@ -47,18 +45,15 @@ export SCHOOLHOUSE_STATE_DIR=/tmp/doris-schoolhouse-state
export UPLOAD_TMP_DIR=/tmp/doris-schoolhouse-uploads export UPLOAD_TMP_DIR=/tmp/doris-schoolhouse-uploads
``` ```
Production target remains shared Postgres plus Paperless/n8n integration. Production runtime uses shared Postgres plus Paperless/n8n integration.
`SCHOOLHOUSE_STORAGE_BACKEND=auto` prefers Postgres when psycopg and a PostgreSQL DSN are available, and falls back to JSON state for local iteration. `SCHOOLHOUSE_STORAGE_BACKEND=auto` prefers Postgres when psycopg and a PostgreSQL DSN are available, and falls back to JSON state for local iteration.
Repo contents still include:
- route stubs - route handlers and templates
- templates - schema/bootstrap helpers
- schema draft
- compose/env examples - compose/env examples
- D2L / Paperless integration placeholders - D2L / Paperless / n8n integration code
It is not deployed yet.
## Layout ## Layout
@@ -159,7 +154,7 @@ That image includes:
So even if the host is missing `ffmpeg`, the containerized runtime can still do WAV → MP3 conversion. So even if the host is missing `ffmpeg`, the containerized runtime can still do WAV → MP3 conversion.
## Future deploy shape ## Deploy shape
Expected live deploy flow on NOMAD: Expected live deploy flow on NOMAD:
@@ -172,7 +167,7 @@ docker compose --env-file .env config
docker compose --env-file .env up -d --build docker compose --env-file .env up -d --build
``` ```
Not doing that yet. No live deployment should happen without explicit rollout work. That flow is the reference for refreshing the live runtime from repo source.
## Live integration notes ## Live integration notes