feat(n8n): add school paperless intake pipeline

2026-05-13 20:53:00 +00:00
parent a04b2eecd7
commit 3269cc1fc5
11 changed files with 895 additions and 3 deletions
--- a/docs/operations/SCHOOL_PAPERLESS_INTAKE.md
+++ b/docs/operations/SCHOOL_PAPERLESS_INTAKE.md
@@ -0,0 +1,111 @@
+# School Paperless Intake Workflow
+
+Workflows:
+- `n8n-workflows/18-school-paperless-intake.json`
+- `n8n-workflows/19-paperless-school-metadata-enrichment.json`
+
+This workflow is meant for a Telegram chat-driven upload pipeline that sends schoolwork into Paperless-NGX while recording intake metadata in shared Postgres.
+
+## What it does
+
+1. Accepts a multipart upload at `POST /webhook/school/intake/upload`
+2. Requires form fields:
+   - `class_name`
+   - `assignment_name`
+   - `submission_kind`
+3. Accepts optional fields:
+   - `semester`
+   - `course_code`
+   - `paper_kind`
+   - `notes`
+   - `source` (defaults to `telegram`)
+   - `telegram_chat_id`
+   - `telegram_message_id`
+4. Generates a deterministic `intake_id` from date + class + assignment + file checksum
+5. Stores/updates the intake row in Postgres first
+6. Uploads the file to Paperless using the deterministic intake ID in the filename
+7. Marks the row as `uploaded` after Paperless accepts the upload
+
+## Shared Postgres expectations
+
+- Use the shared Postgres network/credentials pattern already used by the automation stack.
+- The workflow expects an n8n Postgres credential named `Shared Postgres`.
+- Recommended target database: `school_intake`
+- Tracked schema: `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql`
+
+## Required environment variables
+
+Add to `automation/.env`:
+
+```env
+PAPERLESS_BASE_URL=https://paperless.paccoco.com
+PAPERLESS_API_TOKEN=CHANGE_ME
+LITELLM_API_KEY=CHANGE_ME
+PAPERLESS_CORRESPONDENT_SCHOOL=
+PAPERLESS_DOCUMENT_TYPE_SCHOOL=
+PAPERLESS_TAG_SCHOOL=
+PAPERLESS_TAG_ASSIGNMENTS=
+PAPERLESS_TAG_TELEGRAM=
+NODE_FUNCTION_ALLOW_BUILTIN=crypto
+```
+
+Add to the `n8n` service `environment:` block in `automation/docker-compose.yaml`:
+
+```yaml
+      PAPERLESS_BASE_URL: ${PAPERLESS_BASE_URL}
+      PAPERLESS_API_TOKEN: ${PAPERLESS_API_TOKEN}
+      PAPERLESS_CORRESPONDENT_SCHOOL: ${PAPERLESS_CORRESPONDENT_SCHOOL}
+      PAPERLESS_DOCUMENT_TYPE_SCHOOL: ${PAPERLESS_DOCUMENT_TYPE_SCHOOL}
+      PAPERLESS_TAG_SCHOOL: ${PAPERLESS_TAG_SCHOOL}
+      PAPERLESS_TAG_ASSIGNMENTS: ${PAPERLESS_TAG_ASSIGNMENTS}
+      PAPERLESS_TAG_TELEGRAM: ${PAPERLESS_TAG_TELEGRAM}
+      NODE_FUNCTION_ALLOW_BUILTIN: crypto
+```
+
+Notes:
+- `PAPERLESS_CORRESPONDENT_SCHOOL` and `PAPERLESS_DOCUMENT_TYPE_SCHOOL` should be numeric IDs if you want those fields auto-assigned.
+- Leave optional Paperless IDs blank if you prefer Paperless rules to classify later.
+- `NODE_FUNCTION_ALLOW_BUILTIN=crypto` is required because the Code node hashes the uploaded file.
+- `N8N_BLOCK_ENV_ACCESS_IN_NODE` must remain `false`.
+
+## Paperless metadata behavior
+
+The upload uses:
+- filename: `<intake_id>.<ext>`
+- title: `<class_name> — <assignment_name> — <submission_kind>`
+
+Workflow 18 deliberately keeps the Paperless upload minimal and safe:
+- multipart `document`
+- deterministic title
+
+The deterministic filename/title preserve the `intake_id` for later correlation.
+
+Workflow 19 handles the follow-up metadata pass after Paperless finishes processing:
+- fetch Paperless document by webhook document id
+- resolve `intake_id` from the stored filename
+- look up the intake record in shared Postgres
+- re-apply deterministic title
+- merge generic school tags plus optional class/submission-kind mappings
+- optionally set document type / correspondent from JSON env mappings
+
+## Testing
+
+Use the example curl request in:
+- `n8n-workflows/fixtures/school-paperless-intake-webhook.curl.example.txt`
+
+Minimum test checklist:
+1. Import the workflow into n8n.
+2. Attach the `Shared Postgres` credential.
+3. Confirm the target DB/table exists.
+4. Send the sample multipart request.
+5. Verify a row appears in `school_paperless_intake` with `status=uploaded`.
+6. Verify the uploaded Paperless document title and stored filename both include the deterministic intake ID.
+7. Trigger workflow 19 with the Paperless document id and verify the row moves to `status=enriched`.
+8. Re-submit the same file and metadata to confirm the same `intake_id` is reused and the Postgres row is upserted.
+
+## Known assumptions / caveats
+
+- The workflow is repo-only and was not live-tested against the running n8n/Paperless stack.
+- The Postgres node uses SQL expressions inline rather than parameter binding because exported n8n node JSON can differ by version.
+- If your n8n build blocks `require('crypto')`, add `NODE_FUNCTION_ALLOW_BUILTIN=crypto` to the container env.
+- If Paperless needs tags as repeated `tags[]` fields instead of a comma-separated `tags` field, adjust the HTTP Request node after import.
--- a/docs/planning/SCHOOL_INTAKE_PIPELINE.md
+++ b/docs/planning/SCHOOL_INTAKE_PIPELINE.md
@@ -0,0 +1,85 @@
+# School Work Intake Pipeline (Telegram → Postgres → Paperless → n8n)
+
+## Goal
+
+Give Doris a reliable way to ingest school work from Telegram while preserving explicit metadata:
+- class
+- assignment
+- submission kind (`draft`, `final`, etc.)
+- optional semester / instructor / due date / paper kind
+
+The design must be expandable for new classes and future non-paper workflows.
+
+## Recommended architecture
+
+1. **Telegram chat-driven intake**
+   - John sends file(s) to Doris on Telegram.
+   - Doris asks follow-up questions until required metadata is complete.
+
+2. **n8n intake webhook**
+   - Doris submits the file + metadata to an intake webhook.
+   - The webhook creates a durable intake record in shared Postgres.
+   - The webhook stages/uploads the file into Paperless with a deterministic `intake_id` embedded in the filename.
+
+3. **Paperless post-consumption webhook**
+   - After Paperless finishes processing the file, a second workflow fetches the Paperless document.
+   - That workflow extracts `intake_id`, looks up the Postgres intake record, and applies metadata.
+
+4. **Metadata + AI enrichment**
+   - Deterministic metadata first: class tags, version tags, title rules, optional correspondent/document type IDs.
+   - AI second: document summary/note and any low-risk enrichment.
+   - Ambiguous cases can be routed to a review queue.
+
+## Why this is better than folder-based consume ingestion
+
+Folder paths are a decent hint, but they are not durable enough to be the only source of truth.
+
+Problems with folder-only inference:
+- folder context can disappear after import
+- mixed uploads are easy to mislabel
+- retries/reprocessing can grab the wrong document
+- future use-cases (draft review, assignment history, instructor-specific rules) need explicit metadata anyway
+
+## Shared DB requirements
+
+Use shared Postgres, not SQLite.
+
+Schema artifact:
+- `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql`
+
+Core tables:
+- `school_paperless_intake`
+- `school_paperless_intake_events`
+
+## Config-driven class profiles
+
+Class/course behavior should live in config, not hardcoded workflow branches.
+
+Suggested profile fields:
+- `class_key`
+- `display_name`
+- `course_code`
+- `default_tags`
+- Paperless IDs for correspondent/document type/storage path when needed
+- title template override
+
+## Initial rollout target
+
+- ENGL-1010
+- HIST-2020
+
+## OpenClaw helper artifact
+
+Local helper:
+- `school/intake/submit_to_n8n.js`
+
+This gives Doris a simple handoff point after the Telegram conversation is complete.
+
+## Live work still needed
+
+- import/activate the new n8n intake workflow
+- import/activate the Paperless enrichment workflow
+- create shared Postgres database/user/schema
+- configure class profiles with real Paperless IDs
+- decide Telegram delivery wording/confirmation behavior
+- test with real ENGL + HIST documents
--- a/docs/planning/TODO.md
+++ b/docs/planning/TODO.md
@@ -41,6 +41,9 @@
 - [ ] Add sudoers NOPASSWD rule for n8n watchdog (see n8n-workflows/README.md)
 - [ ] Import and activate upgraded workflows: 04, 08 (01 done — v1.4 active and tested)
 - [ ] Import and activate new workflows: 14, 15 (16 already active)
+- [ ] Build Telegram chat-driven school-work intake for Paperless using shared Postgres, deterministic `intake_id`, and class/version metadata
+- [ ] Configure class profiles + Paperless metadata mapping for current courses (ENGL-1010, HIST-2020, future classes)
+- [ ] Set up DB and Docker backups to Serenity following homelab SOP

 ## Completed
 - [x] Deploy Gotify (Phase 1) — 2026-05-05
--- a/docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql
+++ b/docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql
@@ -0,0 +1,47 @@
+-- Shared Postgres schema for Telegram-driven school-work intake.
+-- Apply in a dedicated `school_intake` database or adapt to an existing shared DB.
+
+CREATE TABLE IF NOT EXISTS school_paperless_intake (
+  intake_id              TEXT PRIMARY KEY,
+  class_name             TEXT NOT NULL,
+  assignment_name        TEXT NOT NULL,
+  submission_kind        TEXT NOT NULL,
+  semester               TEXT,
+  course_code            TEXT,
+  paper_kind             TEXT,
+  notes                  TEXT,
+  source                 TEXT NOT NULL DEFAULT 'telegram',
+  telegram_chat_id       TEXT,
+  telegram_message_id    TEXT,
+  original_filename      TEXT NOT NULL,
+  stored_filename        TEXT NOT NULL,
+  mime_type              TEXT NOT NULL,
+  file_size_bytes        BIGINT NOT NULL,
+  checksum_sha256        TEXT NOT NULL,
+  status                 TEXT NOT NULL DEFAULT 'received',
+  paperless_task_id      TEXT,
+  paperless_document_id  BIGINT,
+  paperless_title        TEXT,
+  received_at            TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+  uploaded_at            TIMESTAMPTZ,
+  enriched_at            TIMESTAMPTZ,
+  error_text             TEXT,
+  created_at             TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+  updated_at             TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+CREATE INDEX IF NOT EXISTS idx_school_paperless_intake_status ON school_paperless_intake(status);
+CREATE INDEX IF NOT EXISTS idx_school_paperless_intake_doc_id ON school_paperless_intake(paperless_document_id);
+CREATE INDEX IF NOT EXISTS idx_school_paperless_intake_assignment ON school_paperless_intake(class_name, assignment_name);
+CREATE INDEX IF NOT EXISTS idx_school_paperless_intake_checksum ON school_paperless_intake(checksum_sha256);
+
+CREATE TABLE IF NOT EXISTS school_paperless_intake_events (
+  id              BIGSERIAL PRIMARY KEY,
+  intake_id       TEXT NOT NULL REFERENCES school_paperless_intake(intake_id) ON DELETE CASCADE,
+  event_type      TEXT NOT NULL,
+  event_payload   JSONB NOT NULL DEFAULT '{}'::jsonb,
+  created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+CREATE INDEX IF NOT EXISTS idx_school_paperless_intake_events_intake_id
+  ON school_paperless_intake_events(intake_id);