fizzlepoof/truenas-stacks
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(n8n): add school paperless intake pipeline

Browse Source
This commit is contained in:
Fizzlepoof
2026-05-13 20:53:00 +00:00
parent 5584de38e0
commit 56026cf542
11 changed files with 895 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

111
docs/operations/SCHOOL_PAPERLESS_INTAKE.md Normal file
View File

@@ -0,0 +1,111 @@
# School Paperless Intake Workflow
Workflows:
- `n8n-workflows/18-school-paperless-intake.json`
- `n8n-workflows/19-paperless-school-metadata-enrichment.json`
This workflow is meant for a Telegram chat-driven upload pipeline that sends schoolwork into Paperless-NGX while recording intake metadata in shared Postgres.
## What it does
1. Accepts a multipart upload at `POST /webhook/school/intake/upload`
2. Requires form fields:
- `class_name`
- `assignment_name`
- `submission_kind`
3. Accepts optional fields:
- `semester`
- `course_code`
- `paper_kind`
- `notes`
- `source` (defaults to `telegram`)
- `telegram_chat_id`
- `telegram_message_id`
4. Generates a deterministic `intake_id` from date + class + assignment + file checksum
5. Stores/updates the intake row in Postgres first
6. Uploads the file to Paperless using the deterministic intake ID in the filename
7. Marks the row as `uploaded` after Paperless accepts the upload
## Shared Postgres expectations
- Use the shared Postgres network/credentials pattern already used by the automation stack.
- The workflow expects an n8n Postgres credential named `Shared Postgres`.
- Recommended target database: `school_intake`
- Tracked schema: `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql`
## Required environment variables
Add to `automation/.env`:
```env
PAPERLESS_BASE_URL=https://paperless.paccoco.com
PAPERLESS_API_TOKEN=CHANGE_ME
LITELLM_API_KEY=CHANGE_ME
PAPERLESS_CORRESPONDENT_SCHOOL=
PAPERLESS_DOCUMENT_TYPE_SCHOOL=
PAPERLESS_TAG_SCHOOL=
PAPERLESS_TAG_ASSIGNMENTS=
PAPERLESS_TAG_TELEGRAM=
NODE_FUNCTION_ALLOW_BUILTIN=crypto
```
Add to the `n8n` service `environment:` block in `automation/docker-compose.yaml`:
```yaml
PAPERLESS_BASE_URL: ${PAPERLESS_BASE_URL}
PAPERLESS_API_TOKEN: ${PAPERLESS_API_TOKEN}
PAPERLESS_CORRESPONDENT_SCHOOL: ${PAPERLESS_CORRESPONDENT_SCHOOL}
PAPERLESS_DOCUMENT_TYPE_SCHOOL: ${PAPERLESS_DOCUMENT_TYPE_SCHOOL}
PAPERLESS_TAG_SCHOOL: ${PAPERLESS_TAG_SCHOOL}
PAPERLESS_TAG_ASSIGNMENTS: ${PAPERLESS_TAG_ASSIGNMENTS}
PAPERLESS_TAG_TELEGRAM: ${PAPERLESS_TAG_TELEGRAM}
NODE_FUNCTION_ALLOW_BUILTIN: crypto
```
Notes:
- `PAPERLESS_CORRESPONDENT_SCHOOL` and `PAPERLESS_DOCUMENT_TYPE_SCHOOL` should be numeric IDs if you want those fields auto-assigned.
- Leave optional Paperless IDs blank if you prefer Paperless rules to classify later.
- `NODE_FUNCTION_ALLOW_BUILTIN=crypto` is required because the Code node hashes the uploaded file.
- `N8N_BLOCK_ENV_ACCESS_IN_NODE` must remain `false`.
## Paperless metadata behavior
The upload uses:
- filename: `<intake_id>.<ext>`
- title: `<class_name> — <assignment_name> — <submission_kind>`
Workflow 18 deliberately keeps the Paperless upload minimal and safe:
- multipart `document`
- deterministic title
The deterministic filename/title preserve the `intake_id` for later correlation.
Workflow 19 handles the follow-up metadata pass after Paperless finishes processing:
- fetch Paperless document by webhook document id
- resolve `intake_id` from the stored filename
- look up the intake record in shared Postgres
- re-apply deterministic title
- merge generic school tags plus optional class/submission-kind mappings
- optionally set document type / correspondent from JSON env mappings
## Testing
Use the example curl request in:
- `n8n-workflows/fixtures/school-paperless-intake-webhook.curl.example.txt`
Minimum test checklist:
1. Import the workflow into n8n.
2. Attach the `Shared Postgres` credential.
3. Confirm the target DB/table exists.
4. Send the sample multipart request.
5. Verify a row appears in `school_paperless_intake` with `status=uploaded`.
6. Verify the uploaded Paperless document title and stored filename both include the deterministic intake ID.
7. Trigger workflow 19 with the Paperless document id and verify the row moves to `status=enriched`.
8. Re-submit the same file and metadata to confirm the same `intake_id` is reused and the Postgres row is upserted.
## Known assumptions / caveats
- The workflow is repo-only and was not live-tested against the running n8n/Paperless stack.
- The Postgres node uses SQL expressions inline rather than parameter binding because exported n8n node JSON can differ by version.
- If your n8n build blocks `require('crypto')`, add `NODE_FUNCTION_ALLOW_BUILTIN=crypto` to the container env.
- If Paperless needs tags as repeated `tags[]` fields instead of a comma-separated `tags` field, adjust the HTTP Request node after import.

85
docs/planning/SCHOOL_INTAKE_PIPELINE.md Normal file
View File

@@ -0,0 +1,85 @@
# School Work Intake Pipeline (Telegram → Postgres → Paperless → n8n)
## Goal
Give Doris a reliable way to ingest school work from Telegram while preserving explicit metadata:
- class
- assignment
- submission kind (`draft`, `final`, etc.)
- optional semester / instructor / due date / paper kind
The design must be expandable for new classes and future non-paper workflows.
## Recommended architecture
1. **Telegram chat-driven intake**
- John sends file(s) to Doris on Telegram.
- Doris asks follow-up questions until required metadata is complete.
2. **n8n intake webhook**
- Doris submits the file + metadata to an intake webhook.
- The webhook creates a durable intake record in shared Postgres.
- The webhook stages/uploads the file into Paperless with a deterministic `intake_id` embedded in the filename.
3. **Paperless post-consumption webhook**
- After Paperless finishes processing the file, a second workflow fetches the Paperless document.
- That workflow extracts `intake_id`, looks up the Postgres intake record, and applies metadata.
4. **Metadata + AI enrichment**
- Deterministic metadata first: class tags, version tags, title rules, optional correspondent/document type IDs.
- AI second: document summary/note and any low-risk enrichment.
- Ambiguous cases can be routed to a review queue.
## Why this is better than folder-based consume ingestion
Folder paths are a decent hint, but they are not durable enough to be the only source of truth.
Problems with folder-only inference:
- folder context can disappear after import
- mixed uploads are easy to mislabel
- retries/reprocessing can grab the wrong document
- future use-cases (draft review, assignment history, instructor-specific rules) need explicit metadata anyway
## Shared DB requirements
Use shared Postgres, not SQLite.
Schema artifact:
- `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql`
Core tables:
- `school_paperless_intake`
- `school_paperless_intake_events`
## Config-driven class profiles
Class/course behavior should live in config, not hardcoded workflow branches.
Suggested profile fields:
- `class_key`
- `display_name`
- `course_code`
- `default_tags`
- Paperless IDs for correspondent/document type/storage path when needed
- title template override
## Initial rollout target
- ENGL-1010
- HIST-2020
## OpenClaw helper artifact
Local helper:
- `school/intake/submit_to_n8n.js`
This gives Doris a simple handoff point after the Telegram conversation is complete.
## Live work still needed
- import/activate the new n8n intake workflow
- import/activate the Paperless enrichment workflow
- create shared Postgres database/user/schema
- configure class profiles with real Paperless IDs
- decide Telegram delivery wording/confirmation behavior
- test with real ENGL + HIST documents

3
docs/planning/TODO.md
View File

@@ -41,6 +41,9 @@
- [ ] Add sudoers NOPASSWD rule for n8n watchdog (see n8n-workflows/README.md)
- [ ] Import and activate upgraded workflows: 04, 08 (01 done — v1.4 active and tested)
- [ ] Import and activate new workflows: 14, 15 (16 already active)
- [ ] Build Telegram chat-driven school-work intake for Paperless using shared Postgres, deterministic `intake_id`, and class/version metadata
- [ ] Configure class profiles + Paperless metadata mapping for current courses (ENGL-1010, HIST-2020, future classes)
- [ ] Set up DB and Docker backups to Serenity following homelab SOP
## Completed
- [x] Deploy Gotify (Phase 1) — 2026-05-05

47
docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql Normal file
View File

@@ -0,0 +1,47 @@
-- Shared Postgres schema for Telegram-driven school-work intake.
-- Apply in a dedicated `school_intake` database or adapt to an existing shared DB.
CREATE TABLE IF NOT EXISTS school_paperless_intake (
intake_id TEXT PRIMARY KEY,
class_name TEXT NOT NULL,
assignment_name TEXT NOT NULL,
submission_kind TEXT NOT NULL,
semester TEXT,
course_code TEXT,
paper_kind TEXT,
notes TEXT,
source TEXT NOT NULL DEFAULT 'telegram',
telegram_chat_id TEXT,
telegram_message_id TEXT,
original_filename TEXT NOT NULL,
stored_filename TEXT NOT NULL,
mime_type TEXT NOT NULL,
file_size_bytes BIGINT NOT NULL,
checksum_sha256 TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'received',
paperless_task_id TEXT,
paperless_document_id BIGINT,
paperless_title TEXT,
received_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
uploaded_at TIMESTAMPTZ,
enriched_at TIMESTAMPTZ,
error_text TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_school_paperless_intake_status ON school_paperless_intake(status);
CREATE INDEX IF NOT EXISTS idx_school_paperless_intake_doc_id ON school_paperless_intake(paperless_document_id);
CREATE INDEX IF NOT EXISTS idx_school_paperless_intake_assignment ON school_paperless_intake(class_name, assignment_name);
CREATE INDEX IF NOT EXISTS idx_school_paperless_intake_checksum ON school_paperless_intake(checksum_sha256);
CREATE TABLE IF NOT EXISTS school_paperless_intake_events (
id BIGSERIAL PRIMARY KEY,
intake_id TEXT NOT NULL REFERENCES school_paperless_intake(intake_id) ON DELETE CASCADE,
event_type TEXT NOT NULL,
event_payload JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_school_paperless_intake_events_intake_id
ON school_paperless_intake_events(intake_id);