diff --git a/docs/operations/SCHOOL_PAPERLESS_INTAKE.md b/docs/operations/SCHOOL_PAPERLESS_INTAKE.md index 5922fd1..4158f77 100644 --- a/docs/operations/SCHOOL_PAPERLESS_INTAKE.md +++ b/docs/operations/SCHOOL_PAPERLESS_INTAKE.md @@ -1,7 +1,7 @@ # School Paperless Intake Workflow Current workflow versions: -- `n8n-workflows/18-school-paperless-intake-v1.0.json` +- `n8n-workflows/18-school-paperless-intake-v1.1.json` - `n8n-workflows/19-paperless-school-metadata-enrichment-v1.0.json` This workflow set is meant for a Telegram chat-driven upload pipeline that sends schoolwork into Paperless-NGX while recording intake metadata in shared Postgres. @@ -15,7 +15,7 @@ When these workflow artifacts change, the version must change too: ## What it does -### Workflow 18 — v1.0 +### Workflow 18 — v1.1 1. Accepts a multipart upload at `POST /webhook/school/intake/upload` 2. Requires form fields: @@ -39,7 +39,9 @@ When these workflow artifacts change, the version must change too: - Use the shared Postgres network/credentials pattern already used by the automation stack. - The workflow expects an n8n Postgres credential named `Shared Postgres`. -- Recommended target database: `school_intake` +- Preferred target database: dedicated `school_intake` +- Current fast-path/live bootstrap also works against the existing `n8n` database, provided the schema tables are created there. +- If you are using the current PD bootstrap path, point the n8n credential at database `n8n` with the existing Postgres account. - Tracked schema: `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql` ## Required environment variables @@ -83,7 +85,7 @@ Notes: - `PAPERLESS_CORRESPONDENT_SCHOOL` and `PAPERLESS_DOCUMENT_TYPE_SCHOOL` should be numeric IDs if you want those fields auto-assigned. - `PAPERLESS_TAG_CLASS_MAP_JSON`, `PAPERLESS_TAG_SUBMISSION_KIND_MAP_JSON`, `PAPERLESS_DOCUMENT_TYPE_PAPER_KIND_MAP_JSON`, and `PAPERLESS_CORRESPONDENT_CLASS_MAP_JSON` are optional JSON maps used by workflow 19. - Leave optional Paperless IDs and JSON maps blank/default if you prefer to start with deterministic title + generic tags only. -- `NODE_FUNCTION_ALLOW_BUILTIN=crypto` is required because the Code node hashes the uploaded file. +- Workflow 18 v1.1 no longer requires `crypto`, so `NODE_FUNCTION_ALLOW_BUILTIN=crypto` is optional instead of required. - `N8N_BLOCK_ENV_ACCESS_IN_NODE` must remain `false`. ## Paperless metadata behavior @@ -116,7 +118,7 @@ Use the example curl request in: Minimum test checklist: 1. Import the workflow into n8n. 2. Attach the `Shared Postgres` credential. -3. Confirm the target DB/table exists. +3. Confirm the target DB/table exists in either `school_intake` or `n8n`, depending on which credential/database you are using. 4. Send the sample multipart request. 5. Verify a row appears in `school_paperless_intake` with `status=uploaded`. 6. Verify the uploaded Paperless document title and stored filename both include the deterministic intake ID. @@ -127,5 +129,5 @@ Minimum test checklist: - The workflow is repo-only and was not live-tested against the running n8n/Paperless stack. - The Postgres node uses SQL expressions inline rather than parameter binding because exported n8n node JSON can differ by version. -- If your n8n build blocks `require('crypto')`, add `NODE_FUNCTION_ALLOW_BUILTIN=crypto` to the container env. +- Workflow 18 v1.0 needed `require('crypto')`; workflow 18 v1.1 removed that dependency for easier live rollout. - If Paperless needs tags as repeated `tags[]` fields instead of a comma-separated `tags` field, adjust the HTTP Request node after import. diff --git a/n8n-workflows/18-school-paperless-intake-v1.1.json b/n8n-workflows/18-school-paperless-intake-v1.1.json new file mode 100644 index 0000000..4c3eb5b --- /dev/null +++ b/n8n-workflows/18-school-paperless-intake-v1.1.json @@ -0,0 +1,238 @@ +{ + "name": "Telegram School Intake → Postgres → Paperless v1.1", + "active": false, + "isArchived": false, + "nodes": [ + { + "parameters": { + "httpMethod": "POST", + "path": "school/intake/upload", + "responseMode": "lastNode", + "responseData": "allEntries", + "options": { + "rawBody": true + } + }, + "id": "school-intake-webhook", + "name": "School Intake Webhook", + "type": "n8n-nodes-base.webhook", + "typeVersion": 2, + "position": [ + -1180, + 0 + ] + }, + { + "parameters": { + "jsCode": "\nconst req = $input.first().json;\nconst body = req.body || {};\nconst binary = $input.first().binary || {};\nconst upload = binary.data || Object.values(binary)[0];\nif (!upload) throw new Error('Expected multipart file upload in the webhook request.');\nconst required = ['class_name', 'assignment_name', 'submission_kind'];\nfor (const key of required) {\n if (!body[key] || !String(body[key]).trim()) {\n throw new Error('Missing required form field: ' + key + '. Required fields: class_name, assignment_name, submission_kind.');\n }\n}\nconst sanitize = (value, fallback = 'item') => String(value || fallback)\n .trim()\n .toLowerCase()\n .replace(/[^a-z0-9]+/g, '-')\n .replace(/^-+|-+$/g, '')\n .slice(0, 80) || fallback;\nconst stableHash = (str) => {\n let h1 = 0xdeadbeef;\n let h2 = 0x41c6ce57;\n for (let i = 0; i < str.length; i++) {\n const ch = str.charCodeAt(i);\n h1 = Math.imul(h1 ^ ch, 2654435761);\n h2 = Math.imul(h2 ^ ch, 1597334677);\n }\n h1 = Math.imul(h1 ^ (h1 >>> 16), 2246822507) ^ Math.imul(h2 ^ (h2 >>> 13), 3266489909);\n h2 = Math.imul(h2 ^ (h2 >>> 16), 2246822507) ^ Math.imul(h1 ^ (h1 >>> 13), 3266489909);\n return ((h2 >>> 0).toString(16).padStart(8, '0') + (h1 >>> 0).toString(16).padStart(8, '0'));\n};\nconst originalName = upload.fileName || body.filename || 'upload.bin';\nconst extension = (() => {\n const m = String(originalName).match(/(\\.[A-Za-z0-9]{1,10})$/);\n return m ? m[1].toLowerCase() : '';\n})();\nconst fileBytes = Buffer.from(upload.data, 'base64');\nconst checksum = stableHash([originalName, upload.mimeType || '', String(fileBytes.length), upload.data].join('|'));\nconst now = new Date();\nconst intakeId = [\n 'school',\n now.toISOString().slice(0,10).replace(/-/g,''),\n sanitize(body.class_name),\n sanitize(body.assignment_name),\n checksum.slice(0, 12)\n].join('-');\nconst storedFilename = intakeId + extension;\nconst title = [body.class_name, body.assignment_name, body.submission_kind].map(v => String(v).trim()).join(' — ');\nreturn [{\n json: {\n intake_id: intakeId,\n class_name: String(body.class_name).trim(),\n assignment_name: String(body.assignment_name).trim(),\n submission_kind: String(body.submission_kind).trim(),\n semester: body.semester ? String(body.semester).trim() : null,\n course_code: body.course_code ? String(body.course_code).trim() : null,\n paper_kind: body.paper_kind ? String(body.paper_kind).trim() : null,\n notes: body.notes ? String(body.notes).trim().slice(0, 4000) : null,\n source: body.source ? String(body.source).trim() : 'telegram',\n telegram_chat_id: body.telegram_chat_id ? String(body.telegram_chat_id).trim() : null,\n telegram_message_id: body.telegram_message_id ? String(body.telegram_message_id).trim() : null,\n original_filename: originalName,\n stored_filename: storedFilename,\n mime_type: upload.mimeType || 'application/octet-stream',\n file_size_bytes: fileBytes.length,\n checksum_sha256: checksum,\n paperless_title: title,\n received_at: now.toISOString()\n },\n binary: {\n data: {\n ...upload,\n fileName: storedFilename,\n mimeType: upload.mimeType || 'application/octet-stream'\n }\n }\n}];" + }, + "id": "normalize-request", + "name": "Normalize Intake Request", + "type": "n8n-nodes-base.code", + "typeVersion": 2, + "position": [ + -920, + 0 + ] + }, + { + "parameters": { + "operation": "executeQuery", + "query": "INSERT INTO school_paperless_intake (\n intake_id,\n class_name,\n assignment_name,\n submission_kind,\n semester,\n course_code,\n paper_kind,\n notes,\n source,\n telegram_chat_id,\n telegram_message_id,\n original_filename,\n stored_filename,\n mime_type,\n file_size_bytes,\n checksum_sha256,\n status,\n received_at\n)\nVALUES (\n '{{ $json.intake_id }}',\n '{{ $json.class_name.replace(/'/g, \"''\") }}',\n '{{ $json.assignment_name.replace(/'/g, \"''\") }}',\n '{{ $json.submission_kind.replace(/'/g, \"''\") }}',\n {{ $json.semester ? \"'\" + $json.semester.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n {{ $json.course_code ? \"'\" + $json.course_code.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n {{ $json.paper_kind ? \"'\" + $json.paper_kind.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n {{ $json.notes ? \"'\" + $json.notes.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n '{{ $json.source.replace(/'/g, \"''\") }}',\n {{ $json.telegram_chat_id ? \"'\" + $json.telegram_chat_id.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n {{ $json.telegram_message_id ? \"'\" + $json.telegram_message_id.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n '{{ $json.original_filename.replace(/'/g, \"''\") }}',\n '{{ $json.stored_filename.replace(/'/g, \"''\") }}',\n '{{ $json.mime_type.replace(/'/g, \"''\") }}',\n {{ $json.file_size_bytes }},\n '{{ $json.checksum_sha256 }}',\n 'received',\n '{{ $json.received_at }}'::timestamptz\n)\nON CONFLICT (intake_id) DO UPDATE SET\n class_name = EXCLUDED.class_name,\n assignment_name = EXCLUDED.assignment_name,\n submission_kind = EXCLUDED.submission_kind,\n semester = EXCLUDED.semester,\n course_code = EXCLUDED.course_code,\n paper_kind = EXCLUDED.paper_kind,\n notes = EXCLUDED.notes,\n source = EXCLUDED.source,\n telegram_chat_id = EXCLUDED.telegram_chat_id,\n telegram_message_id = EXCLUDED.telegram_message_id,\n original_filename = EXCLUDED.original_filename,\n stored_filename = EXCLUDED.stored_filename,\n mime_type = EXCLUDED.mime_type,\n file_size_bytes = EXCLUDED.file_size_bytes,\n checksum_sha256 = EXCLUDED.checksum_sha256,\n status = 'received',\n received_at = EXCLUDED.received_at,\n updated_at = NOW();" + }, + "id": "pg-insert-received", + "name": "Postgres Upsert Intake Received", + "type": "n8n-nodes-base.postgres", + "typeVersion": 2.6, + "position": [ + -660, + 0 + ], + "credentials": { + "postgres": { + "id": "SETUP_REQUIRED", + "name": "Shared Postgres" + } + } + }, + { + "parameters": { + "method": "POST", + "url": "={{($env.PAPERLESS_BASE_URL || 'https://paperless.paccoco.com').replace(/\\/$/, '') + '/api/documents/post_document/'}}", + "sendHeaders": true, + "headerParameters": { + "parameters": [ + { + "name": "Authorization", + "value": "={{'Token ' + $env.PAPERLESS_API_TOKEN}}" + } + ] + }, + "sendBody": true, + "contentType": "multipart-form-data", + "bodyParameters": { + "parameters": [ + { + "name": "document", + "value": "data", + "parameterType": "formBinaryData" + }, + { + "name": "title", + "value": "={{$json.paperless_title}}" + } + ] + }, + "options": { + "timeout": 120000 + } + }, + "id": "paperless-upload", + "name": "Upload to Paperless", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.2, + "position": [ + -400, + 0 + ] + }, + { + "parameters": { + "jsCode": "\nconst intake = $('Normalize Intake Request').first().json;\nconst response = $json;\nconst taskId = response.task_id || response.id || response.document?.id || null;\nlet paperlessDocumentId = response.document_id || response.document?.id || null;\nif (!paperlessDocumentId && typeof response === 'object' && response?.id && !response?.task_id) {\n paperlessDocumentId = response.id;\n}\nreturn [{ json: {\n ...intake,\n paperless_task_id: taskId ? String(taskId) : null,\n paperless_document_id: paperlessDocumentId ? String(paperlessDocumentId) : null,\n paperless_response: response,\n uploaded_at: new Date().toISOString()\n}}];" + }, + "id": "capture-paperless-response", + "name": "Capture Paperless Response", + "type": "n8n-nodes-base.code", + "typeVersion": 2, + "position": [ + -140, + 0 + ] + }, + { + "parameters": { + "operation": "executeQuery", + "query": "UPDATE school_paperless_intake\nSET\n paperless_task_id = {{ $json.paperless_task_id ? \"'\" + $json.paperless_task_id.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n paperless_document_id = {{ $json.paperless_document_id ? \"'\" + $json.paperless_document_id.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n paperless_title = '{{ $json.paperless_title.replace(/'/g, \"''\") }}',\n status = 'uploaded',\n uploaded_at = '{{ $json.uploaded_at }}'::timestamptz,\n updated_at = NOW()\nWHERE intake_id = '{{ $json.intake_id }}';" + }, + "id": "pg-mark-uploaded", + "name": "Postgres Mark Uploaded", + "type": "n8n-nodes-base.postgres", + "typeVersion": 2.6, + "position": [ + 120, + 0 + ], + "credentials": { + "postgres": { + "id": "SETUP_REQUIRED", + "name": "Shared Postgres" + } + } + }, + { + "parameters": { + "jsCode": "\nreturn [{\n json: {\n ok: true,\n intake_id: $json.intake_id,\n status: 'uploaded',\n paperless_task_id: $json.paperless_task_id,\n paperless_document_id: $json.paperless_document_id,\n paperless_title: $json.paperless_title,\n stored_filename: $json.stored_filename,\n received_at: $json.received_at,\n uploaded_at: $json.uploaded_at\n }\n}];" + }, + "id": "success-response", + "name": "Return Success Payload", + "type": "n8n-nodes-base.code", + "typeVersion": 2, + "position": [ + 360, + 0 + ] + } + ], + "connections": { + "School Intake Webhook": { + "main": [ + [ + { + "node": "Normalize Intake Request", + "type": "main", + "index": 0 + } + ] + ] + }, + "Normalize Intake Request": { + "main": [ + [ + { + "node": "Postgres Upsert Intake Received", + "type": "main", + "index": 0 + } + ] + ] + }, + "Postgres Upsert Intake Received": { + "main": [ + [ + { + "node": "Upload to Paperless", + "type": "main", + "index": 0 + } + ] + ] + }, + "Upload to Paperless": { + "main": [ + [ + { + "node": "Capture Paperless Response", + "type": "main", + "index": 0 + } + ] + ] + }, + "Capture Paperless Response": { + "main": [ + [ + { + "node": "Postgres Mark Uploaded", + "type": "main", + "index": 0 + } + ] + ] + }, + "Postgres Mark Uploaded": { + "main": [ + [ + { + "node": "Return Success Payload", + "type": "main", + "index": 0 + } + ] + ] + } + }, + "settings": { + "executionOrder": "v1", + "binaryMode": "separate" + }, + "staticData": null, + "meta": { + "templateCredsSetupCompleted": false, + "artifactVersion": "v1.1" + }, + "pinData": {}, + "tags": [ + { + "name": "school" + }, + { + "name": "paperless" + }, + { + "name": "postgres" + }, + { + "name": "telegram" + } + ] +} diff --git a/n8n-workflows/README.md b/n8n-workflows/README.md index 506c02d..52ed661 100644 --- a/n8n-workflows/README.md +++ b/n8n-workflows/README.md @@ -15,7 +15,7 @@ | 14 | Paperless Inbox Reminder | Scheduled (Monday 9 AM) | Queries Paperless for untagged documents; sends Gotify reminder with oldest items listed, or a ✅ clear if inbox is empty | | 15 | n8n Self-Health Monitor | Scheduled (every 15 min) | Checks n8n execution history for failures in the past hour; deduplicates via Postgres; pushes Gotify alert with workflow name and error summary | | 16 | NFS Mount Watchdog | Scheduled (every 5 min) | Probes NFS mount points on PD; if missing: runs mount script, re-probes, restarts affected containers (Plex, Audiobookshelf, Immich); notifies via Gotify on heal or escalation | -| 18 | Telegram School Intake → Postgres → Paperless v1.0 | `POST /webhook/school/intake/upload` | Accepts multipart schoolwork uploads plus class/assignment metadata, writes an intake record to shared Postgres, then uploads into Paperless with deterministic `intake_id` in the filename/title | +| 18 | Telegram School Intake → Postgres → Paperless v1.1 | `POST /webhook/school/intake/upload` | Accepts multipart schoolwork uploads plus class/assignment metadata, writes an intake record to shared Postgres, then uploads into Paperless with deterministic `intake_id` in the filename/title | | 19 | Paperless School Intake Metadata Enrichment v1.0 | `POST /webhook/school/intake/paperless-enrich` | Handles Paperless post-consumption webhooks, looks up the intake record by deterministic filename, then applies deterministic title/tags and optional document-type/correspondent mapping | ## How to Import @@ -289,9 +289,9 @@ curl -X POST https://n8n.paccoco.com/webhook/git/push \ }' ``` -## Workflow 18 — Telegram School Intake → Postgres → Paperless v1.0 +## Workflow 18 — Telegram School Intake → Postgres → Paperless v1.1 -Import `18-school-paperless-intake-v1.0.json` for a chat-driven schoolwork intake flow. Supporting notes live at `docs/operations/SCHOOL_PAPERLESS_INTAKE.md` and the tracked schema lives at `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql`. +Import `18-school-paperless-intake-v1.1.json` for a chat-driven schoolwork intake flow. Supporting notes live at `docs/operations/SCHOOL_PAPERLESS_INTAKE.md` and the tracked schema lives at `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql`. Highlights: - multipart upload endpoint: `POST /webhook/school/intake/upload` @@ -299,6 +299,7 @@ Highlights: - optional fields: `semester`, `course_code`, `paper_kind`, `notes`, `telegram_chat_id`, `telegram_message_id` - writes intake metadata to shared Postgres before upload - sends deterministic `intake_id` in the Paperless filename and title correlation data +- v1.1 removes the `require('crypto')` dependency so the workflow can run without adding `crypto` to `NODE_FUNCTION_ALLOW_BUILTIN` Testing fixture: - `fixtures/school-paperless-intake-webhook.curl.example.txt` @@ -306,7 +307,7 @@ Testing fixture: ## Workflow Versioning Convention For n8n workflow artifacts in this repo: -- include the version in the **filename** (example: `18-school-paperless-intake-v1.0.json`) +- include the version in the **filename** (example: `18-school-paperless-intake-v1.1.json`) - include the version in the workflow's internal **`name`** field - bump the version whenever behavior, schema expectations, webhook payload handling, or env requirements change - update `n8n-workflows/README.md` and `n8n-workflows/TESTING-STATUS.md` when introducing a new workflow version diff --git a/n8n-workflows/TESTING-STATUS.md b/n8n-workflows/TESTING-STATUS.md index 42a43b5..afc4d04 100644 --- a/n8n-workflows/TESTING-STATUS.md +++ b/n8n-workflows/TESTING-STATUS.md @@ -108,13 +108,14 @@ _Last updated: 2026-05-13_ --- -### 18 — Telegram School Intake → Postgres → Paperless v1.0 +### 18 — Telegram School Intake → Postgres → Paperless v1.1 **What it does:** Accepts multipart school-work upload plus metadata (`class_name`, `assignment_name`, `submission_kind`, optional semester/course_code/paper_kind), records it in shared Postgres, and uploads it into Paperless with a deterministic `intake_id`-based filename. **Local artifact checks completed:** - [x] Workflow JSON validates with `python3 -m json.tool` - [x] Companion helper script validates with `node --check school/intake/submit_to_n8n.js` - [x] Class profile example JSON validates with `python3 -m json.tool` +- [x] v1.1 removes the `require('crypto')` dependency for easier live rollout **Pre-test checklist:** - [ ] Shared Postgres table `school_paperless_intake` created from `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql`