feat(n8n): add school paperless intake pipeline
This commit is contained in:
Fizzlepoof
@@ -3,3 +3,15 @@ TZ=America/Chicago
|
|||||||
N8N_HOST=n8n.paccoco.com
|
N8N_HOST=n8n.paccoco.com
|
||||||
N8N_DB_PASS=CHANGE_ME
|
N8N_DB_PASS=CHANGE_ME
|
||||||
N8N_ENCRYPTION_KEY=CHANGE_ME
|
N8N_ENCRYPTION_KEY=CHANGE_ME
|
||||||
|
PAPERLESS_BASE_URL=https://paperless.paccoco.com
|
||||||
|
PAPERLESS_API_TOKEN=CHANGE_ME
|
||||||
|
PAPERLESS_CORRESPONDENT_SCHOOL=
|
||||||
|
PAPERLESS_DOCUMENT_TYPE_SCHOOL=
|
||||||
|
PAPERLESS_TAG_SCHOOL=
|
||||||
|
PAPERLESS_TAG_ASSIGNMENTS=
|
||||||
|
PAPERLESS_TAG_TELEGRAM=
|
||||||
|
PAPERLESS_TAG_CLASS_MAP_JSON={}
|
||||||
|
PAPERLESS_TAG_SUBMISSION_KIND_MAP_JSON={}
|
||||||
|
PAPERLESS_DOCUMENT_TYPE_PAPER_KIND_MAP_JSON={}
|
||||||
|
PAPERLESS_CORRESPONDENT_CLASS_MAP_JSON={}
|
||||||
|
NODE_FUNCTION_ALLOW_BUILTIN=crypto
|
||||||
|
|||||||
@@ -40,6 +40,18 @@ services:
|
|||||||
EXECUTIONS_DATA_MAX_AGE: 336
|
EXECUTIONS_DATA_MAX_AGE: 336
|
||||||
N8N_BLOCK_ENV_ACCESS_IN_NODE: "false"
|
N8N_BLOCK_ENV_ACCESS_IN_NODE: "false"
|
||||||
LITELLM_API_KEY: ${LITELLM_API_KEY}
|
LITELLM_API_KEY: ${LITELLM_API_KEY}
|
||||||
|
PAPERLESS_BASE_URL: ${PAPERLESS_BASE_URL}
|
||||||
|
PAPERLESS_API_TOKEN: ${PAPERLESS_API_TOKEN}
|
||||||
|
PAPERLESS_CORRESPONDENT_SCHOOL: ${PAPERLESS_CORRESPONDENT_SCHOOL}
|
||||||
|
PAPERLESS_DOCUMENT_TYPE_SCHOOL: ${PAPERLESS_DOCUMENT_TYPE_SCHOOL}
|
||||||
|
PAPERLESS_TAG_SCHOOL: ${PAPERLESS_TAG_SCHOOL}
|
||||||
|
PAPERLESS_TAG_ASSIGNMENTS: ${PAPERLESS_TAG_ASSIGNMENTS}
|
||||||
|
PAPERLESS_TAG_TELEGRAM: ${PAPERLESS_TAG_TELEGRAM}
|
||||||
|
PAPERLESS_TAG_CLASS_MAP_JSON: ${PAPERLESS_TAG_CLASS_MAP_JSON}
|
||||||
|
PAPERLESS_TAG_SUBMISSION_KIND_MAP_JSON: ${PAPERLESS_TAG_SUBMISSION_KIND_MAP_JSON}
|
||||||
|
PAPERLESS_DOCUMENT_TYPE_PAPER_KIND_MAP_JSON: ${PAPERLESS_DOCUMENT_TYPE_PAPER_KIND_MAP_JSON}
|
||||||
|
PAPERLESS_CORRESPONDENT_CLASS_MAP_JSON: ${PAPERLESS_CORRESPONDENT_CLASS_MAP_JSON}
|
||||||
|
NODE_FUNCTION_ALLOW_BUILTIN: ${NODE_FUNCTION_ALLOW_BUILTIN}
|
||||||
volumes:
|
volumes:
|
||||||
- /mnt/tank/docker/appdata/n8n:/home/node/.n8n
|
- /mnt/tank/docker/appdata/n8n:/home/node/.n8n
|
||||||
- /mnt/data/class-recordings:/data/class-recordings
|
- /mnt/data/class-recordings:/data/class-recordings
|
||||||
|
|||||||
111
docs/operations/SCHOOL_PAPERLESS_INTAKE.md
Normal file
111
docs/operations/SCHOOL_PAPERLESS_INTAKE.md
Normal file
@@ -0,0 +1,111 @@
|
|||||||
|
# School Paperless Intake Workflow
|
||||||
|
|
||||||
|
Workflows:
|
||||||
|
- `n8n-workflows/18-school-paperless-intake.json`
|
||||||
|
- `n8n-workflows/19-paperless-school-metadata-enrichment.json`
|
||||||
|
|
||||||
|
This workflow is meant for a Telegram chat-driven upload pipeline that sends schoolwork into Paperless-NGX while recording intake metadata in shared Postgres.
|
||||||
|
|
||||||
|
## What it does
|
||||||
|
|
||||||
|
1. Accepts a multipart upload at `POST /webhook/school/intake/upload`
|
||||||
|
2. Requires form fields:
|
||||||
|
- `class_name`
|
||||||
|
- `assignment_name`
|
||||||
|
- `submission_kind`
|
||||||
|
3. Accepts optional fields:
|
||||||
|
- `semester`
|
||||||
|
- `course_code`
|
||||||
|
- `paper_kind`
|
||||||
|
- `notes`
|
||||||
|
- `source` (defaults to `telegram`)
|
||||||
|
- `telegram_chat_id`
|
||||||
|
- `telegram_message_id`
|
||||||
|
4. Generates a deterministic `intake_id` from date + class + assignment + file checksum
|
||||||
|
5. Stores/updates the intake row in Postgres first
|
||||||
|
6. Uploads the file to Paperless using the deterministic intake ID in the filename
|
||||||
|
7. Marks the row as `uploaded` after Paperless accepts the upload
|
||||||
|
|
||||||
|
## Shared Postgres expectations
|
||||||
|
|
||||||
|
- Use the shared Postgres network/credentials pattern already used by the automation stack.
|
||||||
|
- The workflow expects an n8n Postgres credential named `Shared Postgres`.
|
||||||
|
- Recommended target database: `school_intake`
|
||||||
|
- Tracked schema: `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql`
|
||||||
|
|
||||||
|
## Required environment variables
|
||||||
|
|
||||||
|
Add to `automation/.env`:
|
||||||
|
|
||||||
|
```env
|
||||||
|
PAPERLESS_BASE_URL=https://paperless.paccoco.com
|
||||||
|
PAPERLESS_API_TOKEN=CHANGE_ME
|
||||||
|
LITELLM_API_KEY=CHANGE_ME
|
||||||
|
PAPERLESS_CORRESPONDENT_SCHOOL=
|
||||||
|
PAPERLESS_DOCUMENT_TYPE_SCHOOL=
|
||||||
|
PAPERLESS_TAG_SCHOOL=
|
||||||
|
PAPERLESS_TAG_ASSIGNMENTS=
|
||||||
|
PAPERLESS_TAG_TELEGRAM=
|
||||||
|
NODE_FUNCTION_ALLOW_BUILTIN=crypto
|
||||||
|
```
|
||||||
|
|
||||||
|
Add to the `n8n` service `environment:` block in `automation/docker-compose.yaml`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
PAPERLESS_BASE_URL: ${PAPERLESS_BASE_URL}
|
||||||
|
PAPERLESS_API_TOKEN: ${PAPERLESS_API_TOKEN}
|
||||||
|
PAPERLESS_CORRESPONDENT_SCHOOL: ${PAPERLESS_CORRESPONDENT_SCHOOL}
|
||||||
|
PAPERLESS_DOCUMENT_TYPE_SCHOOL: ${PAPERLESS_DOCUMENT_TYPE_SCHOOL}
|
||||||
|
PAPERLESS_TAG_SCHOOL: ${PAPERLESS_TAG_SCHOOL}
|
||||||
|
PAPERLESS_TAG_ASSIGNMENTS: ${PAPERLESS_TAG_ASSIGNMENTS}
|
||||||
|
PAPERLESS_TAG_TELEGRAM: ${PAPERLESS_TAG_TELEGRAM}
|
||||||
|
NODE_FUNCTION_ALLOW_BUILTIN: crypto
|
||||||
|
```
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
- `PAPERLESS_CORRESPONDENT_SCHOOL` and `PAPERLESS_DOCUMENT_TYPE_SCHOOL` should be numeric IDs if you want those fields auto-assigned.
|
||||||
|
- Leave optional Paperless IDs blank if you prefer Paperless rules to classify later.
|
||||||
|
- `NODE_FUNCTION_ALLOW_BUILTIN=crypto` is required because the Code node hashes the uploaded file.
|
||||||
|
- `N8N_BLOCK_ENV_ACCESS_IN_NODE` must remain `false`.
|
||||||
|
|
||||||
|
## Paperless metadata behavior
|
||||||
|
|
||||||
|
The upload uses:
|
||||||
|
- filename: `<intake_id>.<ext>`
|
||||||
|
- title: `<class_name> — <assignment_name> — <submission_kind>`
|
||||||
|
|
||||||
|
Workflow 18 deliberately keeps the Paperless upload minimal and safe:
|
||||||
|
- multipart `document`
|
||||||
|
- deterministic title
|
||||||
|
|
||||||
|
The deterministic filename/title preserve the `intake_id` for later correlation.
|
||||||
|
|
||||||
|
Workflow 19 handles the follow-up metadata pass after Paperless finishes processing:
|
||||||
|
- fetch Paperless document by webhook document id
|
||||||
|
- resolve `intake_id` from the stored filename
|
||||||
|
- look up the intake record in shared Postgres
|
||||||
|
- re-apply deterministic title
|
||||||
|
- merge generic school tags plus optional class/submission-kind mappings
|
||||||
|
- optionally set document type / correspondent from JSON env mappings
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
Use the example curl request in:
|
||||||
|
- `n8n-workflows/fixtures/school-paperless-intake-webhook.curl.example.txt`
|
||||||
|
|
||||||
|
Minimum test checklist:
|
||||||
|
1. Import the workflow into n8n.
|
||||||
|
2. Attach the `Shared Postgres` credential.
|
||||||
|
3. Confirm the target DB/table exists.
|
||||||
|
4. Send the sample multipart request.
|
||||||
|
5. Verify a row appears in `school_paperless_intake` with `status=uploaded`.
|
||||||
|
6. Verify the uploaded Paperless document title and stored filename both include the deterministic intake ID.
|
||||||
|
7. Trigger workflow 19 with the Paperless document id and verify the row moves to `status=enriched`.
|
||||||
|
8. Re-submit the same file and metadata to confirm the same `intake_id` is reused and the Postgres row is upserted.
|
||||||
|
|
||||||
|
## Known assumptions / caveats
|
||||||
|
|
||||||
|
- The workflow is repo-only and was not live-tested against the running n8n/Paperless stack.
|
||||||
|
- The Postgres node uses SQL expressions inline rather than parameter binding because exported n8n node JSON can differ by version.
|
||||||
|
- If your n8n build blocks `require('crypto')`, add `NODE_FUNCTION_ALLOW_BUILTIN=crypto` to the container env.
|
||||||
|
- If Paperless needs tags as repeated `tags[]` fields instead of a comma-separated `tags` field, adjust the HTTP Request node after import.
|
||||||
85
docs/planning/SCHOOL_INTAKE_PIPELINE.md
Normal file
85
docs/planning/SCHOOL_INTAKE_PIPELINE.md
Normal file
@@ -0,0 +1,85 @@
|
|||||||
|
# School Work Intake Pipeline (Telegram → Postgres → Paperless → n8n)
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Give Doris a reliable way to ingest school work from Telegram while preserving explicit metadata:
|
||||||
|
- class
|
||||||
|
- assignment
|
||||||
|
- submission kind (`draft`, `final`, etc.)
|
||||||
|
- optional semester / instructor / due date / paper kind
|
||||||
|
|
||||||
|
The design must be expandable for new classes and future non-paper workflows.
|
||||||
|
|
||||||
|
## Recommended architecture
|
||||||
|
|
||||||
|
1. **Telegram chat-driven intake**
|
||||||
|
- John sends file(s) to Doris on Telegram.
|
||||||
|
- Doris asks follow-up questions until required metadata is complete.
|
||||||
|
|
||||||
|
2. **n8n intake webhook**
|
||||||
|
- Doris submits the file + metadata to an intake webhook.
|
||||||
|
- The webhook creates a durable intake record in shared Postgres.
|
||||||
|
- The webhook stages/uploads the file into Paperless with a deterministic `intake_id` embedded in the filename.
|
||||||
|
|
||||||
|
3. **Paperless post-consumption webhook**
|
||||||
|
- After Paperless finishes processing the file, a second workflow fetches the Paperless document.
|
||||||
|
- That workflow extracts `intake_id`, looks up the Postgres intake record, and applies metadata.
|
||||||
|
|
||||||
|
4. **Metadata + AI enrichment**
|
||||||
|
- Deterministic metadata first: class tags, version tags, title rules, optional correspondent/document type IDs.
|
||||||
|
- AI second: document summary/note and any low-risk enrichment.
|
||||||
|
- Ambiguous cases can be routed to a review queue.
|
||||||
|
|
||||||
|
## Why this is better than folder-based consume ingestion
|
||||||
|
|
||||||
|
Folder paths are a decent hint, but they are not durable enough to be the only source of truth.
|
||||||
|
|
||||||
|
Problems with folder-only inference:
|
||||||
|
- folder context can disappear after import
|
||||||
|
- mixed uploads are easy to mislabel
|
||||||
|
- retries/reprocessing can grab the wrong document
|
||||||
|
- future use-cases (draft review, assignment history, instructor-specific rules) need explicit metadata anyway
|
||||||
|
|
||||||
|
## Shared DB requirements
|
||||||
|
|
||||||
|
Use shared Postgres, not SQLite.
|
||||||
|
|
||||||
|
Schema artifact:
|
||||||
|
- `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql`
|
||||||
|
|
||||||
|
Core tables:
|
||||||
|
- `school_paperless_intake`
|
||||||
|
- `school_paperless_intake_events`
|
||||||
|
|
||||||
|
## Config-driven class profiles
|
||||||
|
|
||||||
|
Class/course behavior should live in config, not hardcoded workflow branches.
|
||||||
|
|
||||||
|
Suggested profile fields:
|
||||||
|
- `class_key`
|
||||||
|
- `display_name`
|
||||||
|
- `course_code`
|
||||||
|
- `default_tags`
|
||||||
|
- Paperless IDs for correspondent/document type/storage path when needed
|
||||||
|
- title template override
|
||||||
|
|
||||||
|
## Initial rollout target
|
||||||
|
|
||||||
|
- ENGL-1010
|
||||||
|
- HIST-2020
|
||||||
|
|
||||||
|
## OpenClaw helper artifact
|
||||||
|
|
||||||
|
Local helper:
|
||||||
|
- `school/intake/submit_to_n8n.js`
|
||||||
|
|
||||||
|
This gives Doris a simple handoff point after the Telegram conversation is complete.
|
||||||
|
|
||||||
|
## Live work still needed
|
||||||
|
|
||||||
|
- import/activate the new n8n intake workflow
|
||||||
|
- import/activate the Paperless enrichment workflow
|
||||||
|
- create shared Postgres database/user/schema
|
||||||
|
- configure class profiles with real Paperless IDs
|
||||||
|
- decide Telegram delivery wording/confirmation behavior
|
||||||
|
- test with real ENGL + HIST documents
|
||||||
@@ -41,6 +41,9 @@
|
|||||||
- [ ] Add sudoers NOPASSWD rule for n8n watchdog (see n8n-workflows/README.md)
|
- [ ] Add sudoers NOPASSWD rule for n8n watchdog (see n8n-workflows/README.md)
|
||||||
- [ ] Import and activate upgraded workflows: 04, 08 (01 done — v1.4 active and tested)
|
- [ ] Import and activate upgraded workflows: 04, 08 (01 done — v1.4 active and tested)
|
||||||
- [ ] Import and activate new workflows: 14, 15 (16 already active)
|
- [ ] Import and activate new workflows: 14, 15 (16 already active)
|
||||||
|
- [ ] Build Telegram chat-driven school-work intake for Paperless using shared Postgres, deterministic `intake_id`, and class/version metadata
|
||||||
|
- [ ] Configure class profiles + Paperless metadata mapping for current courses (ENGL-1010, HIST-2020, future classes)
|
||||||
|
- [ ] Set up DB and Docker backups to Serenity following homelab SOP
|
||||||
|
|
||||||
## Completed
|
## Completed
|
||||||
- [x] Deploy Gotify (Phase 1) — 2026-05-05
|
- [x] Deploy Gotify (Phase 1) — 2026-05-05
|
||||||
|
|||||||
47
docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql
Normal file
47
docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql
Normal file
@@ -0,0 +1,47 @@
|
|||||||
|
-- Shared Postgres schema for Telegram-driven school-work intake.
|
||||||
|
-- Apply in a dedicated `school_intake` database or adapt to an existing shared DB.
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS school_paperless_intake (
|
||||||
|
intake_id TEXT PRIMARY KEY,
|
||||||
|
class_name TEXT NOT NULL,
|
||||||
|
assignment_name TEXT NOT NULL,
|
||||||
|
submission_kind TEXT NOT NULL,
|
||||||
|
semester TEXT,
|
||||||
|
course_code TEXT,
|
||||||
|
paper_kind TEXT,
|
||||||
|
notes TEXT,
|
||||||
|
source TEXT NOT NULL DEFAULT 'telegram',
|
||||||
|
telegram_chat_id TEXT,
|
||||||
|
telegram_message_id TEXT,
|
||||||
|
original_filename TEXT NOT NULL,
|
||||||
|
stored_filename TEXT NOT NULL,
|
||||||
|
mime_type TEXT NOT NULL,
|
||||||
|
file_size_bytes BIGINT NOT NULL,
|
||||||
|
checksum_sha256 TEXT NOT NULL,
|
||||||
|
status TEXT NOT NULL DEFAULT 'received',
|
||||||
|
paperless_task_id TEXT,
|
||||||
|
paperless_document_id BIGINT,
|
||||||
|
paperless_title TEXT,
|
||||||
|
received_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||||
|
uploaded_at TIMESTAMPTZ,
|
||||||
|
enriched_at TIMESTAMPTZ,
|
||||||
|
error_text TEXT,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||||
|
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_school_paperless_intake_status ON school_paperless_intake(status);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_school_paperless_intake_doc_id ON school_paperless_intake(paperless_document_id);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_school_paperless_intake_assignment ON school_paperless_intake(class_name, assignment_name);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_school_paperless_intake_checksum ON school_paperless_intake(checksum_sha256);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS school_paperless_intake_events (
|
||||||
|
id BIGSERIAL PRIMARY KEY,
|
||||||
|
intake_id TEXT NOT NULL REFERENCES school_paperless_intake(intake_id) ON DELETE CASCADE,
|
||||||
|
event_type TEXT NOT NULL,
|
||||||
|
event_payload JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_school_paperless_intake_events_intake_id
|
||||||
|
ON school_paperless_intake_events(intake_id);
|
||||||
237
n8n-workflows/18-school-paperless-intake.json
Normal file
237
n8n-workflows/18-school-paperless-intake.json
Normal file
@@ -0,0 +1,237 @@
|
|||||||
|
{
|
||||||
|
"name": "Telegram School Intake → Postgres → Paperless",
|
||||||
|
"active": false,
|
||||||
|
"isArchived": false,
|
||||||
|
"nodes": [
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"httpMethod": "POST",
|
||||||
|
"path": "school/intake/upload",
|
||||||
|
"responseMode": "lastNode",
|
||||||
|
"responseData": "allEntries",
|
||||||
|
"options": {
|
||||||
|
"rawBody": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"id": "school-intake-webhook",
|
||||||
|
"name": "School Intake Webhook",
|
||||||
|
"type": "n8n-nodes-base.webhook",
|
||||||
|
"typeVersion": 2,
|
||||||
|
"position": [
|
||||||
|
-1180,
|
||||||
|
0
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"jsCode": "\nconst crypto = require('crypto');\nconst req = $input.first().json;\nconst body = req.body || {};\nconst binary = $input.first().binary || {};\nconst upload = binary.data || Object.values(binary)[0];\nif (!upload) throw new Error('Expected multipart file field named \\\"data\\\".');\nconst required = ['class_name', 'assignment_name', 'submission_kind'];\nfor (const key of required) {\n if (!body[key] || !String(body[key]).trim()) {\n throw new Error('Missing required form field: ' + key + '. Required fields: class_name, assignment_name, submission_kind.');\n }\n}\nconst sanitize = (value, fallback = 'item') => String(value || fallback)\n .trim()\n .toLowerCase()\n .replace(/[^a-z0-9]+/g, '-')\n .replace(/^-+|-+$/g, '')\n .slice(0, 80) || fallback;\nconst originalName = upload.fileName || body.filename || 'upload.bin';\nconst extension = (() => {\n const m = String(originalName).match(/(\\.[A-Za-z0-9]{1,10})$/);\n return m ? m[1].toLowerCase() : '';\n})();\nconst fileBytes = Buffer.from(upload.data, 'base64');\nconst checksum = crypto.createHash('sha256').update(fileBytes).digest('hex');\nconst now = new Date();\nconst intakeId = [\n 'school',\n now.toISOString().slice(0,10).replace(/-/g,''),\n sanitize(body.class_name),\n sanitize(body.assignment_name),\n checksum.slice(0, 12)\n].join('-');\nconst storedFilename = intakeId + extension;\nconst title = [body.class_name, body.assignment_name, body.submission_kind].map(v => String(v).trim()).join(' — ');\nreturn [{\n json: {\n intake_id: intakeId,\n class_name: String(body.class_name).trim(),\n assignment_name: String(body.assignment_name).trim(),\n submission_kind: String(body.submission_kind).trim(),\n semester: body.semester ? String(body.semester).trim() : null,\n course_code: body.course_code ? String(body.course_code).trim() : null,\n paper_kind: body.paper_kind ? String(body.paper_kind).trim() : null,\n notes: body.notes ? String(body.notes).trim().slice(0, 4000) : null,\n source: body.source ? String(body.source).trim() : 'telegram',\n telegram_chat_id: body.telegram_chat_id ? String(body.telegram_chat_id).trim() : null,\n telegram_message_id: body.telegram_message_id ? String(body.telegram_message_id).trim() : null,\n original_filename: originalName,\n stored_filename: storedFilename,\n mime_type: upload.mimeType || 'application/octet-stream',\n file_size_bytes: fileBytes.length,\n checksum_sha256: checksum,\n paperless_title: title,\n received_at: now.toISOString()\n },\n binary: {\n data: {\n ...upload,\n fileName: storedFilename,\n mimeType: upload.mimeType || 'application/octet-stream'\n }\n }\n}];"
|
||||||
|
},
|
||||||
|
"id": "normalize-request",
|
||||||
|
"name": "Normalize Intake Request",
|
||||||
|
"type": "n8n-nodes-base.code",
|
||||||
|
"typeVersion": 2,
|
||||||
|
"position": [
|
||||||
|
-920,
|
||||||
|
0
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"operation": "executeQuery",
|
||||||
|
"query": "INSERT INTO school_paperless_intake (\n intake_id,\n class_name,\n assignment_name,\n submission_kind,\n semester,\n course_code,\n paper_kind,\n notes,\n source,\n telegram_chat_id,\n telegram_message_id,\n original_filename,\n stored_filename,\n mime_type,\n file_size_bytes,\n checksum_sha256,\n status,\n received_at\n)\nVALUES (\n '{{ $json.intake_id }}',\n '{{ $json.class_name.replace(/'/g, \"''\") }}',\n '{{ $json.assignment_name.replace(/'/g, \"''\") }}',\n '{{ $json.submission_kind.replace(/'/g, \"''\") }}',\n {{ $json.semester ? \"'\" + $json.semester.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n {{ $json.course_code ? \"'\" + $json.course_code.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n {{ $json.paper_kind ? \"'\" + $json.paper_kind.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n {{ $json.notes ? \"'\" + $json.notes.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n '{{ $json.source.replace(/'/g, \"''\") }}',\n {{ $json.telegram_chat_id ? \"'\" + $json.telegram_chat_id.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n {{ $json.telegram_message_id ? \"'\" + $json.telegram_message_id.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n '{{ $json.original_filename.replace(/'/g, \"''\") }}',\n '{{ $json.stored_filename.replace(/'/g, \"''\") }}',\n '{{ $json.mime_type.replace(/'/g, \"''\") }}',\n {{ $json.file_size_bytes }},\n '{{ $json.checksum_sha256 }}',\n 'received',\n '{{ $json.received_at }}'::timestamptz\n)\nON CONFLICT (intake_id) DO UPDATE SET\n class_name = EXCLUDED.class_name,\n assignment_name = EXCLUDED.assignment_name,\n submission_kind = EXCLUDED.submission_kind,\n semester = EXCLUDED.semester,\n course_code = EXCLUDED.course_code,\n paper_kind = EXCLUDED.paper_kind,\n notes = EXCLUDED.notes,\n source = EXCLUDED.source,\n telegram_chat_id = EXCLUDED.telegram_chat_id,\n telegram_message_id = EXCLUDED.telegram_message_id,\n original_filename = EXCLUDED.original_filename,\n stored_filename = EXCLUDED.stored_filename,\n mime_type = EXCLUDED.mime_type,\n file_size_bytes = EXCLUDED.file_size_bytes,\n checksum_sha256 = EXCLUDED.checksum_sha256,\n status = 'received',\n received_at = EXCLUDED.received_at,\n updated_at = NOW();"
|
||||||
|
},
|
||||||
|
"id": "pg-insert-received",
|
||||||
|
"name": "Postgres Upsert Intake Received",
|
||||||
|
"type": "n8n-nodes-base.postgres",
|
||||||
|
"typeVersion": 2.6,
|
||||||
|
"position": [
|
||||||
|
-660,
|
||||||
|
0
|
||||||
|
],
|
||||||
|
"credentials": {
|
||||||
|
"postgres": {
|
||||||
|
"id": "SETUP_REQUIRED",
|
||||||
|
"name": "Shared Postgres"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"method": "POST",
|
||||||
|
"url": "={{($env.PAPERLESS_BASE_URL || 'https://paperless.paccoco.com').replace(/\\/$/, '') + '/api/documents/post_document/'}}",
|
||||||
|
"sendHeaders": true,
|
||||||
|
"headerParameters": {
|
||||||
|
"parameters": [
|
||||||
|
{
|
||||||
|
"name": "Authorization",
|
||||||
|
"value": "={{'Token ' + $env.PAPERLESS_API_TOKEN}}"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"sendBody": true,
|
||||||
|
"contentType": "multipart-form-data",
|
||||||
|
"bodyParameters": {
|
||||||
|
"parameters": [
|
||||||
|
{
|
||||||
|
"name": "document",
|
||||||
|
"value": "data",
|
||||||
|
"parameterType": "formBinaryData"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "title",
|
||||||
|
"value": "={{$json.paperless_title}}"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"options": {
|
||||||
|
"timeout": 120000
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"id": "paperless-upload",
|
||||||
|
"name": "Upload to Paperless",
|
||||||
|
"type": "n8n-nodes-base.httpRequest",
|
||||||
|
"typeVersion": 4.2,
|
||||||
|
"position": [
|
||||||
|
-400,
|
||||||
|
0
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"jsCode": "\nconst intake = $('Normalize Intake Request').first().json;\nconst response = $json;\nconst taskId = response.task_id || response.id || response.document?.id || null;\nlet paperlessDocumentId = response.document_id || response.document?.id || null;\nif (!paperlessDocumentId && typeof response === 'object' && response?.id && !response?.task_id) {\n paperlessDocumentId = response.id;\n}\nreturn [{ json: {\n ...intake,\n paperless_task_id: taskId ? String(taskId) : null,\n paperless_document_id: paperlessDocumentId ? String(paperlessDocumentId) : null,\n paperless_response: response,\n uploaded_at: new Date().toISOString()\n}}];"
|
||||||
|
},
|
||||||
|
"id": "capture-paperless-response",
|
||||||
|
"name": "Capture Paperless Response",
|
||||||
|
"type": "n8n-nodes-base.code",
|
||||||
|
"typeVersion": 2,
|
||||||
|
"position": [
|
||||||
|
-140,
|
||||||
|
0
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"operation": "executeQuery",
|
||||||
|
"query": "UPDATE school_paperless_intake\nSET\n paperless_task_id = {{ $json.paperless_task_id ? \"'\" + $json.paperless_task_id.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n paperless_document_id = {{ $json.paperless_document_id ? \"'\" + $json.paperless_document_id.replace(/'/g, \"''\") + \"'\" : 'NULL' }},\n paperless_title = '{{ $json.paperless_title.replace(/'/g, \"''\") }}',\n status = 'uploaded',\n uploaded_at = '{{ $json.uploaded_at }}'::timestamptz,\n updated_at = NOW()\nWHERE intake_id = '{{ $json.intake_id }}';"
|
||||||
|
},
|
||||||
|
"id": "pg-mark-uploaded",
|
||||||
|
"name": "Postgres Mark Uploaded",
|
||||||
|
"type": "n8n-nodes-base.postgres",
|
||||||
|
"typeVersion": 2.6,
|
||||||
|
"position": [
|
||||||
|
120,
|
||||||
|
0
|
||||||
|
],
|
||||||
|
"credentials": {
|
||||||
|
"postgres": {
|
||||||
|
"id": "SETUP_REQUIRED",
|
||||||
|
"name": "Shared Postgres"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"jsCode": "\nreturn [{\n json: {\n ok: true,\n intake_id: $json.intake_id,\n status: 'uploaded',\n paperless_task_id: $json.paperless_task_id,\n paperless_document_id: $json.paperless_document_id,\n paperless_title: $json.paperless_title,\n stored_filename: $json.stored_filename,\n received_at: $json.received_at,\n uploaded_at: $json.uploaded_at\n }\n}];"
|
||||||
|
},
|
||||||
|
"id": "success-response",
|
||||||
|
"name": "Return Success Payload",
|
||||||
|
"type": "n8n-nodes-base.code",
|
||||||
|
"typeVersion": 2,
|
||||||
|
"position": [
|
||||||
|
360,
|
||||||
|
0
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"connections": {
|
||||||
|
"School Intake Webhook": {
|
||||||
|
"main": [
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"node": "Normalize Intake Request",
|
||||||
|
"type": "main",
|
||||||
|
"index": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"Normalize Intake Request": {
|
||||||
|
"main": [
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"node": "Postgres Upsert Intake Received",
|
||||||
|
"type": "main",
|
||||||
|
"index": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"Postgres Upsert Intake Received": {
|
||||||
|
"main": [
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"node": "Upload to Paperless",
|
||||||
|
"type": "main",
|
||||||
|
"index": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"Upload to Paperless": {
|
||||||
|
"main": [
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"node": "Capture Paperless Response",
|
||||||
|
"type": "main",
|
||||||
|
"index": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"Capture Paperless Response": {
|
||||||
|
"main": [
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"node": "Postgres Mark Uploaded",
|
||||||
|
"type": "main",
|
||||||
|
"index": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"Postgres Mark Uploaded": {
|
||||||
|
"main": [
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"node": "Return Success Payload",
|
||||||
|
"type": "main",
|
||||||
|
"index": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"settings": {
|
||||||
|
"executionOrder": "v1",
|
||||||
|
"binaryMode": "separate"
|
||||||
|
},
|
||||||
|
"staticData": null,
|
||||||
|
"meta": {
|
||||||
|
"templateCredsSetupCompleted": false
|
||||||
|
},
|
||||||
|
"pinData": {},
|
||||||
|
"tags": [
|
||||||
|
{
|
||||||
|
"name": "school"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "paperless"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "postgres"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "telegram"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
294
n8n-workflows/19-paperless-school-metadata-enrichment.json
Normal file
294
n8n-workflows/19-paperless-school-metadata-enrichment.json
Normal file
@@ -0,0 +1,294 @@
|
|||||||
|
{
|
||||||
|
"name": "Paperless School Intake Metadata Enrichment",
|
||||||
|
"active": false,
|
||||||
|
"isArchived": false,
|
||||||
|
"nodes": [
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"httpMethod": "POST",
|
||||||
|
"path": "school/intake/paperless-enrich",
|
||||||
|
"responseMode": "lastNode",
|
||||||
|
"responseData": "allEntries"
|
||||||
|
},
|
||||||
|
"id": "paperless-school-webhook",
|
||||||
|
"name": "Paperless School Webhook",
|
||||||
|
"type": "n8n-nodes-base.webhook",
|
||||||
|
"typeVersion": 2,
|
||||||
|
"position": [
|
||||||
|
-1160,
|
||||||
|
0
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"jsCode": "\nconst input = $input.first().json || {};\nconst body = input.body || input;\nconst candidates = [\n body.document_id,\n body.id,\n body.doc_id,\n body.pk,\n body.document_id?.id,\n body.document?.id,\n body.document\n].filter(v => v !== undefined && v !== null);\nlet raw = candidates[0];\nif (typeof raw === 'object' && raw !== null) raw = raw.id ?? raw.document_id ?? raw.pk ?? null;\nif (typeof raw === 'string' && /\\{.+\\}/.test(raw)) {\n throw new Error('Received unresolved template string instead of document id.');\n}\nconst documentId = Number(raw);\nif (!Number.isFinite(documentId) || documentId <= 0) {\n throw new Error('Could not resolve Paperless document id from webhook payload.');\n}\nreturn [{ json: { document_id: documentId } }];"
|
||||||
|
},
|
||||||
|
"id": "normalize-paperless-webhook",
|
||||||
|
"name": "Normalize Paperless Webhook",
|
||||||
|
"type": "n8n-nodes-base.code",
|
||||||
|
"typeVersion": 2,
|
||||||
|
"position": [
|
||||||
|
-900,
|
||||||
|
0
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"method": "GET",
|
||||||
|
"url": "={{($env.PAPERLESS_BASE_URL || 'https://paperless.paccoco.com').replace(/\\/$/, '') + '/api/documents/' + $json.document_id + '/'}}",
|
||||||
|
"sendHeaders": true,
|
||||||
|
"headerParameters": {
|
||||||
|
"parameters": [
|
||||||
|
{
|
||||||
|
"name": "Authorization",
|
||||||
|
"value": "={{'Token ' + $env.PAPERLESS_API_TOKEN}}"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Accept",
|
||||||
|
"value": "application/json"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"options": {
|
||||||
|
"timeout": 120000
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"id": "fetch-paperless-document",
|
||||||
|
"name": "Fetch Paperless Document",
|
||||||
|
"type": "n8n-nodes-base.httpRequest",
|
||||||
|
"typeVersion": 4.2,
|
||||||
|
"position": [
|
||||||
|
-640,
|
||||||
|
0
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"jsCode": "\nconst doc = $input.first().json || {};\nconst originalFilename = doc.original_file_name || doc.original_filename || '';\nconst baseName = String(originalFilename).replace(/\\.[^.]+$/, '');\nif (!baseName || !baseName.startsWith('school-')) {\n throw new Error('Document original filename does not contain expected intake id prefix: ' + originalFilename);\n}\nreturn [{ json: {\n intake_id: baseName,\n document_id: doc.id,\n current_title: doc.title || '',\n current_tags: Array.isArray(doc.tags) ? doc.tags : [],\n original_filename: originalFilename\n} }];"
|
||||||
|
},
|
||||||
|
"id": "extract-intake-id",
|
||||||
|
"name": "Extract Intake ID",
|
||||||
|
"type": "n8n-nodes-base.code",
|
||||||
|
"typeVersion": 2,
|
||||||
|
"position": [
|
||||||
|
-380,
|
||||||
|
0
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"operation": "executeQuery",
|
||||||
|
"query": "SELECT intake_id, class_name, assignment_name, submission_kind, semester, course_code, paper_kind, notes, source, telegram_chat_id, telegram_message_id, original_filename, stored_filename, checksum_sha256, status, paperless_task_id, paperless_document_id, paperless_title, received_at, uploaded_at, enriched_at FROM school_paperless_intake WHERE intake_id = '{{ $json.intake_id.replace(/'/g, \"''\") }}' LIMIT 1;"
|
||||||
|
},
|
||||||
|
"id": "pg-fetch-intake-record",
|
||||||
|
"name": "Postgres Fetch Intake Record",
|
||||||
|
"type": "n8n-nodes-base.postgres",
|
||||||
|
"typeVersion": 2.6,
|
||||||
|
"position": [
|
||||||
|
-120,
|
||||||
|
0
|
||||||
|
],
|
||||||
|
"credentials": {
|
||||||
|
"postgres": {
|
||||||
|
"id": "SETUP_REQUIRED",
|
||||||
|
"name": "Shared Postgres"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"jsCode": "\nconst intake = $input.first().json || {};\nif (!intake.intake_id) throw new Error('No intake record found in Postgres.');\nconst doc = $('Fetch Paperless Document').first().json || {};\nconst classTagMap = JSON.parse($env.PAPERLESS_TAG_CLASS_MAP_JSON || '{}');\nconst kindTagMap = JSON.parse($env.PAPERLESS_TAG_SUBMISSION_KIND_MAP_JSON || '{}');\nconst paperKindDocTypeMap = JSON.parse($env.PAPERLESS_DOCUMENT_TYPE_PAPER_KIND_MAP_JSON || '{}');\nconst classCorrespondentMap = JSON.parse($env.PAPERLESS_CORRESPONDENT_CLASS_MAP_JSON || '{}');\nconst currentTags = Array.isArray(doc.tags) ? doc.tags.map(Number).filter(Number.isFinite) : [];\nconst merged = new Set(currentTags);\nfor (const id of [$env.PAPERLESS_TAG_SCHOOL, $env.PAPERLESS_TAG_ASSIGNMENTS, $env.PAPERLESS_TAG_TELEGRAM]) {\n const n = Number(id);\n if (Number.isFinite(n) && n > 0) merged.add(n);\n}\nconst classTag = Number(classTagMap[intake.class_name]);\nif (Number.isFinite(classTag) && classTag > 0) merged.add(classTag);\nconst kindKey = String(intake.submission_kind || '').trim().toLowerCase();\nconst kindTag = Number(kindTagMap[kindKey]);\nif (Number.isFinite(kindTag) && kindTag > 0) merged.add(kindTag);\nconst title = [intake.class_name, intake.assignment_name, intake.submission_kind].filter(Boolean).join(' \u2014 ');\nconst payload = { title, tags: Array.from(merged) };\nconst documentType = Number(paperKindDocTypeMap[intake.paper_kind]);\nif (Number.isFinite(documentType) && documentType > 0) payload.document_type = documentType;\nconst correspondent = Number(classCorrespondentMap[intake.class_name]);\nif (Number.isFinite(correspondent) && correspondent > 0) payload.correspondent = correspondent;\nreturn [{ json: {\n intake_id: intake.intake_id,\n document_id: doc.id,\n payload,\n title,\n final_tag_ids: payload.tags,\n class_name: intake.class_name,\n assignment_name: intake.assignment_name,\n submission_kind: intake.submission_kind\n} }];"
|
||||||
|
},
|
||||||
|
"id": "build-paperless-update",
|
||||||
|
"name": "Build Paperless Update",
|
||||||
|
"type": "n8n-nodes-base.code",
|
||||||
|
"typeVersion": 2,
|
||||||
|
"position": [
|
||||||
|
140,
|
||||||
|
0
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"method": "PATCH",
|
||||||
|
"url": "={{($env.PAPERLESS_BASE_URL || 'https://paperless.paccoco.com').replace(/\\/$/, '') + '/api/documents/' + $json.document_id + '/'}}",
|
||||||
|
"sendHeaders": true,
|
||||||
|
"headerParameters": {
|
||||||
|
"parameters": [
|
||||||
|
{
|
||||||
|
"name": "Authorization",
|
||||||
|
"value": "={{'Token ' + $env.PAPERLESS_API_TOKEN}}"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Content-Type",
|
||||||
|
"value": "application/json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Accept",
|
||||||
|
"value": "application/json"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"sendBody": true,
|
||||||
|
"contentType": "json",
|
||||||
|
"jsonBody": "={{$json.payload}}",
|
||||||
|
"options": {
|
||||||
|
"timeout": 120000
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"id": "update-paperless-document",
|
||||||
|
"name": "Update Paperless Document",
|
||||||
|
"type": "n8n-nodes-base.httpRequest",
|
||||||
|
"typeVersion": 4.2,
|
||||||
|
"position": [
|
||||||
|
400,
|
||||||
|
0
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"operation": "executeQuery",
|
||||||
|
"query": "UPDATE school_paperless_intake SET paperless_document_id = {{ Number($json.id || $('Build Paperless Update').first().json.document_id) }}, paperless_title = '{{ $('Build Paperless Update').first().json.title.replace(/'/g, \"''\") }}', status = 'enriched', enriched_at = NOW(), updated_at = NOW() WHERE intake_id = '{{ $('Build Paperless Update').first().json.intake_id.replace(/'/g, \"''\") }}';"
|
||||||
|
},
|
||||||
|
"id": "pg-mark-enriched",
|
||||||
|
"name": "Postgres Mark Enriched",
|
||||||
|
"type": "n8n-nodes-base.postgres",
|
||||||
|
"typeVersion": 2.6,
|
||||||
|
"position": [
|
||||||
|
660,
|
||||||
|
0
|
||||||
|
],
|
||||||
|
"credentials": {
|
||||||
|
"postgres": {
|
||||||
|
"id": "SETUP_REQUIRED",
|
||||||
|
"name": "Shared Postgres"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"parameters": {
|
||||||
|
"jsCode": "\nconst update = $('Build Paperless Update').first().json;\nreturn [{ json: {\n ok: true,\n intake_id: update.intake_id,\n document_id: update.document_id,\n title: update.title,\n final_tag_ids: update.final_tag_ids,\n status: 'enriched'\n} }];"
|
||||||
|
},
|
||||||
|
"id": "return-enrichment-success",
|
||||||
|
"name": "Return Enrichment Success",
|
||||||
|
"type": "n8n-nodes-base.code",
|
||||||
|
"typeVersion": 2,
|
||||||
|
"position": [
|
||||||
|
920,
|
||||||
|
0
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"connections": {
|
||||||
|
"Paperless School Webhook": {
|
||||||
|
"main": [
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"node": "Normalize Paperless Webhook",
|
||||||
|
"type": "main",
|
||||||
|
"index": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"Normalize Paperless Webhook": {
|
||||||
|
"main": [
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"node": "Fetch Paperless Document",
|
||||||
|
"type": "main",
|
||||||
|
"index": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"Fetch Paperless Document": {
|
||||||
|
"main": [
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"node": "Extract Intake ID",
|
||||||
|
"type": "main",
|
||||||
|
"index": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"Extract Intake ID": {
|
||||||
|
"main": [
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"node": "Postgres Fetch Intake Record",
|
||||||
|
"type": "main",
|
||||||
|
"index": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"Postgres Fetch Intake Record": {
|
||||||
|
"main": [
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"node": "Build Paperless Update",
|
||||||
|
"type": "main",
|
||||||
|
"index": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"Build Paperless Update": {
|
||||||
|
"main": [
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"node": "Update Paperless Document",
|
||||||
|
"type": "main",
|
||||||
|
"index": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"Update Paperless Document": {
|
||||||
|
"main": [
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"node": "Postgres Mark Enriched",
|
||||||
|
"type": "main",
|
||||||
|
"index": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"Postgres Mark Enriched": {
|
||||||
|
"main": [
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"node": "Return Enrichment Success",
|
||||||
|
"type": "main",
|
||||||
|
"index": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"settings": {
|
||||||
|
"executionOrder": "v1"
|
||||||
|
},
|
||||||
|
"staticData": null,
|
||||||
|
"meta": {
|
||||||
|
"templateCredsSetupCompleted": false
|
||||||
|
},
|
||||||
|
"pinData": {},
|
||||||
|
"tags": [
|
||||||
|
{
|
||||||
|
"name": "school"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "paperless"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "postgres"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "metadata"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -15,6 +15,8 @@
|
|||||||
| 14 | Paperless Inbox Reminder | Scheduled (Monday 9 AM) | Queries Paperless for untagged documents; sends Gotify reminder with oldest items listed, or a ✅ clear if inbox is empty |
|
| 14 | Paperless Inbox Reminder | Scheduled (Monday 9 AM) | Queries Paperless for untagged documents; sends Gotify reminder with oldest items listed, or a ✅ clear if inbox is empty |
|
||||||
| 15 | n8n Self-Health Monitor | Scheduled (every 15 min) | Checks n8n execution history for failures in the past hour; deduplicates via Postgres; pushes Gotify alert with workflow name and error summary |
|
| 15 | n8n Self-Health Monitor | Scheduled (every 15 min) | Checks n8n execution history for failures in the past hour; deduplicates via Postgres; pushes Gotify alert with workflow name and error summary |
|
||||||
| 16 | NFS Mount Watchdog | Scheduled (every 5 min) | Probes NFS mount points on PD; if missing: runs mount script, re-probes, restarts affected containers (Plex, Audiobookshelf, Immich); notifies via Gotify on heal or escalation |
|
| 16 | NFS Mount Watchdog | Scheduled (every 5 min) | Probes NFS mount points on PD; if missing: runs mount script, re-probes, restarts affected containers (Plex, Audiobookshelf, Immich); notifies via Gotify on heal or escalation |
|
||||||
|
| 18 | Telegram School Intake → Postgres → Paperless | `POST /webhook/school/intake/upload` | Accepts multipart schoolwork uploads plus class/assignment metadata, writes an intake record to shared Postgres, then uploads into Paperless with deterministic `intake_id` in the filename/title |
|
||||||
|
| 19 | Paperless School Intake Metadata Enrichment | `POST /webhook/school/intake/paperless-enrich` | Handles Paperless post-consumption webhooks, looks up the intake record by deterministic filename, then applies deterministic title/tags and optional document-type/correspondent mapping |
|
||||||
|
|
||||||
## How to Import
|
## How to Import
|
||||||
|
|
||||||
@@ -67,9 +69,16 @@ Two steps required — both the `.env` file AND the `docker-compose.yaml` `envir
|
|||||||
|
|
||||||
| Variable | Used by | How to get |
|
| Variable | Used by | How to get |
|
||||||
|----------|---------|------------|
|
|----------|---------|------------|
|
||||||
| `PAPERLESS_API_TOKEN` | 04, 08 | Paperless → Settings → API token |
|
| `PAPERLESS_API_TOKEN` | 04, 08, 18, 19 | Paperless → Settings → API token |
|
||||||
| `PAPERLESS_TAG_TRANSCRIPTION` | 04 | Tag ID from Paperless API |
|
| `PAPERLESS_TAG_TRANSCRIPTION` | 04 | Tag ID from Paperless API |
|
||||||
| `PAPERLESS_TAG_LECTURE_NOTES` | 08 | Tag ID from Paperless API |
|
| `PAPERLESS_TAG_LECTURE_NOTES` | 08 | Tag ID from Paperless API |
|
||||||
|
| `PAPERLESS_TAG_SCHOOL` | 19 | Generic school tag ID from Paperless API |
|
||||||
|
| `PAPERLESS_TAG_ASSIGNMENTS` | 19 | Generic assignments tag ID from Paperless API |
|
||||||
|
| `PAPERLESS_TAG_TELEGRAM` | 19 | Tag ID for Telegram-submitted items |
|
||||||
|
| `PAPERLESS_TAG_CLASS_MAP_JSON` | 19 | JSON object mapping class name → Paperless tag ID |
|
||||||
|
| `PAPERLESS_TAG_SUBMISSION_KIND_MAP_JSON` | 19 | JSON object mapping submission kind → Paperless tag ID |
|
||||||
|
| `PAPERLESS_DOCUMENT_TYPE_PAPER_KIND_MAP_JSON` | 19 | JSON object mapping paper kind → Paperless document type ID |
|
||||||
|
| `PAPERLESS_CORRESPONDENT_CLASS_MAP_JSON` | 19 | JSON object mapping class name → Paperless correspondent ID |
|
||||||
| `N8N_API_KEY` | 15 | n8n → Settings → API → Create API Key |
|
| `N8N_API_KEY` | 15 | n8n → Settings → API → Create API Key |
|
||||||
| `LITELLM_API_KEY` | 01, 04, 08 | LiteLLM virtual key (already in .env) |
|
| `LITELLM_API_KEY` | 01, 04, 08 | LiteLLM virtual key (already in .env) |
|
||||||
|
|
||||||
@@ -81,6 +90,13 @@ Two steps required — both the `.env` file AND the `docker-compose.yaml` `envir
|
|||||||
PAPERLESS_API_TOKEN: ${PAPERLESS_API_TOKEN}
|
PAPERLESS_API_TOKEN: ${PAPERLESS_API_TOKEN}
|
||||||
PAPERLESS_TAG_TRANSCRIPTION: ${PAPERLESS_TAG_TRANSCRIPTION}
|
PAPERLESS_TAG_TRANSCRIPTION: ${PAPERLESS_TAG_TRANSCRIPTION}
|
||||||
PAPERLESS_TAG_LECTURE_NOTES: ${PAPERLESS_TAG_LECTURE_NOTES}
|
PAPERLESS_TAG_LECTURE_NOTES: ${PAPERLESS_TAG_LECTURE_NOTES}
|
||||||
|
PAPERLESS_TAG_SCHOOL: ${PAPERLESS_TAG_SCHOOL}
|
||||||
|
PAPERLESS_TAG_ASSIGNMENTS: ${PAPERLESS_TAG_ASSIGNMENTS}
|
||||||
|
PAPERLESS_TAG_TELEGRAM: ${PAPERLESS_TAG_TELEGRAM}
|
||||||
|
PAPERLESS_TAG_CLASS_MAP_JSON: ${PAPERLESS_TAG_CLASS_MAP_JSON}
|
||||||
|
PAPERLESS_TAG_SUBMISSION_KIND_MAP_JSON: ${PAPERLESS_TAG_SUBMISSION_KIND_MAP_JSON}
|
||||||
|
PAPERLESS_DOCUMENT_TYPE_PAPER_KIND_MAP_JSON: ${PAPERLESS_DOCUMENT_TYPE_PAPER_KIND_MAP_JSON}
|
||||||
|
PAPERLESS_CORRESPONDENT_CLASS_MAP_JSON: ${PAPERLESS_CORRESPONDENT_CLASS_MAP_JSON}
|
||||||
N8N_API_KEY: ${N8N_API_KEY}
|
N8N_API_KEY: ${N8N_API_KEY}
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -136,10 +152,11 @@ In Paperless-NGX, set up post-consumption webhooks to POST to:
|
|||||||
```
|
```
|
||||||
https://n8n.paccoco.com/webhook/paperless/new-document
|
https://n8n.paccoco.com/webhook/paperless/new-document
|
||||||
https://n8n.paccoco.com/webhook/paperless/rag-ingest
|
https://n8n.paccoco.com/webhook/paperless/rag-ingest
|
||||||
|
https://n8n.paccoco.com/webhook/school/intake/paperless-enrich
|
||||||
```
|
```
|
||||||
with body: `{"document_id": <id>}`
|
with body: `{"document_id": <id>}`
|
||||||
|
|
||||||
You can trigger both from the same Paperless event — workflow 03 classifies/tags the document, workflow 05 ingests it into RAG.
|
You can trigger all three from the same Paperless event. Workflow 03 handles AI summary/tagging, workflow 05 handles RAG ingest, and workflow 19 re-applies deterministic school metadata for intake-managed documents.
|
||||||
|
|
||||||
### Grafana Webhook (already configured)
|
### Grafana Webhook (already configured)
|
||||||
|
|
||||||
@@ -156,6 +173,19 @@ Works with Gitea, Forgejo, GitHub, and GitLab webhook payloads.
|
|||||||
|
|
||||||
## Usage Examples
|
## Usage Examples
|
||||||
|
|
||||||
|
### Upload schoolwork from Telegram intake:
|
||||||
|
```bash
|
||||||
|
curl -X POST https://n8n.paccoco.com/webhook/school/intake/upload \
|
||||||
|
-F "document=@essay-draft.pdf" \
|
||||||
|
-F "class_name=English 101" \
|
||||||
|
-F "assignment_name=Essay Draft 1" \
|
||||||
|
-F "submission_kind=homework" \
|
||||||
|
-F "semester=Fall 2026" \
|
||||||
|
-F "course_code=ENG101" \
|
||||||
|
-F "paper_kind=essay"
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
### Ingest a document into RAG:
|
### Ingest a document into RAG:
|
||||||
```bash
|
```bash
|
||||||
curl -X POST https://n8n.paccoco.com/webhook/rag/ingest \
|
curl -X POST https://n8n.paccoco.com/webhook/rag/ingest \
|
||||||
@@ -259,6 +289,20 @@ curl -X POST https://n8n.paccoco.com/webhook/git/push \
|
|||||||
}'
|
}'
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Workflow 18 — Telegram School Intake → Postgres → Paperless
|
||||||
|
|
||||||
|
Import `18-school-paperless-intake.json` for a chat-driven schoolwork intake flow. Supporting notes live at `docs/operations/SCHOOL_PAPERLESS_INTAKE.md` and the tracked schema lives at `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql`.
|
||||||
|
|
||||||
|
Highlights:
|
||||||
|
- multipart upload endpoint: `POST /webhook/school/intake/upload`
|
||||||
|
- required fields: `class_name`, `assignment_name`, `submission_kind`
|
||||||
|
- optional fields: `semester`, `course_code`, `paper_kind`, `notes`, `telegram_chat_id`, `telegram_message_id`
|
||||||
|
- writes intake metadata to shared Postgres before upload
|
||||||
|
- sends deterministic `intake_id` in the Paperless filename and title correlation data
|
||||||
|
|
||||||
|
Testing fixture:
|
||||||
|
- `fixtures/school-paperless-intake-webhook.curl.example.txt`
|
||||||
|
|
||||||
## Customizing RSS Feeds (Workflow 06)
|
## Customizing RSS Feeds (Workflow 06)
|
||||||
|
|
||||||
Edit the "Define RSS Feeds" Code node to add/remove feeds. Default feeds:
|
Edit the "Define RSS Feeds" Code node to add/remove feeds. Default feeds:
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
# n8n Workflow Testing Status
|
# n8n Workflow Testing Status
|
||||||
_Last updated: 2026-05-10_
|
_Last updated: 2026-05-13_
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -105,3 +105,38 @@ _Last updated: 2026-05-10_
|
|||||||
- [ ] `PAPERLESS_TRIAGE_APPLY_SAFE=false` during initial validation
|
- [ ] `PAPERLESS_TRIAGE_APPLY_SAFE=false` during initial validation
|
||||||
|
|
||||||
**Test approach:** Upload a harmless test document to Paperless, POST `{"document_id": <id>}` to the webhook, and confirm `doris-dashboard/data/paperless_review.json` contains a sanitized review item. Use a bill/due-date fixture to verify forced human review.
|
**Test approach:** Upload a harmless test document to Paperless, POST `{"document_id": <id>}` to the webhook, and confirm `doris-dashboard/data/paperless_review.json` contains a sanitized review item. Use a bill/due-date fixture to verify forced human review.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 18 — Telegram School Intake → Postgres → Paperless v1.0
|
||||||
|
**What it does:** Accepts multipart school-work upload plus metadata (`class_name`, `assignment_name`, `submission_kind`, optional semester/course_code/paper_kind), records it in shared Postgres, and uploads it into Paperless with a deterministic `intake_id`-based filename.
|
||||||
|
|
||||||
|
**Local artifact checks completed:**
|
||||||
|
- [x] Workflow JSON validates with `python3 -m json.tool`
|
||||||
|
- [x] Companion helper script validates with `node --check school/intake/submit_to_n8n.js`
|
||||||
|
- [x] Class profile example JSON validates with `python3 -m json.tool`
|
||||||
|
|
||||||
|
**Pre-test checklist:**
|
||||||
|
- [ ] Shared Postgres table `school_paperless_intake` created from `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql`
|
||||||
|
- [ ] `PAPERLESS_API_TOKEN` exposed to n8n through env vars
|
||||||
|
- [ ] Postgres credential `Shared Postgres` created in n8n and wired into workflow 18
|
||||||
|
- [ ] Webhook path `https://n8n.paccoco.com/webhook/school/intake/upload` reachable from Doris/OpenClaw helper
|
||||||
|
|
||||||
|
**Test approach:** POST a small DOCX/PDF through the intake webhook, confirm a row appears in `school_paperless_intake`, and verify the document lands in Paperless with filename prefix `school-...`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 19 — Paperless School Intake Metadata Enrichment v1.0
|
||||||
|
**What it does:** Receives Paperless post-consumption webhook for intake-managed documents, resolves the deterministic `intake_id` from the stored filename, loads the intake record from Postgres, then reapplies deterministic title/tags and optional document type/correspondent mapping.
|
||||||
|
|
||||||
|
**Local artifact checks completed:**
|
||||||
|
- [x] Workflow JSON validates with `python3 -m json.tool`
|
||||||
|
|
||||||
|
**Pre-test checklist:**
|
||||||
|
- [ ] Paperless webhook configured to POST `https://n8n.paccoco.com/webhook/school/intake/paperless-enrich`
|
||||||
|
- [ ] Workflow 18 already working so intake-managed filenames exist in Paperless
|
||||||
|
- [ ] `PAPERLESS_API_TOKEN`, `PAPERLESS_TAG_SCHOOL`, `PAPERLESS_TAG_ASSIGNMENTS`, and `PAPERLESS_TAG_TELEGRAM` exposed to n8n
|
||||||
|
- [ ] Optional mapping env vars supplied when ready: `PAPERLESS_TAG_CLASS_MAP_JSON`, `PAPERLESS_TAG_SUBMISSION_KIND_MAP_JSON`, `PAPERLESS_DOCUMENT_TYPE_PAPER_KIND_MAP_JSON`, `PAPERLESS_CORRESPONDENT_CLASS_MAP_JSON`
|
||||||
|
- [ ] Postgres credential `Shared Postgres` created in n8n and wired into workflow 19
|
||||||
|
|
||||||
|
**Test approach:** After a workflow-18 upload is consumed by Paperless, POST `{"document_id": <id>}` to `/webhook/school/intake/paperless-enrich` and verify title/tags are updated and the DB row status becomes `enriched`.
|
||||||
|
|||||||
@@ -0,0 +1,12 @@
|
|||||||
|
curl -X POST https://n8n.paccoco.com/webhook/school/intake/upload \
|
||||||
|
-F "document=@essay-draft.pdf" \
|
||||||
|
-F "class_name=English 101" \
|
||||||
|
-F "assignment_name=Essay Draft 1" \
|
||||||
|
-F "submission_kind=homework" \
|
||||||
|
-F "semester=Fall 2026" \
|
||||||
|
-F "course_code=ENG101" \
|
||||||
|
-F "paper_kind=essay" \
|
||||||
|
-F "notes=Submitted from Telegram chat intake" \
|
||||||
|
-F "source=telegram" \
|
||||||
|
-F "telegram_chat_id=123456789" \
|
||||||
|
-F "telegram_message_id=987654321"
|
||||||
Reference in New Issue
Block a user