truenas-stacks/n8n-workflows/README.md

# n8n AI Workflows for Homelab

## Workflows Included

| # | Workflow | Trigger | What It Does |
|---|---------|---------|--------------|
| 01 | Grafana Alert → AI → Gotify | `POST /webhook/grafana-alert` | Receives Grafana alerts, AI-summarizes, deduplicates within 30 min, suppresses low-priority alerts overnight — pushes to Gotify |
| 02 | RAG Pipeline | `POST /webhook/rag/ingest` + `POST /webhook/rag/query` | Ingest docs into Qdrant, query them with AI-powered answers |
| 03 | Paperless AI Processing | `POST /webhook/paperless/new-document` | Auto-classifies & summarizes new Paperless docs, applies AI-suggested tags back to Paperless, notifies via Gotify |
| 04 | Whisper Transcription | `POST /webhook/transcribe` + `POST /webhook/transcribe/summarize` | Transcribe audio → text, optionally with AI summary; notifies via Gotify when done and saves transcript to Paperless |
| 05 | Paperless → RAG | `POST /webhook/paperless/rag-ingest` | Auto-ingests Paperless documents into Qdrant for RAG search; notifies via Gotify |
| 06 | RSS Digest | Scheduled (every 6h) + `POST /webhook/scrape/summarize` | Fetches RSS feeds, deduplicates via Postgres, AI-summarizes per-feed, pushes to Gotify |
| 07 | Git Commit Summarizer | `POST /webhook/git/push` | Receives git push webhooks, AI-summarizes changes, notifies via Gotify |
| 08 | Class Recording → RAG | `POST /webhook/class/upload` | Transcribes uploaded .wav, extracts key notes, ingests into RAG by class, saves notes to Paperless, notifies via Gotify |
| 14 | Paperless Inbox Reminder | Scheduled (Monday 9 AM) | Queries Paperless for untagged documents; sends Gotify reminder with oldest items listed, or a ✅ clear if inbox is empty |
| 15 | n8n Self-Health Monitor | Scheduled (every 15 min) | Checks n8n execution history for failures in the past hour; deduplicates via Postgres; pushes Gotify alert with workflow name and error summary |
| 16 | NFS Mount Watchdog | Scheduled (every 5 min) | Probes NFS mount points on PD; if missing: runs mount script, re-probes, restarts affected containers (Plex, Audiobookshelf, Immich); notifies via Gotify on heal or escalation |
| 17 | Paperless Intake Triage → Doris Review Queue | `POST /webhook/paperless/intake-triage` | Fetches a Paperless document by `document_id`, runs conservative AI triage, writes a Doris Dashboard review queue JSON file, and keeps safe auto-apply disabled unless explicitly enabled |
| 18 | Telegram School Intake → Postgres → Paperless v1.2 | `POST /webhook/school/intake/upload` | Accepts multipart schoolwork uploads plus class/assignment metadata, uploads into Paperless with deterministic `intake_id` in the filename/title, and returns the resulting task/document context |
| 19 | Paperless School Intake Metadata Enrichment v1.1 | `POST /webhook/school/intake/paperless-enrich` | Handles Paperless post-consumption webhooks, resolves metadata from deterministic filename/title, then applies deterministic title/tags and optional correspondent mapping |

## How to Import

1. Open n8n at https://n8n.paccoco.com
2. Go to **Workflows** → click the **⋮** menu → **Import from File**
3. Select each `.json` file from this folder
4. Activate the workflow (toggle in top-right)

## Gotify Channel Map

Each workflow uses a dedicated Gotify app credential. Create these apps in Gotify and add matching credentials in n8n → Credentials:

| Gotify App | Used by |
|------------|---------|
| Gotify account | 01 (Grafana Alerts) |
| Paperless | 03 |
| Whisper | 04 |
| Paperless RAG | 05 |
| RSS Feed | 06 |
| Git Commits | 07 |
| Class Recordings | 08 |
| Reminders | 14 |
| n8n Health | 15 |
| Uptime / Infra | 16 |

## One-Time Setup

### Postgres Dedup Tables (required for workflows 01, 15)

```sql
-- Run via: sudo docker exec -it shared-postgres psql -U postgres -d n8n
CREATE TABLE IF NOT EXISTS grafana_alert_dedup (
  fingerprint TEXT PRIMARY KEY,
  last_seen   TIMESTAMPTZ DEFAULT NOW(),
  alert_name  TEXT,
  host        TEXT
);
CREATE TABLE IF NOT EXISTS n8n_health_dedup (
  fingerprint    TEXT PRIMARY KEY,
  last_seen      TIMESTAMPTZ DEFAULT NOW(),
  workflow_names TEXT
);
```

### n8n Environment Variables

Two steps required — both the `.env` file AND the `docker-compose.yaml` `environment:` section must be updated, then n8n restarted with `sudo docker compose --env-file .env down n8n && sudo docker compose --env-file .env up -d n8n`.

**automation/.env** — add these:

| Variable | Used by | How to get |
|----------|---------|------------|
| `PAPERLESS_API_TOKEN` | 04, 08, 18, 19 | Paperless → Settings → API token |
| `PAPERLESS_TAG_TRANSCRIPTION` | 04 | Tag ID from Paperless API |
| `PAPERLESS_TAG_LECTURE_NOTES` | 08 | Tag ID from Paperless API |
| `PAPERLESS_TAG_SCHOOL` | 19 | Generic school tag ID from Paperless API |
| `PAPERLESS_TAG_ASSIGNMENTS` | 19 | Generic assignments tag ID from Paperless API |
| `PAPERLESS_TAG_TELEGRAM` | 19 | Tag ID for Telegram-submitted items |
| `PAPERLESS_TAG_CLASS_MAP_JSON` | 19 | JSON object mapping class name → Paperless tag ID |
| `PAPERLESS_TAG_SUBMISSION_KIND_MAP_JSON` | 19 | JSON object mapping submission kind → Paperless tag ID |
| `PAPERLESS_DOCUMENT_TYPE_PAPER_KIND_MAP_JSON` | 19 | JSON object mapping paper kind → Paperless document type ID |
| `PAPERLESS_CORRESPONDENT_CLASS_MAP_JSON` | 19 | JSON object mapping class name → Paperless correspondent ID |
| `PAPERLESS_TRIAGE_MODEL` | 17 | Optional LiteLLM model override; defaults to `medium` |
| `PAPERLESS_TRIAGE_REVIEW_QUEUE_PATH` | 17 | Writable queue path inside the n8n container, e.g. `/data/paperless-triage/paperless_review.json` |
| `PAPERLESS_TRIAGE_REVIEW_MAX_ITEMS` | 17 | Optional queue size cap; defaults to `200` |
| `PAPERLESS_TRIAGE_APPLY_SAFE` | 17 | Keep `false` unless you intentionally want title-only safe auto-apply enabled |
| `N8N_API_KEY` | 15 | n8n → Settings → API → Create API Key |
| `LITELLM_API_KEY` | 01, 04, 08 | LiteLLM virtual key (already in .env) |

**automation/docker-compose.yaml** — add to n8n `environment:` section:

```yaml
      N8N_BLOCK_ENV_ACCESS_IN_NODE: "false"
      LITELLM_API_KEY: ${LITELLM_API_KEY}
      PAPERLESS_API_TOKEN: ${PAPERLESS_API_TOKEN}
      PAPERLESS_TAG_TRANSCRIPTION: ${PAPERLESS_TAG_TRANSCRIPTION}
      PAPERLESS_TAG_LECTURE_NOTES: ${PAPERLESS_TAG_LECTURE_NOTES}
      PAPERLESS_TAG_SCHOOL: ${PAPERLESS_TAG_SCHOOL}
      PAPERLESS_TAG_ASSIGNMENTS: ${PAPERLESS_TAG_ASSIGNMENTS}
      PAPERLESS_TAG_TELEGRAM: ${PAPERLESS_TAG_TELEGRAM}
      PAPERLESS_TAG_CLASS_MAP_JSON: ${PAPERLESS_TAG_CLASS_MAP_JSON}
      PAPERLESS_TAG_SUBMISSION_KIND_MAP_JSON: ${PAPERLESS_TAG_SUBMISSION_KIND_MAP_JSON}
      PAPERLESS_DOCUMENT_TYPE_PAPER_KIND_MAP_JSON: ${PAPERLESS_DOCUMENT_TYPE_PAPER_KIND_MAP_JSON}
      PAPERLESS_CORRESPONDENT_CLASS_MAP_JSON: ${PAPERLESS_CORRESPONDENT_CLASS_MAP_JSON}
      PAPERLESS_TRIAGE_MODEL: ${PAPERLESS_TRIAGE_MODEL}
      PAPERLESS_TRIAGE_REVIEW_QUEUE_PATH: ${PAPERLESS_TRIAGE_REVIEW_QUEUE_PATH}
      PAPERLESS_TRIAGE_REVIEW_MAX_ITEMS: ${PAPERLESS_TRIAGE_REVIEW_MAX_ITEMS}
      PAPERLESS_TRIAGE_APPLY_SAFE: ${PAPERLESS_TRIAGE_APPLY_SAFE}
      N8N_API_KEY: ${N8N_API_KEY}
```

> **Note:** `N8N_BLOCK_ENV_ACCESS_IN_NODE: "false"` is required for workflows to read `$env.*` variables. Without it, env var access is silently denied even if the variable exists in the container.

### SSH Credential (required for workflow 16)

Workflow 16 SSHes into PD to probe mounts and restart containers. Set up key-based auth:

```bash
# Run on PD — generate a dedicated key for n8n
ssh-keygen -t ed25519 -f ~/.ssh/n8n_watchdog -N ""
cat ~/.ssh/n8n_watchdog.pub >> ~/.ssh/authorized_keys
cat ~/.ssh/n8n_watchdog   # paste this into n8n → Credentials → SSH (Private Key)
```

In n8n: Credentials → New → SSH (Private Key) → host `10.5.1.6`, user `truenas_admin`, paste private key. Name it `PD SSH`.

Workflow 16 also runs `sudo docker restart` and the mount script over SSH. Add a NOPASSWD sudoers rule so it doesn't hang:

```bash
sudo visudo -f /etc/sudoers.d/n8n-watchdog
# Add:
# truenas_admin ALL=(ALL) NOPASSWD: /usr/bin/docker, /bin/bash /mnt/tank/docker/scripts/mount-unraid-nfs.sh
```

### Qdrant Collection (required for workflows 02, 05, 08)

Create the `knowledge_base` collection before using the RAG workflows:

```bash
curl -X PUT http://10.5.1.6:6333/collections/knowledge_base \
  -H 'Content-Type: application/json' \
  -d '{
    "vectors": {
      "size": 768,
      "distance": "Cosine"
    }
  }'
```

(768 dimensions matches `nomic-embed-text`. If you use a different embedding model, adjust the size.)

### Pull nomic-embed-text into Ollama (required for workflows 02, 05, 08)

```bash
curl http://10.5.1.6:11434/api/pull -d '{"name": "nomic-embed-text"}'
```

### Paperless Webhook (required for workflows 03, 05, 17, and 19)

In Paperless-NGX, set up post-consumption webhooks to POST to:
```
https://n8n.paccoco.com/webhook/paperless/new-document
https://n8n.paccoco.com/webhook/paperless/rag-ingest
https://n8n.paccoco.com/webhook/paperless/intake-triage
https://n8n.paccoco.com/webhook/school/intake/paperless-enrich
```
with body: `{"document_id": <id>}`

You can trigger all four from the same Paperless event. Workflow 03 handles general AI summary/tagging, workflow 05 handles RAG ingest, workflow 17 feeds the Doris Dashboard review queue, and workflow 19 re-applies deterministic school metadata for intake-managed documents.

### Grafana Webhook (already configured)

Your Grafana contact point is already set to `https://n8n.paccoco.com/webhook/grafana-alert` — workflow 01 will work immediately once activated.

### Git Webhook (required for workflow 07)

In your Gitea/Forgejo/GitHub repo settings, add a webhook:
- URL: `https://n8n.paccoco.com/webhook/git/push`
- Content type: `application/json`
- Events: Push only

Works with Gitea, Forgejo, GitHub, and GitLab webhook payloads.

## Usage Examples

### Upload schoolwork from Telegram intake:
```bash
curl -X POST https://n8n.paccoco.com/webhook/school/intake/upload \
  -F "document=@essay-draft.pdf" \
  -F "class_name=English 101" \
  -F "assignment_name=Essay Draft 1" \
  -F "submission_kind=homework" \
  -F "semester=Fall 2026" \
  -F "course_code=ENG101" \
  -F "paper_kind=essay"
```


### Ingest a document into RAG:
```bash
curl -X POST https://n8n.paccoco.com/webhook/rag/ingest \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "Your document content here...",
    "source": "my-notes",
    "metadata": {"topic": "homelab"}
  }'
```

### Query your knowledge base:
```bash
curl -X POST https://n8n.paccoco.com/webhook/rag/query \
  -H 'Content-Type: application/json' \
  -d '{"query": "How do I configure Qdrant?"}'
```

### Transcribe audio:
```bash
curl -X POST https://n8n.paccoco.com/webhook/transcribe \
  -F "data=@recording.mp3"
```

### Transcribe + AI summarize:
```bash
curl -X POST https://n8n.paccoco.com/webhook/transcribe/summarize \
  -F "data=@meeting.mp3"
```

### Add a class recording (just drop a file):
```
/mnt/tank/class-recordings/
├── CS101-Java/
│   ├── lecture-01-intro.wav
│   ├── lecture-02-variables.wav
│   └── lecture-03-loops.wav
├── MATH201/
│   └── lecture-01-derivatives.wav
└── ENG102/
    └── essay-workshop.wav
```

Workflow 08 uses a webhook trigger (`POST /webhook/class/upload`), not a folder watcher. Upload the file via curl and include the class name and lecture title as form fields (see curl example below).

> **Note:** The folder structure above shows suggested organization on disk. The workflow does not watch the folder — you must POST to the webhook.

### Query RAG for class-specific content:
```bash
curl -X POST https://n8n.paccoco.com/webhook/rag/query \
  -H 'Content-Type: application/json' \
  -d '{"query": "What did my CS101 lecture say about inheritance in Java?"}'
```

The RAG system stores class name metadata with each chunk, so when you mention a class name in your query it will prioritize results from that class.

### Class Recordings Setup (Workflow 08)

Create class subfolders on PD (already done for CS101-Java):
```bash
sudo mkdir -p /mnt/tank/class-recordings/CS101-Java
sudo mkdir -p /mnt/tank/class-recordings/MATH201
```

Add to n8n's **automation/.env**:
```
PAPERLESS_TAG_LECTURE_NOTES=11
```

Add to n8n's **automation/docker-compose.yaml** under the n8n `environment:` section:
```yaml
      PAPERLESS_TAG_LECTURE_NOTES: ${PAPERLESS_TAG_LECTURE_NOTES}
```

Then restart n8n and import/activate `08-class-recording-rag-v1.1.json`.

To upload a class recording:
```bash
curl -X POST https://n8n.paccoco.com/webhook/class/upload \
  -F "data=@lecture-01-intro.wav" \
  -F "class_name=CS101-Java" \
  -F "lecture_title=lecture-01-intro"
```

### Trigger RSS digest manually:
```bash
curl -X POST https://n8n.paccoco.com/webhook/scrape/summarize
```

### Test git webhook:
```bash
curl -X POST https://n8n.paccoco.com/webhook/git/push \
  -H 'Content-Type: application/json' \
  -d '{
    "ref": "refs/heads/main",
    "pusher": {"name": "john"},
    "repository": {"full_name": "john/my-project"},
    "commits": [
      {"id": "abc1234", "message": "Fix null pointer in UserService", "author": {"name": "john"}, "added": [], "modified": ["src/UserService.java"], "removed": []}
    ]
  }'
```

## Workflow 18 — Telegram School Intake → Postgres → Paperless v1.2

Import `18-school-paperless-intake-v1.2.json` for a chat-driven schoolwork intake flow. Supporting notes live at `docs/operations/SCHOOL_PAPERLESS_INTAKE.md` and the tracked schema lives at `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql`.

Highlights:
- multipart upload endpoint: `POST /webhook/school/intake/upload`
- required fields: `class_name`, `assignment_name`, `submission_kind`
- optional fields: `semester`, `course_code`, `paper_kind`, `notes`, `telegram_chat_id`, `telegram_message_id`
- sends deterministic `intake_id` in the Paperless filename and title correlation data
- returns upload success context immediately from the webhook
- v1.1 removed the `require('crypto')` dependency so the workflow can run without adding `crypto` to `NODE_FUNCTION_ALLOW_BUILTIN`
- v1.2 fixes the multipart Paperless upload node to use n8n's correct `inputDataFieldName` pattern for binary form uploads

Testing fixture:
- `fixtures/school-paperless-intake-webhook.curl.example.txt`

## Workflow Versioning Convention

For n8n workflow artifacts in this repo:
- include the version in the **filename** (example: `18-school-paperless-intake-v1.1.json`)
- include the version in the workflow's internal **`name`** field
- bump the version whenever behavior, schema expectations, webhook payload handling, or env requirements change
- update `n8n-workflows/README.md` and `n8n-workflows/TESTING-STATUS.md` when introducing a new workflow version

Do not overwrite a workflow artifact with silent behavior changes and no version bump. That way lies confusion and sweaty guesswork. Ha ha.

## Customizing RSS Feeds (Workflow 06)

Edit the "Define RSS Feeds" Code node to add/remove feeds. Default feeds:
- Self-Hosted Show
- r/selfhosted
- r/homelab  
- Hacker News

Add any RSS feed URL you want monitored.

---

## Workflow 17 — Paperless Intake Triage → Doris Review Queue

Import `17-paperless-intake-triage.json` for a review-first Paperless intake workflow. It is intentionally conservative: it writes a sanitized local review queue for Doris Dashboard and does **not** delete documents. Automatic Paperless updates are disabled by default.

### Trigger

Configure a Paperless-NGX post-consumption webhook to POST to:

```text
https://n8n.paccoco.com/webhook/paperless/intake-triage
```

Body:

```json
{"document_id": 12345}
```

Fixture: `fixtures/paperless-intake-webhook.example.json`.

### Environment variables

Add these to `automation/.env` and expose them in the n8n `environment:` block:

| Variable | Required | Default | Purpose |
|----------|----------|---------|---------|
| `PAPERLESS_BASE_URL` | yes | `https://paperless.paccoco.com` | Paperless API/UI base URL |
| `PAPERLESS_API_TOKEN` | yes | none | Paperless API token; used as `Token ...` |
| `LITELLM_BASE_URL` | yes | `http://10.5.1.6:4000` | LiteLLM OpenAI-compatible endpoint |
| `LITELLM_API_KEY` | yes | none | LiteLLM key |
| `PAPERLESS_TRIAGE_MODEL` | no | `gpt-4o-mini` | Model for strict JSON triage |
| `PAPERLESS_TRIAGE_REVIEW_QUEUE_PATH` | yes | `/data/paperless_review.json` | Sanitized JSON queue path mounted for Doris Dashboard |
| `PAPERLESS_TRIAGE_REVIEW_MAX_ITEMS` | no | `200` | Max queue entries retained |
| `PAPERLESS_TRIAGE_APPLY_SAFE` | no | `false` | Enables low-risk title-only update branch when `true` |
| `GOTIFY_URL` | no | none | Enables urgent-doc notification only when token also set |
| `GOTIFY_TOKEN` | no | none | Gotify app token for urgent docs |
| `NODE_FUNCTION_ALLOW_BUILTIN` | yes for queue file | none | Must include `fs,path` for the Code node that writes JSON |
| `N8N_BLOCK_ENV_ACCESS_IN_NODE` | yes | existing note | Must be `false` so Code nodes can read `$env` |

Example environment additions:

```yaml
      N8N_BLOCK_ENV_ACCESS_IN_NODE: "false"
      NODE_FUNCTION_ALLOW_BUILTIN: fs,path
      PAPERLESS_BASE_URL: ${PAPERLESS_BASE_URL}
      PAPERLESS_API_TOKEN: ${PAPERLESS_API_TOKEN}
      LITELLM_BASE_URL: ${LITELLM_BASE_URL}
      LITELLM_API_KEY: ${LITELLM_API_KEY}
      PAPERLESS_TRIAGE_MODEL: ${PAPERLESS_TRIAGE_MODEL}
      PAPERLESS_TRIAGE_REVIEW_QUEUE_PATH: /workspace/doris-dashboard/data/paperless_review.json
      PAPERLESS_TRIAGE_APPLY_SAFE: "false"
```

Mount the Doris dashboard data directory read/write into the n8n container if it is not already mounted. The workflow writes only sanitized display data: document IDs, titles, filenames, archive URL, current metadata names/IDs, triage summary, review reason, urgency, confidence, and safety flags. It must not contain tokens.

### Safety gates

The LLM must return strict JSON:

```json
{
  "summary": "plain English summary",
  "suggested_title": "clean title",
  "document_type": "bill|receipt|school|medical|tax|insurance|legal|manual|letter|other",
  "suggested_correspondent": "string or null",
  "suggested_tags": ["..."],
  "due_date": "YYYY-MM-DD or null",
  "needs_review": true,
  "urgency": "none|low|medium|high",
  "reason": "why review/why not",
  "confidence": "high|medium|low"
}
```

The workflow forcibly marks `needs_review=true` for:

- legal, medical, school, tax, or insurance documents
- any due date
- bill/payment/invoice language
- low confidence
- urgency medium or high

Only high-confidence, low-risk docs with no due date and urgency `none|low` become eligible for automatic update, and even then the branch is disabled unless `PAPERLESS_TRIAGE_APPLY_SAFE=true`. The safe update branch currently updates **title only**. It does not delete, archive, or apply high-risk metadata.

### Test payload and expected result

- Webhook payload: `fixtures/paperless-intake-webhook.example.json`
- Example Paperless API document: `fixtures/paperless-intake-document.example.json`
- Expected triage JSON: `fixtures/paperless-intake-expected-triage.example.json`

The fixture is a utility bill with a due date, so the safety gate must force human review.

### Import/deploy steps

1. Import `17-paperless-intake-triage.json` in n8n.
2. Add env vars and required volume mount.
3. Restart n8n using the normal automation compose procedure.
4. Keep `PAPERLESS_TRIAGE_APPLY_SAFE=false` for initial testing.
5. Use a known test document and POST `{"document_id": <id>}` to `/webhook/paperless/intake-triage`.
6. Confirm `doris-dashboard/data/paperless_review.json` updates and contains no secrets.
7. Activate the Paperless webhook only after the test queue file is correct.

### Rollback

Deactivate the workflow and remove the Paperless webhook target. If needed, move `doris-dashboard/data/paperless_review.json` aside; it is display-only derived data. No Paperless deletion path exists in this workflow.