truenas-stacks/docs/planning/doris-schoolhouse-technical-spec.md

Doris Schoolhouse — Technical Spec

Goal

Build a desktop-first local web app on NOMAD for school intake and tracking.

Primary jobs:

• upload assignment turn-ins
• upload class recordings
• suggest metadata from D2L
• confirm metadata before Paperless submission
• transcribe and summarize recordings
• track assignment lifecycle through grading
• append grades and professor comments into Paperless notes

Deployment shape

• Source path: home/doris-schoolhouse
• Live NOMAD runtime target: /opt/doris-schoolhouse
• Access: local + Pangolin-authenticated
• Networks: pangolin, ix-databases_shared-databases
• DB: shared Postgres
• Archive/document system: Paperless-NGX
• Workflow helpers: existing n8n workflows for school intake and class recording processing when appropriate

Source of truth

Domain	Source of truth
Course roster	D2L sync cache
Assignment roster / due dates	D2L sync cache with manual override support
Submission workflow state	Doris Schoolhouse DB
Uploaded files / transcripts / summaries	Paperless
Grades / professor comments	D2L on refresh, cached in Doris Schoolhouse DB

The app DB owns workflow state and linkage. Paperless owns archived artifacts.

Existing reusable assets

D2L

Existing workspace assets already prove the login/scrape path:

• ~/.openclaw/workspace/school/d2l_scraper_final.js
• ~/.openclaw/workspace/school/d2l_school_watcher.js
• ~/.openclaw/workspace/school/d2l_current_snapshot.json
• ~/.openclaw/workspace/school/d2l_grades.json

Current confirmed D2L coverage:

• courses
• grades
• grade items
• some material/date counts
• calendar/ICS-derived schedule

Gap to close for Schoolhouse:

• normalize D2L data into app DB tables
• stronger assignment/date extraction
• deterministic matching between D2L assignment rows and Paperless-backed submissions

Paperless / n8n

Existing repo assets:

• n8n-workflows/08-class-recording-rag-v1.2.json
• n8n-workflows/18-school-paperless-intake-v1.2.json
• n8n-workflows/19-paperless-school-metadata-enrichment-v1.1.json
• docs/operations/SCHOOL_PAPERLESS_INTAKE.md
• docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql

Current confirmed behavior from docs/testing notes:

• school uploads can already be pushed to Paperless with deterministic filename/title correlation
• class recording pipeline already has a webhook path and downstream Paperless/RAG behavior
• workflow 19 already does deterministic post-consumption enrichment

App architecture

Use a server-rendered Python app for the first build:

• Backend: FastAPI
• Templates: Jinja2
• Frontend: vanilla JS + server-rendered forms/pages
• DB access: SQLAlchemy / psycopg
• Background work: app jobs can hand off long-running audio processing to existing n8n recording/transcription pipeline

Why this split:

• easier large-upload handling
• easy form-heavy UI
• less frontend build nonsense
• plays nicely with existing Python-heavy repo bits

Core pages

1. Dashboard /

Shows:

• upcoming assignments
• unsubmitted assignments
• recent uploads pending confirmation
• newly graded work
• recording processing status
• sync health

2. Assignment Upload /assignments/upload

Form fields:

• class
• assignment
• submitted date
• version
• notes
• file upload / drag-drop / paste

Behavior:

1. choose file
2. app suggests class + assignment from D2L cache
3. user confirms/edits metadata
4. upload goes to Paperless intake path
5. app stores linkage row
6. assignment state becomes submitted

3. Recording Upload /recordings/upload

Form fields:

• class
• session date
• optional related assignment
• notes
• .wav upload

Behavior:

1. accept large WAV
2. persist upload job row
3. convert WAV → MP3
4. delete WAV only after MP3 success
5. hand MP3 to transcription/summarization flow
6. send transcript + summary to Paperless
7. store resulting Paperless document links

4. Classes /classes and /classes/{id}

Shows:

• assignments
• due dates
• grades
• comments
• linked submissions
• linked recordings

5. D2L Sync /admin/d2l

Shows:

• last sync time
• sync status
• changed courses/assignments/grades
• unmatched rows requiring manual mapping

API surface

Health

• GET /api/health

D2L

• POST /api/d2l/sync
• GET /api/d2l/courses
• GET /api/d2l/assignments?course_id=
• GET /api/d2l/grades?course_id=

Assignments

• GET /api/assignments
• POST /api/assignments/intake
• POST /api/assignments/{id}/confirm-paperless
• POST /api/assignments/{id}/status

Recordings

• GET /api/recordings
• POST /api/recordings/intake
• POST /api/recordings/{id}/reprocess

Paperless linkage

• POST /api/paperless/webhook/intake-enriched
• POST /api/paperless/webhook/recording-complete
• POST /api/paperless/assignments/{id}/append-grade-note

Data model

schoolhouse_course

• id
• d2l_course_id
• name
• code
• semester
• instructor_name
• active
• first_seen_at
• last_synced_at

schoolhouse_assignment

• id
• course_id
• d2l_assignment_key
• title
• category
• due_at
• status (assigned|in_progress|submitted|graded|late|missing)
• grade_text
• grade_numeric
• professor_comments
• source (d2l|manual|template)
• created_at
• updated_at

schoolhouse_submission

• id
• assignment_id
• version_label
• original_filename
• mime_type
• checksum_sha256
• submitted_at
• suggested_metadata_json
• confirmed_metadata_json
• paperless_document_id
• paperless_title
• paperless_note_appended_at
• status (draft|uploaded|confirmed|failed)

schoolhouse_recording

• id
• course_id
• assignment_id nullable
• session_date
• original_filename
• mp3_filename
• duration_seconds
• transcription_status (queued|processing|ready|failed)
• summary_status (queued|processing|ready|failed)
• transcript_paperless_document_id
• summary_paperless_document_id
• recording_paperless_document_id nullable
• notes

schoolhouse_d2l_sync_run

• id
• started_at
• completed_at
• status
• raw_snapshot_path
• changes_summary_json
• error_text

schoolhouse_assignment_mapping

• id
• course_id
• d2l_grade_item_name
• normalized_key
• assignment_id
• confidence
• mapping_source (auto|manual)

This table is the bastard that keeps D2L grade items tied to internal assignments cleanly.

Matching rules

Assignment matching priority

1. exact D2L assignment key when available
2. normalized title within course
3. filename heuristics
4. latest due date proximity
5. manual confirmation

Grade/comment update policy

• D2L refresh updates cached grade/comment fields for mapped assignments
• if a submission has a Paperless document ID and grade/comment changed, append note once per distinct change hash
• never overwrite user notes in Paperless; append a dated Schoolhouse block

Recording pipeline contract

Input limits

• accept WAV up to at least 1 GB
• support up to 90 minutes
• reject unsupported formats with clear message

Processing steps

1. store upload metadata row
2. convert WAV → MP3
3. delete WAV only after MP3 success
4. submit MP3 to recording workflow
5. receive transcript/summary callback
6. archive transcript + summary into Paperless

Secrets/config

Required env:

• SCHOOLHOUSE_HOST
• SCHOOLHOUSE_PORT
• DATABASE_URL
• PAPERLESS_BASE_URL
• PAPERLESS_API_TOKEN
• N8N_BASE_URL
• SCHOOLHOUSE_N8N_ASSIGNMENT_WEBHOOK
• SCHOOLHOUSE_N8N_RECORDING_WEBHOOK
• D2L_SCRAPER_WORKDIR
• D2L_SCRAPER_COMMAND
• UPLOAD_TMP_DIR
• MAX_UPLOAD_MB
• SESSION_SECRET

Do not commit live values. Use .env.example only.

Repo / deployment plan

Repo source path:

• home/doris-schoolhouse

Expected live NOMAD runtime path later:

• /opt/doris-schoolhouse

Expected live data path later:

• /opt/doris-schoolhouse/data

Expected compose validation before deploy:

cd /opt/doris-schoolhouse
docker compose --env-file .env config

Build phases

Phase 1

• app scaffold
• schema
• D2L import command/service
• assignment upload UI
• Paperless confirmation flow

Phase 2

• recording upload flow
• MP3 conversion
• recording job tracking
• transcript/summary callbacks

Phase 3

• grade sync
• Paperless note append logic
• dashboard views

Phase 4

• manual mappings UI
• recurring assignment templates
• mobile cleanup

Verification checklist

• Python modules compile
• schema file present and readable
• compose file parses
• route map loads without import failure
• README explains runtime/deploy expectations
• no secrets committed