392 lines
8.4 KiB
Markdown
392 lines
8.4 KiB
Markdown
# Doris Schoolhouse — Technical Spec
|
|
|
|
## Goal
|
|
|
|
Build a desktop-first local web app on NOMAD for school intake and tracking.
|
|
|
|
Primary jobs:
|
|
|
|
- upload assignment turn-ins
|
|
- upload class recordings
|
|
- suggest metadata from D2L
|
|
- confirm metadata before Paperless submission
|
|
- transcribe and summarize recordings
|
|
- track assignment lifecycle through grading
|
|
- append grades and professor comments into Paperless notes
|
|
|
|
## Deployment shape
|
|
|
|
- **Source path:** `home/doris-schoolhouse`
|
|
- **Live NOMAD runtime target:** `/opt/doris-schoolhouse`
|
|
- **Access:** local + Pangolin-authenticated
|
|
- **Networks:** `pangolin`, `ix-databases_shared-databases`
|
|
- **DB:** shared Postgres
|
|
- **Archive/document system:** Paperless-NGX
|
|
- **Workflow helpers:** existing n8n workflows for school intake and class recording processing when appropriate
|
|
|
|
## Source of truth
|
|
|
|
| Domain | Source of truth |
|
|
|---|---|
|
|
| Course roster | D2L sync cache |
|
|
| Assignment roster / due dates | D2L sync cache with manual override support |
|
|
| Submission workflow state | Doris Schoolhouse DB |
|
|
| Uploaded files / transcripts / summaries | Paperless |
|
|
| Grades / professor comments | D2L on refresh, cached in Doris Schoolhouse DB |
|
|
|
|
The app DB owns workflow state and linkage. Paperless owns archived artifacts.
|
|
|
|
## Existing reusable assets
|
|
|
|
### D2L
|
|
|
|
Existing workspace assets already prove the login/scrape path:
|
|
|
|
- `~/.openclaw/workspace/school/d2l_scraper_final.js`
|
|
- `~/.openclaw/workspace/school/d2l_school_watcher.js`
|
|
- `~/.openclaw/workspace/school/d2l_current_snapshot.json`
|
|
- `~/.openclaw/workspace/school/d2l_grades.json`
|
|
|
|
Current confirmed D2L coverage:
|
|
|
|
- courses
|
|
- grades
|
|
- grade items
|
|
- some material/date counts
|
|
- calendar/ICS-derived schedule
|
|
|
|
Gap to close for Schoolhouse:
|
|
|
|
- normalize D2L data into app DB tables
|
|
- stronger assignment/date extraction
|
|
- deterministic matching between D2L assignment rows and Paperless-backed submissions
|
|
|
|
### Paperless / n8n
|
|
|
|
Existing repo assets:
|
|
|
|
- `n8n-workflows/08-class-recording-rag-v1.2.json`
|
|
- `n8n-workflows/18-school-paperless-intake-v1.2.json`
|
|
- `n8n-workflows/19-paperless-school-metadata-enrichment-v1.1.json`
|
|
- `docs/operations/SCHOOL_PAPERLESS_INTAKE.md`
|
|
- `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql`
|
|
|
|
Current confirmed behavior from docs/testing notes:
|
|
|
|
- school uploads can already be pushed to Paperless with deterministic filename/title correlation
|
|
- class recording pipeline already has a webhook path and downstream Paperless/RAG behavior
|
|
- workflow 19 already does deterministic post-consumption enrichment
|
|
|
|
## App architecture
|
|
|
|
Use a server-rendered Python app for the first build:
|
|
|
|
- **Backend:** FastAPI
|
|
- **Templates:** Jinja2
|
|
- **Frontend:** vanilla JS + server-rendered forms/pages
|
|
- **DB access:** SQLAlchemy / psycopg
|
|
- **Background work:** app jobs can hand off long-running audio processing to existing n8n recording/transcription pipeline
|
|
|
|
Why this split:
|
|
|
|
- easier large-upload handling
|
|
- easy form-heavy UI
|
|
- less frontend build nonsense
|
|
- plays nicely with existing Python-heavy repo bits
|
|
|
|
## Core pages
|
|
|
|
### 1. Dashboard `/`
|
|
|
|
Shows:
|
|
|
|
- upcoming assignments
|
|
- unsubmitted assignments
|
|
- recent uploads pending confirmation
|
|
- newly graded work
|
|
- recording processing status
|
|
- sync health
|
|
|
|
### 2. Assignment Upload `/assignments/upload`
|
|
|
|
Form fields:
|
|
|
|
- class
|
|
- assignment
|
|
- submitted date
|
|
- version
|
|
- notes
|
|
- file upload / drag-drop / paste
|
|
|
|
Behavior:
|
|
|
|
1. choose file
|
|
2. app suggests class + assignment from D2L cache
|
|
3. user confirms/edits metadata
|
|
4. upload goes to Paperless intake path
|
|
5. app stores linkage row
|
|
6. assignment state becomes `submitted`
|
|
|
|
### 3. Recording Upload `/recordings/upload`
|
|
|
|
Form fields:
|
|
|
|
- class
|
|
- session date
|
|
- optional related assignment
|
|
- notes
|
|
- `.wav` upload
|
|
|
|
Behavior:
|
|
|
|
1. accept large WAV
|
|
2. persist upload job row
|
|
3. convert WAV → MP3
|
|
4. delete WAV only after MP3 success
|
|
5. hand MP3 to transcription/summarization flow
|
|
6. send transcript + summary to Paperless
|
|
7. store resulting Paperless document links
|
|
|
|
### 4. Classes `/classes` and `/classes/{id}`
|
|
|
|
Shows:
|
|
|
|
- assignments
|
|
- due dates
|
|
- grades
|
|
- comments
|
|
- linked submissions
|
|
- linked recordings
|
|
|
|
### 5. D2L Sync `/admin/d2l`
|
|
|
|
Shows:
|
|
|
|
- last sync time
|
|
- sync status
|
|
- changed courses/assignments/grades
|
|
- unmatched rows requiring manual mapping
|
|
|
|
## API surface
|
|
|
|
### Health
|
|
- `GET /api/health`
|
|
|
|
### D2L
|
|
- `POST /api/d2l/sync`
|
|
- `GET /api/d2l/courses`
|
|
- `GET /api/d2l/assignments?course_id=`
|
|
- `GET /api/d2l/grades?course_id=`
|
|
|
|
### Assignments
|
|
- `GET /api/assignments`
|
|
- `POST /api/assignments/intake`
|
|
- `POST /api/assignments/{id}/confirm-paperless`
|
|
- `POST /api/assignments/{id}/status`
|
|
|
|
### Recordings
|
|
- `GET /api/recordings`
|
|
- `POST /api/recordings/intake`
|
|
- `POST /api/recordings/{id}/reprocess`
|
|
|
|
### Paperless linkage
|
|
- `POST /api/paperless/webhook/intake-enriched`
|
|
- `POST /api/paperless/webhook/recording-complete`
|
|
- `POST /api/paperless/assignments/{id}/append-grade-note`
|
|
|
|
## Data model
|
|
|
|
### `schoolhouse_course`
|
|
|
|
- id
|
|
- d2l_course_id
|
|
- name
|
|
- code
|
|
- semester
|
|
- instructor_name
|
|
- active
|
|
- first_seen_at
|
|
- last_synced_at
|
|
|
|
### `schoolhouse_assignment`
|
|
|
|
- id
|
|
- course_id
|
|
- d2l_assignment_key
|
|
- title
|
|
- category
|
|
- due_at
|
|
- status (`assigned|in_progress|submitted|graded|late|missing`)
|
|
- grade_text
|
|
- grade_numeric
|
|
- professor_comments
|
|
- source (`d2l|manual|template`)
|
|
- created_at
|
|
- updated_at
|
|
|
|
### `schoolhouse_submission`
|
|
|
|
- id
|
|
- assignment_id
|
|
- version_label
|
|
- original_filename
|
|
- mime_type
|
|
- checksum_sha256
|
|
- submitted_at
|
|
- suggested_metadata_json
|
|
- confirmed_metadata_json
|
|
- paperless_document_id
|
|
- paperless_title
|
|
- paperless_note_appended_at
|
|
- status (`draft|uploaded|confirmed|failed`)
|
|
|
|
### `schoolhouse_recording`
|
|
|
|
- id
|
|
- course_id
|
|
- assignment_id nullable
|
|
- session_date
|
|
- original_filename
|
|
- mp3_filename
|
|
- duration_seconds
|
|
- transcription_status (`queued|processing|ready|failed`)
|
|
- summary_status (`queued|processing|ready|failed`)
|
|
- transcript_paperless_document_id
|
|
- summary_paperless_document_id
|
|
- recording_paperless_document_id nullable
|
|
- notes
|
|
|
|
### `schoolhouse_d2l_sync_run`
|
|
|
|
- id
|
|
- started_at
|
|
- completed_at
|
|
- status
|
|
- raw_snapshot_path
|
|
- changes_summary_json
|
|
- error_text
|
|
|
|
### `schoolhouse_assignment_mapping`
|
|
|
|
- id
|
|
- course_id
|
|
- d2l_grade_item_name
|
|
- normalized_key
|
|
- assignment_id
|
|
- confidence
|
|
- mapping_source (`auto|manual`)
|
|
|
|
This table is the bastard that keeps D2L grade items tied to internal assignments cleanly.
|
|
|
|
## Matching rules
|
|
|
|
### Assignment matching priority
|
|
|
|
1. exact D2L assignment key when available
|
|
2. normalized title within course
|
|
3. filename heuristics
|
|
4. latest due date proximity
|
|
5. manual confirmation
|
|
|
|
### Grade/comment update policy
|
|
|
|
- D2L refresh updates cached grade/comment fields for mapped assignments
|
|
- if a submission has a Paperless document ID and grade/comment changed, append note once per distinct change hash
|
|
- never overwrite user notes in Paperless; append a dated Schoolhouse block
|
|
|
|
## Recording pipeline contract
|
|
|
|
### Input limits
|
|
|
|
- accept WAV up to at least 1 GB
|
|
- support up to 90 minutes
|
|
- reject unsupported formats with clear message
|
|
|
|
### Processing steps
|
|
|
|
1. store upload metadata row
|
|
2. convert WAV → MP3
|
|
3. delete WAV only after MP3 success
|
|
4. submit MP3 to recording workflow
|
|
5. receive transcript/summary callback
|
|
6. archive transcript + summary into Paperless
|
|
|
|
## Secrets/config
|
|
|
|
Required env:
|
|
|
|
- `SCHOOLHOUSE_HOST`
|
|
- `SCHOOLHOUSE_PORT`
|
|
- `DATABASE_URL`
|
|
- `PAPERLESS_BASE_URL`
|
|
- `PAPERLESS_API_TOKEN`
|
|
- `N8N_BASE_URL`
|
|
- `SCHOOLHOUSE_N8N_ASSIGNMENT_WEBHOOK`
|
|
- `SCHOOLHOUSE_N8N_RECORDING_WEBHOOK`
|
|
- `D2L_SCRAPER_WORKDIR`
|
|
- `D2L_SCRAPER_COMMAND`
|
|
- `UPLOAD_TMP_DIR`
|
|
- `MAX_UPLOAD_MB`
|
|
- `SESSION_SECRET`
|
|
|
|
Do not commit live values. Use `.env.example` only.
|
|
|
|
## Repo / deployment plan
|
|
|
|
Repo source path:
|
|
|
|
- `home/doris-schoolhouse`
|
|
|
|
Expected live NOMAD runtime path later:
|
|
|
|
- `/opt/doris-schoolhouse`
|
|
|
|
Expected live data path later:
|
|
|
|
- `/opt/doris-schoolhouse/data`
|
|
|
|
Expected compose validation before deploy:
|
|
|
|
```bash
|
|
cd /opt/doris-schoolhouse
|
|
docker compose --env-file .env config
|
|
```
|
|
|
|
## Build phases
|
|
|
|
### Phase 1
|
|
|
|
- app scaffold
|
|
- schema
|
|
- D2L import command/service
|
|
- assignment upload UI
|
|
- Paperless confirmation flow
|
|
|
|
### Phase 2
|
|
|
|
- recording upload flow
|
|
- MP3 conversion
|
|
- recording job tracking
|
|
- transcript/summary callbacks
|
|
|
|
### Phase 3
|
|
|
|
- grade sync
|
|
- Paperless note append logic
|
|
- dashboard views
|
|
|
|
### Phase 4
|
|
|
|
- manual mappings UI
|
|
- recurring assignment templates
|
|
- mobile cleanup
|
|
|
|
## Verification checklist
|
|
|
|
- Python modules compile
|
|
- schema file present and readable
|
|
- compose file parses
|
|
- route map loads without import failure
|
|
- README explains runtime/deploy expectations
|
|
- no secrets committed
|