fizzlepoof/truenas-stacks
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Doris Schoolhouse and clean pending homelab changes

Browse Source
This commit is contained in:
Fizzlepoof
2026-05-15 21:58:42 +00:00
parent 3aad8d037b
commit 383cd41ae7
60 changed files with 3145 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
docs/planning/TODO.md
View File

@@ -5,9 +5,10 @@
- [x] Regenerate N.O.M.A.D. Newt secret (was exposed in chat) — completed 2026-05-15
## Infrastructure
- [ ] Deploy Pi-hole 3-node HA cluster (PD + N.O.M.A.D. + RPi4) — see docs/planning/DEPLOY_PIHOLE.md
- PD primary stack scaffolded repo-side under `pihole/` on 2026-05-15
- RPi4 IP still TBD — fill in before deploying
- [x] Deploy Pi-hole HA on PD + N.O.M.A.D. with VIP `10.5.1.53` — completed 2026-05-15; see docs/planning/DEPLOY_PIHOLE.md
- PD primary live at `/mnt/docker-ssd/docker/compose/pihole`
- NOMAD backup live at `/opt/pihole-nomad`
- RPi4 remains optional future BACKUP2 work if you want a third node later
- [x] Deploy Kima-Hub — completed 2026-05-06; docs/planning/DEPLOY_KIMA_HUB.md now serves as deployment/repair reference
- [ ] Set up off-site backup for vital data (appdata, databases, config repo, photos)
- [ ] Remove dead Unraid-Cloudflared-Tunnel container from Serenity
@@ -40,10 +41,10 @@
- Pelican/Wings-managed game containers stay and are documented as permanent game-server infrastructure on NOMAD.
## n8n Workflows
- [ ] Create Gotify apps and n8n credentials: Reminders, n8n Health, Uptime / Infra
- [ ] Add remaining env vars to n8n docker-compose + .env: `PAPERLESS_API_TOKEN`, `PAPERLESS_TAG_TRANSCRIPTION`, `PAPERLESS_TAG_LECTURE_NOTES`, `N8N_API_KEY` (see n8n-workflows/README.md for docker-compose.yaml changes required)
- [ ] Set up `PD SSH` credential in n8n for workflow 16 (see n8n-workflows/README.md)
- [ ] Add sudoers NOPASSWD rule for n8n watchdog (see n8n-workflows/README.md)
- [x] Create Gotify apps and n8n credentials: Reminders, n8n Health, Uptime / Infra — completed 2026-05-15
- [x] Add remaining env vars to n8n docker-compose + .env: `PAPERLESS_API_TOKEN`, `PAPERLESS_TAG_TRANSCRIPTION`, `PAPERLESS_TAG_LECTURE_NOTES`, `N8N_API_KEY` — completed 2026-05-15
- [x] Set up `PD SSH` credential in n8n for workflow 16 — completed 2026-05-15
- [x] Add sudoers NOPASSWD rule for n8n watchdog — completed 2026-05-15
- [x] Import and activate upgraded workflows: 04, 08 (01 done — v1.4 active and tested) — completed 2026-05-15
- [x] Import and activate new workflows: 14, 15 (16 already active) — completed 2026-05-15
- [x] Configure class profiles + Paperless metadata mapping for current courses (ENGL-1010, HIST-2020, future classes) — completed 2026-05-15

391
docs/planning/doris-schoolhouse-technical-spec.md Normal file
View File

@@ -0,0 +1,391 @@
# Doris Schoolhouse — Technical Spec
## Goal
Build a desktop-first local web app on NOMAD for school intake and tracking.
Primary jobs:
- upload assignment turn-ins
- upload class recordings
- suggest metadata from D2L
- confirm metadata before Paperless submission
- transcribe and summarize recordings
- track assignment lifecycle through grading
- append grades and professor comments into Paperless notes
## Deployment shape
- **Source path:** `home/doris-schoolhouse`
- **Live NOMAD runtime target:** `/opt/doris-schoolhouse`
- **Access:** local + Pangolin-authenticated
- **Networks:** `pangolin`, `ix-databases_shared-databases`
- **DB:** shared Postgres
- **Archive/document system:** Paperless-NGX
- **Workflow helpers:** existing n8n workflows for school intake and class recording processing when appropriate
## Source of truth
| Domain | Source of truth |
|---|---|
| Course roster | D2L sync cache |
| Assignment roster / due dates | D2L sync cache with manual override support |
| Submission workflow state | Doris Schoolhouse DB |
| Uploaded files / transcripts / summaries | Paperless |
| Grades / professor comments | D2L on refresh, cached in Doris Schoolhouse DB |
The app DB owns workflow state and linkage. Paperless owns archived artifacts.
## Existing reusable assets
### D2L
Existing workspace assets already prove the login/scrape path:
- `~/.openclaw/workspace/school/d2l_scraper_final.js`
- `~/.openclaw/workspace/school/d2l_school_watcher.js`
- `~/.openclaw/workspace/school/d2l_current_snapshot.json`
- `~/.openclaw/workspace/school/d2l_grades.json`
Current confirmed D2L coverage:
- courses
- grades
- grade items
- some material/date counts
- calendar/ICS-derived schedule
Gap to close for Schoolhouse:
- normalize D2L data into app DB tables
- stronger assignment/date extraction
- deterministic matching between D2L assignment rows and Paperless-backed submissions
### Paperless / n8n
Existing repo assets:
- `n8n-workflows/08-class-recording-rag-v1.2.json`
- `n8n-workflows/18-school-paperless-intake-v1.2.json`
- `n8n-workflows/19-paperless-school-metadata-enrichment-v1.1.json`
- `docs/operations/SCHOOL_PAPERLESS_INTAKE.md`
- `docs/reference/SCHOOL_INTAKE_POSTGRES_SCHEMA.sql`
Current confirmed behavior from docs/testing notes:
- school uploads can already be pushed to Paperless with deterministic filename/title correlation
- class recording pipeline already has a webhook path and downstream Paperless/RAG behavior
- workflow 19 already does deterministic post-consumption enrichment
## App architecture
Use a server-rendered Python app for the first build:
- **Backend:** FastAPI
- **Templates:** Jinja2
- **Frontend:** vanilla JS + server-rendered forms/pages
- **DB access:** SQLAlchemy / psycopg
- **Background work:** app jobs can hand off long-running audio processing to existing n8n recording/transcription pipeline
Why this split:
- easier large-upload handling
- easy form-heavy UI
- less frontend build nonsense
- plays nicely with existing Python-heavy repo bits
## Core pages
### 1. Dashboard `/`
Shows:
- upcoming assignments
- unsubmitted assignments
- recent uploads pending confirmation
- newly graded work
- recording processing status
- sync health
### 2. Assignment Upload `/assignments/upload`
Form fields:
- class
- assignment
- submitted date
- version
- notes
- file upload / drag-drop / paste
Behavior:
1. choose file
2. app suggests class + assignment from D2L cache
3. user confirms/edits metadata
4. upload goes to Paperless intake path
5. app stores linkage row
6. assignment state becomes `submitted`
### 3. Recording Upload `/recordings/upload`
Form fields:
- class
- session date
- optional related assignment
- notes
- `.wav` upload
Behavior:
1. accept large WAV
2. persist upload job row
3. convert WAV → MP3
4. delete WAV only after MP3 success
5. hand MP3 to transcription/summarization flow
6. send transcript + summary to Paperless
7. store resulting Paperless document links
### 4. Classes `/classes` and `/classes/{id}`
Shows:
- assignments
- due dates
- grades
- comments
- linked submissions
- linked recordings
### 5. D2L Sync `/admin/d2l`
Shows:
- last sync time
- sync status
- changed courses/assignments/grades
- unmatched rows requiring manual mapping
## API surface
### Health
- `GET /api/health`
### D2L
- `POST /api/d2l/sync`
- `GET /api/d2l/courses`
- `GET /api/d2l/assignments?course_id=`
- `GET /api/d2l/grades?course_id=`
### Assignments
- `GET /api/assignments`
- `POST /api/assignments/intake`
- `POST /api/assignments/{id}/confirm-paperless`
- `POST /api/assignments/{id}/status`
### Recordings
- `GET /api/recordings`
- `POST /api/recordings/intake`
- `POST /api/recordings/{id}/reprocess`
### Paperless linkage
- `POST /api/paperless/webhook/intake-enriched`
- `POST /api/paperless/webhook/recording-complete`
- `POST /api/paperless/assignments/{id}/append-grade-note`
## Data model
### `schoolhouse_course`
- id
- d2l_course_id
- name
- code
- semester
- instructor_name
- active
- first_seen_at
- last_synced_at
### `schoolhouse_assignment`
- id
- course_id
- d2l_assignment_key
- title
- category
- due_at
- status (`assigned|in_progress|submitted|graded|late|missing`)
- grade_text
- grade_numeric
- professor_comments
- source (`d2l|manual|template`)
- created_at
- updated_at
### `schoolhouse_submission`
- id
- assignment_id
- version_label
- original_filename
- mime_type
- checksum_sha256
- submitted_at
- suggested_metadata_json
- confirmed_metadata_json
- paperless_document_id
- paperless_title
- paperless_note_appended_at
- status (`draft|uploaded|confirmed|failed`)
### `schoolhouse_recording`
- id
- course_id
- assignment_id nullable
- session_date
- original_filename
- mp3_filename
- duration_seconds
- transcription_status (`queued|processing|ready|failed`)
- summary_status (`queued|processing|ready|failed`)
- transcript_paperless_document_id
- summary_paperless_document_id
- recording_paperless_document_id nullable
- notes
### `schoolhouse_d2l_sync_run`
- id
- started_at
- completed_at
- status
- raw_snapshot_path
- changes_summary_json
- error_text
### `schoolhouse_assignment_mapping`
- id
- course_id
- d2l_grade_item_name
- normalized_key
- assignment_id
- confidence
- mapping_source (`auto|manual`)
This table is the bastard that keeps D2L grade items tied to internal assignments cleanly.
## Matching rules
### Assignment matching priority
1. exact D2L assignment key when available
2. normalized title within course
3. filename heuristics
4. latest due date proximity
5. manual confirmation
### Grade/comment update policy
- D2L refresh updates cached grade/comment fields for mapped assignments
- if a submission has a Paperless document ID and grade/comment changed, append note once per distinct change hash
- never overwrite user notes in Paperless; append a dated Schoolhouse block
## Recording pipeline contract
### Input limits
- accept WAV up to at least 1 GB
- support up to 90 minutes
- reject unsupported formats with clear message
### Processing steps
1. store upload metadata row
2. convert WAV → MP3
3. delete WAV only after MP3 success
4. submit MP3 to recording workflow
5. receive transcript/summary callback
6. archive transcript + summary into Paperless
## Secrets/config
Required env:
- `SCHOOLHOUSE_HOST`
- `SCHOOLHOUSE_PORT`
- `DATABASE_URL`
- `PAPERLESS_BASE_URL`
- `PAPERLESS_API_TOKEN`
- `N8N_BASE_URL`
- `SCHOOLHOUSE_N8N_ASSIGNMENT_WEBHOOK`
- `SCHOOLHOUSE_N8N_RECORDING_WEBHOOK`
- `D2L_SCRAPER_WORKDIR`
- `D2L_SCRAPER_COMMAND`
- `UPLOAD_TMP_DIR`
- `MAX_UPLOAD_MB`
- `SESSION_SECRET`
Do not commit live values. Use `.env.example` only.
## Repo / deployment plan
Repo source path:
- `home/doris-schoolhouse`
Expected live NOMAD runtime path later:
- `/opt/doris-schoolhouse`
Expected live data path later:
- `/opt/doris-schoolhouse/data`
Expected compose validation before deploy:
```bash
cd /opt/doris-schoolhouse
docker compose --env-file .env config
```
## Build phases
### Phase 1
- app scaffold
- schema
- D2L import command/service
- assignment upload UI
- Paperless confirmation flow
### Phase 2
- recording upload flow
- MP3 conversion
- recording job tracking
- transcript/summary callbacks
### Phase 3
- grade sync
- Paperless note append logic
- dashboard views
### Phase 4
- manual mappings UI
- recurring assignment templates
- mobile cleanup
## Verification checklist
- Python modules compile
- schema file present and readable
- compose file parses
- route map loads without import failure
- README explains runtime/deploy expectations
- no secrets committed