From Project to Product: How cant.study → wehelpyou.study Transforms Student Note‑Taking and Meetings with Open‑Source LLMs

For three semesters I worked as an on‑campus student note‑taker, clocking in at 8 a.m. every Monday, Wednesday, and Friday. While the paycheck helped, the routine exposed a deeper accessibility gap: many courses never secure a note‑taker, leaving NTID classmates to piece together scattered slides, hurried phone photos, or—too often—nothing at all.

Why can’t note‑taking be automated, reliable, and context‑aware?

That question became CANT — the Context‑Aware Note Taker. (http://cant.study/)

From Manual Notes to Multimodal Intelligence

CANT’s earliest prototype hinged on three technical insights:

1. Context Correction Layer
   Raw audio is first transcribed by NVIDIA Canary. A Context Correction Layer then aligns jargon, acronyms, and speaker disfluencies with the instructor’s actual slides and course materials, transforming transcripts into task‑ready text.

2. Iterative Chunk Summarization
   Lectures are streamed in ~10 k‑token chunks. Each chunk is summarized, then adjacent summaries are recursively refined to build a cohesive document—reducing LLM hallucinations and keeping sections tight and well‑labeled.

3. 24‑Hour Retention Window
   To balance study needs with privacy, only the generated structured notes are retained on the server after the first 24 hours. The system performs an auto‑purge unless a student explicitly opts to archive them.

Design Decisions & Why They Won

A (Very) Brief Tour of the Deployment Pipeline

1. Frontend – React 18 + Tailwind. A single “Start Session” button streams 30 s audio chunks and visualizes progress.

2. API Layer (Node.js + Express) – Handles JWT auth, writes Langfuse traces, handles other business logic.

3. Sandbox Container

   • Speech: NVIDIA Canary

   • Context Parser: OCR for slides/PDFs

   • LLM: Llama 3 8B‑Instruct‑128k behind an Nginx reverse proxy

4. Storage – MongoDB for transcripts & structured notes (ephemeral collections respect the 24‑hour policy).

5. CI/CD – GitHub Actions → Docker Buildx → zero‑downtime blue‑green deploys on Linode’s dedicated GPU cloud.

Launch Day: 30 April 2025 — Capstone Poster & Public Release

We unveiled CANT during the RIT Data Science Capstone poster session on 30 April 2025 and simultaneously opened public access:

First‑week impact

• 42 new registered users

• 100 + unique visitors

• 8 + hours of audio captured

• 14 structured note sets generated

Improvements and Results

What’s Next?

1. Federated Learning
   Train the Context Correction Layer on‑device so each cohort’s jargon improves future transcripts—without centralizing raw data.

2. Edge Deployment using Service Workers
   Packaging the full pipeline for offline field trips and patchy Wi‑Fi environments.

3. Open‑Source Modules
   Releasing the chunk‑iterative summarizer, Context Aware ASR Correction so researchers can plug in custom LLMs.

Thank you for following our journey from sleepy lecture halls to an AI‑powered accessibility platform. Have feedback or want to contribute? Visit cant.study—because when you CANT.study, WEHELPYOU.study.

Email: gmail@sanathswaroop.com

Contact: +1 (585)-565-2567