Transform any lecture into an interactive AI-powered learning experience with avatar tutors, auto-generated quizzes, and 3Blue1Brown-style explainer videos.
Professors can't give personalized 1-on-1 attention to hundreds of students. Office hours are limited, and students often struggle with concepts without immediate help.
An AI-powered professor toolkit that:
- Clones the professor as an AI avatar that can tutor students individually
- Generates animated explainers from any lecture (3Blue1Brown style)
- Creates interactive quizzes delivered via Zoom Team Chat
- Provides real-time analytics on student understanding
- Professor's likeness cloned as interactive AI avatar
- Real-time lip-sync and natural conversation
- Joins Zoom breakout rooms to tutor students 1-on-1
- Context-aware responses using lecture transcripts
Turn any YouTube lecture into animated educational content:
YouTube URL → Transcribe → Scene Split → Manim Animations → Voice Clone → Final Video
- Whisper transcription via Dedalus API
- LLM scene planning - intelligently splits lectures into concept-based scenes
- Auto-generated Manim code - creates 3Blue1Brown-style animations
- Voice cloning with PocketTTS - maintains the professor's voice
- Parallel rendering - generates multiple scenes concurrently
- Zoom Team Chat Chatbot - students type
/makequizto start - Auto-generated questions from lecture concepts using Cerebras LLM
- Interactive button cards - A/B/C/D answer buttons
- Video on wrong answer - plays the relevant Manim explainer scene
- Progress tracking - scores and concepts to review
- Zoom RTMS (Real-Time Media Streams) for live transcription
- WebSocket architecture - Render service broadcasts to local dashboard
- Live transcript accumulation per meeting
- Demeanor/engagement analysis (extensible)
- Frosted glass Electron UI
- One-click session start
- Real-time student analytics
- Quiz trigger buttons
- Meeting management
┌─────────────────────────────────────────────────────────────────────────────────┐
│ PROFESSOR DASHBOARD │
│ (Electron + React + Tailwind) │
└───────────────────────────────────┬─────────────────────────────────────────────┘
│ WebSocket
▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│ PYTHON BACKEND │
│ (FastAPI + SQLite) │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │
│ │ Zoom API │ │ HeyGen API │ │ Cerebras LLM│ │ Quiz Session Manager │ │
│ │ (meetings) │ │ (avatars) │ │ (generation)│ │ (state per student) │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────────────────┘ │
└───────────────────────────────────┬─────────────────────────────────────────────┘
│
┌───────────────────────────┼───────────────────────────┐
▼ ▼ ▼
┌───────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Zoom Meeting │ │ Render (RTMS) │ │ Zoom Team Chat │
│ │◄───────►│ Node.js │◄───────►│ Chatbot │
│ - Breakouts │ RTMS │ - Webhooks │ WS │ - /makequiz │
│ - Avatars │ WS │ - Transcripts │ │ - Buttons │
└───────────────┘ └─────────────────┘ └─────────────────┘
| Layer | Technology |
|---|---|
| Frontend | Electron, React, TypeScript, Tailwind CSS |
| Backend | Python 3.11+, FastAPI, SQLAlchemy, asyncio |
| RTMS Service | Node.js, Express, WebSocket |
| Video Pipeline | Manim, FFmpeg, PocketTTS |
| Database | SQLite (dev), PostgreSQL (prod) |
| Deployment | Render (cloud), uv (Python pkg mgmt) |
| Service | Purpose |
|---|---|
| Zoom REST API | Meeting creation, breakout rooms, user management |
| Zoom RTMS | Real-time audio/video/transcript streams |
| Zoom Team Chat | Chatbot for interactive quizzes |
| HeyGen | AI avatar generation and streaming |
| Deepgram | Speech-to-text, text-to-speech |
| Cerebras | Fast LLM inference (Llama 3.3 70B) |
| Dedalus | Whisper API for transcription |
| PocketTTS | Voice cloning for narration |
| HuggingFace | Model hosting for TTS |
TreeHacks2026/
├── src/ # Manim video pipeline
│ ├── pipeline.py # Main orchestration
│ ├── downloader.py # YouTube audio download
│ ├── transcribe.py # Whisper transcription
│ ├── scene_splitter.py # LLM-based scene planning
│ ├── clip_generator.py # Manim code generation
│ ├── voice.py # TTS with voice cloning
│ └── stitcher.py # Final video assembly
│
├── backend/ # Python backend
│ ├── app.py # FastAPI main app
│ ├── run_chatbot_client.py # Quiz WebSocket client
│ ├── services/
│ │ ├── render_ws_client.py # Connects to Render WebSocket
│ │ ├── chatbot_ws_handler.py # Handles /makequiz commands
│ │ ├── quiz_generator.py # LLM quiz generation
│ │ ├── quiz_session_manager.py # Per-student quiz state
│ │ ├── zoom_chatbot_service.py # Zoom API message sending
│ │ ├── heygen_controller.py # Avatar management
│ │ ├── session_orchestrator.py # Meeting lifecycle
│ │ └── llm_service.py # Cerebras/OpenAI wrapper
│ └── models/ # SQLAlchemy models
│
├── rtms-zoom-official/ # Render-deployed Node.js service
│ ├── index.js # Express + webhook handlers
│ ├── frontendWss.js # WebSocket broadcasting
│ └── library/ # RTMS SDK wrappers
│
├── frontend/ # Electron app (if separate)
│
├── outputs/ # Generated content (gitignored)
│ └── {topic-name}/
│ ├── audio.mp3
│ ├── transcript.txt
│ ├── scene_plan.json
│ ├── quiz_questions.json
│ └── videos/
│
├── prompts/ # LLM prompt templates
│
└── docs/ # Documentation
├── QUIZ_INTEGRATION.md # Quiz system guide
├── QUICKSTART.md
└── DEPLOYMENT.md
- Python 3.11+
- Node.js 18+
- uv (Python package manager)
- FFmpeg
- LaTeX (for Manim)
git clone https://github.com/ChristmasSun/TreeHacks2026.git
cd TreeHacks2026
# Python dependencies
uv sync
# Node dependencies (for RTMS service)
cd rtms-zoom-official && npm install && cd ..# Backend
cp backend/.env.example backend/.env
# Edit backend/.env with your API keys
# RTMS Service
cp rtms-zoom-official/.env.example rtms-zoom-official/.env
# Edit with Zoom credentialsRequired API keys:
ZOOM_CLIENT_ID,ZOOM_CLIENT_SECRET,ZOOM_ACCOUNT_IDZOOM_CHATBOT_CLIENT_ID,ZOOM_CHATBOT_CLIENT_SECRET,ZOOM_BOT_JIDHEYGEN_API_KEYCEREBRAS_API_KEYDEDALUS_API_KEYHF_TOKEN(HuggingFace for PocketTTS)
export DEDALUS_API_KEY="your-key"
export HF_TOKEN="your-huggingface-token"
uv run python -c "
import asyncio
from src.pipeline import run
asyncio.run(run(
'https://www.youtube.com/watch?v=YOUR_VIDEO_ID',
'outputs/your-topic',
clip_concurrency=4
))
"# Set quiz data directory
export QUIZ_DATA_DIR=outputs/your-topic
# Start the WebSocket client
python backend/run_chatbot_client.pyThen in Zoom Team Chat, message your bot with /makequiz.
# Terminal 1: Backend
cd backend && uvicorn app:app --reload --host 0.0.0.0 --port 8000
# Terminal 2: RTMS Service (or deploy to Render)
cd rtms-zoom-official && node index.js
# Terminal 3: Frontend
npm run dev- Download - Extracts audio from YouTube video
- Transcribe - Whisper API converts speech to text with timestamps
- Scene Split - LLM analyzes transcript, identifies key concepts, plans scenes
- Generate Code - LLM writes Manim Python code for each scene
- Voice Clone - PocketTTS extracts speaker voice sample, generates narration
- Render - Manim renders animations, FFmpeg merges with voiceover
- Stitch - Combines all scenes into final video
- User types
/makequizin Zoom Team Chat - Zoom sends webhook to Render
- Render broadcasts via WebSocket to local Python
- Python loads quiz JSON, creates session, sends intro card
- User clicks "Start Quiz" button
- Python sends first question with A/B/C/D buttons
- User clicks answer
- If wrong → Python triggers video playback, sends explanation
- If right → Python sends next question
- At end → Python sends score summary
Zoom Webhook → Render (HTTPS) → WebSocket broadcast → Local Python
↓
Zoom API (send messages)
This allows the Python backend to run locally while receiving Zoom events through Render.
| Document | Description |
|---|---|
| QUIZ_INTEGRATION.md | Complete guide to video + quiz integration |
| QUICKSTART.md | Step-by-step setup guide |
| DEPLOYMENT.md | Deploy to Render |
| PLAN.md | Project roadmap and phases |
Videos generated from:
- Think Fast, Talk Smart (Stanford communication lecture) - 14 scenes
- Mathematics Gives You Wings (fluid dynamics lecture) - 16 scenes
- Human Behavioral Biology (Sapolsky lecture) - 10 scenes
Each generates:
- Animated Manim videos per concept
- Quiz questions linked to videos
- Voice-cloned narration
MIT
- Manim Community - Animation engine
- 3Blue1Brown - Inspiration for visual style
- Zoom Developer Platform - Meeting & chat APIs
- HeyGen - AI avatar technology
- Cerebras - Fast LLM inference
