ALEX - Augmented Lecture Explainer

Main Interface, showing an interaction with ALEX
Code Creation Example
Live Setup Example
Repition Setup Example
Lecture Question Example
Graphic Example
Structure of Project

Inspiration

Have you ever found yourself in a lecture with a burning question but hesitant to interrupt the professor and disrupt the flow? Or have you ever attended live-streamed lectures without a way to ask questions or get quick and vivid clarification? Enter Alex—our Augmented Lecture Explainer, designed to bridge this gap.

What it does

Try it out : ALEX Alex combines transcript from live-streamed lectures with lecture slides as a knowledge base, enabling students to ask questions using chat functionality during live sessions and also for past lectures. It provides answers and, when relevant, offers visualizations to clarify concepts.

How we built it

We used whisper model to convert lecture videos into transcriptions and then synchronized them with timestamps and corresponding slides. The processed data is stored in a vector database for seamless retrieval. By leveraging RAG (Retrieval-Augmented Generation), the system provides accurate, context-aware responses. For this project, we utilized a self-hosted Llama 3.2 Vision Model.

Challenges we ran into

Matching video frames with the current slides
Generating accurate visualizations

Accomplishments that we're proud of

Successfully interpreting video content and slides, with support for multiple slides in a single session
Seamlessly integrating a real-time chat feature for interactive learning
Providing accurate, context-aware answers to students' questions during lectures
Generating visualizations to simplify complex algorithms or address intricate questions

What we learned

Through this project, we gained valuable experience in working with vision models to process and interpret visual content. We also learned how to effectively handle and analyze video data, ensuring seamless synchronization with other elements. Additionally, we developed the skills to integrate a language model with code execution, enabling dynamic and context-aware responses.

What's next for ALEX

We aim to integrate the system with learning platforms like moodle and automate slide processing for a seamless experience. Additionally, we plan to connect with TUM-live , making the solution accessible to all TUM students.