Kanting 看听 | Devpost

Website frontend
Streaming frontend web
Streaming frontend phone

Inspiration

More often than not, we will inevitably get distracted during a 2-hour-long lecture or presentation. Or even when we are paying attention, we can't focus on the presenter while frantically trying to copy down the slides. That's why we created Kanting. (See-Hear in Chinese)

What it does

Kanting is an audiovisual note taking website, powered by the newest AI technology. It automatically captures unique key frames from slideshows and gives notes generated from real time transcription of what is being discussed in lectures, meetings, and presentations. It implements a state-of-the-art RAG database pipeline for the user to retrieve not only actual notes but also the exact moment in the lecture recording where the lecturer answers the question by talking to our chatbot.

How we built it

We built an interactive frontend website with Next.js and Tailwind. The backend consists of Deepgram STT which is fed to Claude to summarize it. We used FFpmeg to combine the audio and video streams into a .mp4 to be saved to our S3 databases. For keyframe capturing during the livestream, we utilized the the Structural Similarity Index Measure to filter out minor changes and store the last captured frame before major changes occur. We implement an efficient RAG pipeline using the power of ChromoDB and add our customized JSON retrieval mechanism to improve the efficiency.

Challenges we ran into

It was hard getting all the endpoints to work together, especially we have three major components in the backend: audio processing, video processing, notes retrieval. We also have two frontend that needs to work together.