TREVA | Devpost

Inspiration

We were inspired by the quest to streamline the endless sea of lecture videos into digestible knowledge nuggets. This sparked the creation of TREVA.

What it does

TREVA is a tool that takes a video lecture and is able to generate an srt file from it with detailed timestamps with wonderful state of the art speech recognition capabilities. We have then produced custom AI models to perform analysis on these files and render a more distilled, engaging set of lecture notes that can be used at your convenience.

How we built it

Our first step involved providing users with multiple to input their content. Whether it's a video link, an SRT file, or a video file uploaded directly from their device, our platform can handle it all. Additionally, we offer a convenient browser extension that utilises Whisper WebAssembly tools, enabling users to convert videos into SRT files effortlessly.

The second stage focused on backend processing. Once we received the content, we generated the final SRT file and ensured that the output data was meticulously processed and prepared for the subsequent stages. The 3rd stage - the crux of our project - was where we implemented advanced custom algorithms to generate accurate punctuation and derive paragraph structures from the SRT file. By leveraging this technology, we were able to achieve remarkable accuracy and efficiency in the transcript processing

Finally, we brought all the components together to create a cohesive and user-friendly output. By combining the SRT file with the readable text, we generate chapter information that includes the beginning time, end time, and a concise summary for each chapter. This feature allows users to effortlessly navigate through the content, making it easier to find and reference specific sections within the transcription.

Challenges we ran into

We found that one of the biggest challenges was to create something from the ground up without being forced to reinvent the wheel. Specifically, quite a few of the libraries and existing codebases we sought had limited documentation and/or inconsistencies that were difficult to reoconcile. In hindsight, given the inherent complexity of our web extension-enabled transcription tool, there were limited resources we could relying on for help when debugging or troubleshooting a series of cascading issues. However, our team still managed to pull through and produce something we are all proud of within the hackathon deadline.

Accomplishments that we're proud of

We utilised existing codebases to develop a rather distinctive offering. While the market features audio transcribers and summarisers integrated with web UI frameworks, our innovation condenses this functionality into a single web extension, allowing our app to run locally. This method not only ensures true scalability but we also feel it sets our project apart, especially given that executing such inference tasks directly in a browser has traditionally been seen as a nontrivial task.

What we learned

We gained substantial insights in how to developing infrastructure for projects we care about, and then scaling them beyond our personal devices, making them accessible to a broader audience. Another driving force for our efforts was the concept of scalability. Our friends and mentors taught us that scalability encompasses far more than just computing resources or speed. It's about the collective benefit derived from an expanding user base. Analysing trends, such as the most transcribed lectures or frequently summarised sections, demonstrates how effectively our system leverages the ability to flourish as the reach we have grows larger and larger.

What's next for TREVA

We hope TREVA can contribute toward a paradigm shift within classroom learning! As it continues to grow, we hope to make it available to a growing user base who wish to use a tool that frees them from the shackles of agonising over the minutiae; we hope it can be a tool which boosts productivity and enables a deeper engagement with the core teachings of lectures.