ScholarScribe | Devpost

Inspiration

Our inspiration behind ScholarScribe is to enhance the learning process and make it more efficient and effective for both students and educators. We found that student’s actually achieve 13% higher test achievements than not taking notes, especially combined with cued lectures (study from University of Ohio), and by summarizing the main points covered in a lecture, this app can help students better understand and retain the material, as well as save them time when reviewing and studying for tests and exams. We were also inspired by creating a more inclusive environment for individuals with disabilities hence this software could be used by, for example, blind or low vision individuals who can use speech-to-text to hear written text read aloud. People with mobility disabilities can also use speech-to-text for text generation without physical typing. This accessibility helps to remove barriers and creates a more inclusive digital environment.

What it does

This app would allow students to take effective lecture notes without having to spend hours listening to and transcribing lecture videos. All the user would have to do is upload the lecture video and the app would generate a summary of the video as notes. The notes would be comprehensive, covering all the key points of the lecture. They would also be organized into sections, making it easy to review and study the material. Additionally, the notes could be shared and edited collaboratively, making it a great tool for group study.

How we built it

ScholarScribe uses facebook’s BART, NLTK, GPT3, React.Js, Vite.Js, OpenAI whisper, Python, Notion API, HTML, CSS

We developed ScholarScribe utilizing Python in the backend, React.Js and Vite.Js in the front end. For the backend, we employed OpenAI's whisper model to transcribe lecture videos or audio, then passing the transcription to Facebook's NLP model BART, which blends Google's BERT model, which is bidirectional, and OpenAI's GPT3 model, which is auto-regressive, to generate a summarized transcription that retains context and emphasizes the most important concepts of the lecture. The output is then transmitted to the front-end, where the summarized text is displayed and connected to the Notion API for displaying the notes.

Challenges we ran into

During our development process, we encountered an issue with BART, which has a maximum sequence length of 1024 sub-tokens and is unable to summarize longer transcripts. To resolve this, we imported the NLTK library and used it to chunk the transcript into smaller parts that could be processed by the summarizer. NLTK enabled us to separate sentences taking into account English abbreviations and periods, and we divided the total number of words in the document by the maximum sequence length to determine the number of chunks required. We ran into other limitations in APIs we were planning on using and had to quickly adapt to the API or change to a new API.

Accomplishments that we're proud of

We are proud of several accomplishments in our project that combines machine learning, web application development, and adaptability. Firstly, we successfully implemented a machine learning model that can accurately transcribe then summarize a whole lecture. Secondly, we developed a user-friendly web application that allows for easy access and interaction with the application by a wide range of users. Finally, we are proud of our adaptability in the face of challenges and our ability work around any issues with our own problem solving skills. Overall, this project demonstrates our team's expertise in utilizing cutting-edge technology to create practical and effective solutions.

What we learned

During the development of ScholarScribe, we learned a lot about machine learning implementation, integration and system design. In machine learning, we learn about the various models implemented and their requirements and outputs while also learning about preprocessing data before sending it in. In integration we learned about several very useful APIs for our project and how to properly assess documentation and apply it to the application. Finally, in system design we learned a lot about communicating between different parts of the application and how to effectively and efficiently do so.

What's next for ScholarScribe

We see ScholarScribe as a tool that is incredibly useful for teachers and students alike and so we want to tailor the future of ScholarScribe to the future of education. In the future we are planning on implementing Google Docs and Slides integration so that students can more easily obtain the notes in any format comfortable to them. The Slides would also be populated with images related to each slide using text-to-image generation, making it even better for education. We also think that ScholarScribe will expand to be able to transcribe and take notes in real time during the lecture based on content classification.