Scribes

Enhances the note-taking process for students with learning disabilities by providing an annotatable live transcript of lectures which is then categorized and summarized using advanced ML algorithms.

Comment

Inspiration

Many students have learning disabilities that negatively impact their education. After doing some research, we learned that there were nearly 150,000 students in Ontario alone! UofT offers note-taking services to students that are registered with accessibility services, but the service relies on the quality and availability of fellow student volunteers. After learning about Cohere, we realized that we could create a product that could more reliably provide this support to students.

What it does

Scribes aims to address these issues by providing a service that simplifies and enhances accessibility in the note-taking process. The web app was built with accessibility as the main priority, delivered through high contrast, readable fonts and clear hierarchies.

To start saving lecture sessions, the user can simply sign up for free with their school email! Scribes allows students to record live lectures on either their phone or laptop, and then receive a live transcript that can be highlighted and commented on in real-time. Once class is done, the annotated transcript is summarized by Cohere's advanced NLP algorithms to provide an easily digestible overview of the session material.

The student is also presented with definitions and additional context to better understand key terms and difficult concepts. The recorded sessions and personalized notes are always accessible through the student's tailored dashboard, where they can organize their study material through the selection of tags and folders.

How we built it

Designers:

  • Conducted research to gain a better understanding of our target demographic, their pain points, and our general project scope.
  • Produced wireframes to map out the user experience.
  • Gathered visual inspiration and applied our own design system toward creating a final design for the web app. Devs:
  • Used Python and AssemblyAI API to convert audio files into text transcription
  • Used Cohere to summarize text, and adjusted hyperparameters in order to generate accurate and succinct summarizations of the input text
  • Used Flask to create a backend API to send data from the frontend to the backend and to retrieve data from the Cohere API Team:
  • Brainstormed potential project ideas and features specific to this flow.
  • Shared ideas and portions of the project to combine into one overall project.

Challenges we ran into

  • Troubleshooting with code: some of our earlier approaches required components that were out of date/no longer hosted, and we had to adjust by shifting the input type
  • The short timeframe was a barrier preventing us from implementing stylized front-end code. To make the most out of our time, we designed the interface on Figma, developed the back-end to transcribe the sessions, and created a simple front-end document to showcase the functionality and potential methods of integration.
  • Figuring out which platform and which technologies to use such that our project would be reflective of our original idea, easy and fast to develop, and also extensible for future improvements

Accomplishments that we're proud of

Over the course of 36 hours, we’ve managed to work together and create an effective business pitch, a Figma prototype for our web app, and a working website that transcribes and summarizes audio files.

What we learned

Our team members learned a great deal this weekend, including: creating pitches, working in a tight timeframe, networking, learning about good accessibility practices in design when designing for those with learning needs, how to work with and train advanced machine learning models, python dependencies, working with APIs and Virtual Environments.

What's next for Scribes

If provided with more time, we plan to implement other features — such as automatically generated cue cards, a bot to answer questions regarding session content, and collaborative notes. As we prioritize accessibility and ease-of-use, we would also conduct usability testing to continue ensuring that our users are at the forefront of our product.

To cover these additional goals, we may apply to receive funding dedicated to accessibility, such as the Government of Canada’s Enabling Education Fund. We could also partner with news platforms, wiki catalogs, and other informational websites to receive more funding and bridge the gap in accessing more knowledge online. We believe that everyone is equally deserving of receiving proper access to education and the necessary support it takes to help them make the most of it.

Built With

Share this project:

Updates