Inspiration
At our school, we observed a classmate who is part of the hard-of-hearing community struggling to keep up with the teacher’s pace. This student frequently had difficulty understanding the teacher’s lessons and instructions, leading us to believe that they felt excluded. We began to wonder how many other students might be facing similar challenges, especially those with whom we had personal connections.
In response to this issue, we developed a program aimed at enhancing communication and accessibility for individuals who are hard of hearing. Our hope is that our project will not only positively transform our classmate’s classroom experience but also make a significant difference for many others in similar situations.
What it does
SignBridge is an innovative application designed to enhance communication and accessibility in academic environments for deaf and hard-of-hearing students. Leveraging cutting-edge real-time sign language to speech conversion, SignBridge allows students to communicate with professors using a camera, providing unparalleled mobility and immediacy. This functionality ensures that students can engage in dynamic, moving interactions without being confined to static text-to-speech systems. Furthermore, SignBridge offers an additional feature that generates detailed notes from the professor's audio, helping students maintain comprehensive records of lectures and discussions. This combination of real-time communication and automatic note-taking makes SignBridge a powerful tool for fostering inclusive and efficient learning experiences.
How we built it
We used HTML, CSS, JS and Python to create this website. We used many different packages to make it work, but notable ones include OpenCV and OpenAI. Prompts are either fed into ChatGPT API or PlayHT API to generate text and speech.
Challenges we ran into
The biggest difficulty was making the sign language hand-tracking work. OpenCV would often not work properly. However, after many hours of trying, we managed to make it function properly.
Accomplishments that we're proud of
Making the Hand-Tracking for OpenCV and learning how to use multiple different APIs were the biggest ones.
What we learned
We learned a lot more about using OpenCV and other ML models.
What's next for SignBridge
We will add features such as live note-taking translation and reduce latency. We will also upgrade the website to make it more functional.
Log in or sign up for Devpost to join the conversation.