SiGnL | Devpost

Inspiration 💭

Embarrassment. Humiliation. Discomfort. These are felt by deaf people, who wish to participate in the chatter and fully comprehend their surroundings with the rarely encountered sign language. Here at SiGnL, pronounced ‘signal,’ we understand how hard it is to pick up a new language. We also understand that someone who learns the language of signs might not use it on a regular basis, further discouraging its uptake.

You might be asking, why not just text? Preserving sign language is important because it encompasses body and facial expressions. This sustains the rapport between individuals as it helps bridge the communication gap between a hearing-impaired person and a hearing person. SiGnL strives for equal accessibility and raises awareness of the power of signing.

What it does 👀

Not only can users use this software as a standalone app, but it can also be employed on a video call with someone who needs sign language translation. SiGnL can output sign language into audio translation, which is done by using the python library pyttsx3. The real-time detection uses pose estimation which stores key points of the signs. On our intuitive UI, users will see a detection box and a matching score that gauges the accuracy of the translations.

How we built it 💡

SiGnL uses Python to run both its GUI and AI model. Using the Tensorflow Object Detection API, we employed transfer learning using SSD MobileNet. We trained our model with a Kaggle dataset for ASL (American Sign Language) combined with our own images, which were fed through Label img. We implemented Mediapipe for real-time pose estimation on CPUs and GPUs paired with OpenCV. This allowed us to detect sign language poses in real-time using a local webcam. As for the front end, we used Tkinter to actualize the program window. We also used Pillow to incorporate images into the window.

Challenges we ran into 🌋

One of our most time-consuming challenges was the decision regarding a method of execution for the program that would optimize time efficiency and performance. We bounced between creating a Chrome extension to making a website to programming an app—it was befuddling, to say the least. While programming, we had to find a viable option to pass audio through to a virtual microphone and outputting audio to the correct sound device. We also had to figure out how to merge the Tensorflow model with Python GUI.

What we learned 🤠

• How to perform computer vision analysis on the web through a user's webcam

• Finding/ creating a suitable database to train the ai

• Figured out how to create a virtual camera to pass camera feed through, since two applications can’t use the same camera at the same time

What's next for SiGnL

We hope to carry out a wider vocab library, including complex signs that incorporate a lot of movement. This will lead to greater sensory accuracy, which is achievable with more database elements per word. Furthermore, applying TensorFlow to detect changes in facial expression and body language. Incorporation of more languages and other variations of sign language.