Sign2Voice

An assistive web application that helps deaf people communicate with others.

Comment

Inspiration

We wanted to create a web application that had significant social impact, and we decided that enabling a particular group of people (i.e. deaf people/people hard of hearing) to be more self-reliant was a good starting point.

What it does

Sign2Voice is a web application that is able to recognize American sign language gestures and translate them into text, and eventually into speech, bridging the communication gap between deaf and non-deaf individuals.

How we built it

We mainly used JavaScript in our web app to receive input from a client's webcam, make API requests to classify the gestures, and convert the resulting text into audio. We trained a custom model on Google's AutoML Vision for classification, and we used Watson's text-to-speech API to eventually translate it into speech.

Challenges we ran into

We faced a few technical challenges: Most notably, it was difficult to train a sufficiently-accurate classifier for the gestures, given the limited time and compute power that we had available. Additionally, it was a challenge to get the user interface exactly the way we wanted it to look and perform.

Accomplishments that we're proud of

We're proud of the usability and overall aesthetic of the user interface of our web application. Additionally, we're proud that we achieved a minimal viable product in a short span of time.

What we learned

We learned how to create a good user interface for a web application, about how to create efficient computer-to-computer interfaces, and picked up a few new technologies (e.g. Google Cloud SDK). We also learned how to work more effectively with other developers.

What's next for Sign2Voice

In the future, we hope to improve the accuracy and minmize the latency of our gesture classifier. We also plan to do more usability testing to improve the visual and interaction design of our user interface.

Built With

Share this project:

Updates