SignBuddy | Devpost

Main menu
Learn
Learn 2
Profile
Login Screen
Sign Language Detection

Inspiration

Two of our members are first-time hackers, so we didn’t come into this competition expecting much more than to learn something new and to build something cool. That being said, we opted for more than that after some brainstorming. Rather than just a neat technical project, we chose to create something that could truly help people. That’s how we came up with SignBuddy, an educational app that uses Computer Vision to make learning sign language more accessible and more convenient for people afflicted with deafness, for those close to them, and anybody who might want to learn to sign.

What it does

Our project is designed to make sign language education easy, accessible, and affordable to people of all kinds–deaf and hearing, young and old. The app includes two interactive activities: Practice Mode and Play Mode.

In Practice Mode, the app displays flashcards containing a word and its sign to the user, who must in turn repeat that sign to the camera. The app uses Computer Vision to recognize when the user has correctly made the sign, and move on to the next card. When the user encounters a sign enough times, it is added to their Sign Dictionary.

In Play Mode, the app displays words from the Sign Dictionary but does not reveal the sign to the user, who has a limited time to guess and make the sign to the camera. If they cannot make the sign in time, they lose the game.

How we built it

First, we created a dataset with which to adapt a generic object-recognition model to our particular application of recognizing hand signs. For this, we drafted Khalil to take dozens of pictures of himself making hand signs to his webcam, then to use a software called LabelImg to label them with what word was being signed. The labeled pictures were then used as training data for a Machine Learning model, which would use that experience to recognize those same hand signs from live camera input.

We used React to build the frontend, and handled user authentication with MongoDB. We also used Tailwind to pretty it up a little.

Challenges we ran into

Perhaps the biggest challenge our team faced was staying on the same page. It was critical to have a continuous stream of communication and update each other regularly on our progress. This way, we minimized the chance of “integration hell” at the end of the project, where the various parts of the project have deviated far from one another and become difficult to combine into one coherent product. The team was split between tasks such as designing the app, training the model, creating the frontend, and the user authentication system. Our solution to this problem was to have each person on the team work on more than one task, and to ensure that all team members were familiar with the design of the app before beginning their work.

Another challenge was working with unfamiliar technologies. Most of us found ourselves working with something we had never used before, simply because we needed it to make the project work. Working and learning along the way was as exciting as it was challenging.

Accomplishments that we're proud of

As mentioned above, one of the biggest challenges we faced was coming together as a team and integrating all of our individual work into one, seamless product. Although it was difficult, we ended up combining our three important elements; the backend, which is readily available on the website, the website itself, and the MongoDB login feature. Each plays a pivotal role in delivering the accessible, and simple-to-use sign language experience we aim to provide. Not only was merging three sizeable features like this a difficult task, but considering the complexity of each one, we thought it to be impossible given the time we had. We were combining an object-recognition program with a web-app and authentication system. We were proud, in the end, to have pulled it off.

What we learned

We were out of our comfort zones for most of the event. Two group members working on the frontend used React for the first time, and the ones training the model had never done image recognition or used Tensorflow before. This project forced us to power through our inexperience and to do our research at the same time as our work. What should ordinarily take days of preparation became an improvised effort.

What's next for SignBuddy

The central feature of SignBuddy is its recognition of hand signs. Properly trained Machine Learning algorithms can take hours, or even days of training, and training can be quite demanding on hardware. They also require large and diverse datasets in order to perform well in many different contexts. While a hackathon is not the appropriate place to address these issues, an app like SignBuddy requires far more data, training time, and computing power to be an effective product.