HandScript | Devpost

Home-Page

Inspiration

One of our common teachers, Mr. Bernard, has trouble conversing in spoken language due to his advanced age and deafness. We have frequently seen him communicate with others in American Sign Language, and were inspired to build an easily accessible Sign Language translator.

What it does

Our framework uses images captured from one's computer and employs a pose-estimation algorithm to label landmarks in your hand. AI in the form of Neural Networks is then employed to predict the specific character that is being made based on the landmarks. The entire framework is packaged into a user-friendly website that also provides information about American Sign Language.

How we built it

On the backend, we employed a model that detected if a hand was in frame and generated a set of 63 landmarks if a hand was detected. These landmarks were inputted into a Neural Network which classified the landmarks into one of 26 english characters. On the front-end, we used HTML, CSS, JS, React, and Nodejs.

Challenges we ran into

Our original framework that we were leveraging wasn’t achieving satisfactory performance in a real-world setting on character classification. To ramify this, we developed an ensemble framework with a Pose Estimation intermediate step.
Our initial dataset didn’t contain diverse images and backgrounds which made it hard to generalize. We ended up pooling images from multiple datasets and used image augmentations to generalize the model.
We ran into many cross-platform issues with running code in different environments and programming languages but cleaned up the code which mitigated many of these issues.
We had problems integrating the front-end and back-end; Initially we tried to use ExpressJS and Sockets but we ended up using Flask and http.