Inspiration
Live translation of spoken English to closed captions is fairly common at this point: however, very few, if any such tools exist for American Sign Language. Additionally, to our knowledge, no such tool exists that does not require the signer to wear specialized gloves. We aim to address this with our project.
What it does
At the current moment, our project is able to translate ASL fingerspelling to the English alphabet in real-time.
How we built it
Our backend is built in Python and Flask, and we leverage Mediapipe and OpenCV to help with hand position detection.
Challenges we ran into
Initially, we planned on training a convolutional neural network to classify which letter was being signed. However, we ran into challenges with finding and creating a suitable dataset to train our model on. Additionally, we felt that training the network would take too long given the amount of time that we had, as well as our available hardware. Thus, we pivoted to using Mediapipe.
Accomplishments that we're proud of
Coming up with a working product!
What we learned
We learned how to use Mediapipe and OpenCV, two very useful tools for real-time computer vision and machine learning.
What's next for ASL Fingerspelling Translator
In the future, we aim to expand the translator beyond just fingerspelling. We hope to leverage large language models, such as GPT-3, as well as common deep learning architectures used in computer vision, such as convolutional neural networks and/or vision transformers, to build a fully fleshed out, real-time translation software for ASL.
Log in or sign up for Devpost to join the conversation.