Signlingo

• 

• 

• 

• 

  System Diagram

• 

  Entity Relationship Diagram

• 

• 

Inspiration

Signlingo was inspired by the growing gap in communication accessibility for the Deaf and hard-of-hearing communities. Many deaf individuals and their families desire to learn American Sign Language (ASL) to bridge this gap but often find existing platforms to be either expensive, unengaging, or lacking in structured learning paths. Families especially need consistent, engaging, and adaptive learning tools to build proficiency. This app aims to make ASL learning accessible, efficient, and enjoyable through real-time feedback and interactive practice.

What it Does

Signlingo is a web app designed to teach users ASL through bite-sized lessons, real-time gesture recognition, and gamified learning experiences. The app customizes content to the user’s skill level, creating a learning path that includes daily challenges and skill progress tracking. Users can practice signs relevant to everyday interactions, receive instant feedback on gestures, and play games that reinforce both vocabulary and comprehension of ASL.

How We Built It

The app's architecture leverages a React frontend with Next.js for enhanced SEO and server-side rendering. On the backend, we used Flask for API endpoints, Cloudflare for managing edge routing and caching, and Supabase for user authentication and real-time data storage.

For ASL recognition, we implemented a custom convolutional neural network (CNN) model using Keras and TensorFlow. The model was trained on an extensive ASL Alphabet dataset, containing over 87,000 images across 29 classes representing letters A–Z and additional gestures (space, delete, and nothing). Each image was resized to 200x200 pixels to balance performance with real-time accuracy. To address variations in lighting and background, we used robust data augmentation techniques, such as random brightness adjustments, flips, rotations, and noise injection. This improved the model’s generalization on real-world video data significantly.

Challenges We Faced

Training a model to reach high real-time accuracy (aiming for over 99%) posed several challenges:

The model initially overfit due to limited background diversity in the training dataset, leading to reduced accuracy in real-world settings. Differences in lighting and background across user environments heavily impacted prediction reliability. We introduced real-time adaptive filters to stabilize recognition under these varying conditions. Streaming and processing live video data required high responsiveness, which we optimized using WebSockets for real-time data streaming. The video frames were streamed directly to a Cloudflare Worker, which allocated processing to a local inference server for minimal latency.

Accomplishments We're Proud Of

Achieving real-time ASL recognition with high accuracy across varied user environments is a major accomplishment. Our model can robustly handle challenging lighting and background conditions, thanks to both our augmented training dataset and adaptive filtering. Additionally, we’re proud of optimizing video streaming and inference through WebSocket connections, which significantly reduced latency and improved the overall user experience. This setup enables Signlingo to process and provide feedback on ASL gestures almost instantaneously, allowing for a more interactive and satisfying learning experience.

What We Learned

Building Signlingo taught us the importance of accessibility and user-centered design in language-learning applications. Through this project, we developed strategies to combat overfitting and enhance generalization by leveraging extensive data augmentation and model tuning. Additionally, we learned the value of optimizing real-time video processing for responsiveness and accuracy—a key factor for any app that requires instant feedback.

What's Next for Signlingo

Looking ahead, we plan to expand Signlingo’s capabilities by:

Expanding beyond ASL to incorporate other sign languages, creating a universal learning platform. Introducing real-time multiplayer modes, allowing users to practice with friends or family, or engage in friendly challenges with other learners. Leveraging AR to superimpose sign hints in real-time for better immersion and learning retention. Incorporating reinforcement learning to further personalize learning paths based on user strengths and weaknesses, adapting dynamically as the user progresses. Signlingo aims to redefine sign language education through advanced AI, interactive learning, and a deep focus on accessibility and inclusivity for the Deaf and hard-of-hearing communities

Built With

• cloudflare
• flask
• nextjs
• python
• supabase
• tailwindcss
• websockets