Inspiration
The original inspiration for Unsubtle came from personal experiences supporting students in special-needs classes and wanting to find better ways to help them feel understood. As the idea developed, it became clear that the same underlying challenge—difficulty expressing or interpreting subtle non-verbal cues—affects many people, especially those who struggle with verbal communication. This realization expanded the project’s purpose. Unsubtle evolved into a way to empower individuals whose communication styles don’t fit traditional expectations, enabling them to express themselves more openly and be better understood by the people around them.
What it does
Unsubtle uses computer vision to detect and interpret body language in real time. By integrating with wearables, XR glasses, and headsets, the system analyzes non-verbal cues—posture, gestures, movement patterns—and translates them into visual, accessible messages. Users can also send quick signals or emotional indicators through microgestures, enabling discreet, low-effort communication. This bi-directional interaction helps teachers, caregivers, or communication partners better understand what a user may be trying to express when spoken communication is difficult. Our goal is to make non-verbal communication more visible, intuitive, empathetic, and inclusive.
How we built it
We integrated computer vision models trained on body-language datasets with sensor inputs from wearables and XR devices. Leveraging pose estimation and gesture recognition pipelines, the system detects key movement patterns and classifies them into meaningful emotional or communicative states. A lightweight interface for XR devices then presents simple visual cues and notifications, translating subtle gestures into clear, representative messages. Users can respond with microgestures to send emojis or short phrases (e.g., “I need personal space”), which are shared across devices via a colocation-based networking layer.
The system was prototyped using a combination of open-source computer vision frameworks, fine-tuned custom models, and real-time streaming connections to wearable hardware. A FastAPI backend hosts the vision models, receiving image data from the VR headset via POST requests for inference. The backend returns JSON responses with detected body parts and recognized poses (e.g., crossing arms to indicate defensiveness). The headset then renders the user’s skeleton based on this data, dynamically changing colors to reflect the detected body language.
Challenges we ran into
One major challenge was achieving reliable body-language interpretation across diverse environments and movement styles. Non-verbal communication is inherently nuanced, requiring a careful balance between technical accuracy and ethical considerations. Variations in lighting conditions and video quality also impacted the model’s ability to accurately detect body parts for the final body language classification. To address this, we fine-tuned our models using videos captured directly on a Meta Quest 3 headset to better reflect real-world XR conditions.
Another significant hurdle was setting up the colocation environment, which involved selecting the correct Unity version and ensuring proper project importing within the IDE. A key challenge was synchronizing the menu system across colocated users, ensuring both the display data (emojis, thought bubbles, notifications) and UI visibility remained consistent for all participants.
Additionally, we initially planned to run the computer vision model directly on the headset using Unity Sentis, but technical limitations and stability issues led us to pivot. Instead, we deployed the model on a FastAPI backend server, where image data is sent for inference, and detection results are returned for real-time rendering on the headset. This hybrid architecture allowed more robust and scalable processing while maintaining responsive, visually intuitive body language feedback.
Accomplishments that we're proud of
We’re proud that Unsubtle stayed grounded in empathy throughout the entire development process. What began as a personal idea grew into a tool that can genuinely help people feel seen and supported. We successfully built a functional prototype that interprets real-time body language and visualizes it in an accessible way—a challenging task both technically and conceptually. Most importantly, we built something with the potential to help people feel more understood, especially those whose communication needs are often overlooked.
What we learned
We learned how powerful technology can be when it’s designed with inclusivity at its core. We deepened our understanding of computer vision, wearable integration, and human-centered design. We also learned how complex body language interpretation truly is, and how important it is to treat these signals as supportive cues rather than definitive diagnoses. Building Unsubtle reinforced the importance of listening to users, designing ethically, and balancing technological ambition with real human needs.
What's next for Unsubtle
Next, we plan to refine the CV model for improved accuracy across diverse body types, movement patterns, and communication styles. We aim to further integrate microgesture outputs with the colocation system, define clearer interaction roles (e.g., teacher vs. learner), and deliver role-specific XR experiences.
We also want to expand support for additional wearables and enhance the XR interface to allow more customizable, context-aware visualizations. In the long term, we plan to collaborate with educators, therapists, and neurodivergent communities to ensure Unsubtle becomes a tool that genuinely serves its users. Ultimately, we envision a future where non-verbal communication barriers are reduced, so everyone can see more than the subtle cues.
Log in or sign up for Devpost to join the conversation.