Accessible Addons

Inspiration

Our inspiration for this project was to focus on assisting conditions that usually get overlooked - Developmental disorders like Autism and ADHD. We wanted to create tools that provide real-time, practical assistance in everyday life, empowering neurodivergent individuals to navigate social situations more comfortably.

What it does

We developed a lens for Snap AR's Spectacles that enables users to detect the emotions of the person they are interacting with. This feature helps users better understand and interpret the facial cues of others, creating easier and more meaningful social interactions.

How we built it

We utilized the 3d modelling capabilities of Lens Studio along with the scripting capabilities we could add to objects. In order to differentiate the facial expressions, we trained a Machine Learning algorithm using Tensorflow Lite technology. The training for this model used over 32,000 images. We bounded a halo effect along with text displaying the emotion using facial mesh technology

Challenges we ran into

We ran into numerous problems attempting to import our Machine Learning models into Lens Studio. There was a large struggle getting used to the new development environment and hardware as neither of us had used any AR technologies before this. Being on the bleeding edge of this new technology there was a large lack of documentation compared to older Technlogies, causing us to struggle to make meaningful forward progress with our project.

Accomplishments that we're proud of

Successfully created an emotion detection machine learning algorithm and created AR elements that display what emotion is detected. Developed on the new SnapAR Spectacles!

What we learned

We learned a lot about AR development and how much harder it is to develop when on the bleeding edge of technology. We had many issues trying to get, what we though were simple things, working to within the degree we expected. There was a large learning curve overall with this project for both of us.

What's next for Accessibility Addons

Develop even more accessibility features including ADHD Filters. Blurring out things outside of your focus zone to help maintain your focus. A Speach to text setting to allow those hard of hearing to be able read and maintain normal conversation, even more emotions and more polished Face mapping for our currently existing Neurodivergence assistant, and a Dyslexia filter to overlay onto reading that might otherwise be difficult to decipher.

Built With

• javascript
• machine-learning
• python
• snapar
• tensorflow
• typescript