contextAR

Enhance nonverbal communication through ambient mood cues - helping others understand when someone is open to interaction or needs space

Comment

Inspiration

We were inspired to create this project after having a first hour of getting to know other developers at the hackathon. We noticed many hackers were a little quiet in the morning, and slowly after having breakfast, they started to open up more. Ideally, we started talking to hackers throughout the hackathon, but talking to them as they walked through the door didn't exactly seem like the best move, at least when understanding their morning perspective. So, we wanted to revolutionize how people communicate with each other, but in an appropriate environment, making sure that two people are in the mood to and are available to speak with each other, in AR!

What it does

When users enter the app, an instance of the user is instantiated, and we invite users to speak into the Spectacles Mic to describe their mood, availability, and desire to talk to others. Based on their voice input we transcribe this and send the transcription to our AI, which decides if the user's status is one of four colors: Red representing angry, Blue representing being focused and locked in (DND mode), Green representing an eagerness to meet others and be approachable, and Yellow representing a weaker desire to communicate, potentially causing irritation if approached. Once the AI chooses the best color based on their voice input, a circular colored status is shown above their heads on a box, allowing other users in the environment to see their status. This truly helps people know who to talk to and who not to bother.

How we built it

We built our project using Spectacles, Lens Studio, and OpenAI Whisper (tried to use the VoiceLLM module but had to improvise).

Challenges we ran into

Figuring out how to set up the VoiceLLM package in Lens Studio, allowing users’ speech to be transcribed and sent to our AI. Learning how to set up our objects underneath the head binding, allowing them to move relative to the user.

What we learned

We were able to dive deep into Lens Studio's capabilities and become much more comfortable and familiar with its numerous features. Working with this new technology has allowed us to build a good foundation of knowledge that we will continue to build and expand upon.

Built With

Share this project:

Updates