GlimpseAI

Inspiration

The biggest weakness modern is AI doesn't have any context about our daily lives. With our GlimpseAI glasses, we give AI the power to use audio and visual cues learn and understand from the world around us and provide customized help to enrich our lives.

What it does

GlimpseAI glasses use a powerful combination of audio input paired with TogetherAI's robust LLMs and visual input put together a with custom object classification computer vision model to comprehend what goes on in the user's life. Later, when the user has a question about something that happened earlier (like whether they took their pills, or where they put their keys), the AI can look back through the day and help the user answer their question. GlimpseAI also helps ease the user's life by giving the ability to perform a series of different actions for the user. For example, GlimpseAI's integration with Google Calendar allows it to schedule meetings whenever it recognizes the user wants to schedule one, and its integration with Slack allows it to send out important messages just by vocalizing the thought.

How we built it

For the backend, we used a Flask router to interact with the audio input. It transcribes the user's voice to text using Google Cloud's speech to text model, and then leverages the LLMs from TogetherAI to contextualize what's going on. The backend uses Selenium to integrate with common technologies like Google Calendar and Slack. To give a visualized summary of what the AI is understanding and contextualizing, we also have a Next.js/React frontend using Axios to send requests to the Flask backend. Finally, the vision model uses Pytorch to understand the world in front of us.

Challenges we ran into

There were several challenges with integrating software like Slack and Google Calendar. It was also difficult to train the Computer Vision model. We had a lot of difficulty with the hardware equipment here, however we overcame a lot of the difficulties by coming up with workarounds such as using our laptop mic & attaching an external webcam to create a prototype.

Accomplishments that we're proud of

We're proud of our ability to improvise & change things on the spot to make our product better. We pivoted a lot throughout the hackathon & despite the slow wifi speeds and lack of hardware equipment, we made do with what we had and persevered which we are really proud of.

What's next for GlimpseAI

Next, we will integrate more actions and integrate a more robust vision model to understand more common objects and things. We would create a developer sdk so other people could build more actions upon our work in an app store-like marketplace. Finally, we would try to bring more offline features into the glasses and create a prototype with better hardware so that the glasses could be worn in an everyday setting.

Built With

• ai
• computer
• flask
• javascript
• next
• python
• pytorch
• react
• vision