RizzVision | Devpost

Inspiration

Have you ever walked into a room, stepped into a meeting, or even gone on a date, only to feel completely out of place? It's like everything is going wrong—you say the wrong things, thinking you’ve got it under control, but in reality, you don’t. You’re left wondering what went wrong and wishing you had known what to do differently. Well, we have gotten the fix for you.

What it does

We first open our mobile app, which contains settings and a profile of your conversations and uses. Using a camera and microphone, we capture real-time data from your conversation. This includes live audio transcription, capturing spoken words, and performing semantic analysis on facial expressions. All this data is stored in a conversational database. During the conversation, our state-of-the-art AI inference system analyzes the sentences spoken, providing a score and a brief explanation for it. Additionally, it offers live suggestions for responses, leveraging past context to craft the next optimal question or statement. At the end of the conversation, we format the conversation data similar to a chess game analysis, providing an overall summary overview. This includes key insights on what went well and areas for improvement, with specific pointers highlighting the strengths and missteps throughout the conversation. All of this is comfortably displayed and centralized all in the portability of your phone inside the mobile app.

It also notes of the blunder you made when you asked them if they liked Jazz.

How we built it

Input Processing: We utilized a Raspberry Pi as the central hub for input processing. This compact yet powerful device efficiently connects to various capturing devices (likely including a camera and microphone) as well as output devices.

AI Inference System: The core of our solution is an external service running a constant AI inference system. We leveraged Langchain as our framework and Groq for high-performance AI computations. At the heart of this system, we implemented the Mistral 7B language model, known for its efficiency and strong natural language understanding capabilities.

Mobile Application: To make our solution accessible and user-friendly, we developed a mobile app using React Native. This cross-platform approach allows us to reach both iOS and Android users with a single codebase.

Data Management: All conversational data is stored and managed using MongoDB, a flexible and scalable NoSQL database. This allows us to efficiently handle the real-time nature of our application while maintaining a robust data structure for analysis and retrieval.

This architecture allows us to capture real-time audio and visual data, process it through our AI system for instant analysis and suggestions, and present the results to users through an intuitive mobile interface. The combination of edge computing on the Raspberry Pi and cloud-based AI processing enables us to balance performance, latency, and scalability effectively.

Challenges we ran into

Due to the use of large language models (LLMs), predictions can occasionally be noisy, leading to inaccuracies or "hallucinations" in determining the correct score or analysis. This can affect the precision of the real-time feedback and suggestions. Additionally, the process of integrating and migrating software to hardware is a significant challenge. Hardware, in general, presents numerous issues, from linking devices to ensuring seamless communication between inputs/outputs and the service. Establishing stable connections, handling device compatibility, and ensuring real-time processing all contributed to the primary hurdles we faced in the development of this product. Overcoming these technical obstacles was crucial in shaping the final solution.

Accomplishments that we're proud of

We take pride in the challenge we set for ourselves, pushing through an intensive process to bring this unique idea to life. Special thanks to Eddy for staying awake for over 24 hours and helping us push through when it mattered most.

What we learned

Hardware is not easy. Software isn't either, but integrating and migrating these 2 worlds is really really not easy.

It's also not about the outcome but the friends we made along the wa-

What's next for RizzVision

Although we initially developed this tool to help with the thousands of struggling, desperate students at Waterloo who may never find a date, the potential for expansion on this project is limitless. This analysis-based tool can be applied to a wide range of scenarios, from analyzing conversations in business meetings to evaluating how you interact with an employer during a job interview. Sentiment analysis and live suggestions can be valuable across various aspects of life, particularly in the career of an engineer.