🚀 SAVMA - Simplifying App Navigation with AI 🧠

🌠 Inspiration

Our team member Nina, had a frustrating experience with her grandma on Facetime: an app designed to be simplistic. In her own words: “I was talking to a trash can the whole time”.

Even with Nina giving instructions, her grandma could not identify where the flip camera button was. This resulted in an hour long call with trash and recyclables. This got our group thinking about the accessibility of mobile apps. Everyone can download an app, but can everyone truly use it?

⚙️ How it works

Enter SAVMA, an AI assistant that senses user confusion and provides real-time, in-app guidance. When users struggle, SAVMA triggers a voice assistant 🗣️ that guides them by highlighting relevant UI elements. For developers, SAVMA can comprehend and instruct on the relationship between various UI elements, offering a smoother user experience. 📱

🏗️ How we built it

• Hume facial detection model for emotion recognition 😕
• Flask framework to host backend and manage REST API 🐍
• Swift for iOS development and hosting our in-app voice assistant 🍎

🚧 Challenges

We faced numerous challenges:

• There were no precedents for integrating Hume with iOS apps, so we engineered it ourselves 🧩
• Three of our four team members were new to Swift and iOS development 📚
• Numerous API calls and intermittent internet issues slowed our debugging process 🐌

💪 Accomplishments

Despite the obstacles, we're proud to have created the first use case of the Hume API with iOS apps. This breakthrough is a result of intense learning, experimentation, and dedication from our team. 🏆

🎓 What we learned

We acquired several skills:

• Creating Swift UI and triggering the Swift voice assistant 🎨
• Sending API requests from Python backend to Swift ⛓️
• Quickly understanding documentation to effectively use GuardRails and Langchain 📖
• Analyzing technologies like MindsDB and Vocode to select the best for our use case 🔍

🔜 What's next for SAVMA

Post-hackathon, we'll integrate SAVMA into FaceTime to ensure no one talks to trash cans again! 🗑️

Our ultimate goal, however, is to standardize SAVMA as an add-on service for any app. We plan to open-source it, inviting all Swift developers to enhance SAVMA's capabilities. 🌐

👨‍💻 Built With

• Backend: Python and Flask Framework 🐍
• Open-Source Libraries: LangChain, Guardrails 📚
• APIs: Hume AI, OpenAI 🤖
• Mobile App: Swift 🍎

Built With

• flask
• guardrails
• hume
• langchain
• python
• swift