CookQuest

• GIF

Inspiration

We’ve all been there: you're hungry, you've got ingredients in the house, but... you have no idea what to make. Maybe you've got some eggs, rice, or veggies sitting around, but nothing sounds right. That's where CookQuest comes in. With just a screenshot of the ingredients you have, taken right from your Meta Quest, you can let the AI work its magic. It’ll analyze what you’ve got and recommend a dish you can make, complete with a detailed recipe and step-by-step instructions. No more guessing games in the kitchen. CookQuest makes cooking easier and more fun, turning your ingredients into meals with just a snap.

What it does

Simply lay out the ingredients you already have, take a quick screenshot with the Meta Quest, and let the AI assistant recognize them. In seconds, it recommends recipes you can make, tailored to what’s in front of you! If you don’t like a suggestion, you can give it a quick thumbs down, and the AI adjusts its idea to better fit your tastes. Once you’ve chosen a recipe, CookQuest guides you step by step with text to speech .

How we built it

CookQuest was built using a blend of AI, VR, and design technologies, combining powerful backend intelligence with an immersive front-end experience:

• Backend (Python + FastAPI) → Handles user sessions, recipe generation, and AI interactions through asynchronous endpoints, allowing image analysis, reasoning, and voice generation to run in parallel for smooth performance. The backend is hosted on DigitalOcean, ensuring reliable cloud deployment, scalable API performance, and secure data handling

• Google Gemini API → Powers the core intelligence of CookQuest. Using its multimodal reasoning, it identifies ingredients from Meta Quest camera input and generates adaptive, step-by-step recipes, blending vision and language tasks into one cohesive workflow.

• ElevenLabs Voice AI→ Brings the digital chef to life with dynamic speech generation, turning static text instructions into a hands-free conversational experience that feels natural and engaging.

• Unity 2023 LTS + XR Interaction Toolkit + Meta SDK→ Creates the immersive VR kitchen, where users can interact with ingredients, view generated recipes, and follow real-time AI guidance within a spatial environment.

• AR Foundation + Quest 3 Passthrough → Enables ingredient recognition via live camera feed or simulated detection for demos, bridging real-world and virtual cooking.

• Figma UI Design→ Ensures a clean, intuitive interface that prioritizes clarity, immersion, and flow between real and virtual experiences. Aided in creation of our pitch slide deck as well as the product logo.

Challenges we ran into

• Connecting the frontend (Unity) with the Python/FastAPI backend

• Retrieving data from the backend and converting the voice into an MP3 using ElevenLabs

• None of the team members had experience with Meta or Unity — learning XR development was challenging due to limited documentation

• Wi-Fi connection issues when trying to connect to the Meta Quest

• Losing nearly ¾ of our time to long Unity build times

• We had limited API credits, which restricted us to only two simultaneous API calls — a constraint that slowed down performance.

Accomplishments that we're proud of

We successfully integrated Google Gemini, ElevenLabs, and Unity to build a fully functional VR cooking assistant . We’re proud to have combined AI, VR, and voice technology into one cohesive and immersive experience.

What we learned

• Gained hands-on experience with Unity and XR development

• Improved time management in response to long build times and connectivity issues

• Learned advanced API integration and asynchronous workflows for smooth, real-time performance

What's next for CookQuest

• Add a menu for users to select an avatar and customize its voice.

• Make the overall UI design for a more intuitive, visually engaging, and accessible cooking experience.

• A more robust user feedback loop to continuously improve AI recommendations and guidance quality

• Enhanced voice interactions, allowing users to chat naturally with the AI chef, ask questions, get tips, and adjust recipes in real time

• Add a built-in wrist timer feature for convenience and precise control during hands-on cooking sessions

• Collaborative multi-agent system

• With additional credits, we can significantly improve processing speed and responsiveness across tasks like image analysis and recipe generation

Built With

• ar
• digitalocean
• elevenlabs
• fastapi
• figma
• gemini
• geminiapi
• github
• metaquest
• metasdk
• python
• unity
• vr