UMacro

Snap, Scan, Savor! Combining AI photo recognition & UMass Dining data, our app uses a quick photo to give students instant calorie/nutrient info for their meal, taking the guesswork out of nutrition!

Comment

Inspiration

Studies show that individuals underestimate portion sizes by 20-50%, particularly with high-calorie foods, leading to significant inaccuracies in caloric and macronutrient intake. Additionally, only 30% of Americans correctly identify their daily caloric needs, making it harder to achieve health goals. At UMass, where busy student lifestyles often lead to inconsistent eating habits, this issue is even more pronounced. We wanted to solve this problem by leveraging AI to make nutritional tracking effortless and precise. UMacro was inspired by the need to bridge the gap between visual estimations and accurate nutritional data for UMass students.

What it does

Students take a photo of their meal, and the app uses AI to identify the dish and its ingredients. It then provides a detailed breakdown of macronutrients and calories, tailored to UMass Dining options.

How we built it

We used the OpenAI API to classify dishes and food items from the input image, which then accesses our MongoDB database. This database is on UMass dining data collected via web-scraping using BeautifulSoup. Using MongoDB atlas vector search we extract the nutritional information and present our users with them. We used FastAPI for the backend and we deployed our server in a Docker container on fly.io. Additionally, we used Figma to help us design and prototype the UI/UX. We went to the UMass dining halls to take pictures of the food items to help train/validate/test our model

Challenges we ran into

  1. Initially, we were using a computer vision ML classification algorithm known as Yolo v4 (object detection). The major challenge was training the model to be accurate because we didn't have a large enough dataset of UMass dining food images.

  2. Another major challenge was not only identifying individual dishes but also estimating their depth and quantity to provide precise serving sizes. Although our model was able to function, we needed more data and integrating this complex real-time analysis and optimizing MongoDB’s vector database for fast, accurate data retrieval posed significant technical hurdles.

Accomplishments that we're proud of

We came into this hackathon as friends, primarily looking to have fun and gain some experience, and for many of us, it was our first hackathon. From brainstorming problems and discussing potential solutions to diving into the technical components, the entire journey was incredibly fulfilling!

One of the highlights was tackling a real problem we've personally faced: underestimating calories and struggling to get our macros right. Our team members are all very passionate about nutrition health and working out, so seeing our idea transform into a working product that could genuinely help students at UMass was an amazing accomplishment. Along the way, we learned new technologies through the HackUmass workshops too like FastAPI, MongoDB’s vector search, and the OpenAI API, which we in-fact ended up using!

What we learned

Through the openAI developer platform, we gained hands-on experience with the OpenAI API, learning how to integrate this into our idea.

We also learned how to use Figma to design and prototype our UI/UX, which we didn't have any experience with before.

We learned how to create a database using MongoDB to store the embeddings and we were able to use their feature atlas vector search to extract nutrition information from our database.

What's next for UMacro

Portion Size Analysis: Train a custom ML model to analyze the specific portion size of each food item, providing more accurate calorie and macronutrient data.

Personalized Dietary Recommendations: Offer tailored suggestions based on users’ dietary goals, preferences, and past meal history.

Share this project:

Updates