LetMeCook

Snap your fridge, discover a recipe. A computer vision mobile app that helps you find personalized recipes based on your ingredients and dietary needs.

Comment 2

Inspiration

Everyone loves to eat. But whether you’re a college student, a fitness enthusiast trying to supplement your gains, or have dietary restrictions, it can be hard to come up with meal ideas. LetMeCook is an innovative computer vision-powered web application that combines a scan of a user’s fridge or cupboard with dietary needs to generate personalized recipes based on the ingredients they have.

What it does

When opening LetMeCook, users are first prompted to take an image of their fridge or cupboard. After this, the taken image is sent to a backend server where it is entered into an object segmentation and image classification machine-learning algorithm to classify the food items being seen. Next, the app sends this data to the Edamam API, which then returns comprehensive nutritional facts for each ingredient. After this, users are presented with an option to add custom dietary needs or go directly to the recipe page. When adding dietary needs, users fill out a questionnaire regarding allergies, dietary preferences (such as vegetarian or vegan), or specific nutritional goals (like high-protein or low-carb). They are also prompted to select a meal type (like breakfast or dinner), time-to-prepare limit, and tools available for preparation (like microwave or stove). Next, the dietary criteria, classified ingredients, and corresponding nutritional facts are sent to the OpenAI API, and a personalized recipe is generated to match the user's needs. Finally, LetMeCook displays the recipe and step-by-step instructions for preparation onscreen. If users are unsatisfied with the recipe, they can add a comment and generate a new recipe.

How we built it

The frontend was designed using React with Tailwind for styling. This was done to allow the UI to be dynamic and adjust seamlessly regardless of varying devices. A component library called Radix-UI was used for prefabricating components and Lucide was used for icon components. To use the device's local camera in the app, a library called react-dashcam was utilized. To edit the photos, a library called react-image-crop was used. After the initial image and dietary restrictions are entered, the image is encoded to base64 and entered as a parameter in an HTTP request to the backend server. The backend server is hosted using ngrok and passes the received image to the Google Cloud Vision API. A response containing the classified ingredients is then passed to the Edamam API where nutritional facts are stored about each respective ingredient. All of the information gathered until this point (ingredients, nutritional facts, dietary needs) is then passed to the OpenAI API where a custom recipe is generated and returned. Finally, a response containing the meal name, ingredients, step-by-step instructions for preparation, and nutritional information is returned to the interface and displayed onscreen.

Challenges we ran into

One of the biggest challenges we ran into was creating the model to accurately and rapidly classify the objects in the taken picture. Because we were trying to classify multiple objects from the same image, we sought to create an object segmentation and classification model, but this required hardware capabilities incompatible with our laptops. As a result, we had to switch to using Google Cloud's Vision API, which would allow us to perform the same data extraction necessary. Additionally, we ran into many issues when working on the frontend and allowing it to be responsive regardless of device type, size, or orientation. Finally, we had to troubleshoot the sequence of HTTP communication between the interface and the backend server for specific data types and formatting.

Accomplishments that we're proud of

We are proud to have recognized a very prevalent problem around us and engineered a seamless and powerful tool to solve it. We all enjoyed the bittersweet experience of discovering bugs, editing troublesome code, and staying up overnight working to overcome the various challenges we faced. Additionally, we are proud to have learned many new tools and technologies to create a successful mobile application. Ultimately, our efforts and determination culminated in an innovative, functional product we are all very proud of and excited to present. Lastly, we are proud to have created a product that could reduce food waste and revolutionize the home cooking space around the world.

What we learned

First and foremost, we've learned the profound impact that technology can have on simplifying everyday challenges. In researching the problem, we learned how pervasive the problem of "What to make?" is in home cooking around the world. It can be painstakingly difficult to make home-cooked meals with limited ingredients and numerous dietary criteria. However, we also discovered how effective intelligent-recipe generation can be when paired with computer vision and user-entered dietary needs. Finally, the hackathon motivated us to learn a lot about the technologies we worked with - whether it be new errors or desired functions, new ideas and strategies had to be employed to make the solution work.

What's next for LetMeCook

There is much potential for LetMeCook's functionality and interfacing. First, the ability to take photos of multiple food storages will be implemented. Additionally, we will add the ability to manually edit ingredients after scanning, such as removing detected ingredients or adding new ingredients. A feature allowing users to generate more detailed recipes with currently unavailable ingredients would also be useful for users willing to go to a grocery store. Overall, there are many improvements that could be made to elevate LetMeCook's overall functionality.

Built With

+ 1 more
Share this project:

Updates