Smart Portion | Devpost

Dinner plate analyzer
Analyzing a healthy dinner

Inspiration

As students, it can sometimes be difficult to eat healthy consistently and have a nice balanced plate of nutritious food. We thus decided to make it easier for people to analyze and visualize what they can have on their plate, and make adjustments and substitutions virtually to get the updated nutritional information. Our solution is Smart Portion, the AI driven food plate analyzer and suggestion engine.

What it does

Smart Portion uses proven AI techniques, namely K-means and visual object recognition, to analyze the picture of a dinner plate, segment it into the components of the dinner (portions) and then recognize these portions to retrieve their nutritional information. Furthermore, the user can swap portions for some other food alternative and see how that would impact the nutritional value, encouraging new healthier ideas for the next food plate to cook and building better eating habits.

How I built it

In order to process the uploaded dinner plate picture we used the OpenCV 3 computer vision library in C++, running in the background to isolate the food portions. With the plate segmented in portions, we then use the Google Vision API to query each portion as a separate image and get back labels of what type of food the portion would most likely be composed of. then we retrieve the calories for the identified food. The front end consists of a _ node.js _ web app that allows to upload an image, display the portions and query the nutritional information.

Challenges I ran into

Using OpenCV proved to be difficult, first tried the python library before running into issues and having to go to the original C++ implementation of the library. It was tricky to achieve proper segmentation of the plate into portions whille avoiding background noise.Another setback we encountered was the relative inaccuracy of Google Vision API to recognize isolated food items, with the labels being returned often not being precise enough to get nutritional information.

Accomplishments that I'm proud of

The K-means clustering approach to segmenting the plate into portions is the bulwark of this project and required a lot of trial and error along with different image pre-processing techniques to get it right so it is what we are most proud of.

What I learned

This project taught us a lot about the fascinating fields of computer vision and object recognition, none of us was very familiar with complex libraries like OpenCV and figuring them out and being able to use them properly to do what we wanted was very satisfying.

What's next for Smart Portion

The next challenge for Smart Portion is to solve Google Vision API's inaccuracies by training our own food recognition neural network (CNN) as to be able to identify the segmented portion much more accurately. Furthermore, we would like to use a trusted nutritional information API to display more than just calories as well as improve the segmentation into portions.