Check it out! | Devpost

Deep learning grocery detection
Our rig, in the wild
MobileNetV2 Architecture used
3D CAD model of our ]rig

The story started with an observation

Americans spend $750 billion on grocery shopping per year. However, checking out fresh produce requires manual look-up through a messy database, which is inefficient for the consumer and costly for the business. A 2018 study of 1000 grocery shoppers found that 74% of shoppers were dissatisfied with the length of lines, and a smooth checkout experience was voted as one of the top 3 most important factors.

Despite this, existing self-checkout systems are confusing and unintuitive, leading to a frustrating user experience and requiring employee oversight. Slapping a barcode sticker on 100,000 items and paying cashiers to work 10 check-out lanes requires lots of tedious manual work and wasteful packaging. Is there a better way?

Our computer vision-powered solution

demo_gif We identified a way to streamline the checkout experience that speeds up lines while reducing man-hours for grocery stores and chains. Our idea is simple but transformative. Using state-of-the-art computer vision object detectors, we automatically scan and classify groceries without the need for manual look-up. Our system is modular and can be easily integrated into existing grocery stores of any size, expediting the checkout experience for the customer while saving costs for businesses. Moreover, with the way modern packaging and transport have been optimized, significantly greater waste has opted for ease of checkout. Using ML to detect groceries, we can reduce the need for wasteful, durable plastic with more recyclable as-you-need bags.

Features

Modular, easy integration into existing check-out systems. ✅
Lightweight classifier and detector using MobileNetV2, with robust accuracy. ✅
Increased security in self-checkout lanes with an additional overhead camera. ✅

Who we benefit

A smoother checkout experience for shoppers ✅
Easy, lightweight system for small and chain stores ✅
Reduced cost, manpower, and increased customer satisfaction for grocery chains ✅

How we built it

detection Our system comprises of three main components:

A checkout counter with an integrated scale.
An overhead camera.
A processing unit performing automatic detection and classification of grocery items

To showcase our idea, we used a combination of hardware and software to build a small demo, involving everything from neural network training to laser cutting and assembling the CAD design.

Deep learning back-end: We compiled a selection of images from publicly available datasets across 5 common food categories with 1000 images each. Then, we used transfer learning on the MobileNetV2 architecture pre-trained on ImageNet and added our own output classification layer. Our classification model trained on our custom dataset reached 99.85% accuracy. For object detection, we fine-tuned a pre-trained model from TensorFlow Hub on select images from the COCO2017 dataset. Again, we chose a MobileNetV2 backbone for lightweight inference and used it with the single-shot detector model(SSD). We deployed our models in Flask to create a REST-API, and build our demo application around it. The demo application was built with Bootstrap, jQuery, Ajax, HTML5, and CSS.

Building the rig: We drove our design based on physical limitations as well as software necessities. Due to the limited and fast-paced nature of this event, we didn’t have many options like we would in a conventional engineering design cycle. Looking at the design problem we found and its necessities, we discovered the following primary design conditions:

Watermelon length = 16in
Color contrast = full scale
Weight condition = 10lbs
Weight precisions = .5 grams
“Camera” weight = 120 grams
Viewpoint = vertical

From here we created a mounting stand to achieve the necessary loading condition for our camera as well as meet the aforementioned design conditions. Calculating the view range of our camera, we used the specified 7mm focal length from the Galaxy S21 combined with a 16in diameter necessary to allow our 144-degree FOV to completely encapsulate the watermelon at an 18-inch height. To be able to have a circulate measurement area, our camera thus required a horizontal offset leading to the shelf style of the mount. Using Solidworks simulation to discover the stability of the system, we propagated the connections down and reduced side component material volume. rig

What's next for Check it out!

We would like to build a full-scale version of our integrated system, with a more comprehensive deep learning model. Firstly, we would like to propose our design to our beloved university store CharMar, that's right, Charles Street Market. Charmar, you've been selling us potatoes and bananas at price per count, instead of price by weight. As a result, the smaller produce is sitting and rotting on the shelf, leading to food waste and student complaints. It's okay, we get it, an integrated checkout system that weighs the groceries is a pain. But boy do we have the solution for you. We literally will make you a checkout register.