AutoStore | Devpost

Inspiration

Checking out of a store can be a difficult task when there are long line-ups, and when few cashiers are available to operate their desk... but self-checkout is a scary task when never done before

What it does

AutoStore is responsible for making checking out a lot quicker and easier, by no longer requiring the need of a cashier or barcodes. The items are immediately identified when taken out of a shopping cart, and added to your bill. Its as simple as taking your item from your shopping cart and into your bags to take home!

How we built it

I built this application by first creating a flask based web application user interface (UI), which is responsible for displaying a video stream of the shopping cart. When an item is taken out, using OpenCV, the shopping cart is segmented before and after the exit of the item, which are then subtracted to return a contour difference. The largest contour then specifies where the item was taken. With the item out, the rest of the shopping cart image is cropped to zoom into only the IMPORTANT section of the image: the location of the taken item. This cropped image is then fed into a pretrained ResNet model, which is used to identify the item taken. Once the item was found, it was simply added to the receipt on the UI!

Picture -> Classify -> Receipt! As simple as that!

Challenges we ran into

The segmentation proved to be a challenge as the colours were incredibly sensitive, and for some reason, when an image was taken out, the section was never clearly differentiated. Furthermore, when it came to the pretrained ResNet model, I had to create my own quick and simple dataset to perform transfer learning. Unfortunately, due to having such little time, I was only able to create a dataset for 3 items, and I did not have nearly as many images to be incredibly accurate. Lastly, the Flask application proved to be tedious when creating a shopping cart frequently asked for a local database, but that was then no longer needed when a rather strange solution was implemented to create a global array to store the previously added items.

Accomplishments that we're proud of

I am proud for having been able to complete by first entirely software oriented McHacks! With a mix of tools I have used in my years at school, notably when I worked on my final engineering project, I found myself ready to get cracking! Lastly, this proved to work!

What we learned

I learned to segment images and take the PROPER X-Y COORDINATES! Silly me reversed them and didn't know what was going on. Furthermore, I also learned to enjoy the time put into the work, and take my time, as I realized that I found out numerous of my solutions when I took a break! I have also learned that everybody at McHacks is very cool!!

What's next for AutoStore

Next in store for AutoStore is to implement an automated image capture functionality to avoid the need of manually selecting an image capture button.
This would fully automate the entire experience from start to finish. Next, more data must be found and applied to a pretrained model to better classify images.
Furthermore, the web application needs some.... "work"... It isn't quire the prettiest and doesn't allow to remove specific items from the cart.
The image segmentation can be drastically improved to acquire better readings in the differences, such that better cropping can be performed from more precise and larger contours.