Swift Shop

An app to scan items and then in augment reality, the details of that product will appear.

Comment 1

Inspiration

We wanted to make shopping _ swifter _. Rather than having to wait to purchase your item at the checkout, you can instead scan the item you want and purchase by using your voice.

What it does

Swift Shopper uses machine learning to identify the various objects for sale and then when it recognises one it'll show a popup with further information of the product .

Below is the basic rundown of the process

Step 1. : Scan what you want. Step 2. : look at the details that've popped up Step 3. : confirm your purchase with your voice!

How we built it

  • IOS App: Swift
  • Content Technology: ARKit,CoreML
  • Machine Learning Model : Keras, Python, AWS, Jupyter,
  • Tested on iPhone 8 plus running iOS 12.0.1

Challenges we ran into

Having never written an iOS app before it was quite hard to get used to the syntax and the design paradigms as well as the integration of AR kit.

Training the neural network was also quite challenging, we had various problems with accuracy and training time as well as having problems with trying to convert the model to a coreML compatible one.

Accomplishments that we're proud of

The combination of multiple cool technologies. Including Machine Learning, ARKit and Speech to Text, the only thing that's missing is its own blockchain ;)

What we learned

That trying to train your own ML model, in 24 hours may not be the best idea. You probably shouldn't try to make an iOS app, if only one of you has a Mac. The basics of IOS app development and how to use ARKIT, CoreML and the IOS speech framework.

What's next for Swift Shop

So much more! The app is just a basic example of the grand idea we have. We'd like to improve the ML model, make the AR tracking much smoother as well as making the voice recognition more accurate. Another feature we'd like to add is social media integration so when you make a purchase everyone knows! We'd also like to add support for "Multiplayer" in that when shopping the business owner can see what items you're looking at and pop over to provide assistance.

Built With

Share this project:

Updates