mrshopper | Devpost

Real world view with 3D grid for profile elements
HoloLens view for profile inside Unity playground
HoloLens view for products inside Unity playground
HoloLens view for recipes inside Unity playground
Real world view with 3D grid for nutrition and ingredient information
Real world view with 3D grid for recipe selection

Inspiration

These are peculiar times. The world is plagued by the coronavirus pandemic and there is a huge uptick in the demand for remote activities. While this sounds uber comfortable, it can also be detrimental to one's social, physical and mental health. One also has to consider the fact that more often than not, walking to the grocery store is often a person's only "fitness" incentive during the day. Take this away and we aren't left with a lot of activities to keep up our physical fitness. Keeping this in mind, we want to present an end to end solution that marries something as banal as grocery shopping with fitness challenges and nurtures users towards healthy and more responsible choices.

When you think of online shopping, what is the first thing that comes to your mind? Let me take a guess. You are probably imagining a person endlessly through products, perusing, questioning life decisions, asking themselves what they are doing with their lives. Wondering how life might have been if I had said "Yes" to that 1 month South America travel escapade my friend had invited me to in February before the travel bans. Err...I digress. You get the gist. Online shopping is comfortable, very very comfortable, but it isn't necessarily a lot of fun. And then you have online grocery shopping, which in my opinion, is one of the most tedious things to do. Scrolling through a plethora of products without any structure or meaning does not appeal to me. Tell me, what sends your stomach into a frenzy? Looking at onions, garlic and chicken or a picture of freshly cooked butter chicken curry?

Good looking food usually comes with a lot of fitness caveats. And then there is your diet plan. You really want to eat that butter chicken but how exactly do you fit it in your plan? What about your allergies? Can you still eat it on a Keto diet? These are some of the problems we aim to tackle with our solution. Great, the food is sorted but what about exercising now? Based on a study by Barte and several others, we know that there is a strong relationship between exercising and incentives. But how can one be incentivised to exercise during online grocery shopping? This is where we took the concept of "Adding to Cart" quite literally. If Pokemon Go was any indication, most people don't mind doing physical tasks to gain rewards, financial or not. Similarly, using the power of Mixed Reality devices such as HoloLens, a user can be incentvised to perform fitness tasks within the confines of their home to gain rewards such as discount vouchers and bonus coupons while accomplishing the goal of shopping conveniently and safely.

What it does

Reimagines online shopping by harnessing mixed reality
Integrates fitness training with shopping and gamifies the experience of monotonous buying
Remote shopping and exercising, keeping you safe from the clutches of Coronavirus!
Enables you to search and retrieve recipes using images of the delicious food images you see in the web with Computer Vision algorithms.
Empowers you to shop based on what you want to eat instead of forcing you to think what to eat based on what you have shopped.
Aids in making better food choices for your health and also the environment
Personalised recommendation of products based on your dietary preferences and allergies
Connects to your fitness devices and leverages this data to improve your nutrition choices
Rewards you for better choices

How I built it

We used Microsoft's HoloLens 2 as our choice of platform to present our solution.
For the datalake, we used data from vendors such as Migros, Eaternity. byte.ai, Eatfit and OpenFoodFacts.
We used Azure to deploy our backend

Challenges I ran into

The major challenges came in the form of leveraging the humongous amounts of information at our disposal through a multitude of APIs and filtering them to use the most relevant data points. This often lead to slowdowns and demanded novel ways of optimising data retrieval techniques. Another major hurdle was to build 3D elements for the HoloLens and render it in a user-friendly manner. Surprisingly, the most obvious challenge of collaborating within ourselves (since one of us was working remotely) didn't turn out to be as difficult as one would imagine. Hurrah to the wonders of modern technology!