Show Me The Money

VR gameshow to teach you how much you really spend

Comment

Inspiration

Ever wondered how much you spent on Tims? McDonalds? Groceries? Felt you were spending too little when your wallet says otherwise? Felt #girlmath was a great option?

Moneylingo to the rescue!

Members of our GenZ team are shopaholics, guilty of money burning a hole in their pocket. So we had an idea to create a hyper personal assistant that we can interact with and help visualize our spending, tangibly in the VR space. This helps emphasize the expenses associated with everyday life and provide an incentive to begin your financial literacy journey.

In the future, the same VR application can also be fine-tuned to target persons with disabilities who may face issues travelling to a physical branch or people with lower tech literacy who would prefer a physical interaction over an app or a virtual chatbot. It can also help reduce the load on existing bank tellers, freeing them to handle the tasks that really matter.

What it does

  • Teach financial literacy in a tangible way and confront you with your spending habits
  • Provide access to your bank features in an accessible and inclusive way
  • Improve productivity of bank branches

Show Me the Money is a game designed to confront you with your spending habits in a way you absolutely cannot ignore! It is a VR game in which your financial assistant will ask you to report on your perceived spending. It then compares this reported quantity to the actual amount spent based on real financial data, and manifests a physical representation of your spending in a way that is easy to understand and assess.

How we built it

The frontend is a Unity VR app, using the Oculus SDK for VR integration with the Quest 2 and OpenAI’s Whisper running locally for dictation recognition. The user is presented with a realistic virtual environment and a “personal” banking advisor (duolingo free 3d model google search) and input is naturally taken using speech recognition.

The backend was developed using a combination of Python, Google Cloud, and a Firebase database. The database contains a manual export of real financial data from one of our group mates, to be replaced with a bank API integration for future iterations. It also connects to the ChatGPT 3.5 Turbo model to generate realistic dialogs for the game.

Challenges we ran into

We initially thought about using Wit.ai for voice detection, but setup was too clunky and hard for us to do in 36 hours. We couldn’t use Windows’ built in voice recognition properly either, but luckily there was an implementation of Whisper for Unity that worked out of the box.

While dropping coins and food items, we ran into frame drops caused by having to calculate so many physics collisions. Tweaking the physics timestep helped a lot, but the main performance gains were from deactivating physics on an object after a few seconds, to prevent a “settling” effect having to recalculate collisions on many objects.

We were also planning to integrate the game with Google DialogFlow to make the communication more open ended and immersive. However we faced challenges in implementing the Webhook functionality of DialogFlow that would fetch dynamic information like spending details or LLM output from the internet.

Accomplishments that we're proud of

We’re proud of how immersive the VR bank space looked (green owl aside). We’re also very proud of the various integrations we were able to include within our application.

We even managed to get some decent sleep when all was said and done!

What we learned

On the game side, we learned that OpenAI made Whisper open source and runnable locally. We also learned how to create more “immersive” scenes and how to optimize collision physics for thousands of items. We also learnt how to engineer prompts for the LLMs and DialogFlow.

What's next for Show Me The Money

This app concept could be expanded to include more impactful visualizations of spending, as well as expanded to include banking and financial planning services such as setting budgets.

Built With

Share this project:

Updates