theARcade

Inspiration

AR experiences are great, but most of them require you to have a specific device (Android, iOS, Magic Leap.) TheARcade aims to build an online web-based platform for people to stream AR experiences and connect with friends, playing games where everyone can join in on the fun based on open source software.

What it does

I built three games highlighting the different range of AR experiences. Single player, playing with friends in person, and playing with friends online. They are also twists on classic games, so that people can learn them quickly and focus on having fun. As a bonus, they involve moving your body, so people are getting healthy while they are having fun!

How I built it

I built it on top of TensorFlow.js and several out of the box pre trained models, person segmentation and pose inference. This enables Machine Learning inference to process data to pipe to my JavaScript frontend which renders the games for players. Since this runs in the JavaScript engine available on any device with a web browser, you keep all your data securely on your device, and you can have a truly cross platform experience based on the web.

Challenges I ran into

Trying to build a lot of projects at once. Also intense array manipulation dealing with large matrices. There are challenges supporting multiple platforms, and designing game mechanics that are intuitive, fun, the right level of challenge, and available on different devices with different screen resolutions, camera resolutions, and processing power.

Accomplishments that I'm proud of

Flappy Fly - Very interesting to work out the mechanics of flappy bird. It seems so simple, yet there is a lot to it Tetwist - This one especially! Involved a lot of math, and I think I got it working really well despite there being more room for improvements algorithm wise, and a lot of more features possible MoonJump - Got a minimum version this Monday and all of the frontend ready to plug it into a pub/sub or websockets server so that everyone can play together. I also started working on a multiple people on one device logic.

What I learned

I got to apply some Linear Algebra heavily during Tetwist. Also got a better feel for the human segmentation model. It was interesting balancing the challenges of Flappier. I definitely wish I spent more time on MoonJump, because when I told people about the concept it seemed to resonate the most with people. Although it was challenging doing the planning, coding, design, and deployment all by myself I think being able to show communication online between devices would have really sold the benefits of 5G and gotten people excited about the future of multiplayer AR.

What's next for theARcade

More integrations! People should be able to log in, share easily with their friends, and play more interactive experiences online. There can be so many optimizations to make the games smoother, and the graphics better. Also there can be a huge improvement by training custom models for the particular games instead of using generic human estimation models, since they are trained for people doing ordinary things and not jumping around, flapping their arms like a maniac, and posing like tetris blocks.

I also would like to make more optimizations for mobile responsiveness, only Tetwist is heavily optimized for that.

Obviously cleaning up the code and possibly open sourcing the whole thing.

Also, I would like to put in analytics since it is available online.

Built With

• tensorflow.js