Shapeshifters

Online Multiplayer, Mixed Reality Pose Game Inspired by Hole in the Wall! Optimised for low latency and developed with raw-networking protocols. Come and try, upstairs at Table 23!

Comment

Inspiration

Shapeshifters is a mixed reality, online networked game similar to the TV Show 'Hole in the Wall' - where players have to hold poses to fit through the wall. We wanted to promote stretching, and at home exercise in a sociable way without the need for specialist hardware, only WiFi and a webcam! You can think of it similarly to an online Wii/Kinect game.

We use a dedicated raw-UDP client to talk to the computer vision, from the game with a separate, azure-networked endpoint for storing game data. We also have a raw-TCP relay server to provide low latency multiplayer, which acts as a low latency proxy. We chose low-level data transmission to minimise latency and increase gameplay fluidity & responsiveness.

What it Does

Computer Vision: Shapeshifters uses MoveNet to calculate human pose estimates from the webcam. MoveNet initially returns a list of positions on the body, however we decided to convert these to angles to ensure that people of different heights and dimensions are all portrayed similarly.

Game Design: Shapeshifters was created in GameMakerStudio 2. All assets and music were made by us! Learning Garageband was especially fun, and also designing interesting poses - call us a bunch of yogis?

Networking: Angles retrieved from the computer vision server are sent to the GameMaker client over a raw (P)UDP. connection. A raw-TCP Relay connection was established between an Azure server and GameMaker clients to constantly transfer pose data across clients. This ensures you can see your opponent's live pose on the screen, at a low latency. Furthermore, a second TCP Server was established to transfer extra information about the game, such as game states, accuracies & score!

How We Built It

We split the project into three main components, and assigned each member a component: Game Design & Pose Rendering (Toby), Networking (Eden), and Computer Vision (Amber).

Accomplishments That We're Proud Of

  • Learned LOTS about Networking, Latency and low-level data transmission.
  • Actually had lots of fun, up until the hours of 2am - 6am (purgatory)
  • Learned the value of HCI - especially in mixed reality applications.
  • We decided to do this at the start, and we actually did it? (goat?).
  • Online Multiplayer works with CV clients being represented with low latency, and accurately!
  • Game Design actually turned out okay.
  • Music was good (especially as we hadn't done it before!)
  • Computer Vision works well, and translates to the game at a very low latency.

Challenges we Ran Into

Networking: Eduroam! We also made several attempts to establish a REST API between the GameMaker client and Azure server. However, due to limitations of Azure, this was not possible. Therefore, we ultimately decided to establish TCP connections which ultimately resulted in a significant performance increase! Additionally, our extensive networking issues led to us creating a novel communication protocol: PUDP (Polite UDP)!

We also had a really, really annoying issue between JSON 'latin-1' encoding and 'utf-8' with what seemed like randomly generated amounts of binary preamble. This took the entire team between 2am to 6:30am to fix.

Computer Vision: Finding an initial suitable model was difficult: balancing accuracy and speed - whilst trying not to spend the whole weekend dancing in front of the webcam chasing perfection. Once we found a suitable model, we spent lots of time ensuring that data gathered is as reliable as possible.

Game Design: Rendering the data retrieved from the computer vision model into GameMaker was a significant challenge which required a lot of calculations and debugging. We wanted to ensure that we added plenty of personal touches to the game, even learning GarageBand to create bespoke tracks!

Built With

Share this project:

Updates