6ix6tep | Devpost

Inspiration

As a group of 4 rhythm game enjoyers, we wanted to touch grass and live life more actively. Additionally, we found that there aren't a lot of accessible ways to play DDR in Markham. Thus, we took matters into our own hands and created 6ix6tep, a portable dance game that promotes an active lifestyle. 6ix6tep is a combination of a classic rhythm game format, and live 3D camera tracking via raspberry pi's running on the QNX operating system.

What It Does

6ix6tep is a foot-controlled rhythm game that supports the existing osu!mania beatmat format. This means you can seamlessly import any song from that game without any hassle. Instead of using a keyboard, players play by moving their feet at the right time. Our dual-Raspberry Pi camera system cleanly simulates and tracks player's movement in 3D.

How We Built It

2x Raspberry Pi 4s, each with a camera attachment. Both are running QNX, a real-time OS. The 3D tracking is done through OpenCV + stereo camera calibration, detecting humanoid bodies. Each Pi transmits its camera information through sockets to the game instance. The game itself was built using Godot, and we support osu!mania maps with a Python pre-processing script.

Challenges We Ran Into

QNX was a massive struggle, but we were willing to rise to the challenge. One of our brave teammates specifically dove into the terminal for hours on end, wrangling to create QNX support with numpy, OpenCV, and MediaPipe. Additionally, we were worried that just 1 camera would result in possible drops in camera scanning. We decided in 2 cameras for a 3D scan with greater uptime, but the calibration process and the trickiness of handled 2 Raspberry Pis turned out to be a much greater barrier than we thought. Finally, we also struggled connecting everything together. We needed to link the 2 cameras together and process them on the Raspberry Pis, and then use network sockets to transfer the processed scan to our game front-end.

Accomplishments That We're Proud Of

For our group, this was our first major hardware hack. We have sparse experience working with breadboards, but this was the first time we built something with such a complex technical flow. Additionally, we're all very proud of each other for fighting through to the bitter end! Despite being composed of hackathon veterans, Hack The 6ix pushed us to our absolute limits to get something working produced before hacking ends.

What We Learned

The main takeaways from 6ix6tep was the knowledge of how to integrate real-time data with computer vision, and the fundamentals of stereo vision. Namely, turning multiple 2D images of the same object into a reliable 3D render. Also, we have become VERY familiar with both the highs and the lows of a hardware hack.

What's Next for 6ix6tep?

In the future, we're looking into increasing the song selection into a wide library. Since we support osu!mania beatmaps, this shouldn't be too difficult. Additionally, we're looking forwards to integrating local multiplayer.

https://docs.google.com/presentation/d/19_v9ghiDveJsRD4udkbWYnFXEzZBLwu8pk4Ldo-CLAM/edit?usp=sharing