Disembodied

Control your disembodied hands with your... hands.

Comment 4

Inspiration

I've worked on quest games before. My last game (Twistex) was big on passthrough. But I'd never worked with hand tracking before, so I knew I wanted to do something with that. My initial thought was to have a small character walking around where you control it's legs with your index and middle fingers, but I quickly realized that hand tracking isn't good enough for that.

I'd had another idea for a while of a third person game, where you control a small character's arms using just controller movement, like a third person Gorilla Tag. So I decided to put the 2 ideas together and use hand tracking to control 2 disembodied hands, where the fingers make a difference to the movement too.

What it does

Disembodied is a third person MR platform game where the main character is your disembodied hands. Using just hand tracking you must manoeuvre them through small levels featuring various mechanics and challenges. The hands will copy what you do 1:1 (with a restriction on the distance they can move apart), so it's very easy to pick up and feels natural to control.

You start with some basic levels which will teach you to cycle your real hands to make your disembodied hands walk along. Players are encouraged to move around the play area, following the hands, which makes it easier to walk in different directions.

When you fall from the level you will land on the floor, where you can then walk around your room and interact with your real environment. People who have tested the game have found this to be a lot of fun, and I plan on doing more with this in the future. To continue the level just push the reset button on any of the podiums to spawn back to the last checkpoint.

In each level you must collect orbs to unlock the exit, then push the exit button to move on.

A red power up will give you the ability to punch through walls. Once you've collected it, making a fist with either hand will turn it red. Then with enough force you can destroy walls blocking your path, and watch the pieces collide with your real environment.

Rails will scoop up the hands and transport them to another area. Along the way you may have to position your hands to collect orbs. They can also drop you in a different orientation, so you have to work out how to walk backwards or sideways.

Interaction SDK

I've gone all in on hand tracking. I use the physics capsules to interact with menu buttons. I use a pinch to set up the play area. I use some custom hand gestures to grab the height lever, and trigger power ups. And of course I use it to mimic your hand movements to the 'character' hands, using various other prefabs and features of the SDK.

Spatial Anchors

When you set the size and position of your play area, the location will be saved as a spatial anchor so it's always in the same place next time you play.

Passthrough

Obviously.

Depth/Occlusion

I use the occlusion hands from the depth API sample (which works surprisingly well on Quest 2), as well as the Quest 3 depth sensor for occluding your arms and anything else that goes in front of graphics. Both can be disabled in the menu.

Scene Mesh

Objects and the disembodied hands will collide with your scene mesh. So when you fall out of the level you can walk around your room. You could even setup some ramps and such to make a course around your playspace. This is something I want to explore much more in the future.

Challenges I ran into

The biggest challenge was getting the fingers to have an effect on the movement. I started by following Justin's video on Gorilla Tag movement and replaced the controllers with hand tracking. I had hand models that would copy your hand pose, and moved them towards your hand positions using physics, then send forces from collisions to the main player rigidbody (which was the camera rig). I then converted this to a small third person model which takes the positions of your hands relative to each other, and applies them to a different rig, which then becomes the main rigidbody and 2 target positions for the hands.

It was fairly easy up to this point, but I really wanted to be able to walk on the fingers of one hand which wasn't happening. To achieve this I had to use the tips of each finger (from the physics hand models) as targets for 10 other physics objects, which in turn would add collision forces to the main rididbody, as well as the hand they are attached to. Working out how to apply the extra forces and tweaking all the values to a point where it felt right took a long time, with lots of trial and error.

It's still far from perfect, and between meta hand tracking and unity physics (your 2 greatest enemies in this game) it's still pretty janky at times. I need to spend a lot longer on this to get it perfect, but I felt like I could have spent the whole hackathon doing that, so I needed to move on. It isn't too bad now. It's possible to pull 1 hand along using your fingers, and even walk along using 2 fingers if you're careful (thumb and index is quite easy).

I also had issues with the hands losing tracking, as when they do they go to zero position and rotation and the pose goes to a default straight hand. This would cause havoc with the physics, so to minimize this I have a separate copy of the hands which records the rotation of each bone (the bone structure is a bit different on the ones that get spawned for your actual hands). The rotations then gets copied to the physics hands, at which point when you lose tracking I can keep the last known position, rotation and hand pose. Again this still isn't perfect as when a hand gets occluded by the other one, the pose often goes wrong before it loses tracking all together. But it helps a lot.

Accomplishments that I'm proud of

I'm pretty proud of the hand movement despite it not being perfect yet. Another problem I had was using scale. I wanted to be able to change the size of the play area to accommodate different sized rooms, but changing the scale of the hands caused the physics to behave differently.

So I had to come up with a scale system where pretty much everything has 2 copies. 1 with physics and scripts, and 1 with renderers. Everything in a level scales the graphics down to the right size and keeps the rest a constant size. The podiums and obviously your room are always the same scale, but for things to collide with them I need to change the scale of the colliders to an inverse of the level scale.

This also made coming up with level mechanics a lot harder, as for instance disabling a game object means disabling both versions of it, so I have to have everything referenced in a complex script, which also has to copy the movement of all non static objects. At the moment it creates the copies and sets it all up every time you load a level, which causes a few dropped frames, so I need to do that a bit better there. But all in all it works really well.

What I learned

I still consider myself a novice in most aspects of game development, and I've definitely learned a lot from this experience. I now have a much better understanding of the logistics behind a mixed reality game. I'd say the biggest thing I learnt though is time management. This is the first time I've had a strict deadline to hit, and I think I've managed it well, spending enough time on each thing to get it working well, but not over thinking everything and adding features that no-one will ever notice.

What's next for Disembodied

I will definitely be continuing development on this. I think it's an original idea that's not been done before and I'm excited to get it out there. I've got lots of ideas of where to take this. There are loads of mechanics I want add. Like lasers to avoid, objects you can pick up, and more power ups using hand gestures. I'd love to eventually add a level editor, where you can make normal levels to share, or add ramps and platforms to your real environment, and have levels across your tables and furniture. I even have some ideas for a simple story where characters will enter your room and talk to you.

Thank you to Devpost for hosting this, and the judges for your time.

There's a bit more gameplay in the unreleased launch trailer https://youtu.be/2jJNow_9v9g

Built With

+ 2 more
Share this project:

Updates