Team 30 - Spell Vision

Inspiration

Driven by our interest in computer vision and gaming, we wanted to create a fun project which unified these interests.

What it does

• The website recognises hand landmarks, and classifies the current hand position into a set of gestures.
• The user does a combination of gestures in quick succession to cast a spell
• It is a compeitive 1v1 game
• Each user gets 100 mana points and 100 health points
• There are three spells: fire, water and grass
• Fire inflicts a repeated damage of 5 damage per second for 10 seconds on the opponent and costs 25 mana
• Water extinguishes fire, heals 5 health and costs 33 mana
• Grass heals 30 health, and costs 50 mana
• Mana regnerates at a rate of 10 points per second
• Health doesn't auto-regenerate

How we built it

• React with typescript, vite and tailwindCSS for the frontend
• Mediapipe for the hand gesture recognition
• WebRTC for the peer-to-peer connection
• And Firebase for the WebRTC signalling server

Challenges we ran into

• We had great difficulty porting the gesture recognition from vanilla TS into react.
• WebRTC was tricky to implement in a performant manner
• We couldn't make the video feed combine with the hand recognition in a non-hacky way
• The Mana mechanic was difficult to implement

Accomplishments that we're proud of

• We made it work!
• We linked our skills together well, such as machine learning, react principles and how WebRTC works
• It's very responsive, even on old hardware
• Most importantly, it's fun and challenging to play

What we learned

• How WebRTC works
• How Mediapipe works
• Don't trust claude to write your hand recognition mechanism 🙃

What's next for Team 30 - Spell Vision

• We would like to add more mechanics, like two-handed mode where each hand can perform a different gesture, making for more fast-paced gameplay

Built With

• firebase
• mediapipe
• react
• tailwindcss
• typescript
• vite
• webrtc