ARcane! | Devpost

Controller Overlay!
Blast Effect!
The Army...
Josh 0_0

Inspiration

We’ve always been fascinated by the idea of casting spells with just a flick of the wrist, like stepping straight into a fantasy world. With Snap Spectacles and Bluetooth controller support, we saw an opportunity to merge immersive AR visuals with real-world motion tracking to create something magical. We were also inspired to push the limits of hardware controllers by implementing our hardware that could interact with the Spectacles software interface!

What it does

ARcane transforms an ordinary motion-tracked wand into a spellcasting device. By moving the wand in specific patterns, users can conjure elemental spells (fireballs, ice shards, poison blasts) that appear in augmented reality and can be launched from their hand. Each gesture becomes a spell, turning the physical world into a magical battlefield.

The project has support for two different types of controllers; a traditional X-Box controller, and a special Bluetooth wand. The X-Box controller relies on the various built-in buttons for picking, launching and blocking spells with seamless software-hardware codesign to integrate the controller with the AR experience and enhance the capabilities of the controller. . The Bluetooth wand relies on movement patterns to cast spells, providing a super immersive experience like a true wizard.

How we built it

Hardware: An ESP32 microcontroller paired with an IMU (accelerometer + gyroscope) was assembled within a wand-shaped package. The ESP32 ran Bluetooth Low-Energy (BLE) communication with the Snap Spectacles to provide a secondary form of user input.

Software: We built a motion recognition system to detect gesture patterns in various directions mapping them to specific spells. These spells would be transmitted from the ESP32's C code, to the typescript running on the Snap Spectacles. A similar driver existed on the Spectacles to interpret the X-Box button and gyroscope data to trigger AR actions.

AR Lens: In Lens Studio, we created AR visual effects for each spell, which appeared in the user’s hand and could be launched into the world at incoming enemies using one of the two controller inputs.

Challenges we ran into

Designing a gesture detection system that’s both fast and reliable on limited hardware.
Achieving smooth communication between the wand and Snap Spectacles using Bluetooth HID.
Allowing both the wand and X-Box controller to communicate with the Spectacles at the same time.
Balancing complexity—wanting advanced gestures like circles and squares—while ensuring performance was stable for a hackathon demo.
Accurately mapping AR objects to their real life counterpart (i.e. controller, wand)
Creating AR effects that felt magical while still running efficiently in real time.
Using a brand new engine with a new language and limited gamedev experience on the team.

Accomplishments that we're proud of

Building a fully functional Bluetooth wand controller that felt natural to use.
Successfully mapping physical motions into spell effects in AR.
Creating a system that feels like a game, but could also inspire accessibility or educational AR tools in the future.

The first time we saw a fireball appear in AR after a wand flick, it felt like magic was real!

What we learned

Throughout our journey at Hack the North this year, we learned a great deal about how to leverage AR technology with real-time Bluetooth hardware to create truly immersive user experiences. For some of us, this was our first exposure to AR app development, and we're happy to see that it worked!

What's next for ARcane!

We had a lot of ambitious ideas that we weren't able to get to, but in the future, we'd love to:

Expanding the spell library with more complex gestures (e.g. triangles, spirals, runes).
Adding haptics and lights to the wand for richer feedback.
Support multiplayer duels to battle your friends with AR spells!
Exploring accessibility applications using gesture-based input to assist users with visual or physical impairments.