El Mask | Devpost

Introduction

El Mask is an interactive AR experience, mostly our attempt on making something goofy and fun while experimenting with technologies like computer vision and graphics. Stemming as initially a ray marching project, this evolved into something unique: using parallax as a technique to create a window to our AR world. By moving our head in 3D space and using real-time facial landmark detection models, we are able pass this precise data into Three.js rendering engine, transforming head movements into a dynamic, intuitive controller. The final showcases are a couple different demos; main ones being the parallax view and the endless AR game. Another cool thing is that since its the app is web-based, you can run it from any platform: macOS, windows, Android, IOS etc. If your wondering, the name has no meaning at all ... kinda random we were feeling spanish in the moment.

Check it out: https://elmask.tech

We are the future of technologia. We are the future of interaction. We are the intersection of digital real-world modeling, and fun. We are - elmask.tech.

How we built it

We use Google's MediaPipe to give the computer "eyes" to see your face in real-time. Then, we pipe that data into Three.js, which renders 3D graphics and worlds you see on screen. Then a little bit of math and you have interactive world you can controlling by your real-world head movements to the virtual camera.

Challenges

The whole team had never done anything related to computer graphics before so it was a lil hard grasping some of essential concepts.
Spent a lot of time figuring out how to correctly map a real world camera (our heads) and modify planes of view on the screen. Involved a lot of maths, vectors and trigonometry!!

Accomplishments

Turning on the endless game and vibing to some music while dogging obstacles in VERY FUN!! (look at the video demo).
We decided to submit this because why not lol. Parallax is a cool technique we learned.