KnightBlocks | Devpost

WebGL Environment with blocks
WebGL Environment with blocks and a block selected
Cursor moved by face movement
The left wink, would be used to place a block

Inspiration

Our inspiration came from someone close to one of our team members. This person loved building with LEGOs, but now struggles with arthritis. We wanted to bring the joy of building with LEGOs to people who experience limited motor function.

What it does

The goal was for our project to allow users to build hands free. Users can select bricks using the movement of their face and place them through winking. There are various buttons on the interface to allow users to simulate building better.

How we built it

Using Google's MediaPipe framework, we deployed a face landmark detection model to control the cursor of the application. The face landmark detection model can be used to track various points on a user's face to figure out where the user is looking. For the virtual environment and assets, we used the Universal Render Pipeline in Unity and used WebGL to keep it browser-based.

Challenges we ran into

We spent a long time trying to get a different models working that just focused on the user's iris to make the cursor more precise. What we realized on the second night of the hackathon was that the face landmark detection model could do exactly what we were searching for. Two of us had also never worked with Unity or C#, posing another challenge for us. We faced many HTTP errors at the end too to get the webcam working. Unfortunately, the final product could not come to life.

Accomplishments that we're proud of

We were able to bounce ideas off each other easily and we hit the ground running after a short brainstorm session. We're proud of the brick design and the ability to overcome the "bricks" in our way (not knowing C#).

What we learned

We learned how to utilize a computer vision model and connect it to a virtual environment. We also learned about efficient team collaboration in a high pressure environment.

What's next for KnightBlocks

We're hoping to get the application working and implement users being able to save their creations and share them with others and AI-powered instructions (users can generate instructions for an idea).