Mind=Controlled

All Hands Off Board

Comment

Inspiration

I've always wanted to make a computer vision project, but was hesitant to do so. FullyHacks gave me the excuse to created something, so I did just that. The inspiration behind the idea is helping people that do not have access to hands, making their lives easier when it comes down to using computers.

What it does

Mind=Controlled is an alien themed accessibility-focused desktop app that lets users control their computer using only head movements and voice commands — no hands required.

How I built it

It is built withPython3.10, and several libraries such as: OpenCV, MediaPipe, Vosk, and CustomTkinter. It combines real-time gaze-based cursor control with offline speech recognition, enabling actions like clicking, typing, scrolling, and navigating the desktop through natural movements and voice.

Challenges I ran into

I first started with using Dlib for getting the facial structure of the person, but then ran into very complicated math. At first I wanted to use pupil eye tracking, but noticed that eye movement is very little, and I was not at that level yet. I compromised with using head movement to move the cursor which proved to be a lot smoother and precise. Instead of reinventing the wheel, I settled with using Google's MediaPipe which proved to be the best in tracking and precision.

Accomplishments that I am proud of

I am proud that this is my first solo hackathon, and that I decided to challenge myself with something I was interested in, but was hesitant in starting. This hackathon gave me an excuse to explain new domains such as computer vision and speech-to-text models, and I am proud that I made something that actually works!

What I learned

I've learned that there is A LOT of math when it comes to face detection, and that terms such as "yaw, pitch, roll" were terms I've never heard of before today. I've also learned that doing UI/UX in python is very tedious.

What's next for Mind=Controlled

For Mind=Controlled, I want to deploy it as a desktop application that is cross-compatible Windows and Linux, maybe even MacOS. I've done an alien theme because I thought it would be funny, and that it was related to the Space Theme that FullyHacks provided. Maybe the next iteration of Mind=Controlled would be a more professional version without the silly gimmicks, focusing fully on optimizing the features.

Built With

  • customtkinter
  • mediapipe
  • opencv
  • pyautogui
  • pynput
  • python
  • speechrecognition
  • tkinter
  • vosk
Share this project:

Updates