Wynk

• 

  Metrics produced by the program.

• 

  Facial landmark detection.

• 

  Blinking eye example.

• 

  Open mouth example.

What it does

Wynk is a mouse replacement which uses facial landmark detection to mirror head movements onto the mouse. It utilizes facial movements like blinking and mouth opening to trigger mouse events.

How we built it

We primarily used OpenCV, dlib and a trained face shape predictor model. Once we had all the points set, we used euclidian distances to find the aspect ratios with regards to the eyes and mouth to tell whether they are open or closed. The pyautogui framework was used for moving the mouse and triggering click events.

Challenges we ran into

We had a lot of issues with dependencies early on. A large amount of the code only works on modern machines. Ultimately we only had one machine to test with. There were also limitation to the technology we used. Many of the eye detection methods did not meet the specifications required for the project. We ended up manually making calculations using facial landmarks in order to get the metrics we needed.

Accomplishments that we're proud of

Using this application you are able to navigate web pages and do many other simple tasks using only your face. It is a little finicky, but highly functional.

What we learned

Check dependencies first and do more research! Our main issue was with dependencies. Not all of our laptops were recent enough to use the required software. For future projects, we will probably check with all frameworks first and have a back up plan in case it doesn't work.

What's next for Wynk

A custom trained model would provide a lot more accuracy when it comes to blinking. The current model is learned from people with open eyes, so the model does not like estimating the eye position as closed. If we provided our own data, we could most likely have get a much better blink detector. We will most likely keep working on this. Ultimately, we would like to have a complete, easy-to-use mouse replacement for people suffering from physical disabilities.

Inspiration

As students interested in machine learning and computer vision as a broad topic, we were curious about how we could apply these theoretical constructs to real world applications to help out people who may be unable to use traditional methods of computer input

Built With

• dlib
• opencv
• pyautogui
• python