IDLE

IDLE is a computer vision powered application that monitors your stress and posture in your virtual work environment!

Comment

Inspiration

For the past several months, each of us have been working almost exclusively from our computers. However, extended periods of bad posture and high stress levels as a result of an unfavorable situation has made us realize that it's important for someone to make us aware of our own health while we're sitting at a computer for long periods of time. This inspired us to make IDLE, someone that will do this for us!

What it does

IDLE is a web app that monitors various metrics regarding your stress levels, happiness, and posture. IDLE makes use of a variety of Google Cloud services in order to evaluate these metrics based on the features of your face, and the roll angle of your head. At a set interval time, IDLE takes a snapshot of the the webcam and uses these cloud services to determine the exact angle of the user's head and their likelihood of happiness or stress.

How we built it

All of the information extracted from a picture of the user's face was done via Google Cloud. The Cloud Vision API gave us face detection abilities, giving us values for roll, pan, and tilt angles of the user's head, and the likelihood that the user is experiencing a pre-determined set of emotions. While Cloud Vision API gives us a lot of information regarding the features of a user's face, we still needed something a little more tailored for our own purposes to determine the user's stress. This is where Google Cloud's AutoML comes in. Using AutoML, we were able to create our own machine learning model trained with hundreds of pictures of both stressed and unstressed faces. As a result, we were able to make a call to our model to also tell if the user is stressed given a threshold confidence level.

Challenges we ran into

Throughout the development process, we ran into various challenges using Google Cloud. For example, we needed to find an efficient workflow in order to debug our Cloud Functions that wouldn't involve zipping and uploading the code each time we made a change. This is where we made use of the gcloud CLI. Furthermore, we encountered challenges on our frontend. Initially, we planned to integrate our application as a Chrome extensions, however, due to known developer bugs in granting camera permission, we had to change our plans to make a web app at the last minute.

Accomplishments that we're proud of

Successfully deploying an app using the Google Cloud stack and creating a project that seems very relevant and applicable to the current situation. Since Google Cloud was new to us, we were proud that we could learn the interface and deploy something relatively quickly.

What we learned

Since this was our first time using Google Cloud, we learned a lot regarding the various features of Google Cloud as well as cloud services in general.

What's next for IDLE

A comprehensive UI, more in-depth health feature tracking, Chrome/Firefox extension integration. There is a lot of room to make IDLE a complete production-ready package, and these are the first steps we can take to accomplishing it.

Built With

Share this project:

Updates