Auralytics

Providing students with mental health insights using deep learning and cloud technology | Providing solutions to businesses to improve the customer experience with mutli-faceted emotional analysis

Comment

Inspiration

As CS students at large research universities, we all know classmates of ours who suffer from mental health issues, especially depression. Many of whom don't even realize that their mental health is declining until it is too late. As a result, we decided to create a service called Self by Auralytics to help students monitor and provide deeper insights into how their online actions affect their well-being.

After realizing the multiple uses of our backend technology, we decided the best approach would be to create a larger data science and emotional analytics project called Auralytics that would be composed of Self, Aware, and any other projects we decide to continue with in the future.

After coming up with the initial idea for Self, we realized that the same technology can be applied to improving the customer experience at businesses. We decided to make another service as part of our larger Auaralytics project called Aware that monitors customers emotions to provide a sense of the customer experience in a business so that business owners can learn how to improve their environment. This service currently lives in our backend ready to be hooked up to a UI.

What it does

Self by Auralytics utilizes the front-facing camera on the user’s laptop to analyze their emotional state as they use their computer. After analyzing this state, the program generates an Emotional Trend Line (ETL) which is displayed to the user. If used in tandem with our Google Chrome extension, Self correlates websites, such as Facebook or Reddit, to the user’s ETL, providing insights to how these websites affect their health. While providing health and habit insight is a major focus of our program, we specifically intended for this software to serve as an early warning sign for students falling victim to mental health issues.

Aware uses the same emotion recognition technology and ETL system to analyze the sentiment of up to 100 customers within view of a camera in a business. It uses the data it generates to gage the overall customer satisfaction at any given time. If the software detects a drastic positive or negative ETL spike, it marks and saves the time period the business owner’s further analysis to understand what affected the customer experience so dramatically.

How we built it

The product contains a few main technological components: data aggregation, emotional analysis, and frontend UI/UX.

To collect and aggregate image data, we use a combination of Go and OpenCV. These images are then sent to Azure Function endpoints for further processing and collection. We then use Azure’s Cognitive Services and FACE API to detect and classify emotion from the pictures in real-time. This data is used to calculate our ETL dataset. To augment this dataset, a Google Chrome plugin records all web requests made by a user during a session. This data is aggregated and presented alongside the ETL. Our backend storage utilizes Azure Tables for timeseries and image classification storage, and we use Azure BLOB storage for our raw image data. Finally, Azure Functions keep our infrastructure highly scalable and agile. A React UI presents our data to the user in a unique graph form accompanied by a list of potentially problematic digital habits. The React app communicates, again with stateless Azure Functions, to our Azure Cloud backend.

Challenges we ran into

Finding a project that was compatible across most of our skillsets was initially difficult for us. However, our biggest challenges involved integrating our various programs that we wrote in different languages to generate a cohesive application. We also worked extensively with various libraries for every language we used, so managing how they all related to each other presented many challenges.

Accomplishments that we're proud of

We are incredibly proud to have created software that can serve the community and help combat the mental health epidemic that often plagues universities. We see this project as something that could greatly benefit our classmates and could help them realize the mental health problems they may face before those problems reach a critical level.

In addition, we’re proud of the fact that we were able to create such versatile and flexible software usable in a wide range of applications involving mental health and human interactions. When our backend software is combined with an additional source of data, incredibly insightful analysis can be made of various scenarios. For example, we see possible applications of this software in sports for analyzing team mentality in relation to certain events in a match, such as: being scored against or having a penalty called. The same model could be applied to analyzing psychology study participants, generating additional data specifically related to emotional reaction. The fact that our software is so versatile is an incredible personal accomplishment for all of us. Human emotion correlated with a timed data source may bring incredible insights.

What we learned

Every one of use was exposed to a new language or framework. Be it the Azure Cloud, Python in a large project, Machine Learning and Facial Recognition, or beautiful React UI frameworks, we all learned something. As a result, this project involved an incredible amount of fast-paced learning and problem solving, true to the Hackathon spirit. In addition, this was our first project with Azure and with the general theme of machine learning!

What's next for Auralytics

What we would like to do is turn Self into a more comprehensive software suite tailored made to an industry’s need. Users use a variety of devices and platforms. Different collection methods across those mobile and computer platforms can help provide a more complete mental health profile. We also want to recognize specific tasks within programs or websites that can affect a student’s emotional level, such as viewing grades or programming with certain language (JavaScript :P).

For Aware, we would like to develop a piece of hardware for businesses to use in house or to make Aware compatible with existing camera systems and streams. On a broader scale however, we would love to explore how we could combine our backend software with various other data sets to gain a deeper insight into various fields.

Share this project:

Updates