#72 Knowtion | Devpost

Our Emotion Detector Model
Our Front Page

Inspiration

Mental health is often overlooked, yet it plays a crucial role in our daily lives. We wanted to create a tool that could provide users with real-time insights into their emotional and stress levels without requiring active input. The idea was to integrate technology seamlessly into everyday life, helping individuals track and understand their mental well-being.

What it does

Knowtion is a mental health tracker that monitors your stress and mood swings in the background. It processes data such as audio cues and physiological signals to provide timestamped reports on your emotional state throughout the day. This helps users identify patterns in their mood and stress levels, enabling them to make informed decisions about their mental health.

How we built it

We used the Librosa library to analyze audio data for emotion detection. We incorporated datasets like RAVDESS, TESS, and SAVEE for speech emotion recognition and WESAD for heartbeat stress detection. After experimenting with various ML models, we chose XGBoost due to its performance and accuracy. The user interface was built using HTML, CSS, and React for a clean and interactive experience. Flask powered the backend, handling data processing and communication between the frontend and the ML models.

Challenges we ran into

We faced a few challenges along the way, like dealing with Librosa not being compatible with one of our datasets, which forced us to adapt on the fly. Training our machine learning models, especially XGBoost, was also time-consuming, making it tough to iterate and test quickly. On top of that, some features we were excited about—like stress detection from heartbeats and speech-to-text logging—didn't pan out as expected within the tight timeframe.

Accomplishments that we're proud of

We’re proud of successfully integrating multiple datasets and using them for emotion and stress detection, as well as creating a end-to-end pipeline that takes data from collection to user reporting. Most importantly, we managed to build a functional and user-friendly web app within a short timeframe, which felt like a significant accomplishment.

What we learned

We learnt that working with audio data is a lot harder than we had envisioned, and we learnt that datasets take a lot more space than we thought. As far as actual technical gains go, we learned a lot about audio data analysis, becoming well-versed with Librosa and the intricacies of working with audio datasets. We also gained hands-on experience with boosting techniques, particularly XGBoost, and got a better understanding of its strengths and limitations in terms of performance. Perhaps most importantly, we learned how to adapt quickly to unexpected technical challenges and still deliver a functional product.

What's next for Knowtion

So if we were able to realize our full vision for Knowtion, it would be an app embedded into a wearable device, something like a smartwatch. You ideally have it on all the time for maximum results, but the goal is to get a sense of the user's typical workday, and how it affects their headspace. The app will track their heartbeat, stress, tone, emotions and detect strong language. These instances would be recorded into a daily report, along with their timestamp. We would have a separate healthcare professional portal, where the user could share their reports with their healthcare professionals. We would allow for ease of communication between different professionals, allowing them to work on a patient together. This idea was a bit too ambitious to achieve most of our goals within two days, but we have made a good start, I think.