Emoti

Track your feelings using a fine-tuned AI and analyze your facial expressions throughout the day

Comment

Inspiration

We wanted to help one of our developer's brothers, who has mild autism, in recognizing the facial expressions of others and himself, as well as track how he's feeling throughout a week. Through combining facial expression detection with an emotion tracker, we introduce Emoti, an app that motivates users to take care of their mental wellbeing as well as recognize their own emotions.

What it does

Emoti is a web and mobile app where users take selfies of their face, submit them for categorization into seven distinct emotions, and add them to a growing history of their own facial expressions over time. The app features a login page, a camera and emotion detecting page, and a statistics page that features a calendar and a star graph to track emotions over time. The statistics tab is also able to inform the user if they are experiencing an excess of a particular emotion.

The problem it solves

Autism comes in many forms. One common difficulty in people with autism is recognizing emotions through facial cues. In a contributing developer's brother's case (who has mild autism), he had extreme difficulty with this task through childhood. We believe Emoti can help people like him in detecting the emotions of others and themselves.

Furthermore, Emoti's statistics tracker promotes mental health awareness for all. Beyond its specific use case for autism, the app encourages all users to practice emotional literacy—the ability to identify, understand, and express emotions. This is a foundational skill for mental health, helping individuals manage stress and build resilience.

How it aligns with the Social Good theme

By providing a statistics tracker, Emoti empowers users to take an active role in their own mental health. It transforms subjective feelings into objective data, allowing users to identify patterns and triggers. This self-awareness is the first step toward making positive changes, whether it's through personal habits or by providing concrete data to a mental health professional. Because its free, Emoti can help those who might not have access to professional care to participate in tracking their mental health, making mental wellbeing more accessible.

Tools or Technologies Used

The app is hosted on a frontend expo server. The login page is built with MongoDB and a JavaScript server, and the camera page is built with TypeScript and React-Native. Users take photos and submit them to a backend Flask server, where our fine-tuned model classifies them. Then, we used an ngrok instance to tunnel all three of our servers together. We fine-tuned an imagenet21k model on seven human emotions using Torch to improve accuracy and avoid overfitting.

Challenges we ran into

  • End-to-End server connection: using ngrok to ensure user ease-of-access to the app through any WIFI, as well as smooth communication with the backend expo, login, and Flask servers, is crucial in maintaining the functionality of the app.
  • Fine-tuning: we struggled in fine-tuning a pre-trained model that was trained on 21,000 classifications and fine-tuning it to seven basic human expressions, so we trained multiple models and utilized different techniques in improving validation accuracy. The machine learning side of the report can be found below.
  • Storing user data: Designing a intuitive statistics tracker that gives insight into the user's emotions over time, as well as developing a functional database that stores those statistics behind the scenes, was a challenge.

Accomplishments that we're proud of

  • Successfully fine-tuning a machine learning model that accurately classifies a face into one of seven basic emotions after experimenting with differing epochs, dropout rates, and hyperparameters.
  • Integrating the inference function to identify a face and downsize the image to 224x224, which are the dimensions that the model is trained on
  • Creating not only a functional but clear statistics tracker that allows the user to infer patterns about their emotions over time
  • Successfully integrating ngrok server tunnels over diverse web frameworks (Flask, expo, etc)

What we learned

We gained experience in end-to-end server development, fine-tuning in machine learning, and web & mobile app development. This process taught us how to decrease common ML pitfalls like model overfitting, where the model fails to generalize to unseen data, streamline inter-server communication using ngrok, and optimize API response times and frontend rendering for a seamless user experience.

What's next for Emoti

Enhancing Core Features: Based on the emotion, the app could offer gentle, evidence-based suggestions. For example, if a user consistently feels "sad," Emoti could suggest a short breathing exercise or a journaling prompt. We could also allow users to add short text notes or tags (e.g., "work," "family," "exercise") to their emotion entries. Over time, the statistics page could reveal powerful correlations, like "You tend to feel happiest on days you tag 'exercise'." We also plan to add multiple file support, where the user can upload selfies in bulk instead of only supporting one photo at a time.

Improvements in Machine Learning: Upgrade the model to not only classify an emotion but also its intensity (e.g., from "slightly happy" to "ecstatic"). This would provide more nuanced data for the user. Additionally, while the seven basic emotions are a great start, we plan to expand the model to recognize more complex or subtle states like "thoughtful," "tired," or "confused."

Scalability: Migrate from ngrok to a scalable cloud hosting solution like AWS, Google Cloud, or Heroku for the backend Flask server. This would eliminate the need for (in its current state) the user to host their own frontend and backend.

Share this project:

Updates