Face2Learn

Face2Learn helps you understand social interactions when it can be hard. It uses face recognition to detect anchor emotions on YouTube videos to teach individuals with social impairments.

Comment 1

Our Inspiration

Our team had dual sources of inspiration. On one hand, our team finds new AI technology a fascinating and exciting pursuit! However, AI tends to require many resources to be used effectively that make it inaccessible to the general public and developers. On the other hand, our team has connections to family members that work to overcome social disabilities. These family members often have very limited access to guidance with professional help only being available once or twice a week. In this, our team saw an opportunity for more 'academic' AI models to have a real-world practical application. By leveraging light weight research models, we realized that we could bring more accessible help to those with challenges that require specialized guidance.

What Does It Do?

We built Face2Learn as a Chrome extension that brings real-time emotion awareness to YouTube learning. For emotion recognition, we used an open-source CNN by Atulapra, trained on the Kaggle facial expression dataset with TensorFlow to classify seven emotions at about 65% accuracy. We paired this with Haar cascade face detection to track faces in the video feed and deliver live mood recognition as users watch educational content.

On the front end, the Chrome extension (Manifest V3) captures video frames, runs detection locally, and renders color-coded bounding boxes with smooth animations using the Canvas API. We designed the UI with Tailwind CSS and accessible typography to stay clear and non-intrusive over YouTube videos.

To enhance learning, we integrated the Gemini API. When a face is clicked on, Face2Learn generates context-aware information on the current situation. For example, a person is seen laughing with someone else, Gemini will explain that this is laughter, and this is how people connect. This combination of computer vision, AI guidance, and seamless browser integration makes the tool interactive and supportive while keeping everything lightweight and privacy focused.

How We Built It!

Frontend: Chrome Extension (Manifest V3) built with html and Tailwind CSS, and Js for a clean, accessible, and responsive interface.

Backend: Python Flask API for running the local emotion classification model (trained on the Kaggle facial expression dataset, ~65% accuracy), tensor flow, and managing AI-driven features(Gemini).

Computer Vision: Haar cascade face detection to process video frames in real time.

AI Integration: Google Gemini API for generating dynamic, context-aware content based on detected emotions.

UI Rendering: Canvas API for smooth, non-intrusive overlays and color-coded feedback on YouTube videos.

Version Control: Git & GitHub for collaborative development and code management.

Development Workflow: Python virtual environment for isolating backend dependencies.

Challenges We Ran Into

In terms of technological challenges, we had to deal with several limitations. One of the biggest problems we realized were the limited training context of many free light weight models. While this strict standardization of our data could not be strictly overcome, our team worked to leverage the strengths the models did possess and minimizing the impact of their weaknesses. Not all challenges with AI models were strictly from their weight's limitations. Gemini, while very accessible, proved to have very inconvenient response times. Outside of tech challenges, our team found the format of hackathon to be a challenge of one's stamina, challenging us to give the extra ten percent and finding the drive to keep hacking!

Our Accomplishments!

We have built a fully working prototype of our idea where we used multiple different technologies, such as TensorFlow, Flask, Haar Cascades, Canvas API, Gemini API, all at one time. We also prioritized a user first experience which really made us think about the user while thinking up different ideas. We are also proud to build something that can impact people's lives in a positive way.

What We Learned!

For most of our team, this was our very first Hackathon! Our team viewed the event as something of a marathon, but in spite of the challenges we faced, we feel that we have greatly improved as a team. We have gained a much greater understanding of AI's integration as a core feature, gaining a strong understanding of both the benefits and weakness's that come from AI being a central feature. Our team also familiarized ourselves with new tools like Torch and TensorFlow. Our team was also able to effectively practice at dividing work and having proper teamwork etiquette. In general, our team has gained valuable experience in both AI integration and teamwork practices.

What's next for Face2Learn?

Going forward, our team is excited to continue to pursue learning about and applying modern AI advancements! With more time and the skills, we have gained, we will pursue two aligned avenues. Firstly, our team will work to promote the use of common accessible and light weight AI research to empower those with social disabilities. These additional tools will be integrated into Face2Learn to expand its capabilities. Secondly, our team will work to improve the free public data collections available for researchers, especially for tasks overly reliant on stock photos and limited context! Doing so will not only improve research, but the quality of light weight models that can be used to help people in their day to day lives. With this gathered data, we will be able to train and improve the models currently behind the many different features within Face2Learn. With better data collections and tools to help people with their daily lives, our team is committed to building a better brighter future for everyone!

Built With

Share this project:

Updates