FaceMelody

Inspiration

We were considering people with disabilities and saw how it may be hard for them to use or even interact with technology depending on how severe the situation is. This may affect their mood since they may not be able to do much. Music has the ability to uplift and heal and would be of help to them, since it can lift their mood or help them control it. However, they may not be able to switch or even play music by themselves, and if they have a caretaker it they may not be able to constantly monitor the person's states and expressions to change the music. This inspired us to create FaceMelody to help automate the music.

What it does

FaceMelody is a website application that uses a machine learning model to read the facial expressions of the user and play music based off of their expression. It will observe the user and switch the music at certain intervals to meet the needs of the user.

How we built it

FaceMelody is built using Javascript, HTML, and CSS for the user interface and is connected to an MTCNN and VGG-Face Mode trained machine learning model to recognize facial expressions in real time. We used some simple Javascript to integrate everything and create the final product.

Challenges we ran into

This project posed some unique challenges. We had to deal with real life problems, such as how the user would even interact or use the product. We discussed the issue and came to the solution to have minimal physical interaction from the user after the web application has been started, most likely from the caretaker.

Accomplishments we are proud of

This was a daunting task that we did not have much experience in, and we are proud to have completed the project that will help better the lives of those around us.

What's next for FaceMelody

Moving forward, FaceMelody would extend to more than just improving playing the right music. We would like to use the facial expression AI to extend to activating other features, such as notifying caretakers.

Resources

https://developers.google.com/youtube/iframe_api_reference https://benetech.org/blog/designing-disability-inclusive-data-collection/ https://www.bloomberg.com/news/articles/2021-07-29/the-apps-and-technology-helping-people-with-disabilities https://www.pewresearch.org/short-reads/2021/09/10/americans-with-disabilities-less-likely-than-those-without-to-own-some-digital-devices/ https://cdt.org/insights/disabled-people-need-critical-privacy-protections-in-adppa/ https://bootstrapmade.com/softland-bootstrap-app-landing-page-template/

Built With

• computer-vision
• css
• gcp
• html
• javascript
• machine-learning
• replit
• youtubeapiv3