Sentimeant (aka EmotionCATch)

nice

Inspiration

Wanting to understand how we can increase trust in our conversations, to see the intent and sentiment of conversation changes. To understand the verbal cues have a significant impact on the nature of the conversation. Trying to understand the conversations killers and the things that get people hooked. Inspired by psychological and sociological profiling techniques.

What it does

Essentially, Sentimeant/EmotionCATch acquires a video file, extracts the audio component, and breaks it down to evaluate even subtle fluctuations in pitch, volume, frequencies, and voice intonations! Presenting, Sentimeant/EmotionCatch: where our software, captures a speaker's emotions, thereby providing more information of a their deeper intents and inner thoughts!

Our program leverages industry-standard API's to provide advanced sentiment analysis and speech diarization to help better understand the conversational interaction between individuals.

Real World Applicability: This technology can revolutionize the prosecution process and will prove truly useful in a court setting to analyze the authenticity of the words of a suspect! It can also be used to aid recruiters in the hiring process as it can help them gauge the competence of an applicant, using the Natural Language analytics that our platform provides.

How we built it

We leveraged React.js for the front-end which essentially takes in any youtube link and then streams it to our backend Flask server. We then split the audio from the corresponding video and utilize Google Cloud Platform to analyze the audio's intents and retrieve a transcription of the conversation. Additionally, we aimed to use GraphQL in amalgamation with the sentiment analysis of GCP's Natural Language Processing to map the conversational relationships between the individuals.

Challenges we ran into

Initially, we had some difficulty using Microsoft Cognitive Services but we found the google API to be a better solution. A challenge we ran into was that google restricted the length of the video to one min, it limited our flexibility on working with longer videos/conversations.

Accomplishments that we're proud of

The Sentimeant/EmotionCATch Team prides itself on our ability to garner psychological insights on the mere basis of sound patterns. Ultimately, the task of calling numerous APIs was undoubtedly a demanding yet rewarding one; working with Google Cloud’s API, we worked with technology that still has a strong potential for growth and this fact meant we were restricted to only audio analysis on small files (particularly, content smaller than 1 minute). We’re happy to have done our best on working around such issues and bringing together a successful, final product.

What's next for Sentimeant/EmotionCATch

We still have a lot of plans for our Cat to grow into a Lion! In the future, we hope to enable video acceptances of lengths exceeding one minute. In particular, we’d love to implement an audio analysis in conjunction with video analysis, whereby our algorithm interprets changes in body language/gesture, as well as facial expressions. By utilizing image recognition, we can, frame by frame, comprehend even the most subtle indications of deception and pretense.