Inspiration
Do you ever forgot your phone number? Do you ever forgot your name? Do you ever forgot who you siblings are? No?
Maybe you think that´s crazy, but for nearly 50 million people worldwide that is reality. These people suffer from the disease Alzheimer. Major effects of Alzheimer are that you forget who the people are, how to eat and even how to go to the toilet. All the things that we take for granted.
What it does
Our software is a smart bot that interacts with the patient in a social way via voice. It will remind the patient to take his medicine, to do some physical exercise and can play a mental test with the patient. It will monitor the patient activity and can detect the emotion of the patient.
How we built it
We built the tool mostly with Azure and in Python. The app listens to the voice of the patiente by using speech to text. We then get the intent of the message by using LUIS. After that we send send the intent of the messege to our chatbot. Depending of the intent of the messege the chatbot we get information from the Azure SQL database. These data could be for example the time, at which the patient has to take the medicine. From there the chatbot will send the message back to LUIS and back to the patient using text-to-speech-technology. While the patient is talking, it uses a camera to record the patient's face and can thus determine the patient's mood during his speaking time. This works by using the Python libary face recognition to distinguish between five different moods. We store the information about the emotional state together with the text messages in the database. To enable an emergency call for the patient, we have automated the call using Twilio. Twilio is started as soon as LUIS detects the corresponding intent. The heart of the project is the database. There we match the emotional state with the topics the patient has talked about. In this way we can determine which topics make the patient happy or sad. Based on these evaluations the doctor will be informed about the negative topics and it will no longer talk to the patient about the negative topics and will only focus on the positive topics. We built this function in Python.
Challenges we ran into
We ran into some challenges on the way. For example, we had some problem to get the text to speech engine working in the beginning.
Accomplishments that we're proud of
We are proud of making the bot smart by combining facial recognition to determine the emotion of the patient while talking about different topics. With this information the bot makes intelegent decisions.
What we learned
We learned how to use Azure services and how you can automate processes.
What's next for us
Now we are looking for input from menatl health expert so that we can optimized the activities with the patient and buld a helpfull social bot that will revolutionize the mental care area.
Log in or sign up for Devpost to join the conversation.