MemAid

We wanted to improve the quality of life for people with dementia, who can suffer from memory issues. Our product will serve as a second brain to help users remember people and important information.

Comment

Inspiration

One of our team members has a family member who deals with dementia on a daily basis. From this, we got the inspiration to make a memory aid to improve the lives of those with this condition.

What it does

Our app helps those with memory-altering ailments by recording video and audio of interactions the user has with those around them. By learning what the user sees and hears it can keep track of important information about people they encounter. This can help users link names with faces as well as help them remember key details from previous interactions. This information is relayed to the user through a headset, like an airpod or other Bluetooth device.

How we built it

The application consists of two main parts, a frontend mobile app and a backend server. We built the mobile app using Flutter and Dart, which we enabled to stream audio and video to a server. The backend server is written in Python 3 and handles the processing of all audio and image data. We used computer vision with neural networks in order to detect and recognize faces. We utilized Google Cloud APIs for speech to text, text to speech, and natural language processing. We keep track of each sentence recorded, and then when a new face is recognized, we can pull from recent sentences and see if that person mentioned their name.

Challenges we ran into

We had some difficulty figuring out how to convert speech to text as a constant stream of audio instead of using saved recordings. We also had to optimize our facial recognition processing to quickly yet accurately recognize faces.

Accomplishments that we're proud of

We were able to successfully stream video and audio between the device and the server, allowing us to process the information on stronger hardware. We also constructed an aesthetically pleasing UI for the app.

What we learned

We learned how to send video and audio streams to and from a server. We also learned how to locate and recognize faces. Additionally, we learned how to extract important information from text using natural language processing.

What's next for MemAid

The next step for MemAid is to add additional utility by keeping track of additional information. This could involve events associated with a person, to-do items, relation to the user, and more! In addition, we would like to integrate IoT devices, such as an external camera so the application is more discrete.

Share this project:

Updates