MelodicMind | Devpost

Home Page
Memories
Add Memory
SupportBot
NFT Mint PolygonScan

Inspiration

Mental health is an important issue. In 2017, over 1 in 8 people suffer from mental health issues. I wanted to solve the difficulties of dealing with mental health, and I realized that there are not many apps that have a concrete plan to help make the burden a bit easier. Therefore, I wanted to create a solution that was a simple plan that anyone could follow, and use my application to promote mental health and help become happier and more successful. I wanted to create a proof of concept that could be turned into a viable product, as well as help myself and others that have struggled with mental health issues to feel better!

What it does

MelodicMind is an application that provides a simple three step process: create a journal entry, remember happy memories, and talk about it with an AI chatbot. It is a soothing and relaxing application to make the process of destressing easier, and calm the mind after a stressful situation. This three step process works because you first document the situation to see if you can reflect on it, then remember some happier moments, and finally, you can talk to a bot about your problems, and they can give you feedback about what you should do!

How we built it

For the machine learning portion of our app, we fine tuned GPT-3.5 and LLama for the Chatbot, stable diffusion and OpenAI image generation, MusicLM and MusicGen training, sentiment analysis hyperparameter tuning and pipeline workflows using HuggingFace transformers, tokenizers, and embedding layers. This was all hosted on streamlit. For blockchain tools, we used web3.storage’s API for IPFS and Filecoin decentralized storage, the VerbWire API to create an NFT and mint it to 7 mainnet and 5 testnet chains. Finally, we used the T3 stack (Next.JS, Tailwind, Typescript, Prisma ORM) as well as a firebase DB. We modified Vercel’s AI interface, while editing the GPT stream to allow a vector store to store previous context (a POC was also deployed on Streamlit). Some other things we attempted to do was use OpenCV for facial recognition. Finally, we used Ngrok and Flask to deploy an API endpoint in order to run generative image AI. We looked into papers about sentiment analysis, GPT fine tuning, and music generation training using quantum layers (musiclm-pytorch), as well as CI/CD methods regarding MLOps like Wandb and pytorch lighting, which we used to run sweeps while fine tuning. I created several Colab and Jupyter notebooks to train and test models, as well as bypass some implementation issues with python environments.

We have three levels for our product. Our first level is mainly a testing environment using Google Colab and Jupyter Notebooks, which shows off the integrations, setup, and API routing of the external parts of our application. Next, the Streamlit app showcases our proof of concept, and how we are implementing these tools in a full stack web app. Finally , the web app is an MVP, which contains some integrations, but is a fully working sleek UI to showcase the features of our POC.

Challenges we ran into

I had trouble with the scope of this project, since I had to figure out a lot of tools and technologies in a short amount of time, as well as come up with a novel solution for a pretty involved problem. I had trouble with setting up a lot of tools, as some of the python packages and docker images were outdated, deprecated, or had issues I had to work around. I had to get around GPU limits with Colab, as well as training times for several of the tools I ended up using. Finally, even though I have previously worked with Flask and Streamlit, I had little experience with the T3 stack, so I had to learn it on the spot, and connect it with the backend in a short amount of time, as well as interact between three different tools. Understanding the ML jargon and implementing it was also quite difficult: I’d have to read about something, find a suitable model, fine tune it, implement it, load and deploy it into streamlit, and make sure it was connected to the UI. I also had to learn a LOT about music generation, which was super interesting, as well as a comparison between two SOTA music generation models, and their limitations with limited GPU access. I also had to learn a lot about IPFS protocols and decentralized storage, and especially about the Verbwire API, where I had to implement over 12 chains! Through this hackathon, I've understood the power and importance of Verbwire and how I can use it further in this application, through the IPFS store HTTP request, sending NFTs cross chain, and even deploying a smart contract, which can be used for an NFT studio as a future feature.

Accomplishments that we're proud of

I’m proud of creating a product, and a firm three step process that I and anyone else can follow, in order to gain a better understanding of how to tackle mental health issues. I’m proud of overcoming multiple hurdles and learning several technologies in the span of a hackathon, as well as learning about the technical details of them. I created a full stack web app with a frontend, backend, and DB, with ML and Web3 integrations, as well as being optimized for accuracy as well as latency.

What we learned

We learned that mental health is quite an involved problem, and what we know and tried to tackle couldn’t possibly encompass even a small amount of the problems with mental health. However, we learned that by making small strides and by sharing and educating others on our story, we can influence and inspire others around the World to maybe become a bit happier, to enjoy life a bit more, and maybe help some tackle their mental health issues, using our simple three step process.

What's next for MelodicMind

We want to implement some more features into MelodicMind, including openCV facial recognition, a decentralized autonomous network so users are incentivized to help others, a chat or video service for users to talk to each other, a TTS service for the AI Chatbot, as well as more educational tools to help users tackle mental health (a four step process?). We want to improve our ML models with sentiment analysis accuracy, add other sentiments like finding a use case for specific models like EmoRoBERTa, fine tune LLama further for our use case and use possibly Langchain and AutoGPT with vector databases for searching (rather than a traditional DB), as well as much more integration with NFT features using the Verbwire API and smart contract deployment interactions. We wanted to work on this after the hackathon, and implement the full list of features in our POC into our full stack web app, and possibly launch on Product Hunt. We want to make this an application that is easy to use, relaxing, and can help others grasp how important mental health is, and a simple way to go about trying to make life less stressful.