Dr.HowAI | Devpost

A sample question you could ask to the Doctor
A sample question with an image response

Inspiration

Three months ago, Vyas was preparing for wrist surgery. When he asked the doctor for a detailed explanation, the response was unclear and difficult to understand. This left Vyas with many unanswered questions. After searching for more information online, he found that the resources lacked the specificity he needed. Even chatbots like ChatGPT didn’t help—some even provided made-up sources. This experience motivated us to create a platform that makes medical information more accessible, breaking down complex yet essential concepts for the average person.

What it does

We’ve built a Question-Answer platform where users can ask Dr. HowAI their medical questions. Dr. HowAI consults a curated list of trusted sources—including publications, medical textbooks, and journals—using Multi-Modal Retrieval Augmented Generation (RAG) to generate accurate answers. Users can also hover over groups of sentences in the response to see which source contributed most to that portion. for the utmost transparency, Additionally, users can “like” or “dislike” specific sentence groups, helping Dr. HowAI adjust its future responses by weighting the sources accordingly. Dr.HowAI is also honest - if it is asked about a concept that is not within its sources, it will honestly say that it does not know the answer.

How we built it

The website was built using React, leveraging its strengths in component reusability, performance, and ease of integration with external APIs. Tailwind CSS was used for efficient styling. The frontend communicates with a Flask API, which acts as a middleman between the user and the AI via HTTP requests.

To answer user queries, we implemented Multi-Modal Retrieval Augmented Generation (RAG) alongside a vector database. First, we used CLIP, a visual-language model, to encode our resources—textbooks, articles, and images—into the Chroma vector database. When a user asks a question, it is also encoded by the CLIP model, and we query the encoding in the vector database to find relevant answers.

Instead of traditional cosine-similarity matching, we implemented a custom algorithm inspired by Reinforcement Learning. Specifically, we use a variation of the Upper Confidence Bound (UCB) algorithm to determine the weight each source should contribute to the response. The UCB algorithm encourages exploration by prioritizing sources that haven’t been used much, while gradually favoring those that receive positive feedback from users.

We then multiply the UCB weighting by the cosine similarity scores to select the best text chunks for the model to use. Finally, we use Claude 3.5 Sonnet to generate intelligent responses based on the selected chunks. If the highest weighting score is below a threshold, the Claude model will be prompted to say that it does not know the answer to a query,.

Challenges we ran into

One of the biggest challenges was integrating our UCB weighting system with the RAG pipeline, as most libraries abstract many of these processes. This required us to gain an in-depth understanding of embedding spaces and transformer self-attention mechanisms in a short period of time. This way, we could integrate our custom vector-querying system with the ChromaDB pipeline.

On the front end, designing a user-friendly interface was no easy task. We went through four different iterations of the Question-Answer system, struggling to decide how users should interact with the responses and rate each source.

Accomplishments that we're proud of

We are really proud of how we integrated many complex components into a cohesive solution. Writing the custom RAG code was particularly rewarding, and the overall user experience of the website turned out to be smooth and intuitive.

What we learned

We gained a lot of knowledge in front-end design, particularly with React and Tailwind. On the technical side, we learned about machine learning techniques and tools such as PyTorch, ChromaDB, and multi-modal analysis. We also developed a deeper understanding of the medical field and how to make complex information accessible.

What's next for Dr.HowAI

Looking ahead, we aim to integrate more medical sources and potentially crowdsource resources from trusted medical professionals. We also plan to launch the platform online to gather more user feedback, which will help train and improve our weighting algorithm. Dr. HowAI’s mission is to make medical knowledge more transparent, and we are excited to continue that journey beyond this Hackathon.