MED-iator | Devpost

Inspiration

The inspiration behind our project stemmed from how many patients and doctors often have unhelpful conversations, both from the patient's side, when they often exaggerate or mislocate their symptoms for a condition, and from the doctor's side, when they might prescribe the wrong medication or the right one but the wrong dosage.

What it does

To help alleviate the problem, we wanted to create an actively listening (live recording of conversation) AI model that listens to the conversations between patients and doctors and identifies contradictions. In a perfect world, a patient thoroughly describes his/her symptoms, and the doctor, from their extensive medical experience, responds with either the perfect medication and dosage amounts or a structured treatment plan to solve the patient's medical challenge. While that covers patient-doctor potential medication errors, another problem that we wanted to solve was potential risks and unwanted mistakes during a surgical operation on a patient. In a similar fashion to the previous screen, we achieved this by using an AI-trained model designed to carefully listen to the conversations exchanged between anesthesiologists and surgeons throughout the whole operation, and generating questions on our web app to be shown to a mediator. The mediator will receive these questions and will be able to immediately inform the surgeons of any potential mistakes they might be making during the operation to hopefully avert a problem that can be avoided, and once again in a perfect world, our AI will not create any questions for the mediator to answer as the operation is moving smoothly.

How we built it

The framework for this app included numerous technologies, including React JS for the frontend framework, a Python Flask server, Langchain, and OpenAI chatbots for the backend and Artificial Intelligence framework and models, and even a 3D design tool called Spline that can be embedded on the frontend for an unimaginable interactive and visual user experience on a web app.

Challenges we ran into

One of the biggest challenges that we ran into was embedding the Spline framework into our web app, as at first, we simply wanted to take an embedding that Spline provided us and we hoped we could make design edits through the Spline editor itself. However, that wasn't the case, and instead we needed to develop code into our React web app to bring this Spline 3D interactive framework to life. The point of doing this was because we designed 8 small circles on the Spline UI web screen, designed specifically for a user to interact with our web app post-operation, allowing them to see vibrant colors, improve their moods, and even give them an activity for a couple hours before they are hopefully discharged. Almost like a video game, the user needs to move a ball around to hop on one of eight circle tiles, where each one provides either a key detail from their surgical operation (where the circle tile is highlighted in red) or a next step to take in terms of medication and prescriptions to receive a speedy recovery (where the circle tile is highlighted in green).

Accomplishments that we're proud of

An accomplishment that my team and I are proud of were using Spline in our React framework and creating a live-recording Artificial Intelligence model that is capable of accurately listening to a complete conversation and give the best feedback that it can in terms of the scenario, whether during operation or even just a simple patient-doctor scheduled appointment.

What we learned

We improved our skills in using React JS, Python Flask, ChatGPT chatbots, and we learned how to use the OpenAI Whisker API Model (which is the AI model used to listen onto conversations to take as an input file for its AI processing) and Spline.

What's next for MED-iator

Hopefully our app, MED-iator, receives outstanding feedback from our judges, with a greater focus on doctors from The Johns Hopkins Hospital itself, allowing us to improve our highly technical AI models and interactive Spline framework embeddings to take this medically focused application to a production level.