-
-
Customized Body Model from Scans
-
Surgical Demonstration and Examination
-
Patient Examination of Their Own Organs
-
Children's View
Inspiration
Just one medical horror story or statistic is enough to deter many of us from seeking critical medical care. Even if there’s only a 1 in 10,000 chance of a complication, our complex human brains will find every possible way we’re going to be that 1—especially if we don’t know what will happen to us once we go under. According to the National Institute of Health (NIH), up to 92% of patients experience anxiety and doubt before going under an operation. There are many physiological symptoms of heightened anxiety, but the most impactful one is psychological: patient mistrust of doctors. When patients do not trust their medical professionals, they end up refusing life-saving care—which can have permanent consequences.
Patient distrust stems from a lack of transparency. The flood of medical jargon that many receive before surgery only increases their fear, rather than builds understanding. We see XR as a connective technology that can help make some of these discussions more intuitive and less intimidating.
What it does
Dr. Trust is a Mixed Reality application built on Meta’s Presence Platform that helps facilitate difficult medical conversations. It will allow doctors to more effectively educate and reassure patients by clearly visualizing what the problem is and how it can be treated.
For this hackathon, we specialized Dr.Trust in visualizing the pre-surgery consultation of a lung removal surgery. Doctors and patients can annotate a 3D model of the body, calling attention to specific areas and lifting models of organs out to highlight their relevance. It supports both an avatar appearance and a pass-through real-life appearance for the doctor and the patient to accommodate both in-person consultations and remote virtual consultations.
How we built it
We used Meta’s Presence Platform and OpenXR on Unity to support Mixed Reality features, and the XR interaction toolkit to support a variety of features on Oculus Touch Controllers (preferred for their high degree of precision).
We also wanted to create a real-time collaborative experience for patients, doctors, and family members, whether they’re in the same room or across the world. We used Normcore to host the app on a remote server, and enable live voice and hand interaction between multiple people.
Challenges we ran into
First, trying out new technologies! We challenged ourselves to try spatial design software that was new to us, such as Bezi. This turned out to be the best kind of challenge - we clicked with Bezi amazingly well, and it was a valuable asset for our Unity development.
On the flip-side, we tried the new building blocks in Unity and ran into substantial challenges. After spending nearly 5 hours on Friday night, we decided to pivot after learning that other teams were facing the same issues. This adaptability allowed us to adjust and build a well-functioning project within the time constraint.
Accomplishments that we're proud of
We’re most proud of our team dynamic and the way we were able to seamlessly work together. Our project’s name is Dr. Trust, and trust in each other is exactly what the doctor ordered for this hackathon! Some of us took a backseat in areas where we usually lead, others trailblazed in unfamiliar areas to them. The result was a group of people that wholeheartedly trusted each other - even as exhaustion from an all-nighter set in.
This allowed us to move faster and not be scrambling at the last minute the way most of our prior hackathon experiences have been. And the vibes within the team were impeccable throughout.
What we learned
First, it can be fun to take a backseat from areas one is typically experienced in. People who go to multiple hackathons sometimes tend to fall into niches - “I’m a technical person”, “I’m a designer”, “I’m an idea guy”. By relinquishing that self-framing and trying new things, we can grow so much more.
Second, things don’t always go as planned in the demo as they do in a live test. Even after setting up our project’s real-time collaboration to work seamlessly when we tried it out, when we ran it for judges while everyone was using the WiFi, there was sometimes lag or disconnection.
What's next for Dr. Trust
Our vision for Dr. Trust is not just limited to pre-surgery consultations. Visibly seeing the condition of their body is an incredibly powerful experience, and that can be expanded to all medical diagnoses.
Log in or sign up for Devpost to join the conversation.