-
-
Dr. Andy
Inspiration
A soldier lies wounded on the battlefield, miles from the nearest hospital. In a remote village, a child suffers from a life-threatening condition with no solution in sight. But what if the world’s best surgeons could operate from anywhere—no matter the distance? Enter Dr. Andy: The Surgeon in a Box. With just a VR headset and robotic arms, life-saving medical procedures can be applied across the globe in real time. The future of remote medicine starts now. In crisis zones and remote areas, access to surgical expertise is often a matter of life and death. Our team was inspired by the idea of eliminating geographical barriers in medicine, allowing highly skilled surgeons to perform life-saving procedures anywhere in the world. With advancements in robotics and VR hand tracking, we saw an opportunity to create a system that could translate a surgeon's precise hand movements into robotic actions with minimal latency.
What it does
Dr. Andy – The Surgeon in a Box consists of two components. In the field, a pair of robotic arms and stereo camera system that interfaces with a surgeon or other operator wearing any available VR headset that supports the widespread and open-source OpenXR standard. Dr. Andy allows skilled professionals to apply their skills whenever and wherever needed, without putting their lives in danger. The system leverages high-fidelity hand tracking and haptic feedback to ensure precise, real-time control. It’s designed for delicate mechanical procedures, making remote operations in war zones, disaster areas, or underserved regions a reality.
How we built it
We integrated Meta Quest 3’s hand-tracking capabilities with a robotic system using real-time kinematic modeling. Our software pipeline processes the surgeon’s hand movements and translates them into commands for the robotic arms with near-zero latency. We used ROS (Robot Operating System) for communication between the VR input and robotic actuators and implemented a machine-learning-based stabilization algorithm to enhance precision.
Challenges we ran into
One major challenge was minimizing latency between the surgeon’s movements and the robotic arms’ response. Even a slight delay could introduce risks in a surgical setting. Additionally, ensuring smooth, natural motion for complex procedures required fine-tuning our inverse kinematics algorithms. Finally, developing a user-friendly interface that felt intuitive to surgeons unfamiliar with VR control was a significant hurdle.
Accomplishments that we're proud of
We successfully demonstrated real-time remote control of robotic arms with sub-50ms latency, an achievement that pushes the boundaries of remote surgery. The system’s precision exceeded our initial expectations, allowing for fine motor control necessary for surgical tasks. Seeing our prototype perform delicate procedures—like tying a suture—was a huge milestone for our team.
What we learned
We gained deep insights into real-time robotics control, VR interface design, and the practical challenges of telemedicine. Understanding the importance of low-latency communication in medical applications was eye-opening. Additionally, we learned the value of interdisciplinary collaboration, as our project required expertise from robotics, software engineering, and even medical professionals.
What's next for Dr. Andy
The next step is refining the system for clinical-grade reliability and conducting user testing with surgeons. We plan to integrate AI-assisted stabilization to compensate for small errors in hand movements and explore 5G and satellite connectivity for global accessibility. Ultimately, we aim to partner with medical institutions and humanitarian organizations to bring Dr. Andy into real-world surgical applications.
Log in or sign up for Devpost to join the conversation.