SpectaCare | Devpost

Inspiration

Doctors often need to look up symptoms and conditions after consultations, which can detract from the time and attention they give to their patients during the visit. The burden of paperwork, sometimes completed hours after the consultation, can result in key details being missed or overlooked. This not only adds to the stress and workload but also contributes greatly to burnout among medical professionals. SpectaCare aims to address these challenges by providing real-time support during consultations and streamlining the documentation process after a patient consultation through generating a preliminary report.

What it does

SpectaCare enhances the consultation experience by transcribing conversations between doctors and patients, providing key points and potential diagnoses in real time based on the discussion. It allows doctors to capture key moments with the Snap Spectacles camera, saving them as screenshots. And at the end of each consultation, SpectaCare generates a comprehensive, AI-powered report that can be edited before being saved as a PDF, helping doctors maintain accurate records effortlessly.

How we built it

We built our application using a multitude of services/frameworks/tool:

Front End: React.js was used for the core user interface, along with React-PDF for the PDF generation functionality. We also used Spline to create custom 3D models for a more engaging user experience.
Back End: We built our backend using FastAPI with Python, leveraging ChromaDB for data management. Our logic relies on multiple large language models (LLMs), including OpenAI, Claude, and Toolhouse, for accurate transcription and diagnosis generation.
Spectacles: We utilized Snap Spectacles and developed custom interactions using JavaScript in Lens Studio. Additionally, we employed SnapML for camera-based functionalities like image capture and real-time object recognition during consultations.

Challenges we ran into

This project was full of firsts for our team. It was our first time developing an AR application, and it was Byron's first experience as a designer (the challenge of having a team of all developers). None of us had ever used Snap Spectacles or worked with Lens Studio before, and the newness of the technology meant that documentation was limited. This presented challenges throughout development, from figuring out marker tracking to something as simple as getting the glasses set up. Huge shoutout to the Snap sponsors who were incredibly supportive and eager to help, which made a big difference in overcoming these hurdles and bringing our vision to life.

Accomplishments that we're proud of

Getting a finished product! Initially, our plan involved using a Meta Quest 3 w/ Unity, but we had to pivot quickly to try our hand with the Snap Spectacles and Lens Studio—technologies that none of us had used before. Despite these adjustments, we managed to deliver a dynamic and engaging front end, integrated it with a complex backend that incorporates multiple LLMs and tools, and linked everything up to bring our vision to the new AR platform.

What's next for SpectaCare

For the future of SpectaCare, we plan to refine our real-time diagnosis algorithms to ensure even greater accuracy. We also would want to expand the system to integrate directly with electronic medical record (EMR) systems will make SpectaCare an even more powerful tool for healthcare professionals.