Inspiration
There is an urgent necessity to advance the quality and speed of X-ray diagnosis. As it currently stands, there is an average delay of 14-days between performing an X-ray scan and an appointment with a specialist to discuss results. This gap not only hinders the diagnostic process but also leaves patients in a state of uncertainty and anxiety while awaiting further consultation with healthcare specialists. Considering that up to 25% of Xrays are rejected or must be repeated, this issue is exacerbated. Thus, we hope to tackle the problem of the absence of immediate rehabilitative guidance for patients throughout prevalent X-ray procedures.
What it does
Our tool combines supervised learning in image classification, and Retrieval-Augmented Generation (RAG) using the Gemini LLM, to detect and diagnose injuries based on X-ray images for patients. By applying transfer learning with specialized datasets on DenseNet and EfficientNet convolutional neural networks (CNNs), which were pre-trained on the ImageNet dataset, the tool approaches expert-level accuracy in detecting and classifying various skeletal fractures. Then, the RAG-powered rehabilitation agent allows for informative plans and recommendations, using clinically verified documents from a comprehensive medical database (PubMed).
How we built it
We divided our tool into two main development divisions: 1) X-ray fracture identification and classification, and 2) diagnosis and rehabilitation information generation. The first part of the tool uses a two-step computer vision pipeline, starting with a binary classifier to determine whether or not a fracture exists. The second step then identifies the location of the fracture (e.g, ankle). For the second part of the tool, we developed a RAG pipeline using LangChain to generate useful rehabilitation information related to the previously identified injury. To ensure relevant responses, we created a series of pre-defined prompt templates called "flows". So far, we have 4 primary flows: base diagnosis, heat & icing procedures, movement and lifestyle restrictions, and expected recovery timeline.
Challenges we ran into
In the development of the tool, our primary challenges involved ensuring a high degree of accuracy. With the CV models, we attempted to address low accuracy numbers by further performing extensive hyperparameter tuning. Additionally, we have identified overfitting concerns that we have made steps towards fully fixing. With our RAG pipeline, we identified accuracy issues and low-quality responses at times. We discovered that this was mainly due to our document retrieval, and fixed this issue by experimenting with various chunking methods, which resulted in far higher quality responses.
Accomplishments that we're proud of
We are very proud of our tool so far, as well as everything we learned along the way. Our results demonstrate that our initial goal of producing a pipeline capable of diagnosing a fracture given an X-ray was successful. For the CV models (identifying the presence of a fracture and classifying a fracture’s location), the DenseNet and EfficientNet models demonstrated high-quality results: following training and hyperparameter tuning, the models were able to achieve an accuracy of 86.60% and 91.30% respectively on an unseen ’test’ dataset. For the RAG pipeline, we are proud of the fact that we were able to generate high-quality rehabilitation responses, and are proud of the various pre-defined prompt templates (flows) that we created.
What we learned
The waiting period that exists between having diagnostic imaging taken and consultation with a doctor can be bridged, providing better patient care, speeding up recovery and reducing the period of uncertainty that follows injuries. In the development of our RAG pipeline, we learned a lot about various techniques to improve the quality of results and document retrieval, such as various chunking methods, as well as optimizing the different parameters. We learned a lot about maintaining abstract code through the usage of LangChain, making it easier to test various LLMs and embedding models. Additionally, in the development of our CV pipeline, we learned a lot about hyperparameter tuning
What's next for X-Care
Looking ahead, our focus will be on improving our machine learning models and user-facing website for practical application in healthcare. This will entail engaging with clinicians in real hospitals, and medical researchers at Universities to deeply understand their needs and how our product can be best integrated into clinical workflows. We aim to refine our model’s accuracy and architecture through rigorous testing and iterative development. Simultaneously, we plan to broaden our portfolio of tools, ensuring a comprehensive suite that addresses a wide range of medical needs.
Log in or sign up for Devpost to join the conversation.