-
-
Sample Xray
-
Dashboard
Inspiration
Nadia was initially inspired by everyone getting sick with no official diagnosis. One of the team members is sick right now and the other two got sick twice in a row. None of us went to the doctor to get a diagnosis to see if we had something more serious, and we naturally realized that those who don't have easy to a doctor can't either. After further digging, we saw that 3.9 million deaths per year are attributable to lower respiratory infections globally (WHO, 2023). The primary clinical tool for detecting internal conditions — stethoscope listening — requires a trained physician, specialized hardware, and an in-person clinical visit. For the vast majority of the global population, especially those in rural or underserved areas, this creates a hard barrier between symptom onset and diagnosis.
What it does
Nadia, short for Not A Doctor but an Inference Assistant, is a web-based AI diagnostic tool that takes in wavelength recordings of users breathing from a stethoscope and coughing and outputs what disease a user most likely has with a confidence score. It also outputs sample X-rays of lungs that have the disease and labels of what differentiates that from a normal set of lungs.
How we built it
We first normalized and standardized data from Kaggle using pandas that had wavelength recordings of stethoscope data and coughs. Then, we applied audio augmentation techniques like adding Gaussian white noise, time stretch, pitch shift, and time shift so that we could get 6000+ samples to train a neural network on. Then, we applied a Fourier transform to all those samples to get 6000+ spectrograms that had the metadata (age, gender, ID). Then, we trained an EfficientNet CNN model on that data. We also trained another neural network on the metadata on age and gender and combined the results to get a cohesive conclusion. We fetched the result from the model and used a hash map to match the disease to a labelled X-ray about what in the lungs indicate the disease. To create the labeled X-ray, we passed in a PNG file of a sample X-ray to Gemini and type of disease and it returned where and what about the X-ray signaled a disease. Then, we used Figma Make to create the actual labelled X-ray, using the coordinates Gemini gave to create a coherent label for each disease in the model. We used React for the frontend and FastAPI for the backend.
Challenges we ran into
Challenges we ran into was that the model wasn't generalizing well to new samples outside of the training set, and we eventually found out that the model was only trained on half of what it was intended. We realized that there was a problem with how the file names were created which caused old files to be rewritten with new files with the same file name, decreasing the number of spectrograms for the model to be trained on by 50%. Also, no one on the team was proficient in backend development so we had to learn how to connect everything after we got the model and Figma Make designs finished.
Accomplishments that we're proud of
We're proud of being able to make quick architectural changes on the fly, especially relating to the model and the data it took in. We're also proud of finding a way to use Gemini in a unique way outside of just a GPT wrapper.
What we learned
We learned how to perform data augmentation in audio data with the Gaussian white noise. We also learned how to fine-tune and use a pre-existing CNN model for an application with real impact. We also learned how to link everything using FastAPI and we strengthened our skills in React.
What's next for Nadia
We want Nadia to be able to provide a more cohesive output. We also want to be able to expand the diseases that can be predicted by the model because options the common cold aren't available because we couldn't find data on Kaggle for the common cold. We also want to be able to create 3D models of what each disease would look like since using Figma Make for them was very rigid and a bit unintuitive. We also want to be able to create a real representation of what the user's breathing pattern looks like and model any discrepancies caused by mucus in the lungs.
Log in or sign up for Devpost to join the conversation.