PharmFill

From doctor's note to pharmacy 📜, using AI for quick prescription transcription and patient info storage 🚀💊

Comment

Inspiration 💡

Every day, doctors across the globe use advanced expertise and decades of medical breakthroughs to diagnose patients and craft unique prescriptions. The inspiration for this mobile application stems from the irony that the result of such precision is without fail, the chicken scratch found on a doctor’s note. The physician not only entrusts that the patient will keep track of the crumpled paper but they require that the pharmacist on-call will understand their professional scribble. The plan is to create a platform that leverages technology to streamline the prescription filling process, making it easier for doctors to authenticate their work and for patients and pharmacists alike to be confident in their prescriptions.

What it does 🚀

The mobile application is designed to streamline prescription filling for patients, physicians and pharmacists. It starts with the written doctor’s note which is scanned via the mobile app and transcribed in real-time, allowing for them to confirm the prescription, directly edit it or retake the scan. This allows physicians to authenticate the interpretation of their prescription before shipping it off to the pharmacy via a shared patient database. During registration, the patient volunteers personal information which populates the database and ensures that general questions such as age, address and insurance are only to be answered once. In conjunction with the transcribed prescription, this information will be used to fill any necessary pharmaceutical forms that are scanned via the app. With completed paperwork, a transcribed description and verified patient information, pharmacists are significantly less likely to make errors fulfilling patient orders.

How we built it 🏗️

To produce this mobile application, we utilized a diverse technology stack to integrate various components and create an uninterrupted product experience. Using our defined user types (patient, physician and pharmacist), we derived the necessary functions for each, prioritizing and placing them to create an intuitive UI. This paved the way for early design wireframes and an eventual high-fidelity Figma prototype which directed our front-end development in ReactNative. On the back-end, the light-weight Python framework, Flask, was used to handle registration, data transferring and transcription. Our application required keeping track of a large amount of data, which we stored/accessed within a Redis Cloud database. In order to accurately interpret text from forms and notes provided by doctors/pharmacies, we utilised the Google Cloud Vision OCR API as well as Gemini Pro, providing users with accurate transcriptions within images that were processed with the Pillow API. Then, in order to autocomplete forms with efficiency and accuracy, we deployed the OpenAI API as an LLM model, generating prompts through information found within the Redis Cloud database to fill out forms with the correct answers. The back-end of our project was developed parallel to our front-end, which was entirely built using react-native, capable of supporting both android and IOS devices. By using the navigation library, the various components of the application are split into their own pages with styling and functions unique to each one.

Form filling process:

  • Read form
    • Extract text and location using Google Cloud OCR
    • Group text into coherent groups (Vertical + horizontal coordinate comparison and LLMs
    • Detect fillable fields and clean punctuation (LLM and Python)
  • Answer fields (LLM)
  • Write to form (Python Pillow library)

Challenges we ran into 🧩

Throughout our project, we faced several obstacles which we ultimately overcame with perseverance and targeted learning. Before writing a single line of code, we discussed our interests, skillsets and project ambitions, finding clear differences which would necessitate a degree of compromise. Moreover, we decided to implement a tech stack with Flask on the backend and React-native on the frontend, which proved to be frustrating as they could have been much more complementary. As a result, we had a handful of avoidable complications given a better-composed tech stack. Given the shorter nature of this particular hackathon, we were very much forced to stick with the decisions we made early on without much room to pivot, greatly improving our critical thinking and debugging skills. Another issue we ran into was the inconsistency that we found within practices in the medical industry, as doctors notes and pharmaceutical forms often differed widely from one person/company to another, meaning we had to build scripts that would satisfy a wide variety of possibilities.

Accomplishments that we're proud of 🏆

Ultimately, we are extremely proud that we were able to successfully build a full-stack mobile application within a 24-hour window, especially given that 3 of our members were not particularly experienced in app development before this hackathon. However, we were all able to find ways to contribute to the project whether it be through design elements, programming, or pitching. We are also extremely happy with the number of different libraries/APIs that we were able to put to use in this project. Having the experience of working with these APIs, many of which were for the first time, was extremely exciting for us and allowed us to build a product that we found to be extremely interesting.

What we learned 🧠

Through this experience, all members of our group came away with an enhanced skillset. We each improved our developing expertise as despite our lack of experience with the chosen technologies and tools, we ultimately built a strong project that applied those tools well. Beyond just coding, we also improved our product thinking, prioritizing the needs of each end-user to design a well-rounded and truly valuable mobile application. Above all, we learned the importance of programming as a collaborative process, finding success by sharding responsibility between members and optimizing workloads with everyone's unique skill set. We are beyond excited to apply our updated mindset to participate in future hackathons.

What's next for PharmFill 🚀

We are interested in continuing our work with PharmFill, with a focus on adding more features and polishing those already implemented. Our belief in PharmFill's potential to significantly impact the medical industry fuels our enthusiasm. A key area of development is the creation of more complex form-filling algorithms, capable of handling all types of forms. This advancement will not only enhance the app's functionality but also solidify its position as a transformative tool in healthcare.

Built With

Share this project:

Updates