Inspiration Digitizing medical records is essential for safety. Handwritten prescriptions remain a major source of disorder and risk in healthcare. Our inspiration came from the critical need to mitigate preventable medical errors—such as misinterpreting "150mg" as "15mg" or confusing similarly named drugs. In high-stakes clinical environments, illegible handwriting isn't just an annoyance; it's a safety hazard. We wanted to build a digital "second pair of eyes" for pharmacists to automate data entry and prevent severe consequences.
What it does Intelligent Prescription Digitization is a specialized OCR tool designed to convert analog physical records into structured, searchable digital data. Pharmacy Automation: It deciphers handwriting to reduce the manual data entry burden on pharmacists, allowing them to focus on patient advice. Error Reduction: It acts as a safety layer, flagging potential interpretation errors before they reach the patient. EMR Integration: It bridges the gap between paper and digital, unlocking billions of old paper prescriptions for data analysis in modern Electronic Medical Record (EMR) systems.
How we built it Our methodology used a Resource-Efficient Transfer Learning approach, specifically moving away from GenAI to a specialized PaddleOCR pipeline. Phase I: Lexical Priming (1 Epoch): We tackled the "cold start" problem by training on a custom dataset of 3,000 augmented samples (medical dosages and cropped drug abbreviations) to prepare the model for specific medical character shapes. Phase II: "Gold Standard" Fine-Tuning (2 Epochs): We used a Weak Supervision method. We collected 130 complete prescription images and used the Google Vision API to generate "Gold Standard" pseudo-labels. Distillation: We fine-tuned our lightweight model on this high-quality data, effectively distilling the strengths of a large commercial model into our resource-efficient design.
Challenges we ran into Generative AI vs. Accuracy: Our biggest hurdle was the realization that "Prompt Engineering" with VLMs (Vision-Language Models) was fundamentally unsafe for this task. The models would hallucinate drug names or dosages that looked plausible but were incorrect. We had to make the difficult decision to scrap that approach and pivot to a deterministic OCR pipeline. Extreme Handwriting Variability: "Doctor's handwriting" is notoriously difficult to read. We faced significant issues where standard OCR models failed to distinguish between similar-looking characters (like 'l' vs '1', or 'q' vs 'g') when written in hurried cursive styles common in clinical settings. Noisy Input Data: Real-world prescriptions are often crumpled, stained, or photographed under poor lighting. Pre-processing these images to remove noise without erasing faint pencil strokes required extensive trial and error with various image processing filters.
Accomplishments that we're proud of Drastic Accuracy Boost: We improved performance from a baseline (Raw OCR) of 30%-50% accuracy to a final fine-tuned accuracy of ~90%. Efficiency: We achieved high performance on limited hardware by emphasizing data quality over model size. Successful Distillation: We successfully proved that we could match the utility of massive commercial APIs using a smaller, fine-tuned local model via our two-stage curriculum learning strategy.
What we learned Generative AI isn't always the answer: We learned that for high-stakes, data-entry tasks, deterministic models (specialized OCR) are safer and more accurate than generative ones (LLMs/VLMs). Curriculum Learning Works: We gained deep insight into how "priming" a model with augmented data before fine-tuning on real data can significantly boost performance with very few epochs.
What's next for Intelligent Prescription Digitization Scaling Data Quality: While our weak supervision method worked well for the prototype, we need to curate and train on a much larger dataset of human-verified, high-quality medical prescriptions to push accuracy from 90% to 99%. Custom Loss Function: We plan to design and implement a custom loss function specifically tailored for medical imagery. This will penalize the model more heavily for misclassifying medically critical characters (like dosage numbers or units) compared to standard text errors. Real-time Deployment: Optimizing the lightweight model for edge devices so pharmacists can scan prescriptions via smartphone instantly
Log in or sign up for Devpost to join the conversation.