Pharmascribee

PharmaScribe: Decode Prescriptions Instantly with AI-Powered Insights, Maps, and Voice

Comment 2

🩺 PharmaScribe: Decoding Prescriptions with AI Transforming handwritten prescriptions into clear, safe, and actionable healthcare data using LLMs + Voice AI.

33

🩺 Inspiration

Handwritten medical prescriptions are notoriously difficult to interpret, often leading to confusion or even medication errors. Patients, caregivers, and pharmacists frequently struggle to identify drug names, dosages, or instructions correctly. This challenge inspired us to create PharmaScribe — an AI-powered prescription decoder that transforms messy handwritten prescriptions into clear, structured, and actionable medical data.

Our goal is to empower everyone in the healthcare ecosystem — from patients and doctors to pharmacists — by reducing interpretation errors, improving medication compliance, and enhancing health literacy through AI-driven understanding and dietary guidance.

💡 What It Does

PharmaScribe offers a fast, intelligent, and accessible way to understand and act on medical prescriptions:

  • 📷 Prescription Upload:

    • Drag-and-drop or file select to upload handwritten or printed prescriptions.

  • 🔍 AI Extraction:

    • Extracts drug names, dosages, and frequencies.
    • Identifies diagnosis or medical conditions.
    • Provides drug benefits and use cases.
    • Flags food and drug warnings, interactions, and safety tips.

  • 🧠 Powered by Google Gemini 1.5 Flash:

    • Multimodal LLM for both image and text interpretation.

  • 🗺️ Location Services:

    • Detects real-time user location.
    • Displays nearby pharmacies on a live map with custom markers.
    • Provides one-click Google Maps navigation to the selected pharmacy.

  • 🎧 Voice Assistant:

    • AI chatbot responses are read aloud using ElevenLabs TTS.
    • Designed for elderly, visually impaired, or low-literacy users.

  • 📄 Enhanced PDF Reports:

    • Medication summary, interactions, contraindications.
    • Dietary do’s and don’ts, doctor’s instructions, and disclaimers.
    • Auto-download upon completion.

  • 💎 Premium Access Control:

    • Voice assistant and map features available only under subscription.

  • 🔐 Security & Compliance:

    • HIPAA-aware data handling.
    • HTTPS and CORS-secured architecture.

🛠️ How We Built It

  • Frontend: React + TypeScript + Tailwind CSS
  • Backend: FastAPI (Python)
  • OCR Engine: Google Gemini 1.5 Flash (Multimodal)
  • PDF Export: jsPDF + html2canvas
  • Deployment: Render (Backend), Vercel (Frontend)
  • Voice: ElevenLabs API
  • Map/Location: Leaflet.js + OpenStreetMap + Browser Geolocation
  • Search & Verification: Google CSE + Gemini Vision
  • Security: HTTPS, custom CORS, HIPAA-aware practices

🚧 Challenges We Ran Into

  • Structuring accurate and aesthetically clean PDF reports with real-time data.
  • Maintaining OCR precision with highly varied handwriting styles.
  • Enabling real-time voice responses and low-latency map interactions.

🏆 Accomplishments We’re Proud Of

  • 🎯 Achieved high-accuracy OCR on real handwritten prescriptions using Google Gemini.
  • 📄 Created automated, readable PDF reports including drug, diet, and safety information.
  • 💬 Integrated chatbot, image search, map, and voice modules into a seamless user experience.
  • 📱 Designed a responsive, user-friendly UI for both desktop and mobile platforms.

🔮 What’s Next for PharmaScribe

  • 💊 AI-powered reminder system for dosage alerts and medication schedules.
  • 🔐 End-to-End Encryption for medical data.

🩺 Challenge Compliance

  • **🎤 Voice AI Challenge:

Used ElevenLabs to integrate a voice assistant into PharmaScribe, enabling real-time audio responses for users.

This helps visually impaired, elderly, or low-literacy users interact with the app more easily, improving accessibility and engagement.

  • **🚀 Deploy Challenge:

Deployed the PharmaScribe app using Netlify for seamless front-end hosting and Render for backend services.

Built With

Share this project:

Updates