PillSnap

Snap a photo to identify any pill, save its details, and get drug interaction warnings, including with food, so you never misuse your medication or forget what it was!

Comment 1

Inspiration

Around 3.5 billion people take medication each year, many of whom lose their bottle or forget to take their meds. This greatly impacts adults ages 65+ with 40% of older adults regularly taking 5+ medications and 57% of older adults forgetting to take their medicine in the United States. PillSnap was inspired by the everyday struggles of managing medications—misplacing pills, forgetting their names, or missing doses. Whether it's a single prescription or a handful of vitamins, keeping track can be overwhelming. We wanted to create a simple, tech-powered solution that helps people feel more in control. By using computer vision to identify pills from a photo, PillSnap ensures users always know what they’re taking, stores that information securely with Auth0 login, and provides smart warnings on drug interactions, both between medications and with food. You can also easily search through your saved data by name, shape, color, or dosage. It's peace of mind in your pocket.

What it does

PillSnap allows the user to take a photo of any pill, and we will identify it using our fine tuned Gemini API for image description and drugs.com to get accurate up to date FDA drug information. We provide the most likely drug candidates and the user can select one to save. PillSnap will also alert users to any potential drug interactions, including warnings about how certain medications might interact with food. Users can search through their saved pill data by name, shape, color, or dosage to quickly find what they need. We created an easy to navigate UI with high contrast values and a clear font to be disability friendly. This is available as an app and on our website: pillsnap.tech

How we built it

We built PillSnap using React and TypeScript on the frontend, and Python and Flask on the backend. We used and hosted our website using Expo, and got our domain through .TECH domains. We integrated Auth0 for secure login. We added a camera feature so users can take photos. We fine tuned Gemini Flash 2.0 API using Vertex.AI to get a description of the drug in our image, and queried the result by web scraping drugs.com with selenium to get accurate up to date FDA drug information. We saved this information to asynchronous data storage so users can access the information whenever they want.

Challenges we ran into

Since it was our first time working with Flask, we faced challenges in configuring the server to correctly handle requests and responses. Setting up the endpoints, managing JSON data, and ensuring compatibility between the frontend and backend required significant troubleshooting. We also had trouble launching our website onto the .TECH domain, as we had difficulty finding where to host it since we had both a frontend and a backend to run. We also had to spend many hours on testing the website due to it functioning differently than locally.

Accomplishments that we're proud of

We're proud of making an interactive, reader-friendly website that lets any user identify their pills whether online or through our app. It was important to us that users could immediately understand how to use PillSnap and why it matters, so we used a clean, intuitive design with high contrast and easy to read fonts for disability friendly access. On the technical side, we built a full-stack application that combines frontend and backend development, computer vision, and live data integration. Our team successfully connected React and TypeScript on the frontend with a Flask and Python backend, even though it was our first time working with Flask. We were able to overcome significant challenges in setting up routes, handling requests, and ensuring smooth communication between the two ends of the stack.

We also implemented a working system that allows users to take a photo of any pill and receive a likely match, using a fine tuned Gemini API paired with live data from drugs.com. From there, users can save pill details, search through their saved list, and receive alerts about drug interactions. We’re especially proud of how seamless and practical the experience is as users can go from snapping a photo to getting useful, personalized alerts in just a few steps. All of this is available both on our mobile friendly website and through our app, and we even secured our own domain, pillsnap.tech, to make the experience feel complete and professional. We even managed to add Auth0 for security purposes. This project pushed us out of our comfort zones and helped us build something meaningful, with the potential to help millions of real people stay on top of their health.

What we learned

We learned many things about the medical industry and programming. Thanks to the Patient Safety Workshop and our outside research, we learned about the challenges patients face when managing multiple medications, the risks associated with pill misidentification, and how technology can play a key role in reducing medication errors. We also gained a deeper understanding of the importance of accurate drug information, secure data handling, and building accessible tools that can empower users to take control of their health. We realized just how important this issue is to the disabled, elderly, and the poor. On the technical side, we improved our skills in full-stack development, integrating AI tools like Gemini and Vertex, and connecting data pipelines from web scraping with selenium to drug and food interaction warnings. We learned how to use Auth0 and new languages. Most importantly, we learned how impactful technology can be when it's designed with empathy and real world use cases in mind.

What's next for PillSnap

We hope to add even more functionality to PillSnap by adding trackers and reminders so users can get notified when they miss taking a pill.

Built With

Share this project:

Updates