PictoPill

Helping the elderly identify and understand their medication, aiding them with their dosage schedule using augmented reality and visual recognition.

Comment

Inspiration

The beauty of a hackathon revolves around creating something uniquely powerful and innovative. But a whole segment of the population disproportionately misses out on this experience. This isn't because they don't want to, or because they are uninterested, but because they can't or, even worse, we don't allow them to. I am referring to the elderly who, despite the continuous cycle of invention we live in today, find themselves getting increasingly left behind. The older you get, the harder it is to find employment if you were to lose your job because of the steep, technologic learning curve. As we as people get older, our vision worsens, our motor function becomes limited, and our memory starts to fragment. Technology can solve these problems, and many amazing developers and teams try, but sometimes there isn't enough attention dedicated to this issue.

This is why we wanted to focus on the elderly, and this is why we decided to make our project about their healthcare.

What it does

Our app, PictoPill, identifies prescription drugs through the barcode on the bottle/box and, with the information that comes with it, becomes a tool to aid older people with their medication. Once a label is scanned, an automatic dosage schedule is created with notifications to remind the user when to take their daily prescription. A simple UI easily lets the user know what medications they are on and the schedules for them.

How we built it

PictoPill is an Android app built with Firebase and AR Core

In order to do so, we researched and implemented solutions in multiple different APIs, including:

-DailyMed (Drug Code Lookup)

-Good RX

-Room Persistent Database (Local Data Storage)

-Firebase ML Kit (Image Processing)

-Camera2 (Better Runtime and Memory Usage)

-Google AR Core (AR)

Challenges we ran into

Our workflow and version control was the greatest challenge that we faced during the development of this hack. Since one of our team members had never used git before, we helped him set it up, but not before he began working in a project directory different from that of our repository. Since he was also the one with the most android experience out of all of us, this meant that we ended up with two halves of an app that needed to be adapted to work with each other. This added substantial time and difficulty to our project which could have been avoided. Working with AR and databases proved to be a lot more challenging than expected.

We know this is a lot of reading but some good news next:

Accomplishments that we're proud of

Victories are personal. So, in order of first names, here is our team to explain what we're individually proud of accomplishing.

Cesar: This is my first hackathon. I came in with no knowledge of coding, without any friends, without a team, and without an idea. After a little more than a day, I have all four, including a rudimentary knowledge of .xml and Android App development, and a personal mission.

Karan: Being able to develop a full backend and working persistent database system from one very undocumented set of information. Downloading and processing data in Dailymed was a time-consuming process and being able to learn how to create a room database backend would be something that made the results much cleaner.

Nitin: Managed to understand and apply complex API's to create a powerful project. Responsible for setting up the ArCore, Camera, and Firebase Vision API. Learned how to enable these API's to interact with each other smoothly even though the interfaces were originally not configured to interact with each other. Other back-end work included setting up methods to pass image sensor data straight to Firebase so that it could be parsed and then projected utilizing ArCore.

Robert: Laying out the foundation for the database and trying to help make the two parts of our project interface in a meaningful way.

What we learned

We learned how to work with Android Studio in app development and we gained practical experience with databases and datasets. We also gained practical knowledge with integrating APIs and learned of the complexities that come with working with Augmented Reality. On a more technical sense, we experimented a lot with software and languages that were new to some of us like Firebase and Google AR Core.

What's next for PictoPill

Aside from fully developing the augmented reality and scheduling capabilities, an important part of this app is accessibility, so we chose a color scheme that was AAA and AA compliant. However, we believe we can do better, so through user testing, we hope to create a set of themes were the user can select their preferred colors, all of which have a good contrast ratio to make using the app easier for the visually impaired. Along those lines, we want to have options for people with different forms of color blindness, like protanopia, deuteranopia, and more. Finally, we want to fully leverage the set of features that are already available for app developers, like options to adjust font sizes and spoken narration, in a way that still makes the app easy to use.

We have some other ideas that go beyond the main scope of the app, but these ideas completely depend on the feedback from the community we are trying to help. If we have the potential to truly help those with accessibility needs, then that is enough of a reason to keep moving forward and thinking bigger.

We may also change our name :)

Built With

Share this project:

Updates