Inspiration
PathTel encompasses a shared vision inspired by Telus' mission to accommodate and remove barriers for societal benefit: to transform the way people navigate indoors and interact with their surroundings. For many of us, the ability to move safely and confidently through an unfamiliar environment is something we take for granted. However, countless individuals face barriers due to visual impairments or disorientation in complex settings. We wanted to use generative AI to enable people with impaired vision to have a better sense of their surroundings.
What it does
PathTel is a smart assistant that uses your smartphone to observe and interpret your surroundings, enabling safer and more informed navigation. It combines object detection, QR-based localization, and facial recognition to provide a personalized and interactive experience. It personalizes the experience by allowing your friends to register their faces to be in your network and the app will tell you if they are in your line of sight!
Features
- Object Detection and Localization: The app identifies objects and hazards in your environment using your phone’s camera and QR codes strategically placed in the area.
- Real-Time Routing: PathTel maps out a safe path to your destination and allows you to navigate within a floor plan.
- Voice Aided Navigation: It provides audio instructions and descriptions of what’s happening around you, helping you build a mental picture of your environment.
- Facial Recognition for Connections: With built-in facial recognition, your friends and family can register their profiles. The app alerts you when someone in your network is in your line of sight.
How We Built It
- Object Detection: Utilizing YOLOv8 for efficient real-time object recognition.
- Video Scene Processing: GPT-4 Vision for processing video scenes.
- QR-Based Localization: Generating and reading QR codes to establish precise positional references in indoor and outdoor spaces.
- A* for Path Route Calculation
- Facial Recognition API: Implementing OpenCV for secure and reliable friend identification.
- Speech Processing: Using Google Cloud Text-to-Speech for natural-sounding voice guidance.
Challenges we ran into
Real-time location proved to be a big obstacle to our app. For navigating inside a building, extremely precise location is necessary to ensure that the directions given to the user are accurate. Collecting this data without having bluetooth emitters or another device was challenging. Our pivot was to use qr codes strategically placed in corners and and key areas of rooms so that the user's camera will capture them. By calculating the user's distance from qr codes, a real-time location can be computed.
Accomplishments that we're proud of
The friend of a user feature is a great addition to the app. Our hope is that it can remove a barrier to social interactions for visually impaired people. This feature can let users know if their friends are nearby, allowing them to call out their name and have a conversation that might never have happened otherwise.
What we learned
As mentioned above, obtaining real time location data to localise users is difficult! We were able to learn to use react to build a multi-page application. We also learned how to integrate front-end and back-end development to capture images and store them in the facial recognition database. It was very exciting to link our UI with our APIs.
What's next for PathTel
We believe putting up QR codes in public spaces and buildings would enable a straightforward scalability for PathTel. Awarding accessibility certifications would be a great way to incentivize people to put up the QR codes.
Log in or sign up for Devpost to join the conversation.