Les yeux | Devpost

Inspiration

For people who are visually impaired, they would need help to find objects in a room as well as to navigate themselves when they are outdoor.

What it does

Through audio input, the user enters what s/he wants to find, be it water bottle, phone, or even elevators. The app will then scan the surrounding to try to detect that particular object. Once detected, the app beeps to indicate the direction.

How we built it

For the backend, we use YOLOv8n for general detection, and trained our own custom-trained model for STM elevator sign detection. We built a dataset which originally includes 102 photos, mostly taken by ourselves and labeled by our own, powered by roboflow. Preprocessing and Augmentation steps were introduced to improve its generalization ability. For the frontend, we use VueJs to design an user interface that ensures maximum ease-of-use. The accesses camera and microphone through APIs, and communicates with the backend through the agreed-upon endpoint. Additionally, we use the state management feature of vuejs to keep track of application status such as isRecording.

Challenges we ran into

We initially thought of building an Android mobile app; however, speech recognition is not possible on the devices we have. Therefore, we decided to build a web application instead.
The accuracy of the model decreases drastically when we first integrated the frontend and backend together. Patches were added to the frontend in order to make the app work.
Training a model is time consuming. Instead of merging the new feature (STM elevator sign) into the original model, we use two models (YOLOv8n and our model) in parallel. The outcome is the same as there is one detecting model.

Accomplishments that we're proud of

made our own dataset
trained our own cv model for STM elevator sign

What we learned

Create a dataset and label
Train a CV model
Create a frontend using vueJS ## What's next for Les yeux Deploy the app as a shortcut that's available on phones.