Inspiration
Our product draws inspiration from innovations like Meta Glasses and Humane AI. As portable technology continues to advance, we envision a future where blind individuals can navigate the world independently and confidently, without relying on a walking stick. By simply clipping a lightweight, wearable device to their clothing, users can receive real-time narration of their surroundings, turn-by-turn navigation to any destination, and seamless access to an integrated personal assistant that can answer questions about their surroundings. We believe this approach represents not only a powerful assistive solution but also a significant step toward greater accessibility, freedom, and dignity for the visually impaired community.
What it does
Nova Vision serves as a navigation tool for blind people. Powered by Livekit's AI agents, users are able to receive step-by-step bluetooth audio directions to take them from point A to point B. This, coupled with the integration of a YOLO Object Detection model trained on identifying people, stairs, and crosswalks, ensures that safe navigation is no longer a hope but a reality for those who are visually impaired.
How we built it
At its heart, Nova Vision is built using an ESP32-S3-CAM microcontroller. The backend, where the object detection, LiveKit API calls, and navigation services occur, is built with Python. Communication between the ESP and the backend is achieved using MQTT. Finally, multithreading is essential for smooth and seamless interactions between the user and the different features Nova Vision has to offer as well as obstacle detection running always in the background for safety.
Challenges we ran into
A major challenge we ran into was the lack of WiFi in the main hacking space. This significantly hindered our ability to make progress, as, from training the object detection model to getting the ESP to stream its camera input, a lot of this project required a working internet connection. What made things worse was discovering that even when the WiFi started to work, the ESP was simply unable to connect to it. Eventually, we had to resort to connecting it to our personal hotspot. For our navigation algorithm, we discovered that laptops use wifi triangulation which isn't too accurate, so we pivoted to use our mobile phones as a GPS source to calculate step by step directions with distances. Another issue we faced was optimizing the program for different APIs and figuring out an algorithm for dynamic multithreading with shifting priorities. Lastly, we also faced numerous CAD and 3D printing issues, where our CAD model doesn't print the way we speculated it would and the 4 total printers that were available all had some unexpected technical issues. Also, going through the documentation of different APIs and incorporating them for our use case was not easy!
Accomplishments that we're proud of
We take pride in several key achievements that exceeded our initial expectations. Among them is the successful design of a compact and intuitive solid model that maximizes functionality while minimizing unnecessary space. We also developed and trained an advanced object detection model, enabling Nova Vision to gain spatial awareness and accurately identify and interpret objects and obstructions in its surroundings. In addition, we successfully implemented voice-directed GPS guidance, allowing users to receive real-time navigation instructions through spoken feedback, another major step toward making Nova Vision a fully independent assistive companion.
What we learned
Throughout the process of developing Nova Vision from the ground up, we encountered numerous challenges, from GPS and internet connectivity issues while testing the map guidance feature to CAD modeling difficulties where our SolidWorks design faced tolerance and fitting problems. Despite these setbacks, we learned that innovation requires patience, persistence, and adaptability. We also learnt the valuable skill of pivoting to other tasks if one takes too long. The journey ultimately taught us that even when progress feels slow or frustrating, staying committed to your vision and continuously improving upon it is what ultimately turns ambitious ideas into reality.
What's next for Nova Vision
In-House Detection: Instead of relying on an external device to run the object detection model, future versions of Nova Vision will perform all processing directly on the ESP itself. This advancement will not only improve performance but also enhance privacy and data security, ensuring user information stays fully local.
Smart Memory: We plan to integrate a memory feature that automatically logs the location of essential personal items, such as keys, wallets, and AirPods, so users can easily locate misplaced belongings. This capability would make Nova Vision even more practical for everyday use.
Adaptive Model Switching: To ensure reliability in all environments, Nova Vision will include an intelligent model-switching system. When network connectivity is limited or unavailable, the device will automatically transition to a lightweight offline model that focuses on proximity alerts, warning users of nearby obstacles and keeping them safe from potential collisions or injuries
Log in or sign up for Devpost to join the conversation.