Inspiration
The inspiration behind Sightra came from the everyday challenges faced by people with visual impairments when they need to understand what is around them. We wanted to explore how AI and computer vision could act as an accessible “second pair of eyes,” helping users independently interpret their surroundings and access visual information through audio.
What it does
Sightra is an AI-powered visual assistant that uses a camera to understand a user's surroundings in real time. It detects objects and interprets scenes using computer vision, then uses AI to generate contextual descriptions that are delivered through voice. This can help users identify objects, understand environments, and become more aware of what is happening around them.
How we built it
Sightra combines OpenCV and YOLO-based computer vision for real-time visual processing and object detection, with Gemini multimodal AI for understanding scenes and generating contextual descriptions. Text-to-speech converts the AI's responses into audio, creating a hands-free interaction experience. The system was developed as a lightweight prototype focused on real-time accessibility and practical usability.
Challenges we ran into
One of our biggest challenges was balancing real-time performance with AI accuracy. Vision models need computational resources, while multimodal AI introduces latency and depends on network connectivity. We also had to design the experience so that descriptions are useful rather than overwhelming, especially when multiple objects or visual elements are present simultaneously.
Accomplishments that we're proud of
We successfully brought multiple AI components together into a working accessibility prototype. Sightra can capture visual information, detect objects, interpret scenes, generate meaningful descriptions, and communicate the results through speech. We're particularly proud of demonstrating how existing AI technologies can be combined into a solution addressing a real-world accessibility challenge.
What we learned
We learned that building an AI accessibility solution requires more than simply achieving good model accuracy. Latency, clarity, context, usability, and accessibility are equally important. We also gained practical experience integrating computer vision, multimodal AI, and speech technologies into a single real-time pipeline.
What's next for Sightra
Our next step is to make Sightra more reliable and useful in real-world environments. We plan to improve real-time performance, add richer scene understanding, navigation assistance, text and document recognition, and offline capabilities. We also want to conduct user testing with people with visual impairments to ensure the product is genuinely useful, intuitive, and accessible.
Log in or sign up for Devpost to join the conversation.