Inspiration
The global disparity in access to essential services for the visually impaired was a compelling reason to create SightCom 2. The aim is to leverage cutting-edge technologies to make everyday tasks more accessible for people with visual impairments.
What it does
SightCom 2 is a software prototype designed for Smart Glasses. It integrates various OpenAI and Clarifai technologies to provide features such as real-time speech-to-text, image captioning, OCR, and color recognition. With the help of a camera and microphone, SightCom 2 narrates the surrounding environment to the user through speakers.
How we built it
We used a combination of OpenAI's Whisper for speech transcription, GPT-3.5 as a large language model, and DALL-E for image generation. For image captioning, OCR, and color recognition, we integrated the Clarifai API. The software is hosted on Streamlit Cloud and will eventually be designed to run on a microprocessor that can be integrated into Smart Glasses.
Challenges we ran into
The primary challenges were designing an optimized algorithm that could seamlessly integrate the various technologies and ensuring that these components could work together in a prototype application that users could actually test.
Accomplishments that we're proud of
We successfully built a prototype that effectively combines multiple advanced technologies. Preliminary tests show promising results, with the software performing reliably under various conditions.
What we learned
We learned how to integrate various APIs and technologies into a cohesive solution. We also gained insights into the specific needs and challenges faced by visually impaired individuals.
What's next for SightCom 2
Our future plans include replacing the current LLM routing chain with a more efficient language-classification model. We also aim to train custom AI models to be deployed on dedicated servers for faster API calls. Future updates will incorporate new accessibility features, such as currency recognition.
Log in or sign up for Devpost to join the conversation.