Sight Buddy

Your voice, your web, your way.Imagine a world where every website is easily navigable and understood, empowering blind users to connect with digital world. Sight Buddy makes that fantasy a reality.

Comment

Inspiration

Sight Buddy is more than just a tool - it's a gateway to independence and knowledge for those with visual impairments to explore the vast digital landscape with confidence and ease. It breaks down barriers, unlocks information, and opens up new possibilities for learning, working, and connecting. Sight Buddy demonstrates that visual ability is not a requirement for understanding and engaging with digital content. In essence, Sight Buddy is paving the way for a more equitable digital society, ensuring that everyone, regardless of visual ability, can fully participate in our increasingly online world.

What it does

Sight Buddy is an AI-powered assistant designed to enhance web accessibility for visually impaired users. It consists of two main components:

  1. A browser extension that:

    • Navigates websites
    • Summarizes web content
    • Performs actions on web pages
    • Facilitates further navigation
    • Closes the browser when requested

  2. A terminal-based assistant that:

    • Listens to general queries and provides answers using a large language model
    • Drafts and sends emails using AI-generated content
    • Opens desired websites on command

How we built it

We developed Sight Buddy using a combination of technologies:

  1. For the browser extension:
    • JavaScript for extension development
    • Chrome Extension API for browser integration
    • Web Speech API for text-to-speech functionality
    • Custom AI models for content summarization and action prediction
  2. For the terminal assistant:
    • Python for the core functionality
    • Speech recognition library for voice input
    • Gemini AI model for natural language processing and generation
    • SMTP library for email functionality

Challenges we ran into

  1. Accurate Content Interpretation : Summarizing the diverse webpage content and having required information like navigation links, possible actions.
  2. Avoiding Information Overload : Providing too much information can be overwhelming for users. Striking the right balance between comprehensive descriptions and concise, relevant summaries is crucial.
  3. User Experience Considerations: Designing an interface that caters to the visually impaired people

Accomplishments that we're proud of

  1. Successfully integrating voice commands with AI-powered responses
  2. Creating a seamless experience between the terminal assistant and browser extension
  3. Implementing a system that can perform actions on websites based on voice commands

What we learned

Through this project, we gained valuable insights into:

  1. The complexities of web accessibility and the challenges faced by visually impaired users
  2. Usage of Gen AI for desired task
  3. Browser extension development and its limitations
  4. The importance of user-centric design in assistive technology

What's next for Sight Buddy

We can further improve performance and accuracy using natural language processing, to better understand different dialects, languages and provide accurate result for complex websites or commands.

We can also add additional actions which can be performed by sight buddy in the system. The actions can be - read the data.pdf under downloads folder, play my favorite songs, etc.

The project can be further extended to analyze and summarize content like images, audio, video, other files and dynamic ui components.

Share this project:

Updates