QuackSupport | Devpost

QuackSupport is always there to help
QuackSupport tells you where to click

Inspiration 🌟

My grandmother once called me because she was frustrated when she couldn’t figure out how to adjust her screen brightness. That call stuck with me—it wasn’t just about brightness; it was about how technology, meant to empower, often leaves the elderly feeling helpless.

For millions like her, even simple digital tasks are overwhelming. Systems prioritising efficiency over accessibility exclude the elderly and tech-illiterate and burden them. Mistakes, like printing the wrong ticket or missing an online step, lead to unnecessary frustration and loss of independence.

What it does 🔍

Our agent provides step-by-step, visual guidance by spotlighting where users should move their cursor. It empowers them to easily navigate technology, reducing reliance on others and restoring confidence.

How we built it 🛠️

By leveraging Anthropic’s computer-use, community tools (computer agent) built on top of it, and PyQt, we enabled the system to spotlight interactive elements and guide users visually.

PyQt enabled transparent, screen-wide guidance with smooth, anti-aliased highlights and intuitive visual feedback, seamlessly directing users to key elements.

A threaded agent loop powered real-time updates, ensuring the system refined its actions based on user feedback for an engaging and responsive experience.

Challenges we ran into 🚧

Ensuring accuracy in cursor-guided instructions was challenging, especially when handling diverse screen layouts and applications.

Latency issues with tool execution and balancing usability with system limitations required careful iteration and optimization.

Accomplishments that we're proud of 🏆

We successfully implemented a human-in-the-loop design, ensuring users always feel in control while receiving guidance. By leveraging human-computer interaction principles, we created an intuitive system where spotlighting highlights actionable elements on the screen, empowering users to complete tasks independently.

Using an agentic workflow allows the system to dynamically adapt to user needs, offering personalized, step-by-step assistance.

What we learned 📚

Varun: "Don't use PyQt (it is not *that" bad)"
Zach: "Don't be afraid of computer-use tools"
Justus: "Don't reinvent the wheel"
Nicolas: "Don't be afraid to delete the repo four times during a hackathon"

(Advisor Gabriel: "Don't spend more than 10% of your hackathon free-food savings on anthropic credits")

What's next for QuackSupport 🦆

We plan to expand the system’s capabilities by integrating multilingual support and voice-based interactions to reach a broader audience. Technical improvements will focus on enhancing task consistency and reducing latency in agentic workflows.

Discussions with social agencies are underway to deploy the solution in community centres and eldercare facilities, bridging the digital divide for marginalized groups.

Additionally, we aim to collaborate with developers to refine human-in-the-loop validation, leveraging advanced contextual adaptability for diverse use cases.