86 Hands | Devpost

Main UI
Dark Mode
Typical console output

Inspiration

There are many nights I lay in bed with my favorite show on with unfortunate ads that pop up and the only way to skip them is to get up out of my warm, cozy spot just to skip it and go right back in bed. I wanted to have the power to do all that and more, without having to get up all the time.

What it does

86 Hands is a Chrome extension that allows the user to speak basic Chrome browser functionalities to life, instead of manually doing so through mouse/keyboard input.

How we built it

Building a Google Chrome extension, we found, is similar to creating a regular website. It uses the typical HTML/CSS/JavaScript stack that websites uses, with an added manifest.json file to configure it for a Chrome extension specifically. Additionally, we implemented the Google Gemini API to interpret user speech and produce system-acceptable commands to execute certain commands in the browser.

Challenges we ran into

The most challenging implementation we ran into was the final part we worked on, the speech to text to Chrome commands. What started out with one problem, quickly grew into many subproblems that once solved, led to more subproblems than we had before.

Accomplishments that we're proud of

We are proud of the progress we have made in such a short time, especially with our small backgrounds in the tools and languages we used to try to make this happen. Getting the functionality of commands to work posed minor problems, but we were able to overcome those, and that was so refreshing after running into problem after problem.

What we learned

Extension service workers are not workers! They're slackers!

What's next for 86 Hands

Our team feels really excited about this project, we can see ourselves continuing this project in the future if something like this is really feasible as a browser extension. If so, we would hope to finish implementations of speech recognition, and then eventually build a hand gesture feature where users can use specific hand signals/signs to perform certain actions in the browser. This is a far reach at this point but we can see it becoming an incredible accessibility tool as well, for those impaired in eyesight and hearing.