Chrome Control | Devpost

Inspiration

I was inspired to make this hack after see how many young and old people alike that have physical and/or visual impairments struggle to use their computers to their full potential because of how challenging it can be to use a browser since that requires fine motor controls for navigation and ability to see all the small details on a page.

What it does

It uses a Google Home as a voice interface to browse the web through an accompanying Chrome Extension that manipulates Google Chrome. Its current features include: go forward/back, invert colors, mute/unmute tab, new tab, open link, reload page, restore window, scroll down/up, close tab, site search, pause/play youtube video, and zoom in/out.

How I built it

I made a Google Cloud Function that runs server-side logic for processing user requests through the Home. One of the things the GCF does is it updates Firebase values for a user so that the Chrome Extension can pick up on the updates and change Chrome's state accordingly. Also, Dialogflow is used for the NLP aspect of understanding user requests.

Challenges I ran into

syncing code folder to GCF bucket
finding a way to synchronize actions/values between Action and Extension
authenticating/getting Google user on both platforms (OAuth 2)
figuring out what commands would be most useful for the end-users
getting the Google Home connected to the internet because the password-protected WiFi was making this difficult

Accomplishments that I'm proud of

Overcoming the aforementioned challenges! This was a non-trivial project to make work considering the various parts needed to be built.

What I learned

Some of the things I learned include: how to use Firebase in a creative way, how OAuth2 works, how to make a Chrome Extension, and how various Chrome APIs work.

What's next for Chrome Control

More features: There are so many more features that can be added. As long as there's a Chrome API for manipulating some aspect of Chrome, I believe I can find a way to make it accessible through voice.
Deploying to other common voice platforms such as Alexa and Cortana to increase potential user base.
Also adding the totally different feature of eye-tracking based navigation for scenarios where voice-based navigation may not be preferable.