Hello Buddy!

Control Windows systems using voice commands and hand gestures.

Comment

Inspiration

On this occasion of ThanksGiving, we created a lazy script for lazy hackers to control their whole windows system using just voice commands and hand gestures without touching the keyboard and mouse.

The theme of this hackathon i.e. to build something that can help coders and assist them while they are feeling lazy and want to do things without coming close to their PC. This project can be extended to create an OS for disabled people.

What it does

An assistant that can take voice instructions and perform small tasks as well as be able to control the mouse control using fingers in front of the webcam. Some of the tasks performed as of now are:

  1. Play Music
  2. Log Off/Turn off/Restart or put their computer on sleep.
  3. Open/close the task manager in case your system is frozen.
  4. Delete Temp Files if you think your system needs some extra space.
  5. Close all Processes.
  6. Activate/Deactivate virtual windows keyboard
  7. Activate Virtual mouse to turn on the camera and use mouse pointer from hand gestures.

How we built it

We have used Google Speech-to-Text API to get the transcript of the user's voice commands lives by streaming the voice. As soon as the user says "hello buddy", and the rest of the scripts start working. To stop the script you just need to say exit/quit/okay/bye. The python script is running as a service in a windows machine and as soon as the system boots, the service starts running in the background.

Enabled camera based mouse control using palm/fingers detection through media pipe library and open CV.

Challenges we ran into

We wanted to add more features during the hacking period but due to time constraints, we had to limit.

Accomplishments that we're proud of

We wanted to work on automation this weekend as we all were new to this technology and we are proud that our script is completely working.

What we learned

We learned about Google's speech-to-text API.

What's next for Hello Buddy!

  1. Run python script as Service in Windows
  2. Create an Executable file to Install script as a Service in Windows.
  3. Script for Other OS.
  4. Add other features like getting Current Time/Weather, Close processes using PID. Upload Files to google cloud bucket through voice commands,etc.

Built With

Share this project:

Updates