Inspiration
The inspiration behind creating JarWiz likely stemmed from a desire to address the limitations of traditional keyboard and mouse inputs, particularly in the realm of accessibility and user interaction. Observing the challenges individuals with motor impairments faced and the need for more intuitive and inclusive computing solutions, we were motivated to develop a product that could empower users to interact with computers naturally and seamlessly. Additionally, recognizing the potential for gesture recognition technology to revolutionize not only accessibility but also productivity and hardware control, we were inspired to embark on a journey to create JarWiz. This groundbreaking software redefines how we engage with technology. Through innovation and a commitment to inclusivity, our vision for JarWiz was born, driven by a passion for making computing more accessible, efficient, and enjoyable for all users
What it does
Introducing "JarWiz" – revolutionizing computer accessibility with gesture and action recognition software. Embrace a new era of interaction as JarWiz liberates users from traditional keyboard and mouse inputs, offering intuitive gestures for seamless control. With its immense implications for accessibility and innovative hardware control, JarWiz is poised to transform how we interact with computers. JarWiz is more than just a computer accessibility tool. It is a generic gesture control interface. The technology behind gesture and action recognition can be applied in many other fields, especially to operate hardware more intuitively and annotate without any instruments.
How we built it
Our implementation of JarWiz involved harnessing the power of OpenCV (cv2) to map hand gestures in real time, enabling users to interact with computers intuitively and without traditional input devices. By leveraging computer vision techniques, we developed algorithms to detect and track hand movements, recognize gestures, and translate them into actionable commands. This allowed users to navigate user interfaces, control applications, and perform tasks simply by gesturing, revolutionizing the way they interacted with their computers.
In addition to hand gesture recognition, we integrated Whisper, a cutting-edge speech-to-text library, to further enhance the user experience. Whisper enabled users to dictate commands, input text, and control their computers through voice commands, providing an alternative mode of interaction that complemented gesture recognition. By seamlessly combining both hand gestures and speech recognition capabilities, JarWiz offered users a versatile and intuitive interface that catered to diverse preferences and accessibility needs.
We successfully integrated a hardware controlling interface, allowing us to intuitively manipulate the Claw we designed through gestures. Furthermore, we've achieved a groundbreaking feat by enabling the ability to play an invisible air guitar, yielding extraordinary and melodious results.
Challenges we ran into
Particular challenges we ran into were:
- real-time audio detection
- Intuitive navigation by mapping the position of fingers to the actual desktop screen
- incorporating varying yet comfortable gestures to perform different operations
- ensure real-time lag-free computation with such hefty image and action detection algorithms
Accomplishments that we're proud of
We are proud of incorporating JarWiz not only as a software accessibility and productivity tool but also as a hardware control system. We managed to effectively use the available time to integrate our program for various purposes.
What we learned
We learned about the possibilities provided by opencv and mediapipe, and how gesture control can be used for far more use cases than we previously though.
What's next for JarWiz
We recognized some realms where we can further improve the performance and scalability of JarWiz
- Precision: make the navigation more precise
- More gestures: incorporate more gestures for easier computing
- Eye tracking: eye tracking to further enhance accessibility
- Better audio recognition: perfectly real-time audio detection, reducing computation time and increasing efficiency
- User detection: detecting the user of the laptop to enhance privacy and limit unrestricted use.
Log in or sign up for Devpost to join the conversation.