Our Vision
Let's say you have a friend named John who has cerebral palsy. John has difficulty using his hands, so he has to use a special keyboard that is designed for people with disabilities. This keyboard is very slow and difficult to use, and it makes it hard for John to keep up with his friends and family.
But if John had GestureCon, he would be able to use his facial expressions and gestures to control his mouse pointer. This would make it much easier for him to use his computer and stay connected with the people he loves.
Now imagine a world where technology is inclusive and accessible to everyone. A world where you can control your computer with just your face and perform tasks that were once out of reach. This is the world that GestureCon is going to create.
Some conditions like spinal cord injury or muscular dystrophy can make it difficult or impossible for people to use traditional input devices. But, GestureCon can provide a much-needed way to interact with computers.
Our Inspiration
The software is designed with interactivity as its sole purpose. By featuring a user-friendly interface and customizable settings, we cater to every individual's need. Whether you have a physical disability or simply prefer a more natural way of interacting with your computer, GestureCon has got you covered.
At the heart of GestureCon is a belief that technology should be accessible to everyone, regardless of their physical, mental, or social ability. We understand the frustrations and limitations that traditional computer interfaces carry, but we're dedicated towards creating a more inclusive world. Join us on our mission to make technology more accessible and support those who need it the most.
What GestureCon does
GestureCon is more than just a neural networking project - it's a game-changer for people who face challenges daily due to their physical limitations. With GestureCon, you can browse the internet, open files, and send emails easily, all with the simple movement of your face. By simply tracking one's facial movement, GestureCon interacts with the computer in a way that is intuitive and natural to the customer, thereby reducing frustration & increasing productivity. Hence, through GestureCon, we provide much-needed support to people who face daily challenges due to their physical limitations and give them a sense of being.
How we built it
The core functionality and stand-out features of GestureCon involve two Computer Vision Tasks, namely facial authentication and pointer tracking.
For facial authentication, we take a quick scan of the user's face, map out the key features, and store this information as an NFT. This is done using a custom protocol on the goerli blockchain, ensuring a secure authentication and smooth user login experience. The mouse tracking feature utilizes dlib's model composed of 68 key facial features, which are mapped onto a 3D plane to determine the difference in distance, executing specific keystrokes and cursor movement. GestureCon's image and video processing is based upon OpenCV, therefore, providing us with accurate and efficient results. Three.js front ends our website to create an engaging and interactive 3D interface. The use of Three.js to develop stunning animations and incorporate a user-friendly interface promotes a seamless user experience.
Challenges we ran into
There were issues every step of the way. Our initial idea was to track the user's movement through gaze detection. But this failed, as there was no solution that could provide fruitful results in 36 hours, which made us change our direction to track a person's nose to move the cursor and use the eyes to act as control buttons. We also ran into problems while storing the facial data securely. Our reason for picking a blockchain over a simple SQL database was the security of the person's facial data. This made our work harder as we had to mint each person's data as an NFT on a custom address and also find a way to access all NFTs minted on that address. Integrating Flask with OpenCV was a situation where we ran into a bunch of drawbacks. The answer: threading.
Accomplishments that we're proud of
We were able to create software that allows users to control their mouse pointer through facial recognition and gesture control. This is a significant accomplishment because it allows users to interact with their computers in a more natural and intuitive way. Also, the apt use of a variety of cutting-edge technologies, such as OpenCV, dlib, Flask, Python, pyautogui, verbwire, NFT, blockchain, html5, css3, mongoDB, twillio, JavaScript, and Three.js to create GestureCon shows that we are a team of talented and experienced developers who are able to use the latest technologies to create innovative and societal solutions.
Overcoming the challenges & problems faced are some of the biggest accomplishments we have made here at HackPrinceton. But there is one accomplishment that we are trying to accomplish with all our might, i.e., to create an accessible as well as interactable technology that provides people with disabilities the support they are entitled to.
What we learned
Hack-a-thons are not only places to create or idealize softwares. But they give you the tools as well as the motivation to develop technologies that could last years or even decades.
Moreover, the purpose of a hack-a-thon is not about who creates the best-looking or best-functioning software. It's about realizing "what is the purpose of my technology?" or "how is it going to help a certain group of people?". And we understand that fact. So, we created GestureCon - maybe not the best looking or not the best functioning, BUT the best purpose-serving software.
What's next for GestureCon
We are proud of the accomplishments that we have made while creating GestureCon. We believe that this software has the potential to make a significant impact on the lives of people with physical disabilities. We are committed to continuing towards the development and improvement of GestureCon so that it can be used seamlessly by as many people as possible.
Discord IDs
- Tanay Shah: shishimaru11#3683
- Kushal Kapoor: kushhhh#6943
- Aryan Agrawal: aryan5276#1284
Log in or sign up for Devpost to join the conversation.