HandAuth | Devpost

Inspiration

The frustration of forgotten passwords, stolen credit cards, and the inconvenience of traditional payment authentication methods. We wanted to create something as seamless as Face ID but even more unique—using the palm of your hand, which contains distinct biometric patterns that are nearly impossible to replicate.

What it does

HandAuth is a revolutionary biometric authentication system that uses advanced AI-powered palm recognition to verify identity for secure payments. By analyzing 21 unique hand keypoints and measuring distances between finger knuckles, it creates an individual "palm signature" that's as unique as a fingerprint—enabling users to authenticate payments with just a wave of their hand.

How we built it

-Tech Stack - Frontend: React, TypeScript, Vite, Tailwind CSS, React Router DOM, MediaPipe Hands (real-time hand tracking) - Backend: Express.js with Node.js, Twilio Verify API for SMS authentication, JWT for session management - AI/ML Pipeline: YOLOv11-Pose for hand keypoint detection, Ultralytics framework, PyTorch, OpenCV, NumPy - Training Infrastructure: Modal Labs GPU instances (A10G) for accelerated model training - Data Processing: Custom dataset of 4,000 hand images (3,000 training, 1,000 validation) with 21 keypoint annotations per hand - UI Components: Radix UI, Lucide React icons, custom animated wave backgrounds - Palm Recognition Algorithm: - Detects 21 hand landmarks (wrist, finger knuckles, fingertips) - Calculates Euclidean distances between knuckle points - Normalizes measurements for scale-invariance - Generates SHA-256 palm signatures for matching - Stores templates in persistent database with pickle serialization

Challenges we ran into

Training a YOLO pose estimation model from scratch required extensive GPU resources—we leveraged Modal Labs to solve this but faced dataset preprocessing challenges. Getting MediaPipe's hand tracking to work smoothly in real-time with the overlay alignment was tricky, requiring precise coordinate transformations between screen space and SVG space. The auto-capture countdown system needed careful state management to prevent race conditions. Integrating Twilio's SMS verification had rate limiting issues during testing. Creating a scale-invariant palm signature algorithm that could handle different hand sizes and camera distances required multiple iterations.

Accomplishments that we're proud of

Successfully trained a custom YOLO11-pose model that accurately detects 21 hand keypoints in real-time. Built a beautiful, production-ready React dashboard with smooth animations and excellent UX. Created a novel palm recognition algorithm that generates unique biometric signatures from hand geometry. Implemented a complete authentication flow from phone verification to palm registration. The auto-capture feature that detects proper hand positioning and triggers a countdown automatically—making the user experience incredibly smooth. Most importantly, we built something that could actually revolutionize how people make secure payments in the real world.

What we learned

We learned how to work with pose estimation models and the intricacies of keypoint detection. Understanding coordinate system transformations between video feeds, screen space, and SVG overlays was a deep dive into computer graphics. We gained expertise in Modal Labs for distributed GPU training and learned how to optimize YOLO models for specific use cases. The Twilio Verify API taught us about production-grade SMS authentication. We developed a stronger appreciation for biometric security—understanding why certain measurements work better than others for creating unique signatures. Most importantly, we learned how to integrate complex AI systems into user-friendly web applications.

What's next for Untitled

If we were to continue with HandAuth, we would first deploy our backend API to AWS or Google Cloud with a production database (PostgreSQL) to replace the in-memory storage. We'd integrate with actual payment processors like Stripe or Square to enable real-world transactions. Next, we'd expand the palm database capabilities with advanced features like multiple palm registration per user, liveness detection to prevent spoofing, and encrypted palm signature storage with zero-knowledge architecture. We'd also optimize the YOLO model further with quantization for mobile deployment, add support for iOS/Android apps using React Native, and implement analytics dashboards to track authentication success rates and improve the matching algorithm. Finally, we'd pursue partnerships with retail stores and fintech companies to pilot the technology in real payment scenarios.