Blinktype

Inspiration

Most dedicated gaze-tracking hardware costs upwards of $1,000, creating a steep financial barrier for individuals with motor impairments or situational disabilities. BlinkType was inspired by a simple goal: democratize digital accessibility by transforming any standard laptop webcam into a precise, hands-free typing terminal without requiring external sensors or expensive hardware.

What it does

BlinkType transforms any standard, built-in webcam into a hands-free typing device without requiring $1,000+ proprietary eye-tracking hardware. Spatial Calibration: Launches with a fast 9-point interactive calibration routine that maps spatial gaze coordinates to the individual user's head position and environment. Gaze-Driven Navigation: Leverages 468-point facial mesh tracking to follow iris trajectory across a high-contrast QWERTY key matrix in real time. Blink & Dwell Selection: Combines Eye Aspect Ratio (EAR) thresholding with a 750\text{ms} dwell-time timer to execute letter clicks, spacebar inputs, and deletions while filtering out involuntary blinks. Predictive Text Engine: Features dynamic word suggestion slots above the keyboard matrix to accelerate typing speed and reduce eye fatigue. Telemetry HUD: Displays real-time system performance—including camera FPS, active mesh point count, and estimated WPM—within a distraction-free dark workspace. 100% Local & Private: Runs entirely inside the browser using client-side JavaScript, ensuring video feeds and typed messages never leave the user's local device.

How we built it

BlinkType was engineered using a lightweight, pure client-side architecture to deliver zero latency and complete user privacy: Interface & HUD: Built using modular HTML5 and custom CSS Grid, designed around a high-contrast, dark-mode workspace. Landmark Extraction: Integrated MediaPipe Face Mesh to capture 468 3D facial landmarks and 10 refined iris tracking points at 60 FPS. Computer Vision Pipeline: Structured across modular JavaScript files (camera.js, tracker.js, gaze.js, app.js). Blink Detection Math: Calculated real-time Eye Aspect Ratio (EAR) to separate natural eye movement from deliberate selection triggers:EAR = \frac{\Vert{}p_2 - p_6\Vert{} + \Vert{}p_3 - p_5\Vert{}}{2 \Vert{}p_1 - p_4\Vert{}} where p_1, \dots, p_6 represent the 2D spatial coordinates of eye contour landmarks. A sustained drop below the threshold (EAR < 0.20) registers as an intentional action. Selection Dwell Logic: Implemented bounding box collision checking where continuous gaze duration T \ge 750\text{ms} triggers a visual key highlight and outputs characters to the terminal.

Challenges we ran into

Optical Drift & Distortion: Standard webcams lack depth sensors and suffer from focal distortion. We resolved coordinate offset by building a full-screen, 9-point interactive spatial calibration routine prior to loading the workspace. False-Positive Filtering: Natural blinking and squints initially caused erratic typing errors. Implementing rolling frame-buffers and dynamic thresholding eliminated involuntary noise. Performance Optimization: Processing heavy computer vision models alongside continuous DOM updates risks freezing browser threads. Modularizing the core logic into lightweight ES6 scripts prevented memory chokes and maintained high frame rates.

Accomplishments that we're proud of

Hardware Independence: Built a functional, real-time computer vision system that runs on standard consumer webcams, eliminating financial barriers to hands-free digital tools. 60 FPS Client-Side Vision: Successfully integrated MediaPipe iris tracking directly inside standard browser JavaScript, maintaining stable frame rates and low memory consumption without external backend servers. Software-Based Drift Correction: Engineered a custom 9-point spatial calibration algorithm that corrects optical camera distortion and lighting variations without physical sensors. High-End Assistive Design: Created an ultra-minimalist, dark-mode HUD interface that elevates accessibility software into a modern, tactile, and empowering user experience. Zero-Latency Privacy: Delivered an end-to-end local pipeline where gaze calculations, selection triggers, and terminal rendering happen instantly with complete data privacy.

What we learned

Human-Computer Interaction (HCI): Fine-tuning system latency—such as calibration sensitivity and dwell duration (t_{\text{dwell}})—is just as critical to user experience as raw tracking accuracy. Client-Side Power: Modern browser engines combined with WebAssembly enable complex, real-time spatial applications to run completely locally, keeping video data entirely private

What's next for Blinktype

Looking ahead, the roadmap for BlinkType focuses on three core areas to transform our hackathon project into a production-ready accessibility suite: Engine & Performance Optimization: Moving the MediaPipe vision loop to a WebWorker via OffscreenCanvas to offload heavy image processing from the main UI thread and lock DOM rendering to a constant 60 FPS. We also plan to implement 1D Kalman filtering on iris coordinates to eliminate natural eye jitter (saccades) and introduce continuous, passive recalibration. Context-Aware AI Prediction: Upgrading our basic word dictionary to a lightweight, client-side language model running via WebAssembly/ONNX. This will enable full-sentence autocompletion and dynamic target scaling—automatically enlarging high-probability keys to reduce eye fatigue. Browser Overlay & Ecosystem Integration: Packaging the vision engine into a Chrome Extension so users can inject the BlinkType gaze keyboard over any web page (such as Gmail, Twitter, or Notion), alongside building a customizable quick-phrase dashboard for rapid, everyday communication.

Built With

• computer-vision
• css3
• html5
• javascript
• mediapipe
• webassembly
• webrtc