Inspiration
People with motor neuron diseases, such as Parkinson's, often experience involuntary tremors that can make typing difficult and error-prone. Typing mistakes disrupt communication and can cause frustration. We wanted to create a tool that helps users with Parkinson's type more accurately and smoothly, reducing errors and improving their experience on the web.
What it does
Steadytype is an application that provides real-time autocorrect assistance for individuals with motor neuron difficulties. When enabled, it monitors typing in text fields and automatically corrects errors caused only by their tremors, enabling users with tremors or poor motor function to type with greater confidence and helping hand.
How we built it
We built our Steadytype application by integrating several key technologies and components into a seamless system. At its core, we implemented a global keyboard monitoring system using pynput that captures keystrokes across all applications in real-time, while a sophisticated text processing engine maintains an accurate buffer of typed text and keystroke errors. The time between two keystrokes is known as latency, and whenever this latency is below a threshold of 50 milliseconds, we can be 99% sure this is a typing error. This allows us to distinguish between intentional misspelling and unintentional errors, and correct the latter. The autocorrect intelligence comes from Google's Gemini AI model, which provides fast text corrections. We developed a pause detection system that monitors natural typing breaks, waiting for 2 seconds of inactivity before initiating corrections. To ensure smooth operation, we implemented thread-safe mechanisms that prevent concurrent corrections. The text replacement system works by carefully simulating backspace keystrokes to remove incorrect text and then inserting the corrected version, directly in the input field where the errors were made. The application operates non-intrusively, only making corrections during natural pauses. The program runs on Python, requiring only an internet connection for the Gemini service and appropriate system-level permissions for input monitoring, making it accessible and easy to set up.
Challenges we ran into
- Databases: Initially, we had hoped to find data retrieved from research on Parkinson's patients' typing errors. Something we were looking for in particular was ideally a dataset of results from typing tests that showed both target words and actual input from the user. As we tried to implement several models for autocorrection, we realized that no such models had higher accuracy than gemini-1.5-flash.
- Model Latency: Integrating real-time correction with a language model proved challenging due to internet latency. Optimizing the response time was crucial to make the corrections feel smooth and unintrusive.
- Handling Errors Gracefully: We ran into many hurdles with edge cases and errors we did not account for. Some of these included losing punctuation and capitalization, or accidentally sending Gemini a prompt with nothing to correct. In response, we fine-tuned our program's flow and our AI prompt to prevent extraneous results.
- Delivery Apparatus: Originally, our intention was to implement this service as a Google Chrome extension. However, this method presented many more obstacles in our access to keystroke information and would limit the situations where the user could access our service.
Accomplishments that we're proud of
- Successfully implemented real-time autocorrect for users with tremors available system-wide.
- Created a smooth user experience with a customizable interface, allowing users to enable or disable autocorrect per domain.
What we learned
Throughout the development of our Steadytype application, we gained crucial insights across multiple aspects of software development. We learned that Gemini 1.5 Flash offers the best balance of speed and accuracy for real-time text correction, while the implementation of a 2-second pause detection proved essential for non-intrusive corrections. Our experience with thread management and background processes highlighted the importance of proper synchronization to prevent race conditions. Building a globally operating application taught us the significance of robust error handling for various scenarios like network failures and API timeouts. Most importantly, we discovered that technical excellence must be carefully balanced with user experience - the application needs to feel natural and helpful rather than intrusive, a principle that guided every aspect of our development process from timing to text replacement mechanics.
What's next for Steadytype
- Improving Correction Accuracy: We aim to fine-tune our algorithm by examining more specific data on people with Parkinson’s and their typing patterns.
- Expanding Accessibility Options: We plan to add more customizable options for users with different types of accessibility needs. This of course comes with the changes to the UI and frontend which allows users to select their autocorrect type based on their needs.
- Publishing on Chrome Web Store: Once fully optimized, we plan to publish Steadytype to the Web, making it downloadable to the public.
- Conducting our own research: We hope to be able to eventually use the application to gather data from users, in order to better explore research that fits our tool's purpose (including the potential labeled data for an AI-powered future version).
Log in or sign up for Devpost to join the conversation.