Scrible | Devpost

Landing Screen
Choice of font
Live photo capture/Photo dropbox

Inspiration

Handwriting is a skill that’s easy to take for granted! However, many motor, muscular, neurological, and developmental health conditions impair individuals’ abilities to write freely with pen and paper. These disadvantaged populations deserve more than diversity: they deserve inclusion. Enabling ethical & effective education means using new innovations to meet the needs of all communities. Novel technologies like AI can allow the target population to engage with written activities in the same way as their able-bodied peers.

What it does

Scrible extracts handwriting, written by someone with the aforementioned conditions, from an uploaded file, or from a photo captured using the device's webcam. It will then transcribe the handwriting into a more legible text, allowing the user to effectively convey their ideas through written media!

How we built it

We used a CNN (Convolutional Neural Network), RNN (Recurrent Neural Network) and CTC (Connectionist Temporal Classification)-loss function architecture to break words into their individual characters. For example, if the word "Hello" was fed into the CRNN, the CNN would be able to detect regions where the probability of a single character was the highest. The RNN would then decode, using context clues and stroke patterns, the actual letters themselves. The resulting information was then properly aligned and labeled with the use of the CTC-loss function. This was then fed to Generative AI to clean up anything that the CRNN model missed.

Challenges we ran into

Formatting for the front-end was rather tedious and gave us quite the headache. The CRNN was very tricky as we had to adjust our approach multiple times as we had to adapt our strategies on how we wanted the CRNN to decode handwriting into text, especially text that's already difficult to read with the human eye.

Accomplishments that we're proud of

A beautiful UI, with lots of character and a joyful bounce to it! We are also proud to have made a coherent CRNN model to train on handwriting. Most of all, we truly feel that we developed our project for a relevant issue that affects many populations today, even if it meant that we encountered more roadblocks!

What we learned

We got more comfortable with front-end as we used a lot of React, TypeScript, and Tailwind. Learned a lot about full-stack development, connecting the front-end to the back-end using Flask.

What's next for Scrible

Further training and getting more accurate results from our model. Reaching out to our actual target demographics for a more accurate understanding of user needs. Consulting educational and developmental experts on how our platform could be integrated into existing workflows.