StickyAI | Devpost

Drag to select text
Selected text

Inspiration

The reading experience is far superior on a physical book compared to a pdf (also prevents eye strain), but taking notes is often a really tedious process.

Say you wanted to write down a quote from a book -- you'd have to put down your book, pull up your computer to type up the quote or alternatively, write it down on paper.

Both take time -- making note-taking an inconvenient process, and distracting you from the actual reading experience.

Introducing StickyAI, a tool that allows you to capture notes with a simple hand gesture, simplifying your note-taking process.

All you need is a book, your hand, and a camera -- you can drag your fingers along the page to select a passage to save. While the program does all the heavylifting, you can continue enjoying your book.

What it does

Webcam continuously captures the pages from the book
Placing your index and middle fingers together, you can drag across the passage you want to save from the book
Moving your middle finger away, the camera takes a picture of the current page -> sends it to the backend
Shows the selected passage within the green frame
Saves the quote (generated from OCR) into a text file

How we built it

OpenCV detect contours and adds bounding boxes for each word
Words are sorted into lines based on their y-position and height, allowing the program to determine where each word is located and its order
Using Mediapipe for hand landmarks, the coordinates of the index and middle finger tips are located between these points, the camera recognizes the gestures for starting and finishing the capture
Using the distance between these points, the camera recognizes the gestures for starting and finishing the capture
Pytesseract takes each of the bounding boxes and turns it from an image into a word, allowing the reader to select a specific passage of the page and save it into a text file (also gives higher accuracy compared to sending the entire page into Pytesseract)