Ink-Keys turns drawings into instruments using a webcam and hand tracking, offering a portable, affordable way to make music. Accessible for kids, beginners, and creatives alike!

Watch the demo

The idea for Ink-Keys came from the desire to make music accessible, affordable, and portable for everyone. The hefty price tag on pianos often makes music education inaccessible, discouraging beginners. Ink-Keys combines a child's love of drawing with the joy of playing music, fostering creativity and learning through fun. We wanted to eliminate the barriers by allowing users to draw their own piano and bring it to life anywhere, anytime.

• Uses a webcam, phone, or any camera available to detect a printed or drawn piano on paper.
• Allows users to learn, create, and play around with a piano system, with real-time feedback.

• Used ReactJS and TailwindCSS to form an easy-to-follow hierarchy, displaying various components used and reused across the website.

• Flask: Created a SocketIO stream, updating key presses in real-time.
• OpenCV: Accessed the webcam to detect the piano keys by finding contours in the video, filtering by shape and aspect ratio.
• TensorFlow and MediaPipe: Used for hand and fingertip tracking to allow easy interaction with the virtual keyboard.
• Python: Implemented backend logic for key detection, hand tracking, and input recognition.

Follow the steps below to set up and run InkKeys on your machine:

Download the code from our GitHub repository by running:

git clone https://github.com/obljubej/ink-keys.git

We developed this project using:

• Python 3.10.0
• Node.js v22.13.0

Ensure these versions or compatible ones are installed on your system.

Navigate to the frontend folder:

cd virtual-keyboard/frontend/virtual-keyboard

Install the necessary dependencies using npm:

npm install

Navigate to the backend folder:

cd virtual-keyboard/backend

Install the Python dependencies:

pip install opencv-python numpy mediapipe flask flask-cors opencv-python-headless

Open a terminal and navigate to the backend folder:

cd virtual-keyboard/backend

Run the backend server:

python main.py

Open a separate terminal and navigate to the frontend folder:

cd virtual-keyboard/frontend/virtual-keyboard

Start the frontend server:

npm start

1. Place the keyboard paper you want to play on in view of your camera.
2. Open the site in your browser (usually available at http://localhost:3000).
3. Use the Spacebar to adjust the keys if necessary.
4. Start playing your virtual keyboard!

• Real-time Integration: The initial system of constantly pinging the backend was inefficient, so we switched to a Flask SocketIO stream for real-time updates.
• Noise Issues in Detection: Separating keyboard keys based on drawn lines involved overcoming challenges with overlap detection, rectangle detection, and shape filters.

• Developed a fully interactive and responsive keyboard.
• The website’s display reflects the keys being pressed in real-time.
• Successfully implemented OpenCV, TensorFlow, and MediaPipe for hand tracking and piano detection/key separation mechanisms.

• Gained experience using OpenCV, MediaPipe, and TensorFlow for object detection and tracking.
• Learned how to use Flask StreamIO for seamless frontend-backend integration in real-time.

• Community Features: Users can save their info, enabling others to play and learn their custom tunes.
• Precise Hand Tracking: Enhance tracking to detect whether a finger is pressed down for a key press.
• Mobile App: Adapt Ink-Keys into an app with a user-friendly UI for mobile users.