PaperBand | Devpost

Homescreen
Scanned Paper Piano

Inspiration

Music has been shown to be an incredibly strong tool for both learning and expression, and a way to connect people. Yet, it is difficult for many individuals to access musical instruments due to their high cost, need for regular maintenance, and lack of availability. These limitations have resulted in some large gaps in areas such as classrooms, in underserved communities, and in aging populations, which have experienced significant cuts to creative resources.

PaperBand began with a basic question: Can someone draw a picture of anything on paper and, using a camera, produce music?

Our objective was to provide a method of creating music that would be affordable, accessible, and easy to understand, all while being practical in a 36-hour hackathon format.

What it does

PaperBand uses pictures you draw on paper to create music. You simply sketch simple shapes on your paper — rectangles represent piano keys, and circles represent drum pads. Then, when you place a finger on a rectangle or circle that you have sketched onto the paper, and a camera detects the placement of that finger on the drawn shape, you will immediately hear the sound associated with that drawn shape in real time. No hardware is needed; no money is required; just the ability to think creatively.

PaperBand is a browser-based, interactive music system that creates music in real time and utilizes a live camera feed of a piece of paper to:

Observe a sheet of paper via a live camera feed]
Recognize hand-drawn shapes on that paper:
Rectangles to piano keys (including overlapping black keys)
Circles to drum pads
Continuously tracks the motion of fingers
Recognizes when a finger is pressed into a shape
Resolves the overlap of shapes appropriately (for example, black piano keys take precedence)
Immediately plays the correct musical sound

How we built it

Computer Vision & Interaction:

We utilized a number of real-time computer vision techniques to combine:

MediaPipe Hands for rapid, reliable finger tracking
OpenCV.js for contour detection and shape classification
Custom geometric logic to:
Determine whether a rectangle or a circle was drawn
Deal with imperfectly-drawn shapes
Prioritize overlapping areas (black piano keys over white keys)
Unlike detecting shapes each frame, we detect the shapes once and then track them efficiently over time to maintain the responsiveness and stability of the system.
When a press is detected, it is detected by the fingertip entering a shape area and crossing a velocity/threshold, thus preventing accidental hover detection.

Audio Generation

To generate sound directly in the browser, we employed the Web Audio API:

Piano keys are mapped to synthesized tones across a musical scale
Drum pads are mapped to percussive sounds
All audio is generated locally in the browser to prevent latency

Since we did not call the server for any reason, this ensured that the system worked offline and was scalable for widespread usage.

Challenges we ran into

Dirty Shapes: Since hand-drawn shapes are generally dirty and inconsistent, we were forced to develop a tolerant shape detection logic.

Geometry Overlap: Black piano keys sit atop white keys, and therefore we had to carefully determine how to prioritize them.

Latency: In order for the system to feel musical, we had to ensure that it responded nearly instantaneously, even the smallest delay can ruin the experience.

Stability in Real-Time: Another major technical obstacle was maintaining the continuous tracking of hands and shapes without jitter and/or false triggers.

Each of these challenges caused us to make strategic decisions regarding the trade-offs between accuracy, performance, and feasibility within the time constraints of the hackathon.

Accomplishments that we're proud of

The outcome is a paper instrument that is responsive enough to allow you to actually "play" it, not merely a proof-of-concept demonstration.

What we learned

Real-time interaction involves far more than pure accuracy, but also includes perceived responsiveness.

Mental models that are simple (drawing -> pressing -> sound) can greatly enhance usability.

Artificial intelligence can be a meaningful contributor to creative activities without diminishing the value of human expression.

Accessibility and sustainability are not merely moral obligations, but also contribute to the improvement of the design of products.

What's next for PaperBand

PaperBand will help to improve long term quality of life by reducing the financial burden of musical expression to zero.

Some potential implementations for PaperBand include:

Providing music education for students in under-funded schools

Creating therapeutic music programs for senior citizens and students with disabilities

Designing printable, open source instrument kits

Developing low bandwidth and/or offline creative tools for worldwide communities

In the long run, PaperBand may serve as a platform for providing accessible creative tools, demonstrating that powerful creative experiences do not require expensive hardware, only imagination.