Flairbrush

A web-based, smartphone-controlled 3D drawing app, without the cost of VR.

Comment

Inspiration

We wanted to make a 3D painting app similar to Tilt Brush for the Vive, but accessible to everyone and without the cost of a VR setup. Flairbrush requires nothing more than a gyro-enabled phone, a computer, and an Internet connection.

What it does

Flairbrush uses a combination of touch controls and the gyroscope sensor in a smartphone to track user input in 3 dimensions. This data is sent to a computer paired via a shared code, and displayed as a 3D point cloud.

How to use

Visit the project link with a smartphone as well as a computer. Enter on the computer the code that appears on the phone. Move the cursor by moving the phone, tap to draw, and slide up or down on the phone screen to control depth. Drag on the computer screen to move the camera around, and scroll to zoom.

Note: the yellow 3d cursor might appear behind the camera initially, so zoom out if you don't see it.

How We built it

We started by developing the code for tracking input. The x and y axes are taken from gyroscope input, while the user manipulates the z axis via touch controls. To facilitate communication between the computer and the phone, we used a Flask web server with socket.io connections for real-time updates. We also used EC2 to host and test an online prototype of our hack. For the 3D graphics, we used three.js, a developer-friendly WebGL library.

Challenges We ran into

Tracking the user's hand in 3 dimensions using smartphone sensors. We initially tried to integrate accelerometer data twice, but this led to significant drift issues that made our data unusable. After more thinking, we decided that the gyros would likely be more accurate for what we wanted to do. By using the gyroscope and measuring the tilt of the user's wrist as input, we were finally able to obtain surprisingly accurate 3D positioning.

Accomplishments that We're proud of

The tracking of the user's 3D inputs using only sensors available on most smartphones. With this hack, we were able to eliminate the need for expensive 3D-tracked controllers such as those used with VR headsets.

What We learned

We learned how to use three.js to quickly produce vibrant 3D graphics. We gained additional experience using smartphone sensors in web applications. We became familiar with the Flask web server and its libraries.

What's next for Flairbrush

Adding smooth lines instead of discrete points. Saving and sharing drawings. Collaborative drawings. Cardboard integration. Better instructions. User-selectable colors. Possibly 3D printing drawings.

Share this project:

Updates