As part of the "Hack the Party" category, HandScape is an interactive art installation designed for the Sunday night party at ManRay. It allows users to connect through artistic expression.
Inspiration
Since the theme of MIT Reality Hack 2024 is connection, we wanted to make a community project that emphasizes our connections to each other and to the rest of the digital and physical worlds.
The initial project idea was to create a website that combined drawings from a digital graffiti wall with user-submitted photos, videos, and text. We were heavily inspired by Cambridge’s Graffiti Alley, a public walkway where artists can leave any painted message (from smiley faces and their signatures, to full-fledged works of art). To gather inspiration, a team member headed to Central Square and took the photographs below!
At Graffiti Alley, one's creation might not last too long since anywhere on the wall is fair game for artistic use. We wanted to create a similar experience with HandScape, letting it serve as an ever-changing art gallery that hackers could interact with.
What It Does
By incorporating Ultraleap's hand tracking camera and software, users could use hand gestures to "paint" and leave messages. These messages were digitally captured and displayed on a large physical wall, with the help of a laser projector. Every 60 seconds, the wall refreshed - but not before capturing and saving the current painting. We also prompted many users with the question: "how did you connect at MIT reality hack?" Many participants posted photos of their contribution to Instagram...where all answers could be combined together to make a digital MIT Reality Hack Quilt. As a result, HandScape evolved to be a dynamic art installation, creating a time capsule for the future of the digital world.
How We Built It
We combined Ultraleap's hand tracking camera and software with Unity to set up a baseline experience that allows users to pinch their fingers and generate digital paint strokes. Using C#, we experimented with writing scripts that adjusted the paint stroke's colors, width, and appearance. The Ultraleap documentation provides many useful Unity plugins and example scenes for setting up XR experiences. The hand tracking camera was also extremely accurate and had great field of view. Below is one of our early prototypes.
When thinking about the night club this art installation would be displayed in, we later decided to add sound interactivity using Keijiro Takahashi's Reaktion, an audio reactive animation toolkit for Unity. With this, we were able to make the paint strokes of each message "dance" with the beat drop.
Challenges We Ran Into
Given our team's inexperience with Unity, the learning curve associated with the interface, features, and workflows was quite steep. Many of us downloaded the game engine at the hackathon itself, so we were met with challenges when importing the necessary libraries and setting up scenes. Half the battle was learning to use a new tool and getting accustomed to the interface. Overcoming this learning curve while simultaneously working on the project's implementation was possible with the help of Reality Hack's brilliant mentors and sponsors. Special shout out to Kris Pilcher for always pointing us in the right direction regarding hardware, software, and user experience; Gregory Osborne for his sick Unity skills and explanations; Pip, Lazlo, and Rory from the Ultraleap team for promptly coming to the rescue to debug our code & for sharing a whole bunch of links to their plugins, documentation, and 3rd party tooling details! Pip even drew us this handy user flow suggestion about best placement of the camera, ensuring that interacting with the wall would feel natural to the user :-)
Accomplishments We Are Proud Of
Overall, we are proud of creating something that unites and brings the community of hackers together. Over a hundred people interacted with this piece at the party, and we received great feedback! The interactive sound element made it perfect for the night club environment. With only 2 hours to install in a new space, we are also proud of our last-minute camera holding contraption, which was thrown together with a plastic cup and gaff tape.
What We Learned
By working together, we learned how to code with Unity and use the amazing hand-tracking devices of Ultraleap in a project. We also learned the importance of onboarding the user, such as providing a themed prompt for the experience. Very cool!
What's Next for HandScape
HandScape welcomes sequels in future MIT hacks...maybe it could be left up longer than a few hours! Beyond the transient and interactive nature of this project, we also want to figure out a more concrete way to host everyone’s messages. We experimented with creating the backend for a website and setting up a WebXR / Unity socket, but abandoned the idea due to time constraints. It would be great to permanently host drawings, photos, videos, and text - therefore preserving the connections and memories made at MIT Reality Hack 2024!
Log in or sign up for Devpost to join the conversation.