Inspiration
Our inspiration came from right here at FIU, while waiting in line for the sponsor fair we began to play with the flood lights and create shadow puppets to be projected to the roof, we saw how this entertained everyone around us. This sparked our imagination to create a similar experience but with computer vision and machine learning.
What it does
Shadow Vision recognizes classic hand shadow-puppet poses in real time and streams them into TouchDesigner. With a quick pinch you enter selection mode, strike a pose, and watch the visuals on screen change instantly.
How we built it
The first hurdle was data — there wasn’t a ready-made dataset for shadow puppets. So we built our own. We filmed and photographed ourselves holding puppet poses, spliced the footage into frames, and labeled everything. That forced us to really think about what kind of data an ML model needs to learn patterns reliably.
From there we ran the dataset through MediaPipe to pull landmark vectors and engineered compact feature representations. We trained a k-NN classifier for speed and accuracy, then built an OSC bridge so the predictions drive real-time visuals in TouchDesigner.
Challenges we ran into
Building and labeling a dataset from scratch took more time and care than expected. Getting reliable performance on low-light and “messy” validation images. Keeping the pipeline responsive enough for real-time use. Connecting Python and TouchDesigner smoothly without lag.
Accomplishments that we're proud of
Creating our own dataset and using it successfully to train a working model. Designing a pinch-to-select gesture that makes the interaction intuitive. Achieving strong accuracy on challenging validation data while keeping frame rates high. Building a system that’s both technically sound and fun to demo.
What we learned
We learned the full loop: collecting data, labeling it, training a model, and deploying it into a real-time system. We also learned how important data quality is — more than just code, the dataset itself makes or breaks the model. Finally, we saw how tight the trade-off is between accuracy and latency when you want something to run live in front of people.
What's next for Shadow Vision
We want to expand the gesture library, add motion-based interactions, and package our system so TouchDesigner artists can drop it into their own projects. We also see possibilities for education, accessibility, and live performance.
Log in or sign up for Devpost to join the conversation.