LookSmith

LookSmith empowers anyone, regardless of motor ability, to shape 3D art using only facial movement, transforming creative expression into an inclusive, accessible, and hands-free experience.

Comment

🎨 LookSmith

Accessible 3D Modeling Through Facial Tracking

🧠 Inspiration

We wanted to create a tool that opens the world of 3D art to people who have the imagination but not the motor control to use traditional modeling software. LookSmith was inspired by artists and creatives with disabilities who often face barriers in digital creation.

Our goal was to let anyone shape 3D models — not with their hands, but with their expressions and movements.

💡 What It Does

LookSmith is an accessible facial-tracking 3D modeling platform built with Godot and Python OpenCV and Mediapipe. Users sculpt and shape 3D objects using only their head and facial movements, tracked through MediaPipe.

The program translates subtle gestures into precise modeling inputs — no mouse, stylus, or keyboard required.

What makes LookSmith truly unique is its direct export integration: users can instantly export their .glb models from Godot to our companion monetization site, where they can showcase, share, or sell their creations. This seamless connection forms a complete creative-to-commerce pipeline — from facial expression to fully marketable 3D model.

🛠️ How We Built It

  • Engine: The engine used to make 3d models was Godot. The engine was not used at its full capabilities, but instead used through a set of predefined commands which were hooked to facial gestures. The user could perform gestures (such as tilting head to the left, blinking thrice etc.) to initiate particular commands for a given shape. The commands are transferred from the python vision engine to the 3D engine using a sustained websocket connection, and specific commands in json format to maximize information transmitted per websocket query.
  • Tracking: Built the real-time facial motion-capture using MediaPipe with OpenCV to provide preprocessing (stabilization + temporal smoothing) which was streamed through a Python↔Godot to move the cursor based on facial motions.
  • Modeling System: Designed our own mesh manipulation and gesture-mapping framework
  • Export Pipeline: Built an automated .glb exporter that uploads models directly to the companion site
  • Web Companion: Developed a platform for upload, monetization, and creator portfolios
  • Design Focus: Prioritized accessibility-first UX and real-time feedback for intuitive, hands-free control

🚧 Challenges We Ran Into

  • Motion Stability: Filtering and smoothing noisy face-tracking data without adding latency
  • Gesture Mapping: Designing natural, precise mappings between facial motion and modeling tools
  • Performance: Maintaining smooth, real-time rendering in Godot while streaming live tracking data
  • Cross-Platform Assets: Ensuring consistent .glb export and upload between the Godot app and the web platform
  • Marketplace Integration: Handling authentication, uploads, and asset verification between systems

📚 What We Learned

  • How to integrate computer vision with a real-time game engine
  • How to design for accessibility and inclusivity from the start
  • How accessibility-driven technology can unlock new career paths in digital art
  • That merging art, empathy, and engineering can redefine what it means to create

🌍 The Impact

LookSmith provides millions of people of determination the opportunity to interact with art in their favorite way. It empowers people with limited or no fine motor control to become digital artists, for recreational and entrepreneurial purposes. By enabling direct export to a monetized platform, it provides both creative freedom and financial opportunity.

We’re redefining digital art as something anyone can make — proving that innovation, accessibility, and creativity can exist hand in hand.

Built With

Share this project:

Updates