Inspiration We wanted to create a magical interface that merges the physical and digital, moving beyond the keyboard and mouse. Our inspiration was to build a tool that lets you create art using your most natural inputs: your hand gestures and your voice, enhanced by the power of AI.
What it does ChromaScribe is a 3D generative art tool where you draw with your hand gestures and color with your voice. You can then type a text prompt (e.g., "a dragon made of fire") and click "Evolve" (⚡) to transform your 3D sketch into a photorealistic AI-generated image.
How we built it We built a serverless React/Vite app using react-three/fiber for the 3D canvas and Tailwind CSS for the UI. We used MediaPipe for real-time hand tracking (including fist detection), the Web Audio API for voice analysis, and Puter.js for the free, serverless AI image generation.
Challenges we ran into Our biggest challenge was the AI backend, which almost stopped the project. We first hit a paywall (402 Error) with Replicate, then a long series of deprecated models (410 Gone) and 404 errors with Hugging Face's unreliable free API.
Accomplishments that we're proud of We're proud of building a fully functional, multi-modal AI app that works end-to-end. Our biggest accomplishment was our persistence; instead of giving up at the API paywall, we researched and implemented three different backend solutions until we found a serverless one (Puter.js) that worked.
What we learned We learned how to integrate real-time hand and voice data into a react-three/fiber 3D scene, which was a major technical challenge. Most importantly, we learned how to quickly pivot and debug complex, unreliable API issues, finding a serverless solution when our initial plan failed.
What's next for ChromaScribe We plan to add real-time multiplayer support using WebSockets, allowing users to draw together in the same 3D space. We also want to add more gesture-based tools (like changing brushes with a peace sign) and eventually rebuild the app for WebXR to create a fully immersive VR/AR art experience.
Built With
- api
- javascript
- mediapipe
- react
- typescript
Log in or sign up for Devpost to join the conversation.