CinePhone

Turn your phone into a cinematic recorder

Comment

Inspiration

CinePhone was inspired by a common problem in creative film making. It's super expensive. Filmmaking overall is one of the most paywalled creative industries currently. To shoot even a short scene you need a camera, a crew, a location, lighting, actors, and a budget. While everyone has a camera, without a set and crew it's hard to make something you imagined to film... But with CinePhone, everyone has a set. We wanted to make a fun project to prove film making can be done in a different format. Using phones to create a storyboard and AI to render your full movie. Giving everyone the capacity to film even without the budget of a Hollywood director.

What it does

CinePhone turns your phone into a virtual film studio. You pick characters and environments from a preset library, place them into a scene, and then direct the whole thing with your voice. With CinePhone directing is as easy as speaking, with a voice input powered AI you can direct your characters to move and act without needing to write a script. Once you're done, you're able to record the film and watch your characters in action before you finally render the full movie using AI. We pass the segments/ scenes to be rendered into a full film using AI and the original unedited clips to produce a high quality cinematic film with consistent character and object placement.

How we built it

CinePhone is a web app, which kept us in one language end to end and made the result instantly shareable. The frontend is built with React, TypeScript and Vite, with the entire 3D scene authored in React Three Fiber on top of Three.js. The set, characters and camera all live in the browser, so the core filming experience runs with no server round-trip.

Characters and animations came from a preset repo of 3d models, using voice you can animate these characters to perform actions in the scene. Voice directing runs on the Web Speech API for the command grammar, with an AI layer powered by Claude Haiku interpreting the free-form description you speak between commands and mapping it to the right performance. The phone's motion drives the virtual camera so you frame and move your shots by hand, and we record each take as both video and a camera pose track, so your shot choices are captured as data rather than baked pixels.

For the backend we used Amazon Aurora Serverless (PostgreSQL) with the RDS Data API enabled, queried over HTTPS from serverless functions so there are no persistent connections to manage. Projects, scenes, actions and takes are stored relationally with full foreign-key integrity. Amazon S3 holds the recorded takes, camera tracks and rendered films, with the frontend uploading directly via presigned URLs. The whole thing deploys on Vercel, with the AI render pipeline assembling your selected takes into the final cut.

Challenges we ran into

Voice was the next hurdle. Turning a continuous stream of speech into clean, discrete commands without false triggers, and while still capturing the description spoken in between took real tuning. Recording was also deceptively hard: we wanted the camera movement stored as reusable data alongside the video, not just baked into the footage, so the render stage could work from your exact framing.

On the backend, getting Aurora to behave well with serverless functions meant leaning on the Data API instead of traditional connections, and configuring the cluster to scale to zero so a hackathon-scale project wouldn't quietly burn through credit while idle. And we had to be honest about the fidelity ceiling of AI rendering keeping a character consistent across cuts is genuinely hard so we scoped the render around what AI does reliably rather than overpromising.

Additionally, we struggled with luma-ai api and decided to switch to fal.ai for easier model deployment and more consistent api flow. This was a last second decision which brought the project closer to full.

Accomplishments that we're proud of

We're proud that the architecture is real, not a demo shell a relational Aurora backend with proper data integrity, S3 storage, direct uploads, and camera movement stored as data so the render can rebuild your shots faithfully. We also kept the interface intentional and soft rather than cluttered, so a first-time user can place a character, speak an action, and watch it perform within seconds. And we shipped something runnable on a real phone, not just on a laptop.

What we learned

The hardest part about AI integration isn't getting the AI to do more but preventing it from doing what it should not be able to. Applying guardrails and safety nets to the LLM was fairly hard but allowed for a smooth experience where the user can focus on creating rather than learning to use the app.

What's next for CinePhone

Full positional tracking, so you can physically walk around and between your characters, not just look around them. A higher-fidelity AI render tier that restyles hero shots toward photoreal footage while keeping characters consistent across cuts. A shot-suggestion co-pilot that proposes framing and coverage as you direct. Custom character and environment import, plus a marketplace of preset packs. Collaborative directing, so multiple people can crew the same virtual shoot.

Built With

Share this project:

Updates