Composition | Devpost

Inspiration

I built composition for creatives that don't want to spend hours prompting an idea they can see so clearly. Also, for those who want an aid to 3D modelling and those who don't want to give away all of their creative freedom.

What it does

Composition is an agentic filmmaking studio that lets you physically step inside a virtual scene. Using ARKit and the phone’s motion sensors, your real-world position and orientation control the virtual camera, walk forward, circle a character, tilt your phone, or frame a close-up, and Composition records that movement as an editable camera take.

Gemini Live acts as a voice-controlled director, Gemini plans scene edits and develops the visual style, and ElevenLabs voices the Director’s final responses. Your camera and character movements are preserved as reusable animation blocks, then used to guide video generation with Seedance or Gemini Veo.

How we built it

The filmmaking pipeline

Direct with Gemini and ElevenLabs. Gemini receives scene state and a viewport image, returning typed actions for objects, cameras, and animation. Gemini Live carries the conversational voice direction; after a reply is finalized, ElevenLabs synthesizes it as audio through a server-side voice proxy. Replies play completely and sequentially, and Director waits for the next turn while audio is preparing or playing. If information is missing, it asks one concise question and waits rather than guessing. Hunyuan supplies generated character motion. Changes are validated and approved before application.
Film and preserve movement. ARKit sends phone position and orientation through a WebSocket relay. The Three.js editor maps those poses to the virtual camera and saves interpolated motion as editable timeline blocks. Blocks retain source keyframes when split, retimed, or reused.
Render the composition. The saved camera and animation tracks become a clean 720p, 30 fps motion guide. This captures staging, timing, and camera movement independently of the phone's preview stream.
Develop the look with Gemini. Gemini refines artistic prompts and generates styled first and last frames. Gemini receives the styled first image when creating the last, providing a shared appearance reference while retaining the ending composition.
Generate with Seedance or Gemini Veo. Seedance receives the motion guide, selected Gemini images, and direction prompt: the guide supplies motion and framing while the images supply appearance. Gemini Veo 3.1 is an alternate Gemini API path for 4-, 6-, or 8-second 720p clips; it uses the direction and selected baseline/reference images, describes movement in the prompt, and returns generated audio. Veo does not receive the phone motion guide, so the user can intentionally choose authored motion (Seedance) or prompt-and-image-driven generation (Veo).
Review and iterate. The pipeline saves returned video for review and download. Users can revise movement or staging, or reuse the composition for another visual treatment.

Solo development with Codex

Codex made Composition manageable as a solo build by connecting repository access, terminal execution, browser research, MCP tools, and Git diff inspection in one development environment.

The most useful features were the ones that made the development process repeatable:

Project instructions and reusable skills. AGENTS.md defined the repository's validation commands, while a separate SKILL.md workflow required scoped checks, diff inspection, and a commit for each completed change.
Interactive steering. I could refine requirements during ongoing work while keeping the conversation's implementation context.
Persistent implementation notes. Codex helped document rotation continuity, stale Gemini proposals, and recoverable generation requests, so later work could build on previous debugging.
Tool use grounded in the repository. Codex could connect an architectural claim to the actual renderer, relay, or provider implementation, then make and review the corresponding edit.

Challenges we ran into

Handheld mode was initially too slow because camera tracking and video previews competed for the same connection. I separated them: the phone streams motion at up to 30 fps, while the desktop sends back a lightweight preview at roughly 6 fps and drops old frames when the connection falls behind.

Gemini also needed to make safe, predictable edits. I gave it a compact set of typed actions, validated every response, and tied each proposal to the exact scene it was created for. If the scene changes, Composition asks Gemini to refresh the proposal instead of applying an outdated edit.

Accomplishments that we're proud of

Building a solo project spanning spatial editing, phone tracking, voice direction, and media generation with Codex.
Using Gemini across conversation, scene planning, prompt refinement, image generation, and Veo video generation.
Adding ElevenLabs as a reliable spoken response layer for Director, with complete, non-overlapping playback and clarification turns that wait for the user's next message.
Validating a live Gemini object proposal and converting a two-second Hunyuan performance into 19 editable tracks.
Preserving authored motion through reusable blocks and a rendered generation guide.

What we learned

Gemini becomes more useful when it operates on explicit scene state and visual context. Codex becomes more useful when work has clear boundaries, reusable instructions, and reviewable results.

What's next for Composition

My next development pass focuses on:

Finished-film audio: extend the current ElevenLabs Director voice layer into optional narration or dialogue tracks mixed into exported video through FFmpeg.
Codex cloud sandboxes: compare Gemini appearance-conditioning approaches in isolated environments using the same composition.
Task forks, parallel agents, and worktree handoff: branch an investigation with its conversation context, give independent implementation tasks separate checkouts, and bring a task and its code back to the local checkout for integration.
Browser annotations and inline review: point Codex at specific timeline or director-panel elements, reproduce browser interactions, and attach code feedback to exact diff lines.
Overnight automations backed by skills: package rotation, tracking-loss, and export checks into a repeatable workflow, schedule it, and review failures in the morning.
Remote control from my phone: steer development tasks and inspect diffs while away from the desktop during handheld testing.