Dreaming the Sound of Contact
Leveraging video and audio generation for zero-shot force-aware manipulation and data generation.
Experiments
Video for motion, audio for force.
Pick a task and a run. Each row plays the generated reference video + audio, our force-aware execution on a Franka arm that tracks the audio-derived force profile, and a kinematic-only baseline for comparison. Turn the sound on — the contact audio is the force signal.
The problem
A video shows where, not how hard.
Video generators can now dream up a robot doing a task. Turn those frames into a trajectory and a real robot can copy the motion. But a video only shows where the arm goes, not how hard it presses. For wiping, peeling, or pressing a button, that missing force is the difference between finishing the job and hovering a few millimeters above the surface, or slamming into it and triggering a safety stop.
The generated audio fills the gap. Contact sounds get louder as contact gets harder, so we use loudness as a proxy for force. From one task description we generate video and audio, take the motion from the video and the force profile from the audio, and run both on a Franka Panda with a force regulator in the loop. Those executions are good enough to become training data, too.
The pipeline
Four steps from a sentence to contact.
- GenerateOne RGB image plus a text prompt goes to Seedance 2.0, which returns a video with audio (the exact prompts are at the bottom).
- Motion from videoFind the gripper and object (MolmoPoint), segment them (SAM 2), and track them in 3D (TAPIP3D) to get the end-effector path, the contact moments, and the force direction.
- Force from audioIsolate the contact sound (SAM-Audio) and turn its loudness into a force-magnitude profile over the contact interval.
- ExecuteA 1 kHz impedance controller follows the path while a force regulator tracks the audio-derived profile during contact.
Audio check
Ask for a harder hit, get a louder sound.
The whole idea only works if the generator's audio responds to force, so we tested that directly. Same starting image of a hand holding a hammer, one prompt asking for two strikes on the table: one at 5 N and one at 20 N. We generated ten videos with the soft strike first and ten with the hard strike first, then measured how loud each strike is.
In 18 of 20 videos the strike prompted as 20 N is the louder one, 9 of 10 in each order. The two misses are near-ties. This shows the generator follows a prompted force order. It does not show that loudness measures physical force.
The 2nd strike, prompted as 20 N, is 6.8 LU louder, about 1.6× as loud to the ear.
The 1st strike, prompted as 20 N, is 13.7 LU louder, about 2.6× as loud to the ear.
Bars show the peak loudness of each strike over a 400 ms window, drawn relative to the louder strike. One further generation was excluded because it showed three strikes.
Sanity check
Does the shape of the force matter?
If any contact force works, we could just press with a constant 10 N and skip the audio. So we tried that. On wiping and peeling we ran the same trajectories with two audio-shaped profiles (peaking at 10 or 15 N) and two constant profiles (10 and 15 N), ten trials each.
Constant force fails at first contact: the command jumps from 0 to 10 N in one step, the measured force overshoots toward the 20 N limit, and the robot stops itself. The audio-shaped profile starts low and ramps up after contact, so it never trips the limit, and it finishes with a lower peak force and less total impulse.
| Task | Profile | Success | Peak (N) | Impulse (N·s) |
|---|---|---|---|---|
| Wiping | Variable, Fmax = 10 | 7/10 | 9.31 ± 1.43 | 15.20 ± 4.42 |
| Variable, Fmax = 15 | 10/10 | 12.38 ± 1.13 | 21.50 ± 3.32 | |
| Constant, 10 N | 8/10 | 12.86 ± 0.45 | 25.71 ± 5.11 | |
| Constant, 15 N | 2/10 | 15.36 ± 0.30 | 30.28 ± 6.32 | |
| Peeling | Variable, Fmax = 10 | 9/10 | 8.76 ± 1.65 | 18.34 ± 5.16 |
| Variable, Fmax = 15 | 9/10 | 11.39 ± 1.52 | 27.88 ± 9.74 | |
| Constant, 10 N | 8/10 | 12.98 ± 1.42 | 39.16 ± 11.13 | |
| Constant, 15 N | 6/10 | 14.79 ± 0.93 | 106.42 ± 61.97 |
Peak force and impulse: mean ± std over all 10 trials, including failures.
Then train on it
The pipeline is also a data engine.
Every successful execution is a demonstration with images, poses, and a force profile. So we let the pipeline collect 50 demonstrations per task at random object positions and trained a Diffusion Policy on them: two camera views plus the end-effector pose in, poses out. We trained two versions, one that also sees the measured contact force and one that doesn't.
At test time there is no force regulator. The policy just commands poses on new object placements and appearances, and the contact force comes from how deep it learned to press. Without force input it succeeds in 29/40 trials; with force input, 34/40. On most tasks the policy's force stays within about 2 N of the demonstrations. Peeling is the exception: the policies press harder than the demos, and the one with force input presses hardest, which peels more reliably.
| Wiping | Peeling | Stacking | Pressing | Total | |
|---|---|---|---|---|---|
| DP w/o force input | 8/10 | 8/10 | 8/10 | 5/10 | 29/40 |
| DP w/ force input | 9/10 | 10/10 | 8/10 | 7/10 | 34/40 |
Appendix
The prompts we used.
Every generation call is one shared system prompt with the task description dropped into it.
Video: The camera remains stationary while <TASK SPECIFIC PROMPT>. Audio: Emphasize the sound of physical interaction and contact force central to the action, while suppressing all ambient, background, and unrelated sounds.
The gripper, holding the eraser, slowly wipes the black marks off the whiteboard from right to left once, then retracts at the end.
The left gripper, holding the yellow peeler, slowly peels off a layer of the carrot skin, while the right gripper remains still.
The gripper's left finger slides fully under the chocolate box, forms a fully stable grasp, and slowly puts it on the pile of books.
The gripper performs a single press on the white button on the table and then rises back up, while remaining closed.
Strike the table exactly twice with a hammer: first with <F1> of force, then with <F2>. Each strike must produce exactly one clean contact without bouncing. (F1, F2) = (5 N, 20 N) or (20 N, 5 N)
Audio separation uses one SAM-Audio prompt for every task: “The sound of contact between gripper and <object>”.
