Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators

Motion Guidance is a method to achieve motion-based editing. Given an image to edit and a flow field, indicating where each pixel should go, we produce a new image with the desired motion.

Our method is zero-shot, and supports motions such as rotations, translations, stretches, scaling, shrinking, homographies, general deformations, and works on both generated and real images.

(If you are interested in using the interactive flow visualizations from this page for your own project, a minimum working example is provided here.)

To achieve motion-based editing, we propose using guidance [1] during sampling in a diffusion model. At each time step in the reverse process, we perturb the noisy estimate in the direction that minimizes some loss function. As a loss function, we propose using the difference between the desired motion and the current motion of the noisy sample, with respect to the original image, as estimated by an off-the-shelf (differentiable) optical flow network [2]. Effectively, we find a sample that is likely under the diffusion model, while attaining a low loss.

In order to achieve good results, we find we also need to use a few tricks, including color regularization, reconstruction guidance [3] [4], occlusion masking, and edit masking. Please see the paper for additional details.

A GUI for Flow Construction

All optical flows used in this paper (with the exception of Motion Transfer results) are generated by composing elementary flows together, and masking with segmentations masks from SAM. These elementary flows consist of translations, rotations, scaling, and more complex deformations. We show examples of how these flows can be created using a simple UI below:

Construction of elementary flows with a GUI is fairly straightforward, using a click and drag interface to specify translations, rotations, scaling, and stretching. More complex deformations can be constructed by composing or interpolating these flows:

Below, we show various translations, scalings, and stretches to the same source image.

In some cases, we can extract motion from video and apply that motion to images. For example, below we extract to motion from the video of the Earth spinning, and apply it to various, real animal images. The extracted flow is not perfectly accurate, and does not perfectly align with the source image, but because we optimize a soft objective, our method is able to produce reasonable results.