I've been wanting to get to writing more often, so I started a little blog attached to my personal website.
My first blog post is a tutorial on building TPU flash attention from the ground up. Let me know what you think!
A much needed step in world modeling. By generating 360° panos, world models will truly track everything happening around them, serving as a better form of memory.
Huge shoutout to @Dazitu_616 for spearheding this awesome work. It was amazing to work with you!
📢Introducing 360Anything, our method for lifting any perspective image or video to gravity-aligned 360° panoramas without using any camera or 3D information. This enables consistent novel view synthesis and 3D scene reconstruction.
Project page: 360anything.github.io
🧵
[[THREAD]] Happy to announce 4DiM, our diffusion model for novel view synthesis of scenes! 4DiM allows camera+time control with as few as one input image.
Joint work with @srbhsxn* @lala_yi_li* @taiyasaki@fleet_dj
*equal contribution
Very excited to share VIDIM, our diffusion model for video interpolation. Cheap and high quality video diffusion will become a reality. I have also believed for a while now that using pixels as inputs is severely underrated-- there is so much more signal in there v.s. text.
VIDIM: Video Frame Interpolation Using Diffusion Models. Watch out for our generative frame interpolation magic in CVPR 2024. With Siddhant Jain, Daniel Watson, Eric Tabellion, Aleksander Holynski, and Ben Poole arxiv.org/abs/2404.01203
Check out @clark_kev’s and my paper on fine-tuning diffusion models on differentiable rewards! We present DRaFT, which computes gradients through diffusion sampling. DRaFT is efficient & works across many reward functions.
With @kswersk, @fleet_dj
arXiv: arxiv.org/abs/2309.17400
