Pinned
What can unsupervised representation learning learn from deep RL? As it turned out, we can learn representations (unsupervised) by making plans!
Check out our new work at
Ge Yang
2,315 posts
Ge Yang
@EpisodeYang
I am planting acorns one at a time with policy gradient.
- Sora from @OpenAI is super impressive, but how consistent are the geometries? We ran this through our fast 3DGS pipeline, and here are some of the early results. This is a reconstruction 👉 1/n
00:00 - That 675M is 24% of the company -- 675M round at 2.8B valuation. In the meantime, enjoy what we achieved with a nimble team of just six people :) paper link: arxiv.org/abs/2402.16796 w/@xiaolonw @xuxin_cheng @JiYandong @RchalYang
00:00OpenAI + humanoid robots — we’re collaborating with @Figure_robot to expand our multimodal models to robotic perception, reasoning, and interaction. prnewswire.com/news-releases/… - Finally came out 🔥 Humanoid code name "Sonnie" (from @DavidSFWilson) @RoboticsSciSys - @Tesla is probably going to copy their design with Optimus - Lots of good design decisions - An agile controller trained w/ RL in #isaacgym by @nvidia (not shown) The company was founded in
00:00 - Fun fact: Do you see the white box on top? We stripped an Apple M1 Mac to its core to run visual processing onboard the robot : ) @xiaolonw @RchalYangMIT researchers taught a robot dog to perceive a 3D world using Neural Volumetric Memory (NVM). This technique lets the bot climb stairs, step over gaps & run autonomously w/a single neural network. Paper: bit.ly/3LNYWfy Video: bit.ly/3LSciaA w/@UCSanDiego
00:00 - Folks at Unitree are incredibly focused, and they work incredibly hard.
00:00 - Today 4pm ET, @TheGregYang from @MSFTResearch #physics and #ML will be giving a talk at the Embodied Intelligence @MIT_CSAIL on "Tuning GPT-3 on a Single GPU via Zero-Shot Hyper-parameter Transfer"! Live-streaming from: youtube.com/channel/UCnXGb……, co-hosted with @du_yilun !
- We "bake" SE(3) equivariance into the robot's world representation by training. Seeing how well this worked, in real-life, with just onboard camera and compute, is just SURREAL. A lot more to be done, in closing the gap between vision + control, on dynamic and contact-richThe robot climbs stairs🏯, steps over stones 🪨, and runs in the wild🏞️, all in one policy, without any remote control! Our #CVPR2023 Highlight paper achieves this by using RL + a 3D Neural Volumetric Memory (NVM) trained with view synthesis! rchalyang.github.io/NVM/
00:00 - Replying to @EpisodeYangIn the other videos, the scenes sure look nice, but the photogrammetry pipeline fails because there isn't enough overlap and baseline separation between the camera poses. Does this mean sora failed to model geometry? 3/n
00:00 - Replying to @EpisodeYangJust got back to my desk. Here is what it looks like when you move around. We noticed that the 3D reconstruction using our photogrammetry pipeline only works well when it has a circular camera path like in this Big Sur video. In the other videos 2/n
00:00 - You can tell there is a lot of attention to the details. The UMI gripper can be printed without support! Great work @chichengcc 👍
- Man, I love MIT.
- My favorite part of research is you get to work on interesting problems together, and make friends along the way.
