Ever wondered if the RL in RLHF is really needed? Worried that you might really need to understand how PPO works?
Worry no more, Direct Preference Optimization (DPO) allows you to fine-tune LMs directly from preferences via a simple classification loss, no RL required.
🧵 ->
I don't have a paper to write this in but there is an interesting property when thinking about iterative RL(HF) algorithms. It seems natural to use an improved policy to sample new data online when training LLMs -- turns out that this just lowers the weight on the KL constraint!
High-quality human feedback for RLHF is expensive 💰. AI feedback is emerging as a scalable alternative, but are we using AI feedback effectively?
Not yet; RLAIF improves perf *only* when LLMs are SFT'd on a weak teacher. Simple SFT on a strong teacher can outperform RLAIF! 🧵->
I have been waiting for this to be announced, it’s so amazing to see such elegant scaling of the Deep Think system where the same system can now achieve a gold at IMO!
Gemini 2.5 Deep Think is out!! We were able to improve the model substantially since our announcement at I/O, and it is a faster variation of the system that got Gold 🥇at IMO (still getting bronze level performance🥉!!)
The model is p good at detailed creative tasks too!
I cannot emphasize how good Mistral 7B is. It has been crushing some of the experiments I have been doing, makes such a huge difference in the fine-tuned performance. Kudos @MistralAI for creating and open-sourcing this model!
HELM v0.4.0 is out!
1) We have a new frontend (thanks to community contribution from Mike Lay).
2) We have added Mistral 7B, which really is punching above its weight (see crfm.stanford.edu/helm/v0.4.0/#/…), rivaling models an order of magnitude larger on the 16 core scenarios:
Embodied agents such as humans and robots live in a continual non-episodic world. Why do we continue to develop RL algorithms in episodic settings? This discrepancy also presents a practical challenge -- algorithms rely on extrinsic interventions (often humans) to learn ..
This is the “right” way to interpret DPO, ie. setting Z(x) to 1 and removing the DoF.
But, I think the narrative of removing RL from RLHF has played its part, so I want to course-correct a bit this new year — is DPO still RL(HF), which requires answering, what is RL? 🧵->
DPO's method of removing RL from RLHF is so based
> previously "forced" to sample trajectories with RL bc direct optimization would require an intractable partition fn Z(x)
> observe that the bradley-terry model has a few extra degrees of freedom
> simply set Z(x)=1
So, since we are posting science straight to twitter now, @ericmitchellai and I have some updates for potential overfitting in DPO.
TL;DR: we compared DPO to IPO and cDPO (DPO + label noise) on 3 different datasets, and we didn't observe any significant advantage (yet). 🧵->
I wrote an accessible post on why unsupervised learning is important for reinforcement learning and robotics, and how our work on DADS is improving skill discovery and making it feasible in real life. Check it out here!
Introducing DADS, a novel unsupervised #ReinforcementLearning algorithm for discovering task-agnostic skills, based on their predictability and diversity, that can be applied to learn a broad range of complex behaviors. Learn more at: goo.gle/2ZMPRxe
