Eric Zhao (@ericzhao28) / X

Eric Zhao

69 posts

Eric Zhao

@ericzhao28

llms and reasoning @openai previously: @googleai, @berkeley_ai, @caltech

San Francisco

Joined July 2020

Eric Zhao
@ericzhao28
Mar 17, 2025
Thinking for longer (e.g. o1) is only one of many axes of test-time compute. In a new @Google_AI paper, we instead focus on scaling the search axis. By just randomly sampling 200x & self-verifying, Gemini 1.5 ➡️ o1 performance. The secret: self-verification is easier at scale!
Eric Zhao
@ericzhao28
Mar 17, 2025
Replying to @ericzhao28
3 ways to burn through compute: 1️⃣ search over more solutions (e.g., sample many responses in parallel), 2️⃣ spend more time reasoning through each solution (e.g., use o1-style RL or COT prompts), and 3️⃣ improve/adapt your model (e.g. use GPT4.5 instead of GPT4, or finetune).
Eric Zhao
@ericzhao28
Mar 17, 2025
Replying to @ericzhao28
Our paper focuses on this search axis and its scaling trends. For example, by just randomly sampling 200 responses and self-verifying, Gemini 1.5 (an ancient early 2024 model!) beats o1-Preview and approaches o1. This is without finetuning, RL, or ground-truth verifiers.
Eric Zhao
@ericzhao28
Mar 17, 2025
Replying to @ericzhao28
This was surprising: search is bottlenecked by verification, and models are notoriously bad at self-verifying (think hallucinations) and self-consistency doesn't scale. The magic is that self-verification naturally becomes easier at scale! You'd expect that picking out a correct
Eric Zhao
@ericzhao28
Mar 17, 2025
Replying to @ericzhao28
While longer reasoning has been the 🌶️ topic, we can't forget search! Search is arguably more fundamental, can be scaled ad hoc, is embarrassingly parallel, and is the only axis that can be scaled by itself without a ceiling. (+ these axes are all complementary anyways!) There's
Eric Zhao
@ericzhao28
Mar 17, 2025
Replying to @ericzhao28
Check out the paper and an accompanying blog at these 🔗s! eric-zhao.com/blog/sampling arxiv.org/abs/2502.01839 (with Pranjal Awasthi and Sreenivas Gollapudi)
Eric Zhao
@ericzhao28
Mar 17, 2025
Replying to @ericzhao28
Reason 1: Scaling search doesn't just increase the probability of a correct attempt, it also increases the probability of an obviously correct attempt.
Eric Zhao
@ericzhao28
Mar 17, 2025
Replying to @ericzhao28
Reason 2: Models are better at self-verifying when they can compare between many attempts. Why? Because the diff between attempts strongly hints at the locations of errors, and self-verification capability skyrockets if you tell models where errors/hallucinations may lie.
Eric Zhao
@ericzhao28
Mar 17, 2025
Replying to @ericzhao28
+ You can also use search to scale self-verification. All these advantages stack to empower models to correctly pick out singular correct attempts from amongst hundreds of incorrect ones. It's also worth noting that we focused on scaling a minimum viable form of search +
Eric Zhao
@ericzhao28
Mar 11, 2025
Finetuning is great for customizing model behavior but notoriously bad at teaching knowledge 📚. But what's the actual difference between these regimes? We trained 700 Gemini 1.5 models on specially tailored datasets to find out. Here's what we saw ⬇️
Eric Zhao
@ericzhao28
Mar 6, 2025
(More news!) I wrote a new blog post on our current understanding of multi-distribution learning (MDL) in 2025. I give a gentle intro, 🌶️ but belated updates to our COLT open problem, and discuss some fundamental unresolved questions. Link ⬇️
Eric Zhao
@ericzhao28
Mar 18, 2025
Replying to @ericzhao28
*Oops I meant to tag @GoogleAI and @berkeley_ai; I've apparently tagged a spam account instead. (thanks @calebsirak for the catch)
Eric Zhao
@ericzhao28
Mar 6, 2025
Asking for calibration is asking to be lied to. AI alignment is hard in forecasting! Short 🧵 on the difficulty of (+ a solution to) evaluating forecasters so that you can both trust their forecasts and not incentivize them to lie. (new 📄 with @MingdaQiao + blog post ⬇️)
Eric Zhao
@ericzhao28
Mar 18, 2025
Replying to @dbmikus @ziv_ravid and @Google_AI
Good q! The most cost-efficient balance of search+reasoning+model size scaling is pretty sensitive to your problem and model in my experience scaling gemini 2.0 thinking from 60k -> 600k tokens is not as efficient doing search by sampling 60k 10x, but scaling 8k->60k is worth it