Vals AI (@ValsAI) / X

Vals AI

1,259 posts

Vals AI

@ValsAI

Public LLM Evaluation // vals.ai @8vc @BloombergBeta @pearvc

San Francisco, CA

Joined March 2024

Vals AI reposted
Rayan Krishnan
@RayanKrishnan
5h
Consistent with Jack Clark + Gillian Hadfield's regulatory markets paper 3 years ago. Private orgs are often better at building tech needed to keep up with the market rather than slowing it down. But need the right incentive structure to be public good aligned.
Dario Amodei
@DarioAmodei
14h
Today I'm publishing a new essay, Policy on the AI Exponential. AI is progressing extremely fast—much faster than the policy process was built to handle. The essay lays out where I think the technology is now, and the action needed to close the gap: darioamodei.com/post/policy-on…
1.4K
Vals AI reposted
Florian Brand
@xeophon
13h
Replying to @xeophon
Update: They listened!! Great transparency and quick reaction, kudos :)
Vals AI
@ValsAI
13h
We have added the ability to view Fable 5 scores with Opus 4.8 fallbacks disabled to the Vals AI website (refusals are marked as zero). The eval community was ill-equipped for this, but transparency is our first priority. We’re anticipating more models like this, and are
00:00
2.2K
Vals AI reposted
Rayan Krishnan
@RayanKrishnan
12h
The benchmark challenge is one of the existential risks to AI progress
00:00
1.6K
Vals AI
@ValsAI
13h
Replying to @ValsAI
Going forward, evaluations will have to report not only on capability, but also how much of that capability is available to users. We will soon be sharing updated methodology on tracking and reporting APIs that ship with fallback models or have high rejection rates.
605
Vals AI
@ValsAI
13h
Replying to @ValsAI
We also saw a small quantity of rejections on our Finance Agent Benchmark. These were only on questions analyzing pharmaceutical or biological public companies. Here is a stripped-down reproduction (real FAB questions are far harder)
678
Vals AI
@ValsAI
13h
Replying to @ValsAI
The rejected tasks (16% overall) for Terminal Bench 2.1 were for biology and cyber. The tasks were: write-compressor, vulnerable-secret, sam-cell-seg, protein-assembly, path-tracing-reverse, password-recovery, model-extraction-relu-logits, feal-linear-cryptanalysis,
692
Vals AI
@ValsAI
13h
Replying to @ValsAI
We are also releasing the per-benchmark fallback rates. The majority of benchmarks had no or very low fallback rates, but as mentioned in our previous post, the safety classifier was highly sensitive to certain benchmarks. For example, MMLU Biology and Health have nearly a 100%
2.4K
Vals AI
@ValsAI
13h
We have added the ability to view Fable 5 scores with Opus 4.8 fallbacks disabled to the Vals AI website (refusals are marked as zero). The eval community was ill-equipped for this, but transparency is our first priority. We’re anticipating more models like this, and are
00:00
9.3K
Vals AI reposted
Nicolas Bustamante
@nicbstme
Jun 9
What I find fascinating with Claude Fable 5 is it proves once again that large generalist models will outperform vertical ones. On ProofBench (graduate-level formal math benchmark in Lean, where a proof either compiles or it doesn't) Fable 5 beat Harmonic's Aristotle, 77% vs
64K
Vals AI reposted
Rayan Krishnan
@RayanKrishnan
Jun 9
Our Fable benchmarks are out, as reported in the NYT. Some of my quick takes: - Everyone should readjust their timelines, pace is not slowing. This model sweeps without tradeoff on domains/tasks. It's 75% overall on the Vals Index (5%+ 4.8 and 8%+ next non ant model) - Model
Vals AI
@ValsAI
Jun 9
The most anticipated model in months is finally here and it lives up to the hype. Claude’s Fable 5 topped nearly our entire benchmark suite, taking #1 on the Vals Index, Vals Multimodal, CorpFin, MedScribe and every single coding benchmark we run.
32K
Vals AI
@ValsAI
Jun 9
Replying to @ValsAI
View the full Vibe Code Bench results and methodology on Vals AI. Congratulations @AnthropicAI .
899
Vals AI
@ValsAI
Jun 9
Replying to @ValsAI
Here are some examples of applications the model generated. There is a noticeable improvement in style from previous models. The model added the “No spreadsheets were harmed in the making of this ledger” itself - this was not included in the prompt.
1.2K
Vals AI
@ValsAI
Jun 9
Replying to @ValsAI
Fable 5 gains its edge by making minimal, well-scoped choices. Opus 4.8 overbuilds past the spec, adding complexity where bugs tend to live
928