Replying to @ValsAI
Congrats to the @Xiaomi team on the release. Full benchmark-by-benchmark results vals.ai/models/xiaomi_… vals.ai/models/xiaomi_…
Vals AI
1,266 posts
Vals AI
@ValsAI
- Replying to @ValsAISettings: temperature=1, top_p=0.95, 128k max output tokens via Xiaomi's API. The model has a 1M-token context window. Both models are cheap to run: MiMo 2.5 at $0.04/test and the Pro at $0.09/test, putting them among the more cost-efficient options on the index.
- Replying to @ValsAIBoth models will sometimes reason, and occasionally answer, in Chinese regardless of the prompt language. We saw a related issue on SAGE, where the model skips the system prompt and just solves the underlying question instead of grading the student response. It showed up more
- Vals AI reposted
Consistent with Jack Clark + Gillian Hadfield's regulatory markets paper 3 years ago. Private orgs are often better at building tech needed to keep up with the market rather than slowing it down. But need the right incentive structure to be public good aligned.
Today I'm publishing a new essay, Policy on the AI Exponential. AI is progressing extremely fast—much faster than the policy process was built to handle. The essay lays out where I think the technology is now, and the action needed to close the gap: darioamodei.com/post/policy-on… - Vals AI reposted
Replying to @xeophonUpdate: They listened!! Great transparency and quick reaction, kudos :) - Vals AI repostedThe benchmark challenge is one of the existential risks to AI progress
00:00 - Replying to @ValsAIGoing forward, evaluations will have to report not only on capability, but also how much of that capability is available to users. We will soon be sharing updated methodology on tracking and reporting APIs that ship with fallback models or have high rejection rates.
- Replying to @ValsAIThe rejected tasks (16% overall) for Terminal Bench 2.1 were for biology and cyber. The tasks were: write-compressor, vulnerable-secret, sam-cell-seg, protein-assembly, path-tracing-reverse, password-recovery, model-extraction-relu-logits, feal-linear-cryptanalysis,
- Replying to @ValsAIWe are also releasing the per-benchmark fallback rates. The majority of benchmarks had no or very low fallback rates, but as mentioned in our previous post, the safety classifier was highly sensitive to certain benchmarks. For example, MMLU Biology and Health have nearly a 100%
