Kevin Meng

@mengk20

@TransluceAI

mengk.me

Joined August 2016

Replies Reposts Media

Pinned
Kevin Meng
@mengk20
Oct 23, 2024
why do language models think 9.11 > 9.9? at @TransluceAI we stumbled upon a surprisingly simple explanation - and a bugfix that doesn't use any re-training or prompting. turns out, it's about months, dates, September 11th, and... the Bible?
00:00
Transluce
@TransluceAI
Oct 23, 2024
Monitor: An Observability Interface for Language Models Research report: transluce.org/observability-… Live interface: monitor.transluce.org (optimized for desktop)
68
Kevin Meng
@mengk20
Aug 31
It's been fantastic watching @_ddjohnson, @cogconfluence, and team build out this very important evaluation. It sets several precedents I'm excited about: - Rich, publicly-verifiable evals with all the data available through an interactive experience (next up: reproducible!) -
Transluce
@TransluceAI
Aug 31
Today, Transluce is releasing the most expansive independent evaluation to date of how AI systems respond to users experiencing mental health crises. We evaluated 77 model variants from OpenAI, Anthropic, Google, Meta, SpaceXAI, Thinking Machines, DeepSeek and Moonshot AI.
00:00
1
Kevin Meng
@mengk20
Aug 14, 2025
Go check @SolaAI_ out, they're doing insanely cool work!! Congrats @NeilDeshmukh @jess123555 :)
Neil Deshmukh
@NeilDeshmukh
Aug 14, 2025
Excited to announce that @SolaAI_ has raised a $17.5M Series A led by @a16z with support from @conviction @ycombinator, bringing total funding to $21M 🚀 From the start, we set out to reimagine human-AI interaction to push the boundaries of process automation. Our agents watch
00:00
Kevin Meng
@mengk20
Mar 28, 2025
i'm really excited about our Docent roadmap :) we're developing: - open protocols, schemas, and interfaces for interpreting AI agent traces - automated systems that can propose and verify general hypotheses about model behaviors, using eval results come work with us! roles 👇
Transluce
@TransluceAI
Mar 24, 2025
Replying to @TransluceAI
If you want to help build Docent and other AI tools at Transluce, we’re hiring for our product team. Apply below! jobs.gem.com/transluce/am9i…
6
Kevin Meng
@mengk20
Mar 25, 2025
AI models are *not* solving problems the way we think using Docent, we find that Claude solves *broken* eval tasks - memorizing answers & hallucinating them! details in 🧵 we really need to look at our data harder, and it's time to rethink how we do evals...
Transluce
@TransluceAI
Mar 24, 2025
00:44
To interpret AI benchmarks, we need to look at the data. Top-level numbers don't mean what you think: there may be broken tasks, unexpected behaviors, or near-misses. We're introducing Docent to accelerate analysis of AI agent transcripts. It can spot surprises in seconds. 🧵👇
16