Appen Research

@AppenResearch

Human data for frontier AI. Research and insights from Appen.

Washington, USA

appen.com

Joined May 2026

217 Following

1,097 Followers

Replies Reposts Media

Appen Research
@AppenResearch
Oct 29
"Benchmaxxed: Why agents that ace the eval still fail the job." Sergio Bruccoleri @SergioBrucc took the stage at Bench Summit by @kaggle yesterday. His point: a pass can hide skipped work, and how you score cheating runs alone can move a headline result about 6x. He closed with
1
Appen Research
@AppenResearch
Oct 26
An agent can ace the eval and still fail the job. So what was the benchmark actually measuring? Sergio Bruccoleri @SergioBrucc takes that on at @kaggle Bench Summit in SF, Wed Oct 28, 3 PM PT, on browser agents in enterprise workflows. Bring counterexamples.
Sergio Bruccoleri
1
Appen Research
@AppenResearch
Oct 23
A checker can verify 180 million lines of proof. It can't tell you if the spec says what you meant. Evals have the same shape. For MedTerm-90 we reviewed 12,326 near-miss pairs under one rule: accept only what a clinician would sign off on. 21.2% passed. The rule did the work.
Anthropic
@AnthropicAI
Oct 22
New on our Frontier Red Team blog: using Claude to formally verify software at large scale. Formal verification can prove that code is free of certain bugs, but historically it has only been feasible for small codebases (or required a lot of expert time). With AI models,
2
Appen Research
@AppenResearch
Oct 22
RL tasks are getting more expensive while environments are getting cheaper. That isn't a contradiction. Nobody was really paying for the sandbox. You pay for the grader: one that rejects answers that look right, still works after the environment is patched, and has had its
Marco Mascorro
@Mascobot
Oct 21
The market price for a single top top-quality RL environment task has gone bananas. It's just wild.
Appen Research
@AppenResearch
Oct 22
Human vs synthetic labels for alignment evals usually gets framed as either/or. In practice it splits by the kind of question. Synthetic labels hold up well on clear rule violations: leaked data, a refusal that should have happened, a broken format. They're much weaker on