Ziang Xiao

@ZiangXiao

Assist. Prof. @JHUCompSci / HCI

Baltimore, MD

ziangxiao.com

Joined December 2013

650 Following

1,329 Followers

Replies Reposts Media

Pinned
Ziang Xiao
@ZiangXiao
Oct 22
The field should shift the focus from benchmark scores to item response is essential for building the science for AI eval and trustworthy eval reporting! We are actively building OpenEval to serve as the foundational data layer for the science of AI eval.
Han Jiang
@SalomeJiang7
Oct 22
AI Eval Should Require Standardized Item-Level Data Releases In our #NeurIPS2026 position paper, we argue the science of AI evaluation requires shifting focus from benchmark scores to item-level data. We present OpenEval to enable such releases. 🔗open-eval.com #AI #NLP
Ziang Xiao
@ZiangXiao
Oct 27
Good to see the idea from HCI being picked up by other fields. However, interview as a method is way more complex than it seems. An AI interviewer may be easy (?) to build but difficult to evaluate properly. Read @Jonathan_Ivey8's InterviewPlayground (arxiv.org/html/2610.1202…).
The Review of Economic Studies
@RevEconStudies
Oct 26
Can AI conduct qualitative interviews with thousands of people? In a new REStud paper, Friedrich Geiecke and @XJaravel develop an open-source approach that combines the depth of qualitative interviews with the scale of surveys. 1/6
1
Ziang Xiao
@ZiangXiao
Oct 24
Qualitative interview has always been a fascinating research methods to me. In this work, we tried to understand qualitative interview. Glad this paper is selected as oral. Find @Jonathan_Ivey8 #EMNLP26!!
Jonathan Ivey @ EMNLP 26
@Jonathan_Ivey8
May 19
What makes a good participant response in qualitative research? In our new paper, we investigate this by analyzing 16,940 responses across 343 interviews to see which characteristics of a response make it most likely to contribute to a study’s findings. Here’s what we found …
Ziang Xiao
@ZiangXiao
Oct 21
Many AI evaluations now use IRT to select test items or estimate model capability. But we found that common IRT models can produce unreliable item-level and ranking inferences when applied to AI benchmarks. The science of AI evaluation needs a stronger foundation!
Han Jiang
@SalomeJiang7
Oct 20
#IRT is increasingly adopted in #AI evaluation, but can we trust it when AI benchmark data differs from human test data❓ Our 18K simulations highlight the need to examine IRT assumptions and estimation settings before drawing conclusions. #NeurIPS2026 🔗arxiv.org/abs/2607.15190
Ziang Xiao
@ZiangXiao
Oct 19
CEDI creates ecologically valid AI evaluation through context grounded dynamic probing. Think CEDI as an experienced oral examiner. Let’s rethink benchmarking. #neurips2026
William Yijiang Li
@Williamiumli
Oct 19
Are static benchmarks enough to evaluate VLMs in the real world? 🤔 In our #NeurIPS2026 paper, we move beyond traditional static benchmarks and introduce CEDI: Give AI an oral exam ! We introduce CEDI, a framework that evaluates VLMs through dynamic, contextualized,

Ziang Xiao

Ziang Xiao’s posts