About⁠

I lead R&D for self-improving AI scientists at the Allen Institute for AI (Ai2) and am a lead on the Asta AI-for-science project.

I have published 35+ peer-reviewed papers at top AI and human-computer interaction (HCI) conferences and journals over 15+ years, with 5+ paper honors, and mentored 20+ PhD students and postdocs. I try to make AI useful for society through my work on AI for science and AI for accessibility, which has been covered by Nature.

I pioneered self-improving crowdsourcing and collective intelligence systems, which led to my PhD dissertation (Self-Improving Crowdsourcing…) and a book chapter (MIT Press 2015).

Self-improvement and orchestration of modern AI agents revisit many of the same core challenges, including intelligent task prioritization (HCOMP 2013) and routing (HCOMP 2014), worker testing and monitoring (AAMAS 2016), answer deliberation (HCOMP 2016, CHI 2019), and requirements elicitation (UIST 2018).

Rigorous evaluation is essential for measuring progress, especially in self-improving systems; I have studied evaluation for agents (ICLR 2026), scientific prediction (arXiv 2026), hard-to-verify tasks (TACL 2026), and meta-learning (NeurIPS 2021) — and how to scale reliable human judgments through collective intelligence (EMNLP 2022).

My work on HCI aims to help people steer and work with AI.

Previously, I was a postdoc at Stanford, earned my Ph.D. from the University of Washington, and received my A.B. from Harvard College. I am also a professionally trained musician with an M.M. from New England Conservatory.

Tools (selected)⁠

Asta plugins allenai/asta-plugins⁠ AIAI for sciencePlugins and skills that give an agent scientific capabilities. Released 2026 · Lead
AstaBench allenai/asta-bench⁠ AIAI for scienceEvaluationAn evaluation framework for measuring the scientific research abilities of AI agents, with over 2,400 examples across 11 benchmarks, and a public leaderboard. Released 2025 · Lead Blog Blog (new results, spring 2026) ICLR 2026
agent-baselines allenai/agent-baselines⁠ AIAI for scienceEvaluationA growing set of agents runnable as Inspect solvers, including the Asta agents and the baselines reported in AstaBench. Released 2025 · Lead ICLR 2026

Show 4 more toolsShow fewer tools

Asta⁠ AIAI for scienceAi2’s hosted agent for scientific research. Released 2025 · Lead Vision
Paper+Figure QA⁠ AIAI for scienceAccessibilityAsk a question about any paper, answered from the full content including figures and tables. Released 2025 · Mentor · Led by Arnavi Chheda-Kothary Chrome extension ASSETS 2026
agent-eval allenai/agent-eval⁠ AIEvaluationRuns agents against a suite of Inspect-formatted evals, extracting token usage and cost from logs and submitting results to a leaderboard. Used by AstaBench. Released 2025 · Lead ICLR 2026
Pier allenai/pier⁠ AIEvaluationAn interactive workspace manager for coding agents — Claude Code, Codex, Gemini CLI, OpenHands, Aider and others — for solving and developing Harbor tasks. Released 2026 · Lead

Publications⁠

Pinned

AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite⁠

AIAI for scienceEvaluationConference paper

Jonathan Bragg, Mike D’Arcy, Nishant Balepur, Dan Bareket, Bhavana Dalvi, Daniel S. Weld

ICLR 2026 · Oral ·

Oral — top 1.2% of submissions

223 given, of 18,949 submissions. ICLR 2026 accepted 5,339 overall (28.2%).

acceptance figures · honor count

PDF AstaBench agent-baselines agent-eval Blog Blog (new results, spring 2026)

AI agents hold the potential to revolutionize scientific productivity by automating literature reviews, replicating experiments, analyzing data, and even proposing new directions of inquiry; indeed, there are now many such agents, ranging from general-purpose “deep research” systems to specialized science-specific agents, such as AI Scientist and AIGS. Rigorous evaluation of these agents is critical for progress. Yet existing benchmarks fall short on several fronts: they often (1) lack reproducible agent tools necessary for a controlled comparison of core agentic capabilities; (2) do not account for confounding variables such as model cost and tool access; (3) do not provide standardized interfaces for quick agent prototyping and evaluation; (4) fail to provide holistic, product-informed measures of real-world use cases such as science research; and (5) lack comprehensive baseline agents necessary to identify true advances. In response, we define principles and tooling for more rigorously benchmarking agents. Using these, we present AstaBench, a suite that provides a holistic measure of agentic ability to perform scientific research, comprising 2400+ problems spanning the entire scientific discovery process and multiple scientific domains, and including many problems inspired by actual user requests to deployed Asta agents. Our suite comes with the first scientific research environment with production-grade search tools that enable controlled, reproducible evaluation, better accounting for confounders. Alongside, we provide a comprehensive suite of nine science-optimized classes of Asta agents and numerous baselines. Our extensive evaluation of 57 agents across 22 agent classes reveals several interesting findings, most importantly that despite meaningful progress on certain individual aspects, AI remains far from solving the challenge of science research assistance.
```
@inproceedings{bragg-iclr26-astabench,
  title = {Asta{B}ench: Rigorous Benchmarking of {AI} Agents with a Scientific Research Suite},
  author = {Bragg, Jonathan and D'Arcy, Mike and Balepur, Nishant and Bareket, Dan and Dalvi, Bhavana and Feldman, Sergey and Haddad, Dany and Hwang, Jena D. and Jansen, Peter and Kishore, Varsha and Majumder, Bodhisattwa Prasad and Naik, Aakanksha and Rahamimov, Sigal and Richardson, Kyle and Singh, Amanpreet and Surana, Harshit and Tiktinsky, Aryeh and Vasu, Rosni and Wiener, Guy and Anastasiades, Chloe and Candra, Stefan and Dunkelberger, Jason and Emery, Dan and Evans, Rob and Hamada, Malachi and Huff, Regan and Kinney, Rodney and Latzke, Matt and Lochner, Jaron and Lozano-Aguilera, Ruben and Nguyen, Cecile and Rao, Smita and Tanaka, Amber and Vlahos, Brooke and Clark, Peter and Downey, Doug and Goldberg, Yoav and Sabharwal, Ashish and Weld, Daniel S.},
  booktitle = {Proceedings of the Fourteenth International Conference on Learning Representations},
  series = {{ICLR}},
  year = {2026},
  month = apr,
  day = {23},
  url = {https://openreview.net/forum?id=M7TNf5J26u},
  eprint = {2510.21652},
  archivePrefix = {arXiv}
}
```

2026

Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarks⁠

AIAI for scienceEvaluationJournal article

Jena D. Hwang, Varsha Kishore, Amanpreet Singh, Dany Haddad, Aakanksha Naik, Malachi Hamada, Jonathan Bragg, Sergey Feldman

TACL 2026 · To appear

PDF (arXiv) HTML (arXiv)

Recent advances have made long-form report-generating systems widely available. This has prompted evaluation frameworks that use LLM-as-judge protocols and claim verification, along with meta-evaluation frameworks that seek to validate these methods. Many of the meta-evaluations estimate an evaluation quality’s by comparing its assessments against human pairwise preferences. Prior work, however, suggests that human pairwise preference may be overly simplistic and can fail to capture nuances of expert expectations. We conduct a case study in meta-evaluation for long-form QA benchmarks using ScholarQA-CS2, a benchmark designed for assessing retrieval-augmented deep-research QA in the scientific domain. We comprehensively validate the benchmark through human pairwise preference judgments, then critically examine the strengths, weaknesses, and confounders of this approach. We show that pairwise preference rankings are best suited for system-level evaluation, while explicit metric-wise annotations and expert annotators are critical for reliable metric-level assessment, with subjectivity remaining a key challenge. Based on our findings, we offer practical guidelines for designing future meta-evaluations that better align evaluation methods, annotator expertise, and reporting practices. By surfacing these methodological challenges, we aim to advance evaluation standards for deep-research systems.
```
@article{hwang-tacl26-deep-research,
  author = {Hwang, Jena D. and Kishore, Varsha and Singh, Amanpreet and Haddad, Dany and Naik, Aakanksha and Hamada, Malachi and Bragg, Jonathan and D'Arcy, Mike and Weld, Daniel S. and Wang, Lucy Lu and Downey, Doug and Feldman, Sergey},
  title = {Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form {QA} Benchmarks},
  journal = {Trans. Assoc. Comput. Linguistics},
  series = {{TACL}},
  year = {2026},
  month = dec,
  eprint = {2603.06942},
  archivePrefix = {arXiv}
}
```
LitPivot: Developing Well-Situated Research Ideas Through Dynamic Contextualization and Critique within the Literature Landscape⁠

HCIAIAI for scienceConference paper

Hita Kambhamettu, Bhavana Dalvi Mishra, Andrew Head, Jonathan Bragg, Aakanksha Naik, Joseph Chee Chang, Pao Siangliulue

UIST 2026 · To appear

PDF (arXiv) HTML (arXiv)

Developing a novel research idea is hard. It must be distinct enough from prior work to claim a contribution while also building on it. This requires iteratively reviewing literature and refining an idea based on what a researcher reads; yet when an idea changes, the literature that matters often changes with it. Most tools offer limited support for this interplay: literature tools help researchers understand a fixed body of work, while ideation tools evaluate ideas against a static, pre-curated set of papers. We introduce literature-initiated pivots, a mechanism where engagement with literature prompts revision to a developing idea, and where that revision changes which literature is relevant. We operationalize this in LitPivot, where researchers concurrently draft and vet an idea. LitPivot dynamically retrieves clusters of papers relevant to a selected part of the idea and proposes literature-informed critiques for how to revise it. A lab study (n = 17) shows researchers produced higher-rated ideas with stronger self-reported understanding of the literature space; an open-ended study (n = 5) reveals how researchers use LitPivot to iteratively evolve their own ideas.
```
@inproceedings{kambhamettu-uist26-litpivot,
  author = {Kambhamettu, Hita and Mishra, Bhavana Dalvi and Head, Andrew and Bragg, Jonathan and Naik, Aakanksha and Chang, Joseph Chee and Siangliulue, Pao},
  title = {LitPivot: Developing Well-Situated Research Ideas Through Dynamic Contextualization and Critique within the Literature Landscape},
  booktitle = {Proceedings of the 39th Annual {ACM} Symposium on User Interface Software and Technology},
  series = {{UIST}},
  year = {2026},
  month = nov,
  eprint = {2604.02600},
  archivePrefix = {arXiv}
}
```
Ream: Unfolding Mutual Awareness in Human-Agent Workspaces⁠

HCIAIAI for sciencePreprint

Peiling Jiang, Sangho Suh, Varsha Kishore, Jonathan Bragg, Joseph Chee Chang

arXiv, Oct 2026

PDF HTML

As AI agents work alongside humans in shared workspaces, a mutual awareness challenge arises: agents act at speeds that outpace human monitoring, and users’ evolving interests are not always expressed in chat. This challenge is especially pressing in literature review, where both parties retrieve, read, and synthesize a growing body of papers. We present Ream, a literature review workspace that supports mutual awareness through structured artifacts, bidirectional engagement tracking, and localized visualizations. Users can see each party’s activity within these documents, and agents can retrieve the same history to guide their work. In studies with eighteen researchers, participants used these traces to inspect evidence, steer agents, communicate through annotations, and reflect on their research focus. Shared histories also helped agents build on earlier work. These findings inform how engagement traces within shared documents can support transparency, personalized assistance, and coordination in human-agent knowledge work.
```
@article{jiang-2026-ream,
  author = {Jiang, Peiling and Suh, Sangho and Kishore, Varsha and Bragg, Jonathan and Xia, Haijun and Siangliulue, Pao and Weld, Daniel S. and Zhang, Amy X. and Chang, Joseph Chee},
  title = {Ream: Unfolding Mutual Awareness in Human-Agent Workspaces},
  journal = {CoRR},
  volume = {abs/2610.09497},
  year = {2026},
  url = {https://doi.org/10.48550/arXiv.2610.09497},
  doi = {10.48550/ARXIV.2610.09497},
  eprint = {2610.09497},
  month = oct,
  archivePrefix = {arXiv}
}
```
Querying Multimodal Scientific Papers with AI: Practices and Preferences Across Blind, Low-Vision, and Sighted Scientists⁠

HCIAccessibilityAIAI for scienceConference paper

Arnavi Chheda-Kothary, Lucy Lu Wang, Joseph Chee Chang, Jonathan Bragg

ASSETS 2026 · To appear

PDF (arXiv) HTML (arXiv) Paper+Figure QA

Visual diagrams, figures, and tables are central to scientific papers, and convey information beyond what is captured in text. While blind or low-vision (BLV) scientists have traditionally relied on static alternative text to access figures in papers, the rise of artificial intelligence (AI) has made interactive question-answering (QA) a feasible paradigm for visual exploration; yet little is known about how scientists use visual QA in practice or how to improve its accessibility. In this work, we interview five BLV and five sighted scientists across different STEM fields to understand how they use two AI tools, ChatGPT and Gemini, to query multimodal scientific documents. Our findings characterize how scientists review multimodal content, including existing practices (along with accessibility workarounds) for engaging with visuals, and feedback on the suitability of AI-generated responses to multimodal queries. We further find that vague or incomplete image descriptions, as well as incorrect AI outputs more broadly, can cause both BLV and sighted scientists to abandon AI workflows. To support future research, we additionally contribute a dataset of 115 queries and responses from our participants’ interactions with the AI tools for papers in their field. We close by discussing implications for AI-powered scientific QA systems, emphasizing considerations for access across abilities and domains.
```
@inproceedings{chheda-kothary-assets26-multimodal,
  author = {Chheda-Kothary, Arnavi and Wang, Lucy Lu and Chang, Joseph Chee and Bragg, Jonathan},
  title = {Querying Multimodal Scientific Papers with {AI}: Practices and Preferences Across Blind, Low-Vision, and Sighted Scientists},
  booktitle = {Proceedings of the 28th International {ACM} {SIGACCESS} Conference on Computers and Accessibility},
  series = {{ASSETS}},
  year = {2026},
  month = oct,
  eprint = {2607.18514},
  archivePrefix = {arXiv}
}
```
Scientific Reasoning Does Not Reliably Translate into Scientific Forecasting in Frontier AI⁠

AIAI for scienceEvaluationPreprint

Sean Wu, Pan Lu, Yupeng Chen, Jonathan Bragg, Junchi Yu

arXiv, May 2026

PDF HTML Code Data

AI systems are increasingly used to support forward-looking scientific judgment, but it remains unclear whether they can form reliable expectations about future scientific advances. Here we show that strong scientific reasoning does not reliably translate into accurate forecasting of future scientific advances. To study this question, we introduce CUSP, a temporally grounded evaluation suite for event-level scientific forecasting across eight scientific disciplines. Across six frontier AI models, we observe a striking asymmetry in forecasting performance together with systematic error patterns. Models often identify plausible mechanisms underlying future scientific advances, yet perform near chance on feasibility assessment, generate solution strategies that only weakly align with realized advances, and systematically predict scientific advances later than they become publicly observable. Providing additional pre-cutoff scientific knowledge improves performance but does not eliminate these forecasting limitations. These findings suggest that current AI systems possess substantial retrospective scientific competence but limited forward-looking predictive capability. Scientific forecasting should therefore be evaluated as a complementary dimension of AI scientific capability when deploying AI systems for research prioritization and scientific decision-making.
```
@article{wu-2026-forecasting,
  author = {Wu, Sean and Lu, Pan and Chen, Yupeng and Bragg, Jonathan and Yamada, Yutaro and Clark, Peter and Clifton, David A. and Torr, Philip and Zou, James and Yu, Junchi},
  title = {Scientific Reasoning Does Not Reliably Translate into Scientific Forecasting in Frontier {AI}},
  journal = {CoRR},
  volume = {abs/2605.22681},
  year = {2026},
  url = {https://doi.org/10.48550/arXiv.2605.22681},
  doi = {10.48550/ARXIV.2605.22681},
  eprint = {2605.22681},
  month = may,
  archivePrefix = {arXiv}
}
```
Cocoa: Co-Planning and Co-Execution with AI Agents⁠

HCIAIAI for scienceConference paper

K. J. Kevin Feng, Kevin Pu, Matt Latzke, Tal August, Pao Siangliulue, Jonathan Bragg, Daniel S. Weld, Amy X. Zhang, Joseph Chee Chang

CHI 2026 · Best Paper

PDF HTML

As AI agents take on increasingly long-running tasks involving sophisticated planning and execution, there is a corresponding need for novel interaction designs that enable deeper human-agent collaboration. However, most prior works leverage human interaction to fix “autonomous” workflows that have yet to become fully autonomous or rigidly treat planning and execution as separate stages. Based on a formative study with 9 researchers using AI to support their work, we propose a design that affords greater flexibility in collaboration, so that users can 1) delegate agency to the user or agent via a collaborative plan where individual steps can be assigned; and 2) interleave planning and execution so that plans can adjust after partial execution. We introduce Cocoa, a system that takes design inspiration from computational notebooks to support complex research tasks. A lab study (n = 16) found that Cocoa enabled steerability without sacrificing ease-of-use, and a week-long field deployment (n = 7) showed how researchers collaborated with Cocoa to accomplish real-world tasks.
```
@inproceedings{feng-chi26-cocoa,
  author = {Feng, K. J. Kevin and Pu, Kevin and Latzke, Matt and August, Tal and Siangliulue, Pao and Bragg, Jonathan and Weld, Daniel S. and Zhang, Amy X. and Chang, Joseph Chee},
  title = {Cocoa: Co-Planning and Co-Execution with {AI} Agents},
  booktitle = {Proceedings of the 2026 {CHI} Conference on Human Factors in Computing Systems},
  series = {{CHI}},
  pages = {16:1--16:23},
  year = {2026},
  month = apr,
  day = {13},
  url = {https://doi.org/10.1145/3772318.3791673},
  doi = {10.1145/3772318.3791673},
  eprint = {2412.10999},
  archivePrefix = {arXiv}
}
```
Omakase: Proactive Assistance with Actionable Suggestions for Evolving Scientific Research Projects⁠

HCIAIAI for sciencePreprint

Pao Siangliulue, Jonathan Bragg, Doug Downey, Joseph Chee Chang, Daniel S. Weld

arXiv, Apr 2026

PDF HTML

As AI agents become increasingly capable of complex knowledge tasks, the lack of context limits their capability to proactively reason about a user’s latent needs throughout a long evolving project. In scientific research, many researchers still manually query a deep research system and compress their rich project contexts into short, targeted queries. Further, a deep research system produces exhaustive reports, making it difficult to identify concrete actions. To explore the opportunities of research assistants that are proactive throughout a research project, we conducted several studies (N=42) with a technology probe and an iterative prototype. The latest iteration of our system, Omakase, is a research assistant that monitors a user’s project documents to infer timely queries to a deep research system. Omakase then distills long reports into suggestions contextualized to their evolving projects. Our evaluations showed that participants found the generated queries to be useful and timely, and rated Omakase’s suggestions as significantly more actionable than the original reports.
```
@article{siangliulue-2026-omakase,
  author = {Siangliulue, Pao and Bragg, Jonathan and Downey, Doug and Chang, Joseph Chee and Weld, Daniel S.},
  title = {Omakase: Proactive Assistance with Actionable Suggestions for Evolving Scientific Research Projects},
  journal = {CoRR},
  volume = {abs/2604.08898},
  year = {2026},
  url = {https://doi.org/10.48550/arXiv.2604.08898},
  doi = {10.48550/ARXIV.2604.08898},
  eprint = {2604.08898},
  month = apr,
  archivePrefix = {arXiv}
}
```
Understanding Usage and Engagement in AI-Powered Scientific Research Tools: The Asta Interaction Dataset⁠

HCIAIAI for sciencePreprint

Dany Haddad, Dan Bareket, Joseph Chee Chang, Jay DeYoung, Jonathan Bragg, Doug Downey

arXiv, Feb 2026

PDF HTML Data Blog

AI-powered scientific research tools are rapidly being integrated into research workflows, yet the field lacks a clear lens into how researchers use these systems in real-world settings. We present and analyze the Asta Interaction Dataset, a large-scale resource comprising over 200,000 user queries and interaction logs from two deployed tools (a literature discovery interface and a scientific question-answering interface) within an LLM-powered retrieval-augmented generation platform. Using this dataset, we characterize query patterns, engagement behaviors, and how usage evolves with experience. We find that users submit longer and more complex queries than in traditional search, and treat the system as a collaborative research partner, delegating tasks such as drafting content and identifying research gaps. Users treat generated responses as persistent artifacts, revisiting and navigating among outputs and cited evidence in non-linear ways. With experience, users issue more targeted queries and engage more deeply with supporting citations, although keyword-style queries persist even among experienced users. We release the anonymized dataset and analysis with a new query intent taxonomy to inform future designs of real-world AI research assistants and to support realistic evaluation.
```
@article{haddad-2026-asta-interaction,
  author = {Haddad, Dany and Bareket, Dan and Chang, Joseph Chee and DeYoung, Jay and Hwang, Jena D. and Katz, Uri and Polak, Mark and Suh, Sangho and Surana, Harshit and Tiktinsky, Aryeh and Atmakuri, Shriya and Bragg, Jonathan and D'Arcy, Mike and Feldman, Sergey and Hassan-Ali, Amal and Lozano, Rub{\'e}n and Majumder, Bodhisattwa Prasad and McGrady, Charles and Singh, Amanpreet and Vlahos, Brooke and Goldberg, Yoav and Downey, Doug},
  title = {Understanding Usage and Engagement in {AI}-Powered Scientific Research Tools: The Asta Interaction Dataset},
  journal = {CoRR},
  volume = {abs/2602.23335},
  year = {2026},
  url = {https://doi.org/10.48550/arXiv.2602.23335},
  doi = {10.48550/ARXIV.2602.23335},
  eprint = {2602.23335},
  month = feb,
  archivePrefix = {arXiv}
}
```

2025

SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks⁠

AIAI for scienceEvaluationConference paper

Yilun Zhao^*, Kaiyan Zhang^*, Tiansheng Hu^*, Sihong Wu, Ronan Le Bras, Taira Anderson, Jonathan Bragg, Arman Cohan ^*equal contribution

NeurIPS 2025 · Spotlight ·

Spotlight — top 3.2% of submissions

683 given, of 21,575 submissions. NeurIPS 2025 accepted 5,290 overall (24.5%).

acceptance figures · honor count

PDF HTML (arXiv) Code Data Blog

We present SciArena, an open and collaborative platform for evaluating foundation models on scientific literature-grounded tasks. Unlike traditional benchmarks for scientific literature understanding and synthesis, SciArena engages the research community directly, following the Chatbot Arena evaluation approach of community voting on model comparisons. By leveraging collective intelligence, SciArena offers a community-driven evaluation of model performance on open-ended scientific tasks that demand literature-grounded, long-form responses. The platform currently supports 47 foundation models and has collected over 20,000 votes from human researchers across diverse scientific domains. Our analysis of the data collected so far confirms its high quality. We discuss the results and insights based on the model ranking leaderboard. To further promote research in building model-based automated evaluation systems for literature tasks, we release SciArena-Eval, a meta-evaluation benchmark based on collected preference data. It measures the accuracy of models in judging answer quality by comparing their pairwise assessments with human votes. Our experiments highlight the benchmark’s challenges and emphasize the need for more reliable automated evaluation methods.
```
@inproceedings{zhao-neurips25-sciarena,
  author = {Zhao, Yilun and Zhang, Kaiyan and Hu, Tiansheng and Wu, Sihong and Bras, Ronan Le and Anderson, Taira and Bragg, Jonathan and Chang, Joseph Chee and Dodge, Jesse and Latzke, Matt and Liu, Yixin and McGrady, Charles and Tang, Xiangru and Wang, Zihang and Zhao, Chen and Hajishirzi, Hannaneh and Downey, Doug and Cohan, Arman},
  title = {SciArena: An Open Evaluation Platform for Non-Verifiable Scientific
                    Literature-Grounded Tasks},
  booktitle = {39th Conference on Neural Information Processing Systems ({NeurIPS} 2025) Track on Datasets and Benchmarks},
  series = {{NeurIPS}},
  year = {2025},
  month = dec,
  url = {https://openreview.net/forum?id=am6RR85mnc},
  eprint = {2507.01001},
  archivePrefix = {arXiv}
}
```
PaperPing: A Socially-aware AI Agent that Recommends Academic Papers to Research Group Chats with Contextualized Explanations⁠

HCIAIAI for scienceDemo

Ruotong Wang, Xinyi Zhou, Lin Qiu, Joseph Chee Chang, Jonathan Bragg, Amy X. Zhang

CSCW Companion 2025

PDF HTML

AI agents are entering group spaces to facilitate or contribute to conversations. However, without an understanding of group context, these contributions can be irrelevant or even intrusive. In this demo, we introduce a socially-aware AI agent that proactively contributes to research group chats by posting context-aware messages that contain paper recommendations and contextualized explanations. To achieve this, PaperPing instantiates Social-RAG [14]. It retrieves relevant interactions in the group context, builds a social knowledge base, and extracts social signals to contextualize the generated messages. PaperPing has been deployed in 18 channels as part of a user study for at least 3 months, reaching 500+ researchers. As part of the demo, we will demonstrate PaperPing in a Slack workspace open to all CSCW attendees during and after the conference, allowing researchers to explore the interactions in real settings with the side benefits of fostering collaborative learning within the community.
```
@inproceedings{wang-cscw25-paperping,
  author = {Wang, Ruotong and Zhou, Xinyi and Qiu, Lin and Chang, Joseph Chee and Bragg, Jonathan and Zhang, Amy X.},
  title = {PaperPing: {A} Socially-aware {AI} Agent that Recommends Academic Papers to Research Group Chats with Contextualized Explanations},
  booktitle = {Companion Publication of the 2025 Conference on Computer-Supported Cooperative Work and Social Computing},
  series = {{CSCW} Companion},
  pages = {532--535},
  year = {2025},
  month = oct,
  url = {https://doi.org/10.1145/3715070.3757230},
  doi = {10.1145/3715070.3757230}
}
```
Social-RAG: Retrieving from Group Interactions to Socially Ground AI Generation⁠

HCIAIAI for scienceConference paper

Ruotong Wang, Xinyi Zhou, Lin Qiu, Joseph Chee Chang, Jonathan Bragg, Amy X. Zhang

CHI 2025 ·

Accepted — 24.9% of submissions

1,249 of 5,020 submissions to CHI 2025.

acceptance figures

PDF HTML

AI agents are increasingly tasked with making proactive suggestions in online spaces where groups collaborate, yet risk being unhelpful or even annoying if they fail to match group preferences or behave in socially inappropriate ways. Fortunately, group spaces have a rich history of prior interactions and affordances for social feedback that can support grounding an agent’s generations to a group’s interests and norms. We present Social-RAG, a workflow for socially grounding agents that retrieves context from prior group interactions, selects relevant social signals, and feeds them into a language model to generate messages in a socially aligned manner. We implement this in PaperPing, a system for posting paper recommendations in group chat, leveraging social signals determined from formative studies with 39 researchers. From a three-month deployment in 18 channels reaching 500+ researchers, we observed PaperPing posted relevant messages in groups without disrupting their existing social practices, fostering group common ground.
```
@inproceedings{wang-chi25-social-rag,
  author = {Wang, Ruotong and Zhou, Xinyi and Qiu, Lin and Chang, Joseph Chee and Bragg, Jonathan and Zhang, Amy X.},
  title = {Social-RAG: Retrieving from Group Interactions to Socially Ground
                    {AI} Generation},
  booktitle = {Proceedings of the 2025 {CHI} Conference on Human Factors in Computing
                    Systems, {CHI} 2025, YokohamaJapan, 26 April 2025- 1 May 2025},
  series = {{CHI}},
  pages = {162:1--162:25},
  year = {2025},
  url = {https://doi.org/10.1145/3706598.3713749},
  doi = {10.1145/3706598.3713749},
  month = apr,
  eprint = {2411.02353},
  archivePrefix = {arXiv}
}
```
Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference⁠

AIEvaluationConference paper

Mingqi Gao, Yixin Liu, Xinyu Hu, Xiaojun Wan, Jonathan Bragg, Arman Cohan

NAACL 2025 ·

Accepted — 15.0% of submissions

477 of 3,185 submissions to NAACL 2025.

acceptance figures

PDF HTML (arXiv)

Evaluating and ranking the capabilities of different LLMs is crucial for understanding their performance and alignment with human preferences. Due to the high cost and timeconsuming nature of human evaluations, an automatic LLM bencher (i.e., an automatic evaluation framework that aims to rank LLMs based on their alignment with human preferences) is indispensable. An automatic LLM bencher consists of four components: the input set (e.g., a user instruction), the evaluation model (e.g., an LLM), the evaluation type (e.g., pairwise comparison), and the aggregation method (e.g., the ELO rating system).However, previous work has not thoroughly explored how to select these components or how their different combinations influence the results. In this work, through controlled experiments, we provide a series of recommendations on how to choose each component to better automate the evaluation of LLMs. Furthermore, we discovered that when evaluating LLMs with similar performance, the performance of the automatic LLM bencher declines sharply, underscoring the limitations of current benchers and calling for future work. Lastly, we found that the evaluation models’ performance at the instance level (e.g., the accuracy of selecting the best output) does not always align with their effectiveness when used as a component of a bencher, highlighting the importance of dedicated system-level evaluation of benchers.
```
@inproceedings{gao-naacl25-llm-ranking,
  author = {Gao, Mingqi and Liu, Yixin and Hu, Xinyu and Wan, Xiaojun and Bragg, Jonathan and Cohan, Arman},
  title = {Re-evaluating Automatic {LLM} System Ranking for Alignment with Human
                    Preference},
  booktitle = {Findings of the Association for Computational Linguistics: {NAACL}
                    2025, Albuquerque, New Mexico, USA, April 29 - May 4, 2025},
  series = {NAACL},
  pages = {4605--4629},
  doi = {10.18653/v1/2025.findings-naacl.260},
  year = {2025},
  url = {https://aclanthology.org/2025.findings-naacl.260/},
  month = apr,
  eprint = {2501.00560},
  archivePrefix = {arXiv}
}
```
IdeaSynth: Iterative Research Idea Development Through Evolving and Composing Idea Facets with Literature-Grounded Feedback⁠

HCIAIAI for scienceConference paper

Kevin Pu, K. J. Kevin Feng, Tovi Grossman, Tom Hope, Bhavana Dalvi Mishra, Matt Latzke, Jonathan Bragg, Joseph Chee Chang, Pao Siangliulue

CHI 2025 ·

Accepted — 24.9% of submissions

1,249 of 5,020 submissions to CHI 2025.

acceptance figures

PDF HTML

Research ideation involves broad exploring and deep refining ideas. Both require deep engagement with literature. Existing tools focus primarily on broad idea generation, yet offer little support for iterative specification, refinement, and evaluation needed to further develop initial ideas. To bridge this gap, we introduce IdeaSynth, a research idea development system that uses LLMs to provide literature-grounded feedback for articulating research problems, solutions, evaluations, and contributions. IdeaSynth represents these idea facets as nodes on a canvas, and allow researchers to iteratively refine them by creating and exploring variations and combinations. Our lab study (N = 20) showed that participants, while using IdeaSynth, explored more alternative ideas and expanded initial ideas with more details compared to a strong LLM-based baseline. Our deployment study (N = 7) demonstrated that participants effectively used IdeaSynth for real-world research projects at various ideation stages from developing initial ideas to revising framings of mature manuscripts, highlighting the possibilities to adopt IdeaSynth in researcher’s workflows.
```
@inproceedings{pu-chi25-ideasynth,
  author = {Pu, Kevin and Feng, K. J. Kevin and Grossman, Tovi and Hope, Tom and Mishra, Bhavana Dalvi and Latzke, Matt and Bragg, Jonathan and Chang, Joseph Chee and Siangliulue, Pao},
  title = {IdeaSynth: Iterative Research Idea Development Through Evolving and
                    Composing Idea Facets with Literature-Grounded Feedback},
  booktitle = {Proceedings of the 2025 {CHI} Conference on Human Factors in Computing
                    Systems, {CHI} 2025, YokohamaJapan, 26 April 2025- 1 May 2025},
  series = {{CHI}},
  pages = {145:1--145:31},
  year = {2025},
  url = {https://doi.org/10.1145/3706598.3714057},
  doi = {10.1145/3706598.3714057},
  month = apr,
  eprint = {2410.04025},
  archivePrefix = {arXiv}
}
```
Facets, Taxonomies, and Syntheses: Navigating Structured Representations in LLM-Assisted Literature Review⁠

HCIAIAI for sciencePreprint

Raymond Fok, Joseph Chee Chang, Marissa Radensky, Pao Siangliulue, Jonathan Bragg, Amy X. Zhang, Daniel S. Weld

arXiv, Apr 2025

PDF HTML

Comprehensive literature review requires synthesizing vast amounts of research — a labor intensive and cognitively demanding process. Most prior work focuses either on helping researchers deeply understand a few papers (e.g., for triaging or reading), or retrieving from and visualizing a vast corpus. Deep analysis and synthesis of large paper collections (e.g., to produce a survey paper) is largely conducted manually with little support. We present DimInd, an interactive system that scaffolds literature review across large paper collections through LLM-generated structured representations. DimInd scaffolds literature understanding with multiple levels of compression, from papers, to faceted literature comparison tables with information extracted from individual papers, to taxonomies of concepts, to narrative syntheses. Users are guided through these successive information transformations while maintaining provenance to source text. In an evaluation with 23 researchers, DimInd supported participants in extracting information and conceptually organizing papers with less effort compared to a ChatGPT-assisted baseline workflow.
```
@article{fok-2025-facets,
  author = {Fok, Raymond and Chang, Joseph Chee and Radensky, Marissa and Siangliulue, Pao and Bragg, Jonathan and Zhang, Amy X. and Weld, Daniel S.},
  title = {Facets, Taxonomies, and Syntheses: Navigating Structured Representations
                    in {LLM}-Assisted Literature Review},
  journal = {CoRR},
  volume = {abs/2504.18496},
  year = {2025},
  url = {https://doi.org/10.48550/arXiv.2504.18496},
  doi = {10.48550/ARXIV.2504.18496},
  eprint = {2504.18496},
  month = apr,
  archivePrefix = {arXiv}
}
```

2024

Accelerating Scientific Paper Skimming with Augmented Intelligence Through Customizable Faceted Highlights⁠

HCIAIAI for scienceJournal article

Raymond Fok, Luca Soldaini, Jonathan Bragg, Kyle Lo, Marti A. Hearst, Andrew Head, Daniel S. Weld

TIIS 14(4) 2024

PDF HTML

Scholars need to keep up with an exponentially increasing flood of scientific papers. To aid this challenge, we introduce Scim, a novel intelligent interface that helps scholars skim papers to rapidly review and gain a cursory understanding of its contents. Scim supports the skimming process by highlighting salient content within a paper, directing a scholar’s attention. These automatically-extracted highlights are faceted by content type, evenly-distributed across a paper, and have a density configurable by scholars. We evaluate Scim with an in-lab usability study and a longitudinal diary study, revealing how its highlights facilitate the more efficient construction of a conceptualization of a paper. Finally, we describe the process of scaling highlights from their conception within Scim, a research prototype, to production on over 521,000 papers within the Semantic Reader, a publicly-available augmented reading interface for scientific papers. We conclude by discussing design considerations and tensions for the design of future skimming tools with augmented intelligence.
```
@article{fok-tiis24-scim,
  author = {Fok, Raymond and Soldaini, Luca and Trier, Cassidy and Bransom, Erin and MacMillan, Kelsey and Cheng, Evie and Kambhamettu, Hita and Bragg, Jonathan and Lo, Kyle and Hearst, Marti A. and Head, Andrew and Weld, Daniel S.},
  title = {Accelerating Scientific Paper Skimming with Augmented Intelligence Through Customizable Faceted Highlights},
  series = {{TIIS}},
  year = {2024},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  issn = {2160-6455},
  url = {https://doi.org/10.1145/3665648},
  volume = {14},
  number = {4},
  pages = {1--30},
  doi = {10.1145/3665648},
  journal = {{ACM} Trans. Interact. Intell. Syst.},
  month = dec
}
```
The Semantic Reader Project⁠

HCIAccessibilityAIAI for scienceMagazine article

Kyle Lo, Joseph Chee Chang, Andrew Head, Jonathan Bragg, Daniel S. Weld

CACM 67(10) 2024

PDF HTML

Scholarly publications are key to the transfer of knowledge from scholars to others. However, research papers are information-dense, and as the volume of the scientific literature grows, the greater the need for new technology to support scholars. In contrast to the process of finding papers, which has been transformed by Internet technology, the experience of reading research papers has changed little in decades. For instance, the PDF format for sharing papers remains widely used due to its portability but has significant downsides, inter alia, static content and poor accessibility for low-vision readers. This paper explores the question “Can recent advances in AI and HCI power intelligent, interactive, and accessible reading interfaces — even for legacy PDFs?” We describe the Semantic Reader Project, a collaborative effort across multiple institutions to explore automatic creation of dynamic reading interfaces for research papers. Through this project, we’ve developed a collection of novel reading interfaces and evaluated them with study participants and real-world users to show improved reading experiences for scholars. We’ve also released a production research paper reading interface that will continuously incorporate novel features from our research as they mature. We structure this paper around five key opportunities for AI assistance in scholarly reading — discovery, efficiency, comprehension, synthesis, and accessibility — and present an overview of our progress and discuss remaining open challenges.
```
@article{lo-cacm24-semantic-reader,
  author = {Lo, Kyle and Chang, Joseph Chee and Head, Andrew and Bragg, Jonathan and Zhang, Amy X. and Trier, Cassidy and Anastasiades, Chloe and August, Tal and Authur, Russell and Bragg, Danielle and Bransom, Erin and Cachola, Isabel and Candra, Stefan and Chandrasekhar, Yoganand and Chen, Yen{-}Sung and Cheng, Evie Yu{-}Yen and Chou, Yvonne and Downey, Doug and Evans, Rob and Fok, Raymond and Hu, Fangzhou and Huff, Regan and Kang, Dongyeop and Kim, Tae Soo and Kinney, Rodney and Kittur, Aniket and Kang, Hyeonsu B. and Klevak, Egor and Kuehl, Bailey and Langan, Michael and Latzke, Matt and Lochner, Jaron and MacMillan, Kelsey and Marsh, Eric and Murray, Tyler and Naik, Aakanksha and Nguyen, Ngoc{-}Uyen and Palani, Srishti and Park, Soya and Paulic, Caroline and Rachatasumrit, Napol and Rao, Smita and Sayre, Paul and Shen, Zejiang and Siangliulue, Pao and Soldaini, Luca and Tran, Huy and van Zuylen, Madeleine and Wang, Lucy Lu and Wilhelm, Christopher and Wu, Caroline and Yang, Jiangjiang and Zamarron, Angele and Hearst, Marti A. and Weld, Daniel S.},
  title = {The Semantic Reader Project},
  journal = {Commun. {ACM}},
  volume = {67},
  number = {10},
  pages = {50--61},
  series = {{CACM}},
  year = {2024},
  url = {https://doi.org/10.1145/3659096},
  doi = {10.1145/3659096},
  month = sep,
  eprint = {2303.14334},
  archivePrefix = {arXiv}
}
```
ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews⁠

AIAI for scienceConference paper

Mike D’Arcy, Alexis Ross, Erin Bransom, Bailey Kuehl, Jonathan Bragg, Tom Hope, Doug Downey

ACL 2024 ·

Accepted — 22.2% of submissions

976 of 4,407 submissions to ACL 2024.

acceptance figures

PDF HTML (arXiv)

We introduce the task of automatically revising scientific papers based on peer feedback and release ARIES, a dataset of review comments and their corresponding paper edits. The data is drawn from real reviewer-author interactions from computer science, and we provide labels linking each reviewer comment to the specific paper edits made by the author in response. We automatically create a high-precision silver training set, as well as an expert-labeled test set that shows high inter-annotator agreement. In experiments with 10 models covering the state of the art, we find that they struggle even to identify which edits correspond to a comment — especially when the relationship between the edit and the comment is indirect and requires reasoning to uncover. We also extensively analyze GPT-4’s ability to generate edits given a comment and the original paper. We find that it often succeeds on a superficial level, but tends to rigidly follow the wording of the feedback rather than the underlying intent, and lacks technical details compared to human-written edits.
```
@inproceedings{darcy-acl24-aries,
  author = {D'Arcy, Mike and Ross, Alexis and Bransom, Erin and Kuehl, Bailey and Bragg, Jonathan and Hope, Tom and Downey, Doug},
  title = {{ARIES:} {A} Corpus of Scientific Paper Edits Made in Response to
                    Peer Reviews},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  series = {{ACL}},
  url = {https://doi.org/10.18653/v1/2024.acl-long.377},
  doi = {10.18653/v1/2024.acl-long.377},
  year = {2024},
  month = aug,
  pages = {6985--7001},
  eprint = {2306.12587},
  archivePrefix = {arXiv}
}
```
Papers-to-Posts: Supporting Detailed Long-Document Summarization with an Interactive LLM-Powered Source Outline⁠

HCIAIAI for sciencePreprint

Marissa Radensky, Daniel S. Weld, Joseph Chee Chang, Pao Siangliulue, Jonathan Bragg

arXiv, Jun 2024

PDF HTML

Compressing long and technical documents (e.g., >10 pages) into shorter-form articles (e.g., <2 pages) is critical for communicating information to different audiences, for example, blog posts of scientific research paper or legal briefs of dense court proceedings. While large language models (LLMs) are powerful tools for condensing large amounts of text, current interfaces to these models lack support for understanding and controlling what content is included in a detailed summarizing article. Such capability is especially important for detail- and technical-oriented domains, in which tactical selection and coherent synthesis of key details is critical for effective communication to the target audience. For this, we present interactive reverse source outlines, a novel mechanism for controllable long-form summarization featuring outline bullet points with automatic point selections that the user can iteratively adjust to obtain an article with the desired content coverage. We implement this mechanism in Papers-to-Posts, a new LLM-powered system for authoring research-paper blog posts. Through a within-subjects lab study (n=20) and a between-subjects deployment study (n=37 blog posts, 26 participants), we compare Papers-to-Posts to a strong baseline tool that provides an LLM-generated draft and access to free-form prompting. Under time constraints, Papers-to-Posts significantly increases writer satisfaction with blog post quality, particularly with respect to content coverage. Furthermore, quantitative results showed an increase in editing power (change in text for an amount of time or writing actions) while using Papers-to-Posts, and qualitative results showed that participants found incorporating key research-paper insights in their blog posts easier while using Papers-to-Posts.
```
@article{radensky-2024-blog,
  author = {Radensky, Marissa and Weld, Daniel S. and Chang, Joseph Chee and Siangliulue, Pao and Bragg, Jonathan},
  title = {Papers-to-Posts: Supporting Detailed Long-Document Summarization with
                    an Interactive {LLM}-Powered Source Outline},
  journal = {CoRR},
  volume = {abs/2406.10370},
  year = {2024},
  url = {https://doi.org/10.48550/arXiv.2406.10370},
  doi = {10.48550/ARXIV.2406.10370},
  eprint = {2406.10370},
  month = jun,
  archivePrefix = {arXiv}
}
```
PaperWeaver: Enriching Topical Paper Alerts by Contextualizing Recommended Papers with User-collected Papers⁠

HCIAIAI for scienceConference paper

Yoonjoo Lee, Hyeonsu B. Kang, Matt Latzke, Juho Kim, Jonathan Bragg, Joseph Chee Chang, Pao Siangliulue

CHI 2024 ·

Accepted — 26.2% of submissions

1,060 of 4,046 submissions to CHI 2024.

acceptance figures

PDF HTML

With the rapid growth of scholarly archives, researchers subscribe to “paper alert” systems that periodically provide them with recommendations of recently published papers that are similar to previously collected papers. However, researchers sometimes struggle to make sense of nuanced connections between recommended papers and their own research context, as existing systems only present paper titles and abstracts. To help researchers spot these connections, we present PaperWeaver, an enriched paper alerts system that provides contextualized text descriptions of recommended papers based on user-collected papers. PaperWeaver employs a computational method based on Large Language Models (LLMs) to infer users’ research interests from their collected papers, extract context-specific aspects of papers, and compare recommended and collected papers on these aspects. Our user study (N=15) showed that participants using PaperWeaver were able to better understand the relevance of recommended papers and triage them more confidently when compared to a baseline that presented the related work sections from recommended papers.
```
@inproceedings{lee-chi24-paperweaver,
  author = {Lee, Yoonjoo and Kang, Hyeonsu B. and Latzke, Matt and Kim, Juho and Bragg, Jonathan and Chang, Joseph Chee and Siangliulue, Pao},
  title = {PaperWeaver: Enriching Topical Paper Alerts by Contextualizing Recommended
                    Papers with User-collected Papers},
  booktitle = {Proceedings of the {CHI} Conference on Human Factors in Computing Systems},
  series = {{CHI}},
  doi = {10.1145/3613904.3642196},
  pages = {19:1--19:19},
  year = {2024},
  month = may,
  url = {https://doi.org/10.1145/3613904.3642196},
  eprint = {2403.02939},
  archivePrefix = {arXiv}
}
```
Mitigating Barriers to Public Social Interaction with Meronymous Communication⁠

HCIAI for scienceConference paper

Nouran Soliman, Hyeonsu B. Kang, Matthew Latzke, Jonathan Bragg, Joseph Chee Chang, Amy X. Zhang, David R. Karger

CHI 2024 · Best Paper ·

Best Paper — top 1.0% of submissions

40 given, of 4,046 submissions. CHI 2024 accepted 1,060 overall (26.2%). The share is computed from the counts below, since the conference doesn't publish it.

acceptance figures · honor count

PDF HTML

In communities with social hierarchies, fear of judgment can discourage communication. While anonymity may alleviate some social pressure, fully anonymous spaces enable toxic behavior and hide the social context that motivates people to participate and helps them tailor their communication. We explore a design space of meronymous communication, where people can reveal carefully chosen aspects of their identity and also leverage trusted endorsers to gain credibility. We implemented these ideas in a system for scholars to meronymously seek and receive paper recommendations on Twitter and Mastodon. A formative study with 20 scholars confirmed that scholars see benefits to participating but are deterred due to social anxiety. From a month-long public deployment, we found that with meronymity, junior scholars could comfortably ask “newbie” questions and get responses from senior scholars who they normally found intimidating. Responses were also tailored to the aspects about themselves that junior scholars chose to reveal.
```
@inproceedings{soliman-chi24-meronymous,
  author = {Soliman, Nouran and Kang, Hyeonsu B. and Latzke, Matthew and Bragg, Jonathan and Chang, Joseph Chee and Zhang, Amy X. and Karger, David R.},
  title = {Mitigating Barriers to Public Social Interaction with Meronymous Communication},
  booktitle = {Proceedings of the {CHI} Conference on Human Factors in Computing Systems},
  series = {{CHI}},
  doi = {10.1145/3613904.3642241},
  pages = {151:1--151:26},
  year = {2024},
  month = may,
  url = {https://doi.org/10.1145/3613904.3642241},
  eprint = {2402.17847},
  archivePrefix = {arXiv}
}
```
FigurA11y: AI Assistance for Writing Scientific Alt Text⁠

HCIAccessibilityAIAI for scienceConference paper

Nikhil Singh, Lucy Lu Wang, Jonathan Bragg

IUI 2024

PDF HTML Code

High-quality alt text is crucial for making scientific figures accessible to blind and low-vision readers. Crafting complete, accurate alt text is challenging even for domain experts, as published figures often depict complex visual information and readers have varied informational needs. These challenges, along with high diversity in figure types and domain-specific details, also limit the usefulness of fully automated approaches. Consequently, the prevalence of high-quality alt text is very low in scientific papers today. We investigate whether and how human-AI collaborative editing systems can help address the difficulty of writing high-quality alt text for complex scientific figures. We present FigurA11y, an interactive system that generates draft alt text and provides suggestions for author revisions using a pipeline driven by extracted figure and paper metadata. We test two versions, motivated by prior work on visual accessibility and writing support. The base Draft+Revise version provides authors with an automatically generated draft description to revise, along with extracted figure metadata and figure-specific alt text guidelines to support the revision process. The full Interactive Assistance version further adds contextualized suggestions: text snippets to iteratively produce descriptions, and hypothetical user questions with possible answers to reveal potential ambiguities and resolutions. In a study of authors (N=14), we found the system assisted them in efficiently producing descriptive alt text. Generated drafts and interface elements enabled authors to quickly initiate and edit detailed descriptions. Additionally, interactive suggestions from the full system prompted more iteration and highlighted aspects for authors to consider, resulting in greater deviation from the drafts without increased average cognitive load or manual effort.
```
@inproceedings{singh-iui24-figura11y,
  author = {Singh, Nikhil and Wang, Lucy Lu and Bragg, Jonathan},
  title = {Figur{A}11y: {AI} Assistance for Writing Scientific Alt Text},
  booktitle = {Proceedings of the 29th International Conference on Intelligent
                    User Interfaces, {IUI} 2024, Greenville, SC, USA, March 18-21,
                    2024},
  series = {{IUI}},
  year = {2024},
  month = mar,
  pages = {886--906},
  url = {https://doi.org/10.1145/3640543.3645212},
  doi = {10.1145/3640543.3645212}
}
```

2023

RCT Rejection Sampling for Causal Estimation Evaluation⁠

AIEvaluationJournal article

Katherine A. Keith, Sergey Feldman, David Jurgens, Jonathan Bragg, Rohit Bhattacharya

TMLR 2023

PDF HTML (arXiv)

Confounding is a significant obstacle to unbiased estimation of causal effects from observational data. For settings with high-dimensional covariates — such as text data, genomics, or the behavioral social sciences — researchers have proposed methods to adjust for confounding by adapting machine learning methods to the goal of causal estimation. However, empirical evaluation of these adjustment methods has been challenging and limited. In this work, we build on a promising empirical evaluation strategy that simplifies evaluation design and uses real data: subsampling randomized controlled trials (RCTs) to create confounded observational datasets while using the average causal effects from the RCTs as ground-truth. We contribute a new sampling algorithm, which we call RCT rejection sampling, and provide theoretical guarantees that causal identification holds in the observational data to allow for valid comparisons to the ground-truth RCT. Using synthetic data, we show our algorithm indeed results in low bias when oracle estimators are evaluated on the confounded samples, which is not always the case for a previously proposed algorithm. In addition to this identification result, we highlight several finite data considerations for evaluation designers who plan to use RCT rejection sampling on their own datasets. As a proof of concept, we implement an example evaluation pipeline and walk through these finite data considerations with a novel, real-world RCT — which we release publicly — consisting of approximately 70k observations and text data as high-dimensional covariates. Together, these contributions build towards a broader agenda of improved empirical evaluation for causal estimation.
```
@article{keith-tmlr23-rct,
  title = {{RCT} Rejection Sampling for Causal Estimation Evaluation},
  author = {Keith, Katherine A. and Feldman, Sergey and Jurgens, David and Bragg, Jonathan and Bhattacharya, Rohit},
  journal = {Transactions on Machine Learning Research},
  issn = {2835-8856},
  year = {2023},
  month = nov,
  url = {https://openreview.net/forum?id=F74ZZk5hPa},
  series = {{TMLR}},
  eprint = {2307.15176},
  archivePrefix = {arXiv}
}
```
Papeos: Augmenting Research Papers with Talk Videos⁠

HCIAIAI for scienceConference paper

Tae Soo Kim, Matt Latzke, Jonathan Bragg, Amy X. Zhang, Joseph Chee Chang

UIST 2023 ·

Accepted — 25.1% of submissions

121 of 483 submissions to UIST 2023.

acceptance figures

PDF HTML

Research consumption has been traditionally limited to the reading of academic papers — a static, dense, and formally written format. Alternatively, pre-recorded conference presentation videos, which are more dynamic, concise, and colloquial, have recently become more widely available but potentially under-utilized. In this work, we explore the design space and benefits for combining academic papers and talk videos to leverage their complementary nature to provide a rich and fluid research consumption experience. Based on formative and co-design studies, we present Papeos, a novel reading and authoring interface that allow authors to augment their papers by segmenting and localizing talk videos alongside relevant paper passages with automatically generated suggestions. With Papeos, readers can visually skim a paper through clip thumbnails, and fluidly switch between consuming dense text in the paper or visual summaries in the video. In a comparative lab study (n=16), Papeos reduced mental load, scaffolded navigation, and facilitated more comprehensive reading of papers.
```
@inproceedings{kim-uist23-papeos,
  author = {Kim, Tae Soo and Latzke, Matt and Bragg, Jonathan and Zhang, Amy X. and Chang, Joseph Chee},
  title = {Papeos: Augmenting Research Papers with Talk Videos},
  booktitle = {Proceedings of the 36th Annual {ACM} Symposium on User Interface Software
                    and Technology, {UIST} 2023, San Francisco, CA, USA, 29 October 2023-
                    1 November 2023},
  series = {{UIST}},
  pages = {15:1--15:19},
  year = {2023},
  month = oct,
  url = {https://doi.org/10.1145/3586183.3606770},
  doi = {10.1145/3586183.3606770},
  eprint = {2308.15224},
  archivePrefix = {arXiv}
}
```
Paper Plain: Making Medical Research Papers Approachable to Healthcare Consumers with Natural Language Processing⁠

HCIAIAI for scienceJournal article

Tal August, Lucy Lu Wang, Jonathan Bragg, Marti A. Hearst, Andrew Head, Kyle Lo

TOCHI 30(5) 2023

PDF HTML

When seeking information not covered in patient-friendly documents, healthcare consumers may turn to the research literature. Reading medical papers, however, can be a challenging experience. To improve access to medical papers, we explore four features enabled by natural language processing: definitions of unfamiliar terms, in-situ plain language section summaries, a collection of key questions that guides readers to answering passages, and plain language summaries of those passages. We embody these features into a prototype system, Paper Plain. We evaluate Paper Plain, finding that participants who used the prototype system had an easier time reading research papers without a loss in paper comprehension compared to those who used a typical PDF reader. Altogether, the study results suggest that guiding readers to relevant passages and providing plain language summaries alongside the original paper content can make reading medical papers easier and give readers more confidence to approach these papers.
```
@article{august-tochi23-paperplain,
  author = {August, Tal and Wang, Lucy Lu and Bragg, Jonathan and Hearst, Marti A. and Head, Andrew and Lo, Kyle},
  title = {Paper {P}lain: Making Medical Research Papers Approachable to Healthcare
                    Consumers with Natural Language Processing},
  journal = {{ACM} Trans. Comput. Hum. Interact.},
  series = {{TOCHI}},
  volume = {30},
  number = {5},
  pages = {74:1--74:38},
  year = {2023},
  month = sep,
  url = {https://doi.org/10.1145/3589955},
  doi = {10.1145/3589955},
  eprint = {2203.00130},
  archivePrefix = {arXiv}
}
```
Relatedly: Scaffolding Literature Reviews with Existing Related Work Sections⁠

HCIAIAI for scienceConference paper

Srishti Palani, Aakanksha Naik, Doug Downey, Amy X. Zhang, Jonathan Bragg, Joseph Chee Chang

CHI 2023 ·

Accepted — 27.6% of submissions

879 of 3,182 submissions to CHI 2023.

acceptance figures

PDF HTML

Scholars who want to research a scientific topic must take time to read, extract meaning, and identify connections across many papers. As scientific literature grows, this becomes increasingly challenging. Meanwhile, authors summarize prior research in papers’ related work sections, though this is scoped to support a single paper. A formative study found that while reading multiple related work paragraphs helps overview a topic, it is hard to navigate overlapping and diverging references and research foci. In this work, we design a system, Relatedly, that scaffolds exploring and reading multiple related work paragraphs on a topic, with features including dynamic re-ranking and highlighting to spotlight unexplored dissimilar information, auto-generated descriptive paragraph headings, and low-lighting of redundant information. From a within-subjects user study (n=15), we found that scholars generate more coherent, insightful, and comprehensive topic outlines using Relatedly compared to a baseline paper list.
```
@inproceedings{palani-chi23-relatedly,
  author = {Palani, Srishti and Naik, Aakanksha and Downey, Doug and Zhang, Amy X. and Bragg, Jonathan and Chang, Joseph Chee},
  title = {Relatedly: Scaffolding Literature Reviews with Existing Related Work
                    Sections},
  booktitle = {Proceedings of the 2023 {CHI} Conference on Human Factors in Computing
                    Systems, {CHI} 2023, Hamburg, Germany, April 23-28, 2023},
  series = {{CHI}},
  pages = {742:1--742:20},
  year = {2023},
  month = apr,
  url = {https://doi.org/10.1145/3544548.3580841},
  doi = {10.1145/3544548.3580841},
  eprint = {2302.06754},
  archivePrefix = {arXiv}
}
```
ComLittee: Literature Discovery with Personal Elected Author Committees⁠

HCIAIAI for scienceConference paper

Hyeonsu B. Kang, Nouran Soliman, Matt Latzke, Joseph Chee Chang, Jonathan Bragg

CHI 2023 ·

Accepted — 27.6% of submissions

879 of 3,182 submissions to CHI 2023.

acceptance figures

PDF HTML

In order to help scholars understand and follow a research topic, significant research has been devoted to creating systems that help scholars discover relevant papers and authors. Recent approaches have shown the usefulness of highlighting relevant authors while scholars engage in paper discovery. However, these systems do not capture and utilize users’ evolving knowledge of authors. We reflect on the design space and introduce ComLittee, a literature discovery system that supports author-centric exploration. In contrast to paper-centric interaction in prior systems, ComLittee’s author-centric interaction supports curating research threads from individual authors, finding new authors and papers using combined signals from a paper recommender and the curated authors’ authorship graphs, and understanding them in the context of those signals. In a within-subjects experiment that compares to a paper-centric discovery system with author-highlighting, we demonstrate how ComLittee improves author and paper discovery.
```
@inproceedings{kang-chi23-comlittee,
  author = {Kang, Hyeonsu B. and Soliman, Nouran and Latzke, Matt and Chang, Joseph Chee and Bragg, Jonathan},
  title = {ComLittee: Literature Discovery with Personal Elected Author Committees},
  booktitle = {Proceedings of the 2023 {CHI} Conference on Human Factors in Computing
                    Systems, {CHI} 2023, Hamburg, Germany, April 23-28, 2023},
  series = {{CHI}},
  pages = {738:1--738:20},
  year = {2023},
  month = apr,
  url = {https://doi.org/10.1145/3544548.3581371},
  doi = {10.1145/3544548.3581371},
  eprint = {2302.06780},
  archivePrefix = {arXiv}
}
```
CiteSee: Augmenting Citations in Scientific Papers with Persistent and Personalized Historical Context⁠

HCIAI for scienceConference paper

Joseph Chee Chang, Amy X. Zhang, Jonathan Bragg, Andrew Head, Kyle Lo, Doug Downey, Daniel S. Weld

CHI 2023 · Best Paper ·

Best Paper — top 1.1% of submissions

35 given, of 3,182 submissions. CHI 2023 accepted 879 overall (27.6%). The share is computed from the counts below, since the conference doesn't publish it.

acceptance figures · honor count

PDF HTML

When reading a scholarly article, inline citations help researchers contextualize the current article and discover relevant prior work. However, it can be challenging to prioritize and make sense of the hundreds of citations encountered during literature reviews. This paper introduces CiteSee, a paper reading tool that leverages a user’s publishing, reading, and saving activities to provide personalized visual augmentations and context around citations. First, CiteSee connects the current paper to familiar contexts by surfacing known citations a user had cited or opened. Second, CiteSee helps users prioritize their exploration by highlighting relevant but unknown citations based on saving and reading history. We conducted a lab study that suggests CiteSee is significantly more effective for paper discovery than three baselines. A field deployment study shows CiteSee helps participants keep track of their explorations and leads to better situational awareness and increased paper discovery via inline citation when conducting real-world literature reviews.
```
@inproceedings{chang-chi23-citesee,
  author = {Chang, Joseph Chee and Zhang, Amy X. and Bragg, Jonathan and Head, Andrew and Lo, Kyle and Downey, Doug and Weld, Daniel S.},
  title = {CiteSee: Augmenting Citations in Scientific Papers with Persistent
                    and Personalized Historical Context},
  booktitle = {Proceedings of the 2023 {CHI} Conference on Human Factors in Computing
                    Systems, {CHI} 2023, Hamburg, Germany, April 23-28, 2023},
  series = {{CHI}},
  pages = {737:1--737:15},
  year = {2023},
  month = apr,
  url = {https://doi.org/10.1145/3544548.3580847},
  doi = {10.1145/3544548.3580847},
  eprint = {2302.07302},
  archivePrefix = {arXiv}
}
```
Beyond Summarization: Designing AI Support for Real-World Expository Writing Tasks⁠

HCIAIAI for scienceWorkshop paper

Zejiang Shen, Tal August, Pao Siangliulue, Kyle Lo, Jonathan Bragg, Jeff Hammerbacher, Doug Downey, Joseph Chee Chang, David A. Sontag

In2Writing @ CHI 2023

PDF (arXiv) HTML (arXiv)

Large language models have introduced exciting new opportunities and challenges in designing and developing new AI-assisted writing support tools. Recent work has shown that leveraging this new technology can transform writing in many scenarios such as ideation during creative writing, editing support, and summarization. However, AI-supported expository writing — including real-world tasks like scholars writing literature reviews or doctors writing progress notes — is relatively understudied. In this position paper, we argue that developing AI supports for expository writing has unique and exciting research challenges and can lead to high real-world impacts. We characterize expository writing as evidence-based and knowledge-generating: it contains summaries of external documents as well as new information or knowledge. It can be seen as the product of authors’ sensemaking process over a set of source documents, and the interplay between reading, reflection, and writing opens up new opportunities for designing AI support. We sketch three components for AI support design and discuss considerations for future research.
```
@inproceedings{shen-in2writing23-expository-writing,
  author = {Shen, Zejiang and August, Tal and Siangliulue, Pao and Lo, Kyle and Bragg, Jonathan and Hammerbacher, Jeff and Downey, Doug and Chang, Joseph Chee and Sontag, David A.},
  title = {Beyond Summarization: Designing {AI} Support for Real-World Expository
                    Writing Tasks},
  booktitle = {Proceedings of the 2nd Workshop on Intelligent and Interactive Writing Assistants},
  series = {{In2Writing} @ {CHI}},
  year = {2023},
  month = apr,
  url = {https://doi.org/10.48550/arXiv.2304.02623},
  eprint = {2304.02623},
  archivePrefix = {arXiv}
}
```
Scim: Intelligent Skimming Support for Scientific Papers⁠

HCIAIAI for scienceConference paper

Raymond Fok, Hita Kambhamettu, Luca Soldaini, Jonathan Bragg, Kyle Lo, Marti A. Hearst, Andrew Head, Daniel S. Weld

IUI 2023

PDF HTML

Scholars need to keep up with an exponentially increasing flood of scientific papers. To aid this challenge, we introduce Scim, a novel intelligent interface that helps experienced researchers skim — or rapidly review — a paper to attain a cursory understanding of its contents. Scim supports the skimming process by highlighting salient paper contents in order to direct a reader’s attention. The system’s highlights are faceted by content type, evenly distributed across a paper, and have a density configurable by readers at both the global and local level. We evaluate Scim with both an in-lab usability study and a longitudinal diary study, revealing how its highlights facilitate the more efficient construction of a conceptualization of a paper. We conclude by discussing design considerations and tensions for the design of future intelligent skimming tools.
```
@inproceedings{fok-iui23-scim,
  author = {Fok, Raymond and Kambhamettu, Hita and Soldaini, Luca and Bragg, Jonathan and Lo, Kyle and Hearst, Marti A. and Head, Andrew and Weld, Daniel S.},
  title = {Scim: Intelligent Skimming Support for Scientific Papers},
  booktitle = {Proceedings of the 28th International Conference on Intelligent User
                    Interfaces, {IUI} 2023, Sydney, NSW, Australia, March 27-31, 2023},
  series = {{IUI}},
  pages = {476--490},
  year = {2023},
  month = mar,
  url = {https://doi.org/10.1145/3581641.3584034},
  doi = {10.1145/3581641.3584034},
  eprint = {2205.04561},
  archivePrefix = {arXiv}
}
```
The Semantic Scholar Open Data Platform⁠

AIAI for sciencePreprint

Rodney Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Daniel S. Weld

arXiv, Jan 2023

PDF HTML

The volume of scientific output is creating an urgent need for automated tools to help scientists keep up with developments in their field. Semantic Scholar (S2) is an open data platform and website aimed at accelerating science by helping scholars discover and understand scientific literature. We combine public and proprietary data sources using state-of-the-art techniques for scholarly PDF content extraction and automatic knowledge graph construction to build the Semantic Scholar Academic Graph, the largest open scientific literature graph to-date, with 200M+ papers, 80M+ authors, 550M+ paper-authorship edges, and 2.4B+ citation edges. The graph includes advanced semantic features such as structurally parsed text, natural language summaries, and vector embeddings. In this paper, we describe the components of the S2 data processing pipeline and the associated APIs offered by the platform. We will update this living document to reflect changes as we add new data offerings and improve existing services.
```
@article{kinney-2023-s2-api,
  author = {Kinney, Rodney and Anastasiades, Chloe and Authur, Russell and Beltagy, Iz and Bragg, Jonathan and Buraczynski, Alexandra and Cachola, Isabel and Candra, Stefan and Chandrasekhar, Yoganand and Cohan, Arman and Crawford, Miles and Downey, Doug and Dunkelberger, Jason and Etzioni, Oren and Evans, Rob and Feldman, Sergey and Gorney, Joseph and Graham, David and Hu, Fangzhou and Huff, Regan and King, Daniel and Kohlmeier, Sebastian and Kuehl, Bailey and Langan, Michael and Lin, Daniel and Liu, Haokun and Lo, Kyle and Lochner, Jaron and MacMillan, Kelsey and Murray, Tyler and Newell, Chris and Rao, Smita and Rohatgi, Shaurya and Sayre, Paul and Shen, Zejiang and Singh, Amanpreet and Soldaini, Luca and Subramanian, Shivashankar and Tanaka, Amber and Wade, Alex D. and Wagner, Linda and Wang, Lucy Lu and Wilhelm, Chris and Wu, Caroline and Yang, Jiangjiang and Zamarron, Angele and van Zuylen, Madeleine and Weld, Daniel S.},
  title = {The {S}emantic {S}cholar Open Data Platform},
  journal = {CoRR},
  volume = {abs/2301.10140},
  year = {2023},
  month = jan,
  url = {https://doi.org/10.48550/arXiv.2301.10140},
  doi = {10.48550/ARXIV.2301.10140},
  eprint = {2301.10140},
  archivePrefix = {arXiv}
}
```

2022

GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation⁠

AIEvaluationCollective intelligenceConference paper

Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, Daniel S. Weld

EMNLP 2022 ·

Accepted — 13.1% of submissions

549 of 4,190 submissions to EMNLP 2022.

acceptance figures

PDF HTML (arXiv) Code Application (archived)

While often assumed a gold standard, effective human evaluation of text generation remains an important, open area for research. We revisit this problem with a focus on producing consistent evaluations that are reproducible — over time and across different populations. We study this goal in different stages of the human evaluation pipeline. In particular, we consider design choices for the annotation interface used to elicit human judgments and their impact on reproducibility. Furthermore, we develop an automated mechanism for maintaining annotator quality via a probabilistic model that detects and excludes noisy annotators. Putting these lessons together, we introduce GENIE: a system for running standardized human evaluations across different generation tasks. We instantiate GENIE with datasets representing four core challenges in text generation: machine translation, summarization, commonsense reasoning, and machine comprehension. For each task, GENIE offers a leaderboard that automatically crowdsources annotations for submissions, evaluating them along axes such as correctness, conciseness, and fluency. We have made the GENIE leaderboards publicly available, and have already ranked 50 submissions from 10 different research groups. We hope GENIE encourages further progress toward effective, standardized evaluations for text generation.
```
@inproceedings{khashabi-emnlp22-genie,
  author = {Khashabi, Daniel and Stanovsky, Gabriel and Bragg, Jonathan and Lourie, Nicholas and Kasai, Jungo and Choi, Yejin and Smith, Noah A. and Weld, Daniel S.},
  title = {{GENIE:} Toward Reproducible and Standardized Human Evaluation for
                    Text Generation},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural
                    Language Processing, {EMNLP} 2022, Abu Dhabi, United Arab Emirates,
                    December 7-11, 2022},
  series = {{EMNLP}},
  pages = {11444--11458},
  year = {2022},
  month = dec,
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.787},
  doi = {10.18653/V1/2022.EMNLP-MAIN.787},
  eprint = {2101.06561},
  archivePrefix = {arXiv}
}
```
Exploring Team-Sourced Hyperlinks to Address Navigation Challenges for Low-Vision Readers of Scientific Papers⁠

HCIAccessibilityAI for scienceConference paper

Soya Park, Jonathan Bragg, Michael Chang, Kevin Larson, Danielle Bragg

CSCW 2022

PDF

Reading academic papers is a fundamental part of higher education and research, but navigating these information-dense texts can be challenging. In particular, low-vision readers using magnification encounter additional barriers to quickly skimming and visually locating information. In this work, we explored the design of interfaces to enable readers to: 1) navigate papers more easily, and 2) input the required navigation hooks that AI cannot currently automate. To explore this design space, we ran two exploratory studies. The first focused on current practices of low-vision paper readers, the challenges they encounter, and the interfaces they desire. During this study, low-vision participants were interviewed, and tried out four new paper navigation prototypes. Results from this study grounded the design of our end-to-end system prototype Ocean, which provides an accessible front-end for low-vision readers, and enables all readers to contribute to the backend by leaving traces of their reading paths for others to leverage. Our second study used this exploratory interface in a field study with groups of low-vision and sighted readers to probe the user experience of reading and creating traces. Our findings suggest that it may be possible for readers of all abilities to organically leave traces in papers, and that these traces can be used to facilitate navigation tasks, in particular for low-vision readers. Based on our findings, we present design considerations for creating future paper-reading tools that improve access, and organically source the required data from readers.
```
@article{park-cscw22-ocean,
  author = {Park, Soya and Bragg, Jonathan and Chang, Michael and Larson, Kevin and Bragg, Danielle},
  title = {Exploring Team-Sourced Hyperlinks to Address Navigation Challenges
                    for Low-Vision Readers of Scientific Papers},
  journal = {Proc. {ACM} Hum. Comput. Interact.},
  series = {{CSCW}},
  volume = {6},
  number = {{CSCW2}},
  pages = {1--23},
  year = {2022},
  month = nov,
  url = {https://doi.org/10.1145/3555629},
  doi = {10.1145/3555629}
}
```
Paper to HTML: A Publicly Available Web Tool for Converting Scientific PDFs into Accessible HTML⁠

HCIAccessibilityAIAI for scienceNewsletter article

Lucy Lu Wang, Jonathan Bragg, Daniel S. Weld

SIGACCESS Access. Comput. 2022

HTML Application (archived)

Most scientific papers are distributed in PDF format, which is by default inaccessible to blind and low vision audiences and people who use assistive reading technology. These access barriers hinder and may even deter members of these groups from pursuing careers or opportunities that necessitate the reading of technical documents. In cases where no accessible versions of papers are made available by publishers or authors, the gold standard for PDF document accessibility is PDF remediation. Remediation is the process by which a PDF is made accessible by fixing accessibility errors, for example, tagging headings, specifying reading order, adding alt text to images, and so on, such that a reader can navigate and engage with the resulting content using assistive reading technology such as screen readers.
```
@article{wang-sigaccess22-paper-to-html,
  author = {Wang, Lucy Lu and Bragg, Jonathan and Weld, Daniel S.},
  title = {Paper to {HTML}: {A} Publicly Available Web Tool for Converting Scientific
                    {PDF}s into Accessible {HTML}},
  journal = {{SIGACCESS} Access. Comput.},
  number = {134},
  pages = {1--1},
  series = {{SIGACCESS} Access. Comput.},
  year = {2022},
  month = oct,
  url = {https://doi.org/10.1145/3582298.3582299},
  doi = {10.1145/3582298.3582299}
}
```
FeedLens: Polymorphic Lenses for Personalizing Exploratory Search over Knowledge Graphs⁠

HCIAIAI for scienceConference paper

Harmanpreet Kaur, Doug Downey, Amanpreet Singh, Evie Yu-Yen Cheng, Daniel S. Weld, Jonathan Bragg

UIST 2022 ·

Accepted — 26.3% of submissions

98 of 372 submissions to UIST 2022.

acceptance figures

PDF HTML

The vast scale and open-ended nature of knowledge graphs (KGs) make exploratory search over them cognitively demanding for users. We introduce a new technique, polymorphic lenses, that improves exploratory search over a KG by obtaining new leverage from the existing preference models that KG-based systems maintain for recommending content. The approach is based on a simple but powerful observation: in a KG, preference models can be re-targeted to recommend not only entities of a single base entity type (e.g., papers in the scientific literature KG, products in an e-commerce KG), but also all other types (e.g., authors, conferences, institutions; sellers, buyers). We implement our technique in a novel system, FeedLens, which is built over Semantic Scholar, a production system for navigating the scientific literature KG. FeedLens reuses the existing preference models on Semantic Scholar — people’s curated research feeds — as lenses for exploratory search. Semantic Scholar users can curate multiple feeds/lenses for different topics of interest, e.g., one for human-centered AI and another for document embeddings. Although these lenses are defined in terms of papers, FeedLens re-purposes them to also guide search over authors, institutions, venues, etc. Our system design is based on feedback from intended users via two pilot surveys (n = 17 and n = 13, respectively). We compare FeedLens and Semantic Scholar via a third (within-subjects) user study (n = 15) and find that FeedLens increases user engagement while reducing the cognitive effort required to complete a short literature review task. Our qualitative results also highlight people’s preference for this more effective exploratory search experience enabled by FeedLens.
```
@inproceedings{kaur-uist22-feedlens,
  author = {Kaur, Harmanpreet and Downey, Doug and Singh, Amanpreet and Cheng, Evie Yu{-}Yen and Weld, Daniel S. and Bragg, Jonathan},
  title = {FeedLens: Polymorphic Lenses for Personalizing Exploratory Search
                    over Knowledge Graphs},
  booktitle = {The 35th Annual {ACM} Symposium on User Interface Software and Technology,
                    {UIST} 2022, Bend, OR, USA, 29 October 2022 - 2 November 2022},
  series = {{UIST}},
  pages = {97:1--97:15},
  year = {2022},
  month = oct,
  url = {https://doi.org/10.1145/3526113.3545631},
  doi = {10.1145/3526113.3545631},
  eprint = {2208.07531},
  archivePrefix = {arXiv}
}
```
A Dataset of Alt Texts from HCI Publications: Analyses and Uses Towards Producing More Descriptive Alt Texts of Data Visualizations in Scientific Papers⁠

HCIAccessibilityAIAI for scienceConference paper

Sanjana Shivani Chintalapati, Jonathan Bragg, Lucy Lu Wang

ASSETS 2022 ·

Accepted — 26.5% of submissions

35 of 132 submissions to ASSETS 2022.

acceptance figures

PDF HTML Data

Figures in scientific publications contain important information and results, and alt text is needed for blind and low vision readers to engage with their content. We conduct a study to characterize the semantic content of alt text in HCI publications based on a framework introduced by Lundgard and Satyanarayan [30]. Our study focuses on alt text for graphs, charts, and plots extracted from HCI and accessibility publications; we focus on these communities due to the lack of alt text in papers published outside of these disciplines. We find that the capacity of author-written alt text to fulfill blind and low vision user needs is mixed; for example, only 50% of alt texts in our sample contain information about extrema or outliers, and only 31% contain information about major trends or comparisons conveyed by the graph. We release our collected dataset of author-written alt text, and outline possible ways that it can be used to develop tools and models to assist future authors in writing better alt text. Based on our findings, we also discuss recommendations that can be acted upon by publishers and authors to encourage inclusion of more types of semantic content in alt text.
```
@inproceedings{chintalapati-assets22-alt-text-quality,
  author = {Chintalapati, Sanjana Shivani and Bragg, Jonathan and Wang, Lucy Lu},
  title = {A Dataset of Alt Texts from {HCI} Publications: Analyses and Uses
                    Towards Producing More Descriptive Alt Texts of Data Visualizations
                    in Scientific Papers},
  booktitle = {Proceedings of the 24th International {ACM} {SIGACCESS} Conference
                    on Computers and Accessibility, {ASSETS} 2022, Athens, Greece, October
                    23-26, 2022},
  series = {{ASSETS}},
  pages = {30:1--30:12},
  year = {2022},
  month = oct,
  url = {https://doi.org/10.1145/3517428.3544796},
  doi = {10.1145/3517428.3544796},
  eprint = {2209.13718},
  archivePrefix = {arXiv}
}
```
From Who You Know to What You Read: Augmenting Scientific Recommendations with Implicit Social Networks⁠

HCIAIAI for scienceConference paper

Hyeonsu B. Kang, Rafal Kocielnik, Andrew Head, Jiangjiang Yang, Matt Latzke, Aniket Kittur, Daniel S. Weld, Doug Downey, Jonathan Bragg

CHI 2022 ·

Accepted — 24.7% of submissions

637 of 2,579 submissions to CHI 2022.

acceptance figures

PDF HTML

The ever-increasing pace of scientific publication necessitates methods for quickly identifying relevant papers. While neural recommenders trained on user interests can help, they still result in long, monotonous lists of suggested papers. To improve the discovery experience we introduce multiple new methods for augmenting recommendations with textual relevance messages that highlight knowledge-graph connections between recommended papers and a user’s publication and interaction history. We explore associations mediated by author entities and those using citations alone. In a large-scale, real-world study, we show how our approach significantly increases engagement — and future engagement when mediated by authors — without introducing bias towards highly-cited authors. To expand message coverage for users with less publication or interaction history, we develop a novel method that highlights connections with proxy authors of interest to users and evaluate it in a controlled lab study. Finally, we synthesize design implications for future graph-based messages.
```
@inproceedings{kang-chi22-implicit-social-networks,
  author = {Kang, Hyeonsu B. and Kocielnik, Rafal and Head, Andrew and Yang, Jiangjiang and Latzke, Matt and Kittur, Aniket and Weld, Daniel S. and Downey, Doug and Bragg, Jonathan},
  title = {From Who You Know to What You Read: Augmenting Scientific Recommendations
                    with Implicit Social Networks},
  booktitle = {{CHI} '22: {CHI} Conference on Human Factors in Computing Systems,
                    New Orleans, LA, USA, 29 April 2022 - 5 May 2022},
  series = {{CHI}},
  pages = {302:1--302:23},
  year = {2022},
  month = apr,
  url = {https://doi.org/10.1145/3491102.3517470},
  doi = {10.1145/3491102.3517470},
  eprint = {2204.10254},
  archivePrefix = {arXiv}
}
```
CiteRead: Integrating Localized Citation Contexts into Scientific Paper Reading⁠

HCIAIAI for scienceConference paper

Napol Rachatasumrit, Jonathan Bragg, Amy X. Zhang, Daniel S. Weld

IUI 2022

PDF HTML

When reading a scholarly paper, scientists oftentimes wish to understand how follow-on work has built on or engages with what they are reading. While a paper itself can only discuss prior work, some scientific search engines can provide a list of all subsequent citing papers; unfortunately, they are undifferentiated and disconnected from the contents of the original reference paper. In this work, we introduce a novel paper reading experience that integrates relevant information about follow-on work directly into a paper, allowing readers to learn about newer papers and see how a paper is discussed by its citing papers in the context of the reference paper. We built a tool, called CiteRead, that implements the following three contributions: 1) automated techniques for selecting important citing papers, building on results from a formative study we conducted, 2) an automated process for localizing commentary provided by citing papers to a place in the reference paper, and 3) an interactive experience that allows readers to seamlessly alternate between the reference paper and information from citing papers (e.g., citation sentences), placed in the margins. Based on a user study with 12 scientists, we found that in comparison to having just a list of citing papers and their citation sentences, the use of CiteRead while reading allows for better comprehension and retention of information about follow-on work.
```
@inproceedings{rachatasumrit-iui22-citeread,
  author = {Rachatasumrit, Napol and Bragg, Jonathan and Zhang, Amy X. and Weld, Daniel S.},
  title = {CiteRead: Integrating Localized Citation Contexts into Scientific
                    Paper Reading},
  booktitle = {{IUI} 2022: 27th International Conference on Intelligent User Interfaces,
                    Helsinki, Finland, March 22 - 25, 2022},
  series = {{IUI}},
  pages = {707--719},
  year = {2022},
  month = mar,
  url = {https://doi.org/10.1145/3490099.3511162},
  doi = {10.1145/3490099.3511162}
}
```

2021

FLEX: Unifying Evaluation for Few-Shot NLP⁠

AIEvaluationConference paper

Jonathan Bragg^*, Arman Cohan^*, Kyle Lo, Iz Beltagy ^*equal contribution

NeurIPS 2021 ·

Accepted — 25.6% of submissions

2,334 of 9,122 submissions to NeurIPS 2021.

acceptance figures

PDF HTML (arXiv) Code Covered byruder.io

Few-shot NLP research is highly active, yet conducted in disjoint research threads with evaluation suites that lack challenging-yet-realistic testing setups and fail to employ careful experimental design. Consequently, the community does not know which techniques perform best or even if they outperform simple baselines. In response, we formulate the FLEX Principles, a set of requirements and best practices for unified, rigorous, valid, and cost-sensitive few-shot NLP evaluation. These principles include Sample Size Design, a novel approach to benchmark design that optimizes statistical accuracy and precision while keeping evaluation costs manageable. Following the principles, we release the FLEX benchmark, which includes four few-shot transfer settings, zero-shot evaluation, and a public leaderboard that covers diverse NLP tasks. In addition, we present UniFew, a prompt-based model for few-shot learning that unifies pretraining and finetuning prompt formats, eschewing complex machinery of recent prompt-based approaches in adapting downstream task formats to language model pretraining objectives. We demonstrate that despite simplicity, UniFew achieves results competitive with both popular meta-learning and prompt-based approaches.
```
@inproceedings{bragg-neurips21-flex,
  author = {Bragg, Jonathan and Cohan, Arman and Lo, Kyle and Beltagy, Iz},
  title = {{FLEX:} Unifying Evaluation for Few-Shot {NLP}},
  booktitle = {Advances in Neural Information Processing Systems 34: Annual Conference
                    on Neural Information Processing Systems 2021, NeurIPS 2021, December
                    6-14, 2021, virtual},
  series = {{NeurIPS}},
  pages = {15787--15800},
  year = {2021},
  month = dec,
  url = {https://proceedings.neurips.cc/paper/2021/hash/8493eeaccb772c0878f99d60a0bd2bb3-Abstract.html},
  eprint = {2107.07170},
  archivePrefix = {arXiv}
}
```
SciA11y: Converting Scientific Papers to Accessible HTML⁠

HCIAccessibilityAIAI for scienceDemo

Lucy Lu Wang^*, Isabel Cachola^*, Jonathan Bragg, Evie Yu-Yen Cheng, Daniel S. Weld ^*equal contribution

ASSETS Posters and Demos 2021 · Best Artifact ·

Best Artifact

Not a share of submissions: delegate voting selected a top five and a panel of judges chose among them.

how it was decided

PDF HTML Application (archived) Covered byNature

We present SciA11y, a system that renders inaccessible scientific paper PDFs into HTML. SciA11y uses machine learning models to extract and understand the content of scientific PDFs, and reorganizes the resulting paper components into a form that better supports skimming and scanning for blind and low vision (BLV) readers. SciA11y adds navigation features such as tagged headings, a table of contents, and bidirectional links between inline citations and references, which allow readers to resolve citations without losing their context. A set of 1.5 million open access papers are processed and available at https://scia11y.org/. This system is a first step in addressing scientific PDF accessibility, and may significantly improve the experience of paper reading for BLV users.
```
@inproceedings{wang-assets21-scia11y,
  author = {Wang, Lucy Lu and Cachola, Isabel and Bragg, Jonathan and Cheng, Evie Yu{-}Yen and Haupt, Chelsea and Latzke, Matt and Kuehl, Bailey and van Zuylen, Madeleine N. and Wagner, Linda and Weld, Daniel S.},
  title = {SciA11y: Converting Scientific Papers to Accessible {HTML}},
  booktitle = {{ASSETS} '21: The 23rd International {ACM} {SIGACCESS} Conference
                    on Computers and Accessibility, Virtual Event, USA, October 18-22,
                    2021},
  series = {{ASSETS} Posters and Demos},
  pages = {85:1--85:4},
  year = {2021},
  month = oct,
  url = {https://doi.org/10.1145/3441852.3476545},
  doi = {10.1145/3441852.3476545}
}
```
Improving the Accessibility of Scientific Documents: Current State, User Needs, and a System Solution to Enhance Scientific PDF Accessibility for Blind and Low Vision Users⁠

HCIAccessibilityAIAI for sciencePreprint

Lucy Lu Wang^*, Isabel Cachola^*, Jonathan Bragg, Evie Yu-Yen Cheng, Daniel S. Weld ^*equal contribution

arXiv, Apr 2021

PDF HTML Application (archived) Covered byNature

The majority of scientific papers are distributed in PDF, which pose challenges for accessibility, especially for blind and low vision (BLV) readers. We characterize the scope of this problem by assessing the accessibility of 11,397 PDFs published 2010–2019 sampled across various fields of study, finding that only 2.4% of these PDFs satisfy all of our defined accessibility criteria. We introduce the SciA11y system to offset some of the issues around inaccessibility. SciA11y incorporates several machine learning models to extract the content of scientific PDFs and render this content as accessible HTML, with added novel navigational features to support screen reader users. An intrinsic evaluation of extraction quality indicates that the majority of HTML renders (87%) produced by our system have no or only some readability issues. We perform a qualitative user study to understand the needs of BLV researchers when reading papers, and to assess whether the SciA11y system could address these needs. We summarize our user study findings into a set of five design recommendations for accessible scientific reader systems. User response to SciA11y was positive, with all users saying they would be likely to use the system in the future, and some stating that the system, if available, would become their primary workflow. We successfully produce HTML renders for over 12M papers, of which an open access subset of 1.5M are available for browsing at https://scia11y.org/
```
@article{wang-2021-scia11y,
  author = {Wang, Lucy Lu and Cachola, Isabel and Bragg, Jonathan and Cheng, Evie Yu{-}Yen and Haupt, Chelsea and Latzke, Matt and Kuehl, Bailey and van Zuylen, Madeleine and Wagner, Linda and Weld, Daniel S.},
  title = {Improving the Accessibility of Scientific Documents: Current State,
                    User Needs, and a System Solution to Enhance Scientific {PDF} Accessibility
                    for Blind and Low Vision Users},
  journal = {CoRR},
  volume = {abs/2105.00076},
  year = {2021},
  month = apr,
  url = {https://doi.org/10.48550/arXiv.2105.00076},
  eprint = {2105.00076},
  archivePrefix = {arXiv}
}
```

2019

Fake It Till You Make It: Learning-Compatible Performance Support⁠

AIConference paper

Jonathan Bragg, Emma Brunskill

UAI 2019 ·

Accepted — 26.2% of submissions

118 of 450 submissions to UAI 2019.

acceptance figures

PDF Code

A longstanding goal of artificial intelligence is to develop technologies that augment or assist humans. Current approaches to developing agents that can assist humans focus on adapting behavior of the assistant, and do not consider the potential for assistants to support human learning. We argue that in many cases it is worthwhile to provide assistance in a manner that also promotes task learning or skill maintenance. We term such assistance Learning-Compatible Performance Support, and present the Stochastic Q Bumpers algorithm for greatly improving learning outcomes while still providing high levels of performance support. We demonstrate the effectiveness of our approach in multiple domains, including a complex flight control task.
```
@inproceedings{bragg-uai19-performance-support,
  author = {Bragg, Jonathan and Brunskill, Emma},
  title = {Fake It Till You Make It: Learning-Compatible Performance Support},
  booktitle = {Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial
                    Intelligence, {UAI} 2019, Tel Aviv, Israel, July 22-25, 2019},
  series = {{UAI}},
  pages = {915--924},
  year = {2019},
  month = jul,
  url = {https://proceedings.mlr.press/v115/bragg20a.html}
}
```
Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing⁠

HCICollective intelligenceConference paper

Quanze Chen, Jonathan Bragg, Lydia B. Chilton, Daniel S. Weld

CHI 2019 ·

Accepted — 23.8% of submissions

703 of 2,958 submissions to CHI 2019.

acceptance figures

PDF HTML

Traditional approaches for ensuring high quality crowdwork have failed to achieve high-accuracy on difficult problems. Aggregating redundant answers often fails on the hardest problems when the majority is confused. Argumentation has been shown to be effective in mitigating these drawbacks. However, existing argumentation systems only support limited interactions and show workers general justifications, not context-specific arguments targeted to their reasoning. This paper presents Cicero, a new workflow that improves crowd accuracy on difficult tasks by engaging workers in multi-turn, contextual discussions through real-time, synchronous argumentation. Our experiments show that compared to previous argumentation systems which only improve the average individual worker accuracy by 6.8 percentage points on the Relation Extraction domain, our workflow achieves 16.7 percentage point improvement. Furthermore, previous argumentation approaches don’t apply to tasks with many possible answers; in contrast, Cicero works well in these cases, raising accuracy from 66.7% to 98.8% on the Codenames domain.
```
@inproceedings{chen-chi19-cicero,
  author = {Chen, Quanze and Bragg, Jonathan and Chilton, Lydia B. and Weld, Daniel S.},
  title = {Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing},
  booktitle = {Proceedings of the 2019 {CHI} Conference on Human Factors in Computing
                    Systems, {CHI} 2019, Glasgow, Scotland, UK, May 04-09, 2019},
  series = {{CHI}},
  pages = {531:1--531:14},
  year = {2019},
  month = may,
  url = {https://doi.org/10.1145/3290605.3300761},
  doi = {10.1145/3290605.3300761},
  eprint = {1810.10733},
  archivePrefix = {arXiv}
}
```

2018

Automatic Curriculum Generation Applied to Teaching Novices a Short Bach Piano Segment⁠

AIDemo

Emma Brunskill, Tong Mu, Karan Goel, Jonathan Bragg

NeurIPS Demo 2018

@inproceedings{brunskill-neurips18-bach,
  title = {Automatic {{Curriculum Generation Applied}} to {{Teaching Novices}} a {{Short Bach Piano Segment}}},
  booktitle = {{{NeurIPS}} '18 {{Demonstration Track}}},
  author = {Brunskill, Emma and Mu, Tong and Goel, Karan and Bragg, Jonathan},
  series = {{NeurIPS} Demo},
  year = {2018},
  month = dec
}

Sprout: Crowd-Powered Task Design for Crowdsourcing⁠

HCICollective intelligenceConference paper

Jonathan Bragg, Mausam, Daniel S. Weld

UIST 2018 ·

Accepted — 21.3% of submissions

80 of 375 submissions to UIST 2018.

acceptance figures

PDF Slides Video

While crowdsourcing enables data collection at scale, ensuring high-quality data remains a challenge. In particular, effective task design underlies nearly every reported crowdsourcing success, yet remains difficult to accomplish. Task design is hard because it involves a costly iterative process: identifying the kind of work output one wants, conveying this information to workers, observing worker performance, understanding what remains ambiguous, revising the instructions, and repeating the process until the resulting output is satisfactory. To facilitate this process, we propose a novel meta-workflow that helps requesters optimize crowdsourcing task designs and Sprout, our open-source tool, which implements this workflow. Sprout improves task designs by (1) eliciting points of confusion from crowd workers, (2) enabling requesters to quickly understand these misconceptions and the overall space of questions, and (3) guiding requesters to improve the task design in response. We report the results of a user study with two labeling tasks demonstrating that requesters strongly prefer Sprout and produce higher-rated instructions compared to current best practices for creating gated instructions (instructions plus a workflow for training and testing workers). We also offer a set of design recommendations for future tools that support crowdsourcing task design.
```
@inproceedings{bragg-uist18-sprout,
  author = {Bragg, Jonathan and Mausam and Weld, Daniel S.},
  title = {Sprout: Crowd-Powered Task Design for Crowdsourcing},
  booktitle = {The 31st Annual {ACM} Symposium on User Interface Software and Technology,
                    {UIST} 2018, Berlin, Germany, October 14-17, 2018},
  series = {{UIST}},
  pages = {165--176},
  year = {2018},
  month = oct,
  url = {https://doi.org/10.1145/3242587.3242598},
  doi = {10.1145/3242587.3242598}
}
```

Self-Improving Crowdsourcing: Near-Effortless Design of Adaptive Distributed Work⁠

AIHCICollective intelligenceThesis

Jonathan Bragg

PhD Dissertation, University of Washington, USA, Oct 2018

PDF

@phdthesis{bragg-phdthesis18,
  author = {Bragg, Jonathan},
  title = {Self-Improving Crowdsourcing: Near-Effortless Design of Adaptive Distributed
                    Work},
  school = {University of Washington, {USA}},
  year = {2018},
  url = {https://hdl.handle.net/1773/43019}
}

2017

Subcontracting Microwork⁠

HCICollective intelligenceConference paper

Meredith Ringel Morris, Jeffrey P. Bigham, Robin Brewer, Jonathan Bragg, Anand Kulkarni, Jessie Li, Saiph Savage

CHI 2017 ·

Accepted — 25.0% of submissions

600 of 2,400 submissions to CHI 2017.

acceptance figures

PDF

Mainstream crowdwork platforms treat microtasks as indivisible units; however, in this article, we propose that there is value in re-examining this assumption. We argue that crowdwork platforms can improve their value proposition for all stakeholders by supporting subcontracting within microtasks. After describing the value proposition of subcontracting, we then define three models for microtask subcontracting: real-time assistance, task management, and task improvement, and reflect on potential use cases and implementation considerations associated with each. Finally, we describe the outcome of two tasks on Mechanical Turk meant to simulate aspects of subcontracting. We reflect on the implications of these findings for the design of future crowd work platforms that effectively harness the potential of subcontracting workflows.
```
@inproceedings{morris-chi17-subcontracting,
  author = {Morris, Meredith Ringel and Bigham, Jeffrey P. and Brewer, Robin and Bragg, Jonathan and Kulkarni, Anand and Li, Jessie and Savage, Saiph},
  title = {Subcontracting Microwork},
  booktitle = {Proceedings of the 2017 {CHI} Conference on Human Factors in Computing
                    Systems, Denver, CO, USA, May 06-11, 2017},
  series = {{CHI}},
  pages = {1867--1876},
  year = {2017},
  month = may,
  url = {https://doi.org/10.1145/3025453.3025687},
  doi = {10.1145/3025453.3025687}
}
```
Worker-Owned Cooperative Models for Training Artificial Intelligence⁠

HCICollective intelligencePoster

Anand Sriraman, Jonathan Bragg, Anand Kulkarni

CSCW Companion 2017

PDF

Artificial intelligence (AI) is widely expected to reduce the need for human labor in a variety of sectors. Workers on virtual labor marketplaces accelerate this process by generating training data for AI systems. We propose a new model where workers earn ownership of trained AI systems, allowing them to draw a long-term royalty from a tool that replaces their labor. This concept offers benefits for workers and requesters alike, reducing the upfront costs of model training while increasing longer-term rewards to workers. We identify design and technical problems associated with this new concept, including finding market opportunities for trained models, financing model training, and compensating workers fairly for training contributions. A survey of workers on Amazon Mechanical Turk about this idea finds that workers are willing to give up 25% of their earnings in exchange for an investment in the future performance of a machine learning system.
```
@inproceedings{sriraman-cscw17-worker-cooperatives,
  author = {Sriraman, Anand and Bragg, Jonathan and Kulkarni, Anand},
  title = {Worker-Owned Cooperative Models for Training Artificial Intelligence},
  booktitle = {Proceedings of the 2017 {ACM} Conference on Computer Supported Cooperative
                    Work and Social Computing, {CSCW} 2017, Portland, OR, USA, February
                    25 - March 1, 2017, Companion Volume},
  series = {{CSCW} Companion},
  pages = {311--314},
  year = {2017},
  month = feb,
  url = {https://doi.org/10.1145/3022198.3026356},
  doi = {10.1145/3022198.3026356}
}
```

2016

MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy⁠

AICollective intelligenceConference paper

Ryan Drapeau, Lydia B. Chilton, Jonathan Bragg, Daniel S. Weld

HCOMP 2016

PDF

Crowd workers are human and thus sometimes make mistakes. In order to ensure the highest quality output, requesters often issue redundant jobs with gold test questions and sophisticated aggregation mechanisms based on expectation maximization (EM). While these methods yield accurate results in many cases, they fail on extremely difficult problems with local minima, such as situations where the majority of workers get the answer wrong. Indeed, this has caused some researchers to conclude that on some tasks crowdsourcing can never achieve high accuracies, no matter how many workers are involved. This paper presents a new quality-control workflow, called MicroTalk, that requires some workers to Justify their reasoning and asks others to Reconsider their decisions after reading counter-arguments from workers with opposing views. Experiments on a challenging NLP annotation task with workers from Amazon Mechanical Turk show that (1) argumentation improves the accuracy of individual workers by 20%, (2) restricting consideration to workers with complex explanations improves accuracy even more, and (3) our complete MicroTalk aggregation workflow produces much higher accuracy than simpler voting approaches for a range of budgets.
```
@inproceedings{drapeau-hcomp16-microtalk,
  author = {Drapeau, Ryan and Chilton, Lydia B. and Bragg, Jonathan and Weld, Daniel S.},
  title = {MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy},
  booktitle = {Proceedings of the Fourth {AAAI} Conference on Human Computation and
                    Crowdsourcing, {HCOMP} 2016, 30 October - 3 November, 2016, Austin,
                    Texas, {USA}},
  series = {{HCOMP}},
  pages = {32--41},
  volume = {4},
  year = {2016},
  month = oct,
  url = {https://doi.org/10.1609/hcomp.v4i1.13270},
  doi = {10.1609/HCOMP.V4I1.13270}
}
```
Effective Crowd Annotation for Relation Extraction⁠

AICollective intelligenceConference paper

Angli Liu, Stephen Soderland, Jonathan Bragg, Christopher H. Lin, Xiao Ling, Daniel S. Weld

NAACL 2016 ·

Accepted — 26.8% of submissions

182 of 680 submissions to NAACL 2016.

acceptance figures

PDF

Can crowdsourced annotation of training data boost performance for relation extraction over methods based solely on distant supervision? While crowdsourcing has been shown effective for many NLP tasks, previous researchers found only minimal improvement when applying the method to relation extraction. This paper demonstrates that a much larger boost is possible, e.g., raising F1 from 0.40 to 0.60. Furthermore, the gains are due to a simple, generalizable technique, Gated Instruction, which combines an interactive tutorial, feedback to correct errors during training, and improved screening.
```
@inproceedings{liu-naacl16-relation-extraction,
  author = {Liu, Angli and Soderland, Stephen and Bragg, Jonathan and Lin, Christopher H. and Ling, Xiao and Weld, Daniel S.},
  title = {Effective Crowd Annotation for Relation Extraction},
  booktitle = {{NAACL} {HLT} 2016, The 2016 Conference of the North American Chapter
                    of the Association for Computational Linguistics: Human Language Technologies,
                    San Diego California, USA, June 12-17, 2016},
  series = {{NAACL}},
  pages = {897--906},
  year = {2016},
  month = jun,
  url = {https://doi.org/10.18653/v1/n16-1104},
  doi = {10.18653/V1/N16-1104}
}
```
Optimal Testing for Crowd Workers⁠

AICollective intelligenceConference paper

Jonathan Bragg, Mausam, Daniel S. Weld

AAMAS 2016

PDF Code Slides

Requesters on crowdsourcing platforms, such as Amazon Mechanical Turk, routinely insert gold questions to verify that a worker is diligent and is providing high-quality answers. However, there is no clear understanding of when and how many gold questions to insert. Typically, requesters mix a flat 10-30% of gold questions into the task stream of every worker. This static policy is arbitrary and wastes valuable budget — the exact percentage is often chosen with little experimentation, and, more importantly, it does not adapt to individual workers, the current mixture of spamming vs. diligent workers, or the number of tasks workers perform before quitting. We formulate the problem of balancing between (1) testing workers to determine their accuracy and (2) actually getting work performed as a partially-observable Markov decision process (POMDP) and apply reinforcement learning to dynamically calculate the best policy. Evaluations on both synthetic data and with real Mechanical Turk workers show that our agent learns adaptive testing policies that produce up to 111% more reward than the non-adaptive policies used by most requesters. Furthermore, our method is fully automated, easy to apply, and runs mostly out of the box.
```
@inproceedings{bragg-aamas16-worker-testing,
  author = {Bragg, Jonathan and Mausam and Weld, Daniel S.},
  title = {Optimal Testing for Crowd Workers},
  booktitle = {Proceedings of the 2016 International Conference on Autonomous Agents
                    {\&} Multiagent Systems, Singapore, May 9-13, 2016},
  series = {{AAMAS}},
  pages = {966--974},
  doi = {10.65109/pxbk4897},
  year = {2016},
  month = may,
  url = {https://dl.acm.org/doi/10.5555/2936924.2937066}
}
```
Toward Automatic Bootstrapping of Online Communities Using Decision-theoretic Optimization⁠

HCICollective intelligenceConference paper

Shih-Wen Huang^*, Jonathan Bragg^*, Isaac Cowhey, Oren Etzioni, Daniel S. Weld ^*equal contribution

CSCW 2016 ·

Accepted — 24.9% of submissions

142 of 571 submissions to CSCW 2016.

acceptance figures

PDF Slides

Successful online communities (e.g., Wikipedia, Yelp, and StackOverflow) can produce valuable content. However, many communities fail in their initial stages. Starting an online community is challenging because there is not enough content to attract a critical mass of active members. This paper examines methods for addressing this cold-start problem in datamining-bootstrappable communities by attracting non-members to contribute to the community. We make four contributions: 1) we characterize a set of communities that are “datamining-bootstrappable” and define the bootstrapping problem in terms of decision-theoretic optimization, 2) we estimate the model parameters in a case study involving the Open AI Resources website, 3) we demonstrate that non-members’ predicted interest levels and request design are important features that can significantly affect the contribution rate, and 4) we ran a simulation experiment using data generated with the learned parameters and show that our decision-theoretic optimization algorithm can generate as much community utility when bootstrapping the community as our strongest baseline while issuing only 55% as many contribution requests.
```
@inproceedings{huang-cscw16-community-bootstrapping,
  author = {Huang, Shih{-}Wen and Bragg, Jonathan and Cowhey, Isaac and Etzioni, Oren and Weld, Daniel S.},
  title = {Toward Automatic Bootstrapping of Online Communities Using Decision-theoretic
                    Optimization},
  booktitle = {Proceedings of the 19th {ACM} Conference on Computer-Supported Cooperative
                    Work {\&} Social Computing, {CSCW} 2016, San Francisco, CA, USA, February
                    27 - March 2, 2016},
  series = {{CSCW}},
  pages = {582--594},
  year = {2016},
  month = feb,
  url = {https://doi.org/10.1145/2818048.2820024},
  doi = {10.1145/2818048.2820024}
}
```

2015

Artificial Intelligence and Collective Intelligence⁠

AICollective intelligenceBook chapter

Daniel S. Weld, Mausam, Christopher H. Lin, Jonathan Bragg

Handbook of Collective Intelligence 2015

PDF

@incollection{weld-handbook15-ci,
  title = {Artificial {{Intelligence}} and {{Collective Intelligence}}},
  booktitle = {Handbook of Collective Intelligence},
  author = {Weld, Daniel S. and {Mausam} and Lin, Christopher H. and Bragg, Jonathan},
  editor = {Malone, Thomas W. and Bernstein, Michael S.},
  series = {Handbook of Collective Intelligence},
  year = {2015},
  month = oct,
  pages = {89--114},
  publisher = {{The MIT Press}},
  isbn = {0-262-02981-2},
  url = {https://mitpress.mit.edu/9780262545846/handbook-of-collective-intelligence/}
}

Learning on the Job: Optimal Instruction for Crowdsourcing⁠

AICollective intelligenceWorkshop paper

Jonathan Bragg, Mausam, Daniel S. Weld

Crowdsourcing and Machine Learning @ ICML 2015

PDF

@inproceedings{bragg-icml15-instruction,
  title = {Learning on the Job: Optimal Instruction for Crowdsourcing},
  booktitle = {{ICML} '15 Workshop on Crowdsourcing and Machine Learning},
  author = {Bragg, Jonathan and {Mausam} and Weld, Daniel S.},
  series = {Crowdsourcing and Machine Learning @ {ICML}},
  year = {2015},
  month = jul,
  url = {https://crowdml.cc/icml2015/papers/CrowdML-Paper17.pdf}
}

2014

Parallel Task Routing for Crowdsourcing⁠

AICollective intelligenceConference paper

Jonathan Bragg, Andrey Kolobov, Mausam, Daniel S. Weld

HCOMP 2014

PDF Slides

An ideal crowdsourcing or citizen-science system would route tasks to the most appropriate workers, but the best assignment is unclear because workers have varying skill, tasks have varying difficulty, and assigning several workers to a single task may significantly improve output quality. This paper defines a space of task routing problems, proves that even the simplest is NP-hard, and develops several approximation algorithms for parallel routing problems. We show that an intuitive class of requesters’ utility functions is submodular, which lets us provide iterative methods for dynamically allocating batches of tasks that make near-optimal use of available workers in each round. Experiments with live oDesk workers show that our task routing algorithm uses only 48% of the human labor compared to the commonly used round-robin strategy. Further, we provide versions of our task routing algorithm which enable it to scale to large numbers of workers and questions and to handle workers with variable response times while still providing significant benefit over common baselines.
```
@inproceedings{bragg-hcomp14-task-routing,
  author = {Bragg, Jonathan and Kolobov, Andrey and Mausam and Weld, Daniel S.},
  title = {Parallel Task Routing for Crowdsourcing},
  booktitle = {Proceedings of the Second {AAAI} Conference on Human Computation and
                    Crowdsourcing, {HCOMP} 2014, November 2-4, 2014, Pittsburgh, Pennsylvania,
                    {USA}},
  series = {{HCOMP}},
  pages = {11--21},
  volume = {2},
  year = {2014},
  month = nov,
  url = {https://doi.org/10.1609/hcomp.v2i1.13170},
  doi = {10.1609/HCOMP.V2I1.13170}
}
```

2013

Crowdsourcing Multi-Label Classification for Taxonomy Creation⁠

AICollective intelligenceConference paper

Jonathan Bragg, Mausam, Daniel S. Weld

HCOMP 2013 · Best Paper

PDF

Recent work has introduced CASCADE, an algorithm for creating a globally-consistent taxonomy by crowdsourcing microwork from many individuals, each of whom may see only a tiny fraction of the data (Chilton et al. 2013). While CASCADE needs only unskilled labor and produces taxonomies whose quality approaches that of human experts, it uses significantly more labor than experts. This paper presents DELUGE, an improved workflow that produces taxonomies with comparable quality using significantly less crowd labor. Specifically, our method for crowdsourcing multi-label classification optimizes CASCADE’s most costly step (categorization) using less than 10% of the labor required by the original approach. DELUGE’s savings come from the use of decision theory and machine learning, which allow it to pose microtasks that aim to maximize information gain.
```
@inproceedings{bragg-hcomp13-multi-label,
  author = {Bragg, Jonathan and Mausam and Weld, Daniel S.},
  title = {Crowdsourcing Multi-Label Classification for Taxonomy Creation},
  booktitle = {Proceedings of the First {AAAI} Conference on Human Computation and
                    Crowdsourcing, {HCOMP} 2013, November 7-9, 2013, Palm Springs, CA,
                    {USA}},
  series = {{HCOMP}},
  pages = {25--33},
  volume = {1},
  year = {2013},
  month = nov,
  url = {https://doi.org/10.1609/hcomp.v1i1.13091},
  doi = {10.1609/HCOMP.V1I1.13091}
}
```

2011

Neo-Riemannian Cycle Detection with Weighted Finite-State Transducers⁠

AIMusicConference paper

Jonathan Bragg, Elaine Chew, Stuart M. Shieber

ISMIR 2011 ·

PDF Data Poster

@inproceedings{bragg-ismir11-neo-riemannian,
  author = {Bragg, Jonathan and Chew, Elaine and Shieber, Stuart M.},
  title = {Neo-Riemannian Cycle Detection with Weighted Finite-State Transducers},
  booktitle = {Proceedings of the 12th International Society for Music Information
                    Retrieval Conference, {ISMIR} 2011, Miami, Florida, USA, October 24-28,
                    2011},
  series = {{ISMIR}},
  pages = {399--404},
  year = {2011},
  month = oct,
  doi = {10.5281/zenodo.1416152},
  url = {https://ismir2011.ismir.net/papers/PS3-8.pdf}
}

2010

Mathematics and Computation in Music 2009: John Clough Memorial Conference⁠

AIMusicJournal article

Jonathan Bragg, Cheng-Zhi Anna Huang

Computer Music Journal 34(1) 2010 · Conference review

PDF HTML

@article{bragg-cmj10-mcm,
  author = {Bragg, Jonathan and Huang, Cheng{-}Zhi Anna},
  title = {Mathematics and Computation in Music 2009: John Clough Memorial Conference},
  journal = {Comput. Music. J.},
  volume = {34},
  number = {1},
  pages = {100--102},
  series = {Computer Music Journal},
  year = {2010},
  month = mar,
  url = {https://doi.org/10.1162/comj.2010.34.1.100},
  doi = {10.1162/COMJ.2010.34.1.100}
}

Detection of Neo-Riemannian Cycles: A Finite State Approach⁠

AIMusicThesis

Jonathan Bragg

Honors Undergraduate Thesis, Departments of Computer Science and Music, Harvard University, Mar 2010

PDF (author copy)

@thesis{bragg-abthesis10,
  type = {Honors Undergraduate Thesis},
  title = {Detection of {{Neo-Riemannian Cycles}}: {{A Finite State Approach}}},
  author = {Bragg, Jonathan},
  year = {2010},
  school = {Departments of Computer Science and Music, Harvard University},
  url = {https://id.lib.harvard.edu/alma/990125611760203941/catalog}
}

Mentees⁠

Carrie Yuan · ⁠ Ai2 InternFrom: UW CSE PhD 2026
Arnavi Chheda-Kothary · ⁠ Ai2 InternFrom: UW CSE PhDASSETS 2026 2025
Kevin Feng · ⁠ UW StudentFrom: UW HCDE PhDCHI 2026
Peiling Jiang · ⁠ Ai2 InternFrom: UCSD Design Lab PhDarXiv 2026
Hita Kambhamettu · ⁠ Ai2 InternFrom: Penn CIS PhDUIST 2026
Kevin Feng · ⁠ Ai2 InternFrom: UW HCDE PhDCHI 2025 2024
Mingqi Gao · ⁠ Yale StudentFrom: Yale NLP VisitingNAACL 2025
Kevin Pu · ⁠ Ai2 InternFrom: Toronto CS PhDCHI 2025
Ruotong Wang · ⁠ UW StudentFrom: UW CSE PhDCHI 2025 CSCW Companion 2025
Jeongyeon Kim · ⁠ Ai2 InternFrom: Stanford CS PhD 2023
Yoonjoo Lee · ⁠ Ai2 InternFrom: KAIST CS PhDCHI 2024
Marissa Radensky · ⁠ Ai2 InternFrom: UW CSE PhDarXiv 2024
Nikhil Singh · ⁠ Ai2 InternFrom: MIT Media Lab PhDIUI 2024
Nayha Auradkar · ⁠ Ai2 InternFrom: UW CSE Masters 2022
Hyeonsu Kang · ⁠ Ai2 InternFrom: CMU HCII PhDCHI 2023
Tae Soo Kim · ⁠ Ai2 InternFrom: KAIST CS PhDUIST 2023
Srishti Palani · ⁠ Ai2 InternFrom: UCSD Cog Sci PhDCHI 2023
Nouran Soliman · ⁠ Ai2 InternFrom: MIT CSAIL PhDCHI 2023 CHI 2024
Tal August · ⁠ Ai2 InternFrom: UW CSE PhDTOCHI 2023 2021
Sanjana Chintalapati · ⁠ Ai2 InternFrom: UW CSE UndergradASSETS 2022
Raymond Fok · ⁠ Ai2 InternFrom: UW CSE PhDIUI 2023 TIIS 2024 arXiv 2025
Hyeonsu Kang · ⁠ Ai2 InternFrom: CMU HCII PhDCHI 2022
Soya Park · ⁠ MSR InternFrom: MIT CSAIL PhDCSCW 2022
Napol Rachatasumrit · ⁠ Ai2 InternFrom: CMU HCII PhDIUI 2022
Isabel Cachola · ⁠ Ai2 Predoctoral Young InvestigatorarXiv 2021 ASSETS Posters and Demos 2021 arXiv 2023 2020
Harmanpreet Kaur · ⁠ Ai2 InternFrom: Michigan CSE PhDUIST 2022
Rafal Kocielnik · ⁠ Ai2 InternFrom: UW HCDE PhDCHI 2022
Lucy Lu Wang · ⁠ Ai2 Postdoctoral Young InvestigatorarXiv 2021 ASSETS Posters and Demos 2021 ASSETS 2022 SIGACCESS Access. Comput. 2022 arXiv 2023 TOCHI 2023 IUI 2024 2019
Quan Ze Chen · ⁠ UW StudentFrom: UW CSE PhDCHI 2019 2018
Ryan Drapeau · ⁠ UW StudentFrom: UW CSE MastersHCOMP 2016 2016