| Vol-2008 urn:nbn:de:0074-2008-1 Copyright © 2017 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.

# EVIA 2017
8th International Workshop on Evaluating Information Access

### Proceedings of the 8th International Workshop on Evaluating Information Access
co-located with the 13th NTCIR Conference on the Evaluation of Information Access Technologies (NTCIR 2017)

### Tokyo, Japan, December 5, 2017.

Edited by

### Nicola Ferro *
Ian Soboroff **

* University of Padua, Department of Information Engineering, Via G. Gradenigo 6/B, 35131 Padova, Italy
*** National Institute of Standards and Technology, (NIST), 100 Bureau Drive, Gaithersburg, MD 20899, USA

## Table of Contents

- Preface i-ii
  Nicola Ferro, Ian Soboroff

### Evaluation Measures and Test Collections

- Test Collections and Measures for Evaluating Customer-Helpdesk Dialogues 1-9
  Zhaohao Zeng, Cheng Luo, Lifeng Shang, Hang Li, Tetsuya Sakai
- An Interval-Like Scale Property for IR Evaluation Measures 10-15
  Marco Ferrante, Nicola Ferro, Silvia Pontarollo
- Evaluating Evaluation Measures with Worst-Case Confidence Interval Widths 16-19
  Tetsuya Sakai

### Assessment and Automatic Evaluation

- Automatic Evaluation of World History Essay Using Chronological and Geographical Measures 20-23
  Kotaro Sakamoto, Hideyuki Shibuki, Madoka Ishioroshi, Akira Fujita, Yoshinobu Kano, Teruko Mitamura, Tatsunori Mori, Noriko Kando
- Towards Automatic Evaluation of Multi-Turn Dialogues: A Task Design that Leverages Inherently Subjective Annotations 24-30
  Tetsuya Sakai
- The Effect of Inter-Assessor Disagreement on IR System Evaluation: A Case Study with Lancers and Students 31-38
  Tetsuya Sakai
- Unanimity-Aware Gain for Highly Subjective Assessments 39-42
  Tetsuya Sakai

2017-11-28: submitted by Nicola Ferro, metadata incl. bibliographic data published under Creative Commons CC0
2017-11-29: published on CEUR-WS.org |valid HTML5|