As explained in our paper, we re-construct the ASQA data and collect human feedback for it. We name the resulting dataset as qa-feedback, as you are viewing here.

The original train and dev ASQA sets are re-split into our train (3,853), dev (500) and test (948) sets. To run RLHF experiments, we initialize our policy model with 1K supervised training. These 1K examples are in train_1k.json. The grounding passages were obtained by mapping the oracle context information and answers, provided by ASQA, to the given passage corpus.

We collect feedback for the remaining 2,853 training examples and 500 dev examples. They can be found in train_feedback.json and dev_feedback.json. For each question, you will see four model predicted outputs, which come from sampling decoding by the initial policy model. We collect fine-grained feedback (field feedback) for the first model prediction only (field prediction 1), and preference feedback (field preference) among the four outputs.

The fine-grained feedback consist of span errors (field errors), missing information (field missing-info) and corrected response (field corrected-prediction). See the annotation interfaces Fig 5&6 in our paper. Each span error contains the start and end character indices in the text of prediction 1 and the error type. You will also see explanation for some error types, which are not used in our experiments. But in case you are interested in using it, it means the previous span being repeated for "redundant" (or repetition) errors, or the passage and sentence ids for "wrong grounding" (or inconsistent fact) errors. Each missing info includes the missing info type, and the passage and sentence ids that contain the information (but missed in the model prediction). The corrected response is the response after human edits that address the labeled errors.

The preference feedback are 6 pairwise comparisons among the 4 model outputs (1vs2, 1vs3, 1vs4, 2vs3, 2vs4, 3vs4). 1 means the first one is preferred and 2 means the second is preferred. For example, for the pair "3 vs 4", 1 means "3" is preferred and 2 means "4" is preferred. 0 means no preference.

Please see our paper and reward model data processing scripts for more details.