Missing observations in paired comparison data

Abstract

This paper considers the analysis of paired comparison experiments in the presence of missing responses. Various scenarios for how missing data might arise in paired comparisons are considered, and it is suggested that the most common types of missing data mechanism would be either missing completely at random or missing not at random. A new model is then proposed based on the paired comparison set of responses augmented by a set of missing data indicators for each comparison. Taking a sample selection approach, the proposed new method is based on the classical Bradley-Terry model for the response outcomes and a multinomial model for the missing indicators. Different models for the two missing data mechanisms—missing completely at random (MCAR) and missing not at random (MNAR)—are then discussed and a blockwise composite link formulation is used to construct the likelihood. Additionally, an extension to account for dependence between the paired comparison items is introduced. The methodology is illustrated by a survey paired comparison experiment on five distinct teaching qualities of teachers. We show that there is little evidence of a MNAR process in this dataset. A discussion on the sizes of problems that can be fitted using this approach concludes the paper.