Modelling covariate effects in observer agreement studies: The case of nominal scale agreement

Abstract

Correction for chance-expected agreement has become an accepted technique in the analysis of observer agreement data and may be particularly useful when the level of agreement achieved in different populations is compared. However, formal methods for making comparisons of chance-corrected agreement or, more generally, for studying the effects of covariates on chance-corrected agreement have not received much attention. For nominal scale agreement data we show how Tanner and Young's model for observer agreement can be applied to this problem. The models discussed can be fitted using existing software and certain model parameters have interpretations in terms of positive and negative agreement odds ratios. The proposed methodology facilitates investigation of issues such as confounding of covariate effects and interaction between covariates in their effect on chance-corrected agreement. The methods outlined therefore allow observer agreement data to be analyzed in a manner strongly analogous to the logistic modelling of the association between disease and suspected risk factors. The methods are illustrated using data on the comparability of primary and proxy respondent reports of the primary respondents participation in physically vigorous leisure time activity.