A Framework for Assessing Broad Sense Agreement Between Ordinal and Continuous Measurements

Abstract

Conventional agreement studies have been confined to addressing the sense of reproducibility, and therefore are limited to assessing measurements on the same scale. In this work, we propose a new concept, called “broad sense agreement,” which extends the classical framework of agreement to evaluate the capability of interpreting a continuous measurement in an ordinal scale. We present a natural measure for broad sense agreement. Nonparametric estimation and inference procedures are developed for the proposed measure along with theoretical justifications. We also consider longitudinal settings which involve agreement assessments at multiple time points. Simulation studies have demonstrated good performance of the proposed method with small sample sizes. We illustrate our methods via an application to a mental health study.