Evaluations of the sum-score-based and item response theory-based tests of group mean differences under various simulation conditions.

Mian Wang,Bryce B Reeve

doi:10.1177/09622802211043263

Abstract

The use of patient-reported outcomes measures is gaining popularity in clinical trials for comparing patient groups. Such comparisons typically focus on the differences in group means and are carried out using either a traditional sum-score-based approach or item response theory (IRT)-based approaches. Several simulation studies have evaluated different group mean comparison approaches in the past, but the performance of these approaches remained unknown under certain uninvestigated conditions (e.g. under the impact of differential item functioning (DIF)). By incorporating some of the uninvestigated simulation features, the current study examines Type I error, statistical power, and effect size estimation accuracy associated with group mean comparisons using simple sum scores, IRT model likelihood ratio tests, and IRT expected-a-posteriori scores. Manipulated features include sample size per group, number of items, number of response categories, strength of discrimination parameters, location of thresholds, impact of DIF, and presence of missing data. Results are summarized and visualized using decision trees.

Highlights

In the past two decades, the importance of patient-reported outcomes (PROs) has been well recognized in the medical and healthcare research community, and an increasing number of clinical trials have included endpoints utilizing PRO instruments.[1,2] Typically, a PRO instrument consists of multiple survey questions that are intended to assess patients’ levels on a latent construct that is often not directly observable
A preliminary decision tree based on the full Type I error summary table showed a clear separation in splitting patterns between conditions with and without differential item functioning (DIF), which indicated a strong impact of DIF
The current study extended the work of previous simulations on the performance of different group mean comparison methods, by incorporating four new manipulated features

Summary

Introduction

In the past two decades, the importance of patient-reported outcomes (PROs) has been well recognized in the medical and healthcare research community, and an increasing number of clinical trials have included endpoints utilizing PRO instruments.[1,2] Typically, a PRO instrument consists of multiple survey questions that are intended to assess patients’ levels on a latent construct that is often not directly observable (e.g. post-treatment life satisfaction) Based on such an assessment, sample means can be estimated for predefined patient groups in a clinical trial, and group comparisons can be carried out. Item response theory (IRT) models have gained popularity in the PRO field,[3,4] and IRT-model-based group mean comparisons are not uncommon in applications.[5,6] Under an IRT model, the probability of observing each item response is expressed as a function of both item parameters and person scores on a latent factor

Methods

Results

Discussion

Conclusion