Practical consequences of model misfit when using rating scales to assess the severity of attention problems in children.

Daniela R Crișan,Rob B.K Wanders,Jorge N Tendeiro,Catharina A Hartman,Rob R Meijer,Don Van Ravenzwaaij

doi:10.1002/mpr.1795

Abstract

ObjectivesIn this study, we examined the consequences of ignoring violations of assumptions underlying the use of sum scores in assessing attention problems (AP) and if psychometrically more refined models improve predictions of relevant outcomes in adulthood.MethodsTracking Adolescents' Individual Lives data were used. AP symptom properties were examined using the AP scale of the Child Behavior Checklist at age 11. Consequences of model violations were evaluated in relation to psychopathology, educational attainment, financial status, and ability to form relationships in adulthood.ResultsResults showed that symptoms differed with respect to information and difficulty. Moreover, evidence of multidimensionality was found, with two groups of items measuring sluggish cognitive tempo and attention deficit hyperactivity disorder symptoms. Item response theory analyses indicated that a bifactor model fitted these data better than other competing models. In terms of accuracy of predicting functional outcomes, sum scores were robust against violations of assumptions in some situations. Nevertheless, AP scores derived from the bifactor model showed some superiority over sum scores.ConclusionThese findings show that more accurate predictions of later‐life difficulties can be made if one uses a more suitable psychometric model to assess AP severity in children. This has important implications for research and clinical practice.

Highlights

The Child Behavior Checklist (CBCL/6–18; Achenbach, 1991a; Achenbach, Dumenci, & Rescorla, 2003) is an inventory often used in practice to assess children on behavioral and emotional problems and competencies, including attention problems (AP)
We considered the participants' ability to function in several life areas as young adults, with the following specific areas measured with the TRacking Adolescents' Individual Lives Survey (TRAILS) survey at T5: (a) education achievement—a single question asking participants to indicate their latest obtained diploma by choosing one of the 15 available options representative for different levels of education in the Netherlands
As an illustration of how item response theory (IRT) location parameters relate to AP severity, a symptom severity level of 1.74 standard deviations above the mean is necessary for an individual in the clinical cohort to answer at least 1 to CBCL80, with 4.1% of the individuals being expected to endorse this symptom. These results show that the CBCL symptoms differ with respect to the level of information they provide to measuring AP severity

Summary

| INTRODUCTION

The Child Behavior Checklist (CBCL/6–18; Achenbach, 1991a; Achenbach, Dumenci, & Rescorla, 2003) is an inventory often used in practice to assess children on behavioral and emotional problems and competencies, including attention problems (AP). IRT was used for purposes such as differential item functioning analysis (e.g., Flora, Curran, Hussong, & Edwards, 2008; Lambert et al, 2007; Stevanovic et al, 2017), test score linking (e.g., Kaat et al, 2018), item selection (Lambert et al, 2003), or examining item properties over time (e.g., Petersen, Bates, Dodge, Lansford, & Pettit, 2016) These empirical studies showed that symptoms differ with respect to the information (related to measurement precision) they provide across the severity continuum and with respect to their level of difficulty (i.e., some symptoms are endorsed more often than others). Evidence collected to study our hypothesis includes several categories of difficulties associated with childhood AP

| METHODS

| RESULTS

Findings

| DISCUSSION