Confronting the Base Rate Problem: More Ups and Downs for Cognitive Scatter Analysis

Abstract

Within the professional literature, it is frequently suggested that interpretation of cognitive profile scatter may be useful for generating a host of clinical inferences. To wit, Hale et al. (Psychology in the Schools, 45, 838–858, (2008)) posit that cognitive scatter is a defining characteristic of specific learning disability and that individuals with learning disabilities may have higher levels of scatter compared to normal controls. To investigate the tenability of this claim, the present study employed diagnostic efficiency statistics and other recommended psychometric methods (e.g., receiver operative characteristic curve, Bayesian nomogram) to test whether cognitive scatter could accurately distinguish between individuals with and without a known learning disability (LD) diagnosis in the Kaufman Assessment Battery for Children-Second Edition KABC-II; Kaufman and Kaufman (2004a) normative sample. Results indicated that increasing levels of cognitive profile scatter identified individuals with LD at no better than chance levels. The current negative results add to a growing corpus of research questioning the utility of many of the interpretive procedures that are utilized by school psychologists for commercial ability measures. In particular, it is suggested that clinicians who interpret cognitive profile scatter may risk diagnostic overconfidence and in the case of LD identification, unacceptable levels of false positive decisions attributable to error. Implications for evidence-based assessment in school psychology are discussed.