Modeling the reliability and predictive validity of risk assessment in child protective services

Abstract

In a time of shrinking resources policy makers and administrators in Child Protective Services are increasingly turning to tools such as structured risk assessment to manage service demand. The reliability and predictive validity of risk assessment is questionable, however, and concerns continue about the validity of using lists of explicit criteria in protective services decision-making. In this research the issues of reliability and validity are addressed using an explicated confirmatory factor analysis model. A sample of 239 cases that included 432 children brought to CPS attention for allegations of physical abuse, neglect and child/family problems are evaluated for risk of abuse or neglect using the Washington State Risk Assessment Matrix (WARM). The study employed a three-wave panel design. Results show that a widely used risk assessment instrument exhibits high levels of measurement error and increasing stability over time, which limit the instrument's capacity to predict new allegations of abuse and neglect. Measurement error reduces the instrument's reliability while stability, in light of changes in allegation status and service intensity, reveals a consistency or stiffness that weakens predictive validity. Recommendations are offered for constructing risk assessments that are both psychometrically sound and diagnostically useful.