Training and Maintaining System-Wide Reliability in Outcome Management

Melanie A Barwick,Julia E Moore,Diana J Urajnik

doi:10.1007/s10826-012-9694-x

Abstract

The Child and Adolescent Functional Assessment Scale (CAFAS) is widely used for outcome management, for providing real time client and program level data, and the monitoring of evidence-based practices. Methods of reliability training and the assessment of rater drift are critical for service decision-making within organizations and systems of care. We assessed two approaches for CAFAS training: external technical assistance and internal technical assistance. To this end, we sampled 315 practitioners trained by external technical assistance approach from 2,344 Ontario practitioners who had achieved reliability on the CAFAS. To assess the internal technical assistance approach as a reliable alternative training method, 140 practitioners trained internally were selected from the same pool of certified raters. Reliabilities were high for both practitioners trained by external technical assistance and internal technical assistance approaches (.909–.995, .915–.997, respectively). 1 and 3-year estimates showed some drift on several scales. High and consistent reliabilities over time and training method has implications for CAFAS training of behavioral health care practitioners, and the maintenance of CAFAS as a global outcome management tool in systems of care.

Highlights

Outcome management systems are rapidly becoming a core component of healthcare services (e.g. Knaup et al 2009)
We sampled 315 practitioners trained by external technical assistance approach from 2,344 Ontario practitioners who had achieved reliability on the Child and Adolescent Functional Assessment Scale (CAFAS)
To assess the internal technical assistance approach as a reliable alternative training method, 140 practitioners trained internally were selected from the same pool of certified raters

Summary

Introduction

Outcome management systems are rapidly becoming a core component of healthcare services (e.g. Knaup et al 2009). Outcome measures can be used to determine the appropriateness of a treatment plan, the need for further treatment, and can serve as a key indicator of treatment progression or lack thereof (Howard et al 1996). This latter point is further strengthened by Whipple et al (2003), who found that clients at risk for a negative outcome were less likely to deteriorate, more likely to stay in treatment longer, and twice as likely to achieve a clinically significant change when their therapists had access to outcome and alliance information

Methods

Results

Conclusion