In Search of Reliability: Expert-Informed Training Methods for Conducting Observational Coding of Cognitive Behavioral Therapy Fidelity.

Abstract

Observational coding is the gold standard for measuring treatment fidelity; however, the intensive training needed for reliable and valid measurement has not been carefully scrutinized. A systematic review concluded there is a lack of information in published studies on how to approach training raters, and the available content suggests widely variable approaches are taken across research teams. The quality and comparability of the data produced from these treatment fidelity measures is undermined by heterogeneous training approaches occurring both within and between coding teams. No guidance for training teams to do observational coding for fidelity is currently available. The aims of the current study were to: (a) characterize expert-informed processes for observational coding regarding training, coding, and achieving reliability, and (b) generate expert recommendations for training and conducting observational coding. A semi-structured interview was used to explore international CBT expert's (N = 11) perspectives and experiences regarding observational coding training processes. A qualitative content analysis approach was used to analyze the data. Experts in the study provided information about their training, coding, and reliability processes. Results from this study informed recommendations on how to: (a) introduce scale items, (b) anchor ratings, and (c) drift prevention.