Test-Retest and Inter-Rater Reliability of a Process Skills Assessment

Abstract

The Process Skills Assessment is an observational assessment designed to evaluate process skills as demonstrated during the performance of a self-selected task. Retest reliability results indicated that the assessment can be used to measure changes in clients' process skills after therapy if the same task is used upon re-administration. Although all assessment tasks meet specific guidelines, each one places slightly different demands on the subject's process skills. The different process skill demands of tasks must be addressed to ensure generalizability of the assessment results beyond the specific task. Inter-rater reliability results indicated that two raters observing the same subject will reach a moderate level of agreement when identifying general process skill deficits. The findings suggest that increased specificity of the individual rating scale items and procedures to ensure raters' adherence to assessment training protocol will improve the consistency of ratings across raters.