A Preliminary Assessment of the Validity and Reliability of the Performance Diagnostic Checklist-Human Services

Abstract

ABSTRACTThe Performance Diagnostic Checklist-Human Services (PDC-HS) is an informant-based tool used to identify the environmental variables contributing to employee performance problems in human service settings. Although previous research has suggested that the tool is useful to identify an intervention which can be used to improve performance, no studies have formally examined the validity and reliability of the tool. In this study, we used video vignettes to assess the validity and reliability of the PDC-HS. We created three videos; each described a performance problem in one or more of the four PDC-HS domains. Twenty-one participants then watched the videos and evaluated the performance problem depicted using the PDC-HS. We measured the validity of the tool by calculating the percentage of participants who correctly identified the PDC-HS domain responsible for the performance problem. Two to four weeks later, we had participants re-watch the videos and complete the PDC-HS to obtain a measure of test re-test reliability. We also compared PDC-HS item responses across participants to evaluate inter-rater reliability. Results show that all participants correctly identified the problematic PDC-HS domain(s) in each video. PDC-HS validity values were above 93% across all three videos and domains; mean test re-test and interrater reliability values were above 95%. The results are discussed in terms of the preliminary nature of this analysis; although these data are limited by the amount and clarity of the information provided in the videos, this study represents a necessary first step in the assessment of the reliability and validity of the PDC-HS tool.