In the Same Ballpark or a Whole New Ball Game? Staff as Raters of Youth Behavior

Abstract

There is a lack of psychometrically sound tools for measuring youth outcomes in out-of-school time (OST) settings. Consequently, behavior ratings completed by OST staff are being scored as though the raters were teachers, even though cross-informant correlations are notoriously low (meta-analysis r = .27). Across 26 schools, 227 students were assessed by both teachers and OST staff using the Devereux Student Strengths Assessment (DESSA) to measure Social Emotional Competence. These 4th and 5th grade students were 50% male; 53% 5th graders; and 51% Latino, 20% Mixed/Other, 11% Black, 11% Asian, and 7% White. In the full sample, OST staff rated children’s behavior more harshly than teachers (p < .001; d = .32), although the scores were associated (r = .31, p < .001). Among the ratings completed within the same week, teacher and staff distributions were not statistically different. Teacher and staff ratings had a “medium” correlation (r = .42; p = .01) and a classification consistency (88%) that exceeded chance by a “moderate” amount (κ = .43). Few, if any, studies have previously compared the ratings of the same children by teachers and OST providers. Cross-informant inter-rater reliability between teachers and OST staff was higher than expected on the DESSA.