Abstract
Intraclass correlation coefficients (ICCs) often are used to index the level of interrater agreement for job analysis ratings. Typically, ICCs are calculated from a one-way ANOVA design for each of several rated job characteristics in which multiple job analysts rate multiple jobs. ICCs are highest when between mean square (BMS) is large relative to within mean square (WMS), but they can be low either when (a) WMS is large, in which case ICCs appropriately indicate low interrater agreement, or (b) BMS is small, in which case ICCs reflect the attenuating influence of range restriction on the grouping factor (i.e., jobs). Using two job analysis data sets, we found that (a) ICCs covaried significantly and negatively with WMS components and (b) ICCs covaried significantly and positively with BMS. These restults indicate that ICCs cannot be interpreted unambiguously as indices of interrater agreement because they are substantially affected by the level of between-group variance. Interpretational confounding for ICCs can be circumvented by (a) using statistics designed specifically to index within-group interrater agreement, or (b) applying corrections for attenuation due to range restriction.
Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: Proceedings of the Human Factors Society Annual Meeting
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
