Interview judgments: Using external criteria to compare behavioral and graphic scale ratings

Abstract

A study was conducted which examined the psychometric properties of interview ratings. The results of a project to develop behavioral rating scales for interviews were used to compose interview scripts of hypothetical candidates for the job of resident assistant in a college dormitory. Following Borman (1975), the scaled values of behavioral examples served as the basis for external performance criteria, relative to which traditional rating errors were assessed. Audio recordings of five structured interviews were rated by 112 subjects randomly assigned to one of four conditions in a factorial design. Independent variables included type of scale with which ratings were made (behavioral or graphic) and training to avoid rating errors (received or not received). Results showed that behavioral scale ratings were significantly more accurate (relative to the external performance criteria), were less subject to halo and leniency errors, and exhibited greater interrater reliability than graphic scale ratings. No effect for training was found. The finding that behavioral scales were psychometrically superior to graphic scales when raters were inexperienced suggests the possibility that experienced interviewers may use behavioral scales to similar advantage.