The Latent Structure of Multi-Observer Ratings

Abstract

This study estimated the effects of the following 5 factors on the multi-observer ratings ofratees' job performance; the ratee's general level of performance, the ratee's performanceon the specific dimension, the rater's idiosyncratic rating tendencies, the rater'sorganizational perspective, and random error. The job performance ratings were gatheredfor developmental or administrative purposes in Japanese organizations with anassessment tool; MOA-M (Multi-Observer Assessment provided by Recruit Management Solutions Co., Ltd.). The data set analyzed in this study consists of 1380 managers' dataeach rated by 2 bosses, 2 peers, 2 subordinates, and self. The results indicated thatidiosyncratic rater effect dominated over half of rating variance in the data set. The rateeperformance levels in both general and dimensional factors explained less than 25% of thevariance. The average random error effect was 15%. These results were considered togenerally coincide with the prior research results reported by Scullen et al.(2000). This isthe first study focusing on the latent structure of job performance ratings in Japan.