Using proprietary data from the internal audit department of a large multinational organization, we provide the first empirical evidence of the role that “calibration committees” play in subjective performance evaluation systems. Specifically, we examine how distributional properties of ratings, supervisor rating credibility, and structural characteristics of the calibration process influence the likelihood, direction, and magnitude of calibration committee adjustments to supervisors’ subjective ratings of subordinate auditors. We find that calibration committees make adjustments that remove inter-rater differences in the distribution of initial ratings, but have asymmetric preferences for the downward adjustment of relatively higher ratings versus the upward adjustment of relatively lower ratings. In doing so, calibration committees mitigate leniency bias, while exacerbating centrality bias. Contrary to expectation, we find no association between calibration committee adjustments and our proxies for the credibility of supervisor ratings. However, we do find that committee adjustments are decreasing in the hierarchical distance between the committee and the subordinate auditor being rated, consistent with the theoretical prediction that decision rights are optimally collocated with the decision-maker possessing the greatest relevant knowledge. We also find that adjustments are less likely when the rating supervisor serves on the committee. This study contributes to the literature on subjective performance evaluation by providing novel insights into the organizational dynamics of subjective performance evaluation systems when decision rights span hierarchical levels of the organization.
Read full abstract