Inter-rater agreement in trait judgements from faces.

Robin S S Kramer,Kay L Ritchie,Mila Mileva

doi:10.1371/journal.pone.0202655

Robin S S Kramer, Kay L Ritchie + Show 1 more

Open Access

https://doi.org/10.1371/journal.pone.0202655

Copy DOI

Journal: PloS one	Publication Date: Aug 17, 2018
Citations: 31	License type: CC BY 4.0

Affiliation: University of Lincoln, University of York

Abstract

Researchers have long been interested in how social evaluations are made based upon first impressions of faces. It is also important to consider the level of agreement we see in such evaluations across raters and what this may tell us. Typically, high levels of inter-rater agreement for facial judgements are reported, but the measures used may be misleading. At present, studies commonly report Cronbach’s α as a way to quantify agreement, although problematically, there are various issues with the use of this measure. Most importantly, because researchers treat raters as items, Cronbach’s α is inflated by larger sample sizes even when agreement between raters is fixed. Here, we considered several alternative measures and investigated whether these better discriminate between traits that were predicted to show low (parental resemblance), intermediate (attractiveness, dominance, trustworthiness), and high (age, gender) levels of agreement. Importantly, the level of inter-rater agreement has not previously been studied for many of these traits. In addition, we investigated whether familiar faces resulted in differing levels of agreement in comparison with unfamiliar faces. Our results suggest that alternative measures may prove more informative than Cronbach’s α when determining how well raters agree in their judgements. Further, we found no apparent influence of familiarity on levels of agreement. Finally, we show that, like attractiveness, both trustworthiness and dominance show significant levels of private taste (personal or idiosyncratic rater perceptions), although shared taste (perceptions shared with other raters) explains similar levels of variance in people’s perceptions. In conclusion, we recommend that researchers investigating social judgements of faces consider alternatives to Cronbach’s α but should also be prepared to examine both the potential value and origin of private taste as these might prove informative.

Highlights

It has often been argued that we ‘automatically’ form first impressions from photos of faces
One interesting and commonly reported aspect of social evaluation is that people seem to agree with each other’s social trait ratings despite limited evidence supporting their accuracy ([11] cf. [12,13])
This high consensus implies that our judgements are based on certain types of physical information in the face, and all current social evaluation models use data-driven approaches to extract this information

Summary

Introduction

It has often been argued that we ‘automatically’ form first impressions from photos of faces. One interesting and commonly reported aspect of social evaluation is that people seem to agree with each other’s social trait ratings despite limited evidence supporting their accuracy ([11] cf [12,13]). This high consensus implies that our judgements are based on certain types of physical information in the face, and all current social evaluation models use data-driven approaches to extract this information (e.g., deriving key social dimensions from the ratings themselves [14]).

Methods

Results

Conclusion