Internal Representations of Facial Expressions Reveal Cultural Diversity

Abstract

We recently (Jack et al., 2009) challenged one of the most widely held beliefs in psychological research – the universality of facial expressions of emotion. Merging behavioural and novel spatio-temporal eye movement analyses, we showed that East Asian (EA) observers decode expressions with a culture-specific strategy that is inadequate to reliably distinguish ‘universal’ expressions of ‘Fear’ and ‘Disgust.’ Using a model observer, we demonstrated that EA observers persistently sample ambiguous eye information, while neglecting the mouth, thereby systematically confusing ‘Fear’ with ‘Surprise,’ and ‘Disgust’ with ‘Anger.’ Our rejection of universality thus raises the question - how are facial expressions represented across cultures? To investigate, we reconstructed the internal representations of Western Caucasian (WC) and EA observers, using a reverse correlation technique. On each trial, we added white noise to a racially ambiguous neutral expression face, producing a perceptively different expressive face (Figure 1 in Supplemental Materials shows sample stimuli). We instructed 15 WC and 15 EA naive observers to categorize stimuli (12,000 trials per observer) according to the 6 Ekman expressions (i.e., ‘Happy,’ ‘Surprise,’ ‘Fear,’ ‘Disgust,’ ‘Anger’ and ‘Sad’). We then reconstructed the internal representations of each expression by summing the noise templates across trials before adding the neutral face (Figure 2 in Supplemental Materials illustrates the procedure; Figure 3 shows examples of reconstructed representations). Using complementary statistical image processing tools to examine the signal properties of the representations, we reveal that while certain expressive facial features are common across cultures (e.g., wide opened eyes for ‘Surprise’), others are culture-specific (e.g., only WC observers showed the mouth for ‘Surprise’). Furthermore, we show modified gaze direction is unique to EA observers (Figure 4 in Supplemental Materials shows examples). For the first time, our results demonstrate that culture shapes the representations of facial expressions, thereby refuting their universality as signals of human social communication.