Naturalistic Face Research Articles

Real Face Value: The Processing of Naturalistic Facial Expressions in the Macaque Inferior Temporal Cortex.

For primates, expressions of fear are thought to be powerful social signals. In laboratory settings, faces with fearful expressions have reliably evoked valence effects in inferior temporal cortex. However, because macaques use so called "fear grins" in a variety of different contexts, the deeper question is whether the macaque inferior temporal cortex is tuned to the prototypical fear grin, or to conspecifics signaling fear? In this study, we combined neuroimaging with the results of a behavioral task to investigate how macaques encode a wide variety of fearful facial expressions. In Experiment 1, we identified two sets of macaque face stimuli using different approaches; we selected faces based on the emotional context (i.e., calm vs. fearful), and we selected faces based on the engagement of action units (i.e., neutral vs. fear grins). We also included human faces in Experiment 1. Then, using fMRI, we found that the faces selected based on context elicited a larger valence effect in the inferior temporal cortex than faces selected based on visual appearance. Furthermore, human facial expressions only elicited weak valence effects. These observations were further supported by the results of a two-alternative, forced-choice task (Experiment 2), suggesting that fear grins vary in their perceived pleasantness. Collectively, these findings indicate that the macaque inferior temporal cortex is more involved in social intelligence than commonly assumed, encoding emergent properties in naturalistic face stimuli that transcend basic visual features. These results demand a rethinking of theories surrounding the function and operationalization of primate inferior temporal cortex.

Modeling naturalistic face processing in humans with deep convolutional neural networks.

Deep convolutional neural networks (DCNNs) trained for face identification can rival and even exceed human-level performance. The ways in which the internal face representations in DCNNs relate to human cognitive representations and brain activity are not well understood. Nearly all previous studies focused on static face image processing with rapid display times and ignored the processing of naturalistic, dynamic information. To address this gap, we developed the largest naturalistic dynamic face stimulus set in human neuroimaging research (700+ naturalistic video clips of unfamiliar faces). We used this naturalistic dataset to compare representational geometries estimated from DCNNs, behavioral responses, and brain responses. We found that DCNN representational geometries were consistent across architectures, cognitive representational geometries were consistent across raters in a behavioral arrangement task, and neural representational geometries in face areas were consistent across brains. Representational geometries in late, fully connected DCNN layers, which are optimized for individuation, were much more weakly correlated with cognitive and neural geometries than were geometries in late-intermediate layers. The late-intermediate face-DCNN layers successfully matched cognitive representational geometries, as measured with a behavioral arrangement task that primarily reflected categorical attributes, and correlated with neural representational geometries in known face-selective topographies. Our study suggests that current DCNNs successfully capture neural cognitive processes for categorical attributes of faces but less accurately capture individuation and dynamic features.

Comprehensive Social Trait Judgments From Faces in Autism Spectrum Disorder.

Processing social information from faces is difficult for individuals with autism spectrum disorder (ASD). However, it remains unclear whether individuals with ASD make high-level social trait judgments from faces in the same way as neurotypical individuals. Here, we comprehensively addressed this question using naturalistic face images and representatively sampled traits. Despite similar underlying dimensional structures across traits, online adult participants with self-reported ASD showed different judgments and reduced specificity within each trait compared with neurotypical individuals. Deep neural networks revealed that these group differences were driven by specific types of faces and differential utilization of features within a face. Our results were replicated in well-characterized in-lab participants and partially generalized to more controlled face images (a preregistered study). By investigating social trait judgments in a broader population, including individuals with neurodevelopmental variations, we found important theoretical implications for the fundamental dimensions, variations, and potential behavioral consequences of social cognition.

Naturalistic face adaptation: How we adapt to freckles fast and sustainably.

While sunbathing, our skin becomes susceptible to quite remarkable changes in visual appearance, that is, freckles appear or increase in intensity-most obviously on the face. Research on face adaptation repeatedly showed that the inspection of manipulated versions of faces (so-called adaptor faces) leads to robust and sustainable changes in the perception of subsequently presented faces. Therefore, during the adaptation phase of the present study, participants saw faces with either strongly increased or decreased intensities of freckles. After a 5-minute break, during the test phase, participants had to identify the veridical (non-manipulated) face out of two faces (a slightly manipulated face combined with a non-manipulated face). Results showed strong adaptation effects to increased and decreased levels of freckles. We conclude that updating facial representations in memory is relatively fast, and these representation updates seem to sustain over a certain time span (at least 5 minutes). Face-specificity of our effects will be discussed. The results align with our everyday experience that the appearance of freckles in spring is a salient change in a familiar face; however, we seem to not register these changes after a few exposures due to a loss of information quality.

A novel face-name mnemonic discrimination task with naturalistic stimuli

Difficulty remembering faces and names is a common struggle for many people and gets more difficult as we age. Subtle changes in appearance from day to day, common facial characteristics across individuals, and overlap of names may contribute to the difficulty of learning face-name associations. Computational models suggest the hippocampus plays a key role in reducing interference across experiences with overlapping information by performing pattern separation, which enables us to encode similar experiences as distinct from one another. Thus, given the nature of overlapping features within face-name associative memory, hippocampal pattern separation may be an important underlying mechanism supporting this type of memory. Furthermore, cross-species approaches find that aging is associated with deficits in hippocampal pattern separation. Mnemonic discrimination tasks have been designed to tax hippocampal pattern separation and provide a more sensitive measure of age-related cognitive decline compared to traditional memory tasks. However, traditional face-name associative memory tasks do not parametrically vary overlapping features of faces and names to tax hippocampal pattern separation and often lack naturalistic facial features (e.g., hair, accessories, similarity of features, emotional expressions). Here, we developed a face-name mnemonic discrimination task where we varied face stimuli by similarity, race, sex, and emotional expression as well as the similarity of name stimuli. We tested a sample of healthy young and older adults on this task and found that both age groups showed worsening performance as face-name interference increased. Overall, older adults struggled to remember faces and face-name pairs more than young adults. However, while young adults remembered emotional faces better than neutral faces, older adults selectively remembered positive faces. Thus, the use of a face-name association memory task designed with varying levels of face-name interference as well as the inclusion of naturalistic face stimuli across race, sex, and emotional expressions provides a more nuanced approach relative to traditional face-name association tasks toward understanding age-related changes in memory.

Intersensory redundancy impedes face recognition in 12-month-old infants.

This study examined the role of intersensory redundancy on 12-month-old infants' attention to and processing of face stimuli. Two experiments were conducted. In Experiment 1, 72 12-month-olds were tested using an online platform called Lookit. Infants were familiarized with two videos of an actor reciting a children's story presented simultaneously. A soundtrack either matched one of the videos (experimental condition) or neither of the videos (control condition). Visual-paired comparison (VPC) trials were completed to measure looking preferences for the faces presented synchronously and asynchronously during familiarization and for novel faces. Neither group displayed looking preferences during the VPC trials. It is possible that the complexity of the familiarization phase made the modality-specific face properties (i.e., facial characteristics and configuration) difficult to process. In Experiment 2, 56 12-month-old infants were familiarized with the video of only one actor presented either synchronously or asynchronously with the soundtrack. Following familiarization, participants completed a VPC procedure including the familiar face and a novel face. Results from Experiment 2 showed that infants in the synchronous condition paid more attention during familiarization than infants in the asynchronous condition. Infants in the asynchronous condition demonstrated recognition of the familiar face. These findings suggest that the competing face stimuli in the Experiment 1 were too complex for the facial characteristics to be processed. The procedure in Experiment 2 led to increased processing of the face in the asynchronous presentation. These results indicate that intersensory redundancy in the presentation of synchronous audiovisual faces is very salient, discouraging the processing of modality-specific visual properties. This research contributes to the understanding of face processing in multimodal contexts, which have been understudied, although a great deal of naturalistic face exposure occurs multimodally.

Fine-scale dynamics of functional connectivity in the face-processing network during movie watching

Mapping the human face-processing network is typically done during rest or using isolated, static face images, overlooking widespread cortical interactions obtained in response to naturalistic face dynamics and context. To determine how inter-subject functional correlation (ISFC) relates to face recognition scores, we measure cortical connectivity patterns in response to a dynamic movie in typical adults (N= 517). We find a positive correlation with recognition scores in edges connecting the occipital visual and anterior temporal regions and a negative correlation in edges connecting the attentional dorsal, frontal default, and occipital visual regions. We measure the inter-subject stimulus-evoked response at a single TR resolution and demonstrate that co-fluctuations in face-selective edges are related to activity in core face-selective regions and that the ISFC patterns peak during boundaries between movie segments rather than during the presence of faces. Our approach demonstrates how face processing is linked to fine-scale dynamics in attentional, memory, and perceptual neural circuitry.

Encoding of dynamic facial information in the middle dorsal face area

Faces in motion reveal a plethora of information through visual dynamics. Faces can move in complex patterns while transforming facial shape, e.g., during the generation of different emotional expressions. While motion and shape processing have been studied extensively in separate research enterprises, much less is known about their conjunction during biological motion. Here, we took advantage of the discovery in brain-imaging studies of an area in the dorsal portion of the macaque monkey superior temporal sulcus (STS), the middle dorsal face area (MD), with selectivity for naturalistic face motion. To gain mechanistic insights into the coding of facial motion, we recorded single-unit activity from MD, testing whether and how MD cells encode face motion. The MD population was highly sensitive to naturalistic facial motion and facial shape. Some MD cells responded only to the conjunction of facial shape and motion, others were selective for facial shape even without movement, and yet others were suppressed by facial motion. We found that this heterogeneous MD population transforms face motion into a higher dimensional activity space, a representation that would allow for high sensitivity to relevant small-scale movements. Indeed, we show that many MD cells carry such sensitivity for eye movements. We further found that MD cells encode motion of head, mouth, and eyes in a separable manner, requiring the use of multiple reference frames. Thus, MD is a bona fide face-motion area that uses highly heterogeneous cell populations to create codes capturing even complex facial motion trajectories.

First impressions from faces in the real world: Commentary on Sutherland and Young (2022).

The study of first impressions from faces now emphasizes the need to understand trait inferences made to naturalistic face images (British Journal of Psychology, 113, 2022, 1056). Face recognition algorithms based on deep convolutional neural networks simultaneously represent invariant, changeable and environmental variables in face images. Therefore, we suggest them as a comprehensive 'face space' model of first impressions of naturalistic faces. We also suggest that to understand trait inferences in the real world, a logical next step is to consider trait inferences made to whole people (faces and bodies). On the role of cultural contributions to trait perception, we think it is important for the field to begin to consider the way in which trait inferences motivate (or not) behaviour in independent and interdependent cultures.

Dynamic, naturalistic faces embedded in a narrative elicit responses in the distributed face processing system

Dynamic, naturalistic faces embedded in a narrative elicit responses in the distributed face processing system

Not so fast: Limited validity of deep convolutional neural networks as in silico models for human naturalistic face processing

Not so fast: Limited validity of deep convolutional neural networks as in silico models for human naturalistic face processing

Does learning different script systems affect configural visual processing? ERP evidence from early readers of Chinese and German.

AbstractReading is a complex cultural skill requiring considerable training, apparently affecting also the processing of non‐linguistic visual stimuli. We examined whether the different visual demands involved in reading different script systems—alphabetic German versus logographic Chinese script—would differentially influence configural visual processing. Our main dependent measure was the N170 component of the ERP, which is considered as a signature of configural processing. In the present study, German and Chinese children (N = 28 vs. 27) who had received about one year of formal instruction in their native script system, worked on a series of one‐back tasks with naturalistic faces, two‐tone Mooney faces and doodles, and on an adaptation task with pairs of faces were either identical or differed in their second‐order relations. Chinese children showed larger N170 amplitudes than German children for naturalistic and Mooney faces, specifically indicating superior holistic processing in Chinese children. In contrast, there was no superiority in Chinese children on the second‐order adaptation effect at the N170, providing no evidence for differences in second‐order relations processing of facial configurations between the groups. Given the sensitivity of the visual system to reading acquisition, these findings suggest that these group differences in holistic processing might be due to the extensive training with the highly complex logographic script system learned by Chinese children, imposing high demands on higher‐order visual perception.

Working memory guidance of visual attention to threat in offenders.

Aggression and trait anger have been linked to attentional biases toward angry faces and attribution of hostile intent in ambiguous social situations. Memory and emotion play a crucial role in social-cognitive models of aggression but their mechanisms of influence are not fully understood. Combining a memory task and a visual search task, this study investigated the guidance of attention allocation toward naturalistic face targets during visual search by visual working memory (WM) templates in 113 participants who self-reported having served a custodial sentence. Searches were faster when angry faces were held in working memory regardless of the emotional valence of the visual search target. Higher aggression and trait anger predicted increased working memory modulated attentional bias. These results are consistent with the Social-Information Processing model, demonstrating that internal representations bias attention allocation to threat and that the bias is linked to aggression and trait anger.

Naturalistic Face Learning in Infants and Adults

Everyday face recognition presents a difficult challenge because faces vary naturally in appearance as a result of changes in lighting, expression, viewing angle, and hairstyle. We know little about how humans develop the ability to learn faces despite natural facial variability. In the current study, we provide the first examination of attentional mechanisms underlying adults’ and infants’ learning of naturally varying faces. Adults (n = 48) and 6- to 12-month-old infants (n = 48) viewed videos of models reading a storybook; the facial appearance of these models was either high or low in variability. Participants then viewed the learned face paired with a novel face. Infants showed adultlike prioritization of face over nonface regions; both age groups fixated the face region more in the high- than low-variability condition. Overall, however, infants showed less ability to resist contextual distractions during learning, which potentially contributed to their lack of discrimination between the learned and novel faces. Mechanisms underlying face learning across natural variability are discussed.

From eye to arrow: Attention capture by direct gaze requires more than just the eyes

Human attention is strongly attracted by direct gaze and sudden onset motion. The sudden direct-gaze effect refers to the processing advantage for targets appearing on peripheral faces that suddenly establish eye contact. Here, we investigate the necessity of social information for attention capture by (sudden onset) ostensive cues. Six experiments involving 204 participants applied (1) naturalistic faces, (2) arrows, (3) schematic eyes, (4) naturalistic eyes, or schematic facial configurations (5) without or (6) with head turn to an attention-capture paradigm. Trials started with two stimuli oriented towards the observer and two stimuli pointing into the periphery. Simultaneous to target presentation, one direct stimulus changed to averted and one averted stimulus changed to direct, yielding a 2 × 2 factorial design with direction and motion cues being absent or present. We replicated the (sudden) direct-gaze effect for photographic faces, but found no corresponding effects in Experiments 2–6. Hence, a holistic and socially meaningful facial context seems vital for attention capture by direct gaze.Statement of significanceThe present study highlights the significance of context information for social attention. Our findings demonstrate that the direct-gaze effect, that is, the prioritization of direct gaze over averted gaze, critically relies on the presentation of a meaningful holistic and naturalistic facial context. This pattern of results is evidence in favor of early effects of surrounding social information on attention capture by direct gaze.

Linear Integration of Sensory Evidence over Space and Time Underlies Face Categorization.

Visual object recognition relies on elaborate sensory processes that transform retinal inputs to object representations, but it also requires decision-making processes that read out object representations and function over prolonged time scales. The computational properties of these decision-making processes remain underexplored for object recognition. Here, we study these computations by developing a stochastic multifeature face categorization task. Using quantitative models and tight control of spatiotemporal visual information, we demonstrate that human subjects (five males, eight females) categorize faces through an integration process that first linearly adds the evidence conferred by task-relevant features over space to create aggregated momentary evidence and then linearly integrates it over time with minimum information loss. Discrimination of stimuli along different category boundaries (e.g., identity or expression of a face) is implemented by adjusting feature weights of spatial integration. This linear but flexible integration process over space and time bridges past studies on simple perceptual decisions to complex object recognition behavior.SIGNIFICANCE STATEMENT Although simple perceptual decision-making such as discrimination of random dot motion has been successfully explained as accumulation of sensory evidence, we lack rigorous experimental paradigms to study the mechanisms underlying complex perceptual decision-making such as discrimination of naturalistic faces. We develop a stochastic multifeature face categorization task as a systematic approach to quantify the properties and potential limitations of the decision-making processes during object recognition. We show that human face categorization could be modeled as a linear integration of sensory evidence over space and time. Our framework to study object recognition as a spatiotemporal integration process is broadly applicable to other object categories and bridges past studies of object recognition and perceptual decision-making.

Examining cultural differences in naturalistic face scanning: A data-driven approach to analysing head-mounted eye-tracking data

Examining cultural differences in naturalistic face scanning: A data-driven approach to analysing head-mounted eye-tracking data

Electrophysiological evidence for attentional capture by irrelevant angry facial expressions: Naturalistic faces

Recently, research on lateralized event related potentials (ERPs) in response to irrelevant distractors has revealed that angry but not happy schematic distractors capture spatial attention. Whether this effect occurs in the context of the natural expression of emotions is unknown. To fill this gap, observers were asked to judge the gender of a natural face surrounded by a color singleton among five other face identities. In contrast to previous studies, the similarity between the task-relevant feature (color) and the distractor features was low. On some trials, the target was displayed concurrently with an irrelevant angry or happy face. The lateralized ERPs to these distractors were measured as a marker of spatial attention. Our results revealed that angry face distractors, but not happy face distractors, triggered a PD, which is a marker of distractor suppression. Subsequent to the PD, angry distractors elicited a larger N450 component, which is associated with conflict detection. We conclude that threatening expressions have a high attentional priority because of their emotional value, resulting in early suppression and late conflict detection.

Developmental changes in infants' attention to naturalistic faces and visual saliency

Developmental changes in infants' attention to naturalistic faces and visual saliency

Personality judgments from everyday images of faces.

People readily make personality attributions to images of strangers' faces. Here we investigated the basis of these personality attributions as made to everyday, naturalistic face images. In a first study, we used 1000 highly varying “ambient image” face photographs to test the correspondence between personality judgments of the Big Five and dimensions known to underlie a range of facial first impressions: approachability, dominance, and youthful-attractiveness. Interestingly, the facial Big Five judgments were found to separate to some extent: judgments of openness, extraversion, emotional stability, and agreeableness were mainly linked to facial first impressions of approachability, whereas conscientiousness judgments involved a combination of approachability and dominance. In a second study we used average face images to investigate which main cues are used by perceivers to make impressions of the Big Five, by extracting consistent cues to impressions from the large variation in the original images. When forming impressions of strangers from highly varying, naturalistic face photographs, perceivers mainly seem to rely on broad facial cues to approachability, such as smiling.