Real Face Images Research Articles

Active Vision: From Theory to Application

An overview of known works in active vision area and our recent results on application of the foveal visual preprocessor to detect the head motion parameters are presented. In overview, the main directions of research and development in the field of artificial foveal active vision have been considered. It is justified that: (i) for a successful solution of complex problems in this area and creation of universal systems based on active foveal vision, it is necessary to develop new technologies and platforms for the experimental study of various aspects of active foveal vision in detail; (ii) at present, software implementations of the foveal principles of visual information processing already contribute to solving particular applied problems of computer vision. In computer simulation, detection of the initial moment of head motion was evaluated by means of foveal visual preprocessor. In this neural network, each pair of excitatory and inhibitory neurons has common center, different sizes of their receptive fields and time delay. To test network performance, synthetic video of facial image sequences from SYLAHP database monitoring the head motion were used. It was shown that the UE amplitude and polarity qualitatively correspond to face motion amplitude and direction. The initial front of UE changes corresponding to quick motion of head was equal to 12 ms in all cases (n = 46). Video of real face images with graduated turns was tested too to estimate quantitative relation between output function of excitatory neuron and turn degree. It was revealed that this relation is equal to 40 UE/degree. Future steps of research in this direction have been shortly discussed.

Identity-Preserving Face Recovery from Stylized Portraits

Given an artistic portrait, recovering the latent photorealistic face that preserves the subject's identity is challenging because the facial details are often distorted or fully lost in artistic portraits. We develop an Identity-preserving Face Recovery from Portraits (IFRP) method that utilizes a Style Removal network (SRN) and a Discriminative Network (DN). Our SRN, composed of an autoencoder with residual block-embedded skip connections, is designed to transfer feature maps of stylized images to the feature maps of the corresponding photorealistic faces. Owing to the Spatial Transformer Network (STN), SRN automatically compensates for misalignments of stylized portraits to output aligned realistic face images. To ensure the identity preservation, we promote the recovered and ground truth faces to share similar visual features via a distance measure which compares features of recovered and ground truth faces extracted from a pre-trained FaceNet network. DN has multiple convolutional and fully-connected layers, and its role is to enforce recovered faces to be similar to authentic faces. Thus, we can recover high-quality photorealistic faces from unaligned portraits while preserving the identity of the face in an image. By conducting extensive evaluations on a large-scale synthesized dataset and a hand-drawn sketch dataset, we demonstrate that our method achieves superior face recovery and attains state-of-the-art results. In addition, our method can recover photorealistic faces from unseen stylized portraits, artistic paintings, and hand-drawn sketches.

Symmetrical Viewpoint Representations in Face-Selective Regions Convey an Advantage in the Perception and Recognition of Faces.

Learning new identities is crucial for effective social interaction. A critical aspect of this process is the integration of different images from the same face into a view-invariant representation that can be used for recognition. The representation of symmetrical viewpoints has been proposed to be a key computational step in achieving view-invariance. The aim of this study was to determine whether the representation of symmetrical viewpoints in face-selective regions is directly linked to the perception and recognition of face identity. In Experiment 1, we measured fMRI responses while male and female human participants viewed images of real faces from different viewpoints (-90, -45, 0, 45, and 90° from full-face view). Within the face regions, patterns of neural response to symmetrical views (-45 and 45° or -90 and 90°) were more similar than responses to nonsymmetrical views in the fusiform face area and superior temporal sulcus, but not in the occipital face area. In Experiment 2, participants made perceptual similarity judgements to pairs of face images. Images with symmetrical viewpoints were reported as being more similar than nonsymmetric views. In Experiment 3, we asked whether symmetrical views also convey an advantage when learning new faces. We found that recognition was best when participants were tested with novel face images that were symmetrical to the learning viewpoint. Critically, the pattern of perceptual similarity and recognition across different viewpoints predicted the pattern of neural response in face-selective regions. Together, our results provide support for the functional value of symmetry as an intermediate step in generating view-invariant representations.SIGNIFICANCE STATEMENT The recognition of identity from faces is crucial for successful social interactions. A critical step in this process is the integration of different views into a unified, view-invariant representation. The representation of symmetrical views (e.g., left profile and right profile) has been proposed as an important intermediate step in computing view-invariant representations. We found view symmetric representations were specific to some face-selective regions, but not others. We also show that these neural representations influence the perception of faces. Symmetric views were perceived to be more similar and were recognized more accurately than nonsymmetric views. Moreover, the perception and recognition of faces at different viewpoints predicted patterns of response in those face regions with view symmetric representations.

Face liveness detection using convolutional-features fusion of real and deep network generated face images

Conventionally, classifiers designed for face liveness detection are trained on real-world images, where real-face images and corresponding face presentation attacks (PA) are very much overlapped. However, a little research has been carried out in utilization of the combination of real-world face images and face images generated by deep convolutional neural networks (CNN) for face liveness detection. In this paper, we evaluate the adaptive fusion of convolutional-features learned by convolutional layers from real-world face images and deep CNN generated face images for face liveness detection. Additionally, we propose an adaptive convolutional-features fusion layer that adaptively balance the fusion of convolutional-features of real-world face images and face images generated by deep CNN during training. Our extensive experiments on the state-of-the-art face anti-spoofing databases, i.e., CASIA, OULU and Replay-Attack face anti-spoofing databases with both intra-database and cross-database scenarios indicate promising performance of the proposed method on face liveness detection compared to state-of-the-art methods.

Deep Learning Based Computer Generated Face Identification Using Convolutional Neural Network

Generative adversarial networks (GANs) describe an emerging generative model which has made impressive progress in the last few years in generating photorealistic facial images. As the result, it has become more and more difficult to differentiate between computer-generated and real face images, even with the human’s eyes. If the generated images are used with the intent to mislead and deceive readers, it would probably cause severe ethical, moral, and legal issues. Moreover, it is challenging to collect a dataset for computer-generated face identification that is large enough for research purposes because the number of realistic computer-generated images is still limited and scattered on the internet. Thus, a development of a novel decision support system for analyzing and detecting computer-generated face images generated by the GAN network is crucial. In this paper, we propose a customized convolutional neural network, namely CGFace, which is specifically designed for the computer-generated face detection task by customizing the number of convolutional layers, so it performs well in detecting computer-generated face images. After that, an imbalanced framework (IF-CGFace) is created by altering CGFace’s layer structure to adjust to the imbalanced data issue by extracting features from CGFace layers and use them to train AdaBoost and eXtreme Gradient Boosting (XGB). Next, we explain the process of generating a large computer-generated dataset based on the state-of-the-art PCGAN and BEGAN model. Then, various experiments are carried out to show that the proposed model with augmented input yields the highest accuracy at 98%. Finally, we provided comparative results by applying the proposed CNN architecture on images generated by other GAN researches.

LR-GAN for degraded Face Recognition

Most Face Recognition (FR) systems provide acceptable performances under controlled environments, but fail to perform well when subjected to severe degradations. The major bottle-neck occurs due to severe degradation in the image quality captured by surveillance cameras, compared to the high-quality mug-shot images used for training. In this work, we propose a deep network based on a generative adversarial network (GAN), termed LR-GAN, which helps to reconstruct realistic mugshot images from low-resolution probe samples. These synthesized but realistic (in quality) face images facilitate superior matching performance for FR. The main contribution of the paper is in the adversarial training procedure of LR-GAN to optimize a “multi-scale reconstruction loss”, based on Multi-Scale Structural Similarity Index (MS-SSIM), patch-wise mean-squared error (PMSE), modified Jensen–Shannon Divergence (JSD) and Mutual Variation in Information (MVI). The discriminator in LR-GAN preserves the identity of each subject, based on the categorical information learned implicitly during training. Ablation studies show that the model converges to an equilibrium faster than the recent GAN based networks. Our model provides rich performances for FR, as evident by the high ( > 90%) rank-1 recognition rates, over 4 real-world degraded face datasets.

3D-Aided Dual-Agent GANs for Unconstrained Face Recognition.

Synthesizing realistic profile faces is beneficial for more efficiently training deep pose-invariant models for large-scale unconstrained face recognition, by augmenting the number of samples with extreme poses and avoiding costly annotation work. However, learning from synthetic faces may not achieve the desired performance due to the discrepancy betwedistributions of the synthetic and real face images. To narrow this gap, we propose a Dual-Agent Generative Adversarial Network (DA-GAN) model, which can improve the realism of a face simulator's output using unlabeled real faces while preserving the identity information during the realism refinement. The dual agents are specially designed for distinguishing real versus fake and identities simultaneously. In particular, we employ an off-the-shelf 3D face model as a simulator to generate profile face images with varying poses. DA-GAN leverages a fully convolutional network as the generator to generate high-resolution images and an auto-encoder as the discriminator with the dual agents. Besides the novel architecture, we make several key modifications to the standard GAN to preserve pose, texture as well as identity, and stabilize the training process: (i) a pose perception loss; (ii) an identity perception loss; (iii) an adversarial loss with a boundary equilibrium regularization term. Experimental results show that DA-GAN not only achieves outstanding perceptual results but also significantly outperforms state-of-the-arts on the large-scale and challenging NIST IJB-A and CFP unconstrained face recognition benchmarks. In addition, the proposed DA-GAN is also a promising new approach for solving generic transfer learning problems more effectively. DA-GAN is the foundation of our winning entry to the NIST IJB-A face recognition competition in which we secured the $1^{st}$ places on the tracks of verification and identification.

Measuring social variables in real and artificial faces

Human observers rapidly make social judgments about faces, including estimates of trustworthiness, aggression, and many other personality states and traits. Are social variables like these estimated effectively from artificial faces? If not, this could easily compromise interactions between humans and social agents. We examined how a set of personality variables were estimated from real face images and artificial face images synthesized from the original photographs using a morphable model of facial appearance. This stimulus set provided us with an identity-matched set of real and artificial agents, allowing for a close comparison of how social variables are estimated as a function of animacy. We found that observers’ ratings of social variables in both cases were well-described using a two-factor model of social “face space” based on valence and dominance, but real vs. artificial scores were only correlated along the valence axis. Further, we found that correlations between real and synthetic face social variables depended critically on sex categories, suggesting that social evaluation in artificial agents depends on male/female appearance.

Individual differences in hyper-realistic mask detection

Hyper-realistic masks present a new challenge to security and crime prevention. We have recently shown that people’s ability to differentiate these masks from real faces is extremely limited. Here we consider individual differences as a means to improve mask detection. Participants categorized single images as masks or real faces in a computer-based task. Experiment 1 revealed poor accuracy (40%) and large individual differences (5–100%) for high-realism masks among low-realism masks and real faces. Individual differences in mask categorization accuracy remained large when the Low-realism condition was eliminated (Experiment 2). Accuracy for mask images was not correlated with accuracy for real face images or with prior knowledge of hyper-realistic face masks. Image analysis revealed that mask and face stimuli were most strongly differentiated in the region below the eyes. Moreover, high-performing participants tracked the differential information in this area, but low-performing participants did not. Like other face tasks (e.g. identification), hyper-realistic mask detection gives rise to large individual differences in performance. Unlike many other face tasks, performance may be localized to a specific image cue.

R-BTN: Cross-Domain Face Composite and Synthesis From Limited Facial Patches

Recent face composite and synthesis related works have shown promising results in generating realistic face images from deep convolutional networks. However, these works either do not generate consistent results when the constituent patches contain large domain variations (i.e., from face and sketch domains) or cannot generate high-resolution images with limited facial patches (e.g., the inpainting approach tends to blur the generated region when the missing area is more than 50%). Motivated by the mental imagery and simulation in human cognition, we exploit the potential of deep learning networks in filling large missing region (e.g., as high as 95% missing) and generating realistic faces with high fidelity in cross domains.We propose the recursive generation by bidirectional transformation networks (r-BTN) that recursively generates a whole face/sketch from a small sketch/face patch. The large missing area and domain variations make it difficult to generate satisfactory results using a unidirectional cross-domain learning structure. We explore that the bidirectional transformation network can lead to the consistent result by minimizing the forward and backward errors in the cross-domain scenario. On the other hand, a forward and backward bidirectional learning between the face and sketch domains would enable recursive estimation of the missing region in an incremental manner to yield appealing results. r-BTN also adopts an adversarial constraint to encourage the generation of realistic faces/sketches. Extensive experiments have been conducted to demonstrate the superior performance from r-BTN as compared to existing potential solutions.

Semantic analysis on faces using deep neural networks.

In this paper we address the problem of automatic emotion recognition and classification through video. Nowadays there are excellent results focused on lab-made datasets, with posed facial expressions. On the other hand there is room for a lot of improvement in the case of `in the wild' datasets, where light, face angle to the camera, etc. are taken into account. In these cases it could be very harmful to work with a small dataset. Currently, there are not big enough datasets of adequately labeled faces for the task.\\ We use Generative Adversarial Networks in order to train models in a semi-supervised fashion, generating realistic face images in the process, allowing the exploitation of a big cumulus of unlabeled face images.

Improved Boundary Equilibrium Generative Adversarial Networks

Boundary equilibrium generative adversarial networks (BEGANs) can generate impressively realistic face images, but there is a trade-off between the quality and the diversity of generated images. Based on BEGANs, we propose an effective approach to generate images with higher quality and better diversity. By adding a second loss function (a denoising loss) to the discriminator, the discriminator can learn more useful information about the distribution of real images. Naturally, the ability of discriminator in distinguishing between real and generated images is improved, which further guides the generator to produce more realistic images to confuse the discriminator. We also find that using technique of batch normalization in BEGANs architecture can improve the diversity of generated images. By using batch normalization and adding a denoising loss to the objective of discriminator, we achieve comparative generations on CIFAR-10 and CelebA data sets. In addition, we evaluate the effect of several techniques on BEGANs framework through ”Inception-Score”, a measure which has been found to correlate well with human assessment of generated samples.

Hybrid sensing face detection and registration for low-light and unconstrained conditions.

The capability to track, detect, and identify human targets in highly cluttered scenes under extreme conditions, such as in complete darkness or on the battlefield, has been one of the primary tactical advantages in military operations. In this paper, we propose a new collaborative, multi-spectrum sensing method to achieve face detection and registration under low-light and unconstrained conditions. We design and prototype a novel type of hybrid sensor by combining a pair of near-infrared (NIR) cameras and a thermal camera (a long-wave infrared camera). We strategically surround each NIR sensor with a ring of LED IR flashes to capture the "red-eye," or more precisely, the "bright-eye" effect of the target. The "bright-eyes" are used to localize the 3D position of eyes and face. The recovered 3D information is further used to warp the thermal face imagery to a frontal-parallel pose so that additional tasks, such as face recognition, can be reliably conducted, especially with the assistance of accurate eye locations. Experiments on real face images are provided to demonstrate the merit of our method.

Face, hairstyle and clothing colour de‐identification in video sequences

The authors introduce a system for person de-identification in video data that de-identifies biometric and non-biometric features, namely faces, hairstyles and clothing colours. The authors’ system detects human faces and silhouettes in the input video and replaces the detected faces with random synthesised faces obtained using a deep convolutional generative adversarial network. Alternative hairstyles are rendered over the synthesised faces, and the human silhouette is recoloured so that skin hues are preserved and clothing hues are altered. Through the use of artificially synthesised faces that look realistic, they ensure that the de-identified image looks natural and at the same time avoid ethical and legal considerations present when using real face images as replacement faces. As they address non-biometric feature de-identification, their system offers a considerably higher level of privacy protection than commonly employed solutions that use simple image processing techniques such as blurring. Qualitative and quantitative evaluation suggests that their system produces de-identified images that look natural, at the same time being resistant to re-identification attacks.

Frankenstein: Learning Deep Face Representations Using Small Data.

Deep convolutional neural networks have recently proven extremely effective for difficult face recognition problems in uncontrolled settings. To train such networks, very large training sets are needed with millions of labeled images. For some applications, such as near-infrared (NIR) face recognition, such large training data sets are not publicly available and difficult to collect. In this paper, we propose a method to generate very large training data sets of synthetic images by compositing real face images in a given data set. We show that this method enables to learn models from as few as 10 000 training images, which perform on par with models trained from 500 000 images. Using our approach, we also obtain state-of-the-art results on the CASIA NIR-VIS2.0 heterogeneous face recognition data set.

Improved self-paced learning framework for nonnegative matrix factorization

Nonnegative matrix factorization (NMF) has been attracting intensive attention due to its wide applications. However due to the non-convexity of the NMF models, most of the existing methods are easily stuck into a bad local minima, especially in presence of noise or outliers. To alleviate this deficiency, in this paper we propose a novel NMF method by incorporating self-paced learning (SPL) methodology with traditional NMF model, to sequentially include matrix elements into NMF training from easy to complex, which draws the merits of SPL that have been demonstrated to be beneficial in avoiding bad local minima. To make the SPL methodology play a more stable and efficient role in NMF, we suggest to base on multicriteria to select training elements. The effectiveness of the proposed multicriteria self-paced NMF (MSPNMF) method is demonstrated by a series of numerical experiments on synthetic and real face image data. We also discuss the effects of different initializations on MSPNMF. Experimental results show that MSPNMF is sensitive to the starting values and different initializations should be adopted for MSPNMF based on different situations.

Simplified and efficient face recognition system on real image set and synthesised data

This paper presents experimental results related to a simplified and efficient face recognition system using a basic webcam. More specifically, the purpose of this work is two-fold: 1) to detect constantly the individual's face in front of the computer, in an uncontrolled environment; 2) to send an alert to the system manager if another individual's face is recognised. Sixteen distances based on 14 points of the face between eyes, nose and mouth are considered. Experimental results carried out on two different training sets are presented. The first database has been constructed on 500 real face images of ten individuals (50 faces each one), while the second database has been created on the same 500 previous images and 500 new synthetic data obtained through a crossover operation. For experimental evaluation, a 5-NN classifier is used. Finally, results show the performance in terms of a new correlation score for the recognition task.

Simplified and efficient face recognition system on real image set and synthesised data

This paper presents experimental results related to a simplified and efficient face recognition system using a basic webcam. More specifically, the purpose of this work is two-fold: 1) to detect constantly the individual's face in front of the computer, in an uncontrolled environment; 2) to send an alert to the system manager if another individual's face is recognised. Sixteen distances based on 14 points of the face between eyes, nose and mouth are considered. Experimental results carried out on two different training sets are presented. The first database has been constructed on 500 real face images of ten individuals (50 faces each one), while the second database has been created on the same 500 previous images and 500 new synthetic data obtained through a crossover operation. For experimental evaluation, a 5-NN classifier is used. Finally, results show the performance in terms of a new correlation score for the recognition task.

Brain responses to facial attractiveness induced by facial proportions: evidence from an fMRI study.

Brain responses to facial attractiveness induced by facial proportions are investigated by using functional magnetic resonance imaging (fMRI), in 41 young adults (22 males and 19 females). The subjects underwent fMRI while they were presented with computer-generated, yet realistic face images, which had varying facial proportions, but the same neutral facial expression, baldhead and skin tone, as stimuli. Statistical parametric mapping with parametric modulation was used to explore the brain regions with the response modulated by facial attractiveness ratings (ARs). The results showed significant linear effects of the ARs in the caudate nucleus and the orbitofrontal cortex for all of the subjects, and a non-linear response profile in the right amygdala for only the male subjects. Furthermore, canonical correlation analysis was used to learn the most relevant facial ratios that were best correlated with facial attractiveness. A regression model on the fMRI-derived facial ratio components demonstrated a strong linear relationship between the visually assessed mean ARs and the predictive ARs. Overall, this study provided, for the first time, direct neurophysiologic evidence of the effects of facial ratios on facial attractiveness and suggested that there are notable gender differences in perceiving facial attractiveness as induced by facial proportions.

Bidirectional Gender Face Aftereffects: Evidence Against Normative Facial Coding.

Facial appearance can be altered, not just by restyling but also by sensory processes. Exposure to a female face can, for instance, make subsequent faces look more masculine than they would otherwise. Two explanations exist. According to one, exposure to a female face renormalizes face perception, making that female and all other faces look more masculine as a consequence-a unidirectional effect. According to that explanation, exposure to a male face would have the opposite unidirectional effect. Another suggestion is that face gender is subject to contrastive aftereffects. These should make some faces look more masculine than the adaptor and other faces more feminine-a bidirectional effect. Here, we show that face gender aftereffects are bidirectional, as predicted by the latter hypothesis. Images of real faces rated as more and less masculine than adaptors at baseline tended to look even more and less masculine than adaptors post adaptation. This suggests that, rather than mental representations of all faces being recalibrated to better reflect the prevailing statistics of the environment, mental operations exaggerate differences between successive faces, and this can impact facial gender perception.