Facial Occlusion Research Articles

End2End Occluded Face Recognition by Masking Corrupted Features.

With the recent advancement of deep convolutional neural networks, significant progress has been made in general face recognition. However, the state-of-the-art general face recognition models do not generalize well to occluded face images, which are exactly the common cases in real-world scenarios. The potential reasons are the absences of large-scale occluded face data for training and specific designs for tackling corrupted features brought by occlusions. This article presents a novel face recognition method that is robust to occlusions based on a single end-to-end deep neural network. Our approach, named FROM (Face Recognition with Occlusion Masks), learns to discover the corrupted features from the deep convolutional neural networks, and clean them by the dynamically learned masks. In addition, we construct massive occluded face images to train FROM effectively and efficiently. FROM is simple yet powerful compared to the existing methods that either rely on external detectors to discover the occlusions or employ shallow models which are less discriminative. Experimental results on the LFW, Megaface challenge 1, RMF2, AR dataset and other simulated occluded/masked datasets confirm that FROM dramatically improves the accuracy under occlusions, and generalizes well on general face recognition.

Hybrid attention network and center-guided non-maximum suppression for occluded face detection

Hybrid attention network and center-guided non-maximum suppression for occluded face detection

Emotional context can reduce the negative impact of face masks on inferring emotions.

While face masks prevent the spread of disease, they occlude lower face parts and thus impair facial emotion recognition. Since emotions are often also contextually situated, it remains unknown whether providing a descriptive emotional context alongside the facial emotion may reduce some of the negative impact of facial occlusion on emotional communication. To address this question, here we examined how emotional inferences were affected by facial occlusion and the availability of emotional context. Participants were presented with happy or sad emotional faces who were either fully visible or partially obstructed by an opaque surgical mask. The faces were shown either within an emotionally congruent (e.g., “Her cat was found/lost yesterday afternoon”) or neutral (“Get ready to see the next person”) context. Participants were asked to infer the emotional states of the protagonists by rating their emotional intensity and valence. Facial occlusion by masks impacted the ratings, such that protagonists were judged to feel less intense and more neutral emotions when they wore masks relative to when their face was fully visible. Importantly, this negative impact of visual occlusion by mask was reduced but not fully eliminated when the faces were presented within a congruent emotional context. Thus, visual occlusion of facial emotions impairs understanding of emotions, with this negative effect of face masks partially mitigated by the availability of a larger emotional context.

Dual-position features fusion for head pose estimation for complex scene

Head pose estimation (HPE) is widely used in attention detection, behavior analysis, and expression recognition. Nevertheless, in some complex scenes (such as facial occlusion, large head deflection angle, and multi-person in one scene), HPE still has the problem of low estimation accuracy. To solve this problem, we propose a dual position feature fusion method for estimating head pose. First, the RGB input is replaced with a standard luminance, which reduces the effect of extraneous light factors. Subsequently, the center offset loss is used to detect the head and body position, and dynamic adjustment strategy is used to deflate the border, aiming to not only obtain the best confidence level but also improve the capability of multi-person HPE. Finally, the estimate results under head position and body position are fused to further reduce the estimate loss. We tested our approach on the popular public AFLW2000, BIWI, and UPNA datasets, the results show the superiority of our approach in solving the occlusion, deflection, and multi-person scene problems.

Adaptive Multilayer Perceptual Attention Network for Facial Expression Recognition

In complex real-world situations, problems such as illumination changes, facial occlusion, and variant poses make facial expression recognition (FER) a challenging task. To solve the robustness problem, this paper proposes an adaptive multilayer perceptual attention network (AMP-Net) that is inspired by the facial attributes and the facial perception mechanism of the human visual system. AMP-Net extracts global, local, and salient facial emotional features with different fine-grained features to learn the underlying diversity and key information of facial emotions. Different from existing methods, AMP-Net can adaptively guide the network to focus on multiple finer and distinguishable local patches with robustness to occlusion and variant poses, improving the effectiveness of learning potential facial diversity information. In addition, the proposed global perception module can learn different receptive field features in the global perception domain, and AMP-Net also supplements salient facial region features with high emotion correlation based on prior knowledge to capture key texture details and avoid important information loss. Many experiments show that AMP-Net achieves good generalizability and state-of-the-art results on several real-world datasets, including RAF-DB, AffectNet-7, AffectNet-8, SFEW 2.0, FER-2013, and FED-RO, with accuracies of 89.25%, 64.54%, 61.74%, 61.17%, 74.48%, and 71.75%, respectively. All codes and training logs are publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/liuhw01/AMP-Net</uri> .

The limitations for expression recognition in computer vision introduced by facial masks

Facial Expression recognition is a computer vision problem that took relevant benefit from the research in deep learning. Recent deep neural networks achieved superior results, demonstrating the feasibility of recognizing the expression of a user from a single picture or a video recording the face dynamics. Research studies reveal that the most discriminating portions of the face surfaces that contribute to the recognition of facial expressions are located on the mouth and the eyes. The restrictions for COVID pandemic reasons have also revealed that state-of-the-art solutions for the analysis of the face can severely fail due to the occlusions of using the facial masks. This study explores to what extend expression recognition can deal with occluded faces in presence of masks. To a fairer comparison, the analysis is performed in different occluded scenarios to effectively assess if the facial masks can really imply a decrease in the recognition accuracy. The experiments performed on two public datasets show that some famous top deep classifiers expose a significant reduction in accuracy in presence of masks up to half of the accuracy achieved in non-occluded conditions. Moreover, a relevant decrease in performance is also reported also in the case of occluded eyes but the overall drop in performance is not as severe as in presence of the facial masks, thus confirming that, like happens for face biometric recognition, occluded faces by facial mask still represent a challenging limitation for computer vision solutions.

Single Camera Face Position-Invariant Driver's Gaze Zone Classifier Based on Frame-Sequence Recognition Using 3D Convolutional Neural Networks.

Estimating the driver’s gaze in a natural real-world setting can be problematic for different challenging scenario conditions. For example, faces will undergo facial occlusions, illumination, or various face positions while driving. In this effort, we aim to reduce misclassifications in driving situations when the driver has different face distances regarding the camera. Three-dimensional Convolutional Neural Networks (CNN) models can make a spatio-temporal driver’s representation that extracts features encoded in multiple adjacent frames that can describe motions. This characteristic may help ease the deficiencies of a per-frame recognition system due to the lack of context information. For example, the front, navigator, right window, left window, back mirror, and speed meter are part of the known common areas to be checked by drivers. Based on this, we implement and evaluate a model that is able to detect the head direction toward these regions having various distances from the camera. In our evaluation, the 2D CNN model had a mean average recall of 74.96% across the three models, whereas the 3D CNN model had a mean average recall of 87.02%. This result show that our proposed 3D CNN-based approach outperforms a 2D CNN per-frame recognition approach in driving situations when the driver’s face has different distances from the camera.

Eye Localisation using Cascaded U-Net for Autism Spectrum Disorder

Many computer vision applications Computer Aided Diagnosis require an accurate and efficient eye detector. We represent, in this work, an efficient approach for determining the position of the eye in images presenting faces. First, a series of candidate regions are proposed; next, a set of cascaded U-net is used to detect the eye regions; then edge detection methods are used to detect eyes lids and iris boundary and thus helping determining the gaze direction of person having ASD. Our proposed approach achieved a precision of detection that is better than the current most recent methods in trials utilizing GI4E, BioID, and columbiaGaze datasets. The proposed approach is robust to picture alterations, such as changes in external illumination, facial occlusion, and image modality.

Color Occlusion Face Recognition Method Based on Quaternion Non-Convex Sparse Constraint Mechanism.

As the acquisition and application of color images become more and more extensive, color face recognition technology has also been vigorously developed, especially the recognition methods based on convolutional neural network, which have excellent performance. However, with the increasing depth and complexity of network models, the number of calculated parameters also increases, which means the training of most high-performance models depends on large-scale samples and expensive equipment. Therefore, the key to the current research is to realize a lightweight model while ensuring the recognition accuracy. At present, PCANet, a typical lightweight framework for deep learning, has achieved good results in most of the image recognition tasks, but its recognition accuracy for color face images, especially under occlusion, still needs to be improved. Therefore, a color occlusion face recognition method based on quaternion non-convex sparse constraint mechanism is proposed in this paper. Firstly, a quaternion non-convex sparse principal component analysis network model was constructed based on Lp regularization of strong sparsity. Secondly, the fixed point iteration method and coordinate descent method were established to solve the non-convex optimization problem. Finally, the occlusion recognition performance of the proposed method was verified on Georgia Tech, Color FERET, AR, and LFW-A Color face datasets.

LocalFace: Learning significant local features for deep face recognition

As a widely mentioned topic in face recognition, the margin-based loss function enhances the discriminability of face recognition models by applying margin between class decision boundaries. However, there is still room to improve the representation of face features. Local face feature extraction has been employed in traditional face recognition methods, but with the increase of network depth in deep learning, the traditional approach requires a large number of computational resources. In this paper, we propose a novel face recognition architecture called LocalFace to extract local face features. First, by analyzing the distribution of significant features in face images, we propose an efficient face fixed-point local feature extraction approach and improve this method to propose a more effective face dynamic local feature extraction scheme. Subsequently, we propose a block-based random occlusion method for the limitations of the random face occlusion method to better simulate the occlusion situation in real scenes. In the end, we present a detailed discussion on the channel attention method that is more appropriate for face recognition and classification tasks. Our method enhances the representation of face features by ensembling local features into global features without extra parameters, which is efficient and easy to implement. Extensive experiments on various benchmarks demonstrate the superiority of our LocalFace, and part of the experimental results achieve SOTA results.

A New Design of Occlusion Invariant Face Recognition Using Optimal Pattern Extraction and CNN with GRU-Based Architecture

Deep learning networks are considered as an important technique for face recognition and image recognition. Convolutional Neural Networks (CNN) is regarded as a problem solver in face recognition challenges. To solve the challenges of occlusion and noise in the image, more clarification is needed to acquire high accuracy. Hence, a deep learning model is developed in this paper. The proposed model covers four main steps: (a) Data acquisition, (b) Pre-processing, (c) pattern extraction, and (d) classification. The benchmark datasets with occluded faces is gathered from public source. Further, the pre-processing of the images is performed by contrast enhancement and Gabor filtering. With these pre-processed images, pattern extraction is done by the optimal local mesh ternary pattern. By inputting the pattern extracted image, a deep learning model “CNN with Gated Recurrent Unit (GRU)” network performs the recognition process. The experimental results are obtained and the proposed model gives better classification accuracy.

Facial expression recognition under constrained conditions using stacked generalized convolution neural network

A cognitive-analysis of facial features can make facial expression recognition system more robust and efficient for Human-Machine Interaction (HMI) applications. Through this work, we propose a new methodology to improve accuracy of facial expression recognition system even with the constraints like partial hidden faces or occlusions for real time applications. As a first step, seven independent facial segments: Full-Face, half-face (left/right), upper half face, lower half face, eyes, mouth and nose are considered to recognize facial expression. Unlike the work reported in literature, where arbitrarily generated patch type occlusions on facial regions are used, in this work a detailed analysis of each facial feature is explored. Using the results thus obtained, these seven sub models are combined using a Stacked Generalized ensemble method with deep neural network as meta-learner to improve accuracy of facial expression recognition system even in occluded state. The accuracy of the proposed model improved up to 30% compared to individual model accuracies for cross-corpus seven model datasets. The proposed system uses CNN with RPA compliance and is also configured on Raspberry Pi, which can be used for HRI and Industry 4.0 applications which involve face occlusion and partially hidden face challenges.

Facial expression recognition in facial occlusion scenarios: A path selection multi-network

The occlusion scenarios including wearing mask, wearing sunglasses, wearing hat, etc. Thus, parts of face are occluded, e.g. nose and mouth are obscured if a mask is worn. Facial expression recognition (FER) tasks have been widely researched. However, less attention has been paid to FER in occlusion scenarios, which are not uncommon in the real world. In this paper, we propose a method that structures a path selection multi-network model to achieve the FER in the above three types of facial occlusion scenarios. The method contains two parts. For the multi-network, we segment the labels in one database which results in three new sub-databases to train three Subnets, respectively. For the path selection, which is an integration method of multi-network, we merge groups of labels in one database to train an initial network called BeginNet. The prediction of BeginNet selects one of the Subnets to make the final prediction. We concatenate four popular databases Fer2013, JAFFE, KDEF and Raf-DB databases into one larger database, and use the combined database to verify our method effectiveness. The experimental results show that our method has better results in coping with the expression recognition task in multiple types of facial occlusion scenarios.

Research on occlusion block face recognition based on feature point location

AbstractAiming at improving the poor face recognition accuracy under occlusion, a multipose block occlusion face recognition method based on feature point location is proposed. Face segmentation is carried out according to the location results of face feature points and occlusion areas. The local features of each face block are extracted by deep convolution neural network. The dynamic adaptive weighting method is used to give different weights to the face block information, and the occluded face recognition is completed according to the results of face segmentation, which effectively reduces the impact of pose change and occlusion on face recognition. The experiment is analyzed from two aspects: frontal occlusion and multipose face occlusion. The results show that fewer blocks in the frontal occlusion experiment are conducive to maintaining good local integrity and relatively good recognition performance; When the proportion of frontal occlusion reaches 50%, the recognition rate of our algorithm can still be up to 92.68%, which is significantly better than other algorithms.

Driver Fatigue Detection Method Based on Human Pose Information Entropy

Driver fatigue detection (DFD) is an effective method to prevent traffic accidents. The existing research on DFD using facial features is an effective and noninvasive fatigue detection method. However, this approach is affected by facial occlusions (glasses, sunglasses, masks, etc.) and the large facial pose deformations in the extraction of effective fatigue features. In this paper, we introduce a novel DFD method using human pose information entropy. The method first estimates human pose from video sequences and then uses them as clues to extract multiple fatigue-related features which can reduce the influence of facial occlusion and head pose deformation. Information entropy and sliding window algorithm are applied to analyse and calculate sufficient consecutive video frames to obtain more robust and accurate fatigue-related values than by using a single frame. These information entropy values are combined resorting to the support vector machine (SVM) to recognize the driver fatigue state. Experimental results show that the method can achieve much higher accuracy and robustness, and the detection speed meets the requirements of real time.

The effect of facial occlusion on facial impressions of trustworthiness and dominance

Recognizing the role that facial appearance plays in guiding social interactions, here we investigated how occlusions of the bottom-face region affect facial impressions of trustworthiness and dominance. Previous studies suggesting that different facial features impact inferences on these traits sustain the hypothesis that wearing a face mask will differently affect each trait inference. And specifically, that trustworthiness impressions will be more disrupted by this type of face occlusion than dominance impressions. In two studies, we addressed this possibility by occluding the bottom face region of faces that were previously shown to convey different levels of dominance and trustworthiness, and tested differences in the ability to discriminate between these trait levels across occlusion conditions. In Study 1 faces were occluded by a mask, and in Study 2 by a square image. In both studies, results showed that although facial occlusions generally reduced participants’ confidence on their trait judgments, the ability to discriminate facial trustworthiness was more strongly affected than the ability to discriminate facial dominance. Practical and theoretical implications of these findings are discussed.

Face occlusion removal for face recognition using the related face by structural similarity index measure and principal component analysis

Facial occlusions like sunglasses, masks, caps etc. have severe consequences when reconstructing the partially occluded regions of a facial image. This paper proposes a novel hybrid machine learning approach for occlusion removal based on Structural Similarity Index Measure (SSIM) and Principal Component Analysis (PCA), called SSIM_PCA. The proposed system comprises two stages. In the first stage, a Face Similar Matrix (FSM) guided by the Structural Similarity Index Measure is generated to provide the necessary information to recover from the lost regions of the face image. The FSM generates Related Face (RF) images similar to the probe image. In the second stage, these RF images are considered as related information and used as input data to generate eigenspaces using PCA to reconstruct the occluded face region exploiting the relationship between the occluded region and related face images, which contain relevant data to recover from the occluded area. Experimental results with five standard datasets viz. Caspeal-R1, IMFDB, and FEI have proven that the proposed method works well under illumination changes and occlusion of facial images.

Collaborative Attention Transformer on facial expression recognition under partial occlusion

Recent research on facial expression recognition (FER) in the wild shows challenges still remain. Different from laboratory-controlled expression in the past, images in the wild contain more uncertainties, such as different forms of face information occlusion, ambiguous facial images, noisy labels, and so on. Among them, real-world facial occlusion is the most general and crucial challenge for FER. In addition, because of the COVID-19 disease epidemic, people have to wear masks in public, which brings new challenges to FER tasks. Due to the recent success of the Transformer on numerous computer vision tasks, we propose a Collaborative Attention Transformer (CAT) network that first uses Cross-Shaped Window Transformer as the backbone for the FER task. Meanwhile, two attention modules are collaborated. Channel-Spatial Attention Module is designed to increase the attention of the network to global features. Moreover, Window Attention Gate is used to enhance the ability of the model to focus on local details. The proposed method is evaluated on two public in-the-wild facial expression datasets, RAF-DB and FERPlus, and the results demonstrate that our CAT performs superior to the state-of-the-art methods.

A new occluded face recognition framework with combination of both Deocclusion and feature filtering methods.

Face recognition plays the significant role in many human-computer interaction decvices and applications, whose access control systems are based on the verification of face biometrical features. Though great improvement in the recognition performances have been achieved, when under some specific conditions like faces with occlusions, the performance would suffer a severe drop. Occlusion is one of the most significant reasons for the performance degrade of the existing general face recognition systems. The biggest problem in occluded face recognition (OFR) lies in the lack of the occluded face data. To mitigate this problem, this paper has proposed one new OFR network DOMG-OFR (Dynamic Occlusion Mask Generator based Occluded Face Recognition), which keeps trying to generate the most informative occluded face training samples on feature level dynamically, in this way, the recognition model would always be fed with the most valuable training samples so as to save the labor in preparing the synthetic data while simultaneously improving the training efficiency. Besides, this paper also proposes one new module called Decision Module (DM) in an attempt to combine both the merits of the two mainstream methodologies in OFR which are face image reconstruction based methodologies and the face feature filtering based methodologies. Furthermore, to enable the existing face deocclusion methods that mostly target at near frontal faces to work well on faces under large poses, one head pose aware deocclusion pipeline based on the Condition Generative Adversarial Network (CGAN) is proposed. In the experimental parts, we have also investigated the effects of the occlusions upon face recognition performance, and the validity and the efficiency of our proposed Decision based OFR pipeline has been fully proved. Through comparing both the verification and the recognition performance upon both the real occluded face datasets and the synthetic occluded face datasets with other existing works, our proposed OFR architecture has demonstrated obvious advantages over other works.

Fast Facial Landmark Detection and Applications: A Survey

&#x0D; Dense facial landmark detection is a key element of face processing pipeline. In this paper we survey and analyze modern neural-network-based facial landmark detection algorithms, focus on approaches that have led to a significant increase in quality over the past few years on datasets with large pose and emotion variability, significant face occlusion, all of which are typical in real-world scenarios. We summarize the improvements into categories, provide quality comparison on difficult and modern in-the-wild datasets: 300 Faces in-the-wild (300W), Annotated Facial Landmarks in-the-wild (AFLW), Wider Facial Landmarks in-the-wild (WFLW), Caltech Occluded Faces in-the-wild (COFW). Additionally, we compare algorithm speed on desktop central and graphical processors, mobile devices. For completeness, we also briefly touch on established methods with open implementations available. Besides, we cover applications and vulnerabilities of the landmark detection algorithms. We hope that generalizations that we make will lead to further algorithm improvements.&#x0D;