Local Facial Features Research Articles

A Lightweight Model Enhancing Facial Expression Recognition with Spatial Bias and Cosine-Harmony Loss

This paper proposes a facial expression recognition network called the Lightweight Facial Network with Spatial Bias (LFNSB). The LFNSB model effectively balances model complexity and recognition accuracy. It has two key components: a lightweight feature extraction network (LFN) and a Spatial Bias (SB) module for aggregating global information. The LFN introduces combined channel operations and depth-wise convolution techniques, effectively reducing the number of parameters while enhancing feature representation capability. The Spatial Bias module enables the model to focus on local facial features while capturing the dependencies between different facial regions. Additionally, a new loss function called Cosine-Harmony Loss is designed. This function optimizes the relative positions of feature vectors in high-dimensional space, resulting in better feature separation and clustering. Experimental results on the AffectNet and RAF-DB datasets demonstrate that the LFNSB model achieves competitive recognition accuracy, with 63.12% accuracy on AffectNet-8, 66.57% accuracy on AffectNet-7, and 91.07% accuracy on RAF-DB, while significantly reducing the model complexity.

A multi-aware graph convolutional network for driver drowsiness detection

Driver drowsiness is a leading cause of traffic accidents. Utilizing deep neural networks, facial feature-based methods have achieved promising results in drowsiness detection. However, these methods suffer from two limitations. Firstly, they only focus on features from one or two facial regions, thus overlooking the relationships between features across different facial regions. Secondly, these methods struggle to account for individual driver variability, a common phenomenon where drivers may display dissimilar signs of drowsiness. These limitations lead to inaccurate drowsiness detection. To address these issues, in this paper we propose a multi-aware graph convolutional network (MAGCN). At the heart of MAGCN are two feature extractors: the class- and attention-aware extractor (CAAE), and the composite temporal-aware extractor (CTAE). The CAAE explores interdependencies within global and local facial features, while the CTAE leverages temporal information to capture dynamic changes in features. Moreover, a task-oriented graph convolutional network is designed to refine the drowsiness feature space for precise detection. Experiment results show that the proposed MAGCN exhibits competitive detection performance, when compared with state-of-the-art drowsiness detection approaches on two public datasets. In summary, the proposed model not only learns and analyzes correlations between features from various facial regions, but also tackles individual driver variability.

POST: Prototype‐oriented similarity transfer framework for cross‐domain facial expression recognition

AbstractFacial expression recognition (FER) is one of the popular research topics in computer vision. Most deep learning expression recognition methods perform well on a single dataset, but may struggle in cross‐domain FER applications when applied to different datasets. FER under cross‐dataset also suffers from difficulties such as feature distribution deviation and discriminator degradation. To address these issues, we propose a prototype‐oriented similarity transfer framework (POST) for cross‐domain FER. The bidirectional cross‐attention Swin Transformer (BCS Transformer) module is designed to aggregate local facial feature similarities across different domains, enabling the extraction of relevant cross‐domain features. The dual learnable category prototypes is designed to represent potential space samples for both source and target domains, ensuring enhanced domain alignment by leveraging both cross‐domain and specific domain features. We further introduce the self‐training resampling (STR) strategy to enhance similarity transfer. The experimental results with the RAF‐DB dataset as the source domain and the CK+, FER2013, JAFFE and SFEW 2.0 datasets as the target domains, show that our approach achieves much higher performance than the state‐of‐the‐art cross‐domain FER methods.

SwinDPSR: Dual-Path Face Super-Resolution Network Integrating Swin Transformer

Whether to use face priors in the face super-resolution (FSR) methods is a symmetry problem.Various face priors are used to describe the overall and local face features, making the generation of super-resolution face images expensive and laborious. FSR methods that do not require any prior information tend to focus too much on the local features of the face, ignoring the modeling of global information. To solve this problem, we propose a dual-path facial image super-resolution network (SwinDPSR) fused with Swin Transformer. The network does not require additional face priors, and it learns global face shape and local face components through two independent branches. In addition, the channel attention ECA module is used to aggregate the global and local face information in the above dual-path sub-networks, which can generate corresponding high-quality face images. The results of face super-resolution reconstruction experiments on public face datasets and a real-scene face dataset show that SwinDPSR is superior to previous advanced methods both in terms of visual effects and objective indicators. The reconstruction results are evaluated with four evaluation metrics: peak signal-to-noise ratio (PSNR), structural similarity (SSIM), learned perceptual image patch similarity (LPIPS), and mean perceptual score (MPS).

G2L-CariGAN: Caricature Generation from Global Structure to Local Features

Existing GAN-based approaches to caricature generation mainly focus on exaggerating a character’s global facial structure. This often leads to the failure in highlighting significant facial features such as big eyes and hook nose. To address this limitation, we propose a new approach termed as G2L-CariGAN, which uses feature maps of spatial dimensions instead of latent codes for geometric exaggeration. G2L-CariGAN first exaggerates the global facial structure of the character on a low-dimensional feature map and then exaggerates its local facial features on a high-dimensional feature map. Moreover, we develop a caricature identity loss function based on feature maps, which well retains the character's identity after exaggeration. Our experiments have demonstrated that G2L-CariGAN outperforms the state-of-arts in terms of the quality of exaggerating a character and retaining its identity.

Gaze-Swin: Enhancing Gaze Estimation with a Hybrid CNN-Transformer Network and Dropkey Mechanism

Gaze estimation, which seeks to reveal where a person is looking, provides a crucial clue for understanding human intentions and behaviors. Recently, Visual Transformer has achieved promising results in gaze estimation. However, dividing facial images into patches compromises the integrity of the image structure, which limits the inference performance. To tackle this challenge, we present Gaze-Swin, an end-to-end gaze estimation model formed with a dual-branch CNN-Transformer architecture. In Gaze-Swin, we adopt the Swin Transformer as the backbone network due to its effectiveness in handling long-range dependencies and extracting global features. Additionally, we incorporate a convolutional neural network as an auxiliary branch to capture local facial features and intricate texture details. To further enhance robustness and address overfitting issues in gaze estimation, we replace the original self-attention in the Transformer branch with Dropkey Assisted Attention (DA-Attention). In particular, this DA-Attention treats keys in the Transformer block as Dropout units and employs a decay Dropout rate schedule to preserve crucial gaze representations in deeper layers. Comprehensive experiments on three benchmark datasets demonstrate the superior performance of our method in comparison to the state of the art.

Real-Time Noise Suppression Method Using Deep Learning Algorithm

In this study, we present an innovative approach to real-time facial feature detection utilizing the MTCNN (MultiTask Cascaded Neural Network) deep learning architecture. Unlike traditional methods, our novel framework combines advanced techniques to achieve unparalleled precision and efficiency in facial feature localization. Through a meticulous exploration of MTCNN's capabilities, we unveil a transformative methodology that significantly enhances the speed and accuracy of real-time facial detection. Our research focuses on pushing the boundaries of existing facial recognition technologies, introducing a fresh perspective on the application of MTCNN. By leveraging its unique architecture, we not only address the challenges associated with real-time detection but also enhance the overall robustness of the system. The proposed approach showcases the untapped potential of MTCNN, establishing it as a key player in the realm of facial feature detection. Through rigorous experimentation and evaluation, we demonstrate the superiority of our approach over conventional methods, highlighting its effectiveness in diverse scenarios. This work contributes to the ongoing evolution of deep learning applications in computer vision, with implications for security, surveillance, and various human-computer interaction domains. Our findings open new avenues for researchers and practitioners seeking cutting-edge solutions in the dynamic field of real-time facial feature detection.

Face Tracking and Recognition with the Use of Particle-Filtered Local Features

A consistent particle filtering-based framework for the purpose of parallel face tracking and recognition from video sequences is proposed. A novel approach to defining randomized, particle filtering-driven local face features for the purpose of recognition is proposed. The potential of cumulating classification decisions based on the proposed feature set definition is evaluated. By applying cumulation mechanisms to the classification results determined from single frames and with the use of particle-filtered features, good recognition rates are obtained at the minimal computational cost. The proposed framework can operate in real-time on a typical modern PC. Additionally, the application of cumulation mechanisms makes the framework resistant to brief visual distortions, such as occlusions, head rotations or face expressions. A high performance is also obtained on low resolution images (video frames). Since the framework is based on the particle filtering principle, it is easily tunable to various application requirements (security level, hardware constraints).

Attention‐guided evolutionary attack with elastic‐net regularization on face recognition

In recent years, face recognition has achieved promising results along with the development of advanced Deep Neural Networks (DNNs). The existing face recognition systems are vulnerable to adversarial examples, which brings potential security risks. Evolutionary Attack (EA) has been successfully used to fool face recognition by inducing a minimum perturbation to a face image with few queries. However, EA employs the global information of face images but ignores their local characteristics. In addition, restricting the ℓ2-norm of adversarial perturbations hinders the diversity of adversarial perturbations. To solve the above problems, we propose Attention-guided Evolutionary Attack with Elastic-Net Regularization (ERAEA) for attacking face recognition. ERAEA extracts local facial characteristics by attention mechanism, effectively improving the attack effect and image perception quality. In particular, ERAEA adopts an attention mechanism to guide evolutionary direction, which operates on the covariance matrix as it contains crucial information about the evolutionary path. Furthermore, we design an adaptive elastic-net regularization to diversify the adversarial perturbation, accelerating the optimization performance. Extensive experiments obtained on three benchmarks demonstrate that our proposed method achieves better perturbation norm than the state-of-the-art methods with limited queries on face recognition and generates adversarial face images with higher perceptual quality. Besides, ERAEA requires fewer queries to achieve a fixed adversarial perturbation norm.

Local Corner and Motion Key Point Trajectory Extraction for Facial Forgery Identification

At present, the development of deep forgery technology has brought new challenges to media content forensics, and the use of deep forgery identification methods to identify forged audio and video has become a significant focus of research and difficulty. Deep forgery technology and forensic technology play a mutual game and promote each other’s development. This paper proposes a spatiotemporal local feature abstraction (STLFA) framework for facial forgery identification to solve the media industry challenges of deep forgery technology. To adequately utilize local facial features, we combine facial key points, key point movement, and facial corner points to detect forgery content. This paper establishes a spatiotemporal relation, which realizes face forgery detection by identifying abnormalities of facial keypoints and corner points for interframe judgments. Meanwhile, we utilize RNNs to predict the sequences from facial key point movement abnormalities and corner points for interframe. Experimental results show that our method achieves better performance than some existing methods and good anticompression forgery face detection performance on FF++.

Clinical accuracy of infrared temperature measurement devices: a comparison against non-invasive core-body temperature

During the coronavirus 2019 (COVID-19) pandemic, the implementation of non-contact infrared thermometry (NCIT) became an increasingly popular method of screening body temperature. However, data on the accuracy of these devices and the standardisation of their use are limited. In the current study, the body temperature of non-febrile volunteers was measured using infrared (IR) thermography, IR tympanic thermometry and IR gun thermometry at different facial feature locations and distances and compared with SpotOn core-body temperature. Poor agreement was found between all IR devices and SpotOn measurements (intra-class correlation coefficient <0.8). Bland-Alman analysis showed the narrowest limits of agreement with the IR gun at 3 cm from the forehead (bias = 0.19°C, limits of agreement (LOA): -0.58°C to 0.97°C) and widest with the IR gun at the nose (bias = 1.40°C, LOA: -1.15°C to 3.94°C). Thus, our findings challenge the established use of IR thermometry devices within hospital settings without adequate standard operating procedures to reduce operator error.

Integrating audio and visual modalities for multimodal personality trait recognition via hybrid deep learning.

Recently, personality trait recognition, which aims to identify people's first impression behavior data and analyze people's psychological characteristics, has been an interesting and active topic in psychology, affective neuroscience and artificial intelligence. To effectively take advantage of spatio-temporal cues in audio-visual modalities, this paper proposes a new method of multimodal personality trait recognition integrating audio-visual modalities based on a hybrid deep learning framework, which is comprised of convolutional neural networks (CNN), bi-directional long short-term memory network (Bi-LSTM), and the Transformer network. In particular, a pre-trained deep audio CNN model is used to learn high-level segment-level audio features. A pre-trained deep face CNN model is leveraged to separately learn high-level frame-level global scene features and local face features from each frame in dynamic video sequences. Then, these extracted deep audio-visual features are fed into a Bi-LSTM and a Transformer network to individually capture long-term temporal dependency, thereby producing the final global audio and visual features for downstream tasks. Finally, a linear regression method is employed to conduct the single audio-based and visual-based personality trait recognition tasks, followed by a decision-level fusion strategy used for producing the final Big-Five personality scores and interview scores. Experimental results on the public ChaLearn First Impression-V2 personality dataset show the effectiveness of our method, outperforming other used methods.

Local features drive identity responses in macaque anterior face patches

Humans and other primates recognize one another in part based on unique structural details of the face, including both local features and their spatial configuration within the head and body. Visual analysis of the face is supported by specialized regions of the primate cerebral cortex, which in macaques are commonly known as face patches. Here we ask whether the responses of neurons in anterior face patches, thought to encode face identity, are more strongly driven by local or holistic facial structure. We created stimuli consisting of recombinant photorealistic images of macaques, where we interchanged the eyes, mouth, head, and body between individuals. Unexpectedly, neurons in the anterior medial (AM) and anterior fundus (AF) face patches were predominantly tuned to local facial features, with minimal neural selectivity for feature combinations. These findings indicate that the high-level structural encoding of face identity rests upon populations of neurons specialized for local features.

LocalFace: Learning significant local features for deep face recognition

As a widely mentioned topic in face recognition, the margin-based loss function enhances the discriminability of face recognition models by applying margin between class decision boundaries. However, there is still room to improve the representation of face features. Local face feature extraction has been employed in traditional face recognition methods, but with the increase of network depth in deep learning, the traditional approach requires a large number of computational resources. In this paper, we propose a novel face recognition architecture called LocalFace to extract local face features. First, by analyzing the distribution of significant features in face images, we propose an efficient face fixed-point local feature extraction approach and improve this method to propose a more effective face dynamic local feature extraction scheme. Subsequently, we propose a block-based random occlusion method for the limitations of the random face occlusion method to better simulate the occlusion situation in real scenes. In the end, we present a detailed discussion on the channel attention method that is more appropriate for face recognition and classification tasks. Our method enhances the representation of face features by ensembling local features into global features without extra parameters, which is efficient and easy to implement. Extensive experiments on various benchmarks demonstrate the superiority of our LocalFace, and part of the experimental results achieve SOTA results.

Driving Detection Based on the Multifeature Fusion

In order to solve the problems of facial feature localization and driver fatigue state identification methods in driving fatigue detection, a driving detection method based on the multifeature fusion was proposed. This method uses a supervised descent algorithm to simultaneously locate multiple facial features of drivers. On the basis of blink, yawn and nod judgment, multiple characteristic values of blink frequency, yawn frequency, and nod frequency of drivers were extracted to establish a fatigue detection sample database, and a naive Bayes classifier was constructed to judge fatigue. When the driver appears fatigue driving, warning information is given in time in order to prevent traffic accidents. The experimental results show that two sample videos were selected for testing. The accuracy rate of video sample 1 and video sample 2 was 94.74% and 95.00%, respectively. Conclusion. In the actual driving environment video test results, the discriminant average accuracy of a driver fatigue state reaches 94.87%, which has a good performance.

Advanced Analysis of 3D Kinect Data: Supervised Classification of Facial Nerve Function via Parallel Convolutional Neural Networks

In this paper, we designed a methodology to classify facial nerve function after head and neck surgery. It is important to be able to observe the rehabilitation process objectively after a specific brain surgery, when patients are often affected by face palsy. The dataset that is used for classification problems in this study only contains 236 measurements of 127 patients of complex observations using the most commonly used House–Brackmann (HB) scale, which is based on the subjective opinion of the physician. Although there are several traditional evaluation methods for measuring facial paralysis, they still suffer from ignoring facial movement information. This plays an important role in the analysis of facial paralysis and limits the selection of useful facial features for the evaluation of facial paralysis. In this paper, we present a triple-path convolutional neural network (TPCNN) to evaluate the problem of mimetic muscle rehabilitation, which is observed by a Kinect stereovision camera. A system consisting of three modules for facial landmark measure computation and facial paralysis classification based on a parallel convolutional neural network structure is used to quantitatively assess the classification of facial nerve paralysis by considering facial features based on the region and the temporal variation of facial landmark sequences. The proposed deep network analyzes both the global and local facial movement features of a patient’s face. These extracted high-level representations are then fused for the final evaluation of facial paralysis. The experimental results have verified the better performance of TPCNN compared to state-of-the-art deep learning networks.

Real-Time Face Recognition and Detection Using Python

Abstract: While humans can recognize faces without much effort, facial recognition is a challenging pattern recognition problem in computing. Facial recognition systems attempt to identify a human face, which is three-dimensional and changes in appearance with lighting and facial expression, based on its two-dimensional image. To accomplish this computational task, facial recognition systems perform four steps. First face detection is used to segment the face from the image background. In the second step the segmented face image is aligned to account for face pose, image size and photographic properties, such as illumination and grayscale. The purpose of the alignment process is to enable the accurate localization of facial features in the third step, the facial nature extraction. Features such as eyes, nose and mouth are pinpointed and measured in the image to represent the face. The so established feature vector of the face is then, in the fourth step, matched against a database of faces. Keywords: face, detection, recognition, system, OpenCV

Part-facial relational and modality-style attention networks for heterogeneous face recognition

The purpose of heterogeneous face recognition (HFR) is to match face images of the same person from different modalities. Most HFR methods bridge the cross-modality variations with feature alignment by global feature representation learning, but ignore the content information of local features and modality-style information of face image for each modality, which limits the performance for HFR. The content information of local features not only contains the invariance of modality face features, but also can improve the stability of global face features, i.e., local features such as eyes, nose and mouth are steady and invariant. With this motivation, we propose a cross-modality dual-constraint (CMDC) approach that includes the part-facial relational attention network (PRAN) and modality-style attention network (MSAN). First, PRAN is designed to estimate the intrinsic structural relationships of local content features on each modality. It can extract discriminative local face features by capturing correlations within the face space of individual modality, and strengthen representations by contextual relationships across modalities. Secondly, we design the MSAN to capture the modality-style information for each modality, and then reduce the inter-modality differences by minimizing the distance of two modality-style features. Thirdly, to alleviate cross-modality variances and enhance intra-class compactness and inter-class divisibility, we propose the cross-modality dual-constrained loss (DCLoss) in the CMDC approach, which adds a global constraint to each sample distribution in the embedding space. Meanwhile, on the basis of focusing on modality-style information, DCLoss emphasizes the significance of category information. Extensive experiments on four datasets demonstrate the superior performance of our approach over the existing state-of-the-art. The code is available at https://github.com/JianYu777/CMDC.

Facial Expression Recognition With Deeply-Supervised Attention Network

Facial expression recognition (FER) is crucial for social communication. However, current studies present limitations when addressing facial expression difference due to demographic variation, such as race, gender, and age, etc. In this article, we first propose a deeply-supervised attention network (DSAN) to recognize human emotions based on facial images automatically. Based on DSAN, a two-stage training scheme is designed, taking full advantage of the race/gender/age-related information. In our DSAN framework, multi-scale features are leveraged to capture more discriminative information from the deep layers to the shallow layers. Furthermore, we adopt the attention block to highlight the essential local facial characteristics; it performs well when it is incorporated into the deeply-supervised framework. Finally, we combine the complementary characteristics of multiple convolutional layers in deeply-supervised manner and ensemble the intermediate predicted scores. Our experimental results have shown that our proposed framework can (i) effectively integrate demographic information in improving the performance of a variety of FER tasks, (ii) learn informative feature representations with a visual explanation by capturing the regions of interests (ROI), (iii) achieve superior performance for both the posed and the spontaneous FER databases, each containing pictures of human facial expressions varied in gender, age or race.

Video Face Detection Technology and Its Application in Health Information Management System

Computer face detection, as an early step and prerequisite for applications such as face recognition and face analysis, has attracted people's attention for a long time. With the popularization of computer applications, the improvement of performance, and the gradual maturity of research in the field of image processing and pattern recognition, face-related applications have become more and more a reality, so the research on face detection and positioning is also receiving more and more attention. Face detection and positioning are an important part of face analysis technology. Its goal is to search for the location of facial features (such as eyes, nose, mouth, and ears) in images or image sequences. It can be widely used in the fields of face tracking, face recognition, gesture recognition, facial expression recognition, head image compression and reconstruction, facial animation, etc. Based on the health information management system, this study mainly discusses the application of face recognition technology in video systems. Compared with other biological characteristics, such as fingerprints and eye masks, human faces are easier to obtain. In research and exploration, stable and effective face detection and face recognition algorithms have been proposed, which can achieve good recognition results even in real-time video surveillance. Aiming at the automatic face recognition technology in video surveillance, this study introduces in detail the video face detection technology in the health information management system of video image collection, image preprocessing, face detection, and face recognition. The prototype system of hygiene management is recognized.