Face Alignment Research Articles

A Review of Facial Expression Recognition Based on Convolutional Neural Networks

Abstract. The intelligent processing of human facial expressions has gained popularity due to the growth of big data and the expanding knowledge of AI and machine learning. In this paper, the author reviews the current mainstream methods for face recognition using convolutional neural network (CNN) models in deep learning, providing insights and future directions. By collecting and analyzing research findings, the characteristics, strengths and weaknesses of each model are discussed. The results indicate that CNN-based face recognition systems typically involve several steps: face detection, face alignment, face representation, face matching, and post-processing. By examining expression recognition with models like LeNet-5, AlexNet, VGGNet, and GoogleNet, the author concludes that CNNs possess features such as being data-driven, capable of feature learning, multi-level structured, adaptable to different domains, capable of real-time processing, and supporting multi-task learning. This technology has made significant strides and offers considerable potential for further development.

VoiceStyle: Voice-Based Face Generation via Cross-Modal Prototype Contrastive Learning

Can we predict a person’s appearance solely based on their voice? This article explores this question by focusing on generating a face from an unheard voice segment. Our proposed method, VoiceStyle, combines cross-modal representation learning with generation modeling, enabling us to incorporate voice semantic cues into the generated face. In the first stage, we introduce cross-modal prototype contrastive (CMPC) learning to establish the association between voice and face. Recognizing the presence of false negative and deviate positive instances in real-world unlabeled data, we not only use voice–face pairs in the same video but also construct additional semantic positive pairs through unsupervised clustering, enhancing the learning process. Moreover, we recalibrate instances based on their similarity to cluster centers in the other modality. In the second stage, we harness the powerful generative capabilities of StyleGAN to produce faces. We optimize the latent code in StyleGAN’s latent space, guided by the learned voice–face alignment. To address the importance of selecting an appropriate starting point for optimization, we aim to automatically find an optimal starting point by utilizing the face prototype derived from the voice input. The entire pipeline can be implemented in a self-supervised manner, eliminating the need for manually labeled annotations. Through extensive experiments, we demonstrate the effectiveness and performance of our VoiceStyle method in both cross-modal representation learning and voice-based face generation.

GANs-generated synthetic datasets for face alignment algorithms in complex environments

Face alignment has matured over the past several decades, but privacy violations or data abuse have also triggered global controversy. Moreover, existing face algorithms are still challenging in complex environments. For the question: ”Can synthetic datasets introduce novel variations in real-world data?”. We proposed a new research direction concerning key point detection tasks utilizing synthetic datasets, aiming to reduce the model’s reliance on real-world datasets. Considering the differences between synthetic and real-world data, our work proposed two different transfer ways based on GANs: (1) S→R model converts the synthetic face images generated by the Face middleware 3D model (FaceGen) into more realistic face images for training face alignment. (2) R→S model converts the real-world face images into a synthetic style image for testing face alignment. Extensive experiments explored the synthetic data complementarity and availability.

Design and Optimization Analysis of an Adaptive Knee Exoskeleton

To solve the problem of undesired relative motion of human-machine interaction positions caused by misalignment of the human-machine joints rotation axis of the knee exoskeleton, this study designed an adaptive knee exoskeleton based on a gear-link mechanism (GLM) by considering the human body as a component of the exoskeleton mechanism. Simultaneously, the concept of the wearable area (WA) was proposed, which transformed the operation of aligning the exoskeleton rotation axis with the human knee joint rotation axis into a "face alignment point" in the sagittal plane, reducing the difficulty of aligning the human-machine joint rotation axis. In the kinematic analysis of GLM, the phenomenon of instantaneous movement of the central axis of the human knee joint was considered. Based on the kinematic model, the WA, velocity transfer ratio, and initial position static stiffness of GLM were analyzed. The NSGA-II optimization algorithm was used to optimize the size parameters of GLM, which increased the WA by 18.4%, the average velocity transfer ratio by 4.98%, and the average initial position static stiffness by 6.01%. Finally, the ability of the exoskeleton to absorb movement displacement (MD) was verified through simulation, and the good human-machine kinematic compatibility of the exoskeleton was verified through wearable tests conducted on the initial mechanism principle prototype.

Gender and Age Detection using Deep Learning

The demand for automatic gender and age prediction has surged with the proliferation of social media and online platforms. Despite advancements in related fields like facial recognition, the effectiveness of current technologies in real-world scenarios remains limited. This paper explores the application of deep convolutional neural networks (CNNs) to address this challenge. Our proposed method involves a five-step process: facial recognition, background removal, face alignment, application of multiple CNN models, and a voting mechanism to enhance prediction accuracy. We evaluate our approach using the recent Audience-Face benchmark dataset, focusing on gender detection and age estimation. The implementation is carried out using Python. Keywords: Convolutional neural networks (CNN), Deep learning, Facial recognition, Computer vision, Region of interest (ROI).

Learning facial structural dependency in 3D aligned space for face alignment

Facial structure's statistical characteristics offer pivotal prior information in facial landmark prediction, forming inter-dependencies among different landmarks. Such inter-dependencies ensure that predictions adhere to the shape distribution typical of natural faces. In challenging scenarios like occlusions or extreme facial poses, this structure becomes indispensable, which can help to predict elusive landmarks based on more discernible ones. While current deep learning methods do capture these landmark dependencies, it's often an implicit process heavily reliant on vast training datasets. We contest that such implicit modeling approaches fail to manage more challenging situations. In this paper, we propose a new method that harnesses the facial structure and explicitly explores inter-dependencies among facial landmarks in an end-to-end fashion. We propose a Structural Dependency Learning Module (SDLM). It uses 3D face information to map facial features into a canonical UV space, in which the facial structure is explicitly 3D semantically aligned. Besides, to explore the global relationships between facial landmarks, we take advantage of the self-attention mechanism in the image and UV spaces. We name the proposed method Facial Structure-based Face Alignment (FSFA). FSFA reinforces the landmark structure, especially under challenging conditions. Extensive experiments demonstrate that FSFA achieves state-of-the-art performance on the WFLW, 300W, AFLW, and COFW68 datasets.

Precise facial landmark detection by Dynamic Semantic Aggregation Transformer

At present, deep neural network methods have played a dominant role in face alignment field. However, they generally use predefined network structures to predict landmarks, which tends to learn general features and leads to mediocre performance, e.g., they perform well on neutral samples but struggle with faces exhibiting large poses or occlusions. Moreover, they cannot effectively deal with semantic gaps and ambiguities among features at different scales, which may hinder them from learning efficient features. To address the above issues, in this paper, we propose a Dynamic Semantic-Aggregation Transformer (DSAT) for more discriminative and representative feature (i.e., specialized feature) learning. Specifically, a Dynamic Semantic-Aware (DSA) model is first proposed to partition samples into subsets and activate the specific pathways for them by estimating the semantic correlations of feature channels, making it possible to learn specialized features from each subset. Then, a novel Dynamic Semantic Specialization (DSS) model is designed to mine the homogeneous information from features at different scales for eliminating the semantic gap and ambiguities and enhancing the representation ability. Finally, by integrating the DSA model and DSS model into our proposed DSAT in both dynamic architecture and dynamic parameter manners, more specialized features can be learned for achieving more precise face alignment. It is interesting to show that harder samples can be handled by activating more feature channels. Extensive experiments on popular face alignment datasets demonstrate that our proposed DSAT outperforms state-of-the-art models in the literature. Our code is available at https://github.com/GERMINO-LiuHe/DSAT.

A smartphone application for personalized facial aesthetic monitoring.

Methods available at home for capturing facial images to track changes in skin quality and evaluate skincare treatments are limited. In this study, we developed a smartphone camera application (app) for personalized facial aesthetic monitoring. A face alignment indicators (FAIN) system utilizing facial landmark detection, an artificial intelligence technique, to estimate key facial parts, was implemented into the app to maintain a consistent facial appearance during image capture. The FAIN system is composed of a fixed target indicator and an alignment indicator that dynamically changes its shape according to the user's face position, size, and orientation. Users align their faces to match the alignment indicator with the fixed target indicator, and the image is automatically captured when alignment is achieved. We investigated the app's effectiveness in ensuring a consistent facial appearance by analyzing both geometric and colorimetric data. Geometric information from captured faces and colorimetric data from stickers applied to the faces were utilized. The coefficients of variation (CVs) for the L*, a*, and b* values of the stickers were higher compared to those measured by a colorimeter, with CVs of 14.9 times, 8.14 times, and 4.41 times for L*, a*, and b*, respectively. To assess the feasibility of the app for facial aesthetic monitoring, we tracked changes in pseudo-skin color on the cheek of a participant using skin-colored stickers. As a result, we observed the smallest color difference ∆Eab of 1.901, which can be considered as the experimentally validated detection limit using images acquired by the app. While the current monitoring method is a relative quantification approach, it contributes to evidence-based evaluations of skincare treatments.

Age and Gender voice Recognition using Deep learning

Since the advent of social media, there has been an increased interest in automatic age and gender classification through facial images. So, the process of age and gender classification is a crucial stage for many applications such as face verification, aging analysis, ad targeting and targeting of interest groups. Yet most age and gender classification systems still have some problems in real-world applications. This work involves an approach to age and gender classification using multiple convolutional neural networks (CNN). The proposed method has 5 phases as follows: face detection, remove background, face alignment, multiple CNN and voting systems. The multiple CNN model consists of three different CNN in structure and depth; the goal of this difference It is to extract various features for each network. Each network is trained separately on the AGFW dataset, and then we use the Voting system to combine predictions to get the result.

LDDMM-Face: Large deformation diffeomorphic metric learning for cross-annotation face alignment

We propose an innovative, flexible, and consistent cross-annotation face alignment framework, LDDMM-Face, the key contribution of which is a deformation layer that naturally embeds facial geometry in a diffeomorphic way. This enables and solves cross-annotation face alignment tasks that were impossible in the existing works. Instead of predicting facial landmarks via a heatmap or coordinate regression, we formulate the face alignment task in a diffeomorphic registration manner and predict momenta that uniquely parameterize the deformation between the initial boundary and true boundary. We then perform large deformation diffeomorphic metric mapping (LDDMM) simultaneously for curve and landmark to localize the facial landmarks. The novel embedding of LDDMM into a deep network allows LDDMM-Face to consistently annotate facial landmarks without ambiguity and flexibly handle various annotation schemes, and can even predict dense annotations from sparse ones. To the best of our knowledge, this is the first study to leverage learning-based diffeomorphic mapping for face alignment. Our method can be easily integrated into various face alignment networks. We extensively evaluate LDDMM-Face on five benchmark datasets: 300W, WFLW, HELEN, COFW-68, and AFLW. LDDMM-Face distinguishes itself with outstanding performance when dealing with within-dataset cross-annotation learning (sparse-to-dense) and cross-dataset learning (different training and testing datasets). In addition, LDDMM-Face shows promising results on the most challenging task of cross-dataset cross-annotation learning (different training and testing datasets with different annotations). Our codes are available at https://github.com/CRazorback/LDDMM-Face.

Dynamics of manual impaction instruments during total hip arthroplasty.

Manual impaction, with a mallet and introducer, remains the standard method of installing cementless acetabular cups during total hip arthroplasty (THA). This study aims to quantify the accuracy and precision of manual impaction strikes during the seating of an acetabular component. This understanding aims to help improve impaction surgical techniques and inform the development of future technologies. Posterior approach THAs were carried out on three cadavers by an expert orthopaedic surgeon. An instrumented mallet and introducer were used to insert cementless acetabular cups. The motion of the mallet, relative to the introducer, was analyzed for a total of 110 strikes split into low-, medium-, and high-effort strikes. Three parameters were extracted from these data: strike vector, strike offset, and mallet face alignment. The force vector of the mallet strike, relative to the introducer axis, was misaligned by an average of 18.1°, resulting in an average wasted strike energy of 6.1%. Furthermore, the mean strike offset was 19.8 mm from the centre of the introducer axis and the mallet face, relative to the introducer strike face, was misaligned by a mean angle of 15.2° from the introducer strike face. The direction of the impact vector in manual impaction lacks both accuracy and precision. There is an opportunity to improve this through more advanced impaction instruments or surgical training.

Are 3D Face Shapes Expressive Enough for Recognising Continuous Emotions and Action Unit Intensities?

Recognising continuous emotions and action unit (AU) intensities from face videos, requires a spatial and temporal understanding of expression dynamics. Existing works primarily rely on 2D face appearance features to extract such dynamics. This work focuses on a promising alternative based on parametric 3D face alignment models, which disentangle different factors of variation, including expression-induced shape variations. We aim to understand how expressive 3D face shapes are in estimating valence-arousal and AU intensities compared to the state-of-the-art 2D appearance-based models. We benchmark five recent 3D face models: ExpNet, 3DDFA-V2, RingNet, DECA, and EMOCA. In valence-arousal estimation, expression features of 3D face models consistently surpassed previous works and yielded an average concordance correlation of. 745 and. 574 on SEWA and AVEC 2019 CES corpora, respectively. We also study how 3D face shapes performed on AU intensity estimation on BP4D and DISFA datasets, and report that 3D face features were on par with 2D appearance features in recognising AUs 4, 6, 10, 12, and 25, but not the entire set of AUs. To understand this discrepancy, we conduct a correspondence analysis between valence-arousal and AUs, which points out that accurate prediction of valence-arousal may require the knowledge of only a few AUs.

Structural Dependence Learning Based on Self-attention for Face Alignment

Structural Dependence Learning Based on Self-attention for Face Alignment

Refined dense face alignment through image matching

Refined dense face alignment through image matching

One level deep convolutional neural network for facial key points detection

Facial landmark detection has a lot of applications in face recognition, face alignment, facial expression recognition, video surveillance and security systems. In the existing literature, there are multiple methods utilizing convolutional neural networks (CNNs) that address this problem in various ways. In many cases, the models use a tree-like structure of CNNs to achieve better results. This paper proposes a combination of three parallel deep convolutional neural networks (DCNNs) to estimate the accurate localization of each keypoint. The first one focuses on the whole face to outperform five points, including the eyes, nose, and mouth corners. The second one focuses on the eyes-nose parts to outperform three points, specifically the eyes and nose. The last one focuses on the nose-mouth parts to outperform three points, namely the nose and mouth corners. Further, we combine all outputs of the three DCNNs and take the average value of each detected key point as the final output. In the first step, we improvthe the parameter efficiency and accuracy of each DCNNs through a set of experiments using the labeled face parts in-the-wild database (LFPW) and the helen facial feature dataset (Helen). Then, we demonstrate that our approach yields more accurate estimations of facial key points than two state-of-the-art methods and commercial software in terms of accuracy.

An approach for handwritten Chinese text recognition unifying character segmentation and recognition

Text line recognition methods can be categorized into explicit segmentation based and implicit segmentation based ones. Explicit segmentation based methods require character-level annotation during training, while implicit segmentation based methods, trained on line-level annotated data, face alignment drift challenges. Though some methods have been proposed to address these challenges using weakly supervised object detection, they often rely on cumbersome pseudo-box generation processes and complex decoding. In this paper, we propose a unified framework to overcome these challenges, achieving high accuracy in text recognition and character segmentation. To eliminate the need of character-level annotated real text line data in training, we introduce a novel training paradigm that utilizes character-level annotated synthetic data and line-level annotated real data jointly. For synthetic data, candidate characters are explicitly aligned with labeled characters to generate hard labels for supervising model training. For real data, implicit alignment is produced by Connectionist Temporal Classification (CTC) mapping to provide soft labels for weakly-supervised model training. And for inference, we propose two decoding strategies leveraging the advantages of Non-Maximum Suppression (NMS) and CTC decoding. Extensive experiments on benchmark datasets demonstrate the superior performance of our method in text recognition and character localization, even with minimal amounts of character-level annotated line data.

Research on facial recognition system based on deep learning

This essay explores the integration of deep learning techniques in facial recognition, delving into the advancements, challenges, and prospects in future of this transformative technology. Deep learning, a subset of artificial intelligence, has revolutionized facial recognition by enabling automatic feature learning and robust representations from raw facial data. This paper examines the key components of deep learning in facial recognition, including feature extraction, facial alignment, and face recognition. It highlights the advantages of using deep learning over traditional methods by analysing related research. It also addresses the challenges deep learning faces in facial recognition, such as data bias, privacy concerns, and adversarial attacks. The essay also explores the promising future of deep learning in facial recognition, with ongoing research aiming to improve robustness, fairness, and privacy preservation.

Acoustic and visual geometry descriptor for multi-modal emotion recognition fromvideos

<span>Recognizing human emotions simultaneously from multiple data modalities (e.g., face, and speech) has drawn significant research interest, and numerous research contributions have been investigated in the affective computing community. However, most methods concentrate less on facial alignment and keyframe selection for audio-visual input. Hence, this paper proposed a new audio-visual descriptor, mainly concentrating on describing the emotion through only a few frames. For this purpose, we proposed a new self-similarity distance matrix (SSDM), which computes the spatial, and temporal distances through landmark points on the facial image. The audio signal is described through an asset of composite features, including statistical features, spectral features, formant frequencies, and energies. A support vector machine (SVM) algorithm is employed to classify both models, and the final results are fused to predict the emotion. Surrey audio-visual expressed emotion (SAVEE) and Ryerson multimedia research lab (RML) datasets are utilized for experimental validation, and the proposed method has shown significant improvement from the state of art methods.</span>

Computationally intelligent real-time security surveillance system in the education sector using deep learning.

Real-time security surveillance and identity matching using face detection and recognition are central research areas within computer vision. The classical facial detection techniques include Haar-like, MTCNN, AdaBoost, and others. These techniques employ template matching and geometric facial features for detecting faces, striving for a balance between detection time and accuracy. To address this issue, the current research presents an enhanced FaceNet network. The RetinaFace is employed to perform expeditious face detection and alignment. Subsequently, FaceNet, with an improved loss function is used to achieve face verification and recognition with high accuracy. The presented work involves a comparative evaluation of the proposed network framework against both traditional and deep learning techniques in terms of face detection and recognition performance. The experimental findings demonstrate that an enhanced FaceNet can successfully meet the real-time facial recognition requirements, and the accuracy of face recognition is 99.86% which fulfills the actual requirement. Consequently, the proposed solution holds significant potential for applications in face detection and recognition within the education sector for real-time security surveillance.

IMetaTown: A Metaverse System with Multiple Interactive Functions Based on Virtual Reality.

This work aims to pioneer the development of a real-time interactive and immersive Metaverse Human-Computer Interaction (HCI) system leveraging Virtual Reality (VR). The system incorporates a three-dimensional (3D) face reconstruction method, grounded in weakly supervised learning, to enhance player-player interactions within the Metaverse. The proposed method, two-dimensional (2D) face images, are effectively employed in a 2D Self-Supervised Learning (2DASL) approach, significantly optimizing 3D model learning outcomes and improving the quality of 3D face alignment in HCI systems. The work outlines the functional modules of the system, encompassing user interactions such as hugs and handshakes and communication through voice and text via blockchain. Solutions for managing multiple simultaneous online users are presented. Performance evaluation of the HCI system in a 3D reconstruction scene indicates that the 2DASL face reconstruction method achieves noteworthy results, enhancing the system's interaction capabilities by aiding 3D face modeling through 2D face images. The experimental system achieves a maximum processing speed of 18 frames of image data on a personal computer, meeting real-time processing requirements. User feedback regarding social acceptance, action interaction usability, emotions, and satisfaction with the VR interactive system reveals consistently high scores. The designed VR HCI system exhibits outstanding performance across diverse applications.