Head Pose Changes Research Articles

Enhanced Hybrid Vision Transformer with Multi-Scale Feature Integration and Patch Dropping for Facial Expression Recognition.

Convolutional neural networks (CNNs) have made significant progress in the field of facial expression recognition (FER). However, due to challenges such as occlusion, lighting variations, and changes in head pose, facial expression recognition in real-world environments remains highly challenging. At the same time, methods solely based on CNN heavily rely on local spatial features, lack global information, and struggle to balance the relationship between computational complexity and recognition accuracy. Consequently, the CNN-based models still fall short in their ability to address FER adequately. To address these issues, we propose a lightweight facial expression recognition method based on a hybrid vision transformer. This method captures multi-scale facial features through an improved attention module, achieving richer feature integration, enhancing the network's perception of key facial expression regions, and improving feature extraction capabilities. Additionally, to further enhance the model's performance, we have designed the patch dropping (PD) module. This module aims to emulate the attention allocation mechanism of the human visual system for local features, guiding the network to focus on the most discriminative features, reducing the influence of irrelevant features, and intuitively lowering computational costs. Extensive experiments demonstrate that our approach significantly outperforms other methods, achieving an accuracy of 86.51% on RAF-DB and nearly 70% on FER2013, with a model size of only 3.64 MB. These results demonstrate that our method provides a new perspective for the field of facial expression recognition.

An effective cross-scenario remote heart rate estimation network based on global-local information and video transformer.

Remote photoplethysmography (rPPG) technology is a non-contact physiological signal measurement method, characterized by non-invasiveness and ease of use. It has broad application potential in medical health, human factors engineering, and other fields. However, current rPPG technology is highly susceptible to variations in lighting conditions, head pose changes, and partial occlusions, posing significant challenges for its widespread application. In order to improve the accuracy of remote heart rate estimation and enhance model generalization, we propose PulseFormer, a dual-path network based on transformer. By integrating local and global information and utilizing fast and slow paths, PulseFormer effectively captures the temporal variations of key regions and spatial variations of the global area, facilitating the extraction of rPPG feature information while mitigating the impact of background noise variations. Heart rate estimation results on the popular rPPG dataset show that PulseFormer achieves state-of-the-art performance on public datasets. Additionally, we establish a dataset containing facial expressions and synchronized physiological signals in driving scenarios and test the pre-trained model from the public dataset on this collected dataset. The results indicate that PulseFormer exhibits strong generalization capabilities across different data distributions in cross-scenario settings. Therefore, this model is applicable for heart rate estimation of individuals in various scenarios.

FSErasing: Improving Face Recognition with Data Augmentation Using Face Parsing

We propose original semantic labels for detailed face parsing to improve the accuracy of face recognition by focusing on parts in a face. The part labels used in conventional face parsing are defined based on biological features, and thus, one label is given to a large region, such as skin. Our semantic labels are defined by separating parts with large areas based on the structure of the face and considering the left and right sides for all parts to consider head pose changes, occlusion, and other factors. By utilizing the capability of assigning detailed part labels to face images, we propose a novel data augmentation method based on detailed face parsing called Face Semantic Erasing (FSErasing) to improve the performance of face recognition. FSErasing is to randomly mask a part of the face image based on the detailed part labels, and therefore, we can apply erasing‐type data augmentation to the face image that considers the characteristics of the face. Through experiments using public face image datasets, we demonstrate that FSErasing is effective for improving the performance of face recognition and face attribute estimation. In face recognition, adding FSErasing in training ResNet‐34 with Softmax using CelebA improves the average accuracy by 0.354 points and the average equal error rate (EER) by 0.312 points, and with ArcFace, the average accuracy and EER improve by 0.752 and 0.802 points, respectively. ResNet‐50 with Softmax using CASIA‐WebFace improves the average accuracy by 0.442 points and the average EER by 0.452 points, and with ArcFace, the average accuracy and EER improve by 0.228 points and 0.500 points, respectively. In face attribute estimation, adding FSErasing as a data augmentation method in training with CelebA improves the estimation accuracy by 0.54 points. We also apply our detailed face parsing model to visualize face recognition models and demonstrate its higher explainability than general visualization methods.

Point CNN:3D Face Recognition with Local Feature Descriptor and Feature Enhancement Mechanism.

Three-dimensional face recognition is an important part of the field of computer vision. Point clouds are widely used in the field of 3D vision due to the simple mathematical expression. However, the disorder of the points makes it difficult for them to have ordered indexes in convolutional neural networks. In addition, the point clouds lack detailed textures, which makes the facial features easily affected by expression or head pose changes. To solve the above problems, this paper constructs a new face recognition network, which mainly consists of two parts. The first part is a novel operator based on a local feature descriptor to realize the fine-grained features extraction and the permutation invariance of point clouds. The second part is a feature enhancement mechanism to enhance the discrimination of facial features. In order to verify the performance of our method, we conducted experiments on three public datasets: CASIA-3D, Bosphorus, and Lock3Dface. The results show that the accuracy of our method is improved by 0.7%, 0.4%, and 0.8% compared with the latest methods on these three datasets, respectively.

Facial Expression Recognition Based on Fine-Tuned Channel-Spatial Attention Transformer.

Facial expressions help individuals convey their emotions. In recent years, thanks to the development of computer vision technology, facial expression recognition (FER) has become a research hotspot and made remarkable progress. However, human faces in real-world environments are affected by various unfavorable factors, such as facial occlusion and head pose changes, which are seldom encountered in controlled laboratory settings. These factors often lead to a reduction in expression recognition accuracy. Inspired by the recent success of transformers in many computer vision tasks, we propose a model called the fine-tuned channel-spatial attention transformer (FT-CSAT) to improve the accuracy of recognition of FER in the wild. FT-CSAT consists of two crucial components: channel-spatial attention module and fine-tuning module. In the channel-spatial attention module, the feature map is input into the channel attention module and the spatial attention module sequentially. The final output feature map will effectively incorporate both channel information and spatial information. Consequently, the network becomes adept at focusing on relevant and meaningful features associated with facial expressions. To further improve the model's performance while controlling the number of excessive parameters, we employ a fine-tuning method. Extensive experimental results demonstrate that our FT-CSAT outperforms the state-of-the-art methods on two benchmark datasets: RAF-DB and FERPlus. The achieved recognition accuracy is 88.61% and 89.26%, respectively. Furthermore, to evaluate the robustness of FT-CSAT in the case of facial occlusion and head pose changes, we take tests on Occlusion-RAF-DB and Pose-RAF-DB data sets, and the results also show that the superior recognition performance of the proposed method under such conditions.

Siamese PointNet: 3D Head Pose Estimation with Local Feature Descriptor

Head pose estimation is an important part of the field of face analysis technology. It can be applied to driver attention monitoring, passenger monitoring, effective information screening, etc. However, illumination changes and partial occlusion interfere with the task, and due to the non-stationary characteristic of the head pose change process, normal regression networks are unable to achieve very accurate results on large-scale synthetic training data. To address the above problems, a Siamese network based on 3D point clouds was proposed, which adopts a share weight network with similar pose samples to constrain the regression process of the pose’s angles; meanwhile, a local feature descriptor was introduced to describe the local geometric features of the objects. In order to verify the performance of our method, we conducted experiments on two public datasets: the Biwi Kinect Head Pose dataset and Pandora. The results show that compared with the latest methods, our standard deviation was reduced by 0.4, and the mean error was reduced by 0.1; meanwhile, our network also maintained a good real-time performance.

Tracking of rigid head motion during MRI using anEEG system.

PurposeTo demonstrate a novel method for tracking of head movements during MRI using electroencephalography (EEG) hardware for recording signals induced by native imaging gradients.Theory and MethodsGradient switching during simultaneous EEG–fMRI induces distortions in EEG signals, which depend on subject head position and orientation. When EEG electrodes are interconnected with high‐impedance carbon wire loops, the induced voltages are linear combinations of the temporal gradient waveform derivatives. We introduce head tracking based on these signals (CapTrack) involving 3 steps: (1) phantom scanning is used to characterize the target sequence and a fast calibration sequence; (2) a linear relation between changes of induced signals and head pose is established using the calibration sequence; and (3) induced signals recorded during target sequence scanning are used for tracking and retrospective correction of head movement without prolonging the scan time of the target sequence. Performance of CapTrack is compared directly to interleaved navigators.ResultsHead‐pose tracking at 27.5 Hz during echo planar imaging (EPI) was demonstrated with close resemblance to rigid body alignment (mean absolute difference: [0.14 0.38 0.15]‐mm translation, [0.30 0.27 0.22]‐degree rotation). Retrospective correction of 3D gradient‐echo imaging shows an increase of average edge strength of 12%/−0.39% for instructed/uninstructed motion with CapTrack pose estimates, with a tracking interval of 1561 ms and high similarity to interleaved navigator estimates (mean absolute difference: [0.13 0.33 0.12] mm, [0.28 0.15 0.22] degrees).ConclusionMotion can be estimated from recordings of gradient switching with little or no sequence modification, optionally in real time at low computational burden and synchronized to image acquisition, using EEG equipment already found at many research institutions.

Head Pose Estimation in Complex Environment Based on Four-Branch Feature Selective Extraction and Regional Information Exchange Fusion Network

Under the severe situation of the COVID-19 pandemic, masks cover most of the effective facial features of users, and their head pose changes significantly in a complex environment, which makes the accuracy of head pose estimation in some systems such as safe driving systems and attention detection systems impossible to guarantee. To this end, we propose a powerful four-branch feature selective extraction network (FSEN) structure, in which three branches are used to extract three independent discriminative features of pose angles, and one branch is used to extract composite features corresponding to multiple pose angles. By reducing the dimension of high-dimensional features, our method significantly reduces the amount of computation while improving the estimation accuracy. Our convolution method is an improved spatial channel dynamic convolution (SCDC) that initially enhances the extracted features. Additionally, we embed a regional information exchange network (RIEN) after each convolutional layer in each branch to fully mine the potential semantic correlation between regions from multiple perspectives and learn and fuse this correlation to further enhance feature expression. Finally, we fuse the independent discriminative features of each pose angle and composite features from the three directions of channel, space, and pixel to obtain perfect feature expression for each pose angle, and then obtain the head pose angle. We conducted extensive experiments on the controlled environment datasets and a self-built real complex environment dataset (RCE) and the results showed that our method outperforms state-of-the-art single-modality methods and performs on par with multimodality-based methods. This shows that our network meets the requirements of accurate head-pose estimation in real complex environments such as complex illumination and partial occlusion.

Estimation of in-scanner head pose changes during structural MRI using a convolutional neural network trained on eye tracker video

IntroductionIn-scanner head motion is a common cause of reduced image quality in neuroimaging, and causes systematic brain-wide changes in cortical thickness and volumetric estimates derived from structural MRI scans. There are few widely available methods for measuring head motion during structural MRI. Here, we train a deep learning predictive model to estimate changes in head pose using video obtained from an in-scanner eye tracker during an EPI-BOLD acquisition with participants undertaking deliberate in-scanner head movements. The predictive model was used to estimate head pose changes during structural MRI scans, and correlated with cortical thickness and subcortical volume estimates. Methods21 healthy controls (age 32 ± 13 years, 11 female) were studied. Participants carried out a series of stereotyped prompted in-scanner head motions during acquisition of an EPI-BOLD sequence with simultaneous recording of eye tracker video. Motion-affected and motion-free whole brain T1-weighted MRI were also obtained. Image coregistration was used to estimate changes in head pose over the duration of the EPI-BOLD scan, and used to train a predictive model to estimate head pose changes from the video data. Model performance was quantified by assessing the coefficient of determination (R2). We evaluated the utility of our technique by assessing the relationship between video-based head pose changes during structural MRI and (i) vertex-wise cortical thickness and (ii) subcortical volume estimates. ResultsVideo-based head pose estimates were significantly correlated with ground truth head pose changes estimated from EPI-BOLD imaging in a hold-out dataset. We observed a general brain-wide overall reduction in cortical thickness with increased head motion, with some isolated regions showing increased cortical thickness estimates with increased motion. Subcortical volumes were generally reduced in motion affected scans. ConclusionsWe trained a predictive model to estimate changes in head pose during structural MRI scans using in-scanner eye tracker video. The method is independent of individual image acquisition parameters and does not require markers to be to be fixed to the patient, suggesting it may be well suited to clinical imaging and research environments. Head pose changes estimated using our approach can be used as covariates for morphometric image analyses to improve the neurobiological validity of structural imaging studies of brain development and disease.

A novel eye center localization method for multiview faces

Existing studies on eye center localization have mostly applied localization methods on faces viewed from frontal angles or faces with small yaw-rotation angles. This study proposed a novel eye center localization method that can effectively localize the eye centers under various situations on multiview faces with large yaw-rotation angles (from +67.5° to −67.5°). First, this study developed a multiview face detector that can be applied to large yaw angles and can flexibly detect and precisely capture the face region. The face detector can facilitate the generation of satisfactory results by a complete representation generative adversarial network (CR-GAN). Because a large yaw-rotation angle can substantially reduce the completeness of eye representation in an image or even cause the eyes to disappear within the face image, this study used a CR-GAN to produce frontal face images to solve the problem of incomplete representation in multiview face images. Furthermore, this study proposed a new iris-ripple filter to increase the accuracy and robustness of gradient localization. Finally, a new depth corresponding-points conversion method was proposed to automatically estimate the rotation variable between two faces and the conversion relationship between the eye centers. This method can effectively solve the problem of instability resulting from a CR-GAN during eye generation and can ensure localization accuracy for eyeballs with subtle changes. According to the experimental results, compared with other advanced methods, the proposed method exhibited higher accuracy and robustness in eye center localization when applied to images obtained from four databases that involved various challenging situations, including large yaw-rotation angle, illumination change, head pose change, gaze interaction, and complete occlusion of eyes.

A Novel Eye Center Localization Method for Head Poses With Large Rotations.

Eye localization is undoubtedly crucial to acquiring large amounts of information. It not only helps people improve their understanding of others but is also a technology that enables machines to better understand humans. Although studies have reported satisfactory accuracy for frontal faces or head poses at limited angles, large head rotations generate numerous defects (e.g., disappearance of the eye), and existing methods are not effective enough to accurately localize eye centers. Therefore, this study makes three contributions to address these limitations. First, we propose a novel complete representation (CR) pipeline that can flexibly learn and generate two complete representations, namely the CR-center and CR-region, of the same identity. We also propose two novel eye center localization methods. This first method employs geometric transformation to estimate the rotational difference between two faces and an unknown-localization strategy for accurate transformation of the CR-center. The second method is based on image translation learning and uses the CR-region to train the generative adversarial network, which can then accurately generate and localize eye centers. Five image databases are employed to verify the proposed methods, and tests reveal that compared with existing methods, the proposed method can more accurately and robustly localize eye centers in challenging images, such as those showing considerable head rotation (both yaw rotation of -67.5° to +67.5° and roll rotation of +120° to -120°), complete occlusion of both eyes, poor illumination in addition to head rotation, head pose changes in the dark, and various gaze interaction.

Hybrid Force Tracking Impedance Control-Based Autonomous Robotic System for Tooth Brushing Assistance of Disabled People

Due to the increase of the world ageing population and several forms of disabilities, assistive robots have received a lot of attention lately. Although, different assistive robotic systems have been developed to perform essential tasks around the head, these existing robotic systems are either manually controlled, shared controlled or interactive ones. Although, fully autonomous robotic systems can have the potential of eliminating several limitations of the existing systems, such systems are limited in design due to critical issues of increased complexities and the associated risks. In this article, a hybrid impedance controller-based autonomous assistive robotic system for tooth brushing assistance is developed. A fixed base manipulator with an attached automatic tooth brush and Kinect RGB-D sensor are proposed as a platform to achieve the brushing task. An online face detection, mouth detection, head pose estimation and trajectory planning for the brushing task are achieved. Additionally, online re-planning the trajectories in case of patients’ head pose changes is taken into account. The controller performance is evaluated regarding tracking the generated brushing trajectories and the desired force trajectory by co-simulating the manipulator’s dynamics in ADAMS and SIMULINK. The results demonstrate the capability of the proposed robotic system to achieve tooth brushing autonomously.

Thermal Face Recognition under Spatial Variation Conditions

This paper proposes a novel method for degraded thermal face recognition using Hu Li moments. The method deals with spatial variations of thermal images resulting from low resolution and changes in head pose. We describe thermal images with a set of scalar quantities that can capture its significant features at component level. Each thermal image is divided into components where the statistical features of these components are combined using a fusion method. The method finds a combination of multiple statistical patterns to produce an integrated result that is enhanced in terms of information content for correct classification. We show that local representations using Hu Li moments provide higher discriminability and offer robustness against variability due to spatial changes. To evaluate the performance of the proposed method, we conduct thorough experiments and detailed analysis on a database consisting of 7500 images of different poses and variable spatial resolutions. This database is generated from an initial one that originally consists of 1500 images, by down-sampling the original images to four different low resolution levels beside the original level. The results show that the proposed method achieves significantly high recognition rates over various conditions resembling practical scenarios.

On visual BMI analysis from facial images

Automatically assessing body mass index (BMI) from facial images is an interesting and challenging problem in computer vision. Facial feature extraction is an important step for visual BMI estimation. This work studies the visual BMI estimation problem based on the characteristics and performance of different facial representations, which has not been well studied yet. Various facial representations, including geometry based representations and deep learning based, are comprehensively evaluated and analyzed from three perspectives: the overall performance on visual BMI prediction, the redundancy in facial representations and the sensitivity to head pose changes. The experiments are conducted on two databases: a new dataset we collected, called the FIW-BMI and an existing large dataset Morph II. Our studies provide some deep insights into the facial representations for visual BMI analysis: 1) The deep model based methods perform better than geometry based methods. Among them, the VGG-Face and Arcface show more robustness than others in most cases; 2) Removing the redundancy in VGG-Face representation can increase the accuracy and efficiency in BMI estimation; 3) Large head poses lead to low performance for BMI estimation. The Arcface, VGG-Face and PIGF are more robust than the others to head pose variations.

Eye center localization in a facial image based on geometric shapes of iris and eyelid under natural variability

The estimation of the human eye centers is one of the important step in several computer vision applications such as driver drowsiness detection, eye tracking, face recognition etc. Most of the existing techniques are able to localize eyes in frontal faces only while they fail to localize eye pairs in complex scenarios such as changes in head pose, scale, and illumination. In this paper, an eye localization method has been proposed that can locate the eye centers more precisely in facial images captured under the above-mentioned complexities. The proposed method consists of three stages: eye candidate detection, eye candidate verification, and post-processing. In eye candidate detection, the possible eye candidates are extracted using two new features namely Semi-Circular Edge Shape (sCES) and Semi-Ellipse Edge Shape (sEES) features. These features take into consideration the semi-circular and semi-ellipse edges of iris and eyelid and hence are able to localize eye centers more precisely. In verification, the extracted eye candidates are verified using a Support Vector Machine (SVM) based classifier. A scale-space framework is also included in the verification stage to handle the scale variations of images. In post-processing, the eye centers are paired using some geometrical constraints and then a modified gradient-based method is proposed to detect the required eye pair. The proposed system is evaluated on different databases to check its robustness to changes in head pose, scale, illumination etc. The experimental results suggest that the proposed method shows better accuracy in challenging environments and also outperforms some state-of-the-art methods.

Estimating Audience Engagement to Predict Movie Ratings

While watching movies, audience members exhibit both subtle and coarse gestures (e.g., smiles, head-pose change, fidgeting, stretching) which convey sentiment (i.e., engaged or disengaged) during feature length movies. Noticing these behaviors using computer vision systems is a very challenging problem—especially in a movie theatre environment. The environment is dark and contains views of people at different scales and viewpoints. Feature length movies typically run 80-120 minutes, and tracking people uninterrupted for this duration is still an unsolved problem. Facial expressions of audience members are subtle, short, and sparse; making it difficult to detect and recognize activities. Finally, annotating audience sentiment at the frame-level is prohibitively time consuming. To circumvent these issues, we use an infrared illuminated test-bed to obtain a visually uniform input of audiences watching feature length movies. We present a method which can automatically detect the change in behavior (key-gestures) using “key-frames”, which can convey audience sentiment. As the number of key-frames are many orders of magnitudes lower than the number of frames, the annotation problem is reduced to assigning a sentiment label for each key-frame. Using these discovered key-gestures, we create a movie rating classifier from crowd-sourced ratings and demonstrate its predictive capability. Our dataset consists of over 50 hours of audience behavior collected across 237 subjects.

Effect of head motion on MRI B0 field distribution.

To identify and characterize the sources of B0 field changes due to head motion, to reduce motion sensitivity in human brain MRI. B0 fields were measured in 5 healthy human volunteers at various head poses. After measurement of the total field, the field originating from the subject was calculated by subtracting the external field generated by the magnet and shims. A subject-specific susceptibility model was created to quantify the contribution of the head and torso. The spatial complexity of the field changes was analyzed using spherical harmonic expansion. Minor head pose changes can cause substantial and spatially complex field changes in the brain. For rotations and translations of approximately 5 º and 5 mm, respectively, at 7 T, the field change that is associated with the subject's magnetization generates a standard deviation (SD) of about 10 Hz over the brain. The stationary torso contributes to this subject-associated field change significantly with a SD of about 5 Hz. The subject-associated change leads to image-corrupting phase errors in multi-shot -weighted acquisitions. The B0 field changes arising from head motion are problematic for multishot -weighted imaging. Characterization of the underlying sources provides new insights into mitigation strategies, which may benefit from individualized predictive field models in addition to real-time field monitoring and correction strategies.

RGB-D face recognition under various conditions via 3D constrained local model

This research proposes a method for 3D face recognition in various conditions using 3D constrained local model (CLM-Z). In this method, a combination of 2D images (RGBs) and depth images (Ds) captured by Kinect has been used. After detecting the face and smoothing the depth image, CLM-Z model has been used to model and detect the important points of the face. These points are described using Histogram of Oriented Gradients (HOG), Local Binary Patterns (LBP), and 3D Local Binary Patterns (3DLBP). Finally, each face is recognized by a Support Vector Machine (SVM). The challenging situations are changes of lighting, facial expression and head pose. The results on CurtinFaces and IIIT-D datasets demonstrate that the proposed method outperformed state-of-the-art methods under illumination, expression and pitch pose conditions and comparable results were obtained in other cases. Additionally, our proposed method is robust even when the training data has not been carefully collected.

Real-time 3D eyelids tracking from semantic edges

State-of-the-art real-time face tracking systems still lack the ability to realistically portray subtle details of various aspects of the face, particularly the region surrounding the eyes. To improve this situation, we propose a technique to reconstruct the 3D shape and motion of eyelids in real time. By combining these results with the full facial expression and gaze direction, our system generates complete face tracking sequences with more detailed eye regions than existing solutions in real-time. To achieve this goal, we propose a generative eyelid model which decomposes eyelid variation into two low-dimensional linear spaces which efficiently represent the shape and motion of eyelids. Then, we modify a holistically-nested DNN model to jointly perform semantic eyelid edge detection and identification on images. Next, we correspond vertices of the eyelid model to 2D image edges, and employ polynomial curve fitting and a search scheme to handle incorrect and partial edge detections. Finally, we use the correspondences in a 3D-to-2D edge fitting scheme to reconstruct eyelid shape and pose. By integrating our fast fitting method into a face tracking system, the estimated eyelid results are seamlessly fused with the face and eyeball results in real time. Experiments show that our technique applies to different human races, eyelid shapes, and eyelid motions, and is robust to changes in head pose, expression and gaze direction.

Conceiving Human Interaction by Visualising Depth Data of Head Pose Changes and Emotion Recognition via Facial Expressions

Affective computing in general and human activity and intention analysis in particular comprise a rapidly-growing field of research. Head pose and emotion changes present serious challenges when applied to player’s training and ludology experience in serious games, or analysis of customer satisfaction regarding broadcast and web services, or monitoring a driver’s attention. Given the increasing prominence and utility of depth sensors, it is now feasible to perform large-scale collection of three-dimensional (3D) data for subsequent analysis. Discriminative random regression forests were selected in order to rapidly and accurately estimate head pose changes in an unconstrained environment. In order to complete the secondary process of recognising four universal dominant facial expressions (happiness, anger, sadness and surprise), emotion recognition via facial expressions (ERFE) was adopted. After that, a lightweight data exchange format (JavaScript Object Notation (JSON)) is employed, in order to manipulate the data extracted from the two aforementioned settings. Motivated by the need to generate comprehensible visual representations from different sets of data, in this paper, we introduce a system capable of monitoring human activity through head pose and emotion changes, utilising an affordable 3D sensing technology (Microsoft Kinect sensor).