Face Video Research Articles

Towards Realistic Landmark-Guided Facial Video Inpainting Based on GANs

Towards Realistic Landmark-Guided Facial Video Inpainting Based on GANs

LSTPNet: Long short-term perception network for dynamic facial expression recognition in the wild

In-the-wild dynamic facial expression recognition (DFER) is a very challenging task, and previous methods based on convolutional neural networks (CNNs), recurrent neural networks (RNNs), or Transformers emphasize the extraction of either short-term temporal information or long-term temporal information from facial video sequences. Different from existing methods, this paper proposes a long short-term perceptimon network (LSTPNet) for dynamic facial expression recognition, taking into account the joint perception of the above two temporal cues to benefit the DFER task. Specifically, we propose a long short-term temporal Transformer (LSTformer) which can perceive both long-term and short-term temporal information effectively. In addition, we introduce a temporal channel excitation (TCE) module extended from the previous notable efficient channel attention (ECA) module, in order to establish temporal attention for intermediate features within the backbone network, and obtain more temporally representative features. Experimental results on three benchmark datasets demonstrate the state-of-the-art performance of the proposed LSTPNet. The code will be available at https://github.com/LLFabiann/LSTPNet/.

Vital signs estimation in elderly using camera-based photoplethysmography

In the context of Ambient Assisted Living, the demand for healthcare technologies development has been increased in the last two years by the current global health situation. The contagious nature of COVID-19 warrants inevitability to the thought of easing a continuous measurement of vital signs such as Heart Rate, Breath Rate, and SpO2 in a non-invasive manner, contributing of fundamental importance in the assessment of health status especially among frail and/or elderly individuals. A widely diffused method for contactless vital signs monitoring is remote photoplethysmography from facial video streams that, contrary to traditional contact measurement techniques, allows the measurement of vital parameters without the need for wearable sensors (generally considered uncomfortable, especially by the elderly), even with commercial and low-cost digital cameras. This paper proposes the design and implementation of a new pipeline for estimating Heart Rate, Breath Rate, and SpO2 values, and its integration on Raspberry Pi 4 as an elaboration unit. The pipeline provides specific algorithmic blocks to improve vital signs estimation in elderly subjects as it is made more difficult by the skin tone and the presence on the face of wrinkles, folds, and moles. Quantitative evaluations on our dataset containing the acquisition of only elderly older than 65 years of age demonstrate the validity of the proposed pipeline. For validation against the state of the art, tests were also conducted on three standard benchmark datasets containing video with subjects of varying ages. Again, the pipeline proved to be robust concerning the estimation of vital signs considered in the present work.

Contactless face video based vital signs detection framework for continuous health monitoring using feature optimization and hybrid neural network.

Continuous monitoring of vital signs such as respiration and heart rate is essential to detect and predict conditions that may affect the patient's well-being. To detect these vital signs most medical systems use contact sensors. They are not feasible for long term monitoring and are not repeatable. Vital signs using facial video-noncontact monitoring are becoming increasingly important. Researchers in the last few years although considerable progress has been made, challenging datasets absence timing of assessment process and the technology still has some limitations such as time consuming nature and lack of computer portability. To solve those problems, we propose a contactless video based vital signs detection framework for continuous health monitoring using feature optimization and hybrid neural network. In the proposed technique, modified war strategy optimization algorithm is proposed to segment the face portion from the input video frames. Then, we utilize the known data acquisition models to extract vital signs from the segmented face portions are heart rate, blood pressure, respiratory rate and oxygen saturation. An improved neural network structure (Lifting Net) is further used to achieve the adaptive extraction of deep hidden features for specific signs, for realizing the high precision of human health monitoring. The Hughes effect or dimensionality issue affects detection accuracy in sign classification when there are fewer training instances relative to the number of spectral features. The problem can be overcome through feature optimization here Northern goshawk optimization algorithm is used to select optimal best features which reduces the data dimensionality issue. Furthermore, hybrid deep ensemble reinforcement learning classifier is proposed for the human vital sign detection and classification which ensures the early detection of patient abnormality. Finally, we validate our framework using benchmark video datasets such as TokyoTechrPPG, PURE and COHFACE. To proves the effectiveness of proposed technique using simulation results and comparative analysis.

Monitoring respiratory state from near-infrared face video images

In this paper, we propose an algorithm for estimating respiratory state using near-infrared facial video images. Estimation of respiratory state is an important indicator for early detection of respiratory diseases. In particular, there is a demand for monitoring respiratory state during the night. One method of monitoring respiratory state is to use contact-type sensors. However, this method requires the installation of many sensors and a visit to a hospital, which place a burden on patients. Therefore, we propose to acquire respiratory-induced features from near-infrared face video images and investigate their similarity to measurements obtained with a respirometer for non-contact monitoring of respiratory state in the dark. Respiratory-induced features were obtained from pulse wave signals extracted from the face video images. The results showed correlations in several respiratory states. This study opens some perspectives in non-contact monitoring of respiratory states.

Attention-based 3D convolutional recurrent neural network model for multimodal emotion recognition.

Multimodal emotion recognition has become a hot topic in human-computer interaction and intelligent healthcare fields. However, combining information from different human different modalities for emotion computation is still challenging. In this paper, we propose a three-dimensional convolutional recurrent neural network model (referred to as 3FACRNN network) based on multimodal fusion and attention mechanism. The 3FACRNN network model consists of a visual network and an EEG network. The visual network is composed of a cascaded convolutional neural network-time convolutional network (CNN-TCN). In the EEG network, the 3D feature building module was added to integrate band information, spatial information and temporal information of the EEG signal, and the band attention and self-attention modules were added to the convolutional recurrent neural network (CRNN). The former explores the effect of different frequency bands on network recognition performance, while the latter is to obtain the intrinsic similarity of different EEG samples. To investigate the effect of different frequency bands on the experiment, we obtained the average attention mask for all subjects in different frequency bands. The distribution of the attention masks across the different frequency bands suggests that signals more relevant to human emotions may be active in the high frequency bands γ (31-50 Hz). Finally, we try to use the multi-task loss function Lc to force the approximation of the intermediate feature vectors of the visual and EEG modalities, with the aim of using the knowledge of the visual modalities to improve the performance of the EEG network model. The mean recognition accuracy and standard deviation of the proposed method on the two multimodal sentiment datasets DEAP and MAHNOB-HCI (arousal, valence) were 96.75 ± 1.75, 96.86 ± 1.33; 97.55 ± 1.51, 98.37 ± 1.07, better than those of the state-of-the-art multimodal recognition approaches. The experimental results show that starting from the multimodal information, the facial video frames and electroencephalogram (EEG) signals of the subjects are used as inputs to the emotion recognition network, which can enhance the stability of the emotion network and improve the recognition accuracy of the emotion network. In addition, in future work, we will try to utilize sparse matrix methods and deep convolutional networks to improve the performance of multimodal emotion networks.

Nonlinear dynamical system iteration applied in video face feature extraction and recognition

Nonlinear dynamical system iteration applied in video face feature extraction and recognition

Research on the Advantages of Short Video and the Way to Revive Long Video

Short video is one of the fastest growing industries at present, and the spread of short video on social media is very large. Long videos not only provide more selectivity, but also change the way humans watch videos. However, in the face of massive content competition, it becomes more difficult to attract viewers, leading to more people choosing to watch short videos. This paper explores the advantages of short video in modern communication, and proposes the revival of long video. Short videos are popular for their concise and clear features, which grab viewers' attention in a fast-paced society. They are easily shared and disseminated across various platforms, expanding the reach of information. Moreover, short videos provide a rich audio-visual experience, making communication more lively and interesting. On the other hand, long videos face challenges such as the dominance of short video platforms, rising membership fees, and copyright restrictions. To revive long videos, platforms can open up copyrights, reduce fees, and improve member benefits. Creators should focus on producing premium content with appealing themes, while the government can promote the import of foreign films to enrich viewing options. Viewers can use their free time to watch long-form videos in theaters or on TV. By addressing these challenges and promoting quality content, long-form videos can thrive alongside short-form videos in the modern media landscape.

Pulse rate estimation based on facial videos: an evaluation and optimization of the classical methods using both self-constructed and public datasets

Pulse rate estimation based on facial videos: an evaluation and optimization of the classical methods using both self-constructed and public datasets

Recovering Physiological Signals from Facial Videos: Recent Advances and Applications in Intelligent Vehicles

Recovering Physiological Signals from Facial Videos: Recent Advances and Applications in Intelligent Vehicles

Facial Action Unit Representation Based on Self-Supervised Learning With Ensembled Priori Constraints.

Facial action units (AUs) focus on a comprehensive set of atomic facial muscle movements for human expression understanding. Based on supervised learning, discriminative AU representation can be achieved from local patches where the AUs are located. Unfortunately, accurate AU localization and characterization are challenged by the tremendous manual annotations, which limits the performance of AU recognition in realistic scenarios. In this study, we propose an end-to-end self-supervised AU representation learning model (SsupAU) to learn AU representations from unlabeled facial videos. Specifically, the input face is decomposed into six components using auto-encoders: five photo-geometric meaningful components, together with 2D flow field AUs. By constructing the canonical neutral face, posed neutral face, and posed expressional face gradually, these components can be disentangled without supervision, therefore the AU representations can be learned. To construct the canonical neutral face without manually labeled ground truth of emotion state or AU intensity, two priori knowledge based assumptions are proposed: 1) identity consistency, which explores the identical albedos and depths of different frames in a face video, and helps to learn the camera color mode as an extra cue for canonical neutral face recovery. 2) average face, which enables the model to discover a 'neutral facial expression' of the canonical neutral face and decouple the AUs in representation learning. To the best of our knowledge, this is the first attempt to design self-supervised AU representation learning method based on the definition of AUs. Substantial experiments on benchmark datasets have demonstrated the superior performance of the proposed work in comparison to other state-of-the-art approaches, as well as an outstanding capability of decomposing input face into meaningful factors for its reconstruction. The code is made available at https://github.com/Sunner4nwpu/SsupAU.

Contact-Free Atrial Fibrillation Screening With Attention Network.

Atrial Fibrillation (AF) screening from face videos has become popular with the trend of telemedicine and telehealth in recent years. In this study, the largest facial image database for camera-based AF detection is proposed. There are 657 participants from two clinical sites and each of them is recorded for about 10 minutes of video data, which can be further processed as over 10 000 segments around 30 seconds, where the duration setting is referred to the guideline of AF diagnosis. It is also worth noting that, 2 979 segments are segment-wise labeled, that is, every rhythm is independently labeled with AF or not. Besides, all labels are confirmed by the cardiologist manually. Various environments, talking, facial expressions, and head movements are involved in data collection, which meets the situations in practical usage. Specific to camera-based AF screening, a novel CNN-based architecture equipped with an attention mechanism is proposed. It is capable of fusing heartbeat consistency, heart rate variability derived from remote photoplethysmography, and motion features simultaneously to reliable outputs. With the proposed model, the performance of intra-database evaluation comes up to 96.62% of sensitivity, 90.61% of specificity, and 0.96 of AUC. Furthermore, to check the capability of adaptation of the proposed method thoroughly, the cross-database evaluation is also conducted, and the performance also reaches about 90% on average with the AUCs being over 0.94 in both clinical sites.

Identity-Aware and Shape-Aware Propagation of Face Editing in Videos.

The development of deep generative models has inspired various facial image editing methods, but many of them are difficult to be directly applied to video editing due to various challenges ranging from imposing 3D constraints, preserving identity consistency, ensuring temporal coherence, etc. To address these challenges, we propose a new framework operating on the StyleGAN2 latent space for identity-aware and shape-aware edit propagation on face videos. In order to reduce the difficulties of maintaining the identity, keeping the original 3D motion, and avoiding shape distortions, we disentangle the StyleGAN2 latent vectors of human face video frames to decouple the appearance, shape, expression, and motion from identity. An edit encoding module is used to map a sequence of image frames to continuous latent codes with 3D parametric control and is trained in a self-supervised manner with identity loss and triple shape losses. Our model supports propagation of edits in various forms: I. direct appearance editing on a specific keyframe, II. implicit editing of face shape via a given reference image, and III. existing latent-based semantic edits. Experiments show that our method works well for various forms of videos in the wild and outperforms an animation-based approach and the recent deep generative techniques.

Analysis and Benchmarking of Extending Blind Face Image Restoration to Videos.

Recent progress in blind face restoration has resulted in producing high-quality restored results for static images. However, efforts to extend these advancements to video scenarios have been minimal, partly because of the absence of benchmarks that allow for a comprehensive and fair comparison. In this work, we first present a fair evaluation benchmark, in which we first introduce a Real-world Low-Quality Face Video benchmark (RFV-LQ), evaluate several leading image-based face restoration algorithms, and conduct a thorough systematical analysis of the benefits and challenges associated with extending blind face image restoration algorithms to degraded face videos. Our analysis identifies several key issues, primarily categorized into two aspects: significant jitters in facial components and noise-shape flickering between frames. To address these issues, we propose a Temporal Consistency Network (TCN) cooperated with alignment smoothing to reduce jitters and flickers in restored videos. TCN is a flexible component that can be seamlessly plugged into the most advanced face image restoration algorithms, ensuring the quality of image-based restoration is maintained as closely as possible. Extensive experiments have been conducted to evaluate the effectiveness and efficiency of our proposed TCN and alignment smoothing operation.

Quantifying Environmental Sensitivity Using Morphing Facial Expression Videos

Quantifying Environmental Sensitivity Using Morphing Facial Expression Videos

The Emotional Toll of HIV: Exploring Facial and Auditory Emotion Recognition and Emotion Regulation in People Living With HIV

Abstract People living with Human Immunodeficiency Virus (PLHIV) have been reported to show poor facial emotion recognition. However, these studies presented participants with facial emotion photographs whereas in real life facial emotion recognition hardly involves inferring emotions from static faces. Moreover, emotion recognition from other sensory modalities, such as auditory, has hardly been explored. There’s also a dearth of studies examining emotion regulation difficulties in this group. The present study, thus, explored facial (using facial emotion videos) and auditory emotion recognition as well as difficulties in emotion regulation (using the Hindi version of Difficulties in Emotion Regulation Scale) in 60 PLHIV and 60 people without HIV (PWoHIV). Additionally, the association of HIV duration (duration since diagnosis of HIV), viral load, and Clusters of differentiation 4 (CD4) count with emotion recognition and regulation difficulties in PLHIV was explored. Findings from one-way ANCOVA (with education and socioeconomic status as covariates) revealed significantly impaired auditory emotion recognition (particularly for fear) among PLHIV than PWoHIV. The former also showed significantly poorer facial emotion recognition for surprise. PLHIV also self-reported significantly more emotion regulation difficulties than PWoHIV, specifically Nonacceptance of their response to negative emotions and limited access to emotion regulation Strategies. CD4 count was negatively correlated with emotion regulation difficulties, particularly for accomplishing goal-directed behaviour when experiencing negative emotions (Goals) and Strategies. Besides the novel addition to the literature regarding impaired auditory emotion recognition in PLHIV, these findings can help develop targeted interventions to improve emotion recognition and emotion regulation for PLHIV.

Enhanced Face Detection in Videos Based on Integrating Spatial Features (LBP, CS-LBP) with CNN Technique

Face detection is a crucial aspect of computer vision and image processing, in order to enable the automatic detection and identification of human faces in video streams, face detection is an essential component of computer vision and image processing. Applications for facial recognition, video analytics, security systems, and surveillance all depend on it. Face identification techniques face many obstacles and issues, such as positional fluctuations, illumination changes, resolution and scale issues, facial emotions, and cosmetics. Robust algorithms are required for efficient face detection. This field looks at the feature extraction process using a variety of techniques. These consist of the center symmetric local binary patterns (CS_LBP) approach and the local binary patterns (LBP) method. The YouTube Face database provided the video frames that we used for our study. In order to train the convolutional neural network (CNN) to detect human faces in the video and draw a bounding box around them. The experimental results of the suggested approaches show that. The accuracy rate was 94% higher with the LBP techniques. However, the CS_LBP technique showed the best level of accuracy in both face detection and face rectangle recognition, with an accuracy rate of 95%.

Trial Selection Tensor Canonical Correlation Analysis (TSTCCA) for Depression Recognition with Facial Expression and Pupil Diameter.

Facial expressions have been widely used for depression recognition because it is intuitive and convenient to access. Pupil diameter contains rich emotional information that is already reflected in facial video streams. However, the spatiotemporal correlation between pupillary changes and facial behavior changes induced by emotional stimuli has not been explored in existing studies. This paper presents a novel multimodal fusion algorithm - Trial Selection Tensor Canonical Correlation Analysis (TSTCCA) to optimize the feature space and build a more robust depression recognition model, which innovatively combines the spatiotemporal relevance and complementarity between facial expression and pupil diameter features. TSTCCA explores the interaction between trials and obtains an effective fusion representation of two modalities from a trial subset related to depression. The experimental results show that TSTCCA achieves the highest accuracy of 78.81% with the subset of 25 trials.

Non-contact Heart Rate Measurement from Face Video Sequences using SIFT and FastICA.

Non-contact Heart Rate Measurement from Face Video Sequences using SIFT and FastICA.

Continuously Controllable Facial Expression Editing in Talking Face Videos

Continuously Controllable Facial Expression Editing in Talking Face Videos