Expression Recognition In Videos Research Articles

Multi-geometry embedded transformer for facial expression recognition in videos

Dynamic facial expressions in videos express more realistic emotional states, and recognizing emotions from in-the-wild facial expression videos is a challenging task due to the changeable posture, partial occlusion and various light conditions. Although current methods have designed transformer-based models to learn spatial–temporal features, they cannot explore useful local geometry structures from both spatial and temporal views to capture subtle emotional features for the videos with varied poses and facial occlusion. To this end, we propose a novel multi-geometry embedded transformer (MGET), which adapts multi-geometry knowledge into transformers and excavates spatial–temporal geometry information as complementary to learn effective emotional features. Specifically, from a new perspective, we first design a multi-geometry distance learning (MGDL) to capture emotion-related geometry structure knowledge under Euclidean and Hyperbolic spaces. Especially based on the advantages of hyperbolic geometry, it finds the more subtle emotional changes among local spatial and temporal features. Secondly, we combine MGDL with transformer to design spatial–temporal MGETs, which capture important spatial and temporal multi-geometry features to embed them into their corresponding original features, and then perform cross-regions and cross-frame interaction on these multi-level features. Finally, MGET gains superior performance on DFEW, FERV39k and AFEW datasets, where the unweighted average recall (UAR) and weighted average recall (WAR) are 58.65%/69.91%, 41.91%/50.76% and 53.23%/55.40%, respectively, and the gained improvements are 2.55%/0.66%, 3.69%/2.63% and 3.66%/1.14% compared to M3DFEL, Logo-Forme and EST methods.

From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos

From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos

Facial expression recognition in videos using hybrid CNN & ConvLSTM.

The three-dimensional convolutional neural network (3D-CNN) and long short-term memory (LSTM) have consistently outperformed many approaches in video-based facial expression recognition (VFER). The image is unrolled to a one-dimensional vector by the vanilla version of the fully-connected LSTM (FC-LSTM), which leads to the loss of crucial spatial information. Convolutional LSTM (ConvLSTM) overcomes this limitation by performing LSTM operations in convolutions without unrolling, thus retaining useful spatial information. Motivated by this, in this paper, we propose a neural network architecture that consists of a blend of 3D-CNN and ConvLSTM for VFER. The proposed hybrid architecture captures spatiotemporal information from the video sequences of emotions and attains competitive accuracy on three FER datasets open to the public, namely the SAVEE, CK + , and AFEW. The experimental results demonstrate excellent performance without external emotional data with the added advantage of having a simple model with fewer parameters. Moreover, unlike the state-of-the-art deep learning models, our designed FER pipeline improves execution speed by many factors while achieving competitive recognition accuracy. Hence, the proposed FER pipeline is an appropriate candidate for recognizing facial expressions on resource-limited embedded platforms for real-time applications.

Dynamic FERNet: Deep learning with optimal feature selection for face expression recognition in video

SummaryThis work concentrates on the recognition of facial emotion from video sequences with deep learning. Once the input video is converted into frames, the face detection is performed on each frame using the viola–jones face detection algorithm. Then, the feature extraction is performed by three well‐performing feature extraction techniques like modified local directional pattern, spatio‐temporal features, and scale‐invariant feature transforms. The extracted features from all the frames of the video are concatenated. To reduce the feature‐length for decreasing the training complexity, and enhance the recognition performance, the optimal feature selection is accomplished with the distance‐based tunicate swarm algorithm. These selected features are processed to an innovative deep learning model termed a heuristically modified recurrent neural network. The same D‐TSA improves the performance of RNN by optimally tuning its hidden neurons. Experimental results on a widely used benchmark dataset and manually collected dataset show that the classification performance is improved using spatio‐temporal features, SIFT, M‐LDP, and optimal feature selection, and thus, the proposed model with HM‐RNN outperforms the other existing models.

Micro-Facial Expression Recognition in Video Based on Optimal Convolutional Neural Network (MFEOCNN) Algorithm

Facial expression is a standout amongst the most imperative features of human emotion recognition. For demonstrating the emotional states facial expressions are utilized by the people. In any case, recognition of facial expressions has persisted a testing and intriguing issue with regards to PC vision. Recognizing the Micro-Facial expression in video sequence is the main objective of the proposed approach. For efficient recognition, the proposed method utilizes the optimal convolution neural network. Here the proposed method considering the input dataset is the CK+ dataset. At first, by means of Adaptive median filtering preprocessing is performed in the input image. From the preprocessed output, the extracted features are Geometric features, Histogram of Oriented Gradients features and Local binary pattern features. The novelty of the proposed method is, with the help of Modified Lion Optimization (MLO) algorithm, the optimal features are selected from the extracted features. In a shorter computational time it has the benefits of rapidly focalizing and effectively acknowledging with the aim of getting an overall arrangement or idea. Finally the recognition is done by Convolution Neural network (CNN). Then the performance of the proposed MFEOCNN method is analyzed in terms of false measures and recognition accuracy. This kind of emotion recognition is mainly used in medicine, marketing, E-learning, entertainment, law and monitoring. From the simulation, we know that the proposed approach achieves maximum recognition accuracy of 99.2% with minimum Mean Absolute Error (MAE) value. This results are compared with the existing for Micro-Facial Expression Based Deep-Rooted Learning (MFEDRL), Convolutional Neural Network with Lion Optimization (CNN+LO) and Convolutional Neural Network (CNN) without optimization. The simulation of the proposed method is done in the working platform of MATLAB.

Facial Expression Recognition in Video with Multiple Feature Fusion

Video based outward appearance affirmation has been a long standing issue and pulled in creating thought starting late. The way to a productive outward appearance affirmation system is to manhandle the conceivable outcomes of shifting media modalities and structure incredible highlights to effectively depict the facial appearance and course of action changes achieved by facial developments. We propose a powerful system to address this issue in this paper. In our examination, both visual modalities (face pictures) and sound modalities (talk) are used. Another part descriptor known as bar chart of familiarized Gradients from 3 Orthogonal Planes (HOG-TOP) is projected to free dynamic surfaces from video courses of action to depict facial look changes. In addition, another effective geometric element got from the curve change of facial places of interest is proposed to get facial plan changes.

Facial Expression Recognition in Video with Multiple Feature Fusion

Video based facial expression recognition has been a long standing problem and attracted growing attention recently. The key to a successful facial expression recognition system is to exploit the potentials of audiovisual modalities and design robust features to effectively characterize the facial appearance and configuration changes caused by facial motions. We propose an effective framework to address this issue in this paper. In our study, both visual modalities (face images) and audio modalities (speech) are utilized. A new feature descriptor called Histogram of Oriented Gradients from Three Orthogonal Planes (HOG-TOP) is proposed to extract dynamic textures from video sequences to characterize facial appearance changes. And a new effective geometric feature derived from the warp transformation of facial landmarks is proposed to capture facial configuration changes. Moreover, the role of audio modalities on recognition is also explored in our study. We applied the multiple feature fusion to tackle the video-based facial expression recognition problems under lab-controlled environment and in the wild, respectively. Experiments conducted on the extended Cohn-Kanade (CK+) database and the Acted Facial Expression in Wild (AFEW) 4.0 database show that our approach is robust in dealing with video-based facial expression recognition problems under lab-controlled environment and in the wild compared with the other state-of-the-art methods.

Illumination invariants in deep video expression recognition

In this paper we present architectures based on deep neural nets for expression recognition in videos, which are invariant to local scaling. We amalgamate autoencoder and predictor architectures using an adaptive weighting scheme coping with a reduced size labeled dataset, while enriching our models from enormous unlabeled sets. We further improve robustness to lighting conditions by introducing a new adaptive filter based on temporal local scale normalization. We provide superior results over known methods, including recent reported approaches based on neural nets.

Collaborative discriminative multi-metric learning for facial expression recognition in video

Facial expression recognition in video has been an important and relatively new topic in human face analysis and attracted growing interests in recent years. Unlike conventional image-based facial expression recognition methods which recognize facial expression category from still images, facial expression recognition in video is more challenging because there are usually larger intra-class variations among facial frames within a video. This paper presents a collaborative discriminative multi-metric learning (CDMML) for facial expression recognition in video. We first compute multiple feature descriptors for each face video to describe facial appearance and motion information from different aspects. Then, we learn multiple distance metrics with these extracted multiple features collaboratively to exploit complementary and discriminative information for recognition. Experimental results on the Acted Facial Expression in Wild (AFEW) 4.0 and the extended Cohn–Kanada (CK+) datasets are presented to demonstrate the effectiveness of our proposed method.

A new framework with multiple tasks for detecting and locating pain events in video

Automatically detecting and locating pain events in video is an important task in medical assessment. It is a challenging problem in facial expression analysis due to spontaneous faces, head movements and pose variations. In this paper, we explore the role of facial information at various time scales (frame, segment and sequence) and propose a new framework for pain event detection and locating in video. We introduce a feature-level fusion method for pain event detection and a multiple-task fusion method for locating pain events, respectively. Both spatial and spatial–temporal features are utilized in our study. At first, we employ the histogram of oriented gradients (HOG) of fiducial points (P-HOG) to extract spatial features from each video frame and train an SVM as a frame-based pain event detector. Secondly, HOG from Three Orthogonal Planes (named as HOG-TOP) is used to characterize the dynamic textures of a video segment, a segment-based classifier (SVM) is then trained for segment-level detection. We further apply a max pooling strategy to obtain the global P-HOG and HOG-TOP to represent the whole video sequence and a multiple kernel fusion is employed to find an optimal feature-level fusion. An SVM with multiple kernels is trained to perform sequence-level (pain event) detection. Finally, an effective probabilistic fusion method is proposed to integrate the detection results of the three different tasks (frame-level, segment-level and sequence-level detection) to locate pain events in video. Extensive experiments conducted on the UNBC-McMaster Shoulder Pain database show that our proposed method outperforms other state-of-the-art methods both in pain event detection and locating in video. Our sequence-level event detection method has also been applied to facial expression recognition in video with good results.

Facial Expression Recognition in Video using Adaboost and SVM

In human-computer interaction facial expression is the characteristic and proficient method for correspondence, and has been acknowledged as essential input of such interface. In this paper, we present an enhancement in facial expression recognition for image sequence. The most important step is to extract essential features from face to efficiently determine facial expression. Experimentation shows that LBP method performs well while extracting facial features. We further found that Boosted-LBP extracts most distinct features and the best recognition is calculated by SVM classifier.,