Japanese Female Facial Expression Research Articles

Emotion Detection Using Deep Learning Algorithm

Automatic emotion detection is a prime task in computerized human behaviour analysis. The proposed system is an automatic emotion detection using convolution neural network. The proposed end-to-end CNN is therefore named as ENet. Keeping in mind the computational efficiency, the deep network makes use of trained weight parameters of the MobileNet to initialize the weight parameters of ENet. On top of the last convolution layer of ENet, the authors place global average pooling layer to make it independent of the input image size. The ENet is validated for emotion detection using two benchmark datasets: Cohn-Kanade+ (CK+) and Japanese female facial expression (JAFFE). The experimental results show that the proposed ENet outperforms the other existing methods for emotion detection.

EMOTION DETECTION USING DEEP LEARNING ALGORITHM

Automatic emotion detection is a key task in human machine interaction,where emotion detection makes system more natural. In this paper, we propose an emotion detection using deep learning algorithm. The proposed algorithm uses end to end CNN. To increase computational efficiency of the deep network, we make use of trained weight parameters of the MobileNet to initialize the weight parameters of our system. To make our system independent of the input image size, we place global average pooling layer On top of the last convolution layer of it. Proposed system is validated for emotion detection using two benchmark datasets viz. Cohn–Kanade+ (CK+) and Japanese female facial expression (JAFFE). The experimental results show that the proposed method outperforms the other existing methods for emotion detection.

Facial expression analysis using local directional stigma mean patterns and convolutional neural networks

This paper represents automatic facial expression analysis method named Local Directional Stigma Mean Patterns (LDSMP) for automatic facial expression analysis and image retrieval using content based facial expression image retrieval and CNN. The traditional local patterns such as Local Binary Patterns (LBP) and Local Ternary Patterns (LTP) are applied for face recognition and expression analysis, calculated using relationship between the center pixel and neighboring pixels. The proposed method calculates the eight directional difference values then divided into the three ranges based on threshold values. Thus, the values are substituted with basic three positive values (+3, +2, +1) and three negative values (-3, -2, -1) to get more sensitive information from an image rather than aforementioned methods. The threshold can be select either static which is selected by user or dynamic is evaluated from image itself and supports to improve the efficiency. The performance of the proposed method is further improved by giving this patterns as input to the Convolutional Neural Networks (CNN) and compared with the existing methods LBP, LTP, and Directional Binary Code (DBC) in terms of Average Precision (AP), Average Recall (AR), and Average Retrieval Rate (ARR) using standard databases COREL 10K (DB1) and JAFFE (The Japanese Female Facial Expression) (DB2) and Extended Cohn-Kanade (CK +) (DB3) dataset.

An ensemble approach of improved quantum inspired gravitational search algorithm and hybrid deep neural networks for computational optimization

In this paper, an autonomous ensemble approach of improved quantum inspired gravitational search algorithm (IQI-GSA) and hybrid deep neural networks (HDNN) is proposed for the optimization of computational problems. The IQI-GSA is a combinational variant of gravitational search algorithm (GSA) and quantum computing (QC). The improved variant enhances the diversity of mass collection for retaining the stochastic attributes and handling the local trapping of mass agents. Further, the hybrid deep neural network encompasses the convolutional and recurrent neural networks (HDCR-NN) which analyze the relational & temporal dependencies among the different computational components for optimization. The proposed ensemble approach is evaluated for the application of facial expression recognition by experimentation on Karolinska Directed Emotional Faces (KDEF) and Japanese Female Facial Expression (JAFFE) datasets. The experimentation evaluations evidently exhibit the outperformed recognition rate of the proposed ensemble approach in comparison with state-of-the-art techniques.

Deep neural network for coded mask cryptographical imaging.

We proposed a novel cryptographic imaging scheme that is the combination of optical encryption and computational decryption. To prevent personal privacy from being spied upon amid the imaging formation process, in this study we applied a coded mask to optically encrypt the scene and utilized the deep neural network for computational decryption. For encryption, the sensor recorded a new representation of the original signal, not being distinguishable by humans on purpose. For decryption, we successfully reconstructed the image with the mean squared error equal to 0.028, and 100% for the classification through the Japanese Female Facial Expression dataset. By means of the feature visualization, we found that the coded mask served as a linear operator to synthesize the spatial fidelity of the original scene, but kept the features for the post-recognition process. We believe the proposed framework can inspire more possibilities for the unconventional imaging system.

Facial expression recognition using singular values and wavelet‐based LGC‐HD operator

Facial expression recognition (FER) is a significant research area in the human–computer interaction. The performance of FER systems depends on an efficient feature extraction method. This research work proposes a robust method for feature extraction using local gradient coding based on horizontal and diagonal (LGC-HD) operator, wavelet transform and singular values. The proposed framework consists of three steps in feature extraction. First, the wavelet transform is applied on facial images and features are extracted by applying the LGC-HD operator on low–low bands at different levels of decomposition. Second, the singular values of the facial image are calculated by applying singular value decomposition and these are used as features. Third, the features from the previous two steps are concatenated to obtain the final feature for FER systems. The proposed method combines the two advantages: (1) the integration of wavelet transform and LGC-HD operator provides discriminating and robust features. (2) The singular values are not sensitive to grey scale changes caused by the noise. After obtaining the features, the support vector machine is used for expression recognition. Extensive experimentation on Japanese Female Facial Expression database, Cohn–Kanade, FEI face and facial expression research group databases achieves good recognition rates of 84.2%, 92.3%, 98.2% and 97.9%, respectively.

Fusion of multi representation and multi descriptors for facial expression recognition

Facial Expression Recognition has become vital for efficient Human Computer Interaction. In this paper, we propose effective facial expression recognition approachfor recognizing six basic facial expressions. Our approach consists of three main phases which are: (1) face detection and pre-processing, (2) features extraction and (3) facial expression classification. The face pre-processing phase is performed using the facial landmarks. After the face is aligned and cropped, facial regions of interest (eyes, nose and mouth) are detected. In the features extraction phase, we used Histogram of oriented gradients (HOG), Local Binary Pattern (LBP) and the fusion of the two features. For the last step, Support Vector Machine (SVM) is used to recognize the facial expression. To evaluate the performance of our approach, we used three popular datasets which are The Extended Cohn-Kanade (CK+), The Japanese Female Facial Expression (JAFFE) and Oulu-CASIA NIR-VIS dataset (CASIA), In addition, 10 folds cross-validation scheme is used to evaluate the performance of our approach. Our proposed fusion of multi representations and multi descriptors achieves better or competitive performance compared with the state-of-the-art methods. The accuracies of our approach are 99.18%, 95.77% and 99.09% for CK+, JAFFE and CASIA, respectively. The results prove the efficiency of our approach although the challenging conditions from one dataset to another.

Patch Attention Layer of Embedding Handcrafted Features in CNN for Facial Expression Recognition.

Recognizing facial expression has attracted much more attention due to its broad range of applications in human–computer interaction systems. Although facial representation is crucial to final recognition accuracy, traditional handcrafted representations only reflect shallow characteristics and it is uncertain whether the convolutional layer can extract better ones. In addition, the policy that weights are shared across a whole image is improper for structured face images. To overcome such limitations, a novel method based on patches of interest, the Patch Attention Layer (PAL) of embedding handcrafted features, is proposed to learn the local shallow facial features of each patch on face images. Firstly, a handcrafted feature, Gabor surface feature (GSF), is extracted by convolving the input face image with a set of predefined Gabor filters. Secondly, the generated feature is segmented as nonoverlapped patches that can capture local shallow features by the strategy of using different local patches with different filters. Then, the weighted shallow features are fed into the remaining convolutional layers to capture high-level features. Our method can be carried out directly on a static image without facial landmark information, and the preprocessing step is very simple. Experiments on four databases show that our method achieved very competitive performance (Extended Cohn–Kanade database (CK+): 98.93%; Oulu-CASIA: 97.57%; Japanese Female Facial Expressions database (JAFFE): 93.38%; and RAF-DB: 86.8%) compared to other state-of-the-art methods.

FER-net: facial expression recognition using deep neural net

Automatic facial expression recognition (FER) is one of the most challenging tasks in computer vision. FER admits a wide range of applications in human–computer interaction, behavioral psychology, and human expression synthesis. Extensive works have been reported in this field, mainly, based on handcrafted features. However, it is a challenging task to accurately extract all the correlated handcrafted features due to the effect of variations caused by emotional state. Therefore, there is a quest for further research on accurately extracting relevant features that can capture changes in facial expressions (FEs) with high fidelity. In this study, we propose FER-net: a convolution neural network to distinguish FEs efficiently with the help of the softmax classifier. We implement our method FER-net along with twenty-one state-of-the-art methods and test them on five benchmarking datasets, namely FER2013, Japanese Female Facial Expressions, Extended CohnKanade, Karolinska Directed Emotional Faces, and Real-world Affective Faces. Seven FEs, namely neutral, anger, disgust, fear, happiness, sadness, and surprise, are considered in this work. The average accuracies on these datasets are 78.9%, 96.7%, 97.8%, 82.5% and 81.68%, respectively. The obtained results demonstrate that FER-net is preeminent in comparison with twenty-one state-of-the-art methods.

A hybrid feature descriptor with Jaya optimised least squares SVM for facial expression recognition

Facial expression recognition has been a long-standing problem in the field of computer vision. This paper proposes a new simple scheme for effective recognition of facial expressions based on a hybrid feature descriptor and an improved classifier. Inspired by the success of stationary wavelet transform in many computer vision tasks, stationary wavelet transform is first employed on the pre-processed face image. The pyramid of histograms of orientation gradient features is then computed from the low-frequency stationary wavelet transform coefficients to capture more prominent details from facial images. The key idea of this hybrid feature descriptor is to exploit both spatial and frequency domain features which at the same time are robust against illumination and noise. The relevant features are subsequently determined using linear discriminant analysis. A new least squares support vector machine parameter tuning strategy is proposed using a contemporary optimisation technique called Jaya optimisation for classification of facial expressions. Experimental evaluations are performed on Japanese female facial expression and the Extended Cohn–Kanade (CK+) datasets, and the results based on 5-fold stratified cross-validation test confirm the superiority of the proposed method over state-of-the-art approaches.

Facial Expression Recognition of Instructor Using Deep Features and Extreme Learning Machine.

Classroom communication involves teacher's behavior and student's responses. Extensive research has been done on the analysis of student's facial expressions, but the impact of instructor's facial expressions is yet an unexplored area of research. Facial expression recognition has the potential to predict the impact of teacher's emotions in a classroom environment. Intelligent assessment of instructor behavior during lecture delivery not only might improve the learning environment but also could save time and resources utilized in manual assessment strategies. To address the issue of manual assessment, we propose an instructor's facial expression recognition approach within a classroom using a feedforward learning model. First, the face is detected from the acquired lecture videos and key frames are selected, discarding all the redundant frames for effective high-level feature extraction. Then, deep features are extracted using multiple convolution neural networks along with parameter tuning which are then fed to a classifier. For fast learning and good generalization of the algorithm, a regularized extreme learning machine (RELM) classifier is employed which classifies five different expressions of the instructor within the classroom. Experiments are conducted on a newly created instructor's facial expression dataset in classroom environments plus three benchmark facial datasets, i.e., Cohn–Kanade, the Japanese Female Facial Expression (JAFFE) dataset, and the Facial Expression Recognition 2013 (FER2013) dataset. Furthermore, the proposed method is compared with state-of-the-art techniques, traditional classifiers, and convolutional neural models. Experimentation results indicate significant performance gain on parameters such as accuracy, F1-score, and recall.

Facial Expression Recognition Using Local Gravitational Force Descriptor-Based Deep Convolution Neural Networks

An image is worth a thousand words; hence, a face image illustrates extensive details about the specification, gender, age, and emotional states of mind. Facial expressions play an important role in community-based interactions and are often used in the behavioral analysis of emotions. Recognition of automatic facial expressions from a facial image is a challenging task in the computer vision community and admits a large set of applications, such as driver safety, human-computer interactions, health care, behavioral science, video conferencing, cognitive science, and others. In this work, a deep-learning-based scheme is proposed for identifying the facial expression of a person. The proposed method consists of two parts. The former one finds out local features from face images using a local gravitational force descriptor, while, in the latter part, the descriptor is fed into a novel deep convolution neural network (DCNN) model. The proposed DCNN has two branches. The first branch explores geometric features, such as edges, curves, and lines, whereas holistic features are extracted by the second branch. Finally, the score-level fusion technique is adopted to compute the final classification score. The proposed method along with 25 state-of-the-art methods is implemented on five benchmark available databases, namely, Facial Expression Recognition 2013, Japanese Female Facial Expressions, Extended CohnKanade, Karolinska Directed Emotional Faces, and Real-world Affective Faces. The databases consist of seven basic emotions: neutral, happiness, anger, sadness, fear, disgust, and surprise. The proposed method is compared with existing approaches using four evaluation metrics, namely, accuracy, precision, recall, and f1-score. The obtained results demonstrate that the proposed method outperforms all state-of-the-art methods on all the databases.

ExpressionHash: Securing Telecare Medical Information Systems Using BioHashing

The COVID-19 outbreak and its medical distancing phenomenon have effectively turned the global healthcare challenge into an opportunity for Telecare Medical Information Systems. Such systems employ the latest mobile and digital technologies and provide several advantages like minimal physical contact between patient and healthcare provider, easy mobility, easy access, consistent patient engagement, and cost-effectiveness. Any leakage or unauthorized access to users’ medical data can have serious consequences for any medical information system. The majority of such systems thus rely on biometrics for authenticated access but biometric systems are also prone to a variety of attacks like spoofing, replay, Masquerade, and stealing of stored templates. In this article, we propose a new cancelable biometric approach which has tentatively been named as “Expression Hash” for Telecare Medical Information Systems. The idea is to hash the expression templates with a set of pseudo-random keys which would provide a unique code (expression hash). This code can then be serving as a template for verification. Different expressions would result in different sets of expression hash codes, which could be used in different applications and for different roles of each individual. The templates are stored on the server-side and the processing is also performed on the server-side. The proposed technique is a multi-factor authentication system and provides advantages like enhanced privacy and security without the need for multiple biometric devices. In the case of compromise, the existing code can be revoked and can be directly replaced by a new set of expression hash code. The well-known JAFFE (The Japanese Female Facial Expression) dataset has been for empirical testing and the results advocate for the efficacy of the proposed approach.

Facial Expression Recognition with LBP and ORB Features.

Emotion plays an important role in communication. For human–computer interaction, facial expression recognition has become an indispensable part. Recently, deep neural networks (DNNs) are widely used in this field and they overcome the limitations of conventional approaches. However, application of DNNs is very limited due to excessive hardware specifications requirement. Considering low hardware specifications used in real-life conditions, to gain better results without DNNs, in this paper, we propose an algorithm with the combination of the oriented FAST and rotated BRIEF (ORB) features and Local Binary Patterns (LBP) features extracted from facial expression. First of all, every image is passed through face detection algorithm to extract more effective features. Second, in order to increase computational speed, the ORB and LBP features are extracted from the face region; specifically, region division is innovatively employed in the traditional ORB to avoid the concentration of the features. The features are invariant to scale and grayscale as well as rotation changes. Finally, the combined features are classified by Support Vector Machine (SVM). The proposed method is evaluated on several challenging databases such as Cohn-Kanade database (CK+), Japanese Female Facial Expressions database (JAFFE), and MMI database; experimental results of seven emotion state (neutral, joy, sadness, surprise, anger, fear, and disgust) show that the proposed framework is effective and accurate.

Optimal feature selection-based biometric key management for identity management system: Emotion oriented facial biometric system

Identity management systems with biometric key binding make digital transactions secure and reliable. A novel methodology is proposed to develop an intelligent key management system using facial emotions. Key binding with facial emotions makes use of an intrinsic user specific trait facilitating a more natural computer to human interaction. The proposed system utilizes metaheuristic swarm intelligence based optimization techniques to extract optimal features. The work demonstrates key binding by encrypting an image with a secret key bound to optimal features extracted from facial emotions. Efficiency and correctness of proposed key management is validated by successful decryption at receiving end with any one of the enrolled emotions given as input. Deer Hunting Optimization Algorithm and Chicken Swarm Optimization are merged to select optimal features from facial emotions. The derived algorithm is called Fitness Sorted Deer Hunting Optimization Algorithm with Rooster Update. Seven facial emotions — anger, disgust, fear, happiness, sadness, surprise and neutral are used to extract optimal features from Japanese Female Facial Expressions and Yale Facial datasets to train the neural network. Proposed work achieved better performance results over state-of-art optimization algorithms such as whale optimization algorithm, grey wolf optimization, chicken swarm optimization and deer hunting optimization algorithm. Accuracy of proposed model is 2.2% better than deer hunting optimization algorithm and 12.3% better than chicken swarm optimization for a key length 80.

Classifying Facial Expression using Convolution Neural Network

The human-computer interaction system is a success by deriving an effective facial expression recognition function. But it remains a difficult activity to understand facial speech. This paper sets out a novel Recognition of facial expression approach to the task. The approach proposed is motivated by the performance of the Convolutional Neural Networks (CNN) on the face trouble with identification. Unlike other plays, we focus on having good accuracy while requiring only a small sample data for training. The proposed approach is tested on Japanese Female Facial Expression (JAFFE). The accuracy increased compared with state-of-art results on the JAFEE dataset, where it achieved 95%.

Accurate computing of facial expression recognition using a hybrid feature extraction technique

Facial expression recognition (FER) serves as an essential tool for understanding human emotional behaviors. Facial expressions provide a wealth of information about intentions, emotions, and other inner states. Over the past two decades, the development of an automatic FER device has become one of the most demanding multimedia research areas in human–computer interaction systems. Several automatic systems have been introduced and have achieved precise identification accuracies. Due to the complex nature of the human face, however, problems still exist. Researchers are still struggling to develop effective methods for extracting features from images because of unclear features. This work proposes a methodology that improves high-performance computing in terms of the facial expression recognition accuracy. To achieve the goal of high accuracy, a hybrid method is proposed using the dual-tree m-band wavelet transform (DTMBWT) algorithm based on energy, entropy, and gray-level co-occurrence matrix (GLCM). It is accompanied by the use of a Gaussian mixture model (GMM) as the classification scheme to provide efficient identification of database images in terms of facial expressions. Using the DTMBWT, it is possible to derive many expression features from decomposition levels 1 to 6. Moreover, along with the GLCM features, the contrast and homogeneity features can be retrieved. All the features are eventually categorized and recognized with the aid of the GMM classifier. The proposed algorithms are tested using Japanese Female Facial Expression (JAFFE) database with seven different facial expressions: happiness, sadness, anger, fear, neutral, surprise, and disgust. The results of the experiments show that the highest precision of the proposed technique is 99.53%, which is observed at the 4th decomposition level of the DTMBWT.

A multi-scale and rotation-invariant phase pattern (MRIPP) and a stack of restricted Boltzmann machine (RBM) with preprocessing for facial expression classification

In facial expression recognition applications, the classification accuracy decreases because of the blur, illumination and localization problems in images. Therefore, a robust emotion recognition technique is needed. In this work, a Multi-scale and Rotation-Invariant Phase Pattern (MRIPP) is proposed. The MRIPP extracts the features from facial images, and the extracted patterns are blur-insensitive, rotation-invariant and robust. The performance of classification algorithms like Fisher faces, Support Vector Machine (SVM), Extreme Learning Machine (ELM), Convolutional Neural Network (CNN) and Deep Neural Network (DNN) are analyzed. In order to reduce the time for classification, an OPTICS-based pre-processing of the features is proposed that creates a non-redundant and compressed training set to classify the test set. Ten-fold cross validation is used in experimental analysis and the performance metric classification accuracy is used. The proposed approach has been evaluated with six datasets Japanese Female Facial Expression (JAFFE), Cohn Kanade (CK +), Multi- media Understanding Group (MUG), Static Facial Expressions in the Wild (SFEW), Oulu-Chinese Academy of Science, Institute of Automation (Oulu-CASIA) and Man–Machine Interaction (MMI) datasets to meet a classification accuracy of 98.2%, 97.5%, 95.6%, 35.5%, 87.7% and 82.4% for seven class emotion detection using a stack of Restricted Boltzmann Machines(RBM), which is high when compared to other latest methods.

AutoFER: PCA and PSO based automatic facial emotion recognition

Automatic emotion recognition is a critical part of human-machine interactions. Reflection of emotions and to develop its understanding is crucial to provide dealings across human beings and machine frameworks. This work determines an automatic system that distinguishes different emotions connoted on the face. The framework is deliberated to apply the hybridization of feature extraction and optimization using PCA and PSO, respectively, to accomplish a high precision rate. PCA is used to get high-quality feature vectors for each category of emotion. Swarm intelligence, optimization is applied to get an optimized feature vector which is essential for classifying the features in the testing phase. For exploratory work, the authors have considered the Japanese Female Facial Expression (JAFFE) dataset. A maximum classification rate of 94.97% is achieved with the proposed technique. The proposed framework execution is assessed in terms of the false rejection rate, false acceptance rate, and accuracy.

Automatic facial recognition using VGG16 based transfer learning model

Detecting and analyzing emotions from human facial movement is a problem defined and developed over many years for the benefits it brings. Facial expression is a crux of the human-computer interaction (HCI) research area. Researchers are exploring its application in security, medical science, and to know the behavior of a person or community. In this paper, we have proposed a deep learning-based framework using transfer learning for facial expression. This approach uses the existing VGG16 model for a modified trained model and concatenates additional layers on it. VGG16 model is already trained on ImageNet, which has 1000 classes. After this, the model has been verified on CK+, JAFFE benchmark datasets. Extended Cohn-Kanade (CK+) and Japanese Female Facial Expression (JAFFE) are popular Facial Expression Dataset. The proposed model has shown 94.8% accuracy on CK+ and 93.7% on the JAFFE dataset and found superior to existing techniques. We have implemented proposed technique on Google Colab-GPU that has helped us to process these data.