Facial Expressions Classification based on Broad Learning Network

Human facial behaviour recognition can be defined as the process of identifying human internal feelings or mood from the classification of facial expression and gesture. Human facial expression and gesture recognition have a many real world applications such as Human Machine Intelligent Interaction (HMII), Smart rooms, Advance Driver Assistance Systems (ADAS), Intelligent Robotics, Monitoring and Surveillance, Gaming, Research on pain and depression, Health support appliances. Facial Expression Recognition is challenging problem up till now because of many reasons, moreover, it consists of three sub challenging tasks face detection, facial feature extraction and expression classification. Soft computing is a computer science field that applies to the problem whose solution is unpredictable or inexact. Digital image processing works effectively together with soft computing techniques to improve efficiency of recognizing human facial behaviour through machine. This paper gives a review on the mechanisms of human facial behavior recognition using soft computing techniques, which includes a brief detail on framework, literature survey and key issues in facial behaviour recognition using soft computing.

How to build a system for robust classification and recognition of facial expressions has been one of the most important research issues for successful interactive computing applications. However, previous datasets and studies mainly focused on facial expression recognition in a controlled/lab setting, therefore, could hardly be generalized in a more practical and real-life environment. The Affective Behavior Analysis in-the-wild (ABAW) 2022 competition released a dataset consisting of various video clips of facial expressions in-the-wild. In this paper, we propose a method based on the ensemble of multi-head cross attention networks to address the facial expression classification task introduced in the ABAW 2022 competition. We built a uni-task approach for this task, achieving the average F1-score of 34.60 on the validation set and 33.77 on the test set, ranking second place on the final leaderboard.

In recent facial expression recognition competitions, top approaches were using either geometric relationships that best captured facial dynamics or an accurate registration technique to develop texture features. These two methods capture two different types of facial information that is similar to how the human visual system divides information when perceiving faces. This paper discusses a framework of a fully automated comprehensive facial expression detection and classification. We study the capture of facial expressions through geometric and texture-based features, and demonstrate that a simple concatenation of these features can lead to significant improvement in facial expression classification. Each type of expression has individual differences in the commonality of facial expression features due to differences in appearance and other factors. The geometric feature tends to emphasize the facial parts that are changed from the neutral and peak expressions, which can represent the common features of the expression, thus reducing the influence of the difference in appearance and effectively eliminating the individual differences. Meanwhile, the consolidation of gradient-level normalized cross correlation and Gabor wavelet is utilized to present the texture features. We perform experiments using the well-known extended Cohn-Kanade (CK+) database, compared to the other state of the art algorithms, the proposed method achieved provide better performance with an average accuracy of 95.3%.KeywordsFacial expression recognitionGeometry featureTexture feature

The classification of facial expression has been extensively studied using adult facial images which are not appropriate ground truths for classifying facial expressions in children. The state-of-the-art deep learning approaches have been successful in the classification of facial expressions in adults. A deep learning model may be better able to learn the subtle but important features underlying child facial expressions and improve upon the performance of traditional machine learning and feature extraction methods. However, unlike adult data, only a limited number of ground truth images exist for training and validating models for child facial expression classification and there is a dearth of literature in child facial expression analysis. Recent advances in transfer learning methods have enabled the use of deep learning architectures, trained on adult facial expression images, to be tuned for classifying child facial expressions with limited training samples. The network will learn generic facial expression patterns from adult expressions which can be fine-tuned to capture representative features of child facial expressions. This work proposes a transfer learning approach for multi-class classification of the seven prototypical expressions including the ‘neutral’ expression in children using a recently published child facial expression data set. This work holds promise to facilitate the development of technologies that focus on children and monitoring of children throughout their developmental stages to detect early symptoms related to developmental disorders, such as Autism Spectrum Disorder (ASD).

Facial expressions classification (FEC) software has usually been based upon the analysis of visible-spectrum images. Little work has been done on the use of infrared thermal imaging (IRTI) in this area. We report ongoing work on the use of IRTI for FEC. We have identified thermally significant points on human faces, termed facial thermal feature points (FTFPs) and have discovered that variances in thermal intensity values (TIVs) recorded at these FTFPs can help classify common intentional facial expressions. Using multivariate tests and linear discriminant analysis, we examined whether it is possible to distinguish between faces on the basis of TIVs for FEC. Results show that TIVs provide a viable set of thermal data that can be used to classify intentional facial expressions of happiness, sadness and disgust. IRTI may provide an alternative, or be complementary, to visible-spectrum based FEC techniques. IRTI also promises nonintrusive facial feature extraction and FEC in low illumination and image quality conditions.

Humans use their faces to express their emotions and intentions in a simple and natural way. Face expressions are the essential components of nonverbal communication. In human-computer interaction and affective computing, facial expression recognition has various applications. There are several methods devised for recognition and classification of facial expression; still, the accurate recognition is the challenging task. Hence, in this research an automatic facial expression recognition and classification based on deep learning is introduced. Initially the input image is collected from facial expression recognition dataset. Then, the collected image is fed into pre-processing using Gaussian filtering which is used for noise reduction. Then the pre-processed images are given to feature extraction phase using Gray-Level Co-Occurrence Matrix (GLCM). GLCM is used to extract texture features for the facial expression recognition. Then the EfficientNet-B7 is utilized for the recognition and classification of facial expression due to the enhanced outcome with faster inference and smaller size. The proposed IC_EfficientNet method combines the gannet’s ability to capture food with the coot bird’s ability to forage. The optimization technique achieves higher convergence rates due to this hybridization, which improves the EfficientNet-B7 model's parameter tuning. In comparison to existing techniques, the hybrid Improved Coot (IC) algorithm balances exploration and exploitation, leading to a quicker and more effective optimization process. The proposed IC_EfficientNet provides better results compared to existing methods such as Deep Neural Network (DNN), MultiLayer Perceptron (MLP) neural network, Facial Detection using a Convolutional Neural Network (FD-CNN) and Convolutional Neural Network (CNN). Thus, the proposed IC_EfficientNet provides the better outcome in terms of Accuracy, Specificity, Precision, Recall, F1-Measure, and MSE acquired the better outcome of 99.13, 98.80, 97.80, 99.13, 98.44, and 0.87 respectively.

This paper addresses the problems of recognition and the classification of the facial expressions from videos. Currently there are excellent results focusing on the control environments, where artificial facial expressions are found. It is by far the largest database of facial expression, valence, and arousal in the wild enabling research in automated facial expression recognition in two different emotion models. On the other hand, much remains to be improved when it comes to the uncontrolled environments, in which variations in lighting, camera angle, face framing, make the small amount of labelled data available in impediment when the training models of automated learning. In order to attack this difficulty, the Reproductive Confrontational Networks technique was used in an innovative way, which allows a large number of unlabelled images to be used with a semi-supervised training style. In this paper; nearly half of the retrieved images were manually annotated for the presence of seven discrete facial expressions and the intensity of valence and arousal. From facial expressions, as well as the primary theoretical frameworks that have been offered to explain these patterns, we propose that this is an area of inquiry that would benefit from an ecological approach in which contextual elements are more explicitly considered and reflected in experimental methods and may suggest heretofore unexplored underlying mechanisms.

This paper addresses the problems of recognition and the classification of the facial expressions from videos. Currently, there are excellent results focusing on the control environments where artificial facial expressions are found. It is by far the largest database of facial expression, valence, and arousal in the wild enabling research in automated facial expression recognition in two different emotion models. On the other hand, much remains to be improved when it comes to the uncontrolled environments, in which variations in lighting, camera angle, and face framing make the small amount of labeled data available in impediment when the training models of automated learning. In order to attack this difficulty, the reproductive confrontational networks technique was used in an innovative way, which allows a large number of unlabeled images to be used with a semi-supervised training style. In this paper, nearly half of the retrieved images were manually annotated for the presence of seven discrete facial expressions and the intensity of valence and arousal. From facial expressions, as well as the primary theoretical frameworks that have been offered to explain these patterns, we propose that this is an area of inquiry that would benefit from an ecological approach in which contextual elements are more explicitly considered and reflected in experimental methods and may suggest heretofore unexplored underlying mechanisms.KeywordsFacial expressionBiometric systemArtificial neural networksGANSSFEW

Multimodal information-based broad and deep learning model (MIBDL) for emotion understanding is proposed, in which facial expression and body gesture are used to achieve emotional states recognition for emotion understanding. It aims to understand coexistence multimodal information in human-robot interaction by using different processing methods of deep network and broad network, which obtains the features of depth and width dimensions. Moreover, random mapping in the initial broad learning network could cause information loss and its shallow layer network is difficult to cope with complex tasks. To address this problem, we use principal component analysis to generate the nodes of the broad learning, and the stacked broad learning network is adapted to make it easier for the existing broad learning networks to cope with complex tasks by creating deep variations of the existing network. To verify the effectiveness of the proposal, experiments completed on benchmark database of spontaneous emotion expressions are developed, and experimental results show that the proposal outperforms the state-of-the-art methods. According to the simulation experiments on the FABO database, by using the proposed method, the multimodal recognition rate is 17,54%, 1.24%, and 0.23% higher than those of the temporal normalized motion and appearance features(TN), the multi-channel CNN (MCCNN), and the hierarchical classification fusion strategy (HCFS), respectively.

In this article, we introduce a new neonatal facial expression database for pain analysis. This database, called facial expression of neonatal pain (FENP), contains 11,000 neonatal facial expression images associated with 106 Chinese neonates from two children's hospitals, i.e., the Children's Hospital Affiliated to Nanjing Medical University and Second Affiliated Hospital Affiliated to Nanjing Medical University in China. The facial expression images cover four categories of facial expressions, i.e., severe pain expression, mild pain expression, crying expression and calmness expression, where each category contains 2750 neonatal facial expression images. Based on this database, we also investigate the pain facial expression recognition problem using several state-of-the-art facial expression features and expression recognition methods, such as Gabor+SVM, LBP+SVM, HOG+SVM, LBP+HOG+SVM, and several Convolutional Neural Network (CNN) methods (including AlexNet, VGGNet, GoogLeNet, ResNet and DenseNet). The experimental results indicate that the proposed neonatal pain facial expression database is very suitable for the study of both neonatal pain and facial expression recognition. Moreover, the FENP database is publicly available after signing a license agreement (the users can contact Jingjie Yan (yanjingjie@njupt.edu.cn), Guanming Lu (lugm@njupt.edu.cn)) or Xiaonan Li (xnli@njmu.edu.cn).

The recognition and classification of facial expressions using artificial intelligence (AI) presents a promising avenue for early detection and monitoring ofneurodegenerative disorders. This narrative review critically examines the current state of AI-driven facial expression analysis in the context of neurodegenerative diseases, such as Alzheimer's and Parkinson's. We discuss thepotential of AI techniques, including deep learning andcomputer vision, to accurately interpret and categorize subtle changes in facial expressions associated with thesepathological conditions. Furthermore, we explore theroleof facial expression recognition as a noninvasive, cost-effective tool for screening, disease progression tracking, and personalized intervention in neurodegenerative disorders. The review also addresses the challenges, ethical considerations, and future prospects of integrating AI-based facial expression analysis into clinical practice forearly intervention and improved quality of life for individuals at risk ofor affected by neurodegenerative diseases.

Intelligent facial expression recognition and classification using optimal deep transfer learning model

Just as humans use body language or nonverbal language such as gestures and facial expressions in communication, computers will also be able to communicate with humans. In medical engineering, it is possible that recognition of facial expression can be applied to support communication with persons who have trouble communicating verbally such as infants and mental patients. The purpose of this study is to enable recognition of human emotions by facial expressions. Our observations of facial expressions found that recognizing facial expressions by identifying changes in important facial segments such as the eyebrow, the eyes and the mouth by using sequences of images is important. Self-organizing maps, which are neural networks, are used to extract features of image sequences. The image sequences of six types of facial expressions are recorded on VTR and made into image sequences consisting of 30 images per second. Gray levels of each segment are input into the self-organizing map corresponding to each segment. The neuron in the output layer, called the victory neuron, reacts to the feature nearest the input segment. Our analysis of the changes in victory neurons demonstrates that they have characteristic features which correspond to each of the six facial expressions.

Cameras constantly capture and track facial images and videos on cell phones, webcams etc. In the past decade, facial expression classification and recognition has been the topic of interest as facial expression analysis has a wide range of applications such as intelligent tutoring system, systems for psychological studies etc. This study reviews the latest advances in the algorithms and techniques used in distinct phases of real-time facial expression recognition. Though there are state-of-art approaches to address facial expression identification in real-time, many issues such as subjectivity-removal, occlusion, pose, low resolution, scale, variations in illumination level and identification of baseline frame still remain unaddressed. Attempts to deal with such issues for higher accuracy lead to a trade-off in efficiency. Furthermore, the goal of this study is to elaborate on these issues and highlight the solutions provided by the current approaches. This survey has helped the authors to understand that there is a need for a better strategy to address these issues without having to trade-off performance in real-time.

Quantum-inspired meta-heuristic algorithms with deep learning for facial expression recognition under varying yaw angles