Expression Recognition Task Research Articles

Accurately detecting happy facial expressions associates with perceptions of flirtatiousness

Emotional facial expressions are relevant to flirtation because they provide information on an individual’s intentions or motivations. Individual differences in the ability to accurately detect and discriminate between normative facially expressed emotions could lead to misperceptions of the level of sexual interest being conveyed, which has been linked to sexual assault and harassment. To explore this notion, we recruited a national sample of college aged male and female participants ( N = 219) who completed a novel facial expression recognition task used to detect accuracy in processing facial emotions of happiness, surprise, anger, and disgust. Participants also viewed multiple video clips of blind dates between two different-sex participants and rated each partner on their degree of flirtatiousness. Consistent with predictions, we found that individuals who misidentified other facial emotions for happiness appeared to overestimate flirtation. Though not predicted, participants who failed to accurately identify happy faces also overestimated flirtation, whereas individuals who took longer to respond to emotional facial expressions and misidentified an emotion as conveying happiness made greater errors in perceptions of flirtatiousness. Overall, these findings suggest that individual differences in the ability to detect and discriminate happiness through facial expressions are relevant to misperceptions of flirtatious behavior, and more broadly illuminates the role of basic emotion recognition on perceptions of flirtatiousness.

A DIGITAL PRINTING APPLICATION AS AN EXPRESSION IDENTIFICATION SYSTEM

Human Computer Interaction (HCI), a growing research field in science and engineering, aims to provide a natural way for humans to use computers as tools. Humans prefer to interact with each other mainly through speech, but also through facial expressions and gestures, for certain parts of the speech and displays of emotions. The identity, age, gender, and emotional state of a person can be obtained from his face. The impression we receive from the expression reflected on the face affects our interpretation of the spoken word and even our attitude towards the speaker himself. Although emotion recognition is an easy task for humans, it still proves to be a difficult task for computers to recognize user`s emotional state. Advances in this area promise to arm our technological environment by means for more effective interactions with humans, and hopefully the impact of facial expressions on cognition will increase rapidly in the future. Will do. In recent years, the adoption of digital has increased rapidly, and the quality has improved significantly. Digital printing has resulted in fast delivery and needs-based costs. This article describes a sophisticated combination classifier approach, an empirical study of ensembles, stacking, and voting. These three approaches were tested on Nave Bayes (NB), Kernel Naive Bayes (kNB), Neural Network (NN), Auto MultiLayer Perceptron (Auto MLP), and Decision Tree (DT), respectively. The main contribution of this paper is the improvement of the classification accuracy of facial expression recognition tasks. In both persondependent and nonpersondependent experiments we showed that using a combination of these classifier combinations gave significantly better results than using individual classifiers. It has been observed from experiments that the overall voting technique by voting achieves the best classification accuracy.

FACIAL EMOTION RECOGNITION SYSTEM USING CONVOLUTION NEURAL NETWORK

Automatic facial expression recognition is an actively emerging research in Emotion Recognition. This project extends the deep Convolutional Neural Network (CNN) approach to facial expression recognition task. This task is done by detecting the occurrence of facial Action Units(AUs) as a sub part of Facial Action Coding System (FACS) which represents human emotion. In the CNN fully-connected layers. This research uses the extended Cohn Kanade (CK+) dataset which is collected for facial expression recognition experiment

A DIGITAL PRINTING APPLICATION AS AN EXPRESSION IDENTIFICATION SYSTEM.

Human Computer Interaction (HCI), a growing research field in science and engineering, aims to provide a natural way for humans to use computers as tools. Humans prefer to interact with each other mainly through speech, but also through facial expressions and gestures, for certain parts of the speech and displays of emotions. The identity, age, gender, and emotional state of a person can be obtained from his face. The impression we receive from the expression reflected on the face affects our interpretation of the spoken word and even our attitude towards the speaker himself. Although emotion recognition is an easy task for humans, it still proves to be a difficult task for computers to recognize user`s emotional state. Advances in this area promise to arm our technological environment by means for more effective interactions with humans, and hopefully the impact of facial expressions on cognition will increase rapidly in the future. Will do. In recent years, the adoption of digital has increased rapidly, and the quality has improved significantly. Digital printing has resulted in fast delivery and needs-based costs. This article describes a sophisticated combination classifier approach, an empirical study of ensembles, stacking, and voting. These three approaches were tested on Nave Bayes (NB), Kernel Naive Bayes (kNB), Neural Network (NN), Auto MultiLayer Perceptron (Auto MLP), and Decision Tree (DT), respectively. The main contribution of this paper is the improvement of the classification accuracy of facial expression recognition tasks. In both persondependent and nonpersondependent experiments we showed that using a combination of these classifier combinations gave significantly better results than using individual classifiers. It has been observed from experiments that the overall voting technique by voting achieves the best classification accuracy.

A DIGITAL PRINTING APPLICATION AS AN EXPRESSION IDENTIFICATION SYSTEM.

Human Computer Interaction (HCI), a growing research field in science and engineering, aims to provide a natural way for humans to use computers as tools. Humans prefer to interact with each other mainly through speech, but also through facial expressions and gestures, for certain parts of the speech and displays of emotions. The identity, age, gender, and emotional state of a person can be obtained from his face. The impression we receive from the expression reflected on the face affects our interpretation of the spoken word and even our attitude towards the speaker himself. Although emotion recognition is an easy task for humans, it still proves to be a difficult task for computers to recognize user`s emotional state. Advances in this area promise to arm our technological environment by means for more effective interactions with humans, and hopefully the impact of facial expressions on cognition will increase rapidly in the future. Will do. In recent years, the adoption of digital has increased rapidly, and the quality has improved significantly. Digital printing has resulted in fast delivery and needs-based costs. This article describes a sophisticated combination classifier approach, an empirical study of ensembles, stacking, and voting. These three approaches were tested on Nave Bayes (NB), Kernel Naive Bayes (kNB), Neural Network (NN), Auto MultiLayer Perceptron (Auto MLP), and Decision Tree (DT), respectively. The main contribution of this paper is the improvement of the classification accuracy of facial expression recognition tasks. In both persondependent and nonpersondependent experiments we showed that using a combination of these classifier combinations gave significantly better results than using individual classifiers. It has been observed from experiments that the overall voting technique by voting achieves the best classification accuracy.

Facial expression recognition in facial occlusion scenarios: A path selection multi-network

The occlusion scenarios including wearing mask, wearing sunglasses, wearing hat, etc. Thus, parts of face are occluded, e.g. nose and mouth are obscured if a mask is worn. Facial expression recognition (FER) tasks have been widely researched. However, less attention has been paid to FER in occlusion scenarios, which are not uncommon in the real world. In this paper, we propose a method that structures a path selection multi-network model to achieve the FER in the above three types of facial occlusion scenarios. The method contains two parts. For the multi-network, we segment the labels in one database which results in three new sub-databases to train three Subnets, respectively. For the path selection, which is an integration method of multi-network, we merge groups of labels in one database to train an initial network called BeginNet. The prediction of BeginNet selects one of the Subnets to make the final prediction. We concatenate four popular databases Fer2013, JAFFE, KDEF and Raf-DB databases into one larger database, and use the combined database to verify our method effectiveness. The experimental results show that our method has better results in coping with the expression recognition task in multiple types of facial occlusion scenarios.

A Loss Function Base on Softmax for Expression Recognition

Benefiting from deep learning, the accuracy of face expression recognition tasks based on convolutional neural networks has been greatly improved. However, the traditional SoftMax activation function lacks the ability to discriminate between classes. To solve this problem, the industry has proposed several activation functions based on softmax, such as A-softmax, LMCL, etc. We investigate the geometric significance of the weights from a fully connected layer and consider the weights as the class centers. By extracting the feature vector of several samples and extending the corresponding means to the weights, the model can develop the ability to recognize custom classes without training, while maintaining the accuracy of the original classification. On the expression task, the original seven-category classification is validated to obtain 97.10% accuracy on the CK+ dataset and 88% accuracy on the custom dataset.

Early Influence of Emotional Scenes on the Encoding of Fearful Expressions With Different Intensities: An Event-Related Potential Study.

Contextual affective information influences the processing of facial expressions at the relatively early stages of face processing, but the effect of the context on the processing of facial expressions with varying intensities remains unclear. In this study, we investigated the influence of emotional scenes (fearful, happy, and neutral) on the processing of fear expressions at different levels of intensity (high, medium, and low) during the early stages of facial recognition using event-related potential (ERP) technology. EEG data were collected while participants performed a fearful facial expression recognition task. The results showed that (1) the recognition of high-intensity fear expression was higher than that of medium- and low-intensity fear expressions. Facial expression recognition was the highest when faces appeared in fearful scenes. (2) Emotional scenes modulated the amplitudes of N170 for fear expressions with different intensities. Specifically, the N170 amplitude, induced by high-intensity fear expressions, was significantly higher than that induced by low-intensity fear expressions when faces appeared in both neutral and fearful scenes. No significant differences were found between the N170 amplitudes induced by high-, medium-, and low-intensity fear expressions when faces appeared in happy scenes. These results suggest that individuals may tend to allocate their attention resources to the processing of face information when the valence between emotional context and expression conflicts i.e., when the conflict is absent (fear scene and fearful faces) or is low (neutral scene and fearful faces).

Impact of visual impairment on balance and visual processing functions in students with special educational needs.

IntroductionVision is critical for children’s development. However, prevalence of visual impairment (VI) is high in students with special educational needs (SEN). Other than VI, SEN students are prone to having functional deficits. Whether visual problems relate to these functional deficits is unclear. This study aimed to assess the impact of vision on visual processing functions and balance in SEN students through a community service.MethodsVisual acuity (VA) and contrast sensitivity were measured in a total of 104 (aged 14.3±4.3) SEN students as the visual outcomes, followed by retinoscopy. Visual processing function assessment included facial expression recognition by card matching examiner’s facial expression matching, and visual orientation recognition. Dynamic balance, by Timed Up and Go test, and static standing balance (postural sway in double-legged standing with feet-together and tandem-stance for open-eye and closed-eye conditions) were assessed. Static balance was presented in terms of the maximal medial-lateral and antero-posterior sways.ResultsOf the 104 students, 62 (59.6%) were classified as visually impaired according to WHO classification of visual impairment based on presenting distance acuity. Ocular problems (e.g. optic nerve anomaly, uncorrected/ under-corrected refractive errors) and neurological anomalies were the major causes of vision loss. VA was positively associated with visual processing functions (all p ≤ 0.01), as SEN students with better vision tended to perform better in visual orientation and facial expression recognition tasks, as well as dynamic balance function (p = 0.04). For the static balance, postural sway and VA showed a positive relationship under open-eye and tandem stance conditions. However, the relationship between postural sway and VA became negative under closed-eye and tandem stance conditions.ConclusionThis study found a high prevalence of SEN students with visual impairment, in which many of them were undetected. Optometric examination is important to improve their visual function to minimize the effect of vision on functional performance. Vision is critical in visual processing as well as playing an important role in maintaining balance in SEN students.

On-Road Driver Emotion Recognition Using Facial Expression

With the development of intelligent automotive human-machine systems, driver emotion detection and recognition has become an emerging research topic. Facial expression-based emotion recognition approaches have achieved outstanding results on laboratory-controlled data. However, these studies cannot represent the environment of real driving situations. In order to address this, this paper proposes a facial expression-based on-road driver emotion recognition network called FERDERnet. This method divides the on-road driver facial expression recognition task into three modules: a face detection module that detects the driver’s face, an augmentation-based resampling module that performs data augmentation and resampling, and an emotion recognition module that adopts a deep convolutional neural network pre-trained on FER and CK+ datasets and then fine-tuned as a backbone for driver emotion recognition. This method adopts five different backbone networks as well as an ensemble method. Furthermore, to evaluate the proposed method, this paper collected an on-road driver facial expression dataset, which contains various road scenarios and the corresponding driver’s facial expression during the driving task. Experiments were performed on the on-road driver facial expression dataset that this paper collected. Based on efficiency and accuracy, the proposed FERDERnet with Xception backbone was effective in identifying on-road driver facial expressions and obtained superior performance compared to the baseline networks and some state-of-the-art networks.

Multi-level knowledge distillation for low-resolution object detection and facial expression recognition

Recently, remarkable object detection and facial expression recognition (FER) approaches have been made by researchers. However, all of these models are trained and tested on high-resolution images without considering that low-resolution images are more common in practical application. Therefore, to relieve this issue, in this paper we aim to propose a knowledge distillation approach to transfer the learned high-resolution features from a teacher network to a simpler structured student network trained on low-resolution inputs. In our approach, instead of transferring knowledge from the same layers of the teacher and student network, we chose multi-level knowledge of the teacher network to supervise single-level output of the student network. Furthermore, we do not directly use the knowledge from the teacher network, instead, before knowledge transfer we concatenate different level features of the teacher network and figure our what kind of information is important and what is redundant and set different weight value to them. Then we use these knowledge with different weights to guide the output of single layer student network to extract abundant features from the low-resolution images. To evaluate the effectiveness of our proposed approach, we apply this approach to two models for object detection and facial expression recognition tasks. Through our experiments, we find that, in object detection task, the CornerNet achieves an accuracy of 40.6% on the original MC COCO dataset, while this index drops dramatically to only 34.2% on the resolution degraded images. By comparison, our proposed model trained by our knowledge distillation approach achieves 35.4% and 33.4% on the original and resolution degraded datasets, respectively. At the same time, compared to CornerNet the number of layers of the proposed network has been reduced by about 60%. Furthermore, in the task of facial expression recognition and image classification, the similar experimental results can also be observed.

Facial expression recognition based on deep learning

Background and objectiveFacial expression recognition technology will play an increasingly important role in our daily life. Autonomous driving, virtual reality and all kinds of robots integrated into our life depend on the development of facial expression recognition technology. Many tasks in the field of computer vision are based on deep learning technology and convolutional neural network. The paper proposes an occluded expression recognition model based on the generated countermeasure network. The model is divided into two modules, namely, occluded face image restoration and face recognition. MethodsFirstly, this paper summarizes the research status of deep facial expression recognition methods in recent ten years and the development of related facial expression database. Then, the current facial expression recognition methods based on deep learning are divided into two categories: Static facial expression recognition and dynamic facial expression recognition. The two methodswill be introduced and summarized respectively. Aiming at the advanced deep expression recognition algorithms in the field, the performance of these algorithms on common expression databases is compared, and the strengths and weaknesses of these algorithms are analyzed in detail. Discussion and resultsAs the task of facial expression recognition is gradually transferred from the controlled laboratory environment to the challenging real-world environment, with the rapid development of deep learning technology, deep neural network can learn discriminative features, and is gradually applied to automatic facial expression recognition task. The current deep facial expression recognition system is committed to solve the following two problems: (1) Overfitting due to lack of sufficient training data; (2) In the real world environment, other variables that have nothing to do with expression bring interference problems. ConclusionFrom the perspective of algorithm, combining other expression models, such as facial action unit model and pleasure arousal dimension model, as well as other multimodal models, such as audio mode, 3D face depth information and human physiological information, can make expression recognition more practical.

Identity-Aware Facial Expression Recognition Via Deep Metric Learning Based on Synthesized Images

Person-dependent facial expression recognition has received considerable research attention in recent years. Unfortunately, different identities can adversely influence recognition accuracy, and the recognition task becomes challenging. Other adverse factors, including limited training data and improper measures of facial expressions, can further contribute to the above dilemma. To solve these problems, a novel identity-aware method is proposed in this study. Furthermore, this study also represents the first attempt to fulfill the challenging person-dependent facial expression recognition task based on deep metric learning and facial image synthesis techniques. Technically, a StarGAN is incorporated to synthesize facial images depicting different but complete basic emotions for each identity to augment the training data. Then, a deep-convolutional-neural-network-based network is employed to automatically extract latent features from both real facial images and all synthesized facial images. Next, a Mahalanobis metric network trained based on extracted latent features outputs a learned metric that measures facial expression differences between images, and the recognition task can thus be realized. Extensive experiments based on several well-known publicly available datasets are carried out in this study for performance evaluations. Person-dependent datasets, including CK+, Oulu (all 6 subdatasets), MMI, ISAFE, ISED, etc., are all incorporated. After comparing the new method with several popular or state-of-the-art facial expression recognition methods, its superiority in person-dependent facial expression recognition can be proposed from a statistical point of view.

Attenuated alpha-gamma coupling in emotional dual pathways with right-Amygdala predicting ineffective antidepressant response.

AimsThe diversity of treatment outcomes for major depressive disorder (MDD) remains uncertain in neuropathology. The current study aimed at exploring electrophysiological biomarkers associated with treatment response.MethodsThe present study recruited 130 subjects including 100 MDD patients and 30 healthy controls. All subjects participated in a sad expression recognition task while their magnetoencephalography data were recorded. Patients who had a reduction of at least 50% in disorder severity at endpoint (>2 weeks) were considered as responders. Within‐frequency power and phase‐amplitude coupling were measured for the brain regions involved in the emotional visual information processing pathways.ResultsThe significant alpha–gamma decoupling from the right thalamus to the right amygdala in unconscious processing and from right orbital frontal cortices to the right amygdala in conscious processing was found in non‐responders relative to responders and healthy controls. These kinds of dysregulation could also predict the potential treatment response.ConclusionThe attenuated alpha–gamma coupling in dual pathways indicated increased sensitivity to the negative emotional information and reduced moderated effect of the amygdala, which might cause insensitivity to antidepressant treatment and could be regarded as potential neural mechanisms for treatment response prediction.

Impairment of recognition of facial expressions in different types of dementia

AbstractBackgroundImpairment of facial expression recognition in dementia is one of the cognitive deficits that could affect the social life of dementia patients. The severity of the facial expression recognition in different types of dementia has not been fully addressed. Previous studies showed that EEG event‐related oscillation (ERO) studies could reveal the brain dynamics during successful facial expression recognition. Furthermore, the impairment of facial recognition in Alzheimer's disease patients were represented with abnormal EROs (Güntekin et al., 2019). The present study aims to compare the impaired facial expression recognition between different types of dementia by analysis of event‐related delta responses. The role of delta responses in facial expression and emotional paradigms was reported previously by several researchers (Güntekin and Başar, 2014).Method25 Healthy elderly controls (HC), 25 Mild Cognitive Impairment patients (MCI), 25 patients with Alzheimer's disease (AD), 15 patients with Parkinson's disease with Mild Cognitive Impairment (PDMCI) and 16 patients with Parkinson's disease Dementia (PDD) were included in the study. The electroencephalographic activity was recorded during the perception of the facial expression recognition task (Angry, Happy, Neutral). Delta power and phase‐locking were computed for each facial recognition stimuli and compared among the groups (ANOVA, p<0.05).ResultBoth delta power and delta phase locking were lower in dementia groups than healthy controls being worst in PDD. Figure 1 shows the grand average of delta power in all groups for the right occipital location. PDD (p<0.0001) and AD (p<0.05) had lower delta power than the healthy controls, the reduction of delta responses in PDD and AD were especially found over occipital locations. MCI and PDMCI subjects were found less impaired facial expression recognition compared to AD and PDMCI.ConclusionThe present study showed that dementia patients had severe facial recognition deficits that increased cognitive impairment. Event‐related delta responses successfully showed the impaired recognition of facial expression in different types of dementia groups. Among all dementia subjects, PDD patients had the most reduced delta responses that could be an electrophysiological biomarker of impaired facial expression recognition. Acknowledgments: This work (grant number 218S314) was supported by TUBITAK.

Sub-second transient activated patterns to sad expressions in major depressive disorders discovered via hidden Markov model.

The pathological mechanisms of major depressive disorders (MDDs) is associated with the overexpression of negative emotions, and the fast transient-activated patterns underlying overrepresentation in depression still remain to be revealed to date. We hypothesized that the aberrant spatiotemporal attributes of the process of sad expressions are related to the neuropathology of MDD and help to detect the depression severity. We enrolled a total of 96 subjects including 47 patients with MDD and 49 healthy controls (HCs), and recorded their magnetoencephalography data under a sad expression recognition task. A hidden Markov model (HMM) was applied to separate the whole neural activity into several brain states, then to characterize the dynamics. To find the disrupted temporal-spatial characteristics, power estimations and fractional occupancy (FO) of each state were estimated and contrasted between MDDs and HCs. Three states were found over the period of emotional stimuli processing procedure. The early visual stage (0-270ms) was mainly manifested by state 1, and the emotional information processing stage (270-600ms) was manifested by state 2, while the state 3 remained a steady proportion across the whole period. MDDs activated statistically more in limbic system during state 2 (p=0.0045) and less in frontoparietal control network during state 3 (p=5.38×10-5 ) relative to HCs. Hamilton Depression Rating Scale scores were significantly correlated with the predicted disorder severity using FO values (p=0.0062, r=0.3933). Relative to HCs, MDDs perceived the sad contents quickly and spent more time overexpressing the negative emotions. These phenomena indicated MDD patients might easily indulge in negative emotion and neglect other things. Furthermore, temporal descriptors built by HMM could be potential biomarkers for identifying the severity of depression disorders.

Training Emotion Recognition Accuracy: Results for Multimodal Expressions and Facial Micro Expressions.

Nonverbal emotion recognition accuracy (ERA) is a central feature of successful communication and interaction, and is of importance for many professions. We developed and evaluated two ERA training programs—one focusing on dynamic multimodal expressions (audio, video, audio-video) and one focusing on facial micro expressions. Sixty-seven subjects were randomized to one of two experimental groups (multimodal, micro expression) or an active control group (emotional working memory task). Participants trained once weekly with a brief computerized training program for three consecutive weeks. Pre-post outcome measures consisted of a multimodal ERA task, a micro expression recognition task, and a task about patients' emotional cues. Post measurement took place approximately a week after the last training session. Non-parametric mixed analyses of variance using the Aligned Rank Transform were used to evaluate the effectiveness of the training programs. Results showed that multimodal training was significantly more effective in improving multimodal ERA compared to micro expression training or the control training; and the micro expression training was significantly more effective in improving micro expression ERA compared to the other two training conditions. Both pre-post effects can be interpreted as large. No group differences were found for the outcome measure about recognizing patients' emotion cues. There were no transfer effects of the training programs, meaning that participants only improved significantly for the specific facet of ERA that they had trained on. Further, low baseline ERA was associated with larger ERA improvements. Results are discussed with regard to methodological and conceptual aspects, and practical implications and future directions are explored.

Auto-FERNet: A Facial Expression Recognition Network With Architecture Search

Deep convolutional neural networks have achieved great success in facial expression datasets both under laboratory conditions and in the wild. However, most of these related researches use general image classification networks (e.g., VGG, GoogLeNet) as backbones, which leads to inadaptability while applying to Facial Expression Recognition (FER) task, especially those in the wild. In the meantime, these manually designed networks usually have large parameter size. To tackle with these problems, we propose an appropriative and lightweight Facial Expression Recognition Network Auto-FERNet, which is automatically searched by a differentiable Neural Architecture Search (NAS) model directly on FER dataset. Furthermore, for FER datasets in the wild, we design a simple yet effective relabeling method based on Facial Expression Similarity (FES) to alleviate the uncertainty problem caused by natural factors and the subjectivity of annotators. Experiments have shown the effectiveness of the searched Auto-FERNet on FER task. Concretely, our architecture achieves a test accuracy of 73.78% on FER2013 without ensemble or extra training data. And noteworthily, experimental results on CK+ and JAFFE outperform the state-of-the-art with an accuracy of 98.89% (10 folds) and 97.14%, respectively, which also validate the robustness of our system.

Emotion perception and theory of mind in adolescents with major depression.

The research of theory of mind (ToM) and emotion perception (EP) in adolescents with major depressive disorder (MDD) is scarce, and no study to date has investigated the association between EP and long-term outcomes of adolescents with MDD. The aim of the current study was to evaluate ToM and EP in adolescents with MDD, as compared to healthy controls (HCs). In addition, we aimed to assess the association between impairment in ToM and EP, depressive symptom severity, and long-term outcome in the MDD group. We compared the performance of 14 adolescents with MDD and 25 HC in the Facial Expression Recognition Task (FERT) and the Interpersonal Perception Task. We followed up with the MDD group 2 years later to assess the level of their depressive symptoms using the Children's Depression Rating Scale-Revised (CDRS-R). No differences were found between adolescents with MDD and HC in the ToM and FERT tasks. Also, within the MDD group, there was no association between the severity of depressive symptoms and task performance. In the MDD group, there was a significant correlation between lower levels of accuracy in the FERT during the index depressive episode and lower CDRS-R scores on follow-up 2 years later (r2 = 0.35, p = 0.021). EP impairments in adolescents with MDD might predict worse long-term outcome. Further research is needed to verify our findings and to assess for a possible neurobiological underpinning for the state and trait impairments in EP in adolescents with MDD.

The Facial Expression Recognition Method Based on Image Fusion and CNN

Facial expression recognition (FER) is an important task in the field of human-computer interaction. However, the traditional facial expression recognition task needs to be based on the hand-crafted features, and the feature extraction method is single; the facial expression recognition task based on deep learning cannot extract local texture features of the image and loss more information. Therefore, a facial expression recognition method based on image fusion and convolution neural network (FERFC) is proposed in this paper. Which fused the facial expression images after extracted by the local binary pattern (LBP) with the original images. It can effectively improved the utilization of images. Firstly, some image pre-processing approaches are used in this paper, such as data augmentation, face detection and data normalization. Secondly, the images of local texture features extracted by the LBP and the original images are fused in this step. Finally, the task of facial expression features learning and classification is completed by convolution neural network (CNN). The results show that the method proposed in this paper can accomplish the facial expression recognition task accurately. The recognition rate of reference database ‘Jaffe’, ‘CK+’ and ‘FER2013’ is 91.9%, 95.6% and 75.9%. The results show that the FERFC has significant advantages than traditional facial expression recognition. At the same time, the number of training samples is small, the FERFC still has obvious advantages and a higher robustness.