Facial Expression Recognition Technology Research Articles

An Ensemble Kernelized-based Approach for Precise Emotion Recognition in Depressed People

As the COVID-19 pandemic created serious challenges for mental health worldwide, with a noticeable increase in depression cases, it has become important to quickly and accurately assess emotional states. Facial expression recognition technology is a key tool for this task. To address this need, this study proposes a new approach to emotion recognition using the Ensemble Kernelized Learning System (EKLS). Nonverbal cues, such as facial expressions, are crucial in showing emotional states. This study uses the Extended Cohn-Kanade (CK+) dataset, which was enhanced with images and videos from the COVID-19 era related to depression. Each of these images and videos is manually labeled with the corresponding emotions, creating a strong dataset for training and testing the proposed model. Facial feature detection techniques were used along with key facial measurements to aid in emotion recognition. EKLS is a flexible machine-learning framework that combines different techniques, including Support Vector Machines (SVMs), Self-Organizing Maps (SOMs), kernel methods, Random Forest (RF), and Gradient Boosting (GB). The ensemble model was thoroughly trained and fine-tuned to ensure high accuracy and consistency. EKLS is a powerful tool for real-time emotion recognition in both images and videos, achieving an impressive accuracy of 99.82%. This study offers a practical and effective approach to emotion recognition and makes a significant contribution to the field.

Research Advances in Facial Expression Recognition Technology

As a key area in artificial intelligence and computer vision, facial expression analysis technology has been taking hold rapidly, yielding dazzling advances. This technology extracts even the subtlest facial movements to identify when a human is sad, happy, or angry. It has extensive application value in markets like mental diagnosis, secure monitoring, and intelligent interaction. The paper will provide a brief overview of the kernel technology of facial expressions recognition: geometric features, appearance features, and deep learning. And compared the experimental results of different experiments, as a goal, the experiments compare functions of several methods in a multi-dataset context, concentrating the primary problems in the tasks like dataset bias, real-time requirements, and different light levels and partial occlusions in various environments. Eventually, the paper forecasts the imminent growth trend in data fusion and deep learning improvement. This also indicates that emerging technologies, especially in terms of applications, will have the upper hand.

Introducing a novel dataset for facial emotion recognition and demonstrating significant enhancements in deep learning performance through pre-processing techniques

Facial expression recognition (FER) plays a pivotal role in various applications, ranging from human-computer interaction to psychoanalysis. To improve the accuracy of facial emotion recognition (FER) models, this study focuses on enhancing and augmenting FER datasets. It comprehensively analyzes the Facial Emotion Recognition dataset (FER13) to identify defects and correct misclassifications. The FER13 dataset represents a crucial resource for researchers developing Deep Learning (DL) models aimed at recognizing emotions based on facial features. Subsequently, this article develops a new facial dataset by expanding upon the original FER13 dataset. Similar to the FER + dataset, the expanded dataset incorporates a wider range of emotions while maintaining data accuracy. To further improve the dataset, it will be integrated with the extended Cohn-Kanade (CK+) dataset.This paper investigates the application of modern DL models to enhance emotion recognition in human faces. By training a new dataset, the study demonstrates significant performance gains compared with its counterparts. Furthermore, the article examines recent advances in FER technology and identifies critical requirements for DL models to overcome the inherent challenges of this task effectively. The study explores several DL architectures for emotion recognition in facial image datasets, with a particular focus on convolutional neural networks (CNNs). Our findings indicate that complex architecture, such as EfficientNetB7, outperforms other DL architectures, achieving a test accuracy of 78.9 %. Notably, the model surpassed the EfficientNet-XGBoost model, especially when used with the new dataset. Our approach leverages EfficientNetB7 as a backbone to build a model capable of efficiently recognizing emotions from facial images. Our proposed model, EfficientNetB7-CNN, achieved a peak accuracy of 81 % on the test set despite facing challenges such as GPU memory limitations. This demonstrates the model's robustness in handling complex facial expressions. Furthermore, to enhance feature extraction and attention mechanisms, we propose a new hybrid model, CBAM-4CNN, which integrates the convolutional block attention module (CBAM) with a custom 4-layer CNN architecture. The results showed that the CBAM-4CNN model outperformed existing models, achieving higher accuracy, precision, and recall metrics across multiple emotion classes. The results highlight the critical role of comprehensive and diverse data in enhancing model performance for facial emotion recognition.

BFFN: A novel balanced feature fusion network for fair facial expression recognition

Facial expression recognition (FER) technology has become increasingly mature and applicable in recent years. However, it still suffers from the bias of expression class, which can lead to unfair decisions for certain expression classes in applications. This study aims to mitigate expression class bias through both pre-processing and in-processing approaches. First, we analyze the output of existing models and demonstrate the existence of obvious class bias, particularly for underrepresented expressions. Second, we develop a class-balanced dataset constructed through data generation, mitigating unfairness at the data level. Then, we propose the Balanced Feature Fusion Network (BFFN), a class fairness-enhancing network. The BFFN mitigates the class bias by adding facial action units (AU) to enrich expression-related features and allocating weights in the AU feature fusion process to improve the extraction ability of underrepresented expression features. Finally, extensive experiments on datasets (RAF-DB and AffectNet) provide evidence that our BFFN outperforms existing FER models, improving the fairness by at least 16%.

A facial expression recognition network based on attention double branch enhanced fusion.

The facial expression reflects a person's emotion, cognition, and even physiological or mental state to a large extent. It has important application value in medical treatment, business, criminal investigation, education, and human-computer interaction. Automatic facial expression recognition technology has become an important research topic in computer vision. To solve the problems of insufficient feature extraction, loss of local key information, and low accuracy in facial expression recognition, this article proposes a facial expression recognition network based on attention double branch enhanced fusion. Two parallel branches are used to capture global enhancement features and local attention semantics respectively, and the fusion and complementarity of global and local information is realized through decision-level fusion. The experimental results show that the features extracted by the network are made more complete by fusing and enhancing the global and local features. The proposed method achieves 89.41% and 88.84% expression recognition accuracy on the natural scene face expression datasets RAF-DB and FERPlus, respectively, which is an excellent performance compared with many current methods and demonstrates the effectiveness and superiority of the proposed network model.

A lightweight and continuous dimensional emotion analysis system of facial expression recognition under complex background

Facial expression recognition technology has a brilliant prospect in applying the Internet of Things systems. Concerning the limited hardware computing capability and high real-time processing requirements, this paper proposes a lightweight emotion analysis system based on edge computing, which could be deployed on edge devices. To further improve the accuracy of dimensional emotion analysis, we propose a modified network structure of MobileNetV3 to measure the intensity of dimensional emotion. The optimization scheme includes introducing the improved efficient channel attention mechanism and the feature pyramid network, adjusting the structure of the model, and optimizing the loss function. Furthermore, the system uses Intel’s OpenVINO toolkit to make the model more suitable for stable operation and provides an operable human–computer interaction interface. The experimental results show that the system has the advantages of few parameters, high recognition accuracy, and low latency. The optimized network size is reduced by 67% compared with the original, the root mean square error of potency and activation is 0.413 and 0.389, and the latency is up to 9ms in Myriad. This work meets the requirements of practical applications and has essential significance with the demand for continuous dimensional emotion analysis. The source code is available at https://github.com/SCNU-RISLAB/Lightweight-and-Continuous-Dimensional-Emotion-Analysis-System-of-Facial-Expression-Recognition/.

An Empirical Study of Portrait Rights on Emotional Evolution in Virtual Social Scenarios by Transformer and Cloud Computing

Facial expression recognition technology can improve the possibility of fugitives and persons subject to execution being discovered by public security and judicial personnel in the virtual world, thereby improving the actual effectiveness and credibility of the law. There may also be a risk of portrait rights infringement with this technology, and the user of the technology needs to inform the person whose facial expression is being extracted in advance and obtain permission. With the rapid progression of deep learning and artificial intelligence, 3D facial expression modeling has garnered increased significance in computer vision and graphics. We propose an innovative approach combining Transformer models with cloud computing to simulate facial expression evolution in virtual social environments. Leveraging Transformer-based feature extraction, our model integrates emotional cues from various modalities to accurately capture subtle changes over time.

Revolutionizing online education: Advanced facial expression recognition for real-time student progress tracking via deep learning model

AbstractThis paper presents a groundbreaking online educational platform that utilizes facial expression recognition technology to track the progress of students within the classroom environment. Through periodic image capture and facial data extraction, the platform employs ResNet50, CBAM, and TCNs for enhanced facial expression recognition. Achieving accuracies of 91.86%, 91.71%, 95.85%, and 97.08% on the RAF-DB, FER2013, CK + , and KDEF expression datasets, respectively, the proposed model surpasses the initial ResNet50 model in accuracy and detection of students' learning states. Comparative evaluations against state-of-the-art models using the RAF-DB, FER2013, CK + , and KDEF datasets underscore the significance of the results for educational institutions. By enhancing emotion recognition accuracy, improving feature relevance, capturing temporal dynamics, enabling real-time monitoring, and ensuring robustness and adaptability in online classroom environments, this approach offers valuable insights for educators to enhance teaching strategies and student outcomes. The combined capabilities of ResNet50, CBAM, and TCNs contribute uniquely to capturing dynamic changes in facial expressions over time, thereby facilitating accurate interpretation of students' emotions and engagement levels for more effective monitoring of learning behaviors in real-time.

Una investigación sobre un sistema de recomendación musical basado en expresiones faciales mediante mecanismos de aprendizaje profundo

In this study, we propose a new music recommendation system (MRS) that combines facial expression recognition technology and deep learning algorithms to respond to the changing music industry environment and provide personalized music recommendations based on the user's emotional state. Our approach includes a thorough study of facial expression recognition, emotion-based music recommendation systems, and deep learning engines, as well as a detailed presentation of the MRS design, system architecture, and deep learning engines used. Through extensive experiments, we evaluate MRS's ability to accurately recognize facial expressions, filter music based on emotional states, and effectively recommend music to users. We analyze the results of follow-up experiments to identify the strengths and limitations of MRS compared to existing approaches, and conduct a comparative study with the latest music recommendation systems based on deep learning and emotion. This comparison highlights the originality and potential of the proposed MRS system to improve user experience and promote the development of artificial intelligence-based music recommendation systems. This study demonstrates the problem of accurately determining a user's emotional state from facial expressions, which requires the integration of facial expression recognition systems, deep learning, and music recommendation systems. Using advanced deep learning techniques and a comprehensive experimental setup, the proposed MRS provides a solution to this problem by facilitating accurate emotional state identification and personalized music recommendations. Overall, MRS represents a powerful and innovative response to the growing demand for accurate and reliable music recommendations and shows significant potential for future collaboration and development of AI-based music recommendation systems.

Application of facial expression recognition technology based on feature fusion in teaching

With the development of artificial intelligence technology, the digital transformation of student-oriented education becomes particularly important. How to promote real-time interaction between teachers and students in the classroom is an urgent issue which is needed to pay attention to. Based on the facial expression features of students in a classroom, this paper analyzes the changes in angles between facial expression feature points using Dlib. Additionally, this paper proposes a novel algorithm for extracting variable scale template edge trend features. The algorithm adaptively processes the template based on the edge trend features of expression feature points, and use the proposed template slope normalization algorithm to achieve multi-scale template edge trend extraction. Then, DNN are used to recognize different listening expressions. The experimental results show that the proposed algorithm has faster recognition speed and better robustness when applied to classroom expression recognition. By identifying students’ class status to remind teachers to adjust their class progress, the goal of improving classroom learning effectiveness is achieved

The positive effect of blue luminescent pathways on urban park visitor’s affective states: A virtual reality online study measuring facial expressions and self-reports

The colour blue often elicits feelings of calmness and contentment, for which evidence has largely been provided in daytime settings. It is unclear whether pathways illuminated in blue, for example, in urban recreational park areas at night confers the same positive impact on night time park visitors. To tackle this open empirical question, we investigated how adding blue self-luminous pavement to park lighting at night time affects park visitors’ emotions compared to conventional white street light illumination. Our goal is to inform design decisions aimed at enhancing the emotional well-being of people outdoors at night in urban environments. Participants’ emotional response was captured at four different time points while watching a video of a walk in a virtual urban park at night, which was lit with white street lights only or with the addition of blue luminescent pavement on the walked paths (between-subject design). To capture visitor’s emotions, we used a simplified version of the Geneva Emotion Wheel (GEW) instrument and online facial expression recognition technology as subjective (self-reports) and objective (physiological) measures of emotion, respectively. The results of the GEW self-reports showed that the addition of a blue self-luminous pavement in a park during night time yielded more positive affect than standard white lighting in park visitors for the first half of the walk. In the second half of the walk through the park, participants’ affective states seemed to equalize between the two lighting conditions. In contrast, sensory data on facial expressions indicated no difference between participants’ emotional states over the whole walk in the two experimental conditions. Consistent with the positive emotional state perceived in the second half of the walk, the state of relaxation experienced after the walk also did not differ between the two lighting conditions. Furthermore, participants’ relaxation judgements after the park walk were more negative overall for females than the more neutral ratings of males. Our results highlight the importance of lighting colour at night for the design of future affect-smart cities that may consider individual and group characteristics with the ultimate intent of promoting public well-being.

Retracted: Construction of Facial Expression Recognition Technology of Intelligent Service Robot Based on Nanofiber Flexible Wearable Electronic Skins

Retracted: Construction of Facial Expression Recognition Technology of Intelligent Service Robot Based on Nanofiber Flexible Wearable Electronic Skins

PAIN ASSESSMENT AND MANAGEMENT FOR OLDER ADULTS WITH DEMENTIA IN LONG-TERM CARE: A SCOPING REVIEW

Abstract Self-reporting is essential for assessing pain. However, owing to their cognitive impairment, older adults with dementia may not always express pain. As such, older adults with dementia are at risk of living with undertreated pain. This scoping review aimed to explore and identify current publication trends and strategies regarding pain assessment and management for older adults with dementia in long-term care. Using the scoping review method of Arksey and O’Malley, two reviewers conducted structured searches for articles published from 2012 to 2022 on four databases (CINAHL, Cochrane, PubMed, RISS) and then independently reviewed the full text for eligibility criteria. The review included 40 articles for analysis and identified 20 observational pain tools used in the studies. A few studies assessed pain in older adults with dementia using facial expression recognition technology and electroencephalography. A common pain management strategy identified for older adults with dementia was the application of protocols or clinical guidelines that included a pain assessment and treatment phase. In addition to pharmacological and non-pharmacological treatment, a therapeutic robot (PARO) and play activities were investigated to determine whether the pain was alleviated in older adults with dementia. Although studies have been conducted to assess and manage pain in older adults with dementia in long-term care, the review highlighted the need for education and training on pain tools and treatment approaches, as well as for the development of pain management strategies and non-pharmacological interventions.

Recognizing facial emotion leveraging convolutional neural network

Facial expression recognition technology is a technology based on the combination of artificial intelligence technology and biology, and it is interdisciplinary research. The appearance of this technology shows the diversity of the development direction and application fields of computer technology, but it also means that the development of facial expression recognition technology needs not only the support of computer technology but also the exploration and progress of biology. At present, to recognize facial expressions a system usually contains three stages, namely, face detection stage, feature extraction stage, and face expression recognition stage. The facial expression recognition algorithm is transformed into a classification problem, and a convolutional neural network (CNN) is adopted, where features of the facial expressions could be automatically extracted. It takes as input a preprocessed image and can directly produce encoded features and predictions, leveraging the end-to-end strategy. It could discard the complicated intermediate modeling process of traditional machine learning. In this work a CNN is implemented for the recognition of facial expression on Fer2013 dataset. Moreover, the effectiveness of different numbers of CNN layers are testified on this problem. It could be concluded that with deeper architectures, the CNN tends to perform better.

Facial expression-enhanced recommendation for virtual fitting rooms

With the development of Augmented Reality (AR) technology in the retail industry, virtual fitting room (VFR) are considered promising enhancement of e-commerce by providing users with an immersive environment to try on new products, especially fashion products. While allowing users having more vivid impression of products, virtual fitting rooms also offer sellers more channels to collect information on user preferences, which can be used to enhance recommender systems. This study proposes to leverage facial expression recognition technology together with fine-grained human-computer interactions in virtual fitting rooms to personalize product recommendations. This paper proposes a recommendation algorithm based on confidence setting, negative feedback sampling, and matrix factorization to model user behaviors in virtual fitting rooms. We conduct an experiment on 81 subjects to evaluate the proposed method. Experimental results show the proposed method outperforms existing methods using traditional behavior information. Our study provides a strong support to the value of AR in enhancing e-commerce.

Facial Expression Recognition Model Based on CNN and Data Augmentation Method

Face expressions are essential in expressing human emotions, and it is accomplished by separating features and categorizing them. Facial expression recognition technology has been widely employed in human-computer interaction, telemedicine, mental health, and criminal investigation detection. In recent years, significant advances in deep learning have facilitated the development of facial expression recognition, making it increasingly accessible. Convolutional neural networks (CNN) are the foundation of the face expression recognition model presented in this article. In order to maximize the final accuracy, an abundance of sample photographs are required for training and optimization. As a result, the 35,886 facial expression photos from the FER2013 dataset, which contains all seven emotions, were used for both training and testing. The photos were scaled down to 4848 pixels during data preparation, and data augmentation was carried out. To get accurate face expression recognition results, various optimizers and parameters were chosen for the training network, which was a bespoke structure based on the VGG network design. The model constructed in this study achieved an accuracy of 73.16% during prediction.

Designing Effective Visual Feedback for Facial Rehabilitation Exercises: Investigating the Role of Shape, Transparency, and Age on User Experience.

Facial expression recognition technology has been utilized both for entertainment purposes and as a valuable aid in rehabilitation and facial exercise assistance. This technology leverages artificial intelligence models to predict facial landmark points and provide visual feedback, thereby facilitating users' facial movements. However, feedback designs that disregard user preferences may cause discomfort and diminish the benefits of exercise. This study aimed to develop a feedback design guide for facial rehabilitation exercises by investigating user responses to various feedback design methods. We created a facial recognition mobile application and designed six feedback variations based on shape and transparency. To evaluate user experience, we conducted a usability test involving 48 participants (24 subjects in their 20s and 24 over 60 years of age), assessing factors such as feedback, assistance, disturbance, aesthetics, cognitive ease, and appropriateness. The experimental results revealed significant differences in transparency, age, and the interaction between transparency and age. Consequently, it is essential to consider both transparency and user age when designing facial recognition feedback. The findings of this study could potentially inform the design of more effective and personalized visual feedback for facial motion, ultimately benefiting users in rehabilitation and exercise contexts.

Facial Expression Recognition Based on Pruning Optimization Technology

Facial expression recognition is based on deep learning, which has become one of the main topics in artificial intelligence. Nevertheless, the complex network structure causes numerous parameter redundancy, which leads to low recognition efficiency. Therefore, researching how to reduce parameter redundancy makes sense. In addition, today's machine expression recognition has some limitations and constraints, while the generalization ability is poor. Aiming at addressing the problems that excessive parameter redundancy brings, for example, low efficiency and massive limitations. This paper will put forward a facial expression recognition technology that is based on pruning optimization to improve existing models. Based on VGGNet, ResNet and SENet, these three convolutional neural networks modify the convolution layer of the network, then add the attention module and pruning optimization. Through face detection, face area feature representation, expression recognition, and other steps. Realize the recognition of multiple facial expressions. The CK+ dataset and JAFFE dataset are independently divided into 4/5 for training and 1/5 for testing. Experimental results show that after inserting CBAM, except ResNet18, the accuracy of the other two networks all improved, SENet18 improved the most significantly. After pruning, the accuracy and speed of VGG16 and ResNet18 improved, but the accuracy of SENet18 decreased. In summary, the optimized model has good applicability and generalization ability.

Facial Motion Capture System Based on Facial Electromyogram and Electrooculogram for Immersive Social Virtual Reality Applications.

With the rapid development of virtual reality (VR) technology and the market growth of social network services (SNS), VR-based SNS have been actively developed, in which 3D avatars interact with each other on behalf of the users. To provide the users with more immersive experiences in a metaverse, facial recognition technologies that can reproduce the user's facial gestures on their personal avatar are required. However, it is generally difficult to employ traditional camera-based facial tracking technology to recognize the facial expressions of VR users because a large portion of the user's face is occluded by a VR head-mounted display (HMD). To address this issue, attempts have been made to recognize users' facial expressions based on facial electromyogram (fEMG) recorded around the eyes. fEMG-based facial expression recognition (FER) technology requires only tiny electrodes that can be readily embedded in the HMD pad that is in contact with the user's facial skin. Additionally, electrodes recording fEMG signals can simultaneously acquire electrooculogram (EOG) signals, which can be used to track the user's eyeball movements and detect eye blinks. In this study, we implemented an fEMG- and EOG-based FER system using ten electrodes arranged around the eyes, assuming a commercial VR HMD device. Our FER system could continuously capture various facial motions, including five different lip motions and two different eyebrow motions, from fEMG signals. Unlike previous fEMG-based FER systems that simply classified discrete expressions, with the proposed FER system, natural facial expressions could be continuously projected on the 3D avatar face using machine-learning-based regression with a new concept named the virtual blend shape weight, making it unnecessary to simultaneously record fEMG and camera images for each user. An EOG-based eye tracking system was also implemented for the detection of eye blinks and eye gaze directions using the same electrodes. These two technologies were simultaneously employed to implement a real-time facial motion capture system, which could successfully replicate the user's facial expressions on a realistic avatar face in real time. To the best of our knowledge, the concurrent use of fEMG and EOG for facial motion capture has not been reported before.

Facial Expression Recognition Using Multi-Branch Attention Convolutional Neural Network

Facial expression recognition technology was extensively used. This paper develops a multi-branch attention convolutional neural network based on a multiple-branch structure to recognize facial expressions. First, features are extracted from facial images in a multi-branch architecture, and features from three branches are fused. Second, to address the issue of insufficient feature extraction and poor recognition performance, the Convolutional Block Attention Module is added as attention module. Third, our model reduces parameters and computation loads by using depth-wise separable convolutions. The experiments on the FER2013, FERPLUS, and CK+ datasets show that the recognition rates of the proposed model are 69.49%, 84.633%, and 99.39%, respectively. The proposed method has a higher efficiency in extracting image features than traditional deep learning counterparts and achieves high accuracy without complicated artificial feature technology.