Expression Recognition Technology Research Articles

An Ensemble Kernelized-based Approach for Precise Emotion Recognition in Depressed People

As the COVID-19 pandemic created serious challenges for mental health worldwide, with a noticeable increase in depression cases, it has become important to quickly and accurately assess emotional states. Facial expression recognition technology is a key tool for this task. To address this need, this study proposes a new approach to emotion recognition using the Ensemble Kernelized Learning System (EKLS). Nonverbal cues, such as facial expressions, are crucial in showing emotional states. This study uses the Extended Cohn-Kanade (CK+) dataset, which was enhanced with images and videos from the COVID-19 era related to depression. Each of these images and videos is manually labeled with the corresponding emotions, creating a strong dataset for training and testing the proposed model. Facial feature detection techniques were used along with key facial measurements to aid in emotion recognition. EKLS is a flexible machine-learning framework that combines different techniques, including Support Vector Machines (SVMs), Self-Organizing Maps (SOMs), kernel methods, Random Forest (RF), and Gradient Boosting (GB). The ensemble model was thoroughly trained and fine-tuned to ensure high accuracy and consistency. EKLS is a powerful tool for real-time emotion recognition in both images and videos, achieving an impressive accuracy of 99.82%. This study offers a practical and effective approach to emotion recognition and makes a significant contribution to the field.

Research Advances in Facial Expression Recognition Technology

As a key area in artificial intelligence and computer vision, facial expression analysis technology has been taking hold rapidly, yielding dazzling advances. This technology extracts even the subtlest facial movements to identify when a human is sad, happy, or angry. It has extensive application value in markets like mental diagnosis, secure monitoring, and intelligent interaction. The paper will provide a brief overview of the kernel technology of facial expressions recognition: geometric features, appearance features, and deep learning. And compared the experimental results of different experiments, as a goal, the experiments compare functions of several methods in a multi-dataset context, concentrating the primary problems in the tasks like dataset bias, real-time requirements, and different light levels and partial occlusions in various environments. Eventually, the paper forecasts the imminent growth trend in data fusion and deep learning improvement. This also indicates that emerging technologies, especially in terms of applications, will have the upper hand.

BFFN: A novel balanced feature fusion network for fair facial expression recognition

Facial expression recognition (FER) technology has become increasingly mature and applicable in recent years. However, it still suffers from the bias of expression class, which can lead to unfair decisions for certain expression classes in applications. This study aims to mitigate expression class bias through both pre-processing and in-processing approaches. First, we analyze the output of existing models and demonstrate the existence of obvious class bias, particularly for underrepresented expressions. Second, we develop a class-balanced dataset constructed through data generation, mitigating unfairness at the data level. Then, we propose the Balanced Feature Fusion Network (BFFN), a class fairness-enhancing network. The BFFN mitigates the class bias by adding facial action units (AU) to enrich expression-related features and allocating weights in the AU feature fusion process to improve the extraction ability of underrepresented expression features. Finally, extensive experiments on datasets (RAF-DB and AffectNet) provide evidence that our BFFN outperforms existing FER models, improving the fairness by at least 16%.

Attention-Based Spatiotemporal-Aware Network for Fine-Grained Visual Recognition

On public benchmarks, current macro facial expression recognition technologies have achieved significant success. However, in real-life scenarios, individuals may attempt to conceal their true emotions. Conventional expression recognition often overlooks subtle facial changes, necessitating more fine-grained micro-expression recognition techniques. Different with prevalent facial expressions, weak intensity and short duration are the two main obstacles for perceiving and interpreting a micro-expression correctly. Meanwhile, correlations between pixels of visual data in spatial and channel dimensions are ignored in most existing methods. In this paper, we propose a novel network structure, the Attention-based Spatiotemporal-aware network (ASTNet), for micro-expression recognition. In ASTNet, we combine ResNet and ConvLSTM as a holistic framework (ResNet-ConvLSTM) to extract the spatial and temporal features simultaneously. Moreover, we innovatively integrate two level attention mechanisms, channel-level attention and spatial-level attention, into the ResNet-ConvLSTM. Channel-level attention is used to discriminate the importance of different channels because the contributions for the overall presentation of micro-expression vary between channels. Spatial-level attention is leveraged to dynamically estimate weights for different regions due to the diversity of regions’ reflections to micro-expression. Extensive experiments conducted on two benchmark datasets demonstrate that ASTNet achieves performance improvements of 4.25–16.02% and 0.79–12.93% over several state-of-the-art methods.

Exploring the effects of multiple online interaction on emotions of L2 learners in synchronous online classes

The decline in both the quantity and quality of interaction has emerged as a notable challenge in online learning. However, the definition of interaction quality remains unclear. This study clarifies it as a decrease in the breadth of interaction, which refers to interaction that can only cover a smaller number of learners. To address this, a synchronous interaction modality, termed Multiple Online Interaction (MOI), based on Zoom's interactive tools, was introduced. In a quasi-experiment involving 58 Chinese L2 learners (with 30 beginner and 28 intermediate students), emotions were assessed using the Brief Mood Introspection Scale (BMIS) while real-time emotional dynamics were revealed through the analysis of 5129 facial expression images during a 35-min synchronous class. MOI participants reported higher levels of Lively and Happy but also experienced more Nervous and less Calm. These emotional dynamics, tracked through expression recognition technology, demonstrate that MOI's impact is primarily observed during the first Grammar & Practice section of the teaching. The empirical findings of this study provide practical insights for educators aiming to conduct effective online teaching in the future.

A facial expression recognition network based on attention double branch enhanced fusion.

The facial expression reflects a person's emotion, cognition, and even physiological or mental state to a large extent. It has important application value in medical treatment, business, criminal investigation, education, and human-computer interaction. Automatic facial expression recognition technology has become an important research topic in computer vision. To solve the problems of insufficient feature extraction, loss of local key information, and low accuracy in facial expression recognition, this article proposes a facial expression recognition network based on attention double branch enhanced fusion. Two parallel branches are used to capture global enhancement features and local attention semantics respectively, and the fusion and complementarity of global and local information is realized through decision-level fusion. The experimental results show that the features extracted by the network are made more complete by fusing and enhancing the global and local features. The proposed method achieves 89.41% and 88.84% expression recognition accuracy on the natural scene face expression datasets RAF-DB and FERPlus, respectively, which is an excellent performance compared with many current methods and demonstrates the effectiveness and superiority of the proposed network model.

A lightweight and continuous dimensional emotion analysis system of facial expression recognition under complex background

Facial expression recognition technology has a brilliant prospect in applying the Internet of Things systems. Concerning the limited hardware computing capability and high real-time processing requirements, this paper proposes a lightweight emotion analysis system based on edge computing, which could be deployed on edge devices. To further improve the accuracy of dimensional emotion analysis, we propose a modified network structure of MobileNetV3 to measure the intensity of dimensional emotion. The optimization scheme includes introducing the improved efficient channel attention mechanism and the feature pyramid network, adjusting the structure of the model, and optimizing the loss function. Furthermore, the system uses Intel’s OpenVINO toolkit to make the model more suitable for stable operation and provides an operable human–computer interaction interface. The experimental results show that the system has the advantages of few parameters, high recognition accuracy, and low latency. The optimized network size is reduced by 67% compared with the original, the root mean square error of potency and activation is 0.413 and 0.389, and the latency is up to 9ms in Myriad. This work meets the requirements of practical applications and has essential significance with the demand for continuous dimensional emotion analysis. The source code is available at https://github.com/SCNU-RISLAB/Lightweight-and-Continuous-Dimensional-Emotion-Analysis-System-of-Facial-Expression-Recognition/.

An Empirical Study of Portrait Rights on Emotional Evolution in Virtual Social Scenarios by Transformer and Cloud Computing

Facial expression recognition technology can improve the possibility of fugitives and persons subject to execution being discovered by public security and judicial personnel in the virtual world, thereby improving the actual effectiveness and credibility of the law. There may also be a risk of portrait rights infringement with this technology, and the user of the technology needs to inform the person whose facial expression is being extracted in advance and obtain permission. With the rapid progression of deep learning and artificial intelligence, 3D facial expression modeling has garnered increased significance in computer vision and graphics. We propose an innovative approach combining Transformer models with cloud computing to simulate facial expression evolution in virtual social environments. Leveraging Transformer-based feature extraction, our model integrates emotional cues from various modalities to accurately capture subtle changes over time.

Revolutionizing online education: Advanced facial expression recognition for real-time student progress tracking via deep learning model

AbstractThis paper presents a groundbreaking online educational platform that utilizes facial expression recognition technology to track the progress of students within the classroom environment. Through periodic image capture and facial data extraction, the platform employs ResNet50, CBAM, and TCNs for enhanced facial expression recognition. Achieving accuracies of 91.86%, 91.71%, 95.85%, and 97.08% on the RAF-DB, FER2013, CK + , and KDEF expression datasets, respectively, the proposed model surpasses the initial ResNet50 model in accuracy and detection of students' learning states. Comparative evaluations against state-of-the-art models using the RAF-DB, FER2013, CK + , and KDEF datasets underscore the significance of the results for educational institutions. By enhancing emotion recognition accuracy, improving feature relevance, capturing temporal dynamics, enabling real-time monitoring, and ensuring robustness and adaptability in online classroom environments, this approach offers valuable insights for educators to enhance teaching strategies and student outcomes. The combined capabilities of ResNet50, CBAM, and TCNs contribute uniquely to capturing dynamic changes in facial expressions over time, thereby facilitating accurate interpretation of students' emotions and engagement levels for more effective monitoring of learning behaviors in real-time.

Una investigación sobre un sistema de recomendación musical basado en expresiones faciales mediante mecanismos de aprendizaje profundo

In this study, we propose a new music recommendation system (MRS) that combines facial expression recognition technology and deep learning algorithms to respond to the changing music industry environment and provide personalized music recommendations based on the user's emotional state. Our approach includes a thorough study of facial expression recognition, emotion-based music recommendation systems, and deep learning engines, as well as a detailed presentation of the MRS design, system architecture, and deep learning engines used. Through extensive experiments, we evaluate MRS's ability to accurately recognize facial expressions, filter music based on emotional states, and effectively recommend music to users. We analyze the results of follow-up experiments to identify the strengths and limitations of MRS compared to existing approaches, and conduct a comparative study with the latest music recommendation systems based on deep learning and emotion. This comparison highlights the originality and potential of the proposed MRS system to improve user experience and promote the development of artificial intelligence-based music recommendation systems. This study demonstrates the problem of accurately determining a user's emotional state from facial expressions, which requires the integration of facial expression recognition systems, deep learning, and music recommendation systems. Using advanced deep learning techniques and a comprehensive experimental setup, the proposed MRS provides a solution to this problem by facilitating accurate emotional state identification and personalized music recommendations. Overall, MRS represents a powerful and innovative response to the growing demand for accurate and reliable music recommendations and shows significant potential for future collaboration and development of AI-based music recommendation systems.

Application of facial expression recognition technology based on feature fusion in teaching

With the development of artificial intelligence technology, the digital transformation of student-oriented education becomes particularly important. How to promote real-time interaction between teachers and students in the classroom is an urgent issue which is needed to pay attention to. Based on the facial expression features of students in a classroom, this paper analyzes the changes in angles between facial expression feature points using Dlib. Additionally, this paper proposes a novel algorithm for extracting variable scale template edge trend features. The algorithm adaptively processes the template based on the edge trend features of expression feature points, and use the proposed template slope normalization algorithm to achieve multi-scale template edge trend extraction. Then, DNN are used to recognize different listening expressions. The experimental results show that the proposed algorithm has faster recognition speed and better robustness when applied to classroom expression recognition. By identifying students’ class status to remind teachers to adjust their class progress, the goal of improving classroom learning effectiveness is achieved

The positive effect of blue luminescent pathways on urban park visitor’s affective states: A virtual reality online study measuring facial expressions and self-reports

The colour blue often elicits feelings of calmness and contentment, for which evidence has largely been provided in daytime settings. It is unclear whether pathways illuminated in blue, for example, in urban recreational park areas at night confers the same positive impact on night time park visitors. To tackle this open empirical question, we investigated how adding blue self-luminous pavement to park lighting at night time affects park visitors’ emotions compared to conventional white street light illumination. Our goal is to inform design decisions aimed at enhancing the emotional well-being of people outdoors at night in urban environments. Participants’ emotional response was captured at four different time points while watching a video of a walk in a virtual urban park at night, which was lit with white street lights only or with the addition of blue luminescent pavement on the walked paths (between-subject design). To capture visitor’s emotions, we used a simplified version of the Geneva Emotion Wheel (GEW) instrument and online facial expression recognition technology as subjective (self-reports) and objective (physiological) measures of emotion, respectively. The results of the GEW self-reports showed that the addition of a blue self-luminous pavement in a park during night time yielded more positive affect than standard white lighting in park visitors for the first half of the walk. In the second half of the walk through the park, participants’ affective states seemed to equalize between the two lighting conditions. In contrast, sensory data on facial expressions indicated no difference between participants’ emotional states over the whole walk in the two experimental conditions. Consistent with the positive emotional state perceived in the second half of the walk, the state of relaxation experienced after the walk also did not differ between the two lighting conditions. Furthermore, participants’ relaxation judgements after the park walk were more negative overall for females than the more neutral ratings of males. Our results highlight the importance of lighting colour at night for the design of future affect-smart cities that may consider individual and group characteristics with the ultimate intent of promoting public well-being.

Retracted: Construction of Facial Expression Recognition Technology of Intelligent Service Robot Based on Nanofiber Flexible Wearable Electronic Skins

Retracted: Construction of Facial Expression Recognition Technology of Intelligent Service Robot Based on Nanofiber Flexible Wearable Electronic Skins

Exploring the Innovation of Informative English Teaching Evaluation in the Age of Mathematical Intelligence

Abstract With the informatization of education, the traditional English teaching evaluation method is increasingly unable to meet the needs of cultivating talents in colleges and universities. In this paper, we constructed an informatized English teaching evaluation system using two modules: students’ classroom learning status evaluation and teachers’ teaching quality evaluation. Using informationized face detection and expression recognition technology, the evaluation axis of students’ classroom learning status is established as “students’ head-up rate - students’ expression - students’ concentration - classroom learning status”. The fuzzy comprehensive judgment model based on a genetic algorithm was established by constructing a judgment matrix and calculating the weights of teaching quality evaluation indexes using a hierarchical analysis method. The experimental and control groups analyzed the English teaching evaluation system’s application. Regarding English achievement, the post-experimental scores of the students in the experimental class were 8.57 points higher than those of the control class, with a highly significant difference (P<0.01). In terms of student literacy, students in the experimental class showed significant improvement in four indicators, including learning motivation, learning situation and cognition, and their attitudes toward the quality of English teaching were in the range between satisfaction and comparative satisfaction, and the overall performance of the application practice of the English education evaluation system was good.

A Comprehensive Evaluation Scheme of Students' Classroom Learning Status Based on Analytic Hierarchy Process

The state of students' classroom learning refers to whether students are concentrating and listening in the class, whether they are interested in the content of the lecture, and whether they understand and master the content of the teacher’s psychological state. The research on the state of students’ classroom learning is of great significance in teaching activities. Usually, measuring students’ classroom learning status is a subjective evaluation, and it is difficult to have a quantitative description. With the development of artificial intelligence (AI)technology, a large number of modern information technologies have been applied in education and teaching. People use video images, eye tracking technology, orfacial expression recognition technology in deep learning to describe students' attention and interest in classroom learning. However, with the complexity of students' learning process, no single information technology can accurately describe students' learning status and learning performance. Therefore, when evaluating students' learning status, it is necessary to consider the students' learning status, learning process, and learning performance. To this end, we used the Analytic Hierarchy Process (AHP)to establish a mathematical model in the four dimensions of interest, pleasure, concentration, and classroom performance and homomorphically adjusted the corresponding weights of the four dimensions in evaluating students' learning status. The proposed evaluation scheme of students' classroom learning status evaluated students' classroom learning status in real-time, providing modern information technology support for improving classroom learning efficiency and the quality of education and teaching.

PAIN ASSESSMENT AND MANAGEMENT FOR OLDER ADULTS WITH DEMENTIA IN LONG-TERM CARE: A SCOPING REVIEW

Abstract Self-reporting is essential for assessing pain. However, owing to their cognitive impairment, older adults with dementia may not always express pain. As such, older adults with dementia are at risk of living with undertreated pain. This scoping review aimed to explore and identify current publication trends and strategies regarding pain assessment and management for older adults with dementia in long-term care. Using the scoping review method of Arksey and O’Malley, two reviewers conducted structured searches for articles published from 2012 to 2022 on four databases (CINAHL, Cochrane, PubMed, RISS) and then independently reviewed the full text for eligibility criteria. The review included 40 articles for analysis and identified 20 observational pain tools used in the studies. A few studies assessed pain in older adults with dementia using facial expression recognition technology and electroencephalography. A common pain management strategy identified for older adults with dementia was the application of protocols or clinical guidelines that included a pain assessment and treatment phase. In addition to pharmacological and non-pharmacological treatment, a therapeutic robot (PARO) and play activities were investigated to determine whether the pain was alleviated in older adults with dementia. Although studies have been conducted to assess and manage pain in older adults with dementia in long-term care, the review highlighted the need for education and training on pain tools and treatment approaches, as well as for the development of pain management strategies and non-pharmacological interventions.

Recognizing facial emotion leveraging convolutional neural network

Facial expression recognition technology is a technology based on the combination of artificial intelligence technology and biology, and it is interdisciplinary research. The appearance of this technology shows the diversity of the development direction and application fields of computer technology, but it also means that the development of facial expression recognition technology needs not only the support of computer technology but also the exploration and progress of biology. At present, to recognize facial expressions a system usually contains three stages, namely, face detection stage, feature extraction stage, and face expression recognition stage. The facial expression recognition algorithm is transformed into a classification problem, and a convolutional neural network (CNN) is adopted, where features of the facial expressions could be automatically extracted. It takes as input a preprocessed image and can directly produce encoded features and predictions, leveraging the end-to-end strategy. It could discard the complicated intermediate modeling process of traditional machine learning. In this work a CNN is implemented for the recognition of facial expression on Fer2013 dataset. Moreover, the effectiveness of different numbers of CNN layers are testified on this problem. It could be concluded that with deeper architectures, the CNN tends to perform better.

Facial expression-enhanced recommendation for virtual fitting rooms

With the development of Augmented Reality (AR) technology in the retail industry, virtual fitting room (VFR) are considered promising enhancement of e-commerce by providing users with an immersive environment to try on new products, especially fashion products. While allowing users having more vivid impression of products, virtual fitting rooms also offer sellers more channels to collect information on user preferences, which can be used to enhance recommender systems. This study proposes to leverage facial expression recognition technology together with fine-grained human-computer interactions in virtual fitting rooms to personalize product recommendations. This paper proposes a recommendation algorithm based on confidence setting, negative feedback sampling, and matrix factorization to model user behaviors in virtual fitting rooms. We conduct an experiment on 81 subjects to evaluate the proposed method. Experimental results show the proposed method outperforms existing methods using traditional behavior information. Our study provides a strong support to the value of AR in enhancing e-commerce.

Facial Expression Recognition Model Based on CNN and Data Augmentation Method

Face expressions are essential in expressing human emotions, and it is accomplished by separating features and categorizing them. Facial expression recognition technology has been widely employed in human-computer interaction, telemedicine, mental health, and criminal investigation detection. In recent years, significant advances in deep learning have facilitated the development of facial expression recognition, making it increasingly accessible. Convolutional neural networks (CNN) are the foundation of the face expression recognition model presented in this article. In order to maximize the final accuracy, an abundance of sample photographs are required for training and optimization. As a result, the 35,886 facial expression photos from the FER2013 dataset, which contains all seven emotions, were used for both training and testing. The photos were scaled down to 4848 pixels during data preparation, and data augmentation was carried out. To get accurate face expression recognition results, various optimizers and parameters were chosen for the training network, which was a bespoke structure based on the VGG network design. The model constructed in this study achieved an accuracy of 73.16% during prediction.

A high‐performance and lightweight framework for real‐time facial expression recognition

AbstractFacial expression recognition technology has become a powerful tool for conveying human emotions and intentions and is widely used in areas such as assisted driving and intelligent medical care. Due to the limited computing power of current hardware devices and the real‐time requirements of application scenarios, this paper proposes a high‐performance and lightweight framework for real‐time facial expression recognition framework to solve the problem of real‐time completion of expression recognition tasks under low hardware costs. To address these issues, this paper first designs a RepVGG and mobileNetV2 dual‐channel structure in the feature extraction. It is then input into the MobileViT Block for global feature modelling. Finally, the position vector of the capsule network is used to replace the output of the global pooling, preserving the spatial relationship of the salient features and enhancing the classification effect. Compared with the mainstream facial expression recognition algorithm that cannot get good classification results under low complexity conditions, the model has a significant accuracy improvement while ensuring lightweight. With only 294.60M FLOPS and 0.95M parameters, it achieved an accuracy of 97.53% on the KDEF dataset and 85.56% on the RAF‐DB, demonstrating the advanced nature of the algorithm.