Facial Expressions Research Articles

사전학습 모델 기반 발화 동영상 멀티 모달 감정 인식

Recently, as the demand for non-face-to-face counseling has rapidly increased, the need for emotion recognition technology that combines various aspects such as text, voice, and facial expressions is being emphasized. In this paper, we address issues such as the dominance of non-Korean data and the imbalance of emotion labels in existing datasets like FER-2013, CK+, and AFEW by using Korean video data. We propose methods to enhance multimodal emotion recognition performance in videos by integrating the strengths of image modality with text modality. A pre-trained model is used to overcome the limitations caused by small training data. A GPT-4-based LLM model is applied to text, and a pre-trained model based on VGG-19 architecture is fine-tuned to facial expression images. The method of extracting representative emotions by combining the emotional results of each aspect extracted using a pre-trained model is as follows. Emotion information extracted from text was combined with facial expression changes in a video. If there was a sentiment mismatch between the text and the image, we applied a threshold that prioritized the text-based sentiment if it was deemed trustworthy. Additionally, as a result of adjusting representative emotions using emotion distribution information for each frame, performance was improved by 19% based on F1-Score compared to the existing method that used average emotion values for each frame.

Accident Prevention System for Cars, Buses, Trucks

Driver drowsiness is a significant factor in road accidents, contributing to a large number of crashes annually. To enhance road safety, the development of a Drivers Drowsiness Detection System is essential. This system builds to detect early signs of lazyness in drivers and provide timely alerts to prevent accidents. The proposed system uses to track key indicators of drowsiness, such as eye closure, head position, yawning, and blink rate. By utilizing a camera-based real-time monitoring system, it continuously analyzes the driver’s facial expressions. The system employs cadcade algorithms to distinguish a featurebased object detection algorithm to detect objects from images between alert and drowsy states based on collected data. When drowsiness is detected, the system triggers alerts like audio warnings, seat vibrations, or even vehicle slowing mechanisms, depending on the integration. This technology has the potential to significantly reduce the number of road accidents related to driver fatigue and could be applied in both personal vehicles and commercial fleets. The implementation of such systems represents a critical step toward improving road safety and decreasing driver-related incidents. Nowadays it is very challenging to stay active all the time due to busy schedules. Falling asleep while driving can lead to serious consequences, accidents, and even death. This situation is much more common and therefore it is very important to solve this problem. So, to solve this problem, we developed a sleep alarm system for drivers. This system alerts the user when he falls asleep at the wheel, thus the main concern is preventing accidents and saving lives. This system is useful for long- distance travelers and late-night drivers.

Task-irrelevant inputs alter ensemble representations of faces within the spatial focus of attention.

Spatial attention enhances processing of information, but how does unattended and task-irrelevant information influence visual processing within the spatial focus of attention? We tested this by asking participants to extract the average emotional expression of a set of sequentially presented faces while simultaneously presenting task-irrelevant faces at a spatially unattended and task-irrelevant location. Across several experiments, we found that participants' reports of the emotional expression of faces at the attended location were biased toward the task-irrelevant faces. For example, when happier faces were presented at the unattended location, participants were biased to perceive the attended faces as happier. A control experiment in which participants were asked to also detect probes at cued and uncued locations showed that spatial attention was directed towards the cued location as instructed. Together, these results reveal that unattended and task-irrelevant inputs do not only affect the efficiency of target processing, for example by slowing responses or lowering accuracies, but that they can systematically bias ensemble representations within the spatial focus of attention. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Prefrontal cortex activation and working memory performance in individuals with non-clinical depression: Insights from fNIRS.

Previous research has extensively explored cognitive and neural deficits in clinically diagnosed depression, but the early stages of depression, where symptoms do not meet clinical thresholds, are less explored. This study investigated neurocognitive markers in individuals with non-clinical depression. The study assessed working memory (WM) performance and hemodynamic responses of prefrontal cortex (PFC) in 30 individuals with non-clinical depression and 41 healthy controls using two-back tasks with four stimulus types: numbers, letters, shapes, and emotional facial expressions. Hemodynamic responses were measured via oxyhemoglobin (HbO) using functional near-infrared spectroscopy. Results showed that individuals with non-clinical depression have significantly lower WM performance compare to healthy controls, particularly for shape- and emotional facial expression-based tasks. No differences were observed for the number- and letter- based tasks. Additionally, individual with non-clinical depression exhibited elevated HbO levels, indicating increased PFC activation. Specifically, significant HbO differences observed in the bilateral ventrolateral PFC during shape-based tasks, and in the left medial, bilateral orbital, and bilateral ventrolateral PFCs during emotional facial expression-based tasks. In conclusion, individuals with non-clinical depression may experience WM deficits and PFC dysregulation, even without a clinical diagnosis. This study highlights the role of stimulus type in understanding WM performance and PFC activation in depression.

User emotion recognition and indoor space interaction design: a CNN model optimized by multimodal weighted networks

In interior interaction design, achieving intelligent user-interior interaction is contingent upon understanding the user’s emotional responses. Precise identification of the user’s visual emotions holds paramount importance. Current visual emotion recognition methods rely solely on singular features, predominantly facial expressions, resulting in inadequate coverage of visual characteristics and low recognition rates. This study introduces a deep learning-based multimodal weighting network model to address this challenge. The model initiates with a convolutional attention module, employing a self-attention mechanism within a convolutional neural network (CNN). As a result, the multimodal weighting network model is integrated to optimize weights during training. Finally, a weight network classifier is derived from these optimized weights to facilitate visual emotion recognition. Experimental outcomes reveal a 77.057% correctness rate and a 74.75% accuracy rate in visual emotion recognition. Comparative analysis against existing models demonstrates the superiority of the multimodal weight network model, showcasing its potential to enhance human-centric and intelligent indoor interaction design.

Facial emotion recognition using deep quantum and advanced transfer learning mechanism

IntroductionFacial expressions have become a common way for interaction among humans. People cannot comprehend and predict the emotions or expressions of individuals through simple vision. Thus, in psychology, detecting facial expressions or emotion analysis demands an assessment and evaluation of decisions for identifying the emotions of a person or any group during communication. With the recent evolution of technology, AI (Artificial Intelligence) has gained significant usage, wherein DL (Deep Learning) based algorithms are employed for detecting facial expressions.MethodsThe study proposes a system design that detects facial expressions by extracting relevant features using a Modified ResNet model. The proposed system stacks building-blocks with residual connections and employs an advanced extraction method with quantum computing, which significantly reduces computation time compared to conventional methods. The backbone stem utilizes a quantum convolutional layer comprised of several parameterized quantum-filters. Additionally, the research integrates residual connections in the ResNet-18 model with the Modified up Sampled Bottle Neck Process (MuS-BNP), retaining computational efficacy while benefiting from residual connections.ResultsThe proposed model demonstrates superior performance by overcoming the issue of maximum similarity within varied facial expressions. The system’s ability to accurately detect and differentiate between expressions is measured using performance metrics such as accuracy, F1-score, recall, and precision.DiscussionThis performance analysis confirms the efficacy of the proposed system, highlighting the advantages of quantum computing in feature extraction and the integration of residual connections. The model achieves quantum superiority, providing faster and more accurate computations compared to existing methodologies. The results suggest that the proposed approach offers a promising solution for facial expression recognition tasks, significantly improving both speed and accuracy.

AUTOMATED GESTURE RECOGNITION USING APPLIED LINGUISTICS WITH DATA-DRIVEN DEEP LEARNING FOR ARABIC SPEECH TRANSLATION

Gesture recognition for Arabic speech translation includes developing advanced technologies that correctly translate body and hand movements corresponding to Arabic sign language (ArSL) into spoken Arabic. This leverages machine learning and computer vision techniques in complex systems simulation platforms to scrutinize the gestures utilized in ArSL, detecting mild differences in facial expressions, hand shapes, and movements. Sign Language Recognition (SLR) is paramount in assisting communication for the Deaf and Hard-of-Hearing communities. It includes using vision-based methods and Surface Electromyography (sEMG) signals. The sEMG signal is crucial for recognizing hand gestures and capturing muscular activities in sign language. Researchers have comprehensively shown the capability of EMG signals to approach specific details, mainly in classifying hand gestures. This progression is a stimulating feature in extracting the interpretation and recognition of sign languages and investigating the phonology of signed language. Leveraging machine learning algorithms and signal processing techniques in complex systems simulation platforms, researchers aim to extract relevant traits from the sEMG signals that correspond to different ArSL gestures. This study introduces an Enhanced Dwarf Mongoose Algorithm with a Deep Learning-Driven Arabic Sign Language Detection (EDMODL-ASLD) technique on sEMG data. In the initial phase, the presented EDMODL-ASLD model is subjected to data preprocessing to change the input sEMG data into an attuned format. In the next stage, feature extraction with fractal theories is used to gather relevant and nonredundant data from the EMG window to construct a feature vector. In this study, the absolute envelope (AE), energy (E), root-mean square (RMS), standard deviation (STD), and mean absolute value (MAV) are the five time-domain extracted features for the EMG window observation. Meanwhile, the dilated convolutional long short-term memory (ConvLSTM) technique is used to identify distinct sign languages. To improve the results of the dilated ConvLSTM model, the hyperparameter selection process is executed using the EDMO model. To illustrate the significance of the EDMODL-ASLD technique, a brief experimental validation is made on the Arabic SLR database. The experimental validation of the EDMODL-ASLD technique portrayed a superior accuracy value of 96.47% over recent DL approaches.

Effect of observer's cultural background and masking condition of target face on facial expression recognition for machine-learning dataset.

Facial expression recognition (FER) is significantly influenced by the cultural background (CB) of observers and the masking conditions of the target face. This study aimed to clarify these factors' impact on FER, particularly in machine-learning datasets, increasingly used in human-computer interaction and automated systems. We conducted an FER experiment with East Asian participants and compared the results with the FERPlus dataset, evaluated by Western raters. Our novel analysis approach focused on variability between images and participants within a "majority" category and the eye-opening rate of target faces, providing a deeper understanding of FER processes. Notable findings were differences in "fear" perception between East Asians and Westerners, with East Asians more likely to interpret "fear" as "surprise." Masking conditions significantly affected emotion categorization, with "fear" perceived by East Asians for non-masked faces interpreted as "surprise" for masked faces. Then, the emotion labels were perceived as different emotions across categories in the masking condition, rather than simply lower recognition rates or confusion as in existing studies. Additionally, "sadness" perceived by Westerners was often interpreted as "disgust" by East Asians. These results suggest that one-to-one network learning models, commonly trained using majority labels, might overlook important minority response information, potentially leading to biases in automated FER systems. In conclusion, FER dataset characteristics differ depending on the target face's masking condition and the diversity among evaluation groups. This study highlights the need to consider these factors in machine-learning-based FER that relies on human-judged labels, to contribute to the development of more nuanced and fair automated FER systems. Our findings emphasize the novelty of our approach compared to existing studies and the importance of incorporating a broader range of human variability in FER research, setting the stage for future evaluations of machine learning classifiers on similar data.

P1 evoked by facial expression images is enhanced in Parkinson’s disease patients with depressive symptoms

IntroductionDepressive symptoms are most common non-motor symptoms in Parkinson’s disease (PD), which is often overlooked due to absence of rapid and objective diagnostic biomarkers. Electroencephalography (EEG)-based event-related potentials (ERPs) is commonly used to assess emotional processes. The aim of this study was to investigate changes in ERPs in PD patients exhibiting depressive symptoms and to provide a reliable biomarker for assisting in the diagnosis of PD with depressive symptoms.MethodsWe conducted a case–control study involving 30 PD patients with (dPD group) or without depressive symptoms (nPD group) and 13 age matched healthy controls (HC). We recorded EEG of the patients during the emotional picture stimulation task and analyzed the difference in the early ERPs potentials (P1, N170, early posterior negativity) and their correlation with the severity of symptoms in PD patients.ResultsOur results found that P1 amplitude in the occipital region of the dPD group in response to emotional faces was significantly higher than that of nPD and HC group, and it was positively correlated with severity of depressive symptoms in PD patients.ConclusionOur study shows that facial expression-induced enhancement of P1 amplitude can be utilized as a rapid and objective indicator to screen for depressive symptoms in PD.

Measuring palatability of pet food products: Sensory components, evaluations, challenges, and opportunities.

The pet food industry is a growing business launching a variety of new products in the market. The acceptability or preference of pet food samples has traditionally been measured using either one-bowl or two-bowl tests. Academic researchers and professionals in the pet food industry have explored other methods, including the cognitive palatability assessment protocols and the ranking test, to evaluate more than two samples. A variety of approaches and perspectives were also utilized to predict palatability and key sensory attributes of pet foods, including descriptive sensory analysis by human-trained panelists and pet food caregivers' perceptions of pet food. This review article examined a range of testing methods for evaluating the palatability of pet foods, specifically targeting products for dogs and/or cats. It outlined the advantages and disadvantages of each method. Additionally, the review provided in-depth insights into the key sensory attributes of pet foods and the methodologies for assessing palatability. It also explored pets' behavioral responses and facial expressions in relation to different pet foods. Furthermore, this review discussed current challenges and future opportunities in pet food development, including the use of instrumental analyses and artificial intelligence-based approaches.

Facial Expression Recognition-You Only Look Once-Neighborhood Coordinate Attention Mamba: Facial Expression Detection and Classification Based on Neighbor and Coordinates Attention Mechanism

In studying the joint object detection and classification problem for facial expression recognition (FER) deploying the YOLOX framework, we introduce a novel feature extractor, called neighborhood coordinate attention Mamba (NCAMamba) to substitute for the original feature extractor in the Feature Pyramid Network (FPN). NCAMamba combines the background information reduction capabilities of Mamba, the local neighborhood relationship understanding of neighborhood attention, and the directional relationship understanding of coordinate attention. The resulting FER-YOLO-NCAMamba model, when applied to two unaligned FER benchmark datasets, RAF-DB and SFEW, obtains significantly improved mean average precision (mAP) scores when compared with those obtained by other state-of-the-art methods. Moreover, in ablation studies, it is found that the NCA module is relatively more important than the Visual State Space (VSS), a version of using Mamba for image processing, and in visualization studies using the grad-CAM method, it reveals that regions around the nose tip are critical to recognizing the expression; if it is too large, it may lead to erroneous prediction, while a small focused region would lead to correct recognition; this may explain why FER of unaligned faces is such a challenging problem.

From Movement to Meaning: Real-time Detection of Behavioural Patterns in Human Psychology using Deep Learning and Time Series Sequence

Understanding human behaviour and psychology is complex and involves various fields, including cognitive science, neurology, and artificial intelligence (AI). In this study, we introduce a cutting-edge system that detects both manual (like hand gestures) and non-manual (such as facial expressions or body movements) features in real time, using advanced deep learning techniques and time-series analysis. This system translates these human movements into text, providing a powerful tool to analyse subtle behavioural and psychological signals. By processing real-time data, our system significantly advances psychology by detecting small expressions, involuntary gestures, and behaviour patterns often associated with psychological and neurological conditions. For instance, it can help identify repetitive actions or atypical gestures in people with autism spectrum disorder (ASD), which are critical for early diagnosis. It can also track signs of neurodegenerative diseases like Parkinson’s or Alzheimer’s, detecting tremors, posture changes, or facial expressions, and offer insights for early intervention

What is in a look? The accountability of gaze in trajectories to conflict

This study investigates the role of gaze in initiating episodes of conflict by examining, using multimodal conversation analysis, a set of cases in which a recipient is prompted to speak by another’s extended gaze. In these cases, this recipient response may be, e.g., “What,” or a more elaborate demand for an account, such as “Why are you looking at me like that for?” Here we investigate the characteristics of the gaze that prompts such responses, and what actions such responses constitute. While “What” compositionally resembles other-initiated repair, its sequential position characterizes it as a so-called “go-ahead” action. In these cases, the sequential positioning of such gazes, constituting it structurally as a so-called “pre,” alongside its durational characteristics and facial expression, are examined to identify the normative associations of gaze and subsequent conduct that make such gazes accountable.

Virtual Children’s Hospital for Learning Pediatric Nursing

Abstract Background In the context of a funded Erasmus+ program, a platform of a Virtual Children’s Hospital for Learning Pediatric Nursing - VLPN is being developed. The education of nurses working with children, adolescents and families, needs to include specialized knowledge, skills, attitudes and behaviors that will empower them to provide optimal and high-quality care. The virtual platform enables the improvement of pediatric nursing competences of the nursing students through interactive and self-paced activities and content. Objectives 1. To develop an immersive and interactive virtual reality learning environment (VRLE), a virtual children’s hospital and new teaching and learning methods and practices, utilizing virtual reality and simulation in teaching. 2. To promote the international collaboration of students and lecturers through the VRLE Results The results are preliminary. 1. The VRLE includes educational materials developed to be integrated in the pediatric nursing studies, easy to access and use by the students and teachers with innovative self-pace learning material i.e. videos, 3D-models, interactive tests. Digital patients can react to the provided care with speech, movement, facial expressions and gestures, something that is not possible with the manikins in the Labs. 2. Nursing students and lecturers from different countries can participate in joint remote seminars or workshops in the VRLE, sharing and comparing their country and culture-specific differences and similarities in the nursing practices. VRLE and digital materials enables synchronous and asynchronous learning. Conclusions VRLE is highly conducive to clinical and procedure-focused training by enabling simulation. VR methods offer nursing students the opportunity to practice skills in a safe and risk-free environment in an immersive way. It can be accessed by multiple students, teachers or nurses at the same time from different locations in different countries. Key messages • New innovative VRLE enhances nursing students’ knowledge and skills that are needed when working with sick children and their families. • International collaboration, interaction and mobilization are enabled by using a virtual platform instead of traveling, thus strengthening the co-operation in European higher educational institutes.

Audio-Driven Facial Animation with Deep Learning: A Survey

Audio-driven facial animation is a rapidly evolving field that aims to generate realistic facial expressions and lip movements synchronized with a given audio input. This survey provides a comprehensive review of deep learning techniques applied to audio-driven facial animation, with a focus on both audio-driven facial image animation and audio-driven facial mesh animation. These approaches employ deep learning to map audio inputs directly onto 3D facial meshes or 2D images, enabling the creation of highly realistic and synchronized animations. This survey also explores evaluation metrics, available datasets, and the challenges that remain, such as disentangling lip synchronization and emotions, generalization across speakers, and dataset limitations. Lastly, we discuss future directions, including multi-modal integration, personalized models, and facial attribute modification in animations, all of which are critical for the continued development and application of this technology.

Autism Identification Based on the Intelligent Analysis of Facial Behaviors: An Approach Combining Coarse- and Fine-Grained Analysis

Background: Facial behavior has emerged as a crucial biomarker for autism identification. However, heterogeneity among individuals with autism poses a significant obstacle to traditional feature extraction methods, which often lack the necessary discriminative power. While deep-learning methods hold promise, they are often criticized for their lack of interpretability. Methods: To address these challenges, we developed an innovative facial behavior characterization model that integrates coarse- and fine-grained analyses for intelligent autism identification. The coarse-grained analysis provides a holistic view by computing statistical measures related to facial behavior characteristics. In contrast, the fine-grained component uncovers subtle temporal fluctuations by employing a long short-term memory (LSTM) model to capture the temporal dynamics of head pose, facial expression intensity, and expression types. To fully harness the strengths of both analyses, we implemented a feature-level attention mechanism. This not only enhances the model’s interpretability but also provides valuable insights by highlighting the most influential features through attention weights. Results: Upon evaluation using three-fold cross-validation on a self-constructed autism dataset, our integrated approach achieved an average recognition accuracy of 88.74%, surpassing the standalone coarse-grained analysis by 8.49%. Conclusions: This experimental result underscores the improved generalizability of facial behavior features and effectively mitigates the complexities stemming from the pronounced intragroup variability of those with autism, thereby contributing to more accurate and interpretable autism identification.

Implementation of the YOLO Method for Detection of Human Emotions Based on Facial Mimics

Emotion detection through facial expression recognition plays an important role in everyday life, such as how to respond correctly to emotional expressions in social interactions, so that you can establish and build verbal or nonverbal communication with other people and so on. Facial expressions are facial changes in response to a person's emotional state, intentions, or social communication. Face detection is the first step that must be taken in facial analysis, including facial expression recognition. There are many methods that can be used to carry out the face detection process, such as the YOLO method. This YOLO method reframes object detection as a single regression problem, directly from image pixels to bounding box coordinates and class probabilities. By using the YOLO method, the process only needs to look once at the input image, to predict what objects are in the image and where those objects are. Based on the results of the tests carried out, the YOLO method can be used to detect human facial expressions with a success rate of 80%, with neutral, surprise and disgust facial expressions having a good level of accuracy and fear facial expressions having a poor accuracy level. The YOLO method is able to detect facial expressions of humans who wear accessories such as glasses.

Human-machine interaction with intelligent machine learning algorithms for smart classrooms

ABSTRACT The modern technology of learning and teaching in education has moved on to smart classes. The introduction of artificial intelligence (AI) in the education sector has essentially taken over traditional classrooms and transformed how education leads to admiration. Various data types, such as videos, images, sound, text, and biometric data like eye-tracking potentially captured through smart classrooms, pose new issues and challenges for artificial intelligence algorithms. Hence, this paper suggests the Machine Learning Assisted Intelligent Human-Machine Interaction (MLIHMI) model for the smart classroom monitoring system. The real-time interactive dimension signifies the capability to support the Smart classroom’s teaching and human-computer interactions, including smooth interaction, convenient operation, and interactive monitoring. This paper used ML and HMI models to introduce a new effective database for online learning and smart classroom environments utilising the learners’ body postures, facial expressions, and hand gestures. The numerical findings show that the suggested system achieves less error rate of 10.1% and enhances the accuracy rate of 96.5%, with the highest classification ratio of 95.6%, prediction rate of 93.6%, the detection rate of 97.3%, and engagement rate of 81.3% in student gesture and postures recognition and the identification of facial expressions with the high performance of 93.3% when compared to other existing models.

Decoding the influence of emotional and attentional states on self-control using facial analysis

Self-control plays a pivotal role in pursuing long-term goals related to health and financial well-being. While ample evidence suggests that humans are prone to occasional self-control lapses, little is known about how changes in emotional and attentional states affect the ability to maintain self-control. In two studies (N1 = 109 and N2 = 90), we used emotion recognition software to decode participants’ facial expressions while manipulating their attentional and emotional states during a Psychomotor Vigilance Task (PVT) before exerting self-control in a subsequent task. Our findings reveal dissociable roles of attention and valence in maintaining self-control, depending on the distinct demands of the task. Specifically, performance in a subsequent cognitive task was predominantly associated with changes in attentional states during the PVT rather than valence. Conversely, preferences in a subsequent social task were associated with changes in valence states during the PVT, while attention showed no effect. This dissociation between attention-invoked and emotion-invoked lapses in self-control holds significant implications for psychological and economic models of self-control, ultimately contributing to the development of policies to mitigate the detrimental consequences of individual self-control lapses.

Attendance Marking using Face Detection

The demand for automated attendance management systems has significantly increased in educational institutions and corporate environments where manual methods are prone to inefficiencies and errors. This paper presents a real-time, facial recognition-based attendance system developed using Flask, OpenCV, and machine learning techniques. The system employs OpenCV’s Haar Cascade classifier for face detection and the K-Nearest Neighbours (KNN) algorithm for face recognition. During user registration, the system captures 10 images per individual to ensure robust recognition across various facial expressions. The web interface, built using Flask, facilitates user interaction for managing attendance records stored in CSV format. The system achieves an accuracy of 92% under optimal lighting conditions, though performance decreases to 80% in low-light environments. Recognition is completed in approximately 2 seconds, making the system suitable for real-time applications in classrooms or offices. Despite handling real-time image streaming efficiently, challenges such as reduced accuracy when handling obstructions like masks or glasses remain. Future enhancements will explore the integration of deep learning models, such as Convolutional Neural Networks (CNNs), to improve robustness and scalability. This system demonstrates the potential of facial recognition technology for automating attendance tracking, with significant applications in both educational and corporate sectors