Facial Expression Patterns Research Articles

AI-Based Visual Early Warning System

Facial expressions are a universally recognised means of conveying internal emotional states across diverse human cultural and ethnic groups. Recent advances in understanding people’s emotions expressed through verbal and non-verbal communication are particularly noteworthy in the clinical context for the assessment of patients’ health and well-being. Facial expression recognition (FER) plays an important and vital role in health care, providing communication with a patient’s feelings and allowing the assessment and monitoring of mental and physical health conditions. This paper shows that automatic machine learning methods can predict health deterioration accurately and robustly, independent of human subjective assessment. The prior work of this paper is to discover the early signs of deteriorating health that align with the principles of preventive reactions, improving health outcomes and human survival, and promoting overall health and well-being. Therefore, methods are developed to create a facial database mimicking the underlying muscular structure of the face, whose Action Unit motions can then be transferred to human face images, thus displaying animated expressions of interest. Then, building and developing an automatic system based on convolution neural networks (CNN) and long short-term memory (LSTM) to recognise patterns of facial expressions with a focus on patients at risk of deterioration in hospital wards. This research presents state-of-the-art results on generating and modelling synthetic database and automated deterioration prediction through FEs with 99.89% accuracy. The main contributions to knowledge from this paper can be summarized as (1) the generation of visual datasets mimicking real-life samples of facial expressions indicating health deterioration, (2) improvement of the understanding and communication with patients at risk of deterioration through facial expression analysis, and (3) development of a state-of-the-art model to recognize such facial expressions using a ConvLSTM model.

Computer Vision for E-learning Student Face Reaction Detection for Improving Learning Rate

In the advent of e-learning, understanding student engagement and reaction is crucial for improving the quality of education and enhancing the learning rate. With the advancement of computer vision technologies, there is a significant opportunity to analyze and interpret student reactions in a non-intrusive manner. This study proposes a novel framework employing Faster R-CNN integrated with DenseNet architecture for real-time detection of student facial reactions during e-learning sessions. The proposed method leverages the strengths of Faster R-CNN in generating high-quality region proposals for object detection tasks, coupled with the DenseNet’s efficiency in feature propagation and reduction in the number of parameters, which is well-suited for processing the intricate patterns in facial expressions. Our approach begins with the application of Faster R-CNN to extract potential facial regions with high accuracy and reduced computational cost. The integration of DenseNet as a backbone for feature extraction within Faster R-CNN capitalizes on its densely connected convolutional networks, ensuring maximum information flow between layers in the network. By doing so, the system becomes exceptionally adept at recognizing subtle changes in facial features that indicate various student reactions, such as confusion, engagement, or boredom. We conducted a series of experiments using a diverse dataset of e-learning interactions, collected under various lighting conditions and involving multiple ethnicities to ensure robustness and generalizability. The model was trained and validated on this dataset, and the results demonstrate a significant improvement in detection rates of student reactions compared to existing methods.

An Emotional Artificial Intelligence Based Attendance System for Class Room

Attendance through a camera using Artificial Intelligence (AI) and Deep Learning (DL) is a mod-ern approach to monitoring and tracking attendance in various settings such as schools, offices, and man-ufacturing facilities. It uses computer vision techniques and deep learning algorithms to automatically detect and identify individuals in an image or video captured by the camera. The process typically in-volves training the system on a dataset of labelled images of the individuals who will be taking attend-ance. This training data can include images of the individual's face, iris, or fingerprints, depending on the specific approach used. Once the system is trained, it can then use this knowledge to recognize these in-dividuals in new images captured by the camera. When an individual is recognized, the system can log their attendance in a database or other record-keeping system. This can be done in real-time, allowing for immediate tracking of attendance, or it can be done at a later time for batch processing. However, AI has been expanding its horizon and face recognition with deep learning techniques can be augmented with emotion recognition as well. Sometimes, students feel really sad and overwhelmed by their school work and other responsibilities. They might feel like they can't keep up or that they are not good enough. When these feelings last for a long time, it's called depression. Depression can make it hard for students to do their school work, be with friends and family, or even get out of bed in the morning. When students are feeling really sad and hopeless, they might think about hurting themselves or ending their lives. This is called suicide. It's important to know that suicide is preventable and there are people who can help. It's important to take care of our mental health, just like we take care of our physical health. If you ever feel sad or overwhelmed, it's important to talk to someone you trust and get help. One way to prevent suicide is to detect if someone is feeling sad and hopeless, which can be a sign of depression. One way to do this is by using a CCTV camera and AI and DL technology to analyze the person's facial expression, body language, and speech patterns. Emotional AI and DL can be a powerful tool in detecting depression but it's not a substitute for a professional diagnosis. It's always advisable to consult with a mental health pro-fessional if you suspect that you or someone you know may be struggling with depression.

USE OF VIDEO-BASED DATA TO INFORM PSYCHOSOCIAL INTERVENTIONS FOR PEOPLE LIVING WITH DEMENTIA

Abstract The pandemic has accelerated technological developments in care services. Emerging evidence suggests feasibility of delivering evidence-based psychosocial interventions such as group cognitive stimulation therapy using information and communication technology (ICT) platforms. A unique opportunity offered by ICT is the more readily available data at session level to investigate nuanced, theoretically based outcome predictors, to improve efficacy of these complex interventions. Theoretically, the cognitive and wellbeing benefits of group cognitive stimulation therapy is related to the level of person-centred engagement. As part of an ongoing randomised controlled trial involving ICT-delivered group cognitive stimulation therapy, this study examines the use of big data analytics of session behavioural data to develop predictive algorithms of cognitive and wellbeing outcomes. In 78 people living with mild to moderate dementia from 26 groups, we create automatically estimated markers of facial expression and eye movement pattern, and hand-coded markers (using time-sampling method) of engagement level and speech/verbal production information, to autobiographical information presented in the session. These markers are used to predict changes in cognitive and wellbeing outcomes. Pilot analysis using hand-coded speech sample suggested strong correlation between time-sampled engagement level with cognitive benefits, and changes in information content and fluency with cognitive benefits after the group intervention (ranging from r=0.76 to 0.97, p< 0.001). The markers we develop in this study using video-based data from ICT-delivered therapy sessions, including deep learning-based emotion recognition and eye movement analysis with hidden Markov model, allow further application in understanding the mechanisms of action in different psychosocial interventions.

Edge Artificial Intelligence-Based Facial Pain Recognition During Myocardial Infarction

Medical history highlights that Myocardial Infarction is one of the leading factors of death in human beings. Angina pectoris is a prominent vital sign of Myocardial Infraction. Medical reports suggests that experiencing a chest pain during heart attacks causes changes in facial muscles resulting in variations in patterns of facial expression. The present work intends to develop an automatic facial expression detection to identify the severity of chest pain as a vital signs of MI using algorithmic approach that is implemented with a state of art Convolution Neural Networks CNN. The advanced object detection light weight Convolutional Neural Network models: Single Shot Detector Mobile Net V2 and Single Shot Detector Inception V2 are utilized for designing the vital signs MI model from 500 Red Blue Green Color images private dataset. The authors develop a cardiac emergency health monitoring care using an Edge Artificial Intelligence, “Edge AI” using NVIDIA’s Jetson Nano embedded GPU platform. The proposed model is mainly focused on the low cost and less power consumption factors for onboard real time detection of vital signs of Myocardial infarction. The evaluated metrics achieves a mean Average Precision of 85.18%, Average Recall of 88.32% and 6.85 frames per second for the generated detections.

An Efficient Multi-modal Facial Gesture-based Ensemble Classification and Reaction to Sound Framework for Large Video Sequences

Machine learning-based feature extraction and classification models play a vital role in evaluating and detecting patterns in multivariate facial expressions. Most conventional feature extraction and multi-modal pattern detection models are independent of filters for multi-class classification problems. In traditional multi-modal facial feature extraction models, it is difficult to detect the dependent correlated feature sets and use ensemble classification processes. This study used advanced feature filtering, feature extraction measures, and ensemble multi-class expression prediction to optimize the efficiency of feature classification. A filter-based multi-feature ranking-based voting framework was implemented on different multiple-based classifiers. Experimental results were evaluated on different multi-modal facial features for the automatic emotions listener using a speech synthesis library. The evaluation results showed that the proposed model had better feature classification, feature selection, prediction, and runtime than traditional approaches on heterogeneous facial databases.

Hybrid heuristic mechanism for occlusion aware facial expression recognition scheme using patch based adaptive CNN with attention mechanism

In computer vision, the recognition of expressions from partially occluded faces is one of the serious problems. By the prior recognizing techniques it can solve the issue by various assumptions. A benchmark-guided branch was proposed for detecting and eliminating the manipulated features from the occluded regions since the human visual system is proficient for eliminating the occlusion and the appropriate focal point was obtained on the non-occluded areas. In recent years deep learning has attained a great place in the recognition of facial reactions Still, the precision of facial expression is affected by the occlusion and large skew. In this research work, a deep structure-based occlusionaware facial expression recognition mechanism is introduced to provide superior recognition results. Firstly, the required image is taken from publically provided online sources and the gathered images are subjected to the face extraction method. The face extraction method is done via the ViolaJones method for the extraction of redundant patterns from the original images. Secondly, the extracted face features are given to the pattern recognition stage, where the Adaptive CNN with Attention Mechanism (ACNN-AM) is introduced. This mechanism automatically forms the occluded region of the face and the focal point was on the most discriminative un-occluded regions. Moreover, the hidden patterns in the Occlusion aware facial expressions are identified through the Hybrid Galactic Swarm Yellow Saddle Goatfish Optimization (HGSYSGO). Finally, the overall effectiveness of the developed occlusion aware facial expression recognition model is examined through the comparative analysis of different existing baseline recognition techniques.

Classification of multiple emotional states from facial expressions in head-fixed mice using a deep learning-based image analysis.

Facial expressions are widely recognized as universal indicators of underlying internal states in most species of animals, thereby presenting as a non-invasive measure for assessing physical and mental conditions. Despite the advancement of artificial intelligence-assisted tools for automated analysis of voluminous facial expression data in human subjects, the corresponding tools for mice still remain limited so far. Considering that mice are the most prevalent model animals for studying human health and diseases, a comprehensive characterization of emotion-dependent patterns of facial expressions in mice could extend our knowledge on the basis of emotions and the related disorders. Here, we present a framework for the development of a deep learning-powered tool for classifying facial expressions in head-fixed mouse. We demonstrate that our machine vision was capable of accurately classifying three different emotional states from lateral facial images in head-fixed mouse. Moreover, we objectively determined how our classifier characterized the differences among the facial images through the use of an interpretation technique called Gradient-weighted Class Activation Mapping. Importantly, our machine vision presumably discerned the data by leveraging multiple facial features. Our approach is likely to facilitate the non-invasive decoding of a variety of emotions from facial images in head-fixed mice.

Распознавание утомления человека на основе анализа его речи с помощью нейросетевых технологий

Qualitative psychophysiological research studies are associated with the creation of accessible and well-organized databases that require a lot of preliminary work on the development of measuring complexes, including not only tools for measuring the psychophysiological parameters of a human, but also their emotional state, which is displayed in facial expression, speech and behavioral patterns. Measuring systems should also include the means of experimental material processing. The purpose of the study was to conduct an experiment on creating a prototype of the Speech Data Base of Russian-speaking respondents and to obtain answers to some methodological questions that arise among specialists when they use the database for the task of recognizing the state of human fatigue. A hardware and software complex has been developed that allows to synchronously register psychophysiological parameters, video recordings of behavioral reactions and audio recordings of human speech. As a model of physical fatigue, a cardio-respiratory test with physical activity (load) was used. Before and after completing the test, volunteers read out a set of standard phonetically representative texts. The obtained audio recordings were processed using a specialized neural network capable of analyzing the integral spectral characteristics of sound. The results of the experiment showed the possibility of recognizing the state of human fatigue based on speech analysis, which makes it possible to proceed to the creation of a large bank of audio recordings and the improvement of algorithms for recognizing the state of fatigue.

The Effect of Layer Batch Normalization and Droupout of CNN model Performance on Facial Expression Classification

One of the implementations of face recognition is facial expression recognition in which a machine can recognize facial expression patterns from the observed data. This study used two models of convolutional neural network, model A and model B. The first model A was without batch normalization and dropout layers, while the second model B used batch normalization and dropout layers. It used an arrangement of 4 layer models with activation of ReLU and Softmax layers as well as 2 fully connected layers for 5 different classes of facial expressions of angry, happy, normal, sad, and shock faces. Research Metodology are 1). Data Analysis, 2). Preprocessing grayscaling, 3). Convolutional Neural Network (CNN), 4). Model validation Testing, Obtained an accuracy of 64.8% for training data and accuracy of 63.3% for validation data. The use of dropout layers and batch normalization could maintain the stability of both training data and validation data so that there was no overfitting. By dividing the batch size on the training data into 50% with 200 iterations, aiming to make the load on each training model lighter, by using the learning rate to be 0.001 which works to improve the weight value, thus making the training model work to be fast without crossing the minimum error limit. Accuracy results in the classification of ekp facial receipts from the distance of the camera to the face object about 30 cm in the room with the use of bright enough lighting by 78%.

Phase Space Reconstruction Driven Spatio-Temporal Feature Learning for Dynamic Facial Expression Recognition

Automatic Dynamic Facial Expression Recognition (DFER) is a challenging task, since how to effectively capture facial temporal dynamics is still an open problem. In this article, we regard variations of facial expressions as a dynamic system in accord with certain rules, and try to explore the fundamental temporal properties for recognizing dynamic expressions. Inspired by the phase space reconstruction method for time series analysis, we propose a novel network named Phase Space Reconstruction Network (PSRNet) for learning spatio-temporal features of facial expressions. First, 3D convolutional neural networks are used to extract spatial and short-term temporal features, which indicate the state of each frame and are termed as observations in the phase space. All the observations compose the trajectory of the dynamical system. Then, a data-driven across-correlation matrix is inferred to reveal the relationship of the observations. With this matrix, the phase space reconstruction module reconstructs the trajectory by aggregating the observations adaptively in the phase space. Reconstructed observations represent the gradual process of dynamic facial expressions, which is beneficial to recognize these expressions. The experiment results on three databases (Oulu, MMI, and CK+) demonstrate that the proposed PSRNet can extract more informative and representative spatio-temporal features for DFER. Moreover, the visualization of intermediate features reveals that the reconstructed features have global consistency in facial regions and the underlying evolutionary pattern of dynamic facial expression.

Static and Motion Facial Analysis for Craniofacial Assessment and Diagnosing Diseases.

Deviation from a normal facial shape and symmetry can arise from numerous sources, including physical injury and congenital birth defects. Such abnormalities can have important aesthetic and functional consequences. Furthermore, in clinical genetics distinctive facial appearances are often associated with clinical or genetic diagnoses; the recognition of a characteristic facial appearance can substantially narrow the search space of potential diagnoses for the clinician. Unusual patterns of facial movement and expression can indicate disturbances to normal mechanical functioning or emotional affect. Computational analyses of static and moving 2D and 3D images can serve clinicians and researchers by detecting and describing facial structural, mechanical, and affective abnormalities objectively. In this review we survey traditional and emerging methods of facial analysis, including statistical shape modeling, syndrome classification, modeling clinical face phenotype spaces, and analysis of facial motion and affect.

Harlequin Ichthyosis: Folklore of Demon

Background: Harlequin ichthyosis (HI) is a severe and rare autosomal recessive congenital ichthyosis, characterized by severe hyperkeratosis, extensive fissuring and massive, diamond-shaped scales which limit the child’s movements. The term harlequin comes from the baby’s facial expression and diamond-shaped pattern of the scales like the 17th century entertainers, harlequins. The underlying genetic abnormality has been identified as a mutation in the lipid-transporter gene ABCA 12 on chromosome 2q35. Aim and objectives: To provide a comprehensive knowledge of severe forms of congenital ichthyosis, the harlequin ichthyosis and abolish the superstitious notion regarding the appearance of the neonate. Materials and Methods: The present report is case of premature girl baby delivered at MIMS, Mandya. The neonate presented plaques of rigid fixed skin, separated by deep red fissures, facial features were obliterated by thickened skin, undeveloped nose and pinna, severe ectropion and eclabium. Intensive care was given and survival was prolonged by systemic retinoids. The neonate succumbed to respiratory failure few days later. Results and conclusion: The present study gives a comprehensive knowledge of harlequin ichthyosis and provides emphasis on sensitization of the disease to prevent any misconception or superstition. Also lays emphasis on Genetic counseling of the couple to prevent further occurrences. KEY WORDS: Eclabium, Ectropion, Harlequin ichthyosis.

Парный модельный эксперимент «верю/не верю» — возможность выделения невербальных признаков достоверности/недостоверности сообщаемой информации

As markers of reliability/inaccuracy of the reported information, there have been repeated attempts to use the partial facial features identified by experts. Our research proposes a transition to analysis at the level of automatically identified integral mimic patterns. To solve this problem, a model paired experiment was conducted in which participants had to repeatedly make a decision to tell the truth/lie. The analysis of facial expressions was performed by automatically calculating the coordinates of key points and further extracting integral facial expression patterns using the method of principal components. At the level of loads of the main components, signs of making a decision “to lie/tell the truth” for the sample as a whole and for individual subjects are highlighted. It was found that holistic patterns of facial expressions associated with the reliability/unreliability of the reported information at “low rates” are characterized by high individual variability. As a direction for further research on facial expressions, a transition to the study of the individual repertoire of integral patterns of facial expressions is proposed.

Finding Facial Expression Patterns on Videos based on Smile and Eyes-Open Confidence Values

Facial expression recognition is one of the types of non-verbal communication that is not only commons for human but also plays an essential role in everyday lives. The development of science and technology allows the machine to automatically detect human facial expressions based on images and videos. Numerous facial expression detection methods have been proposed in the literature. This paper presents a method to find three basic facial expressions (neutral, happy, and angry) from two parameter values: smile and eyes-open. The analysis involves a preprocessing step using a combination of pre-designed proprietary algorithm and Luxand library. Firstly, the parameters were mapped into two-dimensional space and then grouped into three clusters using K-means, a popular heuristic clustering method. Secondly, more than 50,000 frames for each video were experimented using the proprietary research data. The result shows that the proposed method successfully performed a simple video analysis of facial expressions.

Comparing self-reported emotions and facial expressions of joy in heterosexual romantic couples

To better understand individual differences in the expression of emotion within intimate relationships, we evaluated and compared patterns in facial expressions of joy against patterns in self-reported expressions of emotions in romantic couples. Using conversational data from 44 heterosexual romantic couples discussing four different topics, we examined the impact of stress on emotion expression, similarity in emotion expression between partners, and the influence of one partner's facial expressions on the self-reported expressions of the other partner. Overall, we found large differences between patterns in facial expressions of joy and patterns in self-reported emotions. First, using social relations analysis and generalizability analysis, we found that self-reported positive and negative emotions changed between stressful and non-stressful conversational topics, whereas facial expressions of joy remained stable. Second, we found similarities between romantic partners were common for self-reported positive emotions, less common for self-reported negative emotions, and uncommon for facial expressions. Finally, using Actor-Partner Interdependence Models, we found facial expressions of joy were unrelated to self-reported positive and negative emotions, and were non-significant predictors of partner's self-reported emotions. Our results challenge the use of only one methodology when measuring emotional experiences, as patterns observed for self-reported emotional data and facial expression data were not the same.

Investigating appraisal-driven facial expression and inference in emotion communication.

Theory and research on emotion expression, both on production and recognition, has been dominated by a categorical emotion approach suggesting that discrete emotions are elicited and expressed via prototypical facial muscle configurations that can then be recognized by observers, presumably via template matching. This tradition is increasingly challenged by alternative theoretical approaches. In particular, appraisal theorists have suggested that specific elements of facial expressions are directly produced by the result of certain appraisals and have made detailed predictions about the facial patterns to be expected for these appraisal configurations. This approach has been extended to emotion perception, with theorists claiming that observers first infer individual appraisals and only then make categorical emotion judgments from the estimated appraisal patterns, using semantic inference rules. Here we report two studies that empirically examine the two central hypotheses proposed by this theoretical position: (a) that specific appraisals produce predicted patterns of facial muscle expressions and (b) that observers can infer a person's appraisals of ongoing events from the predicted facial expression configurations. The results show that (a) professional actors use many of the predicted facial action unit patterns to enact, in a realistic scenario setting, appraisal outcomes specified by experimental design, and (b) observers systematically infer specific appraisals from ecologically valid video recordings of marketing research participants as they view TV commercials (selected according to the likelihood of eliciting specific appraisals). The patterns of facial action units identified in these studies correspond largely to prior predictions and encourage further research on appraisal-driven expression and inference. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Facial emotion mimicry in older adults with and without cognitive impairments due to Alzheimer's disease.

Facial expression of humans is one of the main channels of everyday communication. The reported research work investigated communication regarding the pattern of emotional expression of healthy older adults and with mild cognitive impairments (MCI) or Alzheimer's disease (AD). It focuses on mimicking of displayed emotional facial expression on a sample of 25 older adults (healthy, MCI and AD patients). The adequacy of the patients' individual facial expressions in six basic emotions was measured with the Kinect 3D recording of the participants' facial expressions and compared to their own typical emotional facial expressions. The reactions were triggered by mimicking 49 still pictures of emotional facial expressions. No statistically significant differences in terms of frequency nor adequacy of emotional facial expression were reported in healthy and MCI groups. Unique patterns of emotional expressions have been observed in the AD group. Further investigating the pattern of older adults' facial expression may decrease the misunderstandings and increase the quality of life of the patients.

Sixteen facial expressions occur in similar contexts worldwide.

Understanding the degree to which human facial expressions co-vary with specific social contexts across cultures is central to the theory that emotions enable adaptive responses to important challenges and opportunities1-6. Concrete evidence linking social context to specific facial expressions is sparse and is largely based on survey-based approaches, which are often constrained by language and small sample sizes7-13. Here, by applying machine-learning methods to real-world, dynamic behaviour, we ascertain whether naturalistic social contexts (for example, weddings or sporting competitions) are associated with specific facial expressions14 across different cultures. In two experiments using deep neural networks, we examined the extent to which 16 types of facial expression occurred systematically in thousands of contexts in 6million videos from 144 countries. We found that each kind of facial expression had distinct associations with a set of contexts that were 70% preserved across 12world regions. Consistent with these associations, regions varied in how frequently different facial expressions were produced as a function of which contexts were most salient. Our results reveal fine-grained patterns in human facial expressions that are preserved across the modern world.

Digital Behavioral Phenotyping Detects Atypical Pattern of Facial Expression in Toddlers with Autism.

Commonly used screening tools for autism spectrum disorder (ASD) generally rely on subjective caregiver questionnaires. While behavioral observation is more objective, it is also expensive, time-consuming, and requires significant expertise to perform. As such, there remains a critical need to develop feasible, scalable, and reliable tools that can characterize ASD risk behaviors. This study assessed the utility of a tablet-based behavioral assessment for eliciting and detecting one type of risk behavior, namely, patterns of facial expression, in 104 toddlers (ASD N = 22) and evaluated whether such patterns differentiated toddlers with and without ASD. The assessment consisted of the child sitting on his/her caregiver's lap and watching brief movies shown on a smart tablet while the embedded camera recorded the child's facial expressions. Computer vision analysis (CVA) automatically detected and tracked facial landmarks, which were used to estimate head position and facial expressions (Positive, Neutral, All Other). Using CVA, specific points throughout the movies were identified that reliably differentiate between children with and without ASD based on their patterns of facial movement and expressions (area under the curves for individual movies ranging from 0.62 to 0.73). During these instances, children with ASD more frequently displayed Neutral expressions compared to children without ASD, who had more All Other expressions. The frequency of All Other expressions was driven by non-ASD children more often displaying raised eyebrows and an open mouth, characteristic of engagement/interest. Preliminary results suggest computational coding of facial movements and expressions via a tablet-based assessment can detect differences in affective expression, one of the early, core features of ASD. LAY SUMMARY: This study tested the use of a tablet in the behavioral assessment of young children with autism. Children watched a series of developmentally appropriate movies and their facial expressions were recorded using the camera embedded in the tablet. Results suggest that computational assessments of facial expressions may be useful in early detection of symptoms of autism.