Recognition Of Affective States Research Articles

Recognizing affective states from the expressive behavior of tennis players using convolutional neural networks

This study describes an AI model by leveraging advanced Convolutional Neural Networks (CNNs) to recognize affective states in real-world sports settings, particularly tennis matches. In contrast to prior studies that primarily utilized data acquired from actors and rudimentary statistical methods, the present research emphasizes the analysis of bodily expressions in real-life contexts, aiming for a more naturalistic representation of human emotions. Our CNN-based models demonstrate an accuracy rate of up to 68.9 %, outperforming or matching human observers in many instances. Intriguingly, both the machine learning models and human observers exhibited a shared propensity to more effectively identify negative affective states, which may be attributed to the more intense and straightforward expression of these states. These results not only advance the state of the art in affective state recognition but also pave the way for broader applications, including in healthcare and automotive safety sectors, thereby constituting a significant advancement in the development of sophisticated and universally applicable emotional recognition systems.

Multi-Corpus Learning for Audio–Visual Emotions and Sentiment Recognition

Recognition of emotions and sentiment (affective states) from human audio–visual information is widely used in healthcare, education, entertainment, and other fields; therefore, it has become a highly active research area. The large variety of corpora with heterogeneous data available for the development of single-corpus approaches for recognition of affective states may lead to approaches trained on one corpus being less effective on another. In this article, we propose a multi-corpus learned audio–visual approach for emotion and sentiment recognition. It is based on the extraction of mid-level features at the segment level using two multi-corpus temporal models (a pretrained transformer with GRU layers for the audio modality and pre-trained 3D CNN with BiLSTM-Former for the video modality) and on predicting affective states using two single-corpus cross-modal gated self-attention fusion (CMGSAF) models. The proposed approach was tested on the RAMAS and CMU-MOSEI corpora. To date, our approach has outperformed state-of-the-art audio–visual approaches for emotion recognition by 18.2% (78.1% vs. 59.9%) for the CMU-MOSEI corpus in terms of the Weighted Accuracy and by 0.7% (82.8% vs. 82.1%) for the RAMAS corpus in terms of the Unweighted Average Recall.

In-the-Wild Affect Analysis of Children with ASD Using Heart Rate.

Recognizing the affective state of children with autism spectrum disorder (ASD) in real-world settings poses challenges due to the varying head poses, illumination levels, occlusion and a lack of datasets annotated with emotions in in-the-wild scenarios. Understanding the emotional state of children with ASD is crucial for providing personalized interventions and support. Existing methods often rely on controlled lab environments, limiting their applicability to real-world scenarios. Hence, a framework that enables the recognition of affective states in children with ASD in uncontrolled settings is needed. This paper presents a framework for recognizing the affective state of children with ASD in an in-the-wild setting using heart rate (HR) information. More specifically, an algorithm is developed that can classify a participant's emotion as positive, negative, or neutral by analyzing the heart rate signal acquired from a smartwatch. The heart rate data are obtained in real time using a smartwatch application while the child learns to code a robot and interacts with an avatar. The avatar assists the child in developing communication skills and programming the robot. In this paper, we also present a semi-automated annotation technique based on facial expression recognition for the heart rate data. The HR signal is analyzed to extract features that capture the emotional state of the child. Additionally, in this paper, the performance of a raw HR-signal-based emotion classification algorithm is compared with a classification approach based on features extracted from HR signals using discrete wavelet transform (DWT). The experimental results demonstrate that the proposed method achieves comparable performance to state-of-the-art HR-based emotion recognition techniques, despite being conducted in an uncontrolled setting rather than a controlled lab environment. The framework presented in this paper contributes to the real-world affect analysis of children with ASD using HR information. By enabling emotion recognition in uncontrolled settings, this approach has the potential to improve the monitoring and understanding of the emotional well-being of children with ASD in their daily lives.

Weighted knowledge distillation of attention-LRCN for recognizing affective states from PPG signals

The recognition of affective states is important for regulating stress levels and maintaining mental health, and it is known that affective states can be inferred from physiological signals. However, in practice, the problem of affect recognition contains many challenges due to various types of external noise and different individual characteristics. This paper proposes a deep learning model called Attention-LRCN for recognizing affective states from photoplethysmography (PPG) signals. We construct a long-term recurrent convolutional network to extract temporal features from spectrograms, and a novel attention module is introduced to alleviate the effect of noise components in PPG signals. Moreover, to improve the recognition accuracy, we propose a weighted knowledge distillation technique, which is a teacher–student learning framework. We quantify the uncertainty of teacher’s predictions, and the predictive uncertainty is utilized to adaptively compute the weight of the distillation loss. To demonstrate the effectiveness of the proposed method, experiments were conducted on the WESAD dataset, which is a public dataset for stress and affect detection. We also collected our own dataset from 34 subjects to verify the accuracy of the proposed method. Experimental results demonstrate that the proposed method significantly outperforms previous algorithms on both the public and real-world datasets. The code is available at https://github.com/ziiho08/Attention-LRCN.

Can You Tell Who Scores? An Assessment of the Recognition of Affective States Based on the Nonverbal Behavior of Amateur Tennis Players in Competitive Matches.

The purpose of the present study was to assess whether the recognition of tennis players' affective state associated with their nonverbal behavior would be influenced by (a)the importance of the situation, (b)the point outcome, and (c)the tennis expertise of the observer. Two hundred sixty-nine participants (Mage = 30.51years; 116 female; 79 tennis club members) watched video excerpts showing the nonverbal behavior of amateur tennis players during competitive matches immediately after the end of a rally and were asked to estimate whether the player had just won or lost the point. Results indicate that the recognition rates were higher for situations closer to the end of a game, closer to the end of a set, and with a tighter score during a game. Moreover, recognition rates were higher for lost than for won points, while the tennis expertise of participants had no influence on the recognition rates.

Multimodal Affective States Recognition Based on Multiscale CNNs and Biologically Inspired Decision Fusion Model

There has been an encouraging progress in the affective states recognition models based on the single-modality signals as electroencephalogram (EEG) signals or peripheral physiological signals in recent years. However, multimodal physiological signals-based affective states recognition methods have not been thoroughly exploited yet. Here we propose Multiscale Convolutional Neural Networks (Multiscale CNNs) and a biologically inspired decision fusion model for multimodal affective states recognition. Firstly, the raw signals are pre-processed with baseline signals. Then, the High Scale CNN and Low Scale CNN in Multiscale CNNs are utilized to predict the probability of affective states output for EEG and each peripheral physiological signal respectively. Finally, the fusion model calculates the reliability of each single-modality signals by the Euclidean distance between various class labels and the classification probability from Multiscale CNNs, and the decision is made by the more reliable modality information while other modalities information is retained. We use this model to classify four affective states from the arousal valence plane in the DEAP and AMIGOS dataset. The results show that the fusion model improves the accuracy of affective states recognition significantly compared with the result on single-modality signals, and the recognition accuracy of the fusion result achieve 98.52% and 99.89% in the DEAP and AMIGOS dataset respectively.

Analysis of Facial Occlusion Challenge in Thermal Images for Human Affective State Recognition.

Several studies have been conducted using both visual and thermal facial images to identify human affective states. Despite the advantages of thermal facial images in recognizing spontaneous human affects, few studies have focused on facial occlusion challenges in thermal images, particularly eyeglasses and facial hair occlusion. As a result, three classification models are proposed in this paper to address the problem of thermal occlusion in facial images, with six basic spontaneous emotions being classified. The first proposed model in this paper is based on six main facial regions, including the forehead, tip of the nose, cheeks, mouth, and chin. The second model deconstructs the six main facial regions into multiple subregions to investigate the efficacy of subregions in recognizing the human affective state. The third proposed model in this paper uses selected facial subregions, free of eyeglasses and facial hair (beard, mustaches). Nine statistical features on apex and onset thermal images are implemented. Furthermore, four feature selection techniques with two classification algorithms are proposed for a further investigation. According to the comparative analysis presented in this paper, the results obtained from the three proposed modalities were promising and comparable to those of other studies.

A Sparse Representation Classification Scheme for the Recognition of Affective and Cognitive Brain Processes in Neuromarketing.

In this work, we propose a novel framework to recognize the cognitive and affective processes of the brain during neuromarketing-based stimuli using EEG signals. The most crucial component of our approach is the proposed classification algorithm that is based on a sparse representation classification scheme. The basic assumption of our approach is that EEG features from a cognitive or affective process lie on a linear subspace. Hence, a test brain signal can be represented as a linear (or weighted) combination of brain signals from all classes in the training set. The class membership of the brain signals is determined by adopting the Sparse Bayesian Framework with graph-based priors over the weights of linear combination. Furthermore, the classification rule is constructed by using the residuals of linear combination. The experiments on a publicly available neuromarketing EEG dataset demonstrate the usefulness of our approach. For the two classification tasks offered by the employed dataset, namely affective state recognition and cognitive state recognition, the proposed classification scheme manages to achieve a higher classification accuracy compared to the baseline and state-of-the art methods (more than 8% improvement in classification accuracy).

Explainable automated recognition of emotional states from canine facial expressions: the case of positive anticipation and frustration

In animal research, automation of affective states recognition has so far mainly addressed pain in a few species. Emotional states remain uncharted territories, especially in dogs, due to the complexity of their facial morphology and expressions. This study contributes to fill this gap in two aspects. First, it is the first to address dog emotional states using a dataset obtained in a controlled experimental setting, including videos from (n = 29) Labrador Retrievers assumed to be in two experimentally induced emotional states: negative (frustration) and positive (anticipation). The dogs’ facial expressions were measured using the Dogs Facial Action Coding System (DogFACS). Two different approaches are compared in relation to our aim: (1) a DogFACS-based approach with a two-step pipeline consisting of (i) a DogFACS variable detector and (ii) a positive/negative state Decision Tree classifier; (2) An approach using deep learning techniques with no intermediate representation. The approaches reach accuracy of above 71% and 89%, respectively, with the deep learning approach performing better. Secondly, this study is also the first to study explainability of AI models in the context of emotion in animals. The DogFACS-based approach provides decision trees, that is a mathematical representation which reflects previous findings by human experts in relation to certain facial expressions (DogFACS variables) being correlates of specific emotional states. The deep learning approach offers a different, visual form of explainability in the form of heatmaps reflecting regions of focus of the network’s attention, which in some cases show focus clearly related to the nature of particular DogFACS variables. These heatmaps may hold the key to novel insights on the sensitivity of the network to nuanced pixel patterns reflecting information invisible to the human eye.

Analysis of infoware and software for human affective states recognition

The article presents an analytical review of research in the affective computing field. This research direction is a component of artificial intelligence, and it studies methods, algorithms and systems for analyzing human affective states during interactions with other people, computer systems or robots. In the field of data mining, the definition of affect means the manifestation of psychological reactions to an exciting event, which can occur both in the short and long term, and also have different intensity. The affects in this field are divided into 4 types: affective emotions, basic emotions, sentiment and affective disorders. The manifestation of affective states is reflected in verbal data and non-verbal characteristics of behavior: acoustic and linguistic characteristics of speech, facial expressions, gestures and postures of a person. The review provides a comparative analysis of the existing infoware for automatic recognition of a person’s affective states on the example of emotions, sentiment, aggression and depression. The few Russian-language, affective databases are still significantly inferior in volume and quality compared to electronic resources in other world languages. Thus, there is a need to consider a wide range of additional approaches, methods and algorithms used in a limited amount of training and testing data, and set the task of developing new approaches to data augmentation, transferring model learning and adapting foreign-language resources. The article describes the methods of analyzing unimodal visual, acoustic and linguistic information, as well as multimodal approaches for the affective states recognition. A multimodal approach to the automatic affective states analysis makes it possible to increase the accuracy of recognition of the phenomena compared to single-modal solutions. The review notes the trend of modern research that neural network methods are gradually replacing classical deterministic methods through better quality of state recognition and fast processing of large amount of data. The article discusses the methods for affective states analysis. The advantage of multitasking hierarchical approaches is the ability to extract new types of knowledge, including the influence, correlation and interaction of several affective states on each other, which potentially leads to improved recognition quality. The potential requirements for the developed systems for affective states analysis and the main directions of further research are given.

The recognition of affective states associated with players’ non-verbal behavior in volleyball

The visible non-verbal behavior of volleyball players after a point might indicate their affective state. The purpose of the present study was to investigate whether the point outcome (i.e., win vs. loss) and the importance of the situation (i.e., first vs. fifth set) influence how well the affective states (indexed through non-verbal behavior) can be recognized. For this purpose, 111 participants were presented 60 video excerpts showing the non-verbal behavior of volleyball teams immediately after a rally and asked whether the shown team had just won or lost a point. The results indicated that the recognition rate was significantly higher for won (80.09%) than for lost points (65.56%) as well as for points of the fifth set (77.84%) than for points of the first set (68.51%). Furthermore, an explorative analysis showed that the recognition rate was significantly higher for won points of the fifth set (89.19%) than for those of the first set (72.13%).

A Review of AI Cloud and Edge Sensors, Methods, and Applications for the Recognition of Emotional, Affective and Physiological States.

Affective, emotional, and physiological states (AFFECT) detection and recognition by capturing human signals is a fast-growing area, which has been applied across numerous domains. The research aim is to review publications on how techniques that use brain and biometric sensors can be used for AFFECT recognition, consolidate the findings, provide a rationale for the current methods, compare the effectiveness of existing methods, and quantify how likely they are to address the issues/challenges in the field. In efforts to achieve the key goals of Society 5.0, Industry 5.0, and human-centered design better, the recognition of emotional, affective, and physiological states is progressively becoming an important matter and offers tremendous growth of knowledge and progress in these and other related fields. In this research, a review of AFFECT recognition brain and biometric sensors, methods, and applications was performed, based on Plutchik’s wheel of emotions. Due to the immense variety of existing sensors and sensing systems, this study aimed to provide an analysis of the available sensors that can be used to define human AFFECT, and to classify them based on the type of sensing area and their efficiency in real implementations. Based on statistical and multiple criteria analysis across 169 nations, our outcomes introduce a connection between a nation’s success, its number of Web of Science articles published, and its frequency of citation on AFFECT recognition. The principal conclusions present how this research contributes to the big picture in the field under analysis and explore forthcoming study trends.

EEG-based affective state recognition from human brain signals by using Hjorth-activity

EEG-based emotion recognition enables investigation of human brain activity, which is recognized as an important factor in brain-computer interface. In recent years, several methods have been studied to find optimal features from brain signals. The main limitation of existing studies is that either they consider very few emotion classes or they employ a large feature set. To overcome these issues, we propose a novel Hjorth-feature-based emotion recognition model. Unlike other methods, our proposed method explores a wider set of emotion classes in the arousal-valence domain. To reduce the dimension of the feature set, we employ Hjorth parameters (HPs) and analyze the parameters in the frequency domain. At the same time, our study was focused to maintain the accuracy of emotion recognition for four emotional classes. The average accuracy was approximately 69%, 76%, 85%, 59%, and 87% for DEAP, SEED-IV, DREAMER, SELEMO, and ASCERTAIN, respectively. Results show that the features from HP activity with random forest outperforms all the classic methods of EEG-based emotion recognition.

CNN-XGBoost fusion-based affective state recognition using EEG spectrogram image analysis

Recognizing emotional state of human using brain signal is an active research domain with several open challenges. In this research, we propose a signal spectrogram image based CNN-XGBoost fusion method for recognising three dimensions of emotion, namely arousal (calm or excitement), valence (positive or negative feeling) and dominance (without control or empowered). We used a benchmark dataset called DREAMER where the EEG signals were collected from multiple stimulus along with self-evaluation ratings. In our proposed method, we first calculate the Short-Time Fourier Transform (STFT) of the EEG signals and convert them into RGB images to obtain the spectrograms. Then we use a two dimensional Convolutional Neural Network (CNN) in order to train the model on the spectrogram images and retrieve the features from the trained layer of the CNN using a dense layer of the neural network. We apply Extreme Gradient Boosting (XGBoost) classifier on extracted CNN features to classify the signals into arousal, valence and dominance of human emotion. We compare our results with the feature fusion-based state-of-the-art approaches of emotion recognition. To do this, we applied various feature extraction techniques on the signals which include Fast Fourier Transformation, Discrete Cosine Transformation, Poincare, Power Spectral Density, Hjorth parameters and some statistical features. Additionally, we use Chi-square and Recursive Feature Elimination techniques to select the discriminative features. We form the feature vectors by applying feature level fusion, and apply Support Vector Machine (SVM) and Extreme Gradient Boosting (XGBoost) classifiers on the fused features to classify different emotion levels. The performance study shows that the proposed spectrogram image based CNN-XGBoost fusion method outperforms the feature fusion-based SVM and XGBoost methods. The proposed method obtained the accuracy of 99.712% for arousal, 99.770% for valence and 99.770% for dominance in human emotion detection.

Is It Me or the Robot? A Critical Evaluation of Human Affective State Recognition in a Cognitive Task.

A key goal in human-robot interaction (HRI) is to design scenarios between humanoid robots and humans such that the interaction is perceived as collaborative and natural, yet safe and comfortable for the human. Human skills like verbal and non-verbal communication are essential elements as humans tend to attribute social behaviors to robots. However, aspects like the uncanny valley and different technical affinity levels can impede the success of HRI scenarios, which has consequences on the establishment of long-term interaction qualities like trust and rapport. In the present study, we investigate the impact of a humanoid robot on human emotional responses during the performance of a cognitively demanding task. We set up three different conditions for the robot with increasing levels of social cue expressions in a between-group study design. For the analysis of emotions, we consider the eye gaze behavior, arousal-valence for affective states, and the detection of action units. Our analysis reveals that the participants display a high tendency toward positive emotions in presence of a robot with clear social skills compared to other conditions, where we show how emotions occur only at task onset. Our study also shows how different expression levels influence the analysis of the robots' role in HRI. Finally, we critically discuss the current trend of automatized emotion or affective state recognition in HRI and demonstrate issues that have direct consequences on the interpretation and, therefore, claims about human emotions in HRI studies.

Correction: Neethirajan, S. Affective State Recognition in Livestock—Artificial Intelligence Approaches. Animals 2022, 12, 759

The authors wish to make the following correction to the original paper [...].

Affective State Recognition with Convolutional Autoencoders.

The aim of this study was to create a robust generalizable model to classify person's affective state based on physiological signals obtained using wearable sensor devices. Traditional machine learning methods require manual feature extraction from time sequences. Deep learning methods, such as Convolutional Neural Networks (CNN), can automatically extract features from time sequences. However, CNN models can be prone to overfitting, especially when the dataset is small. We apply a novel idea of using unsupervised convolutional autoencoders to automatically extract features from time-series data that are then fed to supervised classifier to classify people's affective state. We achieve almost 3% accuracy increase over traditional CNN model using all physio data from WESAD dataset, 2% increase using chest only physio data, and 8% increase using wrist only physio data while classifying neutral, stress, and amusement states. Code to reproduce the results can be found at https://github.com/srovins/wesad Clinical Relevance- A high-performing affective state recognition system can be utilized for various medical applications, ranging from patient monitoring to cognitive therapy.

Intact prioritization of fearful faces during continuous flash suppression in psychopathy.

Affective state recognition and in particular the identification of fear is known to be impaired in psychopathy. It is unclear, however, whether this reflects a deficit in basic perception ('fear blindness') or a deficit in later cognitive processing. To test for a perceptual deficit, 63 male incarcerated offenders, assessed with the Hare Psychopathy Checklist-Revised (PCL-R), detected fearful, neutral, and happy facial expressions rendered invisible through continuous flash suppression (CFS). Fearful faces were detected faster than neutral and happy faces. There was no reduction of the fear advantage in the 20 offenders diagnosed with psychopathy according to the PCL-R, and there was no correlation between the fear advantage and PCL-R scores. Deficits in the processing of fearful facial expressions in psychopathy may thus not reflect fear blindness, but impairments at later postperceptual processing stages. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Context Awareness in Recognition of Affective States: A Systematic Mapping of the Literature

Studies indicate the growth and relevance of systems that can recognize affective states in the most different areas. A tendency to more natural environments could be noticed that considering physical or physiological signs and the individual’s relationship with the environment, other people, and daily activities. Therefore, what is the importance of combining ubiquitous computing, affective computing, and the contributions of context-aware information to provide more accurate and intelligent affective systems? This study presents a systematic literature mapping about the use of contextual information to identify affective states, which covered articles published between 2010 and October 2021, resulting in 1.638 studies. After applying filters, which we explain further in this article, we selected 49 works to answer a set of research questions. The results indicate that physiological data was the main parameter for recognizing affective signs (62.3%, 33/53), followed by visual data (32.1%, 17/53). The links between context and affective signs presented a more significant occurrence in the combination of contexts related to activities and physiological data (34%, 18/53).

The examination of factors influencing the recognition of affective states associated with tennis players’ non-verbal behaviour

In tennis, the non-verbal behaviours shown after a rally may indicate the affective state of players. The purpose of the present study was to assess whether (a) the point outcome, (b) the duration of video-excerpts, and (c) the tennis expertise of the participants would influence the recognition rates of the affective state. To that end, 115 participants were shown non-verbal behaviour of tennis players after a point and asked to rate whether the player had just won or lost a point. The results indicate that the recognition rates were higher for lost than for won points. Moreover, participants who were members of a tennis club had a higher recognition rate. Finally, there was no difference in the recognition rate regarding the duration of video excerpts. The findings point to a negativity bias and the bio-cultural framework in relation to the recognition of affective states associated with non-verbal behaviour.