Emotion Recognition Applications Research Articles

A deep spatiotemporal interaction network for multimodal sentimental analysis and emotion recognition

One of the challenges of sentiment analysis and emotion recognition is how to effectively fuse the multimodal inputs. The transformer-based models have achieved great success in applications of multimodal sentiment analysis and emotion recognition recently. However, the transformer-based model often neglects the coherence of human emotion due to its parallel structure. Additionally, a low-rank bottleneck created by multi- attention-head causes an inadequate fitting ability of models. To tackle these issues, a Deep Spatiotemporal Interaction Network (DSIN) is proposed in this study. It consists of two main components, i.e., a cross-modal transformer with a cross-talking attention module and a hierarchically temporal fusion module, where the cross-modal transformer is used to model the spatial interactions between different modalities and the hierarchically temporal fusion network is utilized to model the temporal coherence of emotion. Therefore, the DSIN can model the spatiotemporal interactions of multimodal inputs by incorporating the time-dependency into the parallel structure of transformer and decrease the redundancy of embedded features by implanting their spatiotemporal interactions into a hybrid memory network in a hierarchical manner. The experimental results on two benchmark datasets indicate that DSIN achieves superior performance compared with the state-of-the-art models, and some useful insights are derived from the results.

Voice Tone Emotion Recognition for the Depressed: A review

This study investigates the application of voice tone emotion recognition for individuals with depression, utilizing advanced machine learning techniques. The research focuses on analyzing voice tone to detect emotional states, providing a non-invasive method to monitor and potentially diagnose depression. Audio data from participants were preprocessed through noise reduction and normalization, followed by feature extraction, including pitch, tone, and frequency. The extracted features were then analyzed using a combination of traditional machine learning and deep learning models. Our model achieved an accuracy of 85% in emotion recognition, with high precision and recall for neutral and happy tones. The findings highlight the model's potential in enhancing mental health monitoring and interventions. By enabling continuous, remote assessment of emotional states, this technology can aid in early detection, personalized therapy, and timely interventions for depression. The study also addresses existing challenges, such as variability in speech patterns and ethical considerations, proposing future research directions to improve accuracy and applicability. This research underscores the transformative impact of emotion recognition technology in mental health, offering promising applications in telehealth and personalized care for individuals with depression. Keywords: Voice tone, emotion recognition, depression, mental health, machine learning.

Emotion Recognition Based on Handwriting Using Generative Adversarial Networks and Deep Learning

The quality of people’s lives is closely related to their emotional state. Positive emotions can boost confidence and help overcome difficulties, while negative emotions can harm both physical and mental health. Research has shown that people’s handwriting is associated with their emotions. In this study, audio-visual media were used to induce emotions, and a dot-matrix digital pen was used to collect neutral text data written by participants in three emotional states: calm, happy, and sad. To address the challenge of limited samples, a novel conditional table generative adversarial network called conditional tabular-generative adversarial network (CTAB-GAN) was used to increase the number of task samples, and the recognition accuracy of task samples improved by 4.18%. The TabNet (a neural network designed for tabular data) with SimAM (a simple, parameter-free attention module) was employed and compared with the original TabNet and traditional machine learning models; the incorporation of the SimAm attention mechanism led to a 1.35% improvement in classification accuracy. Experimental results revealed significant differences between negative (sad) and nonnegative (calm and happy) emotions, with a recognition accuracy of 80.67%. Overall, this study demonstrated the feasibility of emotion recognition based on handwriting with the assistance of CTAB-GAN and SimAm-TabNet. It provides guidance for further research on emotion recognition or other handwriting-based applications.

Automatic detection of expressed emotion from Five-Minute Speech Samples: Challenges and opportunities.

Research into clinical applications of speech-based emotion recognition (SER) technologies has been steadily increasing over the past few years. One such potential application is the automatic recognition of expressed emotion (EE) components within family environments. The identification of EE is highly important as they have been linked with a range of adverse life events. Manual coding of these events requires time-consuming specialist training, amplifying the need for automated approaches. Herein we describe an automated machine learning approach for determining the degree of warmth, a key component of EE, from acoustic and text natural language features. Our dataset of 52 recorded interviews is taken from recordings, collected over 20 years ago, from a nationally representative birth cohort of British twin children, and was manually coded for EE by two researchers (inter-rater reliability 0.84-0.90). We demonstrate that the degree of warmth can be predicted with an F1-score of 64.7% despite working with audio recordings of highly variable quality. Our highly promising results suggest that machine learning may be able to assist in the coding of EE in the near future.

A Comprehensive Evaluation of Features and Simple Machine Learning Algorithms for Electroencephalographic-Based Emotion Recognition

The study of electroencephalographic (EEG) signals has gained popularity in recent years because they are unlikely to intentionally fake brain activity. However, the reliability of the results is still subject to various noise sources and potential inaccuracies inherent to the acquisition process. Analyzing these signals involves three main processes: feature extraction, feature selection, and classification. The present study extensively evaluates feature sets across domains and their impact on emotion recognition. Feature selection improves results across the different domains. Additionally, hybrid models combining features from various domains offer a superior performance when applying the public DEAP dataset for emotion classification using EEG signals. Time, frequency, time–frequency, and spatial domain attributes and their combinations were analyzed. The effectiveness of the input vectors for the classifiers was validated using SVM, KNN, and ANN, which are simple classification algorithms selected for their widespread use and better performance in the state of the art. The use of simple machine learning algorithms makes the findings particularly valuable for real-time emotion recognition applications where the computational resources and processing time are often limited. After the analysis stage, feature vector combinations were proposed to identify emotions in four quadrants of the valence–arousal representation space using the DEAP dataset. This research achieved a classification accuracy of 96% using hybrid features in the four domains and the ANN classifier. A lower computational cost was obtained in the frequency domain.

MGFKD: A semi-supervised multi-source domain adaptation algorithm for cross-subject EEG emotion recognition

Currently, most models rarely consider the negative transfer problem in the research field of cross-subject EEG emotion recognition. To solve this problem, this paper proposes a semi-supervised domain adaptive algorithm based on few labeled samples of target subject, which called multi-domain geodesic flow kernel dynamic distribution alignment (MGFKD). It consists of three modules: 1) GFK common feature extractor: projects the feature distribution of source and target subjects to the Grassmann manifold space, and obtains the latent common features of the two feature distributions through GFK method. 2) Source domain selector: obtains pseudo-labels of the target subject through weak classifier, finds "golden source subjects" by using few known labels of target subjects. 3) Label corrector: uses a dynamic distribution balance strategy to correct the pseudo-labels of the target subject. We conducted comparison experiments on the SEED and SEED-IV datasets, and the results show that MGFKD outperforms unsupervised and semi-supervised domain adaptation algorithms, achieving an average accuracy of 87.51±7.68% and 68.79±8.25% on the SEED and SEED-IV datasets with only one labeled sample per video for target subject. Especially when the number of source domains is set as 6 and the number of known labels is set as 5, the accuracy increase to 90.20±7.57% and 69.99±7.38%, respectively. The above results prove that our proposed algorithm can efficiently and quickly improve the cross-subject EEG emotion classification performance. Since it only need a small number of labeled samples of new subjects, making it has strong application value in future EEG-based emotion recognition applications.

Human-computer interaction: Evaluation of trust and emotional

With the advancement of information technology, human-computer interaction is becoming increasingly intelligent. This progress has led to the transformation of the car cabin from a simple driving space to an intelligent mobile life terminal. At the same time, more and more people have psychological problems in modern society, and timely adjustment of their negative emotions is of great practical significance to social harmony and stability. Traditional methods of emotion regulation require a lot of manpower and energy. Hence, this paper presents an approach that utilizes text dialogue and somatosensory interaction to detect the users emotions and subsequently adjust them. By examining and analyzing the current research landscape, the objective of this study is to offer insights for designing trustworthy and emotionally-responsive interactions in the automotive context. It can bring a better driving experience to smart car users and bring benefits to many companies. It is of great scientific and application value to carry out this research in depth to promote the development of human-computer interaction. Through literature review, case study, theoretical analysis and other research methods. This paper sorts out the establishment of trust degree and recognition of emotional changes in human-computer interaction, and puts forward the role, defects and improvements of trust establishment in human-car cooperation, as well as the reasons affecting user emotions, and how to improve the defects of the application of voice emotion recognition. It also discusses difficulties in establishing trust and misperception of emotions.

Trusted emotion recognition based on multiple signals captured from video and its application in intelligent education

<abstract> <p>The emotional variation can reflect shifts in mental and emotional states. It plays an important role in the field of intelligent education. Emotion recognition can be used as cues for teachers to evaluate the learning state, analyze learning motivation, interest, and efficiency. Although research on emotion recognition has been ongoing for a long time, there has been a restricted emphasis on analyzing the credibility of the recognized emotions. In this paper, the origin, development, and application of emotion recognition were introduced. Then, multiple signals captured from video that could reflect emotion changes were described in detail and their advantages and disadvantages were discussed. Moreover, a comprehensive summary of the pertinent applications and research endeavors of emotion recognition technology in the field of education was provided. Last, the trend of emotion recognition in the field of education was given.</p> </abstract>

Leveraging a Hybrid Deep Learning Architecture for Efficient Emotion Recognition in Audio Processing

This paper presents a novel hybrid deep learning architecture for emotion recognition from speech signals, which has garnered significant interest in recent years due to its potential applications in various fields such as healthcare, psychology, and entertainment. The proposed architecture combines modified ResNet-34 and RoBERTa models to extract meaningful features from speech signals and classify them into different emotion categories. The model is evaluated on five standard emotion recognition datasets, including RAVDESS, EmoDB, SAVEE, CREMA-D, and TESS, and achieves state-of-the-art performance on all datasets. The experimental results show that the proposed hybrid architecture outperforms existing emotion recognition models, achieving high accuracy and F1 scores for emotion classification. The proposed architecture is promising for real-time emotion recognition applications and can be applied in various domains such as speech-based emotion recognition systems, human-computer interaction, and virtual assistants.

Self-aware face emotion accelerated recognition algorithm: a novel neural network acceleration algorithm of emotion recognition for international students.

With an increasing number of human-computer interaction application scenarios, researchers are looking for computers to recognize human emotions more accurately and efficiently. Such applications are desperately needed at universities, where people want to understand the students' psychology in real time to avoid catastrophes. This research proposed a self-aware face emotion accelerated recognition algorithm (SFEARA) that improves the efficiency of convolutional neural networks (CNNs) in the recognition of facial emotions. SFEARA will recognize that critical and non-critical regions of input data perform high-precision computation and convolutive low-precision computation during the inference process, and finally combine the results, which can help us get the emotional recognition model for international students. Based on a comparison of experimental data, the SFEARA algorithm has 1.3× to 1.6× higher computational efficiency and 30% to 40% lower energy consumption than conventional CNNs in emotion recognition applications, is better suited to the real-time scenario with more background information.

Facial Sentiment Recognition using artificial intelligence techniques.

Facial emotion recognition technology is used to analyze and recognize human emotions based on facial expressions. This technology uses deep learning models to classify facial expressions, eyes, eyebrows, mouth, and other facial expressions to determine a person's emotions. The application of facial emotion recognition in the field of education is a potential way to evaluate the level of student absorption after each class period. Using cameras and emotion recognition technology, the system can record and analyze students' facial expressions during class. In this paper, we use the Convolutional Neural Network (CNN) algorithm combined with the linear regression analysis method to build a model to predict students' facial emotions over a period of time camera recorded.

EEG SIGNAL-DRIVEN HUMAN–COMPUTER INTERACTION EMOTION RECOGNITION MODEL USING AN ATTENTIONAL NEURAL NETWORK ALGORITHM

The level of human–machine interaction experience is raising its bar as artificial intelligence develops quickly. An important trend in this application is the improvement of the friendliness, harmony, and simplicity of human–machine communication. Electroencephalogram (EEG) signal-driven emotion identification has recently gained popularity in the area of human–computer interaction (HCI) because of its advantages of being simple to extract, difficult to conceal, and real-time differences. The corresponding research is ultimately aimed at imbuing computers with feelings to enable fully harmonic and organic human–computer connections. This study applies three-dimensional convolutional neural networks (3DCNNs) and attention mechanisms to an environment for HCI and offers a dual-attention 3D convolutional neural networks (DA-3DCNNs) model from the standpoint of spatio-temporal convolution. With the purpose of extracting more representative spatio-temporal characteristics, the new model first thoroughly mines the spatio-temporal distribution information of EEG signals using 3DCNN, taking into account the temporal fluctuation of EEG data. Yet, a dual-attention technique based on EEG channels is utilized at the same time to strengthen or weaken the feature information and understand the links between various brain regions and emotional activities, highlighting the variations in the spatiotemporal aspects of various emotions. Finally, three sets of experiments were planned on the DEAP dataset for cross-subject emotion classification experiments, channel selection experiments, and ablation experiments, respectively, to show the validity and viability of the DA-3DCNN model for HCI emotion recognition applications. The outcomes show that the new model may significantly increase the model’s accuracy in recognizing emotions, acquire the spatial relationship of channels, and more thoroughly extract dynamic information from EEG.

Automatic Classification of Emotions Based on Cardiac Signals: A Systematic Literature Review.

Emotions play a pivotal role in human cognition, exerting influence across diverse domains of individuals' lives. The widespread adoption of artificial intelligence and machine learning has spurred interest in systems capable of automatically recognizing and classifying emotions and affective states. However, the accurate identification of human emotions remains a formidable challenge, as they are influenced by various factors and accompanied by physiological changes. Numerous solutions have emerged to enable emotion recognition, leveraging the characterization of biological signals, including the utilization of cardiac signals acquired from low-cost and wearable sensors. The objective of this work was to comprehensively investigate the current trends in the field by conducting a Systematic Literature Review (SLR) that focuses specifically on the detection, recognition, and classification of emotions based on cardiac signals, to gain insights into the prevailing techniques employed for signal acquisition, the extracted features, the elicitation process, and the classification methods employed in these studies. A SLR was conducted using four research databases, and articles were assessed concerning the proposed research questions. Twenty seven articles met the selection criteria and were assessed for the feasibility of using cardiac signals, acquired from low-cost and wearable devices, for emotion recognition. Several emotional elicitation methods were found in the literature, including the algorithms applied for automatic classification, as well as the key challenges associated with emotion recognition relying solely on cardiac signals. This study extends the current body of knowledge and enables future research by providing insights into suitable techniques for designing automatic emotion recognition applications. It emphasizes the importance of utilizing low-cost, wearable, and unobtrusive devices to acquire cardiac signals for accurate and accessible emotion recognition.

API Face Value

Emotion recognition application programming interface (API) is a recent advancement in computing technology that synthesizes computer vision, machine-learning algorithms, deep-learning neural networks, and other information to detect and label human emotions. The strongest iterations of this technology are produced by technology giants with large, cloud infrastructure (i.e., Google, and Microsoft), bolstering high true positive rates. We review the current status of applications of emotion recognition API in psychological research and find that, despite evidence of spatial, age, and race bias effects, API is improving the accessibility of clinical and educational research. Specifically, emotion detection software can assist individuals with emotion-related deficits (e.g., Autism Spectrum Disorder, Attention Deficit-Hyperactivity Disorder, Alexithymia). API has been incorporated in various computer-assisted interventions for Autism, where it has been used to diagnose, train, and monitor emotional responses to one's environment. We identify AP's potential to enhance interventions in other emotional dysfunction populations and to address various professional needs. Future work should aim to address the bias limitations of API software and expand its utility in subfields of clinical, educational, neurocognitive, and industrial-organizational psychology.

Intelligent emotion recognition application in university students’ psychological intervention by using mobile voice terminal

Intelligent emotion recognition application in university students’ psychological intervention by using mobile voice terminal

Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition

EEG signals have been reported to be informative and reliable for emotion recognition in recent years. However, the inter-subject variability of emotion-related EEG signals still poses a great challenge for the practical applications of EEG-based emotion recognition. Inspired by recent neuroscience studies on inter-subject correlation, we proposed a Contrastive Learning method for Inter-Subject Alignment (CLISA) to tackle the cross-subject emotion recognition problem. Contrastive learning was employed to minimize the inter-subject differences by maximizing the similarity in EEG signal representations across subjects when they received the same emotional stimuli in contrast to different ones. Specifically, a convolutional neural network was applied to learn inter-subject aligned spatiotemporal representations from EEG time series in contrastive learning. The aligned representations were subsequently used to extract differential entropy features for emotion classification. CLISA achieved state-of-the-art cross-subject emotion recognition performance on our THU-EP dataset with 80 subjects and the publicly available SEED dataset with 15 subjects. It could generalize to unseen subjects or unseen emotional stimuli in testing. Furthermore, the spatiotemporal representations learned by CLISA could provide insights into the neural mechanisms of human emotion processing.

SSTM-IS: simplified STM method based on instance selection for real-time EEG emotion recognition.

EEG signals can non-invasively monitor the brain activities and have been widely used in brain-computer interfaces (BCI). One of the research areas is to recognize emotions objectively through EEG. In fact, the emotion of people changes over time, however, most of the existing affective BCIs process data and recognize emotions offline, and thus cannot be applied to real-time emotion recognition. In order to solve this problem, we introduce the instance selection strategy into transfer learning and propose a simplified style transfer mapping algorithm. In the proposed method, the informative instances are firstly selected from the source domain data, and then the update strategy of hyperparameters is also simplified for style transfer mapping, making the model training more quickly and accurately for a new subject. To verify the effectiveness of our algorithm, we carry out the experiments on SEED, SEED-IV and the offline dataset collected by ourselves, and achieve the recognition accuracies up to 86.78%, 82.55% and 77.68% in computing time of 7s, 4s and 10s, respectively. Furthermore, we also develop a real-time emotion recognition system which integrates the modules of EEG signal acquisition, data processing, emotion recognition and result visualization. Both the results of offline and online experiments show that the proposed algorithm can accurately recognize emotions in a short time, meeting the needs of real-time emotion recognition applications.

A COMPARATIVE ANALYSIS OF EDA DECOMPOSITION METHODS FOR IMPROVED EMOTION RECOGNITION

This study analyzed five decomposition algorithms for separating electrodermal activity (EDA) into tonic and phasic components to identify different emotions using machine learning algorithms. We used EDA signals from the Continuously Annotated Signals of Emotion dataset for this analysis. First, we decomposed the EDA signals into tonic and phasic components using five decomposition methods: continuous deconvolution analysis, discrete deconvolution analysis, convex optimization-based EDA, nonnegative sparse deconvolution (SparsEDA), and BayesianEDA. We extracted time, frequency, and time-frequency domain features from each decomposition method’s tonic and phasic components. Finally, various machine learning algorithms such as logistic regression (LR), support vector machine, random forest, extreme gradient boosting, and multilayer perceptron were applied to evaluate the performance of the decomposition methods. Our results show that the considered decomposition methods successfully split the EDA signal into tonic and phasic components. The SparsEDA decomposition method outperforms the other decomposition methods considered in the study. In addition, LR with features extracted from the tonic component of the SparsEDA achieved highest average classification accuracy of 95.83%. This study can be used to identify the optimal decomposition methods suitable for emotion recognition applications.

FERFM: An Enhanced Facial Emotion Recognition System Using Fine-tuned MobileNetV2 Architecture

Deep learning techniques have recently drawn a lot of attention towards automatic facial emotion recognition (FER) applications such as diagnosis of psychiatric disorders, mental health monitoring, and other human–computer interaction applications. But the models that have been developed, have still not been generalized due to the absence of substantial emotion datasets and high operational costs. To address these issues, we propose a novel technique called facial emotion recognition using fine-tuned MobileNetV2 architecture (FERFM). It uses the concept of transfer learning to improve the performance of FER systems for mobile devices. It transfers the knowledge from pre-trained ImageNet dataset to profile view RGB-KDEF dataset for diversity among learnt features. Initially, data descriptive analysis is performed on input KDEF images to automate the necessary knowledge. Then the pipeline strategy is introduced, where the pre-trained MobileNetV2 architecture is fine tuned by eliminating the last six layers and adding dropout regularization, max pooling and dense layer. To test the performance of the proposed work, we ran several tests using the KDEF-RGB dataset, FER13 dataset and real-time facial expression images to classify the results into seven discrete categories of emotions. It was also verified on VGG-16 pretrained model and demonstrated remarkable performance comparatively. Proposed FERFM achieves 85.7% accuracy and requires 1,510,599 number of trainable parameters, 43 ms running time per image that proves the enhancement of FER performance and outperforms the existing state of art methods. Also, the ability to achieve low operational costs while maintaining high accuracy demonstrates proficiency in mobile applications.

Spatio-temporal deep forest for emotion recognition based on facial electromyography signals

Emotion recognition is a key component of human–computer interaction technology, for which facial electromyogram (fEMG) is an important physiological modality. Recently, deep-learning-based emotion recognition using fEMG signals has drawn increased attention. However, the ability of effective feature extraction and the demand of large-scale training data are two dominant factors that restrict the performance of emotion recognition. In this paper, a novel spatio-temporal deep forest (STDF) model is proposed to classify three categories of discrete emotions (neutral, sadness, and fear) using multi-channel fEMG signals.The feature extraction module fully extracts effective spatio-temporal features of fEMG signals using a combination of 2D frame sequences and multi-grained scanning. Meanwhile, a cascade forest-based classifier is designed to provide optimal structures for different scales of training data via automatically adjusting the number of cascade layers.The proposed model and five comparison methods were evaluated on our in-house fEMG dataset that included three discrete emotions and three channels of fEMG electrodes with a total of twenty-seven subjects. Experimental results demonstrate that the proposed STDF model achieves the best recognition performance with an average accuracy of 97.41%. Besides, our proposed STDF model can reduced the scale of training data to 50% while the average accuracy of emotion recognition is only reduced by about 5%. Our proposed model offers an effective solution for practical applications of fEMG-based emotion recognition.