Emotion Recognition Approach Research Articles

Cross-Modal Harmony: Low-Rank Fusion for Enhanced Artificial Emotion Recognition

Emotion recognition has become a subject of considerable interest in recent times, owing to its diverse and far-reaching applications in various fields. These applications span from enhancing human-computer interactions to assessing mental health and improving entertainment systems. The proposed study presents a novel approach for emotion recognition by fusing audio and video modalities using low-rank fusion techniques. The proposed methodology leverages the complementary nature of audio and video data in capturing emotional cues. Audio data often encapsulates tone, speech patterns, and vocal nuances, while video data captures facial expressions, body language, and gestures. However, the challenge lies in effectively integrating these two modalities to enhance recognition accuracy. To address the challenge, it employs low-rank fusion, a dimensionality reduction technique that extracts the most informative features from both modalities while minimizing redundancy. Furthermore, it presents the implementation of the chosen low-rank fusion algorithm in a real-world emotion recognition system. The results can contribute to advancing the field of emotion recognition by providing a practical and efficient solution for combining audio and video data to achieve more robust and accurate emotion classification. Keywords: Deep Learning; Emotion Recognition; Human-Computer Interaction; Low-Rank Fusion; Multimodal Fusion.

EEG emotion recognition approach using multi-scale convolution and feature fusion

EEG emotion recognition approach using multi-scale convolution and feature fusion

A Hybrid Gabor Filter-Convolutional Neural Networks Model for Facial Emotion Recognition System

Objectives: This research aims to develop a hybrid facial emotion recognition system using integrated machine learning feature extraction techniques with convolutional neural networks model for improving the accuracy of facial emotion recognition (FER). Methods: This study introduces a novel approach that integrates various machine learning feature extraction techniques (HoG, LBP, SIFT and Gabor Filters) with Convolutional Neural Networks (CNNs). The study utilized a set of 10,500 facial emotion images of the CK+48 dataset with six different facial emotion recognition. Findings: The result from the experiments shows that the proposed Gabor Filter-CNN approach for estimating FER had higher accuracy as compared to other CNN, HoG-CNN, LBP-CNN, and SIFT-CNN. The specific approaches is based on the Gabor Filter – CNN paired results with an accuracy rate of 99.85% outstanding other models. Novelty: The proposed approach would ensure the combination of the ML extraction techniques for capturing the details of the textures/features and CNNs for other deep hierarchical features and further better prediction. Keywords: Hybrid Model, Facial expressions, Emotion recognition approach, Machine Learning, Deep Learning, Gabor filter approach, Feature extraction, Convolutional neural network

Verse1-Chorus-Verse2 Structure: A Stacked Ensemble Approach for Enhanced Music Emotion Recognition

In this study, we present a novel approach for music emotion recognition that utilizes a stacked ensemble of models integrating audio and lyric features within a structured song framework. Our methodology employs a sequence of six specialized base models, each designed to capture critical features from distinct song segments: verse1, chorus, and verse2. These models are integrated into a meta-learner, resulting in superior predictive performance, achieving an accuracy of 96.25%. A basic stacked ensemble model was also used in this study to independently run the audio and lyric features for each song segment. The six-input stacked ensemble model surpasses the capabilities of models analyzing song parts in isolation. The pronounced enhancement underscores the importance of a bimodal approach in capturing the full spectrum of musical emotions. Furthermore, our research not only opens new avenues for studying musical emotions but also provides a foundational framework for future investigations into the complex emotional aspects of music.

Multimodal machine learning approach for emotion recognition using physiological signals

This study explores a novel approach to emotion recognition through machine learning, addressing the limitations of previous methods. While deep learning has shown promise in this field, it often requires significant computational resources and time. In response, we propose a multimodal approach utilizing Feature-level Fusion (FLF) and Decision-level Fusion (DLF) to enhance performance while reducing complexity. The study focuses on integrating electroencephalogram (EEG), electromyography (EMG), and electrooculogram (EOG) signals. Signal preprocessing involves extracting statistical features, power spectral density (PSD), and incremental entropy analysis. Recursive Feature Elimination (RFE) is employed as a feature selector, facilitating the fusion of different signal features. Three fusion strategies are explored: EEG with EOG, EEG with EMG, and a combination of EEG with EOG and EMG. For classification, the Bagging Classifier and K-Nearest Neighbors Algorithm are chosen. Results demonstrate promising accuracy rates, with 95.7% for arousal, 96.41% for valence in subject-dependent classification, 93.68% for arousal, and 93.23% for valence in subject-independent classification. This approach presents a viable alternative to deep learning methods, offering improved performance with reduced computational burden.

Sentiment Analysis-Based Automatic Stress and Emotion Recognition using Weighted Fused Fusion-Based Cascaded DTCN with Attention Mechanism from EEG Signal

When loaded with difficulties in fulfilling daily requirements, a lot of people in today’s world experience an emotional pressure known as stress. Stress that lasts for a short duration of time has more advantages as they are good for mental health. But, the persistence of stress for a long duration of time may lead to serious health impacts in individuals, such as high blood pressure, cardiovascular disease, stroke and so on. Long-term stress, if unidentified and not treated, may also result in personality disorder, depression and anxiety. The initial detection of stress has become more important to prevent the health issues that arise due to stress. Detection of stress based on brain signals for analysing the emotion in humans leads to accurate detection outcomes. Using EEG-based detection systems and disease, disability and disorders can be identified from the brain by utilising the brain waves. Sentiment Analysis (SA) is helpful in identifying the emotions and mental stress in the human brain. So, a system to accurately and precisely detect depression in human based on their emotion through the utilisation of SA is of high necessity. The development of a reliable and precise Emotion and Stress Recognition (ESR) system in order to detect depression in real-time using deep learning techniques with the aid of Electroencephalography (EEG) signal-based SA is carried out in this paper. The essentials needed for performing stress and emotion detection are gathered initially from benchmark databases. Next, the pre-processing procedures, like the removal of artifacts from the gathered EEG signal, are carried out on the implemented model. The extraction of the spectral attributes is carried out from the pre- processed. The extracted spectral features are considered the first set of features. Then, with the aid of a Conditional Variational Autoencoder (CVA), the deep features are extracted from the pre-processed signals forming a second set of features. The weights are optimised using the Adaptive Egret Swarm Optimisation Algorithm (AESOA) so that the weighted fused features are obtained from these two sets of extracted features. Then, a Cascaded Deep Temporal Convolution Network with Attention Mechanism (CDTCN-AM) is used to recognise stress and emotion. The validation of the results from the developed stress and emotion recognition approach is carried out against traditional models in order to showcase the effectiveness of the suggested approach.

Emotion classification using electrocardiogram and machine learning: A study on the effect of windowing techniques

Automated emotion recognition using physiological signals has gained significant attention in recent years due to its potential applications in human–computer interaction, healthcare, and psychology. Electrocardiogram (ECG) signals are widely used due to their non-invasiveness, high temporal resolution, and direct relationship with the autonomic nervous system. In this study, we propose a novel approach for ECG-based emotion recognition using time-series to image encoding techniques and texture-based features combined with machine learning algorithms and deep learning architectures. The ECG data used in this study were obtained from the Continuously Annotated Signals of Emotion (CASE) and Wearable Stress and Affect Detection (WESAD) datasets. We categorized emotional states based on valence and arousal annotations into four classes: High-Valence High-Arousal, High-Valence Low-Arousal, Low-Valence High-Arousal, and Low-Valence Low-Arousal. The ECGs were segmented into 5 and 7-window segments and transformed into 2D representations using Gramian Angular Summation Field, Markov Transition Field, Recurrence Plot (RP), and the triple-channel fusion of all these images. Total of 85 textural features based on the Gray-Level Co-occurrence Matrix (GLCM), Gray-Level Run Length Matrix, Zernike’s Moments, Hu’s Moments, Fractal Dimension Texture Analysis (FDTA), and First-Order Statistics were extracted. Classifiers, namely Random Forest, Support Vector Machine (SVM), eXtreme Gradient Boosting (XGB), 1D Convolutional Neural Network (CNN), and Multi-head Attention Network were considered to classify the emotional states. The performance of the classifiers varied depending on the time-series to image encoding technique, segmentation approaches, and the classifier employed. We achieved the highest Weighted F-measure (F-m) of 94.91% (RP + XGB) and 86.78% (RP + SVM) using the 7-window and 5-window approaches, respectively. Our proposed 1D CNN architecture achieved the highest classification metrics (F-m = 92.52%, Balanced accuracy = 92.0%, Recall = 91.96%, and Precision = 93.16%) with RP images in a 7-window approach. GLCM and FDTA features made significant contributions to the classification of emotional states. Overall, our results suggest that the proposed method holds promise for developing more accurate and efficient emotion recognition systems.

A robust semi-supervised deep learning approach for emotion recognition using EEG signals

A robust semi-supervised deep learning approach for emotion recognition using EEG signals

Research on Speech Emotion Recognition System Using Machine Learning

Speech Emotion Recognition (SER) remains a hot topic in the domain of affective computing, drawing considerable research interest. Its growing potential, advancements in algorithms, and real-world applications contribute to this interest. Human speech carries paralinguistic cues that can be quantitatively represented through features like pitch, intensity, and Mel-Frequency Cepstral Coefficients (MFCC). SER typically involves three main stages: data processing, feature extraction/selection, and classification based on emotional features present. These steps, tailored to the unique attributes of human speech, make machine learning methods a fitting choice for SER implementation. While recent studies in affective computing leverage various ML techniques for SER tasks, few delve into the techniques aiding these core steps. Moreover, the challenges within these stages and cutting-edge approaches addressing them often receive limited discussion or are overlooked in these works.This project introduces a pioneering Speech Emotion Recognition System leveraging BiLSTM Algorithm and Image Processing techniques, developed for implementation on the Raspberry Pi platform using Python. The system analyses recorded audio input, employing three LEDs for mood detection and interfacing with Raspberry Pi programming facilitated by an SD card. This abstract outlines an all-encompassing approach for real-time emotion recognition via audio analysis, catering to varied applications in emotional AI and human-computer interaction. KEYWORDS: Speech Emotion Recognition (SER), Machine Learning, Image Processing, Raspberry Pi.

Using transformers for multimodal emotion recognition: Taxonomies and state of the art review

Emotion recognition is an aspect of human-computer interaction, affective computing, and social robotics. Conventional unimodal approaches for emotion recognition, depending on single data sources such as facial expressions or speech signals often fall short in capturing the complexity and context-dependent nature of emotions. Multimodal Emotion Recognition (MER), which integrates information from multiple modalities, has emerged as a promising solution to overcome these limitations. In recent years, Transformers-based approaches have gathered significant attention in the fields of natural language processing and computer vision, highlighting their ability to capture long-range dependencies and semantic representations. These models have rapidly achieved the MER state-of-the-art. However, current survey papers that cover MER lack a specific focus on Transformer-based techniques. To bridge this research gap, this review paper provides a comprehensive investigation of Transformers-based approaches for MER. It explores various Transformer architectures and proposes several scenarios for using Transformers at different stages of MER process. In addition, it examines datasets suitable for MER, discusses fusion mechanisms, and introduces novel taxonomies in both MER and Transformer technologies. The review also addresses challenges and future research directions. Through this review, we aim to provide researchers with an inclusive understanding of the current state-of-the-art in Transformers-based approaches for MER, paving the way for further advancements in this rapidly developing field.

Modeling a Novel Approach for Emotion Recognition Using Learning and Natural Language Processing

Various facts, including politics, entertainment, industry, and research fields, are connected to analyzing the audience's emotions. Sentiment Analysis (SA) is a Natural Language Processing (NLP) concept that uses statistical and lexical forms as well as learning techniques to forecast how different types of content in social media will express the audience's neutral, positive, and negative emotions. There is lack of an adequate tool to quantify the characteristics and independent text for assessing the primary audience emotion from the available online social media dataset. The focus of this research is on modeling a cutting-edge method for decoding the connectivity among social media texts and assessing audience emotions. Here, a novel dense layer graph model (DLG-TF) for textual feature analysis is used to analyze the relevant connectedness inside the complex media environment to forecast emotions. The information from the social media dataset is extracted using some popular convolution network models, and the predictions are made by examining the textual properties. The experimental results show that, when compared to different standard emotions, the proposed DLG-TF model accurately predicts a greater number of possible emotions. The macro-average of baseline is 58%, the affective is 55%, the crawl is 55%, and the ultra-dense is 59%, respectively. The feature analysis comparison of baseline, affective, crawl, ultra-dense and DLG-TF using the unsupervised model based on EmoTweet gives the precision, recall, and F1-score of the anticipated model are explained. The micro- and macro-average based on these parameters are compared and analyzed. The macro-average of baseline is 47%, the affective is 46%, the crawl is 50%, and the ultra-dense is 85%, respectively. It makes precise predictions using the social media dataset that is readily available. A few criteria, including accuracy, recall, precision, and F-measure, are assessed and contrasted with alternative methods.

Optimizing Speech Emotion Recognition with Deep Learning and Grey Wolf Optimization: A Multi-Dataset Approach

Machine learning and speech emotion recognition are rapidly evolving fields, significantly impacting human-centered computing. Machine learning enables computers to learn from data and make predictions, while speech emotion recognition allows computers to identify and understand human emotions from speech. These technologies contribute to the creation of innovative human–computer interaction (HCI) applications. Deep learning algorithms, capable of learning high-level features directly from raw data, have given rise to new emotion recognition approaches employing models trained on advanced speech representations like spectrograms and time–frequency representations. This study introduces CNN and LSTM models with GWO optimization, aiming to determine optimal parameters for achieving enhanced accuracy within a specified parameter set. The proposed CNN and LSTM models with GWO optimization underwent performance testing on four diverse datasets—RAVDESS, SAVEE, TESS, and EMODB. The results indicated superior performance of the models compared to linear and kernelized SVM, with or without GWO optimizers.

Modelling an efficient hybridized approach for facial emotion recognition using unconstraint videos and deep learning approaches

Modelling an efficient hybridized approach for facial emotion recognition using unconstraint videos and deep learning approaches

Multimodal emotion recognition based on the fusion of vision, EEG, ECG, and EMG signals

This paper presents a novel approach for emotion recognition (ER) based on Electroencephalogram (EEG), Electromyogram (EMG), Electrocardiogram (ECG), and computer vision. The proposed system includes two different models for physiological signals and facial expressions deployed in a real-time embedded system. A custom dataset for EEG, ECG, EMG, and facial expression was collected from 10 participants using an Affective Video Response System. Time, frequency, and wavelet domain-specific features were extracted and optimized, based on their Visualizations from Exploratory Data Analysis (EDA) and Principal Component Analysis (PCA). Local Binary Patterns (LBP), Local Ternary Patterns (LTP), Histogram of Oriented Gradients (HOG), and Gabor descriptors were used for differentiating facial emotions. Classification models, namely decision tree, random forest, and optimized variants thereof, were trained using these features. The optimized Random Forest model achieved an accuracy of 84%, while the optimized Decision Tree achieved 76% for the physiological signal-based model. The facial emotion recognition (FER) model attained an accuracy of 84.6%, 74.3%, 67%, and 64.5% using K-Nearest Neighbors (KNN), Random Forest, Decision Tree, and XGBoost, respectively. Performance metrics, including Area Under Curve (AUC), F1 score, and Receiver Operating Characteristic Curve (ROC), were computed to evaluate the models. The outcome of both results, i.e., the fusion of bio-signals and facial emotion analysis, is given to a voting classifier to get the final emotion. A comprehensive report is generated using the Generative Pretrained Transformer (GPT) language model based on the resultant emotion, achieving an accuracy of 87.5%. The model was implemented and deployed on a Jetson Nano. The results show its relevance to ER. It has applications in enhancing prosthetic systems and other medical fields such as psychological therapy, rehabilitation, assisting individuals with neurological disorders, mental health monitoring, and biometric security.

ECG-based emotion recognition using random convolutional kernel method

The recognition of human emotions based on physiological signals has garnered significant attention from various fields such as human–computer interaction, cognitive-behavioral science, and the treatment of emotion-related diseases. Using the easily accessible ECG signal for emotion recognition presents a viable approach. However, method relying on hand-crafted features necessitate task-specific domain knowledge and consume substantial time to design suitable features, and studies employing deep learning technologies incur greater computational complexity and extended training durations. To address these problems, we proposed an emotion recognition method that employs random convolutional kernels on ECG signals. Following ECG signal preprocessing, a large number of random convolutional kernels were used to transform the signals, and the resulting features are utilized to classify the various emotion states. In addition to accurately and automatically extracting the multi-scale features from ECG signals, combining the random convolutional kernels with ECG signal alone also reduces the computational complexity and training time compared to methods using multiple physiological signals or deep neural networks. Validation on three publicly available datasets revealed that the proposed method achieved recognition accuracies of 91.5%, 96% and 92.5% for valence, arousal and dominance in AMIGOS dataset, and 95.1%, 98.8% and 90.2% for same labels in DREAMER dataset. The accuracy for valence, arousal and affective states in WESAD dataset are 94.5%, 91.8% and 91.8%. The results show that the proposed method outperforms previous researches in multi-class emotion recognition, thus offering a new and promising approach for swift and precise emotion recognition.

Automated EEG based Emotion Detection using Bonobo Optimizer with Deep Learning on Human Computer Interaction

Recently, Emotion detection utilizing EEG signals develops popularity in domain of Human-Computer Interaction (HCI). EEG (electroencephalography) is a non-invasive approach, which processes electrical action from the brain through electrodes located in the scalp. An emotion recognition approach could not only be significant for healthy people among them disabled persons for detecting emotional changes and is utilized for different applications. It is significant to realize that emotion recognition in EEG indications is a difficult task owing to difficult and subjective nature of emotions. In recent times, Machine learning (ML) algorithms like Random Forests or Support Vector Machines (SVM) and Deep Learning (DL) systems namely Recurrent Neural Network (RNN) or Convolutional Neural Network (CNN) are trained on EEG feature extracted and connected emotional labels for classifying the user emotional state. This study presents an Automated EEG-based Emotion Detection using Bonobo Optimizer with Deep Learning (AEEGED-BODL) technique on HCI applications. The goal of the study is to analyze the EEG signals for the classification of several kinds of emotions in HCI applications. To achieve this, the AEEGED-BODL technique uses Higuchi fractal dimension (HFD) approach for extracting features in the EEG signals. Besides, the AEEGED-BODL technique makes use of the quasi-recurrent neural network (QRNN) approach for the detection and classification of distinct kinds of emotions. Furthermore, the BO system was demoralized for optimum hyperparameter selection of QRNN model, which helps in attaining an improved detection rate. The simulation validation of AEEGED-BODL algorithm was simulated on EEG signal database. The comprehensive result stated best outcome of the AEEGED-BODL algorithm over other recent approaches

A Transfer Learning Approach for Facial Emotion Recognition Using a Deep Learning Model

The facial expression recognition (FER) system is the process of identifying the emotional state of a person. Emotion recognition from facial expressions is a rapidly growing area of research with numerous applications, including psychology, marketing, and human-computer interaction. This paper presents a novel approach to FER employing deep learning techniques, specifically leveraging the power of transfer learning methodology. To address the challenge of accurately recognizing different emotions, including angry, disgusted, afraid, happy, sad, surprised, and neutral, from facial expressions, a pre-trained model MobileNetV2 architecture has been fine-tuned for improving accuracy. It is a lightweight convolutional neural network architecture, specifically designed for efficient on-device inference. For evaluating the performance of the proposed FER model, the state-of-the-art FER-2013 dataset, along random images and video clips has been employed. Experimental results demonstrate that the proposed FER model attained a remarkable accuracy rate exceeding 99% when tested on diverse sets of random images and video clips. Moreover, the system achieved a notable accuracy of 61% when evaluated against the FER-2013 dataset. Overall, this approach presents a significant advancement in the field of real-time facial expression recognition using the MobileNetV2 architecture with FER2013 dataset, which may improve the quality of human-computer interactions.

GraphCFC: A Directed Graph Based Cross-Modal Feature Complementation Approach for Multimodal Conversational Emotion Recognition

Emotion Recognition in Conversation (ERC) plays a significant part in Human-Computer Interaction (HCI) systems since it can provide empathetic services. Multimodal ERC can mitigate the drawbacks of uni-modal approaches. Recently, Graph Neural Networks (GNNs) have been widely used in a variety of fields due to their superior performance in relation modeling. In multimodal ERC, GNNs are capable of extracting both long-distance contextual information and inter-modal interactive information. Unfortunately, since existing methods such as MMGCN directly fuse multiple modalities, redundant information may be generated and diverse information may be lost. In this work, we present a directed Graph based Cross-modal Feature Complementation (GraphCFC) module that can efficiently model contextual and interactive information. GraphCFC alleviates the problem of heterogeneity gap in multimodal fusion by utilizing multiple subspace extractors and Pair-wise Cross-modal Complementary (PairCC) strategy. We extract various types of edges from the constructed graph for encoding, thus enabling GNNs to extract crucial contextual and interactive information more accurately when performing message passing. Furthermore, we design a GNN structure called GAT-MLP, which can provide a new unified network framework for multimodal learning. The experimental results on two benchmark datasets show that our GraphCFC outperforms the state-of-the-art (SOTA) approaches.

A Multimodal Low Complexity Neural Network Approach for Emotion Recognition

This paper introduces a neural network‐based model designed for classifying emotional states by leveraging multimodal physiological signals. The model utilizes data from the AMIGOS and SEED‐V databases. The AMIGOS database integrates inputs from electroencephalogram (EEG), electrocardiogram (ECG), and galvanic skin response (GSR) to analyze emotional responses, while the SEED‐V database continuously updates EEG signals. We implemented a sequential neural network architecture featuring two hidden layers, which underwent substantial hyperparameter tuning to achieve optimal performance. Our model’s effectiveness was tested through binary classification tasks focusing on arousal and valence, as well as a more complex four‐class classification that delineates emotional quadrants for the emotional tags: happy, sad, neutral, and disgust. In these varied scenarios, the model consistently demonstrated accuracy levels ranging from 79% to 86% in the AMIGOS database and up to 97% in SEED‐V. A notable aspect of our approach is the model’s ability to accurately recognize emotions without the need for extensive signal preprocessing, a common challenge in multimodal emotion analysis. This feature enhances the practical applicability of our model in real‐world scenarios where rapid and efficient emotion recognition is essential.

A novel PSO-ViT approach for facial emotion recognition

ABSTRACT Facial Emotion Recognition is gaining interest from researchers in various fields due to its numerous applications. The Vision Transformer (ViT) outperforms convolutional neural network (CNN)-based systems significantly in terms of performance when compared to contemporary image categorisation algorithms. In this context, the primary objective of this paper is to present an innovative and efficient deep learning model for facial emotion recognition. This hybrid intelligent model combines the Vision Transformer with Particle Swarm Optimization (PSO). The proposed model accurately identified the majority of emotions in images, achieving an accuracy of 95.93% for the Fer2013plus dataset and 100% for the CK+ dataset, as demonstrated through experiments conducted on both datasets. According to the results, the proposed facial expression recognition approach has either outperformed or closely matched the performance of state-of-the-art methods.