Affective Computing Research Articles

High-Accuracy Classification of Multiple Distinct Human Emotions Using EEG Differential Entropy Features and ResNet18

The high-accuracy detection of multiple distinct human emotions is crucial for advancing affective computing, mental health diagnostics, and human–computer interaction. The integration of deep learning networks with entropy measures holds significant potential in neuroscience and medicine, especially for analyzing EEG-based emotion states. This study proposes a method combining ResNet18 with differential entropy to identify five types of human emotions (happiness, sadness, fear, disgust, and neutral) from EEG signals. Our approach first calculates the differential entropy of EEG signals to capture the complexity and variability of the emotional states. Then, the ResNet18 network is employed to learn feature representations from the differential entropy measures, which effectively captures the intricate spatiotemporal dynamics inherent in emotional EEG patterns using residual connections. To validate the efficacy of our method, we conducted experiments on the SEED-V dataset, achieving an average accuracy of 95.61%. Our findings demonstrate that the combination of ResNet18 with differential entropy is highly effective in classifying multiple distinct human emotions from EEG signals. This method shows robust generalization and broad applicability, indicating its potential for extension to various pattern recognition tasks across different domains.

Automated recognition of emotional states of horses from facial expressions.

Animal affective computing is an emerging new field, which has so far mainly focused on pain, while other emotional states remain uncharted territories, especially in horses. This study is the first to develop AI models to automatically recognize horse emotional states from facial expressions using data collected in a controlled experiment. We explore two types of pipelines: a deep learning one which takes as input video footage, and a machine learning one which takes as input EquiFACS annotations. The former outperforms the latter, with 76% accuracy in separating between four emotional states: baseline, positive anticipation, disappointment and frustration. Anticipation and frustration were difficult to separate, with only 61% accuracy.

DEEP LEARNING BASED EMOTION DETECTION SYSTEM

Emotion detection plays a crucial role in human-computer interaction, affective computing, and psychological research. Traditional methods of emotion detection often rely on manual annotations or subjective assessments, which can be time- consuming and prone to biases. In this paper, we propose a Deep Learning-Based Emotion Detection System that leverages advanced neural network architectures to automatically recognize emotions from facial expressions in real-time. The system utilizes convolutional neural networks (CNNs) for feature extraction from facial images and recurrent neural networks (RNNs) for sequence modeling of temporal dynamics in facial expressions. Our proposed system addresses key challenges in emotion detection, including variations in facial expressions, lighting conditions, and occlusions. By training on large-scale datasets of labeled facial images, the system learns to accurately classify emotions into predefined categories such as happiness, sadness, anger, fear, disgust, and surprise. We evaluate the performance of the system using standard metrics such as accuracy, precision, recall, and F1-score, demonstrating its effectiveness in real- world scenarios. The results indicate that the Deep Learning-Based Emotion Detection System achieves state-of-the-art performance in recognizing emotions from facial expressions, outperforming traditional methods. The system's ability to analyze facial dynamics in real-time opens up opportunities for applications in human-computer interaction, virtual reality, mental health monitoring, and personalized user experiences. Keywords: Deep Learning, Emotion Detection, Facial Expression Recognition, Convolutional Neural Networks, Recurrent Neural Networks, Affective Computing. Keywords: emotion Detection, Machine Learning, Classifier, Natural Language Processing

Cross-corpus speech emotion recognition with transformers: Leveraging handcrafted features and data augmentation

Speech emotion recognition (SER) stands as a prominent and dynamic research field in data science due to its extensive application in various domains such as psychological assessment, mobile services, and computer games, mobile services. In previous research, numerous studies utilized manually engineered features for emotion classification, resulting in commendable accuracy. However, these features tend to underperform in complex scenarios, leading to reduced classification accuracy. These scenarios include: 1. Datasets that contain diverse speech patterns, dialects, accents, or variations in emotional expressions. 2. Data with background noise. 3. Scenarios where the distribution of emotions varies significantly across datasets can be challenging. 4. Combining datasets from different sources introduce complexities due to variations in recording conditions, data quality, and emotional expressions. Consequently, there is a need to improve the classification performance of SER techniques. To address this, a novel SER framework was introduced in this study. Prior to feature extraction, signal preprocessing and data augmentation methods were applied to augment the available data, resulting in the derivation of 18 informative features from each signal. The discriminative feature set was obtained using feature selection techniques which was then utilized as input for emotion recognition using the SAVEE, RAVDESS, and EMO-DB datasets. Furthermore, this research also implemented a cross-corpus model that incorporated all speech files related to common emotions from three datasets. The experimental outcomes demonstrated the superior performance of SER framework compared to existing frameworks in the field. Notably, the framework presented in this study achieved remarkable accuracy rates across various datasets. Specifically, the proposed model obtained an accuracy of 95%, 94%,97%, and 97% on SAVEE, RAVDESS, EMO-DB and cross-corpus datasets respectively. These results underscore the significant contribution of our proposed framework to the field of SER.

TRNet: Two-level Refinement Network leveraging speech enhancement for noise robust speech emotion recognition

One persistent challenge in Speech Emotion Recognition (SER) is the ubiquitous environmental noise, which frequently results in deteriorating SER performance in practice. In this paper, we introduce a Two-level Refinement Network, dubbed TRNet, to address this challenge. Specifically, a pre-trained speech enhancement module is employed for front-end noise reduction and noise level estimation. Later, we utilize clean speech spectrograms and their corresponding deep representations as reference signals to refine the spectrogram distortion and representation shift of enhanced speech during model training. Experimental results validate that the proposed TRNet substantially promotes the robustness of the proposed system in both matched and unmatched noisy environments, without compromising its performance in noise-free environments.

Bridging computer and education sciences: A systematic review of automated emotion recognition in online learning environments

Emotions play an important role in the learning process. With intelligent technology support, identification and intervention of learners’ cognition have made great achievement, but the care of emotion has been in the absence for a long time. In recent years, the use of affective computing technology to solve affective loss in online education has become a key research topic. To date, a growing number of studies have investigated automated emotion recognition (AER) in online environments. However, AER has been mainly studied from the perspective of computer science focusing on technical characteristics of developing AI technology while its pedagogical value and educational application has been overlooked. Therefore, this systematic literature review aimed to bring together educational and technical aspects of AER. Following PRISMA methodology, a comprehensive search of AER research from 2010 to 2024 in three databases (Web of Science, Science Direct and IEEE Xplore) identified 117 studies that met inclusion criteria. The articles were coded for report characteristics, educational characteristics (tech platform, pedagogy, assessment, content), technical characteristics (emotion model, emotion category, emotion measurement channel, database, algorithm model) and outcome characteristics (technical result, educational application). We found that the primary purpose of these studies was to develop and evaluate systems for AER, rather than implementing these systems in real online learning environments. Furthermore, our findings indicated a lack of integration between computer science and educational science in the realm of AER. Despite the fact that most algorithm models demonstrated high accuracy in AER, the interpretability of the results was significantly constrained by the quality of the databases used, along with the scarcity of studies focusing on the effective and real-time application of AER results. These findings provide essential guidance for shaping future research and development pathways in this field.

Speech emotion recognition using the novel SwinEmoNet (Shifted Window Transformer Emotion Network)

Speech emotion recognition using the novel SwinEmoNet (Shifted Window Transformer Emotion Network)

Feature-Enhanced Multi-Task Learning for Speech Emotion Recognition Using Decision Trees and LSTM

Speech emotion recognition (SER) plays an important role in human-computer interaction (HCI) technology and has a wide range of application scenarios in medical medicine, psychotherapy, and other applications. In recent years, with the development of deep learning, many researchers have combined feature extraction technology with deep learning technology to extract more discriminative emotional information. However, a single speech emotion classification task makes it difficult to effectively utilize feature information, resulting in feature redundancy. Therefore, this paper uses speech feature enhancement (SFE) as an auxiliary task to provide additional information for the SER task. This paper combines Long Short-Term Memory Networks (LSTM) with soft decision trees and proposes a multi-task learning framework based on a decision tree structure. Specifically, it trains the LSTM network by computing the distances of features at different leaf nodes in the soft decision tree, thereby achieving enhanced speech feature representation. The results show that the algorithm achieves 85.6% accuracy on the EMO-DB dataset and 81.3% accuracy on the CASIA dataset. This represents an improvement of 11.8% over the baseline on the EMO-DB dataset and 14.9% on the CASIA dataset, proving the effectiveness of the method. Additionally, we conducted cross-database experiments, real-time performance analysis, and noise environment analysis to validate the robustness and practicality of our method. The additional analyses further demonstrate that our approach performs reliably across different databases, maintains real-time processing capabilities, and is robust to noisy environments.

Emotion recognition in user‐generated videos with long‐range correlation‐aware network

AbstractEmotion recognition in user‐generated videos plays an essential role in affective computing. In general, visual information directly affects human emotions, so the visual modality is significant for emotion recognition. Most classic approaches mainly focus on local temporal information of videos, which potentially restricts their capacity to encode the correlation of long‐range context. To address this issue, a novel network is proposed to recognize emotions in videos. To be specific, a spatio‐temporal correlation‐aware block is designed to depict the long‐range correlations between input tokens, where the convolutional layers are used to learn the local correlations and the inter‐image cross‐attention is designed to learn the long‐range and spatio‐temporal correlations between input tokens. To generate diverse and challenging samples, a dual‐augmentation fusion layer is devised, which fuses each frame with its corresponding frame in the temporal domain. To produce rich video clips, a long‐range sampling layer is designed, which generates clips in a wide range of spatial and temporal domains. Extensive experiments are conducted on two challenging video emotion datasets, namely VideoEmotion‐8 and Ekman‐6. The experimental results demonstrate that the proposed method obtains better performance than baseline methods. Moreover, the proposed method achieves state‐of‐the‐art results on the two datasets. The source code of the proposed network is available at: https://github.com/JinChow/LRCANet.

Enhanced speech emotion recognition using averaged valence arousal dominance mapping and deep neural networks

Enhanced speech emotion recognition using averaged valence arousal dominance mapping and deep neural networks

Speech emotion recognition based on multi-feature speed rate and LSTM

Correctly recognizing speech emotions is of significant importance in various fields, such as healthcare and human–computer interaction (HCI). However, the complexity of speech signal features poses challenges for speech emotion recognition. This study introduces a novel multi-feature method for speech emotion recognition that combines short-and rhythmic features. Utilizing short-time energy, zero-crossing rate, and average amplitude difference, the proposed approach effectively addressed overfitting concerns by reducing feature dimensionality. Employing an (LSTM) network, the experiment achieved notable accuracy across diverse datasets. Specifically, the proposed method achieved an impressive accuracy of up to 98.47% on the CASIA dataset, 100% on the Emo-DB dataset, and 98.87% on the EMOVO dataset, demonstrating its capability to accurately discern speaker emotions across different languages and emotion classes. These findings underscore the significance of incorporating speech rate for emotional content recognition, which holds promise for application in HCI and auxiliary medical diagnostics.

Exploring the potential of Wav2vec 2.0 for speech emotion recognition using classifier combination and attention-based feature fusion

Exploring the potential of Wav2vec 2.0 for speech emotion recognition using classifier combination and attention-based feature fusion

Harnessing AI and NLP Tools for Innovating Brand Name Generation and Evaluation: A Comprehensive Review

The traditional approach of single-word brand names faces constraints due to trademarks, prompting a shift towards fusing two or more words to craft unique and memorable brands, exemplified by brands such as SalesForce© or SnapChat©. Furthermore, brands such as Kodak©, Xerox©, Google©, Häagen-Dazs©, and Twitter© have become everyday names although they are not real words, underscoring the importance of brandability in the naming process. However, manual evaluation of the vast number of possible combinations poses challenges. Artificial intelligence (AI), particularly natural language processing (NLP), is emerging as a promising solution to address this complexity. Existing online brand name generators often lack the sophistication to comprehensively analyze meaning, sentiment, and semantics, creating an opportunity for AI-driven models to fill this void. In this context, the present document reviews AI, NLP, and text-to-speech tools that might be useful in innovating the brand name generation and evaluation process. A systematic search on Google Scholar, IEEE Xplore, and ScienceDirect was conducted to identify works that could assist in generating and evaluating brand names. This review explores techniques and datasets used to train AI models as well as strategies for leveraging objective data to validate the brandability of generated names. Emotional and semantic aspects of brand names, which are often overlooked in traditional approaches, are discussed as well. A list with more than 75 pivotal datasets is presented. As a result, this review provides an understanding of the potential applications of AI, NLP, and affective computing in brand name generation and evaluation, offering valuable insights for entrepreneurs and researchers alike.

A novel solution for the development of a sentimental analysis chatbot integrating ChatGPT

AbstractIn today’s business landscape, Chatbots play a pivotal role in innovation and process optimization. In this paper, we introduced a novel advanced Emotional Chatbot AI, introducing sentiment analysis for human chatbot conversations. Adding an emotional component within the human-computer interaction, can in fact dramatically improve the quality of the final conversation between Chatbots and humans. More specifically, in our paper, we provided a practical evaluation of the EmoROBERTA software, introducing it into a novel implementation of an Emotional Chatbot. The pipeline we present is novel, and we developed it within a business context in which the use of sentimental and emotional responses can act in a significant and fundamental way toward the final success and use of the Chatbot itself. The architecture enriches user experience with real-time updates on the topic of interest, maintaining a user-centric design, toward an affective-response enhancement of the interaction established between the Chatbot and the user. The source code is fully available on GitHub: https://github.com/filippoflorindi/F-One.

AI-Driven UX/UI Design: Empirical Research and Applications in FinTech

This study explores the transformative impact of AI-driven UX/UI design in the FinTech sector, examining current practices, user preferences, and emerging trends. Through a mixed-methods approach, including surveys, interviews, and case studies, the research reveals significant adoption of AI technologies in FinTech UX/UI design, with 78% of surveyed companies implementing such solutions. Personalization emerges as a dominant trend, with 76% of FinTech apps utilizing AI for tailored user interfaces. The study demonstrates a strong correlation between AI-enhanced features and improved user engagement, with apps incorporating advanced AI features showing a 41% increase in daily active users. Ethical considerations, including data privacy and algorithmic bias, are addressed as critical challenges in AI implementation. The research contributes a conceptual framework for AI-driven UX/UI design in FinTech, synthesizing findings from diverse data sources. Future trends, including emotional AI and augmented reality integration, are explored. The study concludes that while AI-driven UX/UI design offers significant potential for enhancing user experiences in FinTech, balancing innovation with ethical considerations is crucial for responsible implementation and user trust.

Decoding Emotions: Machine Learning Approach to Speech Emotion Recognition

Speech Emotion Recognition (SER) stands at the forefront of human-computer interaction, offering profound implications for fields such as healthcare, education, and entertainment. This project report delves into the application of Machine Learning (ML) techniques for SER, aiming to discern the emotional content from speech signals. The report begins with an overview of the significance of SER in various domains, emphasizing the need for accurate and robust emotion detection systems. Following this,a detailed exploration of the methodologies employed in SER is presented, encompassing feature extraction techniques, classification algorithms, and model evaluation metrics. In the implementation phase, diverse ML algorithms such as Support Vector Machine (SVM), Convolutional Neural Networks (CNN), and Recurrent Neural Networks (RNN) are employed to classify emotional states from speech data. Feature sets including prosodic features, spectral features, and deep learning-based representations are extracted from the speech signals to capture nuanced emotional cues.

Emotion Recognition in Lhasa Tibetan Speech based on Bi-LSTM Graph Convolutional Networks

Speech Emotion Recognition (SER) is a crucial component in the field of Human-Computer Interaction (HCI), with significant research and practical application implications. However, due to the complexity of the Tibetan language and the scarcity of datasets caused by the difficulty in collecting various dialects, there are not many research achievements in Tibetan speech recognition. Based on the foundation of constructing a TBLS1 dataset containing 6,000 Tibetan-language speech samples, an approach was devised for Tibetan speech emotion recognition. This approach leverages MFCC features and incorporates a Bi-directional Long Short-Term Memory (Bi-LSTM) network within a graph convolutional neural network. Finally, by comparing the performance of different models on this dataset, we demonstrated the feasibility of our model for Tibetan speech emotion recognition.

EEG Emotion Recognition Employing RGPCN-BiGRUAM: ReliefF-Based Graph Pooling Convolutional Network and BiGRU Attention Mechanism

Emotion recognition plays a crucial role in affective computing, and electroencephalography (EEG) signals are increasingly applied in this field due to their effectiveness in reflecting brain activity. In this paper, we propose a novel EEG emotion recognition model that combines the ReliefF-based Graph Pooling Convolutional Network and BiGRU Attention Mechanisms (RGPCN-BiGRUAM). RGPCN-BiGRUAM effectively integrates the advantages of graph convolutional networks and recurrent neural networks. By incorporating ReliefF weights and an attention mechanism into graph pooling, our model enhances the aggregation of high-quality features while discarding irrelevant ones, thereby improving the efficiency of information transmission. The implementation of a multi-head attention mechanism fusion in BiGRU addresses the limitations of single-output features, achieving optimal selection of global features. Comparative experiments on public datasets SEED and DEAP demonstrate that our proposed RGPCN-BiGRUAM significantly improves classification performance compared to classic algorithms, achieving state-of-the-art results. Ablation studies further validate the design principles of our model. The results of this study indicate that RGPCN-BiGRUAM has strong potential for EEG emotion recognition, offering substantial possibilities for future applications.

Multi-level attention fusion network assisted by relative entropy alignment for multimodal speech emotion recognition

Multi-level attention fusion network assisted by relative entropy alignment for multimodal speech emotion recognition

Speech Emotion Recognition Incorporating Relative Difficulty and Labeling Reliability.

Emotions in speech are expressed in various ways, and the speech emotion recognition (SER) model may perform poorly on unseen corpora that contain different emotional factors from those expressed in training databases. To construct an SER model robust to unseen corpora, regularization approaches or metric losses have been studied. In this paper, we propose an SER method that incorporates relative difficulty and labeling reliability of each training sample. Inspired by the Proxy-Anchor loss, we propose a novel loss function which gives higher gradients to the samples for which the emotion labels are more difficult to estimate among those in the given minibatch. Since the annotators may label the emotion based on the emotional expression which resides in the conversational context or other modality but is not apparent in the given speech utterance, some of the emotional labels may not be reliable and these unreliable labels may affect the proposed loss function more severely. In this regard, we propose to apply label smoothing for the samples misclassified by a pre-trained SER model. Experimental results showed that the performance of the SER on unseen corpora was improved by adopting the proposed loss function with label smoothing on the misclassified data.