Emotion Classification Method Research Articles

A 4D spontaneous micro‐expression database: Establishment and evaluation

AbstractMicro‐expressions are spontaneous and unconscious facial movements that reveal individuals’ genuine inner emotions. They hold significant potential in various psychological testing fields. As the face is a 3D deformation object, the emergence of facial expression leads to spatial deformation of the face. However, existing databases primarily offer 2D video sequences, limiting descriptions of 3D spatial information related to micro‐expressions. Here, a new micro‐expression database is proposed, which contains 2D image sequences and corresponding 3D point cloud sequences. These samples were classified using both an objective method based on the facial action coding system and a non‐objective emotion classification method that considers video contents and participants’ self‐reports. A variety of feature extraction techniques are applied to 2D data, including traditional algorithms and deep learning methods. Additionally, a novel local curvature‐based algorithm is developed to extract 3D spatio‐temporal deformation features from the 3D data. The authors evaluated the classification accuracies of these two features individually and their fusion results under leave‐one‐subject‐out (LOSO) and tenfold cross‐validation. The results demonstrate that fusing 3D features with 2D features results in improved recognition performance compared to using 2D features alone.

Research progress on emotion recognition by combining virtual reality environment and electroencephalogram signals

Emotion recognition refers to the process of determining and identifying an individual's current emotional state by analyzing various signals such as voice, facial expressions, and physiological indicators etc. Using electroencephalogram (EEG) signals and virtual reality (VR) technology for emotion recognition research helps to better understand human emotional changes, enabling applications in areas such as psychological therapy, education, and training to enhance people's quality of life. However, there is a lack of comprehensive review literature summarizing the combined researches of EEG signals and VR environments for emotion recognition. Therefore, this paper summarizes and synthesizes relevant research from the past five years. Firstly, it introduces the relevant theories of VR and EEG signal emotion recognition. Secondly, it focuses on the analysis of emotion induction, feature extraction, and classification methods in emotion recognition using EEG signals within VR environments. The article concludes by summarizing the research's application directions and providing an outlook on future development trends, aiming to serve as a reference for researchers in related fields.

Research on Image Perception of Tourist Destinations Based on the BERT-BiLSTM-CNN-Attention Model

Image perception of tourism destinations plays an important role in destination marketing and management. Considering that long-distance feature information of travel review text is difficult to capture and local key information is ignored, BiLSTM and CNN are improved to propose a travel text classification method based on BERT-BiLSTM-CNN-Attention hybrid neural network model. Taking Sanya City as the research object, we adopt the emotion classification and content analysis methods and construct the tourism destination image perception analysis framework based on the “cognitive-emotional” three-dimensional model, providing a research perspective for the sustainable development of tourism in Sanya City. The results show that the accuracy of the proposed model reaches 93.18%, which is better than other models. Tourists’ perception of destination image includes four aspects: tourism resources, tourism environment perception, tourism infrastructure and supporting services, as well as tourism activities. Positive emotions dominate emotional image, and negative emotions are mainly focused on tourism infrastructure and supporting services. On the overall image perception, tourists have a high evaluation of the tourism image of Sanya City. This research has some implications for tourism destinations, such as improving their management programs, enhancing their marketing strategies, and achieving long-term sustainable development of their destinations.

Improvements in Similarity Measurement Method for Emotion Classification of Japanese Sentences

This study proposes the following three improvements to address the shortfalls of existing methods for emotion classification based on similarity measures between Japanese sentences. (1) The proposed method utilizes, as a similarity measure, an automatic evaluation metric for machine translation based on correlation considering global word order. Similarity measures used in existing methods are too strict or lax in their consideration of the order of words during the processing of Japanese sentences. They fail to incorporate the fact that Japanese exhibits a higher degree of freedom in word order compared to English. (2) Rather than word-level alignment, the proposed method aligns two sentences at the level of a base phrase constituted by one or more content words followed by zero or more function words. Word-level alignment, used by existing methods, may cause inappropriate matching, because it independently processes content words and function words which constitute a base phrase. (3) The proposed method integrates, using linear interpolation, similarity scores by the alignment of word-level and those of base phrase-level. Integration can avoid data sparsity, namely a problem that a similarity score between two sentences becomes zero, occurring when base phrase-level alignment is made. The experimental results found that (1) the proposed method demonstrated significantly higher classification accuracy than existing alternatives; (2) the base phrase-level alignment was effective because the alignment exhibited higher classification accuracy than the word-level alignment; and (3) integration of the word-level and base phrase-level alignments was valid because the integrated method outperformed the base phrase-level alignment. Key Words: Emotion classification, Similarity measure, Japanese, Word order, Base phrase, Linear interpolation

A Modified MFCC-Based Deep Learning Method for Emotion Classification from Speech

Speech, which is one of the most effective methods of communication, varies according to the emotions experienced by people and includes not only vocabulary but also information about emotions. With developing technologies, human-machine interaction is also improving. Emotional information to be extracted from voice signals is valuable for this interaction. For these reasons, studies on emotion recognition systems are increasing. In this study, sentiment analysis is performed using the Toronto Emotional Speech Set (TESS) created by University of Toronto. The voice data in the dataset is first preprocessed and then a new CNN-based deep learning method on it is compared. The voice files in the TESS dataset have been first obtained feature maps using the MFCC method, and then classification has been performed with this method based on the proposed neural network model. Separate models have been created with CNN and LSTM models for the classification process. The experiments show that the MFCC-applied CNN model achieves a better result with an accuracy of 99.5% than the existing methods for the classification of voice signals. The accuracy value of the CNN model shows that the proposed CNN model can be used for emotion classification from human voice data.

Channel Semantic Enhancement-Based Emotional Recognition Method Using SCLE-2D-CNN

The existing EEG emotion classification methods have some problems, such as insufficient emotion representation and lack of targeted channel enhancement module due to feature redundancy. To this end, a novel EEG emotion recognition method (SCLE-2D-CNN) combining scaled convolutional layer (SCLs), enhanced channel module and two-dimensional convolutional neural network (2D-CNN) is proposed. Firstly, the time-frequency features of multi-channel EEG emotional signals were extracted by stacking scl layer by layer. Secondly, channel enhancement module is used to reassign different importance to all EEG physical channels. Finally, 2D-CNN was used to obtain deep local spatiotemporal features and complete emotion classification. The experimental results show that the accuracy of SEED data set and F1 are 98.09% and 97.00%, respectively, and the binary classification accuracy of DEAP data set is 98.06% and 96.83%, respectively, which are superior to other comparison methods. The proposed method has a certain application prospect in the recognition of human mental state.

A Parallel Two-Channel Emotion Classification Method for Chinese Text

A Parallel Two-Channel Emotion Classification Method for Chinese Text

Emotion Analysis Using Improved Cat Swarm Optimization with Machine Learning for Speech-impaired People

Emotion analysis is an effective method for improving interaction and understanding for speech-impaired people. We can provide the best interaction and support emotional well-being by analyzing the emotion conveyed through speech. Using deep learning (DL) or machine learning algorithms for training an emotion classification method. This might include training classifiers namely random forests, deep neural networks, or support vector machines. It is noteworthy that emotion analysis could be effective; however, it is crucial to consider individual differences and context while interpreting emotion. Furthermore, ensuring data protection and privacy and obtaining consent are vital features to consider while working with sensitive speech data. Therefore, this study presents an emotion analysis approach using improved cat swarm optimization with machine learning (EA-ICSOML) technique. The EA-ICSOML technique applies the concepts of computer vision and DL to identify various types of emotions. For feature vector generation, the ShuffleNet model is used in this work. To adjust the hyperparameters compared to the ShuffleNet system, the ICSO algorithm is used. Finally, the recognition and classification of emotions are performed using the Transient Chaotic Neural Network approach. The performance validation of the EA-ICSOML technique is validated on facial emotion databases. The simulation result inferred the improved emotion recognition results of the EA-ICSOML approach compared to other recent models in terms of different evaluation measures.

A Fine-Grained Approach for EEG-Based Emotion Recognition Using Clustering and Hybrid Deep Neural Networks

Emotion recognition, as an important part of human-computer interaction, is of great research significance and has already played a role in the fields of artificial intelligence, healthcare, and distance education. In recent times, there has been a growing trend in using deep learning techniques for EEG emotion recognition. These methods have shown higher accuracy in recognizing emotions when compared with traditional machine learning methods. However, most of the current EEG emotion recognition performs multi-category single-label prediction, and is a binary classification problem based on the dimensional model. This simplifies the fact that human emotions are mixed and complex. In order to adapt to real-world applications, fine-grained emotion recognition is necessary. We propose a new method for building emotion classification labels using linguistic resource and density-based spatial clustering of applications with noise (DBSCAN). Additionally, we integrate the frequency domain and spatial features of emotional EEG signals and feed these features into a serial network that combines a convolutional neural network (CNN) and a long short-term memory (LSTM) recurrent neural network (RNN) for EEG emotion feature learning and classification. We conduct emotion classification experiments on the DEAP dataset, and the results show that our method has an average emotion classification accuracy of 92.98% per subject, validating the effectiveness of the improvements we have made to our emotion classification method. Our method for emotion classification holds potential for future use in the domain of affective computing, such as mental health care, education, social media, and so on. By constructing an automatic emotion analysis system using our method to enable the machine to understand the emotional implications conveyed by the subjects’ EEG signals, it can provide healthcare professionals with valuable information for effective treatment outcomes.

Image emotion multi-label classification based on multi-graph learning

Images contain rich information and can induce various emotions in the audience. Image emotion classification aims to identify the emotion categories that images can evoke. It is widely used in mental health assessment, human–computer interaction, etc. There are two main problems in existing image emotion classification methods: (1) Most of them only focus on a single emotion label; (2) The global structural relationship among semantic objects in the image is ignored. Therefore, this paper proposes an Image Emotion classification method based on Multi-Graph Multi-Label learning (IE-MGML). In contrast to the existing approaches, the image is transformed into a graph-based representation by extracting the emotional features of semantic objects and calculating the similarity between the features. The local (semantic objects) features and global structure (relationship among semantic objects) features of the image are fused by the relationship between nodes. Furthermore, the graph representation of an image from the perspective of multiple emotional features is pooled and modeled as a graph bag containing multiple graphs (i.e., multi-graph). In multi-graph learning, the graph kernel directly evaluates a graph-label dependency score to avoid the loss of structural information caused by graph-instance degradation. The bag(image)-label dependency score is obtained by aggregating the graph-label dependency score from different perspectives through the aggregation function. The problem of error accumulation in the learning process is handled by proposing a threshold-based ranking loss objective function. Moreover, the non-convex optimization problem is addressed using a subgradient descent algorithm to deal with the required high-dimensional space computation. Experimental results on three general image emotion datasets show that the proposed method outperforms the state-of-the-art methods.

Enhancing EEG-based emotion recognition using PSD-Grouped Deep Echo State Network

Emotions are a crucial aspect of daily life and play a vital role in shaping human inter-actions. The purpose of this paper is to introduce a novel approach to recognize human emotions through the use of electroencephalogram (EEG) signals. To recognize these signals for emotion prediction, we employ a paradigm of Reservoir Computing (RC), called Echo State Network (ESN). In our analysis, we focus on two specific classes of emotion recognition: H/L Arousal and H/L Valence. We suggest using the Deep ESN model in conjunction with the Welch Power Spectral Density (Wlech PSD) method for emotion classification and feature extraction. Furthermore, we feed the selected features to a grouped ESN for recognizing emotions. Our approach is validated on the well-known DEAP benchmark, which includes the EEG data from 32 participants. The proposed model achieved 89.32% accuracy for H/L Arousal and 91.21% accuracy for H/L Valence on the DEAP dataset. The obtained results demonstrate the effectiveness of our approach, which yields good performance compared to existing models of emotion analysis based on EEG.

Analyze IMDb movies by sentiment and topic analysis

Movie is an important cultural form, carrying multiple levels and meanings such as art, entertainment and social value. Movie review and rating data sets are huge, and deep learning and natural language processing methods are widely used today. Advances in big data and deep learning offer unprecedented opportunities to understand moviegoer behavior and preferences while providing a cost-effective way to gain insights relevant to the entertainment industry. This project conducts sentiment analysis, topic modeling, and visual statistical analysis based on the IMDb movie data set to identify key factors and deeper insights that influence successful decision-making in film production. This project first uses the word embedding method to vectorize the movie review text, and then uses Bidirectional Long Short-Term Memory (Bi-LSTM) to perform sentiment classification. In addition, statistical methods such as visualization were used to discover conclusions such as the highest average number of movies released in November, and identify trends, patterns and relationships between the variables of IMDb movies. Finally, the Latent Dirichlet Allocation (LDA) topic modeling model was constructed to find out that the important topic with increased demand is light entertainment movies, highlighting the commercial feasibility of comedy movies as a profitable business model. In summary, this project uses an emotion-topic fusion analysis method based on the Bi-LSTM emotion classification method and the LDA topic modeling method. The results show that the Bi-LSTM model can better identify positive and negative emotions in movie reviews, and the LDA topic model performs well in mining popular topics.

Simple But Powerful, a Language-Supervised Method for Image Emotion Classification

Image emotion classification is an important computer vision task to extract emotions from images. The methods for image emotion classification (IEC) are primarily based on label or distribution as a supervision signal, which neither has enough accessibility nor diversity, limiting the development of IEC research. Inspired by psychology research and the recent booming of large-scale pretrained language models. We figure out a language-supervised paradigm, which can cleverly combine the features of language and visual emotion to drive the visual model to gain stronger emotional discernment with language prompts. To practice the paradigm, we present a conceptually simple while empirically powerful framework for image emotion classification, SimEmotion. That we propose a prompt-based fine-tuning strategy to learn task-specific representations by composing a template with the emotion-level concept and entity-level information. Evaluations on four widely-used affective datasets, namely, Flickr and Instagram (FI), EmotionROI, Twitter I, and Twitter II, demonstrate that the proposed algorithm outperforms the state-of-the-art methods with a large margin ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , 8.42% absolute accuracy gain on EmotionROI) on image emotion classification tasks. Our codes will be publicly available for research purposes.

Evading text based emotion detection mechanism via adversarial attacks

Textual Emotion Analysis (TEA) seeks to extract and assess the emotional states of users from the text. Various Deep Learning (DL) algorithms have emerged rapidly and demonstrated success in numerous disciplines, including audio, image, and natural language processing. Thetrend has shifted a growing number of researchers from standard machine learning to DL for scientific study. Using DL approaches, we offer an overview of TEA in this paper. After introducing the background for emotion analysis, including the definition of emotion, emotion classification methods, and application domains of emotion analysis, we demonstrated that, despite the immense success of deep learning models in NLP-related tasks, they are susceptible to adversarial attacks, which can lead to incorrect emotion classification. An adversarial text is constructed by altering a few words or characters so as to keep the overall semantic similarity of emotion for a human reader while tricking the machine into making erroneous predictions. This study demonstrates the vulnerability of emotion categorization by generating adversarial text using a variety of cutting-edge attack techniques. Comprehensive experiments are performed to assess the effectiveness of the attack methods against several widely-used models, such as Word-CNN, Bi-LSTM, and four powerful transformer models, namely BERT, DistilBERT, ALBERT, and RoBERTa. These models were trained on an emotion dataset utilized for the purpose of emotion classification. We evaluated and analyzed the behavior of different models under a variety of attack conditions to determine which is the most and least vulnerable. Also, we determine which perturbation technique affects transformer models the most. Using Attack Success Rates (ASR) as our evaluation metric, we have assessed the potential outcomes. The findings reveal that methodologies for classifying emotion predictioncan be circumvented, which has implications for existing policy measures.

Retracted: Music Emotion Classification Method Based on Deep Learning and Explicit Sparse Attention Network.

[This retracts the article DOI: 10.1155/2022/3920663.].

An artificial intelligence-based classifier for musical emotion expression in media education.

Music can serve as a potent tool for conveying emotions and regulating learners' moods, while the systematic application of emotional assessment can help to improve teaching efficiency. However, existing music emotion analysis methods based on Artificial Intelligence (AI) rely primarily on pre-marked content, such as lyrics and fail to adequately account for music signals' perception, transmission, and recognition. To address this limitation, this study first employs sound-level segmentation, data frame processing, and threshold determination to enable intelligent segmentation and recognition of notes. Next, based on the extracted audio features, a Radial Basis Function (RBF) model is utilized to construct a music emotion classifier. Finally, correlation feedback was used to label the classification results further and train the classifier. The study compares the music emotion classification method commonly used in Chinese music education with the Hevner emotion model. It identifies four emotion categories: Quiet, Happy, Sad, and Excited, to classify performers' emotions. The testing results demonstrate that audio feature recognition time is a mere 0.004 min, with an accuracy rate of over 95%. Furthermore, classifying performers' emotions based on audio features is consistent with conventional human cognition.

Fusion-based speech emotion classification using two-stage feature selection

Speech emotion recognition plays an important role in human–computer interaction, which uses speech signals to determine the emotional state. Previous studies have proposed various features and feature selection methods. However, few studies have investigated the two-stage feature selection method for speech emotion classification. In this study, we propose a novel speech emotion classification algorithm based on two-stage feature selection and two fusion strategies. Specifically, three types of features are extracted from speech signals: constant-Q spectrogram-based histogram of oriented gradients, openSMILE, and wavelet packet decomposition-based features. Then, two-stage feature selection using random forest and grey wolf optimization is applied to reduce feature dimension and model training time and improve the classification performance. In addition, both early and late fusion strategies are explored aiming to further improve the performance. Experimental results indicate that early fusion with two-stage feature selection can achieve the best performance. The highest classification accuracy for RAVDESS, SAVEE, EMOVO, and EmoDB is 86.97%, 88.79%, 89.24%, and 95.29%, respectively.

From gram to attention matrices: a monotonicity constrained method for eeg-based emotion classification

From gram to attention matrices: a monotonicity constrained method for eeg-based emotion classification

Research on emotion classification technology of movie reviews based on topic attention mechanism and dual channel long short term memory.

In recent years, with the popularity of the Internet, more and more people like to comment on movies they have watched on the film platform after watching them. These reviews hide the reviewers' feedback on films. Mining the emotional orientation information in these reviews can provide consumers with shopping references and help businesses optimize film works and improve business strategies. Therefore, the emotional classification of film reviews has high research value because few emotion dictionaries and analysis tools are available for reference and use in film reviews. The accuracy of emotion classification still needs to be improved. This study introduces the attention mechanism and dual channel long short term memory (DC-LSTM) while building the emotion dictionary in the field of Chinese film review. It classifies Chinese film reviews in terms of topic-based fine-grained emotion. First, the emotion vector is constructed using the constructed movie review emotion lexicon. The semantic vector obtained by the Word2vector tool is input to LSTM to encode the comment text. Then, the topic attention module is used to decode. Finally, the final emotion classification result is obtained through the softmax function of the entire link layer and the output layer. The thematic attention modules constructed in this study are independent of each other for attention parameter adjustment and learning. One attention module corresponds to one film theme. In this study, eight themes, including "plot," "special effects," "original work," "music," "thought," "theme," "acting skills," and "joke," were extracted, and each theme was classified into three types of emotions: "positive," "neutral," and "negative." The experimental results on the crawled Chinese film review dataset show that the proposed algorithm is superior to some existing algorithms and models in accuracy, precision, recall and F1 measure. The DCLSTM based on the thematic attention mechanism (DCLSTM-TAM) model constructed in this study introduces the emotion vector into the network and adds the theme attention mechanism. It can not only classify the emotion for different topics of a film review but also effectively deal with film reviews with fuzzy emotional tendencies. It realizes the fine-grained emotion classification of film topics and improves the accuracy of emotion classification of film reviews. The emotion classification method and model proposed in this study have good transferability, and the change of training corpus is also applicable to other short text fields.

Emotion Classification from Multi-Band Electroencephalogram Data Using Dynamic Simplifying Graph Convolutional Network and Channel Style Recalibration Module.

Because of its ability to objectively reflect people's emotional states, electroencephalogram (EEG) has been attracting increasing research attention for emotion classification. The classification method based on spatial-domain analysis is one of the research hotspots. However, most previous studies ignored the complementarity of information between different frequency bands, and the information in a single frequency band is not fully mined, which increases the computational time and the difficulty of improving classification accuracy. To address the above problems, this study proposes an emotion classification method based on dynamic simplifying graph convolutional (SGC) networks and a style recalibration module (SRM) for channels, termed SGC-SRM, with multi-band EEG data as input. Specifically, first, the graph structure is constructed using the differential entropy characteristics of each sub-band and the internal relationship between different channels is dynamically learned through SGC networks. Second, a convolution layer based on the SRM is introduced to recalibrate channel features to extract more emotion-related features. Third, the extracted sub-band features are fused at the feature level and classified. In addition, to reduce the redundant information between EEG channels and the computational time, (1) we adopt only 12 channels that are suitable for emotion classification to optimize the recognition algorithm, which can save approximately 90.5% of the time cost compared with using all channels; (2) we adopt information in the θ, α, β, and γ bands, consequently saving 23.3% of the time consumed compared with that in the full bands while maintaining almost the same level of classification accuracy. Finally, a subject-independent experiment is conducted on the public SEED dataset using the leave-one-subject-out cross-validation strategy. According to experimental results, SGC-SRM improves classification accuracy by 5.51-15.43% compared with existing methods.