Emotion Recognition Performance Research Articles

Age differences in mental state inference of sarcasm: Contributions of facial emotion recognition and cognitive performance.

Older age is associated with poorer ability to accurately infer mental states, but some mental states are more complex than others. Sarcasm is a complex mental state because the literal and intended meaning of a speaker's words are in opposition. Individuals must rely on additional cues (e.g., facial expressions, intonation) for accurate inference. We hypothesized that understanding of sarcastic versus sincere exchanges would be more sensitive to age-related difficulty in mental state understanding. We examined accuracy at identifying sarcasm among 263 adults (ages 18-90 years) using videos of social interactions in The Awareness of Social Inference Test. Hypotheses were tested using a logistic linear mixed effects model predicting correct/incorrect trial-level responses. To characterize why sarcasm differed with age, we measured two abilities commonly implicated in mental state understanding: facial emotion recognition and cognitive performance. Sarcasm understanding declined with age, whereas understanding of sincere exchanges did not. Both better emotion recognition and cognitive performance related to better understanding of sarcastic but not sincere exchanges. Only cognitive performance showed an age-related effect such that the cognitive performance among the oldest participants facilitated their understanding of both sarcastic and sincere exchanges. We showed that individual variation related to age and social and cognitive performance is more pronounced when the use of multiple mental state cues is more (sarcasm) or less (sincerity) necessary for accurate understanding of social interactions. Naturalistic paradigms involving multiple mental state cues can address important questions about how older adults make decisions in the real world.

Effects of a social network enhancement intervention for older adults: a feasibility study

BackgroundSocial networks play a critical role in the mental health of older adults. This pilot study investigates the feasibility of a newly developed intervention to enhance older adults’ social networks. This intervention was designed on the Theory of Mind’s foundation and aimed to enhance older adults’ social interaction motivation through theoretical explanations. Furthermore, the courses fostered more social opportunities for the participants through group-based sessions.MethodsThe feasibility of this intervention was tested using a double-blind, two-arm, non-randomized grouping approach. Older individuals residing in two separate residential buildings (n = 31, mean age = 66.81, 48% women) were divided into an intervention group (n = 15) and a control group (n = 16). They attended daily group sessions at a designated location and completed homework assignments. The primary outcomes of this pilot study were the feasibility of the intervention, and secondary outcomes included Theory of Mind levels and social network indicators. Additional outcomes encompassed levels of global mental health and depression.ResultsAll participants completed the pilot intervention and completed assessments. The primary outcomes indicated that the intervention had excellent feasibility, including compliance (attendance and homework completion rates met the standards) and satisfaction (average ratings for all items ranged from 4.47 to 5.00 on a 5-point scale). Interview results revealed that participants in the intervention group found the intervention beneficial for their daily lives and expressed a desire to participate in a formal intervention. Regarding secondary and additional outcomes, compared to the control group, the intervention group exhibited a significant improvement in emotional recognition performance of Theory of Mind. There was a significant increase in the whole network density in the intervention group. There were no significant differences in other social network indicators, global mental health, and depression levels in the intervention group compared to the control group.ConclusionsThe social network enhancement intervention for older adults is feasible. This pilot study has identified several improvements in the courses and tests. It is necessary to carry out a formal course to examine the effectiveness of the intervention on social networks in older adults.Trial registrationRegistration number: ChiCTR2100053779; Reg Date: 29/11/2021.

사전학습 모델 기반 발화 동영상 멀티 모달 감정 인식

Recently, as the demand for non-face-to-face counseling has rapidly increased, the need for emotion recognition technology that combines various aspects such as text, voice, and facial expressions is being emphasized. In this paper, we address issues such as the dominance of non-Korean data and the imbalance of emotion labels in existing datasets like FER-2013, CK+, and AFEW by using Korean video data. We propose methods to enhance multimodal emotion recognition performance in videos by integrating the strengths of image modality with text modality. A pre-trained model is used to overcome the limitations caused by small training data. A GPT-4-based LLM model is applied to text, and a pre-trained model based on VGG-19 architecture is fine-tuned to facial expression images. The method of extracting representative emotions by combining the emotional results of each aspect extracted using a pre-trained model is as follows. Emotion information extracted from text was combined with facial expression changes in a video. If there was a sentiment mismatch between the text and the image, we applied a threshold that prioritized the text-based sentiment if it was deemed trustworthy. Additionally, as a result of adjusting representative emotions using emotion distribution information for each frame, performance was improved by 19% based on F1-Score compared to the existing method that used average emotion values for each frame.

Multi-source domain adaptation for EEG emotion recognition based on inter-domain sample hybridization.

Electroencephalogram (EEG) is widely used in emotion recognition due to its precision and reliability. However, the nonstationarity of EEG signals causes significant differences between individuals or sessions, making it challenging to construct a robust model. Recently, domain adaptation (DA) methods have shown excellent results in cross-subject EEG emotion recognition by aligning marginal distributions. Nevertheless, these methods do not consider emotion category labels, which can lead to label confusion during alignment. Our study aims to alleviate this problem by promoting conditional distribution alignment during domain adaptation to improve cross-subject and cross-session emotion recognition performance. This study introduces a multi-source domain adaptation common-branch network for EEG emotion recognition and proposes a novel sample hybridization method. This method enables the introduction of target domain data information by directionally hybridizing source and target domain samples without increasing the overall sample size, thereby enhancing the effectiveness of conditional distribution alignment in domain adaptation. Cross-subject and cross-session experiments were conducted on two publicly available datasets, SEED and SEED-IV, to validate the proposed model. In cross-subject emotion recognition, our method achieved an average accuracy of 90.27% on the SEED dataset, with eight out of 15 subjects attaining a recognition accuracy higher than 90%. For the SEED-IV dataset, the recognition accuracy also reached 73.21%. Additionally, in the cross-session experiment, we sequentially used two out of the three session data as source domains and the remaining session as the target domain for emotion recognition. The proposed model yielded average accuracies of 94.16 and 75.05% on the two datasets, respectively. Our proposed method aims to alleviate the difficulties of emotion recognition from the limited generalization ability of EEG features across subjects and sessions. Though adapting the multi-source domain adaptation and the sample hybridization method, the proposed method can effectively transfer the emotion-related knowledge of known subjects and achieve accurate emotion recognition on unlabeled subjects.

Multimodal Dataset Construction and Validation for Driving-Related Anger: A Wearable Physiological Conduction and Vehicle Driving Data Approach

Anger impairs a driver’s control and risk assessment abilities, heightening traffic accident risks. Constructing a multimodal dataset during driving tasks is crucial for accurate anger recognition. This study developed a multimodal physiological -vehicle driving dataset (DPV-MFD) based on drivers’ self-reported anger during simulated driving tasks. In Experiment 1, responses from 624 participants to anger-inducing videos and driving scenarios were collected via questionnaires to select appropriate materials. In Experiments 2 and 3, multimodal dynamic data and self-reported SAM emotion ratings were collected during simulated and real-vehicle tasks, capturing physiological and vehicle responses in neutral and anger states. Spearman’s correlation coefficient analysis validated the DPV-MFD’s effectiveness and explored the relationships between multimodal data and emotional dimensions. The CNN-LSTM deep learning network was used to assess the emotion recognition performance of the DPV-MFD across different time windows, and its applicability in real-world driving scenarios was validated. Compared to using EEG data alone, integrating multimodal data significantly improved anger recognition accuracy, with accuracy and F1 scores rising by 4.49% and 9.14%, respectively. Additionally, real-vehicle data closely matched simulated data, confirming the dataset’s effectiveness for real-world applications. This research is pivotal for advancing emotion-aware human–machine- interaction and intelligent transportation systems.

Introducing a novel dataset for facial emotion recognition and demonstrating significant enhancements in deep learning performance through pre-processing techniques

Facial expression recognition (FER) plays a pivotal role in various applications, ranging from human-computer interaction to psychoanalysis. To improve the accuracy of facial emotion recognition (FER) models, this study focuses on enhancing and augmenting FER datasets. It comprehensively analyzes the Facial Emotion Recognition dataset (FER13) to identify defects and correct misclassifications. The FER13 dataset represents a crucial resource for researchers developing Deep Learning (DL) models aimed at recognizing emotions based on facial features. Subsequently, this article develops a new facial dataset by expanding upon the original FER13 dataset. Similar to the FER + dataset, the expanded dataset incorporates a wider range of emotions while maintaining data accuracy. To further improve the dataset, it will be integrated with the extended Cohn-Kanade (CK+) dataset.This paper investigates the application of modern DL models to enhance emotion recognition in human faces. By training a new dataset, the study demonstrates significant performance gains compared with its counterparts. Furthermore, the article examines recent advances in FER technology and identifies critical requirements for DL models to overcome the inherent challenges of this task effectively. The study explores several DL architectures for emotion recognition in facial image datasets, with a particular focus on convolutional neural networks (CNNs). Our findings indicate that complex architecture, such as EfficientNetB7, outperforms other DL architectures, achieving a test accuracy of 78.9 %. Notably, the model surpassed the EfficientNet-XGBoost model, especially when used with the new dataset. Our approach leverages EfficientNetB7 as a backbone to build a model capable of efficiently recognizing emotions from facial images. Our proposed model, EfficientNetB7-CNN, achieved a peak accuracy of 81 % on the test set despite facing challenges such as GPU memory limitations. This demonstrates the model's robustness in handling complex facial expressions. Furthermore, to enhance feature extraction and attention mechanisms, we propose a new hybrid model, CBAM-4CNN, which integrates the convolutional block attention module (CBAM) with a custom 4-layer CNN architecture. The results showed that the CBAM-4CNN model outperformed existing models, achieving higher accuracy, precision, and recall metrics across multiple emotion classes. The results highlight the critical role of comprehensive and diverse data in enhancing model performance for facial emotion recognition.

Multi-view brain functional connectivity and hierarchical fusion for EEG-based emotion recognition

Emotional states evolve gradually from their onset to full manifestation, reflected in changes in brain functional connectivity. Existing research often focuses only on the final emotional state, missing the rich, dynamic information of emotional evolution. This paper proposes a novel network framework leveraging global and local brain functional connectivity through a multi-view approach. We introduce a global and local connectivity modeling method using electroencephalogram signals to simulate brain connectivity changes across time and specific moments. A residual-based local detail feature extractor combined with a coordinate attention module captures long-range dependencies and maintains positional accuracy. A global feature extractor is also designed for the effective extraction of emotional information. Finally, a hierarchical multi-scale cross-attention feature fusion module integrates features across different levels to enhance emotion recognition performance and robustness. Extensive experiments on three publicly available datasets demonstrate the framework’s superiority and generalization capability.

EAV: EEG-Audio-Video Dataset for Emotion Recognition in Conversational Contexts

Understanding emotional states is pivotal for the development of next-generation human-machine interfaces. Human behaviors in social interactions have resulted in psycho-physiological processes influenced by perceptual inputs. Therefore, efforts to comprehend brain functions and human behavior could potentially catalyze the development of AI models with human-like attributes. In this study, we introduce a multimodal emotion dataset comprising data from 30-channel electroencephalography (EEG), audio, and video recordings from 42 participants. Each participant engaged in a cue-based conversation scenario, eliciting five distinct emotions: neutral, anger, happiness, sadness, and calmness. Throughout the experiment, each participant contributed 200 interactions, which encompassed both listening and speaking. This resulted in a cumulative total of 8,400 interactions across all participants. We evaluated the baseline performance of emotion recognition for each modality using established deep neural network (DNN) methods. The Emotion in EEG-Audio-Visual (EAV) dataset represents the first public dataset to incorporate three primary modalities for emotion recognition within a conversational context. We anticipate that this dataset will make significant contributions to the modeling of the human emotional process, encompassing both fundamental neuroscience and machine learning viewpoints.

Hybrid CNN-BiLSTM architecture with multiple attention mechanisms to enhance speech emotion recognition

During recent years, the concept of attention in deep learning has been increasingly used to boost the performance of Speech Emotion Recognition (SER) models. However, these models for SER exhibit shortcomings in jointly emphasizing the time-frequency and dynamic sequential variations, often under-utilizing the rich contextual emotion-related information. We propose a hybrid deep learning model for SER using Convolutional Neural Networks (CNN) and Bidirectional Long Short-Term Memory Networks (BiLSTM) with multiple attention mechanisms. Our model utilizes features from the speech waveform viz. Mel spectrograms and Mel Frequency Cepstral Coefficients (MFCC), along with their time derivatives as input to the CNN and BiLSTM modules, respectively. A Time–Frequency Attention (TFA) mechanism, optimally incorporated into CNN, helps to selectively focus on emotion-related energy–time–frequency variations in Mel spectrograms. Attention-based BiLSTM uses MFCC and its time derivatives to identify the positional information of emotion for addressing the dynamic sequential variations. Finally, we fuse the attention-learned features from the CNN and BiLSTM modules and feed them to a Deep Neural Network (DNN) for SER. The experiments were carried out using three different datasets: Emo-DB and IEMOCAP, which are public datasets, and Amritaemo_Arabic; a private dataset. The hybrid model exhibited superior performance on both the public and private datasets, generating an average SER accuracy of 94.62%, 67.85%, and 95.80% with Emo-DB, IEMOCAP, and Amritaemo_Arabic datasets, respectively, effectively outperforming several state-of-the-art models.

Advancing emotion recognition in facial expressions through PCA, RFE, and MLP Integration

Emotion recognition through facial expressions is vital for enhancing human-computer interaction, making systems more intuitive and responsive to user needs. This study introduces an innovative approach to emotion detection, leveraging Principal Component Analysis (PCA) and Recursive Feature Elimination (RFE) for feature extraction and optimization. The methodology focuses on refining facial feature representations to improve classification accuracy, which is critical for accurately detecting emotions like happiness, sadness, fear, and surprise. The approach was applied to two widely recognized facial emotion datasets: the Cohn-Kanade (CK) and the Japanese Female Facial Expression (JAFFE) datasets. By integrating PCA and RFE, the model efficiently selects the most relevant features, enhancing the overall performance of emotion recognition. A comparative analysis with existing deep learning models highlights the advantages of the proposed method. The effectiveness of this approach is further supported by the results, where the model achieves an accuracy of 98.49% on the CK dataset and 96.03% on the JAFFE dataset. These results demonstrate a significant improvement over recent methods, indicating the model's potential for real-world applications.

SFT-SGAT: A semi-supervised fine-tuning self-supervised graph attention network for emotion recognition and consciousness detection

Emotional recognition is highly important in the field of brain-computer interfaces (BCIs). However, due to the individual variability in electroencephalogram (EEG) signals and the challenges in obtaining accurate emotional labels, traditional methods have shown poor performance in cross-subject emotion recognition. In this study, we propose a cross-subject EEG emotion recognition method based on a semi-supervised fine-tuning self-supervised graph attention network (SFT-SGAT). First, we model multi-channel EEG signals by constructing a graph structure that dynamically captures the spatiotemporal topological features of EEG signals. Second, we employ a self-supervised graph attention neural network to facilitate model training, mitigating the impact of signal noise on the model. Finally, a semi-supervised approach is used to fine-tune the model, enhancing its generalization ability in cross-subject classification. By combining supervised and unsupervised learning techniques, the SFT-SGAT maximizes the utility of limited labeled data in EEG emotion recognition tasks, thereby enhancing the model’s performance. Experiments based on leave-one-subject-out cross-validation demonstrate that SFT-SGAT achieves state-of-the-art cross-subject emotion recognition performance on the SEED and SEED-IV datasets, with accuracies of 92.04% and 82.76%, respectively. Furthermore, experiments conducted on a self-collected dataset comprising ten healthy subjects and eight patients with disorders of consciousness (DOCs) revealed that the SFT-SGAT attains high classification performance in healthy subjects (maximum accuracy of 95.84%) and was successfully applied to DOC patients, with four patients achieving emotion recognition accuracies exceeding 60%. The experiments demonstrate the effectiveness of the proposed SFT-SGAT model in cross-subject EEG emotion recognition and its potential for assessing levels of consciousness in patients with DOC.

Multimodal insights into granger causality connectivity: Integrating physiological signals and gated eye-tracking data for emotion recognition using convolutional neural network

This study introduces a groundbreaking method to enhance the accuracy and reliability of emotion recognition systems by combining electrocardiogram (ECG) with electroencephalogram (EEG) data, using an eye-tracking gated strategy. Initially, we propose a technique to filter out irrelevant portions of emotional data by employing pupil diameter metrics from eye-tracking data. Subsequently, we introduce an innovative approach for estimating effective connectivity to capture the dynamic interaction between the brain and the heart during emotional states of happiness and sadness. Granger causality (GC) is estimated and utilized to optimize input for a highly effective pre-trained convolutional neural network (CNN), specifically ResNet-18. To assess this methodology, we employed EEG and ECG data from the publicly available MAHNOB-HCI database, using a 5-fold cross-validation approach. Our method achieved an impressive average accuracy and area under the curve (AUC) of 91.00 % and 0.97, respectively, for GC-EEG-ECG images processed with ResNet-18. Comparative analysis with state-of-the-art studies clearly shows that augmenting ECG with EEG and refining data with an eye-tracking strategy significantly enhances emotion recognition performance across various emotions.

Deficits of Facial Emotion Recognition in Elderly Individuals with Mild Cognitive Impairment.

The study of facial emotion recognition is under-explored in subjects with mild cognitive impairment (MCI). We investigated whether deficits in facial emotion recognition are present in patients with MCI. We also analyzed the relationship between facial emotion recognition and different domains of cognitive function. This study included 300 participants aged 60 years or older with cognitive decline. We evaluated 181 MCI and 119 non-MCI subjects using the Seoul Neuropsychological Screening Battery-Core (SNSB-C) and facial emotion recognition task using six facial expressions (anger, disgust, fear, happiness, sadness and surprise). A Generalized Linear Model (GLM) was used to assess the association between cognitive performance and accuracy of facial emotion recognition and to compare facial emotion recognition in the MCI group based on the impairment of five different domains of cognitive function. The model was adjusted for age, sex, years of education, and depressive symptoms. Patients with MCI had a lower score for accurately recognizing total facial emotion (0.48 vs. 0.53; ρ = 0.0003) and surprise (0.73 vs. 0.81; ρ = 0.0215) when compared to cognitively healthy subjects. We also discovered that frontal/executive function domain (Digit Symbol Coding [DSC, 0.38 vs. 0.49; p < 0.0001], Controlled Oral Word Association Test [COWAT, 0.42 vs. 0.49; p = 0.0001], Korean-Trail Making Test [K-TMT, 0.37 vs. 0.48; p = 0.0073], Korean-Color Word Stroop Test [K-CWST, 0.43 vs. 0.49; p = 0.0219]) and language domain (Korean-Boston Naming Test [S-K-BNT, 0.46 vs. 0.47; p = 0.003]) were statistically associated with the deficits of facial emotion recognition in patients with MCI. We observed a significant association between deficits in facial emotion recognition and cognitive impairment in elderly individuals.

EEG emotion recognition using EEG-SWTNS neural network through EEG spectral image

The rapid development in deep learning models provide considerable advancements in emotion recognition by electroencephalogram (EEG). However, existing approaches primarily focus on temporal and frequency domain features of EEG signals, neglecting spatial domain features in model information integration. Therefore, this study proposed a novel EEG input format, called EEG spectral image (ESI), which integrates spatial domain features using Azimuthal Equidistant Projection (AEP) and frequency domain features through differential entropy (DE). To better utilize this fine-grained data format, we propose the EEG Swin Transformer (EEG-SWTNS), which combines the window attention mechanism with shifted window partitioning. Window attention mechanism can concentrate on affective information within various regions and use shifted window partitioning to disrupt aggregated emotional representations for more granular features. Different from traditional graph neural networks, this method leverages the spatial locality of region information, leading to enhanced performance in EEG-based emotion recognition. Experiments conducted on SEED and SEED IV datasets demonstrate superior performance compared to baseline methods and the state-of-the-art models. Relative improvements of 0.6% and 0.08% are observed in subject-dependent experiments, while accuracies of 80.07% and 66.72% are achieved in subject-independent experiments without utilizing transfer learning techniques.

Evolutionary Ensemble Learning for EEG-Based Cross-Subject Emotion Recognition.

Electroencephalogram (EEG) has been widely utilized in emotion recognition due to its high temporal resolution and reliability. However, the individual differences and non-stationary characteristics of EEG, along with the complexity and variability of emotions, pose challenges in generalizing emotion recognition models across subjects. In this paper, an end-to-end framework is proposed to improve the performance of cross-subject emotion recognition. A novel evolutionary programming (EP)-based optimization strategy with neural network (NN) as the base classifier termed NN ensemble with EP (EPNNE) is designed for cross-subject emotion recognition. The effectiveness of the proposed method is evaluated on the publicly available DEAP, FACED, SEED, and SEED-IV datasets. Numerical results demonstrate that the proposed method is superior to state-of-the-art cross-subject emotion recognition methods. The proposed end-to-end framework for cross-subject emotion recognition aids biomedical researchers in effectively assessing individual emotional states, thereby enabling efficient treatment and interventions.

Optimizing Emotional Insight through Unimodal and Multimodal Long Short-term Memory Models

The field of multimodal emotion recognition is increasingly gaining popularity as a research area. It involves analyzing human emotions across multiple modalities, such as acoustic, visual, and language. Emotion recognition is more effective as a multimodal learning task than relying on a single modality. In this paper, we present an unimodal and multimodal long short-term memory model with a class weight parameter technique for emotion recognition on the CMU-Multimodal Opinion Sentiment and Emotion Intensity dataset. In addition, a critical challenge lies in selecting the most effective fusion method for integrating multiple modalities. To address this, we applied four different fusion techniques: Early fusion, late fusion, deep fusion, and tensor fusion. These fusion methods improved the performance of multimodal emotion recognition compared to unimodal approaches. With the highly imbalanced number of samples per emotion class in the MOSEI dataset, adding a class weight parameter technique leads our model to outperform the state of the art on all three modalities — acoustic, visual, and language — as well as on all the fusion models. The challenges of class imbalance, which can lead to biased model performance, and using an effective fusion method for integrating multiple modalities often result in decreased accuracy in recognizing less frequent emotion classes. Our proposed model shows 2–3% performance improvement in the unimodal and 2% in the multimodal over the state-of-the-art achieved results.

Facial emotion recognition and judgment of affective scenes in Parkinson's disease

Emotional dysfunctions in Parkinson's disease (PD) remain a controversial issue. While previous investigations showed compromised recognition of expressive faces in PD, no studies evaluated potential deficits in recognizing the emotional valence of affective scenes. This study aimed to investigate both facial emotion recognition performance and the ability to judge affective scenes in PD patients.Forty PD patients (mean age ± SD: 64.50 ± 8.19 years; 27 men) and forty healthy subjects (64.95 ± 8.25 years; 27 men) were included. Exclusion criteria were previous psychiatric disorders, previous Deep Brain Stimulation, and cognitive impairment. Participants were evaluated through the Ekman 60-Faces test and the International Affective Picture System. The accuracy in recognizing the emotional valence of facial expressions and affective scenes was compared between groups using linear mixed models. Pearson's correlation was performed to test the association between accuracy measures.The groups did not differ in sex, age, education, and Mini-Mental State Examination scores. Patients showed a lower recognition accuracy of facial expressions (68.54 % ± 15.83 %) than healthy participants (78.67 % ± 12.04 %; p < 0.001). Specifically, the PD group was characterized by lower recognition of faces expressing fear, sadness, and anger than the control group (all p < 0.020). No difference was detected for faces expressing disgust, surprise, and happiness (all p ≥ 0.25). Furthermore, patients showed lower accuracy in recognizing the emotional valence of affective scenes (66.75 % ± 14.59 %) than healthy subjects (74.83 % ± 12.65 %; p = 0.010). Pearson's correlations indicated that higher accuracy in recognizing the emotional facial expressions was associated with higher accuracy in classifying the valence of affective scenes in patients (r = 0.57, p < 0.001) and control participants (r = 0.57, p < 0.001).Our study suggested maladaptive affective processing in PD, leading patients to misinterpret both facial expressions and the emotional valence of complex evocative scenes.

ConTL: CNN, Transformer 및 LSTM의 결합을 통한 EEG 기반 감정인식 성능 개선

ConTL: CNN, Transformer 및 LSTM의 결합을 통한 EEG 기반 감정인식 성능 개선

Facial emotion recognition function and white matter microstructural alterations in drug-naive, comorbidity-free autism.

Individuals with autism spectrum disorder have deficits in facial emotion recognition and white matter microstructural alterations. Nonetheless, most previous studies were confounded by different variables, such as psychiatric comorbidities and psychotropic medications used by ASD participants. Also, it remains unclear how exactly FER deficits are related to white matter microstructural alterations in ASD. Accordingly, we aimed to investigate the FER functions, white matter microstructure, and their relationship in drug-naive and comorbidity-free ASD individuals. 59 ASD individuals and 59 typically developed individuals were included, where 46 ASD and 50 TD individuals completed FER tasks. Covariance analysis showed scores were lower in both basic and complex FER tasks in the ASD group. Tract-Based Spatial Statistics showed FA values in widespread white matter fibers were lower in the ASD group than in the TD group, including forceps major and forceps minor of the corpus callosum, anterior thalamic radiation, corticospinal tract, cingulum, inferior frontal-occipital fasciculus, inferior longitudinal fasciculus, superior longitudinal fasciculus. Moreover, in the TD group but not the ASD group, the performance in the complex FER task was negatively correlated with the FA value in some white matter fibers, including forceps major of the corpus callosum, ATR, CT, cingulum, IFOF, ILF, SLF. Our study suggests children with ASD may experience deficits in facial emotion recognition and exhibit alterations in white matter microstructure. More importantly, our study indicates that white matter microstructural alterations may be involved in FER deficits in children with ASD.

Joint enhancement and classification constraints for noisy speech emotion recognition

In the natural environment, the received speech signal is often interfered by noise, which reduces the performance of speech emotion recognition (SER) system. To this end, a noisy SER method based on joint constraints, including enhancement constraint and arousal-valence classification constraint (EC-AVCC), is proposed. This method introduces speech enhancement features and constraints into emotion recognition for the first time, which helps to improve the performance of noisy SER. It does not require denoising pre-processing for noisy speech, thus saving testing time. Specifically, it uses multi-domain statistical features (MDSF) composed of emotional and enhanced features as the input of the SER model based on convolution neural network (CNN) and long short-term memory-attention (ALSTM). In addition, the model is constrained jointly by speech enhancement and arousal-valence classification to obtain the robust and discriminative deep emotion features. Furthermore, in the auxiliary speech enhancement task, a joint loss function that simultaneously constrains the error of the ideal ratio mask and the error of the corresponding MDSF is introduced to obtain more robust features. The experimental results on the CASIA Chinese emotional dataset and the EMO-DB German emotional database show that compared with the CNN baseline, the proposed method improves the accuracy of SER in white noise and babble noise by 4.7%-9.9%.