Recognition Of Mental States Research Articles

MCNN-CMCA: A multiscale convolutional neural networks with cross-modal channel attention for physiological signal-based mental state recognition

Human mental state recognition (MSR) has significant implications for human-machine interactions. Although mental state recognition models based on single-modality signals, such as electroencephalogram (EEG) or peripheral physiological signals (PPS), have achieved encouraging progress, methods leveraging multimodal physiological signals still need to be explored. In this study, we present MCNN-CMCA, a generic model that employs multiscale convolutional neural networks (CNNs) with cross-modal channel attention to realize physiological signals-based MSR. Specifically, we first design an innovative cross-modal channel attention mechanism that adaptively adjusting the weights of each signal channel, effectively learning both intra-modality and inter-modality correlation and expanding the channel information to the depth dimension. Additionally, the study utilizes multiscale temporal CNNs for obtaining short-term and long-term time-frequency features across different modalities. Finally, the multimodal fusion module integrates the representations of all physiological signals and the classification layer implements sparse connections by setting the mask weights to 0. We evaluate the proposed method on the SEED-VIG, DEAP, and self-made datasets, achieving superior results compared to existing state-of-the-art methods. Furthermore, we conduct ablation studies to demonstrate the effectiveness of each component in the MCNN-CMCA and show the use of multimodal physiological signals outperforms single-modality signals.

Empathy among autistic and non-autistic adolescents: The importance of informant effects.

Empathy is the ability to recognize the emotions of others (cognitive empathy) and to share in those emotions while maintaining a self-other distinction (emotional empathy). Previous research often, but not always, showed that autistic adults and children have lower levels of overall and cognitive empathy than non-autistic individuals. Yet how empathy manifests during adolescence, a developmental period marked by physiological, social, and cognitive change, is largely unclear. As well, we aimed to compare self versus parents' perceptions regarding adolescents' empathy. To do so, parents (N = 157) of 10-16-year-olds (N = 59 autistic) and their children (N = 133) completed empathy questionnaires. Adolescents also completed a measure of mental state recognition (Reading the Mind in the Eyes Test; RMET) and parents reported on their child's autistic traits. The tasks were completed twice ~six months apart. We found that autistic adolescents reported having lower empathic concern and higher personal distress than their non-autistic peers, whereas parents of autistic adolescents perceived them as having overall lower levels of empathy. Performance on the mental state recognition task of autistic and non-autistic adolescents' was comparable. The gap between self and parent reports regarding adolescents' empathy was explainable by parent-reported autistic traits, mainly communication difficulties. Empathy remains stable across the study's two time points. Thus, the findings do not support previous views of autistic people as having less empathy and these are possibly explainable by informant effects.

Improving mentalizing deficits in older age with region-specific transcranial direct current stimulation

Older adults have difficulties to detect the intentions, thoughts, and feelings of others, indicating an age-associated decline of socio-cognitive abilities that are known as “mentalizing”. These deficits in mental state recognition are driven by neurofunctional alterations in brain regions that are implicated in mentalizing, such as the right temporo-parietal junction (rTPJ) and the dorso-medial prefrontal cortex (dmPFC). We tested whether focal transcranial current stimulation (tDCS) of the rTPJ and dmPFC has the potential to eliminate mentalizing deficits in older adults. Mentalizing deficits were assessed with a novel mindreading task that required the recognition of mental states in child faces. Older adults (n = 60) performed worse than younger adults (n = 30) on the mindreading task, indicating age-dependent deficits in mental state recognition. These mentalizing deficits were ameliorated in older adults who received sham-controlled andodal tDCS over the rTPJ (n = 30) but remained unchanged in older adults who received sham-controlled andodal tDCS over the dmPFC (n = 30). We, thus, showed for the first time that anodal tDCS over the rTPJ has the potential to remediate age-dependent mentalizing deficits in a region-specific way. This provides a rationale for exploring stimulation-based interventions targeting mentalizing deficits in older age.

Using Multimodal Methods and Machine Learning to Recognize Mental Workload: Distinguishing Between Underload, Moderate Load, and Overload

Mental workload recognition is of great significance in preventing human errors and accidents. This study constructed a multimodal recognition scheme to recognize three mental workload states: underload, moderate load, and overload. Based on driving scenarios, these three states were induced in this study by changing the driving modes and situations. Multimodal recognition of underload, moderate load, and overload was performed using electroencephalography (EEG), electrocardiography (ECG), and pupillometry. In addition, various machine learning methods were used to evaluate the recognition performance of different feature combinations. The results showed that the random forest method, trained using spectral power, pupil diameter, and heart rate variability, achieved the highest recognition accuracy of 83.13% for the three mental workload states. This study provides valuable reference information for multimodal recognition of mental workload states.

EF-Net: Mental State Recognition by Analyzing Multimodal EEG-fNIRS via CNN.

Analysis of brain signals is essential to the study of mental states and various neurological conditions. The two most prevalent noninvasive signals for measuring brain activities are electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). EEG, characterized by its higher sampling frequency, captures more temporal features, while fNIRS, with a greater number of channels, provides richer spatial information. Although a few previous studies have explored the use of multimodal deep-learning models to analyze brain activity for both EEG and fNIRS, subject-independent training-testing split analysis remains underexplored. The results of the subject-independent setting directly show the model's ability on unseen subjects, which is crucial for real-world applications. In this paper, we introduce EF-Net, a new CNN-based multimodal deep-learning model. We evaluate EF-Net on an EEG-fNIRS word generation (WG) dataset on the mental state recognition task, primarily focusing on the subject-independent setting. For completeness, we report results in the subject-dependent and subject-semidependent settings as well. We compare our model with five baseline approaches, including three traditional machine learning methods and two deep learning methods. EF-Net demonstrates superior performance in both accuracy and F1 score, surpassing these baselines. Our model achieves F1 scores of 99.36%, 98.31%, and 65.05% in the subject-dependent, subject-semidependent, and subject-independent settings, respectively, surpassing the best baseline F1 scores by 1.83%, 4.34%, and 2.13% These results highlight EF-Net's capability to effectively learn and interpret mental states and brain activity across different and unseen subjects.

Mental fatigue state recognition method based on convolution neural network and long short-term memory

The pace of modern life is accelerating, the pressure of life is gradually increasing, and the long-term accumulation of mental fatigue poses a threat to health. By analyzing physiological signals and parameters, this paper proposes a method that can identify the state of mental fatigue, which helps to maintain a healthy life. The method proposed in this paper is a new recognition method of psychological fatigue state of electrocardiogram signals based on convolutional neural network and long short-term memory. Firstly, the convolution layer of one-dimensional convolutional neural network model is used to extract local features, the key information is extracted through pooling layer, and some redundant data is removed. Then, the extracted features are used as input to the long short-term memory model to further fuse the ECG features. Finally, by integrating the key information through the full connection layer, the accurate recognition of mental fatigue state is successfully realized. The results show that compared with traditional machine learning algorithms, the proposed method significantly improves the accuracy of mental fatigue recognition to 96.3%, which provides a reliable basis for the early warning and evaluation of mental fatigue.

Enhanced performance of EEG-based brain-computer interfaces by joint sample and feature importance assessment.

Electroencephalograph (EEG) has been a reliable data source for building brain-computer interface (BCI) systems; however, it is not reasonable to use the feature vector extracted from multiple EEG channels and frequency bands to perform recognition directly due to the two deficiencies. One is that EEG data is weak and non-stationary, which easily causes different EEG samples to have different quality. The other is that different feature dimensions corresponding to different brain regions and frequency bands have different correlations to a certain mental task, which is not sufficiently investigated. To this end, a Joint Sample and Feature importance Assessment (JSFA) model was proposed to simultaneously explore the different impacts of EEG samples and features in mental state recognition, in which the former is based on the self-paced learning technique while the latter is completed by the feature self-weighting technique. The efficacy of JSFA is extensively evaluated on two EEG data sets, i.e., SEED-IV and SEED-VIG. One is a classification task for emotion recognition and the other is a regression task for driving fatigue detection. Experimental results demonstrate that JSFA can effectively identify the importance of different EEG samples and features, leading to enhanced recognition performance of corresponding BCI systems.

Channel Semantic Enhancement-Based Emotional Recognition Method Using SCLE-2D-CNN

The existing EEG emotion classification methods have some problems, such as insufficient emotion representation and lack of targeted channel enhancement module due to feature redundancy. To this end, a novel EEG emotion recognition method (SCLE-2D-CNN) combining scaled convolutional layer (SCLs), enhanced channel module and two-dimensional convolutional neural network (2D-CNN) is proposed. Firstly, the time-frequency features of multi-channel EEG emotional signals were extracted by stacking scl layer by layer. Secondly, channel enhancement module is used to reassign different importance to all EEG physical channels. Finally, 2D-CNN was used to obtain deep local spatiotemporal features and complete emotion classification. The experimental results show that the accuracy of SEED data set and F1 are 98.09% and 97.00%, respectively, and the binary classification accuracy of DEAP data set is 98.06% and 96.83%, respectively, which are superior to other comparison methods. The proposed method has a certain application prospect in the recognition of human mental state.

Mental Health Status Detection Model Based on LSTM Neural Network

In today's society where mental health is increasingly concerned, how to effectively monitor and evaluate an individual's mental health status has become a research focus. However, traditional mental health evaluation methods have disadvantages such as strong subjectivity and strong dependence on professionals, and there is an urgent need to establish an automatic evaluation method based on objective data. This article is based on the LSTM (Long Short-Term Memory) mental health state detection model, which inputs the text sequence data to be detected into the LSTM model, and obtains the corresponding mental health state prediction results through forward transmission. A threshold can be set to determine whether the mental health status is normal, or the predicted results can be compared with real markers to evaluate the precision and performance of the model. Multiple features such as emotional analysis with textual features were integrated to construct a new model for evaluating the role of integrated features in mental health state recognition. Only text features had an accuracy rate of 0.87, an precision rate of 0.89, and a recall rate of 0.86. The accuracy rate under text feature+sentiment analysis was 0.90; the precision rate was 0.92; the recall rate was 0.89. This article provided reference and tool support for the detection of mental health status.

Spatial-Frequency Characteristics of EEG Associated With the Mental Stress in Human-Machine Systems.

Accurate assessment of user mental stress in human-machine system plays a crucial role in ensuring task performance and system safety. However, the underlying neural mechanisms of stress in human-machine tasks and assessment methods based on physiological indicators remain fundamental challenges. In this paper, we employ a virtual unmanned aerial vehicle (UAV) control experiment to explore the reorganization of functional brain network patterns under stress conditions. The results indicate enhanced functional connectivity in the frontal theta band and central beta band, as well as reduced functional connectivity in the left parieto-occipital alpha band, which is associated with increased mental stress. Evaluation of network metrics reveals that decreased global efficiency in the theta and beta bands is linked to elevated stress levels. Subsequently, inspired by the frequency-specific patterns in the stress brain network, a cross-band graph convolutional network (CBGCN) model is constructed for mental stress brain state recognition. The proposed method captures the spatial-frequency topological relationships of cross-band brain networks through multiple branches, with the aim of integrating complex dynamic patterns hidden in the brain network and learning discriminative cognitive features. Experimental results demonstrate that the neuro-inspired CBGCN model improves classification performance and enhances model interpretability. The study suggests that the proposed approach provides a potentially viable solution for recognizing stress states in human-machine system by using EEG signals.

EEG-Based Cross-Subject Driver Drowsiness Recognition With an Interpretable Convolutional Neural Network.

In the context of electroencephalogram (EEG)-based driver drowsiness recognition, it is still challenging to design a calibration-free system, since EEG signals vary significantly among different subjects and recording sessions. Many efforts have been made to use deep learning methods for mental state recognition from EEG signals. However, existing work mostly treats deep learning models as black-box classifiers, while what have been learned by the models and to which extent they are affected by the noise in EEG data are still underexplored. In this article, we develop a novel convolutional neural network combined with an interpretation technique that allows sample-wise analysis of important features for classification. The network has a compact structure and takes advantage of separable convolutions to process the EEG signals in a spatial-temporal sequence. Results show that the model achieves an average accuracy of 78.35% on 11 subjects for leave-one-out cross-subject drowsiness recognition, which is higher than the conventional baseline methods of 53.40%-72.68% and state-of-the-art deep learning methods of 71.75%-75.19%. Interpretation results indicate the model has learned to recognize biologically meaningful features from EEG signals, e.g., alpha spindles, as strong indicators of drowsiness across different subjects. In addition, we also explore reasons behind some wrongly classified samples with the interpretation technique and discuss potential ways to improve the recognition accuracy. Our work illustrates a promising direction on using interpretable deep learning models to discover meaningful patterns related to different mental states from complex EEG signals.

Correction to: Comparative Analysis of Signal Processing Techniques for Mental State Recognition in Brain-Computer Interfaces (BCI)

Correction to: Comparative Analysis of Signal Processing Techniques for Mental State Recognition in Brain-Computer Interfaces (BCI)

Comparative Analysis of Signal Processing Techniques for Mental State Recognition in Brain-Computer Interfaces (BCI)

Comparative Analysis of Signal Processing Techniques for Mental State Recognition in Brain-Computer Interfaces (BCI)

The FACE test: a new neuropsychological task to assess the recognition of complex mental states from faces

BackgroundSocial cognition deficits are reported in several neurodegenerative diseases, including Parkinson’s disease (PD). However, the availability of tasks for the clinical assessment is still limited, preventing the full characterization of socio-cognitive dysfunctions in neurological patients. This study aims to present a new task to assess the recognition of complex mental states from faces (FACE test), reporting normative data for the Italian population and an example of its clinical application to 40 PD patients.MethodsTwo-hundred twenty-nine Italian participants with at least 5 years of education were enrolled. Data were analyzed according to the method of equivalent scores; test-retest reliability and convergent validity were assessed. Two short versions of the FACE test were defined for clinical and research purposes. The prevalence of deficits in the FACE test was computed in the PD sample, as well as correlations with cognitive performance and diagnostic accuracy.ResultsRegression analyses revealed significant effects of demographic variables on FACE performance, with younger and more educated individuals showing higher scores. Twenty-eight percent of PD patients showed borderline/pathological performance, which was correlated with emotion recognition/attribution abilities, and attentive-executive functions. The FACE test was accurate (80%) in distinguishing PD patients with socio-cognitive dysfunctions from both controls and PD patients without emotion recognition/attribution difficulties.ConclusionThe FACE test represents a new tool assessing the ability to recognize complex mental states from facial expressions. Overall, these results support its use in both clinical and research settings, as well as the presence of affective processing deficits in a subsample of PD patients.

Social Competence in Adolescents with Different Level of Theory of Mind

The aim of this study was to explore the role of theory of mind - the ability to attribute mental states to other people to explain their behavior - in the development of social competence and prosocial behavior in adolescence. It is assumed that there is a difference in the manifestation of prosocial behavior and social competence in adolescents with different levels of theory of mind. Participants were 95 adolescents aged 12–15 years (M=13.8; SD=9.5; 46 girls). We used “Reading the Mind in the Eyes” test to assess the social-perceptual component of theory of mind (assessment of the recognition of mental states by facial expressions) and the task of understanding higher-order false beliefs to assess the socio-cognitive component. We assessed the adolescent’s prosocial behavior during a group game of puzzles and calculated the ratio of the number of prosocial acts to time in seconds. Teachers assessed the social behavior of adolescents using a specially designed questionnaire. We also used a sociometric method to study the popularity of adolescents in the group. The results obtained testify to the different contribution of the socio-perceptual and socio-cognitive components of theory of mind to the development of social competence of adolescents. Adolescents who better understand the mental states of others and predict their behavior are more likely to act prosocially, are more persuasive and successful in mastering the school curriculum. Teachers are rated those who better recognize mental states by facial expressions as more helpful and popular among their peers. Adolescents with low level of theory of mind are rejected by their peers.

Modulating mental state recognition by anodal tDCS over the cerebellum

Increasing evidence from neuroimaging and clinical studies has demonstrated cerebellar involvement in social cognition components, including the mentalizing process. The aim of this study was to apply transcranial direct current stimulation (tDCS) to modulate cerebellar excitability to investigate the role the cerebellum plays in mental state recognition. Forty-eight healthy subjects were randomly assigned to different groups in which anodal, cathodal, or sham tDCS (2 mA for 20 min) was delivered centering the electrode on the vermis to stimulate the posterior portion of the cerebellum. The ability to attribute mental states to others was tested before and after tDCS using a digital version of the 'Reading the Mind in the Eyes test', which includes visual perceptive and motor stimuli as control conditions. Correct response and reaction times (RTs) were recorded. The results revealed a significant reduction in RTs between the baseline and post-stimulation sessions after cerebellar anodal tDCS only for mental state stimuli (Wilcoxon test p = 0.00055), whereas no significant effect was found in the cathodal or sham conditions or for visual perceptive and motor stimuli. Overall, our study suggests that cerebellar anodal tDCS might selectively improve mental state recognition and constitute an effective strategy to positively modulate the mentalizing process.

Research on mental load state recognition based on combined information sources

Whether from a theoretical research perspective or a practical situation, mental loads lead to lower productivity, worse job quality, and industrial accidents, and evaluating the mental loads of workers is important for task assignment, job performance, and production safety. In this study, impact sensitivity experiments were used to induce emotions such as fear and tension in participants, thus causing them to develop mental load. Prospective time-distance estimation tests and subjective questionnaires were used to validate the induction experiment. EEG signals from three brain regions and eye movements of the subjects made up four single variables, and combinations of different brain regions and eye movements were used as combined variables. Thus, a total of 15 variables were used for the analysis of mental load states. The Relief algorithm was applied to the 15 variables for feature extraction to construct an optimized feature set. Then, 4 types of classifiers were used to identify mental load states. The combination of occipital area and eye movement variables achieved better mental load assessment results than single variables. With this combination variable, the SVM (94.9 %) and RF (94.3 %) classifiers were effective. However, the SVM classification accuracy gradually increased with the number of training sessions, while the RF classifier did not show this trend. Therefore, the SVM classifier was the best mental load classification method. The results of this study can be used to assess the mental loads of workers, reducing the number of accidents and improving productivity.

Cloud Data System Design for Mental Crisis State Recognition of College Students Based on Machine Learning

At present, Chinese college students are facing a lot of psychological pressure; whether it is teaching pressure or life pressure, it will have a certain adverse impact on college students’ psychological state, and if timely guidance is not provided, it will result in some adverse consequences. Therefore, it is necessary to timely identify the psychological crisis situation of college students, but the existing form of manual identification has high limitations, which cannot obtain the psychological state of students more accurately and efficiently, so it is necessary to optimize and improve with the help of network technology. Cloud computing data system is one of the mature big data systems at present. The combination of cloud computing system and machine learning technology is effectively applied to the field of psychological crisis analysis, which can quickly screen the psychological status of college students and report abnormal data in a timely manner, so as to help college psychological teachers identify the state of college students’ psychological crisis and intervene in a timely manner to promote the physical and mental health of students. By applying machine learning technology for the establishment of a cloud computing data system and putting the system into the field of psychological crisis identification of Chinese college students, this study lays a theoretical and practical foundation for preventing students from the psychological crisis.

Buddy System: An Adaptive Mental State Support System Based on Active Inference and Free-Energy Principles

To support a healthy human mental state, controlling the environment is one of the best-known solutions. However, it is difficult to design an environmental control system because of the uniqueness of individual preferences and mental state fluctuations. Here, we propose “Buddy system” as an adaptive mental state support solution that controls devices in the environment, depending on the recognized user’s mental state at the time and how it could be improved, serving a role similar to a “buddy” to individuals. The recognition of mental states and the locus of actions to control one’s surrounding environment are implemented on the basis of a brain-derived theory of computation known as active inference and free-energy principles, which provide biologically plausible computations for perceptions and behavior in a changing world. For the generation of actions, we modify the general calculations of active inference to adjust to individual environmental preferences. In the experiments, the Buddy system sought to maintain the participants’ concentration while a calculation task was conducted. As a result, the task performance for most of the participants was improved through the aid of the Buddy system. The results indicated that the Buddy system can adaptively support to improve the mental states of individual users.

General Aviation Loss of Control in Flight Accidents: Causal and Contributory Factors

Loss of control in flight is the primary fatal accident category in general aviation. Forty-six fixed-wing United Kingdom accidents from 2018 and 2019 were analyzed to identify precursors, human factors, and possible reasons for unsuccessful recovery. Most of the events were nonfatal (82.6%), and most occurred during low-altitude flight phases: particularly, landings and go-arounds. Pilots under the age of 40 and over the age of 75 were disproportionately more likely to experience loss of control in flight. It was mostly precipitated by ineffective recovery from an upset, inadequate energy management, abnormal/inadvertent control inputs or maneuvers, or improper procedures. Insufficient height above the ground was a factor in most unsuccessful recoveries, followed by limited pilot capability. Fatal accidents were much more likely to be unrecoverable due to a hazardous mental or physical state or incorrect recognition of the situation. Decision- and skill-based human errors contributed to most events; more than half of the cases involved both errors. Fatal accidents were more complex in terms of preflight and latent human errors. These results informed a new definition of loss of control in flight for general aviation combined with a conceptual framework to inform future intervention strategies.