Mental Workload Classification Research Articles

The diagnosticity of psychophysiological signatures: Can we disentangle mental workload from acute stress with ECG and fNIRS?

The ability to identify reliable and sensitive physiological signatures of psychological dimensions is key to developing intelligent adaptive systems that may in turn help to mitigate human error in complex operations. The challenge of this endeavor lies with diagnosticity. Despite different underlying causes, the physiological correlates of workload and acute psychological stress manifest in rather similar ways and can be easily confounded. The current work aimed to build a diagnostic model of mental state through the simultaneous classification of mental workload (varied through three levels of the n-back task) and acute stress (the presence/absence of aversive sounds) with machine learning. Using functional near infrared spectroscopy (fNIRS) and electrocardiography (ECG), the model's classifiers was above-chance to disentangle variations of mental workload from variations of acute stress. Both ECG and fNIRS could predict mental workload level, the best accuracy resulted from the two measures in combination. Stress level could not be accurately diagnosed through ECG alone, only with fNIRS or ECG and fNIRS combined. Individual calibration may be important since stress classification was more accurate for those with higher subjective state anxiety, perhaps due to a greater sensitivity to stress. Mental workload and stress were both better classified with activity in lateral prefrontal regions of the cortex than the medial areas, and the HbO2 signal generally lead to better classification than HHB. The current model represents a step forward to finely discriminate different mental states despite their rather analog physiological correlates.

Decoding Mental Workload in Virtual Environments: A fNIRS Study using an Immersive n-back Task.

Virtual Reality (VR) has emerged as a novel paradigm for immersive applications in training, entertainment, rehabilitation, and other domains. In this paper, we investigate the automatic classification of mental workload from brain activity measured through functional near-infrared spectroscopy (fNIRS) in VR. We present results from a study which implements the established n-back task in an immersive visual scene, including physical interaction. Our results show that user workload can be detected from fNIRS signals in immersive VR tasks both person-dependently and -adaptively.

Spectral and Temporal Feature Learning With Two-Stream Neural Networks for Mental Workload Assessment.

People's mental workload profoundly affects their work efficiency and health. Mental workload assessment can be used to effectively avoid serious accidents caused by excessive mental workload. Both electroencephalogram (EEG) spectral features and its temporal features have proven to be useful in addressing this problem. The fusion of the two types of features can provide rich distinguishing information for improving mental workload assessment. Benefiting from the progress of deep learning, this study proposes the two-stream neural networks (TSNN) for fusing the two types of EEG features. Compared with hand-crafted features, the TSNN can learn and fuse EEG features from the spectral and temporal dimensions automatically without prior knowledge. The TSNN includes a spectral stream and a temporal stream. Each stream consists of a convolutional neural network (CNN) and a temporal convolutional network (TCN) to learn spectral or temporal features from EEG topographic maps. To fuse the learned spectral and temporal information, we concatenate the output of the two streams prior to the fully connected layer. EEG data were collected from 17 subjects who performed n-back tasks with easy, medium, and hard difficulty levels, leading to a three-class mental workload classification. The results show that the TSNN achieves an average accuracy of 91.9%, which is a significant improvement over baseline classifiers based on hand-crafted features. The TSNN also outperforms state-of-the-art deep learning methods developed for EEG classification. The results indicate that the proposed structure is promising for fusing spectral and temporal features for mental workload assessment. In addition, it provides a high-precision approach for potential applications during cognitive activities.

Assessment and Classification of Mental Workload in the Prefrontal Cortex (PFC) Using Fixed-Value Modified Beer-Lambert Law

Optical-neuro-imaging based functional Near-Infrared Spectroscopy (fNIRS) has been in use for several years in the fields of brain research to measure the functional response of brain activity and apply it in fields such as Neuro-rehabilitation, Brain-Computer Interface (BCI) and Neuro-ergonomics. In this paper we have enhanced the classification accuracy of a Mental workload task using a novel Fixed-Value Modified Beer-Lambert law (FV-MBLL) method. The hemodynamic changes corresponding to mental workload are measured from the Prefrontal Cortex (PFC) using fNIRS. The concentration changes of oxygenated and deoxygenated hemoglobin (Δc HbO (t) and Δc HbR (t)) of 20 participants are recorded for mental workload and rest. The statistical analysis shows that data obtained from fNIRS is statistically significant with p 1.97 at confidence level of 0.95. The Support Vector Machine (SVM) classifier is used to discriminate mental math (coding) task from rest. Four features, namely mean, peak, slope and variance, are calculated on data processed through two different variants of Beer-lambert Law i.e., MBLL and FVMBLL for tissue blood flow. The optimal combination of the mean and peak values classified by SVM yielded the highest accuracy, 75%. This accuracy is further enhanced using the same feature combination, to 94% when those features are calculated using the novel algorithm FV-MBLL (with its optical density modelled form the first 4 sec stimulus data). The proposed technique can be effectively used with greater accuracies in the application of fNIRS for functional brain imaging and Brain-Machine Interface.

Learning Spatial-Spectral-Temporal EEG Features With Recurrent 3D Convolutional Neural Networks for Cross-Task Mental Workload Assessment.

Mental workload assessment is essential for maintaining human health and preventing accidents. Most research on this issue is limited to a single task. However, cross-task assessment is indispensable for extending a pre-trained model to new workload conditions. Because brain dynamics are complex across different tasks, it is difficult to propose efficient human-designed features based on prior knowledge. Therefore, this paper proposes a concatenated structure of deep recurrent and 3D convolutional neural networks (R3DCNNs) to learn EEG features across different tasks without prior knowledge. First, this paper adds frequency and time dimensions to EEG topographic maps based on a Morlet wavelet transformation. Then, R3DCNN is proposed to simultaneously learn EEG features from the spatial, spectral, and temporal dimensions. The proposed model is validated based on the EEG signals collected from 20 subjects. This paper employs a binary classification of low and high mental workload across spatial n-back and arithmetic tasks. The results show that the R3DCNN achieves an average accuracy of 88.9%, which is a significant increase compared with that of the state-of-the-art methods. In addition, the visualization of the convolutional layers demonstrates that the deep neural network can extract detailed features. These results indicate that R3DCNN is capable of identifying the mental workload levels for cross-task conditions.

Imbalanced classification of mental workload using a cost-sensitive majority weighted minority oversampling strategy

Identifying the temporal variations in mental workload level (MWL) is crucial for enhancing the safety of human---machine system operations, especially when there is cognitive overload or inattention of human operator. This paper proposed a cost-sensitive majority weighted minority oversampling strategy to address the imbalanced MWL data classification problem. Both the inter-class and intra-class imbalance problems are considered. For the former, imbalance ratio is defined to determine the number of the synthetic samples in the minority class. The latter problem is addressed by assigning different weights to borderline samples in the minority class based on the distance and density meaures of the sample distribution. Furthermore, multi-label classifier is designed based on an ensemble of binary classifiers. The results of analyzing 21 imbalanced UCI multi-class datasets showed that the proposed approach can effectively cope with the imbalanced classification problem in terms of several performance metrics including geometric mean (G-mean) and average accuracy (ACC). Moreover, the proposed approach was applied to the analysis of the EEG data of eight experimental participants subject to fluctuating levels of mental workload. The comparative results showed that the proposed method provides a competing alternative to several existing imbalanced learning algorithms and significantly outperforms the basic/referential method that ignores the imbalance nature of the dataset.

Feature Weight Driven Interactive Mutual Information Modeling for Heterogeneous Bio-Signal Fusion to Estimate Mental Workload.

Many people suffer from high mental workload which may threaten human health and cause serious accidents. Mental workload estimation is especially important for particular people such as pilots, soldiers, crew and surgeons to guarantee the safety and security. Different physiological signals have been used to estimate mental workload based on the n-back task which is capable of inducing different mental workload levels. This paper explores a feature weight driven signal fusion method and proposes interactive mutual information modeling (IMIM) to increase the mental workload classification accuracy. We used EEG and ECG signals to validate the effectiveness of the proposed method for heterogeneous bio-signal fusion. The experiment of mental workload estimation consisted of signal recording, artifact removal, feature extraction, feature weight calculation, and classification. Ten subjects were invited to take part in easy, medium and hard tasks for the collection of EEG and ECG signals in different mental workload levels. Therefore, heterogeneous physiological signals of different mental workload states were available for classification. Experiments reveal that ECG can be utilized as a supplement of EEG to optimize the fusion model and improve mental workload estimation. Classification results show that the proposed bio-signal fusion method IMIM can increase the classification accuracy in both feature level and classifier level fusion. This study indicates that multi-modal signal fusion is promising to identify the mental workload levels and the fusion strategy has potential application of mental workload estimation in cognitive activities during daily life.

A deep learning scheme for mental workload classification based on restricted Boltzmann machines

The mental workload (MWL) classification is a critical problem for quantitative assessment and analysis of operator functional state in many safety-critical situations with indispensable human–machine cooperation. The MWL can be measured by psychophysiological signals. In this work, we propose a novel restricted Boltzmann machine (RBM) architecture for MWL classification. In relation to this architecture, we examine two main issues: the optimal structure of RBM and selection of the most important EEG channels (electrodes) for MWL classification. The trial-and-error and entropy-based pruning methods are compared for the RBM structure identification. The degree of importance of EEG channels is calculated from the weights in a well-trained network in order to select the most relevant channels for classification task. Extensive comparative results showed that the selected EEG channels lead to accurate MWL classification across subjects.

Nonlinear Dynamic Classification of Momentary Mental Workload Using Physiological Features and NARX-Model-Based Least-Squares Support Vector Machines

This paper designs a pattern classifier based on a Nonlinear AutoRegressive model with eXogenous inputs (NARX) to reveal intricate nonlinear dynamical correlation between mental workload (MWL) of a human operator and psychophysiological features. The salient electroencephalogram and electrocardiogram features were selected as inputs to the NARX model, whose continuous output was discretized in terms of five MWL classes at each time instant. The orders of the NARX model were determined using an objective function to achieve a good tradeoff between model accuracy and complexity via a least-squares support vector machine. The physiological features from different measurement channels (electrodes) and frequency bands were compared in terms of multiclass MWL classification performance. The classification results showed that the locality projection preservation technique can maintain sufficiently high MWL classification accuracy (with the highest five-class correct classification rate of 88%) with a significantly reduced computational complexity. The comparative results of classification performance also demonstrated the superiority of the proposed dynamic model to a widely-used static model.

Cross-subject mental workload classification using kernel spectral regression and transfer learning techniques

Due to the poor generalizability of the subject-specific mental workload (MWL) classifier, we propose a cross-subject MWL recognition framework in this paper. Firstly, we use fuzzy mutual information-based wavelet-packet transform (FMI-WPT) technique to extract the salient physiological features of the MWL. Then, we combine kernel spectral regression (KSR) and transferable discriminative dimensionality reduction (TDDR) methods to reduce the dimensionality of the feature vector and to transfer the classifier model across subjects. Finally, the measured data analysis results are presented to show the enhanced performance of the proposed framework for multi-class MWL recognition.

Pattern Classification of Instantaneous Mental Workload Using Ensemble of Convolutional Neural Networks

In this paper, we consider the Mental Workload (MWL) classification problem based on the measured physiological data. Firstly we discussed the optimal classifier structure from two perspectives of Convolutional Neural Network (CNN) depth (i.e., the number of hidden layers) and parameter optimization algorithm. The base CNNs designed were tested according to Accuracy and the model training time required. Then we developed an Ensemble Convolutional Neural Network (ECNN) to enhance the robustness and accuracy of a single CNN model. For the ECNN design, three model aggregation approaches (weighted averaging, majority voting and stacking) were examined and a resampling strategy was used to enhance the diversity of individual CNN models. The results of a series of MWL classification performance comparison indicated that the proposed ECNN framework can effectively improve the MWL classification performance and is characterized by entirely automatic feature extraction and MWL classification, when compared with traditional machine learning methods.

Pattern Recognition of Momentary Mental Workload Based on Multi-Channel Electrophysiological Data and Ensemble Convolutional Neural Networks

In this paper, we deal with the Mental Workload (MWL) classification problem based on the measured physiological data. First we discussed the optimal depth (i.e., the number of hidden layers) and parameter optimization algorithms for the Convolutional Neural Networks (CNN). The base CNNs designed were tested according to five classification performance indices, namely Accuracy, Precision, F-measure, G-mean, and required training time. Then we developed an Ensemble Convolutional Neural Network (ECNN) to enhance the accuracy and robustness of the individual CNN model. For the ECNN design, three model aggregation approaches (weighted averaging, majority voting and stacking) were examined and a resampling strategy was used to enhance the diversity of individual CNN models. The results of MWL classification performance comparison indicated that the proposed ECNN framework can effectively improve MWL classification performance and is featured by entirely automatic feature extraction and MWL classification, when compared with traditional machine learning methods.

Task-Independent Mental Workload Classification Based Upon Common Multiband EEG Cortical Connectivity.

Efficient classification of mental workload, an important issue in neuroscience, is limited, so far to single task, while cross-task classification remains a challenge. Furthermore, network approaches have emerged as a promising direction for studying the complex organization of the brain, enabling easier interpretation of various mental states. In this paper, using two mental tasks (N-back and mental arithmetic), we present a framework for cross- as well as within-task workload discrimination by utilizing multiband electroencephalography (EEG) cortical brain connectivity. In detail, we constructed functional networks in EEG source space in different frequency bands and considering the individual functional connections as classification features, we identified salient feature subsets based on a sequential feature selection algorithm. These connectivity subsets were able to provide accuracy of 87% for cross-task, 88% for N-back task, and 86% for mental arithmetic task. In conclusion, our method achieved to detect a small number of discriminative interactions among brain areas, leading to high accuracy in both within-task and cross-task classifications. In addition, the identified functional connectivity features, the majority of which were detected in frontal areas in theta and beta frequency bands, helped delineate the shared as well as the distinct neural mechanisms of the two mental tasks.

Mental workload classification with concurrent electroencephalography and functional near-infrared spectroscopy

A brain-computer interface that measures the mental workload level of operators has applications in human-computer interactions (HCI) for reducing human error and improving work efficiency. In this study, concurrently recorded electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) were combined at the decision fusion stage for the classification of three mental workload levels induced by an n-back working-memory task. An average three-class classification accuracy of 42, 43, and 49% has been achieved across 13 participants for the fNIR-alone, EEG-alone, and EEG-fNIRS combined approach, respectively. The current study demonstrated a multimodality-based approach to decode human mental workload levels that may potentially be used for adaptive HCI applications.

A method for cross-task mental workload classification based on brain connectivity

A method for cross-task mental workload classification based on brain connectivity

Mental Workload Recognition by Combining Wavelet Packet Transform and Kernel Spectral Regression Techniques

A Mental Workload (MWL) recognition system was developed based on psychophysiological data to assess temporal variations in MWL levels. Salient EEG features were first extracted by using fuzzy mutual-information-based wavelet-packet transform (FMI-WPT). Then we adopted the kernel spectral regression linear discriminant analysis (KSRDA) to reduce the EEG feature dimensionality and to simultaneously enhance the inter-class discrimination capacity of the MWL classifiers. By combining FMIWPT and KSRDA techniques, we designed, evaluated and compared different types of MWL classifiers. The results demonstrated a improvement of the MWL classification accuracy by the proposed feature reduction method and classifier design framework. Particularly, it was shown by extensive comparative studies that the KNN and SVM outperform other classifiers.

A comparison of ERP spatial filtering methods for optimal mental workload estimation.

Mental workload estimation is of crucial interest for user adaptive interfaces and neuroergonomics. Its estimation can be performed using event-related potentials (ERPs) extracted from electroencephalographic recordings (EEG). Several ERP spatial filtering methods have been designed to enhance relevant EEG activity for active brain-computer interfaces. However, to our knowledge, they have not yet been used and compared for mental state monitoring purposes. This paper presents a thorough comparison of three ERP spatial filtering methods: principal component analysis (PCA), canonical correlation analysis (CCA) and the xDAWN algorithm. Those methods are compared in their performance to allow for an accurate classification of mental workload when applied in an otherwise similar processing chain. The data of 20 healthy participants that performed a memory task for 10 minutes each was used for classification. Two levels of mental workload were considered depending on the number of digits participants had to memorize (2/6). The highest performances were obtained using the CCA filtering and the xDAWN algorithm respectively with 98% and 97% of correct classification. Their performances were significantly higher than that obtained using the PCA filtering (88%).

Recognition of Mental Workload Levels Under Complex Human–Machine Collaboration by Using Physiological Features and Adaptive Support Vector Machines

In order to detect human operator performance degradation or breakdown, this paper proposes an adaptive support vector machine-based method to classify operator mental workload (MWL) into few discrete levels based on psychophysiological measures. Electroencephalogram, electrocardiogram, and electrooculography signals were recorded continuously while the operator was performing safety-critical process control operations in a simulated human-machine system. In coarse-grained analysis, the adaptive exponential smoothing (AES) technique is used to smooth the psychophysiological data and to remove strong artifacts without requiring templates. The MWL level is classified every 30 s by using bounded support vector machine (BSVM) and tenfold cross-validation techniques. Locality preservation projection (LPP) technique is utilized to derive salient psychophysiological features by means of feature reduction. By combining the AES-LPP and BSVM methods, the accuracy of the coarse-grained MWL classification was significantly improved by 11-13%. On the other hand, to perform MWL classification with higher temporal resolution and cross-subject and cross-trial generalizability, finer-grained data analysis is also conducted to recognize MWL levels every 5 s based on a combination of adaptive BSVM (ABSVM) and AES techniques. In comparison with the use of the BSVM algorithm alone, a significant performance improvement by 10-20% is achieved by using the AES-ABSVM method in the finer-grained MWL classification.

Adaptive training using an artificial neural network and EEG metrics for within- and cross-task workload classification

Adaptive training using neurophysiological measures requires efficient classification of mental workload in real time as a learner encounters new and increasingly difficult levels of tasks. Previous investigations have shown that artificial neural networks (ANNs) can accurately classify workload, but only when trained on neurophysiological exemplars from experienced operators on specific tasks. The present study examined classification accuracies for ANNs trained on electroencephalographic (EEG) activity recorded while participants performed the same (within task) and different (cross) tasks for short periods of time with little or no prior exposure to the tasks. Participants performed three working memory tasks at two difficulty levels with order of task and difficulty level counterbalanced. Within-task classification accuracies were high when ANNs were trained on exemplars from the same task or a set containing the to-be-classified task, (M=87.1% and 85.3%, respectively). Cross-task classification accuracies were significantly lower (average 44.8%) indicating consistent systematic misclassification for certain tasks in some individuals. Results are discussed in terms of their implications for developing neurophysiologically driven adaptive training platforms.

DISCRIMINATION OF MENTAL WORKLOAD LEVELS IN HUMAN SUBJECTS WITH FUNCTIONAL NEAR-INFRARED SPECTROSCOPY

We have applied functional near-infrared spectroscopy (fNIRS) to the human forehead to distinguish different levels of mental workload on the basis of hemodynamic changes occurring in the prefrontal cortex. We report data on 3 subjects from a protocol involving 3 mental workload levels based on to working memory tasks. To quantify the potential of fNIRS for mental workload discrimination, we have applied a 3-nearest neighbor classification algorithm based on the amplitude of oxyhemoglobin ( HbO 2) and deoxyhemoglobin ( HbR ) concentration changes associated with the working memory tasks. We have found classification success rates in the range of 44–72%, which are significantly higher than the corresponding chance level (for random data) of 19.1%. This work shows the potential of fNIRS for mental workload classification, especially when more parameters (rather than just the amplitude of concentration changes used here) and more sophisticated classification algorithms (rather than the simple 3-nearest neighbor algorithm used here) are considered and optimized for this application.