Cognitive Load Classification Research Articles

Exploring cognitive load through neuropsychological features: an analysis using fNIRS-eye tracking.

Cognition is crucial to brain function, and accurately classifying cognitive load is essential for understanding the psychological processes across tasks. This paper innovatively combines functional near-infrared spectroscopy (fNIRS) with eye tracking technology to delve into the classification of cognitive load at the neurocognitive level. This integration overcomes the limitations of a single modality, addressing challenges such as feature selection, high dimensionality, and insufficient sample capacity. We employ fNIRS-eye tracking technology to collect neural activity and eye tracking data during various cognitive tasks, followed by preprocessing. Using the maximum relevance minimum redundancy algorithm, we extract the most relevant features and evaluate their impact on the classification task. We evaluate the classification performance by building models (naive Bayes, support vector machine, K-nearest neighbors, and random forest) and employing cross-validation. The results demonstrate the effectiveness of fNIRS-eye tracking, the maximum relevance minimum redundancy algorithm, and machine learning techniques in discriminating cognitive load levels. This study emphasizes the impact of the number of features on performance, highlighting the need for an optimal feature set to improve accuracy. These findings advance our understanding of neuroscientific features related to cognitive load, propelling neural psychology research to deeper levels and holding significant implications for future cognitive science.

Classification of Driver Cognitive Load in Conditionally Automated Driving: Utilizing Electrocardiogram-Based Spectrogram with Lightweight Neural Network

With the development of conditionally automated driving, drivers will be allowed to perform non-driving-related tasks. Under such circumstances, continuous monitoring of driver cognitive load will play an increasingly important role in ensuring that drivers have sufficient mental resources to take over control of the vehicle should the driving automation fail. However, estimation of cognitive load is challenging because of the difficulties in identifying high-level feature representation and accounting for interindividual differences. Physiological measures are believed to be promising candidates for cognitive load estimation in partially automated vehicles. However, current estimation methods are mainly based on the manual feature extraction of time- or frequency-domain indicators from physiological signals, which may not adapt to dynamic driving conditions. With the development of deep learning, the neural network has shown good performance in automatically capturing high-level features from input data. Inspired by this, we adopted a novel approach to classify driver cognitive load based on electrocardiogram (ECG) spectrograms, in which the driver’s ECG signal was collected and transformed into a 2D spectrogram by a short-time Fourier transform. A squeeze-and-excitation network-based deep-learning framework that can capture high-level features and pays more attention to the cognition-related features of the spectrogram was proposed for classification. Experiments on a publicly available dataset demonstrated that our model achieved an accuracy of 96.76% in differentiating two levels of cognitive load for a within-subject evaluation and 71.50% accuracy with an across-subjects evaluation. The results demonstrated the feasibility of detecting drivers’ cognitive load through deep learning using ECG spectrogram alone.

Cross-Task and Cross-Participant Classification of Cognitive Load in an Emergency Simulation Game

Assessment of cognitive load is a major step towards adaptive interfaces. However, non-invasive assessment is rather subjective as well as task specific and generalizes poorly, mainly due to methodological limitations. Additionally, it heavily relies on performance data like game scores or test results. In this study, we present an eye-tracking approach that circumvents these shortcomings and allows for effective generalizing across participants and tasks. First, we established classifiers for predicting cognitive load individually for a typical working memory task (n-back), which we then applied to an emergency simulation game by considering the similar ones and weighting their predictions. Standardization steps helped achieve high levels of cross-task and cross-participant classification accuracy between 63.78% and 67.25% for the distinction between easy and hard levels of the emergency simulation game. These very promising results could pave the way for novel adaptive computer-human interaction across domains and particularly for gaming and learning environments.

Fusion of Spatial, Temporal, and Spectral EEG Signatures Improves Multilevel Cognitive Load Prediction

Cognitive load prediction is one of the most important issues in the nascent field of neuroergonomics, and it has significant value in real-world applications. Most of the previous studies of cognitive load prediction only utilized electroencephalography (EEG)-based spectral signatures or interchannel connectivity, ignoring abundant temporal microstate features, which may represent the transient topologies of EEG signals. Furthermore, previous studies have mostly focused on the binary-level classification of cognitive load for single-type cognitive tasks. To date, there are few studies on the multilevel prediction of cognitive load during mixed cognitive tasks. Here, we first designed a new paradigm termed the “finding fault game,” mixing multiple tasks of memory, counting, and visual search, and then developed a multidimensional analysis framework to improve cognitive load prediction using a fusion of spatial, temporal, and spectral EEG features. Specifically, EEG-based functional connectivity, microstates and power spectral densities (PSD) were calculated for three cognitive load levels. Twelve adult subjects participated in the study. The experimental results show that increased cognitive load was associated with elevated theta and degraded alpha power and significant changes in interchannel connectivity and microstates, and that fusing the three types of EEG features improved the performance of three-level cognitive load prediction, achieving the accuracies of greater than 80% in the cross-validation, real-time, and over-time prediction. The findings suggest that all three types of EEG features can serve as signatures of cognitive load and that their fusion can improve multilevel prediction.

The Role of EEG Signals: SVM Classification of Cognitive Load as a Support for UX Evaluation

Cognitive load is the mental effort that needs to be applied to working memory to process information received over a period of time. Cognitive load can be viewed as the level of mental energy required to process a given amount of information. In user experience design, cognitive load is considered as the mental processing power required to use a product. If the amount of information processed exceeds the user's ability to process it, the overall performance will be disrupted. An EEG device is needed that is used to record electrical activity that occurs in the brain by channeling brain electrical waves to cables and modulators that are sensitive to electrical waves. The object of this research is the EEG Beta signal with the attention wave type from UX testing activities on students aged 21-24 years with a frequency level of 13-30 Hz. The EEG tool records the activity of the respondent's wave signal by collecting data on the activity of working on a questionnaire about evaluating the WhatsApp application using the Google Form application. The classification of cognitive load studied is unencumbered and burdened. Unencumbered represents the ease that is felt when interacting with the application, while burdened represents the difficulty or confusion that is felt when interacting with the application. Testing is done with the Confusion matrix. The best accuracy results among the kernel types in the SVM method are linear kernel types with an accuracy result of 89% consisting of 1 data that is categorized as an unencumbered label and 8 data labels that are loaded

Cognitive Load Prediction from Multimodal Physiological Signals using Multiview Learning.

Predicting cognitive load is a crucial issue in the emerging field of human-computer interaction and holds significant practical value, particularly in flight scenarios. Although previous studies have realized efficient cognitive load classification, new research is still needed to adapt the current state-of-the-art multimodal fusion methods. Here, we proposed a feature selection framework based on multiview learning to address the challenges of information redundancy and reveal the common physiological mechanisms underlying cognitive load. Specifically, the multimodal signal features (EEG, EDA, ECG, EOG, & eye movements) at three cognitive load levels were estimated during multiattribute task battery (MATB) tasks performed by 22 healthy participants and fed into a feature selection-multiview classification with cohesion and diversity (FS-MCCD) framework. The optimized feature set was extracted from the original feature set by integrating the weight of each view and the feature weights to formulate the ranking criteria. The cognitive load prediction model, evaluated using real-time classification results, achieved an average accuracy of 81.08% and an average F1-score of 80.94% for three-class classification among 22 participants. Furthermore, the weights of the physiological signal features revealed the physiological mechanisms related to cognitive load. Specifically, heightened cognitive load was linked to amplified δ and θ power in the frontal lobe, reduced α power in the parietal lobe, and an increase in pupil diameter. Thus, the proposed multimodal feature fusion framework emphasizes the effectiveness and efficiency of using these features to predict cognitive load.

Preprocessing of PPG and EDA signals for detection of emotional and cognitive states via physiological signals

Presented in the current paper is a methodology for approaching the preprocessing of Photoplethysmography and Electrodermal activity for the detection of emotional and cognitive states in humans via physiological signals. Examined closely are the effects of downsampling and segmentation of the PPG, the segmentation and separation of the Skin Conductance Level (SCL), and Skin Conductance Response (SCR) components of the EDA signal with both median and low pass filters. The results from the research indicate that the most appropriate preprocessing with regard to emotions and cognitive load classification is segmentation of 2 minutes which is the recommended length for frequency analysis of heart rate variability. Recommended, furthermore, is the downsampling of the PPG to 64 Hz, which proved to be the lowest sampling frequency that doesn’t introduce errors in the systolic peak detection, neither does it drastically affect the length of the Inter Beat Intervals (IBIs). Proposed, as to the separation of the SCL component of the EDA, is the usage of median filter with window length of 75% of the sampling frequency, which introduces negligible artefacts, mainly at the start of the signal.

Classification of Driver Cognitive Load: Exploring the Benefits of Fusing Eye-Tracking and Physiological Measures

In-vehicle infotainment systems can increase cognitive load and impair driving performance. These effects can be alleviated through interfaces that can assess cognitive load and adapt accordingly. Eye-tracking and physiological measures that are sensitive to cognitive load, such as pupil diameter, gaze dispersion, heart rate (HR), and galvanic skin response (GSR), can enable cognitive load estimation. The advancement in cost-effective and nonintrusive sensors in wearable devices provides an opportunity to enhance driver state detection by fusing eye-tracking and physiological measures. As a preliminary investigation of the added benefits of utilizing physiological data along with eye-tracking data in driver cognitive load detection, this paper explores the performance of several machine learning models in classifying three levels of cognitive load imposed on 33 drivers in a driving simulator study: no external load, lower difficulty 1-back task, and higher difficulty 2-back task. We built five machine learning models, including k-nearest neighbor, support vector machine, feedforward neural network, recurrent neural network, and random forest (RF) on (1) eye-tracking data only, (2) HR and GSR, (3) eye-tracking and HR, (4) eye-tracking and GSR, and (5) eye-tracking, HR, and GSR. Although physiological data provided 1%–15% lower classification accuracies compared with eye-tracking data, adding physiological data to eye-tracking data increased model accuracies, with an RF classifier achieving 97.8% accuracy. GSR led to a larger boost in accuracy (29.3%) over HR (17.9%), with the combination of the two factors boosting accuracy by 34.5%. Overall, utilizing both physiological and eye-tracking measures shows promise for driver state detection applications.

Vision-Based Driver’s Cognitive Load Classification Considering Eye Movement Using Machine Learning and Deep Learning

Due to the advancement of science and technology, modern cars are highly technical, more activity occurs inside the car and driving is faster; however, statistics show that the number of road fatalities have increased in recent years because of drivers’ unsafe behaviors. Therefore, to make the traffic environment safe it is important to keep the driver alert and awake both in human and autonomous driving cars. A driver’s cognitive load is considered a good indication of alertness, but determining cognitive load is challenging and the acceptance of wire sensor solutions are not preferred in real-world driving scenarios. The recent development of a non-contact approach through image processing and decreasing hardware prices enables new solutions and there are several interesting features related to the driver’s eyes that are currently explored in research. This paper presents a vision-based method to extract useful parameters from a driver’s eye movement signals and manual feature extraction based on domain knowledge, as well as automatic feature extraction using deep learning architectures. Five machine learning models and three deep learning architectures are developed to classify a driver’s cognitive load. The results show that the highest classification accuracy achieved is 92% by the support vector machine model with linear kernel function and 91% by the convolutional neural networks model. This non-contact technology can be a potential contributor in advanced driver assistive systems.

Wayfinding Information Cognitive Load Classification Based on Functional Near-Infrared Spectroscopy

AbstractAmid the rapid development of building information technologies, wayfinding information has become more accessible to building users and first responders. As a result, a realistic risk of c...

Robust cognitive load detection from wrist-band sensors

In recent years, the detection of cognitive load has received a lot of attention. Understanding the circumstances in which cognitive load occurs and reliably predicting such occurrences, offers the potential for considerable advances in the field of Human-Computer Interaction (HCI). Numerous HCI applications, ranging from medical and health-related solutions to (smart) automotive environments, would directly benefit from the reliable detection of cognitive load. However, this task still remains highly challenging. We present a machine learning (ML) approach based on ensemble learning for robust cognitive load classification. The features used by the proposed solution are generated from the interpretation of physiological measurements (e.g., heart rate, r-r interval, skin temperature, and skin response) from a wearable device. Hence, our approach consists of two steps: (1) transforming the original data into discriminative features and (2) training an ensemble model to accurately and robustly predict cognitive load. The empirical results confirm that our method has a superior performance compared to various state-of-the-art baselines on the original and transformed data. Moreover, in the open-data CogLoad@UbiComp 2020 Competition, the proposed approach achieved the best results among 17 competing approaches and outperformed all participating competitors by a considerable margin.

Objective Measures of Cognitive Load Using Deep Multi-Modal Learning

The capability of measuring human performance objectively is hard to overstate, especially in the context of the instructor and student relationship within the process of learning. In this work, we investigate the automated classification of cognitive load leveraging the aviation domain as a surrogate for complex task workload induction. We use a mixed virtual and physical flight environment, given a suite of biometric sensors utilizing the HTC Vive Pro Eye and the E4 Empatica. We create and evaluate multiple models. And we have taken advantage of advancements in deep learning such as generative learning, multi-modal learning, multi-task learning, and x-vector architectures to classify multiple tasks across 40 subjects inclusive of three subject types --- pilots, operators, and novices. Our cognitive load model can automate the evaluation of cognitive load agnostic to subject, subject type, and flight maneuver (task) with an accuracy of over 80%. Further, this approach is validated with real-flight data from five test pilots collected over two test and evaluation flights on a C-17 aircraft.

Unsupervised fNIRS feature extraction with CAE and ESN autoencoder for driver cognitive load classification

Objective. Understanding the cognitive load of drivers is crucial for road safety. Brain sensing has the potential to provide an objective measure of driver cognitive load. We aim to develop an advanced machine learning framework for classifying driver cognitive load using functional near-infrared spectroscopy (fNIRS). Approach. We conducted a study using fNIRS in a driving simulator with the N-back task used as a secondary task to impart structured cognitive load on drivers. To classify different driver cognitive load levels, we examined the application of convolutional autoencoder (CAE) and Echo State Network (ESN) autoencoder for extracting features from fNIRS. Main results. By using CAE, the accuracies for classifying two and four levels of driver cognitive load with the 30 s window were 73.25% and 47.21%, respectively. The proposed ESN autoencoder achieved state-of-art classification results for group-level models without window selection, with accuracies of 80.61% and 52.45% for classifying two and four levels of driver cognitive load. Significance. This work builds a foundation for using fNIRS to measure driver cognitive load in real-world applications. Also, the results suggest that the proposed ESN autoencoder can effectively extract temporal information from fNIRS data and can be useful for other fNIRS data classification tasks.

Cognitive Load Monitoring With Wearables–Lessons Learned From a Machine Learning Challenge

To further extend the applicability of wearable sensors, methods for accurately extracting subtle psychological information from the sensor data are required. However, accessing subjective information in everyday life, such as cognitive load, remains challenging. To bring consensus on methods for cognitive load monitoring, a machine learning challenge is organized. The participants developed machine learning methods for cognitive load classification using wrist-worn physiological sensors' data, namely heart rate, R-R intervals, skin conductance, and skin temperature. The data from subjects solving cognitive tasks of varying difficulty is used for the challenge. This article presents a systematic comparison and multi-strategic performance evaluation of the thirteen methods submitted to this challenge. A systematic comparison of preprocessing techniques, classification algorithms, and implementation techniques is presented. Performance variations for different task difficulty levels, different subjects, and different experiment periods are evaluated. The results indicate that the most robust methods used multimodal sensor data, classical classification approaches such as decision trees and support vector machines or their ensembles, and Bayesian hyperparameter optimization for hyperparameter tuning. The most accurate models used handcrafted features that are further selected using sequential backward floating search and evaluated using stratified person-aware cross-validation strategy. Moreover, the results indicated better classification performance for specific test subjects, the tasks with the highest difficulty, and in some cases, the time elapsed since the start of the experiment. This dependency is likely due to model overfitting or due to the subjective nature of the psychophysiological process. The intersubject variability in responses is challenging to be captured through objective binary labels for cognitive load, thereby warranting more sophisticated annotation approaches.

Classification of Cognitive Load Based on Neurophysiological Features From Functional Near-Infrared Spectroscopy and Electrocardiography Signals on n-Back Task

Cognitive load can be estimated using individuals' task performance, their subjective measures, and neurophysiological measures. Neurophysiological measures, which among others include brain activation signals obtained with various brain imaging techniques, such as the functional near-infrared spectroscopy (fNIRS), and signals from the peripheral physiology, such as the electrocardiography (ECG) signal, allow an objective and continuous estimation of cognitive load. In this article, the fNIRS and ECG signals were simultaneously collected from 32 participants and used to classify three levels of cognitive load on n-back task. A set of 30 fNIRS and ECG features proposed in this article enables the classification of different levels of cognitive load on n-back task using the support vector machine (SVM), k-nearest neighbors (KNN), and linear discriminant analysis (LDA) classification models. When combining the fNIRS and ECG features, three difficulties of the n-back task were classified with the mean accuracies ranging from 61% to 67%, while two difficulties were classified with the mean accuracy ranging from 70% to 84%. The most important features in the classification are discussed. The results presented in this article extend the existing empirical evidence that combining brain imaging and peripheral physiology features increases the accuracy of multi-level cognitive load classification, thus further underscoring the importance of multimodal approach to cognitive load classification.

Pattern Recognition of Cognitive Load Using EEG and ECG Signals.

The matching of cognitive load and working memory is the key for effective learning, and cognitive effort in the learning process has nervous responses which can be quantified in various physiological parameters. Therefore, it is meaningful to explore automatic cognitive load pattern recognition by using physiological measures. Firstly, this work extracted 33 commonly used physiological features to quantify autonomic and central nervous activities. Secondly, we selected a critical feature subset for cognitive load recognition by sequential backward selection and particle swarm optimization algorithms. Finally, pattern recognition models of cognitive load conditions were constructed by a performance comparison of several classifiers. We grouped the samples in an open dataset to form two binary classification problems: (1) cognitive load state vs. baseline state; (2) cognitive load mismatching state vs. cognitive load matching state. The decision tree classifier obtained 96.3% accuracy for the cognitive load vs. baseline classification, and the support vector machine obtained 97.2% accuracy for the cognitive load mismatching vs. cognitive load matching classification. The cognitive load and baseline states are distinguishable in the level of active state of mind and three activity features of the autonomic nervous system. The cognitive load mismatching and matching states are distinguishable in the level of active state of mind and two activity features of the autonomic nervous system.

Towards Intelligent Data Analytics: A Case Study in Driver Cognitive Load Classification.

One debatable issue in traffic safety research is that the cognitive load by secondary tasks reduces primary task performance, i.e., driving. In this paper, the study adopted a version of the n-back task as a cognitively loading secondary task on the primary task, i.e., driving; where drivers drove in three different simulated driving scenarios. This paper has taken a multimodal approach to perform ‘intelligent multivariate data analytics’ based on machine learning (ML). Here, the k-nearest neighbour (k-NN), support vector machine (SVM), and random forest (RF) are used for driver cognitive load classification. Moreover, physiological measures have proven to be sophisticated in cognitive load identification, yet it suffers from confounding factors and noise. Therefore, this work uses multi-component signals, i.e., physiological measures and vehicular features to overcome that problem. Both multiclass and binary classifications have been performed to distinguish normal driving from cognitive load tasks. To identify the optimal feature set, two feature selection algorithms, i.e., sequential forward floating selection (SFFS) and random forest have been applied where out of 323 features, a subset of 42 features has been selected as the best feature subset. For the classification, RF has shown better performance with F1-score of 0.75 and 0.80 than two other algorithms. Moreover, the result shows that using multicomponent features classifiers could classify better than using features from a single source.

Non-contact-based driver’s cognitive load classification using physiological and vehicular parameters

Classification of cognitive load for vehicular drivers is a complex task due to underlying challenges of the dynamic driving environment. Many previous works have shown that physiological sensor signals or vehicular data could be a reliable source to quantify cognitive load. However, in driving situations, one of the biggest challenges is to use a sensor source that can provide accurate information without interrupting diverging tasks. In this paper, instead of traditional wire-based sensors, non-contact camera and vehicle data are used that have no physical contact with the driver and do not interrupt driving. Here, four machine learning algorithms, logistic regression (LR), support vector machine (SVM), linear discriminant analysis (LDA) and neural networks (NN), are investigated to classify the cognitive load using the collected data from a driving simulator study. In this paper, physiological parameters are extracted from facial video images, and vehicular parameters are collected from controller area networks (CAN). The data collection was performed in close collaboration with industrial partners in two separate studies, in which study-1 was designed with a 1-back task and study-2 was designed with both 1-back and 2-back task. The goal of the experiment is to investigate how accurately the machine learning algorithms can classify drivers’ cognitive load based on the extracted features in complex dynamic driving environments. According to the results, for the physiological parameters extracted from the facial videos, the LR model with logistic function outperforms the other three classification methods. Here, in study-1, the achieved average accuracy for the LR classifier is 94% and in study-2 the average accuracy is 82%.In addition, the classification accuracy for the collected physiological parameters was compared with reference wire-sensor signals. It is observed that the classification accuracies between the sensor and the camera are very similar; however, better accuracy is achieved with the camera data due to having lower artefacts than the sensor data.

A physiological data‐driven model for learners' cognitive load detection using HRV‐PRV feature fusion and optimized XGBoost classification

SummaryDue to the increasing attention to online learning, cognitive load has been recently considered as a crucial indicator for judging teenagers' learning state so as to improve both learning and teaching effects. However, some traditional cognitive load measurement methods such as subjective measurement are easily influenced by subjective sensation deviation of subjects. None of them can reflect the cognitive load of learners more precisely. Recently, machine learning–based data modeling has gained more importance in the scenarios of various smart wearables and Internet of things applications. Meanwhile, physiological signals have proven to contribute much to human health assessment. On the basis of the above considerations, this paper proposes a physiological data‐driven model for learners' cognitive load detection under the application of smart wearables. The model consists of four modules: physiological signal acquisition, signal preprocessing, heart rate variability and pulse rate variability feature fusion, and cognitive load classification through an optimized extreme gradient boosting classifier in which hyperparameters are adaptively tuned with sequential model–based optimization. Furthermore, we design an experimental paradigm for signal acquisition in a learning environment, and the experimental results demonstrate that the proposed model for cognitive load detection outperforms conventional approaches that only employ either heart rate variability or pulse rate variability for modeling. We also compare the effects of different feature fusion algorithms combined with different classification algorithms, which demonstrates that the proposed model achieves the highest accuracy of cognitive load detection due to its optimal combination of feature fusion and classification.

Hemodynamic Analysis for Cognitive Load Assessment and Classification in Motor Learning Tasks Using Type-2 Fuzzy Sets

This paper addresses a novel approach to assess and classify the cognitive load of subjects from their hemodynamic response while engaged in motor learning tasks, such as vehicle driving. A set of complex motor-activity-learning stimuli for braking, steering-control, and acceleration is prepared to experimentally measure and classify the cognitive load of the car drivers in three distinct classes: high, medium, and low. New models of general and interval type-2 fuzzy classifiers are proposed to reduce the scope of uncertainty in cognitive load classification due to the fluctuation of the hemodynamic features within and across sessions. The proposed classifiers offer high classification accuracy over 96%, leaving behind the traditional type-1/type-2 fuzzy and other standard classifiers. Experiments undertaken also offer a deep biological insight concerning the shift of brain activations from the orbitofrontal to the ventrolateral prefrontal cortex during the high-to-low transition in cognitive load. Furthermore, the activation of the dorsolateral prefrontal cortex is also reduced during the low cognitive load of subjects. The proposed research work outcome may directly be utilized to identify driving learners with a low cognitive load for difficult motor learning tasks, such as taking a U-turn in a narrow space and motion control on the top of a bridge to avoid possible collision with the car ahead.