Monitoring Of Cognitive Workload Research Articles

Brainwaves in the Cloud: Cognitive Workload Monitoring Using Deep Gated Neural Network and Industrial Internet of Things

Monitoring and classifying cognitive workload in real time is vital for optimizing human–machine interactions and enhancing performance while ensuring safety, particularly in industrial scenarios. Considering this significance, the authors aim to formulate a cognitive workload monitoring system (CWMS) by leveraging the deep gated neural network (DGNN), a hybrid model integrating bi-directional long short-term memory (Bi-LSTM) and gated recurrent unit (GRU) networks. In our experimental setup, each of the four virtual users is equipped with a Raspberry Pi Zero W module to ensure efficient data transmission, thereby enhancing the reliability and efficacy of the monitoring process. This seamless monitoring framework utilizes the constrained application protocol (CoAP) and the Things Board platform to evaluate cognitive workload in real time. The most popular EEG benchmark dataset, the STEW is utilized for workload classification in this study. We employ the short-time Fourier transformation (STFT) to extract frequency bands corresponding to users in both high and low cognitive workload modes. The proposed DGNN models achieve a perfect accuracy of 99.45%, outperforming every previous state-of-the-art model. We meticulously monitored critical parameters, including latency, classification processing time, and cognitive workload levels. This research demonstrates the importance of continuous monitoring for increasing productivity and safety in industries by introducing a novel method of real-time cognitive workload monitoring. The implementation codes for each experiment are documented and made available for reproducibility.

Next Generation Computing and Communication Hub for First Responders in Smart Cities.

This paper contributes to the development of a Next Generation First Responder (NGFR) communication platform with the key goal of embedding it into a smart city technology infrastructure. The framework of this approach is a concept known as SmartHub, developed by the US Department of Homeland Security. The proposed embedding methodology complies with the standard categories and indicators of smart city performance. This paper offers two practice-centered extensions of the NGFR hub, which are also the main results: first, a cognitive workload monitoring of first responders as a basis for their performance assessment, monitoring, and improvement; and second, a highly sensitive problem of human society, the emergency assistance tools for individuals with disabilities. Both extensions explore various technological-societal dimensions of smart cities, including interoperability, standardization, and accessibility to assistive technologies for people with disabilities. Regarding cognitive workload monitoring, the core result is a novel AI formalism, an ensemble of machine learning processes aggregated using machine reasoning. This ensemble enables predictive situation assessment and self-aware computing, which is the basis of the digital twin concept. We experimentally demonstrate a specific component of a digital twin of an NGFR, a near-real-time monitoring of the NGFR cognitive workload. Regarding our second result, a problem of emergency assistance for individuals with disabilities that originated as accessibility to assistive technologies to promote disability inclusion, we provide the NGFR specification focusing on interactions based on AI formalism and using a unified hub platform. This paper also discusses a technology roadmap using the notion of the Emergency Management Cycle (EMC), a commonly accepted doctrine for managing disasters through the steps of mitigation, preparedness, response, and recovery. It positions the NGFR hub as a benchmark of the smart city emergency service.

Adaptive Human-Robotic Interaction for Robotic-assisted Surgical Settings.

Increased complexity in robotic-assisted surgical system interfaces introduces problems with human-robot collaboration that result in excessive mental workload (MWL), adversely impacting a surgeon's task performance and increasing error probability. Real-time monitoring of the operator's MWL will aid in identifying when and how interventions can be best provided to moderate MWL. In this study, an MWL-based adaptive automation system is constructed and evaluated for its effectiveness during robotic-assisted surgery. This study recruited 10 participants firstto perform surgical tasks under different cognitive workload levels. Physiological signals were obtained and employed to build a real-time system for cognitive workload monitoring. To evaluate the effectiveness of the proposed system, 15 participants were recruited to perform the surgical task with and without the proposed system. The participants' task performance and perceived workload were collected and compared. The proposed neural network model achieved an accuracy of 77.9% in cognitive workload classification. In addition, better task performance and lower perceived workload were observed when participants completed the experimental task under the task condition supplemented with adaptive aiding using the proposed system. The proposed MWL monitoring system successfully diminished the perceived workload of participants and increased their task performance under high-stress conditions via interventions by a semi-autonomous suction tool. The preliminary results from the comparative study show the potential impact of automated adaptive aiding systems in enhancing surgical task performance via cognitive workload-triggered interventions in robotic-assisted surgery.

Developing cognitive workload and performance evaluation models using functional brain network analysis

Cognition, defined as the ability to learn, remember, sustain attention, make decisions, and solve problems, is essential in daily activities and in learning new skills. The purpose of this study was to develop cognitive workload and performance evaluation models using features that were extracted from Electroencephalogram (EEG) data through functional brain network and spectral analyses. The EEG data were recorded from 124 brain areas of 26 healthy participants conducting two cognitive tasks on a robot simulator. The functional brain network and Power Spectral Density features were extracted from EEG data using coherence and spectral analyses, respectively. Participants reported their perceived cognitive workload using the SURG-TLX questionnaire after each exercise, and the simulator generated actual performance scores. The extracted features, actual performance scores, and subjectively assessed cognitive workload values were used to develop linear models for evaluating performance and cognitive workload. Furthermore, the Pearson correlation was used to find the correlation between participants’ age, performance, and cognitive workload. The findings demonstrated that combined EEG features retrieved from spectral analysis and functional brain networks can be used to evaluate cognitive workload and performance. The cognitive workload in conducting only Matchboard level 3, which is more challenging than Matchboard level 2, was correlated with age (0.54, p-value = 0.01). This finding may suggest playing more challenging computer games are more helpful in identifying changes in cognitive workload caused by aging. The findings could open the door for a new era of objective evaluation and monitoring of cognitive workload and performance.

Assessing Cognitive Workload Using Cardiovascular Measures and Voice.

Monitoring cognitive workload has the potential to improve both the performance and fidelity of human decision making. However, previous efforts towards discriminating further than binary levels (e.g., low/high or neutral/high) in cognitive workload classification have not been successful. This lack of sensitivity in cognitive workload measurements might be due to individual differences as well as inadequate methodology used to analyse the measured signal. In this paper, a method that combines the speech signal with cardiovascular measurements for screen and heartbeat classification is introduced. For validation, speech and cardiovascular signals from 97 university participants and 20 airline pilot participants were collected while cognitive stimuli of varying difficulty level were induced with the Stroop colour/word test. For the trinary classification scheme (low, medium, high cognitive workload) the prominent result using classifiers trained on each participant achieved 15.17 ± 0.79% and 17.38 ± 1.85% average misclassification rates indicating good discrimination at three levels of cognitive workload. Combining cardiovascular and speech measures synchronized to each heartbeat and consolidated with short-term dynamic measures might therefore provide enhanced sensitivity in cognitive workload monitoring. The results show that the influence of individual differences is a limiting factor for a generic classification and highlights the need for research to focus on methods that incorporate individual differences to achieve even better results. This method can potentially be used to measure and monitor workload in real time in operational environments.

Machine-Learning Based Monitoring of Cognitive Workload in Rescue Missions With Drones.

In search and rescue missions, drone operations are challenging and cognitively demanding. High levels of cognitive workload can affect rescuers' performance, leading to failure with catastrophic outcomes. To face this problem, we propose a machine learning algorithm for real-time cognitive workload monitoring to understand if a search and rescue operator has to be replaced or if more resources are required. Our multimodal cognitive workload monitoring model combines the information of 25 features extracted from physiological signals, such as respiration, electrocardiogram, photoplethysmogram, and skin temperature, acquired in a noninvasive way. To reduce both subject and day inter-variability of the signals, we explore different feature normalization techniques, and introduce a novel weighted-learning method based on support vector machines suitable for subject-specific optimizations. On an unseen test set acquired from 34 volunteers, our proposed subject-specific model is able to distinguish between low and high cognitive workloads with an average accuracy of 87.3% and 91.2% while controlling a drone simulator using both a traditional controller and a new-generation controller, respectively.

Real-Time EEG-Based Cognitive Workload Monitoring on Wearable Devices.

Cognitive workload monitoring (CWM) can enhance human-machine interaction by supporting task execution assistance considering the operator's cognitive state. Therefore, we propose a machine learning design methodology and a data processing strategy to enable CWM on resource-constrained wearable devices. Our CWM solution is built upon edge computing on a simple wearable system, with only four peripheral channels of electroencephalography (EEG). We assess our solution on experimental data from 24 volunteers. Moreover, to overcome the system's memory constraints, we adopt an optimization strategy for model size reduction and a multi-batch data processing scheme for optimizing RAM memory footprint. Finally, we implement our data processing strategy on a state-of-the-art wearable platform and assess its execution and system battery life. We achieve an accuracy of 74.5% and a 74.0% geometric mean between sensitivity and specificity for CWM classification on unseen data. Besides, the proposed model optimization strategy generates a 27.5x smaller model compared to the one generated with default parameters, and the multi-batch data processing scheme reduces RAM memory footprint by 14x compared to a single batch data processing. Finally, our algorithm uses only 1.28% of the available processing time, thus, allowing our system to achieve 28.5 hours of battery life. We provide a reliable and optimized CWM solution using wearable devices, enabling more than a day of operation on a single battery charge. The proposed methodology enables real-time data processing on resource-constrained devices and supports real-time wearable monitoring based on EEG for applications as CWM in human-machine interaction.

MBioTracker: Multimodal Self-Aware Bio-Monitoring Wearable System for Online Workload Detection.

Cognitive workload affects operators' performance principally in high-risk or time-demanding situations and when multitasking is required. An online cognitive workload monitoring system can provide valuable inputs to decision-making instances, such as the operator's state of mind and resulting performance. Therefore, it can allow potential adaptive support to the operator. This work presents a new design of a wearable embedded system for online cognitive workload monitoring. This new wearable system consists of, on the hardware side, a multi-channel physiological signals acquisition (respiration cycles, heart rate, skin temperature, and pulse waveform) and a low-power processing platform. Further, on the software side, our wearable embedded system includes a novel energy-aware bio-signal processing algorithm. We also use the concept of application self-awareness to enable energy-scalable embedded machine learning algorithms and methods for online subjects' cognitive workload monitoring. Our results show that this new wearable system can continuously monitor multiple bio-signals, compute their key features, and provide reliable detection of high and low cognitive workload levels with a time resolution of 1 minute and a battery lifetime of 14.58 h in our experimental conditions. It achieves a detection accuracy of 76.6% (2.6% lower than analogous offline computer-based analysis) with a sensitivity of 77.04% and a specificity of 81.75%, on a simulated drone rescue mission task. Moreover, by applying our self-aware monitoring to exploit different energy-scalable modes, we can increase battery lifetime by 51.6% (up to 22.11 hours) while incurring an insignificant accuracy loss of 1.07%.

The Impacts of Temporal Variation and Individual Differences in Driver Cognitive Workload on ECG-Based Detection.

This paper aimed to investigate the robustness of driver cognitive workload detection based on electrocardiogram (ECG) when considering temporal variation and individual differences in cognitive workload. Cognitive workload is a critical component to be monitored for error prevention in human-machine systems. It may fluctuate instantaneously over time even in the same tasks and differ across individuals. A driving simulation study was conducted to classify driver cognitive workload underlying four experimental conditions (baseline, N-back, texting, and N-back + texting distraction) in two repeated 1-hr blocks. Heart rate (HR) and heart rate variability (HRV) were compared among the experimental conditions and between the blocks. Random forests were built on HR and HRV to classify cognitive workload in different blocks and for different individuals. HR and HRV were significantly different between repeated blocks in the study, demonstrating the time-induced variation in cognitive workload. The performance of cognitive workload classification across blocks and across individuals was significantly improved after normalizing HR and HRV in each block by the corresponding baseline. The temporal variation and individual differences in cognitive workload affects ECG-based cognitive workload detection. But normalization approaches relying on the choice of appropriate baselines help compensate for the effects of temporal variation and individual differences. The findings provide insight into the value and limitations of ECG-based driver cognitive workload monitoring during prolonged driving for individual drivers.

Sensor Location Optimization of Wireless Wearable fNIRS System for Cognitive Workload Monitoring Using a Data-Driven Approach for Improved Wearability.

Functional Near-Infrared Spectroscopy (fNIRS) is a hemodynamic modality in human cognitive workload assessment receiving popularity due to its easier implementation, non-invasiveness, low cost and other benefits from the signal-processing point of view. Wearable wireless fNIRS systems used in research have promisingly shown that fNIRS could be used in cognitive workload assessment in out-of-the-lab scenarios, such as in operators’ cognitive workload monitoring. In such a scenario, the wearability of the system is a significant factor affecting user comfort. In this respect, the wearability of the system can be improved if it is possible to minimize an fNIRS system without much compromise of the cognitive workload detection accuracy. In this study, cognitive workload-related hemodynamic changes were acquired using an fNIRS system covering the whole forehead, which is the region of interest in most cognitive workload-monitoring studies. A machine learning approach was applied to explore how the mean accuracy of the cognitive workload classification accuracy varied across various sensing locations on the forehead such as the Left, Mid, Right, Left-Mid, Right-Mid and Whole forehead. The statistical significance analysis result showed that the Mid location could result in significant cognitive workload classification accuracy compared to Whole forehead sensing, with a statistically insignificant difference in the mean accuracy. Thus, the wearable fNIRS system can be improved in terms of wearability by optimizing the sensor location, considering the sensing of the Mid location on the forehead for cognitive workload monitoring.

A Review of Measuring the Cognitive Workload of Electronic Health Records.

The To Err Is Human report stated that 98 000 patients die yearly because of medical errors, and that medication errors kill more people than workplace injuries. The inadequate design and utilization of the electronic health record have been identified as major contributing factors to medical errors. Increased cognitive workload of clinicians has consistently been linked to the occurrence of medical errors. The purpose of this article was to synthesize the current state of the science on measuring clinicians' cognitive workload associated with using electronic health records in order to inform evidence-based guidelines. The major considerations identified in the literature involve the use of psychometric instruments, using efficiency as a proxy for cognitive workload, and eye tracking. The National Aeronautics and Space Administration Task Load Index was the most used psychometric instrument, but reliability measures were not reported. It is important to evaluate reliability of psychometric instruments because the consistency of the instrument can change when administered to different populations. Efficiency is an observable measure defined by the total time to complete a task and the total number of physical interactions with the user interface. Efficiency can allow the use of statistical modeling, but it does not directly evaluate the mental activity associated with using an electronic health record interface. Eye tracking has been used extensively in the literature to measure cognitive workload via changes in pupil size related to mental activity, but it is not often used to measure the cognitive workload associated with using the electronic health record. Eye tracking is very useful for continuous monitoring of cognitive workload.

The role of working memory capacity in cardiovascular monitoring of cognitive workload

Cardiovascular measures have been found to be sensitive to task onset and offset, but are less sensitive to adjacent levels of increasing cognitive workload. A potential confound in the literature is the disregard of individual differences in cardiovascular reactivity. In particular, the individuals‘ working memory capacity (WMC) is likely to play a role in cardiovascular reactivity to workload. A total of 98 university students performed four cognitive tasks that varied in their level of workload. The operation span (OSPAN) task was used to measure the participants‘ WMC. A variety of cardiovascular measures were gathered in real time during the experiment. Derived measures of blood pressure regulation were also calculated. In line with what was hypothesized, cardiovascular measures detected workload onset and offset but did not consistently distinguish between the individual task levels. Furthermore, a significant interaction between workload levels and WMC showed that cardiovascular profile varied depending on WMC scores. In addition, WMC negatively predicted subjective ratings of task difficulty as well as task performance, with subjective estimation of task difficulty and error increasing as WMC decreased. The results suggest that WMC may play a critical role in determining how individuals react to increased cognitive workload.

Monitoring Cognitive Workload Using Vocal Tract and Voice Source Features

Monitoring cognitive workload from speech signals has received much attention from researchers in the past few years as it has the potential to improve performance and fidelity in human decision making. The bulk of the research has focused on classifying speech from talkers participating in cognitive workload experiments using simple reading tasks, memory span tests and the Stroop test, typically into three levels of low, medium and high cognitive workload. This study focuses on using parameters extracted from the vocal tract and the voice source components of the speech signal for cognitive workload monitoring. The experiment used in this study contains 98 participants, the levels were obtained by using a reading task and three Stroop tasks which were randomly ordered for each participant and an adequate rest time was used inbetween tasks to mitigate the effect of cognitive workload from one task affecting the subsequent one. Vocal tract features were obtained from the first three formants and voice source features were extracted using signal analysis on the inverse filtered speech signal. The results show that on their own, the vocal tract features outperform the voice source features. The MCR of 33.92% ± 1.05 was achieved with a SVM classifier. A weighted combination of vocal tract and voice source features classified with SWM classifier fused at the output level achieved the lowest MCR of 32.5%.

Combining EEG with Pupillometry to Improve Cognitive Workload Detection

Combining signals from traditional electroencephalography with those of novel physiological measures such as pupil dilation could enable more accurate and robust real-time monitoring of cognitive workload.

Brain enhancement through cognitive training: a new insight from brain connectome.

Owing to the recent advances in neurotechnology and the progress in understanding of brain cognitive functions, improvements of cognitive performance or acceleration of learning process with brain enhancement systems is not out of our reach anymore, on the contrary, it is a tangible target of contemporary research. Although a variety of approaches have been proposed, we will mainly focus on cognitive training interventions, in which learners repeatedly perform cognitive tasks to improve their cognitive abilities. In this review article, we propose that the learning process during the cognitive training can be facilitated by an assistive system monitoring cognitive workloads using electroencephalography (EEG) biomarkers, and the brain connectome approach can provide additional valuable biomarkers for facilitating leaners’ learning processes. For the purpose, we will introduce studies on the cognitive training interventions, EEG biomarkers for cognitive workload, and human brain connectome. As cognitive overload and mental fatigue would reduce or even eliminate gains of cognitive training interventions, a real-time monitoring of cognitive workload can facilitate the learning process by flexibly adjusting difficulty levels of the training task. Moreover, cognitive training interventions should have effects on brain sub-networks, not on a single brain region, and graph theoretical network metrics quantifying topological architecture of the brain network can differentiate with respect to individual cognitive states as well as to different individuals’ cognitive abilities, suggesting that the connectome is a valuable approach for tracking the learning progress. Although only a few studies have exploited the connectome approach for studying alterations of the brain network induced by cognitive training interventions so far, we believe that it would be a useful technique for capturing improvements of cognitive functions.