Adaptive Aiding Research Articles

Adaptive Human-Robotic Interaction for Robotic-assisted Surgical Settings.

Increased complexity in robotic-assisted surgical system interfaces introduces problems with human-robot collaboration that result in excessive mental workload (MWL), adversely impacting a surgeon's task performance and increasing error probability. Real-time monitoring of the operator's MWL will aid in identifying when and how interventions can be best provided to moderate MWL. In this study, an MWL-based adaptive automation system is constructed and evaluated for its effectiveness during robotic-assisted surgery. This study recruited 10 participants firstto perform surgical tasks under different cognitive workload levels. Physiological signals were obtained and employed to build a real-time system for cognitive workload monitoring. To evaluate the effectiveness of the proposed system, 15 participants were recruited to perform the surgical task with and without the proposed system. The participants' task performance and perceived workload were collected and compared. The proposed neural network model achieved an accuracy of 77.9% in cognitive workload classification. In addition, better task performance and lower perceived workload were observed when participants completed the experimental task under the task condition supplemented with adaptive aiding using the proposed system. The proposed MWL monitoring system successfully diminished the perceived workload of participants and increased their task performance under high-stress conditions via interventions by a semi-autonomous suction tool. The preliminary results from the comparative study show the potential impact of automated adaptive aiding systems in enhancing surgical task performance via cognitive workload-triggered interventions in robotic-assisted surgery.

Optimizing aid activation in adaptive and non-adaptive aiding systems: A framework for design and validation

Development of adaptive aids to support human performance in complex systems is a cornerstone of human factors. Research in this area has led to a diversity of ideas regarding potential activation methods. However, little guidance has been provided on how to select among aid activation methods, and this lack of guidance could hinder adaptive aid development and deployment. Within the current paper, we review available methods of aid activation and describe a process for developing and validating adaptive aiding systems. We focus on supporting system designers who wish to select the ideal aid activation method for an intended application. The process that we recommend is an empirical approach to evaluate the feasibility, costs, and benefits of various potential methods of aid activation. This methodological framework will support practitioners making critical decisions about the design of aiding systems.

Cross-Task Consistency of Electroencephalography-Based Mental Workload Indicators: Comparisons Between Power Spectral Density and Task-Irrelevant Auditory Event-Related Potentials.

Mental workload (MWL) estimators based on ongoing electroencephalography (EEG) and event-related potentials (ERPs) have shown great potentials to build adaptive aiding systems for human–machine systems by estimating MWL in real time. However, extracting EEG features which are consistent in indicating MWL across different tasks is still one of the critical challenges. This study attempts to compare the cross-task consistency in indexing MWL variations between two commonly used EEG-based MWL indicators, power spectral density (PSD) of ongoing EEG and task-irrelevant auditory ERPs (tir-aERPs). The verbal N-back and the multi-attribute task battery (MATB), both with two difficulty levels, were employed in the experiment, along with task-irrelevant auditory probes. EEG was recorded from 17 subjects when they were performing the tasks. The tir-aERPs elicited by the auditory probes and the relative PSDs of ongoing EEG between two consecutive auditory probes were extracted and statistically analyzed to reveal the effects of MWL and task type. Discriminant analysis and support vector machine were employed to examine the generalization of tir-aERP and PSD features in indexing MWL variations across different tasks. The results showed that the amplitudes of tir-aERP components, N1, early P3a, late P3a, and the reorienting negativity, significantly decreased with the increasing MWL in both N-back and MATB. Task type had no obvious influence on the amplitudes and topological layout of the MWL-sensitive tir-aERP features. The relative PSDs in θ, α, and low β bands were also sensitive to MWL variations. However, the MWL-sensitive PSD features and their topological patterns were significantly affected by task type. The cross-task classification results based on tir-aERP features also significantly outperformed the PSD features. These results suggest that the tir-aERPs should be potentially more consistent MWL indicators across very different task types when compared to PSD. The current study may provide new insights to our understanding of the common and distinctive neuropsychological essences of MWL across different tasks.

Using Functional Near Infrared Spectroscopy to Assess Cognitive Workload

Quantification of mental workload is a significant aspect of monitoring and adaptive aiding systems that are intended to improve the efficiency and safety of human–machine systems. Functional near Infrared (fNIR) spectroscopy is a field-deployable brain monitoring device that provides a measures of cerebral hemodynamic within the prefrontal cortex. The purpose of this study was to assess the cognitive load by using Performance (reaction time), Behavioral metrics (NASA TLX) and Neuro-Cognitive Measures (Hemodynamic response). To observe the activation in prefrontal cortex, we employed Functional Near Infrared (fNIR) Spectroscopy with a Standard Stroop task. A total of 25 healthy participants (N 18 Male and N 07 Female, M Age 25.5 SD 7.6), participated in the study. For statistical analysis, a repeated measure t-test was computed to compare the Oxy (Δ[HbO2]) and De-Oxy (Δ[hHb]) changes under Congruent and In-Congruent task conditions. For Classification, Binary logistic regression model applied to identify how accurately classifying the varied workload conditions. The finding shows that fNIR measures had adequate predictive power for estimating task performance in workload conditions. In this paper, we have found evidence that fNIR can be used as indicator of cognitive load which is important for optimal human performance.

Task-irrelevant Auditory Event-related Potentials as Mental Workload Indicators: A Between-task Comparison Study.

Real-time monitoring of mental workload (MWL) is a crucial step to build closed-loop adaptive aiding systems for human-machine systems. MWL estimators based on spontaneous electroencephalography (EEG) and event-related potentials (ERPs) have shown great potentials to achieve this goal. However, the previous studies show that the between-task robustness of these EEG/ERP-based MWL estimators is still an unsolved intractable question. This study attempts to examine the task-irrelevant auditory event-related potentials (tir-aERPs) as MWL indicators. A working memory task (verbal n-back) and a visuo-motor task (multi-attribute task battery, MATB), both with two difficulty levels (easy and hard), were used in the experiment, along with task-irrelevant auditory probes that did not need any response from the participants. EEG was recorded from ten participants when they were performing the tasks. The tir-aERPs elicited by the auditory probes were extracted and analyzed. The results show that the amplitudes of N1, early P3a (eP3a) and the late reorienting negativity (RON) significantly decreased with the increasing MWL in both n-back and MATB. Task type has no obvious influence on the amplitudes and topological layout of the MWL-sensitive tir-aERPs features. These results suggest that the tir-aERPs are potentially more constant MWL indicators across very different task types. Therefore, the tir-aERPs should be taken into consideration in future task-independent MWL monitoring studies.

Adaptive aiding with an individualized workload model based on psychophysiological measures

Potential benefits of technology such as automation are oftentimes negated by improper use and application. Adaptive systems provide a means to calibrate the use of technological aids to the operator’s state, such as workload state, which can change throughout the course of a task. Such systems require a workload model which detects workload and specifies the level at which aid should be rendered. Workload models that use psychophysiological measures have the advantage of detecting workload continuously and relatively unobtrusively, although the inter-individual variability in psychophysiological responses to workload is a major challenge for many models. This study describes an approach to workload modeling with multiple psychophysiological measures that was generalizable across individuals, and yet accommodated inter-individual variability. Under this approach, several novel algorithms were formulated. Each of these underwent a process of evaluation which included comparisons of the algorithm’s performance to an at-chance level, and assessment of algorithm robustness. Further evaluations involved the sensitivity of the shortlisted algorithms at various threshold values for triggering an adaptive aid.

Change detection support for supervisory controllers of highly automated systems: Effects on performance, mental workload, and recovery of situation awareness following interruptions

Dynamic Positioning (DP) is a computer-controlled process to automatically keep a floating vessel at a specific position or to follow a pre-defined path (tracking) by using its own propellers and thrusters. The human supervisory controller has no direct need to constantly know what the status is of all parts of the automation and the system it is controlling, because the highly automated DPS is controlling all components itself. Only after a failure arises, the operator needs to take over manual control and take appropriate action(s) to prevent the failure from harming the operation. As the supervisory controller may be out of the loop, swiftly taking over control may be problematic when failures arise. The purpose of the current study was to investigate whether automation of change detection enables human operators with low awareness of the automation and the system it is controlling to quickly recover awareness in emergency take-over situations. A 2 by 2 within subjects experiment was conducted using a DP simulation (n = 22). Within-subjects factors were support (Yes, No) and interruption (Yes, No). Results showed that change detection support helps in the process of recovering situation awareness after it has been reduced, due to an interruption of the primary task of overseeing the automation. Interestingly, support was not beneficial to the participants in all conditions. In non-interrupted conditions the support unexpectedly resulted in higher workload, raising questions whether supervisory controllers should be supported continuously or only when it is required. Relevance to industry: The results show that change detection support has potential value in operational maritime environments, especially in situations where the DP operator has low situation awareness. Future research should investigate whether adaptive aiding could alleviate some of the negative effects of non-adaptive operator support in maritime environments.

Cognitive and Affective Probing for Neuroergonomics

Cognitive and Affective Probing for Neuroergonomics

Personalized Driver Workload Inference by Learning From Vehicle Related Measurements

Adapting in-vehicle systems (e.g., advanced driver assistance systems and in-vehicle information systems) to individual drivers’ workload can enhance both safety and convenience. To make this possible, it is a prerequisite to infer driver workload so that adaptive aiding can be provided to the driver at the right time and in an appropriate manner. Rather than developing an average model for all drivers, a personalized driver workload inference (PDWI) system considering individual drivers driving characteristics is developed using machine learning techniques via easily accessed vehicle related measurements (VRMs). The proposed PDWI system comprises two stages. In offline training, individual drivers workload is first automatically splitted into different categories according to its inherent data characteristics using fuzzy C-means (FCM) clustering. Then an implicit mapping between VRMs and different levels of workload is constructed via classification algorithms. In online implementation, VRMs samples are classified into different clusters, consequently driver workload type can be successfully inferred. A recently collected dataset from real-world naturalistic driving experiments is drawn to validate the proposed PDWI system. Comparative experimental results indicate that the proposed framework integrating FCM clustering and support vector machine classifier provides a promising workload recognition performance in terms of accuracy, precision, recall, ${F_{1}}$ -score, and prediction time. The interindividual differences in term of workload are also identified and can be accommodated by the proposed framework due to its adaptiveness.

Using physiological signals to measure operator’s mental workload in shipping – an engine room simulator study

ABSTRACTMental workload (MWL) is one of core elements of human factor construct reflecting arousal level, and its optimisation is crucial to maintain favourable operator functional state. However, sensible, reliable, and diagnostic measurement of MWL is essential for applications of adaptive aiding system design, usability testing, and seafarers’ training. To develop robust MWL measures, 10 participants voluntarily participated in a simulator-based experiment study. During this study, the participants carried out standard four-level calibration tasks and simulated four-level maritime operation tasks, their heart rate and electroencephalogram (EEG) were continuously measured using a heart rate sensor and an ambulatory EEG device, which includes an accelerometer to distinguish signal corruption epochs induced by body movement artefact. After each task, NASA Task Load Index was collected as subjective measurement. One-way analysis of variance was used to test the sensitivity of MWL measures and Pearson’s correlation coefficients were calculated. The significantly sensitive indices for n-back task and simulator-based maritime operation task were different, supporting the limited cognitive resource pool theory. EEG features showed higher sensitivity than heart-rate-related measures.

Efficient and Robust Pupil Size and Blink Estimation From Near-Field Video Sequences for Human–Machine Interaction

Monitoring pupil and blink dynamics has applications in cognitive load measurement during human-machine interaction. However, accurate, efficient, and robust pupil size and blink estimation pose significant challenges to the efficacy of real-time applications due to the variability of eye images, hence to date, require manual intervention for fine tuning of parameters. In this paper, a novel self-tuning threshold method, which is applicable to any infrared-illuminated eye images without a tuning parameter, is proposed for segmenting the pupil from the background images recorded by a low cost webcam placed near the eye. A convex hull and a dual-ellipse fitting method are also proposed to select pupil boundary points and to detect the eyelid occlusion state. Experimental results on a realistic video dataset show that the measurement accuracy using the proposed methods is higher than that of widely used manually tuned parameter methods or fixed parameter methods. Importantly, it demonstrates convenience and robustness for an accurate and fast estimate of eye activity in the presence of variations due to different users, task types, load, and environments. Cognitive load measurement in human-machine interaction can benefit from this computationally efficient implementation without requiring a threshold calibration beforehand. Thus, one can envisage a mini IR camera embedded in a lightweight glasses frame, like Google Glass, for convenient applications of real-time adaptive aiding and task management in the future.

Coadaptive Aiding and Automation Enhance Operator Performance

In this work, we expand on the theory of adaptive aiding by measuring the effectiveness of coadaptive aiding, wherein we explicitly allow for both system and user to adapt to each other. Adaptive aiding driven by psychophysiological monitoring has been demonstrated to be a highly effective means of controlling task allocation and system functioning. Psychophysiological monitoring is uniquely well suited for coadaptation, as malleable brain activity may be used as a continuous input to the adaptive system. To establish the efficacy of the coadaptive system, physiological activation of adaptation was directly compared with manual activation or no activation of the same automation and cuing systems. We used interface adaptations and automation that are plausible for real-world operations, presented in the context of a multi-remotely piloted aircraft control simulation. Each participant completed 3 days of testing during 1 week. Performance was assessed via proportion of targets successfully engaged. In the first 2 days of testing, there were no significant differences in performance between the conditions. However, in the third session, physiological adaptation produced the highest performance. By extending the data collection across multiple days, we offered enough time and repeated experience for user adaptation as well as online system adaptation, hence demonstrating coadaptive aiding. The results of this work may be employed to implement more effective adaptive workstations in a variety of work domains.

Classifying human operator functional state based on electrophysiological and performance measures and fuzzy clustering method.

The human operator's ability to perform their tasks can fluctuate over time. Because the cognitive demands of the task can also vary it is possible that the capabilities of the operator are not sufficient to satisfy the job demands. This can lead to serious errors when the operator is overwhelmed by the task demands. Psychophysiological measures, such as heart rate and brain activity, can be used to monitor operator cognitive workload. In this paper, the most influential psychophysiological measures are extracted to characterize Operator Functional State (OFS) in automated tasks under a complex form of human-automation interaction. The fuzzy c-mean (FCM) algorithm is used and tested for its OFS classification performance. The results obtained have shown the feasibility and effectiveness of the FCM algorithm as well as the utility of the selected input features for OFS classification. Besides being able to cope with nonlinearity and fuzzy uncertainty in the psychophysiological data it can provide information about the relative importance of the input features as well as the confidence estimate of the classification results. The OFS pattern classification method developed can be incorporated into an adaptive aiding system in order to enhance the overall performance of a large class of safety-critical human-machine cooperative systems.

A Human Cognitive Performance Measure Based on Available Options for Adaptive Aiding

The concept of adaptive automation stipulates dynamically changing levels of automation as solution for the conflict between advantages offered by different static levels of automation. The use of a task performance measure to invoke adaptive automation enables adapting not only to the current situation but additionally to the specific needs of different operators. This paper presents an approach to measure the task performance by comparing the observed interaction behavior with the options available. This takes account of the fact that the achievable result strongly depends on the current situation and the available interaction options. The presented approach is based on a Coloured-Petri-Net (CPN) simulation environment and generates action sequences in the Petri Nets state space which represent the consequences of alternative options. Thus, evaluating operators' actions taking into account the available options becomes possible. This paper further discusses the advantages and disadvantages of the proposed approach regarding its application for adaptive automation. On the one hand, the method is sensitive to the environmental conditions and is able to measure the performance at any time. On the other hand, a drawback of the method is its high computational demand so that it has been used only for retrospective analyses up to now.

Erratum

de Visser, E., & Parasuraman, R. (2011). Adaptive aiding of human-robot teaming: Effects of imperfect automation on performance, trust, and workload. Journal of Cognitive Engineering and Decision Making, 5: 209–231. (Original DOI: 10.1177/1555343411410160).

Optical brain monitoring for operator training and mental workload assessment

An accurate measure of mental workload in human operators is a critical element of monitoring and adaptive aiding systems that are designed to improve the efficiency and safety of human-machine systems during critical tasks. Functional near infrared (fNIR) spectroscopy is a field-deployable non-invasive optical brain monitoring technology that provides a measure of cerebral hemodynamics within the prefrontal cortex in response to sensory, motor, or cognitive activation. In this paper, we provide evidence from two studies that fNIR can be used in ecologically valid environments to assess the: 1) mental workload of operators performing standardized (n-back) and complex cognitive tasks (air traffic control--ATC), and 2) development of expertise during practice of complex cognitive and visuomotor tasks (piloting unmanned air vehicles--UAV). Results indicate that fNIR measures are sensitive to mental task load and practice level, and provide evidence of the fNIR deployment in the field for its ability to monitor hemodynamic changes that are associated with relative cognitive workload changes of operators. The methods reported here provide guidance for the development of strategic requirements necessary for the design of complex human-machine interface systems and assist with assessments of human operator performance criteria.

Adaptive Aiding of Human-Robot Teaming

In many emerging civilian and military operations, human operators are increasingly being tasked to supervise multiple robotic uninhabited vehicles (UVs) with the support of automation. As 100% automation reliability cannot be assured, it is important to understand the effects of automation imperfection on performance. In addition, adaptive aiding may help counter any adverse effects of static (fixed) automation. Using a high-fidelity multi-UV simulation involving both air and ground vehicles, two experiments examined the effects of automation reliability and adaptive automation on human-system performance with different levels of task load. In Experiment 1, participants performed a reconnaissance mission while assisted with an automatic target recognition (ATR) system whose reliability was low, medium, or high. Overall human-robot team performance was higher than with either human or ATR performance alone. In Experiment 2, participants performed a similar reconnaissance mission with no ATR, static automation, or with adaptive automation keyed to task load. Participant trust and self-confidence were higher and workload was lower for adaptive automation compared with the other conditions. The results show that human-robot teams can benefit from imperfect static automation even in high task load conditions and that adaptive automation can provide additional benefits in trust and workload.

Using Transcranial Doppler Sonography to Measure Cognitive Load in a Command and Control Task

Previous work has explored the possibility of measuring the functional state of the operator to drive the implementation of physiological adaptive aiding. Transcranial Doppler Sonography (TCD), which has shown promise as an index of cognitive resource utilization in vigilance or sustained attention tasks, may provide a time and cost efficient alternative to traditional measures used to assess operator functional state. In the current study, participants performed a command and control simulation under varying levels of task load: a low task load condition in which enemy threats incurred at a steady pace, and a high workload trial in which the number of enemy threats increased unpredictably at two points within the scenario. Reaction time to engage and destroy enemies, and the efficiency of protection of a no-fly zone, were superior in the low than in the high load condition. Furthermore, an automated decision aid facilitated better performance in both task load conditions. As the demands of the task increased unpredictably in the high task load condition, cerebral blood flow velocity (CBFV) increased in a similar manner for the first task load transition, but not for the second. Results suggest that the TCD measure may be useful in monitoring the dynamic changes of operator workload in unpredictable environments, but additional studies are needed to validate its use for physiologically-driven adaptive automation.

Adaptive Inter-Loop Aiding for Performance Improvements in Low CNIR Environments

Use of inter-loop aiding for improving tracking performance has been widely researched in recent years. However, most of the previously proposed aiding schemes rely on the assumption that the aiding loop remains unaffected by received interference. This paper argues that this may not always be the case. It is likely that the performance of the aiding loop may also degrade in the case where interference is received at the aiding carrier’s frequency resulting in performance degradation of both aiding and aided loops. This paper proposes an aiding scheme that offers performance improvements in case interference corrupts both frequencies. Also, an algorithm is proposed that continuously updates the aided loop bandwidth to keep its jitter at a minimum. A relationship between the quality of the aiding signal and its effects on the performance of the aided loop is analysed. An adaptive Kalman filter-based implementation of the aiding architecture is proposed to improve the quality of aiding estimates. This implementation offers an improved margin against received interference. Simulation and real data results are presented that show improvements of 7 and 5 dB-Hz in this margin by employing the proposed aiding scheme with and without an adaptive Kalman filter.

Design Principles for Adaptive Automation and Aiding

Adaptive automation technologies and the associated research have advanced significantly since their inception in the early 1970s. Here, we present the history of this research and the lessons that have been learned from it. We also present several design guidelines for the successful implementation of adaptive automation in order to serve the community of designers who will be implementing adaptive automation.