Adaptive Automation Research Articles

Verification and Validation of Cognitive Workload Models for Adaptive Automation Tasks

Technological advances that seek to address future operational challenges abound. While advanced capabilities are being developed, there is an important space for human design considerations, including cognitive workload. One proposed solution to improve cognitive workload management is adaptive automation (AA). This research used a novel, model-based approach to assess the impacts of AA on cognitive workload. This assessment modeled the tasks in NASA’s Multi-attribute Task Battery-II (MATB-II) using the Improved Performance Research Integration Tool (IMPRINT). The effort sought to investigate the relationship of cognitive workload, situation awareness, and performance through three human-in-the-loop studies with 120 participants using MATB-II. The research also attempted to validate cognitive workload models from IMPRINT. The IMPRINT models were representative of the MATB tasks with statistically significant differences between workload conditions, which mirrored the models’ predictions. The results demonstrated that AA system task models can be developed using IMPRINT to provide design recommendations.

Measuring aviator workload using EEG: an individualized approach to workload manipulation.

Measuring an operator's physiological state and using that data to predict future performance decrements has been an ongoing goal in many areas of transportation. Regarding Army aviation, the realization of such an endeavor could lead to the development of an adaptive automation system which adapts to the needs of the operator. However, reaching this end state requires the use of experimental scenarios similar to real-life settings in order to induce the state of interest that are able to account for individual differences in experience, exposure, and perception to workload manipulations. In the present study, we used an individualized approach to manipulating workload in order to account for individual differences in response to workload manipulations, while still providing an operationally relevant flight experience. Eight Army aviators participated in the study, where they completed two visits to the laboratory. The first visit served the purpose of identifying individual workload thresholds, with the second visit resulting in flights with individualized workload manipulations. EEG data was collected throughout both flights, along with subjective ratings of workload and flight performance. Both EEG data and workload ratings suggested a high workload. Subjective ratings were higher during the high workload flight compared to the low workload flight (p < 0.001). Regarding EEG, frontal alpha (p = 0.04) and theta (p = 0.01) values were lower and a ratio of beta/(alpha+theta) (p = 0.02) were higher in the baseline flight scenario compared to the high workload scenario. Furthermore, the data were compared to that collected in previous studies which used a group-based approach to manipulating workload. The individualized method demonstrated higher effect sizes in both EEG and subjective ratings, suggesting the use of this method may provide a more reliable way of producing high workload in aviators.

Разработка системы управления натяжным устройством ленточных конвейеров

Belt conveyors are one of the most common conveying machines with high productivity, economy, ease of maintenance and reliability. The range of their application is very wide and therefore the improvement of their operational characteristics is an urgent task. The tensioning device of the belt conveyor is accepted as an object of modernization. The essence lies in the development of adaptive automation of the tension process by installing a hydraulic drive. In the article, based on the analysis of known designs of tensioning devices, a variant of a hydraulic tensioning station is proposed that provides an adaptive change in the tension of the belt. The practical value of the material is to reduce the wear of the conveyor belt

Adaptable automation for a more human-centered work design? Effects on human perception and behavior

This experiment systematically examines whether, in safety-critical environments such as Air Traffic Control, the negative effects of increasing automation associated with static automation concepts can be mitigated by adaptable automation. Adaptable automation is a form of flexible automation in which the human operator (rather than the system as in adaptive automation) can decide when and to what extent to delegate tasks. A special focus is on its effects on human perception in terms of perceived autonomy and competence, satisfaction, and human role perception. We conducted two online studies using the same dual-task paradigm. Study 1 was conducted with a novice sample via Prolific (N = 93) and study 2 with an expert sample of Air Traffic Controllers (N = 126). Participants were either supported by static information automation, static decision automation, or an adaptable solution that allowed them to switch between the two automation stages. The findings of both studies are similar. Results indicated that, even when humans rarely switched, adaptable automation could increase perceived autonomy, led to high satisfaction, and had positive effects on role perceptions without impairing performance or workload. Furthermore, satisfaction was found to correlate with performance. From a human-centered perspective, flexible concepts seem to be particularly suitable when automation increasingly takes over parts of a job task not only at the stage of information analysis but also at the stage of decision-making.

Rig Automation Empowers Well Construction in Ecuador

_ This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper SPE 216249, “Digital Innovation of Well-Construction Process in Ecuador Through Rig Automation,” by Karen Peña, Kevin Etcheverry, and Hugo Quevedo, SPE, SLB, et al. The paper has not been peer reviewed. _ Artificial intelligence (AI)-based digital drilling technology was implemented in two mature fields in Ecuador that represent 33% of the country’s oil production. The drilling campaign’s main strategy included the deployment of a novel automation solution on two rigs, resulting in the optimization of the well-construction process. In the complete paper, the authors present the results of implementation of a rig-automation solution applied to 20 wells in 2022. Scope of the Digital Solution The authors detail the validation, optimization, and implementation of a drilling software to automate work flows and the installation and commissioning of associated hardware components. The following key indicators and objectives were established: - Validation of drilling-software work flows - Achievement of a 75% average in automation control - Improvement of rate of penetration (ROP) by 5% - Reduction of pre- and post-connection times by at least 30% - Real-time tuning of parameters - Training of all personnel involved in the use of the drilling-software package - Installation and commission of hardware components Drilling-Software Implementation The digital solution is an advanced software system that brings automation capabilities to drilling rigs, analogous to the different levels of automation found in automobiles. By drawing a parallel to car automation, one can better understand the system’s functionality and benefits. Level 1 consists of task automation, comparable to automatic driving features such as cruise control. In this context, the drilling software takes on the role of automating specific tasks within the drilling process. Level 2 involves process automation, similar to a car’s ability to autonomously park itself. The digital solution automates selected processes in drilling operations that enable efficient and precise execution. Level 3 encompasses adaptive automation, akin to a self-driving car that dynamically adapts to its surroundings. The drilling software continuously analyzes the real-time conditions of the wellbore and suggests optimal drilling parameters based on proprietary algorithms that leverage the well-known DeTournay model principle. The drilling software actively controls four key rig machines: the drawworks, topdrive, automated driller, and mud pumps. The solution adjusts drilling parameters intelligently, optimizing the drilling process and improving overall efficiency and drilling stability.

Influence of automation level of human-machine system on operators’ mental load

The appropriate automation in armored vehicles is vital for the operational efficiency and personnel security of operators. In this study, fifty subjects conducted over-the-horizon strike and N-back tests at different automation levels based on a virtual simulation system for armored vehicles. Physiological signals and subjective assessments were recorded. The mental load and task performance of operators were related to different automation levels. Results suggested that the mental load decreased with the increase of automation levels. Apart from object destruction time, heart rate and standard deviation of NN intervals (SDNN), other indexes were all significantly affected by the automation level of subtasks (p ​< ​0.01). The NASA-TLX scores, object destruction time, response time of abnormal states, and reaction time in N-back tests decreased by at least 2.9 ​%, 8.2 ​%, 11.2 ​% and 1.3 ​% respectively, while the mean accuracy in N-back tests increased by 0.1 ​%. Furthermore, there existed several automation levels of tasks where the task performance remained almost unchanged under normal operation. The function of task automation on decreasing mental load reduced in the following order: A3-B3-C2-D2-E2, A2-B2-C2-D2-E2, and A3-B3-C1-D1-E1. The main contribution of this research was to provide a qualitative method and framework for the evaluation of influences of automation level on operators’ mental load, and the design of human-machine interaction and adaptive automation in automated systems.

Adaptive Human-Robotic Interaction for Robotic-assisted Surgical Settings.

Increased complexity in robotic-assisted surgical system interfaces introduces problems with human-robot collaboration that result in excessive mental workload (MWL), adversely impacting a surgeon's task performance and increasing error probability. Real-time monitoring of the operator's MWL will aid in identifying when and how interventions can be best provided to moderate MWL. In this study, an MWL-based adaptive automation system is constructed and evaluated for its effectiveness during robotic-assisted surgery. This study recruited 10 participants firstto perform surgical tasks under different cognitive workload levels. Physiological signals were obtained and employed to build a real-time system for cognitive workload monitoring. To evaluate the effectiveness of the proposed system, 15 participants were recruited to perform the surgical task with and without the proposed system. The participants' task performance and perceived workload were collected and compared. The proposed neural network model achieved an accuracy of 77.9% in cognitive workload classification. In addition, better task performance and lower perceived workload were observed when participants completed the experimental task under the task condition supplemented with adaptive aiding using the proposed system. The proposed MWL monitoring system successfully diminished the perceived workload of participants and increased their task performance under high-stress conditions via interventions by a semi-autonomous suction tool. The preliminary results from the comparative study show the potential impact of automated adaptive aiding systems in enhancing surgical task performance via cognitive workload-triggered interventions in robotic-assisted surgery.

Toward Workload-Based Adaptive Automation: The Utility of fNIRS for Measuring Load in Multiple Resources in the Brain

We investigate the utility of functional near-infrared spectroscopy (fNIRS) for workload-based adaptive automation through the lens of multiple resource theory. We focus on the criteria of unobtrusiveness, responsiveness, load sensitivity (low vs high load), and load diagnosticity (differentiating types of load). We report a large meta-review, in which we conclude that only a few studies were suitable for evaluating sensitivity and diagnosticity in complex real-world tasks. While these reveal that the fNIRS signal is adequately sensitive to gradations of load level changes (sensitivity), the diagnosticity of fNIRS to different sources of cognitive load remained uncertain. We manipulated mental load of a complex shape sorting task via working memory load (WM) and visual perceptual load (VL), while a secondary auditory task was present throughout. We measured the effect of these manipulations at the group-level using conventional secondary and eyetracking workload measures, as well as hemodynamic response in specific functional regions in the brain, including regions involved in multi-tasking (MT), VL, WM, and auditory load (AL). Our findings revealed that fNIRS is both sensitive and diagnostic to load in complex tasks, with greater sensitivity revealed by deoxyhemoglobin than oxyhemoglobin and the brain regions associated with diagnosticity align with neuroscience literature on perceptual load, WM, and goal-directed multitasking.

Preventing mind-wandering during driving: Predictions on potential interventions using a cognitive model

In this study, we made predictions on the effects of different interventions by adaptive automation systems designed to prevent mind-wandering while driving. Although cognitive load associated with secondary tasks tends to affect driving negatively, a simple secondary task can improve driving performance when the driving scenario is mundane. Nijboer and colleagues (2016) have hypothesized that if the driving task is simple, people might start mind-wandering, which interferes with driving. Furthermore, the authors proposed that a simple secondary task, which imposes less workload than mind-wandering, could prevent this from happening and thereby improve driving performance. Automation systems that are informed about and adapt to the cognitive state of the driver could leverage this effect by inducing mild cognitive load during mundane driving scenarios. To test suitable interventions, we combined an existing driver model with an existing model of mind-wandering implemented in the cognitive architecture ACT-R as executable theories of mind-wandering and cognitive processing in driving. Using these different models we show how mind-wandering harms driving performance at the behavioral level and that interventions eliciting mild cognitive load can mitigate this behavioral effect. However, the model indicates that some interventions incur a significant cognitive processing cost that adaptive automation systems must account for.

Ergonomic Design of an Adaptive Automation Assembly System

Ergonomics is a key factor in the improvement of health and productivity in workplaces. Its use in improving the performance of a manufacturing process and its positive effects on productivity and human performance is drawing the attention of researchers and practitioners in the field of industrial engineering. This paper proposes an ergonomic design approach applied to an innovative prototype of an adaptive automation assembly system (A3S) equipped with Microsoft Kinect™ for real-time adjustment. The system acquires the anthropometric measurements of the operator by means of the 3-D sensing device and changes its layout, arranging the mobile elements accordingly. The aim of this study was to adapt the assembly workstation to the operator dimensions, improving the ergonomics of the workstation and reducing the risks of negative effects on workers’ health and safety. The case study of an assembly operation of a centrifugal electric pump is described to validate the proposed approach. The assembly operation was simulated at a traditional fixed workstation and at the A3S. The shoulder flexion angle during the assembly tasks at the A3S reduced between 18% and 47%. The ergonomic risk assessment confirmed the improvement of the ergonomic conditions and the ergonomic benefits of the A3S.

Evaluating the dissociation between drivers’ self-perceived and objective need for vehicle assistance during obstacle avoidance tasks

Driver-assistance systems are becoming more commonplace; however, the realized safety benefits of these technologies depend on whether a person accepts and adopts automated driving aids. One challenge to adoption could be a preference-performance dissociation (PPD), which is a mismatch between a self-perceived desire and an objective need for assistance. Research has reported PPD in driving but has not extensively leveraged driving performance data to confirm its existence. Thus, the goal of this study was to compare drivers’ self-reported need for vehicle assistance to their objective driving performance. Twenty-one participants drove on a simulated road and traversed challenging, real-world roadway obstacles. Afterwards, they were asked about their preference for automated vehicle assistance (e.g., steering and braking) during their drive. Overall, some participants exhibited PPD that included both over- and underestimating their need for a particular type of automated assistance. Findings can be used to develop shared control and adaptive automation strategies tailored to particular users and contexts across various safety-critical environments.

Spectral methods for capillary surfaces described by bounded generating curves

We consider capillary surfaces that are constructed by bounded generating curves. This class of surfaces includes radially symmetric and lower dimensional fluid-fluid interfaces. We use the arc-length representation of the differential equations for these surfaces to allow for vertical points and inflection points along the generating curve. These considerations admit fluids in capillary tubes, sessile drops, and fluids in annular tubes as well as other examples.We present a pseudo-spectral method for approximating solutions of the associated boundary value problems based on interpolation by Chebyshev polynomials. This method is observably more stable than the traditional shooting method and it is computationally lean and fast. The algorithm is also adaptive, but does not use the adaptive automation in Chebfun.

Connected, Cooperative and Automated Driving: Stepping away from dynamic function allocation towards human-machine collaboration

Implementing the concept of Connected, Cooperative and Automated Driving (CCAD) has many potential societal and individual benefits. However, to do so successfully fundamental challenges need to be tackled, one being the integration of automation and the human driver. The decisive problems with safety surrounding this topic are well-known, in particular when the human is expected to intervene in situations that the automation is unable to handle. Here we argue that the leading human-machine interaction paradigm for automated driving was and continues to be dynamic function allocation and adaptive automation and that this gave rise to many of the issues currently seen in CCAD. As an alternative, we propose the concept of collaboration which has been discussed in other domains. We believe it may in the future allow tackling the issues associated with the integration of CCAD and the human.

Model Predictive Degree of Automation Regulation for Mobile Robots Using Robot Vitals and Robot Health

Environmental adversities can severely impact the performance of human-robot teams, potentially even leading to task failure. If the operator and the robot automation are not equally affected, adjusting the degree of automation to shift control authority between them is a means of maintaining the performance of the human-robot team. The robot vitals and robot health framework is a recent approach to quantifying runtime performance degradation in robots. This framework can serve as a methodological foundation for the adjustment of the degree of automation based on the human-robot system's state. In this paper, we contribute two model predictive adaptive automation systems that can adjust either the level or the degree of automation of a robot. These systems optimize robot health to ensure optimal performance of the human-robot team when exposed to adversities. Feasibility studies in simulation showcase the ability of our systems to manage the level and degree of automation, thus allowing for an optimal task execution by the human-robot team.

A Comparative Study of Situation Awareness-Based Decision-Making Model Reinforcement Learning Adaptive Automation in Evolving Conditions

Situation-awareness-based decision-making (SABDM) models constructed using cognitive maps and goal-direct task analysis techniques have been successfully used in decision support systems in safety-critical and mission-critical environments such as air traffic control and electrical energy distribution. Reinforcement learning (RL) and other machine learning techniques are used to automate situational awareness mental model parameter adjustments, reducing the expert work on the initial configuration and long-term maintenance without affecting the mental model’s structure and maintaining the situation-awareness-based decision-making model’s cognitive and explainability characteristics. Real-world models should evolve to cope with changes in the environmental conditions. This study evaluates the application of reinforcement learning as an online adaptive technique to adjust the situational-awareness mental model’s parameters under evolving conditions. We conducted evaluation experiments using real-world public datasets to compare the performance of the SABDM model with that of the reinforcement learning adaptation technique (SABDM/RL) and other adaptive machine learning methods under distinct concept drift evolving conditions. We measured the techniques’ overall and dynamic performance to understand how well they adapt to evolving environmental conditions. The experiments show that SABDM/RL performs similarly to modern online adaptive machine learning classification methods with the support of concept drift detection techniques while maintaining the mental model strength of the situation awareness-based systems.

Return-to-Manual Performance can be Predicted Before Automation Fails.

This study aimed to examine operator state variables (workload, fatigue, and trust in automation) that may predict return-to-manual (RTM) performance when automation fails in simulated air traffic control. Prior research has largely focused on triggering adaptive automation based on reactive indicators of performance degradation or operator strain. A more direct and effective approach may be to proactively engage/disengage automation based on predicted operator RTM performance (conflict detection accuracy and response time), which requires analyses of within-person effects. Participants accepted and handed-off aircraft from their sector and were assisted by imperfect conflict detection/resolution automation. To avoid aircraft conflicts, participants were required to intervene when automation failed to detect a conflict. Participants periodically rated their workload, fatigue and trust in automation. For participants with the same or higher average trust than the sample average, an increase in their trust (relative to their own average) slowed their subsequent RTM response time. For participants with lower average fatigue than the sample average, an increase in their fatigue (relative to own average) improved their subsequent RTM response time. There was no effect of workload on RTM performance. RTM performance degraded as trust in automation increased relative to participants' own average, but only for individuals with average or high levels of trust. Study outcomes indicate a potential for future adaptive automation systems to detect vulnerable operator states in order to predict subsequent RTM performance decrements.

An Adaptive Human-Robotic Interaction Architecture for Augmenting Surgery Performance Using Real-Time Workload Sensing-Demonstration of a Semi-autonomous Suction Tool.

This study developed and evaluated a mental workload-based adaptive automation (MWL-AA) that monitors surgeon cognitive load and assist during cognitively demanding tasks and assists surgeons in robotic-assisted surgery (RAS). The introduction of RAS makes operators overwhelmed. The need for precise, continuous assessment of human mental workload (MWL) states is important to identify when the interventions should be delivered to moderate operators' MWL. The MWL-AA presented in this study was a semi-autonomous suction tool. The first experiment recruited ten participants to perform surgical tasks under different MWL levels. The physiological responses were captured and used to develop a real-time multi-sensing model for MWL detection. The second experiment evaluated the effectiveness of the MWL-AA, where nine brand-new surgical trainees performed the surgical task with and without the MWL-AA. Mixed effect models were used to compare task performance, objective- and subjective-measured MWL. The proposed system predicted high MWL hemorrhage conditions with an accuracy of 77.9%. For the MWL-AA evaluation, the surgeons' gaze behaviors and brain activities suggested lower perceived MWL with MWL-AA than without. This was further supported by lower self-reported MWL and better task performance in the task condition with MWL-AA. A MWL-AA systems can reduce surgeons' workload and improve performance in a high-stress hemorrhaging scenario. Findings highlight the potential of utilizing MWL-AA to enhance the collaboration between the autonomous system and surgeons. Developing a robust and personalized MWL-AA is the first step that can be used do develop additional use cases in future studies. The proposed framework can be expanded and applied to more complex environments to improve human-robot collaboration.

Advances in Artificial Intelligence Methods Applications in Industrial Control Systems: Towards Cognitive Self-Optimizing Manufacturing Systems

Industrial control systems play a central role in today’s manufacturing systems. Ongoing trends towards more flexibility and sustainability, while maintaining and improving production capacities and productivity, increase the complexity of production systems drastically. To cope with these challenges, advanced control algorithms and further developments are required. In recent years, developments in Artificial Intelligence (AI)-based methods have gained significantly attention and relevance in research and the industry for future industrial control systems. AI-based approaches are increasingly explored at various industrial control systems levels ranging from single automation devices to the real-time control of complex machines, production processes and overall factories supervision and optimization. Thereby, AI solutions are exploited with reference to different industrial control applications from sensor fusion methods to novel model predictive control techniques, from self-optimizing machines to collaborative robots, from factory adaptive automation systems to production supervisory control systems. The aim of the present perspective paper is to provide an overview of novel applications of AI methods to industrial control systems on different levels, so as to improve the production systems’ self-learning capacities, their overall performance, the related process and product quality, the optimal use of resources and the industrial systems safety, and resilience to varying boundary conditions and production requests. Finally, major open challenges and future perspectives are addressed.

A User Study for the Evaluation of Adaptive Interaction Systems for Inclusive Industrial Workplaces

In recent years, production systems have become highly sophisticated and complex. As a result, while on the one hand, the least-skilled labor has been partially displaced by machines, high-skilled labor is more required to supervise and control advanced automation systems. In many cases, the complexity of machines implies an increased complexity of human–machine interfaces (HMIs), which are the main point of contact between the operator and the machine. To enable effective use of HMIs and to enable their usage by workers with different knowledge and capabilities, novel design approaches have been proposed. In particular, in this article, we consider the approach developed in the framework of the European research project INCLUSIVE, which aimed at designing industrial HMIs that adapt to the skills and capabilities of human operators. As a case study, we consider an adaptive interaction system for the woodworking industry and present an extensive evaluation carried out in real production environment with shopfloor workers. The effectiveness of the INCLUSIVE approach has been assessed with subjective and objective measurements and compared to that of interaction systems customarily used in industry. Results have shown that users appreciated the INCLUSIVE system and largely preferred it over the customary system. Moreover, with regard to objective performance-related measurements, they performed better when using the INCLUSIVE system since they received tailored guidance during the considered working tasks. Note to Practitioners— This article was motivated by the fact that advanced automation systems are often highly complex for human operators. To address this problem, we focus on the importance of designing the automation system around users and discuss an approach for adaptive automation, called INCLUSIVE. Its main feature is that it adapts the human–machine interface (HMI) according to operator’s skills, capabilities, and current mental fatigue. In this article, we provide an extensive evaluation of the INCLUSIVE system, considering a company producing woodworking machines as a use case. Assessment was carried out in the company shopfloor, considering real workers, and the INCLUSIVE system was compared with the customary HMI running on the company machines. The results of our study suggest that adapting the interaction to operator’s needs allows better working performance while letting workers more satisfied with the use of the system.

Working Memory Load Impact on Effective Connectivity: a Dynamic Causal Modeling Study

This study aims to examine the roles of the brain regions in working memory (WM) processing and the modulatory effects of WM load. Electroencephalogram (EEG) signals were recorded when participants perform a letter-version n-back task with three different levels of WM load. The directional causal connections between brain regions were estimated using Dynamic Causal Modeling (DCM). The directions and strengths of the connections were compared for different WM load conditions. The results showed a right-lateralized, backward-only connection pattern for the high WM load condition. The results also showed changes in the roles of the brain regions when the WM load increases. These findings of the modulatory effects of WM load may be utilized in measuring cognitive states, and designing adaptive automation in augmented cognition programs.