Human Operator Performance Research Articles

Performance-Aware Trust Modeling within a Human–Multi-Robot Collaboration Setting

In this study, a novel time-driven mathematical model for trust is developed considering human–multi-robot performance for a Human–Robot Collaboration (HRC) framework. For this purpose, a model is developed to quantify human performance considering the effects of physical and cognitive constraints and factors such as muscle fatigue and recovery, muscle isometric force, human (cognitive and physical) workload, workloads due to the robots’ mistakes, and task complexity. The performance of multi-robot in the HRC setting is modeled based upon the rate of task assignment and completion as well as the mistake probabilities of the individual robots. The human trust in HRC setting with single and multiple robots is modeled over different operation regions, namely unpredictable region, predictable region, dependable region, and faithful region. The relative performance difference between the human operator and the robot is used to analyze the effect on the human operator’s trust in robots’ operation. The developed model is simulated for a manufacturing workspace scenario considering different task complexities and involving multiple robots to complete shared tasks. The simulation results indicate that for a constant multi-robot performance in operation, the human operator’s trust in robots’ operation improves whenever the comparative performance of the robots improves with respect to the human operator performance. The impact of robot hypothetical learning capabilities on human trust in the same HRC setting is also analyzed. The results confirm that a hypothetical learning capability allows robots to reduce human workloads, which improves human performance. The simulation result analysis confirms that the human operator’s trust in the multi-robot operation increases faster with the improvement of the multi-robot performance when the robots have a hypothetical learning capability. An empirical study was conducted involving a human operator and two collaborator robots with two different performance levels in a software-based HRC setting. The experimental results closely followed the pattern of the developed mathematical models when capturing human trust and performance in terms of human–multi-robot collaboration.

A Real-time Assessment System for Cognitive Performances of Armored Vehicle Crews Based upon the Multi-source Information Fusion

In the human-machine system of armored vehicles, the cognitive performance state of crews is crucial for the personnel security and combat efficiency. The purpose of this research was to establish a real-time assessment system for cognitive performances of armored vehicle crews, consisting of the data input module, data processing module, data visualization module, and scheduling module. Forty subjects were recruited to cooperate and execute the cross-platform strike task in a virtual simulation platform. The physiological data and operation behavior data was collected during the experiment process. To realize the accurate classification of different cognitive performance states, a multi-source information fusion algorithm was developed based on linear discriminant analysis (LDA) and D-S evidence theory, which included the information collection module, the feature extraction module, and the information fusion module. The results indicated that there existed a significant correlation between the extractive feature indicators (i.e., EOG, ECG, and task performance indicators) and the cognitive performance. The recognition accuracy and the data efficiency of the proposed assessment system were 91.25% and 96.69% respectively by using the complementarity of different evidences, which were higher than the others using partial information sources. This study can provide a reference for the comprehensive assessment of cognitive performance of human operators in military and industrial domains.

Retrospective and predictive analysis of human operator performance with event report data of a nuclear reactor

Retrospective and predictive analysis of human operator performance with event report data of a nuclear reactor

Additive Sensory Noise Effects on Operator Performance in a Lunar Landing Simulation.

INTRODUCTION: Adding noise to a system to improve a weak signal's detectability is known as stochastic resonance (SR). SR has been shown to improve sensory perception and cognitive performance in certain individuals, but it is unknown whether this performance improvement can translate to meaningful macrocognitive enhancements in performance for complex, operational tasks.OBJECTIVE: We investigated human operator performance in a lunar landing simulation while applying auditory white noise and/or noisy galvanic vestibular stimulation.METHODS: We measured performance (N = 16 subjects) while completing simulation trials in our Aerospace Research Simulator. Trials were completed with and without the influence of auditory white noise, noisy galvanic vestibular stimulation, and both simultaneously in a multimodal fashion. Performance was observed holistically and across subdimensions of the task, which included flight skill and perception. Subjective mental workload was collected after completing four trials in each treatment.RESULTS: We did not find broad operator improvement under the influence of noise, but a significant interaction was identified between subject and noise treatment, indicating that some subjects were impacted by additive noise. We also found significant interactions between subject and noise treatment in performance subdimensions of flight skill and perception. We found no significant main effects on mental workload.CONCLUSIONS: This study investigated the utility of using additive sensory noise to induce SR for complex tasks. While SR has been shown to improve aspects of performance, our results suggest additive noise does not yield operational performance changes for a broad population, but specific individuals may be affected.Sherman SO, Shen Y-Y, Gutierrez-Mendoza D, Schlittenhart M, Watson C, Clark TK, Anderson AP. Additive sensory noise effects on operator performance in a lunar landing simulation. Aerosp Med Hum Perform. 2023; 94(10):770-779.

The Impacts of Unreliable Autonomy in Human-Robot Collaboration on Shared and Supervisory Control for Remote Manipulation

This work compared human-robot shared and supervisory control of remote robots for dexterous manipulation, and examined how the reliability of robot autonomy affects human operator performance, workload, and preference for robot assistance. Specifically, we implemented two human-robot collaboration (HRC) paradigms for remote manipulation: (1) shared control, where humans controlled gross manipulation and the robot autonomy controlled precise manipulation actions, and (2) supervisory control, where the robot autonomy controlled both gross and precise manipulation actions but relied on humans to detect and correct errors. We conducted two user studies: one to compare the effectiveness of the two HRC paradigms when assistive autonomy is reliable, and the other to examine the impact of error type and frequency on tasks and human operators in the two HRC paradigms when assistive autonomy is unreliable. Our results show that: (1) the interface with a higher level of reliable autonomy yields significantly better performance, lower workload, and higher user preference but lower engagement, and (2) the frequency and type of the error have significant impacts on the task performance and human workload but only partially affects the operator's preference and usage of autonomy.

Shaping Human Movement via Bimanually-Dependent Haptic Force Feedback.

Haptic feedback can enhance training and performance of human operators; however, the design of haptic feedback for bimanual coordination in robot-assisted tasks (e.g., control of surgical robots) remains an open problem. In this study, we present four bimanually-dependent haptic force feedback conditions aimed at shaping bimanual movement according to geometric characteristics: the number of targets, direction, and symmetry. Haptic conditions include a virtual spring, damper, combination spring-damper, and dual springs placed between the hands. We evaluate the effects of these haptic conditions on trajectory shape, smoothness, and speed. We hypothesized that for subjects who perform worse with no haptic feedback (1) a spring will improve the shape of parallel trajectories, (2) a damper will improve the shape of point symmetric trajectories, (3) dual springs will improve the shape of trajectories with one target, and (4) a damper will improve smoothness for all trajectories. Hypotheses (1) and (2) were statistically supported at the p < 0.001 level, but hypotheses (3) and (4) were not supported. Moreover, bimanually-dependent haptic feedback tended to improve shape accuracy for movements that subjects performed worse on under no haptic condition. Thus, bimanual haptic feedback based on geometric trajectory characteristics shows promise to improve performance in robot-assisted motor tasks.

Enhancing Human Operator Performance With Long Short-Term Memory Networks in Adaptively Controlled Systems

Enhancing Human Operator Performance With Long Short-Term Memory Networks in Adaptively Controlled Systems

Comparative Analysis of Human Operators and Advanced Technologies in the Visual Inspection of Aero Engine Blades

Background—Aircraft inspection is crucial for safe flight operations and is predominantly performed by human operators, who are unreliable, inconsistent, subjective, and prone to err. Thus, advanced technologies offer the potential to overcome those limitations and improve inspection quality. Method—This paper compares the performance of human operators with image processing, artificial intelligence software and 3D scanning for different types of inspection. The results were statistically analysed in terms of inspection accuracy, consistency and time. Additionally, other factors relevant to operations were assessed using a SWOT and weighted factor analysis. Results—The results show that operators’ performance in screen-based inspection tasks was superior to inspection software due to their strong cognitive abilities, decision-making capabilities, versatility and adaptability to changing conditions. In part-based inspection however, 3D scanning outperformed the operator while being significantly slower. Overall, the strength of technological systems lies in their consistency, availability and unbiasedness. Conclusions—The performance of inspection software should improve to be reliably used in blade inspection. While 3D scanning showed the best results, it is not always technically feasible (e.g., in a borescope inspection) nor economically viable. This work provides a list of evaluation criteria beyond solely inspection performance that could be considered when comparing different inspection systems.

Empirical estimation of human error probabilities based on the complexity of proceduralized tasks in an analog environment

The contribution of degraded human performance (e.g., human errors) is significant for the safety of diverse social-technical systems. Therefore, it is crucial to understand when and why the performance of human operators could be degraded. In this study, the occurrence probability of human errors was empirically estimated based on the complexity of proceduralized tasks. To this end, Logistic regression analysis was conducted to correlate TACOM (Task Complexity) scores with human errors collected from the full-scope training simulator of nuclear power plants equipped with analog devices (analog environment). As a result, it was observed that the occurrence probability of both errors of commission and errors of omission can be soundly estimated by TACOM scores. Since the effect of diverse performance influencing factors on the occurrence probabilities of human errors could be soundly distinguished by TACOM scores, it is also expected that TACOM scores can be used as a tool to explain when and why the performance of human operators starts to be degraded.

Teleoperation of wheeled mobile robots subject to longitudinal slipping and lateral sliding by time-domain passivity controller

Wheeled mobile robot (WMR)-based exploration on soft terrains has created new difficulties for its tele-control system. A critical problem is that wheel-terrain contact areas may induce both longitudinal slipping and lateral sliding. In this paper, a new approach for tele-driving a WMR that addresses the difficulties simultaneously induced by both longitudinal slipping and lateral sliding is presented. In the augmented coordination between the master and slave site, the translational and angular velocities of the slave WMR were modified by the wheel sliding angle. Moreover, they were separately coordinated with the positions of the master haptic interface. To improve the command-tracking performance with the augmented coordination, the teleoperator was designed with feedforward of the sliding angle. To compensate for the non-passivity induced by longitudinal slipping and lateral sliding, a time-domain passivity controller also is proposed. The stability and transparency of the proposed approach were analyzed based on the passivity theory. Numerical experiments validate that this proposed approach can effectively improve the teleoperation performance of human operators for the WMR on soft terrains.

Identification of mental fatigue levels in a language understanding task based on multi-domain EEG features and an ensemble convolutional neural network

In this paper, electroencephalogram (EEG) is used to assess mental fatigue of operators in a human-computer system aiming at preventing increasing risk of human operator performance degradation. We present an experimental design for fatigue identification in a language understanding task. The EEG signals of 14 channels from 15 healthy participants were collected via a wireless brain computer interface device to indicate instantaneous fatigue level. By extracting EEG features as temporal statistics, power spectral density and entropy indicators, we build four different spatial feature maps that restructure feature vectors. Further, a bootstrap-aggregating ensemble convolutional neural network of multi-domain features (ensCNN-MD) is proposed to improve the fatigue recognition accuracy. By examining seven different feature combinations, ensCNN-MD is significantly superior to classical shallow and deep classifiers. The highest classification accuracy under participant-specific training and testing paradigm is achieved at 87.69%. The results demonstrate both the effectiveness of experimental design and the ensCNN-MD of learning high-level spatial feature abstractions related to mental fatigue variations.

Human performance evaluation of user interfaces for spatially-constrained text portrayal and access

To maximize visualization effectiveness, graphical data are commonly augmented with text to provide detailed information and define specific values. This text is often displayed in a pop-up dialog box pertaining to an object, permitting simultaneous display of the object and associated alphanumeric information. However, a human operator performance cost may be incurred when the resulting portrayal occludes critical information within the visualization. To address this issue, we developed and evaluated three alternative, spatially constrained, text portrayal techniques. These techniques and their associated access interface were designed to reduce occlusion while providing rapid access to desired alphanumeric data. Each technique was evaluated against the pop-up dialog using a dual-task human performance paradigm. Performance measures included accuracy, response time, display occlusion, and subjective feedback. The basis for the spatially-constrained text access technique design, their implementation affordances, and limitations are discussed. The alternative techniques and their user interface concepts resulted in mixed accuracy and response time performance compared to the pop-up dialog. Specific design features reduced data access time to one third of the time required to access the spatially-constrained text access techniques. Overall, equivalent performance was obtained among the variants while the potential for occlusion was reduced during use of the novel designs.

Autonomous recommender system for reconnaissance tasks using a swarm of UAVs and asynchronous shepherding

Several artificial intelligence-based systems have been proposed to help human operators in safety-critical domains such as air-traffic control systems, health and farming. In scenarios where multiple autonomous systems are utilised, the cognitive capacity of humans hits its limits quickly, and sometimes the success of these systems is obstructed. Designing systems to sustain human performance over longer periods through balancing the load between human operators and autonomous systems can reduce errors and improve overall performance. Future human-autonomy systems need to be designed in a way that makes them scalable in terms of capabilities and safety, yet flexible enough to adapt to the cognitive workload of human operators. In this paper, we propose an adaptive human swarm interaction system for maintaining human operator performance in supervisory-control aerial-surveillance tasks. A bio-inspired guidance and control shepherding agent is proposed to support the human operator to guide multiple assets simultaneously using a single point of control. The proposed system employs an adjustable autonomous recommender system to provide the human operator with timely recommendations in real-time sittings. The proposed system is designed in a flexible way that allows it to adapt its rate of providing these recommendations in response to the needs of human operators. Simulation-based experiments are conducted to validate and evaluate the proposed recommender system. Experimental results demonstrate the effectiveness and efficiency of the proposed recommender system in minimising control errors while maintaining low computational costs that allow it to be used in a real-time applications.

Design and characterization of an instrumented hand-held power tool to capture dynamic interaction with the workpiece during manual operations

In recent years, robots have contributed extensively to the automation of repetitive tasks for which position control-based approaches represent effective solutions. However, other contact tasks such as finishing, deburring and grinding require both position and force control. To date, despite the availability of cost-effective robotic solutions, such tasks are still carried out manually by skilled operators mainly because programming is time consuming and not sufficiently flexible to be readapted to product or task changes. An alternative approach is teaching by demonstration, by instrumenting hand-held tools for capturing both the force and the contact point while an expert operator performs tooling tasks. This paper presents a novel approach for instrumenting hand-held tools for polishing/grinding used in monitoring the performance of skilled human operators which can be, in future work, translated into planning strategies for robot programming and control. More specifically, the instrumented tool is designed to monitor interaction forces with the workpiece and point of contact, where these interaction forces arise. The key element in our design is a flexible coupler which, ideally, only transmits rotation torques to spin the polishing/grinding wheel while all the remaining torque and force components are transmitted through a parallel structure and sensed by a 6-axis loadcell. Sensing torques, in addition to forces, also allows for a dynamic estimation of the point of contact.

Agent-based vulnerability assessment at airport security checkpoints: A case study on security operator behavior

Despite enormous investments in airport security, terrorists have been able to find and exploit vulnerabilities at security checkpoints. Existing vulnerability assessment methodologies struggle with accounting for human behavior, and agent-based modelling forms a promising technique to overcome this limitation.This paper investigated how the decision-making and performance of human operators can be taken into account while assessing vulnerability at an airport security checkpoint. To this end, an agent-based model was designed, in which the performance of security operators was modelled using a functional state model, while decision making was modelled using decision field theory. Passengers and an attacker that brings a weapon to the security checkpoint were also explicitly modelled as agents. Simulation results indicate that the highest skilled operators outperformed their lowest skilled counterparts on analyzing X-ray images, but performed worse on both searching luggage and performing patdowns. Furthermore, results showed that a high focus on speed of security operators leads to a decrease in luggage searches and therefore increased vulnerability.More work is needed to calibrate and validate the simulation results, but initial results are promising. The agent-based model can be used by airport regulators and managers to understand the workings of their security checkpoint better and ultimately to reduce vulnerabilities.

An empirical study of the personality effects on diagnosis performance of human operators in unexpected plant conditions of NPPs

Personality traits have been recognized as an important means to mitigate psychological and physiological effects on human performance. In the nuclear industry, personality traits have been utilized only to screen out potential employees of mental illnesses. Although cognitive characteristics are considered in second generation HRA methods, there is no or superficial consideration about personality itself. Thus, an experiment was conducted to examine the relationship between personality traits and diagnosis performance in unexpected plant conditions of NPPs. As a result, five personality facets correlated with diagnosis performance were identified. The empirical data were used to derive an equation for diagnosis performance and a multiplier of personality factor through a regression analysis. This study proved that personality traits can be utilized in human resource management to maximize operators’ performance, and the results can be utilized as a useful empirical source for HRA experts to judge effects of personality traits on human performance.

Human Factors Contributions to the Development of Standards for Displays of Unmanned Aircraft Systems in Support of Detect-and-Avoid.

The aim is to provide a high-level synthesis of human factors research that contributed to the development of detect-and-avoid display requirements for unmanned aircraft systems (UAS). The integration of UAS into the U.S. National Airspace System is a priority under the Federal Aviation Administration's Modernization and Reform Act. For UAS to have routine access to the National Airspace System, UAS must have detect-and-avoid capabilities. One human factors challenge is to determine how to display information effectively to remote pilots for performing detect-and-avoid tasks. A high-level review of research informing the display requirements for UAS detect-and-avoid is provided. In addition, description of the contributions of human factors researchers in the writing of the requirements is highlighted. Findings from human-in-the-loop simulations are used to illustrate how evidence-based guidelines and requirements were established for the display of information to assist pilots in performing detect-and-avoid. Implications for human factors are discussed. Human factors researchers and engineers made many contributions to generate the data used to justify the detect-and-avoid display requirements. Human factors researchers must continue to be involved in the development of standards to ensure that requirements are evidence-based and take into account human operator performance and human factors principles and guidelines. The research presented in this paper is relevant to the design of UAS, the writing of standards and requirements, and the work in human-systems integration.

Application of Cardio-Forecasting for Evaluation of Human-Operator Performance.

The paper presents the results of the development of the cardio-forecasting technology, which introduces a new method to monitor the state of human-operator, which is characteristic for the given production conditions and for individual operators, to predict the moment of exhaustion of his/her working capacity. The work aims to demonstrate the unique, distinctive features of the cardio-forecasting technology for predicting an individual limit of his/her working capacity for each person. A unique methodology for predicting individually for each person the moment when he/she reaches the limit of his/her working capacity is based on a spectral analysis of a human phonocardiogram in order to isolate the frequency component located at the heart contraction frequency. The trend of the amplitude of this component is approximated by its model; consequently, the coefficients of the trend model are determined. They include the operator’s operating time until his/her working capacity is exhausted. A methodology for predicting the moment when he/she reaches the limit of his/her working capacity for each person individually and assessment based on this degree of criticality of their condition will be realized as a software application for smartphones using the Android operating system.

Operator Performance Prediction based on Fuzzy Modeling Approach

Abstract In this paper, physiological signals were measured from five participants, each participating in two sessions of experiment with identical experimental procedure. A simulation platform, AutoCAMS (Automation-enhanced Cabin Air Management System), was used to simulate a complex task environment of human-machine shared process control. Fuzzy models were constructed to quantitatively predict the human operator performance based on three EEG input features. The incremental-PID-controlled particle swarm optimization (IPID-PSO) algorithm was utilized to optimize the parameters of fuzzy models. The IPID-PSO algorithm incorporated incremental-PID-controlled search strategy to speed up the convergence of standard PSO algorithm. The operator performance modeling results are given to show the effectiveness of the IPID-PSO-tuned fuzzy modeling approach proposed to momentary operator performance assessment problem under consideration.

Operator-in-the-Loop Deep Sequential Multi-Camera Feature Fusion for Person Re-Identification

Given a target image as query, person re-identification systems retrieve a ranked list of candidate matches on a per-camera basis. In deployed systems, a human operator scans these lists and labels sighted targets by touch or mouse-based selection. However, classical re-id approaches generate per-camera lists independently. Therefore, target identifications by operator in a subset of cameras cannot be utilized to improve ranking of the target in remaining set of network cameras. To address this shortcoming, we propose a novel sequential multi-camera re-id approach. The proposed approach can accommodate human operator inputs and provides early gains via a monotonic improvement in target ranking. At the heart of our approach is a fusion function which operates on deep feature representations of query and candidate matches. We formulate an optimization procedure custom-designed to incrementally improve query representation. Since existing evaluation methods cannot be directly adopted to our setting, we also propose two novel evaluation protocols. The results on two large-scale re-id datasets (Market-1501, DukeMTMC-reID) demonstrate that our multi-camera method significantly outperforms baselines and other popular feature fusion schemes. Additionally, we conduct a comparative subject-based study of human operator performance. The superior operator performance enabled by our approach makes a compelling case for its integration into deployable video-surveillance systems.