Single Human Operator Research Articles

Rule-Based Verification of Autonomous Unmanned Aerial Vehicles

Automation enhances the capabilities of unmanned aerial vehicles (UAVs) by enabling self-determined behavior, while reducing the need for extensive human involvement. Future concepts envision a single human operator commanding multiple autonomous UAVs with minimal supervision. Despite advances in automation, there remains a demand for a “human in command” to assume overall responsibility, driven by concerns about UAV safety and regulatory compliance. In response to these challenges, a method for runtime verification of UAVs using a knowledge-based system is introduced. This method empowers human operators to identify unsafe behavior without assuming full control of the UAV. Aspects of automated formalization, updating and processing of knowledge elements at runtime, coupled with an automatic reasoning process, are considered. The result is an ontology-based approach for runtime verification, addressing the growing complexity of UAVs and the need to ensure safety in the context of evolving aviation regulations.

An Artificial Intelligence Generated Automated Algorithm to Measure Total Kidney Volume in ADPKD

IntroductionAccurate tools to inform individual prognosis in patients with autosomal dominant polycystic kidney disease (ADPKD) are lacking. Here, we report an artificial intelligence (AI) generated method for routinely measuring total kidney volume (TKV). MethodsAn ensemble U-net algorithm was created using the nnUNet approach. The training and internal cross-validation cohort consisted of all 1.5T MRI data acquired using 5 different MRI scanners (454 kidneys, 227 scans) in the CYSTic consortium which was first manually segmented by a single human operator. As an independent validation cohort, we utilised 48 sequential clinical MRI scans with reference results of manual segmentation acquired by 6 individual analysts at a single centre. The tool was then implemented for clinical use and its performance analysed. ResultsThe training / internal validation cohort was younger (mean age 44.0 vs 51.5 years) and the female-male ratio higher (1.2 v 0.94) compared to the clinical validation cohort. The majority of CYSTic patients had PKD1 mutations (79%) and typical disease (Mayo Imaging Class 1, 86%). The median DICE score on the clinical validation dataset between the algorithm and human analysts was 0.96 for left and right kidneys with a median TKV error of -1.8%. The time taken to manually segment kidneys in the CYSTic dataset was 56 (±28) min whereas manual corrections of the algorithm output took 8.5 (±9.2) min per scan. ConclusionsOur AI-based algorithm demonstrates performance comparable to manual segmentation. Its rapidity and precision in real-world clinical cases demonstrate its suitability for clinical application.

Human-Collective Collaborative Target Selection

Robotic collectives are composed of hundreds or thousands of distributed robots using local sensing and communication that encompass characteristics of biological spatial swarms, colonies, or a combination of both. Interactions between the individual entities can result in emergent collective behaviors. Human operators in future disaster response or military engagement scenarios are likely to deploy semi-autonomous collectives to gather information and execute tasks within a wide area, while reducing the exposure of personnel to danger. This article presents and evaluates two action selection models in an experiment consisting of a single human operator supervising four simulated collectives. The action selection models have two parts: (1) a best-of- n decision-making model that attempts to choose the highest-quality target from a set of n targets and (2) a quorum sensing task sequencing model that enables autonomous target site occupation. An original biologically inspired insect colony decision model is compared to a bias-reducing model that attempts to reduce environmental bias, which can negatively influence collective best-of- n decisions when poorer-quality targets are easier to evaluate than higher-quality targets. The collective decision-making models are compared in both supervised and unsupervised trials. The bias-reducing model without human supervision is slower than the original model but is 57% more accurate for decisions where evaluating the optimal target is more difficult. Human-collective teams using the bias-reducing model require less operator influence and achieve 25% higher accuracy with difficult decisions compared to the teams using the original model.

Adaptive Human-Robot Interactions for Multiple Unmanned Aerial Vehicles

Advances in unmanned aircraft systems (UAS) have paved the way for progressively higher levels of intelligence and autonomy, supporting new modes of operation, such as the one-to-many (OTM) concept, where a single human operator is responsible for monitoring and coordinating the tasks of multiple unmanned aerial vehicles (UAVs). This paper presents the development and evaluation of cognitive human-machine interfaces and interactions (CHMI2) supporting adaptive automation in OTM applications. A CHMI2 system comprises a network of neurophysiological sensors and machine-learning based models for inferring user cognitive states, as well as the adaptation engine containing a set of transition logics for control/display functions and discrete autonomy levels. Models of the user’s cognitive states are trained on past performance and neurophysiological data during an offline calibration phase, and subsequently used in the online adaptation phase for real-time inference of these cognitive states. To investigate adaptive automation in OTM applications, a scenario involving bushfire detection was developed where a single human operator is responsible for tasking multiple UAV platforms to search for and localize bushfires over a wide area. We present the architecture and design of the UAS simulation environment that was developed, together with various human-machine interface (HMI) formats and functions, to evaluate the CHMI2 system’s feasibility through human-in-the-loop (HITL) experiments. The CHMI2 module was subsequently integrated into the simulation environment, providing the sensing, inference, and adaptation capabilities needed to realise adaptive automation. HITL experiments were performed to verify the CHMI2 module’s functionalities in the offline calibration and online adaptation phases. In particular, results from the online adaptation phase showed that the system was able to support real-time inference and human-machine interface and interaction (HMI2) adaptation. However, the accuracy of the inferred workload was variable across the different participants (with a root mean squared error (RMSE) ranging from 0.2 to 0.6), partly due to the reduced number of neurophysiological features available as real-time inputs and also due to limited training stages in the offline calibration phase. To improve the performance of the system, future work will investigate the use of alternative machine learning techniques, additional neurophysiological input features, and a more extensive training stage.

Online Replanning With Human-in-the-Loop for Non-Prehensile Manipulation in Clutter — A Trajectory Optimization Based Approach

We are interested in the problem where a number of robots, in parallel, are trying to solve reaching through clutter problems in a simulated warehouse setting. In such a setting, we investigate the performance increase that can be achieved by using a human-in-the-loop providing guidance to robot planners. These manipulation problems are challenging for autonomous planners as they have to search for a solution in a high-dimensional space. In addition, physics simulators suffer from the uncertainty problem where a valid trajectory in simulation can be invalid when executing the trajectory in the real-world. To tackle these problems, we propose an online-replanning method with a human-in-the-loop. This system enables a robot to plan and execute a trajectory autonomously, but also to seek high-level suggestions from a human operator if required at any point during execution. This method aims to minimize the human effort required, thereby increasing the number of robots that can be guided in parallel by a single human operator. We performed experiments in simulation and on a real robot, using an experienced and a novice operator. Our results show a significant increase in performance when using our approach in a simulated warehouse scenario and six robots.

Development of an Operability Evaluation Framework for Remotely Controlled Ground Combat Vehicles in a Simulated Environment

This paper proposes a systematic framework for operability evaluation of remotely controlled ground combat systems (RGCS) in a simulated environment. The popular human-robot interaction metric used in unmanned vehicle systems is called fan-out (FO) and represents the maximum number of robots/vehicles that could be controlled by a single human operator. However, FO is inappropriate for systems with a lower level of automation where vehicles are remotely controlled by a human, such as RGCS. The theoretical background of the suggested framework is based on McRuer’s crossover model that was initially developed in the aviation domain for explaining pilot handling issues. In this study, an evaluation/analysis software prototype was developed, known as the RGCS operability evaluation tool in a simulated environment (ROPES). The ROPES was designed to be a simple tool for use by officers or researchers who only have intuitive understanding on the human adaptability. The ROPES includes two sub-modules; 1) an interactive interface for the configuration of the RGCS dynamic parameters and user interfaces and 2) a time-varying graphical display of system and human performance. Examples case studies demonstrate the advantage of the ROPES, and improvement points were identified for future development.

Agile Coordination and Assistive Collision Avoidance for Quadrotor Swarms Using Virtual Structures

This paper presents a method for controlling a swarm of quadrotors to perform agile interleaved maneuvers while holding a fixed relative formation, or transitioning between different formations. The method prevents collisions within the swarm, as well as between the quadrotors and static obstacles in the environment. The method is built upon the existing notion of a virtual structure, which serves as a framework with which to plan and execute complex interleaved trajectories, and also gives a simple, intuitive interface for a single human operator to control an arbitrarily large aerial swarm in real time. The virtual structure concept is integrated with differential flatness-based feedback control to give an end-to-end integrated swarm teleoperation system. Collision avoidance is achieved by using multiple layered potential fields. Our method is demonstrated in hardware experiments with groups of 3–5 quadrotors teleoperated by a single human operator, and simulations of 200 quadrotors teleoperated by a single human operator.

Human-autonomy teaming interface design considerations for multi-unmanned vehicle control

ABSTRACTFuture applications are envisioned in which a single human operator manages multiple heterogeneous unmanned vehicles (UVs) by working together with an autonomy teammate that consists of several intelligent decision-aiding agents/services. This article describes recent advancements in developing a new interface paradigm that will support human-autonomy teaming for air, ground, and surface (sea craft) UVs in defence of a military base. Several concise and integrated candidate control station interfaces are described by which the operator determines the role of autonomy in UV management using an adaptable automation control scheme. An extended play calling based control approach is used to support human-autonomy communication and teaming in managing how UV assets respond to potential threats (e.g. asset allocation, routing, and execution details). The design process for the interfaces is also described including: analysis of a base defence scenario used to guide this effort, consideration of ecological interface design constructs, and generation of UV and task-related pictorial symbology.

Fast, Automated Indoor Light Detection, Classification, and Measurement

Lighting is one of the largest power consumers in the United States and around the globe. To better understand how much energy lighting uses in a building, a lighting audit can be performed. Typically, this is a long and manual process, current solutions require significant effort on the part of the auditor. This paper develops a system using commercially available hardware and custom algorithms that enable a single human operator to quickly cover a large area while estimating light positions, type, and surface area. These tasks are accomplished with an error rate of 6.9% and 13.9%, respectively, with surface area estimation within about a factor of two.

Designing for human–agent collectives: display considerations

The adoption of unmanned systems is growing at a steady rate, with the promise of improved task effectiveness and decreased costs associated with an increasing multitude of operations. The added flexibility that could potentially enable a single operator to control multiple unmanned platforms is thus viewed as a potential game-changer in terms of both cost and effectiveness. The use of advanced technologies that facilitate the control of multiple systems must lie within control frameworks that allow the delegation of authority between the human and the machine(s). Agent-based systems have been used across different domains in order to offer support to human operators, either as a form of decision support offered to the human or to directly carry out behaviours that lead to the achievement of a defined goal. This paper discusses the need for adopting a human–agent interaction paradigm in order to facilitate an effective human–agent partnership. An example of this is discussed, in which a single human operator may supervise and control multiple unmanned platforms within an emergency response scenario.

Ecological Interface for Collaboration of Multiple UAVs in Remote Areas

Unmanned Aerial Vehicles (UAVs) can be used to access remote areas, e.g., for surveillance missions. Collaboration between them can help overcome communication constraints by building airborne relay networks that allow beyond line of sight communication. This research investigates whether a single human operator can supervise multiple UAVs in a collaborative surveillance task under communication constraints. We designed an ecological interface to support operators in their task and increase system flexibility. A preliminary humanin-the-loop study was done to investigate operator task performance and evaluate interface components. It was shown that operators are able to successfully operate surveillance missions under communication- and battery constraints. Participants did, however, not succeed to do this without separation conflicts and communication losses, which indicates that the interface lacks elements representing endurance and separation assurance. To an extent, the interface design turned out to be scalable, with a few remaining visualizations that cause clutter for large numbers of UAVs. More advanced ways of displaying information on request and grouping of select information is warranted to further improve the interface.

Scheduling Tasks for Human Operators in Monitoring and Surveillance Applications

Abstract: We consider a scheduling problem that arises in the context of surveillance or reconnaissance missions by multiple unmanned aerial vehicles (UAVs) where human operators would be present to perform tasks that require a high level of expertise and reasoning. Each UAV performing the mission provides a video stream of the targets it monitors to the human operator who is required to take critical decisions based on the video stream. A single human operator often examines the video stream from multiple UAVs constantly; this can cause his/her visual attention to drop below an acceptable level while performing the tasks. We propose an automated system to ease the operator's work load by breaking up the video streams into parts and scheduling a subset of them for the operator's inspection. We present the architecture for the system, identify the various components involved to build such a system, formulate the associated scheduling problem as a mixed-integer linear program (MILP) and solve the same. Preliminary results are also presented to corroborate the effectiveness of the formulation.

Dynamic Operator Overload: A Model for Predicting Workload During Supervisory Control

Crandall <etal xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"/> and Cummings & Mitchell introduced fan-out as a measure of the maximum number of robots a single human operator can supervise in a given single-human-multiple-robot system. Fan-out is based on the time constraints imposed by limitations of the robots and of the supervisor, e.g., limitations in attention. Adapting their work, we introduced a dynamic model of operator overload that predicts failures in supervisory control in real time, based on fluctuations in time constraints and in the supervisor's allocation of attention, as assessed by eye fixations. Operator overload was assessed by damage incurred by unmanned aerial vehicles when they traversed hazard areas. The model generalized well to variants of the baseline task. We then incorporated the model into the system where it predicted in real time, when an operator would fail to prevent vehicle damage and alerted the operator to the threat at those times. These model-based adaptive cues reduced the damage rate by one-half relative to a control condition with no cues.

Multi-UAV Velocity and Trajectory Scheduling Strategies for Target Classification by a Single Human Operator

This work addresses the problem of enabling a single human operator to individually inspect targets for a fixed amount of time in a reconnaissance mission. The task of the operator is to classify t...

Scheduling Multiple Uninhabited Aerial Vehicles for Target Classification by Single Human Operator

Scheduling Multiple Uninhabited Aerial Vehicles for Target Classification by Single Human Operator

Defining the Challenges Operators Face when Controlling Multiple Unmanned Vehicles

To achieve effective human-robot interaction (HRI) it is important to determine what types of supervisory control interfaces lead to optimal human-robot teaming. Research in HRI has demonstrated that operators controlling fewer robots against opponents of equal strength face greater challenges when control is restricted to only automation. Using human-in-the-loop evaluations of delegation-type interfaces, the present study examined the challenges and outcomes of a single operator supervising (1) more or less robots than a simulated adversary, with either a (2) flexible or restricted control interface. Testing was conducted with 12 paid participants using the RoboFlag simulation environment. Results from this experiment support past findings of execution timing deficiencies related to automation brittleness, and present new findings that indicate that successful teaming between a single human operator and a robotic team is affected by the number of robots and the type of interface.

Algorithms for cooperative multisensor surveillance

The Video Surveillance and Monitoring (VSAM) team at Carnegie Mellon University (CMU) has developed an end-to-end, multicamera surveillance system that allows a single human operator to monitor activities in a cluttered environment using a distributed network of active video sensors. Video understanding algorithms have been developed to automatically detect people and vehicles, seamlessly track them using a network of cooperating active sensors, determine their three-dimensional locations with respect to a geospatial site model, and present this information to a human operator who controls the system through a graphical user interface. The goal is to automatically collect and disseminate real-time information to improve the situational awareness of security providers and decision makers. The feasibility of real-time video surveillance has been demonstrated within a multicamera testbed system developed on the campus of CMU. This paper presents an overview of the issues and algorithms involved in creating this semiautonomous, multicamera surveillance system.