Manned-unmanned Teaming Research Articles

Battle Simulator for Multi-Robot Mission Simulation and Reinforcement Learning

As AI technology advances, interest in performing multi-robot autonomous missions for manned-unmanned teaming (MUM-T) is increasing. In order to develop autonomous mission performance technology for multiple robots, simulation technology that reflects the characteristics of real robots and can flexibly apply various missions is needed. Additionally, in order to solve complex non-linear tasks, an API must be provided to apply multi-robot reinforcement learning technology, which is currently under active research. In this study, we propose the campaign model to flexibly simulate the missions of multiple robots. We then discuss the results of developing a simulation environment that can be edited and run and provides a reinforcement learning API including acceleration performance. The proposed simulated control module and simulated environment were verified using an enemy infiltration scenario, and parallel processing performance for efficient reinforcement learning was confirmed through experiments.

Fault-tolerant security-efficiency combined authentication scheme for manned-unmanned teaming

Manned-unmanned teaming (MUM-T) is an emerging network system which interconnects manned aerial vehicles (MAVs) with unmanned aerial vehicles (UAVs) to enhance mission effectiveness and reduce workloads. Like other wireless systems, MUM-T is prone to attacks from the open communication channel. Therefore, an authentication scheme is required to establish secure and trusted communication between the MAV and the UAV. However, existing schemes fail to provide adequate security features and necessary efficiency, mostly due to their incomprehensive threat modeling and improper use of authentication techniques. Moreover, traditional centralized architecture of the authentication server leads to the single point of failure (SPOF) problem, which jeopardizes the robustness and scalability of MUM-T. In this paper, we propose a fault-tolerant authentication scheme for MUM-T that will solve these problems. To enhance its security, we construct a new threat model consisting of adversary's capabilities, security features, and security challenges to ensure the security of a scheme under combination attacks. To preserve the highest possible efficiency, we propose the design principle of using lightweight primitives for authentication and applying public key operations in key establishment. To address the SPOF problem, we employ a distributed fault-tolerant mechanism to share registration information within authentication servers and defend against faulty nodes. As is demonstrated by the security proof and performance comparison, our scheme succeeds in improving security and reducing overall costs, which provides a better solution than existing schemes.

Modelling situated intent for human-autonomy teaming: a human-centric approach

Entering an era where humans and synthetic agents are supposed to collaborate and cooperate, adequate models of human intent are crucial for coordinated teamwork. Unfortunately, although there is a need for such models, the concept of intent is ambiguous and approaches to model intent from a human-centric perspective are scarce. Building upon theoretical and methodological foundations, this study aims to address these gaps by presenting a conceptualisation of intent alongside an approach. Specifically, leveraging the six levels of cognitive control outlined in the Joint Control Framework, a provisional model of human intent alongside a defined and operationalised concept is presented. Building on these foundations, a novel approach is proposed. Utilising seven scenario-based interviews, the value of these contributions is demonstrated through an example case in the context of Manned-Unmanned Teaming. It is concluded that intent should be understood as a multi-faceted concept shaped by situated constraints, where intent is formed through a commitment to choices by context-situation and means-end reasoning. It is also concluded that the approach is useful, particularly since it can glean insights from choices considered and committed, both being essential in the design of synthetic teammates’ capability to adapt to their human partner’s agency.

My synthetic wingman must understand me: modelling intent for future manned–unmanned teaming

With advances in artificial intelligence, machine learning, and cognitive modelling, unmanned aircraft are expected to act as human-like wingmen in the near future. For fluent and effective manned–unmanned teaming, synthetic wingmen must be able to account for and adapt to their partners’ intent with little or no communication. To enable such abilities, it becomes crucial to identify the requirements that makes intent explainable to synthetic wingmen, necessitating approaches to describe and analyse intent from a human-centric perspective. To address this issue, this paper reports on findings from using Work Domain Analysis to design and analyse models of situated intent with six levels of cognitive control (frames, effects, values, generic, implementations, and physical). Through a literature review and seven subject matter expert interviews, a synthesized model was designed to represent fighter pilots’ intent in a manned–unmanned teaming scenario. Using the synthesized model as the context, a transfer of control and a link loss situation were further described and analysed. Experiences show that Work Domain Analysis can provide a practical and applicable means to model situated intent, particularly since designed models can be re-utilised to model intent in similar situations. Furthermore, the model analyses show the importance of accounting for fighter pilots’ adopted frames since small variations of the framing of the situations can propagate throughout the model resulting in conflicting or inconsistent intent. The paper concludes that synthetic wingmen must be able to reason about all six levels of cognitive control, requiring a more holistic approach to make intent explainable.

Effectiveness evaluation of complex air combat system based on capability-service model

The core feature of a complex air combat system is manned-unmanned teaming (MUM-T). It will be the basic air combat formation of the future Air Force. The effectiveness evaluation of a complex air combat system is a challenging task. It will profoundly affect the future direction of air combat systems. Firstly, this paper describes the concept, components, and winning mechanism of complex air combat. Secondly, drawing on the ideas of mosaic warfare effectiveness evaluation, building an agent-based capability-service effectiveness evaluation model, solving the model by heuristic rules. We define an effectiveness evaluation metric to measure the effectiveness of a complex air combat system from different perspectives. Finally, simulation results under different scenarios show that the platforms’ orchestration relationship is essential in developing a complex air combat system.

Heuristics for Multi-Vehicle Routing Problem Considering Human-Robot Interactions

Autonomous mobile robots (AMRs) are being used extensively in civilian and military applications for applications such as underground mining, nuclear plant operations, planetary exploration, intelligence, surveillance and reconnaissance (ISR) missions and manned-unmanned teaming. We consider a multi-objective, multiple-vehicle routing problem in which teams of manned ground vehicles (MGVs) and AMRs are deployed respectively in a leader-follower framework to execute missions with differing requirements for MGVs and AMRs while considering human-robot interactions (HRI). HRI studies highlight the costs of managing a team of follower AMRs by a leader MGV. This paper aims to compute feasible visit sequences, replenishments, team compositions and number of MGV-AMR teams deployed such that the requirements for MGVs and AMRs for the missions are met and the routing, replenishment, HRI and team deployment costs are at minimum. The problem is first modeled as a a mixed-integer linear program (MILP) that can be solved to optimality by off-the-shelf commercial solvers for small-sized instances. For larger instances, a variable neighborhood search algorithm is offered to compute near optimal solutions and address the challenges that arise when solving the combinatorial multi-objective routing optimization problem. Finally, computational experiments that corroborate the effectiveness of the proposed algorithms are presented.

Development of an MUM-T Integrated Simulation Platform

In this paper, we present a platform that can simulate Manned-UnManned Teaming (MUM-T) situations using various hardware and software being integrated. MUM-T, which combines unmanned aerial vehicles (UAVs) with an existing manned vehicle (MV), is growing into an indispensable strategy thanks to enormous advantages it provides. In order to test this indispensable strategy which involves flight control algorithms being applied for various MUM-T situations, it is certainly needed to run and evaluate three-dimensional (3D) MV-UAVs combined simulations in realistic environments. However, there is no such simulator that enables such 3D MUM-T simulations as well as the Human Interface (HI) for both MV pilot and the mission manager at a ground station. In this paper, we propose an MUM-T integrated simulation platform (MUM-T ISP) that combines a single full-scale helicopter simulator (for MV simulation) and multiple Iris/PX4-Autopilot (for UAVs simulation), along with specially developed HI devices (TM and TV). Tactical Monitor (TM), one of the HI devices, is a touchable tablet which allows the MV pilot to interact with UAVs by simple actions (e.g. touching a button), and the other HI device called Tactical Viewer (TV) allows the mission manager at a ground station to closely monitor the MUM-T situation via realistic 3D displays of MV and UAVs. The proposed MUM-T ISP is tested on several mission scenarios and then evaluated by actual pilots and the mission manager based on sensible criteria. It is expected that the proposed simulation platform may serve well as a training tool for prospective MV pilots working with UAVs, and also as a research tool for flight engineers developing control algorithms that make MV and UAVs work together.

The Development of Rule-based AI Engagement Model for Air-to-Air Combat Simulation

Since the concept of Manned-UnManned Teaming(MUM-T) and Unmanned Aircraft System(UAS) can efficiently respond to rapidly changing battle space, many studies are being conducted as key components of the mosaic warfare environment. In this paper, we propose a rule-based AI engagement model based on Basic Fighter Maneuver(BFM) capable of Within-Visual-Range(WVR) air-to-air combat and a simulation environment in which human pilots can participate. In order to develop a rule-based AI engagement model that can pilot a fighter with a 6-DOF dynamics model, tactical manuals and human pilot experience were configured as knowledge specifications and modeled as a behavior tree structure. Based on this, we improved the shortcomings of existing air combat models. The proposed model not only showed a 100 % winning rate in engagement with human pilots, but also visualized decision-making processes such as tactical situations and maneuvering behaviors in real time. We expect that the results of this research will serve as a basis for development of various AI-based engagement models and simulators for human pilot training and embedded software test platform for fighter.

Human-Machine Shared Situational Awareness

Abstract As unmanned platforms are more and more present in battlefield, alongside humans, in different configurations (from tele-operated to fully autonomous platforms, manned-unmanned teaming, swarming, etc.) and domains (space, air, land, sea, and underwater), it is also necessary for decision-making process to adapt to this new reality. This process will no longer be exclusively human and this requires that humans and machines share a common, meaningful, and timely understanding of the context in which they act and interact, namely a common situational awareness. We will investigate how this objective could be achieved by means of AI-related techniques of inference and reasoning based on ontologies that will enable all levels of information sharing (data, knowledge, and models) among all participants, humans and non-humans. The success of this undertaking shall be reflected by the achievement of a high level of interoperability between heterogeneous entities, within which they will be able to take advantage of each other’s best developed abilities.

행위자 기반 모델링을 이용한 유·무인 복합체계의 전투효과 분석

This study tried to suggest implications for effective team composition in the manned-unmanned teaming system by analyzing the effect of the combination of manned and unmanned platforms on the combat effect in the manned-unmanned teaming system. In this study, by changing the ratio of amphibious operation vehicles (KAAV), unmanned ground vehicles (UGV), and unmanned aerial vehicles (UAV), the combat survival rate was calculated in a standard operation scenario using AnyLogic, an agent-based modeling program. As a result of the analysis, it was confirmed that an appropriate combination is possible to optimize the combat efficiency of the unmanned weapon system, the unmanned aerial vehicle (UGV) and the unmanned aerial vehicle (UAV), compared to the use of the current manned amphibious assault vehicle (KAAV) alone. It is expected that the research results will be a reference for M&S analysis for each stage of weapon system acquisition and establish the operational concept of the manned-unmanned teaming system that will be promoted to foster the advanced science and technology forces pursued by Defense Innovation 4.0.

Towards Human Objective Real-Time Trust of Autonomy Measures for Combat Aviation

ABSTRACT Objective Can objective trust indicators be measured and computed in real time in a within visual range aerial combat “dogfight” scenario consisting of a manned autonomous own-ship versus an adversary? Background Previous research has focused on human trust of automation during simulated ground combat and human trust of civil aviation automation. Those studies largely consisted of subjective measures of trust analyzed post-hoc. Human trust of autonomy in high-risk situations such as dogfighting had not yet been studied. Method Nine evaluation pilots participated in an experiment that consisted of nine operationally relevant live-flight vignettes during which they conducted a mission manager task while simultaneously monitoring an autonomous agent controlling their own-ship in an aerial dogfight. Eye tracking, objective real-time mental workload, and mission manager task performance were all recorded. Results Potential indicators of trust were objectively measured in real-time as a function of autonomy disengagement, human cross-check ratio of autonomy, and real-time objective mental workload, as validated by traditional subjective trust measures. Measured trust still needs to be categorized as appropriate trust, overtrust, or undertrust in future work. Conclusion This first-ever real-time measure of objective indicators of trust in operationally relevant live-flight paves the way for determining appropriate human trust of autonomy in high-risk situations such as dogfighting. These measures could have utility in high-risk manned-unmanned teaming applications such as working with robots and automated trading.

The Development of Air Escort Tactics for High-Value Airborne Assets Using Manned-Unmanned Teaming and the Study on Effective Force Disposition Using M&S

As the role of high-value air assets(e.g., AWACS, JSTARS, Rivet Joint, E-2) becomes more critical in modern warfare, the air escort for these assets blocking attacks from any potential enemy fighter also becomes vital. Without the escort, the operations of the assets become restricted. However, such an escort is not always possible due to the limited flight time of the escort fighters. In this paper, we introduce an escort tactics for high-value air assets performed by the manned-unmanned teaming composed of a transport aircraft and UAVs(unmanned aerial vehicles). In this tactics, the transport aircraft plays the role of an aircraft carrier, which carries, launches, and retrieves the UAVs. The missions of UAVs in this tactics are to detect and engage enemy fighters. We also introduce the simulation result of this tactics to identify the UAVs' required capabilities and optimal maneuvering.

On the Robust Network Design for MUM-T

Manned-unmanned teaming (MUM-T) is defined as the coupling of one or more manned platforms with a network of unmanned platforms to perform a certain mission. In this article, we are concerned with designing robust MUM-T networks in terms of a network stability margin measure. First, we consider an original network of unmanned platforms, and then try to find nodes such that replacing them with manned platforms (leaders) maximally increases the resultant MUM-T network stability margin. We also do mathematical analysis and find several bounds on the stability margin of MUM-T, using which greedy and degree-based schemes for the sake of computational efficiency are proposed. In the end, several numerical tests are carried out to demonstrate the correctness of the performed analysis as well as the effectiveness of the proposed schemes.

The Specification of Air-to-Air Combat Tactics Using UML Sequence Diagram

Air force air-to-air combat tactics are occurring at a high speed in three-dimensional space. The specification of the tactics requires dealing with a quite amount of information, which makes it a challenge to accurately describe the maneuvering procedure of the tactics. The specification of air-to-air tactics using natural languages is not suitable because of the intrinsic ambiguity of natural languages. Therefore, this paper proposes an approach of using UML Sequence Diagram to describe air-to-air combat tactics. Since the current Sequence Diagram notation is not sufficient to express all aspects of the tactics, we extend the syntax of the Sequence Diagram to accommodate the required features of air-to-air combat tactics. We evaluate the applicability of the extended Sequence Diagram to air-to-air combat tactics using a case example, that is the manned-unmanned teaming combat tactic. The result shows that Sequence Diagram specification is more advantageous than natural language specification in terms of readability, conciseness, and accuracy. However, the expressiveness of the Sequence Diagram is evaluated to be less powerful than natural language, requiring further study to address this issue.

Manned-Unmanned Teaming: Research and Applications Panel

This panel will discuss issues related to Manned-Unmanned Teaming (MUMT) technologies. Panelists were selected to represent diverse topics and each will provide a unique perspective on the MUMT challenge space. Joseph Lyons will frame the discussion and introduce the panelists. Each panelist will provide an overview of the MUMT research/applications they are involved in. Chris Miller will discuss an ongoing project looking at MUMT applications broadly across the enterprise and is seeking to identify the core systemic tenants of MUMT and metrics to gauge MUMT effectiveness. Jay Shively will discuss MUMT challenges in the context of UAS operations in the National aerospace. Nancy Cooke will discuss several MUMT research projects that emphasize teaming and associated research challenges. Col. Dan Javorsek will discuss recent MUMT programs at DARPA as well as where MUMT technologies can support Air Force applications. Phillip Walker will discuss the DARPA OFFSET program and human-swarm interactions, including human factor considerations of large swarm demonstration events.

Mental Workload Estimation Based on Physiological Features for Pilot-UAV Teaming Applications.

Manned-Unmanned Teaming (MUM-T) can be defined as the teaming of aerial robots (artificial agents) along with a human pilot (natural agent), in which the human agent is not an authoritative controller but rather a cooperative team player. To our knowledge, no study has yet evaluated the impact of MUM-T scenarios on operators' mental workload (MW) using a neuroergonomic approach (i.e., using physiological measures), nor provided a MW estimation through classification applied on those measures. Moreover, the impact of the non-stationarity of the physiological signal is seldom taken into account in classification pipelines, particularly regarding the validation design. Therefore this study was designed with two goals: (i) to characterize and estimate MW in a MUM-T setting based on physiological signals; (ii) to assess the impact of the validation procedure on classification accuracy. In this context, a search and rescue (S&R) scenario was developed in which 14 participants played the role of a pilot cooperating with three UAVs (Unmanned Aerial Vehicles). Missions were designed to induce high and low MW levels, which were evaluated using self-reported, behavioral and physiological measures (i.e., cerebral, cardiac, and oculomotor features). Supervised classification pipelines based on various combinations of these physiological features were benchmarked, and two validation procedures were compared (i.e., a traditional one that does not take time into account vs. an ecological one that does). The main results are: (i) a significant impact of MW on all measures, (ii) a higher intra-subject classification accuracy (75%) reached using ECG features alone or in combination with EEG and ET ones with the Adaboost, Linear Discriminant Analysis or the Support Vector Machine classifiers. However this was only true with the traditional validation. There was a significant drop in classification accuracy using the ecological one. Interestingly, inter-subject classification with ecological validation (59.8%) surpassed both intra-subject with ecological and inter-subject with traditional validation. These results highlight the need for further developments to perform MW monitoring in such operational contexts.

Experimental evaluation of tasking and teaming design patterns for human delegation of unmanned vehicles

This work discusses different approaches for the cooperation between humans as a supervisor and multiple unmanned vehicles (UVs). We evaluated the most promising approach experimentally with expert pilots of the German Air Force. The co-agency of humans and highly automated unmanned systems (i.e., human autonomy teaming, HAT) is described by the use of a design and description language for HAT design patterns. This design language is used to differentiate control modes for tasking, teaming, and swarming of UVs. The different control modes are then combined in a planner agent (PA) design pattern that further enables the UV guidance on scalable delegation levels from a single individual up to a team. The desired system behavior and interaction concept of the PA for these scalable delegation levels is then transferred to the domain of manned-unmanned teaming in fighter aircraft missions. To demonstrate the applicability of the system, we implemented the concept into our fast-jet simulator of the Institute of Flight Systems (IFS) and conducted an experimental campaign with expert pilots. The results of the experiment showed (1) task delegation with the PA design pattern is faster and reduces the error potential; (2) scalable delegation levels enable a pilot and situation-specific task delegation; (3) the delegation of teams is faster and reduces the error potential; however, in some situations, deeper access through the scalable delegation levels is needed; (4) the concept is intuitive and the transparency and trust in UVs and swarms were very high; and (5) the pilots could imagine operating such systems in the future. Overall speaking the presented PA design pattern is suited for the guidance of UVs and the scalable delegation levels are beneficial.

SEAD 임무 수행을 위한 유·무인기 협업 체계의 임무적합도 평가 방법론 연구

본 연구에서는 협업기반의 임무 수행 시 유·무인전투기 각각의 역할을 도출하고 복합 운용방안을 제시하기 위한 임무 적합도 평가 방법론을 제시하였다. 유·무인전투기 복합운용을 위한 대표 임무로 SEAD 임무를 선정하고 기체 성능, 장비성능 그리고 자율성 수준을 고려한 성능평가표를 작성하여 유인전투기와 무인전투기의 성능을 비교하였다. SEAD의 세부 임무절차를 구분하여 성능평가표의 항목과 결합된 임무적합도 평가지표를 설계하고, 전문가 설문을 통해 임무적합도를 평가하였다. 평가결과를 통해 세부 임무의 위험성과 중요도에 따라 무인기와 유인기의 적정 역할을 설정할 수 있었고, 이를 바탕으로 조종사의 임무부담을 줄이도록 임무, 통제권, 유·무인전투기의 임무 및 기동을 반영한 시간 기반의 임무계획을 수립할 수 있었다.

Friend or frenemy? The role of trust in human-machine teaming and lethal autonomous weapons systems

ABSTRACT This article explores the imprecise boundary between Lethal Autonomous Weapons Systems (LAWS) and Human-Machine Teaming – as a subset of Human-Machine Interaction – and the extent both are emerging as a point of concern (and option) in military and security policy debates. As the development of Human-Machine Teaming relates to artificial intelligence (AI) capabilities there also exists an area of concern pertaining to reliability and confidence, particularly in the heat of battle. Also known as Manned-Unmanned Teaming, Human-Machine Teaming attempts to engender trust and collaborative partnerships with robots and algorithms. Clearly the prospect of LAWS in recent times, or so-called ‘killer robots,’ has raised questions relating to the degree such devices can be trusted to select and engage targets without further human intervention. Aside from examining the ‘trust factor,’ the article also considers security threats posed by both state and non-state actors and the complicit yet inadvertent role multinational corporations play in such developments where civilian technology is modified for dual-purposes. The effectiveness of government regulation over AI, including whether AI can be ‘nationalised’ for national security reasons, will also be examined as part of AI non-proliferation.

Transparency for a Workload-Adaptive Cognitive Agent in a Manned–Unmanned Teaming Application

This study focuses on the transparent design of a cognitive agent to enhance situation awareness in two aspects of a human-agent teaming application: assisted system management and mixed-initiative mission planning. Adaptive and complex agent behavior might result in the failure to comprehend resulting interventions, a decrease in trust, and a loss of overall situation awareness. This study describes and validates a concept for transparent agent design by adopting the transparency strategies proposed by the “situation awareness-based agent transparency model.” The overall objective was to improve the human operator's perception, comprehension, and projection of the agent's support. The concept was applied to the prototype of a workload-adaptive cognitive agent, which supports a helicopter crew during mission planning and execution in complex and dynamically changing multi-vehicle missions. A human-in-the-loop experiment revealed enhancements in situation awareness and performance. Subjective trust measures implied an increase in human-like characteristics of the cognitive agent. The results and the potential for further research are discussed.