Human-swarm Interaction Research Articles

Extracting Human Levels of Trust in Human–Swarm Interaction Using EEG Signals

Extracting Human Levels of Trust in Human–Swarm Interaction Using EEG Signals

Human–Swarm Interaction Simulation Platform to Assess Sense of Agency

Human–Swarm Interaction Simulation Platform to Assess Sense of Agency

Game-Based Approximate Optimal Motion Planning for Safe Human-Swarm Interaction.

Safety as a fundamental requirement for human-swarm interaction has attracted a lot of attention in recent years. Most existing approaches solve a constrained optimization problem at each time step, which has a high real-time requirement. To deal with this challenge, this article formulates the safe human-swarm interaction problem as a Stackerberg-Nash game, in which the optimization is performed over the entire time domain. The leader robot is supposed to be in a dominant position, interacting directly with the human operator to realize trajectory tracking and responsible for guiding the swarm to avoid obstacles. The follower robots always take their best responses to leader's behavior with the purpose of achieving the desired formation. Following the bottom-up principle, we first design the best-response controllers, that is, Nash equilibrium strategies, for the followers. Then, a Lyapunov-like control barrier function-based safety controller and a learning-based formation tracking controller for the leader are designed to realize safe and robust cooperation. We show that the designed controllers can make the robotic swarms move in a desired geometric formation following the human command and modify their motion trajectories autonomously when the human command is unsafe. The effectiveness of the proposed approach is verified through simulation and experiments. The experiment results further show that safety can still be guaranteed even when there exists a dynamic obstacle.

Integrated system architecture with mixed-reality user interface for virtual-physical hybrid swarm simulations

This paper introduces a hybrid robotic swarm system architecture that combines virtual and physical components and enables human–swarm interaction through mixed reality (MR) devices. The system comprises three main modules: (1) the virtual module, which simulates robotic agents, (2) the physical module, consisting of real robotic agents, and (3) the user interface (UI) module. To facilitate communication between the modules, the UI module connects with the virtual module using Photon Network and with the physical module through the Robot Operating System (ROS) bridge. Additionally, the virtual and physical modules communicate via the ROS bridge. The virtual and physical agents form a hybrid swarm by integrating these three modules. The human–swarm interface based on MR technology enables one or multiple human users to interact with the swarm in various ways. Users can create and assign tasks, monitor real-time swarm status and activities, or control and interact with specific robotic agents. To validate the system-level integration and embedded swarm functions, two experimental demonstrations were conducted: (a) two users playing planner and observer roles, assigning five tasks for the swarm to allocate the tasks autonomously and execute them, and (b) a single user interacting with the hybrid swarm consisting of two physical agents and 170 virtual agents by creating and assigning a task list and then controlling one of the physical robots to complete a target identification mission.

Autonomous UAV Swarms: Powering Up with AI

Abstract: This study offers a thorough analysis of autonomous UAV swarming, focusing on its merits, important technologies, applications, difficulties, and prospects for the future. The idea of autonomous UAV swarming entails the coordination and decentralised cooperation of several UAVs to accomplish common objectives. The foundations of UAV swarming are thoroughly covered, including its essential ideas, advantages, architectures, and swarm behaviour. We emphasise the role of artificial intelligence (AI), more especially reinforcement learning (RL) and particle swarm optimisation (PSO), in promoting thoughtful decision-making, task allocation, and coordination within UAV swarms. The study examines the many uses of autonomous UAV swarming in fields including infrastructure inspection, search and rescue, agriculture, and disaster response. It also discusses UAV swarming's difficulties and restrictions, such as scalability, communication, fault tolerance, and ethical issues. In addition to standardisation initiatives, the study suggests future research possibilities for swarm intelligence, edge computing, cognitive capacities, and human swarm interaction.

Swarm Metaverse for Multi-Level Autonomy Using Digital Twins.

Robot swarms are becoming popular in domains that require spatial coordination. Effective human control over swarm members is pivotal for ensuring swarm behaviours align with the dynamic needs of the system. Several techniques have been proposed for scalable human-swarm interaction. However, these techniques were mostly developed in simple simulation environments without guidance on how to scale them up to the real world. This paper addresses this research gap by proposing a metaverse for scalable control of robot swarms and an adaptive framework for different levels of autonomy. In the metaverse, the physical/real world of a swarm symbiotically blends with a virtual world formed from digital twins representing each swarm member and logical control agents. The proposed metaverse drastically decreases swarm control complexity due to human reliance on only a few virtual agents, with each agent dynamically actuating on a sub-swarm. The utility of the metaverse is demonstrated by a case study where humans controlled a swarm of uncrewed ground vehicles (UGVs) using gestural communication, and via a single virtual uncrewed aerial vehicle (UAV). The results show that humans could successfully control the swarm under two different levels of autonomy, while task performance increases as autonomy increases.

A methodological approach for the analysis and design of Human–Swarm interactions based upon feedback loops

A new era of humans interacting seamlessly with devices has began. Systems composed of humans and devices are being deployed over a wide range of domains such as smart grids, smart cities, industry 4.0, among others. In many cases, the algorithms proposed to achieve the intelligent decision-making are based on decentralized and collective behaviours including swarm intelligence. Although clear benefits can be attributed to these approaches, the analysis and design of such systems is a difficult task. Furthermore, enabling humans (e.g. operators) to guide and influence the swarms is still an open research question. In this paper we propose a methodological approach to enable swarms to be influenced by humans with minimal intervention or modification (if any) to the original underlying principles. This approach allows to translate high level goals, as conceptualized by human operators, into influencing factors to the swarm algorithms; and thus allowing humans to guide and interpret the resolution process. To illustrate and evaluate this proposal, we apply the methodology to a swarm of drones using Particle Swarm Optimization (PSO) algorithm for search and rescue operations. Experimentations show that, by using the resulting system, humans are able to influence the PSO algorithm overall results using high level abstractions. Even more, the PSO algorithm mechanics are not modified and influences are derived solely by following the proposed steps. Finally, we discuss the limitations of the approach and application to other swarm intelligence algorithms.

Distributed Control for Human-Swarm Interaction In Non-Convex Environments using Gaussian Mixture Models

This paper presents a novel implementation of a human-swarm interface that allows humans to define an area with desired shape to be reached by a multi-robot system. Human-swarm interaction can be useful in order to exploit human intelligence and knowledge for the operation of swarm robots. The proposed work deals with limitations usually met when dealing with real-world implementation, e.g. limited sensing capabilities of the agents and hard conditions where communication is difficult or even completely denied. Gaussian Mixture Models are exploited in order to define an appropriate probability density function of the environment based on the area selected by a human operator. Then, velocity input for each robot is calculated in a distributed manner using Voronoi tessellation and Lloyd's algorithm. Finally, results of both virtual and real-world tests are presented, showing the final configuration reached by the multi-robot system in comparison with the desired region defined on the graphical interface.

Opportunities and Challenges in Human-Swarm Interaction: Systematic Review and Research Implications

We conducted a Systematic Literature Review on scientific papers that examined the interaction between operators and drone swarms based on the use of a command and control center. We present the results of a meta-analysis of nine scientific papers published in the ACM DL and IEEE Xplore databases. Our findings show that research on human-drone swarm interaction shows a disproportionate interest in hand gestures compared to other input modalities for drone swarm control. Furthermore, all articles reviewed exclusively explored gestures and the size of the swarm used in the studies was limited, with a median of 3.0 and an average of 3.8 drones per study. We compiled an inventory of interaction modalities, recognition techniques, and application types from the scientific literature, which is presented in this paper. On the basis of our findings, we propose four areas for future research that can guide scientific investigations and practical developments in this field.

The Effect of Asset Degradation on Trust in Swarms: A Reexamination of System-Wide Trust in Human-Swarm Interaction.

The effects of asset degradation on trust in human-swarm interaction were investigated through the lens of system-wide trust theory. Researchers have begun investigating contextual features that shape human interactions with robotic swarms-systems comprising assets that coordinate behavior based on their nearest neighbors. Recent work has begun investigating how human trust toward swarms is affected by asset degradation through the lens of system-wide trust theory, but these studies have been marked by several limitations. In an online study, the current work manipulated asset degradation and measured trust-relevant criteria in a within-subjects design and addressed the limitations of past work. Controlling for swarm performance (i.e., target acquisition), asset degradation and trust (i.e., reliance intentions) in swarms were negatively related. In addition, as degradation increased, perceptions of swarm cohesion, obstacle avoidance, target acquisition, and terrain exploration efficiency decreased, the latter two of which (coupled with the reliance intentions criterion) support the tenets of system-wide trust theory as well as replicate and extend past work on the effects of asset degradation on trust in swarms. Human-swarm interaction is a context in which system-wide trust is relevant, and future work ought to investigate how to calibrate human trust toward swarm systems. Based on these findings, design professionals should prioritize ways to depict swarm performance and system health such that humans do not abandon trust in systems that are still functional yet not over-trust those systems which are indeed performing poorly.

Characterizing motion prediction in small autonomous swarms

The use of robotic swarms has become increasingly common in research, industrial, and military domains for tasks such as collective exploration, coordinated movement, and collective localization. Despite the expanded use of robotic swarms, little is known about how swarms are perceived by human operators. To characterize human-swarm interactions, we evaluate how operators perceive swarm characteristics, including movement patterns, control schemes, and occlusion. In a series of experiments manipulating movement patterns and control schemes, participants tracked swarms on a computer screen until they were occluded from view, at which point participants were instructed to estimate the spatiotemporal dynamics of the occluded swarm by mouse click. In addition to capturing mouse click responses, eye tracking was used to capture participants eye movements while visually tracking swarms. We observed that manipulating control schemes had minimal impact on the perception of swarms, and that swarms are easier to track when they are visible compared to when they were occluded. Regarding swarm movements, a complex pattern of data emerged. For example, eye tracking indicates that participants more closely track a swarm in an arc pattern compared to sinusoid and linear movement patterns. When evaluating behavioral click-responses, data show that time is underestimated, and that spatial accuracy is reduced in complex patterns. Results suggest that measures of performance may capture different patterns of behavior, underscoring the need for multiple measures to accurately characterize performance. In addition, the lack of generalizable data across different movement patterns highlights the complexity involved in the perception of swarms of objects.

Robot swarms as embodied extensions of humans

In order to harness the potential of robot swarms, humans must be able to control decentralized and self-organizing multi-agent systems. The field of human-swarm interaction investigates how humans with their centralized information and control demands can be paired with the decentralized architectures of robot swarms. Recently, we have proposed an architecture which aims at balancing the trade-offs between centralized and decentralized control and communication in human-swarm interaction while joining the relative capabilities of human cognition and swarm intelligence. In this article, we expand on this architecture by discussing a list of design requirements that must be met in order to make robot swarms embodied extensions of humans that act and feel like natural body parts. We also conceptually show how some of the design requirements could be implemented by simple rules when we take the world’s structure into account.

Augmented reality for human–swarm interaction in a swarm-robotic chemistry simulation

We present a novel augmented reality (AR) framework to show relevant information about swarm dynamics to a user in the absence of markers by using blinking frequency to distinguish between groups in the swarm. In order to distinguish between groups, clusters of the same group are identified by blinking at a specific time interval that is distinct from the time interval at which their neighbors blink. The problem is thus to find blinking sequences that are distinct for each group with respect to the group’s neighbors. Selecting an appropriate sequence is an instance of the distributed graph coloring problem, which can be solved in $$O(\log (n))$$ time with n being the number of robots involved. We demonstrate our approach using a swarm chemistry simulation in which robots simulate individual atoms that form molecules following the rules of chemistry. An AR display is then used to display information about the internal state of individual swarm members as well as their topological relationship, corresponding to molecular bonds in a context that uses robot swarms to teach basic chemistry concepts.

Characterization of Indicators for Adaptive Human-Swarm Teaming.

Swarm systems consist of large numbers of agents that collaborate autonomously. With an appropriate level of human control, swarm systems could be applied in a variety of contexts ranging from urban search and rescue situations to cyber defence. However, the successful deployment of the swarm in such applications is conditioned by the effective coupling between human and swarm. While adaptive autonomy promises to provide enhanced performance in human-machine interaction, distinct factors must be considered for its implementation within human-swarm interaction. This paper reviews the multidisciplinary literature on different aspects contributing to the facilitation of adaptive autonomy in human-swarm interaction. Specifically, five aspects that are necessary for an adaptive agent to operate properly are considered and discussed, including mission objectives, interaction, mission complexity, automation levels, and human states. We distill the corresponding indicators in each of the five aspects, and propose a framework, named MICAH (i.e., Mission-Interaction-Complexity-Automation-Human), which maps the primitive state indicators needed for adaptive human-swarm teaming.

Designing a Virtual Reality Testbed for Direct Human-Swarm Interaction in Aquatic Species Monitoring

Swarm robotics is a natural candidate for monitoring large and complex environments. With the aid of a human who is directly involved in monitoring and search using a teleoperated robot, swarms of autonomous robots can solve problems related to navigation and changing objectives. In this context, understanding the role of nonverbal interactions have the potential to enable rapid and bidirectional communication between humans and robots. Quantifying such interactions however, requires a repeatable and engaging experimental setup that can track human action and cognition. This paper describes a desktop virtual reality testbed designed specifically to quantify human actions, perception, and cognitive load in real time during a monitoring mission. Motivated by recent deployment of underwater robots to monitor invasive species, the testbed is designed to mimic an underwater environment within the Great Lakes with five species of fish whose appearance, locomotion, and behavior are modeled based on videos from the field. Brain activity and pupillometry data are recorded synchronously in real time to aid in the measurement of cognitive load of the human operator. To quantify human perception, empirical data of visual acuity from the literature is used to model virtual object recognition. The capabilities of the testbed are evaluated in terms of frame rate achieved as a function of number of fish and robots in the environment, and demonstrated through two examples highlighting possible uses in human-swarm interaction studies.

Designing a User-Centered Interaction Interface for Human–Swarm Teaming

A key challenge in human–swarm interaction is to design a usable interface that allows the human operators to monitor and control a scalable swarm. In our study, we restrict the interactions to only one-to-one communications in local neighborhoods between UAV-UAV and operator-UAV. This type of proximal interactions will decrease the cognitive complexity of the human–swarm interaction to O(1). In this paper, a user study with 100 participants provides evidence that visualizing a swarm as a heat map is more effective in addressing usability and acceptance in human–swarm interaction. We designed an interactive interface based on the users’ preference and proposed a controlling mechanism that allows a human operator to control a large swarm of UAVs. We evaluated the proposed interaction interface with a complementary user study. Our testbed and results establish a benchmark to study human–swarm interaction where a scalable swarm can be managed by a single operator.

“Is something amiss?” Investigating individuals’ competence in estimating swarm degradation

Robotic swarms comprise component assets operating via local control algorithms which emulate natural swarming behaviors. Scientists are beginning to focus on the human-centered topic of human-swarm interaction. In a novel within-subjects design, we followed this comparatively nascent focus and investigated whether people can detect swarm degradations in assets flocking via consensus, their accuracy in estimating those degradations, and their confidence in those estimates. We also assessed open-ended responses to shed light on the strategies people may use to detect swarm degradations. Participants were recruited online and viewed 21 randomized simulations, each 30 seconds in duration with varying proportions of asset degradation. Results showed that the proportion of asset degradation did have an effect on the aforementioned criteria. Qualitative themes showed preliminary evidence that participants used common strategies to detect the time and degree of swarm degradation. However, we did not find evidence of a linear effect of the degradation manipulation on criteria of interest, which did not support our expectations. We discuss limitations and future research perspectives in detail, which we believe provide fodder for future work to investigate human-swarm interaction at a more granular level.

Exploring the Effects of Swarm Degradations on Trustworthiness Perceptions, Reliance Intentions, and Reliance Behaviors

Swarms comprise robotic assets operating autonomously through local control laws. Research on human-swarm interaction (HSwI) investigates how human operators collaborate with swarms to accomplish shared goals. Researchers have begun to investigate the role of trust in HSwI, specifically which aspects of robotic swarms affect human trust. Through a human factors lens, the present research builds on earlier HSwI work and investigates the effect of swarm asset degradations on trustworthiness perceptions, reliance intentions, and reliance behaviors. Results showed that trustworthiness perceptions of and intentions to rely on swarms (but not reliance behaviors) were correlated, demonstrating the relation between theoretically relevant antecedents to trust in HSwI contexts. Contrary to past work, the results showed no statistical evidence that asset degradations differentially affect trustworthiness perceptions, reliance intentions, or reliance behaviors. Limitations of the current work (e.g., heterogeneity of post-intervention foraging behavior, sample size) are discussed and followed with future research suggestions.

Manned-Unmanned Teaming: Research and Applications Panel

This panel will discuss issues related to Manned-Unmanned Teaming (MUMT) technologies. Panelists were selected to represent diverse topics and each will provide a unique perspective on the MUMT challenge space. Joseph Lyons will frame the discussion and introduce the panelists. Each panelist will provide an overview of the MUMT research/applications they are involved in. Chris Miller will discuss an ongoing project looking at MUMT applications broadly across the enterprise and is seeking to identify the core systemic tenants of MUMT and metrics to gauge MUMT effectiveness. Jay Shively will discuss MUMT challenges in the context of UAS operations in the National aerospace. Nancy Cooke will discuss several MUMT research projects that emphasize teaming and associated research challenges. Col. Dan Javorsek will discuss recent MUMT programs at DARPA as well as where MUMT technologies can support Air Force applications. Phillip Walker will discuss the DARPA OFFSET program and human-swarm interactions, including human factor considerations of large swarm demonstration events.

The design of self-organizing human–swarm intelligence

Human–swarm interaction is a frontier in the realms of swarm robotics and human-factors engineering. However, no holistic theory has been explicitly formulated that can inform how humans and robot swarms should interact through an interface while considering real-world demands, the relative capabilities of the components, as well as the desired joint-system behaviours. In this article, we apply a holistic perspective that we refer to as joint human–swarm loops, that is, a cybernetic system made of human, swarm and interface. We argue that a solution for human–swarm interaction should make the joint human–swarm loop an intelligent system that balances between centralized and decentralized control. The swarm-amplified human is suggested as a possible design that combines perspectives from swarm robotics, human-factors engineering and theoretical neuroscience to produce such a joint human–swarm loop. Essentially, it states that the robot swarm should be integrated into the human’s low-level nervous system function. This requires modelling both the robot swarm and the biological nervous system as self-organizing systems. We discuss multiple design implications that follow from the swarm-amplified human, including a computational experiment that shows how the robot swarm itself can be a self-organizing interface based on minimal computational logic.