Heterogeneous Multi-robot System Research Articles

Toward a Generic Framework for Mission Planning and Execution with a Heterogeneous Multi-Robot System.

This paper presents a comprehensive framework for mission planning and execution with a heterogeneous multi-robot system, specifically designed to coordinate unmanned ground vehicles (UGVs) and unmanned aerial vehicles (UAVs) in dynamic and unstructured environments. The proposed architecture evaluates the mission requirements, allocates tasks, and optimizes resource usage based on the capabilities of the available robots. It then executes the mission utilizing a decentralized control strategy that enables the robots to adapt to environmental changes and maintain formation stability in both 2D and 3D spaces. The framework's architecture supports loose coupling between its components, enhancing system scalability and maintainability. Key features include a robust task allocation algorithm, and a dynamic formation control mechanism, using a ROS 2 communication protocol that ensures reliable information exchange among robots. The effectiveness of this framework is demonstrated through a case study involving coordinated exploration and data collection tasks, showcasing its ability to manage missions while optimizing robot collaboration. This work advances the field of heterogeneous robotic systems by providing a scalable and adaptable solution for multi-robot coordination in challenging environments.

Real-time reactive task allocation and planning of large heterogeneous multi-robot systems with temporal logic specifications

Existing methods for the task allocation and planning (TAP) of multi-robot systems with temporal logic specifications mainly rely on optimization-based approaches or graph search techniques applied to the product automaton. However, these methods suffer from high computational cost and scale poorly with the number of robots and the complexity of temporal logic tasks, thus limiting the applicability in real-time implementation, especially for large multi-robot systems. To address these challenges, this work develops a novel TAP framework that can solve reactive temporal logic planning problems for large-scale heterogeneous multi-robot systems (HMRS) in real time. Specifically, we develop a planning decision tree (PDT) to represent the task progression and task allocation specialized for HMRS with temporal logic specifications. Based on the PDT, we develop two key search algorithms—the planning decision tree search (PDTS) and the interactive planning decision tree search (IPDTS)—where PDTS generates an offline plan which will be modified online by PDTS and IPDTS jointly to enable fast reactive planning if environmental changes or temporary tasks occur. Such a design can generate satisfying plan for HMRS with multiple orders of magnitude more robots than those that existing methods can manipulate. Rigorous analysis shows that the PDT-based planning is feasible (i.e., the generated plan is applicable) and complete (i.e., a feasible plan, if exits, is guaranteed to be found). The algorithm complexity further indicates that the solution time is only linearly proportional to the robot numbers and types. Simulation and experiment results demonstrate that reactive plan can be generated for large HMRS in real-time, which outperforms the state-of-the-art methods.

Efficiency of Fuzzy Task Assignment in Heterogeneous Multi-robot System

Efficiency of Fuzzy Task Assignment in Heterogeneous Multi-robot System

A decoupled solution to heterogeneous multi-formation planning and coordination for object transportation

Multi-robot formations have numerous applications, such as cooperative object transportation in intelligent warehouses. In this context, robots are tasked with delivering objects in formation while avoiding intra- and inter-formation collisions. This necessitates the development of solutions for multi-robot task allocation, formation generation, rigid formation route planning, and formation coordination. In this paper, we present a cooperative formation object transportation system for heterogeneous multi-robot systems which captures robot dynamics and avoids inter-formation collisions. Accounting for heterogeneous formations expands the applicability of the proposed robotic transport system. For formation generation, we propose an approach based on conflict-based search, which integrates high-level path planning with low-level trajectory optimisation. For heterogeneous formation planning, we present a two-stage iterative trajectory optimisation framework which adheres to the kinematic constraints of our heterogeneous multi-robot system while retaining formation rigidity. For multi-formation coordination, we use a loosely-coupled algorithm which can guarantee collision-free and deadlock-free formation navigation under minimal assumptions. We demonstrate the efficacy of our approach in simulation.

Extension of Counting LTL and Its Application to a Path Planning Problem for Heterogeneous Multi-Robot Systems

Extension of Counting LTL and Its Application to a Path Planning Problem for Heterogeneous Multi-Robot Systems

Development of an Attention Mechanism for Task-Adaptive Heterogeneous Robot Teaming

The allure of team scale and functional diversity has led to the promising adoption of heterogeneous multi-robot systems (HMRS) in complex, large-scale operations such as disaster search and rescue, site surveillance, and social security. These systems, which coordinate multiple robots of varying functions and quantities, face the significant challenge of accurately assembling robot teams that meet the dynamic needs of tasks with respect to size and functionality, all while maintaining minimal resource expenditure. This paper introduces a pioneering adaptive cooperation method named inner attention (innerATT), crafted to dynamically configure teams of heterogeneous robots in response to evolving task types and environmental conditions. The innerATT method is articulated through the integration of an innovative attention mechanism within a multi-agent actor–critic reinforcement learning framework, enabling the strategic analysis of robot capabilities to efficiently form teams that fulfill specific task demands. To demonstrate the efficacy of innerATT in facilitating cooperation, experimental scenarios encompassing variations in task type (“Single Task”, “Double Task”, and “Mixed Task”) and robot availability are constructed under the themes of “task variety” and “robot availability variety.” The findings affirm that innerATT significantly enhances flexible cooperation, diminishes resource usage, and bolsters robustness in task fulfillment.

Digital Battle: A Three-Layer Distributed Simulation Architecture for Heterogeneous Robot System Collaboration

In this paper, we propose a three-layer distributed simulation network architecture, which consists of a distributed virtual simulation network, a perception and control subnetwork, and a cooperative communication service network. The simulation architecture runs on a distributed platform, which can provide unique virtual scenarios and multiple simulation services for the verification of basic perception, control, and planning algorithms of a single-robot system and can verify the distributed collaboration algorithms of heterogeneous multirobot systems. Further, we design simulation experimental scenarios for classic heterogeneous robotic systems such as unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs). Through the analysis of experimental measurement data, we draw several important conclusions: firstly, the replication time characteristics and update frequency characteristics of entity synchronization in our system indicate that the replication time of entity synchronization in our system is relatively short, and the update frequency can meet the needs of multirobot collaboration and ensure the real-time use and accuracy of the system; secondly, we analyze the bandwidth usage of data frames in the whole session and observe that the server side occupies almost half of the data throughput during the whole session, which indicates that the allocation and utilization of data transmission in our system is reasonable; and finally, we construct a bandwidth estimation surface model to estimate the bandwidth requirements of the current model when scaling the server-side scale and synchronization-state scale, which provides an important reference for better planning and optimizing of the resource allocation and performance of the system. Based on this distributed simulation framework, future research will improve the key technical details, including further refining the coupling object dynamic model update method to support the simulation theory of the coupling relationship between system objects, studying the impact of spatiotemporal consistency of distributed systems on multirobot control and decision making, and in-depth research on the impact of collaborative frameworks combined with multirobot systems for specific tasks.

A Distributed Dynamic Framework to Allocate Collaborative Tasks Based on Capability Matching in Heterogeneous Multirobot Systems

Collaboration among a group of robots with heterogeneous capabilities is an important research problem that enables to combine different robot functionalities, and thus, conducts complex tasks that may be difficult to achieve by a single robot with limited resources. In this paper, we propose a new distributed task allocation framework based on the capability matching of heterogeneous robots. The framework is composed of an ontological dynamic knowledge graph model and a hardware control scheme to model the capability and optimize resource utilization for collaborative tasks. We introduce an intuitive hardware control scheme based on a dynamic knowledge graph that resolves possible conflicts between the hardware control of different types of robots. Action sequences are produced by a task and motion planning algorithm to collaboratively perform the assigned task. The performance of the proposed methodology is evaluated by both simulations and hardware experiments.

Resiliency Through Collaboration in Heterogeneous Multi-Robot Systems

Resiliency Through Collaboration in Heterogeneous Multi-Robot Systems

Asymmetric Self-Play-Enabled Intelligent Heterogeneous Multirobot Catching System Using Deep Multiagent Reinforcement Learning

Aiming to develop a more robust and intelligent heterogeneous system for adversarial catching in security and rescue tasks, in this article, we discuss the specialities of applying asymmetric self-play and curriculum learning techniques to deal with the increasing heterogeneity and number of different robots in modern heterogeneous multirobot systems (HMRS). Our method, based on actor-critic multiagent reinforcement learning, provides a framework that can enable cooperative behaviors among heterogeneous multirobot teams. This leads to the development of an HMRS for complex catching scenarios that involve several robot teams and real-world constraints. We conduct simulated experiments to evaluate different mechanisms' influence on our method's performance, and real-world experiments to assess our system's performance in complex real-world catching problems. In addition, a bridging study is conducted to compare our method with a state-of-the-art method called S2M2 in heterogeneous catching problems, and our method performs better in adversarial settings. As a result, we show that the proposed framework, through fusing asymmetric self-play and curriculum learning during training, is able to successfully complete the HMRS catching task under realistic constraints in both simulation and the real world, thus providing a direction for future large-scale intelligent security & rescue HMRS.

Strength Learning Particle Swarm Optimization for Multiobjective Multirobot Task Scheduling

Cooperative heterogeneous multirobot systems have attracted increasing attention in recent years. They use multiple heterogeneous robots to execute complex tasks in a coordinated way. The allocation of heterogeneous robots to cooperative tasks is a significant and challenging optimization problem. However, little work has gone into scheduling large-scale cooperative tasks with precedence constraints and multiple conflicting optimization objectives. Existing methods are insufficient to address the issue. We propose a multiobjective model and develop strength learning particle swarm optimization (SLPSO) to optimize multiple objectives. In this article, the problem is converted into a two-step problem of task permutation construction and robot subset selection. In order to coordinate with the time-extended property of the problem, SLPSO utilizes a hybrid encode scheme: an element-based representation for task permutations and a binary representation for robot coalitions. A strength learning strategy with heuristic information guides particles to enhance their best-performing objectives for improving swarm convergence. In addition, an estimation-based local search is developed to improve spare solutions for enhancing swarm diversity, which determines the search direction by estimating fitness improvements. Experimental results on thirty problem instances are elaborated to demonstrate that the proposed SLPSO significantly outperforms the state-of-the-art algorithms in terms of inverted generational distance and hypervolume metrics. The proposed SLPSO can obtain a set of high-quality and diversified solutions.

Decentralized Robust Collision-Avoidance for Cooperative Multirobot Systems: A Gaussian Process-Based Control Barrier Function Approach

Data-based Machine Learning methods have been successfully applied to control system design in recent years. However, safety during the learning and control process is difficult to guarantee due to the inherent uncertainties. In this work, we propose a barrier function-based robust cooperative collision avoidance control framework for heterogeneous multirobot systems with model learning. First, a new control barrier function (CBF) design is proposed for cooperative collision avoidance, which leads to less conservativeness of the feasible control actions. Then, decentralized robust CBF conditions are derived which incorporate the estimation of the individual model uncertainty using Gaussian Process models such that each robot can guarantee safety with a high probability. Finally, a quadratic programming problem is formulated to obtain a controller which is minimally invasive to the nominal controller and satisfies the CBF and velocity constraints simultaneously. The decentralized implementation of the robust collision avoidance control strategy is explicitly shown and proven. Two simulation examples are given to demonstrate the effectiveness of the proposed control framework.

Vision-Based Autonomous Following of a Moving Platform and Landing for an Unmanned Aerial Vehicle

Interest in Unmanned Aerial Vehicles (UAVs) has increased due to their versatility and variety of applications, however their battery life limits their applications. Heterogeneous multi-robot systems can offer a solution to this limitation, by allowing an Unmanned Ground Vehicle (UGV) to serve as a recharging station for the aerial one. Moreover, cooperation between aerial and terrestrial robots allows them to overcome other individual limitations, such as communication link coverage or accessibility, and to solve highly complex tasks, e.g., environment exploration, infrastructure inspection or search and rescue. This work proposes a vision-based approach that enables an aerial robot to autonomously detect, follow, and land on a mobile ground platform. For this purpose, ArUcO fiducial markers are used to estimate the relative pose between the UAV and UGV by processing RGB images provided by a monocular camera on board the UAV. The pose estimation is fed to a trajectory planner and four decoupled controllers to generate speed set-points relative to the UAV. Using a cascade loop strategy, these set-points are then sent to the UAV autopilot for inner loop control. The proposed solution has been tested both in simulation, with a digital twin of a solar farm using ROS, Gazebo and Ardupilot Software-in-the-Loop (SiL); and in the real world at IST Lisbon's outdoor facilities, with a UAV built on the basis of a DJ550 Hexacopter and a modified Jackal ground robot from DJI and Clearpath Robotics, respectively. Pose estimation, trajectory planning and speed set-point are computed on board the UAV, using a Single Board Computer (SBC) running Ubuntu and ROS, without the need for external infrastructure.

Implementation of a Heterogeneous Multi-Robot System for a Construction Task

Cooperative robotic systems are becoming increasingly important, as cooperation between two or more robots makes it possible to tackle tasks that are difficult, if not impossible, to perform with a single robot. Among the various cooperative robotic systems, in this paper we focus on a heterogeneous robotic system to perform a building activity where large and heavy blocks are used. The proposed robotic system is based on a lifting mechanism and a robotic arm. This system exploits the characteristics of the lifting cable mechanism to hold most of the weight of the block, while the use of the rigid robot allows to obtain the desired precision during the fine placement. In this paper, we first derive the dynamic model of the system under consideration. Then, the kinematics analysis of the multi-robot system is investigated. Based on these two previous results, a control scheme is designed to ensure safe and correct cooperation between the two robotic systems. Experimental results with an in-scale prototype of the proposed robotic solution show the efficiency of our approach.

A BPMN-driven framework for Multi-Robot System development

Programming robotic systems is often a challenging task requiring advanced skills, especially when the goal is to ensure loosely-coupled coordination in heterogeneous Multi-Robot Systems (MRSs). Model-driven approaches for robotic system engineering have shown their benefits in facilitating the development of robots’ behavior, controllers, and system components. However, the state of the art still lacks contributions addressing crucial aspects of the model-driven approach applied to MRSs, such as developing robots’ distributed cooperation through models supporting the communication among robots.In this paper, we present a novel framework for modeling, configuring and enacting the cooperative behaviors of MRSs through collaboration diagrams as provided by the BPMN 2.0 standard. The advantages of our solution lie, indeed, in the use of BPMN, which provides easily understandable and highly expressive diagrams for representing the cooperation among distributed robots, and benefits from a wide list of supporting tools. Starting from the selection of BPMNelements, we define a set of guidelines for driving the developer in modeling an MRS mission using BPMN. The developer configures the resulting collaboration diagram to link elements in the model to the robotic middleware, ROS2 in the toolchain we implemented. Finally, the configured model is enacted by BPMN engines integrated into the ROS2 middleware run by each robot involved in the MRS, thus obtaining a fully distributed cooperation. We assess our framework’s effectiveness through experiments in simulated and real environments.

Temporal Logic Task Allocation in Heterogeneous Multirobot Systems

We consider the problem of optimally allocating tasks, expressed as global linear temporal logic (LTL) specifications, to teams of heterogeneous mobile robots of different types. Each task may require robots of multiple types. To obtain a scalable solution, we propose a hierarchical approach that first allocates specific robots to tasks using the information about the tasks contained in the nondeterministic B <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\ddot{\text{u}}$</tex-math></inline-formula> chi automaton (NBA) that captures the LTL specification and then designs low-level paths for robots that respect the high-level assignment. Specifically, motivated by “lazy collision checking” methods in robotics, we first prune and relax the NBA by removing all negative atomic propositions, which simplifies the planning problem by checking constraint satisfaction only when needed. Then, we extract sequences of subtasks from the relaxed NBA along with their temporal orders and formulate a mixed integer linear program to allocate these subtasks to robots. Finally, we define generalized multirobot path planning problems to obtain low-level paths that satisfy both the high-level task allocation and the constraints captured by the negative atomic propositions in the original NBA. We show that our method is complete for a subclass of LTL that covers a broad range of tasks and present numerical simulations demonstrating that it can generate paths with lower cost, considerably faster than existing methods.

Edge Intelligence Enabled Heterogeneous Multi-Robot Networks: Hybrid Framework, Communication Issues, and Potential Solutions

Heterogeneous multi-robot systems have been increasingly employed to collaboratively execute tasks in harsh and unknown environments. High-quality collaboration requires robots to efficiently communicate with each other and have ample intelligence to deal with uncertainties and emergencies. However, these demands are still difficult to be satisfied under existing single-mode networking architectures and onboard computing paradigms. Therefore, this article first develops a hybrid edge-aided framework for multi-robot collaboration, where the centralized and decentralized modes are elegantly combined to support a robust and efficient network, while an edge intelligence platform with powerful servers and base stations is employed to provide abundant intelligence. Under this framework, we discuss three critical communication issues that keep robots coordinated and guarantee efficient information sharing. The issues include network construction among heterogenous robots, bandwidth-efficient communication protocol design, and link transmission adaption. For each issue, potential solutions are provided and the challenges that need to be further explored are also discussed.

A Multirobot System in an Assisted Home Environment to Support the Elderly in Their Daily Lives.

The increasing isolation of the elderly both in their own homes and in care homes has made the problem of caring for elderly people who live alone an urgent priority. This article presents a proposed design for a heterogeneous multirobot system consisting of (i) a small mobile robot to monitor the well-being of elderly people who live alone and suggest activities to keep them positive and active and (ii) a domestic mobile manipulating robot that helps to perform household tasks. The entire system is integrated in an automated home environment (AAL), which also includes a set of low-cost automation sensors, a medical monitoring bracelet and an Android application to propose emotional coaching activities to the person who lives alone. The heterogeneous system uses ROS, IoT technologies, such as Node-RED, and the Home Assistant Platform. Both platforms with the home automation system have been tested over a long period of time and integrated in a real test environment, with good results. The semantic segmentation of the navigation and planning environment in the mobile manipulator for navigation and movement in the manipulation area facilitated the tasks of the later planners. Results about the interactions of users with the applications are presented and the use of artificial intelligence to predict mood is discussed. The experiments support the conclusion that the assistance robot correctly proposes activities, such as calling a relative, exercising, etc., during the day, according to the user’s detected emotional state, making this is an innovative proposal aimed at empowering the elderly so that they can be autonomous in their homes and have a good quality of life.

Stronger Together: Air-Ground Robotic Collaboration Using Semantics

In this work, we present an end-to-end heterogeneous multi-robot system framework where ground robots are able to localize, plan, and navigate in a semantic map created in real time by a high-altitude quadrotor. The ground robots choose and deconflict their targets independently, without any external intervention. Moreover, they perform cross-view localization by matching their local maps with the overhead map using semantics. The communication backbone is opportunistic and distributed, allowing the entire system to operate with no external infrastructure aside from GPS for the quadrotor. We extensively tested our system by performing different missions on top of our framework over multiple experiments in different environments. Our ground robots travelled over 6 km autonomously with minimal intervention in the real world and over 96 km in simulation without interventions.

GMP: A Genetic Mission Planner for Heterogeneous Multirobot System Applications.

The use of multiagent systems (MASs) in real-world applications keeps increasing, and diffuses into new domains, thanks to technological advances, increased acceptance, and demanding productivity requirements. Being able to automate the generation of mission plans for MASs is critical for managing complex missions in realistic settings. In addition, finding the right level of abstraction to represent any generic MAS mission is important for being able to provide general solution to the automated planning problem. In this article, we show how a mission for heterogeneous MASs can be cast as an extension of the traveling salesperson problem (TSP), and we propose a mixed-integer linear programming formulation. In order to solve this problem, a genetic mission planner (GMP), with a local plan refinement algorithm, is proposed. In addition, the comparative evaluation of CPLEX and GMP is presented in terms of timing and optimality of the obtained solutions. The algorithms are benchmarked on a proposed set of different problem instances. The results show that, in the presence of timing constraints, GMP outperforms CPLEX in the majority of test instances.