Complex Unknown Environments Research Articles

Deep reinforcement learning for adaptive path planning and control of an autonomous underwater vehicle

Research into intelligent motion planning methods has been driven by the growing autonomy of autonomous underwater vehicles (AUV) in complex unknown environments. Deep reinforcement learning (DRL) algorithms with actor-critic structures are optimal adaptive solutions that render online solutions for completely unknown systems. The present study proposes an adaptive motion planning and obstacle avoidance technique based on deep reinforcement learning for an AUV. The research employs a twin-delayed deep deterministic policy algorithm, which is suitable for Markov processes with continuous actions. Environmental observations are the vehicle's sensor navigation information. Motion planning is carried out without having any knowledge of the environment. A comprehensive reward function has been developed for control purposes. The proposed system is robust to the disturbances caused by ocean currents. The simulation results show that the motion planning system can precisely guide an AUV with six-degrees-of-freedom dynamics towards the target. In addition, the intelligent agent has appropriate generalization power.

E 3 MoP: Efficient Motion Planning Based on Heuristic-Guided Motion Primitives Pruning and Path Optimization With Sparse-Banded Structure

To solve the autonomous navigation problem in complex environments, an efficient motion planning approach is newly presented in this paper. Considering the challenges from large-scale, partially unknown complex environments, a three-layer motion planning framework is elaborately designed, including global path planning, local path optimization, and time-optimal velocity planning. Compared with existing approaches, the novelty of this work is twofold: 1) a novel heuristic-guided pruning strategy of motion primitives is proposed and fully integrated into the state lattice-based global path planner to further improve the computational efficiency of graph search, and 2) a new soft-constrained local path optimization approach is proposed, wherein the sparse-banded system structure of the underlying optimization problem is fully exploited to efficiently solve the problem. We validate the safety, smoothness, flexibility, and efficiency of our approach in various complex simulation scenarios and challenging real-world tasks. It is shown that the computational efficiency is improved by 66.21% in the global planning stage and the motion efficiency of the robot is improved by 22.87% compared with the recent quintic Bézier curve-based state space sampling approach. We name the proposed motion planning framework E <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathbf {^{3}} $ </tex-math></inline-formula> MoP, where the number 3 not only means our approach is a three-layer framework but also means the proposed approach is efficient in three stages. Note to Practitioners—This paper is motivated by the challenges of motion planning problems of mobile robots. A three-layer motion planning framework is proposed by combining global path planning, local path optimization, and time-optimal velocity planning. For mobile robot navigation applications in semi-structured environments, optimization-based local planners are recommended. Extensive simulation and experimental results show the effectiveness of the proposed motion planning framework. However, due to the non-convexity of the path optimization formulation, the proposed local planner may get stuck in local optima. In future research, we will concentrate on extending the proposed local path optimization approach with the theory of homology classes to maintain several homotopically distinct local paths and seek global optima.

Teleoperation for Urban Search and Rescue Applications

An important application of field robotics research is robotic assistance for search and rescue operations. The problem of robotic search and rescue requires techniques to map, navigate, and search unknown complex environments. In subterranean domains such as tunnels, caves, and underground urban environments these activities are made more difficult due to communication constraints and unavailability of global positioning systems. We present here Coordinated Robotics participation in the Urban Circuit of the Defense Advanced Research Projects Agency (DARPA) Subterranean Challenge which addresses these problems in the underground urban environment. Our Teleoperation strategy serves as a baseline approach by which to compare autonomous solutions. Our aim is to provide insight into our system design and our lessons learned from the competition.

IRLSOT: Inverse reinforcement learning for scene‐oriented trajectory prediction

Forecasting pedestrians' future trajectory in unknown complex environments is essential to autonomous navigation in real-world applications, for example, for self-driving cars and collision warnings. However, modern observed trajectory-based prediction methods may easily over-fit to complex or rare scenes because they do not entirely understand the correlations between scenes and trajectories. To address the over-fitting problem, an Inverse Reinforcement Learning for Scene-oriented Trajectory Prediction (IRLSOT) is proposed in this work. The authors' method can be divided into three modules. First, the inverse reinforcement learning module generates the optimal policy by extracting features from scenes and pedestrians' observed trajectories. A lightweight ENet is used to extract features from scenes. Afterwards, the path sampling module introduces a Gumbel Softmax Trick (GST) to improve the accuracy of optimal policy sampling. Different paths are generated on the basis of the optimal policies. Finally, the information fusion module uses the proposed Scene Based Attention (SBA) to fuse the path and trajectory information, then outputs the predicted trajectories. Comparison results show that IRLSOT improves performance on Stanford Drone Database(SDD) by 5.9 % $\%$ . Furthermore, the authors' test IRLSOT on multi-agent scenarios and the authors' own data sets, and results demonstrate that IRLSOT can enhance the generalization of trajectory prediction to rare or new scenes.

Safe-Nav: learning to prevent PointGoal navigation failure in unknown environments

Training robots to safely navigate (Safe-Nav) in uncertain complex environments using the RGB-D sensor is quite challenging as it involves the performance of different tasks such as obstacle avoidance, optimal path planning, and control. Traditional navigation approaches cannot generate suitable paths which guarantee enough visible features. Recent learning-based methods are still not mature enough due to their proneness to collisions and prohibitive computational cost. This paper focuses on generating safe trajectories to the desired goal while avoiding collisions and tracking failure in unknown complex environments. We present Safe-Nav, a hierarchical framework composed of the visual simultaneous localization and mapping (SLAM) module, the global planner module and the local planner module. The visual SLAM module generates the navigation map and the robot pose. The global planner module plans a local waypoint on the real-time navigation map. In the local planner module, a deep-reinforcement-learning-based (DRL-based) policy is presented for taking safe actions towards local waypoints. Our DRL-based policy can learn different navigation skills (e.g., avoiding collisions and avoiding tracking failure) through specialized modes without any supervisory signals when the PointGoal-navigation-specied reward is provided. We have demonstrated the performance of our proposed Safe-Nav in the Habitat simulation environment. Our approach outperforms the recent learning-based method and conventional navigation approach with relative improvements of over 205% (0.55 vs. 0.18) and 139% (0.55 vs. 0.23) in the success rate, respectively.

Multi-Target Coordinated Search Algorithm for Swarm Robotics Considering Practical Constraints.

In order to deal with the multi-target search problems for swarm robots in unknown complex environments, a multi-target coordinated search algorithm for swarm robots considering practical constraints is proposed in this paper. Firstly, according to the target detection situation of swarm robots, an ideal search algorithm framework combining the strategy of roaming search and coordinated search is established. Secondly, based on the framework of the multi-target search algorithm, a simplified virtual force model is combined, which effectively overcomes the real-time obstacle avoidance problem in the target search of swarm robots. Finally, in order to solve the distributed communication problem in the multi-target search of swarm robots, a distributed neighborhood communication mechanism based on a time-varying characteristic swarm with a restricted random line of sight is proposed, and which is combined with the multi-target search framework. For the swarm robot kinematics, obstacle avoidance, and communication constraints of swarm robots, the proposed multi-target search strategy is more stable, efficient, and practical than the previous methods. The effectiveness of this proposed method is verified by numerical simulations.

Review of wheeled mobile robot collision avoidance under unknown environment.

Recently, the working scenes of the robot have been emerging as diversity and complexity with gradually mature of robotic control technology. The challenge of robot adaptability emerges, especially in complicated and unknown environments. Among the numerous researches on improving the adaptability of robots, aiming at avoiding collision between robot and external environment, obstacle avoidance has drawn much attention. Compared to the global circumvention requiring the environmental information that is known, the local obstacle avoidance is a promising method due to the environment is possibly dynamic and unknown. This study is aimed at making a review of research progress about local obstacle avoidance methods for wheeled mobile robots (WMRs) under complex unknown environment in the last 20 years. Sensor-based obstacle perception and identification is first introduced. Then, obstacle avoidance methods related to WMRs' motion control are reviewed, mainly including artificial potential field (APF)-based, population-involved meta heuristic-based, artificial neural network (ANN)-based, fuzzy logic (FL)-based and quadratic optimization-based, etc. Next, the relevant research on Unmanned Ground Vehicles (UGVs) is surveyed. Finally, conclusion and prospection are given. Appropriate obstacle avoidance methods should be chosen based on the specific requirements or criterion. For the moment, effective fusion of multiple obstacle avoidance methods is becoming a promising method.

Research on Motion Planning Based on Flocking Control and Reinforcement Learning for Multi-Robot Systems

Robots have poor adaptive ability in terms of formation control and obstacle avoidance control in unknown complex environments. To address this problem, in this paper, we propose a new motion planning method based on flocking control and reinforcement learning. It uses flocking control to implement a multi-robot orderly motion. To avoid the trap of potential fields faced during flocking control, the flocking control is optimized, and the strategy of wall-following behavior control is designed. In this paper, reinforcement learning is adopted to implement the robotic behavioral decision and to enhance the analytical and predictive abilities of the robot during motion planning in an unknown environment. A visual simulation platform is developed in this paper, on which researchers can test algorithms for multi-robot motion control, such as obstacle avoidance control, formation control, path planning and reinforcement learning strategy. As shown by the simulation experiments, the motion planning method presented in this paper can enhance the abilities of multi-robot systems to self-learn and self-adapt under a fully unknown environment with complex obstacles.

Relevant experience learning: A deep reinforcement learning method for UAV autonomous motion planning in complex unknown environments

Unmanned Aerial Vehicles (UAVs) play a vital role in military warfare. In a variety of battlefield mission scenarios, UAVs are required to safely fly to designated locations without human intervention. Therefore, finding a suitable method to solve the UAV Autonomous Motion Planning (AMP) problem can improve the success rate of UAV missions to a certain extent. In recent years, many studies have used Deep Reinforcement Learning (DRL) methods to address the AMP problem and have achieved good results. From the perspective of sampling, this paper designs a sampling method with double-screening, combines it with the Deep Deterministic Policy Gradient (DDPG) algorithm, and proposes the Relevant Experience Learning-DDPG (REL-DDPG) algorithm. The REL-DDPG algorithm uses a Prioritized Experience Replay (PER) mechanism to break the correlation of continuous experiences in the experience pool, finds the experiences most similar to the current state to learn according to the theory in human education, and expands the influence of the learning process on action selection at the current state. All experiments are applied in a complex unknown simulation environment constructed based on the parameters of a real UAV. The training experiments show that REL-DDPG improves the convergence speed and the convergence result compared to the state-of-the-art DDPG algorithm, while the testing experiments show the applicability of the algorithm and investigate the performance under different parameter conditions.

FUEL: Fast UAV Exploration Using Incremental Frontier Structure and Hierarchical Planning

Autonomous exploration is a fundamental problem for various applications of unmanned aerial vehicles(UAVs). Existing methods, however, were demonstrated to insufficient exploration rate, due to the lack of efficient global coverage, conservative motion plans and low decision frequencies. In this letter, we propose FUEL, a hierarchical framework that can support Fast UAV ExpLoration in complex unknown environments. We maintain crucial information in the entire space required by exploration planning by a frontier information structure (FIS), which can be updated incrementally when the space is explored. Supported by the FIS, a hierarchical planner plans exploration motions in three steps, which find efficient global coverage paths, refine a local set of viewpoints and generate minimum-time trajectories in sequence. We present extensive benchmark and real-world tests, in which our method completes the exploration tasks with unprecedented efficiency (3-8 times faster) compared to state-of-the-art approaches. Our method will be made open source to benefit the community <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">11</sup> To be released at https://github.com/HKUST-Aerial-Robotics/FUEL..

Joint Multitarget Detection and Tracking in Multipath Environment Using Expectation Maximization Algorithm

Joint detection and tracking is fundamentally important in signal processing, navigation, and radar applications. Especially, multitarget detection and tracking in multipath environments is a promising issue and broadening range for joint detection and tracking. However, its wide adoption in real-world systems is challenged by the unknown data association, complex and changeable motion environment, and data mutual coupling. To address those problems, we propose a joint detection and tracking scheme based on expectation maximization (EM) algorithm. By alternately computing the complete likelihood function and optimizing it with respect to target existence state (detection) and target kinematic state (tracking), the proposed scheme iteratively optimizes estimation and data association. Furthermore, we provide a hybrid forward and backward algorithm during the M-step to deal with the coupling of the target existence state and target kinematic state. We also provide a convergence speed-up algorithm based on K-means method and stochastic initialization method to accelerate convergence speed. Finally, we verify the effectiveness of the proposed joint detection and tracking scheme by conducting extensive simulations with different tracking scenarios. The simulation results show that the proposed algorithm not only improves the detection and tracking performance, but also stays stable performance in complex and changeable motion environment and high noise background.

Autonomous PSO-DVSF&lt;SUP align="right"&gt;2&lt;/SUP&gt;: an optimised force field mobile robot motion planning approach for unknown dynamic environments

PSO-DVSF2 (Ziadi et al., 2016) is a PSO optimised mobile robot motion planning strategy based on the force field approach. PSO-DVSF2 has been previously developed to optimally guide the mobile robot in known dynamic environments. Autonomous PSO-DVSF2, the subject of this paper, is an improvement of PSO-DVSF2 to deal with unknown dynamic environments. In this new real-time PSO optimised mobile robot motion planning approach, the robot has to update F2 parameters all along the trajectory and not once in the beginning of the navigation as has been the case with the previous version. A comparison with the autonomous PSO-CF2 has been applied in various unknown environments (static and dynamic) using the 3D virtual Webots simulator. The robot localisation based on sensor readings with a local motion planning, and the variation of angular and linear speeds ensure the robot collision-free motion. Simulation results prove the efficiency of the autonomous PSO-DVSF2 to guide the robot along the shortest and safest path in complex unknown dynamic environments.

A Novel End Effector of Environmental Exploring Robot with Fully Spherical Workspace

Yu, S.; Zhang, J., and Li, W., 2020. A novel end effector of environmental exploring robot with fully spherical workspace. In: Hu, C. and Cai, M. (eds.), Geo-informatics and Oceanography. Journal of Coastal Research, Special Issue No. 105, pp. 159–164. Coconut Creek (Florida), ISSN 0749-0208.A large workspace is an important parameter to improve the robot's mobility and flexibility. Based on decoupled 2-degree of freedom spherical parallel mechanism, two novel end effectors of the environmental exploring robot with a fully spherical workspace was presented. Through workspace analysis, the conditions of the end effectors to achieve a fully spherical workspace were determined. Through kinematic analysis, the kinematic equations of the two end effectors were established. After size optimization and performance comparison, the input and output performance of the RR&P5R end effector is better. Then the statics analysis and finite element analysis of the RR&P5R end effector were carried out. The results show that the material selection and structural design are reasonable. This research provides a technical support for complex unknown environment exploration.

Learning-Based Real-Time Process-Aware Anomaly Monitoring for Assured Autonomy

Various learning-based approaches have been proposed recently for robot control in complex unknown environments. However, ensuring safety in these systems is essential. Learning-based approaches are usually black-box systems which are difficult to interpret/verify. We propose a real-time continuous safety checking system which monitors the validity of the mapping from the sensor input to the actuator command (i.e., controller-focused anomaly detection) and from the actuator command to the sensor input (i.e., system-focused anomaly detection). Our controller-focused anomaly monitor is an image conditioned energy based generative adversarial network in which the discriminator acts as an energy function attributing low energies to proper actuator commands and high energy to anomalous actuator commands. Our system-focused anomaly monitor is based on an action condition video prediction framework, which finds the action conditioned future frame most similar to the actual future frame and compares it with the true action to detect anomaly. We experimentally demonstrate the effectiveness of our framework for an autonomous navigation system in indoor (on datasets from our ground vehicle) and outdoor environments (on Udacity dataset). Furthermore, we show through experimental evaluations that our proposed framework is resilient to various data corruptions common in a vision based system.

Multi-Agent Collaborative Path Planning Based on Staying Alive Policy

Modern mobile robots tend to be used in numerous exploration and search and rescue applications. Essentially they are coordinated by human operators and collaborate with inspection or rescue teams. Over the time, robots became more advanced and capable for various autonomous collaborative scenarios. Recent advances in the field of collaborative exploration and coverage provide different approaches to solve this objective. Thus scope of this article is to present a novel collaborative approach for multi-agent coordination in exploration and coverage of unknown complex indoor environments. Fundamentally, the task of collaborative exploration can be divided into two core components. The principal one is a sensor based exploration scheme that aims to guarantee complete area exploration and coverage. The second core component proposed is a staying alive policy that takes under consideration the battery charge level limitation of the agents. From this perspective the path planner assigns feasible tasks to each of the agents, including the capability of providing reachable, collision free paths. The overall efficacy of the proposed approach was extensively evaluated by multiple simulation results in a complex unknown environments.

Event-Based Path-Planning and Path-Following in Unknown Environments for Underactuated Autonomous Underwater Vehicles

The paper addresses path planning and path-following problems in an unknown complex environment for an underactuated autonomous underwater vehicle (AUV). The AUV is required to follow a given reference path represented as a sequence of smoothly joined lines and arcs, bypassing obstacles encountered on the path. A two-level control system is proposed with an upper level for event-driven path planning and a lower level for path-following. A discrete event system is designed to identify situations that require planning a new path. An improved waypoint guidance algorithm and a Dubins curves based algorithm are proposed to build paths that allow the AUV to avoid collision with obstacles and to return to the reference path respectively. Both algorithms generate paths that meet the minimum turning radius constraint. A robust parameter-varying controller is designed using sublinear vector Lyapunov functions to solve the path-following problem. The performance of the developed event-based control system is demonstrated in three different simulation scenarios: with a sharp-edged obstacle, with a U-shaped obstacle, and with densely scattered obstacles. The proposed scheme does not require significant computing resources and allows for easy implementation on board.

Reinforcement based mobile robot path planning with improved dynamic window approach in unknown environment

Mobile robot path planning in an unknown environment is a fundamental and challenging problem in the field of robotics. Dynamic window approach (DWA) is an effective method of local path planning, however some of its evaluation functions are inadequate and the algorithm for choosing the weights of these functions is lacking, which makes it highly dependent on the global reference and prone to fail in an unknown environment. In this paper, an improved DWA based on Q-learning is proposed. First, the original evaluation functions are modified and extended by adding two new evaluation functions to enhance the performance of global navigation. Then, considering the balance of effectiveness and speed, we define the state space, action space and reward function of the adopted Q-learning algorithm for the robot motion planning. After that, the parameters of the proposed DWA are adaptively learned by Q-learning and a trained agent is obtained to adapt to the unknown environment. At last, by a series of comparative simulations, the proposed method shows higher navigation efficiency and successful rate in the complex unknown environment. The proposed method is also validated in experiments based on XQ-4 Pro robot to verify its navigation capability in both static and dynamic environment.

Multitarget Search of Swarm Robots in Unknown Complex Environments

When searching for multiple targets in an unknown complex environment, swarm robots should firstly form a number of subswarms autonomously through a task division model and then each subswarm searches for a target in parallel. Based on the probability response principle and multitarget division strategy, a closed-loop regulation strategy is proposed, which includes target type of member, target response intensity evaluation, and distance to the corresponding individuals. Besides, it is necessary to make robots avoid other robots and convex obstacles with various shapes in the unknown complex environment. By decomposing the multitarget search behavior of swarm robots, a simplified virtual-force model (SVF-Model) is developed for individual robots, and a control method is designed for swarm robots searching for multiple targets (SRSMT-SVF). The simulation results indicate that the proposed method keeps the robot with a good performance of collision avoidance, effectively reducing the collision conflicts among the robots, environment, and individuals.

Multiobjective coordinated search algorithm for swarm of UAVs based on 3D-simplified virtual forced model

This paper aims to tackle the problem of multiobjective search for swarms of UAVs in unknown complex environments and proposes a multiobjective coordinated search algorithm based on a 3D-simplified virtual forced model (MOCS-3D-SVFM). First, it decomposes the search behaviour into the roaming search state and coordinated search state based on the detection of target signals by a swarm of UAVs. Second, a nearest neighbour exclusion diffusion (NNED) algorithm is introduced for the UAV of the wander search state, and a 3D adaptive inertia weight extended particle swarm algorithm (IAEPSO) is proposed by combining the motion characteristics of UAV with a 3D particle swarm algorithm aiming at the UAV with coordinated search state. Finally, the 3D-simplified virtual force model proposed based on the concept of the 2D-simplified virtual force model by the rotation matrix is introduced to solve the model parameters and the control strategy under the UAV of wander search state and coordinated search state is established, which effectively solves the real-time obstacle avoidance problem. Moreover, this paper sets the comparison mode of the three search methods; compared to Mode1, the search time T and energy consumption S can be significantly reduced, and the numerical simulations verify its effectiveness.

On the consensus of nonlinear agents in unknown cluttered environments using random planning

The consensus of multi-agent dynamic systems is a metaphor for many different tasks involving group agreement. However, ensuring consensus in real-world scenarios, with non-convex obstacles and kinodynamic motion constraints, proves to be a hard task, since it is quite difficult to model such a problem analytically. Therefore, this paper studies the problem of state agreement for Multi-Robot Systems (MRS) in unknown cluttered complex environments. Here, we propose and analyze a distributed consensus algorithm combined with a Rapidly-exploring Random Tree-based planner, which allows linear and nonlinear systems to reach a common target on their states inside bi- or three-dimensional spaces filled with static obstacles. We demonstrate that, with enough time, our planning strategy ensures probabilistic completeness convergence independently of the topological communication network employed, since some connectivity constraints are observed. Simulated results with linear and nonlinear models are provided, showing the effectiveness of our proposed method in comparison with the state-of-the-art literature for the specific case of position consensus (rendezvous) missions.