Navigation Policy Research Articles

Efficient autonomous navigation for mobile robots using machine learning

The ability to navigate autonomously from the start to its final goal is the crucial key to mobile robots. To ensure complete navigation, it is mandatory to do heavy programming since this task is composed of several subtasks such as path planning, localization, and obstacle avoidance. This paper simplifies this heavy process by making the robot more intelligent. The robot will acquire the navigation policy from an expert in navigation using machine learning. We used the expert A*, which is characterized by generating an optimal trajectory. In the context of robotics, learning from demonstration (LFD) will allow robots, in general, to acquire new skills by imitating the behavior of an expert. The expert will navigate in different environments, and our robot will try to learn its navigation strategy by linking states and suitable actions taken. We find that our robot acquires the navigation policy given by A* very well. Several tests were simulated with environments of different complexity and obstacle distributions to evaluate the flexibility and efficiency of the proposed strategies. The experimental results demonstrate the reliability and effectiveness of the proposed method.

Eligibility traces in an autonomous soccer robot with obstacle avoidance and navigation policy

As the thematic literature of machine learning suggests, reinforcement learning falls between the two methods of supervised learning and unsupervised learning. In this method, the learning agent receives a reward or punishment from the environment according to its action. Therefore, the learning agent interacts with the environment through trial and error and learns to choose the optimal action to achieve its goal. In the meantime, the eligibility traces are considered as one of the main mechanisms of reinforcement learning in receiving delayed rewards. In the use of conventional reinforcement learning methods, when the learning agent achieves a goal, only the value function of the last state-action pair changes, but all the trace states are affected based on the eligibility traces. In other words, the delayed rewards are distributed throughout the trace. Like the ant pheromone effect, this method can increase learning speed empirically to some extent. A soccer robot on the field encounters moving obstacles such as balls, rival robots, and home robots, and fixed obstacles such as gates and flags, so its environment is in a very dynamic state. Therefore, due to the dynamic environment, it is an important issue for autonomous soccer robots to avoid obstacles and should be considered in real play. The main idea of this study is to determine the appropriate traces for the robot to move towards the ball and ultimately to score with the approach of avoiding obstacles in simulating a real soccer match. The desired results obtained from a game played indicate a high level of online experience and decision-making power in the face of new situations.

DGMem: learning visual navigation policy without any labels by dynamic graph memory

DGMem: learning visual navigation policy without any labels by dynamic graph memory

Risk assessment of maritime autonomous surface ships collisions using an FTA-FBN model

Maritime autonomous surface ships (MASS), presenting the future of maritime transport, are attracting increasing attention from the international maritime community. The collision risk analysis of MASS reveals unsolved challenges, which without appropriate solutions, will result in the error prone development of the relevant risk control measures and policies. Among the challenges, two significant ones in the existing literature are the lack of historical failure data to realise quantitative risk assessment, and 2) the complex causal relationship among the relevant risk factors. This paper aims to develop a new Fault Tree Analysis-Fuzzy Bayesian Network (FTA-FBN) model to conduct the collision risk assessment of MASS with uncertainty in data. First, it establishes a causal relationship among the risk factors through an FTA. Secondly, mapping the obtained FTA diagram into a BN allows fault diagnosis and the identification of the most important factors influencing MASS collisions. In this process, a survey is conducted to collect the primary data for configuring the conditional probabilities of the relevant influential factors and quantifying the developed BN for risk diagnosis and prediction. Finally, the new model is verified by using sensitivity analysis and three axioms and then applied to conduct scenario-based risk prediction and diagnosis to generate insightful findings to guide MASS navigation safety. The results demonstrate that the FTA-FBN model realizes the simplification of the expert scoring process, reduces computational complexity, and addresses the challenge of constructing causal relationships between MASS collisions and their risk factors due to the scarcity of historical accident data. Additionally, the BN backward reasoning identifies key collision risks, including external physical attacks, inadequate training of shore-based operators, insufficient maintenance of ship equipment and systems, and cyber-security threats. The new model when being adapted, can provide a reference for the formulation of safe navigation policies and provide important insights for shipping companies to ensure the safe navigation of their ships and shipbuilders to optimise ship design.

A Soft Actor-Critic Deep Reinforcement-Learning-Based Robot Navigation Method Using LiDAR

When there are dynamic obstacles in the environment, it is difficult for traditional path-generation algorithms to achieve desired obstacle-avoidance results. To solve this problem, we propose a robot navigation control method based on SAC (Soft Actor-Critic) Deep Reinforcement Learning. Firstly, we use a fast path-generation algorithm to control the robot to generate expert trajectories when the robot encounters danger as well as when it approaches a target, and we combine SAC reinforcement learning with imitation learning based on expert trajectories to improve the safety of training. Then, for the hybrid data consisting of agent data and expert data, we use an improved prioritized experience replay method to improve the learning efficiency of the policies. Finally, we introduce RNN (Recurrent Neural Network) units into the network structure of the SAC Deep Reinforcement-Learning navigation policy to improve the agent’s transfer inference ability in a new environment and obstacle-avoidance ability in dynamic environments. Through simulation and practical experiments, it is fully verified that our method has a higher training efficiency and navigation success rate compared to state-of-the-art reinforcement-learning algorithms, which further enhances the obstacle-avoidance capability of the robot system.

Building Category Graphs Representation with Spatial and Temporal Attention for Visual Navigation

Given an object of interest, visual navigation aims to reach the object’s location based on a sequence of partial observations. To this end, an agent needs to (1) acquire specific knowledge about the relations of object categories in the world during training and (2) locate the target object based on the pre-learned object category relations and its trajectory in the current unseen environment. In this article, we propose a Category Relation Graph (CRG) to learn the knowledge of object category layout relations and a Temporal-Spatial-Region attention (TSR) architecture to perceive the long-term spatial-temporal dependencies of objects, aiding navigation. We establish CRG to learn prior knowledge of object layout and deduce the positions of specific objects. Subsequently, we propose the TSR architecture to capture relationships among objects in temporal, spatial, and regions within observation trajectories. Specifically, we implement a Temporal attention module (T) to model the temporal structure of the observation sequence, implicitly encoding historical moving or trajectory information. Then, a Spatial attention module (S) uncovers the spatial context of the current observation objects based on CRG and past observations. Last, a Region attention module (R) shifts the attention to the target-relevant region. Leveraging the visual representation extracted by our method, the agent accurately perceives the environment and easily learns a superior navigation policy. Experiments on AI2-THOR demonstrate that our CRG-TSR method significantly outperforms existing methods in both effectiveness and efficiency. The supplementary material includes the code and will be publicly available.

Multi – Robot Exploration Supported by Enhanced Localization with Reduction of Localization Error Using Particle Swarm Optimization

Exploration of an area by a group of robots is an active research field of robotics as multi-robot exploration is applied extensively in several real life scenarios. The major challenges in such exploration are the availability of communication infrastructure as communication plays a key role in the coordination of team of robots for effective coverage of the area under exploration. But in disaster affected scenarios, there will be no existing communication infrastructure available and this makes the exploration ineffective and time consuming. Another challenge is in the localization process each robot is carrying out to update the map as well as for exchange of information with other robots. In this paper, an enhanced Multi-robot exploration strategy is introduced. The base of the exploration strategy is two techniques. The first one being localization of each robot involved in the exploration and this is done with the help of trilateration where three anchors are required which will be setup before the exploration starts. The second part is navigation and avoiding overlapping or missing out sectors while exploring. This is done with help of a navigation policy called frontier cell based approach. Further to this, the exploration strategy is supported with localization error reduction scheme in which the localization error is reduced with the help of Particle Swarm Optimization (PSO). The entire scheme is simulated and exploration time is analyzed for the same environment in different obstacle density and different number of robots to perform exploration. The results show the scheme is better than many existing multi-robot exploration strategies. Precisely, the proposed scheme is able to reduce the localization error to a threshold level of 0.02cm or below which can be considered as novel contribution towards the exploration strategies.

CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments

Audio-visual navigation of an agent towards locating an audio goal is a challenging task especially when the audio is sporadic or the environment is noisy. In this paper, we present CAVEN, a Conversation-based Audio-Visual Embodied Navigation framework in which the agent may interact with a human/oracle for solving the task of navigating to an audio goal. Specifically, CAVEN is modeled as a budget-aware partially observable semi-Markov decision process that implicitly learns the uncertainty in the audio-based navigation policy to decide when and how the agent may interact with the oracle. Our CAVEN agent can engage in fully-bidirectional natural language conversations by producing relevant questions and interpret free-form, potentially noisy responses from the oracle based on the audio-visual context. To enable such a capability, CAVEN is equipped with: i) a trajectory forecasting network that is grounded in audio-visual cues to produce a potential trajectory to the estimated goal, and (ii) a natural language based question generation and reasoning network to pose an interactive question to the oracle or interpret the oracle's response to produce navigation instructions. To train the interactive modules, we present a large scale dataset: AVN-Instruct, based on the Landmark-RxR dataset. To substantiate the usefulness of conversations, we present experiments on the benchmark audio-goal task using the SoundSpaces simulator under various noisy settings. Our results reveal that our fully-conversational approach leads to nearly an order-of-magnitude improvement in success rate, especially in localizing new sound sources and against methods that use only uni-directional interaction.

SOCIALGYM 2.0: Simulator for Multi-Robot Learning and Navigation in Shared Human Spaces

We present Social Gym 2.0, a simulator for multi-agent navigation research. Our simulator enables navigation for multiple autonomous agents, replicating real-world dynamics in complex indoor environments, including doorways, hallways, intersections, and roundabouts. Unlike current simulators that concentrate on single robots in open spaces, Social Gym 2.0 employs multi-agent reinforcement learning (MARL) to develop optimal navigation policies for multiple robots with diverse, dynamic constraints in complex environments. Social Gym 2.0 also departs from the accepted software design standards by employing a configuration-over-convention paradigm providing the capability to benchmark different MARL algorithms, as well as customize observation and reward functions. Users can additionally create their own environments and evaluate various algorithms, based on both deep reinforcement learning as well as classical navigation, using a broad range of social navigation metrics.

ANYmal parkour: Learning agile navigation for quadrupedal robots.

Performing agile navigation with four-legged robots is a challenging task because of the highly dynamic motions, contacts with various parts of the robot, and the limited field of view of the perception sensors. Here, we propose a fully learned approach to training such robots and conquer scenarios that are reminiscent of parkour challenges. The method involves training advanced locomotion skills for several types of obstacles, such as walking, jumping, climbing, and crouching, and then using a high-level policy to select and control those skills across the terrain. Thanks to our hierarchical formulation, the navigation policy is aware of the capabilities of each skill, and it will adapt its behavior depending on the scenario at hand. In addition, a perception module was trained to reconstruct obstacles from highly occluded and noisy sensory data and endows the pipeline with scene understanding. Compared with previous attempts, our method can plan a path for challenging scenarios without expert demonstration, offline computation, a priori knowledge of the environment, or taking contacts explicitly into account. Although these modules were trained from simulated data only, our real-world experiments demonstrate successful transfer on hardware, where the robot navigated and crossed consecutive challenging obstacles with speeds of up to 2 meters per second.

The Fittest Wins: A Multistage Framework Achieving New SOTA in ViZDoom Competition

This paper offers an integrated solution for First-Person Shooter (FPS) games to train agents with adaptive strategies. Solving such complex decision tasks requires generalization ability and adaptive strategies. We develop a framework using a novel Adaptive Strategic Control (ASC) algorithm combined with advanced techniques like the hindsight experience replay (HER), multi-agent reinforcement learning (MARL), and league training. The approach adopts a multi-stage learning scheme, consisting of learning a goal-conditioned navigation policy, then transferring to learn sophisticated shooting skills by playing against a league of players, and finally learning adaptive strategies. Our agent achieves the SOTA result in past ViZDoom AI Competitions, surpassing previous top-ranked agents (never seen during training) by a large margin. We provide comprehensive analysis and experiments to elaborate the effect of each component in affecting the agent performance and demonstrate that the proposed and adopted techniques are essential to achieve superior performance in ViZDoom Competition and potentially valuable for general end-to-end FPS games.

Learning Heterogeneous Relation Graph and Value Regularization Policy for Visual Navigation.

The goal of visual navigation is steering an agent to find a given target object with current observation. It is crucial to learn an informative visual representation and robust navigation policy in this task. Aiming to promote these two parts, we propose three complementary techniques, heterogeneous relation graph (HRG), a value regularized navigation policy (VRP), and gradient-based meta learning (ML). HRG integrates object relationships, including object semantic closeness and spatial directions, e.g., a knife is usually co-occurrence with bowl semantically or located at the left of the fork spatially. It improves visual representation learning. Both VRP and gradient-based ML improve robust navigation policy, regulating this process of the agent to escape from the deadlock states such as being stuck or looping. Specifically, gradient-based ML is a type of supervision method used in policy network training, which eliminates the gap between the seen and unseen environment distributions. In this process, VRP maximizes the transformation of the mutual information between visual observation and navigation policy, thus improving more informed navigation decisions. Our framework shows superior performance over the current state-of-the-art (SOTA) in terms of success rate and success weighted by length (SPL). Our HRG outperforms the Visual Genome knowledge graph on cross-scene generalization with ≈ 56% and ≈ 39% improvement on Hits@ 5* (proportion of correct entities ranked in top 5) and MRR * (mean reciprocal rank), respectively. Our code and HRG datasets will be made publicly available in the scientific community.

Robust Strategy Generation for Automatic Navigation of Unmanned Surface Vehicles through Improved DDPG Algorithm

Automatic navigation with collision‐free navigation has become a critical challenge for unmanned surface vehicles (USVs) to expand their application scenarios. Conventional methods for achieving automatic navigation of USVs typically rely on finely modeling the environment, thus exhibiting poor generalization capabilities. Methods based on deep reinforcement learning possess powerful learning abilities and have achieved promising results in USV‐automatic navigation‐tasks. However, the increase in the complexity of network model structures has led to instability during the training process. Therefore, generating more robust navigation strategies, namely ensuring robust reward‐score trends during training and smoother action trajectories of the USV, is crucial for automatic navigation and constitutes the main research question of this study. In this paper, an improved deep deterministic policy gradient (DDPG) algorithm has been proposed for stable automatic navigation of USVs in complex environments. In this algorithm, first, we construct a stable training framework that incorporates the stable feature‐sharing module with constrained gradient backpropagation, which bolsters the USV’s scene memorization capacity, reduces model training fluctuations during navigation policy learning, and improves the training stability of the navigation model. Second, we ensure the decision adaptability of the USV by constraining the extent of action change between adjacent time‐steps by using a reward‐function, which improves the USV‐action smoothly. Finally, we design typical USV‐automatic‐navigation‐scenarios to validate the performance of the Algorithm. Experimental results validate our algorithm’s capability to achieve collision‐free navigation, outperforming the traditional DDPG algorithm in terms of convergence speed, effective sailing distance, and rudder angle maneuver consumption, among other performance metrics.

Hierarchical Relational Graph Learning for Autonomous Multirobot Cooperative Navigation in Dynamic Environments

As a specific kind of Cyber-Physical Systems (CPSs), autonomous robot clusters play an important role in various intelligent manufacturing fields. However, due to the increasing design complexity of robot clusters, it is becoming more and more challenging to guarantee the safety and efficiency for multi-robot cooperative navigation in dynamic and complex environments. Although Deep Reinforcement Learning (DRL) shows great potential in learning multi-robot cooperative navigation policies, existing DRL-based approaches suffer from scalability issues and rarely consider the transferability of trained policies to new tasks. To address these problems, this paper presents a novel DRL-based multi-robot cooperative navigation approach named HRMR-Navi that equips each robot with both a two-layered hierarchical graph network model and an attention-based communication model. In our approach, the hierarchical graph network model can efficiently figure out hierarchical relations among all agents that either cooperate for efficiency or avoid obstacles for safety to derive more advanced strategies, and the communication model can accurately form a global view of the environment for a specific robot, thus the multi-robot cooperation efficiency can be further strengthened. Meanwhile, we propose an improved Proximal Policy Optimization (PPO) algorithm based on the Maximum Entropy reinforcement learning, named MEPPO, to enhance the robot exploration ability. Comprehensive experimental results demonstrate that, compared with state-of-the-art approaches, HRMR-Navi can achieve more efficient cooperative navigation with less time cost, lower collision rate, higher scalability, and better knowledge transferability.

Image-based Navigation in Real-World Environments via Multiple Mid-level Representations: Fusion Models, Benchmark and Efficient Evaluation

AbstractRobot visual navigation is a relevant research topic. Current deep navigation models conveniently learn the navigation policies in simulation, given the large amount of experience they need to collect. Unfortunately, the resulting models show a limited generalization ability when deployed in the real world. In this work we explore solutions to facilitate the development of visual navigation policies trained in simulation that can be successfully transferred in the real world. We first propose an efficient evaluation tool to reproduce realistic navigation episodes in simulation. We then investigate a variety of deep fusion architectures to combine a set of mid-level representations, with the aim of finding the best merge strategy that maximize the real world performances. Our experiments, performed both in simulation and on a robotic platform, show the effectiveness of the considered mid-level representations-based models and confirm the reliability of the evaluation tool. The 3D models of the environment and the code of the validation tool are publicly available at the following link: https://iplab.dmi.unict.it/EmbodiedVN/.

Reinforcement learning for high-quality reality mapping of indoor construction using unmanned ground vehicles

Recent advances in reality capture technology focused on automating reality capture and devising robust computational models to convert the collected data into usable formats. However, these modern approaches are still challenged by the insufficient capacity of the resulting information to relay accurate and complete representations of the construction state due to the data's poor visual quality. In addition, the complexity of robotic path planning –especially for indoor construction– hinders automatic visual data collection due to cluttered, narrow, and dynamic construction spaces. This work targets both challenges by presenting a reinforcement learning model that optimizes indoor data collection policy for acquiring high-quality visual data using camera-equipped unmanned ground rovers. Results from three learned navigation policies show the capability of the method to provide high visual quality for the collected data. The learned policies reduced the data collection duration by 38.23% on average compared to the currently used automatic data collection strategies. The policies also provided a 31.04% average reduction in data collection distance compared to lawn-mower patterns.

SafeCrowdNav: safety evaluation of robot crowd navigation in complex scenes.

Navigating safely and efficiently in dense crowds remains a challenging problem for mobile robots. The interaction mechanisms involved in collision avoidance require robots to exhibit active and foresighted behaviors while understanding the crowd dynamics. Deep reinforcement learning methods have shown superior performance compared to model-based approaches. However, existing methods lack an intuitive and quantitative safety evaluation for agents, and they may potentially trap agents in local optima during training, hindering their ability to learn optimal strategies. In addition, sparse reward problems further compound these limitations. To address these challenges, we propose SafeCrowdNav, a comprehensive crowd navigation algorithm that emphasizes obstacle avoidance in complex environments. Our approach incorporates a safety evaluation function to quantitatively assess the current safety score and an intrinsic exploration reward to balance exploration and exploitation based on scene constraints. By combining prioritized experience replay and hindsight experience replay techniques, our model effectively learns the optimal navigation policy in crowded environments. Experimental outcomes reveal that our approach enables robots to improve crowd comprehension during navigation, resulting in reduced collision probabilities and shorter navigation times compared to state-of-the-art algorithms. Our code is available at https://github.com/Janet-xujing-1216/SafeCrowdNav.

Multi-target landmark detection with incomplete images via reinforcement learning and shape prior embedding.

Medical images are generally acquired with limited field-of-view (FOV), which could lead to incomplete regions of interest (ROI), and thus impose a great challenge on medical image analysis. This is particularly evident for the learning-based multi-target landmark detection, where algorithms could be misleading to learn primarily the variation of background due to the varying FOV, failing the detection of targets. Based on learning a navigation policy, instead of predicting targets directly, reinforcement learning (RL)-based methods have the potential to tackle this challenge in an efficient manner. Inspired by this, in this work we propose a multi-agent RL framework for simultaneous multi-target landmark detection. This framework is aimed to learn from incomplete or (and) complete images to form an implicit knowledge of global structure, which is consolidated during the training stage for the detection of targets from either complete or incomplete test images. To further explicitly exploit the global structural information from incomplete images, we propose to embed a shape model into the RL process. With this prior knowledge, the proposed RL model can not only localize dozens of targets simultaneously, but also work effectively and robustly in the presence of incomplete images. We validated the applicability and efficacy of the proposed method on various multi-target detection tasks with incomplete images from practical clinics, using body dual-energy X-ray absorptiometry (DXA), cardiac MRI and head CT datasets. Results showed that our method could predict whole set of landmarks with incomplete training images up to 80% missing proportion (average distance error 2.29 cm on body DXA), and could detect unseen landmarks in regions with missing image information outside FOV of target images (average distance error 6.84 mm on 3D half-head CT). Our code will be released via https://zmiclab.github.io/projects.html.

Efficient Focus Autoencoders for Fast Autonomous Flight in Intricate Wild Scenarios

The autonomous navigation of aerial robots in unknown and complex outdoor environments is a challenging problem that typically requires planners to generate collision-free trajectories based on human expert rules for fast navigation. Presently, aerial robots suffer from high latency in acquiring environmental information, which limits the control strategies that the vehicle can implement. In this study, we proposed the SAC_FAE algorithm for high-speed navigation in complex environments using deep reinforcement learning (DRL) policies. Our approach consisted of a soft actor–critic (SAC) algorithm and a focus autoencoder (FAE). Our end-to-end DRL navigation policy enabled a flying robot to efficiently accomplish navigation tasks without prior map information by relying solely on the front-end depth frames and its own pose information. The proposed algorithm outperformed existing trajectory-based optimization approaches at flight speeds exceeding 3 m/s in multiple testing environments, which demonstrates its effectiveness and efficiency.

Training a Non-Cooperator to Identify Vulnerabilities and Improve Robustness for Robot Navigation

Autonomous mobile robots have become popular in various applications coexisting with humans, which requires robots to navigate efficiently and safely in crowd environments with diverse pedestrians. Pedestrians may cooperate with the robot by avoiding it actively or ignoring the robot during their walking, while some pedestrians, denoted as non-cooperators, may try to block the robot. It is also challenging to identify potential vulnerabilities of a navigation policy, i.e., situations that the robot may cause a collision, in various crowd environments, which reduces the reliability and the safety of the robot. In this paper, we propose a deep reinforcement learning (DRL) approach to train a policy simulating the behavior of non-cooperators, which can effectively identify vulnerabilities of a navigation policy. We evaluate the approach both on the Robot Operating System (ROS) navigation stack with dynamic window approach (DWA) and a DRL-based navigation policy, which identifies useful vulnerabilities of both navigation policies for further improvements. Moreover, these non-cooperators play a game with the DRL-based navigation policy, then we can improve the robustness of such navigation policy by retraining it in the sense of asymmetric self-play. We evaluate the retrained navigation policy in various crowd environments with diverse pedestrians. The experimental results show that the approach can improve the robustness of the navigation policy. The source code for the training and the simulation platform is released online at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/DRL-Navigation</uri> .