Navigation Policy Research Articles

Learning to Navigate Sidewalks in Outdoor Environments

Outdoor navigation on sidewalks in urban environments is the key technology behind important human assistive applications, such as last-mile delivery or neighborhood patrol. This letter aims to develop a quadruped robot that follows a route plan generated by public map services, while remaining on sidewalks and avoiding collisions with obstacles and pedestrians. We devise a two-staged learning framework, which first trains a teacher agent in an abstract world with privileged ground-truth information, and then applies Behavior Cloning to teach the skills to a student agent who only has access to realistic sensors. The main research effort of this letter focuses on overcoming challenges when deploying the student policy on a quadruped robot in the real world. We propose methodologies for designing sensing modalities, network architectures, and training procedures to enable zero-shot policy transfer to unstructured and dynamic real outdoor environments. We evaluate our learning framework on a quadrupedal robot navigating sidewalks in the city of Atlanta, USA ( <xref ref-type="fig" rid="fig1" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Fig. 1</xref> ). Using the learned navigation policy and its onboard sensors, the robot is able to walk 3.2 kilometers with a limited number of human interventions.

Efficient localization in warehouse logistics: a comparison of LMS approaches for 3D multilateration of passive UHF RFID tags

The Fourth Industrial Revolution, or Industry 4.0, aims at automating traditional manufacturing and industrial practices using modern smart technology. Autonomous mobile robots equipped with different sensors and employing different techniques have been proposed in the literature in the logistics and inventory warehouse management contexts. Efficient robot motion and planning depend on precise localization, mapping, and awareness of the environment. To properly localize items, recent attempts have been made employing radio frequency identification (RFID) in a 3D environment. This manuscript introduces four least mean squares methods to estimate the 3D positions of tags employing synthetic apertures and phase unwrapping. The proposed methods approach the localization problem solving a system of equations typical of multilateration methods to find the intersections of multiple hyperboloids. The novelty introduced here is the use of unwrapped phase distances to compute pseudo ranges for the multilateration problem. The use of such a technique is feasible thanks to precise mobile robot localization and custom navigation policies. All the analyzed methods are suitable for online localization due to reduced computation timings and have been tested on three different datasets. Two of them contain virtual data that has been generated by simulation, while the other one comes from an indoor experimental setup. Simulated tests show that combined antenna motions in 3D space (including Z-axis) improve the localization obtaining errors under the centimeter for 3D localization. Experimental tests obtained results as low as 13 cm of mean accuracy for 3D localization and 0.21 cm for 2D localization.

Vision-Based 2D Navigation of Unmanned Aerial Vehicles in Riverine Environments with Imitation Learning

There have been many researchers studying how to enable unmanned aerial vehicles (UAVs) to navigate in complex and natural environments autonomously. In this paper, we develop an imitation learning framework and use it to train navigation policies for the UAV flying inside complex and GPS-denied riverine environments. The UAV relies on a forward-pointing camera to perform reactive maneuvers and navigate itself in 2D space by adapting the heading. We compare the performance of a linear regression-based controller, an end-to-end neural network controller and a variational autoencoder (VAE)-based controller trained with data aggregation method in the simulation environments. The results show that the VAE-based controller outperforms the other two controllers in both training and testing environments and is able to navigate the UAV with a longer traveling distance and a lower intervention rate from the pilots.

A ship navigation information service system for the Arctic Northeast Passage using 3D GIS based on big Earth data

ABSTRACT Research on Arctic passages has mainly focused on navigation policies, sea ice extraction models, and navigation of Arctic sea routes. It is difficult to quantitatively address the specific problems encountered by ships sailing in the Arctic in real time through traditional manual approaches. Additionally, existing sea ice information service systems focus on data sharing and lack online calculation and analysis capabilities, making it difficult for decision-makers to derive valuable information from massive amounts of data. To improve navigation analysis through intelligent information service, we built an advanced Ship Navigation Information Service System (SNISS) using a 3D geographic information system (GIS) based on big Earth data. The SNISS includes two main features: (1) heuristic algorithms were developed to identify the optimal navigation route of the Arctic Northeast Passage (NEP) from a macroscale perspective for the past 10 years to the next 100 years, and (2) for key sea straits along the NEP, online local sea-ice images can be retrieved to provide a fully automatic sea ice data processing workflow, solving the problems of poor flexibility and low availability of real sea ice remote sensing data extraction. This work can potentially enhance the safety of shipping navigation along the NEP.

Multigoal Visual Navigation With Collision Avoidance via Deep Reinforcement Learning

Learning to map the images acquired by a moving agent equipped with a camera sensor to motion commands for multigoal navigation is challenging. Most existing approaches are still struggling against collision avoidance, faster convergence, and generalization. In this article, a novel actor&#x2013;critic architecture is presented to learn the optimal navigation policy. We introduce single-step reward observation and collision penalty to reshape the reinforcement learning (RL) reward function. The collision perception can be obtained by the reshaped reward function and treated as measurement information from the visual observation to avoid obstacles. Besides, expert trajectories are used to generate subgoals. A subgoal reward shaping is then proposed to accelerate policy learning with the expert knowledge of subgoals. In order to generate human-aware navigation policies, an observation-action consistency (OAC) model is introduced to ensure that the agent reaches the subgoals in turn, and moves toward the target. The whole training process is performed on a self-supervised RL approach, accompanied by an expert supervision signal. This method balances the exploration and exploitation, helping the proposed model to generalize to unseen goals. The training experiments on AI2-THOR show better performance and faster convergence speed, compared with the existing approaches. For the generalization capacity to unseen goals, the proposed method achieves the state-of-the-art success rate, with at least a 30&#x0025; improvement of average episode collision.

An Open-Source Low-Cost Mobile Robot System With an RGB-D Camera and Efficient Real-Time Navigation Algorithm

Currently, mobile robots are developing rapidly and are finding numerous applications in the industry. However, several problems remain related to their practical use, such as the need for expensive hardware and high power consumption levels. In this study, we build a low-cost indoor mobile robot platform that does not include a LiDAR or a GPU. Then, we design an autonomous navigation architecture that guarantees real-time performance on our platform with an RGB-D camera and a low-end off-the-shelf single board computer. The overall system includes SLAM, global path planning, ground segmentation, and motion planning. The proposed ground segmentation approach extracts a traversability map from raw depth images for the safe driving of low-body mobile robots. We apply both rule-based and learning-based navigation policies using the traversability map. Running sensor data processing and other autonomous driving components simultaneously, our navigation policies perform rapidly at a refresh rate of 18 Hz for control command, whereas other systems have slower refresh rates. Our methods show better performances than current state-of-the-art navigation approaches within limited computation resources as shown in 3D simulation tests. In addition, we demonstrate the applicability of our mobile robot system through successful autonomous driving in an indoor environment.

Learning efficient navigation in vortical flow fields

Efficient point-to-point navigation in the presence of a background flow field is important for robotic applications such as ocean surveying. In such applications, robots may only have knowledge of their immediate surroundings or be faced with time-varying currents, which limits the use of optimal control techniques. Here, we apply a recently introduced Reinforcement Learning algorithm to discover time-efficient navigation policies to steer a fixed-speed swimmer through unsteady two-dimensional flow fields. The algorithm entails inputting environmental cues into a deep neural network that determines the swimmer’s actions, and deploying Remember and Forget Experience Replay. We find that the resulting swimmers successfully exploit the background flow to reach the target, but that this success depends on the sensed environmental cue. Surprisingly, a velocity sensing approach significantly outperformed a bio-mimetic vorticity sensing approach, and achieved a near 100% success rate in reaching the target locations while approaching the time-efficiency of optimal navigation trajectories.

A Sim-to-Real Pipeline for Deep Reinforcement Learning for Autonomous Robot Navigation in Cluttered Rough Terrain

Robots that autonomously navigate real-world 3D cluttered environments need to safely traverse terrain with abrupt changes in surface normals and elevations. In this letter, we present the development of a novel sim-to-real pipeline for a mobile robot to effectively learn how to navigate real-world 3D rough terrain environments. The pipeline uses a deep reinforcement learning architecture to learn a navigation policy from training data obtained from the simulated environment and a unique combination of strategies to directly address the reality gap for such environments. Experiments in the real-world 3D cluttered environment verified that the robot successfully performed point-to-point navigation from arbitrary start and goal locations while traversing rough terrain. A comparison study between our DRL method, classical, and deep learning-based approaches showed that our method performed better in terms of success rate, and cumulative travel distance and time in a 3D rough terrain environment.

DYNAMIC ROUTING FOR NAVIGATION IN CHANGING UNKNOWN MAPS USING DEEP REINFORCEMENT LEARNING

Abstract. In this work, we propose an approach for an autonomous agent that learns to navigate in an unknown map in a real-world environment. Recognizing that the real-world environment is changing overtime such as road-closure happening due to construction work, a key contribution of our paper is adopt the dynamic adaptation characteristic of the reinforcement learning approach and develop a dynamic routing ability for our agent. Our method is based on the Q-learning algorithm and modifies it into a double-critic Qlearning model (DCQN) that only uses visual input without other aids such as GPS. Our treatment of the problem enables the agent to learn the navigation policy while interacting with the environment. We demonstrate that the agent can learn navigating to the destination kilometers away from the starting point in a real world scenario and has the ability to respond to environment changes while learning to adjust the routing plan dynamically by adjusting the old knowledge. The supplementary video can be accessed at the following link: https://www.youtube.com/watch?v=tknsxVuNwkg.

Improving Autonomous Robotic Navigation Using Imitation Learning.

Autonomous navigation to a specified waypoint is traditionally accomplished with a layered stack of global path planning and local motion planning modules that generate feasible and obstacle-free trajectories. While these modules can be modified to meet task-specific constraints and user preferences, current modification procedures require substantial effort on the part of an expert roboticist with a great deal of technical training. In this paper, we simplify this process by inserting a Machine Learning module between the global path planning and local motion planning modules of an off-the shelf navigation stack. This model can be trained with human demonstrations of the preferred navigation behavior, using a training procedure based on Behavioral Cloning, allowing for an intuitive modification of the navigation policy by non-technical users to suit task-specific constraints. We find that our approach can successfully adapt a robot’s navigation behavior to become more like that of a demonstrator. Moreover, for a fixed amount of demonstration data, we find that the proposed technique compares favorably to recent baselines with respect to both navigation success rate and trajectory similarity to the demonstrator.

Model Recognition as Planning

Given a partially observed plan execution, and a set of possible planning models (models that share the same state variables but different action schemata), model recognition is the task of identifying the model that explains the observation. The paper formalizes this task and introduces a novel method that estimates the probability of a STRIPS model to produce an observation of a plan execution. This method builds on top of off-the-shelf classical planning algorithms and it is robust to missing actions and intermediate states in the observation. The effectiveness of the method is tested in three experiments, each encoding a set of different STRIPS models and all using empty-action observations: (1) a classical string classification task; (2) identification of the model that encodes a failure present in an observation; and (3) recognition of a robot navigation policy.

Learn to Navigate Autonomously Through Deep Reinforcement Learning

In this article, we propose a deep reinforcement learning (DRL) algorithm as well as a novel tailored neural network architecture for mobile robots to learn navigation policies autonomously. We first introduce a new feature extractor to better acquire critical spatiotemporal features from raw depth images. In addition, we present a double-source scheme so that the experiences are collected from both the proposed model and a conventional planner alternatively based on a switching criterion to provide more diverse and comprehensive samples for learning. Moreover, we also propose a dual-soft-actor–critic architecture to train two sets of networks with different purposes simultaneously. Specifically, the primary network aims to learn the autonomous navigation policy, while the secondary network aims to learn the depth feature extractor. In this way, the learning performance can be improved through decoupling representation learning from policy learning and training the feature extractor separately with more specific goals. Experimental results have demonstrated the remarkable performance of the proposed model. It outperforms both the conventional and the state-of-the-art DRL-based methods consistently in terms of success rate. Meanwhile, it also exhibits higher trajectory quality and better generalization capability compared to existing DRL-based methods. Videos of our experiments are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://youtu.be/evjU6bOU3UY</uri> or OneDrive.

Target‐driven visual navigation in indoor scenes using reinforcement learning and imitation learning

Here, the challenges of sample efficiency and navigation performance in deep reinforcement learning for visual navigation are focused and a deep imitation reinforcement learning approach is proposed. Our contributions are mainly three folds: first, a framework combining imitation learning with deep reinforcement learning is presented, which enables a robot to learn a stable navigation policy faster in the target-driven navigation task. Second, the surrounding images is taken as the observation instead of sequential images, which can improve the navigation performance for more information. Moreover, a simple yet efficient template matching method is adopted to determine the stop action, making the system more practical. Simulation experiments in the AI-THOR environment show that the proposed approach outperforms previous end-to-end deep reinforcement learning approaches, which demonstrate the effectiveness and efficiency of our approach.

Visual Navigation Among Humans With Optimal Control as a Supervisor

Real world visual navigation requires robots to operate in unfamiliar, human-occupied dynamic environments. Navigation around humans is especially difficult because it requires anticipating their future motion, which can be quite challenging. We propose an approach that combines learning-based perception with model-based optimal control to navigate among humans based only on monocular, first-person RGB images. Our approach is enabled by our novel data-generation tool, HumANav, that allows for photorealistic renderings of indoor environment scenes with humans in them, which are then used to train the perception module entirely in simulation. Through simulations and experiments on a mobile robot, we demonstrate that the learned navigation policies can anticipate and react to humans without explicitly predicting future human motion, generalize to previously unseen environments and human behaviors, and transfer directly from simulation to reality. Videos describing our approach and experiments, as well as a demo of HumANav are available on the project website. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>

Automated Labeling for Robotic Autonomous Navigation Through Multi-Sensory Semi-Supervised Learning on Big Data

Imitation learning holds the promise to address challenging robotic tasks such as autonomous navigation. It however requires a human supervisor to oversee the training process and send correct control commands to robots without feedback, which is always prone to error and expensive. To minimize human involvement and avoid manual labeling of data in the robotic autonomous navigation with imitation learning, this paper proposes a novel semi-supervised imitation learning solution based on a multi-sensory design. This solution includes a suboptimal sensor policy based on sensor fusion to automatically label states encountered by a robot to avoid human supervision during training. In addition, a recording policy is developed to throttle the adversarial affect of learning too much from the suboptimal sensor policy. As a result, this solution allows the robot to learn a navigation policy in a self-supervised manner without human intervention after the initial data collection. With extensive experiments in indoor environments, this solution can achieve near human performance in most of the tasks and even surpasses human performance in case of unexpected events such as hardware failures or human operation errors. To best of our knowledge, this is the first work that synthesizes sensor fusion and imitation learning to enable robotic autonomous navigation in the real world without human supervision.

A Behavior-Based Mobile Robot Navigation Method with Deep Reinforcement Learning

Deep reinforcement learning-based mobile robot navigation has attracted some recent interest. In the single-agent case, a robot can learn to navigate autonomously without a map of the environment. In the multi-agent case, robots can learn to avoid collisions with each other. In this work, we propose a behavior-based mobile robot navigation method which directly maps the raw sensor data and goal information to the control command. The learned navigation policy can be applied in both single-agent and multi-agent scenarios. Two basic navigation behaviors are considered in our method, which are goal reaching and collision avoidance. The two behaviors are fused based on the risk-level estimation of the current state. The navigation task is decomposed using the behavior-based framework, which is capable of reducing the complexity of the learning process. The simulations and real-world experiments demonstrate that the proposed method can enable the collision-free autonomous navigation of multiple mobile robots in unknown environments.

Simultaneous Navigation and Radio Mapping for Cellular-Connected UAV With Deep Reinforcement Learning

Cellular-connected unmanned aerial vehicle (UAV) is a promising technology to unlock the full potential of UAVs in the future by reusing the cellular base stations (BSs) to enable their air-ground communications. However, how to achieve ubiquitous three-dimensional (3D) communication coverage for the UAVs in the sky is a new challenge. In this paper, we tackle this challenge by a new coverage-aware navigation approach, which exploits the UAV’s controllable mobility to design its navigation/trajectory to avoid the cellular BSs’ coverage holes while accomplishing their missions. To this end, we formulate an UAV trajectory optimization problem to minimize the weighted sum of its mission completion time and expected communication outage duration, which, however, cannot be solved by the standard optimization techniques due to the lack of an accurate and tractable end-to-end communication model in practice. To overcome this difficulty, we propose a new solution approach based on the technique of deep reinforcement learning (DRL). Specifically, by leveraging the state-of-the-art dueling double deep Q network (dueling DDQN) with multi-step learning, we first propose a UAV navigation algorithm based on direct RL, where the signal measurement at the UAV is used to directly train the action-value function of the navigation policy. To further improve the performance, we propose a new framework called simultaneous navigation and radio mapping (SNARM) , where the UAV’s signal measurement is used not only for training the DQN directly, but also to create a radio map that is able to predict the outage probabilities at all locations in the area of interest. This enables the generation of simulated UAV trajectories and predicting their expected returns, which are then used to further train the DQN via Dyna technique, thus greatly improving the learning efficiency.

Learning With Stochastic Guidance for Robot Navigation.

Due to the sparse rewards and high degree of environmental variation, reinforcement learning approaches, such as deep deterministic policy gradient (DDPG), are plagued by issues of high variance when applied in complex real-world environments. We present a new framework for overcoming these issues by incorporating a stochastic switch, allowing an agent to choose between high- and low-variance policies. The stochastic switch can be jointly trained with the original DDPG in the same framework. In this article, we demonstrate the power of the framework in a navigation task, where the robot can dynamically choose to learn through exploration or to use the output of a heuristic controller as guidance. Instead of starting from completely random actions, the navigation capability of a robot can be quickly bootstrapped by several simple independent controllers. The experimental results show that with the aid of stochastic guidance, we are able to effectively and efficiently train DDPG navigation policies and achieve significantly better performance than state-of-the-art baseline models.

Autonomous Unmanned Aerial Vehicle for Post-Disaster Management With Cognitive Radio Communication

Unmanned aerial vehicles (UAV) play a significant role in finding victims affected in the post-disaster zone, where a man cannot risk his life under a critical condition of the disaster environment. The proposed design incorporates autonomous vision-based navigation through the disaster environment based on general graph theory with dynamic changes on the length between two or multiple nodes, where a node is a pathway. Camera fixed on it continuously captures the surrounding footage, processing it frame by frame on-site using image processing technique based on a SOC. Identifies victims in the zone and the pathways available for traversal. UAV uses an ultrasonic rangefinder to avoid collision with obstacles. The system alerts the rescue team if any victim detected and transmits the frames using CRN to the off-site console. UAV learns navigation policy that achieves high accuracy in real-time environments; communication using CRN is uninterrupted and useful during such emergencies.

Integrating Multiple Policies for Person-Following Robot Training Using Deep Reinforcement Learning

Given a training environment which follows Markov decision process for a specific task, a deep reinforcement learning (DRL) agent is able to find possible optimal policies which map states of the environment to appropriate actions by repeatedly trying various actions to maximize training rewards. However, the learned policies cannot be reused directly in the training process for other new tasks resulting wasted precious time and resources. To solve this problem, we propose a DRL-based method for training an agent capable of selecting the appropriate policy for current state of the environment from a set of previously trained optimal policies for a given task which can be decomposed into other sub tasks. We implement our proposed method to a person-following robot task training that can be broken down into three sub tasks, namely: navigation, left attending, and right attending. Using the proposed method, the previously learned optimal navigation policy obtained from our previous work is integrated to attending policies which are trained in this study. We also introduce the use of weight-scheduled action smoothing which is able to stabilize actions generated by the agent in the attending task training. Our experiment results show that the proposed method is able to integrate all sub policies using the action smoothing method even though the navigation and the attending policies have dissimilar input structures, unalike output ranges, and are trained in different ways. Moreover, our proposed method shows better results compared to training from scratch and training using transfer learning strategy.