Proposed Reward Function Research Articles

Mapless navigation via Hierarchical Reinforcement Learning with memory-decaying novelty

Hierarchical Reinforcement Learning (HRL) has shown superior performance for mapless navigation tasks. However, it remains limited in unstructured environments that might contain terrains like long corridors and dead corners, which can lead to local minima. This is because most HRL-based mapless navigation methods employ a simplified reward setting and exploration strategy. In this work, we propose a novel reward function for training the high-level (HL) policy, which contains two components: extrinsic reward and intrinsic reward. The extrinsic reward encourages the robot to move towards the target location, while the intrinsic reward is computed based on novelty, episode memory and memory decaying, making the agent capable of accomplishing spontaneous exploration. We also design a novel neural network structure that incorporates an LSTM network to augment the agent with memory and reasoning capabilities. We test our method in unknown environments and specific scenarios prone to the local minimum problem to evaluate the navigation performance and local minimum resolution ability. The results show that our method significantly increases the success rate when compared to advanced RL-based methods, achieving a maximum improvement of nearly 28%. Our method demonstrates effective improvement in addressing the local minimum issue, especially in cases where the baselines fail completely. Additionally, numerous ablation studies consistently confirm the effectiveness of our proposed reward function and neural network structure.

Path planning of 6-DOF free-floating space robotic manipulators using reinforcement learning

This paper presents a study on path planning for 6-DOF free-floating space robotic manipulators using Deep Deterministic Policy Gradient-based Reinforcement Learning. The focus is the development of a novel reward function tailored to address critical requirements for efficient and effective manipulation in space. These requirements include accurate pose alignment between the end-effector and the target, collision avoidance with both the target and other links of the manipulator, smoothing of joint velocities, adaptability to strong dynamic coupling between the manipulator and its base spacecraft due to high manipulator-spacecraft mass ratio, and resilience to noise in the state observations. Uniquely, the proposed reward function employs quaternions for orientation control to reduce pose misalignments and dynamic singularities, as opposed to traditional Euler angles. Our findings demonstrate that the Reinforcement Learning algorithm, when guided by this new reward function that integrates these enhancements and constraints, not only achieves the desired path planning objectives more efficiently but also exhibits faster convergence. Furthermore, the Reinforcement Learning successfully manages significant dynamic coupling effects caused by a high mass ratio between the robotic manipulator and the base spacecraft. Even under the challenge of noisy state observations, the trained agent successfully completes the path planning task, proving the Reinforcement Learning's applicability to real-space mission designs where the noise in observation is inevitable. The study highlights the critical role of reward function design in the Reinforcement Learning training process and its consequential impact on the solution quality, providing a solid foundation for future advancements in free-floating space robotic missions.

Reinforcement learning for path planning of free-floating space robotic manipulator with collision avoidance and observation noise

This study introduces a novel approach for the path planning of a 6-degree-of-freedom free-floating space robotic manipulator, focusing on collision and obstacle avoidance through reinforcement learning. It addresses the challenges of dynamic coupling between the spacecraft and the robotic manipulator, which significantly affects control and precision in the space environment. An innovative reward function is introduced in the reinforcement learning framework to ensure accurate alignment of the manipulator’s end effector with its target, despite disturbances from the spacecraft and the need for obstacle and collision avoidance. A key feature of this study is the use of quaternions for orientation representation to avoid the singularities associated with conventional Euler angles and enhance the training process’ efficiency. Furthermore, the reward function incorporates joint velocity constraints to refine the path planning for the manipulator joints, enabling efficient obstacle and collision avoidance. Another key feature of this study is the inclusion of observation noise in the training process to enhance the robustness of the agent. Results demonstrate that the proposed reward function enables effective exploration of the action space, leading to high precision in achieving the desired objectives. The study provides a solid theoretical foundation for the application of reinforcement learning in complex free-floating space robotic operations and offers insights for future space missions.

Towards learning-based energy-efficient online coordinated virtual network embedding framework

Network virtualization is a highly effective technology for resource sharing within data centers, enabling the coexistence of multiple heterogeneous virtual networks in a shared substrate network, thus achieving resource multiplexing. The efficient embedding of a virtual network onto a substrate network, known as the virtual network embedding (VNE) problem, has been proven to be NP-hard. In response to this challenge, this paper introduces a novel method, named PPO-VNE, which leverages deep reinforcement learning for virtual network embedding. PPO-VNE employs the Proximal Policy Optimization (PPO) algorithm to generate policies and efficiently coordinate node and link mapping. Furthermore, it adopts a hybrid feature extraction approach that combines handcrafted features with features extracted using graph convolutional networks. The proposed reward function takes multiple objectives into account, guiding the learning process. We implemented a prototype of PPO-VNE and conducted experiments based on the simulation environment, in which the substrate network has 100 nodes, with a probability of 0.1 generating edges between any two node, and eventually there will be about 500 physical links. We evaluate the performance of our PPO-VNE approach from the perspective of overall acceptance rate, overall revenue, revenue-to-cost ratio, maximum energy consumption per unit time and revenue-energy consumption coefficient. Comprehensive simulation results in different scenarios show that our PPO-VNE approach achieves the superior performance on most metrics compared with the existing state-of-the-art approaches, where the overall acceptance rate, overall revenue, revenue-energy consumption coefficient are increased by up to 6.4%, 21.3% and 41.4%, respectively, and the maximum energy consumption per unit time are reduced by up to 22.1%.

PPO-Based Attitude Controller Design for a Tilt Rotor UAV in Transition Process

The complex aerodynamic changes of the tilt-rotor UAV (TRUAV) in the transition process show strong nonlinearity, which brings a great impact on the stability of the vehicle attitude. This study aims to design a PPO-based RL controller for attitude control in the transition process. A reinforcement-learning PPO approach is used to learn the control strategy by interacting directly with the environment. And the reward function is designed and improved for the transition process. The performance of the proposed controller is tested and compared by simulation. The results show that the PPO algorithm is more suitable for the tilt-rotor transition process control than the A2C algorithm. Our proposed reward function improves the attitude control performance and the designed RL controller has good adaptability to changes in the takeoff weight, the diagonal wheelbase and the tilt rate. This study highlights the effectiveness and potential of reinforcement learning for tilt-rotor UAV transition process attitude control. These findings contribute to the advancement of autonomous flight systems by providing insights into the application of reinforcement learning algorithms. These results have important implications for the development of intelligent flight control systems and could guide future research in this area.

Hölder Divergence-Based Reward Function for Poisson RFSs and Application to Multitarget Sensor Management

In this study, we propose a novel information theoretic reward function based on the statistical Hölder divergence. The Hölder divergence is the generalization of Cauchy-Schwarz divergence. We extend the Hölder divergence to the FISST densities, thus making it possible to use in the multi-object applications based on the RFS theory. We derive the analytic expressions for the extended Hölder divergence (EHD) for the case when the multi-target densities have the form of Poisson RFSs and apply it to the PHD filter in a sequential Monte Carlo implementation. We evaluated the performance of the proposed reward function in a multi-target sensor management problem where the next position of a moving observer is decided according to the value of the EHD-based reward function. The performance of the algorithm is compared against, in terms of OSPA metric, and it is shown that the proposed reward function is superior to other similar reward functions in multi-target sensor management literature. We also show that it is possible to adapt the management algorithm to different situations, such as static and dynamic environments.

강화학습 기반 화학 공정 제어 성능 향상을 위한 보상 함수 시뮬레이션 연구

Process control is vital for operating chemical process safely and efficiently while maintaining product quality. Process control has been implemented by utilizing traditional techniques such as PID (Proportional-Integrate-Derivative) control and MPC (Model Predictive Control). However, these approaches have disadvantages when the PID tuning rules do not provide optimal tuning parameters for various operating scenarios or when the model fails to match the actual plant. RL (Reinforcement Learning) is one of the most prominent data-driven techniques for addressing such issues and has gained popularity in recent years. While RL has been extensively studied and succeeds in controlling chemical processes, the reward function has a significant impact on its performance. Therefore, it is essential to specify the reward function in order to efficiently control the process. In this study, we propose a reward function to enhance the performance of RL and compare the control performances of MPC and RLs with four distinct reward functions. The controllers track the setpoint of the product in the Van de Vusse process and are evaluated based on the deviation between the setpoint and output. RL with the proposed reward function outperformed the other controllers. Its performance was 12% greater than that of MPC and 6-30% greater than other RL controllers. In chemical processes, the control performance of RL is enhanced by incorporating the time term and positive reward in the reward function, thus outperforming conventional control approaches.

Cooperative Search Method for Multiple UAVs Based on Deep Reinforcement Learning.

In this paper, a cooperative search method for multiple UAVs is proposed to solve the problem of low efficiency of multi-UAV task execution by using a cooperative game with incomplete information. To improve search efficiency, CBBA (Consensus-Based Bundle Algorithm) is applied to designate the tasks area for each UAV. Then, Independent Deep Reinforcement Learning (IDRL) is used to solve Nash equilibrium to improve UAVs’ collaborations. The proposed reward function is smartly developed to guide UAVs to fly along the path with higher reward value while avoiding the collisions between UAVs during flights. Finally, extensive experiments are carried out to compare our proposed method with other algorithms. Simulation results show that the proposed method can obtain more rewards in the same period of time as other algorithms.

Efficient Detailed Routing for FPGA Back-End Flow Using Reinforcement Learning

Over the past few years, the computation capability of field-programmable gate arrays (FPGAs) has increased tremendously. This has led to the increase in the complexity of the designs implemented on FPGAs and to the time taken by the FPGA back-end flow. The FPGA back-end flow comprises of many steps, and routing is one of the most critical steps among them. Routing normally constitutes more than 50% of the total time taken by the back-end flow and an optimization at this step can lead to overall optimization of the back-end flow. In this work, we propose enhancements to the routing step by incorporating a reinforcement learning (RL)-based framework. In the proposed RL-based framework, we use the ϵ-greedy approach and customized reward functions to speed up the routing step while maintaining similar or better quality of results (QoR) as compared to the conventional negotiation-based congestion-driven routing solution. For experimentation, we use two sets of widely deployed, large heterogeneous benchmarks. Our results show that, for the RL-based framework, the ϵ-greedy greedy approach combined with a modified reward function gives better results as compared to purely greedy or exploratory approaches. Moreover, the incorporation of the proposed reward function in the RL-based framework and its comparison with a conventional routing algorithm shows that the proposed enhancement requires less routing time while giving similar or better QoR. On average, a speedup of 35% is recorded for the proposed routing enhancement as compared to negotiation-based congestion-driven routing solutions. Finally, the speedup of the routing step leads to an overall reduction in the execution time of the back-end flow of 25%.

Toward Observation Based Least Restrictive Collision Avoidance Using Deep Meta Reinforcement Learning

This letter presents the Observation-based Least-Restrictive Collision Avoidance Module (OLR-CAM) that can be added to any autonomous robot working in a shared environment and provide a high-level safety layer to the existing policy for each robot. The OLR-CAM takes raw sensory observations as input, evaluates the agents' safety against dynamic and static obstacles, and only intervenes the default policy when needed - in a least-restrictive fashion - to avoid a potential collision. In our approach, we meta-train the OLR-CAM policy within a “2D Navigation Meta World System”. Furthermore, to endow the policy with a notion of safety in multi-agent environments with obstacles, we propose a novel reward function based on a safety value function derived from the Hamilton-Jacobi reachability theory and a local cost map. The proposed reward function does not need any additional information about the environment's map. This facilitates the adoption of the algorithm in a new environment at the meta test stage. The proposed algorithm is fully meta-trained in simulation and tested on a real multi-agent system without any additional training conducted in the real setting. Our results show that the OLR-CAM success rate outperforms a well-known classical baseline approach by 10 percent on average and reduces the interruptions/changes to the preferred velocity by 15 percent.

Comparing Deep Reinforcement Learning Algorithms' Ability to Safely Navigate Challenging Waters.

Reinforcement Learning (RL) controllers have proved to effectively tackle the dual objectives of path following and collision avoidance. However, finding which RL algorithm setup optimally trades off these two tasks is not necessarily easy. This work proposes a methodology to explore this that leverages analyzing the performance and task-specific behavioral characteristics for a range of RL algorithms applied to path-following and collision-avoidance for underactuated surface vehicles in environments of increasing complexity. Compared to the introduced RL algorithms, the results show that the Proximal Policy Optimization (PPO) algorithm exhibits superior robustness to changes in the environment complexity, the reward function, and when generalized to environments with a considerable domain gap from the training environment. Whereas the proposed reward function significantly improves the competing algorithms’ ability to solve the training environment, an unexpected consequence of the dimensionality reduction in the sensor suite, combined with the domain gap, is identified as the source of their impaired generalization performance.

Control and Simulation of a 6-DOF Biped Robot based on Twin Delayed Deep Deterministic Policy Gradient Algorithm

Objectives: To study an algorithm to control a bipedal robot to walk so that it has a gait close to that of a human. It is known that the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm is a highly efficient algorithm with a few changes compared to the popular algorithm — the commonly used Deep Deterministic Policy Gradient (DDPG) in the continuous action space problem in Reinforcement Learning. Methods: Different from the usual sparse reward function model used, in this study, a reward model combined with a sparse reward function and dense reward function will be proposed. The application of the TD3 algorithm together with the proposed reward function model to control a bipedal robot model with 6 degrees of freedom will be presented. The training process is simulated in Gazebo/Robot Operating System (ROS) environment. Finding: The results show that, when choosing a reward model combined with a sparse reward function and a dense reward function suitable for the robot model, will help it learn faster and achieve better results. The biped robot can walk straight with an almost human-like gait. In the paper, the results from the TD3 algorithm combined with the proposed reward model are also compared with the results from other algorithms. Novelty: Applying the TD3 algorithm combined with the proposed reward model for the 6-DOF biped robot and simulating the robot’s gait in Gazebo/ROS environment, ROS is a middleware that can be used to control a robot in a real environment in the future. Keywords: TD3; biped robot; reinforcement learning; ROS; Gazebo

CURIOSITY-DRIVEN REINFORCEMENT LEARNING AGENT FOR MAPPING UNKNOWN INDOOR ENVIRONMENTS

Abstract. Autonomously exploring and mapping is one of the open challenges of robotics and artificial intelligence. Especially when the environments are unknown, choosing the optimal navigation directive is not straightforward. In this paper, we propose a reinforcement learning framework for navigating, exploring, and mapping unknown environments. The reinforcement learning agent is in charge of selecting the commands for steering the mobile robot, while a SLAM algorithm estimates the robot pose and maps the environments. The agent, to select optimal actions, is trained to be curious about the world. This concept translates into the introduction of a curiosity-driven reward function that encourages the agent to steer the mobile robot towards unknown and unseen areas of the world and the map. We test our approach in explorations challenges in different indoor environments. The agent trained with the proposed reward function outperforms the agents trained with reward functions commonly used in the literature for solving such tasks.

Energy-Efficient UAV-Enabled Data Collection via Wireless Charging: A Reinforcement Learning Approach

In this article, we study the application of unmanned aerial vehicle (UAV) for data collection with wireless charging, which is crucial for providing seamless coverage and improving system performance in the next-generation wireless networks. To this end, we propose a reinforcement learning-based approach to plan the route of UAV to collect sensor data from sensor devices scattered in the physical environment. Specifically, the physical environment is divided into multiple grids, where one spot for UAV hovering as well as the wireless charging of UAV is located at the center of each grid. Each grid has a spot for the UAV to hover, and moreover, there is a wireless charger at the center of each grid, which can provide wireless charging to UAV when it is hovering in the grid. When the UAV lacks energy, it can be charged by the wireless charger at the spot. By taking into account the collected data amount as well as the energy consumption, we formulate the problem of data collection with UAV as a Markov decision problem, and exploit $Q$ -learning to find the optimal policy. In particular, we design the reward function considering the energy efficiency of UAV flight and data collection, based on which $Q$ -table is updated for guiding the route of UAV. Through extensive simulation results, we verify that our proposed reward function can achieve a better performance in terms of the average throughput, delay of data collection, as well as the energy efficiency of UAV, in comparison with the conventional capacity-based reward function.

Deep Reinforcement Learning Multi-UAV Trajectory Control for Target Tracking

In this article, we propose a novel deep reinforcement learning (DRL) approach for controlling multiple unmanned aerial vehicles (UAVs) with the ultimate purpose of tracking multiple first responders (FRs) in challenging 3-D environments in the presence of obstacles and occlusions. We assume that the UAVs receive noisy distance measurements from the FRs which are of two types, i.e., Line of Sight (LoS) and non-LoS (NLoS) measurements and which are used by the UAV agents in order to estimate the state (i.e., position) of the FRs. Subsequently, the proposed DRL-based controller selects the optimal joint control actions according to the Cramér–Rao lower bound (CRLB) of the joint measurement likelihood function to achieve high tracking performance. Specifically, the optimal UAV control actions are quantified by the proposed reward function, which considers both the CRLB of the entire system and each UAV’s individual contribution to the system, called global reward and difference reward, respectively. Since the UAVs take actions that reduce the CRLB of the entire system, tracking accuracy is improved by ensuring the reception of high quality LoS measurements with high probability. Our simulation results show that the proposed DRL-based UAV controller provides a highly accurate target tracking solution with a very low runtime cost.

Diversifying Inference Path Selection: Moving-Mobile-Network for Landmark Recognition.

Deep convolutional neural networks have largely benefited computer vision tasks. However, the high computational complexity limits their real-world applications. To this end, many methods have been proposed for efficient network learning, and applications in portable mobile devices. In this paper, we propose a novel Moving-Mobile-Network, named M2Net, for landmark recognition, equipped each landmark image with located geographic information. We intuitively find that M2Net can essentially promote the diversity of the inference path (selected blocks subset) selection, so as to enhance the recognition accuracy. The above intuition is achieved by our proposed reward function with the input of geo-location and landmarks. We also find that the performance of other portable networks can be improved via our architecture. We construct two landmark image datasets, with each landmark associated with geographic information, over which we conduct extensive experiments to demonstrate that M2Net achieves improved recognition accuracy with comparable complexity.

Reinforcement learning-based optimal complete water-blasting for autonomous ship hull corrosion cleaning system

Routine cleaning of the corroded ship hulls in dry dock maintenance guarantees the smooth operation of the shipping industry. Deploying the autonomous system to remove the corrosion by water-blasting is a feasible approach to ease the burden in manual operation and to reduce water, time, and energy consumption. In this paper, the water-blasting framework is proposed for a novel robot platform named Hornbill with the adhesion mechanism by permanent magnetic, self-localization by sensor fusion to navigate smoothly on a vertical surface. Hence, we propose a complete waypoint path planning (CWPP) to re-blast the self-synthesizing deep convolutional neural network (DCNN) based corrosion heatmap by initial-blasting. The optimal CWPP problem, including the shortest travel distance and shortest travel time to save water, power while ensuring visiting all predefined waypoints by benchmarking output, is modeled as the classic Travel Salesman Problem (TSP). Further, the Pareto-optimal trajectory for given TSP has been driven by the reinforcement learning (RL) technique with a proposed reward function based on the robot's operation during blasting. From the experimental results at the shipyard site, the proposed RL-based CWPP generates the Pareto-optimal trajectory that enables the water-blasting robot to spend about 10% of energy and 9% of water less than the second-best evolutionary-based optimization method in various workspaces.

Reinforcement Q-Learning Control With Reward Shaping Function for Swing Phase Control in a Semi-active Prosthetic Knee.

In this study, we investigated a control algorithm for a semi-active prosthetic knee based on reinforcement learning (RL). Model-free reinforcement Q-learning control with a reward shaping function was proposed as the voltage controller of a magnetorheological damper based on the prosthetic knee. The reward function was designed as a function of the performance index that accounts for the trajectory of the subject-specific knee angle. We compared our proposed reward function to a conventional single reward function under the same random initialization of a Q-matrix. We trained this control algorithm to adapt to several walking speed datasets under one control policy and subsequently compared its performance with that of other control algorithms. The results showed that our proposed reward function performed better than the conventional single reward function in terms of the normalized root mean squared error and also showed a faster convergence trend. Furthermore, our control strategy converged within our desired performance index and could adapt to several walking speeds. Our proposed control structure has also an overall better performance compared to user-adaptive control, while some of its walking speeds performed better than the neural network predictive control from existing studies.

Learning Reward Function with Matching Network for Mapless Navigation.

Deep reinforcement learning (DRL) has been successfully applied in mapless navigation. An important issue in DRL is to design a reward function for evaluating actions of agents. However, designing a robust and suitable reward function greatly depends on the designer’s experience and intuition. To address this concern, we consider employing reward shaping from trajectories on similar navigation tasks without human supervision, and propose a general reward function based on matching network (MN). The MN-based reward function is able to gain the experience by pre-training through trajectories on different navigation tasks and accelerate the training speed of DRL in new tasks. The proposed reward function keeps the optimal strategy of DRL unchanged. The simulation results on two static maps show that the DRL converge with less iterations via the learned reward function than the state-of-the-art mapless navigation methods. The proposed method performs well in dynamic maps with partially moving obstacles. Even when test maps are different from training maps, the proposed strategy is able to complete the navigation tasks without additional training.

Information-Seeking Sensor Selection for Ocean-of-Things

We propose a general sensor selection (SS) methodology for ocean-of-things (OoT) where a sensing network performs multiobject tracking (MOT) under resource constraints. SS methods address the combinatorial problem of determining the best subset of sensors that maximizes a suitable reward function for a fixed cardinality. The novelty of this article is twofold. First, we propose a tractable information-theoretic reward function for MOT-OoT with an unknown and time-varying number of objects such as ocean vessels. A tractable reward function is essential in order to rapidly evaluate a sensor subset, which is crucial in the high-dimensional problems encountered in OoT. Second, we propose a general cross-entropy SS (CE-SS) methodology that efficiently estimates the probabilities of sensor activations and determines the optimal sensor subset according to the proposed reward function and under the imposed cardinality constraint. The CE-SS algorithm avoids exhaustive searching over the space of all sensor subsets, which is intractable for most OoT applications. The CE-SS methodology, coupled with the proposed reward function, is capable of selecting sensors that lead to more accurate estimates than random selection for both the number of vessels and their trajectories. We demonstrate the effectiveness of our method via numerical simulation in serveral scenarios, including multivessel tracking for OoT with an emulated network of acoustic sensors deployed off the coast of Italy.