Reward Shaping Research Articles

Velocity range-based reward shaping technique for effective map-less navigation with LiDAR sensor and deep reinforcement learning.

In recent years, sensor components similar to human sensory functions have been rapidly developed in the hardware field, enabling the acquisition of information at a level beyond that of humans, and in the software field, artificial intelligence technology has been utilized to enable cognitive abilities and decision-making such as prediction, analysis, and judgment. These changes are being utilized in various industries and fields. In particular, new hardware and software technologies are being rapidly applied to robotics products, showing a level of performance and completeness that was previously unimaginable. In this paper, we researched the topic of establishing an optimal path plan for autonomous driving using LiDAR sensors and deep reinforcement learning in a workplace without map and grid coordinates for mobile robots, which are widely used in logistics and manufacturing sites. For this purpose, we reviewed the hardware configuration of mobile robots capable of autonomous driving, checked the characteristics of the main core sensors, and investigated the core technologies of autonomous driving. In addition, we reviewed the appropriate deep reinforcement learning algorithm to realize the autonomous driving of mobile robots, defined a deep neural network for autonomous driving data conversion, and defined a reward function for path planning. The contents investigated in this paper were built into a simulation environment to verify the autonomous path planning through experiment, and an additional reward technique "Velocity Range-based Evaluation Method" was proposed for further improvement of performance indicators required in the real field, and the effectiveness was verified. The simulation environment and detailed results of experiments are described in this paper, and it is expected as guidance and reference research for applying these technologies in the field.

GRL-PS: Graph Embedding-Based DRL Approach for Adaptive Path Selection

Forwarding path selection for data traffic is one of the most fundamental operations in computer networks, whose performance drastically impacts both transmission efficiency and reliability in network domains. Although deep reinforcement learning (DRL) has attracted considerable attention for path selection instead of hand-tuned heuristics, few works have considered how to exploit graph-structured information in networks to improve routing and forwarding efficiency. In fact, generating routes is essentially a process for finding a subgraph in a graph-structured network. To this end, this paper proposes an effective and novel graph embedding-based DRL framework for adaptive path selection (termed GRL-PS), aiming at reducing end-to-end (E2E) latency and promoting network throughput while maintaining stability in dynamically changing environments. Specifically, graph representation learning (GRL) is deployed as an effective enabler for the DRL agent to learn the relational knowledge of interacting entities for route decisions in networks. However, training such an agent in a dynamically changing environment encounters a knowledge acquisition bottleneck, since the DRL agent is always forced to learn every task from scratch. To improve the adaptation of behaviors and acquire skills beyond what the source policy can teach, we introduce potential-based reward shaping as a means of knowledge transfer to guide the agent in unfamiliar conditions with sparse rewards. Experimental results show that compared with baseline methods, our solution can achieve nearly-optimal performance with both latency and throughput, especially in large-scale dynamic networks.

Leveraging Minimum Nodes for Optimum Key Player Identification in Complex Networks: A Deep Reinforcement Learning Strategy with Structured Reward Shaping

The problem of finding key players in a graph, also known as network dismantling, or network disintegration, aims to find an optimal removal sequence of nodes (edges, substructures) through a certain algorithm, ultimately causing functional indicators such as the largest connected component (GCC) or network pair connectivity in the graph to rapidly decline. As a typical NP-hard problem on graphs, recent methods based on reinforcement learning and graph representation learning have effectively solved such problems. However, existing reinforcement-learning-based key-player-identification algorithms often need to remove too many nodes in order to achieve the optimal effect when removing the remaining network until no connected edges remain. The use of a minimum number of nodes while maintaining or surpassing the performance of existing methods is a worthwhile research problem. To this end, a novel algorithm called MiniKey was proposed to tackle such challenges, which employs a specific deep Q-network architecture for reinforcement learning, a novel reward-shaping mechanism based on network functional indicators, and the graph-embedding technique GraphSage to transform network nodes into latent representations. Additionally, a technique dubbed ‘virtual node technology’ is integrated to grasp the overarching feature representation of the whole network. This innovative algorithm can be effectively trained on small-scale simulated graphs while also being scalable to large-scale real-world networks. Importantly, experiments from both six simulated datasets and six real-world datasets demonstrates that MiniKey can achieve optimal performance, striking a perfect balance between the effectiveness of key node identification and the minimization of the number of nodes that is utilized, which holds potential for real-world applications such as curbing misinformation spread in social networks, optimizing traffic in transportation systems, and identifying key targets in biological networks for targeted interventions.

Autonomous Guidance Between Quasiperiodic Orbits in Cislunar Space via Deep Reinforcement Learning

This paper investigates the use of reinforcement learning for the fuel-optimal guidance of a spacecraft during a time-free low-thrust transfer between two libration point orbits in the cislunar environment. To this aim, a deep neural network is trained via proximal policy optimization to map any spacecraft state to the optimal control action. A general-purpose reward is used to guide the network toward a fuel-optimal control law, regardless of the specific pair of libration orbits considered and without the use of any ad hoc reward shaping technique. Eventually, the learned control policies are compared with the optimal solutions provided by a direct method in two different mission scenarios, and Monte Carlo simulations are used to assess the policies’ robustness to navigation uncertainties.

Kappa opioid receptor agonist U50,488 inhibits dopamine more in caudal than rostral nucleus accumbens core.

The nucleus accumbens (NAc) core is involved in regulating stress and shaping reward seeking behaviours. Multiple neuromodulators, including dynorphin/kappa opioid receptor (KOR) and dopamine systems, converge in this area to influence behavioural outcomes. KOR activation acutely inhibits dopamine release and chronically depresses overall dopamine transmission. Recently, studies in the NAc shell have revealed that the impact of KOR activation on behaviour is regionally specific, and these rostro-caudal differences are likely driven by greater control of KORs over dopamine inhibition in the caudal compared with rostral subregion. Given the importance of NAc core, particularly the interaction between KORs and dopamine in regulating reward seeking behaviours, we examined the impact of KOR activation on dopamine release and uptake along the rostro-caudal axis in the NAc core of male and female mice. Using ex vivo fast scan cyclic voltammetry, we observed that KOR mediated inhibition of dopamine release was significantly greater in caudal compared with rostral NAc core with no significant sex differences observed. These data suggest that KORs regulate dopamine release differentially along the rostro-caudal axis, providing a new axis on which to examine the process by which the KOR/dopamine system controls reward encoding.

Deep reinforcement learning for cooperative robots based on adaptive sentiment feedback

Human–robot cooperative tasks have gained importance with the emergence of robotics and artificial intelligence technology. In interactive reinforcement learning techniques, robots learn target tasks by receiving feedback from an experienced human trainer. However, most interactive reinforcement learning studies require a separate process to integrate the trainer’s feedback into the training dataset, making it challenging for robots to learn new tasks from humans in real-time. Furthermore, the types of feedback sentences that trainers can use are limited in previous research. To address these limitations, this paper proposes a robot teaching strategy that uses deep RL via human–robot interaction to learn table balancing tasks interactively. The proposed system employs Deep Q-Network with real-time sentiment feedback delivered through the trainer’s speech to learn cooperative tasks. We designed a novel reward function that incorporates sentiment feedback from human speech in real-time during the learning process. The paper presents an improved reward shaping technique based on subdivided feedback levels and shrinking feedback. This function serves as a guide for the robot to engage in natural interactions with humans and enables it to learn the tasks effectively. Experimental results demonstrate that the proposed interactive deep reinforcement learning model achieved a high success rate of up to 99.06%, outperforming the model without sentiment feedback.

Deep reinforcement learning with reward shaping for tracking control and vibration suppression of flexible link manipulator

This paper puts forward a novel deep reinforcement learning control using deep deterministic policy gradient (DRLC-DDPG) framework to address the reference tracking and vibration suppression problem of rotary flexible link (RFL) manipulator. Specifically, this study attempts to address the continuous action space DRLC problem through DDPG algorithm and presents a Lyapunov function based reward shaping approach for guaranteed deep reinforcement learning (DRL) convergence and enhanced speed of training. The proposed approach synthesizes the hard and soft constraints of the flexible manipulator as a constrained Markov decision problem (MDP) and evaluates the performance of DRLC-DDPG framework through hardware in loop (HIL) testing to realize precise servo tracking and suppressed vibration of the flexible manipulator. For identifying the dynamical model of the RFL, an empirical Auto-Regressive eXogenous (ARX) model using the closed loop identification technique is built. Moreover, to extract the true states (servo angle and deflection angle) from the actual measurements, which typically have the influence of sensor noise, an adaptive Kalman filter (AKF) is augmented with the DRLC scheme. The experimental results of DRLC-DDPG scheme compared with those of the model predictive control (MPC) for several test cases reveal that the proposed scheme is superior to MPC both in terms of trajectory tracking and robustness against the external disturbances and model uncertainty.

Reward shaping using convolutional neural network

In this paper, we propose Value Iteration Network for Reward Shaping (VIN-RS), a potential-based reward shaping mechanism using Convolutional Neural Network (CNN). The proposed VIN-RS embeds a CNN trained on computed labels using the message passing mechanism of the Hidden Markov Model. The CNN processes images or graphs of the environment to predict the shaping values. Recent work on reward shaping still has limitations towards training on a representation of the Markov Decision Process (MDP) and building an estimate of the transition matrix. The advantage of VIN-RS is to construct an effective potential function from an estimated MDP while automatically inferring the environment transition matrix. The proposed VIN-RS estimates the transition matrix through a self-learned convolution filter while extracting environment details from the input frames or sampled graphs. Due to (1) the previous success of using message passing for reward shaping; and (2) the CNN planning behavior, we use these messages to train the CNN of VIN-RS. Experiments are performed on tabular games, Atari 2600 and MuJoCo, for discrete and continuous action space. Our results illustrate promising improvements in the learning speed and maximum cumulative reward compared to the state-of-the-art. The improvement achieved by VIN-RS can only be observed for some of the games due to the underlying nature of some environments. In terms of the studied MuJoCo games, there is on average an increase of 30% in the maximum reward reached during early stages of learning.

Reinforcement Learning for Mobile Robotics Exploration: A Survey.

Efficient exploration of unknown environments is a fundamental precondition for modern autonomous mobile robot applications. Aiming to design robust and effective robotic exploration strategies, suitable to complex real-world scenarios, the academic community has increasingly investigated the integration of robotics with reinforcement learning (RL) techniques. This survey provides a comprehensive review of recent research works that use RL to design unknown environment exploration strategies for single and multirobots. The primary purpose of this study is to facilitate future research by compiling and analyzing the current state of works that link these two knowledge domains. This survey summarizes: what are the employed RL algorithms and how they compose the so far proposed mobile robot exploration strategies; how robotic exploration solutions are addressing typical RL problems like the exploration-exploitation dilemma, the curse of dimensionality, reward shaping, and slow learning convergence; and what are the performed experiments and software tools used for learning and testing. Achieved progress is described, and a discussion about remaining limitations and future perspectives is presented.

Policy Gradient From Demonstration and Curiosity.

With reinforcement learning, an agent can learn complex behaviors from high-level abstractions of the task. However, exploration and reward shaping remain challenging for existing methods, especially in scenarios where extrinsic feedback is sparse. Expert demonstrations have been investigated to solve these difficulties, but a tremendous number of high-quality demonstrations are usually required. In this work, an integrated policy gradient algorithm is proposed to boost exploration and facilitate intrinsic reward learning from only a limited number of demonstrations. We achieved this by reformulating the original reward function with two additional terms, where the first term measured the Jensen-Shannon divergence between current policy and the expert's demonstrations, and the second term estimated the agent's uncertainty about the environment. The presented algorithm was evaluated by a range of simulated tasks with sparse extrinsic reward signals, where only limited demonstrated trajectories were provided to each task. Superior exploration efficiency and high average return were demonstrated in all tasks. Furthermore, it was found that the agent could imitate the expert's behavior and meanwhile sustain high return.

Social learning in self-organizing systems for complex assembly tasks

Self-organizing systems have the potential to adapt to situation changes and gracefully deal with system degradations. Designing such systems, however, requires devising self-organizing knowledge into the individual agents of the system through either explicitly modelling the knowledge or allowing the agents to learn it by themselves through reinforcement learning. For tasks that are complex to a certain extent, the research has shown that developing proper reward functions with the help of reward shaping is an effective approach for successful multiagent reinforcement learning and hence for self-organizing systems design. In this case, agents often are individuals who learn by themselves through directly interacting with the task environment with little sense of the existence of other agents. As the task becomes more complex, explicitly considering the interactions among the agents during their learning process has evolved as a rich resource to explore for further eliciting the power of self-organizing systems. In this paper, social learning in its limited form—i.e., observing the actions of other agents—has been introduced to the multiagent reinforcement learning process, and a multiagent social deep Q-learning (MASo-DQL) has been developed and tested in the context of an assembly task complicated with obstacles. The experiment results have demonstrated that social learning can be effectively integrated into multiagent reinforcement learning frameworks through MASo-DQL; it has a positive impact on the training process and the task performance of the trained teams when the task complexity becomes high; and it entails costs and may lead to the inferior system performance for low complexity tasks.

Graph convolution with topology refinement for Automatic Reinforcement Learning

Reinforcement learning faces the challenge of sparse rewards. Existing research utilizes reward shaping based on graph convolutional neural networks (GCNs) to address this challenge. However, the automatic construction of optimal graph has been a long standing issue. Here we propose Graph Convolution with Topology Refinement for Automatic Reinforcement Learning (GTR), based on the construction of new latent graph to replace the original input graph for more effective reward shaping. It is found from this work that, the most suitable state node can be extracted through the graph entropy. Subsequently we map the original graph to subset of nodes adaptively to form a new and more compact latent graph. Since GTR utilizes trainable projection vectors for projecting all node features into one-dimensional representation, the inter-connections between the nodes of the newly constructed latent graph are consistent with the original ones. The proposed GTR stems from mathematical grounds, and preliminary experiments have shown that the proposed GTR has considerable improvement on Atari benchmark and Mujoco benchmark. Further experiment and ablation analysis have given further supports to this work.

Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learning

In reinforcement learning (RL), the ability to utilize prior knowledge from previously solved tasks can allow agents to quickly solve new problems. In some cases, these new problems may be approximately solved by composing the solutions of previously solved primitive tasks (task composition). Otherwise, prior knowledge can be used to adjust the reward function for a new problem, in a way that leaves the optimal policy unchanged but enables quicker learning (reward shaping). In this work, we develop a general framework for reward shaping and task composition in entropy-regularized RL. To do so, we derive an exact relation connecting the optimal soft value functions for two entropy-regularized RL problems with different reward functions and dynamics. We show how the derived relation leads to a general result for reward shaping in entropy-regularized RL. We then generalize this approach to derive an exact relation connecting optimal value functions for the composition of multiple tasks in entropy-regularized RL. We validate these theoretical contributions with experiments showing that reward shaping and task composition lead to faster learning in various settings.

Exploiting Multiple Abstractions in Episodic RL via Reward Shaping

One major limitation to the applicability of Reinforcement Learning (RL) to many practical domains is the large number of samples required to learn an optimal policy. To address this problem and improve learning efficiency, we consider a linear hierarchy of abstraction layers of the Markov Decision Process (MDP) underlying the target domain. Each layer is an MDP representing a coarser model of the one immediately below in the hierarchy. In this work, we propose a novel form of Reward Shaping where the solution obtained at the abstract level is used to offer rewards to the more concrete MDP, in such a way that the abstract solution guides the learning in the more complex domain. In contrast with other works in Hierarchical RL, our technique has few requirements in the design of the abstract models and it is also tolerant to modeling errors, thus making the proposed approach practical. We formally analyze the relationship between the abstract models and the exploration heuristic induced in the lower-level domain. Moreover, we prove that the method guarantees optimal convergence and we demonstrate its effectiveness experimentally.

SafeLight: A Reinforcement Learning Method toward Collision-Free Traffic Signal Control

Traffic signal control is safety-critical for our daily life. Roughly one-quarter of road accidents in the U.S. happen at intersections due to problematic signal timing, urging the development of safety-oriented intersection control. However, existing studies on adaptive traffic signal control using reinforcement learning technologies have focused mainly on minimizing traffic delay but neglecting the potential exposure to unsafe conditions. We, for the first time, incorporate road safety standards as enforcement to ensure the safety of existing reinforcement learning methods, aiming toward operating intersections with zero collisions. We have proposed a safety-enhanced residual reinforcement learning method (SafeLight) and employed multiple optimization techniques, such as multi-objective loss function and reward shaping for better knowledge integration. Extensive experiments are conducted using both synthetic and real-world benchmark datasets. Results show that our method can significantly reduce collisions while increasing traffic mobility.

Soft Action Priors: Towards Robust Policy Transfer

Despite success in many challenging problems, reinforcement learning (RL) is still confronted with sample inefficiency, which can be mitigated by introducing prior knowledge to agents. However, many transfer techniques in reinforcement learning make the limiting assumption that the teacher is an expert. In this paper, we use the action prior from the Reinforcement Learning as Inference framework - that is, a distribution over actions at each state which resembles a teacher policy, rather than a Bayesian prior - to recover state-of-the-art policy distillation techniques. Then, we propose a class of adaptive methods that can robustly exploit action priors by combining reward shaping and auxiliary regularization losses. In contrast to prior work, we develop algorithms for leveraging suboptimal action priors that may nevertheless impart valuable knowledge - which we call soft action priors. The proposed algorithms adapt by adjusting the strength of teacher feedback according to an estimate of the teacher's usefulness in each state. We perform tabular experiments, which show that the proposed methods achieve state-of-the-art performance, surpassing it when learning from suboptimal priors. Finally, we demonstrate the robustness of the adaptive algorithms in continuous action deep RL problems, in which adaptive algorithms considerably improved stability when compared to existing policy distillation methods.

Multi-objective reward shaping for global and local trajectory planning of wing-in-ground crafts based on deep reinforcement learning

AbstractThe control of a wing-in-ground craft (WIG) usually allows for many needs, like cruising, speed, survival and stealth. Various degrees of emphasis on these requirements result in different trajectories, but there has not been a way of integrating and quantifying them yet. Moreover, most previous studies on other vehicles’ multi-objective trajectory is planned globally, lacking for local planning. For the multi-objective trajectory planning of WIGs, this paper proposes a multi-objective function in a polynomial form, in which each item represents an independent requirement and is adjusted by a linear or exponential weight. It uses the magnitude of weights to demonstrate how much attention is paid relatively to the corresponding demand. Trajectories of a virtual WIG model above the wave trough terrain are planned using reward shaping based on the introduced multi-objective function and deep reinforcement learning (DRL). Two conditions are considered globally and locally: a single scheme of weights is assigned to the whole environment, and two different schemes of weights are assigned to the two parts of the environment. Effectiveness of the multi-object reward function is analysed from the local and global perspectives. The reward function provides WIGs with a universal framework for adjusting the magnitude of weights, to meet different degrees of requirements on cruising, speed, stealth and survival, and helps WIGs guide an expected trajectory in engineering.

Reward shaping with hierarchical graph topology

Reward shaping using GCNs is a popular research area in reinforcement learning. However, it is difficult to shape potential functions for complicated tasks. In this paper, we develop Reward Shaping with Hierarchical Graph Topology (HGT). HGT propagates information about the rewards through the message passing mechanism, which can be used as potential functions for reward shaping. We describe reinforcement learning by a probability graph model. Then we generate a underlying graph with each state is a node and edges represent transition probabilities between states. In order to prominently shape potential functions for complex environments, HGT divides the underlying graph constructed from states into multiple subgraphs. Since these subgraphs provide a representation of multiple logical relationships between states in the Markov decision process, the aggregation process rich correlation information between nodes, which makes the propagated messages more powerful. When compared to cutting-edge RL techniques, HGT achieves faster learning rates in experiments on Atari and Mujoco tasks.

Coordinated Slicing and Admission Control Using Multi-Agent Deep Reinforcement Learning

5G Cloud Radio Access Networks (C-RANs) facilitate new forms of flexible resource management as dynamic RAN function splitting and placement. Virtualized RAN functions can be placed at different sites in the substrate network based on resource availability and slice constraints. Due to limited resources in the substrate network and variability in revenue of slices, the Infrastructure Provider (InP) must perform network slicing in a strategic manner, and accept or reject slice-requests to maximize long-term revenue. In this paper, we propose to use multi-agent Deep Reinforcement Learning (DRL) to jointly solve the problems of network slicing and slice Admission Control (AC). Multi-agent DRL along with reward shaping is a promising choice, which is well-suited to problems where multiple distinct tasks have to be performed optimally. The proposed DRL approach can learn the dynamics of slice-request traffic and effectively address these joint problems. We compare multi-agent DRL to approaches that use: (i) simple heuristics to address the problems, and (ii) DRL to address either slicing or AC. Our results show that the proposed approach achieves up to 30% and 5.18% gain in long-term InP revenue when compared to approaches (i) and (ii), respectively. Additionally, we show that multi-agent DRL is preferable to a single-agent DRL approach for the joint problems in terms of convergence time and InP revenue. Finally, we evaluate the robustness of the trained agents in scenarios that differ from training, such as different arrival rates and real dynamic traffic patterns.

MultiselfGAN: A Self-Guiding Neural Architecture Search Method for Generative Adversarial Networks With Multicontrollers

In recent years, Reinforcement Learning and Gradient optimization were applied with Neural Architecture Search algorithms in Generative Adversarial Network to achieve their state-of-the-art (SOTA) performance. However, the existing RL-based methods utilised the calculation of Inception Score or Fréchet Inception Distance as the reward value to guide the controller, which actually wasted much of searching time. In order to improve the search efficiency without degradation of performance, this paper proposes recycling the discriminator to evaluate the performance of architectures, in other words, we propose to self-guide the search process. In the mean time, we introduce new concept of multiple controllers and the method of reward shaping to independently and effectively search the cell architectures. The experiments demonstrate the effectiveness and efficiency of our Multi-Self GAN and the ablation study also exhibits its robustness.