Reward Problem Research Articles

Rapid-Learning Collaborative Pushing and Grasping via Deep Reinforcement Learning and Image Masking

When multiple objects are positioned close together or stacked, pre-grasp operations such as pushing objects can be used to create space for the grasp, thereby improving the grasping success rate. This study develops a model based on a deep Q-learning network architecture and introduces a fully convolutional network to accurately identify pixels in the workspace image that correspond to target locations for exploration. In addition, this study incorporates image masking to limit the exploration area of the robotic arm, ensuring that the agent consistently explores regions containing objects. This approach effectively addresses the sparse reward problem and improves the convergence rate of the model. Experimental results from both simulated and real-world environments show that the proposed method accelerates the learning of effective grasping strategies. When image masking is applied, the success rate in the grasping task reaches 80% after 600 iterations. The time required to reach 80% success rate is 25% shorter when image masking is used compared to when it is not used. The main finding of this study is the direct integration of image masking technique with a deep reinforcement learning (DRL) algorithm, which offers significant advancement in robotic arm control. Furthermore, this study shows that image masking technique can substantially reduce training time and improve the object grasping success rate. This innovation enables the robotic arm to better adapt to scenarios that conventional DRL methods cannot handle, thereby improving training efficiency and performance in complex and dynamic industrial applications.

Trajectory tracking control based on deep reinforcement learning and ensemble random network distillation for robotic manipulator

Abstract In general, the trajectory tracking of robotic manipulator is exceptionally challenging due to the complex and strongly coupled mechanical architecture. In this paper, precise track control of the robotic manipulator is formulated as a dense reward problem for reinforcement learning(RL). A deep RL(DRL) approach combining the soft actor-critic (SAC) algorithm and ensemble random network distillation (ERND) is proposed to address the tracking control problem for robotic manipulator. Firstly, an ERND model is designed, consisting of a module list of multiple RND models. Each RND model obtains the error by learning the target features and the predicted features of the environment. The resulting error serves as internal rewards that drive the robotic agent to explore unknown and unpredictable environmental states. The ensemble model obtains the total internal reward by summing the internal rewards of each RND model, thereby obtaining more accurately reflecting the characteristics of the manipulator in tracking control tasks and improving control performance. Secondly, combining the SAC algorithm with ERND facilitates more robust exploration capabilities in environments with input saturation and joint angle constraints, thereby enabling faster learning of effective policies and enhancing the performance and efficiency of robotic manipulator tracking control tasks. Finally, the simulation results demonstrate that the robotic manipulator tracking control task is effectively completed in dense reward problems through the combination of the SAC algorithm and ERND.

Deep Model-Based Reinforcement Learning for Predictive Control of Robotic Systems with Dense and Sparse Rewards

Sparse rewards and sample efficiency are open areas of research in the field of reinforcement learning. These problems are especially important when considering applications of reinforcement learning to robotics and other cyber-physical systems. This is so because in these domains many tasks are goal-based and naturally expressed with binary successes and failures, action spaces are large and continuous, and real interactions with the environment are limited. In this work, we propose Deep Value-and-Predictive-Model Control (DVPMC), a model-based predictive reinforcement learning algorithm for continuous control that uses system identification, value function approximation and sampling-based optimization to select actions. The algorithm is evaluated on a dense reward and a sparse reward task. We show that it can match the performance of a predictive control approach to the dense reward problem, and outperforms model-free and model-based learning algorithms on the sparse reward task on the metrics of sample efficiency and performance. We verify the performance of an agent trained in simulation using DVPMC on a real robot playing the reach-avoid game. Video of the experiment can be found here: https://youtu.be/0Q274kcfn4c.

Hierarchical deep reinforcement learning for self-adaptive economic dispatch

It is challenging to accurately model the overall uncertainty of the power system when it is connected to large-scale intermittent generation sources such as wind and photovoltaic generation due to the inherent volatility, uncertainty, and indivisibility of renewable energy. Deep reinforcement learning (DRL) algorithms are introduced as a solution to avoid modeling the complex uncertainties and to adapt the fluctuation of uncertainty by interacting with the environment and using feedback to continuously improve their strategies. However, the large-scale nature and uncertainty of the system lead to the sparse reward problem and high-dimensional space issue in DRL. A hierarchical deep reinforcement learning (HDRL) scheme is designed to decompose the process of solving this problem into two stages, using the reinforcement learning (RL) agent in the global stage and the heuristic algorithm in the local stage to find optimal dispatching decisions for power systems under uncertainty. Simulation studies have shown that the proposed HDRL scheme is efficient in solving power system economic dispatch problems under both deterministic and uncertain scenarios thanks to its adaptation system uncertainty, and coping with the volatility of uncertain factors while significantly improving the speed of online decision-making.

Collective Transport Behavior in a Robotic Swarm with Hierarchical Imitation Learning

Swarm robotics is the study of how a large number of relatively simple physically embodied robots can be designed such that a desired collective behavior emerges from local interactions. Furthermore, reinforcement learning (RL) is a promising approach for training robotic swarm controllers. However, the conventional RL approach suffers from the sparse reward problem in some complex tasks, such as key-to-door tasks. In this study, we applied hierarchical imitation learning to train a robotic swarm to address a key-to-door transport task with sparse rewards. The results demonstrate that the proposed approach outperforms the conventional RL method. Moreover, the proposed method outperforms the conventional hierarchical RL method in its ability to adapt to changes in the training environment.

Security Development Lifecycle-Based Adaptive Reward Mechanism for Reinforcement Learning in Continuous Integration Testing Optimization

Continuous automated testing throughout each cycle can ensure the security of the continuous integration (CI) development lifecycle. Test case prioritization (TCP) is a critical factor in optimizing automated testing, which prioritizes potentially failed test cases and improves the efficiency of automated testing. In CI automated testing, the TCP is a continuous decision-making process that can be solved with reinforcement learning (RL). RL-based CITCP can continuously generate a TCP strategy for each CI development lifecycle, with the reward mechanism as the core. The reward mechanism consists of the reward function and the reward strategy. However, there are new challenges to RL-based CITCP in real-industry CI testing. With high-frequency iteration, the reward function is often calculated with a fixed length of historical information, ignoring the spatial characteristics of the current cycle. Therefore, the dynamic time window (DTW)-based reward function is proposed to perform the reward calculation, which adaptively adjusts the recent historical information range based on the integration cycle. Moreover, with low-failure testing, the reward strategy usually only rewards failure test cases, which creates a sparse reward problem in RL. To address this issue, the similarity-based reward strategy is proposed, which increases the reward objects of some passed test cases, similar to the failure test cases. The DTW-based reward function and the similarity-based reward strategy together constitute the proposed adaptive reward mechanism in RL-based CITCP. To validate the effectiveness of the adaptive reward mechanism, experimental verification is carried out on 13 industrial data sets. The experimental results show that the adaptive reward mechanism can improve the TCP effect, where the average NAPFD is maximally improved by 7.29%, the average Recall is maximally improved by 6.04% and the average TTF is improved by 6.81 positions with a maximum of 63.77.

Learning with sparse reward in a gap junction network inspired by the insect mushroom body.

Animals can learn in real-life scenarios where rewards are often only available when a goal is achieved. This 'distal' or 'sparse' reward problem remains a challenge for conventional reinforcement learning algorithms. Here we investigate an algorithm for learning in such scenarios, inspired by the possibility that axo-axonal gap junction connections, observed in neural circuits with parallel fibres such as the insect mushroom body, could form a resistive network. In such a network, an active node represents the task state, connections between nodes represent state transitions and their connection to actions, and current flow to a target state can guide decision making. Building on evidence that gap junction weights are adaptive, we propose that experience of a task can modulate the connections to form a graph encoding the task structure. We demonstrate that the approach can be used for efficient reinforcement learning under sparse rewards, and discuss whether it is plausible as an account of the insect mushroom body.

An indoor blind area-oriented autonomous robotic path planning approach using deep reinforcement learning

Deep reinforcement learning (DRL) provides a new solution for autonomous robotic path planning in a known indoor environment. Previous studies mainly focused on robot path optimization but ignored blind areas in the indoor exploration, naturally result in low coverage rate and low exploration efficiency. The blind areas exploration is a crucial issue of the indoor environment. This work proposes an indoor blind area-oriented autonomous robotic path planning approach using DRL methods. First, the method optimization is based on a double deep Q-network (DDQN) with prioritized experience replay (PER). Then the Blocking and Blind Angle mechanism (BBA) is proposed to explore blind areas, assisted in selecting the optimal exploration points of next moment. Meanwhile it solves the common sparse reward problem in DRL. Finally, the presented method is successfully applied in simulation environment using a cleaning robot. Experiments show that the proposed BBA-PER-DDQN not only explores the blind areas, but also accelerates the convergence speed. The results show that the training time is reduced from more than one hour to 36 minutes, and the coverage rate is increased by 11.37% higher than that of the baseline algorithms.

Optimal bipartite graph matching-based goal selection for policy-based hindsight learning

The sparse reward problem stands as a significant challenge in the field of reinforcement learning. Hindsight Experience Replay (HER) addresses this by goal relabeling, allowing the agent to learn from unsuccessful experiences. Some studies combine policy gradient methods with HER, resulting in policy-based hindsight learning algorithms. However, Policy-based hindsight learning involves the use of importance sampling, where the distribution of hindsight goals and the distribution of desired goals contribute to the computation of importance weights. When there is a significant difference between the two distributions, importance weights may become skewed, thereby impacting the evaluation of the policy and leading to suboptimal policies. To address this, we propose modeling the goal selection as an optimization problem for distribution matching. After we augment the original desired goals using Kernel Density Estimation (KDE), we further convert the optimization problem for distribution matching into a bipartite graph matching problem that minimizes the sum of weights. Our optimal bipartite graph matching-based hindsight goal selection method can select hindsight goals that are the most closely aligned with the original goals. Experimental results show that algorithms combined with the optimal bipartite graph matching-based hindsight goal selection outperform the original algorithms. Visualizations also demonstrate the superiority of our method in selecting hindsight goals.

Support for redistribution is shaped by motives of egalitarian division and coercive redistribution

The three-player evolutionary model of support for redistribution is compatible with a fairness motive; however, existing research has found near-zero effects of fairness. Here we propose an egalitarian division fairness motive, solving the problem of reward for collaboration and impacting support for redistribution. Study 1 (N = 403) showed egalitarian division fairness had additional predictive power predicting support for redistribution (β = 0.14), as well as discriminant validity from self-interest, compassion, and envy. Robustness was supported by a replication (N = 402), yielding a significant and larger effect size (β = 0.25) of egalitarian division with support for redistribution. We also examined support for coercive redistribution. In both studies, willingness to use coercive redistribution was predicted by egalitarian division fairness (S1 β = 0.15, S2: β = 0.31) and, independently, by instrumental harm (S1 β = 0.21, S2: β = 0.16). These motives expand the three-player model to include fairness and coercive enforcement, and suggest applications of evolution in developing better political, economic, and ethical knowledge. Evolved motives accounted for ~45 % of support for redistribution.

Inspection Robot Navigation Based on Improved TD3 Algorithm.

The swift advancements in robotics have rendered navigation an essential task for mobile robots. While map-based navigation methods depend on global environmental maps for decision-making, their efficacy in unfamiliar or dynamic settings falls short. Current deep reinforcement learning navigation strategies can navigate successfully without pre-existing map data, yet they grapple with issues like inefficient training, slow convergence, and infrequent rewards. To tackle these challenges, this study introduces an improved two-delay depth deterministic policy gradient algorithm (LP-TD3) for local planning navigation. Initially, the integration of the long-short-term memory (LSTM) module with the Prioritized Experience Re-play (PER) mechanism into the existing TD3 framework was performed to optimize training and improve the efficiency of experience data utilization. Furthermore, the incorporation of an Intrinsic Curiosity Module (ICM) merges intrinsic with extrinsic rewards to tackle sparse reward problems and enhance exploratory behavior. Experimental evaluations using ROS and Gazebo simulators demonstrate that the proposed method outperforms the original on various performance metrics.

Research on game strategy of spacecraft chase and escape based on adaptive augmented random search

To solve the problem of the survival differential policy interception between a spacecraft and a non-cooperative target pursuit game, the pursuit game policy is studied based on reinforcement learning, and the adaptive-augmented random search algorithm is proposed. Firstly, to solve the sparse reward problem of sequential decision making, an exploration method based on the spatial perturbation of parameters of the policy is designed, thus accelerating its convergence speed. Secondly, to avoid the possibility of falling into local optimum prematurely, a novelty degree function is designed to guide the policy update, enhancing the efficiency of data utilization. Finally, the effectiveness and advancement of the exploration method are verified with numerical simulations and compared with those of the augmented random search algorithm, the proximal policy optimization algorithm and the deep deterministic policy gradient algorithm.

Autonomous Driving of Mobile Robots in Dynamic Environments Based on Deep Deterministic Policy Gradient: Reward Shaping and Hindsight Experience Replay.

In this paper, we propose a reinforcement learning-based end-to-end learning method for the autonomous driving of a mobile robot in a dynamic environment with obstacles. Applying two additional techniques for reinforcement learning simultaneously helps the mobile robot in finding an optimal policy to reach the destination without collisions. First, the multifunctional reward-shaping technique guides the agent toward the goal by utilizing information about the destination and obstacles. Next, employing the hindsight experience replay technique to address the experience imbalance caused by the sparse reward problem assists the agent in finding the optimal policy. We validated the proposed technique in both simulation and real-world environments. To assess the effectiveness of the proposed method, we compared experiments for five different cases.

Boosting Reinforcement Learning via Hierarchical Game Playing With State Relay.

Due to its wide application, deep reinforcement learning (DRL) has been extensively studied in the motion planning community in recent years. However, in the current DRL research, regardless of task completion, the state information of the agent will be reset afterward. This leads to a low sample utilization rate and hinders further explorations of the environment. Moreover, in the initial training stage, the agent has a weak learning ability in general, which affects the training efficiency in complex tasks. In this study, a new hierarchical reinforcement learning (HRL) framework dubbed hierarchical learning based on game playing with state relay (HGR) is proposed. In particular, we introduce an auxiliary penalty to regulate task difficulty, and one training mechanism, the state relay mechanism, is designed. The relay mechanism can make full use of the intermediate states of the agent and expand the environment exploration of low-level policy. Our algorithm can improve the sample utilization rate, reduce the sparse reward problem, and thereby enhance the training performance in complex environments. Simulation tests are carried out on two public experiment platforms, i.e., MazeBase and MuJoCo, to verify the effectiveness of the proposed method. The results show that HGR significantly benefits the reinforcement learning (RL) area.

Game on! A state-of-the-art overview of doing business with gamification

Gamification is the act of applying game-design elements to transform activities, products, services, and systems in a way that provides the kind of experiences similar to those offered by games. These elements include badges, points, and leaderboards to motivate and reward problem solving activities and processes. Both businesses and business researchers are increasingly interested in the application of gamification. In this regard, this review aims to offer a state-of-the-art overview of gamification in business, revealing its current trends and future directions. Using business research on gamification published in the last decade (2012–2022) found on Scopus and Web of Science, we conduct a performance analysis to illuminate the field's performance (publication productivity and impact) alongside its key contributors (journals, authors, and countries) as well as a science mapping using a co-citation analysis to locate knowledge foundations, and bibliographic coupling and keyword co-occurrence analysis to reveal major themes in business gamification. Our exploration has unearthed gamification's pervasive influence across business domains, from enhancing learning and fostering innovation to empowering marketing strategies and catalyzing transformation. Gamification has also made significant inroads into management practices, driving engagement, influencing behavioral outcomes, and ushering sustainability. Peering into the future, our density-centrality quadrant analysis casts light on emerging frontiers. Noteworthily, we expect the next wave of gamification to be shaped by the allure of video games and the metaverse, bespoke industry-specific designs, confluence with e-commerce trends, emphasis on sustainable development, expanded social responsibilities, innovations in virtual reality, intersections with human urbanism, and the rise of smart cities. As gamification continues to integrate and reshape myriad facets of the business world, it holds the promise of not only enriching current practices but also charting the course for the future of innovative, sustainable, and transformative business strategies.

Progressive reinforcement learning for video summarization

Video summarization addresses generating video summaries to help watchers grasp the content of a video without watching it entirely. Many methods have engaged in automatic video summarization. Although these methods have performed well, they still suffer from limited training data and sparse reward problems. We propose a Progressive Reinforcement Learning Video Summarization structure (PRLVS) with an unsupervised reward. The reward measures the information and quality the selected frames convey without annotations. Striving to earn higher rewards, our PRLVS adopts a “T”-type human thinking paradigm: choosing some key frames and checking if their adjacent frames are better than them. To simulate this paradigm, we decompose the flat strategy into a hierarchical strategy consisting of a horizontal policy and a vertical policy. These two policies are optimized alternatively, which densifies the reward while reducing the exploration space. Their cooperation also makes the agent capture the context information of the whole video at every step. Extensive experimental results on two benchmark databases (i.e., SumMe, TVSum) show that our PRLVS outperforms the comparisons and approaches the supervised methods, which indicates that it is significant to integrate our unsupervised reward into the progressive reinforcement learning structure to address limited annotation and sparse reward problems.

A Real-Time and Optimal Hypersonic Entry Guidance Method Using Inverse Reinforcement Learning

The mission of hypersonic vehicles faces the problem of highly nonlinear dynamics and complex environments, which presents challenges to the intelligent level and real-time performance of onboard guidance algorithms. In this paper, inverse reinforcement learning is used to address the hypersonic entry guidance problem. The state-control sample pairs and state-rewards sample pairs obtained by interacting with hypersonic entry dynamics are used to train the neural network by applying the distributed proximal policy optimization method. To overcome the sparse reward problem in the hypersonic entry problem, a novel reward function combined with a sophisticated discriminator network is designed to generate dense optimal rewards continuously, which is the main contribution of this paper. The optimized guidance methodology can achieve good terminal accuracy and high success rates with a small number of trajectories as datasets while satisfying heating rate, overload, and dynamic pressure constraints. The proposed guidance method is employed for two typical hypersonic entry vehicles (Common Aero Vehicle-Hypersonic and Reusable Launch Vehicle) to demonstrate the feasibility and potential. Numerical simulation results validate the real-time performance and optimality of the proposed method and indicate its suitability for onboard applications in the hypersonic entry flight.

Multi-Agent Collaborative Target Search Based on the Multi-Agent Deep Deterministic Policy Gradient with Emotional Intrinsic Motivation

Multi-agent collaborative target search is one of the main challenges in the multi-agent field, and deep reinforcement learning (DRL) is a good way to learn such a task. However, DRL always faces the problem of sparse reward, which to some extent reduces its efficiency in task learning. Introducing intrinsic motivation has proved to be a useful way to make the sparse reward in DRL. So, based on the multi-agent deep deterministic policy gradient (MADDPG) structure, a new MADDPG algorithm with the emotional intrinsic motivation name MADDPG-E is proposed in this paper for the multi-agent collaborative target search. In MADDPG-E, a new emotional intrinsic motivation module with three emotions, joy, sadness, and fear, is designed. The three emotions are defined by corresponding psychological knowledge to the multi-agent embodied situations in an environment. An emotional steady-state variable function H is then designed to help judge the goodness of the emotions. Based on H, an emotion-based intrinsic reward function is finally proposed. With the designed emotional intrinsic motivation module, the multi-agent system always tries to make itself joy, which means it always learns to search the target. To show the effectiveness of the proposed MADDPG-E algorithm, two kinds of simulation experiments with a determined initial position and random initial position, respectively, are carried out, and comparisons are performed with MADDPG as well as MADDPG-ICM (MADDPG with an intrinsic curiosity module). The results show that with the designed emotional intrinsic motivation module, MADDPG-E has a higher learning speed and better learning stability, and the advantage is more obvious when facing complex situations.

End-to-End Autonomous Navigation Based on Deep Reinforcement Learning with a Survival Penalty Function.

An end-to-end approach to autonomous navigation that is based on deep reinforcement learning (DRL) with a survival penalty function is proposed in this paper. Two actor-critic (AC) frameworks, namely, deep deterministic policy gradient (DDPG) and twin-delayed DDPG (TD3), are employed to enable a nonholonomic wheeled mobile robot (WMR) to perform navigation in dynamic environments containing obstacles and for which no maps are available. A comprehensive reward based on the survival penalty function is introduced; this approach effectively solves the sparse reward problem and enables the WMR to move toward its target. Consecutive episodes are connected to increase the cumulative penalty for scenarios involving obstacles; this method prevents training failure and enables the WMR to plan a collision-free path. Simulations are conducted for four scenarios-movement in an obstacle-free space, in a parking lot, at an intersection without and with a central obstacle, and in a multiple obstacle space-to demonstrate the efficiency and operational safety of our method. For the same navigation environment, compared with the DDPG algorithm, the TD3 algorithm exhibits faster numerical convergence and higher stability in the training phase, as well as a higher task execution success rate in the evaluation phase.

SafeCrowdNav: safety evaluation of robot crowd navigation in complex scenes.

Navigating safely and efficiently in dense crowds remains a challenging problem for mobile robots. The interaction mechanisms involved in collision avoidance require robots to exhibit active and foresighted behaviors while understanding the crowd dynamics. Deep reinforcement learning methods have shown superior performance compared to model-based approaches. However, existing methods lack an intuitive and quantitative safety evaluation for agents, and they may potentially trap agents in local optima during training, hindering their ability to learn optimal strategies. In addition, sparse reward problems further compound these limitations. To address these challenges, we propose SafeCrowdNav, a comprehensive crowd navigation algorithm that emphasizes obstacle avoidance in complex environments. Our approach incorporates a safety evaluation function to quantitatively assess the current safety score and an intrinsic exploration reward to balance exploration and exploitation based on scene constraints. By combining prioritized experience replay and hindsight experience replay techniques, our model effectively learns the optimal navigation policy in crowded environments. Experimental outcomes reveal that our approach enables robots to improve crowd comprehension during navigation, resulting in reduced collision probabilities and shorter navigation times compared to state-of-the-art algorithms. Our code is available at https://github.com/Janet-xujing-1216/SafeCrowdNav.