Robot Reinforcement Learning Research Articles

Design and analysis of parallel quantum transfer fractal priority replay with dynamic memory algorithm in quantum reinforcement learning for robotics

Design and analysis of parallel quantum transfer fractal priority replay with dynamic memory algorithm in quantum reinforcement learning for robotics

GeneWorker: An end-to-end robotic reinforcement learning approach with collaborative generator and worker networks

Reinforcement learning aided by the skill conception exhibits potent capabilities in guiding autonomous agents toward acquiring meaningful behaviors. However, in the current landscape of reinforcement learning, a skill is often merely a rudimentary abstraction of a sequence of primitive actions, serving as a component of the input to policy networks with fixed network parameters. This rigid methodology presents obstacles when attempting to integrate with burgeoning techniques such as meta-learning and large language models. To address this issue, we introduce a unique neural skill representation that abstracts the activation of neurons in each neural layer. Based on this, a novel end-to-end robotic reinforcement learning algorithm is proposed, in which two sub-networks, i.e., generator and worker networks, implement collaborative inferences via neural skills. Specifically, the generator produces a series of multi-spatial neural skills, providing efficient guidance for subsequent decision-making; by integrating these skills, the worker can determine its own network weights and biases to cope with various environmental conditions. Therefore, actions can be sampled with flexibly changeable network parameters through the collaboration between generator and worker networks. The experiments demonstrate that GeneWorker can achieve a mean success rate of over 90.67% on continuous robotic tasks and outperforms previous state-of-the-art methods by a minimum of 54% on the pick-and-place task.

Guiding real-world reinforcement learning for in-contact manipulation tasks with Shared Control Templates

The requirement for a high number of training episodes has been a major limiting factor for the application of Reinforcement Learning (RL) in robotics. Learning skills directly on real robots requires time, causes wear and tear and can lead to damage to the robot and environment due to unsafe exploratory actions. The success of learning skills in simulation and transferring them to real robots has also been limited by the gap between reality and simulation. This is particularly problematic for tasks involving contact with the environment as contact dynamics are hard to model and simulate. In this paper we propose a framework which leverages a shared control framework for modeling known constraints defined by object interactions and task geometry to reduce the state and action spaces and hence the overall dimensionality of the reinforcement learning problem. The unknown task knowledge and actions are learned by a reinforcement learning agent by conducting exploration in the constrained environment. Using a pouring task and grid-clamp placement task (similar to peg-in-hole) as use cases and a 7-DoF arm, we show that our approach can be used to learn directly on the real robot. The pouring task is learned in only 65 episodes (16 min) and the grid-clamp placement task is learned in 75 episodes (17 min) with strong safety guarantees and simple reward functions, greatly alleviating the need for simulation.

EXPLORING THE FUSION OF INTELLIGENT AGENTS AND UNITY ML: A COMPREHENSIVE ANALYSIS OF ROBOT MOTION SIMULATION

In this article usage of procedural generation for reinforcement learning is discussed. Procedural generation is a technique in computer graphics and game design that creates content algorithmically, often generating terrain, levels, or assets dynamically, rather than relying on pre-made, static data. Thus, in the context of reinforcement learning for robots, procedural generation can be used to create dynamic and diverse training environments, allowing robots to learn and adapt to a wide range of scenarios through trial and error. Making it possible to train robots in a virtual sophisticated environment resembling what is present in the world. Keywords: Reinforcement Learning (RL), Procedural Generation (PG), Unity3D, Environment, Robotics.

Evaluating Reliability and Economics of EV Charging Configurations and Deep Reinforcement Learning in Robotics and Autonomy

Growing EV popularity drives companies to focus on reliable charging station designs despite challenges in maintaining reliability. A proposed 36-ported design combines uniform and non-uniform port arrangements, tested with 50-350 kW systems. Failure rates are estimated using MILHDBK217F and MILHBK-338B standards, assessing port reliability and station success rates through binomial distribution and cost analysis. This design improves voltage stability and reduces maintenance costs through enhanced port reliability. In robotics and autonomous systems, Deep Reinforcement Learning (DRL) excels but faces challenges from unsafe policies leading to hazardous decisions. This study introduces a reliability assessment framework for DRL-controlled systems, using formal neural network analysis. A two-level verification approach evaluates safety locally using reachability tools and globally by aggregating local safety metrics across tasks. Experimental validation confirms the framework's effectiveness in enhancing RAS safety.

A memory and attention-based reinforcement learning for musculoskeletal robots with prior knowledge of muscle synergies

PurposeLimited by the types of sensors, the state information available for musculoskeletal robots with highly redundant, nonlinear muscles is often incomplete, which makes the control face a bottleneck problem. The aim of this paper is to design a method to improve the motion performance of musculoskeletal robots in partially observable scenarios, and to leverage the ontology knowledge to enhance the algorithm’s adaptability to musculoskeletal robots that have undergone changes.Design/methodology/approachA memory and attention-based reinforcement learning method is proposed for musculoskeletal robots with prior knowledge of muscle synergies. First, to deal with partially observed states available to musculoskeletal robots, a memory and attention-based network architecture is proposed for inferring more sufficient and intrinsic states. Second, inspired by muscle synergy hypothesis in neuroscience, prior knowledge of a musculoskeletal robot’s muscle synergies is embedded in network structure and reward shaping.FindingsBased on systematic validation, it is found that the proposed method demonstrates superiority over the traditional twin delayed deep deterministic policy gradients (TD3) algorithm. A musculoskeletal robot with highly redundant, nonlinear muscles is adopted to implement goal-directed tasks. In the case of 21-dimensional states, the learning efficiency and accuracy are significantly improved compared with the traditional TD3 algorithm; in the case of 13-dimensional states without velocities and information from the end effector, the traditional TD3 is unable to complete the reaching tasks, while the proposed method breaks through this bottleneck problem.Originality/valueIn this paper, a novel memory and attention-based reinforcement learning method with prior knowledge of muscle synergies is proposed for musculoskeletal robots to deal with partially observable scenarios. Compared with the existing methods, the proposed method effectively improves the performance. Furthermore, this paper promotes the fusion of neuroscience and robotics.

Deep reinforcement learning in mobile robotics – a concise review

Deep reinforcement learning in mobile robotics – a concise review

Continual Reinforcement Learning for Quadruped Robot Locomotion.

The ability to learn continuously is crucial for a robot to achieve a high level of intelligence and autonomy. In this paper, we consider continual reinforcement learning (RL) for quadruped robots, which includes the ability to continuously learn sub-sequential tasks (plasticity) and maintain performance on previous tasks (stability). The policy obtained by the proposed method enables robots to learn multiple tasks sequentially, while overcoming both catastrophic forgetting and loss of plasticity. At the same time, it achieves the above goals with as little modification to the original RL learning process as possible. The proposed method uses the Piggyback algorithm to select protected parameters for each task, and reinitializes the unused parameters to increase plasticity. Meanwhile, we encourage the policy network exploring by encouraging the entropy of the soft network of the policy network. Our experiments show that traditional continual learning algorithms cannot perform well on robot locomotion problems, and our algorithm is more stable and less disruptive to the RL training progress. Several robot locomotion experiments validate the effectiveness of our method.

Automatic Curriculum Determination for Deep Reinforcement Learning in Reconfigurable Robots

Automatic Curriculum Determination for Deep Reinforcement Learning in Reconfigurable Robots

Application of fuzzy reinforcement learning in IoT-based robotics for autonomous navigation and collision avoidance

In this study, we investigate the viability of applying fuzzy reinforcement learning (FRL) to Internet of Things-based robots for purposes of autonomous navigation and collision avoidance. The proposed approach utilises FRL, IoT, and a sensor network to give the robot the ability to learn from its environment and act in accordance with those policies. The authors used FRL to train a mobile robot with wheels to move around and avoid obstacles, and then they put the robot through its paces in a virtual world. Results showed that the FRL-based technique improved the robot’s navigation and collision avoidance performance compared to traditional rule-based approaches. The results of this study indicate that FRL may be a viable technique for enabling autonomous navigation and obstacle avoidance in IoT-based robotics.

A Novel Approach for Target Attraction and Obstacle Avoidance of a Mobile Robot in Unknown Environments Using a Customized Spiking Neural Network

In recent years, implementing reinforcement learning in autonomous mobile robots (AMRs) has become challenging. Traditional methods face complex trials, long convergence times, and high computational requirements. This paper introduces an innovative strategy using a customized spiking neural network (SNN) for autonomous learning and control of mobile robots (AMR) in unknown environments. The model combines spike-timing-dependent plasticity (STDP) with dopamine modulation for learning. It utilizes the Izhikevich neuron model, leading to biologically inspired and computationally efficient control systems that adapt to changing environments. The performance of the model is evaluated in a simulated environment, replicating real-world scenarios with obstacles. In the initial training phase, the model faces significant challenges. Integrating brain-inspired learning, dopamine, and the Izhikevich neuron model adds complexity. The model achieves an accuracy rate of 33% in reaching its target during this phase. Collisions with obstacles occur 67% of the time, indicating the struggle of the model to adapt to complex obstacles. However, the model’s performance improves as the study progresses to the testing phase after the robot has learned. Its accuracy surges to 94% when reaching the target, and collisions with obstacles reduce it to 6%. This shift demonstrates the adaptability and problem-solving capabilities of the model in the simulated environment, making it more competent for real-world applications.

A human-centered safe robot reinforcement learning framework with interactive behaviors.

Deployment of Reinforcement Learning (RL) algorithms for robotics applications in the real world requires ensuring the safety of the robot and its environment. Safe Robot RL (SRRL) is a crucial step toward achieving human-robot coexistence. In this paper, we envision a human-centered SRRL framework consisting of three stages: safe exploration, safety value alignment, and safe collaboration. We examine the research gaps in these areas and propose to leverage interactive behaviors for SRRL. Interactive behaviors enable bi-directional information transfer between humans and robots, such as conversational robot ChatGPT. We argue that interactive behaviors need further attention from the SRRL community. We discuss four open challenges related to the robustness, efficiency, transparency, and adaptability of SRRL with interactive behaviors.

A Novel Actor—Critic Motor Reinforcement Learning for Continuum Soft Robots

Reinforcement learning (RL) is explored for motor control of a novel pneumatic-driven soft robot modeled after continuum media with a varying density. This model complies with closed-form Lagrangian dynamics, which fulfills the fundamental structural property of passivity, among others. Then, the question arises of how to synthesize a passivity-based RL model to control the unknown continuum soft robot dynamics to exploit its input–output energy properties advantageously throughout a reward-based neural network controller. Thus, we propose a continuous-time Actor–Critic scheme for tracking tasks of the continuum 3D soft robot subject to Lipschitz disturbances. A reward-based temporal difference leads to learning with a novel discontinuous adaptive mechanism of Critic neural weights. Finally, the reward and integral of the Bellman error approximation reinforce the adaptive mechanism of Actor neural weights. Closed-loop stability is guaranteed in the sense of Lyapunov, which leads to local exponential convergence of tracking errors based on integral sliding modes. Notably, it is assumed that dynamics are unknown, yet the control is continuous and robust. A representative simulation study shows the effectiveness of our proposal for tracking tasks.

Sim2Real Transfer of Reinforcement Learning for Concentric Tube Robots

Sim2Real Transfer of Reinforcement Learning for Concentric Tube Robots

Variational meta reinforcement learning for social robotics

Variational meta reinforcement learning for social robotics

Cross-domain policy adaptation with dynamics alignment

The implementation of robotic reinforcement learning is hampered by problems such as an unspecified reward function and high training costs. Many previous works have used cross-domain policy transfer to obtain the policy of the problem domain. However, these researches require paired and aligned dynamics trajectories or other interactions with the environment. We propose a cross-domain dynamics alignment framework for the problem domain policy acquisition that can transfer the policy trained in the source domain to the problem domain. Our framework aims to learn dynamics alignment across two domains that differ in agents’ physical parameters (armature, rotation range, or torso mass) or agents’ morphologies (limbs). Most importantly, we learn dynamics alignment between two domains using unpaired and unaligned dynamics trajectories. For these two scenarios, we propose a cross-physics-domain policy adaptation algorithm (CPD) and a cross-morphology-domain policy adaptation algorithm (CMD) based on our cross-domain dynamics alignment framework. In order to improve the performance of policy in the source domain so that a better policy can be transferred to the problem domain, we propose the Boltzmann TD3 (BTD3) algorithm. We conduct diverse experiments on agent continuous control domains to demonstrate the performance of our approaches. Experimental results show that our approaches can obtain better policies and higher rewards for the agents in the problem domains even when the dataset of the problem domain is small.

Deep reinforcement learning for cooperative robots based on adaptive sentiment feedback

Human–robot cooperative tasks have gained importance with the emergence of robotics and artificial intelligence technology. In interactive reinforcement learning techniques, robots learn target tasks by receiving feedback from an experienced human trainer. However, most interactive reinforcement learning studies require a separate process to integrate the trainer’s feedback into the training dataset, making it challenging for robots to learn new tasks from humans in real-time. Furthermore, the types of feedback sentences that trainers can use are limited in previous research. To address these limitations, this paper proposes a robot teaching strategy that uses deep RL via human–robot interaction to learn table balancing tasks interactively. The proposed system employs Deep Q-Network with real-time sentiment feedback delivered through the trainer’s speech to learn cooperative tasks. We designed a novel reward function that incorporates sentiment feedback from human speech in real-time during the learning process. The paper presents an improved reward shaping technique based on subdivided feedback levels and shrinking feedback. This function serves as a guide for the robot to engage in natural interactions with humans and enables it to learn the tasks effectively. Experimental results demonstrate that the proposed interactive deep reinforcement learning model achieved a high success rate of up to 99.06%, outperforming the model without sentiment feedback.

Composable energy policies for reactive motion generation and reinforcement learning

In this work, we introduce composable energy policies (CEP), a novel framework for multi-objective motion generation. We frame the problem of composing multiple policy components from a probabilistic view. We consider a set of stochastic policies represented in arbitrary task spaces, where each policy represents a distribution of the actions to solve a particular task. Then, we aim to find the action in the configuration space that optimally satisfies all the policy components. The presented framework allows the fusion of motion generators from different sources: optimal control, data-driven policies, motion planning, and handcrafted policies. Classically, the problem of multi-objective motion generation is solved by the composition of a set of deterministic policies, rather than stochastic policies. However, there are common situations where different policy components have conflicting behaviors, leading to oscillations or the robot getting stuck in an undesirable state. While our approach is not directly able to solve the conflicting policies problem, we claim that modeling each policy as a stochastic policy allows more expressive representations for each component in contrast with the classical reactive motion generation approaches. In some tasks, such as reaching a target in a cluttered environment, we show experimentally that CEP additional expressivity allows us to model policies that reduce these conflicting behaviors. A field that benefits from these reactive motion generators is the one of robot reinforcement learning. Integrating these policy architectures with reinforcement learning allows us to include a set of inductive biases in the learning problem. These inductive biases guide the reinforcement learning agent towards informative regions or improve collision safety while exploring. In our work, we show how to integrate our proposed reactive motion generator as a structured policy for reinforcement learning. Combining the reinforcement learning agent exploration with the prior-based CEP, we can improve the learning performance and explore safer.

Human-aligned reinforcement learning for autonomous agents and robots

Human-aligned reinforcement learning for autonomous agents and robots

A review on reinforcement learning for contact-rich robotic manipulation tasks

Research and application of reinforcement learning in robotics for contact-rich manipulation tasks have exploded in recent years. Its ability to cope with unstructured environments and accomplish hard-to-engineer behaviors has led reinforcement learning agents to be increasingly applied in real-life scenarios. However, there is still a long way ahead for reinforcement learning to become a core element in industrial applications. This paper examines the landscape of reinforcement learning and reviews advances in its application in contact-rich tasks from 2017 to the present. The analysis investigates the main research for the most commonly selected tasks for testing reinforcement learning algorithms in both rigid and deformable object manipulation. Additionally, the trends around reinforcement learning associated with serial manipulators are explored as well as the various technological challenges that this machine learning control technique currently presents. Lastly, based on the state-of-the-art and the commonalities among the studies, a framework relating the main concepts of reinforcement learning in contact-rich manipulation tasks is proposed. The final goal of this review is to support the robotics community in future development of systems commanded by reinforcement learning, discuss the main challenges of this technology and suggest future research directions in the domain.