Reinforcement Learning Process Research Articles

AutoRL X: Automated Reinforcement Learning on the Web

Reinforcement Learning (RL) is crucial in decision optimization, but its inherent complexity often presents challenges in interpretation and communication. Building upon AutoDOViz — an interface that pushed the boundaries of Automated RL for Decision Optimization — this paper unveils an open-source expansion with a web-based platform for RL. Our work introduces a taxonomy of RL visualizations and launches a dynamic web platform, leveraging backend flexibility for AutoRL frameworks like ARLO and Svelte.js for a smooth interactive user experience in the front end. Since AutoDOViz is not open-source, we present AutoRL X, a new interface designed to visualize RL processes. AutoRL X is shaped by the extensive user feedback and expert interviews from AutoDOViz studies, and it brings forth an intelligent interface with real-time, intuitive visualization capabilities that enhance understanding, collaborative efforts, and personalization of RL agents. Addressing the gap in accurately representing complex real-world challenges within standard RL environments, we demonstrate our tool's application in healthcare, explicitly optimizing brain stimulation trajectories. A user study contrasts the performance of human users optimizing electric fields via a 2D interface with RL agents’ behavior that we visually analyze in AutoRL X, assessing the practicality of automated RL. All our data and code is openly available at: https://github.com/lorifranke/autorlx .

Local connection reinforcement learning method for efficient robotic peg-in-hole assembly

Traditional robotic peg-in-hole assembly methods rely on complex contact state analysis. Reinforcement learning (RL) is gradually becoming a preferred method of controlling robotic peg-in-hole assembly tasks. However, the training process of RL is quite time-consuming because RL methods are always globally connected, which means all state components are assumed to be the input of policies for all action components, thus increasing action space and state space to be explored. In this paper, we first define continuous space serialized Shapley value (CS3) and construct a connection graph to clarify the correlativity of action components on state components. Then we propose a local connection reinforcement learning (LCRL) method based on the connection graph, which eliminates the influence of irrelevant state components on the selection of action components. The simulation and experiment results demonstrate that the LCRL method can achieve the same average reward as the traditional RL method in only 49% of episodes. In the final episode, the LCRL method's final reward is 35% higher than that of the traditional RL method, which guarantees the rapidity and stability of the assembly process.

Eliminating Primacy Bias in Online Reinforcement Learning by Self-Distillation.

Excessive invalid explorations at the beginning of training lead deep reinforcement learning process to fall into the risk of overfitting, further resulting in spurious decisions, which obstruct agents in the following states and explorations. This phenomenon is termed primacy bias in online reinforcement learning. This work systematically investigates the primacy bias in online reinforcement learning, discussing the reason for primacy bias, while the characteristic of primacy bias is also analyzed. Besides, to learn a policy generalized to the following states and explorations, we develop an online reinforcement learning framework, termed self-distillation reinforcement learning (SDRL), based on knowledge distillation, allowing the agent to transfer the learned knowledge into a randomly initialized policy at regular intervals, and the new policy network is used to replace the original one in the following training. The core idea for this work is distilling knowledge from the trained policy to another policy can filter biases out, generating a more generalized policy in the learning process. Moreover, to avoid the overfitting of the new policy due to excessive distillations, we add an additional loss in the knowledge distillation process, using L2 regularization to improve the generalization, and the self-imitation mechanism is introduced to accelerate the learning on the current experiences. The results of several experiments in DMC and Atari 100k suggest the proposal has the ability to eliminate primacy bias for reinforcement learning methods, and the policy after knowledge distillation can urge agents to get higher scores more quickly.

Adaptive reinforcement learning-based control using proximal policy optimization and slime mould algorithm with experimental tower crane system validation

This paper presents a novel optimal reference tracking control approach resulted from the combination of a popular policy gradient Reinforcement Learning (RL) algorithm, namely Proximal Policy Optimization (PPO), and a metaheuristic Slime Mould Algorithm (SMA). One of the most important parameters in the PPO-based RL process is the learning rate, which has a big impact on how the parameters of the actor neural network (NN) are iteratively updated. In every episode of the RL process, the weights and the biases of the actor NN are multiplied with the learning rate, determining how much the learning agent will step into a certain direction computed based on previous experiences. The classical PPO algorithm usually relies on fixed values for the learning rates which rarely change, or not at all, during the learning process. However, its main drawback is that the learning agent cannot take advantage of positive momentum in the learning process by accelerating towards good learning experiences or slow down and quickly change the direction in the case of consecutive negative learning experiences. The main objective of the combination proposed in this paper is to create an adaptive SMA-based PPO approach applied to control systems, which instead of using fixed learning rate values, it uses the SMA to compute optimal values of the learning rates in each time step of the learning process based on the progress of the learning agent. This paper investigates if the adaptive SMA-based PPO control approach can be considered as an alternative to the classical PPO version, which employs fixed values of the learning rate. A comparison is carried out using control system performance indices gathered while performing an optimal reference tracking control task on tower crane system laboratory equipment.

Software Reliability Model Estimation for an Indeterministic Crime Cluster through Reinforcement Learning

The software reliability model estimates the probability of data failure in a specific environment, significantly impacting reliability and trustworthiness. The paper study focuses on cluster crime data, i.e., indeterministic in Neutrosophic Logic, using a software reliability model. The study utilizes reinforcement learning, Neutrosophic logic, and non-homogeneous Poisson process crime data to estimate indeterministic cluster data in crime. The "Non-homogeneous Poisson Process with Neutrosophic Logic" technique performs well in evaluating and deterring crime based on crime data analysis. The crime cluster involving offenders correctly classified as failure to accomplish does better than uncertain cluster reliability estimation with least squares and logistic regression analysis. The method enables crime prediction and prevention by using concave growth models to create an uncertain crime cluster, penalizing the correct person.

Fully Data-Driven Robust Output Formation Tracking Control for Heterogeneous Multiagent System With Multiple Leaders and Actuator Faults.

This article investigates a fully data-driven method to solve the robust output formation tracking control problem for the multiagent system (MAS) under actuator faults. The outputs of the followers are controlled to track those of multiple leaders with respect to a convex point while achieving an expected time-varying formation. To obviate the requirement of various system prior knowledge in typical MAS control, a hierarchical frame is developed with three learning and control stages using the online measured data. First, a distributed adaptive observer is designed to coordinate the state convex of multiple leaders while estimating unknown dynamics. The adaptive mechanism relaxes the demand for global topology. Second, by collecting and reusing the online system data, an off-policy reinforcement learning (RL) method is proposed in a continuous form to acquire nominal feedback gains from partial observations of the followers. Essential system models are learned along with the RL process, while solutions to the output regulation equations are implicitly obtained. Third, a comprehensive robust controller is further presented based on the previous learning results. To address the actuator faults with efficiency loss and bias, the adaptive neural networks and robust compensations are utilized in a model-free manner. The output formation tracking is achieved under a derived feasibility condition while stabilities of the learning and control methods are analyzed. Finally, simulation results demonstrate the validity of this fully data-driven control frame.

An online hyper‐volume action bounding approach for accelerating the process of deep reinforcement learning from multiple controllers

AbstractThis paper fuses ideas from reinforcement learning (RL), Learning from Demonstration (LfD), and Ensemble Learning into a single paradigm. Knowledge from a mixture of control algorithms (experts) are used to constrain the action space of the agent, enabling faster RL refining of a control policy, by avoiding unnecessary explorative actions. Domain‐specific knowledge of each expert is exploited. However, the resulting policy is robust against errors of individual experts, since it is refined by a RL reward function without copying any particular demonstration. Our method has the potential to supplement existing RLfD methods when multiple algorithmic approaches are available to function as experts, specifically in tasks involving continuous action spaces. We illustrate our method in the context of a visual servoing (VS) task, in which a 7‐DoF robot arm is controlled to maintain a desired pose relative to a target object. We explore four methods for bounding the actions of the RL agent during training. These methods include using a hypercube and convex hull with modified loss functions, ignoring actions outside the convex hull, and projecting actions onto the convex hull. We compare the training progress of each method using expert demonstrators, employing one expert demonstrator with the DAgger algorithm, and without using any demonstrators. Our experiments show that using the convex hull with a modified loss function not only accelerates learning but also provides the most optimal solution compared with other approaches. Furthermore, we demonstrate faster VS error convergence while maintaining higher manipulability of the arm, compared with classical image‐based VS, position‐based VS, and hybrid‐decoupled VS.

The selective serotonin reuptake inhibitor sertraline alters learning from aversive reinforcements in patients with depression: evidence from a randomized controlled trial.

Selective serotonin reuptake inhibitors (SSRIs) are first-line pharmacological treatments for depression and anxiety. However, little is known about how pharmacological action is related to cognitive and affective processes. Here, we examine whether specific reinforcement learning processes mediate the treatment effects of SSRIs. The PANDA trial was a multicentre, double-blind, randomized clinical trial in UK primary care comparing the SSRI sertraline with placebo for depression and anxiety. Participants (N = 655) performed an affective Go/NoGo task three times during the trial and computational models were used to infer reinforcement learning processes. There was poor task performance: only 54% of the task runs were informative, with more informative task runs in the placebo than in the active group. There was no evidence for the preregistered hypothesis that Pavlovian inhibition was affected by sertraline. Exploratory analyses revealed that in the sertraline group, early increases in Pavlovian inhibition were associated with improvements in depression after 12 weeks. Furthermore, sertraline increased how fast participants learned from losses and faster learning from losses was associated with more severe generalized anxiety symptoms. The study findings indicate a relationship between aversive reinforcement learning mechanisms and aspects of depression, anxiety, and SSRI treatment, but these relationships did not align with the initial hypotheses. Poor task performance limits the interpretability and likely generalizability of the findings, and highlights the critical importance of developing acceptable and reliable tasks for use in clinical studies. This article presents research supported by NIHR Program Grants for Applied Research (RP-PG-0610-10048), the NIHR BRC, and UCL, with additional support from IMPRS COMP2PSYCH (JM, QH) and a Wellcome Trust grant (QH).

5-HT 2A and 5-HT 2C receptor antagonism differentially modulate reinforcement learning and cognitive flexibility: behavioural and computational evidence

RationaleCognitive flexibility, the ability to adapt behaviour in response to a changing environment, is disrupted in several neuropsychiatric disorders, including obsessive–compulsive disorder and major depressive disorder. Evidence suggests that flexibility, which can be operationalised using reversal learning tasks, is modulated by serotonergic transmission. However, how exactly flexible behaviour and associated reinforcement learning (RL) processes are modulated by 5-HT action on specific receptors is unknown.ObjectivesWe investigated the effects of 5-HT2A receptor (5-HT2AR) and 5-HT2C receptor (5-HT2CR) antagonism on flexibility and underlying RL mechanisms.MethodsThirty-six male Lister hooded rats were trained on a touchscreen visual discrimination and reversal task. We evaluated the effects of systemic treatments with the 5-HT2AR and 5-HT2CR antagonists M100907 and SB-242084, respectively, on reversal learning and performance on probe trials where correct and incorrect stimuli were presented with a third, probabilistically rewarded, stimulus. Computational models were fitted to task choice data to extract RL parameters, including a novel model designed specifically for this task.Results5-HT2AR antagonism impaired reversal learning only after an initial perseverative phase, during a period of random choice and then new learning. 5-HT2CR antagonism, on the other hand, impaired learning from positive feedback. RL models further differentiated these effects. 5-HT2AR antagonism decreased punishment learning rate (i.e. negative feedback) at high and low doses. The low dose also decreased reinforcement sensitivity (beta) and increased stimulus and side stickiness (i.e., the tendency to repeat a choice regardless of outcome). 5-HT2CR antagonism also decreased beta, but reduced side stickiness.ConclusionsThese data indicate that 5-HT2A and 5-HT2CRs both modulate different aspects of flexibility, with 5-HT2ARs modulating learning from negative feedback as measured using RL parameters and 5-HT2CRs for learning from positive feedback assessed through conventional measures.

Explainable Reinforcement Learning: A Survey and Comparative Review

Explainable reinforcement learning (XRL) is an emerging subfield of explainable machine learning that has attracted considerable attention in recent years. The goal of XRL is to elucidate the decision-making process of reinforcement learning (RL) agents in sequential decision-making settings. Equipped with this information, practitioners can better understand important questions about RL agents (especially those deployed in the real world), such as what the agents will do and why. Despite increased interest, there exists a gap in the literature for organizing the plethora of papers—especially in a way that centers the sequential decision-making nature of the problem. In this survey, we propose a novel taxonomy for organizing the XRL literature that prioritizes the RL setting. We propose three high-level categories: feature importance, learning process and Markov decision process, and policy-level. We overview techniques according to this taxonomy, highlighting challenges and opportunities for future work. We conclude by using these gaps to motivate and outline a roadmap for future work.

STPA-RL: Integrating Reinforcement Learning into STPA for Loss Scenario Exploration

Experience-based methods like reinforcement learning (RL) are often deemed less suitable for the safety field due to concerns about potential safety issues. To bridge this gap, we introduce STPA-RL, a methodology that integrates RL with System-Theoretic Process Analysis (STPA). STPA is a safety analysis technique that identifies causative factors leading to unsafe control actions and system hazards through loss scenarios. In the context of STPA-RL, we formalize the Markov Decision Process based on STPA analysis results to incorporate control algorithms into the system environment. The agent learns safe actions through reward-based learning, tracking potential hazard paths to validate system safety. Specifically, by analyzing various loss scenarios related to the Platform Screen Door, we assess the applicability of the proposed approach by evaluating hazard trajectory graphs and hazard frequencies in the system. This paper streamlines the RL process for loss scenario identification through STPA, contributing to self-guided loss scenarios and diverse system modeling. Additionally, it offers effective simulations for proactive development to enhance system safety and provide practical assistance in the safety field.

Explore 3D Dance Generation via Reward Model from Automatically-Ranked Demonstrations

This paper presents an Exploratory 3D Dance generation framework, E3D2, designed to address the exploration capability deficiency in existing music-conditioned 3D dance generation models. Current models often generate monotonous and simplistic dance sequences that misalign with human preferences because they lack exploration capabilities.The E3D2 framework involves a reward model trained from automatically-ranked dance demonstrations, which then guides the reinforcement learning process. This approach encourages the agent to explore and generate high quality and diverse dance movement sequences. The soundness of the reward model is both theoretically and experimentally validated. Empirical experiments demonstrate the effectiveness of E3D2 on the AIST++ dataset.

Reinforcement learning in autonomous driving

Automatic driving technology has become a highly researched field in recent years, aiming to achieve vehicle driving without human intervention. In this regard, reinforcement learning techniques have played a crucial role. This study discusses and analyses the use of reinforcement learning in automatic driving methods. The research begins with the process of reinforcement learning. In the architectural framework, there is a special emphasis on designing innovative reward functions to encourage safe and socially acceptable driving behaviour, while considering uncertainty factors through advanced Bayesian neural networks. This paper primarily focuses on aspects such as scene understanding, localization and mapping, planning and driving strategies, and control. Furthermore, the paper analyses the key elements of automatic driving and delves into the specific complexities associated with each element. It highlights the utilization of reinforcement learning within the realm of autonomous driving. Reinforcement learning assists autonomous vehicles in understanding the surrounding environment, accurately identifying paths, making intelligent driving decisions, and safely controlling the vehicle. Reinforcement learning especially working with deep learning plays a crucial role in realizing and continuously improving automatic driving. Finally, the paper concludes with a summary and outlook on the entire study.

Human skill knowledge guided global trajectory policy reinforcement learning method.

Traditional trajectory learning methods based on Imitation Learning (IL) only learn the existing trajectory knowledge from human demonstration. In this way, it can not adapt the trajectory knowledge to the task environment by interacting with the environment and fine-tuning the policy. To address this problem, a global trajectory learning method which combinines IL with Reinforcement Learning (RL) to adapt the knowledge policy to the environment is proposed. In this paper, IL is proposed to acquire basic trajectory skills, and then learns the agent will explore and exploit more policy which is applicable to the current environment by RL. The basic trajectory skills include the knowledge policy and the time stage information in the whole task space to help learn the time series of the trajectory, and are used to guide the subsequent RL process. Notably, neural networks are not used to model the action policy and the Q value of RL during the RL process. Instead, they are sampled and updated in the whole task space and then transferred to the networks after the RL process through Behavior Cloning (BC) to get continuous and smooth global trajectory policy. The feasibility and the effectiveness of the method was validated in a custom Gym environment of a flower drawing task. And then, we executed the learned policy in the real-world robot drawing experiment.

Walking control of semi-passive robot via a modified Q-learning algorithm

The analysis and control of stability of passive biped robot has been the subject of an in-depth study because of its unique characteristics. This work gives a new perspective aiming at a force applied on foot of collision phase of walking process of passive robot, which is a completely new way of applying impulse force that is different from the previous. Therefore, the new equation has been derived due to this force which is not parallel to the support leg to provide further analysis for the study. Besides, this paper combines the characteristics of passive walking with the process of reinforcement learning, and an improved algorithm thus was designed to calculate the value of the control force. The simulation results show that control method designed in this article can not only make the unsteady passive robot achieve stable walking, but also have a quick control speed. Furthermore, the selection range of initial values is expanded by using this method, which provides a convenient reference for the later research work.

Contextual panel conditioning and reward models in large language models

Direct preference optimization (DPO) aims to match human preferences while reducing the complexity of reinforcement learning. Traditional methods such as reinforcement learning with human feedback (RLHF) first match reward models with cues and preferences, and then use reinforcement learning (RL) to find policies that maximize rewards. In contrast, DPO simplifies the process by directly optimizing the policy to satisfy preferences without explicit reward functions or RL processes. DPO is a more direct and potentially more efficient way to fine-tune a language model to remain consistent with human feedback. Additionally, OpenAI mentioned that they trained the model by imitating human ratings to help improve RLHF. The next step is to fit the model to a data set containing rich "conditions". For example, the training model generates a panel containing memories, conditions, goals, plans, and future tasks, and uses this panel for training. These conditions transform the "creative writing task" into the task of "distributing materials", reducing entropy in creative writing. Conditional reinforcement learning fine-tuning (C-RLFT) enables large language models to understand and generate human-like text, adapt to new information, and personalize responses while maintaining relevance and coherence. Future improvements include improving conditional panels using RLHF or RLAIF, iteration between datasets and models, aligning models with real-world needs, and building new base models based on 0-order optimization. These directions aim to make large language models more efficient, consistent with human preferences, and able to run in a variety of environments, including edge computing devices. Hello, here is some text without a meaning. This text should show what a printed text will look like at this place. If you read this text, you will get no information. Really? Is there no information? Is there a difference between this text and some nonsense like "Huardest gefburn"? Kjift – not at all! A blind text like this gives you information about the selected font, how the letters are written and an impression of the look. This text should contain all letters of the alphabet and it should be written in the original language. There is no need for special content, but the length of words should match the language.

Application of machine learning to SRAM circuit design

This paper discusses the application of machine learning in Static Random Access Memory (SRAM) circuit design. It mentions several aspects of application analysis, including SRAM design for machine learning applications, SRAM PUF design for machine learning modelling attacks, lightweight fault-tolerant mechanism for NVDLA-based edge AI chips, interconnection resource allocation of SRAM FPGAs using reinforcement learning and Markov decision process, and SRAM power optimization and privacy security using machine learning. This article conclude that machine learning can be used to model and optimize SRAM performance, improve system reliability and stability through fault detection and fault tolerance mechanisms, provide a more comprehensive and efficient approach to testing and verification, and automatically correct and repair errors. The significance of machine learning in SRAM design is to provide an intelligent method and tool that can speed up the design process, improve system performance, increase reliability, and drive the advancement of chip manufacturing and processes.

Intelligent reflecting surface‐assisted UAV inspection system based on transfer learning

AbstractIntelligent reflective surface (IRS) provides an effective solution for reconfiguring air‐to‐ground wireless channels, and intelligent agents based on reinforcement learning can dynamically adjust the reflection coefficient of IRS to adapt to changing channels. However, most exiting IRS configuration schemes based on reinforcement learning require long training time and are difficult to be industrially deployed. This paper, proposes a model‐free IRS control scheme based on reinforcement learning and adopts transfer learning to accelerate the training process. A knowledge base of the source tasks has been constructed for transfer learning, allowing accumulation of experience from different source tasks. To mitigate potential negative effects of transfer learning, quantitative analysis of task similarity through unmanned aerial vehicle (UAV) flight path is conducted. After identifying the most similar source task to the target task, parameters of the source task model are used as the initial values for the target task model to accelerate the convergence process of reinforcement learning. Simulation results demonstrate that the proposed method can increase the convergence speed of the traditional DDQN algorithm by up to 60%.

Fractional-order fuzzy sliding mode control of uncertain nonlinear MIMO systems using fractional-order reinforcement learning

This paper introduces a novel approach aimed at enhancing the control performance of a specific class of unknown multiple-input and multiple-output nonlinear systems. The proposed method involves the utilization of a fractional-order fuzzy sliding mode controller, which is implemented through online fractional-order reinforcement learning (FOFSMC-FRL). First, the proposed approach leverages two Takagi–Sugeno–Kang (TSK) fuzzy neural network actors. These actors approximate both the equivalent and switch control parts of the sliding mode control. Additionally, a critic TSK fuzzy neural network is employed to approximate the value function of the reinforcement learning process. Second, the FOFSMC-FRL parameters undergo online adaptation using an innovative fractional-order Levenberg–Marquardt learning method. This adaptive mechanism allows the controller to continuously update its parameters based on the system’s behavior, optimizing its control strategy accordingly. Third, the stability and convergence of the proposed approach are rigorously examined using Lyapunov theorem. Notably, the proposed structure offers several key advantages as it does not depend on knowledge of the system dynamics, uncertainty bounds, or disturbance characteristics. Moreover, the chattering phenomenon, often associated with sliding mode control, is effectively eliminated without compromising the system’s robustness. Finally, a comparative simulation study is conducted to demonstrate the feasibility and superiority of the proposed method over other control methods. Through this comparison, the effectiveness and performance advantages of the approach are validated.

Prioritized Experience-Based Reinforcement Learning With Human Guidance for Autonomous Driving.

Reinforcement learning (RL) requires skillful definition and remarkable computational efforts to solve optimization and control problems, which could impair its prospect. Introducing human guidance into RL is a promising way to improve learning performance. In this article, a comprehensive human guidance-based RL framework is established. A novel prioritized experience replay mechanism that adapts to human guidance in the RL process is proposed to boost the efficiency and performance of the RL algorithm. To relieve the heavy workload on human participants, a behavior model is established based on an incremental online learning method to mimic human actions. We design two challenging autonomous driving tasks for evaluating the proposed algorithm. Experiments are conducted to access the training and testing performance and learning mechanism of the proposed algorithm. Comparative results against the state-of-the-art methods suggest the advantages of our algorithm in terms of learning efficiency, performance, and robustness.