Combines Reinforcement Learning Research Articles

Autonomous driving decision-making control algorithm based on hierarchical reinforcement learning

In the decision-making model of autonomous driving based on learning, the reinforcement learning model has the problems of slow convergence and single application scenario, while the imitation learning model has the problem of poor generalization. In order to solve this problem, a two-layer reinforcement learning framework is proposed to replace the decision layer and control layer in the autonomous driving task. The decision layer divides the driving behavior into lane keeping, left lane change and right lane change. After the decision layer selects the corresponding behavior, it completes the behavior by changing the input of the control layer. Then, a new method for training the control layer, RL_COE (Reinforcement learning combined with online experts), is proposed by combining reinforcement learning and online experts. Finally, a highway simulation environment is built in Carla to verify the proposed algorithm and compare it with the reinforcement learning baseline algorithm. The results show that this method greatly improves the convergence speed and stability of the algorithm and can better complete the driving task.

A guided twin delayed deep deterministic reinforcement learning for vaccine allocation in human contact networks

This manuscript introduces an innovative approach to optimizing the distribution of a limited vaccine resource within a population modeled as a contact network, aiming to mitigate the spread of infectious diseases. The study develops a novel methodology that combines reinforcement learning and graph neural networks. To understand the dynamics of disease propagation, the study constructs an analytical model that outlines conditions for disease eradication or endemic states. This model supports a series of simulation experiments across various scenarios, demonstrating the proposed method’s superiority over random and centrality-based approaches in reducing the average number of infections per individual during an outbreak. The adaptability of the proposed method is further emphasized by its robust performance across networks of diverse sizes and configurations, highlighting its real-world applicability. The findings of this study have significant implications for public health policy and resource allocation, offering a promising framework for managing infectious disease outbreaks in complex and dynamic environments.

Energy management method for battery swapping station combining reinforcement learning and mathematical optimization

Abstract The rapid proliferation of electric vehicles has spurred the expansion of scalable Battery Swapping Station (BSS) to cater to the demand for swift charging. However, the current energy management faces challenges in coping with the fluctuation of charging batteries in scalable BSS, the unpredictability of electricity prices and battery demand, as well as the intricacy of demand response. So, this paper presents a two-layer optimization framework for energy management in scalable BSS. The framework decomposes the power scheduling problem in BSS into two subproblems and solves them with deep reinforcement learning and mathematical optimization. The upper layer uses deep reinforcement learning to schedule the BSS’s total power, while the lower layer uses mathematical optimization to allocate the power to each charging pile. The two layers cooperate to achieve an efficient solution. The experiments demonstrate that the proposed method can reduce the operating cost of the BSS, satisfy the safety and user’s demand, and facilitate grid demand response, in BSSs of different sizes. The method is an effective solution for power scheduling of scalable BSS.

The guidance of neutral human populations maintains cooperation in the prisoner's dilemma game

In game theory, the emergence and maintenance of cooperative behavior within a group is a significant topic in evolutionary game theory and complex network theory. However, the limitations of a single mechanism in traditional networks restrict a thorough analysis of the sustenance and development of cooperative behavior, given the challenges posed by the diversity of social groups. To address this issue, this paper combines reinforcement learning game strategies with traditional prisoner's dilemma strategies based on two-layer coupled network to investigate the transmission of cooperative behavior among individuals in games. In our research, we study the evolutionary pattern and phase transitions using the Monte Carlo method. We use the prisoner's dilemma game as a mathematical model, establishing two subpopulations in each layer, with mutually payoff-neutral players between different subpopulations. This configuration results in intriguing spatiotemporal dynamics and patterns, leading to the spontaneous emergence of a cyclic dominance, where defectors from one group become prey for cooperators in another group, and vice versa. By simulating game evolution, we explore individual strategy changes and the impact of individual abilities on cooperative behavior in reinforcement learning. Extensive validations indicate that, in social dilemmas, adjusting the abilities of groups through effective guidance can sustain cooperative behavior. This guidance enables us to comprehend the stability of cooperation under adverse conditions. Simultaneously, the coexistence of two subpopulations greatly amplifies the complexity of evolutionary dynamics, causing a increase in cooperation rate.

The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Models

In this work, we review research studies that combine Reinforcement Learning (RL) and Large Language Models (LLMs), two areas that owe their momentum to the development of Deep Neural Networks (DNNs). We propose a novel taxonomy of three main classes based on the way that the two model types interact with each other. The first class, RL4LLM, includes studies where RL is leveraged to improve the performance of LLMs on tasks related to Natural Language Processing (NLP). RL4LLM is divided into two sub-categories depending on whether RL is used to directly fine-tune an existing LLM or to improve the prompt of the LLM. In the second class, LLM4RL, an LLM assists the training of an RL model that performs a task that is not inherently related to natural language. We further break down LLM4RL based on the component of the RL training framework that the LLM assists or replaces, namely reward shaping, goal generation, and policy function. Finally, in the third class, RL+LLM, an LLM and an RL agent are embedded in a common planning framework without either of them contributing to training or fine-tuning of the other. We further branch this class to distinguish between studies with and without natural language feedback. We use this taxonomy to explore the motivations behind the synergy of LLMs and RL and explain the reasons for its success, while pinpointing potential shortcomings and areas where further research is needed, as well as alternative methodologies that serve the same goal.

Modular control architecture for safe marine navigation: Reinforcement learning with predictive safety filters

Many autonomous systems are safety-critical, making it essential to have a closed-loop control system that satisfies constraints arising from underlying physical limitations and safety aspects in a robust manner. However, this is often challenging to achieve for real-world systems. For example, autonomous ships at sea have nonlinear and uncertain dynamics and are subject to numerous time-varying environmental disturbances such as waves, currents, and wind. There is increasing interest in using machine learning-based approaches to adapt these systems to more complex scenarios, but there are few standard frameworks that guarantee the safety and stability of such systems. Recently, predictive safety filters (PSF) have emerged as a promising method to ensure constraint satisfaction in learning-based control, bypassing the need for explicit constraint handling in the learning algorithms themselves. The safety filter approach leads to a modular separation of the problem, allowing the use of arbitrary control policies in a task-agnostic way. The filter takes in a potentially unsafe control action from the main controller and solves an optimization problem to compute a minimal perturbation of the proposed action that adheres to both physical and safety constraints. In this work, we combine reinforcement learning (RL) with predictive safety filtering in the context of marine navigation and control. The RL agent is trained on path-following and safety adherence across a wide range of randomly generated environments, while the predictive safety filter continuously monitors the agents' proposed control actions and modifies them if necessary. The combined PSF/RL scheme is implemented on a simulated model of Cybership II, a miniature replica of a typical supply ship. Safety performance and learning rate are evaluated and compared with those of a standard, non-PSF, RL agent. It is demonstrated that the predictive safety filter is able to keep the vessel safe, while not prohibiting the learning rate and performance of the RL agent.

An overview: Attention mechanisms in multi-agent reinforcement learning

In recent years, in the field of Multi-Agent Systems (MAS), significant progress has been made in the research of algorithms that combine Reinforcement Learning (RL) with Attention Mechanism (AM). However, there is a lack of comprehensive reviews in this field. Based on this, this paper does the following work. Firstly, it reviews the classical algorithms of RL and AM; Secondly, it systematically introduces the combination of RL and AM; Thirdly, it sorts out their application in the field of single-agent and multi-agent, and pays attention to and looks forward to the challenges and future research prospects in this field; The last part offers a comprehensive analysis of the challenges encountered by research in this field and anticipates future research paths. The research offers a conceptual understanding and theoretical foundation for future applications of RL using AM and facilitates further in-depth study in this field for researchers.

Combining reinforcement learning algorithm and genetic algorithm to solve the traveling salesman problem

AbstractWith the growing recognition of the unique advantages of reinforcement learning and genetic algorithms in addressing combinatorial optimization problems, this study aims to integrate these two methods to collectively tackle the classic combinatorial optimization challenge of the travelling salesman problem (TSP). The TSP stands as a quintessential combinatorial optimization challenge, tasked with determining the shortest path among designated cities. This paper introduces an innovative approach by amalgamating reinforcement learning's path selection prowess with genetic algorithms' global search strategy, aiming to uncover superior solutions in TSP. Specifically, the experiment employs a dual Q‐learning algorithm within reinforcement learning to identify multiple optimal paths, serving as progenitors for the genetic algorithm to further enhance performance. The paper meticulously outlines the problem modelling process, elucidating TSP instance definitions, descriptions, and precise objective function definitions. Experimental findings underscore the substantial enhancements achievable in TSP optimization through this comprehensive approach, offering a fresh perspective and methodology for tackling combinatorial optimization challenges.

AiTO: Simultaneous gate sizing and buffer insertion for timing optimization with GNNs and RL

Gate sizing and buffer insertion for timing optimization are performed extensively in electronic design automation (EDA) flows. Both of them aim to adjust the upstream and downstream capacitances of gates/buffers to minimize delay. However, most of existing work focuses on gate sizing or buffer insertion independently. This paper proposes a learning-based timing optimization framework, AiTO, that combines reinforcement learning with graph neural network, to perform simultaneously gate sizing and buffer insertion. We model buffer insertion as a special gate sizing by determining possible buffer locations in advance and treating the buffer insertion and gate sizing as an RL process. Experimental results on 10 real designs (28-nm and 110-nm) show that, AiTO can achieve better worst negative slack (WNS) optimization results than OpenROAD while being able to improve the results of the commercial tool, Innovus, to some extent. Moreover, ablation studies demonstrate the benefits of performing simultaneous gate sizing and buffer insertion for timing optimization.

Combining Reinforcement Learning and Tensor Networks, with an Application to Dynamical Large Deviations.

We present a framework to integrate tensor network (TN) methods with reinforcement learning (RL) for solving dynamical optimization tasks. We consider the RL actor-critic method, a model-free approach for solving RL problems, and introduce TNs as the approximators for its policy and value functions. Our "actor-critic with tensor networks" (ACTeN) method is especially well suited to problems with large and factorizable state and action spaces. As an illustration of the applicability of ACTeN we solve the exponentially hard task of sampling rare trajectories in two paradigmatic stochastic models, the East model of glasses and the asymmetric simple exclusion process, the latter being particularly challenging to other methods due to the absence of detailed balance. With substantial potential for further integration with the vast array of existing RL methods, the approach introduced here is promising both for applications in physics and to multi-agent RL problems more generally.

Reinforcement learning based blockchain model for revoking unauthorized access in Virtualized Network Functions‐based Internet of Things Mobile Edge Computing

AbstractVNFs boost data processing efficiency in Mobile Edge Computing (MEC)‐driven Internet of Things (IoT) for healthcare, smart cities, and industrial automation. VNF‐based IoT MEC systems encounter a significant security threat due to unauthorized access, posing risks to data privacy and system integrity. Existing approaches struggle to adapt to dynamic environments and lack tamper‐proof enforcement mechanisms. In this work, we propose a novel system combining Reinforcement Learning (RL) and blockchain technology to revoke unauthorized access in VNF‐based IoT MEC. We introduce the Integrated Action‐selection DRL Algorithm for Unauthorized Access Revocation (IASDRL‐UAR), a novel RL approach that excels in dynamic environments by handling both continuous and discrete actions, enabling real‐time optimization of security risk, execution time, and energy consumption. A behavior control contract (BCC) is proposed and integrated into the RL system, automating behavior checks and enforcement, streamlining security management, and reducing manual intervention. RL feedback plays a pivotal role in steering dynamic security adjustments, gaining valuable perspectives from user behavior via trust scores in the behavior contract. The security features of the proposed method are analyzed. Performance comparisons reveal a substantial improvement, with the proposed system outperforming existing methods by 30% in terms of throughput, 21.7% in system stability, and 26% in access revocation latency. Additionally, the system demonstrates a higher security index, energy efficiency, and scalability.

Adaptive composite learning control of a flexible two‐link manipulator with unknown spatiotemporally varying disturbance

AbstractThis article presents a novel adaptive composite learning (ACL) control strategy combining reinforcement learning and a disturbance observer (DOB) to address vibration issues in a flexible two‐link manipulator (FTLM) system affected by unknown spatiotemporally varying disturbances. Based on the assumed mode method, the FTLM system is initially transformed into an ordinary differential equation model, while effectively capturing the elastic deformation and vibration characteristics of the flexible link. A composite learning controller, based on the actor‐critic algorithm and DOB, is then developed to achieve trajectory tracking and vibration suppression in the FTLM system. The DOB in the controller compensates for unknown disturbances resulting in reduced system error. It is noting that the proposed optimal control strategy is continuously gathering system experience and evaluating the current policy's effectiveness. The stability and robustness of the closed‐loop system incorporating the composite controller are analyzed using Lyapunov's direct method, and the semi‐global uniform ultimate boundedness of the tracking and vibration errors are also demonstrated. To validate the effectiveness and superiority of the proposed ACL controller, comparative simulations and experiments are conducted on the Quanser experimental platform.

Reinforcement learning for addressing the cold-user problem in recommender systems

Recommender systems are widely used in webshops because of their ability to provide users with personalized recommendations. However, the cold-user problem (i.e., recommending items to new users) is an important issue many webshops face. With the recent General Data Protection Regulation in Europe, the use of additional user information such as demographics is not possible without the user’s explicit consent. Several techniques have been proposed to solve the cold-user problem. Many of these techniques utilize Active Learning (AL) methods, which let cold users rate items to provide better recommendations for them. In this research, we propose two novel approaches that combine reinforcement learning with AL to elicit the users’ preferences and provide them with personalized recommendations. We compare reinforcement learning approaches that are either AL-based or item-based, where the latter predicts users’ ratings of an item by using their ratings of similar items. Differently than many of the existing approaches, this comparison is made based on implicit user information. Using a large real-world dataset, we show that the item-based strategy is more accurate than the AL-based strategy as well as several existing AL strategies.

Model predictive control combined reinforcement learning for automatic vehicles applied in intelligent transportation system

Model predictive control combined reinforcement learning for automatic vehicles applied in intelligent transportation system

Integrating Reinforcement Learning and Monte Carlo Tree Search for enhanced neoantigen vaccine design.

Recent advances in cancer immunotherapy have highlighted the potential of neoantigen-based vaccines. However, the design of such vaccines is hindered by the possibility of weak binding affinity between the peptides and the patient's specific human leukocyte antigen (HLA) alleles, which may not elicit a robust adaptive immune response. Triggering cross-immunity by utilizing peptide mutations that have enhanced binding affinity to target HLA molecules, while preserving their homology with the original one, can be a promising avenue for neoantigen vaccine design. In this study, we introduced UltraMutate, a novel algorithm that combines Reinforcement Learning and Monte Carlo Tree Search, which identifies peptide mutations that not only exhibit enhanced binding affinities to target HLA molecules but also retains a high degree of homology with the original neoantigen. UltraMutate outperformed existing state-of-the-art methods in identifying affinity-enhancing mutations in an independent test set consisting of 3660 peptide-HLA pairs. UltraMutate further showed its applicability in the design of peptide vaccines for Human Papillomavirus and Human Cytomegalovirus, demonstrating its potential as a promising tool in the advancement of personalized immunotherapy.

Optimization of high-speed fixed-wing UAV penetration strategy based on deep reinforcement learning

The penetration decision of unmanned aerial vehicles (UAVs) is one of the key components for UAVs to detect or strike crucial targets in adversarial environments. Presently, most methods lack research on penetration decisions under unknown interception information. The executing UAV for decision-making is generally a simplified three-degree-of-freedom model. The decision format is often simplified to path point optimization, lacking control over the stability of complex model attitudes and exhibiting relatively poor robustness. This paper proposes a penetration relative motion theory and aircraft control method based on the six-degrees-of-freedom UAV model to reduce response errors between algorithm control decisions and actual flight control. Using a Markov decision process, a UAV penetration decision control method based on an attitude-overload control loop is designed. This method combines reinforcement learning techniques to enable autonomous penetration decision-making and control for UAVs facing active defense scenarios, enhancing the autonomy, accuracy, and generalization of UAV online decision-making. The paper concludes by implementing autonomous decision-making and control for UAV penetration through the integration of interceptor trajectory prediction with proximal policy optimization. This method reduces the dependency of penetration strategies on interceptor information, ensuring both the generalization ability of penetration strategy models under uncertain information and the consistency of decision stability. Through simulations and experiments, we verified that the penetration strategy based on the pre-sampling PPO method improves the penetration effect by 15.44 % on average and the hit rate by 18.48 %. Moreover, the probability of penetration in different scenarios exceeds 70 %, and the penetration effect is improved by more than 10 % in the case of multiple interceptors. This paper also analyzes the impact of interceptor spatial distribution and the quantity of interceptors on the effectiveness of UAV penetration decision-making.

Decoding topological XYZ 2 codes with reinforcement learning based on attention mechanisms

Quantum error correction, a technique that relies on the principle of redundancy to encode logical information into additional qubits to better protect the system from noise, is necessary to design a viable quantum computer. For this new topological stabilizer code-XYZ 2 code defined on the cellular lattice, it is implemented on a hexagonal lattice of qubits and it encodes the logical qubits with the help of stabilizer measurements of weight six and weight two. However topological stabilizer codes in cellular lattice quantum systems suffer from the detrimental effects of noise due to interaction with the environment. Several decoding approaches have been proposed to address this problem. Here, we propose the use of a state-attention based reinforcement learning decoder to decode XYZ 2 codes, which enables the decoder to more accurately focus on the information related to the current decoding position, and the error correction accuracy of our reinforcement learning decoder model under the optimisation conditions can reach 83.27% under the depolarizing noise model, and we have measured thresholds of 0.18856 and 0.19043 for XYZ 2 codes at code spacing of 3–7 and 7–11, respectively. our study provides directions and ideas for applications of decoding schemes combining reinforcement learning attention mechanisms to other topological quantum error-correcting codes.

Combining hybrid metaheuristic algorithms and reinforcement learning to improve the optimal control of nonlinear continuous-time systems with input constraints

This paper proposes an innovative method for achieving optimal tracking control in nonlinear continuous-time systems with input constraints. The method combines reinforcement learning and hybrid metaheuristics to enhance the controller’s performance. Specifically, a hybrid metaheuristic algorithm is employed to optimize the hyperparameters of a critic neural network, which serves as the system’s controller. The proposed approach is evaluated through extensive simulation studies on a nonlinear system with input constraints. Results demonstrate its superiority over traditional control techniques in terms of accuracy, timeliness, and stability. Notably, the approach effectively eliminates overshoot and steady-state error while providing precise and prompt tracking and showcasing remarkable robustness against model uncertainties. By synergistically integrating reinforcement learning and hybrid metaheuristics, this approach represents a significant advancement in enhancing the control performance of complex nonlinear systems. The simulation studies confirm superiority of the proposed approach over existing techniques, offering a promising solution for achieving optimal tracking control in nonlinear systems with input constraints. This approach holds potential for a wide range of applications, including robotics, aerospace, and manufacturing, where precise and prompt tracking control is critical.

A Sequential Sampling Approach to the Integration of Habits and Goals

Habits often conflict with goal-directed behaviors and this phenomenon continues to attract interests from neuroscientists, experimental psychologists, and applied health psychologists. Recent computational models explain habit-goal conflicts as the competitions between two learning systems, arbitrated by a central unit. Based on recent research that combined reinforcement learning and sequential sampling, we show that habit-goal conflicts can be more parsimoniously explained by a dynamic integration of habit and goal values in a sequential sampling model, without any arbitration. A computational model was developed by extending the multialternative decision field theory with the assumptions that habits bias starting points of preference accumulation, and that goal importance and goal relevance determine sampling probabilities of goal-related attributes. Simulation studies demonstrated our approach’s ability to qualitatively reproduce important empirical findings from three paradigms – classic devaluation, devaluation with a concurrent schedule, and reversal learning, and to predict gradual changes in decision times. In addition, a parameter recovery exercise using approximate Bayesian computation showcased the possibility of fitting the model to empirical data in future research. Implications of our work for habit theories and applications are discussed.

Reinforcement Learning of Bipedal Walking Using a Simple Reference Motion

In this paper, a novel reinforcement learning method that enables a humanoid robot to learn bipedal walking using a simple reference motion is proposed. Reinforcement learning has recently emerged as a useful method for robots to learn bipedal walking, but, in many studies, a reference motion is necessary for successful learning, and it is laborious or costly to prepare a reference motion. To overcome this problem, our proposed method uses a simple reference motion consisting of three sine waves and automatically sets the waveform parameters using Bayesian optimization. Thus, the reference motion can easily be prepared with minimal human involvement. Moreover, we introduce two means to facilitate reinforcement learning: (1) we combine reinforcement learning with inverse kinematics (IK), and (2) we use the reference motion as a bias for the action determined via reinforcement learning, rather than as an imitation target. Through numerical experiments, we show that our proposed method enables bipedal walking to be learned based on a small number of samples. Furthermore, we conduct a zero-shot sim-to-real transfer experiment using a domain randomization method and demonstrate that a real humanoid robot, KHR-3HV, can walk with the controller acquired using the proposed method.