Classical Reinforcement Learning Research Articles

The Effect of Hyperparameters on the Model Convergence Rate of Cliff Walking Problem Based on Q-Learning

With the rapid development of AI, machine learning has become a hot topic. Among them, reinforcement learning is an important branch of machine learning. With the continuous efforts of scholars, various algorithms emerge in an endless stream. Q-Learning algorithm is a very classic reinforcement learning algorithm, which is the basis of many algorithms. Basically, the Q-table is updated by iteration, so that the agent can choose the best action in the corresponding situation, so as to get closer to the optimal solution. In essence, Q-Learning is sequential difference of different strategies. In the process of learning different strategies, there are two different strategies, goal strategy and behavior strategy. In order to balance the relationship between exploration and exploitation, the -greedy strategy is selected to maintain a certain exploratory property of the agent, and relevant hyperparameters such as learning rate (alpha) and discount factor (gamma) are set. However, the research on Q-Learning hyperparameters is not clear enough. In this paper, the author will study the influence of Q-Learning algorithm hyperparameters on its convergence speed under a relatively simple model.

A Multi-Strategy Co-Evolutionary Particle Swarm Optimization Algorithm with Its Convergence Analysis

Compared to the single-strategy particle swarm optimization (PSO) algorithm, the multi-strategy PSO shows potential advantages in solving complex optimization problems. In this study, a novel framework of the multi-strategy co-evolutionary PSO (M-PSO) is first proposed in which a matrix parameter pool scheme is introduced. In the scheme, multiple strategies are taken into account in the matrix parameter pool and new hybrid strategies can be generated. Then, the convergence analysis is made and the convergence conditions are provided for the co-evolutionary PSO framework when some operators are specified. Subsequently, based on the PSO framework, a novel multi-strategy co-evolutionary PSO is developed using [Formula: see text]-learning which is a classical reinforcement learning technique. In the proposed M-PSO, both the parameter optimization by the orthogonal method and the convergence conditions are embedded to improve the performance of the algorithm. Finally, the experiments are conducted on two test suites, CEC2017 and CEC2019, and the results indicate that M-PSO outperforms several meta-heuristic algorithms on most of the test problems.

Risk-averse supply chain management via robust reinforcement learning

Classical reinforcement learning (RL) may suffer performance degradation when the environment deviates from training conditions, limiting its application in risk-averse supply chain management. This work explores using robust RL in supply chain operations to hedge against environment inconsistencies and changes. Two robust RL algorithms, Qˆ-learning and β-pessimistic Q-learning, are examined against conventional Q-learning and a baseline order-up-to inventory policy. Furthermore, this work extends RL applications from forward to closed-loop supply chains. Two case studies are conducted using a supply chain simulator developed with agent-based modeling. The results show that Q-learning can outperform the baseline policy under normal conditions, but notably degrades under environment deviations. By comparison, the robust RL models tend to make more conservative inventory decisions to avoid large shortage penalties. Specifically, fine-tuned β-pessimistic Q-learning can achieve good performance under normal conditions and maintain robustness against moderate environment inconsistencies, making it suitable for risk-averse decision-making.

Dopamine release plateau and outcome signals in dorsal striatum contrast with classic reinforcement learning formulations

We recorded dopamine release signals in centromedial and centrolateral sectors of the striatum as mice learned consecutive versions of visual cue-outcome conditioning tasks. Dopamine release responses differed for the centromedial and centrolateral sites. In neither sector could these be accounted for by classic reinforcement learning alone as classically applied to the activity of nigral dopamine-containing neurons. Medially, cue responses ranged from initial sharp peaks to modulated plateau responses; outcome (reward) responses during cue conditioning were minimal or, initially, negative. At centrolateral sites, by contrast, strong, transient dopamine release responses occurred at both cue and outcome. Prolonged, plateau release responses to cues emerged in both regions when discriminative behavioral responses became required. At most sites, we found no evidence for a transition from outcome signaling to cue signaling, a hallmark of temporal difference reinforcement learning as applied to midbrain dopaminergic neuronal activity. These findings delineate a reshaping of striatal dopamine release activity during learning and suggest that current views of reward prediction error encoding need review to accommodate distinct learning-related spatial and temporal patterns of striatal dopamine release in the dorsal striatum.

Decision Transformer-Based Efficient Data Offloading in LEO-IoT.

Recently, the Internet of Things (IoT) has witnessed rapid development. However, the scarcity of computing resources on the ground has constrained the application scenarios of IoT. Low Earth Orbit (LEO) satellites have drawn people's attention due to their broader coverage and shorter transmission delay. They are capable of offloading more IoT computing tasks to mobile edge computing (MEC) servers with lower latency in order to address the issue of scarce computing resources on the ground. Nevertheless, it is highly challenging to share bandwidth and power resources among multiple IoT devices and LEO satellites. In this paper, we explore the efficient data offloading mechanism in the LEO satellite-based IoT (LEO-IoT), where LEO satellites forward data from the terrestrial to the MEC servers. Specifically, by optimally selecting the forwarding LEO satellite for each IoT task and allocating communication resources, we aim to minimize the data offloading latency and energy consumption. Particularly, we employ the state-of-the-art Decision Transformer (DT) to solve this optimization problem. We initially obtain a pre-trained DT through training on a specific task. Subsequently, the pre-trained DT is fine-tuned by acquiring a small quantity of data under the new task, enabling it to converge rapidly, with less training time and superior performance. Numerical simulation results demonstrate that in contrast to the classical reinforcement learning approach (Proximal Policy Optimization), the convergence speed of DT can be increased by up to three times, and the performance can be improved by up to 30%.

Design and analysis of parallel quantum transfer fractal priority replay with dynamic memory algorithm in quantum reinforcement learning for robotics

AbstractThis paper introduces the parallel quantum transfer fractal priority reply with dynamic memory (P‐QTFPR‐DM) algorithm, an innovative approach that combines quantum computing and reinforcement learning (RL) to enhance decision‐making in autonomous vehicles. Leveraging quantum principles such as superposition and entanglement, P‐QTFPR‐DM optimises Q‐value approximation through a custom quantum circuit (UQC), facilitating efficient exploration and exploitation in high‐dimensional state‐action spaces. This algorithm utilises a quantum neural network (QNN) with 4 qubits to encode and process Q‐values. The autonomous vehicle, equipped with GPS for real‐time navigation, uses P‐QTFPR‐DM to reach a predefined destination with coordinates 12.82,514,234,148 latitude and 80.0,451,311,962,242 longitude. Through extensive numerical simulations, P‐QTFPR‐DM demonstrates a 30% reduction in decision‐making time and a 25% improvement in navigation accuracy compared to classical RL methods. The QNN‐based approach achieves a 95% success rate in reaching the destination within a 5‐m accuracy threshold, whereas traditional RL methods achieve only an 85% success rate. Dynamic memory management in P‐QTFPR‐DM optimises computational resources, enhancing the vehicle's adaptability to environmental changes. These results highlight the potential of quantum computing to significantly advance autonomous vehicle technology by improving both efficiency and effectiveness in complex navigation tasks. Future research will focus on refining the algorithm and exploring its real‐world applications to enhance autonomous vehicle performance.

Ex-RL: Experience-based reinforcement learning

Reinforcement learning (RL) has achieved significant success across various tasks. However, generalizing RL for similar tasks remains a challenge. This study leverages expertise from related tasks to introduce a novel algorithm, Ex-RL, for executing transfer learning in tabular RL. The methodology concentrates on abstracting previous experiences into descriptive data and utilizing such data for similar tasks. The research focuses on classic RL solutions for balancing and anti-balancing, which improve the sample efficiency of the learning process. Studies indicate that weak learners, such as Q-learning, require fewer learning episodes, resulting in a 50% improvement and a higher success rate in the learning process. An online virtual lab was developed to facilitate the execution of the experiments. The code is available at Github.

Humans forage for reward in reinforcement learning tasks.

How do we make good decisions in uncertain environments? In psychology and neuroscience, the classic answer is that we calculate the value of each option and then compare the values to choose the most rewarding, modulo some exploratory noise. An ethologist, conversely, would argue that we commit to one option until its value drops below a threshold, at which point we start exploring other options. In order to determine which view better describes human decision-making, we developed a novel, foraging-inspired sequential decision-making model and used it to ask whether humans compare to threshold ("Forage") or compare alternatives ("Reinforcement-Learn" [RL]). We found that the foraging model was a better fit for participant behavior, better predicted the participants' tendency to repeat choices, and predicted the existence of held-out participants with a pattern of choice that was almost impossible under RL. Together, these results suggest that humans use foraging computations, rather than RL, even in classic reinforcement learning tasks.

An Improved Dyna-Q Algorithm Inspired by the Forward Prediction Mechanism in the Rat Brain for Mobile Robot Path Planning.

The traditional Model-Based Reinforcement Learning (MBRL) algorithm has high computational cost, poor convergence, and poor performance in robot spatial cognition and navigation tasks, and it cannot fully explain the ability of animals to quickly adapt to environmental changes and learn a variety of complex tasks. Studies have shown that vicarious trial and error (VTE) and the hippocampus forward prediction mechanism in rats and other mammals can be used as key components of action selection in MBRL to support "goal-oriented" behavior. Therefore, we propose an improved Dyna-Q algorithm inspired by the forward prediction mechanism of the hippocampus to solve the above problems and tackle the exploration-exploitation dilemma of Reinforcement Learning (RL). This algorithm alternately presents the potential path in the future for mobile robots and dynamically adjusts the sweep length according to the decision certainty, so as to determine action selection. We test the performance of the algorithm in a two-dimensional maze environment with static and dynamic obstacles, respectively. Compared with classic RL algorithms like State-Action-Reward-State-Action (SARSA) and Dyna-Q, the algorithm can speed up spatial cognition and improve the global search ability of path planning. In addition, our method reflects key features of how the brain organizes MBRL to effectively solve difficult tasks such as navigation, and it provides a new idea for spatial cognitive tasks from a biological perspective.

Co-evolutionary traffic signal control using reinforcement learning for road networks under stochastic capacity

A co-evolutionary traffic signal control using reinforcement learning approach (CORLA) is proposed for time-varying road networks under stochastic capacity. Classic reinforcement learning based traffic signal control cannot effectively reduce traffic congestion for large-scale road networks while standard evolutionary metaheuristics often suffer from significantly high computational cost. A co-evolutionary decomposition algorithm (CODA) is proposed to improve traffic mobility for urban road networks with time-varying traffic flow. To capture time-varying spatial evolution of traffic flow inside road links, a stochastic traffic model is presented. To fully consider road users’ response, a stochastic bi-level optimization problem (SBOP) is given where road users’ route choice can be fully taken into account. To efficiently implement CORLA in a large-scale road network against high-consequence realization for stochastic capacity, a coordinated co-evolutionary two phase control (CCTPC) is proposed. Numerical experiments are performed at a real-data city road network and various sizable traffic grids. As compared to state-of-the-art traffic signal control for various traffic conditions, obtained results showed that CCTPC exhibits sufficient gain of achieving road network performance and suffers from the least computational cost in all cases.

An adaptable and personalized framework for top-N course recommendations in online learning

In recent years, the proliferation of Massive Open Online Courses (MOOC) platforms on a global scale has been remarkable. Learners can now meet their learning demands with the help of MOOC. However, learners might not understand the course material well if they have access to a lot of information due to their inadequate expertise and cognitive ability. Personalized Recommender Systems (RSs), a cutting-edge technology, can assist in addressing this issue. It greatly increases resource acquisition through personalized availability for various people of all ages. Intelligent learning methods, such as machine learning and Reinforcement Learning (RL) can be used in RS challenges. However, machine learning needs supervised data and classical RL is not suitable for multi-task recommendations in online learning platforms. To address these challenges, the proposed framework integrates a Deep Reinforcement Learning (DRL) and multi-agent approach. This adaptive system personalizes the learning experience by considering key factors such as learner sentiments, learning style, preferences, competency, and adaptive difficulty levels. We formulate the interactive RS problem using a DRL-based Actor-Critic model named DRR, treating recommendations as a sequential decision-making process. The DRR enables the system to provide top-N course recommendations and personalized learning paths, enriching the student's experience. Extensive experiments on a MOOC dataset such as the 100 K Coursera course review validate the proposed DRR model, demonstrating its superiority over baseline models in major evaluation metrics for long-term recommendations. The outcomes of this research contribute to the field of e-learning technology, guiding the design and implementation of course RSs, to facilitate personalized and relevant recommendations for online learning students.

UISA: User Information Separating Architecture for Commodity Recommendation Policy with Deep Reinforcement Learning

Commodity recommendation contributes an important part of individuals’ daily life. In this context, deep reinforcement learning methods have demonstrated substantial efficacy in enhancing recommender systems’ performance. Nevertheless, several recommender systems directly utilize original feature information as a foundational element for decision-making, which seems simplistic and low efficient. Furthermore, the incorporation of sequential decision-making adds complexity to the task of recommendation. In pursuit of maximizing the long-term sequential returns of recommender systems, our study introduces a novel architecture, named User Information Separating Architecture (UISA). This framework is tailored to align with classic reinforcement learning algorithms and aims to extract the user’s interest value through the discrete processing of both static and dynamic user information. Through integration with deep reinforcement learning, the architecture is oriented towards the maximization of long-term profit and is applicable in sequential recommendation scenarios. We conduct experimental assessments by combining the proposed architecture with proximal policy optimization (PPO) and deep deterministic policy gradient (DDPG) algorithms. The outcomes illustrate marked improvements in commodity recommendation, showcasing enhancements ranging from approximately 5% to 40% in both reward and click-through rate metrics across a self-constructed JDEnv environment and the Virtual Taobao environment. Through comparison experiments, the UISA models demonstrate comparable performance.

OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning

Model-based offline reinforcement learning (RL) algorithms have emerged as a promising paradigm for offline RL. These algorithms usually learn a dynamics model from a static dataset of transitions, use the model to generate synthetic trajectories, and perform conservative policy optimization within these trajectories. However, our observations indicate that policy optimization methods used in these model-based offline RL algorithms are not effective at exploring the learned model and induce biased exploration, which ultimately impairs the performance of the algorithm. To address this issue, we propose Offline Conservative ExplorAtioN (OCEAN), a novel rollout approach to model-based offline RL. In our method, we incorporate additional exploration techniques and introduce three conservative constraints based on uncertainty estimation to mitigate the potential impact of significant dynamic errors resulting from exploratory transitions. Our work is a plug-in method and can be combined with classical model-based RL algorithms, such as MOPO, COMBO, and RAMBO. Experiment results of our method on the D4RL MuJoCo benchmark show that OCEAN significantly improves the performance of existing algorithms.

BadRL: Sparse Targeted Backdoor Attack against Reinforcement Learning

Backdoor attacks in reinforcement learning (RL) have previously employed intense attack strategies to ensure attack success. However, these methods suffer from high attack costs and increased detectability. In this work, we propose a novel approach, BadRL, which focuses on conducting highly sparse backdoor poisoning efforts during training and testing while maintaining successful attacks. Our algorithm, BadRL, strategically chooses state observations with high attack values to inject triggers during training and testing, thereby reducing the chances of detection. In contrast to the previous methods that utilize sample-agnostic trigger patterns, BadRL dynamically generates distinct trigger patterns based on targeted state observations, thereby enhancing its effectiveness. Theoretical analysis shows that the targeted backdoor attack is always viable and remains stealthy under specific assumptions. Empirical results on various classic RL tasks illustrate that BadRL can substantially degrade the performance of a victim agent with minimal poisoning efforts (0.003% of total training steps) during training and infrequent attacks during testing. Code is available at: https://github.com/7777777cc/code.

A Comprehensive Analysis of Game theory on Multi-Agent Reinforcement

In recent years, reinforcement learning has gradually emerged as a popular research field in artificial intelligence. However, for multi-agent systems, due to their complex environment and an abundance of agents, the learning target of maximizing the anticipated value of cumulative rewards for specific agents frequently fails to converge. Introducing game theory into reinforcement learning can effectively address the interactions among intelligent agents, provide a rationale for convergence points corresponding to strategies. To address the issue of non-existence of optimal solutions for certain tasks in multi-agent settings, this paper takes a game-theoretic perspective and summarizes classical reinforcement learning algorithms developed in recent years. The basic theory of multi-agent reinforcement learning, essential game theory, categorization of reinforcement learning using multiple agents game strategies, primary worries are addressed in the study. It also analyzes the challenges that game-theoretic multi-agent reinforcement learning algorithms may encounter in the future, along with the relevant optimization directions.

Hybrid actor-critic algorithm for quantum reinforcement learning at CERN beam lines

Free energy-based reinforcement learning (FERL) with clamped quantum Boltzmann machines (QBM) was shown to significantly improve the learning efficiency compared to classical Q-learning with the restriction, however, to discrete state-action space environments. In this paper, the FERL approach is extended to multi-dimensional continuous state-action space environments to open the doors for a broader range of real-world applications. First, free energy-based Q-learning is studied for discrete action spaces, but continuous state spaces and the impact of experience replay on sample efficiency is assessed. In a second step, a hybrid actor-critic (A-C) scheme for continuous state-action spaces is developed based on the deep deterministic policy gradient algorithm combining a classical actor network with a QBM-based critic. The results obtained with quantum annealing (QA), both simulated and with D-Wave QA hardware, are discussed, and the performance is compared to classical reinforcement learning methods. The environments used throughout represent existing particle accelerator beam lines at the European Organisation for Nuclear Research. Among others, the hybrid A-C agent is evaluated on the actual electron beam line of the Advanced Wakefield Experiment (AWAKE).

Research advanced in the integration of federated learning and reinforcement learning

Reinforcement learning (RL) and federated learning (FL) are two important machine learning paradigms. Reinforcement learning is concerned with enabling intelligence to learn optimal policies when interacting with an environment, while federated learning is concerned with collaboratively training models on distributed equipment while preserving data privacy. In recent years, the fusion and complementarity of reinforcement learning, and federated learning have attracted increasing research interest, providing new directions for the development of the machine learning community. Focusing on the integration of reinforcement learning and federated learning, this paper introduces in detail the latest technological developments in the integration of reinforcement learning and federated learning, and discusses the main challenges, existing methods and future directions of this intersection. Specifically, based on the introduction of classical reinforcement learning and federated learning. In addition, this document introduces cutting-edge results on the integration of reinforcement learning and joint learning and discusses the problems and future directions of the integration.

A Supervised Reinforcement Learning Algorithm for Controlling Drone Hovering

The application of drones carrying different devices for aerial hovering operations is becoming increasingly widespread, but currently there is very little research relying on reinforcement learning methods for hovering control, and it has not been implemented on physical machines. Drone’s behavior space regarding hover control is continuous and large-scale, making it difficult for basic algorithms and value-based reinforcement learning (RL) algorithms to have good results. In response to this issue, this article applies a watcher-actor-critic (WAC) algorithm to the drone’s hover control, which can quickly lock the exploration direction and achieve high robustness of the drone’s hover control while improving learning efficiency and reducing learning costs. This article first utilizes the actor-critic algorithm based on behavioral value Q (QAC) and the deep deterministic policy gradient algorithm (DDPG) for drone hover control learning. Subsequently, an actor-critic algorithm with an added watcher is proposed, in which the watcher uses a PID controller with parameters provided by a neural network as the dynamic monitor, transforming the learning process into supervised learning. Finally, this article uses a classic reinforcement learning environment library, Gym, and a current mainstream reinforcement learning framework, PARL, for simulation, and deploys the algorithm to a practical environment. A multi-sensor fusion strategy-based autonomous localization method for unmanned aerial vehicles is used for practical exercises. The simulation and experimental results show that the training episodes of WAC are reduced by 20% compared to the DDPG and 55% compared to the QAC, and the proposed algorithm has a higher learning efficiency, faster convergence speed, and smoother hovering effect compared to the QAC and DDPG.

GREEN PATH: an expert system for space planning and design by the generation of human trajectories

Public space is usually conceived as where people live, perceive, and interact with other people. The environment affects people in several different ways as well. The impact of environmental problems on humans is significant, affecting all human activities, including health and socio-economic development. Thus, there is a need to rethink how space is used. Dealing with the important needs raised by climate emergency, pandemic and digitization, the contributions of this paper consist in the creation of opportunities for developing generative approaches to space design and utilization. It is proposed GREEN PATH, an intelligent expert system for space planning. GREEN PATH uses human trajectories and deep learning methods to analyse and understand human behaviour for offering insights to layout designers. In particular, a Generative Adversarial Imitation Learning (GAIL) framework hybridised with classical reinforcement learning methods is proposed. An example of the classical reinforcement learning method used is continuous penalties, which allow us to model the shape of the trajectories and insert a bias, which is necessary for the generation, into the training. The structure of the framework and the formalisation of the problem to be solved allow for the evaluation of the results in terms of generation and prediction. The use case is a chosen retail domain that will serve as a demonstrator for optimising the layout environment and improving the shopping experience. Experiments were assessed on shoppers’ trajectories obtained from four different stores, considering two years.

Distributional reinforcement learning in prefrontal cortex

The prefrontal cortex is crucial for learning and decision-making. Classic reinforcement learning (RL) theories center on learning the expectation of potential rewarding outcomes and explain a wealth of neural data in the prefrontal cortex. Distributional RL, on the other hand, learns the full distribution of rewarding outcomes and better explains dopamine responses. In the present study, we show that distributional RL also better explains macaque anterior cingulate cortex neuronal responses, suggesting that it is a common mechanism for reward-guided learning.