Reinforcement Learning Theory Research Articles

Inferring source of learning by chimpanzees in cognitive tasks using reinforcement learning theory

Inferring source of learning by chimpanzees in cognitive tasks using reinforcement learning theory

Analysis and Implementation of Web Advertisement Optimization Using UCB Reinforcement Learning

Web advertisements are vital in digital marketing for connecting businesses with customers; yet optimizing their placement faces challenges from evolving user preferences. This project explores leveraging the Upper Confidence Bound (UCB) algorithm, a reinforcement learning technique, to automate ad placement strategies based on real-time feedback. Through analysis and implementation, this study investigates UCB's potential for web advertisement optimization. It examines existing methods, delves into reinforcement learning theory, and develops a simulation environment reflecting real-world dynamics. Experimental results comparing the UCB approach against baselines demonstrate its efficacy in improving ad placement and enhancing online advertising campaigns.

Bayesian Reinforcement Learning With Limited Cognitive Load.

All biological and artificial agents must act given limits on their ability to acquire and process information. As such, a general theory of adaptive behavior should be able to account for the complex interactions between an agent's learning history, decisions, and capacity constraints. Recent work in computer science has begun to clarify the principles that shape these dynamics by bridging ideas from reinforcement learning, Bayesian decision-making, and rate-distortion theory. This body of work provides an account of capacity-limited Bayesian reinforcement learning, a unifying normative framework for modeling the effect of processing constraints on learning and action selection. Here, we provide an accessible review of recent algorithms and theoretical results in this setting, paying special attention to how these ideas can be applied to studying questions in the cognitive and behavioral sciences.

Neural Network Approximation for Pessimistic Offline Reinforcement Learning

Deep reinforcement learning (RL) has shown remarkable success in specific offline decision-making scenarios, yet its theoretical guarantees are still under development. Existing works on offline RL theory primarily emphasize a few trivial settings, such as linear MDP or general function approximation with strong assumptions and independent data, which lack guidance for practical use. The coupling of deep learning and Bellman residuals makes this problem challenging, in addition to the difficulty of data dependence. In this paper, we establish a non-asymptotic estimation error of pessimistic offline RL using general neural network approximation with C-mixing data regarding the structure of networks, the dimension of datasets, and the concentrability of data coverage, under mild assumptions. Our result shows that the estimation error consists of two parts: the first converges to zero at a desired rate on the sample size with partially controllable concentrability, and the second becomes negligible if the residual constraint is tight. This result demonstrates the explicit efficiency of deep adversarial offline RL frameworks. We utilize the empirical process tool for C-mixing sequences and the neural network approximation theory for the Holder class to achieve this. We also develop methods to bound the Bellman estimation error caused by function approximation with empirical Bellman constraint perturbations. Additionally, we present a result that lessens the curse of dimensionality using data with low intrinsic dimensionality and function classes with low complexity. Our estimation provides valuable insights into the development of deep offline RL and guidance for algorithm model design.

A Comprehensive Analysis of Game theory on Multi-Agent Reinforcement

In recent years, reinforcement learning has gradually emerged as a popular research field in artificial intelligence. However, for multi-agent systems, due to their complex environment and an abundance of agents, the learning target of maximizing the anticipated value of cumulative rewards for specific agents frequently fails to converge. Introducing game theory into reinforcement learning can effectively address the interactions among intelligent agents, provide a rationale for convergence points corresponding to strategies. To address the issue of non-existence of optimal solutions for certain tasks in multi-agent settings, this paper takes a game-theoretic perspective and summarizes classical reinforcement learning algorithms developed in recent years. The basic theory of multi-agent reinforcement learning, essential game theory, categorization of reinforcement learning using multiple agents game strategies, primary worries are addressed in the study. It also analyzes the challenges that game-theoretic multi-agent reinforcement learning algorithms may encounter in the future, along with the relevant optimization directions.

Efficient energy management in smart grids with finite horizon Q-learning

Efficient energy distribution in smart grids is an important problem driven by the need to manage increasing power consumption across the globe. This problem has been studied in the past using different frameworks including Markov Decision Processes (MDP) and Reinforcement Learning. However, existing algorithms in Reinforcement Learning Theory largely deal with infinite horizon decision making. On the other hand, smart grid problems are inherently finite horizon in nature and so can best be addressed using Finite Horizon algorithms. We therefore analyze the smart grid setup using a finite horizon MDP model and develop, for the first time, a finite horizon version of the popular Q-learning algorithm. We observe that our algorithm shows good empirical performance on the smart grid setting. We also theoretically analyze the full convergence of the algorithm by proving both the stability and the convergence of the same. Our analysis of stability and convergence of finite horizon Q-learning is based entirely on the ordinary differential equations (O.D.E) method. Apart from smart grids, we additionally demonstrate the performance of our algorithm on a setting of random MDPs indicating that the algorithm is more generally applicable and can be studied on other settings in the future.

Distributional reinforcement learning in prefrontal cortex

The prefrontal cortex is crucial for learning and decision-making. Classic reinforcement learning (RL) theories center on learning the expectation of potential rewarding outcomes and explain a wealth of neural data in the prefrontal cortex. Distributional RL, on the other hand, learns the full distribution of rewarding outcomes and better explains dopamine responses. In the present study, we show that distributional RL also better explains macaque anterior cingulate cortex neuronal responses, suggesting that it is a common mechanism for reward-guided learning.

Brain Emotion Perception Inspired EEG Emotion Recognition With Deep Reinforcement Learning.

Inspired by the well-known Papez circuit theory and neuroscience knowledge of reinforcement learning, a double dueling deep Q network (DQN) is built incorporating the electroencephalogram (EEG) signals of the frontal lobe as prior information, which is named frontal lobe double dueling DQN (FLD3QN). The framework of FLD3QN is constructed in accord with the brain emotion mechanism which takes the frontal lobe and the thalamus as the core, in which the part of the Papez circuit is simulated by the bifrontal lobe residual convolution neural network (BiFRCNN). Moreover, a step penalty factor is designed to constrain the number of mistakes of the agent. The ablation studies results on the public EEG emotion dataset DEAP verified the important roles of the frontal lobe and the Papez circuit in modeling the procedure of learning rewards during the perception of emotions, with a great increase in the average accuracies by 25.24% and 23.31% in valence and arousal dimensions.

Data-Driven Optimal Bipartite Consensus Control for Second-Order Multiagent Systems via Policy Gradient Reinforcement Learning.

This article investigates the optimal bipartite consensus control (OBCC) problem for unknown second-order discrete-time multiagent systems (MASs). First, the coopetition network is constructed to describe the cooperative and competitive relationships between agents, and the OBCC problem is proposed by the tracking error and related performance index function. Based on the distributed policy gradient reinforcement learning (RL) theory, a data-driven distributed optimal control strategy is obtained to guarantee the bipartite consensus of all agents' position and velocity states. In addition, the offline data sets ensure the learning efficiency of the system. These data sets are generated by running the system in real time. Besides, the designed algorithm is an asynchronous version, which is essential to solve the challenge caused by the computational ability difference between nodes in MASs. Then, by means of the functional analysis and Lyapunov theory, the stability of the proposed MASs and the convergence of the learning process are analyzed. Furthermore, an actor-critic structure containing two neural networks is used to implement the proposed methods. Finally, a numerical simulation shows the effectiveness and validity of the results.

Distributed Neural Learning Algorithms for Multiagent Reinforcement Learning

In this paper, the fully distributed neural learning algorithms by neural network approximation for networked multi-agent reinforcement learning (NMARL) are studied. To tackle the convergence analysis of methods in NMARL with tremendous state-action space, most of the existing distributed algorithms are designed by linear function approximation, which however would fall into a situation of poor expression. To conquer such limitation, the distributed neural learning algorithms are developed by using a novel neural network approximation that bridges the theory and practice of deep multi-agent reinforcement learning (DMARL). Specifically, inspired by the overparametrization method for minimizing mean-squared projected bellman error (MSPBE), the distributed neural learning algorithms with population semigradients and stochastic semigradients are respectively proposed to solve the NMARL problem. Furthermore, the convergence of the proposed algorithms are strictly given by employing the overparametrization method to establish the approximate stationary point of MSPBE to characterizes the algorithms toward the global optimum. Finally, some numerical simulations demonstrate the effectiveness of the distributed neural learning algorithms.

Research on Anthropomorphic Obstacle Avoidance Trajectory Planning for Adaptive Driving Scenarios Based on Inverse Reinforcement Learning Theory

The forward design of trajectory planning strategies requires preset trajectory optimization functions, resulting in poor adaptability of the strategy and an inability to accurately generate obstacle avoidance trajectories that conform to real driver behavior habits. In addition, owing to the strong time-varying dynamic characteristics of obstacle avoidance scenarios, it is necessary to design numerous trajectory optimization functions and adjust the corresponding parameters. Therefore, an anthropomorphic obstacle-avoidance trajectory planning strategy for adaptive driving scenarios is proposed. First, numerous expert-demonstrated trajectories are extracted from the HighD natural driving dataset. Subsequently, a trajectory expectation feature-matching algorithm is proposed that uses maximum entropy inverse reinforcement learning theory to learn the extracted expert-demonstrated trajectories and achieve automatic acquisition of the optimization function of the expert-demonstrated trajectory. Furthermore, a mapping model is constructed by combining the key driving scenario information that affects vehicle obstacle avoidance with the weight of the optimization function, and an anthropomorphic obstacle avoidance trajectory planning strategy for adaptive driving scenarios is proposed. Finally, the proposed strategy is verified based on real driving scenarios. The results show that the strategy can adjust the weight distribution of the trajectory optimization function in real time according to the “emergency degree” of obstacle avoidance and the state of the vehicle. Moreover, this strategy can generate anthropomorphic trajectories that are similar to expert-demonstrated trajectories, effectively improving the adaptability and acceptability of trajectories in driving scenarios.

Applying real options with reinforcement learning to assess commercial CCU deployment

Carbon capture and utilization (CCU), which emerged as a means to reduce anthropogenic carbon emissions, has been highlighted to close the carbon cycle and combat climate change. CCU involves utilizing or converting captured CO2 to create value-added products that can replace or supplement fossil fuel-derived products. In order to meet climate goals, commercial-scale CCU facilities need to be built and their capacities increased, but barriers to large-scale CCU deployment still exist, primarily caused by large uncertainties in product/energy prices, markets, technologies, and policies. Conventional techno-economic analysis (TEA) methods cannot appropriately assess viability of CCU deployment projects which include dynamic uncertainties and deployment strategies. More flexible methods that can account for both time-varying uncertainties and dynamic capacity building are needed. We propose a systematic framework for the evaluation of commercial CCU deployment using the theory of real options and reinforcement learning (RL). RL is needed as considering options of adding capacities or delaying/abandoning the project at multiple time points under dynamic uncertainties lead to a large-scale stochastic optimal control problem which cannot be solved using conventional optimization methods like stochastic programming. The framework consists of three steps: surrogate modeling, uncertainty modeling, and assessment modeling. The framework is dependent on the type of CCU technology, uncertainties, real options and RL algorithm. We demonstrate the application of the framework through a case study of CO2 hydrogenation-to-methanol process in Europe, which is a late-stage, i.e., high technology readiness level (TRL), technology.

Design of anti‐jamming decision‐making for cognitive radar

AbstractWith the development of electronic warfare, anti‐jamming measure becomes more and more complex. There have been certain research results on jamming strategies, but only a few research materials on anti‐jamming strategies. It is difficult to simulate the real jamming environment, and there is no appropriate anti‐jamming decision‐making model for research. Cognitive radar can perceive the environment and receive feedback, which provides the possibility to solve the problem of anti‐jamming decision‐making. This article regards the anti‐jamming measure as a kind of interaction behaviour and establishes the cognitive radar antagonistic environment model and uses the reinforcement learning algorithm to solve the problem of anti‐jamming decision‐making. Finally, this article verifies the feasibility of applying reinforcement learning theory on making anti‐jamming decision in the radar antagonistic environment model. The performance of different reinforcement learning algorithms is compared, and their advantages and disadvantages are discussed.

Mean-Field Learning for Edge Computing in Mobile Blockchain Networks

Blockchain has been leveraged to secure transactions for the m-commerce. However, the intensive computation in the mining process restricts the participation of mobile devices. Currently, some studies have deployed edge computing services to support the mining process, where edge servers managed by one Service Provider (SP) are considered. This paper investigates a more practical scenario with multiple SPs, where servers managed by different SPs have distinct capacities and prices, making miners’ offloading decisions rather complicated. To tackle the above challenges, we consider task offloading, block propagation and miner mobility comprehensively to maximize utilities of miners. Specifically, we first formulate a Markov game, and then design a learning-based offloading algorithm for off-chain computation, where a novel learning model is constructed by integrating Deep Reinforcement Learning (DRL) and Mean Field Theory (MFT) to guarantee a Nash equilibrium. Different from existing studies, each miner merely needs to respond to the average effect from others in our system, insteading of knowing policies of others. Finally, both theoritical and performance results show that our designed algorithm has superiority on average miner utilities and algorithm convergence time compared with other representative algorithms.

A novel intelligent collision avoidance algorithm based on deep reinforcement learning approach for USV

Enhancing the efficiency of unmanned surface vehicles (USVs) collision avoidance can yield a significant impact, as it can result in safer navigation and lower energy consumption. This paper introduces a robust approach employing deep reinforcement learning theory to facilitate informed collision avoidance decisions within intricate maritime environments. The restrictions on USV maneuverability and international regulations for preventing collisions at sea are studied and quantified, particularly focusing on the shape and size changes of the ship’s domain caused by USV speed. Based on the deep Q network, an improved methodology is designed, incorporating a noisy network, prioritized experience replay, dueling neural network architecture, and double Q learning, resulting in a highly efficient sampling, exploration, and learning process. To curtail computational expenses associated with USVs, a novel dynamic area restriction technique is proposed. Furthermore, an innovative USV state clipping method is introduced to mitigate training complexities. By utilizing the Unity platform, a virtual environment characterized by complexity and stochasticity is constructed for training and testing the collision avoidance of USVs This novel approach surpasses the performance of the pre-improvement algorithm across multiple collision avoidance effectiveness indicators and performance metrics.

Reward expectation enhances action-related activity of nigral dopaminergic and two striatal output pathways

Neurons comprising nigrostriatal system play important roles in action selection. However, it remains unclear how this system integrates recent outcome information with current action (movement) and outcome (reward or no reward) information to achieve appropriate subsequent action. We examined how neuronal activity of substantia nigra pars compacta (SNc) and dorsal striatum reflects the level of reward expectation from recent outcomes in rats performing a reward-based choice task. Movement-related activity of direct and indirect pathway striatal projection neurons (dSPNs and iSPNs, respectively) were enhanced by reward expectation, similarly to the SNc dopaminergic neurons, in both medial and lateral nigrostriatal projections. Given the classical basal ganglia model wherein dopamine stimulates dSPNs and suppresses iSPNs through distinct dopamine receptors, dopamine might not be the primary driver of iSPN activity increasing following higher reward expectation. In contrast, outcome-related activity was affected by reward expectation in line with the classical model and reinforcement learning theory, suggesting purposive effects of reward expectation.

Value iteration for streaming data on a continuous space with gradient method in an RKHS

The classical theory of reinforcement learning focused on the tabular setting when states and actions are finite, or for linear representation of the value function in a finite-dimensional approximation. Establishing theory on general continuous state and action space requires a careful treatment of complexity theory of appropriately chosen function spaces and the iterative update of the value function when stochastic gradient descent (SGD) is used. For the classical prediction problem in reinforcement learning based on i.i.d. streaming data in the framework of reproducing kernel Hilbert spaces, we establish polynomial sample complexity taking into account the smoothness of the value function. In particular, we prove that the gradient descent algorithm efficiently computes the value function with appropriately chosen step sizes, with a convergence rate that can be close to 1/N, which is the best possible rate for parametric SGD. The advantages of using the gradient descent algorithm include its computational convenience and it can naturally deal with streaming data.

Poor learning or hyper-exploration?: A commentary on Hulsbosch etal. (2023).

In an outstanding paper, Hulsbosch etal. (Journal of Child Psychology and Psychiatry, 2023) tested major reinforcement learning theories of attention-deficit/hyperactivity disorder (ADHD). On a reinforcement learning task, children with ADHD needed more trials to reach the criterion in the learning phase and chose the target less often in the extinction phase. These results only partially support either of the theories and may be interpreted as reflecting a general deficit in learning. Still, distinguishing between learning and decision-making stages suggests that poor performance on reinforcement learning tasks by children with ADHD is accounted for by increased exploration or decreased choice consistency rather than deficient learning.

Intrinsic rewards explain context-sensitive valuation in reinforcement learning.

When observing the outcome of a choice, people are sensitive to the choice's context, such that the experienced value of an option depends on the alternatives: getting $1 when the possibilities were 0 or 1 feels much better than when the possibilities were 1 or 10. Context-sensitive valuation has been documented within reinforcement learning (RL) tasks, in which values are learned from experience through trial and error. Range adaptation, wherein options are rescaled according to the range of values yielded by available options, has been proposed to account for this phenomenon. However, we propose that other mechanisms-reflecting a different theoretical viewpoint-may also explain this phenomenon. Specifically, we theorize that internally defined goals play a crucial role in shaping the subjective value attributed to any given option. Motivated by this theory, we develop a new "intrinsically enhanced" RL model, which combines extrinsically provided rewards with internally generated signals of goal achievement as a teaching signal. Across 7 different studies (including previously published data sets as well as a novel, preregistered experiment with replication and control studies), we show that the intrinsically enhanced model can explain context-sensitive valuation as well as, or better than, range adaptation. Our findings indicate a more prominent role of intrinsic, goal-dependent rewards than previously recognized within formal models of human RL. By integrating internally generated signals of reward, standard RL theories should better account for human behavior, including context-sensitive valuation and beyond.

A Method to Plan the Path of a Robot Utilizing Deep Reinforcement Learning and Multi-Sensory Information Fusion

ABSTRACT Nowadays, mobile robots are being widely employed in various settings, including factories, homes, and everyday tasks. Achieving successful implementation of autonomous robot movement largely depends on effective route planning. Therefore, it is not surprising that there is a growing trend in studying and improving the intelligence of this technology. Deep reinforcement learning has shown remarkable performance in decision-making problems and can be effectively utilized to address path planning challenges faced by mobile robots. This manuscript focuses on investigating path planning problems using deep reinforcement learning and multi-sensing information fusion technology. The manuscript elaborates on the significance of path planning, providing comprehensive research encompassing path planning algorithms, deep reinforcement learning, and multi-sensing information fusion. Also, the fundamental theory of deep reinforcement learning is introduced, followed by the design of a multimodal perception module based on image and lidar. A semantic segmentation approach is employed to bridge the gap between simulated and real environments. To enhance strategy, a lightweight multimodal data fusion network model is carefully developed, incorporating modality separation learning. Overall, in this paper, we explore the use of a deep reinforcement learning architecture for conducting path planning experiments with mobile robots. The results obtained from these experiments demonstrate promising outcomes.