Reinforcement Learning System Research Articles

Model-Free and Model-Based Influences in Addiction-Related Behaviors

BackgroundDisruptions in the decision-making processes that guide action selection are a core feature of many psychiatric disorders, including addiction. Decision making is influenced by the goal-directed and habitual systems that can be computationally characterized using model-based and model-free reinforcement learning algorithms, respectively. Recent evidence suggests an imbalance in the influence of these reinforcement learning systems on behavior in individuals with substance dependence, but it is unknown whether these disruptions are a manifestation of chronic drug use and/or are a preexisting risk factor for addiction. MethodsWe trained adult male rats on a multistage decision-making task to quantify model-free and model-based processes before and after self-administration of methamphetamine or saline. ResultsIndividual differences in model-free, but not model-based, learning prior to any drug use predicted subsequent methamphetamine self-administration; rats with lower model-free behavior took more methamphetamine than rats with higher model-free behavior. This relationship was selective to model-free updating following a rewarded, but not unrewarded, choice. Both model-free and model-based learning were reduced in rats following methamphetamine self-administration, which was due to a decrement in the ability of rats to use unrewarded outcomes appropriately. Moreover, the magnitude of drug-induced disruptions in model-free learning was not correlated with disruptions in model-based behavior, indicating that drug self-administration independently altered both reinforcement learning strategies. ConclusionsThese findings provide direct evidence that model-free and model-based learning mechanisms are involved in select aspects of addiction vulnerability and pathology, and they provide a unique behavioral platform for conducting systems-level analyses of decision making in preclinical models of mental illness.

Effective Experiences Collection and State Aggregation in Reinforcement Learning

In reinforcement learning systems, learning agents cluster a large number of experiences by identifying similarities in terms of domain knowledge and replace the groups with a representative prototype. Our method addresses two key challenges in reinforcement learning: the difficulty of transferring continuous state domain into the discrete state space, and the need for a good compromise between exploration and exploitation. To tackle the former challenge, the adaptive state aggregation algorithm with a decision tree helps the agent to learn the optimal policy in continuous state space. For the latter challenge, this paper proposes an adaptive state aggregation algorithm, which uses information entropy to evaluate the probability of state-action-dependent exploration and the value for e is determined using the information entropy instead of manual tuning. Meanwhile, a Tabu search is used to lift the efficiency of exploration. For the planning algorithm, the time required for global exploration depends only on the metric resolution, and not on the size of the state space. The simulation experiments demonstrate the effectiveness of the proposed method, which adaptively partitions the state space into exclusive subspaces and, meanwhile, obtains a good compromise between exploration and exploitation.

Electrophysiological correlates of feedback processing in subarachnoid hemorrhage patients.

Patients with subarachnoid hemorrhage (SAH) secondary to anterior communicating artery (AComA) aneurysm rupture often experience deficits in executive functioning and decision-making. Effective decision-making is based on the subjects' ability to adjust their performance based on feedback processing, ascribing either positive or negative value to the actions performed reinforcing the most adaptive behavior in an appropriate temporal framework. A crucial brain structure associated to feedback processing is the medial prefrontal cortex (mPFC), a brain region frequently damaged after AComA aneurysm rupture. In the present study, we recorded electrophysiological responses (event-related potentials (ERPs') and oscillatory activity (time frequency analysis) during a gambling task in a series of 15 SAH patients. Previous studies have identified a feedback related negativity (FRN) component associated with an increase on frontal medial theta power in response to negative feedback or monetary losses, which is thought to reflect the degree of negative prediction error. Our findings show a decreased FRN component in response to negative feedback and a delayed increase of theta oscillatory activity in the SAH patient group when compared to the healthy controls, indicating a reduced sensitivity to negative feedback processing and an effortful signaling of cognitive control and monitoring processes lengthened in time, respectively. These results provide us with novel neurophysiological markers regarding feedback processing and performance monitoring patterns in SAH patients, illustrating a dysfunctional reinforcement learning system probably contributing to the maladaptive day-to-day functioning in these patients.

Cooperative Deep Q-Learning With Q-Value Transfer for Multi-Intersection Signal Control

The problem of adaptive traffic signal control in the multi-intersection system has attracted the attention of researchers. Among the existing methods, reinforcement learning has shown to be effective. However, the complex intersection features, heterogeneous intersection structures, and dynamic coordination for multiple intersections pose challenges for reinforcement learning-based algorithms. This paper proposes a cooperative deep Q-network with Q-value transfer (QT-CDQN) for adaptive multi-intersection signal control. In QT-CDQN, a multi-intersection traffic network in a region is modeled as a multi-agent reinforcement learning system. Each agent searches the optimal strategy to control an intersection by a deep Q-network that takes the discrete state encoding of traffic information as the network inputs. To work cooperatively, the agent considers the influence of the latest actions of its adjacencies in the process of policy learning. Especially, the optimal Q-values of the neighbor agents at the latest time step are transferred to the loss function of the Q-network. Moreover, the strategy of the target network and the mechanism of experience replay are used to improve the stability of the algorithm. The advantages of QT-CDQN lie not only in the effectiveness and scalability for the multi-intersection system but also in the versatility to deal with the heterogeneous intersection structures. The experimental studies under different road structures show that the QT-CDQN is competitive in terms of average queue length, average speed, and average waiting time when compared with the state-of-the-art algorithms. Furthermore, the experiments of recurring congestion and occasional congestion validate the adaptability of the QT-CDQN to dynamic traffic environments.

Reinforcement Learning for Service Function Chain Reconfiguration in NFV-SDN Metro-Core Optical Networks

With the advent of 5G technology, we are witnessing the development of increasingly bandwidth-hungry network applications, such as enhanced mobile broadband, massive machine-type communications and ultra-reliable low-latency communications. Software Defined Networking (SDN), Network Function Virtualization (NFV) and Network Slicing (NS) are gaining momentum not only in research but also in IT industry representing the drivers of 5G. NS is an approach to network operations allowing the partition of a physical topology into multiple independent virtual networks, called network slices (or slices). Within a single slice, a set of Service Function Chains (SFCs) is defined and the network resources, e.g. bandwidth, can be provisioned dynamically on demand according to specific Quality of Service (QoS) and Service Level Agreement (SLA) requirements. Traditional schemes for network resources provisioning based on static policies may lead to poor resource utilization and suffer from scalability issues. In this article, we investigate the application of Reinforcement Learning (RL) for performing dynamic SFC resources allocation in NFV-SDN enabled metro-core optical networks. RL allows to build a self-learning system able to solve highly complex problems by employing RL agents to learn policies from an evolving network environment. In particular, we build an RL system able to optimize the resources allocation of SFCs in a multi-layer network (packet over flexi-grid optical layer). The RL agent decides if and when to reconfigure the SFCs, given state of the network and historical traffic traces. Numerical simulations show significant advantages of our RL-based optimization over rule-based optimization design.

Deep reinforcement learning using compositional representations for performing instructions

AbstractSpoken language is one of the most efficientways to instruct robots about performing domestic tasks. However, the state of the environment has to be considered to plan and execute actions successfully. We propose a system that learns to recognise the user’s intention and map it to a goal. A reinforcement learning (RL) system then generates a sequence of actions toward this goal considering the state of the environment. A novel contribution in this paper is the use of symbolic representations for both input and output of a neural Deep Q-network (DQN), which enables it to be used in a hybrid system. To show the effectiveness of our approach, the Tell-Me-Dave corpus is used to train an intention detection model and in a second step an RL agent generates the sequences of actions towards the detected objective, represented by a set of state predicates. We show that the system can successfully recognise command sequences fromthis corpus aswell as train the deep- RL network with symbolic input.We further show that the performance can be significantly increased by exploiting the symbolic representation to generate intermediate rewards.

Neurochemical and Behavioral Dissections of Decision-Making in a Rodent Multistage Task

Flexible decision-making in dynamic environments requires both retrospective appraisal of reinforced actions and prospective reasoning about the consequences of actions. These complementary reinforcement-learning systems can be characterized computationally with model-free and model-based algorithms, but how these processes interact at a neurobehavioral level in normal and pathological states is unknown. Here, we developed a translationally analogous multistage decision-making (MSDM) task to independently quantify model-free and model-based behavioral mechanisms in rats. We provide the first direct evidence that male rats, similar to humans, use both model-free and model-based learning when making value-based choices in the MSDM task and provide novel analytic approaches for independently quantifying these reinforcement-learning strategies. Furthermore, we report that ex vivo dopamine tone in the ventral striatum and orbitofrontal cortex correlate with model-based, but not model-free, strategies, indicating that the biological mechanisms mediating decision-making in the multistage task are conserved in rats and humans. This new multistage task provides a unique behavioral platform for conducting systems-level analyses of decision-making in normal and pathological states.SIGNIFICANCE STATEMENT Decision-making is influenced by both a retrospective "model-free" system and a prospective "model-based" system in humans, but the biobehavioral mechanisms mediating these learning systems in normal and disease states are unknown. Here, we describe a translationally analogous multistage decision-making task to provide a behavioral platform for conducting neuroscience studies of decision-making in rats. We provide the first evidence that choice behavior in rats is influenced by model-free and model-based systems and demonstrate that model-based, but not model-free, learning is associated with corticostriatal dopamine tone. This novel behavioral paradigm has the potential to yield critical insights into the mechanisms mediating decision-making alterations in mental disorders.

The Optimal Agent: The Future of Autonomous Vehicles & Liability Theory

Autonomous Vehicles (“AVs”) are rapidly disrupting the $4 trillion auto industry. Indeed, questions surrounding AV regulation are some of the most important to be answered in the Twenty-First Century. Yet, legislators have yet to address or even identify some of the most critical issues relating to AV regulation. This paper explains the unique issues that deep reinforcement learning systems pose for AV technology, policy, and law. Additionally, this paper identifies two important regulatory problems that legislators and scholars need to address in the context of AV development. Legal scholars have made clear that there is a demanding need for some sort of regulatory system for AVs. However, those arguments focus on short term regulation and generally misunderstand the evolutionary rate of AV technology. This paper takes an informatics based approach to analyzing issues in AV regulation with a specific emphasis on the technical aspects of AV systems. Further, this paper discusses and explains the formal models that are currently being used as a foundation for AV development. Ultimately, AV technology will change the way humans move throughout the world and legislators must prepare immediately for the endeavor ahead in regulating AVs.

The Perils & Promises of Artificial General Intelligence

Artificial General Intelligence (“AGI”) - an Artificial Intelligence (AI) capable of achieving any goal - is the greatest existential threat humanity faces. Indeed, the questions surrounding the regulation of AGI are the most important the millennial generation will answer. The capabilities of current AI systems are evolving at accelerating rates. Yet, legislators and scholars have yet to address or identify critical issues relating to AI regulation. Instead, legislators and scholars have focused narrowly on short term AI policy. This paper takes a contrarian approach to analyzing AI regulation with a specific emphasis on deep reinforcement learning systems, a relatively recent breakthrough in AI technology. Additionally, this paper identifies three important regulatory issues legislators and scholars need to address in the context of AI development. AI and legal scholars have made the demanding need for an AI regulatory system clear. However, those arguments focus on the regulation of current AI systems and generally ignore or dismiss the possibility of AGI. Further, previous scholarship has yet to grapple specifically with the regulation of deep reinforcement learning systems, which many AI scholars argue provides a direct path to AGI. Ultimately, legislators must consider and address the perils and promises of AGI when developing and evolving AI regulatory frameworks.

High-intensity interval exercise impairs neuroelectric indices of reinforcement-learning

A single bout of high-intensity interval exercise (HIIE) improves behavioural measures of cognitive function; however, investigations using event-related potentials (ERPs) to examine the systems that underlie these cognitive improvements are lacking. The reward positivity is a positive-going ERP component that indexes reward processing evoked by ‘win’ feedback and is a candidate marker of an underlying human reinforcement learning system. While HIIE improves behavioural measures of learning, it is unknown how HIIE affects the amplitude of the reward positivity. Therefore, the purpose of this study was to investigate how HIIE affects reward positivity amplitude in response to reward feedback in university students. Using a single-group randomly assigned counterbalance crossover design, 25 healthy university students performed HIIE and control visits on separate days. Electroencephalographic data was recorded before (pre-intervention) and 10 min after (post-intervention) the intervention period while participants played a novel gambling task. The HIIE intervention consisted of 4 separate body-weight exercises totaling 11 min in duration, including rest. The control visit intervention consisted of quietly watching a nature documentary for 11 min. Heart rate (HR) was measured at the same time intervals in both trials. Analysis revealed that HIIE significantly diminished the amplitude of the reward positivity whereas it remained unaffected in the control condition. HR was significantly higher following HIIE compared to control during post-intervention testing. These findings suggest that mechanisms of reinforcement learning are impaired shortly after HIIE cessation, possibly due to persistent, suboptimal arousal as evidenced by elevated HR post-HIIE.

Two-phase selective decentralization to improve reinforcement learning systems with MDP

Two-phase selective decentralization to improve reinforcement learning systems with MDP

Model-free and model-based reward prediction errors in EEG

Learning theorists posit two reinforcement learning systems: model-free and model-based. Model-based learning incorporates knowledge about structure and contingencies in the world to assign candidate actions with an expected value. Model-free learning is ignorant of the world's structure; instead, actions hold a value based on prior reinforcement, with this value updated by expectancy violation in the form of a reward prediction error. Because they use such different learning mechanisms, it has been previously assumed that model-based and model-free learning are computationally dissociated in the brain. However, recent fMRI evidence suggests that the brain may compute reward prediction errors to both model-free and model-based estimates of value, signalling the possibility that these systems interact. Because of its poor temporal resolution, fMRI risks confounding reward prediction errors with other feedback-related neural activity. In the present study, EEG was used to show the presence of both model-based and model-free reward prediction errors and their place in a temporal sequence of events including state prediction errors and action value updates. This demonstration of model-based prediction errors questions a long-held assumption that model-free and model-based learning are dissociated in the brain.

Increased conflict-induced slowing, but no differences in conflict-induced positive or negative prediction error learning in patients with schizophrenia

People with schizophrenia (PSZ) often fail to pursue rewarding activities despite largely intact in-the-moment hedonic experiences. Deficits in effort-based decision making in PSZ may be related to enhanced effects of cost or reduced reward, i.e., through the amplification of negative prediction errors or by dampened positive prediction errors (here, positive and negative prediction errors refer to outcomes that are better or worse than expected respectively). We administered a modified Simon task to people with schizophrenia (PSZ; N = 46) and healthy controls (N = 32). The modification included a reinforcement learning component, where positive and negative prediction errors are dampened or boosted through the use of cognitively-effortful response conflict. EEG was recorded concurrently to investigate potential differences in conflict enhanced mid-frontal theta power between PSZ and controls. We found an enhanced effect of response conflict on response time in people with schizophrenia, but no discernible difference in conflict processing as reflected by the lack of a difference in theta-power enhancement to conflict in mid-frontal regions. Using the reinforcement learning transfer phase of the modified Simon task, PSZ also showed clear deficits in selecting the most rewarding stimulus during the 'easy' (most discriminable in terms of value) stimulus contrasts. However, we failed to find a difference between patients and controls in their gain or avoidance learning bias, nor did these biases correlate with negative symptoms. Previous studies had failed to find significant conflict effects on the Simon task likely due to its modest effect size. Our results show that PSZ do indeed possess subtle impairments in response-conflict, suggesting an increase in cognitive effort required for appropriate responding. In addition, while the lack of an overt positive or negative prediction error bias (i.e., a bias towards punishment or reward learning) was unexpected, it is consistent with recent work showing intact estimation of value when the reinforcement learning system is isolated from other contributors to value learning.

Credit Assignment in a Motor Decision Making Task Is Influenced by Agency and Not Sensory Prediction Errors.

Failures to obtain reward can occur from errors in action selection or action execution. Recently, we observed marked differences in choice behavior when the failure to obtain a reward was attributed to errors in action execution compared with errors in action selection (McDougle et al., 2016). Specifically, participants appeared to solve this credit assignment problem by discounting outcomes in which the absence of reward was attributed to errors in action execution. Building on recent evidence indicating relatively direct communication between the cerebellum and basal ganglia, we hypothesized that cerebellar-dependent sensory prediction errors (SPEs), a signal indicating execution failure, could attenuate value updating within a basal ganglia-dependent reinforcement learning system. Here we compared the SPE hypothesis to an alternative, "top-down" hypothesis in which changes in choice behavior reflect participants' sense of agency. In two experiments with male and female human participants, we manipulated the strength of SPEs, along with the participants' sense of agency in the second experiment. The results showed that, whereas the strength of SPE had no effect on choice behavior, participants were much more likely to discount the absence of rewards under conditions in which they believed the reward outcome depended on their ability to produce accurate movements. These results provide strong evidence that SPEs do not directly influence reinforcement learning. Instead, a participant's sense of agency appears to play a significant role in modulating choice behavior when unexpected outcomes can arise from errors in action execution.SIGNIFICANCE STATEMENT When learning from the outcome of actions, the brain faces a credit assignment problem: Failures of reward can be attributed to poor choice selection or poor action execution. Here, we test a specific hypothesis that execution errors are implicitly signaled by cerebellar-based sensory prediction errors. We evaluate this hypothesis and compare it with a more "top-down" hypothesis in which the modulation of choice behavior from execution errors reflects participants' sense of agency. We find that sensory prediction errors have no significant effect on reinforcement learning. Instead, instructions influencing participants' belief of causal outcomes appear to be the main factor influencing their choice behavior.

Contribution of explicit processes to reinforcement-based motor learning.

Despite increasing interest in the role of reward in motor learning, the underlying mechanisms remain ill defined. In particular, the contribution of explicit processes to reward-based motor learning is unclear. To address this, we examined subjects' ( n = 30) ability to learn to compensate for a gradually introduced 25° visuomotor rotation with only reward-based feedback (binary success/failure). Only two-thirds of subjects ( n = 20) were successful at the maximum angle. The remaining subjects initially followed the rotation but after a variable number of trials began to reach at an insufficiently large angle and subsequently returned to near-baseline performance ( n = 10). Furthermore, those who were successful accomplished this via a large explicit component, evidenced by a reduction in reach angle when they were asked to remove any strategy they employed. However, both groups displayed a small degree of remaining retention even after the removal of this explicit component. All subjects made greater and more variable changes in reach angle after incorrect (unrewarded) trials. However, subjects who failed to learn showed decreased sensitivity to errors, even in the initial period in which they followed the rotation, a pattern previously found in parkinsonian patients. In a second experiment, the addition of a secondary mental rotation task completely abolished learning ( n = 10), while a control group replicated the results of the first experiment ( n = 10). These results emphasize a pivotal role of explicit processes during reinforcement-based motor learning, and the susceptibility of this form of learning to disruption has important implications for its potential therapeutic benefits. NEW & NOTEWORTHY We demonstrate that learning a visuomotor rotation with only reward-based feedback is principally accomplished via the development of a large explicit component. Furthermore, this form of learning is susceptible to disruption with a secondary task. The results suggest that future experiments utilizing reward-based feedback should aim to dissect the roles of implicit and explicit reinforcement learning systems. Therapeutic motor learning approaches based on reward should be aware of the sensitivity to disruption.

Within- and across-trial dynamics of human EEG reveal cooperative interplay between reinforcement learning and working memory

Learning from rewards and punishments is essential to survival and facilitates flexible human behavior. It is widely appreciated that multiple cognitive and reinforcement learning systems contribute to decision-making, but the nature of their interactions is elusive. Here, we leverage methods for extracting trial-by-trial indices of reinforcement learning (RL) and working memory (WM) in human electro-encephalography to reveal single-trial computations beyond that afforded by behavior alone. Neural dynamics confirmed that increases in neural expectation were predictive of reduced neural surprise in the following feedback period, supporting central tenets of RL models. Within- and cross-trial dynamics revealed a cooperative interplay between systems for learning, in which WM contributes expectations to guide RL, despite competition between systems during choice. Together, these results provide a deeper understanding of how multiple neural systems interact for learning and decision-making and facilitate analysis of their disruption in clinical populations.

Reinforcement learning and convolutional neural network system for firefighting rescue robot

In this paper, we combine the machine learning and neural network to build some modules for the fire rescue robot application. In our research, we build the robot legs module with Q-learning. We also finish the face detection with color sensors and infrared sensors. It is usual that image fusion is done when we want to use two kinds of sensors. Kalman filter is chosen to meet our requirement. After we finish some indispensable steps, we use sliding windows to choose our region of interest to make the system’s calculation lower. The least step is convolutional neural network. We design a seven layers neural network to find the face feature and distinguish it or not.

Deep reinforcement learning based finite-horizon optimal tracking control for nonlinear system

Abstract Reinforcement learning (RL) can be used to obtain an approximate numerical solution to the Hamilton-Jacobi-Bellman (HJB) equation. Recent advances in machine learning community enable the use of deep neural networks (DNNs) to approximate high-dimensional nonlinear functions as those that occur in RL, accurately without any domain knowledge. In the standard RL setting, both system and cost structures are unknown, and the amount of data needed to obtain an accurate approximation can be impractically large. Meanwhile, when the structures are known, they can be used to solve the HJB equation efficiently. Herein, the model-based globalized dual heuristic programming (GDHP) is proposed, in which the HJB equation is separated into value, costate, and policy functions. A particular class of interest in this research is finite horizon optimal tracking control (FHOC) problem. Additional issues that arise, such as time-varying functions, terminal constraints, and delta-input formulation, are addressed in the context of FHOC. The DNN structure and training algorithm suitable for FHOC are presented. A benchmark continuous reactor example is provided to illustrate the proposed approach.

Encouraging Physical Activity in Patients With Diabetes: Intervention Using a Reinforcement Learning System

BackgroundRegular physical activity is known to be beneficial for people with type 2 diabetes. Nevertheless, most of the people who have diabetes lead a sedentary lifestyle. Smartphones create new possibilities for helping people to adhere to their physical activity goals through continuous monitoring and communication, coupled with personalized feedback.ObjectiveThe aim of this study was to help type 2 diabetes patients increase the level of their physical activity.MethodsWe provided 27 sedentary type 2 diabetes patients with a smartphone-based pedometer and a personal plan for physical activity. Patients were sent short message service messages to encourage physical activity between once a day and once per week. Messages were personalized through a Reinforcement Learning algorithm so as to improve each participant’s compliance with the activity regimen. The algorithm was compared with a static policy for sending messages and weekly reminders.ResultsOur results show that participants who received messages generated by the learning algorithm increased the amount of activity and pace of walking, whereas the control group patients did not. Patients assigned to the learning algorithm group experienced a superior reduction in blood glucose levels (glycated hemoglobin [HbA1c]) compared with control policies, and longer participation caused greater reductions in blood glucose levels. The learning algorithm improved gradually in predicting which messages would lead participants to exercise.ConclusionsMobile phone apps coupled with a learning algorithm can improve adherence to exercise in diabetic patients. This algorithm can be used in large populations of diabetic patients to improve health and glycemic control. Our results can be expanded to other areas where computer-led health coaching of humans may have a positive impact. Summary of a part of this manuscript has been previously published as a letter in Diabetes Care, 2016.

Cost-Benefit Arbitration Between Multiple Reinforcement-Learning Systems

Human behavior is sometimes determined by habit and other times by goal-directed planning. Modern reinforcement-learning theories formalize this distinction as a competition between a computationally cheap but inaccurate model-free system that gives rise to habits and a computationally expensive but accurate model-based system that implements planning. It is unclear, however, how people choose to allocate control between these systems. Here, we propose that arbitration occurs by comparing each system’s task-specific costs and benefits. To investigate this proposal, we conducted two experiments showing that people increase model-based control when it achieves greater accuracy than model-free control, and especially when the rewards of accurate performance are amplified. In contrast, they are insensitive to reward amplification when model-based and model-free control yield equivalent accuracy. This suggests that humans adaptively balance habitual and planned action through on-line cost-benefit analysis.