Reinforcement Learning Parameters Research Articles

Regulation of reinforcement learning parameters captures long-term changes in rat behaviour.

In uncertain environments in which resources fluctuate continuously, animals must permanently decide whether to stabilise learning and exploit what they currently believe to be their best option, or instead explore potential alternatives and learn fast from new observations. While such a trade-off has been extensively studied in pretrained animals facing non-stationary decision-making tasks, it is yet unknown how they progressively tune it while learning the task structure during pretraining. Here, we compared the ability of different computational models to account for long-term changes in the behaviour of 24 rats while they learned to choose a rewarded lever in a three-armed bandit task across 24 days of pretraining. We found that the day-by-day evolution of rat performance and win-shift tendency revealed a progressive stabilisation of the way they regulated reinforcement learning parameters. We successfully captured these behavioural adaptations using a meta-learning model in which either the learning rate or the inverse temperature was controlled by the average reward rate.

Tuning Reinforcement Learning Parameters for Cluster Selection to Enhance Evolutionary Algorithms.

The ability to find optimal molecular structures with desired properties is a popular challenge, with applications in areas such as drug discovery. Genetic algorithms are a common approach to global minima molecular searches due to their ability to search large regions of the energy landscape and decrease computational time via parallelization. In order to decrease the amount of unstable intermediate structures being produced and increase the overall efficiency of an evolutionary algorithm, clustering was introduced in multiple instances. However, there is little literature detailing the effects of differentiating the selection frequencies between clusters. In order to find a balance between exploration and exploitation in our genetic algorithm, we propose a system of clustering the starting population and choosing clusters for an evolutionary algorithm run via a dynamic probability that is dependent on the fitness of molecules generated by each cluster. We define four parameters, MFavOvrAll-A, MFavClus-B, NoNewFavClus-C, and Select-D, that correspond to a reward for producing the best structure overall, a reward for producing the best structure in its own cluster, a penalty for not producing the best structure, and a penalty based on the selection ratio of the cluster, respectively. A reward increases the probability of a cluster's future selection, while a penalty decreases it. In order to optimize these four parameters, we used a Gaussian distribution to approximate the evolutionary algorithm performance of each cluster and performed a grid search for different parameter combinations. Results show parameter MFavOvrAll-A (rewarding clusters for producing the best structure overall) and parameter Select-D (appearance penalty) have a significantly larger effect than parameters MFavClus-B and NoNewFavClus-C. In order to produce the most successful models, a balance between MFavOvrAll-A and Select-D must be made that reflects the exploitation vs exploration trade-off often seen in reinforcement learning algorithms. Results show that our reinforcement-learning-based method for selecting clusters outperforms an unclustered evolutionary algorithm for quinoline-like structure searches.

5-HT 2A and 5-HT 2C receptor antagonism differentially modulate reinforcement learning and cognitive flexibility: behavioural and computational evidence

RationaleCognitive flexibility, the ability to adapt behaviour in response to a changing environment, is disrupted in several neuropsychiatric disorders, including obsessive–compulsive disorder and major depressive disorder. Evidence suggests that flexibility, which can be operationalised using reversal learning tasks, is modulated by serotonergic transmission. However, how exactly flexible behaviour and associated reinforcement learning (RL) processes are modulated by 5-HT action on specific receptors is unknown.ObjectivesWe investigated the effects of 5-HT2A receptor (5-HT2AR) and 5-HT2C receptor (5-HT2CR) antagonism on flexibility and underlying RL mechanisms.MethodsThirty-six male Lister hooded rats were trained on a touchscreen visual discrimination and reversal task. We evaluated the effects of systemic treatments with the 5-HT2AR and 5-HT2CR antagonists M100907 and SB-242084, respectively, on reversal learning and performance on probe trials where correct and incorrect stimuli were presented with a third, probabilistically rewarded, stimulus. Computational models were fitted to task choice data to extract RL parameters, including a novel model designed specifically for this task.Results5-HT2AR antagonism impaired reversal learning only after an initial perseverative phase, during a period of random choice and then new learning. 5-HT2CR antagonism, on the other hand, impaired learning from positive feedback. RL models further differentiated these effects. 5-HT2AR antagonism decreased punishment learning rate (i.e. negative feedback) at high and low doses. The low dose also decreased reinforcement sensitivity (beta) and increased stimulus and side stickiness (i.e., the tendency to repeat a choice regardless of outcome). 5-HT2CR antagonism also decreased beta, but reduced side stickiness.ConclusionsThese data indicate that 5-HT2A and 5-HT2CRs both modulate different aspects of flexibility, with 5-HT2ARs modulating learning from negative feedback as measured using RL parameters and 5-HT2CRs for learning from positive feedback assessed through conventional measures.

DQN and dynamic feedback for multitask scheduling optimization in engineering management

Abstract Within the realm of complex project management, the prevailing multitask scheduling optimization algorithm grapples with the dual constraints of sluggish convergence and exorbitant computational overhead. To address this challenge, this paper introduces a novel multitask scheduling optimization algorithm anchored in Deep Q Network (DQN) and dynamic feedback mechanisms. This innovative approach endeavors to ameliorate the algorithm’s learning prowess and scheduling efficiency through a reinforcement learning framework and dynamic feedback mechanisms. To substantiate the efficacy of the proposed algorithm, comprehensive experiments are conducted utilizing a sizable project management dataset. Comparative analyses are performed against established algorithms such as the ant colony algorithm, particle swarm algorithm, and adaptive genetic algorithm. Evaluation metrics encompass convergence speed and algorithmic runtime. Notably, experimental results unveil the superiority of the proposed algorithm across these metrics, underscoring its prowess in multitask scheduling within project management contexts. Looking ahead, future endeavors may entail further optimization of reinforcement learning parameters and their application in larger-scale engineering projects. Such endeavors hold the promise of augmenting the algorithm’s stability and efficacy, thereby fostering advancements in multitask scheduling optimization within the project management domain.

Harnessing the flexibility of neural networks to predict dynamic theoretical parameters underlying human choice behavior.

Reinforcement learning (RL) models are used extensively to study human behavior. These rely on normative models of behavior and stress interpretability over predictive capabilities. More recently, neural network models have emerged as a descriptive modeling paradigm that is capable of high predictive power yet with limited interpretability. Here, we seek to augment the expressiveness of theoretical RL models with the high flexibility and predictive power of neural networks. We introduce a novel framework, which we term theoretical-RNN (t-RNN), whereby a recurrent neural network is trained to predict trial-by-trial behavior and to infer theoretical RL parameters using artificial data of RL agents performing a two-armed bandit task. In three studies, we then examined the use of our approach to dynamically predict unseen behavior along with time-varying theoretical RL parameters. We first validate our approach using synthetic data with known RL parameters. Next, as a proof-of-concept, we applied our framework to two independent datasets of humans performing the same task. In the first dataset, we describe differences in theoretical RL parameters dynamic among clinical psychiatric vs. healthy controls. In the second dataset, we show that the exploration strategies of humans varied dynamically in response to task phase and difficulty. For all analyses, we found better performance in the prediction of actions for t-RNN compared to the stationary maximum-likelihood RL method. We discuss the use of neural networks to facilitate the estimation of latent RL parameters underlying choice behavior.

A series of unfortunate events: Do those who catastrophize learn more after negative outcomes?

AbstractCatastrophizing is a transdiagnostic construct that has been suggested to precipitate and maintain a multiplicity of psychiatric disorders, including anxiety, depression, post‐traumatic stress disorder, and obsessive‐compulsive disorder. However, the underlying cognitive mechanisms that result in catastrophizing are unknown. Relating reinforcement learning model parameters to catastrophizing may allow us to further understand the process of catastrophizing. Using a modified four‐armed bandit task, we aimed to investigate the relationship between reinforcement learning parameters and self‐report catastrophizing questionnaire scores to gain a mechanistic understanding of how catastrophizing may alter learning. We recruited 211 participants to complete a computerized four‐armed bandit task and tested the fit of six reinforcement learning models on our data, including two novel models which both incorporated a scaling factor related to a history of negative outcomes variable. We investigated the relationship between self‐report catastrophizing scores and free parameters from the overall best‐fitting model, along with the best‐fitting model to include history, using Pearson's correlations. Subsequently, we reassessed these relationships using multiple regression analyses to evaluate whether any observed relationships were altered when relevant IQ and mental health covariates were applied. Model‐agnostic analyses indicated there were effects of outcome history on reaction time and accuracy, and that the effects on accuracy related to catastrophizing. The overall model of best fit was the Standard Rescorla–Wagner Model and the best‐fitting model to include history was a model in which learning rate was scaled by history of negative outcome. We found no effect of catastrophizing on the scaling by history of negative outcome parameter (r = 0.003, p = 0.679), the learning rate parameter (r = 0.026, p = 0.703), or the inverse temperature parameter (r = 0.086, p = 0.220). We were unable to relate catastrophizing to any of the reinforcement learning parameters we investigated. This implies that catastrophizing is not straightforwardly linked to any changes to learning after a series of negative outcomes are received. Future research could incorporate further exploration of the space of models which include a history parameter.

Test-retest reliability of reinforcement learning parameters.

It has recently been suggested that parameter estimates of computational models can be used to understand individual differences at the process level. One area of research in which this approach, called computational phenotyping, has taken hold is computational psychiatry. One requirement for successful computational phenotyping is that behavior and parameters are stable over time. Surprisingly, the test-retest reliability of behavior and model parameters remains unknown for most experimental tasks and models. The present study seeks to close this gap by investigating the test-retest reliability of canonical reinforcement learning models in the context of two often-used learning paradigms: a two-armed bandit and a reversal learning task. We tested independent cohorts for the two tasks (N = 69 and N = 47) via an online testing platform with a between-test interval of five weeks. Whereas reliability was high for personality and cognitive measures (with ICCs ranging from .67 to .93), it was generally poor for the parameter estimates of the reinforcement learning models (with ICCs ranging from .02 to .52 for the bandit task and from .01 to .71 for the reversal learning task). Given that simulations indicated that our procedures could detect high test-retest reliability, this suggests that a significant proportion of the variability must be ascribed to the participants themselves. In support of that hypothesis, we show that mood (stress and happiness) can partly explain within-participant variability. Taken together, these results are critical for current practices in computational phenotyping and suggest that individual variability should be taken into account in the future development of the field.

Reliability of gamified reinforcement learning in densely sampled longitudinal assessments.

Reinforcement learning is a core facet of motivation and alterations have been associated with various mental disorders. To build better models of individual learning, repeated measurement of value-based decision-making is crucial. However, the focus on lab-based assessment of reward learning has limited the number of measurements and the test-retest reliability of many decision-related parameters is therefore unknown. In this paper, we present an open-source cross-platform application Influenca that provides a novel reward learning task complemented by ecological momentary assessment (EMA) of current mental and physiological states for repeated assessment over weeks. In this task, players have to identify the most effective medication by integrating reward values with changing probabilities to win (according to random Gaussian walks). Participants can complete up to 31 runs with 150 trials each. To encourage replay, in-game screens provide feedback on the progress. Using an initial validation sample of 384 players (9729 runs), we found that reinforcement learning parameters such as the learning rate and reward sensitivity show poor to fair intra-class correlations (ICC: 0.22-0.53), indicating substantial within- and between-subject variance. Notably, items assessing the psychological state showed comparable ICCs as reinforcement learning parameters. To conclude, our innovative and openly customizable app framework provides a gamified task that optimizes repeated assessments of reward learning to better quantify intra- and inter-individual differences in value-based decision-making over time.

Graph neural network architecture search for rotating machinery fault diagnosis based on reinforcement learning

In order to improve the accuracy of fault diagnosis, researchers are constantly trying to develop new diagnostic models. However, limited by the inherent thinking of human beings, it has always been difficult to build a pioneering architecture for rotating machinery fault diagnosis. In order to solve this problem, this paper uses reinforcement learning algorithm based on adjacency matrix to carry out network architecture search (NAS) of rotating machinery fault diagnosis model. A reinforcement learning agent for deep deterministic policy gradient (DDPG) is developed based on actor–critic neural networks. The observation state of reinforcement learning is used to develop the graph neural network (GNN) diagnosis model, and the diagnosis accuracy is fed back to the agent as a reward for updating the reinforcement learning parameters. The MFPT bearing fault datasets and the developed gear pitting fault experimental data are used to validate the proposed network architecture search method based on reinforcement learning (RL-NAS). The proposed method is proved to be practical and effective in various aspects such as fault diagnosis ability, search space, search efficiency and multi-working condition performance.

A core component of psychological therapy causes adaptive changes in computational learning mechanisms.

Cognitive distancing is an emotion regulation strategy commonly used in psychological treatment of various mental health disorders, but its therapeutic mechanisms are unknown. 935 participants completed an online reinforcement learning task involving choices between pairs of symbols with differing reward contingencies. Half (49.1%) of the sample was randomised to a cognitive self-distancing intervention and were trained to regulate or 'take a step back' from their emotional response to feedback throughout. Established computational (Q-learning) models were then fit to individuals' choices to derive reinforcement learning parameters capturing clarity of choice values (inverse temperature) and their sensitivity to positive and negative feedback (learning rates). Cognitive distancing improved task performance, including when participants were later tested on novel combinations of symbols without feedback. Group differences in computational model-derived parameters revealed that cognitive distancing resulted in clearer representations of option values (estimated 0.17 higher inverse temperatures). Simultaneously, distancing caused increased sensitivity to negative feedback (estimated 19% higher loss learning rates). Exploratory analyses suggested this resulted from an evolving shift in strategy by distanced participants: initially, choices were more determined by expected value differences between symbols, but as the task progressed, they became more sensitive to negative feedback, with evidence for a difference strongest by the end of training. Adaptive effects on the computations that underlie learning from reward and loss may explain the therapeutic benefits of cognitive distancing. Over time and with practice, cognitive distancing may improve symptoms of mental health disorders by promoting more effective engagement with negative information.

Stimulating human prefrontal cortex increases reward learning

Work in computational psychiatry suggests that mood disorders may stem from aberrant reinforcement learning processes. Specifically, it has been proposed that depressed individuals believe that negative events are more informative than positive events, resulting in higher learning rates from negative outcomes (Pulcu and Browning, 2019). In this proof-of-concept study, we investigated whether transcranial direct current stimulation (tDCS) applied to dorsolateral prefrontal cortex, as commonly used in depression treatment trials, might change learning rates for affective outcomes. Healthy adults completed an established reinforcement learning task (Pulcu and Browning, 2017) in which the information content of reward and loss outcomes was manipulated by varying the volatility of stimulus-outcome associations. Learning rates on the tasks were quantified using computational models. Stimulation over dorsolateral prefrontal cortex (DLPFC) but not motor cortex (M1) increased learning rates specifically for reward outcomes. The effects of prefrontal tDCS were cognitive state-dependent: tDCS applied during task performance increased learning rates for wins; tDCS applied before task performance decreased both win and loss learning rates. A replication study confirmed the key finding that tDCS to DLPFC during task performance increased learning rates specifically for rewards. Taken together, these findings demonstrate the potential of tDCS for modulating computational parameters of reinforcement learning that are relevant to mood disorders.

Hosting Capacity Assessment Strategies and Reinforcement Learning Methods for Coordinated Voltage Control in Electricity Distribution Networks: A Review

Increasing connection rates of rooftop photovoltaic (PV) systems to electricity distribution networks has become a major concern for the distribution network service providers (DNSPs) due to the inability of existing network infrastructure to accommodate high levels of PV penetration while maintaining voltage regulation and other operational requirements. The solution to this dilemma is to undertake a hosting capacity (HC) study to identify the maximum penetration limit of rooftop PV generation and take necessary actions to enhance the HC of the network. This paper presents a comprehensive review of two topics: HC assessment strategies and reinforcement learning (RL)-based coordinated voltage control schemes. In this paper, the RL-based coordinated voltage control schemes are identified as a means to enhance the HC of electricity distribution networks. RL-based algorithms have been widely used in many power system applications in recent years due to their precise, efficient and model-free decision-making capabilities. A large portion of this paper is dedicated to reviewing RL concepts and recently published literature on RL-based coordinated voltage control schemes. A non-exhaustive classification of RL algorithms for voltage control is presented and key RL parameters for the voltage control problem are identified. Furthermore, critical challenges and risk factors of adopting RL-based methods for coordinated voltage control are discussed.

Feedback control approaches for restoration of power grids from blackouts

The automated restoration of power systems with variable energy resources is a timely problem to tackle. Automated restoration advice can support operators in deciding on strategic actions to restore power grids from a blackout with a mix of conventional and renewable generation resources. To this end, this paper frames the restoration process of power grids with solar resources as a nonlinear dynamic model with algebraic constraints in discrete time which is steered by feedback control loops. We discuss two feedback-control strategies based on greedy and reinforcement learning algorithms, and contrast their performance with restoration plans generated by a mixed-integer linear program. We found that the reinforcement learning algorithm infers restoration actions faster than the greedy one. However, the tuning process of the reinforcement learning parameters is slower than for the greedy one.

Reinforcement Learning Disruptions in Individuals With Depression and Sensitivity to Symptom Change Following Cognitive Behavioral Therapy

Major depressive disorder is prevalent and impairing. Parsing neurocomputational substrates of reinforcement learning in individuals with depression may facilitate a mechanistic understanding of the disorder and suggest new cognitive therapeutic targets. To determine associations among computational model-derived reinforcement learning parameters, depression symptoms, and symptom changes after treatment. In this mixed cross-sectional-cohort study, individuals performed reward and loss variants of a probabilistic learning task during functional magnetic resonance imaging at baseline and follow-up. A volunteer sample with and without a depression diagnosis was recruited from the community. Participants were assessed from July 2011 to February 2017, and data were analyzed from May 2017 to May 2021. Computational model-based analyses of participants' choices assessed a priori hypotheses about associations between components of reward-based and loss-based learning with depression symptoms. Changes in both learning parameters and symptoms were then assessed in a subset of participants who received cognitive behavioral therapy (CBT). Of 101 included adults, 69 (68.3%) were female, and the mean (SD) age was 34.4 (11.2) years. A total of 69 participants with a depression diagnosis and 32 participants without a depression diagnosis were included at baseline; 48 participants (28 with depression who received CBT and 20 without depression) were included at follow-up (mean [SD] of 115.1 [15.6] days). Computational model-based analyses of behavioral choices and neural data identified associations of learning with symptoms during reward learning and loss learning, respectively. During reward learning only, anhedonia (and not negative affect or arousal) was associated with model-derived learning parameters (learning rate: posterior mean regression β = -0.14; 95% credible interval [CrI], -0.12 to -0.03; outcome sensitivity: posterior mean regression β = 0.18; 95% CrI, 0.02 to 0.37) and neural learning signals (moderation of association between striatal prediction error and expected value signals: t97 = -2.10; P = .04). During loss learning only, negative affect (and not anhedonia or arousal) was associated with learning parameters (outcome shift: posterior mean regression β = -0.11; 95% CrI, -0.20 to -0.01) and disrupted neural encoding of learning signals (association with subgenual anterior cingulate prediction error signals: r = -0.28; P = .005). Symptom improvement following CBT was associated with normalization of learning parameters that were disrupted at baseline (reward learning rate: posterior mean regression β = 0.15; 90% CrI, 0.001 to 0.41; loss outcome shift: posterior mean regression β = 0.42; 90% CrI, 0.09 to 0.77). In this study, the mapping of reinforcement learning components to symptoms of major depression revealed mechanistic features associated with these symptoms and points to possible learning-based therapeutic processes and targets.

Impaired Learning From Negative Feedback in Stimulant Use Disorder: Dopaminergic Modulation.

BackgroundDrug-induced alterations to the dopamine system in stimulant use disorder (SUD) are hypothesized to impair reinforcement learning (RL). Computational modeling enables the investigation of the latent processes of RL in SUD patients, which could elucidate the nature of their impairments.MethodsWe investigated RL in 44 SUD patients and 41 healthy control participants using a probabilistic RL task that assesses learning from reward and punishment separately. In an independent sample, we determined the modulatory role of dopamine in RL following a single dose of the dopamine D2/3 receptor antagonist amisulpride (400 mg) and the agonist pramipexole (0.5 mg) in a randomised, double-blind, placebo-controlled, crossover design. We analyzed task performance using computational modelling and hypothesized that RL impairments in SUD patients would be differentially modulated by a dopamine D2/3 receptor antagonist and agonist.ResultsComputational analyses in both samples revealed significantly reduced learning rates from punishment in SUD patients compared with healthy controls, whilst their reward learning rates were not measurably impaired. In addition, the dopaminergic receptor agents modulated RL parameters differentially in both groups. Both amisulpride and pramipexole impaired RL parameters in healthy participants, but ameliorated learning from punishment in SUD patients.ConclusionOur findings suggest that RL impairments seen in SUD patients are associated with altered dopamine function.

Reinforcement learning for the traveling salesman problem with refueling

The traveling salesman problem (TSP) is one of the best-known combinatorial optimization problems. Many methods derived from TSP have been applied to study autonomous vehicle route planning with fuel constraints. Nevertheless, less attention has been paid to reinforcement learning (RL) as a potential method to solve refueling problems. This paper employs RL to solve the traveling salesman problem With refueling (TSPWR). The technique proposes a model (actions, states, reinforcements) and RL-TSPWR algorithm. Focus is given on the analysis of RL parameters and on the refueling influence in route learning optimization of fuel cost. Two RL algorithms: Q-learning and SARSA are compared. In addition, RL parameter estimation is performed by Response Surface Methodology, Analysis of Variance and Tukey Test. The proposed method achieves the best solution in 15 out of 16 case studies.

Design of model-free reinforcement learning control for tunable vibration absorber system based on magnetorheological elastomer

This paper proposes a novel model-free reinforcement learning (RL) for vibration control of a magnetorheological elastomer (MRE)-based application. Because the modeling of the MRE stiffness is nonlinear and time-varying depending on various environmental factors, this paper approached the MRE control issue via a model-free learning based method. In this study, an RL model is designed for the MRE-based tunable vibration absorber (TVA) which can control the optimal stiffness of MRE to maximize the vibration suppression. The designed RL algorithm continuously learns and updates the optimal control input of the MRE stiffness adaptively to the dynamic environment without using any prior knowledge of the MRE modeling. From analyzing the mechanism of MRE TVA, the RL algorithm and parameters are carefully designed for a high vibration performance. Also, this study proposed several ideas to make the RL model simpler and converge at a high rate. The experiments confirmed that the proposed RL model showed a rapid convergence to the optimal policy which could minimize the vibration level with respect to the dynamic excitation disturbance. Results showed that the RL model had a similar performance as the conventional tuning method and suppressed the vibration level as much as 57% compared to the one without the controller. Also, the proposed RL algorithm was able to estimate the actual dynamics of the MRE TVA by learning from the environment. Thus, this study showed the feasibility of implementing a model-free RL model to realize an adaptive controller for applications based on highly nonlinear MRE.

DEEP REINFORCEMENT LEARNING WITH SPARSE DISTRIBUTED MEMORY FOR “WATER WORLD” PROBLEM SOLVING

Context. Machine learning is one of the actively developing areas of data processing. Reinforcement learning is a class of machine learning methods where the problem involves mapping the sequence of environmental states to agent’s actions. Significant progress in this area has been achieved using DQN-algorithms, which became one of the first classes of stable algorithms for learning using deep neural networks. The main disadvantage of this approach is the rapid growth of RAM in real-world tasks. The approach proposed in this paper can partially solve this problem.
 Objective. The aim is to develop a method of forming the structure and nature of access to the sparse distributed memory with increased information content to improve reinforcement learning without additional memory.
 Method. A method of forming the structure and modification of sparse distributed memory for storing previous transitions of the actor in the form of prototypes is proposed. The method allows increasing the informativeness of the stored data and, as a result, to improve the process of creating a model of the studied process by intensifying the learning of the deep neural network. Increasing the informativeness of the stored data is the result of this sequence of actions. First, we compare the new transition and the last saved transition. To perform this comparison, this method introduces a rate estimate for the distance between transitions. If the distance between the new transition and the last saved transition is smaller than the specified threshold, the new transition is written in place of the previous one without increasing the amount of memory. Otherwise, we create a new prototype in memory while deleting the prototype that has been stored in memory the longest.
 Results. The work of the proposed method was studied during the solution of the popular “Water World” test problem. The results showed a 1.5-times increase in the actor’s survival time in a hostile environment. This result was achieved by increasing the informativeness of the stored data without increasing the amount of RAM.
 Conclusions. The proposed method of forming and modifying the structure of sparse distributed memory allowed to increase the informativeness of the stored data. As a result of this approach, improved reinforcement learning parameters on the example of the “Water World” problem by increasing the accuracy of the model of the physical process represented by a deep neural network.

Individual differences in experienced and observational decision-making illuminate interactions between reinforcement learning and declarative memory

Decision making can be shaped both by trial-and-error experiences and by memory of unique contextual information. Moreover, these types of information can be acquired either by means of active experience or by observing others behave in similar situations. The interactions between reinforcement learning parameters that inform decision updating and memory formation of declarative information in experienced and observational learning settings are, however, unknown. In the current study, participants took part in a probabilistic decision-making task involving situations that either yielded similar outcomes to those of an observed player or opposed them. By fitting alternative reinforcement learning models to each subject, we discerned participants who learned similarly from experience and observation from those who assigned different weights to learning signals from these two sources. Participants who assigned different weights to their own experience versus those of others displayed enhanced memory performance as well as subjective memory strength for episodes involving significant reward prospects. Conversely, memory performance of participants who did not prioritize their own experience over others did not seem to be influenced by reinforcement learning parameters. These findings demonstrate that interactions between implicit and explicit learning systems depend on the means by which individuals weigh relevant information conveyed via experience and observation.

Autonomic responses to choice outcomes: Links to task performance and reinforcement-learning parameters

Previous work has shown that autonomic responses to choice feedback can predict subsequent decision-making. In this study, we tested whether skin conductance responses (SCRs) and heart rate (HR) decelerations following the presentation of choice outcomes predict Iowa Gambling Task performance in nonclinical participants (n = 64). We also examined how these signals related to parameters of a reinforcement-learning (RL) model. Feedback SCRs and HR decelerations were greater following outcomes that included losses and choices from the bad decks defined by their negative expected value. In addition, SCRs predicted task performance. A hierarchical Bayesian RL model indicated that greater feedback SCR for the bad decks compared to good decks was associated with stronger loss aversion and a lower learning rate, both of which predicted higher performance. These results suggest that feedback-related SCRs are linked to individual differences in outcome evaluation and learning processes that guide reinforcement-learning.