Rescorla-Wagner Model Research Articles

Associative Learning and Active Inference.

Associative learning is a behavioral phenomenon in which individuals develop connections between stimuli or events based on their co-occurrence. Initially studied by Pavlov in his conditioning experiments, the fundamental principles of learning have been expanded on through the discovery of a wide range of learning phenomena. Computational models have been developed based on the concept of minimizing reward prediction errors. The Rescorla-Wagner model, in particular, is a well-known model that has greatly influenced the field of reinforcement learning. However, the simplicity of these models restricts their ability to fully explain the diverse range of behavioral phenomena associated with learning. In this study, we adopt the free energy principle, which suggests that living systems strive to minimize surprise or uncertainty under their internal models of the world. We consider the learning process as the minimization of free energy and investigate its relationship with the Rescorla-Wagner model, focusing on the informational aspects of learning, different types of surprise, and prediction errors based on beliefs and values. Furthermore, we explore how well-known behavioral phenomena such as blocking, overshadowing, and latent inhibition can be modeled within the active inference framework. We accomplish this by using the informational and novelty aspects of attention, which share similar ideas proposed by seemingly contradictory models such as Mackintosh and Pearce-Hall models. Thus, we demonstrate that the free energy principle, as a theoretical framework derived from first principles, can integrate the ideas and models of associative learning proposed based on empirical experiments and serve as a framework for a better understanding of the computational processes behind associative learning in the brain.

A psychological mechanism for the development of anxiety.

Although numerous behavioral constructs have been proposed to account for anxiety disorders, how these disorders develop within an individual has been difficult to predict. In this perspective, I selectively review clinical and experimental evidence suggesting that avoidance (i.e., safety) behavior increases beliefs of threat or fear. The experimental evidence has been replicated numerous times, with different parameters, and shows that when human participants emit avoidance responses in the presence of a neutral stimulus, they later show heightened expectations of threat in the presence of the neutral stimulus. I interpret these findings as resulting from prediction errors as anticipated by the Rescorla-Wagner model, although other animal learning theories can also predict the phenomenon. I discuss some implications and offer a few novel predictions. The analysis presented here sheds light on a phenomenon of theoretical and clinical relevance which is accommodated by basic associative learning theory. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Humans adaptively select different computational strategies in different learning environments.

The Rescorla-Wagner rule remains the most popular tool to describe human behavior in reinforcement learning tasks. Nevertheless, it cannot fit human learning in complex environments. Previous work proposed several hierarchical extensions of this learning rule. However, it remains unclear when a flat (nonhierarchical) versus a hierarchical strategy is adaptive, or when it is implemented by humans. To address this question, current work applies a nested modeling approach to evaluate multiple models in multiple reinforcement learning environments both computationally (which approach performs best) and empirically (which approach fits human data best). We consider 10 empirical data sets (N = 407) divided over three reinforcement learning environments. Our results demonstrate that different environments are best solved with different learning strategies; and that humans adaptively select the learning strategy that allows best performance. Specifically, while flat learning fitted best in less complex stable learning environments, humans employed more hierarchically complex models in more complex environments. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Incorporating uncertainty within dynamic interoceptive learning.

Interoception, the perception of the internal state of the body, has been shown to be closely linked to emotions and mental health. Of particular interest are interoceptive learning processes that capture associations between environmental cues and body signals as a basis for making homeostatically relevant predictions about the future. One method of measuring respiratory interoceptive learning that has shown promising results is the Breathing Learning Task (BLT). While the original BLT required binary predictions regarding the presence or absence of an upcoming inspiratory resistance, here we extended this paradigm to capture continuous measures of prediction (un)certainty. Sixteen healthy participants completed the continuous version of the BLT, where they were asked to predict the likelihood of breathing resistances on a continuous scale from 0.0 to 10.0. In order to explain participants' responses, a Rescorla-Wagner model of associative learning was combined with suitable observation models for continuous or binary predictions, respectively. For validation, we compared both models against corresponding null models and examined the correlation between observed and modeled predictions. The model was additionally extended to test whether learning rates differed according to stimuli valence. Finally, summary measures of prediction certainty as well as model estimates for learning rates were considered against interoceptive and mental health questionnaire measures. Our results demonstrated that the continuous model fits closely captured participant behavior using empirical data, and the binarised predictions showed excellent replicability compared to previously collected data. However, the model extension indicated that there were no significant differences between learning rates for negative (i.e. breathing resistance) and positive (i.e. no breathing resistance) stimuli. Finally, significant correlations were found between fatigue severity and both prediction certainty and learning rate, as well as between anxiety sensitivity and prediction certainty. These results demonstrate the utility of gathering enriched continuous prediction data in interoceptive learning tasks, and suggest that the updated BLT is a promising paradigm for future investigations into interoceptive learning and potential links to mental health.

Bayesian reinforcement learning: A basic overview

We and other animals learn because there is some aspect of the world about which we are uncertain. This uncertainty arises from initial ignorance, and from changes in the world that we do not perfectly know; the uncertainty often becomes evident when our predictions about the world are found to be erroneous. The Rescorla-Wagner learning rule, which specifies one way that prediction errors can occasion learning, has been hugely influential as a characterization of Pavlovian conditioning and, through its equivalence to the delta rule in engineering, in a much wider class of learning problems. Here, we review the embedding of the Rescorla-Wagner rule in a Bayesian context that is precise about the link between uncertainty and learning, and thereby discuss extensions to such suggestions as the Kalman filter, structure learning, and beyond, that collectively encompass a wider range of uncertainties and accommodate a wider assortment of phenomena in conditioning.

Variation in the effectiveness of reinforcement and nonreinforcement in generating different conditioned behaviors

Rat autoshaping procedures generate two readily measurable conditioned responses: During lever presentations that have previously signaled food, rats approach the food well (called goal-tracking) and interact with the lever itself (called sign-tracking). We investigated how reinforced and nonreinforced trials affect the overall and temporal distributions of these two responses across 10-second lever presentations.In two experiments, reinforced trials generated more goal-tracking than sign-tracking, and nonreinforced trials resulted in a larger reduction in goal-tracking than sign-tracking. The effect of reinforced trials was evident as an increase in goal-tracking and reduction in sign-tracking across the duration of the lever presentations, and nonreinforced trials resulted in this pattern transiently reversing and then becoming less evident with further training.These dissociations are consistent with a recent elaboration of the Rescorla-Wagner model, HeiDI (Honey, R.C., Dwyer, D.M., & Iliescu, A.F. (2020a). HeiDI: A model for Pavlovian learning and performance with reciprocal associations. Psychological Review, 127, 829–852.), a model in which responses related to the nature of the unconditioned stimulus (e.g., goal-tracking) have a different origin than those related to the nature of the conditioned stimulus (e.g., sign-tracking).

Social bond dynamics and the evolution of helping

Empiricists often struggle to apply game theory models to real-life cases of animal cooperation. One reason is that many examples of cooperation occur in stable groups, where individuals form social bonds that influence exchanges of help in ways that are not well described by previous models, including the extent of reciprocity and how relationships are initiated. We present a game theory model exploring the conditions under which social bonds between group members promote cooperation. In the model, bonds build up from exchanges of help in a similar way as the strength of association increases in learning, as in the Rescorla-Wagner rule. The bonds in turn affect partner choice and influence helping amounts. The model has a mechanism of reciprocity for bonded pairs, which can evolve toward either loose or strict reciprocation. Several aspects of the model are inspired by observations of food sharing in vampire bats. We find that small social neighborhoods are required for the evolutionary stability of helping, either as small group sizes, or if bonded members of larger groups can form temporary (daily) smaller groupings. The costs of helping need to be fairly low, while the benefits can be substantial. The form of reciprocity that evolves is neither immediate nor very strict. Individuals in need request help based on bond strength, but there is also an evolved preference for initiating bonds with new group members. In contrast, if different groups come into temporary contact, the evolved tendency is to avoid forming bonds between groups.

A series of unfortunate events: Do those who catastrophize learn more after negative outcomes?

AbstractCatastrophizing is a transdiagnostic construct that has been suggested to precipitate and maintain a multiplicity of psychiatric disorders, including anxiety, depression, post‐traumatic stress disorder, and obsessive‐compulsive disorder. However, the underlying cognitive mechanisms that result in catastrophizing are unknown. Relating reinforcement learning model parameters to catastrophizing may allow us to further understand the process of catastrophizing. Using a modified four‐armed bandit task, we aimed to investigate the relationship between reinforcement learning parameters and self‐report catastrophizing questionnaire scores to gain a mechanistic understanding of how catastrophizing may alter learning. We recruited 211 participants to complete a computerized four‐armed bandit task and tested the fit of six reinforcement learning models on our data, including two novel models which both incorporated a scaling factor related to a history of negative outcomes variable. We investigated the relationship between self‐report catastrophizing scores and free parameters from the overall best‐fitting model, along with the best‐fitting model to include history, using Pearson's correlations. Subsequently, we reassessed these relationships using multiple regression analyses to evaluate whether any observed relationships were altered when relevant IQ and mental health covariates were applied. Model‐agnostic analyses indicated there were effects of outcome history on reaction time and accuracy, and that the effects on accuracy related to catastrophizing. The overall model of best fit was the Standard Rescorla–Wagner Model and the best‐fitting model to include history was a model in which learning rate was scaled by history of negative outcome. We found no effect of catastrophizing on the scaling by history of negative outcome parameter (r = 0.003, p = 0.679), the learning rate parameter (r = 0.026, p = 0.703), or the inverse temperature parameter (r = 0.086, p = 0.220). We were unable to relate catastrophizing to any of the reinforcement learning parameters we investigated. This implies that catastrophizing is not straightforwardly linked to any changes to learning after a series of negative outcomes are received. Future research could incorporate further exploration of the space of models which include a history parameter.

The effects of extinction and an explicitly unpaired treatment on the reinforcing properties of a Pavlovian conditioned stimulus

This series of experiments examined the effects of extinction and an explicitly unpaired treatment on the ability of a conditioned stimulus (CS) to function as a reinforcer. Rats were trained to lever press for food, exposed to pairings of a noise CS and food, and, finally, tested for their willingness to lever press for the CS in the absence of the food. Experiment 1 provided a demonstration of conditioned reinforcement (using controls that were only exposed to unpaired presentations of the CS and food) and showed that it was equivalent after one or four sessions of CS-food pairings. Experiments 2 and 3 showed that, after one session of CS-food pairings, repeated presentations of the CS alone reduced its reinforcing properties; but after four sessions of CS-food pairings, repeated presentations of the CS alone had no effect on these properties. Experiment 4 showed that, after four sessions of CS-food pairings, explicitly unpaired presentations of the CS and food completely undermined conditioned reinforcement. Finally, Experiment 5 provided within-experiment evidence that, after four sessions of CS-food pairings, the reinforcing properties of the CS were disrupted by explicitly unpaired presentations of the CS and food but spared by repeated presentations of the CS alone. Together, these findings indicate that the effectiveness of extinction in undermining the reinforcing properties of a CS depends on its level of conditioning; and that, where extinction fails to disrupt these properties, they are successfully undermined by an explicitly unpaired treatment. They are discussed with respect to findings in the literature on Pavlovian-to-instrumental transfer; and the Rescorla-Wagner model, which anticipates that an explicitly unpaired treatment will be more effective than extinction in reversing the effects of conditioning.

Why is the Rescorla-Wagner model so influential?

The influence of the Rescorla-Wagner model cannot be overestimated, despite that (1) the model does not differ much computationally from its predecessors and competitors, and (2) its shortcomings are well-known in the learning community. Here we discuss the reasons behind its widespread influence in the cognitive and neural sciences, and argue that it is the constant search for general-process theories by learning scholars which eventually produced a model whose application spans many different areas of research to this day. We focus on the theoretical and empirical background of the model, the theoretical connections that it has with later developments across Marr’s levels of analysis, as well as the broad variety of research that it has guided and inspired.

The Rescorla-Wagner model, prediction error, and fear learning

The Rescorla-Wagner model remains one of the most important and influential theoretical accounts of the conditions under which Pavlovian learning occurs. Moreover, the experimental approaches that inspired the model continue to provide powerful behavioral tools to advance mechanistic understanding of how we and other animals learn to fear and learn to reduce fear. Here we consider key features of the Rescorla-Wagner model as applied to study of fear learning. We review evidence for key insights of the model. First, learning to fear and learning to reduce fear are governed by a common, signed prediction error. Second, this error drives variations in effectiveness of the shock US that are causal to whether and how much fear is learned or lost during a conditioning trial. We also consider behavioral and neural findings inconsistent with the model and which will be essential to understand and advance understanding of fear learning.

How much data do we need to estimate computational models of decision-making? The COMPASS toolbox.

How much data are needed to obtain useful parameter estimations from a computational model? The standard approach to address this question is to carry out a goodness-of-recovery study. Here, the correlation between individual-participant true and estimated parameter values determines when a sample size is large enough. However, depending on one's research question, this approach may be suboptimal, potentially leading to sample sizes that are either too small (underpowered) or too large (overcostly or unfeasible). In this paper, we formulate a generalized concept of statistical power and use this to propose a novel approach toward determining how much data is needed to obtain useful parameter estimates from a computational model. We describe a Python-based toolbox (COMPASS) that allows one to determine how many participants are needed to fit one specific computational model, namely the Rescorla-Wagner model of learning and decision-making. Simulations revealed that a high number of trials per person (more than the number of persons) are a prerequisite for high-powered studies in this particular setting.

Overshadowing, but not relative validity, between the elements of an outcome during human associative learning

Overshadowing and relative validity constitute two phenomena that inspired the development of the Rescorla-Wagner model in 1972. They demonstrate that cues will interact with one another for an association with the presence or absence of an outcome. Here, three experiments sought to explore whether these two effects extended to outcomes using a food allergist paradigm with human participants. In Experiment 1 (overshadowing) participants received trials in which a cue was followed by a compound of two outcomes (A-O1O2). Test trials revealed that participants learned less about the A-O2 association than they did between a control cue C, which had been paired with O2 in isolation (C-O2) in training – thus demonstrating an outcome overshadowing effect. In Experiment 2 (relative validity) participants received true discrimination trials, in which A was paired with an O1O3 compound and B was paired with an O2O3 compound, and pseudo discrimination trials, in which C and D were paired on 50% of the trials with an O4O6 compound and on the remaining trials with an O5O6 compound. Consequently, O3 is less well predicted by A and B relative to O1 and O2, whereas O6 is equally well predicted by C and D relative to O4 and O5. Despite the relative validity of A and B for O3 being less than the relative validity of C and D for O6, the ratings of A and B for O3 were the same as C and D for O6. This failure to observe an outcome relative validity effect was reproduced in Experiment 3, which replicated Experiment 2, but with an adjustment made to the number of training trials given to participants. These results are discussed in terms of a real-time development of the Rescorla-Wagner model provided by Wagner (1981).

Error‐Correction Mechanisms in Language Learning: Modeling Individuals

AbstractSince its first adoption as a computational model for language learning, evidence has accumulated that Rescorla–Wagner error‐correction learning (Rescorla & Wagner, 1972) captures several aspects of language processing. Whereas previous studies have provided general support for the Rescorla–Wagner rule by using it to explain the behavior of participants across a range of tasks, we focus on testing predictions generated by the model in a controlled natural language learning task and model the data at the level of the individual learner. By adjusting the parameters of the model to fit the trial‐by‐trial behavioral choices of participants, rather than fitting a one‐for‐all model using a single set of default parameters, we show that the model accurately captures participants’ choices, time latencies, and levels of response agreement. We also show that gender and working memory capacity affect the extent to which the Rescorla–Wagner model captures language learning.

Incorporating social knowledge structures into computational models

To navigate social interactions successfully, humans need to continuously learn about the personality traits of other people (e.g., how helpful or aggressive is the other person?). However, formal models that capture the complexities of social learning processes are currently lacking. In this study, we specify and test potential strategies that humans can employ for learning about others. Standard Rescorla-Wagner (RW) learning models only capture parts of the learning process because they neglect inherent knowledge structures and omit previously acquired knowledge. We therefore formalize two social knowledge structures and implement them in hybrid RW models to test their usefulness across multiple social learning tasks. We name these concepts granularity (knowledge structures about personality traits that can be utilized at different levels of detail during learning) and reference points (previous knowledge formalized into representations of average people within a social group). In five behavioural experiments, results from model comparisons and statistical analyses indicate that participants efficiently combine the concepts of granularity and reference points—with the specific combinations in models depending on the people and traits that participants learned about. Overall, our experiments demonstrate that variants of RW algorithms, which incorporate social knowledge structures, describe crucial aspects of the dynamics at play when people interact with each other.

Obsessive-compulsive disorder is characterized by decreased Pavlovian influence on instrumental behavior.

Obsessive-compulsive disorder (OCD) is characterized by uncontrollable repetitive actions thought to rely on abnormalities within fundamental instrumental learning systems. We investigated cognitive and computational mechanisms underlying Pavlovian biases on instrumental behavior in both clinical OCD patients and healthy controls using a Pavlovian-Instrumental Transfer (PIT) task. PIT is typically evidenced by increased responding in the presence of a positive (previously rewarded) Pavlovian cue, and reduced responding in the presence of a negative cue. Thirty OCD patients and thirty-one healthy controls completed the Pavlovian Instrumental Transfer test, which included instrumental training, Pavlovian training for positive, negative and neutral cues, and a PIT phase in which participants performed the instrumental task in the presence of the Pavlovian cues. Modified Rescorla-Wagner models were fitted to trial-by-trial data of participants to estimate underlying computational mechanism and quantify individual differences during training and transfer stages. Bayesian hierarchical methods were used to estimate free parameters and compare the models. Behavioral and computational results indicated a weaker Pavlovian influence on instrumental behavior in OCD patients than in HC, especially for negative Pavlovian cues. Our results contrast with the increased PIT effects reported for another set of disorders characterized by compulsivity, substance use disorders, in which PIT is enhanced. A possible reason for the reduced PIT in OCD may be impairment in using the contextual information provided by the cues to appropriately adjust behavior, especially when inhibiting responding when a negative cue is present. This study provides deeper insight into our understanding of deficits in OCD from the perspective of Pavlovian influences on instrumental behavior and may have implications for OCD treatment modalities focused on reducing compulsive behaviors.

The Rescorla-Wagner Model: The culmination of Hume’s theory of causation.

The associative learning theory of Robert Rescorla and Allan Wagner has been duly celebrated for its 50-year reign as the predominant model in learning science. One special recognition is warranted: its close correspondence with David Hume's associative theory of causality judgment. Hume's rules by which causes come to suggest effects are not only embraced by the Rescorla-Wagner model, but their mechanistic account makes precise quantitative predictions that can be assessed by empirical evidence rather than by speculation and argumentation. Framed in this way, the Rescorla-Wagner model truly represents the scientific culmination of Hume's philosophical theory of causation. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Explaining the Return of Fear with Revised Rescorla-Wagner Models.

Exposure therapy - exposure to a feared stimulus without harmful consequences - can reduce fear responses in many mental disorders. However, such relief is often partial and temporary: fear can return after the therapy has ended. Conditioning research has identified three mechanisms for the return of fear, viz. change in physical context (renewal), the passage of time (spontaneous recovery), and an encounter with the fear-producing unconditioned stimulus (reinstatement). To understand why fear returns and thereby develop more effective therapies, we develop mathematical learning models based on that of Rescorla and Wagner. According to this model, context cues present during extinction become conditioned inhibitors (i.e. safety signals) which prevent total erasure of the threat association. Adding various mechanisms to the model allows it to explain different facets of the return of fear. Among these mechanisms is decay of inhibitory associations, which provides a novel explanation for spontaneous recovery. To make the benefits of exposure robust and permanent, one must minimize the degree to which the extinction context becomes inhibitory in order to maximize unlearning. We simulate several experimental paradigms that reduce the return of fear and explain them according to this principle.

An adaptation algorithm for personalised virtual reality exposure therapy

BackgroundAnxiety disorders are highly prevalent in mental health problems. The lives of people suffering from an anxiety disorder can be severely impaired. Virtual Reality Exposure Therapy (VRET) is an effective treatment, which immerses patients in a controlled Virtual Environment (VE). This creates the opportunity to confront feared stimuli and learn how to deal with them, which may result in the reduction of anxiety. The configuration of these VEs requires extensive effort to maximise the potential of Virtual Reality (VR) and the effectiveness of the therapy. Manual configuration becomes infeasible when the number of possible virtual stimuli combinations is infinite. Due to the growing complexity, acquiring the skills to truly master a VR system is difficult and it increases the threshold for psychotherapists to use such useful systems. We therefore developed a prototype of a supportive algorithm to facilitate the use of VRET in a clinical setting. This automatised system assists psychotherapists to use the wide range of functionalities without burdening them with technical challenges. Thus, psychotherapists can focus their attention on the patient. MethodsIn this paper both the prototype of the algorithm and a first proof of concept are described. The algorithm suggests environment configurations for VRET, tailored to the individual therapeutic needs of each patient. The system aims to maximise learning during exposure therapy for different combinations of stimuli by using the Rescorla–Wagner model as a predictor for learning. In a first proof of concept, the VE configurations suggested by the algorithm for three anonymised clinical vignettes were compared with prior manual configurations by two psychotherapists. ResultsThe prototype of the algorithm and a first proof of concept are described. The first proof of concept demonstrated the relevance and potential of the proposed system, as it managed to propose similar configurations for the clinical vignettes compared to those made by therapists. Nonetheless, because of the exploratory nature of the study, no claims can yet be made about its efficacy. ConclusionsWith the increasing ubiquity of immersive technologies, this technology for assisted configuration of VEs could make VRET a valuable tool for psychotherapists.

Spontaneous recovery from overexpectation in an insect

In associative learning in mammals, it is widely accepted that learning is determined by the prediction error, i.e., the error between the actual reward and the reward predicted by the animal. However, it is unclear whether error-based learning theories are applicable to the learning occurring in other non-mammalian species. Here, we examined whether overexpectation, a phenomenon that supports error-based learning theories, occurs in crickets. Crickets were independently trained with two different conditioned stimuli (CSs), an odour and a visual pattern, that were followed by an appetitive unconditioned stimulus (US). Then the two CSs were presented simultaneously as a compound, followed by the same US. This treatment resulted in a reduced conditioned response to the odour CS when tested immediately after training. However, the response to the CS was partially recovered after 1 day. These results are the first to show overexpectation and its spontaneous recovery in an invertebrate species. While the results showing overexpectation are in agreement with the prediction by the Rescorla-Wagner model, a major form of error-based learning theories, the ones showing spontaneous recovery are not. Our results suggest that conventional error-based learning models account for some, but not for all essential features of Pavlovian conditioning in crickets.