Animal Learning Theory Research Articles

Developments in associative theory: A tribute to the contributions of Robert A. Rescorla.

The field of associative learning theory was forever changed by the contributions of Robert A. Rescorla. He created an organizational structure that gave us a framework for thinking about the key questions surrounding learning theory: what are the conditions that produce learning?, what is the content of that learning?, and how is that learning expressed in performance? He gave us beautifully sophisticated experimental designs that tackled deep theoretical problems in experimentally clever and elegant ways. And he left us with a collection of work that fundamentally altered the way we as a field think about basic learning processes. Few scientists have impacted their field in the way that Rescorla impacted animal learning theory. In this paper, we introduce this special issue (Developments in Associative Theory: A Tribute to Robert A. Rescorla) by considering some of the many ways in which Rescorla's empirical and theoretical contributions impacted learning theory over his almost 50-year career. We conclude by identifying multiple fundamental issues we think he would have found especially fruitful to pursue as we continue to move forward. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

A response function that maps associative strengths to probabilities.

Bridging associative and normative theories of animal learning, I show that an associative system can behave as if performing probabilistic inference by using the function f(V) = 1 - e-cV to transform associative strengths (V) into response probabilities. For example, using this function, an associative system can respond normatively to a compound stimulus AB, given previous separate experiences with the components A and B. The CR probability formulae that result from the proposed function have a normative interpretation in terms of statistical decision theory. The formulae also suggest a normative interpretation of stimulus generalization as a heuristic to infer whether different stimuli are likely to convey redundant or independent information about reinforcement. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Cross-species studies on orbitofrontal control of inference-based behavior.

Many decisions are guided by expectations about their outcomes. These expectations can arise from two fundamentally different sources: from direct experience with outcomes and the events and actions that precede them or from mental simulations and inferences when direct experience is missing. Here we discuss four elegant tasks from animal learning theory (devaluation, sensory preconditioning, Pavlovian-to-instrumental transfer, and Pavlovian overexpectation) and how they can be used to isolate behavior that is based on such mental simulations from behavior that can be based solely on experience. We then review findings from studies in rodents, nonhuman primates, and humans that use these tasks in combination with neural recording and loss-of-function experiments to understand the role of the orbitofrontal cortex (OFC) in outcome inference. The results of these studies show that activity in the OFC is correlated with inferred outcome expectations and that an intact OFC is necessary for inference-based behavior and learning. In summary, these findings provide converging cross-species support for the idea that the OFC is critical for behavior that is based on inferred outcomes, whereas it is not required when expectations can be based on direct experience alone. This conclusion may have important implications for our understanding of the role of OFC in psychiatric disorders and how we may be able to treat them. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

A-learning: A new formulation of associative learning theory.

We present a new mathematical formulation of associative learning focused on non-human animals, which we call A-learning. Building on current animal learning theory and machine learning, A-learning is composed of two learning equations, one for stimulus-response values and one for stimulus values (conditioned reinforcement). A third equation implements decision-making by mapping stimulus-response values to response probabilities. We show that A-learning can reproduce the main features of: instrumental acquisition, including the effects of signaled and unsignaled non-contingent reinforcement; Pavlovian acquisition, including higher-order conditioning, omission training, autoshaping, and differences in form between conditioned and unconditioned responses; acquisition of avoidance responses; acquisition and extinction of instrumental chains and Pavlovian higher-order conditioning; Pavlovian-to-instrumental transfer; Pavlovian and instrumental outcome revaluation effects, including insight into why these effects vary greatly with training procedures and with the proximity of a response to the reinforcer. We discuss the differences between current theory and A-learning, such as its lack of stimulus-stimulus and response-stimulus associations, and compare A-learning with other temporal-difference models from machine learning, such as Q-learning, SARSA, and the actor-critic model. We conclude that A-learning may offer a more convenient view of associative learning than current mathematical models, and point out areas that need further development.

Eliot S. Hearst (1932-2018).

Presents an obituary for Eliot S. Hearst (1932-2018). Educated in psychology at Columbia University, Hearst received his Bachelor of Arts degree summa cum laude in 1953, his Master of Arts in 1954, and his doctoral degree in 1956, under William N. "Nat" Schoenfeld. From 1958 to 1964, Hearst was a Senior Experimental Psychologist at the Clinical Neuropharmacology Research Center, a joint unit of the National Institute of Mental Health (NIMH) and Saint Elizabeth's Hospital. Focused on conditioning, his experimental work included pharmacological, neuroanatomical, genetic, and biochemical correlates of behavior as well. Hearst was appointed professor of psychology in 1965 at the University of Missouri. In 1966, he was awarded his first NIMH grant, beginning a string of 22 annual renewals. Moving to Indiana University (IU) 5 years later, he taught courses in animal behavior, learning theory, and history and systems of psychology. An approachable yet demanding mentor, he patiently guided hundreds of students. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

An Analysis of Decision under Risk in Rats

In 1979, Daniel Kahneman and Amos Tversky published a ground-breaking paper titled "Prospect Theory: An Analysis of Decision under Risk," which presented a behavioral economic theory that accounted for the ways in which humans deviate from economists' normative workhorse model, Expected Utility Theory [1, 2]. For example, people exhibit probability distortion (they overweight low probabilities), loss aversion (losses loom larger than gains), and reference dependence (outcomes are evaluated as gains or losses relative to an internal reference point). We found that rats exhibited many of these same biases, using a task in which rats chose between guaranteed and probabilistic rewards. However, prospect theory assumes stable preferences in the absence of learning, an assumption at odds with alternative frameworks such as animal learning theory and reinforcement learning [3-7]. Rats also exhibited trial history effects, consistent with ongoing learning. A reinforcement learning model in which state-action values were updated by the subjective value of outcomes according to prospect theory reproduced rats' nonlinear utility and probability weighting functions and also captured trial-by-trial learning dynamics.

Context-dependent learning and causal structure.

The notion of "context" has played an important but complicated role in animal learning theory. Some studies have found that contextual stimuli (e.g., conditioning chamber) act much like punctate stimuli, entering into competition with other cues as would be predicted by standard associative learning theories. Other studies have found that contextual stimuli act more like "occasion setters," modulating the associative strength of punctate stimuli without themselves acquiring associative strength. Yet other studies have found that context is often largely ignored, resulting in transfer of performance across context changes. This article argues that these diverse functions of context arise in part from different causal interpretations of the environment. A Bayesian theory is presented that infers which causal interpretation best explains an animal's training history, and hence which function of context is appropriate. The theory coherently accounts for a number of disparate experimental results, and quantitatively predicts the results of a new experiment designed to directly test the theory.

Overcoming associative learning.

Thorndike (1898, 1911) rejected the idea that animal behavior was the consequence of reasoning, and suggested instead that the gradual acquisition of associations formed the basis of behavior-a contention that has had a significant impact on the development of animal learning theory. Despite this, comparative psychology provides a number of examples of behaviors that have been considered to be above and beyond the explanation of associative-, or reinforcement-learning mechanisms. These behaviors have motivated some researchers to propose higher-order cognitive abilities in animals, including (but not limited to) reasoning, sensitivity to ambiguity, and metacognition. However, other authors have questioned this claim, and provided alternative explanations for these behaviors from an associative perspective. With relevant examples, the steps that must be taken in order to overcome an associative explanation of behavior are described. (PsycINFO Database Record

Dopamine Reward Prediction Error Responses Reflect Marginal Utility

SummaryBackgroundOptimal choices require an accurate neuronal representation of economic value. In economics, utility functions are mathematical representations of subjective value that can be constructed from choices under risk. Utility usually exhibits a nonlinear relationship to physical reward value that corresponds to risk attitudes and reflects the increasing or decreasing marginal utility obtained with each additional unit of reward. Accordingly, neuronal reward responses coding utility should robustly reflect this nonlinearity.ResultsIn two monkeys, we measured utility as a function of physical reward value from meaningful choices under risk (that adhered to first- and second-order stochastic dominance). The resulting nonlinear utility functions predicted the certainty equivalents for new gambles, indicating that the functions’ shapes were meaningful. The monkeys were risk seeking (convex utility function) for low reward and risk avoiding (concave utility function) with higher amounts. Critically, the dopamine prediction error responses at the time of reward itself reflected the nonlinear utility functions measured at the time of choices. In particular, the reward response magnitude depended on the first derivative of the utility function and thus reflected the marginal utility. Furthermore, dopamine responses recorded outside of the task reflected the marginal utility of unpredicted reward. Accordingly, these responses were sufficient to train reinforcement learning models to predict the behaviorally defined expected utility of gambles.ConclusionsThese data suggest a neuronal manifestation of marginal utility in dopamine neurons and indicate a common neuronal basis for fundamental explanatory constructs in animal learning theory (prediction error) and economic decision theory (marginal utility).

Pigeons as a model species for cognitive neuroscience

Deeper understanding of the neuronal basis of behavior and its evolution requires investigation of model organisms taken from different taxonomic groups. The merits of this comparative approach are highlighted by research on birds: while their cognitive capacities have long been underestimated, research on avian model systems more recently has begun to provide central insights into the functional organization of the brain. In particular, domesticated homing pigeons (Columba livia) have been used as a model for the study of the psychological processes underlying learning, memory, and choice behavior, and much of current animal learning theory is based on findings with pigeons. Moreover, the vast amount of available behavioral and anatomical data has rendered the pigeon one of the key model species of behavioral and comparative neuroscience. This article illustrates some insights gained from research with pigeons with applicability beyond the class of aves.

Socially learned habituation to human observers in wild chimpanzees

Habituation to human observers is an essential tool in animal behaviour research. Habituation occurs when repeated and inconsequential exposure to a human observer gradually reduces an animal's natural aversive response. Despite the importance of habituation, little is known about the psychological mechanisms facilitating it in wild animals. Although animal learning theory offers some account, the patterns are more complex in natural than in laboratory settings, especially in large social groups in which individual experiences vary and individuals influence each other. Here, we investigate the role of social learning during the habituation process of a wild chimpanzee group, the Waibira community of Budongo Forest, Uganda. Through post hoc hypothesis testing, we found that the immigration of two well-habituated, young females from the neighbouring Sonso community had a significant effect on the behaviour of non-habituated Waibira individuals towards human observers, suggesting that habituation is partially acquired via social learning.

Reward Contexts Extend Dopamine Signals to Unrewarded Stimuli

SummaryBasic tenets of sensory processing emphasize the importance of accurate identification and discrimination of environmental objects [1]. Although this principle holds also for reward, the crucial acquisition of reward for survival would be aided by the capacity to detect objects whose rewarding properties may not be immediately apparent. Animal learning theory conceptualizes how unrewarded stimuli induce behavioral reactions in rewarded contexts due to pseudoconditioning and higher-order context conditioning [2, 3, 4, 5, 6]. We hypothesized that the underlying mechanisms may involve context-sensitive reward neurons. We studied short-latency activations of dopamine neurons to unrewarded, physically salient stimuli while systematically changing reward context. Dopamine neurons showed substantial activations to unrewarded stimuli and their conditioned stimuli in highly rewarded contexts. The activations decreased and often disappeared entirely with stepwise separation from rewarded contexts. The influence of reward context suggests that dopamine neurons respond to real and potential reward. The influence of reward context is compatible with the reward nature of phasic dopamine responses. The responses may facilitate rapid, default initiation of behavioral reactions in environments usually containing reward. Agents would encounter more and miss less reward, resulting in survival advantage and enhanced evolutionary fitness.

Use of Reinforcement Learning as a Challenge: A Review

Reinforcement learning has its origin from the animal learning theory. RL does not require prior knowledge but can autonomously get optional policy with the help of knowledge obtained by trial-and-error and continuously interacting with the dynamic environment. Due to its characteristics of self improving and online learning, reinforcement learning has become one of intelligent agent’s core technologies. This paper gives an introduction of reinforcement learning, discusses its basic model, the optimal policies used in RL , the main reinforcement optimal policy that are used to reward the agent including model free and model based policies – Temporal difference method, Q-learning , average reward, certainty equivalent methods, Dyna , prioritized sweeping , queue Dyna . At last but not the least this paper briefly describe the applications of reinforcement leaning and some of the future research scope in Reinforcement Learning. General Terms Reinforcement Learning, Q-Learning, temporal difference, robot control.

PROBLEMAS Y ENFOQUES DE LA PSICOLOGÍA COMPARADA DEL APRENDIZAJE

Comparative psychology originated during the 19th Century because of the influence of evolutionary theory and its application to the human species. It was to provide support for one of Darwin's central hypothesis: psychological continuity between species. After a period of rapid development, it concentrated in few species, an almost exclusive use of laboratory situations as analitical tools, an emphasis on proximal causes, and the exclusive influence of associationism in animal learning theory. During the sixties, comparative psychology was criticized on two grounds: 1) that the traditional anagenetic basis of comparative research had no basis in evolutionary biology; 2) that traditional principIes of learning were not general (e.g., acquired food aversions in rodents; song-Iearning in birds). A critical review of both issues is presented; the conclusion is that both fundamen­tal approaches (anagenesis and cladogenesis) to the comparative study of learning point to relatively independent problems and are, therefore, compelmentary rather than com­peting approaches.

Hippocampal learning and cognitive maps as products of hierarchical latent variable models

Event Abstract Back to Event Hippocampal learning and cognitive maps as products of hierarchical latent variable models Adam Johnson1*, Zachary Varberg1 and Paul Schrater2 1 Bethel University , United States 2 University of Minnesota, United States The hippocampus supports inference within early learning regimes and on a variety of complex spatial/context memory tasks. The defining characteristic of hippocampus dependent learning is that it provides the basis for a broad class of inferences in domains where sampling is limited relative to the potential dimensionality of inputs. Standard approaches to spatial learning and context learning have used neural network models to describe generalization behavior at the level of place cell maps and at the level of animal behavior. However, these models often overfit training data and compromise inference for novel test data. And in many cases, pattern classification dynamics observed in neural network models can be easily accommodated within probabilistic inference frameworks. We develop a model-based Bayesian approach to learning hierarchical latent variable structure and dynamics for spatial and context learning tasks. This approach expands probabilistic treatments of classical conditioning (Courville et al., 2003) to spatial learning domains for inference related to context dependent learning rules (e.g. "odor/reward position" paired-associate learning). These context dependent learning rules can be found by computing the posterior probabilities over latent variable hyperparameters. As a consequence, resultant behavioral inferences for novel test items reflect a mixture of latent models and are generally more robust. We model the spatial paired-associate task developed by Morris and colleagues (Day, Langston and Morris, 2003; Tse et al., 2007). Tse et al. (2007) showed that following initial training, rats learn new paired associates in a single trial. Single trial learning was dependent on the hippocampus and the consistency of the previously learned paired-associates within a given context. We show that initial training tunes the hyperparameter posterior density and consequently facilitates learning subsequent novel paired-associates - both in the search for newly learned paired-associate food-sites and for the avoidance of known food-sites following presentation of a novel odor. Furthermore, we show that the structure of the task requires the use of a full conjunctive representation for learning new paired-associates but does not require a full conjunctive representation for retrieval of previously learned paired-associates. Together these results show how the hippocampus facilitates new learning but is not always necessary for retrieval of previously learned information. Morris and colleagues suggest the hippocampus supports schema representations that facilitate new learning within a context. While this perspective has provided important insights into animal learning theory, it has lacked a computational counterpart. We suggest that hippocampus dependent schematic representations and cognitive maps are the consequence of a process that probabilistically infers hierarchical latent structure from input data. This statistical approach supports model optimization for inference across test data rather than model optimization across training data. As a result, the latent variable approach provides a much more robust inference engine than standard neural network models. Finally, the extraction of latent structure from input data allows for a simple explanation of hippocampal function and its important role within early learning regimes. Conference: Computational and Systems Neuroscience 2010, Salt Lake City, UT, United States, 25 Feb - 2 Mar, 2010. Presentation Type: Poster Presentation Topic: Poster session II Citation: Johnson A, Varberg Z and Schrater P (2010). Hippocampal learning and cognitive maps as products of hierarchical latent variable models. Front. Neurosci. Conference Abstract: Computational and Systems Neuroscience 2010. doi: 10.3389/conf.fnins.2010.03.00213 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 04 Mar 2010; Published Online: 04 Mar 2010. * Correspondence: Adam Johnson, Bethel University, St. Paul, United States, adam-johnson@bethel.edu Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Adam Johnson Zachary Varberg Paul Schrater Google Adam Johnson Zachary Varberg Paul Schrater Google Scholar Adam Johnson Zachary Varberg Paul Schrater PubMed Adam Johnson Zachary Varberg Paul Schrater Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Animal learning.

Pavlov and Thorndike pioneered the experimental study of animal learning and provided psychologists with powerful tools to unveil its underlying mechanisms. Today's research developments and theoretical analyses owe much to the pioneering work of these early investigators. Nevertheless, in the evolution of our knowledge about animal learning, some initial conceptions have been challenged and revised. We first review the original experimental procedures and findings of Pavlov and Thorndike. Next, we discuss critical research and consequent controversies which have greatly shaped animal learning theory. For example, although contiguity seemed to be the only condition that is necessary for learning, we now know that it is not sufficient; the conditioned stimulus (CS) also has to provide information about the occurrence of the unconditioned stimulus (US). Also, animals appear to learn different things about the same stimuli when circumstances vary. For instance, when faced with situations in which the meaning of a CS changes, as in the case of acquisition and later extinction, animals seem to preserve the original knowledge (CS-US) in addition to learning about the new conditions (CS-noUS). Finally, we discuss how parallels among Pavlovian conditioning, operant conditioning, and human causal judgment suggest that causal knowledge may lie at the root of both human and animal learning. All of these empirical findings and theoretical developments prove that animal learning is more complex and intricate than was once imagined. Copyright © 2009 John Wiley & Sons, Ltd. For further resources related to this article, please visit the WIREs website.

An Infinite Mixture Model of Context-dependent Learning and Extinction

Event Abstract Back to Event An Infinite Mixture Model of Context-dependent Learning and Extinction Dominant theories of human and animal learning posit that in classical conditioning, one of the most basic forms of learning, animals learn to make associations between different stimuli in the environment. These associations are then used to predict future events based on current observations. Although this framework is powerful, we argue that a wealth of behavioral evidence is more consistent with an account of learning in which animals make more complex inferences about the environment. Most notably, responding to a conditioned stimulus appears to depend not only on the reinforcement history of the stimulus but also on its relationship to contextual stimuli (such as the experimental chamber). We thus model learning as inference over latent causes of observed stimuli, rather than of direct causal relationships between them. We focus on two extensively-studied animal conditioning paradigms, latent inhibition and renewal. In latent inhibition, the acquisition of a conditioned response to a cue that is paired with reinforcement is retarded if the cue was previously presented without reinforcement. A change of context between the initial exposure to the unreinforced cue and the subsequent training reduces this retardation. In renewal, the cue is first presented paired with reinforcement (conditioning) in one context and then presented without reinforcement (extinction) in another context, leading to loss of conditioned responding. Returning the animal to the conditioning context renews conditioned responding. Renewal also occurs when the animal is tested in a completely novel context. In both latent inhibition and renewal, lesioning the hippocampus prior to the second phase of training eliminates context-dependence. A similar pattern is observed in infant rats, whose behavior seems to not be sensitive to context. We suggest that these and other context-dependent learning phenomena can be explained by a normative statistical model in which the animal performs inference of latent causes in an infinite mixture model. In contrast to classical associative theories, the infinite mixture model assumes that each observation (trial) is generated by a single latent cause, selected from an unbounded set of latent causes. This identifies the computational problem being solved by the animal as that of how to assign observations to discrete latent causes (ie, clustering) and use these to predict reinforcement. The infinite capacity of the model enables the animal to infer new clusters as it collects observations. Our simulations show that the model predicts the context-dependence of latent inhibition and renewal. We also show how restricting the model's ability to infer new clusters explains developmental changes in learning as well as the effects of hippocampal damage. The key difference between the infinite mixture model and previous models is the explanation of learning as inference over latent causes rather than the formation of associations between observed stimuli, giving rise to novel predictions regarding the effects of training length on learning and context effects, and suggesting manipulations that could reverse behavioral phenomena that were previously regarded as fixed constraints on learning. The model also has implications for understanding context-dependent relaspe in drug addiction and anxiety disorders. Conference: Computational and systems neuroscience 2009, Salt Lake City, UT, United States, 26 Feb - 3 Mar, 2009. Presentation Type: Oral Presentation Topic: Oral Presentations Citation: (2009). An Infinite Mixture Model of Context-dependent Learning and Extinction. Front. Syst. Neurosci. Conference Abstract: Computational and systems neuroscience 2009. doi: 10.3389/conf.neuro.06.2009.03.323 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 04 Feb 2009; Published Online: 04 Feb 2009. Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Google Google Scholar PubMed Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Hearth and Hotmail

Hearth and Hotmail

Reinforcement learning: Computational theory and biological mechanisms

Reinforcement learning is a computational framework for an active agent to learn behaviors on the basis of a scalar reward signal. The agent can be an animal, a human, or an artificial system such as a robot or a computer program. The reward can be food, water, money, or whatever measure of the performance of the agent. The theory of reinforcement learning, which was developed in an artificial intelligence community with intuitions from animal learning theory, is now giving a coherent account on the function of the basal ganglia. It now serves as the "common language" in which biologists, engineers, and social scientists can exchange their problems and findings. This article reviews the basic theoretical framework of reinforcement learning and discusses its recent and future contributions toward the understanding of animal behaviors and human decision making.

Neurophysiology of Motivated Learning: Adaptive Mechanisms Underlying Cognitive Bias in Depression

Cognitive Behavior Therapy for depression focuses on changing cognitive appraisals of the self and life events. This approach contrasts with behaviorist approaches to clinical disorders, which focused on elementary learning explanations and assumed that cognitive mediation is irrelevant. A better integration of behavioral analysis with cognitive therapy could be facilitated by the theoretical advances in animal learning theory over the last several decades. Animals have been found to respond to events in relation to mental representations of the reward value of those events. The evidence now shows that learning in mammals cannot be explained by passive behaviorist association principles, but requires a theoretical understanding of the animal’s hedonic expectancies and its adaptive response to discrepancies from those expectancies. The mechanisms of cognitive representation and information handling that are implied by modern learning theory have presented clear and specific challenges to neurophysiological theories of motivational control of learning and memory. The mechanisms of human motivational control appear to be similar to those of other mammals, suggesting that progress with understanding neurophysiological mechanisms of hedonic evaluation, frustration, and action regulation will soon provide the opportunity for a comprehensive neuropsychological model to guide diagnosis and treatment of depressive disorders.