Value Of Available Options Research Articles

Preferences reveal dissociable encoding across prefrontal-limbic circuits.

Individual preferences for the flavor of different foods and fluids exert a strong influence on behavior. Most current theories posit that preferences are integrated with other state variables in the orbitofrontal cortex (OFC), which is thought to derive the relative subjective value of available options to guide choice behavior. Here, we report that instead of a single integrated valuation system in the OFC, another complementary one is centered in the ventrolateral prefrontal cortex (vlPFC) in macaques. Specifically, we found that the OFC and vlPFC preferentially represent outcome flavor and outcome probability, respectively, and that preferences are separately integrated into value representations in these areas. In addition, the vlPFC, but not the OFC, represented the probability of receiving the available outcome flavors separately, with the difference between these representations reflecting the degree of preference for each flavor. Thus, both the vlPFC and OFC exhibit dissociable but complementary representations of subjective value, both of which are necessary for decision-making.

An internal expectation guides Drosophila egg-laying decisions.

To better understand how animals make ethologically relevant decisions, we studied egg-laying substrate choice in Drosophila. We found that flies dynamically increase or decrease their egg-laying rates while exploring substrates so as to target eggs to the best, recently visited option. Visiting the best option typically yielded inhibition of egg laying on other substrates for many minutes. Our data support a model in which flies compare the current substrate's value with an internally constructed expectation on the value of available options to regulate the likelihood of laying an egg. We show that dopamine neuron activity is critical for learning and/or expressing this expectation, similar to its role in certain tasks in vertebrates. Integrating sensory experiences over minutes to generate an estimate of the quality of available options allows flies to use a dynamic reference point for judging the current substrate and might be a general way in which decisions are made.

Value representations in the rodent orbitofrontal cortex drive learning, not choice.

Humans and animals make predictions about the rewards they expect to receive in different situations. In formal models of behavior, these predictions are known as value representations, and they play two very different roles. Firstly, they drive choice: the expected values of available options are compared to one another, and the best option is selected. Secondly, they support learning: expected values are compared to rewards actually received, and future expectations are updated accordingly. Whether these different functions are mediated by different neural representations remains an open question. Here, we employ a recently developed multi-step task for rats that computationally separates learning from choosing. We investigate the role of value representations in the rodent orbitofrontal cortex, a key structure for value-based cognition. Electrophysiological recordings and optogenetic perturbations indicate that these representations do not directly drive choice. Instead, they signal expected reward information to a learning process elsewhere in the brain that updates choice mechanisms.

The orbitofrontal cortex in temporal cognition.

One of the most important factors in decision-making is estimating the value of available options. Subregions of the prefrontal cortex, including the orbitofrontal cortex (OFC), have been deemed essential for this process. Value computations require a complex integration across numerous dimensions, including, reward magnitude, effort, internal state, and time. The importance of the temporal dimension is well illustrated by temporal discounting tasks, in which subjects select between smaller-sooner versus larger-later rewards. The specific role of OFC in telling time and integrating temporal information into decision-making remains unclear. Based on the current literature, in this review we reevaluate current theories of OFC function, accounting for the influence of time. Incorporating temporal information into value estimation and decision-making requires distinct, yet interrelated, forms of temporal information including the ability to tell time, represent time, create temporal expectations, and the ability to use this information for optimal decision-making in a wide range of tasks, including temporal discounting and wagering. We use the term "temporal cognition" to refer to the integrated use of these different aspects of temporal information. We suggest that the OFC may be a critical site for the integration of reward magnitude and delay, and thus important for temporal cognition. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Reward foraging task and model-based analysis reveal how fruit flies learn value of available options.

Foraging animals have to evaluate, compare and select food patches in order to increase their fitness. Understanding what drives foraging decisions requires careful manipulation of the value of alternative options while monitoring animals choices. Value-based decision-making tasks in combination with formal learning models have provided both an experimental and theoretical framework to study foraging decisions in lab settings. While these approaches were successfully used in the past to understand what drives choices in mammals, very little work has been done on fruit flies. This is despite the fact that fruit flies have served as model organism for many complex behavioural paradigms. To fill this gap we developed a single-animal, trial-based decision making task, where freely walking flies experienced optogenetic sugar-receptor neuron stimulation. We controlled the value of available options by manipulating the probabilities of optogenetic stimulation. We show that flies integrate reward history of chosen options and forget value of unchosen options. We further discover that flies assign higher values to rewards experienced early in the behavioural session, consistent with formal reinforcement learning models. Finally, we also show that the probabilistic rewards affect walking trajectories of flies, suggesting that accumulated value is controlling the navigation vector of flies in a graded fashion. These findings establish the fruit fly as a model organism to explore the genetic and circuit basis of reward foraging decisions.

Investigating the origin and consequences of endogenous default options in repeated economic choices.

Classical value-based decision theories state that economic choices are solely based on the value of available options. Experimental evidence suggests, however, that individuals' choices are biased towards default options, prompted by the framing of decisions. Although the effects of default options created by exogenous framing-such as how choice options are displayed-are well-documented, little is known about the potential effects and properties of endogenous framing, that is, originating from an individual's internal state. In this study, we investigated the existence and properties of endogenous default options in a task involving choices between risky lotteries. By manipulating and examining the effects of three experimental features-time pressure, time spent on task and relative choice proportion towards a specific option-, we reveal and dissociate two features of endogenous default options which bias individuals' choices: a natural tendency to prefer certain types of options (natural default), and the tendency to implicitly learn a default option from past choices (learned default). Additional analyses suggest that while the natural default may bias the standard choice process towards an option category, the learned default effects may be attributable to a second independent choice process. Overall, these investigations provide a first experimental evidence of how individuals build and apply diverse endogenous default options in economic decision-making and how this biases their choices.

The Value of Persistent Value

In this issue of Neuron, Bari etal. (2019) show that neurons in medial frontal cortex, but not a nearby premotor area, encode the relative value of available options with long-lasting persistent activity states during naturalistic foraging. These long-lasting activity states serve to preferentially guide choices to more likely rewarded options.

Area-Specificity and Plasticity of History-Dependent Value Coding During Learning

Decision making is often driven by the subjective value of available options, a value which is formed through experience. To support this fundamental behavior, the brain must encode and maintain the subjective value. To investigate the area specificity and plasticity of value coding, we trained mice in a value-based decision task and imaged neural activity in 6 cortical areas with cellular resolution. History- and value-related signals were widespread across areas, but their strength and temporal patterns differed. Inexpert mice, the retrosplenial cortex (RSC) uniquely encoded history- and value-related signals with persistent population activity patterns across trials. This unique encoding of RSC emerged during task learning with a strong increase in more distant historysignals. Acute inactivation of RSC selectively impaired the reward-history-based behavioral strategy. Our results indicate that RSC flexibly changes its history coding and persistently encodes value-related signals to support adaptive behaviors.

Ventromedial Prefrontal Cortex Tracks Multiple Environmental Variables during Search.

To make efficient foraging decisions, we must combine information about the values of available options with nonvalue information. Some accounts of ventromedial PFC (vmPFC) suggest that it has a narrow role limited to evaluating immediately available options. We examined responses of neurons in area 14 (a putative macaque homolog of human vmPFC) as 2 male macaques performed a novel foraging search task. Although many neurons encoded the values of immediately available offers, they also independently encoded several other variables that influence choice, but that are conceptually distinct from offer value. These variables include average reward rate, number of offers viewed per trial, previous offer values, previous outcome sizes, and the locations of the currently attended offer. We conclude that, rather than serving as specialized economic value center, vmPFC plays a broad role in integrating relevant environmental information to drive foraging decisions.SIGNIFICANCE STATEMENT Decision makers must often choose whether to take an immediately available option or continue to search for a better one. We hypothesized that this process, which is integral to foraging theory, leaves neural signatures in the brain region ventromedial PFC. Subjects performed a novel foraging task in which they searched through differently valued options and attempted to balance their reward threshold with various time costs. We found that neurons not only encode the values of immediately available offers, but multiplexed these with environmental variables, including reward rate, number of offers viewed, previous offer values, and spatial information. We conclude that vmPFC plays a rich role in encoding and integrating multiple foraging-related variables during economic decisions.

Decisions in Motion: Decision Dynamics during Intertemporal Choice reflect Subjective Evaluation of Delayed Rewards.

People tend to discount rewards or losses that occur in the future. Such delay discounting has been linked to many behavioral and health problems, since people choose smaller short-term gains over greater long-term gains. We investigated whether the effect of delays on the subjective value of rewards is expressed in how people move when they make choices. Over 600 patrons of the RISK LAB exhibition hosted by the Science Gallery DublinTM played a short computer game in which they used a computer mouse to choose between amounts of money at various delays. Typical discounting effects were observed and decision dynamics indicated that choosing smaller short-term rewards became easier (i.e., shorter response times, tighter trajectories, less vacillation) as the delays until later rewards increased. Based on a sequence of choices, subjective values of delayed outcomes were estimated and decision dynamics during initial choices predicted these values. Decision dynamics are affected by subjective values of available options and thus provide a means to estimate such values.

Dopaminergic Drugs Modulate Learning Rates and Perseveration in Parkinson's Patients in a Dynamic Foraging Task

Making appropriate choices often requires the ability to learn the value of available options from experience. Parkinson's disease is characterized by a loss of dopamine neurons in the substantia nigra, neurons hypothesized to play a role in reinforcement learning. Although previous studies have shown that Parkinson's patients are impaired in tasks involving learning from feedback, they have not directly tested the widely held hypothesis that dopamine neuron activity specifically encodes the reward prediction error signal used in reinforcement learning models. To test a key prediction of this hypothesis, we fit choice behavior from a dynamic foraging task with reinforcement learning models and show that treatment with dopaminergic drugs alters choice behavior in a manner consistent with the theory. More specifically, we found that dopaminergic drugs selectively modulate learning from positive outcomes. We observed no effect of dopaminergic drugs on learning from negative outcomes. We also found a novel dopamine-dependent effect on decision making that is not accounted for by reinforcement learning models: perseveration in choice, independent of reward history, increases with Parkinson's disease and decreases with dopamine therapy.

Adapting to Dynamic Stimulus-Response Values: Differential Contributions of Inferior Frontal, Dorsomedial, and Dorsolateral Regions of Prefrontal Cortex to Decision Making

Dorsomedial prefrontal cortex (dmPFC), dorsolateral prefrontal cortex (dlPFC), and inferior frontal gyrus (IFG) have all been implicated in resolving decision conflict whether this conflict is generated by having to select between responses of similar value or by making selections following a reversal in reinforcement contingencies. However, work distinguishing their individual functional contributions remains preliminary. We used functional magnetic resonance imaging to delineate the functional role of these systems with regard to both forms of decision conflict. Within dmPFC and dlPFC, blood oxygen level-dependent responses increased in response to decision conflict regardless of whether the conflict occurred in the context of a reduction in the difference in relative value between objects, or an error following a reversal of reinforcement contingencies. Conjunction analysis confirmed that overlapping regions of dmPFC and dlPFC were activated by both forms of decision conflict. Unlike these regions, however, activity in IFG was not modulated by reductions in the relative value of available options. Moreover, although all three regions of prefrontal cortex showed enhanced activity to reversal errors, only dmPFC and dlPFC were also modulated by the magnitude of value change during the reversal. These data are interpreted with reference to models of dmPFC, dlPFC, and IFG functioning.

Computational implications of a normalized value representation in decision circuits

Event Abstract Back to Event Computational implications of a normalized value representation in decision circuits Value information is a critical component of decision-making, and a growing number of studies report value-related signals in decision-related neural circuits. Decision theorists have long hypothesized that choices are guided by a relative reward representation, but the specific form in which the brain encodes value remains unclear. Using two- and three-target choice tasks, our recent work suggests that visuomotor neurons in the monkey lateral intraparietal area (LIP) encode a relative measure of saccade value, normalized across available options: activity is negatively correlated to target values outside the response field while positively correlated to target value in the response field. Interestingly, this value normalization is well-described by a divisive normalization model that also characterizes nonlinear phenomena such as gain control and cross-orientation suppression in visual cortex: FR = K*[B+V] / [S+Vt] where the firing rate of an LIP neuron (FR) is a function of the value associated with a saccade to the target in the response field (V), the total value of all available options (Vt), and three additional parameters (K,B,S). How might such a value-coding scheme affect choice behavior? Here, we explore the computational implications of divisive normalization for value-based decision-making. Specifically, we examine how relative reward encoding interacts with noisy population estimates of value and the resulting effects on choice. The motivating intuition is straightforward: as the total value of available options increases, the separation between the distributions of firing rates representing two differently valued options decreases; if variance doesn’t decrease appropriately, the options will be increasingly difficult to distinguish. To examine this computationally, we examine the choice between two differently valued options as additional lower valued options are added to the choice set. We simulate the decision process as the selection of the highest valued option, with the value of each option drawn from a distribution of firing rates around the option-specific mean firing rate. The mean rate for each option is determined using the value-based divisive normalization equation, with parameters fit to the observed monkey physiological data. Noisy population coding is modeled by including two general sources of noise: 1) a mean-rate dependent noise term, which models the considerable intrinsic response variability observed in cortical neurons (response variance typically ~1-1.5 times the mean response), and 2) a mean-rate independent additive noise term. We find that the presence of either form of noise leads to increasing errors as the choice set size increases, and characterize this phenomenon as a function of the number and value of additional alternatives as well as different degrees of noise. We conclude that a relative representation of reward, mediated by divisive normalization, can lead to suboptimal choice behavior. These results suggest a computational explanation for why and how choice behavior responds to multiple options and changing choice sets, and highlight the importance of understanding both the specific mechanism of cortical representation and the nature of the population code. Conference: Computational and systems neuroscience 2009, Salt Lake City, UT, United States, 26 Feb - 3 Mar, 2009. Presentation Type: Poster Presentation Topic: Poster Presentations Citation: (2009). Computational implications of a normalized value representation in decision circuits. Front. Syst. Neurosci. Conference Abstract: Computational and systems neuroscience 2009. doi: 10.3389/conf.neuro.06.2009.03.218 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 03 Feb 2009; Published Online: 03 Feb 2009. Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Google Google Scholar PubMed Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.