Reward Uncertainty Research Articles

Uncertainty of treatment efficacy moderates placebo effects on reinforcement learning

The placebo-reward hypothesis postulates that positive effects of treatment expectations on health (i.e., placebo effects) and reward processing share common neural underpinnings. Moreover, experiments in humans and animals indicate that reward uncertainty increases striatal dopamine, which is presumably involved in placebo responses and reward learning. Therefore, treatment uncertainty analogously to reward uncertainty may affect updating from rewards after placebo treatment. Here, we address whether different degrees of uncertainty regarding the efficacy of a sham treatment affect reward sensitivity. In an online between-subjects experiment with N = 141 participants, we systematically varied the provided efficacy instructions before participants first received a sham treatment that consisted of listening to binaural beats and then performed a probabilistic reinforcement learning task. We fitted a Q-learning model including two different learning rates for positive (gain) and negative (loss) reward prediction errors and an inverse gain parameter to behavioral decision data in the reinforcement learning task. Our results yielded an inverted-U-relationship between provided treatment efficacy probability and learning rates for gain, such that higher levels of treatment uncertainty, rather than of expected net efficacy, affect presumably dopamine-related reward learning. These findings support the placebo-reward hypothesis and suggest harnessing uncertainty in placebo treatment for recovering reward learning capabilities.

Novelty and uncertainty differentially drive exploration across development.

Across the lifespan, individuals frequently choose between exploiting known rewarding options or exploring unknown alternatives. A large body of work has suggested that children may explore more than adults. However, because novelty and reward uncertainty are often correlated, it is unclear how they differentially influence decision-making across development. Here, children, adolescents, and adults (ages 8-27 years, N = 122) completed an adapted version of a recently developed value-guided decision-making task that decouples novelty and uncertainty. In line with prior studies, we found that exploration decreased with increasing age. Critically, participants of all ages demonstrated a similar bias to select choice options with greater novelty, whereas aversion to reward uncertainty increased into adulthood. Computational modeling of participant choices revealed that whereas adolescents and adults demonstrated attenuated uncertainty aversion for more novel choice options, children's choices were not influenced by reward uncertainty.

Unpredictable, intermittent access to sucrose or water promotes increased reward pursuit in rats

Reward uncertainty can sensitize reward pathways, promoting increased reward-seeking and -taking behaviours. This is relevant to human conditions such as pathological gambling, eating disorders and drug addiction. In the context of addiction, preclinical self-administration procedures have been developed to model the intermittency of human drug use. These intermittent-access (IntA) procedures involve intermittent but predictable access to drug during self-administration sessions. However, human drug use typically involves intermittent and unpredictable drug access. We introduce a new procedure modeling unpredictable, intermittent access (UIntA) to a reinforcer, and we compare it to procedures that provide predictable reinforcer availability; continuous (ContA) or intermittent (IntA) access. Female rats self-administered water or liquid sucrose in daily hour-long sessions. IntA and ContA rats had access to a fixed volume of water or sucrose (0.1 ml), under a fixed ratio 3 schedule of reinforcement. IntA rats had predictable 5-min reinforcer ON and 25-min reinforcer OFF periods. ContA rats had 60 min of reinforcer access during each session. For UIntA rats, variation in the length of ON and OFF periods (1, 5 or 9 min/period), response requirement (variable ratio 3 schedule of reinforcement), reinforcer probability (50%) and quantity (0, 0.1 or 0.2 ml) introduced reward uncertainty. Following 14 daily self-administration sessions, UIntA rats showed the highest levels of responding for water or sucrose under progressive ratio conditions, responding under extinction conditions, and cue-induced reinstatement of sucrose seeking. Thus, unpredictable, intermittent reward access promotes increased reward pursuit. This has implications for modeling addiction and other disorders of increased reward seeking.

Dopaminergic and prefrontal dynamics co-determine mouse decisions in a spatial gambling task

The neural mechanisms by which animals initiate goal-directed actions, choose between options, or explore opportunities remain unknown. Here, we develop a spatial gambling task in which mice, to obtain intracranial self-stimulation rewards, self-determine the initiation, direction, vigor, and pace of their actions based on their knowledge of the outcomes. Using electrophysiological recordings, pharmacology, and optogenetics, we identify a sequence of oscillations and firings in the ventral tegmental area (VTA), orbitofrontal cortex (OFC), and prefrontal cortex (PFC) that co-encodes and co-determines self-initiation and choices. This sequence appeared with learning as an uncued realignment of spontaneous dynamics. Interactions between the structures varied with the reward context, particularly the uncertainty associated with the different options.We suggest that self-generated choices arise from a distributed circuit based on an OFC-VTA core determining whether to wait for or initiate actions, while the PFC is specifically engaged by reward uncertainty in action selection and pace.

Token Incentives in a Volatile Crypto Market: The Effects of Token Price Volatility on User Contribution

ABSTRACT Crypto tokens, issued and managed via smart contracts, function as rewards in blockchain systems to encourage user participation. Distinct from monetary incentives, token incentives are uncertain in reward magnitude due to the large swings in token prices on crypto markets. By focusing on token price volatility, this study investigates how the reward uncertainty affects user contribution in a tokenized digital platform. Our empirical setting is Steemit, a platform where bloggers write posts and share token rewards based on their posts’ popularity. We find that while high token price volatility induces a large volume of blog posts, it diminishes post quality. The dichotomous effects are explained via two mechanisms: users’ direct reactions to reward uncertainties and their indirect reactions, mediated by altered token preference amid volatility shocks. Deeply exploring this dynamic process, our results reveal that token price volatility facilitates a platform’s network short-term effect but impairs long-term user creativity. Our empirical findings thus extend the literature on blockchain economics and cryptocurrencies and have practical implications for the design of incentive mechanisms on tokenized digital platforms.

Reinforcement Learning Under Uncertainty: Expected Versus Unexpected Uncertainty and State Versus Reward Uncertainty

Two prominent types of uncertainty that have been studied extensively are expected and unexpected uncertainty. Studies suggest that humans are capable of learning from reward under both expected and unexpected uncertainty when the source of variability is the reward. How do people learn when the source of uncertainty is the environment’s state and the rewards themselves are deterministic? How does their learning compare with the case of reward uncertainty? The present study addressed these questions using behavioural experimentation and computational modelling. Experiment 1 showed that human subjects were generally able to use reward feedback to successfully learn the task rules under state uncertainty, and were able to detect a non-signalled reversal of stimulus-response contingencies. Experiment 2, which combined all four types of uncertainties—expected versus unexpected uncertainty, and state versus reward uncertainty—highlighted key similarities and differences in learning between state and reward uncertainties. We found that subjects performed significantly better in the state uncertainty condition, primarily because they explored less and improved their state disambiguation. We also show that a simple reinforcement learning mechanism that ignores state uncertainty and updates the state-action value of only the identified state accounted for the behavioural data better than both a Bayesian reinforcement learning model that keeps track of belief states and a model that acts based on sampling from past experiences. Our findings suggest a common mechanism supports reward-based learning under state and reward uncertainty.

Temporal Dynamics Underlying Prelimbic Prefrontal Cortical Regulation of Action Selection and Outcome Evaluation during Risk/Reward Decision-Making.

Risk/reward decision-making is a dynamic process that includes periods of deliberation before action selection and evaluation of the action outcomes that bias subsequent choices. Inactivation of the prelimbic (PL) cortex has revealed its integral role in updating decision biases in the face of changes in probabilistic reward contingencies, yet how phasic PL signals during different phases of the decision process influence choice remains unclear. We used temporally specific optogenetic inhibition to selectively disrupt PL activity coinciding with action selection and outcome phases to examine how these signals influence choice. Male rats expressing the inhibitory opsin eArchT within PL excitatory neurons were well trained on a probabilistic discounting task, entailing choice between small/certain versus large/risky rewards, the probability of which varied over a session (50-12.5%). During testing, brief light pulses suppressed PL activity before choice or after different outcomes. Prechoice suppression reduced bias toward more preferred/higher utility options and disrupted how recent outcomes influenced subsequent choice. Inhibition during risky losses induced a similar profile, but here, the impact of reward omissions were either amplified or diminished, relative to the context of the estimated profitability of the risky option. Inhibition during large or small reward receipt reduced risky choice when this option was more profitable, suggesting these signals can both reinforce rewarded risky choices and also act as a relative value comparator signal that augments incentive for larger rewards. These findings reveal multifaceted contributions by the PL in implementing decisions and integrating action-outcome feedback to assign context to the decision space.SIGNIFICANCE STATEMENT The PL prefrontal cortex plays an integral role in guiding risk/reward decisions, but how activity in this region during different phases of the decision process influences choice is unclear. By using temporally specific optogenetic manipulations of this activity, the present study unveiled previously uncharacterized and differential contributions by PL in implementing decision policies and how evaluation of decision outcomes shape subsequent choice. These findings provide novel insight into the dynamic processes engaged by the PL that underlie action selection in situations involving reward uncertainty that may aid in understanding the mechanism underlying normal and aberrant decision-making processes.

Drug-Induced Gambling Disorder: Epidemiology, Neurobiology, and Management.

Problematic gambling has been suggested to be a possible consequence of dopaminergic medications used mainly in neurological conditions, i.e. pramipexole and ropinirole, and possibly by one antipsychotic compound, aripiprazole. Patients with Parkinson's disease, restless legs syndrome and other conditions potentially treated with dopamine agonists, as well as patients treated for psychotic disorders, are vulnerable patient groups with theoretically increased risk of developing gambling disorder (GD), for example due to higher rates of mental ill-health in these groups. The aim of the present paper is to review the epidemiological, clinical, and neurobiological evidence of the association between dopaminergic medications and GD, and to describe risk groups and treatment options. The neurobiology of GD involves the reward and reinforcement system, based mainly on mesocorticolimbic dopamine projections, with the nucleus accumbens being a crucial area for developing addictions to substances and behaviors. The addictive properties of gambling can perhaps be explained by the reward uncertainty that activates dopamine signaling in a pathological manner. Since reward-related learning is mediated by dopamine, it can be altered by dopaminergic medications, possibly leading to increased gambling behavior and a decreased impulse control. A causal relationship between the medications and GD seems likely, but the molecular mechanisms behind this association have not been fully described yet. More research is needed in order to fully outline the clinical picture of GD developing in patient groups with dopaminergic medications, and data are needed on the differentiation of risk in different compounds. In addition, very few interventional studies are available on the management of GD induced by dopaminergic medications. While GD overall can be treated, there is need for treatment studies testing the effectiveness of tapering of the medication or other gambling-specific treatment modalities in these patient groups.

Reward Uncertainty and Expected Value Enhance Generalization of Episodic Memory.

Previous research has revealed some mechanisms underlying the generalization of reward expectation of generalization stimuli, but little is known about the generalization of episodic memory for rewarding events, its consolidation, and how reward components such as expected value and reward uncertainty affect it. Participants underwent a Pavlovian reward-conditioning task to test whether reward conditioning would enhance episodic memory generalization and which reward components would directly affect it. Counterbalanced across participants, one semantic category was paired with a reward, while the other was never paired. Following a delay of either 5 min or 24 h, participants took a memory test consisting of old, highly similar, and new items. We found that participants were more likely to falsely recognize lure items as old in the reward-paired category after 5 min and 24 h delays. These results indicate that reward conditioning enhanced the generalization of episodic memory, but this effect was not necessarily dependent on consolidation. The composite score and raw data of generalization further showed that the uncertainty and expected value enhanced generalization. Together, these findings revealed an effect of reward conditioning on episodic memory generalization and supported the enhancement effects of expected value and uncertainty.

Characterization of continuum robot arms under reinforcement learning and derived improvements.

In robotics, soft continuum robot arms are a promising prospect owing to their redundancy and passivity; however, no comprehensive study exists that examines their characteristics compared to rigid manipulators. In this study, we examined the advantages of a continuum robot arm as compared to a typical and rigid seven-degree-of-freedom (7-DoF) robot manipulator in terms of performing various tasks through reinforcement learning. We conducted simulations for tasks with different characteristics that require control over position and force. Common tasks in robot manipulators, such as reaching, crank rotation, object throwing, and peg-in-hole were considered. The initial conditions of the robot and environment were randomized, aiming for evaluations including robustness. The results indicate that the continuum robot arm excels in the crank-rotation task, which is characterized by uncertainty in environmental conditions and cumulative rewards. However, the rigid robot arm learned better motions for the peg-in-hole task than the other tasks, which requires fine motion control of the end-effector. In the throwing task, the continuum robot arm scored well owing to its good handling of anisotropy. Moreover, we developed a reinforcement-learning method based on the comprehensive experimental results. The proposed method successfully improved the motion learning of a continuum robot arm by adding a technique to regulate the initial state of the robot. To the best of our knowledge, ours is the first reinforcement-learning experiment with multiple tasks on a single continuum robot arm and is the first report of a comparison between a single continuum robot arm and rigid manipulator on a wide range of tasks. This simulation study can make a significant contribution to the design of continuum arms and specification of their applications, and development of control and reinforcement learning methods.

Dissecting the Neural Substrate of Decreased Reward Sensitivity in a Model of Chronic Pain

Chronic pain is a debilitating condition that shades the lives of approximately 20% of the global population. Patients with chronic pain often display impairments in cognitive function, such as cognitive inflexibility, that negatively affect treatment efficacy. Understanding the mechanism underlying chronic pain and its behavioral impacts is therefore crucial for the development of successful treatments on symptoms beyond the physical aspect. However, current study on the circuitous underpinning of pain-associated behavioral inflexibility is still insufficient. Chronic pain is associated with a hypodopaminergic state in patients and animal models; the medial prefrontal cortex (mPFC) and nucleus accumbens shell (NAcSh) are two brain regions that receive dense dopaminergic innervation, and dopamine exquisitely modulates synaptic plasticity and excitability in this pathway. Projections from the medial prefrontal cortex (mPFC) to nucleus accumbens shell (NAcSh) specifically are essential for encoding reward associations and flexibly updating behavior in response to reward. Therefore, we hypothesize that a pain-associated hypodopaminergic state would modulate function of the mPFC–NAcSh pathway and lead to impairment in reward-related behavioral flexibility. Whether maladaptive plasticity of the mPFC–NAc pathway contributes to the behavioral inflexibility associated with chronic pain has not been well understood. Here, we will test the hypothesis that decreased synaptic strength of the mPFC–NAcSh pathway leads to decreased reward-based effort allocation following induction of a chronic pain and impaired behavioral flexibility under conditions of reward uncertainty. These behavioral deficits were associated with decreased excitatory synaptic transmission into the nucleus accumbens in a cell-type-specific manner, providing a potential neural substrate of these effects. Together, these results provide evidence linking a pain-associated hypodopaminergic state to downstream dopamine-dependent adaptations in cortico-striatal circuits that could contribute to impairments in reward-based behavior. In the future, it is possible that this pathway could be harnessed for neuromodulation strategies to treat cognitive or affective symptoms of chronic pain. Grant support from Rita Allen Scholar Award in Pain (to M.C.C.). Chronic pain is a debilitating condition that shades the lives of approximately 20% of the global population. Patients with chronic pain often display impairments in cognitive function, such as cognitive inflexibility, that negatively affect treatment efficacy. Understanding the mechanism underlying chronic pain and its behavioral impacts is therefore crucial for the development of successful treatments on symptoms beyond the physical aspect. However, current study on the circuitous underpinning of pain-associated behavioral inflexibility is still insufficient. Chronic pain is associated with a hypodopaminergic state in patients and animal models; the medial prefrontal cortex (mPFC) and nucleus accumbens shell (NAcSh) are two brain regions that receive dense dopaminergic innervation, and dopamine exquisitely modulates synaptic plasticity and excitability in this pathway. Projections from the medial prefrontal cortex (mPFC) to nucleus accumbens shell (NAcSh) specifically are essential for encoding reward associations and flexibly updating behavior in response to reward. Therefore, we hypothesize that a pain-associated hypodopaminergic state would modulate function of the mPFC–NAcSh pathway and lead to impairment in reward-related behavioral flexibility. Whether maladaptive plasticity of the mPFC–NAc pathway contributes to the behavioral inflexibility associated with chronic pain has not been well understood. Here, we will test the hypothesis that decreased synaptic strength of the mPFC–NAcSh pathway leads to decreased reward-based effort allocation following induction of a chronic pain and impaired behavioral flexibility under conditions of reward uncertainty. These behavioral deficits were associated with decreased excitatory synaptic transmission into the nucleus accumbens in a cell-type-specific manner, providing a potential neural substrate of these effects. Together, these results provide evidence linking a pain-associated hypodopaminergic state to downstream dopamine-dependent adaptations in cortico-striatal circuits that could contribute to impairments in reward-based behavior. In the future, it is possible that this pathway could be harnessed for neuromodulation strategies to treat cognitive or affective symptoms of chronic pain. Grant support from Rita Allen Scholar Award in Pain (to M.C.C.).

How firms can create delightful customer experience? Contrasting roles of future reward uncertainty

The recent emergence of customer experience management has sparked renewed interest in customer delight. Building upon social comparison theory and the uncertainty literature, this research examines a dual-path model (surprise and exclusivity) for creating delightful customer experience through bolstering self-enhancement and a contrasting moderating effect of future reward uncertainty in reward programs. We posit that a surprise (vs. exclusive) reward has a positive effect on customer delight through self-enhancement, but only when perceived uncertainty of receiving a similar reward in the future is high (vs. low). We verify our predictions with data from a laboratory experiment and a field survey, supplemented with multiple waves of customers’ actual behavioral loyalty data and confirm the impacts of customer delight on attitudinal (repeat and WOM intentions) and behavioral (both short- and long-term) loyalty. Our findings offer companies actionable managerial guidelines for building customer loyalty through creating delightful customer experience.

Distinct Medial Orbitofrontal-Striatal Circuits Support Dissociable Component Processes of Risk/Reward Decision-Making.

The medial orbitofrontal cortex (mOFC) regulates a variety of cognitive functions, including refining action selection involving reward uncertainty. This region sends projections to numerous subcortical targets, including the ventral and dorsal striatum, yet how these corticostriatal circuits differentially regulate risk/reward decision-making is unknown. The present study examined the contribution of mOFC circuits linking the nucleus accumbens (NAc) and dorsomedial striatum (DMS) to risk/reward decision-making using pharmacological disconnections. Male rats were well trained on a probabilistic discounting task involving choice between small/certain or large/risky rewards, with the probability of obtaining the larger reward decreasing or increasing over a session. Disconnection of mOFC-striatal pathways was achieved using infusions of GABA agonists inactivating the mOFC in one hemisphere, combined with NAc or DMS inactivation in the contralateral or ipsilateral hemisphere. Perturbing mOFC → NAc circuits induced suboptimal, near-random patterns of choice that manifested as a flattening of the discounting curve. Animals were equally likely to stay or shift following rewarded/nonrewarded choices, suggesting this pathway mediates use of information about reward history to stabilize decision biases. In contrast, mOFC → DMS disconnection impaired adjustments in decision biases, causing opposing changes in risky choice depending on how probabilities varied over time. This was driven by alterations in lose-shift behavior, suggesting mOFC → DMS circuits track volatility in nonrewarded actions to adjust choice in accordance with changes in profitability. Thus, separate mOFC-striatal projection pathways regulate dissociable processes underlying decision-making, with mOFC → NAc circuits aiding in establishing and stabilizing tasks states and mOFC → DMS circuits facilitating transitions across states to promote flexible reward seeking.SIGNIFICANCE STATEMENT The medial orbitofrontal cortex regulates a variety of goal-directed behaviors, yet the functional circuits through which it mediates higher order decision-making functions are unclear. The present study revealed that different mOFC projection pathways facilitate diverse aspects of decision-making involving risks and rewards by engaging separate networks of neurons that interface with distinct ventral and dorsal striatal targets. These findings clarify some of the normal functions of these corticostriatal pathways and may have implications for understanding how dysfunction in these circuits relate to certain psychiatric disorders.

Doing good for (maybe) nothing: How reward uncertainty shapes observer responses to prosocial behavior

When firms or individuals stand to benefit from doing good, observers often question their motivations and discount their good deeds. We propose that this attribution process is sensitive not only to the presence of extrinsic incentives, but also to their prior likelihoods. Across eleven studies, observers treat uncertain rewards (vs. equally valuable certain rewards) as weaker signals of extrinsic motivation. Consequently, observers judge actors who do good when facing uncertain incentives as more purely motivated, benevolent, and likable, and they prefer products from brands that incur profit uncertainty when launching CSR initiatives. Even actors who are handsomely rewarded for doing good are judged favorably if rewards were uncertain at the outset. These effects may stem from more general processes of counterfactual attribution: Actors who do good knowing they might not be rewarded for it may seem more like they would have been willing to act without any incentive at all.

Quantifying the instrumental and noninstrumental underpinnings of Pavlovian responding with the Price equation.

The Price equation is a mathematical expression of selectionist and non-selectionist pressures on biological, cultural, and behavioral change. We use it here to specify instrumental and noninstrumental behaviors as they arise within the context of the Pavlovian autoshaping procedure, for rats trained under reward certainty and reward uncertainty. The point of departure for this endeavor is that some portion of autoshaped behavior referred to as goal-tracking appears instrumental-a function of resource attainment (the individual approaches the location where the unconditioned stimulus is to be delivered). By contrast, some other portion of autoshaped behavior referred to as sign-tracking is noninstrumental-irrelevant to making contact with the to-be-delivered unconditioned stimulus. A Price equation model is proposed that unifies our understanding of Pavlovian autoshaping behavior by isolating operant and respondent influences on goal-tracking (instrumental) and sign-tracking (noninstrumental) behavior.

"Just not knowing" can make life sweeter (and saltier): Reward uncertainty alters the sensory experience and consumption of palatable food and drinks.

Reward uncertainty can prompt exploration and learning, strengthening approach and consummatory behaviors. For humans, these phenomena are exploited in marketing promotions and gambling products, sometimes spurring hedonic consumption. Here, in four experiments, we sought to identify whether reward uncertainty-as a state of "not knowing" that exists between an action and a positively valanced outcome-enhances the in-the-moment consumption and experience of other palatable food and drink rewards. In Experiment 1, we demonstrate that reward uncertainty can increase consumption of commercial alcoholic drinks and energy-dense savory snacks. In Experiment 2, we show that reward uncertainty is unlikely to promote consumption through gross increases in impulsivity (expressed as higher discounting rates) or risk tolerance (expressed as lower probability discounting rates). In Experiment 3, we find that reward uncertainty intensifies the taste of, and hedonic responses to, sucrose solutions in a concentration-dependent manner among individuals with heightened preferences for sweet tastes. Finally, in Experiment 4, we replicate and extend these findings by showing that reward uncertainty intensifies the taste of palatable foods and drinks in ways that are independent of individuals' discounting rates, motor control, reflection impulsivity, and momentary happiness but are strongly moderated by recent depressive symptoms. These data suggest a working hypothesis that (incidental) reward uncertainty, as a state of not knowing, operates as a mood-dependent "taste intensifier" of palatable food and drink rewards, possibly sustaining reward seeking and consumption. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Dissociating the effect of reward uncertainty and timing uncertainty on neural indices of reward prediction errors: A reward positivity (RewP) event-related potential (ERP) study

Accurate reward predictions include forecasting both what a reward will be and when a reward will occur. We tested how variations in the certainty of reward outcome and certainty in timing of feedback presentation modulate neural indices of reward prediction errors using the reward positivity (RewP) component of the scalp-recorded brain event-related potential (ERP). In a within-subjects design, seventy-three healthy individuals completed two versions of a cued doors task; one cued the probability of a reward outcome while the other cued the probability of a delay before feedback. Replicating previous results, RewP amplitude was larger for uncertain feedback compared to certain feedback. Additionally, RewP amplitude was differentially associated with uncertainty of presence/absence of reward, but not uncertainty of feedback timing. Findings suggest a dissociation in that RewP amplitude is modulated by reward prediction certainty but is less affected by certainty surrounding timing of feedback.

How reward uncertainty influences subsequent donations: The role of mental accounting

It is common in the marketplace for merchants to reward consumers whose spending reaches a certain level. This study investigates how reward uncertainty affects consumers’ intentions to donate money to charity after they receive a reward. Across one field experiment and three laboratory experiments, this study demonstrates that consumers are more likely to donate after receiving uncertain rewards than certain ones, because they treat uncertain rewards as financial gains, separate from their payment accounts; certain rewards are expected during consumption, and thus perceived as payment deductions. Therefore, uncertain rewards (vs. certain rewards) induce lower (vs. higher) psychological pain from donating money, which leads to higher (vs. lower) donation intentions. These effects do not apply to people with a calculative mindset (i.e., those who calculated their true expenses before being asked to donate).

Medial orbitofrontal cortex dopamine D1/D2 receptors differentially modulate distinct forms of probabilistic decision-making.

Efficient decision-making involves weighing the costs and benefits associated with different actions and outcomes to maximize long-term utility. The medial orbitofrontal cortex (mOFC) has been implicated in guiding choice in situations involving reward uncertainty, as inactivation in rats alters choice involving probabilistic rewards. The mOFC receives considerable dopaminergic input, yet how dopamine (DA) modulates mOFC function has been virtually unexplored. Here, we assessed how mOFC D1 and D2 receptors modulate two forms of reward seeking mediated by this region, probabilistic reversal learning and probabilistic discounting. Separate groups of well-trained rats received intra-mOFC microinfusions of selective D1 or D2 antagonists or agonists prior to task performance. mOFC D1 and D2 blockade had opposing effects on performance during probabilistic reversal learning and probabilistic discounting. D1 blockade impaired, while D2 blockade increased the number of reversals completed, both mediated by changes in errors and negative feedback sensitivity apparent during the initial discrimination of the task, which suggests changes in probabilistic reinforcement learning rather than flexibility. Similarly, D1 blockade reduced, while D2 blockade increased preference for larger/risky rewards. Excess D1 stimulation had no effect on either task, while excessive D2 stimulation impaired probabilistic reversal performance, and reduced both profitable risky choice and overall task engagement. These findings highlight a previously uncharacterized role for mOFC DA, showing that D1 and D2 receptors play dissociable and opposing roles in different forms of reward-related action selection. Elucidating how DA biases behavior in these situations will expand our understanding of the mechanisms regulating optimal and aberrant decision-making.

Decreased Key Pecking in Response to Reward Uncertainty and Surprising Delay Extension in Pigeons

The Pavlovian autoshaping paradigm has often been used to assess the behavioral effects of reward omission on behavior. We trained pigeons to receive a food reward (unconditioned stimulus or UCS) following illumination of a response key (conditioned stimulus or CS). In Experiment 1, one group of pigeons was trained with two 100% predictive CS-UCS associations (reward certainty) and another group with two 25% predictive CS-UCS associations (reward uncertainty) for 12 sessions. In both groups, the two CS durations were 8 s. Then, in each group, the duration of one CS remained unchanged and that of the other CS was suddenly extended from 8 to 24 s for 6 sessions. In Experiment 2, some experienced individuals (from Experiment 1) and naïve individuals formed two groups trained with a 24-s CS throughout for 18 sessions. Our results show that pigeons (a) pecked less at the uncertain than the certain CS, (b) decreased and then increased CS-pecking after extending CS duration, especially in the certainty condition, (c) were unresponsive to the 24-s CS in the absence of previous experience, and (d) decreased their response rate close to the end of a trial irrespective of the reinforcement condition, CS duration, and amount of training. These results are discussed in relation to several theoretical frameworks.