Prior Reward Research Articles

Persistent threat avoidance following negative reinforcement is not associated with elevated state anxiety.

Obsessive-Compulsive Disorder (OCD) is a debilitating illness consisting of obsessions and compulsions. OCD severity and treatment response are correlated with avoidant behaviors thought be performed to alleviate obsession-related anxiety. However, little is known about either the role of avoidance in the development of OCD or the interplay between anxiety states and avoidance behaviors. We have developed an instrumental negative reinforcement (i.e. active avoidance) paradigm in which mice must lever-press to avoid upcoming foot shocks. We show that mice (both sexes) can learn this task with high acquisition rates (75%) and that this behavior is largely stable when introducing uncertainty and modifying task structure. Furthermore, mice continue to perform avoidance responses on trials where lever pressing is not reinforced and increase response rates as they are maintained on this paradigm. With this paradigm, we did not find a relationship between negative reinforcement history and anxiety-related behaviors in well-established anxiety assays. Finally, we performed exploratory analyses to identify candidate regions involved in well-trained negative reinforcement using expression of the immediate early gene c-Fos. We detected correlated c-Fos expression in 1) cortico-striatal regions which regulate active avoidance in other paradigms and 2) amygdala circuits known to regulate conditioned defensive behaviors.Significance Statement Studies in patients with OCD suggest that compulsions are performed to avoid perceived threats and modulate anxiety tied to obsessions and/or compulsions. The negative reinforcement of avoidance and alleviated anxiety could therefore be a key driver of compulsive behaviors. However, there are still outstanding questions concerning the relationship between these two behaviors and the neural circuits involved in mediating negative reinforcement. We have developed an operant negative reinforcement paradigm in mice with discrete avoid and escape behaviors that can be learned without prior reward training with high throughput (75% acquisition) with responding that persists during nonreinforced trials. However, no differences were observed between negative reinforcement vs. unshocked and inescapably shocked controls, suggesting that continued negative reinforcement did not impact anxiety.

Touchscreen response precision is sensitive to the explore/exploit tradeoff.

The explore/exploit tradeoff is a fundamental property of choice selection during reward-guided decision making. In perceptual decision making, higher certainty decisions are more motorically precise, even when the decision does not require motor accuracy. However, while we can parametrically control uncertainty in perceptual tasks, we do not know what variables - if any - shape motor precision and reflect subjective certainty during reward-guided decision making. Touchscreens are increasingly used across species to measure choice, but provide no tactile feedback on whether an action is precise or not, and therefore provide a valuable opportunity to determine whether actions differ in precision due to explore/exploit state, reward, or individual variables. We find all three of these factors exert independent drives towards increased precision. During exploit states, successive touches to the same choice are closer together than those made in an explore state, consistent with exploit states reflecting higher certainty and/or motor stereotypy in responding. However, exploit decisions might be expected to be rewarded more frequently than explore decisions. We find that exploit choice precision is increased independently of a separate increase in precision due to immediate past reward, suggesting multiple mechanisms regulating choice precision. Finally, we see evidence that male mice in general are less precise in their interactions with the touchscreen than females, even when exploiting a choice. These results suggest that as exploit behavior emerges in reward-guided decision making, individuals become more motorically precise reflecting increased certainty, even when decision choice does not require additional motor accuracy, but this is influenced by individual differences and prior reward. These data uncover the hidden potential for touchscreen tasks in any species to uncover the latent neural states that unite cognition and movement.

Reward history guides focal attention in whisker somatosensory cortex.

Prior reward is a potent cue for attentional capture, but the underlying neurobiology is largely unknown. In a novel whisker touch detection task, we show that mice flexibly shift attention between specific whiskers on a trial-by-trial timescale, guided by the recent history of stimulus-reward association. Two-photon calcium imaging and spike recordings revealed a robust neurobiological correlate of attention in the somatosensory cortex (S1), boosting sensory responses to the attended whisker in L2/3 and L5, but not L4. Attentional boosting in L2/3 pyramidal cells was topographically precise and whisker-specific, and shifted receptive fields toward the attended whisker. L2/3 VIP interneurons were broadly activated by whisker stimuli, motion, and arousal but did not carry a whisker-specific attentional signal, and thus did not mediate spatially focused tactile attention. Together, these findings establish a new model of focal attention in the mouse whisker tactile system, showing that the history of stimuli and rewards in the recent past can dynamically engage local modulation in cortical sensory maps to guide flexible shifts in ongoing behavior.

Sex differences in discrimination behavior and orbitofrontal engagement during context-gated reward prediction.

Animals, including humans, rely on contextual information to interpret ambiguous stimuli. Impaired context processing is a hallmark of several neuropsychiatric disorders, including schizophrenia, autism spectrum disorders, post-traumatic stress disorder, and addiction. While sex differences in the prevalence and manifestations of these disorders are well established, potential sex differences in context processing remain uncertain. Here, we examined sex differences in the contextual control over cue-evoked reward seeking and its neural correlates, in rats. Male and female rats were trained in a bidirectional occasion-setting preparation in which the validity of two auditory reward-predictive cues was informed by the presence, or absence, of a visual contextual feature (LIGHT: X+/DARK: X-/LIGHT: Y-/DARK: Y+). Females were significantly slower to acquire contextual control over cue-evoked reward seeking. However, once established, the contextual control over behavior was more robust in female rats; it showed less within-session variability (less influence of prior reward) and greater resistance to acute stress. This superior contextual control achieved by females was accompanied by an increased activation of the orbitofrontal cortex (OFC) compared to males. Critically, these behavioral and neural sex differences were specific to the contextual modulation process and not observed in simple, context-independent, reward prediction tasks. These results indicate a sex-biased trade-off between the speed of acquisition and the robustness of performance in the contextual modulation of cued reward seeking. The different distribution of sexes along the fast learning ↔ steady performance continuum might reflect different levels of engagement of the OFC, and might have implications for our understanding of sex differences in psychiatric disorders.

Sex differences in discrimination behavior and orbitofrontal engagement during context-gated reward prediction

Animals, including humans, rely on contextual information to interpret ambiguous stimuli. Impaired context processing is a hallmark of several neuropsychiatric disorders, including schizophrenia, autism spectrum disorders, post-traumatic stress disorder, and addiction. While sex differences in the prevalence and manifestations of these disorders are well established, potential sex differences in context processing remain uncertain. Here, we examined sex differences in the contextual control over cue-evoked reward seeking and its neural correlates, in rats. Male and female rats were trained in a bidirectional occasion-setting preparation in which the validity of two auditory reward-predictive cues was informed by the presence, or absence, of a visual contextual feature (LIGHT: X+/DARK: X−/LIGHT: Y−/DARK: Y+). Females were significantly slower to acquire contextual control over cue-evoked reward seeking. However, once established, the contextual control over behavior was more robust in female rats; it showed less within-session variability (less influence of prior reward) and greater resistance to acute stress. This superior contextual control achieved by females was accompanied by an increased activation of the orbitofrontal cortex (OFC) compared to males. Critically, these behavioral and neural sex differences were specific to the contextual modulation process and not observed in simple, context-independent, reward prediction tasks. These results indicate a sex-biased trade-off between the speed of acquisition and the robustness of performance in the contextual modulation of cued reward seeking. The different distribution of sexes along the fast learning ↔ steady performance continuum might reflect different levels of engagement of the OFC, and might have implications for our understanding of sex differences in psychiatric disorders.

Localized and global representation of prior value, sensory evidence, and choice in male mouse cerebral cortex.

Adaptive behavior requires integrating prior knowledge of action outcomes and sensory evidence for making decisions while maintaining prior knowledge for future actions. As outcome- and sensory-based decisions are often tested separately, it is unclear how these processes are integrated in the brain. In a tone frequency discrimination task with two sound durations and asymmetric reward blocks, we found that neurons in the medial prefrontal cortex of male mice represented the additive combination of prior reward expectations and choices. The sensory inputs and choices were selectively decoded from the auditory cortex irrespective of reward priors and the secondary motor cortex, respectively, suggesting localized computations of task variables are required within single trials. In contrast, all the recorded regions represented prior values that needed to be maintained across trials. We propose localized and global computations of task variables in different time scales in the cerebral cortex.

Dynamic Weights and Prior Reward in Policy Fusion for Compound Agent Learning

In Deep Reinforcement Learning (DRL) domain, a compound learning task is often decomposed into several sub-tasks in a divide-and-conquer manner, each trained separately and then fused concurrently to achieve the original task, referred to as policy fusion. However, the state-of-the-art (SOTA) policy fusion methods treat the importance of sub-tasks equally throughout the task process, eliminating the possibility of the agent relying on different sub-tasks at various stages. To address this limitation, we propose a generic policy fusion approach, referred to as Policy Fusion Learning with Dynamic Weights and Prior Reward (PFLDWPR), to automate the time-varying selection of sub-tasks. Specifically, PFLDWPR produces a time-varying one-hot vector for sub-tasks to dynamically select a suitable sub-task and mask the rest throughout the entire task process, enabling the fused strategy to optimally guide the agent in executing the compound task. The sub-tasks with the dynamic one-hot vector are then aggregated to obtain the action policy for the original task. Moreover, we collect sub-tasks’s rewards at the pre-training stage as a prior reward, which, along with the current reward, is used to train the policy fusion network. Thus, this approach reduces fusion bias by leveraging prior experience. Experimental results under three popular learning tasks demonstrate that the proposed method significantly improves three SOTA policy fusion methods in terms of task duration, episode reward, and score difference.

34 Neurocomputational Mechanisms of Social Reward Processing in Combat-Exposed Veterans

Objective:Combat exposure is associated with higher rates of depressive symptoms, including anhedonia (i.e., a reduced ability to seek and experience rewards) and feelings of social disconnectedness. While these symptoms are commonly documented in combat-exposed Veterans following deployment, the cognitive mechanisms underlying this pathology is less well understood. Computational modeling can provides detailed mechanistic insights into complex cognition, which may be particularly useful to understand how social reward processing is altered following combat exposure. Here, we use a Bayesian learning model framework to address this question.Participants and Methods:Thirty-three Operation Enduring Freedom (OEF)/ Operation Iraqi Freedom (OIF)/Operation New Dawn (OND) Veterans (25 Male, 8 Female) between the ages of 18-65 years old (M = 41.61, SD = 10.49) participated in this study. In both classic/monetary and social reward conditions, participants completed a 2-arm bandit task, in which they must choose on each trial between two options (i.e., slot machine vs social partner) with unknown reward rates. While they received monetary outcomes in the classic condition, participants received compliments from different fictitious partners in the social condition. We first compared a learning-independent Win-stay/Lose-shift (WSLS) heuristic and either a Rescorla-Wagner Q-learning or a Bayesian learning model (Dynamic Belief Model/DBM) paired with a Softmax reward maximization policy. DBM+Softmax provided the best fit of the data for most participants (31/33). Individual DBM parameters of prior reward expectation, reward learning (i.e., perceived stability of reward rates), and Softmax reward maximization were estimated and compared across conditions.Results:Participants did not differ in their reward learning parameters across monetary and social conditions (t(30)= -0.70, p = 0.490), suggesting similar perception of reward stability in both modalities. However, higher Bayesian prior mean (i.e., initial belief of reward rate; t(30)= -2.31, p = 0.028, d=0.42) and greater reward maximization (i.e., Softmax parameter; t(30)= -2.26, p = 0.031, d=0.41) were observed in response to social vs monetary rewards. In the social reward condition, higher self-reported social connectedness was associated with greater model fit of our DBM model (i.e., smaller Bayesian Information Criterion/BIC; r = -0.38, p = 0.041). In this condition, those expecting higher reward rates when initiating reward exploration (those with higher DBM prior mean) endorsed lower self-esteem (Spearman's ρ = -0.43, p = 0.078) and lower positive affect (ρ = -0.32, p = 0.078).Conclusions:A Bayesian learning modeling framework can characterize mechanistic differences in the processing of social vs non-social reward among combat-exposed Veterans. Individuals with higher social connectedness were more model-based in their performance, consistent with the notion that they are more likely to estimate and anticipate how much social peers have to offer. Combat-exposed individuals with lower self-esteem and positive affect appear to have higher initial expectations of reward from unknown partners, which could reflect greater need for mood and/or self-esteem repair in those individuals. Overall, Bayesian modeling of social reward behavior provides a useful quantitative framework to predict clinically relevant construct of functional outcomes in military populations.

Beta traveling waves in monkey frontal and parietal areas encode recent reward history

Brain function depends on neural communication, but the mechanisms of this communication are not well understood. Recent studies suggest that one form of neural communication is through traveling waves (TWs)—patterns of neural oscillations that propagate within and between brain areas. We show that TWs are robust in microarray recordings in frontal and parietal cortex and encode recent reward history. Two adult male monkeys made saccades to obtain probabilistic rewards and were sensitive to the (statistically irrelevant) reward on the previous trial. TWs in frontal and parietal areas were stronger in trials that followed a prior reward versus a lack of reward and, in the frontal lobe, correlated with the monkeys’ behavioral sensitivity to the prior reward. The findings suggest that neural communication mediated by TWs within the frontal and parietal lobes contribute to maintaining information about recent reward history and mediating the impact of this history on the monkeys’ expectations.

Reward-motivated memories influence new learning across development.

Previously rewarding experiences can influence choices in new situations. Past work has demonstrated that existing reward associations can either help or hinder future behaviors and that there is substantial individual variability in the transfer of value across contexts. Developmental changes in reward sensitivity may also modulate the impact of prior reward associations on later goal-directed behavior. The current study aimed to characterize how reward associations formed in the past affected learning in the present from childhood to adulthood. Participants completed a reinforcement learning paradigm using high- and low-reward stimuli from a task completed 24 h earlier, as well as novel stimuli, as choice options. We found that prior high-reward associations impeded learning across all ages. We then assessed how individual differences in the prioritization of high- versus low-reward associations in memory impacted new learning. Greater high-reward memory prioritization was associated with worse learning performance for previously high-reward relative to low-reward stimuli across age. Adolescents also showed impeded early learning regardless of individual differences in high-reward memory prioritization. Detrimental effects of previous reward on choice behavior did not persist beyond learning. These findings indicate that prior reward associations proactively interfere with future learning from childhood to adulthood and that individual differences in reward-related memory prioritization influence new learning across age.

Psychoanalytical Considerations of Emotion Regulation Disorders in Multiple Complex-Traumatized Children-A Study Protocol of the Prospective Study MuKi.

Studies in adults with mental disorders suggest that the experience of early and chronic trauma is associated with changes in reward expectancy and processing. In addition, severe childhood trauma has been shown to contribute to the development of mental disorders in general. Data on effects of early childhood trauma on reward expectancy and processing in middle childhood currently appear insufficient. The present study aims to fill this research gap by examining the effects of developmental trauma disorder (DTD) on reward expectancy and processing in children aged 8–12 years, testing the hypothesis that children with multiple complex traumas exhibit altered reward processing as a result of prior disappointing reward experiences. One main feature of developmental trauma disorder is early experiences of multiple separation from important and close relationships alongside other experiences of emotional or physical harm. In the sequel children often show affect regulation disorders. To investigate this, we have developed an adapted version of the Monetary Incentive Delay (MID) Task, which examines children’s expectation of reward or frustration. In this first study, behavioral data will be collected from N = 40 children (n = 20 experimental group and n = 20 healthy controls) using this adapted version of the MID Task. Children in the experimental group will be recruited from youth welfare centers in Frankfurt a.M., Germany. Healthy control subjects will be recruited from after-school-care facilities. A brief trauma screening will be conducted for both groups, experimental and control. If children show signs of trauma, the presence of a developmental trauma disorder will be further delineated by a German translation of the Developmental Trauma Disorder Structured Interview for Children (DTDSI-C) which was translated the first time in German by our research group. We hypothesize that children in the experimental group will be less accurate in performing the Monetary Incentive Delay Task because of their impaired emotion regulation skills due to emotional avoidance following developmental trauma. If the results of our initial behavioral study are promising, the MID task will be used in a future study to elucidate the relationship between trauma developmental disorder, reward expectancy and processing, and neurobiological processes in middle childhood.

The influence of reward history on goal-directed visual search.

The attentional priority of a stimulus is influenced by both its relationship to task goals and reward history. While experiments probing goal-directed visual search typically dictate which stimuli should receive attentional priority by virtue of highly constrained task demands, individuals in real-world scenarios choose what to search for. In such complex and dynamic environments, it is unclear how reward history might influence the strategic control of attention. In the present study, participants completed a modified version of the Adaptive Choice Visual Search (ACVS) task integrated with the value-driven attentional capture design. In a training phase, participants learned to associate one of two possible target colors with monetary reward. In a subsequent test phase, they completed the ACVS task in which a target in both a previously rewarded and unrewarded color was present on each trial and participants could choose how to search. Our results reveal that participants were biased to voluntarily search through the previously reward-associated color regardless of whether the distribution of stimuli made it optimal to do so, which came at a cost in performance when searching through the previously rewarded color was a suboptimal strategy. In the absence of prior reward training, in contrast, search strategy was inconsistent with respect to color. Our findings provide evidence that reward history biases voluntary search processes.

Disruptions of Sustained Spatial Attention Can Be Resistant to the Distractor's Prior Reward Associations.

Attention can be involuntarily biased toward reward-associated distractors (value-driven attentional capture, VDAC). Yet past work has primarily demonstrated this distraction phenomenon during a particular set of circumstances: transient attentional orienting to potentially relevant stimuli occurring in our visual environment. Consequently, it is not well-understood if reward-based attentional capture can occur under other circumstances, such as during sustained visuospatial attention. Using EEG, we investigated whether associating transient distractors with reward value would increase their distractibility and lead to greater decrements in concurrent sustained spatial attention directed elsewhere. Human participants learned to associate three differently colored, laterally presented squares with rewards of varying magnitude (zero, small, and large). These colored squares were then periodically reintroduced as distractors at the same lateral locations during a demanding sustained-attention rapid-serial-visual-presentation (RSVP) task at the midline. Behavioral and neural evidence indicated that participants had successfully learned and maintained the reward associations to the distractors. During the RSVP task, consistent with prior work, we found that the distractors generated dips in the instantaneous amplitude of the steady-state visual evoked potentials (SSVEPs) elicited by the midline RSVP stimuli, indicating that the distractors were indeed transiently disrupting sustained spatial attention. Contrary to our hypotheses, however, the magnitude of this dip did not differ by the magnitude of the distractor’s reward associations. These results indicate that while sustained spatial attention can be impaired by the introduction of distractors at another location, the main distraction process is resistant to the distractors’ reward associations, thus providing evidence of an important boundary condition to value-driven attentional capture.

Chronic Augmentation of Endocannabinoid Levels Persistently Increases Dopaminergic Encoding of Reward Cost and Motivation.

Motivational deficits characterized by an unwillingness to overcome effortful costs are a common feature of neuropsychiatric and neurologic disorders that are insufficiently understood and treated. Dopamine (DA) signaling in the nucleus accumbens (NAc) facilitates goal-seeking, but how NAc DA release encodes motivationally salient stimuli to influence effortful investment is not clear. Using fast-scan cyclic voltammetry in male and female mice, we find that NAc DA release diametrically responds to cues signaling increasing cost of reward, while DA release to the reward itself is unaffected by its cost. Because endocannabinoid (eCB) signaling facilitates goal seeking and NAc DA release, we further investigated whether repeated augmentation of the eCB 2-arachidonoylglycerol with a low dose of a monoacylglycerol lipase (MAGL) inhibitor facilitates motivation and DA signaling without the development of tolerance. We find that chronic MAGL treatment stably facilitates goal seeking and DA encoding of prior reward cost, providing critical insight into the neurobiological mechanisms of a viable treatment for motivational deficits.SIGNIFICANCE STATEMENT Decades of work has established a fundamental role for dopamine neurotransmission in motivated behavior and cue-reward learning, but how dopaminergic encoding of cues associates with motivated action has remained unclear. Specifically, how dopamine neurons signal future and prior reward cost, and whether this can be modified to influence motivational set points is not known. The current study provides important insight into how dopamine neurons encode motivationally relevant stimuli to influence goal-directed action and supports cannabinoid-based therapies for treatment of motivational disorders.

Dopamine release and its control over early Pavlovian learning differs between the NAc core and medial NAc shell

Dopamine neurons respond to cues to reflect the value of associated outcomes. These cue-evoked dopamine responses can encode the relative rate of reward in rats with extensive Pavlovian training. Specifically, a cue that always follows the previous reward by a short delay (high reward rate) evokes a larger dopamine response in the nucleus accumbens (NAc) core relative to a distinct cue that always follows the prior reward by a long delay (low reward rate). However, it was unclear if these reward rate dopamine signals are evident during early Pavlovian training sessions and across NAc subregions. To address this, we performed fast-scan cyclic voltammetry recordings of dopamine levels to track the pattern of cue- and reward-evoked dopamine signals in the NAc core and medial NAc shell. We identified regional differences in the progression of cue-evoked dopamine signals across training. However, the dopamine response to cues did not reflect the reward rate in either the NAc core or the medial NAc shell during early training sessions. Pharmacological experiments found that dopamine-sensitive conditioned responding emerged in the NAc core before the medial NAc shell. Together, these findings illustrate regional differences in NAc dopamine release and its control over behavior during early Pavlovian learning.

Prior Reward Conditioning Dampens Hippocampal and Striatal Responses during an Associative Memory Task.

Offering reward during encoding typically leads to better memory [Adcock, R. A., Thangavel, A., Whitfield-Gabrieli, S.,Knutson, B., & Gabrieli, J. D. E. Reward-motivated learning: Mesolimbic activation precedes memory formation. Neuron, 50, 507-517, 2006]. Whether such memory benefit persists when tested in a different task context remains, however, largely understudied [Wimmer, G. E., & Buechel, C. Reactivation of reward-related patterns from single past episodes supports memory-based decision making. Journal of Neuroscience, 36, 2868-2880, 2016]. Here, we ask whether reward at encoding leads to a generalized advantage across learning episodes, a question of high importance for any everyday life applications, from education to patient rehabilitation. Although we confirmed that offering monetary reward increased responses in the ventral striatum and pleasantness judgments for pictures used as stimuli, this immediate beneficial effect of reward did not carry over to a subsequent and different picture-location association memory task during which no reward was delivered. If anything, a trend for impaired memory accuracy was observed for the initially high-rewarded pictures as compared to low-rewarded ones. In line with this trend in behavioral performance, fMRI activity in reward (i.e., ventral striatum) and in memory (i.e., hippocampus) circuits was reduced during the encoding of new associations using previously highly rewarded pictures (compared to low-reward pictures). These neural effects extended to new pictures from same, previously highly rewarded semantic category. Twenty-four hours later, delayed recall of associations involving originally highly rewarded items was accompanied by decreased functional connectivity between the hippocampus and two brain regions implicated in value-based learning, the ventral striatum and the ventromedial PFC. We conclude that acquired reward value elicits a downward value-adjustment signal in the human reward circuit when reactivated in a novel nonrewarded context, with a parallel disengagement of memory-reward (hippocampal-striatal) networks, likely to undermine new associative learning. Although reward is known to promote learning, here we show how it may subsequently hinder hippocampal and striatal responses during new associative memory formation.

Memory guidance of value-based decision making at an abstract level of representation

Value-based decisions about alternatives we have never experienced can be guided by associations between current choice options and memories of prior reward. A critical question is how similar memories need to be to the current situation to effectively guide decisions. We address this question in the context of associative learning of faces using a sensory preconditioning paradigm. We find that memories of reward spread along established associations between faces to guide decision making. While memory guidance is specific for associated facial identities, it does not only occur for the specific images that were originally encountered. Instead, memory guidance generalizes across different images of the associated identities. This suggests that memory guidance does not rely on a pictorial format of representation but on a higher, view-invariant level of abstraction. Thus, memory guidance operates on a level of representation that neither over- nor underspecifies associative relationships in the context of obtaining reward.

Deep Reinforcement Learning by Balancing Offline Monte Carlo and Online Temporal Difference Use Based on Environment Experiences

Owing to the complexity involved in training an agent in a real-time environment, e.g., using the Internet of Things (IoT), reinforcement learning (RL) using a deep neural network, i.e., deep reinforcement learning (DRL) has been widely adopted on an online basis without prior knowledge and complicated reward functions. DRL can handle a symmetrical balance between bias and variance—this indicates that the RL agents are competently trained in real-world applications. The approach of the proposed model considers the combinations of basic RL algorithms with online and offline use based on the empirical balances of bias–variance. Therefore, we exploited the balance between the offline Monte Carlo (MC) technique and online temporal difference (TD) with on-policy (state-action–reward-state-action, Sarsa) and an off-policy (Q-learning) in terms of a DRL. The proposed balance of MC (offline) and TD (online) use, which is simple and applicable without a well-designed reward, is suitable for real-time online learning. We demonstrated that, for a simple control task, the balance between online and offline use without an on- and off-policy shows satisfactory results. However, in complex tasks, the results clearly indicate the effectiveness of the combined method in improving the convergence speed and performance in a deep Q-network.

Neighborhood Deprivation Shapes Motivational-Neurocircuit Recruitment in Children

Implementing motivated behaviors on the basis of prior reward is central to adaptive human functioning, but aberrant reward-motivated behavior is a core feature of neuropsychiatric illness. Children from disadvantaged neighborhoods have decreased access to rewards, which may shape motivational neurocircuits and risk for psychopathology. Here, we leveraged the unprecedented neuroimaging data from the Adolescent Brain Cognitive Development (ABCD) study to test the hypothesis that neighborhood socioeconomic disadvantage shapes the functional recruitment of motivational neurocircuits in children. Specifically, via the ABCD study’s monetary-incentive-delay task (N = 6,396 children; age: 9–10 years), we found that children from zip codes with a high Area Deprivation Index demonstrate blunted recruitment of striatum (dorsal and ventral nuclei) and pallidum during reward anticipation. In fact, blunted dorsal striatal recruitment during reward anticipation mediated the association between Area Deprivation Index and increased attention problems. These data reveal a candidate mechanism driving elevated risk for psychopathology in children from socioeconomically disadvantaged neighborhoods.

A novel test of flexible planning in relation to executive function and language in young children.

In adult humans, decisions involving the choice and use of tools for future events typically require episodic foresight. Previous studies suggest some non-human species are capable of future planning; however, these experiments often cannot fully exclude alternative learning explanations. Here, we used a novel tool-use paradigm aiming to address these critiques to test flexible planning in 3- to 5-year-old children, in relation to executive function and language abilities. In the flexible planning task, children were not verbally cued during testing, single trials avoided consistent exposure to stimulus–reward relationships, and training trials provided experience of a predictable return of reward. Furthermore, unlike most standard developmental studies, we incorporated short delays before and after tool choice. The critical test choice included two tools with equal prior reward experience—each only functional in one apparatus. We tested executive function and language abilities using several standardized tasks. Our results echoed standard developmental research: 4- and 5-year-olds outperformed 3-year-olds on the flexible planning task, and 5-year-old children outperformed younger children in most executive function and language tasks. Flexible planning performance did not correlate with executive function and language performance. This paradigm could be used to investigate flexible planning in a tool-use context in non-human species.