Nonrewarded Trials Research Articles

Event-related potential activity in the basal ganglia differentiates rewards from nonrewards: response to commentary.

We recently demonstrated that the feedback negativity may be better understood as a reward-related positivity that is absent on nonreward trials, and source localization revealed that this reward response may reflect activity in the striatum. In a commentary on our report, Cohen et al. argue against this latter finding, claiming it is unlikely that the striatum contributes to the scalp-recorded event-related potential. We disagree with the line of reasoning presented by Cohen et al., and we respond here to each of their points. Based on all the available evidence, we argue that the striatum is a plausible generator of a reward-related response observed at the scalp, and this possibility warrants further investigation.

Pattern Classification of Working Memory Networks Reveals Differential Effects of Methylphenidate, Atomoxetine, and Placebo in Healthy Volunteers

Stimulant and non-stimulant drugs can reduce symptoms of attention deficit/hyperactivity disorder (ADHD). The stimulant drug methylphenidate (MPH) and the non-stimulant drug atomoxetine (ATX) are both widely used for ADHD treatment, but their differential effects on human brain function remain unclear. We combined event-related fMRI with multivariate pattern recognition to characterize the effects of MPH and ATX in healthy volunteers performing a rewarded working memory (WM) task. The effects of MPH and ATX on WM were strongly dependent on their behavioral context. During non-rewarded trials, only MPH could be discriminated from placebo (PLC), with MPH producing a similar activation pattern to reward. During rewarded trials both drugs produced the opposite effect to reward, that is, attenuating WM networks and enhancing task-related deactivations (TRDs) in regions consistent with the default mode network (DMN). The drugs could be directly discriminated during the delay component of rewarded trials: MPH produced greater activity in WM networks and ATX produced greater activity in the DMN. Our data provide evidence that: (1) MPH and ATX have prominent effects during rewarded WM in task-activated and -deactivated networks; (2) during the delay component of rewarded trials, MPH and ATX have opposing effects on activated and deactivated networks: MPH enhances TRDs more than ATX, whereas ATX attenuates WM networks more than MPH; and (3) MPH mimics reward during encoding. Thus, interactions between drug effects and motivational state are crucial in defining the effects of MPH and ATX.

Color modulates olfactory learning in honeybees by an occasion-setting mechanism

A sophisticated form of nonelemental learning is provided by occasion setting. In this paradigm, animals learn to disambiguate an uncertain conditioned stimulus using alternative stimuli that do not enter into direct association with the unconditioned stimulus. For instance, animals may learn to discriminate odor rewarded from odor nonrewarded trials if these two situations are indicated by different colors that do not themselves become associated with the reward. Despite a growing interest in nonelemental learning in insects, no study has so far attempted to study occasion setting in restrained honeybees, although this would allow direct access to the neural basis of nonelemental learning. Here we asked whether colors can modulate olfactory conditioning of the proboscis extension reflex (PER) via an occasion-setting mechanism. We show that intact, harnessed bees are not capable of learning a direct association between color and sucrose. Despite this incapacity, bees solved an occasion-setting discrimination in which colors set the occasion for appropriate responding to an odor that was rewarded or nonrewarded depending on the color. We therefore provide the first controlled demonstration of bimodal (color-odor) occasion setting in harnessed honeybees, which opens the door for studying the neural basis of such bimodal, nonelemental discriminations in insects.

Lateralization for visual processes: eye preference in Campbell’s monkeys (Cercopithecus c. campbelli)

Brain lateralization has been the matter of extensive research over the last centuries, but it remains an unsolved issue. While hand preferences have been extensively studied, very few studies have investigated laterality of eye use in non-human primates. We examined eye preference in 14 Campbell's monkeys (Cercopithecus c. campbelli). We assessed eye preference to look at a seed placed inside a tube using monocular vision. Eye use was recorded for 100 independent and non-rewarded trials per individual. All of the 14 monkeys showed very strong preferences in the choice of the eye used to look inside the tube (mean preference: 97.6%). Eight subjects preferred the right eye and six subjects preferred the left eye. The results are discussed in light of previous data on eye preference in primates, and compared to data on hand preference from these subjects. Our findings would support the hypothesis for an early emergence of lateralization for perceptual processes compared to manual motor functions.

Striatal dopamine transmission in healthy humans during a passive monetary reward task

Research on dopamine (DA) transmission has emphasized the importance of increased phasic DA cell firing in the presence of unpredictable rewards. Using [ 11C]raclopride PET, we previously reported that DA transmission was both suppressed and enhanced in different regions of the striatum during an unpredictable reward task [Zald, D.H., Boileau, I., El Dearedy, W., Gunn, R., McGlone, F., Dichter, G.S. et al. (2004). Dopamine transmission in the human striatum during monetary reward tasks. J. Neurosci. 24, 4105–4112]. However, it was unclear if reductions in DA release during this task reflected a response to the high proportion of nonrewarding trials, and whether the behavioral demands of the task influenced the observed response. To test these issues, we presented 10 healthy subjects with an automated (passive) roulette wheel game in which the amount of reward and its timing were unpredictable and the rewarding trials greatly outnumbered the nonrewarding ones. As in the previous study, DA transmission in the putamen was significantly suppressed relative to a predictable control condition. A similar suppression occurred when subjects were presented with temporally unpredictable novel pictures and sounds. At present, models of DA functioning during reward do not account for this suppression, but given that it has been observed in two different studies using different reward paradigms, this phenomenon warrants attention. Neither the unpredictable reward nor the novelty conditions produced consistent increases in striatal DA transmission. These data suggest that active behavioral engagement may be necessary to observe robust statewise increases in DA release in the striatum.

Impulsivity and risk-taking behavior in focal frontal lobe lesions

Frontal lobe dysfunction may underlie excessively impulsive and risky behavior observed in a range of neurological disorders. We devised a gambling task to examine these behavior tendencies in a sample of patients who had sustained focal damage to the frontal lobes or nonfrontal cortical regions as well as in a matched sample of healthy control subjects. The main objectives of the study were: (1) to behaviorally dissociate impulsivity and risk-taking; (2) to examine potential associations between specific frontal lesion sites and impulsivity or risk-taking; (3) to investigate the influence of reinforcement and trial timing on both behaviors. Our results indicated that patients and controls were equally likely to perform impulsively. Risk-taking performance strategies, however, were related to left ventrolateral and orbital lesion sites. Moreover, risk-taking was also associated with blunted response alteration following a nonrewarded trial. Patients and control subjects showed identical responses to reward-timing manipulations consistent with formal decision-making theory. These findings suggest that ventrolateral and orbital lesions are related to the reward-based aspects of decision-making (risk-taking) rather than to simple response disinhibition (impulsivity). Reduced reaction to the negative consequences of one's actions may underlie this behavior pattern.

Effects of parafascicular excitotoxic lesions on two-way active avoidance and odor-discrimination

To investigate whether the parafascicular (PF) nucleus of the thalamus is involved in different learning and memory tasks, two experiments were carried out in adult male Wistar rats that were submitted to pre-training bilateral N-methyl- d-aspartate PF infusions (0.15 M, pH 7.4; 1.2 μl/side, 0.2 μl/min). In Experiment 1, we evaluated the effects of PF lesions in two identical 30-trial training sessions, separated by a 24-h interval, of a two-way active avoidance conditioning. PF-lesioned rats exhibited impaired performance in both sessions, measured by number of avoidance responses. In Experiment 2, the effects of PF lesions were assessed in a training session (5 trials) and a 24-h retention test (2 retention trials and 2 relearning trials) of an odor-discrimination task. PF lesions did not significantly disrupt the acquisition or the first retention trial, which was not rewarded. However, lesioned animals’ performance was clearly affected in subsequent trials, following the introduction of the single non-rewarded trial. Current data are discussed considering evidence that lesions of the PF nucleus affect learning and memory functions mediated by anatomically related areas of the frontal cortex and striatum.

Memories and anticipations control responding by rats (Rattus norvegicus) in a Pavlovian procedure

In Experiment 1 each rat received two different fixed series of three trials each. The unconditioned stimulus occurred on Trial 1 of one series and on Trial 3 of the other series, all other trials being nonreinforced. Previous Pavlovian investigations have shown that rats can remember the immediately prior reward outcome and anticipate the immediately subsequent reward outcome. Experiment 1 demonstrated that rats could remember and anticipate even more remote reward outcomes. In Experiment 2 two groups received a series of two nonrewarded trials followed by a rewarded trial. It was demonstrated that a change in the conditioned stimulus (CS) from Trial 2 to Trial 3, which occurred in one group, produced weaker responding than in the other group that did not experience such CS change. On the basis of these findings it was suggested that the rats organized the trials of a series into a unit or chunk. This was concluded for two reasons. First, remembering and anticipating remote reward outcomes strongly suggests that responding is being controlled by events extending beyond the current trial. Secondly, the experimental manipulations employed in the Pavlovian situation here are similar to those used in prior human learning and animal instrumental learning investigations concerned with chunking. Thus, it would appear that chunking is a ubiquitous phenomenon appearing in human serial learning (e.g., Bower and Winzenz 1969; Crowder 1976), in animal instrumental learning (e.g., Capaldi 1992; Hulse and Dorsky 1977; Terrace 1987), and now in Pavlovian learning.

The brain metabolic enhancer methylene blue improves discrimination learning in rats

Methylene blue (MB) is a metabolic enhancer that has been demonstrated to improve memory retention when given post-training in low doses in a variety of tasks in rats, including inhibitory avoidance, spatial memory (in both normal and metabolically-impaired subjects), object recognition, and habituation to a familiar environment. MB has been also shown to improve memory retention of extinction of fear conditioning in the rat. No experiments have been conducted to determine the effects of MB on more complex learning such as in discrimination tasks that require repeated days of training. This study examined the effects of daily MB on spatial discrimination memory in a baited holeboard maze. Following three days of discrimination training, subjects treated daily with post-training MB (1 mg/kg) reliably discriminated between rewarded (baited) and non-rewarded (unbaited) trials as indicated by a greater number of correct responses on rewarded trials than non-rewarded trials during the last three days of discrimination training. No such discrimination effects were observed in the saline-treated control group during the same training period. To determine whether the memory-enhancing effects of MB are associated with an increase in metabolic energy capacity in the brain, cytochrome c oxidation was measured in brains from rats treated with 1 mg/kg MB or saline for three days. The number of daily injections was chosen based on the behavioral data which revealed group differences three days after the beginning of MB treatment. Brain cytochrome oxidase activity in the MB-treated group was approximately 70% higher than in saline-treated rats. The findings suggest that repeated post-training MB may improve memory consolidation between days of learning by an induction in the enzyme cytochrome oxidase, leading to increased metabolic capacity in brain regions requiring more energy during discrimination learning.

Reward modulation of prefrontal and visual association cortex during an incentive working memory task

Cognitive performance differs with motivation, but little direct evidence exists regarding the neural mechanisms of the influence of reward motivation on working memory (WM). We tested the effects of motivation on the top-down control in visual WM. Encoding relevant stimuli for maintenance, while suppressing inappropriate inputs is considered a core process in cognition. Prior functional magnetic resonance imaging (fMRI) results demonstrated that stimulus-specific visual association cortex serves as a marker of activation differences for task-relevant and task-irrelevant inputs, such that enhanced activity occurs when attention is directed to relevant stimuli and suppressed activity occurs when attention is directed away from irrelevant stimuli [Gazzaley, A., Cooney, J., McEvoy, K., Knight, R.T., and D'Esposito, M. J. Cogn. Neurosci. 17, 507–517]. We used fMRI to test whether differential WM performance, indexed by lowered response times on a delayed-recognition task, was associated with amplification of enhancement and suppression effects during stimulus encoding within visual association cortex. Our results indicate that enhancement and suppression are amplified for trials with the highest reward level relative to non-rewarded trials for a scene-selective cortical region. In a face-selective region, similar modulation of enhancement for the highest reward level relative to non-rewarded trials was found. Prefrontal cortex also showed enhanced activity during high reward trials. Overall these results reveal that reward motivation can play a pivotal role in driving performance through top-down signaling in frontal regions involved in WM, as well as visual association regions selective to processing the perceptual inputs of the items to be remembered.

Sensitivity of the nucleus accumbens to violations in expectation of reward

This study examined whether ventral frontostriatal regions differentially code expected and unexpected reward outcomes. We parametrically manipulated the probability of reward and examined the neural response to reward and nonreward for each probability condition in the ventral striatum and the orbitofrontal cortex (OFC). By late trials of the experiment, subjects showed slower behavioral responses for the condition with the lowest probability of reward, relative to the condition with the highest probability of reward. At the neural level, both the nucleus accumbens (NAcc) and OFC showed greater activation to rewarded relative to nonrewarded trials, but the accumbens appeared to be most sensitive to violations in expected reward outcomes. These data suggest distinct roles for frontostriatal circuitry in reward prediction and in responding to violations in expectations.

Cues produced by reward and nonreward and temporal cues influence responding in the intertrial interval and to the conditioned stimulus

Rats in a Pavlovian situation were trained under three different reward schedules, at either a 30 s or a 90 s intertrial interval (ITI): Consistent reward (C), 50% irregular reward (I), and single alternation of reward and nonrewarded trials (SA). Activity was recorded to the conditioned stimulus (CS) and in all 10 s bins in each ITI except the first 10 s bin. Responding increased progressively over the bins of the 90 s ITI indicating a role for temporal cues in the ITI. Single alternation patterning (SAP), more vigorous responding on rewarded than on nonrewarded trials, occurred in the ITI as well as to the CS and differed little at the 30 s and 90 s ITI. The occurrence of SAP to the CS and, as shown here for the first time, in the ITI along with the findings for the C and I schedules, indicate three matters of importance. First, along with other cues such as temporal cues, behavior is influenced by cues associated with reward and nonreward both in the ITI and when the CS is presented. Second, as the percentage of reward associated with these cues increases so too does vigor of responding. Finally, if the cues produced by reward events are memories the retrieval cues for such memories are present in the ITI as well as when the CS is presented. Considerations are advanced indicating that reward produced cues may influence responding in the Pavlovian situation not only under regular schedules such as the SA schedule, but under all schedules. Indeed, such cues may more validly signal reward and nonreward than the CS itself.

Effects of forced-choice runway variations on rats’ T-maze serial pattern learning

Rats learned an ordered RNR/RNN serial pattern task in a T-maze where they were shifted to a different runway on Trial 3 only in the RNR series (shift-win/stay-lose group) or only in the RNN series (stay-win/shift-lose group). The shift-win/stay-lose group developed faster speeds on Trial 3 of the RNR than on Trial 3 of the RNN series more easily than the stay-win/shift-lose group. This difference occurred whether all rats were forced onto the same runway on the first two trials (Experiment 1) or onto a different runway on Trial 2 from that on Trial 1 in each series (Experiment 2). Posttraining probe tests revealed that the shift-win/stay-lose group in each experiment relied on the runway shift event in Trial 3 or on the series position to anticipate the second reward within a series. Such reward expectancies were greater when the runway shift occurred in the same series position as during training. These probe tests revealed that the stay-win/shift-lose group relied only on the series position in Experiment 2. Our findings do not support predictions based on an associative predictive validity model. Rather, they reflect rats' predisposition to spontaneously alternate choices in the T-maze, a tendency corresponding to their inherent win-shift foraging strategy. Rats in each group also reduced their speeds less on the nonrewarded Trial 2 when it preceded a rewarded rather than a nonrewarded Trial 3. This effect suggests that rats were able to determine which series contained a second rewarded trial. We discuss the theoretical implications of this Trial 2 speed effect in terms of rats' uncertainty about where this second rewarded trial might occur in the RNR series.

A model to explain the emergence of reward expectancy neurons using reinforcement learning and neural network

In an experiment of multi-trial task to obtain a reward, reward expectancy neurons, which responded only in the non-reward trials that are necessary to advance toward the reward, have been observed in the anterior cingulate cortex of monkeys. In this paper, to explain the emergence of the reward expectancy neuron in terms of reinforcement learning theory, a model that consists of a recurrent neural-network trained based on reinforcement learning is proposed. The analysis of the hidden layer neurons of the model during the learning suggests that the reward expectancy neurons emerge to realize smooth temporal increase of the state value by complementing the neuron that responds only in the reward trial.

Honeybee (Apis mellifera) vision can discriminate between and recognise images of human faces

Recognising individuals using facial cues is an important ability. There is evidence that the mammalian brain may have specialised neural circuitry for face recognition tasks, although some recent work questions these findings. Thus, to understand if recognising human faces does require species-specific neural processing, it is important to know if non-human animals might be able to solve this difficult spatial task. Honeybees (Apis mellifera) were tested to evaluate whether an animal with no evolutionary history for discriminating between humanoid faces may be able to learn this task. Using differential conditioning, individual bees were trained to visit target face stimuli and to avoid similar distractor stimuli from a standard face recognition test used in human psychology. Performance was evaluated in non-rewarded trials and bees discriminated the target face from a similar distractor with greater than 80% accuracy. When novel distractors were used, bees also demonstrated a high level of choices for the target face, indicating an ability for face recognition. When the stimuli were rotated by 180 degrees there was a large drop in performance, indicating a possible disruption to configural type visual processing.

Effects of runway shift and stay rules on rats' serial pattern learning in the T-maze.

Rats received three-trial series on a T-maze consisting of extended visually distinct left-black and right-striped side runways. During the first phase of training, when allowed to select baited runways within these series, they predominantly alternated their choices. During the second phase, rats received forced-choice serial pattern training of series consisting of two rewarded (R) trials and one nonrewarded (N) trial in two fixed orders, RRN and RNR. In Experiment 1, the rats in the runway shift rule group always received the second R trial when forced down a runway opposite that on the preceding trial in the series and the N trial when forced down the same runway. The rats in the runway stay rule group always received the second R trial when forced down the same runway and the N trial when forced down the opposite runway. In Experiment 2, each rat was conditionally trained with both runway outcome rules as determined by the central alley lighting and the type of food in the side alleys. The rats took longer to reduce their running speed on the N trial within each sequence under the runway stay rule than under the runway shift rule. They also took longer to acquire serial pattern responding for the RNR than for the RRN series only under the runway stay rule condition. When subsequently reexposed to series of free-choice trials on the final phase, rats maintained spontaneous alternating choice patterns under the runway shift rule conditions but either seldom alternated their choices (Experiment 1) or greatly reduced choice alternations (Experiment 2) under the runway stay rule condition. We discussed these effects in terms of rats' natural foraging strategies and as a factor that interacts with other within- and between-series variables that affect serial pattern behavior.

Recognition memory for complex visual discriminations is influenced by stimulus interference in rodents with perirhinal cortex damage.

Rats with perirhinal cortex (PRC), hippocampal, or sham lesions were trained on a successive discrimination go/no-go task to examine recognition memory for an array of visual objects with varying interference among the objects in the array. Rats were trained to recognize a target array consisting of four particular objects that could be presented in any one of four possible configurations to cover baited food wells. If the four target objects were presented, the rat should displace each object to receive food. However, if a novel object replaced any one or more of the target objects, then the rat should withhold its response. The number of novel objects presented on nonrewarded trials varied from one to four. The fewer the number of novel objects in the array, the more interference the array shared with the target array, therefore increasing task difficulty. An increased number of novel objects should result in less interference with the target array and decreased task difficulty. Although accuracy was slightly lower in rats with hippocampal lesions compared with controls, the learning of the groups was not statistically different. In contrast, rats with PRC lesions were significantly impaired in learning compared with both control and hippocampal-lesioned rats. The results suggest that recognition memory for complex visual discriminations is affected by stimulus interference in rodents with PRC damage.

Serial learning vs. reward schedules: What is learned?

Rats were employed in a runway to investigate whether different conditions for presenting reward events affect what is learned. One view is that, depending on how reward events are presented, animals may learn either simple associations or encode rules (rule view). Another view is that under all conditions for presenting reward events, animals employ the memory of prior reward events as signals for current reward events (memory view). To test these views, in each of three investigations reported here, rats were shifted to a single alternation series of rewarded and nonrewarded trials following various types of initial training. Concern was invested in whether in shift, immediate single alternation pattern responding would occur consisting of faster running on rewarded than on nonrewarded trials. All of the initial training conditions employed here were considered to be too minimal to support immediate single alternation pattern responding according to the rule view. This was true of some but not all conditions of initial training according to the memory view. Contrary to the rule view and consistent with the memory view, immediate single alternation pattern responding occurred following some types of minimal initial training but not others. Finally, we discussed the possibility that only the mechanisms within the memory view can similarly deal with all varieties of reward schedule phenomena which have previously been attributed to different, separate theories.

Stimulus control of anticipatory responding in instrumental learning as revealed in serial learning tasks

Items in a fixed series A-B-C can be anticipated employing either item information (e.g., ItemA cues signal Item B) or position information (e.g., Position 2 cues signal Item B). Identical fixed series of rewarded and nonrewarded trials were employed in investigations reported by Capaldi and Miller (1988), Capaldi, Alptekin, Miller, and Birmingham (1997), and Burns, Dunkman, and Detloff (1999). These series were learned employing item cues according to Capaldi and colleagues and position cues according to Burns et al. (1999). Burns et al.’s (1999) conclusions were based on rats’ behavior when shifted to novel series, which could reveal position learning but not item learning. Employing the acquisition conditions of Burns and co-workers but shifts to series that could reveal item learning, evidence for item learning was obtained in two experiments. The most reasonable interpretation of the above set of findings is that rats employ a compound of both types of cues to signal items. To our knowledge, no set of prior findings so strongly suggests a similar conclusion.

Molar vs Molecular Approaches to Reward Schedule and Serial Learning Phenomena

In Experiment 1 all rats received identical series of rewarded and nonrewarded trials in a black runway and in a white runway. A grouping cue, a change in runway brightness, was introduced on a rewarded trial that followed either a single nonrewarded trial (Group N1) or four successive nonrewarded trials (Group N4). Over a series of four nonrewarded trials terminating in reward, Group N1 ran slower than Group N4 on Trial 2 but faster than Group N4 on Trial 4. In Experiment 2, slower running occurred when the grouping cue occurred on Trial 4 of a consistent reward schedule rather than on Trial 4 of a partial reward schedule. These findings were shown to be inconsistent with three theories that attempt to explain reward schedule data in terms of some overall characteristic of the schedule such as percentage of reward (molar theories). The data are consistent with the sequential view that recommends decomposing reward schedules into more elementary memory units (a molecular theory). In particular the data demonstrate that the effects of an overall reward schedule on behavior are determined by the more specific reward schedules associated with each memory component of the schedule. Importantly, the findings suggest that reward schedule investigations and serial learning investigations are, theoretically speaking, identical. Accordingly, the findings strongly discourage the common practice of reward schedule theories ignoring serial learning data and of serial learning theories ignoring reward schedule data.