Reward Signal Research Articles

Multi-objective ω-Regular Reinforcement Learning

The expanding role of reinforcement learning (RL) in safety-critical system design has promoted ω-automata as a way to express learning requirements—often non-Markovian—with greater ease of expression and interpretation than scalar reward signals. However, real-world sequential decision making situations often involve multiple, potentially conflicting, objectives. Two dominant approaches to express relative preferences over multiple objectives are: (1) weighted preference , where the decision maker provides scalar weights for various objectives, and (2) lexicographic preference , where the decision maker provides an order over the objectives such that any amount of satisfaction of a higher-ordered objective is preferable to any amount of a lower-ordered one. In this article, we study and develop RL algorithms to compute optimal strategies in Markov decision processes against multiple ω-regular objectives under weighted and lexicographic preferences. We provide a translation from multiple ω-regular objectives to a scalar reward signal that is both faithful (maximising reward means maximising probability of achieving the objectives under the corresponding preference) and effective (RL quickly converges to optimal strategies). We have implemented the translations in a formal reinforcement learning tool, Mungojerrie , and we present an experimental evaluation of our technique on benchmark learning problems.

Autonomous Obstacle Avoidance in Crowded Ocean Environment Based on COLREGs and POND

In crowded waters with unknown obstacle motion information, traditional methods often fail to ensure safe and autonomous collision avoidance. To address the challenges of information acquisition and decision delay, this study proposes an optimized autonomous navigation strategy that combines deep reinforcement learning with internal and external rewards. By incorporating random network distillation (RND) with proximal policy optimization (PPO), the interest of autonomous ships in exploring unknown environments is enhanced. Additionally, the proposed approach enables the autonomous generation of intrinsic reward signals for actions. For multi-ship collision avoidance scenarios, an environmental reward is designed based on the International Regulations for Preventing Collision at Sea (COLREGs). This reward system categorizes dynamic obstacles into four collision avoidance situations. The experimental results demonstrate that the proposed algorithm outperforms the popular PPO algorithm by achieving more efficient and safe collision avoidance decision-making in crowded ocean environments with unknown motion information. This research provides a theoretical foundation and serves as a methodological reference for the route deployment of autonomous ships.

The gut-brain axis mediates bacterial driven modulation of reward signaling

ObjectiveOur goal is to investigate if microbiota composition modulates reward signaling and assess the role of the vagus in mediating microbiota to brain communication. MethodsMale germ-free Fisher rats were colonized with gastrointestinal contents from chow (low fat (LF) ConvLF) or HF (ConvHF) fed rats. ResultsFollowing colonization, ConvHF rats consumed significantly more food than ConvLF animals. ConvHF rats displayed lower feeding-induced extracellular DOPAC levels (a metabolite of dopamine) in the Nucleus Accumbens (NAc) as well as reduced motivation for HF foods compared to ConvLF rats. Dopamine receptor 2 (DDR2) expression levels in the NAc were also significantly lower in ConvHF animals. Similar deficits were observed in conventionally raised HF fed rats, showing that diet-driven alteration in reward can be initiated via microbiota. Selective gut to brain deafferentation restored DOPAC levels, DRD2 expression, and motivational drive in ConvHF rats. ConclusionsWe concluded from these data that a HF-type microbiota is sufficient to alter appetitive feeding behavior and that bacteria to reward communication is mediated by the vagus nerve.

Cogito Ergo Summ: Abstractive Summarization of Biomedical Papers via Semantic Parsing Graphs and Consistency Rewards

The automatic synthesis of biomedical publications catalyzes a profound research interest elicited by literature congestion. Current sequence-to-sequence models mainly rely on the lexical surface and seldom consider the deep semantic interconnections between the entities mentioned in the source document. Such superficiality translates into fabricated, poorly informative, redundant, and near-extractive summaries that severely restrict their real-world application in biomedicine, where the specialized jargon and the convoluted facts further emphasize task complexity. To fill this gap, we argue that the summarizer should acquire semantic interpretation over input, exploiting structured and unambiguous representations to capture and conserve the most relevant parts of the text content. This paper presents CogitoErgoSumm, the first framework for biomedical abstractive summarization equipping large pre-trained language models with rich semantic graphs. Precisely, we infuse graphs from two complementary semantic parsing techniques with different goals and granularities—Event Extraction and Abstract Meaning Representation, also designing a reward signal to maximize information content preservation through reinforcement learning. Extensive quantitative and qualitative evaluations on the CDSR dataset show that our solution achieves competitive performance according to multiple metrics, despite using 2.5x fewer parameters. Results and ablation studies indicate that our joint text-graph model generates more enlightening, readable, and consistent summaries. Code available at: https://github.com/disi-unibo-nlp/cogito-ergo-summ.

Rhythmic oscillations in the midbrain dopaminergic nuclei in mice.

Dopamine release in the forebrain by midbrain ventral tegmental nucleus (VTA) and substantia nigra pars compacta (SNc) neurons is implicated in reward processing, goal-directed learning, and decision-making. Rhythmic oscillations of neural excitability underlie coordination of network processing, and have been reported in these dopaminergic nuclei at several frequency bands. This paper provides a comparative characterization of several frequencies of oscillations of local field potential and single unit activity, highlighting some behavioral correlates. We recorded from optogenetically identified dopaminergic sites in four mice training in operant olfactory and visual discrimination tasks. Rayleigh and Pairwise Phase Consistency (PPC) analyses revealed some VTA/SNc neurons phase-locked to each frequency range, with fast spiking interneurons (FSIs) prevalent at 1-2.5 Hz (slow) and 4 Hz bands, and dopaminergic neurons predominant in the theta band. More FSIs than dopaminergic neurons were phase-locked in the slow and 4 Hz bands during many task events. The highest incidence of phase-locking in neurons was in the slow and 4 Hz bands, and occurred during the delay between the operant choice and trial outcome (reward or punishment) signals. These data provide a basis for further examination of rhythmic coordination of activity of dopaminergic nuclei with other brain structures, and its impact for adaptive behavior.

Neural dissociation between reward and salience prediction errors through the lens of optimistic bias.

The question of how the brain represents reward prediction errors is central to reinforcement learning and adaptive, goal-directed behavior. Previous studies have revealed prediction error representations in multiple electrophysiological signatures, but it remains elusive whether these electrophysiological correlates underlying prediction errors are sensitive to valence (in a signed form) or to salience (in an unsigned form). One possible reason concerns the loose correspondence between objective probability and subjective prediction resulting from the optimistic bias, that is, the tendency to overestimate the likelihood of encountering positive future events. In the present electroencephalography (EEG) study, we approached this question by directly measuring participants' idiosyncratic, trial-to-trial prediction errors elicited by subjective and objective probabilities across two experiments. We adopted monetary gain and loss feedback in Experiment 1 and positive and negative feedback as communicated by the same zero-value feedback in Experiment 2. We provided electrophysiological evidence in time and time-frequency domains supporting both reward and salience prediction error signals. Moreover, we showed that these electrophysiological signatures were highly flexible and sensitive to an optimistic bias and various forms of salience. Our findings shed new light on multiple presentations of prediction error in the human brain, which differ in format and functional role.

Perinatal opioid exposure leads to decreased social play in adolescent male and female rats: Potential role of oxytocin signaling in brain regions associated with social reward

Over the last two decades, the number of infants exposed to opioids in utero has quadrupled in the United States, with some states reporting rates as high as 55 infants per 1000 births. Clinical studies report that children previously exposed to opioids during gestation show significant deficits in social behavior, including an inability to form friendships or other social relationships. To date, the neural mechanisms whereby developmental opioid exposure disrupts social behavior remain unknown. Using a novel paradigm of perinatal opioid administration, we tested the hypothesis that chronic opioid exposure during critical developmental periods would disrupt juvenile play. As oxytocin is a major regulator of sociability, the impact of perinatal morphine exposure on oxytocin peptide expression was also examined. Juvenile play was assessed in vehicle- or morphine-exposed male and female rats at P25, P35, and P45. Classical features of juvenile play were measured, including time spent engaged in social play, time not in contact, number of pins, and number of nape attacks. We report that morphine-exposed males and females spend less time engaged in play behavior than control males and females, with a corresponding increase in time spent alone. Morphine-exposed males and females also initiated fewer pins and nape attacks. Together, these data suggest that male and female rats exposed to morphine during critical developmental periods are less motivated to participate in social play, potentially due to alterations in oxytocin-mediated reward signaling.

Online POI Recommendation: Learning Dynamic Geo-Human Interactions in Streams

Online POI recommendation strives to recommend future visitable places to users over time. It is crucial to improve the user experience of location-based social networking applications (e.g., Google Maps, Yelp). While numerous studies focus on capturing user visit preferences, geo-human interactions are ignored. However, users make visit decisions based on the status of geospatial contexts. In the meantime, such visits will change the status of geospatial contexts. Therefore, disregarding such geo-human interactions in streams will result in inferior recommendation performance. To fill this gap, in this paper, we propose a novel deep interactive reinforcement learning framework to model geo-human interactions. Specifically, this framework has two main parts: the representation module and the imitation module. The purpose of the representation module is to capture geo-human interactions and convert them into embedding vectors (state). The imitation module is a reinforced agent whose job is to imitate user visit behavior by recommending next-visit POI (action) based on the state. Imitation performance is regarded as a reward signal to optimize the whole interactive framework. When the model converges, the imitation module can precisely perceive users and geospatial contexts to provide accurate POI recommendations. Finally, we conduct extensive experiments to validate the superiority of our framework.

Roles of octopamine neurons in the vertical lobe of the mushroom body for the execution of a conditioned response in cockroaches

Aminergic neurons mediate reward signals in mammals and insects. In crickets, we showed that blockade of synaptic transmission from octopamine neurons (OANs) impairs conditioning of an odor (conditioned stimulus, CS) with water or sucrose (unconditioned stimulus, US) and execution of a conditioned response (CR) to the CS. It has not yet been established, however, whether findings in crickets can be applied to other species of insects. In this study, we investigated the roles of OANs in conditioning of salivation, monitored by activities of salivary neurons, and in execution of the CR in cockroaches (Periplaneta americana). We showed that injection of epinastine (an OA receptor antagonist) into the head hemolymph impaired both conditioning and execution of the CR, in accordance with findings in crickets. Moreover, local injection of epinastine into the vertical lobes of the mushroom body (MB), the center for associative learning and control of the CR, impaired execution of the CR, whereas injection of epinastine into the calyces of the MB or the antennal lobes (primary olfactory centers) did not. We propose that OANs in the MB vertical lobes play critical roles in the execution of the CR in cockroaches. This is analogous to the fact that midbrain dopamine neurons govern execution of learned actions in mammals.

Reinforcement learning in a spiking neural model of striatum plasticity

The basal ganglia (BG), and more specifically the striatum, have long been proposed to play an essential role in action-selection based on a reinforcement learning (RL) paradigm. However, some recent findings, such as striatal spike-timing-dependent plasticity (STDP) or striatal lateral connectivity, require further research and modelling as their respective roles are still not well understood. Theoretical models of spiking neurons with homeostatic mechanisms, lateral connectivity, and reward-modulated STDP have demonstrated a remarkable capability to learn sensorial patterns that statistically correlate with a rewarding signal. In this article, we implement a functional and biologically inspired network model of the striatum, where learning is based on a previously proposed learning rule called spike-timing-dependent eligibility (STDE), which captures important experimental features in the striatum. The proposed computational model can recognize complex input patterns and consistently choose rewarded actions to respond to such sensorial inputs. Moreover, we assess the role different neuronal and network features, such as homeostatic mechanisms and lateral inhibitory connections, play in action-selection with the proposed model. The homeostatic mechanisms make learning more robust (in terms of suitable parameters) and facilitate recovery after rewarding policy swapping, while lateral inhibitory connections are important when multiple input patterns are associated with the same rewarded action. Finally, according to our simulations, the optimal delay between the action and the dopaminergic feedback is obtained around 300 ms, as demonstrated in previous studies of RL and in biological studies.

Collaborative museum heist with reinforcement learning

AbstractNon‐playable characters (NPCs) play a crucial role in enhancing immersion in video games. However, traditional NPC behaviors are often hard‐coded using methods such as Finite State Machines, Decision and Behavior trees. This has a few limitations; namely, it is quite difficult to implement complex cooperative behaviors and secondly this makes it easy for human players to identify and exploit patterns in behavior. To overcome these challenges, Reinforcement learning (RL) can be used to generate dynamic and real‐time NPC responses to human player actions. In this paper, we report on first results of applying RL techniques to a Non‐Zero Sum, adversarial asymmetric game, using a multi‐agent team. The game environment simulates a museum heist, where the objective of the successfully trained team of robbers with different skills (Locksmith, Technician) is to steal valuable items from the museum without being detected by the scripted security guards and cameras. Both agents were trained concurrently with separate policies and received both individual and group reward signals. Through this training process, the agents learned to cooperate effectively and use their skills to maximize both individual and team benefits. These results demonstrate the feasibility of realizing the full game where both robbers and security guards are trained at the same time to achieve their adversarial goals.

Rewarding-unrewarding prediction signals under a bivalent context in the primate lateral hypothalamus

Animals can expect rewards under equivocal situations. The lateral hypothalamus (LH) is thought to process motivational information by producing valence signals of reward and punishment. Despite rich studies using rodents and non-human primates, these signals have been assessed separately in appetitive and aversive contexts; therefore, it remains unclear what information the LH encodes in equivocal situations. To address this issue, macaque monkeys were conditioned under a bivalent context in which reward and punishment were probabilistically delivered, in addition to appetitive and aversive contexts. The monkeys increased approaching behavior similarly in the bivalent and appetitive contexts as the reward probability increased. They increased avoiding behavior under the bivalent and aversive contexts as the punishment probability increased, but the mean frequency was lower under the bivalent context than under the aversive context. The population activity correlated with these mean behaviors. Moreover, the LH produced fine prediction signals of reward expectation, uncertainty, and predictability consistently in the bivalent and appetitive contexts by recruiting context-independent and context-dependent subpopulations of neurons, while it less produced punishment signals in the aversive and bivalent contexts. Further, neural ensembles encoded context information and “rewarding-unrewarding” and “reward-punishment” valence. These signals may motivate individuals robustly in equivocal environments.

An Intelligent Algorithm for USVs Collision Avoidance Based on Deep Reinforcement Learning Approach with Navigation Characteristics

Many achievements toward unmanned surface vehicles have been made using artificial intelligence theory to assist the decisions of the navigator. In particular, there has been rapid development in autonomous collision avoidance techniques that employ the intelligent algorithm of deep reinforcement learning. A novel USV collision avoidance algorithm based on deep reinforcement learning theory for real-time maneuvering is proposed. Many improvements toward the autonomous learning framework are carried out to improve the performance of USV collision avoidance, including prioritized experience replay, noisy network, double learning, and dueling architecture, which can significantly enhance the training effect. Additionally, considering the characteristics of the USV collision avoidance problem, two effective methods to enhance training efficiency are proposed. For better training, considering the international regulations for preventing collisions at sea and USV maneuverability, a complete and reliable USV collision avoidance training system is established, demonstrating an efficient learning process in complex encounter situations. A reward signal system in line with the USV characteristics is designed. Based on the Unity maritime virtual simulation platform, an abundant simulation environment for training and testing is designed. Through detailed analysis, verification, and comparison, the improved algorithm outperforms the pre-improved algorithm in terms of stability, average reward, rules learning, and collision avoidance effect, reducing 26.60% more accumulated course deviation and saving 1.13% more time.

Cytosine methylation in GABA B1 receptor identifies alcohol-related changes for men in blood and brain tissues

Alcohol use alters the reward signaling processes contributing to the development of addiction. We studied the effects of alcohol use disorder (AUD) on brain regions and blood of deceased women and men to examine sex-dependent differences in epigenetic changes associated with AUD. We investigated the effects of alcohol use on the gene promoter methylation of GABBR1 coding for GABAB receptor subunit 1 in blood and brain. We chose six brain regions associated with addiction and the reward pathway (nucleus arcuatus, nucleus accumbens, the mamillary bodies, amygdala, hippocampus and anterior temporal cortex) and performed epigenetic profiling of the proximal promoter of the GABBR1 gene of post-mortem brain and blood samples of 17 individuals with AUD pathology (4 female, 13 male) and 31 healthy controls (10 female, 21 male). Our results show sex-specific effects of AUD on GABBR1 promoter methylation. Especially, CpG -4 showed significant tissue-independent changes and significantly decreased methylation levels for the AUD group in the amygdala and the mammillary bodies of men. We saw prominent and consistent change in CpG-4 across all investigated tissues. For women, no significant loci were observed. We found sex-dependent differences in GABBR1 promoter methylation in relation to AUD. CpG-4 hypomethylation in male individuals with AUD is consistent for most brain regions. Blood shows similar results without reaching significance, potentially serving as a peripheral marker for addiction-associated neuronal adaptations. Further research is needed to discover more contributing factors in the pathological alterations of alcohol addiction to offer sex-specific biomarkers and treatment.

Octopamine neurons mediate reward signals in social learning in an insect

Social learning is found in many animals, but its mechanisms are not understood. We previously showed that a cricket that was trained to observe a conspecific staying at a drinking apparatus exhibited an increased preference for the odor of that drinking apparatus. Here we investigated a hypothesis that this learning is achieved by second-order conditioning (SOC), i.e., by associating conspecifics at a drinking bottle with water reward during group drinking in the rearing stage and then associating an odor with a conspecific in training. Injection of an octopamine receptor antagonist before training or testing impaired the learning or response to the learned odor, as we reported for SOC, thereby supporting the hypothesis. Notably, the SOC hypothesis predicts that octopamine neurons that respond to water in the group-rearing stage also respond to a conspecific in training, without the learner itself drinking water, and such mirror-like activities mediate social learning. This awaits future investigation.

Leveraging transition exploratory bonus for efficient exploration in Hard-Transiting reinforcement learning problems

In reinforcement learning (RL), agents learn policies from spatiotemporal data generated by interaction with the environment. However, spatiotemporal data containing reward signals are often sparse, the agent only gets a reward signal when it reaches the goal state. This kind of problem is technically challenging since rewards are the basis of policy learning. Recently, the prior knowledge of the task structure has been used to solve such problems such as hierarchical learning and goal-conditioned learning. In this paper, we consider a common task structure called Hard-Transiting. In a Hard-Transiting task, the difficulty of moving forward of the agent increases as it approaches the goal state. We formalize the Hard-Transiting RL problem with sparse rewards in which the transition probability decreases as the agent approaches the goal state. Inspired by reward bonus in interactive spatiotemporal data, we propose two novel algorithms with efficient exploration to solve such problems. For tabular setting, we adopt Transition Exploratory Bonus (TEB) to encourage exploration in Hard-Transiting problems and propose Model-Based Interval Estimation-TEB (MBIE-TEB) in which TEB is considered in the value iteration phase of the conventional MBIE algorithm. And under the performance metric of sample complexity, we give a theoretical proof of the upper bound of MBIE-TEB. For non-tabular setting, we propose Deep Q-Network-TEB (DQN-TEB) in which TEB is used as the intrinsic motivation in DQN. We test the proposed algorithms on two numerical tasks and one large-scale task. And the experimental results demonstrate that with the transition exploratory bonus, the proposed algorithms outperform the compared algorithms.

Reward, motivation and brain imaging in human healthy participants - A narrative review.

Over the past 20 years there has been an increasing number of brain imaging studies on the mechanisms underlying reward motivation in humans. This narrative review describes studies on the neural mechanisms associated with reward motivation and their relationships with cognitive function in healthy human participants. The brain's meso-limbic dopamine reward circuitry in humans is known to control reward-motivated behavior in humans. The medial and lateral Pre-Frontal Cortex (PFC) integrate motivation and cognitive control during decision-making and the dorsolateral PFC (dlPFC) integrates and transmits signals of reward to the mesolimbic and meso-cortical dopamine circuits and initiates motivated behavior. The thalamus and insula influence incentive processing in humans and the motor system plays a role in response to action control. There are reciprocal relationships between reward motivation, learning, memory, imagery, working memory, and attention. The most common method of assessing reward motivation is the monetary incentive delay task (DMRT) and there are several meta-analyses of this paradigm. Genetics modulates motivation reward, and dopamine provides the basis for the interaction between motivational and cognitive control. There is some evidence that male adolescents take more risky decisions than female adolescents and that the lateralization of reward-related DA release in the ventral striatum is confined to men. These studies have implications for our understanding of natural reward and psychiatric conditions like addiction, depression and ADHD. Furthermore, the association between reward and memory can help develop treatment techniques for drug addiction that interfere with consolidation of memory. Finally, there is a lack of research on reward motivation, genetics and sex differences and this can improve our understanding of the relationships between reward, motivation and the brain.

Multiclass Reinforced Active Learning for Droplet Pinch-Off Behaviors Identification in Inkjet Printing

Abstract Inkjet printing (IJP) is one of the promising additive manufacturing techniques that yield many innovations in electronic and biomedical products. In IJP, the products are fabricated by depositing droplets on substrates, and the quality of the products is highly affected by the droplet pinch-off behaviors. Therefore, identifying pinch-off behaviors of droplets is critical. However, annotating the pinch-off behaviors is burdensome since a large amount of images of pinch-off behaviors can be collected. Active learning (AL) is a machine learning technique which extracts human knowledge by iteratively acquiring human annotation and updating the classification model for the pinch-off behaviors identification. Consequently, a good classification performance can be achieved with limited labels. However, during the query process, the most informative instances (i.e., images) are varying and most query strategies in AL cannot handle these dynamics since they are handcrafted. Thus, this paper proposes a multiclass reinforced active learning (MCRAL) framework in which a query strategy is trained by reinforcement learning (RL). We designed a unique intrinsic reward signal to improve the classification model performance. Moreover, how to extract the features from images for pinch-off behavior identification is not trivial. Thus, we used a graph convolutional network for droplet image feature extraction. The results show that MCRAL excels AL and can reduce human efforts in pinch-off behavior identification. We further demonstrated that, by linking the process parameters to the predicted droplet pinch-off behaviors, the droplet pinch-off behavior can be adjusted based on MCRAL.

Pseudo-model-free hedging for variable annuities via deep reinforcement learning

This paper proposes a two-phase deep reinforcement learning approach, for hedging variable annuity contracts with both GMMB and GMDB riders, which can address model miscalibration in Black-Scholes financial and constant force of mortality actuarial market environments. In the training phase, an infant reinforcement learning agent interacts with a pre-designed training environment, collects sequential anchor-hedging reward signals, and gradually learns how to hedge the contracts. As expected, after a sufficient number of training steps, the trained reinforcement learning agent hedges, in the training environment, equally well as the correct Delta while outperforms misspecified Deltas. In the online learning phase, the trained reinforcement learning agent interacts with the market environment in real time, collects single terminal reward signals, and self-revises its hedging strategy. The hedging performance of the further trained reinforcement learning agent is demonstrated via an illustrative example on a rolling basis to reveal the self-revision capability on the hedging strategy by online learning.

Actions and Consequences of Insulin in the Striatum.

Insulin crosses the blood-brain barrier to enter the brain from the periphery. In the brain, insulin has well-established actions in the hypothalamus, as well as at the level of mesolimbic dopamine neurons in the midbrain. Notably, insulin also acts in the striatum, which shows abundant expression of insulin receptors (InsRs) throughout. These receptors are found on interneurons and striatal projections neurons, as well as on glial cells and dopamine axons. A striking functional consequence of insulin elevation in the striatum is promoting an increase in stimulated dopamine release. This boosting of dopamine release involves InsRs on cholinergic interneurons, and requires activation of nicotinic acetylcholine receptors on dopamine axons. Opposing this dopamine-enhancing effect, insulin also increases dopamine uptake through the action of insulin at InsRs on dopamine axons. Insulin acts on other striatal cells as well, including striatal projection neurons and astrocytes that also influence dopaminergic transmission and striatal function. Linking these cellular findings to behavior, striatal insulin signaling is required for the development of flavor-nutrient learning, implicating insulin as a reward signal in the brain. In this review, we discuss these and other actions of insulin in the striatum, including how they are influenced by diet and other physiological states.