Reward Signal Research Articles

Reinforcement learning using fully connected, attention, and transformer models in knapsack problem solving

SummaryKnapsack is a combinatorial optimization problem that involves a variety of resource allocation challenges. It is defined as non‐deterministic polynomial time (NP) hard and has a wide range of applications. Knapsack problem (KP) has been studied in applied mathematics and computer science for decades. Many algorithms that can be classified as exact or approximate solutions have been proposed. Under the category of exact solutions, algorithms such as branch‐and‐bound and dynamic programming and the approaches obtained by combining these algorithms can be classified. Due to the fact that exact solutions require a long processing time, many approximate methods have been introduced for knapsack solution. In this research, deep Q‐learning using models containing fully connected layers, attention, and transformer as function estimators were used to provide the solution for KP. We observed that deep Q‐networks, which continued their training by observing the reward signals provided by the knapsack environment we developed, optimized the total reward gained over time. The results showed that our approaches give near‐optimum solutions and work about 40 times faster than an exact algorithm using dynamic programming.

The Unity of Opposites: The Flux Between Impulsiveness and Compulsiveness Mediated by the Endocannabinoid System

Background: Impulsiveness and compulsiveness, though traditionally assumed to represent the opposite poles of a spectrum, have increasingly been understood to have common, rather than disparate, features. Objective: We investigated the influence of the ECS on the intricate interplay between impulsiveness and compulsiveness. Method: We conducted a search in PubMed, PsyhInfo, and Scopus by using the keywords compulsiveness, Obsessive-Compulsive Disorder (OCD), impulsiveness, Impulse Control Disorders (ICDs), addiction, ECS, serotonin, and dopamine. Results: The ECS is famous for its role in motivational processes and reward signaling, contributing to the hedonic effect elicited by pleasurable activities. Impulsiveness involves an inability to delay pleasure; compulsiveness, in contrast, is associated with a postponement of present pleasure due to an increased sense of menace. Impulsiveness and compulsiveness work together to result in addiction. The manipulation of the ECS through pharmacological or genetic methods is effective on the manifestations of OCD, ICDs, and impulsiveness. The ECS is distributed throughout the basal ganglia, prefrontal cortex, amygdala, and hippocampus and interacts with serotonergic, dopaminergic, noradrenergic, GABAergic, and glutamatergic neural circuits which are all involved in impulsive and compulsive symptomatology. The ECS or cannabinoids exhibit divergent effects depending on environmental conditions, dose, and duration of exposure. The endocannabinoids are retrograde messengers inhibiting the presynaptic neurons (causing a braking effect) and released “on demand”, generating a fine-tuning adjustment of the neural activity. Conclusion: How impulsiveness and compulsiveness, two apparently opposite traits, interoperate might be accounted for by the specific functions of the ECS.

Constrained Dual-Level Bandit for Personalized Impression Regulation in Online Ranking Systems

Impression regulation plays an important role in various online ranking systems, e.g. , e-commerce ranking systems always need to achieve local commercial demands on some pre-labeled target items like fresh item cultivation and fraudulent item counteracting while maximizing its global revenue. However, local impression regulation may cause “butterfly effects” on the global scale, e.g. , in e-commerce, the price preference fluctuation in initial conditions (overpriced or underpriced items) may create a significantly different outcome, thus affecting shopping experience and bringing economic losses to platforms. To prevent “butterfly effects”, some researchers define their regulation objectives with global constraints, by using contextual bandit at the page-level that requires all items on one page sharing the same regulation action, which fails to conduct impression regulation on individual items. To address this problem, in this article, we propose a personalized impression regulation method that can directly makes regulation decisions for each user-item pair. Specifically, we model the regulation problem as a C onstrained D ual-level B andit (CDB) problem, where the local regulation action and reward signals are at the item-level while the global effect constraint on the platform impression can be calculated at the page-level only. To handle the asynchronous signals, we first expand the page-level constraint to the item-level and then derive the policy updating as a second-order cone optimization problem. Our CDB approaches the optimal policy by iteratively solving the optimization problem. Experiments are performed on both offline and online datasets, and the results, theoretically and empirically, demonstrate CDB outperforms state-of-the-art algorithms.

Fear of Missing Out Predicts Distraction by Social Reward Signals Displayed on a Smartphone in Difficult Driving Situations.

Smartphones are particularly likely to elicit driver distraction with obvious negative repercussions on road safety. Recent selective attention models lead to expect that smartphones might be very effective in capturing attention due to their social reward history. Hence, individual differences in terms of Fear of Missing Out (FoMO) – i.e., of the apprehension of missing out on socially rewarding experiences – should play an important role in driver distraction. This factor has already been associated with self-reported estimations of greater attention paid to smartphones while driving, but the potential link between FoMO and smartphone-induced distraction has never been tested empirically. Therefore, we conducted a preliminary study to investigate whether FoMO would modulate attentional capture by reward distractors displayed on a smartphone. First, participants performed a classical visual search task in which neutral stimuli (colored circles) were associated with high or low social reward outcomes. Then, they had to detect a pedestrian or a roe deer in driving scenes with various levels of fog density. The social reward stimuli were displayed as distractors on the screen of a smartphone embedded in the pictures. The results showed a significant three-way interaction between FoMO, social reward distraction, and task difficulty. More precisely, under attention-demanding conditions (i.e., high-fog density), individual FoMO scores predicted attentional capture by social reward distractors, with longer reaction times (RTs) for high rather than low social reward distractors. These results highlight the importance to consider reward history and FoMO when investigating smartphone-based distraction. Limitations are discussed, notably regarding our sample characteristics (i.e., mainly young females) that might hamper the generalization of our findings to the overall population. Future research directions are provided.

Spiking neural network-based multi-task autonomous learning for mobile robots

Spiking Neural Networks (SNNs) are the new generation of artificial neural networks that closely mimic the time encoding and information processing aspects of the human brain. In this work, a multi-task autonomous learning paradigm is proposed for the mobile robot application, which employs a SNN to construct the controlling system of the mobile robot. The Reward-modulated Spiking-time-dependent Plasticity learning rule is developed for the SNN-based controller, which aims to achieve the capability of autonomous learning under multiple tasks. Reward signals are generated based on the instantaneous frequencies of pre- and post-synaptic spikes, which adapts to the sensory stimuli and environmental feedback. Meanwhile, inspired by lateral inhibition connections, a task switch mechanism is designed to enable the controller to switch the operations between multiple tasks. Two tasks of obstacle avoidance and target tracking are used for performance evaluation and results demonstrate that the mobile robot with the proposed paradigm is able to autonomously learn, switch and complete the tasks.

Deep reinforcement learning methods for structure-guided processing path optimization

A major goal of materials design is to find material structures with desired properties and in a second step to find a processing path to reach one of these structures. In this paper, we propose and investigate a deep reinforcement learning approach for the optimization of processing paths. The goal is to find optimal processing paths in the material structure space that lead to target-structures, which have been identified beforehand to result in desired material properties. There exists a target set containing one or multiple different structures, bearing the desired properties. Our proposed methods can find an optimal path from a start structure to a single target structure, or optimize the processing paths to one of the equivalent target-structures in the set. In the latter case, the algorithm learns during processing to simultaneously identify the best reachable target structure and the optimal path to it. The proposed methods belong to the family of model-free deep reinforcement learning algorithms. They are guided by structure representations as features of the process state and by a reward signal, which is formulated based on a distance function in the structure space. Model-free reinforcement learning algorithms learn through trial and error while interacting with the process. Thereby, they are not restricted to information from a priori sampled processing data and are able to adapt to the specific process. The optimization itself is model-free and does not require any prior knowledge about the process itself. We instantiate and evaluate the proposed methods by optimizing paths of a generic metal forming process. We show the ability of both methods to find processing paths leading close to target structures and the ability of the extended method to identify target-structures that can be reached effectively and efficiently and to focus on these targets for sample efficient processing path optimization.

Self-Operating Stock Exchange – A Deep Reinforcement Learning Approach

Stock trading approaches play an important role in equity. However, it is tough to create a financially beneficial approach in a complicated and evolving stock market. In this manuscript, we suggest an epsilon greedy policy in our DQN prototype that allows you to get effective policy for the agent this could optimize the predicted values of the total reward across any sequential steps ranging from the present state i. E. To maximize the state-action-value function through engaging with the environment q (s, a) to recommend when to buy, sell or hold. In this prototype, the state depends on routine principles of buy, sell or hold of existing data and the state alter as the buying and selling session alters. The prototype is able to grow rapidly the responses to market on reward signals but by agents which will allow us to understand about the holding and buying of the stocks.

Dopaminergic signaling supports auditory social learning

Explicit rewards are commonly used to reinforce a behavior, a form of learning that engages the dopaminergic neuromodulatory system. In contrast, skill acquisition can display dramatic improvements from a social learning experience, even though the observer receives no explicit reward. Here, we test whether a dopaminergic signal contributes to social learning in naïve gerbils that are exposed to, and learn from, a skilled demonstrator performing an auditory discrimination task. Following five exposure sessions, naïve observer gerbils were allowed to practice the auditory task and their performance was assessed across days. We first tested the effect of an explicit food reward in the observer’s compartment that was yoked to the demonstrator’s performance during exposure sessions. Naïve observer gerbils with the yoked reward learned the discrimination task significantly faster, as compared to unrewarded observers. The effect of this explicit reward was abolished by administration of a D1/D5 dopamine receptor antagonist during the exposure sessions. Similarly, the D1/D5 antagonist reduced the rate of learning in unrewarded observers. To test whether a dopaminergic signal was sufficient to enhance social learning, we administered a D1/D5 receptor agonist during the exposure sessions in which no reward was present and found that the rate of learning occurred significantly faster. Finally, a quantitative analysis of vocalizations during the exposure sessions suggests one behavioral strategy that contributes to social learning. Together, these results are consistent with a dopamine-dependent reward signal during social learning.

Anatomical dissociation of intracerebral signals for reward and punishment prediction errors in humans

Whether maximizing rewards and minimizing punishments rely on distinct brain systems remains debated, given inconsistent results coming from human neuroimaging and animal electrophysiology studies. Bridging the gap across techniques, we recorded intracerebral activity from twenty participants while they performed an instrumental learning task. We found that both reward and punishment prediction errors (PE), estimated from computational modeling of choice behavior, correlate positively with broadband gamma activity (BGA) in several brain regions. In all cases, BGA scaled positively with the outcome (reward or punishment versus nothing) and negatively with the expectation (predictability of reward or punishment). However, reward PE were better signaled in some regions (such as the ventromedial prefrontal and lateral orbitofrontal cortex), and punishment PE in other regions (such as the anterior insula and dorsolateral prefrontal cortex). These regions might therefore belong to brain systems that differentially contribute to the repetition of rewarded choices and the avoidance of punished choices.

Presynaptic inhibition of dopamine neurons controls optimistic bias.

Regulation of reward signaling in the brain is critical for appropriate judgement of the environment and self. In Drosophila, the protocerebral anterior medial (PAM) cluster dopamine neurons mediate reward signals. Here, we show that localized inhibitory input to the presynaptic terminals of the PAM neurons titrates olfactory reward memory and controls memory specificity. The inhibitory regulation was mediated by metabotropic gamma-aminobutyric acid (GABA) receptors clustered in presynaptic microdomain of the PAM boutons. Cell type-specific silencing the GABA receptors enhanced memory by augmenting internal reward signals. Strikingly, the disruption of GABA signaling reduced memory specificity to the rewarded odor by changing local odor representations in the presynaptic terminals of the PAM neurons. The inhibitory microcircuit of the dopamine neurons is thus crucial for both reward values and memory specificity. Maladaptive presynaptic regulation causes optimistic cognitive bias.

Gym-ANM: Reinforcement learning environments for active network management tasks in electricity distribution systems

Active network management (ANM) of electricity distribution networks include many complex stochastic sequential optimization problems. These problems need to be solved for integrating renewable energies and distributed storage into future electrical grids. In this work, we introduce Gym-ANM, a framework for designing reinforcement learning (RL) environments that model ANM tasks in electricity distribution networks. These environments provide new playgrounds for RL research in the management of electricity networks that do not require an extensive knowledge of the underlying dynamics of such systems. Along with this work, we are releasing an implementation of an introductory toy-environment, ANM6-Easy, designed to emphasize common challenges in ANM. We also show that state-of-the-art RL algorithms can already achieve good performance on ANM6-Easy when compared against a model predictive control (MPC) approach. Finally, we provide guidelines to create new Gym-ANM environments differing in terms of (a) the distribution network topology and parameters, (b) the observation space, (c) the modeling of the stochastic processes present in the system, and (d) a set of hyperparameters influencing the reward signal. Gym-ANM can be downloaded at https://github.com/robinhenry/gym-anm.

Ibudilast, a neuroimmune modulator, reduces heavy drinking and alcohol cue-elicited neural activation: a randomized trial

Ibudilast, a neuroimmune modulator which selectively inhibits phosphodiesterases (PDE)-3, -4, -10, and -11, and macrophage migration inhibitory factor (MIF), shows promise as a novel pharmacotherapy for alcohol use disorder (AUD). However, the mechanisms of action underlying ibudilast’s effects on the human brain remain largely unknown. Thus, the current study examined the efficacy of ibudilast to improve negative mood, reduce heavy drinking, and attenuate neural reward signals in individuals with AUD. Fifty-two nontreatment-seeking individuals with AUD were randomized to receive ibudilast (n = 24) or placebo (n = 28). Participants completed a 2-week daily diary study during which they filled out daily reports of their past day drinking, mood, and craving. Participants completed an functional magnetic resonance imaging (fMRI) alcohol cue-reactivity paradigm half-way through the study. Ibudilast did not have a significant effect on negative mood (β = −0.34, p = 0.62). However, ibudilast, relative to placebo, reduced the odds of heavy drinking across time by 45% (OR = 0.55, (95% CI: 0.30, 0.98)). Ibudilast also attenuated alcohol cue-elicited activation in the ventral striatum (VS) compared to placebo (F(1,44) = 7.36, p = 0.01). Alcohol cue-elicited activation in the VS predicted subsequent drinking in the ibudilast group (F(1,44) = 6.39, p = 0.02), such that individuals who had attenuated ventral striatal activation and took ibudilast had the fewest number of drinks per drinking day in the week following the scan. These findings extend preclinical and human laboratory studies of the utility of ibudilast to treat AUD and suggest a biobehavioral mechanism through which ibudilast acts, namely, by reducing the rewarding response to alcohol cues in the brain leading to a reduction in heavy drinking.

Brain Networks Underlying Strategy Execution and Feedback Processing in an Efficient Functional Magnetic Resonance Imaging Neurofeedback Training Performed in a Parallel or a Serial Paradigm.

Neurofeedback (NF) is a complex learning scenario, as the task consists of trying out mental strategies while processing a feedback signal that signifies activation in the brain area to be self-regulated and acts as a potential reward signal. In an attempt to dissect these subcomponents, we obtained whole-brain networks associated with efficient self-regulation in two paradigms: parallel, where the task was performed concurrently, combining feedback with strategy execution; and serial, where the task was performed consecutively, separating feedback processing from strategy execution. Twenty participants attempted to control their anterior midcingulate cortex (aMCC) using functional magnetic resonance imaging (fMRI) NF in 18 sessions over 2 weeks, using cognitive and emotional mental strategies. We analyzed whole-brain fMRI activations in the NF training runs with the largest aMCC activation for the serial and parallel paradigms. The equal length of the strategy execution and the feedback processing periods in the serial paradigm allows a description of the two task subcomponents with equal power. The resulting activation maps were spatially correlated with functionally annotated intrinsic connectivity brain maps (BMs). Brain activation in the parallel condition correlates with the basal ganglia (BG) network, the cingulo-opercular network (CON), and the frontoparietal control network (FPCN); brain activation in the serial strategy execution condition with the default mode network (DMN), the FPCN, and the visual processing network; while brain activation in the serial feedback processing condition predominantly with the CON, the DMN, and the FPCN. Additional comparisons indicate that BG activation is characteristic to the parallel paradigm, while supramarginal gyrus (SMG) and superior temporal gyrus (STG) activations are characteristic to the serial paradigm. The multifaceted view of the subcomponents allows describing the cognitive processes associated with strategy execution and feedback processing independently in the serial feedback task and as combined processes in the multitasking scenario of the conventional parallel feedback task.

Reward tampering problems and solutions in reinforcement learning: a causal influence diagram perspective

Can humans get arbitrarily capable reinforcement learning (RL) agents to do their bidding? Or will sufficiently capable RL agents always find ways to bypass their intended objectives by shortcutting their reward signal? This question impacts how far RL can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we study when an RL agent has an instrumental goal to tamper with its reward process, and describe design principles that prevent instrumental goals for two different types of reward tampering (reward function tampering and RF-input tampering). Combined, the design principles can prevent reward tampering from being an instrumental goal. The analysis benefits from causal influence diagrams to provide intuitive yet precise formalizations.

Foresee then Evaluate: Decomposing Value Estimation with Latent Future Prediction

Value function is the central notion of Reinforcement Learning (RL). Value estimation, especially with function approximation, can be challenging since it involves the stochasticity of environmental dynamics and reward signals that can be sparse and delayed in some cases. A typical model-free RL algorithm usually estimates the values of a policy by Temporal Difference (TD) or Monte Carlo (MC) algorithms directly from rewards, without explicitly taking dynamics into consideration. In this paper, we propose Value Decomposition with Future Prediction (VDFP), providing an explicit two-step understanding of the value estimation process: 1) first foresee the latent future, 2) and then evaluate it. We analytically decompose the value function into a latent future dynamics part and a policy-independent trajectory return part, inducing a way to model latent dynamics and returns separately in value estimation. Further, we derive a practical deep RL algorithm, consisting of a convolutional model to learn compact trajectory representation from past experiences, a conditional variational auto-encoder to predict the latent future dynamics and a convex return model that evaluates trajectory representation. In experiments, we empirically demonstrate the effectiveness of our approach for both off-policy and on-policy RL in several OpenAI Gym continuous control tasks as well as a few challenging variants with delayed reward.

Meta-Learning Effective Exploration Strategies for Contextual Bandits

In contextual bandits, an algorithm must choose actions given ob- served contexts, learning from a reward signal that is observed only for the action chosen. This leads to an exploration/exploitation trade-off: the algorithm must balance taking actions it already believes are good with taking new actions to potentially discover better choices. We develop a meta-learning algorithm, Mêlée, that learns an exploration policy based on simulated, synthetic con- textual bandit tasks. Mêlée uses imitation learning against these simulations to train an exploration policy that can be applied to true contextual bandit tasks at test time. We evaluate Mêlée on both a natural contextual bandit problem derived from a learning to rank dataset as well as hundreds of simulated contextual ban- dit problems derived from classification tasks. Mêlée outperforms seven strong baselines on most of these datasets by leveraging a rich feature representation for learning an exploration strategy.

WCSAC: Worst-Case Soft Actor Critic for Safety-Constrained Reinforcement Learning

Safe exploration is regarded as a key priority area for reinforcement learning research. With separate reward and safety signals, it is natural to cast it as constrained reinforcement learning, where expected long-term costs of policies are constrained. However, it can be hazardous to set constraints on the expected safety signal without considering the tail of the distribution. For instance, in safety-critical domains, worst-case analysis is required to avoid disastrous results. We present a novel reinforcement learning algorithm called Worst-Case Soft Actor Critic, which extends the Soft Actor Critic algorithm with a safety critic to achieve risk control. More specifically, a certain level of conditional Value-at-Risk from the distribution is regarded as a safety measure to judge the constraint satisfaction, which guides the change of adaptive safety weights to achieve a trade-off between reward and safety. As a result, we can optimize policies under the premise that their worst-case performance satisfies the constraints. The empirical analysis shows that our algorithm attains better risk control compared to expectation-based methods.

Improving Sample Efficiency in Model-Free Reinforcement Learning from Images

Training an agent to solve control tasks directly from high-dimensional images with model-free reinforcement learning (RL) has proven difficult. A promising approach is to learn a latent representation together with the control policy. However, fitting a high-capacity encoder using a scarce reward signal is sample inefficient and leads to poor performance. Prior work has shown that auxiliary losses, such as image reconstruction, can aid efficient representation learning. However, incorporating reconstruction loss into an off-policy learning algorithm often leads to training instability. We explore the underlying reasons and identify variational autoencoders, used by previous investigations, as the cause of the divergence. Following these findings, we propose effective techniques to improve training stability. This results in a simple approach capable of matching state-of-the-art model-free and model-based algorithms on MuJoCo control tasks. Furthermore, our approach demonstrates robustness to observational noise, surpassing existing approaches in this setting. Code, results, and videos are anonymously available at https://sites.google.com/view/sac-ae/home.

Parameterized reinforcement learning for optical system optimization

Engineering a physical system to feature designated characteristics states an inverse design problem, which is often determined by several discrete and continuous parameters. If such a system must feature a particular behavior, the mentioned combination of both, discrete and continuous, parameters results in a challenging optimization problem that requires an extensive search for an optimal system design. However, if the corresponding inverse design problem can be reformulated as a parameterized Markov decision process, reinforcement learning (RL) provides a heuristic framework to solve it. In this work, we use multi-layer thin films as an example of the aforementioned optimization problems and consider three design parameters: Each of the thin film layer’s dielectric material (discrete) and thickness (continuous), as well as the total number of layers (discrete). While recent methods merely determine the optimal thicknesses and—less commonly—the layers’ materials, our approach optimizes the total number of stacked layers as well. In summary, we further develop a Q-learning variant to solve inverse design optimization and thereby outperform human experts and current approaches like needle-point optimization or naive RL. For this purpose, we propose an exponentially transformed reward signal that eases policy search and enables constrained optimization. Moreover, the learned Q-values contain information about the optical properties of multi-layer thin films, which allows us a physical interpretation or what-if analysis and thus enables explainability.

Sequential Generative Exploration Model for Partially Observable Reinforcement Learning

Many challenging partially observable reinforcement learning problems have sparse rewards and most existing model-free algorithms struggle with such reward sparsity. In this paper, we propose a novel reward shaping approach to infer the intrinsic rewards for the agent from a sequential generative model. Specifically, the sequential generative model processes a sequence of partial observations and actions from the agent's historical transitions to compile a belief state for performing forward dynamics prediction. Then we utilize the error of the dynamics prediction task to infer the intrinsic rewards for the agent. Our proposed method is able to derive intrinsic rewards that could better reflect the agent's surprise or curiosity over its ground-truth state by taking a sequential inference procedure. Furthermore, we formulate the inference procedure for dynamics prediction as a multi-step forward prediction task, where the time abstraction that has been incorporated could effectively help to increase the expressiveness of the intrinsic reward signals. To evaluate our method, we conduct extensive experiments on challenging 3D navigation tasks in ViZDoom and DeepMind Lab. Empirical evaluation results show that our proposed exploration method could lead to significantly faster convergence than various state-of-the-art exploration approaches in the testified navigation domains.