Reinforcement Learning Task Research Articles

DeRL: Coupling Decomposition in Action Space for Reinforcement Learning Task

DeRL: Coupling Decomposition in Action Space for Reinforcement Learning Task

Reward Learning as a Potential Mechanism for Improvement in Schizophrenia Spectrum Disorders Following Cognitive Remediation: Protocol for a Clinical, Nonrandomized, Pre-Post Pilot Study.

Cognitive impairment is common with schizophrenia spectrum disorders. Cognitive remediation (CR) is effective in improving global cognition, but not all individuals benefit from this type of intervention. A better understanding of the potential mechanism of action of CR is needed. One proposed mechanism is reward learning (RL), the cognitive processes responsible for adapting behavior following positive or negative feedback. It is proposed that the structure of CR enhances RL and motivation to engage in increasingly challenging tasks, and this is a potential mechanism by which CR improves cognitive functioning in schizophrenia. Our primary objective is to examine reward processing in individuals with schizophrenia before and after completing CR and to compare this with a group of matched clinical controls. We will assess whether RL mediates the relationship between CR and improved cognitive function and reduced negative symptoms. Potential differences in social RL and nonsocial RL in individuals with schizophrenia will also be investigated and compared with a healthy matched control group. We propose a clinical, nonrandomized, pre-post pilot study comparing the impact of CR on RL and neurocognitive outcomes. The study will use a combination of objective and subjective measures to assess neurocognitive, psychiatric symptoms, and neurophysiological domains. A total of 40 individuals with schizophrenia spectrum disorders (aged 18-35 years) will receive 12 weeks of CR therapy (n=20) or treatment as usual (n=20). Reward processing will be evaluated using a reinforcement learning task with 2 conditions (social reward vs nonsocial reward) at baseline and the 12-week follow-up. Functional magnetic resonance imaging responses will be measured during this task. To validate the reinforcement learning task, RL will also be assessed in 20 healthy controls, matched for age, sex, and premorbid functioning. Mixed-factorial ANOVAs will be conducted to evaluate treatment group differences. For the functional magnetic resonance imaging analysis, computational modeling will allow the estimation of learning parameters at each point in time, during each task condition, for each participant. We will use a variational Bayesian framework to measure how learning occurred during the experimental task and the subprocesses that underlie this learning. Second-level group analyses will examine how learning in patients differs from that observed in control participants and how CR alters learning efficiency and the underlying neural activity. As of September 2023, this study has enrolled 15 participants in the CR group, 1 participant in the treatment-as-usual group, and 11 participants in the healthy control group. Recruitment is expected to be completed by September 2024. Data analysis is expected to be completed and published in early 2025. The results of this study will contribute to the knowledge of CR and RL processes in severe mental illness and the understanding of the systems that impact negative symptoms and cognitive impairments within this population. DERR1-10.2196/52505.

Disentangling the contribution of individual and social learning processes in human advice-taking behavior

The study of social learning examines how individuals learn from others by means of observation, imitation, or compliance with advice. However, it still remains largely unknown whether social learning processes have a distinct contribution to behavior, independent from non-social trial-and-error learning that often occurs simultaneously. 153 participants completed a reinforcement learning task, where they were asked to make choices to gain rewards. Advice from an artificial teacher was presented in 60% of the trials, allowing us to compare choice behavior with and without advice. Results showed a strong and reliable tendency to follow advice (test-retest reliability ~0.73). Computational modeling suggested a unique contribution of three distinct learning strategies: (a) individual learning (i.e., learning the value of actions, independent of advice), (b) informed advice-taking (i.e., learning the value of following advice), and (c) non-informed advice-taking (i.e., a constant bias to follow advice regardless of outcome history). Comparing artificial and empirical data provided specific behavioral regression signatures to both informed and non-informed advice taking processes. We discuss the theoretical implications of integrating internal and external information during the learning process.

Exploring the steps of learning: computational modeling of initiatory-actions among individuals with attention-deficit/hyperactivity disorder

Attention-deficit/hyperactivity disorder (ADHD) is characterized by difficulty in acting in a goal-directed manner. While most environments require a sequence of actions for goal attainment, ADHD was never studied in the context of value-based sequence learning. Here, we made use of current advancements in hierarchical reinforcement-learning algorithms to track the internal value and choice policy of individuals with ADHD performing a three-stage sequence learning task. Specifically, 54 participants (28 ADHD, 26 controls) completed a value-based reinforcement-learning task that allowed us to estimate internal action values for each trial and stage using computational modeling. We found attenuated sensitivity to action values in ADHD compared to controls, both in choice and reaction-time variability estimates. Remarkably, this was found only for first-stage actions (i.e., initiatory actions), while for actions performed just before outcome delivery the two groups were strikingly indistinguishable. These results suggest a difficulty in following value estimation for initiatory actions in ADHD.

Study of a dual RIS-assisted beam assignment algorithm in a high-speed moving scenario

In order to improve the frequent Doppler bias in the high-speed mobile scenario, improve the communication quality on HSR (High-Speed Railway). An algorithm can improve communication performance through RIS (intelligent metasurface) phase optimization. The optimization process is modeled as a reinforcement learning task, where the agent in the channel learns to select the optimal RIS phase by interacting with the environment to reduce the effect of Doppler shift.The model covers characteristic parameters such as signal propagation path and frequency related to RIS, and the agent influences the beam direction by adjusting the phase, and learns the optimal phase configuration in the high-speed moving scene.The RIS phase optimization method implemented based on the deep reinforcement learning algorithm has high convergence and has significant advantages in improving the Doppler frequency shift problem. Compared with the traditional frequency bias estimation method, this method can significantly improve the transmission performance. The effectiveness of the proposed algorithm is verified by performing large-scale training and evaluation in the simulation environment.

Guided Cooperation in Hierarchical Reinforcement Learning via Model-Based Rollout.

Goal-conditioned hierarchical reinforcement learning (HRL) presents a promising approach for enabling effective exploration in complex, long-horizon reinforcement learning (RL) tasks through temporal abstraction. Empirically, heightened interlevel communication and coordination can induce more stable and robust policy improvement in hierarchical systems. Yet, most existing goal-conditioned HRL algorithms have primarily focused on the subgoal discovery, neglecting interlevel cooperation. Here, we propose a novel goal-conditioned HRL framework named Guided Cooperation via Model-Based Rollout (GCMR; code is available at https://github.com/HaoranWang-TJ/GCMR_ACLG_official), aiming to bridge interlayer information synchronization and cooperation by exploiting forward dynamics. First, the GCMR mitigates the state-transition error within off-policy correction via model-based rollout, thereby enhancing sample efficiency. Second, to prevent disruption by the unseen subgoals and states, lower level Q -function gradients are constrained using a gradient penalty with a model-inferred upper bound, leading to a more stable behavioral policy conducive to effective exploration. Third, we propose a one-step rollout-based planning, using higher level critics to guide the lower level policy. Specifically, we estimate the value of future states of the lower level policy using the higher level critic function, thereby transmitting global task information downward to avoid local pitfalls. These three critical components in GCMR are expected to facilitate interlevel cooperation significantly. Experimental results demonstrate that incorporating the proposed GCMR framework with a disentangled variant of hierarchical reinforcement learning guided by landmarks (HIGL), namely, adjacency constraint and landmark-guided planning (ACLG), yields more stable and robust policy improvement compared with various baselines and significantly outperforms previous state-of-the-art (SOTA) algorithms.

Data Quality-Aware Mixed-Precision Quantization via Hybrid Reinforcement Learning.

Mixed-precision quantization mostly predetermines the model bit-width settings before actual training due to the non-differential bit-width sampling process, obtaining suboptimal performance. Worse still, the conventional static quality-consistent training setting, i.e., all data is assumed to be of the same quality across training and inference, overlooks data quality changes in real-world applications which may lead to poor robustness of the quantized models. In this article, we propose a novel data quality-aware mixed-precision quantization framework, dubbed DQMQ, to dynamically adapt quantization bit-widths to different data qualities. The adaption is based on a bit-width decision policy that can be learned jointly with the quantization training. Concretely, DQMQ is modeled as a hybrid reinforcement learning (RL) task that combines model-based policy optimization with supervised quantization training. By relaxing the discrete bit-width sampling to a continuous probability distribution that is encoded with few learnable parameters, DQMQ is differentiable and can be directly optimized end-to-end with a hybrid optimization target considering both task performance and quantization benefits. Trained on mixed-quality image datasets, DQMQ can implicitly select the most proper bit-width for each layer when facing uneven input qualities. Extensive experiments on various benchmark datasets and networks demonstrate the superiority of DQMQ against existing fixed/mixed-precision quantization methods.

Reduced brain connectivity underlying value-based choices and outcomes in stimulant use disorder

BACKGROUND: Patients with stimulant use disorder (SUD) show impairments when making value-based choices that are associated with disruptions in neural processing across brain networks. Making optimal choices requires learning from outcomes to update knowledge and further optimise ongoing behaviour. The optimal functioning of neural systems that underpin the ability to make favourable choices is an essential component for life functioning, and successful recovery in patients with SUD. Therefore, we sought to investigate the neural processes that underpin value-based choices in SUD patients. We hypothesise that patients with SUD have reduced functional connectivity while making financial choices during a probabilistic reinforcement learning task. METHODS: We investigated connectivity associated with loss and reward value-based choices and their outcomes in patients with SUD and healthy control participants during a pharmacological magnetic resonance imaging study. Participants received a single dose of a dopamine receptor agonist, pramipexole, and a dopamine receptor antagonist, amisulpride, in a randomised, double-blind, placebo-controlled, balanced, crossover design. Functional task-related connectivity was analysed taking a whole brain connectomics approach to identify networks that are differentially modulated by dopaminergic receptor functioning. RESULTS: SUD patients showed widespread reductions in connectivity during both reward and loss value-based choices and outcomes, which were negatively correlated with the duration of stimulant drug use. Disturbances to functional brain connectivity in SUD patients during task performance were not modulated acutely by either amisulpride or pramipexole. CONCLUSIONS: Reductions in brain connectivity, particularly when making value-based choices and processing outcomes, may underlie learning impairments in SUD patients. Given that acute dopaminergic modulation did not improve brain connectivity impairments in SUD patients, it is likely that alternative treatments are needed.

Deep reinforcement learning task scheduling method based on server real-time performance.

Server load levels affect the performance of cloud task execution, which is rooted in the impact of server performance on cloud task execution. Traditional cloud task scheduling methods usually only consider server load without fully considering the server's real-time load-performance mapping relationship, resulting in the inability to evaluate the server's real-time processing capability accurately. This deficiency directly affects the efficiency, performance, and user experience of cloud task scheduling. Firstly, we construct a performance platform model to monitor server real-time load and performance status information in response to the above problems. In addition, we propose a new deep reinforcement learning task scheduling method based on server real-time performance (SRP-DRL). This method introduces a real-time performance-aware strategy and adds status information about the real-time impact of task load on server performance on top of considering server load. It enhances the perception capability of the deep reinforcement learning (DRL) model in cloud scheduling environments and improves the server's load-balancing ability under latency constraints. Experimental results indicate that the SRP-DRL method has better overall performance regarding task average response time, success rate, and server average load variance compared to Random, Round-Robin, Earliest Idle Time First (EITF), and Best Fit (BEST-FIT) task scheduling methods. In particular, the SRP-DRL is highly effective in reducing server average load variance when numerous tasks arrive within a unit of time, ultimately optimizing the performance of the cloud system.

Adversarial Stress Test for Autonomous Vehicle Via Series Reinforcement Learning Tasks With Reward Shaping

Adversarial Stress Test for Autonomous Vehicle Via Series Reinforcement Learning Tasks With Reward Shaping

Maternal selenium dietary supplementation alters sociability and reinforcement learning deficits induced by in utero exposure to maternal immune activation in mice

Maternal immune activation (MIA) during pregnancy increases the risk for the unborn foetus to develop neurodevelopmental conditions such as autism spectrum disorder and schizophrenia later in life. MIA mouse models recapitulate behavioural and biological phenotypes relevant to both conditions, and are valuable models to test novel treatment approaches. Selenium (Se) has potent anti-inflammatory properties suggesting it may be an effective prophylactic treatment against MIA. The aim of this study was to determine if Se supplementation during pregnancy can prevent adverse effects of MIA on offspring brain and behaviour in a mouse model.Selenium was administered via drinking water (1.5 ppm) to pregnant dams from gestational day (GD) 9 to birth, and MIA was induced at GD17 using polyinosinic:polycytidylic acid (poly-I:C, 20 mg/kg via intraperitoneal injection). Foetal placenta and brain cytokine levels were assessed using a Luminex assay and brain elemental nutrients assessed using inductively coupled plasma- mass spectrometry. Adult offspring were behaviourally assessed using a reinforcement learning paradigm, the three-chamber sociability test and the open field test. MIA elevated placental IL-1β and IL-17, and Se supplementation successfully prevented this elevation. MIA caused an increase in foetal brain calcium, which was prevented by Se supplement. MIA caused in offspring a female-specific reduction in sociability, which was recovered by Se, and a male-specific reduction in social memory, which was not recovered by Se. Exposure to poly-I:C or selenium, but not both, reduced performance in the reinforcement learning task. Computational modelling indicated that this was predominantly due to increased exploratory behaviour, rather than reduced rate of learning the location of the food reward.This study demonstrates that while Se may be beneficial in ameliorating sociability deficits caused by MIA, it may have negative effects in other behavioural domains. Caution in the use of Se supplementation during pregnancy is therefore warranted.

Recent Opioid Use Impedes Range Adaptation in Reinforcement Learning in Human Addiction

BackgroundDrugs like opioids are potent reinforcers thought to co-opt value-based decisions by overshadowing other rewarding outcomes, but how this happens at a neurocomputational level remains elusive. Range adaptation is a canonical process of fine-tuning representations of value based on reward context. Here, we tested whether recent opioid exposure impacts range adaptation in opioid use disorder, potentially explaining why shifting decision making away from drug taking during this vulnerable period is so difficult. MethodsParticipants who had recently (<90 days) used opioids (n = 34) or who had abstained from opioid use for ≥ 90 days (n = 20) and comparison control participants (n = 44) completed a reinforcement learning task designed to induce robust contextual modulation of value. Two models were used to assess the latent process that participants engaged while making their decisions: 1) a Range model that dynamically tracks context and 2) a standard Absolute model that assumes stationary, objective encoding of value. ResultsControl participants and ≥90-days-abstinent participants with opioid use disorder exhibited choice patterns consistent with range-adapted valuation. In contrast, participants with recent opioid use were more prone to learn and encode value on an absolute scale. Computational modeling confirmed the behavior of most control participants and ≥90-days-abstinent participants with opioid use disorder (75%), but a minority in the recent use group (38%), was better fit by the Range model than the Absolute model. Furthermore, the degree to which participants relied on range adaptation correlated with duration of continuous abstinence and subjective craving/withdrawal. ConclusionsReduced context adaptation to available rewards could explain difficulty deciding about smaller (typically nondrug) rewards in the aftermath of drug exposure.

Reinforced Imitative Graph Learning for Mobile User Profiling

Mobile user profiling refers to the efforts of extracting users’ characteristics from mobile activities. In order to capture the dynamic varying of user characteristics for generating effective user profiling, we propose an imitation-based mobile user profiling framework. Considering the objective of teaching an autonomous agent to imitate user mobility based on the user’s profile, the user profile is the most accurate when the agent can perfectly mimic the user behavior patterns. The profiling framework is formulated into a reinforcement learning task, where an agent is a next-visit planner, an action is a POI that a user will visit next, and the state of the environment is a fused representation of a user and spatial entities. An event in which a user visits a POI will construct a new state, which helps the agent predict users’ mobility more accurately. In the framework, we introduce a spatial Knowledge Graph (KG) to characterize the semantics of user visits over connected spatial entities. Additionally, we develop a mutual-updating strategy to quantify the state that evolves over time. Along these lines, we develop a reinforcement imitative graph learning framework for mobile user profiling. Finally, we conduct extensive experiments to demonstrate the superiority of our approach.

Solving Partially Observable 3D-Visual Tasks with Visual Radial Basis Function Network and Proximal Policy Optimization

Visual Reinforcement Learning (RL) has been largely investigated in recent decades. Existing approaches are often composed of multiple networks requiring massive computational power to solve partially observable tasks from high-dimensional data such as images. Using State Representation Learning (SRL) has been shown to improve the performance of visual RL by reducing the high-dimensional data into compact representation, but still often relies on deep networks and on the environment. In contrast, we propose a lighter, more generic method to extract sparse and localized features from raw images without training. We achieve this using a Visual Radial Basis Function Network (VRBFN), which offers significant practical advantages, including efficient and accurate training with minimal complexity due to its two linear layers. For real-world applications, its scalability and resilience to noise are essential, as real sensors are subject to change and noise. Unlike CNNs, which may require extensive retraining, this network might only need minor fine-tuning. We test the efficiency of the VRBFN representation to solve different RL tasks using Proximal Policy Optimization (PPO). We present a large study and comparison of our extraction methods with five classical visual RL and SRL approaches on five different first-person partially observable scenarios. We show that this approach presents appealing features such as sparsity and robustness to noise and that the obtained results when training RL agents are better than other tested methods on four of the five proposed scenarios.

Instance Weighted Incremental Evolution Strategies for Reinforcement Learning in Dynamic Environments.

Evolution strategies (ESs), as a family of black-box optimization algorithms, recently emerge as a scalable alternative to reinforcement learning (RL) approaches such as Q-learning or policy gradient and are much faster when many central processing units (CPUs) are available due to better parallelization. In this article, we propose a systematic incremental learning method for ES in dynamic environments. The goal is to adjust previously learned policy to a new one incrementally whenever the environment changes. We incorporate an instance weighting mechanism with ES to facilitate its learning adaptation while retaining scalability of ES. During parameter updating, higher weights are assigned to instances that contain more new knowledge, thus encouraging the search distribution to move toward new promising areas of parameter space. We propose two easy-to-implement metrics to calculate the weights: instance novelty and instance quality. Instance novelty measures an instance's difference from the previous optimum in the original environment, while instance quality corresponds to how well an instance performs in the new environment. The resulting algorithm, instance weighted incremental evolution strategies (IW-IESs), is verified to achieve significantly improved performance on challenging RL tasks ranging from robot navigation to locomotion. This article thus introduces a family of scalable ES algorithms for RL domains that enables rapid learning adaptation to dynamic environments.

Deep reinforcement learning with significant multiplications inference

We propose a sparse computation method for optimizing the inference of neural networks in reinforcement learning (RL) tasks. Motivated by the processing abilities of the brain, this method combines simple neural network pruning with a delta-network algorithm to account for the input data correlations. The former mimics neuroplasticity by eliminating inefficient connections; the latter makes it possible to update neuron states only when their changes exceed a certain threshold. This combination significantly reduces the number of multiplications during the neural network inference for fast neuromorphic computing. We tested the approach in popular deep RL tasks, yielding up to a 100-fold reduction in the number of required multiplications without substantial performance loss (sometimes, the performance even improved).

Approach-avoidance reinforcement learning as a translational and computational model of anxiety-related avoidance.

Although avoidance is a prevalent feature of anxiety-related psychopathology, differences in the measurement of avoidance between humans and non-human animals hinder our progress in its theoretical understanding and treatment. To address this, we developed a novel translational measure of anxiety-related avoidance in the form of an approach-avoidance reinforcement learning task, by adapting a paradigm from the non-human animal literature to study the same cognitive processes in human participants. We used computational modelling to probe the putative cognitive mechanisms underlying approach-avoidance behaviour in this task and investigated how they relate to subjective task-induced anxiety. In a large online study (n = 372), participants who experienced greater task-induced anxiety avoided choices associated with punishment, even when this resulted in lower overall reward. Computational modelling revealed that this effect was explained by greater individual sensitivities to punishment relative to rewards. We replicated these findings in an independent sample (n = 627) and we also found fair-to-excellent reliability of measures of task performance in a sub-sample retested 1 week later (n = 57). Our findings demonstrate the potential of approach-avoidance reinforcement learning tasks as translational and computational models of anxiety-related avoidance. Future studies should assess the predictive validity of this approach in clinical samples and experimental manipulations of anxiety.

Approach-avoidance reinforcement learning as a translational and computational model of anxiety-related avoidance

Although avoidance is a prevalent feature of anxiety-related psychopathology, differences in the measurement of avoidance between humans and non-human animals hinder our progress in its theoretical understanding and treatment. To address this, we developed a novel translational measure of anxiety-related avoidance in the form of an approach-avoidance reinforcement learning task, by adapting a paradigm from the non-human animal literature to study the same cognitive processes in human participants. We used computational modelling to probe the putative cognitive mechanisms underlying approach-avoidance behaviour in this task and investigated how they relate to subjective task-induced anxiety. In a large online study (n = 372), participants who experienced greater task-induced anxiety avoided choices associated with punishment, even when this resulted in lower overall reward. Computational modelling revealed that this effect was explained by greater individual sensitivities to punishment relative to rewards. We replicated these findings in an independent sample (n = 627) and we also found fair-to-excellent reliability of measures of task performance in a sub-sample retested 1 week later (n = 57). Our findings demonstrate the potential of approach-avoidance reinforcement learning tasks as translational and computational models of anxiety-related avoidance. Future studies should assess the predictive validity of this approach in clinical samples and experimental manipulations of anxiety.

Hybrid Policy-Based Reinforcement Learning of Adaptive Energy Management for the Energy Transmission-Constrained Island Group

This paper proposes a hybrid policy-based reinforcement learning (HPRL) adaptive energy management to realize the optimal operation for the island group energy system with energy transmission-constrained environment. An island energy hub (IEH) model that can realize the energy cascade utilization is proposed. Compared with the traditional model, the IEH can satisfy the special energy demand of island meanwhile ensure the energy supply of island. Moreover, an energy management model of islands group (EMIG) based on the IEH is formulated which comprehensively considers the inverse distribution of energy demand and resources, as well as the limited energy transmission. Since the environment model of island is difficult to construct due to the increase of proportion of renewable energy generation and civilian load, the EMIG is transformed into a reinforcement learning (RL) task which features model-free. Considering the limitations of traditional RL in discrete-continuous hybrid action space, HPRL is proposed to achieve optimal operation without simplifying the model. Numerical simulations demonstrate the effectiveness of the proposed adaptive energy management.

A Graph-Based PPO Approach in Multi-UAV Navigation for Communication Coverage

Multi-Agent Reinforcement Learning (MARL) is widely used to solve various problems in real life. In the multi-agent reinforcement learning tasks, there are multiple agents in the environment, the existing Proximal Policy Optimization (PPO) algorithm can be applied to multi-agent reinforcement learning. However, it cannot deal with the communication problem between agents. In order to resolve this issue, we propose a Graph-based PPO algorithm, this approach can solve the communication problem between agents and it can enhance the exploration efficiency of agents in the environment and speed up the learning process. We apply our algorithms to the task of multi-UAV navigation for communication coverage to verify the functionality and performance of our proposed algorithms.