Reinforcement Learning Research Articles

Distributed representations of temporally accumulated reward prediction errors in the mouse cortex.

Reward prediction errors (RPEs) quantify the difference between expected and actual rewards, serving to refine future actions. Although reinforcement learning (RL) provides ample theoretical evidence suggesting that the long-term accumulation of these error signals improves learning efficiency, it remains unclear whether the brain uses similar mechanisms. To explore this, we constructed RL-based theoretical models and used multiregional two-photon calcium imaging in the mouse dorsal cortex. We identified a population of neurons whose activity was modulated by varying degrees of RPE accumulation. Consequently, RPE-encoding neurons were sequentially activated within each trial, forming a distributed assembly. RPE representations in mice aligned with theoretical predictions of RL, emerging during learning and being subject to manipulations of the reward function. Interareal comparisons revealed a region-specific code, with higher-order cortical regions exhibiting long-term encoding of RPE accumulation. These results present an additional layer of complexity in cortical RPE computation, potentially augmenting learning efficiency in animals.

Adaptive weight adjustment technology for reinforcement learning reward indicators based on the new-type power system

Current RL-based scheduling methods struggle with designing effective reward functions that balance cost reduction, safety and renewable energy integration. To tackle these issues, this paper proposes a novel hybrid optimization framework that combines Particle Swarm Optimization and Reinforcement Learning (PSO-RL), focusing on three key aspects: (1) Minimizing Generation Costs: Systematically reducing operational costs for economic efficiency. (2) Enhancing System Safety: Improving reliability by incorporating operational constraints. (3) Increasing Renewable Energy Proportion: Emphasizing the integration of renewable energy sources. The comprehensive mathematical model encapsulates intricate constraints and multi-objective characteristics of the new-type power system, featuring reward functions tailored for dynamic responsiveness. Empirical validations using the Grid2Op scheduler affirm the efficacy of our framework. Results highlight superior convergence speed and optimization precision of the PSO-RL approach over traditional RL methods. In summary, this study advances scheduling methodologies for the new-type power system, fostering renewable integration and smart grid technology evolution.

Target Search and Navigation in Heterogeneous Robot Systems with Deep Reinforcement Learning

Target Search and Navigation in Heterogeneous Robot Systems with Deep Reinforcement Learning

Optimising multi-element cut-off grades for a strategic production plan under geological uncertainty

ABSTRACT Mines operate in uncertain heterogenous environments, in which the most profitable material must be delineated from lower value-material. Traditionally, single-element cut-off grades provide this delineation and methods for optimising date back to the 1960s. Recent advancements have included geological uncertainty, multiple elements, and production scheduling; however, no method combines all three improvements. Two multi-element cut-off grade definitions are proposed herein, along with a reinforcement learning framework for optimising long-term multi-element cut-off grades under geological uncertainty for an optimal production plan. The method is applied to a gold-copper mining complex, and the performance of both cut-off grade definitions is compared.

Deep Reinforcement Learning for automated scheduling of mining earthwork equipment with spatio-temporal safety constraints

Deep Reinforcement Learning for automated scheduling of mining earthwork equipment with spatio-temporal safety constraints

Reinforcement Learning is Impaired in the Sub-acute Post-stroke Period.

In humans, most spontaneous recovery from motor impairment after stroke occurs in the first 3 months. Studies in animal models show higher responsiveness to training over a similar time-period. Both phenomena are often attributed to a milieu of heightened plasticity, which may share some mechanistic overlap with plasticity associated with normal motor learning. Given that neurorehabilitation approaches are frequently predicated on motor learning principles, here we asked if the sensitivity of trial-to-trial learning for 2 kinds of motor learning processes often involved during rehabilitation is also enhanced early post-stroke. In a cross-sectional design, we compared (1) reinforcement and (2) error-based learning in 2 groups: 1 tested within 3 months after stroke (early group, N = 35) another tested more than 6 months after stroke (late group, N = 30). These 2 forms of motor learning were assessed with variations of the same visuomotor rotation task. Critically, motor execution was matched between the 2 groups. Reinforcement learning was impaired in the early but not the late group, whereas error-based learning was unimpaired in either group. These findings could not be attributed to differences in baseline execution, cognitive impairment, gender, age, or lesion volume and location. The presence of a deficit in reinforcement motor learning in the first 3 months after stroke has important implications for rehabilitation. It might be necessary to either increase reinforcement feedback given early after stroke, increase the dose of rehabilitation to compensate, or delay onset of rehabilitation approaches that may rely on reinforcement, for example, constraint-induced movement therapy, and instead emphasize other forms of motor training in the subacute time period.

Breast radiation therapy fluence painting with multi-agent deep reinforcement learning.

The electronic compensation (ECOMP) technique for breast radiation therapy provides excellent dose conformity and homogeneity. However, the manual fluence painting process presents a challenge for efficient clinical operation. To facilitate the clinical treatment planning automation of breast radiation therapy, we utilized reinforcement learning (RL) to develop an auto-planning tool that iteratively edits the fluence maps under the guidance of clinically relevant objectives. With institutional review board (IRB) approval, 70 patients treated with 6MV tangential photon beams with ECOMP technique were retrospectively collected and included in this study (20/50 for training/testing). Each pixel in the fluence map was assigned a reinforcement learning agent to perform independent action. Beam-eye-view projected dose profiles were generated to form state information as the input of the RL network. By predicting the Q value, pixel-wise actions were selected to modify specific pixel value in the fluence maps to improve overall plan quality. After dose calculation, reward signal calculated from the variation of target coverage and dose homogeneity was fed back to the RL framework and used to update network parameters. The RL generated plans were evaluated with dose distribution and dosimetric endpoints (i.e., Breast PTV V90%, Breast PTV V95%, Breast PTV V105%, Lung V20Gy, Heart V5Gy, Dmax) and compared with clinical plans. The RL agent took around 90 s to generate a ECOMP treatment plan. The RL plans exhibited plan quality comparable to clinical plans in terms of isodose distribution and dosimetric endpoints. The mean Breast PTV V95%, Breast PTV V105% of RL plans are and , compared to and cc of clinical plans. The developed RL framework efficiently generates breast ECOMP plans with clinical acceptable plan quality.

Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML

Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML

Computational Predictor–Corrector Homing Guidance for Constrained Impact

This paper proposes a computational predictor–corrector guidance algorithm for homing missiles to constrain the terminal impact angle and impact time. The guidance algorithm developed leverages a cascaded design philosophy to cater to the terminal constraints and considers the variation of the moving speed of the missile subject to both drag and gravity forces. The inner loop employs a deep neural network as a predictor to accurately forecast the impact angle, and a reinforcement learning algorithm serves as a corrector that generates a biased guidance command to eliminate the impact angle error. The outer loop uses a nested neural predictor–corrector that considers the inner loop command as the baseline guidance command to address the constraint on impact time. Extensive numerical simulation and several comparisons validate the performance of the proposed guidance algorithm in this paper.

A graph reinforcement learning framework for real-time distributed multi-robot task allocation

A graph reinforcement learning framework for real-time distributed multi-robot task allocation

Gradient-Based Framework for Bilevel Optimization of Black-Box Functions: Synergizing Model-Free Reinforcement Learning and Implicit Function Differentiation

Gradient-Based Framework for Bilevel Optimization of Black-Box Functions: Synergizing Model-Free Reinforcement Learning and Implicit Function Differentiation

Action Correction-Enhanced Multi-Agent Reinforcement Learning for Path Planning in Urban Environments

Action Correction-Enhanced Multi-Agent Reinforcement Learning for Path Planning in Urban Environments

Optimizing Biomimetic 3D Disordered Fibrous Network Structures for Lightweight, High-Strength Materials via Deep Reinforcement Learning.

3D disordered fibrous network structures (3D-DFNS), such as cytoskeletons, collagen matrices, and spider webs, exhibit remarkable material efficiency, lightweight properties, and mechanical adaptability. Despite their widespread in nature, the integration into engineered materials is limited by the lack of study on their complex architectures. This study addresses the challenge by investigating the structure-property relationships and stability of biomimetic 3D-DFNS using large datasets generated through procedural modeling, coarse-grained molecular dynamics simulations, and machine learning. Based on these datasets, a network deep reinforcement learning (N-DRL) framework is developed to optimize its stability, effectively balancing weight reduction with the maintenance of structural integrity. The results reveal a pronounced correlation between the total fiber length in 3D-DFNS and its mechanical properties, where longer fibers enhance stress distribution and stability. Additionally, fiber orientation is also considered as a potential factor influencing stress growth values. Furthermore, the N-DRL model demonstrates superior performance compared to traditional approaches in optimizing network stability while minimizing mass and computational cost. Structural integrity is significantly improved through the addition of triple junctions and the reduction of higher-order nodes. In summary, this study leverages machine learning to optimize biomimetic 3D-DFNS, providing novel insights into the design of lightweight, high-strength materials.

Reinforcement learning-based vehicle travel path reconstruction from sparse automatic licence plate recognition data

ABSTRACT Automatic licence plate recognition (ALPR) data is a vital source for acquiring large-scale vehicle trajectory data in urban transportation research. However, the sparse distribution of ALPR sensors often results in incomplete vehicle trajectories with unobserved travel paths between adjacent ALPR sensors. To address this challenge, this study proposes a novel travel path reconstruction model based on reinforcement learning. First, the maximum entropy inverse reinforcement learning (MaxEnt IRL) method is employed to learn vehicle path selection strategy, i.e. the reward function, from the real data. Then, a comprehensive reward function is formulated by integrating destination rewards. Finally, the Q-learning algorithm is utilized to reconstruct vehicle travel paths between adjacent ALPR sensors based on the comprehensive reward function. Evaluation on a randomly selected set of 100 pairs of adjacent ALPR sensors demonstrates that the proposed model significantly outperforms two baseline methods, achieving reductions of 24.3% and 22.5% in deviation from the real paths.

Time-optimal Flight in Cluttered Environments via Safe Reinforcement Learning

Time-optimal Flight in Cluttered Environments via Safe Reinforcement Learning

Delta opioid receptors affect acoustic features of song during vocal learning in zebra finches

Delta-opioid receptors (δ-ORs) are known to be involved in associative learning and modulating motivational states. We wanted to study if they were also involved in naturally-occurring reinforcement learning behaviors such as vocal learning, using the zebra finch model system. Zebra finches learn to vocalize early in development and song learning in males is affected by factors such as the social environment and internal reward, both of which are modulated by endogenous opioids. Pairs of juvenile male siblings (35-day-old) were systemically administered a δ-OR-selective antagonist naltrindole or vehicle (controls) for a period of 10 days. The acoustic structure of songs differed across treated and control groups at adulthood (120 days). Naltrindole-treated birds had a significantly lower pitch, mean frequency, and frequency modulation than controls, whereas there was no difference in the number of songs in naltrindole-treated and control siblings. Since the opioid and dopaminergic systems interact, we decided to study whether blocking δ-ORs during the sensitive period led to changes in dopaminoceptive neurons in Area X, a song control nucleus in the basal ganglia. Interestingly, compared with controls, naltrindole-treated birds had higher numbers of DARPP-32-positive medium spiny neurons and potentially excitatory synapses in Area X. We show that manipulating δ-OR signaling during the learning phase resulted in alterations in the acoustic features of song and had long term effects on dopaminergic targets within the basal ganglia in adulthood. Our results suggest that endogenous opioids regulate the development of cognitive processes and the underlying neural circuitry during the sensitive period for learning.

Accelerating materials recipe acquisition via LLM-mediated reinforcement learning

Accelerating materials recipe acquisition via LLM-mediated reinforcement learning

Cost-effective dynamic sampling in high dimensional online monitoring with advantage actor-critic

Industry 5.0 puts a spotlight on the team effort between people and machines, making the fast-paced development of IoT technology and sensor systems crucial. However, limitations such as the high cost of data acquisition, constrained bandwidth, and computational capacity can often hinder the real-time processing needed for human machine collaboration. To address this issue, we propose an integrated framework using Advantage Actor-Critic (A2C) and statistical process control (SPC) for economically monitoring high-dimensional data streams online. Our method leverages deep reinforcement learning with SPC charts to quickly identify system issues by smartly watching a few essential data streams. This approach not only saves resources but also boosts both the sustainability and efficiency of monitoring systems. Specifically, we construct a nonparametric monitoring statistic for each stream and develop a reinforcement learning framework to automatically identify the most informative and minimal number of data streams for observation at each time epoch. Our strategy is proven effective through simulations and a practical example in smart manufacturing, showcasing its superiority in reducing detection delay and minimising resources use compared to state-of-the-art algorithms.

An end-to-end decentralised scheduling framework based on deep reinforcement learning for dynamic distributed heterogeneous flowshop scheduling

Heterogeneity among factories in distributed manufacturing significantly expands the solution space, complicating optimisation. Traditional centralised scheduling methods lack the scalability to adapt to varying factory scales. This paper proposes an end-to-end decentralised scheduling framework based on deep reinforcement learning (DRL) for dynamic distributed heterogeneous permutation flowshop scheduling problem (DDHPFSP) with random job arrivals. The framework utilises a multi-agent architecture, where each factory operates as an independent agent, enabling efficient, robust, and scalable scheduling. Specifically, the DDHPFSP is formulated as a partially observable Markov decision process (POMDP), with a state space reflecting heterogeneity and permutation characteristics and a new tailored reward function addressing sparse rewards and high reward variance. An end-to-end policy network with dual-layer architecture is developed, incorporating a feature extraction network to capture intrinsic relationships between jobs and heterogeneous factories, enhancing the agent's self-learning and policy evolution. Moreover, a backward swap search (BSS) method based on greedy heuristics optimises the pre-scheduling plan during the online phase with minimal computation time. Experimental results demonstrate the framework outperforms the best comparison methods by 39.76% on 540 baseline instances and 59.95% on 2430 generalisation instances. Furthermore, the framework's effectiveness improves by 68.9% with the introduction of the BSS method.

Research on the structural design of exoskeleton assisted transport robot combined with reinforcement learning algorithm under the background of artificial intelligence

Research on the structural design of exoskeleton assisted transport robot combined with reinforcement learning algorithm under the background of artificial intelligence