Reinforcement Learning Research Articles

Time-optimal Flight in Cluttered Environments via Safe Reinforcement Learning

Time-optimal Flight in Cluttered Environments via Safe Reinforcement Learning

Delta opioid receptors affect acoustic features of song during vocal learning in zebra finches

Delta-opioid receptors (δ-ORs) are known to be involved in associative learning and modulating motivational states. We wanted to study if they were also involved in naturally-occurring reinforcement learning behaviors such as vocal learning, using the zebra finch model system. Zebra finches learn to vocalize early in development and song learning in males is affected by factors such as the social environment and internal reward, both of which are modulated by endogenous opioids. Pairs of juvenile male siblings (35-day-old) were systemically administered a δ-OR-selective antagonist naltrindole or vehicle (controls) for a period of 10 days. The acoustic structure of songs differed across treated and control groups at adulthood (120 days). Naltrindole-treated birds had a significantly lower pitch, mean frequency, and frequency modulation than controls, whereas there was no difference in the number of songs in naltrindole-treated and control siblings. Since the opioid and dopaminergic systems interact, we decided to study whether blocking δ-ORs during the sensitive period led to changes in dopaminoceptive neurons in Area X, a song control nucleus in the basal ganglia. Interestingly, compared with controls, naltrindole-treated birds had higher numbers of DARPP-32-positive medium spiny neurons and potentially excitatory synapses in Area X. We show that manipulating δ-OR signaling during the learning phase resulted in alterations in the acoustic features of song and had long term effects on dopaminergic targets within the basal ganglia in adulthood. Our results suggest that endogenous opioids regulate the development of cognitive processes and the underlying neural circuitry during the sensitive period for learning.

Market-aware Long-term Job Skill Recommendation with Explainable Deep Reinforcement Learning

Continuously learning new skills is essential for talents to gain a competitive advantage in the labor market. Despite extensive efforts on relevance- or preference-based skill recommendations, little attention has been given to the practical effects of job skills in the market. To bridge this gap, we propose an explainable personalized skill learning recommendation system that considers the long-term learning benefits and costs. Specifically, we model skill learning utilities based on salary and learning cost associated with job positions and propose a multi-objective deep reinforcement learning framework to model and maximize long-term utilities. Furthermore, we propose a Self-explaining Skill Recommendation Deep Q-network (SeSRDQN) that captures and prototypes prevalent skill sets in the market into representative exemplars for decision-making. SeSRDQN quantitatively decomposes the talent’s long-term learning utility into contributions from each exemplar, offering a comprehensive and multi-factorial explanation across various skill learning options. To tackle the combinatorial complexity of the skill space, we develop an MCTS-based optimization-decoding iterative training procedure for explanation fidelity and human understandability. In this way, talents will receive a tailored roadmap of essential skills, complemented by exemplar-based explanations, to effectively plan their careers. Extensive experiments on a real-world dataset validate the effectiveness and explainability of our approach.

Accelerating materials recipe acquisition via LLM-mediated reinforcement learning

Accelerating materials recipe acquisition via LLM-mediated reinforcement learning

Image-Based Tactile Deformation Simulation and Pose Estimation for Robot Skill Learning

The TacTip is a cost-effective, 3D-printed optical tactile sensor commonly used in deep learning and reinforcement learning for robotic manipulation. However, its specialized structure, which combines soft materials of varying hardnesses, makes it challenging to simulate the distribution of numerous printed markers on pins. This paper aims to create an interpretable, AI-applicable simulation of the deformation of TacTip under varying pressures and interactions with different objects, addressing the black-box nature of learning and simulation in haptic manipulation. The research focuses on simulating the TacTip sensor’s shape using a fully tunable, chain-based mathematical model, refined through comparisons with real-world measurements. We integrated the WRS system with our theoretical model to evaluate its effectiveness in object pose estimation. The results demonstrated that the prediction accuracy for all markers across a variety of contact scenarios exceeded 92%.

Carbon Emission Reduction in Traffic Control: A Signal Timing Optimization Method Based on Rainbow DQN

To improve intersection traffic flow and reduce vehicle energy consumption and emissions at intersections, a signal optimization method based on deep reinforcement learning (DRL) is proposed. The algorithm uses Rainbow DQN as the core framework, incorporating vehicle position, speed, and acceleration information into the state space. The reward function simultaneously considers two objectives: reducing vehicle waiting times and minimizing carbon emissions, with the vehicle queue length as a weighted factor. Additionally, an ACmix module, which integrates self-attention mechanisms and convolutional layers, is introduced to enhance the model’s feature extraction and information representation capabilities, improving computational efficiency. The model is tested using an actual intersection as the study object, with a signal intersection simulation built in SUMO. The proposed approach is compared with traditional Webster signal timing, actuated signal timing, and control strategies based on DQN and D3QN models. The results show that the proposed strategy, through real-time signal timing adjustments, reduces the average vehicle waiting time by approximately 27.58% and the average CO2 emissions by about 7.34% compared with the actuated signal timing method. A comparison with DQN and D3QN models further demonstrates the superiority of the proposed model, achieving a 15% reduction in average waiting time and a 6.5% reduction in CO2 emissions. The model’s applicability is validated under various scenarios, including different proportions of electric vehicles and traffic volumes. This study aims to provide a flexible signal control strategy to enhance intersection vehicle flow and reduce carbon emissions. It offers a reference for the development of green, intelligent transportation systems and holds practical significance for promoting urban carbon reduction efforts.

A Reinforcement Learning-Based Solution for the Capacitated Electric Vehicle Routing Problem from the Last-Mile Delivery Perspective

The growth of the urban population and the increase in e-commerce activities have resulted in challenges for last-mile delivery. On the other hand, electric vehicles (EVs) have been introduced to last-mile delivery as an alternative to fossil fuel vehicles. Electric vehicles (EVs) not only play a pivotal role in reducing greenhouse gas emissions and air pollution but also contribute significantly to the development of more energy-efficient and environmentally sustainable urban transportation systems. Within these dynamics, the Electric Vehicle Routing Problem (EVRP) has begun to replace the Vehicle Routing Problem (VRP) in last-mile delivery. While classic vehicle routing ignores fueling, both the location of charging stations and charging time should be included in the Electric Vehicle Routing Problem due to the long recharging time. This study addresses the Capacitated EVRP (CEVRP) with a novel Q-learning algorithm. Q-learning is a model-free reinforcement learning algorithm designed to maximize an agent’s cumulative reward over time by selecting optimal actions. Additionally, a new dataset is also published for the EVRP considering field constraints. For the design of the dataset, real geographical positions have been used, located in the province of Eskisehir, Türkiye. It also includes environmental information, such as streets, intersections, and traffic density, unlike classical EVRP datasets. Optimal solutions are obtained for each instance of the EVRP by using the mathematical model. The results of the proposed Q-learning algorithm are compared with the optimal solutions of the presented dataset. Test results show that the proposed algorithm provides remarkable advantages in obtaining routes in a shorter time for EVs.

Study on the Multi-Equipment Integrated Scheduling Problem of a U-Shaped Automated Container Terminal Based on Graph Neural Network and Deep Reinforcement Learning

Intelligent Guided Vehicles (IGVs) in U-shaped automated container terminals (ACTs) have longer travel paths than those in conventional vertical layout ACTs, and their interactions with double trolley quay cranes (DTQCs) and double cantilever rail cranes (DCRCs) are more frequent and complex, so the scheduling strategy of a traditional ACT cannot easily be applied to a U-shaped ACT. With the aim of minimizing the maximum task completion times within a U-shaped ACT, this study investigates the integrated scheduling problem of DTQCs, IGVs and DCRCs under the hybrid “loading and unloading” mode, expresses the problem as a Markovian decision-making process, and establishes a disjunctive graph model. A deep reinforcement learning algorithm based on a graph neural network combined with a proximal policy optimization algorithm is proposed. To verify the superiority of the proposed models and algorithms, instances of different scales were stochastically generated to compare the proposed method with several heuristic algorithms. This study also analyses the idle time of the equipment under two loading and unloading modes, and the results show that the hybrid mode can enhance the operational effectiveness. of the U-shaped ACT.

Research on UAV Trajectory Planning Algorithm Based on Adaptive Potential Field

For multi-obstacle complex scenarios, the traditional artificial potential field method suffers from the defects of potential field imbalance, its capability to easily fall into the local minima, and encounter unreachable targets in complex navigation environments. Therefore, this paper proposes a three-dimensional adaptive potential field algorithm (SAPF) based on multi-agent reinforcement learning. First, in this paper, the gravitational function in the artificial potential field (APF) is modified to weaken the gravitational effect on the UAV in the region far away from the target point in order to reduce the risk of collision between the UAV and the obstacles during the moving process. Second, in the region close to the target point, this paper improves the artificial potential field function to ensure that the UAV can reach the target point smoothly and realize path convergence by considering the relative distance between the UAV’s current position and the target point. Finally, for the characteristics of UAV trajectory planning, a 3D state space is designed based on the 3D coordinates of the UAV, the distance between the UAV and the nearest obstacle, and the distance between the UAV and the target point; an action space is designed based on the displacement increment of the UAV in the three coordinate axes; and the specific formulas for collision penalties and path optimization rewards are re-designed, which effectively avoids the UAV from entering the local minimal points. The experimental results show that the artificial potential field method designed with reinforcement learning can plan shorter paths and exhibit better planning results. In addition, the method is more adaptable in complex scenes and has better anti-interference.

A reinforcement learning methodology to hierarchical sliding‐mode surface H∞$$ {H}_{\infty } $$ control of nonlinear systems via a dynamic event‐triggered mechanism

SummaryThis paper addresses the problem of a hierarchical sliding mode surface (HSMS) control design for nonlinear systems via a dynamic event‐triggered mechanism. Initially, the HSMS containing the system states is constructed to enhance the system's response rate and robustness. By assigning a cost function associated with the HSMS, such an control problem is equivalently transformed into a zero‐sum game problem, where the control policy and the exogenous disturbance are treated as two players with opposite interests. Afterwards, a novel dynamic event‐triggered mechanism is designed, where the triggering condition depends on HSMS variables. To solve the corresponding event‐triggered Hamilton–Jacobi–Isaacs equation, a single‐critic reinforcement learning algorithm is developed, which removes the error generated by approximating the actor network in the actor‐critic network. According to the Lyapunov stability theory, all signals of the considered system are strictly proved to be bounded. Finally, the validity of the proposed control method is demonstrated through simulations of a tunnel diode circuit system and a mass‐spring‐damper system.

Cost-effective dynamic sampling in high dimensional online monitoring with advantage actor-critic

Industry 5.0 puts a spotlight on the team effort between people and machines, making the fast-paced development of IoT technology and sensor systems crucial. However, limitations such as the high cost of data acquisition, constrained bandwidth, and computational capacity can often hinder the real-time processing needed for human machine collaboration. To address this issue, we propose an integrated framework using Advantage Actor-Critic (A2C) and statistical process control (SPC) for economically monitoring high-dimensional data streams online. Our method leverages deep reinforcement learning with SPC charts to quickly identify system issues by smartly watching a few essential data streams. This approach not only saves resources but also boosts both the sustainability and efficiency of monitoring systems. Specifically, we construct a nonparametric monitoring statistic for each stream and develop a reinforcement learning framework to automatically identify the most informative and minimal number of data streams for observation at each time epoch. Our strategy is proven effective through simulations and a practical example in smart manufacturing, showcasing its superiority in reducing detection delay and minimising resources use compared to state-of-the-art algorithms.

An end-to-end decentralised scheduling framework based on deep reinforcement learning for dynamic distributed heterogeneous flowshop scheduling

Heterogeneity among factories in distributed manufacturing significantly expands the solution space, complicating optimisation. Traditional centralised scheduling methods lack the scalability to adapt to varying factory scales. This paper proposes an end-to-end decentralised scheduling framework based on deep reinforcement learning (DRL) for dynamic distributed heterogeneous permutation flowshop scheduling problem (DDHPFSP) with random job arrivals. The framework utilises a multi-agent architecture, where each factory operates as an independent agent, enabling efficient, robust, and scalable scheduling. Specifically, the DDHPFSP is formulated as a partially observable Markov decision process (POMDP), with a state space reflecting heterogeneity and permutation characteristics and a new tailored reward function addressing sparse rewards and high reward variance. An end-to-end policy network with dual-layer architecture is developed, incorporating a feature extraction network to capture intrinsic relationships between jobs and heterogeneous factories, enhancing the agent's self-learning and policy evolution. Moreover, a backward swap search (BSS) method based on greedy heuristics optimises the pre-scheduling plan during the online phase with minimal computation time. Experimental results demonstrate the framework outperforms the best comparison methods by 39.76% on 540 baseline instances and 59.95% on 2430 generalisation instances. Furthermore, the framework's effectiveness improves by 68.9% with the introduction of the BSS method.

Research on the structural design of exoskeleton assisted transport robot combined with reinforcement learning algorithm under the background of artificial intelligence

Research on the structural design of exoskeleton assisted transport robot combined with reinforcement learning algorithm under the background of artificial intelligence

Trajectory Tracking Control Based on Deep Reinforcement Learning for a Robotic Manipulator with an Input Deadzone

Trajectory Tracking Control Based on Deep Reinforcement Learning for a Robotic Manipulator with an Input Deadzone

Improvised robotic navigation using deep reinforcement learning (DRL) towards safer integration in real-time complex environments

Improvised robotic navigation using deep reinforcement learning (DRL) towards safer integration in real-time complex environments

Mechanisms of increased pain discrimination by contingent reinforcement: a perceptual decision-making and instrumental learning account.

Recent evidence highlights that monetary rewards can increase the precision at which healthy human volunteers can detect small changes in the intensity of thermal noxious stimuli, contradicting the idea that rewards exert a broad inhibiting influence on pain perception. This effect was stronger with contingent rewards compared with noncontingent rewards, suggesting a successful learning process. In the present study, we implemented a model comparison approach that aimed to improve our understanding of the mechanisms that underlie thermal noxious discrimination in humans. In a between-subject design, 54 healthy human volunteers took part in a pain discrimination task with monetary rewards either contingent or noncontingent on successful discrimination of small changes in the intensity painful heat stimulation. We used models from 2 traditions in decision-making research, perceptual decision-making, and instrumental learning. Replicating the previous findings, only rewards contingent on behavior enhanced pain discrimination. Drift diffusion modelling revealed increased sensory signal strength and decreased response caution and nondecision times as mechanisms underlying this effect of contingent rewards on pain discrimination. In addition, reinforcement learning models indicated a temporal evolution of discriminative abilities reflected by a trial-by-trial increase of perceived signal strength only with contingent rewards but not with noncontingent rewards. Modelling of separate learning rates for positive and negative prediction errors indicated that this temporal evolution of discriminative abilities was driven by positive reward prediction errors. These results might indicate increased sensitivity towards better-than-expected outcomes in the temporal adaptation of pain discrimination abilities to a rewarding context in humans.

Stackelberg game-based anti-disturbance control for unmanned surface vessels via integrative reinforcement learning

Stackelberg game-based anti-disturbance control for unmanned surface vessels via integrative reinforcement learning

Dynamic Foraging in Swarm Robotics: A Hybrid Approach with Modular Design and Deep Reinforcement Learning Intelligence

This paper proposes a hybrid approach that combines intelligent algorithms and modular design to solve a foraging problem within the context of swarm robotics. Deep reinforcement learning (RL) and particle swarm optimization (PSO) are deployed in the proposed modular architecture. They are utilized to search for many resources that vary in size and exhibit a dynamic nature with unpredictable movements. Additionally, they transport the collected resources to the nest. The swarm comprises 8 E-Puck mobile robots, each equipped with light sensors. The proposed system is built on a 3D environment using the Webots simulator. Through a modular approach, we address complex foraging challenges characterized by a non-static environment and objectives. This architecture enhances manageability, reduces computational demands, and facilitates debugging processes. Our simulations reveal that the RL-based model outperforms PSO in terms of task completion time, efficiency in collecting resources, and adaptability to dynamic environments, including moving targets. Notably, robots equipped with RL demonstrate enhanced individual learning and decision-making abilities, enabling a level of autonomy that fosters collective swarm intelligence. In PSO, the individual behavior of the robots is more heavily influenced by the collective knowledge of the swarm. The findings highlight the effectiveness of a modular design and deep RL for advancing autonomous robotic systems in complex and unpredictable environments.

Rewards transiently and automatically enhance sustained attention.

Our ability to maintain a consistent attentional state is essential to many aspects of daily life. Still, despite our best efforts, attention naturally fluctuates between more and less vigilant states. Previous work has shown that offering performance-based rewards or incentives can help to buffer against attentional lapses. However, such work is generally focused on long timescales and, critically, does not dissociate between task-based motivation (i.e., where reward is contingent on attention performance) versus more generic motivation or arousal accounts of reward effects. Here, we investigated the influence of reward feedback on attentional vigilance during a simultaneous sustained attention and reinforcement learning (RL) task. Crucially, rewards were tied only to the RL task rather than to attentional performance. We assessed the impact of two core components of RL-reward and surprise-on short-term fluctuations in attentional vigilance. In two experiments (N = 161), we demonstrated that intermittent, attention-independent rewards transiently boosted vigilance on a timescale of seconds. We did not find consistent evidence that surprises modulated vigilance. In a third experiment (N = 135), we observed that even passively received rewards elicit transient boosts in sustained attention. Together, these findings suggest that rewards transiently buffer against attentional lapses to improve vigilance, likely through generic increases in arousal or motivation. These results point to a fundamental relationship between reward and sustained attention. (PsycInfo Database Record (c) 2025 APA, all rights reserved).

Advanced Optimization Algorithm Combining a Fuzzy Inference System for Vehicular Communications

The use of a static modulation coding scheme (MCS), such as 7, and resource keep probability (Prk) value, such as 0.8, was proven to be insufficient to achieve the best packet reception ratio (PRR) performance. Various adaptation techniques have been used in the following years. This work introduces a novel optimization algorithm approach called the fuzzy inference reinforcement learning (FIRL) sequence for adaptive parameter configuration in cellular vehicle-to-everything (C-V2X) mode-4 communication networks. This innovative method combines a Sugeno-type fuzzy inference system (FIS) control system with a Q-learning reinforcement learning algorithm to optimize the PRR as the key metric for overall network performance. The FIRL sequence generates adaptive configuration parameters for Prk and MCS index values each time the Long-Term Evolution (LTE) packet is generated. Simulation results demonstrate the effectiveness of this optimization algorithm approach, achieving up to a 169.83% improvement in performance compared to static baseline parameters.