Reinforcement Learning Research Articles

Dissociating social reward learning and behavior in alcohol use disorder

BackgroundAlcohol use disorder (AUD) is associated with deficits in social cognition and behavior, but why these deficits are acquired is unknown. We hypothesized that a reduced association between actions and outcomes for others, i.e., social reward learning, would explain prevalent social deficiencies in AUD.MethodsWe conducted one laboratory study (n = 234) and one confirmatory online study (n = 258), comparing young adults with AUD to age-, gender-, and education-matched healthy controls on a standardized reward learning task. In the task, participants learned to maximize reward for another person and for oneself. To elucidate the potential relation between reward learning and social behavior in AUD, we administered two measures: a dictator game task and a self-report measure. Finally, we applied reinforcement learning models to examine the computational properties of learning.ResultsSocial and individual learning, expressed in choice behavior, were comparable in individuals with AUD and healthy controls. Individual differences in learning were not associated with reduced social behavior in AUD. Computational modeling suggested that the learning mechanisms are comparable in AUD and healthy controls and indifferent to whether learning maximizes reward for another person or oneself.ConclusionsIndividuals with AUD demonstrated preserved reward learning abilities that do not vary with social behavior. Together, these results indicate that reward processes may not be relevant for understanding compromised social behavior in AUD.

Optimization Strategies in Consumer Choice Behavior for Personalized Recommendation Systems Based on Deep Reinforcement Learning

In domains such as e-commerce and media recommendations, personalized recommendation systems effectively alleviate the issue of information overload. However, existing systems still face challenges in multimodal data processing, data sparsity, and dynamic changes in user preferences. This paper proposes a Hierarchical Generative Reinforcement Learning Recommendation Optimization framework (HG-RLRO) that addresses these issues by integrating multimodal data, Generative Adversarial Networks (GAN), Inverse Reinforcement Learning (IRL), and Hierarchical Temporal Difference Learning (HTD). HG-RLRO employs a multi-agent architecture to handle textual and image data and utilizes GAN to generate simulated user behavior data to mitigate data sparsity. IRL dynamically infers user preferences across multiple time scales.

Distributed representations of temporally accumulated reward prediction errors in the mouse cortex.

Reward prediction errors (RPEs) quantify the difference between expected and actual rewards, serving to refine future actions. Although reinforcement learning (RL) provides ample theoretical evidence suggesting that the long-term accumulation of these error signals improves learning efficiency, it remains unclear whether the brain uses similar mechanisms. To explore this, we constructed RL-based theoretical models and used multiregional two-photon calcium imaging in the mouse dorsal cortex. We identified a population of neurons whose activity was modulated by varying degrees of RPE accumulation. Consequently, RPE-encoding neurons were sequentially activated within each trial, forming a distributed assembly. RPE representations in mice aligned with theoretical predictions of RL, emerging during learning and being subject to manipulations of the reward function. Interareal comparisons revealed a region-specific code, with higher-order cortical regions exhibiting long-term encoding of RPE accumulation. These results present an additional layer of complexity in cortical RPE computation, potentially augmenting learning efficiency in animals.

Controlling a Mecanum-Wheeled Robot with Multiple Swivel Axes Controlled by Three Commands

The Mecanum-wheeled robot has four special wheels. It can control four wheels independently and has seven turning axes. The robot can translate in all directions and travel in curves without changing its direction by means of the control commands for turning ratio, speed, and direction of travel. However, no model has been proposed that can accurately simulate the output of the actual machine for the three types of inputs, even when the characteristics of the motor and motor driver are unknown. In this study, we synthesized and simplified transfer functions and estimated the undetermined coefficients that minimize the sum of squared errors to construct a model of the robot that can output the position and posture equivalent to those of the actual robot for the input commands for turning ratio, speed, and the direction of travel. We modeled a Mecanum-wheeled robot using the proposed modeling method and parameter determination method and compared the outputs of the real robot to the step and ramp inputs. The results showed that the errors between the two outputs were very small and accurate enough to simulate AI learning, such as reinforcement learning, using the model of the robot.

Adaptive weight adjustment technology for reinforcement learning reward indicators based on the new-type power system

Current RL-based scheduling methods struggle with designing effective reward functions that balance cost reduction, safety and renewable energy integration. To tackle these issues, this paper proposes a novel hybrid optimization framework that combines Particle Swarm Optimization and Reinforcement Learning (PSO-RL), focusing on three key aspects: (1) Minimizing Generation Costs: Systematically reducing operational costs for economic efficiency. (2) Enhancing System Safety: Improving reliability by incorporating operational constraints. (3) Increasing Renewable Energy Proportion: Emphasizing the integration of renewable energy sources. The comprehensive mathematical model encapsulates intricate constraints and multi-objective characteristics of the new-type power system, featuring reward functions tailored for dynamic responsiveness. Empirical validations using the Grid2Op scheduler affirm the efficacy of our framework. Results highlight superior convergence speed and optimization precision of the PSO-RL approach over traditional RL methods. In summary, this study advances scheduling methodologies for the new-type power system, fostering renewable integration and smart grid technology evolution.

Reinforcement learning based route optimization model to enhance energy efficiency in internet of vehicles

The Internet of Vehicles (IoV) transforms the automobile industry through connected vehicles with communication infrastructure that improves traffic control, safety and information, and entertainment services. However, some issues remain, like data protection, privacy, compatibility with other protocols and systems, and the availability of stable and continuous connections. Specific problems are related to energy consumption for transmitting information, distributing energy loads across the vehicle’s sensors and communication units, and designing energy-efficient approaches to processing received data and making decisions in the context of the IoV environment. In the realm of IoV, we propose OptiE2ERL, an advanced Reinforcement Learning (RL) based model designed to optimize energy efficiency and routing. Our model leverages a reward matrix and the Bellman equation to determine the optimal path from source to destination, effectively managing communication overhead. The model considers critical parameters such as Remaining Energy Level (REL), Bandwidth and Interference Level (BIL), Mobility Pattern (MP), Traffic Condition (TC), and Network Topological Arrangement (NTA), ensuring a comprehensive approach to route optimization. Extensive simulations were conducted using NS2 and Python, demonstrating that OptiE2ERL significantly outperforms existing models like LEACH, PEGASIS, and EER-RL across various performance metrics. Specifically, our model extends the network lifetime, delays the occurrence of the first dead node, and maintains a higher residual energy rate. Furthermore, OptiE2ERL enhances network scalability and robustness, making it a superior choice for IoV applications. The simulation results highlight the effectiveness of our model in achieving energy-efficient routing while maintaining network performance under different scenarios. By incorporating a diverse set of parameters and utilizing RL techniques, OptiE2ERL provides a robust solution for the challenges faced in IoV networks. This research contributes to the field by presenting a model that optimizes energy consumption and ensures reliable and efficient communication in dynamic vehicular environments.

Development of deep reinforcement learning-based fault diagnosis method for actuator faults in unmanned aerial vehicles

Abstract Actuator faults in unmanned aerial vehicles (UAVs) can have significant and potentially adverse effects on their safety and performance, highlighting the critical importance of fault diagnosis in UAV design. Ensuring the reliability of these systems in various applications often requires the use of advanced diagnostic algorithms. Artificial intelligence methods, such as deep learning and machine learning techniques, enable fault diagnosis through sample-based learning without the need for prior knowledge of fault mechanisms or physics-based models. However, UAV fault datasets are typically small due to stringent safety standards, which presents challenges for achieving high-performance fault diagnosis. To address this, deep reinforcement learning (DRL) algorithms offer a unique advantage by combining deep learning’s automatic feature extraction with reinforcement learning’s interactive learning approach, improving both learning capabilities and robustness. In this study, we propose and evaluate two DRL-based fault diagnosis models, which demonstrate remarkable accuracy in fault diagnosis, consistently exceeding $99{\rm{\% }}$ . Notably, under small sample scenarios, the proposed models significantly outperform traditional classifiers such as decision trees, support vector machines, and multilayer perceptron neural networks. These findings suggest that the integration of DRL enhances fault diagnosis performance, particularly in data-limited environments.

A Systematic Review on Reinforcement Learning for Industrial Combinatorial Optimization Problems

This paper presents a systematic review on reinforcement learning approaches for combinatorial optimization problems based on real-world industrial applications. While this topic is increasing in popularity, explicit implementation details are not always available in the literature. The main objective of this paper is characterizing the agent–environment interactions, namely, the state space representation, action space mapping and reward design. Also, the main limitations for practical implementation and the needed future developments are identified. The literature selected covers a wide range of industrial combinatorial optimization problems, found in the IEEE Xplore, Scopus and Web of Science databases. A total of 715 unique papers were extracted from the query. Then, out-of-scope applications, reviews, surveys and papers with insufficient implementation details were removed. This resulted in a total of 298 papers that align with the focus of the review with sufficient implementation details. The state space representation shows the most variety, while the reward design is based on combinations of different modules. The presented studies use a large variety of features and strategies. However, one of the main limitations is that even with state-of-the-art complex models the scalability issues of increasing problem complexity cannot be fully solved. No methods were used to assess risk of biases or automatically synthesize the results.

AI-Enhanced Digital Twins for Energy Efficiency and Carbon Footprint Reduction in Smart City Infrastructure

The combination of Artificial Intelligence (AI) and Digital Twin technology provides disruptive potential to increase energy efficiency and carbon footprint in smart city infrastructure. Digital Twins virtual copies of the real-world systems are augmented by AI algorithms that enable continuous monitoring, predictive analysis and optimization. In this paper, we explore the use of AI-based Digital Twins on smart buildings, transport networks and smart grids to save significant amounts of energy and drive sustainability. This is done through machine learning and reinforcement learning algorithms which identify patterns of energy use with high precision and helps to reduce the energy usage in smart buildings by 25-30%. For transportation, AI-enabled traffic infrastructure reduced carbon emissions by 20% and enhanced EV infrastructure efficiency by 18%. The smart grids were better served by predictive energy distribution, which allowed for a 15% decrease in losses and a 20% rise in the use of renewable energy. All of these results point towards the potential of AI-augmented Digital Twins to reshape city planning, optimise resource consumption and play a key role in achieving global sustainability targets. This research underscores the need to embrace high-tech solutions for the next smart city projects to combat climate change and promote sustainable development.

Improving Quadrotor Tracking Performance in Wind Disturbances Using Hybrid Reinforcement Learning Control

The control performance of model-based control approaches depends on the accuracy of the system dynamics. However, due to unknown environmental disturbances, quadrotor dynamics are time-varying during real-world flight, and real-time updating of the dynamics is required to avoid degradation of control performance due to model errors. To address this challenging issue, this paper proposes a hybrid controller that combines model predictive control (MPC) with model-based reinforcement learning (MBRL) to improve quadrotor trajectory tracking accuracy in unknown wind conditions. The MPC controller uses simple dynamics identified from offline flight data. The MBRL controller learns MPC-augmented dynamics represented by the Gaussian process (GP) and the residual policy from online flight data. The RL controller outputs additional actions to compensate for MPC control errors, while MPC can ensure the safety of RL exploration. The computational cost of GP when dealing with a large dataset affects the real-time performance. We design a priority data selection criterion to maintain a small dataset, balancing model accuracy and computational time. The accurate tracking results for different reference trajectories under various wind conditions demonstrate the robustness of the proposed approach. Comparative results with nominal MPC and related baselines show the advantages of the proposed approach in terms of control and real-time performance.

Target Search and Navigation in Heterogeneous Robot Systems with Deep Reinforcement Learning

Target Search and Navigation in Heterogeneous Robot Systems with Deep Reinforcement Learning

Research on the Decision-Making Model of Carbon Quota Trading Based on Deep Reinforcement Learning

This study addresses the complexities of carbon quota trading markets amidst global warming concerns, proposing a deep reinforcement learning (DRL)-based decision-making model to enhance trading strategies. Acknowledging the limitations of conventional methods in navigating volatile carbon prices, policy shifts, and informational disparities, the research integrates DRL's advanced capabilities. It commences with an overview of DRL principles and its successful applications, followed by an analysis of market dynamics and trading nuances. A DRL model is then formulated, delineating state-action spaces and a tailored reward function for optimized learning within the carbon trading context. Model refinement involves hyperparameter tuning for superior performance. The summary concludes with an evaluation of the model's efficacy, highlighting its adaptability and computational demands, while outlining avenues for further enhancement and real-world implementation to combat climate change through improved carbon market operations.

Optimising multi-element cut-off grades for a strategic production plan under geological uncertainty

ABSTRACT Mines operate in uncertain heterogenous environments, in which the most profitable material must be delineated from lower value-material. Traditionally, single-element cut-off grades provide this delineation and methods for optimising date back to the 1960s. Recent advancements have included geological uncertainty, multiple elements, and production scheduling; however, no method combines all three improvements. Two multi-element cut-off grade definitions are proposed herein, along with a reinforcement learning framework for optimising long-term multi-element cut-off grades under geological uncertainty for an optimal production plan. The method is applied to a gold-copper mining complex, and the performance of both cut-off grade definitions is compared.

Intelligent energy pairing scheduler (InEPS) for heterogeneous HPC clusters

In recent years, energy consumption has become a limiting factor in the evolution of high-performance computing (HPC) clusters in terms of environmental concern and maintenance cost. The computing power of these clusters is increasing, together with the demands of the workloads they execute. A key component in HPC systems is the workload manager, whose operation has a substantial impact on the performance and energy consumption of the clusters. Recent research has employed machine learning techniques to optimise the operation of this component. However, these attempts have focused on homogeneous clusters where all the cores are pooled together and considered equal, disregarding the fact that they are contained in nodes and that they can have different performances. This work presents an intelligent job scheduler based on deep reinforcement learning that focuses on reducing energy consumption of heterogeneous HPC clusters. To this aim it leverages information provided by the users as well as the power consumption specifications of the compute resources of the cluster. The scheduler is evaluated against a set of heuristic algorithms showing that it has potential to give similar results, even in the face of the extra complexity of the heterogeneous cluster.

Emerging Practices in Risk-Based Maintenance Management Driven by Industrial Transitions: Multi-Case Studies and Reflections

Risk-based maintenance is a well-known maintenance concept in the oil and gas sector, especially in the Norwegian continental shelf and has been supported by the NORSOK Z-008 standard since 2011. Green transition and Industry 4.0 with its digitization and technology transformations have enforced the risk-based maintenance program to be dynamic and emerging. Thus, the purpose of this paper is to identify the impact of Industry 4.0 on risk-based maintenance and define the new emerging version of risk-based maintenance. The study is based on seven case studies that have been conducted between 2017 and 2024 within the Norwegian continental shelf. It is found that Industry 4.0 impacts the entire management loop of risk-based maintenance, it enforces more toward the project phase and built-in features, it enhances the data collection and their fitness to purpose, and it enables reinforcement learning for the entire maintenance management program. On the other hand, it creates a new data flow and increases the system’s complexity, uncertainty, and associated risks.

Towards Robust Routing: Enabling Long-Range Perception with the Power of Graph Transformers and Deep Reinforcement Learning in Software-Defined Networks

Deep Reinforcement Learning (DRL) has demonstrated promising capabilities for routing optimization in Software-Defined Networks (SDNs). However, existing DRL-based routing algorithms are struggling to extract graph-structured information and constrained to a fixed topology, suffering from the lack of robustness. In this paper, we strengthen the advantages of Graph Neural Networks (GNNs) for DRL-based routing optimization and propose a novel algorithm named Graph Transformer Star Routing (GTSR) to enhance robustness against topology changes. GTSR utilizes the multi-agent architecture to enable each node to make routing decisions independently, and introduces a Graph Transformer to equip agents with the capabilities of handling topology changes. Furthermore, we carefully design a global message-passing mechanism with a virtual star node and a path-based readout method, enhancing the long-range perception of traffic and the detection of potential congestion for routing decision-making. Moreover, we construct a multi-agent cooperation mechanism to facilitate the learning of universal perceptual strategies and reduce the amount of computation. Extensive experiments on multiple real-world network topologies demonstrate that GTSR is capable of adapting to unseen topology changes without retraining and decreases end-to-end latency by at least 47% and packet loss rate by at least 10% compared to all baselines, highlighting strong robustness.

Deep Reinforcement Learning for automated scheduling of mining earthwork equipment with spatio-temporal safety constraints

Deep Reinforcement Learning for automated scheduling of mining earthwork equipment with spatio-temporal safety constraints

Unlocking robotic perception: comparison of deep learning methods for simultaneous localization and mapping and visual simultaneous localization and mapping in robot

Abstract Simultaneous Localization and Mapping (SLAM) and Visual SLAM are crucial technologies in robotics, allowing autonomous systems to navigate and comprehend their environment. Deep learning (DL) has become a powerful tool in driving progress in these areas, providing solutions that improve accuracy, efficiency, and resilience. This article thoroughly analyzes different deep learning techniques designed explicitly for SLAM and Visual SLAM applications in robotic systems. This work provides a detailed overview of DL roles in SLAM and VSLAM and emphasizes the differences between these two fields. Five powerful DL methods are investigated: Convolutional Neural Networks in extracting features and understanding meaning, Recurrent Neural Network in modeling temporal relationships, Deep Reinforcement Learning in developing exploration strategies, Graph Neural Network in modeling spatial relationships, and Attention Mechanisms in selectively processing information. In this research, we will examine the advantages and disadvantages of each approach in relation to robotic applications, taking into account issues such as real-time performance, resource restrictions, and adaptability to various situations. This article seeks to guide researchers and practitioners in selecting suitable deep learning algorithms to improve the capabilities of SLAM and Visual SLAM in robotic systems by combining ideas from recent research and actual implementations. The popular types of each concerned DL will be synthesized with the discussion of pros and cons.

Research on optimization of power system load response strategy and cost–benefit analysis based on electricity price algorithm model

Abstract This article proposes an innovative framework that amalgamates deep reinforcement learning (DRL) with cost–benefit analysis (CBA). The enhanced actor–critic DRL algorithm simultaneously addresses short-term price fluctuations and long-term system benefits, facilitating optimization across multiple time scales. Furthermore, it establishes a dynamic, multidimensional CBA model that encompasses a comprehensive evaluation of economic, social, and technological benefits, employing a fuzzy comprehensive evaluation method for quantitative analysis. The integration of DRL and CBA forms a closed-loop system that continuously refines strategy optimization through real-time adjustments of the reward function and evaluation weights. Experimental results validate the efficacy of this approach.

Reinforcement Learning is Impaired in the Sub-acute Post-stroke Period.

In humans, most spontaneous recovery from motor impairment after stroke occurs in the first 3 months. Studies in animal models show higher responsiveness to training over a similar time-period. Both phenomena are often attributed to a milieu of heightened plasticity, which may share some mechanistic overlap with plasticity associated with normal motor learning. Given that neurorehabilitation approaches are frequently predicated on motor learning principles, here we asked if the sensitivity of trial-to-trial learning for 2 kinds of motor learning processes often involved during rehabilitation is also enhanced early post-stroke. In a cross-sectional design, we compared (1) reinforcement and (2) error-based learning in 2 groups: 1 tested within 3 months after stroke (early group, N = 35) another tested more than 6 months after stroke (late group, N = 30). These 2 forms of motor learning were assessed with variations of the same visuomotor rotation task. Critically, motor execution was matched between the 2 groups. Reinforcement learning was impaired in the early but not the late group, whereas error-based learning was unimpaired in either group. These findings could not be attributed to differences in baseline execution, cognitive impairment, gender, age, or lesion volume and location. The presence of a deficit in reinforcement motor learning in the first 3 months after stroke has important implications for rehabilitation. It might be necessary to either increase reinforcement feedback given early after stroke, increase the dose of rehabilitation to compensate, or delay onset of rehabilitation approaches that may rely on reinforcement, for example, constraint-induced movement therapy, and instead emphasize other forms of motor training in the subacute time period.