Reinforcement Learning System Research Articles

Childhood anhedonia symptoms and stressful life events predict the development of reward-related brain activity across adolescence.

The reward positivity (RewP) is an event-related potential that indexes reinforcement learning and reward system activation. The RewP has been shown to increase across adolescence; however, most studies have examined the RewP across two assessments, and no studies have examined within-person changes across adolescence into young adulthood. Moreover, the RewP has been identified as a neurobiological risk factor for adolescent-onset depression, but it is unclear whether childhood psychosocial risk factors might predict RewP development across adolescence. In a sample of 317 8- to 14-year-old girls (Mage = 12.4, SD = 1.8), the present study examined self-report measures of depression symptoms and stressful life events at baseline and the ΔRewP during the doors guessing task across three timepoints. Growth modeling indicated that, across all participants, the ΔRewP did not demonstrate linear change across adolescence. However, baseline anhedonia symptoms predicted within-person changes in the ΔRewP, such that individuals with low anhedonia symptoms demonstrated a linear increase in the ΔRewP, but individuals with high anhedonia symptoms had no change in the ΔRewP across adolescence. Similar patterns were observed for stressful life events. The present study suggests that childhood risk factors impact the development of reward-related brain activity, which might subsequently increase risk for psychopathology.

Reinforcement learning based blockchain model for revoking unauthorized access in Virtualized Network Functions‐based Internet of Things Mobile Edge Computing

AbstractVNFs boost data processing efficiency in Mobile Edge Computing (MEC)‐driven Internet of Things (IoT) for healthcare, smart cities, and industrial automation. VNF‐based IoT MEC systems encounter a significant security threat due to unauthorized access, posing risks to data privacy and system integrity. Existing approaches struggle to adapt to dynamic environments and lack tamper‐proof enforcement mechanisms. In this work, we propose a novel system combining Reinforcement Learning (RL) and blockchain technology to revoke unauthorized access in VNF‐based IoT MEC. We introduce the Integrated Action‐selection DRL Algorithm for Unauthorized Access Revocation (IASDRL‐UAR), a novel RL approach that excels in dynamic environments by handling both continuous and discrete actions, enabling real‐time optimization of security risk, execution time, and energy consumption. A behavior control contract (BCC) is proposed and integrated into the RL system, automating behavior checks and enforcement, streamlining security management, and reducing manual intervention. RL feedback plays a pivotal role in steering dynamic security adjustments, gaining valuable perspectives from user behavior via trust scores in the behavior contract. The security features of the proposed method are analyzed. Performance comparisons reveal a substantial improvement, with the proposed system outperforming existing methods by 30% in terms of throughput, 21.7% in system stability, and 26% in access revocation latency. Additionally, the system demonstrates a higher security index, energy efficiency, and scalability.

Reinforcement Learning Techniques in Optimizing Energy Systems

Reinforcement learning (RL) techniques have emerged as powerful tools for optimizing energy systems, offering the potential to enhance efficiency, reliability, and sustainability. This review paper provides a comprehensive examination of the applications of RL in the field of energy system optimization, spanning various domains such as energy management, grid control, and renewable energy integration. Beginning with an overview of RL fundamentals, the paper explores recent advancements in RL algorithms and their adaptation to address the unique challenges of energy system optimization. Case studies and real-world applications demonstrate the efficacy of RL-based approaches in improving energy efficiency, reducing costs, and mitigating environmental impacts. Furthermore, the paper discusses future directions and challenges, including scalability, interpretability, and integration with domain knowledge. By synthesizing the latest research findings and identifying key areas for further investigation, this paper aims to inform and inspire future research endeavors in the intersection of reinforcement learning and energy system optimization.

Towards Efficient Coordination in Multi-Agent Reinforcement Learning through Hybrid Information-Driven Approaches

Abstract: This paper introduces a novel framework to enhance coordination in multi-agent reinforcement learning (MARL) systems by integrating mutual information (MMI) principles with information-driven strategies. Firstly, we propose a variational approach leveraging MMI to promote coordinated behaviors among agents by regulating the cumulative return alongside the simultaneous mutual information between multi-agent actions. Through the introduction of a latent variable inducing nonzero mutual information and the application of a variational bound, a tractable lower bound is derived for the MMI-regularized objective function. This bound combines maximum entropy reinforcement learning with reducing uncertainty in other agents' actions. Subsequently, we present a practical algorithm, Variational Maximum Mutual Information Multi-Agent Actor-Critic (VM3-AC), utilizing policy iteration to maximize the derived lower bound, following a centralized learning with decentralized execution (CTDE) paradigm. Secondly, we explore the challenges of large state spaces and limited computational resources in distributed multi-agent systems, proposing a hybrid information-driven MARL approach. This approach integrates informationtheoretic models as heuristics to aid navigation in sparse state spaces, complemented by information-based rewards within an RL framework to learn higher-level policies efficiently. Our preliminary findings suggest that this hybrid approach could enhance exploration efficiency significantly, demonstrating approximately three orders of magnitude improvement over naive baseline metrics. Although still in its early stages, this work presents a promising direction for future research in achieving efficient coordination in MARL systems

How to Design Reinforcement Learning Methods for the Edge: An Integrated Approach toward Intelligent Decision Making

Extensive research has been carried out on reinforcement learning methods. The core idea of reinforcement learning is to learn methods by means of trial and error, and it has been successfully applied to robotics, autonomous driving, gaming, healthcare, resource management, and other fields. However, when building reinforcement learning solutions at the edge, not only are there the challenges of data-hungry and insufficient computational resources but also there is the difficulty of a single reinforcement learning method to meet the requirements of the model in terms of efficiency, generalization, robustness, and so on. These solutions rely on expert knowledge for the design of edge-side integrated reinforcement learning methods, and they lack high-level system architecture design to support their wider generalization and application. Therefore, in this paper, instead of surveying reinforcement learning systems, we survey the most commonly used options for each part of the architecture from the point of view of integrated application. We present the characteristics of traditional reinforcement learning in several aspects and design a corresponding integration framework based on them. In this process, we show a complete primer on the design of reinforcement learning architectures while also demonstrating the flexibility of the various parts of the architecture to be adapted to the characteristics of different edge tasks. Overall, reinforcement learning has become an important tool in intelligent decision making, but it still faces many challenges in the practical application in edge computing. The aim of this paper is to provide researchers and practitioners with a new, integrated perspective to better understand and apply reinforcement learning in edge decision-making tasks.

Toward Optimized In‐Memory Reinforcement Learning: Leveraging 1/f Noise of Synaptic Ferroelectric Field‐Effect‐Transistors for Efficient Exploration

Reinforcement learning (RL), exhibiting outstanding performance in various fields, requires large amounts of data for high performance. While exploration techniques address this requirement, conventional exploration methods have limitations: complexity of hardware implementation and significant hardware burden. Herein, in‐memory RL systems leveraging intrinsic 1/f noise of synaptic ferroelectric field‐effect‐transistors (FeFETs) for efficient exploration are proposed. The electrical characteristics of fabricated FeFETs with low‐power operation capability verify their suitability for neuromorphic systems. The proposed system achieves comparable performance to the conventional exploration method without additional circuits. The intrinsic 1/f noise of the FeFETs facilitates efficient exploration and offers significant advantages: efficiency in hardware implementation and simplicity in adjusting the 1/f noise level for optimal performance. This approach effectively addresses the challenges of conventional exploration methods. The operation mechanism of the exploration method utilizing the 1/f noise is systematically analyzed. The proposed in‐memory RL system demonstrates robustness and reliability to the device‐to‐device variation and the initial conductance distribution. This work provides further insights into the exploration methods of RL, paving the way for advanced in‐memory RL systems.

Deep Reinforcement Learning Model for Stock Portfolio Management Based on Data Fusion

Deep reinforcement learning (DRL) can be used to extract deep features that can be incorporated into reinforcement learning systems to enable improved decision-making; DRL can therefore also be used for managing stock portfolios. Traditional methods cannot fully exploit the advantages of DRL because they are generally based on real-time stock quotes, which do not have sufficient features for making comprehensive decisions. In this study, in addition to stock quotes, we introduced stock financial indices as additional stock features. Moreover, we used Markowitz mean-variance theory for determining stock correlation. A three-agent deep reinforcement learning model called Collaborative Multi-agent reinforcement learning-based stock Portfolio management System (CMPS) was designed and trained based on fused data. In CMPS, each agent was implemented with a deep Q-network to obtain the features of time-series stock data, and a self-attention network was used to combine the output of each agent. We added a risk-free asset strategy to CMPS to prevent risks and referred to this model as CMPS-Risk Free (CMPS-RF). We conducted experiments under different market conditions using the stock data of China Shanghai Stock Exchange 50 and compared our model with the state-of-the-art models. The results showed that CMPS could obtain better profits than the compared benchmark models, and CMPS-RF was able to accurately recognize the market risk and achieved the best Sharpe and Calmar ratios. The study findings are expected to aid in the development of an efficient investment-trading strategy.

Implementation of a reinforcement learning system with deep q network algorithm in the amc dash mark i game

Reinforcement learning is a branch of artificial intelligence that trains algorithms using a trial-and-error system. Reinforcement learning interacts with its environment and observes the consequences of its actions in response to rewards or punishments received. Reinforcement Learning uses information from every interaction with its environment to update its knowledge. The problem identified from this research is the lack of consistency, which is not always the same for Non-Player Characters (Agents) in the process of exploring an environment (Game environment). This research uses the Software Development Life Cycle (SDLC) Waterfall model method to train Non Player Characters (Agents) in the Amc Dash Mark I Game which uses the Deep Q Network (DQN) algorithm in several stages. Training results show improvements in model performance over time. The average duration of the episode and average reward episode showed an increase of 7.75 to 24.7, while the exploration rate decreased to 0.05. This indicates that the model has experienced learning and is improving to achieve better rewards by performing fewer actions. The lower loss also shows that the model has succeeded in reducing prediction errors and improving prediction capabilities.

Multi-granularity coverage criteria for deep reinforcement learning systems

Deep reinforcement learning (DRL) systems are progressively being deployed in safety- and security-critical domains, such as self-driving cars and unmanned aerial vehicles, raising concerns about their trustworthiness. DRL, a integration of deep learning (DL) and reinforcement learning (RL) principles, represents a machine learning technique. Existing DL coverage criteria primarily focus on neuron coverage, overlooking the unique features of RL, thus falling short in assessing the test adequacy of DRL systems.This paper introduces the first set of coverage criteria designed to systematically measure the elements of DRL systems exercised by test inputs. DRL elements, including states, actions, and policies, are leveraged to define coverage criteria that consider multi-levels and multi-granularities. Furthermore, these coverage criteria undergo optimization through the application of combinatorial coverage principles. State coverage is employed as feedback to guide test case selection for DRL systems. Empirical studies have been conducted to assess the performance of the proposed coverage criteria on five well-known DRL environments. The experiments demonstrate the effectiveness of these coverage criteria in detecting unexpected behaviors, highlighting that the proposed test case selection guided by state coverage serves as an effective strategy.

Investigating the properties of neural network representations in reinforcement learning

In this paper we investigate the properties of representations learned by deep reinforcement learning systems. Much of the early work on representations for reinforcement learning focused on designing fixed-basis architectures to achieve properties thought to be desirable, such as orthogonality and sparsity. In contrast, the idea behind deep reinforcement learning methods is that the agent designer should not encode representational properties, but rather that the data stream should determine the properties of the representation—good representations emerge under appropriate training schemes. In this paper we bring these two perspectives together, empirically investigating the properties of representations that support transfer in reinforcement learning. We introduce and measure six representational properties over more than 25,000 agent-task settings. We consider Deep Q-learning agents with different auxiliary losses in a pixel-based navigation environment, with source and transfer tasks corresponding to different goal locations. We develop a method to better understand why some representations work better for transfer, through a systematic approach varying task similarity and measuring and correlating representation properties with transfer performance. We demonstrate the generality of the methodology by investigating representations learned by a Rainbow agent that successfully transfers across Atari 2600 game modes.

Blockchain and Federated Reinforcement Learning for Vehicle-to-Everything Energy Trading in Smart Grids

The proliferation of electric vehicles (EVs) and the advancement of Vehicle-to-Everything (V2X) energy trading systems are expected to play a crucial role in alleviating the stress on the electric grid during peak hours. However, the wide adoption of these paradigms requires intelligent mechanisms that protect the security and privacy of EV users. This paper proposes a novel Federated Reinforcement Learning (FRL) system combined with blockchain technology to maximize EV users' utility while preserving the security and privacy of trading transactions. Furthermore, we develop the concept of Proof of State of Charge (PoSOC) as a consensus mechanism to determine the winning EVs and reward them as block miners in the blockchain. The proposed system is validated through comprehensive simulation experiments utilizing a real-world dataset. The model is implemented on the Avalanche blockchain platform to demonstrate its real-world feasibility. The test results show that the proposed scheme improves EV users' utility significantly compared to the existing studies. The obtained simulation results indicate the effectiveness and robustness of the proposed system.

Coverage-guided fuzzing for deep reinforcement learning systems

While the past decade has witnessed a growing demand for employing deep reinforcement learning (DRL) in various domains to solve real-world problems, the reliability of DRL systems has become more of a concern. In particular, DRL agents are often trained on data from a potentially biased distribution over environmental settings, causing the trained agents to fail in certain cases despite high average-case performance. Hence, it is necessary and urgent to adequately test DRL agents to ensure the reliability of practical DRL systems. However, due to the fundamental difference in the programming paradigm and the development process, traditional software testing methodology cannot be applied directly to DRL systems. Given that, we introduce a novel testing framework for DRL systems, aiming to generate diverse test cases that can drive a DRL system to fail. Specifically, we design, implement and evaluate DRLFuzz, which is a coverage-guided fuzzing (CGF) framework for systematically testing DRL systems. Experimental results demonstrate that DRLFuzz can efficiently discover diverse failures in different DRL systems for various benchmark tasks. Compared with a random search baseline, DRLFuzz can generate 60% more failed cases in general. Additionally, the diversity of failed cases generated by DRLFuzz is increased by 4.6%∼14.1% in terms of mean pairwise distance (MPD). Furthermore, our experiments also indicate that the failed cases generated by DRLFuzz can be utilized to fine-tune the DRL agent to eliminate the failures resulting from inadequate exploration during training and thus improve the reliability of DRL systems.

A survey of multi-agent deep reinforcement learning with communication

Communication is an effective mechanism for coordinating the behaviors of multiple agents, broadening their views of the environment, and to support their collaborations. In the field of multi-agent deep reinforcement learning (MADRL), agents can improve the overall learning performance and achieve their objectives by communication. Agents can communicate various types of messages, either to all agents or to specific agent groups, or conditioned on specific constraints. With the growing body of research work in MADRL with communication (Comm-MADRL), there is a lack of a systematic and structural approach to distinguish and classify existing Comm-MADRL approaches. In this paper, we survey recent works in the Comm-MADRL field and consider various aspects of communication that can play a role in designing and developing multi-agent reinforcement learning systems. With these aspects in mind, we propose 9 dimensions along which Comm-MADRL approaches can be analyzed, developed, and compared. By projecting existing works into the multi-dimensional space, we discover interesting trends. We also propose some novel directions for designing future Comm-MADRL systems through exploring possible combinations of the dimensions.

Suppression of negative tweets using reinforcement learning systems

In recent years, damage caused by negative tweets has become a social problem. In this paper, we consider a method of suppressing negative tweets by using reinforcement learning. In particular, we consider the case where tweet writing is modeled as a multi-agent environment. Numerical experiments verify the effects of suppression using various reinforcement learning methods. We will also verify robustness to environmental changes. We compared the results of Profit Sharing (PS) and Q-learning (QL) as reinforcement learning methods to confirm the effectiveness of PS, and confirmed the behavior of the rationality theorem in a multi-agent environment. Furthermore, in experiments regarding the ability to follow environmental changes, it was confirmed that PS is more robust than QL. If machines can appropriately intervene and interact with posts made by humans, we can expect that negative tweets and even blow-ups can be suppressed automatically without the need for costly human eye monitoring.

Deep reinforcement learning with light-weight vision model for sequential robotic object sorting

Training an agent to generate cooperative joint learning policies for executing object sorting in cluttered scenes is complex. This paper presents a comparative model-free deep reinforcement learning (DRL) system built from variants of end-to-end lightweight deep neural networks that jointly complement three primitive policies (pushing, grasping, and placing) that operate from visual observations to actions. The end goal of the proposed DRL systems is to perform object sorting in very dynamic and complex environmental scenarios of cluttered and random objects (irregular and regular blocks) existing in different color variations. For this, we investigated DRL methods built from 12 custom instances of Pixel-wise Q-valued Critic Networks (PQCN) from four main backbone networks (DenseNet121, DenseNet169, MobileNetV3, and SqueezeNet) individually combined with custom fully convolutional neural networks (FCN) for learning the image-based observation space through affordance mapping paradigm while considering both dual and single transfer learning schemes. Additionally, we explored three categories of gradient-based optimization methods and considered custom reward functions with varying discount factors. The results show that the PQCN constructed by integrating DenseNet121 or MobileNetV3 with FCN and trained by a dual transfer learning scheme (with a discount factor of γ=0.70) yielded the best performance in most categories of our evaluation in the training phases. Overall, the PQCN-DenseNet121, when trained with a dual transfer learning process, yielded the best generalization (sorting success rate) in most of the evaluation metrics across all object categories in the testing phase. Furthermore, our study provides a new benchmark for validating deep reinforcement learning algorithms while sorting object blocks under varying degrees of complexity. This research finds practical relevance in industrial fields such as manufacturing and packaging, building construction, plant sorting, automobile assembly lines, and waste sorting. Here is the article’s source code: https://github.com/Emmanuel-Okafor/DRL_for_Object_Sorting.git.

The neurocognitive role of working memory load when Pavlovian motivational control affects instrumental learning.

Research suggests that a fast, capacity-limited working memory (WM) system and a slow, incremental reinforcement learning (RL) system jointly contribute to instrumental learning. Thus, situations that strain WM resources alter instrumental learning: under WM loads, learning becomes slow and incremental, the reliance on computationally efficient learning increases, and action selection becomes more random. It is also suggested that Pavlovian learning influences people's behavior during instrumental learning by providing hard-wired instinctive responses including approach to reward predictors and avoidance of punishment predictors. However, it remains unknown how constraints on WM resources affect instrumental learning under Pavlovian influence. Thus, we conducted a functional magnetic resonance imaging (fMRI) study (N = 49) in which participants completed an instrumental learning task with Pavlovian-instrumental conflict (the orthogonalized go/no-go task) both with and without extra WM load. Behavioral and computational modeling analyses revealed that WM load reduced the learning rate and increased random choice, without affecting Pavlovian bias. Model-based fMRI analysis revealed that WM load strengthened RPE signaling in the striatum. Moreover, under WM load, the striatum showed weakened connectivity with the ventromedial and dorsolateral prefrontal cortex when computing reward expectations. These results suggest that the limitation of cognitive resources by WM load promotes slow and incremental learning through the weakened cooperation between WM and RL; such limitation also makes action selection more random, but it does not directly affect the balance between instrumental and Pavlovian systems.

Integration of deep reinforcement learning and multi-agent system for dynamic scheduling of re-entrant hybrid flow shop considering worker fatigue and skill levels

In real-life manufacturing systems, production management is often affected by urgent demands and unexpected interruptions, such as new job insertions, machine breakdowns and operator unavailability. In this context, agent-based techniques are useful and able to respond quickly to dynamic disturbances. The ability of agents to recognize their environment and make decisions can be further enhanced by deep reinforcement learning (DRL). This paper investigates a novel dynamic re-entrant hybrid flow shop scheduling problem (DRHFSP) considering worker fatigue and skill levels to minimize the total tardiness of all production tasks. An integrated architecture of DRL and MAS (DRL-MAS) is proposed for real-time scheduling in dynamic environments. Two DRL models are proposed for different sub-decisions, where a reward-shaping technique combining long-term and short-term returns is proposed for the job sequence and machine selection sub-decisions, and an attention-based network is proposed for the worker assignment sub-decision for efficient feature extraction and decision making. Numerical experiments and case studies demonstrate the superior performance of the proposed DRL models compared with existing scheduling strategies.

A deep reinforcement learning system for the allocation of epidemic prevention materials based on DDPG

COVID-19 has spread rapidly around the world since the end of 2019. Consequently, the demand for epidemic prevention materials (e.g., medical-grade masks) has increased drastically. The medical-grade masks have become a necessary item for everyone. Referring to the shortage of medical-grade masks in Taiwan, the government collected and managed them at the early stage and sold them for a fixed price. However, the government must distribute masks around the country to prevent the movement of people who can cause infection with COVID-19. Moreover, the demands for medical supplies during the pandemic have become more complex and dynamic. This study proposes a robust system for allocating epidemic prevention materials. The proposed approach adopts the reinforcement learning framework, which takes the daily supply and demand for masks as the environment, with the Deep Deterministic Policy Gradient (DDPG) algorithm for agent updates and the daily shortage as rewards and punishments. The proposed system is compared with the traditional machine learning approach used for supply chain demand forecasting through experiments. The results indicate that our proposed method is superior regarding the number of pharmacies with over-stocked masks and rewards. Moreover, our proposed system performs consistently under different total numbers of masks.

Explainable reinforcement learning (XRL): a systematic literature review and taxonomy

In recent years, reinforcement learning (RL) systems have shown impressive performance and remarkable achievements. Many achievements can be attributed to combining RL with deep learning. However, those systems lack explainability, which refers to our understanding of the system’s decision-making process. In response to this challenge, the new explainable RL (XRL) field has emerged and grown rapidly to help us understand RL systems. This systematic literature review aims to give a unified view of the field by reviewing ten existing XRL literature reviews and 189 XRL studies from the past five years. Furthermore, we seek to organize these studies into a new taxonomy, discuss each area in detail, and draw connections between methods and stakeholder questions (e.g., “how can I get the agent to do _?”). Finally, we look at the research trends in XRL, recommend XRL methods, and present some exciting research directions for future research. We hope stakeholders, such as RL researchers and practitioners, will utilize this literature review as a comprehensive resource to overview existing state-of-the-art XRL methods. Additionally, we strive to help find research gaps and quickly identify methods that answer stakeholder questions.

Distinct value computations support rapid sequential decisions

The value of the environment determines animals’ motivational states and sets expectations for error-based learning1–3. How are values computed? Reinforcement learning systems can store or cache values of states or actions that are learned from experience, or they can compute values using a model of the environment to simulate possible futures3. These value computations have distinct trade-offs, and a central question is how neural systems decide which computations to use or whether/how to combine them4–8. Here we show that rats use distinct value computations for sequential decisions within single trials. We used high-throughput training to collect statistically powerful datasets from 291 rats performing a temporal wagering task with hidden reward states. Rats adjusted how quickly they initiated trials and how long they waited for rewards across states, balancing effort and time costs against expected rewards. Statistical modeling revealed that animals computed the value of the environment differently when initiating trials versus when deciding how long to wait for rewards, even though these decisions were only seconds apart. Moreover, value estimates interacted via a dynamic learning rate. Our results reveal how distinct value computations interact on rapid timescales, and demonstrate the power of using high-throughput training to understand rich, cognitive behaviors.