Reinforcement Leaning Research Articles

Basal ganglia: Uniting circuit logic between matrix and striosome

A new study has identified a novel direct-indirect circuit architecture connecting the striosome compartment of the striatum with midbrain dopamine neurons. This circuit has the potential to integrate limbic and sensorimotor functions and to exert substantial control over biological reinforcement leaning.

Zero-Trust Zero-Communication Defence against Hybrid Cyberattacks in Distributed Energy Resources Using Mean Field Reinforcement Leaning

As the evolution of smart grids accelerates, distributed energy resources (DERs) emerge as key elements in the transformation of global energy systems. However, the integration of these technologies introduces significant cybersecurity vulnerabilities, notably false data injection (FDI) and a direct load-altering attack (DLAA). Traditional load-altering attacks require a huge attack load and, thus, are not practical to implement. In contrast, in modern DER environments where households become “prosumers” with high-power energy generation, the implications of such attacks are substantially amplified. This paper considers a hybrid cyberattack that includes both FDI and a DLAA, and presents a hierarchical, optimal load adjustment framework that addresses these security concerns. A centralized optimizer first calculates the ideal load-shedding strategies for each substation, which are then securely broadcast to households. To address the complexities at the individual household level, we introduce a novel reinforcement learning algorithm termed Mean Field Deep Deterministic Policy Gradients (MF-DDPG). This algorithm employs mean-field game theory to enable decentrally coordinated decision-making among each household, making it particularly effective in zero-trust scenarios. Through this multifaceted approach, we offer a robust countermeasure against load-altering attacks, thereby enhancing the resilience and stability of advanced smart grids.

A New AI Approach by Acquisition of Characteristics in Human Decision-Making Process

Planning and decision making are closely interconnected processes that often occur in tandem, influence and informing each other. Planning usually precedes decision making in the chronological sequence, and it can be viewed as a strategy to make decisions. A comprehensive planning or decision strategy can facilitate effective decisions. Thus, understanding and learning human decision-making strategies has drawn intensive attention from the AI community. For example, applying planning algorithms into reinforcement leaning (RL) can simulate the consequence of different actions and select optimal decisions based on learned models, while inverse reinforcement learning (IRL) learns a reward function and policy from expert demonstration and applies them into new scenarios. Most of these methods work based on learning human decision strategies by using modeling of a Markovian decision-making process (MDP). In this paper, we argue that the property of MDP is not fit for human decision-making processes in the real-world and it is insufficient to capture human decision strategies. To tackle this challenge, we propose a new approach to identify the characteristics of human decision-making processes as a decision map, where the decision strategy is defined by the probability distribution of human decisions that are adaptive to the dynamic changes in the environment. The proposed approach was inspired by imitation learning (IL) but with fundamental differences: (a) Instead of aiming to learn an optimal policy based on expert’s demonstrations, we aimed to estimate the distribution of decisions of any group of people. (b) Instead of modeling the environment by an MDP, we used an ambiguity probability model to consider the uncertainty of each decision. (c) The participant trajectory was obtained by categorizing each decision of a participant to a certain cluster based on the commonness in the distribution of decisions. The result shows a feasible way to capture human long-term decision dependency, which provides a complement to the existing machine learning methods for understanding and learning human decision strategies.

Digital twin-based reinforcement learning framework: application to autonomous mobile robot dispatching

ABSTRACT This paper proposes a new framework for embedding an Intelligent Digital Twin (DT) in a production system with the objective of achieving more efficient real-time production planning and control. For that purpose, the Intelligence Layer is based on Reinforcement Leaning (RL) and Deep RL (DRL) algorithms. The use of this control instead of parametric simulation-based optimization approach allows to benefit from the separation between the training and execution phase. To ensure consistency and reusability, this work presents a standardized framework, based on a formal methodology, that specifies how the various components of the DT-based RL architecture interact over time to achieve essential real-time concurrency and synchronization aspects. Experiments are conducted in a small-scale production system where material handling operations are performed by an Autonomous Mobile Robot (AMR) in an Industry 4.0 Laboratory. Results showed how synchronized state updates between the Physical and Cyber World are used within the Decision Layer to ensure real-time response for the AMR dispatching requests. Finally, to deal with continuous and high-dimensional state spaces, the Deep Q-Network is implemented. The findings of an extensive computational study reveal that implementing the DT-based DRL solution leads to improved efficiency and robustness when compared to conventional dispatching rules.

AI-Assisted Decision-Making and Risk Evaluation in Uncertain Environment Using Stochastic Inverse Reinforcement Learning: American Football as a Case Study

In this work, we focus on the development of an AI technology to support decision making for people in leadership positions while facing uncertain environments. We demonstrate an efficient approach based on a stochastic inverse reinforcement leaning (IRL) algorithm constructed by hybridizing the conventional Max-entropy IRL and mixture density network (MDN) for the prediction of transition probability. We took the case study of American football, a sports game with stochastic environment, since the number of yards gainable on the next offence in real American football is usually uncertain during strategy planning and decision making. The expert data for IRL are built using the American football 2017 season data in National Football League (NFL). The American football simulation environment was built by training MDN using the annual NFL data to generate the state transition probability for IRL. Under the framework of Max-Entropy IRL, optimal strategy was successfully obtained through a learnt reward function by trial-and-error communication with the MDN environment. To precisely evaluate the validity of the learnt policy, we have conducted a risk-return analysis and revealed that the trained IRL agent showed higher return and lower risk than the expert data, indicating that it is possible for the proposed IRL algorithm to learn superior policy than the one derived directly from the expert teaching data. Decision-making in an uncertain environment is a general issue, ranging from business operation to management. Our work presented here will likely serve as a general framework for optimal business operation and risk management and contribute especially to the portfolio’s optimization in the financial and energy trading market.

Intelligent Virtual Resource Allocation of QoS-Guaranteed Slices in B5G-Enabled VANETs for Intelligent Transportation Systems

5G communication technologies and networks help researchers and engineers look into intelligent transportation systems (ITS) with a new eye, including vehicular ad hoc networks (VANET) application. Network function virtualization (NFV) and network slicing (NS) are accepted as two most promising technologies towards the agile and elastic network architecture of 5G and beyond 5G (B5G). However, previous researchers studied NFV and NS separately. In addition, learning technologies, such as reinforcement leaning (RL), graph-based learning, emerge so as to enhance the network intelligence and resource allocation in recent years. Inspired from these, we jointly explore intelligent resource allocation issue within B5G-enabled VANETs. At first, the novel virtual resource allocation framework supporting NFV and NS for providing quality of service (QoS)-guaranteed slices is constructed. Then, we formulate the virtual resource allocation of slices as the optimization problem, having the goals of providing guaranteed QoS performance and maximizing the net profit. Considering the non convex attributes of the formulated optimization problem, we propose one intelligent and feasible algorithm instead, including the details of the proposed intelligent algorithm. We record the results in order to validate the feasibility and highlights of our proposed algorithm. For example, our intelligent algorithm has the slice acceptance advantage of 5%, comparing with the best existing work.

ULMR: An Unsupervised Learning Framework for Mismatch Removal.

Due to radiometric and geometric distortions between images, mismatches are inevitable. Thus, a mismatch removal process is required for improving matching accuracy. Although deep learning methods have been proved to outperform handcraft methods in specific scenarios, including image identification and point cloud classification, most learning methods are supervised and are susceptible to incorrect labeling, and labeling data is a time-consuming task. This paper takes advantage of deep reinforcement leaning (DRL) and proposes a framework named unsupervised learning for mismatch removal (ULMR). Resorting to DRL, ULMR firstly scores each state–action pair guided by the output of classification network; then, it calculates the policy gradient of the expected reward; finally, through maximizing the expected reward of state–action pairings, the optimal network can be obtained. Compared to supervised learning methods (e.g., NM-Net and LFGC), unsupervised learning methods (e.g., ULCM), and handcraft methods (e.g., RANSAC, GMS), ULMR can obtain higher precision, more remaining correct matches, and fewer remaining false matches in testing experiments. Moreover, ULMR shows greater stability, better accuracy, and higher quality in application experiments, demonstrating reduced sampling times and higher compatibility with other classification networks in ablation experiments, indicating its great potential for further use.

Data-based decentralized learning scheme for nonlinear systems with mismatched interconnections

In this paper, the decentralized learning scheme for nonlinear systems with mismatched interconnections is developed by using the off-policy integral reinforcement leaning algorithm. First, the decentralized control of the overall system is transformed into the optimal control of each subsystem by introducing an auxiliary control. In order to relax the knowledge of system dynamics, a model-free policy iteration algorithm is derived based on the off-policy integral reinforcement learning. Then, the model-free policy iteration algorithm is used to solve the related Hamilton–Jacobi-Bellman equations, where only the collected system data is required. For implementation purpose, neural networks are employed to approximate the optimal cost functions and the optimal control policies, respectively. Moreover, the least squares method and the experience replay technique are combined to learn neural network weights. Finally, a mismatched interconnected system and a photovoltaic power system are presented to verify the effectiveness of the proposed algorithm.

Fuzzy Reinforcement Learning for energy efficient task offloading in Vehicular Fog Computing

Vehicular Fog Computing (VFC) has been envisioned as a potential fog computing paradigm which aims to offload delay sensitive tasks to mobile fog vehicles instead of remote cloud in order to facilitate computational demands of smart villages close to rural highways. There exists challenges related to task offloading in VFC that need to be addressed. Most often, Road Side Units (RSUs) deployed along rural highways are energy constrained and they need to provide energy efficient scheduling services for the allocation of tasks to fog vehicles. On the other hand, energy consumption optimization is challenging, since scheduling decision of local processing of tasks incur computation cost while the allocation of tasks to fog vehicles incurs communication cost. Although the task offloading to VFC reduces response latency, it leads to higher RSU energy consumption contributed by the communication of task data to fog vehicles. Therefore, this paper presents an energy efficient vehicle scheduling problem for offloading of tasks to mobile fog nodes subject to satisfy constraints of task deadline and resource availability. To resolve high dimensionality issue caused by increased number of vehicles in RSU coverage, we propose an on-policy reinforcement leaning based scheduling algorithm combined with fuzzy logic based greedy heuristic, named as Fuzzy Reinforcement Learning (FRL). This greedy heuristic not only accelerates learning process, but also improves long term reward when compared to Q-learning algorithm. Extensive experiments have been performed to evaluate the proposed algorithm and the simulation results show that the proposed FRL algorithm outperforms other scheduling algorithms such as First Come First Serve (FCFS), Rate Monotonic Scheduling (RMS), Fuzzy and Distributed Task Allocation with Distributed Process (DTA_DP).

Quantum Learning-Enabled Green Communication for Next-Generation Wireless Systems

Next generation wireless systems have witnessed significant R&D attention from academia and industries to enable wide range of applications for connected environment around us. The technical design of next generation wireless systems in terms of relay and transmit power control is very critical due to the ever-reducing size of these sensor enabled systems. The growing demand of computation capability in these systems for smart decision making further diversified the significance of relay and transmit power control. Towards harnessing the benefits of Quantum Reinforcement Leaning (QRL) in the design of next generation wireless systems, this article presents a framework for joint optimal Relay and transmit Power Selection (QRL-RPS). In QRL-RPS, each sensor node learns using its present and past local state's knowledge to take optimal decision in relay and transmit power selection. Firstly, RPS problem is modelled as a Markov Decision Process (MDP), and then QRL optimization aspect of the MDP problem is formulated focusing on joint optimization of energy consumption and throughput as network utility. Secondly, a QRL-RPS algorithm is developed based on Grover's iteration to solve the MDP problem. The comparative performance evaluation attests the benefit of the proposed framework as compared to the state-of-the-art techniques.

Particle Swarm Optimization on Deep Reinforcement Learning for Detecting Social Spam Bots and Spam-Influential Users in Twitter Network

In online social networks (OSNs), detection of malicious social bots is an important research challenge to provide legitimacy of user profiles and trustworthy service ratings. Further, spam-influential users must be minimal to control the fake information-spread in OSNs. Learning from example patterns using supervised learning may not provide accurate results in cases where existing data items are biased and bot behavior dynamically changes over a period of time. Moreover, deep reinforcement learning provides improved learning by repeated interactions with the environment. However, a typical deep reinforcement leaning algorithm converges slower to find an optimal sequence of actions to reach out a goal state. In this article, we design a particle swarm optimization (PSO) based deep Q-learning algorithm for detecting social spam bots by integrating PSO with Q-value function. In addition, spam-influential users are identified using the proposed spam influence minimization model and it helps in restricting the flow of illegitimate tweets in Twitter network. Further, an influential community detection algorithm has been proposed to reduce the spreading of spam content through influential communities in Twitter network. Experimental results illustrate the efficacy of our proposed algorithms by considering two Twitter datasets and performance metrics such as precision, recall, and modularity.

Deep learning and case-based reasoning for predictive and adaptive traffic emergency management

An efficient traffic signal control system (TSCS) should not only be reactive to the current traffic but also be predictive by anticipating future traffic disturbances. In this study, we investigate the potential of using convolution neural network (CNN) in detecting emergency cases and forecasting events that can interrupt the traffic flow. Case-based reasoning (CBR) is then exploited to react to detected and forecasted events. We further develop an adapted Reinforcement Leaning (RL) algorithm in building and enhancing the case bases. The proposed system inherits the advantages of CNN, CBR, and RL, which allow detection, prediction, control, evaluation, and learning in a unified framework. To assess the proposed TSCS, we compare our approach with a set of state-of-art algorithms (e.g., multi-agent preemptive case-based reasoning algorithm and multi-agent preemptive longest queue first—maximal weight matching). The proposed TSCS outperforms the benchmarking algorithms through experiments in various traffic scenarios.

Reinforcement-Learning-Based Energy Storage System Operation Strategies to Manage Wind Power Forecast Uncertainty

Currently, renewable-energy-based power generation is rapidly developing to tackle climate change; however, the use of renewable energy is limited owing to the uncertainty related to renewable energy sources. In particular, energy storage systems (ESSs), which are critical for implementing wind power generation (WPG), entail a wide uncertainty range. Herein, a reinforcement leaning (RL)-based ESS operation strategy is investigated for managing the WPG forecast uncertainty. First, a WPG forecast uncertainty minimization problem is formulated with respect to the ESS operation, subject to ESS constraints, and then, the problem is presented as a Markov decision process (MDP) model, with the state-action space limited by the ESS characteristics. To achieve the optimal solution of the MDP model, an expected state–action–reward–state–action (SARSA) method, which is robust toward the dispersion of the system environment, is employed. Further, frequency-domain data screening based on the k-mean clustering method is implemented to improve learning performance by reducing the variance of the WPG forecast uncertainty. Extensive simulations are conducted based on practical WPG generation data and forecasting. Results indicate that the proposed clustered RL-based ESS operation strategy can manage the WPG forecast uncertainty more effectively than conventional Q-learning-based methods; additionally, the proposed method demonstrates a near-optimal performance within a 1%-point analysis gap to the optimal solution, which requires complete information, including future values.

Інтелектуальне керування орієнтацією космічних апаратів із використанням навчання з підкріпленням

Інтелектуальне керування орієнтацією космічних апаратів із використанням навчання з підкріпленням

Smart fog based workflow for traffic control networks

In this paper, we propose a novel traffic control architecture which is based on fog computing paradigm and reinforcement leaning technologies. We firstly provide an overview of this framework and detail the components and workflows designed to relieve traffic congestion. These workflows, which are connecting traffic lights, vehicles, Fog nodes and traffic cloud, aim to generate traffic light control flow and communication flow for each intersection to avoid a traffic jam. In order to make the whole city’s traffic highly efficient, the fog computing paradigm and a distributed reinforcement learning algorithm is designed to overcome communication bandwidth limitation and local optimal traffic control flow, respectively. We also demonstrate that our framework outperforms traditional systems and provides high practicability in future research for building the intelligent transportation system.

Disrupted reinforcement learning during post-error slowing in ADHD.

ADHD is associated with altered dopamine regulated reinforcement learning on prediction errors. Despite evidence of categorically altered error processing in ADHD, neuroimaging advances have largely investigated models of normal reinforcement learning in greater detail. Further, although reinforcement leaning critically relies on ventral striatum exerting error magnitude related thresholding influences on substantia nigra (SN) and dorsal striatum, these thresholding influences have never been identified with neuroimaging. To identify such thresholding influences, we propose that error magnitude related activities must first be separated from opposite activities in overlapping neural regions during error detection. Here we separate error detection from magnitude related adjustment (post-error slowing) during inhibition errors in the stop signal task in typically developing (TD) and ADHD adolescents using fMRI. In TD, we predicted that: 1) deactivation of dorsal striatum on error detection interrupts ongoing processing, and should be proportional to right frontoparietal response phase activity that has been observed in the SST; 2) deactivation of ventral striatum on post-error slowing exerts thresholding influences on, and should be proportional to activity in dorsal striatum. In ADHD, we predicted that ventral striatum would instead correlate with heightened amygdala responses to errors. We found deactivation of dorsal striatum on error detection correlated with response-phase activity in both groups. In TD, post-error slowing deactivation of ventral striatum correlated with activation of dorsal striatum. In ADHD, ventral striatum correlated with heightened amygdala activity. Further, heightened activities in locus coeruleus (norepinephrine), raphe nucleus (serotonin) and medial septal nuclei (acetylcholine), which all compete for control of DA, and are altered in ADHD, exhibited altered correlations with SN. All correlations in TD were replicated in healthy adults. Results in TD are consistent with dopamine regulated reinforcement learning on post-error slowing. In ADHD, results are consistent with heightened activities in the amygdala and non-dopaminergic neurotransmitter nuclei preventing reinforcement learning.

Deep Multi-User Reinforcement Learning for Distributed Dynamic Spectrum Access

We consider the problem of dynamic spectrum access for network utility maximization in multichannel wireless networks. The shared bandwidth is divided into $K$ orthogonal channels. In the beginning of each time slot, each user selects a channel and transmits a packet with a certain transmission probability. After each time slot, each user that has transmitted a packet receives a local observation indicating whether its packet was successfully delivered or not (i.e., ACK signal). The objective is a multi-user strategy for accessing the spectrum that maximizes a certain network utility in a distributed manner without online coordination or message exchanges between users. Obtaining an optimal solution for the spectrum access problem is computationally expensive, in general, due to the large-state space and partial observability of the states. To tackle this problem, we develop a novel distributed dynamic spectrum access algorithm based on deep multi-user reinforcement leaning. Specifically, at each time slot, each user maps its current state to the spectrum access actions based on a trained deep-Q network used to maximize the objective function. Game theoretic analysis of the system dynamics is developed for establishing design principles for the implementation of the algorithm. The experimental results demonstrate the strong performance of the algorithm.

강화학습 기반의 다단계 공급망 분배계획

강화학습 기반의 다단계 공급망 분배계획

Research on Leaning and Evolutionary Algorithm of Agent for Task Oriented

The leaning and evolutionary (L&E) algorithm of Agent for task oriented is deeply researched in this paper. Based on the relationship between tasks and the executive Agent, the importance of the research has been elaborated. Moreover, the algorithm is improved by considering the effect of environment and network structure. Reinforcement leaning and complex network have been introduced into the nonlinear genetic algorithm. Finally, some simulations of equipment acquisition tasks are made to test the validity and capability of the algorithm.

Off-policy reinforcement learning for H∞ control design.

The H∞ control design problem is considered for nonlinear systems with unknown internal system model. It is known that the nonlinear H∞ control problem can be transformed into solving the so-called Hamilton-Jacobi-Isaacs (HJI) equation, which is a nonlinear partial differential equation that is generally impossible to be solved analytically. Even worse, model-based approaches cannot be used for approximately solving HJI equation, when the accurate system model is unavailable or costly to obtain in practice. To overcome these difficulties, an off-policy reinforcement leaning (RL) method is introduced to learn the solution of HJI equation from real system data instead of mathematical system model, and its convergence is proved. In the off-policy RL method, the system data can be generated with arbitrary policies rather than the evaluating policy, which is extremely important and promising for practical systems. For implementation purpose, a neural network (NN)-based actor-critic structure is employed and a least-square NN weight update algorithm is derived based on the method of weighted residuals. Finally, the developed NN-based off-policy RL method is tested on a linear F16 aircraft plant, and further applied to a rotational/translational actuator system.