Adaptive Reinforcement Learning Research Articles

Evaluating crisis perturbations on urban mobility using adaptive reinforcement learning

• An adaptive reinforcement learning model is developed to learn human travel behaviors in regular situations. • The model is capable of simulating the traffic conditions and evaluate the perturbations of disasters on urban mobility. • A case study of the 2017 Hurricane Harvey in Houston is conducted to examine the application of the model. • The model and outcomes of applications can inform the public and decision-makers about the response strategies and resilience planning to reduce the impacts of crises on urban mobility. The objective of this study is to propose and test an adaptive reinforcement learning model that can learn the patterns of human mobility in a normal context and simulate the mobility during perturbations caused by crises, such as flooding, wildfire, and hurricanes. Understanding and evaluating human mobility patterns, such as destination and trajectory selection, can inform emerging congestion and road closures raised by disruptions in emergencies. Data related to human movement trajectories are scarce, especially in the context of emergencies, which places a limitation on applications of existing urban mobility models learned from empirical data. Models with the capability of learning the mobility patterns from data generated in normal situations and which can adapt to emergency situations are needed to inform emergency response and urban resilience assessments. To address this gap, this study creates and tests an adaptive reinforcement learning model that can predict the destinations of movements, estimate the trajectory for each origin and destination pair, and examine the impact of perturbations on humans’ decisions related to destinations and movement trajectories. Employing millions of trajectory data from INRIX, the application of the proposed model is shown in the context of Houston and the flooding scenario caused by Hurricane Harvey in August 2017. The results show that the model can achieve more than 76% precision and recall at the model learning stage. The mean percentage error of the travel distance in predicted trajectories is 4.29%, compared to the travel distances in empirical data. In addition, predicted density of vehicles in grid cells are negatively associated with the traffic speed on road segments and inundation intensity in grid cells during the flooding. The results from the simulation further show that the model could predict traffic patterns and congestion resulting from urban flooding. The outcomes of the analysis demonstrate the capabilities of the model for analyzing urban mobility during crises, which can inform the public and decision-makers about the response strategies and resilience planning to reduce the impacts of crises on urban mobility.

Deep reinforcement learning for building honeypots against runtime DoS attack

Honeypot is a network environment utilized to protect proper network sources from attacks. Honeypot makes an environment that attracts the attacker to pay their operations to steal sources. Denial of Service (DoS) attacks are efficiently noticed using the proposed honeypot method. The issues of the previous technique are that the DoS attack is a malicious act with the goal of interrupting the access to a computer network. The result of the DoS attack can cause the computers on the network to squander their resources to serve illegitimate requests that result in a disruption of the network's services to legitimate users. To overcome these challenges this method is proposed. In this manuscript, the Deep Adaptive Reinforcement Learning for Honeypots (DARLH) is proposed. Here, honeypot environment, the proposed DARLHs system implements Deep Adaptive Reinforcement Learning (DARL) with Intrusion Detection System (IDS) agents and Deep Recurrent Neural Network (DRNN) with IDS agent for observing multiruntime DoS attack. In the next level, the system creates DRNN and DARL IDS agent integration modules for effective runtime attack detections. The Knowledge Data Discovery data set pattern, UNSW-NB20, and Bot-IoT data sets are used to the scenario of DoS attack. The method is executed in Python 3.7. The experimental outcomes are likened through different existing methods, such as Game and Naïve-Bayes Honeypot, Block Chain Honeypot, and Recurrent Neural Network-based Signature Generation and Detection. The proposed method is compared with External DoS Attack, Internal DoS attack, Brute-force attack, DoS attack, Web attack, and Botnet attacks with the existing methods. From the comparison, the proposed method offers 5%–10% better outcomes than another existing method. Lastly, the test results determine that the proposed method performance is most efficient with another existing system.

Adaptive Reinforcement Learning Framework for NOMA-UAV Networks

We propose an adaptive reinforcement learning (A-RL) framework to maximize the sum-rate for non-orthogonal multiple access-unmanned aerial vehicle (NOMA-UAV) network. In this framework, Mamdani fuzzy inference system (MFIS) supervises a reinforcement learning (RL) policy based on multi-armed bandits (MAB). UAV as learning agent serves an internet of things (IoT) region. It manages an interference affected, channel block for NOMA uplink. Sum-rate, rate outage probability and average bit error rate (BER) for far-user are compared. Simulations reveal superior performance of A-RL, compared to non-adaptive RL counterpart. Joint maximum likelihood detection (JMLD) and successive interference cancellation (SIC) are also compared for BER performance and implementation complexity.

Adaptive fault-tolerant tracking control of flying-wing unmanned aerial vehicle with system input saturation and state constraints

In this paper, an adaptive fault-tolerant attitude tracking controller based on reinforcement learning is developed for flying-wing unmanned aerial vehicle subjected to actuator faults and saturation. At first, the attitude dynamic model is separated into two dynamic subsystems as slow and fast dynamic subsystems based on the principle of time scale separation. Secondly, backstepping technique is adopted to design the controller. For the purpose of attitude angle constraints, the control technique based on Barrier Lyapunov is used to design controller of slow dynamic subsystem. Considering the optimization of the fast dynamic subsystem, this paper introduces an adaptive reinforcement learning control method in which neural network is used to approximate the long-term performance index and lumped fault dynamic. It is shown that this control algorithm can satisfy the requirements of attitude tracking subjected to the control constraints and the stability of the system is proved from Lyapunov stability theory. The simulation results demonstrate that the developed fault-tolerant scheme is useful and has more smooth control effect compared with fault-tolerant controller based on sliding mode theory.

Adaptive Reinforcement Learning-Enhanced Motion/Force Control Strategy for Multirobot Systems

This paper presents an adaptive reinforcement learning- (ARL-) based motion/force tracking control scheme consisting of the optimal motion dynamic control law and force control scheme for multimanipulator systems. Specifically, a new additional term and appropriate state vector are employed in designing the ARL technique for time-varying dynamical systems with online actor/critic algorithm to be established by minimizing the squared Bellman error. Additionally, the force control law is designed after obtaining the computation of constraint force coefficient by the Moore–Penrose pseudo-inverse matrix. The tracking effectiveness of the ARL-based optimal control is verified in the closed-loop system by theoretical analysis. Finally, simulation studies are conducted on a system of three manipulators to validate the physical realization of the proposed optimal tracking control design.

A power allocation algorithm based on cooperative Q-learning for multi-agent D2D communication networks

In the multi-agent device to device (D2D) communication networks, the scene of the multi-agent will change due to its mobility. To address communication interference and energy overconsumption problems caused by the lack of adaptability to changeful scenes, a power allocation algorithm based on scenes adaptive cooperative Q-learning (SACL) is proposed in the paper. Specifically, the scene variable is added into the state space, and the reward function in the algorithm is improved to achieve a larger system capacity with less power. Then, in order to improve the convergence speed of SACL algorithm, the balance factor is introduced based on the location distribution of multiple agents, and a fast scene adaptive reinforcement learning (FSACL) algorithm is proposed. Simulation experiments verify the adaptability of SACL and FSACL algorithm when the scene is changed. Compared with traditional cooperative Q-learning algorithm (CL) and independent Q-learning (IL) algorithms, the SACL and FSACL algorithm can obtain larger system capacity with smaller power to some extent. In addition, the FSACL algorithm converges faster than the CL, IL and the SACL algorithm.

Sliding Variable-based Online Adaptive Reinforcement Learning of Uncertain/Disturbed Nonlinear Mechanical Systems

In this work, the trajectory tracking control scheme is the framework of optimal control and robust integral of the sign of the error (RISE); sliding mode control technique for an uncertain/disturbed nonlinear robot manipulator without holonomic constraint force is presented. The sliding variable combining with RISE enables to deal with external disturbance and reduced the order of closed systems. The adaptive reinforcement learning technique is proposed by tuning simultaneously the actor–critic network to approximate the control policy and the cost function, respectively. The convergence of weight as well as tracking control problem was determined by theoretical analysis. Finally, the numerical example is investigated to validate the effectiveness of proposed control scheme.

Adaptive Reinforcement Learning Strategy with Sliding Mode Control for Unknown and Disturbed Wheeled Inverted Pendulum

This paper develops a novel adaptive integral sliding-mode control (SMC) technique to improve the tracking performance of a wheeled inverted pendulum (WIP) system, which belongs to a class of continuous time systems with input disturbance and/or unknown parameters. The proposed algorithm is established based on an integrating between the advantage of online adaptive reinforcement learning control and the high robustness of integral sliding-mode control (SMC) law. The main objective is to find a general structure of integral sliding mode control law that can guarantee the system state reaching a sliding surface in finite time. An adaptive/approximate optimal control based on the approximate/adaptive dynamic programming (ADP) is responsible for the asymptotic stability of the closed loop system. Furthermore, the convergence possibility of proposed output feedback optimal control was determined without the convergence of additional state observer. Finally, the theoretical analysis and simulation results validate the performance of the proposed control structure.

Imprecise neural computations as a source of adaptive behaviour in volatile environments.

In everyday life, humans face environments that feature uncertain and volatile or changing situations. Efficient adaptive behaviour must take into account uncertainty and volatility. Previous models of adaptive behaviour involve inferences about volatility that rely on complex and often intractable computations. Because such computations are presumably implausible biologically, it is unclear how humans develop efficient adaptive behaviours in such environments. Here, we demonstrate a counterintuitive result: simple, low-level inferences confined to uncertainty can produce near-optimal adaptive behaviour, regardless of the environmental volatility, assuming imprecisions in computation that conform to the psychophysical Weber law. We further show empirically that this Weber-imprecision model explains human behaviour in volatile environments better than optimal adaptive models that rely on high-level inferences about volatility, even when considering biologically plausible approximations of such models, as well as non-inferential models like adaptive reinforcement learning.

On Stability of Perturbed Nonlinear Switched Systems with Adaptive Reinforcement Learning

In this paper, a tracking control approach is developed based on an adaptive reinforcement learning algorithm with a bounded cost function for perturbed nonlinear switched systems, which represent a useful framework for modelling these converters, such as DC–DC converter, multi-level converter, etc. An optimal control method is derived for nominal systems to solve the tracking control problem, which results in solving a Hamilton–Jacobi–Bellman (HJB) equation. It is shown that the optimal controller obtained by solving the HJB equation can stabilize the perturbed nonlinear switched systems. To develop a solution to the translated HJB equation, the proposed neural networks consider the training technique obtaining the minimization of square of Bellman residual error in critic term due to the description of Hamilton function. Theoretical analysis shows that all the closed-loop system signals are uniformly ultimately bounded (UUB) and the proposed controller converges to optimal control law. The simulation results of two situations demonstrate the effectiveness of the proposed controller.

An Olympic curling robot

Artificial Intelligence The Olympic sport of curling is played on an ice sheet that is prone to changes throughout a match. The uncertain conditions of the ice sheet add further complexity to the game, beyond the prerequisites of accurate throwing and strategic thinking, making it a difficult sport to master and a challenging problem to solve for roboticists. Won et al. developed a curling robot, which they called Curly, that uses an adaptive deep reinforcement learning framework to react to past errors and adapt its throws to the current ice sheet conditions. Curly demonstrated human-like performance by winning three out of four matches against top-ranked human curling teams. Sci. Robot. 5 , eabb9764 (2020).

A Non-intrusive home load identification method based on adaptive reinforcement learning algorithm

At present, the load data collection of residential users mainly starts from the lower acquisition frequency, so the non-intrusive home load identification method based on low frequency sampling has attracted wide attention. However, low-frequency sampling has a low recognition accuracy when the training data set is small. Therefore, a non-intrusive home load identification method based on adaptive KNN reinforcement learning algorithm is proposed. The method firstly analyzes the state of the electrical appliance by KNN to obtain the initial HMM model, and then solves the HMM model by adaptive KNN reinforcement learning algorithm to obtain the optimal state transition strategy. This method reduces the model pair data. The dependence improves the recognition accuracy of the model and the adaptability to new data. Finally, the experimental verification is carried out by the low frequency data set AMPds. The results show that the method improves the state recognition accuracy of the electrical appliance and enhances the adaptability of the algorithm to new data.

Assessing the Use of Reinforcement Learning for Integrated Voltage/Frequency Control in AC Microgrids

The main purpose of this paper is to present a novel algorithmic reinforcement learning (RL) method for damping the voltage and frequency oscillations in a micro-grid (MG) with penetration of wind turbine generators (WTG). First, the continuous-time environment of the system is discretized to a definite number of states to form the Markov decision process (MDP). To solve the modeled discrete RL-based problem, Q-learning method, which is a model-free and simple iterative solution mechanism is used. Therefore, the presented control strategy is adaptive and it is suitable for the realistic power systems with high nonlinearities. The proposed adaptive RL controller has a supervisory nature that can improve the performance of any kind of controllers by adding an offset signal to the output control signal of them. Here, a part of Denmark distribution system is considered and the dynamic performance of the suggested control mechanism is evaluated and compared with fuzzy-proportional integral derivative (PID) and classical PID controllers. Simulations are carried out in two realistic and challenging scenarios considering system parameters changing. Results indicate that the proposed control strategy has an excellent dynamic response compared to fuzzy-PID and traditional PID controllers for damping the voltage and frequency oscillations.

A Learning-Based Power Management Method for Networked Microgrids Under Incomplete Information

This paper presents an approximate Reinforcement Learning (RL) methodology for bi-level power management of networked Microgrids (MG) in electric distribution systems. In practice, the cooperative agent can have limited or no knowledge of the MG asset behavior and detailed models behind the Point of Common Coupling (PCC). This makes the distribution systems unobservable and impedes conventional optimization solutions for the constrained MG power management problem. To tackle this challenge, we have proposed a bi-level RL framework in a price-based environment. At the higher level, a cooperative agent performs function approximation to predict the behavior of entities under incomplete information of MG parametric models; while at the lower level, each MG provides power-flow-constrained optimal response to price signals. The function approximation scheme is then used within an adaptive RL framework to optimize the price signal as the system load and solar generation change over time. Numerical experiments have verified that, compared to previous works in the literature, the proposed privacy-preserving learning model has better adaptability and enhanced computational speed.

NN Reinforcement Learning Adaptive Control for a Class of Nonstrict-Feedback Discrete-Time Systems.

This article investigates an adaptive reinforcement learning (RL) optimal control design problem for a class of nonstrict-feedback discrete-time systems. Based on the neural network (NN) approximating ability and RL control design technique, an adaptive backstepping RL optimal controller and a minimal learning parameter (MLP) adaptive RL optimal controller are developed by establishing a novel strategic utility function and introducing external function terms. It is proved that the proposed adaptive RL optimal controllers can guarantee that all signals in the closed-loop systems are semiglobal uniformly ultimately bounded (SGUUB). The main feature is that the proposed schemes can solve the optimal control problem that the previous literature cannot deal with. Furthermore, the proposed MPL adaptive optimal control scheme can reduce the number of adaptive laws, and thus the computational complexity is decreased. Finally, the simulation results illustrate the validity of the proposed optimal control schemes.

Fault-Tolerant Control of Degrading Systems with On-Policy Reinforcement Learning

We propose a novel adaptive reinforcement learning control approach for fault tolerant control of degrading systems that is not preceded by a fault detection and diagnosis step. Therefore, a priori knowledge of faults that may occur in the system is not required. The adaptive scheme combines online and offline learning of the on-policy control method to improve exploration and sample efficiency, while guaranteeing stable learning. The offline learning phase is performed using a data-driven model of the system, which is frequently updated to track the system’s operating conditions. We conduct experiments on an aircraft fuel transfer system to demonstrate the effectiveness of our approach.

Multigradient recursive reinforcement learning NN control for affine nonlinear systems with unmodeled dynamics

SummaryIn this paper, an adaptive reinforcement learning approach is developed for a class of discrete‐time affine nonlinear systems with unmodeled dynamics. The multigradient recursive (MGR) algorithm is employed to solve the local optimal problem, which is inherent in gradient descent method. The MGR radial basis function neural network approximates the utility functions and unmodeled dynamics, which has a faster rate of convergence than that of the gradient descent method. A novel strategic utility function and cost function are defined for the affine systems. Finally, it concludes that all the signals in the closed‐loop system are semiglobal uniformly ultimately bounded through differential Lyapunov function method, and two simulation examples are presented to demonstrate the effectiveness of the proposed scheme.

An intelligent decision support system prototype for hinterland port logistics

Port logistics is characterised by a high degree of fragmentation, uncertainty and complexity. In a context with these characteristics, decision support can be of significant value. This study presents the prototype of an intelligent decision support system (DSS) that leads to horizontal and vertical cooperation among freight agents involved in port logistics. A multi-agent simulation model is presented where heterogeneous actions are enabled as a result of an adaptive reinforcement learning algorithm that is inspired by human decision-making strategies. The model combines optimisation modelling and decision theory to operate in a dynamic environment characterised by information asymmetry among agents and dynamic changes over time. A simulation demonstrates the dynamics and convergence to equilibrium of the interactions of heterogeneous agents for delivery and pick up of import/export shipments.In particular, we answer two specific research questions. What is the likely impact of an intelligent DSS in hinterland container transport as a value-added service of port community system (PCS)? How can an optimum cooperative strategy be formulated to meet the dynamic demand and supply of freight agents in hinterland container transport and achieve agility in the age of hyper-competition?By addressing these two research questions we make a specific contribution by developing an agent-based simulation model in a real-world and large-scale case study, by using a reinforcement learning model based on probability matching theory that allows simulating realistically the adaptive behaviour of agents. The results of the simulation of two weeks of container movements indicate huge savings in total transport costs and distance travelled as well as higher utilisation of trucks from the sharing of resources. Moreover, we show that it is strictly better for smaller freight agents to cooperate – a conclusion generated endogenously inside the model as a trade-off between exploration and exploitation. As a result of this prototype simulation, major freight agents will not necessarily benefit from the DSS mainly because they are already using the economies of scale.

Parallel Optimal Tracking Control Schemes for Mode-Dependent Control of Coupled Markov Jump Systems via Integral RL Method

This article is concerned with the optimal tracking control problem of the coupled Markov jump system (CMJS) by using the reinforcement learning (RL) technique. Based on the conventional optimal tracking architecture, an offline tracking iteration algorithm is first designed to solve the coupled algebraic Riccati equation that can hardly be solved by mathematical methods directly. To overcome the crucial requirements and existing shortcomings in the offline tracking method, a novel integral RL (IRL) tracking algorithm is first proposed for CMJS, which develops a transition-probability-free optimal tracking control scheme with a reconstructed augmented system and discounted cost function. Both the requirements of transition probability $\pi _{ij}$ and system matrix $A_{i}$ are avoided via the designed IRL algorithm. The stability and convergence of the novel schemes are proved by the Lyapunov theory, and the tracking objective is achieved as desired. Finally, we apply the designed algorithms in a fourth-order Markov jump control problem and the stochastic mass, spring, and damper system to track continuous sinusoidal waveforms, and the simulation results are provided to show the effectiveness and applicability. Note to Practitioners —In the practical engineering systems, many useful signals and interference vary randomly. Therefore, the tracking control of stochastic systems and dynamics, such as the Markovion, Ito’s, Wiener, and Martingale processes, plays an important role in the modern industry. As a matter of fact, it is always desired to reduce the requirement of exact information and transition probability in the homogeneous Markovian process, which is very difficult to obtain accurate measurements. One way is integrating the adaptive reinforcement learning (RL) technique into the Markovian systems to learn this implicit information. However, a major restriction of the RL technique is that the control policy should be related to the finite performance index, which generally invalidates the optimal tracking solutions. In order to tackle this difficulty, by designing a novel parallel scheme via integral RL (IRL) technique, the solution of the coupled algebraic Riccati equation is solved, and the transition probability can be completely unknown during the learning process.

Research on Fuzzy Enhanced Learning Model of Multienhanced Signal Learning Automata

With the continuous iteration of information technology, information presents an explosive development trend, so how to effectively classify, retrieve, and summarize information is very important. Data classification and mining technology as the key technology to solve the aforementioned problems is of great significance. At present, in the fields of data processing, information classification and machine learning reinforcement learning, it is usually based on the learning automata, but the traditional learning automata algorithm cannot meet the development of information technology in terms of convergence and learning rate. In order to improve the convergence stability and learning efficiency of the learning automata algorithm in random environment and enhance its learning ability in dealing with large-scale space and action, this paper innovatively proposes an adaptive fuzzy reinforcement learning model of the learning automata, which makes full use of the randomness of gradient information to solve its stability problem, and abandons the original reinforcement learning method. For the linear function, an adaptive fuzzy numerical superposition algorithm is used to further enhance its learning rate and improve its stability. The experimental results show that the proposed learning automata fuzzy reinforcement learning model has practical application value, and can solve the problems of unstable convergence and poor learning efficiency of the current learning automata.