Combination Of Reinforcement Learning Research Articles

Artificial Intelligence for Real-Time Tolerance to Critical Flight Data Errors in Large Aircraft

The environment in the cockpit of large transport aircraft is highly complex due to an increasing number of automation systems. This complexity can cause pilots to become less aware of how systems interact. It becomes a severe issue when sensor or data failures occur, as such failures can contribute to a situation in which it is difficult for a pilot to assess what actually is happening and, possibly, how to resolve the problem. This paper presents a method, based on artificial intelligence, for identifying incorrect critical flight control data in real-time. A novel combination of reinforcement learning and a denoising autoencoder is proposed to identify failures and to provide inputs to the aircraft’s flight control and guidance systems, allowing for the correct maneuver to counter the failure and/or to avoid or recover from flight upsets. Tests in stall conditions with a partially blocked pitot tube show that the proposed method results in successful detection and recovery. The performance of the system without an autoencoder is compared to highlight the significant advantages, how this relates to creating systems with AI to improve situational awareness for pilots, and to execute appropriate automatic maneuvers to successfully counter the effect of sensor failures.

Optimized leader‐follower consensus control using combination of reinforcement learning and sliding mode mechanism for multiple robot manipulator system

AbstractThis article is to develop an optimized leader‐follower consensus control for multiple robot manipulator system by combining sliding mode control (SMC) and reinforcement learning (RL). The SMC mechanism aims to steer both position and velocity states of multiple manipulator reaching the predefined trajectory. And the RL is designed and performed under identifier‐critic‐actor architecture for the achievement of optimized control performance. Compared to traditional optimal control, this proposed method has two main advantages: (i) the RL updating laws for training both the actor and critic networks are simpler; (ii) the optimized control can not require the complete dynamic knowledge because the adaptive identifier is designed into the RL learning. Consequently, this optimized control method can smoothly steer the multiple robot manipulator system to achieve the leader‐follower consensus. Finally, feasibility of this control method is verified through both theory and simulation.

Environment-Aware Adaptive Reinforcement Learning-Based Routing for Vehicular Ad Hoc Networks.

With the rapid development of the intelligent transportation system (ITS), routing in vehicular ad hoc networks (VANETs) has become a popular research topic. The high mobility of vehicles in urban streets poses serious challenges to routing protocols and has a significant impact on network performance. Existing topology-based routing is not suitable for highly dynamic VANETs, thereby making location-based routing protocols the preferred choice due to their scalability. However, the working environment of VANETs is complex and interference-prone. In wireless-network communication, the channel contention introduced by the high density of vehicles, coupled with urban structures, significantly increases the difficulty of designing high-quality communication protocols. In this context, compared to topology-based routing protocols, location-based geographic routing is widely employed in VANETs due to its avoidance of the route construction and maintenance phases. Considering the characteristics of VANETs, this paper proposes a novel environment-aware adaptive reinforcement routing (EARR) protocol aimed at establishing reliable connections between source and destination nodes. The protocol adopts periodic beacons to perceive and explore the surrounding environment, thereby constructing a local topology. By applying reinforcement learning to the vehicle network's route selection, it adaptively adjusts the Q table through the perception of multiple metrics from beacons, including vehicle speed, available bandwidth, signal-reception strength, etc., thereby assisting the selection of relay vehicles and alleviating the challenges posed by the high dynamics, shadow fading, and limited bandwidth in VANETs. The combination of reinforcement learning and beacons accelerates the establishment of end-to-end routes, thereby guiding each vehicle to choose the optimal next hop and forming suboptimal routes throughout the entire communication process. The adaptive adjustment feature of the protocol enables it to address sudden link interruptions, thereby enhancing communication reliability. In experiments, the EARR protocol demonstrates significant improvements across various performance metrics compared to existing routing protocols. Throughout the simulation process, the EARR protocol maintains a consistently high packet-delivery rate and throughput compared to other protocols, as well as demonstrates stable performance across various scenarios. Finally, the proposed protocol demonstrates relatively consistent standardized latency and low overhead in all experiments.

Physics-constrained deep reinforcement learning for flow field denoising

A multi-agent deep reinforcement learning (DRL)-based model is presented in this study to reconstruct flow fields from noisy data. A combination of reinforcement learning with pixel-wise rewards, physical constraints represented by the momentum equation and the pressure Poisson equation, and the known boundary conditions is used to build a physics-constrained deep reinforcement learning (PCDRL) model that can be trained without the target training data. In the PCDRL model, each agent corresponds to a point in the flow field and learns an optimal strategy for choosing pre-defined actions. The proposed model is efficient considering the visualisation of the action map and the interpretation of the model operation. The performance of the model is tested by using direct numerical simulation-based synthetic noisy data and experimental data obtained by particle image velocimetry. Qualitative and quantitative results show that the model can reconstruct the flow fields and reproduce the statistics and the spectral content with commendable accuracy. Furthermore, the dominant coherent structures of the flow fields can be recovered by the flow fields obtained from the model when they are analysed using proper orthogonal decomposition and dynamic mode decomposition. This study demonstrates that the combination of DRL-based models and the known physics of the flow fields can potentially help solve complex flow reconstruction problems, which can result in a remarkable reduction in the experimental and computational costs.

Automated Antenna Design via Domain Knowledge-Informed Reinforcement Learning and Imitation Learning

In this study, we propose a novel approach for automating antenna design through the utilization of domain knowledge-informed reinforcement learning (RL) and imitation learning (IL). The proposed method allows the RL agent to learn from its interactions with the environment and to take actions that maximize the reward signal, resulting in the optimal policy. Despite the attractive advantages, the learning process of RL can be challenging and time-consuming, as the agent must explore a vast range of states, actions, and policies. To enhance the efficiency of the design process, we incorporate two key components into the proposed method. Firstly, we utilize domain knowledge, such as the scaling property of antenna, to initialize the antenna parameters. This not only provides interpretability to the design process but also reduces the solution space. Additionally, we employ IL to pretrain the decision-making network within the RL algorithm, which significantly mitigates the data inefficiency issue for RL and reduces the time required to explore feasible solutions. The proposed method has been evaluated using the design of a slot filtering antenna as a case study. Simulation results indicate that the proposed method can achieve the antenna design for different frequencies and bandwidth metrics with at least 50% fewer adjusting steps than the traditional reinforcement learning method, such as deep deterministic policy gradient (DDPG). The combination of RL and IL also results in a considerable performance boost compared to both pure IL and pure RL for antenna designs.

Reinforcement Learning Recommendation Algorithm Based on Label Value Distribution

Reinforcement learning is an important machine learning method and has become a hot popular research direction topic at present in recent years. The combination of reinforcement learning and a recommendation system, is a very important application scenario and application, and has always received close attention from researchers in all sectors of society. In this paper, we first propose a feature engineering method based on label distribution learning, which analyzes historical behavior is analyzed and constructs, whereby feature vectors are constructed for users and products via label distribution learning. Then, a recommendation algorithm based on value distribution reinforcement learning is proposed. We first designed the stochastic process of the recommendation process, described the user’s state in the interaction process (by including the information on their explicit state and implicit state), and dynamically generated product recommendations through user feedback. Next, by studying hybrid recommendation strategies, we combined the user’s dynamic and static information to fully utilize their information and achieve high-quality recommendation algorithms. Finally, the algorithm was designed and validated, and various relevant baseline models were compared to demonstrate the effectiveness of the algorithm in this study. With this study, we actually tested the remarkable advantages of relevant design models based on nonlinear expectations compared to other homogeneous individual models. The use of recommendation systems with nonlinear expectations has considerably increased the accuracy, data utilization, robustness, model convergence speed, and stability of the systems. In this study, we incorporated the idea of nonlinear expectations into the design and implementation process of recommendation systems. The main practical value of the improved recommendation model is that its performance is more accurate than that of other recommendation models at the same level of computing power level. Moreover, due to the higher amount of information that the enhanced model contains, it provides theoretical support and the basis for an algorithm that can be used to achieve high-quality recommendation services, and it has many application prospects.

A Combined Artificial-Intelligence Aerodynamic Design Method for a Transonic Compressor Rotor Based on Reinforcement Learning and Genetic Algorithm

An aircraft engine’s performance depends largely on the compressors’ aerodynamic design, which aims to achieve higher stage pressure, efficiency, and an acceptable stall margin. Existing design methods require substantial prior knowledge and different optimization algorithms to determine the 2D and 3D features of the blades, in which the design policy needs to be more readily systematized. With the development of artificial intelligence (AI), deep reinforcement learning (RL) has been successfully applied to complex design problems in different domains and provides a feasible method for compressor design. In addition, the applications of AI methods in compressor research have progressively developed. This paper described a combined artificial-intelligence aerodynamic design method based on a modified deep deterministic policy gradient algorithm and a genetic algorithm (GA) and integrated the GA into the RL framework. The trained agent learned the design policy and used it to improve the GA optimization result of a single-stage transonic compressor rotor. Consequently, the rotor exhibited a higher pressure ratio and efficiency owing to the sweep feature, lean feature, and 2D airfoil angle changes. The separation near the tip and the secondary flow decreased after the GA process, and at the same time, the shockwave was weakened, providing improved efficiency. Most of these beneficial flow field features remained after agent modification to improve the pressure ratio, showing that the policy learned by the agent was generally universal. The combination of RL and other design optimization methods is expected to benefit the future development of compressor designs by merging the advantages of different methods.

Reinforcement Learning and its application in making Recommendation System

Reinforcement learning (RL) is an area of machine learning concerned with how intelligent agents are made to take actions in an environment in order to maximize the total reward. RL works on Markov Decision Process which leads to Q-learning. MDP provides a mechanism to maximize the reward in a given environment. Deep reinforcement learning is the combination of reinforcement learning (RL) and deep learning. DRL has applications in many fields like medicine, robotics, games, etc. Combining DL and RL leads to the formation of Deep QNetworks. Another application of RL and the focus of this seminar is personalized recommendation systems. Recommendation systems are trained on user-item interaction to predict the next item that a user can be interested in. In personalized recommendation systems, it is important to consider a lot of factors like user-item interactions, clicks, purchases, loss, etc. DRL does a very good job of reducing the loss in recommendation systems and recommending relevant items. I will explore a technique called self-supervised learning in which a model is made to predict data for itself for its use in recommendation systems. I also summarize 2 frameworks namely SelfSupervised Q-learning (SQN) and Self-Supervised Actor-Critic (SAC) based on self-supervised learning and try to understand how they work differently as compared to non-RL recommendation systems.

Plume: Lightweight and generalized congestion control with deep reinforcement learning

Congestion control (CC) is always an important issue in the field of networking, and the enthusiasm for its research has never diminished in both academia and industry. In current years, due to the rapid development of machine learning (ML), the combination of reinforcement learning (RL) and CC has a striking effect. However, These complicated schemes lack generalization and are too heavyweight in storage and computing to be directly implemented in mobile devices. In order to address these problems, we propose Plume, a high-performance, lightweight and generalized RL-CC scheme. Plume proposes a lightweight framework to reduce the overheads while preserving the original performance. Besides, Plume innovatively modifies the framework parameters of the reward function during the retraining process, so that the algorithm can be applied to a variety of scenarios. Evaluation results show that Plume can retain almost all the performance of the original model but the size and decision latency can be reduced by more than 50% and 20%, respectively. Moreover, Plume has better performances in some special scenes.

Reservoirs Optimal Operation Based on Reinforcement Learning

Reinforcement learning has been widely used to solve the problem of optimal operation of a single reservoir. However, the research on the combination of reinforcement learning and optimal operation of reservoir groups is still blank. The complexity of reservoir groups brings difficulties to the application of reinforcement learning. In order to solve the complex problem of optimal operation of the reservoir group, Q-learning combined with the penalty method is used to generate the optimal operation scheme of the reservoir group by exploring the optimization Q value of the operation sequence. The experimental results show that this method can achieve the optimal solution in theory. Furthermore, Q-learning combined with a feasible direction method is used to establish time feasible state table and state-feasible action hash table according to the constraints of the discrete four-reservoirs problem, so that the Q-learning algorithm can shorten the optimization time and obtain the optimal solution.

Learning for MPC with stability & safety guarantees

The combination of learning methods with Model Predictive Control (MPC) has attracted a significant amount of attention in the recent literature. The hope of this combination is to reduce the reliance of MPC schemes on accurate models, and to tap into the fast developing machine learning and reinforcement learning tools to exploit the growing amount of data available for many systems. In particular, the combination of reinforcement learning and MPC has been proposed as a viable and theoretically justified approach to introduce explainable, safe and stable policies in reinforcement learning. However, a formal theory detailing how the safety and stability of an MPC-based policy can be maintained through the parameter updates delivered by the learning tools is still lacking. This paper addresses this gap. The theory is developed for the generic robust MPC case, and applied in simulation in the robust tube-based linear MPC case, where the theory is fairly easy to deploy in practice. The paper focuses on reinforcement learning as a learning tool, but it applies to any learning method that updates the MPC parameters online.

Mitigating Routing Update Overhead for Traffic Engineering by Combining Destination-Based Routing With Reinforcement Learning

Traffic Engineering (TE) is a widely-adopted network operation to optimize network performance and resource utilization. Destination-based routing is supported by legacy routers and more readily deployed than flow-based routing, where the forwarding entries could be frequently updated by TE to accommodate traffic dynamics. However, as the network size grows, destination-based TE could render high time complexity when generating and updating many forwarding entries, which may limit the responsiveness of TE and degrade network performance. In this paper, we propose a novel destination-based TE solution called FlexEntry, which leverages emerging Reinforcement Learning (RL) to reduce the time complexity and routing update overhead while achieving good network performance simultaneously. For each traffic matrix, FlexEntry only updates a few forwarding entries called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">critical entries</i> for redistributing a small portion of the total traffic to improve network performance. These critical entries are intelligently selected by RL with traffic split ratios optimized by Linear Programming (LP). We find out that the combination of RL and LP is very effective. Our simulation results on six real-world network topologies show that FlexEntry reduces up to 99.3% entry updates on average and generalizes well to unseen traffic matrices with near-optimal load balancing performance.

Automated Reinforcement Learning (AutoRL): A Survey and Open Problems

The combination of Reinforcement Learning (RL) with deep learning has led to a series of impressive feats, with many believing (deep) RL provides a path towards generally capable agents. However, the success of RL agents is often highly sensitive to design choices in the training process, which may require tedious and error-prone manual tuning. This makes it challenging to use RL for new problems and also limits its full potential. In many other areas of machine learning, AutoML has shown that it is possible to automate such design choices, and AutoML has also yielded promising initial results when applied to RL. However, Automated Reinforcement Learning (AutoRL) involves not only standard applications of AutoML but also includes additional challenges unique to RL, that naturally produce a different set of methods. As such, AutoRL has been emerging as an important area of research in RL, providing promise in a variety of applications from RNA design to playing games, such as Go. Given the diversity of methods and environments considered in RL, much of the research has been conducted in distinct subfields, ranging from meta-learning to evolution. In this survey, we seek to unify the field of AutoRL, provide a common taxonomy, discuss each area in detail and pose open problems of interest to researchers going forward.

Research on Dynamic Spectrum Allocation Algorithm Based on Cyclic Neural Network

Due to the wide application of cognitive wireless network, the network structure is becoming more and more complex. It is difficult to establish the corresponding mathematical model to simulate the high complexity network environment. The algorithm based on recurrent neural network in deep reinforcement learning can effectively solve this problem. In addition, with the rise of deep learning in recent years, the combination of reinforcement learning and deep learning shows excellent ability in dealing with complex problems and data operation. This paper is aimed at studying dynamic spectrum allocation based on cyclic neural network. This paper briefly introduces MATLAB, sets up the network environment of algorithm simulation, then analyzes the overall performance of the improved genetic algorithm, and explores the influence of genetic algorithm-related parameters and network environment-related parameters on the performance of the algorithm. The results show the improved genetic algorithm. The network efficiency can be improved by about 2%, but the spectrum switching frequency can be reduced by 69%. When the number of primary users in the network is large, the network benefit of improving the genetic algorithm is superior to the other two algorithms. In addition, when the crossover probability is 0.6 and 0.1, the fitness value is higher than the crossover probability of 0.9 and 0.5; the interference of authorized users in the network initially has less impact on the secondary user.

An improved marine predators algorithm for the optimal design of hybrid renewable energy systems

Microgrid technologies are exciting energy sources that are economically feasible for current and future applications in light of increased energy demand and the depletion of traditional sources. This article focuses on the latest metaheuristic algorithm, the marine predators algorithm (MPA), in the field of energy. To investigate a method for reducing the system’s investment costs in Minia, Egypt, the combination of reinforcement learning (RL) with MPA is used to build a new method, Deep-MPA, where RL principles are applied to adjust and enhance the lack of MPA in global searching. The exploration/exploitation ratio is regulated by varying the step size, which affects the efficiency of the MPA. Instead of updating the value of the parameter for all agents in the same manner, RL principles are used to update it on the basis of the current individual state. Additionally, to resolve the common challenge of using RL to determine the appropriate global search parameters for MPA, Deep-MPA is used to design a hybrid renewable energy microgrid system, which includes photovoltaic panels, a wind turbine system, a diesel generator, and battery storage systems. These are some of the criteria and constraints that this system may require to ensure its stability, robustness, performance, and load satisfaction. The proposed Deep-MPA is verified by contrasting the results with different algorithms in the CEC’2017 test. Moreover, Wilcoxon’s test validates the statistical significance of the Deep-MPA. The energy cost is reduced by 6% of total consumption.

Meta Q-network: a combination of reinforcement learning and meta learning

Deep reinforcement learning develops rapidly by using neural network to approximate the learning data of reinforcement learning, which makes the sequential decision in continuous space making achieve preliminary results. However, deep reinforcement learning is over-dependent on huge amount of training and requires accurate reward. For many problems in the real world, such as robot learning, there is generally no good reward and no unlimited training, which requires the ability to learn quickly. In this paper, we propose a deep reinforcement learning model with meta-learning, which we call meta Q-network (MQN). The model uses a LSTM-based meta-learner to update the Q-network. This optimises a series of problems such as the difficulty in stability of Q-network in deep reinforcement learning model, and we have proved this improved performance through experiments. It is not optimal, though, still the combination of meta-learning and reinforcement learning is very desirable.

Meta Q-network: a combination of reinforcement learning and meta learning

Meta Q-network: a combination of reinforcement learning and meta learning

Approach to Automation of Line Heating by Combination of Reinforcement Learning and Finite Element Method Simulation

Abstract In recent years, ship hulls have very complicated shapes in order to reduce frictional resistance and wave resistance during navigation. In particular, in the bow and stern, curved skin plates with complex shapes are used. Line heating is used to produce such complex shapes. Line heating is a bending technique using plastic deformation due to heating. The relationship between the heat input and the deformation is nonlinear, which may lead to difficulty in making a heating plan for forming the target shape. Thus, skilled workers are necessary in line heating, and the work time and dimensional accuracy depend on their skills. Another problem is the transfer of this technique to future generations. In order to overcome these problems, automation of the line heating process has been investigated urgently. On the other hand, artificial intelligence (AI) technology has been rapidly developed in recent years. An AI system can deal with nonlinear relationships and ambiguous feature quantities, which are difficult to express mathematically. By using AI, automation of the planning of the heating line can be expected. The purpose of the present study is to obtain the optimal heat input conditions for forming an arbitrary shape in line heating. In order to accomplish this, we constructed an AI system that integrated deep layer reinforcement learning and line heating simulation. The proposed system was applied to the formation of fundamental shapes of line heating, including the bowl shape, the saddle shape, and the twisted shape. As a result, the proposed system was found to be able to generate heating plans for these shapes with fewer heating lines.

E xploration E xploitation Problem in Policy Based Deep Reinforcement Learning for Episodic and Continuous Environments

Reinforcement learning is an artificial intelligence paradigm that enables intelligent agents to accrue environmental incentives to get superior results. It is concerned with sequential decision-making problems which offer limited feedback. Reinforcement learning has roots in cybernetics and research in statistics, psychology, neurology, and computer science. It has piqued the interest of the machine learning and artificial intelligence groups in the last five to ten years. It promises that it allows you to train agents using rewards and penalties without explaining how the task will be completed. The RL issue may be described as an agent that must make decisions in a given environment to maximize a specified concept of cumulative rewards. The learner is not taught which actions to perform but must experiment to determine which acts provide the greatest reward. Thus, the learner has to actively choose between exploring its environment or exploiting it based on its knowledge. The exploration-exploitation paradox is one of the most common issues encountered while dealing with Reinforcement Learning algorithms. Deep reinforcement learning is the combination of reinforcement learning (RL) and deep learning. We describe how to utilize several deep reinforcement learning (RL) algorithms for managing a Cartpole system used to represent episodic environments and Stock Market Trading, which is used to describe continuous environments in this study. We explain and demonstrate the effects of different RL ideas such as Deep Q Networks (DQN), Double DQN, and Dueling DQN on learning performance. We also look at the fundamental distinctions between episodic and continuous activities and how the exploration-exploitation issue is addressed in their context.

Robotic architectural assembly with tactile skills: Simulation and optimization

Construction is an industry that could benefit significantly from automation, yet still relies heavily on manual human labor. Thus, we investigate how a robotic arm can be used to assemble a structure from predefined discrete building blocks autonomously. Since assembling structures is a challenging task that involves complex contact dynamics, we propose to use a combination of reinforcement learning and planning for this task. In this work, we take a first step towards autonomous construction by training a controller to place a single building block in simulation. Our evaluations show that trial-and-error algorithms that have minimal prior knowledge about the problem to be solved, so called model-free deep reinforcement learning algorithms, can be successfully employed. We conclude that the achieved results, albeit imperfect, serve as a proof of concept and indicate the directions for further research to enable more complex assemblies involving multiple building elements.