Proposed Reinforcement Learning Research Articles

A reconfigurable RF front-end for 5G direct sampling receivers with an optimized calibration scheme

This paper presents a reconfigurable radio frequency front-end (RFFE) tailored for direct RF sampling receivers operating within Frequency Range 1 (FR-1) of the 5G spectrum. It consists of a balun-LNA, a noise-cancelling, current-reuse, Q-enhanced filter, and a programmable gain amplifier (PGA). Fabricated in 22-nm FD-SOI technology, the RFFE covers the entire frequency range from 1.7 to 6.4 GHz with a tunable bandwidth ranging from 50 MHz to 1.2 GHz. The experimental results confirm that the RFFE achieves 3.9 dB NF, 55 dB ultimate OOB rejection, −4 dBm IB-IIP3, and 23 dBm OOB-IIP3. Furthermore, a digital calibration scheme is proposed to compensate for process, voltage, and temperature (PVT) variations, achieving lower than 2 % error in 6.61 µs on average. Subsequently, a proximal policy optimization (PPO) agent is employed to choose the optimal policy for the successive adjustments of quality factors and resonance frequencies of the filter’s resonators. As a result, the proposed reinforcement learning algorithm reduces the calibration’s convergence time by 59 %.

A Model-based Reinforcement Learning Algorithm for Multi-agent Cooperation Nash Equilibrium with Unstable Communication

Solving Nash equilibrium is important to multi-agent systems, in which the communication is an important factor. However, many proposed reinforcement learning (RL) based algorithms take into account the communication factors by assuming stable communication conditions, which does not hold in the real environment. In this paper, we analyze the effect of a typical RL algorithm in the cases of unstable communication and communication failure, which causes information loss between agents, leading to isolation of agents and affecting algorithm convergence. Then, we propose a model-based RL algorithm to solve Nash equilibrium for multi-agent systems when agents are isolated, and prove its convergence and rationality through mathematical proofs. The simulations results show the effectiveness of the proposed algorithm.

Reinforcement Learning-Based Power Control for Reliable Mission-Critical Wireless Transmission

In this paper, we investigate sequential power allocation over fast varying channels for mission-critical applications, aiming to minimize the expected sum power while guaranteeing the transmission success probability. In particular, a reinforcement learning framework is constructed with appropriate reward design so that the optimal policy maximizes the Lagrangian of the primal problem, where the maximizer of the Lagrangian is shown to have several good properties. For the model-based case, a fast converging algorithm is proposed to find the optimal Lagrange multiplier and thus the corresponding optimal policy. For the model-free case, we develop a three-stage strategy, composed in order of online sampling, offline learning, and online operation, where a backward Q-learning with full exploitation of sampled channel realizations is designed to accelerate the learning process. According to our simulation, the proposed reinforcement learning framework can solve the primal optimization problem from the dual perspective. Moreover, the model-free strategy achieves a performance close to that of the optimal model-based algorithm.

Automated Design of Metaheuristics Using Reinforcement Learning Within a Novel General Search Framework

Metaheuristic algorithms have been investigated intensively to address highly complex combinatorial optimisation problems. However, most metaheuristic algorithms have been designed manually by researchers of different expertise without a consistent framework to support effective algorithm design. This paper proposes a general search framework to formulate in a unified way a range of different metaheuristics. This framework defines generic algorithmic components, including selection heuristics and evolution operators. The unified general search framework aims to serve as the basis of analysing algorithmic components for automated algorithm design. With the established new general search framework, two reinforcement learning based methods, deep Q-network based and proximal policy optimisation based methods, have been developed to automatically design a new general population-based algorithm. The proposed reinforcement learning based methods are able to intelligently select and combine appropriate algorithmic components during different stages of the optimisation process. The effectiveness and generalization of the proposed reinforcement learning based methods are validated comprehensively across different benchmark instances of the capacitated vehicle routing problem with time windows. This study contributes to making a key step towards automated algorithm design with a general framework supporting fundamental analysis by effective machine learning.

A reinforcement learning algorithm for scheduling parallel processors with identical speedup functions

In this study, we investigate a real-time system where computationally intensive tasks are executed using cloud computing platforms in data centers. These data centers are designed to provide prompt responses to incoming demands. To achieve this objective, an efficient scheduling system is crucial for determining the assignment of jobs to processors and the optimal starting times for each job’s execution. In this paper, we propose a novel reinforcement learning algorithm that introduces a new state variable and utilizes a set of virtual bins to classify jobs based on their remaining processing times. Our objective is to minimize the total slowdown, which is defined as the sum of completion time ratios to job demand sizes. We conduct a performance evaluation by comparing our developed algorithm with commonly used and highly efficient dispatching rules found in existing literature. The computational results demonstrate that our proposed reinforcement learning approach outperforms other solution approaches available in the literature. Furthermore, we illustrate the generalization capability of our algorithm and its ability to achieve superior results compared to dispatching rules when applied to new test instances.

Reinforcement Learning With Model-Based Assistance for Shape Control in Sendzimir Rolling Mills

As one of the most popular tandem cold rolling mills, the Sendzimir rolling mill (ZRM) aims to obtain a flat steel strip shape by properly allocating the rolling pressure. To improve the performance of the ZRM, it is meaningful to adopt recently emerging deep reinforcement learning (DRL) that is powerful for difficult-to-solve and challenging problems. However, the direct application of DRL techniques may be impractical because of a serious singularity, partial observability, and even safety issues inherent in mill systems. In this brief, we propose an effective hybridization approach that integrates a model-based assistant into model-free DRL to resolve such practical issues. For the model-based assistant, a model-based optimization problem is first constructed and solved for the static part of the mill model. Then, the obtained static model-based coarse assistant, or controller, is improved by the proposed reinforcement learning, considering the remaining dynamic part of the mill model. The serious singularity can be resolved using the model-based approach, and the issue of partial observability is addressed by the long short-term memory (LSTM) state estimator in the proposed method. In simulation results, the proposed method successfully learns a highly performing policy for the ZRM, achieving a higher reward than pure model-free DRL. It is also observed that the proposed method can safely improve the shape controller of the mill system. The demonstration results strongly confirm the high applicability of DRL to other cold multiroll mills, such as four-high, six-high, and cluster mills.

Optimal Control Algorithm for Subway Train Operation by Proximal Policy Optimization

With the increasing scale of the urban subway, the total energy consumption of the subway has increased dramatically and poses a great challenge to the comfort of passengers and the punctuality of train operation. In order to ensure on-time train operation and passenger comfort, and at the same time reduce the energy consumption of subway operation, this paper proposes a Proximal Policy Optimization (PPO)-based optimization algorithm for the optimal control of subway train operation. Firstly, a reinforcement learning architecture for optimal control of subway train operation is constructed with the position and speed of train operation as the reinforcement learning state, energy consumption and comfort as the optimization objectives, and train operation time as the constraint. The proposed reinforcement learning model is trained by the PPO algorithm, and the reward scaling is added to the training process to accelerate the training speed and improve the efficiency of the algorithm. The experimental results show that the proposed PPO with reward scaling algorithm can effectively reduce train energy consumption and improve passenger comfort while ensuring on-time train operation.

Wireless Resources Cooperation of Assembled Small UAVs for Data Collections of IoT

Small Unmanned Air Vehicles (UAVs) have many advantages, including low cost, and flexible deployment. And they play an important role to collect the sensing data of IoT (Internet of Things). However, limited by the load capability, it is a big challenge for them to perform long-term, large range, or far distance tasks. In order to tackle these challenges, we propose to use assembly UAVs, in which we can jointly optimize the resource management, especially the energy resource. The system model, energy cyclic cooperation, and one of the typical applications based on assembly UAVs are introduced. The energy cooperation problems are formulated and an in-air replenishing strategy (IA-RS) is proposed. Simulations show that the performances of the proposed IA-RS outperform those of the traditional on-ground replenishing strategy (OG-RS). The working time of task UAV (UAV-T) could reduce 8.3%-19.5%, and the freshness of the collected data and the collecting efficiency of the UAV-T can be improved. We also optimize the path of the replenishing UAVs (UAV-Rs). Simulations show that the proposed reinforcement learning (RL) algorithm has the best performances and acceptable complexity. Consequently, the efficiency of the IoT data collection task is improved by the proposed assembly UAVs.

RSCAT: Towards zero touch congestion control based on actor–critic reinforcement learning and software-defined networking

Network congestion is a phenomenon present in contemporaneous data centers (DCs) independently of scale and underlying technologies. The small-scale presence of congestion causes information delay for the DC hosted services, while in large-scale data losses or even service downtimes are not uncommon. Several congestion control algorithms out in the market have no network support, and as such, they must act and rely only on the data provided by the servers they are running on. Some algorithms have used network’s feedback to improve the performance, however these solutions depend on tuning parameters for marking queued packets, an arduous task on DCs shared by multiple applications with different workloads. In turn, Software-Defined Networking (SDN) created an opportunity to avoid congestion once the centralized controller can gather ongoing and historical information from all network switches and flows. The data gathered is enormous, and fast-computing algorithms are crucial for decision-making. In this sense, this work proposes Reinforcement Learning and SDN-aided Congestion Avoidance Tool (RSCAT), which uses data classification to determine if the network is congested and actor–critic reinforcement learning to find better Transmission Control Protocol (TCP) parameters. We evaluated RSCAT based on three workloads: a diversified DC traffic simulation; high-performance computing from NAS benchmark; and Hadoop TeraSort application. Our results show RSCAT fundamentals (reinforcement learning, data classification, and SDN) are potential candidates to achieve zero touch network congestion control. Specifically, our experimental analysis shows RSCAT, running alongside with traditional TCP congestion control algorithms like DCTCP and CUBIC, can decrease the flow completion time and applications’ runtimes in several cases, without requiring any software update on DC end-points.

Efficient and Private Scheduling of Wireless Electric Vehicles Charging Using Reinforcement Learning

Future vehicle-to-grid (V2G) systems require more flexible scheduling to adjust and flatten the peak energy demand. For efficient scheduling and energy trading, the utility provider (UP) needs to keep track of the state of charge (SoC) of vehicle batteries (VBs). However, sharing of SoC of VBs from electric vehicles (EVs) to UP may compromise owner privacy by analyzing the electricity usage in EVs. Therefore, we propose Reinforcement learning (RL)-based demand-side energy management using a rechargeable battery (RB) for enhanced cost-friendly privacy of EVs, efficient scheduling, and accurate billing. With existing Q-Learning-based RL (using <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\epsilon $ </tex-math></inline-formula> -greedy exploration and exploitation), we find that the reward maximization of efficient and private scheduling is often sluggish and incurs convergence issues. Therefore, we develop a genetic algorithm (GA)-based exploration and exploitation, which solves the convergence problems. We develop theoretical analysis and implement numerical results to demonstrate that the proposed GA-based RL framework accelerates convergence and enhances cost-friendly privacy considerably.

Self-learning swimming of a three-disk microrobot in a viscous and stochastic environment using reinforcement learning

Artificial Intelligence (AI) is a novel and fascinating tool in the control of robots. In this study, using a reinforcement learning approach, we have trained an artificial three-disk microrobot in order to learn how to swim at low Reynolds number flow (Re≪1) like the behavior of young microorganisms. Here, as the first contribution, we introduced a discrete model of a novel microswimmer. As the second contribution, we demonstrated that, using the proposed intelligent method, the microswimmer will be able to not only escape the physical constraints of swimming in a viscous medium, but also find the best sequence of actuation to reach the target point in a stochastic environment. The proposed reinforcement learning strategy can be applied to any other type of self-propelled microswimmers consisting of disks and actuators, which swim with continuous state space in a perturbed environment. In the end, we investigated the effect of learning hyperparameters on the performance of the swimmer’s training, and showed that, in the current agent and environment, the optimal values for the discount factor γ and learning rate α are 0.7 and 0.001 respectively which should be properly set for other types of mechanisms and environments.

Development of improved reinforcement learning smart charging strategy for electric vehicle fleet

Due to its environmental and energy sustainability, electric vehicles (EV) have emerged as the preferred option in the current transportation system. Uncontrolled EV charging, however, can raise consumers; charging costs and overwhelm the grid. Smart charging coordination systems are required to prevent the grid overload caused by charging too many electric vehicles at once. In light of the baseload that is present in the power grid, this research suggests an improved reinforcement learning charging management system. An optimization method, however, requires some knowledge in advance, such as the time the vehicle departs and how much energy it will need when it arrives at the charging station. Therefore, under realistic operating conditions, our improved Reinforcement Learning method with Double Deep Q-learning approach provides an adjustable, scalable, and flexible strategy for an electric car fleet. Our proposed approach provides fair value which solves the overestimation action value problem in deep Q-learning. Then, a number of different charging strategies are compared to the Reinforcement Learning algorithm. The proposed Reinforcement Learning technique minimizes the variance of the overall load by 68 % when compared to an uncontrolled charging strategy.

Asset correlation based deep reinforcement learning for the portfolio selection

Portfolio selection is an important application of AI in the financial field, which has attracted considerable attention from academia and industry alike. One of the great challenges in this application is modeling the correlation among assets in the portfolio. However, current studies cannot deal well with this challenge because it is difficult to analyze complex nonlinearity in the correlation. This paper proposes a policy network that models the nonlinear correlation by utilizing the self-attention mechanism to better tackle this issue. In addition, a deterministic policy gradient recurrent reinforcement learning method based on Monte Carlo sampling is constructed with the objective function of cumulative return to train the policy network. In most existing reinforcement learning-based studies, the state transition probability is generally regarded as unknown, so the value function of the policy can only be estimated. Based on financial backtest experiments, we analyze that the state transition probability is known in the portfolio, and value function can be directly obtained by sampling, further theoretically proving the optimality of the proposed reinforcement learning method in the portfolio. Finally, the superiority and generality of our approach are demonstrated through comprehensive experiments on the cryptocurrency dataset, S&P 500 stock dataset, and ETF dataset.

Real-Time Pricing Method for Spot Cloud Services with Non-Stationary Excess Capacity

Cloud operators face massive unused excess computing capacity with a stochastic non-stationary nature due to time-varying resource utilization with peaks and troughs. Low-priority spot (pre-emptive) cloud services with real-time pricing have been launched by many cloud operators, which allow them to maximize excess capacity revenue while keeping the right to reclaim capacities when resource scarcity occurs. However, real-time spot pricing with the non-stationarity of excess capacity has two challenges: (1) it faces incomplete peak–trough and pattern shifts in excess capacity, and (2) it suffers time and space inefficiency in optimal spot pricing policy, which needs to search over the large space of history-dependent policies in a non-stationary state. Our objective was to develop a real-time pricing method with a spot pricing scheme to maximize expected cumulative revenue under a non-stationary state. We first formulated the real-time spot pricing problem as a non-stationary Markov decision process. We then developed an improved reinforcement learning algorithm to obtain the optimal solution for real-time pricing problems. Our simulation experiments demonstrate that the profitability of the proposed reinforcement learning algorithm outperforms that of existing solutions. Our study provides both efficient optimization algorithms and valuable insights into cloud operators’ excess capacity management practices.

Reinforcement Learning-Based Pricing and Incentive Strategy for Demand Response in Smart Grids

International agreements support the modernization of electricity networks and renewable energy resources (RES). However, these RES affect market prices due to resource variability (e.g., solar). Among the alternatives, Demand Response (DR) is presented as a tool to improve the balance between electricity supply and demand by adapting consumption to available production. In this sense, this work focuses on developing a DR model that combines price and incentive-based demand response models (P-B and I-B) to efficiently manage consumer demand with data from a real San Juan—Argentina distribution network. In addition, a price scheme is proposed in real time and by the time of use in relation to the consumers’ influence in the peak demand of the system. The proposed schemes increase load factor and improve demand displacement compared to a demand response reference model. In addition, the proposed reinforcement learning model improves short-term and long-term price search. Finally, a description and formulation of the market where the work was implemented is presented.

Reinforcement learning based-adaptive tracking control for a class of semi-markov non-Lipschitz uncertain system with unmatched disturbances

A reinforcement learning based-adaptive tracking control method has been proposed for a class of semi-markov non-Lipschitz uncertain system with unmatched disturbances. Not only does the considered system contain unknown nonlinear functions and multisource disturbances, but parameter matrix in the system varies with the semi-Markov progress, which create difficulties in the design of the controller. Firstly, to handle the non-Lipschitz nonlinear terms, an adaptive control law based on reinforcement learning has been designed. Secondly, according to the reinforcement learning result, an adaptive disturbance observer has been constructed to offset the adverse influence generated by the multiple disturbances in the considered system. Finally, aiming at the parameter matrixes in the system varying with semi-Markov progress, a novel adaptive tracking controller integrating the reinforcement learning approximation, the adaptive disturbance observer and the tracking control law, has been proposed. With the given adaptive tracking control law based on reinforcement learning, through the Lyapunov direct method, it has been proven that the closed-loop system are stable and convergent and the tracking errors are ultimately uniformly bounded. In addition, several numerical experiments are provided to illustrate the feasibility and effectiveness of the proposed reinforcement learning based adaptive tracking control law.

The reinforcement learning model with heterogeneous learning rate in activity-driven networks

Agent’s learning behavior usually presents biased judgments influenced by many internal and external reasons, we incorporate an improved [Formula: see text]-learning algorithm in the reinforcement learning which is examined with the prisoner’s dilemma game in an activity-driven networks. The heterogeneous learning rate and [Formula: see text]-greedy exploration mechanism are taken into account while modeling decision-making of agents. Simulation results show the proposed reinforcement learning mechanism is conducive to the emergence of defective behavior, i.e. it could maximize one’s expected payoff regardless of its neighbors’ strategy. In addition, we find the temptation gain, vision level and the number of connected edges of activated agents are proportional to the density of defectors. Interestingly, when the inherent learning rate is small, the increase of exploration rate can demote the appearance of defectors, and the decrease of defectors is insignificant by increasing of exploration rate conversely.

Deep Reinforcement Learning Based Resource Allocation Approach for Wireless Networks Considering Network Slicing Paradigm

In this paper, we present an approach for resource scheduling in wireless networks based on the Network Slicing (NS) paradigm using Double Deep Q-Network (DDQN) Reinforcement Learning (RL) algorithm. More specifically, we propose a joint power and Scheduling Block (SB) allocation algorithm for networks with NS. The reinforcement learning algorithm applied to the resource allocation problem is formulated using state transitions regarding the system dynamics. We also present an algorithm, namely Network Slicing based on Reinforcement Learning (NSRL) that combines the proposed reinforcement learning based resource allocation with an approach based on reservation and sharing of resources among the slices where each RL agent acts in one slice. Simulations are carried out considering User Equipments (UEs) within a small cell coverage area - (Small Cells) with different Modulation and Coding Schemes (MCS) standardized by the 3rd Generation Partnership Project (3GPP) based on a simplified NS scenario with fifth generation wireless network (5G) characteristics. In the simulations, two slices are considered for the UEs: one considering Ultra-reliable and Low Latency Communications (URLLC) and other related to enhanced Mobile Broadband (eMBB) services. Simulation results show that the NSRL algorithm efficiently allocates power and SBs, outperforming other algorithms in the literature.

An actor-critic framework based on deep reinforcement learning for addressing flexible job shop scheduling problems.

With the rise of Industry 4.0, manufacturing is shifting towards customization and flexibility, presenting new challenges to meet rapidly evolving market and customer needs. To address these challenges, this paper suggests a novel approach to address flexible job shop scheduling problems (FJSPs) through reinforcement learning (RL). This method utilizes an actor-critic architecture that merges value-based and policy-based approaches. The actor generates deterministic policies, while the critic evaluates policies and guides the actor to achieve the most optimal policy. To construct the Markov decision process, a comprehensive feature set was utilized to accurately represent the system's state, and eight sets of actions were designed, inspired by traditional scheduling rules. The formulation of rewards indirectly measures the effectiveness of actions, promoting strategies that minimize job completion times and enhance adherence to scheduling constraints. The experimental evaluation conducted a thorough assessment of the proposed reinforcement learning framework through simulations on standard FJSP benchmarks, comparing the proposed method against several well-known heuristic scheduling rules, related RL algorithms and intelligent algorithms. The results indicate that the proposed method consistently outperforms traditional approaches and exhibits exceptional adaptability and efficiency, particularly in large-scale datasets.

Robust state and output feedback prescribed performance interval type‐3 fuzzy reinforcement learning controller for an unmanned aerial vehicle with actuator saturation

AbstractThis paper presents a novel adaptive reinforcement learning control method with interval type‐3 fuzzy neural networks to improve the trajectory tracking control performance of quadrotor unmanned aerial vehicles in challenging flight conditions. The proposed reinforcement learning controller is independent of the system's dynamics, and only relies on measurable signals of the system. An adaptive robust controller in collaboration with the suggested reinforcement learning method is designed to significantly improve the robustness of the control system. The maximum overshoot/undershoot, convergence rate and final tracking accuracy are ensured a priori by the prescribed performance control methodology. To develop the proposed controller and to achieve a high‐performance closed‐loop system, a high‐gain observer is employed in order to estimate the velocity and acceleration of the quadrotor unmanned aerial vehicles system. The uniform ultimate boundedness stability of the proposed control algorithm is achieved by a Lyapunov‐based stability analysis. Finally, in the simulation section, it is shown that the presented intelligent controller with the proposed learning algorithm result in a better performance in contrast to the other kind of conventional control techniques.