Proposed Reinforcement Learning Algorithm Research Articles

A reconfigurable RF front-end for 5G direct sampling receivers with an optimized calibration scheme

This paper presents a reconfigurable radio frequency front-end (RFFE) tailored for direct RF sampling receivers operating within Frequency Range 1 (FR-1) of the 5G spectrum. It consists of a balun-LNA, a noise-cancelling, current-reuse, Q-enhanced filter, and a programmable gain amplifier (PGA). Fabricated in 22-nm FD-SOI technology, the RFFE covers the entire frequency range from 1.7 to 6.4 GHz with a tunable bandwidth ranging from 50 MHz to 1.2 GHz. The experimental results confirm that the RFFE achieves 3.9 dB NF, 55 dB ultimate OOB rejection, −4 dBm IB-IIP3, and 23 dBm OOB-IIP3. Furthermore, a digital calibration scheme is proposed to compensate for process, voltage, and temperature (PVT) variations, achieving lower than 2 % error in 6.61 µs on average. Subsequently, a proximal policy optimization (PPO) agent is employed to choose the optimal policy for the successive adjustments of quality factors and resonance frequencies of the filter’s resonators. As a result, the proposed reinforcement learning algorithm reduces the calibration’s convergence time by 59 %.

An iterative gradient descent-based reinforcement learning policy for active control of structural vibrations

A contemporary policy gradient-based optimization scheme is presented for active structural control by exerting the concept of reinforcement learning (RL). The RL-based control algorithm is demonstrated both in proportional (P) state, and proportional-integral (PI) state-output feedback approaches, wherein, the latter strategy is deliberated based on the theory of servo-mechanism. The search for optimal P and PI controller parameters in a training sequence is attained by engaging an efficient gradient descent-based optimization strategy. The utilization of gradient-based sequence within the RL framework accelerates the learning protocol, ensuring effective dissipation of structural energy, and achieving suboptimal control. The proposed algorithms are validated through numerical experiments on two different structural systems: (i) a quarter car model subjected to periodic road excitation, having a single actuator and presented in continuous time, and (ii) an 8-story building model subjected to random seismic excitation, having multiple actuators and presented in discrete time. Practical implementation concerns of control strategies are thoroughly investigated by considering perturbations in model parameters and input forces. The findings from this study indicate that the RL-based P and PI controllers exhibit high stabilizing performance, and are applicable in both analog and digital domains. Finally, it has affirmed that the proposed RL algorithms exhibit relatively low computational complexity in real-time, and thus, open up a wide range of perspectives for its application in large-scale complex structures.

Reconfigurable tolerant control of nonlinear Euler–Lagrange systems under actuator fault: A reinforcement learning-based fixed-time approach

This paper presents a novel fixed-time adaptive Fault Tolerant Control (FTC) framework for MIMO nonlinear Euler-Lagrange systems using sliding mode-based strategy, reinforcement learning (RL) and fixed-time disturbance observer. The primary objective is to enhance system reliability in the presence of actuator faults, uncertainties and disturbances. The proposed RL algorithm incorporates an actor-critic neural network (NN), where the actor NN estimates the uncertainty and the critic NN evaluates the performance cost function. Additionally, a fixed-time adaptive observer is designed to estimate the lumped term of faults and disturbances. To achieve high precision trajectory tracking within a fixed-time interval, a nonsingular fast terminal sliding mode scheme is designed. This scheme ensures fixed-time convergence of the tracking error and facilitates disturbance attenuation and fault mitigation, which are key features of the proposed fixed-time secure control strategy. Furthermore, the closed-loop system's fixed-time stability is analyzed using Lyapunov theory. Experimental results demonstrate the effectiveness of the proposed FTC framework in mitigating the adverse effects of faults, uncertainties and disturbances, thereby enhancing system performance and reliability.

Robust safe reinforcement learning control of unknown continuous-time nonlinear systems with state constraints and disturbances

Industrial processes must operate safely and optimally in practice, which typically entails solving a constrained optimal control (COC) problem. Finding a control policy for such a problem, however, is challenging especially when the system dynamics are unknown. In this paper, a novel safe reinforcement learning (RL) algorithm is proposed to deal with the COC problem for the continuous-time nonlinear system with unknown dynamics and disturbances. The presented method can ensure that the system operates within a predefined safe region in the presence of system model uncertainties and external disturbances, which goes beyond the results of typical RL methods. To handle the system state constraints, the problem is first transformed into an unconstrained one via the proposed data-driven slack function approach. Then an improved model-based RL method is devised to learn a near-optimal and safe control policy in real-time. Finally, a robust compensator and the composite neural-network updating rule are designed to eliminate the influence of disturbances and uncertainties. Theoretical analysis based on the Lyapunov approach is conducted to prove the closed-loop system stability and the algorithm convergence, which further guarantees the satisfaction of safety constraints. The effectiveness of the proposed RL algorithm is verified through simulations.

A Reinforcement Learning Framework for Vehicular Network Routing Under Peak and Average Constraints

Providing provable performance guarantees in vehicular network routing problems is crucial to ensure safely and timely delivery of information in an environment characterized by high mobility, dynamic network conditions, and frequent topology changes. While Reinforcement Learning (RL) has shown great promise in network routing, existing RL-based solutions typically support decision-making with either peak constraints or average constraints, but not both. For network routing in intelligent transportation, such as advanced vehicle control and safety, both peak constraints (e.g., maximum latency or minimum bandwidth guarantees) and average constraints (e.g., average transmit power or data rate constraints) must be satisfied. In this paper, we propose a holistic framework for RL-based vehicular network routing, which maximizes routing decisions under both average and peak constraints. The routing problem is modeled as a Constrained Markov Decision Process and recast into an optimization based on Constraint Satisfaction Problems (CSPs). We prove that the optimal policy of a given CSP can be learned by an extended Q-learning algorithm while satisfying both peak and average latency constraints. To improve the scalability of our framework, we further turn it into a decentralized implementation through a cluster-based learning structure. Applying the proposed RL algorithm to vehicular network routing problems under both peak and average latency constraints, simulation results show that our algorithm achieves much higher rewards than heuristic baselines with over 40% improvement in average transmission rate, while resulting in zero violation in terms of both peak and average constraints.

Reinforcement Learning with Dual Safety Policies for Energy Savings in Building Energy Systems

Reinforcement learning (RL) is being gradually applied in the control of heating, ventilation and air-conditioning (HVAC) systems to learn the optimal control sequences for energy savings. However, due to the “trial and error” issue, the output sequences of RL may cause potential operational safety issues when RL is applied in real systems. To solve those problems, an RL algorithm with dual safety policies for energy savings in HVAC systems is proposed. In the proposed dual safety policies, the implicit safety policy is a part of the RL model, which integrates safety into the optimization target of RL, by adding penalties in reward for actions that exceed the safety constraints. In explicit safety policy, an online safety classifier is built to filter the actions outputted by RL; thus, only those actions that are classified as safe and have the highest benefits will be finally selected. In this way, the safety of controlled HVAC systems running with proposed RL algorithms can be effectively satisfied while reducing the energy consumptions. To verify the proposed algorithm, we implemented the control algorithm in a real existing commercial building. After a certain period of self-studying, the energy consumption of HVAC had been reduced by more than 15.02% compared to the proportional–integral–derivative (PID) control. Meanwhile, compared to the independent application of the RL algorithm without safety policy, the proportion of indoor temperature not meeting the demand is reduced by 25.06%.

Real-Time Pricing Method for Spot Cloud Services with Non-Stationary Excess Capacity

Cloud operators face massive unused excess computing capacity with a stochastic non-stationary nature due to time-varying resource utilization with peaks and troughs. Low-priority spot (pre-emptive) cloud services with real-time pricing have been launched by many cloud operators, which allow them to maximize excess capacity revenue while keeping the right to reclaim capacities when resource scarcity occurs. However, real-time spot pricing with the non-stationarity of excess capacity has two challenges: (1) it faces incomplete peak–trough and pattern shifts in excess capacity, and (2) it suffers time and space inefficiency in optimal spot pricing policy, which needs to search over the large space of history-dependent policies in a non-stationary state. Our objective was to develop a real-time pricing method with a spot pricing scheme to maximize expected cumulative revenue under a non-stationary state. We first formulated the real-time spot pricing problem as a non-stationary Markov decision process. We then developed an improved reinforcement learning algorithm to obtain the optimal solution for real-time pricing problems. Our simulation experiments demonstrate that the profitability of the proposed reinforcement learning algorithm outperforms that of existing solutions. Our study provides both efficient optimization algorithms and valuable insights into cloud operators’ excess capacity management practices.

Deep Reinforcement Learning-Based Joint Optimization Control of Indoor Temperature and Relative Humidity in Office Buildings

Indoor temperature and relative humidity control in office buildings is crucial, which can affect thermal comfort, work efficiency, and even health of the occupants. In China, fan coil units (FCUs) are widely used as air-conditioning equipment in office buildings. Currently, conventional FCU control methods often ignore the impact of indoor relative humidity on building occupants by focusing only on indoor temperature as a single control object. This study used FCUs with a fresh-air system in an office building in Beijing as the research object and proposed a deep reinforcement learning (RL) control algorithm to adjust the air supply volume for the FCUs. To improve the joint control satisfaction rate of indoor temperature and relative humidity, the proposed RL algorithm adopted the deep Q-network algorithm. To train the RL algorithm, a detailed simulation environment model was established in the Transient System Simulation Tool (TRNSYS), including a building model and FCUs with a fresh-air system model. The simulation environment model can interact with the RL agent in real time through a self-developed TRNSYS–Python co-simulation platform. The RL algorithm was trained, tested, and evaluated based on the simulation environment model. The results indicate that compared with the traditional on/off and rule-based controllers, the RL algorithm proposed in this study can increase the joint control satisfaction rate of indoor temperature and relative humidity by 12.66% and 9.5%, respectively. This study provides preliminary direction for a deep reinforcement learning control strategy for indoor temperature and relative humidity in office building heating, ventilation, and air-conditioning (HVAC) systems.

Robotic Peg-in-Hole Assembly Strategy Research Based on Reinforcement Learning Algorithm

To improve the robotic assembly effects in unstructured environments, a reinforcement learning (RL) algorithm is introduced to realize a variable admittance control. In this article, the mechanisms of a peg-in-hole assembly task and admittance model are first analyzed to guide the control strategy and experimental parameters design. Then, the admittance parameter identification process is defined as the Markov decision process (MDP) problem and solved with the RL algorithm. Furthermore, a fuzzy reward system is established to evaluate the action–state value to solve the complex reward establishment problem, where the fuzzy reward includes a process reward and a failure punishment. Finally, four sets of experiments are carried out, including assembly experiments based on the position control, fuzzy control, and RL algorithm. The necessity of compliance control is demonstrated in the first experiment. The advantages of the proposed algorithms are validated by comparing them with different experimental results. Moreover, the generalization ability of the RL algorithm is tested in the last two experiments. The results indicate that the proposed RL algorithm effectively improves the robotic compliance assembly ability.

Mitigating the risk of musculoskeletal disorders during human robot collaboration: a reinforcement learning approach

Work-related musculoskeletal disorders (MSDs) are often observed in human-robot collaboration (HRC), a common work configuration in modern factories. In this study, we aim to reduce the risk of MSDs in HRC scenarios by developing a novel model-free reinforcement learning (RL) method to improve workers’ postures. Our approach follows two steps: first, we adopt a 3D human skeleton reconstruction method to calculate workers’ Rapid Upper Limb Assessment (RULA) scores; next, we devise an online gradient-based RL algorithm to dynamically improve the RULA score. Compared with previous model-based studies, the key appeals of the proposed RL algorithm are two-fold: (i) the model-free structure allows it to “learn” the optimal worker postures without need any specific biomechanical models of tasks or workers, and (ii) the data-driven nature makes it accustomed to arbitrary users by providing personalized work configurations. Results of our experiments confirm that the proposed method can significantly improve the workers’ postures.

Optimized Backstepping Tracking Control Using Reinforcement Learning for Quadrotor Unmanned Aerial Vehicle System

In this article, an optimized tracking control scheme is studied for the quadrotor unmanned aerial vehicle (QUAV) system by combining both reinforcement learning (RL) and the backstepping technique. The RL aims to overcome the difficulty coming from solving the Hamilton–Jacobi–Bellman (HJB) equation, and it is performed via iterating both critic and actor each other, where the critic is for improving the control performance and the actor is for executing the control behavior. In mathematics, a QUAV system is composed of two connected subsystems that are, respectively, modeled by the translational and rotational dynamic equations, which are coupled via a rotation matrix; hence, the optimized tracking scheme is composed of two interconnected individual controls corresponding to the position and attitude, respectively. To achieve the two optimized position and attitude controls, the RL is constructed on the basis of the neural network (NN) approximation of the HJB equation’s solution by utilizing NN’s outstanding function approximation ability. Particularly, the position control is accomplished by introducing an intermediate control because the translational dynamic is an underactuated system. Since the proposed RL algorithm is significantly simple in comparison with the published methods, the optimized QUAV control can be easily executed in practical applications. Finally, the results are demonstrated by a Lyapunov stability analysis and a numerical simulation.

Security State Estimation for Cyber-Physical Systems against DoS Attacks via Reinforcement Learning and Game Theory

This paper addressed the optimal policy selection problem of attacker and sensor in cyber-physical systems (CPSs) under denial of service (DoS) attacks. Since the sensor and the attacker have opposite goals, a two-player zero-sum game is introduced to describe the game between the sensor and the attacker, and the Nash equilibrium strategies are studied to obtain the optimal actions. In order to effectively evaluate and quantify the gains, a reinforcement learning algorithm is proposed to dynamically adjust the corresponding strategies. Furthermore, security state estimation is introduced to evaluate the impact of offensive and defensive strategies on CPSs. In the algorithm, the ε-greedy policy is improved to make optimal choices based on sufficient learning, achieving a balance of exploration and exploitation. It is worth noting that the channel reliability factor is considered in order to study CPSs with multiple reasons for packet loss. The reinforcement learning algorithm is designed in two scenarios: reliable channel (that is, the reason for packet loss is only DoS attacks) and unreliable channel (the reason for packet loss is not entirely from DoS attacks). The simulation results of the two scenarios show that the proposed reinforcement learning algorithm can quickly converge to the Nash equilibrium policies of both sides, proving the availability and effectiveness of the algorithm.

Dynamic Multichannel Access via Multi-Agent Reinforcement Learning: Throughput and Fairness Guarantees

We consider a multichannel random access system in which each user accesses a single channel at each time slot to communicate with an access point (AP). Users arrive to the system at random and be activated for a certain period of time slots and then disappear from the system. Under such dynamic network environment, we propose a distributed multichannel access protocol based on multi-agent reinforcement learning (RL) to improve both throughput and fairness between active users. Unlike the previous approaches adjusting channel access probabilities at each time slot, the proposed RL algorithm deterministically selects a set of channel access policies for several consecutive time slots. To effectively reduce the complexity of the proposed RL algorithm, we adopt a branching dueling Q-network architecture and propose an efficient training methodology for producing proper Q-values over time-varying user sets. We perform extensive simulations on realistic traffic environments and demonstrate that the proposed online learning improves both throughput and fairness compared to the conventional RL approaches and centralized scheduling policies.

Image quality assessment by overlapping task-specific and task-agnostic measures: application to prostate multiparametric MR images for cancer segmentation

Image quality assessment (IQA) in medical imaging can be used to ensure that downstream clinical tasks can be reliably performed. Quantifying the impact of an image on the specific target tasks, also named as task amenability, is needed. A <em>task-specific</em> IQA has recently been proposed to learn an image-amenability-predicting controller simultaneously with a target task predictor. This allows for the trained IQA controller to measure the impact an image has on the target task performance, when this task is performed using the predictor, e.g. segmentation and classification neural networks in modern clinical applications. In this work, we propose an extension to this task-specific IQA approach, by adding a <em>task-agnostic</em> IQA based on auto-encoding as the target task. Analysing the intersection between low-quality images, deemed by both the task-specific and task-agnostic IQA, may help to differentiate the underpinning factors that caused the poor target task performance. For example, common imaging artefacts may not adversely affect the target task, which would lead to a low task-agnostic quality and a high task-specific quality, whilst individual cases considered clinically challenging, which can not be improved by better imaging equipment or protocols, is likely to result in a high task-agnostic quality but a low task-specific quality. We first describe a flexible reward shaping strategy which allows for the adjustment of weighting between task-agnostic and task-specific quality scoring. Furthermore, we evaluate the proposed reinforcement learning algorithm, using a clinically challenging target task of prostate tumour segmentation on multiparametric magnetic resonance (mpMR) images. Based on experimental results using mpMR images from 850 patients, it was found that <em>a</em>) The task-agnostic IQA may identify artefacts, but with limited impact on the accuracy of cancer segmentation networks. A Dice score of 0.367±0.017 was obtained after rejecting 10% of low quality images, compared to 0.354±0.016 from a non-selective baseline; <em>b</em>} The task-specific IQA alone improved the performance to 0.415±0.020, at the same rejection ratio. However, this system indeed rejected both images that impact task performance due to imaging defects and due to being clinically challenging; and <em>c</em>) The proposed reward shaping strategy, when the task-agnostic and task-specific IQA are weighted appropriately, successfully identified samples that need re-acquisition due to defected imaging process, as opposed to clinically challenging cases due to low contrast in pathological tissues or other equivocacy in radiological presentation.<br>Our code is available at https://github.com/s-sd/task-amenability/tree/v1

Optimal Resource Allocation Considering Non-Uniform Spatial Traffic Distribution in Ultra-Dense Networks: A Multi-Agent Reinforcement Learning Approach

Recently, the demand for small cell base stations (SBSs) has been exploding to accommodate the explosive increase in mobile data traffic. In ultra-dense small cell networks (UDSCNs), because the spatial and temporal traffic distributions are significantly disproportionate, the efficient management of the energy consumption of SBSs is crucial. Therefore, we herein propose a multi-agent distributed Q-learning algorithm that maximizes energy efficiency (EE) while minimizing the number of outage users. Through intensive simulations, we demonstrate that the proposed algorithm outperforms conventional algorithms in terms of EE and the number of outage users. Even though the proposed reinforcement learning algorithm has significantly lower computational complexity than the centralized approach, it is shown that it can converge to the optimal solution.

Omnidirectional-Wheel Conveyor Path Planning and Sorting Using Reinforcement Learning Algorithms

In this paper, path planning and sorting of packages for Omnidirectional-Wheel conveyor are presented using Reinforcement Learning (RL). Q-learning, Double Q-learning, Deep Q-learning, and the Double Deep Q-learning algorithms are investigated. The RL algorithms enable the conveyor to self-learn the packages path and sort them without using conventional control or path planning theories. The RL algorithms are used for two different case studies on conveyors structures with different numbers of cells to compare and evaluate their performances in large- and small-scale sized structures. To explore the proposed methods response to external environment effects, two types of collisions between multiple packages were considered, the proposed RL algorithms showed their ability to resolve both types successfully. Comparative study between multiple RL algorithms for path planning showed that the Q-learning and Double Q-learning algorithms had outperformed their Deep learning versions for path planning in the two case studies. Furthermore, the proposed RL methods are compared experimentally to classic control and path planning theories using a hardware prototype for one of the presented case studies. The hardware experimental results showed that the proposed RL methods were as successful as the conventional methods in path planning and sorting in much less processing time. Two types of sorting scenarios (Type I and II) were tested for same package type and for multiple ones. For Type I sorting the Q-learning algorithm performed better than the Q-learning with weights approach, achieving better mean and minimum rewards while maximum rewards remain the same for both techniques. As for Type II sorting, only the Q-learning with weights approach was able to achieve it and converge in a reasonable time.

Deep Deterministic Policy Gradient to Regulate Feedback Control Systems Using Reinforcement Learning

Controlling feedback control systems in continuous action spaces has always been a challenging problem. Nevertheless, reinforcement learning is mainly an area of artificial intelligence (AI) because it has been used in process control for more than a decade. However, the existing algorithms are unable to provide satisfactory results. Therefore, this research uses a reinforcement learning (RL) algorithm to manage the control system. We propose an adaptive speed control of the motor system based on depth deterministic strategy gradient (DDPG). The actor-critic scenario using DDPG is implemented to build the RL agent. In addition, a framework has been created for traditional feedback control systems to make RL implementation easier for control systems. The RL algorithms are robust and proficient in using trial and error to search for the best strategy. Our proposed algorithm is a deep deterministic policy gradient, in which a large amount of training data trains the agent. Once the system is trained, the agent can automatically adjust the control parameters. The algorithm has been developed using Python 3.6 and the simulation results are evaluated in the MATLAB/Simulink environment. The performance of the proposed RL algorithm is compared with a proportional integral derivative (PID) controller and a linear quadratic regulator (LQR) controller. The simulation results of the proposed scheme are promising for the feedback control problems.

Optimizing Packet Forwarding Performance in Multiband Relay Networks via Customized Reinforcement Learning

In next-generation wireless networks, relay-based packet forwarding, emerged as an appealing technique to extend network coverage while maintaining the required service quality. The incorporation of multiple frequency bands, ranging from MHz/GHz to THz frequencies, and their opportunistic and/or simultaneous exploitation by relay nodes can significantly improve system capacity, however at the risk of increased packet latency. Since a relay node can use different bands to send and receive packets, there is a pressing need to design an efficient channel allocation algorithm without a central oracle. While existing greedy heuristics and game-theoretic techniques, which were developed for multi-band channel assignment to relay nodes, achieve minimum packet latency, their performance drops significantly when network dynamism (i.e., user mobility, non-quasi-static channel conditions) is introduced. Since this problem involves multiple relay nodes, we model it as a Markov Decision Process (MDP) involving various stages, which essentially means that achieving an optimal and stable solution is a computationally hard problem. Since solving the MDP, traditionally, consumes a great deal of time and is intractable for relay nodes, we explore how to approximate the optimal solution in a distributed manner by reformulating a reinforcement learning-based, smart channel adaptation problem in the considered multi-band relay network. By customizing a Q-Learning algorithm that adopts an epsilon-greedy policy, we can solve this re-formulated reinforcement learning problem. Extensive computer-based simulation results demonstrate that the proposed reinforcement learning algorithm outperforms the existing methods in terms of transmission time, buffer overflow, and effective throughput. We also provide the convergence analysis of the proposed model by systematically finding and setting the appropriate parameters.

A New Covid-19 Tracing Approach using Machine Learning and Drones Enabled Wireless Network

<p class="0abstract">The continuous advancements in wireless network systems have reshaped the healthcare systems towards using emerging communication technologies at different levels. This paper makes two major contributions. Firstly, a new monitoring and tracking wireless system is developed to handle the COVID-19 spread problem. Unmanned aerial vehicles (UAVs), i.e., drones, are used as base stations as well as data collection points from Internet of Things (IoT) devices on the ground. These UAVs are also able to exchange data with other UAVs and cloud servers. Secondly, this paper introduces a new reinforcement learning (RL) framework for learning the optimal signal-aware UAV trajectories under quality of service constraints. The proposed RL algorithm is instrumental in making the UAV movement decisions that maximize the signal power at the receiver and the data collected from the ground agents. Simulation experiments confirm that the system overcomes conventional wireless monitoring systems and demonstrates efficiency especially in terms of flexible continues connectivity, line-of sight visibility, and collision avoidance.</p>

Design of model-free reinforcement learning control for tunable vibration absorber system based on magnetorheological elastomer

This paper proposes a novel model-free reinforcement learning (RL) for vibration control of a magnetorheological elastomer (MRE)-based application. Because the modeling of the MRE stiffness is nonlinear and time-varying depending on various environmental factors, this paper approached the MRE control issue via a model-free learning based method. In this study, an RL model is designed for the MRE-based tunable vibration absorber (TVA) which can control the optimal stiffness of MRE to maximize the vibration suppression. The designed RL algorithm continuously learns and updates the optimal control input of the MRE stiffness adaptively to the dynamic environment without using any prior knowledge of the MRE modeling. From analyzing the mechanism of MRE TVA, the RL algorithm and parameters are carefully designed for a high vibration performance. Also, this study proposed several ideas to make the RL model simpler and converge at a high rate. The experiments confirmed that the proposed RL model showed a rapid convergence to the optimal policy which could minimize the vibration level with respect to the dynamic excitation disturbance. Results showed that the RL model had a similar performance as the conventional tuning method and suppressed the vibration level as much as 57% compared to the one without the controller. Also, the proposed RL algorithm was able to estimate the actual dynamics of the MRE TVA by learning from the environment. Thus, this study showed the feasibility of implementing a model-free RL model to realize an adaptive controller for applications based on highly nonlinear MRE.