Reinforcement Learning-based Scheme Research Articles

Enhancing Production in Robot-Enabled Manufacturing Systems: A Dynamic Model and Moving Horizon Control Strategy for Mobile Robot Assignment

Abstract This paper presents a dynamic mathematical model for a robot-enabled manufacturing system, where mobile robots independently manage workstation tasks. Each robot possesses one or multiple skills, enabling collaborative work at workstations. A real-time robot assignment problem is formulated to maximize production of the system, and a novel control strategy is developed to address this problem. Leveraging system properties derived from the model and moving window downtime prediction, the problem of maximizing system production is transformed into a more tractable control problem focused on identifying and achieving a defined ideal clean configurations. The proposed solution significantly outperforms various benchmarks, including a pure reinforcement learning-based strategy, underscoring the importance of system understanding and its crucial role in enhancing flexibility and productivity in manufacturing systems.

Vehicle Collaborative Partial Offloading Strategy in Vehicular Edge Computing

Vehicular Edge Computing (VEC) is a crucial application of Mobile Edge Computing (MEC) in vehicular networks. In VEC networks, the computation tasks of vehicle terminals (VTs) can be offloaded to nearby MEC servers, overcoming the limitations of VTs’ processing power and reducing latency caused by distant cloud communication. However, a mismatch between VTs’ demanding tasks and MEC servers’ limited resources can overload MEC servers, impacting Quality of Service (QoS) for computationally intensive tasks. Additionally, vehicle mobility can disrupt communication with static MEC servers, further affecting VTs’ QoS. To address these challenges, this paper proposes a vehicle collaborative partial computation offloading model. This model allows VTs to offload tasks to two types of service nodes: collaborative vehicles and MEC servers. Factors like a vehicle’s mobility, remaining battery power, and available computational power are also considered when evaluating its suitability for collaborative offloading. Furthermore, we design a deep reinforcement learning-based strategy for collaborative partial computation offloading that minimizes overall task delay while meeting individual latency constraints. Experimental results demonstrate that compared to traditional approaches without vehicle collaboration, this scheme significantly reduces latency and achieves a significant reduction (around 2%) in the failure rate under tighter latency constraints.

Personalizing renal replacement therapy initiation in the intensive care unit: a reinforcement learning-based strategy with external validation on the AKIKI randomized controlled trials.

The timely initiation of renal replacement therapy (RRT) for acute kidney injury (AKI) requires sequential decision-making tailored to individuals' evolving characteristics. To learn and validate optimal strategies for RRT initiation, we used reinforcement learning on clinical data from routine care and randomized controlled trials. We used the MIMIC-III database for development and AKIKI trials for validation. Participants were adult ICU patients with severe AKI receiving mechanical ventilation or catecholamine infusion. We used a doubly robust estimator to learn when to start RRT after the occurrence of severe AKI for three days in a row. We developed a "crude strategy" maximizing the population-level hospital-free days at day 60 (HFD60) and a "stringent strategy" recommending RRT when there is significant evidence of benefit for an individual. For validation, we evaluated the causal effects of implementing our learned strategies versus following current best practices on HFD60. We included 3748 patients in the development set and 1068 in the validation set. Through external validation, the crude and stringent strategies yielded an average difference of 13.7 [95% CI -5.3 to 35.7] and 14.9 [95% CI -3.2 to 39.2] HFD60, respectively, compared to current best practices. The stringent strategy led to initiating RRT within 3 days in 14% of patients versus 38% under best practices. Implementing our strategies could improve the average number of days that ICU patients spend alive and outside the hospital while sparing RRT for many. We developed and validated a practical and interpretable dynamic decision support system for RRT initiation in the ICU.

An improved spectrum sharing strategy evaluation over wireless network framework to perform error free communications

The possible application of wireless sensor networks is hampered and the widespread use of this novel method is slowed, according to recent surveys conducted within the field of automation in industry, which identified that accuracy pertains indicate currently among the primary obstacles to the dissemination of wireless networking for recognizing and regulating applications. In order to overcome these constraints, it is necessary to raise public understanding of the reasons for dependability issues and the potential approaches to resolving them. Low-power communications of sensor nodes are, in reality, quite susceptible to adverse channel conditions and can readily be affected by transmissions of other co-located devices, making them seem unreliable. In this dissertation, I explore several strategies that may be used to either eliminate interference altogether or reduce its negative consequences. In this paper, we study the creation and modeling of a brand-new spectrum allocation mechanism for wireless sensor networks. Cognitive radio technology can detect spectrum holes in the environment, learn from its surroundings using artificial intelligence, adjust the system’s operating parameters in real-time, and use the secondary spectrum to increase efficiency. In this study, we present a reinforcement learning-based strategy for choosing the power of transmission and frequency that can help individual sensors learn from their prior decisions and those of their peers. Our suggested approach is multiple agents decentralized and adaptable to both the data needs from source to sink and the amount of energy that sensing devices in the network have left over. In comparison to different resource allocation algorithms, the results reveal a dramatic increase in the lifespan of the network.

Energy management optimization for connected hybrid electric vehicle using offline reinforcement learning

Energy management strategy (EMS) is critical to ensure the long-term energy economy of hybrid electric vehicles. The classical deep reinforcement learning algorithms exist many issues such as the safety constraint and the simulation-to-real gap, resulting in difficulties in applications to industrial tasks. Thus, this paper proposes a novel EMS and a policy updating method based on the offline deep reinforcement learning algorithm to address the energy optimization problem. Firstly, the batch-constrained deep Q-learning algorithm is applied to provide a solution training the control strategy based on the existing datasets without interaction with the environment. Secondly, the EMS updating method is proposed to improve the adaptability under complicated driving cycles. The Jensen–Shannon divergence is introduced to determine when offline control strategy updates in real time. Finally, the optimality and effectiveness of the proposed EMS are validated, and the real-time and adaptive performance of the proposed method is also verified. The results indicate that the proposed EMS can learn a superior policy from fixed data, and the proposed updating method can utilize the real-time data to update the offline policy approaching the online deep reinforcement learning-based strategy in energy consumption.

A reinforcement learning-based strategy updating model for the cooperative evolution

A reinforcement learning-based strategy updating model for the cooperative evolution

SECHO: A deep reinforcement learning-based scheme for secure handover in mobile edge computing

The experience of users in Mobile Edge Computing (MEC) highly depends on data offloading whose major bottle neck is the design of handover scheme. Existing research advocates that offloading security and quality of service (QoS) should be both considered for an optimal handover. However, recent studies concentrate on improving QoS while neglect security. Due to the complexity of MEC scenarios, generating an optimal handover scheme improving both QoS and security remains a challenge. To solve this problem, we propose SECHO, a secure handover scheme for blockchain-based single-user mobile edge computing system. Then, we design a security policy selection algorithm of offloading tasks to guarantees security performance and controls overhead. Finally, we customize the neural network for a deep reinforcement learning algorithm to realize optimal handover, where a security policy selection method is engaged to collaboratively enhance offloading QoS while decreasing underlying risks. Simulation results show that SECHO leads a 29.6%–302% advantage in overall performance than existing approaches. It proves that our proposal is effective to achieve secure handover while not degrading offloading QoS.

RSSI Map-Based Trajectory Design for UGV Against Malicious Radio Source: A Reinforcement Learning Approach

Trajectory design is of great significance for the intelligent Unmanned Ground Vehicle (UGV) when performing various ground tasks. Though obstacle avoidance, speed control and other movement issues in the UGV navigation have been considered by the current research, the UGV path planning against malicious radio source is off the beaten path. To address such a research gap, we propose a reinforcement learning-based scheme to design UGV trajectory against malicious radio source as well as minimize the movement cost. Firstly, the malicious radio source detection and localization models are introduced after the Received Signal Strength Indicator (RSSI) map establishment. Then, the RSSI Map-based UGV trajectory design problem is formulated, where the movement cost and security risk are both concerned. To solve the formed problem, we propose a reinforcement learning-based trajectory design scheme, whose complexities are analyzed in detail. Finally, experiments are conducted under various parameter settings, where the simulation results evaluate the correctness and effectiveness of the proposed algorithm.

Reinforcement Learning based position control of shell-fetching manipulator with extreme random trees

The automatic loading system of artillery includes the gun-fetching manipulator, which is crucial. The stability and control accuracy of the manipulator, however, are comparatively subpar when following the position trajectory as a result of the changes in the settings of the permanent magnet synchronous motor. This research suggests a reinforcement learning-based strategy for controlling a gun manipulator’s motor behavior in light of the current scenario. In this algorithm, the state variable is the feedback output of the control variable, and the reward function is utilized to calculate the associated reward value. The reinforcement learning agent then makes decisions based on the environment’s state and reward value, adjusting the manipulator’s trajectory in real-time. By comparing different simulation results, the gun-taking manipulator can achieve more accurate position trajectory control based on reinforcement learning.

Transfer Deep Reinforcement Learning-Based Energy Management Strategy for Plug-In Hybrid Electric Heavy-Duty Trucks under Segmented Usage Scenarios

Energy management strategy (EMS) is a way to reduce the energy consumption of hybrid power systems. This article proposes a unique deep reinforcement learning- (DRL-) based EMS for plug-in hybrid electric heavy-duty trucks (PHETs), combining driving cycle pattern recognition (DPR) and deep transfer learning (DTL). The proposed EMS can cope well with the complex usage scenarios of PHETs and the difficulty of generating EMS. While ensuring the minimum overall driving cost, the strategy can improve the convergence speed of the DRL method and the generalizability under segmented usage scenarios. Firstly, representative driving cycles that reflect different usage scenarios are constructed based on a naturalistic data-driven method. Secondly, a plug-in hybrid electric heavy-duty truck (PHET) driving pattern recognizer based on a learning vector quantization neural network (LVQ) is built. Thirdly, the deep deterministic policy gradient (DDPG) algorithm is innovatively combined with the DTL algorithm. The pretrained neural network in the corresponding usage scenarios is transferred to the natural driving cycles based on DTL. Moreover, the proposed EMS gives an emphasized consciousness on the battery degradation cost. Finally, the strategy is tested under natural driving cycles in different usage scenarios and proven through comparison with the current state-of-the-art techniques, deep reinforcement learning-based strategy, and dynamic programming (DP). The results show that the proposed strategy outperforms existing cutting-edge deep reinforcement learning techniques in terms of convergence speed, battery life extension, fuel consumption, and overall driving cost reduction. The proposed control strategy can improve the convergence speed by nearly 50%, while effectively extending the battery life and reducing the overall driving cost compared to the existing state-of-the-art strategies. The battery degradation rate is reduced by 48.46%, 57.95%, and 36.99%, and the driving cost is reduced by 17.76%, 8.51%, and 7.12%, respectively, under each usage scenario.

Resource Allocation Based on Digital Twin-Enabled Federated Learning Framework in Heterogeneous Cellular Network

Federated learning (FL) allows user devices (UDs) to upload local model parameters to participate in a global model training, which protects UDs' data privacy. Nevertheless, FL still faces challenges such as core network congestion, UDs' limited resources and less efficient mapping between devices and cyber systems. Therefore, in this article, we integrate the digital twin (DT) and the mobile edge computing (MEC) technologies into a hierarchical FL framework in the heterogeneous cellular network scenario. When the UDs are not in the service range of the small base stations (SBSs), the framework allows macro base stations to assist UDs' local computation, thus reducing the transmission delay. It also protects the user privacy and allows more users to join in the training in order to improve the FL accuracy. In addition, we propose a deep reinforcement learning-based scheme to solve the joint optimization problem of dynamic UDs-stations association and resource allocation, thereby minimizing the energy consumption within a limited time delay. Simulation results show that our proposed scheme not only effectively reduces the task transmission failure rate and energy consumption compared with the baseline scheme, but also saves the communication cost through the DT network.

A multiple neural network and reinforcement learning-based strategy for process control

In this study, we propose a novel process control strategy by assimilating multiple neural network (MNN) and reinforcement learning (RL) to generate an MNNRL controller. The artificial neural networks (ANNs) constituting the MNN are trained using repositories of optimal control and state data to predict control actions to attain or stay at a set-point. This ability is significantly improved by developing an RL component for the MNNRL controller to provide enhanced set-point tracking, robustness against disturbances, and real-time learning to successfully adapt in the presence of persistent parameter upsets. The controller performance is examined in three different case studies utilizing the simulation of continuous processes with varying degrees of nonlinearity. Relative to both MNN and nonlinear model predictive control (NMPC), the MNNRL controller is found to provide better performance with about 32% less lower integral absolute error (IAE), and 50% reduced settling time with suppressed over-/under-shoots in controlled variables. Further, it is observed that the controller is fairly robust against disturbances, and can adapt via real-time learning to altered process dynamics caused by any persistent process upsets. The controller computation time is observed to be about an order of magnitude less in comparison to NMPC.

A Reinforcement Learning-Based Strategy of Path Following for Snake Robots with an Onboard Camera.

For path following of snake robots, many model-based controllers have demonstrated strong tracking abilities. However, a satisfactory performance often relies on precise modelling and simplified assumptions. In addition, visual perception is also essential for autonomous closed-loop control, which renders the path following of snake robots even more challenging. Hence, a novel reinforcement learning-based hierarchical control framework is designed to enable a snake robot with an onboard camera to realize autonomous self-localization and path following. Specifically, firstly, a path following policy is trained in a hierarchical manner, in which the RL algorithm and gait knowledge are well combined. On this basis, the training efficiency is sufficiently optimized, and the path following performance of the control policy is greatly improved, which can then be implemented on a practical snake robot without any additional training. Subsequently, in order to promote visual self-localization during path following, a visual localization stabilization item is added to the reward function that trains the path following strategy, which endows a snake robot with smooth steering ability during locomotion, thereby guaranteeing the accuracy of visual localization and facilitating practical applications. Comparative simulations and experimental results are illustrated to exhibit the superior performance of the proposed hierarchical path following the control method in terms of convergence speed and tracking accuracy.

Deep reinforcement learning-based strategy for charging station participating in demand response

The trend of zero-carbonization has accelerated the prevalence of electric vehicles (EVs) owing to their advantages of low carbon emissions and high energy efficiency. The stochastic and high charging load of EVs results in a non-negligible challenge that may cause grid overload. A promising approach is the participation of charging stations in demand response as load aggregators by coordinating the charging power of electric vehicles. However, improper coordination of charging load may lead to unfulfilled charging demand, which would cause dissatisfaction on the demand side. In this study, the incentive-based and time-varying demand response mechanism is considered when charging stations coordinate charging of multiple EVs. A decentralized decision-making framework is innovatively applied to provide charging power of each EV. The charging process is modeled as a Markov decision process, and a virtual price is designed to help decide the charging power. Deep reinforcement learning algorithms such as deep deterministic policy gradient are applied to determine the charging strategy of multiple and heterogeneous EVs. Numerical experiments are performed to validate the effectiveness of the proposed method. A comparison with an optimal charging strategy and a heuristic rule-based method shows that the proposed method can trade off the revenue from demand response and user satisfaction, as well as reduce the peak load of the charging station. Furthermore, a test with inaccurate departure information indicates the robustness of the proposed method.

Deep Reinforcement Learning-Based Trading Strategy for Load Aggregators on Price-Responsive Demand.

With the development of the Internet of things and smart grid technologies, modern electricity markets seamlessly connect demand response to the spot market through price-responsive loads, in which the trading strategy of load aggregators plays a crucial role in profit capture. In this study, we propose a deep reinforcement learning-based strategy for purchasing and selling electricity based on real-time electricity prices and real-time demand data in the spot market, which maximizes the revenue of load aggregators. The deep deterministic policy gradient (DDPG) is applied through a bidirectional long- and short-term memory (BiLSTM) network to extract the market state features that are used to make trading decisions. The effectiveness of the method is validated using datasets from the New England electricity market and Australian electricity market by introducing a bidirectional LSTM structure into the actor-critic network structure to learn hidden states in partially observable Markov states through memory inference. Comparative experiments of the method show that the method can provide greater yield results.

Secondary Frequency Control of Microgrids: An Online Reinforcement Learning Approach

In this article, we present a reinforcement learning-based scheme for secondary frequency control of lossy inverter-based microgrids. Compared with the existing methods in the literature, we relax the common restrictions on the system, i.e., being lossless, and the transmission lines and loads to have known constant impedances. The proposed secondary frequency control scheme does not require <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a priori</i> information about system parameters and can achieve frequency synchronization within an ultimate bound in the presence of dominantly resistive and/or inductive line and load impedances, model parameter uncertainties, and time varying loads and disturbances. First, using Lyapunov theory, a feedback control is formulated based on the unknown dynamics of the microgrid. Next, a performance function is defined based on cumulative costs toward achieving convergence to the nominal frequency. The performance function is approximated by a critic neural network in real-time. An actor network is then simultaneously learning a parameterized approximation of the nonlinear dynamics and optimizing the approximated performance function obtained from the critic network. Furthermore, using the Lyapunov approach, the uniformly ultimate boundedness of the closed-loop frequency error dynamics and the networks' weight estimation errors are shown.

Automatic Double-Auction Mechanism for Federated Learning Service Market in Internet of Things

In the past decade, the Artificial Intelligent & Internet of Things (AIoT) has become the disruptive force reshaping our lives and works. Billions of AIoT devices around the world are now connected to the Internet. This widespread deployment of AIoT devices presents huge challenges for the centralized AI training architecture (e.g., security issues, privacy issues). Recently, with the advance of federated learning (FL) technology, the integration of AIoT and FL is considered as a promising solution. FL enables distributed devices to collaboratively train a shared AI model while keeping all the training data contained in local. However, unlike the centralized architecture, the distributed learners are independent, uncontrollable, and self-interested. Designing effective incentive schemes to stimulate the distributed devices to actively participate in training tasks and contribute high-quality training models is the priority question. In this paper, we design a double auction mechanism for FL service market, where the trained models can be automatically traded between AIoT devices and FL platforms. Then, we propose an iterative double auction (IDA) algorithm, where a controller is implemented to guide participants to adjust their policies for the sake of maximizing individual participates utilities as well as social welfare. In addition, we propose a reinforcement learning-based scheme, termed as the Experience Weighted Attraction Learning-Double Auction (EWA-DA) algorithm. In EWA-DA, the participants can learn to improve their strategies directly by participating in the auction continuously, without the controller participating.

Towards reinforcement learning in UAV relay for anti-jamming maritime communications

Maritime communications with sea surface reflections and sea wave occlusions are susceptible to jamming attacks due to the wide geographical area and intensive wireless communication services. Unmanned Aerial Vehicles (UAVs) help relay messages to improve communication performance, but the relay policy that depends on the rapidly changing maritime environments is difficult to optimize. In this paper, a reinforcement learning-based UAV relay policy for maritime communications is proposed to resist jamming attacks. Based on previous transmission performance, the relay location, the received power of the transmitted signal and the received jamming power, this scheme optimizes the UAV trajectory and relay power to save the energy consumption and decrease the Bit-Error-Rate (BER) of the maritime signals. A deep reinforcement learning-based scheme is also proposed, which designs a deep neural network with dueling architecture to further improve the communication performance and computational complexity. The performance bounds regarding the signal to interference plus noise ratio, energy consumption and the communication utility are provided based on the Nash equilibrium of the game against jamming, and the computational complexity of the proposed schemes is analyzed. Simulation results show that the proposed schemes improve the energy efficiency and decrease the BER compared with the benchmark.

Resource allocation for UAV-aided energy harvesting-powered D2D communications: A reinforcement learning-based scheme

Unmanned Aerial Vehicle (UAV) has become one of the most significant component in future wireless networks since its on-demand and cost-effective deployment. Meanwhile, Device-to-Device (D2D) communications with the nature of providing proximity-based service to improve network performance has emerged as an important feature in 5th generation cellular networks. However, the explosive growth of smart devices and bandwidth-hungry applications cause enormous energy consumption. Energy Harvesting (EH) as a potential solution of improving energy efficiency has shown great importance. Consequently, in this paper, we investigate the resource allocation problem in UAV-aided EH-powered D2D Cellular Networks (UAV-EH-DCNs). Our objective is to maximize the energy efficiency while guaranteeing the satisfaction of ground users (GUs). Owning to the non-convexity of the problem, we formulate the problem as a Markov decision process. Afterward, Deep Deterministic Policy Gradient (DDPG) is proposed to find the optimal strategy. Additionally, Long Short-Term Memory (LSTM) network is employed to facilitate the convergence speed by extracting the previous information of GUs satisfaction to determine the current resource allocation strategy. Numerical results adduce the validity of the proposed DDPG+LSTM algorithm as compared to the DDPG and deep Q-learning algorithms.

Resource Allocation for Cellular Zero-Touch Deterministic Industrial M2M Networks: A Reinforcement Learning-Based Scheme

In this letter, we investigate the resource allocation problem for cellular zero-touch deterministic industrial machine-to-machine networks. We consider a scenario in which multiple self-adaptive industrial machine-to-machine (M2M) networks are deployed underlaying cellular networks to emulate a variety of vertical use cases in future Industry 4.0 with the objective of guaranteeing the determinacy of latency by learning-inspired resource allocation policy. Specifically, in order to figure out the relationship between latency determinacy and the utilization of wireless resources meanwhile modeling the uncertainty of the stochastic environment, we first formulate the resource allocation problem as a Markov game by jointly considering transmission power control, frequency spectrum allocation, and the selection of base stations. Afterward, a random graph-based sparse long short-term memory network is proposed to solve the optimization problem while reducing the computational complexity. Finally, numerical result demonstrates the effectiveness of the proposed scheme.