Reinforcement Learning-based Approach Research Articles

Multi-Lane Differential Variable Speed Limit Control via Deep Neural Networks Optimized by an Adaptive Evolutionary Strategy.

In advanced transportation-management systems, variable speed limits are a crucial application. Deep reinforcement learning methods have been shown to have superior performance in many applications, as they are an effective approach to learning environment dynamics for decision-making and control. However, they face two significant difficulties in traffic-control applications: reward engineering with delayed reward and brittle convergence properties with gradient descent. To address these challenges, evolutionary strategies are well suited as a class of black-box optimization techniques inspired by natural evolution. Additionally, the traditional deep reinforcement learning framework struggles to handle the delayed reward setting. This paper proposes a novel approach using covariance matrix adaptation evolution strategy (CMA-ES), a gradient-free global optimization method, to handle the task of multi-lane differential variable speed limit control. The proposed method uses a deep-learning-based method to dynamically learn optimal and distinct speed limits among lanes. The parameters of the neural network are sampled using a multivariate normal distribution, and the dependencies between the variables are represented by a covariance matrix that is optimized dynamically by CMA-ES based on the freeway's throughput. The proposed approach is tested on a freeway with simulated recurrent bottlenecks, and the experimental results show that it outperforms deep reinforcement learning-based approaches, traditional evolutionary search methods, and the no-control scenario. Our proposed method demonstrates a 23% improvement in average travel time and an average of a 4% improvement in CO, HC, and NOx emission.Furthermore, the proposed method produces explainable speed limits and has desirable generalization power.

Reinforcement Learning-based Approach for Click-Through Rate Optimization in Real-Time

Reinforcement Learning-based Approach for Click-Through Rate Optimization in Real-Time

Energy Efficiency Maximization in RIS-Assisted SWIPT Networks With RSMA: A PPO-Based Approach

This paper investigates reconfigurable intelligent surface (RIS)-assisted simultaneous wireless information and power transfer (SWIPT) networks with rate splitting multiple access (RSMA). An energy efficiency (EE) maximization problem is formulated subject to the power budget at the transmitter and the quality of service (QoS) requirements of both information communication and energy harvesting, where the beamforming vectors, the power splitting (PS) ratios, the common message rates, and the discrete phase shifts are jointly optimized. To tackle the non-convex problem with both discrete and continuous variables, a deep reinforcement learning-based approach is proposed with the proximal policy optimization (PPO) framework. Different from traditional optimization approaches which optimizes the beamforming vectors and phase shifts separately and alternatively, our proposed PPO-based approach optimizes all the variables in unison. Besides, to perform beamforming design in action space, the beamforming vectors for the common stream and the private stream are respectively designed based on the maximum-ratio transmission and the zero forcing to enhance both energy and information transmission. To evaluate the performance of the PPO-based approach, a successive convex approximation (SCA) and Dinkelbach’s method based solution scheme (named SCA-D scheme) is also presented. Simulation results show that the system EE obtained by the proposed PPO-based approach is close to that obtained by the SCA-D scheme while outperforming various benchmarks. The RSMA contributes to the EE of the system greatly compared with traditional scheme. As for the case of time-varying channels, the proposed PPO-based approach is with much smaller running time by only sacrificing a slight EE performance compared with the SCA-D scheme.

A Case Study of Automated Simulation for Excavator Cycle Time Estimation

Efficient planning of earthworks is critical for the success of construction projects, accounting for approximately 30% of the project budget. However, traditional approaches, relying on managers' experience, or using standardized estimating systems and equipment manufacturer data, are regarded as inaccurate due to the fact that such methods may are limited in incorporating various site conditions that influence excavator operation, such as digging depth, swing angle, and terrain shape. This study proposes a reinforcement learning-based simulation approach for predicting the cycle time of an excavator given field conditions. For the training of an excavator agent, terrain point clouds of a site are collected and used to build a 3D virtual simulation environment representing a real site where the agent learns how to operate the excavation through trial and error processes. The proposed approach achieves a prediction accuracy of 88.89%, which is higher than that of a traditional method (77.06%). It is thus expected that the proposed approach may allow for improving the accuracy of cycle time estimation for earthmoving planning in practice.

A reinforcement learning-based optimal control approach for managing an elective surgery backlog after pandemic disruption.

Contagious disease pandemics, such as COVID-19, can cause hospitals around the world to delay nonemergent elective surgeries, which results in a large surgery backlog. To develop an operational solution for providing patients timely surgical care with limited health care resources, this study proposes a stochastic control process-based method that helps hospitals make operational recovery plans to clear their surgery backlog and restore surgical activity safely. The elective surgery backlog recovery process is modeled by a general discrete-time queueing network system, which is formulated by a Markov decision process. A scheduling optimization algorithm based on the piecewise decaying [Formula: see text]-greedy reinforcement learning algorithm is proposed to make dynamic daily surgery scheduling plans considering newly arrived patients, waiting time and clinical urgency. The proposed method is tested through a set of simulated dataset, and implemented on an elective surgery backlog that built up in one large general hospital in China after the outbreak of COVID-19. The results show that, compared with the current policy, the proposed method can effectively and rapidly clear the surgery backlog caused by a pandemic while ensuring that all patients receive timely surgical care. These results encourage the wider adoption of the proposed method to manage surgery scheduling during all phases of a public health crisis.

Reinforcement Learning-Based Approach for Minimizing Energy Loss of Driving Platoon Decisions.

Reinforcement learning (RL) methods for energy saving and greening have recently appeared in the field of autonomous driving. In inter-vehicle communication (IVC), a feasible and increasingly popular research direction of RL is to obtain the optimal action decision of agents in a special environment. This paper presents the application of reinforcement learning in the vehicle communication simulation framework (Veins). In this research, we explore the application of reinforcement learning algorithms in a green cooperative adaptive cruise control (CACC) platoon. Our aim is to train member vehicles to react appropriately in the event of a severe collision involving the leading vehicle. We seek to reduce collision damage and optimize energy consumption by encouraging behavior that conforms to the platoon's environmentally friendly aim. Our study provides insight into the potential benefits of using reinforcement learning algorithms to improve the safety and efficiency of CACC platoons while promoting sustainable transportation. The policy gradient algorithm used in this paper has good convergence in the calculation of the minimum energy consumption problem and the optimal solution of vehicle behavior. In terms of energy consumption metrics, the policy gradient algorithm is used first in the IVC field for training the proposed platoon problem. It is a feasible training decision-planning algorithm for solving the minimization of energy consumption caused by decision making in platoon avoidance behavior.

A reinforcement learning-based approach for online bus scheduling

Bus Scheduling Problem (BSP) is vital to save operational cost and ensure service quality. Existing approaches typically generate a bus scheduling scheme in an offline manner and then schedule vehicles according to the scheme. In practice, uncertain events such as traffic congestion occur frequently, which may make the originally planned bus scheduling scheme infeasible. This study proposes a Reinforcement Learning-based Bus Scheduling Approach (RL-BSA) for online bus scheduling. In RL-BSA, each departure time in a bus timetable is regarded as a decision point, and an agent makes a decision at the departure time to select a vehicle to depart at the time. The BSP is modeled as a Markov Decision Process (MDP) for the first time in literature. The state features are devised, which consist of real-time information of vehicles, including remaining working time, remaining driving time, rest time, number of executed trips and vehicle type. A reward function combining a final reward and a step-wise reward is devised. An invalid action masking approach is used to avoid the agent from selecting vehicles not meeting constraints. The agent is trained by interacting with a simulation environment and then the trained agent can schedule vehicles in an online manner. Experiments on real-world BSP instances show that RL-BSA can significantly reduce the number of vehicles used compared with the manual scheduling approach and Adaptive Large Neighborhood Search (ALNS). Under uncertain environment, RL-BSA can cover all departure times in the timetable without increasing the number of vehicles used.

Deep reinforcement learning-based approach for rumor influence minimization in social networks.

Spreading malicious rumors on social networks such as Facebook, Twitter, and WeChat can trigger political conflicts, sway public opinion, and cause social disruption. A rumor can spread rapidly across a network and can be difficult to control once it has gained traction.Rumor influence minimization (RIM) is a central problem in information diffusion and network theory that involves finding ways to minimize rumor spread within a social network. Existing research on the RIM problem has focused on blocking the actions of influential users who can drive rumor propagation. These traditional static solutions do not adequately capture the dynamics and characteristics of rumor evolution from a global perspective. A deep reinforcement learning strategy that takes into account a wide range of factors may be an effective way of addressing the RIM challenge. This study introduces the dynamic rumor influence minimization (DRIM) problem, a step-by-step discrete time optimization method for controlling rumors. In addition, we provide a dynamic rumor-blocking approach, namely RLDB, based on deep reinforcement learning. First, a static rumor propagation model (SRPM) and a dynamic rumor propagation model (DRPM) based on of independent cascade patterns are presented. The primary benefit of the DPRM is that it can dynamically adjust the probability matrix according to the number of individuals affected by rumors in a social network, thereby improving the accuracy of rumor propagation simulation. Second, the RLDB strategy identifies the users to block in order to minimize rumor influence by observing the dynamics of user states and social network architectures. Finally, we assess the blocking model using four real-world datasets with different sizes. The experimental results demonstrate the superiority of the proposed approach on heuristics such as out-degree(OD), betweenness centrality(BC), and PageRank(PR).

A Deep Reinforcement Learning-Based Approach in Porker Game

<p>Recent years have witnessed the big success deep reinforcement learning achieved in the domain of card and board games, such as Go, chess and Texas Hold’em poker. However, Dou Di Zhu, a traditional Chinese card game, is still a challenging task for deep reinforcement learning methods due to the enormous action space and the sparse and delayed reward of each action from the environment. Basic reinforcement learning algorithms are more effective in the simple environments which have small action spaces and valuable and concrete reward functions, and unfortunately, are shown not be able to deal with Dou Di Zhu satisfactorily. This work introduces an approach named Two-steps Q-Network based on DQN to playing Dou Di Zhu, which compresses the huge action space through dividing it into two parts according to the rules of Dou Di Zhu and fills in the sparse rewards using inverse reinforcement learning (IRL) through abstracting the reward function from experts’ demonstrations. It is illustrated by the experiments that two-steps Q-network gains great advancements compared with DQN used in Dou Di Zhu.</p> <p> </p>

An AR-assisted Deep Reinforcement Learning-based approach towards mutual-cognitive safe human-robot interaction

With the emergence of Industry 5.0, the human-centric manufacturing paradigm requires manufacturing equipment (robots, etc.) interactively assist human workers to deal with dynamic and complex production tasks. To achieve symbiotic human–robot interaction (HRI), the safety issue serves as a prerequisite foundation. Regarding the growing individualized demand of manufacturing tasks, the conventional rule-based safe HRI measures could not well address the safety requirements due to inflexibility and lacking synergy. To fill the gap, this work proposes a mutual-cognitive safe HRI approach including worker visual augmentation, robot velocity control, Digital Twin-enabled motion preview and collision detection , and Deep Reinforcement Learning-based robot collision avoidance motion planning in the Augmented Reality-assisted manner. Finally, the feasibility of the system design and the performance of the proposed approach are validated by establishing and executing the prototype HRI system in a practical scene. • Introduced an AR-assisted system architecture for mutual-cognitive safe HRI. • Implemented distance-based robot velocity control and area-based workers’ visual aids functions. • Developed the robot DT-enabled motion preview for workers to enhance safe cognition and for robots to detect collision. • Proposed a curriculum learning-based DRL motion planning policy for collision avoidance.

Deep Reinforcement Learning-Based Approach for Efficient and Reliable Droplet Routing on MEDA Biochips

The micro-electrode-dot-array (MEDA) architecture provides precise droplet control and real-time sensing in digital microfluidic biochips. Previous work has shown that trapped charge under microelectrodes (MCs) leads to droplets being stuck and failures in fluidic operations. A recent approach utilizes real-time sensing of MC health status, and attempts to avoid degraded electrodes during droplet routing. However, the problem with this solution is that the computational complexity is unacceptable for MEDA biochips of realistic size. Consequently, in this work, we introduce a deep reinforcement learning (DRL)-based approach to bypass degraded electrodes and enhance the reliability of routing. The DRL model utilizes the information of health sensing in real time to proactively reduce the likelihood of charge trapping and avoid using degraded MCs. Simulation results show that our approach provides effective routing strategies for COVID-19 testing protocols. We also validate our DRL-based approach using fabricated prototype biochips. Experimental results show that the developed DRL model completed the routing tasks using a fewer number of clock cycles and shorter total execution time, compared with a baseline routing method. Moreover, our DRL-based approach provides reliable routing strategies even in the presence of degraded electrodes. Our experimental results show that the proposed DRL-based routing is robust to occurrences of electrode faults, as well as increases the lifetime and usability of microfluidic biochips compared to existing strategies.

Heterogeneous Crowd Simulation using Parametric Reinforcement Learning.

Agent-based synthetic crowd simulation affords the cost-effective large-scale simulation and animation of interacting digital humans. Model-based approaches have successfully generated a plethora of simulators with a variety of foundations. However, prior approaches have been based on statically defined models predicated on simplifying assumptions, limited video-based datasets, or homogeneous policies. Recent works have applied reinforcement learning to learn policies for navigation. However, these approaches may learn static homogeneous rules, are typically limited in their generalization to trained scenarios, and limited in their usability in synthetic crowd domains. In this paper, we present a multi-agent reinforcement learning-based approach that learns a parametric predictive collision avoidance and steering policy. We show that training over a parameter space produces a flexible model across crowd configurations. That is, our goal-conditioned approach learns a parametric policy that affords heterogeneous synthetic crowds. We propose a model-free approach without centralization of internal agent information, control signals, or agent communication. The model is extensively evaluated. The results show policy generalization across unseen scenarios, agent parameters, and out-of-distribution parameterizations. The learned model has comparable computational performance to traditional methods. Qualitatively the model produces both expected (laminar flow, shuffling, bottleneck) and unexpected (side-stepping) emergent qualitative behaviours, and quantitatively the approach is performant across measures of movement quality.

Cooperative Task Offloading and Block Mining in Blockchain-Based Edge Computing With Multi-Agent Deep Reinforcement Learning

The convergence of mobile edge computing (MEC) and blockchain is transforming the current computing services in mobile networks, by offering task offloading solutions with security enhancement empowered by blockchain mining. Nevertheless, these important enabling technologies have been studied separately in most existing works. This article proposes a novel cooperative task offloading and block mining (TOBM) scheme for a blockchain-based MEC system where each edge device not only handles data tasks but also deals with block mining for improving the system utility. To address the latency issues caused by the blockchain operation in MEC, we develop a new Proof-of-Reputation consensus mechanism based on a lightweight block verification strategy. A multi-objective function is then formulated to maximize the system utility of the blockchain-based MEC system, by jointly optimizing offloading decision, channel selection, transmit power allocation, and computational resource allocation. We propose a novel distributed deep reinforcement learning-based approach by using a multi-agent deep deterministic policy gradient algorithm. We then develop a game-theoretic solution to model the offloading and mining competition among edge devices as a potential game, and prove the existence of a pure Nash equilibrium. Simulation results demonstrate the significant system utility improvements of our proposed scheme over baseline approaches.

A Reinforcement Learning-Based Congestion Control Approach for V2V Communication in VANET

Vehicular ad hoc networks (VANETs) are crucial components of intelligent transportation systems (ITS) aimed at enhancing road safety and providing additional services to vehicles and their users. To achieve reliable delivery of periodic status information, referred to as basic safety messages (BSMs) and event-driven alerts, vehicles need to manage the conflicting requirements of situational awareness and congestion control in a dynamic environment. To address this challenge, this paper focuses on controlling the message transmission rate through a Markov decision process (MDP) and solves it using a novel reinforcement learning (RL) algorithm. The proposed RL approach selects the most suitable transmission rate based on the current channel conditions, resulting in a balanced performance in terms of packet delivery and channel congestion, as shown by simulation results for different traffic scenarios. Additionally, the proposed approach offers increased flexibility for adaptive congestion control through the design of an appropriate reward function.

Online job scheduling scheme for low-carbon data center operation: An information and energy nexus perspective

As the digitalization of the economy and society accelerates, the enormous and fast-growing energy consumption of data centers is becoming a global concern. With the unique power consumption flexibility introduced by computing job scheduling, data centers could play an important role in enhancing the capability to integrate renewable generation as a demand-side resource. In this paper, we propose an online job scheduling scheme for low-carbon data center operation from an information and energy nexus perspective. We formulate the job scheduling problem as a Markov decision process in which job dependencies, job heterogeneity, and quality of service are considered comprehensively. To address the challenges of large-scale heterogeneous computing jobs, we propose a deep reinforcement learning-based approach to solve the energy-aware scheduling problem and achieve an optimal online policy. The case study results based on real-world data illustrate that the proposed scheme can effectively reduce the carbon footprint and energy cost of a data center while maintaining the quality of service for cloud products.

Deep reinforcement learning for data-efficient weakly supervised business process anomaly detection

The detection of anomalous behavior in business process data is a crucial task for preventing failures that may jeopardize the performance of any organization. Supervised learning techniques are impracticable because of the difficulties of gathering huge amounts of labeled business process anomaly data. For this reason, unsupervised learning techniques and semi-supervised learning approaches trained on entirely labeled normal data have dominated this domain for a long time. However, these methods do not work well because of the absence of prior knowledge of true anomalies. In this study, we propose a deep weakly supervised reinforcement learning-based approach to identify anomalies in business processes by leveraging limited labeled anomaly data. The proposed approach is intended to use a small collection of labeled anomalous data while exploring a huge set of unlabeled data to find new classes of anomalies that are outside the scope of the labeled anomalous data. We created a unique reward function that combined the supervisory signal supplied by a variational autoencoder trained on unlabeled data with the supervisory signal provided by the environment’s reward. To further reduce data deficiency, we introduced a sampling method to allow the effective exploration of the unlabeled data and to address the imbalanced data problem, which is a common problem in the anomaly detection field. This approach depends on the proximity between the data samples in the latent space of the variational autoencoder. Furthermore, to efficiently model the sequential nature of business process data and to handle the long-term dependences, we used a long short-term memory network combined with a self-attention mechanism to develop the agent of our reinforcement learning model. Multiple scenarios were used to test the proposed approach on real-world and synthetic datasets. The findings revealed that the proposed approach outperformed five competing approaches by efficiently using the few available anomalous examples.

Deep reinforcement learning-based propofol infusion control for anesthesia: A feasibility study with a 3000-subject dataset

In this work, we present a deep reinforcement learning-based approach as a baseline system for autonomous propofol infusion control. Specifically, design an environment for simulating the possible conditions of a target patient based on input demographic data and design our reinforcement learning model-based system so that it effectively makes predictions on the proper level of propofol infusion to maintain stable anesthesia even under dynamic conditions that can affect the decision-making process, such as the manual control of remifentanil by anesthesiologists and the varying patient conditions under anesthesia. Through an extensive set of evaluations using patient data from 3000 subjects, we show that the proposed method results in stabilization in the anesthesia state, by managing the bispectral index (BIS) and effect-site concentration for a patient showing varying conditions.

Federated Learning for Intelligent Transmission with Space-Air-Ground Integrated Network toward 6G

The future intelligent devices requires ultra-low communication delay and high QoS requirement for the following beyond-5G network. Space-air-ground Integrated Network (SAGIN) integrated with satellite networks in space, aerial networks, and terrestrial networks is an advanced network framework to expand the networking overage and improve connectivity for intelligent applications. However, due to the heterogeneous structure and high dynamic of SAGIN, the resource management and transmission strategy should be carefully designed to adjust to the unbalanced resources and varying environments. Federated learning is an innovative distributed learning method to intelligently manage the resource scheduling problem in SAGIN with security and guarantee of user privacy. In this paper, we introduce the advantages and potential usage directions of using federated learning in SAGIN in terms of different optimization objectives. To better illustrate the potential deployment of federated learning in SAGIN, we further provide a case study of a federated reinforcement learning-based traffic offloading approach in SAGIN.

A Reinforcement Learning-based Approach for Active Optimization of High Proportional Distributed Power Supply Access to Rural Distribution Networks

In the process of actively implementing the rural revitalization strategy and the “double carbon” goal, distributed power sources are connected to the rural distribution network on a large scale, resulting in a gradual transition to a new type of active distribution network, which has a greater impact on the distribution network in terms of voltage, network loss, and protection, thus posing greater demands on the distribution network in terms of planning and control, power quality management, and operation. This significantly impacts the voltage, network loss, and protection of the distribution network, thus putting higher requirements on the planning and control, power quality management, operation, and maintenance of distribution networks. Since the traditional OPF algorithm has great defects in solving large-scale decentralized, time-series, and stochastic nonlinear problems, in order to realize real-time control of distribution networks under more complex network conditions, this paper proposes an optimization algorithm based on reinforcement learning and applies it to real-time control of active distribution networks, taking the optimal network loss as the objective function and considering network voltage, line load factor, and distributed power supply. As a de-modeling artificial intelligence method, reinforcement learning has great advantages in solving large problems with complex mathematical models. The method’s control effect and convergence time can be improved by introducing imitation learning and migration learning, expanding the knowledge matrix, and introducing risk assessment mechanisms. Based on MATLAB R2017a simulation platform for the IEEE 33-node improvement model, the simulation results show that the reinforcement learning method has a better control effect in solving the real-time control problem of active distribution network, significantly alleviates the voltage crossing limit, effectively reduces the network loss, and has good real-time characteristics.

Simultaneous locomotion and manipulation control of quadruped robots using reinforcement learning-based adaptive fractional-order sliding-mode control

This paper investigates a model-free reinforcement learning-based approach that enables the quadruped robot to manipulate objects while maintaining its balance and dynamic stability during walking. At first, the dynamics of quadruped robots in two sub-spaces, position control space and force control space, are developed. Then, a new long-term performance index is introduced, and a radial basis function neural network as a critic network is presented to estimate the unobtainable long-term performance index. Based on the exported reinforcement signal, the actor neural network is introduced to generate the feedforward compensation term to cope with the nonlinear dynamics and the system uncertainties. The robustness of the actor-critic reinforcement learning algorithm is enhanced by using a fractional-order sliding-mode controller in the closed-loop system. The online adaptive laws for both the critic and actor-network weights are obtained using the Lyapunov stability theory. As a result, the uniformly ultimately boundedness of the position and the force tracking errors are proven. Finally, numerical simulations are conducted to illustrate the feasibility and effectiveness of the proposed adaptive actor-critic learning-based control scheme.