Deep Reinforcement Learning Framework Research Articles

Long-Term Tracking of Evasive Urban Target Based on Intention Inference and Deep Reinforcement Learning.

Unmanned aerial vehicles (UAVs) have been widely used in urban target-tracking tasks, where long-term tracking of evasive targets is of great significance for public safety. However, the tracked targets are easily lost due to the evasive behavior of the targets and the unstructured characteristics of the urban environment. To address this issue, this article proposes a hybrid target-tracking approach based on target intention inference and deep reinforcement learning (DRL). First, a target intention inference model based on convolution neural networks (CNNs) is built to infer target intentions by fusing urban environment information and observed target trajectory. Then, the prediction of the target trajectory can be inspired by the inferred target intentions, which can further provide effective guidance to the target search process. In order to fully explore the policy space, the target search policy is developed under a DRL framework, where the search policy is modeled as a deep neural network (DNN) and trained by interacting with the task environment. The simulation results show that the inference of the target intentions can effectively guide the UAV to search for the target and significantly improve the target-tracking performance. Meanwhile, the generalization results indicate that the proposed DRL-based search policy has high robustness to the uncertainty of the target behavior.

A study of natural robustness of deep reinforcement learning algorithms towards adversarial perturbations

Deep reinforcement learning (DRL) has been shown to have numerous potential applications in the real world. However, DRL algorithms are still extremely sensitive to noise and adversarial perturbations, hence inhibiting the deployment of RL in many real-life applications. Analyzing the robustness of DRL algorithms to adversarial attacks is an important prerequisite to enabling the widespread adoption of DRL algorithms. Common perturbations on DRL frameworks during test time include perturbations to the observation and the action channel. Compared with observation channel attacks, action channel attacks are less studied; hence, few comparisons exist that compare the effectiveness of these attacks in DRL literature. In this work, we examined the effectiveness of these two paradigms of attacks on common DRL algorithms and studied the natural robustness of DRL algorithms towards various adversarial attacks in hopes of gaining insights into the individual response of each type of algorithm under different attack conditions.

Multi-Agent Deep Reinforcement Learning for Spectral Efficiency Optimization in Vehicular Optical Camera Communications

In this paper, we propose a vehicular optical camera communication system that can meet low bit error rate (BER) and ultra-low latency constraints. First, we formulate a sum spectral efficiency optimization problem that aims at finding the speed of vehicles and the modulation order that maximizes the sum spectral efficiency subject to reliability and latency constraints. This problem is mixed-integer programming with nonlinear constraints, and even for a small set of modulation orders, is NP-hard. To overcome the entailed high computational and time complexity which prevents its solution with traditional methods, we first model the optimization problem as a partially observable Markov decision process. We then solve it using an independent Q-learning framework, where each vehicle acts as an independent agent. Since the state-action space is large we then adopt deep reinforcement learning (DRL) to solve it efficiently. As the problem is constrained, we employ the Lagrange relaxation approach prior to solving it using the DRL framework. Simulation results demonstrate that the proposed DRL-based optimization scheme can effectively learn how to maximize the sum spectral efficiency while satisfying the BER and ultra-low latency constraints. The evaluation further shows that our scheme can achieve superior performance compared to radio frequency-based vehicular communication systems and other vehicular OCC variants of our scheme.

Structure-Enhanced DRL for Optimal Transmission Scheduling

Remote state estimation of large-scale distributed dynamic processes plays an important role in Industry 4.0 applications. In this paper, we focus on the transmission scheduling problem of a remote estimation system. First, we derive some structural properties of the optimal sensor scheduling policy over fading channels. Then, building on these theoretical guidelines, we develop a structure-enhanced deep reinforcement learning (DRL) framework for optimal scheduling of the system to achieve the minimum overall estimation mean-square error (MSE). In particular, we propose a structure-enhanced action selection method, which tends to select actions that obey the policy structure. This explores the action space more effectively and enhances the learning efficiency of DRL agents. Furthermore, we introduce a structure-enhanced loss function to add penalties to actions that do not follow the policy structure. The new loss function guides the DRL to converge to the optimal policy structure quickly. Our numerical experiments illustrate that the proposed structure-enhanced DRL algorithms can save the training time by 50% and reduce the remote estimation MSE by 10% to 25%, when compared to benchmark DRL algorithms. In addition, we show that the derived structural properties exist in a wide range of dynamic scheduling problems that go beyond remote state estimation.

Combining Reinforcement Learning and In Situ Current Velocity Measurements for Efficient UUV Guidance

Conventional guidance algorithms for unmanned underwater vehicles commonly operate on geometric principles that do not directly take into account local environmental conditions. We describe here a deep reinforcement learning (RL) framework that partially addresses this issue by generating energy-minimizing guidance systems. Specifically, the proposed framework outputs trained policy networks (agents) capable of mapping histories of local current velocity measurements to heading and depth commands that yield nearly-optimal, minimum-energy transits through spatially varying current fields. Numerical experiments in three spatial dimensions on realistic ocean currents, where direction and speed vary non-monotonically in depth, demonstrate that trained agents reduce the energy costs of transits when compared against dead-reckoning, a common in-service guidance heuristic. In particular, numerical results suggest that trained agents learn an explore-exploit strategy according to which they initially execute an exploratory dive, followed by a return to a depth with optimally favorable currents, which they exploit for the remainder of the transit.

Learning-based Integrated Cooperative Motion Planning and Control of Multi-AUVs

This paper introduces a learning-based solution tailored for the integrated motion planning and control of Multiple Autonomous Underwater Vehicles (AUVs). Tackling the complexities of cooperative motion planning, encompassing tasks such as waypoint tracking and self/obstacle collision avoidance, becomes challenging in a rule-based algorithmic paradigm due to the diverse and unpredictable situations encountered, necessitating a proliferation of if-then conditions in the implementation. Recognizing the limitations of traditional approaches that are heavily dependent on models and geometry of the system, our solution ofers an innovative paradigm shift. This study proposes an integrated motion planning and control strategy that leverages sensor and navigation outputs to generate longitudinal and lateral control outputs dynamically. At the heart of this cutting-edge methodology lies a continuous action Deep Reinforcement Learning (DRL) framework, specifically based on the Twin Delayed Deep Deterministic Policy Gradient (TD3). This algorithm surpasses traditional limitations by embodying an elaborated reward function, enabling the seamless execution of control actions essential for maneuvering multiple AUVs. Through simulation tests under both nominal and perturbed conditions, considering obstacles and underwater current disturbances, the obtained results demonstrate the feasibility and robustness of the proposed technique.

A Task-Adaptive Deep Reinforcement Learning Framework for Dual-Arm Robot Manipulation

A Task-Adaptive Deep Reinforcement Learning Framework for Dual-Arm Robot Manipulation

An Improved SAC-Based Deep Reinforcement Learning Framework for Collaborative Pushing and Grasping in Underwater Environments

An Improved SAC-Based Deep Reinforcement Learning Framework for Collaborative Pushing and Grasping in Underwater Environments

TinyDeepUAV: A Tiny Deep Reinforcement Learning Framework for UAV Task Offloading in Edge-Based Consumer Electronics

TinyDeepUAV: A Tiny Deep Reinforcement Learning Framework for UAV Task Offloading in Edge-Based Consumer Electronics

Intelligent Predetermination of Generator Tripping Scheme: Knowledge Fusion-based Deep Reinforcement Learning Framework

Intelligent Predetermination of Generator Tripping Scheme: Knowledge Fusion-based Deep Reinforcement Learning Framework

A Deep Reinforcement Learning Framework for Control of Robotic Manipulators in Simulated Environments

A Deep Reinforcement Learning Framework for Control of Robotic Manipulators in Simulated Environments

An Azimuth Aware Deep Reinforcement Learning Framework for Active SAR Target Recognition

An Azimuth Aware Deep Reinforcement Learning Framework for Active SAR Target Recognition

Joint MCS Adaptation and RB Allocation in Cellular Networks Based on Deep Reinforcement Learning With Stable Matching

Joint modulation-coding scheme (MCS) adaptation and resource block (RB) allocation is an effective approach to guarantee different quality of service (QoS) requirements of all UEs under dynamic network environments. In this paper, we consider a fifth generation (5 G) cellular network with time-varying wireless channels, in which the BS serves multiple user equipments (UEs) under limited available RBs. We aim to minimize the total RB consumption subject to the rigorous constraints of each UE's QoS requirement. To attain this objective, this paper puts forth an online learning technique, referred to as integrated Deep Reinforcement learning and stable Matching (DeepRM), in the sense that the MCS adaptation and RB allocation decisions are conducted without acquiring the real-time channel quality indicator (CQI) feedback. DeepRM is a closed-loop framework, where the output of deep reinforcement learning (DRL) is imported into the stable matching to guide optimal RB allocation whilst the output of stable matching is fed into the DRL framework to assist efficient MCS decision-making. Specifically, in DeepRM, we first develop a powerful DRL algorithm, termed as Action-and-Reward Branching Deep Q-network (ARBDQ), by incorporating the action branch architecture into conventional DRL and modifying the traditional deep neural network training mechanism, to perform judicious MCS decisions on different links in parallel. Then, a new many-to-one stable matching algorithm, called adaptive deferred acceptance, is exploited to dynamically adjust the RB quota of each UE in a computationally efficient fashion. Simulation results demonstrate that compared with ACO-HM, OLLA-ADA, and ARBDQ-Random algorithms, DeepRM induces much less RB consumption while guaranteeing the QoS requirements of all UEs in various network scenarios. Furthermore, under miscellaneous QoS requirement, number of UEs, and CQI reporting period setups, DeepRM is more robust than other baselines.

Solving flexible job shop scheduling problems via deep reinforcement learning

Flexible job shop scheduling problem (FJSSP), as a variant of the job shop scheduling problem, has a larger solution space. Researchers are always looking for good methods to solve this problem. In recent years, the deep reinforcement learning (DRL) has been applied to solve various shop scheduling problems due to its advantages that fast solving speed and strong generalization ability. In this paper, we first propose a new DRL framework to realize representation learning and policy learning. The new framework adopts a lightweight multi-layer perceptron (MLP) as the state embedding network to extract state information, which reduces the computational complexity of the algorithm to some extent. Next, we design a new state representation and define a new action space. The new state representation can directly reflect the state features of candidate actions, which is conducive for the agent to capture more effective state information and improve its decision-making ability. The new definition of action space can solve the two subproblems of the FJSSP simultaneously with only one action space. Finally, we evaluate the performance of the policy model on four public datasets: Barnes dataset, Brandimarte dataset, Dauzere dataset and Hurink dataset. Extensive experimental results on these public datasets show that the proposed method achieves a better compromise in terms of optimization ability and applicability compared to the composite priority dispatching rules and the existing state-of-the-art models.

Double RISs assisted task offloading for NOMA-MEC with action-constrained deep reinforcement learning

Reconfigurable intelligent surface (RIS) is expected to enhance task offloading performance in non-line-of-sight mobile edge computing (MEC) scenarios. This paper aims at reducing the long-term energy consumption for task offloading in a MEC system assisted by double RISs while considering non-orthogonal multiple access (NOMA). Due to the non-convexity of the formulated problem, we propose an action-constrained deep reinforcement learning (DRL) framework based on twin delayed deep deterministic policy gradient (TD3) algorithm that includes inner and outer optimization processes, which greatly reduces the action space size of the DRL agent, making it easy to implement and achieve fast convergence. During the inner optimization phase, by utilizing theoretical derivations, we propose a low-complexity method to optimally derive the transmit power, the local computing frequency, and the computation resource allocation in the base station. In the outer optimization phase, building on the solution derived from the inner optimization, we further use the TD3 algorithm to jointly determine the phase shifting of RIS elements, the offloading ratio, and the transmission time for each time slot. Experiment results demonstrate that the proposed algorithm achieves rapid convergence performance. In comparison to single RIS assisted offloading, the double RISs assisted offloading scheme proposed in this paper can reduce energy consumption by 42.8% on average.

Joint Task Offloading and Resource Allocation for Intelligent Reflecting Surface-Aided Integrated Sensing and Communication Systems Using Deep Reinforcement Learning Algorithm.

This paper investigates an intelligent reflecting surface (IRS)-aided integrated sensing and communication (ISAC) framework to cope with the problem of spectrum scarcity and poor wireless environment. The main goal of the proposed framework in this work is to optimize the overall performance of the system, including sensing, communication, and computational offloading. We aim to achieve the trade-off between system performance and overhead by optimizing spectrum and computing resource allocation. On the one hand, the joint design of transmit beamforming and phase shift matrices can enhance the radar sensing quality and increase the communication data rate. On the other hand, task offloading and computation resource allocation optimize energy consumption and delay. Due to the coupled and high dimension optimization variables, the optimization problem is non-convex and NP-hard. Meanwhile, given the dynamic wireless channel condition, we formulate the optimization design as a Markov decision process. To tackle this complex optimization problem, we proposed two innovative deep reinforcement learning (DRL)-based schemes. Specifically, a deep deterministic policy gradient (DDPG) method is proposed to address the continuous high-dimensional action space, and the prioritized experience replay is adopted to speed up the convergence process. Then, a twin delayed DDPG algorithm is designed based on this DRL framework. Numerical results confirm the effectiveness of proposed schemes compared with the benchmark methods.

Safety-oriented dynamic speed harmonization of mixed traffic flow in nonrecurrent congestion

Freeway nonrecurrent congestion adversely affects collision risks, travel time, emissions, and fuel consumption. Optimizing mixed flow involving Human-Driven Vehicles (HDVs) and Connected Automated vehicles (CAVs) is expected to exist as a challenging issue for a long time prior to full adoption of CAVs. To address this, a safety-oriented Dynamic Speed Harmonization (DSH) strategy for mixed traffic flow in nonrecurrent congestion is proposed in this study. This research utilizes a Deep Reinforcement Learning (DRL) framework to investigate the effects of DSH on the measures of effectiveness in microscopic simulation. A holistic evaluation of HDVs and CAVs integrated with DSH highlights the synergies and trade-offs across different metrics. The results reveal that the implementation of DSH can further improve safety and enhance mobility with increasing CAV penetration rates. While special events may exacerbate congestion, their impact can be mitigated to some extent through speed controls. Spatiotemporal patterns of speed variations at the bottleneck demonstrate that the DRL controller has the capability to dampen oscillations and smoothen traffic flow. A series of sensitivity analyses also indicate the adaptability of the agent under adverse weather scenarios, and the differences of surrogate safety measurements in response to various parametric thresholds.

The Impact of LiDAR Configuration on Goal-Based Navigation within a Deep Reinforcement Learning Framework.

Over the years, deep reinforcement learning (DRL) has shown great potential in mapless autonomous robot navigation and path planning. These DRL methods rely on robots equipped with different light detection and range (LiDAR) sensors with a wide field of view (FOV) configuration to perceive their environment. These types of LiDAR sensors are expensive and are not suitable for small-scale applications. In this paper, we address the performance effect of the LiDAR sensor configuration in DRL models. Our focus is on avoiding static obstacles ahead. We propose a novel approach that determines an initial FOV by calculating an angle of view using the sensor's width and the minimum safe distance required between the robot and the obstacle. The beams returned within the FOV, the robot's velocities, the robot's orientation to the goal point, and the distance to the goal point are used as the input state to generate new velocity values as the output action of the DRL. The cost function of collision avoidance and path planning is defined as the reward of the DRL model. To verify the performance of the proposed method, we adjusted the proposed FOV by ±10° giving a narrower and wider FOV. These new FOVs are trained to obtain collision avoidance and path planning DRL models to validate the proposed method. Our experimental setup shows that the LiDAR configuration with the computed angle of view as its FOV performs best with a success rate of 98% and a lower time complexity of 0.25 m/s. Additionally, using a Husky Robot, we demonstrate the model's good performance and applicability in the real world.

Deep Attention Q-Network for Personalized Treatment Recommendation.

Tailoring treatment for severely ill patients is crucial yet challenging to achieve optimal healthcare outcomes. Recent advances in reinforcement learning offer promising personalized treatment recommendations. However, they often rely solely on a patient's current physiological state, which may not accurately represent the true health status of the patient. This limitation hampers policy learning and evaluation, undermining the effectiveness of the treatment. In this study, we propose Deep Attention Q-Network for personalized treatment recommendation, leveraging the Transformer architecture within a deep reinforcement learning framework to efficiently integrate historical observations of patients. We evaluated our proposed method on two real-world datasets: sepsis and acute hypotension patients, demonstrating its superiority over state-of-the-art methods. The source code for our model is available at https://github.com/stevenmsm/RL-ICU-DAQN.

A multi-agent reinforcement learning-based longitudinal and lateral control of CAVs to improve traffic efficiency in a mandatory lane change scenario

Bottleneck areas are prone to severe traffic congestion due to the sudden drop in capacity. To improve traffic efficiency in the bottleneck area, this paper proposes a multi-agent deep reinforcement learning framework integrating collision avoidance strategies to improve traffic efficiency in a mandatory lane change scenario. The proposed method considers distance-keeping and lane-changing coordination in a connected autonomous vehicle (CAV) environment, by controlling vehicles' longitudinal and lateral movement to effectively reduce traffic congestion in a mandatory lane change scenario. This framework was trained and tested in a simulation environment that is the same as the natural driving environment. Compared with real-world data and the benchmark model (a Dueling Double Deep Q-Network-based model), the proposed model shows better performance in terms of average speed, travel time, throughput, and safety in the bottleneck area. The results show that the proposed model can effectively reduce traffic congestion and improve traffic efficiency in a mandatory lane change scenario.