Deep Reinforcement Learning Framework Research Articles

Heuristics Integrated Deep Reinforcement Learning for Online 3D Bin Packing

Online 3D Bin Packing Problem (3D-BPP) has a wide range of industrial applications and there is an emerging research interest in learning optimal bin packing policy and deploying it for real logistics applications. From the heuristic methods to the deep reinforcement learning (DRL) methods, the previous works have proposed many solutions to solve the online 3D-BPP. However, none of them have studied what and how heuristics can be modelled into DRL to build a more effective and practical bin packing pipeline. In this work, we thoroughly investigate what heuristics can be used in online 3D-BPP and how to effectively integrate the heuristics with the DRL. First, we design 3 different heuristics based on the physical rules of the real world and the experiences of the human packers, including the Physics-Heuristics, the Packing-Heuristics and the Unpacking-Heuristics. Second, we model the 3 types of heuristics into the DRL framework and propose a novel heuristic DRL method to solve the online 3D-BPP. Extensive experimental results show that our method achieves state-of-the-art bin packing performance and the resulting real-world system is able to reliably finish the bin packing task in real logistics scenarios. Supplementary video is available at https://www.youtube.com/watch?v=x8GpmEELq18. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The rapid growth of e-commerce has significantly increased the burden of human packers in logistic warehouses, where the workers need to pick the products from a conveyor and pack them into bins (i.e. the online 3D bin packing). Thus it is of great importance to develop intelligent robotic systems to replace human labor, which is a long-standing topic in the field of control and automation science. This paper makes a substantial contribution to the related field by studying the online 3D bin packing in terms of both the theory and practice. On the one hand, the simulated experiments suggest that the presented algorithm significantly improves the space utilization of bin packing. On the other hand, the robotic system developed based on the proposed method can favourably finish the bin packing task in real logistics scenarios, demonstrating the practical use of our approach. Consequently, the approach proposed in this paper is totally applicable in logistic warehouses and is promising to drastically improve the working efficiency of the product packing in real warehouses. In the future, we will extend the presented approach to pack irregular-shaped objects and then facilitate more logistics applications.

Long-Term Tracking of Evasive Urban Target Based on Intention Inference and Deep Reinforcement Learning.

Unmanned aerial vehicles (UAVs) have been widely used in urban target-tracking tasks, where long-term tracking of evasive targets is of great significance for public safety. However, the tracked targets are easily lost due to the evasive behavior of the targets and the unstructured characteristics of the urban environment. To address this issue, this article proposes a hybrid target-tracking approach based on target intention inference and deep reinforcement learning (DRL). First, a target intention inference model based on convolution neural networks (CNNs) is built to infer target intentions by fusing urban environment information and observed target trajectory. Then, the prediction of the target trajectory can be inspired by the inferred target intentions, which can further provide effective guidance to the target search process. In order to fully explore the policy space, the target search policy is developed under a DRL framework, where the search policy is modeled as a deep neural network (DNN) and trained by interacting with the task environment. The simulation results show that the inference of the target intentions can effectively guide the UAV to search for the target and significantly improve the target-tracking performance. Meanwhile, the generalization results indicate that the proposed DRL-based search policy has high robustness to the uncertainty of the target behavior.

A study of natural robustness of deep reinforcement learning algorithms towards adversarial perturbations

Deep reinforcement learning (DRL) has been shown to have numerous potential applications in the real world. However, DRL algorithms are still extremely sensitive to noise and adversarial perturbations, hence inhibiting the deployment of RL in many real-life applications. Analyzing the robustness of DRL algorithms to adversarial attacks is an important prerequisite to enabling the widespread adoption of DRL algorithms. Common perturbations on DRL frameworks during test time include perturbations to the observation and the action channel. Compared with observation channel attacks, action channel attacks are less studied; hence, few comparisons exist that compare the effectiveness of these attacks in DRL literature. In this work, we examined the effectiveness of these two paradigms of attacks on common DRL algorithms and studied the natural robustness of DRL algorithms towards various adversarial attacks in hopes of gaining insights into the individual response of each type of algorithm under different attack conditions.

Multi-Agent Deep Reinforcement Learning for Spectral Efficiency Optimization in Vehicular Optical Camera Communications

In this paper, we propose a vehicular optical camera communication system that can meet low bit error rate (BER) and ultra-low latency constraints. First, we formulate a sum spectral efficiency optimization problem that aims at finding the speed of vehicles and the modulation order that maximizes the sum spectral efficiency subject to reliability and latency constraints. This problem is mixed-integer programming with nonlinear constraints, and even for a small set of modulation orders, is NP-hard. To overcome the entailed high computational and time complexity which prevents its solution with traditional methods, we first model the optimization problem as a partially observable Markov decision process. We then solve it using an independent Q-learning framework, where each vehicle acts as an independent agent. Since the state-action space is large we then adopt deep reinforcement learning (DRL) to solve it efficiently. As the problem is constrained, we employ the Lagrange relaxation approach prior to solving it using the DRL framework. Simulation results demonstrate that the proposed DRL-based optimization scheme can effectively learn how to maximize the sum spectral efficiency while satisfying the BER and ultra-low latency constraints. The evaluation further shows that our scheme can achieve superior performance compared to radio frequency-based vehicular communication systems and other vehicular OCC variants of our scheme.

Structure-Enhanced DRL for Optimal Transmission Scheduling

Remote state estimation of large-scale distributed dynamic processes plays an important role in Industry 4.0 applications. In this paper, we focus on the transmission scheduling problem of a remote estimation system. First, we derive some structural properties of the optimal sensor scheduling policy over fading channels. Then, building on these theoretical guidelines, we develop a structure-enhanced deep reinforcement learning (DRL) framework for optimal scheduling of the system to achieve the minimum overall estimation mean-square error (MSE). In particular, we propose a structure-enhanced action selection method, which tends to select actions that obey the policy structure. This explores the action space more effectively and enhances the learning efficiency of DRL agents. Furthermore, we introduce a structure-enhanced loss function to add penalties to actions that do not follow the policy structure. The new loss function guides the DRL to converge to the optimal policy structure quickly. Our numerical experiments illustrate that the proposed structure-enhanced DRL algorithms can save the training time by 50% and reduce the remote estimation MSE by 10% to 25%, when compared to benchmark DRL algorithms. In addition, we show that the derived structural properties exist in a wide range of dynamic scheduling problems that go beyond remote state estimation.

Combining Reinforcement Learning and In Situ Current Velocity Measurements for Efficient UUV Guidance

Conventional guidance algorithms for unmanned underwater vehicles commonly operate on geometric principles that do not directly take into account local environmental conditions. We describe here a deep reinforcement learning (RL) framework that partially addresses this issue by generating energy-minimizing guidance systems. Specifically, the proposed framework outputs trained policy networks (agents) capable of mapping histories of local current velocity measurements to heading and depth commands that yield nearly-optimal, minimum-energy transits through spatially varying current fields. Numerical experiments in three spatial dimensions on realistic ocean currents, where direction and speed vary non-monotonically in depth, demonstrate that trained agents reduce the energy costs of transits when compared against dead-reckoning, a common in-service guidance heuristic. In particular, numerical results suggest that trained agents learn an explore-exploit strategy according to which they initially execute an exploratory dive, followed by a return to a depth with optimally favorable currents, which they exploit for the remainder of the transit.

Learning-based Integrated Cooperative Motion Planning and Control of Multi-AUVs

This paper introduces a learning-based solution tailored for the integrated motion planning and control of Multiple Autonomous Underwater Vehicles (AUVs). Tackling the complexities of cooperative motion planning, encompassing tasks such as waypoint tracking and self/obstacle collision avoidance, becomes challenging in a rule-based algorithmic paradigm due to the diverse and unpredictable situations encountered, necessitating a proliferation of if-then conditions in the implementation. Recognizing the limitations of traditional approaches that are heavily dependent on models and geometry of the system, our solution ofers an innovative paradigm shift. This study proposes an integrated motion planning and control strategy that leverages sensor and navigation outputs to generate longitudinal and lateral control outputs dynamically. At the heart of this cutting-edge methodology lies a continuous action Deep Reinforcement Learning (DRL) framework, specifically based on the Twin Delayed Deep Deterministic Policy Gradient (TD3). This algorithm surpasses traditional limitations by embodying an elaborated reward function, enabling the seamless execution of control actions essential for maneuvering multiple AUVs. Through simulation tests under both nominal and perturbed conditions, considering obstacles and underwater current disturbances, the obtained results demonstrate the feasibility and robustness of the proposed technique.

Enhancing High-Speed Cruising Performance of Autonomous Vehicles Through Integrated Deep Reinforcement Learning Framework

Enhancing High-Speed Cruising Performance of Autonomous Vehicles Through Integrated Deep Reinforcement Learning Framework

A Task-Adaptive Deep Reinforcement Learning Framework for Dual-Arm Robot Manipulation

A Task-Adaptive Deep Reinforcement Learning Framework for Dual-Arm Robot Manipulation

An Improved SAC-Based Deep Reinforcement Learning Framework for Collaborative Pushing and Grasping in Underwater Environments

An Improved SAC-Based Deep Reinforcement Learning Framework for Collaborative Pushing and Grasping in Underwater Environments

TinyDeepUAV: A Tiny Deep Reinforcement Learning Framework for UAV Task Offloading in Edge-Based Consumer Electronics

TinyDeepUAV: A Tiny Deep Reinforcement Learning Framework for UAV Task Offloading in Edge-Based Consumer Electronics

Intelligent Predetermination of Generator Tripping Scheme: Knowledge Fusion-based Deep Reinforcement Learning Framework

Intelligent Predetermination of Generator Tripping Scheme: Knowledge Fusion-based Deep Reinforcement Learning Framework

A Deep Reinforcement Learning Framework for Control of Robotic Manipulators in Simulated Environments

A Deep Reinforcement Learning Framework for Control of Robotic Manipulators in Simulated Environments

HGFF: A Deep Reinforcement Learning Framework for Lifetime Maximization in Wireless Sensor Networks

HGFF: A Deep Reinforcement Learning Framework for Lifetime Maximization in Wireless Sensor Networks

An Azimuth Aware Deep Reinforcement Learning Framework for Active SAR Target Recognition

An Azimuth Aware Deep Reinforcement Learning Framework for Active SAR Target Recognition

Joint MCS Adaptation and RB Allocation in Cellular Networks Based on Deep Reinforcement Learning With Stable Matching

Joint modulation-coding scheme (MCS) adaptation and resource block (RB) allocation is an effective approach to guarantee different quality of service (QoS) requirements of all UEs under dynamic network environments. In this paper, we consider a fifth generation (5 G) cellular network with time-varying wireless channels, in which the BS serves multiple user equipments (UEs) under limited available RBs. We aim to minimize the total RB consumption subject to the rigorous constraints of each UE's QoS requirement. To attain this objective, this paper puts forth an online learning technique, referred to as integrated Deep Reinforcement learning and stable Matching (DeepRM), in the sense that the MCS adaptation and RB allocation decisions are conducted without acquiring the real-time channel quality indicator (CQI) feedback. DeepRM is a closed-loop framework, where the output of deep reinforcement learning (DRL) is imported into the stable matching to guide optimal RB allocation whilst the output of stable matching is fed into the DRL framework to assist efficient MCS decision-making. Specifically, in DeepRM, we first develop a powerful DRL algorithm, termed as Action-and-Reward Branching Deep Q-network (ARBDQ), by incorporating the action branch architecture into conventional DRL and modifying the traditional deep neural network training mechanism, to perform judicious MCS decisions on different links in parallel. Then, a new many-to-one stable matching algorithm, called adaptive deferred acceptance, is exploited to dynamically adjust the RB quota of each UE in a computationally efficient fashion. Simulation results demonstrate that compared with ACO-HM, OLLA-ADA, and ARBDQ-Random algorithms, DeepRM induces much less RB consumption while guaranteeing the QoS requirements of all UEs in various network scenarios. Furthermore, under miscellaneous QoS requirement, number of UEs, and CQI reporting period setups, DeepRM is more robust than other baselines.

Solving flexible job shop scheduling problems via deep reinforcement learning

Flexible job shop scheduling problem (FJSSP), as a variant of the job shop scheduling problem, has a larger solution space. Researchers are always looking for good methods to solve this problem. In recent years, the deep reinforcement learning (DRL) has been applied to solve various shop scheduling problems due to its advantages that fast solving speed and strong generalization ability. In this paper, we first propose a new DRL framework to realize representation learning and policy learning. The new framework adopts a lightweight multi-layer perceptron (MLP) as the state embedding network to extract state information, which reduces the computational complexity of the algorithm to some extent. Next, we design a new state representation and define a new action space. The new state representation can directly reflect the state features of candidate actions, which is conducive for the agent to capture more effective state information and improve its decision-making ability. The new definition of action space can solve the two subproblems of the FJSSP simultaneously with only one action space. Finally, we evaluate the performance of the policy model on four public datasets: Barnes dataset, Brandimarte dataset, Dauzere dataset and Hurink dataset. Extensive experimental results on these public datasets show that the proposed method achieves a better compromise in terms of optimization ability and applicability compared to the composite priority dispatching rules and the existing state-of-the-art models.

Double RISs assisted task offloading for NOMA-MEC with action-constrained deep reinforcement learning

Reconfigurable intelligent surface (RIS) is expected to enhance task offloading performance in non-line-of-sight mobile edge computing (MEC) scenarios. This paper aims at reducing the long-term energy consumption for task offloading in a MEC system assisted by double RISs while considering non-orthogonal multiple access (NOMA). Due to the non-convexity of the formulated problem, we propose an action-constrained deep reinforcement learning (DRL) framework based on twin delayed deep deterministic policy gradient (TD3) algorithm that includes inner and outer optimization processes, which greatly reduces the action space size of the DRL agent, making it easy to implement and achieve fast convergence. During the inner optimization phase, by utilizing theoretical derivations, we propose a low-complexity method to optimally derive the transmit power, the local computing frequency, and the computation resource allocation in the base station. In the outer optimization phase, building on the solution derived from the inner optimization, we further use the TD3 algorithm to jointly determine the phase shifting of RIS elements, the offloading ratio, and the transmission time for each time slot. Experiment results demonstrate that the proposed algorithm achieves rapid convergence performance. In comparison to single RIS assisted offloading, the double RISs assisted offloading scheme proposed in this paper can reduce energy consumption by 42.8% on average.

Joint Task Offloading and Resource Allocation for Intelligent Reflecting Surface-Aided Integrated Sensing and Communication Systems Using Deep Reinforcement Learning Algorithm.

This paper investigates an intelligent reflecting surface (IRS)-aided integrated sensing and communication (ISAC) framework to cope with the problem of spectrum scarcity and poor wireless environment. The main goal of the proposed framework in this work is to optimize the overall performance of the system, including sensing, communication, and computational offloading. We aim to achieve the trade-off between system performance and overhead by optimizing spectrum and computing resource allocation. On the one hand, the joint design of transmit beamforming and phase shift matrices can enhance the radar sensing quality and increase the communication data rate. On the other hand, task offloading and computation resource allocation optimize energy consumption and delay. Due to the coupled and high dimension optimization variables, the optimization problem is non-convex and NP-hard. Meanwhile, given the dynamic wireless channel condition, we formulate the optimization design as a Markov decision process. To tackle this complex optimization problem, we proposed two innovative deep reinforcement learning (DRL)-based schemes. Specifically, a deep deterministic policy gradient (DDPG) method is proposed to address the continuous high-dimensional action space, and the prioritized experience replay is adopted to speed up the convergence process. Then, a twin delayed DDPG algorithm is designed based on this DRL framework. Numerical results confirm the effectiveness of proposed schemes compared with the benchmark methods.

Safety-oriented dynamic speed harmonization of mixed traffic flow in nonrecurrent congestion

Freeway nonrecurrent congestion adversely affects collision risks, travel time, emissions, and fuel consumption. Optimizing mixed flow involving Human-Driven Vehicles (HDVs) and Connected Automated vehicles (CAVs) is expected to exist as a challenging issue for a long time prior to full adoption of CAVs. To address this, a safety-oriented Dynamic Speed Harmonization (DSH) strategy for mixed traffic flow in nonrecurrent congestion is proposed in this study. This research utilizes a Deep Reinforcement Learning (DRL) framework to investigate the effects of DSH on the measures of effectiveness in microscopic simulation. A holistic evaluation of HDVs and CAVs integrated with DSH highlights the synergies and trade-offs across different metrics. The results reveal that the implementation of DSH can further improve safety and enhance mobility with increasing CAV penetration rates. While special events may exacerbate congestion, their impact can be mitigated to some extent through speed controls. Spatiotemporal patterns of speed variations at the bottleneck demonstrate that the DRL controller has the capability to dampen oscillations and smoothen traffic flow. A series of sensitivity analyses also indicate the adaptability of the agent under adverse weather scenarios, and the differences of surrogate safety measurements in response to various parametric thresholds.