Deep Reinforcement Learning System Research Articles

Multi-granularity coverage criteria for deep reinforcement learning systems

Deep reinforcement learning (DRL) systems are progressively being deployed in safety- and security-critical domains, such as self-driving cars and unmanned aerial vehicles, raising concerns about their trustworthiness. DRL, a integration of deep learning (DL) and reinforcement learning (RL) principles, represents a machine learning technique. Existing DL coverage criteria primarily focus on neuron coverage, overlooking the unique features of RL, thus falling short in assessing the test adequacy of DRL systems.This paper introduces the first set of coverage criteria designed to systematically measure the elements of DRL systems exercised by test inputs. DRL elements, including states, actions, and policies, are leveraged to define coverage criteria that consider multi-levels and multi-granularities. Furthermore, these coverage criteria undergo optimization through the application of combinatorial coverage principles. State coverage is employed as feedback to guide test case selection for DRL systems. Empirical studies have been conducted to assess the performance of the proposed coverage criteria on five well-known DRL environments. The experiments demonstrate the effectiveness of these coverage criteria in detecting unexpected behaviors, highlighting that the proposed test case selection guided by state coverage serves as an effective strategy.

Investigating the properties of neural network representations in reinforcement learning

In this paper we investigate the properties of representations learned by deep reinforcement learning systems. Much of the early work on representations for reinforcement learning focused on designing fixed-basis architectures to achieve properties thought to be desirable, such as orthogonality and sparsity. In contrast, the idea behind deep reinforcement learning methods is that the agent designer should not encode representational properties, but rather that the data stream should determine the properties of the representation—good representations emerge under appropriate training schemes. In this paper we bring these two perspectives together, empirically investigating the properties of representations that support transfer in reinforcement learning. We introduce and measure six representational properties over more than 25,000 agent-task settings. We consider Deep Q-learning agents with different auxiliary losses in a pixel-based navigation environment, with source and transfer tasks corresponding to different goal locations. We develop a method to better understand why some representations work better for transfer, through a systematic approach varying task similarity and measuring and correlating representation properties with transfer performance. We demonstrate the generality of the methodology by investigating representations learned by a Rainbow agent that successfully transfers across Atari 2600 game modes.

Security and Privacy Issues in Deep Reinforcement Learning: Threats and Countermeasures

Deep Reinforcement Learning (DRL) is an essential subfield of Artificial Intelligence (AI), where agents interact with environments to learn policies for solving complex tasks. In recent years, DRL has achieved remarkable breakthroughs in various tasks, including video games, robotic control, quantitative trading, and autonomous driving. Despite its accomplishments, security and privacy-related issues still prevent us from deploying trustworthy DRL applications. For example, by manipulating the environment, an attacker can influence an agent’s actions, misleading it to behave abnormally. Additionally, an attacker can infer private training data and environmental information by maliciously interacting with DRL models, causing a privacy breach. In this survey, we systematically investigate the recent progress of security and privacy issues in the context of DRL. First, we present a holistic review of security-related attacks within DRL systems from the perspectives of single-agent and multi-agent systems and review privacy-related attacks. Second, we review and classify defense methods used to address security-related challenges, including robust learning, anomaly detection, and game theory approaches. Third, we review and classify privacy-preserving technologies, including encryption, differential privacy, and policy confusion. We conclude the survey by discussing open issues and possible directions for future research in this field.

Coverage-guided fuzzing for deep reinforcement learning systems

While the past decade has witnessed a growing demand for employing deep reinforcement learning (DRL) in various domains to solve real-world problems, the reliability of DRL systems has become more of a concern. In particular, DRL agents are often trained on data from a potentially biased distribution over environmental settings, causing the trained agents to fail in certain cases despite high average-case performance. Hence, it is necessary and urgent to adequately test DRL agents to ensure the reliability of practical DRL systems. However, due to the fundamental difference in the programming paradigm and the development process, traditional software testing methodology cannot be applied directly to DRL systems. Given that, we introduce a novel testing framework for DRL systems, aiming to generate diverse test cases that can drive a DRL system to fail. Specifically, we design, implement and evaluate DRLFuzz, which is a coverage-guided fuzzing (CGF) framework for systematically testing DRL systems. Experimental results demonstrate that DRLFuzz can efficiently discover diverse failures in different DRL systems for various benchmark tasks. Compared with a random search baseline, DRLFuzz can generate 60% more failed cases in general. Additionally, the diversity of failed cases generated by DRLFuzz is increased by 4.6%∼14.1% in terms of mean pairwise distance (MPD). Furthermore, our experiments also indicate that the failed cases generated by DRLFuzz can be utilized to fine-tune the DRL agent to eliminate the failures resulting from inadequate exploration during training and thus improve the reliability of DRL systems.

Deep reinforcement learning with light-weight vision model for sequential robotic object sorting

Training an agent to generate cooperative joint learning policies for executing object sorting in cluttered scenes is complex. This paper presents a comparative model-free deep reinforcement learning (DRL) system built from variants of end-to-end lightweight deep neural networks that jointly complement three primitive policies (pushing, grasping, and placing) that operate from visual observations to actions. The end goal of the proposed DRL systems is to perform object sorting in very dynamic and complex environmental scenarios of cluttered and random objects (irregular and regular blocks) existing in different color variations. For this, we investigated DRL methods built from 12 custom instances of Pixel-wise Q-valued Critic Networks (PQCN) from four main backbone networks (DenseNet121, DenseNet169, MobileNetV3, and SqueezeNet) individually combined with custom fully convolutional neural networks (FCN) for learning the image-based observation space through affordance mapping paradigm while considering both dual and single transfer learning schemes. Additionally, we explored three categories of gradient-based optimization methods and considered custom reward functions with varying discount factors. The results show that the PQCN constructed by integrating DenseNet121 or MobileNetV3 with FCN and trained by a dual transfer learning scheme (with a discount factor of γ=0.70) yielded the best performance in most categories of our evaluation in the training phases. Overall, the PQCN-DenseNet121, when trained with a dual transfer learning process, yielded the best generalization (sorting success rate) in most of the evaluation metrics across all object categories in the testing phase. Furthermore, our study provides a new benchmark for validating deep reinforcement learning algorithms while sorting object blocks under varying degrees of complexity. This research finds practical relevance in industrial fields such as manufacturing and packaging, building construction, plant sorting, automobile assembly lines, and waste sorting. Here is the article’s source code: https://github.com/Emmanuel-Okafor/DRL_for_Object_Sorting.git.

A deep reinforcement learning system for the allocation of epidemic prevention materials based on DDPG

COVID-19 has spread rapidly around the world since the end of 2019. Consequently, the demand for epidemic prevention materials (e.g., medical-grade masks) has increased drastically. The medical-grade masks have become a necessary item for everyone. Referring to the shortage of medical-grade masks in Taiwan, the government collected and managed them at the early stage and sold them for a fixed price. However, the government must distribute masks around the country to prevent the movement of people who can cause infection with COVID-19. Moreover, the demands for medical supplies during the pandemic have become more complex and dynamic. This study proposes a robust system for allocating epidemic prevention materials. The proposed approach adopts the reinforcement learning framework, which takes the daily supply and demand for masks as the environment, with the Deep Deterministic Policy Gradient (DDPG) algorithm for agent updates and the daily shortage as rewards and punishments. The proposed system is compared with the traditional machine learning approach used for supply chain demand forecasting through experiments. The results indicate that our proposed method is superior regarding the number of pharmacies with over-stocked masks and rewards. Moreover, our proposed system performs consistently under different total numbers of masks.

Designing A WSNs-based Smart Home Monitoring System through Deep Reinforcement Learning

The technology of smart home systems has developed rapidly and provides convenience for human life. Several smart home technologies, especially monitoring systems, have been developed by integrating several aspects, including security systems, fuzzy methods, and energy saving methods. However, the problem is how to build a smart home system that is accurate, convenient, and low-cost. In this research, the development of a smart home monitoring system that integrates wireless sensor networks (WSNs) and deep reinforcement learning (DRL) is carried out based on three parameters, i.e. temperature, humidity and CO2 level. The experimental method is carried out by (1) validating the accuracy quality of WSNs; (2) determining the best model implemented in the system; and (3) measuring the quality of the DRL system on the smart home monitoring system. Based on the test results, several indicators were obtained: (1) WSN testing resulted in an accuracy of 98.52%; (2) the accuracy of the modeling results implemented in the system is 97.70%; and (3) DRL system test on the smart home monitoring system through 21 test scenarios resulted in an accuracy of 95.52%. The indicators of testing this smart monitoring system prove that the developed system provides the advantages of accuracy, ease of use, and low cost.

Selective real‐time adversarial perturbations against deep reinforcement learning agents

AbstractRecent work has shown that deep reinforcement learning (DRL) is vulnerable to adversarial attacks, so that exploiting vulnerabilities in DRL systems through adversarial attack techniques has become a necessary prerequisite for building robust DRL systems. Compared to traditional deep learning systems, DRL systems are characterised by long sequential decisions rather than one‐step decision, so attackers must perform multi‐step attacks on them. To successfully attack a DRL system, the number of attacks must be minimised to avoid detecting by the victim agent and to ensure the effectiveness of the attack. Some selective attack methods proposed in recent researches, that is, attacking an agent at partial time steps, are not applicable to real‐time attack scenarios, although they can avoid detecting by the victim agent. A real‐time selective attack method that is applicable to environments with discrete action spaces is proposed. Firstly, the optimal attack threshold T for performing selective attacks in the environment Env is determined. Then, the observation states corresponding to when the value of the action preference function of the victim agent in multiple eposides exceeds the threshold T are added to the training set according to this threshold. Finally, a universal perturbation is generated based on this training set, and it is used to perform real‐time selective attacks on the victim agent. Comparative experiments show that our attack method can perform real‐time attacks while maintaining the attack effect and stealthiness.

Satellite and Aerial Image Restoration Using Deep Reinforcement Learning

In this paper, we present a deep reinforcement learning-based method for effectively denoising satellite and aerial imagery data. Noise of various kinds and with varying noise levels contaminates satellite imagery data. The image’s quality and readability suffer when there is noise present. Therefore, it is crucial to create a network that can effectively and efficiently remove noise from the image while also preserving its quality and signal components. This paper evaluates the denoising capabilities of the deep reinforcement learning system. The proposed network is trained using the training set from the “dataset of object detection in aerial images (DOTA) dataset,” and its hyperparameters were adjusted for optimum performance. The training set from the aforementioned dataset was used to train the proposed network. The trained network was given the test set of unseen images for denoising. Statistical denoising, a common denoising technique, was used on the test dataset, and the outcomes were assessed. The same unseen images were also given to existing CNN-based denoising algorithms like denoising using CNN (DnCNN), U-shaped DnCNN (UDnCNN), and dilated U-shaped DnCNN (DUDnCNN), designed specifically for image denoising. Runtime and structural similarity index (SSIM) as well as peak signal-to-noise ratio have both been used as evaluation metrics to compare the effectiveness of various approaches. It is discovered that, when comparing the performance of various systems, the suggested system outperforms both statistical- and CNN-based denoising in terms of the evaluation metrics, PSNR and SSIM.

Deep Reinforcement Learning Verification: A Survey

Deep reinforcement learning (DRL) has proven capable of superhuman performance on many complex tasks. To achieve this success, DRL algorithms train a decision-making agent to select the actions that maximize some long-term performance measure. In many consequential real-world domains, however, optimal performance is not enough to justify an algorithm’s use—for example, sometimes a system’s robustness, stability, or safety must be rigorously ensured. Thus, methods for verifying DRL systems have emerged. These algorithms can guarantee a system’s properties over an infinite set of inputs, but the task is not trivial. DRL relies on deep neural networks (DNNs). DNNs are often referred to as “black boxes” because examining their respective structures does not elucidate their decision-making processes. Moreover, the sequential nature of the problems DRL is used to solve promotes significant scalability challenges. Finally, because DRL environments are often stochastic, verification methods must account for probabilistic behavior. To address these complications, a new subfield has emerged. In this survey, we establish the foundations of DRL and DRL verification, define a taxonomy for DRL verification methods, describe approaches for dealing with stochasticity, characterize considerations related to writing specifications, enumerate common testing tasks/environments, and detail opportunities for future research.

Research on Prediction and Regulation of Thermal Dissatisfaction Rate Based on Personalized Differences

Thermal discomfort body language has been shown to be a psychological representation of personnel’s particular thermal comfort. Individual thermal comfort differences are ignored in public building settings with random personnel flow. To solve this issue, we suggested a Bayesian group thermal dissatisfaction rate prediction model based on thermal discomfort body language expression and subsequently implemented intelligent indoor temperature and humidity control. The PMV-PPD model was utilized to represent the group’s overall thermal comfort and to create a prior distribution of thermal dissatisfaction rate. To acquire the dynamic distribution of temperature discomfort body language, data on thermal discomfort body language expression were collected in a real-world office setting experiment. Based on Bayesian theory, we used personalized thermal discomfort body language expressions to modify the group’s universal thermal comfort and realized the assessment of the thermal dissatisfaction rate by combining commonality and personalization. Finally, a deep reinforcement learning system was employed to achieve intelligent indoor temperature and humidity control. The results show that when commonality and personalized thermal comfort differences are combined, real-time prediction of thermal dissatisfaction rate has high prediction accuracy and good model performance, and the prediction model provides a reference basis for reasonable indoor temperature and humidity settings.

A Distributed-GPU Deep Reinforcement Learning System for Solving Large Graph Optimization Problems

Graph optimization problems (such as minimum vertex cover, maximum cut, traveling salesman problems) appear in many fields including social sciences, power systems, chemistry, and bioinformatics. Recently, deep reinforcement learning (DRL) has shown success in automatically learning good heuristics to solve graph optimization problems. However, the existing RL systems either do not support graph RL environments or do not support multiple or many GPUs in a distributed setting. This has compromised the ability of reinforcement learning in solving large-scale graph optimization problems due to lack of parallelization and high scalability. To address the challenges of parallelization and scalability, we develop RL4GO , a high-performance distributed-GPU DRL framework for solving graph optimization problems. RL4GO focuses on a class of computationally demanding RL problems, where both the RL environment and policy model are highly computation intensive. Traditional reinforcement learning systems often assume either the RL environment is of low time complexity or the policy model is small. In this work, we distribute large-scale graphs across distributed GPUs and use the spatial parallelism and data parallelism to achieve scalable performance. We compare and analyze the performance of the spatial parallelism and data parallelism and show their differences. To support graph neural network (GNN) layers that take as input data samples partitioned across distributed GPUs, we design parallel mathematical kernels to perform operations on distributed 3D sparse and 3D dense tensors. To handle costly RL environments, we design a parallel graph environment to scale up all RL-environment-related operations. By combining the scalable GNN layers with the scalable RL environment, we are able to develop high-performance RL4GO training and inference algorithms in parallel. Furthermore, we propose two optimization techniques—replay buffer on-the-fly graph generation and adaptive multiple-node selection—to minimize the spatial cost and accelerate reinforcement learning. This work also conducts in-depth analyses of parallel efficiency and memory cost and shows that the designed RL4GO algorithms are scalable on numerous distributed GPUs. Evaluations on large-scale graphs show that (1) RL4GO training and inference can achieve good parallel efficiency on 192 GPUs, (2) its training time can be 18 times faster than the state-of-the-art Gorila distributed RL framework [ 34 ], and (3) its inference performance achieves a 26 times improvement over Gorila.

A Survey on Deep Reinforcement Learning Algorithms for Robotic Manipulation.

Robotic manipulation challenges, such as grasping and object manipulation, have been tackled successfully with the help of deep reinforcement learning systems. We give an overview of the recent advances in deep reinforcement learning algorithms for robotic manipulation tasks in this review. We begin by outlining the fundamental ideas of reinforcement learning and the parts of a reinforcement learning system. The many deep reinforcement learning algorithms, such as value-based methods, policy-based methods, and actor-critic approaches, that have been suggested for robotic manipulation tasks are then covered. We also examine the numerous issues that have arisen when applying these algorithms to robotics tasks, as well as the various solutions that have been put forth to deal with these issues. Finally, we highlight several unsolved research issues and talk about possible future directions for the subject.

Framework for Connected and Automated Bus Rapid Transit with Sectionalized Speed Guidance based on deep reinforcement learning: Field test in Sejong City

Nowadays, Automated Vehicle (AV) technology is gaining attention as a candidate to improve the efficiency of Bus Rapid Transit (BRT) systems. However, there are still some challenges in AV technology including limited perception range and lack of cooperation capability in mixed traffic situations with drivers. The emerging Connected and Automated Vehicles (CAVs) and Cooperative Intelligent Transportation System (C-ITS) offer an unprecedented opportunity to solve such challenges. As a result, this study presents a framework for Connected and Automated BRT (CA-BRT), including a cloud-based architecture and a deep reinforcement learning system for Sectionalized Speed Guidance (SSG) system designed for CAVs. The proposed framework is field-tested in Sejong City in South Korea, where there are various road environments such as bus stops, overpasses, underground tunnels, intersections, and crosswalks. The driving performance of the proposed system is compared with different types of control scenarios, and the results from the field tests show that the proposed system improves the driving performance of the AVs in various aspects including driving safety, ride comfort, and energy efficiency with downstream information obtained from road infrastructures.

DeepAPP: A Deep Reinforcement Learning Framework for Mobile Application Usage Prediction

This paper aims to predict a set of apps a user will open on her mobile device in the next time slot. Such an information is essential for many smartphone operations, e.g., app pre-loading and content pre-caching, to improve user experience. However, it is hard to build an explicit model that accurately captures the complex environment context and predicts a set of apps at one time. This paper presents a deep reinforcement learning framework, named as DeepAPP, which learns a model-free predictive neural network from historical app usage data. Meanwhile, an online updating strategy is designed to adapt the predictive network to the time-varying app usage behavior. To transform DeepAPP into a practical deep reinforcement learning system, several challenges are addressed by developing a context representation method for complex contextual environment, a general agent for overcoming data sparsity and a lightweight personalized agent for minimizing the prediction time. Extensive experiments on a large-scale anonymized app usage dataset reveal that DeepAPP provides high accuracy (precision 70.6 percent and recall of 62.4 percent) and reduces the prediction time of the state-of-the-art by 6.58×. A field experiment of 29 participants demonstrates DeepAPP can effectively reduce launch time of apps.

Safe Decision Controller for Autonomous DrivingBased on Deep Reinforcement Learning inNondeterministic Environment.

Autonomous driving systems are crucial complicated cyber-physical systems that combine physical environment awareness with cognitive computing. Deep reinforcement learning is currently commonly used in the decision-making of such systems. However, black-box-based deep reinforcement learning systems do not guarantee system safety and the interpretability of the reward-function settings in the face of complex environments and the influence of uncontrolled uncertainties. Therefore, a formal security reinforcement learning method is proposed. First, we propose an environmental modeling approach based on the influence of nondeterministic environmental factors, which enables the precise quantification of environmental issues. Second, we use the environment model to formalize the reward machine's structure, which is used to guide the reward-function setting in reinforcement learning. Third, we generate a control barrier function to ensure a safer state behavior policy for reinforcement learning. Finally, we verify the method's effectiveness in intelligent driving using overtaking and lane-changing scenarios.

Information Retrieval and Optimization in Distribution and Logistics Management Using Deep Reinforcement Learning

Resource balance is one of the most critical concerns in the existing logistic domain within dynamic transport networks. Modern solutions are used to maximize demand and supply prediction in collaboration with these problems. However, the great difficulty of transportation networks, profound uncertainties of potential demand and availability, and non-convex market limits make conventional resource management main paths. Hence, this paper proposes an integrated deep reinforcement learning-based logistics management model (DELLMM) to increase and optimize the logistic distribution. An optimization approach can be used in inventors and price control applications. This research methodology gives the fundamentals of information retrieval and the scope of blockchain integration. The conceptual framework of use cases for an efficient logistic management system with blockchain has been discussed. This research designs the deep reinforcement learning system that can boost optimization and other business operations due to impressive improvements in generic self-learning algorithms for optimal management. Thus, the experimental results show that DELLMM improves logistics management and optimized distribution compared to other methods with the highest operability of 94.35%, latency reduction of 97.12%, efficiency of 98.01%, trust enhancement of 96.37%, and sustainability of 97.80%.

Attacking Deep Reinforcement Learning With Decoupled Adversarial Policy

While Deep Reinforcement Learning (DRL) has achieved outstanding performance in extensive applications, exploiting its vulnerability with adversarial attacks is essential towards building robust DRL systems. In this work, we aim to propose a novel Decoupled Adversarial Policy (DAP) for attacking the DRL mechanism, whereas the adversarial agent can decompose the adversarial policy into two separate sub-policies: 1) the switch policy which determines if an attacker should launch the attack, and 2) the lure policy which determines the action an attacker induces the victim to take. If the adversarial agent samples an injection action from the switch policy, the attacker can query the pre-constructed database for universal perturbation in the real-time manner, misleading the victim to take the induced action sampled from the lure policy. To train the adversarial agent to learn DAP, we utilize those samples wherein both of the sub-actions from DAP are not restricted by each other or by the external constraint, but can actually affect the attacker’s behaviors. Therefore, we propose trajectory clipping and padding in data pruning, and Decoupled Proximal Policy Optimization (DPPO) in optimizing. Extensive experiments on different Atari games demonstrate the effectiveness of our proposed method. In addition, it can simultaneously implement the real-time and few-steps attack, which outperforms the existing counterparts.

Deep Q-network for social robotics using emotional social signals.

Social robotics represents a branch of human-robot interaction dedicated to developing systems to control the robots to operate in unstructured environments with the presence of human beings. Social robots must interact with human beings by understanding social signals and responding appropriately to them. Most social robots are still pre-programmed, not having great ability to learn and respond with actions adequate during an interaction with humans. Recently more elaborate methods use body movements, gaze direction, and body language. However, these methods generally neglect vital signs present during an interaction, such as the human emotional state. In this article, we address the problem of developing a system to turn a robot able to decide, autonomously, what behaviors to emit in the function of the human emotional state. From one side, the use of Reinforcement Learning (RL) represents a way for social robots to learn advanced models of social cognition, following a self-learning paradigm, using characteristics automatically extracted from high-dimensional sensory information. On the other side, Deep Learning (DL) models can help the robots to capture information from the environment, abstracting complex patterns from the visual information. The combination of these two techniques is known as Deep Reinforcement Learning (DRL). The purpose of this work is the development of a DRL system to promote a natural and socially acceptable interaction among humans and robots. For this, we propose an architecture, Social Robotics Deep Q-Network (SocialDQN), for teaching social robots to behave and interact appropriately with humans based on social signals, especially on human emotional states. This constitutes a relevant contribution for the area since the social signals must not only be recognized by the robot but help him to take action appropriated according to the situation presented. Characteristics extracted from people’s faces are considered for extracting the human emotional state aiming to improve the robot perception. The development and validation of the system are carried out with the support of SimDRLSR simulator. Results obtained through several tests demonstrate that the system learned satisfactorily to maximize the rewards, and consequently, the robot behaves in a socially acceptable way.

Adversarial Attacks on Heterogeneous Multi-Agent Deep Reinforcement Learning System with Time-Delayed Data Transmission

This paper studies the gradient-based adversarial attacks on cluster-based, heterogeneous, multi-agent, deep reinforcement learning (MADRL) systems with time-delayed data transmission. The structure of the MADRL system consists of various clusters of agents. The deep Q-network (DQN) architecture presents the first cluster’s agent structure. The other clusters are considered as the environment of the first cluster’s DQN agent. We introduce two novel observations in data transmission, termed on-time and time-delay observations. The proposed observations are considered when the data transmission channel is idle, and the data is transmitted on time or delayed. By considering the distance between the neighboring agents, we present a novel immediate reward function by appending a distance-based reward to the previously utilized reward to improve the MADRL system performance. We consider three types of gradient-based attacks to investigate the robustness of the proposed system data transmission. Two defense methods are proposed to reduce the effects of the discussed malicious attacks. We have rigorously shown the system performance based on the DQN loss and the team reward for the entire team of agents. Moreover, the effects of the various attacks before and after using defense algorithms are demonstrated. The theoretical results are illustrated and verified with simulation examples.