Deep Reinforcement Learning Approach Research Articles

A hybrid deep reinforcement learning approach for a proactive transshipment of fresh food in the online–offline channel system

To reduce the waste of fresh foods, one of the e-commerce companies in South Korea utilizes lateral transshipment in the network of online platforms and offline shops, which is called the online–offline channel system (OOCS). Even though the OOCS has achieved success in real practice, there is room for further study on this system with regard to deriving a transshipment policy. For this reason, this study aims to develop a solution approach that could derive a promising policy and analyze the impacts of transshipment in the OOCS. The main contributions are summarized as follows. First, we propose a model to deal with the proactive transshipment of perishable products in the OOCS. In particular, this is the first study that introduces the concept of the heterogeneous shelf life considering different properties of online and offline channels. Second, we develop the hybrid deep reinforcement learning (DRL) approach by combining the soft actor–critic algorithm with two novel acceleration methods. The developed method could obtain a promising policy without assumptions about demand distribution and mitigate computational burdens by reducing action spaces. On a set of experiments carried out on real-world demand data, the transshipment policy derived from the hybrid DRL approach could obtain the best profit compared to existing algorithms. Third, we examine the impacts of transshipment by differing types of demand and varying the unit transshipment cost parameter. We find that transshipment substantially reduces the outdating cost by allowing the offline channel to make good use of the old products that will be discarded in the online channel, which is new to the literature.

A deep reinforcement learning approach for dynamic task scheduling of flight tests

A deep reinforcement learning approach for dynamic task scheduling of flight tests

Energy and throughput efficient mobile wireless sensor networks: A deep reinforcement learning approach

AbstractThe efficient development of Mobile Wireless Sensor Networks (MWSNs) relies heavily on optimizing two key parameters: Throughput and Energy Consumption. The proposed work investigates network connectivity issues with MWSN and proposes two routing algorithms, namely Self‐Organising Maps based‐Optimised Link State Routing (SOM‐OLSR) and Deep Reinforcement Learning based‐Optimised Link State Routing (DRL‐OLSR) for MWSNs. The primary objective of the proposed algorithms is to achieve energy‐efficient routing while maximizing throughput. The proposed algorithms are evaluated through simulations by considering various performance metrics, including connection probability (CP), end‐to‐end delay, overhead, network throughput, and energy consumption. The simulation analysis is discussed under three scenarios. The first scenario undertakes ‘no optimisation’, the second considers SOM‐OLSR, and the third undertakes DRL‐OLSR. A comparison between DRL‐OLSR and SOM‐OLSR reveals that the former surpasses the latter in terms of low latency and prolonged network lifetime. Specifically, DRL‐OLSR demonstrates a 47% increase in throughput, a 67% reduction in energy consumption, and a CP three times higher than SOM‐OLSR. Furthermore, when contrasted with the ‘no optimisation’ scenario, DRL‐OLSR achieves a remarkable 69.7% higher throughput and nearly 89% lower energy consumption. These findings highlight the effectiveness of the DRL‐OLSR approach in wireless sensor networks.

IRS Integrated UAV Communication to Enhance Physical Layer Security: A Deep Reinforcement Learning approach

The paper investigates the security of an intelligent reflecting surface (IRS) assisted unmanned aerial vehicle (UAV) network, where a base station (BS) transmits confidential information to the ground user (GU) via IRS-assisted UAV. The study explores using UAVs with IRS to enhance the security and reliability of wireless communication systems, particularly in the presence of eavesdroppers and friendly jammers. The beamforming at IRS-assisted UAV and UAV trajectory are jointly formulated as a non-convex optimization problem, which is solved by the deep reinforcement learning (DRL) algorithm to maximize the sum secrecy rate of GU with the aid of jammer. We proposed a dual-DDPG (D3PG) algorithm that utilizes the deep deterministic policy gradient (DDPG) structure to effectively address these dual non-convex problems of the UAV-trajectory optimization and the UAV-IRS beamforming optimization. The proposed algorithm's effectiveness and robustness are demonstrated through simulation results, with the IRS significantly enhancing the sum secrecy rate. Extensive simulations show that the proposed DRL-based D3PG scheme outperforms the traditional optimization schemes and ordinary DQN schemes.

A Multipolicy Deep Reinforcement Learning Approach for Multiobjective Joint Routing and Scheduling in Deterministic Networks

A Multipolicy Deep Reinforcement Learning Approach for Multiobjective Joint Routing and Scheduling in Deterministic Networks

Oneshot Deep Reinforcement Learning Approach to Network Slicing for Autonomous IoT Systems

Oneshot Deep Reinforcement Learning Approach to Network Slicing for Autonomous IoT Systems

Deep Reinforcement Learning for Optimizing Restricted Access Window in IEEE 802.11ah MAC Layer.

The IEEE 802.11ah standard is introduced to address the growing scale of internet of things (IoT) applications. To reduce contention and enhance energy efficiency in the system, the restricted access window (RAW) mechanism is introduced in the medium access control (MAC) layer to manage the significant number of stations accessing the network. However, to achieve optimized network performance, it is necessary to appropriately determine the RAW parameters, including the number of RAW groups, the number of slots in each RAW, and the duration of each slot. In this paper, we optimize the configuration of RAW parameters in the uplink IEEE 802.11ah-based IoT network. To improve network throughput, we analyze and establish a RAW parameters optimization problem. To effectively cope with the complex and dynamic network conditions, we propose a deep reinforcement learning (DRL) approach to determine the preferable RAW parameters to optimize network throughput. To enhance learning efficiency and stability, we employ the proximal policy optimization (PPO) algorithm. We construct network environments with periodic and random traffic in an NS-3 simulator to validate the performance of the proposed PPO-based RAW parameters optimization algorithm. The simulation results reveal that using the PPO-based DRL algorithm, optimized RAW parameters can be obtained under different network conditions, and network throughput can be improved significantly.

Discrete structural design synthesis: a hierarchical-inspired deep reinforcement learning approach considering topological and parametric actions

Abstract Structural design synthesis considering discrete elements can be formulated as a sequential decision process solved using deep reinforcement learning (DRL), shown in prior work to naturally accommodate discrete actions and efficiently provide adept solutions to a variety of problems. By modeling structural design synthesis as a Markov decision process (MDP), the states correspond to specific structural designs, the actions correspond to specific design alterations, and the rewards are related to the improvement in the altered design's performance with respect to the design objective and specified constraints. Here, the MDP action definition is extended by integrating parametric design grammars that further enable the design agent to not only alter a given structural design's topology, but also its element parameters. In considering topological and parametric actions, both the dimensionality of the state and action space and the diversity of the action types available to the agent in each state significantly increase, making the overall MDP learning task more challenging. Hence, this paper also addresses discrete design synthesis problems with large state and action spaces by significantly extending the framework. A hierarchical-inspired deep neural network architecture is developed to equip the agent to learn the type of action, topological or parametric, to apply, thus reducing the complexity of possible action choices in a given state. This extended framework is applied to the design synthesis of planar structures considering both discrete elements and cross-sectional areas, and it is observed to adeptly learn policies that synthesize high performing design solutions.

Collaborative Computation Offloading and Resource Management in Space–Air–Ground Integrated Networking: A Deep Reinforcement Learning Approach

With the increasing dissemination of the Internet of Things and 5G, mobile edge computing has become a novel scheme to assist terminal devices in executing computation tasks. To elevate the coverage and computation capability of edge computing, a collaborative computation offloading and resource management architecture was proposed in space–air–ground integrated networking (SAGIN). In this manuscript, we established a novel model considering the computation offloading cost constraints of the communication, computing and cache model in the SAGIN. To be specific, the joint optimization problem of collaborative computation offloading and resource management was modeled as a mixed integer nonlinear programming problem. To address this issue, this paper proposed a computation offloading and resource allocation strategy based on deep reinforcement learning (DRL). Differing from traditional methods, DRL does not need a well-established formulation or previous information, and it is capable of revising the strategy adaptively according to the environment. The simulation results demonstrate the proposed approach can achieve the optimal reward values in the case of different terminal device numbers. Furthermore, this manuscript provided the analysis with variant parameters of the proposed approach.

An adaptable and personalized framework for top-N course recommendations in online learning

In recent years, the proliferation of Massive Open Online Courses (MOOC) platforms on a global scale has been remarkable. Learners can now meet their learning demands with the help of MOOC. However, learners might not understand the course material well if they have access to a lot of information due to their inadequate expertise and cognitive ability. Personalized Recommender Systems (RSs), a cutting-edge technology, can assist in addressing this issue. It greatly increases resource acquisition through personalized availability for various people of all ages. Intelligent learning methods, such as machine learning and Reinforcement Learning (RL) can be used in RS challenges. However, machine learning needs supervised data and classical RL is not suitable for multi-task recommendations in online learning platforms. To address these challenges, the proposed framework integrates a Deep Reinforcement Learning (DRL) and multi-agent approach. This adaptive system personalizes the learning experience by considering key factors such as learner sentiments, learning style, preferences, competency, and adaptive difficulty levels. We formulate the interactive RS problem using a DRL-based Actor-Critic model named DRR, treating recommendations as a sequential decision-making process. The DRR enables the system to provide top-N course recommendations and personalized learning paths, enriching the student's experience. Extensive experiments on a MOOC dataset such as the 100 K Coursera course review validate the proposed DRR model, demonstrating its superiority over baseline models in major evaluation metrics for long-term recommendations. The outcomes of this research contribute to the field of e-learning technology, guiding the design and implementation of course RSs, to facilitate personalized and relevant recommendations for online learning students.

A Deep Recurrent Reinforcement Learning Approach for Enhanced MPPT in PV Systems

Under partial shading condition (PSC), for efficient usage, solar photovoltaic (PV) systems are required to operate at global maximum power point (GMPP). A maximum power point tracking (MPPT) controller is usually employed for this task. In this work, for MPPT under PSC, a recurrent deep reinforcement learning (DRL) approach has been developed and investigated. The optimized neural network, augmented by long short-term memory (LSTM), is trained using the proximal policy optimization (PPO) DRL algorithm. The MPPT performance of the proposed scheme was assessed using the GMPP attainment accuracy (%), on several realistic static and dynamic test cases. The presented investigations revealed the proposed controller's average MPPT accuracy of 97.79% for the considered random static PSCs. Similarly, average MPPT accuracies of 95.69% under changing irradiance, 95.70% under change in temperature and 95.77% under changes in both irradiance and temperature were recorded. These accuracies are found to be significantly superior to recent reported works. The resulting performance enhancement is attributed to the inclusion of LSTM, which allowed the agent to retain information about past states and actions, enabling it to make informed decisions. Therefore, based on the presented investigations, it is concluded that the proposed PPO-LSTM approach is an excellent MPPT alternative for PV systems.

Collaborative Computing in Non-Terrestrial Networks: A Multi-Time-Scale Deep Reinforcement Learning Approach

Collaborative Computing in Non-Terrestrial Networks: A Multi-Time-Scale Deep Reinforcement Learning Approach

Reinforcement learning-based heuristic planning for optimized energy management in power-split hybrid electric heavy duty vehicles

In this work, we systematically integrate relevant expertise, specifically on the optimal brake-specific fuel consumption (BSFC) curve, battery characteristics and terrain information into the development of an energy management plan for heavy-duty power-split hybrid electric vehicles. We utilize deep deterministic policy gradient (DDPG) algorithm as one of the most sophisticated reinforcement learning technique. Initially, we begin by explaining the vehicle configuration's system modeling. Subsequently, we then present an energy management approach based on deep Q-learning concepts. A novel algorithm, Deep Deterministic Policy Gradient, for energy management control, have been created to combat the "curse of dimensionality" in reinforcement learning. The new AMSGrad optimization technique is used by DDPG algorithm to update the weight of neural networks. We robustly train the proposed control system in realistic driving environment. The Knowledge Incorporation (KI) based DDPG based system is compared systematically to the conventional DDDPG methodology and the benchmark Dynamic Programming (DP) method, the latter of which usually uses the RMSProp Optimizer in its formulation. The results show that as compared to the typical DQL policy, deep reinforcement learning approaches, notably expert Knowledge incorporation KI-DDPG and terrain information with the AMSGrad optimizer, achieve faster training speeds and reduced fuel consumption while maintaining the terminal state of charge (SOC).

An Adaptive Route Guidance Model Considering the Effect of Traffic Signals Based on Deep Reinforcement Learning

Navigation or route guidance systems are designed to provide drivers with real-time travel information and the associated recommended routes for their trips. Classical route choice models typically rely on utility theory to represent drivers’ route choice behavior. Such choices, however, may not be optimal from both the individual and the system perspectives. This is simply due to the fact that drivers usually have imperfect knowledge about the time-varying traffic conditions. In this article, we explore and propose a new model-free deep reinforcement learning (DRL) approach to solving the adaptive route guidance problem based on microsimulation. The proposed approach consists of three interconnected algorithms, including a network edge labeling algorithm, a routing plan identification algorithm, and an adaptive route guidance algorithm. Simulation experiments on both a toy network and a real-world network of Suzhou, China, are performed to demonstrate the effectiveness of the proposed approach in terms of guiding a single vehicle as well as multiple vehicles through complex traffic environments. Comparative results confirm that the DRL approach outperforms the traditional shortest path method by further reducing the average travel time in the network.

Towards efficient RAN slicing: A deep hierarchical reinforcement learning approach

Radio access network (RAN) slicing can significantly improve network flexibility and resource utilization efficiency. Generally, deep reinforcement learning (DRL) is a prevailing approach to implementing practicable resource management in RAN slicing. However, this demand-aware resource allocation suffers from long training time and slow execution efficiency. This can lead to the inability to quickly achieve the desired spectral efficiency and service level agreement satisfaction rate. To tackle this issue, we propose a novel RAN slicing resource allocation framework based on a deep hierarchical RL framework for efficient resource scheduling. Structurally, our framework consists of a policy selection network and a policy evaluation network. In particular, a newly built action acceleration unit can achieve quick reward accumulation, thus speeding up the optimal policy search. Extensive simulations show that our proposal has higher system utility and faster convergence compared to the state-of-the-art DRL algorithms.

Data distribution inference attack in federated learning via reinforcement learning support

Federated Learning(FL) is currently a widely used collaborative learning framework, and the distinguished feature of FL is that the clients involved in training do not need to share raw data, but only transfer the model parameters to share knowledge, and finally get a global model with improved performance. However, recent studies have found that sharing model parameters may still lead to privacy leakage. From the shared model parameters, local training data can be reconstructed and thus lead to a threat to individual privacy and security. We observed that most of the current attacks are aimed at client-specific data reconstruction, while limited attention is paid to the information leakage of the global model. In our work, we propose a novel FL attack based on shared model parameters that can deduce the data distribution of the global model. Different from other FL attacks that aim to infer individual clients’ raw data, the data distribution inference attack proposed in this work shows that the attackers can have the capability to deduce the data distribution information behind the global model. We argue that such information is valuable since the training data behind a well-trained global model indicates the common knowledge of a specific task, such as social networks and e-commerce applications. To implement such an attack, our key idea is to adopt a deep reinforcement learning approach to guide the attack process, where the RL agent adjusts the pseudo-data distribution automatically until it is similar to the ground truth data distribution. By a carefully designed MDP process, our implementation ensures our attack can have stable performance and experimental results verify the effectiveness of our proposed inference attack.

Federated Multiagent Deep Reinforcement Learning Approach via Physics-Informed Reward for Multimicrogrid Energy Management.

The utilization of large-scale distributed renewable energy (RE) promotes the development of the multimicrogrid (MMG), which raises the need of developing an effective energy management method to minimize economic costs and keep self energy sufficiency. The multiagent deep reinforcement learning (MADRL) has been widely used for the energy management problem because of its real-time scheduling ability. However, its training requires massive energy operation data of microgrids (MGs), while gathering these data from different MGs would threaten their privacy and data security. Therefore, this article tackles this practical yet challenging issue by proposing a federated MADRL (F-MADRL) algorithm via the physics-informed reward. In this algorithm, the federated learning (FL) mechanism is introduced to train the F-MADRL algorithm, thus ensures the privacy and the security of data. In addition, a decentralized MMG model is built, and the energy of each participated MG is managed by an agent, which aims to minimize economic costs and keep self energy sufficiency according to the physics-informed reward. At first, MGs individually execute the self-training based on local energy operation data to train their local agent models. Then, these local models are periodically uploaded to a server and their parameters are aggregated to build a global agent, which will be broadcasted to MGs and replace their local agents. In this way, the experience of each MG agent can be shared and the energy operation data are not explicitly transmitted, thus protecting the privacy and ensuring data security. Finally, experiments are conducted on Oak Ridge National Laboratory distributed energy control communication laboratory MG (ORNL-MG) test system, and the comparisons are carried out to verify the effectiveness of introducing the FL mechanism and the outperformance of our proposed F-MADRL.

Optimization for Dynamic Laser Inter-Satellite Link Scheduling With Routing: A Multi-Agent Deep Reinforcement Learning Approach

Optimization for Dynamic Laser Inter-Satellite Link Scheduling With Routing: A Multi-Agent Deep Reinforcement Learning Approach

Dynamic resource matching in manufacturing using deep reinforcement learning

Matching plays an important role in the logical allocation of resources across a wide range of industries. The benefits of matching have been increasingly recognized in manufacturing industries. In particular, capacity sharing has received much attention recently. In this paper, we consider the problem of dynamically matching demand-capacity types of manufacturing resources. We formulate the multi-period, many-to-many manufacturing resource-matching problem as a sequential decision process. The formulated manufacturing resource-matching problem involves large state and action spaces, and it is not practical to accurately model the joint distribution of various types of demands. To address the curse of dimensionality and the difficulty of explicitly modeling the transition dynamics, we use a model-free deep reinforcement learning approach to find optimal matching policies. Moreover, to tackle the issue of infeasible actions and slow convergence due to initial biased estimates caused by the maximum operator in Q-learning, we introduce two penalties to the traditional Q-learning algorithm: a domain knowledge-based penalty based on a prior policy and an infeasibility penalty that conforms to the demand-supply constraints. We establish theoretical results on the convergence of our domain knowledge-informed Q-learning providing performance guarantee for small-size problems. For large-size problems, we further inject our modified approach into the deep deterministic policy gradient (DDPG) algorithm, which we refer to as domain knowledge-informed DDPG (DKDDPG). In our computational study, including small- and large-scale experiments, DKDDPG consistently outperformed traditional DDPG and other RL algorithms, yielding higher rewards and demonstrating greater efficiency in time and episodes.

Joint Offloading and Resource Allocation for Hybrid Cloud and Edge Computing in SAGINs: A Decision Assisted Hybrid Action Space Deep Reinforcement Learning Approach

Joint Offloading and Resource Allocation for Hybrid Cloud and Edge Computing in SAGINs: A Decision Assisted Hybrid Action Space Deep Reinforcement Learning Approach