Reinforcement Learning Approach Research Articles

Aircraft routing using dynamic programming and reinforcement learning: A customer-centric approach

Over the past few decades, the airline industry has evolved into a sophisticated business, with conventional objectives such as operational efficiency, effective fleet assignment, flight punctuality, and equitable crew scheduling being the primary focus. Nevertheless, given the growing accessibility to customer feedback (either through public forums or private data), it is crucial to make business decisions that cater to the customers’ requirements in addition to conventional goals. Within aircraft operations, there is a series of decisions to be made, such as flight scheduling, fleet assignment, aircraft routing, and crew scheduling. After designating a fleet type to each flight leg, airlines ascertain the sequence of flight legs, or strings, to be operated by individual aircrafts, while complying with obligatory maintenance checks. This constitutes the aircraft routing problem.We introduce a novel approach to aircraft routing problem, by infusing traditional methodologies with insightful customer feedback to create a more responsive and customer-centric model. Unlike conventional techniques of constructing routes, in our work, we construct maintenance feasible strings. Traditional aircraft routing methodologies typically concentrate on minimizing propagated delay. However, by integrating customer feedback, more effective prioritization to minimize delays for flight legs with higher customer dissatisfaction can be achieved. We propose an optimization model that strives to minimize propagated delay while prioritizing flights based on customer feedback, thus resulting in customer-aware routes. To solve the proposed optimization model, we design an iterative alternating optimization scheme where feasible strings are first constructed from the pool of available flight legs using a string generation procedure, followed by solving an integer linear optimization problem over only the feasible set of strings obtained in the first stage. To construct feasible strings of flights at each iteration, we use two different methods. The first method is based on a dynamic programming approach which is useful for mean delay information. To handle practical cases where delays are uncertain, we propose a reinforcement learning (RL) approach to construct the strings. We provide experiments of our proposed methods on synthetic and real data sets and demonstrate the effectiveness of our RL model in comparison to dynamic programming based model. Our experiments affirm that routing decisions informed by customer feedback prioritize minimizing delays for specific flight legs, resulting in routes that are inherently customer-friendly.

Fast Pareto set approximation for multi-objective flexible job shop scheduling via parallel preference-conditioned graph reinforcement learning

The Multi-Objective Flexible Job Shop Scheduling Problem (MOFJSP) is a complex challenge in manufacturing, requiring balancing multiple, often conflicting objectives. Traditional methods, such as Multi-Objective Evolutionary Algorithms (MOEA), can be time-consuming and unsuitable for real-time applications. This paper introduces a novel Graph Reinforcement Learning (GRL) approach, named Preference-Conditioned GRL, which efficiently approximates the Pareto set for MOFJSP in a parallelized manner. By decomposing the MOFJSP into distinct sub-problems based on preferences and leveraging a parallel multi-objective training algorithm, our method efficiently produces high-quality Pareto sets, significantly outperforming MOEA methods in both solution quality and speed, especially for large-scale problems. Extensive experiments demonstrate the superiority of our approach, with remarkable results on large instances, showcasing its potential for real-time scheduling in dynamic manufacturing environments. Notably, for large instances (50 × 20), our approach outperforms MOEA baselines with remarkably shorter computation time (less than 1% of that of MOEA baselines). The robust generalization performance across various instances also highlights the practical value of our method for decision-makers seeking optimized production resource utilization.

Agricultural land use modeling and climate change adaptation: A reinforcement learning approach

AbstractThis paper provides a novel approach to integrate farmers' behavior in spatially explicit agricultural land use modeling to investigate climate change adaptation strategies. More specifically, we develop and apply a computationally efficient machine learning approach based on reinforcement learning to simulate the adoption of agroforestry practices. Using data from an economic experiment with crop farmers in Southeast Germany, our results show that a change in climate, market, and policy conditions shifts the spatial distribution of the uptake of agroforestry systems. Our modeling approach can be used to advance currently used models for ex ante policy analysis by upscaling existing knowledge about farmers behavioral characteristics and combine it with spatially explicit environmental and farm structural data. The approach presents a potential solution for researchers who aim to upscale information, potentially enriching and complementing existing land use modeling approaches.

Research on a Personalized Decision Control Algorithm for Autonomous Vehicles Based on the Reinforcement Learning from Human Feedback Strategy

To address the shortcomings of previous autonomous decision models, which often overlook the personalized features of users, this paper proposes a personalized decision control algorithm for autonomous vehicles based on RLHF (reinforcement learning from human feedback). The algorithm combines two reinforcement learning approaches, DDPG (Deep Deterministic Policy Gradient) and PPO (proximal policy optimization), and divides the control scheme into three phases including pre-training, human evaluation, and parameter optimization. During the pre-training phase, an agent is trained using the DDPG algorithm. In the human evaluation phase, different trajectories generated by the DDPG-trained agent are scored by individuals with different styles, and the respective reward models are trained based on the trajectories. In the parameter optimization phase, the network parameters are updated using the PPO algorithm and the reward values given by the reward model to achieve personalized autonomous vehicle control. To validate the control algorithm designed in this paper, a simulation scenario was built using CARLA_0.9.13 software. The results demonstrate that the proposed algorithm can provide personalized decision control solutions for different styles of people, satisfying human needs while ensuring safety.

Synergizing Transfer Learning and Multi-Agent Systems for Thermal Parametrization in Induction Traction Motors

Maintaining optimal temperatures in the critical parts of an induction traction motor is crucial for railway propulsion systems. A reduced-order lumped-parameter thermal network (LPTN) model enables computably inexpensive, accurate temperature estimation; however, it requires empirically based parameter estimation exercises. The calibration process is typically performed in labs in a controlled experimental setting, which is associated with a lot of supervised human efforts. However, the exploration of machine learning (ML) techniques in varied domains has enabled the model parameterization in the drive system outside the laboratory settings. This paper presents an innovative use of a multi-agent reinforcement learning (MARL) approach for the parametrization of an LPTN model. First, a set of reinforcement learning agents are trained to estimate the optimized thermal parameters using the simulated data in several driving cycles (DCs). The selection of a reinforcement learning agent and the level of neurons in the RL model is made based on variability of the driving cycle data. Furthermore, transfer learning is performed on a new driving cycle data collected on the measurement setup. Statistical analysis and clustering techniques are proposed for the selection of an RL agent that has been pre-trained on the historical data. It is established that by synergizing within reinforcement learning techniques, it is possible to refine and adjust the RL learning models to effectively capture the complexities of thermal dynamics. The proposed MARL framework shows its capability to accurately reflect the motor’s thermal behavior under various driving conditions. The transfer learning usage in the proposed approach could yield significant improvement in the accuracy of temperature prediction in the new driving cycles data. This approach is proposed with the aim of developing more adaptive and efficient thermal management strategies for railway propulsion systems.

Data-driven constrained reinforcement learning algorithm for path tracking control of hovercraft

This paper presents a novel Constrained Backstepping Reinforcement Learning (CBRL) approach designed for path tracking of hovercraft. The approach addresses the challenges posed by complex modeling, multiple constraints, and large sideslip angles during high-speed maneuvers of underactuated hovercraft. Initially, a unique state constraint function is formulated, incorporating constraint boundaries related to navigation speed and path curvature. Additionally, transformations are applied to the yaw angular velocity and virtual yaw control law, ensuring that the yaw angular velocity remains within safety limits. Subsequently, a data-driven backstepping reinforcement learning optimal approach, coupled with a line-of-sight guidance law, is employed to guide the hovercraft along the intended path by regulating its yaw angle. Simultaneously, the backstepping reinforcement learning optimal control scheme is used to regulate the surge speed of the hovercraft above the resistance peak speed, thereby preventing excessive sideslip angles during path tracking. Neural network observers are integrated into the design process of both the yaw and surge controllers to monitor system dynamics in the presence of interference. Through simulation, the effectiveness and feasibility of the proposed CBRL algorithm are demonstrated.

Corruption-Robust Exploration in Episodic Reinforcement Learning

We initiate the study of episodic reinforcement learning (RL) under adversarial corruptions in both the rewards and the transition probabilities of the underlying system, extending recent results for the special case of multiarmed bandits. We provide a framework that modifies the aggressive exploration enjoyed by existing reinforcement learning approaches based on optimism in the face of uncertainty by complementing them with principles from action elimination. Importantly, our framework circumvents the major challenges posed by naively applying action elimination in the RL setting, as formalized by a lower bound we demonstrate. Our framework yields efficient algorithms that (a) attain near-optimal regret in the absence of corruptions and (b) adapt to unknown levels of corruption, enjoying regret guarantees that degrade gracefully in the total corruption encountered. To showcase the generality of our approach, we derive results for both tabular settings (where states and actions are finite) and linear Markov decision process settings (where the dynamics and rewards admit a linear underlying representation). Notably, our work provides the first sublinear regret guarantee that accommodates any deviation from purely independent and identically distributed transitions in the bandit-feedback model for episodic reinforcement learning. Supplemental Material: The online appendix is available at https://doi.org/10.1287/moor.2021.0202 .

A hybrid deep reinforcement learning approach for a proactive transshipment of fresh food in the online–offline channel system

To reduce the waste of fresh foods, one of the e-commerce companies in South Korea utilizes lateral transshipment in the network of online platforms and offline shops, which is called the online–offline channel system (OOCS). Even though the OOCS has achieved success in real practice, there is room for further study on this system with regard to deriving a transshipment policy. For this reason, this study aims to develop a solution approach that could derive a promising policy and analyze the impacts of transshipment in the OOCS. The main contributions are summarized as follows. First, we propose a model to deal with the proactive transshipment of perishable products in the OOCS. In particular, this is the first study that introduces the concept of the heterogeneous shelf life considering different properties of online and offline channels. Second, we develop the hybrid deep reinforcement learning (DRL) approach by combining the soft actor–critic algorithm with two novel acceleration methods. The developed method could obtain a promising policy without assumptions about demand distribution and mitigate computational burdens by reducing action spaces. On a set of experiments carried out on real-world demand data, the transshipment policy derived from the hybrid DRL approach could obtain the best profit compared to existing algorithms. Third, we examine the impacts of transshipment by differing types of demand and varying the unit transshipment cost parameter. We find that transshipment substantially reduces the outdating cost by allowing the offline channel to make good use of the old products that will be discarded in the online channel, which is new to the literature.

A deep reinforcement learning approach for dynamic task scheduling of flight tests

A deep reinforcement learning approach for dynamic task scheduling of flight tests

Energy and throughput efficient mobile wireless sensor networks: A deep reinforcement learning approach

AbstractThe efficient development of Mobile Wireless Sensor Networks (MWSNs) relies heavily on optimizing two key parameters: Throughput and Energy Consumption. The proposed work investigates network connectivity issues with MWSN and proposes two routing algorithms, namely Self‐Organising Maps based‐Optimised Link State Routing (SOM‐OLSR) and Deep Reinforcement Learning based‐Optimised Link State Routing (DRL‐OLSR) for MWSNs. The primary objective of the proposed algorithms is to achieve energy‐efficient routing while maximizing throughput. The proposed algorithms are evaluated through simulations by considering various performance metrics, including connection probability (CP), end‐to‐end delay, overhead, network throughput, and energy consumption. The simulation analysis is discussed under three scenarios. The first scenario undertakes ‘no optimisation’, the second considers SOM‐OLSR, and the third undertakes DRL‐OLSR. A comparison between DRL‐OLSR and SOM‐OLSR reveals that the former surpasses the latter in terms of low latency and prolonged network lifetime. Specifically, DRL‐OLSR demonstrates a 47% increase in throughput, a 67% reduction in energy consumption, and a CP three times higher than SOM‐OLSR. Furthermore, when contrasted with the ‘no optimisation’ scenario, DRL‐OLSR achieves a remarkable 69.7% higher throughput and nearly 89% lower energy consumption. These findings highlight the effectiveness of the DRL‐OLSR approach in wireless sensor networks.

Dynamic programming with meta-reinforcement learning: a novel approach for multi-objective optimization

AbstractMulti-objective optimization (MOO) endeavors to identify optimal solutions from a finite array of possibilities. In recent years, deep reinforcement learning (RL) has exhibited promise through its well-crafted heuristics in tackling NP-hard combinatorial optimization (CO) problems. Nonetheless, current methodologies grapple with two key challenges: (1) They primarily concentrate on single-objective optimization quandaries, rendering them less adaptable to the more prevalent MOO scenarios encountered in real-world applications. (2) These approaches furnish an approximate solution by imbibing heuristics, lacking a systematic means to enhance or substantiate optimality. Given these challenges, this study introduces an overarching hybrid strategy, dynamic programming with meta-reinforcement learning (DPML), to resolve MOO predicaments. The approach melds meta-learning into an RL framework, addressing multiple subproblems inherent to MOO. Furthermore, the precision of solutions is elevated by endowing exact dynamic programming with the prowess of meta-graph neural networks. Empirical results substantiate the supremacy of our methodology over previous RL and heuristics approaches, bridging the chasm between theoretical underpinnings and real-world applicability within this domain.

IRS Integrated UAV Communication to Enhance Physical Layer Security: A Deep Reinforcement Learning approach

The paper investigates the security of an intelligent reflecting surface (IRS) assisted unmanned aerial vehicle (UAV) network, where a base station (BS) transmits confidential information to the ground user (GU) via IRS-assisted UAV. The study explores using UAVs with IRS to enhance the security and reliability of wireless communication systems, particularly in the presence of eavesdroppers and friendly jammers. The beamforming at IRS-assisted UAV and UAV trajectory are jointly formulated as a non-convex optimization problem, which is solved by the deep reinforcement learning (DRL) algorithm to maximize the sum secrecy rate of GU with the aid of jammer. We proposed a dual-DDPG (D3PG) algorithm that utilizes the deep deterministic policy gradient (DDPG) structure to effectively address these dual non-convex problems of the UAV-trajectory optimization and the UAV-IRS beamforming optimization. The proposed algorithm's effectiveness and robustness are demonstrated through simulation results, with the IRS significantly enhancing the sum secrecy rate. Extensive simulations show that the proposed DRL-based D3PG scheme outperforms the traditional optimization schemes and ordinary DQN schemes.

A Multipolicy Deep Reinforcement Learning Approach for Multiobjective Joint Routing and Scheduling in Deterministic Networks

A Multipolicy Deep Reinforcement Learning Approach for Multiobjective Joint Routing and Scheduling in Deterministic Networks

Oneshot Deep Reinforcement Learning Approach to Network Slicing for Autonomous IoT Systems

Oneshot Deep Reinforcement Learning Approach to Network Slicing for Autonomous IoT Systems

Peak shaving and self-consumption maximization in home energy management systems: A combined integer programming and reinforcement learning approach

This paper proposes a novel framework for Home Energy Management System based on the combination of integer programming and Reinforcement Learning (RL) for achieving efficient home-based Demand Response (DR). In particular, RL is exploited to manage the charge and discharge of Battery Energy Storage System (BESS), and Mixed Integer Linear Programming is exploited for load scheduling. The idea is to focus the RL specifically on BESS management, as its behavior is stochastic and is mainly affected by Photovoltaic (PV) production and user behavior changes. The scheduling decisions of household appliances, Electric Vehicles (EVs), and charging/discharging batteries can be subsequently obtained through the newly developed framework, of which the objective is dual, i.e., to minimize the electricity bill as well as the DR-induced dissatisfaction. Simulations are performed on a residential house level with multiple home appliances, an EV, PV panels, and electric storage. The test results demonstrate the effectiveness of the proposed home energy management framework under the application of different demand-side flexibility strategies.

Deep Reinforcement Learning for Optimizing Restricted Access Window in IEEE 802.11ah MAC Layer.

The IEEE 802.11ah standard is introduced to address the growing scale of internet of things (IoT) applications. To reduce contention and enhance energy efficiency in the system, the restricted access window (RAW) mechanism is introduced in the medium access control (MAC) layer to manage the significant number of stations accessing the network. However, to achieve optimized network performance, it is necessary to appropriately determine the RAW parameters, including the number of RAW groups, the number of slots in each RAW, and the duration of each slot. In this paper, we optimize the configuration of RAW parameters in the uplink IEEE 802.11ah-based IoT network. To improve network throughput, we analyze and establish a RAW parameters optimization problem. To effectively cope with the complex and dynamic network conditions, we propose a deep reinforcement learning (DRL) approach to determine the preferable RAW parameters to optimize network throughput. To enhance learning efficiency and stability, we employ the proximal policy optimization (PPO) algorithm. We construct network environments with periodic and random traffic in an NS-3 simulator to validate the performance of the proposed PPO-based RAW parameters optimization algorithm. The simulation results reveal that using the PPO-based DRL algorithm, optimized RAW parameters can be obtained under different network conditions, and network throughput can be improved significantly.

Discrete structural design synthesis: a hierarchical-inspired deep reinforcement learning approach considering topological and parametric actions

Abstract Structural design synthesis considering discrete elements can be formulated as a sequential decision process solved using deep reinforcement learning (DRL), shown in prior work to naturally accommodate discrete actions and efficiently provide adept solutions to a variety of problems. By modeling structural design synthesis as a Markov decision process (MDP), the states correspond to specific structural designs, the actions correspond to specific design alterations, and the rewards are related to the improvement in the altered design's performance with respect to the design objective and specified constraints. Here, the MDP action definition is extended by integrating parametric design grammars that further enable the design agent to not only alter a given structural design's topology, but also its element parameters. In considering topological and parametric actions, both the dimensionality of the state and action space and the diversity of the action types available to the agent in each state significantly increase, making the overall MDP learning task more challenging. Hence, this paper also addresses discrete design synthesis problems with large state and action spaces by significantly extending the framework. A hierarchical-inspired deep neural network architecture is developed to equip the agent to learn the type of action, topological or parametric, to apply, thus reducing the complexity of possible action choices in a given state. This extended framework is applied to the design synthesis of planar structures considering both discrete elements and cross-sectional areas, and it is observed to adeptly learn policies that synthesize high performing design solutions.

Collaborative Computation Offloading and Resource Management in Space–Air–Ground Integrated Networking: A Deep Reinforcement Learning Approach

With the increasing dissemination of the Internet of Things and 5G, mobile edge computing has become a novel scheme to assist terminal devices in executing computation tasks. To elevate the coverage and computation capability of edge computing, a collaborative computation offloading and resource management architecture was proposed in space–air–ground integrated networking (SAGIN). In this manuscript, we established a novel model considering the computation offloading cost constraints of the communication, computing and cache model in the SAGIN. To be specific, the joint optimization problem of collaborative computation offloading and resource management was modeled as a mixed integer nonlinear programming problem. To address this issue, this paper proposed a computation offloading and resource allocation strategy based on deep reinforcement learning (DRL). Differing from traditional methods, DRL does not need a well-established formulation or previous information, and it is capable of revising the strategy adaptively according to the environment. The simulation results demonstrate the proposed approach can achieve the optimal reward values in the case of different terminal device numbers. Furthermore, this manuscript provided the analysis with variant parameters of the proposed approach.

An adaptable and personalized framework for top-N course recommendations in online learning

In recent years, the proliferation of Massive Open Online Courses (MOOC) platforms on a global scale has been remarkable. Learners can now meet their learning demands with the help of MOOC. However, learners might not understand the course material well if they have access to a lot of information due to their inadequate expertise and cognitive ability. Personalized Recommender Systems (RSs), a cutting-edge technology, can assist in addressing this issue. It greatly increases resource acquisition through personalized availability for various people of all ages. Intelligent learning methods, such as machine learning and Reinforcement Learning (RL) can be used in RS challenges. However, machine learning needs supervised data and classical RL is not suitable for multi-task recommendations in online learning platforms. To address these challenges, the proposed framework integrates a Deep Reinforcement Learning (DRL) and multi-agent approach. This adaptive system personalizes the learning experience by considering key factors such as learner sentiments, learning style, preferences, competency, and adaptive difficulty levels. We formulate the interactive RS problem using a DRL-based Actor-Critic model named DRR, treating recommendations as a sequential decision-making process. The DRR enables the system to provide top-N course recommendations and personalized learning paths, enriching the student's experience. Extensive experiments on a MOOC dataset such as the 100 K Coursera course review validate the proposed DRR model, demonstrating its superiority over baseline models in major evaluation metrics for long-term recommendations. The outcomes of this research contribute to the field of e-learning technology, guiding the design and implementation of course RSs, to facilitate personalized and relevant recommendations for online learning students.

Reinforcement Learning Approaches for Exploring Discrepancy Variations: Implications for Supply Chain Optimization

Combinatorics discrepancy theory studies the ways in which a given state deviates from the ideal. A set system's disparity in its classical form has been shown to be bound in certain situations. Though they may not hold for other types of discrepancy, no mathematician has been able to produce an example that defies these limits as of yet. Due to the large number of set systems (kmn for m sets and n elements, each choosing one of k possible values), it is not viable to use traditional techniques, such as human reasoning or computer brute forcing, to identify the few set systems with a significant discrepancy. However, in a 2021 preprint, Adam Zsolt Wagner shows how the deep cross-entropy method—a popular Neural Network (NNs)-based Reinforcement-Learning (RL) technique—could effectively find examples that disprove open conjectures in two other combinatorial subfields: pattern avoidance and graph theory. Therefore, in light of these encouraging findings, we wondered if Wagner's method could help us find examples that defy the upper bound of discrepancy variations. To start answering this question, we examined whether Wagner's approach might be extended to find set systems that exhibit significant inconsistencies, either as classical or variant (prefix and fractional in our instance) systems. Our findings could help approximation algorithms, which employ discrepancy theory to tackle certain problems, as well as discrepancy theory itself.