Sequential Decision Problems Research Articles

OSTTD: Offloading of Splittable Tasks With Topological Dependence in Multi-Tier Computing Networks

With the arrival of the Internet of Things (IoT) era, multi-tier computing has attracted significant attention. The multi-tier computing can organize many computing devices and provide sufficient computing resources to support various IoT applications. However, due to the complex architecture and the dynamic system status of the multi-tier computing network, task offloading for multi-tier computing is still challenging. This paper proposes a novel task offloading method named OSTTD, to deal with the Offloading of Splittable Tasks with Topological Dependence in multi-tier computing networks. OSTTD formulates the task offloading as a sequential decision-making problem and learns the task offloading policy by Deep Reinforcement Learning (DRL). Compared with existing task offloading approaches, OSTTD is the first method in which the topological dependence among sub-tasks of the splittable task is fully considered. OSTTD makes offloading decisions intelligently based on the dynamic system status and can be applied to various multi-tier network topology structures. To verify the effectiveness of OSTTD, we extend and build a latency-aware multi-tier computing simulation platform. Extensive simulations show that OSTTD can significantly reduce the task processing time, thus, improving the overall task processing efficiency in multi-tier computing networks.

Collision avoidance for autonomous ship using deep reinforcement learning and prior-knowledge-based approximate representation

Reinforcement learning (RL) has shown superior performance in solving sequential decision problems. In recent years, RL is gradually being used to solve unmanned driving collision avoidance decision-making problems in complex scenarios. However, ships encounter many scenarios, and the differences in scenarios will seriously hinder the application of RL in collision avoidance at sea. Moreover, the iterative speed of trial-and-error learning for RL in multi-ship encounter scenarios is slow. To solve this problem, this study develops a novel intelligent collision avoidance algorithm based on approximate representation reinforcement learning (AR-RL) to realize the collision avoidance of maritime autonomous surface ships (MASS) in a continuous state space environment involving interactive learning capability like a crew in navigation situation. The new algorithm uses an approximate representation model to deal with the optimization of collision avoidance strategies in a dynamic target encounter situation. The model is combined with prior knowledge and International Regulations for Preventing Collisions at Sea (COLREGs) for optimal performance. This is followed by a design of an online solution to a value function approximation model based on gradient descent. This approach can solve the problem of large-scale collision avoidance policy learning in static-dynamic obstacles mixed environment. Finally, algorithm tests were constructed though two scenarios (i.e., the coastal static obstacle environment and the static-dynamic obstacles mixed environment) using Tianjin Port as an example and compared with multiple groups of algorithms. The results show that the algorithm can improve the large-scale learning efficiency of continuous state space of dynamic obstacle environment by approximate representation. At the same time, the MASS can efficiently and safely avoid obstacles enroute to reaching its target destination. It therefore makes significant contributions to ensuring safety at sea in a mixed traffic involving both manned and MASS in near future.

Joint condition-based maintenance and spare provisioning policy for a K-out-of-N system with failures during inspection intervals

Recent advances in dedicated sensors and surveillance technology have given rise to condition-based maintenance. In addition to the degradation condition of the system, maintenance schedules should take into account the availability of spare parts. By incorporating spare part order information, planners can make more precise arrangements for both maintenance and spare parts replenishment, leading to a reduction in operating cost. This paper investigates the joint optimization of condition-based maintenance and spare provisioning for a K-out-of-N system considering failures during inspection intervals. The degradation states of the system are revealed at periodic inspection epochs where planners have the opportunity to replace components and order spare parts. The resulting sequential decision problem is formulated as a Markov decision process that integrates the comprehensive information of the components and spare parts orders into decision making. We propose an optimal joint policy by using the value iteration algorithm and compare it with two separated policies that consider part of the information. Sensitivity experiments are conducted to analyze the impact of the parameters on the effect of the proposed policy with the comprehensive information. The results show that the comprehensive information is more important to the ordering decisions than the replacing decisions.

An electrical vehicle-assisted demand response management system: A reinforcement learning method

With the continuous progress of urbanization, determining the charging and discharging strategy for randomly parked electric vehicles to help the peak load shifting without affecting users’ travel is a key problem. This paper design a reinforcement learning-based method for the electric vehicle-assisted demand response management system. Specifically, we formalize the charging and discharging sequential decision problem of the parking lot into the Markov process, in which the state space is composed of the state of parking spaces, electric vehicles, and the total load. The charging and discharging decision of each parking space acts as the action space. The reward comprises the penalty term that guarantees the user’s travel and the sliding average value of the load representing peak load shifting. After that, we use a Deep Q-Network (DQN)-based reinforcement learning architecture to solve this problem. Finally, we conduct a comprehensive evaluation with real-world power usage data. The results show that our proposed method will reduce the peak load by 10% without affecting the travel plan of all electric vehicles. Compared with random charging and discharging scenarios, we have better performance in terms of state-of-charge (SoC) achievement rate and peak load shifting effect.

Online portfolio management via deep reinforcement learning with high-frequency data

Recently, models that based on Transformer (Vaswani et al., 2017) have yielded superior results in many sequence modeling tasks. The ability of Transformer to capture long-range dependencies and interactions makes it possible to apply it in the field of portfolio management (PM). However, the built-in quadratic complexity of the Transformer prevents its direct application to the PM task. To solve this problem, in this paper, we propose a deep reinforcement learning-based PM framework called LSRE-CAAN, with two important components: a long sequence representations extractor and a cross-asset attention network. Direct Policy Gradient is used to solve the sequential decision problem in the PM process. We conduct numerical experiments in three aspects using four different cryptocurrency datasets, and the empirical results show that our framework is more effective than both traditional and state-of-the-art (SOTA) online portfolio strategies, achieving a 6x return on the best dataset. In terms of risk metrics, our framework has an average volatility risk of 0.46 and an average maximum drawdown risk of 0.27 across the four datasets, both of which are lower than the vast majority of SOTA strategies. In addition, while the vast majority of SOTA strategies maintain a poor turnover rate of approximately greater than 50% on average, our framework enjoys a relatively low turnover rate on all datasets, efficiency analysis illustrates that our framework no longer has the quadratic dependency limitation.

Variational Information Bottleneck Regularized Deep Reinforcement Learning for Efficient Robotic Skill Adaptation

Deep Reinforcement Learning (DRL) algorithms have been widely studied for sequential decision-making problems, and substantial progress has been achieved, especially in autonomous robotic skill learning. However, it is always difficult to deploy DRL methods in practical safety-critical robot systems, since the training and deployment environment gap always exists, and this issue would become increasingly crucial due to the ever-changing environment. Aiming at efficiently robotic skill transferring in a dynamic environment, we present a meta-reinforcement learning algorithm based on a variational information bottleneck. More specifically, during the meta-training stage, the variational information bottleneck first has been applied to infer the complete basic tasks for the whole task space, then the maximum entropy regularized reinforcement learning framework has been used to learn the basic skills consistent with that of basic tasks. Once the training stage is completed, all of the tasks in the task space can be obtained by a nonlinear combination of the basic tasks, thus, the according skills to accomplish the tasks can also be obtained by some way of a combination of the basic skills. Empirical results on several highly nonlinear, high-dimensional robotic locomotion tasks show that the proposed variational information bottleneck regularized deep reinforcement learning algorithm can improve sample efficiency by 200-5000 times on new tasks. Furthermore, the proposed algorithm achieves substantial asymptotic performance improvement. The results indicate that the proposed meta-reinforcement learning framework makes a significant step forward to deploy the DRL-based algorithm to practical robot systems.

Procedural- and Reinforcement-Learning-Based Automation Methods for Analog Integrated Circuit Sizing in the Electrical Design Space

Analog integrated circuit sizing is notoriously difficult to automate due to its complexity and scale; thus, it continues to heavily rely on human expert knowledge. This work presents a machine learning-based design automation methodology comprising pre-defined building blocks such as current mirrors or differential pairs and pre-computed look-up tables for electrical characteristics of primitive devices. Modeling the behavior of primitive devices around the operating point with neural networks combines the speed of equation-based methods with the accuracy of simulation-based approaches and, thereby, brings quality of life improvements for analog circuit designers using the gm/Id method. Extending this procedural automation method for human design experts, we present a fully autonomous sizing approach. Related work shows that the convergence properties of conventional optimization approaches improve significantly when acting in the electrical domain instead of the geometrical domain. We, therefore, formulate the circuit sizing task as a sequential decision-making problem in the alternative electrical design space. Our automation approach is based entirely on reinforcement learning, whereby abstract agents learn efficient design space navigation through interaction and without expert guidance. These agents’ learning behavior and performance are evaluated on circuits of varying complexity and different technologies, showing both the feasibility and portability of the work presented here.

A General Framework for Bandit Problems Beyond Cumulative Objectives

The stochastic multiarmed bandit (MAB) problem is a common model for sequential decision problems. In the standard setup, a decision maker has to choose at every instant between several competing arms; each of them provides a scalar random variable, referred to as a “reward.” Nearly all research on this topic considers the total cumulative reward as the criterion of interest. This work focuses on other natural objectives that cannot be cast as a sum over rewards but rather, more involved functions of the reward stream. Unlike the case of cumulative criteria, in the problems we study here, the oracle policy, which knows the problem parameters a priori and is used to “center” the regret, is not trivial. We provide a systematic approach to such problems and derive general conditions under which the oracle policy is sufficiently tractable to facilitate the design of optimism-based (upper confidence bound) learning policies. These conditions elucidate an interesting interplay between the arm reward distributions and the performance metric. Our main findings are illustrated for several commonly used objectives, such as conditional value-at-risk, mean-variance trade-offs, Sharpe ratio, and more. Funding: This work was partially funded by the Israel Science Foundation [Contract 2199/20] and by the European Community’s Seventh Framework Programme FP7/2007–2013 [Grant 306638 (Scaling Up Reinforcement Learning: Structure Learning, Skill Acquisition, and Reward Shaping)].

GUBS criterion: Arbitrary trade-offs between cost and probability-to-goal in stochastic planning based on Expected Utility Theory

Stochastic Shortest Path MDPs (SSP-MDPs) are used to model probabilistic sequential decision problems where the objective is to minimize the expected accumulated cost to goal. However, in the presence of dead-ends, the conventional criterion for SSP-MDPs, which minimizes the expected accumulated cost, can become ill-defined. Lexicographic criteria can solve this by preferring policies that reach the goal with the highest possible probability. Other criteria can instead make a trade-off between some cost measure and probability-to-goal. However, both of these approaches can lead to policies that might not represent the choice of a real decision-maker. In this work, we propose the GUBS criterion to address these problems. GUBS combines goal prioritization over histories with Expected Utility Theory and is the only criterion between all criteria analyzed that not only allows for a trade-off between a large accumulated cost and a small loss in probability-to-goal, but also guarantees arbitrary trade-offs that can be tuned from its parameters without previous knowledge of the problem being solved. We also propose eGUBS, which is a particular case of GUBS when the exponential utility function is used, and two algorithms for optimally solving these problems: eGUBS-VI, a VI-based algorithm; and eGUBS-AO*, a heuristic search algorithm. Results indicate that, when there is a good heuristic function available or when the state space is too large, eGUBS-AO* can perform better than eGUBS-VI by doing an efficient search. In other cases, eGUBS-VI's simpler approach might have better results.

Estimating Link Flows in Road Networks With Synthetic Trajectory Data Generation: Inverse Reinforcement Learning Approach

While traffic volume data from loop detectors have been the common data source for link flow estimation, the detectors only cover a subset of links. These days, other data sources such as vehicle trajectory data collected from vehicle tracking sensors are also incorporated. However, trajectory data are often sparse in that the observed trajectories only represent a small subset of the whole population, where the exact route sampling rate is unknown and may vary over space and time. In this paper, we develop a method that leverage these two limited data sources to enhance link flow estimation. This study proposes a novel generative modelling framework, where we formulate a vehicle’s link-to-link movements as a sequential decision-making problem using the Markov Decision Process framework. We propose an Inverse Reinforcement Learning-based method, based on which synthetic population vehicle trajectories can be generated to estimate link flows across the whole network. The proposed method ensures the generated population vehicle trajectories are consistent with the observed traffic volume and trajectory data. The proposed generative modelling framework is compared to two existing methods in a synthetic road network and validated in a real road network.

Reinforcement learning for automatic detection of effective strategies for self-regulated learning

Self-regulated learning (SRL) is an essential skill for achieving one's learning goals, particularly in Digital Learning Environments (DLEs) where system support is often limited compared to traditional classroom settings. However, research has found that learners often struggle to adapt their behaviour to the self-regulatory demands of DLEs. Furthermore, existing SRL analysis tools have limited utility for real-time or individualized prescriptive support of a learner's SRL strategy during a study session.In response to these challenges, we propose a novel approach using reinforcement learning as a framework to optimize the sequence of SRL processes for a learning task. This framework allows us to model and optimize the SRL strategy as a sequential decision-making problem, where each decision corresponds to an SRL process. The goal is to find an optimal sequence of decisions that maximizes performance in learning outcomes such as assessment score or learning gains.We compare the performance of our reinforcement learning framework with other sequential machine learning tools, such as Long Short-Term Memory (LSTM) neural networks and Genetic Algorithms (GA). The results of our study show that our reinforcement learning model outperforms GA and LSTM models in optimizing SRL strategy.The contributions of this work can facilitate the development of a tool which can detect sub-optimal SRL strategy in real-time and enable individualized SRL focused scaffolding.

Hybrid dynamic optimal tracking control of hydraulic cylinder speed in injection molding industry process

<p style='text-indent:20px;'>Injection speed control is one of the most essential components in the injection molding process. It is critical to implement fast and reliable injection speed optimization control for the effective manufacturing of injection molding products. In this paper, the speed tracking control problem of the hydraulic cylinder of an injection molding machine driven by a typical servo motor is studied. An efficient two-stage optimization framework that leverages the ideas of both control parameterization (CP) and particle swarm optimization (PSO) is proposed for generating the optimal signal input to achieve the required injection speed tracking over a certain time. To this end, the dynamic model of the hydraulic system in the injection molding equipment is firstly established and the optimal controller design problem is transformed into sequential optimal parameter decision problems, and the explicit expressions of the gradient information of the objective function as well as the constraint on the parameters of the decision variables are derived. Then, the PSO combined with the CP and gradient-based algorithm is utilized to to solve the parameters of the controller efficiently. The two-stage optimization framework proposed in this paper can improve the convergence speed of the CP method and the precision of the PSO and easy-to-implement in engineering and industrial deployment. Finally, the feasibility and effectiveness of the proposed design method are verified by experimental simulations.</p>

Novel Adaptive Transmission Scheme for Effective URLLC Support in 5G NR: A Model-Based Reinforcement Learning Solution

Future industrial Internet of Things (IIoT) applications demand trustworthy ultra-reliable and low-latency communications (URLLC) service. In this letter, we jointly design available reliability and latency mechanisms in 5G NR to maximize the probability of successful data delivery subject to a strict latency constraint. Particularly, we propose to optimally select numerology, mini-slot size, and modulation and coding scheme for each transmission/retransmission attempt, considering channel quality and remaining latency budget. To obtain the optimal policy for this sequential decision-making problem, we apply model-based reinforcement learning technique and formulate and solve a finite-step MDP problem. Through selected numerical examples, we show that the proposed joint design can achieve considerable performance gain over conventional scheme.

An anytime algorithm for constrained stochastic shortest path problems with deterministic policies

Sequential decision-making problems arise in every arena of daily life and pose unique challenges for research in decision-theoretic planning. Although there has been a wide variety of research in this field, most of the studies have largely focused on single objective problem without constraints. In many real-world applications, however, it is often desirable to bound certain costs or resources under some predefined level. Constrained stochastic shortest path problem (C-SSP), one of the most well-known mathematical frameworks for stochastic decision-making problems with constraints, can formally model such problems, by incorporating constraints in the model formulation. However, it remains an open challenge to produce a deterministic optimal policy with desirable computation time due to its intrinsic complexity.In this paper, we propose a method that produces an optimal and deterministic policy for a C-SSP based on the Lagrangian duality theory and the heuristic forward search method. To address the intrinsic complexity of C-SSP, the proposed method is designed to have an anytime property. In other words, the proposed algorithm tries to find a feasible but decent solution quickly, then improves the solution incrementally until it converges to a true optimal solution. An extensive experimental evaluation on three problem domains shows that the proposed method outperforms the state-of-the-art methods in terms of the near-optimal solution with an optimality gap of less than 0.1%.

STACoRe: Spatio-temporal and action-based contrastive representations for reinforcement learning in Atari

With the development of deep learning technology, deep reinforcement learning (DRL) has successfully built intelligent agents in sequential decision-making problems through interaction with image-based environments. However, learning from unlimited interaction is impractical and sample inefficient because training an agent requires many trial and error and numerous samples. One response to this problem is sample-efficient DRL, a research area that encourages learning effective state representations in limited interactions with image-based environments. Previous methods could effectively surpass human performance by training an RL agent using self-supervised learning and data augmentation to learn good state representations from a given interaction. However, most of the existing methods only consider similarity of image observations so that they are hard to capture semantic representations. To address these challenges, we propose spatio-temporal and action-based contrastive representation (STACoRe) learning for sample-efficient DRL. STACoRe performs two contrastive learning to learn proper state representations. One uses the agent’s actions as pseudo labels, and the other uses spatio-temporal information. In particular, when performing the action-based contrastive learning, we propose a method that automatically selects data augmentation techniques suitable for each environment for stable model training. We train the model by simultaneously optimizing an action-based contrastive loss function and spatio-temporal contrastive loss functions in an end-to-end manner. This leads to improving sample efficiency for DRL. We use 26 benchmark games in Atari 2600 whose environment interaction is limited to only 100k steps. The experimental results confirm that our method is more sample efficient than existing methods. The code is available at https://github.com/dudwojae/STACoRe.

Sequential dynamic resource allocation in multi-beam satellite systems: A learning-based optimization method

Multi-beam antenna and beam hopping technologies are an effective solution for scarce satellite frequency resources. One of the primary challenges accompanying with Multi-Beam Satellites (MBS) is an efficient Dynamic Resource Allocation (DRA) strategy. This paper presents a learning-based Hybrid-Action Deep Q-Network (HADQN) algorithm to address the sequential decision-making optimization problem in DRA. By using a parameterized hybrid action space, HADQN makes it possible to schedule the beam pattern and allocate transmitter power more flexibly. To pursue multiple long-term QoS requirements, HADQN adopts a multi-objective optimization method to decrease system transmission delay, loss ratio of data packets and power consumption load simultaneously. Experimental results demonstrate that the proposed HADQN algorithm is feasible and greatly reduces in-orbit energy consumption without compromising QoS performance.

Reinforcement Learning based Recommender Systems: A Survey

Recommender systems (RSs) have become an inseparable part of our everyday lives. They help us find our favorite items to purchase, our friends on social networks, and our favorite movies to watch. Traditionally, the recommendation problem was considered to be a classification or prediction problem, but it is now widely agreed that formulating it as a sequential decision problem can better reflect the user-system interaction. Therefore, it can be formulated as a Markov decision process (MDP) and be solved by reinforcement learning (RL) algorithms. Unlike traditional recommendation methods, including collaborative filtering and content-based filtering, RL is able to handle the sequential, dynamic user-system interaction and to take into account the long-term user engagement. Although the idea of using RL for recommendation is not new and has been around for about two decades, it was not very practical, mainly because of scalability problems of traditional RL algorithms. However, a new trend has emerged in the field since the introduction of deep reinforcement learning (DRL) , which made it possible to apply RL to the recommendation problem with large state and action spaces. In this paper, a survey on reinforcement learning based recommender systems (RLRSs) is presented. Our aim is to present an outlook on the field and to provide the reader with a fairly complete knowledge of key concepts of the field. We first recognize and illustrate that RLRSs can be generally classified into RL- and DRL-based methods. Then, we propose an RLRS framework with four components, i.e., state representation, policy optimization, reward formulation, and environment building, and survey RLRS algorithms accordingly. We highlight emerging topics and depict important trends using various graphs and tables. Finally, we discuss important aspects and challenges that can be addressed in the future.

Distributional reinforcement learning for inventory management in multi-echelon supply chains

Reinforcement Learning (RL) is an effective method to solve stochastic sequential decision-making problems. This is a problem description common to supply chain operations, however, most RL algorithms are tailored for game-based benchmarks. Here, we propose a deep RL method tailored for supply chain problems. The proposed algorithm deploys a derivative free approach to balance exploration and exploitation of the neural policy’s parameter space, providing means to avoid low quality local optima. Furthermore, the method allows consideration of risk-sensitive formulations to learn a policy that optimizes, for example, the conditional value-at-risk. The capabilities of our algorithm are tested on a multi-echelon supply chain problem, and several combinatorial optimization problems. The results empirically demonstrate the method’s improved sample efficiency compared to the benchmark algorithm proximal policy optimization, and superior performance to shrinking horizon mixed integer formulations. Additionally, its risk-sensitive policy can offer protection from low probability, high severity scenarios. Finally, we provide a sensitivity analysis for technical intuition.

Dynamic parameters in Sequential Decision Making

Sequential Decision Making (SDM) problems optimize over the sequence of actions (or, decisions) taken to minimize the underlying cumulative cost. These sequence of actions are referred to as the policy of the SDM. Often these problems comprise of additional (fixed and manipulable) parameters; and the objective is to determine the optimal policy as well as the manipulable parameters that minimizes the SDM cost. In this paper we address the class of SDM problems that are characterized by dynamic parameters; where the dynamics is pre-specified for a subset of parameters and manipulable for others. The objective is to determine the manipulable parameter dynamics as well as the time-varying policy such that the associated SDM cost gets minimized at each time instant. To this end, we develop a control-theoretic framework to design the manipulable parameter dynamics such that it tracks the optimal values of the parameters, and simultaneously determines the time-varying optimal policy. Our methodology builds upon a Maximum Entropy Principle (MEP) based framework that addresses SDMs. More precisely, the above framework results into a smooth approximation of the SDM cost which we utilize as a control Lyapunov function. We show that under the resulting control law the parameters asymptotically track the local optimal, the proposed control law is Lipschitz continuous and bounded, and the policy of the SDM is optimal for a given set of parameter values. The simulations demonstrate the efficacy of our proposed methodology.

Multi-Robot Learning Dynamic Obstacle Avoidance in Formation With Information-Directed Exploration

This paper presents an algorithm that generates distributed collision-free velocities for multi-robot while maintain formation as much as possible. The adaptive formation problem is cast as a sequential decision-making problem, which is solved using reinforcement learning that trains several distributed policies to avoid dynamic obstacles on the top of consensus velocities. We construct the policy with Bayesian Linear Regression based on a neural network (called BNL) to compute the state-action value uncertainty efficiently for sequential decision making. The information-directed sampling is applied in our BNL policy to achieve efficient exploration. By further combining the distributional reinforcement learning, we can estimate the intrinsic uncertainty of the state-action value globally and more accurately. For continuous control tasks, efficient exploration can be achieved by optimizing a policy with the sampled action value function from a BNL model. Through our experiments in some contextual Bandit and sequential decision-making tasks, we show that exploration with the BNL model has improved efficiency in both computation and training samples. By augmenting the consensus velocities with our BNL policy, experiments on Multi-Robot navigation demonstrate that adaptive formation is achieved.