Stochastic Shortest Path Problem Research Articles

Finding the Shortest Path in Stochastic Vehicle Routing: A Cardinality Minimization Approach

This paper aims at solving the stochastic shortest path problem in vehicle routing, the objective of which is to determine an optimal path that maximizes the probability of arriving at the destination before a given deadline. To solve this problem, we propose a data-driven approach, which directly explores the big data generated in traffic. Specifically, we first reformulate the original shortest path problem as a cardinality minimization problem directly based on samples of travel time on each road link, which can be obtained from the GPS trajectory of vehicles. Then, we apply an $\ell_{1}$ - norm minimization technique and its variants to solve the cardinality problem. Finally, we transform this problem into a mixed-integer linear programming problem, which can be solved using standard solvers. The proposed approach has three advantages over traditional methods. First, it can handle various or even unknown travel time probability distributions, while traditional stochastic routing methods can only work on specified probability distributions. Second, it does not rely on the assumption that travel time on different road segments is independent of each other, which is usually the case in traditional stochastic routing methods. Third, unlike other existing methods which require that deadlines must be larger than certain values, the proposed approach supports more flexible deadlines. We further analyze the influence of important parameters to the performances, i.e., accuracy and time complexity. Finally, we implement the proposed approach and evaluate its performance based on a real road network of Munich city. With real traffic data, the results show that it outperforms traditional methods.

New reformulations of distributionally robust shortest path problem

This paper considers a stochastic version of the shortest path problem, namely the Distributionally Robust Stochastic Shortest Path Problem (DRSSPP) on directed graphs. In this model, each arc has a deterministic cost and a random delay. The mean vector and the second-moment matrix of the uncertain data are assumed to be known, but the exact information of the distribution is unknown. A penalty occurs when the given delay constraint is not satisfied. The objective is to minimize the sum of the path cost and the expected path delay penalty. As this problem is NP-hard, we propose new reformulations and approximations using a sequence of semidefinite programming problems which provide tight lower bounds. Finally, numerical tests are conducted to illustrate the tightness of the bounds and the value of the proposed distributionally robust approach.

Modeling wildfire propagation with the stochastic shortest path: A fast simulation approach

Wildfires have significant environmental and economic effects. Since containment of wildfires involves deciding under tight time constraints, there is an increasing need for accurate yet computationally efficient wildfire prediction models. We consider the problem of finding the fire traversal time across a landscape considering wind speed as an unpredictable phenomenon. The landscape is represented as a graph network and fire propagation time is modeled as the Stochastic Shortest Path problem. Monte-Carlo simulation is utilized to determine the fire travel-time distribution. A network size reduction methodology is introduced to quicken the simulation time by eliminating the unimportant parts of the network. This methodology is implemented in Java to simulate the wildfire propagation on a study area located in Massachusetts. This method shows the capability of substantially reducing the simulation time without affecting prediction accuracy, enabling the algorithm to serve as a fast and reliable tool for fire prediction.

Heuristic Search in Dual Space for Constrained Stochastic Shortest Path Problems

We consider the problem of generating optimal stochastic policies for Constrained Stochastic Shortest Path problems, which are a natural model for planning under uncertainty for resource-bounded agents with multiple competing objectives. While unconstrained SSPs enjoy a multitude of efficient heuristic search solution methods with the ability to focus on promising areas reachable from the initial state, the state of the art for constrained SSPs revolves around linear and dynamic programming algorithms which explore the entire state space. In this paper, we present i-dual, which, to the best of our knowledge, is the first heuristic search algorithm for constrained SSPs. To concisely represent constraints and efficiently decide their violation, i-dual operates in the space of dual variables describing the policy occupation measures. It does so while retaining the ability to use standard value function heuristics computed by well-known methods. Our experiments on a suite of PPDDL problems augmented with constraints show that these features enable i-dual to achieve up to two orders of magnitude improvement in run-time and memory over linear programming algorithms.

General Error Bounds in Heuristic Search Algorithms for Stochastic Shortest Path Problems

We consider recently-derived error bounds that can be used to bound the quality of solutions found by heuristic search algorithms for stochastic shortest path problems. In their original form, the bounds can only be used for problems with positive action costs. We show how to generalize the bounds so that they can be used in solving any stochastic shortest path problem, regardless of cost structure. In addition, we introduce a simple new heuristic search algorithm that performs as well or better than previous algorithms for this class of problems, while being easier to implement and analyze.

A note on detecting unbounded instances of the online shortest path problem

The online shortest path problem is a type of stochastic shortest path problem in which certain arc costs are revealed en route, and the path is updated accordingly to minimize expected cost. This note addresses the open problem of determining whether a problem instance admits a finite optimal solution in the presence of negative arc costs. We formulate the problem as a Markov decision process and show ways to detect such instances in the course of solving the problem using standard algorithms such as value and policy iteration. © 2016 Wiley Periodicals, Inc. NETWORKS, Vol. 67(4), 270–276 2016

Urban Freight Truck Routing under Stochastic Congestion and Emission Considerations

Freight trucks are known to be a major source of air pollutants as well as greenhouse gas emissions in U.S. metropolitan areas, and they have significant effects on air quality and global climate change. Emissions from freight trucks during their deliveries should be considered by the trucking service sector when they make routing decisions. This study proposes a model that incorporates total delivery time, various emissions including CO2, VOC, NOX, and PM from freight truck activities, and a penalty for late or early arrival into the total cost objective of a stochastic shortest path problem. We focus on urban transportation networks in which random congestion states on each link follows an independent probability distribution. Our model finds the best truck routing on a given network so as to minimize the expected total cost. This problem is formulated into a mathematical model, and two solution algorithms including a dynamic programming approach and a deterministic shortest path heuristic are proposed. Numerical examples show that the proposed approach performs very well even for the large-size U.S. urban networks.

Efficient Bounds in Heuristic Search Algorithms for Stochastic Shortest Path Problems

Fully observable decision-theoretic planning problems are commonly modeled as stochastic shortest path (SSP) problems. For this class of planning problems, heuristic search algorithms (including LAO*, RTDP, and related algorithms), as well as the value iteration algorithm on which they are based, lack an efficient test for convergence to an ε-optimal policy (except in the special case of discounting). We introduce a simple and efficient test for convergence that applies to SSP problems with positive action costs. The test can detect whether a policy is proper, that is, whether it achieves the goal state with probability 1. If proper, it gives error bounds that can be used to detect convergence to an ε-optimal solution. The convergence test incurs no extra overhead besides computing the Bellman residual, and the performance guarantee it provides substantially improves the utility of this class of planning algorithms.

PH-graphs for analyzing shortest path problems with correlated traveling times

This paper presents a new approach to model weighted graphs with correlated weights at the edges. Such models are important to describe many real world problems like routing in computer networks or finding shortest paths in traffic models under realistic assumptions. Edge weights are modeled by phase type distributions (PHDs), a versatile class of distributions based on continuous time Markov chains (CTMCs). Correlations between edge weights are introduced by adding dependencies between the PHDs of adjacent edges using transfer matrices. The new model class, denoted as PH graphs (PHGs), allows one to formulate many shortest path problems as the computation of an optimal policy in a continuous time Markov decision process (CTMDP). The basic model class is defined, methods to parameterize the required PHDs and transfer matrices based on measured data are introduced and the formulation of basic shortest path problems as solutions of CTMDPs with the corresponding solution algorithms are also provided. Numerical examples for some typical stochastic shortest path problems demonstrate the usability of the new approach.

A Stochastic Shortest Path Framework for Quantifying the Value and Lifetime of Battery Energy Storage Under Dynamic Pricing

This paper aims at quantifying the value of a lifetime-constrained battery energy storage system (BESS) operated by a consumer who faces fluctuating electricity prices. We define the lifetime of the BESS as the serving duration within which the BESSs capacity stays above a certain threshold of its initial capacity and define the value of the BESS as the total peak-shaving value within its entire lifetime. Under the assumption that the price dynamics are Markovian, we show that maximizing the average value of the BESS can be formulated as a stochastic shortest path (SSP) problem, and the average lifetime corresponds to the average number of steps before being absorbed in the SSP problem. We propose an efficient parallel value iteration algorithm to solve the proposed SSP problem with guarantees of achieving optimality and a fast convergence. We also derive a closed form expression for the average lifetime based on the principle of an embedded absorbing Markov chain. We validate our model and algorithm on a practical BESS via real price data sets from two different markets. Comparison of the computational efficiency between the standard Gauss-Seidel value iteration and our parallel algorithm is also illustrated through extensive simulation.

Depth-based short-sighted stochastic shortest path problems

Stochastic Shortest Path Problems (SSPs) are a common representation for probabilistic planning problems. Two approaches can be used to solve SSPs: (i) consider all probabilistically reachable states and (ii) plan only for a subset of these reachable states. Closed policies, the solutions obtained in the former approach, require significant computational effort, and they do not require replanning, i.e., the planner is never re-invoked. The second approach, employed by replanners, computes open policies, i.e., policies for a subset of the probabilistically reachable states. Therefore, when a state is reached in which the open policy is not defined, the replanner is reinvoked to compute a new open policy. In this article, we introduce a special case of SSPs, the depth-based short-sighted SSPs, in which every state has a nonzero probability of being reached using at most t actions. We also introduce the novel algorithm Short-Sighted Probabilistic Planner (SSiPP), which solves SSPs through depth-based short-sighted SSPs and guarantees that at least t actions can be executed without replanning. Therefore, SSiPP can compute both open and closed policies: as t increases, the returned policy approaches the behavior of a closed policy, and for t large enough, the returned policy is closed. Moreover, we present two extensions to SSiPP: Labeled-SSiPP and SSiPP-FF. The former extension incorporates a labeling mechanism to avoid revisiting states that have already converged. The latter extension combines SSiPP and determinizations to improve the performance of SSiPP in problems without dead ends. We also performed an extensive empirical evaluation of SSiPP and its extensions in several problems against state-of-the-art planners. The results show that (i) Labeled-SSiPP outperforms SSiPP and the considered planners in the task of finding the optimal solution when the problems have a low percentage of relevant states; and (ii) SSiPP-FF outperforms SSiPP in the task of quickly finding suboptimal solutions to problems without dead ends while performing similarly in problems with dead ends.

Applying Dijkstra's algorithm for general shortest path problem with normal probability distribution arc length

We present a general treatment of shortest path problems by dynamic program in networks having normal probability distributions as arc lengths. We consider a network that may contains cycles. Two operators of sum and comparison need to be adapted for the proposed Dijkstra’s algorithm. Because of inefficiency of the methods such as maximum likelihood estimation and moment generating function due to long computational efforts and inaccurate solution results, convolution approach is used to sum two normal probability distributions being employed in the Dijkstra’s algorithm. Generally, stochastic shortest path problems are treated using expected values of the arc probabilities, but in the proposed method using distributed observed past data as arc lengths, an integrated value is obtained as the shortest path length. The objective of this paper is to extend the general shortest path problem in dynamic and stochastic networks where link travel times are defined as normal probability distributions. We solve a numerical example to show efficiency of our proposed approach and after finding the shortest path in the network we obtain the shortest path’s density function, too.

Finding shortest path in a combined exponential-gamma probability distribution arc length

We propose a dynamic programme to find the shortest path in a network having exponential and gamma probability distributions as arc lengths. Two operators of sum and comparison need to be adapted for the proposed dynamic programme. Convolution approach is used to sum any two probability distributions being employed in the dynamic programme. Generally, stochastic shortest path problems are treated using expected values of the arc probabilities, but in the proposed method using distributed observed past data as arc lengths, an integrated value is obtained as the shortest path length. The objective of this paper is to extend the shortest path problem in dynamic and stochastic networks where link travel times are defined as gamma or exponential probability distributions.

Improved Self-Adaptive Genetic Algorithm and its Application on the Dynamic Stochastic Shortest Path Problem

In reality transportation network is dynamic and stochastic. Studies on the dynamic stochastic shortest path problem are of great research and application value. This paper introduces basic genetic algorithm into it. Then an improved self-adaptive genetic algorithm is proposed by improving population initialization, selection strategy, crossover strategy and mutation strategy. In addition, the improved genetic algorithm adjusts the crossover and mutation factors adaptively. The results of simulation experiment show that the improved genetic algorithm proposed by this paper has much higher capacity of global optimization than Dijkstra and A* algorithm in the dynamic stochastic shortest path problem.

Dynamic Markov Decision Policies for Delay Constrained Wireless Scheduling

We consider a one-hop wireless system with a small number of delay constrained users and a larger number of users without delay constraints. We develop a scheduling algorithm that reacts to time varying channels and maximizes throughput utility (to within a desired proximity), stabilizes all queues, and satisfies the delay constraints. The problem is solved by reducing the constrained optimization to a set of weighted stochastic shortest path problems, which act as natural generalizations of max-weight policies to Markov decision networks. We also present approximation results for the corresponding shortest path problems, and discuss the additional complexity and delay incurred as compared to systems without delay constraints. The solution technique is general and applies to other constrained stochastic decision problems.

An outer approximation algorithm for the robust shortest path problem

This paper describes a new algorithm for the stochastic shortest path problem where path costs are a weighted sum of expected cost and cost standard deviation. We allow correlation between link costs, subject to a regularity condition excluding unbounded solutions. The chief complication in this variant is that path costs are not an additive sum of link costs. In this paper, we reformulate this problem as a conic quadratic program, and develop an outer-approximation algorithm based on this formulation. Numerical experiments show that the outer-approximation algorithm significantly outperforms standard integer programming algorithms implemented in solvers.

A Mean-Risk Model for the Traffic Assignment Problem with Stochastic Travel Times

Heavy and uncertain traffic conditions exacerbate the commuting experience of millions of people across the globe. When planning important trips, commuters typically add an extra buffer to the expected trip duration to ensure on-time arrival. Motivated by this, we propose a new traffic assignment model that takes into account the stochastic nature of travel times. Our model extends the traditional model of Wardrop competition when uncertainty is present in the network. The focus is on strategic risk-averse users who capture the trade-off between travel times and their variability in a mean-standard deviation objective, defined as the mean travel time plus a risk-aversion factor times the standard deviation of travel time along a path. We consider both infinitesimal users, leading to a nonatomic game, and atomic users, leading to a discrete finite game. We establish conditions that characterize an equilibrium traffic assignment and find when it exists. The main challenge is posed by the users' risk aversion, since the mean-standard deviation objective is nonconvex and nonseparable, meaning that a path cannot be split as a sum of edge costs. As a result, even an individual user's subproblem—a stochastic shortest path problem—is a nonconvex optimization problem for which no polynomial time algorithms are known. In turn, the mathematical structure of the traffic assignment model with stochastic travel times is fundamentally different from the deterministic counterpart. In particular, an equilibrium characterization requires exponentially many variables, one for each path in the network, since an edge flow has multiple possible path-flow decompositions that are not equivalent. Because of this, characterizing the equilibrium and the socially optimal assignment, which minimizes the total user cost, is more challenging than in the traditional deterministic setting. Nevertheless, we prove that both can be encoded by a representation with just polynomially many paths. Finally, under the assumption that the standard deviations of travel times are independent from edge loads, we show that the worst-case ratio between the social cost of an equilibrium and that of an optimal solution is not higher than the analogous ratio in the deterministic setting. In other words, uncertainty does not further degrade the system performance in addition to strategic user behavior alone.

Distributionally robust stochastic shortest path problem

Abstract This paper considers a stochastic version of the shortest path problem, the Distributionally Robust Stochastic Shortest Path Problem(DRSSPP) on directed graphs. In this model, the arc costs are deterministic, while each arc has a random delay. The mean vector and the second-moment matrix of the uncertain data are assumed known, but the exact information of the distribution is unknown. A penalty occurs when the given delay constraint is not satisfied. The objective is to minimize the sum of the path cost and the expected path delay penalty. As it is NP-hard, we approximate the DRSSPP with a semidefinite programming (SDP for short) problem, which is solvable in polynomial time and provides tight lower bounds.

A Mean-Risk Model for the Traffic Assignment Problem with Stochastic Travel Times

Heavy and uncertain traffic conditions exacerbate the commuting experience of millions of people across the globe. When planning important trips, commuters typically add an extra buffer to the expected trip duration to ensure on-time arrival. Motivated by this, we propose a new traffic assignment model that takes into account the stochastic nature of travel times. Our model extends the traditional model of Wardrop competition when uncertainty is present in the network. The focus is on strategic risk-averse users who capture the tradeoff between travel times and their variability in a mean-standard deviation (mean-stdev) objective, defined as the mean travel time plus a risk-aversion factor times the standard deviation of travel time along a path. We consider both infinitesimal users, leading to a non-atomic game, and atomic users, leading to a discrete finite game.We establish conditions that characterize an equilibrium traffic assignment and find when it exists. The main challenge is posed by the users’ risk aversion, since the mean-stdev objective is non-convex and non-separable, meaning that a path cannot be split as a sum of edge costs. As a result, even an individual user’s sub-problem — a stochastic shortest path problem — is a non-convex optimization problem for which no polynomial time algorithms are known. In turn, the mathematical structure of the traffic assignment model with stochastic travel times is fundamentally different from the deterministic counterpart. In particular, an equilibrium characterization requires exponentially many variables, one for each path in the network, since an edge-flow has multiple possible path-flow decompositions that are not equivalent. Because of this, characterizing the equilibrium and the socially-optimal assignment, which minimizes the total user cost, is more challenging than in the traditional deterministic setting. Nevertheless, we prove that both can be encoded by a representation with just polynomially-many paths. Finally, for the case of uncertainty parameters that are independent from edge loads, we show that although an equilibrium assignment results in a total user cost that is higher than that of the socially-optimal one, it is not higher than the analogous ratio in the deterministic setting. In other words, uncertainty does not further degrade the system performance in addition to strategic user behavior alone.

On Boundedness of Q-Learning Iterates for Stochastic Shortest Path Problems

We consider a totally asynchronous stochastic approximation algorithm, Q-learning, for solving finite space stochastic shortest path (SSP) problems, which are undiscounted, total cost Markov decision processes with an absorbing and cost-free state. For the most commonly used SSP models, existing convergence proofs assume that the sequence of Q-learning iterates is bounded with probability one, or some other condition that guarantees boundedness. We prove that the sequence of iterates is naturally bounded with probability one, thus furnishing the boundedness condition in the convergence proof by Tsitsiklis [Tsitsiklis JN (1994) Asynchronous stochastic approximation and Q-learning. Machine Learn. 16:185–202] and establishing completely the convergence of Q-learning for these SSP models.