Stochastic Shortest Path Problem Research Articles

Efficient Constraint Generation for Stochastic Shortest Path Problems

Current methods for solving Stochastic Shortest Path Problems (SSPs) find states’ costs-to-go by applying Bellman backups, where state-of-the-art methods employ heuristics to select states to back up and prune. A fundamental limitation of these algorithms is their need to compute the cost-to-go for every applicable action during each state backup, leading to unnecessary computation for actions identified as sub-optimal. We present new connections between planning and operations research and, using this framework, we address this issue of unnecessary computation by introducing an efficient version of constraint generation for SSPs. This technique allows algorithms to ignore sub-optimal actions and avoid computing their costs-to-go. We also apply our novel technique to iLAO* resulting in a new algorithm, CG-iLAO*. Our experiments show that CG-iLAO* ignores up to 57% of iLAO*’s actions and it solves problems up to 8x and 3x faster than LRTDP and iLAO*.

Minimizing File Transfer Time in Opportunistic Spectrum Access Model

We study the file transfer problem in opportunistic spectrum access (OSA) model, which has been widely studied in throughput-oriented applications for max-throughput strategies and in delay-related works that commonly assume identical channel rates and fixed file sizes. Our work explicitly considers minimizing the file transfer time for a given file in a set of heterogeneous-rate Bernoulli channels, showing that max-throughput policy doesn't minimize file transfer time in general. We formulate a mathematical framework for static extend to dynamic policies by mapping our file transfer problem to a stochastic shortest path problem. We analyze the performance of our proposed static and dynamic optimal policies over the max-throughput policy. We propose a mixed-integer programming formulation as an efficient alternative way to obtain the dynamic optimal policy and show a huge reduction in computation time. Then, we propose a heuristic policy that takes into account the performance-complexity tradeoff and consider the online implementation with unknown channel parameters. Furthermore, we present numerical simulations to support our analytical results and discuss the effect of switching delay on different policies. Finally, we extend the file transfer problem to Markovian channels and demonstrate the impact of the correlation of each channel.

Dynamic User-Scheduling and Power Allocation for SWIPT Aided Federated Learning: A Deep Learning Approach

Federated learning (FL) has been considered as a promising paradigm for enabling distributed machine learning (ML) in wireless networks. To address the limited energy capacity of wireless devices, we propose a simultaneous wireless information and power transfer (SWIPT) aided FL, in which one FL server (FLS) co-located at a cellular base station (BS) uses SWIPT to simultaneously broadcast the global model to wireless user-devices (UDs) and provide wireless power transfer to them. The UDs then use the harvested energy to train their local models and further transmit the local models to the FLS for aggregation. To improve the spectrum efficiency, we consider that the UDs form a non-orthogonal multiple access (NOMA) group for simultaneously sending their local models over the same spectrum channel. Taking the UDs' time-varying available energy and channel conditions into account, we propose a dynamic optimization of the UDs-scheduling, the BS's transmit-power allocation, and the UDs' power-splitting factors for SWIPT, with the objective of minimizing the long-term energy consumption while ensuring the FL convergence. The optimization problem, however, is challenging to solve since it is a finite-horizon dynamic programming problem but with an unknown stopping time, and moreover, the action space covers both discrete and continuous variables. To address these difficulties, we first execute a series of equivalent transformations to reduce the number of decision variables and then formulate the problem as a stochastic shortest path problem, based on which we propose an actor-critic deep reinforcement learning algorithm with the proximal policy optimization to efficiently learn the policy that dynamically adjusts the UDs-scheduling for FL as well as the BS's transmit-power for SWIPT. Numerical results validate the effectiveness and performance of our proposed algorithm. The results demonstrate that our proposed algorithm can effectively reduce the long-term energy consumption in comparison with two baseline algorithms.

Dynamic Project Expediting: A Stochastic Shortest-Path Approach

We deal with the problem of managing a project or a complex operational process by controlling the execution pace of the activities it comprises. We consider a setting in which these activities are clearly defined, are subject to precedence constraints, and progress randomly. We formulate a discrete-time, infinite-horizon Markov decision process in which the manager reviews progress in each period and decides which activities to expedite to balance expediting costs with delay costs. We derive structural properties for this dynamic project expediting problem. These enable us then to devise exact solution methods that we show to reduce computational burden significantly. We illustrate how our method generalizes and can be used to tackle a wide range of so-called stochastic shortest-path problems that are characterized by an intuitive property and can capture other applications, including medical decision-making and disease-modeling problems. Moreover, we also deal with the state identification issue for our problem, which is a challenging task in and of itself, owing to precedence constraints. We complement our analytical results with numerical experiments, demonstrating that both our solution and state identification methods significantly outperform extant methods for a supply chain example and for various randomly generated instances. This paper was accepted by Chung Piaw Teo, optimization. Funding: R. Mogre acknowledges support from the U.S.-UK Fulbright Commission and the Lloyd’s Tercentenary Research Foundation through the Fulbright-Lloyd’s Scholar Award, which allowed him to spend an extended period of time at Massachusetts Institute of Technology. Supplemental Material: The data files are available at https://doi.org/10.1287/mnsc.2023.4876 .

A Theory of Merge-and-Shrink for Stochastic Shortest Path Problems

The merge-and-shrink framework is a powerful tool to construct state space abstractions based on factored representations. One of its core applications in classical planning is the construction of admissible abstraction heuristics. In this paper, we develop a compositional theory of merge-and-shrink in the context of probabilistic planning, focusing on stochastic shortest path problems (SSPs). As the basis for this development, we contribute a novel factored state space model for SSPs. We show how general transformations, including abstractions, can be formulated on this model to derive admissible and/or perfect heuristics. To formalize the merge-and-shrink framework for SSPs, we transfer the fundamental merge-and-shrink transformations from the classical setting: shrinking, merging, and label reduction. We analyze the formal properties of these transformations in detail and show how the conditions under which shrinking and label reduction lead to perfect heuristics can be extended to the SSP setting.

An anytime algorithm for constrained stochastic shortest path problems with deterministic policies

Sequential decision-making problems arise in every arena of daily life and pose unique challenges for research in decision-theoretic planning. Although there has been a wide variety of research in this field, most of the studies have largely focused on single objective problem without constraints. In many real-world applications, however, it is often desirable to bound certain costs or resources under some predefined level. Constrained stochastic shortest path problem (C-SSP), one of the most well-known mathematical frameworks for stochastic decision-making problems with constraints, can formally model such problems, by incorporating constraints in the model formulation. However, it remains an open challenge to produce a deterministic optimal policy with desirable computation time due to its intrinsic complexity.In this paper, we propose a method that produces an optimal and deterministic policy for a C-SSP based on the Lagrangian duality theory and the heuristic forward search method. To address the intrinsic complexity of C-SSP, the proposed method is designed to have an anytime property. In other words, the proposed algorithm tries to find a feasible but decent solution quickly, then improves the solution incrementally until it converges to a true optimal solution. An extensive experimental evaluation on three problem domains shows that the proposed method outperforms the state-of-the-art methods in terms of the near-optimal solution with an optimality gap of less than 0.1%.

Optimal decisions in stochastic graphs with uncorrelated and correlated edge weights

Stochastic graph models and algorithms for the computation of different stochastic shortest path problems (SSPPs) continue to play an important role in various fields of computer science and operations research, with recent applications ranging from vehicle routing to social networks and decision problems in finance. In this paper, we first introduce stochastic graph models where weights to pass an edge in a graph are, possibly correlated, random variables. Depending on the application context and the interpretation of uncertain edge weights, random variables either model the time to pass an edge or describe a reward one gains by traversing an edge. For modeling the random variables discrete phase type distributions are used. Several algorithms for the computation of optimal policies which depend on the current weight upon arriving at a vertex while still exploiting weight dependencies among edges are developed. It will be shown that with correlated edge weights the stochastic shortest path problems become PSPACE-complete and the solution of instances with a bounded number of phases is NP-hard such that for an exact computation of the shortest path, algorithms from partially observable Markov decision processes (POMDPs) or mixed integer linear programming (MILP) have to be applied. Alternatively, an efficient heuristic algorithm can be used which often yields good or even optimal results with a small effort.

Cost Partitioning Heuristics for Stochastic Shortest Path Problems

In classical planning, cost partitioning is a powerful method which allows to combine multiple admissible heuristics while retaining an admissible bound. In this paper, we extend the theory of cost partitioning to probabilistic planning by generalizing from deterministic transition systems to stochastic shortest path problems (SSPs). We show that fundamental results related to cost partitioning still hold in our extended theory. We also investigate how to optimally partition costs for a large class of abstraction heuristics for SSPs. Lastly, we analyze occupation measure heuristics for SSPs as well as the theory of approximate linear programming for reward-oriented Markov decision processes. All of these fit our framework and can be seen as cost-partitioned heuristics.

Synthesis of Optimal Multiobjective Attack Strategies for Controlled Systems Modeled by Probabilistic Automata

In this article, we study the security of control systems in the context of the supervisory control layer of stochastic discrete-event systems. Control systems heavily rely on correct communication between the plant and the controller. In this article, we consider that such communication is partially compromised by a malicious attacker. The attacker has the ability to modify a subset of the sensor readings and mislead the supervisor, with the goal of inducing the system into an unsafe state. We consider this problem from the attacker’s viewpoint and investigate the synthesis of an attack strategy for systems modeled as probabilistic automata. Specifically, we investigate the synthesis of attack functions constrained by multiple objectives. We proceed in two steps. First, we quantify each attack strategy based on the likelihood of successfully reaching an unsafe state. Based on this quantification, we study the problem of synthesizing attack functions with the maximum likelihood of successfully reaching an unsafe state. Second, we consider the problem of synthesizing attack functions that have the maximum likelihood of successfully reaching an unsafe state while minimizing a cost function, i.e., the synthesis of attack functions is constrained by multiple objectives. Our solution methodology is based on mapping these problems to optimal control problems for Markov decision processes, specifically, a probabilistic reachability problem and a stochastic shortest path problem.

A framework for efficient dynamic routing under stochastically varying conditions

Despite measures to reduce congestion, occurrences of both recurrent and non-recurrent congestion cause large delays in road networks with important economic implications. Educated use of Intelligent Transportation Systems (ITS) can significantly reduce travel times. We focus on a dynamic stochastic shortest path problem: our objective is to minimize the expected travel time of a vehicle, assuming the vehicle may adapt the chosen route while driving. We introduce a new stochastic process that incorporates ITS information to model the uncertainties affecting congestion in road networks. A Markov-modulated background process tracks traffic events that affect the speed of travelers. The resulting continuous-time routing model allows for correlation between velocities on the arcs and incorporates both recurrent and non-recurrent congestion. Obtaining the optimal routing policy in the resulting semi-Markov decision process using dynamic programming is computationally intractable for realistic network sizes. To overcome this, we present the edsger⋆ algorithm, a Dijkstra-like shortest path algorithm that can be used dynamically with real-time response. We develop additional speed-up techniques that reduce the size of the network model. We quantify the performance of the algorithms by providing numerical examples that use road network detector data for The Netherlands.

A lower bound competitive ratio for the online stochastic shortest path problem

A lower bound competitive ratio for the online stochastic shortest path problem

Stochastic Safest and Shortest Path Problems

Optimal solutions to Stochastic Shortest Path Problems (SSPs) usually require that there exists at least one policy that reaches the goal with probability 1 from the initial state. This condition is very strong and prevents from solving many interesting problems, for instance where all possible policies reach some dead-end states with a positive probability. We introduce a more general and richer dual optimization criterion, which minimizes the average (undiscounted) cost of only paths leading to the goal among all policies that maximize the probability to reach the goal. We present policy update equations in the form of dynamic programming for this new dual criterion, which are different from the standard Bellman equations, but produce the same solution if there exists one policy leading to the goal with probability 1 from the initial state. We demonstrate that our equations converge in infinite horizon without any condition on the structure of the problem or on its policies, which actually extends the class of SSPs that can be solved. We experimentally show that our dual criterion provides well-founded solutions to SSPs that can not be solved by the standard criterion, and that using a discount factor with the latter certainly provides solution policies but which are not optimal considering our well-founded criterion.

Pattern Databases for Stochastic Shortest Path Problems

Stochastic shortest-path problems (SSP) are an important subclass of MDPs for which heuristic search algorithms exist since over a decade. Yet most known heuristic functions rely on determinization so do not actually take the transition probabilities into account. The only exceptions are Trevizan et al.'s heuristics hpom and hroc, which are geared at solving more complex (constrained) MDPs. Here we contribute pattern database (PDB) heuristics for SSPs, including an additivity criterion. These new heuristics turn out to be very competitive, even when using a simple systematic generation of pattern collections up to a fixed size. In our experiments, they beat determinization-based heuristics, and tend to yield better runtimes than hpom and hroc.

Solving in real-time the dynamic and stochastic shortest path problem for electric vehicles by a prognostic decision making strategy

The adoption of Electric Vehicles (EVs) has substantially increased during the last decade, creating the need for customized EV-oriented routing strategies capable of using the enormous amount of historical, and real-time, traffic data that is collected through Intelligent Transport Systems (ITSs). Existing EV routing algorithms, however, concentrate mostly on the usage historical data to compute offline optimal paths, whereas the use of real-time traffic information to compute en-route path updates is still an almost unexplored topic; mainly due to its inherent computational challenges. This research effort proposes a Prognostic Decision Making (PDM) strategy to solve in real-time the Electric Vehicle Dynamic Stochastic Shortest Path Problem (EV-DSSPP); aiming at the simultaneous utilization of historical and real-time traffic data. Factors such as recurring and non-recurring traffic congestion, elevation, velocity and EV’s parameters are incorporated into the decision-making process. The proposed strategy has two hierarchical functional layers. The lower layer consists of a fast-computing routing algorithm that, by construction, guarantees a real-time execution. The higher layer organizes the periodic en-route execution of the first layer to compute en-route path updates during a trip. This strategy can hence serve as an expert router that works jointly with an ITS to assist EV drivers on route. The proposal is validated through a simulation study based on real-world traffic data collected in Santiago, Chile. The results show that periodic en-route path updates can generate a reduction in both travel time and energy consumption, which evidences the benefits of incorporating real-time traffic information into the EV-routing problems.

On scenario construction for stochastic shortest path problems in real road networks

Stochastic shortest path (SSP) computations are often performed under very strict time constraints, so computational efficiency is critical. A major determinant for the CPU time is the number of scenarios used. We demonstrate that by carefully picking the right scenario generation method for finding scenarios, the quality of the computations can be improved substantially over random sampling for a given number of scenarios. We study extensive SSP instances from a freeway network and an urban road network, which involve 10,512 and 37,500 spatially and temporally correlated speed variables, respectively. On the basis of experimental results from a total of 42 origin–destination pairs and 6 typical objective functions for SSP problems, we find that (1) the scenario generation method generates unbiased scenarios and strongly outperforms random sampling in terms of stability (i.e., relative difference and variance) whichever origin–destination pair and objective function is used; (2) to achieve a certain accuracy, the number of scenarios required for scenario generation is much lower than that for random sampling, typically about 6–10 times lower for a stability level of 1% in the freeway network; and (3) different origin–destination pairs and different objective functions could require different numbers of scenarios to achieve a specified stability.

An Exact Algorithm to Make a Trade-Off between Cost and Probability in SSPs

In stochastic sequential decision problems, such as Stochastic Shortest Path Problems, the GUBS (Goal with UtilityBased Semantics) criterion considers a trade-off between probability-to-goal and cost-to-goal using a goal semantics based on Expected Utility Theory (EUT); in such a semantics, goal paths have priority over non-goal paths, but it implies neither the MAXPROB criterion nor the dual optimization criterion that finds the cheapest policy among the policies that maximize goal probability. Whereas evaluation criteria based on a sound theory such as EUT are desirable, optimal policies under GUBS are non-markovian. Non-markovian solutions are undesirable because there is not always a finite representation and even if it can be represented in a finite way, the representation may be too large to be stored. Here we define a special case of GUBS criterion that allows a finite representation to the optimal policy, the eGUBS criterion, where the cost utility function is exponencial. Considering this special case, we contribute with: (i) the proof that the eGUBSoptimal policy has a finite representation; (ii) the first exact algorithm to obtain finite optimal policies for the eGUBS criterion, and (iii) four strategies to find sub-optimal policies. We conduct experiments on one synthetic problem to evaluate each strategy. Although optimal solutions have a high memory cost, sub-optimal policies can save memory space with a small decrease in performance.

On the convergence of reinforcement learning with Monte Carlo Exploring Starts

A basic simulation-based reinforcement learning algorithm is the Monte Carlo Exploring Starts (MCES) method, also known as optimistic policy iteration, in which the value function is approximated by simulated returns and a greedy policy is selected at each iteration. The convergence of this algorithm in the general setting has been an open question. In this paper, we investigate the convergence of this algorithm for the case with undiscounted costs, also known as the stochastic shortest path problem. The results complement existing partial results on this topic and thereby help further settle the open problem.

Dynamic video delivery using deep reinforcement learning for device-to-device underlaid cache-enabled Internet-of-vehicle networks

This paper addresses an Internet-of-vehicle network that utilizes a device-to-device (D2D) underlaid cellular system, where distributed caching at each vehicle is available and the video streaming service is provided via D2D links. Given the spectrum reuse policy, three decisions having different timescales in such a D2D underlaid cache-enabled vehicular network were investigated: 1) The decision on the cache-enabled vehicles for providing contents, 2) power allocation for D2D users, and 3) power allocation for cellular vehicles. Since wireless link activation for video delivery could introduce delays, node association is determined in a larger time scale compared to power allocations. We jointly optimize these delivery decisions by maximizing the average video quality under the constraints on the playback delays of streaming users and the data rate guarantees for cellular vehicles. Depending on the channel and queue states of users, the decision on the cache-enabled vehicle for video delivery is adaptively made based on the frame-based Lyapunov optimization theory by comparing the expected costs of vehicles. For each cache-enabled vehicle, the expected cost is obtained from the stochastic shortest path problem that is solved by deep reinforcement learning without the knowledge of global channel state information. Specifically, the deep deterministic policy gradient (DDPG) algorithm is adopted for dealing with the very large state space, i.e., time-varying channel states. Simulation results verify that the proposed video delivery algorithm achieves all the given goals, i.e., average video quality, smooth playback, and reliable data rates for cellular vehicles.

A sampling method based on distributed learning automata for solving stochastic shortest path problem

This paper studies an iterative stochastic algorithm for solving the stochastic shortest path problem. This algorithm, which uses a distributed learning automata, tries to find the shortest path by taking a sufficient number of samples from the edges of the graph. In this algorithm, which edges to be sampled are determined dynamically as the algorithm proceeds. At each iteration of this algorithm, a distributed learning automata used to determine which edges to be sampled. This sampling method, which uses distributed learning automata, reduces the number of samplings from those edges, which may not be along the shortest path, and resulting in a reduction in the number of the edges to be sampled. In this paper, we propose a new method for analysis of the algorithm. The method proposed in this paper, unlike the previous ones, which were based on the Martingale theory, is based on the sampling. The proof given in this paper, unlike the previous ones, which were for a specific input graph, is not restricted to the input graph and is general. We also show that as the number of samples taken from the edges increases, the probability of finding the shortest path also increases. Experimental results obtained from some benchmark stochastic graphs and some random graphs also confirm the theoretical results.

Multiagent value iteration algorithms in dynamic programming and reinforcement learning

We consider infinite horizon dynamic programming problems, where the control at each stage consists of several distinct decisions, each one made by one of several agents. In an earlier work we introduced a policy iteration algorithm, where the policy improvement is done one-agent-at-a-time in a given order, with knowledge of the choices of the preceding agents in the order. As a result, the amount of computation for each policy improvement grows linearly with the number of agents, as opposed to exponentially for the standard all-agents-at-once method. For the case of a finite-state discounted problem, we showed convergence to an agent-by-agent optimal policy. In this paper, this result is extended to value iteration and optimistic versions of policy iteration, as well as to more general DP problems where the Bellman operator is a contraction mapping, such as stochastic shortest path problems with all policies being proper.