Stochastic Decision Problems Research Articles

Autonomous Behavior Selection For Self-driving Cars Using Probabilistic Logic Factored Markov Decision Processes

ABSTRACT We propose probabilistic logic factored Markov decision processes (PL-fMDPs) as a behavior selection scheme for self-driving cars. Probabilistic logic combines logic programming with probability theory to achieve clear, rule-based knowledge descriptions of multivariate probability distributions, and a flexible mixture of deductive and probabilistic inferences. Factored Markov decision processes (fMDPs) are widely used to generate reward-optimal action policies for stochastic sequential decision problems. For evaluation, we developed a simulated self-driving car with reliable modules for behavior selection, perception, and control. The behavior selection module is composed of a two-level structure of four action policies obtained from PL-fMDPs. Three main tests were conducted focused on the selection of the appropriate actions in specific driving scenarios, and the overtaking of static obstacle vehicles and dynamic obstacle vehicles. We performed 520 repetitions of these tests. The self-driving car completed its task without collisions in 99.2% of the repetitions. Results show the suitability of the overall self-driving strategy and PL-fMDPs to construct safe action policies for self-driving cars.

Multi-period, multi-timescale stochastic optimization model for simultaneous capacity investment and energy management decisions for hybrid Micro-Grids with green hydrogen production under uncertainty

Given the steep rises in renewable energy's proportion in the global energy mix expected over several decades, a systematic way to plan the long-term deployment is needed. The main challenges are how to handle the significant uncertainties in technologies and market dynamics over a large time horizon. The problem is further complicated by the fast-timescale volatility of renewable energy sources, potentially causing grid instability and unfulfilled demands. As a remedy, energy storage and power-to-hydrogen systems are considered in conjunction with energy management system but doing so raises the complexity of the planning problem further. In this work, the long-term capacity planning for a hybrid microgrid (HM) system is formulated as a multi-period stochastic decision problem that considers uncertainties occurring at multiple timescales. Long-term capacity decisions are inherently linked with energy dispatch and storage decisions occurring at fast-timescale and it is best to solve for them simultaneously. However, the computational demand for solving it becomes quickly intractable with problem size. To this end, we propose to develop a Markov decision process (MDP) formulation of the problem and use simulation-based reinforcement learning for multi-period capacity investments of the planning horizon. The MDP includes the policies used for dispatch and storage operation, which are represented by linear programming as a part of the simulation model. The effectiveness of our proposed method is demonstrated with a case study, where decisions over multiple decades are considered along with various uncertainties of multi-timescales. Economic and environmental assessments are performed, providing valuable guidelines for government's energy policy.

Decision Theory and Stochastic Growth

This paper examines connections between stochastic growth and decision problems. We use tools from the theory of large deviations to show that wishful thinking decision problems are equivalent to utility maximization problems, both of which are equivalent to growth maximization under idiosyncratic risk. Rational inattention problems are equivalent to growth-optimal portfolio problems, both of which are equivalent to growth maximization under aggregate risk. Stochastic growth generates extreme inequality, with nearly all wealth eventually held by those who happen to have faced empirical distributions that match the solution to the wishful thinking or rational inattention problem. (JEL D31, D81, D82, D83, G51, O41)

Multi-period design of hybrid microgrid under multi-timescale uncertainties

This study focuses on the deployment planning of hybrid microgrid that uses energy storage and power-to-hydrogen (P2H) systems for combatting the variability and intermittency problem of renewable energy sources (RES). To consider the long-term uncertainties, such as those related to demand growth and installation costs, the deployment problem is formulated as a multi-period stochastic decision problem. Given that deployment decisions are intimately linked with how the system is operated against the short-term variabilities of the energy generated from the RES, the overall decision problem is multi-scale in nature, and computation for solving the problem can quickly become exorbitant. To address this complexity, this study proposes a combination of reinforcement learning and mathematical programming in a complementary, hierarchical way. The proposed method is demonstrated through a case study, where deployment decisions over multiple decades are considered along with hourly operational decisions within each period under various uncertainty types. The deployment plan obtained using the suggested method shows an 11% improvement in the economic performance compared to the two-stage stochastic programming (2SSP) approach.

Deep reinforcement learning-based decision support system for transportation infrastructure management under hurricane events

The high wind, large surge/wave and heavy rainfall during a hurricane may significantly impact the performance of transportation infrastructure and associated traffic safety. Accordingly, stakeholders need to make a sequence of decisions to close (or restrict) the traffic of vulnerable components in the transportation network (e.g., aerodynamics-sensitive long-span bridges, hydrodynamics-sensitive coastal bridges and inundation-sensitive road segments) for the balance of safety and mobility. Due to large uncertainty involved in hurricane weather and traffic conditions, it is essentially a stochastic sequential decision problem and can be effectively formulated as a Markov decision process. This study proposes a deep reinforcement learning (RL)-based decision support system for stakeholders to optimally manage these critical components for the purpose of minimizing the network-level losses induced by hurricanes. Specifically, the RL policy (i.e., mapping from high-dimensional continuous traffic/weather information to management decisions) is represented by a deep neural network (DNN) and the optimal decision (corresponding to a set of DNN weights) is obtained using the deep Q learning algorithm. A case study based on a hypothetical traffic network is utilized to demonstrate the good performance of the developed deep RL-based decision support system in the application of transportation infrastructure management under hurricane conditions.

Deep Reinforcement Learning Aided Platoon Control Relying on V2X Information

The impact of Vehicle-to-Everything (V2X) communications on platoon control performance is investigated. Platoon control is essentially a sequential stochastic decision problem (SSDP), which can be solved by Deep Reinforcement Learning (DRL) to deal with both the control constraints and uncertainty in the platoon leading vehicle’s behavior. In this context, the value of V2X communications for DRL-based platoon controllers is studied with an emphasis on the tradeoff between the gain of including exogenous information in the system state for reducing uncertainty and the performance erosion due to the curse-of-dimensionality. Our objective is to find the specific set of information that should be shared among the vehicles for the construction of the most appropriate state space. SSDP models are conceived for platoon control under different information topologies (IFT) by taking into account ‘just sufficient’ information. Furthermore, theorems are established for comparing the performance of their optimal policies. In order to determine whether a piece of information should or should not be transmitted for improving the DRL-based control policy, we quantify its value by deriving the conditional KL divergence of the transition models. More meritorious information is given higher priority in transmission, since including it in the state space has a higher probability in offsetting the negative effect of having higher state dimensions. Finally, simulation results are provided to illustrate the theoretical analysis.

Multistage stochastic decision problems: Approximation by recursive structures and ambiguity modeling

Stochastic multistage decision problems appear in many - if not all - application areas of Operations Research. While to define such problems is easy, to solve them is quite difficult, since they are of infinite dimension. Numerical solution can only be found by solving an approximate, easier problem. In this paper, we show good approximations can be found, where we emphasize the recursive structure of the involved algorithms and data structures.In a second part, the problem of coping with the model error of approximations is discussed. We present algorithms for finding distributionally robust solutions for the model error problem. We also review some application cases of such situations from the literature.

Geometry of information structures, strategic measures and associated stochastic control topologies

In many areas of applied mathematics, decentralization of information is a ubiquitous attribute affecting how to approach a stochastic optimization, decision and estimation, or control problem. In this review article, we present a general formulation of information structures under a probability theoretic and geometric formulation. We define information structures, place various topologies on them, and study closedness, compactness and convexity properties on the strategic measures induced by information structures and decentralized control/decision policies under varying degrees of relaxations with regard to access to private or common randomness. Ultimately, we present existence and tight approximation results for optimal decision/control policies. We discuss various lower bounding techniques, through relaxations and convex programs ranging from classically realizable and classically non-realizable (such as quantum theoretic and non-signaling) relaxations. For each of these, we establish closedness and convexity properties and also a hierarchy of correlation structures. As a further theme, we review and introduce various topologies on decision/control strategies defined independent of information structures, but for which information structures determine whether the topologies entail utility in arriving at existence, compactness, convexification or approximation results. These approaches, which we term as the strategic measures approach and the control topology approach, lead to complementary results on existence, approximations and upper and lower bounds in optimal decentralized stochastic decision, estimation, and control problems.

Two-stage stochastic standard quadratic optimization

Two-stage stochastic decision problems are characterized by the property that in stage one part of the decision has to be made before relevant data are observed and the rest of the decision can be made in stage two after data are available. It is generally assumed that while the data missing in stage one are not known, their probability distribution is known. In this paper, we consider a special form of such problems where the objective function is quadratic in the first and second stage decisions and the goal is to minimize the expected quadratic cost function. We assume that the decisions must lie in a standard simplex, but do not require convexity of the uncertain objective. It is well known that such quadratic optimization programs are NP-complete and thus belong to the most complex optimization problems. A viable method to attack such problems is to find bounds for the original complicated problem by solving easier approximate problems. We describe such approximations and show that by exploiting their properties, good but not necessarily optimal solutions can be found. We also present possible applications, such as selecting invited speakers for conferences with the aim to generate a community with high coherence. Finally, systematic numerical results are provided.

A Markov decision process for response adaptive designs

The randomized treatment allocation process in a response adaptive clinical trial is formulated as a stochastic sequential decision problem and an algorithm is proposed to approximate the optimal value under the average reward criterion. When the information of previous treatment allocations and associated responses are summarized with sufficient statistics for unknown parameters, the decision process becomes a Markov process, on which a span-contractor operator is defined. It is proven that the average reward under the policy identified from the span-contractor operator converges almost surely to the optimal value. Numerical results reveal that the sequential procedure based on the controlled Markov process shows superior ethical advantage and at the same time produces good statistical power for large sample sizes such as 200 or larger.

Decision making under uncertain and dependent system rates in service systems

In this paper, we propose decision analysis methods for determining the optimal number of agents of a service system where the system rates (arrival, service, and abandonment) are modeled as dependent random variables. In doing so, we take the Bayesian point of view of inference and obtain joint posterior distributions of the system rates. We solve the proposed stochastic staffing decision problem with augmented probability simulation based optimization methods. The novelty of our approach stems from the use of dependent system rates to determine optimal staffing in a constrained optimization setting for stochastic service systems. We demonstrate the implications of ignoring dependence and uncertainty in system rates on simulated data for general service systems, and illustrate the application of the proposed methodology on call center operations.

Building Trust in Home Services—Stochastic Team-Orienteering with Consistency Constraints

In this paper, we consider service applications where drivers serve subscription customers at their homes on a regular basis and at known times. To build trust with customers, the company requires that subscription customers are consistently served by the same driver. In addition to subscription customers, on-demand customers request delivery on a daily basis. For the company, the challenge is to consistently serve the subscription customers while simultaneously maximizing the daily profit from the on-demand customers. We model the problem as a two-stage stochastic decision problem. The first stage determines the assignment of drivers to subscription customers. The second stage is a team-orienteering problem with time windows and mandatory visits by fixed drivers. We present an anticipatory consistent customer assignment policy (ACCA) based on the multiple scenario approach framework. Our computational study shows that ACCA significantly outperforms consistency concepts from the literature, while increasing costs less than 5%.

An Exact Algorithm to Make a Trade-Off between Cost and Probability in SSPs

In stochastic sequential decision problems, such as Stochastic Shortest Path Problems, the GUBS (Goal with UtilityBased Semantics) criterion considers a trade-off between probability-to-goal and cost-to-goal using a goal semantics based on Expected Utility Theory (EUT); in such a semantics, goal paths have priority over non-goal paths, but it implies neither the MAXPROB criterion nor the dual optimization criterion that finds the cheapest policy among the policies that maximize goal probability. Whereas evaluation criteria based on a sound theory such as EUT are desirable, optimal policies under GUBS are non-markovian. Non-markovian solutions are undesirable because there is not always a finite representation and even if it can be represented in a finite way, the representation may be too large to be stored. Here we define a special case of GUBS criterion that allows a finite representation to the optimal policy, the eGUBS criterion, where the cost utility function is exponencial. Considering this special case, we contribute with: (i) the proof that the eGUBSoptimal policy has a finite representation; (ii) the first exact algorithm to obtain finite optimal policies for the eGUBS criterion, and (iii) four strategies to find sub-optimal policies. We conduct experiments on one synthetic problem to evaluate each strategy. Although optimal solutions have a high memory cost, sub-optimal policies can save memory space with a small decrease in performance.

Estimating Private Beliefs of Bayesian Agents Based on Observed Decisions

We consider sequential stochastic decision problems in which, at each time instant, an agent optimizes its local utility by solving a stochastic program and, subsequently, announces its decision to the world. Given this action, we study the problem of estimating the agent’s private belief (i.e., its posterior distribution over the set of states of nature based on its private observations). We demonstrate that it is possible to determine the set of private beliefs that are consistent with public data by leveraging techniques from inverse optimization. We further give a number of useful characterizations of this set; for example, tight bounds by solving a set of linear programs (under concave utility). As an illustrative example, we consider estimating the private belief of an investor in regime-switching portfolio allocation. Finally, our theoretical results are illustrated and evaluated in numerical simulations.

Event-based optimization approach for solving stochastic decision problems with probabilistic constraint

In many practical control systems or management systems, the manager of systems may allow that the statistic probability of system error or parameter deviation occurs within a certain range. The problem of decision optimization under probabilistic constraint is thus an issue needs to be addressed urgently. In this paper, we consider to develop an event-based approach which can solve the probabilistic constrained decision problems in discrete events dynamic systems. The framework of the event-based optimization is first introduced, and then with the methodology of the performance sensitivity analysis, we present an online event-based policy iteration algorithm based on the derived performance gradient formula. We apply the event-based idea and propose the concept of “risk state”, “risk event” and “risk index” which can be used to better describe the nature of the probabilistic constrained problem. Furthermore, by taking the Lagrangian approach, the constrained decision problem can be solved with two steps. Finally, numerical experiments are designed to verify the efficiency of the proposed method.

Designing robust liner shipping schedules: Optimizing recovery actions and buffer times

It is important for liner shipping companies to maintain cost efficient and robust liner shipping networks. Regularly, they set up pro-forma schedules, yet it is difficult to stay on time. We consider the problem of managing the delays. Therefore, we need to determine an optimal recovery policy and buffer time allocation to the ship route in order to minimize the total costs associated with delays and recovery actions, such as increasing sailing speed. We introduce a general framework consisting of a mixed integer programming formulation to solve discrete stochastic decision problems with short and long term decisions and apply this framework to the above described problem. Furthermore, we propose and test four heuristics for this problem. We compared the results of our method with an existing liner shipping route schedule and found a cost decrease of 28.9% after optimizing the buffer time distribution compared to the cost of sailing the current route schedule at constant speed.

A Recursive Scenario Decomposition Algorithm for Combinatorial Multistage Stochastic Optimisation Problems

Stochastic programming is concerned with decision making under uncertainty, seeking an optimal policy with respect to a set of possible future scenarios. This paper looks at multistage decision problems where the uncertainty is revealed over time. First, decisions are made with respect to all possible future scenarios. Secondly, after observing the random variables, a set of scenario specific decisions is taken. Our goal is to develop algorithms that can be used as a back-end solver for high-level modeling languages. In this paper we propose a scenario decomposition method to solve multistage stochastic combinatorial decision problems recursively. Our approach is applicable to general problem structures, utilizes standard solving technology and is highly parallelizable. We provide experimental results to show how it efficiently solves benchmarks with hundreds of scenarios.

Resolving Time Inconsistency in Financial Decision Problems With Non-Expectation Operator: From Internal Conflict to Internal Harmony by Strategy of Self-Coordination

When a stochastic decision problem is time inconsistent, the decision maker would always be troubled by his conflicting decisions “optimally” derived from his time-varying preferences at different time instants. The long-run self (LR) of the decision maker pursues the long-term optimality and thus would like to take the pre-committed strategy. On the other hand, the short-run selves (SRs) of the decision maker at different time instants lack foresight, bow to short-term temptations and would like to take the so-called time consistent strategy. As these two extreme strategies have been criticized for their incomplete viewpoints in dealing with time inconsistency, the emerging literature of self-control game models recognizes an importance to strike a balance between the LR’s and SRs’ interests. While most risk measures in financial decisions are of a non-expectation nature, the existing results of self-control game models are confined to situations where the decision maker’s preferences involve only expectation operators. We take a step forward to develop in this study an operable unified two-tier dual-self game model with commitment by punishment in continuous time setting, which can cope with general time inconsistent stochastic financial decision problems with both expectation and non-expectation operators in the objective function. By attaching to both the preferences of LR and SRs certain punishment terms which quantitatively evaluate the internal conflict among different selves, our game model can align the interests of the LR and SRs to any desired degree. The equilibrium strategy, termed strategy of self-coordination in this paper, achieves some degree of internal harmony among various selves. We demonstrate in the paper successful applications of our dual-self game model to two long standing challenges in the literature, the continuous-time investment and consumption problem with quasi-hyperbolic discounting and the continuous-time mean-variance portfolio selection problem.

Using Ranked Weights and Acceptability Analysis to Construct Composite Indicators: A Case Study of Regional Sustainable Society Index

A variety of published composite indicators, i.e., Energy Trilemma Index and Sustainable Society Index, are commonly aggregated with equal weights. However, this plausible scheme is criticized as eclecticism and ignores the discriminating power of the different indicators. Differing from the traditional methods that assign weights to each indicator for the purpose of aggregation, this paper proposes a new mechanism to construct composite indicators using ranked weights and stochastic multicriteria acceptability analysis. More specifically, this study comprehensively consider all possible preferences among the indicators. Under each preference, we develop a sophisticated mathematical transformation to calculate the least and most favorable scores of each entity, which formulates the lower and upper bounds of the intervals. Then an interval decision matrix, alternatively described as a stochastic decision problem, is formulated to construct the composite indicators. Holistic acceptability indices are generated and regarded as a new composite indicator, which is capable of providing a comprehensive and robust composite indicator with more discriminating power. We apply the proposed method to modify the regional sustainable society index and present the obtained results and comparisons.

The Dating Problem and Optimal Ordering of Sequential Opportunities

In this paper, we discuss a stochastic sequential decision problem of optimally selecting the order in which to try n opportunities that may yield an uncertain reward in the future. The motivation came out from curiosity, after an informal conversation about what could be the best way to date friends. The problem structure turned out to be suitable also for other situations, such as the problem of optimally selecting the order of submission of a paper to journals. Despite the seemingly combinatorial nature of the problem, we show that optimal-tradeoff solutions can be found by simply ordering a sequence of n real numbers.