Multistage Decision Process Research Articles

A generalized outcome-adaptive sequential multiple assignment randomized trial design.

A dynamic treatment regime (DTR) is a mathematical representation of a multistage decision process. When applied to sequential treatment selection in medical settings, DTRs are useful for identifying optimal therapies for chronic diseases such as AIDs, mental illnesses, substance abuse, and many cancers. Sequential multiple assignment randomized trials (SMARTs) provide a useful framework for constructing DTRs and providing unbiased between-DTR comparisons. A limitation of SMARTs is that they ignore data from past patients that may be useful for reducing the probability of exposing new patients to inferior treatments. In practice, this may result in decreased treatment adherence or dropouts. To address this problem, we propose a generalized outcome-adaptive (GO) SMART design that adaptively unbalances stage-specific randomization probabilities in favor of treatments observed to be more effective in previous patients. To correct for bias induced by outcome adaptive randomization, we propose G-estimators and inverse-probability-weighted estimators of DTR effects embedded in a GO-SMART and show analytically that they are consistent. We report simulation results showing that, compared to a SMART, Response-Adaptive SMART and SMART with adaptive randomization, a GO-SMART design treats significantly more patients with the optimal DTR and achieves a larger number of total responses while maintaining similar or better statistical power.

Post-disaster repair optimization method for traction power supply system of electrified railways based on train operation loss

The traction power supply system (TPSS) is the critical facility that provides power to electrified railways. However, TPSS is highly susceptible to extreme events. Therefore, it is essential to develop reasonable post-disaster repair strategies to mitigate losses. In this context, this paper proposes a post-disaster repair optimization method for TPSS based on train operation loss. The hazard characterization and equipment fragility analysis are introduced to simulate post-disaster fault scenarios of TPSSs. By analyzing the changes in component states during the repair process, the component state transition model based on the multi-stage decision process is established and the state transition relationship is derived. According to the component state and TPSS topology, the model for evaluating the power supply state of the traction network is constructed. On this basis, the indicator of train operation loss is defined, which is used as the objective to formulate the post-disaster repair optimization model of TPSSs. The case study using real-world high-speed railway data demonstrates that, compared to traditional repair strategies, the proposed method effectively reduces the restoration time of TPSS and significantly decreases train operation loss. This method can provide maintenance personnel with theoretical bases for formulating post-disaster repair strategies.

A Dynamic Programming Approach to the Collision Avoidance of Autonomous Ships

The advancement of autonomous capabilities in maritime navigation has gained significant attention, with a trajectory moving from decision support systems to full autonomy. This push towards autonomy has led to extensive research focusing on collision avoidance, a critical aspect of safe navigation. Among the various possible approaches, dynamic programming is a promising tool for optimizing collision avoidance maneuvers. This paper presents a DP formulation for the collision avoidance of autonomous vessels. We set up the problem framework, formulate it as a multi-stage decision process, define cost functions and constraints focusing on the actual requirements a marine maneuver must comply with, and propose a solution algorithm leveraging parallel computing. Additionally, we present a greedy approximation to reduce algorithm complexity. We put the proposed algorithms to the test in realistic navigation scenarios and also develop an extensive test on a large set of randomly generated scenarios, comparing them with the RRT* algorithm using performance metrics proposed in the literature. The results show the potential benefits of an autonomous navigation or decision support framework.

Modeling heterogeneous and adaptive homeowners’ decision-making for green retrofit

Green retrofitting (GR) of existing residential buildings is widely acknowledged as a potential solution for addressing climate and energy challenges. However, the realization of GR potential depends on homeowners’ decision-making, as they play a pivotal role in initiating GR and are the primary users of retrofitted homes. Previous studies have explored homeowners’ GR decisions from various perspectives. The majority of these studies may have encountered limitations, such as the use of static analysis methods, homogeneity assumptions, and oversimplified decision processes. These limitations can potentially hinder a comprehensive understanding of homeowners’ GR adoption, indicating the need for further research. To address this research gap, this study develops a decision-making model for homeowners that incorporates multiple dimensions: heterogeneity (e.g., individual attributes and GR motivations), a multi-stage decision process (needs-utility-willingness-adoption), and dynamic adaptability (both at individual and overall networks). The proposed model is empirically validated through Python simulations based on a case study in China. The findings of this study reveal the critical roles of GR needs, perceived (actual) utility, and perceived behavioral control in homeowners’ adoption of GR. Moreover, this study highlights the significance of coordinating homeowners’ overall perceived utility with the specific sub-utility associated with their motivations. Furthermore, empirical evidence underscores the influence of social factors on GR adoption and emphasizes the existence of gaps across different stages of the decision-making process. Drawing from these research findings, policy implications are proposed for promoting housing GR. This research contributes to the advancement of decision models for building GR and provides valuable insights for policymakers to promote housing GR.

Distributionally Robust Multiperiod Inventory Model for Omnichannel Retailing Considering Buy-Online, Pickup-in-Store and Out-of-Stock, Home-Delivery Services

This article proposes the distributionally robust multiperiod inventory model incorporating the buy-online-pickup-in-store (BOPIS) and out-of-stock-home-delivery-service (OSHDS), which are the representative services of omnichannel retailing. Under this omnichannel system, the retailer operates both online and brick-and-mortar (B&M) stores simultaneously, which allow interactive flows of customer demands and desired products. The BOPIS allows customers who buy products through the online store to pick them up in the B&M store. Meanwhile, the OSHDS allows customers who find the product they want out of stock in a B&M store to receive it later, through express delivery from the online store. To capture the correlated uncertain demands of the BOPIS and OSHDS, we adopt a factor-based demand model that is affinely dependent on predefined uncertain factors. To handle a multistage decision process under uncertain demands, we utilize a rule-based approximation and distributionally robust bound to derive a tractable formulation. Computational results achieved in this article offer some insights that BOPIS and OSHDS play a role in providing retailer with flexibility, which could handle unsatisfied demands efficiently.

Long-term optimal coordination of hydro-wind-thermal energy generation using stochastic dynamic programming

ABSTRACT Clean alternative energy and a greater focus on climate change aim to increase the integration of Renewable Energy Sources (RES) into power system networks. As a relatively inexpensive renewable energy, wind energy is integrated into the electrical network to reduce its operating costs. A long-term optimal scheduling model for hydro-wind-thermal in a hybrid generation system is established to find the minimum cost trajectory of energy generation at each period under various constraints. Based on the proposed model and different types of power plants, the original complex problem decomposed into hydro-wind-thermal subproblems. The stochastic Dynamic programming technique (SDP) is employed to solve the complete optimization. In this research, the SDP technique is preferred. This technique handles multistage decision processes by splitting problems down into sequential stages. Because it can incorporate nonlinear and stochastic features into a dynamic programming problem, it has been successful in this hybrid system. A penalty factor was added to the model to reduce outflow variations. As can be seen from the results, outflows are very high during peak demand periods and very low during high inflows. Furthermore, the cost decreases as demand increases, from 40,082.26 $/ GWh in May when demand is 10,275 Gwh to 16,536.32 $/ GWh in January when demand is 17,503 Gwh .

Improvement of Reinforcement Learning With Supermodularity.

Reinforcement learning (RL) is a promising approach to tackling learning and decision-making problems in a dynamic environment. Most studies on RL focus on the improvement of state evaluation or action evaluation. In this article, we investigate how to reduce action space by using supermodularity. We consider the decision tasks in the multistage decision process as a collection of parameterized optimization problems, where state parameters dynamically vary along with the time or stage. The optimal solutions of these parameterized optimization problems correspond to the optimal actions in RL. For a given Markov decision process (MDP) with supermodularity, the monotonicity of the optimal action set and the optimal selection with respect to state parameters can be obtained by using the monotone comparative statics. Accordingly, we propose a monotonicity cut to remove unpromising actions from the action space. Taking bin packing problem (BPP) as an example, we show how the supermodularity and monotonicity cut work in RL. Finally, we evaluate the monotonicity cut on the benchmark datasets reported in the literature and compare the proposed RL with some popular baseline algorithms. The results show that the monotonicity cut can effectively improve the performance of RL.

Crossing multiple solar energy gaps: A Dutch case study on intermediation for building-integrated photovoltaics

Building-integrated photovoltaics are a promising technology to enhance renewable energy production in the built environment while improving the aesthetics of buildings at the same time. Several challenges hinder this technology's uptake, such as information asymmetry and limited value chain coordination. Prior work demonstrates that the support of intermediaries can play a crucial role in coping with these challenges, but this aspect has not yet been previously investigated for this technology. A comprehensive overview is lacking of how various intermediaries can support a multi-stage decision process. Rather than focusing on specific intermediary actors, we explore the Dutch system for building-integrated photovoltaics, identifying which actors act or can act as an intermediary, and what intermediation activities can support the decision process. This article identifies the need for intermediation at the various stages of the decision process and the actors best suited to providing this. Drawing from our empirical findings from 26 in-depth interviews and the literature on innovation adoption and intermediaries, the results reveal that a dynamic ‘ecology of intermediaries’ is necessary to perform various intermediation activities at different system levels in the multi-stage decision process. As these activities and actors are highly interrelated and interdependent, we argue that it is vital to assess intermediation in a holistic way. These findings are significant for suppliers, potential intermediaries, and governments because they can support improving the decision process with the help of intermediation. The present paper contributes to innovation and intermediation studies by demonstrating that intermediation is an interrelated, multi-level, and variegated phenomenon.

Vectors of indicators and pointer function in the Multistage Bipolar Method

The Multistage Bipolar Method considered in the paper deals with multistage decision processes. Multistage alternatives are not compared directly to each other, but they are confronted with the stage sets of reference objects—desirable and non-acceptable. In the paper vectors and pointer functions are defined. The aim of the paper is to apply them to classify and rank multistage alternatives and search for the final solution. Our method simplifies the procedure of finding the final solution and allows to use single criterion dynamic programming to solve the problem.

Generalization of ALMM Based Learning Method for Planning and Scheduling

This paper refers to a machine learning method for solving NP-hard discrete optimization problems, especially planning and scheduling. The method utilizes a special multistage decision process modeling paradigm referred to as the Algebraic Logical Metamodel based learning methods of Multistage Decision Processes (ALMM). Hence, the name of the presented method is the ALMM Based Learning method. This learning method utilizes a specifically built local multicriterion optimization problem that is solved by means of scalarization. This paper describes both the development of such local optimization problems and the concept of the learning process with the fractional derivative mechanism itself. It includes proofs of theorems showing that the ALMM Based Learning method can be defined for a much broader problem class than initially assumed. This significantly extends the range of the prime learning method applications. New generalizations for the prime ALMM Based Learning method, as well as some essential comments on a comparison of Reinforcement Learning with the ALMM Based Learning, are also presented.

Comparing Assessment Methods of Attribute Importance in Teachers’ Decisions: The Importance of Different Criteria for Tracking Recommendations after Primary School

The importance of different criteria for tracking recommendations is usually inferred using regression weights as a cross-student measure. The few studies that have applied alternative approaches or differentiated between student groups sometimes reach different conclusions. According to research on judgment and decision making (JDM), different methods operationalize different facets of importance. Given this, we investigate whether the importance of criteria for tracking recommendations depends on a direct vs. indirect operationalization (regression weights vs. ratings). A total of 181 teachers selected four students from their most recent fourth-grade class using a 2 × 2 design (certain vs. uncertain qualification for the Realschule (vocational track) vs. the Gymnasium (academic track)). Then, they reported on the level and the importance of predetermined criteria for each student. Contrary to JDM research, we found few method-related differences, but striking differences between cases with a certain vs. an uncertain qualification. For the latter, the importance of the criteria is more homogeneous, the regression prediction is less successful and the importance varies with the dependent variable in the regression (actual recommendation vs. perceived qualification). We conclude that further research should focus on uncertain cases rather than method-related differences and suspect that, in uncertain cases, the formation of the recommendation is a multistage decision process.

A Stochastic Dual Dynamic Integer Programming for the Uncapacitated Lot-Sizing Problem with Uncertain Demand and Costs

We study the uncapacitated lot-sizing problem with uncertain demand and costs. We consider a multi-stage decision process and rely on a scenario tree to represent the uncertainty. We propose to solve this stochastic combinatorial optimization problem thanks to a new extension of the stochastic dual dynamic integer programming algorithm. Our results show that this approach can provide good quality solutions in a reasonable time for large-size instances.

Stochastic single machine scheduling problem as a multi-stage dynamic random decision process

In this work, we study a stochastic single machine scheduling problem in which the features of learning effect on processing times, sequence-dependent setup times, and machine configuration selection are considered simultaneously. More precisely, the machine works under a set of configurations and requires stochastic sequence-dependent setup times to switch from one configuration to another. Also, the stochastic processing time of a job is a function of its position and the machine configuration. The objective is to find the sequence of jobs and choose a configuration to process each job to minimize the makespan. We first show that the proposed problem can be formulated through two-stage and multi-stage Stochastic Programming models, which are challenging from the computational point of view. Then, by looking at the problem as a multi-stage dynamic random decision process, a new deterministic approximation-based formulation is developed. The method first derives a mixed-integer non-linear model based on the concept of accessibility to all possible and available alternatives at each stage of the decision-making process. Then, to efficiently solve the problem, a new accessibility measure is defined to convert the model into the search of a shortest path throughout the stages. Extensive computational experiments are carried out on various sets of instances. We discuss and compare the results found by the resolution of plain stochastic models with those obtained by the deterministic approximation approach. Our approximation shows excellent performances both in terms of solution accuracy and computational time.

Multiobjective dynamic programming in bipolar multistage method

The multicriteria Bipolar method can be extended and used to control multicriteria, multistage decision processes. In this extension, at each stage of the given multistage process two sets of reference points are determined, constituting a reference system for the evaluation of stage alternatives. Multistage alternatives, which are compositions of stage alternatives, are assigned to one of six predefined hierarchical classes and then ranked. The aim of this paper is to show the possibility of finding the best multistage alternative, using Bellman’s optimality principle and optimality equations. Of particular importance is a theorem on the non-dominance of the best multistage alternative, proven here. The methodology proposed allows to avoid reviewing each multistage alternative, which is important in large-size problems. The method is illustrated by a numerical example and a brief description of the sustainable regional development problem. The problem can be solved by means of the proposed procedure.

Economic and emission impacts of energy storage systems on power-system long-term expansion planning when considering multi-stage decision processes

Energy storage systems (ESS) are becoming a key component for power systems due to their capability to store energy generation surpluses and supply them whenever needed. However, adding ESS might eventually have unexpected long-term consequences and may not necessarily help in reducing CO2 emissions; mainly because they can store energy from any source and add some inefficiencies. Thus, it is important to accurately assess the impact they could have on the long-term prospects of the overall power system.In this work, those effects are assessed using a new adaptive two-stage generation, storage, and transmission expansion planning model, which includes constraints related to the system's flexibility, commonly disregarded in long-term expansion analysis. The effects of including an adaptive step in the proposed planning model, where investments can be adjusted to long-term demand trends, and those of including or ignoring ramp and reserve constraints are analyzed.The methodology is illustrated using a 45-bus representation of the Chilean power system. We use this case study to illustrate a counterexample where ESS can help reducing total system costs, but increasing CO2 emissions. We do not argue this would be mostly the case, but this counterexample helps to illustrate that this situation is possible. Likewise, we show that including the adaptive (two-step) approach significantly changes the generation and transmission expansion mix, as well as the emission levels. Finally, disregarding ramps does reduce total costs and slightly increases emissions in the Chilean case, but dismissing power reserves does not significantly affect the results.

A Turbo Q-Learning (TQL) for Energy Efficiency Optimization in Heterogeneous Networks

In order to maximize energy efficiency in heterogeneous networks (HetNets), a turbo Q-Learning (TQL) combined with multistage decision process and tabular Q-Learning is proposed to optimize the resource configuration. For the large dimensions of action space, the problem of energy efficiency optimization is designed as a multistage decision process in this paper, according to the resource allocation of optimization objectives, the initial problem is divided into several subproblems which are solved by tabular Q-Learning, and the traditional exponential increasing size of action space is decomposed into linear increase. By iterating the solutions of subproblems, the initial problem is solved. The simple stability analysis of the algorithm is given in this paper. As to the large dimension of state space, we use a deep neural network (DNN) to classify states where the optimization policy of novel Q-Learning is set to label samples. Thus far, the dimensions of action and state space have been solved. The simulation results show that our approach is convergent, improves the convergence speed by 60% while maintaining almost the same energy efficiency and having the characteristics of system adjustment.

Robust optimization for transmission defense against multi-period attacks with uncertainties

The power system is under increasing threat of terrorist attacks, and it is important to develop efficient methods to improve the power system resiliency for defending against the attacks. In this paper, the defense strategy of the transmission system in case of multi-period attacks considering uncertainties is investigated. The problem formulation involves a multi-stage decision process: first, the defender chooses the critical branches to harden for the purpose of minimizing the expected damage; second, the attacker may launch multiple attacks to disrupt the branches aiming to maximize the damage; third, following each of the attacks, the operator performs the optimal power re-dispatch to minimize the load shedding. A novel robust optimization-based Defender-Sequential-Attacker-Operator (DSAO) model is developed, which considers game-theoretic interactions between different agents (the defender, attacker and operator), selection of attacking targets, as well as the probabilities of attacks and correlations between attacks. A hybrid solution approach consisting of both robust optimization and stochastic programming is developed to solve the DSAO model. Case studies based on the IEEE RTS79 and 118-bus systems validate the proposed approach. In addition, detailed comparisons are made with the explicit enumeration method and the recent trilevel defender-attacker-defender (DAD) model, which demonstrates the accuracy and computational efficiency of the proposed approach.

Economical user-generated content (UGC) marketing for online stores based on a fine-grained joint model of the consumer purchase decision process

User-generated content (UGC) is influential in reducing customer perceived risk and determining online store sales. E-sellers spend huge costs and efforts to improve UGC for it serves as a convenient and persuasive alternative for marketing and advertising purposes. Considering that consumers may set lower and/or upper limits (i.e., psychological thresholds) in which the good is expected to be, and purchase decisions are considered as a multi-stage decision process, yet models in previous research cannot uncover this decision-making process. Therefore, exploring the impact of UGC at each decision-making stage and detecting the psychological thresholds on various aspects of UGC (i.e., the fine-grained effects of UGC) contribute to optimizing the UGC with the best cost to boost sales. To this end, a fine-grained joint two-stage decision model, zero-inflated negative binomial regression (ZINB-P) model is proposed to support economical UGC marketing. Specifically, we compile a factors system composed of various types of aggregate-level statistics of UGC, which can impact risk perception. Afterward, change point analysis is used to find multi-level consumer psychological thresholds on UGC factors and consumers’ risk perception model is constructed to measure purchasing probabilities in the first decision-making stage. On the basis of consumers’ risk perception model, the ZINB-P model is built to fully capture the fine-grained effects of UGC factors on each stage of the consumer purchase decision. It integrates two stages of consumer decision: the consumer risk perception and non-compensatory choice in the first stage, and the second compensatory stage. A genetic algorithm is constructed to jointly estimate the parameters in ZINB-P model. Finally, an experiment on a kind of fresh produce from Taobao.com evidences the precision of our model. We demonstrate how our model can provide with economical UGC marketing strategies using a decision support table, in which some scenarios are identified. E-sellers can use this table to find the scenarios they are located in and identify the critical UGC factors that impede the sales in each scenario, and thus economical UGC marketing strategies can be obtained by improving these critical UGC factors.

Dynamic optimization of compaction process for rockfill materials

The automatic adjustment of compaction parameters during a compaction process for increased efficiency is an important part in intelligent compaction (IC). However, there is a lack of effective methods for optimizing compaction parameters. This study proposes a compaction process-dynamic optimization method (CPDOM) based on a genetic algorithm (GA). The purpose of the CPDOM is to determine the optimal compaction plan based on the current compaction state to complete compaction within the shortest time. Therefore, field compaction tests with rockfill materials were conducted with various roller speeds and frequencies. Further, a relative density incremental function (RDIF) was established via a nonlinear multiple regression method. Based on the RDIF and the concept of a multistage decision process, CPDOM optimizes the compaction process globally via a GA to minimize the remaining compaction time. The main advantage of CPDOM is that the compaction parameters are optimized considering the overall optimal solution. Moreover, contrast tests were conducted in the Qianping reservoir. According to the results, the compaction efficiency improved by 13.1% through CPDOM optimization. Thus, CPDOM can be employed for engineering applications.

The multi-stage dynamic stochastic decision process with unknown distribution of the random utilities

We consider a decision maker who performs a stochastic decision process over a multiple number of stages, where the choice alternatives are characterized by random utilities with unknown probability distribution. The decisions are nested each other, i.e. the decision taken at each stage is affected by the subsequent stage decisions. The problem consists in maximizing the total expected utility of the overall multi-stage stochastic dynamic decision process. By means of some results of the extreme values theory, the probability distribution of the total maximum utility is derived and its expected value is found. This value is proportional to the logarithm of the accessibility of the decision maker to the overall set of alternatives in the different stages at the start of the decision process. It is also shown that the choice probability to select alternatives becomes a Nested Multinomial Logit model.