Multistage Decision Problem Research Articles

Reinforcement learning for energy management of an islanded microgrid: Analysis of state, state space and reinforcement function

The energy management (EM) of a microgrid is a challenging problem because of the stochasticity of renewable sources and loads, which can be formulated as a multi-stage decision-making problem. Reinforcement learning (RL), is found to be suitable for solving multi-stage decision-making problems under uncertainty. With the recent advancements in deferrable types of loads like that of electric vehicles, there is flexibility to manage peak demand, thereby saving the microgrid's operation cost. In this paper, we systematically explain for the first time how RL can be used to solve the scheduling problem of an isolated MG. A novel Q-learning algorithm employing a state-dependent action set is proposed to solve an isolated microgrid's EM problem in the presence of critical and deferrable loads. Various scenarios of wind turbine output power are generated to test the algorithm under a stochastic environment. In this work, the action set is divided into two: one with all the actions included and the second with a reduced number of actions. The entire state space is also divided into two: one set in which the deferrable load needs to be allotted and other where it need not be allotted. The advantage of such a method is that it substantially reduces the computational burden even if the state space's dimension is higher. Also, the judicious selection of reinforcement function to meet all the objectives and constraints is illustrated with case studies. The proposed method yields optimized scheduling with significant cost reductions compared to existing work.

Global Trajectory Optimization of Multispacecraft Successive Rendezvous Using Multitree Search

This paper investigates the global optimization of multispacecraft successive rendezvous trajectories, which is divided here into three subproblems: target assignment, sequence optimization, and rendezvous time optimization. A method consisting of two novel algorithms is proposed to solve these subproblems. First, a multitree search framework is developed to assign multiple targets to each spacecraft and simultaneously optimize the rendezvous sequence for every single spacecraft. Specifically, a novel algorithm of local search combined with beam search is proposed. Second, this paper converts the rendezvous time optimization problem into a multistage decision problem. Based on a critical rendezvous-epoch-dependent characteristic found in this subproblem, the number of state variables is thereby reduced. A novel dual dynamic programming algorithm is proposed and combined with dynamic programming to solve for the globally optimal rendezvous epochs efficiently. Global optimality is guaranteed by Bellman’s principle of optimality, which is the first time in such a problem to our knowledge. The proposed method achieves state-of-the-art performance in several typical fuel-optimal scenarios of active debris removal. This open-sourced method is non-database-dependent and contains only one design stage, which is expected to be adopted in other successive rendezvous missions.

Z-relation-based multistage decision making

Multistage decision problems have attracted interest in various fields. Starting from the pioneering work of Bellman and Zadeh, a large class of comprehensive studies for fuzzy, stochastic and other settings were proposed. The main issue missed in the existing works is partial reliability of information. Indeed, partial reliability of information about states and actions is important for multistage decisions. Particularly, information about future state of a system resulting from current strategies is partially reliable due to uncertainty. To account for partial reliability aspect, Zadeh introduced the concept of a Z-number. In this paper, we use this concept to consider multistage decision making under partially reliable information. States, strategies, and goals are described by using Z-numbers. State transition matrix that describes evolution of a system in time is formed by using Z-number-based relations. This description leads to a problem of dynamic programming in Z-environment. We use Z-numbers to account for the fact that information related to state transitions is partially reliable – we are partially sure in a result of each transition. The reason behind partial reliability is complexity and uncertainty of real-world problems. A typical example and an application study are used to illustrate the approach.

An intelligent control method for the spraying amount of a self-propelled weeder based on a dynamic programming algorithm

When the self-propelled weeder reaches an obstacle, the amount of pesticide spraying does not change according to the actual situation, leading to an application imbalance. Therefore, an intelligent control method of spraying output of a self-propelled weeder based on a dynamic programming algorithm (DPA) was proposed. The dynamic norm algorithm transformed obstacle avoidance path planning into a multi-stage decision-making problem. The obstacle avoidance path was planned, and the pesticide dosage control equation was designed by using the PID control algorithm to realize the obstacle avoidance and variable application of self-walking weed killer. The experimental results show that the designed self-walking weeder can effectively avoid obstacles, and the maximum average dosage error is 1.4%. Therefore, the method can effectively apply the dosage according to the actual situation and guarantee the balance of spraying quantity.

Enumeration optimization of open pit production scheduling based on mobile capacity search domain

The optimization of open pit mine production scheduling is not only a multistage decision-making problem but also involves space–time dynamic action among multiple factors, which makes it difficult to optimize production capacity, mining sequence, mining life, and other factors simultaneously in optimizing design. In addition, the production capacity is disorderly expanded, the calculation scale is large, and the optimization time is long. Therefore, this article designs a mobile capacity search domain method to improve computing efficiency without omitting the optimal production capacity. At the same time, taking the maximum net present value as the objective function, an enumeration method is used to optimize the possible paths in different capacity domains and calculate the infrastructure investment and facility idle cost required to meet the maximum production capacity on each possible path to control the disorderly expansion and violent fluctuation of production capacity. The research shows that the open pit mine production scheduling optimization algorithm proposed in this article can not only realize the simultaneous optimization of the three elements of production capacity, mining sequence, and mining life but also improve the computing efficiency by 200 times. Furthermore, the production capacity fluctuation is less than 1.4%. The mining life of the mine is extended by 13 years, and the overall economic benefit is increased by 18%.

The timing and mode of foreign exit from conflict zones: A behavioral perspective

We examine the timing and mode of firm exits from host-country conflict zones. We argue that timing and mode are interdependent decisions where decision ordering matters, and show that a firm’s prioritizing of either exit timing or mode is dependent on the relative salience of two behavioral stimuli: (1) the firm’s own experience (i.e., its performance shortfall), and (2) the experience of peer firms (i.e., their exits). Using instrumental variables modeling on a sample of 101 Japanese MNE exits from 11 conflict-afflicted countries between 1991 and 2005, we demonstrate that, when mode is prioritized over timing, partial exits tend to occur earlier and whole exits later. However, when timing is prioritized over mode, the decision choices reverse: earlier exits tend to be whole and later exits partial. The outcome of one decision therefore affects that of the other in a unique and predictable manner, such that the ordering of the decisions both produces and precludes strategic choices. Our findings, based on a multidecision problem that has traditionally been treated as a single decision (i.e., foreign exit), delineate expanded boundary conditions for satisficing, as well as reconcile optimizing and satisficing behaviors.

Multi-agent deep reinforcement learning for online request scheduling in edge cooperation networks

Edge computing as a complementary paradigm of cloud computing has gained more attention by providing mobile users with diversified services at the network edge. However, the increasingly complex mobile applications put a heavier load on edge networks. It is challenging to provide concurrency requests with high-quality service processing, especially when the edge networks are dynamically changing. To address the above issues, this paper investigates the online concurrent user requests scheduling optimization problem in edge cooperation networks. We model it as an online multi-stage decision-making problem, where requests are divided into a group of independent and logically related sub-tasks. We proposed a centralized training distributed execution based multi-agent deep reinforcement learning technique to realize the implicit cooperation scheduling decision-making policy learning among edge nodes. At the centralized training stage of the proposed mechanism, a value-decomposition-based policy learning technique is adopted to improve the long-term system performance, while at the distributed execution stage, only local environment status information is needed for each edge node to make the request scheduling decision. Extensive experiments are conducted, and simulation results demonstrate that the proposed mechanism outperforms other request scheduling mechanisms in reducing the long-term average system delay and energy consumption while improving the throughput rate of the system.

Bayesian Optimization for Cascade-Type Multistage Processes.

Complex processes in science and engineering are often formulated as multistage decision-making problems. In this letter, we consider a cascade process, a type of multistage decision-making process. This is a multistage process in which the output of one stage is used as an input for the subsequent stage. When the cost of each stage is expensive, it is difficult to search for the optimal controllable parameters for each stage exhaustively. To address this problem, we formulate the optimization of the cascade process as an extension of the Bayesian optimization framework and propose two types of acquisition functions based on credible intervals and expected improvement. We investigate the theoretical properties of the proposed acquisition functions and demonstrate their effectiveness through numerical experiments. In addition, we consider suspension setting, an extension in which we are allowed to suspend the cascade process at the middle of the multistage decision-making process that often arises in practical problems. We apply the proposed method in a test problem involving a solar cell simulator, the motivation for this study.

Improved progressive optimality algorithm and its application to determination of optimal release trajectory of long-term power generation operation of cascade reservoirs

Dynamic programming (DP) is an effective method for solving multi-stage decision-making problems and has been extensively applied to the optimization of reservoir operations. However, the method is limited by the dimensionality problem; hence, it cannot be directly applied to optimizing the operations of a reservoir system with more than three reservoirs at current computing power. The progressive optimality algorithm (POA) is a classic variant of DP and has been widely used for optimizing multi-reservoir operations. Despite the use of a static variable decoupling strategy to ease DP's dimensionality problem, the POA's performance is reduced by an inherent drawback of the blind search of the static variable decoupling strategy and the dimensionality problem in two-stage solutions. To enhance the POA's performance, we propose an improved version of the POA known as the dipole optimization procedure (DOP) for optimizing cascade reservoir operations. In the improved algorithm, we use a dynamic variable decoupling strategy to obtain search directions that are more targeted than those of the POA so as to improve the quality of the solutions. The dynamic decoupling of variables was achieved by constructing and solving a dipole optimization problem using a DP model developed in this study. Moreover, a perturbation mechanism was introduced to address the POA's dimensionality problem so that the algorithm can be extended to larger reservoir systems. The results of the simulation of a hypothetical four-reservoir system and a real-world cascade reservoir system showed the superiority of the DOP over the POA and other five available alternatives. The comparison was based on the quality of solutions and solving efficiency. The results indicate that the DOP is rational and feasible and has the potential to be applied to optimizing the operation of large-scale multi-reservoir systems with numerous reservoirs.

Design of an intelligent support system for fabric quality inspection

Efficient quality management in production process is the key factor for firm’s permanence and prosperity. The rapped globalization development and shortage of resources leading to enhance the efforts toward good raw material exploitation. The present work aims to develop a Decision Support System (DSS) that may provide and facilitate one of the most difficult multi-decision problems that quality managers, in textile manufacturing firm face. In addition, the DSS is developed for a textile manufacturer and it will automate a variety of tasks to improve rolls quality; control the defect distribution on rolls by inventing anew cutting scenarios with respect to inspection results, rolls length, and number of assemblies. The DSS tool, applied for more than 100 fabric lots, will be demonstrated through a short selection of practical case studies.

Multistage stochastic decision problems: Approximation by recursive structures and ambiguity modeling

Stochastic multistage decision problems appear in many - if not all - application areas of Operations Research. While to define such problems is easy, to solve them is quite difficult, since they are of infinite dimension. Numerical solution can only be found by solving an approximate, easier problem. In this paper, we show good approximations can be found, where we emphasize the recursive structure of the involved algorithms and data structures.In a second part, the problem of coping with the model error of approximations is discussed. We present algorithms for finding distributionally robust solutions for the model error problem. We also review some application cases of such situations from the literature.

Dynamic Allocation of Medical Resources During the Outbreak of Epidemics

During the outbreak of epidemics such as coronavirus disease (COVID-19), the local hospitals often withstand a sharp increase of patient influx, which renders the healthcare system on the verge of collapse. To alleviate the situation, the effective allocation of scarce medical resources during the pandemic plays a vital role. The essence of the healthcare system in time of emergency is to stay functional, and to be able to diagnose and hospitalize as many patients as possible. Fangcang shelter hospital, as a novel way to temporarily increase the capacity of the local healthcare system, is proven to be effective against the COVID-19 pandemic. To improve the performance of the healthcare system with Fangcang, many practical factors need to be taken into account, such as the patient deterioration during waiting to be admitted, the referral mechanism according to the severity of the patients, and the selective admission regulations. To address the high volatility and time-varying feature of the COVID-19, a multistage and multi-type medical service network model is established, and a dynamic allocation strategy of the medical resources at each stage is proposed based on a stochastic optimization problem, which is then solved via the fluid queueing approximation. Combined with the real data collected from Wuhan, it is revealed that the proposed algorithm could help with the allocation of medical resources during the outbreak of epidemics. Even with limited medical resources available, the method could still maintain a guaranteed service level while keeping the healthcare system operational. Furthermore, the simulation analysis validates that our method can effectively allocate medical resources at each stage, so as to stabilize the system performance and fulfill the medical demand for multiple types of patients. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —To fend off the outbreak of epidemics, the lessons have to be learned from the past. The successful control of the spread of COVID-19 in Wuhan (China) is a classical example of applying modern medical practices and management tools. In the present article, the treating procedure of COVID-19 in Wuhan is modeled as a multistage decision problem, which includes the screening with nucleic acid testing, the further testing, the treatment of patients with mild/severe symptoms, or even critical patients. The introduction of Fangcang shelter hospital is crucial for winning the battle against COVID-19. The current study attempts to determine the timing of introducing the Fangcang shelter hospital during the outbreak of a major epidemic, and helps allocate the medical resources needed to contain the spread of the virus. It is discovered that the actual number of beds in the Fangcang shelter hospital is far more than what is necessary, and it would be better to have built the Fangcang some time in advance. In the meantime, the number of designated hospitals for COVID-19 is in line with the results obtained via the optimal staffing strategy proposed here, but it is also noticeable that these hospitals should be released of duty sooner to fight against not only COVID-19 but also other diseases in reality.

Decision programming for mixed-integer multi-stage optimization under uncertainty

Influence diagrams are widely employed to represent multi-stage decision problems in which each decision is a choice from a discrete set of alternative courses of action, uncertain chance events have discrete outcomes, and prior decisions may influence the probability distributions of uncertain chance events endogenously. In this paper, we develop the Decision Programming framework which extends the applicability of influence diagrams by developing mixed-integer linear programming formulations for such problems. In particular, Decision Programming makes it possible to (i) solve problems in which earlier decisions cannot necessarily be recalled later, for instance, when decisions are taken by agents who cannot communicate with each other; (ii) accommodate a broad range of deterministic and chance constraints, including those based on resource consumption, logical dependencies or risk measures such as Conditional Value-at-Risk; and (iii) determine all non-dominated decision strategies in problems which multiple value objectives. In project portfolio selection problems, Decision Programming allows scenario probabilities to depend endogenously on project decisions and can thus be viewed as a generalization of Contingent Portfolio Programming (Gustafsson & Salo, 2005). We present several illustrative examples, evidence on the computational performance of Decision Programming formulations, and directions for further development.

Optimal manpower recruitment and promotion policies for the finitely graded systems using dynamic programming

BackgroundAt the heart in the development of any organization or nation is human resource. Over the years the world over, a sharp increase in too many numbers of qualified persons is being experienced yearly. This has impacted on recruitment and promotion costs to increase immensely, thereby affecting negatively manpower system costs. Dynamic Programming (DP) approach to optimal manpower recruitment and promotion policies for the two grade system has been proposed. MethodsConsidering the fact that contemporary manpower systems are not limited to just two grades - a kind of “switch-approach” to manpower systems, we first establish the link between a manpower planning problem and a dynamic decision-making process. This linkage resulted to a multistage real-life decision-making problem whose solution demands that decisions be made sequentially at different levels and at different points in time and space. Dynamic Programming is a mathematical technique well appropriate for the optimization of multistage decision problems. This allows us to give a generalization to manpower systems by modifying the model to finite grades which came out to be more robust and actionable, a constrained deterministic Dynamic Programming (CDDP) found to function computationally as the very well-known Wagner-Whitin Model in inventory/production management. Five cost variables associated with manpower planning were identified and used as inputs to the modified deterministic DP model. Results/RelevanceThe data shows yearly recruitments and promotions totaling 507 and 266 staff respectively for a ten-period (year) planning horizon. Total manpower system cost (in ooo's of Nigerian Naira) occasioned by yearly recruitments and promotions exercises for the period is 11334 (7092 for recruiting, 4100 for promoting, and 142 for overstaffing). Our DP model minimizes the manpower system cost to 9462 making a significant reduction of 1872. The optimal policy for the planning period calls for recruiting and promoting respectively 79 and 41 in period 1 only, 86 and 24 in period 2 for periods 2 and 3, 86 and 46 in period 4 for periods 4 and 5, 89 and 29 in period 6 only, 85 and 70 in period 7 for periods 7 and 8, and 82 and 56 in period 9 for periods 9 and 10.The study will contribute to the growing literature on applications of OR models to problems in manpower planning. The model outcomes would provide the basis for evaluating decision policies aimed at conducting recruitment promptly and to eliminate over-stagnated promotions. Conclusion/Further researchWe formulate decisions making in a finitely-graded manpower system as a multistage decision-making optimization problem which are best handled by dynamic programming. Five cost variables associated with manpower planning were identified and used as inputs to the modified deterministic DP model. Our model is resolute for minimizing manpower system costs occasioned by recruitments and promotions exercises in a wide range of multi-graded manpower systems instead of just two grades. The study's limitations and scope for future research work are presented in the end.

An Research on the design and optimization of shipping routes in the Arctic

In the design and research of the Arctic route, the route design can be interfered by other factors. This paper studies the route design and optimization after considering the interference of external factors. According to the external environment factors to design the arctic route weather route, the first arctic route design model was established, using the method of dynamic programming. Therefore, it is necessary to continuously determine the shortest sailing time of the ship, namely the actual sailing speed of the ship. According to the sailing conditions of the ship in different environments, so that the highest degree of navigation safety can be achieved, and the route optimization problem is transformed into a multi-stage decision-making problem. The global optimal solution is obtained by looking for the local optimal solution of each stage. Combined with the original Arctic route model, the meteorological route is optimized. By separating the present period from the future period, the present and future benefits are considered together. By determining the stage index function and the optimal index function, the basic equation of dynamic programming is established. The Pareto front curve corresponding to the requirements of the shortest voyage time and the voyage safety is given, that is, the optimal solution function curve of the two objectives. The influence between the shortest voyage time and the voyage safety is weighed comprehensively, and the optimal solution is obtained by combining the actual situation of the ship during the voyage, that is, the route optimization is completed. Finally, the optimized Arctic route is simulated and evaluated in an electronic computer simulation system with similar environment.

Decision-Making Under Uncertainty for a Digital Thread-Enabled Design Process

AbstractDigital thread is a data-driven architecture that links together information from all stages of the product lifecycle. Despite increasing application in manufacturing, maintenance/operations, and design related tasks, a principled formulation of analyzing the decision-making problem under uncertainty for the digital thread remains absent. The contribution of this article is to present a formulation using Bayesian statistics and decision theory. First, we address how uncertainty propagates in the product lifecycle and how the digital thread evolves based on the decisions we make and the data we collect. By using these mechanics, we explore designing over multiple product generations or iterations and provide an algorithm to solve the underlying multistage decision problem. We illustrate our method on an example structural design problem where our method can quantify and optimize different types and sequences of decisions, ranging from experimentation, manufacturing, and sensor placement/selection, to minimize total accrued costs.

A Noval Fuzzy Financial Investment Algorithm based on Combinatorial Entropy Value Method

According to the multi-objective and multi-decision problem of traditional financial Investment, a Financial Fuzzy Investment model based on Combinatorial Entropy Value Method is proposed. The company financial index system and index calculation model are constructed, an improved entropy method is proposed to calculate the index weight value, which can comprehensively consider the influence of historical evaluation data and current evaluation data, and the fuzzy membership function for investment satisfaction is proposed. Finally, an example is used to verify the effectiveness of the Investment model.

Triangular Neutrosophic Cognitive Map for Multistage Sequential Decision-Making Problems

Nowadays, fuzzy cognitive maps (FCMs) are one of the most efficient artificial intelligence techniques for modeling large and complex systems. However, traditional FCMs have the limitation of not representing the indeterminacy situations presented in many decision-making problems. To overcome this limitation, neutrosophic cognitive maps (NCMs) were proposed as a new extension of traditional FCMs. Nevertheless, the way that NCMs reported in the bibliography handle the indeterminacy is still insufficient since they cannot quantify the degree of indeterminacy. Moreover, there are decision-making problems in which decisions should be considered as a sequence of decisions hardly interconnected in sequential order. This situation is presented in scenarios such as projects evaluation characterized by the existence of multiple interconnected processes (diagnosis, decision, and prediction). The lack of a suitable FCMs topology for modeling this kind of decision-making problems constitutes another challenging issue of FCMs. This paper presents a new neutrosophic cognitive map based on triangular neutrosophic numbers for multistage sequential decision-making problems (MS-TrNCM). In the proposed model, all the map’s connections are represented by triangular neutrosophic numbers, making it possible for decision makers to express their preferences considering the truth, indeterminacy, and falsity degrees. Furthermore, a new topology for representing multistage sequential processes is introduced. The suggested MS-TrNCM is applied to make diagnoses, decisions, and predictions during the evaluation of 1011 projects records from project evaluation database ”uci-gp-eval-201903051137” provided by the University of Informatics Sciences. In validation process, the superiority of the proposed MS-TrNCM over the NCMs and traditional FCMs has been demonstrated.

Multiagent Reinforcement Learning: Rollout and Policy Iteration

We discuss the solution of complex multistage decision problems using methods that are based on the idea of policy iteration (PI), i.e., start from some base policy and generate an improved policy. Rollout is the simplest method of this type, where just one improved policy is generated. We can view PI as repeated application of rollout, where the rollout policy at each iteration serves as the base policy for the next iteration. In contrast with PI, rollout has a robustness property: it can be applied on-line and is suitable for on-line replanning. Moreover, rollout can use as base policy one of the policies produced by PI, thereby improving on that policy. This is the type of scheme underlying the prominently successful AlphaZero chess program. In this paper we focus on rollout and PI-like methods for problems where the control consists of multiple components each selected (conceptually) by a separate agent. This is the class of multiagent problems where the agents have a shared objective function, and a shared and perfect state information. Based on a problem reformulation that trades off control space complexity with state space complexity, we develop an approach, whereby at every stage, the agents sequentially (one-at-a-time) execute a local rollout algorithm that uses a base policy, together with some coordinating information from the other agents. The amount of total computation required at every stage grows linearly with the number of agents. By contrast, in the standard rollout algorithm, the amount of total computation grows exponentially with the number of agents. Despite the dramatic reduction in required computation, we show that our multiagent rollout algorithm has the fundamental cost improvement property of standard rollout: it guarantees an improved performance relative to the base policy. We also discuss autonomous multiagent rollout schemes that allow the agents to make decisions autonomously through the use of precomputed signaling information, which is sufficient to maintain the cost improvement property, without any on-line coordination of control selection between the agents. For discounted and other infinite horizon problems, we also consider exact and approximate PI algorithms involving a new type of one-agent-at-a-time policy improvement operation. For one of our PI algorithms, we prove convergence to an agent-by-agent optimal policy, thus establishing a connection with the theory of teams. For another PI algorithm, which is executed over a more complex state space, we prove convergence to an optimal policy. Approximate forms of these algorithms are also given, based on the use of policy and value neural networks. These PI algorithms, in both their exact and their approximate form are strictly off-line methods, but they can be used to provide a base policy for use in an on-line multiagent rollout scheme.

Deduplication-Oriented Mutual-Assisted Cooperative Video Upload for Mobile Crowd Sensing

Deduplication (redundancy elimination) and cooperative video delivery are two effective ways to save the bandwidth and energy consumption and ensure video collection in damaged networks. However, deduplication in mobile crowd sensing (MSC) is primarily performed on texts and images. Furthermore, most of deduplication technologies require global information and are separated from video routing. To solve such problems, this paper propose a cooperative upload method for sensing videos, which performs the local video deduplication without excessive comparison and feature exchange. Also, we combine the content-aware deduplication with the dynamic relay selection to avoid the propagation of redundant items caused by the content-free video routing. Besides, we integrate a novel mutual-assisted mechanism into our method to motivate relay cooperation and load balance. We formulate the deduplication-supported cooperative video upload as a multi-stage decision problem. To solve the uncertainty of destinations in the decision problem, we develop a stepwise Mutual-Assisted Video Upload Algorithm (MAVU) to schedule video chunks and remove duplicates. Extensive experiments are conducted to compare MAVU with the existing algorithms. The numerical results validate that our MAVU has advantages over the other algorithms in collected video size and upload latency