Stochastic Optimization Problem Research Articles

Co-evolutionary traffic signal control using reinforcement learning for road networks under stochastic capacity

A co-evolutionary traffic signal control using reinforcement learning approach (CORLA) is proposed for time-varying road networks under stochastic capacity. Classic reinforcement learning based traffic signal control cannot effectively reduce traffic congestion for large-scale road networks while standard evolutionary metaheuristics often suffer from significantly high computational cost. A co-evolutionary decomposition algorithm (CODA) is proposed to improve traffic mobility for urban road networks with time-varying traffic flow. To capture time-varying spatial evolution of traffic flow inside road links, a stochastic traffic model is presented. To fully consider road users’ response, a stochastic bi-level optimization problem (SBOP) is given where road users’ route choice can be fully taken into account. To efficiently implement CORLA in a large-scale road network against high-consequence realization for stochastic capacity, a coordinated co-evolutionary two phase control (CCTPC) is proposed. Numerical experiments are performed at a real-data city road network and various sizable traffic grids. As compared to state-of-the-art traffic signal control for various traffic conditions, obtained results showed that CCTPC exhibits sufficient gain of achieving road network performance and suffers from the least computational cost in all cases.

Regression based stochastic energy management strategy for a grid connected microgrid using Artificial Electric Field Algorithm

Recent decade has witnessed steep technological advancement in the renewable energy sector due to growing concern of climate change and emission cut target. The lack of availability of fossil fuels and the high price of the same have made the use of Renewable Energy Sources (RES) as a promising approach for generating clean and sustainable energy at local level. This persuades the implementation of Microgrid (MG) supported with RES, to cater various types of load demand. However, planning and optimizing a MG operation have become challenging due to the intermittent nature of RES and uncertain loads. To address this issue, this paper introduces a Stochastic Energy Management Strategy (SEMS) for a grid connected MG comprising Photovoltaic, Wind Turbine, Fuel Cell, Micro Turbine, Battery Energy Storage and electrical as well as heat energy demand. The volatilities of RES power output and energy demand are modeled via Gaussian Process Regression (GPR) and Monte Carlo Simulation (MCS) respectively. The problem is framed as non-linear (NL) dynamic stochastic optimization problem and solved using ‘Artificial Electric Field Algorithm (AEFA)’. Here, minimization of expected value of operational and emission cost are taken as objective functions; while various practicality aspects are modeled as constraints in the problem. Further, the efficacy of the proposed approach is validated around the first central moment of the load. Various case studies are performed to estimate the day ahead optimum operating schedule of the Distributed Energy Resources (DER) and to assess the impact of load uncertainties on DER operation and total MG operation cost. Further, the efficacy of the GPR is verified through comparison with MCS based result.

Approximate propagation of normal distributions for stochastic optimal control of nonsmooth systems

We present a method for the approximate propagation of mean and covariance of a probability distribution through ordinary differential equations (ODE) with discontinuous right-hand side. For piecewise affine systems, a normalization of the propagated probability distribution at every time step allows us to analytically compute the expectation integrals of the mean and covariance dynamics while explicitly taking into account the discontinuity. This leads to a natural smoothing of the discontinuity such that for relevant levels of uncertainty the resulting ODE can be integrated directly with standard schemes and it is neither necessary to prespecify the switching sequence nor to use a switch detection method. We then show how this result can be employed in the more general case of piecewise smooth functions based on a structure preserving linearization scheme. The resulting dynamics can be straightforwardly used within standard formulations of stochastic optimal control problems with chance constraints.

Decomposition methods for monotone two-time-scale stochastic optimization problems

Decomposition methods for monotone two-time-scale stochastic optimization problems

Nonparametric statistical inference for stochastic optimal control problems and its applications for financial investment

Against a backdrop of financial decision-making, this study examines nonparametric statistical inference problems in stochastic optimal control problems. First, we construct a nonparametric estimation model using the wavelet estimation method for drift and diffusion coefficients. In this study, we focus on stochastic linear–quadratic (SLQ) problems, solved using Riccati equations, and apply this to a cash management problem within the linear–quadratic framework. The results show the effectiveness of the wavelet method using a numerical case.

On the adaptation of the Lagrange formalism to continuous time stochastic optimal control: A Lagrange-Chow redux

We show how the classical Lagrangian approach to solving constrained optimization problems from standard calculus can be extended to solve continuous time stochastic optimal control problems. Connections to mainstream approaches such as the Hamilton-Jacobi-Bellman equation and the stochastic maximum principle are drawn. Our approach is linked to the stochastic maximum principle, but more direct and tied to the classical Lagrangian principle, avoiding the use of backward stochastic differential equations in its formulation. Using infinite dimensional functional analysis, we formalize and extend the approach first outlined in Chow (1992) within a rigorous mathematical setting using infinite dimensional functional analysis. We provide examples that demonstrate the usefulness and effectiveness of our approach in practice. Further, we demonstrate the potential for numerical applications facilitating some of our key equations in combination with Monte Carlo backward simulation and linear regression, therefore illustrating a completely different and new avenue for the numerical application of Chow's methods.

Constrained minimum variance and covariance steering based on affine disturbance feedback control parameterization

AbstractThis paper deals with finite‐horizon minimum‐variance and covariance steering problems subject to constraints. The goal of the minimum variance problem is to steer the state mean of an uncertain system to a prescribed vector while minimizing the trace of its terminal state covariance whereas the goal in the covariance steering problem is to steer the covariance matrix of the terminal state to a prescribed positive definite matrix. The paper proposes a solution approach that relies on a stochastic version of the affine disturbance feedback control parametrization. In this control policy parametrization, the control input at each stage is expressed as an affine function of the history of disturbances that have acted upon the system. It is shown that this particular parametrization reduces the stochastic optimal control problems considered in this paper into tractable convex programs or difference of convex functions programs with essentially the same decision variables. In addition, the paper proposes a variation of this control parametrization that relies on truncated histories of past disturbances, which allows for sub‐optimal controllers to be designed that strike a balance between performance and computational cost. The suboptimality of the truncated policies is formally analyzed and closed form expressions are provided for the performance loss due to the use of the truncation scheme. Finally, the paper concludes with a comparative analysis of the truncated versions of the proposed policy parametrization and other standard policy parametrizations through numerical simulations.

Parallel interior-point solver for block-structured nonlinear programs on SIMD/GPU architectures

ABSTRACT We investigate how to port the standard interior-point method to new exascale architectures for block-structured nonlinear programs with state equations. Computationally, we decompose the interior-point algorithm into two successive operations: the evaluation of the derivatives and the solution of the associated Karush-Kuhn-Tucker (KKT) linear system. Our method accelerates both operations using two levels of parallelism. First, we distribute the computations on multiple processes using coarse parallelism. Second, each process uses SIMD/GPU accelerators locally to accelerate the operations using fine-grained parallelism. The KKT system is reduced by eliminating the inequalities and the state variables from the corresponding equations. We demonstrate our method's capability on the supercomputer Polaris, a testbed for the future exascale Aurora system. Each node is equipped with four GPUs, a setup amenable to our two-level approach. Our experiments on the stochastic optimal power flow problem show that the reduction method is 50x faster than the sparse linear solver HSL MA57 running in serial on the CPU, and 6x faster than Pardiso running in parallel on CPU on the same number of processes.

Decentralized stochastic linear-quadratic optimal control with risk constraint and partial observation

This paper addresses a risk-constrained decentralized stochastic linear-quadratic optimal control problem with one remote controller and one local controller, where the risk constraint is posed on the cumulative state weighted variance in order to reduce the oscillation of system trajectory. In this model, local controller can only partially observe the system’s state, and sends the estimate of state to remote controller through an unreliable channel, whereas the channel from remote controller to local controller is perfect; here, the unreliable channel means the Bernoulli channel, that is, when data is transmitted on this channel, there is a certain probability of failure. For the considered constrained optimization problem with Bernoulli channel, we first punish the risk constraint into cost function through Lagrange multiplier method, and the resulting augmented cost function will include a quadratic mean-field term of state. In the sequel, for any but fixed multiplier, explicit solutions to finite-horizon and infinite-horizon mean-field decentralized linear-quadratic problems are derived together with necessary and sufficient condition on the mean-square stability of optimal system. Then, approach to find the optimal Lagrange multiplier is presented based on bisection method. Finally, two numerical examples are given to show the efficiency of the obtained results.

Robust Parameter Design on Dual Stochastic Response Models With Constrained Bayesian Optimization

In engineering system design, minimizing the variations of the quality measurements while guaranteeing their overall quality up to certain levels, namely the robust parameter design (RPD), is crucial. Recent works have dealt with the design of a system whose response-control variables relationship is a deterministic function with a complex shape and function evaluation is expensive. In this work, we propose a Bayesian optimization method for the RPD of stochastic functions. Dual stochastic response models are carefully designed for stochastic functions. The heterogeneous variance of the sample mean is addressed by the predictive mean of the log variance surrogate model in a two-step approach. We establish an acquisition function that favors exploration across the feasible and optimality-improvable regions to effectively and efficiently solve the stochastic constrained optimization problem. The performance of our proposed method is demonstrated by the extensive numerical and case studies. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Many manufacturing processes involve undesirable variations, which create variations in the final products. For example, many emerging manufacturing processes, such as nanomanufacturing, involve complex physical and chemical dynamics and transformation, creating variations in the manufacturing output. In such processes, it is crucial to design the manufacturing processes or products so that they have minimum variations in their quality. Meanwhile, it is also important to maintain the overall quality of the designed processes or products. Furthermore, acquiring data from many advanced manufacturing processes is often very costly, especially in the designing stage. In this work, we propose a data-driven method that automatically finds the best setting of manufacturing processes or products with the minimum variations of quality and a given constraint on the average quality satisfied. Our proposed method is used before conducting every experiment; It analyzes the historical data from previous experiments and provides a setting to be used in the next experiment. Our proposed method efficiently utilizes the historical data, and thus finds the best robust setting by conducting only a small number of experiments.

Data-Driven Stochastic Optimal Control With Safety Constraints Using Linear Transfer Operators

We provide a data-driven framework for optimal control of a continuous-time control-affine stochastic dynamical system. The proposed framework relies on the linear operator theory involving Perron-Frobenius (P-F) and Koopman operators. Our first result involving the P-F operator provides a convex formulation to the optimal control problem in the dual space of densities. This convex formulation of the stochastic optimal control problem leads to an infinite-dimensional convex program. The finite-dimensional approximation of the convex program is obtained using a data-driven approximation of the linear operators. We provide rigorous error bounds and convergence rate for the data-driven approximation of the linear operators and comment on the convergence rate of the optimal control w.r.t. data size. Our second results demonstrate using the Koopman operator, dual to the P-F operator, for the stochastic optimal control design. We show that the Hamilton Jacobi Bellman (HJB) equation can be expressed using the Koopman operator. We provide an iterative procedure along the lines of a popular policy iteration algorithm based on the data-driven approximation of the Koopman operator for solving the HJB equation. The two formulations, namely the convex formulation involving the P-F operator and the Koopman-based formulation using the HJB equation, can be viewed as dual to each other where the duality follows due to the dual nature of the P-F and Koopman operators. Finally, we present examples to demonstrate the efficacy of the developed framework and verify the convergence rates for the operator and optimal control numerically.

Generally Applicable Q-Table Compression Method and Its Application for Constrained Stochastic Graph Traversal Optimization Problems

We analyzed a special class of graph traversal problems, where the distances are stochastic, and the agent is restricted to take a limited range in one go. We showed that both constrained shortest Hamiltonian pathfinding problems and disassembly line balancing problems belong to the class of constrained shortest pathfinding problems, which can be represented as mixed-integer optimization problems. Reinforcement learning (RL) methods have proven their efficiency in multiple complex problems. However, researchers concluded that the learning time increases radically by growing the state- and action spaces. In continuous cases, approximation techniques are used, but these methods have several limitations in mixed-integer searching spaces. We present the Q-table compression method as a multistep method with dimension reduction, state fusion, and space compression techniques that project a mixed-integer optimization problem into a discrete one. The RL agent is then trained using an extended Q-value-based method to deliver a human-interpretable model for optimal action selection. Our approach was tested in selected constrained stochastic graph traversal use cases, and comparative results are shown to the simple grid-based discretization method.

Controlled Reflected McKean–Vlasov SDEs and Neumann Problem for Backward SPDEs

This paper is concerned with the stochastic optimal control problem of a 1-dimensional McKean–Vlasov stochastic differential equation (SDE) with reflection, of which the drift coefficient and diffusion coefficient can be both dependent on the state of the solution process along with its law and control. One backward stochastic partial differential equation (BSPDE) with the Neumann boundary condition can represent the value function of this control problem. Existence and uniqueness of the solution to the above equation are obtained. Finally, the optimal feedback control can be constructed by the BSPDE.

A solution to multi Objective Stochastic Optimal Power Flow problem using mutualism and elite strategy based Pelican Optimization Algorithm

Wind Energy is rapidly growing into one of the most popular energy options for power generation. However, the unpredictability of wind necessitates the employment of a distribution function when describing the wind environment. In the depiction of wind speeds, the two significant parameters of the Weibull distribution, such as the scale parameter and the shape parameter are often utilized. Therefore, it is essential to choose the most appropriate approach for estimating these parameters. Furthermore, the uncertainty in delegated power from variable wind speeds makes system management challenging. In this paper, the Weibull distribution parameters were determined using the Pelican Optimization Algorithm (POA). POA has gained extensive popularity among the research community and is currently used to address a wide range of optimization problems. However, the exploration and exploitation are not properly balanced in this algorithm. To overcome this problem a novel hybrid algorithm namely Mutualism phase based POA (MPOA) is proposed with the help of mutualism phase of Symbiotic Organisms Search (SOS) algorithm. To demonstrate the efficacy and robustness MPOA is applied on multi-objective Optimal Power Flow (MO-OPF) problems. For this study IEEE 30 bus standard system and modified IEEE 30 bus system consisting of two wind farms are taken. The purpose of this work is to examine the impact of changing the Weibull parameters, reverse and penalty cost coefficients on the overall generation cost while considering all the security aspects.

Contextual Pandora’s Box

Pandora’s Box is a fundamental stochastic optimization problem, where the decision-maker must find a good alternative, while minimizing the search cost of exploring the value of each alternative. In the original formulation, it is assumed that accurate distributions are given for the values of all the alternatives, while recent work studies the online variant of Pandora’s Box where the distributions are originally unknown. In this work, we study Pandora’s Box in the online setting, while incorporating context. At each round, we are presented with a number of alternatives each having a context, an exploration cost and an unknown value drawn from an unknown distribution that may change at every round. Our main result is a no-regret algorithm that performs comparably well against the optimal algorithm which knows all prior distributions exactly. Our algorithm works even in the bandit setting where the algorithm never learns the values of the alternatives that were not explored. The key technique that enables our result is a novel modification of the realizability condition in contextual bandits that connects a context to a sufficient statistic of each alternative’s distribution (its reservation value) rather than its mean.

Technical Note—On the Convergence Rate of Stochastic Approximation for Gradient-Based Stochastic Optimization

Finite-Time Performance of Gradient-Based Stochastic Approximation Algorithms Stochastic approximation (SA) algorithms are among the most commonly used methods for addressing stochastic optimization problems. In “Technical Note: On the Convergence Rate of Stochastic Approximation for Gradient-Based Stochastic Optimization,” J. Hu and M. C. Fu provide a new error bound for SA algorithms when applied to a class of problems with convex differentiable structures. The bound allows a detailed characterization of an algorithm’s finite-time performance through directly analyzing the bias and variance of the employed gradient estimator. The main result is then applied to study and compare the efficiency of traditional Kiefer-Wolfowitz algorithms and those based on random direction finite-difference gradient estimators such as simultaneous perturbation SA. The analysis leads to new insights into when it may be advantageous to use such randomized gradient estimates under various problem and algorithm parameter settings.

Performance analysis of machine learning based prediction models in assessing optimal operation of microgrid under uncertainty

Of late, the exponential rise in the global population is driving higher energy demand. However, the rapid depletion of conventional fossil fuels and growing environmental concerns have prompted the evolution of alternative energy sources. To this end, Microgrid (MG) with Renewable Energy Sources (RES) has emerged as popular means of small-scale localized power grid. However, planning of MG operation poses challenges due to the inherent variability and stochasticity in RES power output and energy demand. On account of this, the present study introduces a Stochastic Energy Management Strategy (SEMS) for a grid-connected MG incorporating Micro-Turbine, Fuel-Cell, RES, Battery Energy Storage, and electrical and heat energy demand. The stochasticity of RES is forecasted through a hybrid prediction model (sARIMA-GRU) and the uncertain demand is estimated via 'Monte Carlo Simulation.' The proposed problem is formulated as a dynamic non-linear stochastic optimization problem. It seeks to minimize the expected value of MG operational cost satisfying the practical constraints. Addressing this, a newly developed ‘Artificial Electric Field Algorithm (AEFA)' is utilized. Several case studies are performed to assess MG operation under varied operating conditions. Moreover, the present study analyses the impact of uncertainty on energy contribution from DER, grid dependency, and MG operation cost. Comparative analysis reveals that sARIMA-GRU outperforms other contemporary prediction models. It is noteworthy that the superior prediction accuracy of sARIMA-GRU leads to lower MG operation costs. Moreover, statistical analysis and convergence confirm the proficiency of applied AEFA over state-of-the-art Grey Wolf Optimization and Firefly Algorithm in solving the proposed problem.

Joint resource allocation and beamforming design for RIS-aided symbiotic radio networks: A DRL approach

In this paper, we investigate a Reconfigurable Intelligent Surface (RIS)-assisted secure Symbiosis Radio (SR) network to address the information leakage of the primary transmitter (PTx) to potential eavesdroppers. Specifically, the RIS serves as a secondary transmitter in the SR network to ensure the security of the communication between the PTx and the Primary Receiver (PRx), and simultaneously transmits its information to the PTx concurrently by configuring the phase shifts. Considering the presence of multiple eavesdroppers and uncertain channels in practical scenarios, we jointly optimize the active beamforming of PTx and the phase shifts of RIS to maximize the secrecy energy efficiency of RIS-supported SR networks while satisfying the quality of service requirement and the secure communication rate. To solve this complicated non-convex stochastic optimization problem, we propose a secure beamforming method based on Proximal Policy Optimization (PPO), which is an efficient deep reinforcement learning algorithm, to find the optimal beamforming strategy against eavesdroppers. Simulation results show that the proposed PPO-based method is able to achieve fast convergence and realize the secrecy energy efficiency gain by up to 22% when compared to the considered benchmarks.

Optimal Sizing of Hybrid Renewable Energy System Using Two-Stage Stochastic Programming

Stochastic programming has become increasingly vital in energy applications, especially in the context of the growing need for renewable energy solutions. This paper presents a significant advancement in this field by introducing an efficient and robust algorithm for optimally sizing hybrid renewable energy systems. Utilizing a two-stage stochastic programming approach, the proposed algorithm addresses the challenges posed by the unpredictability of renewable energy sources. The proposed solution leverages the three-block alternating direction method of multipliers (ADMM), a cutting-edge technique that facilitates parallel computation and enhances computational efficiency. The distinctiveness of this method lies in its ability to solve complex stochastic optimization problems without compromising the mathematical integrity of the model. This is achieved by applying first-order optimality conditions, ensuring both robustness and efficacy. To demonstrate the practical applicability and superiority of the algorithm, a case study was conducted in a rural area of South Africa. The proposed algorithm was applied to design an optimal hybrid renewable energy system, and its performance was compared against traditional methods such as progressive hedging and Monte Carlo techniques. Results affirm the superiority of the approach, saving approximately 8.16% capital cost when compared to progressive hedging. In addition, the proposed algorithm outperforms the Monte Carlo method both in terms of CPU time and the number of cost function evaluations.

A two-stage stochastic collaborative planning approach for data centers and distribution network incorporating demand response and multivariate uncertainties

As a critical infrastructure to support the development of digital economy, the scale of data center (DC) power consumption has been growing continuously in recent years. The plug-in of a large number of DCs to the distribution network (DN) presents both opportunities and challenges for the secure, stable, and sustainable operation of the system. In order to address the planning issues arising from the integration of DCs into the DN, this paper proposes a two-stage (planning stage and operation stage) collaborative planning method for DCs and the DN while considering the source-demand uncertainty. Firstly, spatial-temporal demand response characteristic of DCs is explicitly elaborated, and based on the energy consumption model, a two-stage stochastic planning model considering conditional value at risk (CVaR) for DCs and the DN is constructed using probabilistic stochastic simulation, which obeys low-carbon and security, and satisfies the source-network-load constraints of the system. The siting and sizing for DCs and renewable generation, feeder expansion for the DN are determined to minimize the investment cost during the planning stage, while the power output for the devices considering uncertainties from load and renewable energy are optimized to minimize the operation cost during the operation stage. Secondly, the L-shaped decomposition algorithm is proposed to solve the two-stage stochastic optimization problem by reconstructing the problem into a master problem and subproblems, where the dual multiplies are calculated to generate feasibility cuts and optimality cuts to achieve iterative solving. Finally, numerical case studies on a 33-bus distribution system demonstrate that the participation of DCs in demand response can achieve a significant reduction in the investment and operation cost of the DN, and further promote the consumption of distributed renewable energy sources.