Data-driven Stochastic Optimization Research Articles

BM-RCWTSG: An integrated matching framework for electric vehicle ride-hailing services under stochastic guidance

Electric vehicles (EVs) are playing an increasingly vital role in sustainable ride-hailing services, primarily due to their potential to reduce carbon emissions and enhance environmental well-being. A core service of ride-hailing platforms is to match riders with nearby drivers within a short time window. In this mandate, it is crucial to ensure a high matching rate and low rider waiting time since these are key indicators of customer satisfaction and the platform’s operational effectiveness. However, existing matching mechanisms in the literature failed to consider important aspects related to EVs, including the range limit, charging detour time, and charging waiting time. These factors can significantly affect the system matching rate, rider waiting time, and vehicle utilization rate. To address the new challenges of the increased adoption of EVs in ride-hailing services, we propose a data-driven stochastic optimization framework for EV ride-hailing matching. The proposed framework integrates two stochastic optimization models, namely the stochastic guidance (SG) model and the EV matching (EVM) model. The SG model optimizes EV relocation and improves the system matching rate by proactively guiding idle EVs to the designated regions based on probabilistic rider demand forecasting. The EVM model utilizes the relocated EVs and current rider demands to provide the optimal matching solution that minimizes the overall cost associated with rider waiting time, EV charging waiting time, and charging detours. We conduct experimental validation of this approach using two real-world data sets from New York City. Comparative results suggest that by utilizing the proposed framework, the matching rate can be increased up to 22.9%, and the rider and EV waiting times can be reduced up to 14.9% and 73.6%, respectively, compared to the benchmark models.

A data-driven multi-stage stochastic robust optimization model for dynamic optimal power flow problem

The uncertainty caused by the distributed generations(DG) with inconspicuous patterns has been an essential subject in the optimization scheduling for the distribution network. We propose a novel data-driven approach to deal with the dynamic optimal power flow(DOPF) problem which contains uncertain variables with their unknown probability distribution. The data-driven model is made to learn the joint probability distribution of the uncertain variables and use robust optimization(RO) to solve the multi-stage stochastic linear DOPF by averaging the worst case from each uncertainty set. In contrast to the motivation for traditional RO to find solutions that perform well on the worst-case realization, our proposed approach adds robustness to the historical data as a tool to avoid overfitting as the number of data points tends to infinity. The application verification for the AC OPF problem is presented for the IEEE-33 system. The simulation verifies the feasibility and robustness of the proposed approach and its results are compared with those of other data-driven stochastic optimization methods. We prove that the proposed approach can effectively solve the overvoltage problem caused by the high permeability of photovoltaic generation and achieve a better out-of-sample performance guarantee, and also has obvious economic advantages over other data-driven methods.

Efficient Approximation of Jacobian Matrices Involving a Non-Uniform Fast Fourier Transform (NUFFT).

There is growing interest in learning Fourier domain sampling strategies (particularly for magnetic resonance imaging, MRI) using optimization approaches. For non-Cartesian sampling, the system models typically involve non-uniform fast Fourier transform (NUFFT) operations. Commonly used NUFFT algorithms contain frequency domain interpolation, which is not differentiable with respect to the sampling pattern, complicating the use of gradient methods. This paper describes an efficient and accurate approach for computing approximate gradients involving NUFFTs. Multiple numerical experiments validate the improved accuracy and efficiency of the proposed approximation. As an application to computational imaging, the NUFFT Jacobians were used to optimize non-Cartesian MRI sampling trajectories via data-driven stochastic optimization. Specifically, the sampling patterns were learned with respect to various model-based image reconstruction (MBIR) algorithms. The proposed approach enables sampling optimization for image sizes that are infeasible with standard auto-differentiation methods due to memory limits. The synergistic acquisition and reconstruction design leads to remarkably improved image quality. In fact, we show that model-based image reconstruction methods with suitably optimized imaging parameters can perform nearly as well as CNN-based methods.

A study of data-driven distributionally robust optimization with incomplete joint data under finite support

Missing data is a common issue for many practical data-driven stochastic programming problems. The state-of-the-art approaches first estimate the missing data values and then separately solve the corresponding stochastic programming. Accurate estimation of missing values is typically inaccessible as it requires enormous data and sophisticated statistical methods. Therefore, this paper proposes an integrated approach, a distributionally robust optimization (DRO) framework, that simultaneously tackles the missing data problem and data-driven stochastic optimization by hedging against the uncertainties of the missing values. This paper adds to the DRO literature by considering the practical scenario where the data can be incomplete and partially observable; it particularly focuses on data distributions with finite support. We construct several classes of ambiguity sets for our DRO model utilizing the incomplete data sets, maximum likelihood estimation method, and different metrics. We prove the statistical consistency and finite sample guarantees of the corresponding models and provide tractable reformulations of our model for different scenarios. We perform computational studies on the multi-item inventory control problem and portfolio optimization using synthetic and real-world data. We validate that our method outperforms the traditional estimate-then-optimized approaches.

Data-driven stochastic optimization for distributional ambiguity with integrated confidence region

We discuss stochastic optimization problems under distributional ambiguity. The distributional uncertainty is captured by considering an entire family of distributions. Because we assume the existence of data, we can consider confidence regions for the different estimators of the parameters of the distributions. Based on the definition of an appropriate estimator in the interior of the resulting confidence region, we propose a new data-driven stochastic optimization problem. This new approach applies the idea of a-posteriori Bayesian methods to the confidence region. We are able to prove that the expected value, over all observations and all possible distributions, of the optimal objective function of the proposed stochastic optimization problem is bounded by a constant. This constant is small for a sufficiently large i.i.d. sample size and depends on the chosen confidence level and the size of the confidence region. We demonstrate the utility of the new optimization approach on a Newsvendor and a reliability problem.

Data-driven stochastic optimization for power grids scheduling under high wind penetration

To address the environmental concern and improve the economic efficiency, the wind power is rapidly integrated into smart grids. However, the inherent uncertainty of wind energy raises operational challenges. To ensure the cost-efficient, reliable and robust operation, it is critically important to find the optimal decision that can correctly and rigorously hedge against all sources of uncertainty. In this paper, we propose data-driven stochastic unit commitment (SUC) to guide the power grids scheduling. Specifically, given the finite historical data, the posterior predictive distribution is developed to quantify the wind power prediction uncertainty accounting for both inherent stochastic uncertainty of wind power generation and input model estimation error. For complex power grid systems, a finite number of scenarios is used to estimate the expected cost in the planning horizon. To further control the impact of finite sampling error induced by using the sample average approximation (SAA), we propose a parallel computing based optimization solution methodology, which can quickly find the reliable optimal unit commitment decision hedging against various sources of uncertainty. The empirical study over six-bus and 118-bus systems demonstrates that our approach can provide more cost-efficient and robust performance than the existing deterministic and stochastic unit commitment approaches.

Data-driven stochastic service restoration in unbalanced active distribution networks with multi-terminal soft open points

Service restoration is critical in ensuring the reliability of distribution networks after the isolation of faults. Multi-terminal soft open points (MTSOPs) can realize power flow among multiple feeders and provide voltage support for power outage areas. This mechanism can help improve the restoration ability of distribution systems. A distribution network is typically unbalanced when the load, distributed generation, and line parameters are unbalanced. This study proposes accordingly a service restoration model for unbalanced active distribution networks (UADNs) with MTSOP. The power control principles of MTSOP in the fault condition and a service restoration strategy in MTSOP-based UADN are first analyzed. Then, a deterministic service restoration model is established. The maximized weighting restored load and the minimized voltage unbalance are used as the objective functions. The original optimization model is converted into a mixed integer second-order cone programming problem by linearizing the three-phase power flow equation, the big-M method, and the second-order cone relaxation, which can be solved by commercial solvers. A two-stage data-driven stochastic optimization model for service restoration in UADN is established by considering the uncertainty of load demand and the generation of renewable energy. The IEEE 33-node, Taiwan Power Company, and modified IEEE 123-node systems are used to verify the proposed model.

Data-driven stochastic optimization on manifolds for additive manufacturing

Laser powder-bed fusion is an increasingly attractive modality for additive manufacturing. Simulating the precise behavior of the process is infeasible, leading researchers to seek proxy solutions in machine learning. We demonstrate a novel adaptation of a technique called diffusion maps to infer the dependence structure between build parameters and material properties of interest on an approximated Riemannian manifold, when attaining a sufficient number of samples is cost prohibitive. We perform stochastic optimization to learn the efficient frontier of multi-objective combinations as well as to locate the set of parameters (hatch width, laser speed, and power) that putatively minimize the chance of failure, herein defined as lower bounds on the material properties (ultimate strength, yield strength, and elongation). Out-of-sample validation is performed and the results confirm our model’s efficacy in optimizing coupons, and perhaps complex geometries in the future.

Data-driven stochastic optimization approaches to determine decision thresholds for risk estimation models

The increasing availability of data has popularized risk estimation models in many industries, especially healthcare. However, properly utilizing these models for accurate diagnosis decisions remains challenging. Our research aims to determine when a risk estimation model provides sufficient evidence to make a positive or negative diagnosis, or if the model is inconclusive. We formulate the Two-Threshold Problem (TTP) as a stochastic program which maximizes sensitivity and specificity while constraining false-positive and false-negative rates. We characterize the optimal solutions to TTP as either two-threshold or one-threshold and show that its optimal solution can be derived from a related linear program (TTP*). We also derive utility-based and multi-class classification frameworks for which our analytical results apply. We solve TTP* using data-driven methods: quantile estimation (TTP*-Q) and distributionally robust optimization (TTP*-DR). Through simulation, we characterize the feasibility, optimality, and computational burden of TTP*-Q and TTP*-DR and compare TTP*-Q to an optimized single threshold. Finally, we apply TTP* to concussion assessment data and find that it achieves greater accuracy at lower misclassification rates compared with traditional approaches. This data-driven framework can provide valuable decision support to clinicians by identifying “easy” cases which can be diagnosed immediately and “hard” cases which may require further evaluation before diagnosing.

Data-Driven Models for Capacity Allocation of Inpatient Beds in a Chinese Public Hospital.

Hospital beds are a critical but limited resource shared between distinct classes of elective patients. Urgent elective patients are more sensitive to delays and should be treated immediately, whereas regular patients can wait for an extended time. Public hospitals in countries like China need to maximize their revenue and at the same time equitably allocate their limited bed capacity between distinct patient classes. Consequently, hospital bed managers are under great pressure to optimally allocate the available bed capacity to all classes of patients, particularly considering random patient arrivals and the length of patient stay. To address the difficulties, we propose data-driven stochastic optimization models that can directly utilize historical observations and feature data of capacity and demand. First, we propose a single-period model assuming known capacity; since it recovers and improves the current decision-making process, it may be deployed immediately. We develop a nonparametric kernel optimization method and demonstrate that an optimal allocation can be effectively obtained with one year's data. Next, we consider the dynamic transition of system state and extend the study to a multiperiod model that allows random capacity; this further brings in substantial improvement. Sensitivity analysis also offers interesting managerial insights. For example, it is optimal to allocate more beds to urgent patients on Mondays and Thursdays than on other weekdays; this is in sharp contrast to the current myopic practice.

Duality-Free Decomposition Based Data-Driven Stochastic Security-Constrained Unit Commitment

To incorporate the superiority of both stochastic and robust approaches, a data-driven stochastic optimization is employed to solve the security-constrained unit commitment model. This approach makes the most use of the historical data to generate a set of possible probability distributions for wind power outputs and then it optimizes the unit commitment under the worst-case probability distribution. However, this model suffers from huge computational burden, as a large number of scenarios are considered. To tackle this issue, a duality-free decomposition method is proposed in this paper. This approach does not require doing duality, which can save a large set of dual variables and constraints, and therefore reduces the computational burden. In addition, the inner max-min problem has a special mathematical structure, where the scenarios have the similar constraint. Thus, the max-min problem can be decomposed into independent subproblems to be solved in parallel, which further improves the computational efficiency. A numerical study on an IEEE 118-bus system with practical data of a wind power system has demonstrated the effectiveness of the proposal.

A Bayesian Risk Approach to Data-driven Stochastic Optimization: Formulations and Asymptotics

A large class of stochastic programs involve optimizing an expectation taken with respect to an underlying distribution that is unknown in practice. One popular approach to addressing the distributional uncertainty, known as the distributionally robust optimization, is to hedge against the worst case over an uncertainty set of candidate distributions. However, it has been observed that inappropriate construction of the uncertainty set can sometimes result in overconservative solutions. To explore the middle ground between optimistically ignoring the distributional uncertainty and pessimistically fixating on the worst-case scenario, we propose a Bayesian risk optimization (BRO) framework for parametric underlying distributions, which is to optimize a risk functional applied to the posterior distribution of an unknown distribution parameter. Of our particular interest are four risk functionals: mean, mean-variance, value-at-risk, and conditional value-at-risk. To unravel the implication of BRO, we establish...

A Data-Driven Stochastic Optimization Approach to the Seasonal Storage Energy Management

Several studies in the literature have shown the potential energy savings emerging from the cooperative management of the aggregated building energy demands. Sophisticated predictive control schemes have recently been developed that achieve these gains by exploiting the energy generation, conversion, and storage equipment shared by the building community. A common difficulty with all these methods is integrating knowledge about the long term evolution of the disturbances affecting the system dynamics (e.g., ambient temperature and solar radiation). In this context, the seasonal storage capabilities of the system are difficult to be optimally managed. This letter addresses this issue by exploiting available historical data to: (i) construct bounds that confine with high probability the optimal charging trajectory of the seasonal storage and (ii) generate a piece-wise affine approximation of the value function of the energy stored in the seasonal storage at each time step. Using these bounds and value functions, we formulate a multistage stochastic optimization problem to minimize the total energy consumption of the system. In a numerical study based on a realistic system configuration, the proposed method is shown to operate the system close to global optimality.