Variance Reduction Research Articles

Sensitivity analysis and robust optimization to high-dimensional uncertainties of compressors with active subspace method

The performance of compressors often deviates significantly from the design value due to the substantial geometric deviations arising from manufacturing errors and in-service degradation. Robust design optimization (RDO) plays a crucial role in ensuring the high performance and reliability of compressors. The study proposes an active subspace-based dimensionality-reduction RDO method for high-dimensional uncertainties of compressors. By employing the active subspace (AS) method, a dimensionality-reduction artificial neural network (ANN) is raised to solve the "dimension disaster" problem for high-dimensional uncertainty quantification (UQ) analysis. Additionally, to decrease the computation cost of the original AS method, a data-driven method is utilized to calculate the gradient of the quantity of interest (QoI) within the AS dimensionality-reduction framework. Furthermore, Shapley method is applied to study the sensitivity of geometric variables and their variation direction on the mean and variance of the performance, and the robustness improvement method by only modifying key geometric variables based on Shapley results is discussed. The RDO of high dimensional geometrical uncertainties of a transonic compressor stage is studied by using this method. The results demonstrate a reduction from the original 24-dimensional uncertainties to a 4-dimensional space, achieved through this method. Consequently, the required sample size is reduced by 75 % while maintaining nearly unchanged model accuracy. Furthermore, the findings reveal that after RDO, the mean efficiency increases by 1.04 %, with a 42 % reduction in variance. Compared to RDO, which requires simultaneous optimization of all 24 geometric variables, the robustness improvement based on Shapley only necessitates modifying 4 key geometric variables, resulting in a substantial increase in both the mean and variance of efficiency.

Variance Reduction Techniques for Stochastic Proximal Point Algorithms

AbstractIn the context of finite sums minimization, variance reduction techniques are widely used to improve the performance of state-of-the-art stochastic gradient methods. Their practical impact is clear, as well as their theoretical properties. Stochastic proximal point algorithms have been studied as an alternative to stochastic gradient algorithms since they are more stable with respect to the choice of the step size. However, their variance-reduced versions are not as well studied as the gradient ones. In this work, we propose the first unified study of variance reduction techniques for stochastic proximal point algorithms. We introduce a generic stochastic proximal-based algorithm that can be specified to give the proximal version of SVRG, SAGA, and some of their variants. For this algorithm, in the smooth setting, we provide several convergence rates for the iterates and the objective function values, which are faster than those of the vanilla stochastic proximal point algorithm. More specifically, for convex functions, we prove a sublinear convergence rate of O(1/k). In addition, under the Polyak-łojasiewicz condition, we obtain linear convergence rates. Finally, our numerical experiments demonstrate the advantages of the proximal variance reduction methods over their gradient counterparts in terms of the stability with respect to the choice of the step size in most cases, especially for difficult problems.

A novel logistic regression-embedded 0–1 mixed-integer probabilistic robust design model to obtain optimum sustainable factor settings

Quality by design (QbD) focuses on satisfying the needs of products or processes when aiming for variance reduction. On the other side, sustainability is associated with carrying on to conduct so over time. The research gap is also addressed in sustainability and QbD in this paper. Next, robust design (RD) is a suitable approach to QbD to reduce the process or product variation regarding quantitative and qualitative design factors. For a number of situations, a binomial random response variable is considered to decrease the variability of the production process. Thus, this paper is four-fold. First of all, a modified distance-based criterion is developed to generate the design matrix when dealing with quantitative and qualitative design factors. Second, a logistic regression technique is presented for modeling responses when the random response variable is binomial. Third, a new logistic regression-based 0–1 mixed-integer probabilistic RD model is proposed to find optimum sustainable design factors. Fourth, an Industry 4.0 associated case study is performed in order to design sustainable process development, and the comparison and verification studies are also presented to show the effectiveness of the proposed probabilistic model. Further, managerial insights are provided based on the sustainable development goals (SDGs).

Numerical methods for distributed stochastic compositional optimization problems with aggregative structure

ABSTRACT The paper studies the distributed stochastic compositional optimization problems over networks, where all the agents' inner-level function is the sum of each agent's private expectation function. Focusing on the aggregative structure of the inner-level function, we employ the hybrid variance reduction method to obtain the information on each agent's private expectation function, and apply the dynamic consensus mechanism to track the information on each agent's inner-level function. Then by combining with the standard distributed stochastic gradient descent method, we propose a distributed aggregative stochastic compositional gradient descent method. When the objective function is smooth, the proposed method achieves the convergence rate O ( K − 1 / 2 ) . We further combine the proposed method with the communication compression and propose the communication compressed variant distributed aggregative stochastic compositional gradient descent method. The compressed variant of the proposed method maintains the convergence rate O ( K − 1 / 2 ) . Simulated experiments on decentralized reinforcement learning verify the effectiveness of the proposed methods.

Research on stochastic error suppression method of FOG based on quasi-proportional information

Abstract In order to improve the precision of fiber optic gyro (FOG), it is necessary to suppress the stochastic error effectively. Therefore, the stochastic error suppression of FOG is studied. Firstly, the data of FOG at room temperature and variable temperature are analyzed. Then, a general form of FOG random error model is derived, and an adaptive filtering method based on quasi-proportional information is proposed to estimate the process noise covariance effectively. At last, the room temperature experiment, the variable temperature experiment and the dynamic experiment are carried out. Compared with the recently proposed adaptive filter algorithm, the variance reduction of the proposed algorithm at the average time of 33 s, 54 s and 55 s is more than 70%, and the suppression effect of the algorithm on the random walk coefficient is improved by more than 2%, showing strong robustness.

Using machine learning for efficient flexible regression adjustment in economic experiments

This study investigates the optimal use of covariates in reducing variance when analyzing experimental data. We show that finding the variance-minimizing strategy for making use of pre-treatment observables is equivalent to estimating the conditional expectation function of the outcome given all available pre-randomization observables. This is a pure prediction problem, which recent advances in machine learning (ML) are well-suited to tackling. Through a number of empirical examples, we show how ML-based regression adjustments can feasibly be implemented in practical settings. We compare our proposed estimator to other standard variance reduction techniques in the literature. Two important advantages of our ML-based regression adjustment estimator are that (i) they improve asymptotic efficiency relative to other alternatives and (i i) they can be implemented automatically, with relatively little tuning from the researcher, which limits the scope for data-snooping.

Conditional Mixture Path Guiding for Differentiable Rendering

The efficiency of inverse optimization in physically based differentiable rendering heavily depends on the variance of Monte Carlo estimation. Despite recent advancements emphasizing the necessity of tailored differential sampling strategies, the general approaches remain unexplored. In this paper, we investigate the interplay between local sampling decisions and the estimation of light path derivatives. Considering that modern differentiable rendering algorithms share the same path for estimating differential radiance and ordinary radiance, we demonstrate that conventional guiding approaches, conditioned solely on the last vertex, cannot attain this density. Instead, a mixture of different sampling distributions is required, where the weights are conditioned on all the previously sampled vertices in the path. To embody our theory, we implement a conditional mixture path guiding that explicitly computes optimal weights on the fly. Furthermore, we show how to perform positivization to eliminate sign variance and extend to scenes with millions of parameters. To the best of our knowledge, this is the first generic framework for applying path guiding to differentiable rendering. Extensive experiments demonstrate that our method achieves nearly one order of magnitude improvements over state-of-the-art methods in terms of variance reduction in gradient estimation and errors of inverse optimization. The implementation of our proposed method is available at https://github.com/mollnn/conditional-mixture.

Genetic constitution and variability in synthetic populations of intermediate wheatgrass, an outcrossing perennial grain crop.

Intermediate wheatgrass (IWG) is a perennial grass that produces nutritious grain while offering substantial ecosystem services. Commercial varieties of this crop are mostly synthetic panmictic populations that are developed by intermating a few selected individuals. As development and generation advancement of these synthetic populations is a multiyear process, earlier synthetic generations are tested by the breeders and subsequent generations are released to the growers. A comparison of generations within IWG synthetic cultivars is currently lacking. In this study, we used simulation models and genomic prediction to analyze population differences and trends of genetic variance in 4 synthetic generations of MN-Clearwater, a commercial cultivar released by the University of Minnesota. Little to no differences were observed among the 4 generations for population genetic, genetic kinship, and genome-wide marker relationships measured via linkage disequilibrium. A reduction in genetic variance was observed when 7 parents were used to generate synthetic populations while using 20 led to the best possible outcome in determining population variance. Genomic prediction of plant height, free threshing ability, seed mass, and grain yield among the 4 synthetic generations showed a few significant differences among the generations, yet the differences in values were negligible. Based on these observations, we make 2 major conclusions: (1) the earlier and latter synthetic generations of IWG are mostly similar to each other with minimal differences and (2) using 20 genotypes to create synthetic populations is recommended to sustain ample genetic variance and trait expression among all synthetic generations.

Gradient-Based Monte Carlo Methods for Relaxation Approximations of Hyperbolic Conservation Laws

Particle methods based on evolving the spatial derivatives of the solution were originally introduced to simulate reaction-diffusion processes, inspired by vortex methods for the Navier–Stokes equations. Such methods, referred to as gradient random walk methods, were extensively studied in the ’90s and have several interesting features, such as being grid-free, automatically adapting to the solution by concentrating elements where the gradient is large, and significantly reducing the variance of the standard random walk approach. In this work, we revive these ideas by showing how to generalize the approach to a larger class of partial differential equations, including hyperbolic systems of conservation laws. To achieve this goal, we first extend the classical Monte Carlo method to relaxation approximation of systems of conservation laws, and subsequently consider a novel particle dynamics based on the spatial derivatives of the solution. The methodology, combined with asymptotic-preserving splitting discretization, yields a way to construct a new class of gradient-based Monte Carlo methods for hyperbolic systems of conservation laws. Several results in one spatial dimension for scalar equations and systems of conservation laws show that the new methods are very promising and yield remarkable improvements compared to standard Monte Carlo approaches, either in terms of variance reduction as well as in describing the shock structure.

Efficient risk-based collection of biospecimens in cohort studies: Designing a prospective study of diagnostic performance for multicancer detection tests.

In cohort studies, it can be infeasible to collect specimens on an entire cohort. For example, to estimate sensitivity of multiple Multi-Cancer Detection (MCD) assays, we desire an extra 80mL of cell-free DNA (cfDNA) blood, but this much extra blood is too expensive for us to collect on everyone. We propose a novel epidemiologic study design that efficiently oversamples those at highest baseline disease risk from whom to collect specimens, to increase the number of future cases with cfDNA blood collection. The variance reduction ratio from our risk-based subsample versus a simple random (sub)sample (SRS) depends primarily on the ratio of risk model sensitivity to the fraction of the cohort selected for specimen collection subject to constraining the risk model specificity. In a simulation where we chose 34% of Prostate, Lung, Colorectal, and Ovarian Screening Trial cohort at highest risk of lung cancer for cfDNA blood collection, we could enrich the number of lung cancers 2.42-fold and the standard deviation of lung-cancer MCD sensitivity was 31-33% reduced versus SRS. Risk-based collection of specimens on a subsample of the cohort could be a feasible and efficient approach to collecting extra specimens for molecular epidemiology.

Debiasing Welch’s method for spectral density estimation

Abstract Welch’s method provides an estimator of the power spectral density that is statistically consistent. This is achieved by averaging over periodograms calculated from overlapping segments of a time series. For a finite-length time series, while the variance of the estimator decreases as the number of segments increases, the magnitude of the estimator’s bias increases: a bias-variance trade-off ensues when setting the segment number. We address this issue by providing a novel method for debiasing Welch’s method that maintains the computational complexity and asymptotic consistency, and leads to improved finite-sample performance. Theoretical results are given for fourth-order stationary processes with finite fourth-order moments and an absolutely convergent fourth-order cumulant function. The significant bias reduction is demonstrated with numerical simulation and an application to real-world data. Our estimator also permits irregular spacing over frequency and we demonstrate how this may be employed for signal compression and further variance reduction. The code accompanying this work is available in R and python.

New developments in the MCUNED-Plus code for radiation transport and coupled transport-activation computational simulations in accelerator-based facilities

The radiation hazard is one of the major aspects of concern in the design of a facility where it exists a risk of exposure to ionizing radiation. The nuclear analysis for radioprotection purposes requires an accurate description of the radiation fields. This is especially true in accelerator-based facilities where several types of particles are producing the radiation fields. Radiation fields exist during the accelerator operation, but delayed radiation may also be significant for a long time after operation ends. The radiation fields of concern present during the operation are due to the secondary neutrons and photons emitted by the interaction of accelerated particles with the accelerator components (beam leakage) or with the target. The radiation present after the operation is the residual photon field produced by activated materials, these materials being activated by both accelerated particles and secondary neutrons. This residual radiation field is relevant in high intensity accelerators like IFMIF-DONES.In order to address these complex radiation transport simulations, the D1SUNED code has been updated and the new code release renamed as MCUNED-Plus. The new developments include improvements in the light-ion transport like the implementation of a variance reduction for the production of secondary particles, and a new kinematics to reproduce the angular distribution of secondary particles emitted after deuteron breakup reaction. The calculation of the residual photon field in accelerator facility has also been improved by allowing to evaluate both light-ions and secondary neutrons induced shutdown dose rate in a single coupled simulation.

Reliability assessment combining importance resampling and the cross entropy method

In the ongoing transition towards a sustainable energy system, the electric power system increases in complexity and must adapt to rapid changes and new uncertainties. Thus, development and application of appropriate probabilistic methods is all the more important. Reliability analysis based on Monte Carlo simulation (MCS) has become increasingly popular, and its strength lies in the ability to account for a large number of random variables, general stochastic processes and assessing the probability distribution of the output. However, power system reliability is governed by relatively rare interruption events which poses a fundamental challenge to MCS. This paper presents a variance reduction technique for Monte Carlo based reliability analysis which combines resampling and the cross entropy (CE) method. The motivation for the work is to mitigate the computational burden related to rare event sampling, while at the same time preserving the flexibility of MCS by introducing few assumptions on the stochastic model. The method is demonstrated on a synthetic test system and gives a speedup of about 10 times compared to a crude simulation.

A naïve Bayes regularized logistic regression estimator for low-dimensional classification

To reduce the estimator's variance and prevent overfitting, regularization techniques have attracted great interest from the statistics and machine learning communities. Most existing regularized methods rely on the sparsity assumption that a model with fewer parameters predicts better than one with many parameters. This assumption works particularly well in high-dimensional problems. However, the sparsity assumption may not be necessary when the number of predictors is relatively small compared to the number of training instances. This paper argues that shrinking the coefficients towards a low-variance data-driven estimate could be a better regularization strategy for such situations. For low-dimensional classification problems, we propose a naïve Bayes regularized logistic regression (NBRLR) that shrinks the logistic regression coefficients toward the naïve Bayes estimate to provide a reduction in variance. Our approach is primarily motivated by the fact that naïve Bayes is functionally equivalent to logistic regression if naïve Bayes' conditional independence assumption holds. Under standard conditions, we prove the consistency of the NBRLR estimator. Extensive simulation and empirical experimental results show that NBRLR is a competitive alternative to various state-of-the-art classifiers, especially on low-dimensional datasets.

The Use of Variance Reduction Techniques for Photon Dose Estimation in the Design of Radiotherapy Facilities at 10 and 15 MeV via Monte Carlo Simulations

In the context of designing radiotherapy facilities, typical dose estimation methods involve analytical approaches, as outlined in International Atomic Energy Agency (IAEA) Safety Reports Series No. 47 (IAEA 47). These methods are known for their ease of use and rapid calculations, but they could lead to either overestimation or underestimation of radiation doses. Hence, the integration of Monte Carlo (MC) methods is considered valuable. In this particular study, a radiotherapy facility was modeled using MCNP version 6.2, and dose calculations were conducted using analytical techniques following both IAEA 47 guidelines and MC simulations. The study focused on monoenergetic photon cone beams with energies of 10 and 15 MeV. Notably, the beam’s orientation prevented primary radiation from reaching the dose location at the entrance of the maze, allowing only scatter radiation to contribute to the tally. Given the challenges associated with obtaining reliable and accurate results through standard MCNP calculations, the investigation focused on the use of weight windows as a variance reduction technique. The findings revealed that the IAEA method tends to provide conservative results only when the same conditions were replicated in the MC simulations. In fact, approximately 50% of the final dose estimated through MC methods accounted for factors that were not considered in the analytical calculations. The primary contributor to scattering (averaging around 30%) was identified as the floor and ceiling. This study underscores the need for caution when relying solely on the analytical approach, as it may not consistently yield conservative outcomes.

Optimal two-phase sampling for comparing correlated areas under the ROC curves of two screening tests in the presence of verification bias

ABSTRACT The accuracy of a screening test is often measured by the area under the receiver characteristic (ROC) curve (AUC) of a screening test. Two-phase designs have been widely used in diagnostic studies for estimating one single AUC and comparing two AUCs where the screening test results are measured for a large sample (Phase one sample) while the disease status is only verified for a subset of Phase one sample (Phase two sample) by a gold standard. In this paper, we consider the optimal two-phase sampling design for comparing the performance of two ordinal screening tests in classifying disease status. Specifically, we derive an analytical variance formula for the AUC difference estimator and use it to find the optimal sampling probabilities that minimize the variance formula for the AUC difference estimator. According to the proposed optimal two-phase design, the strata with the levels of two tests far apart from each other should be over-sampled while the strata with the levels of two tests close to each other should be under-sampled. Simulation results indicate that two-phase sampling under optimal allocation (OA) achieves a substantial amount of variance reduction, compared with two-phase sampling under proportional allocation (PA). Furthermore, in comparison with a one-phase random sampling, two-phase sampling under OA or PA has a clear advantage in reducing the variance of AUC difference estimator when the variances of the two screening test results in the disease population differ greatly from their counterparts in non-disease population.

Level-set topology optimization of heat sinks with phase-change material

Phase-change materials (PCMs) excel in storing significant thermal energy through the latent heat of fusion during phase changes. However, they often suffer from low thermal conductivity, requiring the incorporation of materials with higher thermal conductivity to overcome this limitation. In this study, we employ the level-set method to topologically optimize the distribution of both PCM and high thermal conductivity materials within heat sinks.The heat transfer process is modeled as an unsteady diffusion problem, with PCM integrated using the apparent heat capacity method. The finite element method, implemented using FEniCS, is utilized to compute the temperature distribution at each time step. The optimization problem is solved using ParaLeSTO, a modular topology optimization software written in C++, seamlessly interfacing with FEniCS through its Python interface.To compute boundary point sensitivities for level-set topology optimization, the discrete adjoint method is employed, combining a perturbation scheme with automatic differentiation. The proposed methodology is first applied to a two-dimensional example of temperature oscillation minimization with a heat flux boundary condition, drawn from existing literature. Subsequently, a two-dimensional example with multiple volumetric heat sources is studied, followed by an extension to a three-dimensional design domain.Comparisons with optimized designs without PCM, serving as a reference, highlight the superior thermal performance of heat sink designs with PCM. The results showcase a maximum temperature reduction of up to 42%, a mean temperature reduction of up to 42%, and a temperature variance reduction of up to 94%. This research contributes to heat sink design by offering discrete solutions that significantly improve thermal performance as compared to designs without PCM. Its impact extends to addressing crucial challenges in thermal management across various applications, including automotive cooling systems, aerospace components, and consumer electronics.

Within-well patterns in bigeye tuna catch composition and implications for purse-seine port-sampling and catch estimation for the Eastern Pacific Ocean

In response to concerns about the stock status of bigeye tuna (BET) in the Eastern Pacific Ocean, the Inter-American Tropical Tuna Commission adopted additional management measures for BET in 2021, including an individual-vessel catch threshold system for the purse-seine fishery. Development of an enhanced port-sampling program for estimation of trip-level BET catch was also mandated to support member countries and their purse-seine vessels in their conservation efforts. To this end, a pilot study was conducted in the ports of Manta and Posorja, Ecuador, in 2022. In these ports, well unloading often takes place with small containers, which are used to transport fish from inside the well to the vessel’s wet deck. Using a high-frequency systematic sampling protocol, 63 wells of 37 trips with catch exclusively from floating-object (OBJ) sets were sampled from the start to the end of unloading, at a fixed interval from a random starting point, sampling about 10 % of all the containers of fish unloaded from each well. Notable large-scale pattern in the proportion of BET per container was found for 38 of the 56 wells that had BET catch. The proportion of BET per container, which could be high at the beginning, middle or end of the well unloading, sometimes varied by 0.40 or more. The magnitude of this large-scale within-well pattern had a significant increasing relationship with the number of OBJ sets associated with the well catch, consistent with variability in species composition among OBJ sets being an important factor in determining variability in species composition during catch unloading. Simulation studies demonstrated that such large-scale within-well pattern has potential implications for OBJ-set well-level sampling design, where, as expected from sampling theory, a systematic sampling protocol for containers of fish generally resulted in lower mean squared error (MSE) on the proportion of BET in the well, compared to simple random sampling of containers of fish. The reduction in MSE was largely due to a reduction in variance. Simulations also showed that the within-well variance component of the trip-level estimator for the proportion of BET can be larger than the among-well variance component, unless the within-well sampling coverage is relatively high. To minimize the negative impact of within-well variability on the precision of the estimator, results indicate that for a sampling protocol with one systematic sample per well, the within-well coverage should be at least 2 % of containers of fish unloaded from the well, and preferably more than 3 %. Finally, simulation studies were conducted with observer well plan data from 2013 to 2022 to put the well-level results in a larger context. Results indicated that the level of within-well sampling coverage may affect the variance on fleet-level estimates of the proportion of BET in the catch, with potentially substantially higher variance when the within-well sampling coverage is low. Taken together, these results suggest that both the type of within-well sampling and the level of within-well sampling coverage have important implications for the variance on BET catch estimates, from the well level to the fleet level.

Improving compressed matrix multiplication using control variate method

The seminal work by Pagh [1] proposed a matrix multiplication algorithm for real-valued squared matrices called Compressed Matrix Multiplication (CMM) having a sparse matrix output product. The algorithm is based on a popular sketching technique called Count-Sketch [2] and Fast Fourier Transform (FFT). For input square matrices A and B of order n and the product matrix AB with Frobenius norm ||AB||F, the algorithm offers an unbiased estimate for each entry, i.e., (AB)i,j of the product matrix AB with a variance bounded by ||AB||F2/b, where b is the compressed bucket size. Thus, the variance will eventually become high for a small bucket size. In this work, we address the high variance problem of CMM with the help of a simple and practical technique based on classical variance reduction methods in statistics. Our techniques rely on the Control Variate (CV) method. We suggest rigorous theoretical analysis for variance reduction and complement it via supporting empirical evidence.

Global sensitivity analysis with multifidelity Monte Carlo and polynomial chaos expansion for vascular haemodynamics.

Computational models of the cardiovascular system are increasingly used for the diagnosis, treatment, and prevention of cardiovascular disease. Before being used for translational applications, the predictive abilities of these models need to be thoroughly demonstrated through verification, validation, and uncertainty quantification. When results depend on multiple uncertain inputs, sensitivity analysis is typically the first step required to separate relevant from unimportant inputs, and is key to determine an initial reduction on the problem dimensionality that will significantly affect the cost of all downstream analysis tasks. For computationally expensive models with numerous uncertain inputs, sample-based sensitivity analysis may become impractical due to the substantial number of model evaluations it typically necessitates. To overcome this limitation, we consider recently proposed Multifidelity Monte Carlo estimators for Sobol' sensitivity indices, and demonstrate their applicability to an idealized model of the common carotid artery. Variance reduction is achieved combining a small number of three-dimensional fluid-structure interaction simulations with affordable one- and zero-dimensional reduced-order models. These multifidelity Monte Carlo estimators are compared with traditional Monte Carlo and polynomial chaos expansion estimates. Specifically, we show consistent sensitivity ranks for both bi- (1D/0D) and tri-fidelity (3D/1D/0D) estimators, and superior variance reduction compared to traditional single-fidelity Monte Carlo estimators for the same computational budget. As the computational burden of Monte Carlo estimators for Sobol' indices is significantly affected by the problem dimensionality, polynomial chaos expansion is found to have lower computational cost for idealized models with smooth stochastic response.