Stochastic Variance Research Articles

Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme

Stochastic variance reduced gradient (SVRG) methods are important approaches to minimize the average of a large number of cost functions frequently arising in machine learning and many other applications. In this paper, based on SVRG, we propose a SVRG-TR method which employs a trust-region-like scheme for selecting stepsizes. It is proved that the SVRG-TR method is linearly convergent in expectation for smooth strongly convex functions and enjoys a faster convergence rate than SVRG methods. In order to overcome the difficulty of tuning stepsizes by hand, we propose to combine the Barzilai–Borwein (BB) method to automatically compute stepsizes for the SVRG-TR method, named as the SVRG-TR-BB method. By incorporating mini-batching scheme with SVRG-TR and SVRG-TR-BB, respectively, we further propose two extended methods mSVRG-TR and mSVRG-TR-BB. Linear convergence and complexity of mSVRG-TR are given. Numerical experiments on some standard datasets show that SVRG-TR and SVRG-TR-BB are generally better than or comparable to SVRG with best-tuned stepsizes and some modern stochastic gradient methods, while mSVRG-TR and mSVRG-TR-BB are very competitive with mini-batch variants of recent successful stochastic gradient methods.

A Stochastic Time-Series Model for Solar Irradiation

We propose a novel stochastic time series model able to explain the stylized features of daily irradation level data in 5 cities in Germany. The model is suitable for applications to risk management of photovoltaic power production in renewable energy markets. The suggested dynamics is a low order autoregressive time series with seasonal level given by an atmospheric clear-sky model. Moreover, we detect a skewness property in the residuals which we explain by a winter-summer regime switch. The stochastic variance is modelled by a seasonally varying GARCH-dynamics. The winter and summer standardized residuals are proposed to be a Gaussian mixture model to capture the bimodal distributions. We estimate the model on the observed data, and perform a validation study. An application to energy markets studying the production at risk for a PV-producer is presented.

Zeroth-Order Diffusion Adaptive Filter Over Networks

Most existing diffusion-based estimation algorithms require the explicit expression of the cost function in order to evaluate the stochastic gradient. In this work, we first discuss the zeroth-order (ZO) gradient for diffusion strategies, and present the ZO-diffusion algorithm that is suitable for applications where the explicit expression of the cost function is unavailable. In addition, to improve the convergence rate of the ZO-diffusion algorithm, we introduce a time-averaging stochastic variance reduced gradient (TA-SVRG) strategy, which is a variant of SVRG algorithm and designed to address online learning problems, and propose a ZO-TA-SVRG diffusion algorithm. Then, we analyze the mean and mean-square stability of the proposed ZO-TA-SVRG diffusion algorithm. Finally, simulation results are provided that demonstrate the performance and effectiveness of the proposed algorithms.

Efficient Asynchronous Semi-Stochastic Block Coordinate Descent Methods for Large-Scale SVD

Eigenvector computation such as Singular Value Decomposition (SVD) is one of the most fundamental problems in machine learning, optimization and numerical linear algebra. In recent years, many stochastic variance reduction algorithms and randomized coordinate descent algorithms have been developed to efficiently solve the leading eigenvalue problem. By taking full advantage of both variance reduction and randomized coordinate descent techniques, this paper proposes a novel Semi-stochastic Block Coordinate Descent algorithm (SBCD-SVD), which is more suitable than existing algorithms for large-scale leading eigenvalue problems of SVD, and can obtain linear convergence. Unlike existing stochastic variance reduction and randomized coordinate descent methods, our algorithm inherits their advantages. Moreover, we propose a new Asynchronous parallel Semi-stochastic Block Coordinate Descent algorithm (ASBCD-SVD) and one new Asynchronous parallel Sparse approximated Variance Reduction algorithm (ASVR-SVD) for large-scale dense and sparse datasets, respectively. Finally, we prove that both dense and sparse asynchronous parallel variants can converge linearly. Extensive experimental results show that our algorithms attain high parallel speedup and achieve almost the same performance with significantly shorter time, and thus they can be widely used in various practice applications.

Variance Reduced Methods for Non-Convex Composition Optimization.

This paper explores the non-convex composition optimization consisting of inner and outer finite-sum functions with a large number of component functions. This problem arises in important applications such as nonlinear embedding and reinforcement learning. Although existing approaches such as stochastic gradient descent (SGD) and stochastic variance reduced gradient (SVRG) descent can be applied to solve this problem, their query complexities tend to be high, especially when the number of inner component functions is large. Therefore, to significantly improve the query complexity of current approaches, we have devised the stochastic composition via variance reduction (SCVR). What's more, we analyze the query complexity under different numbers of inner function and outer function. Based on different kinds of estimation of inner component function, we also present the SCVRII algorithm, though the order of query complexities are the same with SCVR. Additionally, we propose an extension to handle the mini-batch cases, which improve the query complexity under the optimal mini-batch size. The experimental results validate our proposed algorithms and theoretical analyses.

A Stochastic Variance Reduced Primal Dual Fixed Point Method for Linearly Constrained Separable Optimization

A Stochastic Variance Reduced Primal Dual Fixed Point Method for Linearly Constrained Separable Optimization

Propagating residual biases in masked cosmic shear power spectra

In this paper we derive a full expression for the propagation of weak lensing shape measurement biases into cosmic shear power spectra including the effect of missing data. We show using simulations that terms higher than first order in bias parameters can be ignored and the impact of biases can be captured by terms dependent only on the mean of the multiplicative bias field. We identify that the B-mode power contains information on the multiplicative bias. We find that without priors on the residual multiplicative bias $\delta m$ and stochastic ellipticity variance $\sigma_e$ that constraints on the amplitude of the cosmic shear power spectrum are completely degenerate, and that when applying priors the constrained amplitude $A$ is slightly biased low via a classic marginalisation paradox. Using all-sky Gaussian random field simulations we find that the combination of $(1+2\delta m)A$ is unbiased for a joint EE and BB power spectrum likelihood if the error and mean (precision and accuracy) of the stochastic ellipticity variance is known to better than $\sigma(\sigma_e)\leq 0.05$ and $\Delta\sigma_e\leq 0.01$, or the multiplicative bias is known to better than $\sigma(m)\leq 0.07$ and $\Delta m\leq 0.01$.

Large-Scale Phase Retrieval via Stochastic Reweighted Amplitude Flow

Phase retrieval, recovering a signal from phaseless measurements, is generally considered to be an NP-hard problem. This paper adopts an amplitude-based nonconvex optimization cost function to develop a new stochastic gradient algorithm, named stochastic reweighted phase retrieval (SRPR). SRPR is a stochastic gradient iteration algorithm, which runs in two stages: First, we use a truncated sample stochastic variance reduction algorithm to initialize the objective function. The second stage is the gradient refinement stage, which uses continuous updating of the amplitude-based stochastic weighted gradient algorithm to improve the initial estimate. Because of the stochastic method, each iteration of the two stages of SRPR involves only one equation. Therefore, SRPR is simple, scalable, and fast. Compared with the state-of-the-art phase retrieval algorithm, simulation results show that SRPR has a faster convergence speed and fewer magnitude-only measurements required to reconstruct the signal, under the real- or complex- cases.

Variance reduced optimization with implicit gradient transport

The question of how to readily reduce variance in stochastic optimization methods is challenging. The most widely used approach of reducing variance in stochastic optimization is to employ past gradients. Although utilizing second order information in stochastic optimization has recently been proved to be an effective way to track gradients, research on taking second order information to track gradients is quite limited due to a high computational burden. In this paper, we introduce the modified form of the implicit gradient transport (MIGT) technique into the classical stochastic variance reduced methods, i.e., SVRG, which leads to a new method called SVRG–MIGT. The SVRG–MIGT method enjoys second order information to provide faster variance reduction, but without computing the Hessian explicitly. We analyze the theoretical convergence properties of the SVRG–MIGT method for strongly convex objective functions and prove that it is comparable to state-of-the-art methods. To further confirm the effectiveness of MIGT, we propose Acc-Prox-SVRG–MIGT, a novel variant of the accelerated mini-batch Prox-SVRG (Acc-Prox-SVRG) method, by introducing it into Acc-Prox-SVRG. In addition, to greatly reduce the computational complexity, we propose L-SVRG–MIGT, by introducing MIGT to loopless SVRG (L-SVRG), which uses the full gradient with a small probability. Moreover, we establish a convergence analysis for L-SVRG–MIGT. Experimental results on a wide range of data sets demonstrate the efficacy and robustness of our proposed methods.

Who Wants to Be a Volatility Timer?

The exact conditions under which volatility timing strategies yield value are documented. These conditions include: the ability to correctly forecast next period stochastic variance, and violation of Merton's ICAPM. Using reasonable parameter values plugged into the derived formulae, the results show that extreme leverage is often required for success. A method of tempering leverage is proposed, which is somewhat able to loosen the requirement of high leverage while still maintaining a good performance level.

Efficient simulation of generalized SABR and stochastic local volatility models based on Markov chain approximations

We propose a novel Monte Carlo simulation method for two-dimensional stochastic differential equation (SDE) systems based on approximation through continuous-time Markov chains (CTMCs). Specifically, we propose an efficient simulation framework for asset prices under general stochastic local volatility (SLV) models arising in finance, which includes the Heston and the stochastic alpha beta rho (SABR) models as special cases. Our simulation algorithm is constructed based on approximating the latent stochastic variance process by a CTMC. Compared with time-discretization schemes, our method exhibits several advantages, including flexible boundary condition treatment, weak continuity conditions imposed on coefficients, and a second order convergence rate in the spatial grids of the approximating CTMC under suitable regularity conditions. Replacing the stochastic variance process with a discrete-state approximation greatly simplifies the direct sampling of the integrated variance, thus enabling a highly efficient simulation scheme. Extensive numerical examples illustrate the accuracy and efficiency of our estimator, which outperforms both biased and unbiased simulation estimators in the literature in terms of root mean squared error (RMSE) and computational time. This paper is focused primarily on the simulation of SDEs which arise in finance, but this new simulation approach has potential for applications in other contextual areas in operations research, such as queuing theory.

Efficient Simulation of Generalized SABR and Stochastic Local Volatility Models based on Markov Chain Approximations

We propose a novel Monte Carlo simulation method for two-dimensional stochastic differential equation (SDE) systems based on approximation through continuous-time Markov chains (CTMCs). Specifically, we propose an efficient simulation framework for asset prices under general stochastic local volatility (SLV) models arising in finance, which includes the Heston and the stochastic alpha beta rho (SABR) models as special cases. Our simulation algorithm is constructed based on approximating the latent stochastic variance process by a CTMC. Compared with time-discretization schemes, our method exhibits several advantages, including flexible boundary condition treatment, weak continuity conditions imposed on coefficients, and a second order convergence rate in the spatial grids of the approximating CTMC under suitable regularity conditions. Replacing the stochastic variance process with a discrete-state approximation greatly simplifies the direct sampling of the integrated variance, thus enabling a highly efficient simulation scheme. Extensive numerical examples illustrate the accuracy and efficiency of our estimator, which outperforms both biased and unbiased simulation estimators in the literature in terms of root mean squared error (RMSE) and computational time. This paper is focused primarily on the simulation of SDEs which arise in finance, but this new simulation approach has potential for applications in other contextual areas in operations research, such as queuing theory.

Second order of stochastic dominance efficiency vs mean variance efficiency

In this paper, we compare two of the main paradigms of portfolio theory: mean variance analysis and expected utility. In particular, we show empirically that mean variance efficient portfolios are typically sub-optimal for non satiable and risk averse investors. We illustrate that the second order stochastic dominance (SSD) efficient set is the solution of a multi-objective optimization problem. We further show that the market portfolio is not necessarily a solution to this optimization problem. We also conduct an empirical analysis, examining the ex ante and ex post performance of SSD and mean variance efficient portfolios, using a bootstrap approach. In an ex ante analysis, we compare empirical moments, the level of diversification and set distances of mean variance and SSD efficient sets. We also show that the global minimum variance (GMV) portfolio and the part of the mean variance efficient frontier (MVEF) composed of highly diversified portfolios is second order stochastically dominated. This result also provides a possible alternative explanation for the diversification puzzle. Conducting an ex post analysis, we construct second order stochastic dominating strategies that outperform the GMV portfolio in terms of wealth and various other performance measures, producing a positive ex post opportunity cost.

Stochastic Recursive Gradient Support Pursuit and Its Sparse Representation Applications

In recent years, a series of matching pursuit and hard thresholding algorithms have been proposed to solve the sparse representation problem with -norm constraint. In addition, some stochastic hard thresholding methods were also proposed, such as stochastic gradient hard thresholding (SG-HT) and stochastic variance reduced gradient hard thresholding (SVRGHT). However, each iteration of all the algorithms requires one hard thresholding operation, which leads to a high per-iteration complexity and slow convergence, especially for high-dimensional problems. To address this issue, we propose a new stochastic recursive gradient support pursuit (SRGSP) algorithm, in which only one hard thresholding operation is required in each outer-iteration. Thus, SRGSP has a significantly lower computational complexity than existing methods such as SG-HT and SVRGHT. Moreover, we also provide the convergence analysis of SRGSP, which shows that SRGSP attains a linear convergence rate. Our experimental results on large-scale synthetic and real-world datasets verify that SRGSP outperforms state-of-the-art related methods for tackling various sparse representation problems. Moreover, we conduct many experiments on two real-world sparse representation applications such as image denoising and face recognition, and all the results also validate that our SRGSP algorithm obtains much better performance than other sparse representation learning optimization methods in terms of PSNR and recognition rates.

Efficient Simulation of Stochastic Differential Equations Based on Markov Chain Approximations With Applications

We propose a novel Monte Carlo simulation method for two-dimensional stochastic differential equation (SDE) systems based on approximation through continuous-time Markov chains (CTMCs). Specifically, we propose an efficient simulation framework for asset prices under general stochastic local volatility (SLV) models arising in finance, which includes the Heston and the stochastic alpha beta rho (SABR) models as special cases. Our simulation algorithm is constructed based on approximating the latent stochastic variance process by a CTMC. Compared with time-discretization schemes, our method exhibits several advantages, including flexible boundary condition treatment, weak continuity conditions imposed on coefficients, and a second order convergence rate in the spatial grids of the approximating CTMC under suitable regularity conditions. Replacing the stochastic variance process with a discrete-state approximation greatly simplifies the direct sampling of the integrated variance, thus enabling a highly efficient simulation scheme. Extensive numerical examples illustrate the accuracy and efficiency of our estimator, which outperforms \textit{both biased and unbiased} simulation estimators in the literature in terms of root mean squared error (RMSE) and computational time. This paper is focused primarily on the simulation of SDEs which arise in finance, but this new simulation approach has potential for applications in other contextual areas in operations research, such as queuing theory. Note: this is an earlier version of the work Efficient Simulation of Generalized SABR and Stochastic Local Volatility Models based on Markov Chain Approximations.

A Markov chain approximation scheme for option pricing under skew diffusions

In this paper, we propose a general valuation framework for option pricing problems related to skew diffusions based on a continuous-time Markov chain approximation to the underlying stochastic process. We obtain an explicit closed-form approximation of the transition density of a general skew diffusion process, which facilitates the unified valuation of various financial contracts written on assets with natural boundary behavior, e.g. in the foreign exchange market with target zones, and equity markets with psychological barriers. Applications include valuation of European call and put options, barrier and Bermudan options, and zero-coupon bonds. Motivated by the presence of psychological barriers in the market volatility, we also propose a novel ‘skew stochastic volatility’ model, in which the latent stochastic variance follows a skew diffusion process. Numerical results demonstrate that our approach is accurate and efficient, and recovers various benchmark results in the literature in a unified fashion.

Does the variance of surface EMG signals during isometric contractions follow an inverse gamma distribution?

In this paper, the validity of the stochastic model-based variance distribution of surface electromyogram (EMG) signals during isometric contraction is investigated. In the model, the EMG variance is considered as a random variable following an inverse gamma distribution, thereby allowing the representation of variations in the variance. This inverse gamma-based model for the EMG variance is experimentally validated through comparison with the empirical distribution of variances. The difference between the model distribution and the empirical distribution is quantified using the Kullback- Leibler divergence. Additionally, regression analysis is conducted between the model parameters and the statistics calculated from the empirical distribution of EMG variances. Experimental results showed that the inverse gamma-based model is potentially suitable and that its parameters can be used to evaluate the stochastic properties of the EMG variance.

Differentially Private Distributed Learning

The rich data used to train learning models increasingly tend to be distributed and private. It is important to efficiently perform learning tasks without compromising individual users’ privacy even considering untrusted learning applications and, furthermore, understand how privacy-preservation mechanisms impact the learning process. To address the problem, we design a differentially private distributed algorithm based on the stochastic variance reduced gradient (SVRG) algorithm, which prevents the learning server from accessing and inferring private training data with a theoretical guarantee. We quantify the impact of the adopted privacy-preservation measure on the learning process in terms of convergence rate, by which it indicates noises added at each gradient update results in a bounded deviation from the optimum. To further evaluate the impact on the trained models, we compare the proposed algorithm with SVRG and stochastic gradient descent using logistic regression and neural nets. The experimental results on benchmark data sets show that the proposed algorithm has minor impact on the accuracy of trained models under a moderate amount of privacy budget.

Optimal decision policy for real options under general Markovian dynamics

The Least-Squares Monte Carlo Method (LSM) has become the standard tool to solve real options modeled as an optimal switching problem. The method has been shown to deliver accurate valuation results under complex and high dimensional stochastic processes; however, the accuracy of the underlying decision policy is not guaranteed. For instance, an inappropriate choice of regression functions can lead to noisy estimates of the optimal switching boundaries or even continuation/switching regions that are not clearly separated. As an alternative to estimate these boundaries, we formulate a simulation-based method that starts from an initial guess of them and then iterates until reaching optimality. The algorithm is applied to a classical mine under a wide variety of underlying dynamics for the commodity price process. The method is first validated under a one-dimensional geometric Brownian motion and then extended to general Markovian processes. We consider two general specifications: a two-factor model with stochastic variance and a rich jump structure, and a four-factor model with stochastic cost-of-carry and stochastic volatility. The method is shown to be robust, stable, and easy-to-implement, converging to a more profitable strategy than the one obtained with LSM.

Organohalide-Respiring Bacteria in Polluted Urban Rivers Employ Novel Bifunctional Reductive Dehalogenases to Dechlorinate Polychlorinated Biphenyls and Tetrachloroethene.

Polluted urban river sediments could be a sink of persistent and toxic polychlorinated biphenyls (PCBs) in urban areas and provide desired growth niches for organohalide-respiring bacteria (OHRB). In this study, microcosms were set up with surface sediments of nationwide polluted urban rivers in China, of which 164 cultures could dechlorinate tetrachloroethene (PCE) to dichloroethenes (DCEs) and to vinyl chloride and/or ethene. Further in vivo tests showed extensive PCB dechlorination with different pathways in 135 PCE pregrown cultures. Taking reductive dechlorination of PCB180 (2345-245-CB) as an example, 121 and 14 cultures preferentially removed flanked para- and meta-chlorines, respectively. Strikingly, all in vitro assays with the 135 PCE pregrown cultures showed identical PCB dechlorination pathways with their living cultures, implying the involvement of bifunctional reductive dehalogenases (RDases) to dechlorinate both PCBs and PCE. Further 16S rRNA and RDase gene-based analyses, together with enantioselective dechlorination of chiral PCBs, suggested that Dehalococcoides and Dehalogenimonas in the 135 cultures largely employed distinctively different novel bifunctional RDases to catalyze PCB/PCE dechlorination. Quantitative assessment of the community assembly process with the modified stochasticity ratio (MST) indicated three different stages in enrichment of OHRB. The second stage, as the only one controlled by stochastic processes (MST > 0.5), required extra attention in monitoring community successional patterns to minimize stochastic variance for enriching the PCB/PCE-dechlorinating OHRB.