Stochastic Gradient Estimation Research Articles

The auxiliary model based hierarchical gradient algorithms and convergence analysis using the filtering technique

On the basis of the auxiliary model identification idea, this paper studies the filtering based parameter estimation issues for a class of multivariable control systems with colored noise. An auxiliary model based hierarchical stochastic gradient (AM-HSG) algorithm is given for comparison and a data filtering AM-HSG identification algorithm is derived by using the data filtering technique. Its main key is to decompose a multivariable system into two subsystems and to coordinate the associate items between two subsystem identification algorithms. The convergence analysis indicates that the parameter estimates given by the presented algorithms converge to the true values under proper conditions by using the stochastic process theory. The simulation results show that the proposed hierarchical stochastic gradient estimation algorithms are effective.

Proactive Optimization of Intelligent-Well Production Using Stochastic Gradient-Based Algorithms

Summary The popularity of intelligent wells (I-wells), which provide layer-by-layer monitoring and control capability of production and injection, is growing. However, the number of available techniques for optimal control of I-wells is limited (Sarma et al. 2006; Alghareeb et al. 2009; Almeida et al. 2010; Grebenkin and Davies 2012). Currently, most of the I-wells that are equipped with interval control valves (ICVs) are operated to enhance the current production and to resolve problems associated with breakthrough of the unfavorable phase. This reactive strategy is unlikely to deliver the long-term optimum production. On the other side, the proactive-control strategy of I-wells, with its ambition to provide the optimum control for the entire well's production life, has the potential to maximize the cumulative oil production. This strategy, however, results in a high-dimensional, nonlinear, and constrained optimization problem. This study provides guidelines on selecting a suitable proactive optimization approach, by use of state-of-the-art stochastic gradient-approximation algorithms. A suitable optimization approach increases the practicality of proactive optimization for real field models under uncertain operational and subsurface conditions. We evaluate the simultaneous-perturbation stochastic approximation (SPSA) method (Spall 1992) and the ensemble-based optimization (EnOpt) method (Chen et al. 2009). In addition, we present a new derivation of the EnOpt by use of the concept of directional derivatives. The numerical results show that both SPSA and EnOpt methods can provide a fast solution to a large-scale and multiple I-well proactive optimization problem. A criterion for tuning the algorithms is proposed and the performance of both methods is compared for several test cases. The used methodology for estimating the gradient is shown to affect the application area of each algorithm. SPSA provides a rough estimate of the gradient and performs better in search environments, characterized by several local optima, especially with a large ensemble size. EnOpt was found to provide a smoother estimation of the gradient, resulting in a more-robust algorithm to the choice of the tuning parameters, and a better performance with a small ensemble size. Moreover, the final optimum operation obtained by EnOpt is smoother. Finally, the obtained criteria are used to perform proactive optimization of ICVs in a real field.

Stochastic variational inference for large-scale discrete choice models using adaptive batch sizes

Discrete choice models describe the choices made by decision makers among alternatives and play an important role in transportation planning, marketing research and other applications. The mixed multinomial logit (MMNL) model is a popular discrete choice model that captures heterogeneity in the preferences of decision makers through random coefficients. While Markov chain Monte Carlo methods provide the Bayesian analogue to classical procedures for estimating MMNL models, computations can be prohibitively expensive for large datasets. Approximate inference can be obtained using variational methods at a lower computational cost with competitive accuracy. In this paper, we develop variational methods for estimating MMNL models that allow random coefficients to be correlated in the posterior and can be extended easily to large-scale datasets. We explore three alternatives: (1) Laplace variational inference, (2) nonconjugate variational message passing and (3) stochastic linear regression. Their performances are compared using real and simulated data. To accelerate convergence for large datasets, we develop stochastic variational inference for MMNL models using each of the above alternatives. Stochastic variational inference allows data to be processed in minibatches by optimizing global variational parameters using stochastic gradient approximation. A novel strategy for increasing minibatch sizes adaptively within stochastic variational inference is proposed.

A Stochastic Variational Framework for Fitting and Diagnosing Generalized Linear Mixed Models

In stochastic variational inference, the variational Bayes objective function is optimized using stochastic gradient approximation, where gradients computed on small random subsets of data are used to approximate the true gradient over the whole data set. This enables complex models to be fit to large data sets as data can be processed in mini-batches. In this article, we extend stochastic variational inference for conjugate-exponential models to nonconjugate models and present a stochastic nonconjugate variational message passing algorithm for fitting generalized linear mixed models that is scalable to large data sets. In addition, we show that diagnostics for prior-likelihood conflict, which are useful for Bayesian model criticism, can be obtained from nonconjugate variational message passing automatically, as an alternative to simulation-based Markov chain Monte Carlo methods. Finally, we demonstrate that for moderate-sized data sets, convergence can be accelerated by using the stochastic version of nonconjugate variational message passing in the initial stage of optimization before switching to the standard version.

STOCHASTIC GRADIENT METHODS FOR UNCONSTRAINED OPTIMIZATION

This papers presents an overview of gradient based methods for minimization of noisy functions. It is assumed that the objective functions is either given with error terms of stochastic nature or given as the mathematical expectation. Such problems arise in the context of simulation based optimization. The focus of this presentation is on the gradient based Stochastic Approximation and Sample Average Approximation methods. The concept of stochastic gradient approximation of the true gradient can be successfully extended to deterministic problems. Methods of this kind are presented for the data fitting and machine learning problems.

Integrated Traffic and Emission Simulation: a Model Calibration Approach Using Aggregate Information

Environmental impacts of road traffic have attracted increasing attention in project-level traffic planning and management. The conventional approach considers emission impact analysis as a separate process in addition to traffic modeling. This paper first introduces our research effort to integrate traffic, emission, and dispersion processes into a common distributed computational framework, which makes it efficient to quantify and analyze correlations among dynamic traffic conditions, emission impacts, and air quality consequences. A model calibration approach is particularly proposed when on-road or in-lab instantaneous emission measurements are not directly available. Microscopic traffic simulation is applied to generate dynamic vehicle states at the second-by-second level. Using aggregate emission estimation as standard reference, a numerical optimization scheme on the basis of a stochastic gradient approximation algorithm is applied to find optimal parameters for the dynamic emission model. The calibrated model has been validated on several road networks with traffic states generated by the same simulation model. The results show that with proper formulation of the optimization objective function, the estimated dynamic emission model can capture the trends of aggregate emission patterns of traffic fleets and predict local emission and air quality at higher temporal and spatial resolutions.

Enhancing Stochastic Kriging Metamodels with Gradient Estimators

Stochastic kriging is a new metamodeling technique for effectively representing the mean response surface implied by a stochastic simulation; it takes into account both stochastic simulation noise and uncertainty about the underlying response surface of interest. We show theoretically, through some simplified models, that incorporating gradient estimators into stochastic kriging tends to significantly improve surface prediction. To address the issue of which type of gradient estimator to use, when there is a choice, we briefly review stochastic gradient estimation techniques; we then focus on the properties of infinitesimal perturbation analysis and likelihood ratio/score function gradient estimators and make recommendations. To conclude, we use simulation experiments with no simplifying assumptions to demonstrate that the use of stochastic kriging with gradient estimators provides more reliable prediction results than stochastic kriging alone.

Maximum likelihood stochastic gradient estimation for Hammerstein systems with colored noise based on the key term separation technique

This paper considers the identification problems of Hammerstein finite impulse response moving average (FIR-MA) systems using the maximum likelihood principle and stochastic gradient method based on the key term separation technique. In order to improve the convergence rate, a maximum likelihood multi-innovation stochastic gradient algorithm is presented. The simulation results show that the proposed algorithms can effectively estimate the parameters of the Hammerstein FIR-MA systems.

Blind Adaptive Constrained Constant-Modulus Reduced-Rank Interference Suppression Algorithms Based on Interpolation and Switched Decimation

This work proposes a blind adaptive reduced-rank scheme and constrained constant-modulus (CCM) adaptive algorithms for interference suppression in wireless communications systems. The proposed scheme and algorithms are based on a two-stage processing framework that consists of a transformation matrix that performs dimensionality reduction followed by a reduced-rank estimator. The complex structure of the transformation matrix of existing methods motivates the development of a blind adaptive reduced-rank constrained (BARC) scheme along with a low-complexity reduced-rank decomposition. The proposed BARC scheme and a reduced-rank decomposition based on the concept of joint interpolation, switched decimation and reduced-rank estimation subject to a set of constraints are then detailed. The proposed set of constraints ensures that the multipath components of the channel are combined prior to dimensionality reduction. We develop low-complexity joint interpolation and decimation techniques, stochastic gradient, and recursive least squares reduced-rank estimation algorithms. A model-order selection algorithm for adjusting the length of the estimators is devised along with techniques for determining the required number of switching branches to attain a predefined performance. An analysis of the convergence properties and issues of the proposed optimization and algorithms is carried out, and the key features of the optimization problem are discussed. We consider the application of the proposed algorithms to interference suppression in DS-CDMA systems. The results show that the proposed algorithms outperform the best known reduced-rank schemes, while requiring lower complexity.

Efficient price sensitivity estimation of financial derivatives by weak derivatives

The stochastic gradient estimation method of weak derivatives (WD) is presented with the aim of constructing efficient algorithms for the estimation of the “Greeks” of financial derivatives. The key idea is to replace the derivative of the probability measure of the underlying model by its WD. The WD method has the same advantageous property of the well-known score function method that the form of the Greek estimator does not depend on the details of the payoff function but only on the probability density of the underlying model. Simulation studies indicate that the WD estimator has significantly lower variance than the score function and finite difference estimator, however, the associated computational burden in certain cases may not be negligible.

Quaternion-Valued Stochastic Gradient-Based Adaptive IIR Filtering

A learning algorithm for the training of quaternion valued adaptive infinite impulse (IIR) filters is introduced. This is achieved by taking into account specific properties of stochastic gradient approximation in the quaternion domain and the recursive nature of the sensitivities within the IIR filter updates, to give the quaternion-valued stochastic gradient algorithm for adaptive IIR filtering (QSG-IIR). Further, to reduce computational complexity, a variant of the QSG-IIR is introduced, which for small stepsizes makes better use of the available information. Stability analysis and simulations on both synthetic and real world 4D data support the approach.

Performance analysis of the auxiliary models based multi-innovation stochastic gradient estimation algorithm for output error systems

This paper combines the multi-innovation identification theory and the auxiliary model identification idea and presents an auxiliary model based multi-innovation stochastic gradient algorithm by expanding the scalar innovation to an innovation vector and introducing the innovation length. Convergence analysis in the stochastic framework indicates that the parameter estimates given by the proposed algorithm can fast converge to their true values. Finally, we illustrate and test the proposed algorithm with an example.

A ROBUST ENSEMBLE LEARNING USING ZERO-ONE LOSS FUNCTION

Classifier is used for pattern recognition in various fields including data mining. Boosting is an ensemble learning method to boost (enhance) an accuracy of single classifier. We propose a new, robust boosting method by using a zero-one step function as a loss function. In deriving the method, the MarginBoost technique is blended with the stochastic gradient approximation algorithm, called Stochastic Noise Reaction (SNR). Based on intensive numerical experiments, we show that the proposed method is actually better than AdaBoost on test error rates in the case of noisy, mislabeled situation.

Direct reinforcement learning, spike time dependent plasticity and the BCM rule

Learning agents, whether natural or artificial, must update their internal parameters in order to improve their behavior over time. In reinforcement learning, this plasticity is influenced by an environmental signal, termed a reward, which directs the changes in appropriate directions. We model a network of spiking neurons as a Partially Observed Markov Decision Process (POMDP) and apply a recently introduced policy learning algorithm from Machine Learning to the network [1]. Based on computing a stochastic gradient approximation of the average reward, we derive a plasticity rule falling in the class of Spike Time Dependent Plasticity (STDP) rules, which ensures convergence to a local maximum of the average reward. The approach is applicable to a broad class of neuronal models, including the Hodgkin-Huxley model. The obtained update rule is based on the correlation between the reward signal and local data available at the synaptic site. This data depends on local activity (e.g., pre and post synaptic spikes) and requires mechanisms that are available at the cellular level. Simulations on several toy problems demonstrate the utility of the approach. Like most stochastic gradient based methods, the convergence rate is slow, even though the percentage of convergence to global maxima is high. Additionally, through statistical analysis we show that the synaptic plasticity rule established is closely related to the widely used BCM rule [2], for which good biological evidence exists. The relation to the BCM rule captures the nature of the relation between pre and post synaptic spiking rates, and in particular the self-regularizing nature of the BCM rule. Compared to previous work in this field, our model is more realistic than the one used in [3], and the derivation of the update rule applies to a broad class of voltage based neuronal models, eliminating some of the additional statistical assumptions required in [4]. Finally, the connection between Reinforcement Learning and the BCM rule is, to the best of our knowledge, new.

Stochastic gradient algorithms for design of minimum error-rate linear dispersion codes in MIMO wireless systems

Linear dispersion (LD) codes are a good candidate for high-data-rate multiple-input multiple-ouput (MIMO) signaling. Traditionally LD codes were designed by maximizing the average mutual information, which cannot guarantee good error performance. This paper presents a new design scheme for LD codes that directly minimizes the block error rate (BLER) in MIMO channels with arbitrary fading statistics and various detection algorithms. For MIMO systems employing LD codes, the error rate does not admit an explicit form. Therefore, we cannot use deterministic optimization methods to design the minimum-error-rate LD codes. In this paper, we propose a simulation-based optimization methodology for the design of LD codes through stochastic approximation and simulation-based gradient estimation. The gradient estimation is done using the score function method originally developed in the discrete-event-system community. The proposed method can be applied to design the minimum-error-rate LD codes for a variety of detector structures including the maximum-likelihood (ML) detector and several suboptimal detectors. It can also design optimal codes under arbitrary fading channel statistics; in particular, it can take into account the knowledge of spatial fading correlation at the transmitter and receiver ends. Simulation results show that codes generated by the proposed new design paradigm generally outperform the codes designed based on algebraic number theory.

Model-Based Search for Combinatorial Optimization: A Critical Survey

In this paper we introduce model-based search as a unifying framework accommodating some recently proposed metaheuristics for combinatorial optimization such as ant colony optimization, stochastic gradient ascent, cross-entropy and estimation of distribution methods. We discuss similarities as well as distinctive features of each method and we propose some extensions.

Constrained SPSA controller for operations processes

Continuous quality improvement calls for employing methodologies that assist in continual reduction of variations in process performance characteristics around their target values. The study considers a case in which some of the operations process parameters/inputs are required to take values in pre-specified ranges. To improve the process performance while accounting for these requirements, the study employs a neural network-based model-free controller along with the penalty function. Simultaneous perturbation stochastic gradient approximation method is used to iteratively estimate the weights of neural network and as a result to estimate the control values. Furthermore, the study uses a special cause control chart to monitor the performance of the controller in reducing the process variations and to signal the change in the process dynamics. Simulation findings indicate that the neural network model-free provides control values that result in fewer nonconforming outputs than when the requirements are not incorporated in optimization process.

A manufacturing system with general stationary failure process: stability and IPA of hedging control policies

This paper concerns a new methodology for the adaptive optimization of piecewise deterministic non-Markovian systems via a simple example of interest in manufacturing. This methodology takes into account the fact that piecewise deterministic systems are rarely Markovian and that classical control theory based on dynamic programming of Markovian systems cannot provide quantitative answers in most realistic situations. The example considered is that of a single-machine/single-part production system, from the point of view of the control theory by Kimemia and Gershwin (1983). We obtain stochastic gradient estimates via the perturbation analysis of Ho and Cao (1991) by the regenerative method of Konstantopoulos and Zazanis (1992), in view of stochastic optimization. Its mathematical justification requires a careful study of the regenerative structure of the process. In particular, we give the necessary and sufficient conditions of stability of this system via the method of Loynes (1962).

Stochastic Approximation Algorithm eith Gradient Averaging and On-Line Stepsize Rules

A new practical stochastic approximation algorithm for finding unconstrained minima of smooth functions is described. It uses an auxillary filter which averages stochastic gradient estimates observed, thus producing directions for subsequent iterations. Stepsize coefficients and filter gains are controlled on-line on the basis of information gathered in the course of computations. Convergence of the method with probability one is proved, asymptotic properties are studied and numerical examples described.