Efficient Stochastic Gradient Research Articles

Efficient mini-batch stochastic gradient descent with Centroidal Voronoi Tessellation for PDE-constrained optimization under uncertainty

The study of optimal control problems under uncertainty plays an important role in scientific numerical simulations. This class of optimization problems is frequently utilized in engineering, biology and finance. Although stochastic gradient descent is a well-known stochastic optimization technique for solving the approximate optimal control problem, it often exhibits a slow convergence rate due to the inherent variance in gradient approximation. To address this issue, we propose a more efficient stochastic optimization algorithm based on Centroidal Voronoi Tessellation sampling. This strategy reduces the error in gradient estimation compared to the conventional mini-batch stochastic gradient descent method. Our approach involves dividing the whole snapshot set into several Voronoi cells with low variance and extracting samples with good uniformity from each region to construct an unbiased estimation of the full gradient. Our method can economically and stably approximate numerical optimal control functions even with a fixed step size. Numerical results demonstrate that the proposed method is a reliable algorithm with great potential for applications in the field of optimization.

Highly efficient online stochastic gradient and sliding window stochastic gradient signal modelling methods for multi-frequency signals

Highly efficient online stochastic gradient and sliding window stochastic gradient signal modelling methods for multi-frequency signals

Highly efficient online stochastic gradient and sliding window stochastic gradient signal modelling methods for multi-frequency signals

Highly efficient online stochastic gradient and sliding window stochastic gradient signal modelling methods for multi-frequency signals

Efficient stochastic parallel gradient descent training for on-chip optical processor

Efficient stochastic parallel gradient descent training for on-chip optical processor

Overlap Removal by Stochastic Gradient Descent With(out) Shape Awareness.

In many 2D visualizations, data points are projected without considering their surface area, although they are often represented as shapes in visualization tools. These shapes support the display of information such as labels or encode data with size or color. However, inappropriate shape and size selections can lead to overlaps that obscure information and hinder the visualization's exploration. Overlap Removal (OR) algorithms have been developed as a layout post-processing solution to ensure that the visible graphical elements accurately represent the underlying data. As the original data layout contains vital information about its topology, it is essential for OR algorithms to preserve it as much as possible. This article presents an extension of the previously published FORBID algorithm by introducing a new approach that models OR as a joint stress and scaling optimization problem, utilizing efficient stochastic gradient descent. The goal is to produce an overlap-free layout that proposes a compromise between compactness (to ensure the encoded data is still readable) and preservation of the original layout (to preserve the structures that convey information about the data). Additionally, this article proposes SORDID, a shape-aware adaptation of FORBID that can handle the OR task on data points having any polygonal shape. Our approaches are compared against state-of-the-art algorithms, and several quality metrics demonstrate their effectiveness in removing overlaps while retaining the compactness and structures of the input layouts.

A novel support vector machine with generalized pinball loss for uncertain data classification

In real‐world problems, data suffer from measurement errors, data staleness, and repeated measurements, which make data uncertain. To consider the uncertainty of data, the concept of giving each data feature as a multidimensional Gaussian distribution has been utilized. In 2021, support vector machines with an insensitive zone pinball loss (UPinSVMs) for uncertain data classification was proposed, where is a positive number. The UPinSVMs bring noise insensitivity, stability for resampling, and increased model sparsity. However, the value of is known to be specified. In order to improve the performance, we combine the generalized pinball loss (( ‐Mod‐Pin‐SVM) into the uncertain classification, term as UGPinSVMs, where are two positive numbers. The generalized pinball loss is an optimal insensitive zone pinball loss and is an extension of existing loss functions that also addresses the issues of noise insensitivity and resampling instability. We solve the primal quadratic programming problems by transforming individuals into unconstrained optimization using an efficient stochastic gradient descent algorithm. More specially, we have introduced verified theorems that are related to our approaches and investigated scatter minimization. The results from several benchmark datasets show that our model outperforms the existing classifier in terms of accuracy and statistical analysis. Furthermore, the application of our framework to the crop recommendation dataset is also examined.

Fair and Energy-Efficient Coverage Optimization for UAV Placement Problem in the Cellular Network

Unmanned Aerial Vehicle (UAV) Base Station (BS) placement optimization is an essential operational task to improve the Quality of Service (QoS) in UAV-aided wireless cellular networks. The existing approaches are almost zeroth order methods, and the few first order methods mainly ignore the allocation fairness, computational efficiency, and backhaul constraints. In this paper, we formulate the UAV placement problem as a constrained optimization problem, with the objective of maximizing the fair coverage versus energy consumption while satisfying the backhaul constraints at different time nodes. To guarantee fair QoS allocation, we introduce a novel fairness index to ensure fair communication opportunity and the novel region coverage ratio to avoid excess QoS on covered spots. An accurate and efficient proximal stochastic gradient descent based alternating algorithm that iteratively executes two optimization steps is proposed to optimize the UAV locations, which enables the fast single point-based first order methods to solve the complex problems with constraints. Experiment results manifest that the proposed algorithm performs well both in synthetic data scenario and in real city scenario. Furthermore, the proposed first order algorithm is more efficient than the existing zeroth order algorithm, typically referring to the meta-heuristic method.

A backward SDE method for uncertainty quantification in deep learning

<p style='text-indent:20px;'>We develop a backward stochastic differential equation based probabilistic machine learning method, which formulates a class of stochastic neural networks as a stochastic optimal control problem. An efficient stochastic gradient descent algorithm is introduced with the gradient computed through a backward stochastic differential equation. Convergence analysis for stochastic gradient descent optimization and numerical experiments for applications of stochastic neural networks are carried out to validate our methodology in both theory and performance.</p>

Learning GANs in Simultaneous Game Using Sinkhorn With Positive Features

Entropy regularized optimal transport (EOT) distance and its symmetric normalization, known as the Sinkhorn divergence, offer smooth and continuous metrized weak-convergence distance metrics. They have excellent geometric properties and are useful to compare probability distributions in some generative adversarial network (GAN) models. Computing them using the original Sinkhorn matrix scaling algorithm is still expensive. The running time is quadratic at $\mathcal {O}(n^{2})$ in the size $n$ of the training dataset. This work investigates the problem of accelerating the GAN training when Sinkhorn divergence is used as a minimax objective. Let $\mathcal {G}$ be a Gaussian map from the ground space onto the positive orthant $\mathbb {R}_{+}^{r}$ with $r \ll n $ . To speed up the divergence computation, we propose the use of $c(x,y)= - \varepsilon \log \left \langle{ \mathcal {G}(x),\mathcal {G}(y) }\right \rangle $ as the ground cost. This approximation, known as Sinkhorn with positive features, brings down the running time of the Sinkhorn matrix scaling algorithm to $\mathcal {O}(r \, n)$ , which is linear in $n$ . To solve the minimax optimization in GAN, we put forward a more efficient simultaneous stochastic gradient descent-ascent (SimSGDA) algorithm in place of the standard sequential gradient techniques. Empirical evidence shows that our model, trained using SimSGDA on the DCGAN neural architecture on tiny-coloured Cats and CelebA datasets, converges to stationary points. These are the local Nash equilibrium points. We carried out numerical experiments to confirm that our model is computationally stable. It generates samples of comparable quality to those produced by prior Sinkhorn and Wasserstein GANs. Further simulations, assessed on the similarity index measures (SSIM), show that our model’s empirical convergence rate is comparable to that of WGAN-GP.

Linear Mixed Effects Models for Non-Gaussian Continuous Repeated Measurement Data

SummaryWe consider the analysis of continuous repeated measurement outcomes that are collected longitudinally. A standard framework for analysing data of this kind is a linear Gaussian mixed effects model within which the outcome variable can be decomposed into fixed effects, time invariant and time-varying random effects, and measurement noise. We develop methodology that, for the first time, allows any combination of these stochastic components to be non-Gaussian, using multivariate normal variance–mean mixtures. To meet the computational challenges that are presented by large data sets, i.e. in the current context, data sets with many subjects and/or many repeated measurements per subject, we propose a novel implementation of maximum likelihood estimation using a computationally efficient subsampling-based stochastic gradient algorithm. We obtain standard error estimates by inverting the observed Fisher information matrix and obtain the predictive distributions for the random effects in both filtering (conditioning on past and current data) and smoothing (conditioning on all data) contexts. To implement these procedures, we introduce an R package: ngme. We reanalyse two data sets, from cystic fibrosis and nephrology research, that were previously analysed by using Gaussian linear mixed effects models.

Deep generative models for reject inference in credit scoring

Credit scoring models based on accepted applications may be biased and their consequences can have a statistical and economic impact. Reject inference is the process of attempting to infer the creditworthiness status of the rejected applications. Inspired by the promising results of semi-supervised deep generative models, this research develops two novel Bayesian models for reject inference in credit scoring combining Gaussian mixtures and auxiliary variables in a semi-supervised framework with generative models. To the best of our knowledge this is the first study coupling these concepts together. The goal is to improve the classification accuracy in credit scoring models by adding reject applications. Further, our proposed models infer the unknown creditworthiness of the rejected applications by exact enumeration of the two possible outcomes of the loan (default or non-default). The efficient stochastic gradient optimization technique used in deep generative models makes our models suitable for large data sets. Finally, the experiments in this research show that our proposed models perform better than classical and alternative machine learning models for reject inference in credit scoring, and that model performance increases with the amount of data used for model training.

Transformed [formula omitted] regularization for learning sparse deep neural networks

Deep Neural Networks (DNNs) have achieved extraordinary success in numerous areas. However, DNNs often carry a large number of weight parameters, leading to the challenge of heavy memory and computation costs. Overfitting is another challenge for DNNs when the training data are insufficient. These challenges severely hinder the application of DNNs in resource-constrained platforms. In fact, many network weights are redundant and can be removed from the network without much loss of performance. In this paper, we introduce a new non-convex integrated transformed ℓ1 regularizer to promote sparsity for DNNs, which removes redundant connections and unnecessary neurons simultaneously. Specifically, we apply the transformed ℓ1 regularizer to the matrix space of network weights and utilize it to remove redundant connections. Besides, group sparsity is integrated to remove unnecessary neurons. An efficient stochastic proximal gradient algorithm is presented to solve the new model. To the best of our knowledge, this is the first work to develop a non-convex regularizer in sparse optimization based method to simultaneously promote connection-level and neuron-level sparsity for DNNs. Experiments on public datasets demonstrate the effectiveness of the proposed method.

Attributed network embedding via subspace discovery

Network embedding aims to learn a latent, low-dimensional vector representations of network nodes, effective in supporting various network analytic tasks. While prior arts on network embedding focus primarily on preserving network topology structure to learn node representations, recently proposed attributed network embedding algorithms attempt to integrate rich node content information with network topological structure for enhancing the quality of network embedding. In reality, networks often have sparse content, incomplete node attributes, as well as the discrepancy between node attribute feature space and network structure space, which severely deteriorates the performance of existing methods. In this paper, we propose a unified framework for attributed network embedding-attri2vec-that learns node embeddings by discovering a latent node attribute subspace via a network structure guided transformation performed on the original attribute space. The resultant latent subspace can respect network structure in a more consistent way towards learning high-quality node representations. We formulate an optimization problem which is solved by an efficient stochastic gradient descent algorithm, with linear time complexity to the number of nodes. We investigate a series of linear and non-linear transformations performed on node attributes and empirically validate their effectiveness on various types of networks. Another advantage of attri2vec is its ability to solve out-of-sample problems, where embeddings of new coming nodes can be inferred from their node attributes through the learned mapping function. Experiments on various types of networks confirm that attri2vec is superior to state-of-the-art baselines for node classification, node clustering, as well as out-of-sample link prediction tasks. The source code of this paper is available at https://github.com/daokunzhang/attri2vec.

Gaussian variational approximation with sparse precision matrices

We consider the problem of learning a Gaussian variational approximation to the posterior distribution for a high-dimensional parameter, where we impose sparsity in the precision matrix to reflect appropriate conditional independence structure in the model. Incorporating sparsity in the precision matrix allows the Gaussian variational distribution to be both flexible and parsimonious, and the sparsity is achieved through parameterization in terms of the Cholesky factor. Efficient stochastic gradient methods which make appropriate use of gradient information for the target distribution are developed for the optimization. We consider alternative estimators of the stochastic gradients which have lower variation and are more stable. Our approach is illustrated using generalized linear mixed models and state space models for time series.

Hilbert maps: Scalable continuous occupancy mapping with stochastic gradient descent

The vast amount of data robots can capture today motivates the development of fast and scalable statistical tools to model the space the robot operates in. We devise a new technique for environment representation through continuous occupancy mapping that improves on the popular occupancy grip maps in two fundamental aspects: (1) it does not assume an a priori discrimination of the world into grid cells and therefore can provide maps at an arbitrary resolution; (2) it captures spatial relationships between measurements naturally, thus being more robust to outliers and possessing better generalization performance. The technique, named Hilbert maps, is based on the computation of fast kernel approximations that project the data in a Hilbert space where a logistic regression classifier is learnt. We show that this approach allows for efficient stochastic gradient optimization where each measurement is only processed once during learning in an online manner. We present results with three types of approximations: random Fourier; Nyström; and a novel sparse projection. We also extend the approach to accept probability distributions as inputs, for example, due to uncertainty over the position of laser scans due to sensor or localization errors. In this extended version, experiments were conducted in two dimensions and three dimensions, using popular benchmark datasets. Furthermore, an analysis of the adaptive capabilities of the technique to handle large changes in the data, such as trajectory update before and after loop closure during simultaneous localization and mapping, is also included.

Robust Rank Reduction Algorithm with Iterative Parameter Optimization and Vector Perturbation

In dynamic propagation environments, beamforming algorithms may suffer from strong interference, steering vector mismatches, a low convergence speed and a high computational complexity. Reduced-rank signal processing techniques provide a way to address the problems mentioned above. This paper presents a low-complexity robust data-dependent dimensionality reduction based on an iterative optimization with steering vector perturbation (IOVP) algorithm for reduced-rank beamforming and steering vector estimation. The proposed robust optimization procedure jointly adjusts the parameters of a rank reduction matrix and an adaptive beamformer. The optimized rank reduction matrix projects the received signal vector onto a subspace with lower dimension. The beamformer/steering vector optimization is then performed in a reduced dimension subspace. We devise efficient stochastic gradient and recursive least-squares algorithms for implementing the proposed robust IOVP design. The proposed robust IOVP beamforming algorithms result in a faster convergence speed and an improved performance. Simulation results show that the proposed IOVP algorithms outperform some existing full-rank and reduced-rank algorithms with a comparable complexity.

Blind Adaptive Constrained Reduced-Rank Parameter Estimation Based on Constant Modulus Design for CDMA Interference Suppression

This paper proposes a multistage decomposition for blind adaptive parameter estimation in the Krylov subspace with the code-constrained constant modulus (CCM) design criterion. Based on constrained optimization of the constant modulus cost function and utilizing the Lanczos algorithm and Arnoldi-like iterations, a multistage decomposition is developed for blind parameter estimation. A family of computationally efficient blind adaptive reduced-rank stochastic gradient (SG) and recursive least squares (RLS) type algorithms along with an automatic rank selection procedure are also devised and evaluated against existing methods. An analysis of the convergence properties of the method is carried out and convergence conditions for the reduced-rank adaptive algorithms are established. Simulation results consider the application of the proposed techniques to the suppression of multiaccess and intersymbol interference in DS-CDMA systems.

Adaptive Interference Suppression for DS-CDMA Systems Based on Interpolated FIR Filters With Adaptive Interpolators in Multipath Channels

In this paper, we propose an adaptive linear-receiver structure based on interpolated finite-impulse response (FIR) filters with adaptive interpolators for direct-sequence code-division multiple-access (DS-CDMA) systems in multipath channels. The interpolated minimum mean-squared error (MMSE) and the interpolated constrained minimum-variance (CMV) solutions are described for a novel scheme, where the interpolator is rendered time-varying in order to mitigate multiple-access interference and multiple-path propagation effects. Based upon the interpolated MMSE and CMV solutions, we present computationally efficient stochastic gradient and exponentially weighted recursive least squares type algorithms for both receiver and interpolator filters in the supervised and blind modes of operation. A convergence analysis of the algorithms and a discussion of the convergence properties of the method are carried out for both modes of operation. Simulation experiments for a downlink scenario show that the proposed structures achieve a superior bit-error-rate convergence and steady-state performance to previously reported reduced-rank receivers at lower complexity.

Efficient Stochastic Gradient Search for Automatic Image Registration

This paper focuses mainly on an efficient stochastic gradient search algorithm for the study of optimal registration transformation. A simultaneous perturbation stochastic approximation technique is successfully implemented on image registration by optimizing mutual information based similarity measures. The hill climbing search and simplex direct search are also conducted in the experiments for the comparative purpose. The registration experiments are associated with the pairs of optical sensor images, synthetic aperture radar images and medical multimodality images, which are misaligned by the rigid or affine transformations. The experimental results show that in general the stochastic gradient search yields significant improvements on the optimal solution over the conventional hill climbing and simplex direct search in terms of accuracy and robustness. The main contribution of this paper is the first accomplishment of an efficient stochastic gradient search strategy on the mutual information based automatic image registration. (Extended paper from the International Symposium CompIMAGE – Coimbra, Portugal, 20-21 October 2006.)

Statistical performance analysis of a fast stochastic gradient for constrained adaptive time delay estimation

An efficient bi-state stochastic gradient is proposed for spontaneous constrained time delay estimation. The quantized stochastic gradient is an approximation of the polarity of the instantaneous delay estimation error. It is adjusted in such a way that it has a much higher probability to move in the correct direction at each iteration so as to enable a speed-up in the delay estimate to converge to global minimum in steady state. The performance of the delay estimator is evaluated statistically and an analytical solution for its convergence behavior is established. It is demonstrated that the proposed algorithm has at least a two-fold improvement in convergence speed when compared with the conventional approach, and this is verified by extensive simulation results.