Gradient Descent Regularization Research Articles

Hebbian Descent: A Unified View on Log-Likelihood Learning.

This study discusses the negative impact of the derivative of the activation functions in the output layer of artificial neural networks, in particular in continual learning. We propose Hebbian descent as a theoretical framework to overcome this limitation, which is implemented through an alternative loss function for gradient descent we refer to as Hebbian descent loss. This loss is effectively the generalized log-likelihood loss and corresponds to an alternative weight update rule for the output layer wherein the derivative of the activation function is disregarded. We show how this update avoids vanishing error signals during backpropagation in saturated regions of the activation functions, which is particularly helpful in training shallow neural networks and deep neural networks where saturating activation functions are only used in the output layer. In combination with centering, Hebbian descent leads to better continual learning capabilities. It provides a unifying perspective on Hebbian learning, gradient descent, and generalized linear models, for all of which we discuss the advantages and disadvantages. Given activation functions with strictly positive derivative (as often the case in practice), Hebbian descent inherits the convergence properties of regular gradient descent. While established pairings of loss and output layer activation function (e.g., mean squared error with linear or cross-entropy with sigmoid/softmax) are subsumed by Hebbian descent, we provide general insights for designing arbitrary loss activation function combinations that benefit from Hebbian descent. For shallow networks, we show that Hebbian descent outperforms Hebbian learning, has a performance similar to regular gradient descent, and has a much better performance than all other tested update rules in continual learning. In combination with centering, Hebbian descent implements a forgetting mechanism that prevents catastrophic interference notably better than the other tested update rules. When training deep neural networks, our experimental results suggest that Hebbian descent has better or similar performance as gradient descent.

Boundary Conditions for Linear Exit Time Gradient Trajectories Around Saddle Points: Analysis and Algorithm

Gradient-related first-order methods have become the workhorse of large-scale numerical optimization problems. Many of these problems involve nonconvex objective functions with multiple saddle points, which necessitates an understanding of the behavior of discrete trajectories of first-order methods within the geometrical landscape of these functions. This paper concerns convergence of first-order discrete methods to a local minimum of nonconvex optimization problems that comprise strict-saddle points within the geometrical landscape. To this end, it focuses on analysis of discrete gradient trajectories around saddle neighborhoods, derives sufficient conditions under which these trajectories can escape strict-saddle neighborhoods in linear time, explores the contractive and expansive dynamics of these trajectories in neighborhoods of strict-saddle points that are characterized by gradients of moderate magnitude, characterizes the non-curving nature of these trajectories, and highlights the inability of these trajectories to re-enter the neighborhoods around strict-saddle points after exiting them. Based on these insights and analyses, the paper then proposes a simple variant of the vanilla gradient descent algorithm, termed Curvature Conditioned Regularized Gradient Descent (CCRGD) algorithm, which utilizes a check for an initial boundary condition to ensure its trajectories can escape strict-saddle neighborhoods in linear time. Convergence analysis of the CCRGD algorithm, which includes its rate of convergence to a local minimum, is also presented in the paper. Numerical experiments are then provided on a test function as well as a low-rank matrix factorization problem to evaluate the efficacy of the proposed algorithm.

Perceptron Collaborative Filtering

Abstract: While multivariate logistic regression classifiers are a great way of implementing collaborative filtering - a method of making automatic predictions about the interests of a user by collecting preferences or taste information from many other users, we can also achieve similar results using neural networks. A recommender system is a subclass of information filtering system that provide suggestions for items that are most pertinent to a particular user. A perceptron or a neural network is a machine learning model designed for fitting complex datasets using backpropagation and gradient descent. When coupled with advanced optimization techniques, the model may prove to be a great substitute for classical logistic classifiers. The optimizations include feature scaling, mean normalization, regularization, hyperparameter tuning and using stochastic/mini-batch gradient descent instead of regular gradient descent. In this use case, we will use the perceptron in the recommender system to fit the parameters i.e., the data from a multitude of users and use it to predict the preference/interest of a particular user

An Accelerated First-Order Method for Non-convex Optimization on Manifolds

We describe the first gradient methods on Riemannian manifolds to achieve accelerated rates in the non-convex case. Under Lipschitz assumptions on the Riemannian gradient and Hessian of the cost function, these methods find approximate first-order critical points faster than regular gradient descent. A randomized version also finds approximate second-order critical points. Both the algorithms and their analyses build extensively on existing work in the Euclidean case. The basic operation consists in running the Euclidean accelerated gradient descent method (appropriately safe-guarded against non-convexity) in the current tangent space, then moving back to the manifold and repeating. This requires lifting the cost function from the manifold to the tangent space, which can be done for example through the Riemannian exponential map. For this approach to succeed, the lifted cost function (called the pullback) must retain certain Lipschitz properties. As a contribution of independent interest, we prove precise claims to that effect, with explicit constants. Those claims are affected by the Riemannian curvature of the manifold, which in turn affects the worst-case complexity bounds for our optimization algorithms.

Free-water DTI estimates from single b-value data might seem plausible but must be interpreted with care.

Free-water elimination DTI (FWE-DTI) has been used widely to distinguish increases of free-water partial-volume effects from tissue's diffusion in healthy aging and degenerative diseases. Because the FWE-DTI fitting is only well-posed for multishell acquisitions, a regularized gradient descent (RGD) method was proposed to enable application to single-shell data, more common in the clinic. However, the validity of the RGD method has been poorly assessed. This study aims to quantify the specificity of FWE-DTI procedures on single-shell and multishell data. Different FWE-DTI fitting procedures were tested on an open-source in vivo diffusion data set and single-shell and multishell synthetic signals, including the RGD and standard nonlinear least-squares methods. Single-voxel simulations were carried out to compare initialization approaches. A multivoxel phantom simulation was performed to evaluate the effect of spatial regularization when comparing between methods. To test the algorithms' specificity, phantoms with two different types of lesions were simulated: with altered mean diffusivity or with modified free water. Plausible parameter maps were obtained with RGD from single-shell in vivo data. The plausibility of these maps was shown to be determined by the initialization. Tests with simulated lesions inserted into the in vivo data revealed that the RGD approach cannot distinguish free water from tissue mean-diffusivity alterations, contrarily to the nonlinear least-squares algorithm. The RGD FWE-DTI method has limited specificity; thus, its results from single-shell data should be carefully interpreted. When possible, multishell acquisitions and the nonlinear least-squares approach should be preferred instead.

Blind Deconvolution Using Modulated Inputs

This paper considers the blind deconvolution of multiple modulated signals/filters, and an arbitrary filter/signal. Multiple inputs $\boldsymbol{s}_1, \boldsymbol{s}_2, \ldots, \boldsymbol{s}_N =: [\boldsymbol{s}_n]$ are modulated (pointwise multiplied) with random sign sequences $\boldsymbol{r}_1, \boldsymbol{r}_2, \ldots, \boldsymbol{r}_N =: [\boldsymbol{r}_n]$ , respectively, and the resultant inputs $(\boldsymbol{s}_n \odot \boldsymbol{r}_n) \in \mathbb {C}^Q, n \in [N]$ are convolved against an arbitrary input $\boldsymbol{h} \in \mathbb {C}^M$ to yield the measurements $\boldsymbol{y}_n = (\boldsymbol{s}_n\odot \boldsymbol{r}_n)\circledast \boldsymbol{h}, n \in [N] := 1,2,\ldots,N,$ where $\odot$ and $\circledast$ denote pointwise multiplication, and circular convolution. Given $[\boldsymbol{y}_n]$ , we want to recover the unknowns $[\boldsymbol{s}_n]$ and $\boldsymbol{h}$ . We make a structural assumption that unknowns $[\boldsymbol{s}_n]$ are members of a known $K$ -dimensional (not necessarily random) subspace, and prove that the unknowns can be recovered from sufficiently many observations using a regularized gradient descent algorithm whenever the modulated inputs $\boldsymbol{s}_n \odot \boldsymbol{r}_n$ are long enough, i.e, $Q \gtrsim KN+M$ (to within logarithmic factors, and signal dispersion/coherence parameters). Under the bilinear model, this is the first result on multichannel ( $N\geq 1$ ) blind deconvolution with provable recovery guarantees under near optimal (in the $N=1$ case) sample complexity estimates, and comparatively lenient structural assumptions on the convolved inputs. A neat conclusion of this result is that modulation of a bandlimited signal protects it against an unknown convolutive distortion. We discuss the applications of this result in passive imaging, wireless communication in unknown environment, and image deblurring. A thorough numerical investigation of the theoretical results is also presented using phase transitions, image deblurring experiments, and noise stability plots.

Training binary neural networks with knowledge transfer

Binary Neural Networks (BNNs) use binary values for both weights and activations instead of 32 bit floating point numbers typically used in deep neural networks. This reduces the memory footprint by a factor of 32 and allows a very efficient implementation in hardware. BNNs are trained using regular gradient descent but are harder to optimise, take longer to train and generally require a more careful tuning of hyperparameters such as the learning rate decay schedule than floating point versions. We propose to use Knowledge Transfer techniques to make it easier to train BNNs. Knowledge transfer is a general technique that tries to transfer the knowledge stored in a large network (the teacher) to a smaller (student) network. In our case the teacher is a network trained with floating point weights and activations while the student is a BNN. We apply different Knowledge Transfer techniques to the task of training a BNN. We introduce a novel similarity based Knowledge Transfer algorithm and show that this technique results in a higher test accuracy on different benchmark datasets compared to training the BNN from scratch.

Regularized gradient descent: a non-convex recipe for fast joint blind deconvolution and demixing

We study the question of extracting a sequence of functions $\{\boldsymbol{f}_i, \boldsymbol{g}_i\}_{i=1}^s$ from observing only the sum of their convolutions, i.e., from $\boldsymbol{y} = \sum_{i=1}^s \boldsymbol{f}_i\ast \boldsymbol{g}_i$. While convex optimization techniques are able to solve this joint blind deconvolution-demixing problem provably and robustly under certain conditions, for medium-size or large-size problems we need computationally faster methods without sacrificing the benefits of mathematical rigor that come with convex methods. In this paper, we present a non-convex algorithm which guarantees exact recovery under conditions that are competitive with convex optimization methods, with the additional advantage of being computationally much more efficient. Our two-step algorithm converges to the global minimum linearly and is also robust in the presence of additive noise. While the derived performance bounds are suboptimal in terms of the information-theoretic limit, numerical simulations show remarkable performance even if the number of measurements is close to the number of degrees of freedom. We discuss an application of the proposed framework in wireless communications in connection with the Internet-of-Things.

Rapid, robust, and reliable blind deconvolution via nonconvex optimization

Abstract We study the question of reconstructing two signals f and g from their convolution y = f ⁎ g . This problem, known as blind deconvolution, pervades many areas of science and technology, including astronomy, medical imaging, optics, and wireless communications. A key challenge of this intricate non-convex optimization problem is that it might exhibit many local minima. We present an efficient numerical algorithm that is guaranteed to recover the exact solution, when the number of measurements is (up to log-factors) slightly larger than the information-theoretical minimum, and under reasonable conditions on f and g. The proposed regularized gradient descent algorithm converges at a geometric rate and is provably robust in the presence of noise. To the best of our knowledge, our algorithm is the first blind deconvolution algorithm that is numerically efficient, robust against noise, and comes with rigorous recovery guarantees under certain subspace conditions. Moreover, numerical experiments do not only provide empirical verification of our theory, but they also demonstrate that our method yields excellent performance even in situations beyond our theoretical framework.

Co-registration of intra-operative brain surface photographs and pre-operative MR images

Brain shift, the change in configuration of the brain after opening the dura mater, is a significant problem for neuronavigation. Brain structures at intra-operative deformed positions must be matched with corresponding structures in the pre-operative 3D planning data. A method to co-register the cortical surface from intra-operative microscope images with pre-operative MRI-segmented data was developed and tested. Automated classification of sulci on MRI-extracted cortical surfaces was tested by comparison with user guided marking of prominent sulci on an intra-operative photography. A variational registration method with a fidelity energy for 3D deformations of the cortical surface in conjunction with a higher-order, linear elastic prior energy was used for the actual registration. The minimization of this energy was performed with a regularized gradient descent scheme using finite elements for spatial discretization. The sulcal classification method was tested on eight different clinical MRI data sets by comparison of the deformed MRI scans with intra-operative photographs of the brain surface. User intervention was required for marking sulci on the photographs demonstrating the potential for incorporating an automatic classifier. The actual registration was validated first on an artificial testbed. The complete algorithm for the co-registration of actual clinical MRI data was successful for eight different patients. Pre-operative MRI scans can be registered to intra-operative brain surface photographs using a surface-to-surface registration method. This co-registration method has potential applications in neurosurgery, particularly during functional procedures.

Integrative analysis of cancer prognosis data with multiple subtypes using regularized gradient descent.

In cancer research, high-throughput profiling studies have been extensively conducted, searching for genes/single nucleotide polymorphisms (SNPs) associated with prognosis. Despite seemingly significant differences, different subtypes of the same cancer (or different types of cancers) may share common susceptibility genes. In this study, we analyze prognosis data on multiple subtypes of the same cancer but note that the proposed approach is directly applicable to the analysis of data on multiple types of cancers. We describe the genetic basis of multiple subtypes using the heterogeneity model that allows overlapping but different sets of susceptibility genes/SNPs for different subtypes. An accelerated failure time (AFT) model is adopted to describe prognosis. We develop a regularized gradient descent approach that conducts gene-level analysis and identifies genes that contain important SNPs associated with prognosis. The proposed approach belongs to the family of gradient descent approaches, is intuitively reasonable, and has affordable computational cost. Simulation study shows that when prognosis-associated SNPs are clustered in a small number of genes, the proposed approach outperforms alternatives with significantly more true positives and fewer false positives. We analyze an NHL (non-Hodgkin lymphoma) prognosis study with SNP measurements and identify genes associated with the three major subtypes of NHL, namely, DLBCL, FL, and CLL/SLL. The proposed approach identifies genes different from using alternative approaches and has the best prediction performance.

An SL(2) Invariant Shape Median

Median averaging is a powerful averaging concept on sets of vector data in finite dimensions. A generalization of the median for shapes in the plane is introduced. The underlying distance measure for shapes takes into account the area of the symmetric difference of shapes, where shapes are considered to be invariant with respect to different classes of affine transformations. To obtain a well-posed problem the perimeter is introduced as a geometric prior. Based on this model, an existence result can be established in the class of sets of finite perimeter. As alternative invariance classes other classical transformation groups such as pure translation, rotation, scaling, and shear are investigated. The numerical approximation of median shapes uses a level set approach to describe the shape contour. The level set function and the parameter sets of the group action on every given shape are incorporated in a joint variational functional, which is minimized based on step size controlled, regularized gradient descent. Various applications show in detail the qualitative properties of the median.

Extracting Grain Boundaries and Macroscopic Deformations from Images on Atomic Scale

Nowadays image acquisition in materials science allows the resolution of grains at atomic scale. Grains are material regions with different lattice orientation which are frequently in addition elastically stressed. At the same time, new microscopic simulation tools allow to study the dynamics of such grain structures. Single atoms are resolved experimentally as well as in simulation results on the data microscale, whereas lattice orientation and elastic deformation describe corresponding physical structures mesoscopically. A qualitative study of experimental images and simulation results and the comparison of simulation and experiment requires the robust and reliable extraction of mesoscopic properties from the microscopic image data. Based on a Mumford---Shah type functional, grain boundaries are described as free discontinuity sets at which the orientation parameter for the lattice jumps. The lattice structure itself is encoded in a suitable integrand depending on a local lattice orientation and one global elastic displacement. For each grain a lattice orientation and an elastic displacement function are considered as unknowns implicitly described by the image microstructure. In addition the approach incorporates solid---liquid interfaces. The resulting Mumford---Shah functional is approximated with a level set active contour model following the approach by Chan and Vese. The implementation is based on a finite element discretization in space and a step size controlled, regularized gradient descent algorithm.