Sparse Gradient Research Articles

Scaling Private Deep Learning with Low-rank and Sparse Gradients

Scaling Private Deep Learning with Low-rank and Sparse Gradients

Optical Flow Computation for Video Under the Dynamic Illumination

Optical flow computation for video under the dynamic illumination is a challenging issue in video multimedia applications. In this paper, we solve this issue by introducing an illumination-invariant framework for variational optical flow estimation. It consists of an illumination-invariance model that handles complex illumination changes and a data enhancement model that guarantees highly accurate optical flow estimation. In this framework, we design a log-correlation descriptor for the data term, which handles complex illumination changes by eliminating the common parameters shared by the neighboring pixels in the corresponding illumination model while improving the accuracy of optical flow estimation by enhancing the discriminability of the data term matching. We also introduce a novel optical flow model with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$L_{0}$</tex-math></inline-formula> regularizer, which reconstructs optical flow field by a sparse flow gradient counting scheme. Different from other edge-preserving regularizers, it does not depend on local motion features, but locates important flow edges globally. Therefore, it will not cause edge blurriness due to avoiding local filtering or average operation. It is particularly effective for enhancing major flow edges while eliminating a manageable degree of low-amplitude motion structures to control smoothing and reduce oversegmentation artifacts. Even small-scale motion structures with high contrast can be preserved remarkably well. The experimental results show our method significantly outperforms previous illumination-robust optical flow methods in handling complex illumination changes, and achieves competitive evaluation results on the challenging MPI-Sintel and Kitti datasets.

Multispectral Remote Sensing Image Deblurring Using Auxiliary Band Gradient Information

Multispectral remote sensing images (RSI), including hyperspectral and multispectral images, contain adequate information of ground objects and areas and play important roles in environmental monitoring, weather forecasting, urban planning, etc. However, due to many inevitable external effects on the remote sensing pathway, remote sensing images are often degraded by blur. The fields of multispectral remote sensing image deblurring have witnessed great improvements in recent years, including both optimization-based and deep-learning-based methods. However, issues are to be addressed in the remote sensing image deblurring field, such as the incompatibility of general regularizations, lack of spectral correlations for multispectral RSIs, demands of blind deblurring for real world applications, and high costs of computation. To address these problems, we incorporate a novel prior exploiting gradient information similarity between different spectral bands, and we name it auxiliary band gradient information (ABGI) prior. We show that the ABGI prior is applicable to all gradient sparsity regularizations by a simple subtract-then-add step. Specifically, we apply ABGI prior to the patch-wise minimal pixel (PMP) prior based deblurring method, and we also prove that the PMP prior exhibits sparsity for clear natural RSIs. We estimate our method on remote sensing image datasets of different spectral types and geographic resolutions. Compared to other state-of-the-art deblurring methods, our method shows superior performance on both simulated and real world blur.

Blind Text Image Deblurring Algorithm Based on Multi-Scale Fusion and Sparse Priors

The goal of blind text image deblurring is to obtain a clean text image from the given blurry text image without knowing the blur kernel. Sparsity-based methods have been shown their effectiveness in various blind text image deblurring models. However, the blur kernel estimation methods based on sparse priors lack of the consideration for the brightness information about the blur kernel, which will affect the restoration effect of the blur kernel. Besides, previous methods seldom apply sparse priors to both spatial domain and transform domain information. We propose a novel blind text image deblurring model based on multi-scale fusion and sparse priors. Besides the sparse gradient prior on the latent clean text image, we add the sparse prior on the high-frequency wavelet coefficients of the latent text image, which will better constrain the solution space and obtain good clean images. The semi-quadratic splitting method is used to alternately optimize the blur kernel and the latent clean image. Meanwhile, we consider the influence of the brightness feature of the restored blur kernel. By multi-scale fusion technique on the basis of Laplacian weight and saliency weight, we fuse the computed blur kernels in three channels to improve the quality of blur kernel. The experimental results show that our algorithm has good results in the restoration of blur kernels and text images.

Online Feature Selection Using Sparse Gradient

Feature Selection (FS) is an important preprocessing step in data analytics. It is used to select a subset of the original feature set such that the selected subset does not affect the classification performance significantly. Its objective is to remove irrelevant and redundant features from the original dataset. FS can be done either in offline mode or in online mode. The basic assumption in the former mode is that the entire dataset has been available for the FS algorithm; and the FS algorithm takes multiple epochs to select optimal feature subset that gives good accuracy. In contrast, the FS algorithms in online mode take input data one instance at a time and accumulate knowledge by learning each one of them. In online mode each instance of the original dataset is considered as training and testing sample as well. The offline FS algorithms require long time periods, if the data to be processed is large such as Big data. Whereas online FS algorithms will take only one epoch to learn the entire data and can produce the results swiftly which is highly desirable in the case of Big data. This paper deals with the online FS problem and provides a novel Feature Selection algorithm which uses the Sparse Gradient method to build a sparse classifier. In this proposed method, an online classifier is built and maintained throughout the learning process and feature weights, which are limited to a particular boundary limit, are reduced in a step by step decrement process. This method creates sparsity in the classifier. Effectively, the built classifier is used to select optimal feature subset from the incoming data. As this method reduces the weights in the classifier in step by step manner, only those important features which have value higher than the boundary survive from this repeated decrement process. The resultant optimal feature subset is formed using these non-zero weighted features. Most significantly, this particular method can be used with any learning algorithm. To show its applicability with different learning algorithms, various online feature selection models have been built using Learning Vector Quantization, Radial Basis Function Networks and Adaptive Resonance Theory MAP. In all these models, the proposed Sparse Gradient method is used. The encouraging results shows the effectiveness of the proposed method with different learning algorithms in medium and large sized benchmark datasets.

Communication-Efficient Federated Learning Over MIMO Multiple Access Channels

Communication efficiency is of importance for wireless federated learning systems. In this paper, we propose a communication-efficient strategy for federated learning over multiple-input multiple-output (MIMO) multiple access channels (MACs). The proposed strategy comprises two components. When sending a locally computed gradient, each device compresses a high dimensional local gradient to multiple lower-dimensional gradient vectors using block sparsification. When receiving a superposition of the compressed local gradients via a MIMO-MAC, a parameter server (PS) performs a joint MIMO detection and the sparse local-gradient recovery. Inspired by the turbo decoding principle, our joint detection-and-recovery algorithm accurately recovers the high-dimensional local gradients by iteratively exchanging their beliefs for MIMO detection and sparse local gradient recovery outputs. We then analyze the reconstruction error of the proposed algorithm and its impact on the convergence rate of federated learning. From simulations, our gradient compression and joint detection-and-recovery methods diminish the communication cost significantly while achieving identical classification accuracy for the case without any compression.

On the Sparse Gradient Denoising Optimization of Neural Network Models for Rolling Bearing Fault Diagnosis Illustrated by a Ship Propulsion System

The drive rolling bearing is an important part of a ship’s system; the detection of the drive rolling bearing is an important component in ship-fault diagnosis, and machine learning methods are now widely used in the fault diagnosis of rolling bearings. However, training methods based on small batches have a disadvantage in that the samples which best represent the gradient descent direction can be disturbed by either other samples in the opposite direction or anomalies. Aiming at this problem, a sparse denoising gradient descent (SDGD) optimization algorithm, based on the impact values of network nodes, was proposed to improve the updating method of the batch gradient. First, the network is made sparse by using the node weight method based on the mean impact value. Second, the batch gradients are clustered via a distribution-density-based clustering method. Finally, the network parameters are updated using the gradient values after clustering. The experimental results show the efficiency and feasibility of the proposed method. The SDGD model can achieve up to a 2.35% improvement in diagnostic accuracy compared to the traditional network diagnosis model. The training convergence speed of the SDGD model improves by 2.16%, up to 17.68%. The SDGD model can effectively solve the problem of falling into the local optimum point while training a network.

Phase Transition of Total Variation Based on Approximate Message Passing Algorithm

In compressed sensing (CS), one seeks to down-sample some high-dimensional signals and recover them accurately by exploiting the sparsity of the signals. However, the traditional sparsity assumption cannot be directly satisfied in most practical applications. Fortunately, many signals-of-interest do at least exhibit a low-complexity representation with respect to a certain transformation. Particularly, total variation (TV) minimization is a notable example when the transformation operator is a difference matrix. Presently, many theoretical properties of total variation have not been completely explored, e.g., how to estimate the precise location of phase transitions and their rigorous understanding is still in its infancy. So far, the performance and robustness of the existing algorithm for phase transition prediction of TV model are not satisfactory. In this paper, we design a new approximate message passing algorithm to solve the above problems, called total variation vector approximate message passing (TV-VAMP) algorithm. To be specific, we first consider the problem from the Bayesian perspective, and formulate it as an optimization problem. Then, the vector factor graph for the TV model is constructed based on the formulized problem. Finally, the TV-VAMP algorithm is derived according to the idea of probabilistic inference in machine learning. Compared with the existing algorithm, our algorithm can be applied to a wider range of measurements distributions, including the non-zero-mean Gaussian distribution measurements matrix and ill-conditioned measurements matrix. Furthermore, in experiments with various settings, including different measurement distribution matrices, signal gradient sparsity, and measurement times, the proposed algorithm can almost reach the target mean squared error (−60 dB) with fewer iterations and better fit the empirical phase transition curve.

Low-frequency seismic random noise attenuation based on EPLL-TV under double prior constraints

The expected patch log likelihood (EPLL) method utilizes the learned Gaussian mixture model (GMM) as a prior for white Gaussian noise removal, so that every patch from noisy data is likely under the prior and the global morphology is remained along with denoising. For suppressing the low-frequency seismic random noise having similar waveforms to the signals, we propose an Expected Patch Log Likelihood-Total Variation (EPLL-TV) algorithm, which is constrained by the EPLL prior and the total variation (TV) prior. In the denoising process, the TV prior introduces the structural features of the current data and restrains the random noise by enhancing the gradient sparsity of seismic data. Hence, the EPLL-TV algorithm is able to choose the Gaussian component which is closest to the features of signals from pre-learned general prior of EPLL, although the features in the patch are severely disturbed by the low-frequency noise. Then the seismic events with complex structures can be accurately restored by the chosen Gaussian component, which provides more reliable structures for denoising via the TV prior. We use the alternating direction method of multipliers (ADMM) to solve the proposed optimization model with double constraints and reconstruct the data in Fourier domain to reduce computational complexity. The results of the synthetic and field seismic data demonstrate that the proposed EPLL-TV method is superior to the other methods mentioned in our paper in terms of recovering the seismic events with complex structures and suppressing the low-frequency seismic random noise which has similar waveforms to the signals.

Zeroth-Order Regularized Optimization (ZORO): Approximately Sparse Gradients and Adaptive Sampling

We consider the problem of minimizing a high-dimensional objective function, which may include a regularization term, using only (possibly noisy) evaluations of the function. Such optimization is also called derivative-free, zeroth-order, or black-box optimization. We propose a new zeroth-order regularized optimization method, dubbed ZORO. When the underlying gradient is approximately sparse at an iterate, ZORO needs very few objective function evaluations to obtain a new iterate that decreases the objective function. We achieve this with an adaptive, randomized gradient estimator, followed by an inexact proximal-gradient scheme. Under a novel approximately sparse gradient assumption and various different convex settings, we show that the (theoretical and empirical) convergence rate of ZORO is only logarithmically dependent on the problem dimension. Numerical experiments show that ZORO outperforms existing methods with similar assumptions, on both synthetic and real datasets.

A Novel Stripe Noise Removal Model for Infrared Images.

Infrared images often carry obvious streak noises due to the non-uniformity of the infrared detector and the readout circuit. These streak noises greatly affect the image quality, adding difficulty to subsequent image processing. Compared with current elimination algorithms for infrared stripe noises, our approach fully utilizes the difference between the stripe noise components and the actual information components, takes the gradient sparsity along the stripe direction and the global sparsity of the stripe noises as regular terms, and treats the sparsity of the components across the stripe direction as a fidelity term. On this basis, an adaptive edge-preserving operator (AEPO) based on edge contrast was proposed to protect the image edge and, thus, prevent the loss of edge details. The final solution was obtained by the alternating direction method of multipliers (ADMM). To verify the effectiveness of our approach, many real experiments were carried out to compare it with state-of-the-art methods in two aspects: subjective judgment and objective indices. Experimental results demonstrate the superiority of our approach.

Adaptive sparse ternary gradient compression for distributed DNN training in edge computing

In edge computing, though distributed training of Deep Neural Networks (DNNs) is expected to exchange massive gradients between parameter servers and working nodes, the high communication cost constrains the training speed. To break this limitation, gradient compression algorithms expect the ultimate compression ratio at the expense of the accuracy of the trained model. Therefore, new gradient compression techniques are necessary to ensure both communication efficiency and model accuracy. This paper introduces a novel technique—an Adaptive Sparse Ternary Gradient Compression (ASTC) scheme, which relies on the number of gradients in model layers to compress gradients. ASTC establishes the model compression selection criterion by gradients’ amount, compresses the network layer that meets the model’s standard, evaluates the gradients’ importance based on entropy to adaptively perform sparse compression, and finally conducts ternary quantization compression and a lossless code scheme on sparse gradients. Using public datasets (MNIST, CIFAR-10, Tiny ImageNet) and deep learning models (CNN, LeNet5, ResNet18) for experimental evaluation, we exhibit excellent results that the training efficiency of ASTC is about 1.6 times, 1.37 times, and 1.1 times higher than that of Top-1, AdaComp, and SBC, respectively. Furthermore, ASTC can be improved by an average of about 1.9% in training accuracy compared with the above approaches.

A Two-Stage Pansharpening Method for the Fusion of Remote-Sensing Images

The pansharpening (PS) of remote-sensing images aims to fuse a high-resolution panchromatic image with several low-resolution multispectral images for obtaining a high-resolution multispectral image. In this work, a two-stage PS model is proposed by integrating the ideas of component replacement and the variational method. The global sparse gradient of the panchromatic image is extracted by variational method, and the weight function is constructed by combining the gradient of multispectral image in which the global sparse gradient can provide more robust gradient information. Furthermore, we refine the results in order to reduce spatial and spectral distortions. Experimental results show that our method had high generalization ability for QuickBird, Gaofen-1, and WorldView-4 satellite data. Experimental results evaluated by seven metrics demonstrate that the proposed two-stage method enhanced spatial details subjective visual effects better than other state-of-the-art methods do. At the same time, in the process of quantitative evaluation, the method in this paper had high improvement compared with that other methods, and some of them can reach a maximal improvement of 60%.

An iterative approach for image fusion with dynamic gradient sparsity and anisotropic spectral–spatial total variation

An iterative approach for image fusion with dynamic gradient sparsity and anisotropic spectral–spatial total variation

Compressive Synthetic Aperture Radar Imaging and Autofocusing by Augmented Lagrangian Methods

We consider the problem of synthetic aperture radar (SAR) image reconstruction from undersampled data in the presence of phase errors. We formulate the problem as one of estimating both the phase errors and the underlying image simultaneously through optimization. Within that optimization framework, we use a combination of priors that have proven useful in radar imaging, including strong sparse scattering in the image domain, as well as piecewise smoothness (i.e., gradient sparsity). To solve the resulting optimization problems, we propose an alternating direction method of multipliers (ADMM), an augmented Lagrangian method, for compressive SAR image reconstruction. We use simulated and real data at varying undersampling rates to study the performance of the proposed method and compare it against existing methods in terms of convergence speed and reconstruction quality. Our method provides significant improvements in terms of image reconstruction quality and computation speed, as well as phase error estimation performance.

Sparse assortment personalization in high dimensions

The data-driven conditional multinomial logit choice model with customer features performs well in the assortment personalization problem when the low-rank structure of the parameter matrix is considered. However, despite recent theoretical and algorithmic advances, parameter estimation in the choice model still poses a challenging task, especially when there are more predictors than observations. For this reason, we suggest a penalized likelihood approach based on a feature matrix to recover the sparse structure from populations and products toward the assortment. Our proposed method considers simultaneously low-rank and sparsity structures, which can further reduce model complexity and improve its estimation and prediction accuracy. A new algorithm, sparse factorial gradient descent (SFGD), was proposed to estimate the parameter matrix, which has high interpretability and efficient computing performance. As a first-order method, the SFGD works well in high-dimensional scenarios because of the absence of the Hessian matrix. Simulation studies show that the SFGD algorithm outperforms state-of-the-art methods in terms of estimation, sparsity recovery, and average regret. We also demonstrate the effectiveness of our proposed method using advertising behavior data analysis.

Low-rank tensor completion using nonconvex total variation

In this work, we study the tensor completion problem in which the main point is to predict the missing values in visual data. To greatly benefit from the smoothness structure and edge-preserving property in visual images, we suggest a tensor completion model that seeks gradient sparsity via the l0-norm. The proposal combines the low-rank matrix factorization which guarantees the low-rankness property and the nonconvex total variation (TV). We present several experiments to demonstrate the performance of our model compared with popular tensor completion methods in terms of visual and quantitative measures.

Image Denoising Using Nonlocal Regularized Deep Image Prior

Deep neural networks have shown great potential in various low-level vision tasks, leading to several state-of-the-art image denoising techniques. Training a deep neural network in a supervised fashion usually requires the collection of a great number of examples and the consumption of a significant amount of time. However, the collection of training samples is very difficult for some application scenarios, such as the full-sampled data of magnetic resonance imaging and the data of satellite remote sensing imaging. In this paper, we overcome the problem of a lack of training data by using an unsupervised deep-learning-based method. Specifically, we propose a deep-learning-based method based on the deep image prior (DIP) method, which only requires a noisy image as training data, without any clean data. It infers the natural images with random inputs and the corrupted observation with the help of performing correction via a convolutional network. We improve the original DIP method as follows: Firstly, the original optimization objective function is modified by adding nonlocal regularizers, consisting of a spatial filter and a frequency domain filter, to promote the gradient sparsity of the solution. Secondly, we solve the optimization problem with the alternating direction method of multipliers (ADMM) framework, resulting in two separate optimization problems, including a symmetric U-Net training step and a plug-and-play proximal denoising step. As such, the proposed method exploits the powerful denoising ability of both deep neural networks and nonlocal regularizations. Experiments validate the effectiveness of leveraging a combination of DIP and nonlocal regularizers, and demonstrate the superior performance of the proposed method both quantitatively and visually compared with the original DIP method.

Estimating Crop Biophysical Parameters Using Machine Learning Algorithms and Sentinel-2 Imagery

Global food security is critical to eliminating hunger and malnutrition. In the changing climate, farmers in developing countries must adopt technologies and farming practices such as precision agriculture (PA). PA-based approaches enable farmers to cope with frequent and intensified droughts and heatwaves, optimising yields, increasing efficiencies, and reducing operational costs. Biophysical parameters such as Leaf Area Index (LAI), Leaf Chlorophyll Content (LCab), and Canopy Chlorophyll Content (CCC) are essential for characterising field-level spatial variability and thus are necessary for enabling variable rate application technologies, precision irrigation, and crop monitoring. Moreover, robust machine learning algorithms offer prospects for improving the estimation of biophysical parameters due to their capability to deal with non-linear data, small samples, and noisy variables. This study compared the predictive performance of sparse Partial Least Squares (sPLS), Random Forest (RF), and Gradient Boosting Machines (GBM) for estimating LAI, LCab, and CCC with Sentinel-2 imagery in Bothaville, South Africa and identified, using variable importance measures, the most influential bands for estimating crop biophysical parameters. The results showed that RF was superior in estimating all three biophysical parameters, followed by GBM which was better in estimating LAI and CCC, but not LCab, where sPLS was relatively better. Since all biophysical parameters could be achieved with RF, it can be considered a good contender for operationalisation. Overall, the findings in this study are significant for future biophysical product development using RF to reduce reliance on many algorithms for specific parameters, thus facilitating the rapid extraction of actionable information to support PA and crop monitoring activities.

Sparse Variational Deterministic Policy Gradient for Continuous Real-Time Control

Recent advancements in deep reinforcement learning for real control tasks have received interest from many researchers and field engineers in a variety of industrial areas. However, in most cases, optimal policies obtained by deep reinforcement learning are difficult to implement on cost-effective and lightweight platforms such as mobile devices. This can be attributed to their computational complexity and excessive memory usage. For this reason, this article proposes an off-policy deep reinforcement learning algorithm called the sparse variational deterministic policy gradient (SVDPG). SVDPG provides highly efficient policy network compression under the standard reinforcement learning framework. The proposed SVDPG integrates Bayesian pruning, which is known as a state-of-the-art neural network compression technique, with the policy update in an actor–critic architecture for reinforcement learning. It is demonstrated that SVDPG achieves a high compression rate of policy networks for continuous control benchmark tasks while preserving a competitive performance. The superiority of SVDPG in low-computing power devices is proven by comparing the level of compression in terms of the memory requirements and computation time on a commercial microcontroller unit. Finally, it is confirmed that the proposed SVDPG is also reliable in real-world scenarios since it can be applied to the swing-up control of an inverted pendulum system.