Descent Step Research Articles

Direct Multi-Material Reconstruction via Iterative Proximal Adaptive Descent for Spectral CT Imaging.

Spectral computed tomography (spectral CT) is a promising medical imaging technology because of its ability to provide information on material characterization and quantification. However, with an increasing number of basis materials, the nonlinearity of measurements causes difficulty in decomposition. In addition, noise amplification and beam hardening further reduce image quality. Thus, improving the accuracy of material decomposition while suppressing noise is pivotal for spectral CT imaging. This paper proposes a one-step multi-material reconstruction model as well as an iterative proximal adaptive decent method. In this approach, a proximal step and a descent step with adaptive step size are designed under the forward-backward splitting framework. The convergence analysis of the algorithm is further discussed according to the convexity of the optimization objective function. For simulation experiments with different noise levels, the peak signal-to-noise ratio (PSNR) obtained by the proposed method increases approximately 23 dB, 14 dB, and 4 dB compared to those of other algorithms. Magnified areas of thorax data further demonstrated that the proposed method has a better ability to preserve details in tissues, bones, and lungs. Numerical experiments verify that the proposed method efficiently reconstructed the material maps, and reduced noise and beam hardening artifacts compared with the state-of-the-art methods.

Boosting auxiliary task guidance: a probabilistic approach

This work aims to introduce a novel approach for auxiliary task guidance (ATG). In this approach, our goal is to achieve effective guidance from a suitable auxiliary task by utilizing the uncertainty in calculated gradients for a mini-batch of samples. Our method calculates a probabilistic fitness factor of the auxiliary task gradient for each of the shared weights to guide the main task at every training step of mini-batch gradient descent. We have shown that this proposed factor incorporates task specific confidence of learning to manipulate ATG in an effective manner. For studying the potency of the method, monocular visual odometry (VO) has been chosen as an application. Substantial experiments have been done on the KITTI VO dataset for solving monocular VO with a simple convolutional neural network (CNN) architecture. Corresponding results show that our ATG method significantly boosts the performance of supervised learning for VO. It also out performs state-of-the-art (SOTA) auxiliary guided methods we applied for VO. The proposed method is able to achieve decent scores (in some cases competitive)compared to existing SOTA supervised monocular VO algorithms, while keeping an exceptionally low parameter space in supervised regime.

Proximal Decoding for LDPC Codes

This paper presents a novel optimization-based decoding algorithm for LDPC codes. The proposed decoding algorithm is based on a proximal gradient method for solving an approximate maximum a posteriori (MAP) decoding problem. The key idea of the proposed algorithm is the use of a code-constraint polynomial to penalize a vector far from a codeword as a regularizer in the approximate MAP objective function. A code proximal operator is naturally derived from a code-constraint polynomial. The proposed algorithm, called proximal decoding, can be described by a simple recursive formula consisting of the gradient descent step for a negative log-likelihood function corresponding to the channel conditional probability density function and the code proximal operation regarding the code-constraint polynomial. Proximal decoding is experimentally shown to be applicable to several non-trivial channel models such as LDPC-coded massive MIMO channels, correlated Gaussian noise channels, and nonlinear vector channels. In particular, in MIMO channels, proximal decoding outperforms known massive MIMO detection algorithms, such as an MMSE detector with belief propagation decoding. The simple optimization-based formulation of proximal decoding allows a way for developing novel signal processing algorithms involving LDPC codes.

Accelerating probabilistic tensor canonical polyadic decomposition with nonnegative factors: An inexact BCD Approach

Recently, Bayesian modeling and variational inference (VI) were leveraged to enable the nonnegative factor matrix learning with automatic rank determination in tensor canonical polyadic decomposition (CPD), which has found various applications in big data analytics. However, since VI inherently performs block coordinate descent (BCD) steps over the functional space, it generally does not allow integration with modern large-scale optimization methods, making the scalability a critical issue. In this paper, it is revealed that the expectations of the variables updated by the VI algorithm is equivalent to the block minimization steps of a deterministic optimization problem. This equivalence further enables the adoption of inexact BCD method for devising a fast nonnegative factor matrix learning algorithm with automatic tensor rank determination. Numerical results using synthetic data and real-world applications show that the performance of the proposed algorithm is comparable with that of the VI-based algorithm, but with computation times reduced significantly.

A Unified Analysis of Descent Sequences in Weakly Convex Optimization, Including Convergence Rates for Bundle Methods

We present a framework for analyzing convergence and local rates of convergence of a class of descent algorithms, assuming the objective function is weakly convex. The framework is general, in the sense that it combines the possibility of explicit iterations (based on the gradient or a subgradient at the current iterate), implicit iterations (using a subgradient at the next iteration, like in the proximal schemes), as well as iterations when the associated subgradient is specially constructed and does not correspond either to the current or the next point (this is the case of descent steps in bundle methods). Under the subdifferential-based error bound on the distance to critical points, linear rates of convergence are established. Our analysis applies, among other techniques, to prox-descent for decomposable functions, the proximal-gradient method for a sum of functions, redistributed bundle methods, and a class of algorithms that can be cast in the feasible descent framework for constrained optimization.

Adversarial gradient-based meta learning with metric-based test

The gradient-based meta learning and its approximation algorithms have been widely used in the few-shot scenarios. In practice, it is common for the trained meta-model to employ uniform settings for gradient descent steps across different tasks. However, the meta-model may be biased toward some tasks. The convergence issue occurs that some tasks may see convergence in a few steps while others fail to approach the optimum in the whole inner loop. The bias problem may cause the trained meta-model works well in some tasks but has unexpected bad performance in other tasks, which hurts the generality of the meta-model. To address this issue, in this paper, we formally establish the approximation between the metric-based strategy and gradient descent in meta-test. By directly calculating similarity to classify data, the trained meta-model avoids the convergence issue. We point out that the metric-based methods can closely approximate the gradient descent in meta-test if the representation capability of the derived features and the convergence of the inner loop during meta-training are guaranteed. Based on such observation, we propose a new meta-learning model GMT2 (Gradient-based Meta-Train with Metric-based meta-Test) by combining gradient descent in meta-training with metric-based methods in meta-test. GMT2 employs a new first-order approximation scheme using the adversarial update strategy which not only enhances the feature representation of inner layers, but also allows enough inner gradient steps without calculating second-order derivatives. Experiments show that GMT2 achieves better efficiency and competitive accuracy comparing with popular meta-learning models.

Distributed Optimization with Gradient Descent and Quantized Communication

In this paper, we consider the unconstrained distributed optimization problem, in which the exchange of information in the network is captured by a directed graph topology, thus, nodes can only communicate with their neighbors. Additionally, in our problem, the communication channels among the nodes have limited bandwidth. In order to alleviate this limitation, quantized messages should be exchanged among the nodes. For solving this distributed optimization problem, we combine a gradient descent method with a distributed quantized consensus algorithm (which requires the nodes to exchange quantized messages and converges in a finite number of steps). Specifically, at every optimization step, each node (i) performs a gradient descent step (i.e., subtracts the scaled gradient from its current estimate), and (ii) performs a finite-time calculation of the quantized average of every node's estimate in the network. As a consequence, this algorithm approximately mimics the centralized gradient descent algorithm. We show that our algorithm asymptotically converges to a neighborhood of the optimal solution with linear convergence rate. The performance of the proposed algorithm is demonstrated via simple illustrative examples.

CG-DRR：digital reconstructed radiograph generation algorithm based on Cycle-GAN

CG-DRR：digital reconstructed radiograph generation algorithm based on Cycle-GAN

Problems and Prospects for the Application of Neural Networks for the Sphere of Education

The purpose of the study is to examine the theory and practice of using neural networks in education, to develop the concept of a neural network adaptive learning environment, and to implement a neural network model of one of the subsystems of this environment (using a model example of creating an adaptive educational trajectory).Materials and methods. The study includes a review of bibliographic sources on the application of neural networks in the field of education. It also includes modeling the structure of a neural network adaptive learning environment. The PyTorch library was used to programmatically implement the neural network.Results. The prospects for the use of neural networks in the field of education are considered, including various tasks of recognition, diagnostics, classification, clustering, forecasting, optimization, etc. A structural model of an adaptive learning NeuroSmart environment is created. This environment includes a number of subsystems for solving the following tasks: biometric identification of the student; determining starting level; choosing an adaptive learning path; determining the current level of competency development; automated verification of students’ work using recognition technology; analysis of the final result of training; monitoring information security incidents in an electronic course, etc. In order to study the possibilities and problems of applying neural network models to the tasks of constructing student adaptive learning trajectories, a model example of a neural network was created. This example illustrates the possibility of using a neural network to select further nodes of the educational trajectory based on the available data on the current learning parameters in an electronic course. To implement the neural network, the PyTorch deep learning library and Pandas library modules were used. SGD, Adam, Rprop were used as an optimizer to perform gradient descent steps. For each of the optimizers, a computer study of the stability of the neural network was carried out by varying the following parameters: the learning rate coefficient, the number of neurons in the hidden layer, and the number of training epochs.Conclusion. It can be assumed that the next stage in the evolution of the use of neural network technologies in the field of education will be their integration into complex multi-component Smart systems capable of automatically supporting student learning at all stages of the implementation of their personal educational trajectory. Obviously, the practical implementation of complex neural network systems of this level is a very difficult task and can be solved so far only at the level of individual subsystems. There are a number of problems associated with computer simulation of the educational environment based on neural network models: the question of the optimal structure of the neural network (for example, the number of neurons and layers in the network) has not been studied enough, there are no clear criteria for the optimality of the adaptive educational trajectory. Nevertheless, it should be noted that the task of developing new forms and technologies of personalized e-learning is in great demand, which makes modeling based on neural networks especially relevant.

Spatiotemporal variational neural network for reconstruction of highly accelerated cardiac cine MRI

Abstract Funding Acknowledgements Type of funding sources: Private company. Main funding source(s): Siemens Healthcare GmbH; NIH Grant. Background Acquisition of cardiac cine MRI usually requires the subject to repeatedly hold their breath for several seconds. Deep learning-based reconstruction of highly undersampled data could allow higher accelerations and has been proposed for several applications. The variational network [1] (VN) was one of the first network architectures for MRI reconstruction, replicating iterative reconstruction with a regularization term learnt from training data. Purpose To expand the concept of VNs to spatiotemporal data from cardiac cine acquisitions. Reconstructions are presented and compared using 3D, 2D spatial-only, and 2D+t spatiotemporal convolutions. Methods A VN with intermediate quantities in k-space [2] was trained to reconstruct cine sequences from retrospectively undersampled data. The network architecture is illustrated in Fig. 1. It consisted of 10 cascades, each resembling a gradient descent step of an iterative reconstruction. Each refinement step contained a Residual U-Net (cf. Fig. 1d). Pre-estimated coil sensitivity maps were used in all transformations between k-space and image space. The convolution operations were implemented as either 2D spatial-only, 3D, or separable 2D+t spatiotemporal convolutions. A 2D+t convolution consisted of a 2D spatial convolution followed by an activation function and a 1D temporal convolution (cf. Fig. 1e). The OCMR dataset [3] was used for training and evaluation. It contains 183 fully sampled cine slices from 74 subjects in different imaging planes with 15 to 38 coils and 15 to 31 temporal frames at 1.5T and 3T. The training set contained 142 slices from 33 individuals, which includes all multi-slice acquisitions from the dataset. The validation set consisted of 22 single slices and the network performance was evaluated on the remaining, previously unseen 19 single-slice acquisitions. The k-space data was retrospectively undersampled using a variable density mask [4]. Sensitivity-based coil combination of the fully sampled k-space was used as ground truth. Coil sensitivity maps were estimated with ESPIRiT. Six networks were trained using the different convolution types mentioned above, each for acceleration factors R=8 and R=12. Results Exemplary reconstructions and quantitative results are given in Fig. 2. At R=8, reconstruction with 2D+t and 3D convolutions reached high structural similarity (SSIM) scores of around 0.95, while the 2D convolutions only reached 0.74. Artifacts are clearly visible in reconstructions learnt with 2D kernels. At R=12, reconstruction quality degraded most considerably with 2D convolution kernels. Conclusion Reconstruction of highly accelerated cardiac cine data using VNs yielded very good preliminary results. Convolutions that exploit temporal correlations are clearly beneficial over spatial-only convolutions. 2D+t spatiotemporal convolutions may be preferable over 3D convolutions due to reduced model capacity with similar reconstruction capability. Network design with 2D+t convolutionExample reconstruction and time profiles

Extensions of the deep Galerkin method

We extend the Deep Galerkin Method (DGM) introduced in Sirignano and Spiliopoulos(2018)[25] to solve a number of partial differential equations (PDEs) that arise in the context of optimal stochastic control and mean field games. First, we consider PDEs where the function is constrained to be positive and integrate to unity, as is the case with Fokker–Planck equations. Our approach involves reparameterizing the solution as the exponential of a neural network appropriately normalized to ensure both requirements are satisfied. This then gives rise to nonlinear a partial integro-differential equation (PIDE) where the integral appearing in the equation is handled by a novel application of importance sampling. Secondly, we tackle a number of Hamilton–Jacobi–Bellman (HJB) equations that appear in stochastic optimal control problems. The key contribution is that these equations are approached in their unsimplified primal form which includes an optimization problem as part of the equation. We extend the DGM algorithm to solve for the value function and the optimal control simultaneously by characterizing both as deep neural networks. Training the networks is performed by taking alternating stochastic gradient descent steps for the two functions, a technique inspired by the policy improvement algorithms (PIA).

MIMO Radar Beampattern Design Based on Manifold Optimization Method

The transmit beampattern design with constant modulus (CM) constraint is a key technical issue in Multiple-Input Multiple-Output (MIMO) radar system. The problem is non-convex and direct solution is impossible in general, due to the CM constraint. Usually, most existing methods indirectly solve this problem by relaxing it to a more tractable form. These methods usually either degrade the performance due to relaxation or consume expensive computational cost. Different from the relaxation methods, a direct method is proposed to solve the non-convex problem without relaxation. In the proposed method, the original problem is first converted to an Unconstrained Quadratic Lagrange Multiplier Problem (UQLMP) over the non-convex complex circle manifold. Then, by efficiently finding the descent direction and step size to ensure the objective function to be non-increasing, a novel Manifold Optimization-based Conjugate Gradient (MOCG) approach is derived to directly solve UQLMP. Compared with current methods, the proposed method has better performance in terms of nulling depth.

IPGM: Inertial Proximal Gradient Method for Convolutional Dictionary Learning

Inspired by the recent success of the proximal gradient method (PGM) and recent efforts to develop an inertial algorithm, we propose an inertial PGM (IPGM) for convolutional dictionary learning (CDL) by jointly optimizing both an ℓ2-norm data fidelity term and a sparsity term that enforces an ℓ1 penalty. Contrary to other CDL methods, in the proposed approach, the dictionary and needles are updated with an inertial force by the PGM. We obtain a novel derivative formula for the needles and dictionary with respect to the data fidelity term. At the same time, a gradient descent step is designed to add an inertial term. The proximal operation uses the thresholding operation for needles and projects the dictionary to a unit-norm sphere. We prove the convergence property of the proposed IPGM algorithm in a backtracking case. Simulation results show that the proposed IPGM achieves better performance than the PGM and slice-based methods that possess the same structure and are optimized using the alternating-direction method of multipliers (ADMM).

Distributed load forecasting using smart meter data: Federated learning with Recurrent Neural Networks

Load forecasting is essential for energy management, infrastructure planning, grid operation, and budgeting. Large scale smart meter deployments have resulted in ability to collect massive energy data and have created opportunities in sensor-based forecasting. Machine learning (ML) has demonstrated great successes in sensor-based load forecasting; however, when prediction is needed on a smart meter level, typically a single model is trained for each smart meter. With a large number of meters, this becomes computationally expensive or even infeasible. On the other hand, with conventional ML, training a single model for several smart meters requires participants to share their data with the central server. Consequently, this paper proposes federated learning for load forecasting with smart meter data: this strategy enables training a single model with all participating smart meters without the need to share local data. Two alternative federated learning strategies are examined: FedSGD, which performs one step of gradient descent on client before merging updates on the server, and FedAVG, which carries out several steps before the merging. Specifically, residential consumers are diverse what makes training a single model challenging as load profiles vary across consumers. The results show that the FedAVG achieves better accuracy than FedSGD while also requiring fewer communication rounds. Comparing to individual models for each meter and a single central models for all meters, FedAVG achieves comparable or better accuracy.

EDCNN: identification of genome-wide RNA-binding proteins using evolutionary deep convolutional neural network.

RNA-binding proteins (RBPs) are a group of proteins associated with RNA regulation and metabolism, and play an essential role in mediating the maturation, transport, localization and translation of RNA. Recently, Genome-wide RNA-binding event detection methods have been developed to predict RBPs. Unfortunately, the existing computational methods usually suffer some limitations, such as high-dimensionality, data sparsity and low model performance. Deep convolution neural network has a useful advantage for solving high-dimensional and sparse data. To improve further the performance of deep convolution neural network, we propose evolutionary deep convolutional neural network (EDCNN) to identify protein-RNA interactions by synergizing evolutionary optimization with gradient descent to enhance deep conventional neural network. In particular, EDCNN combines evolutionary algorithms and different gradient descent models in a complementary algorithm, where the gradient descent and evolution steps can alternately optimize the RNA-binding event search. To validate the performance of EDCNN, an experiment is conducted on two large-scale CLIP-seq datasets, and results reveal that EDCNN provides superior performance to other state-of-the-art methods. Furthermore, time complexity analysis, parameter analysis and motif analysis are conducted to demonstrate the effectiveness of our proposed algorithm from several perspectives. The EDCNN algorithm is available at GitHub: https://github.com/yaweiwang1232/EDCNN. Both the software and the supporting data can be downloaded from: https://figshare.com/articles/software/EDCNN/16803217. Supplementary data are available at Bioinformatics online.

Adaptive Levenberg–Marquardt Algorithm: A New Optimization Strategy for Levenberg–Marquardt Neural Networks

Engineering data are often highly nonlinear and contain high-frequency noise, so the Levenberg–Marquardt (LM) algorithm may not converge when a neural network optimized by the algorithm is trained with engineering data. In this work, we analyzed the reasons for the LM neural network’s poor convergence commonly associated with the LM algorithm. Specifically, the effects of different activation functions such as Sigmoid, Tanh, Rectified Linear Unit (RELU) and Parametric Rectified Linear Unit (PRLU) were evaluated on the general performance of LM neural networks, and special values of LM neural network parameters were found that could make the LM algorithm converge poorly. We proposed an adaptive LM (AdaLM) algorithm to solve the problem of the LM algorithm. The algorithm coordinates the descent direction and the descent step by the iteration number, which can prevent falling into the local minimum value and avoid the influence of the parameter state of LM neural networks. We compared the AdaLM algorithm with the traditional LM algorithm and its variants in terms of accuracy and speed in the context of testing common datasets and aero-engine data, and the results verified the effectiveness of the AdaLM algorithm.

AdaDiff: Adaptive Gradient Descent with the Differential of Gradient

Optimization methods are crucial to train deep neural networks. Adaptive optimization methods, especially Adam, are wildly used because they aren’t sensitive to the selection of learning rate and converge fast. Recent work point out Adam has a performance gap with SGD and even not converge because of the unstable and extreme learning rates. Many variants of Adam are proposed to solve the problem, such as AMSGrad, AdaBound and AdaBelief. In this paper, we propose a new variant of Adam, called AdaDiff. AdaDiff computing gradient descent step size by the exponential moving average(EMA) of gradient and differential of gradients aiming to make the training of networks more stable. We compare our method with other optimizers on various tasks. The results show AdaDiff outperforms Adam and minimizes the performance gap with SGD.

Variable selection for Gaussian process regression through a sparse projection

This article presents a new variable selection approach integrated with Gaussian process regression. We consider a sparse projection of input variables and a general stationary covariance model that depends on the Euclidean distance between the projected features. The sparse projection matrix is considered as an unknown parameter. We propose a forward stagewise approach with embedded gradient descent steps to co-optimize the parameter with other covariance parameters based on the maximization of a non-convex marginal likelihood function with a concave sparsity penalty, and some convergence properties of the algorithm are provided. The proposed model covers a broader class of stationary covariance functions than the existing automatic relevance determination approaches, and the solution approach is more computationally feasible than the existing Markov chain Monte Carlo sampling procedures for the automatic relevance parameter estimation with a sparsity prior. The approach is evaluated for a large number of simulated scenarios. The choice of tuning parameters and the accuracy of the parameter estimation are evaluated with the simulation study. In comparison to some chosen benchmark approaches, the proposed approach has demonstrated improved accuracy in the variable selection. It is applied to an important problem of identifying environmental factors that affect atmospheric corrosion of metal alloys.

Distributed online gradient boosting on data stream over multi-agent networks

In this paper, we study gradient boosting with distributed data streams over multi-agent networks, and propose a distributed online gradient boosting algorithm. Considering limited communication resources and privacy, each node aims to track the minimum of a global, time-varying cost function based on its own data stream and some information of neighbors. We first formulate the global cost function as a sum of local ones, and then convert distributed online gradient boosting into a distributed online optimization problem. At each time step, the local model is updated by a gradient descent step based on the current data, followed by a consensus step with the neighbors. Then, we use a dynamic regret to measure the performance of the proposed algorithm, and prove that the regret has an O(T) bound. Simulations with some practical datasets illustrate the performance of the proposed algorithm.

Learning to Purify Noisy Labels via Meta Soft Label Corrector

Recent deep neural networks (DNNs) can easily overfit to biased training data with noisy labels. Label correction strategy is commonly used to alleviate this issue by identifying suspected noisy labels and then correcting them. Current approaches to correcting corrupted labels usually need manually pre-defined label correction rules, which makes it hard to apply in practice due to the large variations of such manual strategies with respect to different problems. To address this issue, we propose a meta-learning model, aiming at attaining an automatic scheme which can estimate soft labels through meta-gradient descent step under the guidance of a small amount of noise-free meta data. By viewing the label correction procedure as a meta-process and using a meta-learner to automatically correct labels, our method can adaptively obtain rectified soft labels gradually in iteration according to current training problems. Besides, our method is model-agnostic and can be combined with any other existing classification models with ease to make it available to noisy label cases. Comprehensive experiments substantiate the superiority of our method in both synthetic and real-world problems with noisy labels compared with current state-of-the-art label correction strategies.