Gradient Descent Optimization Method Research Articles

NMF-DuNet: Nonnegative Matrix Factorization Inspired Deep Unrolling Networks for Hyperspectral and Multispectral Image Fusion

The fusion of high-resolution multispectral (HrMSI) and low-resolution hyperspectral images (LrHSI) has been acknowledged as a promising method for generating a high-resolution hyperspectral image (HrHSI), which is also termed to be an essential part for precise recognition and cataloguing of the underlying materials. In order to improve the fusion of LrHSI and HrMSI performance, in this article, we propose a novel Nonnegative Matrix Factorization Inspired Deep Unrolling Networks, dubbed NMF-DuNet, for fusing LrHSI and HrMSI. For this aim, initially, a variational fusion model regularized by non-negative sparse prior is proposed and then is solved through the gradient descent optimization method and unrolled towards the deep network. The nonnegative coefficient matrices and orthogonal of the proposed transform coefficients constraints are both incorporated into the proposed method. Moreover, the fusion of HrMSI and LrHSI heavily depends on an imaging model that explains the degeneracy of HSI in the spectral and spatial regions. Practically, the imaging model is often unknown. The degradation model is represented implicitly via a proposed network, and both the degradation model and sparse priors are jointly optimized through the training process of the proposed network. Instead of being hand-crafted, all the parameters of NMF-DuNet are learned end-to-end. Compared to the previous state-of-the-art model-based and learning-based fusion approaches, the hardware-friendly proposed NMF-DuNet outperforms both the model-based and learning-based fusion approaches and requires a far smaller number of trainable parameters and storage space while preserving real-time performance.

Gaussian process latent variable model factorization for context-aware recommender systems

• We propose a novel algorithm named GPLVMF to achieve a Bayesian factorization for Context-aware recommendation. • We model the non-zero mean in Gaussian processes to capture the bias and provide a generalized variational inference. • Our method can flexibly deal with both real-valued and categorical contexts. • Both scaled conjugate gradient and stochastic gradient descent optimization methods are applied to the model. • Experiment results show that GPLVMF not only improves accuracy but can automatically infer the influence of context. Context-aware recommender systems (CARS) have gained increasing attention due to their ability to utilize contextual information. Compared to traditional recommender systems, CARS are, in general, able to generate more accurate recommendations. The latent factors approach accounts for a large proportion of CARS. Recently, a non-linear Gaussian Process (GP) based factorization method was proven to outperform the state-of-the-art methods in CARS. Despite its effectiveness, standard GP model-based methods can suffer from over-fitting and may not be able to determine the impact of each context automatically. In order to address such shortcomings, we propose a Gaussian Process Latent Variable Model Factorization (GPLVMF) method, where we apply an appropriate prior to the original GP model. Our work is primarily inspired by the Gaussian Process Latent Variable Model (GPLVM), which is a non-linear dimensionality reduction method. Since the traditional maximum likelihood approach for standard GP cannot be applied to our model, we adopt a variational inference method to solve the GPLVMF model. As a result, we improve the results on the real datasets significantly as well as capturing the importance of each context. In addition to the general advantages, our method flexibly provides two contributions regarding recommender system settings: (1) addressing the influence of bias by setting a non-zero mean function, and (2) utilizing real-valued contexts by fixing the latent space with real values.

BALFA: A brain storm optimization-based adaptive latent factor analysis model

Information overload in recent years has tremendously sparked recommender systems (RSs). An RSs usually recommends valuable information for users based on historical experience high-dimensional and incomplete (HDI) data. Since each user cannot mark whole items, extracting latent factors (LF) learned by the stochastic gradient descent (SGD) optimization method is frequently used. However, interference from the adjustment of parameters induces an inferior convergence rate and low efficiency. To address this issue, the paper proposes a brain storm optimization (BSO)-based adaptative latent factor analysis (BALFA) model consisting of the following three essential ideas: 1) divergent mechanism to avoid premature convergence, 2) particle retention technique to prevent invalid search, and 3) self-adaptive multidimensional leaning rate for more efficient application on varying data. Moreover, the convergence of the modified BSO used in BALFA is proofed based on the Markov chain. Comparison experiments on six HDI datasets in the SGD-based LF model indicate that the BALFA achieves state-of-art convergence rate and computational efficiency for HDI data analysis.

Experimental and numerical study of energy and thermal management system for a hydrogen fuel cell-battery hybrid electric vehicle

The aims of this study were fourfold: firstly, develop a fuzzy logic control algorithm for enhancing overall energy efficiency of the fuel cell hybrid electric vehicle, secondly, balance the energy consumptions of the battery and fuel cell with aid of gradient descent optimization method, thirdly, use the waste heat of hydrogen vessels by controlling the fuel flow, and finally, operate two energy sources at cold and hot ambient temperatures safely and efficiently. In the current study, a fuzzy logic controller is used to keep of operate the hydrogen fuel cell and battery pack at their optimum temperatures. A detailed mathematical model is established, which includes electric vehicle dynamics, thermal behavior of battery, hydrogen fuel cell and vessel, and the efficiency model of electric motor. Furthermore, numerical simulations related to vessel temperature changes and energy consumption of the hybrid electric vehicle under actual road conditions are validated by experiments. The results demonstrate that total energy consumption including the propulsion system, thermal management system, and fuel supply system is decreased by 9,1% at 18 °C thanks to the fuzzy logic control algorithm. Besides, energy efficiency is increased around 7% and 11% at −10 °C and 35 °C ambient temperatures, respectively, which are extreme scenarios for electric vehicles. Moreover, the developed algorithm provides a successful thermal management strategy regarding waste heat recovery from hydrogen vessels, the temperature of which is controlled by the fuel supply system.

Uniform error estimates for artificial neural network approximations for heat equations

Abstract Recently, artificial neural networks (ANNs) in conjunction with stochastic gradient descent optimization methods have been employed to approximately compute solutions of possibly rather high-dimensional partial differential equations (PDEs). Very recently, there have also been a number of rigorous mathematical results in the scientific literature, which examine the approximation capabilities of such deep learning-based approximation algorithms for PDEs. These mathematical results from the scientific literature prove in part that algorithms based on ANNs are capable of overcoming the curse of dimensionality in the numerical approximation of high-dimensional PDEs. In these mathematical results from the scientific literature, usually the error between the solution of the PDE and the approximating ANN is measured in the $L^p$-sense, with respect to some $p \in [1,\infty )$ and some probability measure. In many applications it is, however, also important to control the error in a uniform $L^\infty $-sense. The key contribution of the main result of this article is to develop the techniques to obtain error estimates between solutions of PDEs and approximating ANNs in the uniform $L^\infty $-sense. In particular, we prove that the number of parameters of an ANN to uniformly approximate the classical solution of the heat equation in a region $ [a,b]^d $ for a fixed time point $ T \in (0,\infty ) $ grows at most polynomially in the dimension $ d \in {\mathbb {N}} $ and the reciprocal of the approximation precision $ \varepsilon> 0 $. This verifies that ANNs can overcome the curse of dimensionality in the numerical approximation of the heat equation when the error is measured in the uniform $L^\infty $-norm.

Gradient Descent Optimization in Deep Learning Model Training Based on Multistage and Method Combination Strategy

Gradient descent is the core and foundation of neural networks, and gradient descent optimization heuristics have greatly accelerated progress in deep learning. Although these methods are simple and effective, how they work remains unknown. Gradient descent optimization in deep learning has become a hot research topic. Some research efforts have tried to combine multiple methods to assist network training, but these methods seem to be more empirical, without theoretical guides. In this paper, a framework is proposed to illustrate the principle of combining different gradient descent optimization methods by analyzing several adaptive methods and other learning rate methods. Furthermore, inspired by the principle of warmup, CLR, and SGDR, the concept of multistage is introduced into the field of gradient descent optimization, and a gradient descent optimization strategy in deep learning model training based on multistage and method combination strategy is presented. The effectiveness of the proposed strategy is verified on the massive deep learning network training experiments.

Gradient Descent Optimization Based Parameter Identification for FCS-MPC Control of LCL-Type Grid Connected Converter

Aging and temperature changes in the passive components of an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LCL</i> -filter grid connected converter system (GCCs) may lead to parameter uncertainties, which can in turn influence its modeling accuracy for finite-control-set model predictive control (FCS-MPC). The presence of model errors will change the resonance point and deteriorate the power quality of the grid current, in turn degrading the active damping performance. In this situation, there is a serious possibility that the GCCs may malfunction and automatically disconnect from the grid, causing great challenges to the system stability. To solve this problem, first, prediction error analysis in FCS-MPC due to the model parameter errors is presented. Second, to achieve high accuracy and fast filter parameter estimation in utility, an adaptive online parameter identification method based on gradient descent optimization (GDO) has been proposed. Finally, to further reduce the searching time needed by the optimal iteration step, a variable iteration step searching method based on the root-mean-square-prop (RMSprop) GDO method is proposed. Experimental studies of an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LCL</i> -GCCs prototype in the laboratory have been conducted to validate the effectiveness of the proposed method.

U-Net ile Çekirdek Segmentasyonunda Hiper Parametre Optimizasyonu Etkisinin Değerlendirilmesi

Tibbi goruntulerin yorumlanarak hasta ve hastalik hakkinda onemli veriler elde edilmesi zaman ve emek acisindan oldukca maliyetlidir. Tibbi goruntulerin yapay zekâ yontemleri ile analiz edilmesi sayesinde hastalik tespitinin yapilmasi, siniflandirilmasi ve bunlarin otomatiklestirilmesi uzmanlarin is yukunu azaltmaktadir. Bu calismada, Data Science Bowl 2018 veri setinden elde edilen tibbi goruntulerdeki cekirdeklerin tespitinin otomatiklestirilmesi yapilmaktadir. Data Science Bowl 2018, cekirdek tespitinin otomatiklestirilmesi amaciyla bir yarisma icin bir araya getirilmis tibbi goruntulerden olusmaktadir. Bu veri seti icerisinde 670 adet egitim goruntusu ve bunlara ait maske goruntuleri ile 65 adet test goruntusu yer almaktadir. Test goruntulerine ait maske goruntuleri olmadigi icin calisma, egitim goruntulerinin %10’unun dogrulama verisi olarak ayrilarak bu veriler uzerinden yapilmaktadir. Calismada, cekirdek tespitinin otomatiklestirilmesinde U-Net Evrisimli Sinir Agi kullanilip en yuksek dogruluk ve en dusuk hata oraninin elde edilmesi amaclanmistir. Cekirdek tespitindeki dogruluk oraninin arttirilmasi ve hata oraninin minimize edilmesi amaclandigi icin, yapilan deneysel calismalar iki bolume ayrilmaktadir. Ilk olarak Adam optimizasyonu ve belirli hiper parametreler kullanilmakta, ikinci bolumde ise sonuclarin en iyilenmesi icin hiper parametre optimizasyonu yapilmaktadir. Hiper parametre optimizasyonu icin Izgara Arama yontemi tercih edilmistir. Deneysel calismalarin ilk asamasinda dogrulama verileri icin elde edilen dogruluk orani 0.9675 olarak bulunmustur. Hiper parametre optimizasyonu yapildiktan sonra ise dogrulama verileri icin en iyi iki sonuc sirasi ile 0.9698 ve 0.9739 olarak bulunmustur. Son olarak, en iyi sonucu veren 0.9739 orani icin belirlenen hiper parametreler Stokastik Gradyan Inis, RMSProp ve AdaDelta optimizasyon yontemleri icin de denenmistir. Bu sayede optimizasyon yontemlerin kiyaslamasi da yapilmis ve en iyi sonucu veren optimizasyon yonteminin Adam optimizasyonu oldugu belirlenmistir.

Pairwise Gaussian Loss for Convolutional Neural Networks

Convolutional neural networks (CNNs) have demonstrated great competence in feature representation, and then, achieved a good performance to many classification tasks. Cross-entropy loss, together with softmax, is arguably one of the most commonly used loss functions in CNNs (that is generally called softmax loss ). However, the softmax loss can result in a weakly discriminative feature representation since it focuses on the interclass separability rather than the intraclass compactness. This article proposes a pairwise Gaussian loss (PGL) for CNNs that can well address the intraclass compactness through significantly penalizing those similar sample pairs with a relatively large distance. At the same time, PGL can still ensure a good interclass separability. Experiments show that PGL can guarantee that CNNs obtain a better classification performance compared to not only the softmax loss but also others often used in CNNs. Our experiments also show that PGL has a stable convergence for the stochastic gradient descent optimization method in CNNs and a good generalization ability for different structures of CNNs.

Accelerating the Training Process of Convolutional Neural Networks for Image Classification by Dropping Training Samples Out

Stochastic gradient descent and other adaptive optimization methods have been proved effective for training deep neural networks. Within each epoch of these methods, the whole training set is involved to train the model. In general, large training data sets have data redundancy among their training samples. In this paper, we present an algorithm to reduce the training time consumption of CNN by dropping certain samples out, which is called the greedy DropSample. For the absence of certain training samples, the distribution of networks’ activations is biased during training. By correcting the mean and variance of batch-normalization layers, this issue is solved. Experimental results over several data sets demonstrate the efficiency of the proposed method. The results show that this method could decrease the training time of multilayer perceptrons (MLPs) and convolutional neural networks (CNNs) significantly. Despite the reduced number of training samples, the accuracies of networks are similar, or even better.

The Cauchy Conjugate Gradient Algorithm with Random Fourier Features

Random Fourier mapping (RFM) in kernel adaptive filters (KAFs) provides an efficient method to curb the linear growth of the dictionary by projecting the original input data into a finite-dimensional space. The commonly used measure in RFM-based KAFs is the minimum mean square error (MMSE), which causes performance deterioration in the presence of non-Gaussian noises. To address this issue, the minimum Cauchy loss (MCL) criterion has been successfully applied for combating non-Gaussian noises in KAFs. However, these KAFs using the well-known stochastic gradient descent (SGD) optimization method may suffer from slow convergence rate and low filtering accuracy. To this end, we propose a novel robust random Fourier features Cauchy conjugate gradient (RFFCCG) algorithm using the conjugate gradient (CG) optimization method in this paper. The proposed RFFCCG algorithm with low complexity can achieve better filtering performance than the KAFs with sparsification, such as the kernel recursive maximum correntropy algorithm with novelty criterion (KRMC-NC), in stationary and non-stationary environments. Monte Carlo simulations conducted in the time-series prediction and nonlinear system identification confirm the superiorities of the proposed algorithm.

Impact of Initialization on Gradient Descent Method in Localization Using Received Signal Strength

In this article we present a localization technique based on received signal strength (RSS) combined with the gradient descent optimization method. The goal of this article is to show the importance of gradient descent in localization domain over the trilateration technique, and that by reducing the number of needed anchor nodes. Furthermore, we demonstrate the effect of the initialization technique on the localization accuracy. Results have shown that the selection of the initialization type (4 types of initialization were tested) has an efficient impact on the accuracy of the target sensors location estimation.

Online Reconfiguration of a Variable-Stiffness Actuator

This paper presents an algorithm to adjust the configuration of a variable-stiffness actuator to reduce its electrical energy requirements during the execution of repetitive tasks. The algorithm is based on the gradient descent optimization method with a modification of the adaptation step size, a forgetting term, and a projection rule to cope with the variation of the actual objective function and signal noise. The performance of the algorithm is validated with experimental results that confirm its capability to reduce the energy requirements of the actuator's driving mechanism. Simulation results illustrate the use of the algorithm in a two-degree-of-freedom system.

A Discrete Version of CMA-ES

Modern machine learning uses more and more advanced optimization techniques to find optimal hyper parameters. Whenever the objective function is non-convex, non continuous and with potentially multiple local minima, standard gradient descent optimization methods fail. A last resource and very different method is to assume that the optimum(s), not necessarily unique, is/are distributed according to a distribution and iteratively to adapt the distribution according to tested points. These strategies originated in the early 1960s, named Evolution Strategy (ES) have culminated with the CMA-ES (Covariance Matrix Adaptation) ES. It relies on a multi variate normal distribution and is supposed to be state of the art for general optimization program. However, it is far from being optimal for discrete variables. In this paper, we extend the method to multivariate binomial correlated distributions. For such a distribution, we show that it shares similar features to the multi variate normal: independence and correlation is equivalent and correlation is efficiently modeled by interaction between different variables. We discuss this distribution in the framework of the exponential family. We prove that the model can estimate not only pairwise interactions among the two variables but also is capable of modeling higher order interactions. This allows creating a version of CMA-ES that can accommodate efficiently discrete variables. We provide the corresponding algorithm and conclude.

Deep Learning for Passive Synthetic Aperture Radar

We introduce a deep learning (DL) framework for inverse problems in imaging, and demonstrate the advantages and applicability of this approach in passive synthetic aperture radar (SAR) image reconstruction. We interpret image recon- struction as a machine learning task and utilize deep networks as forward and inverse solvers for imaging. Specifically, we design a recurrent neural network (RNN) architecture as an inverse solver based on the iterations of proximal gradient descent optimization methods. We further adapt the RNN architecture to image reconstruction problems by transforming the network into a recurrent auto-encoder, thereby allowing for unsupervised training. Our DL based inverse solver is particularly suitable for a class of image formation problems in which the forward model is only partially known. The ability to learn forward models and hyper parameters combined with unsupervised training approach establish our recurrent auto-encoder suitable for real world applications. We demonstrate the performance of our method in passive SAR image reconstruction. In this regime a source of opportunity, with unknown location and transmitted waveform, is used to illuminate a scene of interest. We investigate recurrent auto- encoder architecture based on the 1 and 0 constrained least- squares problem. We present a projected stochastic gradient descent based training scheme which incorporates constraints of the unknown model parameters. We demonstrate through extensive numerical simulations that our DL based approach out performs conventional sparse coding methods in terms of computation and reconstructed image quality, specifically, when no information about the transmitter is available.

Modeling and analysis of dielectric materials by using gradient-descent optimization method

This study presents a numerical method based on parallel RC equivalent circuit model fitting methodology for analysis and modeling of dielectric materials. The proposed method employs gradient-descent optimization method (GDOM) to estimate parallel RC equivalent circuit model from current waveforms by minimizing sum of squared difference (SSD) error. Estimation of parallel RC equivalent circuit parameters from measured current waveforms provides a useful tool for identification, simulation and analysis of dielectric materials. Moreover, applications of the proposed method for time and frequency analyses of dielectric materials are demonstrated numerically. Numerical simulations were presented to discuss efficiency of the proposed method for modeling, analysis and monitoring of insulator materials.

Non-rigid MR-TRUS image registration for image-guided prostate biopsy using correlation ratio-based mutual information

BackgroundTo improve the accuracy of ultrasound-guided biopsy of the prostate, the non-rigid registration of magnetic resonance (MR) images onto transrectal ultrasound (TRUS) images has gained increasing attention. Mutual information (MI) is a widely used similarity criterion in MR-TRUS image registration. However, the use of MI has been challenged because of intensity distortion, noise and down-sampling. Hence, we need to improve the MI measure to get better registration effect.MethodsWe present a novel two-dimensional non-rigid MR-TRUS registration algorithm that uses correlation ratio-based mutual information (CRMI) as the similarity criterion. CRMI includes a functional mapping of intensity values on the basis of a generalized version of intensity class correspondence. We also analytically acquire the derivative of CRMI with respect to deformation parameters. Furthermore, we propose an improved stochastic gradient descent (ISGD) optimization method based on the Metropolis acceptance criteria to improve the global optimization ability and decrease the registration time.ResultsThe performance of the proposed method is tested on synthetic images and 12 pairs of clinical prostate TRUS and MR images. By comparing label map registration frame (LMRF) and conditional mutual information (CMI), the proposed algorithm has a significant improvement in the average values of Hausdorff distance and target registration error. Although the average Dice Similarity coefficient is not significantly better than CMI, it still has a crucial increase over LMRF. The average computation time consumed by the proposed method is similar to LMRF, which is 16 times less than CMI.ConclusionWith more accurate matching performance and lower sensitivity to noise and down-sampling, the proposed algorithm of minimizing CRMI by ISGD is more robust and has the potential for use in aligning TRUS and MR images for needle biopsy.

A Content Sharing and Discovery Framework Based on Semantic and Geographic Partitioning for Vehicular Networks

Recently P2P networks and theirs applications have become increasingly popular. On the other hand, considering ever increasing industrial and scholarly popularity of Vehicular Ad hoc Networks (VANETs), implementation P2P network over VANET has attracted attentions recently. One of the most important applications in P2P networks is content discovery. Regarding the difficulties of structured and unstructured protocols over VANET, this paper presents a new framework for sharing and content discovery which formed a structured overlay to overcome problems likes broadcasting storm which is the main problem of unstructured methods. On the other hand, this paper tries to solve the instability of structured overlay networks as the main problem of them, by applying geographical and semantic partitioning. Simulation results clarified higher performance of proposed framework in comparison to previous protocols. Furthermore, applying G-Network queue network, we have modeled the behavior of proposed framework and then, optimize it by gradient descent optimization method.

Nonlinear Fuzzy Model-base Technique to Compensate Highly Nonlinear Continuum Robot Manipulator

Refer to this research, a gradient descent optimization methodology for position fuzzy- model based computed torque controller (GDFCTC) is proposed for highly nonlinear continuum robot manipulator. The main problem of the pure computed torque controller (CTC) was equivalent problem in uncertain systems. The simulation results exhibit that the CTC works well in certain system. To eliminate the continuum robot manipulator system's dynamic; Mamdani fuzzy inference system is design and applied to CTC. This methodology is based on applied fuzzy logic in equivalent nonlinear dynamic part to estimate unknown parameters. This relatively controller is more plausible to implement in an actual real-time when compared to other techniques of nonlinear controller methodology of continuum arms. Based on the gradient descent optimization method, the PD-gain updating factor has been developed in certain and partly uncertain continuum robots. The new techniques proposed and methodologies adopted in this paper supported by MATLAB/SIMULINK results represent a significant contribution to the field of design an optimized nonlinear computed torque controller for continuum robots.

Variable Structure System with Sliding Mode Controller

This paper describes the application of sliding mode variable structure control (SMVSC) concept for DC motor position control. A mathematical procedure for determining the values of the gains of variable structure system (VSS) controller (a, P) and the slope of switching line (c) are mentioned and obtained mathematically but optimized values, also, achieved by using MATLAB/Simulink design optimization toolbox - signal constraint by applying gradient descent optimization method. The proposed control scheme is derived, simulated and tested thoroughly by using MATLAB/Simulink package software Ver. 7.10/Version 7.5 (R2010a) respectively. The control-system schematics are given and sample results are shown for three types of controllers (Proportional (P), three terms PID and sliding mode variable structure controllers). The SMVSC simulation should be of interest to design engineers who wish to demonstrate and investigate sophisticated position-control methods and their applications. This simulation may also serve as a basis for further applications of SMVSC in other engineering fields.