Mini-batch Gradient Descent Research Articles

Optimize TSK Fuzzy Systems for Regression Problems: Minibatch Gradient Descent With Regularization, DropRule, and AdaBound (MBGD-RDA)

Takagi–Sugeno–Kang (TSK) fuzzy systems are very useful machine learning models for regression problems. However, to our knowledge, there has not existed an efficient and effective training algorithm that ensures their generalization performance and also enables them to deal with big data. Inspired by the connections between TSK fuzzy systems and neural networks, we extend three powerful neural network optimization techniques, i.e., minibatch gradient descent (MBGD), regularization, and AdaBound, to TSK fuzzy systems, and also propose three novel techniques (DropRule, DropMF, and DropMembership) specifically for training TSK fuzzy systems. Our final algorithm, MBGD with regularization, DropRule, and AdaBound, can achieve fast convergence in training TSK fuzzy systems, and also superior generalization performance in testing. It can be used for training TSK fuzzy systems on datasets of any size; however, it is particularly useful for big datasets, on which currently no other efficient training algorithms exist.

Deep nonsmooth nonnegative matrix factorization network with semi-supervised learning for SAR image change detection

In the paper, we propose a deep nonsmooth nonnegative matrix factorization (nsNMF) network with semi-supervised learning for synthetic aperture radar (SAR) image change detection. In most of the existing deep-NMF-based models, the nonnegative matrix is linearly decomposed layer by layer, which may fail to characterize the nonlinearities in complex data. As such, a nonlinear deep nsNMF model is first built for learning hierarchical, nonlinear, and localized data representations. Meanwhile, in view of its good generalization performance and low computational complexity, extreme learning machine (ELM) is integrated into the nonlinear deep nsNMF model to construct a deep nsNMF network for satisfactory classification. More importantly, since it is difficult to acquire more labeled samples in practice, semi-supervised learning strategy is proposed to make use of partially labeled data for training. The learning process of the proposed network consists of pretraining stage and fine-tuning stage, in which the former pretrains all decomposed matrices layer by layer and the latter aims to reduce the total reconstruction error by using the mini-batch gradient descent algorithm. The experimental results on four pairs of SAR images demonstrate the effectiveness of the proposed method.

Batching Soft IoU for Training Semantic Segmentation Networks

The majority of semantic segmentation networks generally employ cross-entropy as a loss function and intersection-over-union (IoU) as the evaluation metric for network performance. Employing IoU as a loss function can solve the mismatch issue between the loss function and the evaluation metric. We propose a Soft IoU training strategy based on mini-batch (mini-batch Soft IoU). Our work has two primary contributions: The first is to extend the IoU loss function to a multi-class segmentation network. The second is to collect various categories of the training samples in every mini-batch, which will ensure that the number of categories equals to the batch size at least. Our method breaks the randomness of the original mini-batch gradient descent (GD) strategy, advancing training samples in the mini-batch much more consistent with the distribution characteristics of the overall data. It solves the instability of IoU loss function. In addition, the experimental results on the PASCAL VOC2012 dataset reveal that our method effectively improves the segmentation accuracy of the network and attains significant improvements beyond state-of-the-art IoU loss function methods.

Development of a denoising convolutional neural network-based algorithm for metal artifact reduction in digital tomosynthesis for arthroplasty: A phantom study.

The present study aimed to develop a denoising convolutional neural network metal artifact reduction hybrid reconstruction (DnCNN-MARHR) algorithm for decreasing metal objects in digital tomosynthesis (DT) for arthroplasty by using projection data. For metal artifact reduction (MAR), we implemented a DnCNN-MARHR algorithm based on a training network (mini-batch stochastic gradient descent algorithm with momentum) to estimate the residual reference (140 keV virtual monochromatic [VM]) and object (70 kV with metal artifacts) images. For this, we used projection data and subtracted the estimated residual images from the object images, involving hybrid and subjectively reconstructed image usage (back projection and maximum likelihood expectation maximization [MLEM]). The DnCNN-MARHR algorithm was compared with the dual-energy material decomposition reconstruction algorithm (DEMDRA), VM, MLEM, established and commonly used filtered back projection (FBP), and a simultaneous algebraic reconstruction technique-total variation (SART-TV) with MAR processing. MAR was compared using artifact index (AI) and texture analysis. Artifact spread functions (ASFs) for images that were out-of-plane and in-focus were evaluated using a prosthesis phantom. The overall performance of the DnCNN-MARHR algorithm was adequate with regard to the ASF, and the derived images showed better results, without being influenced by the metal type (AI was almost equal to the best value for the DEMDRA). In the ASF analysis, the DnCNN-MARHR algorithm generated better MAR compared with that obtained employing usual algorithms for reconstruction using MAR processing. In addition, comparison of the difference (mean square error) between DnCNN-MARHR and the conventional algorithm resulted in the smallest VM. The DnCNN-MARHR algorithm showed the best performance with regard to image homogeneity in the texture analysis. The proposed algorithm is particularly useful for reducing artifacts in the longitudinal direction, and it is not affected by tissue misclassification.

Barzilai–Borwein-based adaptive learning rate for deep learning

Learning rate is arguably the most important hyper-parameter to tune when training a neural network. As manually setting right learning rate remains a cumbersome process, adaptive learning rate algorithms aim at automating such a process. Motivated by the success of the Barzilai–Borwein (BB) step-size method in many gradient descent methods for solving convex problems, this paper aims at investigating the potential of the BB method for training neural networks. With strong motivation from related convergence analysis, the BB method is generalized to adaptive learning rate of mini-batch gradient descent. The experiments showed that, in contrast to many existing methods, the proposed BB method is highly insensitive to initial learning rate, especially in terms of generalization performance. Also, the BB method showed its advantages on both learning speed and generalization performance over other available methods.

Brain-vascular segmentation for SEEG planning via a 3D fully-convolutional neural network.

Three dimensional visualization of vascular structures can assist clinicians in preoperative planning, intra-operative guidance, and post-operative decision-making. The goal of this work is to provide an automatic, accurate and fast method for brain vessels segmentation in Contrast Enhanced Cone Beam Computed Tomography (CE-CBCT) dataset based on a residual Fully Convolutional Neural Network (FCNN). The proposed NN embeds in an encoder-decoder architecture residual elements which decreases the vanishing effect due to deep architecture while accelerating the convergence. Moreover, a two-stage training has been proposed as a countermeasure for the unbalanced nature of the dataset. The FCNN training was performed on 20 CE-CBCT volumes exploiting mini-batch gradient descent and the Adam optimizer. Binary cross-entropy was used as loss function. Performance evaluation was conducted considering 5 datasets. A median value of Dice, Precision and Recall of 0.79, 0.8 and 0.69 were obtained with respect to manual annotations.

Multi-Objective Defocus Robust Source and Mask Optimization Using Sensitive Penalty

The continuous decrease in the size of lithographic technology nodes has led to the development of source and mask optimization (SMO) and also to the control of defocus becoming stringent in the actual lithography process. Due to multi-factor impact, defocusing is always changeable and uncertain in the real exposure process. But conventional SMO assumes the lithography system is ideal, which only compensates the optical proximity effect (OPE) in the best focus plane. Therefore, to solve the inverse lithography problem with more uniformity of pattern in different defocus variations, we proposed a defocus robust SMO (DRSMO) approach that is driven by a defocus sensitivity penalty function for the first time. This multi-objective optimization samples a wide range of defocus disturbances and it can be proceeded by the mini-batch gradient descent (MBGD) algorithm effectively. The simulation results showed that a more robust defocus source and mask can be designed through DRSMO optimization. The defocus sensitivity factor sβ maximally decreased 63.5% compared to conventional SMO, and due to the low error sensitivity and the depth of defocus (DOF), the process window (PW) was further enlarged effectively. Compared to conventional SMO, the exposure latitude (EL) maximally increased from 4.5% to 10.5% and DOF maximally increased 54.5% (EL = 5%), which proved the validity of the DRSMO method in improving the focusing performance.

Similarity learning for person reidentification using projected gradient method

The problem of learning a similarity measure for cross-view person identification in a disjointed camera network is addressed. Our learning framework is based on the projected gradient approach and is suitable for large-scale applications. The hinge loss with the triplet constraints is used as the objective function. Contrary to other approaches where a dual formulation is used, we propose to optimize the objective function in the primal form, with the help of a minibatch gradient descent algorithm. The choice of learning rate and its schedule are nontrivial tasks in such optimization schemes. We have studied empirically the effect of using different strategies for learning rate schedules. Our experimental study includes conventional strategies such as fixed or diminishing learning rates and recently developed adaptive gradient methods such as root-mean-square propagation (RMSProp) for learning rate schedule. The experimental results are presented on three benchmark datasets of person reidentification, namely VIPeR, CUHK Campus, and CUHK03. Our experimental results demonstrate that the RMSProp is well suited for the proposed learning framework. A comparison with recent methods in the domain presents the justification of the proposed approach.

Clinical validation of fully automated segmentation of thigh muscle and adipose tissue cross sectional areas using maching learning with a convolutional neural network

Clinical validation of fully automated segmentation of thigh muscle and adipose tissue cross sectional areas using maching learning with a convolutional neural network

Linked Data Crowdsourcing Quality Assessment based on Domain Professionalism

With the rapid development of Internet technology, crowdsourcing, as a flexible, effective and low-cost problem-solving method, has begun to receive more and more attention. The use of crowdsourcing to evaluate the quality of linked data has also become a research hotspot. This paper proposes the concept of Domain Specialization Test (DST), which uses domain professional testing tasks DSTs to evaluate the professionalism of workers, and combines the idea of Mini-batch Gradient Descent (MBGD) to improve the EM algorithm, and the MBEM algorithm is proposed to achieve efficient and accurate evaluation of task results. The experimental results show that the proposed method can screen out the appropriate workers for the linked data crowdsourcing task and improve the accuracy and iteration efficiency of the results.

FCNN-based axon segmentation for convection-enhanced delivery optimization.

Glioblastoma multiforme treatment is a challenging task in clinical oncology. Convection- enhanced delivery (CED) is showing encouraging but still suboptimal results due to drug leakages. Numerical models can predict drug distribution within the brain, but require retrieving brain physical properties, such as the axon diameter distribution (ADD), through axon architecture analysis. The goal of this work was to provide an automatic, accurate and fast method for axon segmentation in electronic microscopy images based on fully convolutional neural network (FCNN) as to allow automatic ADD computation. The segmentation was performed using a residual FCNN inspired by U-Net and Resnet. The FCNN training was performed exploiting mini-batch gradient descent and the Adam optimizer. The Dice coefficient was chosen as loss function. The proposed segmentation method achieved results comparable with already existing methods for axon segmentation in terms of Information Theoretic Scoring ([Formula: see text]) with a faster training (5h on the deployed GPU) and without requiring heavy post-processing (testing time was 0.2s with a non-optimized code). The ADDs computed from the segmented and ground-truth images were statistically equivalent. The algorithm proposed in this work allowed fast and accurate axon segmentation and ADD computation, showing promising performance for brain microstructure analysis for CED delivery optimization.

Aero-Engine On-Board Model Based on Batch Normalize Deep Neural Network

A new on-board turbo-fan engine modeling method based on a batch normalize (BN) mini-batch gradient descent (MGD) deep neural network (NN) is proposed. This new method adopts BN algorithm, which accelerates the network training speed and overcomes the gradient vanish problem. Hence, using the BN algorithm, the neural network adopts the deeper structure, which means the network has a stronger representation capacity. This mini-batch gradient descent (MGD-NN) algorithm that consumes much less time to update the NN parameters is adopted. Therefore, it is more suitable for training big dataset and establishing a high-accuracy engine model in a large flight envelope. Finally, to verify whether the proposed method could be applied to larger flight envelope, the conventional NN also adopts MGD (called MGD-NN). The turbo-fan engine models based on these two modeling methods are both conducted within a sub-sonic cruise envelope. The simulation results show that the proposed modeling method has much higher accuracy than the MGD-NN. Moreover, the proposed method has the characteristics of less data storage, low computation complexity, and good real-time performance, which are the most importance indices for model realize on-board.

Accurate Underwater ATR in Forward-Looking Sonar Imagery Using Deep Convolutional Neural Networks

Underwater automatic target recognition (ATR) is a challenging task for marine robots due to the complex environment. The existing recognition methods basically use hand-crafted features and classifiers to recognize targets, which are difficult to achieve ideal recognition accuracy. In this paper, we proposed a novel method to realize accurate multiclass underwater ATR by using forward-looking sonar—Echoscope and deep convolutional neural networks (DCNNs). A complete recognition process from data preprocessing to network training and image recognition was realized. Firstly, we established a real, measured Echoscope sonar image dataset. Inspired by the human visual attention mechanism, the suspected target region was extracted via the graph-based manifold ranking method in image preprocessing. Secondly, an end-to-end DCNNs model, named EchoNet, was designed for Echoscope sonar image feature extraction and recognition. Finally, a network training method based on transfer learning was developed to solve the problem of insufficient training data, and mini-batch gradient descent was used for network optimization. Experimental results demonstrated that our method can implement efficiently, and the recognition accuracy on a nine-class underwater ATR task reached 97.3%, outperforming traditional feature-based methods. The proposed method is expected to be a potential novel technology for the intelligent perception of autonomous underwater vehicles.

Differential Privacy for Data and Model Publishing of Medical Data

Combining medical data and machine learning has fully utilized the value of medical data. However, medical data contain a large amount of sensitive information, and the inappropriate handling of data can lead to the leakage of personal privacy. Thus, both publishing data and training data in machine learning may reveal the privacy of patients. To address the above issue, we propose two effective approaches. One combines a differential privacy and decision tree (DPDT) approach to provide strong privacy guarantees for publishing data, which establishes a weight calculation system based on the classification and regression tree (CART) method and takes weights as a new element of differential privacy to participate in privacy protection and reduce the negative impact of differential privacy on data availability. Another uses the differentially private mini-batch gradient descent algorithm (DPMB) to provide strong protection for training data; it tracks the privacy loss and allows the model to satisfy differential privacy in the process of gradient descent to prevent attackers from invading personal privacy with the training data. It is worth mentioning that, in this paper, we adopt the data processed by DPDT as the training data of DPMB to further strengthen the privacy of data.

The improvement of classification accuracy with denoising class autoencoder

Denoising autoencoder is a data compression algorithm in deep learning, and it can successfully extract features which contain main information of the input data. However, this method is trained in an unsupervised manner and cannot aim at extracting the specified information. Therefore, the input data will be classified into a class label by main but not necessary information using traditional denoising autoencoder, and redundant information reduces the classification accuracy. To solve this problem, denoising class autoencoder presented in this paper is adopted to train a classification function and extract class features. Both the proposed approach and traditional methods are trained on MNIST handwritten digit database with mini-batch gradient descent, moreover the recognition result of the proposed approach demonstrates the effectiveness in comparison with conventional methods.

Explore a deep learning multi-output neural network for regional multi-step-ahead air quality forecasts

Timely regional air quality forecasting in a city is crucial and beneficial for supporting environmental management decisions as well as averting serious accidents caused by air pollution. Artificial Intelligence-based models have been widely used in air quality forecasting. The Shallow Multi-output Long Short-Term Memory (SM-LSTM) model is suitable for regional multi-step-ahead air quality forecasting, while it commonly encounters spatio-temporal instabilities and time-lag effects. To overcome these bottlenecks and overfitting issues, this study proposed a Deep Multi-output LSTM (DM-LSTM) neural network model that were incorporated with three deep learning algorithms (i.e., mini-batch gradient descent, dropout neuron and L2 regularization) to configure the model for extracting the key factors of complex spatio-temporal relations as well as reducing error accumulation and propagation in multi-step-ahead air quality forecasting. The proposed DM-LSTM model was evaluated by three time series of PM2.5, PM10, and NOx simultaneously at five air quality monitoring stations in Taipei City of Taiwan. Results indicated that the loss function values (mean-square-error) of the SM-LSTM and DM-LSTM models in the testing stages at horizon t+4 were 0.87 and 0.72, respectively. The Gbench values of the DM-LSTM model in the testing stages for PM2.5, PM10, and NOx reached 0.95 at horizon t+1 and exceeded 0.81 at horizon t+4, respectively. Results demonstrated that the proposed DM-LSTM model incorporated with three deep learning algorithms could significantly improve the spatio-temporal stability and accuracy of regional multi-step-ahead air quality forecasts.

A nonlinear ANN equalizer with mini-batch gradient descent in 40Gbaud PAM-8 IM/DD system

Abstract We propose and validate a novel nonlinear artificial neural network (ANN) equalizer for PAM-8 transmission in IM/DD system. Mini-batch gradient descent is introduced to efficiently train ANN equalizer. Using the proposed ANN equalizer, we successfully transmit a 40Gbaud PAM-8 signal over 4-km SMF with BER under the threshold of 3.8 × 10−3 and over 10-km SMF with BER under the threshold of 1 × 10−2. We also elaborately compare the proposed ANN equalizer with other methods including LMS equalizer, Volterra equalizer and look-up table (LUT). Experimental results indicate that ANN achieves the best performance that is slightly superior to Volterra equalizer with computational complexity half reduced. To the best of our knowledge, this is the first time to adopt mini-batch gradient descent to train ANN equalizer in IM/DD system.

Prediction of PM2.5 using an ensemble of artificial neural networks and regression models

Inhaling particulate matter such as PM2.5 can have a hazardous impact on the human health. In order to predict the PM2.5 concentration, Artificial Neural Networks trained with conjugate gradient descent such as Multi Layer Perceptron (MLP), cascade forward neural network, Elman neural network, Radial Basis Function (RBF) neural network and Non-linear Autoregressive model with exogenous input (NARX) along with regression models such as Multiple Linear Regression (MLR) consisting of batch gradient descent, stochastic gradient descent, mini-batch gradient descent and conjugate gradient descent algorithms and Support Vector Regression (SVR) were implemented. In these models, the concentration of PM2.5 was the dependent variable and the data related to concentrations of PM2.5, SO2, O3 and meteorological data including average Maximum Temperature (MAX T), daily wind speed (WS) for the years 2010–2016 in Houston and New York were the independent variables. For the final forecast, several ensemble models of individual neural network predictors and individual regression predictors have been presented.

Mini-batch optimized full waveform inversion with geological constrained gradient filtering

High computation cost and generating solutions without geological sense have hindered the wide application of Full Waveform Inversion (FWI). Source encoding technique is a way to dramatically reduce the cost of FWI but subject to fix-spread acquisition setup requirement and slow convergence for the suppression of cross-talk. Traditionally, gradient regularization or preconditioning is applied to mitigate the ill-posedness. An isotropic smoothing filter applied on gradients generally gives non-geological inversion results, and could also introduce artifacts. In this work, we propose to address both the efficiency and ill-posedness of FWI by a geological constrained mini-batch gradient optimization method. The mini-batch gradient descent optimization is adopted to reduce the computation time by choosing a subset of entire shots for each iteration. By jointly applying the structure-oriented smoothing to the mini-batch gradient, the inversion converges faster and gives results with more geological meaning. Stylized Marmousi model is used to show the performance of the proposed method on realistic synthetic model.

Unmixing of large-scale hyperspectral data based on projected mini-batch gradient descent

The minimization problem of reconstruction error over large hyperspectral image data is one of the most important problems in unsupervised hyperspectral unmixing. A variety of algorithms based on nonnegative matrix factorization (NMF) have been proposed in the literature to solve this minimization problem. One popular optimization method for NMF is the projected gradient descent (PGD). However, as the algorithm must compute the full gradient on the entire dataset at every iteration, the PGD suffers from high computational cost in the large-scale real hyperspectral image. In this paper, we try to alleviate this problem by introducing a mini-batch gradient descent-based algorithm, which has been widely used in large-scale machine learning. In our method, the endmember can be updated pixel set by pixel set while abundance can be updated band set by band set. Thus, the computational cost is lowered to a certain extent. The performance of the proposed algorithm is quantified in the experiment on synthetic and real data.