Stochastic Gradient Descent Optimization Algorithm Research Articles

An Improved Reacceleration Optimization Algorithm Based on the Momentum Method for Image Recognition

The optimization algorithm plays a crucial role in image recognition by neural networks. However, it is challenging to accelerate the model’s convergence and maintain high precision. As a commonly used stochastic gradient descent optimization algorithm, the momentum method requires many epochs to find the optimal parameters during model training. The velocity of its gradient descent depends solely on the historical gradients and is not subject to random fluctuations. To address this issue, an optimization algorithm to enhance the gradient descent velocity, i.e., the momentum reacceleration gradient descent (MRGD), is proposed. The algorithm utilizes the point division of the current momentum and the gradient relationship, multiplying it with the gradient. It can adjust the update rate and step size of the parameters based on the gradient descent state, so as to achieve faster convergence and higher precision in training the deep learning model. The effectiveness of this method is further proven by applying the reacceleration mechanism to the Adam optimizer, resulting in the MRGDAdam algorithm. We verify both algorithms using multiple image classification datasets, and the experimental results show that the proposed optimization algorithm enables the model to achieve higher recognition accuracy over a small number of training epochs, as well as speeding up model implementation. This study provides new ideas and expansions for future optimizer research.

Semiparametric regression for spatial data via deep learning

In this work, we propose a deep learning-based method to perform semiparametric regression analysis for spatially dependent data. To be specific, we use a sparsely connected deep neural network with rectified linear unit (ReLU) activation function to estimate the unknown regression function that describes the relationship between response and covariates in the presence of spatial dependence. Under some mild conditions, the estimator is proven to be consistent, and the rate of convergence is determined by three factors: (1) the architecture of neural network class, (2) the smoothness and (intrinsic) dimension of true mean function, and (3) the magnitude of spatial dependence. Our method can handle well large data set owing to the stochastic gradient descent optimization algorithm. Simulation studies on synthetic data are conducted to assess the finite sample performance, the results of which indicate that the proposed method is capable of picking up the intricate relationship between response and covariates. Finally, a real data analysis is provided to demonstrate the validity and effectiveness of the proposed method.

A Modified Stochastic Gradient Descent Optimization Algorithm With Random Learning Rate for Machine Learning and Deep Learning

A Modified Stochastic Gradient Descent Optimization Algorithm With Random Learning Rate for Machine Learning and Deep Learning

An Improvised Sentiment Analysis Model on Twitter Data Using Stochastic Gradient Descent (SGD) Optimization Algorithm in Stochastic Gate Neural Network (SGNN).

Sentiment analysis is one of the effective techniques for mining the opinion from shapeless data contains text like review of the products, review of the movie. Sentiment analysis is used as a key to gather response from consumers, reviews of brands, marketing analyses, and political campaigns. In the subject of natural processing, performing sentiment analysis using the data obtained from Twitter is considered as a new study in these days. The dataset is gathered using the Twitter API and the Twitter package. The analysis of Twitter data is a process which takes place automatically by text data analysis to determine the view of public on the specified topic. Here, an improvised sentimental analysis model is proposed to identify the polarity of the tweets such as positive, neutral and negative. In this paper, stochastic gradient descent (SGD) algorithm uses stochastic gradient neural network (SGNN) to categorize the sentiment analysis on basis of tweets provided by the Twitter users and the proposed stochastic gradient descent optimization Algorithm based on stochastic gradient neural network (SGDOA-SGNN) provides better performance when compared with the existing Forest-Whale Optimization Algorithm based on deep neural network F-WOA-DNN model.

Reconstructed quality improvement with a stochastic gradient descent optimization algorithm for a spherical hologram.

The spherical holography is a promising technology to realize a true three-dimensional (3D) display. Compared to plane holography and cylindrical holography, it has an unlimited field of view, which can be observed from all perspectives. However, so far, the reconstructed images from computer-generated spherical holograms (CGSHs) are not of high quality, especially phase-only holograms, which will seriously affect its application. In this paper, an optimization algorithm for a CGSH based on stochastic gradient descent (SGD) is proposed to improve the quality of the reconstructed image. First, a new, to the best of our knowledge, diffraction model used in the process of optimization is proposed by considering the obliquity factor and occlusion culling. Based on our proposed diffraction model, the optimization process includes diffracting the initial random phase to another sphere, calculating the loss between the reconstructed image and the original image, and optimizing the initial phase through the SGD optimization algorithm. Both the correctness of the proposed diffraction model and the effectiveness of the SGD optimization for spherical holograms are verified well by numerical simulations. Through SGD, a high-quality reconstructed image can be achieved, which is 18 dB higher in the PSNR than that of spherical self-diffraction iteration. Meaningfully, our method has broad application prospects in 3D and omnidirectional displays. The SGD optimization algorithm is brought into the CGSH, and remarkable results have been achieved.

Random‐Objective Waveform Inversion of 3D‐9C Shallow‐Seismic Field Data

AbstractRobustness and uncertainty estimation are two challenging topics in full‐waveform inversion (FWI). To overcome these challenges, we present the methodology of random‐objective waveform inversion (ROWI), which adopts a multi‐objective framework and a preconditioned stochastic gradient descent optimization algorithm. The use of one shot per iteration avoids using redundant data and reduces the computational cost. The Pareto solutions represent a group of most likely solutions and their differences quantifies the model uncertainty associated with the trade‐off between conflicting objective functions. Due to the high dimensionality in the data and model spaces, it is prohibitively expensive to check the Pareto optimality of all solutions explicitly. Thus, we decompose the original multi‐objective function into shot‐related subproblems and use the Pareto solutions of the subproblems for trade‐off analysis. We apply ROWI to a field multi‐component shallow‐seismic data set acquired in Rheinstetten, Germany. The 3D near‐surface model is successfully reconstructed by ROWI and the main target, a refilled trench, is delineated. We compare the results estimated by ROWI and a conventional least squares FWI to prove the high efficiency of ROWI. We run six ROWI tests on the field data with different solution paths to prove the robustness of ROWI against the random solution path. The validity of the reconstructed model is verified by multiple 2D ground‐penetrating radar profiles. We estimate 246 Pareto solutions of multi‐objective subproblems for trade‐off analysis. Another four ROWI tests starting from different poor initial models are performed, whose results prove the relatively high robustness of ROWI against the initial model.

An Early Stopping-Based Artificial Neural Network Model for Atmospheric Corrosion Prediction of Carbon Steel

The optimization of network topologies to retain the generalization ability by deciding when to stop overtraining an artificial neural network (ANN) is an existing vital challenge in ANN prediction works. The larger the dataset the ANN is trained with, the better generalization the prediction can give. In this paper, a large dataset of atmospheric corrosion data of carbon steel compiled from several resources is used to train and test a multilayer backpropagation ANN model as well as two conventional corrosion prediction models (linear and Klinesmith models). Unlike previous related works, a grid searchbased hyperparameter tuning is performed to develop multiple hyperparameter combinations (network topologies) to train multiple ANNs with mini-batch stochastic gradient descent optimization algorithm to facilitate the training of a large dataset. After that, one selection strategy for the optimal hyperparameter combination is applied by an early stopping method to guarantee the generalization ability of the optimal network model. The correlation coefficients (R) of the ANN model can explain about 80% (more than 75%) of the variance of atmospheric corrosion of carbon steel, and the root mean square errors (RMSE) of three models show that the ANN model gives a better performance than the other two models with acceptable generalization. The influence of input parameters on the output is highlighted by using the fuzzy curve analysis method. The result reveals that TOW, Cl- and SO2 are the most important atmospheric chemical variables, which have a well-known nonlinear relationship with atmospheric corrosion.

Lower error bounds for the stochastic gradient descent optimization algorithm: Sharp convergence rates for slowly and fast decaying learning rates

The stochastic gradient descent (SGD) optimization algorithm is one of the central tools used to approximate solutions of stochastic optimization problems arising in machine learning and, in particular, deep learning applications. It is therefore important to analyze the convergence behavior of SGD. In this article we consider a simple quadratic stochastic optimization problem and establish for every γ,ν∈(0,∞) essentially matching lower and upper bounds for the mean square error of the associated SGD process with learning rates (γnν)n∈N. This allows us to precisely quantify the mean square convergence rate of the SGD method in dependence on the choice of the learning rates.

A hybrid recommender system using artificial neural networks

Neural network based hybrid recommender system utilizing review metadata is proposed.The system optimizes model hyper-parameters to minimize log-loss.Validate predictive capability of model against heterogeneous business categories. In the context of recommendation systems, metadata information from reviews written for businesses has rarely been considered in traditional systems developed using content-based and collaborative filtering approaches. Collaborative filtering and content-based filtering are popular memory-based methods for recommending new products to the users but suffer from some limitations and fail to provide effective recommendations in many situations. In this paper, we present a deep learning neural network framework that utilizes reviews in addition to content-based features to generate model based predictions for the business-user combinations. We show that a set of content and collaborative features allows for the development of a neural network model with the goal of minimizing logloss and rating misclassification error using stochastic gradient descent optimization algorithm. We empirically show that the hybrid approach is a very promising solution when compared to standalone memory-based collaborative filtering method.