Stochastic Gradient Descent Algorithm Research Articles

Research on multi-context aware recommendation methods based on tensor factorization

Compared to the traditional recommender systems, context-aware recommender systems are more in line with actual application contexts. However, the existing researches are mostly focused on single context-aware recommendation, such as time-aware recommendation or location-aware recommendation, and lack of in-depth research on multi-context-aware recommendation. Therefore, we proposed a recommendation method of high-order tensor factorization based on multi-context-aware. First, on the basis of analyzing the influence of context on users’ interest preferences, the sensitivity of users to multiple contexts was detected using statistical methods. For context-sensitive users, four-dimensional tensors and feature matrices used to solve data sparsity were constructed based on rating matrix and situational information. And then the stochastic gradient descent algorithm was used for iterative calculation to fill in missing data values and carry out parameter optimization. For context-insensitive users, we used matrix factorization to predict users’ interest preferences. Finally, we tested and validated our method on a multi-context-aware movie dataset, and the experimental results show that the proposed method could effectively reduce the prediction error and improve the recommendation quality.

Parameter estimation of multiple‐input single‐output Hammerstein controlled autoregressive system based on improved adaptive moment estimation algorithm

AbstractIn this article, the problem of parameter estimation for a multiple‐input single‐output Hammerstein controlled autoregressive (MISO‐HCAR) model is solved. After establishing the identification model of MISO‐HCAR system, an improved adaptive moment estimation algorithm with decreasing weight (IADAM) is proposed. The improved algorithm transforms the nonlinear system identification problem into a parameter space optimization problem, according to the input and output data obtained from the system, and uses the parallel search ability of the IADAM algorithm to estimate all the parameters of the system simultaneously. Two numerical examples and the case study example of chemical continuously stirred tank reactor (CSTR) system show that the proposed IADAM algorithm with decreasing weight has better identification effect on the MISO‐HCAR model and CSTR system than the stochastic gradient descent (SGD) algorithm and the basic adaptive moment estimation (ADAM) algorithm, and the estimation accuracy and convergence speed are greatly improved. Thus, the IADAM can estimate the parameters of the MISO‐HCAR system and the CSTR system more effectively.

Dynamic layer-wise sparsification for distributed deep learning

Distributed stochastic gradient descent (SGD) algorithms are becoming popular in speeding up deep learning model training by employing multiple computational devices (named workers) parallelly. Top-k sparsification, a mechanism where each worker only communicates a small number of largest gradients (by absolute value) and accumulates the rest locally, is one of the most basic and high-profile practices to reduce communication overhead. However, the theoretical implementation (Global Top-k SGD) ignoring the layer-wise structure of neural networks has low training efficiency, since the top-k operation requiring the whole gradients impedes parallelism of computation and communication. The practical implementation (Layer-wise Top-k SGD) solves the parallelism problem, but hurts the performance of the trained model due to the deviation from the theoretically optimal solution. In this paper, we solve this contradiction by introducing a Dynamic Layer-wise Sparsification (DLS) mechanism and its extensions, DLS(s). DLS(s) efficiently adjusts the sparsity ratios of the layers to make the uploaded threshold of each layer automatically tend to be the unified global one, so as to retain the good performance of Global Top-k SGD and the high efficiency of Layer-wise Top-k SGD. The experimental results show that DLS(s) outperforms Layer-wise Top-k SGD in performance, and performs close to Global Top-k SGD yet have much less training time.

Two-timescale stochastic gradient descent in continuous time with applications to joint online parameter estimation and optimal sensor placement

In this paper, we establish the almost sure convergence of two-timescale stochastic gradient descent algorithms in continuous time under general noise and stability conditions, extending well known results in discrete time. We analyse algorithms with additive noise and those with non-additive noise. In the non-additive case, our analysis is carried out under the assumption that the noise is a continuous-time Markov process, controlled by the algorithm states. The algorithms we consider can be applied to a broad class of bilevel optimisation problems. We study one such problem in detail, namely, the problem of joint online parameter estimation and optimal sensor placement for a partially observed diffusion process. We demonstrate how this can be formulated as a bilevel optimisation problem, and propose a solution in the form of a continuous-time, two-timescale, stochastic gradient descent algorithm. Furthermore, under suitable conditions on the latent signal, the filter, and the filter derivatives, we establish almost sure convergence of the online parameter estimates and optimal sensor placements to the stationary points of the asymptotic log-likelihood and asymptotic filter covariance, respectively. We also provide numerical examples, illustrating the application of the proposed methodology to a partially observed Beneš equation, and a partially observed stochastic advection-diffusion equation.

Safety-Aware Semi-End-to-End Coordinated Decision Model for Voltage Regulation in Active Distribution Network

Prediction plays a vital role in the active distribution network voltage regulation under the high penetration of photovoltaics. Current prediction models aim at minimizing individual prediction errors but overlook their collective impacts on downstream decision-making. Hence, this paper proposes a safety-aware semi-end-to-end coordinated decision model to bridge the gap from the downstream voltage regulation to the upstream multiple prediction models in a coordinated differential way. The semi-end-to-end model maps the input features to the optimal var decisions via prediction, decision-making, and decision-evaluating layers. It leverages the neural network and the second-order cone program (SOCP) to formulate the stochastic PV/load predictions and the var decision-making/evaluating separately. Then the var decision quality is evaluated via the weighted sum of the power loss for economy and the voltage violation penalty for safety, denoted by regulation loss. Based on the regulation loss and prediction errors, this paper proposes the hybrid loss and hybrid stochastic gradient descent algorithm to back-propagate the gradients of the hybrid loss with respect to multiple predictions for enhancing decision quality. Case studies verify the effectiveness of the proposed model with lower power loss for economy and lower voltage violation rate for safety awareness.

Adaptive Optical Closed-Loop Control Based on the Single-Dimensional Perturbation Descent Algorithm.

Modal-free optimization algorithms do not require specific mathematical models, and they, along with their other benefits, have great application potential in adaptive optics. In this study, two different algorithms, the single-dimensional perturbation descent algorithm (SDPD) and the second-order stochastic parallel gradient descent algorithm (2SPGD), are proposed for wavefront sensorless adaptive optics, and a theoretical analysis of the algorithms' convergence rates is presented. The results demonstrate that the single-dimensional perturbation descent algorithm outperforms the stochastic parallel gradient descent (SPGD) and 2SPGD algorithms in terms of convergence speed. Then, a 32-unit deformable mirror is constructed as the wavefront corrector, and the SPGD, single-dimensional perturbation descent, and 2SPSA algorithms are used in an adaptive optics numerical simulation model of the wavefront controller. Similarly, a 39-unit deformable mirror is constructed as the wavefront controller, and the SPGD and single-dimensional perturbation descent algorithms are used in an adaptive optics experimental verification device of the wavefront controller. The outcomes demonstrate that the convergence speed of the algorithm developed in this paper is more than twice as fast as that of the SPGD and 2SPGD algorithms, and the convergence accuracy of the algorithm is 4% better than that of the SPGD algorithm.

Financial Time Series Forecasting: A Data Stream Mining-Based System

Data stream mining (DSM) represents a promising process to forecast financial time series exchange rate. Financial historical data generate several types of cyclical patterns that evolve, grow, decrease, and end up dying. Within historical data, we can notice long-term, seasonal, and irregular trends. All these changes make traditional static machine learning models not relevant to those study cases. The statistically unstable evolution of financial market behavior yields a progressive deterioration in any trained static model. Those models do not provide the required characteristics to evolve continuously and sustain good forecasting performance as the data distribution changes. Online learning without DSM mechanisms can also miss sudden or quick changes. In this paper, we propose a possible DSM methodology, trying to cope with that instability by implementing an incremental and adaptive strategy. The proposed algorithm includes the online Stochastic Gradient Descent algorithm (SGD), whose weights are optimized using the Particle Swarm Optimization Metaheuristic (PSO) to identify repetitive chart patterns in the FOREX historical data by forecasting the EUR/USD pair’s future values. The data trend change is detected using a statistical technique that studies if the received time series instances are stationary or not. Therefore, the sliding window size is minimized as changes are detected and maximized as the distribution becomes more stable. Results, though preliminary, show that the model prediction is better using flexible sliding windows that adapt according to the detected distribution changes using stationarity compared to learning using a fixed window size that does not incorporate any techniques for detecting and responding to pattern shifts.

Fractional Stochastic Search Algorithms: Modelling Complex Systems via AI

The aim of this article is to establish a stochastic search algorithm for neural networks based on the fractional stochastic processes {BtH,t≥0} with the Hurst parameter H∈(0,1). We define and discuss the properties of fractional stochastic processes, {BtH,t≥0}, which generalize a standard Brownian motion. Fractional stochastic processes capture useful yet different properties in order to simulate real-world phenomena. This approach provides new insights to stochastic gradient descent (SGD) algorithms in machine learning. We exhibit convergence properties for fractional stochastic processes.

A modified Adam algorithm for deep neural network optimization

Deep Neural Networks (DNNs) are widely regarded as the most effective learning tool for dealing with large datasets, and they have been successfully used in thousands of applications in a variety of fields. Based on these large datasets, they are trained to learn the relationships between various variables. The adaptive moment estimation (Adam) algorithm, a highly efficient adaptive optimization algorithm, is widely used as a learning algorithm in various fields for training DNN models. However, it needs to improve its generalization performance, especially when training with large-scale datasets. Therefore, in this paper, we propose HN Adam, a modified version of the Adam Algorithm, to improve its accuracy and convergence speed. The HN_Adam algorithm is modified by automatically adjusting the step size of the parameter updates over the training epochs. This automatic adjustment is based on the norm value of the parameter update formula according to the gradient values obtained during the training epochs. Furthermore, a hybrid mechanism was created by combining the standard Adam algorithm and the AMSGrad algorithm. As a result of these changes, the HN_Adam algorithm, like the stochastic gradient descent (SGD) algorithm, has good generalization performance and achieves fast convergence like other adaptive algorithms. To test the proposed HN_Adam algorithm performance, it is evaluated to train a deep convolutional neural network (CNN) model that classifies images using two different standard datasets: MNIST and CIFAR-10. The algorithm results are compared to the basic Adam algorithm and the SGD algorithm, in addition to other five recent SGD adaptive algorithms. In most comparisons, the HN Adam algorithm outperforms the compared algorithms in terms of accuracy and convergence speed. AdaBelief is the most competitive of the compared algorithms. In terms of testing accuracy and convergence speed (represented by the consumed training time), the HN-Adam algorithm outperforms the AdaBelief algorithm by an improvement of 1.0% and 0.29% for the MNIST dataset, and 0.93% and 1.68% for the CIFAR-10 dataset, respectively.

Load balanced locality-aware parallel SGD on multicore architectures for latent factor based collaborative filtering

We investigate the parallelization of Stochastic Gradient Descent (SGD) for matrix completion on multicore architectures. We provide an experimental analysis of current SGD algorithms to find out their bottlenecks and limitations. Grid-based methods suffer from load imbalance among 2D blocks of the rating matrix, especially when datasets are skewed and sparse. Asynchronous methods, on the other hand, can face cache issues due to their memory access pattern. We propose bin-packing-based block balancing methods that are alternative to the recently proposed BaPa method. We then introduce Locality Aware SGD (LASGD), a grid-based asynchronous parallel SGD algorithm that efficiently utilizes cache by changing nonzero update sequence without affecting factor update order and carefully arranging latent factor matrices in the memory. Combined with our proposed load balancing methods, our experiments show that LASGD performs significantly better than alternative approaches in parallel shared-memory systems.

Adaptive proximal SGD based on new estimating sequences for sparser ERM

Estimating sequences introduced by Nesterov is an efficient trick to accelerate gradient descent (GD). The stochastic version of estimating sequences is also successfully used to speed up stochastic gradient descent (SGD). In solving the non-smooth convex optimization problems, the convergence rate of SGD with stochastic estimating sequences is O(1/k). Here k is the number of iterations. In this paper, we present a new way of constructing estimating sequences. The characteristic of the new estimating sequences is to replace the subgradient with the proximal stochastic gradient. The novelty of the new estimating sequences is to replace the fixed learning rate with the adaptive learning rate. The adaptive learning rate is calculated by the exponential moving average of past squared stochastic gradients. Based on the new estimating sequences, we propose an adaptive proximal SGD algorithm, called ES-APSGD, for solving the large-scale ℓ1-norm regularized empirical risk minimization (ERM). The proposed ES-APSGD simplifies the calculation and can obtain a convergence rate of O(1/k2). The significant advantage of ES-APSGD is to strengthen the sparsity of solution by adaptively adjusting the threshold magnitude. Experimental results on Lasso and ℓ1-norm regularized logistic regression show that ES-APSGD speeds up convergence and obtains the sparser optimal solutions.

Implementing arbitrary quantum operations via quantum walks on a cycle graph

The quantum circuit model is the most commonly used model for implementing quantum computers and quantum neural networks whose essential tasks are to realize certain unitary operations. Here we propose an alternative approach; we use a simple discrete-time quantum walk (DTQW) on a cycle graph to model an arbitrary unitary operation $U(N)$ without the need to decompose it into a sequence of gates of smaller sizes. Our model is essentially a quantum neural network based on DTQW. Firstly, it is universal as we show that any unitary operation $U(N)$ can be realized via an appropriate choice of coin operators. Secondly, our DTQW-based neural network can be updated efficiently via a learning algorithm, i.e., a modified stochastic gradient descent algorithm adapted to our network. By training this network, one can promisingly find approximations to arbitrary desired unitary operations. With an additional measurement on the output, the DTQW-based neural network can also implement general measurements described by positive-operator-valued measures (POVMs). We show its capacity in implementing arbitrary 2-outcome POVM measurements via numeric simulation. We further demonstrate that the network can be simplified and can overcome device noises during the training so that it becomes more friendly for laboratory implementations. Our work shows the capability of the DTQW-based neural network in quantum computation and its potential in laboratory implementations.

A polynomial chaos efficient global optimization approach for Bayesian optimal experimental design

This paper proposes a global optimization framework to address the high computational cost and non convexity of Optimal Experimental Design (OED) problems. To reduce the computational burden and the presence of noise in the evaluation of the Shannon expected information gain (SEIG), this framework proposes the coupling of Laplace approximation and polynomial chaos expansions (PCE). The advantage of this procedure is that PCE allows large samples to be employed for the SEIG estimation, practically vanishing the noisy introduced by the sampling procedure. Consequently, the resulting optimization problem may be treated as deterministic. Then, an optimization approach based on Kriging surrogates is employed as the optimization engine to search for the global solution with limited computational budget. Four numerical examples are investigated and their results are compared to state-of-the-art stochastic gradient descent algorithms. The proposed approach obtained better results than the stochastic gradient algorithms in all situations, indicating its efficiency and robustness in the solution of OED problems.

Decentralized Stochastic Optimization With Inherent Privacy Protection

Decentralized stochastic optimization is the basic building block of modern collaborative machine learning, distributed estimation and control, and large-scale sensing. Since involved data usually contain sensitive information like user locations, healthcare records and financial transactions, privacy protection has become an increasingly pressing need in the implementation of decentralized stochastic optimization algorithms. In this paper, we propose a decentralized stochastic gradient descent algorithm which is embedded with inherent privacy protection for every participating agent against other participating agents and external eavesdroppers. This proposed algorithm builds in a dynamics based gradient-obfuscation mechanism to enable privacy protection without compromising optimization accuracy, which is in significant difference from differential-privacy based privacy solutions for decentralized optimization that have to trade optimization accuracy for privacy. The dynamics based privacy approach is encryption-free, and hence avoids incurring heavy communication or computation overhead, which is a common problem with encryption based privacy solutions for de

Makine Öğrenmesi Yöntemleri İle Günümüz Ve Geleceğe Yönelik Meslek Tahminlerinin Değerlendirilmesi : Türkiye'den Ampirik Deliller

For the purpose of evaluating present and future trends of professions within the labor market, text mining approach could be an alternative to more traditional approaches such as employer surveys. Specifically, machine learning algorithms are used for making accurate predictions about the future directions of the professions which consequently will influence professional development of labour force. The aim of this study is to investigate the professions of the future and current in Turkey by the application of supervised learning algorithms and clustering methods to various Turkish data including documents belonging to Turkey's institutions. In this study, the popular professions were predicted with an accuracy rate between ≅0.81 and ≅0.93 thorough various machine learning algorithms. It was discovered that methodologically perceptron and stochastic gradient descent algorithms demonstrated superiority over other algorithms thanks to their intelligence functions. Furthermore, the analysis of current professions in Turkey revealed that the class of "Professional occupations", "Managers" and "Technicians and assistant professional members" were popular, and according to the analysis of the future, information technology-based occupations will be important. Although limited Turkish data sources for the analysis of future, results with an accuracy of nearly 1 were produced.

Novel approach based on stochastic gradient descent for controlling the reconstructed phase randomness of computer-generated holograms

Depth of field, as one of the essential adjustment cues, affects the user experience and restricts the application of holographic displays. However, it is often underappreciated in modern holographic displays. In this study, we propose a novel computer-generated hologram (CGH) method that uses a pre-calculation procedure and a standard deviation-based loss function to generate realistic depth-of-field effects. The depth-of-field size is adjusted by controlling the randomness of the reconstructed phase. The ability of the proposed algorithm to control the reconstructed phase randomness was confirmed. Finally, a holographic display prototype was built to demonstrate the proposed method experimentally. The experimental results were consistent with the simulation results. Our findings may open up a new perspective on the application of existing stochastic gradient descent (SGD) algorithms to traditional methods for image quality optimization by adding random phases.

Research on SGD Algorithm Using Momentum Strategy

With the continuous development of stochastic gradient descent algorithms, many efficient momentum algorithms have appeared. Stochastic gradient descent(SGD) is one of the classic algorithms in optimization. Its accelerated version, the SGD algorithm with momentum strategy, has been a hot research topic in recent years. Therefore, this paper will analyze and summarize these series of algorithms, starting with the classical momentum algorithm, and introduce some improved versions of the momentum algorithm. Numerical experiments on real problems will also be done to evaluate the performance of these algorithms. It is proved that the addition of momentum and adaptive learning rate effectively improve the performance of these algorithms. In future research, some cutting-edge momentum algorithms and other basic network should be analyzed.

An error based adaptive learning rate stochastic gradient descent algorithm in convolutional neural network

If the learning rate of convolutional neural network (CNN) is set improperly, the efficiency and accuracy of the algorithm will be greatly affected. To solve this problem, a learning rate adaptive algorithm is proposed to improve the traditional SGD: based on parameter prediction, the historical training error is used to update the learning rate. Under the condition of given initial learning rate, experiments on classical data sets prove the effectiveness of the above algorithms, and the adaptive learning rate stochastic gradient descent algorithm can keep the convergence of the network. Training accuracy is relatively stable; Shorter training time; And improve the learning accuracy.

Surface morphology inspection for directed energy deposition using small dataset with transfer learning

Directed Energy Deposition (DED) is a branch of Additive Manufacturing (AM) that offers extraordinary controllability and the ability to rapidly shape materials directly into large parts with complex structures. Innovative applications of machine learning and neural network algorithms in additive manufacturing processes have recently been developed, but there are still several barriers to the development and practical application of this technology. The large amount of sample data required for visual inspection using a neural network framework leads to the problem of difficult and costly preparation of surface morphology data. In this study, a transfer learning approach based on a small dataset is proposed to detect surface morphology in the DED procedure. The approach proposed effectively and innovatively addresses the sample scarcity issue that arises during neural network training and eliminates the issue of having to recreate a sizable quantity of the matching surface sample data. It also effectively accelerates the model convergence process through the transfer learning method, which indirectly improves detection accuracy. This work selected a parametric transfer learning approach to train YOLOv7, a self-supervised object detection model, for transfer learning. The experiments utilized an open-source defective dataset (NEU-DEF) for transfer learning pretraining. The training was carried out in combination with a home-made object dataset of 240 small samples after transfer of the model parameters. The input to the model is image data and the output of the training is the classification of surface morphology. The results show that the transfer training model 2 pretrained with the Stochastic Gradient Descent (SGD) algorithm has high accuracy and precision, achieving a Mean Average Precision (MAP) value of 0.62. Transfer learning speeds up the feature extraction process by transferring the parameters of a pretrained model for a similar task directly into the current task model, resulting in improved detection performance.

A state‐of‐the‐art survey of predicting students' performance using artificial neural networks

AbstractPredicting students' performance is one of the most important issue in educational data mining. In order to investigate the state‐of‐the‐art research development in predicting students' performance by using artificial neural networks (ANN), we conducted a survey on 39 important studies on this issue from 2016 to 2021. The results show that: (1) objectives of most prediction model is the performance of learners on the program and course; (2) datasets used for training prediction model are collected from logs of the learning management system; (3) the most commonly used ANN is feedforward neural network; (4) researchers use stochastic gradient descent and Adam algorithm to optimizes the parameters in ANN and configure hyper parameters of ANN manually; (5) feature selection is not necessary because ANN can automatically adjust the weights of artificial neurons; and (6) ANN has better performance than the classical classifiers in predicting student performance. The challenges in previous studies were comprehensively analyzed, suggestions for future research were put forward.