Gradient Descent Optimization Algorithm Research Articles

Prediction method of particle size qualification rate of high-pressure roller mill based on end-edge-cloud synergy

The pass rate of granules is an essential indicator during the high-pressure grinding process, as it accurately reflects the processing quality. Currently, the pass rate of granules is detected primarily based on manual experience judgments or offline inspections. Hence, this article presents a methodology for predicting the pass rate of granularity via an optimized support vector regression approach improved through genetic algorithms. Initially, a time-delay analysis method based on the particle swarm optimization algorithm is applied to mitigate the effects of time delays between the granularity pass rate and other data, thus aligning the dataset on a temporal scale. Subsequently, the feature data were selected using the maximum information coefficient analysis technique, which identified the most significant variables for inclusion in the training and testing sets of the predictive model. Predictions are then made using a support vector machine model that has been enhanced via genetic algorithm optimization. Furthermore, an online prediction model has been established, enabling real-time forecasting of the granularity pass rate and online model updates through root mean square propagation gradient descent optimization algorithm. This method leverages end-edge-cloud collaboration to provide a smart detection mechanism for the throughput rate of particles in high-pressure grinding mills. Experimental results demonstrate that, compared to traditional time-delay analysis, the improved time-delay analysis method proposed in this study is more effective and accurate. Simultaneously, the ɛ-GASVR granularity pass-rate prediction model proposed in this article achieved an R2 of 0.89.

An Improved Reacceleration Optimization Algorithm Based on the Momentum Method for Image Recognition

The optimization algorithm plays a crucial role in image recognition by neural networks. However, it is challenging to accelerate the model’s convergence and maintain high precision. As a commonly used stochastic gradient descent optimization algorithm, the momentum method requires many epochs to find the optimal parameters during model training. The velocity of its gradient descent depends solely on the historical gradients and is not subject to random fluctuations. To address this issue, an optimization algorithm to enhance the gradient descent velocity, i.e., the momentum reacceleration gradient descent (MRGD), is proposed. The algorithm utilizes the point division of the current momentum and the gradient relationship, multiplying it with the gradient. It can adjust the update rate and step size of the parameters based on the gradient descent state, so as to achieve faster convergence and higher precision in training the deep learning model. The effectiveness of this method is further proven by applying the reacceleration mechanism to the Adam optimizer, resulting in the MRGDAdam algorithm. We verify both algorithms using multiple image classification datasets, and the experimental results show that the proposed optimization algorithm enables the model to achieve higher recognition accuracy over a small number of training epochs, as well as speeding up model implementation. This study provides new ideas and expansions for future optimizer research.

Dynamics Power Quality Cost Assessment Based on a Gradient Descent Method

The escalating demand for power load is increasingly prone to triggering power quality (PQ) issues, leading to severe economic losses. Aiming at reducing the economic losses, this paper focuses on the coordinated relationship between PQ and economic costs. Firstly, a multilayer multiple linear stepwise regression method is employed to screen PQ indicators, identifying harmonic and voltage deviation as the primary influencing factors of PQ. Secondly, a gradient descent optimization algorithm based on the Least Absolute Shrinkage and Selection Operator (LASSO) is proposed, enabling rapid computation of the minimum PQ cost. Finally, through validations of two case studies, the results confirm that the proposed method can rapidly calculate the minimum PQ cost based on real-time load demands, enabling the dynamic adjustment of PQ cost to meet the evolving needs of power system development.

Parametric investigation of battery thermal management system with phase change material, metal foam, and fins; utilizing CFD and ANN models

The focus on developing an effective battery thermal management system (BTMS) to maintain optimal temperatures for lithium-ion batteries (LIBs), especially in electric vehicle (EV) applications, has grown significantly. The effective BTMS not only enhances the cooling performance of LIBs but also contributes to increased passenger safety and mileage of EVs. This study investigates BTMS configurations with fins, metal foam, and phase change material (PCM) to minimize temperature of battery during 3C discharging in varying conditions. Additionally, the study explores the impact of different BTMS material combinations and various fins lengths on system performance as a parametric investigation. Moreover, to streamline the analysis process and introduce novelty, artificial intelligence is explored as an alternative to computational fluid dynamics for predicting liquid fraction of PCM and temperature of battery, enhancing the innovative aspect of this study. Numerical simulations, using a non-equilibrium thermal model for metal foam modeling, reveal that the fourth case, integrating all three passive approaches, maintains the lowest temperature and enhances LIB cooling. The optimum BTMS shows a reduction of 3 K compared to BTMS utilizing pure PCM. During discharge process, the temperature difference in the battery decreases by approximately 75 % and 66 % in the fourth case compared to the first case (with pure PCM) under normal and harsh environmental conditions, respectively. Applying copper metal foam and copper fins yields the best results in reducing battery temperature. Increasing the length of fins and adding more fins effectively lower the battery temperature. Finally, an artificial neural network model is developed using the backpropagation learning technique coupled with the gradient descent optimization algorithm. The model exhibits excellent predictive capabilities, achieving high R-squared values of 0.98 for PCM liquid fraction and 0.99 for battery temperature.

Graph Neural Networks for Graph Drawing.

Graph drawing techniques have been developed in the last few years with the purpose of producing esthetically pleasing node-link layouts. Recently, the employment of differentiable loss functions has paved the road to the massive usage of gradient descent and related optimization algorithms. In this article, we propose a novel framework for the development of Graph Neural Drawers (GNDs), machines that rely on neural computation for constructing efficient and complex maps. GND is Graph Neural Networks (GNNs) whose learning process can be driven by any provided loss function, such as the ones commonly employed in Graph Drawing. Moreover, we prove that this mechanism can be guided by loss functions computed by means of feedforward neural networks, on the basis of supervision hints that express beauty properties, like the minimization of crossing edges. In this context, we show that GNNs can nicely be enriched by positional features to deal also with unlabeled vertexes. We provide a proof-of-concept by constructing a loss function for the edge crossing and provide quantitative and qualitative comparisons among different GNN models working under the proposed framework.

An Improved Skin Lesion Classification Using a Hybrid Approach with Active Contour Snake Model and Lightweight Attention-Guided Capsule Networks.

Skin cancer is a prevalent type of malignancy on a global scale, and the early and accurate diagnosis of this condition is of utmost importance for the survival of patients. The clinical assessment of cutaneous lesions is a crucial aspect of medical practice, although it encounters several obstacles, such as prolonged waiting time and misinterpretation. The intricate nature of skin lesions, coupled with variations in appearance and texture, presents substantial barriers to accurate classification. As such, skilled clinicians often struggle to differentiate benign moles from early malignant tumors in skin images. Although deep learning-based approaches such as convolution neural networks have made significant improvements, their stability and generalization continue to experience difficulties, and their performance in accurately delineating lesion borders, capturing refined spatial connections among features, and using contextual information for classification is suboptimal. To address these limitations, we propose a novel approach for skin lesion classification that combines snake models of active contour (AC) segmentation, ResNet50 for feature extraction, and a capsule network with a fusion of lightweight attention mechanisms to attain the different feature channels and spatial regions within feature maps, enhance the feature discrimination, and improve accuracy. We employed the stochastic gradient descent (SGD) optimization algorithm to optimize the model's parameters. The proposed model is implemented on publicly available datasets, namely, HAM10000 and ISIC 2020. The experimental results showed that the proposed model achieved an accuracy of 98% and AUC-ROC of 97.3%, showcasing substantial potential in terms of effective model generalization compared to existing state-of-the-art (SOTA) approaches. These results highlight the potential for our approach to reshape automated dermatological diagnosis and provide a helpful tool for medical practitioners.

High quality holographic 3D display with enhanced focus cues based on multiple directional light reconstruction.

Holographic display faces the trade-off between image quality and focus cues, resulting from the specific choice of phase distribution. In this Letter, we propose a speckle-free holographic display with enhanced focus cues by multiple directional light reconstruction. The uniform phase hologram is first generated by the gradient descent optimization algorithm. The blazed grating phase is used to steer the object light to a specific direction. Multiple sub-holograms with different blazed gratings are refreshed fast to reconstruct the images from different directions. Thus, the defocus blur is improved due to the separation of multiple spots on the defocus plane. The multi-plane reconstruction is also realized by pre-processing the depth images to eliminate image separation. The proposed method provides apparent focus cues while maintaining high image qualities, which is expected to realize comfortable holographic near-eye display in the near future.

Chinese Language and Literature Education and Humanistic Literacy Enhancement in Colleges and Universities under the Background of Education Informatization

Abstract Based on the background of education informatics, this paper uses natural language processing technology to analyze the teaching effect in the Chinese language and literature. By analyzing the basic methods of processing natural language, the CBOW model is constructed using text vectorization. Combined with the LDA model for Chinese language literature text, a keyword is automatically extracted. GRU is chosen as the master control unit of the information feature loop. The CBOW model is optimized using the gradient descent optimization algorithm, which improves the analysis efficiency of education and teaching. Relevant strategies for improving humanistic quality are proposed by analyzing the humanistic quality of Chinese language and literature majors in colleges and universities using natural language processing technology. The results show that the CBOW model performs better in analyzing the textual features of Chinese language and literature, and its MR value is 0.8007±0.0028 compared with the traditional neural network models such as RNN, CNN, etc. Among the strategies for improving humanistic qualities under the education of Chinese language and literature, the effect of strengthening the dissemination of cultural knowledge is 0.8. This study promotes the education of Chinese language and literature in colleges and universities to a certain degree, which is It is conducive to improving students’ humanistic literacy.

A mathematical model predicting the abundance and spatio-temporal dispersal of adfluvial salmonid (Salvelinus malma) fry in a stream

Models describing the spatial and temporal distributions of fishes are necessary for conservation and management of populations and their habitats. The early life stages of fishes are a particularly important period that can influence population dynamics and fisheries productivity. The fry of riverine migratory fish species are particularly influenced by downstream transport although relatively few models have been used to predict spatio-temporal abundance and dispersion of fry in combination with life history parameters in unperturbed habitats. We present a novel model predicting the downstream dispersal of age-0 adfluvial Dolly Varden in a spawning stream draining into Kronotskoye Lake basin, Kamchatka Peninsula, Russia. The model accounted for the number of and distance between known spawning grounds, diel activity, daily fish growth, and intra-specific diversity (i.e., morphotype) to describe how fry abundance changed at a specific stream location over the course of spring and summer. Data were collected using a trapnet monitored periodically during spring and summer to record changes in abundance, size (growth), and life history composition (‘nosed’ or ‘white’ morphotype) of fish between age-0 and -2. The experimental data were fitted to the model using a gradient descent optimization algorithm and a mean squared error loss function was used to verify how well experimental data fit to the theoretical model. The predicted abundance of age-0 fish was 18,831 with a mean dispersion rate of ∼36 m/day and a transit time between spawning area and stream mouth ranging 32–76 days. The catch was mainly comprised (81%) of the slower growing ‘nosed’ morphotype and exhibited variation in abundance over time and decreased considerably by the end of summer. Our comprehensive model will be useful to simulate the downstream dispersal of salmonid fishes whose early life stages inhabit riverine systems.

GTSO: Gradient tangent search optimization enabled voice transformer with speech intelligibility for aphasia

A frequent neurological condition known as aphasia is brought on by injury to language-related brain regions as well as possibly other regions of the brain involved in executive, memory, and attention functions. Due to a lack of speech-language pathologists and the vast expense of treatment, traditional therapy is difficult for aphasia-affected people to access. In this research work, speech intelligibility for aphasia is done by the proposed Gradient Tangent Search Optimization (GTSO) algorithm-enabled voice transformer. Here, the median filter is used for pre-processing the signal to reduce noise. The pre-processed voice signal is allowed for feature extraction and voice enhancement stages. Moreover, nonlinear spectral subtraction is used for voice enhancement and voice transformer is used for voice recognition. Also, the voice transformer is trained by GTSO, which is devised by hybridizing Gradient Descent (GD) Optimization and Tangent Search Algorithm (TSA). Then, the output obtained is fed to the language and pronunciation model for recognizing speech, and at last, the speech recognized is converted to text. Furthermore, the GTSO- enabled voice transformer is analyzed for its performance by three metrics, namely recognition accuracy, Positive Predictive Value (PPV), and Negative Predictive Value (NPV), with superior values of 0.919, 0.919, and 0.915.

Semiparametric regression for spatial data via deep learning

In this work, we propose a deep learning-based method to perform semiparametric regression analysis for spatially dependent data. To be specific, we use a sparsely connected deep neural network with rectified linear unit (ReLU) activation function to estimate the unknown regression function that describes the relationship between response and covariates in the presence of spatial dependence. Under some mild conditions, the estimator is proven to be consistent, and the rate of convergence is determined by three factors: (1) the architecture of neural network class, (2) the smoothness and (intrinsic) dimension of true mean function, and (3) the magnitude of spatial dependence. Our method can handle well large data set owing to the stochastic gradient descent optimization algorithm. Simulation studies on synthetic data are conducted to assess the finite sample performance, the results of which indicate that the proposed method is capable of picking up the intricate relationship between response and covariates. Finally, a real data analysis is provided to demonstrate the validity and effectiveness of the proposed method.

A Modified Stochastic Gradient Descent Optimization Algorithm With Random Learning Rate for Machine Learning and Deep Learning

A Modified Stochastic Gradient Descent Optimization Algorithm With Random Learning Rate for Machine Learning and Deep Learning

SFGDO: Smart flower gradient descent optimization enabled generative adversarial network for recognition of Tamil handwritten character

The Tamil language is complicated to identify, and thus more efforts are devised in the literary works. The objective is to develop a model called the Smart Flower Gradient Descent optimization-based Generative Adversarial Network (SFGDO-based GAN) for recognising Tamil handwriting. The dataset is first used to acquire the input image. The undesired noises are then pre-processed using a bilateral filter, and the binarization procedure is carried out using gradient-based thresholding. The necessary characteristics are then extracted for further categorization of Tamil characters, including character length, character width, statistical features, Local Binary Pattern (LBP), Convolutional Neural Network (CNN), density features, and Histogram of Oriented Gradients (HOG) features. Finally, utilising the proposed SFGDO enabled GAN, Tamil handwritten characters are recognised. The proposed Smart Flower Gradient Descent optimization (SFGDO) algorithm is developed by integrating Smart Flower Optimization Algorithm SFOA) and Gradient Descent Optimization (GDO).

Neural networks singular evolutive interpolated Kalman filter and its application to data assimilation for 2D water pollution model

Abstract The present study promotes a new algorithm for estimating the water pollution propagation with the primary goal of providing more reliable and high quality estimates to decision makers. To date, the widely used variational method suffers from the large computational burden, which limits its application in practice. Moreover, this method, considering the initial state as a control variable, is very sensitive in specifying initial error, especially for unstable dynamical systems. The Neural Network Filter (NNF), proposed in the present paper, is aimed at overcoming these two drawbacks in the variational method: by its nature, the NNF is sequential (no batch large assimilation window used) and stable even for unstable dynamics, with the gain parameters as control variables. The NNF, developed in the present paper, is a Neural Network Filter (NNF) version of the Singular Evolutive Interpolated Kalman Filter (SEIKF). One of the new versions of this NNF is that it uses structure of the gain of SEIKF0 taken by the SEIKF at the first time moment of correction process. To deal with the uncertainty of the system parameters and of the noise covariance, the proposed Neural Network SEIKF0 named by NNSEIKF0 makes use of the covariance of a reduced rank iterated during assimilation process and of some pertinent gain parameters tuned adaptively to yield the minimum prediction error for the system output. The computational burden in implementation of the NNSEIKF0 is reduced drastically due to applying the optimization tool known as a simultaneous perturbation stochastic approximation (SPSA) algorithm, which requires only two integrations of the numerical model. No iterative loop is required at each assimilation instant as usually happens with the standard gradient descent optimization algorithms. Data assimilation experiment, carried out by the SEIKF0 and NNSEIKF0, is implemented for the Thanh Nhan Lake in Hanoi and the performance comparison between the NNSEIKF0 and SEIKF0 is given to show the high efficiency of the proposed NNSEIKF0.

Accelerating AI-Based Battery Management System’s SOC and SOH on FPGA

Lithium battery-based electric vehicles (EVs) are gaining global popularity as an alternative to combat the adverse environmental impacts caused by the utilization of fossil fuels. State of charge (SOC) and state of health (SOH) are vital parameters that assess the battery’s remaining charge and overall health. Precise monitoring of SOC and SOH is critical for effectively operating the battery management system (BMS) in a lithium battery. This article presents an experimental study for the artificial intelligence (AI)-based data-driven prediction of lithium battery parameters SOC and SOH with the help of deep learning algorithms such as Long Short-Term Memory (LSTM) and bidirectional LSTM (BiLSTM). We utilized various gradient descent optimization algorithms with adaptive and constant learning rates with other default parameters. Compared between various gradient descent algorithms, the selection of the optimal one depends on mean absolute error (MAE) and root mean squared error (RMSE) accuracy. We developed an LSTM and BiLSTM model with four hidden layers with 128 LSTM or BiLSTM units per hidden layer that use Panasonic 18650PF Li-ion dataset released by NASA to predict SOC and SOH. Our experimental results advise that the selection of the optimal gradient descent algorithm impacts the model’s accuracy. The article also addresses the problem of overfitting in the LSTM/BiLSTM model. BiLSTM is the best choice to improve the model’s performance but increase the cost. We trained the model with various combinations of parameters and tabulated the accuracies in terms of MAE and RMSE. This optimal LSTM model can predict the SOC of the lithium battery with MAE more minor than 0.0179%, RMSE 0.0227% in the training phase, MAE smaller than 0.695%, and RMSE 0.947% in the testing phase over a 25°C dataset. The BiLSTM can predict the SOC of the 18650PF lithium battery cell with MAE smaller than 0.012% for training and 0.016% for testing. Similarly, using the Adam optimization algorithm, RMSE for training and testing is 0.326% and 0.454% over a 25°C dataset, respectively. BiLSTM with an adaptive learning rate can improve performance. To provide an alternative solution to high power consuming processors such as central processing unit (CPU) and graphics processing unit (GPU), we implemented the model on field programmable gate Aarray (FPGA) PYNQ Z2 hardware device. The LSTM model using FPGA performs better.

A fixed-time disturbance rejection control framework based on artificial intelligence for the flight environment testbed system

A fixed-time disturbance rejection control framework based on artificial intelligence is proposed to achieve rapid tracking control of the flight environment testbed (FET) system in this study. Firstly, to resist the violent disturbance caused by the change in engine operating conditions and other disturbances, fixed-time extended state observers (FESO) are used in this paper to estimate disturbances in a fixed time and improve the anti-disturbance capability of the system. Then, the anti-saturation fixed-time control (FTC) law is used to prevent the system from collapsing or significant tracking errors due to actuator saturation and ensure the fixed-time convergence of tracking errors. Finally, the gradient descent optimization algorithm based on long short-term memory (LSTM) network is used to optimize the controller parameters and further improve the controller’s adaptive ability. Simulation experiments prove the effectiveness of the proposed method.

A novel wind speed forecasting combined model using variational mode decomposition, sparse auto-encoder and optimized fuzzy cognitive mapping network

The nonlinear, random and fluctuating characteristics of wind speed bring great challenges to its accurate forecast, so no model that can adapt to all situations. In order to solve the problem of unbalanced forecast accuracy and stability in the current wind speed forecast model, a novel and advanced wind speed combined forecast model (CFM) is proposed in this study. The CFM adopts a two-phase data processing strategy composed of variable mode decomposition-sparse autoencoder (VMD-SAE) to extract the original wind speed features, high-order fuzzy cognitive mapping (HFCM) neural network modeling and batch gradient descent optimization algorithm to make up for its shortcomings. The two-phase data processing strategy performs smoothing and feature information extraction processing on the original data. The forecast module adopts the SAE-HFCM combination strategy, and utilizes their respective advantages to achieve accurate and stable result output. The results show that this CFM has the best forecast accuracy and generalization performance compared with 7 benchmark models in datasets from three different sites. Compared with the benchmark models, the performance of CFM in point forecasting and interval forecasting, the average improvement percentage of IP_RMSE, IP_MAE and IP_MAPE minimum values are 40.9%, 40.1% and 40.6%, respectively. The performance of interval indicators is also the best, and its application prospects are broad.

A few theoretical results for Laplace and arctan penalized ordinary least squares linear regression estimators

Two new non convex penalty functions – Laplace and arctan – were recently introduced in the literature to obtain sparse models for high-dimensional statistical problems. In this article, we study the theoretical properties of Laplace and arctan penalized ordinary least squares linear regression models. We first illustrate the near-unbiasedness of the non zero regression weights obtained by the new penalty functions, in the orthonormal design case. In the general design case, we present theoretical results in two asymptotic settings: (a) the number of features, p fixed, but the sample size, n → ∞ , and (b) both n and p tend to infinity. The theoretical results shed light onto the differences between the solutions based on the new penalty functions and those based on existing convex and non convex Bridge penalty functions. Our theory also shows that both Laplace and arctan penalties satisfy the oracle property. Finally, we also present results from a brief simulations study illustrating the performance of Laplace and arctan penalties based on the gradient descent optimization algorithm.

Adaptive learning rate optimization algorithms with dynamic bound based on Barzilai-Borwein method

The training effect of the neural network model is directly influenced by optimization algorithms. The Barzilai-Borwein(BB) method is used in the stochastic gradient descent (SGD) and other deep learning optimization algorithms because of its outstanding performance in terms of convergence speed. In order to improve the stability of BB step size and the output quality of network model in deep learning, this paper presents two improved optimization algorithms based on the BB method: BBbound and AdaBBbound. BBbound reduces the floating BB step size by generating the upper bound of the step in the current iteration. It avoids the occasional occurrence of long steps; AdaBBbound is an adaptive gradient method based on BB method with dynamic bound. The algorithm has a fast convergence effect in the early stage, then calculates learning rate and smoothly transitions to SGD by setting the better conditions of the BB method. We analyzed the performance of the two algorithms based on the initial conditions and the learning rate change curve. Meanwhile, we tested our algorithms on popular network models such as ResNet and DenseNet. The results showed that the new optimization algorithms have achieved high stability and significantly improved the performance of the network model.

WGM-dSAGA: Federated Learning Strategies with Byzantine Robustness Based on Weighted Geometric Median

Federated learning techniques accomplish federated modeling and share global models without sharing data. Federated learning offers a good answer to complex data and privacy security issues. Although there are many ways to target federated learning, Byzantine attacks are the ones we concentrate on. Byzantine attacks primarily impede learning by tampering with the local model parameters provided by a client to the master node throughout the federation learning process, leading to a final global model that diverges from the optimal solution. To address this problem, we combine aggregation rules with Byzantine robustness using a gradient descent optimization algorithm based on variance reduction. We propose a WGM-dSAGA method with Byzantine robustness, called weighted geometric median-based distributed SAGA. We replace the original mean aggregation strategy in the distributed SAGA with a robust aggregation rule based on weighted geometric median. When less than half of the clients experience Byzantine attacks, the experimental results demonstrate that our proposed WGM-dSAGA approach is highly robust to different Byzantine attacks. Our proposed WGM-dSAGA algorithm provides the optimal gap and variance under a Byzantine attack scenario.