Adaptive Momentum Research Articles

Adaptive generic prototype network with geodesic distance for cross-domain few-shot fault diagnosis

Since the fault samples of equipment are limited and the working condition is often variable, it is valuable for researching the cross-domain few-shot fault diagnosis method to assure the safe operation of various machines. As the well-known few-shot classification approaches, the traditional prototype networks are difficult to process the complex sample distributions caused by variable operating condition data and the substantial distributional discrepancies between different machines, which seriously affects the accuracy of cross-domain fault diagnosis. To address these issues, this study proposes a new adaptive geodesic prototype network (AGPN), which can extract the category prototypes with enhanced adaptability and generalization capabilities. Firstly, a geodesic distance-driven learning strategy is developed to better measure the distance between complex samples in the embedding space. Secondly, an adaptive area prototype with a dynamic expansion coefficient is proposed, which allows for more flexible representation of different data categories. Furthermore, an adaptive momentum prototype method is put forward via a model-agnostic adaptive momentum factor, which can reduce the prototype oscillation during training and maximize the learning ability of model. The proposed AGPN is successfully applied to fault diagnosis across bearings with different operating conditions. Compared with the existing few-shot diagnosis methods, the proposed method possesses higher diagnostic accuracy and training stability, thus it is more suitable for cross-domain few-shot fault diagnosis.

Optimizing Recurrent Neural Network with Bayesian Algorithm for Behavioural Authentication System

Current studies on behavioural biometrics authentication have been focused on the use of deep learning and keystroke dynamics but the aspect of conscious optimization of the algorithm in order to obtain best outcome has not been considered. This study employed and incorporated Bayesian algorithm into Recurrent Neural Network to build a Keystroke Behavioural Biometric (KBB) authentication model used against social engineering attacks. The model begins with importing the keylogging dataset for data pre-processing, feature extraction, and RNN algorithm was used to build the KBB model. Hyperparameter tuning was done to achieve optimal results. A traditional optimizer called Adaptive Momentum Estimation (Adam) was used and evaluated so as to estimate the impact of optimization in model inferencing. RNN model result with Bayesian optimization technique shows a better performance than the result of RNN model with ADAM optimization. The essence of incorporating and evaluating the best optimization technique is to come up with an effective and accurate model for behavioural biometric authentication, that could mitigate effectively against social engineering attacks.

Improving Deep Neural Networks' Training for Image Classification With Nonlinear Conjugate Gradient-Style Adaptive Momentum.

Momentum is crucial in stochastic gradient-based optimization algorithms for accelerating or improving training deep neural networks (DNNs). In deep learning practice, the momentum is usually weighted by a well-calibrated constant. However, tuning the hyperparameter for momentum can be a significant computational burden. In this article, we propose a novel adaptive momentum for improving DNNs training; this adaptive momentum, with no momentum-related hyperparameter required, is motivated by the nonlinear conjugate gradient (NCG) method. Stochastic gradient descent (SGD) with this new adaptive momentum eliminates the need for the momentum hyperparameter calibration, allows using a significantly larger learning rate, accelerates DNN training, and improves the final accuracy and robustness of the trained DNNs. For instance, SGD with this adaptive momentum reduces classification errors for training ResNet110 for CIFAR10 and CIFAR100 from 5.25% to 4.64% and 23.75% to 20.03%, respectively. Furthermore, SGD, with the new adaptive momentum, also benefits adversarial training and, hence, improves the adversarial robustness of the trained DNNs.

Adaptive momentum equation method for overcoming singularities of dispersed phases

The singularity issue arising from the phase fraction approaching zero in multiphase flow can significantly intensify the solution difficulty and lead to nonphysical results. By employing the conservative form of momentum equations in high-phase-fraction and discontinuity regions and the phase-intensive form of momentum equations in low-phase-fraction regions, computational reliability can be assured while avoiding the singularity issue. Regarding the proposed adaptive momentum equation method, the form of momentum equations for each cell is determined by a conversion bound and a phase fraction discontinuity detector. A comparative analysis is conducted on this method and other singularity-free methods. For discontinuities of dispersed phases, an error estimation method of the conversion bound is presented through theoretical analysis. Computational results demonstrate that the discontinuity detector accurately captures discontinuities in high-phase-fraction regions while disregarding pseudo-discontinuities in low-phase-fraction regions. Compared to the conservative form corrected by the terminal velocity method, the method yields higher-quality flow fields and potentially exhibits an efficiency improvement of over 10 times. Compared to the phase-intensive form, the method benefits from the physical quantity conservation, providing higher computational reliability. When encountering discontinuities, the expected error from the error estimation method aligns well with the actual error, indicating its effectiveness. When the conversion bound is below 1/10 000 of the inlet phase fraction, the errors of the adaptive method are essentially negligible.

Enhancing colorectal cancer histology diagnosis using modified deep neural networks optimizer

Optimizers are the bottleneck of the training process of any Convolutionolution neural networks (CNN) model. One of the critical steps when work on CNN model is choosing the optimal optimizer to solve a specific problem. Recent challenge in nowadays researches is building new versions of traditional CNN optimizers that can work more efficient than the traditional optimizers. Therefore, this work proposes a novel enhanced version of Adagrad optimizer called SAdagrad that avoids the drawbacks of Adagrad optimizer in dealing with tuning the learning rate value for each step of the training process. In order to evaluate SAdagrad, this paper builds a CNN model that combines a fine- tuning technique and a weight decay technique together. It trains the proposed CNN model on Kather colorectal cancer histology dataset which is one of the most challenging datasets in recent researches of Diagnose of Colorectal Cancer (CRC). In fact, recently, there have been plenty of deep learning models achieving successful results with regard to CRC classification experiments. However, the enhancement of these models remains challenging. To train our proposed model, a learning transfer process, which is adopted from a pre-complicated defined model is applied to the proposed model and combined it with a regularization technique that helps in avoiding overfitting. The experimental results show that SAdagrad reaches a remarkable accuracy (98%), when compared with Adaptive momentum optimizer (Adam) and Adagrad optimizer. The experiments also reveal that the proposed model has a more stable training and testing processes, can reduce the overfitting problem in multiple epochs and can achieve a higher accuracy compared with previous researches on Diagnosis CRC using the same Kather colorectal cancer histology dataset.

Convergence of batch gradient training-based smoothing L1 regularization via adaptive momentum for feedforward neural networks

Convergence of batch gradient training-based smoothing L1 regularization via adaptive momentum for feedforward neural networks

Identification for nonlinear systems modelled by deep long short-term memory networks based Wiener model

This paper is concerned with modeling and identification methodology for practical nonlinear system via deep long short-term memory (DLSTM) networks-based Wiener model. To determine the unknown parameters and simplify parameters identification procedure for the Wiener model, a two-step identification scheme is implemented applying hybrid signals involving separable signal and random data. First, the separable signal to separate the two blocks in Wiener model is imported, then parameters of the linear block is estimated using correlation function-based least squares technique, which handles the issue that the intermediate variable information in Wiener model cannot be measured. Moreover, integrating the advantages of adaptive stochastic gradient descent algorithm and root mean square propagation, an adaptive momentum estimation technique is created to optimize the DLSTM networks parameters based on available random data, which improves the accuracy of identified Wiener model. Compared with the other existing schemes, the superiority of the proposed method in terms of predictive performance and control results are illustrated by permanent magnet synchronous motors.

Deep ocular tumor classification model using cuckoo search algorithm and Caputo fractional gradient descent

While digital ocular fundus images are commonly used for diagnosing ocular tumors, interpreting these images poses challenges due to their complexity and the subtle features specific to tumors. Automated detection of ocular tumors is crucial for timely diagnosis and effective treatment. This study investigates a robust deep learning system designed for classifying ocular tumors. The article introduces a novel optimizer that integrates the Caputo fractional gradient descent (CFGD) method with the cuckoo search algorithm (CSA) to enhance accuracy and convergence speed, seeking optimal solutions. The proposed optimizer’s performance is assessed by training well-known Vgg16, AlexNet, and GoogLeNet models on 400 fundus images, equally divided between benign and malignant classes. Results demonstrate the significant potential of the proposed optimizer in improving classification accuracy and convergence speed. In particular, the mean accuracy attained by the proposed optimizer is 86.43%, 87.42%, and 87.62% for the Vgg16, AlexNet, and GoogLeNet models, respectively. The performance of our optimizer is compared with existing approaches, namely stochastic gradient descent with momentum (SGDM), adaptive momentum estimation (ADAM), the original cuckoo search algorithm (CSA), Caputo fractional gradient descent (CFGD), beetle antenna search with ADAM (BASADAM), and CSA with ADAM (CSA-ADAM). Evaluation criteria encompass accuracy, robustness, consistency, and convergence speed. Comparative results highlight significant enhancements across all metrics, showcasing the potential of deep learning techniques with the proposed optimizer for accurately identifying ocular tumors. This research contributes significantly to the development of computer-aided diagnosis systems for ocular tumors, emphasizing the benefits of the proposed optimizer in medical image classification domains.

On the Convergence of an Adaptive Momentum Method for Adversarial Attacks

Adversarial examples are commonly created by solving a constrained optimization problem, typically using sign-based methods like Fast Gradient Sign Method (FGSM). These attacks can benefit from momentum with a constant parameter, such as Momentum Iterative FGSM (MI-FGSM), to enhance black-box transferability. However, the monotonic time-varying momentum parameter is required to guarantee convergence in theory, creating a theory-practice gap. Additionally, recent work shows that sign-based methods fail to converge to the optimum in several convex settings, exacerbating the issue. To address these concerns, we propose a novel method which incorporates both an innovative adaptive momentum parameter without monotonicity assumptions and an adaptive step-size scheme that replaces the sign operation. Furthermore, we derive a regret upper bound for general convex functions. Experiments on multiple models demonstrate the efficacy of our method in generating adversarial examples with human-imperceptible noise while achieving high attack success rates, indicating its superiority over previous adversarial example generation methods.

ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-Order Optimization

Lowering the memory requirement in full-parameter training on large models has become a hot research area. MeZO fine-tunes the large language models (LLMs) by just forward passes in a zeroth-order SGD optimizer (ZO-SGD), demonstrating excellent performance with the same GPU memory usage as inference. However, the simulated perturbation stochastic approximation for gradient estimate in MeZO leads to severe oscillations and incurs a substantial time overhead. Moreover, without momentum regularization, MeZO shows severe over-fitting problems. Lastly, the perturbation-irrelevant momentum on ZO-SGD does not improve the convergence rate. This study proposes ZO-AdaMU to resolve the above problems by adapting the simulated perturbation with momentum in its stochastic approximation. Unlike existing adaptive momentum methods, we relocate momentum on simulated perturbation in stochastic gradient approximation. Our convergence analysis and experiments prove this is a better way to improve convergence stability and rate in ZO-SGD. Extensive experiments demonstrate that ZO-AdaMU yields better generalization for LLMs fine-tuning across various NLP tasks than MeZO and its momentum variants.

An Adaptive Optimization Algorithm in LSTM for SOC Estimation Based on Improved Borges Derivative

This paper presents an optimization algorithm in the long short-term memory (LSTM) network for state of charge (SOC) estimation based on the fractal derivative. The improved Borges derivative as a fractal derivative is introduced to the parameter optimization in LSTM networks, and the integer-order derivatives in the adaptive momentum estimation (Adam) algorithm are generalized to the improved Borges derivatives. To take advantage of the orders, we analyze the relative speed of the improved Borges derivative to the integer-order derivative. Meanwhile, a tuning method for the orders is presented to flexibly adjust the training speed of parameters. The Adam algorithm with the improved Borges derivative (Adam-IB) is designed to estimate the SOC of lithium-ion batteries, and the training speed for SOC estimation is flexibly adjusted via the Adam-IB algorithm. The experiments are carried out under different working conditions. Compared with the Adam algorithm, the Adam-IB algorithm is obviously satisfactory to estimate the SOC.

The Triumvirate of Bespoke Diverse Hybridized Activation Functions, Adaptive Momentum, and Enhanced Entropic Wavelet Energy Spectrum Discernment for Higher Efficacy Detection of Artificial Intelligence-centric Attacks

The Triumvirate of Bespoke Diverse Hybridized Activation Functions, Adaptive Momentum, and Enhanced Entropic Wavelet Energy Spectrum Discernment for Higher Efficacy Detection of Artificial Intelligence-centric Attacks

Primary Multiplicative Noise Suppression in the HTEM System Via Adaptive Momentum Estimation and Elastic-Net Regularized Framework

Primary Multiplicative Noise Suppression in the HTEM System Via Adaptive Momentum Estimation and Elastic-Net Regularized Framework

Multi-Task Momentum Distillation for Multimodal Sentiment Analysis

In the field of Multimodal Sentiment Analysis (MSA), the prevailing methods are devoted to developing intricate network architectures to capture the intra- and inter-modal dynamics, which necessitates numerous parameters and poses more difficulties in terms of interpretability in multimodal modeling. Besides, the heterogeneous nature of multiple modalities (text, audio, and vision) introduces significant modality gaps, thereby making multimodal representation learning an ongoing challenge. To address the aforementioned issues, by considering the learning process of modalities as multiple subtasks, we propose a novel approach named Multi-Task Momentum Distillation (MTMD) which succeeds in reducing the gap among different modalities. Specifically, according to the abundance of semantic information, we treat the subtasks of textual and multimodal representations as the teacher networks while the subtasks of acoustic and visual representations as the student ones to present knowledge distillation, which transfers the sentiment-related knowledge guided by the regression and classification subtasks. Additionally, we adopt unimodal momentum models to explore modality-specific knowledge deeply and employ adaptive momentum fusion factors to learn a robust multimodal representation. Furthermore, we provide a theoretical perspective of mutual information maximization by interpreting MTMD as generating sentiment-related views in various ways. Extensive experiments illustrate the superiority of our approach compared with the state-of-the-art methods in MSA.

Prototype-guided Attention Distillation for Discriminative Person Search.

Person search aims to localize a person of interest in a large image gallery captured by multiple, non-overlapping cameras. Prevalent unified methods have suffered from (1) noisy proposals with mis-detection and occlusion, and (2) large appearance variation within a class, which deteriorates the prototype-based metric learning. To address these problems, we introduce a Prototype-guided Attention Distillation, shortly PAD, which exploits a prototype (a typical representation of an identity) as a guidance to the attention module to consistently highlight identity-inherent regions across different poses. To utilize the knowledge encoded in prototypes for matching unseen IDs, PAD conducts attention distillation to guide student Re-ID queries by deeply mimicking attention maps from the prototype query. Additionally, to address large intra-class variation induced by pose or camera views, we extend PAD with multiple part prototypes representing consistent local regions across different instances. Furthermore, we exploit an adaptive momentum strategy for robust attention distillation in PAD to update more distinct prototypes. Extensive experiments conducted on CUHK-SYSU and PRW demonstrate the effectiveness of PAD, showcasing state-of-the-art performance. Moreover, our distilled attention surprisingly highlights distinguished multiple regions for person search.

Enhancing hyperspectral remote sensing image classification using robust learning technique

Advanced sensor tech integrates into diverse applications, including remote sensing, robotics, and IoT. Combining artificial intelligence (AI) with sensors enhances their capabilities, creating smart sensors, revolutionizing remote sensing and Internet of Things (IoT). This synergy forms a potent technology in the field. This study carries out a comprehensive analysis of the progress made in Hyperspectral sensors and AI-based classification techniques that are employed in remote sensing fields that utilize hyperspectral images. The classification of images obtained from Hyperspectral Sensors (HSS) has emerged as a prominent research subject within the domain of remote sensing. HSS offer a wealth of information across numerous spectral bands, supporting diverse applications such as land cover classification, environmental monitoring, agricultural assessment, change detection, and more. However, the abundance of data present in HSS also poses the challenge called the curse of dimensionality. The reduction of data dimensionality is crucial before applying any machine learning model to achieve optimal results. The present study introduces a new hybrid strategy combining the Back-Propagation algorithm with a variable adaptive momentum (BPVAM) and principal component analysis (PCA) for the purpose of classifying hyperspectral images. PCA is first applied to obtain an optimal set of discriminative features by eliminating highly correlated and redundant features. These features are then fed into the BPVAM model for classification. The addition of the momentum term in the weight update equation of the backpropagation algorithm helped achieve faster convergence with high accuracy. The proposed model was subjected to evaluation through experiments conducted on two benchmark datasets. These results indicated that the hybrid model based on BPVAM with PCA is an efficient technique for HSS classification.

Digging for gold: evaluating the authenticity of saffron (Crocus sativus L.) via deep learning optimization

IntroductionSaffron is one of the most coveted and one of the most tainted products in the global food market. A major challenge for the saffron industry is the difficulty to distinguish between adulterated and authentic dried saffron along the supply chain. Current approaches to analyzing the intrinsic chemical compounds (crocin, picrocrocin, and safranal) are complex, costly, and time-consuming. Computer vision improvements enabled by deep learning have emerged as a potential alternative that can serve as a practical tool to distinguish the pureness of saffron.MethodsIn this study, a deep learning approach for classifying the authenticity of saffron is proposed. The focus was on detecting major distinctions that help sort out fake samples from real ones using a manually collected dataset that contains an image of the two classes (saffron and non-saffron). A deep convolutional neural model MobileNetV2 and Adaptive Momentum Estimation (Adam) optimizer were trained for this purpose.ResultsThe observed metrics of the deep learning model were: 99% accuracy, 99% recall, 97% precision, and 98% F-score, which demonstrated a very high efficiency.DiscussionA discussion is provided regarding key factors identified for obtaining positive results. This novel approach is an efficient alternative to distinguish authentic from adulterated saffron products, which may be of benefit to the saffron industry from producers to consumers and could serve to develop models for other spices.

Medical image diagnosis based on adaptive Hybrid Quantum CNN

Hybrid quantum systems have shown promise in image classification by combining the strengths of both classical and quantum algorithms. These systems leverage the parallel processing power of quantum computers to perform complex computations while utilizing classical algorithms to handle the vast amounts of data involved in imaging. The hybrid approach is intended to improve accuracy and speed compared to traditional classical methods. Further research and development in this area can revolutionize the way medical images are classified and help improve patient diagnosis and treatment. The use of Conventional Neural Networks (CNN) for the classification and diagnosis of medical images using big datasets requires, in most cases, the use of special high-performance computing machines, which are very expensive and hard to access by most researchers. A new form of Machine Learning (ML), Quantum machine learning (QML), is being introduced as an emerging strategy to overcome this problem. A hybrid quantum–classical CNN uses both quantum and classical convolution layers designed to use a parameterized quantum circuit. This means that the computing model utilizes a quantum circuits approach to construct QML algorithms, which are then used to transform the quantum state to extract image hidden features. This computational acceleration is expected to achieve better algorithm performance than classical CNNs. This study intends to evaluate the performance of a Hybrid Quantum CNN (HQCNN) against a conventional CNN. This is followed by some optimizer modifications for both proposed and classical CNN methods to investigate the possible further improvement of their performance. The optimizer modification is based on forcing the optimizer to be directly adaptive to model accuracy. The optimizer adaptiveness is based on the development of an optimizer with a loss base adaptive momentum. Several algorithms are developed to achieve the above-mentioned goals, including CNN, QCNN, CNN with the adaptive optimizer, and QCNN with the Adaptive optimizer. The four algorithms are tested against a Kaggle brain dataset containing over 7000 samples. The test results show the hybrid quantum circuit algorithm outperformed the conventional CNN algorithm. The performance of both algorithms was further improved by using a fully adaptive SGD optimizer.

MUTE: A multilevel-stimulated denoising strategy for single cataractous retinal image dehazing.

In this research, we studied the duality between cataractous retinal image dehazing and image denoising and proposed that the dehazing task for cataractous retinal images can be achieved with the combination of image denoising and sigmoid function. To do so, we introduce the double-pass fundus reflection model in the YPbPr color space and developed a multilevel stimulated denoising strategy termed MUTE. The transmission matrix of the cataract layer is expressed as the superposition of denoised raw images of different levels weighted by pixel-wise sigmoid functions. We further designed an intensity-based cost function that can guide the updating of the model parameters. They are updated by gradient descent with adaptive momentum estimation, which gives us the final refined transmission matrix of the cataract layer. We tested our methods on cataract retinal images from both public and proprietary databases, and we compared the performance of our method with other state-of-the-art enhancement methods. Both visual assessments and objective assessments show the superiority of the proposed method. We further demonstrated three potential applications including blood vessel segmentation, retinal image registrations, and diagnosing with enhanced images that may largely benefit from our proposed methods.

RAdam-DA-NLSTM: A Nested LSTM-Based Time Series Prediction Method for Human–Computer Intelligent Systems

At present, time series prediction methods are widely applied for Human–Computer Intelligent Systems in various fields such as Finance, Meteorology, and Medicine. To enhance the accuracy and stability of the prediction model, this paper proposes a time series prediction method called RAdam-Dual stage Attention mechanism-Nested Long Short-Term Memory (RAdam-DA-NLSTM). First, we design a Nested LSTM (NLSTM), which adopts a new internal LSTM unit structure as the memory cell of LSTM to guide memory forgetting and memory selection. Then, we design a self-encoder network based on the Dual stage Attention mechanism (DA-NLSTM), which uses the NLSTM encoder based on the input attention mechanism, and uses the NLSTM decoder based on the time attention mechanism. Additionally, we adopt the RAdam optimizer to solve the objective function, which dynamically selects Adam and SGD optimizers according to the variance dispersion and constructs the rectifier term to fully express the adaptive momentum. Finally, we use multiple datasets, such as PM2.5 data set, stock data set, traffic data set, and biological signals, to analyze and test this method, and the experimental results show that RAdam-DA-NLSTM has higher prediction accuracy and stability compared with other traditional methods.