Adaptive Momentum Research Articles

Bayesian optimized GoogLeNet based respiratory signal prediction model from empirically decomposed gammatone visualization

Discrimination of normal and adventitious respiratory sounds from stethoscope auscultation by human ears is challenging owing to low frequency characteristics and varying frequency range for inspiration and expiration. This makes the diagnosis of pulmonary disorders a subjective one relying on the experience and hearing capability of the physician. Most computer assisted diagnosis systems formulated to address these limitations fail to capture the inherent acoustic property of respiratory sounds close to human auditory system. To circumvent this problem, this study exploits the gammatone filter banks to evaluate the distribution of all frequency components present in the signal. To categorize the respiratory signals, GoogLeNet based Convolutional Neural Networks (CNN) prediction model is developed through Time-Frequency (TF) visualization of gammatone cepstral coefficients acquired from decomposed Intrinsic Mode Functions (IMFs) in an empirical manner. As the performance of the CNN model is greatly dependent on the learning environment, this study also tends to optimize the values of the hyper parameters to enhance the classification performance of the CNN model. Accordingly, the optimal values of initial learning rate, L2 regularization and momentum are identified for both Stochastic Gradient Descent Momentum (SGDM) and Adaptive Momentum (ADAM) optimizer using Bayesian optimization technique. Experimentation is carried out for three TF visualization methods viz. spectrograms, scalograms and Constant Q spectrograms. Evaluated results show that scalogram method of classification yields high accuracy of 87.37% and 88.32% for default selection of hyperparameters using SGDM and ADAM optimizers and with optimal selection of hyperparameters based on objective function reveal an improved accuracy of 93.68 % and 95.67% for SGDM and ADAM optimizers.

Constant advance replicas method for locating minimum energy paths and transition states.

The chain-of-states (CoS) constant advance replicas (CAR) method and its climbing image variant (CI-CAR) for locating minimum energy paths (MEPs) and transition states are reported. The CAR algorithm applies the Lagrange multiplier method for imposing holonomic constraints on a chain-of-replicas, aiming to maintain equal mass-weighted/scaled root-mean-square (RMS) distances between the adjacent replicas by removing the sliding-down displacements contributed by the potential gradients during path optimization. Two contextual regularization schemes with clear geometrical interpretations are implemented to jointly promote high convergence and numerical robustness of the CAR algorithm. We show that the constrained reaction path can be solved normally within 5 steps of Lagrange multiplier updates with remarkably high numerical precision via the CAR approach. The efficacy of the CAR methods is demonstrated by testing on multiple analytical, classical, and quantum mechanical transition paths: the Müller potential, the alanine dipeptide isomerization, the helix unwinding of the VIVITLVMLKKK 12-mer peptide, and the Baker set of reactions. We also explore the potential of applying adaptive momentum (AdaM) optimizers for locating optimal transition paths under complex conformational changes. Most importantly, we discuss extensively the differences and connections between our newly proposed CAR methods and several related methods, with focuses on the reaction path with holonomic constraints (RPCons) approach of Brokaw et al. [J. Chem. Theory Comput. 2009, 5 (8), 2050-2061] and the state-of-the-art string method (SM) of E et al. [J. Chem. Phys. 2007, 126 (16), 164103]. The CAR approach represents a latest update to the general theoretical framework of reaction path finding algorithms in the two-ended CoS regime.

Autonomous face mask detection using single shot multibox detector, and ResNet-50 with identity retrieval through face matching using deep siamese neural network.

The COVID-19 pandemic poses a global health challenge. The World Health Organization states that face masks are proven to be effective, especially in public areas. Real-time monitoring of face masks is challenging and exhaustive for humans. To reduce human effort and to provide an enforcement mechanism, an autonomous system has been proposed to detect non-masked people and retrieve their identity using computer vision. The proposed method introduces a novel and efficient method that involves fine-tuning the pre-trained ResNet-50 model with a new head layer for classification between masked and non-masked people. The classifier is trained using adaptive momentum optimization algorithm with decaying learning rate and binary cross-entropy loss. Data augmentation and dropout regularization are employed to achieve best convergence. During real-time application of our classifier on videos, a Caffe face detector model based on Single Shot MultiBox Detector is used to extract the face regions of interest from each frame, on which the trained classifier is applied for detecting the non-masked people. The faces of these people are then captured, which is passed on to a deep siamese neural network, based on VGG-Face model for face matching. The captured faces are compared with the reference images from the database, by extracting the features and calculating cosine distance. If the faces match, the details of that person are retrieved from the database and displayed on the web application. The proposed method has secured best results where the trained classifier has achieved 99.74% accuracy, and the identity retrieval model achieved 98.24% accuracy.

Fighting against COVID‐19: Innovations and applications

Fighting against <scp>COVID</scp>‐19: Innovations and applications

Full waveform inversion guided wave tomography with a recurrent neural network

Corrosion quantitative detection of plate or plate-like structure materials is crucial in industrial Non-Destructive Testing (NDT) for determining their remaining life. For doing that, a novel ultrasonic guided wave tomography method, incorporating recurrent neural network (RNN) into full waveform inversion (FWI) called as RNN-FWI, is proposed in this paper. When the wave equation of an acoustic model is solved by a forward model with the cyclic calculation units of an RNN, it is shown that the inversion of the forward model can be obtained iteratively by minimizing a waveform misfit function of quadratic Wasserstein distance between the modeled and measured data. It is also demonstrated that the gradient of the objective function can be obtained by automatic differentiation while the parameters of the waveform velocity model are updated by the adaptive momentum estimation algorithm (Adam). The U-Net deep image prior (DIP) is used as the velocity model regularization in each iteration. The final thickness maps of the plate or plate-like structure materials shown can be archived by the dispersion characteristics of guided waves. Both the numerical simulation and experimental results show that the proposed RNN-FWI tomography method performs better than the conventional time-domain FWI in terms of convergence rate, initial model requirement, and robustness.

Robust Channel Estimation Scheme for Multi-UAV MmWave MIMO Communication with Jittering

In unmanned aerial vehicle (UAV)-assisted millimeter-wave (mmWave) communications, the communication performance is significantly degraded by UAV jitter. We formulate a UAV-assisted mmWave channel model with hybrid beamforming for the impacts of UAV jitter. Then, we derive the distribution of angle of arrivals/departures (AOAs/AODs) with random fluctuation of the UAV attitude angle. We develop an iterative reweight-based robust scheme as the super-resolution AOAs/AODs estimation method. Specifically, we introduce the partially adaptive momentum (Padam) estimation method to optimize the objective function of the jittering UAV mmWave massive multi-input multioutput (MIMO) system. Finally, compared with existing channel estimation schemes, the proposed UAV mmWave channel estimation method can achieve robust super-resolution performance in AOAs/AODs and path gains estimation with numerical results. Therefore, the proposed channel estimation scheme is very suitable for UAV mmWave massive MIMO communications with jittering.

Efficient Algorithms for High-Dimensional Quantum Optimal Control of a Transmon Qubit

Hybrid quantum classical optimization using near-term quantum technology is an emerging direction for exploring quantum advantage in high-dimensional systems. However, precise characterization of all experimental parameters is often impractical and challenging. A viable approach is to use algorithms that rely entirely on black-box inference rather than analytical gradients. Here, we combine randomized perturbation gradient estimation with adaptive momentum gradient updates and propose AdamSPSA and AdamRSGF algorithms. We prove the asymptotic convergence of the proposed algorithms in a convex setting and benchmark them against other gradient-based black-box optimization algorithms on nonconvex quantum optimal control tasks. Our results indicate that these algorithms accelerate the optimization rate, lower the optimization loss, and efficiently tune up high-fidelity Hann-window single-qubit gates from trivial initial conditions with up to 80 variables for a transmon qubit.

Accelerated Sparse Recovery via Gradient Descent with Nonlinear Conjugate Gradient Momentum

This paper applies an idea of adaptive momentum for the nonlinear conjugate gradient to accelerate optimization problems in sparse recovery. Specifically, we consider two types of minimization problems: a (single) differentiable function and the sum of a non-smooth function and a differentiable function. In the first case, we adopt a fixed step size to avoid the traditional line search and establish the convergence analysis of the proposed algorithm for a quadratic problem. This acceleration is further incorporated with an operator splitting technique to deal with the non-smooth function in the second case. We use the convex $$\ell _1$$ and the nonconvex $$\ell _1-\ell _2$$ functionals as two case studies to demonstrate the efficiency of the proposed approaches over traditional methods.

Applications of fractional gradient descent method with adaptive momentum in BP neural networks

A novel fractional gradient descent method with adaptive momentum is presented in this paper to improve the convergence speed and stability for BP neural network training. The fractional Grünwald-Letnikov derivative is used for the fractional gradient. The coefficient of the momentum term is set as an adaptive variable, depending on the fractional gradient of the current step and the weight change of the previous step. We give a detailed convergence proof of the proposed method. Experiments on MNIST data sets and XOR problem demonstrate that the fractional gradient descent method with adaptive momentum term can effectively improve convergence speed, maintain stability of BP neural network training, help escape from local minimum points, and enlarge the selection range of the learning rate.

Attenuating coherent environmental noise in seismic data via the U-net method

Noise attenuation is a key step in seismic data processing to enhance desired signal features, minimize artifacts, and avoid misinterpretation. However, traditional attenuation methods are often time-consuming and require expert parameter selection. Deep learning can successfully suppress various types of noise via a trained neural network, potentially saving time and effort while avoiding mistakes. In this study, we tested a U-net method to assess its usefulness in attenuating repetitive coherent events (e.g., pumpjack noise) and to investigate the influence of gain methods on denoising quality. We used the U-net method because it preserves fine-scale information during training. Its performance is controlled by network parameters and improved by minimizing misfits between true data and network estimates. A gain method is necessary to avoid the network’s parameter optimization being biased toward large values in data. We first generated synthetic seismic data with added noise for training. Next, we recovered amplitudes using an automatic gain control (AGC) or a 2D AGC (using adjacent traces’ amplitudes). Then, a back-propagation algorithm minimized the Euclidean norm cost function to optimize the network parameters for better performance. The updating step size and direction were determined using an adaptive momentum optimization method. Finally, we removed the gain effect and evaluated the denoising quality using a normalized root-mean-square error (RMSE). Based on RMSE, the data pre-processed by the 2D AGC performed better with RMSE decreasing from 0.225 to 0.09. We also assessed the limitations of the network when source wavelets or noise differed from the training set. The denoising quality of the trained network was sensitive to the change in the wavelet and noise type. The noisy data in the limitation test set were not substantially improved. The trained network was also tested on the seismic field data collected at Hussar, Alberta, by the CREWES Project. The data had not only excellent reflection events but also substantial pumpjack noise on some shot gathers. We were able to significantly reduce the noise (favorably in comparison to traditional techniques) to considerably allow greater reflection continuity. U-net noise reduction techniques show considerable promise in seismic data processing.

Autonomous landing scene recognition based on transfer learning for drones

In this paper, we study autonomous landing scene recognition with knowledge transfer for drones. Considering the difficulties in aerial remote sensing, especially that some scenes are extremely similar, or the same scene has different representations in different altitudes, we employ a deep convolutional neural network (CNN) based on knowledge transfer and fine-tuning to solve the problem. Then, LandingScenes-7 dataset is established and divided into seven classes. Moreover, there is still a novelty detection problem in the classifier, and we address this by excluding other landing scenes using the approach of thresholding in the prediction stage. We employ the transfer learning method based on ResNeXt-50 backbone with the adaptive momentum (ADAM) optimization algorithm. We also compare ResNet-50 backbone and the momentum stochastic gradient descent (SGD) optimizer. Experiment results show that ResNeXt-50 based on the ADAM optimization algorithm has better performance. With a pre-trained model and fine-tuning, it can achieve 97.845 0% top-1 accuracy on the LandingScenes-7 dataset, paving the way for drones to autonomously learn landing scenes.

Luminosity rectified blind Richardson-Lucy deconvolution for single retinal image restoration

Background and objectiveDue to imperfect imaging conditions, retinal images can be degraded by uneven/insufficient illumination, blurriness caused by optical aberrations and unintentional motions. Degraded images reduce the effectiveness of diagnosis by an ophthalmologist. To restore the image quality, in this research we propose the luminosity rectified Richardson-Lucy (LRRL) blind deconvolution framework for single retinal image restoration. MethodsWe established an image formation model based on the double-pass fundus reflection feature and developed a differentiable non-convex cost function that jointly achieves illumination correction and blind deconvolution. To solve this non-convex optimization problem, we derived the closed-form expression of the gradients and used gradient descent with Nesterov-accelerated adaptive momentum estimation to accelerate the optimization, which is more efficient than the traditional half quadratic splitting method. ResultsThe LRRL was tested on 1719 images from three public databases. Four image quality matrixes including image definition, image sharpness, image entropy, and image multiscale contrast were used for objective assessments. The LRRL was compared against the state-of-the-art retinal image blind deconvolution methods. ConclusionsOur LRRL corrects the problematic illumination and improves the clarity of the retinal image simultaneously, showing its superiority in terms of restoration quality and implementation efficiency. The MATLAB code is available on Github.

Individualisation method of biomathematical model of fatigue for predicting individual performance in mild and irregular sleep deprivation

Individual biomathematical models of fatigue (BMMF) are promising tools for detecting fatigue and possible incidents. Existing individual BMMFs have been validated in laboratory experiments in which subjects experience total sleep deprivation (TSD) and regular chronic sleep deprivation (CSD). However, some shift populations experience mild sleep deprivation (MSD) or irregular sleep deprivation (ISD) in real life. We employed the adaptive momentum estimation algorithm to adjust the classical SAFTE model for an individual. Model individualisation can be performed in real-time when new performance data are collected. The individual SAFTE model was compared with existing BMMFs in TSD, CSD, MSD, and ISD. The validation results show that the individual SAFTE model has advantages in MSD and ISD datasets collected from officers and truck drivers in real life. This study expands previous research results on the real-time individualisation of BMMFs and exposes individual BMMFs to various sleep-deprivation conditions in the field. Practitioner summary: This study proposes an individual biomathematical models of fatigue to predict human performance in mild and irregular sleep deprivation. The validation results in both laboratory and field show the proposed model has advantages over existing models when predicting officers’ and truck drivers’ performance in real life.

Bold driver and static restart fused adaptive momentum for visual question answering

Bold driver and static restart fused adaptive momentum for visual question answering

The Bidirectional Gate Recurrent Unit Based Attention Mechanism Network for State of Charge Estimation

State-of-charge (SOC) plays an important role in the battery management system, and the accuracy of its estimation directly affects the efficiency and life of the lithium battery. In this paper, a bidirectional gate recurrent unit neural network based on the attention mechanism is proposed for SOC estimation. The nesterov adaptive momentum optimized algorithm is developed to update weight matrices of the neural network. This method has several advantages over the traditional methods and structures: (1) the proposed structure can well catch the dynamics of the SOC when compared with the traditional neural network structures; (2) the proposed algorithm has faster convergence rates than the momentum gradient descent algorithm. The simulation examples show the effectiveness of the proposed algorithm and structure.

Convergence of Batch Gradient Method for Training of Pi-Sigma Neural Network with Regularizer and Adaptive Momentum Term

Convergence of Batch Gradient Method for Training of Pi-Sigma Neural Network with Regularizer and Adaptive Momentum Term

MGML: Momentum group meta-learning for few-shot image classification

At present, image classification covers more and more fields, and it is often difficult to obtain enough data for learning in some specific scenarios, such as medical fields, personalized customization of robots, etc. Few-shot image classification aims to quickly learn new classes of features from few images, and the meta-learning method has become the mainstream due to its good performance. However, the generalization ability of the meta-learning method is still poor and easy to be disturbed by low-quality images. In order to solve the above problems, this paper proposes Momentum Group Meta-Learning (MGML) to achieve a better effect of few-shot learning, which contains Group Meta-Learning module (GML) and Adaptive Momentum Smoothing module (AMS). GML obtains an ensemble model by training multiple episodes in parallel and then grouping them, which can reduce the interference of low-quality samples and improve the stability of meta-learning training. AMS includes the adaptive momentum update rule to further optimally integrate models between different groups, so that the model can memorize experience in more scenarios and enhance the generalization ability. We conduct experiments on miniImageNet and tieredImageNet datasets. The results show that MGML improves the accuracy, stability and cross-domain transfer ability of few-shot image classification, and can be applied to different few-shot learning models.

Incrementally Stochastic and Accelerated Gradient Information Mixed Optimization for Manipulator Motion Planning

This paper introduces a novel motion planner, incrementally stochastic and accelerated gradient information mixed optimization (iSAGO), for robotic manipulators in a narrow workspace. Primarily, we propose the overall scheme of iSAGO informed by the mixed momenta for an efficient constrained optimization based on the penalty method. In the stochastic part, we generate the adaptive stochastic momenta via the random selection of sub-functionals based on the adaptive momentum (Adam) method to solve the body-obstacle stuck case. Due to the slow convergence of the stochastic part, we integrate the accelerated gradient descent (AGD) to improve the planning efficiency. Moreover, we adopt the Bayesian tree inference (BTI) to transform the whole trajectory optimization (SAGO) into an incremental sub-trajectory optimization (iSAGO), which improves the computation efficiency and success rate further. Finally, we tune the key parameters and benchmark iSAGO against the other 5 planners on LBR-iiwa on a bookshelf and AUBO-i5 on a storage shelf. The result shows the highest success rate and moderate solving efficiency of iSAGO.

A pruning algorithm with relaxed conditions for high-order neural networks based on smoothing group [formula omitted] regularization and adaptive momentum

To enhance the sparseness of the network, improve its generalization ability and accelerate its training, we propose a novel pruning approach for sigma-pi-sigma neural network (SPSNN) under the relaxed condition by adding smoothing group L1/2 regularization and adaptive momentum. The main strength of this method is that it can prune both the redundant nodes between groups in the network, and also the redundant weights of the non-redundant nodes within the group, so as to achieve the sparseness of the network. Another strength is that the non-smooth absolute value function in the traditional L1/2 regularization method is replaced by a smooth function. This reduces the oscillations of learning and enables us to more effectively prove the convergence of the proposed algorithm. Finally, the numerical simulation results demonstrate the effectiveness of the proposed algorithm.

Adaptive momentum variance for attention-guided sparse adversarial attacks

The phenomenon that deep neural networks are vulnerable to adversarial examples has been found for several years. Under the black-box setting, transfer-based methods usually produce the adversarial examples on a white-box model, which serves as the surrogate model in the black-box attack, and hope that the same adversarial examples can also fool the black-box model. However, these methods have high success rates for the surrogate model and exhibit weak transferability for the black-box model. In addition, some studies have shown that deep neural networks are also vulnerable to sparse alterations of the input, but existing sparse attacks mainly focus on the number of attacked pixels without restricting the size of the perturbations, which is perceptible to human eyes. To address the above problems, we propose a transfer-based sparse attack method, called adaptive momentum variance based iterative gradient method with a class activation map, where the method considers a simple adaptive momentum variance and a refining perturbation mechanism to improve the transferability of adversarial examples. Also, a class activation map, which is also known as attention mechanism, is employed to explore the relationship between the number of the perturbed pixels and the attack performance in the case of limiting the intensity of perturbation. The proposed method is compared with a number of the state-of-the-art transfer-based adversarial attack methods on the ImageNet dataset, and the empirical results demonstrate that our method achieves a significant increase in transferability with only attacking about 50% of the pixels.