Adaptive Momentum Research Articles

Deterministic convergence analysis via smoothing group Lasso regularization and adaptive momentum for Sigma-Pi-Sigma neural network

In this paper, we propose a sparse and accelerated method for Sigma-Pi-Sigma neural network training based on smoothing group lasso regularization and adaptive momentum. It is shown that group sparsity can more efficiently sparsity the network structure at the group level, and the adaptive momentum term can speed up the learning convergence during the iteration process. Also, another important contribution lies in the analysis of theoretical results. However, the group lasso regularization is not differentiable at the origin. This leads to oscillations observed in numerical experiments and poses a challenge to theoretical analysis. So, we overcome those problems by smoothing techniques. Under suitable assumptions, we strictly proved the monotonicity, and weak and strong convergence theorems of the new algorithm. Finally, the numerical experiments are presented to support our theoretical findings.

Gradient-based Learning Methods Extended to Smooth Manifolds Applied to Automated Clustering

Grassmann manifold based sparse spectral clustering is a classification technique that consists in learning a latent representation of data, formed by a subspace basis, which is sparse. In order to learn a latent representation, spectral clustering is formulated in terms of a loss minimization problem over a smooth manifold known as Grassmannian. Such minimization problem cannot be tackled by one of traditional gradient-based learning algorithms, which are only suitable to perform optimization in absence of constraints among parameters. It is, therefore, necessary to develop specific optimization/learning algorithms that are able to look for a local minimum of a loss function under smooth constraints in an efficient way. Such need calls for manifold optimization methods. In this paper, we extend classical gradient-based learning algorithms on at parameter spaces (from classical gradient descent to adaptive momentum) to curved spaces (smooth manifolds) by means of tools from manifold calculus. We compare clustering performances of these methods and known methods from the scientific literature. The obtained results confirm that the proposed learning algorithms prove lighter in computational complexity than existing ones without detriment in clustering efficacy.

Pneumonia Classification of Thorax Images using Convolutional Neural Networks

The digital image processing technique is a product of computing technology development. Medical image data processing based on a computer is a product of computing technology development that can help a doctor to diagnose and observe a patient. This study aimed to perform classification on the image of the thorax by using Convolutional Neural Network (CNN). The data used in this study is lung thorax images that have previously been diagnosed by a doctor with two classes, namely normal and pneumonia. The amount of data is 2.200, 1.760 for training, and 440 for testing. Three stages are used in image processing, namely scaling, gray scaling, and scratching. This study used Convolutional Neural Network (CNN) method with architecture ResNet-50. In the field of object recognition, CNN is the best method because it has the advantage of being able to find its features of the object image by conducting the convolution process during training. CNN has several models or architectures; one of them is ResNet-50 or Residual Network. The selection of ResNet-50 architecture in this study aimed to reduce the loss of gradients at certain network-level depths during training because the object is a chest image of X-Ray that has a high level of visual similarity between some pathology. Moreover, several visual factors also affect the image so that to produce good accuracy requires a certain level of depth on the CNN network. Optimization during training used Adaptive Momentum (Adam) because it had a bias correction technique that provided better approximations to improve accuracy. The results of this study indicated the thorax image classification with an accuracy of 97.73%.

Pneumonia Classification of Thorax Images using Convolutional Neural Networks

The digital image processing technique is a product of computing technology development. Medical image data processing based on a computer is a product of computing technology development that can help a doctor to diagnose and observe a patient. This study aimed to perform classification on the image of the thorax by using Convolutional Neural Network (CNN). The data used in this study is lung thorax images that have previously been diagnosed by a doctor with two classes, namely normal and pneumonia. The amount of data is 2.200, 1.760 for training, and 440 for testing. Three stages are used in image processing, namely scaling, gray scaling, and scratching. This study used Convolutional Neural Network (CNN) method with architecture ResNet-50. In the field of object recognition, CNN is the best method because it has the advantage of being able to find its features of the object image by conducting the convolution process during training. CNN has several models or architectures; one of them is ResNet-50 or Residual Network. The selection of ResNet-50 architecture in this study aimed to reduce the loss of gradients at certain network-level depths during training because the object is a chest image of X-Ray that has a high level of visual similarity between some pathology. Moreover, several visual factors also affect the image so that to produce good accuracy requires a certain level of depth on the CNN network. Optimization during training used Adaptive Momentum (Adam) because it had a bias correction technique that provided better approximations to improve accuracy. The results of this study indicated the thorax image classification with an accuracy of 97.73%.

Use of trainable wavelet transform with adaptive threshold filtering for noise reduction of speech signals

A method of noise reduction based on an adaptive threshold filtering over trainable wavelet transform was proposed. The restrictions on the trainable transform filter parameters were provided by a set of quadratic regularization terms. As an analog of the “hard” and “soft” threshold functions we used their smooth infinitely differentiable versions. The parameters of trainable wavelet transform, and threshold values were estimated by backpropagation and gradient-based optimization algorithm with an adaptive momentum estimation. Results of the proposed method were compared with an approach based on fixed discrete wavelet transform and non-adaptive global and level-depended threshold algorithms on the model problem. We used signal-to-noise ratio between suppressed and clean signals to numerically estimate the efficiency of noise reduction. We showed that the best results were obtained when the proposed trainable method with Daubechies 4 wavelet filters fine-tuning and adaptive level-dependent thresholding were applied.

State-of-the-Art CNN Optimizer for Brain Tumor Segmentation in Magnetic Resonance Images.

Brain tumors have become a leading cause of death around the globe. The main reason for this epidemic is the difficulty conducting a timely diagnosis of the tumor. Fortunately, magnetic resonance images (MRI) are utilized to diagnose tumors in most cases. The performance of a Convolutional Neural Network (CNN) depends on many factors (i.e., weight initialization, optimization, batches and epochs, learning rate, activation function, loss function, and network topology), data quality, and specific combinations of these model attributes. When we deal with a segmentation or classification problem, utilizing a single optimizer is considered weak testing or validity unless the decision of the selection of an optimizer is backed up by a strong argument. Therefore, optimizer selection processes are considered important to validate the usage of a single optimizer in order to attain these decision problems. In this paper, we provides a comprehensive comparative analysis of popular optimizers of CNN to benchmark the segmentation for improvement. In detail, we perform a comparative analysis of 10 different state-of-the-art gradient descent-based optimizers, namely Adaptive Gradient (Adagrad), Adaptive Delta (AdaDelta), Stochastic Gradient Descent (SGD), Adaptive Momentum (Adam), Cyclic Learning Rate (CLR), Adaptive Max Pooling (Adamax), Root Mean Square Propagation (RMS Prop), Nesterov Adaptive Momentum (Nadam), and Nesterov accelerated gradient (NAG) for CNN. The experiments were performed on the BraTS2015 data set. The Adam optimizer had the best accuracy of 99.2% in enhancing the CNN ability in classification and segmentation.

Performance evaluation of optimized and adaptive neuro fuzzy inference system for predictive modeling in agriculture

The Neural Network has a significant impact in developing predictive models in a wide range of applications. In this paper, a neuro-fuzzy prediction model is developed depending on improving the performance of the traditional artificial neural networks using Adaptive Momentum Optimizer. This optimizer simulates the behavior of the International Trade Analysis in the agriculture industry, and this method is used to determine the optimal parameters of artificial neural networks. The proposed model is compared with the existing models such as Support Vector Machine, Random Forest, Decision Tree and traditional Artificial Neural Network models. To examine the forecasting performance of the proposed approach, agriculture datasets is used. The performance of the models was assessed using different performance evaluation criteria and the empirical results show that the back propagation neural network with Adam optimizer attains favorable prediction accuracy of 96.78%, and a better convergence rate. Compared to other benchmark algorithms, the proposed algorithm performs better, and the result validates the effectiveness of the back propagation with Adam optimizer for Natural Language Processing.

A Situation Assessment Method with an Improved Fuzzy Deep Neural Network for Multiple UAVs

To improve the intelligence and accuracy of the Situation Assessment (SA) in complex scenes, this work develops an improved fuzzy deep neural network approach to the situation assessment for multiple Unmanned Aerial Vehicle(UAV)s. Firstly, this work normalizes the scene data based on time series and use the normalized data as the input for an improved fuzzy deep neural network. Secondly, adaptive momentum and Elastic SGD (Elastic Stochastic Gradient Descent) are introduced into the training process of the neural network, to improve the learning performance. Lastly, in the real-time situation assessment task for multiple UAVs, conventional methods often bring inaccurate results for the situation assessment because these methods don’t consider the fuzziness of task situations. This work uses an improved fuzzy deep neural network to calculate the results of situation assessment and normalizes these results. Then, the degree of trust of the current result, relative to each situation label, is calculated with the normalized results using fuzzy logic. Simulation results show that the proposed method outperforms competitors.

Convolutional neural network modeling for classification of pulmonary tuberculosis disease

Tuberculosis (TB) is caused by bacteria (Mycobacterium tuberculosis) that most often affect the lungs so called Pulmonary Tuberculosis or PTB. Diagnosis using a chest radiograph image manually by doctor requires a long time, even difficult to detect PTB. Convolutional neural network (CNN) is a deep learning method that adopts the performance of human brain neurons called neural network and convolution functions to classify images. CNN can also help classify PTB based on chest radiograph images. This study uses data from the National Library of Medicine, Maryland, USA in collaboration with Shenzhen No.3 People’s Hospital, Guangdong Medical College, Shenzhen, China including 663 images entered into two classes, normal and PTB. This study uses adaptive momentum optimization (Adam) which serves to improve the accuracy of the model. The classification results of the models built were 99.19% for training data and 80.60% for validation data with 75 epochs, and accuracy in the test data was 84% which means that the model was able to qualify 84% of the test data into normal classes and PTB appropriately. 25 correctly classified as normal lungs, 5 incorrectly and 26 correctly classified as PTB and 5 incorrect.

A Robust Quasi-Newton Training with Adaptive Momentum for Microwave Circuit Models in Neural Networks

A Robust Quasi-Newton Training with Adaptive Momentum for Microwave Circuit Models in Neural Networks

Tensor p-shrinkage nuclear norm for low-rank tensor completion

In recent times, low-rank tensor completion (LRTC), which involves completing missing entries in partially observed tensors, is attracting significant attention from researchers. Several previously proposed LRTC methods have been successfully applied to practical applications. However, while the tensor nuclear norm has been successfully applied to solve the LRTC problem, penalizing all singular values equally restricts its recovery accuracy. In this study, we propose an efficient tensor p-shrinkage nuclear norm (p-TNN) based on tensor singular value decomposition (t-SVD). Under the condition −∞<p<1, we theoretically prove that p-TNN outperforms traditional tensor nuclear norm when they are used to approximate the tensor average rank. Based on the proposed p-TNN, we design an LRTC model to obtain an underlying tensor from its partial observations, which provides an upper bound for recovery error. In addition, we introduce an algorithm to solve the nonconvex optimization problem that originates in our developed model and accelerate this algorithm using an adaptive momentum scheme. Theoretical analyses indicate that our algorithm enjoys a globally geometric convergence rate under the smoothness assumption. Numerical experiments conducted on both synthetic and real-world data sets verify our method and demonstrate the superiority of our proposed p-TNN in solving LRTC problems over several state-of-the-art methods

Prediction Model of Material Removal for Polishing Single Crystal Silicon by Cluster Magnetorheological Finishing with Dynamic Magnetic Fields Based on BP Neural Network

Magnetorheological Finishing (MRF) is a new optical surface processing method, which has the advantages that good polishing effect, no subsurface damage, and suitable for complex surface processing. However, the interaction mechanism between the MRF pad and the workpiece is very complicated, so that the existing MRF material removal theoretical model is not accurate enough to establish the relationship between polishing parameters and material removal. In order to improve the processing efficiency and explore the material removal mechanism, a cluster magnetorheological finishing (CMRF) with dynamic magnetic fields method was proposed. Studying CMRF with dynamic magnetic fields material removal model is helpful to explain the removal mechanism more deeply, and improve the processing efficiency. In this study, the CMRF method was used to conduct a multi-factor orthogonal test on 2-inch single crystal silicon wafers. Based on the empirical Preston equation, the relationship between the machining gap and the polishing pressure was explained. Orthogonal experiments were done for a series of speeds, and obtaining the order of the influence of various factors on the average surface roughness <i>Ra</i> of the workpiece was: workpiece rotation speed > polishing disk speed > magnetic poles rotation speed > oscillating speed; the material removal rate (<i>MRR</i>) was: polishing disk speed > workpiece rotation speed > magnetic poles rotation speed > oscillating speed. Then combining with the orthogonal experimental data, and taking the surface roughness <i>Ra</i> and <i>MRR</i> as evaluation criteria, using Adam (Adaptive momentum) optimization algorithm to build a prediction model of <i>Ra</i> and <i>MRR</i> for polishing single crystal silicon by CMRF with dynamic magnetic fields based on BP neural network. For the prediction result, <i>Ra</i> of the maximum error was 7.05%, the minimum was 0.31%; <i>MRR</i> of the maximum error was 10.22%, the minimum was 1.32%. Therefore, the feasibility of this model for predicting the results of CMRF was verified, and it laid a good foundation for the development of CMRF technology and its industrial application.

Content-Based E-Commerce Image Classification Research

The 21st century is the era of big data in the Internet. Online shopping has become a trend, and e-commerce has developed rapidly. With the exponential increase of the amount of commodity image data, the management of massive commodity image database restricts the development of e-commerce to some extent. In order to effectively manage goods and improve the accuracy and efficiency of product image retrieval, this paper uses content-based methods to classify e-commerce images. Aiming at the problems of insufficient classification accuracy and long classification training time in e-commerce image classification, an adaptive momentum learning rate based LBP-DBN training algorithm-AML-LBP-DBN and commodity image classification method based on image local feature multi-level clustering and image-class nearest neighbor classifier are proposed. By simulating the commodity identification dataset RPC, the results show that the proposed method has obvious advantages in the classification training time and classification accuracy of e-commerce images.

Airfoil gust load alleviation by circulation control

This paper presents a numerical investigation on the gust load alleviation effects based on circulation control (CC) via blowing over airfoil trailing-edge Coanda surface. The NACA0012 airfoil was chosen for the study from subsonic to transonic speeds. The field velocity method is introduced to the unsteady Reynolds averaged Navier-Stokes solutions to simulate responses of the airfoil to arbitrarily shaped gust penetrations. For validation, gust responses to a step change in angle of attack, sharp-edged gusts and one-minus-cosine gusts from the numerical solutions are compared against available theories and other numerical references. Experimental data are used to validate the numerical results of CC. Thereafter circulation control via steady blowing with different momentum coefficients are tested for the gust load alleviation effects in terms of lift coefficients. The results show that circulation control can effectively suppress the lift disturbances caused by the gust for subsonic flows. Most importantly, CC has a fast frequency response characteristic as more than 50% of the total change in lift coefficient can be obtained within the non-dimensional time s=U∞t/b=1. Based on the rapid response characteristic, circulation control via unsteady blowing with dynamically adaptive momentum coefficients proportional to the vertical gust velocities is proposed and tested. The results demonstrate that a near constant lift coefficient can be achieved under gust condition for subsonic incoming flow indicating its potential for real-time adaptive load control. It has been demonstrated that CC is also able to reduce gust load at transonic speed, but it is less effective as compared with that at subsonic speed.

Fuzzy Reinforcement Learning and Curriculum Transfer Learning for Micromanagement in Multi-Robot Confrontation

Multi-Robot Confrontation on physics-based simulators is a complex and time-consuming task, but simulators are required to evaluate the performance of the advanced algorithms. Recently, a few advanced algorithms have been able to produce considerably complex levels in the context of the robot confrontation system when the agents are facing multiple opponents. Meanwhile, the current confrontation decision-making system suffers from difficulties in optimization and generalization. In this paper, a fuzzy reinforcement learning (RL) and the curriculum transfer learning are applied to the micromanagement for robot confrontation system. Firstly, an improved Qlearning in the semi-Markov decision-making process is designed to train the agent and an efficient RL model is defined to avoid the curse of dimensionality. Secondly, a multi-agent RL algorithm with parameter sharing is proposed to train the agents. We use a neural network with adaptive momentum acceleration as a function approximator to estimate the state-action function. Then, a method of fuzzy logic is used to regulate the learning rate of RL. Thirdly, a curriculum transfer learning method is used to extend the RL model to more difficult scenarios, which ensures the generalization of the decision-making system. The experimental results show that the proposed method is effective.

Application of Improved BPNN Algorithm in GIS Insulation Defect Type Identification

In this paper, a new BP neural network has been built with GA algorithm, which possesses high effectivity, parallel processing ability and global search feature, in order to overcome the original shortcomings such as the low convergence rate and the confusion of local minimum points. Ten parameters of partial discharge characteristics are obtained as input of BPNN to identify four typical insulation defect physical models designed in this paper for the purpose of measuring the recognition accuracy of the improved BPNN. Experiments show that the improved BPNN has a better performance of convergence speed and recognition accuracy than that of adaptive momentum BP neural network.

KLASIFIKASI PENYAKIT PNEUMONIA MENGGUNAKAN METODE CONVOLUTIONAL NEURAL NETWORK DENGAN OPTIMASI ADAPTIVE MOMENTUM

Pneumonia is an infection of the bacterium Streptococcus pneumoniae which causes inflammation in the air bag in one or both lungs. Pneumonia is a disease that can spread through the patient's air splashes. Pneumonia can be dangerous because it can cause death, therefore it is necessary to have early detection using chest radiograph images to determine the symptoms of pneumonia. Diagnosis using a chest radiograph image manually by medical personnel or a doctor requires a long time, even difficult to detect pneumonia disase. Convolutional neural network (CNN) is a deep learning method that adopts the performance of human brain neurons called neural network and convolution functions to classify images. CNN can also help classify pneumonia based on chest radiograph images. This study used data from Labeled Optical Coherence Tomography (OCT) and Chest X-Ray Images for Classification as many as 5860 images entered into two classes, namely normal and pneumonia, then 2400 data samples were taken using simple random sampling. This study uses adaptive momentum optimization (Adam) which serves to improve the accuracy of the model. Adam optimization is a development of existing optimizations such as Stochastic gradient descent (SGD), AdaGard, and RMSProp. The classification results of the models built were 99.98% for training data with 100 epochs, and accuracy in the test data was 78% which means that the model was able to qualify 78% of the test data into normal classes and pneumonia appropriately.

An Enhanced Optimization Scheme Based on Gradient Descent Methods for Machine Learning

A The learning process of machine learning consists of finding values of unknown weights in a cost function by minimizing the cost function based on learning data. However, since the cost function is not convex, it is conundrum to find the minimum value of the cost function. The existing methods used to find the minimum values usually use the first derivative of the cost function. When even the local minimum (but not a global minimum) is reached, since the first derivative of the cost function becomes zero, the methods give the local minimum values, so that the desired global minimum cannot be found. To overcome this problem, in this paper we modified one of the existing schemes—the adaptive momentum estimation scheme—by adding a new term, so that it can prevent the new optimizer from staying at local minimum. The convergence condition for the proposed scheme and the convergence value are also analyzed, and further explained through several numerical experiments whose cost function is non-convex.

Segmented and Non-Segmented Skin Lesions Classification Using Transfer Learning and Adaptive Moment Learning Rate Technique Using Pretrained Convolutional Neural Network

A skin lesion is a very severe problem, especially in coastal countries. Early detection by a highly reliable classification of skin lesion causes a great reduction in the mortality rate. Recognition of melanoma is a complicated issue due to the high degree of visual similarities between melanoma and non-melanoma lesions. Various studies are carried out to overcome this problem and to obtain accurate screening of skin lesion, where the most recent method for segmenting and classifying the lesion is based on a deep learning algorithm. In this paper, (GoogleNet) and (AlexNet) are employed with transfer learning and optimization gradient descent adaptive momentum learning rate (ADAM). The proposed method is applied on Archive International Skin Imaging Collaboration (ISIC) database to classify images into three main classes (benign, melanoma, seborrheic keratosis) under the two scenarios; segmented and non-segmented lesion images. The overall accuracy of the non-segmented classification database is 92.2% and 89.8% for the non-segmented dataset. Utilizing optimization algorithm (ADAM) leads to a significant improvement in the classification results when they are compared with previous studies.

Strong Convergence of Neuro-Fuzzy Learning With Adaptive Momentum for Complex System

This paper studies a split-complex-valued neuro-fuzzy algorithm for fuzzy inference system, which realizes a frequently used zero-order Takagi-Sugeno-Kang system. Here, adaptive momentum is utilized to speed up the learning convergence. Some strong convergence results are demonstrated based on the weak convergence results, which expresses that the weight sequence of fuzzy parameters converges to a fixed point. Simulation results support the theoretical findings.