Learning Rate Schedule Research Articles

Achieving generalization of deep learning models in a quick way by adapting T-HTR learning rate scheduler

Achieving generalization of deep learning models in a quick way by adapting T-HTR learning rate scheduler

Satellite and Scene Image Classification Based on Transfer Learning and Fine Tuning of ResNet50

Image classification has gained lot of attention due to its application in different computer vision tasks such as remote sensing, scene analysis, surveillance, object detection, and image retrieval. The primary goal of image classification is to assign the class labels to images according to the image contents. The applications of image classification and image analysis in remote sensing are important as they are used in various applied domains such as military and civil fields. Earlier approaches for remote sensing images and scene analysis are based on low-level feature representations such as color- and texture-based features. Vector of Locally Aggregated Descriptors (VLAD) and orderless Bag-of-Features (BoF) representations are the examples of mid-level approaches for remote sensing image classification. Recent trends for remote sensing and scene classification are focused on the use of Convolutional Neural Network (CNN). Keeping in view the success of CNN models, in this research, we aim to fine-tune ResNet50 by using network surgery and creation of network head along with the fine-tuning of hyperparameters. The learning of hyperparameters is tuned by using a linear decay learning rate scheduler known as piecewise scheduler. To tune the optimizer hyperparameter, Stochastic Gradient Descent with Momentum (SGDM) is used with the usage of weight learn and bias learn rate factor. Experiments and analysis are conducted on five different datasets, that is, UC Merced Land Use Dataset (UCM), RSSCN (the remote sensing scene classification image dataset), SIRI-WHU, Corel-1K, and Corel-1.5K. The analysis and competitive results exemplify that our proposed image classification-based model can classify the images in a more effective and efficient manner as compared to the state-of-the-art research.

Deep learning of resting-state electroencephalogram signals for three-class classification of Alzheimer’s disease, mild cognitive impairment and healthy ageing

Objective. This study aimed to produce a novel deep learning (DL) model for the classification of subjects with Alzheimer’s disease (AD), mild cognitive impairment (MCI) subjects and healthy ageing (HA) subjects using resting-state scalp electroencephalogram (EEG) signals. Approach. The raw EEG data were pre-processed to remove unwanted artefacts and sources of noise. The data were then processed with the continuous wavelet transform, using the Morse mother wavelet, to create time-frequency graphs with a wavelet coefficient scale range of 0–600. The graphs were combined into tiled topographical maps governed by the 10–20 system orientation for scalp electrodes. The application of this processing pipeline was used on a data set of resting-state EEG samples from age-matched groups of 52 AD subjects (82.3 ± 4.7 years of age), 37 MCI subjects (78.4 ± 5.1 years of age) and 52 HA subjects (79.6 ± 6.0 years of age). This resulted in the formation of a data set of 16197 topographical images. This image data set was then split into training, validation and test images and used as input to an AlexNet DL model. This model was comprised of five hidden convolutional layers and optimised for various parameters such as learning rate, learning rate schedule, optimiser, and batch size. Main results. The performance was assessed by a tenfold cross-validation strategy, which produced an average accuracy result of 98.9 ± 0.4% for the three-class classification of AD vs MCI vs HA. The results showed minimal overfitting and bias between classes, further indicating the strength of the model produced. Significance. These results provide significant improvement for this classification task compared to previous studies in this field and suggest that DL could contribute to the diagnosis of AD from EEG recordings.

ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques

Pre-trained language models of the BERT family have defined the state-of-the-arts in a wide range of NLP tasks. However, the performance of BERT-based models is mainly driven by the enormous amount of parameters, which hinders their application to resource-limited scenarios. Faced with this problem, recent studies have been attempting to compress BERT into a small-scale model. However, most previous work primarily focuses on a single kind of compression technique, and few attention has been paid to the combination of different methods. When BERT is compressed with integrated techniques, a critical question is how to design the entire compression framework to obtain the optimal performance. In response to this question, we integrate three kinds of compression methods (weight pruning, low-rank factorization and knowledge distillation (KD)) and explore a range of designs concerning model architecture, KD strategy, pruning frequency and learning rate schedule. We find that a careful choice of the designs is crucial to the performance of the compressed model. Based on the empirical findings, our best compressed model, dubbed Refined BERT cOmpreSsion with InTegrAted techniques (ROSITA), is 7.5x smaller than BERT while maintains 98.5% of the performance on five tasks of the GLUE benchmark, outperforming the previous BERT compression methods with similar parameter budget.

Nondestructive multiplex detection of foodborne pathogens with background microflora and symbiosis using a paper chromogenic array and advanced neural network

We have developed an inexpensive, standardized paper chromogenic array (PCA) integrated with a machine learning approach to accurately identify single pathogens (Listeria monocytogenes, Salmonella Enteritidis, or Escherichia coli O157:H7) or multiple pathogens (either in multiple monocultures, or in a single cocktail culture), in the presence of background microflora on food. Cantaloupe, a commodity with significant volatile organic compound (VOC) emission and large diverse populations of background microflora, was used as the model food. The PCA was fabricated from a paper microarray via photolithography and paper microfluidics, into which 22 chromogenic dye spots were infused and to which three red/green/blue color-standard dots were taped. When exposed to VOCs emitted by pathogens of interest, dye spots exhibited distinguishable color changes and pattern shifts, which were automatically segmented and digitized into a ΔR/ΔG/ΔB database. We developed an advanced deep feedforward neural network with a learning rate scheduler, L2 regularization, and shortcut connections. After training on the ΔR/ΔG/ΔB database, the network demonstrated excellent performance in identifying pathogens in single monocultures, multiple monocultures, and in cocktail culture, and in distinguishing them from the background signal on cantaloupe, providing accuracy of up to 93% and 91% under ambient and refrigerated conditions, respectively. With its combination of speed, reliability, portability, and low cost, this nondestructive approach holds great potential to significantly advance culture-free pathogen detection and identification on food, and is readily extendable to other food commodities with complex microflora.

State-of-Charge Estimation of Li-ion Battery Using Gated Recurrent Unit With One-Cycle Learning Rate Policy

Deep learning has gained much traction in application to state-of-charge (SOC) estimation for Li-ion batteries in electric vehicle applications. However, with the vast selection of architectures and hyperparameter combinations, it remains challenging to design an accurate and robust SOC estimation model with a sufficiently low computation cost. Therefore, this study provides a comparative evaluation among commonly used deep learning models from the recurrent, convolutional, and feedforward architecture benchmarked on an openly available Li-ion battery dataset. To evaluate model robustness and generalization capability, we train and test models on different drive cycles at various temperatures and compute the root mean squared error (RMSE) and mean absolute error metric. To evaluate model computation costs, we run models in real-time and record the model size, floating-point operations per second (FLOPS), and run-time duration per datapoint. This study proposes a two-hidden layer stacked gated recurrent unit model trained with a one-cycle policy learning rate scheduler. The proposed model achieves a minimum RMSE of 0.52% on the train dataset and 0.65% on the test dataset while maintaining a relatively low computation cost. Executing the proposed model in real-time takes up approximately 1 MB in disk space, 300K FLOPS, and 0.03 ms run-time per datapoint. This makes the proposed model feasible to be executed on lightweight battery management system processors.

Learning Rate Optimization in CNN for Accurate Ophthalmic Classification

One of the most important hyper-parameters for model training and generalization is the learning rate. Recently, many research studies have shown that optimizing the learning rate schedule is very useful for training deep neural networks to get accurate and efficient results. In this paper, different learning rate schedules using some comprehensive optimization techniques have been compared in order to measure the accuracy of a convolutional neural network CNN model to classify four ophthalmic conditions. In this work, a deep learning CNN based on Keras and TensorFlow has been deployed using Python on a database that contains 1692 images, which consists of four types of ophthalmic cases: Glaucoma, Myopia, Diabetic retinopathy, and Normal eyes. The CNN model has been trained on Google Colab. GPU with different learning rate schedules and adaptive learning algorithms. Constant learning rate, time-based decay, step-based decay, exponential decay, and adaptive learning rate optimization techniques for deep learning have been addressed. Adam adaptive learning rate method. has outperformed the other optimization techniques and achieved the best model accuracy of 92.58% for training set and 80.49% for validation datasets, respectively.

CrackWeb : A modified U-Net based segmentation architecture for crack detection

Classical image processing methods demands heavy feature engineering, as well as they are not that precise, when it comes to manual extraction of relevant features in real life scenarios amid to various lighting conditions and other factors.Thus, detection of cracks using methods based on classical image processing techniques fails to provide satisfactory results always. Hence, we have proposed a deep convolutional neural network, that is not based on manual extraction of features as mentioned above. We proposed a modified U-Net architecture, and replaced all of its convolutional layers with residual blocks, inspired from the ResNet architecture. For evaluation of our model Dice Loss is used as our objective function and F1 score as a metric. Other than that, for better convergence and optimization, a learning rate scheduler and AMSGRAD optimizer was utilized.

An Adaptive Optimization Method Based on Learning Rate Schedule for Neural Networks

Artificial intelligence (AI) is achieved by optimizing the cost function constructed from learning data. Changing the parameters in the cost function is an AI learning process (or AI learning for convenience). If AI learning is well performed, then the value of the cost function is the global minimum. In order to obtain the well-learned AI learning, the parameter should be no change in the value of the cost function at the global minimum. One useful optimization method is the momentum method; however, the momentum method has difficulty stopping the parameter when the value of the cost function satisfies the global minimum (non-stop problem). The proposed method is based on the momentum method. In order to solve the non-stop problem of the momentum method, we use the value of the cost function to our method. Therefore, as the learning method processes, the mechanism in our method reduces the amount of change in the parameter by the effect of the value of the cost function. We verified the method through proof of convergence and numerical experiments with existing methods to ensure that the learning works well.

Convolutional Neural Network With Automatic Learning Rate Scheduler for Fault Classification

Fault classification is vital in smart manufacturing, and convolutional neural network (CNN) has been widely applied in fault classification. But the performance of CNN heavily depends on its learning rate. As the default setting on learning rate cannot guarantee its performance, the learning rate tuning process becomes essential. However, the traditional learning rate tuning methods either cost much time consumption or rely on the experts’ experiences, so it is a considerable barrier for the users. To overcome this drawback, this article proposes a CNN with automatic learning rate scheduler (AutoLR-CNN) for fault classification. First, the long short-term memory (LSTM) is used to extract the features of the past loss of CNN. Then, an agent based on deep deterministic policy gradient (DDPG) is trained to automatically control the learning rate for CNN online. Third, the double CNN structure is developed to enhance the stability of the proposed method. The proposed AutoLR-CNN is tested on two famous bearing data sets and a practical bearing data set on wind turbine. The results of AutoLR-CNN are superior to six commonly used baseline learning rate schedulers in Tensorflow. AutoLR-CNN is also compared with other reported machine learning and deep learning methods. The results show that AutoLR-CNN has achieved the state-of-the-art performance in fault classification.

Performance Investigation of Learning Rate Decay in LMS-Based Equalization

In this letter, we investigate the performances of different learning rate decay schedules in least mean square (LMS) based equalization. The required iteration numbers for convergence are 275, 150 and 150 for time-based, staircase and exponential decay schedules respectively, which are much less than 1000, the required iteration number for constant learning rate schedule in processing our experimental data. Meanwhile, the corresponding training times are reduced to shorter than one third, one fifth and one fourth of that in constant schedule. The system performances in the aspects of carrier to signal power ratio and optical signal to noise ratio are measured in a 110 Gbit/s probabilistically shaped 64 QAM direct detection system to evaluate different learning rate decay schedules.

A New Reinforcement Learning Based Learning Rate Scheduler for Convolutional Neural Network in Fault Classification

Convolutional neural network (CNN) has gained increasing attention in fault classification. However, the performance of CNN is sensitive to its learning rate. Some previous works have been done to tune the learning rate, including the “trial and error” and manual search, which heavily depend on the experts' experiences and should be conducted repeatedly on every dataset. Because of the variety of the fault data, it is time-consuming and labor intensive to use these traditional tuning methods for fault classification. To overcome this problem, in this article, we develop a novel learning rate scheduler based on the reinforcement learning (RL) for convolutional neural network (RL-CNN) in fault classification, which can schedule the learning rate efficiently and automatically. First, a new RL agent is designed to learn the policies about the learning rate adjustment during the training process. Second, a new structure of RL-CNN is developed to balance the exploration and exploitation of the agent. Third, the bagging ensemble version of RL-CNN (RL-CNN-Ens) is presented. Three bearing datasets are used to test the performance of RL-CNN-Ens. The results show that RL-CNN-Ens outperforms the traditional DLs and machine learning methods. Meanwhile, RL-CNN-Ens can find the state-of-the-art learning rate schedulers as human designed, showing its potential in fault classification.

Toward Optimal Learning Rate Schedule in Scene Classification Network

Stochastic gradient descent (SGD) and adaptive methods, including ADAM, RMSProp, AdaDelta, and AdaGrad, are two dominant optimization algorithms for training convolution neural network (CNN) in scene classification tasks. Recent work reveals that these adaptive methods lead to degraded performance in image classification tasks compared with SGD. In this letter, a learning rate schedule named switching from constant to step decay (SCTSD) is proposed to further improve the classification accuracy of SGD. SCTSD begins with a constant learning rate and switches to step decay learning rates when appropriate. Theoretical evidence is provided on the superiority of SCTSD compared with other manually tuned schedules. Comparison experiments have been conducted among adaptive methods, SCTSD, and other manual schedules with three CNN architectures. The experiment results on various scene classification data sets show that SCTSD has the highest accuracy on the test set and it is state of the art. In the end, some suggestions on hyperparameters selection of SCTSD are given for scene classification.

A novel hand-crafted with deep learning features based fusion model for COVID-19 diagnosis and classification using chest X-ray images

COVID-19 pandemic is increasing in an exponential rate, with restricted accessibility of rapid test kits. So, the design and implementation of COVID-19 testing kits remain an open research problem. Several findings attained using radio-imaging approaches recommend that the images comprise important data related to coronaviruses. The application of recently developed artificial intelligence (AI) techniques, integrated with radiological imaging, is helpful in the precise diagnosis and classification of the disease. In this view, the current research paper presents a novel fusion model hand-crafted with deep learning features called FM-HCF-DLF model for diagnosis and classification of COVID-19. The proposed FM-HCF-DLF model comprises three major processes, namely Gaussian filtering-based preprocessing, FM for feature extraction and classification. FM model incorporates the fusion of handcrafted features with the help of local binary patterns (LBP) and deep learning (DL) features and it also utilizes convolutional neural network (CNN)-based Inception v3 technique. To further improve the performance of Inception v3 model, the learning rate scheduler using Adam optimizer is applied. At last, multilayer perceptron (MLP) is employed to carry out the classification process. The proposed FM-HCF-DLF model was experimentally validated using chest X-ray dataset. The experimental outcomes inferred that the proposed model yielded superior performance with maximum sensitivity of 93.61%, specificity of 94.56%, precision of 94.85%, accuracy of 94.08%, F score of 93.2% and kappa value of 93.5%.

Associated Learning: Decomposing End-to-End Backpropagation Based on Autoencoders and Target Propagation.

Backpropagation (BP) is the cornerstone of today's deep learning algorithms, but it is inefficient partially because of backward locking, which means updating the weights of one layer locks the weight updates in the other layers. Consequently, it is challenging to apply parallel computing or a pipeline structure to update the weights in different layers simultaneously. In this letter, we introduce a novel learning structure, associated learning (AL), that modularizes the network into smaller components, each of which has a local objective. Because the objectives are mutually independent, AL can learn the parameters in different layers independently and simultaneously, so it is feasible to apply a pipeline structure to improve the training throughput. Specifically, this pipeline structure improves the complexity of the training time from , which is the time complexity when using BP and stochastic gradient descent (SGD) for training, to , where is the number of training instances and is the number of hidden layers. Surprisingly, even though most of the parameters in AL do not directly interact with the target variable, training deep models by this method yields accuracies comparable to those from models trained using typical BP methods, in which all parameters are used to predict the target variable. Consequently, because of the scalability and the predictive power demonstrated in the experiments, AL deserves further study to determine the better hyperparameter settings, such as activation function selection, learning rate scheduling, and weight initialization, to accumulate experience, as we have done over the years with the typical BP method. In addition, perhaps our design can also inspire new network designs for deep learning. Our implementation is available at https://github.com/SamYWK/Associated_Learning.

Musculoskeletal Abnormality Detection in Medical Imaging Using GnCNNr (Group Normalized Convolutional Neural Networks with Regularization)

Musculoskeletal abnormality detection serves as an advantage to the professionals in the medical domain and also serves as an assistance in the diagnosis as well as the treatment of the abnormalities. This paper mainly focuses on accurately detecting musculoskeletal abnormalities using various deep learning models and techniques. MURA dataset has been used for experimentation. MURA dataset has 14,863 images of finger, wrist, elbow, shoulder, forearm and hand which has been analyzed using deep learning models. In this research paper, authors have proposed GnCNNr model which utilizes group normalization, weight standardization and cyclic learning rate scheduler to enhance the accuracy, precision and other model interpretation metrics. The musculoskeletal abnormality has been detected by using various deep learning models. Accuracy and Cohen Kappa have been taken as the evaluation criteria. The highest accuracy of 85% and Cohen Kappa statistic of 0.698 was achieved by the GnCNNr model in comparison with the conventional deep learning methods like DenseNet, Inception, Inception v2 model.

On the impact of selected modern deep-learning techniques to the performance and celerity of classification models in an experimental high-energy physics use case

Beginning from a basic neural-network architecture, we test the potential benefits offered by a range of advanced techniques for machine learning, in particular deep learning, in the context of a typical classification problem encountered in the domain of high-energy physics, using a well-studied dataset: the 2014 Higgs ML Kaggle dataset. The advantages are evaluated in terms of both performance metrics and the time required to train and apply the resulting models. Techniques examined include domain-specific data-augmentation, learning rate and momentum scheduling, (advanced) ensembling in both model-space and weight-space, and alternative architectures and connection methods.Following the investigation, we arrive at a model which achieves equal performance to the winning solution of the original Kaggle challenge, whilst being significantly quicker to train and apply, and being suitable for use with both GPU and CPU hardware setups. These reductions in timing and hardware requirements potentially allow the use of more powerful algorithms in HEP analyses, where models must be retrained frequently, sometimes at short notice, by small groups of researchers with limited hardware resources. Additionally, a new wrapper library for PYTORCH called LUMIN is presented, which incorporates all of the techniques studied.

Aerial Scene Classification through Fine-Tuning with Adaptive Learning Rates and Label Smoothing

Remote Sensing (RS) image classification has recently attracted great attention for its application in different tasks, including environmental monitoring, battlefield surveillance, and geospatial object detection. The best practices for these tasks often involve transfer learning from pre-trained Convolutional Neural Networks (CNNs). A common approach in the literature is employing CNNs for feature extraction, and subsequently train classifiers exploiting such features. In this paper, we propose the adoption of transfer learning by fine-tuning pre-trained CNNs for end-to-end aerial image classification. Our approach performs feature extraction from the fine-tuned neural networks and remote sensing image classification with a Support Vector Machine (SVM) model with linear and Radial Basis Function (RBF) kernels. To tune the learning rate hyperparameter, we employ a linear decay learning rate scheduler as well as cyclical learning rates. Moreover, in order to mitigate the overfitting problem of pre-trained models, we apply label smoothing regularization. For the fine-tuning and feature extraction process, we adopt the Inception-v3 and Xception inception-based CNNs, as well the residual-based networks ResNet50 and DenseNet121. We present extensive experiments on two real-world remote sensing image datasets: AID and NWPU-RESISC45. The results show that the proposed method exhibits classification accuracy of up to 98%, outperforming other state-of-the-art methods.

Abstract 2100: Deep learning can predict microsatellite instability from histology in colorectal cancer across different ethnic groups

Abstract Microsatellite instability (MSI) is known to be an important genomic marker for prediction of a patient's response to immune checkpoint inhibitors across various types of cancer. Conventional methods to measure DNA mismatch repair (MMR) status are pCR-based panel sequencing or immunohistochemical staining based methods. Recently, deep learning has been proposed as an efficient diagnostic tool in this regard, as it can learn to predict MSI directly from H&E stained slides. However, it is thought that deep learning may not generalize effectively beyond the ethnicities present in the training dataset. Therefore, we investigated the impact of ethnic differences on deep learning performance at MSI classification across a range of model architectures and hyperparameters. A total of 360 WSIs from a subset of the TCGA colorectal cancer (TCGA set) and another 170 WSIs from Gangnam Severance Hospital (Yonsei University College of Medicine, Seoul, Republic of Korea, Gangnam set) were included in this study. 360 H&E stained WSIs from TCGA set were divided into a set of square tiles of size 256x256um (0.5um/pixel) and then color normalized using Macenko normalization. Those tiles were divided into two classes - MSI-H and MSS (Microsatellite stable) based on the patient's MSI status. We trained various models on those tiles using a range of hyperparameters including model architecture, number of trainable layers, learning rate, and learning rate scheduling. The performance of this model was measured using the AUROC in the TCGA set (internal validation) and Gangnam set (external validation). Our results for ResNet18 - a popular convolutional network architecture - on TCGA and Gangnam test datasets are around 0.79 and 0.76 respectively, with a standard deviation on Gangnam of 0.03. Given good hyperparameters and model architecture, the model's performance on the Gangnam test data was within one standard deviation of its performance on the TCGA test data, despite the fact that said model was trained using only the TCGA data containing few asian patients. The lack of a significant performance difference challenges the notion that there are morphological differences in colorectal cancers between patients of Caucasian and Asian ethnicities that would prevent a deep learning system from generalizing effectively. Citation Format: Isaiah S. Pressman, Hongming Xu, Jeonghyun Kang, Yoon Jin Cha, Sung Hak Lee, Tae Hyun Hwang. Deep learning can predict microsatellite instability from histology in colorectal cancer across different ethnic groups [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 2100.

Lung Sound Classification Using Snapshot Ensemble of Convolutional Neural Networks.

We propose a robust and efficient lung sound classification system using a snapshot ensemble of convolutional neural networks (CNNs). A robust CNN architecture is used to extract high-level features from log mel spectrograms. The CNN architecture is trained on a cosine cycle learning rate schedule. Capturing the best model of each training cycle allows to obtain multiple models settled on various local optima from cycle to cycle at the cost of training a single mode. Therefore, the snapshot ensemble boosts performance of the proposed system while keeping the drawback of expensive training of ensembles moderate. To deal with the class-imbalance of the dataset, temporal stretching and vocal tract length perturbation (VTLP) for data augmentation and the focal loss objective are used. Empirically, our system outperforms state-of-the-art systems for the prediction task of four classes (normal, crackles, wheezes, and both crackles and wheezes) and two classes (normal and abnormal (i.e. crackles, wheezes, and both crackles and wheezes)) and achieves 78.4% and 83.7% ICBHI specific micro-averaged accuracy, respectively. The average accuracy is repeated on ten random splittings of 80% training and 20% testing data using the ICBHI 2017 dataset of respiratory cycles.