Cyclical Learning Rate Research Articles

Provable Super-Convergence With a Large Cyclical Learning Rate

Conventional wisdom dictates that learning rate should be in the stable regime so that gradient-based algorithms don't blow up. This letter introduces a simple scenario where an unstably large learning rate scheme leads to a super fast convergence, with the convergence rate depending only logarithmically on the condition number of the problem. Our scheme uses a Cyclical Learning Rate (CLR) where we periodically take one large unstable step and several small stable steps to compensate for the instability. These findings also help explain the empirical observations of [Smith and Topin, 2019] where they show that CLR with a large maximum learning rate can dramatically accelerate learning and lead to so-called "super-convergence". We prove that our scheme excels in the problems where Hessian exhibits a bimodal spectrum and the eigenvalues can be grouped into two clusters (small and large). The unstably large step is the key to enabling fast convergence over the small eigen-spectrum.

An Empirical Analysis of Deep Learning Architectures for Vehicle Make and Model Recognition

Vehicle make and model recognition is an important component of the Intelligent Transport System (ITS), which plays an essential role in the vehicle surveillance and traffic monitoring. In this paper, we evaluated the performance of the recent deep neural networks for vehicle recognition to identify 196 different types of vehicles based on their make, model, and year using Stanford Cars data. Transfer learning has been employed to reduce the training time and also added a dropout of 0.5 at the last dense layer before the output layer for improved generalization performance on an applied dataset. Different augmentation methods such as random erasing, horizontal flip, random crop, resize, rotate, shear, and sharpen were applied to the training dataset to increase the number of images. Besides, optimization methods namely mixup and cyclical learning rate were also investigated to train these deep neural networks to avoid overfitting. Finally, the proposed ensemble learning framework using k-fold cross-validation has resulted in the improved accuracy of image classification for the vehicle make and model recognition. The final achieved accuracy is 93.96% through ensembling homogeneous models of DenseNet201, which is higher than previous state-of-the-art approaches for the vehicle make and model recognition. Simultaneously, it has also improved the performance of other models where MobileNetV2 has shown the highest improvement with an increase of 2.8% in the overall accuracy using the proposed ensemble learning technique.

Optimal hyperparameter selection of deep learning models for COVID-19 chest X-ray classification

The first and most critical response to curbing the spread of the novel coronavirus disease (COVID-19) is to deploy effective techniques to test potentially infected patients, isolate them and commence immediate treatment. However, several test kits currently in use are slow and in a shortage of supply. This paper presents techniques for diagnosing COVID-19 from chest X-ray (CXR) and address problems associated with training deep models with less voluminous datasets and class imbalance as obtained in most available CXR datasets on COVID-19. We used the discriminative fine-tuning approach, which dynamically assigns different learning rates to each layer of the network. The learning rate is set using the cyclical learning rate policy that changes per iteration. This flexibility ensured rapid convergence and avoided being stuck in saddle point plateau. In addition, we addressed the high computational demand of deep models by implementing our algorithm using the memory- and computational-efficient mixed-precision training. Despite the availability of scanty datasets, our model achieved high performance and generalisation. A Validation accuracy of 96.83%, sensitivity and specificity of 96.26% and 95.54% were obtained, respectively. When tested on an entirely new dataset, the model achieves 97% accuracy without further training. Lastly, we presented a visual interpretation of the model’s output to prove that the model can aid radiologists in rapidly screening for the symptoms of COVID-19.

Contrast CT classification of asymptomatic and symptomatic carotids in stroke and transient ischaemic attack with deep learning and interpretability

Abstract Background Convolutional neural networks (CNNs), part of deep learning, are used widely for computer vision tasks and in some medical domains, such as mammography interpretation. The application of deep learning to carotid artery imaging is scarce. We investigated the ability of deep learning to correctly classify contrast CT images of the carotid arteries without the need for prior feature selection. Purpose (1) To assess the ability of deep learning to differentiate symptomatic patients (had prior stroke or transient ischaemic attack [TIA]) from asymptomatic patients (no prior stroke/TIA) using contrast CT scans alone. (2) To investigate whether deep learning can further discriminate between culprit and non-culprit carotid arteries in symptomatic patients. (3) To assess the interpretability of the deep learning models. Methods Carotid contrast CT scans of consented research subjects were included in the study. Symptomatic patients had confirmed carotid artery-related ischaemic stroke or TIA in the 7 days before CT imaging, and asymptomatic patients had no prior cerebrovascular events. The dataset comprised 1148 axial symptomatic slices (covering a 3cm area of each carotid artery in 41 patients; 41 culprit and 41 non-culprit carotids) and 700 asymptomatic slices (from the bilateral carotid arteries of 25 patients). The dataset was split such that 75% was used for training and 25% for testing. A 30x30 bounding box was used to create patches of the carotid arteries from these axial slices for use as input to the CNN, a modified VGG16 architecture initialised with ImageNet weights to leverage transfer learning (the application of a model trained in one domain to a different domain) implemented in Python. Data augmentation was applied to the training set and the model was trained for 100 epochs using a cyclic learning rate, the RMSProp optimizer and binary cross-entropy loss. Class activation heatmaps were generated using the GradCAM method to highlight the areas of the image that were most important to the model for making its classification decision. Results The deep learning model was 92% accurate in correctly identifying carotid arteries from symptomatic patients versus those from asymptomatic patients. Discriminating between culprit versus non-culprit carotid arteries in symptomatic patients alone was 71% accurate. The class activation heatmaps demonstrated how the model learnt to localise the carotid artery within the image patch, and to ignore the arterial lumen when making its classification decision. Conclusions Deep learning can be used to differentiate between symptomatic and asymptomatic carotid CT scans from stroke/TIA subjects without the need for prior feature engineering. The model learns to identify relevant features in the image that predict the patients' symptom state. If further validated, this approach could be used to identify high-risk patients for intensive medical therapy. Funding Acknowledgement Type of funding source: Public grant(s) – National budget only. Main funding source(s): EPVL is undertaking a PhD funded by the Cambridge School of Clinical Medicine and the Medical Research Council's Doctoral Training Partnership

Automatic Segmentation Using Deep Learning for Online Dose Optimization During Adaptive Radiotherapy of Cervical Cancer

A phase II clinical trial was designed to propose daily online dose optimization to eliminate need for ITV expansions in order to reduce toxicity events of EBRT for cervical cancer. This study investigates deep learning models for automatic segmentation. The cervix-uterus, vagina, bladder, rectum, sigmoid, femoral heads, kidneys, spinal cord, and bowel bag were delineated on 223 CT scans from 157 patients. The data were divided into 149 training, 36 validation and 38 test CTs. Two models were investigated: the multiclass 2D DeepLabV3+ and the 2-steps 3D U-net of RayStation. The 2D model was trained with a window optimizer of 3 channels, triangular cyclic learning rate, no dropout and no augmentation. The 3D model was trained with batch normalization, data augmentation (translation, rotation and deformation) and no dropout. Both models were trained with the Adam optimizer. Three IMRT plans were generated on the planning CT of 20 unique cases from the test set using the manual, 2D or 3D model contours following the EMBRACE II constraints with reduced PTVT/N margins of 5/3 mm. Both models were compared with manual contours on the validation and test sets using the Dice coefficient (DC). Using manual contours as reference, clinical dose indices were extracted on the manual, 2D and 3D model optimized doses for comparison. The mean (min – max) of the average DC of all organs on the validation and test sets were 0.86 (0.61 – 0.93) and 0.85 (0.67 – 0.93) for the 2D model and 0.84 (0.70 – 0.93) and 0.82 (0.65 – 0.92) for the 3D model. The sigmoid and vagina were the most challenging organs with a DC range of 0.61 – 0.72 for both models on the validation and test sets. The 2D model provided better DC for the spinal and bowel on all cases (p< 0.001). The 3D model provided a significantly better DC for the sigmoid on the validation set (p = 0.001), otherwise no difference was found. Removing the spinal and bowel from analysis, the 3D model provided a superior DC for 60% of all cases. The mean (min – max) DC of the CTVT on the 20 test cases were 0.89 (0.75 – 0.93) and 0.88 (0.69 – 0.95) for the 2D and 3D models. Using the manual CTVT on the 2D and 3D model optimized doses, 90% and 80% passed the V42.75Gy>98% (p = 0.07). The remaining 10% and 20% cases had a mean V42.75Gy of 95.6% for 2D and 93.7% for 3D model. The V42.75Gy decrease of the 3D model was significantly correlated with the DC of the CTVT and PTVT (R2 = 0.56 and R2 = 0.5 p<0.001). A decrease of 0.1 in DC was equivalent to a decrease of 3.1% and 5.5% of the V42.75Gy for the CTVT and PTVT. For all organs, 91%, 80%, and 80% of clinical constraints were met for the manual, 2D and 3D model optimized doses. The 2D model showed better robustness while the 3D model provided better representation of complex organ shape. Deep learning model-based segmentation is promising for implementing reduced margin online dose optimization during EBRT of cervical cancer.

Musculoskeletal Abnormality Detection in Medical Imaging Using GnCNNr (Group Normalized Convolutional Neural Networks with Regularization)

Musculoskeletal abnormality detection serves as an advantage to the professionals in the medical domain and also serves as an assistance in the diagnosis as well as the treatment of the abnormalities. This paper mainly focuses on accurately detecting musculoskeletal abnormalities using various deep learning models and techniques. MURA dataset has been used for experimentation. MURA dataset has 14,863 images of finger, wrist, elbow, shoulder, forearm and hand which has been analyzed using deep learning models. In this research paper, authors have proposed GnCNNr model which utilizes group normalization, weight standardization and cyclic learning rate scheduler to enhance the accuracy, precision and other model interpretation metrics. The musculoskeletal abnormality has been detected by using various deep learning models. Accuracy and Cohen Kappa have been taken as the evaluation criteria. The highest accuracy of 85% and Cohen Kappa statistic of 0.698 was achieved by the GnCNNr model in comparison with the conventional deep learning methods like DenseNet, Inception, Inception v2 model.

Remote Sensing Scene Classification and Explanation Using RSSCNet and LIME

Classification is needed in disaster investigation, traffic control, and land-use resource management. How to quickly and accurately classify such remote sensing imagery has become a popular research topic. However, the application of large, deep neural network models for the training of classifiers in the hope of obtaining good classification results is often very time-consuming. In this study, a new CNN (convolutional neutral networks) architecture, i.e., RSSCNet (remote sensing scene classification network), with high generalization capability was designed. Moreover, a two-stage cyclical learning rate policy and the no-freezing transfer learning method were developed to speed up model training and enhance accuracy. In addition, the manifold learning t-SNE (t-distributed stochastic neighbor embedding) algorithm was used to verify the effectiveness of the proposed model, and the LIME (local interpretable model, agnostic explanation) algorithm was applied to improve the results in cases where the model made wrong predictions. Comparing the results of three publicly available datasets in this study with those obtained in previous studies, the experimental results show that the model and method proposed in this paper can achieve better scene classification more quickly and more efficiently.

Aerial Scene Classification through Fine-Tuning with Adaptive Learning Rates and Label Smoothing

Remote Sensing (RS) image classification has recently attracted great attention for its application in different tasks, including environmental monitoring, battlefield surveillance, and geospatial object detection. The best practices for these tasks often involve transfer learning from pre-trained Convolutional Neural Networks (CNNs). A common approach in the literature is employing CNNs for feature extraction, and subsequently train classifiers exploiting such features. In this paper, we propose the adoption of transfer learning by fine-tuning pre-trained CNNs for end-to-end aerial image classification. Our approach performs feature extraction from the fine-tuned neural networks and remote sensing image classification with a Support Vector Machine (SVM) model with linear and Radial Basis Function (RBF) kernels. To tune the learning rate hyperparameter, we employ a linear decay learning rate scheduler as well as cyclical learning rates. Moreover, in order to mitigate the overfitting problem of pre-trained models, we apply label smoothing regularization. For the fine-tuning and feature extraction process, we adopt the Inception-v3 and Xception inception-based CNNs, as well the residual-based networks ResNet50 and DenseNet121. We present extensive experiments on two real-world remote sensing image datasets: AID and NWPU-RESISC45. The results show that the proposed method exhibits classification accuracy of up to 98%, outperforming other state-of-the-art methods.

Deep learning-based multi-modal approach using RGB and skeleton sequences for human activity recognition

The deep learning techniques have achieved great success in the application of human activity recognition (HAR). In this paper, we propose a technique for HAR that utilizes the RGB and skeleton information with the help of a convolutional neural network (Convnet) and long short-term memory (LSTM) as a recurrent neural network (RNN). The proposed method has two parts: first, motion representation images like motion history image (MHI) and motion energy image (MEI) have been created from the RGB videos. The convnet has been trained, using these images with feature-level fusion. Second, the skeleton data have been utilized with a proposed algorithm that develops skeleton intensity images, for three views (top, front and side). Each view is first analyzed by a convnet, that generates the set of feature maps, which are fused for further analysis. On top of convnet sub-networks, LSTM has been used to exploit the temporal dependency. The softmax scores from these two independent parts are later combined at the decision level. Apart from the given approach for HAR, this paper also presents a strategy that utilizes the concept of cyclic learning rate to develop a multi-modal neural network by training the model only once to make the system more efficient. The suggested approach privileges for the perfect utilization of RGB and skeleton data available from an RGB-D sensor. The proposed approach has been tested on three famous and challenging multimodal datasets which are UTD-MHAD, CAD-60 and NTU-RGB + D120. Results have shown that the stated method gives a satisfactory result as compared to the other state-of-the-art systems.

Collaborative deep learning across multiple data centers

Valuable training data is often owned by independent organizations and located in multiple data centers. Most deep learning approaches require to centralize the multi-datacenter data for performance purpose. In practice, however, it is often infeasible to transfer all data of different organizations to a centralized data center owing to the constraints of privacy regulations. It is very challenging to conduct the geo-distributed deep learning among data centers without the privacy leaks. Model averaging is a conventional choice for data parallelized training and can reduce the risk of privacy leaks, but its ineffectiveness is claimed by previous studies as deep neural networks are often non-convex. In this paper, we argue that model averaging can be effective in the decentralized environment by using two strategies, namely, the cyclical learning rate (CLR) and the increased number of epochs for local model training. With the two strategies, we show that model averaging can provide competitive performance in the decentralized mode compared to the data-centralized one. In a practical environment with multiple data centers, we conduct extensive experiments using state-of-the-art deep network architectures on different types of data. Results demonstrate the effectiveness and robustness of the proposed method.

State-of-the-Art CNN Optimizer for Brain Tumor Segmentation in Magnetic Resonance Images.

Brain tumors have become a leading cause of death around the globe. The main reason for this epidemic is the difficulty conducting a timely diagnosis of the tumor. Fortunately, magnetic resonance images (MRI) are utilized to diagnose tumors in most cases. The performance of a Convolutional Neural Network (CNN) depends on many factors (i.e., weight initialization, optimization, batches and epochs, learning rate, activation function, loss function, and network topology), data quality, and specific combinations of these model attributes. When we deal with a segmentation or classification problem, utilizing a single optimizer is considered weak testing or validity unless the decision of the selection of an optimizer is backed up by a strong argument. Therefore, optimizer selection processes are considered important to validate the usage of a single optimizer in order to attain these decision problems. In this paper, we provides a comprehensive comparative analysis of popular optimizers of CNN to benchmark the segmentation for improvement. In detail, we perform a comparative analysis of 10 different state-of-the-art gradient descent-based optimizers, namely Adaptive Gradient (Adagrad), Adaptive Delta (AdaDelta), Stochastic Gradient Descent (SGD), Adaptive Momentum (Adam), Cyclic Learning Rate (CLR), Adaptive Max Pooling (Adamax), Root Mean Square Propagation (RMS Prop), Nesterov Adaptive Momentum (Nadam), and Nesterov accelerated gradient (NAG) for CNN. The experiments were performed on the BraTS2015 data set. The Adam optimizer had the best accuracy of 99.2% in enhancing the CNN ability in classification and segmentation.

Accounting for data variability in multi-institutional distributed deep learning for medical imaging.

Sharing patient data across institutions to train generalizable deep learning models is challenging due to regulatory and technical hurdles. Distributed learning, where model weights are shared instead of patient data, presents an attractive alternative. Cyclical weight transfer (CWT) has recently been demonstrated as an effective distributed learning method for medical imaging with homogeneous data across institutions. In this study, we optimize CWT to overcome performance losses from variability in training sample sizes and label distributions across institutions. Optimizations included proportional local training iterations, cyclical learning rate, locally weighted minibatch sampling, and cyclically weighted loss. We evaluated our optimizations on simulated distributed diabetic retinopathy detection and chest radiograph classification. Proportional local training iteration mitigated performance losses from sample size variability, achieving 98.6% of the accuracy attained by centrally hosting in the diabetic retinopathy dataset split with highest sample size variance across institutions. Locally weighted minibatch sampling and cyclically weighted loss both mitigated performance losses from label distribution variability, achieving 98.6% and 99.1%, respectively, of the accuracy attained by centrally hosting in the diabetic retinopathy dataset split with highest label distribution variability across institutions. Our optimizations to CWT improve its capability of handling data variability across institutions. Compared to CWT without optimizations, CWT with optimizations achieved performance significantly closer to performance from centrally hosting. Our work is the first to identify and address challenges of sample size and label distribution variability in simulated distributed deep learning for medical imaging. Future work is needed to address other sources of real-world data variability.

Automated Pavement Crack Segmentation Using U-Net-Based Convolutional Neural Network

Automated pavement crack image segmentation is challenging because of inherent irregular patterns, lighting conditions, and noise in images. Conventional approaches require a substantial amount of feature engineering to differentiate crack regions from non-affected regions. In this paper, we propose a deep learning technique based on a convolutional neural network to perform segmentation tasks on pavement crack images. Our approach requires minimal feature engineering compared to other machine learning techniques. We propose a U-Net-based network architecture in which we replace the encoder with a pretrained ResNet-34 neural network. We use a "one-cycle" training schedule based on cyclical learning rates to speed up the convergence. Our method achieves an F1 score of 96% on the CFD dataset and 73% on the Crack500 dataset, outperforming other algorithms tested on these datasets. We perform ablation studies on various techniques that helped us get marginal performance boosts, i.e., the addition of spatial and channel squeeze and excitation (SCSE) modules, training with gradually increasing image sizes, and training various neural network layers with different learning rates.

Research on Inception Module Incorporated Siamese Convolutional Neural Networks to Realize Face Recognition

Face recognition is an active research subject of biometrics due to its significant research and application prospects. The performance of face recognition can be affected by a series of uncontrollable factors, such as illumination, expression, posture and occlusion, which restricts its real-world applications. Therefore, improving the robustness of face recognition to environmental changes became an urgent problem. In this paper, a simplified deep convolutional neural network structure having high robustness under unlimited conditions is designed for face recognition. This structure can improve training speed and face recognition accuracy, and be suitable for small-scale data sets. Inception Module Incorporated Siamese Convolutional Neural Networks (IMISCNN) is developed based on effective reduction of external interference and better features extraction by adopting the Siamese network structure. A cyclical learning rate strategy is also introduced in IMISCNN for better model convergence. Compared to classical face recognition algorithms, such as PCA, PCA and SVM, CNN, PCANet, and the original SNN et al. The accuracy of IMISCNN in CASIA-webface and Extended Yale B standard face database is 99.36% and 99.21%, respectively. Its feasibility and effectiveness have been verified in our experiments.

Evolving Neural Conditional Random Fields for drilling report classification

Oil and gas prospecting is an important economic activity, besides being expensive and quite complex, thus requiring close monitoring to avoid work accidents and mainly environmental damages. An essential source of information concerns the daily drilling reports that contain operations technical interpretations and additional information from rig sensors. However, only a few works have focused on mining textual information from such reports for providing intelligent-based decision-making mechanisms to aid safety and efficiency concerns in drilling operations. This work proposes a contextual-driven approach based on Recurrent Neural Networks to recognize events in drilling reports that can outperform other related techniques. We also introduce a novel approach based on evolutionary computing to combine partially trained models using cyclical learning rates. Experiments conducted on two unbalanced datasets provided by Petrobras (Petróleo Brasileiro S.A.) show that our model improved Macro-F1 scores over the baseline by more than 47%. Besides, the proposed ensembling technique further enhanced these values by another 3% in the best scenario. Such promising results can shed light over new research directions in the field.11The source code is available at http://github.com/lzfelix/evolving-ncrf.

CLR-based deep convolutional spiking neural network with validation based stopping for time series classification

Huge amount of time series data over several domains such as engineering, biomedical and finance, demands the development of efficient methods for the problem of time series classification. The classification of univariate and multivariate time series together using a single architecture is a very difficult task. In this work, a bio-inspired convolutional spiking neural network (CSNN) is proposed for both univariate and multivariate time series. For this, first we develop a simple transformation to convert raw time series sequences into matrices. The CSNN is a three staged framework which include convolutional feature extraction, spike encoding using soft leaky integrate and fire (Soft-LIf) and classification. As spikes generated are differentiable, thus the learning algorithm for CSNN uses error-backpropagation with cyclical learning rates (CLR) and RMSprop optimizer. Additionally, validation based stopping rules are employed to overcome the overfitting which also provides a set of parameters associated with low validation set loss. Thereafter, to demonstrate the accuracy and robustness of proposed CSNN model, we have used University of California (UCR) univariate as well as University of East Anglia (UEA) multivariate datasets to perform the experiments. Moreover, we conduct comparative empirical performance evaluation with benchmark methods and also with recent deep networks proposed for time series classification. Our results reveal that proposed CSNN advances the baseline methods by achieving higher performance accuracy for both univariate and multivariate datasets. It is shown that the CLR with RMSprop optimizer is able to achieve faster convergence, however CLR and adaptive rates are considered competitive to each other. In addition, we also address the optimal model selection and study the effects of different factors on the performance of proposed CSNN.

A New Snapshot Ensemble Convolutional Neural Network for Fault Diagnosis

Deep learning (DL) has been widely applied in fault diagnosis field. The ensemble of DL can significantly improve the generalization ability of DL. Snapshot ensemble learning, which uses the cyclic learning rate scheduler (CLR) and then combines the local minima in once training to form an ensemble method, has shown powerful performance. However, the learning rate range of CLR should be pre-defined by experiences, which may limit its prediction accuracy. In this paper, a new snapshot ensemble convolutional neural network (SECNN) is proposed, which can find the proper range of learning rate for SECNN automatically when facing a new dataset. First, a max–min cosine cyclic learning rate scheduler (MMCCLR) is designed to avoid learning rate range being affected by other parameters. Then, a new learning rate testing (logLR Test) is applied to estimate the proper learning rate range for MMCCLR. Finally, the SECNN with MMCCLR is developed. The SECNN is tested on three famous datasets, including bearing dataset of Case Western Reserve University, self-priming centrifugal pump dataset, and bearing dataset provided by the Machinery Failure Prevention Technology. The results have been improved a lot by the proposed SECNN methods. The SECNN has also achieved the start-of-the-art prediction accuracies by comparing with other DL and traditional methods.

Snapshot ensembles of non-negative matrix factorization for stability of topic modeling

Recently many topic models such as Latent Dirichlet Allocation (LDA) and Non-negative Matrix Factorization (NMF) have made important progress towards generating high-level knowledge from a large corpus. However, these algorithms based on random initialization generate different results on the same corpus using the same parameters, denoted as instability problem. For solving this problem, ensembles of NMF are known to be much more stable and accurate than individual NMFs. However, training multiple NMFs for ensembling is computationally expensive. In this paper, we propose a novel scheme to obtain the seemingly contradictory goal of ensembling multiple NMFs without any additional training cost. We train a single NMF algorithm with the cyclical learning rate schedule, which can converge to several local minima along its optimization path. We save the results to the ensemble when the model converges, and then restart the optimization with a large learning rate that can help escape the current local minimum. Based on experiments performed on text corpora using a number of measures to assess, our method can reduce instability at no additional training cost, while simultaneously yields more accurate topic models than traditional single methods and ensemble methods.