Learning Rate Schedule Research Articles

A novel hand-crafted with deep learning features based fusion model for COVID-19 diagnosis and classification using chest X-ray images

COVID-19 pandemic is increasing in an exponential rate, with restricted accessibility of rapid test kits. So, the design and implementation of COVID-19 testing kits remain an open research problem. Several findings attained using radio-imaging approaches recommend that the images comprise important data related to coronaviruses. The application of recently developed artificial intelligence (AI) techniques, integrated with radiological imaging, is helpful in the precise diagnosis and classification of the disease. In this view, the current research paper presents a novel fusion model hand-crafted with deep learning features called FM-HCF-DLF model for diagnosis and classification of COVID-19. The proposed FM-HCF-DLF model comprises three major processes, namely Gaussian filtering-based preprocessing, FM for feature extraction and classification. FM model incorporates the fusion of handcrafted features with the help of local binary patterns (LBP) and deep learning (DL) features and it also utilizes convolutional neural network (CNN)-based Inception v3 technique. To further improve the performance of Inception v3 model, the learning rate scheduler using Adam optimizer is applied. At last, multilayer perceptron (MLP) is employed to carry out the classification process. The proposed FM-HCF-DLF model was experimentally validated using chest X-ray dataset. The experimental outcomes inferred that the proposed model yielded superior performance with maximum sensitivity of 93.61%, specificity of 94.56%, precision of 94.85%, accuracy of 94.08%, F score of 93.2% and kappa value of 93.5%.

Associated Learning: Decomposing End-to-End Backpropagation Based on Autoencoders and Target Propagation.

Backpropagation (BP) is the cornerstone of today's deep learning algorithms, but it is inefficient partially because of backward locking, which means updating the weights of one layer locks the weight updates in the other layers. Consequently, it is challenging to apply parallel computing or a pipeline structure to update the weights in different layers simultaneously. In this letter, we introduce a novel learning structure, associated learning (AL), that modularizes the network into smaller components, each of which has a local objective. Because the objectives are mutually independent, AL can learn the parameters in different layers independently and simultaneously, so it is feasible to apply a pipeline structure to improve the training throughput. Specifically, this pipeline structure improves the complexity of the training time from , which is the time complexity when using BP and stochastic gradient descent (SGD) for training, to , where is the number of training instances and is the number of hidden layers. Surprisingly, even though most of the parameters in AL do not directly interact with the target variable, training deep models by this method yields accuracies comparable to those from models trained using typical BP methods, in which all parameters are used to predict the target variable. Consequently, because of the scalability and the predictive power demonstrated in the experiments, AL deserves further study to determine the better hyperparameter settings, such as activation function selection, learning rate scheduling, and weight initialization, to accumulate experience, as we have done over the years with the typical BP method. In addition, perhaps our design can also inspire new network designs for deep learning. Our implementation is available at https://github.com/SamYWK/Associated_Learning.

Musculoskeletal Abnormality Detection in Medical Imaging Using GnCNNr (Group Normalized Convolutional Neural Networks with Regularization)

Musculoskeletal abnormality detection serves as an advantage to the professionals in the medical domain and also serves as an assistance in the diagnosis as well as the treatment of the abnormalities. This paper mainly focuses on accurately detecting musculoskeletal abnormalities using various deep learning models and techniques. MURA dataset has been used for experimentation. MURA dataset has 14,863 images of finger, wrist, elbow, shoulder, forearm and hand which has been analyzed using deep learning models. In this research paper, authors have proposed GnCNNr model which utilizes group normalization, weight standardization and cyclic learning rate scheduler to enhance the accuracy, precision and other model interpretation metrics. The musculoskeletal abnormality has been detected by using various deep learning models. Accuracy and Cohen Kappa have been taken as the evaluation criteria. The highest accuracy of 85% and Cohen Kappa statistic of 0.698 was achieved by the GnCNNr model in comparison with the conventional deep learning methods like DenseNet, Inception, Inception v2 model.

On the impact of selected modern deep-learning techniques to the performance and celerity of classification models in an experimental high-energy physics use case

Beginning from a basic neural-network architecture, we test the potential benefits offered by a range of advanced techniques for machine learning, in particular deep learning, in the context of a typical classification problem encountered in the domain of high-energy physics, using a well-studied dataset: the 2014 Higgs ML Kaggle dataset. The advantages are evaluated in terms of both performance metrics and the time required to train and apply the resulting models. Techniques examined include domain-specific data-augmentation, learning rate and momentum scheduling, (advanced) ensembling in both model-space and weight-space, and alternative architectures and connection methods.Following the investigation, we arrive at a model which achieves equal performance to the winning solution of the original Kaggle challenge, whilst being significantly quicker to train and apply, and being suitable for use with both GPU and CPU hardware setups. These reductions in timing and hardware requirements potentially allow the use of more powerful algorithms in HEP analyses, where models must be retrained frequently, sometimes at short notice, by small groups of researchers with limited hardware resources. Additionally, a new wrapper library for PYTORCH called LUMIN is presented, which incorporates all of the techniques studied.

Aerial Scene Classification through Fine-Tuning with Adaptive Learning Rates and Label Smoothing

Remote Sensing (RS) image classification has recently attracted great attention for its application in different tasks, including environmental monitoring, battlefield surveillance, and geospatial object detection. The best practices for these tasks often involve transfer learning from pre-trained Convolutional Neural Networks (CNNs). A common approach in the literature is employing CNNs for feature extraction, and subsequently train classifiers exploiting such features. In this paper, we propose the adoption of transfer learning by fine-tuning pre-trained CNNs for end-to-end aerial image classification. Our approach performs feature extraction from the fine-tuned neural networks and remote sensing image classification with a Support Vector Machine (SVM) model with linear and Radial Basis Function (RBF) kernels. To tune the learning rate hyperparameter, we employ a linear decay learning rate scheduler as well as cyclical learning rates. Moreover, in order to mitigate the overfitting problem of pre-trained models, we apply label smoothing regularization. For the fine-tuning and feature extraction process, we adopt the Inception-v3 and Xception inception-based CNNs, as well the residual-based networks ResNet50 and DenseNet121. We present extensive experiments on two real-world remote sensing image datasets: AID and NWPU-RESISC45. The results show that the proposed method exhibits classification accuracy of up to 98%, outperforming other state-of-the-art methods.

Abstract 2100: Deep learning can predict microsatellite instability from histology in colorectal cancer across different ethnic groups

Abstract Microsatellite instability (MSI) is known to be an important genomic marker for prediction of a patient's response to immune checkpoint inhibitors across various types of cancer. Conventional methods to measure DNA mismatch repair (MMR) status are pCR-based panel sequencing or immunohistochemical staining based methods. Recently, deep learning has been proposed as an efficient diagnostic tool in this regard, as it can learn to predict MSI directly from H&E stained slides. However, it is thought that deep learning may not generalize effectively beyond the ethnicities present in the training dataset. Therefore, we investigated the impact of ethnic differences on deep learning performance at MSI classification across a range of model architectures and hyperparameters. A total of 360 WSIs from a subset of the TCGA colorectal cancer (TCGA set) and another 170 WSIs from Gangnam Severance Hospital (Yonsei University College of Medicine, Seoul, Republic of Korea, Gangnam set) were included in this study. 360 H&E stained WSIs from TCGA set were divided into a set of square tiles of size 256x256um (0.5um/pixel) and then color normalized using Macenko normalization. Those tiles were divided into two classes - MSI-H and MSS (Microsatellite stable) based on the patient's MSI status. We trained various models on those tiles using a range of hyperparameters including model architecture, number of trainable layers, learning rate, and learning rate scheduling. The performance of this model was measured using the AUROC in the TCGA set (internal validation) and Gangnam set (external validation). Our results for ResNet18 - a popular convolutional network architecture - on TCGA and Gangnam test datasets are around 0.79 and 0.76 respectively, with a standard deviation on Gangnam of 0.03. Given good hyperparameters and model architecture, the model's performance on the Gangnam test data was within one standard deviation of its performance on the TCGA test data, despite the fact that said model was trained using only the TCGA data containing few asian patients. The lack of a significant performance difference challenges the notion that there are morphological differences in colorectal cancers between patients of Caucasian and Asian ethnicities that would prevent a deep learning system from generalizing effectively. Citation Format: Isaiah S. Pressman, Hongming Xu, Jeonghyun Kang, Yoon Jin Cha, Sung Hak Lee, Tae Hyun Hwang. Deep learning can predict microsatellite instability from histology in colorectal cancer across different ethnic groups [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 2100.

Lung Sound Classification Using Snapshot Ensemble of Convolutional Neural Networks.

We propose a robust and efficient lung sound classification system using a snapshot ensemble of convolutional neural networks (CNNs). A robust CNN architecture is used to extract high-level features from log mel spectrograms. The CNN architecture is trained on a cosine cycle learning rate schedule. Capturing the best model of each training cycle allows to obtain multiple models settled on various local optima from cycle to cycle at the cost of training a single mode. Therefore, the snapshot ensemble boosts performance of the proposed system while keeping the drawback of expensive training of ensembles moderate. To deal with the class-imbalance of the dataset, temporal stretching and vocal tract length perturbation (VTLP) for data augmentation and the focal loss objective are used. Empirically, our system outperforms state-of-the-art systems for the prediction task of four classes (normal, crackles, wheezes, and both crackles and wheezes) and two classes (normal and abnormal (i.e. crackles, wheezes, and both crackles and wheezes)) and achieves 78.4% and 83.7% ICBHI specific micro-averaged accuracy, respectively. The average accuracy is repeated on ten random splittings of 80% training and 20% testing data using the ICBHI 2017 dataset of respiratory cycles.

A Novel Learning Rate Schedule in Optimization for Neural Networks and It’s Convergence

The process of machine learning is to find parameters that minimize the cost function constructed by learning the data. This is called optimization and the parameters at that time are called the optimal parameters in neural networks. In the process of finding the optimization, there were attempts to solve the symmetric optimization or initialize the parameters symmetrically. Furthermore, in order to obtain the optimal parameters, the existing methods have used methods in which the learning rate is decreased over the iteration time or is changed according to a certain ratio. These methods are a monotonically decreasing method at a constant rate according to the iteration time. Our idea is to make the learning rate changeable unlike the monotonically decreasing method. We introduce a method to find the optimal parameters which adaptively changes the learning rate according to the value of the cost function. Therefore, when the cost function is optimized, the learning is complete and the optimal parameters are obtained. This paper proves that the method ensures convergence to the optimal parameters. This means that our method achieves a minimum of the cost function (or effective learning). Numerical experiments demonstrate that learning is good effective when using the proposed learning rate schedule in various situations.

PACL: Piecewise Arc Cotangent Decay Learning Rate for Deep Neural Network Training

Deep neural networks (DNNs) are currently the best-performing method for many classification problems. For training DNNs, the learning rate is the most important hyper-parameter, choice of which affects the performance of the model greatly. In recent years, some learning rate schedulers, such as HTD, CLR, and SGDR, have been proposed. These methods, some of which make use of the cycling mechanism to improve the convergence speed and accuracy of DNN, but performance degradation occurs in the convergence process. Others have good accuracy, but their convergence speed is too slow. This paper proposed a new learning rate schedule called piecewise arc cotangent decay learning rate (PACL), which can not only improve the convergence speed and accuracy of DNN but also significantly reduce performance degradation zone caused by the cycling mechanism. It is easy to implement, but almost at no extra computing expense. Finally, we demonstrate the effectiveness of PACL, on training CIFAR-10, CIFAR-100, and Tiny ImageNet with ResNet, DenseNet, WRN, SEResNet, and MobileNet.

Learning Rate Scheduling Policies

With the availability of high processing capability hardwares at less expensive prices, it is possible to successfully train multi-layered neural networks. Since then, several training algorithms have been developed, from algorithms which are statically initialized to algorithms which adaptively change. It is observed that to improve the training process of neural networks, the hyper-parameters are to be fine tuned. Learning Rate, Decay rate, number of epochs, number of hidden layers and number of neurons in the network are some of the hyper-parameters in concern. Of these, the Learning rate plays a crucial role in enhancing the learning capability of the network. Learning rate is the value by which the weights are adjusted in a neural network with respect to the gradient descending towards the expected optimum value. This paper discusses four types of learning rate scheduling which helps to find the best learning rates in less number of epochs. Following these scheduling methods, facilitates to find better initial learning rate value and step-wise updation during the later phase of the training process. In addition the discussed learning rate schedules are demonstrated using COIL-100, Caltech-101 and CIFAR-10 datasets trained on ResNet. The performance is evaluated using the metrics, Precision, Recall and F1-Score. The results analysis show that, depending on the nature of the dataset, the performance of the Learning Rate Scheduling policy varies. Hence the choice of the scheduling policy to train a neural network is made, based on the data.

Online Embedding Compression for Text Classification Using Low Rank Matrix Factorization

Deep learning models have become state of the art for natural language processing (NLP) tasks, however deploying these models in production system poses significant memory constraints. Existing compression methods are either lossy or introduce significant latency. We propose a compression method that leverages low rank matrix factorization during training, to compress the word embedding layer which represents the size bottleneck for most NLP models. Our models are trained, compressed and then further re-trained on the downstream task to recover accuracy while maintaining the reduced size. Empirically, we show that the proposed method can achieve 90% compression with minimal impact in accuracy for sentence classification tasks, and outperforms alternative methods like fixed-point quantization or offline word embedding compression. We also analyze the inference time and storage space for our method through FLOP calculations, showing that we can compress DNN models by a configurable ratio and regain accuracy loss without introducing additional latency compared to fixed point quantization. Finally, we introduce a novel learning rate schedule, the Cyclically Annealed Learning Rate (CALR), which we empirically demonstrate to outperform other popular adaptive learning rate algorithms on a sentence classification benchmark.

Bayesian Functional Optimisation with Shape Prior

Real world experiments are expensive, and thus it is important to reach a target in a minimum number of experiments. Experimental processes often involve control variables that change over time. Such problems can be formulated as functional optimisation problem. We develop a novel Bayesian optimisation framework for such functional optimisation of expensive black-box processes. We represent the control function using Bernstein polynomial basis and optimise in the coefficient space. We derive the theory and practice required to dynamically adjust the order of the polynomial degree, and show how prior information about shape can be integrated. We demonstrate the effectiveness of our approach for short polymer fibre design and optimising learning rate schedules for deep networks.

Similarity learning for person reidentification using projected gradient method

The problem of learning a similarity measure for cross-view person identification in a disjointed camera network is addressed. Our learning framework is based on the projected gradient approach and is suitable for large-scale applications. The hinge loss with the triplet constraints is used as the objective function. Contrary to other approaches where a dual formulation is used, we propose to optimize the objective function in the primal form, with the help of a minibatch gradient descent algorithm. The choice of learning rate and its schedule are nontrivial tasks in such optimization schemes. We have studied empirically the effect of using different strategies for learning rate schedules. Our experimental study includes conventional strategies such as fixed or diminishing learning rates and recently developed adaptive gradient methods such as root-mean-square propagation (RMSProp) for learning rate schedule. The experimental results are presented on three benchmark datasets of person reidentification, namely VIPeR, CUHK Campus, and CUHK03. Our experimental results demonstrate that the RMSProp is well suited for the proposed learning framework. A comparison with recent methods in the domain presents the justification of the proposed approach.

Breast cancer histopathological image classification using convolutional neural networks with small SE-ResNet module.

Although successful detection of malignant tumors from histopathological images largely depends on the long-term experience of radiologists, experts sometimes disagree with their decisions. Computer-aided diagnosis provides a second option for image diagnosis, which can improve the reliability of experts’ decision-making. Automatic and precision classification for breast cancer histopathological image is of great importance in clinical application for identifying malignant tumors from histopathological images. Advanced convolution neural network technology has achieved great success in natural image classification, and it has been used widely in biomedical image processing. In this paper, we design a novel convolutional neural network, which includes a convolutional layer, small SE-ResNet module, and fully connected layer. We propose a small SE-ResNet module which is an improvement on the combination of residual module and Squeeze-and-Excitation block, and achieves the similar performance with fewer parameters. In addition, we propose a new learning rate scheduler which can get excellent performance without complicatedly fine-tuning the learning rate. We use our model for the automatic classification of breast cancer histology images (BreakHis dataset) into benign and malignant and eight subtypes. The results show that our model achieves the accuracy between 98.87% and 99.34% for the binary classification and achieve the accuracy between 90.66% and 93.81% for the multi-class classification.

CLASSIFYING MIXED PATTERNS OF PROTEINSIN MICROSCOPIC IMAGES WITH DEEP NEURAL NETWORKS

Nowadays, accurate diagnosis of diseases, their treatment and prognosis is a very acute problem of modern medi-cine. By studying information about human proteins, you can identify differentially expressed proteins. These proteins are potentially interesting biomarkers that can be used for an accurate diagnosis, prognosis, or selection of individual treatments, especially for cancer. A surprising finding from this research is that we have relatively few proteins that are tissue specific. Almost halfof all pro-teins are categorized as housekeeping proteins, expressed in all cells. Only 2,300 proteins in the human body have been identified as tissue enriched, meaning they have elevated expression levels in certain tissues. Thanks to advances in high-throughput microscopy, images are generated too quickly for manual evaluation. Consequently, theneed for automating the analysis of biomedical images is as great as ever to speed up the understanding of human cells and diseases. Historically, the classification of proteins was limited to individual patterns in one or more cell types, but in order tofully understand the complexity of a human cell, models must classify mixed patterns according to a number of different human cells. The article formulates the problem of image classification in medical research. In this area, classification methods usingdeep convolutional neural networks are actively used. Presented article gives a brief overview of the various approaches and methods of similar research. As a dataset was taken “The Human Protein Atlas”, that presents a tissue-based map of the human proteome, completed in 2014 after 11 years of research. All protein expression profiling data is publicly accessible in an interactive database, enabling tissue-based exploration of the human proteome. It was done an analysis of the work and the methods that were used during the research. To solve this problem, the deep neural network model is proposed taking into account the characteristics of the domain and the sample under study. The neural network model is based on Inception-v3 architecture. Optimization procedure contains combination of several tweaks for fast convergence: stochastic gradient descent with warm restarts (learning rate schedule for exploring different local minima), progressive image resizing (training starts from small resolution and sequentially increases each cycle of SGDR). We propose new method for threshold selection for F1 meas-ure. Developed model can be used to create an instrument integrated into the medical system of intellectual microscopy to determine the location of the protein from a high-performance image.

SphereReID: Deep hypersphere manifold embedding for person re-identification

Many current successful Person Re-Identification (ReID) methods train a model with the softmax loss function to classify images of different persons and obtain the feature vectors at the same time. However, the underlying feature embedding space is ignored. In this paper, we use a modified softmax function, termed Sphere Softmax, to solve the classification problem and learn a hypersphere manifold embedding simultaneously. A balanced sampling strategy is also introduced. Finally, we propose a convolutional neural network called SphereReID adopting Sphere Softmax and training a single model end-to-end with a new warming-up learning rate schedule on four challenging datasets including Market-1501, DukeMTMC-reID, CHHK-03, and CUHK-SYSU. Experimental results demonstrate that this single model outperforms the state-of-the-art methods on all four datasets without fine-tuning or re-ranking. For example, it achieves 94.4% rank-1 accuracy on Market-1501 and 83.9% rank-1 accuracy on DukeMTMC-reID. The code and trained weights of our model will be released.

A New Snapshot Ensemble Convolutional Neural Network for Fault Diagnosis

Deep learning (DL) has been widely applied in fault diagnosis field. The ensemble of DL can significantly improve the generalization ability of DL. Snapshot ensemble learning, which uses the cyclic learning rate scheduler (CLR) and then combines the local minima in once training to form an ensemble method, has shown powerful performance. However, the learning rate range of CLR should be pre-defined by experiences, which may limit its prediction accuracy. In this paper, a new snapshot ensemble convolutional neural network (SECNN) is proposed, which can find the proper range of learning rate for SECNN automatically when facing a new dataset. First, a max–min cosine cyclic learning rate scheduler (MMCCLR) is designed to avoid learning rate range being affected by other parameters. Then, a new learning rate testing (logLR Test) is applied to estimate the proper learning rate range for MMCCLR. Finally, the SECNN with MMCCLR is developed. The SECNN is tested on three famous datasets, including bearing dataset of Case Western Reserve University, self-priming centrifugal pump dataset, and bearing dataset provided by the Machinery Failure Prevention Technology. The results have been improved a lot by the proposed SECNN methods. The SECNN has also achieved the start-of-the-art prediction accuracies by comparing with other DL and traditional methods.

Snapshot ensembles of non-negative matrix factorization for stability of topic modeling

Recently many topic models such as Latent Dirichlet Allocation (LDA) and Non-negative Matrix Factorization (NMF) have made important progress towards generating high-level knowledge from a large corpus. However, these algorithms based on random initialization generate different results on the same corpus using the same parameters, denoted as instability problem. For solving this problem, ensembles of NMF are known to be much more stable and accurate than individual NMFs. However, training multiple NMFs for ensembling is computationally expensive. In this paper, we propose a novel scheme to obtain the seemingly contradictory goal of ensembling multiple NMFs without any additional training cost. We train a single NMF algorithm with the cyclical learning rate schedule, which can converge to several local minima along its optimization path. We save the results to the ensemble when the model converges, and then restart the optimization with a large learning rate that can help escape the current local minimum. Based on experiments performed on text corpora using a number of measures to assess, our method can reduce instability at no additional training cost, while simultaneously yields more accurate topic models than traditional single methods and ensemble methods.

Improving Language Modelling with Noise Contrastive Estimation

Neural language models do not scale well when the vocabulary is large. Noise contrastive estimation (NCE) is a sampling-based method that allows for fast learning with large vocabularies. Although NCE has shown promising performance in neural machine translation, its full potential has not been demonstrated in the language modelling literature. A sufficient investigation of the hyperparameters in the NCE-based neural language models was clearly missing. In this paper, we showed that NCE can be a very successful approach in neural language modelling when the hyperparameters of a neural network are tuned appropriately. We introduced the `search-then-converge' learning rate schedule for NCE and designed a heuristic that specifies how to use this schedule. The impact of the other important hyperparameters, such as the dropout rate and the weight initialisation range, was also demonstrated. Using a popular benchmark, we showed that appropriate tuning of NCE in neural language models outperforms the state-of-the-art single-model methods based on standard dropout and the standard LSTM recurrent neural networks.

A survey of Algorithms and Analysis for Adaptive Online Learning

We present tools for the analysis of Follow-The-Regularized-Leader (FTRL), Dual Averaging, and Mirror Descent algorithms when the regularizer (equivalently, proxfunction or learning rate schedule) ...