Learning Rate Schedule Research Articles

Weight Update Skipping: Reducing Training Time for Artificial Neural Networks

Artificial Neural Networks (ANNs) are known as state-of-the-art techniques in Machine Learning (ML) and have achieved outstanding results in data-intensive applications, such as recognition, classification, and segmentation. These networks mostly use deep layers of convolution and/or fully connected layers with many filters in each layer, demanding a large amount of data and tunable hyperparameters to achieve competitive accuracy. As a result, storage, communication, and computational costs of training (in particular time spent for training) become limiting factors to scale them up. In this paper, we propose a new training methodology for ANNs that exploits the observation of improvement of accuracy shows temporal variations which allow us to skip updating weights when the variation is minuscule. During such time windows, we keep updating bias which ensures the network still trains and avoids overfitting; however, we selectively skip updating weights (and their time-consuming computations). This training approach virtually achieves the same accuracy with considerably less computational cost and reduces the time spent on training. We developed two variations of the proposed training method for selectively updating weights, and call them as i) Weight Update Skipping (WUS), and ii) Weight Update Skipping with Learning Rate Scheduler (WUS+LR). We evaluate these two approaches by analyzing state-of-the-art models, including AlexNet, VGG-11, VGG-16, ResNet-18 on CIFAR datasets. We also use ImageNet dataset for AlexNet, VGG-16, and Resnet-18. On average, WUS and WUS+LR reduced the training time (compared to the baseline) by 54%, and 50% on CPU and 22%, and 21% on GPU, respectively for CIFAR-10; and 43% and 35% on CPU and 22%, and 21% on GPU, respectively for CIFAR-100; and finally 30% and 27% for ImageNet, respectively.

AccuLearning: A User-Friendly Deep Learning Auto-Segmentation Platform for Radiotherapy

Despite the significant success of deep learning (DL) techniques in auto-segmentation of medical images, the application is often limited by various technical difficulties which are costly to implement for medical researchers without any programming experience. To address these issues, we developed a user-friendly deep learning platform, i.e., AccuLearning, to provide accessible DL resources for clinical application to researchers in medical field.The goal of AccuLearning is to simplify the DL model building workflow, hide the complexities of model training, and provide highly integrated, efficient and flexible platform for the implementation of deep learning auto-segmentation technique. To do this, all the major components were integrated into an automatic pipeline. Gradient-based adaptive network structure building and targeted loss function make the training process of the model easier and more stable. To make the platform generalizable, adaptive image pre-processing strategies were employed, such as automatic dataset characteristics extraction and standardization. With an easy-to-use graphical user interface and optimized default parameters, minimal user intervention is needed. To demonstrate the effectiveness, we applied AccuLearning on a total of 9 auto-segmentation tasks using open-access medical image datasets, covering 64 ROIs (including 4 GTVs and 60 OARs) on various anatomy sites (abdomen, head and neck, and pelvis) on CTs or MRIs. The model performance was evaluated using Dice and HD95 compared with ground truth. In addition, the impact of number of training cases were also evaluated.AccuLearning can achieve competitive model accuracy using default settings, with less than 3 hours of training time and 45 seconds of average inference time per patient. The average Dice and HD were 0.81 ± 0.11 and 7.7 ± 7.4mm for all the OARs. 18.3% (11/60) of the OARs have Dice larger than 0.9, while 66.7% (40/60) larger than 0.8. For HD95, 26.7% (16/60) of the OARs have HD95 smaller than 3mm and 73.3% (44/60) smaller than 10mm. The DL model trained with only 20 cases showed no significant difference with the model trained with 160 cases in OARs segmentation. For nasopharyngeal carcinoma GTV auto-segmentation, the average Dice was 0.62. Through adjustments to model dimensions, loss function and learning rate schedule, the average Dice for GTV was increased to 0.69, which is reasonably comparable to the inter-observer variations Dice 0.71.AccuLearning can build robust and accurate DL models for different auto-segmentation tasks on both CT and MRI with minimal user interventions, short model training time and low demand of training sample size. The platform can be implemented into current workflow, enabling fast and accurate target and OARs segmentation for clinical practice in radiotherapy field.

A deep learning model for classifying human facial expressions from infrared thermal images

The analysis of human facial expressions from the thermal images captured by the Infrared Thermal Imaging (IRTI) cameras has recently gained importance compared to images captured by the standard cameras using light having a wavelength in the visible spectrum. It is because infrared cameras work well in low-light conditions and also infrared spectrum captures thermal distribution that is very useful for building systems like Robot interaction systems, quantifying the cognitive responses from facial expressions, disease control, etc. In this paper, a deep learning model called IRFacExNet (InfraRed Facial Expression Network) has been proposed for facial expression recognition (FER) from infrared images. It utilizes two building blocks namely Residual unit and Transformation unit which extract dominant features from the input images specific to the expressions. The extracted features help to detect the emotion of the subjects in consideration accurately. The Snapshot ensemble technique is adopted with a Cosine annealing learning rate scheduler to improve the overall performance. The performance of the proposed model has been evaluated on a publicly available dataset, namely IRDatabase developed by RWTH Aachen University. The facial expressions present in the dataset are Fear, Anger, Contempt, Disgust, Happy, Neutral, Sad, and Surprise. The proposed model produces 88.43% recognition accuracy, better than some state-of-the-art methods considered here for comparison. Our model provides a robust framework for the detection of accurate expression in the absence of visible light.

A Deep Learning Based Object Identification System for Forest Fire Detection

Forest fires are still a large concern in several countries due to the social, environmental and economic damages caused. This paper aims to show the design and validation of a proposed system for the classification of smoke columns with object detection and a deep learning-based approach. This approach is able to detect smoke columns visible below or above the horizon. During the dataset labelling, the smoke object was divided into three different classes, depending on its distance to the horizon, a cloud object was also added, along with images without annotations. A comparison between the use of RetinaNet and Faster R-CNN was also performed. Using an independent test set, an F1-score around 80%, a G-mean around 80% and a detection rate around 90% were achieved by the two best models: both were trained with the dataset labelled with three different smoke classes and with augmentation; Faster R-CNNN was the model architecture, re-trained during the same iterations but following different learning rate schedules. Finally, these models were tested in 24 smoke sequences of the public HPWREN dataset, with 6.3 min as the average time elapsed from the start of the fire compared to the first detection of a smoke column.

An Adaptive Learning Rate Schedule for SIGNSGD Optimizer in Neural Networks

An Adaptive Learning Rate Schedule for SIGNSGD Optimizer in Neural Networks

Privacy-enhanced momentum federated learning via differential privacy and chaotic system in industrial Cyber–Physical systems

By leveraging Industrial Cyber–Physical Systems (ICPSs), deep learning-based methods are applied to address various industrial issues. Due to privacy policy reasons, conventional centralized learning (CL) may be improper for some industrial scenarios with sensitive data, such as smart medicine. Recently, federated learning (FL) as a novel collaboration learning approach has received extensive attention, which can break data barriers between different institutions to improve the model performance. However, the privacy information of the industrial agents may be inferred from their shared parameters. In this paper, we propose a Privacy-Enhanced Momentum Federated Learning framework, named PEMFL, that amalgamates differential privacy (DP), Momentum FL (MFL) and chaos-based encryption method. During the training, differentially privacy is used to disturb the industrial agents’ gradient parameters in order to preserve their privacy information. Meanwhile, each industrial agent uses the chaos system-based encryption method to encrypt the weight parameters of their local models, which has two advantages: (1) the encryption method can enhance privacy protection; (2) the cloud server cannot access the truth value of the global model parameters which is a vital asset to the industrial agents. In addition, Momentum Gradient Descent (MGD) and an adjusting learning rate schedule are adopted to improve training efficiency for the PEMFL. The performance of the PEMFL is evaluated based on two non-i.i.d datasets. Theoretical analysis and experimental results demonstrate the excellent performance of the PEMFL in terms of accuracy and privacy security.

Покращення моделей розпізнавання облич за допомогою навчання подібності, розкладу зміни темпу навчання та аугментацій

Розпізнавання облич — одна з основних задач комп’ютерного зору, актуальна в силу її практичної значущості та викликає велику зацікавленість широкого кола науковців. І хоча дослідження у сфері відбувались з початку розвитку комп’ютерного зору, адекватних результатів змогли досягнути лише за допомогою згорткових нейронних мереж. У даній роботі проведено порівняльний аналіз методів розпізнавання обличчя до згорткових нейронних мереж. Розглянуто метод навчання подібності, аугментації та розклади зміни темпу навчання. Проведено ряд експериментів, виконано порівняльний аналіз розглянутих методів покращення згорткових нейронних мереж, у результаті отримано універсальний алгоритм для навчання моделі розпізнавання облич. У роботі використано SE-ResNet50 як єдину мережу для експериментів. Навчання подібності — це метод, за допомогою якого можливо досягнути достатньої точності. Перенавчання є серйозною проблемою для нейронних мереж, зокрема тому, що мають дуже багато параметрів, а даних зазвичай не настільки багато, щоб гарантувати узагальнюючу можливість моделі. Додаткова розмітка даних може займати багато часу і бути доволі дорогою, тому виник такий підхід, як аугментації. Аугментації штучно збільшують тренувальний набір даних, тому цілком природньо, що метод аугментації у всіх експериментах покращив результати відносно початкового експерименту. Різні степені та більш агресивні форми аугментації у задачі розпізнавання облич у даній роботі приводив до кращих результатів. Як і очікувалось, найкращим розкладом зміни темпу навчання виявився косинусний з розігрівом та оновленнями. Цей розклад має мало параметрів, до того ж зручний у використанні. Загалом, використовуючи різні підходи, отримали точність 93,5 % на досить складному наборі даних, що на 22 % краще за базовий експеримент. У наступних дослідженнях планується розглянути покращення не лише моделі розпізнавання облич, а й детекції. Від якості детекції обличчя безпосередньо залежить точність розпізнавання.

Achieving generalization of deep learning models in a quick way by adapting T-HTR learning rate scheduler

Achieving generalization of deep learning models in a quick way by adapting T-HTR learning rate scheduler

Satellite and Scene Image Classification Based on Transfer Learning and Fine Tuning of ResNet50

Image classification has gained lot of attention due to its application in different computer vision tasks such as remote sensing, scene analysis, surveillance, object detection, and image retrieval. The primary goal of image classification is to assign the class labels to images according to the image contents. The applications of image classification and image analysis in remote sensing are important as they are used in various applied domains such as military and civil fields. Earlier approaches for remote sensing images and scene analysis are based on low-level feature representations such as color- and texture-based features. Vector of Locally Aggregated Descriptors (VLAD) and orderless Bag-of-Features (BoF) representations are the examples of mid-level approaches for remote sensing image classification. Recent trends for remote sensing and scene classification are focused on the use of Convolutional Neural Network (CNN). Keeping in view the success of CNN models, in this research, we aim to fine-tune ResNet50 by using network surgery and creation of network head along with the fine-tuning of hyperparameters. The learning of hyperparameters is tuned by using a linear decay learning rate scheduler known as piecewise scheduler. To tune the optimizer hyperparameter, Stochastic Gradient Descent with Momentum (SGDM) is used with the usage of weight learn and bias learn rate factor. Experiments and analysis are conducted on five different datasets, that is, UC Merced Land Use Dataset (UCM), RSSCN (the remote sensing scene classification image dataset), SIRI-WHU, Corel-1K, and Corel-1.5K. The analysis and competitive results exemplify that our proposed image classification-based model can classify the images in a more effective and efficient manner as compared to the state-of-the-art research.

Deep learning of resting-state electroencephalogram signals for three-class classification of Alzheimer’s disease, mild cognitive impairment and healthy ageing

Objective. This study aimed to produce a novel deep learning (DL) model for the classification of subjects with Alzheimer’s disease (AD), mild cognitive impairment (MCI) subjects and healthy ageing (HA) subjects using resting-state scalp electroencephalogram (EEG) signals. Approach. The raw EEG data were pre-processed to remove unwanted artefacts and sources of noise. The data were then processed with the continuous wavelet transform, using the Morse mother wavelet, to create time-frequency graphs with a wavelet coefficient scale range of 0–600. The graphs were combined into tiled topographical maps governed by the 10–20 system orientation for scalp electrodes. The application of this processing pipeline was used on a data set of resting-state EEG samples from age-matched groups of 52 AD subjects (82.3 ± 4.7 years of age), 37 MCI subjects (78.4 ± 5.1 years of age) and 52 HA subjects (79.6 ± 6.0 years of age). This resulted in the formation of a data set of 16197 topographical images. This image data set was then split into training, validation and test images and used as input to an AlexNet DL model. This model was comprised of five hidden convolutional layers and optimised for various parameters such as learning rate, learning rate schedule, optimiser, and batch size. Main results. The performance was assessed by a tenfold cross-validation strategy, which produced an average accuracy result of 98.9 ± 0.4% for the three-class classification of AD vs MCI vs HA. The results showed minimal overfitting and bias between classes, further indicating the strength of the model produced. Significance. These results provide significant improvement for this classification task compared to previous studies in this field and suggest that DL could contribute to the diagnosis of AD from EEG recordings.

ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques

Pre-trained language models of the BERT family have defined the state-of-the-arts in a wide range of NLP tasks. However, the performance of BERT-based models is mainly driven by the enormous amount of parameters, which hinders their application to resource-limited scenarios. Faced with this problem, recent studies have been attempting to compress BERT into a small-scale model. However, most previous work primarily focuses on a single kind of compression technique, and few attention has been paid to the combination of different methods. When BERT is compressed with integrated techniques, a critical question is how to design the entire compression framework to obtain the optimal performance. In response to this question, we integrate three kinds of compression methods (weight pruning, low-rank factorization and knowledge distillation (KD)) and explore a range of designs concerning model architecture, KD strategy, pruning frequency and learning rate schedule. We find that a careful choice of the designs is crucial to the performance of the compressed model. Based on the empirical findings, our best compressed model, dubbed Refined BERT cOmpreSsion with InTegrAted techniques (ROSITA), is 7.5x smaller than BERT while maintains 98.5% of the performance on five tasks of the GLUE benchmark, outperforming the previous BERT compression methods with similar parameter budget.

Nondestructive multiplex detection of foodborne pathogens with background microflora and symbiosis using a paper chromogenic array and advanced neural network

We have developed an inexpensive, standardized paper chromogenic array (PCA) integrated with a machine learning approach to accurately identify single pathogens (Listeria monocytogenes, Salmonella Enteritidis, or Escherichia coli O157:H7) or multiple pathogens (either in multiple monocultures, or in a single cocktail culture), in the presence of background microflora on food. Cantaloupe, a commodity with significant volatile organic compound (VOC) emission and large diverse populations of background microflora, was used as the model food. The PCA was fabricated from a paper microarray via photolithography and paper microfluidics, into which 22 chromogenic dye spots were infused and to which three red/green/blue color-standard dots were taped. When exposed to VOCs emitted by pathogens of interest, dye spots exhibited distinguishable color changes and pattern shifts, which were automatically segmented and digitized into a ΔR/ΔG/ΔB database. We developed an advanced deep feedforward neural network with a learning rate scheduler, L2 regularization, and shortcut connections. After training on the ΔR/ΔG/ΔB database, the network demonstrated excellent performance in identifying pathogens in single monocultures, multiple monocultures, and in cocktail culture, and in distinguishing them from the background signal on cantaloupe, providing accuracy of up to 93% and 91% under ambient and refrigerated conditions, respectively. With its combination of speed, reliability, portability, and low cost, this nondestructive approach holds great potential to significantly advance culture-free pathogen detection and identification on food, and is readily extendable to other food commodities with complex microflora.

State-of-Charge Estimation of Li-ion Battery Using Gated Recurrent Unit With One-Cycle Learning Rate Policy

Deep learning has gained much traction in application to state-of-charge (SOC) estimation for Li-ion batteries in electric vehicle applications. However, with the vast selection of architectures and hyperparameter combinations, it remains challenging to design an accurate and robust SOC estimation model with a sufficiently low computation cost. Therefore, this study provides a comparative evaluation among commonly used deep learning models from the recurrent, convolutional, and feedforward architecture benchmarked on an openly available Li-ion battery dataset. To evaluate model robustness and generalization capability, we train and test models on different drive cycles at various temperatures and compute the root mean squared error (RMSE) and mean absolute error metric. To evaluate model computation costs, we run models in real-time and record the model size, floating-point operations per second (FLOPS), and run-time duration per datapoint. This study proposes a two-hidden layer stacked gated recurrent unit model trained with a one-cycle policy learning rate scheduler. The proposed model achieves a minimum RMSE of 0.52% on the train dataset and 0.65% on the test dataset while maintaining a relatively low computation cost. Executing the proposed model in real-time takes up approximately 1 MB in disk space, 300K FLOPS, and 0.03 ms run-time per datapoint. This makes the proposed model feasible to be executed on lightweight battery management system processors.

Learning Rate Optimization in CNN for Accurate Ophthalmic Classification

One of the most important hyper-parameters for model training and generalization is the learning rate. Recently, many research studies have shown that optimizing the learning rate schedule is very useful for training deep neural networks to get accurate and efficient results. In this paper, different learning rate schedules using some comprehensive optimization techniques have been compared in order to measure the accuracy of a convolutional neural network CNN model to classify four ophthalmic conditions. In this work, a deep learning CNN based on Keras and TensorFlow has been deployed using Python on a database that contains 1692 images, which consists of four types of ophthalmic cases: Glaucoma, Myopia, Diabetic retinopathy, and Normal eyes. The CNN model has been trained on Google Colab. GPU with different learning rate schedules and adaptive learning algorithms. Constant learning rate, time-based decay, step-based decay, exponential decay, and adaptive learning rate optimization techniques for deep learning have been addressed. Adam adaptive learning rate method. has outperformed the other optimization techniques and achieved the best model accuracy of 92.58% for training set and 80.49% for validation datasets, respectively.

CrackWeb : A modified U-Net based segmentation architecture for crack detection

Classical image processing methods demands heavy feature engineering, as well as they are not that precise, when it comes to manual extraction of relevant features in real life scenarios amid to various lighting conditions and other factors.Thus, detection of cracks using methods based on classical image processing techniques fails to provide satisfactory results always. Hence, we have proposed a deep convolutional neural network, that is not based on manual extraction of features as mentioned above. We proposed a modified U-Net architecture, and replaced all of its convolutional layers with residual blocks, inspired from the ResNet architecture. For evaluation of our model Dice Loss is used as our objective function and F1 score as a metric. Other than that, for better convergence and optimization, a learning rate scheduler and AMSGRAD optimizer was utilized.

An Adaptive Optimization Method Based on Learning Rate Schedule for Neural Networks

Artificial intelligence (AI) is achieved by optimizing the cost function constructed from learning data. Changing the parameters in the cost function is an AI learning process (or AI learning for convenience). If AI learning is well performed, then the value of the cost function is the global minimum. In order to obtain the well-learned AI learning, the parameter should be no change in the value of the cost function at the global minimum. One useful optimization method is the momentum method; however, the momentum method has difficulty stopping the parameter when the value of the cost function satisfies the global minimum (non-stop problem). The proposed method is based on the momentum method. In order to solve the non-stop problem of the momentum method, we use the value of the cost function to our method. Therefore, as the learning method processes, the mechanism in our method reduces the amount of change in the parameter by the effect of the value of the cost function. We verified the method through proof of convergence and numerical experiments with existing methods to ensure that the learning works well.

Convolutional Neural Network With Automatic Learning Rate Scheduler for Fault Classification

Fault classification is vital in smart manufacturing, and convolutional neural network (CNN) has been widely applied in fault classification. But the performance of CNN heavily depends on its learning rate. As the default setting on learning rate cannot guarantee its performance, the learning rate tuning process becomes essential. However, the traditional learning rate tuning methods either cost much time consumption or rely on the experts’ experiences, so it is a considerable barrier for the users. To overcome this drawback, this article proposes a CNN with automatic learning rate scheduler (AutoLR-CNN) for fault classification. First, the long short-term memory (LSTM) is used to extract the features of the past loss of CNN. Then, an agent based on deep deterministic policy gradient (DDPG) is trained to automatically control the learning rate for CNN online. Third, the double CNN structure is developed to enhance the stability of the proposed method. The proposed AutoLR-CNN is tested on two famous bearing data sets and a practical bearing data set on wind turbine. The results of AutoLR-CNN are superior to six commonly used baseline learning rate schedulers in Tensorflow. AutoLR-CNN is also compared with other reported machine learning and deep learning methods. The results show that AutoLR-CNN has achieved the state-of-the-art performance in fault classification.

Performance Investigation of Learning Rate Decay in LMS-Based Equalization

In this letter, we investigate the performances of different learning rate decay schedules in least mean square (LMS) based equalization. The required iteration numbers for convergence are 275, 150 and 150 for time-based, staircase and exponential decay schedules respectively, which are much less than 1000, the required iteration number for constant learning rate schedule in processing our experimental data. Meanwhile, the corresponding training times are reduced to shorter than one third, one fifth and one fourth of that in constant schedule. The system performances in the aspects of carrier to signal power ratio and optical signal to noise ratio are measured in a 110 Gbit/s probabilistically shaped 64 QAM direct detection system to evaluate different learning rate decay schedules.

A New Reinforcement Learning Based Learning Rate Scheduler for Convolutional Neural Network in Fault Classification

Convolutional neural network (CNN) has gained increasing attention in fault classification. However, the performance of CNN is sensitive to its learning rate. Some previous works have been done to tune the learning rate, including the “trial and error” and manual search, which heavily depend on the experts' experiences and should be conducted repeatedly on every dataset. Because of the variety of the fault data, it is time-consuming and labor intensive to use these traditional tuning methods for fault classification. To overcome this problem, in this article, we develop a novel learning rate scheduler based on the reinforcement learning (RL) for convolutional neural network (RL-CNN) in fault classification, which can schedule the learning rate efficiently and automatically. First, a new RL agent is designed to learn the policies about the learning rate adjustment during the training process. Second, a new structure of RL-CNN is developed to balance the exploration and exploitation of the agent. Third, the bagging ensemble version of RL-CNN (RL-CNN-Ens) is presented. Three bearing datasets are used to test the performance of RL-CNN-Ens. The results show that RL-CNN-Ens outperforms the traditional DLs and machine learning methods. Meanwhile, RL-CNN-Ens can find the state-of-the-art learning rate schedulers as human designed, showing its potential in fault classification.

Toward Optimal Learning Rate Schedule in Scene Classification Network

Stochastic gradient descent (SGD) and adaptive methods, including ADAM, RMSProp, AdaDelta, and AdaGrad, are two dominant optimization algorithms for training convolution neural network (CNN) in scene classification tasks. Recent work reveals that these adaptive methods lead to degraded performance in image classification tasks compared with SGD. In this letter, a learning rate schedule named switching from constant to step decay (SCTSD) is proposed to further improve the classification accuracy of SGD. SCTSD begins with a constant learning rate and switches to step decay learning rates when appropriate. Theoretical evidence is provided on the superiority of SCTSD compared with other manually tuned schedules. Comparison experiments have been conducted among adaptive methods, SCTSD, and other manual schedules with three CNN architectures. The experiment results on various scene classification data sets show that SCTSD has the highest accuracy on the test set and it is state of the art. In the end, some suggestions on hyperparameters selection of SCTSD are given for scene classification.