Mini-batch Size Research Articles

A comparative study of hot tensile deformation behavior of 6016 aluminum alloy under LSTM neural network and Arrhenius model

Abstract The isothermal tensile test of 6016-T6 aluminum alloy was carried out on Gleeble-3500 at the temperature range of 400 °C–550 °C and the strain rate range of 0.01–10 s−1. The results show that the thermal deformation mechanism of 6016-T6 is dynamic recovery and dynamic recrystallization. In this paper, the phenomenological Arrhenius constitutive model and the data-driven WOA-LSTM constitutive model for predicting the hot tensile deformation behavior of 6016-T6 aluminum alloy were studied in contrast. The whale optimization algorithm was used to optimize the hyperparameters of LSTM neural network to improve the prediction accuracy of flow stress. The optimization results show that the optimal hidden layer node, maximum training period, initial learning rate and mini batch size of WOA-LSTM are 46, 260, 0.0248 and 16, respectively. In addition, the influence of the number of hidden layers on the results of the network was discussed. The appropriate hidden layer of the network was determined to be 2. The result show that the prediction accuracy of WOA-LSTM constitutive model is better than the Arrhenius constitutive model. The mean absolute error and correlation coefficient are 0.9348% and 0.99952, respectively. Among them, in this study, the Arrhenius constitutive model has low precision and only has high precision within a single temperature range.

Parallel and scalable AI in HPC systems for CFD applications and beyond

This manuscript presents the library AI4HPC with its architecture and components. The library enables large-scale trainings of AI models on High-Performance Computing systems. It addresses challenges in handling non-uniform datasets through data manipulation routines, model complexity through specialized ML architectures, scalability through extensive code optimizations that augment performance, HyperParameter Optimization (HPO), and performance monitoring. The scalability of the library is demonstrated by strong scaling experiments on up to 3,664 Graphical Processing Units (GPUs) resulting in a scaling efficiency of 96%, using the performance on 1 node as baseline. Furthermore, code optimizations and communication/computation bottlenecks are discussed for training a neural network on an actuated Turbulent Boundary Layer (TBL) simulation dataset (8.3 TB) on the HPC system JURECA at the Jülich Supercomputing Centre. The distributed training approach significantly influences the accuracy, which can be drastically compromised by varying mini-batch sizes. Therefore, AI4HPC implements learning rate scaling and adaptive summation algorithms, which are tested and evaluated in this work. For the TBL use case, results scaled up to 64 workers are shown. A further increase in the number of workers causes an additional overhead due to too small dataset samples per worker. Finally, the library is applied for the reconstruction of TBL flows with a convolutional autoencoder-based architecture and a diffusion model. In case of the autoencoder, a modal decomposition shows that the network provides accurate reconstructions of the underlying field and achieves a mean drag prediction error of ≈5%. With the diffusion model, a reconstruction error of ≈4% is achieved when super-resolution is applied to 5-fold coarsened velocity fields. The AI4HPC library is agnostic to the underlying network and can be adapted across various scientific and technical disciplines.

Learning Joint Topic Representation for Detecting Drift in Social Media Text

Social media texts like tweets and blogs are collaboratively created by human interaction. Rapidly changing trends are leading to topic drift in the social media text. This drift is usually associated with words and hashtags. However, geotags play an essential part in determining topic distribution with location context. The rate of change in the distribution of words, hashtags and geotags cannot be considered uniform and must be handled accordingly. This paper builds a topic model that associates the topic with a mixture of distributions of words, hashtags and geotags. Stochastic gradient Langevin dynamic model with varying mini-batch sizes is used to capture the changes due to the asynchronous distribution of words and tags. Topic representations with co-occurrence and location contexts are specified as hashtag context vector and geotag context vector respectively. These two vectors are jointly learned to yield topical word embedding vectors over time conditioned on hashtags and geotags that can predict location-based topical variations effectively. When evaluated with Chennai and UK geolocated Twitter data, the proposed joint topical word embedding model enhanced by the social tags context, outperforms other methods.

Deep SqueezeNet learning model for diagnosis and prediction of maize leaf diseases

The maize leaf diseases create severe yield reductions and critical problems. The maize leaf disease should be discovered early, perfectly identified, and precisely diagnosed to make greater yield. This work studies three main leaf diseases: common rust, blight, and grey leaf spot. This approach involves pre-processing, including sampling and labelling, while ensuring class balance and preventing overfitting via the SMOTE algorithm. The maize leaf dataset with augmentation was used to classify these diseases using several deep-learning pre-trained networks, including VGG16, Resnet34, Resnet50, and SqueezeNet. The model was evaluated using a maize leaf dataset that included various leaf classes, mini-batch sizes, and input sizes. Performance measures, recall, precision, accuracy, F1-score, and confusion matrix were computed for each network. The SqueezeNet learning model produces an accuracy of 97% in classifying four different classes of plant leaf datasets. Comparatively, the SqueezeNet learning model has improved accuracy by 2–5% and reduced the mean square error by 4–11% over VGG16, Resnet34, and Resnet50 deep learning models.

SRPM-ST: Sequential retraining and pseudo-labeling in mini-batches for self-training

An impediment to training accurate classifiers in supervised learning is the scarcity of labeled data. In that respect, semi-supervised learning could help by using both labeled and unlabeled data. A specific form of semi-supervised learning is self-training (ST). In its basic form, ST trains an initial classifier using the labeled data to generate pseudo-labels for the unlabeled set. At this point, either the whole set of pseudo-labeled data or a subset of them with some high confidence scores about the generated pseudo-labels is selected. The selected pseudo-labeled data are then used to update the initial classifier. Although this process can be repeated to generate new pseudo-labels for the unlabeled data, it is typically a tacit assumption up to this point that the classifier is updated once all pseudo-labels are generated—a process to which we refer as the full-batch ST (F-ST) regardless of any confidence score-based subset selection. Here, we show that sequential retraining and pseudo-labeling in mini-batches (SRPM) could potentially improve the performance of the classifier with respect to F-ST. Our empirical results show the existence of a data-dependent mini-batch size for SRPM that is optimal in terms of possessing the least error rate. In practice, this parameter could be treated as a hyperparameter to tune.

DYNAMITE: Dynamic Interplay of Mini-Batch Size and Aggregation Frequency for Federated Learning With Static and Streaming Datasets

DYNAMITE: Dynamic Interplay of Mini-Batch Size and Aggregation Frequency for Federated Learning With Static and Streaming Datasets

Tuning VGG19 hyperparameters for improved pneumonia classification

This research focuses on the classification of chest X-ray (CXR) images using powerful VGG19 convolutional neural network (CNN)architecture. The classification task involves distinguishing between various chest conditions present in X-ray images, with the aim of assisting medical professionals in achieving accurate and efficient diagnoses. This research work explores the use of the VGG19 model for classifying CXR images using three optimization algorithms: Stochastic gradient descent with momentum (SGDM), root mean square propagation (RMSprop), and adaptive moment estimation (Adam). This study investigates the impact of various factors on hyperparameter adjustments, including a learning rate (LR), mini-batch size (MBS) and training epochs. Additionally, two dropout layers are introduced for weight decay with an L2 factor, and data augmentation techniques are applied with various activation functions. This study not only helps optimize for medical image analysis but also offers valuable insights into the comparative efficacy of popular optimization algorithms in deep learning (DL) applications

A stochastic gradient method with variance control and variable learning rate for Deep Learning

In this paper we study a stochastic gradient algorithm which rules the increase of the mini-batch size in a predefined fashion and automatically adjusts the learning rate by means of a monotone or non-monotone line search procedure. The mini-batch size is incremented at a suitable a priori rate throughout the iterative process in order that the variance of the stochastic gradients is progressively reduced. The a priori rate is not subject to restrictive assumptions, allowing for the possibility of a slow increase in the mini-batch size. On the other hand, the learning rate can vary non-monotonically throughout the iterations, as long as it is appropriately bounded. Convergence results for the proposed method are provided for both convex and non-convex objective functions. Moreover it can be proved that the algorithm enjoys a global linear rate of convergence on strongly convex functions. The low per-iteration cost, the limited memory requirements and the robustness against the hyperparameters setting make the suggested approach well-suited for implementation within the deep learning framework, also for GPGPU-equipped architectures. Numerical results on training deep neural networks for multi-class image classification show a promising behaviour of the proposed scheme with respect to similar state of the art competitors.

Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problems

Is Stochastic Gradient Descent (SGD) substantially different from Metropolis Monte Carlo dynamics? This is a fundamental question at the time of understanding the most used training algorithm in the field of Machine Learning, but it received no answer until now. Here we show that in discrete optimization and inference problems, the dynamics of an SGD-like algorithm resemble very closely that of Metropolis Monte Carlo with a properly chosen temperature, which depends on the mini-batch size. This quantitative matching holds both at equilibrium and in the out-of-equilibrium regime, despite the two algorithms having fundamental differences (e.g. SGD does not satisfy detailed balance). Such equivalence allows us to use results about performances and limits of Monte Carlo algorithms to optimize the mini-batch size in the SGD-like algorithm and make it efficient at recovering the signal in hard inference problems.

Binding Activity Classification of Anti-SARS-CoV-2 Molecules using Deep Learning Across Multiple Assays

The coronavirus disease-2019 (COVID-19) pandemic, caused by severe acute respiratory syndrome-coronavirus-2 (SARS-CoV-2), has urgently necessitated effective therapeutic solutions, with a focus on rapidly identifying and classifying potential small-molecule drugs. Given traditional methods’ labor-intensive and time-consuming nature, deep learning has emerged as an essential tool for efficiently processing and extracting insights from complex biological data. To utilize deep learning techniques, particularly deep neural networks (DNN) enhanced with the synthetic minority oversampling technique (SMOTE), to enhance the classification of binding activities in anti-SARS-CoV-2 molecules across various bioassays. We used 11 bioassay datasets covering various SARS-CoV-2 interactions and inhibitory mechanisms. These assays ranged from spike-ACE2 protein-protein interaction to ACE2 enzymatic activity and 3CL enzymatic activity. To address the prevalent class imbalance in these datasets, the SMOTE technique was employed to generate new samples for the minority class. In our model-building approach, we divided the dataset into 80% training and 20% test sets, reserving 10% of the training set for validation. Our approach involved employing a DNN that integrates ReLU and sigmoid activation functions, incorporates batch normalization, and uses Adam optimization. The hyperparameters and architecture of the DNN were optimized through various tests on layers, minibatch sizes, epoch sizes, and learning rates. A 40% dropout rate was incorporated to mitigate overfitting. For model evaluation, we computed performance metrics, such as balanced accuracy (BACC), precision, recall, F1 score, Matthews’ correlation coefficient (MCC), and area under the curve (AUC). The performance of the DNN across 11 bioassay test sets revealed varying outcomes, significantly influenced by the ratios of active-to-inactive compounds. Assays, such as AlphaLISA and CoV-PPE, demonstrated robust performance across various metrics, including BACC, precision, recall, and AUC, when configured with more balanced ratios (1:3 and 1:1, respectively). This suggests the effective identification of active compounds in both cases. In contrast, assays with higher imbalance ratios, such as 3CL (1:38) and cytopathic effect (1:15), demonstrated higher recall but lower precision, highlighting challenges in accurately identifying active compounds among numerous inactive compounds. However, even in these challenging settings, the model achieved favorable BACC and recall scores. Overall, the DNN model generally performed well, as indicated by the BACC, MCC, and AUC values, especially when considering the degree of dataset imbalance in each assay. This study demonstrates the significant impact of deep learning, particularly DNN models enhanced with SMOTE, in improving the identification of active compounds in bioassay datasets for COVID-19 drug discovery, outperforming traditional machine learning models. Furthermore, this study highlights the efficacy of advanced computational techniques in addressing high-throughput screening data imbalances.

Enhancing Skin Disease Classification: A Novel Approach With Tailored Loss Functions And SMOTE Sumeet Ghumare

Although skin disease like skin cancer is a common health problem worldwide, early identification is crucial to its recovery. By using medical photos, the rise of artificial intelligence (AI) has transformed the detection of skin diseases. Despite the reality that various deep learning models have been created with this goal in mind, class imbalance in multi class classification all poses a serious difficulty and imbalance among the classes. This work offers a novel strategy to reduce class imbalance in the categorization of skin diseases, leading to significant accuracy gains. One of the largest skin cancer datasets currently accessible, it consists of 33,106 dermoscopy photos from seven different kinds of skin diseases as a starting point. The main goal is to enhance skin disease classification algorithms’ performance on this imbalance dataset. We are going to use a two-pronged strategy to combat the class disparity and GAN (Generative adversarial network) Based Oversampling or Meta-learning in order to successfully equalize the representation of each class during training, we first design a balanced Dynamic Mini-Batch Size logic, SMOTE (Synthetic Minority Oversampling Technique) and Augmenter. Second, we provide a new loss function designed to take into account the unique features of the dataset. The outcomes are startling. These findings underscore the efficacy of our hybrid approach in training deep convolutional neural networks for imbalanced skin disease datasets. By combining data-level balancing techniques with a carefully designed loss function at the algorithm level, we address the challenge of among the classes, paving the way for more accurate and reliable skin cancer diagnoses. Key Words: Dermoscopy images, DenseNet169, Convolution neural network, SMOTE (Synthetic Minority Over-sampling Technique, GAN(Generative adversarial network), Mini-Batch Logic, Augmenter

Prediction of sloshing pressure using image-based deep learning

This study investigates an image-based deep learning method for predicting various types of sloshing pressure, including regular, weak impulse, random, and peak pressures. The time series of pressure at a specific location on a tank wall are estimated from independent features of the sequential images of free-surface waves in sloshing flow. A dataset of wave images labeled with pressure is inserted into the residual neural network (ResNet) to enable the training of neural networks without encountering the problem of vanishing gradients. The hyperparameters of the ResNet algorithm are tuned to determine the optimal learning rate, mini-batch size, and strength of L1 regularization. Compared with experimental measurements, the prediction performance regarding the time history of pressure is remarkably good in the non-resonance regime, and the predictions exhibit reasonable agreement in the resonance regime. Notably, the method for pressure normalization has a significant impact on the prediction performance for peak pressure in the resonance regime, affecting both the magnitude and profile of the pressure. Furthermore, to enable practical applications of the proposed model, training is conducted using a comprehensive dataset of wave images including all pressure types. The model successfully estimates the time history for the different pressure types that occur sequentially.

Phase transitions in the mini-batch size for sparse and dense two-layer neural networks

The use of mini-batches of data in training artificial neural networks is nowadays very common. Despite its broad usage, theories explaining quantitatively how large or small the optimal mini-batch size should be are missing. This work presents a systematic attempt at understanding the role of the mini-batch size in training two-layer neural networks. Working in the teacher-student scenario, with a sparse teacher, and focusing on tasks of different complexity, we quantify the effects of changing the mini-batch size m. We find that often the generalization performances of the student strongly depend on m and may undergo sharp phase transitions at a critical value mc , such that for m<mc the training process fails, while for m>mc the student learns perfectly or generalizes very well the teacher. Phase transitions are induced by collective phenomena firstly discovered in statistical mechanics and later observed in many fields of science. Observing a phase transition by varying the mini-batch size across different architectures raises several questions about the role of this hyperparameter in the neural network learning process.

An Inexact Sequential Quadratic Programming Method for Learning and Control of Recurrent Neural Networks.

This article considers the two-stage approach to solving a partially observable Markov decision process (POMDP): the identification stage and the (optimal) control stage. We present an inexact sequential quadratic programming framework for recurrent neural network learning (iSQPRL) for solving the identification stage of the POMDP, in which the true system is approximated by a recurrent neural network (RNN) with dynamically consistent overshooting (DCRNN). We formulate the learning problem as a constrained optimization problem and study the quadratic programming (QP) subproblem with a convergence analysis under a restarted Krylov-subspace iterative scheme that implicitly exploits the structure of the associated Karush-Kuhn-Tucker (KKT) subsystem. In the control stage, where a feedforward neural network (FNN) controller is designed on top of the RNN model, we adapt a generalized Gauss-Newton (GGN) algorithm that exploits useful approximations to the curvature terms of the training data and selects its mini-batch step size using a known property of some regularization function. Simulation results are provided to demonstrate the effectiveness of our approach.

ResNet-50 vs. EfficientNet-B0: Multi-Centric Classification of Various Lung Abnormalities Using Deep Learning

Lung abnormalities are among the significant contributors to morbidity and mortality worldwide. It induces symptoms like coughing, sneezing, fever, breathlessness, etc., which, if left untreated, may lead to death. In current clinical practice, chest X-ray (CXR) images are widely preferred to diagnose different lung abnormalities. However, the pathological tests are time-consuming, expensive and require domain experts. On the other hand, diagnosis through CXR images is manual and subject to inter-observer and intra-observer variability. The recent advancement in deep learning (DL) algorithms may be employed to address these challenges. However, the selection of correct algorithms along with finetuned parameters is challenging. In this study, we comprehensively compared the performance of two state-of-the-art DL algorithms, Resnet-50 and Efficient-B0. These two models are pervasively used in literature and have shown promising classification performance. The performance of the used algorithms is validated using the multi-centric dataset from Kaggle (having Covid-19, Normal, Pneumonia classes) and Mendeley chest X-ray dataset (having Normal, Pneumonia-Bacterial, Pneumonia-Viral, Covid-19 classes). Upon training the algorithms with a mini-batch size 32 and a maximum epoch of 40 using training data, we achieved 0.9807 and 0.9874 accuracy for Kaggle Dataset and Mendeley Dataset, respectively. Similarly, we achieved 0.9962 and 9.9978 for Kaggle dataset and Mendeley dataset for test data, respectively. From the result, it is evident that the EfficientNet-B0 model outperformed the multi-centric datasets.

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models.

In deep learning, different kinds of deep networks typically need different optimizers, which have to be chosen after multiple trials, making the training process inefficient. To relieve this issue and consistently improve the model training speed across deep networks, we propose the ADAptive Nesterov momentum algorithm, Adan for short. Adan first reformulates the vanilla Nesterov acceleration to develop a new Nesterov momentum estimation (NME) method, which avoids the extra overhead of computing gradient at the extrapolation point. Then Adan adopts NME to estimate the gradient's first- and second-order moments in adaptive gradient algorithms for convergence acceleration. Besides, we prove that Adan finds an ϵ-approximate first-order stationary point within O(ϵ-3.5) stochastic gradient complexity on the non-convex stochastic problems (e.g.deep learning problems), matching the best-known lower bound. Extensive experimental results show that Adan consistently surpasses the corresponding SoTA optimizers on vision, language, and RL tasks and sets new SoTAs for many popular networks and frameworks, egResNet, ConvNext, ViT, Swin, MAE, DETR, GPT-2, Transformer-XL, and BERT. More surprisingly, Adan can use half of the training cost (epochs) of SoTA optimizers to achieve higher or comparable performance on ViT, GPT-2, MAE, etc, and also shows great tolerance to a large range of minibatch size, e.g.from 1k to 32k. Code is released at https://github.com/sail-sg/Adan, and has been used in multiple popular deep learning frameworks or projects.

Inexact tensor methods and their application to stochastic convex optimization

We propose general non-accelerated [The results for non-accelerated methods first appeared in December 2020 in the preprint (A. Agafonov, D. Kamzolov, P. Dvurechensky, and A. Gasnikov, Inexact tensor methods and their application to stochastic convex optimization, preprint 2020. arXiv:2012.15636)] and accelerated tensor methods under inexact information on the derivatives of the objective, analyse their convergence rate. Further, we provide conditions for the inexactness in each derivative that is sufficient for each algorithm to achieve a desired accuracy. As a corollary, we propose stochastic tensor methods for convex optimization and obtain sufficient mini-batch sizes for each derivative.

Deep learning model to identify potential occurrence of reduced LVEF in patients with implantable loop recorders

Abstract Background Early detection of left ventricular systolic dysfunction (LVSD) using minimally invasive electrocardiographic assessment in an ambulatory setting may be important considering the increasing economic burden of HF. Recently artificial intelligence (AI) algorithms have been reported to classify reduced ejection fraction using snapshot 12-lead ECG measurements, however the ability of AI to identify LVSD using only a single lead ambulatory ECG is unknown. Purpose We aimed to develop a convolution neural network (CNN) based deep learning algorithm using single lead ECG to predict ejection fraction less than 40% in ambulatory patients implanted with implantable loop recorders (ILRs). Methods We linked ILR patients with LVEF measurements from a de-identified database of aggregated electronic health record (EHR) data during the period of 2007-2021 to a manufacturer’s device database with ambulatory single lead ECGS. The routine ECG transmissions from the ILR devices were paired with LVEF measured within a week. The data was pre-processed to obtain the time-frequency components of ECG using wavelet transform. The wavelet coefficients were then used as input to train and validate a 2D-CNN model. An independent test dataset consisting of data from patient not included in training/validation dataset was used to assess the performance of CNN model to classify LVEF&amp;lt; = 40% using the metrics area under the curve (AUC), sensitivity and specificity. Results A total of 35,741 unique LVEF-ECG pairs collected from 2,249 ILR patients were used to train the model; an independent validation dataset consisted of 6,721 unique LVEF ECG pairs from 750 patients and independent test dataset consisted of 6,611 unique LVEF ECG pairs from 750 patients. In our CNN model, cross entropy was used as loss function and adaptive moment estimation (ADAM) was used as the optimizer. The initial learning rate was selected as 0.0001, epochs as 20 and mini batch size as 64. The threshold to delineate reduced ejection fraction (LVEF&amp;lt; = 40%) was set as 0.1 (Figure 1). The model yielded accuracy, sensitivity, specificity, AUC of 75%,70%, 76% and 0.8 respectively in the independent test set (Figure 2). Conclusion A deep learning algorithm applied to ambulatory single lead ECGs acquired by ILR can detect LVEF&amp;lt; = 40%. This continuous ambulatory AI-ECG monitoring may allow to identify longitudinal changes in LVEF post ILR implant via remote monitoring.Figure 1:Model output vs Actual EF rangeFigure 2:ROC

Towards accelerating model parallelism in distributed deep learning systems.

Modern deep neural networks cannot be often trained on a single GPU due to large model size and large data size. Model parallelism splits a model for multiple GPUs, but making it scalable and seamless is challenging due to different information sharing among GPUs with communication overhead. Specifically, we identify two key issues to make the parallelism being inefficient and inaccurate; an efficient pipelining technique is crucial to maximize GPU utilization and normalizations in deep neural networks may affect the performance due to different statistics sharing of mini-batch. In this work, we address these issues by investigating efficient pipelining for model parallelism and effective normalizations in model / data parallelisms when training a model with large mini-batch in multiple GPUs so that the model performance in accuracy can not be compromised. Firstly, we propose a novel method to search for an optimal micro-batch size considering the number of GPUs and memory size for model parallelism. For efficient pipelining, mini-batch is usually divided into smaller batches (called micro-batch). To maximize the utilization of GPU computing resources, training should be performed with the optimal micro-batch size. Our proposed micro-batch size search algorithm achieved increased image throughput by up to 12% and improved trainable mini-batch size by 25% as compared to the conventional model parallelism method. Secondly, we investigate normalizations in distributed deep learning training for different parallelisms. Our experiments using different normalization methods suggested that the performance with batch normalization can be improved by sharing the batch information among GPUs when performing data parallelism. It was also confirmed that group normalization helped minimizing accuracy degradation when performing model parallelism with pipelining and yielded consistent accuracies for diverse mini-batch sizes.

AdaSAM: Boosting sharpness-aware minimization with adaptive learning rate and momentum for training deep neural networks

Sharpness aware minimization (SAM) optimizer has been extensively explored as it can generalize better for training deep neural networks via introducing extra perturbation steps to flatten the landscape of deep learning models. Integrating SAM with adaptive learning rate and momentum acceleration, dubbed AdaSAM, has already been explored empirically to train large-scale deep neural networks without theoretical guarantee due to the triple difficulties in analyzing the coupled perturbation step, adaptive learning rate and momentum step. In this paper, we try to analyze the convergence rate of AdaSAM in the stochastic non-convex setting. We theoretically show that AdaSAM admits a O(1/bT) convergence rate, which achieves linear speedup property with respect to mini-batch size b. Specifically, to decouple the stochastic gradient steps with the adaptive learning rate and perturbed gradient, we introduce the delayed second-order momentum term to decompose them to make them independent while taking an expectation during the analysis. Then we bound them by showing the adaptive learning rate has a limited range, which makes our analysis feasible. To the best of our knowledge, we are the first to provide the non-trivial convergence rate of SAM with an adaptive learning rate and momentum acceleration. At last, we conduct several experiments on several NLP tasks and the synthetic task, which show that AdaSAM could achieve superior performance compared with SGD, AMSGrad, and SAM optimizers.