Mini-batch Size Research Articles

Genome-Wide Prediction of Complex Traits in Two Outcrossing Plant Species Through Deep Learning and Bayesian Regularized Neural Network.

Genomic selection models were investigated to predict several complex traits in breeding populations of Zea mays L. and Eucalyptus globulus Labill. For this, the following methods of Machine Learning (ML) were implemented: (i) Deep Learning (DL) and (ii) Bayesian Regularized Neural Network (BRNN) both in combination with different hyperparameters. These ML methods were also compared with Genomic Best Linear Unbiased Prediction (GBLUP) and different Bayesian regression models [Bayes A, Bayes B, Bayes Cπ, Bayesian Ridge Regression, Bayesian LASSO, and Reproducing Kernel Hilbert Space (RKHS)]. DL models, using Rectified Linear Units (as the activation function), had higher predictive ability values, which varied from 0.27 (pilodyn penetration of 6 years old eucalypt trees) to 0.78 (flowering-related traits of maize). Moreover, the larger mini-batch size (100%) had a significantly higher predictive ability for wood-related traits than the smaller mini-batch size (10%). On the other hand, in the BRNN method, the architectures of one and two layers that used only the pureline function showed better results of prediction, with values ranging from 0.21 (pilodyn penetration) to 0.71 (flowering traits). A significant increase in the prediction ability was observed for DL in comparison with other methods of genomic prediction (Bayesian alphabet models, GBLUP, RKHS, and BRNN). Another important finding was the usefulness of DL models (through an iterative algorithm) as an SNP detection strategy for genome-wide association studies. The results of this study confirm the importance of DL for genome-wide analyses and crop/tree improvement strategies, which holds promise for accelerating breeding progress.

Detection of Bacterial Wilt on Enset Crop Using Deep Learning Approach

Bacterial Wilt disease is the most determinant factor as it results in a serious reduction in the quality and quantity of food produced by Enset crop. Therefore, early detection of Bacterial Wilt disease is important to diagnose and fight the disease. To this end, a deep learning approach that can detect the disease by using healthy and infected leave images of the crop is proposed. In particular, a convolutional neural network architecture is designed to classify the images collected from different farms as diseased or healthy. A total of 4896 images that were captured directly from the farm with the help of experts in the field of agriculture was used to train the proposed model. The proposed model was trained using these images and data augmentation techniques was applied to generate more images. Besides training the proposed model, a pre-trained model namely VGG16 is trained by using our dataset. The proposed model achieved a mean accuracy of 98.5% and the VGG16 pre-trained model achieved a mean accuracy of 96.6% by using a mini-batch size of 32 and a learning rate of 0.001. The preliminary results demonstrated that the effectiveness of the proposed approach under challenging conditions such as illumination, complex background, different resolutions, variable scale, rotation, and orientation of the real scene images.

運転データに基づく建築設備のANNモデル構築手法に関する研究 (第1報)ミニバッチサイズが予測精度へ与える影響の評価

運転データに基づく建築設備のANNモデル構築手法に関する研究 (第1報)ミニバッチサイズが予測精度へ与える影響の評価

Deep Learning for SVD and Hybrid Beamforming

Hybrid beamforming (BF), which divides BF operation into radio frequency (RF) and baseband (BB) domains, will play a critical role in MIMO communication at millimeter-wave (mmW) frequencies. In principle, we can obtain unconstrained (optimum) beamformers of a transceiver, which approach the maximum achievable data rates, through its singular value decomposition (SVD). Due to the use of finite-precision phase shifters, combined with power constraints, additional challenges are imposed on the problem of designing hybrid beamformers. Motivated by the recent success of machine learning (ML) techniques, particularly in areas such as computer vision and speech recognition, we explore if ML techniques can be effectively used for SVD and hybrid BF. To this end, we first present a data-driven approach to compute the SVD. We propose three deep neural network (DNN) architectures to approximate the SVD, with varying levels of complexity. The methodology for training these DNN architectures is inspired by the fundamental property of SVD, i.e., it can be used to obtain low-rank approximations. We next explicitly take the constraints of hybrid BF into account (such as quantized phase shifters, power constraints), and propose a novel DNN based approach for the design of hybrid BF systems. To validate the DNN based approach, we present simulation results for both approximating the SVD as well as for hybrid BF. Our results show that DNNs can be an attractive and efficient solution for estimating SVD in a data-driven manner. For the simulations of hybrid BF, we first consider the geometric channel model. We show that the DNN based hybrid BF improves rates by up to 50 - 70% compared to conventional hybrid BF algorithms and achieves 10 - 30% gain in rates compared with the state-of-art ML-aided hybrid BF algorithms. We also discuss the impact of the choice of hyperparameters, such as the number of hidden layers, mini-batch size, and training iterations on the accuracy of DNNs. Furthermore, we provide time complexity and memory requirement analyses for the proposed approach and state-of-the-art approaches.

Inexact SARAH algorithm for stochastic optimization

We develop and analyse a variant of the SARAH algorithm, which does not require computation of the exact gradient. Thus this new method can be applied to general expectation minimization problems rather than only finite sum problems. While the original SARAH algorithm, as well as its predecessor, SVRG, requires an exact gradient computation on each outer iteration, the inexact variant of SARAH (iSARAH), which we develop here, requires only stochastic gradient computed on a mini-batch of sufficient size. The proposed method combines variance reduction via sample size selection and iterative stochastic gradient updates. We analyse the convergence rate of the algorithms for strongly convex and non-strongly convex cases, under smooth assumption with appropriate mini-batch size selected for each case. We show that with an additional, reasonable, assumption iSARAH achieves the best-known complexity among stochastic methods in the case of non-strongly convex stochastic functions.

Properties of the stochastic approximation EM algorithm with mini-batch sampling

To deal with very large datasets a mini-batch version of the Monte Carlo Markov Chain Stochastic Approximation Expectation–Maximization algorithm for general latent variable models is proposed. For exponential models the algorithm is shown to be convergent under classical conditions as the number of iterations increases. Numerical experiments illustrate the performance of the mini-batch algorithm in various models. In particular, we highlight that mini-batch sampling results in an important speed-up of the convergence of the sequence of estimators generated by the algorithm. Moreover, insights on the effect of the mini-batch size on the limit distribution are presented. Finally, we illustrate how to use mini-batch sampling in practice to improve results when a constraint on the computing time is given.

Optimal Scree-CNN for Detecting NS1 Molecular Fingerprint from Salivary SERS Spectra.

Dengue fever (DF) is a viral infection with possible fatal consequence. NS1 is a recent antigen based biomarker for dengue fever (DF), as an alternative to current serum and antibody based biomarkers. Convolutional Neural Network (CNN) has demonstrated impressive performance in machine learning problems. Our previous research has captured NS1 molecular fingerprint in saliva using Surface Enhanced Raman Spectroscopy (SERS) with great potential as an early, noninvasive detection method. SERS is an enhanced variant of Raman spectroscopy, with extremely high amplification that enables spectra of low concentration matter, such as NS1 in saliva, readable. The spectrum contains 1801 features per sample, at a total of 284 samples. Principal Component Analysis (PCA) transforms high dimensional correlated signal to a lower dimension uncorrelated principal components (PCs), at no sacrifice of the original signal content. This paper aims to unravel an optimal Scree-CNN model for classification of salivary NS1 SERS spectra. Performances of a total of 490 classifier models were examined and compared in terms of performance indicators [accuracy, sensitivity, specificity, precision, kappa] against a WHO recommended clinical standard test for DF, enzyme-linked immunosorbent assay (ELISA). Effects of CNN parameters on performances of the classifier models were also observed. Results showed that Scree-CNN classifier model with learning rate of 0.01, mini-batch size of 64 and validation frequency of 50, reported an across-the-board 100% for all performance indicators.

Scalable and Practical Natural Gradient for Large-Scale Deep Learning.

Large-scale distributed training of deep neural networks results in models with worse generalization performance as a result of the increase in the effective mini-batch size. Previous approaches attempt to address this problem by varying the learning rate and batch size over epochs and layers, or ad hoc modifications of batch normalization. We propose scalable and practical natural gradient descent (SP-NGD), a principled approach for training models that allows them to attain similar generalization performance to models trained with first-order optimization methods, but with accelerated convergence. Furthermore, SP-NGD scales to large mini-batch sizes with a negligible computational overhead as compared to first-order methods. We evaluated SP-NGD on a benchmark task where highly optimized first-order methods are available as references: training a ResNet-50 model for image classification on ImageNet. We demonstrate convergence to a top-1 validation accuracy of 75.4 percent in 5.5 minutes using a mini-batch size of 32,768 with 1,024 GPUs, as well as an accuracy of 74.9 percent with an extremely large mini-batch size of 131,072 in 873 steps of SP-NGD.

Stochastic quasi-gradient methods: variance reduction via Jacobian sketching

We develop a new family of variance reduced stochastic gradient descent methods for minimizing the average of a very large number of smooth functions. Our method—JacSketch—is motivated by novel developments in randomized numerical linear algebra, and operates by maintaining a stochastic estimate of a Jacobian matrix composed of the gradients of individual functions. In each iteration, JacSketch efficiently updates the Jacobian matrix by first obtaining a random linear measurement of the true Jacobian through (cheap) sketching, and then projecting the previous estimate onto the solution space of a linear matrix equation whose solutions are consistent with the measurement. The Jacobian estimate is then used to compute a variance-reduced unbiased estimator of the gradient. Our strategy is analogous to the way quasi-Newton methods maintain an estimate of the Hessian, and hence our method can be seen as a stochastic quasi-gradient method. Our method can also be seen as stochastic gradient descent applied to a controlled stochastic optimization reformulation of the original problem, where the control comes from the Jacobian estimates. We prove that for smooth and strongly convex functions, JacSketch converges linearly with a meaningful rate dictated by a single convergence theorem which applies to general sketches. We also provide a refined convergence theorem which applies to a smaller class of sketches, featuring a novel proof technique based on a stochastic Lyapunov function. This enables us to obtain sharper complexity results for variants of JacSketch with importance sampling. By specializing our general approach to specific sketching strategies, JacSketch reduces to the celebrated stochastic average gradient (SAGA) method, and its several existing and many new minibatch, reduced memory, and importance sampling variants. Our rate for SAGA with importance sampling is the current best-known rate for this method, resolving a conjecture by Schmidt et al. (Proceedings of the eighteenth international conference on artificial intelligence and statistics, AISTATS 2015, San Diego, California, 2015). The rates we obtain for minibatch SAGA are also superior to existing rates and are sufficiently tight as to show a decrease in total complexity as the minibatch size increases. Moreover, we obtain the first minibatch SAGA method with importance sampling.

Deep learning-based roadway crack classification using laser-scanned range images: A comparative study on hyperparameter selection

In recent years, deep learning-based crack detection methods have been widely explored and applied due to their high versatility and adaptability. In civil engineering applications, recent research on crack detection through deep convolutional neural network (DCNN) includes road pavement crack detection, bridge inspection, defects detection in shield tunnel lining, etc. Despite the increasing popularity of DCNN on crack detection, many challenges have yet to be properly addressed. For crack detection using three-dimensional (3D) range (i.e., elevation) images, disturbances such as surface variation can negatively affect the detection performance. Besides, some typical non-crack patterns such as grooves can be easily misidentified as cracks, i.e., false positives. Another issue lies in the selection of hyperparameters related with the design of a DCNN architecture. For example, the hyperparameters which are related with network structure (e.g., kernel size, network depth and width) and training (e.g., mini-batch size and learning rate) can impact the network performance to a significant extent. Therefore, they need to be properly determined for optimal performance. However, for deep learning-based roadway crack classification using laser-scanned range images, a comprehensive discussion on the hyperparameter selection/tuning has not been thoroughly performed. This study develops a hyperparameter selection process involving a series of experiments on laser-scanned range images with high diversities, investigating the optimal joint hyperparameter configuration on network structure and training for DCNN-based roadway crack classification. In a comparative study, 36 DCNN architectures with varying layouts are developed for crack classification. These architecture candidates differ in kernel sizes (e.g., 3 × 3, 7 × 7, and 11 × 11), network depths (from 5 to 8 weight layers), and widths (from 16 to 96 kernels in each convolutional layer). The 7-layer DCNN with constant 7 × 7 kernels and increasing network widths yields the highest classification performance among the proposed 36 DCNN classifiers, which may be because it can best reflect the complexity of the acquired laser-scanned roadway range images. Once the optimal architecture layout is determined, further discussion on the selection of min-batch sizes, learning rates, dropout factor and leaky rectified linear unit (LReLU) factor is performed. Experimental results show the optimal architecture with associated training configuration can achieve consistent and accurate performance, under the contamination of surface variations and grooved patterns in laser-scanned range images. Discussion on the hyperparameter selection can provide insights for the development of DCNN in similar applications using laser-scanned range images.

Crowding prediction on mass rapid transit systems using a weighted bidirectional recurrent neural network

Crowding on Mass Rapid Transit (MRT) Systems has been the subject of intense scrutiny from a multitude of academic fields. Crowding prediction can assist system managers in finding effective ways to ease traffic pressure. Forecasting crowding levels is a challenging task due to a series of complex factors. Deep learning models are a common approach often applied to traffic prediction, but due to the imbalance within available data, these models that are based on historical data cannot forecast crowding in Rapid Transit Systems effectively. To solve this problem, this study proposes a model named the weighted resample bidirectional recurrent neural network (WRBRNN). First, the training data were split into different sub-datasets according to certain predetermined labels. During this training time, every mini-batch size sequence was weighed and resampled from different sub-datasets. In this study, the authors carefully arranged the traffic data attributes into several time series, allowing the bidirectional time series information and the model to make reliable predictions. This work performed a case study of the Bay Area Rapid Transit, US system with a year's worth of historical data from 2017. Their results reveal that the model WRBRNN performed well when predicting crowding in an MRT system.

FPDeep: Scalable Acceleration of CNN Training on Deeply-Pipelined FPGA Clusters

Deep Neural Networks (DNNs) have revolutionized numerous applications, but the demand for ever more performance remains unabated. Scaling DNN computations to larger clusters is generally done by distributing tasks in batch mode using methods such as distributed synchronous SGD. Among the issues with this approach is that to make the distributed cluster work with high utilization, the workload distributed to each node must be large, which implies nontrivial growth in the SGD mini-batch size. In this paper, we propose a framework called FPDeep, which uses a hybrid of model and layer parallelism to configure distributed reconfigurable clusters to train DNNs. This approach has numerous benefits. First, the design does not suffer from batch size growth. Second, novel workload and weight partitioning leads to balanced loads of both among nodes. And third, the entire system is a fine-grained pipeline. This leads to high parallelism and utilization and also minimizes the time features need to be cached while waiting for back-propagation. As a result, storage demand is reduced to the point where only on-chip memory is used for the convolution layers. We evaluate FPDeep with the Alexnet, VGG-16, and VGG-19 benchmarks. Experimental results show that FPDeep has good scalability to a large number of FPGAs, with the limiting factor being the FPGA-to-FPGA bandwidth. With 6 transceivers per FPGA, FPDeep shows linearity up to 83 FPGAs. Energy efficiency is evaluated with respect to GOPs/J. FPDeep provides, on average, 6.36x higher energy efficiency than comparable GPU servers.

A 2.9–33.0 TOPS/W Reconfigurable 1-D/2-D Compute-Near-Memory Inference Accelerator in 10-nm FinFET CMOS

A 10-nm compute-near-memory (CNM) accelerator augments SRAM with multiply accumulate (MAC) units to reduce interconnect energy and achieve 2.9 8b-TOPS/W for matrix–vector computation. The CNM provides high memory bandwidth by accessing SRAM subarrays to enable low-latency, real-time inference in fully connected and recurrent neural networks with small mini-batch sizes. For workloads with greater arithmetic intensity, such as large-batch convolutional neural networks, the CNM reconfigures into a 2-D systolic array to amortize memory access energy over a greater number of computations. Variable-precision 8b/4b/2b/1b MACs increase throughput by up to $8\times $ for binary operations at 33.0 1b-TOPS/W.

An Approach to Hyperparameter Optimization for the Objective Function in Machine Learning

In machine learning, performance is of great value. However, each learning process requires much time and effort in setting each parameter. The critical problem in machine learning is determining the hyperparameters, such as the learning rate, mini-batch size, and regularization coefficient. In particular, we focus on the learning rate, which is directly related to learning efficiency and performance. Bayesian optimization using a Gaussian Process is common for this purpose. In this paper, based on Bayesian optimization, we attempt to optimize the hyperparameters automatically by utilizing a Gamma distribution, instead of a Gaussian distribution, to improve the training performance of predicting image discrimination. As a result, our proposed method proves to be more reasonable and efficient in the estimation of learning rate when training the data, and can be useful in machine learning.

A Neuronal Morphology Classification Approach Based on Locally Cumulative Connected Deep Neural Networks

Neurons are the basic building and computational units of the nervous system, and have complex and diverse spatial geometric structures. By solving the neuronal classification problem, we can further understand the characteristics of neurons and the process of information transmission. This paper presents a neuronal morphology classification approach based on locally cumulative connected deep neural networks, where 43 geometric features were extracted from two different neuron datasets and applied to classify types of neurons. Then, the effects of different parameters of deep learning networks on the performance of neuron classification were analyzed including mini-batch size, number of intermediate layers, and number of building blocks. The accuracy of the approach was also compared with that of the other mainstream machine learning approaches. The experimental results showed that the proposed approach is effective for solving complex neuronal morphology classification problems.

Random Minibatch Subgradient Algorithms for Convex Problems with Functional Constraints

In this paper we consider non-smooth convex optimization problems with (possibly) infinite intersection of constraints. In contrast to the classical approach, where the constraints are usually represented as intersection of simple sets, which are easy to project onto, in this paper we consider that each constraint set is given as the level set of a convex but not necessarily differentiable function. For these settings we propose subgradient iterative algorithms with random minibatch feasibility updates. At each iteration, our algorithms take a subgradient step aimed at only minimizing the objective function and then a subsequent subgradient step minimizing the feasibility violation of the observed minibatch of constraints. The feasibility updates are performed based on either parallel or sequential random observations of several constraint components. We analyze the convergence behavior of the proposed algorithms for the case when the objective function is strongly convex and with bounded subgradients, while the functional constraints are endowed with a bounded first-order black-box oracle. For a diminishing stepsize, we prove sublinear convergence rates for the expected distances of the weighted averages of the iterates from the constraint set, as well as for the expected suboptimality of the function values along the weighted averages. Our convergence rates are known to be optimal for subgradient methods on this class of problems. Moreover, the rates depend explicitly on the minibatch size and show when minibatching helps a subgradient scheme with random feasibility updates.

Comprehensive techniques of multi-GPU memory optimization for deep learning acceleration

This paper presents a comprehensive suite of techniques for optimized memory management in multi-GPU systems to accelerate deep learning application execution. We employ a hybrid utilization of GPU and CPU memories in a multi-GPU environment by effectively addressing contention issues in the shared interconnect (e.g., PCIe, NVLink). In addition, we designed and implemented an intelligent prefetching algorithm (from CPU memory to GPU) that achieves the highest processing throughput while sustaining a large mini-batch size. We successfully implemented our optimization techniques on TensorFlow, and performed extensive experiments in various multi-GPU environments including traditional PCIe and the latest high-bandwidth interconnect, NVLink. Evaluation results show that our proposed scheme actually improves computing performance by decreasing the I/O bottleneck, and effectively increasing the mini-batch size without sacrificing overall training throughput.

Learning with Type-2 Fuzzy activation functions to improve the performance of Deep Neural Networks

In this study, we propose a novel Interval Type-2 (IT2) Fuzzy activation layer that is composed of Single input IT2 (SIT2) Fuzzy Rectifying Units (FRUs) to improve the learning performances of Deep Neural Networks (DNNs). The novel SIT2-FRU has tunable parameters that not only define the slopes of the positive and negative quadrants but also the characteristic of the input–output mapping of the activation function. The novel SIT2-FRU also alleviates vanishing gradient problem and has a fast convergence rate since it can push the mean activation to around zero by processing the inputs defined in the negative quadrant. Thus, SIT2-FRU gives the opportunity to the DNN to have a better learning behavior as it is capable to express linear or sophisticated input–output mapping by simply tuning the footprint of uncertainty of its IT2 fuzzy sets. In order to examine the performance of the SIT2-FRU, comparative experimental studies are performed on the MNIST, Quickdraw Pictionary and CIFAR-10 benchmark datasets. The proposed SIT2-FRU is compared with the state of the art activation functions which are the Rectified Linear Unit (ReLU), Parametric ReLU (PReLU) and Exponential Linear Unit (ELU). Comparative experimental results and analyses clearly show the enhancement in the learning performance of DNNs that include activation layer(s) composed of SIT2-FRUs. It is shown that the learning performance of the SIT2-FRU is robust to different parameter settings of the learning rates and mini batch sizes. Furthermore, the experimental results show that SIT2-FRU can result with a high performance with or without batch normalization layers unlike the other employed activation units. It is concluded that DNNs with SIT2-FRUs have a satisfactory generalization capability, a robust and high learning performance when compared to the ReLU, PReLU and ELU activation functions.

Projection-domain scatter correction for cone beam computed tomography using a residual convolutional neural network.

Scatter is a major factor degrading the image quality of cone beam computed tomography (CBCT). Conventional scatter correction strategies require handcrafted analytical models with ad hoc assumptions, which often leads to less accurate scatter removal. This study aims to develop an effective scatter correction method using a residual convolutional neural network (CNN). A U-net based 25-layer CNN was constructed for CBCT scatter correction. The establishment of the model consists of three steps: model training, validation, and testing. For model training, a total of 1800 pairs of x-ray projection and the corresponding scatter-only distribution in nonanthropomorphic phantoms taken in full-fan scan were generated using Monte Carlo simulation of a CBCT scanner installed with a proton therapy system. An end-to-end CNN training was implemented with two major loss functions for 100 epochs with a mini-batch size of 10. Image rotations and flips were randomly applied to augment the training datasets during training. For validation, 200 projections of a digital head phantom were collected. The proposed CNN-based method was compared to a conventional projection-domain scatter correction method named fast adaptive scatter kernel superposition (fASKS) method using 360 projections of an anthropomorphic head phantom. Two different loss functions were applied for the same CNN to evaluate the impact of loss functions on the final results. Furthermore, the CNN model trained with full-fan projections was fine-tuned for scatter correction in half-fan scan by using transfer learning with additional 360 half-fan projection pairs of nonanthropomorphic phantoms. The tuned-CNN model for half-fan scan was compared with the fASKS method as well as the CNN-based method without the fine-tuning using additional lung phantom projections. The CNN-based method provides projections with significantly reduced scatter and CBCT images with more accurate Hounsfield Units (HUs) than that of the fASKS-based method. Root mean squared error of the CNN-corrected projections was improved to 0.0862 compared to 0.278 for uncorrected projections or 0.117 for the fASKS-corrected projections. The CNN-corrected reconstruction provided better HU quantification, especially in regions near the air or bone interfaces. All four image quality measures, which include mean absolute error (MAE), mean squared error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM), indicated that the CNN-corrected images were significantly better than that of the fASKS-corrected images. Moreover, the proposed transfer learning technique made it possible for the CNN model trained with full-fan projections to be applicable to remove scatters in half-fan projections after fine-tuning with only a small number of additional half-fan training datasets. SSIM value of the tuned-CNN-corrected images was 0.9993 compared to 0.9984 for the non-tuned-CNN-corrected images or 0.9990 for the fASKS-corrected images. Finally, the CNN-based method is computationally efficient - the correction time for the 360 projections only took less than 5s in the reported experiments on a PC (4.20GHz Intel Core-i7 CPU) with a single NVIDIA GTX 1070 GPU. The proposed deep learning-based method provides an effective tool for CBCT scatter correction and holds significant value for quantitative imaging and image-guided radiation therapy.

BOA: batch orchestration algorithm for straggler mitigation of distributed DL training in heterogeneous GPU cluster

Training deep learning model is a time-consuming job since it usually uses a large amount of data. To reduce the training time, most practitioners train the models in a GPU cluster environment in a distributed way. The synchronous stochastic gradient descent, which is one of the widely used distributed training algorithms, has fast convergence rate with the use of multiple GPU workers; but its speed is tied to the slowest worker, i.e., straggler. In a heterogeneous environment, a static straggler, which has not been mainly focused before, has more impact on the performance than randomly occurring straggler. However, most existing studies for straggler mitigation usually consider a homogeneous environment, so their approaches are limited in practice. In this paper, we scrutinize the straggler problem under heterogeneous environment and define static and dynamic straggler from empirical results. Based on this, we propose a novel approach called batch orchestration algorithm (BOA) for straggler mitigation. It adaptively balances the amount of mini-batch data according to the speed of workers. Therefore, BOA can mitigate both static and dynamic straggler in a modern GPU cluster. BOA uses a Min–Max Integer programming to find the optimal mini-batch size, with the hardware-agnostic performance models. For verification, several experiments are conducted on a cluster having up to six GPUs with three types: GTX 1080, GTX 1060 and Quadro M2000. The results show BOA mitigates both types of stragglers and accelerates the training speed with synchronous SGD compared to other straggler mitigation method.