Mini-batch Size Research Articles

Performance analysis of deep learning architectures for plant leaves disease detection

Plant disease is a major concern in agriculture field. Different pathogens are causes of plant disease. Manual inspection sometimes can cause errors in the disease detection over large cultivation area. Large number of resources and high level of expertise is required for correct prediction of diseases. Sometime the experts are not available in the nearby. Some diseases are so common and have same symptoms even the different experts may give different opinion about the same disease.In this work different deep architectures are studied and tested for plant leaves disease detection. For carrying out the study plant village dataset of 20640 images are used which represents 15 class and 3 species namely pepper, potato, and tomato. Total images in the dataset are divided in three sets i.e., 70% for training, 20% for validation and 10% for testing purpose. MATLAB R2019b is utilized for experimentation. Appropriate value of hyper parameter is determined initially by varying number of epochs, learning rate. Performance of Adam optimizer is compared with Sgdm Optimizer. Finally, accuracy of eight deep learning architectures is calculated at 16 & 32 minibatch size.

A new lightweight deep neural network for surface scratch detection.

This paper aims to develop a lightweight convolutional neural network, WearNet, to realise automatic scratch detection for components in contact sliding such as those in metal forming. To this end, a large surface scratch dataset obtained from cylinder-on-flat sliding tests was used to train the WearNet with appropriate training parameters such as learning rate, gradient algorithm and mini-batch size. A comprehensive investigation on the network response and decision mechanism was also conducted to show the capability of the developed WearNet. It was found that compared with the existing networks, WearNet can realise an excellent classification accuracy of 94.16% with a much smaller model size and faster detection speed. Besides, WearNet outperformed other state-of-the-art networks when a public image database was used for network evaluation. The application of WearNet in an embedded system further demonstrated such advantages in the detection of surface scratches in sheet metal forming processes.

Biomedical Image Classification via Dynamically Early Stopped Artificial Neural Network

It is well known that biomedical imaging analysis plays a crucial role in the healthcare sector and produces a huge quantity of data. These data can be exploited to study diseases and their evolution in a deeper way or to predict their onsets. In particular, image classification represents one of the main problems in the biomedical imaging context. Due to the data complexity, biomedical image classification can be carried out by trainable mathematical models, such as artificial neural networks. When employing a neural network, one of the main challenges is to determine the optimal duration of the training phase to achieve the best performance. This paper introduces a new adaptive early stopping technique to set the optimal training time based on dynamic selection strategies to fix the learning rate and the mini-batch size of the stochastic gradient method exploited as the optimizer. The numerical experiments, carried out on different artificial neural networks for image classification, show that the developed adaptive early stopping procedure leads to the same literature performance while finalizing the training in fewer epochs. The numerical examples have been performed on the CIFAR100 dataset and on two distinct MedMNIST2D datasets which are the large-scale lightweight benchmark for biomedical image classification.

Hyper Parameters Optimization for Effective Brain Tumor Segmentation with YOLO Deep Learning

In the field of neuroimaging, differential diagnosis of brain tumour primarily relies on the visual appearance of the tumor in the MRI. For better analysis of brain tumours segmentation of tumour ROI is indispensable. The objective of this work is to develop a deep network for brain tumour segmentation using YOLO with T1C+ High grade and FLAIR low grade tumour images from the BRATS dataset. As the hyper parameters play a crucial role in the efficacy of the CNN, an exhaustive search is made to arrive at the optimal hyper parameters. The number of anchor boxes, mini batch size and the learning algorithm are carefully evaluated to arrive at the optimal values. The visual analysis of segmentation proves the effectiveness with 4 anchor boxes, mini batch size 16 and Adam optimizer. The mean average precision value of 0.9688 and recall value of 0.9841 is obtained for HGG segmentation. The mean average precision value as 0.8435 and recall value of 0.8552 is obtained for LGG segmentation. The accuracy of the tumor detector computed with Mean IoU score is 100% for HGG and 98% for LGG tumor images. The tumor image segmented using YOLO is further subjected to histogram and region growing techniques to segment the ring and lesion core of the brain tumor which may be used for further analysis of benign and malignant tumor. As YOLO CNN could fix the bounding box with highest accuracy, the traditional skull stripping pre-processing techniques can be skipped for tumor segmentation.

Stochastic privacy-preserving methods for nonconvex sparse learning

Sparse learning is essential in mining high-dimensional data. Iterative hard thresholding (IHT) methods are effective for optimizing nonconvex objectives for sparse learning. However, IHT methods are vulnerable to adversary attacks that infer sensitive data. Although pioneering works attempted to relieve such vulnerability, they confront the issue of high computational cost for large-scale problems. We propose two differentially private stochastic IHT: one based on the stochastic gradient descent method (DP-SGD-HT) and the other based on the stochastically controlled stochastic gradient method (DP-SCSG-HT). The DP-SGD-HT method perturbs stochastic gradients with small Gaussian noise rather than full gradients, which are computationally expensive. As a result, computational complexity is reduced from O(nlog(n)) to a lower O(blog(n)), where n is the sample size and b is the mini-batch size used to compute stochastic gradients. The DP-SCSG-HT method further perturbs the stochastic gradients controlled by large-batch snapshot gradients to reduce stochastic gradient variance. We prove that both algorithms guarantee differential privacy and have linear convergence rates with estimation bias. A utility analysis examines the relationship between convergence rate and the level of perturbation, yielding the best-known utility bound for nonconvex sparse optimization. Extensive experiments show that our algorithms outperform existing methods.

Automated particle recognition for engine soot nanoparticles.

A pre-trained convolution neural network based on residual error functions (ResNet) was applied to the classification of soot and non-soot carbon nanoparticles in TEM images. Two depths of ResNet, one 18 layers deep and the other 50 layers deep, were trained using training-validation sets of increasing size (containing 100, 400 and 1400 images) and were assessed using an independent test set of 200 images. Network training was optimised in terms of mini-batch size, learning rate and training length. In all tests, ResNet18 and ResNet50 had statistically similar performances, though ResNet18 required only 25-35% of the training time of ResNet50. Training using the 100-, 400- and 1400-image training-validation sets led to classification accuracies of 84%, 88% and 95%, respectively. ResNet18 and ResNet50 were also compared for their ability to categorise soot and non-soot nanoparticles via a fivefold cross-validation experiment using the entire set of 800 images of soot and 800 images of non-soot. Cross-validation was repeated 3 times with different training durations. For all cross-validation experiments, classification accuracy exceeded 91%, with no statistical differences between any of the network trainings. The most efficient network was ResNet18 trained for 5 epochs, which reached 91.2% classification after only 84 s of training on 1600 images. Use of ResNet for classification of 1000 images, the amount suggested for reliable characterisation of soot sample, requires <4 s, compared with >30 min for a skilled operator classifying images manually. Use of convolution neural networks for classification of soot and non-soot nanoparticles in TEM images is highly promising, particularly when manually classified data sets have already been established.

Globally Convergent Multilevel Training of Deep Residual Networks

We propose a globally convergent multilevel training method for deep residual networks (ResNets). The devised method can be seen as a novel variant of the recursive multilevel trustregion (RMTR) method, which operates in hybrid (stochastic-deterministic) settings by adaptively adjusting minibatch sizes during the training. The multilevel hierarchy and the transfer operators are constructed by exploiting a dynamical system's viewpoint, which interprets forward propagation through the ResNet as a forward Euler discretization of an initial value problem. In contrast to traditional training approaches, our novel RMTR method also incorporates curvature information on all levels of the multilevel hierarchy by means of the limited-memory SR1 method. The overall performance and the convergence properties of the our multilevel training method are numerically investigated using examples from the field of classification and regression.

Automated multi-class classification of lung diseases from CXR-images using pre-trained convolutional neural networks

According to the World Health Organization (WHO), Pneumonia, COVID-19, Tuberculosis, and Pneumothorax are the leading death causes in the world. Coughing, sneezing, fever, and shortness of breath are common symptoms. To detect them, several tests such as molecular tests (RT-PCR), antigen tests, Monteux tuberculin skin test (TST), and complete blood count (CBC) tests are needed. But these are time-consuming processes and have an error rate of 20% and a sensitivity of 80%. So, radiographic tests like computed tomography (CT) and an X-ray are used to identify lung diseases with the help of a physician. But the risk of these lung diseases’ diagnoses overlapping features in chest radiographs is a worry with chest X-ray or CT-scan images. To accurately classify one of four diseases with healthy images demands the automation of such a process. There is no method for identifying and categorizing these lung diseases. As a result, we were encouraged to use eight pre-trained convolutional neural networks (CNN) to classify various lung diseases into COVID-19, pneumonia, pneumothorax, tuberculosis, and normal images from the chest X-ray image dataset. This classification process is divided into two phases. In the training phase, the CNNs are trained with the Adam optimizer with a maximum epoch of 30 and a mini-batch size of 32. In the classification phase, these trained networks are used to classify diseases. In both phases, the dataset is color preprocessed, resized, and undergoes data augmentation. For this, we used eight pre-trained CNNs: Alexnet,Darknet-19, Darknet-53, Densenet-201, Googlenet, InceptionResnetV2, MobilenetV2, and Resnet-18. Finally, we concluded that the best one to classify these diseases. Among these networks, Densenet-201, achieved the highest accuracy of 97.2%, 94.28% of sensitivity, and 97.92% of specificity for K=5. For K=10, it achieved 97.49% of accuracy, 95.57% of sensitivity, and 97.96% of specificity and for K=15, achieved 97.01% of accuracy, 96.71% of sensitivity, and 97.17% of specificity. Hence, the proposed method outperformed the existing state-of-the-art methods. Finally, our proposed research could aid clinicians in making quick conclusions concerning lung problems so that treatment can proceed.

Using transfer learning for automated microbleed segmentation.

Cerebral microbleeds are small perivascular hemorrhages that can occur in both gray and white matter brain regions. Microbleeds are a marker of cerebrovascular pathology and are associated with an increased risk of cognitive decline and dementia. Microbleeds can be identified and manually segmented by expert radiologists and neurologists, usually from susceptibility-contrast MRI. The latter is hard to harmonize across scanners, while manual segmentation is laborious, time-consuming, and subject to interrater and intrarater variability. Automated techniques so far have shown high accuracy at a neighborhood ("patch") level at the expense of a high number of false positive voxel-wise lesions. We aimed to develop an automated, more precise microbleed segmentation tool that can use standardizable MRI contrasts. We first trained a ResNet50 network on another MRI segmentation task (cerebrospinal fluid vs. background segmentation) using T1-weighted, T2-weighted, and T2* MRIs. We then used transfer learning to train the network for the detection of microbleeds with the same contrasts. As a final step, we employed a combination of morphological operators and rules at the local lesion level to remove false positives. Manual segmentation of microbleeds from 78 participants was used to train and validate the system. We assessed the impact of patch size, freezing weights of the initial layers, mini-batch size, learning rate, and data augmentation on the performance of the Microbleed ResNet50 network. The proposed method achieved high performance, with a patch-level sensitivity, specificity, and accuracy of 99.57, 99.16, and 99.93%, respectively. At a per lesion level, sensitivity, precision, and Dice similarity index values were 89.1, 20.1, and 0.28% for cortical GM; 100, 100, and 1.0% for deep GM; and 91.1, 44.3, and 0.58% for WM, respectively. The proposed microbleed segmentation method is more suitable for the automated detection of microbleeds with high sensitivity.

Convolutional neural networks for approximating electrical and thermal conductivities of Cu-CNT composites

This article explores the deep learning approach towards approximating the effective electrical and thermal conductivities of copper (Cu)-carbon nanotube (CNT) composites with CNTs aligned to the field direction. Convolutional neural networks (CNN) are trained to map the two-dimensional images of stochastic Cu-CNT networks to corresponding conductivities. The CNN model learns to estimate the Cu-CNT composite conductivities for various CNT volume fractions, interfacial electrical resistances, Rc = 20 Ω–20 kΩ, and interfacial thermal resistances, R″t,c = 10−10–10−7 m2K/W. For training the CNNs, the hyperparameters such as learning rate, minibatch size, and hidden layer neurons are optimized. Without iteratively solving the physical governing equations, the trained CNN model approximates the electrical and thermal conductivities within a second with the coefficient of determination (R2) greater than 98%, which may take longer than 100 min for a convectional numerical simulation. This work demonstrates the potential of the deep learning surrogate model for the complex transport processes in composite materials.

Automatic Seizure Detection Using Modified CNN Architecture and Activation Layer

An epileptology expert must visually inspect the EEG to identify abnormal neural activity, which is time-consuming and subject to human errors. The capability of convolution neural networks (CNN) to extract visuospatial features and learn from these discriminative features makes them useful for this task. This paper presents seizure classification based on long-term EEGs using CNN. After filtering, the scalogram is plotted using a 1-second window each. A recently published dataset (TUSZ v1.5.2) was used for the performance evaluation of various CNN-based deep neural networks. The best accuracy obtained for GoogLeNet and AlexNet is 95.88%, and 95.79% respectively with 50 epochs and 32 mini-batch sizes by using the SWISH activation function. The proposed hybrid architecture (AG86) for epoch 50 with mini-batch size 32 has shown the best testing results in terms of accuracy (94.98%) as compared to the SqueezeNet (93.19%), GoogLeNet (92.65%), and AlexNet (94.44%). Similar performance was observed using metrics specificity, sensitivity, Mathew correlation coefficient (MCC), and F1 score. A general inference based on evaluation can be drawn as the proposed hybrid architecture (AG86) showed better test results compared to pre-trained CNN models. Moreover, by replacing ReLU with the SWISH activation function, the performance of AlexNet and GoogLeNet improved.

AMBLE: Adjusting mini-batch and local epoch for federated learning with heterogeneous devices

As data privacy becomes increasingly important, federated learning applied to the training of deep learning models while ensuring the data privacy of devices is entering the spotlight. Federated learning makes it possible to process all data at once while processing data independently from various devices without collecting distributed local data in a central server. However, there are still challenges to overcome for the system of devices in federated learning such as communication overheads and the heterogeneity of the system. In this paper, we propose the Adjusting Mini-Batch and Local Epoch (AMBLE) approach, which adaptively adjusts the local mini-batch and local epoch size for heterogeneous devices in federated learning and updates the parameters synchronously. With AMBLE, we enhance the computational efficiency by removing stragglers and scaling the local learning rate to improve the model convergence rate and accuracy. We verify that federated learning with AMBLE is a stably trained model with a faster convergence speed and higher accuracy than FedAvg and adaptive batch size scheme for both identically and independently distributed (IID) and non-IID cases.

Thai digit handwriting image classification with convolutional neural networks

This paper aims to determine the efficiency in classifying and recognizing Thai digit handwritten using convolutional neural networks (CNN). We created a new dataset called the Thai digit dataset. The performance test was divided into two parts: the first part determines the exact number of epochs, and the second part examines the occurrence of overfits in the model with Keras library's EarlyStoping() function, processed through Cloud Computing with Google Colaboratory, and used a Python programming language. The main parameters for the model were a dropout of 0.75, mini-batch size of 128, the learning rate of 0.0001, and using an Adam optimizer. This study found the model's predictive accuracy was 96.88 and the loss was 0.1075. The results showed that using CNN in image classification and recognition. It has a high level of prediction efficiency. However, the parameters in the model must be adjusted accordingly.

Dorsal Hand Vein Identification using Transfer Learning from AlexNet

Dorsal hand vein pattern is a highly secured biometric system that has been used in many applications due to its non-contact attributes. Prior studies focused on investigation of different deep networks for hand vein classification task using different training parameters. It is the aim of this study to propose the use of systematic fine-tuning system for identifying the best parameters value for enhanced model learning efficiency. In this study, pre-trained AlexNet was trained using Bosphorus hand vein database for identification of 100 users. The experiments were carried out using original images, and preprocessed (cropped) images for comparison. The testing accuracies of these datasets were compared following tuning of training parameters, namely trainingand testing split ratio, number of epochs, mini-batch size and initial learning rate. It was observed that the testing accuracy of the model trained using cropped images is higher than that using the original images. The model from preprocessed dataset shows a good testing accuracy of 96 % using a split ratio of 90:10, epoch 50, mini-batch-size of 10 and an initial learning rate of 0.0001. The performance of our trained model is more superior than many reported results in the field. In future, the performance of this classification system may be further enhanced with automatic search of parameters for improved model training efficiency.

Finite-sum smooth optimization with SARAH

We introduce NC-SARAH for non-convex optimization as a practical modified version of the original SARAH algorithm that was developed for convex optimization. NC-SARAH is the first to achieve two crucial performance properties at the same time—allowing flexible minibatch sizes and large step sizes to achieve fast convergence in practice as verified by experiments. NC-SARAH has a close to optimal asymptotic convergence rate equal to existing prior variants of SARAH called SPIDER and SpiderBoost that either use an order of magnitude smaller step size or a fixed minibatch size. For convex optimization, we propose SARAH++ with sublinear convergence for general convex and linear convergence for strongly convex problems; and we provide a practical version for which numerical experiments on various datasets show an improved performance.

An image based prediction system for early blight disease in tomato plants using deep learning algorithm

The global economic system directly depends on the agricultural and farming industry. However, plant diseases are a peril to crops that deteriorates the quality as well as quantity of agricultural produce which pose huge loss to the economy. Therefore, early detection of diseases is crucial to prevent the wrecking of crops. The main aim of this study is to detect the early blight disease in tomato plants beforehand using a prominent deep learning approach, i.e., Convolutional Neural Network (CNN). The current study uses an image-based Tomato Early Blight Disease (TEBD) dataset to develop a disease prediction model. Various image processing techniques, i.e., Background Removal, Augmentation, Resizing, Noise Removal, and Segmentation were applied to obtain a refined dataset. Further, the modified TEBD dataset was trained using the CNN approach to develop an image-based TEBD prediction model. The impact of hyperparameters such as epochs and mini-batch size were also analyzed for different train-test ratios. Afterwards, the performance of the proposed model was precisely evaluated using various performance metrics, i.e., Accuracy, F1-Score, Test loss, Precision, and Recall. The mean value was calculated of different performance metrics for all nine splitting ratios. The best mean accuracy achieved was 98.10% having batch-size 64 and 15 epochs. This model can be used by farmers to reduce the workload and thus will become feasible for them in early treatment and curing of disease. In this way, the proposed approach will prevent the plants from getting severely affected in the early stage.

Detection and classification of chilli leaf disease using a squeeze-and-excitation-based CNN model

Chilli leaf disease has a destructive effect on the chilli crop yield. Chilli leaf disease can result in a significant decrease in both the quantity and quality of the chilli crop. Early detection, perfect identification and accurately diagnosing the disease will aid in increasing the profit of the cultivator. However, after a comprehensive survey investigation, we discovered that no studies have been previously conducted to compare the classification performance of machine learning and deep learning for the chilli leaf disease classification problem. In this study, five main leaf diseases i.e. down curl of a leaf, Geminivirus, Cercospora leaf spot, yellow leaf disease, and up curl disease were identified, and images were captured using a digital camera and are labelled. These diseases were classified using 12 different pretrained deep learning networks (AlexNet, DarkNet53, DenseNet201, EfficientNetb0, InceptionV3, MobileNetV2, NasNetLarge, ResNet101, ShuffleNet, SqueezeNet, VGG19, and XceptionNet) using chilli leaf data with and without augmentation using deep learning transfer. Performance metrics such as accuracy, recall, precision, F1-score, specificity, and misclassification were calculated for each network. VGG19 had the best accuracy (83.54%) without augmentation, and DarkNet53 had the best result (98.82%) with augmentation among all pretrained deep learning networks in our self-built chilli leaf dataset. The result was enhanced by designing a squeeze-and-excitation-based convolutional neural network (SECNN) model. The model was tested on a chilli leaf dataset with different input sizes and mini-batch sizes. The proposed model produced the best accuracy of 98.63% and 99.12% without and with augmentation, respectively. The SECNN model was also tested on different datasets from the PlantVillage data, including apple, cherry, corn, grape, peach, pepper, potato, strawberry, and tomato leaves, separately and with the chilli dataset. The proposed model achieved an accuracy of 99.28% in classifying 43 different classes of plant leaf datasets.

Provable training of a ReLU gate with an iterative non-gradient algorithm

In this work, we demonstrate provable guarantees on the training of a single ReLU gate in hitherto unexplored regimes. We give a simple iterative stochastic algorithm that can train a ReLU gate in the realizable setting in linear time while using significantly milder conditions on the data distribution than previous such results.Leveraging certain additional moment assumptions, we also show a first-of-its-kind approximate recovery of the true label generating parameters under an (online) data-poisoning attack on the true labels, while training a ReLU gate by the same algorithm. Our guarantee is shown to be nearly optimal in the worst case and its accuracy of recovering the true weight degrades gracefully with increasing probability of attack and its magnitude.For both the realizable and the non-realizable cases as outlined above, our analysis allows for mini-batching and computes how the convergence time scales with the mini-batch size. We corroborate our theorems with simulation results which also bring to light a striking similarity in trajectories between our algorithm and the popular S.G.D. algorithm — for which similar guarantees as here are still unknown.

Deep Learning-Based Brain Computed Tomography Image Classification with Hyperparameter Optimization through Transfer Learning for Stroke.

Brain computed tomography (CT) is commonly used for evaluating the cerebral condition, but immediately and accurately interpreting emergent brain CT images is tedious, even for skilled neuroradiologists. Deep learning networks are commonly employed for medical image analysis because they enable efficient computer-aided diagnosis. This study proposed the use of convolutional neural network (CNN)-based deep learning models for efficient classification of strokes based on unenhanced brain CT image findings into normal, hemorrhage, infarction, and other categories. The included CNN models were CNN-2, VGG-16, and ResNet-50, all of which were pretrained through transfer learning with various data sizes, mini-batch sizes, and optimizers. Their performance in classifying unenhanced brain CT images was tested thereafter. This performance was then compared with the outcomes in other studies on deep learning–based hemorrhagic or ischemic stroke diagnoses. The results revealed that among our CNN-2, VGG-16, and ResNet-50 analyzed by considering several hyperparameters and environments, the CNN-2 and ResNet-50 outperformed the VGG-16, with an accuracy of 0.9872; however, ResNet-50 required a longer time to present the outcome than did the other networks. Moreover, our models performed much better than those reported previously. In conclusion, after appropriate hyperparameter optimization, our deep learning–based models can be applied to clinical scenarios where neurologist or radiologist may need to verify whether their patients have a hemorrhage stroke, an infarction, and any other symptom.

The analysis and optimization of CNN Hyperparameters with fuzzy tree modelfor image classification

The meaningful performance of convolutional neural network (CNN) has enabled the solution of various state-of-the-art problems. Although CNNs achieve satisfactory results in computer-vision problems, they still have some difficulties. As the designed CNN models are deepened to achieve much better accuracy, computational cost and complexity increase. It is significant to train CNNs with suitable topology and training hyperparameters that include initial learning rate, minibatch size, epoch number, filter size, number of filters, etc. because the initialization of hyperparameters affects classification results. On the other hand, it is not possible to make a definite inference for the hyperparameter initialization and there is uncertainty. This study is carried out to model uncertainty using fuzzy inference system (FIS). The designed fuzzy model provides estimation of classification result depending on CNN topology and training hyperparameters. GoogleNet and Inceptionv3 that contain inception-modules, ShuffleNet that contains shuffleblocks, DenseNet201 that contains dense-blocks, EfficientNet, ResNet18, ResNet50, ResNet101, and MobileNetv2 that contain residual-blocks, and InceptionResNetv2 that includes both inception-modules and residual-blocks were evaluated as CNN models. Test sample dataset was obtained by training CNN models with various training hyperparameter combinations. CNN models were trained on Animal Diagnostics Lab (ADL) which is a histopathological dataset includes healthy and inflamed kidney, lung, and spleen images. A new FIS tree model that is more computationally efficient and easier to understand than a single FIS was designed and classification accuracy prediction of CNN models depending on hyperparameter combinations was performed. The best, the worst, and the average classification accuracies obtained with CNN models that use best training hyperparameter set are 97.70%, 93.60%, and 96.30%, respectively. Moreover, Cifar10 and Cifar100 benchmark datasets were experimented to reveal true capability and limitations of the proposed approach. Experimental results indicate that the designed FIS tree model provides a successful hyperparameter evaluation mechanism with an average RMSE value of 1.2652.