Mini-batch Stochastic Gradient Descent Research Articles

Towards Stability and Generalization Bounds in Decentralized Minibatch Stochastic Gradient Descent

Decentralized Stochastic Gradient Descent (D-SGD) represents an efficient communication approach tailored for mastering insights from vast, distributed datasets. Inspired by parallel optimization paradigms, the incorporation of minibatch serves to diminish variance, consequently expediting the optimization process. Nevertheless, as per our current understanding, the existing literature has not thoroughly explored the learning theory foundation of Decentralized Minibatch Stochastic Gradient Descent (DM-SGD). In this paper, we try to address this theoretical gap by investigating the generalization properties of DM-SGD. We establish the sharper generalization bounds for the DM-SGD algorithm with replacement (without replacement) on (non)convex and (non)smooth cases. Moreover, our results consistently recover to the results of Centralized Stochastic Gradient Descent (C-SGD). In addition, we derive generalization analysis for Zero-Order (ZO) version of DM-SGD.

Hierarchical Weight Averaging for Deep Neural Networks.

Despite simplicity, stochastic gradient descent (SGD)-like algorithms are successful in training deep neural networks (DNNs). Among various attempts to improve SGD, weight averaging (WA), which averages the weights of multiple models, has recently received much attention in the literature. Broadly, WA falls into two categories: 1) online WA, which averages the weights of multiple models trained in parallel, is designed for reducing the gradient communication overhead of parallel mini-batch SGD and 2) offline WA, which averages the weights of one model at different checkpoints, is typically used to improve the generalization ability of DNNs. Though online and offline WA are similar in form, they are seldom associated with each other. Besides, these methods typically perform either offline parameter averaging or online parameter averaging, but not both. In this work, we first attempt to incorporate online and offline WA into a general training framework termed hierarchical WA (HWA). By leveraging both the online and offline averaging manners, HWA is able to achieve both faster convergence speed and superior generalization performance without any fancy learning rate adjustment. Besides, we also analyze the issues faced by the existing WA methods, and how our HWA addresses them, empirically. Finally, extensive experiments verify that HWA outperforms the state-of-the-art methods significantly.

Carbon Emission Forecasting Study Based on Influence Factor Mining and Mini-Batch Stochastic Gradient Optimization

With the increasing prominence of the global carbon emission problem, the accurate prediction of carbon emissions has become an increasingly urgent need. Existing carbon emission prediction methods have the problems of slow calculation speed, inaccurate prediction, and insufficient deep mining of influencing factors when dealing with large-scale data. In this study, a comprehensive carbon emission prediction method is proposed. Firstly, multiple influencing factors including economic factors and demographic factors are considered, and a pathway analysis method is introduced to mine the long-term relationship between these factors and carbon emissions. Then, indirect influence terms are added to the multiple regression equation, and the variable is used to represent the indirect influence relationship. Finally, this study proposes the PCA-PA-MBGD method, which applies the results of principal component analysis to the pathway analysis. By reducing the data dimensions and extracting the main influencing factors, and optimizing the carbon emission prediction model by using a mini-batch stochastic gradient descent algorithm, the results show that this method can process a large amount of data quickly and efficiently, and realize an accurate prediction of carbon emissions. This provides strong support for solving the carbon emission problem and offers new ideas and methods for future related research.

Towards Better Generalization of Deep Neural Networks via Non-Typicality Sampling Scheme.

Improving the generalization performance of deep neural networks (DNNs) trained by minibatch stochastic gradient descent (SGD) has raised lots of concerns from deep learning practitioners. The standard simple random sampling (SRS) scheme used in minibatch SGD treats all training samples equally in gradient estimation. In this article, we study a new data selection method based on the intrinsic property of the training set to help DNNs have better generalization performance. Our theoretical analysis suggests that this new sampling scheme, called the nontypicality sampling scheme, boosts the generalization performance of DNNs through biasing the solution toward wider minima, under certain assumptions. We confirm our findings experimentally and show that more variants of minibatch SGD can also benefit from the new sampling scheme. Finally, we discuss an extension of the nontypicality sampling scheme that holds promise to enhance both generalization performance and convergence speed of minibatch SGD.

Convolutional Neural Networks (CNNs) for Pneumonia Classification on Pediatric Chest Radiographs.

Pneumonia is an infectious disease that is especially harmful to those with weak immune systems, such as children under the age of 5. While radiologists' diagnosis of pediatric pneumonia on chest radiographs (CXRs) is often accurate, subtle findings can be missed due to the subjective nature of the diagnosis process. Artificial intelligence (AI) techniques, such as convolutional neural networks (CNNs), can help make the process more objective and precise. However, off-the-shelf CNNs may perform poorly if they are not tuned to their appropriate hyperparameters. Our study aimed to identify the CNNs and their hyperparameter combinations (dropout, batch size, and optimizer) that optimize model performance. Sixty models based on five CNNs (VGG 16, VGG 19, DenseNet 121, DenseNet 169, and InceptionResNet V2) and 12 hyperparameter combinations were tested. Adam, Root Mean Squared Propagation (RmsProp), and Mini-Batch Stochastic Gradient Descent (SGD) optimizers were used. Two batch sizes, 32 and 64, were utilized. A dropout rate of either 0.5 or 0.7 was used in all dropout layers. We used a deidentified CXR dataset of 4200 pneumonia (Figure 1a) and 1600 normal images (Figure 1b).Seventy percent of the CXRs in the dataset were used for training the model, 20% were used for validating the model, and 10% were used for testing the model. All CNNs were trained first on the ImageNet dataset. They were then trained, with frozen weights, on the CXR-containing dataset. Results:Among the 60 models, VGG-19 (dropout of 0.5, batch size of 32, and Adam optimizer) was the most accurate. This model achieved an accuracy of 87.9%. A dropout of 0.5 consistently gave higher accuracy, area under the receiver operating characteristics curve (AUROC), and area under the precision-recall curve (AUPRC) compared to a dropout of 0.7. The CNNs InceptionResNet V2, DenseNet 169, VGG 16, and VGG 19 significantly outperformed the DenseNet121 CNN in accuracy and AUROC. The Adam and RmsProp optimizer had improved AUROC and AUPRC compared to the SGD optimizer. The batch size had no statistically significant effect on model performance. We recommend using low dropout rates (0.5) and RmsProp or Adam optimizer for pneumonia-detecting CNNs. Additionally, we discourage using the DenseNet121 CNN when other CNNs are available. Finally, the batch size may be set to any value, dependent on computational resources.

Breast cancer detection using Histopathology Image with Mini-Batch Stochastic Gradient Descent and Convolutional Neural Network

Cancer is a devastating disease that has far-reaching effects on our culture and economy, in addition to the human lives it takes. Regarding budgetary responsibility, investing just in cancer treatment is not an option. Early diagnosis is a crucial part of the remedy that sometimes gets overlooked. Malignancy is often diagnosed and evaluated using Histopathology Images (HI), which are widely accepted as the gold standard in the field. Yet, even for experienced pathologists, analysing such images is challenging, which raises concerns of inter- and intra-observer variability. The analysis also requires a substantial investment of time and energy. One way that such an examination may be sped up is by making use of computer-assisted diagnostics devices. The purpose of this research is to create a comprehensive cancer detection system using images of breast and prostate histopathology stained with haematoxylin and eosin (H&E). Proposed here is work on improving colour normalisation methods, constructing an integrated model for nuclei segmentation and multiple objects overlap resolution, introducing and evaluating multi-level features for extracting relevant histopathological image and interpretable information, and developing classification algorithms for tasks such as cancer diagnosis, tumor identification, and tumor class labelling. Mini-Batch Stochastic Gradient Descent and Convolutional Neural Network which obtained statistical kappa value for breast cancer histopathology images shows a high degree of consistency in the classification task, with a range of 0.610.80 for benign and low grades and a range of 0.811.0 for medium and high rates. The Support Vector Machine (SVM), on the other hand, shows an almost perfect degree of consistency (0.811.0) across the several breast cancer picture classifications (benign, low, medium, and high).

Hierarchical Aggregation for Numerical Data under Local Differential Privacy.

The proposal of local differential privacy solves the problem that the data collector must be trusted in centralized differential privacy models. The statistical analysis of numerical data under local differential privacy has been widely studied by many scholars. However, in real-world scenarios, numerical data from the same category but in different ranges frequently require different levels of privacy protection. We propose a hierarchical aggregation framework for numerical data under local differential privacy. In this framework, the privacy data in different ranges are assigned different privacy levels and then disturbed hierarchically and locally. After receiving users' data, the aggregator perturbs the privacy data again to convert the low-level data into high-level data to increase the privacy data at each privacy level so as to improve the accuracy of the statistical analysis. Through theoretical analysis, it was proved that this framework meets the requirements of local differential privacy and that its final mean estimation result is unbiased. The proposed framework is combined with mini-batch stochastic gradient descent to complete the linear regression task. Sufficient experiments both on synthetic datasets and real datasets show that the framework has a higher accuracy than the existing methods in both mean estimation and mini-batch stochastic gradient descent experiments.

CG-DRR：digital reconstructed radiograph generation algorithm based on Cycle-GAN

CG-DRR：digital reconstructed radiograph generation algorithm based on Cycle-GAN

Applying a stochastic quasi-Newton optimizer to least-squares reverse time migration

As we keep moving towards large-scale 3D datasets, inversions based on the wave equation, such as least-squares reverse time migration (LSRTM), must rapidly converge to reduce the computational burden and deliver its well-known advantages, compensating the illumination and providing accurate amplitudes. For that, we explore stochastic optimization methods for LSRTM beyond the minibatch stochastic gradient descent (SGD), considered the state of the art in large-scale machine learning problems. We apply a second-order stochastic method to the LSRTM problem, that uses a quasi-Newton approach, known as the Sum of Functions Optimizer (SFO) algorithm. This method also works well with minibatches of data and has shown good performance on optimization of multi-layer neural networks. It can maintain computational tractability and limit memory requirements even for high dimensional optimization problems. As typical for quasi-Newton methods no adjustment of hyperparameters is required. On our experiments presented here, using synthetic data, we demonstrated that the SFO algorithm shows better results and faster convergence than the minibatch SGD.

Drill the Cork of Information Bottleneck by Inputting the Most Important Data.

Deep learning has become the most powerful machine learning tool in the last decade. However, how to efficiently train deep neural networks remains to be thoroughly solved. The widely used minibatch stochastic gradient descent (SGD) still needs to be accelerated. As a promising tool to better understand the learning dynamic of minibatch SGD, the information bottleneck (IB) theory claims that the optimization process consists of an initial fitting phase and the following compression phase. Based on this principle, we further study typicality sampling, an efficient data selection method, and propose a new explanation of how it helps accelerate the training process of the deep networks. We show that the fitting phase depicted in the IB theory will be boosted with a high signal-to-noise ratio of gradient approximation if the typicality sampling is appropriately adopted. Furthermore, this finding also implies that the prior information of the training set is critical to the optimization process, and the better use of the most important data can help the information flow through the bottleneck faster. Both theoretical analysis and experimental results on synthetic and real-world datasets demonstrate our conclusions.

A novel short-term load forecasting method based on mini-batch stochastic gradient descent regression model

Short-term Load Forecasting (STLF) is the basis of smart distribution network system operation, planning, and dispatching. The traditional linear regression prediction method has the problems of slow prediction speed and low prediction accuracy. In order to solve the problem, an improved regression model based on mini-batch stochastic gradient descent is proposed in this paper. Combined with the big data analysis and processing platform, the collected data is conformed, and the parallel computing model Map-Reduce is used to parallelize mini-batch stochastic gradient descent algorithm for improving the processing ability of mini-batch stochastic gradient descent algorithm in big data load forecasting, and shorten load forecasting time. Meanwhile, in order to clean up the duplicated data and bad data generated by the smart meter and sensor before calculation, an adaptive sorted neighborhood method is proposed to detect the repeatedly recorded data, and the K-means clustering method is used to eliminate the noise data .The experimental results show that the parallelized mini-batch stochastic gradient descent algorithm is much faster than the traditional regression analysis algorithm when the data volume is large. The average absolute percentage error of the load forecasting model for Belgium and a transformer station in Baiyin city of Gansu Province in China is 1.902% and 2.058% respectively, which satisfies the requirements of load forecasting.

A comprehensive study of calibration and uncertainty quantification for Bayesian convolutional neural networks — An application to seismic data

Deep neural networks offer numerous potential applications across geoscience; for example, one could argue that they are the state-of-the-art method for predicting faults in seismic data sets. In quantitative reservoir characterization workflows, it is common to incorporate the uncertainty of predictions. Thus, such subsurface models should provide calibrated probabilities and the associated uncertainties in their predictions. It has been shown that popular deep learning-based models often are miscalibrated, and due to their deterministic nature, they provide no means to interpret the uncertainty of their predictions. We compare three different approaches for obtaining probabilistic models based on convolutional neural networks in a Bayesian formalism, namely, deep ensembles, concrete dropout, and stochastic weight averaging Gaussian (SWAG). These methods are consistently applied to fault detection case studies in which deep ensembles use independently trained models to provide fault probabilities and concrete dropout represents an extension to the popular dropout technique to approximate Bayesian neural networks. Finally, we apply SWAG, a recent method that is based on the Bayesian inference equivalence of minibatch stochastic gradient descent. We provide quantitative results in terms of model calibration and uncertainty representation, as well as qualitative results on the synthetic and real seismic data sets. Our results indicate that the approximate Bayesian methods, concrete dropout and SWAG, provide well-calibrated predictions and uncertainty attributes at a lower computational cost compared to the baseline deep ensemble approach. The resulting uncertainties also offer a possibility to further improve the model performance, as well as enhancing the interpretability of the models.

A stochastic gradient descent approach with partitioned-truncated singular value decomposition for large-scale inverse problems of magnetic modulus data

We propose a stochastic gradient descent approach with partitioned-truncated singular value decomposition (SVD) for large-scale inverse problems of magnetic modulus data. Motivated by a uniqueness theorem in gravity inverse problem and realizing the similarity between gravity and magnetic inverse problems, we propose to solve the level-set function modeling the volume susceptibility distribution from the nonlinear magnetic modulus data. To deal with large-scale data, we employ a mini-batch stochastic gradient descent approach with random reshuffling when solving the optimization problem of the inverse problem. We propose a stepsize rule for the stochastic gradient descent according to the Courant–Friedrichs–Lewy condition of the evolution equation. In addition, we develop a partitioned-truncated SVD algorithm for the linear part of the inverse problem in the context of stochastic gradient descent. Numerical examples illustrate the efficacy of the proposed method, which turns out to have the capability of efficiently processing large-scale measurement data for the magnetic inverse problem. A possible generalization to the inverse problem of deep neural network is discussed at the end.

Improved training of deep convolutional networks via minimum-variance regularized adaptive sampling

Fostered by technological and theoretical developments, deep neural networks (DNNs) have achieved great success in many applications, but their training via mini-batch stochastic gradient descent (SGD) can be very costly due to the possibly tens of millions of parameters to be optimized and the large amounts of training examples that must be processed. The computational cost is exacerbated by the inefficiency of the uniform sampling typically used by SGD to form the training mini-batches: since not all training examples are equally relevant for training, sampling these under a uniform distribution is far from optimal, making the case for the study of improved methods to train DNNs. A better strategy is to sample the training instances under a distribution where the probability of being selected is proportional to the relevance of each individual instance; one way to achieve this is through importance sampling (IS), which minimizes the gradients’ variance w.r.t. the network parameters, consequently improving convergence. In this paper, an IS-based adaptive sampling method to improve the training of DNNs is introduced. This method exploits side information to construct the optimal sampling distribution and is dubbed regularized adaptive sampling (RAS). Experimental comparison using deep convolutional networks for classification of the MNIST and CIFAR-10 datasets shows that when compared against SGD and against another sampling method in the state of the art, RAS produces improvements in the speed and variance of the training process without incurring significant overhead or affecting the classification.

Multisource Smart Computer-Aided System for Mining COVID-19 Infection Data.

In this paper, we approach the problem of detecting and diagnosing COVID-19 infections using multisource scan images including CT and X-ray scans to assist the healthcare system during the COVID-19 pandemic. Here, a computer-aided diagnosis (CAD) system is proposed that utilizes analysis of the CT or X-ray to diagnose the impact of damage in the respiratory system per infected case. The CAD was utilized and optimized by hyper-parameters for shallow learning, e.g., SVM and deep learning. For the deep learning, mini-batch stochastic gradient descent was used to overcome fitting problems during transfer learning. The optimal parameter list values were found using the naïve Bayes technique. Our contributions are (i) a comparison among the detection rates of pre-trained CNN models, (ii) a suggested hybrid deep learning with shallow machine learning, (iii) an extensive analysis of the results of COVID-19 transition and informative conclusions through developing various transfer techniques, and (iv) a comparison of the accuracy of the previous models with the systems of the present study. The effectiveness of the proposed CAD is demonstrated using three datasets, either using an intense learning model as a fully end-to-end solution or using a hybrid deep learning model. Six experiments were designed to illustrate the superior performance of our suggested CAD when compared to other similar approaches. Our system achieves 99.94, 99.6, 100, 97.41, 99.23, and 98.94 accuracy for binary and three-class labels for the CT and two CXR datasets.

Classification of Benign and Malignant Lung Nodules Based on Deep Convolutional Network Feature Extraction.

With the rapid development of detection technology, CT imaging technology has been widely used in the early clinical diagnosis of lung nodules. However, accurate assessment of the nature of the nodule remains a challenging task due to the subjective nature of the radiologist. With the increasing amount of publicly available lung image data, it has become possible to use convolutional neural networks for benign and malignant classification of lung nodules. However, as the network depth increases, network training methods based on gradient descent usually lead to gradient dispersion. Therefore, we propose a novel deep convolutional network approach to classify the benignity and malignancy of lung nodules. Firstly, we segmented, extracted, and performed zero-phase component analysis whitening on images of lung nodules. Then, a multilayer perceptron was introduced into the structure to construct a deep convolutional network. Finally, the minibatch stochastic gradient descent method with a momentum coefficient is used to fine-tune the deep convolutional network to avoid the gradient dispersion. The 750 lung nodules in the lung image database are used for experimental verification. Classification accuracy of the proposed method can reach 96.0%. The experimental results show that the proposed method can provide an objective and efficient aid to solve the problem of classifying benign and malignant lung nodules in medical images.

A versatile and scalable single-cell data integration algorithm based on domain-adversarial and variational approximation

Single-cell technologies provide us new ways to profile transcriptomic landscape, chromatin accessibility, spatial expression patterns in heterogeneous tissues at the resolution of single cell. With enormous generated single-cell datasets, a key analytic challenge is to integrate these datasets to gain biological insights into cellular compositions. Here, we developed a domain-adversarial and variational approximation, DAVAE, which can integrate multiple single-cell datasets across samples, technologies and modalities with a single strategy. Besides, DAVAE can also integrate paired data of ATAC profile and transcriptome profile that are simultaneously measured from a same cell. With a mini-batch stochastic gradient descent strategy, it is scalable for large-scale data and can be accelerated by GPUs. Results on seven real data integration applications demonstrated the effectiveness and scalability of DAVAE in batch-effect removing, transfer learning and cell-type predictions for multiple single-cell datasets across samples, technologies and modalities. Availability: DAVAE has been implemented in a toolkit package "scbean" in the pypi repository, and the source code can be also freely accessible at https://github.com/jhu99/scbean. All our data and source code for reproducing the results of this paper can be accessible at https://github.com/jhu99/davae_paper.

Deep Learning for Gravitational-Wave Data Analysis: A Resampling White-Box Approach.

In this work, we apply Convolutional Neural Networks (CNNs) to detect gravitational wave (GW) signals of compact binary coalescences, using single-interferometer data from real LIGO detectors. Here, we adopted a resampling white-box approach to advance towards a statistical understanding of uncertainties intrinsic to CNNs in GW data analysis. We used Morlet wavelets to convert strain time series to time-frequency images. Moreover, we only worked with data of non-Gaussian noise and hardware injections, removing freedom to set signal-to-noise ratio (SNR) values in GW templates by hand, in order to reproduce more realistic experimental conditions. After hyperparameter adjustments, we found that resampling through repeated k-fold cross-validation smooths the stochasticity of mini-batch stochastic gradient descent present in accuracy perturbations by a factor of . CNNs are quite precise to detect noise, for H1 data and for L1 data; but, not sensitive enough to recall GW signals, for H1 data and for L1 data—although recall values are dependent on expected SNR. Our predictions are transparently understood by exploring tthe distribution of probabilistic scores outputted by the softmax layer, and they are strengthened by a receiving operating characteristic analysis and a paired-sample t-test to compare with a random classifier.

A Stochastic Gradient Descent Approach for Hybrid mmWave Beamforming With Blockage and CSI-Error Robustness

In this study, we consider the downlink beamforming problem in millimeter wave (mmWave) systems subjected to both path blockages and imperfect channel state information (CSI), and propose a new robust hybrid sum-outage minimizing design as a solution. We first formulate the problem as an empirical risk minimization (ERM) stochastic learning problem, whose solution can be obtained by the alternate iteration of a baseband digital and a radio frequency (RF) analog Riemann manifold-constrained beamforming updates through a mini-batch stochastic gradient descent (MSGD) approach, with gradient minimizing update rules given in closed-form, and learning rates optimized based on the lower-bounds of the corresponding Lipschitz constants. Unlike existing solutions to the path blockage-robust mmWave beamforming problem, wherein out-of-band side information is required or perfect CSI is assumed, our method relies only on the estimates and statistical knowledge of the channel's angles of departure (AoD) and complex gains, which are simultaneously captured in a Bernoulli-Gaussian model and used to generate the training data for the MSGD-based optimizer. Further, unlike preceding fully-digital or fully-connected hybrid contributions, the proposed scheme assumes a virtually-configured partially-connected setup; therefore, it is compatible with coordinated multipoint (CoMP) architectures, which are known to be crucial in terms of exploiting the full potential of mmWave systems. Simulation results confirm the effectiveness of our MSGD-based robust hybrid CoMP mmWave beamformer in mitigating the effects of path blockage and CSI error, demonstrating its superiority to state-of-the-art (SotA) alternatives.

Exploring Implicit and Explicit Geometrical Structure of Data for Deep Embedded Clustering

Clustering is an essential data analysis technique and has been studied extensively over the last decades. Previous studies have shown that data representation and data structure information are two critical factors for improving clustering performance, and it forms two important lines of research. The first line of research attempts to learn representative features, especially utilizing the deep neural networks, for handling clustering problems. The second concerns exploiting the geometric structure information within data for clustering. Although both of them have achieved promising performance in lots of clustering tasks, few efforts have been dedicated to combine them in a unified deep clustering framework, which is the research gap we aim to bridge in this work. In this paper, we propose a novel approach, Manifold regularized Deep Embedded Clustering (MDEC), to deal with the aforementioned challenge. It simultaneously models data generating distribution, cluster assignment consistency, as well as geometric structure of data in a unified framework. The proposed method can be optimized by performing mini-batch stochastic gradient descent and back-propagation. We evaluate MDEC on three real-world datasets (USPS, REUTERS-10K, and MNIST), where experimental results demonstrate that our model outperforms baseline models and obtains the state-of-the-art performance.