Optimal Subsampling Research Articles

Model constraints independent optimal subsampling probabilities for softmax regression

A prevailing method to alleviate the computational cost is to perform analysis on a subsample of the full data. Optimal subsampling algorithm utilizes non-uniform subsampling probabilities, derived through minimizing the asymptotic mean squared error of the subsample estimator, to acquire a higher estimation efficiency for a given subsample size. The optimal subsampling probabilities for softmax regression have been studied under the baseline constraint which treats one dimension of the multivariate response differently from other dimensions. In this paper, we show that different model constraints lead to different optimal subsampling probabilities, and the summation constraint corresponds to a better subsampling strategy than the baseline constraint in terms of balancing the responses among all categories. Furthermore, we derive the asymptotic distribution of the mean squared prediction error, and minimize its asymptotic expectation to define the optimal subsampling probabilities that are invariant to model constraints. Simulations and a real data example are provided to show the effectiveness of the proposed optimal subsampling probabilities.

Optimal subsampling for functional quantile regression

Subsampling is an efficient method to deal with massive data. In this paper, we investigate the optimal subsampling for linear quantile regression when the covariates are functions. The asymptotic distribution of the subsampling estimator is first derived. Then, we obtain the optimal subsampling probabilities based on the A-optimality criterion. Furthermore, the modified subsampling probabilities without estimating the densities of the response variables given the covariates are also proposed, which are easier to implement in practise. Numerical experiments on synthetic and real data show that the proposed methods always outperform the one with uniform sampling and can approximate the results based on full data well with less computational efforts.

Sampling With Replacement vs Poisson Sampling: A Comparative Study in Optimal Subsampling

Faced with massive data, subsampling is a commonly used technique to improve computational efficiency, and using nonuniform subsampling probabilities is an effective approach to improve estimation efficiency. For computational efficiency, subsampling is often implemented with replacement or through Poisson subsampling. However, no rigorous investigation has been performed to study the difference between the two subsampling procedures such as their estimation efficiency and computational convenience. This paper performs a comparative study on these two different sampling procedures. In the context of maximizing a general target function, we first derive asymptotic distributions for estimators obtained from the two sampling procedures. The results show that the Poisson subsampling may have a higher estimation efficiency. Based on the asymptotic distributions for both subsampling with replacement and Poisson subsampling, we derive optimal subsampling probabilities that minimize the variance functions of the subsampling estimators. These subsampling probabilities further reveal the similarities and differences between subsampling with replacement and Poisson subsampling. The theoretical characterizations and comparisons on the two subsampling procedures provide guidance to select a more appropriate subsampling approach in practice. Furthermore, practically implementable algorithms are proposed based on the optimal structural results, which are evaluated through both theoretical and empirical analyses.

Optimal subsampling for parametric accelerated failure time models with massive survival data.

With increasing availability of massive survival data, researchers need valid statistical inferences for survival modeling whose computation is not limited by computer memories. Existing works focus on relative risk models using the online updating and divide-and-conquer strategies. The subsampling strategy has not been available due to challenges in developing the asymptotic properties of the estimator under semiparametric models with censored data. This article tackles optimal subsampling algorithms to fast approximate the maximum likelihood estimator for parametric accelerate failure time models with massive survival data. We derive the asymptotic distributions of the subsampling estimator and the optimal sampling probabilities that minimize the asymptotic mean squared error of the estimator. A feasible two-step algorithm is proposed where the optimal sampling probabilities in the second step are estimated based on a pilot sample in the first step. The asymptotic properties of the two-step estimator are established. The performance of the estimator is validated in a simulation study. A real data analysis illustrates the usefulness of the methods.

Optimal Dependence of Performance and Efficiency of Collaborative Filtering on Random Stratified Subsampling

Dropping fractions of users or items judiciously can reduce the computational cost of Collaborative Filtering (CF) algorithms. The effect of this subsampling on the computing time and accuracy of CF is not fully understood, and clear guidelines for selecting optimal or even appropriate subsampling levels are not available. In this paper, we present a Density-based Random Stratified Subsampling using Clustering (DRSC) algorithm in which the desired Fraction of Users Dropped (FUD) and Fraction of Items Dropped (FID) are specified, and the overall density during subsampling is maintained. Subsequently, we develop simple models of the Training Time Improvement (TTI) and the Accuracy Loss (AL) as functions of FUD and FID, based on extensive simulations of seven standard CF algorithms as applied to various primary matrices from MovieLens, Yahoo Music Rating, and Amazon Automotive data. Simulations show that both TTI and a scaled AL are bi-linear in FID and FUD for all seven methods. The TTI linear regression of a CF method appears to be same for all datasets. Extensive simulations illustrate that TTI can be estimated reliably with FUD and FID only, but AL requires considering additional dataset characteristics. The derived models are then used to optimize the levels of subsampling addressing the tradeoff between TTI and AL. A simple sub-optimal approximation was found, in which the optimal AL is proportional to the optimal Training Time Reduction Factor (TTRF) for higher values of TTRF, and the optimal subsampling levels, like optimal FID/(1–FID), are proportional to the square root of TTRF.

Sampling-based Gaussian Mixture Regression for Big Data

This paper proposes a nonuniform subsampling method for finite mixtures of regression models to reduce large data computational tasks. A general estimator based on a subsample is investigated, and its asymptotic normality is established. We assign optimal subsampling probabilities to data points that minimize the asymptotic mean squared errors of the general estimator and linearly transformed estimators. Since the proposed probabilities depend on unknown parameters, an implementable algorithm is developed. We first approximate the optimal subsampling probabilities using a pilot sample. After that, we select a subsample using the approximated subsampling probabilities and compute estimates using the subsample. We evaluate the proposed method in a simulation study and present a real data example using appliance energy data.

Optimal subsampling for least absolute relative error estimators with massive data

Due to the limitation of computational resources, traditional statistical methods are no longer applicable to large data sets. Subsampling is a popular method which can significantly reduce computational burden. This paper considers a subsampling strategy based on the least absolute relative error in the multiplicative model for massive data. In addition, we employ the random weighting and the least squares methods to handle the problem that the asymptotic covariance of the estimator is difficult to be estimated directly. Moreover, the comparison among the least absolute relative error, least absolute deviation and least squares under the optimal subsampling strategy are given in simulation studies and real examples.

Adaptive response-dependent two-phase designs: Some results on robustness and efficiency.

Large cohort studies now routinely involve biobanks in which biospecimens are stored for use in future biomarker studies. In such settings, two-phase response-dependent sampling designs involve subsampling individuals in the cohort, assaying their biospecimen to measure an expensive biomarker, and using this data to estimate key parameters of interest under budgetary constraints. When analyses are based on inverse probability weighted estimating functions, recent work has described adaptive two-phase designs in which a preliminary phase of subsampling based on a standard design facilitates approximation of an optimal selection model for a second subsampling phase. In this article, we refine the definition of an optimal subsampling scheme within the framework of adaptive two-phase designs, describe how adaptive two-phase designs can be used when analyses are based on likelihood or conditional likelihood, and consider the setting of a continuous biomarker where the nuisance covariate distribution is estimated nonparametrically at the design stage and analysis stage as required; efficiency and robustness issues are investigated. We also explore these methods for the surrogate variable problem and describe a generalization to accommodate multiple stages of phase II subsampling. A study involving individuals with psoriatic arthritis is considered for illustration, where the aim is to assess the association between the biomarker MMP-3 and the development of joint damage.

A two-stage optimal subsampling estimation for missing data problems with large-scale data

Subsampling is useful to downsize data volumes and speed up calculations for large-scale data and is well studied with completely observed data. In the presence of missing data, computation is more challenging and subsampling becomes more crucial and complex. However, there is still a lack of study on subsampling for missing data problems. This paper fills the gap by studying the subsampling method for a widely used missing data estimator, the augmented inverse probability weighting (AIPW) estimator. The response mean estimation problem with missing responses is discussed for illustration. A two-stage subsampling method is proposed via Poisson sampling framework. A small subsample of expected size n1 is used in the first stage to estimate the parameters in the propensity score and the outcome regression models, while a larger subsample of expected size n2 is used in the computationally simple second stage to calculate the final estimator. An attractive property of the resulting estimator is that its convergence rate is n2−1/2 rather than n1−1/2 when both the propensity score and the outcome regression functions are correctly specified. The rate n2−1/2 is still attainable for some important cases if only one of the two functions is correctly specified. This indicates that using a small subsample in the computationally complex first stage can reduce the computational burden with little impact on the statistical accuracy. Asymptotic normality of the resulting estimator is established and the optimal subsampling probability is derived by minimizing the asymptotic variance of the resulting estimator. Simulations and a real data analysis were conducted to demonstrate the empirical performance of the resulting estimator.

Optimal subsampling for multiplicative regression with massive data

Faced with massive data, subsampling is a popular way to downsize the data volume for reducing computational burden. The key idea of subsampling is to perform statistical analysis on a representative subsample drawn from the full data. It provides a practical solution to extracting useful information from big data. In this article, we develop an efficient subsampling method for large‐scale multiplicative regression model, which can largely reduce the computational burden due to massive data. Under some regularity conditions, we establish consistency and asymptotic normality of the subsample‐based estimator, and derive the optimal subsampling probabilities according to the L‐optimality criterion. A two‐step algorithm is developed to approximate the optimal subsampling procedure. Meanwhile, the convergence rate and asymptotic normality of the two‐step subsample estimator are established. Numerical studies and two real data applications are carried out to evaluate the performance of our subsampling method.

Optimal subsampling for large‐sample quantile regression with massive data

AbstractTo balance the explosive growth of data volume and limited budgets for computational resources, one of the popular methods is downscaling the data volume by subsampling a subdataset that inherits the relevant property of the full data. As an alternative to the mean regression model, the quantile regression model has been studied extensively when the data are independent and the data scale is medium. This article focuses on quantile regression with massive data where the sample size n (greater than in general) is extraordinarily large but the dimension d (smaller than 20 in general) is small. We first formulate the general subsampling procedure and establish the asymptotic property of the resultant estimator. Then, with the help of optimality criteria in experimental design, we derive two subsampling probabilities that are optimal in the sense of smallest asymptotic mean square error. Since the optimal subsampling probabilities depend on the full data estimator, we develop a two‐step optimal subsampling algorithm and study the consistency and asymptotic normality of the resultant estimator. The empirical performance of the optimal subsampling algorithm is evaluated with synthetic and real datasets.

SAT: a Surrogate-Assisted Two-wave case boosting sampling method, with application to EHR-based association studies.

Electronic health records (EHRs) enable investigation of the association between phenotypes and risk factors. However, studies solely relying on potentially error-prone EHR-derived phenotypes (ie, surrogates) are subject to bias. Analyses of low prevalence phenotypes may also suffer from poor efficiency. Existing methods typically focus on one of these issues but seldom address both. This study aims to simultaneously address both issues by developing new sampling methods to select an optimal subsample to collect gold standard phenotypes for improving the accuracy of association estimation. We develop a surrogate-assisted two-wave (SAT) sampling method, where a surrogate-guided sampling (SGS) procedure and a modified optimal subsampling procedure motivated from A-optimality criterion (OSMAC) are employed sequentially, to select a subsample for outcome validation through manual chart review subject to budget constraints. A model is then fitted based on the subsample with the true phenotypes. Simulation studies and an application to an EHR dataset of breast cancer survivors are conducted to demonstrate the effectiveness of SAT. We found that the subsample selected with the proposed method contains informative observations that effectively reduce the mean squared error of the resultant estimator of the association. The proposed approach can handle the problem brought by the rarity of cases and misclassification of the surrogate in phenotype-absent EHR-based association studies. With a well-behaved surrogate, SAT successfully boosts the case prevalence in the subsample and improves the efficiency of estimation.

Optimal subsampling for composite quantile regression model in massive data

Optimal subsampling for composite quantile regression model in massive data

Locally Optimal Subsampling Strategies for Full Matrix Capture Measurements in Pipe Inspection

In ultrasonic non-destructive testing, array and matrix transducers are being employed for applications that require in-field steerability or which benefit from a higher number of insonification angles. Having many transmit channels, on the other hand, increases the measurement time and renders the use of array transducers unfeasible for many applications. In the literature, methods for reducing the number of required channels compared to the full matrix capture scheme have been proposed. Conventionally, these are based on choosing the aperture that is as wide as possible. In this publication, we investigate a scenario from the field of pipe inspection, where cracks have to be detected in specific areas near the weld. Consequently, the width of the aperture has to be chosen according to the region of interest at hand. On the basis of ray-tracing simulations which incorporate a model of the transducer directivity and beam spread at the interface, we derive application specific measures of the energy distribution over the array configuration for given regions of interest. These are used to determine feasible subsampling schemes. For the given scenario, the validity/quality of the derived subsampling schemes are compared on the basis of reconstructions using the conventional total focusing method as well as sparsity driven-reconstructions using the Fast Iterative Shrinkage-Thresholding Algorithm. The results can be used to effectively improve the measurement time for the given application without notable loss in defect detectability.

Optimal subsampling for linear quantile regression models

AbstractSubsampling techniques are efficient methods for handling big data. Quite a few optimal sampling methods have been developed for parametric models in which the loss functions are differentiable with respect to parameters. However, they do not apply to quantile regression (QR) models as the involved check function is not differentiable. To circumvent the non‐differentiability problem, we consider directly estimating the linear QR coefficient by minimizing the Hansen–Hurwitz estimator of the usual loss function for QR. We establish the asymptotic normality of the resulting estimator under a generic sampling method, and then develop optimal subsampling methods for linear QR. In particular, we propose a one‐stage subsampling method, which depends only on the lengths of covariates, and a two‐stage subsampling method, which is a combination of the one‐stage sampling and the ideal optimal subsampling methods. Our simulation and real data based simulation studies show that the two recommended sampling methods always outperform simple random sampling in terms of mean square error, whether the linear QR model is valid or not.

Fast Optimal Subsampling Probability Approximation for Generalized Linear Models

For massive data, subsampling techniques are popular to mitigate computational burden by reducing the data size. In a subsampling approach, subsampling probabilities for each data point are specified to obtain an informative sub-data, and then estimates based on the sub-data are obtained to approximate estimates from the full data. Assigning subsampling probabilities based on minimization of the asymptotic mean squared error of the estimator from a general subsample (A-optimality criterion) is a popular approach, however, it is still computationally demanding to calculate the probabilities under this setting. To efficiently approximate the A-optimal subsampling probabilities for generalized linear models, randomized algorithms are proposed. To develop the algorithms, the Johnson-Lindenstrauss Transform and Subsampled Randomized Hadamard Transform are used. Additionally, optimal subsampling probabilities are derived for the Gaussian linear model in the case where both the regression coefficients and dispersion parameter are of interest, and algorithms are developed to approximate the optimal subsampling probabilities. Simulation studies indicate that the estimators based on the developed algorithms have excellent performance for statistical inference and have substantial savings in computing time compared to the direct calculation of the A-optimal subsampling probabilities.

Optimal subsample selection for massive logistic regression with distributed data

With the emergence of big data, it is increasingly common that the data are distributed. i.e., the data are stored at many distributed sites (machines or nodes) owing to data collection or business operations, etc. We propose a distributed subsampling procedure in such a setting to efficiently approximate the maximum likelihood estimator for the logistic regression. We establish the consistency and asymptotic normality of the subsample estimator given the full data. The optimal subsampling probabilities and optimal allocation sizes are explicitly obtained. We develop a two-step algorithm to approximate the optimal subsampling procedure. Numerical simulations and an application to airline data are presented to evaluate the performance of our subsampling method.

A Review on Optimal Subsampling Methods for Massive Datasets

A Review on Optimal Subsampling Methods for Massive Datasets

Sampling-based estimation for massive survival data with additive hazards model.

For massive survival data, we propose a subsampling algorithm to efficiently approximate the estimates of regression parameters in the additive hazards model. We establish consistency and asymptotic normality of the subsample-based estimator given the full data. The optimal subsampling probabilities are obtained via minimizing asymptotic variance of the resulting estimator. The subsample-based procedure can largely reduce the computational cost compared with the full data method. In numerical simulations, our method has low bias and satisfactory coverage probabilities. We provide an illustrative example on the survival analysis of patients with lymphoma cancer from the Surveillance, Epidemiology, and End Results Program.

Investigating statistical bias correction with temporal subsample of the upper Ping River Basin, Thailand

Abstract This study aims to investigate different statistical bias correction techniques to improve the output of a regional climate model (RCM) of daily rainfall for the upper Ping River Basin in Northern Thailand. Three subsamples are used for each bias correction method, which are (1) using full calibrated 30-year-period data, (2) seasonal subsampling, and (3) monthly subsampling. The bias correction techniques are classified into three groups, which are (1) distribution-derived transformation, (2) parametric transformation, and (3) nonparametric transformation. Eleven bias correction techniques with three different subsamples are used to derive transfer function parameters to adjust model bias error. Generally, appropriate bias correction methods with optimal subsampling are locally dependent and need to be defined specifically for a study area. The study results show that monthly subsampling would be well established by capturing the monthly mean variation after correcting the model's daily rainfall. The results also give the best-fitted parameter set of the different subsamples. However, applying the full calibrated data and the seasonal subsamples cannot substantially improve internal variability. Thus, the effect of internal climate variability of the study region is greater than the choice of bias correction methods. Of the bias correction approaches, nonparametric transformation performed best in correcting daily rainfall bias error in this study area as evaluated by statistics and frequency distributions. Therefore, using a combination of methods between the nonparametric transformation and monthly subsampling offered the best accuracy and robustness. However, the nonparametric transformation was quite sensitive to the calibration time period.