Sample Size Regime Research Articles

An Approximate MSE Expression for Maximum Likelihood and Other Implicitly Defined Estimators of Non-Random Parameters

An approximate mean square error (MSE) expression for the performance analysis of implicitly defined estimators of non-random parameters is proposed. An implicitly defined estimator (IDE) declares the minimizer/maximizer of a selected cost/reward function as the parameter estimate. The maximum likelihood (ML) and the least squares estimators are among the well known examples of this class. In this paper, an exact MSE expression for implicitly defined estimators with a symmetric and unimodal objective function is given. It is shown that the expression reduces to the Cramer-Rao lower bound (CRLB) and misspecified CRLB in the large sample size regime for ML and misspecified ML estimation, respectively. The expression is shown to yield the Ziv-Zakai bound (without the valley filling function) for the maximum a posteriori (MAP) estimator when it is used in a Bayesian setting, that is, when an a-priori distribution is assigned to the unknown parameter. In addition, extension of the suggested expression to the case of nuisance parameters is studied and some approximations are given to ease the computations for this case. Numerical results indicate that the suggested MSE expression not only predicts the estimator performance in the asymptotic region; but it is also applicable for the threshold region analysis, even for IDEs whose objective functions do not satisfy the symmetry and unimodality assumptions. Advantages of the suggested MSE expression are its conceptual simplicity and its relatively straightforward numerical calculation due to the reduction of the estimation problem to a binary hypothesis testing problem, similar to the usage of Ziv-Zakai bounds in random parameter estimation problems.

Some clustering-based exact distribution-free [formula omitted]-sample tests applicable to high dimension, low sample size data

Testing homogeneity of k(≥2) multivariate distributions is a challenging problem in statistics, especially when the dimension of the data is much larger than the sample size. Most of the existing tests often perform poorly in this high dimension, low sample size (HDLSS) regime, and many of them cannot be used at all. In this article, we propose some nonparametric tests for this purpose. These tests have the distribution-free property in finite sample situations. They are based on a high dimensional clustering algorithm that makes a partition of the data to form a contingency table. Using the cell frequencies of that table, we construct the test statistics. We can develop tests based on a k-partition of the data or estimate the number of partitions from the data and construct tests based on it. Under appropriate regularity conditions, we prove the consistency of these tests in the HDLSS asymptotic regime. We also consider a multiscale approach, where the results for different number of partitions are aggregated judiciously. Extensive simulation study and analysis of some benchmark datasets illustrate the superiority of the proposed tests over some existing methods.

On the Optimal Tradeoff Between Computational Efficiency and Generalizability of Oja’s Algorithm

The Oja's algorithm is widely applied for computing principal eigenvectors in real problems, and it is practically useful to understand the theoretical relationships between the learning rate, convergence rate, and generalization error of this algorithm for noisy samples. In this paper, we show that under mild assumptions of sampling noise, both the generalization error and the convergence rate reveal linear relationships with the learning rate in the large sample size and small learning rate regime. In addition, when the algorithm nearly converges, we provide a refined characterization of the generalization error, which suggests the optimal design for the learning rate for data with noise. Moreover, we investigate the minibatch variation of Oja's algorithm and demonstrate that the learning rate of minibatch training is decayed by a factor characterized by the batch size, which provides theoretical insights and guidance for designing the learning rate in minibatch training algorithms. Finally, our theoretical results are validated by experiments on both synthesized data and the MNIST dataset.

Sparse equisigned PCA: Algorithms and performance bounds in the noisy rank-1 setting

Singular value decomposition (SVD) based principal component analysis (PCA) breaks down in the high-dimensional and limited sample size regime below a certain critical eigen-SNR that depends on the dimensionality of the system and the number of samples. Below this critical eigen-SNR, the estimates returned by the SVD are asymptotically uncorrelated with the latent principal components. We consider a setting where the left singular vector of the underlying rank one signal matrix is assumed to be sparse and the right singular vector is assumed to be equisigned, that is, having either only nonnegative or only nonpositive entries. We consider six different algorithms for estimating the sparse principal component based on different statistical criteria and prove that by exploiting sparsity, we recover consistent estimates in the low eigen-SNR regime where the SVD fails. Our analysis reveals conditions under which a coordinate selection scheme based on a sum-type decision statistic outperforms schemes that utilize the $\ell _{1}$ and $\ell _{2}$ norm-based statistics. We derive lower bounds on the size of detectable coordinates of the principal left singular vector and utilize these lower bounds to derive lower bounds on the worst-case risk. Finally, we verify our findings with numerical simulations and a illustrate the performance with a video data where the interest is in identifying objects.

DUAL-GLOW: Conditional Flow-Based Generative Model for Modality Transfer.

Positron emission tomography (PET) imaging is an imaging modality for diagnosing a number of neurological diseases. In contrast to Magnetic Resonance Imaging (MRI), PET is costly and involves injecting a radioactive substance into the patient. Motivated by developments in modality transfer in vision, we study the generation of certain types of PET images from MRI data. We derive new flow-based generative models which we show perform well in this small sample size regime (much smaller than dataset sizes available in standard vision tasks). Our formulation, DUAL-GLOW, is based on two invertible networks and a relation network that maps the latent spaces to each other. We discuss how given the prior distribution, learning the conditional distribution of PET given the MRI image reduces to obtaining the conditional distribution between the two latent codes w.r.t. the two image types. We also extend our framework to leverage "side" information (or attributes) when available. By controlling the PET generation through "conditioning" on age, our model is also able to capture brain FDG-PET (hypometabolism) changes, as a function of age. We present experiments on the Alzheimers Disease Neuroimaging Initiative (ADNI) dataset with 826 subjects, and obtain good performance in PET image synthesis, qualitatively and quantitatively better than recent works.

Heteroscedastic Gaussian processes for uncertainty modeling in large-scale crowdsourced traffic data

Accurately modeling traffic speeds is a fundamental part of efficient intelligent transportation systems. Nowadays, with the widespread deployment of GPS-enabled devices, it has become possible to crowdsource the collection of speed information to road users (e.g. through mobile applications or dedicated in-vehicle devices). Despite its rather wide spatial coverage, crowdsourced speed data also brings very important challenges, such as the highly variable measurement noise in the data due to a variety of driving behaviors and sample sizes. When not properly accounted for, this noise can severely compromise any application that relies on accurate traffic data. In this article, we propose the use of heteroscedastic Gaussian processes (HGP) to model the time-varying uncertainty in large-scale crowdsourced traffic data. Furthermore, we develop a HGP conditioned on sample size and traffic regime (SSRC-HGP), which makes use of sample size information (probe vehicles per minute) as well as previous observed speeds, in order to more accurately model the uncertainty in observed speeds. Using 6 months of crowdsourced traffic data from Copenhagen, we empirically show that the proposed heteroscedastic models produce significantly better predictive distributions when compared to current state-of-the-art methods for both speed imputation and short-term forecasting tasks.

Statistical and computational trade-offs in estimation of sparse principal components

In recent years, sparse principal component analysis has emerged as an extremely popular dimension reduction technique for high-dimensional data. The theoretical challenge, in the simplest case, is to estimate the leading eigenvector of a population covariance matrix under the assumption that this eigenvector is sparse. An impressive range of estimators have been proposed; some of these are fast to compute, while others are known to achieve the minimax optimal rate over certain Gaussian or sub-Gaussian classes. In this paper, we show that, under a widely-believed assumption from computational complexity theory, there is a fundamental trade-off between statistical and computational performance in this problem. More precisely, working with new, larger classes satisfying a restricted covariance concentration condition, we show that there is an effective sample size regime in which no randomised polynomial time algorithm can achieve the minimax optimal rate. We also study the theoretical performance of a (polynomial time) variant of the well-known semidefinite relaxation estimator, revealing a subtle interplay between statistical and computational efficiency.

Learning by Transferring from Unsupervised Universal Sources

Category classifiers trained from a large corpus of annotated data are widely accepted as the sources for (hypothesis) transfer learning. Sources generated in this way are tied to a particular set of categories, limiting their transferability across a wide spectrum of target categories. In this paper, we address this largely-overlooked yet fundamental source problem by both introducing a systematic scheme for generating universal source hypotheses and proposing a principled, scalable approach to automatically tuning the transfer process. Our approach is based on the insights that expressive source hypotheses could be generated without any supervision and that a sparse combination of such hypotheses facilitates recognition of novel categories from few samples. We demonstrate improvements over the state-of-the-art on object and scene classification in the small sample size regime.

Asymptotically Optimal Linear Shrinkage of Sample LMMSE and MVDR Filters

Conventional implementations of the linear minimum mean-square (LMMSE) and minimum variance distortionless response (MVDR) estimators rely on the sample matrix inversion (SMI) technique, i.e., on the sample covariance matrix (SCM). This approach is optimal in the large sample size regime. Nonetheless, in small sample size situations, those sample estimators suffer a large performance degradation. Thus, the aim of this paper is to propose corrections of these sample methods that counteract their performance degradation in the small sample size regime and keep their optimality in large sample size situations. To this aim, a twofold approach is proposed. First, shrinkage estimators are considered, as they are known to be robust to the small sample size regime. Namely, the proposed methods are based on shrinking the sample LMMSE or sample MVDR filters towards a variously called matched filter or conventional (Bartlett) beamformer in array processing. Second, random matrix theory is used to obtain the optimal shrinkage factors for large filters. The simulation results highlight that the proposed methods outperform the sample LMMSE and MVDR. Also, provided that the sample size is higher than the observation dimension, they improve classical diagonal loading (DL) and Ledoit-Wolf (LW) techniques, which counteract the small sample size degradation by regularizing the SCM. Finally, compared to state-of-the-art DL, the proposed methods reduce the computational cost and the proposed shrinkage of the LMMSE obtains performance gains.

Finite-Sample Linear Filter Optimization in Wireless Communications and Financial Systems

We study the problem of linear filter optimization with finite sample size, which has wide applications such as beamformer design in wireless communications and portfolio optimization in finance. Traditional methods in both fields are not robust against the imprecise channel vector and the noise covariance matrix (or the mean return and the covariance of assets in finance) due to finite sample size. We consider estimation errors both in the channel vector and the noise covariance matrix (or the mean return and the covariance) simultaneously. We resort to high-dimensional asymptotics to account for the fact that the observation dimension is of the same order of magnitude as the number of samples, and use the diagonal loading method (or the shrinkage estimator) to improve the robustness. The channel vector (or mean return) and the noise covariance matrix are estimated from the training data, and then corrected under several widely-used criteria. In an asymptotic setting where the number of samples is comparable to the observation dimension, we obtain linear filters that are as good as the optimal filters with a shrinkage structure and a perfect channel vector (or mean return) under different criteria. Monte Carlo simulations show the advantage of our linear filters in the finite sample size regime.

Multi-task learning for pKa prediction

Many compound properties depend directly on the dissociation constants of its acidic and basic groups. Significant effort has been invested in computational models to predict these constants. For linear regression models, compounds are often divided into chemically motivated classes, with a separate model for each class. However, sometimes too few measurements are available for a class to build a reasonable model, e.g., when investigating a new compound series. If data for related classes are available, we show that multi-task learning can be used to improve predictions by utilizing data from these other classes. We investigate performance of linear Gaussian process regression models (single task, pooling, and multi-task models) in the low sample size regime, using a published data set (n = 698, mostly monoprotic, in aqueous solution) divided beforehand into 15 classes. A multi-task regression model using the intrinsic model of co-regionalization and incomplete Cholesky decomposition performed best in 85% of all experiments. The presented approach can be applied to estimate other molecular properties where few measurements are available.

On the Asymptotic Behavior of the Sample Estimates of Eigenvalues and Eigenvectors of Covariance Matrices

This paper analyzes the asymptotic behavior of the sample estimators of the eigenvalues and eigenvectors of covariance matrices. Rather than considering traditional large sample-size asymptotics, in this paper the focus is on limited sample size situations, whereby the number of available observations is comparable in magnitude to the observation dimension. Using tools from random matrix theory, the asymptotic behavior of the traditional sample estimates is investigated under the assumption that both the sample size and the observation dimension tend to infinity, while their quotient converges to a positive quantity. Assuming that an asymptotic eigenvalue splitting condition is fulfilled, closed form asymptotic expressions of these estimators are derived, proving inconsistency of the traditional sample estimators in these asymptotic conditions. The derived expressions are shown to provide a valuable insight into the behavior of the sample estimators in the small sample size regime.

Design and construction of a microcoil NMR probe for the routine analysis of 20‐μL samples

AbstractRecent advances in microcoil NMR have provided commercially available, robust methodologies for analyzing mass and volume limited samples in the low microliter regime, and the technology has been applied in a number of areas. Unfortunately, due to constraints on sample size and the limited solubility of some compounds of interest, the application of this approach to certain areas of development, such as the structural analysis of chromatography eluates, is restricted. A current challenge is to provide an option within a previously unexplored sample size regime (tens of microliters) while still taking advantage of the increase in mass sensitivity afforded by solenoidal microcoil NMR. In this article, we present the design and construction of a microcoil NMR probe with a custom detection cell for the routine analysis of 20‐μL samples. The detection cell is comprised of a CO2‐laser‐heated HF‐etched borosilicate active volume with fused silica transfer lines added to provide sample input and output. This setup produces an enlarged sample bubble within the detection coil and provides easy connection with 1/16 in. standard LC connections, lending itself to applications with HPLC‐NMR, online SPE and similar separation techniques, as well as higher‐throughput robotic automation. NMR performance characteristics determined using standard compounds showed the probe exhibited reasonable resolution (<0.01 ppm), although sensitivity was less than optimal due to tuning constraints. Future improvements and opportunities are also discussed. © 2008 Wiley Periodicals, Inc. Concepts Magn Reson Part B (Magn Reson Engineering) 33B: 1–8, 2008

Anomalous exciton–radiation coupling in the nano-to-bulk crossover regime

Anomalous exciton–radiation coupling in the sample-size regime beyond the long-wavelengthapproximation (LWA) is discussed. In high-quality samples fabricated using thelatest technologies, the long-range excitonic coherence and non-LWA effect cause apeculiar interplay between the spatial structures of the radiation and excitonicwaves, which emerge as an anomalous level structure of the exciton–radiationcoupled system including an extremely large radiative correction. We theoreticallydemonstrated such a situation for a thin film by a microscopic nonlocal theoryin which a size-resonant enhancement of the radiative damping constant andsuccessive interchanges of the confined polariton states can be observed. In order toverify the predicted level structure of confined polaritons, we present the results ofnon-degenerate two-photon scattering spectroscopy investigations and a correspondinganalysis based on the nonlocal theory for the second-order nonlinear process. Thisscheme has successfully elucidated peculiar level structures of confined polaritonsin the non-LWA regime. In the last half of the paper, a new type of resonanthyper-parametric scattering (RHPS) spectroscopy with a non-degenerate inducedexcitation is theoretically proposed as another method to study confined polaritons in thenon-LWA regime. Some calculations for thin films are demonstrated to reveal thepotentiality of this method that provides rich information on confined biexcitons andpolaritons.