sub-Gaussian Distribution Research Articles

Tensor Regression Using Low-Rank and Sparse Tucker Decompositions

This paper studies a tensor-structured linear regression model with a scalar response variable and tensor-structured predictors, such that the regression parameters form a tensor of order $d$ (i.e., a $d$-fold multiway array) in $\mathbb{R}^{n_1 \times n_2 \times \cdots \times n_d}$. It focuses on the task of estimating the regression tensor from $m$ realizations of the response variable and the predictors where $m\ll n = \prod \nolimits_{i} n_i$. Despite the seeming ill-posedness of this problem, it can still be solved if the parameter tensor belongs to the space of sparse, low Tucker-rank tensors. Accordingly, the estimation procedure is posed as a non-convex optimization program over the space of sparse, low Tucker-rank tensors, and a tensor variant of projected gradient descent is proposed to solve the resulting non-convex problem. In addition, mathematical guarantees are provided that establish the proposed method linearly converges to an appropriate solution under a certain set of conditions. Further, an upper bound on sample complexity of tensor parameter estimation for the model under consideration is characterized for the special case when the individual (scalar) predictors independently draw values from a sub-Gaussian distribution. The sample complexity bound is shown to have a polylogarithmic dependence on $\bar{n} = \max \big\{n_i: i\in \{1,2,\ldots,d \} \big\}$ and, orderwise, it matches the bound one can obtain from a heuristic parameter counting argument. Finally, numerical experiments demonstrate the efficacy of the proposed tensor model and estimation method on a synthetic dataset and a collection of neuroimaging datasets pertaining to attention deficit hyperactivity disorder. Specifically, the proposed method exhibits better sample complexities on both synthetic and real datasets, demonstrating the usefulness of the model and the method in settings where $n \gg m$.

Independent Low-Rank Matrix Analysis Based on Time-Variant Sub-Gaussian Source Model for Determined Blind Source Separation

Independent low-rank matrix analysis (ILRMA) is a fast and stable method of blind audio source separation. Conventional ILRMAs assume time-variant (super-)Gaussian source models, which can only represent signals that follow a super-Gaussian distribution. In this article, we focus on ILRMA based on a generalized Gaussian distribution (GGD-ILRMA) and propose a new type of GGD-ILRMA that adopts a time-variant sub-Gaussian distribution for the source model. We propose a new update scheme called generalized iterative projection for homogeneous source models (GIP-HSM) and obtain a convergence-guaranteed update rule for demixing spatial parameters by combining the GIP-HSM scheme and the majorization-minimization (MM) algorithm. Furthermore, a new extension of the MM algorithm is proposed for the convergence acceleration by applying the majorization-equalization algorithm to a multivariate case. In the experimental evaluation, we show the versatility of the proposed method, i.e., the proposed time-variant sub-Gaussian source model can be applied to various types of source signal.

Analysis of Error Dependencies on Newhope

Among many submissions to the NIST post-quantum cryptography (PQC) project, NewHope is a promising key encapsulation mechanism (KEM) based on the Ring-Learning with errors (Ring-LWE) problem. Since NewHope is an indistinguishability (IND)-chosen ciphertext attack secure KEM by applying the Fujisaki-Okamoto transform to an IND-chosen plaintext attack secure public key encryption, accurate calculation of decryption failure rate (DFR) is required to guarantee resilience against attacks that exploit decryption failures. However, the current upper bound of DFR on NewHope is rather loose because the compression noise, the effect of encoding/decoding of NewHope, and the approximation effect of centered binomial distribution are not fully considered. Furthermore, since NewHope is a Ring-LWE based cryptosystem, there is a problem of error dependency among error coefficients, which makes accurate DFR calculation difficult. In this paper, we derive much tighter upper bound on DFR than the current upper bound using constraint relaxation and union bound. Especially, the above-mentioned factors are all considered in derivation of new upper bound and the centered binomial distribution is not approximated to subgaussian distribution. In addition, since the error dependency is considered, the new upper bound is much closer to the real DFR than the previous upper bound. Furthermore, the new upper bound is parameterized by using Chernoff-Cramer bound in order to facilitate calculation of new upper bound for the parameters of NewHope. Since the new upper bound is much lower than the DFR requirement of PQC, this DFR margin is used to improve the security and bandwidth efficiency of NewHope. As a result, the security level of NewHope is improved by 7.2 % or bandwidth efficiency is improved by 5.9 %.

Solving Complex Quadratic Systems With Full-Rank Random Matrices

We tackle the problem of recovering a complex signal $\boldsymbol x\in\mathbb{C}^n$ from quadratic measurements of the form $y_i=\boldsymbol x^*\boldsymbol A_i\boldsymbol x$, where $\boldsymbol A_i$ is a full-rank, complex random measurement matrix whose entries are generated from a rotation-invariant sub-Gaussian distribution. We formulate it as the minimization of a nonconvex loss. This problem is related to the well understood phase retrieval problem where the measurement matrix is a rank-1 positive semidefinite matrix. Here we study the general full-rank case which models a number of key applications such as molecular geometry recovery from distance distributions and compound measurements in phaseless diffractive imaging. Most prior works either address the rank-1 case or focus on real measurements. The several papers that address the full-rank complex case adopt the computationally-demanding semidefinite relaxation approach. In this paper we prove that the general class of problems with rotation-invariant sub-Gaussian measurement models can be efficiently solved with high probability via the standard framework comprising a spectral initialization followed by iterative Wirtinger flow updates on a nonconvex loss. Numerical experiments on simulated data corroborate our theoretical analysis.

Which Factorization Machine Modeling Is Better: A Theoretical Answer with Optimal Guarantee

Factorization machine (FM) is a popular machine learning model to capture the second order feature interactions. The optimal learning guarantee of FM and its generalized version is not yet developed. For a rank k generalized FM of d dimensional input, the previous best known sampling complexity is O[k3d · polylog(kd)] under Gaussian distribution. This bound is sub-optimal comparing to the information theoretical lower bound O(kd). In this work, we aim to tighten this bound towards optimal and generalize the analysis to sub-gaussian distribution. We prove that when the input data satisfies the so-called τ-Moment Invertible Property, the sampling complexity of generalized FM can be improved to O[k2d · polylog(kd)/τ2]. When the second order self-interaction terms are excluded in the generalized FM, the bound can be improved to the optimal O[kd · polylog(kd)] up to the logarithmic factors. Our analysis also suggests that the positive semi-definite constraint in the conventional FM is redundant as it does not improve the sampling complexity while making the model difficult to optimize. We evaluate our improved FM model in real-time high precision GPS signal calibration task to validate its superiority.

Expectile regression for analyzing heteroscedasticity in high dimension

High-dimensional data often display heteroscedasticity and this feature has attracted a lot of attention and discussion. In this paper, we propose regularized expectile regression with SCAD penalty for analyzing heteroscedasticity in high dimension when the error has finite moments. Since the corresponding optimization problem is nonconvex due to the SCAD penalty, we adopt the CCCP (coupling of the concave and convex procedure) algorithm to solve this problem. Under some regular conditions, we can prove that with probability tending to one, the proposed algorithm converges to the oracle estimator after several iterations. We should address that the higher order moment the error has, the higher dimension cardinality our procedure can handle with. If the error follows gaussian or sub-gaussian distribution, our method can be extended to deal with ultra high-dimensional data. Furthermore, by taking different expectile weight level α, we are able to detect heteroscedasticity and explore the entire conditional distribution of the response variable given all the covariates. We investigate the performances of our proposed method through Monte Carlo simulation study and real application and the numerical results show that the resulting estimator by our algorithm enjoys good performance and demonstrate the usefulness of our proposed method to analyze heteroscedasticity.

Phase Retrieval Without Small-Ball Probability Assumptions

In the context of the phase retrieval problem, it is known that certain natural classes of measurements, such as Fourier measurements and random Bernoulli measurements, do not lead to the unique reconstruction of all possible signals, even in combination with certain practically feasible random masks. To avoid this difficulty, the analysis is often restricted to measurement ensembles (or masks) that satisfy a small-ball probability condition, in order to ensure that the reconstruction is unique. This paper shows a complementary result: for random Bernoulli measurements, there is still a large class of signals that can be reconstructed uniquely, namely those signals that are "non-peaky." In fact, this result is much more general: it holds for random measurements sampled from any subgaussian distribution D, without any small-ball conditions. This is demonstrated in two ways: first, a proof of stability and uniqueness, and second, a uniform recovery guarantee for the PhaseLift algorithm. In all of these cases, the number of measurements m approaches the information-theoretic lower bound. Finally, for random Bernoulli measurements with erasures, it is shown that PhaseLift achieves uniform recovery of all signals (including peaky ones).

Optimal and Near-Optimal Detection in Bursty Impulsive Noise

In many practical scenarios, the ambient noise process is known to be impulsive. To combat this, several robust measures have been proposed in the literature. Most of them assume white noise processes, i.e., the noise samples are independent and identically distributed heavy-tailed random variables. However, noise is seldom white in practice and therefore exhibits memory. For impulsive noise, dependency among samples results in outliers clustering together. The process is thus impulsive and bursty. In our work, we employ stationary $\alpha$ -sub-Gaussian noise with memory order $m$ ( $\alpha$ SGN( $m$ )) to model bursty impulsive noise. The model is based on the multivariate $\alpha$ -sub-Gaussian ( $\alpha$ SG) distribution family and statistically characterizes adjacent samples from elliptical distributions. The latter assumption holds well for snapping shrimp noise found in warm shallow underwater channels. We investigate the performance of conventional robust detectors in $\alpha$ SGN( $m$ ) and also propose novel near-optimal detectors. The Neyman–Pearson (NP) approach for binary hypothesis testing is considered and extensive simulation results for the aforementioned detectors are offered. For all instances, we employ an $\alpha$ SGN( $m$ ) process whose parameters are tuned to snapping shrimp noise data sets. By incorporating good signal design rules, it is shown that there is a large performance gap between the new and conventional detectors in various impulsive regimes. Moreover, it is possible to derive a near-optimal detector if one only has information of the temporal statistics of the noise process.

Bivariate sub-Gaussian model for stock index returns

Financial time series are commonly modeled with methods assuming data normality. However, the real distribution can be nontrivial, also not having an explicitly formulated probability density function. In this work we introduce novel parameter estimation and high-powered distribution testing methods which do not rely on closed form densities, but use the characteristic functions for comparison. The approach applied to a pair of stock index returns demonstrates that such a bivariate vector can be a sample coming from a bivariate sub-Gaussian distribution. The methods presented here can be applied to any nontrivially distributed financial data, among others.

A well-conditioned and sparse estimation of covariance and inverse covariance matrices using a joint penalty

We develop a method for estimating well-conditioned and sparse covariance and inverse covariance matrices from a sample of vectors drawn from a sub-Gaussian distribution in high dimensional setting...

An Efficient Two-Stage Sparse Representation Method

There are a large number of methods for solving under-determined linear inverse problems. For large-scale optimization problem, many of them have very high time complexity. We propose a new method called two-stage sparse representation (TSSR) to tackle it. We decompose the representing space of signals into two parts”, the measurement dictionary and the sparsifying basis. The dictionary is designed to obey or nearly obey the sub-Gaussian distribution. The signals are then encoded on the dictionary to obtain the training and testing coefficients individually in the first stage. Then, we design the basis based on the training coefficients to approach an identity matrix, and we apply sparse coding to the testing coefficients over the basis in the second stage. We verify that the projection of testing coefficients onto the basis is a good approximation of the original signals onto the representing space. Since the projection is conducted on a much sparser space, the runtime is greatly reduced. For concrete realization, we provide an instance for the proposed TSSR. Experiments on four biometric databases show that TSSR is effective compared to several classical methods for solving linear inverse problem.

Calibrating nonconvex penalized regression in ultra-high dimension

We investigate high-dimensional nonconvex penalized regression, where the number of covariates may grow at an exponential rate. Although recent asymptotic theory established that there exists a local minimum possessing the oracle property under general conditions, it is still largely an open problem how to identify the oracle estimator among potentially multiple local minima. There are two main obstacles: (1) due to the presence of multiple minima, the solution path is nonunique and is not guaranteed to contain the oracle estimator; (2) even if a solution path is known to contain the oracle estimator, the optimal tuning parameter depends on many unknown factors and is hard to estimate. To address these two challenging issues, we first prove that an easy-to-calculate calibrated CCCP algorithm produces a consistent solution path which contains the oracle estimator with probability approaching one. Furthermore, we propose a high-dimensional BIC criterion and show that it can be applied to the solution path to select the optimal tuning parameter which asymptotically identifies the oracle estimator. The theory for a general class of nonconvex penalties in the ultra-high dimensional setup is established when the random errors follow the sub-Gaussian distribution. Monte Carlo studies confirm that the calibrated CCCP algorithm combined with the proposed high-dimensional BIC has desirable performance in identifying the underlying sparsity pattern for high-dimensional data analysis.

A Blind Receiver with Multiple Antennas in Impulsive Noise Modeled as the Sub-Gaussian Distribution via the MCMC Algorithm

Receivers with an antenna array based on the assumption of Gaussian noise have experienced poor performance in extremely low-frequency/very-low-frequency (ELF/VLF) communication systems, in which the noise is highly impulsive. In this paper, a blind receiver with multiple antennas for channels with impulsive noise is developed to estimate signal and channel parameters jointly by using the Markov chain Monte Carlo (MCMC) algorithm. The impulsive noise is modeled as the sub-Gaussian distribution, which is widely applied. Simulation results show that the blind receiver can rapidly converge, and excellent performance can be achieved.

One-bit compressed sensing with non-Gaussian measurements

In one-bit compressed sensing, previous results state that sparse signals may be robustly recovered when the measurements are taken using Gaussian random vectors. In contrast to standard compressed sensing, these results are not extendable to natural non-Gaussian distributions without further assumptions, as can be demonstrated by simple counter-examples involving extremely sparse signals. We show that approximately sparse signals that are not extremely sparse can be accurately reconstructed from single-bit measurements sampled according to a sub-gaussian distribution, and the reconstruction comes as the solution to a convex program.

Dynamic Cardiac PET Imaging: Extraction of Time-Activity Curves Using ICA and a Generalized Gaussian Distribution Model

Kinetic modeling of metabolic and physiologic cardiac processes in small animals requires an input function (IF) and a tissue time-activity curves (TACs). In this paper, we present a mathematical method based on independent component analysis (ICA) to extract the IF and the myocardium's TACs directly from dynamic positron emission tomography (PET) images. The method assumes a super-Gaussian distribution model for the blood activity, and a sub-Gaussian distribution model for the tissue activity. Our appreach was applied on 22 PET measurement sets of small animals, which were obtained from the three most frequently used cardiac radiotracers, namely: desoxy-fluoro-glucose ((18)F-FDG), [(13)N]-ammonia, and [(11)C]-acetate. Our study was extended to PET human measurements obtained with the Rubidium-82 ((82) Rb) radiotracer. The resolved mathematical IF values compare favorably to those derived from curves extracted from regions of interest (ROI), suggesting that the procedure presents a reliable alternative to serial blood sampling for small-animal cardiac PET studies.

Optimized LSB Matching Steganography Based on Fisher Information

This paper proposes an optimized LSB matching steganography based on Fisher Information. The embedding algorithm is designed to solve the optimization problem, in which Fisher information is the objective function and embedding transferring probabilities are variables to be optimized. Fisher information is the quadratic function of the embedding transferring probabilities, and the coefficients of quadratic term are determined by the joint probability distribution of cover elements. By modeling the groups of elements in a cover image as Gaussian mixture model, the joint probability distribution of cover elements for each cover image is obtained by estimating the parameters of Gaussian mixture distribution. For each sub-Gaussian distribution in Gaussian mixture distribution, the quadratic term coefficients of Fisher information are calculated, and the optimized embedding transferring probabilities are solved by quadratic programming. By maximum posteriori probability principle, cover pixels are classified as the categories corresponding to sub-Gaussian distributions. At last, in order to embed message bits, pixels chose to add or subtract one according to the optimized transferring probabilities of the category. The experiments show that the security performance of this new algorithm is better than the existing LSB matching.

A method of underdetermined blind source separation with an unknown number of sources

Aiming to source number estimation, the recovery of mixing matrix and source signal under underdetermined case, we propose a method of underdetermined blind source separation with an unknown number of sources. Firstly, we introduced an algorithm based on S transform and fuzzy c-means clustering technique to estimate the number of sources and mixing mixtures. Then sources are represented as null space form and the source signals are recovered using an algorithm based on Maximum Likelihood. The simulation results show that the proposed method can separate sources of super- and sub-Gaussian distribution, and it has superior evaluation performance to the conventional methods.

Modeling fat tails in stock returns: a multivariate stable-GARCH approach

In this paper a new multivariate volatility model is proposed. It combines the appealing properties of the stable Paretian distribution to model the heavy tails with the GARCH model to capture the volatility clustering. Returns on assets are assumed to follow a sub-Gaussian distribution, which is a particular multivariate stable distribution. In this way the characteristic function of the fitted returns has a tractable expression and the density function can be recovered by numerical methods. A multivariate GARCH structure is then adopted to model the covariance matrix of the Gaussian vectors underlying the sub-Gaussian system. The model is applied to a bivariate series of daily U.S. stock returns. Value-at-risk for long and short positions is computed and compared with the one obtained using the multivariate normal and the multivariate Student’s t distribution. Finally, exploiting the recent developments in the vast dimensional time-varying covariances modeling, possible feasible extensions of our model to higher dimensions are suggested and an illustrative example using the Dow Jones index components is presented.

Modeling Fat Tails in Stock Returns: A Multivariate Stable-GARCH Approach

In this paper a new multivariate volatility model is proposed. It combines the appealing properties of the stable Paretian distribution to model the heavy tails with the GARCH model to capture the volatility clustering. We assume that multivariate asset-returns of financial stocks follow a sub-Gaussian distribution, which is a particular multivariate stable distribution. In this way the characteristic function of the fitted returns has a tractable expression and the density function can be recovered by numerical methods. A multivariate GARCH structure is then adopted to model the covariance matrix of the Gaussian vectors underlying the sub-Gaussian system. The model is applied to a bivariate series of daily U.S. stock returns. Value-at-Risk for long and short positions is computed and compared with the one obtained using the multivariate normal and the multivariate Student's t distribution. Finally, exploiting the recent developments in the vast dimensional time-varying covariances modeling, possible feasible extensions to higher dimensions are suggested and an illustrative example using the Dow Jones index components is presented.

Comparison of ICA approaches for facial expression recognition

Independent component analysis (ICA) and Gabor wavelets extract the most discriminating features for facial action unit classification by employing either a cosine similarity measure (CSM) classifier or support vector machines (SVMs). So far, only the ICA approach, which is based on the InfoMax principle, has been tested for facial expression recognition. In this paper, in addition to the InfoMax approach, another five ICA approaches extract features from two facial expression databases. In particular, the Extended InfoMax ICA, the undercomplete ICA, and the nonlinear kernel-ICA approaches are exploited for facial expression representation for the first time. When applied to images, ICA treats the images as being mixtures of independent sources and decomposes them into an independent basis and the corresponding mixture coefficients. Two architectures for representing the images can be employed yielding either independent and sparse basis images or independent and sparse distributions of image representation coefficients. After feature extraction, facial expression classification is performed with the help of either a CSM classifier or an SVM classifier. A detailed comparative study is made with respect to the accuracy offered by each classifier. The correlation between the accuracy and the mutual information of independent components or the kurtosis is evaluated. Statistically significant correlations between the aforementioned quantities are identified. Several issues are addressed in the paper: (i) whether features having super- and sub-Gaussian distribution facilitate facial expression classification; (ii) whether a nonlinear mixture of independent sources improves the classification accuracy; and (iii) whether an increased “amount” of sparseness yields more accurate facial expression recognition. In addition, performance enhancements by employing leave-one-set of expressions-out and subspace selection are studied. Statistically significant differences in accuracy between classifiers using several feature extraction methods are also indicated.