High-dimensional Samples Research Articles

A Two-Stage Method Based on Multiobjective Differential Evolution for Gene Selection.

Microarray gene expression data provide a prospective way to diagnose disease and classify cancer. However, in bioinformatics, the gene selection problem, i.e., how to select the most informative genes from thousands of genes, remains challenging. This problem is a specific feature selection problem with high-dimensional features and small sample sizes. In this paper, a two-stage method combining a filter feature selection method and a wrapper feature selection method is proposed to solve the gene selection problem. In contrast to common methods, the proposed method models the gene selection problem as a multiobjective optimization problem. Both stages employ the same multiobjective differential evolution (MODE) as the search strategy but incorporate different objective functions. The three objective functions of the filter method are mainly based on mutual information. The two objective functions of the wrapper method are the number of selected features and the classification error of a naive Bayes (NB) classifier. Finally, the performance of the proposed method is tested and analyzed on six benchmark gene expression datasets. The experimental results verified that this paper provides a novel and effective way to solve the gene selection problem by applying a multiobjective optimization algorithm.

Diproperm: An R Package for the DiProPerm Test.

High-dimensional low sample size (HDLSS) data sets frequently emerge in many biomedical applications. The direction-projection-permutation (DiProPerm) test is a two-sample hypothesis test for comparing two high-dimensional distributions. The DiProPerm test is exact, i.e., the type I error is guaranteed to be controlled at the nominal level for any sample size, and thus is applicable in the HDLSS setting. This paper discusses the key components of the DiProPerm test, introduces the diproperm R package, and demonstrates the package on a real-world data set.

Underwater bubbles recognition based on PCA feature extraction and elastic BP neural network

Aiming at the problem of difficulty in feature extraction and identification of wake echo signals detected by underwater lidar due to instability, an underwater bubbles recognition algorithm based on PCA feature extraction and elastic BP neural network was proposed. First, slice preprocessing was carried out on the echo signals collected continuously. Then the PCA algorithm was used to extract the main features of the spliced high-dimensional samples to determine the number of feature values. Then the parameters of the elastic BP neural network was selected to determine the number of hidden layer node and the number of features that can achieve optimal classification. Finally, an indoor wake detection simulation platform was used to realize the identification of bubbles and interfering targets. The experimental results show that when the hidden node is 12, the increment factor is 1.15, and the decrement factor is 0.55, two eigenvalues can be selected to classify the bubbles, non-bubbles and interfering targets. The recognition rate increases by 13.4% with the increase of bubbles density. At low density, the average recognition rate increases by 6.3% with the increase of laser energy. The recognition rate first increases and then decreases with the increase of distance. When the bubbles distance is 2.2 m, the target peak characteristics are obvious, and the average recognition rate is improved by 3.5%. Compared with adaptive and additional momentum BP, this method can reduce recognition time and achieve 99.1% accuracy. It is proved that this algorithm can be effectively and widely used in the recognition of bubbles in the ships wake by lidar.

The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism

We present scalable hybrid-parallel algorithms for training large-scale 3D convolutional neural networks. Deep learning-based emerging scientific workflows often require model training with large, high-dimensional samples, which can make training much more costly and even infeasible due to excessive memory usage. We solve these challenges by extensively applying hybrid parallelism throughout the end-to-end training pipeline, including both computations and I/O. Our hybrid-parallel algorithm extends the standard data parallelism with spatial parallelism, which partitions a single sample in the spatial domain, realizing strong scaling beyond the mini-batch dimension with a larger aggregated memory capacity. We evaluate our proposed training algorithms with two challenging 3D CNNs, CosmoFlow and 3D U-Net. Our comprehensive performance studies show that good weak and strong scaling can be achieved for both networks using up to 2K GPUs. More importantly, we enable training of CosmoFlow with much larger samples than previously possible, realizing an order-of-magnitude improvement in prediction accuracy.

Conditional out-of-distribution generation for unpaired data using transfer VAE.

While generative models have shown great success in sampling high-dimensional samples conditional on low-dimensional descriptors (stroke thickness in MNIST, hair color in CelebA, speaker identity in WaveNet), their generation out-of-distribution poses fundamental problems due to the difficulty of learning compact joint distribution across conditions. The canonical example of the conditional variational autoencoder (CVAE), for instance, does not explicitly relate conditions during training and, hence, has no explicit incentive of learning such a compact representation. We overcome the limitation of the CVAE by matching distributions across conditions using maximum mean discrepancy in the decoder layer that follows the bottleneck. This introduces a strong regularization both for reconstructing samples within the same condition and for transforming samples across conditions, resulting in much improved generalization. As this amount to solving a style-transfer problem, we refer to the model as transfer VAE (trVAE). Benchmarking trVAE on high-dimensional image and single-cell RNA-seq, we demonstrate higher robustness and higher accuracy than existing approaches. We also show qualitatively improved predictions by tackling previously problematic minority classes and multiple conditions in the context of cellular perturbation response to treatment and disease based on high-dimensional single-cell gene expression data. For generic tasks, we improve Pearson correlations of high-dimensional estimated means and variances with their ground truths from 0.89 to 0.97 and 0.75 to 0.87, respectively. We further demonstrate that trVAE learns cell-type-specific responses after perturbation and improves the prediction of most cell-type-specific genes by 65%. The trVAE implementation is available via github.com/theislab/trvae. The results of this article can be reproduced via github.com/theislab/trvae_reproducibility.

A scalable surrogate [formula omitted] sparse regression method for generalized linear models with applications to large scale data

This paper rigorously studies large sample properties of a surrogate L0 penalization method via iteratively performing reweighted L2 penalized regressions for generalized linear models and develop a scalable implementation of the method for sparse high dimensional massive sample size (sHDMSS) data. We show that for generalized linear models, the limit of the algorithm, referred to as the broken adaptive ridge (BAR) estimator, is consistent for variable selection, enjoys an oracle property for parameter estimation, and possesses a grouping property for highly correlated covariates. We further demonstrate that by taking advantage of an existing efficient implementation of massive L2-penalized generalized linear models, the proposed BAR method can be conveniently implemented for sHDMSS data. An illustration is given using a large sHDMSS data from the Truven MarketScan Medicare (MDCR) database to investigate the safety of dabigatran versus warfarin for treatment of nonvalvular atrial filbrillation in elder patients.

Modified Mahalanobis-Taguchi System based on proper orthogonal decomposition for high-dimensional-small-sample-size data classification.

Mahalanobis-Taguchi System (MTS) is an effective algorithm for dimensionality reduction, feature extraction and classification of data in a multidimensional system. However, when applied to the field of high-dimensional small sample data, MTS has challenges in calculating the Mahalanobis distance due to the singularity of the covariance matrix. To this end, we construct a modified Mahalanobis-Taguchi System (MMTS) by introducing the idea of proper orthogonal decomposition (POD). The constructed MMTS expands the application scope of MTS, taking into account correlations between variables and the influence of dimensionality. It can not only retain most of the original sample information features, but also achieve a substantial reduction in dimensionality, showing excellent classification performance. The results show that, compared with expert classification, individual classifiers such as NB, RF, k-NN, SVM and superimposed classifiers such as Wrapper + RF, MRMR + SVM, Chi-square + BP, SMOTE + Wrapper + RF and SMOTE + MRMR + SVM, MMTS has a better classification performance when extracting orthogonal decomposition vectors with eigenvalues greater than 0.001.

Two-sample Behrens–Fisher problems for high-dimensional data: A normal reference approach

High-dimensional data are frequently encountered with the development of modern data collection techniques. Testing the equality of the mean vectors of two high-dimensional samples with possibly different covariance matrices is usually referred to as a high-dimensional two-sample Behrens–Fisher (BF) problem. In the high-dimensional setting, the classical BF solutions are expected to perform poorly or become inapplicable due to the singularity of the sample covariance matrices. Several approaches have been proposed in the literature to address this challenging issue but they all require strong regularity conditions on the underlying covariance matrices to guarantee that their test statistics are asymptotically normally distributed. To overcome this difficulty, an L2-norm-based test is proposed and studied in this article. It is shown that under some regularity conditions and the null hypothesis, the test statistic and a chi-square-type mixture have the same normal or non-normal limiting distribution. It is then natural to approximate the null distribution of the proposed test using that of the chi-square-type mixture, which is actually obtained from the proposed test statistic when the two high-dimensional samples are normally distributed. The resulting test is then referred to as a normal reference test. The distribution of the chi-square-type mixture can then be well approximated by the Welch–Satterthwaite χ2-approximation with the approximation parameters consistently estimated from the data. The asymptotic power of the proposed test is established. Good performance of the proposed test against several existing competitors is demonstrated via several simulation studies and illustrated by a real data example.

I- flow: High-dimensional integration and sampling with normalizing flows

In many fields of science, high-dimensional integration is required. Numerical methods have been developed to evaluate these complex integrals. We introduce the code i-flow, a Python package that performs high-dimensional numerical integration utilizing normalizing flows. Normalizing flows are machine-learned, bijective mappings between two distributions. i-flow can also be used to sample random points according to complicated distributions in high dimensions. We compare i-flow to other algorithms for high-dimensional numerical integration and show that i-flow outperforms them for high dimensional correlated integrals. The i-flow code is publicly available on gitlab at https://gitlab.com/i-flow/i-flow.

Robust sparse coding for one-class classification based on correntropy and logarithmic penalty function

Similar to binary and multi-class classifiers, one-class classifiers have to face the difficulty of ’curse of dimensionality’ when they are applied to deal with high-dimensional samples. As an efficient dimensionality reduction method, sparse coding tries to learn a set of over-complete bases to represent the given samples. It can effectively overcome the ’curse of dimensionality’ problem. However, the traditional sparse coding only fit for tackling Gaussian noise. When the noise within the given set of samples obey non-Gaussian distribution, the conventional sparse coding cannot obtain accurate coefficient vectors. To make sparse coding more fit for dealing with non-Gaussian noise and enhance the sparseness of the obtained coefficient vectors, correntropy is utilized to substitute its reconstruction error term and logarithmic penalty function is introduced as its regularization term. Furthermore, the obtained sparse coefficient vectors are used as the input vectors for one-class support vector machine (OCSVM). Experimental results on twenty UCI benchmark data sets and one handwritten digit data set demonstrate that the proposed method achieves better anti-noise and generalization abilities in comparison with its related approaches.

Distributed Direct Position Determination

The direct position determination (DPD) algorithms have been shown to outperform conventional two-step methods in localization accuracy under poor observation conditions (e.g., low signal-to-noise ratios or short signal observation durations). In this paper, we consider the distributed DPD of a static source in large-scale sensor networks. Different from the centralized DPD (CDPD) algorithms where all the raw signal samples are transmitted globally to a fusion center to solve for the source position, in our method the signal samples are only exchanged locally between neighboring sensors, and the source position is estimated using the proposed DPD algorithm distributively. The problem-specific cost function, which is different from the cost function of CDPD, is first derived. To obtain the cost function at each sensor, we propose to transmit the low-dimensional sufficient statistics across the sensor networks instead of the original high-dimensional raw signal samples, and to reconstruct the cost function via simple interpolations. Simplification of cost function evaluation is also developed to further reduce complexity. Finally, source position is estimated at each sensor efficiently via the importance sampling algorithm. Simulation results demonstrate the superior performance of the proposed method over existing adaptive distributed DPD as well as the two-step algorithms under poor observation conditions, and its computational efficiency is significantly improved compared with the CDPD.

Shape Adaptive Neighborhood Information-Based Semi-Supervised Learning for Hyperspectral Image Classification

Hyperspectral image (HSI) classification is an important research topic in detailed analysis of the Earth’s surface. However, the performance of the classification is often hampered by the high-dimensionality features and limited training samples of the HSIs which has fostered research about semi-supervised learning (SSL). In this paper, we propose a shape adaptive neighborhood information (SANI) based SSL (SANI-SSL) method that takes full advantage of the adaptive spatial information to select valuable unlabeled samples in order to improve the classification ability. The improvement of the classification mainly relies on two aspects: (1) the improvement of the feature discriminability, which is accomplished by exploiting spectral-spatial information, and (2) the improvement of the training samples’ representativeness which is accomplished by exploiting the SANI for both labeled and unlabeled samples. First, the SANI of labeled samples is extracted, and the breaking ties (BT) method is used in order to select valuable unlabeled samples from the labeled samples’ neighborhood. Second, the SANI of unlabeled samples are also used to find more valuable samples, with the classifier combination method being used as a strategy to ensure confidence and the adaptive interval method used as a strategy to ensure informativeness. The experimental comparison results tested on three benchmark HSI datasets have demonstrated the significantly superior performance of our proposed method.

Unsupervised spectral mapping and feature selection for hyperspectral anomaly detection

Exploring techniques that breakthrough the unknown space or material species is of considerable significance to military and civilian fields, and it is a challenging task without any prior information. Nowadays, the use of material-specific spectral information to detect unknowns has received increasing interest. However, affected by noise and interference, high-dimensional hyperspectral anomaly detection is difficult to meet the requirements of high detection accuracy and low false alarm rate. Besides, there is a problem of insufficient and unbalanced samples. To address these problems, we propose a novel hyperspectral anomaly detection framework based on spectral mapping and feature selection (SMFS) in an unsupervised manner. The SMFS introduces the essential properties of hyperspectral data into an unsupervised neural network to construct the nonlinear mapping relationship from high-dimensional spectral space to low-dimensional deep feature space. And it searches the optimal feature subset from the candidate feature space for standing out anomalies. Because of the compelling characterization of the encoder, we develop it specifically for spectral signatures to reveal the hidden data. Quantitative and qualitative experiments on real hyperspectral datasets indicate that the proposed method can provide the compact features overcoming the problems of noise, interference, redundancy and time-consuming caused by high-dimensionality and limited samples. And it has advantages over some state-of-the-art competitors concerning detecting anomalies of different scales.

An asymptotic and empirical smoothing parameters selection method for smoothing spline ANOVA models in large samples.

Large samples are generated routinely from various sources. Classic statistical models, such as smoothing spline ANOVA models, are not well equipped to analyse such large samples because of high computational costs. In particular, the daunting computational cost of selecting smoothing parameters renders smoothing spline ANOVA models impractical. In this article, we develop an asympirical, i.e., asymptotic and empirical, smoothing parameters selection method for smoothing spline ANOVA models in large samples. The idea of our approach is to use asymptotic analysis to show that the optimal smoothing parameter is a polynomial function of the sample size and an unknown constant. The unknown constant is then estimated through empirical subsample extrapolation. The proposed method significantly reduces the computational burden of selecting smoothing parameters in high-dimensional and large samples. We show that smoothing parameters chosen by the proposed method tend to the optimal smoothing parameters that minimize a specific risk function. In addition, the estimator based on the proposed smoothing parameters achieves the optimal convergence rate. Extensive simulation studies demonstrate the numerical advantage of the proposed method over competing methods in terms of relative efficacy and running time. In an application to molecular dynamics data containing nearly one million observations, the proposed method has the best prediction performance.

Robust Subspace Clustering via Latent Smooth Representation Clustering

Subspace clustering aims to group high-dimensional data samples into several subspaces which they were generated. Among the existing subspace clustering methods, spectral clustering-based algorithms have attracted considerable attentions because of their predominant performances shown in many subspace clustering applications. In this paper, we proposed to apply smooth representation clustering (SMR) to the reconstruction coefficient vectors which were obtained by sparse subspace clustering (SSC). Because the reconstruction coefficient vectors could be regarded as a kind of good representations of original data samples, the proposed method could be considered as a SMR performed in a latent subspace found by SSC and hoped to achieve better performances. For solving the proposed latent smooth representation algorithm (LSMR), we presented an optimization method and also discussed the relationships between LSMR with some related algorithms. Finally, experiments conducted on several famous databases demonstrate that the proposed algorithm dominates the related algorithms.

Distance assessment and analysis of high-dimensional samples using variational autoencoders

An important question in many machine learning applications is whether two samples arise from the same generating distribution. Although an old topic in Statistics, simple accept/reject decisions given by most hypothesis tests are often not enough: it is well known that the rejection of the null hypothesis does not imply that differences between the two groups are meaningful from a practical perspective. In this work, we present a novel nonparametric approach to visually assess the dissimilarity between the datasets that goes beyond two-sample testing. The key idea of our approach is to measure the distance between two (possibly) high-dimensional datasets using variational autoencoders. We also show how this framework can be used to create a formal statistical test to test the hypothesis that both samples arise from the same distribution. We evaluate both the distance measurement and hypothesis testing approaches on simulated and real world datasets. The results show that our approach is useful for data exploration (as it, for instance, allows for quantification of the discrepancy/separability between categories of images), which can be particularly helpful in early phases of the a machine learning pipeline.

Discriminative Reconstruction Constrained Generative Adversarial Network for Hyperspectral Anomaly Detection

The rich and distinguishable spectral information in hyperspectral images (HSIs) makes it possible to capture anomalous samples [i.e., anomaly detection (AD)] that deviate from background samples. However, hyperspectral anomaly detection (HAD) faces various challenges due to high dimensionality, redundant information, and unlabeled and limited samples. To address these problems, this article proposes an unsupervised discriminative reconstruction constrained generative adversarial network for HAD (HADGAN). Our solution is mainly based on the assumption that the number of normal samples is much larger than the number of abnormal ones. The key contribution of this article is to learn a discriminative background reconstruction with anomaly targets being suppressed, which produces the initial detection image (i.e., the residual image between the original image and reconstructed image) with anomaly targets being highlighted and background samples being suppressed. To accomplish this goal, first, by using an autoencoder (AE) network and an adversarial latent discriminator, the latent feature layer learns normal background distribution and AE learns a background reconstruction as much as possible. Second, consistency enhanced representation and shrink constraints are added to the latent feature layer to ensure that anomaly samples are projected to similar positions as normal samples in the latent feature layer. Third, using an adversarial image feature corrector in the input space can guarantee the reliability of the generated samples. Finally, an energy-based spatial and distance-based spectral joint anomaly detector is applied in the residual map to generate the final detection map. Experiments conducted on several data sets over different scenes demonstrate its state-of-the-art performance.

A Fault Diagnosis Model of Marine Diesel Engine Fuel Oil Supply System Using PCA and Optimized SVM

The fuel oil supply system of the marine diesel engine contains many components, which fits plenty of sensors to monitor the condition of all components. A fault sample consists of data collected from all the sensors at certain time, which lead the dimension of the fault sample is very high. When the ship is sailing, there is a randomness in fault categories and fault duration, which leads the fault data unbalanced. This paper proposes an appropriate combinational approach to address the above problems. First, to reduce computational complexity, the high dimensional fault samples are converted into the low dimensional ones using the principal component analysis (PCA). Second, a sample size optimization (SSO) strategy is proposed to address the problem of the learning from the imbalanced datasets, which improve the classification performance of support vector machine (SVM). Third, a three-dimensional Arnold mapping is introduced into the particle swarm optimization (PSO) algorithm to improve its generalization capability. Finally, the SVM optimized by the improved PSO is trained as the classifier to identify the ten faults in the fuel oil supply system. Results demonstrate that the average correct diagnosis ratio can be as high as 93.9%.

Log-sum enhanced sparse deep neural network

How to design deep neural networks (DNNs) for the representation and analysis of high dimensional but small sample size data is still a big challenge. One solution is to construct a sparse network. At present, there exist many approaches to achieve sparsity for DNNs by regularization, but most of them are carried out only in the pre-training process due to the difficulty in the derivation of explicit formulae in the fine-tuning process. In this paper, a log-sum function is used as the regularization terms for both the responses of hidden neurons and the network connections in the loss function of the fine-tuning process. It provides a better approximation to the L0-norm than several often used norms. Based on the gradient formula of the loss function, the fine-tuning process can be executed more efficiently. Specifically, the commonly used gradient calculation in many deep learning research platforms, such as PyTorch or TensorFlow, can be accelerated. Given the analytic formula for calculating gradients used in any layer of DNN, the error accumulated from successive numerical approximations in the differentiation process can be avoided. With the proposed log-sum enhanced sparse deep neural network (LSES-DNN), the sparsity of the responses and the connections can be well controlled to improve the adaptivity of DNNs. The proposed model is applied to MRI data for both the diagnosis of schizophrenia and the study of brain developments. Numerical experiments demonstrate its superior performance among several classical classifiers tested.

Optimized Mahalanobis-Taguchi System for High-Dimensional Small Sample Data Classification.

The Mahalanobis–Taguchi system (MTS) is a multivariate data diagnosis and prediction technology, which is widely used to optimize large sample data or unbalanced data, but it is rarely used for high-dimensional small sample data. In this paper, the optimized MTS for the classification of high-dimensional small sample data is discussed from two aspects, namely, the inverse matrix instability of the covariance matrix and the instability of feature selection. Firstly, based on regularization and smoothing techniques, this paper proposes a modified Mahalanobis metric to calculate the Mahalanobis distance, which is aimed at reducing the influence of the inverse matrix instability under small sample conditions. Secondly, the minimum redundancy-maximum relevance (mRMR) algorithm is introduced into the MTS for the instability problem of feature selection. By using the mRMR algorithm and signal-to-noise ratio (SNR), a two-stage feature selection method is proposed: the mRMR algorithm is first used to remove noise and redundant variables; the orthogonal table and SNR are then used to screen the combination of variables that make great contribution to classification. Then, the feasibility and simplicity of the optimized MTS are shown in five datasets from the UCI database. The Mahalanobis distance based on regularization and smoothing techniques (RS-MD) is more robust than the traditional Mahalanobis distance. The two-stage feature selection method improves the effectiveness of feature selection for MTS. Finally, the optimized MTS is applied to email classification of the Spambase dataset. The results show that the optimized MTS outperforms the classical MTS and the other 3 machine learning algorithms.