Linear Dimensionality Reduction Techniques Research Articles

Fusion of linear and non-linear dimensionality reduction techniques for feature reduction in LSTM-based Intrusion Detection System

Securing networks is becoming increasingly crucial due to the widespread use of information technology. Intrusion Detection System (IDS) plays a crucial role in network security by detecting potential security threats in real-time. In order to create effective IDS, Deep Learning (DL) techniques like Auto Encoder (AE) and Long Short-Term Memory (LSTM) have been widely used. However, the high dimensionality and complexity of network traffic data make it challenging to extract meaningful information. The paper proposes a fusion-based approach that combines AE and Principal Component Analysis (PCA) techniques for dimensionality reduction in IDS. The proposed approach aims to capture both linear and non-linear relationships between features while reducing the dimensionality of the input data. NSL-KDD, UNSW-NB15, CIC-IDS-2017, and MSCAD datasets are used to evaluate this proposed method. The proposed approach has improved accuracy compared to the existing AE+LSTM-based approach by 3% for NSL-KDD and around 1% for UNSW-NB15 and CIC-IDS-2017, while the proposed approach gives comparable results for the MSCAD dataset. The Wilcoxon signed-rank test has been applied to confirm the statistical significance of the result.

NPCA: a linear dimensionality reduction method using a multilayer perceptron.

Background: Linear dimensionality reduction techniques are widely used in many applications. The goal of dimensionality reduction is to eliminate the noise of data and extract the main features of data. Several dimension reduction methods have been developed, such as linear-based principal component analysis (PCA), nonlinear-based t-distributed stochastic neighbor embedding (t-SNE), and deep-learning-based autoencoder (AE). However, PCA only determines the projection direction with the highest variance, t-SNE is sometimes only suitable for visualization, and AE and nonlinear methods discard the linear projection. Results: To retain the linear projection of raw data and generate a better result of dimension reduction either for visualization or downstream analysis, we present neural principal component analysis (nPCA), an unsupervised deep learning approach capable of retaining richer information of raw data as a promising improvement to PCA. To evaluate the performance of the nPCA algorithm, we compare the performance of 10 public datasets and 6 single-cell RNA sequencing (scRNA-seq) datasets of the pancreas, benchmarking our method with other classic linear dimensionality reduction methods. Conclusion: We concluded that the nPCA method is a competitive alternative method for dimensionality reduction tasks.

The dimensionality reductions of environmental variables have a significant effect on the performance of species distribution models.

How to effectively obtain species-related low-dimensional data from massive environmental variables has become an urgent problem for species distribution models (SDMs). In this study, we will explore whether dimensionality reduction on environmental variables can improve the predictive performance of SDMs. We first used two linear (i.e., principal component analysis (PCA) and independent components analysis) and two nonlinear (i.e., kernel principal component analysis (KPCA) and uniform manifold approximation and projection) dimensionality reduction techniques (DRTs) to reduce the dimensionality of high-dimensional environmental data. Then, we established five SDMs based on the environmental variables of dimensionality reduction for 23 real plant species and nine virtual species, and compared the predictive performance of those with the SDMs based on the selected environmental variables through Pearson's correlation coefficient (PCC). In addition, we studied the effects of DRTs, model complexity, and sample size on the predictive performance of SDMs. The predictive performance of SDMs under DRTs other than KPCA is better than using PCC. And the predictive performance of SDMs using linear DRTs is better than using nonlinear DRTs. In addition, using DRTs to deal with environmental variables has no less impact on the predictive performance of SDMs than model complexity and sample size. When the model complexity is at the complex level, PCA can improve the predictive performance of SDMs the most by 2.55% compared with PCC. At the middle level of sample size, the PCA improved the predictive performance of SDMs by 2.68% compared with the PCC. Our study demonstrates that DRTs have a significant effect on the predictive performance of SDMs. Specifically, linear DRTs, especially PCA, are more effective at improving model predictive performance under relatively complex model complexity or large sample sizes.

Toroidal PCA via density ridges

Principal Component Analysis (PCA) is a well-known linear dimension-reduction technique designed for Euclidean data. In a wide spectrum of applied fields, however, it is common to observe multivariate circular data (also known as toroidal data), rendering spurious the use of PCA on it due to the periodicity of its support. This paper introduces Toroidal Ridge PCA (TR-PCA), a novel construction of PCA for bivariate circular data that leverages the concept of density ridges as a flexible first principal component analog. Two reference bivariate circular distributions, the bivariate sine von Mises and the bivariate wrapped Cauchy, are employed as the parametric distributional basis of TR-PCA. Efficient algorithms are presented to compute density ridges for these two distribution models. A complete PCA methodology adapted to toroidal data (including scores, variance decomposition, and resolution of edge cases) is introduced and implemented in the companion R package ridgetorus. The usefulness of TR-PCA is showcased with a novel case study involving the analysis of ocean currents on the coast of Santa Barbara.

Autoencoder-based myoelectric controller for prosthetic hands.

In the past, linear dimensionality-reduction techniques, such as Principal Component Analysis, have been used to simplify the myoelectric control of high-dimensional prosthetic hands. Nonetheless, their nonlinear counterparts, such as Autoencoders, have been shown to be more effective at compressing and reconstructing complex hand kinematics data. As a result, they have a potential of being a more accurate tool for prosthetic hand control. Here, we present a novel Autoencoder-based controller, in which the user is able to control a high-dimensional (17D) virtual hand via a low-dimensional (2D) space. We assess the efficacy of the controller via a validation experiment with four unimpaired participants. All the participants were able to significantly decrease the time it took for them to match a target gesture with a virtual hand to an average of and three out of four participants significantly improved path efficiency. Our results suggest that the Autoencoder-based controller has the potential to be used to manipulate high-dimensional hand systems via a myoelectric interface with a higher accuracy than PCA; however, more exploration needs to be done on the most effective ways of learning such a controller.

Connecting Vibrational Spectroscopy to Atomic Structure via Supervised Manifold Learning: Beyond Peak Analysis.

Vibrational spectroscopy is a nondestructive technique commonly used in chemical and physical analyses to determine atomic structures and associated properties. However, the evaluation and interpretation of spectroscopic profiles based on human-identifiable peaks can be difficult and convoluted. To address this challenge, we present a reliable protocol based on supervised manifold learning techniques meant to connect vibrational spectra to a variety of complex and diverse atomic structure configurations. As an illustration, we examined a large database of virtual vibrational spectroscopy profiles generated from atomistic simulations for silicon structures subjected to different stress, amorphization, and disordering states. We evaluated representative features in those spectra via various linear and nonlinear dimensionality reduction techniques and used the reduced representation of those features with decision trees to correlate them with structural information unavailable through classical human-identifiable peak analysis. We show that our trained model accurately (over 97% accuracy) and robustly (insensitive to noise) disentangles the contribution from the different material states, hence demonstrating a comprehensive decoding of spectroscopic profiles beyond classical (human-identifiable) peak analysis.

Model order reduction techniques to identify submarining risk in a simplified human body model

This work investigates linear and non-linear parametric reduced order models (ROM) capable of replacing computationally expensive high-fidelity simulations of human body models (HBM) through a non-intrusive approach. Conventional crash simulation methods pose a computational barrier that restricts profound analyses such as uncertainty quantification, sensitivity analysis, or optimization studies. The non-intrusive framework couples dimensionality reduction techniques with machine learning-based surrogate models that yield a fast responding data-driven black-box model. A comparative study is made between linear and non-linear dimensionality reduction techniques. Both techniques report speed-ups of a few orders of magnitude with an accurate generalization of the design space. These accelerations make ROMs a valuable tool for engineers.

A non-convex optimization framework for large-scale low-rank matrix factorization

Low-rank matrix factorization problems such as non negative matrix factorization (NMF) can be categorized as a clustering or dimension reduction technique. The latter denotes techniques designed to find representations of some high dimensional dataset in a lower dimensional manifold without a significant loss of information. If such a representation exists, the features ought to contain the most relevant features of the dataset. Many linear dimensionality reduction techniques can be formulated as a matrix factorization. In this paper, we combine the conjugate gradient (CG) method with the Barzilai and Borwein (BB) gradient method, and propose a BB scaling CG method for NMF problems. The new method does not require to compute and store matrices associated with Hessian of the objective functions. Moreover, adopting a suitable BB step size along with a proper nonmonotone strategy which comes by the size of convex parameter ηk, results in a new algorithm that can significantly improve the CPU time, efficiency, the number of function evaluation. Convergence result is established and numerical comparisons of methods on both synthetic and real-world datasets show that the proposed method is efficient in comparison with existing methods and demonstrate the superiority of our algorithms.

Active target classification using a shallow neural network with dimension reduction

Techniques for automatically identifying sonar targets using machine learning algorithms are being actively developed, and modern algorithms based on deep learning are often considered first. However, deep learning based algorithms require a lot of data to train a neural network, and if there is not enough data, overfitting easily occurs and the performance on real data can be degraded. In fact, in the case of active target detection where there is not enough data, it has been reported that using a shallow neural network after extracting appropriately designed features from the raw data showed better detection performance than applying deep learning directly to the raw data. With regard to this, we investigate the performance of a shallow neural network combined with dimensionality reduction techniques for active sonar target classification. In particular, several linear and nonlinear dimension reduction techniques are compared in terms of target classification performance, and the effects of the characteristics of background noise and the presence of reverberation on the target classification performance are discussed. [This work was supported by the research project PES4380 funded by KRISO.]

Visualization of emergency department clinical data for interpretable patient phenotyping

Visual summarization of clinical data collected on patients contained within the electronic health record (EHR) may enable precise and rapid triage at the time of patient presentation to an emergency department (ED). The triage process is critical in the appropriate allocation of resources and in anticipating eventual patient disposition, typically admission to the hospital or discharge home. EHR data are high-dimensional and complex, but offer the opportunity to discover and characterize underlying data-driven patient phenotypes. Data-driven phenotypes are intended to relieve reliance on weak labels like diagnosis codes and to aid in identifying populations of existing patients that are most similar to a specific patient. These phenotypes will enable improved, personalized therapeutic decision making and prognostication. In this work, we focus on the challenge of two-dimensional patient projections. A low dimensional embedding offers visual interpretability lost in higher dimensions. While linear dimensionality reduction techniques such as principal component analysis are often used towards this aim, they are insufficient to describe the variance of patient data. This linear reduction does not account for higher order, non-linear interactions of variables. In this work, we employ the newly-described non-linear embedding technique called uniform manifold approximation and projection (UMAP). UMAP seeks to capture both local and global structures in high-dimensional data. We then use Gaussian mixture models to identify clusters in the embedded data and use the adjusted Rand index (ARI) to establish stability in the discovery of these clusters. This technique is applied to five common clinical chief complaints from a real-world ED EHR dataset, describing the emergent properties of discovered clusters. We observe clinically-relevant cluster attributes, suggesting that visual embeddings of EHR data using non-linear dimensionality reduction is a promising approach to reveal data-driven patient phenotypes. In the five chief complaints, we find between 2 and 6 clusters, with the peak mean pairwise ARI between subsequent training iterations to range from 0.35 to 0.74.

Exploring brand perception in COVID vaccine tweets using TOPSA-NMF model

COVID-19 vaccines are primary biomedical preventive measures to combat the pandemic. Social media can have either a positive or negative impact on vaccine uptake, depending on their opinions about vaccines. An algorithm is proposed that can use a linear dimension reduction technique such as non-negative matrix factorization (NMF) to identify topics while using the sentiment lexicon to classify the tweet topics as positive, negative, and neutral. This study aims to determine public attitudes towards each topic of tweets about COVID-19 vaccines.

Dimensional Reduction Techniques for Huge Volume of Data

Abstract: Huge volume of data and information is needed with the expanding advancement in the current collection of tools, cloud storage, strategic techniques and increasing development of science technology. With the appearance of complete genome successions, the biomedical area has encountered an exceptional progression. This genomics has prompted the advancement of new high-produced strategies techniques that are huge amounts in measures of information and data, which inferred the exponential development of numerous organic and biological data sets. This paper represents different linear and non-linear dimensionality reduction techniques and their validity for different kinds of data information datasets and application regions. Keywords: High dimensional data, Dimensionality reduction, Linear techniques, Non-linear techniques, feature extraction, feature selection, Machine Learning

Potential of functional analysis applied to Sentinel-2 time-series to assess relevant agronomic parameters at the within-field level in viticulture

• Functional analysis methods were able to decompose the yearly trend and spatial structure of the vineyards. • Performance of PCA declines if more than three componets are added. • The results are linked to soil characteristics and other key variables, highlighting several phenomena affecting the vineyard. Sentinel-2 satellite imagery offers a wealth of spectral information combined with a weekly temporal resolution. It is seen as a promising tool to extract spatial information about vineyards and link them to agronomic parameters. Usually, only one or a few images are commonly employed at specific stages like veraison in viticulture. Extracting further information from time-series images may be of interest; however, this remains an issue due to the noisy and complex nature of extracted time-series. The functional analysis proposes a robust continuous representation of these time-series, which can then be used with adapted statistical tools. This paper focuses on extracting relevant information at the within-field level on two vineyards in Spain, which can be jointly interpreted with field observations and measurements. More precisely, it discusses the use of popular linear dimensionality reduction techniques, namely Principal Component Analysis (PCA) and Partial Least Square (PLS), adapted to functional data in order to decompose NDVI time-series into a weighted sum of several functional components. The unsupervised methods, like PCA, decomposed the spatial structure within the vineyards using a few components, resulting in a better and more manageable dataset than the one obtained using simple non-constrained methods. The results show significant correlations with ground-truth data showing the added value of considering the whole NDVI temporal series compared to a single NDVI map at veraison. The proposed approach provided helpful information about each component's yearly trend. Moreover, the results are linked to grapevines' seasonal phenology and management practices, highlighting phenomena affecting the vineyard's development. This method is particularly suited for interactions with field experts, who may derive relevant agronomic information from the decomposition maps.

Dimensionality Reduction and Louvain Agglomerative Hierarchical Clustering for Cluster-Specified Frequent Biomarker Discovery in Single-Cell Sequencing Data.

The major interest domains of single-cell RNA sequential analysis are identification of existing and novel types of cells, depiction of cells, cell fate prediction, classification of several types of tumor, and investigation of heterogeneity in different cells. Single-cell clustering plays an important role to solve the aforementioned questions of interest. Cluster identification in high dimensional single-cell sequencing data faces some challenges due to its nature. Dimensionality reduction models can solve the problem. Here, we introduce a potential cluster specified frequent biomarkers discovery framework using dimensionality reduction and hierarchical agglomerative clustering Louvain for single-cell RNA sequencing data analysis. First, we pre-filtered the features with fewer number of cells and the cells with fewer number of features. Then we created a Seurat object to store data and analysis together and used quality control metrics to discard low quality or dying cells. Afterwards we applied global-scaling normalization method “LogNormalize” for data normalization. Next, we computed cell-to-cell highly variable features from our dataset. Then, we applied a linear transformation and linear dimensionality reduction technique, Principal Component Analysis (PCA) to project high dimensional data to an optimal low-dimensional space. After identifying fifty “significant”principal components (PCs) based on strong enrichment of low p-value features, we implemented a graph-based clustering algorithm Louvain for the cell clustering of 10 top significant PCs. We applied our model to a single-cell RNA sequential dataset for a rare intestinal cell type in mice (NCBI accession ID:GSE62270, 23,630 features and 1872 samples (cells)). We obtained 10 cell clusters with a maximum modularity of 0.885 1. After detecting the cell clusters, we found 3871 cluster-specific biomarkers using an expression feature extraction statistical tool for single-cell sequencing data, Model-based Analysis of Single-cell Transcriptomics (MAST) with a log 2 FC threshold of 0.25 and a minimum feature detection of 25%. From these cluster-specific biomarkers, we found 1892 most frequent markers, i.e., overlapping biomarkers. We performed degree hub gene network analysis using Cytoscape and reported the five highest degree genes (Rps4x, Rps18, Rpl13a, Rps12 and Rpl18a). Subsequently, we performed KEGG pathway and Gene Ontology enrichment analysis of cluster markers using David 6.8 software tool. In summary, our proposed framework that integrated dimensionality reduction and agglomerative hierarchical clustering provides a robust approach to efficiently discover cluster-specific frequent biomarkers, i.e., overlapping biomarkers from single-cell RNA sequencing data.

Quantum algorithm for neighborhood preserving embedding

Neighborhood preserving embedding (NPE) is an important linear dimensionality reduction technique that aims at preserving the local manifold structure. NPE contains three steps, i.e., finding the nearest neighbors of each data point, constructing the weight matrix, and obtaining the transformation matrix. Liang et al. proposed a variational quantum algorithm (VQA) for NPE [Phys. Rev. A 101 032323 (2020)]. The algorithm consists of three quantum sub-algorithms, corresponding to the three steps of NPE, and was expected to have an exponential speedup on the dimensionality n. However, the algorithm has two disadvantages: (i) It is not known how to efficiently obtain the input of the third sub-algorithm from the output of the second one. (ii) Its complexity cannot be rigorously analyzed because the third sub-algorithm in it is a VQA. In this paper, we propose a complete quantum algorithm for NPE, in which we redesign the three sub-algorithms and give a rigorous complexity analysis. It is shown that our algorithm can achieve a polynomial speedup on the number of data points m and an exponential speedup on the dimensionality n under certain conditions over the classical NPE algorithm, and achieve a significant speedup compared to Liang et al.’s algorithm even without considering the complexity of the VQA.

Comparison of dimension reduction techniques applied to the analysis of airborne radionuclide activity concentration

Temporal variation of airborne radionuclide activity concentration is highly influenced by several meteorological parameters. A traditional key tool to perform a combined analysis on these data is Principal Component Analysis (PCA), a linear dimensionality reduction technique that prioritizes the conservation of the data set global structure. While it reveals important information regarding the correlation among the considered variables, the obtained visual representations do not usually allow to clearly discern different clusters of states with common properties. The main goal of this study is applying two recently introduced non linear dimensionality reduction techniques, t-distributed Stochastic Neighbor Embedding (t-SNE) and Uniform Manifold Approximation and Projection (UMAP) to a data set composed by 7Be and gross beta (Aβ) activity concentration and other meteorological data gathered in Mallorca (Spain) between 2004 and 2014. Compared to PCA, both algorithms reveal more details on the local structure of the data set. UMAP allows to clearly identify data clusters with different characteristics that are not clearly identified with the alternative techniques.

Maternal urinary phthalate metabolites are associated with lipidomic signatures among pregnant women in Puerto Rico.

Background:Phthalates have been reported to alter circulating lipid concentrations in animals, and investigation of these associations in humans will provide greater understanding of potential mechanisms for health outcomes.Objective:to explore associations between phthalate metabolite biomarkers and lipidomic profiles among pregnant women (n = 99) in the Puerto Rico PROTECT cohort.Methods:We measured 19 urinary phthalate metabolites during 24–28 weeks of pregnancy. Lipidomic profiles were identified from plasma samples by liquid chromatography-mass spectrometry-based shotgun lipidomics. Relationships between phthalate metabolites and lipid profiles were estimated using compound-by-compound comparisons in multiple linear regression and dimension reduction techniques. We derived sums for each lipid class and sub-class (saturated, monounsaturated, polyunsaturated) which were then regressed on phthalate metabolites. Associations were adjusted for false discovery.Results:After controlling for multiple comparisons, 33 phthalate-lipid associations were identified (False discovery rate adjusted p-value<0.05), and diacylglycerol 40:7 and plasmenyl-phosphatidylcholine 35:1 were the most strongly associated with multiple phthalate metabolites. Metabolites of di-2-ethylhexyl phthalate, bis(2-ethylhexyl) phthalate, dibutyl phthalates, and diisobutyl phthalate were associated with increased ceramides, lysophosphatidylcholines, lysophosphatidylethanolamines, and triacylglycerols, particularly those containing saturated and monounsaturated fatty acid chains.Significance:Characterization of associations between lipidomic markers and phthalate metabolites during pregnancy will yield mechanistic insight for maternal and child health outcomes.

Recovery of linear components: Reduced complexity autoencoder designs

Reducing dimensionality is a key preprocessing step in many data analysis applications to address the negative effects of the curse of dimensionality and collinearity on model performance and computational complexity, to denoise the data or to reduce storage requirements. Moreover, in many applications it is desirable to reduce the input dimensions by choosing a subset of variables that best represents the entire set without any a priori information available. Unsupervised variable selection techniques provide a solution to this second problem. An autoencoder, if properly regularized, can solve both unsupervised dimensionality reduction and variable selection, but the training of large neural networks can be prohibitive in time sensitive applications. We present an approach called Recovery of Linear Components (RLC), which serves as a middle ground between linear and non-linear dimensionality reduction techniques, reducing autoencoder training times while enhancing performance over purely linear techniques. With the aid of synthetic and real world case studies, we show that the RLC, when compared with an autoencoder of similar complexity, shows higher accuracy, similar robustness to overfitting, and faster training times. Additionally, at the cost of a relatively small increase in computational complexity, RLC is shown to outperform the current state-of-the-art for a semiconductor manufacturing wafer measurement site optimization application.

Methods for Measurement and Analysis of Full Hand Angular Kinematics Using Electromagnetic Tracking Sensors

The human hand needs a large number of sensors to measure kinematics owing to its large number of degrees of freedom. Existing devices like data gloves and optical trackers are associated with calibration, line of sight, and accuracy problems. In this paper, we attempt to measure the full hand kinematics using Electromagnetic Tracking Sensors (EMTS) which are accurate, free of line-of-sight problems, and require no calibration. However, EMTS provides output in the form of rotation groups which are defined on a nonlinear manifold. Hence, linear operations required for experimental analysis such as linear dimensionality reduction are not valid. Also, these sensors are expensive, utilize space in terms of cabling, and require a reduced sensor layout. In this paper, we present measurement methods, test the utility of linear and non-linear dimensionality reduction techniques on quaternions and exponential maps. We also performed sensor reduction using a Gini feature selection based on random forest algorithm. The kinematic measurement results show that EMTS yield superior posture reproduction with an error of less than 1 degree. Autoencoder, a nonlinear dimensionality reduction technique, was successfully applied on quaternions which was tuned to perform better than Principal Component Analysis (PCA) in reducing dimensions. The reduced sensor layout with 8 sensors was able to predict full hand kinematics with a Root Mean Square Error (RMSE) of 5.1 degrees.

Supervised dimensionality reduction for big data

To solve key biomedical problems, experimentalists now routinely measure millions or billions of features (dimensions) per sample, with the hope that data science techniques will be able to build accurate data-driven inferences. Because sample sizes are typically orders of magnitude smaller than the dimensionality of these data, valid inferences require finding a low-dimensional representation that preserves the discriminating information (e.g., whether the individual suffers from a particular disease). There is a lack of interpretable supervised dimensionality reduction methods that scale to millions of dimensions with strong statistical theoretical guarantees. We introduce an approach to extending principal components analysis by incorporating class-conditional moment estimates into the low-dimensional projection. The simplest version, Linear Optimal Low-rank projection, incorporates the class-conditional means. We prove, and substantiate with both synthetic and real data benchmarks, that Linear Optimal Low-Rank Projection and its generalizations lead to improved data representations for subsequent classification, while maintaining computational efficiency and scalability. Using multiple brain imaging datasets consisting of more than 150 million features, and several genomics datasets with more than 500,000 features, Linear Optimal Low-Rank Projection outperforms other scalable linear dimensionality reduction techniques in terms of accuracy, while only requiring a few minutes on a standard desktop computer.