Conventional Dimensionality Reduction Research Articles

Remote sensing framework for geological mapping via stacked autoencoders and clustering

Supervised machine learning methods for geological mapping via remote sensing face limitations due to the scarcity of accurately labelled training data that can be addressed by unsupervised learning, such as dimensionality reduction and clustering. Dimensionality reduction methods have the potential to play a crucial role in improving the accuracy of geological maps. Although conventional dimensionality reduction methods may struggle with nonlinear data, unsupervised deep learning models such as autoencoders can model non-linear relationships. Stacked autoencoders feature multiple interconnected layers to capture hierarchical data representations useful for remote sensing data. This study presents an unsupervised machine learning-based framework for processing remote sensing data using stacked autoencoders for dimensionality reduction and k-means clustering for mapping geological units. We use Landsat 8, ASTER, and Sentinel-2 datasets to evaluate the framework for geological mapping of the Mutawintji region in Western New South Wales, Australia. We also compare stacked autoencoders with principal component analysis and canonical autoencoders. Our results reveal that the framework produces accurate and interpretable geological maps, efficiently discriminating rock units. We find that the accuracy of stacked autoencoders ranges from 86.6% to 90%, depending on the remote sensing data type, which is superior to their counterparts. We also find that the generated maps align with prior geological knowledge of the study area while providing novel insights into geological structures.

Dimension Reduction With Prior Information for Knowledge Discovery.

This paper addresses the problem of mapping high-dimensional data to a low-dimensional space, in the presence of other known features. This problem is ubiquitous in science and engineering as there are often controllable/measurable features in most applications. To solve this problem, this paper proposes a broad class of methods, which is referred to as conditional multidimensional scaling (MDS). An algorithm for optimizing the objective function of conditional MDS is also developed. The convergence of this algorithm is proven under mild assumptions. Conditional MDS is illustrated with kinship terms, facial expressions, textile fabrics, car-brand perception, and cylinder machining examples. These examples demonstrate the advantages of conditional MDS over conventional dimension reduction in improving the estimation quality of the reduced-dimension space and simplifying visualization and knowledge discovery tasks. Computer codes for this work are available in the open-source cml R package.

Improving Ultrasonic Testing by Using Machine Learning Framework Based on Model Interpretation Strategy

Ultrasonic testing (UT) is increasingly combined with machine learning (ML) techniques for intelligently identifying damage. Extracting significant features from UT data is essential for efficient defect characterization. Moreover, the hidden physics behind ML is unexplained, reducing the generalization capability and versatility of ML methods in UT. In this paper, a generally applicable ML framework based on the model interpretation strategy is proposed to improve the detection accuracy and computational efficiency of UT. Firstly, multi-domain features are extracted from the UT signals with signal processing techniques to construct an initial feature space. Subsequently, a feature selection method based on model interpretable strategy (FS-MIS) is innovatively developed by integrating Shapley additive explanation (SHAP), filter method, embedded method and wrapper method. The most effective ML model and the optimal feature subset with better correlation to the target defects are determined self-adaptively. The proposed framework is validated by identifying and locating side-drilled holes (SDHs) with 0.5λ central distance and different depths. An ultrasonic array probe is adopted to acquire FMC datasets from several aluminum alloy specimens containing two SDHs by experiments. The optimal feature subset selected by FS-MIS is set as the input of the chosen ML model to train and predict the times of arrival (ToAs) of the scattered waves emitted by adjacent SDHs. The experimental results demonstrate that the relative errors of the predicted ToAs are all below 3.67% with an average error of 0.25%, significantly improving the time resolution of UT signals. On this basis, the predicted ToAs are assigned to the corresponding original signals for decoupling overlapped pulse-echoes and reconstructing high-resolution FMC datasets. The imaging resolution is enhanced to 0.5λ by implementing the total focusing method (TFM). The relative errors of hole depths and central distance are no more than 0.51% and 3.57%, respectively. Finally, the superior performance of the proposed FS-MIS is validated by comparing it with initial feature space and conventional dimensionality reduction techniques.

Smart manufacturing under limited and heterogeneous data: a sim-to-real transfer learning with convolutional variational autoencoder in thermoforming

ABSTRACT Data in advanced manufacturing are often sparse and collected from various sensory devices in a heterogeneous and multi-modal fashion. Thus, for such intricate input spaces, learning robust and reliable predictive models for product quality assessments entails implementing complex nonlinear models such as deep learning. However, these ‘data-greedy’ models require massive datasets for training, and they tend to exhibit poor generalization performance otherwise. To address the data paucity and the data heterogeneity in smart manufacturing applications, this paper introduces a sim-to-real transfer-learning framework. Specifically, using a unified wide-and-deep learning approach, the model pre-processes structured sensory data (wide) as well as high-dimensional thermal images (deep) separately, and then passes the respective concatenated features to a regressor for predicting product quality metrics. Convolutional variational autoencoder (ConvVAE) is utilized to learn concise representations of thermal images in an unsupervised fashion. ConvVAE is trained via a sim-to-real transfer learning approach, backed by theory-based heat transfer simulations. The proposed metamodeling framework was evaluated in an industrial thermoforming process case study. The results suggested that ConvVAE outperforms conventional dimensionality reduction methods despite limited data. A model explainability analysis was conducted and the resulting SHAP values demonstrated the agreement between the model’s predictions, theoretical expectations, and data correlation statistics.

Orthogonal outlier detection and dimension estimation for improved MDS embedding of biological datasets.

Conventional dimensionality reduction methods like Multidimensional Scaling (MDS) are sensitive to the presence of orthogonal outliers, leading to significant defects in the embedding. We introduce a robust MDS method, called DeCOr-MDS (Detection and Correction of Orthogonal outliers using MDS), based on the geometry and statistics of simplices formed by data points, that allows to detect orthogonal outliers and subsequently reduce dimensionality. We validate our methods using synthetic datasets, and further show how it can be applied to a variety of large real biological datasets, including cancer image cell data, human microbiome project data and single cell RNA sequencing data, to address the task of data cleaning and visualization.

Dimensionality reduction and ensemble of LSTMs for antimicrobial resistance prediction

Bacterial resistance to antibiotics has been rapidly increasing, resulting in low antibiotic effectiveness even treating common infections. The presence of resistant pathogens in environments such as a hospital Intensive Care Unit (ICU) exacerbates the critical admission-acquired infections. This work focuses on the prediction of antibiotic resistance in Pseudomonas aeruginosa nosocomial infections at the ICU, using Long Short-Term Memory (LSTM) artificial neural networks as the predictive method. The analyzed data were extracted from the Electronic Health Records (EHR) of patients admitted to the University Hospital of Fuenlabrada from 2004 to 2019 and were modeled as Multivariate Time Series. A data-driven dimensionality reduction method is built by adapting three feature importance techniques from the literature to the considered data and proposing an algorithm for selecting the most appropriate number of features. This is done using LSTM sequential capabilities so that the temporal aspect of features is taken into account. Furthermore, an ensemble of LSTMs is used to reduce the variance in performance. Our results indicate that the patient’s admission information, the antibiotics administered during the ICU stay, and the previous antimicrobial resistance are the most important risk factors. Compared to other conventional dimensionality reduction schemes, our approach is able to improve performance while reducing the number of features for most of the experiments. In essence, the proposed framework achieve, in a computationally cost-efficient manner, promising results for supporting decisions in this clinical task, characterized by high dimensionality, data scarcity, and concept drift.

Shape-supervised Dimension Reduction: Extracting Geometry and Physics Associated Features with Geometric Moments

In shape optimisation problems, subspaces generated with conventional dimension reduction approaches often fail to extract the intrinsic geometric features of the shape that would allow the exploration of diverse but valid candidate solutions. More importantly, they also lack incorporation of any notion of physics against which shape is optimised. This work proposes a shape-supervised dimension reduction approach. To simultaneously tackle these deficiencies, it uses higher-level information about the shape in terms of its geometric integral properties, such as geometric moments and their invariants. Their usage is based on the fact that moments of a shape are intrinsic features of its geometry, and they provide a unifying medium between geometry and physics. To enrich the subspace with latent features associated with shape’s geometrical features and physics, we also evaluate a set of composite geometric moments, using the divergence theorem, for appropriate shape decomposition. These moments are combined with the shape modification function to form a Shape Signature Vector (SSV) uniquely representing a shape. Afterwards, the generalised Karhunen–Loève expansion is applied to SSV, embedded in a generalised (disjoint) Hilbert space, which results in a basis of the shape-supervised subspace retaining the highest geometric and physical variance. Validation experiments are performed for a three-dimensional wing and a ship hull model. Our results demonstrate a significant reduction of the original design space’s dimensionality for both test cases while maintaining a high representation capacity and a large percentage of valid geometries that facilitate fast convergence to the optimal solution. The code developed to implement this approach is available at https://github.com/shahrozkhan66/SSDR.git . • Shape signature vector (SSV) comprises a shape modification vector and geometric moments (GM). • Karhunen–Loève expansion of SSV results in a shape-supervised space (SSS). • GM can induce the notion of physics resulting in a physics-informed SSS. • SSS retains higher geometric variance with fewer latent variables. • SSS is robust and compact, providing accelerated optimisation convergence.

Learning target class eigen subspace (LTC-ES) via eigen knowledge grid

In one-class classification (OCC) tasks, only the target class (class-of-interest (CoI)) samples are well defined during training, whereas the other class samples are totally absent. In OCC algorithms, the high dimensional data adds computational overhead apart from its intrinsic property of curse of dimensionality. For target class learning, conventional dimensionality reduction (DR) techniques are not suitable due to negligence of the unique statistical properties of CoI samples. In this context, the present research proposes a novel target class guided DR technique to extract the eigen knowledge grid that contains the most promising eigenvectors of variance-covariance matrix of CoI samples. In this process the lower and higher eigenvalued eigenvectors are rejected via statistical analysis because the high variance may split the target class itself, whereas the lower variance do not contribute significant information. Furthermore, the identified eigen knowledge grid is utilized to transform high dimensional samples to the lower dimensional eigen subspace. The proposed approach is named as learning target class eigen subspace (LTS-ES) that ensures strong separation of the target class from other classes. To show the effectiveness of transformed lower dimensional eigen subspace, one class support vector machine (OCSVM) has been experimented on wide variety of benchmark datasets in presence of: original feature space, transformed features obtained via eigenvectors of approximately 80%-90% cumulative variance, transformed features obtained via knowledge grid and transformed features obtained via eigenvectors of approximately 50% cumulative variance. Finally, a new performance measure parameter called stability factor is introduced to validate the robustness of the proposed approach.

Joint dimension reduction and clustering analysis of single-cell RNA-seq and spatial transcriptomics data.

Dimension reduction and (spatial) clustering is usually performed sequentially; however, the low-dimensional embeddings estimated in the dimension-reduction step may not be relevant to the class labels inferred in the clustering step. We therefore developed a computation method, Dimension-Reduction Spatial-Clustering (DR-SC), that can simultaneously perform dimension reduction and (spatial) clustering within a unified framework. Joint analysis by DR-SC produces accurate (spatial) clustering results and ensures the effective extraction of biologically informative low-dimensional features. DR-SC is applicable to spatial clustering in spatial transcriptomics that characterizes the spatial organization of the tissue by segregating it into multiple tissue structures. Here, DR-SC relies on a latent hidden Markov random field model to encourage the spatial smoothness of the detected spatial cluster boundaries. Underlying DR-SC is an efficient expectation-maximization algorithm based on an iterative conditional mode. As such, DR-SC is scalable to large sample sizes and can optimize the spatial smoothness parameter in a data-driven manner. With comprehensive simulations and real data applications, we show that DR-SC outperforms existing clustering and spatial clustering methods: it extracts more biologically relevant features than conventional dimension reduction methods, improves clustering performance, and offers improved trajectory inference and visualization for downstream trajectory inference analyses.

Single-cell manifold-preserving feature selection for detecting rare cell populations.

A key challenge in studying organisms and diseases is to detect rare molecular programs and rare cell populations (RCPs) that drive development, differentiation, and transformation. Molecular features such as genes and proteins defining RCPs are often unknown and difficult to detect from unenriched single-cell data, using conventional dimensionality reduction and clustering-based approaches. Here, we propose an unsupervised approach, SCMER (Single-Cell Manifold presERving feature selection), which selects a compact set of molecular features with definitive meanings that preserve the manifold of the data. We applied SCMER in the context of hematopoiesis, lymphogenesis, tumorigenesis, and drug resistance and response. We found that SCMER can identify non-redundant features that sensitively delineate both common cell lineages and rare cellular states. SCMER can be used for discovering molecular features in a high dimensional dataset, designing targeted, cost-effective assays for clinical applications, and facilitating multi-modality integration.

Lightweight CNNs-Based Interleaved Sparse Array Design of Phased-MIMO Radar

Conventional transmit subarray partitioning schemes of phased-MIMO radar will cause several problems, the reduction of array aperture, the increase of feeding network complexity and optimization algorithms time cost. Focus on the problems above, this paper proposes a deep learning-based interleaved sparse transmit subarray partitioning method. Firstly, the training data is generated by phased-MIMO array manifold matrix. Secondly, a dimensionality reduction method for radar data is introduced to reduce the dimensionality of the sample data while minimizing information loss. Then, a lightweight convolutional neural network is constructed for training and the optimal array structures is selected by classification. Finally, linear and plane arrays experiment results show that our proposed method can achieve 97.95% classification accuracy, better than other conventional dimensionality reduction methods and neural networks; compared with the traditional SCP partitioning method, our proposed method has similar beampattern sidelobe level and DOA estimation accuracy, but the time cost is greatly reduced.

Linear and Deep Order-Preserving Wasserstein Discriminant Analysis

Supervised dimensionality reduction for sequence data learns a transformation that maps the observations in sequences onto a low-dimensional subspace by maximizing the separability of sequences in different classes. It is typically more challenging than conventional dimensionality reduction for static data, because measuring the separability of sequences involves non-linear procedures to manipulate the temporal structures. In this paper, we propose a linear method, called order-preserving Wasserstein discriminant analysis (OWDA), and its deep extension, namely DeepOWDA, to learn linear and non-linear discriminative subspace for sequence data, respectively. We construct novel separability measures between sequence classes based on the order-preserving Wasserstein (OPW) distance to capture the essential differences among their temporal structures. Specifically, for each class, we extract the OPW barycenter and construct the intra-class scatter as the dispersion of the training sequences around the barycenter. The inter-class distance is measured as the OPW distance between the corresponding barycenters. We learn the linear and non-linear transformations by maximizing the inter-class distance and minimizing the intra-class scatter. In this way, the proposed OWDA and DeepOWDA are able to concentrate on the distinctive differences among classes by lifting the geometric relations with temporal constraints. Experiments on four 3D action recognition datasets show the effectiveness of OWDA and DeepOWDA.

Handwriting digit classification using PCA-transformed image features

Machine learning model is capable of achieving an incredible great accuracy by training a large amount of data with many dimensions of features in it. To accomplish such a goal, we are currently in desperate need of powerful computers with CPUs (Computation Processing Unit) and GPUs (Graphic Processing Unit) that are competent of the workload of the training process of those machine learning models. Due to the problem that there are multiple disparate computationally intensive tasks that are lengthy period of time during training phase of machine learning, devices like smartphone and personal computer cannot achieve great efficiency during the training phase of machine learning. Instead of enhancing computational performance of devices and improve the machine learning algorithm, another more feasible way to bypass such problem is to apply dimensionality reduction methods on raw data in hope of reducing the number of features that are passing into the machine learning model during the training phase. By doing so, we can shrink the originally large number of features to several dimensions that is accomplishable for our device to compute and meanwhile, sufficient enough to have a model with acceptable accuracy trained. The purpose of this study to explore and investigate the potential benefits and disadvantages of dimensionality reduction on multiple distinct machine learning algorithm by comparing the performances of each machine learning algorithms when passing the features of raw datasets in and when passing the dataset with the features that survived after dimensionality reduction. The performance will be measured by completing a typical machine learning task: handwriting digit classification. Among all the dimensionalities methods that have been discovered, a traditional and conventional dimensionality reduction method, PCA (Principal Components Analysis), is selected. Such method has been proven successful and efficient in generating the set of data with lower dimension features and apply on an image classification task using several supervised machine learning methods, including KNN (k-nearest neighbors), SVM (Support vector machine) and CNN (Convolutional Neural Network).

An efficient variance-based global sensitivity index based on C-DRM and Taylor expansion

Sensitivity analysis plays an important role in quantifying the impact of input uncertainty on model response uncertainty. Through sensitivity analysis, we can grasp the crucial parameters and gain a deeper understanding of the model behavior. In this paper, an effective analytical solution for solving the variance-based global sensitivity index is proposed. Firstly, the original performance function is approximated as the sum of a series of univariate functions using the conventional dimensional reduction method (C-DRM). Then, the Taylor series expansion is used to expand the univariate function as unary linear function and unary quadratic functions. Finally, the analytical solutions of the variance-based global sensitivity index based on unary linear function and unary quadratic functions are derived respectively. The computational cost of the proposed method is completely concentrated on the calculation of the partial derivative information of the performance function with respect to each variable. As long as the partial derivative information is obtained, the variance-based global sensitivity index can be obtained directly by the proposed method without any additional computational cost. For simple explicit performance functions, the derivative information can be directly derived analytically. For complex explicit or implicit performance functions, the derivative information can be estimated by some simple numerical difference methods. Five examples are studied to investigate the accuracy and efficiency of the proposed method.

Guided autoencoder for dimensionality reduction of pedestrian features

Autoencoder and other conventional dimensionality reduction algorithms have achieved great success in dimensionality reduction. In this paper, we present an improved autoencoder structure, which was applied it in the field of pedestrian feature dimensionality reduction. The novel method is also verified on Mnist dataset. High-dimensional deep pedestrian features outperform other descriptors while it is challenging for computing capability and memory in existing systems. The dimensionality reduction method we proposed takes advantages of autoencoder and principal component analysis to achieve high efficiency. A novel weight matrix initialization and an improved reconstruction of autoencoder are proposed. Furthermore, by fusing features labeled with the same pedestrian, the proposed structure minimizes the loss after dimensionality reduction. Experimental results demonstrate that our method outperforms traditional dimensionality reduction methods. In the experiment, the pedestrian features were generated by ResNet and Market-1501 data-set. Our method achieves up to 8.834% mAP increment compared to a principal component analysis, when 2048-dimension pedestrian features are reduced to 16-dimension features.

Joint Spatial Geometric and Max-margin Classifier Constraints for Facial Expression Recognition Using Nonnegative Matrix Factorization

Based on the constrained non-negative matrix factor algorithm, the article presents a new approach to facial recognition recognition. Our proposed method incorporated two tasks in an automatic expression analysis system: facial feature extraction and classification into expressions. To obtain local and geometric structure information in the data as much as possible, we amalgamate max-margin relegation into the constrained NMF optimization, resulting in a multiplicative updating algorithm is additionally proposed for solving optimization quandary. Experimental results on JAFFE dataset demonstrate that the effectiveness of the proposed method with improved performances over the conventional dimension reduction methods.

An Incremental Dimensionality Reduction Method for Visualizing Streaming Multidimensional Data.

Dimensionality reduction (DR) methods are commonly used for analyzing and visualizing multidimensional data. However, when data is a live streaming feed, conventional DR methods cannot be directly used because of their computational complexity and inability to preserve the projected data positions at previous time points. In addition, the problem becomes even more challenging when the dynamic data records have a varying number of dimensions as often found in real-world applications. This paper presents an incremental DR solution. We enhance an existing incremental PCA method in several ways to ensure its usability for visualizing streaming multidimensional data. First, we use geometric transformation and animation methods to help preserve a viewer's mental map when visualizing the incremental results. Second, to handle data dimension variants, we use an optimization method to estimate the projected data positions, and also convey the resulting uncertainty in the visualization. We demonstrate the effectiveness of our design with two case studies using real-world datasets.

Improved PPG-based estimation of the blood pressure using latent space features

Accurate and uninterrupted estimation of the blood pressure is essential for continuous monitoring of patients. We estimate the blood pressure by extracting 21 time parameters from the photoplethysmography signal. The major novelties of this paper include: (1) using a nonlinear mapping to reduce the size of the feature vector and to map the input parameters to a latent space instead of conventional dimensionality reduction schemes, (2) employing a multi-stage noise reduction technique to effectively smooth the input signal. Estimation of the blood pressures is performed by a support vector regressor. The mean absolute errors of our results are 1.21 mmHg and 0.80 mmHg for systolic and diastolic blood pressures, respectively, which are lower than recent researches.

Estimating gene expression from high-dimensional DNA methylation levels in cancer data: A bimodal unsupervised dimension reduction algorithm

Recent molecular and genetic studies have revealed the importance of DNA methylation, a key epigenetic mark, in regulating gene expression and the abnormal profiles of DNA methylation in various diseases including cancer. Here, unsupervised learning methods that are geared towards high-throughput DNA methylation analysis are used to extract useful information from high-dimensional genome wide methylation data in order to provide crucial insights for accurate early diagnosis and treatment of cancer. Herein, these methods are highly dependent on the performance of an earlier step of dimension reduction that aims to find the best subset of attributes to be retained for learning. Widely used algorithms in the literature commonly suffer from resulting in trivial cluster structures and failing to shed light on the relationship between DNA methylation and cancer types due to their myopic and arbitrary search mechanisms. Addressing this issue, we introduce a bimodal unsupervised dimension reduction algorithm (BOUNDER) that identifies the best subset of loci for downstream analysis considering the variability and redundancy across all the samples using bimodal modeling before it feeds into the learning method. BOUNDER models each locus as a bimodal representation using a piecewise linear function with two segments and filters the informative loci based on the fitted line characteristics. To the best of our knowledge, the work presented here is the first study that uses bimodal modeling in unsupervised learning in DNA methylation analysis. BOUNDER is tailored for DNA methylation analysis using a detailed parameter tuning analysis. The performance of BOUNDER is benchmarked against those of widely used conventional algorithms using real lung, breast, kidney, and urological cancer datasets obtained from Gene Expression Omnibus in terms of their accuracies in hierarchical clustering and k-means clustering. Computational experiments reveal that BOUNDER outperforms the PCA and filtering based approach by providing the highest accuracy in 6 out of 9 datasets while providing more interpretable results through a correlation analysis. The BOUNDER algorithm is also shown to be more robust when compared to multiple other conventional dimension reduction algorithms across different datasets.

Textual data dimensionality reduction - a deep learning approach

The growth of Internet has produced a high volume of natural language textual data. Such data can be sparse and may contain uninformative features which increase the dimensions of the data. This high dimensionality in turn, decreases the efficiency of text mining tasks such as clustering. Transforming the high dimensional data into a lower dimension is an important pre-processing step before applying clustering. In this paper, dimensionality reduction method based on deep Autoencoder neural network named as DRDAE, is proposed to provide optimized and robust features for text clustering. DRDAE selects less correlated and salient feature space from the high dimensional feature space. To evaluate proposed algorithm, k-means is used to cluster text documents. The proposed method is tested on five benchmark text datasets. Simulation results demonstrate that the proposed algorithm clearly outperforms other conventional dimensionality reduction methods in the literature in terms of RI measure.