High Dimensional Data Reduction Research Articles

CellTracksColab is a platform that enables compilation, analysis, and exploration of cell tracking data.

In life sciences, tracking objects from movies enables researchers to quantify the behavior of single particles, organelles, bacteria, cells, and even whole animals. While numerous tools now allow automated tracking from video, a significant challenge persists in compiling, analyzing, and exploring the large datasets generated by these approaches. Here, we introduce CellTracksColab, a platform tailored to simplify the exploration and analysis of cell tracking data. CellTracksColab facilitates the compiling and analysis of results across multiple fields of view, conditions, and repeats, ensuring a holistic dataset overview. CellTracksColab also harnesses the power of high-dimensional data reduction and clustering, enabling researchers to identify distinct behavioral patterns and trends without bias. Finally, CellTracksColab also includes specialized analysis modules enabling spatial analyses (clustering, proximity to specific regions of interest). We demonstrate CellTracksColab capabilities with 3 use cases, including T cells and cancer cell migration, as well as filopodia dynamics. CellTracksColab is available for the broader scientific community at https://github.com/CellMigrationLab/CellTracksColab.

Correlation Between Scientific and Technological Innovation and Financial Performance of Seed Enterprises: Application of Principal Component Analysis Algorithm in Performance Evaluation

This study explores the relationship between scientific and technological innovations and the financial performance of startups, focusing on a representative Artificial Intelligence (AI) healthcare startup. Utilizing Principal Component Analysis (PCA), the research aims to dissect the complex interplay between innovation-related metrics—such as R&D spending, patent counts, and technology adoption rates—and financial outcomes like revenue growth, profitability, and market share. The PCA methodology enabled the reduction of high-dimensional data into PCA, which clearly illustrates how various dimensions of innovation impact financial metrics. The approach used in the investigation helps people comprehend more thoroughly how development benefits business viability in several different manners and provides startups with practical advice to use when preparing their revolutionary investments. The objective of the research is to assist the ecosystem of startup consumers (including investors, business owners, and policymakers) in making better decisions that balance technological progress with economic objectives.

Manifold learning-based UMAP method for geochemical anomaly identification

Geochemical data are usually high-dimensional data that could contain dozens of elements. Geochemical distribution patterns and anomalies related to mineralization and lithological features are always hidden in these high-dimensional data, which cannot be directly observed from the data. To solve this problem, a manifold learning-based uniform manifold approximation and projection (UMAP) method, was introduced to recognize mineralization-related geochemical anomalies from high-dimensional geochemical data in this study. The UMAP method is a nonlinear dimensionality reduction method, which is suitable for dimensionality reduction and visualization of high-dimensional data. A case study was conducted to demonstrate the advantages of the UMAP method for identifying ion-adsorbed rare-earth-element (REE) mineralization-related anomalies from high-dimensional data in the Nanling region, China. Factor analysis was used to determine ion-adsorbed REE mineralization-related element combination that consists of 10 elements. High-dimensional geochemical data were reduced to two dimensions based on the UMAP method. The results indicated that the UMAP method can effectively characterize the spatial distributions of ion-adsorbed REE mineralization-related anomalies by dimensionality reduction analysis and visualization analysis of high-dimensional geochemical data in the study area. To illustrate the superiority of the UMAP method, a comparative study was conducted between the UMAP and other three manifold learning methods, namely locally linear embedding (LLE), isometric feature mapping (Isomap) and t-distributed stochastic neighbor embedding (t-SNE). The performance of the four manifold learning methods was evaluated by receiver operating characteristic (ROC) curve and prediction-area (P-A) plot, showing that the performance of the UMAP method is superior to that of the LLE, Isomap and t-SNE methods in terms of recognizing ion-adsorbed REE mineralization-related anomalies and the spatial distributions of the REE-bearing geological bodies in the Nanling belt.

Time dimension feature extraction and classification of high-dimensional large data streams based on unsupervised learning

In order to solve the problem of low accuracy of time dimension feature extraction and classification of high-dimensional large data streams, this paper proposes a time dimension feature extraction and classification algorithm of high-dimensional large data streams based on unsupervised learning. Analyze the trend of high-dimensional data flow changes under machine learning, and achieve dimensionality reduction of high-dimensional large traffic time dimensional data through local save projection. Analyze the spatial relationship between feature attributes and feature space, segment and fit high-dimensional big data streams and time dimensional feature data streams, further segment time dimensional sequences using sliding windows, and complete feature extraction through discrete dyadic wavelet transform. According to the clustering algorithm, cluster the time dimension feature data stream, calculate the cosine similarity of the feature data, model the time dimension feature stream of training samples, use the feature classification function to minimize the classification loss, and use unsupervised learning to achieve the final classification task. The test results show that this method can improve the temporal feature extraction and classification accuracy streams.

Robust locally nonlinear embedding (RLNE) for dimensionality reduction of high-dimensional data with noise

Local Linear Embedding (LLE) is a nonlinear manifold learning method for dimensionality reduction in high-dimensional data. However, when the data is distorted by noise, efficiency of LLE significantly diminishes. This paper proposes a robust locally nonlinear embedding (RLNE) method to alleviate the impact of noise. This is achieved by constructing nonlinear functions between data neighbors in high-dimensional space, and then mapping the relationships to low manifolds. The constrained least squares method is used to obtain more uniform weights to ensure that the neighborhood is approximately located on the local nonlinear patches of the manifold. Theoretical analysis is conducted on the reasons underlying RLNE's robustness to noise. Experimental results on synthetic and real-world data highlight RLNE's ability to preserve the intrinsic structure of data, showcasing robustness across various types data with various levels of noise, as well as with a larger number of nearest neighbors.

Spherical random projection

Abstract We propose a new method for dimension reduction of high-dimensional spherical data based on the nonlinear projection of sphere-valued data to a randomly chosen subsphere. The proposed method, spherical random projection, leads to a probabilistic lower-dimensional mapping of spherical data into a subsphere of the original. In this paper, we investigate some properties of spherical random projection, including expectation preservation and distance concentration, from which we derive an analogue of the Johnson–Lindenstrauss Lemma for spherical random projection. Clustering model selection is discussed as an application of spherical random projection, and numerical experiments are conducted using real and simulated data.

Numerically stable locality-preserving partial least squares discriminant analysis for efficient dimensionality reduction and classification of high-dimensional data

Dimensionality reduction plays a pivotal role in preparing high-dimensional data for classification and discrimination tasks by eliminating redundant features and enhancing the efficiency of classifiers. The effectiveness of a dimensionality reduction algorithm hinges on its numerical stability. When data projections are numerically stable, they lead to enhanced class separability in the lower-dimensional embedding, consequently yielding higher classification accuracy. This paper investigates the numerical attributes of dimensionality reduction and discriminant subspace learning, with a specific focus on Locality-Preserving Partial Least Squares Discriminant Analysis (LPPLS-DA). High-dimensional data frequently introduce singularity in the scatter matrices, posing a significant challenge. To tackle this issue, the paper explores two robust implementations of LPPLS-DA. These approaches not only optimize data projections but also capture more discriminative features, resulting in a marked improvement in classification accuracy. Empirical evidence supports these findings through numerical experiments conducted on synthetic and spectral datasets. The results demonstrate the superior performance of the proposed methods when compared to several state-of-the-art dimensionality reduction techniques in terms of both classification accuracy and dimension reduction.

ALGORITMA PRINCIPAL COMPONENT ANALYSIS UNTUK MENINGKATKAN PERFORMA FUZZY C-MEANS PADA KLASTERISASI DATASET BERDIMENSI TINGGI

High-dimensional data is very difficult to group, because the data growth is exponential in terms of data format and the number of values for each dimension is impossible to calculate. To increase the efficiency and accuracy of processing high-dimensional data, before testing, data cleaning and data reduction processes were carried out using the Principal Component Analysis (PCA) method. There are challenges in comparing the quality of clusters at different membership values resulting in different final clusters, this is due to the presence of noise and outliers in the processed data. Therefore, the PCA method can be used to reduce data sets from high-dimensional data to low-dimensional data and eliminate noise and outliers. The Fuzzy C-Means (FCM) method with different initialization is used to group data into clusters based on similar data, so that related data is placed in the same cluster. Based on this process, a comparison of the results of the method without PCA and with PCA was obtained and the results obtained from the implementation of PCA+FCM with the initialization of a multi-variate Gaussian distribution were higher with an accuracy level of 87.07.

Dual-Graph-Regularization Constrained Nonnegative Matrix Factorization with Label Discrimination for Data Clustering

NONNEGATIVE matrix factorization (NMF) is an effective technique for dimensionality reduction of high-dimensional data for tasks such as machine learning and data visualization. However, for practical clustering tasks, traditional NMF ignores the manifold information of both the data space and feature space, as well as the discriminative information of the data. In this paper, we propose a semisupervised NMF called dual-graph-regularization-constrained nonnegative matrix factorization with label discrimination (DCNMFLD). DCNMFLD combines dual graph regularization and prior label information as additional constraints, making full use of the intrinsic geometric and discriminative structures of the data, and can efficiently enhance the discriminative and exclusionary nature of clustering and improve the clustering performance. The evaluation of the clustering experimental results on four benchmark datasets demonstrates the effectiveness of our new algorithm.

Machine learning: a new direction for assisting the application of deep brain stimulation

In recent years, deep brain stimulation has been widely used in the treatment of various difficult diseases of the nervous system. The rapid development of machine learning algorithms has greatly promoted the application and development of deep brain stimulation. Machine learning can help in preoperative screening of surgical candidates, outcome prediction and surgical planning. It can assist target localization during operation. More importantly, the local field potential signals recorded by macroscopic electrodes can be analyzed in real time after surgery, which provides a basis for the development of a closed-loop stimulation system. Of course, the application of machine learning also has its limitations and challenges, such as dimensionality reduction of high-dimensional data, development and validation of new models and etc, which need to be further explored and improved.

Comparative Analysis of Feature Extraction of High Dimensional Data Reduction Using Machine Learning Techniques

Comparative Analysis of Feature Extraction of High Dimensional Data Reduction Using Machine Learning Techniques

A lightweight IoT intrusion detection model based on improved BERT-of-Theseus

The proliferation of Internet of Things (IoT) technology has resulted in an increase in security vulnerabilities associated with the interconnectivity of IoT devices. As a result, there is a need for intrusion detection mechanisms that can effectively detect attacks on IoT security vulnerabilities. However, due to the resource constraints of IoT deployment devices, intrusion detection schemes must be customized to meet the specific demands of the IoT environment. In this study, we propose a knowledge-distillation-based IoT intrusion detection model named BT-TPF, which is capable of detecting network attacks encountered by IoT devices in an IoT environment with limited computing resources. The proposed BT-TPF model leverages a Siamese network for feature dimensionality reduction of complex high-dimensional network traffic data. Additionally, it employs a large-scale Vision Transformer as a teacher model to guide a small-scale Poolformer model during training, before deploying the trained Poolformer model as a classifier to detect network intrusion traffic. Through knowledge distillation, the final small model obtained in this paper only requires a minimum of 788 parameters, reducing the number of parameters by approximately 90% compared to the large model before knowledge distillation, while maintaining high detection accuracy. Experimental results show that the BT-TPF model achieves over 99% accuracy on both the CIC-IDS2017 and TON_IoT datasets. Furthermore, it exhibits significant advantages compared to traditional Deep Learning methods and recent state-of-the-art models, as evidenced by various evaluation metrics.

Weighted high dimensional data reduction of finite element features: an application on high pressure of an abdominal aortic aneurysm

Weighted high dimensional data reduction of finite element features: an application on high pressure of an abdominal aortic aneurysm

Data-driven physical fields reconstruction of supercritical-pressure flow in regenerative cooling channel using POD-AE reduced-order model

In order to effectively estimate the physical fields of active cooling channel with supercritical pressure hydrocarbon fuel, a novel data-driven reduced-order model framework is firstly proposed in the present work, which is established via combining reduced-order modeling method based on proper orthogonal decomposition and autoencoder neural network (POD-AE) with Kriging surrogate model. This framework mainly contains offline stage and online stage. Preliminary dimension reduction and feature capture of high-dimensional physical fields data is conducted using POD in the offline stage, and low-dimensional reduced-order models (ROMs) are constructed by the linear combination of orthogonal bases and feature coefficients. Specially, feature coefficients of ROMs are used to trained an AE, and then lower-dimensional secondary reduced-order models (2nd-ROMs) are established by secondary compression for ROMs using the trained AE. Furthermore, the Kriging model is trained for implicitly mapping boundary conditions or temperature data from the sensors on the bottom surface to the established 2nd-ROMs. In the online stage, physical fields estimation under new boundary conditions or wall temperature monitoring based on temperature data of sensors can be quickly fulfilled using this framework. Fifty groups of test conditions are considered to demonstrate the accuracy and efficiency of the POD-AE ROM framework. The results prove that it shows good efficiency, accuracy in estimating and monitoring the physical fields of the cooling channel with supercritical pressure n-dodecane within sample space. Compared with the traditional POD-based ROM framework, the proposed framework still achieved acceptable accuracy under all testing conditions when the original data was compressed to extremely low dimensions. And the calculation time only slightly increased.

Optimal supervised reduction of high dimensional transcription data.

The plight of navigating high-dimensional transcription datasets remains a persistent problem. This problem is further amplified for complex disorders, such as cancer as these disorders are often multigenic traits with multiple subsets of genes collectively affecting the type, stage, and severity of the trait. We are often faced with a trade off between reducing the dimensionality of our datasets and maintaining the integrity of our data. To accomplish both tasks simultaneously for very high dimensional transcriptome for complex multigenic traits, we propose a new supervised technique, Class Separation Transformation (CST). CST accomplishes both tasks simultaneously by significantly reducing the dimensionality of the input space into a one-dimensional transformed space that provides optimal separation between the differing classes. Furthermore, CST offers an means of explainable ML, as it computes the relative importance of each feature for its contribution to class distinction, which can thus lead to deeper insights and discovery. We compare our method with existing state-of-the-art methods using both real and synthetic datasets, demonstrating that CST is the more accurate, robust, scalable, and computationally advantageous technique relative to existing methods. Code used in this paper is available on https://github.com/richiebailey74/CST.

CDRKD: An improved density peak algorithm based on kernel fuzzy measure in the overlapping community detection

Compared with other traditional community discovery algorithms, density peak clustering algorithm is more efficient in getting network structures through clustering. However, DPC needs to contain the distance information between all nodes as sources, so it cannot directly processing the complex network represented by the adjacency matrix. DPC introduces truncation distance when calculating the local density of nodes, which is usually set as a fixed value according to experience, and lacks self-adaptability for different network structures. A feasible solution to those problems is to combined rough set theory and kernel fuzzy similarity measures. In this work, we present overlapping community detection algorithm based on improved rough entropy fusion density peak. The algorithm applied rough set theory to attribute reduction of massive high-dimensional data. Another algorithm defines the similarity of sample points by the inner product between two vectors on the basis of fuzzy partition matrix. Finally, a community detection algorithm based on rough entropy and kernel fuzzy density peaks clustering (CDRKD) has proposed by combining the two algorithms above, we perform an extensive set of experiments to verify the effectiveness and feasibility of the algorithm.

Decomposition and Classification of Carbon Star Spectra

Automatic classification of stellar spectra is an important research component of astronomical data processing and is the basis for studying stellar evolution and parameter measurements. As a rare kind of stellar spectra, carbon star spectra put forward more efficient and accurate requirements for classification methods. The traditional manual classification methods have the disadvantages of slow speed and low accuracy, which can no longer meet the practical needs of automatic classification of massive stellar spectra, especially low signal-to-noise ratio stellar spectra, and machine learning algorithms have been widely applied to stellar spectral classification. A distinctive feature of stellar spectra is high data dimensionality, and dimensionality reduction can not only realize feature extraction, but also reduce the computational effort, which is the first task of spectral classification. Traditional linear dimensionality reduction methods such as principal component analysis reduce the spectra only based on the variance, and different types of spectra will appear crossover after projection into the low-dimensional feature space, while streamwise learning can produce excellent classification boundaries, which will avoid overlap and facilitate subsequent classification. In view of the high dimensionality of spectral data, we investigate the distribution of spectral data in high-dimensional space and the principle of dimensionality reduction of high-dimensional linear data by stream shape learning, compare the effect of two-dimensionality reduction methods, t - SNE and principal component analysis, on spectral data, and finally analyze the experimental results and compare and validate them using various machine learning classifiers. The algorithm is implemented using Python language and Scikit - learn third-party library to perform experiments on 1000 low signal-to-noise carbon star spectra from LAMOST, and finally achieve high accuracy automatic processing and classification of the spectral data. The experimental results show that for the dimensionality reduction processing of spectral data, the t - SNE method based on stream shape learning can recover the low-dimensional stream shape structure in the high-dimensional spectral data, and after feature extraction, satisfactory classification accuracy can be achieved on the test dataset using a machine learning classifier.

High Dimensional Data Reduction in Modal Analysis with Stochastic Subspace Identification

Subspace system identification methods are widely used in output-only vibration analysis of civil structures, known as operational modal analysis. With the advent of new sensor technologies, such as video camera-based full field displacement or velocity measurements, the number of measured outputs is quickly increasing. In this paper, we propose principal component analysis-based data size reduction methods for efficient application of subspace methods, while preserving the high spatial resolution of the identified mode shapes for detailed modal analysis.

Utilizing differential characteristics of high dimensional data as a mechanism for dimensionality reduction

• A generally applicable dimensionality reduction method namely CKD has been proposed. • CKD leverages the differential characteristics of the data with respect to a reference data set. • Reference data can be obtained through an independent measurement or from the data themselves. • CKD shows superior performance over existing techniques. Dimensionality reduction and visualization of high dimensional data are fundamental tasks across scientific disciplines. Current dimensionality reduction methods rely on the use of some restrictive mathematical rules in projecting the data onto lower dimensions, which often leads to results of limited accuracy. Here, we present a generally applicable strategy of analyzing high-dimensional data by leveraging the differential characteristics of the data with respect to a reference data set. Depending on the problem, the reference data can be obtained either through an independent measurement or from the data themselves by finding a set of representative data points. The differentiating characteristics between the two sets of data are optimized by correlation transformation and scaling of the covariance matrices . We show the superior performance of the proposed method over existing techniques for a variety of data sets from different biomedical applications .

A CT-Based Radiomics Nomogram to Predict Complete Ablation of Pulmonary Malignancy: A Multicenter Study.

ObjectiveThermal ablation is a minimally invasive procedure for the treatment of pulmonary malignancy, but the intraoperative measure of complete ablation of the tumor is mainly based on the subjective judgment of clinicians without quantitative criteria. This study aimed to develop and validate an intraoperative computed tomography (CT)-based radiomic nomogram to predict complete ablation of pulmonary malignancy.MethodsThis study enrolled 104 individual lesions from 92 patients with primary or metastatic pulmonary malignancies, which were randomly divided into training cohort (n=74) and verification cohort (n=30). Radiomics features were extracted from the original CT images when the study clinicians determined the completion of the ablation surgery. Minimum redundancy maximum relevance (mRMR) and least absolute shrinkage and selection operator (LASSO) were adopted for the dimensionality reduction of high-dimensional data and feature selection. The prediction model was developed based on the radiomics signature combined with the independent clinical predictors by multiple logistic regression analysis. The area under the curve (AUC), accuracy, sensitivity, and specificity were calculated. Receiver operating characteristic (ROC) curves and calibration curves were used to evaluate the predictive performance of the model. Decision curve analysis (DCA) was applied to estimate the clinical usefulness and net benefit of the nomogram for decision making.ResultsThirteen CT features were selected to construct radiomics prediction model, which exhibits good predictive performance for determination of complete ablation of pulmonary malignancy. The AUCs of a CT-based radiomics nomogram that integrated the radiomics signature and the clinical predictors were 0.88 (95% CI 0.80-0.96) in the training cohort and 0.87 (95% CI: 0.71–1.00) in the validation cohort, respectively. The radiomics nomogram was well calibrated in both the training and validation cohorts, and it was highly consistent with complete tumor ablation. DCA indicated that the nomogram was clinically useful.ConclusionA CT-based radiomics nomogram has good predictive value for determination of complete ablation of pulmonary malignancy intraoperatively, which can assist in decision-making.