High-dimensional Data Research Articles

Estimation of the High-Frequency Feature Slope in Gravitational Wave Signals from Core Collapse Supernovae Using Machine Learning

We conducted an in-depth exploration of the use of different machine learning (ML) for regression algorithms, including Linear, Ridge, LASSO, Bayesian Ridge, Decision Tree, and a variety of Deep Neural Network (DNN) architectures, to estimate the slope of the high-frequency feature (HFF), a prominent emergent feature found in the gravitational wave (GW) signals of core collapse supernovae (CCSN). We created a data set of CCSN GW signals generated by an analytical model that mimics the characteristics of the signals obtained from numerical simulations, particularly the HFF. This enabled us to simulate a wide range of HFF slope values and analyze their properties. We opted to employ ML for regression techniques, particularly a supervised learning approach, to analyze the data set due to the parameter chosen for estimating the slope of the HFF. This type of architecture is ideal for this purpose as it can detect the connections between input and output data. In addition, it is suitable for handling high-dimensional input data and produces efficient results with low computational cost. We evaluated the efficiency and performance of the ML algorithms using a set of metrics to measure their ability to accurately predict the HFF slope within the data set. The results showed that a DNN algorithm for regression exhibits the highest accuracy in estimating the slope of the HFF.

Dynamics-preserving compression for modal flow analysis

An efficient compression scheme for modal flow analysis is proposed and validated on data sequences of compressible flow through a linear turbomachinery blade row. The key feature of the compression scheme is a minimal, user-defined distortion of the mutual distance of any snapshot pair in phase space. Through this imposed feature, the model reduction process preserves the temporal dynamics contained in the data sequence, while still decreasing the spatial complexity. The mathematical foundation of the scheme is the fast Johnson–Lindenstrauss transformation (FJLT) which uses randomized projections and a tree-based spectral transform to accomplish the embedding of a high-dimensional data sequence into a lower-dimensional latent space. The compression scheme is coupled to a proper orthogonal decomposition and dynamic mode decomposition analysis of flow through a linear blade row. The application to a complex flow-field sequence demonstrates the efficacy of the scheme, where compression rates of two orders of magnitude are achieved, while incurring very small relative errors in the dominant temporal dynamics. This FJLT technique should be attractive to a wide range of modal analyses of large-scale and multi-physics fluid motion.

Condition Monitoring of Marine Diesel Lubrication System Based on an Optimized Random Singular Value Decomposition Model

As modern marine diesel engine systems become increasingly complex, effective condition monitoring methods are essential for ensuring optimal performance and preventing anomalies. This paper proposes a data-driven condition monitoring approach specifically designed for the lubrication system of marine diesel engines. Unlike traditional methods, the proposed approach eliminates the need for explicit modeling and leverages a novel optimization algorithm for data denoising. Additionally, a new noise-resistant monitoring index is introduced to enhance monitoring reliability. The paper is structured into two main sections for validation. The first section addresses advanced data preprocessing, where the Improved Sparrow Search Algorithm (ISSA) is employed to optimize the parameters of Random Singular Value Decomposition (RSVD). This step effectively minimizes noise, reduces manual intervention, and handles high-dimensional data. The second section focuses on analyzing the data characteristics using the Random Matrix Theory (RMT) and establishing novel condition monitoring indicators to achieve more reliable monitoring outcomes. The proposed methodology captures the intricate relationships among key variables within the system, providing a more robust framework for condition monitoring. Applied to a marine diesel engine lubrication system, the method demonstrates significant improvements in noise immunity and monitoring reliability. Comparative analyses of condition monitoring models before and after denoising reveal that the relative error of the proposed monitoring index under varying noise amplitudes is within 1%, substantially lower than that of other indices. Furthermore, the monitoring accuracy is improved by 4.95% when the proposed index is employed for system condition monitoring.

GDSMOTE: A Novel Synthetic Oversampling Method for High-Dimensional Imbalanced Financial Data

Synthetic oversampling methods for dealing with imbalanced classification problems have been widely studied. However, the current synthetic oversampling methods still cannot perform well when facing high-dimensional imbalanced financial data. The failure of distance measurement in high-dimensional space, error accumulation caused by noise samples, and the reduction of recognition accuracy of majority samples caused by the distribution of synthetic samples are the main reasons that limit the performance of current methods. Taking these factors into consideration, a novel synthetic oversampling method is proposed, namely the gradient distribution-based synthetic minority oversampling technique (GDSMOTE). Firstly, the concept of gradient contribution was used to assign the minority-class samples to different gradient intervals instead of relying on the spatial distance. Secondly, the root sample selection strategy of GDSMOTE avoids the error accumulation caused by noise samples and a new concept of nearest neighbor was proposed to determine the auxiliary samples. Finally, a safety gradient distribution approximation strategy based on cosine similarity was designed to determine the number of samples to be synthesized in each safety gradient interval. Experiments on high-dimensional imbalanced financial datasets show that GDSMOTE can achieve a higher F1-Score and MCC metrics than baseline methods while achieving a higher recall score. This means that our method has the characteristics of improving the recognition accuracy of minority-class samples without sacrificing the recognition accuracy of majority-class samples and has good adaptability to data decision-making tasks in the financial field.

Advancements in Vector Indexing Techniques: KD-Trees, HNSW, and Product Quantization for Vector Databases

Abstract—The growth of high-dimensional data in fields like computer vision, natural language processing, and data mining has necessitated the development of efficient vector indexing techniques. This paper presents a comparative analysis of three state-of-the-art vector indexing techniques—KD-Trees, Hierarchical Navigable Small World (HNSW) graphs, and Product Quantization (PQ). We provide a detailed exploration of each method’s algorithmic foundations, mathematical formulations, advantages, and limitations, followed by a discussion of recent advancements aimed at optimizing these methods for large-scale databases. Through my analysis, I aim to offer a comprehensive understanding of how these techniques perform and can be adapted for various real-world applications. Index Terms—Vector Indexing, KD-Trees, HNSW, Product Quantization, High-Dimensional Data, Nearest Neighbor Search

Research on the abnormal identification method of remote real-time monitoring of power system equipment based on artificial intelligence

ABSTRACT This study addresses the critical challenge of anomaly detection in modern power systems, proposing a novel hybrid approach that combines supervised and unsupervised machine learning techniques. The increasing complexity of power grids and integrating renewable energy sources and smart technologies necessitate more sophisticated methods for identifying and mitigating potential threats to system stability and security. Our model leverages deep learning architectures, including Long Short-Term Memory networks and Convolutional Neural Networks, to process high-dimensional time-series data from various grid sensors. The system demonstrates exceptional performance in detecting diverse anomalies, including cyber-attacks, equipment failures, and environmental disturbances, with an overall accuracy of 97.8% and an F1-score of 0.956. Real-time processing capabilities enable anomaly detection within 1.2 seconds, crucial for preventing cascading failures. A novel interpretability framework provides insights into the model’s decision-making process, enhancing trust and actionable intelligence for operators. Extensive case studies and robustness analyses validate the system’s effectiveness across various operational scenarios. This research significantly enhances power grid resilience and reliability, offering a robust foundation for future advancements in smart grid technologies and adaptive anomaly detection systems.

A REVIEW OF TOOLS, METHODOLOGIES, AND TECHNIQUES FOR PROCESSING, PRE-PROCESSING, AND CLUSTERING ANALYSIS OF GENETIC DATA

Gene expression analysis has become a key component in understanding cellular behavior, disease mechanisms, and drug response. The advent of high-throughput sequencing, particularly single-cell RNA sequencing (scRNAseq), has expanded our ability to study cellular heterogeneity to an unprecedented level. Clustering algorithms needed to group genes or cells with similar expression profiles have become invaluable for analyzing the massive data sets generated by these technologies. This article reviews various clustering methods applied to gene expression data, particularly single-cell RNA sequencing. The analysis covers traditional methods such as hierarchical clustering and k-means, as well as more advanced approaches such as model-based clustering, machine learning-based methods, and deep learning methods. The primary challenges encompass handling high-dimensional data, mitigating noise, and achieving scalability for large datasets. Moreover, new advancements such as multi-omics data integration, deep learning-based clustering, and federated learning offer potential enhancements in accuracy and biological relevance for clustering applications in gene expression research. The review concludes with a discussion of clustering algorithms in handling increasingly complex gene expression data for more accurate biological insights.

Integrative factor-adjusted sparse generalized linear models

With the increasing accessibility of data, sample sizes remain limited in most studies, integrating multiple-source datasets is thus desirable to improve model performance. The high dependence among covariates is a common characteristic of high-dimensional data. However, traditional approaches fail to achieve variable selection consistency when the covariates are highly correlated. To address the issues stemming from correlation, we propose the penalized factor-adjusted approach to reduce the correlations in the integrative analysis within a generalized linear model (GLM) framework for multi-source high-dimensional data. By utilizing the latent factors and idiosyncratic components as predictors, the proposed approach enables model estimation with weakly correlated covariates. We rigorously establish the consistency properties under certain conditions. Simulation demonstrates the superior and competitive performance of the proposed approach in estimation, prediction, and variable selection. Analysis of genetic data on prostate cancer confirms its practical usefulness.

Unveiling the power of high-dimensional cytometry data with cyCONDOR

High-dimensional cytometry (HDC) is a powerful technology for studying single-cell phenotypes in complex biological systems. Although technological developments and affordability have made HDC broadly available in recent years, technological advances were not coupled with an adequate development of analytical methods that can take full advantage of the complex data generated. While several analytical platforms and bioinformatics tools have become available for the analysis of HDC data, these are either web-hosted with limited scalability or designed for expert computational biologists, making their use unapproachable for wet lab scientists. Additionally, end-to-end HDC data analysis is further hampered due to missing unified analytical ecosystems, requiring researchers to navigate multiple platforms and software packages to complete the analysis. To bridge this data analysis gap in HDC we develop cyCONDOR, an easy-to-use computational framework covering not only all essential steps of cytometry data analysis but also including an array of downstream functions and tools to expand the biological interpretation of the data. The comprehensive suite of features of cyCONDOR, including guided pre-processing, clustering, dimensionality reduction, and machine learning algorithms, facilitates the seamless integration of cyCONDOR into clinically relevant settings, where scalability and disease classification are paramount for the widespread adoption of HDC in clinical practice. Additionally, the advanced analytical features of cyCONDOR, such as pseudotime analysis and batch integration, provide researchers with the tools to extract deeper insights from their data. We use cyCONDOR on a variety of data from different tissues and technologies demonstrating its versatility to assist the analysis of high-dimensional data from preprocessing to biological interpretation.

TEMPTED: time-informed dimensionality reduction for longitudinal microbiome studies

Longitudinal studies are crucial for understanding complex microbiome dynamics and their link to health. We introduce TEMPoral TEnsor Decomposition (TEMPTED), a time-informed dimensionality reduction method for high-dimensional longitudinal data that treats time as a continuous variable, effectively characterizing temporal information and handling varying temporal sampling. TEMPTED captures key microbial dynamics, facilitates beta-diversity analysis, and enhances reproducibility by transferring learned representations to new data. In simulations, it achieves 90% accuracy in phenotype classification, significantly outperforming existing methods. In real data, TEMPTED identifies vaginal microbial markers linked to term and preterm births, demonstrating robust performance across datasets and sequencing platforms.

MinLinMo: a minimalist approach to variable selection and linear model prediction

Generating prediction models from high dimensional data often result in large models with many predictors. Causal inference for such models can therefore be difficult or even impossible in practice. The stand-alone software package MinLinMo emphasizes small linear prediction models over highest possible predictability with a particular focus on including variables correlated with the outcome, minimal memory usage and speed. MinLinMo is demonstrated on large epigenetic datasets with prediction models for chronological age, gestational age, and birth weight comprising, respectively, 15, 14 and 10 predictors. The parsimonious MinLinMo models perform comparably to established prediction models requiring hundreds of predictors.

Optimizing Intrusion Detection for DoS, DDoS, and Mirai Attacks Subtypes Using Hierarchical Feature Selection and CatBoost on the CICIoT2023 Dataset

Introduction: Modern networks suffer until unheard of vulnerabilities that need for advanced intrusion detection systems (IDS) given the growing danger presented by DoS, DDoS, and Mirai attacks. Research on the identification of certain attack subtypes is still lacking even with the CICIoT2023 dataset, which offers a complete basis for evaluating these cyber hazards. Usually, aggregating attacks into more general categories, existing research neglects the complex characteristics of specific subtypes, therefore reducing the detection effectiveness.Methods: This work presents a novel IDS model aiming at high accuracy detection of DoS, DDoS, and Mirai attack subtypes. Using hierarchical feature selection and the CatBoost algorithm on the CICIoT2023 dataset, our model addresses the problems of high-dimensional data and emphasizes on keeping the most important features by means of advanced preprocessing methods including Spearman correlation and hierarchical clustering. Furthermore, used is stratified sampling to guarantee in the training and testing stages fair representation of attack types, both common and uncommon.Results: With an amazing Prediction Time per Network Flow of 7.16e-07 seconds, our model shows a breakthrough in intrusion detection performance by means of rigorous stratified cross-valuation, thereby attaining outstanding outcomes in accuracy, recall, and precision.Conclusions: Our method not only closes a significant gap in current knowledge but also establishes a new benchmark in cybersecurity by providing very detailed protection mechanisms against advanced threats. This study marks major progress in network security as it gives companies a more efficient instrument to recognize and minimize certain cyber risks with better precision and effectiveness

Fast spatial simulation of extreme high-resolution radar precipitation data using integrated nested Laplace approximations

Abstract Aiming to deliver improved precipitation simulations for hydrological impact assessment studies, we develop a methodology for modelling and simulating high-dimensional spatial precipitation extremes, focusing on both their marginal distributions and tail dependence structures. Tail dependence is crucial for assessing the consequences of extreme precipitation events, yet most stochastic weather generators do not attempt to capture this property. The spatial distribution of precipitation occurrences is modelled with four competing models, while the spatial distribution of nonzero extreme precipitation intensities are modelled with a latent Gaussian version of the spatial conditional extremes model. Nonzero precipitation marginal distributions are modelled using latent Gaussian models with gamma and generalized Pareto likelihoods. Fast inference is achieved using integrated nested Laplace approximations. We model and simulate spatial precipitation extremes in Central Norway, using 13 years of hourly radar data with a spatial resolution of 1×1km2, over an area of size 6,461km2, to describe the behaviour of extreme precipitation over a small drainage area. Inference on this high-dimensional data set is achieved within hours, and the simulations capture the main trends of the observed precipitation well.

Integrating Statistical Methods and Machine Learning Techniques to Analyze and Classify COVID-19 Symptom Severity

Background/Objectives: The COVID-19 pandemic, caused by Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), led to significant global health challenges, including the urgent need for accurate symptom severity prediction aimed at optimizing treatment. While machine learning (ML) and deep learning (DL) models have shown promise in predicting COVID-19 severity using imaging and clinical data, there is limited research utilizing comprehensive tabular symptom datasets. This study aims to address this gap by leveraging a detailed symptom dataset to develop robust models for categorizing COVID-19 symptom severity, thereby enhancing clinical decision making. Methods: A unique tabular dataset was created using questionnaire responses from 5654 individuals, including demographic information, comorbidities, travel history, and medical data. Both unsupervised and supervised ML techniques were employed, including k-means clustering to categorize symptom severity into mild, moderate, and severe clusters. In addition, classification models, namely, Support Vector Machine (SVM), Adaptive Boosting (AdaBoost), eXtreme Gradient Boosting (XGBoost), random forest, and a deep neural network (DNN) were used to predict symptom severity levels. Feature importance was analyzed using the random forest model for its robustness with high-dimensional data and ability to capture complex non-linear relationships, and statistical significance was evaluated through ANOVA and Chi-square tests. Results: Our study showed that fatigue, joint pain, and headache were the most important features in predicting severity. SVM, AdaBoost, and random forest achieved an accuracy of 94%, while XGBoost achieved an accuracy of 96%. DNN showed robust performance in handling complex patterns with 98% accuracy. In terms of precision and recall metrics, both the XGBoost and DNN models demonstrated robust performance, particularly for the moderate class. XGBoost recorded 98% precision and 97% recall, while DNN achieved 99% precision and recall. The clustering approach improved classification accuracy by reducing noise and dimensionality. Statistical tests confirmed the significance of additional features like Body Mass Index (BMI), age, and dominant variant type. Conclusions: Integrating symptom data with advanced ML models offers a promising approach for accurate COVID-19 severity classification. This method provides a reliable tool for healthcare professionals to optimize patient care and resource management, particularly in managing COVID-19 and potential future pandemics. Future work should focus on incorporating imaging and clinical data to further enhance model accuracy and clinical applicability.

Disentangling genotype and environment specific latent features for improved trait prediction using a compositional autoencoder.

In plant breeding and genetics, predictive models traditionally rely on compact representations of high-dimensional data, often using methods like Principal Component Analysis (PCA) and, more recently, Autoencoders (AE). However, these methods do not separate genotype-specific and environment-specific features, limiting their ability to accurately predict traits influenced by both genetic and environmental factors. We hypothesize that disentangling these representations into genotype-specific and environment-specific components can enhance predictive models. To test this, we developed a compositional autoencoder (CAE) that decomposes high-dimensional data into distinct genotype-specific and environment-specific latent features. Our CAE framework employed a hierarchical architecture within an autoencoder to effectively separate these entangled latent features. Applied to a maize diversity panel dataset, the CAE demonstrated superior modeling of environmental influences and out-performs PCA (principal component analysis), PLSR (Partial Least square regression) and vanilla autoencoders by 7 times for 'Days to Pollen' trait and 10 times improved predictive performance for 'Yield'. By disentangling latent features, the CAE provided a powerful tool for precision breeding and genetic research. This work has significantly enhanced trait prediction models, advancing agricultural and biological sciences.

Enhancing Healthcare Through Metaheuristic Based Deep Learning: Microarray Gene Expression Image Classification Model

Microarray gene expression data analysis has revolutionized genomics by providing insights into the genetic basis of diseases, including cancer. However, the high dimensionality of microarray data poses significant challenges for accurate classification, often leading to overfitting and poor generalization. This study proposes a novel framework that integrates the DNA Bidirectional Encoder Representations from Transformers (DNABERT) with the Mayfly Optimization Algorithm (MFO) to improve the classification accuracy of microarray gene expression data. DNABERT, a transformer-based model pretrained on genomic sequences, excels in learning complex gene interactions through bidirectional contextual embeddings. MFO, a bio-inspired optimization technique, addresses feature selection and hyperparameter tuning by balancing exploration and exploitation in high-dimensional search spaces. The framework, DNABERT-MFO, leverages MFO to identify the most relevant gene subsets and optimize DNABERT's hyperparameters, improving the model’s performance. Evaluations conducted on multiple microarray datasets, demonstrate that DNABERT-MFO with overall accuracy of above 80% significantly outperforms traditional machine learning methods and standalone deep learning models in classification accuracy. This integrated method not only enhances the robustness of gene expression data analysis but also offers a powerful tool for research and clinical applications in genomics. The proposed method addresses key limitations of existing techniques and provides a promising avenue for future advancements in gene expression classification.

Improve Threshold Range of Canopy Clustering Using Optimization Algorithms

Canopy clustering is an effective method for determining the number of clusters dynamically without requiring a predefined cluster count, making it particularly suitable for large and complex datasets. However, its performance is highly dependent on the manual tuning of threshold parameters T1 and T2, which can be time-consuming and inefficient. This study aims to enhance the Canopy clustering algorithm by automating the optimization of threshold ranges using intelligent optimization algorithms. We propose a novel framework that integrates Simulated Annealing (SA), Particle Swarm Optimization (PSO), and Snake Optimization (SO) to automatically determine the optimal values of T1 and T2. Additionally, to address high-dimensional data complexity, we employ dimensionality reduction techniques such as t-SNE, SNE, and Kernel Principal Component Analysis (KPCA). The silhouette coefficient is utilized as the fitness function to evaluate clustering performance. Comprehensive experiments conducted on the Wine, Iris, and MNIST Subset datasets demonstrate that the proposed optimization-based Canopy clustering framework significantly improves clustering accuracy by up to 21% on the Wine dataset and 19% on the Iris dataset compared to traditional methods. Specifically, on the Wine dataset, the optimized Canopy clustering achieved a silhouette coefficient of 0.63, a 21% improvement over the original 0.52. On the Iris dataset, the optimized method outperformed k-means and manual Canopy clustering with silhouette coefficients of 0.62 versus 0.52 and 0.55, respectively. These results highlight the effectiveness of intelligent optimization algorithms in enhancing clustering adaptability and efficiency.

A multidimensional toolkit for elucidating temporal trajectories in cell development in vivo.

Progenitor cells initiate development upon receiving key signals, dynamically altering gene and protein expression to diverge into various lineages and fates. Despite the use of several experimental approaches, including the Fluorescent Timer-based method Timer-of-cell-kinetics-and-activity (Tocky), analysing time-dependent processes at the single-cell level in vivo remains challenging. This study introduces a novel integrated experimental and computational approach, using an advanced multidimensional toolkit. This toolkit facilitates the simultaneous examination of temporal progression and T-cell profiles using high-dimensional flow cytometric data. Employing novel algorithms based on canonical correspondence analysis and network analysis, our toolkit identifies developmental trajectories and analyses dynamic changes in developing cells. The efficacy of this approach is demonstrated by analysing thymic T cells from Nr4a3-Tocky mice, which monitor activities downstream of the T-cell receptor (TCR) signal. Further validation was achieved by deleting the proapoptotic gene Bcl2l11 in Nr4a3-Tocky mice. This revealed dynamic changes in thymic T cells during cellular development and negative selection following TCR signalling. Overall, this study establishes a new method for analysing the temporal dynamics of individual developing cells in response to in vivo signalling cues.

Weighted Linear Discriminant Analysis: An Effective Feature Extraction Method for Multi-Class Imbalanced Datasets

In high-dimensional machine learning tasks, supervised feature extraction is essential for improving model performance, with Linear Discriminant Analysis (LDA) being a common approach. However, LDA tends to deliver suboptimal performance when dealing with class imbalance. To address this issue, we propose a novel feature extraction model, Weighted Linear Discriminant Analysis (WLDA), which integrates cost-sensitive techniques into the traditional LDA framework. By assigning weights inversely proportional to class sample sizes, WLDA achieves effective feature extraction under imbalanced sample conditions. We introduce an efficient solution algorithm for the proposed model and provide a thorough complexity analysis. Experimental results demonstrate the superior performance of WLDA in handling imbalanced datasets, confirming its potential as a robust tool for high-dimensional data scenarios. Overall, WLDA not only improves feature extraction for imbalanced datasets but also enhances classification accuracy across diverse applications.

Testing Spherical Symmetry Based on Statistical Representative Points

This paper introduces a novel chisquare test for spherical symmetry, utilizing statistical representative points. The proposed representative-point-based chisquare statistic is shown, through a Monte Carlo study, to considerably improve the power performance compared to the traditional equiprobable chisquare test in many high-dimensional cases. While the test requires relatively large sample sizes to approximate the chisquare distribution, obtaining critical values from existing chisquare tables is simpler compared to many existing tests for spherical symmetry. A real-data application demonstrates the robustness of the proposed method against different choices of representative points. This paper argues that the use of representative points provides a new perspective in high-dimensional goodness-of-fit testing, offering an alternative approach to evaluating spherical symmetry in such contexts. By leveraging the flexibility of choosing the number of representative points, this method ensures more reliable detection of departures from spherical symmetry, especially in high-dimensional datasets. Overall, this research highlights the practical advantages of the proposed approach in statistical analysis, emphasizing its potential as a powerful tool in goodness-of-fit tests within the realm of high-dimensional data.