High-dimensional Data Research Articles

Human Papillomavirus, Human Immunodeficiency Virus, and Oral Microbiota Interplay in Nigerian Youth (HOMINY): A Prospective Cohort Study Protocol

IntroductionPersistent oral infections with high-risk human papillomavirus (HR-HPV) are a potential cause of most oropharyngeal cancers (OPCs). Oral HR-HPV infection and persistence are significantly higher in people living with HIV...

Identifying patterns differing between high-dimensional datasets with generalized contrastive PCA.

High-dimensional data have become ubiquitous in the biological sciences, and it is often desirable to compare two datasets collected under different experimental conditions to extract low-dimensional patterns enriched in one condition. However, traditional dimensionality reduction techniques cannot accomplish this because they operate on only one dataset. Contrastive principal component analysis (cPCA) has been proposed to address this problem, but it has seen little adoption because it requires tuning a hyperparameter resulting in multiple solutions, with no way of knowing which is correct. Moreover, cPCA uses foreground and background conditions that are treated differently, making it ill-suited to compare two experimental conditions symmetrically. Here we describe the development of generalized contrastive PCA (gcPCA), a flexible hyperparameter-free approach that solves these problems. We first provide analyses explaining why cPCA requires a hyperparameter and how gcPCA avoids this requirement. We then describe an open-source gcPCA toolbox containing Python and MATLAB implementations of several variants of gcPCA tailored for different scenarios. Finally, we demonstrate the utility of gcPCA in analyzing diverse high-dimensional biological data, revealing unsupervised detection of hippocampal replay in neurophysiological recordings and heterogeneity of type II diabetes in single-cell RNA sequencing data. As a fast, robust, and easy-to-use comparison method, gcPCA provides a valuable resource facilitating the analysis of diverse high-dimensional datasets to gain new insights into complex biological phenomena.

Factors associated with underweight, overweight, and obesity in Chinese children aged 3-14 years using ensemble learning algorithms.

Factors underlying the development of childhood underweight, overweight, and obesity are not fully understood. Traditional models have drawbacks in handling large-scale, high-dimensional, and nonlinear data. In this study, we aimed to identify factors responsible for underweight, overweight, and obesity using machine learning methods among Chinese children. Our study participants were children aged 3-14 from 30 kindergartens and 26 schools in Beijing and Tangshan. Weight status was defined per the World Health Organization criteria. We implemented three ensemble learning algorithms and compared their performance and ranked the contributing factors by importance and identified an optimal set. A user-friendly web application was developed to calculate the predicted probability of childhood underweight, overweight, and obesity. We analysed data from 18 503 children aged 3-14, including 1798 underweight, 10 579 of normal weight, 3257 overweight, and 2869 with obesity. Of all algorithms, random forest performed the best, with the area under the receiver operating characteristic reaching 0.759 for underweight, 0.806 for overweight, and 0.849 for obesity, with other metrics also reinforcing this algorithm. Further cumulative analyses showed that, for underweight, the optimal set of six factors included maternal body mass index (BMI), age, paternal BMI, maternal reproductive age, paternal reproductive age, and birth weight. The optimal set for overweight comprised of five factors: age, fast food intake, maternal BMI, paternal BMI, and sedentary time. For obesity, the optimal set included six factors: age, fast food intake, maternal BMI, paternal BMI, sedentary time, and maternal reproductive age. Further logistic regression analyses confirmed the predictive capability of individual top factors. Our findings indicate that random forest is the best ensemble learning algorithm for predicting underweight, overweight, and obesity in children aged 3-14 years. We identified the optimal set of significant factors for each malnutrition status and incorporated them into a web application to support the application of this study's findings.

Multi-head spatial-spectral mamba for hyperspectral image classification

ABSTRACT Spatial-Spectral Mamba (SSM) improves computational efficiency and captures long-range dependencies, addressing the limitations of transformers. However, traditional Mamba models often overlook the rich spectral information in hyperspectral images (HSIs) and struggle with high dimensionality and sequential data. To address these challenges, we propose the Spatial-Spectral Mamba with Multi-Head Self-Attention and Token Enhancement (MHSSMamba). This model integrates spatial and spectral information by enhancing spectral tokens and employing multi-head self-attention to capture complex relationships between spectral bands and spatial locations. It effectively manages long-range dependencies and the sequential nature of HSI data, preserving contextual information across spectral bands. MHSSMamba achieved classification accuracies of 98.56% on the Pavia University dataset, 99.00% on the University of Houston dataset and 98.54% on the Salinas dataset. The source code is available at https://github.com/mahmad000/MHSSMambaGitHub.

Rice yield predictions using remote sensing and machine learning algorithms: A review

Crop yield prediction is becoming increasingly crucial due to global food security concerns, as highlighted by recent reports from the World Health Organization. Accurate early predictions can mitigate famine risks by estimating food supply, which is essential for 820 million people facing hunger globally. Rice is the primary staple food consumed worldwide; therefore, global rice yield and rice area are monitored using emerging technologies such as remote sensing (RS) and machine learning (ML). These technologies provide valuable tools for enhancing rice yield predictions. RS includes critical information on crop health, soil conditions and weather patterns. In contrast, ML algorithms analyze these datasets to identify patterns and relationships that affect yield. Integrating these technologies offers promising improvements in yield forecasting accuracy, with applications showing successful yield predictions 1-3 months before harvest. Various ML techniques, including Random Forest, Support Vector Machines and deep learning models such as LSTM (Long-Short Term Memory), have been employed, often in combination with RS data. However, these models face challenges, such as data quality, managing high-dimensional RS data and accounting for spatial and temporal variability. Despite these challenges, integrating RS and ML has significant potential for advancing precision agriculture and achieving sustainable food production. This study explores the advancements, practical applications and challenges associated with using RS and ML for rice yield prediction, emphasizing the importance of these technologies in addressing global food security and promoting sustainable agricultural practices.

Integrative functional logistic regression model for genome-wide association studies.

Progress in rapid genomic sequencing techniques have transformed the field of disease biomarker identification by offering vast genetic information. The complexity of traits is not only influenced by single genetic loci but also by interactions among multiple genetic loci. When the dimensionality of SNP data is large, identifying a significant number of genetic variants associated with diseases becomes extremely challenging. To address these high-dimensionality issues, we employed functional data analysis techniques. Because there are a lot of ordered genetic variants spread out across a small space, multiple gene variations are handled as a continuous data set rather than discrete variables in some areas. This paper introduces a novel approach for analyzing the association of multiple genes within a region, by employing an integrative functional logistic regression model. The proposed technique has shown promising results in both simulation and real data analysis, indicating its ability to generate smooth signals and accurately estimate the coefficients of the function while recognizing the null regions. Integrative functional logistic regression method adopt functional data analysis and assume that high-dimensional genetic data follow a continuous process. It not only naturally accommodates correlations among adjacent SNPs but also avoids the unstable estimation of a large number of parameters. This is especially desirable with the rapidly increasing dimensions of SNP data but still limited sample size. In summary, the suggested approach offers a valuable new avenue for identifying disease-related genetic variants in GWAS.

Bio-primed machine learning to enhance discovery of relevant biomarkers

Precision medicine relies on identifying reliable biomarkers for gene dependencies to tailor individualized therapeutic strategies. The advent of high-throughput technologies presents unprecedented opportunities to explore molecular disease mechanisms but also challenges due to high dimensionality and collinearity among features. Traditional statistical methods often fall short in this context, necessitating novel computational approaches that harness the full potential of big data in bioinformatics. Here, we introduce a novel machine learning approach extending the Least Absolute Shrinkage and Selection Operator (LASSO) regression framework to incorporate biological knowledge, such as protein-protein interaction databases, into the regularization process. This bio-primed approach prioritizes variables that are both statistically significant and biologically relevant. Applying our method to multiple dependency datasets, we identified biomarkers which traditional methods overlooked. Our biologically informed LASSO method effectively identifies relevant biomarkers from high-dimensional collinear data, bridging the gap between statistical rigor and biological insight. This method holds promise for advancing personalized medicine by uncovering novel therapeutic targets and understanding the complex interplay of genetic and molecular factors in disease.

AccFIT-IDS: accuracy-based feature inclusion technique for intrusion detection system

In today’s digital landscape, maintaining security has become increasingly difficult. Many organizations implement security mechanisms to protect their valuable resources from potential threats. One such mechanism is the Intrusion Detection System (IDS), which examines network traffic to identify and prevent attack. However, the vast amount of network data requires an efficient processing approach. Among various studies from existing literature, Machine Learning (ML) algorithms provide a viable solution to detecting intrusion in voluminous data. ML based IDS still faces challenges, particularly due to high-dimensional data complexity, which affects the performance. To address this, the study proposed AccFIT (Accuracy-based Feature Inclusion Technique) for IDS, combining two-stage Feature Selection Algorithms (FSA). In stage 1, various filter-based feature selection methods are applied extract features from intial dataset. The derived features are concatenated and redundant features are excluded to produce an intermediary feature subset (Fintemediary). In stage 2, the Fintemediary is fed to a wrapper-based selection algorithm to derive an optimal subset Foptimal. Features are included or excluded based on their impact on model accuracy. The proposed AccFIT-IDS achieves 99.86% accuracy, with precision, recall and F1-score of 99.81%, 99.92% and 99.86%, respectively, and a false alarm rate of 0.0021, demonstrating its effectiveness.

A consensus-constrained parsimonious Gaussian mixture model for clustering hyperspectral images

Abstract The use of hyperspectral imaging to investigate food samples has grown due to the improved performance and lower cost of instrumentation. Food engineers use hyperspectral images to classify the type and quality of a food sample, typically using classification methods. In order to train these methods, every pixel in each training image needs to be labelled. Typically, computationally cheap threshold-based approaches are used to label the pixels, and classification methods are trained based on those labels. However, threshold-based approaches are subjective and cannot be generalized across hyperspectral images taken in different conditions and of different foods. Here a consensus-constrained parsimonious Gaussian mixture model (ccPGMM) is proposed to label pixels in hyperspectral images using a model-based clustering approach. The ccPGMM utilizes information that is available on some pixels and specifies constraints on those pixels belonging to the same or different clusters while clustering the rest of the pixels in the image. A latent variable model is used to represent the high-dimensional data in terms of a small number of underlying latent factors. To ensure computational feasibility, a consensus clustering approach is employed, where the data are divided into multiple randomly selected subsets of variables and constrained clustering is applied to each data subset; the clustering results are then consolidated across all data subsets to provide a consensus clustering solution. The ccPGMM approach is applied to simulated datasets and real hyperspectral images of three types of puffed cereal, corn, rice, and wheat. Improved clustering performance and computational efficiency are demonstrated when compared to other current state-of-the-art approaches.

Enhanced Multilinear PCA for Efficient Image Analysis and Dimensionality Reduction: Unlocking the Potential of Complex Image Data

This paper presents an Enhanced Multilinear Principal Component Analysis (EMPCA) algorithm, an improved variant of the traditional Multilinear Principal Component Analysis (MPCA) tailored for efficient dimensionality reduction in high-dimensional data, particularly in image analysis tasks. EMPCA integrates random singular value decomposition to reduce computational complexity while maintaining data integrity. Additionally, it innovatively combines the dimensionality reduction method with the Mask R-CNN algorithm, enhancing the accuracy of image segmentation. Leveraging tensors, EMPCA achieves dimensionality reduction that specifically benefits image classification, face recognition, and image segmentation. The experimental results demonstrate a 17.7% reduction in computation time compared to conventional methods, without compromising accuracy. In image classification and face recognition experiments, EMPCA significantly enhances classifier efficiency, achieving comparable or superior accuracy to algorithms such as Support Vector Machines (SVMs). Additionally, EMPCA preprocessing exploits latent information within tensor structures, leading to improved segmentation performance. The proposed EMPCA algorithm holds promise for reducing image analysis runtimes and advancing rapid image processing techniques.

Identification of Clusters in a Population With Obesity Using Machine Learning: Secondary Analysis of The Maastricht Study.

Modern lifestyle risk factors, like physical inactivity and poor nutrition, contribute to rising rates of obesity and chronic diseases like type 2 diabetes and heart disease. Particularly personalized interventions have been shown to be effective for long-term behavior change. Machine learning can be used to uncover insights without predefined hypotheses, revealing complex relationships and distinct population clusters. New data-driven approaches, such as the factor probabilistic distance clustering algorithm, provide opportunities to identify potentially meaningful clusters within large and complex datasets. This study aimed to identify potential clusters and relevant variables among individuals with obesity using a data-driven and hypothesis-free machine learning approach. We used cross-sectional data from individuals with abdominal obesity from The Maastricht Study. Data (2971 variables) included demographics, lifestyle, biomedical aspects, advanced phenotyping, and social factors (cohort 2010). The factor probabilistic distance clustering algorithm was applied in order to detect clusters within this high-dimensional data. To identify a subset of distinct, minimally redundant, predictive variables, we used the statistically equivalent signature algorithm. To describe the clusters, we applied measures of central tendency and variability, and we assessed the distinctiveness of the clusters through the emerged variables using the F test for continuous variables and the chi-square test for categorical variables at a confidence level of α=.001. We identified 3 distinct clusters (including 4128/9188, 44.93% of all data points) among individuals with obesity (n=4128). The most significant continuous variable for distinguishing cluster 1 (n=1458) from clusters 2 and 3 combined (n=2670) was the lower energy intake (mean 1684, SD 393 kcal/day vs mean 2358, SD 635 kcal/day; P<.001). The most significant categorical variable was occupation (P<.001). A significantly higher proportion (1236/1458, 84.77%) in cluster 1 did not work compared to clusters 2 and 3 combined (1486/2670, 55.66%; P<.001). For cluster 2 (n=1521), the most significant continuous variable was a higher energy intake (mean 2755, SD 506.2 kcal/day vs mean 1749, SD 375 kcal/day; P<.001). The most significant categorical variable was sex (P<.001). A significantly higher proportion (997/1521, 65.55%) in cluster 2 were male compared to the other 2 clusters (885/2607, 33.95%; P<.001). For cluster 3 (n=1149), the most significant continuous variable was overall higher cognitive functioning (mean 0.2349, SD 0.5702 vs mean -0.3088, SD 0.7212; P<.001), and educational level was the most significant categorical variable (P<.001). A significantly higher proportion (475/1149, 41.34%) in cluster 3 received higher vocational or university education in comparison to clusters 1 and 2 combined (729/2979, 24.47%; P<.001). This study demonstrates that a hypothesis-free and fully data-driven approach can be used to identify distinguishable participant clusters in large and complex datasets and find relevant variables that differ within populations with obesity.

Personalizing Dietary Polyphenols for Health Maintenance and Disease Management: A Nutrigenetic Approach.

This literature review provides examples of the influence of certain genetic variants on health outcomes after dietary polyphenol consumption or supplementation. Available evidence is organized according to the major classes of polyphenols (flavonoids, phenolic acids, stilbenes, lignans, and tannins) and their derived subgroups. Nutrigenetic studies have identified mainly single nucleotide polymorphisms located within genes involved in the biotransformation of phenolic acids, stilbenes, lignans and several flavonoid molecules. These genetic variants may affect polyphenol metabolism rates and related predisposition to chronic non-communicable diseases. Moreover, differential cardiometabolic outcomes upon polyphenol supplementation as dietary sources or nutraceuticals have been modulated by specific genotypes. Although current evidence is still limited, growing gene-polyphenol interactions are contributing to systematically elucidate the biological functions of polyphenols; determine individual risk phenotypes to specific diseases or particular responses upon polyphenol exposure; and facilitate the prescription of personalized genotype-based doses of dietary polyphenols to optimize related health benefits. Additionally, the integration of genetics with other omics insights (epigenomics, transcriptomics, metagenomics, and metabolomics) trough biological systems and high-dimensional data analyses and interpretation may provide a more comprehensive understanding of polyphenol metabolism for precision nutrition applications in health and disease.

Deep Learning-Based Point Cloud Classification of Obstacles for Intelligent Vehicles

Intelligent driving research has focused much attention on point cloud obstacles since they are a class of high-dimensional data that can adequately depict the shape and placement of obstacles, unlike picture data. Currently, deep learning technology is primarily employed for vehicle autonomy point cloud obstacle classification tasks. These techniques typically struggle with low classification accuracy, processing efficiency, and model stability. To tackle the abovementioned issues, this paper suggests a novel random forest algorithm that integrates the out-of-bag error theory and can consistently and accurately evaluate the influence of point cloud properties. Then, building on the novel algorithm, this paper suggests a modified PointNet network that incorporates the effects of both global and local features on the classification task, therefore increasing the conventional network’s classification accuracy. To assess the effectiveness of this novel approach in the experimental portion, we set up an evaluation system based on the metrics for average accuracy, overall accuracy, and a confusion matrix. According to the simulation results, the overall accuracy of the proposed network in terms of classification accuracy is 94.4% and the average accuracy is 84.9%, which are then compared to the prototype PointNet and its variants. The classification accuracies for the four types of obstacles are 97.6%, 63.6%, 92.5%, and 86.1%. In addition, the proposed method is effective at improving both the computational complexity and stability of the network.

GEE analysis in joint mean-covariance model for high-dimensional longitudinal data with HPC

High-dimensional covariate longitudinal data is common in various fields. However, estimating a low-dimensional parameter of interest can be challenging due to the presence of high-dimensional nuisance parameters. Traditional methods using generalized estimating equations become ineffective for estimating mean parameters when the working covariance structure is misspecified. In this paper, we first construct generalized estimating equations for regression parameters in a joint mean-covariance model for fixed-dimensional longitudinal data, based on reparameterization of the correlation structure using angles or hyperspherical coordinates. We then utilize a decorrelated matrix to handle parameter estimation for high-dimensional longitudinal data. This model addresses both the parsimonious modelling of correlations without constraints and the challenge of high-dimensional nuisance parameters. The resulting estimators are shown to be consistent and asymptotically normal, with data simulations supporting the effectiveness of the proposed approach.

An improved Grey Wolf Optimization based heuristic initialization algorithm for feature selection in P2P lending default prediction

This research addressed the problem of irrelevant feature and high-dimensional data in Peer-to-Peer (P2P) lending, which made the process hard to predict default accurately. By focusing on the right feature, the model performed better while irrelevant feature just added noise and lower accuracy. High-dimensional data created a problem known as ‘curse of dimensionality.’ This issue increased computational complexity and reduced ability of the model to work well with new data. Therefore, picking the right feature was essential to better predictions and faster computing. In this research, the finding used Grey Wolf Optimization (GWO) algorithm for selecting feature. However, GWO had flaw, starting with suboptimal initial solutions, which were not often the best option. The research incorporated Ant Colony Optimization (ACO) algorithm for better population initialization to overcome this limitation. ACO used pheromone trails and heuristics to find good starting solutions, thereby improving performance of GWO. During the analysis, the proposed model known as improved GWO+ACO, was tested with various configurations of search agents (50, 100, and 250). The tests showed that improved GWO+ACO was better than the standard GWO in terms of accuracy and stability across all configurations. Improved GWO+ACO maintained a steady accuracy of 91% at all search agent levels. In comparison, standard GWO had varying accuracy, including 85%, 90%, and 91% with 50, 100, as well as 250 search agents, respectively. Generally, using ACO for the starting point made the model less dependent on the number of search agents and also improved the optimization process significantly. Therefore, this method proved to be more effective in handling complex P2P lending data and improving default prediction accuracy.

Robustness Analysis and Forecasting of High-Dimensional Financial Time Series Data

Robustness Analysis and Forecasting of High-Dimensional Financial Time Series Data

Generalizing the Cross Product to N Dimensions: A Novel Approach for Multidimensional Analysis and Applications

This paper presents a generalization of the cross product to N dimensions, extending the classical operation beyond its traditional confines in three-dimensional space. By redefining the cross product to accommodate N−1 arguments in N dimensions, a framework has been established that retains the core properties of orthogonality, magnitude, and anticommutativity. The proposed method leverages the determinant approach and introduces the polar sine function to calculate the magnitude of the cross product, linking it directly to the volume of an N-dimensional parallelotope. This generalization not only enriches the theoretical foundation of vector calculus but also opens up new applications in high-dimensional data analysis, machine learning, and multivariate time series. The results suggest that this extension of the cross product could serve as a powerful tool for modeling complex interactions in multi-dimensional spaces, with potential implications across various scientific and engineering disciplines.

Scalable Anomaly Detection with Machine Learning: Techniques for Managing High-Dimensional Data Streams

The increase in big data values from industries, especially in Analytics, Risk, and Management Information systems, offers a great catchment and, at the same time, books with a lot of potential hurdles. This paper provides the conceptual basis for combining unsupervised and deep learning for real-time anomaly detection in high feature-space trajectories and offers practical applications for several industries. Autoencoders, Isolation Forests, and RNNs are tested for their performance and compared with the PCA in finance, manufacturing, healthcare, and cybersecurity applications. Other benefits are measured in specifics, such as cutting fraud detection mistakes by 25% and increasing the effectiveness of predictive maintenance. Further, the examined study focuses on problems like big data, real-time streaming data, distributed computing, edge computing, and incremental learning. Some key contributions are the scalable methods for IoT system-driven scenarios and the possibility of quickly recognizing anomalies with low latency. The paper also outlines emerging aspects of data quality, model interpretability, data privacy, and possible strategies such as Explainable AI and data masking. Examples from fraud detection and equipment failure prediction further explain the various opportunities these models offer for making decisions more effectively in complex and evolving environments.

Predicting electrocatalytic urea synthesis using a two-dimensional descriptor

Electrochemical synthesis routes powered by renewable electricity can provide sustainable chemical commodities by replacing conventional fossil-based processes. Increasing research focuses on value-added chemicals like the indispensable fertilizer urea, which also constitutes a study case for electrochemical CN-coupling. To guide the identification of highly selective catalysts, we aim to provide new insight by analysing existing experimental data on the selectivity of transition metal catalysts towards electrochemically synthesized urea. Firstly, we project high dimensional experimental data using principal component analysis (PCA) to lower dimensions, and thereby confirm that urea selectivity is correlated with the selectivity towards CO and NH3. Furthermore, we identified the most suitable two-dimensional descriptors for selectivity prediction out of various adsorption energies calculated using density functional theory (DFT). We suggest that the adsorption energies of *H and *O on transition metal slabs predict the selectivity towards urea in the co-reduction of CO2 and nitrite (NO2−).

A comparative evaluation of clustering-based outlier detection

Abstract We perform an extensive experimental evaluation of clustering-based outlier detection methods. These methods offer benefits such as efficiency, the possibility to capitalize on more mature evaluation measures, more developed subspace analysis for high-dimensional data and better explainability, and yet they have so-far been neglected in literature. To our knowledge, our work is the first effort to analytically and empirically study their advantages and disadvantages. Our main goal is to evaluate whether or not clustering-based techniques can compete in efficiency and effectiveness against the most studied state-of-the-art algorithms in the literature. We consider the quality of the results, the resilience against different types of data and variations in parameter configuration, the scalability, and the ability to filter out inappropriate parameter values automatically based on internal measures of clustering quality. It has been recently shown that several classic, simple, unsupervised methods surpass many deep learning approaches and, hence, remain at the state-of-the-art of outlier detection. We therefore study 14 of the best classic unsupervised methods, in particular 11 clustering-based methods and 3 non-clustering-based ones, using a consistent parameterization heuristic to identify the pros and cons of each approach. We consider 46 real and synthetic datasets with up to 125k points and 1.5k dimensions aiming to achieve plausibility with the broadest possible diversity of real-world use cases. Our results indicate that the clustering-based methods are on par with (if not surpass) the non-clustering-based ones, and we argue that clustering-based methods like KMeans−− should be included as baselines in future benchmarking studies, as they often offer a competitive quality at a relatively low run time, besides several other benefits.