High-dimensional Data Research Articles

Lipschitz-Regularized Gradient Flows and Generative Particle Algorithms for High-Dimensional Scarce Data

Lipschitz-Regularized Gradient Flows and Generative Particle Algorithms for High-Dimensional Scarce Data

Testing the Mean Vector for High-Dimensional Data

Testing the Mean Vector for High-Dimensional Data

Enhancing Clustering Stability and Efficiency: A Framework for Optimizing K-means, K-medoids, and K-shape with Intelligent Algorithms

Clustering methods like Kmeans often produce inconsistent results due to the random initialization of cluster centroids, even with optimizations such as Kmeans++, which improve centroid selection but fail to eliminate sensitivity to initialization randomness. Additionally, the choice of the number of clusters and distance metrics significantly impacts clustering performance. This paper proposes an enhanced framework combining intelligent optimization algorithms—Sparrow Search Algorithm (SSA), Dung Beetle Optimizer (DBO), and Sine Cosine Algorithm (SCA)—to optimize clustering outcomes for Kmeans, Kmedoids, and Kshape. The framework also incorporates dimensionality reduction techniques, including Principal Component Analysis (PCA), Non-Negative Matrix Factorization (NNMF), and Singular Value Decomposition (SVD), to address high-dimensional data challenges. Experimental results on benchmark datasets demonstrate the proposed framework’s effectiveness, with SSA achieving the highest silhouette score of 0.68 and reducing runtime by 35% compared to traditional methods. This approach enhances clustering stability and accuracy, offering a robust solution for diverse applications.

Analysis of Fertility Determinants and Regional Disparities in Nigeria using Geo-Additive Regression

This study applied geo-additive regression modelling on high-dimensional data with metrical and categorical predictors, using data from the latest Nigeria Demographic and Health Survey (NDHS-6). The sampled data comprises ninety-three predictors of total children ever born by eighty women of age group between fifteen and forty-nine years. Three penalty-based variable selection criteria—Elastic Net, Smoothly Clipped Absolute Deviation (SCAD), and Minimax Concave Penalty (MCP) were employed to reduce the dimensionality of the data. Geo-additive regression model were thereafter applied on the selected predictors to determine the impact of metrical, categorical and spatial predictors on the response variable. Findings revealed that predictors such as contraceptive use, age at first birth, and marital status are major determinants of total children ever born by a woman in Nigeria. Furthermore, spatial analysis revealed regional disparities in fertility rates within Nigeria, with notably higher fertility rate in northeastern states. This study’s findings have broad applications across disciplines. By providing robust methodologies for handling complex datasets, this research supports evidence-based decision-making in public health, agriculture, and environmental policy. Ultimately, these findings contribute to efforts aimed at promoting sustainable development and enhancing maternal and child health outcomes in Nigeria and similar contexts globally.

Exact parallelized dynamic mode decomposition with Hankel matrix for large-scale flow data

An exact parallel algorithm of dynamic mode decomposition (DMD) with Hankel matrices for large-scale flow data is proposed. The proposed algorithm enables the DMD and the Hankel DMD for large-scale data obtained by high-fidelity flow simulations, such as large-eddy simulations or direct numerical simulations using more than a billion grid points, on parallel computations without any approximations. The proposed algorithm completes the computations of the DMD by utilizing block matrices of XTX∈Rk×k (where X∈Rn×k is a large data matrix obtained by high-fidelity simulations, the number of snapshot data is n≳109 rsim 10^9$$\\end{document}]]>, and the number of snapshots is k≲O(103)) without any approximations: for example, the singular value decomposition of X is replaced by the eigenvalue decomposition of XTX. Then, the computation of XTX is parallelized by utilizing the domain decomposition often used in flow simulations, which reduces the memory consumption for each parallel process and wall-clock time in the DMD by a factor approximately equal to the number of parallel processes. The parallel computation with communication is performed only for XTX, allowing for high parallel efficiency under massively parallel computations. Furthermore, the proposed exact parallel algorithm is extended to the Hankel DMD without any additional parallel computations, realizing the Hankel DMD of large-scale data collected by over a billion grid points with comparable cost and memory to the DMD without Hankel matrices. Moreover, this study shows that the Hankel DMD, which has been employed to enrich information and augment rank, is advantageous for large-scale high-dimensional data collected by high-fidelity simulations in data reconstruction and predictions of future states (while prior studies have reported such advantages for low-dimensional data). Several numerical experiments using large-scale data, including laminar and turbulent flows around a cylinder and transonic buffeting flow around a full aircraft configuration, demonstrate that (i) the proposed exact parallel algorithm reproduces the existing non-parallelized Hankel DMD, (ii) the Hankel DMD for large-scale data consisting of over a billion grid points is feasible by using the proposed exact parallel algorithm with high parallel efficiency on more than 6 thousand CPU cores, and (iii) the Hankel DMD has advantages for high-dimensional data such as n≳109 rsim 10^9$$\\end{document}]]>.Graphical abstract

Iris Recognition Using Artificial Neural Network

Iris recognition has emerged as one of the most widely used and effective techniques for personal verification. It leverages the unique features of the human eye, specifically the distinct patterns found in the iris or, more precisely, the pupil. Biometric systems based on iris scanning analyze these unique iris patterns to confirm and authenticate an individual's identity. They are widely used as human eye has unique features particularly the iris. Human iris have random variations and complex texture which are stable with time even visible between identical twins. It’s replication and forgery are almost impossible. For training and testing our model we used CASIA-Iris-Interval database as camera used to collect this data uses a circular NIR LED array [1].Near Infrared LED has a suitable luminous flux for imaging human iris for recognition. Our results demonstrate that the ANN model outperforms traditional machine learning algorithms, such as Support Vector Machines (SVM) and k-Nearest Neighbors (k-NN), in terms of both accuracy and robustness [5]. The model showed high accuracy in identifying iris patterns, even under varying conditions like changes in lighting, pupil dilation, and partial occlusions. This highlights the potential of deep learning techniques to overcome the challenges faced by conventional biometric systems, which often struggle with high-dimensional data and manual feature extraction [18].

Automatic extraction of fine structural information in angle-resolved photoemission spectroscopy by multi-stage clustering algorithm

Unsupervised clustering method has shown strong capabilities in automatically categorizing the ARPES (ARPES: angle-resolved photoemission spectroscopy) spatial mapping dataset. However, there is still room for improvement in distinguishing subtle differences caused by different layers and substrates. Here, we propose a method called Multi-Stage Clustering Algorithm (MSCA). Using the K-means clustering results/metrics for real space in different energy-momentum windows as the input of the second round K-means clustering for momentum space, the energy-momentum windows that exhibit subtle inhomogeneity in real space will be highlighted. It recognizes different types of electronic structures both in real space and momentum space in spatially resolved ARPES dataset. This method can be used to capture the areas of interest, and is especially suitable for samples with complex band dispersions, and can be a practical tool to any high dimensional scientific data analysis.

Optimizing Personalized and Context-Aware Recommendations in Pervasive Computing Environments

The researchers in the current era provided many new recommendation methodologies. Though various recommendation techniques exist, there is a need to develop a unique technique for capturing latent factors and patterns from sparse and high-dimensional data in pervasive environments, specifically for optimizing dynamic recommendations. This study proposes a hybrid approach for optimizing dynamic recommendations in pervasive environments by combining Non-Negative Matrix Factorization (NMF) with deep learning and reinforcement learning techniques. The goal is to overcome the challenge of capturing latent factors and patterns from sparse and high-dimensional data. By leveraging NMF, meaningful latent factors are extracted, while deep learning, specifically Faster recurrent neural networks (FRNNs), learns complex feature representations. Reinforcement learning algorithms optimize the recommendation policy based on user feedback. This Hybrid Context-Aware Optimized Recommendation (HCOR) approach improves recommendation accuracy and relevance in pervasive environments, adapts to changing contexts, and enhances user experiences. The performance benefits are achieved by effectively capturing latent factors and patterns, resulting in improved accuracy and the ability to provide personalized and context-aware recommendations. The performance indicators to validate the research work include the recommendations' accuracy, relevance, and adaptability in pervasive environments. Additionally, metrics, such as precision, recall, and F1-score, are used to evaluate the effectiveness of the hybrid approach in capturing latent factors and patterns. User feedback and satisfaction are also measured to assess the impact on user experiences. The HCOR approach shows substantial performance gains, measuring a precision of 0.932, a recall of 0.922, and an F1-score of 0.943, which indicates the excellent ability of the approach to deliver accurate and personalized recommendations in a pervasive environment.

Riemannian manifold-aided data-driven diagnosis of photovoltaic hot spots

Abstract This paper proposes a novel data-driven approach for hot-spot fault detection in photovoltaic (PV) modules, utilizing a curved Riemannian manifold (RM) to characterize the sample space. The proposed method detects hot spots in PV systems by mining geometric features in high-dimensional data. The RM-based method combines the geometric properties of RMs for fault detection. In addition, it also provides information about the severity of the hot spots. The proposed method has three main advantages: (1) it is capable of actively learning fault characteristics and can be applied to diagnose various degrees of hot spots; (2) unlike other data-driven methods, the proposed method considers non-Euclidean spatial data features, which further improves the accuracy of the method and reduces the false alarm rate; (3) it does not rely on mathematical models or expert knowledge, making it able to meet the real-time monitoring needs of actual PV systems. The effectiveness and superiority of the proposed approach are verified through 12 sets of hot spots tests conducted on a PV experimental platform.

A Lightweight Transformer-Based Spatiotemporal Analysis Prediction Algorithm for High-Dimensional Meteorological Data

High-dimensional meteorological data offer a comprehensive overview of meteorological conditions. Nevertheless, predicting regional high-dimensional meteorological data poses challenges due to the vast scale and rapid changes. Apart from slow conventional numerical weather prediction methods, recently developed deep learning methods often fail to fully integrate spatial information of the high-dimensional data and require a significant amount of computational resources. This paper presents the spatiotemporal analysis fitting prediction algorithm (SA-Fit), an approximation algorithm for regional high-dimensional meteorological data prediction. SA-Fit proposes two key designs to achieve efficient prediction of the high-dimensional data. SA-Fit introduces a lightweight Transformer-based spatiotemporal analysis network to encode spatiotemporal information, which can integrate the interaction information between different coordinates in the data. Furthermore, SA-Fit introduces explicit functions with a lasso penalty to fit variations in high-dimensional meteorological data, achieving the prediction of a large amount of data with minimal prediction values. We performed experiments using the ERA5 dataset from the Shanghai and Xi’an regions. The experimental results show that SA-Fit is comparable to other advanced deep learning prediction methods in overall prediction performance. SA-Fit shortens training time and significantly reduces model parameters while using the Transformer structure to ensure prediction accuracy.

Interpreting single-cell and spatial omics data using deep neural network training dynamics.

Single-cell and spatial omics datasets can be organized and interpreted by annotating single cells to distinct types, states, locations or phenotypes. However, cell annotations are inherently ambiguous, as discrete labels with subjective interpretations are assigned to heterogeneous cell populations on the basis of noisy, sparse and high-dimensional data. Here we developed Annotatability, a framework for identifying annotation mismatches and characterizing biological data structure by monitoring the dynamics and difficulty of training a deep neural network over such annotated data. Following this, we developed a signal-aware graph embedding method that enables downstream analysis of biological signals. This embedding captures cellular communities associated with target signals. Using Annotatability, we address key challenges in the interpretation of genomic data, demonstrated over eight single-cell RNA sequencing and spatial omics datasets, including identifying erroneous annotations and intermediate cell states, delineating developmental or disease trajectories, and capturing cellular heterogeneity. These results underscore the broad applicability of annotation-trainability analysis via Annotatability for unraveling cellular diversity and interpreting collective cell behaviors in health and disease.

An adaptive test of the independence of high-dimensional data based on Kendall rank correlation coefficient

The paper considers the test on the components independence of high-dimensional continuous random vectors based on the Kendall rank correlation coefficient. Under the null hypothesis, it proves that the sum-of-powers type statistics constructed by the Kendall rank correlation coefficient are asymptotically normally distributed and independent when the sum of the orders is odd, and they are also independent with the extreme-value-type test statistics of the Kendall rank correlation coefficient. Based on the asymptotic independence property, it proposes an adaptive test procedure which combines p-values from the sum-of-powers-type test and extreme-value-type test. Some simulations reveal that the proposed test method can significantly enhance the statistical power of the test while keep the type I error well controlled, and it is powerful against a wide range of alternatives (including dense, sparse or moderately sparse signals). To illustrate the use of the proposed adaptive test method, two real data sets are also analysed.

Machine learning-based variable selection for clustered credit risk modeling

AbstractSeveral studies have demonstrated the high prediction accuracy of clustered credit risk modeling. In clustered modeling, borrowers are segmented based on their similarities through cluster analysis, and a separate predictive model is developed for each cluster, resulting in increased predictive accuracy. Unambiguously, its effectiveness depends on the quality of the segmentation, which in turn depends primarily on the choice of variables used in the cluster analysis. However, appropriate variable selection for clustering is a major challenge, particularly for high-dimensional data. In the present study, we propose a machine learning-based variable selection method based on theoretical and regulatory considerations. Formally, the most influential risk drivers from a best-in-class machine learning model are identified using Shapley values and employed as clustering variables. Thus, the information of the explanatory variables crucial for the prediction of the dependent variable is already processed during data segmentation, making each individual predictive model more effective. Through a comparative analysis using two real-world credit default datasets, we show that our proposed approach to clustered modeling leads to the highest prediction accuracy among various clustering models.

Deep interval type-2 generalized fuzzy hyperbolic tangent system for nonlinear regression prediction

Recently, due to the rapid rise of artificial intelligence (AI), considerable progress has been made in the field of nonlinear regression prediction. However, many existing methods suffer from the issues of rule and parameter explosion and poor accuracy, particularly for high-dimensional data with uncertainty. To address these limitations, this paper proposes a deep interval type-2 generalized fuzzy hyperbolic tangent system (DIT2GFHS). First, a novel neural network-based implementation of the interval type-2 fuzzy generalized fuzzy hyperbolic tangent system (IT2GFHS) is introduced to improve the efficiency of system parameter updates and optimization. Then, using a hierarchical and block-based framework, multiple IT2GFHSs are stacked layer by layer from bottom to top to construct the DIT2GFHS, with each layer’s fuzzy subsystems being independent of the others. Additionally, DIT2GFHS incorporates optimization algorithms and the Adam optimizer for training, thereby avoiding the tedious manual parameter tuning process. The detailed analysis of the construction manner and internal mechanisms for DIT2GFHS indicates that it features a reduced number of parameters, a transparent and clear structure, strong capability in handling uncertainty, and favorable accuracy. Notably, the small number of parameters and the explicit structure reduce computational and hardware burdens while maintaining interpretability. Finally, extensive experimental studies on both relatively low-dimensional and high-dimensional datasets are conducted. The results demonstrate that DIT2GFHS achieves excellent performance with fewer parameters compared to many deep-structured models, including deep fuzzy systems and deep learning models. This highlights its potential impact in addressing practical nonlinear regression problems.

Hyperspectral and multispectral images fusion based on pyramid swin transformer

Remote sensing image fusion aims to generate a high spatial resolution hyperspectral image (HR-HSI) by integrating a low spatial resolution hyperspectral image (LR-HSI) and a high spatial resolution multispectral image (HR-MSI). While Convolutional Neural Networks (CNNs) have been widely employed in addressing the HSI-MSI fusion problem, their limited receptive field poses challenges in capturing global relationships within the feature maps. On the other hand, the computational complexity of Transformers hinders their application, especially in dealing with high-dimensional data like hyperspectral images (HSIs). To overcome this challenge, we propose an HSI-MSI fusion method based on the Pyramid Swin Transformer (PSTF). The pyramid design of the PSTF effectively extracts multi-scale information from images. The Spatial-Spectral Crossed Attention (SSCA) module, comprising the Cross Spatial Attention (CSA) and the Spectral Feature Integration (SFI) modules. The CSA module employs a cross-shaped self-attention mechanism, providing greater modeling flexibility for different spatial scales and non-local structures compared to traditional convolutional layers. Meanwhile, the SFI module introduces a global memory block (MB) to select the most relevant low-rank spectral vectors, integrating global spectral information with local spatial–spectral correlation to better extract and preserve spectral information. Additionally, the Separate Feature Extraction (SFE) module enhances the network’s ability to represent image features by independently processing positive and negative parts of shallow features, thus capturing details and structures more effectively and preventing the vanishing gradient problem. Compared with the state-of-the-art (SOTA) methods, experimental results demonstrate the effectiveness of the PSTF method.

A flexible class of priors for orthonormal matrices with basis function-specific structure

Statistical modeling of high-dimensional matrix-valued data motivates the use of a low-rank representation that simultaneously summarizes key characteristics of the data and enables dimension reduction. Low-rank representations commonly factor the original data into the product of orthonormal basis functions and weights, where each basis function represents an independent feature of the data. However, the basis functions in these factorizations are typically computed using algorithmic methods that cannot quantify uncertainty or account for basis function correlation structure a priori. While there exist Bayesian methods that allow for a common correlation structure across basis functions, empirical examples motivate the need for basis function-specific dependence structure. We propose a prior distribution for orthonormal matrices that can explicitly model basis function-specific structure. The prior is used within a general probabilistic model for singular value decomposition to conduct posterior inference on the basis functions while accounting for measurement error and fixed effects. We discuss how the prior specification can be used for various scenarios and demonstrate favorable model properties through synthetic data examples. Finally, we apply our method to two-meter air temperature data from the Pacific Northwest, enhancing our understanding of the Earth system’s internal variability.

Tensor dictionary-based heterogeneous transfer learning to study emotion-related gender differences in brain

In practice, collecting auxiliary labeled data with same feature space from multiple domains is difficult. Thus, we focus on the heterogeneous transfer learning to address the problem of insufficient sample sizes in neuroimaging. Viewing subjects, time, and features as dimensions, brain activation and dynamic functional connectivity data can be treated as high-order heterogeneous data with heterogeneity arising from distinct feature space. To use the heterogeneous priori knowledge from the low-dimensional brain activation data to improve the classification performance of high-dimensional dynamic functional connectivity data, we propose a tensor dictionary-based heterogeneous transfer learning framework. It combines supervised tensor dictionary learning with heterogeneous transfer learning for enhance high-order heterogeneous knowledge sharing. The former can encode the underlying discriminative features in high-order data into dictionaries, while the latter can transfer heterogeneous knowledge encoded in dictionaries through feature transformation derived from mathematical relationship between domains. The primary focus of this paper is gender classification using fMRI data to identify emotion-related brain gender differences during adolescence. Additionally, experiments on simulated data and EEG data are included to demonstrate the generalizability of the proposed method. Experimental results indicate that incorporating prior knowledge significantly enhances classification performance. Further analysis of brain gender differences suggests that temporal variability in brain activity explains differences in emotion regulation strategies between genders. By adopting the heterogeneous knowledge sharing strategy, the proposed framework can capture the multifaceted characteristics of the brain, improve the generalization of the model, and reduce training costs. Understanding the gender specific neural mechanisms of emotional cognition helps to develop the gender-specific treatments for neurological diseases.

QKSAN: A Quantum Kernel Self-Attention Network.

The Self-Attention Mechanism (SAM) excels at distilling important information from the interior of data to improve the computational efficiency of models. Nevertheless, many Quantum Machine Learning (QML) models lack the ability to distinguish the intrinsic connections of information like SAM, which limits their effectiveness on massive high-dimensional quantum data. To tackle the above issue, a Quantum Kernel Self-Attention Mechanism (QKSAM) is introduced to combine the data representation merit of Quantum Kernel Methods (QKM) with the efficient information extraction capability of SAM. Further, a Quantum Kernel Self-Attention Network (QKSAN) framework is proposed based on QKSAM, which ingeniously incorporates the Deferred Measurement Principle (DMP) and conditional measurement techniques to release half of quantum resources by mid-circuit measurement, thereby bolstering both feasibility and adaptability. Simultaneously, the Quantum Kernel Self-Attention Score (QKSAS) with an exponentially large characterization space is spawned to accommodate more information and determine the measurement conditions. Eventually, four QKSAN sub-models are deployed on PennyLane and IBM Qiskit platforms to perform binary classification on MNIST and Fashion MNIST, where the QKSAS tests and correlation assessments between noise immunity and learning ability are executed on the best-performing sub-model. The paramount experimental finding is that the QKSAN subclasses possess the potential learning advantage of acquiring impressive accuracies exceeding 98.05% with far fewer parameters than classical machine learning models. Predictably, QKSAN lays the foundation for future quantum computers to perform machine learning on massive amounts of data while driving advances in areas such as quantum computer vision.

Many-Objective Jaccard-Based Evolutionary Feature Selection for High-Dimensional Imbalanced Data Classification.

Filters and wrappers represent two mainstream approaches to feature selection (FS). Although evolutionary wrapper-based FS outperforms filters in addressing real-world classification problems, extending these methods to high-dimensional, many-objective optimization problems with imbalanced data poses substantial challenges. Overcoming computational costs and identifying suitable performance metrics are vital for navigating search operation complexities. Here, we propose using the Jaccard similarity (JS) in a set-based evolutionary many-objective (JSEMO) FS search, addressing both evolutionary FS and imbalanced classifier choice concurrently. This study highlights the mutual influence between these aspects, impacting overall algorithm performance. JSEMO integrates JS into population initialization, reproduction, and elitism steps, enhancing diversity and avoiding duplicate solutions. The set-based variation operator utilizes intersection and union operators for compatibility with binary coding. We also introduce a double-weighted KNN (KNN2W) classifier with four supportive objectives as a many-objective FS problem to handle imbalanced distributions. Compared with 20 methods on 15 benchmark problems, JSEMO produces distinct optimal features, significantly improving overall accuracy, balance accuracy, and g-mean metrics with comparable feature set size and computational cost. The ablation study underscores the positive impact of all JSEMO components, highlighting the set-based variation operation with JS and KNN2W with relevant evaluation metrics as the most influential aspects.

Conditional Variable Screening for Ultra-High Dimensional Longitudinal Data With Time Interactions.

In recent years, we have been able to gather large amounts of genomic data at a fast rate, creating situations where the number of variables greatly exceeds the number of observations. In these situations, most models that can handle a moderately high dimension will now become computationally infeasible or unstable. Hence, there is a need for a prescreening of variables to reduce the dimension efficiently and accurately to a more moderate scale. There has been much work to develop such screening procedures for independent outcomes. However, much less work has been done for high-dimensional longitudinal data in which the observations can no longer be assumed to be independent. In addition, it is of interest to capture possible interactions between the genomic variable and time in many of these longitudinal studies. In this work, we propose a novel conditional screening procedure that ranks variables according to the likelihood value at the maximum likelihood estimates in a marginal linear mixed model, where the genomic variable and its interaction with time are included in the model. This is to our knowledge the first conditional screening approach for clustered data. We prove that this approach enjoys the sure screening property, and assess the finite sample performance of the method throughsimulations.