High-dimensional Datasets Research Articles

Exploring heterogeneous cell population dynamics in different microenvironments by novel analytical strategy based on images.

Understanding the dynamic states and transitions of heterogeneous cell populations is crucial for addressing fundamental biological questions. High-content imaging provides rich datasets, but it remains increasingly difficult to integrate and annotate high-dimensional and time-resolved datasets to profile heterogeneous cell population dynamics in different microenvironments. Using hepatic stellate cells (HSCs) LX-2 as model, we proposed a novel analytical strategy for image-based integration and annotation to profile dynamics of heterogeneous cell populations in 2D/3D microenvironments. High-dimensional features were extracted from extensive image datasets, and cellular states were identified based on feature profiles. Time-series clustering revealed distinct temporal patterns of cell shape and actin cytoskeleton reorganization. We found LX-2 showed more complex membrane dynamics and contractile systems with an M-shaped actin compactness trend in 3D culture, while they displayed rapid spreading in early 2D culture. This image-based integration and annotation strategy enhances our understanding of HSCs heterogeneity and dynamics in complex extracellular microenvironments.

A beginner's guide to supervised analysis for mass cytometry data in cancer biology.

Mass cytometry enables deep profiling of biological samples at single-cell resolution. This technology is more than relevant in cancer research due to high cellular heterogeneity and complexity. Downstream analysis of high-dimensional datasets increasingly relies on machine learning (ML) to extract clinically relevant information, including supervised algorithms for classification and regression purposes. In cancer research, they are used to develop predictive models that will guide clinical decision making. However, the development of supervised algorithms faces major challenges, such as sufficient validation, before being translated into the clinics. In this work, we provide a framework for the analysis of mass cytometry data with a specific focus on supervised algorithms and practical examples of their applications. We also raise awareness on key issues regarding good practices for researchers curious to implement supervised ML on their mass cytometry data. Finally, we discuss the challenges of supervised ML application to cancer research.

Revolutionizing Optimization: The Game-Changing Power of Quantum Algorithms and Next-Generation Computational Technologies

Quantum algorithms offer promising advancements in solving complex optimization problems that are critical in various fields, including logistics, finance, and machine learning. Classical optimization techniques, while effective, often struggle with large-scale and high-dimensional data sets, leading to inefficiencies and long processing times.Quantum algorithms, leveraging the principles of superposition, entanglement, and quantum parallelism, provide a potential solution by exponentially speeding up certain computational tasks. This paper explores key quantum algorithms for optimization, including Grover's algorithm, the Quantum Approximate Optimization Algorithm (QAOA), and the Quantum Annealing method. We analyze their theoretical foundations, implementation challenges, and practical applications. Special attention is given to the performance comparisons between quantum and classical approaches in solving combinatorial optimization problems, such as the traveling salesman problem and portfolio optimization. We also discuss the limitations of current quantum hardware and the prospects for future improvements that could make quantum optimization algorithms more practical for real-world use.Our findings suggest that, while quantum algorithms are still in their early stages, they hold immense potential for transforming optimization in a variety of domains, offering new pathways toward more efficient problem-solving in both theoretical and applied contexts. we explore how hybrid quantum-classical approaches can enhance optimization results by combining the strengths of both paradigms. Future research directions are highlighted, focusing on improving quantum error correction, scaling quantum systems, and developing more robust quantum algorithms to tackle increasingly complex optimization problems.

Impact of Balanced Exploration and Exploitation on Highdimensional Feature Selection with Hierarchical Whale Optimisation Algorithm

High-dimensional datasets (HDDs) pose significant challenges in feature selection due to their complex nature. While metaheuristic algorithms like the Whale Optimisation Algorithm (WOA) have shown promise in addressing these challenges, stability issues are least addressed. Through an extensive literature review, it was identified that stability is manifested in the equilibrium of exploration and exploitation. Hence, this study introduced the Hierarchical Whale Optimisation Algorithm (HiWOA), an approach designed to enhance the WOA’s stability and performance in HDD feature selection tasks. The HiWOA incorporates a two-phase strategy comprising a nonlinear control parameter based on the arcsine function and a hierarchical position-update mechanism adapted from the Grey Wolf Optimiser. The proposed HiWOA was evaluated through 23 benchmark optimisation functions and feature selection experiments on 11 medical HDDs. The results indicate that the HiWOA outperformed the WOA and a modified variant (mWOA) in terms of a better fitness value, a more balanced exploration-exploitation ratio, and improved classification accuracy with fewer selected features. These findings demonstrate the HiWOA’s effectiveness in enhancing stability, making it a robust solution for high-dimensional optimisation and feature selection.

Enhancing Efficacy in Breast Cancer Screening with Nesterov Momentum Optimization Techniques

In the contemporary landscape of healthcare, machine learning models are pivotal in facilitating precise predictions, particularly in the nuanced diagnosis of complex ailments such as breast cancer. Traditional diagnostic methodologies grapple with inherent challenges, including excessive complexity, elevated costs, and reliance on subjective interpretation, which frequently culminate in inaccuracies. The urgency of early detection cannot be overstated, as it markedly broadens treatment modalities and significantly enhances survival rates. This paper delineates an innovative optimization framework designed to augment diagnostic accuracy by amalgamating momentum-based optimization techniques within a neural network paradigm. Conventional machine learning approaches are often encumbered by issues of overfitting, data imbalance, and the inadequacy of capturing intricate patterns in high-dimensional datasets. To counter these limitations, we propose a sophisticated framework that integrates an adaptive threshold mechanism across an array of gradient-based optimizers, including SGD, RMSprop, adam, adagrad, adamax, adadelta, nadam and Nesterov momentum. This novel approach effectively mitigates oscillatory behavior, refines parameter updates, and accelerates convergence. A salient feature of our methodology is the incorporation of a momentum threshold for early stopping, which ceases training upon the stabilization of momentum below a pre-defined threshold, thereby pre-emptively preventing overfitting. Leveraging the Wisconsin Breast Cancer Dataset, our model achieved a remarkable 99.72% accuracy and 100% sensitivity, significantly curtailing misclassification rates compared to traditional methodologies. This framework stands as a robust solution for early breast cancer diagnosis, thereby enhancing clinical decision making and improving patient outcomes.

Research on the Feasibility of Machine Learning Methods in Stock Price Prediction

Abstract: Stock price prediction has always been an important topic in financial research. Accurate price prediction can not only help investors make informed investment decisions but also enhance market stability and reduce systemic risk. In recent years, with advancements in computing technology and data science, machine learning methods have been increasingly applied in the financial field. Compared to traditional methods, machine learning methods can better handle high-dimensional, nonlinear, and large datasets, thus demonstrating higher prediction accuracy and applicability in stock price prediction.This paper reviews relevant literature and selects five models for empirical research: Support Vector Machine (SVM), Long Short-Term Memory network (LSTM), LightGBM, a combination of LSTM and LightGBM, and Convolutional Neural Network (CNN). The effectiveness of these models in predicting the stock price of Meituan-W (3690) was analyzed and compared in detail. The experimental results show that the LightGBM model performs best in terms of Mean Squared Error (MSE), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE), proving its significant advantages in handling large-scale, high-dimensional, and nonlinear data. By comparing the prediction results of different models, this paper explores the strengths and weaknesses of each model and their feasibility and effectiveness in practical applications. Machine learning methods have significant potential in stock price prediction, model selection needs to comprehensively consider data characteristics, computational resources, and practical application scenarios.

Ensemble-Enhanced Semi-Supervised Learning with Optimized Graph Construction for High-Dimensional Data.

Graph-based methods have demonstrated exceptional performance in semi-supervised classification. However, existing graph-based methods typically construct either a predefined graph in the original space or an adaptive graph within the output space, which often limits their ability to fully utilize prior information and capture the optimal intrinsic data distribution, particularly in high-dimensional data with abundant redundant and noisy features. This paper introduces a novel approach: Semi-Supervised Classification with Optimized Graph Construction (SSC-OGC). SSC-OGC leverages both predefined and adaptive graphs to explore intrinsic data distribution and effectively employ prior information. Additionally, a graph constraint regularization term (GCR) and a collaborative constraint regularization term (CCR) are incorporated to further enhance the quality of the adaptive graph structure and the learned subspace, respectively. To eliminate the negative effect of constructing a predefined graph in the original data space, we further propose a Hybrid Subspace Ensemble-enhanced framework based on the proposed Optimized Graph Construction method (HSE-OGC). Specifically, we construct multiple hybrid subspaces, which consist of meticulously chosen features from the original data to achieve high-quality and diverse space representations. Then, HSE-OGC constructs multiple predefined graphs within hybrid subspaces and trains multiple SSC-OGC classifiers to complement each other, significantly improving the overall performance. Experimental results conducted on various high-dimensional datasets demonstrate that HSE-OGC exhibits outstanding performance.

A feedback matrix based evolutionary multitasking algorithm for high-dimensional ROC convex hull maximization

Multi-objective evolutionary algorithms have shown their competitiveness in solving ROC convex hull maximization. However, due to “the curse of dimensionality”, few of them focus on high-dimensional ROCCH maximization. Therefore, in this paper, a feedback matrix (FM)-based evolutionary multitasking algorithm, termed as FM-EMTA, is proposed. In FM-EMTA, to tackle “the curse of dimensionality”, a feature importance based low-dimensional task construction strategy is designed to transform the high-dimensional ROCCH maximization task into several low-dimensional tasks. Then, each low-dimensional task evolves with a population. To ensure that the low-dimensional task achieves a better ROCCH, an FM-based evolutionary multitasking operator is proposed. Specifically, for each low-dimensional task i, the element FM(i,j) in feedback matrix is defined to measure the degree that the low-dimensional task j could assist task i. Based on it, an FM-based assisted task selection operator and an FM-based knowledge transfer operator are developed to constitute the evolutionary multitasking operator, with which the useful knowledge is transferred among the low-dimensional tasks. After the evolution, the best ROCCHs obtained by the low-dimensional tasks are combined together to achieve the final ROCCH on the original high-dimensional task. Experiments on twelve high-dimensional datasets with different characteristics demonstrate the superiority of the proposed FM-EMTA over the state-of-the-arts in terms of the area under ROCCH, the hypervolume indicator and the running time.

A novel monitoring method based on the fusion of sound and image signals for laser welding penetration status

Accurately identifying penetration status is crucial for ensuring high-quality welding. However, relying on a single sensor to monitor the welding status falls short in accuracy. Thus, a novel monitoring method that fuses sound and image is proposed for assessing the laser welding penetration status. First, a microphone and a high-speed camera are employed to collect sound and image signals during welding. Second, to overcome noise challenges in sound signal, a denoising method is proposed that utilizes Grey Wolf Optimization (GWO) to find the optimal parameters for Variational Mode Decomposition (VMD). The time and frequency domain features from the denoised sound and Local Binary Pattern (LBP) features from the image are extracted. These features constitute a high-dimensional dataset, which is reduced using Principal Component Analysis (PCA). Finally, a back propagation neural network (BPNN) is constructed for fusion monitoring. The results demonstrate that fusion monitoring achieves higher accuracy (97.9%) compared to using sound (79.9%) or image (92.3%) features independently. In comparison to other methods like K-Nearest Neighbor (KNN), Naive Bayesian (NB), Support Vector Machine (SVM), and Decision Tree, BPNN fusion monitoring achieves higher accuracies, with increases of 18.09%, 12.79%, 6.53%, and 25.84% respectively. The proposed method helps to promote the development of welding intelligence.

Reproducibility and interpretability in radiomics: a critical assessment.

Radiomics aims to improve clinical decision making through the use of radiological imaging. However, the field is challenged by reproducibility issues due to variability in imaging and subsequent statistical analysis, which particularly affects the interpretability of the model. In fact, radiomics extracts many highly correlated features that, combined with the small sample sizes often found in radiomics studies, result in high-dimensional datasets. These datasets, which are characterized by containing more features than samples, have different statistical properties than other datasets, thereby complicating their training by machine learning and deep learning methods. This review critically examines the challenges of both reproducibility issues and interpretability, beginning with an overview of the radiomics pipeline, followed by a discussion of the imaging and statistical reproducibility issues. It further highlights how limited model interpretability hinders clinical translation. The discussion concludes that these challenges could be mitigated by following best practices and by creating large, representative, and publicly available datasets.

The Effect of Regularized Regression and Tree-Based Missing Data Imputation Methods on Classification Performance in High Dimensional Data

Missing data is an important problem in the analysis and classification of high dimensional data. The aim of this study is to compare the effects of four different missing data imputation methods on classification performance in high dimensional data. In this study, missing data imputation methods were evaluated using data sets, whose independent variables between mixed correlated with each other, for binary dependent variable, p=500 independent variables, n=150 units and 1000 times running simulation. Missing data structures were created according to different missing rates. Different datasets were obtained by imputing the missing values using different methods. Regularized regression methods such as lasso and elastic net regression were used for imputation, as well as tree-based methods such as support vector machine and classification and regression trees. At the end of simulation, the classification scores of the methods were obtained by gradient boosting machines and the missing data prediction performances were evaluated according to the distance of these scores from the reference. Our simulation demonstrates that regularized regression methods outperform tree-based methods in classifying high dimensional datasets. Additionally, it was found that the increase in the amount of missing values reduced the classification performance of the methods in high dimensional data.

An AutoEncoder enhanced light gradient boosting machine method for credit card fraud detection

Online financial transactions bring convenience to people’s lives, but also present vulnerabilities for criminals to embezzle users’ accounts and trick users into credit card fraud. Although machine learning methods have been adopted to detect anomalous transactions, it’s hard for a single machine learning method to achieve satisfying results with the increasing scale and dimensionality of financial datasets. In addition, for anomaly detection of financial data, there is an obvious imbalance between normal records and abnormal. In this situation, the experimental results cannot be objectively evaluated only by the traditional metrics, such as precision, recall, and accuracy. This paper proposes an AutoEncoder enhanced LightGBM method for credit card detection. The method inherits the advantages of each component, using an AutoEncoder for feature reconstruction on the dataset, and integrating the LightGBM algorithm for improving the GBDT (Gradient Boosting Decison Tree) to detect abnormal data more accurately and efficiently. Besides the traditional evaluation metrics, F-measure, area under curve (AUC), Matthew’s correlation coefficient (MCC), and balanced classification rate (BCR) are also adopted as the evaluation metrics. Two financial datasets were used to validate the performance and robustness of the proposed model. Results obtained from the credit card fraud dataset containing 31 features indicate that our model significantly outperforms other models with a recall of 94.85%, representing a 10.70% improvement compared to the best detection performance model with a recall of only 86%. Additionally, our model’s BCR score is also significantly better than other models, with a BCR score of 97%, as opposed to the best detection performance model’s BCR score of 92%, representing a 5% improvement by our model. Various sampling methods and model combinations were considered in this study. It was found that the SMOTE algorithm combined with the proposed model produced the best results, with an AUC value of 96.83% and an F-measure score of 80.27%. The Santander bank transaction record dataset is a high dimensional large dataset containing 200 features. Experimental results on this dataset reveal that compared to other models, our model significantly improved recall and F-measure results, raising the recall to 94.14% and the F-measure score by 11.51%, surpassing the second-best-performing model. Overall, these findings demonstrate the robustness and superiority of our model in detecting fraudulent transactions and highlight the effectiveness of the SMOTE algorithm in combination with the proposed model.

Using K-NN Algorithm for Evaluating Feature Selection on High Dimensional Datasets

Data mining is the process of using statistics, mathematics, artificial intelligence and machine learning to identify problems that exist in data so as to produce useful information. Based on its function, data mining is grouped into description, estimation, classification, clustering, and association. K-NN is one of the best data mining methods and is widely used in research. K-NN algorithm was introduced by Fix and Hodges in 1951. K-NN algorithm is a simple algorithm and is often used to cluster supervised data. Feature selection attribute selection is a data mining technique used in the pre-processing stage. This technique works by reducing complex attributes that will be managed at the processing and analysis stage. In this study, the most effective feature selection to improve the accuracy of the K-NN algorithm by increasing accuracy by 95.12% on the breast cancer dataset and 88.75% on the prostate cancer dataset.

QCLUE: a quantum clustering algorithm for multi-dimensional datasets

Clustering algorithms are at the basis of several technological applications, and are fueling the development of rapidly evolving fields such as machine learning. In the recent past, however, it has become apparent that they face challenges stemming from datasets that span more spatial dimensions. In fact, the best-performing clustering algorithms scale linearly in the number of points, but quadratically with respect to the local density of points. In this work, we introduce qCLUE, a quantum clustering algorithm that scales linearly in both the number of points and their density. qCLUE is inspired by CLUE, an algorithm developed to address the challenging time and memory budgets of Event Reconstruction (ER) in future High-Energy Physics experiments. As such, qCLUE marries decades of development with the quadratic speedup provided by quantum computers. We numerically test qCLUE in several scenarios, demonstrating its effectiveness and proving it to be a promising route to handle complex data analysis tasks – especially in high-dimensional datasets with high densities of points.

Chain hybrid feature selection algorithm based on improved Grey Wolf Optimization algorithm.

Hybrid feature selection algorithm is a strategy that combines different feature selection methods aiming to overcome the limitations of a single feature selection method and improve the effectiveness and performance of feature selection. In this paper, we propose a new hybrid feature selection algorithm, to be named as Tandem Maximum Kendall Minimum Chi-Square and ReliefF Improved Grey Wolf Optimization algorithm (TMKMCRIGWO). The algorithm consists of two stages: First, the original features are filtered and ranked using the bivariate filter algorithm Maximum Kendall Minimum Chi-Square (MKMC) to form a subset of candidate features S1; Subsequently, S1 features are filtered and sorted to form a candidate feature subset S2 by using ReliefF in tandem, and finally S2 is used in the wrapper algorithm to select the optimal subset. In particular, the wrapper algorithm is an improved Grey Wolf Optimization (IGWO) algorithm based on random disturbance factors, while the parameters are adjusted to vary randomly to make the population variations rich in diversity. Hybrid algorithms formed by combining filter algorithms with wrapper algorithms in tandem show better performance and results than single algorithms in solving complex problems. Three sets of comparison experiments were conducted to demonstrate the superiority of this algorithm over the others. The experimental results show that the average classification accuracy of the TMKMCRIGWO algorithm is at least 0.1% higher than the other algorithms on 20 datasets, and the average value of the dimension reduction rate (DRR) reaches 24.76%. The DRR reached 41.04% for 12 low-dimensional datasets and 0.33% for 8 high-dimensional datasets. It also shows that the algorithm improves the generalization ability and performance of the model.

Distance preserving machine learning for uncertainty aware accelerator capacitance predictions

Abstract Accurate uncertainty estimations are essential for producing reliable machine learning models, especially in safety-critical applications such as accelerator systems. Gaussian process models are generally regarded as the gold standard for this task; however, they can struggle with large, high-dimensional datasets. Combining deep neural networks with Gaussian process approximation techniques has shown promising results, but dimensionality reduction through standard deep neural network layers is not guaranteed to maintain the distance information necessary for Gaussian process models. We build on previous work by comparing the use of the singular value decomposition against a spectral-normalized dense layer as a feature extractor for a deep neural Gaussian process approximation model and apply it to a capacitance prediction problem for the High Voltage Converter Modulators in the Oak Ridge Spallation Neutron Source. Our model shows improved distance preservation and predicts in-distribution capacitance values with less than 1% error.

Methodology for Identifying Mesoscale Weather Patterns from High-Dimensional Climate Datasets

Methodology for Identifying Mesoscale Weather Patterns from High-Dimensional Climate Datasets

Predicting Multiple Outcomes Associated with Frailty based on Imbalanced Multi-label Classification.

Frailty syndrome is prevalent among the elderly, often linked to chronic diseases and resulting in various adverse health outcomes. Existing research has predominantly focused on predicting individual frailty-related outcomes. However, this paper takes a novel approach by framing frailty as a multi-label learning problem, aiming to predict multiple adverse outcomes simultaneously. In the context of multi-label classification, dealing with imbalanced label distribution poses inherent challenges to multi-label prediction. To address this issue, our study proposes a hybrid resampling approach tailored for handling imbalance problems in the multi-label scenario. The proposed resampling technique and prediction tasks were applied to a high-dimensional real-life medical dataset comprising individuals aged 65years and above. Several multi-label algorithms were employed in the experiment, and their performance was evaluated using multi-label metrics. The results obtained through our proposed approach revealed that the best-performing prediction model achieved an average precision score of 83%. These findings underscore the effectiveness of our method in predicting multiple frailty outcomes from a complex and imbalanced multi-label dataset.

Domain adaptation between heterogeneous time series data: A case study on real-time rotary machinery fault diagnosis

Artificial intelligence (AI) in fault diagnosis has emerged as a significant tool for addressing anomalies in rotary machineries (RMs). However, the success of AI models in this context largely depends on situations where the distribution of samples in the training dataset closely aligns with that of the testing dataset. In addition, practically, it is very difficult, costly, or even impossible to collect sufficient time series samples with proper labeling for AI modeling. To address these issues, this paper presents a study developing a structured methodology called frequency-domain supervised domain adaptation (FDSDA) considering heterogeneous data from versatile sensor technologies. In the context of application, two high-dimensional datasets were generated, each following distinct distributions: one being the vibration signals and the other the acoustic signals. The modeling begins with transforming signals from the time domain to the frequency domain through frequency analysis. Subsequently, a domain adaptation model is implemented, adopting an ensemble learning as an estimator to address the covariate shift challenge. Two experiments are conducted with the proposed method for binary and multi-class fault diagnosis of RMs. The results demonstrate that, despite having limited training and target data, the proposed method can effectively detect RMs fault conditions by outperforming the benchmark method.

Machine Learning Approach for Spatiotemporal Multivariate Optimization of Environmental Monitoring Sensor Locations

Abstract Long-term environmental monitoring is critical for managing the soil and groundwater at contaminated sites. Recent improvements in state-of-the-art sensor technology, communication networks, and artificial intelligence have created opportunities to modernize this monitoring activity for automated, fast, robust, and predictive monitoring. In such modernization, it is required that sensor locations be optimized to capture the spatiotemporal dynamics of all monitoring variables as well as to make it cost-effective. The legacy monitoring datasets of the target area are important to perform this optimization. In this study, we have developed a machine-learning approach to optimize sensor locations for soil and groundwater monitoring based on ensemble supervised learning and majority voting. For spatial optimization, Gaussian process regression (GPR) is used for spatial interpolation, while the majority voting is applied to accommodate the multivariate temporal dimension. Results show that the algorithms significantly outperform the random selection of the sensor locations for predictive spatiotemporal interpolation. While the method has been applied to a four-dimensional dataset (with two-dimensional space, time, and multiple contaminants), we anticipate that it can be generalizable to higher-dimensional datasets for environmental monitoring sensor location optimization.