High-dimensional Data Research Articles

A neural network approach for unstructured mesh quality evaluation

PurposeThe quality of the unstructured mesh has a considerable impact on the stability and accuracy of aerodynamic simulation in computational fluid dynamics (CFD). Typically, engineers spend a significant portion of their time on mesh quality evaluation to ensure a valid, high-quality mesh. The extensive manual interaction and a priori knowledge required to undertake an accurate and timely evaluation process have become a bottleneck in the idealized efficient CFD workflow. This paper aims to introduce a neural network-based quality evaluation approach for unstructured meshes to enable higher efficiency and the level of automation.Design/methodology/approachThe paper investigates the capability of deep neural networks for the quality evaluation of unstructured meshes. For training the network, we build a training dataset for mesh quality learning algorithms. The dataset contains a rich variety of unstructured aircraft meshes with different mesh sizes, densities, cell distribution, growth ratios and cell numbers to ensure its diversity and availability. We also design a neural network, AircraftNet, to learn the effect of mesh quality on the convergent properties of the numerical solutions. The proposed network directly manipulates raw point data in mesh source files rather than passing it to an intermediate data representation. During training, AircraftNet extracts non-linear quality features from high-dimensional data spaces and then automatically predicts the overall quality of the input unstructured mesh.FindingsThe paper provides a series of experimental results on GPUs. It shows that AircraftNet is able to effectively analyze the quality-related features like mesh density and distribution from the extracted features and achieve high prediction accuracy on the proposed dataset with even a small number of training runs.Research limitations/implicationsBecause of the limited training dataset, the research results may lack generalizability. Therefore, researchers are encouraged to test the proposed propositions further.Originality/valueThe paper publishes a benchmarking dataset for mesh quality learning algorithms and designs a novel neural network approach for unstructured mesh quality evaluation.

Machine learning methods for propensity and disease risk score estimation in high-dimensional data: a plasmode simulation and real-world data cohort analysis

IntroductionMachine learning (ML) methods are promising and scalable alternatives for propensity score (PS) estimation, but their comparative performance in disease risk score (DRS) estimation remains unexplored.MethodsWe used real-world data comparing antihypertensive users to non-users with 69 negative control outcomes, and plasmode simulations to study the performance of ML methods in PS and DRS estimation. We conducted a cohort study using UK primary care records. Further, we conducted a plasmode simulation with synthetic treatment and outcome mimicking empirical data distributions. We compared four PS and DRS estimation methods: 1. Reference: Logistic regression including clinically chosen confounders. 2. Logistic regression with L1 regularisation (LASSO). 3. Multi-layer perceptron (MLP). 4. Extreme Gradient Boosting (XgBoost). Covariate balance, coverage of the null effect of negative control outcomes (real-world data) and bias based on the absolute difference between observed and true effects (for plasmode) were estimated. 632,201 antihypertensive users and nonusers were included.ResultsML methods outperformed the reference method for PS estimation in some scenarios, both in terms of covariate balance and coverage/bias. Specifically, XgBoost achieved the best performance. DRS-based methods performed worse than PS in all tested scenarios.DiscussionWe found that ML methods could be reliable alternatives for PS estimation. ML-based DRS methods performed worse than PS ones, likely given the rarity of outcomes.

Practical Screening Method for Cancer Gene Diagnosis -How to Choose Cancer and Normal Patients by four Principles

We developed a new theory of discriminant analysis (Theory1). Physicians can use it for practical medical diagnoses. Only Revised IP Optimal-LDF (RIP) obtains the minimum number of misclassification (MNM). RIP can discriminate linearly separable data (LSD) theoretically. It discriminated against 169 microarrays with two classes and found that 169 MNMs are zero and LSD. It can split high-dimensional arrays into many small LSD with less than n (patient’s number) genes that are the candidates of multivariate oncogenes. We completed a new theory of high-dimensional gene data analysis (Theory2). A 100-fold Cross-Validation (Method1) can rank all candidates for the importance of diagnosis. Thus, if physicians firstly use Theory2 as the screening method, they can start their medical studies with the correct small sizes of candidates. This paper analyzes four arrays in detail and proposes correctly choosing cancer and normal patients using four principles.

Matrix Factorization and Prediction for High-Dimensional Co-Occurrence Count Data via Shared Parameter Alternating Zero Inflated Gamma Model

High-dimensional sparse matrix data frequently arise in various applications. A notable example is the weighted word–word co-occurrence count data, which summarizes the weighted frequency of word pairs appearing within the same context window. This type of data typically contains highly skewed non-negative values with an abundance of zeros. Another example is the co-occurrence of item–item or user–item pairs in e-commerce, which also generates high-dimensional data. The objective is to utilize these data to predict the relevance between items or users. In this paper, we assume that items or users can be represented by unknown dense vectors. The model treats the co-occurrence counts as arising from zero-inflated Gamma random variables and employs cosine similarity between the unknown vectors to summarize item–item relevance. The unknown values are estimated using the shared parameter alternating zero-inflated Gamma regression models (SA-ZIG). Both canonical link and log link models are considered. Two parameter updating schemes are proposed, along with an algorithm to estimate the unknown parameters. Convergence analysis is presented analytically. Numerical studies demonstrate that the SA-ZIG using Fisher scoring without learning rate adjustment may fail to find the maximum likelihood estimate. However, the SA-ZIG with learning rate adjustment performs satisfactorily in our simulation studies.

Application of mass cytometry in multiparametric characterization of precancerous cervical lesions.

Cervical cancer (CC) is the fourth most common malignant tumor in women worldwide. Detecting different biomarkers together on single cells by novel method mass cytometry could contribute to more precise screening. Liquid-based cytology (LBC) cervical samples were collected (N = 53) from women categorized as normal and precancerous lesions. Human papillomavirus was genotyped by polymerase chain reaction, while simultaneous examination of the expression of 29 proteins was done by mass cytometry (CyTOF). Differences in cluster abundances were assessed with Spearman's rank correlation as well as high dimensional data analysis (t-SNE, FlowSOM). Cytokeratin (ITGA6, Ck5, Ck10/13, Ck14, Ck7) expression patterns allowed determining the presence of different cells in the cervical epithelium. FlowSOM analysis enabled to phenotype cervical cells in five different metaclusters and find new markers that could be important in CC screening. The markers Ck18, Ck18, and CD63 (Metacluster 3) showed significantly increasing associated with severity of the precancerous lesions (Spearman rank correlation rho 0.304, p = 0.0271), while CD71, KLF4, LRIG1, E-cadherin, Nanog and p53 (Metacluster 1) decreased with severity of the precancerous lesions (Spearman rank correlation rho -0.401, p = 0.0029). Other metaclusters did not show significant correlation, but metacluster 2 (Ck17, MCM, MMP7, CD29, E-cadherin, Nanog, p53) showed higher abundance in low- and high-grade intraepithelial lesion cases. CyTOF appears feasible and should be considered when examining novel biomarkers on cervical LBC samples. This study enabled us to characterize different cells in the cervical epithelium and find markers and populations that could distinguish precancerous lesions.

Shrinkage estimation of gene interaction networks in single-cell RNA sequencing data

BackgroundGene interaction networks are graphs in which nodes represent genes and edges represent functional interactions between them. These interactions can be at multiple levels, for instance, gene regulation, protein-protein interaction, or metabolic pathways. To analyse gene interaction networks at a large scale, gene co-expression network analysis is often applied on high-throughput gene expression data such as RNA sequencing data. With the advance in sequencing technology, expression of genes can be measured in individual cells. Single-cell RNA sequencing (scRNAseq) provides insights of cellular development, differentiation and characteristics at the transcriptomic level. High sparsity and high-dimensional data structures pose challenges in scRNAseq data analysis.ResultsIn this study, a sparse inverse covariance matrix estimation framework for scRNAseq data is developed to capture direct functional interactions between genes. Comparative analyses highlight high performance and fast computation of Stein-type shrinkage in high-dimensional data using simulated scRNAseq data. Data transformation approaches also show improvement in performance of shrinkage methods in non-Gaussian distributed data. Zero-inflated modelling of scRNAseq data based on a negative binomial distribution enhances shrinkage performance in zero-inflated data without interference on non zero-inflated count data.ConclusionThe proposed framework broadens application of graphical model in scRNAseq analysis with flexibility in sparsity of count data resulting from dropout events, high performance, and fast computational time. Implementation of the framework is in a reproducible Snakemake workflow https://github.com/calathea24/ZINBGraphicalModel and R package ZINBStein https://github.com/calathea24/ZINBStein.

Conditional sufficient variable selection with prior information

AbstractDimension reduction and variable selection play crucial roles in high-dimensional data analysis. Numerous existing methods have been demonstrated to attain either or both of these goals. The Minimum Average Variance Estimation (MAVE) method and its variants are effective approaches to estimate directions on the Central Mean Subspace (CMS). The Sparse Minimum Average Variance Estimation (SMAVE) combines the concepts of sufficient dimension reduction and variable selection and has been demonstrated to exhaustively estimate CMS while simultaneously selecting informative variables using LASSO without assuming any specific model or distribution on the predictor variables. In many applications, however, researchers typically possess prior knowledge for a set of predictors that is associated with response. In the presence of a known set of variables, the conditional contribution of additional predictors provides a natural evaluation of the relative importance. Based on this concept, we propose the Conditional Sparse Minimum Average Variance Estimation (CSMAVE) method. By utilizing prior information and creating a meaningful conditioning set for SMAVE, we intend to select variables that will result in a more parsimonious model and a more accurate interpretation than SMAVE. We evaluate our strategy by analyzing simulation examples and comparing them to the SMAVE method. And a real-world dataset validates the applicability and efficiency of our method.

Second Order Extended Ensemble Filter for Non-linear Filtering

Whenever the state of a system must be estimated from noisy information, a state estimator is employed to fuse the data with the model to produce an accurate estimate of the state. When the system dynamics and observation models are linear, the Kalman Filter which is optimal, is used. However, in most applications of interest the system dynamics and observations equations are not- linear and suitable extensions of the Kalman Filter have been developed; for example, the Extended Kalman Filter(EKF). The Extended Kalman Filter is based on linearization by the Taylor series expansion about the mean of the state. This filtering process is however computationally expensive especially in high dimensional data. The cause for this is the high cost of integrating the equation of evolution of covariances. Due to this complexity in integration, new methods were sought known as the particle filters. It replaces linearization of non-linearities with Monte Carlo methods. The particle filter formed a basis for Ensemble Kalman Filter (EnKF) an extension of Kalman filter to non-linear filtering. The EnKF reduced the computational cost but its innovation process does not capture information sufficiently hence there is need to improve its performance. This study has developed a new filter, Second order Extended Ensemble Filter (SoEEF).We derived it from stochastic state models by expansion of expected values to the second order by use of Taylor series together with Monte Carlo method and Matlab. We used Lorenz 63 system of ordinary equations and differential Matlab to test the performance of the new filter. Then we compared its performance with four other filters like Bootstrap Particle Filter (BPF), First order Kalman Bucy Filter (FoEKBF),Second order Kalman Bucy Filter (SoKBF) and First order Extended Ensemble Filter (FoEEF). SoEEKF performs much better than the other four filters.

Association between Radiological and First-Order Statistical Features of the Mammogram, and the Tumor Phenotype in Breast Cancer Patients

Background: Breast cancer is among the most prevalent cancers which can effectively be screened by mammography. The spatial distribution of grey levels of the mammogram known as first-order statistical features (FOSFs) contain higher dimensional data which describe the breast composition. We aim to test these basic measures to differentiate density categories in breast cancer patients, and use them as covariates to investigate the relationship between radiologic and pathologic features of the tumor. Methods: Data from Eighty-five breast cancer subjects, their BI-RADS breast density category (a to d), percentage density (PD), and FOSFs of the mammogram including median, mean, interquartile range, kurtosis, maximum, minimum, standard deviation, skewness, and energy, were extracted. The tumor grade and the percentage of Ki67, ER, PR, and Her2 status were recorded. A linear discriminant analysis, and a support vector machine (SVM) were used to discriminate each density category from others. Then, the relation between variables were investigated using ANCOVA and regression analysis. Results: Density categories a and d were classified by SVM with high accuracy. The key feature of a and d were interquartile range and maximum intensities respectively. Reported tumor margins were related to Her2 overexpression and PR positivity. Spiculated tumor margin predicted percentage of PR expression, with a cumulative odds ratio of 7.85 (CI 2.5- 24.78), when adjusted for age, area of breast, density, and FOSFs, (p=0.0004). Conclusion: The findings of this study suggest that FOSFs can be incorporated in computer-aided systems to adjust for differences in breast composition and to refine risk profiles.

Research on the Feasibility of Machine Learning Methods in Stock Price Prediction

Abstract: Stock price prediction has always been an important topic in financial research. Accurate price prediction can not only help investors make informed investment decisions but also enhance market stability and reduce systemic risk. In recent years, with advancements in computing technology and data science, machine learning methods have been increasingly applied in the financial field. Compared to traditional methods, machine learning methods can better handle high-dimensional, nonlinear, and large datasets, thus demonstrating higher prediction accuracy and applicability in stock price prediction.This paper reviews relevant literature and selects five models for empirical research: Support Vector Machine (SVM), Long Short-Term Memory network (LSTM), LightGBM, a combination of LSTM and LightGBM, and Convolutional Neural Network (CNN). The effectiveness of these models in predicting the stock price of Meituan-W (3690) was analyzed and compared in detail. The experimental results show that the LightGBM model performs best in terms of Mean Squared Error (MSE), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE), proving its significant advantages in handling large-scale, high-dimensional, and nonlinear data. By comparing the prediction results of different models, this paper explores the strengths and weaknesses of each model and their feasibility and effectiveness in practical applications. Machine learning methods have significant potential in stock price prediction, model selection needs to comprehensively consider data characteristics, computational resources, and practical application scenarios.

Crystal Structure Prediction Using Generative Adversarial Network with Data-Driven Latent Space Fusion Strategy.

Crystal structure prediction (CSP) is an important field of material design. Herein, we propose a novel generative adversarial network model, guided by a data-driven approach and incorporating the real physical structure of crystals, to address the complexity of high-dimensional data and improve prediction accuracy in materials science. The model, termed GAN-DDLSF, introduces a novel sampling method called data-driven latent space fusion (DDLSF), which aims to optimize the latent space of generative adversarial networks (GANs) by combining the statistical properties of real data with a standard Gaussian distribution, effectively mitigating the "mode collapse" problem prevalent in GANs. Our approach introduces a more refined generation mechanism specifically for binary crystal structures such as gallium nitride (GaN). By optimizing for the specific crystallographic features of GaN while maintaining structural rationality, we achieve higher precision and efficiency in predicting and designing structures for this particular material system. The model generates 9321 GaN binary crystal structures, with 16.59% reaching a stable state and 24.21% found to be metastable. These results can significantly enhance the accuracy of crystal structure predictions and provide valuable insights into the potential of the GAN-DDLSF approach for the discovery and design of binary, ternary, and multinary materials, offering new perspectives and methods for materials science research and applications.

A regularized Cox hierarchical model for incorporating annotation information in predictive omic studies

BackgroundAssociated with high-dimensional omics data there are often “meta-features” such as biological pathways and functional annotations, summary statistics from similar studies that can be informative for predicting an outcome of interest. We introduce a regularized hierarchical framework for integrating meta-features, with the goal of improving prediction and feature selection performance with time-to-event outcomes.MethodsA hierarchical framework is deployed to incorporate meta-features. Regularization is applied to the omic features as well as the meta-features so that high-dimensional data can be handled at both levels. The proposed hierarchical Cox model can be efficiently fitted by a combination of iterative reweighted least squares and cyclic coordinate descent.ResultsIn a simulation study we show that when the external meta-features are informative, the regularized hierarchical model can substantially improve prediction performance over standard regularized Cox regression. We illustrate the proposed model with applications to breast cancer and melanoma survival based on gene expression profiles, which show the improvement in prediction performance by applying meta-features, as well as the discovery of important omic feature sets with sparse regularization at meta-feature level.ConclusionsThe proposed hierarchical regularized regression model enables integration of external meta-feature information directly into the modeling process for time-to-event outcomes, improves prediction performance when the external meta-feature data is informative. Importantly, when the external meta-features are uninformative, the prediction performance based on the regularized hierarchical model is on par with standard regularized Cox regression, indicating robustness of the framework. In addition to developing predictive signatures, the model can also be deployed in discovery applications where the main goal is to identify important features associated with the outcome rather than developing a predictive model.

Ensemble-Enhanced Semi-Supervised Learning with Optimized Graph Construction for High-Dimensional Data.

Graph-based methods have demonstrated exceptional performance in semi-supervised classification. However, existing graph-based methods typically construct either a predefined graph in the original space or an adaptive graph within the output space, which often limits their ability to fully utilize prior information and capture the optimal intrinsic data distribution, particularly in high-dimensional data with abundant redundant and noisy features. This paper introduces a novel approach: Semi-Supervised Classification with Optimized Graph Construction (SSC-OGC). SSC-OGC leverages both predefined and adaptive graphs to explore intrinsic data distribution and effectively employ prior information. Additionally, a graph constraint regularization term (GCR) and a collaborative constraint regularization term (CCR) are incorporated to further enhance the quality of the adaptive graph structure and the learned subspace, respectively. To eliminate the negative effect of constructing a predefined graph in the original data space, we further propose a Hybrid Subspace Ensemble-enhanced framework based on the proposed Optimized Graph Construction method (HSE-OGC). Specifically, we construct multiple hybrid subspaces, which consist of meticulously chosen features from the original data to achieve high-quality and diverse space representations. Then, HSE-OGC constructs multiple predefined graphs within hybrid subspaces and trains multiple SSC-OGC classifiers to complement each other, significantly improving the overall performance. Experimental results conducted on various high-dimensional datasets demonstrate that HSE-OGC exhibits outstanding performance.

Genomic structural equation modeling reveals latent phenotypes in the human cortex with distinct genetic architecture

Genetic contributions to human cortical structure manifest pervasive pleiotropy. This pleiotropy may be harnessed to identify unique genetically-informed parcellations of the cortex that are neurobiologically distinct from functional, cytoarchitectural, or other cortical parcellation schemes. We investigated genetic pleiotropy by applying genomic structural equation modeling (SEM) to map the genetic architecture of cortical surface area (SA) and cortical thickness (CT) for 34 brain regions recently reported in the ENIGMA cortical GWAS. Genomic SEM uses the empirical genetic covariance estimated from GWAS summary statistics with LD score regression (LDSC) to discover factors underlying genetic covariance, which we are denoting genetically informed brain networks (GIBNs). Genomic SEM can fit a multivariate GWAS from summary statistics for each of the GIBNs, which can subsequently be used for LD score regression (LDSC). We found the best-fitting model of cortical SA identified 6 GIBNs and CT identified 4 GIBNs, although sensitivity analyses indicated that other structures were plausible. The multivariate GWASs of the GIBNs identified 74 genome-wide significant (GWS) loci (p < 5 × 10−8), including many previously implicated in neuroimaging phenotypes, behavioral traits, and psychiatric conditions. LDSC of GIBN GWASs found that SA-derived GIBNs had a positive genetic correlation with bipolar disorder (BPD), and cannabis use disorder, indicating genetic predisposition to a larger SA in the specific GIBN is associated with greater genetic risk of these disorders. A negative genetic correlation was observed between attention deficit hyperactivity disorder (ADHD) and major depressive disorder (MDD). CT GIBNs displayed a negative genetic correlation with alcohol dependence. Even though we observed model instability in our application of genomic SEM to high-dimensional data, jointly modeling the genetic architecture of complex traits and investigating multivariate genetic links across neuroimaging phenotypes offers new insights into the genetics of cortical structure and relationships to psychopathology.

Big data research is everyone's research-Making epilepsy data science accessible to the global community: Report of the ILAE big data commission.

Epilepsy care generates multiple sources of high-dimensional data, including clinical, imaging, electroencephalographic, genomic, and neuropsychological information, that are collected routinely to establish the diagnosis and guide management. Thanks to high-performance computing, sophisticated graphics processing units, and advanced analytics, we are now on the cusp of being able to use these data to significantly improve individualized care for people with epilepsy. Despite this, many clinicians, health care providers, and people with epilepsy are apprehensive about implementing Big Data and accompanying technologies such as artificial intelligence (AI). Practical, ethical, privacy, and climate issues represent real and enduring concerns that have yet to be completely resolved. Similarly, Big Data and AI-related biases have the potential to exacerbate local and global disparities. These are highly germane concerns to the field of epilepsy, given its high burden in developing nations and areas of socioeconomic deprivation. This educational paper from the International League Against Epilepsy's (ILAE) Big Data Commission aims to help clinicians caring for people with epilepsy become familiar with how Big Data is collected and processed, how they are applied to studies using AI, and outline the immense potential positive impact Big Data can have on diagnosis and management.

Digital Economy Oriented Tourism Industry Data Analysis in Semantic IoT

ABSTRACTThe digital economy has revolutionized the tourism industry by integrating Semantic Internet of Things (IoT) technologies for enhanced data analysis. This paper proposes a comprehensive method for analyzing tourism flow using Semantic IoT. The method combines semantic data processing with advanced machine learning techniques, specifically stacked autoencoders and long short‐term memory (LSTM) models, to understand and predict changes in passenger flow. The approach addresses challenges of high dimensionality and complex temporal characteristics in tourism data, offering significant support for tourism management, improving operational efficiency, and enhancing the visitor experience. Experimental results demonstrate the effectiveness of the proposed method in analyzing and predicting tourism passenger flow data, providing accurate and insightful trends that are crucial for decision‐making in the tourism industry.

Early estimation of glutelin to gliadin ratio in wheat grain using high-dimensional and hyperspectral reflectance

Precise and timely estimation of the glutelin-to-gliadin ratio (Glu/Gli) in wheat grain is pivotal for crop monitoring, as it is a crucial quality indicator ensuring the production of high-quality wheat flour. Despite the recognized potential of hyperspectral technology in crop phenotype estimation, its application to estimate Glu/Gli in wheat grains faces challenges due to complex spectral-chemical relationships and the influence of growing seasons. This study addresses this gap by cultivating 11 wheat varieties and collecting high-dimensional hyperspectral data from field experiments during various growth stages of wheat (2018–2019 and 2019–2020). Utilizing vegetation indices (VIs) in conjunction with linear mixed-effects model (LMM) and random forest regression model (RFR), it constructs a robust Glu/Gli estimation model (with a Glu/Gli range of 1.063 to 2.218). Results reveal that a singular VI application suffers from data limitations, while the integration of multiple VIs significantly enhances estimation accuracy. The mid-grain filling period emerges as a critical stage for accurate Glu/Gli estimation, with TCARI (transformed chlorophyll absorption reflectance index) demonstrating notable significance and high correlation. In model performance, RFR (R2 = 0.691, rRMSE = 0.096, RPD = 1.872, RER = 6.028) outperforms LMM (R2 = 0.477, rRMSE = 0.131, RPD = 1.383, RER = 4.453), exhibiting superior accuracy in estimating grain Glu/Gli for diverse wheat varieties. This study introduces a rapid and accurate approach for early wheat grain Glu/Gli estimation, offering valuable insights for wheat value chain and precision farming.

An efficient enhanced stacked auto encoder assisted optimized deep neural network for forecasting Dry Eye Disease

Meibomian Gland Dysfunction (MGD) and Dry Eye Disease (DED) comprise two of the most significant eye diseases, impacting millions of sufferers worldwide. Several etiological factors influence the early symptoms of DED. Early diagnosis and treatment of erectile dysfunction may significantly improve the Quality of Life (QoL) for people. The current study introduces the ESAE-ODNN, an improved stacked autoencoder-aided optimised deep neural network, as a new way to predict DED using feature selection (FS), feature extraction (FE), and classification. The approach described here is novel because it merges chaotic maps into FS, employs SLSTM-STSA for improved classification accuracy (CA), and optimizes with the adaptive quantum rotation of the Enhanced Quantum Bacterial Foraging Optimisation Algorithm (EQBFOA). The present study enhances prediction functions by extracting MGD-related features and complicated relationships from the DED dataset. To ensure essential feature identification, the ESAE minimizes irrelevant and redundant features. To predict the DED, the ESAE first applies FE and then implements an ODNN classifier. This method fine-tunes the ODNN framework to enhance the effectiveness of the classification. The proposed ESAE-ODNN classification system efficiently assists in the early diagnosis of DED. Combining advanced Deep Learning (DL) methods with optimization can help us understand MGD features better and sort the data with the best accuracy (96.34%). The experimental evaluation with relevant performance metrics indicates that the proposed method is efficient in diverse aspects: accurate identification, reduced complexity, and fine-tuned performance. The ESAE-ODNN’s robustness in handling intricate feature indications and high-dimensional data outperforms the existing state-of-the-art techniques.

Development of Two Methods for Estimating High-Dimensional Data in the Case of Multicollinearity and Outliers

High-dimensional problems involve datasets or models characterized by a substantial number of variables or parameters prevalent across various domains such as statistics, machine learning, optimization, physics, and engineering. Challenges in these scenarios include computational complexity, data sparsity, over-fitting, and the curse of dimensionality. This study introduces two innovative techniques that combine the Random Forest machine learning approach with both the least absolute shrinkage and selection operator and the elastic net, which are statistical methodologies tailored to address high-dimensional challenges. We compared performance evaluations of these hybrid methods against traditional statistical approaches and standalone machine learning methods. The assessment is conducted using goodness-of-fit measures and involves both Monte Carlo simulation and a real-world application. The findings show that the strategies proposed in this study exhibit superior performance compared to conventional approaches when tackling high-dimensional challenges.

The Effect of Regularized Regression and Tree-Based Missing Data Imputation Methods on Classification Performance in High Dimensional Data

Missing data is an important problem in the analysis and classification of high dimensional data. The aim of this study is to compare the effects of four different missing data imputation methods on classification performance in high dimensional data. In this study, missing data imputation methods were evaluated using data sets, whose independent variables between mixed correlated with each other, for binary dependent variable, p=500 independent variables, n=150 units and 1000 times running simulation. Missing data structures were created according to different missing rates. Different datasets were obtained by imputing the missing values using different methods. Regularized regression methods such as lasso and elastic net regression were used for imputation, as well as tree-based methods such as support vector machine and classification and regression trees. At the end of simulation, the classification scores of the methods were obtained by gradient boosting machines and the missing data prediction performances were evaluated according to the distance of these scores from the reference. Our simulation demonstrates that regularized regression methods outperform tree-based methods in classifying high dimensional datasets. Additionally, it was found that the increase in the amount of missing values reduced the classification performance of the methods in high dimensional data.