High Dimensionality Issues Research Articles

Leveraging Machine Learning for Sophisticated Rental Value Predictions: A Case Study from Munich, Germany

There has been very limited research conducted to predict rental prices in the German real estate market using an AI-based approach. From a general perspective, conventional approaches struggle to handle large amounts of data and fail to consider the numerous elements that affect rental prices. The absence of sophisticated, data-driven analytical tools further complicates this situation, impeding stakeholders, such as tenants, landlords, real estate agents, and the government, from obtaining the accurate insights necessary for making well-informed decisions in this area. This paper applies novel machine learning (ML) approaches, including ensemble techniques, neural networks, linear regression (LR), and tree-based algorithms, specifically designed for forecasting rental prices in Munich. To ensure accuracy and reliability, the performance of these models is evaluated using the R2 score and root mean squared error (RMSE). The study provides two feature sets for model comparison, selected by particle swarm optimisation (PSO) and CatBoost. These two feature selection methods identify significant variables based on different mechanisms, such as seeking the optimal solution with an objective function and converting categorical features into target statistics (TSs) to address high-dimensional issues. These methods are ideal for this German dataset, which contains 49 features. Testing the performance of 10 ML algorithms on two sets helps validate the robustness and efficacy of the AI-based approach utilising the PyTorch framework. The findings illustrate that ML models combined with PyTorch-based neural networks (PNNs) demonstrate high accuracy compared to standalone ML models, regardless of feature changes. The improved performance indicates that utilising the PyTorch framework for predictive tasks is advantageous, as evidenced by a statistical significance test in terms of both R2 and RMSE (p-values < 0.001). The integration results display outstanding accuracy, averaging 90% across both feature sets. Particularly, the XGB model, which exhibited the lowest performance among all models in both sets, significantly improved from 0.8903 to 0.9097 in set 1 and from 0.8717 to 0.9022 in set 2 after being combined with the PNN. These results showcase the efficacy of using the PyTorch framework, enhancing the precision and reliability of the ML models in predicting the dynamic real estate market. Given that this study applies two feature sets and demonstrates consistent performance across sets with varying characteristics, the methodology may be applied to other locations. By offering accurate projections, it aids investors, renters, property managers, and regulators in facilitating better decision-making in the real estate sector.

Optimisation of synchronisation control parameters for enhanced bilateral teleoperation system

Abstract Bilateral teleoperation systems encounter challenges in achieving synchronisation between master and slave robots due to communication time delays. This paper addresses the instability caused by these delays and proposes a solution through advanced control algorithms. Nonlinear optimisation algorithms might only sometimes deliver solutions in the allotted time, particularly when handling complicated, high-dimensional issues or when optimisation iterations are extensive. The study first develops a comprehensive mathematical model encompassing the dynamics and communication intricacies of both master and slave sides in teleoperation. By recognising the limitations of existing proportional-derivative controllers in compensating for communication errors, a sequential quadratic programming-proportional-integral-derivative (SQP-PID) controller is introduced. This controller accumulates and rectifies synchronisation delay errors, ensuring precise control without steady-state deviations. The proposed SQP-PID controller stands out for its ability to handle steady-state errors effectively, offering swift response and maintaining stability. Leveraging the SQP optimisation algorithm, it intelligently tunes the parameters, minimising synchronisation errors. The approach capitalises on the simplicity, performance, and robustness of the SQP-PID controller, providing a promising avenue for enhancing bilateral teleoperation systems’ accuracy and stability, maintaining initial discrepancy with a best fitness value of 0.98 % in varied operating conditions.

Neutral Benchmarking of Survival Models in Health Sciences: Comparative Study of Classical and Machine Learning Techniques

Survival analysis plays a central role in diverse research fields, especially in health sciences. As an analytical tool, it can be used to help improve patients’ survival time, or at least, reduce the prospects of recurrence in cancer studies. However, approaches to the predictive performance of the current survival models mainly center on clinical data along with the classical survival methods. For censored “omics” data, the performance of survival models has not been thoroughly studied, either often due to their high dimensionality issues or reliance on binarizing the survival time for classification analysis. We aim to present a neural benchmark approach that analyzes and compares a broad range of classical and state-of-the-art machine learning survival models for “omics” and clinical datasets. All the methods considered in our study are evaluated using predictability as a performance measure. The study is systematically designed to make 36 comparisons (9 methods over 4 datasets, i.e., 2 clinical and 2 omics), and shows that, in practice, predictability of survival models does vary across real-world datasets, model choice, as well as the evaluation metric. From our results, we emphasize that performance criteria can play a key role in a balanced assessment of diverse survival models. Moreover, the Multitask Logistic Regression (MTLR) showed remarkable predictability for almost all the datasets. We believe this outstanding performance presents a unique opportunity for a wider use of MTLR for survival risk factors. For translational clinicians and scientists, we hope our findings provide practical guidance for benchmark studies of survival models, as well as highlight potential areas of research interest.

Automotive Consumer Loans Risk Assessment Predictive Modeling Using Generative Adversarial Network and Stacked Autoencoder Neural Networks

In this study, we propose a predictive model for automotive consumer loans risk assessment, leveraging Generative Adversarial Networks (GANs) and Stacked Autoencoder Neural Networks (SAEs). We address issues of high dimensionality and sparsity inherent in big data environments and tackle the class imbalance problem using GANs. Feature selection is effectively carried out using SAEs. Experimental results prove the superiority of our approach over traditional neural networks and our model without GANs or SAEs. The proposed model shows significant potential for application in personal credit risk assessment within automotive finance and beyond. Future work is aimed at extending and improving our model and applying it to other domains.

Physics Guided Deep Learning-Based Model for Short-Term Origin–Destination Demand Prediction in Urban Rail Transit Systems Under Pandemic

Accurate origin–destination (OD) demand prediction is crucial for the efficient operation and management of urban rail transit (URT) systems, particularly during a pandemic. However, this task faces several limitations, including real-time availability, sparsity, and high-dimensionality issues, and the impact of the pandemic. Consequently, this study proposes a unified framework called the physics-guided adaptive graph spatial–temporal attention network (PAG-STAN) for metro OD demand prediction under pandemic conditions. Specifically, PAG-STAN introduces a real-time OD estimation module to estimate real-time complete OD demand matrices. Subsequently, a novel dynamic OD demand matrix compression module is proposed to generate dense real-time OD demand matrices. Thereafter, PAG-STAN leverages various heterogeneous data to learn the evolutionary trend of future OD ridership during the pandemic. Finally, a masked physics-guided loss function (MPG-loss function) incorporates the physical quantity information between the OD demand and inbound flow into the loss function to enhance model interpretability. PAG-STAN demonstrated favorable performance on two real-world metro OD demand datasets under the pandemic and conventional scenarios, highlighting its robustness and sensitivity for metro OD demand prediction. A series of ablation studies were conducted to verify the indispensability of each module in PAG-STAN.

An enhanced Coati Optimization Algorithm for global optimization and feature selection in EEG emotion recognition

Emotion recognition based on Electroencephalography (EEG) signals has garnered significant attention across diverse domains including healthcare, education, information sharing, and gaming, among others. Despite its potential, the absence of a standardized feature set poses a challenge in efficiently classifying various emotions. Addressing the issue of high dimensionality, this paper introduces an advanced variant of the Coati Optimization Algorithm (COA), called eCOA for global optimization and selecting the best subset of EEG features for emotion recognition. Specifically, COA suffers from local optima and imbalanced exploitation abilities as other metaheuristic methods. The proposed eCOA incorporates the COA and RUNge Kutta Optimizer (RUN) algorithms. The Scale Factor (SF) and Enhanced Solution Quality (ESQ) mechanism from RUN are applied to resolve the raised shortcomings of COA. The proposed eCOA algorithm has been extensively evaluated using the CEC’22 test suite and two EEG emotion recognition datasets, DEAP and DREAMER. Furthermore, the eCOA is applied for binary and multi-class classification of emotions in the dimensions of valence, arousal, and dominance using a multi-layer perceptron neural network (MLPNN). The experimental results revealed that the eCOA algorithm has more powerful search capabilities than the original COA and seven well-known counterpart methods related to statistical, convergence, and diversity measures. Furthermore, eCOA can efficiently support feature selection to find the best EEG features to maximize performance on four quadratic emotion classification problems compared to the methods of its counterparts. The suggested method obtains a classification accuracy of 85.17% and 95.21% in the binary classification of low and high arousal emotions in two public datasets: DEAP and DREAMER, respectively, which are 5.58% and 8.98% superior to existing approaches working on the same datasets for different subjects, respectively.

Bi-stage feature selection for crop mapping using grey wolf metaheuristic optimization

Machine learning classifiers have been widely used for crop mapping; however, employing numerous features can reduce classifier accuracy. Feature selection is a widely recognized technique that addresses the issue of high dimensionality by choosing the most relevant subset of features with the highest degree of relevance and practicality. Therefore, the optimal combination of feature selection methods and classifiers for high-precision crop mapping remains an open problem. In this study, we introduce a novel hybrid feature selection approach to obtain the optimal subset of features for effective crop mapping. Initially, spectro-temporal remote sensing features from the dataset were ranked using two distinct feature selection techniques: Mutual Information (MI) and ReliefF filter. The most relevant features from each filter-based approach were combined into a union subset. Subsequently, the Gray Wolf Optimization (GWO), a metaheuristic optimization technique, was applied to enhance the feature set generated in the initial step. In the final stage, a random forest classifier leveraged the optimized feature subset for accurate crop type prediction. The effectiveness of the proposed approach was evaluated in Behiera governorate, Egypt. Comparative analysis against existing crop mapping methods demonstrated the superior performance of the proposed approach, achieving an accuracy rate of 82%. Furthermore, this approach showcased a significant reduction in the feature count, streamlining the set from 308 down to only 26 features.

Deep clustering using 3D attention convolutional autoencoder for hyperspectral image analysis

Deep clustering has been widely applicated in various fields, including natural image and language processing. However, when it is applied to hyperspectral image (HSI) processing, it encounters challenges due to high dimensionality of HSI and complex spatial-spectral characteristics. This study introduces a kind of deep clustering model specifically tailed for HSI analysis. To address the high dimensionality issue, redundant dimension of HSI is firstly eliminated by combining principal component analysis (PCA) with t-distributed stochastic neighbor embedding (t-SNE). The reduced dataset is then input into a three-dimensional attention convolutional autoencoder (3D-ACAE) to extract essential spatial-spectral features. The 3D-ACAE uses spatial-spectral attention mechanism to enhance captured features. Finally, these enhanced features pass through an embedding layer to create a compact data-representation, and the compact data-representation is divided into distinct clusters by clustering layer. Experimental results on three publicly available datasets validate the superiority of the proposed model for HSI analysis.

Chaotic tumbleweed optimization algorithm with stacked deep learning based cyberattack detection in industrial CPS environment

Cyberattacks on cyber-physical systems (CPS) have led to severe concerns, and then it is most significant to identify the attacks in the initial phase. But, there are major problems that are resolved in this area; it contains the capability of the security method for detecting earlier unknown attacks. These challenges are resolved with system behaviour analysis manners and semi-supervised or unsupervised machine learning (ML) approaches. The efficacy of the attack recognition method is strongly dependent upon the databases utilized for training the ML approaches. Anomaly detection is the procedure of recognizing anomalous procedures, which could not equal the predictable system behaviour. It permits the recognition of novel and secret attacks. Presently, anomaly detection systems are frequently executed utilizing ML like shallow (or traditional) learning and deep learning (DL). This article develops a Chaotic Tumbleweed Optimization Algorithm with Stacked Deep Learning based Cyberattack Detection (CTOASDL-CD) approach in the Industrial CPS platform. The CTOASDL-CD technique intends to exploit the FS with an optimal hyperparameter-tuned DL model for the detection of cyberattacks. To resolve the high dimensionality issue, the CTOASDL-CD technique uses CTOA for feature selection (FS) purposes. Besides, the stacked deep belief network (SDBN) model can be utilized for the recognition of cyberattacks. Finally, the tunicate swarm algorithm (TSA) has been deployed for the capable selection of hyperparameter values of the SDBN approach. To determine the improved performance of the CTOASDL-CD approach, a wide-ranging simulation value was performed. The experimental values demonstrated the capable outcome of the CTOASDL-CD methodology in accomplishing security in the industrial CPS environment.

Cross-temporal probabilistic forecast reconciliation: Methodological and practical issues

Forecast reconciliation is a post-forecasting process that involves transforming a set of incoherent forecasts into coherent forecasts which satisfy a given set of linear constraints for a multivariate time series. In this paper, we extend the current state-of-the-art cross-sectional probabilistic forecast reconciliation approach to encompass a cross-temporal framework, where temporal constraints are also applied. Our proposed methodology employs both parametric Gaussian and non-parametric bootstrap approaches to draw samples from an incoherent cross-temporal distribution. To improve the estimation of the forecast error covariance matrix, we propose using multi-step residuals, especially in the time dimension where the usual one-step residuals fail. To address high-dimensionality issues, we present four alternatives for the covariance matrix, where we exploit the two-fold nature (cross-sectional and temporal) of the cross-temporal structure, and introduce the idea of overlapping residuals. We assess the effectiveness of the proposed cross-temporal reconciliation approaches through a simulation study that investigates their theoretical and empirical properties and two forecasting experiments, using the Australian GDP and the Australian Tourism Demand datasets. For both applications, the optimal cross-temporal reconciliation approaches significantly outperform the incoherent base forecasts in terms of the continuous ranked probability score and the energy score. Overall, the results highlight the potential of the proposed methods to improve the accuracy of probabilistic forecasts and to address the challenge of integrating disparate scenarios while coherently taking into account short-term operational, medium-term tactical, and long-term strategic planning.

A Soft Actor-Critic Approach for a Blind Walking Hexapod Robot with Obstacle Avoidance

This paper investigates a path planning approach for the walking and obstacle avoidance of a blind hexapod robot in various field conditions. Hexapod robots often perform field tasks in unstructured environments, and their external sensors are affected by weather and light. This paper proposes the use of internal sensors to sense the terrain and a slightly modified soft actor-critic algorithm to train the motion strategy. A hexapod robot is capable of walking smoothly on rough ground only using internal sensors that are not affected by weather factors, and the soft actor-critic approach is superior for overcoming high-dimensional issues for multi-degree-freedom robot motion in unstructured environments.The experiments showed that the hexapod robot not only traversed rugged terrain at a fixed speed but also possessed obstacle avoidance capabilities.

Adaptive slope reliability analysis method based on sliced inverse regression dimensionality reduction

The response surface model has been widely used in slope reliability analysis owing to its efficiency. However, this method still has certain limitations, especially the curse of high dimensionality when considering the spatial variability of geotechnical parameters. The slice inverse regression dimensionality reduction method is efficient to obtaining the dimensionality-reduction variables from the original soil parameters space, before constructing the response surface. However, the dimensionality reduction process may cause accuracy deficiency due to the loss of variable information. An adaptive slope reliability analysis method is proposed to quantify and correct information loss and errors. Additionally, the slope failure probability based on the response surface in the dimensionality reduction space is modified to an unbiased one based on the finite model in the original space. In this study, two soil slopes considering spatial variability are taken as examples. The results illustrate that this method can effectively reduce the loss of accuracy in the dimensionality reduction process, while obtaining unbiased finite-element-based failure probability effectually. The method addresses the limitation whereby the accuracy of the dimensionality reduction process depends on the sample size and the number of dimensionality-reduction variables. Simultaneously, the proposed method significantly improves the computational efficiency of the sliced inverse regression method and realizes a reasonable dimensionality reduction effect, thereby improving the application of the response surface in practical slope reliability high-dimensional issues.

Regional quantile regression for multiple responses

In this article, we study high-dimensional multiple response quantile regression model for an interval of quantile levels, in which a common set of covariates is used to analyze multiple responses simultaneously. We assume that the underlying quantile coefficient matrix is simultaneously element-wise and row-wise sparse. We address high dimensional issues to identify globally relevant variables for multiple responses when any τth conditional quantile is considered, where τ∈Δ, and Δ is an interval of quantile levels of interest. We develop a novel penalized globally concerned quantile regression with double group Lasso penalties and propose an information criterion for penalty parameter choice. We prove that the proposed method consistently selects both element-wise and row-wise sparsity patterns of the regression coefficient matrix function and that it achieves the oracle convergence rate. Numerical examples and applications to Cancer Cell Line Encyclopedia data illustrate the advantages of the proposed method over separate penalized quantile regression on each response.

Bayesian weighted random forest for classification of high-dimensional genomics data

In this paper, a full Bayesian weighted probabilistic model is developed for random classification trees. The new model Bayesian Weighted Random Classification Forest (BWRCF) arises from the modification of the existing random classification forest in two ways. Firstly, the tree terminal node estimation procedure is replaced with a Bayesian estimation approach. Secondly, a new variable ranking procedure is developed and then hybridized with BWRCF to tackle the high-dimensionality issues. The performance of the proposed method is analyzed using simulated and real-life high-dimensional microarray datasets based on holdout accuracy and misclassification error rates. The results of the analyses showed that the proposed BWRCF is robust in terms of its ability to withstand moderate to large high-dimensionality scenarios. In addition, BWRCF also has improved predictive and efficiency abilities over selected competing methods.

Data Assimilation Using Principal Component Analysis and Artificial Neural Network

Summary Data assimilation (DA) for uncertainty reduction using reservoir simulation models normally demands high computational time; it may take days or even weeks to run a single reservoir application, depending on the reservoir model characteristics. Therefore, it is important to accelerate the process to make it more feasible for practical studies, especially those requiring many simulation runs. One possible way is by using proxy models to replace the reservoir simulator in some time-consuming parts of the procedure. However, the main challenge inherent in proxy models is the inclusion of 3D geostatistical realizations (block-to-block grid properties such as porosity and permeability) as uncertain attributes in the proxy construction. In most cases, it is impossible to treat the values of all grid properties explicitly as input to the proxy building process due to the high dimensionality issue. We present a new methodology for DA combining principal component analysis (PCA) with artificial neural networks (ANN) to solve this problem. The PCA technique is applied to reduce the dimension of the problem, making it possible and feasible to use grid properties in proxy modeling. The trained ANN is used as a proxy for the reservoir simulator, with the goal of reducing the total computational time spent on the application. We run three DA processes using a complex real-field reservoir model for validating the methodology. The first (DA1), used as the reference solution, is the conventional process in which the DA method updates all grid property values explicitly. The second (DA2) is only executed to validate the proposed parameterization via PCA. Both DA1 and DA2 use only the reservoir simulator to generate the reservoir outputs. In the third (DA3), the ANN replaces the reservoir simulator to save computational time. It is important to mention that after DA3, the results (the posterior ensemble) are validated with the reservoir simulator. The DA3, although a little bit less accurate than the DA1, allowed good overall results. Therefore, it seems reasonable to offer the decision-makers the possibility of choosing between the conventional approach (DA1), normally more accurate but slower, and the proposed DA3, much faster than DA1 (with overall good results). This choice may depend on the objective of the reservoir study, available resources, and time to perform the study. The key contribution of this paper is a practical methodology for DA combining PCA [for dimensional reduction (DR)] and ANN (for computational time reduction) applicable in real fields, filling a gap in the literature in this research area.

FUZZY ASSOCIATIVE CLASSIFIER FOR BIG DATA APPLICATIONS

The related blueprint field melds really intriguing cycles for building solid classifiers and any of those methodologies all issues mulled over works of art on 4 top-notch levels. The thought process of this work might be a novel developmental issue for profitably gathering helpful classifiers in huge information. Comfortable supportive hoarding has not been significantly examined inside the structure, anyway familiar classifiers have wrapped up being impeccable in particular genuine space programs. we advance a helpful pivoted delicate auxiliary depiction strategy subject to the Map decrease viewpoint. The framework mishandles a particularly appropriated discretize dependent on woolen entropy for productively making padded bundles of the attributes. Zeroing in on precision, rendition multifaceted nature, assessment time, and adaptability. We spotlight that, despite the way that the correctnesses result to be comparative, the flightiness, assessed with perceive to a number of guidelines, of the classifiers made through the smooth appropriated approach is not exactly the one among the non fuzzy classifiers. the total circuit of padded understanding grams to visual assessment of agreeable association manages a successful appropriated woolen familiar classification model-dependent at the Map diminish demeanor. The preparation assessment first mines an unprecedented arrangement of agreeable association classification directs by utilizing the utilization of a dispersed variant of a smooth development of the putting FP-development tally number. We show the adaptability of our strategy by means of doing selective examinations on an appropriate critical dataset. A fuzzy affiliation rule-based solicitation gadget for high-dimensional issues trouble to two or three territories to advantage a specific and inconsequential delicate standard based absolutely classifier with an espresso computational expense.

Hyperspectral Image Classification via Information Theoretic Dimension Reduction

Hyperspectral images (HSIs) are one of the most successfully used tools for precisely and potentially detecting key ground surfaces, vegetation, and minerals. HSIs contain a large amount of information about the ground scene; therefore, object classification becomes the most difficult task for such a high-dimensional HSI data cube. Additionally, the HSI’s spectral bands exhibit a high correlation, and a large amount of spectral data creates high dimensionality issues as well. Dimensionality reduction is, therefore, a crucial step in the HSI classification pipeline. In order to identify a pertinent subset of features for effective HSI classification, this study proposes a dimension reduction method that combines feature extraction and feature selection. In particular, we exploited the widely used denoising method minimum noise fraction (MNF) for feature extraction and an information theoretic-based strategy, cross-cumulative residual entropy (CCRE), for feature selection. Using the normalized CCRE, minimum redundancy maximum relevance (mRMR)-driven feature selection criteria were used to enhance the quality of the selected feature. To assess the effectiveness of the extracted features’ subsets, the kernel support vector machine (KSVM) classifier was applied to three publicly available HSIs. The experimental findings manifest a discernible improvement in classification accuracy and the qualities of the selected features. Specifically, the proposed method outperforms the traditional methods investigated, with overall classification accuracies on Indian Pines, Washington DC Mall, and Pavia University HSIs of 97.44%, 99.71%, and 98.35%, respectively.

Delay and energy aware task scheduling mechanism for fog-enabled IoT applications: A reinforcement learning approach

With the expansion of the internet of things (IoT) devices and their applications, the demand for executing complex and deadline-aware tasks is growing rapidly. Fog-enabled IoT architecture has evolved to accomplish these tasks at the fog layer. However, fog computing devices have limited power supply and computation resources compared to cloud devices. In delay-sensitive applications of fog-enabled IoT architecture, executing tasks with stringent deadlines while reducing the service latency and energy usage of fog resources is a difficult challenge. This paper presents an effective task scheduling strategy to allocate fog computing resources for IoT requests to meet the deadline of the requests and resource availability. Initially, the scheduling problem is formulated as mixed-integer nonlinear programming (MINLP) to reduce the energy consumption of the fog resources and service time of the tasks subject to the deadline and resource availability constraints. To address the high dimensionality issue of the tasks in a dynamic environment, a fuzzy-based reinforcement learning (FRL) mechanism is employed to reduce the service delay of the tasks and energy usage of the fog nodes. Initially, the tasks are prioritized using fuzzy logic. Then the prioritized tasks are scheduled using the on-policy reinforcement learning technique, which enhances the long-term reward compared to the Q-learning approach. Further, the evaluation outcomes reflect that the proposed task scheduling technique outperforms the existing algorithms with an improvement of up to 23% and 18% regarding service latency and energy consumption, respectively.

Kernelized Similarity Learning and Embedding for Dynamic Texture Synthesis

Dynamic texture (DT) exhibits statistical stationarity in the spatial domain and stochastic repetitiveness in the temporal dimension, indicating that different frames of DT possess a high similarity correlation that is critical prior knowledge. However, existing methods cannot effectively learn a synthesis model for high-dimensional DT from a small number of training samples. In this article, we propose a novel DT synthesis method, which makes full use of similarity as prior knowledge to address this issue. Our method is based on the proposed kernel similarity embedding, which can not only mitigate the high dimensionality and small sample issues, but also has the advantage of modeling nonlinear feature relationships. Specifically, we first put forward two hypotheses that are essential for the DT model to generate new frames using similarity correlations. Then, we integrate kernel learning and the extreme learning machine into a unified synthesis model to learn kernel similarity embeddings for representing DTs. Extensive experiments on DT videos collected from the Internet and two benchmark datasets, i.e., Gatech Graphcut Textures and Dyntex, demonstrate that the learned kernel similarity embeddings can provide discriminative representations for DTs. Further, our method can preserve the long-term temporal continuity of the synthesized DT sequences with excellent sustainability and generalization. Meanwhile, it effectively generates realistic DT videos with higher speed and lower computation than the current state-of-the-art methods. The code and more synthesis videos are available at our project page <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://shiming-chen.github.io/Similarity-page/Similarit.html</uri> .

On high-dimensional Poisson models with measurement error: Hypothesis testing for nonlinear nonconvex optimization.

We study estimation and testing in the Poisson regression model with noisy high dimensional covariates, which has wide applications in analyzing noisy big data. Correcting for the estimation bias due to the covariate noise leads to a non-convex target function to minimize. Treating the high dimensional issue further leads us to augment an amenable penalty term to the target function. We propose to estimate the regression parameter through minimizing the penalized target function. We derive the L1 and L2 convergence rates of the estimator and prove the variable selection consistency. We further establish the asymptotic normality of any subset of the parameters, where the subset can have infinitely many components as long as its cardinality grows sufficiently slow. We develop Wald and score tests based on the asymptotic normality of the estimator, which permits testing of linear functions of the members if the subset. We examine the finite sample performance of the proposed tests by extensive simulation. Finally, the proposed method is successfully applied to the Alzheimer's Disease Neuroimaging Initiative study, which motivated this work initially.