Tuning Parameter Selection Research Articles

Wasserstein regression with empirical measures and density estimation for sparse data.

The problem of modeling the relationship between univariate distributions and one or more explanatory variables lately has found increasing interest. Existing approaches proceed by substituting proxy estimated distributions for the typically unknown response distributions. These estimates are obtained from available data but are problematic when for some of the distributions only few data are available. Such situations are common in practice and cannot be addressed with currently available approaches, especially when one aims at density estimates. We show how this and other problems associated with density estimation such as tuning parameter selection and bias issues can be side-stepped when covariates are available. We also introduce a novel version of distribution-response regression that is based on empirical measures. By avoiding the preprocessing step of recovering complete individual response distributions, the proposed approach is applicable when the sample size available for each distribution varies and especially when it is small for some of the distributions but large for others. In this case, one can still obtain consistent distribution estimates even for distributions with only few data by gaining strength across the entire sample of distributions, while traditional approaches where distributions or densities are estimated individually fail, since sparsely sampled densities cannot be consistently estimated. The proposed model is demonstrated to outperform existing approaches through simulations and Environmental Influences on Child Health Outcomes data.

Selection of Tuning Parameter and Comparison of Lasso and Adaptive Lasso on ZYZ Condition: A Monte Carlo Study

One of the fundamental objectives of statistics is to achieve accurate predictions. In high-dimensional settings (where the number of variables, p, exceeds the number of observations, n), the performance of ordinary least squares (OLS) is often suboptimal due to its high variance, which leads to lower prediction accuracy. Shrinking the variables is a promising approach, and methods such as ridge regression, elastic net, lasso, and adaptive lasso are well-known techniques for this purpose. While variable shrinkage introduces a small bias, it significantly reduces the variance compared to OLS. The effectiveness of shrinkage methods largely depends on the selection of the tuning parameter. Cross-validation and the Bayesian Information Criterion (BIC) are commonly used for this purpose, and an improved version of BIC has shown impressive results.???????? = (5.6,5.6,5.6,0), ???????? = (3,1.5,0,0,2,0,0,0), ???????? = (0.85,0.85,0.85,0) are the multiple regression models which are compared.

Survival Analysis Based Qos Recommendation for Bus Transportation using Deep Learning

Public transportation play an important role in metropolitan cities.Bus services are an important part of public transportation systems, which are frequently chosen because they are convenient, economical, environmentally beneficial, and less taxing on the infrastructure. The temporal patterns of bus transportation incidents are examined in this study using survival analysis, with a focus on the amount of time until certain events like delays, malfunctions, or safe arrivals at destinations. High-dimensional data is being generated at an increasing rate due to technological advancements. As a result, it is quite difficult to analyze such a dataset adequately. Algorithms for machine learning (ML) have become popular tools for modeling complex and nonlinear interactions in a variety of real-world applications, including as the analysis of high-dimensional survival data. Multilayer Deep Neural Network (DNN) models have achieved impressive results recently. Thus, using Keras and TensorFlow, a Cox-based DNN prediction survival model (DNNSurv model) was created. Its findings, however, were limited to survival datasets with large sample sizes or high dimensionality. Using high dimensional survival data, the proposed work will evaluate the DNNSurv model's prediction ability and compare it to a well-known machine learning survival prediction model (random survival forest). The Proposed Work also offers the best settings for a number of hyperparameters, including tuning parameter selection, for this reason. The suggested approach showed through data analysis that, when compared to the ML model, the DNNSurv model performed better overall in terms of the two primary survival prediction evaluation measures (i.e., the concordance index and the time-dependent AUC).

A framework to select tuning parameters for nonparametric derivative estimation.

In this paper, we propose a general framework to select tuning parameters for the nonparametric derivative estimation. The new framework broadens the scope of the previously proposed generalized criterion by replacing the empirical derivative with any other linear nonparametric smoother. We provide the theoretical support of the proposed derivative estimation in a random design and justify it through simulation studies. The practical application of the proposed framework is demonstrated in the study of the age effect on hippocampal gray matter volume in healthy adults from the IXI dataset and the study of the effect of age and body mass index on blood pressure from the Pima Indiansdataset.

Imputation-Based Variable Selection Method for Block-Wise Missing Data When Integrating Multiple Longitudinal Studies

When integrating data from multiple sources, a common challenge is block-wise missing. Most existing methods address this issue only in cross-sectional studies. In this paper, we propose a method for variable selection when combining datasets from multiple sources in longitudinal studies. To account for block-wise missing in covariates, we impute the missing values multiple times based on combinations of samples from different missing pattern and predictors from different data sources. We then use these imputed data to construct estimating equations, and aggregate the information across subjects and sources with the generalized method of moments. We employ the smoothly clipped absolute deviation penalty in variable selection and use the extended Bayesian Information Criterion criteria for tuning parameter selection. We establish the asymptotic properties of the proposed estimator, and demonstrate the superior performance of the proposed method through numerical experiments. Furthermore, we apply the proposed method in the Alzheimer’s Disease Neuroimaging Initiative study to identify sensitive early-stage biomarkers of Alzheimer’s Disease, which is crucial for early disease detection and personalized treatment.

Cross-validation approaches for penalized Cox regression.

Cross-validation is the most common way of selecting tuning parameters in penalized regression, but its use in penalized Cox regression models has received relatively little attention in the literature. Due to its partial likelihood construction, carrying out cross-validation for Cox models is not straightforward, and there are several potential approaches for implementation. Here, we propose a new approach based on cross-validating the linear predictors of the Cox model and compare it to approaches that have been proposed elsewhere. We show that the proposed approach offers an attractive balance of performance and numerical stability, and illustrate these advantages using simulated data as well as analyzing a high-dimensional study of gene expression and survival in lung cancer patients.

Evaluating and improving health equity and fairness of polygenic scores

Polygenic scores (PGSs) are quantitative metrics for predicting phenotypic values, such as human height or disease status. Some PGS methods require only summary statistics of a relevant genome-wide association study (GWAS) for their score. One such method is Lassosum, which inherits the model selection advantages of Lasso to select a meaningful subset of the GWAS single-nucleotide polymorphisms as predictors from their association statistics. However, even efficient scores like Lassosum, when derived from European-based GWASs, are poor predictors of phenotype for subjects of non-European ancestry; that is, they have limited portability to other ancestries. To increase the portability of Lassosum, when GWAS information and estimates of linkage disequilibrium are available for both ancestries, we propose Joint-Lassosum (JLS). In the simulation settings we explore, JLS provides more accurate PGSs compared to other methods, especially when measured in terms of fairness. In analyses of UK Biobank data, JLS was computationally more efficient but slightly less accurate than a Bayesian comparator, SDPRX. Like all PGS methods, JLS requires selection of predictors, which are determined by data-driven tuning parameters. We describe a new approach to selecting tuning parameters and note its relevance for model selection for any PGS. We also draw connections to the literature on algorithmic fairness and discuss how JLS can help mitigate fairness-related harms that might result from the use of PGSs in clinical settings. While no PGS method is likely to be universally portable, due to the diversity of human populations and unequal information content of GWASs for different ancestries, JLS is an effective approach for enhancing portability and reducing predictive bias.

An Enhanced Extreme Learning Machine Based on Square-Root Lasso Method

Extreme learning machine (ELM) is one of the most notable machine learning algorithms with many advantages, especially its training speed. However, ELM has some drawbacks such as instability, poor generalizability and overfitting in the case of multicollinearity in the linear model. This paper introduces square-root lasso ELM (SQRTL-ELM) as a novel regularized ELM algorithm to deal with these drawbacks of ELM. A modified version of the alternating minimization algorithm is used to obtain the estimates of the proposed method. Various techniques are presented to determine the tuning parameter of SQRTL-ELM. The method is compared with the basic ELM, RIDGE-ELM, LASSO-ELM and ENET-ELM on six benchmark data sets. Performance evaluation results show that the SQRTL-ELM exhibits satisfactory performance in terms of testing root mean squared error in benchmark data sets for the sake of slightly extra computation time. The superiority level of the method depends on the tuning parameter selection technique. As a result, the proposed method can be considered a powerful alternative to avoid performance loss in regression problems .

L1 Regularization for High-Dimensional Multivariate GARCH Models

The complexity of estimating multivariate GARCH models increases significantly with the increase in the number of asset series. To address this issue, we propose a general regularization framework for high-dimensional GARCH models with BEKK representations, and obtain a penalized quasi-maximum likelihood (PQML) estimator. Under some regularity conditions, we establish some theoretical properties, such as the sparsity and the consistency, of the PQML estimator for the BEKK representations. We then carry out simulation studies to show the performance of the proposed inference framework and the procedure for selecting tuning parameters. In addition, we apply the proposed framework to analyze volatility spillover and portfolio optimization problems, using daily prices of 18 U.S. stocks from January 2016 to January 2018, and show that the proposed framework outperforms some benchmark models.

Tuning parameter selection for nonparametric derivative estimation in random design

Estimation of a function, or its derivatives via nonparametric regression requires selection of one or more tuning parameters. In the present work, we propose a tuning parameter selection criterion called DCp for nonparametric derivative estimation in random design. Our criterion is general in that it can be applied with any nonparametric estimation method which is linear in the observed outcomes. Charnigo et al. [A generalized C p criterion for derivative estimation. Technometrics. 2011;53(3):238–253] had proposed a GCp criterion for a similar purpose, assuming values of the covariate to be fixed and constant error variance. Here we consider the setting with random design and non-constant error variance since the covariate values will not generally be fixed and equally spaced in real data applications. We justify DCp in this setting both theoretically and by simulation. We also illustrate use of DCp with two economics data sets.

Hierarchical nuclear norm penalization for multi-view data integration.

The prevalence of data collected on the same set of samples from multiple sources (i.e., multi-view data) has prompted significant development of data integration methods based on low-rank matrix factorizations. These methods decompose signal matrices from each view into the sum of shared and individual structures, which are further used for dimension reduction, exploratory analyses, and quantifying associations across views. However, existing methods have limitations in modeling partially-shared structures due to either too restrictive models, or restrictive identifiability conditions. To address these challenges, we propose a new formulation for signal structures that include partially-shared signals based on grouping the views into so-called hierarchical levels with identifiable guarantees under suitable conditions. The proposed hierarchy leads us to introduce a new penalty, hierarchical nuclear norm (HNN), for signal estimation. In contrast to existing methods, HNN penalization avoids scores and loadings factorization of the signals and leads to a convex optimization problem, which we solve using a dual forward-backward algorithm. We propose a simple refitting procedure to adjust the penalization bias and develop an adapted version of bi-cross-validation for selecting tuning parameters. Extensive simulation studies and analysis of the genotype-tissue expression data demonstrate the advantages of our method over existingalternatives.

On the Use of Minimum Penalties in Statistical Learning

Modern multivariate machine learning and statistical methodologies estimate parameters of interest while leveraging prior knowledge of the association between outcome variables. The methods that do allow for estimation of relationships do so typically through an error covariance matrix in multivariate regression which does not generalize to other types of models. In this article we proposed the MinPen framework to simultaneously estimate regression coefficients associated with the multivariate regression model and the relationships between outcome variables using common assumptions. The MinPen framework uses a novel penalty based on the minimum function to simultaneously detect and exploit relationships between responses. An iterative algorithm is proposed as a solution to the nonconvex optimization. Theoretical results such as high-dimensional convergence rates, model selection consistency, and a framework for post selection inference are provided. We extend the proposed MinPen framework to other exponential family loss functions, with a specific focus on multiple binomial responses. Tuning parameter selection is also addressed. Finally, simulations and two data examples are presented to show the finite sample properties of this framework. Supplemental material providing proofs, additional simulations, code, and datasets are available online.

The effect of inflammatory-nutri̇tional prognosti̇c scori̇ng (INPS) system on treatment response and prognosi̇s i̇n pati̇ents wi̇th locally advanced gastroesophageal juncti̇on and gastri̇c cancer wi̇th neoadjuvant systemi̇c treatment.

e16092 Background: We aimed to reveal the prognostic value of inflammatory-nutritional prognostic score (INPS), which is a new scoring system created by using frequently used immune-nutritional markers, as a biomarker, on nutrition and immune status in patients with locally advanced GEJ and gastric cancer. Methods: Patients who were treated in Dokuz Eylul University Medical Oncology Department between 2010-2022 years and diagnosed with locally advanced GEJ and gastric cancer and receiving NACT were included in this study retrospectively. We selected the most valuable biomarkers (Albumin, ABR, CRP, CAR, De-Ritis, HALP, MLR, NLR, PLR, PNI, SII) to develop INPS by the least absolute shrinkage and selection operator (LASSO) Cox regression model. A prognostic nomogram incorporating INPS and other independent clinicopathological factors was developed based on the stepwise multivariate Cox regression method. The retained features with nonzero coefficients were used to establish a novel INPS. The selected four biomarkers were used to construct the novel INPS for our patients. Results: Median follow-up time of 101 patients was 16.7 months (95% CI= 3.7-73.7), DFS was 20.5 months (95%CI= 16.2-24.8 ), OS was 36.6 months (95%CI=25.4-47.8). The 101 patients in the study were randomly assigned to train (n = 71) and test (n = 30) sets in a 7:3 ratio. Using the LASSO Cox regression model, four inflammatory-nutritional biomarkers (INB) with nonzero coefficients, namely, PLR, DeRitis, PNI and LAR, of the 13 parameters were selected. The LASSO coefficient profiles of the 13 INB and 10-fold cross-validation for tuning parameter selection in the LASSO model were presented. The optimal cut-off values for these four biomarkers based on disease-free survival were determined using the Maximally Selected Rank Statistics. The respective cut-off values for PLR, DeRitis, PNI, and LAR were obtained to be 107.93, 1.75, 33.92, and 58.17. Both the patients in the Train set and the in the Test set were classified as low-risk (INPS score ≤ 1) and high-risk (INPS score >1) groups. There was a statistically significant difference in survival probabilities between low and high risk groups in both groups (respectively; p=0.009, p=0.029). A prognostic nomogram including INPS and other independent clinicopathological factors was developed. Conclusions: The search for biomarkers with prognostic value in GEJ and gastric adenocarcinoma receiving NACT continues. With this study, it has been shown for the first time in the literature that the new screening system, INPS, created with nutritional and inflammatory markers, can be used as a prognostic tool to predict disease-free survival in this patient group.

A frequentist approach to dynamic borrowing.

There has been growing interest in leveraging external control data to augment a randomized control group data in clinical trials and enable more informative decision making. In recent years, the quality and availability of real-world data have improved steadily as external controls. However, information borrowing by directly pooling such external controls with randomized controls may lead to biased estimates of the treatment effect. Dynamic borrowing methods under the Bayesian framework have been proposed to better control the false positive error. However, the numerical computation and, especially, parameter tuning, of those Bayesian dynamic borrowing methods remain a challenge in practice. In this paper, we present a frequentist interpretation of a Bayesian commensurate prior borrowing approach and describe intrinsic challenges associated with this method from the perspective of optimization. Motivated by this observation, we propose a new dynamic borrowing approach using adaptive lasso. The treatment effect estimate derived from this method follows a known asymptotic distribution, which can be used to construct confidence intervals and conduct hypothesis tests. The finite sample performance of the method is evaluated through extensive Monte Carlo simulations under different settings. We observed highly competitive performance of adaptive lasso compared to Bayesian approaches. Methods for selecting tuning parameters are also thoroughly discussed based on results from numerical studies and an illustrationexample.

A marginal modelling approach for predicting wildfire extremes across the contiguous United States

This paper details a methodology proposed for the EVA 2021 conference data challenge. The aim of this challenge was to predict the number and size of wildfires over the contiguous US between 1993 and 2015, with more importance placed on extreme events. In the data set provided, over 14% of both wildfire count and burnt area observations are missing; the objective of the data challenge was to estimate a range of marginal probabilities from the distribution functions of these missing observations. To enable this prediction, we make the assumption that the marginal distribution of a missing observation can be informed using non-missing data from neighbouring locations. In our method, we select spatial neighbourhoods for each missing observation and fit marginal models to non-missing observations in these regions. For the wildfire counts, we assume the compiled data sets follow a zero-inflated negative binomial distribution, while for burnt area values, we model the bulk and tail of each compiled data set using non-parametric and parametric techniques, respectively. Cross validation is used to select tuning parameters, and the resulting predictions are shown to significantly outperform the benchmark method proposed in the challenge outline. We conclude with a discussion of our modelling framework, and evaluate ways in which it could be extended.

Tuning parameter selection for penalized estimation via R2

The tuning parameter selection strategy for penalized estimation is crucial to identify a model that is both interpretable and predictive. However, popular strategies (e.g., minimizing average squared prediction error via cross-validation) tend to select models with more predictors than necessary. A simple yet powerful cross validation strategy is proposed which is based on maximizing the squared correlation between the observed and predicted values, rather than minimizing squared error loss for the purposes of support recovery. The strategy can be applied to all penalized least-squares estimators and, under certain conditions, the metric implicitly performs a bias adjustment named the α-modification. When applied to the Lasso estimator, the α-modification is closely related to the relaxed Lasso estimator. The approach is demonstrated on a functional variable selection problem to identify optimal placement of surface electromyogram sensors to control a robotic hand prosthesis.

Variable selection in heterogeneous panel data models with cross‐sectional dependence

AbstractThis paper studies the Bridge estimator for a high‐dimensional panel data model with heterogeneous varying coefficients, where the random errors are assumed to be serially correlated and cross‐sectionally dependent. We establish oracle efficiency and the asymptotic distribution of the Bridge estimator, when the number of covariates increases to infinity with the sample size in both dimensions. A BIC‐type criterion is also provided for tuning parameter selection. We further generalise the marginal Bridge estimator for our model to asymptotically correctly identify the covariates with zero coefficients even when the number of covariates is greater than the sample size under a partial orthogonality condition. The finite sample performance of the proposed estimator is demonstrated by simulated data examples, and an empirical application with the US stock dataset is also provided.

Sparse Estimation in Finite Mixture of Accelerated Failure Time and Mixture of Regression Models with R Package fmrs

Variable selection in large-dimensional data has been extensively studied in different settings over the past decades. In a recent article, Shokoohi et. al. [29, DOI:10.1214/18-AOAS1198] proposed a method for variable selection in finite mixture of accelerated failure time regression models for studies on time-to-event data to capture heterogeneity within the population and account for censoring. In this paper, we introduce the fmrs package, which implements the variable selection methodology for such models. Furthermore, as a byproduct, the fmrs package facilitates variable selection in finite mixture regression models. The package also incorporates a tuning parameter selection mechanism based on component-wise bic. Commonly used penalties, such as Least Absolute Shrinkage and Selection Operator, and Smoothly Clipped Absolute Deviation, are integrated into fmrs. Additionally, the package offers an option for non-mixture regression models. The C language is chosen to boost the optimization speed. We provide an overview of the fmrs principles and the strategies employed for optimization. Hands-on illustrations are presented to help users get acquainted with fmrs. Finally, we apply fmrs to a lung cancer dataset and observe that a two-component mixture model reveals a subgroup with a more aggressive form of the disease, displaying a lower survival time.

Six Phase Transmission Line Protection Using Bat Algorithm Tuned Stacked Sparse Autoencoder

Six-phase transmission lines have the capability to address the continually evolving power demand. It allows upgrading the power transfer capability of the prevailing three-phase double circuit line without major changes in the transmission corridors. However, the operational performance of any six-phase system is highly dependent on its protection scheme. The possibility of larger number of faults in six-phase system complicates the protection task. Furthermore, the harmonics intrusion arising because of nonlinear loading compromises the reliability of the conventional threshold-based protection schemes. In this regard, this article addresses the above-mentioned challenges by developing a protection scheme based on the hybrid frameworks of bat algorithm and stacked sparse autoencoder-deep neural network (SAE-DNN). To overcome the limitation of SAE-DNN regarding optimal selection of architecture and tuning parameters, the selection task has been formulated as an optimization problem and solved using Bat algorithm. The use of raw voltage and current signals as input to the SAE-DNN reduces the overall complexity of the protection scheme. The efficacy of the proposed scheme has been validated for all 120 types of faults under varying operating, loading and fault scenarios. Furthermore, the proposed scheme has been validated for practical settings by performing real-time simulations on OPAL-RT digital simulator.

Analytical performance estimation during code generation on modern GPUs

Automatic code generation is frequently used to create implementations of algorithms specifically tuned to particular hardware and application parameters. The code generation process involves the selection of adequate code transformations, tuning parameters, and parallelization strategies. We propose an alternative to time-intensive autotuning, scenario-specific performance models, or black-box machine learning to select the best-performing configuration.This paper identifies the relevant performance-defining mechanisms for memory-intensive GPU applications through a performance model coupled with an analytic hardware metric estimator. This enables a quick exploration of large configuration spaces to identify highly efficient code candidates with high accuracy.We examine the changes of the A100 GPU architecture compared to the predecessor V100 and address the challenges of how to model the data transfer volumes through the new memory hierarchy.We show how our method can be coupled to the “pystencils” stencil code generator, which is used to generate kernels for a range-four 3D-25pt stencil and a complex two-phase fluid solver based on the Lattice Boltzmann Method. For both, it delivers a ranking that can be used to select the best-performing candidate.The method is not limited to stencil kernels but can be integrated into any code generator that can generate the required address expressions.