Penalized Likelihood Approach Research Articles

Penalized variable selection for accelerated failure time models

The accelerated failure time (AFT) model is a linear model under the log-transformation of survival time that has been introduced as a useful alternative to the proportional hazards (PH) model. In this paper we propose variable-selection procedures of fixed effects in a parametric AFT model using penalized likelihood approaches. We use three popular penalty functions, least absolute shrinkage and selection operator (LASSO), adaptive LASSO and smoothly clipped absolute deviation (SCAD). With these procedures we can select important variables and estimate the fixed effects at the same time. The performance of the proposed method is evaluated using simulation studies, including the investigation of impact of misspecifying the assumed distribution. The proposed method is illustrated with a primary biliary cirrhosis (PBC) data set.

Hypothesis testing via a penalized-likelihood approach

It is illustrated in this paper that hypothesis testing procedures can be derived based on the penalized likelihood approach. Based on this point of view, many traditional hypothesis tests, including the two-sample mean test, score test, and Hotelling’s T2 test are revisited under the penalized likelihood framework. Similar framework is also applicable to the empirical likelihood.

Model-based clustering with sparse covariance matrices

Finite Gaussian mixture models are widely used for model-based clustering of continuous data. Nevertheless, since the number of model parameters scales quadratically with the number of variables, these models can be easily over-parameterized. For this reason, parsimonious models have been developed via covariance matrix decompositions or assuming local independence. However, these remedies do not allow for direct estimation of sparse covariance matrices nor do they take into account that the structure of association among the variables can vary from one cluster to the other. To this end, we introduce mixtures of Gaussian covariance graph models for model-based clustering with sparse covariance matrices. A penalized likelihood approach is employed for estimation and a general penalty term on the graph configurations can be used to induce different levels of sparsity and incorporate prior knowledge. Model estimation is carried out using a structural-EM algorithm for parameters and graph structure estimation, where two alternative strategies based on a genetic algorithm and an efficient stepwise search are proposed for inference. With this approach, sparse component covariance matrices are directly obtained. The framework results in a parsimonious model-based clustering of the data via a flexible model for the within-group joint distribution of the variables. Extensive simulated data experiments and application to illustrative datasets show that the method attains good classification performance and model quality. The general methodology for model-based clustering with sparse covariance matrices is implemented in the R package mixggm, available on CRAN.

Nonparametric Finite Mixture of Gaussian Graphical Models

ABSTRACTGraphical models have been widely used to investigate the complex dependence structure of high-dimensional data, and it is common to assume that observed data follow a homogeneous graphical model. However, observations usually come from different resources and have heterogeneous hidden commonality in real-world applications. Thus, it is of great importance to estimate heterogeneous dependencies and discover a subpopulation with certain commonality across the whole population. In this work, we introduce a novel regularized estimation scheme for learning nonparametric finite mixture of Gaussian graphical models, which extends the methodology and applicability of Gaussian graphical models and mixture models. We propose a unified penalized likelihood approach to effectively estimate nonparametric functional parameters and heterogeneous graphical parameters. We further design an efficient generalized effective expectation-maximization (EM) algorithm to address three significant challenges: high-dimensionality, nonconvexity, and label switching. Theoretically, we study both the algorithmic convergence of our proposed algorithm and the asymptotic properties of our proposed estimators. Numerically, we demonstrate the performance of our method in simulation studies and a real application to estimate human brain functional connectivity from attention deficit hyperactivity disorder (ADHD) imaging data, where two heterogeneous conditional dependencies are explained through profiling demographic variables and supported by existing scientific findings.

Regularized Latent Class Model for Joint Analysis of High-Dimensional Longitudinal Biomarkers and a Time-to-Event Outcome.

Although many modeling approaches have been developed to jointly analyze longitudinal biomarkers and a time-to-event outcome, most of these methods can only handle one or a few biomarkers. In this article, we propose a novel joint latent class model to deal with high dimensional longitudinal biomarkers. Our model has three components: a class membership model, a survival submodel, and a longitudinal submodel. In our model, we assume that covariates can potentially affect biomarkers and class membership. We adopt a penalized likelihood approach to infer which covariates have random effects and/or fixed effects on biomarkers, and which covariates are informative for the latent classes. Through extensive simulation studies, we show that our proposed method has improved performance in prediction and assigning subjects to the correct classes over other joint modeling methods and that bootstrap can be used to do inference for our model. We then apply our method to a dataset of patients with idiopathic pulmonary fibrosis, for whom gene expression profiles were measured longitudinally. We are able to identify four interesting latent classes with one class being at much higher risk of death compared to the other classes. We also find that each of the latent classes has unique trajectories in some genes, yielding novel biological insights.

Inference on phenotype-specific effects of genes using multivariate kernel machine regression.

We consider the problem of assessing the joint effect of a set of genetic markers on multiple, possibly correlated phenotypes of interest. We develop a kernel machine based multivariate regression framework, where the joint effect of the marker set on each of the phenotypes is modeled using prespecified kernel functions with unknown variance components. Unlike most existing methods that mainly focus on the global association between the marker set and the phenotype set, we develop estimation and testing procedures to study phenotype-specific associations. Specifically, we develop an estimation method based on the penalized likelihood approach to estimate phenotype-specific effects and their corresponding standard errors while accounting for possible correlation among the phenotypes. We develop testing procedures for the association of the marker set with any subset of phenotypes using a score-based variance components testing method. We assess the performance of our proposed methodology via a simulation study and demonstrate the utility of the proposed method using the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) data.

Selection by partitioning the solution paths

The performance of penalized likelihood approaches depends profoundly on the selection of the tuning parameter; however, there is no commonly agreed-upon criterion for choosing the tuning parameter. Moreover, penalized likelihood estimation based on a single value of the tuning parameter suffers from several drawbacks. This article introduces a novel approach for feature selection based on the entire solution paths rather than the choice of a single tuning parameter, which significantly improves the accuracy of the selection. Moreover, the approach allows for feature selection using ridge or other strictly convex penalties. The key idea is to classify variables as relevant or irrelevant at each tuning parameter and then to select all of the variables which have been classified as relevant at least once. We establish the theoretical properties of the method, which requires significantly weaker conditions than existing methods in the literature. We also illustrate the advantages of the proposed approach with simulation studies and a data example.

A penalized likelihood approach to model the annual maximum flow with small sample sizes

The aim of this study is to model the annual maximum flow of several sites in Sabah with small sample sizes using the generalized extreme value (GEV) distribution. Previous studies have shown that the standard method of maximum likelihood estimates would give a poor estimation of the GEV parameters and quantiles for small data set. This study will consider the penalized likelihood estimates as an alternative method to improve the inference over the standard method and retains the modeling flexibility. As comparisons, we will illustrate the results of both methods to model the annual maximum flow in Sabah. The results show the implementation of the penalty function had the same effect to the GEV parameter estimates as suggested by previous studies.

Penalised inference for lagged dependent regression in the presence of autocorrelated residuals

This paper deals with linear models for a time-dependent response and explanatory variables in a high-dimensional setting. We account for the time dependency in the data by explicitly adding autoregressive terms to the response variable in the model together with an autoregressive process for the residuals. We present a penalized likelihood approach for parameter estimation and discuss its theoretical properties. Finally, we show the successful application of the proposed methodology on simulated data and on two real applications, where we model air pollution and stock market indices, respectively. We provide an implementation of the method in the R package DREGAR, freely available on CRAN, http://CRAN.R-project.org/package=DREGAR.

A Penalized Inference Approach to Stochastic Block Modelling of Community Structure in the Italian Parliament

SummaryWe analyse bill cosponsorship networks in the Italian Chamber of Deputies. In comparison with other parliaments, a distinguishing feature of the Chamber is the large number of political groups. Our analysis aims to infer the pattern of collaborations between these groups from data on bill cosponsorships. We propose an extension of stochastic block models for edge-valued graphs and derive measures of group productivity and of collaboration between political parties. As the model proposed encloses a large number of parameters, we pursue a penalized likelihood approach that enables us to infer a sparse reduced graph displaying collaborations between political parties.

Multilevel Gaussian graphical model for multilevel networks

Gaussian graphical models have become a popular tool to represent networks among variables such as genes. They use the conditional correlations from the joint distribution to describe the dependencies between gene pairs, and employ the precision matrix of the genes. Because of the sparse nature of the gene networks and small sample sizes in high dimensional genetic data, regularization approaches attracted much attention in aim at obtaining the shrinkage estimates of the precision matrix. However, existing methods have been focused on the Gaussian graphical model among genes; that is, they are only applicable to a single level Gaussian graphical model. It is known that pathways are not independent of each other because of shared genes and interactions among pathways. Developing multipathway analysis has been a challenging problem because of the complex dependence structure among pathways. By considering the dependency among pathways as well as the genes within each pathway, we propose a multilevel Gaussian graphical model (MGGM) in which one level describes the networks for genes and the other for pathways. We have developed a multilevel L1 penalized likelihood approach to achieve the sparseness on both levels. In addition, we have developed an iterative weighted graphical LASSO algorithm for MGGM. Our simulation results supported the advantages of our approach; our method estimated the network more accurately on the pathway level and sparser on the gene level. We also demonstrated the usefulness of our approach using a canine genes-pathways data set.

Penalized Likelihood Approach for Simultaneous Analysis of Survival Time and Binary Longitudinal Outcome

In this paper we consider simultaneous analysis of survival time and binary longitudinal outcome where random effects are introduced to account for the dependence between the two different types of outcomes due to unobserved factors and assumed to follow a Gaussian distribution with mean zero. The estimator based on maximum likelihood approach using an Expectation-Maximization algorithm is consistent and asymptotically normally distributed. However, the EM algorithm may be intensive on numerical integrations with large sample sizes and large numbers of longitudinal observations per subject. We develop a more computationally efficient estimation procedure based on a penalized likelihood obtained by Laplace approximation. Through simulation studies, we compare numerical performances on the computing time, bias, and mean squared error from the proposed penalized likelihood estimation procedure and the EM algorithm of maximum likelihood estimation. We also illustrate the proposed approach with a liver transplantation data set.

Penalized Likelihood Approach for Simultaneous Analysis of Survival Time and Binary Longitudinal Outcome

Penalized Likelihood Approach for Simultaneous Analysis of Survival Time and Binary Longitudinal Outcome

A new method for robust mixture regression.

Finite mixture regression models have been widely used for modelling mixed regression relationships arising from a clustered and thus heterogenous population. The classical normal mixture model, despite its simplicity and wide applicability, may fail in the presence of severe outliers. Using a sparse, case-specific, and scale-dependent mean-shift mixture model parameterization, we propose a robust mixture regression approach for simultaneously conducting outlier detection and robust parameter estimation. A penalized likelihood approach is adopted to induce sparsity among the mean-shift parameters so that the outliers are distinguished from the remainder of the data, and a generalized Expectation-Maximization (EM) algorithm is developed to perform stable and efficient computation. The proposed approach is shown to have strong connections with other robust methods including the trimmed likelihood method and M-estimation approaches. In contrast to several existing methods, the proposed methods show outstanding performance in our simulation studies.

Estimating hazard ratios in cohort data with missing disease information due to death.

In clinical and epidemiological studies information on the primary outcome of interest, that is, the disease status, is usually collected at a limited number of follow-up visits. The disease status can often only be retrieved retrospectively in individuals who are alive at follow-up, but will be missing for those who died before. Right-censoring the death cases at the last visit (ad-hoc analysis) yields biased hazard ratio estimates of a potential risk factor, and the bias can be substantial and occur in either direction. In this work, we investigate three different approaches that use the same likelihood contributions derived from an illness-death multistate model in order to more adequately estimate the hazard ratio by including the death cases into the analysis: a parametric approach, a penalized likelihood approach, and an imputation-based approach. We investigate to which extent these approaches allow for an unbiased regression analysis by evaluating their performance in simulation studies and on a real data example. In doing so, we use the full cohort with complete illness-death data as reference and artificially induce missing information due to death by setting discrete follow-up visits. Compared to an ad-hoc analysis, all considered approaches provide less biased or even unbiased results, depending on the situation studied. In the real data example, the parametric approach is seen to be too restrictive, whereas the imputation-based approach could almost reconstruct the original event history information.

Rapid penalized likelihood-based outlier detection via heteroskedasticity test

ABSTRACTOutlier detection is fundamental to statistical modelling. When there are multiple outliers, many traditional approaches in use are stepwise detection procedures, which can be computationally expensive and ignore stochastic error in the outlier detection process. Outlier detection can be performed by a heteroskedasticity test. In this article, a rapid outlier detection method via multiple heteroskedasticity test based on penalized likelihood approaches is proposed to handle these kinds of problems. The proposed method detects the heteroskedasticity of all data only by one step and estimate coefficients simultaneously. The proposed approach is distinguished from others in that a rapid modelling approach uses a weighted least squares formulation coupled with nonconvex sparsity-including penalization. Furthermore, the proposed approach does not need to construct test statistics and calculate their distributions. A new algorithm is proposed for optimizing penalized likelihood functions. Favourable theoretical properties of the proposed approach are obtained. Our simulation studies and real data analysis show that the newly proposed methods compare favourably with other traditional outlier detection techniques.

Penalized empirical likelihood inference for sparse additive hazards regression with a diverging number of covariates

High-dimensional sparse modeling with censored survival data is of great practical importance, as exemplified by applications in high-throughput genomic data analysis. In this paper, we propose a class of regularization methods, integrating both the penalized empirical likelihood and pseudoscore approaches, for variable selection and estimation in sparse and high-dimensional additive hazards regression models. When the number of covariates grows with the sample size, we establish asymptotic properties of the resulting estimator and the oracle property of the proposed method. It is shown that the proposed estimator is more efficient than that obtained from the non-concave penalized likelihood approach in the literature. Based on a penalized empirical likelihood ratio statistic, we further develop a nonparametric likelihood approach for testing the linear hypothesis of regression coefficients and constructing confidence regions consequently. Simulation studies are carried out to evaluate the performance of the proposed methodology and also two real data sets are analyzed.

Cause-specific hazard regression for competing risks data under interval censoring and left truncation

Inference for cause-specific hazards from competing risks data under interval censoring and possible left truncation has been understudied. Aiming at this target, a penalized likelihood approach for a Cox-type proportional cause-specific hazards model is developed, and the associated asymptotic theory is discussed. Monte Carlo simulations show that the approach performs very well for moderate sample sizes. An application to a longitudinal study of dementia illustrates the practical utility of the method. In the application, the age-specific hazards of AD, other dementia and death without dementia are estimated, and risk factors of all competing risks are studied.

Estimating the order of sinusoidal models using the adaptively penalized likelihood approach: Large sample consistency properties

Recently, the paper [2] introduced a method for model order estimation based on penalizing adaptively the likelihood (PAL). In this paper, we use the PAL based order estimation method for a nonlinear sinusoidal model and study its asymptotic statistical properties. We prove that the estimator of the model order using the PAL rule is consistent. Simulation examples are presented to illustrate the performance of the PAL method for small sample sizes and to compare it with that of three information criterion-based methods.

Sparse estimation of Cox proportional hazards models via approximated information criteria.

We propose a new sparse estimation method for Cox (1972) proportional hazards models by optimizing an approximated information criterion. The main idea involves approximation of the ℓ0 norm with a continuous or smooth unit dent function. The proposed method bridges the best subset selection and regularization by borrowing strength from both. It mimics the best subset selection using a penalized likelihood approach yet with no need of a tuning parameter. We further reformulate the problem with a reparameterization step so that it reduces to one unconstrained nonconvex yet smooth programming problem, which can be solved efficiently as in computing the maximum partial likelihood estimator (MPLE). Furthermore, the reparameterization tactic yields an additional advantage in terms of circumventing postselection inference. The oracle property of the proposed method is established. Both simulated experiments and empirical examples are provided for assessment and illustration.