Penalized Likelihood Estimation Research Articles

Dynamic Networks with Multi-scale Temporal Structure

We describe a novel method for modeling non-stationary multivariate time series, with time-varying conditional dependencies represented through dynamic networks. Our proposed approach combines traditional multi-scale modeling and network based neighborhood selection, aiming at capturing temporally local structure in the data while maintaining sparsity of the potential interactions. Our multi-scale framework is based on recursive dyadic partitioning, which recursively partitions the temporal axis into finer intervals and allows us to detect local network structural changes at varying temporal resolutions. The dynamic neighborhood selection is achieved through penalized likelihood estimation, where the penalty seeks to limit the number of neighbors used to model the data. We present theoretical and numerical results describing the performance of our method, which is motivated and illustrated using task-based magnetoencephalography (MEG) data in neuroscience.

Bias reduction and model selection in misspecified models

This article concerns maximum penalized likelihood estimation in misspecified generalized linear models with independent and identically distributed observations. A new method for simultaneous model selection and estimation with bias reduction is proposed in the framework. A discontinuous penalized likelihood function is used, and an approximate method to solve the discontinuous optimization problem is introduced. The proposed method has model selection consistency in a sparse regression setting in which the dimension of predictors is fixed and the sample size increases. The efficiency of the proposed method is illustrated through a finite simulation study.

A Solution to Separation and Multicollinearity in Multiple Logistic Regression

In dementia screening tests, item selection for shortening an existing screening test can be achieved using multiple logistic regression. However, maximum likelihood estimates for such logistic regression models often experience serious bias or even non-existence because of separation and multicollinearity problems resulting from a large number of highly cor related items. Firth (1993, Biometrika, 80(1),27-38) proposed a penalized likelihood estimator for generalized linear models and it was shown to re duce bias and the non-existence problems. The ridge regression has been used in logistic regression to stabilize the estimates in cases of multicollinear ity. However, neither solves the problems for each other. In this paper, we propose a double penalized maximum likelihood estimator combining Firth’s penalized likelihood equation with a ridge parameter. We present a simu lation study evaluating the empirical performance of the double penalized likelihood estimator in small to moderate sample sizes. We demonstrate the proposed approach using a current screening data from a community-based dementia study.

Penalized Likelihood Estimation of Gamma Distributed Response Variable via Corrected Solution of Regression Coefficients

A Gamma distributed response is subjected to regression penalized likelihood estimations of Least Absolute Shrinkage and Selection Operator (LASSO) and Minimax Concave Penalty via Generalized Linear Models (GLMs). The Gamma related disturbance controls the influence of skewness and spread in the corrected path solutions of the regression coefficients.

Additive models with autoregressive symmetric errors based on penalized regression splines

In this paper additive models with p-order autoregressive conditional symmetric errors based on penalized regression splines are proposed for modeling trend and seasonality in time series. The aim with this kind of approach is try to model the autocorrelation and seasonality properly to assess the existence of a significant trend. A backfitting iterative process jointly with a quasi-Newton algorithm are developed for estimating the additive components, the dispersion parameter and the autocorrelation coefficients. The effective degrees of freedom concerning the fitting are derived from an appropriate smoother. Inferential results and selection model procedures are proposed as well as some diagnostic methods, such as residual analysis based on the conditional quantile residual and sensitivity studies based on the local influence approach. Simulations studies are performed to assess the large sample behavior of the maximum penalized likelihood estimators. Finally, the methodology is applied for modeling the daily average temperature of San Francisco city from January 1995 to April 2020.

A Penalized Likelihood Estimator for Variance Components in Repeated Measurements Model

In this paper, we consider the repeated measurement model. We estimate the parameters of the model by using the maximum likelihood method and compared these estimators with the penalized likelihood estimators by using mean square error. We also determine the corresponding moments of each estimators. Finally, we find bias equations and variance for each estimator.

Manifold Reconstruction of Differences: A Model-Based Iterative Statistical Estimation Algorithm With a Data-Driven Prior.

Manifold learning using deep neural networks been shown to be an effective tool for building sophisticated prior image models that can be applied to noise reduction in low-close CT. A manifold is a low-dimensional space that captures most of the variation in a class of data (e.g. chest CT images). This nonlinear reduction in dimensionality is one way to build up a sophisticated model of the features and structures present in image data. This nonlinear dimensionality reduction tends to learn features associated with the signal rather than the noise when provided with a large training dataset. However, the internal workings operation are typically not easy to understand. The results can be highly nonlinear and object-dependent. After training the model is fixed and during reconstruction is typically no check for fitness between the image estimates and the measured data. We propose a new iterative CT reconstruction algorithm, called Manifold Reconstruction of Differences (MRoD), which combines physical and statistical models with a data-driven prior based on manifold learning. The MRoD algorithm involves estimating a manifold component, approximating common features among all patients, and the difference component which has the freedom to fit the measured data. By applying a sparsity-promoting penalty to the difference image rather than a hard constraint to the manifold, the MRoD algorithm is able to reconstruct features which are not present in the training data. The difference component itself may be independently useful. While the manifold captures typical patient features (e.g. healthy anatomy), the difference image highlights patient-specific elements (e.g. pathology). In this work, we present a description of an optimization framework which combines trained manifold-based modules with physical modules. We present a simulation study using anthropomorphic lung data showing that the MRoD algorithm can both isolate differences between a particular patient and the typical distribution, but also provide significant noise reduction with less bias than a typical penalized likelihood estimator in composite manifold plus difference reconstructions.

Can a penalized-likelihood estimation algorithm be used to reduce the injected dose or the acquisition time in 68Ga-DOTATATE PET/CT studies?

BackgroundImage quality and quantitative accuracy of positron emission tomography (PET) depend on several factors such as uptake time, scanner characteristics and image reconstruction methods. Ordered subset expectation maximization (OSEM) is considered the gold standard for image reconstruction. Penalized-likelihood estimation (PL) algorithms have been recently developed for PET reconstruction to improve quantitation accuracy while maintaining or even improving image quality. In PL algorithms, a regularization parameter β controls the penalization of relative differences between neighboring pixels and determines image characteristics. In the present study, we aim to compare the performance of Q.Clear (PL algorithm, GE Healthcare) and OSEM (3 iterations, 8 subsets, 6-mm post-processing filter) for 68Ga-DOTATATE (68Ga-DOTA) PET studies, both visually and quantitatively.Thirty consecutive whole-body 68Ga-DOTA studies were included. The data were acquired in list mode and were reconstructed using 3D OSEM and Q.Clear with various values of β and various acquisition times per bed position (bp), thus generating images with reduced injected dose (1.5 min/bp: β = 300–1100; 1.0 min/bp: β = 600–1400 and 0.5 min/bp: β = 800–2200). An additional analysis adding β values up to 1500, 1700 and 3000 for 1.5, 1.0 and 0.5 min/bp, respectively, was performed for a random sample of 8 studies. Evaluation was performed using a phantom and clinical data. Two experienced nuclear medicine physicians blinded to the variables assessed the image quality visually.ResultsClinical images reconstructed with Q.Clear, set at 1.5, 1.0 and 0.5 min/bp using β = 1100, 1300 and 3000, respectively, resulted in images with noise equivalence to 3D OSEM (1.5 min/bp) with a mean increase in SUVmax of 14%, 13% and 4%, an increase in SNR of 30%, 24% and 10%, and an increase in SBR of 13%, 13% and 2%. Visual assessment yielded similar results for β values of 1100–1400 and 1300–1600 for 1.5 and 1.0 min/bp, respectively, although for 0.5 min/bp there was no significant improvement compared to OSEM.Conclusion68Ga-DOTA reconstructions with Q.Clear, 1.5 and 1.0 min/bp, resulted in increased tumor SUVmax and in improved SNR and SBR at a similar level of noise compared to 3D OSEM. Q.Clear with β = 1300–1600 enables one-third reduction of acquisition time or injected dose, with similar image quality compared to 3D OSEM.

Penalized likelihood approach for the four-parameter kappa distribution

The four-parameter kappa distribution (K4D) is a generalized form of some commonly used distributions such as generalized logistic, generalized Pareto, generalized Gumbel, and generalized extreme value (GEV) distributions. Owing to its flexibility, the K4D is widely applied in modeling in several fields such as hydrology and climatic change. For the estimation of the four parameters, the maximum likelihood approach and the method of L-moments are usually employed. The L-moment estimator (LME) method works well for some parameter spaces, with up to a moderate sample size, but it is sometimes not feasible in terms of computing the appropriate estimates. Meanwhile, using the maximum likelihood estimator (MLE) with small sample sizes shows substantially poor performance in terms of a large variance of the estimator. We therefore propose a maximum penalized likelihood estimation (MPLE) of K4D by adjusting the existing penalty functions that restrict the parameter space. Eighteen combinations of penalties for two shape parameters are considered and compared. The MPLE retains modeling flexibility and large sample optimality while also improving on small sample properties. The properties of the proposed estimator are verified through a Monte Carlo simulation, and an application case is demonstrated taking Thailand’s annual maximum temperature data.

Bayesian Approaches to Shrinkage and Sparse Estimation

In all areas of human knowledge, datasets are increasing in both size and complexity, creating the need for richer statistical models. This trend is also true for economic data, where high-dimensional and nonlinear/nonparametric inference is the norm in several fields of applied econometric work. The purpose of this paper is to introduce the reader to the world of Bayesian model determination, by surveying modern shrinkage and variable selection algorithms and methodologies. Bayesian inference is a natural probabilistic framework for quantifying uncertainty and learning about model parameters, and this feature is particularly important for inference in modern models of high dimensions and increased complexity. We begin with a linear regression setting in order to introduce various classes of priors that lead to shrinkage/sparse estimators of comparable value to popular penalized likelihood estimators (e.g.\ ridge, lasso). We explore various methods of exact and approximate inference, and discuss their pros and cons. Finally, we explore how priors developed for the simple regression setting can be extended in a straightforward way to various classes of interesting econometric models. In particular, the following case-studies are considered, that demonstrate application of Bayesian shrinkage and variable selection strategies to popular econometric contexts: i) vector autoregressive models; ii) factor models; iii) time-varying parameter regressions; iv) confounder selection in treatment effects models; and v) quantile regression models. A MATLAB package and an accompanying technical manual allow the reader to replicate many of the algorithms described in this review.

Bias approximations for likelihood‐based estimators

AbstractBias approximation has played an important rôle in statistical inference, and numerous bias calculation techniques have been proposed under different contexts. We provide a unified approach to approximating the bias of the maximum likelihood estimator and the l2 penalized likelihood estimator for both linear and nonlinear models, where the design variables are allowed to be random and the sample size can be a stopping time. The proposed method is based on the Woodroofe–Stein identity and is justified by very weak approximations. The accuracy of the derived bias formulas is assessed by simulation for several examples. The bias of the ridge estimator in high‐dimensional settings is also discussed.

Penalized Lq-likelihood estimators and variable selection in linear regression models

Consider a linear regression model where are independent identically distributed (iid) random variables with zero mean and known variance Based on the maximum Lq-likelihood estimator (MLqE) and the penalized likelihood estimator (PLE), we introduce a new parametric estimator which is called penalized Lq-likelihood estimator (PLqE). We investigate its Oracle properties and influence function. Simulation results support the validity of our approach. Furthermore, it is shown that the PLqE is robust, while the PLE is not.

Spectrally Sparse Nonparametric Regression via Elastic Net Regularized Smoothers

Nonparametric regression frameworks, such as generalized additive models (GAMs) and smoothing spline analysis of variance (SSANOVA) models, extend the generalized linear model (GLM) by allowing for unknown functional relationships between an exponential family response variable and a collection of predictor variables. The unknown functional relationships are typically estimated using penalized likelihood estimation, which adds a roughness penalty to the (negative) log-likelihood function. In this article, I propose a spectral parameterization of a smoothing spline, which allows for an efficient application of Elastic Net regression to smooth and select eigenvectors of a kernel matrix. The classic (ridge regression) solution for a smoothing spline is a special case of the proposed kernel eigenvector smoothing and selection operator. Extensions for tensor product smoothers are developed for both the GAM and SSANOVA frameworks. Using simulated and real data examples, I demonstrate that the proposed approach offers practical and computational gains over typical approaches for fitting GAMs, SSANOVA models, and Elastic Net penalized GLMs. Supplementary materials for this article are available online.

Jeffreys-prior penalty, finiteness and shrinkage in binomial-response generalized linear models

Summary Penalization of the likelihood by Jeffreys’ invariant prior, or a positive power thereof, is shown to produce finite-valued maximum penalized likelihood estimates in a broad class of binomial generalized linear models. The class of models includes logistic regression, where the Jeffreys-prior penalty is known additionally to reduce the asymptotic bias of the maximum likelihood estimator, and models with other commonly used link functions, such as probit and log-log. Shrinkage towards equiprobability across observations, relative to the maximum likelihood estimator, is established theoretically and studied through illustrative examples. Some implications of finiteness and shrinkage for inference are discussed, particularly when inference is based on Wald-type procedures. A widely applicable procedure is developed for computation of maximum penalized likelihood estimates, by using repeated maximum likelihood fits with iteratively adjusted binomial responses and totals. These theoretical results and methods underpin the increasingly widespread use of reduced-bias and similarly penalized binomial regression models in many applied fields.

On hazard-based penalized likelihood estimation of accelerated failure time model with partly interval censoring

In survival analysis, the semiparametric accelerated failure time model is an important alternative to the widely used Cox proportional hazard model. The existing methods for accelerated failure time models include least-squares, log rank-based estimating equations and approximations to the nonparametric error distribution. In this paper, we propose another fitting method for the accelerated failure time model, formulated from the hazard function of the exponential error term. Our method can handle partly interval-censored data which contains event time, as well as left, right and interval censoring time. We adopt the maximum penalized likelihood method to estimate all the parameters in the model, including the nonparametric component. The penalty function is used to regularize the nonparametric component of the accelerated failure time model. Asymptotic properties of the penalized likelihood estimate are developed. A simulation study is conducted to investigate the performance of the proposed method and an application of this method to an AIDS study is presented as an example.

Penalized profile quasi-maximum likelihood method of partially linear spatial autoregressive model

In this paper, we develop a class of penalized likelihood method to identify important explanatory variables in parametric component of partially linear spatial autoregressive model. Compared to existing estimation methods, the proposed method can simultaneously select the significant explanatory variables and estimate the nonzero parameters in the parametric component of partially linear spatial autoregressive model. Under appropriate conditions, we establish the consistency, sparsity and asymptotic normality properties of the resulting penalized likelihood estimator. Especially, with proper choice of the penalty function and the regularization parameter, the estimator of the nonzero parameter vector is shown to enjoy the oracle property, in the sense that it is asymptotically normal with the same mean vector and covariance matrix as those it would have if the zero parameters were known in advance. Furthermore, we propose a computationally feasible algorithm to obtain the penalized likelihood estimator. The finite sample performance of the proposed variable selection method is evaluated through extensive simulation studies and illustrated with a real data set.

The exponentiated power exponential semiparametric regression model

We propose a new semiparametric regression model with exponentiated power exponential errors using the B-spline basis for nonlinear effects. We adopt the framework of the generalized additive models for location, scale, and shape to fit this regression model. We obtain the maximum penalized likelihood estimates of the model parameters by considering nonlinear effects. Some global-influence measurements and quantile residuals are also investigated. Various Monte Carlo simulations are performed for inference purposes under different parameter settings, systematic components and sample sizes. The proposals are illustrated by two applications to real data.

Effects of Socialization on Adolescent Sexual Behaviors in Lao PDR

Adolescent risky sexual behaviors are closely associated with adverse health outcomes, and losing future educational and economic opportunities. Communication about sex-related topics with various people influences adolescent sexual behavior through sexual socialization. This study aimed to examine the effects of the sex-related information on sexual behavior by considering three important aspects which are multiple sources, messages, and frequency of communication. Data on this study were from the cross-sectional survey on integrated health of 493 adolescents aged 15–19 from four secondary schools in the Capital of Lao PDR. We used Firth’s penalized likelihood estimation to test hypotheses. The results showed that several messages and sources of sexual information have significant effects on adolescent sexual engagement. Among the six different contents of sex, receiving messages about contraception from media and doctor decreased the probability of having sex. Sibling’s information about pregnancy was significantly associated with having sex. Sexual messages from mother increased the likelihood of having sexual activity. Regardless of sexual contents, partner’s information was closely related to sexual engagement. However, none of the sexual messages from teacher had an impact on adolescents’ sexual behavior. This study provides practical implications to parents, teachers, and policymakers in Lao PDR as well as novel insights to research on sexual behaviors of adolescent.

Empirical likelihood in varying-coefficient quantile regression with missing observations

In this paper, we focus on the partially linear varying-coefficient quantile regression model with observations missing at random (MAR), which include the responses or the responses and covariates MAR. Based on the local linear estimation of the varying-coefficient function in the model, we construct empirical log-likelihood ratio functions for unknown parameter in the linear part of the model, which are proved to be asymptotically weighted chi-squared distributions, further the adjusted empirical log-likelihood ratio functions are verified to converge to standard chi-squared distribution. The asymptotic normality of maximum empirical likelihood estimator for the parameter is also established. In order to do variable selection, we consider penalized empirical likelihood by using smoothly clipped absolute deviationv (SCAD) penalty, and the oracle property of the penalized likelihood estimator of the parameter is proved. Furthermore, Monte Carlo simulation and a real data analysis are undertaken to test the performance of the proposed methods.

Almost sure uniqueness of a global minimum without convexity

This paper establishes the argmin of a random objective function to be unique almost surely. This paper first formulates a general result that proves almost sure uniqueness without convexity of the objective function. The general result is then applied to a variety of applications in statistics. Four applications are discussed, including uniqueness of M-estimators, both classical likelihood and penalized likelihood estimators, and two applications of the argmin theorem, threshold regression and weak identification.