Oracle Property Research Articles

Machine learning model based on non-convex penalized huberized-SVM

The support vector machine (SVM) is a classical machine learning method. Both the hinge loss and least absolute shrinkage and selection operator (LASSO) penalty are usually used in traditional SVMs. However, the hinge loss is not differentiable, and the LASSO penalty does not have the Oracle property. In this paper, the huberized loss is combined with non-convex penalties to obtain a model that has the advantages of both the computational simplicity and the Oracle property, contributing to higher accuracy than traditional SVMs. It is experimentally demonstrated that the two non-convex huberized-SVM methods, smoothly clipped absolute deviation huberized-SVM (SCAD-HSVM) and minimax concave penalty huberized-SVM (MCP-HSVM), outperform the traditional SVM method in terms of the prediction accuracy and classifier performance. They are also superior in terms of variable selection, especially when there is a high linear correlation between the variables. When they are applied to the prediction of listed companies, the variables that can affect and predict financial distress are accurately filtered out. Among all the indicators, the indicators per share have the greatest influence while those of solvency have the weakest influence. Listed companies can assess the financial situation with the indicators screened by our algorithm and make an early warning of their possible financial distress in advance with higher precision.

A flexible stochastic production frontier model with panel data

SummaryWe propose a flexible stochastic production frontier model with fixed effects for the panel data in which the semiparametric frontier is additive with bivariate interactions. To avoid potential misspecification and/or “wrong skew problem” due to distributional assumptions, we model the conditional mean of the inefficiency to depend on environmental variables and to be known up to a vector of parameters. We propose a difference‐based estimator for parameters characterizing the conditional mean of the inefficiency term, a profile series estimator, and a kernel‐based one‐step backfitting estimator for the frontier to facilitate inference. We establish their asymptotic properties and show that each component in the frontier estimated by the kernel‐based backfitting has the same asymptotic distribution as the one estimated with the true knowledge on the other components in the frontier (i.e., the oracle property). Through a Monte Carlo study, we demonstrate that the proposed estimators perform well in finite samples. Utilizing a panel of Chinese firm‐level data in 2000–2006, we apply our method to estimate the frontier and efficiency scores and conclude that export plays a significant role in reducing the efficiency of firms.

Convolution smoothing and non-convex regularization for support vector machine in high dimensions

The support vector machine (SVM) is a well-known statistical learning tool for binary classification. One serious drawback of SVM is that it can be adversely affected by redundant variables, and research has shown that variable selection is crucial and necessary for achieving good classification accuracy. Hence some SVM variable selection studies have been devoted, and they have an unified “empirical hinge loss plus sparse penalty” formulation. However, a noteworthy issue is the computational complexity of existing methods is high especially for large-scale problems, due to the non-smoothness of the hinge loss. To solve this issue, we first propose a convolution smoothing approach, which turns the non-smooth hinge loss into a smooth surrogate one, and they are asymptotically equivalent. Moreover, we construct computationally more efficient SVM variable selection procedure by implementing non-convex penalized convolution smooth hinge loss. In theory, we prove that the resulting variable selection possesses the oracle property when the number of predictors is diverging. Numerical experiments also confirm the good performance of the new method.

Probabilistic electricity price forecasting based on penalized temporal fusion transformer

AbstactIn the deregulated electricity market, it is increasingly important to accurately predict the fluctuating, nonlinear, and high‐frequent electricity price for market decision‐making. However, the uncertainties associated with electricity prices, such as non‐stationarity, nonlinearity, and high volatility, pose critical difficulties for electricity price forecasting (EPF). Unlike point forecasting, which provides only a single, deterministic estimate of future prices, probabilistic forecasting gives a more comprehensive and nuanced picture of future price dynamics, which can help market participants make better‐informed decisions when facing uncertainty. Therefore, in this paper, we propose a robust deep learning method for multi‐step probabilistic forecasting. First, we use the least absolute shrinkage and selection operator (LASSO) in the expert model to generate point forecasts. Second, we introduce the smoothly clipped absolute deviation regularization term, a nonconvex penalty with proven oracle properties in model selection, into temporal fusion transformers. Finally, we employ the proposed model to integrate point forecasts to give probabilistic forecasts. To evaluate the proposed forecasting model, real‐data experiments are conducted in the Nord Pool electricity market and the Polish Power Exchange market. Empirical results show that the proposed model has demonstrated superior probabilistic forecasting performances compared with other competitors and has proven its effectiveness in real‐world applications.

On the Efficiency of the newly Proposed Convex Olanrewaju-Olanrewaju Lo-oλγ(|θ|) Penalized Regression-Type Estimator via GLMs Technique.

In this article, we proposed a novel convex penalized regression-type estimator, termed Olanrewaju-Olanrewaju penalized regression-type estimator, denoted by Lo-oλγ(|θ|) for ultra and high-dimensional data consideration whenever the number of covariates (p) is strictly greater than the sample size(n⁡). The newly proposed penalized estimator was formulated via the unique combination of arctangent and hyperbolic functions. It was noted that the newly proposed estimator could efficiently work under the constraint p<n formulated for, and as well as for the conventional regression methods of n>p. The proposed convex penalized regression-type estimator was formulated via three symmetric distributional noises of Gaussian, Laplace, and Normal Inverse Gaussian (NIG), considered being the three certified symmetric distributions that belong to the exponential family of distributions. The regularity axioms and oracle properties of the newly proposed penalized estimator were worked-out, as well as the parameter estimation for the penalized regression-type estimator for the three symmetric noises via canonical expressible form of Generalized Linear Models (GLMs) and first-order Karush-Kuhn-Tucker (KKT) condition. The Lo-oλγ(|θ|) regression-type estimator was subjected to simulation studies of well-generated symmetric observations. The US Longley macroeconomic dataset with the constraint p<n was fitted to Lo-oλγ(|θ|) regression-type estimator under the umbrella of three symmetric distributional noises. In conclusion, Gaussian noise for Lo-oλγ(|θ|) penalized regression-type estimator under the umbrella of p<n for the simulation and real life analyzes was considered effectively efficient because of its most reduced Mean Square Error (mse) measured error-type in comparison to Gaussian, Laplace, and NIG noises for LASSO and Atan penalties; and as well for Laplace, and NIG noises for Lo-oλγ(|θ|).

Simultaneous variable selection and estimation in semiparametric regression of mixed panel count data.

Mixed panel count data represent a common complex data structure in longitudinal survey studies. A major challenge in analyzing such data is variable selection and estimation while efficiently incorporating both the panel count and panel binary data components. Analyses in the medical literature have often ignored the panel binary component and treated it as missing with the unknown panel counts, while obviously such a simplification does not effectively utilize the original data information. In this research, we put forward a penalized likelihood variable selection and estimation procedure under the proportional mean model. A computationally efficient EM algorithm is developed that ensures sparse estimation for variable selection, and the resulting estimator is shown to have the desirable oracle property. Simulation studies assessed and confirmed the good finite-sample properties of the proposed method, and the method is applied to analyze a motivating dataset from the Health and Retirement Study.

Subgroup detection in the heterogeneous partially linear additive Cox model

In the analysis of survival data, it is crucial to consider individual heterogeneities related to therapy, gender, and genetics as they can impact the validity of conclusions. The heterogeneous partially linear additive Cox model offers a versatile and useful approach for addressing these variations by incorporating both heterogeneous linear components and homogeneous additive components. We propose a subgroup detection method for this model under right-censoring using B-spline smoothing technique to approximate the additive functions. Without imposing any specific subgroup structure, we generate an objective function by combining a log partial likelihood function with a fusion penalty. To estimate parameters and detect subgroups simultaneously, we employ a majorized alternating direction method of multipliers (ADMM) algorithm. Furthermore, we establish oracle properties and model selection consistency for the proposed penalised estimator. Simulation studies and real data analysis on breast cancer demonstrate the performance of our method.

Integrated subgroup identification from multi-source data

Subgroup identification is crucial in dealing with the heterogeneous population and has wide applications in various areas, such as clinical trials and market segmentation. With the prevalence of multi-source data, there is a practical need to identify subgroups based on multi-source data. This paper proposes a working-independence pseudo-loglikelihood and integrates the parameters of each source into a pairwise fusion penalty for simultaneous parameter estimation and subgroup identification. To implement the proposed method, an alternating direction method of multipliers (ADMM) algorithm is derived. Furthermore, the weak oracle properties of parameter estimation are established, illustrating the latent subgroups can be consistently identified. Finally, numerical simulations and an analysis of a randomized trial on reduced nicotine standards for cigarettes are conducted to evaluate the performance of the proposed method.

Fixed and random effect selections in generalized linear mixed models.

Generalized linear mixed models are commonly used to describe relationships between correlated responses and covariates in medical research. In this paper, we propose a simple and easily implementable regularized estimation approach to select both fixed and random effects in generalized linear mixed model. Specifically, we propose to construct and optimize the objective functions using the confidence distributions of model parameters, as opposed to using the observed data likelihood functions, to perform effect selections. Two estimation methods are developed. The first one is to use the joint confidence distribution of model parameters to perform simultaneous fixed and random effect selections. The second method is to use the marginal confidence distributions of model parameters to perform the selections of fixed and random effects separately. With a proper choice of regularization parameters in the adaptive LASSO framework, we show the consistency and oracle properties of the proposed regularized estimators. Simulation studies have been conducted to assess the performance of the proposed estimators and demonstrate computational efficiency. Our method has also been applied to two longitudinal cancer studies to identify demographic and clinical factors associated with patient health outcomes after cancer therapies.

On the Oracle Properties of Bayesian Random Forest for Sparse High-Dimensional Gaussian Regression

Random forest (RF) is a widely used data prediction and variable selection technique. However, the variable selection aspect of RF can become unreliable when there are more irrelevant variables than relevant ones. In response, we introduced the Bayesian random forest (BRF) method, specifically designed for high-dimensional datasets with a sparse covariate structure. Our research demonstrates that BRF possesses the oracle property, which means it achieves strong selection consistency without compromising the efficiency or bias.

Multi-block alternating direction method of multipliers for ultrahigh dimensional quantile fused regression

In this paper, we consider a quantile fused LASSO regression model that combines quantile regression loss with the fused LASSO penalty. Intuitively, this model offers robustness to outliers, thanks to the quantile regression, while also effectively recovering sparse and block coefficients through the fused LASSO penalty. To adapt our proposed method for ultrahigh dimensional datasets, we introduce an iterative algorithm based on the multi-block alternating direction method of multipliers (ADMM). Moreover, we demonstrate the global convergence of the algorithm and derive comparable convergence rates. Importantly, our ADMM algorithm can be easily applied to solve various existing fused LASSO models. In terms of theoretical analysis, we establish that the quantile fused LASSO can achieve near oracle properties with a practical penalty parameter, and additionally, it possesses a sure screening property under a wide class of error distributions. The numerical experimental results support our claims, showing that the quantile fused LASSO outperforms existing fused regression models in robustness, particularly under heavy-tailed distributions.

Subgroup analysis for the functional linear model

Classical functional linear regression models the relationship between a scalar response and a functional covariate, where the coefficient function is assumed to be identical for all subjects. In this paper, the classical model is extended to allow heterogeneous coefficient functions across different subgroups of subjects. The greatest challenge is that the subgroup structure is usually unknown to us. To this end, we develop a penalization-based approach which innovatively applies the penalized fusion technique to simultaneously determine the number and structure of subgroups and coefficient functions within each subgroup. An effective computational algorithm is derived. We also establish the oracle properties and estimation consistency. Extensive numerical simulations demonstrate its superiority compared to several competing methods. The analysis of an air quality dataset leads to interesting findings and improved predictions.

A Group MCP Approach for Structure Identification in Non-Parametric Accelerated Failure Time Additive Regression Model

In biomedical research, identifying genes associated with diseases is of paramount importance. However, only a small fraction of genes are related to specific diseases among the multitude of genes. Therefore, gene selection and estimation are necessary, and the accelerated failure time model is often used to address such issues. Hence, this article presents a method for structural identification and parameter estimation based on a non-parametric additive accelerated failure time model for censored data. Regularized estimation and variable selection are achieved using the Group MCP penalty method. The non-parametric component of the model is approximated using B-spline basis functions, and a group coordinate descent algorithm is employed for model solving. This approach effectively identifies both linear and nonlinear factors in the model. The Group MCP penalty estimation exhibits consistency and oracle properties under regularization conditions, meaning that the selected variable set tends to have a probability of approaching 1 and asymptotically includes the actual predictive factors. Numerical simulations and a lung cancer data analysis demonstrate that the Group MCP method outperforms the Group Lasso method in terms of predictive performance, with the proposed algorithm showing faster convergence rates.

Variable Selection for Length-Biased and Interval-Censored Failure Time Data

Length-biased failure time data occur often in various biomedical fields, including clinical trials, epidemiological cohort studies and genome-wide association studies, and their analyses have been attracting a surge of interest. In practical applications, because one may collect a large number of candidate covariates for the failure event of interest, variable selection becomes a useful tool to identify the important risk factors and enhance the estimation accuracy. In this paper, we consider Cox’s proportional hazards model and develop a penalized variable selection technique with various popular penalty functions for length-biased data, in which the failure event of interest suffers from interval censoring. Specifically, a computationally stable and reliable penalized expectation-maximization algorithm via two-stage data augmentation is developed to overcome the challenge in maximizing the intractable penalized likelihood. We establish the oracle property of the proposed method and present some simulation results, suggesting that the proposed method outperforms the traditional variable selection method based on the conditional likelihood. The proposed method is then applied to a set of real data arising from the Prostate, Lung, Colorectal and Ovarian cancer screening trial. The analysis results show that African Americans and having immediate family members with prostate cancer significantly increase the risk of developing prostate cancer, while having diabetes exhibited a significantly lower risk of developing prostate cancer.

General spatial model meets adaptive shrinkage generalized moment estimation: Simultaneous model and moment selection

Spatial data are widely used in various scenarios of life and are highly valued, and their analysis and research have achieved remarkable results. Spatial data have spatial effects and do not satisfy the assumption of independence; thus, the traditional econometric analysis methods cannot be directly used in spatial models, and the spatial autocorrelation and spatial heterogeneity of spatial data make the research more complicated and difficult. Generalized moment estimation(GMM) is a powerful tool for statistical modeling and inference of spatial data. Considering the case where there is a set of correctly specified moment conditions and another set of possibly misspecified moment conditions for spatial data, this paper proposes a GMM shrinkage method to estimate the unknown parameters for spatial autoregressive model with spatial autoregressive disturbances. The proposed GMM estimators are shown to enjoy oracle properties; i.e., it selects the valid moment conditions consistently from the candidate set and includes them into estimation automatically. The resulting estimator is asymptotically as efficient as the GMM estimator based on all valid moment conditions. Monte Carlo studies show that the method works well in terms of valid moment selection and the finite sample properties of its estimators.

Selective machine learning of doubly robust functionals

Summary While model selection is a well-studied topic in parametric and nonparametric regression or density estimation, selection of possibly high-dimensional nuisance parameters in semiparametric problems is far less developed. In this paper, we propose a selective machine learning framework for making inferences about a finite-dimensional functional defined on a semiparametric model, when the latter admits a doubly robust estimating function and several candidate machine learning algorithms are available for estimating the nuisance parameters. We introduce a new selection criterion aimed at bias reduction in estimating the functional of interest based on a novel definition of pseudo risk inspired by the double robustness property. Intuitively, the proposed criterion selects a pair of learners with the smallest pseudo risk, so that the estimated functional is least sensitive to perturbations of a nuisance parameter. We establish an oracle property for a multi-fold cross-validation version of the new selection criterion that states that our empirical criterion performs nearly as well as an oracle with a priori knowledge of the pseudo risk for each pair of candidate learners. Finally, we apply the approach to model selection of a semiparametric estimator of average treatment effect given an ensemble of candidate machine learners to account for confounding in an observational study that we illustrate in simulations and a data application.

Automatic selection by penalized asymmetric Lq -norm in a high-dimensional model with grouped variables

The paper focuses on the automatic selection of the grouped explanatory variables in a high-dimensional model, when the model blue error is asymmetric. After introducing the model and notations, we define the adaptive group LASSO expectile estimator for which we prove the oracle properties: the sparsity and the asymptotic normality. Afterwards, the results are generalized by considering the asymmetric -norm loss function. The theoretical results are obtained in several cases with respect to the number of variable groups. This number can be fixed or dependent on the sample size n, with the possibility that it is of the same order as n. Note that these new estimators allow us to consider weaker assumptions on the data and on the model errors than the usual ones. Simulation study demonstrates the competitive performance of the proposed penalized expectile regression, especially when the samples size is close to the number of explanatory variables and model errors are asymmetrical. An application on air pollution data is considered.

Penetrating sporadic return predictability

Return predictability has been one of the central research questions in finance for many decades. This paper proposes a predictive regression with multiple structural changes to capture the sporadic predictive ability of potential predictors for the return series. An adaptive group Lasso procedure, augmented with a forward regression for break screening, is adopted to efficiently and consistently identify the structural breaks in the predictive regression, with predictors exhibiting low signal strength and heterogeneous degrees of persistence. To enhance the prediction accuracy, adaptive Lasso is further used to eliminate the irrelevant predictors and is shown to achieve the oracle property. Simulation studies demonstrate the effectiveness of the proposed methods in break detection and predictor selection, and further show that ignoring structural breaks could abate predictability. The application to predicting U.S. equity premium illustrates the practical merits of our methodology in revealing return predictability that changes over time.

High-dimensional variable selection accounting for heterogeneity in regression coefficients across multiple data sources.

When analyzing data combined from multiple sources (e.g., hospitals, studies), the heterogeneity across different sources must be accounted for. In this paper, we consider high-dimensional linear regression models for integrative data analysis. We propose a new adaptive clustering penalty (ACP) method to simultaneously select variables and cluster source-specific regression coefficients with sub-homogeneity. We show that the estimator based on the ACP method enjoys a strong oracle property under certain regularity conditions. We also develop an efficient algorithm based on the alternating direction method of multipliers (ADMM) for parameter estimation. We conduct simulation studies to compare the performance of the proposed method to three existing methods (a fused LASSO with adjacent fusion, a pairwise fused LASSO, and a multi-directional shrinkage penalty method). Finally, we apply the proposed method to the multi-center Childhood Adenotonsillectomy Trial to identify sub-homogeneity in the treatment effects across different study sites.

Partially Functional Linear Models with Linear Process Errors

In this paper, we focus on the partial functional linear model with linear process errors deduced by not necessarily independent random variables. Based on Mercer’s theorem and Karhunen–Loève expansion, we give the estimators of the slope parameter and coefficient function in the model, establish the asymptotic normality of the estimator for the parameter and discuss the weak convergence with rates of the proposed estimators. Meanwhile, the penalized estimator of the parameter is defined by the SCAD penalty and its oracle property is investigated. Finite sample behavior of the proposed estimators is also analysed via simulations.