Penalized Objective Function Research Articles

Mid-quantile regression for discrete panel data

We propose novel quantile regression methods when the response is discrete and the data come from a longitudinal design. The approach is based on conditional mid-quantiles, which have good theoretical properties even in the presence of ties. Optimization of a ridge-type penalized objective function accommodates for the data dependence. We investigate the performance and pertinence of our methods in a simulation study and an original application to macroprudential policies use in more than one hundred countries over a period of seventeen years.

Variables selection using [formula omitted] penalty

The determination of a tuning parameter by the generalized information criterion (GIC) is considered an important issue in variable selection. It is shown that the GIC and the L0 penalized objective functions are equivalent, leading to a new L0 penalized maximum likelihood method for high-dimensional generalized linear models in this article. Based on the technique of the well-known discrete optimization problem in theoretical computer science, a two-step algorithm for local solutions is proposed. The first step optimizes the L0 penalized objective function under a given model size, where only a maximum likelihood algorithm is needed. The second step optimizes the L0 penalized objective function under a candidate set of model sizes, where only the GIC is needed. As the tuning parameter can be fixed, the selection of the tuning parameter can be ignored in the proposed method. The theoretical study shows that the algorithm is polynomial and any resulting local solution is consistent. Thus, it is not necessary to use the global solution in practice. The numerical studies show that the proposed method outperforms its competitors in general.

Exploring the Impact of Big Bang-Big Crunch Algorithm Parameters on Welded Beam Design Problem Resolution

A Metaheuristic Optimization is a group of algorithms that are widely studied and employed in the scientific literature. Typically, metaheuristics algorithms utilize stochastic operators that make each iteration unique, and they frequently contain controlling parameters that have an impact on the convergence process since their impacts are mostly neglected in most optimization literature, making it difficult to draw conclusions. This paper introduced the Big Bang-Big Crunch (BB-BC) metaheuristic algorithm to evaluate the performance of a metaheuristic algorithm in relation to its control parameter. It also demonstrates the effects of varying the values of BB-BC in solving. The "Welded Beam Design problem" is a well-known engineering optimization problem that is classified as a Single-Objective Constrained Optimization issue. Multiple starting parameter values for the BB-BC are evaluated as part of the experimental findings. This is done in an attempt to find the algorithm's optimal starting settings. The lowest, maximum, and mean values of the penalized objective functions are then computed. Finally, the BB-BC results are compared with various metaheuristics algorithms.

Data Integration in High Dimension With Multiple Quantiles

This article deals with the analysis of high dimensional data that come from multiple sources (experiments) and thus have different possibly correlated responses, but share the same set of predictors. The measurements of the predictors may be different across experiments. We introduce a new regression approach with multiple quantiles to select those predictors that affect any of the responses at any quantile level and estimate the nonzero parameters. Our estimator is a minimizer of a penalized objective function, which aggregates the data from the different experiments. We establish model selection consistency and asymptotic normality of the estimator. In addition we present an information criterion, which can also be used for consistent model selection. Simulations and two data applications illustrate the advantages of our method, which takes the group structure induced by the predictors across experiments and quantile levels into account.

Variable Selection with Multiply-Imputed Datasets: Choosing Between Stacked and Grouped Methods

Penalized regression methods are used in many biomedical applications for variable selection and simultaneous coefficient estimation. However, missing data complicates the implementation of these methods, particularly when missingness is handled using multiple imputation. Applying a variable selection algorithm on each imputed dataset will likely lead to different sets of selected predictors. This article considers a general class of penalized objective functions which, by construction, force selection of the same variables across imputed datasets. By pooling objective functions across imputations, optimization is then performed jointly over all imputed datasets rather than separately for each dataset. We consider two objective function formulations that exist in the literature, which we will refer to as “stacked” and “grouped” objective functions. Building on existing work, we (i) derive and implement efficient cyclic coordinate descent and majorization-minimization optimization algorithms for continuous and binary outcome data, (ii) incorporate adaptive shrinkage penalties, (iii) compare these methods through simulation, and (iv) develop an R package miselect. Simulations demonstrate that the “stacked” approaches are more computationally efficient and have better estimation and selection properties. We apply these methods to data from the University of Michigan ALS Patients Biorepository aiming to identify the association between environmental pollutants and ALS risk. Supplementary materials for this article are available online.

A Block Coordinate Descent-Based Projected Gradient Algorithm for Orthogonal Non-Negative Matrix Factorization

This article uses the projected gradient method (PG) for a non-negative matrix factorization problem (NMF), where one or both matrix factors must have orthonormal columns or rows. We penalize the orthonormality constraints and apply the PG method via a block coordinate descent approach. This means that at a certain time one matrix factor is fixed and the other is updated by moving along the steepest descent direction computed from the penalized objective function and projecting onto the space of non-negative matrices. Our method is tested on two sets of synthetic data for various values of penalty parameters. The performance is compared to the well-known multiplicative update (MU) method from Ding (2006), and with a modified global convergent variant of the MU algorithm recently proposed by Mirzal (2014). We provide extensive numerical results coupled with appropriate visualizations, which demonstrate that our method is very competitive and usually outperforms the other two methods.

Variable Selection in Semiparametric Mixed Effect Model for High-Dimension Longitudinal Data

This paper considers the problem of simultaneous variable selection and estimation in a semiparametric mixed-effects model for longitudinal data with normal errors. We approximate the nonparametric function by regression spline and simultaneously estimate and select the variables under the optimization of the penalized objective function. Under some regularity conditions, the asymptotic behaviour of the resulting estimators is established in a high-dimensional framework where the number of parametric covariates increases as the sample size increases. For practical implementation, we use an EM algorithm to selects the significant variables and estimates the nonzero coefficient functions. Simulation studies are carried out to assess the performance of our proposed method, and a real data set is analyzed to illustrate the proposed procedure.

Integrative factorization of bidimensionally linked matrices.

Advances in molecular "omics" technologies have motivated new methodologies for the integration of multiple sources of high-content biomedical data. However, most statistical methods for integrating multiple data matrices only consider data shared vertically (one cohort on multiple platforms) or horizontally (different cohorts on a single platform). This is limiting for data that take the form of bidimensionally linked matrices (eg, multiple cohorts measured on multiple platforms), which are increasingly common in large-scale biomedical studies. In this paper, we propose bidimensional integrative factorization (BIDIFAC) for integrative dimension reduction and signal approximation of bidimensionally linked data matrices. Our method factorizes data into (a) globally shared, (b) row-shared, (c) column-shared, and (d) single-matrix structural components, facilitating the investigation of shared and unique patterns of variability. For estimation, we use a penalized objective function that extends the nuclear norm penalization for a single matrix. As an alternative to the complicated rank selection problem, we use results from the random matrix theory to choose tuning parameters. We apply our method to integrate two genomics platforms (messenger RNA and microRNA expression) across two sample cohorts (tumor samples and normal tissue samples) using the breast cancer data from the Cancer Genome Atlas. We provide R code for fitting BIDIFAC, imputing missing values, and generating simulated data.

Robust Generalized Fuzzy Systems Training From High-Dimensional Time-Series Data Using Local Structure Preserving PLS

Establishing fuzzy models from time-series data with predictive capabilities for numerical targets typically requires dimension reduction techniques to overcome the severe curse of dimensionality effects. Linear projection methods are promising candidates in this context as they—unlike nonlinear dimension reduction techniques—preserve interpretability of the resulting models. However, linear projections do not reveal the inherent (nonlinear, local) cluster structure of the data and are thus not ideally suited for the identification of fuzzy rule bases. To overcome this limitation, in this article, we present a new fuzzy modeling approach that combines generalized fuzzy systems modeling with a local structure preserving variant of partial least squares (PLS). In contrast to ordinary PLS, our approach maps a weighted (adjacency) graph on the directions associated with high covariance with the response in order to emphasize local data structures when constructing the latent variable (LV) space. This operates into the direction from which the (training of the) fuzzy model benefits, as therein local regions are represented by submodels in form of generalized Takagi-Sugeno (TS) fuzzy rules. The local structure preserving LV space is obtained by solving a new penalized objective function, which assures global optimality of the solutions by virtue of the specific properties of the Laplacian matrix. Local regions are characterized in two ways: through nearest neighbor points (assuming fixed local region sizes) and through density regions identified by clustering (achieving variable local region sizes). To establish a robust time-series-based forecast model, the training of a generalized TS fuzzy model is conducted in the LV space with reduced dimensionality. It is realized by an iterative robust version of Gen-Smart-EFS , allowing multiple passes over the complete datasets until convergence of the antecedent space. Consequent parameters are estimated by a fuzzily weighted elastic net approach, embedding a convex combination of ridge regression and Lasso to achieve robust solutions also in case of ill-posed problems and meeting the (more stable and interpretable) local learning spirit. The new approach is termed as LS-PLS-Fuzzy , short for local structure preserving PLS fuzzy regression . It was successfully evaluated on three real-world application scenarios including a total of 11 time-series-based prediction modeling problems with different proportions between the number of training samples and original input dimensionality. Our results show significantly improved model accuracy compared to related State-of-the-Art (SoA) modeling approaches, an alternative dimension reduction technique, and when using conventional PLS and/or nongeneralized fuzzy rules.

Regularized quantile regression for ultrahigh-dimensional data with nonignorable missing responses

The paper concerns the regularized quantile regression for ultrahigh-dimensional data with responses missing not at random. The propensity score is specified by the semiparametric exponential tilting model. We use the Pearson Chi-square type test statistic for identification of the important features in the sparse propensity score model, and employ the adjusted empirical likelihood method for estimation of the parameters in the reduced model. With the estimated propensity score model, we suggest an inverse probability weighted and penalized objective function for regularized estimation using the nonconvex SCAD penalty and MCP functions. Assuming the propensity score model is of low dimension, we establish the oracle properties of the proposed regularized estimators. The new method has several desirable advantages. First, it is robust to heavy-tailed errors or potential outliers in the responses. Second, it can accommodate nonignorable nonresponse data. Third, it can deal with ultrahigh-dimensional data with heterogeneity. Simulation study and real data analysis are given to examine the finite sample performance of the proposed approaches.

Deep multiple kernel least squares support vector regression using PSO

In this paper, we propose a deep multiple kernel least squares support vector regression (DMK-LSSVR) using particle swarm optimization (PSO). Unlike multilayer neural networks (MNN), each LSSVR in the DMK-LSSVR is trained to minimize the penalized objective function. Therefore, the learning of DMK-LSSVR is completely different from that of MNN that minimizes only the final objective function. In DMK-LSSVR the grid search using GCV function is used to find optimal values of hyperparameters of each LSSVR, which has a disadvantage that takes a lot of computational time. And the back propagation algorithm is used for the optimal values of weights and biases, which has a weakness to results in local minima. In DMK-LSSVR which utilizes PSO (DMK-LSSVR-PSO), we find the optimal values of hyperparameters of LSSVRs, weights and biases using PSO in one process. Using PSO, the only needed on hyperparameters are the lower and upper bound, and estimating weights and biases results in global minimizers. Numerical studies show that DMK-LSSVR-PSO has advantages over DMK-LSSVR and other machine learning models that use back propagation algorithm and grid search for regression problems.

Non-linear threshold algorithm based solution for the redundancy allocation problem considering multiple redundancy strategies

PurposeThe purpose of this paper is to introduce an efficient algorithm based on a non-linear accepting threshold to solve the redundancy allocation problem (RAP) considering multiple redundancy strategies. In addition to the components reliability, multiple redundancy strategies are simultaneously considered to vary the reliability of the system. The goal is to determine the optimal selection of elements, redundancy levels and redundancy strategy, which maximizes the system reliability under various system-level constraints.Design/methodology/approachThe mixed RAP considering the use of active and standby components at the subsystem level belongs to the class of NP-hard problems involving selection of elements and redundancy levels, to maximize a specific system performance under a given set of physical and budget constraints. Generally, the authors recourse to meta-heuristic algorithms to solve this type of optimization problem in a reasonable computational time, especially for large-size problems. A non-linear threshold accepting algorithm (NTAA) is developed to solve the tackled optimization problem. Numerical results for test problems from previous research are reported and analyzed to assess the efficiency of the proposed algorithm.FindingsThe comparison with the best solutions obtained in previous studies, namely: genetic algorithm, simulated annealing, memetic algorithm and the particle swarm optimization for 33 different instances of the problem, demonstrated the superiority of the proposed algorithm in finding for all considered instances, a high-quality solution in a minimum computational time.Research limitations/implicationsConsidering multiple redundancy strategies helps to achieve higher reliability levels but increases the complexity of the obtained solution leading to infeasible systems in term of physical design. Technological constraints must be integrated into the model to provide a more comprehensive and realistic approach.Practical implicationsDesigning high performant systems which meet customer requirements, under different economic and functional constraints is the main challenge faced by the manufacturers. The proposed algorithm aims to provide a superior solution of the reliability optimization problem by considering the possibility to adopt multiple redundancy strategies at the subsystem level in a minimum computational time.Originality/valueA NTAA is expanded to the RAP considering multiple redundancy strategies at the subsystem level subject to weight and cost constraints. A procedure based on a penalized objective function is developed to encourage the algorithm to explore toward the feasible solutions area. By outperforming well-known solving technique, the NTAA provides a powerful tool to reliability designers of complex systems where different varieties of redundancies can be considered to achieve high-reliability systems.

Change-point detection in multinomial data with a large number of categories

We consider a sequence of multinomial data for which the probabilities associated with the categories are subject to abrupt changes of unknown magnitudes at unknown locations. When the number of categories is comparable to or even larger than the number of subjects allocated to these categories, conventional methods such as the classical Pearson’s chi-squared test and the deviance test may not work well. Motivated by high-dimensional homogeneity tests, we propose a novel change-point detection procedure that allows the number of categories to tend to infinity. The null distribution of our test statistic is asymptotically normal and the test performs well with finite samples. The number of change-points is determined by minimizing a penalized objective function based on segmentation, and the locations of the change-points are estimated by minimizing the objective function with the dynamic programming algorithm. Under some mild conditions, the consistency of the estimators of multiple change-points is established. Simulation studies show that the proposed method performs satisfactorily for identifying change-points in terms of power and estimation accuracy, and it is illustrated with an analysis of a real data set.

Fusing multimodal microscopy data for improved cell boundary estimation and fluorophore localization of Pseudomonas aeruginosa.

With advances in experimental technologies, the use of biological imaging has grown rapidly and there is need for procedures to combine data arising from different modalities. We propose a procedure to combine yellow fluorescence protein excitation and differential interference contrast microscopy time lapse videos to better estimate the cellular boundary of Pseudomonas aeruginosa (P. aeruginosa) and localization of it's type VI secretion system (T6SS). By approximating the shape by an ellipse, we construct a penalized objective function which accounts for both sources; the minimum of which provides an elliptical approximation to their cellular boundaries. Our approach suggests improved localization of the T6SS on the estimated cell boundary of P. aeruginosa constructed using both sources of data compared to using each in isolation.

Optimal Stratification and Allocation for the June Agricultural Survey

Abstract A computational approach to optimal multivariate designs with respect to stratification and allocation is investigated under the assumptions of fixed total allocation, known number of strata, and the availability of administrative data correlated with thevariables of interest under coefficient-of-variation constraints. This approach uses a penalized objective function that is optimized by simulated annealing through exchanging sampling units and sample allocations among strata. Computational speed is improved through the use of a computationally efficient machine learning method such as K-means to create an initial stratification close to the optimal stratification. The numeric stability of the algorithm has been investigated and parallel processing has been employed where appropriate. Results are presented for both simulated data and USDA’s June Agricultural Survey. An R package has also been made available for evaluation.

Variable Selection in Causal Inference using a Simultaneous Penalization Method

Abstract In the causal adjustment setting, variable selection techniques based only on the outcome or only on the treatment allocation model can result in the omission of confounders and hence may lead to bias, or the inclusion of spurious variables and hence cause variance inflation, in estimation of the treatment effect. We propose a variable selection method using a penalized objective function that is based on both the outcome and treatment assignment models. The proposed method facilitates confounder selection in high-dimensional settings. We show that under some mild conditions our method attains the oracle property. The selected variables are used to form a doubly robust regression estimator of the treatment effect. Using the proposed method we analyze a set of data on economic growth and study the effect of life expectancy as a measure of population health on the average growth rate of gross domestic product per capita.

Scalable approximate Bayesian inference for outlier detection under informative sampling

Government surveys of business establishments receive a large volume of submissions where a small subset contain errors. Analysts need a fast-computing algorithm to ag this subset due to a short time window between collection and reporting. We offer a computationally-scalable optimization method based on non-parametric mixtures of hierarchical Dirichlet processes that allows discovery of multiple industry-indexed local partitions linked to a set of global cluster centers. Outliers are nominated as those clusters containing few observations. We extend an existing approach with a new merge step that reduces sensitivity to hyperparameter settings. Survey data are typically acquired under an informative sampling design where the probability of inclusion depends on the surveyed response such that the distribution for the observed sample is different from the population. We extend the derivation of a penalized objective function to use a pseudo-posterior that incorporates sampling weights that undo the informative design. We provide a simulation study to demonstrate that our approach produces unbiased estimation for the outlying cluster under informative sampling. The method is applied for outlier nomination for the Current Employment Statistics survey conducted by the Bureau of Labor Statistics.

Estimating and Identifying Unspecified Correlation Structure for Longitudinal Data

Identifying correlation structure is important to achieving estimation efficiency in analyzing longitudinal data, and is also crucial for drawing valid statistical inference for large-size clustered data. In this article, we propose a nonparametric method to estimate the correlation structure, which is applicable for discrete longitudinal data. We use eigenvector-based basis matrices to approximate the inverse of the empirical correlation matrix and determine the number of basis matrices via model selection. A penalized objective function based on the difference between the empirical and model approximation of the correlation matrices is adopted to select an informative structure for the correlation matrix. The eigenvector representation of the correlation estimation is capable of reducing the risk of model misspecification, and also provides useful information on the specific within-cluster correlation pattern of the data. We show that the proposed method possesses the oracle property and selects the true correlation structure consistently. The proposed method is illustrated through simulations and two data examples on air pollution and sonar signal studies .

Quadratic Approximation via the SCAD Penalty with a Diverging Number of Parameters

The high-dimensional data arises in diverse fields of sciences, engineering and humanities. Variable selection plays an important role in dealing with high dimensional statistical modelling. In this article, we study the variable selection of quadratic approximation via the smoothly clipped absolute deviation (SCAD) penalty with a diverging number of parameters. We provide a unified method to select variables and estimate parameters for various of high dimensional models. Under appropriate conditions and with a proper regularization parameter, we show that the estimator has consistency and sparsity, and the estimators of nonzero coefficients enjoy the asymptotic normality as they would have if the zero coefficients were known in advance. In addition, under some mild conditions, we can obtain the global solution of the penalized objective function with the SCAD penalty. Numerical studies and a real data analysis are carried out to confirm the performance of the proposed method.

Support vector quantile regression ensemble with bagging

Support vector quantile regression (SVQR) is capable of providing more complete description of the linear and nonlinear relationships among random variables. To improve the estimation performance of SVQR we propose to use SVQR ensemble with bagging (bootstrap aggregating), in which SVQRs are trained independently using the training data sets sampled randomly via a bootstrap method. Then, they are aggregated to obtain the estimator of the quantile regression function using the penalized objective function composed of check functions. Experimental results are then presented, which illustrate the performance of SVQR ensemble with bagging.