Mixture Of Linear Regression Models Research Articles

Comparison of Some Methods for Estimating Mixture of Linear Regression Models with Application

A mixture model is used to model data that come from more than one component. In recent years, it became an effective tool in drawing inferences about the complex data that we might come across in real life. Moreover, it can represent a tremendous confirmatory tool in classification observations based on similarities amongst them. In this paper, several mixture regression-based methods were conducted under the assumption that the data come from a finite number of components. A comparison of these methods has been made according to their results in estimating component parameters. Also, observation membership has been inferred and assessed for these methods. The results showed that the flexible mixture model outperformed the others in most simulation scenarios according to the integrated mean square error and integrated classification error

Weighted likelihood latent class linear regression

A weighted likelihood approach for robust fitting of a finite mixture of linear regression models is proposed. An EM type algorithm and its variant based on the classification likelihood have been developed. The proposed algorithm is characterized by an M-step that is enhanced by the computation of weights aimed at downweighting outliers. The weights are based on the Pearson residuals stemming from the assumption of normality for the error distribution. Formal rules for robust clustering and outlier detection are also defined based on the fitted mixture model. The behavior of the proposed methodologies has been investigated by numerical studies and real data examples in terms of both fitting and classification accuracy and outlier detection.

Semiparametric mixture regression with unspecified error distributions

In fitting a mixture of linear regression models, normal assumption is traditionally used to model the error and then regression parameters are estimated by the maximum likelihood estimators (MLE). This procedure is not valid if the normal assumption is violated. To relax the normal assumption on the error distribution hence reduce the modeling bias, we propose semiparametric mixture of linear regression models with unspecified error distributions. We establish a more general identifiability result under weaker conditions than existing results, construct a class of new estimators, and establish their asymptotic properties. These asymptotic results also apply to many existing semiparametric mixture regression estimators whose asymptotic properties have remained unknown due to the inherent difficulties in obtaining them. Using simulation studies, we demonstrate the superiority of the proposed estimators over the MLE when the normal error assumption is violated and the comparability when the error is normal. Analysis of a newly collected Equine Infectious Anemia Virus data in 2017 is employed to illustrate the usefulness of the new estimator.

Seemingly unrelated clusterwise linear regression

Linear regression models based on finite Gaussian mixtures represent a flexible tool for the analysis of linear dependencies in multivariate data. They are suitable for dealing with correlated response variables when data come from a heterogeneous population composed of two or more sub-populations, each of which is characterised by a different linear regression model. Several types of finite mixtures of linear regression models have been specified by changing the assumptions on the parameters that differentiate the sub-populations and/or the vectors of regressors that affect the response variables. They are made more flexible in the class of models defined by mixtures of seemingly unrelated Gaussian linear regressions illustrated in this paper. With these models, the researcher is enabled to use a different vector of regressors for each dependent variable. The proposed class includes parsimonious models obtained by imposing suitable constraints on the variances and covariances of the response variables in the sub-populations. Details about the model identification and maximum likelihood estimation are given. The usefulness of these models is shown through the analysis of a real dataset. Regularity conditions for the model class are illustrated and a proof is provided that, when these conditions are met, the consistency of the maximum likelihood estimator under the examined models is ensured. In addition, the behaviour of this estimator in the presence of finite samples is numerically evaluated through the analysis of simulated datasets.

Modeling Vehicle Merging Position Selection Behaviors Based on a Finite Mixture of Linear Regression Models

Vehicle merging is a complex and tactical decision process. Merging position selection behavior has been largely ignored in microscopic traffic simulators. Driver heterogeneity has received substantial attention in recent years; however, few studies have focused on the heterogeneity in merging behaviors. To account for the heterogeneity among merging drivers during the merging process and to improve the accuracy of the merging model, a finite mixture of linear regression models was developed for describing the merging position selection model. BIC was used to determine the optimal number of classes, and Latent Gold 5.0 was used to estimate parameters. Based on the US101 data in the NGSIM project, which were provided by FHWA, a 3-class linear regression model was developed. The results demonstrate that the variables differ across the classes, and the sign of each variable may also differ among the classes; hence, the strategies that are used by drivers for merging position selection differ across the classes. Cooperative lane changing of the putative leading vehicle was found to have significant influence on the merging position selection behavior; thus, merging behavior is a two-dimensional behavior that may be influenced by both lateral and longitudinal factors. Compared with previous studies, the proposed model can naturally identify the heterogeneity among drivers and is much more accurate; therefore, the proposed model is a promising tool for microscopic traffic simulation and automatic driving systems or driver assistance systems.

Bayesian curve fitting and clustering with Dirichlet process mixture models for microarray data

In the field of molecular biology, it is often of interest to analyze microarray data for clustering genes based on similar profiles of gene expression to identify genes that are differentially expressed under multiple biological conditions. One of the notable characteristics of a gene expression profile is that it shows a cyclic curve over a course of time. To group sequences of similar molecular functions, we propose a Bayesian Dirichlet process mixture of linear regression models with a Fourier series for the regression coefficients, for each of which a spike and slab prior is assumed. A full Gibbs-sampling algorithm is developed for an efficient Markov chain Monte Carlo (MCMC) posterior computation. Due to the so-called “label-switching” problem and different numbers of clusters during the MCMC computation, a post-process approach of Fritsch and Ickstadt (2009) is additionally applied to MCMC samples for an optimal single clustering estimate by maximizing the posterior expected adjusted Rand index with the posterior probabilities of two observations being clustered together. The proposed method is illustrated with two simulated data and one real data of the physiological response of fibroblasts to serum of Iyer et al. (1999).

Statistical Inference and Applications of Mixture of Varying Coefficient Models

AbstractIn this paper, we consider a new mixture of varying coefficient models, in which each mixture component follows a varying coefficient model and the mixing proportions and dispersion parameters are also allowed to be unknown smooth functions. We systematically study the identifiability, estimation and inference for the new mixture model. The proposed new mixture model is rather general, encompassing many mixture models as its special cases such as mixtures of linear regression models, mixtures of generalized linear models, mixtures of partially linear models and mixtures of generalized additive models, some of which are new mixture models by themselves and have not been investigated before. The new mixture of varying coefficient model is shown to be identifiable under mild conditions. We develop a local likelihood procedure and a modified expectation–maximization algorithm for the estimation of the unknown non‐parametric functions. Asymptotic normality is established for the proposed estimator. A generalized likelihood ratio test is further developed for testing whether some of the unknown functions are constants. We derive the asymptotic distribution of the proposed generalized likelihood ratio test statistics and prove that the Wilks phenomenon holds. The proposed methodology is illustrated by Monte Carlo simulations and an analysis of a CO2‐GDP data set.

A globally convergent algorithm for lasso-penalized mixture of linear regression models

Variable selection is an old and pervasive problem in regression analysis. One solution is to impose a lasso penalty to shrink parameter estimates toward zero and perform continuous model selection. The lasso-penalized mixture of linear regressions model (L-MLR) is a class of regularization methods for the model selection problem in the fixed number of variables setting. A new algorithm is proposed for the maximum penalized-likelihood estimation of the L-MLR model. This algorithm is constructed via the minorization–maximization algorithm paradigm. Such a construction allows for coordinate-wise updates of the parameter components, and produces globally convergent sequences of estimates that generate monotonic sequences of penalized log-likelihood values. These three features are missing in the previously presented approximate expectation–maximization algorithms. The previous difficulty in producing a globally convergent algorithm for the maximum penalized-likelihood estimation of the L-MLR model is due to the intractability of finding exact updates for the mixture model mixing proportions in the maximization-step. This issue is resolved by showing that it can be converted into a simple numerical root finding problem that is proven to have a unique solution. The method is tested in simulation and with an application to Major League Baseball salary data from the 1990s and the present day, where the concept of whether player salaries are associated with batting performance is investigated.

Semiparametric mixture of additive regression models

ABSTRACTIn this article, we propose a semiparametric mixture of additive regression models, in which the regression functions are additive and non parametric while the mixing proportions and variances are constant. Compared with the mixture of linear regression models, the proposed methodology is more flexible in modeling the non linear relationship between the response and covariate. A two-step procedure based on the spline-backfitted kernel method is derived for computation. Moreover, we establish the asymptotic normality of the resultant estimators and examine their good performance through a numerical example.

Annotation Regression for Genome-Wide Association Studies with an Application to Psychiatric Genomic Consortium Data.

Although genome-wide association studies (GWAS) have been successful at finding thousands of disease-associated genetic variants (GVs), identifying causal variants and elucidating the mechanisms by which genotypes influence phenotypes are critical open questions. A key challenge is that a large percentage of disease-associated GVs are potential regulatory variants located in noncoding regions, making them difficult to interpret. Recent research efforts focus on going beyond annotating GVs by integrating functional annotation data with GWAS to prioritize GVs. However, applicability of these approaches is challenged by high dimensionality and heterogeneity of functional annotation data. Furthermore, existing methods often assume global associations of GVs with annotation data. This strong assumption is susceptible to violations for GVs involved in many complex diseases. To address these issues, we develop a general regression framework, named Annotation Regression for GWAS (ARoG). ARoG is based on a finite mixture of linear regressions model where GWAS association measures are viewed as responses and functional annotations as predictors. This mixture framework addresses heterogeneity of effects of GVs by grouping them into clusters and high dimensionality of the functional annotations by enabling annotation selection within each cluster. ARoG further employs permutation testing to evaluate the significance of selected annotations. Computational experiments indicate that ARoG can discover distinct associations between disease risk and functional annotations. Application of ARoG to autism and schizophrenia data from Psychiatric Genomics Consortium led to identification of GVs that significantly affect interactions of several transcription factors with DNA as potential mechanisms contributing to these disorders.

A class of finite mixture of quantile regressions with its applications

Mixture of linear regression models provide a popular treatment for modeling nonlinear regression relationship. The traditional estimation of mixture of regression models is based on Gaussian error assumption. It is well known that such assumption is sensitive to outliers and extreme values. To overcome this issue, a new class of finite mixture of quantile regressions (FMQR) is proposed in this article. Compared with the existing Gaussian mixture regression models, the proposed FMQR model can provide a complete specification on the conditional distribution of response variable for each component. From the likelihood point of view, the FMQR model is equivalent to the finite mixture of regression models based on errors following asymmetric Laplace distribution (ALD), which can be regarded as an extension to the traditional mixture of regression models with normal error terms. An EM algorithm is proposed to obtain the parameter estimates of the FMQR model by combining a hierarchical representation of the ALD. Finally, the iterated weighted least square estimation for each mixture component of the FMQR model is derived. Simulation studies are conducted to illustrate the finite sample performance of the estimation procedure. Analysis of an aphid data set is used to illustrate our methodologies.

Robust estimation of the number of components for mixtures of linear regression models

In this paper, we investigate a robust estimation of the number of components in the mixture of regression models using trimmed information criteria. Compared to the traditional information criteria, the trimmed criteria are robust and not sensitive to outliers. The superiority of the trimmed methods in comparison with the traditional information criterion methods is illustrated through a simulation study. Two real data applications are also used to illustrate the effectiveness of the trimmed model selection methods.

Robust variable selection for mixture linear regression models

In this paper, we propose a robust variable selection to estimate and select relevant covariates for the finite mixture of linear regression models by assuming that the error terms follow a Laplace distribution to the data after trimming the high leverage points. We introduce a revised Expectation-maximization (EM) algorithm for numerical computation. Simulation studies indicate that the proposed method is robust to both the high leverage points and outliers in the y-direction, and can obtain a consistent variable selection in the case of outliers or heavy-tail error distribution. Finally, we apply the proposed methodology to analyze a real data.

Mixtures of regressions with changepoints

We introduce an extension to the mixture of linear regressions model where changepoints are present. Such a model provides greater flexibility over a standard changepoint regression model if the data are believed to not only have changepoints present, but are also believed to belong to two or more unobservable categories. This model can provide additional insight into data that are already modeled using mixtures of regressions, but where the presence of changepoints has not yet been investigated. After discussing the mixture of regressions with changepoints model, we then develop an Expectation/Conditional Maximization (ECM) algorithm for maximum likelihood estimation. Two simulation studies illustrate the performance of our ECM algorithm and we analyze a real dataset.

Robust Mixture of Linear Regression Models

This article proposes a robust model for parameter estimation when modeling in the presence of latent heterogeneity of population. Mixture of regression models are used for modeling when the data are a mixture of subgroups to allow for heterogeneity of the population. Parameter estimates from standard mixture of linear regression models are sensitive to atypical observations. To study mixtures of linear regression models, we introduce a class of robust estimators, called S-estimators. We investigate their breakdown point in mixture of linear regression models. It is expected that the robust S-estimators can achieve the high breakdown point in the contaminated data from the heterogenous populations. This model presents a unified, robust framework and parameter estimation is achieved via an expectation-conditional maximization (ECM) algorithm. This new family of robust mixture models is validated through Monte Carlo simulations. The application to real life data sets has shown that use of robust S-estimators in mixture of linear regression models accommodates the outliers producing stable estimates as compared to maximum likelihood estimators from mixture of linear regression models.

Predicting gene expression in T cell differentiation from histone modifications and transcription factor binding affinities by linear mixture models

BackgroundThe differentiation process from stem cells to fully differentiated cell types is controlled by the interplay of chromatin modifications and transcription factor activity. Histone modifications or transcription factors frequently act in a multi-functional manner, with a given DNA motif or histone modification conveying both transcriptional repression and activation depending on its location in the promoter and other regulatory signals surrounding it.ResultsTo account for the possible multi functionality of regulatory signals, we model the observed gene expression patterns by a mixture of linear regression models. We apply the approach to identify the underlying histone modifications and transcription factors guiding gene expression of differentiated CD4+ T cells. The method improves the gene expression prediction in relation to the use of a single linear model, as often used by previous approaches. Moreover, it recovered the known role of the modifications H3K4me3 and H3K27me3 in activating cell specific genes and of some transcription factors related to CD4+ T differentiation.

Mixtures of regressions with predictor-dependent mixing proportions

We extend the standard mixture of linear regressions model by allowing the mixing proportions to be modeled nonparametrically as a function of the predictors. This framework allows for more flexibility in the modeling of the mixing proportions than the fully parametric mixture of experts model, which we also discuss. We present an EM-like algorithm for estimation of the new model. We also provide simulations demonstrating that our nonparametric approach can provide a better fit than the parametric approach in some instances and can serve to validate and thus reinforce the parametric approach in others. We also analyze and interpret two real data sets using the new method.

Fitting mixtures of linear regressions

In most applications, the parameters of a mixture of linear regression models are estimated by maximum likelihood using the expectation maximization (EM) algorithm. In this article, we propose the comparison of three algorithms to compute maximum likelihood estimates of the parameters of these models: the EM algorithm, the classification EM algorithm and the stochastic EM algorithm. The comparison of the three procedures was done through a simulation study of the performance (computational effort, statistical properties of estimators and goodness of fit) of these approaches on simulated data sets. Simulation results show that the choice of the approach depends essentially on the configuration of the true regression lines and the initialization of the algorithms.

On-Line Expectation–Maximization Algorithm for latent Data Models

SummaryWe propose a generic on-line (also sometimes called adaptive or recursive) version of the expectation–maximization (EM) algorithm applicable to latent variable models of independent observations. Compared with the algorithm of Titterington, this approach is more directly connected to the usual EM algorithm and does not rely on integration with respect to the complete-data distribution. The resulting algorithm is usually simpler and is shown to achieve convergence to the stationary points of the Kullback–Leibler divergence between the marginal distribution of the observation and the model distribution at the optimal rate, i.e. that of the maximum likelihood estimator. In addition, the approach proposed is also suitable for conditional (or regression) models, as illustrated in the case of the mixture of linear regressions model.

Determining the number of components in mixtures of linear models

Methods for determining the number of components in normal mixtures are extended to mixtures of linear regression models. This simulation study evaluates the influence of component separation and mixing proportions on the performance of 22 approximations of measures for determining the number of components in mixtures of linear regression models. Estimated measures based on the maximized log likelihood of the observed data are compared to estimated measures based on the maximized log likelihood of the complete data. Approximations of measures which previously required the convergence rate of the EM algorithm are presented which have no such restriction for their implementation. As an alternative to the EM algorithm, which is known to be sensitive to starting values, differential evolution was the implemented optimization algorithm. This study is further set apart in that the performances of the approximated component measures are explored without assuming the mixing proportions to be equal or assuming equal component variances. Based on the results of the k=1 and 2 component model simulations, the minimum description length, MDL, is the recommended criterion for choosing between one and two component mixtures of linear regression models.