Mixture Regression Research Articles

Bayesian scale mixtures of normals linear regression and Bayesian quantile regression with big data and variable selection

Quantile regression, which estimates various conditional quantiles of a response variable, including the median (0.5th quantile), is particularly useful when the conditional distribution is asymmetric or heterogeneous or fat-tailed or truncated. Bayesian methods for the inference of quantile regression have been receiving increasing attention from both theoretical and empirical viewpoints but facing the challenge of scaling up when the data are too large to be processed by a single machine under many big data environments nowadays. In this paper, we develop a structure link between Bayesian scale mixtures of normals linear regression and Bayesian quantile regression (BQR) via normal-inverse-gamma (NIG) distribution type of likelihood function, prior distribution and posterior distribution. We further explore the detailed methods of BQR for big data, variable selection and posterior prediction. The performance of the proposed techniques is evaluated via simulation studies and a real data analysis.

Hidden Markov Mixture of Gaussian Process Functional Regression: Utilizing Multi-Scale Structure for Time Series Forecasting

The mixture of Gaussian process functional regressions (GPFRs) assumes that there is a batch of time series or sample curves that are generated by independent random processes with different temporal structures. However, in real situations, these structures are actually transferred in a random manner from a long time scale. Therefore, the assumption of independent curves is not true in practice. In order to get rid of this limitation, we propose the hidden-Markov-based GPFR mixture model (HM-GPFR) by describing these curves with both fine- and coarse-level temporal structures. Specifically, the temporal structure is described by the Gaussian process model at the fine level and the hidden Markov process at the coarse level. The whole model can be regarded as a random process with state switching dynamics. To further enhance the robustness of the model, we also give a priori parameters to the model and develop a Bayesian-hidden-Markov-based GPFR mixture model (BHM-GPFR). The experimental results demonstrated that the proposed methods have both high prediction accuracy and good interpretability.

MODELING CELL POPULATIONS MEASURED BY FLOW CYTOMETRY WITH COVARIATES USING SPARSE MIXTURE OF REGRESSIONS.

The ocean is filled with microscopic microalgae, called phytoplankton, which together are responsible for as much photosynthesis as all plants on land combined. Our ability to predict their response to the warming ocean relies on understanding how the dynamics of phytoplankton populations is influenced by changes in environmental conditions. One powerful technique to study the dynamics of phytoplankton is flow cytometry which measures the optical properties of thousands of individual cells per second. Today, oceanographers are able to collect flow cytometry data in real time onboard a moving ship, providing them with fine-scale resolution of the distribution of phytoplankton across thousands of kilometers. One of the current challenges is to understand how these small- and large-scale variations relate to environmental conditions, such as nutrient availability, temperature, light and ocean currents. In this paper we propose a novel sparse mixture of multivariate regressions model to estimate the time-varying phytoplankton subpopulations while simultaneously identifying the specific environmental covariates that are predictive of the observed changes to these subpopulations. We demonstrate the usefulness and interpretability of the approach using both synthetic data and real observations collected on an oceanographic cruise conducted in the northeast Pacific in the spring of 2017.

Lifelong Bayesian Learning Machines for Streaming Industrial Big Data

With the advent of the big data era and the timeliness requirements of data processing, a large amount of streaming industrial big data is continuously obtained in real time. Facing this kind of flowing and time-varying knowledge information form, incremental learning is necessary. Lifelong learning (LL), as a typical incremental learning method, can continuously retain and accumulate old knowledge while learning new knowledge, which is very suitable for streaming industrial big data scenarios. At the same time, Bayesian nonparametric (BNP) models can adjust the complexity of the model based on observation data. Motivated by ideas of BNP and LL, a lifelong Bayesian learning machines framework is proposed in this article, which includes model expansion and model optimization. In general, this framework not only learns new effective knowledge and accumulates knowledge through incremental variational Bayesian under model expansion but also uses optimization steps to avoid model degradation caused by unnecessary component information. As an example, Dirichlet processes Gaussian mixture regression (DPGMR) is utilized for process modeling under this framework. To evaluate the feasibility and efficiency of the developed method, a synthetic and a real industrial case are demonstrated.

Practical State of Health Estimation for LiFePO4 Batteries Based on Gaussian Mixture Regression and Incremental Capacity Analysis

High accuracy, high adaptability and low complexity have always been the goals of on-board state of health (SoH) estimation in a well-designed battery management system for electric vehicles. In this article, we proposed a practical SoH estimation method for LiFePO <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sub> batteries based on Gaussian mixture regression (GMR) and incremental capacity (IC) analysis. According to the close correlation between the SoH and the IC curve of LiFePO <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sub> battery, the peak coordinates of the curve are extracted as capacity feature to train an aging model using GMR. The capacity degradation of batteries is well described by multiple Gaussian distributions, and the estimation accuracy of SoH is improved. Furthermore, the coordinates of three peaks in the curve are separately analyzed as capacity feature to enhance the adaptability of the aging model to different charging regions. Finally, the complexity of the proposed method is compared with that of other methods. The results show that taking the coordinates of any peak as the capacity feature, SoH can be accurately estimated based on GMR even for different tested batteries, and the mean absolute error and root mean square error are both less than 1%.

Mixture Regression for Clustering Atmospheric-Sounding Data: A Study of the Relationship between Temperature Inversions and PM10 Concentrations

Temperature inversions prevent the mixing of air near the surface with the air higher in the atmosphere, contributing to high concentrations of air pollutants. Inversions can be identified by sampling temperature data at different heights, usually done with radiosondes. In our study, we propose using the SMIXS clustering algorithm to cluster radiosonde temperature data as longitudinal data into clusters with distinct temperature profile shapes. We clustered 8 years of early morning radiosonde data from Ljubljana, Slovenia, into 15 clusters and investigated their relationship to PM10 pollution. The results show that high PM10 concentrations (above 50 g/m3, which is the daily limit value) are associated with early morning temperature inversions. The highest concentrations are typical for winter days with the strongest temperature inversions (temperature difference of 5 ∘C or more in the inversion layer) while the lowest concentrations (about 10 g/m3) are typical for days with no early morning temperature inversion. Days with very strong temperature inversions are quite rare. We show that clustering temperature profiles into a distinct number of clusters adds to the interpretability of radiosonde data. It simplifies the characterization of temperature inversions, their frequency, occurrence, and their impact on PM10 concentrations.

Mapping study of papillary thyroid carcinoma in China: Predicting EQ-5D-5L utility values from FACT-H&N.

To develop a mapping algorithm that can be used to predict EQ-5D-5L health utility scores from FACT-H&N and obtain health utility parameters for Chinese patients with papillary thyroid carcinoma (PTC), which can be used for cost-utility analysis in health economic. A total of 1,050 patients with PTC from a tertiary hospital in China were included, and they completed FACT-H&N and EQ-5D-5L. Four mapping algorithms of direct mapping functions were used to derive the models: Ordinary least squares (OLS), Tobit model (Tobit), Two-part model (TPM), and Beta mixture regression model (Beta). The goodness-of-fit of models was assessed by the mean absolute error (MAE), root mean square error (RMSE), Akaike information criteria (AIC), Bayesian information criteria (BIC), and absolute error (AE). A fivefold cross-validation method was used to test the stability of the models. The mean utility value of the EQ-5D-5L was 0.870 ± 0.094. The mean EQ-VAS score was 76.5 ± 13.0. The Beta mixture regression model mapping FACT-H&N to EQ-5D-5L achieved the best performance [fivefold cross-validation MAE = 0.04612, RMSE = 0.06829, AIC = -2480.538, BIC = -2381.137, AE > 0.05 (%) = 32.48, AE > 0.1 (%) = 8.95]. The independent variables in this model were Physical Well-Being (PWB), Emotional Well-Being (EWB), Head & Neck Cancer Subscale (HNCS) scores and its square term and interaction term scores. This study calculated the health utility score of Chinese patients with PTC. The reported algorithms can be used to map the FACT-H&N into the EQ-5D-5L, which can be applied in the cost-utility related study of patients with PTC.

The integral contributions of parental involvement and parenting style to adolescent adjustments: a regression mixture analysis.

The integrative model of parenting has highlighted the integral contributions of parental involvement (quantity) and parenting style (quality) to adolescent psychological adjustments. The first aim of this study was to adopt the person-centered approach to identify profiles of parental involvement (quantity) and parenting styles (quality). The second purpose was to examine the associations between different parenting profiles and adolescent psychological adjustments. A cross-sectional online survey with families (N = 930) that included fathers, mothers, and adolescents (50% female, M age = 14.37 ± 2.31) was conducted in mainland China. The fathers and mothers reported their level of parental involvement; the adolescents rated fathers' and mothers' parenting styles, as well as their own levels of anxiety symptoms, depression symptoms, and loneliness. Latent profile analysis was adopted to identify parenting profiles using the standardized scores of fathers' and mothers' involvement and style (warmth and rejection). The regression mixture model was used to examine the relationships between different parenting profiles and adolescent psychological adjustments. Four classes best characterized the parenting behaviors: warm involvement (52.6%), neglecting noninvolvement (21.4%), rejecting noninvolvement (21.4%), and rejecting involvement (4.6%). Adolescents in the warm involvement group scored lowest on anxiety symptoms, depression symptoms, and loneliness. Adolescents in rejecting involvement group scored highest on psychological adjustment indicators. Adolescents in neglecting noninvolvement group scored lower on anxiety symptoms than those in rejecting noninvolvement group. Adolescents in the warm involvement group adjusted best, while adolescents in the rejecting involvement group adjusted worst among all groups. To promote adolescents' mental health, intervention programs need to consider both parental involvement and parenting styles simultaneously.

Gaussian mixture continuously adaptive regression for multimode processes soft sensing under time-varying virtual drift

Due to time-varying virtual drift in multimode processes, the performance of soft sensors will degrade after online deployment. Traditional adaptive mechanisms have been developed to address this issue, but all have limitations in the real-world scenario. Therefore, this paper explores domain adaptation as a new adaptive mechanism for its unsupervised knowledge calibration effect, and a Gaussian mixture continuously adaptive regression (GMCAR) is proposed for soft sensor modeling. First, we extend Gaussian domain adaptation to Gaussian mixture domain adaptation, in which the target domain could transfer knowledge from multiple Gaussian source domains with different posterior probabilities. Furthermore, a new sensitivity-based continuous domain adaptation is presented to efficiently and naturally update the Gaussian mixture domain adaptation by exploring existing knowledge and using the continuity of data streams. As well, an adaptive Gaussian mixture regression is designed to continuously update the soft sensing model according to the current process state. Finally, experimental results on the TE benchmark process and a real air separation process demonstrate that the proposed GMCAR-based soft sensor achieves state-of-the-art performance under time-varying virtual drift.

Synergism of macrocyclic lactones against Haemonchus contortus.

A possible synergistic effect of macrocyclic lactones' (MLs) combination has been previously described against resistant gastrointestinal nematodes of cattle. In addition to synergism, drug-drug interactions between MLs can also result in additive or antagonistic effect, considering the different MLs pharmacokinetics, pharmacodynamics, and interactions with molecular mechanisms of resistance. Therefore, the aim of the current work was evaluated the effect of different MLs combinations against Haemonchus contortus. Infecting larvae of two isolates (one susceptible and one resistant to ivermectin) were used in the larval migration inhibition test. After estimating the half maximal effective concentration of abamectin (ABA), eprinomectin, (EPR), ivermectin (IVM), and moxidectin (MOX) for both isolates, combinations were delineated by a simplex-centroid mixture experiment, and the mixture regression analysis was applied to the special cubic model. A synergistic effect was found for the EPR + MOX against the susceptible isolate as well as the EPR + MOX, IVM + MOX, and ABA + EPR + IVM against the resistant isolate. An antagonistic effect of ABA + IVM + MOX was found against the susceptible isolate. For the susceptible isolate, a higher inhibition was found with greater proportions of EPR and lower proportions of the other drugs compared to the reference mixture. For the resistant isolate, inhibition greater than that of the reference mixture was found with higher proportions of IVM as well as lower proportions of the other drugs. The synergistic and antagonistic effects were dependent on the following: (a) parasite drug resistance profile, (b) the composition of the combination, and (c) the proportions used, with EPR and IVM exerting a greater impact on these effects.

Who is Most at Risk for Developing Physical Aggression After Playing Violent Video Games? An Individual Differences Perspective From Early Adolescence to Emerging Adulthood.

Many theories of development suggest that playing violent video games would not impact all adolescents the same way, yet empirical research is sparse. To date there have been no within-subjects analyses that examine which adolescents are most at risk for developing aggression after playing violent video games and under what context, and no longitudinal studies encompassing multiple developmental stages (i.e., early adolescence to emerging adulthood) that examine the long-term effects of playing violent video games. To address this gap, the current study used a longitudinal design (spanning 8 years and encompassing multiple developmental periods) with a sample of U.S. adolescents who completed questionnaires on aggression, video gaming, and multiple risk and protective factors for aggression. Participants included 488 adolescents (M age = 13.82, SD = 1.03 at the initial wave, 51% female, with 65% being White, 12% Black, 19% multiethnic, and 4% other). Mixture regression was utilized to model physical aggression over time and to examine how playing violent video games might be related to aggression on an individual level. There were four classes: "Multi-risk", "High Gaming, High Aggression" (both of which had high levels of aggression over time), "Moderate Risk", and "Low Risk, High Privilege" (both of which had lower levels of aggression over time). Individuals were most aggressive with multiple risk factors or higher levels of violent game play and playing violent video games likely increased aggression more in individuals with other gaming problems and in males with low self-regulation. The results are discussed using a balanced perspective and recognize that violent video games do not affect all adolescents the same way. This research has direct implications for designing interventions around reducing aggressive behavior among adolescents.

Sharp global convergence guarantees for iterative nonconvex optimization with random data

We consider a general class of regression models with normally distributed covariates, and the associated nonconvex problem of fitting these models from data. We develop a general recipe for analyzing the convergence of iterative algorithms for this task from a random initialization. In particular, provided each iteration can be written as the solution to a convex optimization problem satisfying some natural conditions, we leverage Gaussian comparison theorems to derive a deterministic sequence that provides sharp upper and lower bounds on the error of the algorithm with sample splitting. Crucially, this deterministic sequence accurately captures both the convergence rate of the algorithm and the eventual error floor in the finite-sample regime, and is distinct from the commonly used “population” sequence that results from taking the infinite-sample limit. We apply our general framework to derive several concrete consequences for parameter estimation in popular statistical models including phase retrieval and mixtures of regressions. Provided the sample size scales near linearly in the dimension, we show sharp global convergence rates for both higher-order algorithms based on alternating updates and first-order algorithms based on subgradient descent. These corollaries, in turn, reveal multiple nonstandard phenomena that are then corroborated by extensive numerical experiments.

Uniform consistency in nonparametric mixture models

We study uniform consistency in nonparametric mixture models as well as closely related mixture of regression (also known as mixed regression) models, where the regression functions are allowed to be nonparametric and the error distributions are assumed to be convolutions of a Gaussian density. We construct uniformly consistent estimators under general conditions while simultaneously highlighting several pain points in extending existing pointwise consistency results to uniform results. The resulting analysis turns out to be nontrivial, and several novel technical tools are developed along the way. In the case of mixed regression, we prove L1 convergence of the regression functions while allowing for the component regression functions to intersect arbitrarily often, which presents additional technical challenges. We also consider generalizations to general (i.e., nonconvolutional) nonparametric mixtures.

Improving the accuracy and internal consistency of regression-based clustering of high-dimensional datasets.

Component-wise Sparse Mixture Regression (CSMR) is a recently proposed regression-based clustering method that shows promise in detecting heterogeneous relationships between molecular markers and a continuous phenotype of interest. However, CSMR can yield inconsistent results when applied to high-dimensional molecular data, which we hypothesize is in part due to inherent limitations associated with the feature selection method used in the CSMR algorithm. To assess this hypothesis, we explored whether substituting different regularized regression methods (i.e. Lasso, Elastic Net, Smoothly Clipped Absolute Deviation (SCAD), Minmax Convex Penalty (MCP), and Adaptive-Lasso) within the CSMR framework can improve the clustering accuracy and internal consistency (IC) of CSMR in high-dimensional settings. We calculated the true positive rate (TPR), true negative rate (TNR), IC and clustering accuracy of our proposed modifications, benchmarked against the existing CSMR algorithm, using an extensive set of simulation studies and real biological datasets. Our results demonstrated that substituting Adaptive-Lasso within the existing feature selection method used in CSMR led to significantly improved IC and clustering accuracy, with strong performance even in high-dimensional scenarios. In conclusion, our modifications of the CSMR method resulted in improved clustering performance and may thus serve as viable alternatives for the regression-based clustering of high-dimensional datasets.

A Gripper-like Exoskeleton Design for Robot Grasping Demonstration

Learning from demonstration (LfD) is a practical method for transferring skill knowledge from a human demonstrator to a robot. Several studies have shown the effectiveness of LfD in robotic grasping tasks to improve the success rate of grasping and to accelerate the development of new robotic grasping tasks. A well-designed demonstration device can effectively represent human grasping motion to transfer grasping skills to robots. In this paper, an improved gripper-like exoskeleton with a data collection system is proposed. First, we present the mechatronic details of the exoskeleton and its motion-tracking system, considering the manipulation flexibility and data acquisition requirements. We then present the capabilities of the device and its data collection system, which collects the position, pose and displacement of the gripper on the exoskeleton. The collected data is further processed by the data acquisition and processing software. Next, we describe the principles of Gaussian mixture model (GMM) and Gaussian mixture regression (GMR) in robot skill learning, which are used to transfer the raw data from demonstrations to robot motions. In the experiment, an optimized trajectory was learned from multiple demonstrations and reproduced on a robot. The results show that the GMR complemented with GMM is able to learn a smooth trajectory from demonstration trajectories with noise.

De Novo Direct Inverse QSPR/QSAR: Chemical Variational Autoencoder and Gaussian Mixture Regression Models.

Herein, we propose a de novo direct inverse quantitative structure-property relationship/quantitative structure-activity relationship (QSPR/QSAR) analysis method, based on the chemical variational autoencoder (VAE) and Gaussian mixture regression (GMR) models, to generate molecules with the desired target variables of interest for properties and activities (y). A data set of molecules was analyzed, and an encoder was used to transform the simplified molecular input line entry system (SMILES) strings to latent variables (x), while a decoder was used to transform x to SMILES strings. A chemical VAE model was used for analysis and a GMR model (between x and y) was constructed for direct inverse analysis. The target y values were input into the GMR model to directly predict the x values. Following this, the predicted x values were input into the decoder associated with the chemical VAE model and the SMILES string representations (or chemical structures of molecules) were obtained as the output, indicating that the proposed method could be used to selectively obtain the molecules that were characterized by the target y values. We confirmed that the proposed method can be used to generate molecules within the target y ranges even when the conventional chemical VAE model failed to generate the target molecules.

Graphical and Computational Tools to Guide Parameter Choice for the Cluster Weighted Robust Model

The Cluster Weighted Robust Model (CWRM) is a recently introduced methodology to robustly estimate mixtures of regressions with random covariates. The CWRM allows users to flexibly perform regression clustering, safeguarding it against data contamination and spurious solutions. Nonetheless, the resulting solution depends on the chosen number of components in the mixture, the percentage of impartial trimming, the degree of heteroscedasticity of the errors around the regression lines and of the clusters in the explanatory variables. Therefore, an appropriate model selection is crucially required. Such a complex modeling task may generate several “legitimate” solutions: each one derived from a distinct hyperparameters specification. The present article introduces a two step-monitoring procedure to help users effectively explore such a vast model space. The first phase uncovers the most appropriate percentages of trimming, whilst the second phase explores the whole set of solutions, conditioning on the outcome derived from the previous step. The final output singles out a set of “top” solutions, whose optimality, stability and validity is assessed. Novel graphical and computational tools—specifically tailored for the CWRM framework—will help the user make an educated choice among the optimal solutions. Three examples on real datasets showcase our proposal in action. Supplementary files for this article are available online.

Flexible mixture regression with the generalized hyperbolic distribution

Flexible mixture regression with the generalized hyperbolic distribution

HETEROGENEITY ANALYSIS VIA INTEGRATING MULTI-SOURCES HIGH-DIMENSIONAL DATA WITH APPLICATIONS TO CANCER STUDIES.

This study has been motivated by cancer research, in which heterogeneity analysis plays an important role and can be roughly classified as unsupervised or supervised. In supervised heterogeneity analysis, the finite mixture of regression (FMR) technique is used extensively, under which the covariates affect the response differently in subgroups. High-dimensional molecular and, very recently, histopathological imaging features have been analyzed separately and shown to be effective for heterogeneity analysis. For simpler analysis, they have been shown to contain overlapping, but also independent information. In this article, our goal is to conduct the first and more effective FMR-based cancer heterogeneity analysis by integrating high-dimensional molecular and histopathological imaging features. A penalization approach is developed to regularize estimation, select relevant variables, and, equally importantly, promote the identification of independent information. Consistency properties are rigorously established. An effective computational algorithm is developed. A simulation and an analysis of The Cancer Genome Atlas (TCGA) lung cancer data demonstrate the practical effectiveness of the proposed approach. Overall, this study provides a practical and useful new way of conducting supervised cancer heterogeneity analysis.

Sparse Estimation in Finite Mixture of Accelerated Failure Time and Mixture of Regression Models with R Package fmrs

Variable selection in large-dimensional data has been extensively studied in different settings over the past decades. In a recent article, Shokoohi et. al. [29, DOI:10.1214/18-AOAS1198] proposed a method for variable selection in finite mixture of accelerated failure time regression models for studies on time-to-event data to capture heterogeneity within the population and account for censoring. In this paper, we introduce the fmrs package, which implements the variable selection methodology for such models. Furthermore, as a byproduct, the fmrs package facilitates variable selection in finite mixture regression models. The package also incorporates a tuning parameter selection mechanism based on component-wise bic. Commonly used penalties, such as Least Absolute Shrinkage and Selection Operator, and Smoothly Clipped Absolute Deviation, are integrated into fmrs. Additionally, the package offers an option for non-mixture regression models. The C language is chosen to boost the optimization speed. We provide an overview of the fmrs principles and the strategies employed for optimization. Hands-on illustrations are presented to help users get acquainted with fmrs. Finally, we apply fmrs to a lung cancer dataset and observe that a two-component mixture model reveals a subgroup with a more aggressive form of the disease, displaying a lower survival time.