Mixture Regression Research Articles

Robust mixture regression modeling based on the generalized M (GM)-estimation method

A robust mixture regression based on the M regression estimation method has already been proposed in literature. However, since the M-estimators are only robust against the outliers in response variables, the resulting mixture regression methods will not be robust against the outliers in explanatory variables (leverage points). In this paper, we propose a robust mixture regression procedure to handle the outliers and the leverage points, simultaneously. Our proposed mixture regression procedure is based on the GM regression estimation method. We give an EM-type algorithm to compute estimates for the parameters of interest. We provide a simulation study and a real data example to assess the robustness performance of the proposed method against the outliers and the leverage points.

Clustering electricity consumers using high‐dimensional regression mixture models

AbstractA massive amount of data about individual electrical consumptions are now provided with new metering technologies and smart grids. These new data are especially useful for load profiling and load modeling at different scales of the electrical network. A new methodology based on mixture of high‐dimensional regression models is used to perform clustering of individual customers. It leads to uncovering clusters corresponding to different regression models. Temporal information is incorporated in order to prepare the next step, the fit of a forecasting model in each cluster. Only the electrical signal is involved, slicing the electrical signal into consecutive curves to consider it as a discrete time series of curves. Interpretation of the models is given on a real smart meter dataset of Irish customers.

Comparing Coefficients Across Subpopulations in Gaussian Mixture Regression Models

When fitting a Gaussian mixture regression model to observed data, estimating a between-group contrast can be a practical issue. One can use the estimate to compare the effects of a particular covariate or a set of covariates across different subpopulations. By applying fiducial generalized pivotal quantities, a small-sample solution is proposed in this paper to obtain interval estimates of between-group contrasts. Specifically, a Markov chain Monte Carlo sampler, which takes the membership uncertainty of each individual into account, is designed to generate realizations from the target distributions for computing the required interval estimates. A plant virus transmission study is first introduced as a motivating example for the present study. Next, the observed data are analyzed to illustrate the proposed method. Based on the simulation results, it is further shown that the proposed method can maintain the empirical coverage rates sufficiently close to the nominal level. Supplementary materials accompanying this paper appear on-line.

Subspace Clustering Under Complex Noise

In this paper, we study the subspace clustering problem under complex noise. A wide class of reconstruction-based methods model the subspace clustering problem by combining a quadratic data-fidelity term and a regularization term. In a statistical framework, the data-fidelity term assumes to be contaminated by a unimodal Gaussian noise, which is a popular setting in most current subspace clustering models. However, the realistic noise is much more complex than our assumptions. Besides, the coarse representation of the data-fidelity term may depress the clustering accuracy, which is often used to evaluate the models. To address this issue, we propose the mixture of Gaussian regression (MoG Regression) for subspace clustering. The MoG Regression seeks a valid way to model the unknown noise distribution, which approaches the real one as far as possible, so that the desired affinity matrix is better at characterizing the structure of data in the real world, and furthermore, improving the performance. Theoretically, the proposed model enjoys the grouping effect, which encourages the coefficients of highly correlated points are nearly equal. Drawing upon the ideal of the minimum message length, a model selection strategy is proposed to estimate the numbers of the Gaussian components that shows a way how to seek the number of Gaussian components besides determining it by empirical value. In addition, the asymptotic property of our model is investigated. The proposed model is evaluated on the challenging datasets. The experimental results show that the proposed MoG Regression model significantly outperforms several state-of-the-art subspace clustering methods.

Data Integrative Bayesian Inference for Mixtures of Regression Models

Summary Modern data collection techniques, which often produce different types of relevant information, call for new statistical learning methods that are adapted to cope with data integration. In the paper Bayesian inference is considered for mixtures of regression models with an unknown number of components, that facilitates data integration and variable selection for high dimensional data. In the approach presented, named data integrative mixture of regressions, data integration is accomplished by introducing a new data allocation scheme that summarizes additional data in the form of an informative prior on latent variables. To cope with high dimensionality, a shrinkage-type prior is assumed on the regression parameters, and a posteriori variable selection is conducted based on Bayesian credible intervals. Posterior estimation is achieved via a Markov chain Monte Carlo algorithm. The method is validated through simulation studies and illustrated by its performance on real data.

Optimization of the proportions of four wastewaters in a blend for the cultivation of microalgae using a mixture design

A 2nd degree mixture design was used to determine the optimal blend prepared from four wastewater streams to produce microalgae-based biomass. The streams consisted of a liquid digestate from an anaerobic digestion process, a landfill leachate, a septic-system sludge treatment plant liquid, and a wastewater treatment plant effluent. The mixture regression analysis indicated that blends with higher proportions of treated effluent and digestate improved cells growth, while the use of leachate was detrimental to the growth. The global solution of the mixture optimization predicted a maximum value of biomass productivity of 22.76 mg L−1 d−1, in a blend consisting of 19% treated effluent, 21% digestate, and 60% water. Proportions of leachate higher than 13.33% were detrimental to the growth. The concentration of ammonia-N in the blends ranged from 0.39 to 150 mg L−1 d−1, and its toxicity effect on the cells diminished with increasing amounts of organic carbon in the cultivation medium.

Motor Skills Learning and Generalization with Adapted Curvilinear Gaussian Mixture Model

This paper is intended to solve the motor skills learning, representation and generalization problems in robot imitation learning. To this end, we present an Adapted Curvilinear Gaussian Mixture Model (AdC-GMM), which is a general extension of the GMM. The proposed model can encode data more compactly. More critically, it is inherently suitable for representing data with strong non-linearity. To infer the parameters of this model, a Cross Entropy Optimization (CEO) algorithm is proposed, where the cross entropy loss of the training data is minimized. Compared with the traditional Expectation Maximization (EM) algorithm, the CEO can automatically infer the optimal number of components. Finally, the generalized trajectories are retrieved by an Adapted Curvilinear Gaussian Mixture Regression (AdC-GMR) model. To encode observations from different frames, the sophisticated task parameterization (TP) technique is introduced. All above proposed algorithms are verified by comprehensive tasks. The CEO is evaluated by a hand writing task. Another goal-directed reaching task is used to evaluate the AdC-GMM and AdC-GMR algorithm. A novel hammer-over-a-nail task is designed to verify the task parameterization technique. Experimental results demonstrate the proposed CEO is superior to the EM in terms of encoding accuracy and the AdC-GMM can achieve more compact representation by reducing the number of components by up to 50%. In addition, the trajectory retrieved by the AdC-GMR is smoother and the approximation error is comparable to the Gaussian process regression (GPR) even far fewer parameters need to be estimated. Because of this, the AdC-GMR is much faster than the GPR. Finally, simulation experiments on the hammer-over-a-nail task demonstrates the proposed methods can be deployed and used in real-world applications.

Segmentation and generalisation for writing skills transfer from humans to robots

In this study, the authors present an enhanced generalised teaching by demonstration technique for a KUKA iiwa robot. Movements are recorded from a human operator, and then the recorded data are sent to be segmented via MATLAB by using the difference method (DV). The outputted trajectories data are used to model a non-linear system named dynamic movement primitive (DMP). For the purpose of learning from multiple demonstrations correctly and accurately, the Gaussian mixture model is employed for the evaluation of the DMP in order to modelling multiple trajectories by the teaching of demonstrator. Furthermore, a synthesised trajectory with smaller position errors in 3D space has been successfully generated by the usage of the Gaussian mixture regression algorithm. The proposed approach has been tested and demonstrated by performing a Chinese characters writing task with a KUKA iiwa robot.

Drug sensitivity prediction with high-dimensional mixture regression.

This paper proposes a mixture regression model-based method for drug sensitivity prediction. The proposed method explicitly addresses two fundamental issues in drug sensitivity prediction, namely, population heterogeneity and feature selection pertaining to each of the subpopulations. The mixture regression model is estimated using the imputation-conditional consistency algorithm, and the resulting estimator is consistent. This paper also proposes an average-BIC criterion for determining the number of components for the mixture regression model. The proposed method is applied to the CCLE dataset, and the numerical results indicate that the proposed method can make a drastic improvement over the existing ones, such as random forest, support vector regression, and regularized linear regression, in both drug sensitivity prediction and feature selection. The p-values for the comparisons in drug sensitivity prediction can reach the order O(10−8) or lower for the drugs with heterogeneous populations.

Middle and Long-Term Runoff Probabilistic Forecasting Based on Gaussian Mixture Regression

Reliable forecasts of middle and long-term runoff can be highly valuable for water resources planning and management. The uncertainty of runoff forecasting is also essential for water resource managers. However, deterministic models only provide single prediction values without uncertainty attached. In this study, Gaussian Mixture Regression (GMR) approach is applied for probabilistic middle and long-term runoff forecasting, which can quantify the predictive uncertainty directly. GMR consists two parts, optimizing the model parameters and hyperparameters of Gaussian Mixture Model (GMM) and forecasting the posterior conditional probability density. GMR is applied to a real-world runoff forecasting case study at Xiangjiaba Station and Panzhihua Station on the Jinsha River. And it is compared with Support Vector Machines and Artificial Neural Network. The experimental results show its excellent performance both on accuracy and reliability. Uncertainty estimation for the probabilistic forecast is also shown, the results demonstrate that GMR is able to handle the heteroscedastic data like runoff and can provide an effective uncertainty estimation.

Skill Learning From Human Demonstrations Using Dynamical Regressive Models for Multitask Applications

This paper is concerned with the motor skill learning from human demonstrations using the framework of dynamic regressive models (DRMs). The DRM-based motion planner is preferred as it generates the end-effector trajectory dynamically based on the current state of the end-effector. Within existing frameworks, a single DRM can learn a single motion profile. In addition, such learned DRMs from the data may not be stable. This paper addresses both these issues in a comprehensive manner. In this paper a single DRM has been used to encode human demonstrations involving multitask profiles and multiple task-equilibriums which is novel. We have introduced the idea that the learned DRM will generate human-like stable motion if the energy dissipation rate (EDR) of the generated trajectory follows that of the human demonstration. Thus, the DRM structure has been modified by adding a continuous guiding signal which can be called as the control signal. This signal has been derived using control theoretic principle to ensure asymptotic stability while maintaining the EDR equivalent to that of the human demonstration. The asymptotic stability of the learned DRM has been established by involving a nonmonotonic Lyapunov function consisting of first derivative of a quadratic function and the energy function associated with the DRM. The proposed framework can be learned using many existing regression techniques in this paper Gaussian mixture regression , locally weighted projection regression , and support vector regression techniques have been used successfully. During the pick and place tasks, human demonstrations involving multiple task profiles and multiple task-equilibriums are generated using a 7 DOF commercial robot manipulator. Experimental validations show that the DRMs learned using these three regression schemes are able to guide the robot along the multitask profiles in a stable manner.

Perceptual Errors Support the Notion of Masking by Object Substitution.

Two experiments examined the effect of object substitution masking (OSM) on the perceptual errors in reporting the orientation of a target. In Experiment 1, a four-dot trailing mask was compared with a simultaneous-noise mask. In Experiment 2, the four-dot and noise masks were factorially varied. Responses were modelled using a mixture regression model and Bayesian inference to deduce whether the relative impacts of OSM on guessing and precision were the same as those of a noise mask, and thus whether the mechanism underpinning OSM is based on increasing noise rather than a substitution process. Across both experiments, OSM was associated with an increased guessing rate when the mask trailed target offset and a reduction in the precision of the target representation (although the latter was less reliable across the two experiments). Importantly, the noise mask also influenced both guessing and precision, but in a different manner, suggesting that OSM is not simply caused by increasing noise. In Experiment 2, the effects of OSM and simultaneous-noise interacted, suggesting the two manipulations involve common mechanisms. Overall results suggest that OSM is often a consequence of a substitution process, but there is evidence that the mask increases noise levels on trials where substitution does not occur.

Programming by Demonstration Using the Teleimpedance Control Scheme: Verification by an sEMG-Controlled Ball-Trapping Robot

Impedance control allows robots to manipulate physical interactions delicately. However, issues associated with path planning and impedance remain unresolved. Herein we propose a path and impedance planning method for impedance control in a robot based on programming by demonstration through telemanipulation using a surface electromyogram. We considered a task that requires quick and precise adjustment of path and impedance, that is, ball trapping. We implemented a teleoperated robot that can deliver an operator's impedance as well as position during the ball trapping task to the slave side. The operators were asked to perform demonstrations of ball-trapping tasks using the implemented teleoperated robot, where the slave side is a vertical robot with one degree of freedom. The path and impedance were recorded and programmed as control input profiles from a set of successful demonstrations using Gaussian mixture regression. The result showed that using the human demonstration, the robot could learn how to catch a dropped ball without rebounding.

Beam-pointing drift prediction in pulsed lasers by a probabilistic learning approach.

In laser systems, it is well known that beam pointing is shifted due to many un-modeled factors, such as vibrations from the hardware platform and air disturbance. In addition, beam-pointing shift also varies with laser sources as well as time, rendering the modeling of shifting errors difficult. While a few works have addressed the problem of predicting shift dynamics, several challenges still remain. Specifically, a generic approach that can be easily applied to different laser systems is highly desired. In contrast to physical modeling approaches, we aim to predict beam-pointing drift using a well-established probabilistic learning approach, i.e., the Gaussian mixture model. By exploiting sampled datapoints (collected from the laser system) comprising time and corresponding shifting errors, the joint distribution of time and shifting error can be estimated. Subsequently, Gaussian mixture regression is employed to predict the shifting error at any query time. The proposed learning scheme is verified in a pulsed laser system (1064nm, Nd:YAG, 100Hz), showing that the drift prediction approach achieves remarkable performances.

Trajectory Clustering Aided Personalized Driver Intention Prediction for Intelligent Vehicles

Early driver intention prediction plays a significant role in intelligent vehicles. Drivers exhibit various driving characteristics impairing the performance of conventional algorithms using all drivers’ data indiscriminatingly. This paper develops a personalized driver intention prediction system at unsignalized T intersections by seamlessly integrating clustering and classification. Polynomial regression mixture (PRM) clustering and Akaike's information criterion are applied to individual drivers trajectories for learning in-depth driving behaviors. Then, various classifiers are evaluated to link low-level vehicle states to high-level driving behaviors. CART classifier with Bayesian optimization excels others in accuracy and computation. The proposed system is validated by a real-world driving dataset. Comparative experimental results indicate that PRM clustering can discover more in-depth driving behaviors than manually defined maneuver due to its fine ability in accounting for both spatial and temporal information; the proposed framework integrating PRM clustering and CART classification provides promising intention prediction performance and is adaptive to different drivers.

Encoding Multiple Sensor Data for Robotic Learning Skills From Multimodal Demonstration

Learning a task such as pushing something, where the constraints of both position and force have to be satisfied, is usually difficult for a collaborative robot. In this work, we propose a multimodal teaching-by-demonstration system which can enable the robot to perform this kind of tasks. The basic idea is to transfer the adaptation of multi-modal information from a human tutor to the robot by taking account of multiple sensor signals (i.e., motion trajectories, stiffness, and force profiles). The human tutor's stiffness is estimated based on the limb surface electromyography (EMG) signals obtained from the demonstration phase. The force profiles in Cartesian space are collected from a force/torque sensor mounted between the robot endpoint and the tool. Subsequently, the hidden semi-Markov model (HSMM) is used to encode the multiple signals in a unified manner. The correlations between position and the other three control variables (i.e., velocity, stiffness and force) are encoded with separate HSMM models. Based on the estimated parameters of the HSMM model, the Gaussian mixture regression (GMR) is then utilized to generate the expected control variables. The learned variables are further mapped into an impedance controller in the joint space through inverse kinematics for the reproduction of the task. Comparative tests have been conducted to verify the effectiveness of our approach on a Baxter robot.

Long Time Sequential Task Learning From Unstructured Demonstrations

Learning from demonstration (LfD), which provides a natural way to transfer skills to robots, has been extensively researched for decades, and an army of methods and applications have been developed and investigated for learning an individual or low-level task. Nevertheless, learning long time sequential tasks is still very difficult as it involves task segmentation and sub-task clustering under an extremely large demonstration variance. Besides, the representation problem should be considered when doing segmentation. This paper presents a new unified framework to solve the problems of segmentation, clustering, and representation in a sequential task. The segmentation algorithm segments unstructured demonstrations into movement primitives (MPs). Then, the MPs are automatically clustered and labeled so that they can be reused in other tasks. Finally, the representation model is leveraged to encode and generalize the learned MPs in new contexts. To achieve the first goal, a change-point detection algorithm based on Bayesian inference is leveraged. It can segment unstructured demonstrations online with minimum prior knowledge requirements. By following the Gaussian distributed assumption in the segmentation model, MPs are encoded by Gaussians or Gaussian mixture models. Thus, the clustering of MPs is formulated as a clustering over cluster (CoC) problem. The Kullback–Leibler divergence is used to measure similarities between MPs, through which the MPs with smaller distance are clustered into the same group. To replay and generalize the task in novel contexts, we use task-parameterized regression models such as the Gaussian mixture regression. We implemented our framework on a sequential open-and-place task. The experiments demonstrate that the segmentation accuracy of our framework can reach 94.3% and the recognition accuracy can reach 97.1%. Comparisons with the state-of-the-art algorithm also indicate that our framework is superior or comparable to their results.

Human Intention Understanding From Multiple Demonstrations and Behavior Generalization in Dynamic Movement Primitives Framework

Human’s interference in the process of skill learning can improve the performance of the robot greatly. However, learning from demonstration to generate a new action with human behavioral characteristics in the varying situation is challenging. Generally, dynamic movement primitives (DMPs) method can generalize the trajectory imitating the demonstration, but cannot integrate the feature of multiple trajectories of different targets. In this paper, the proposed method contains two aspects of learning and generating. The statistical method Gaussian mixture model and Gaussian mixture regression (GMM-GMR) is used to extract the common characteristic and eliminate the uncertainty of the multiple demonstrations. To exert the ability of DMPs to generate a human-like motion to a new goal, and we model the shape parameter with locally weighted regression (LWR) method. To enhance the ability of DMPs in multiple trajectories learning, we propose the multivariate Gaussian process regression (MV-GPR) method to construct the regression model of shape parameters to reflect the human intentions, with respect to the target position. To verify the feasibility of the proposed method, we design a peg-in-hole experiment with proving generalization and obstacle avoidance performance. The results have shown that the strategy integrated the generalization of DMPs and feature regeneration ability of MV-GPR method, and the generated valid trajectory could achieve the peg-in-hole task with 6-DOF whole-arm avoidance.

Soft Sensor Model Development for Cobalt Oxalate Synthesis Process Based on Adaptive Gaussian Mixture Regression

Average particle size is a critical quality variable for optimal operation of the cobalt oxalate synthesis process. However, the measurement for this variable is often achieved from offline laboratory assaying procedure with low sampling rate and low reliability. Therefore, a data-driven soft sensor model based on adaptive Gaussian mixture regression (AGMR) is presented in this paper. Firstly, a GMR based soft sensor model is developed for predicting the average particle size. Secondly, the prediction uncertainty obtained from the GMR model is used to assess the performance of the current soft sensor model. Thirdly, a dual updating algorithm based on the model performance assessment is constructed to track the time-varying behavior of the synthesis process. In the updating method, bias updating and moving window model updating methods are performed in turns based on the results of model performance assessment. The dual updating mechanism can avoid blind updating. Finally, a numerical example and a real industrial cobalt oxalate synthesis process application are used to demonstrate the effectiveness of the proposed method.

Impacts of enquiry-based science teaching on achievement gap between high-and-low SES students: findings from PISA 2015

ABSTRACTThe purpose of this study was to empirically investigate, in a large-scale assessment, the prevalence of enquiry-based teaching, as defined by PISA, implemented in science lessons, and the extent to which it is related to students from different socioeconomic status (SES) family backgrounds in the U.S. The Programme of International Student Assessment (PISA) 2015 data including 5,146 students in the U.S. were used. A Latent Profile Analysis (LPA) was conducted to classify students into different science teaching subgroups. Mixture regression analysis was conducted to examine whether there the association between SES and achievement varied across science teaching subgroups. Four science teaching subgroups were identified, varying in the frequency of and type of teaching practices used. Students’ attitudes, gender, science instruction time, and reading achievement are significant predictors of the subgroup membership. SES-achievement gap also varied across the science teaching subgroups. Discussions for the results and suggestions for future study were presented.