Mixture Of Experts Model Research Articles

Choices and pitfalls concerning mixture-of-experts modeling

This paper deals with some important decisions that ought to be made when building a mixture-of-experts model (MEM). Such decisions are related to aspects such as the clustering method and the gating functions used in the model. Depending on how these decisions are made, different mixtures might be formed, yielding different results. In the present study, we investigate the way such decisions affect the performance of MEM's, when using statistical models to regression problem. The famous Boston housing data is used as illustration for the technique

A Bayesian mixture approach to modeling spatial activation patterns in multisite fMRI data.

We propose a probabilistic model for analyzing spatial activation patterns in multiple functional magnetic resonance imaging (fMRI) activation images such as repeated observations on an individual or images from different individuals in a clinical study. Instead of taking the traditional approach of voxel-by-voxel analysis, we directly model the shape of activation patterns by representing each activation cluster in an image as a Gaussian-shaped surface. We assume that there is an unknown true template pattern and that each observed image is a noisy realization of this template. We model an individual image using a mixture of experts model with each component representing a spatial activation cluster. Taking a nonparametric Bayesian approach, we use a hierarchical Dirichlet process to extract common activation clusters from multiple images and estimate the number of such clusters automatically. We further extend the model by adding random effects to the shape parameters to allow for image-specific variation in the activation patterns. Using a Bayesian framework, we learn the shape parameters for both image-level activation patterns and the template for the set of images by sampling from the posterior distribution of the parameters. We demonstrate our model on a dataset collected in a large multisite fMRI study.

Mixture of experts models to exploit global sequence similarity on biomolecular sequence labeling

BackgroundIdentification of functionally important sites in biomolecular sequences has broad applications ranging from rational drug design to the analysis of metabolic and signal transduction networks. Experimental determination of such sites lags far behind the number of known biomolecular sequences. Hence, there is a need to develop reliable computational methods for identifying functionally important sites from biomolecular sequences.ResultsWe present a mixture of experts approach to biomolecular sequence labeling that takes into account the global similarity between biomolecular sequences. Our approach combines unsupervised and supervised learning techniques. Given a set of sequences and a similarity measure defined on pairs of sequences, we learn a mixture of experts model by using spectral clustering to learn the hierarchical structure of the model and by using bayesian techniques to combine the predictions of the experts. We evaluate our approach on two biomolecular sequence labeling problems: RNA-protein and DNA-protein interface prediction problems. The results of our experiments show that global sequence similarity can be exploited to improve the performance of classifiers trained to label biomolecular sequence data.ConclusionThe mixture of experts model helps improve the performance of machine learning methods for identifying functionally important sites in biomolecular sequences.

Bioprocess hybrid parametric/nonparametric modelling based on the concept of mixture of experts

This paper presents a novel method for bioprocess hybrid parametric/nonparametric modelling based on mixture of experts (ME) and the expectation maximisation (EM) algorithm. The bioreactor system is described by material balance equations whereas the cell population subsystem is described by an adjustable mixture of parametric/nonparametric sub-models inspired in the ME architecture. This idea was motivated by the fact that cellular metabolism has an inherent “modular” structure, organised in metabolic pathways, with complex interactions. This study was supported by simulations using models of different levels of complexity. The proposed method was compared with the conventional hybrid technique employing the multi-layer perceptron (MLP) and the radial basis function (RBF) networks. As main conclusions it can be stated that MEs trained with the EM algorithm are able to systematically detect metabolic shifts with the individual experts developing expertise in describing the individual pathways. The hybrid ME model with thin-plate spline RBF network as experts outperforms both the hybrid MLP model and the hybrid RBF model in its ability to describe metabolic switches.

Using knowledge partitioning to investigate the psychological plausibility of mixtures of experts

Over the years, the presence of knowledge partitioning (KP) in human function learning data has been used to argue that mixture-of-experts models (MOE) constitute a psychologically plausible explanation of human performance, and that the experts used by humans are always linear. These claims recently led to the proposition of the population of linear experts model (POLE). In this paper, variations of the firefighting paradigm developed by Lewandowsky and his colleagues, which initiated research about KP, were used to explore the psychological plausibility of MOE in general and POLE in particular. In a first experiment, these statements were tested by modifying the test display of the firefighting paradigm. The results showed that adding irrelevant information to the display resulted in a smaller proportion of partitioning participants. Also, some participants used non-linear experts to partition the stimulus space. This new type of KP was further explored in a second study, which included more training sessions. The results suggest that linear KP disappears with practice and that non-linear partitioning reflects the incapacity to correctly estimate the position of the function's vertex. It is concluded that MOE are adequate psychological models, but that the linearity and ubiquity claims of the POLE model need to be weakened.

Hybridizing mixtures of experts with support vector machines: Investigation into nonlinear dynamic systems identification

Mixture of experts (ME) models comprise a family of modular neural network architectures aiming at distilling complex problems into simple subtasks. This is done by deploying a separate gating module for softly dividing the input space into overlapping regions to be each assigned to one or more expert networks. Conversely, support vector machines (SVMs) refer to kernel-based methods, neural-network-alike models that constitute an approximate implementation of the structural risk minimization principle. Such learning machines follow the simple, but powerful idea of nonlinearly mapping input data into high-dimensional feature spaces wherein a linear decision surface discriminating different regions is properly designed. In this work, we formally characterize and empirically evaluate a novel approach, named as Mixture of Support Vector Machine Experts (MSVME), whose main purpose is to combine the complementary properties of both SVM and ME models. In the formal characterization, an algorithm based on a maximum likelihood criterion is considered for the MSVME training, and we demonstrate that it is possible to train each expert based on an SVM perspective. Regarding the empirical evaluation, simulation results involving nonlinear dynamic system identification problems are reported, contrasting the performance shown by the MSVME approach with that exhibited by conventional SVM and ME models.

A novel mixture of experts model based on cooperative coevolution

Combining several suitable neural networks can enhance the generalization performance of the group when compared to a single network alone. However, it remains a largely open question, how best to build a suitable combination of individuals. Jacobs and his colleagues proposed the mixture of experts (ME) model, in which a set of neural networks are trained together with a gate network. This tight coupling mechanism enables the system to (i) encourage diversity between the individual neural networks by specializing them in different regions of the input space and (ii) allow for a “good” combination weights of the ensemble members to emerge by training the gate, which computes the dynamic weights together with the classifiers. In this paper, we have wrapped a cooperative coevolutionary (CC) algorithm around the basic ME model. This CC layer allows better exploration of the weight space, and hence, an ensemble with better performance. The results show that CCME is better on average than the original ME on a number of classification problems. We have also introduced a novel mechanism for visualizing the modular structures that emerged from the model.

A mixture of experts committee machine to design compensators for intensity modulated radiation therapy

This paper presents a new algorithm to produce a near optimal mixture of experts model (MEM) architecture for a continuous mapping. The MEM is applied to a new method incorporating photon scatter for designing compensators for intensity modulated radiation therapy. The algorithm utilizes the fuzzy C-means clustering algorithm to partition data before training commences. A reduction in the size of training sets also allows the Levenberg–Marquardt algorithm to be implemented. As a result, both training time and validation error are reduced. A 71% reduction in prediction error compared with that of a single neural network is achieved.

EEG signal classification using wavelet feature extraction and a mixture of expert model

Mixture of experts (ME) is modular neural network architecture for supervised learning. A double-loop Expectation-Maximization (EM) algorithm has been introduced to the ME network structure for detection of epileptic seizure. The detection of epileptiform discharges in the EEG is an important component in the diagnosis of epilepsy. EEG signals were decomposed into the frequency sub-bands using discrete wavelet transform (DWT). Then these sub-band frequencies were used as an input to a ME network with two discrete outputs: normal and epileptic. In order to improve accuracy, the outputs of expert networks were combined according to a set of local weights called the ''gating function''. The invariant transformations of the ME probability density functions include the permutations of the expert labels and the translations of the parameters in the gating functions. The performance of the proposed model was evaluated in terms of classification accuracies and the results confirmed that the proposed ME network structure has some potential in detecting epileptic seizures. The ME network structure achieved accuracy rates which were higher than that of the stand-alone neural network model.

A regularized minimum cross-entropy algorithm on mixtures of experts for time series prediction and curve detection

The well-known mixtures of experts (ME) model has been used in many different areas to account for nonlinearities and other complexities in the data, such as time series prediction. We usually train ME model by expectation maximization (EM) algorithm for maximum likelihood learning. However, the number of experts has to be determined first, which is often hardly known. Derived from regularization theory, a regularized minimum cross-entropy (RMCE) algorithm is proposed to train ME model, which can automatically make model selection. When time series is modeled by ME, it is demonstrated by some climate prediction experiments that RMCE algorithm outperforms EM algorithm. We also compare RMCE algorithm with other regression methods such as back-propagation (BP) and normalized radial basis function (NRBF) networks, and find that RMCE algorithm shows promising results. Moreover, we investigate curve detection problem by ME model with RMCE algorithm, which can detect curves (straight lines or circles) from a binary image. Some simulations and image experiments show that RMCE algorithm can automatically determine the number of straight lines or circles during parameter learning against noise, and in this way our algorithm does better than Hough transform (HT).

Hierarchical Text Categorization Using Neural Networks

This paper presents the design and evaluation of a text categorization method based on the Hierarchical Mixture of Experts model. This model uses a divide and conquer principle to define smaller categorization problems based on a predefined hierarchical structure. The final classifier is a hierarchical array of neural networks. The method is evaluated using the UMLS Metathesaurus as the underlying hierarchical structure, and the OHSUMED test set of MEDLINE records. Comparisons with an optimized version of the traditional Rocchio's algorithm adapted for text categorization, as well as flat neural network classifiers are provided. The results show that the use of the hierarchical structure improves text categorization performance with respect to an equivalent flat model. The optimized Rocchio algorithm achieves a performance comparable with that of the hierarchical neural networks.

Best harmony, unified RPCL and automated model selection for unsupervised and supervised learning on Gaussian mixtures, three-layer nets and ME-RBF-SVM models.

After introducing the fundamentals of BYY system and harmony learning, which has been developed in past several years as a unified statistical framework for parameter learning, regularization and model selection, we systematically discuss this BYY harmony learning on systems with discrete inner-representations. First, we shown that one special case leads to unsupervised learning on Gaussian mixture. We show how harmony learning not only leads us to the EM algorithm for maximum likelihood (ML) learning and the corresponding extended KMEAN algorithms for Mahalanobis clustering with criteria for selecting the number of Gaussians or clusters, but also provides us two new regularization techniques and a unified scheme that includes the previous rival penalized competitive learning (RPCL) as well as its various variants and extensions that performs model selection automatically during parameter learning. Moreover, as a by-product, we also get a new approach for determining a set of 'supporting vectors' for Parzen window density estimation. Second, we shown that other special cases lead to three typical supervised learning models with several new results. On three layer net, we get (i) a new regularized ML learning, (ii) a new criterion for selecting the number of hidden units, and (iii) a family of EM-like algorithms that combines harmony learning with new techniques of regularization. On the original and alternative models of mixture-of-expert (ME) as well as radial basis function (RBF) nets, we get not only a new type of criteria for selecting the number of experts or basis functions but also a new type of the EM-like algorithms that combines regularization techniques and RPCL learning for parameter learning with either least complexity nature on the original ME model or automated model selection on the alternative ME model and RBF nets. Moreover, all the results for the alternative ME model are also applied to other two popular nonparametric statistical approaches, namely kernel regression and supporting vector machine. Particularly, not only we get an easily implemented approach for determining the smoothing parameter in kernel regression, but also we get an alternative approach for deciding the set of supporting vectors in supporting vector machine.

Time-series forecasting through wavelets transformation and a mixture of expert models

This paper describes a system formed by a mixture of expert models (MEM) for time-series forecasting. We deal with several different competing models, such as partial least squares, K-nearest neighbours and carbon copy. The input space, after changing its base using the Haar wavelets transform, is partitioned into disjoint regions by a clustering algorithm. For each region, a benchmark is performed among the different competing models aiming at selecting the most adequate one. MEM has improved the forecast performance when compared with the single models as experimentally demonstrated through two different time series: laser data and exchange rate data.

Data mining for features using scale-sensitive gated experts

Introduces a tool for exploratory data analysis and data mining called scale-sensitive gated experts (SSGE) which can partition a complex nonlinear regression surface into a set of simpler surfaces (which we call features). The set of simpler surfaces has the property that each element of the set can be efficiently modeled by a single feedforward neural network. The degree to which the regression surface is partitioned is controlled by an external scale parameter. The SSGE consists of a nonlinear gating network and several competing nonlinear experts. Although SSGE is similar to the mixture of experts model of Jacobs et al. (1991) the mixture of experts model gives only one partitioning of the input-output space, and thus a single set of features, whereas the SSGE gives the user the capability to discover families of features. One obtains a new member of the family of features for each setting of the scale parameter. We derive the scale-sensitive gated experts and demonstrate its performance on a time series segmentation problem. The main results are: (1) the scale parameter controls the granularity of the features of the regression surface, (2) similar features are modeled by the same expert and different kinds of features are modeled by different experts, and (3) for the time series problem, the SSGE finds different regimes of behavior, each with a specific and interesting interpretation.

Error bounds for functional approximation and estimation using mixtures of experts

We examine some mathematical aspects of learning unknown mappings with the mixture of experts model (MEM). Specifically, we observe that the MEM is at least as powerful as a class of neural networks, in a sense that will be made precise. Upper bounds on the approximation error are established for a wide class of target functions. The general theorem states that /spl par/f-f/sub n//spl par//sub p//spl les/c/n/sup r/d/ for f/spl isin/W/sub p//sup r/(L) (a Sobolev class over [-1,1]/sup d/), and f/sub n/ belongs to an n-dimensional manifold of normalized ridge functions. The same bound holds for the MEM as a special case of the above. The stochastic error, in the context of learning from independent and identically distributed (i.i.d.) examples, is also examined. An asymptotic analysis establishes the limiting behavior of this error, in terms of certain pseudo-information matrices. These results substantiate the intuition behind the MEM, and motivate applications.

RBF nets, mixture experts, and Bayesian Ying–Yang learning

The connections of the alternative model for mixture of experts (ME) to the normalized radial basis function (NRBF) nets and extended normalized RBF (ENRBF) nets are established, and the well-known expectation-maximization (EM) algorithm for maximum likelihood learning is suggested to the two types of RBF nets. This new learning technique determines the parameters of the input layer (including the covariance matrices or so-called receptive fields) and the parameters of the output layer in the RBF nets globally, instead of separately training the input layer by the K-means algorithm and the output layer by the least-squares learning as done in most of the existing RBF learning methods. In addition, coordinated competitive learning (CCL) and adaptive algorithms are proposed to approximate the EM algorithm for considerably speeding up the learning of the original and alternative ME models as well as the NRBF and ENRBF nets. Furthermore, the two ME models are linked to the recent proposed Bayesian Ying–Yang (BYY) learning system and theory such that not only the architecture of ME and RBF nets is shown to be more preferred than multilayer architecture, but also a new model selection criterion has been obtained to determine the number of experts and basis functions. A number of experiments are made on the prediction of foreign exchange rate and trading investment as well as piecewise nonlinear regression and piecewise line fitting. As shown in these experiments, the EM algorithm for NRBF nets and ENRBF nets obviously outperforms the conventional RBF learning technique, CCL speeds up the learning considerably with only a slight sacrifice on performance accuracy, the adaptive algorithm gives significant improvements on financial predication and trading investment, as well as that the proposed criterion can select the number of basis functions successfully. In addition, the ENRBF net and the alternative ME model are also shown to be able to implement curve fitting and detection.