Mixture Of Experts Research Articles

Integrative mixture of experts to combine clinical factors and gene markers.

Motivation: Microarrays are being increasingly used in cancer research to better characterize and classify tumors by selecting marker genes. However, as very few of these genes have been validated as predictive biomarkers so far, it is mostly conventional clinical and pathological factors that are being used as prognostic indicators of clinical course. Combining clinical data with gene expression data may add valuable information, but it is a challenging task due to their categorical versus continuous characteristics. We have further developed the mixture of experts (ME) methodology, a promising approach to tackle complex non-linear problems. Several variants are proposed in integrative ME as well as the inclusion of various gene selection methods to select a hybrid signature.Results: We show on three cancer studies that prediction accuracy can be improved when combining both types of variables. Furthermore, the selected genes were found to be of high relevance and can be considered as potential biomarkers for the prognostic selection of cancer therapy.Availability: Integrative ME is implemented in the R package integrativeME (http://cran.r-project.org/).Contact: k.lecao@uq.edu.auSupplementary information: Supplementary data are available at Bioinformatics online.

Farsi handwritten digit recognition based on mixture of RBF experts

Abstract: In this paper, a new classifier combination model is pre-sented for Farsi handwritten digit recognition. The model is consistedof four RBF neural networks as the experts and another RBF networkas the gating network which learns to split the input space betweenthe experts. Considering the input data, which is an 81-element vectorextracted using the loci characterization method, the gating networkassigns a competence coefficient to each expert. The final output iscomputed as the weighted sum of the outputs of the experts. Therecognition rate of the proposed model is 93.5% which is 3.75% morethan the rate of the mixture of MLPs experts previously ran on thesame database.Keywords: mixture of experts, handwritten digit recognition, locicharacterization methodClassification: Science and engineering for electronics References[1] I. D. Trier and A. K. Jain, “Feature extraction Methods for CharacterRecognition- A Survey,” Pattern Recognition , vol. 29, no. 4, pp. 641–662,1996.[2] L. I. Kuncheva, M. Skurichina, and R. P. W. Duin, “An experimentalStudy on diversity for bagging and boosting with linear classifiers,”

Transient Stability Assessment of a Power System with a UPFC by Mixture of Experts

The power-transfer capability of long transmission lines is usually limited by large signal stability. Economic factors, such as the high cost of long lines and revenue from the delivery of additional power, give strong incentives to explore all economically and technically feasible means of raising the stability limit. The Flexible AC Transmission Systems (FACTS) devices could provide fast control of active and reactive power through a transmission line. We have tried to introduce a new model for transient stability prediction of a power system with a Unified Power Flow Controller (UPFC) to add a contribution to the subject. For this reason we applied so called, Committee Neural Networks (CNNs) methods as tools for Transient Stability Assessment (TSA) of power system. We use the Mixture of Experts (ME) in which, the problem space is divided into several subspaces for the experts, and then the outputs of experts are combined by a gating network to form the final output. In this paper, ME is used to assess the transient stability of power system with a UPFC after faults occur on transmission lines. Simulations were carried out on the IEEE 9-bus with and without a UPFC tests systems considering three phase faults on the systems. The data collected from the time domain simulations are then used as inputs to the ME in which is used as a classifier to determine whether the power systems are stable or unstable.

Improving mixture of experts for view-independent face recognition using teacher-directed learning

In this paper we develop a new learning method, called teacher-directed learning (TDL), for mixture of experts (ME) to perform view-independent face recognition. In the basic form of ME the problem space is automatically divided into several subspaces for the experts, and the outputs of experts are combined by a gating network. In our proposed method, the ME is directed to adapt to a particular partitioning corresponding to predetermined views. To do this, we apply a new learning method to ME, called TDL, in a way that according to the pose of the input training sample, only the weights of the corresponding experts are updated. We apply TDL to MEs, composed of MLP experts and a radial basis function gating network, with different representation schemes: global, single-view and overlapping eigenspace. We test them with previously intermediate unseen views of faces. The experimental results support our claim that directing the experts to a predetermined partitioning of the face space improves the performance of the conventional ME for view-independent face recognition. Comparison with some of the most related methods indicates that the proposed model yields excellent recognition rate in view-independent face recognition.

Applying effective feature selection techniques with hierarchical mixtures of experts for spam classification

E-mail abuse has been steadily increasing during the last decade. E-mail users find themselves targeted by massive quantities of unsolicited bulk e-mail, which often contains offensive language or has fraudulent intentions. Internet Service Providers (ISPs) on the other hand, have to face a considerable system overloading as the incoming mail consumes network and storage resources. Among the plethora of solutions, the most prominent in terms of cost efficiency and complexity are the text filtering approaches. Most of the approaches model the problem using linear statistical models. Despite their popularity - due both to their simplicity and relative ease of interpretation - the non-linearity assumption of data samples is inappropriate in practice. This is mainly due to the inability of other approaches to capture the apparent non-linear relationships, which characterize these samples. In this paper, we propose a margin-based feature selection approach integrated with a Hierarchical Mixtures of Experts (HME) system, which attempts to overcome limitations common to other machine-learning based approaches. By reducing the data dimensionality using effective algorithms for feature selection we evaluated our system with publicly available corpora of e-mails, characterized by very high similarity between legitimate and bulk e-mail (and thus low discriminative potential). We experimented with two different architectures, a hierarchical HME and a perceptron HME. As a result, we confirm the domination of our Spam Filtering (SF) - HME method against other machine learning approaches, which present lesser degree of recall, as well as against traditional rule-based approaches, which lack considerably in the achieved degrees of precision.

Recognition of Persian handwritten digits using Characterization Loci and Mixture of Experts

A method for recognition of Persian handwritten digits based on characterization loci and mixture of experts is proposed. This method utilizes the characterization loci, as the main feature. In the classification stage of our proposed method the mixture of experts are applied. This recognition method is applied to Farsi hand-written digits in the HODA database. The experimental results support our claim that this method improves the performance compare with conventional methods. Evaluating the proposed method with 20000 test samples and ME with 4 experts that each expert has 25 neurons in hidden layer and 5 neurons in hidden layers for gating network the recognition rate of 97.52% is achieved. Comparison with some of the most related methods indicates that the proposed method yields good recognition rate in Persian handwritten digits recognition. .

Differentiation of Two Subtypes of Adult Hydrocephalus by Mixture of Experts

This paper illustrates the use of mixture of experts (ME) network structure to guide model selection for diagnosis of two subtypes of adult hydrocephalus (normal-pressure hydrocephalus-NPH and aqueductal stenosis-AS). The ME is a modular neural network architecture for supervised learning. Expectation-Maximization (EM) algorithm was used for training the ME so that the learning process is decoupled in a manner that fits well with the modular structure. To improve classification accuracy, the outputs of expert networks were combined by a gating network simultaneously trained in order to stochastically select the expert that is performing the best at solving the problem. The classifiers were trained on the defining features of NPH and AS (velocity and flux). Three types of records (normal, NPH and AS) were classified with the accuracy of 95.83% by the ME network structure. The ME network structure achieved accuracy rates which were higher than that of the stand-alone neural network models.

Proton exchange membrane fuel cell modelling based on a mixed moving least squares technique

An accurate model for static and dynamic operation of all components of a fuel cell (FC) system is required in order to optimize its performance. Actually, over the last decades many PEMFC models have been developed with a theoretical or empirical approach. This paper presents a numerical method for fuel cell modelling based on moving least squares method (MLS). This method appears to present a good compromise between response surface accuracy, smoothness, robustness and ease of use. The aim of this work is to develop an adapted PEMFC model, which includes the process variations. Experimental data and analytical equations are used for developing a “mixed” model of the fuel cell (mixture of experts), which mimics closely the real behaviour of the cell. By using the methodology presented in the paper, the PEMFC static characteristics can be approximated and predicted taking into account the reactant pressures, the cell temperature, the membrane humidity, the reactants stoichiometry. An analysis of the results based on the computation of the relative errors and on graphical representations is discussed. The results and the method can be used to analyse and to optimize the fuel cell system operation.

DISAMBIGUATION OF GREEK POLYSEMOUS WORDS USING HIERARCHICAL PROBABILISTIC NETWORKS AND A CHI-SQUARE FEATURE SELECTION STRATEGY

In this paper, we present a robust classification technique to disambiguate Greek polysemous words based on hierarchical probabilistic networks. Assuming that the linguistic data of a polysemous word is classified to equivalent classes according to the number of its senses, we try to disambiguate them by using a hierarchical mixture of experts' probabilistic model — a soft version of neural networks that permits overlapping between classes and which is successfully applied in classification tasks. The model is used in combination with an effective feature selection strategy based on a Chi-square test that enhances disambiguation performance. Due to the absence of previous similar work on Greek linguistic data, for comparison we also implement and apply the very popular naïve Bayes classifier to the same data. Comparing the two systems, we find that the Hierarchical Mixtures-of-Experts (HME) model is superior to the naïve Bayes classifier, mainly because of its ability to permit overlapping and the capture of non-linearity among the data. We believe that the system can be successfully applied on real linguistic data after training on a small amount of data.

Implementing wavelet transform/mixture of experts network for analysis of electrocardiogram beats

Abstract: Mixture of experts (ME) is a modular neural network architecture for supervised learning. This paper illustrates the use of the ME network structure to guide model selection for classification of electrocardiogram (ECG) beats. The expectation maximization algorithm is used for training the ME so that the learning process is decoupled in a manner that fits well with the modular structure. The ECG signals were decomposed into time–frequency representations using discrete wavelet transforms and statistical features were calculated to depict their distribution. The ME network structure was implemented for ECG beats classification using the statistical features as inputs. To improve classification accuracy, the outputs of expert networks were combined by a gating network simultaneously trained in order to stochastically select the expert that is performing the best at solving the problem. Five types of ECG beats (normal beat, congestive heart failure beat, ventricular tachyarrhythmia beat, atrial fibrillation beat, partial epilepsy beat) obtained from the Physiobank database were classified with an accuracy of 96.89% by the ME network structure. The ME network structure achieved accuracy rates which were higher than those of the stand‐alone neural network models.

Eigenvector Methods for Automated Detection of Electrocardiographic Changes in Partial Epileptic Patients

In this paper, the automated diagnostic systems trained on diverse and composite features were presented for detection of electrocardiographic changes in partial epileptic patients. In practical applications of pattern recognition, there are often diverse features extracted from raw data that require recognizing. Methods of combining multiple classifiers with diverse features are viewed as a general problem in various application areas of pattern recognition. Two types (normal and partial epilepsy) of ECG beats (180 records from each class) were obtained from the Physiobank database. The multilayer perceptron neural network (MLPNN), combined neural network (CNN), mixture of experts (ME), and modified mixture of experts (MME) were tested and benchmarked for their performance on the classification of the studied ECG signals, which were trained on diverse or composite features. Decision making was performed in two stages: feature extraction by eigenvector methods and classification using the classifiers trained on the extracted features. The present research demonstrated that the MME trained on the diverse features achieved accuracy rates (total classification accuracy is 99.44%) that were higher than that of the other automated diagnostic systems.

Statistics over features for internal carotid arterial disorders detection

The objective of the present study is to extract the representative features of the internal carotid arterial (ICA) Doppler ultrasound signals and to present the accurate classification model. This paper presented the usage of statistics over the set of the extracted features (Lyapunov exponents and the power levels of the power spectral density estimates obtained by the eigenvector methods) in order to reduce the dimensionality of the extracted feature vectors. Since classification is more accurate when the pattern is simplified through representation by important features, feature extraction and selection play an important role in classifying systems such as neural networks. Mixture of experts (ME) and modified mixture of experts (MME) architectures were formulated and used as basis for detection of arterial disorders. Three types of ICA Doppler signals (Doppler signals recorded from healthy subjects, subjects having stenosis, and subjects having occlusion) were classified. The classification results confirmed that the proposed ME and MME has potential in detecting the arterial disorders.

Diverse and composite features for ECG signals processing

The automated diagnostic systems employing diverse and composite features for electrocardiogram (ECG) signals were analyzed and their accuracies were determined. Because of the importance of making the right decision, classification procedures classifying the ECG signals with high accuracy were investigated. The classification accuracies of multilayer perceptron neural network (MLPNN), recurrent neural network (RNN), and mixture of experts (ME) trained on composite features and modified mixture of experts (MME) trained on diverse features were compared. The inputs of these automated diagnostic systems were composed of diverse or composite features (wavelet coefficients and power levels of the power spectral density estimates obtained by the eigenvector methods) and were chosen according to the network structures. The conclusions of this study demonstrated that the MME trained on diverse features achieved accuracy rates which were higher than that of the other automated diagnostic systems trained on composite features.

An experimental study of the extended NRBF regression model and its enhancement for classification problem

As an extension of the traditional normalized radial basis function (NRBF) model, the extended normalized RBF ( ENRBF) model was proposed by Xu [RBF nets, mixture experts, and Bayesian Ying-Yang learning, Neurocomputing 19 (1998) 223–257]. In this paper, we perform a supplementary study on ENRBF with several properly designed experiments and some further theoretical discussions. It is shown that ENRBF is able to efficiently improve the learning accuracies under some circumstances. Moreover, since the ENRBF model is initially proposed for the regression and function approximation problems, a further step is taken in this work to modify the ENRBF model to deal with the classification problems. Both the original ENRBF model and the new proposed ENRBF classifier (ENRBFC) can be viewed as the special cases of the mixture-of-experts (ME) model that is discussed in Xu et al. [An alternative model for mixtures of experts, in: Advances in Neural Information Processing Systems, MIT Press, Cambridge, MA, 1995]. Experimental results show the potentials of ENRBFC compared to some other related classifiers.

Teacher-directed learning in view-independent face recognition with mixture of experts using overlapping eigenspaces

A model for view-independent face recognition, based on Mixture of Experts, ME, is presented. In the basic form of ME the problem space is automatically divided into several subspaces for the experts, and the outputs of experts are combined by a gating network. In our proposed model, the ME is directed to adapt to a particular partitioning corresponding to predetermined views. To force an expert towards a particular partitioning corresponding to predetermined views, a new representation scheme, overlapping eigenspaces, is introduced, that provides each expert with an eigenspace computed from the faces in the corresponding neighboring views. Furthermore, we use teacher-directed learning, TDL, in a way that according to the pose of the input training sample, only the weights of the corresponding experts are updated. The experimental results support our claim that directing the experts to a predetermined partitioning of the face space improves the performance of the conventional ME for view-independent face recognition. Comparison with some of the most related methods indicates that the proposed model yields excellent recognition rate in view-independent face recognition.

View-independent face recognition with Mixture of Experts

A model for view-independent face recognition, based on Mixture of Experts, ME, is presented. Instead of allowing ME to partition the face space automatically, it is directed to adapt to a particular partitioning corresponding to predetermined views. Experimental results show that this model performs well in recognizing faces of intermediate unseen views. There are neurophysiological evidences that underpin the proposed model, reporting similar mechanisms of pooling the outputs of several view-specific modules to perform view-independent face recognition.

Bioprocess hybrid parametric/nonparametric modelling based on the concept of mixture of experts

This paper presents a novel method for bioprocess hybrid parametric/nonparametric modelling based on mixture of experts (ME) and the expectation maximisation (EM) algorithm. The bioreactor system is described by material balance equations whereas the cell population subsystem is described by an adjustable mixture of parametric/nonparametric sub-models inspired in the ME architecture. This idea was motivated by the fact that cellular metabolism has an inherent “modular” structure, organised in metabolic pathways, with complex interactions. This study was supported by simulations using models of different levels of complexity. The proposed method was compared with the conventional hybrid technique employing the multi-layer perceptron (MLP) and the radial basis function (RBF) networks. As main conclusions it can be stated that MEs trained with the EM algorithm are able to systematically detect metabolic shifts with the individual experts developing expertise in describing the individual pathways. The hybrid ME model with thin-plate spline RBF network as experts outperforms both the hybrid MLP model and the hybrid RBF model in its ability to describe metabolic switches.

Extension of mixture-of-experts networks for binary classification of hierarchical data

For many applied problems in the context of medically relevant artificial intelligence, the data collected exhibit a hierarchical or clustered structure. Ignoring the interdependence between hierarchical data can result in misleading classification. In this paper, we extend the mechanism for mixture-of-experts (ME) networks for binary classification of hierarchical data. Another extension is to quantify cluster-specific information on data hierarchy by random effects via the generalized linear mixed-effects model (GLMM). The extension of ME networks is implemented by allowing for correlation in the hierarchical data in both the gating and expert networks via the GLMM. The proposed model is illustrated using a real thyroid disease data set. In our study, we consider 7652 thyroid diagnosis records from 1984 to early 1987 with complete information on 20 attribute values. We obtain 10 independent random splits of the data into a training set and a test set in the proportions 85% and 15%. The test sets are used to assess the generalization performance of the proposed model, based on the percentage of misclassifications. For comparison, the results obtained from the ME network with independence assumption are also included. With the thyroid disease data, the misclassification rate on test sets for the extended ME network is 8.9%, compared to 13.9% for the ME network. In addition, based on model selection methods described in Section 2, a network with two experts is selected. These two expert networks can be considered as modeling two groups of patients with high and low incidence rates. Significant variation among the predicted cluster-specific random effects is detected in the patient group with low incidence rate. It is shown that the extended ME network outperforms the ME network for binary classification of hierarchical data. With the thyroid disease data, useful information on the relative log odds of patients with diagnosed conditions at different periods can be evaluated. This information can be taken into consideration for the assessment of treatment planning of the disease. The proposed extended ME network thus facilitates a more general approach to incorporate data hierarchy mechanism in network modeling.

Teacher-directed learning in view-independent face recognition with mixture of experts using single-view eigenspaces

Abstract We propose a new model for view-independent face recognition by multiview approach. We use the so-called “mixture of experts”, ME, in which, the problem space is divided into several subspaces for the experts, and the outputs of experts are combined by a gating network. In our model, instead of leaving the ME to partition the face space automatically, the ME is directed to adapt to a particular partitioning corresponding to predetermined views. To force an expert towards a specific view of face, in the representation layer, we provide each expert with its own eigenspace computed from the faces in the corresponding view. Furthermore, we use teacher-directed learning, TDL, in a way that according to the pose of the input training sample, only the weights of the corresponding expert are updated. The experimental results support our claim that directing the experts to a predetermined partitioning of face space improves the performance of the conventional ME for view-independent face recognition. In particular, for 1200 images of unseen intermediate views of faces from 20 subjects, the ME with single-view eigenspaces yields the average recognition rate of 80.51% in 10 trials, which is noticeably increased to 90.29% by applying the TDL method.

Usage of eigenvector methods in implementation of automated diagnostic systems for ECG beats

In this paper, implementation of automated diagnostic systems with diverse and composite features for electrocardiogram (ECG) beats was presented and their accuracies were determined. In practical applications of pattern recognition, there are often diverse features extracted from raw data which needs recognizing. Methods of combining multiple classifiers with diverse features are viewed as a general problem in various application areas of pattern recognition. Because of the importance of making the right decision, better classification procedures were searched for ECG beats. Four types of ECG beats (normal beat, congestive heart failure beat, ventricular tachyarrhythmia beat, atrial fibrillation beat) obtained from the Physiobank database were analyzed. The classification accuracies of six different classifiers, namely multilayer perceptron neural network (MLPNN), combined neural network (CNN), mixture of experts (ME), modified mixture of experts (MME), probabilistic neural network (PNN), and support vector machine (SVM), which were trained on diverse or composite features were compared. Decision making was performed in two stages: feature extraction by eigenvector methods and classification using the classifiers trained on the extracted features. The research demonstrated that the SVM trained on composite feature and the MME trained on diverse features achieved accuracy rates which were higher than that of the other automated diagnostic systems.