Mixture Of Experts Research Articles

Cosmic-ray discrimination capabilities of [formula omitted]– [formula omitted] silicon nuclear telescopes using neural networks

An isotope classifier of cosmic-ray events collected by space detectors has been implemented using a multi-layer perceptron neural architecture. In order to handle a great number of different isotopes a modular architecture of the “mixture of experts” type is proposed. The performance of this classifier has been tested on simulated data and has been compared with a “classical” classifying procedure. The quantitative comparison with traditional techniques shows that the neural approach has classification performances comparable – within 1% – with that of the classical one, with efficiency of the order of 98%. A possible hardware implementation of such a kind of neural architecture in future space missions is considered.

Application of the Mixtures of Experts algorithm for signal processing in a noninvasive glucose monitoring system

The theory of Mixtures of Experts (MOE) [M. Jordan, R. Jacobs, Hierarchical mixtures of experts and the EM algorithm, Neural Computation 6 (2) (1994) 181–214; S.R. Waterhouse, D.J.C. MacKay, et al., in: D.S. Touretzky (Ed.), Bayesian methods for Mixtures of Experts, Advances in Neural Information Processing Systems, Vol. 8, MIT Press, Cambridge, MA, 1996, pp. 351–357; S.R. Waterhouse, Classification and regression using Mixtures of Experts, PhD Thesis, Cambridge University, Cambridge, 1997] was applied to the signal from a noninvasive glucose monitor for the purpose of converting raw signal data into blood glucose values. The MOE algorithm can be described as a generalized predictive method of data analysis. This method uses a superposition of multiple linear regressions, along with a switching algorithm, to predict outcomes. Any number of input/output variables are possible. The unknown coefficients in this method are determined by an optimization technique called the Expectation Maximization (EM) algorithm. The noninvasive GlucoWatch® biographer operation has been described [R.T. Kurnik, B. Berner, et al., Design and simulation of a reverse iontophoretic glucose monitoring device, J. Electrochem. Soc. 145 (12) (1998) 4119–4125]. Briefly, a small electrical current results in the transport of glucose beneath the skin to a hydrogel placed on the skin surface. Within the hydrogel, the glucose reacts with the enzyme glucose-oxidase to produce hydrogen peroxide. This hydrogen peroxide then diffuses to a platinum-based electrode, where it reacts to produce a current. The integral of this current (charge) over the sensing time is the signal used to measure extracted glucose. This process is repeated, yielding up to three measurements per hour. The data used for this analysis were obtained from diabetic subjects wearing the biographer over a 15-h period. The MOE inputs consisted of elapsed time, integrated current, blood glucose value at the calibration point, and a calibrated signal. The output was the value of blood glucose at each measurement. These training data were used to determine the unknown parameters in the MOE by the EM algorithm. Using a 3-h time point for calibrating the biographer, the mean absolute error (MAE) between the actual blood glucose and the blood glucose predicted with the MOE, was 14.4%.

Improved learning algorithms for mixture of experts in multiclass classification

Mixture of experts (ME) is a modular neural network architecture for supervised learning. A double-loop Expectation-Maximization (EM) algorithm has been introduced to the ME architecture for adjusting the parameters and the iteratively reweighted least squares (IRLS) algorithm is used to perform maximization in the inner loop [Jordan, M.I., Jacobs, R.A. (1994). Hierarchical mixture of experts and the EM algorithm, Neural Computation, 6(2), 181–214]. However, it is reported in literature that the IRLS algorithm is of instability and the ME architecture trained by the EM algorithm, where IRLS algorithm is used in the inner loop, often produces the poor performance in multiclass classification. In this paper, the reason of this instability is explored. We find out that due to an implicitly imposed incorrect assumption on parameter independence in multiclass classification, an incomplete Hessian matrix is used in that IRLS algorithm. Based on this finding, we apply the Newton–Raphson method to the inner loop of the EM algorithm in the case of multiclass classification, where the exact Hessian matrix is adopted. To tackle the expensive computation of the Hessian matrix and its inverse, we propose an approximation to the Newton–Raphson algorithm based on a so-called generalized Bernoulli density. The Newton–Raphson algorithm and its approximation have been applied to synthetic data, benchmark, and real-world multiclass classification tasks. For comparison, the IRLS algorithm and a quasi-Newton algorithm called BFGS have also been applied to the same tasks. Simulation results have shown that the use of the proposed learning algorithms avoids the instability problem and makes the ME architecture produce good performance in multiclass classification. In particular, our approximation algorithm leads to fast learning. In addition, the limitation of our approximation algorithm is also empirically investigated in this paper.

On the identifiability of mixtures-of-experts

In mixtures-of-experts (ME) models, “experts” of generalized linear models are combined, according to a set of local weights called the “gating function”. The invariant transformations of the ME probability density functions include the permutations of the expert labels and the translations of the parameters in the gating functions. Under certain conditions, we show that the ME systems are identifiable if the experts are ordered and the gating parameters are initialized. The conditions are validated for Poisson, gamma, normal and binomial experts.

Combining predictors: comparison of five meta machine learning methods

For some years there has been a trend towards combining predictors, and away from monolithic predictors. Two groups of meta machine learning (MML) methods are the ensemble methods and the Mixtures of Experts (ME) methods. In this article, five different representatives are presented, discussed, and compared. Three from the ensemble group, and two from the ME group. The selected methods are Simple ensemble, AdaBoost, Bagging, Hierarchical MEs, and a variation on the ME method found in R.A. Jacobs, M.I. Jordan, S.J. Nowlan, G.E. Hinton (Neural Computation 3 (1) (1991) 79–87), called Dynamic Coefficients ( DynCo). DynCo can use any type of predictors that can be trained with gradient descent. It has a powerful combination method, and it encourages cooperation among the experts. DynCo compares favorably with the other four methods.

Organization of face and object recognition in modular neural network models

There is strong evidence that face processing in the brain is localized. The double dissociation between prosopagnosia, a face recognition deficit occurring after brain damage, and visual object agnosia, difficulty recognizing other kinds of complex objects, indicates that face and non-face object recognition may be served by partially independent neural mechanisms. In this paper, we use computational models to show how the face processing specialization apparently underlying prosopagnosia and visual object agnosia could be attributed to (1) a relatively simple competitive selection mechanism that, during development, devotes neural resources to the tasks they are best at performing, (2) the developing infant's need to perform subordinate classification (identification) of faces early on, and (3) the infant's low visual acuity at birth. Inspired by de Schonen, Mancini and Liegeois' arguments (1998) [de Schonen, S., Mancini, J., Liegeois, F. (1998). About functional cortical specialization: the development of face recognition. In: F. Simon & G. Butterworth, The development of sensory, motor, and cognitive capacities in early infancy (pp. 103–116). Hove, UK: Psychology Press] that factors like these could bias the visual system to develop a processing subsystem particularly useful for face recognition, and Jacobs and Kosslyn's experiments (1994) [Jacobs, R. A., & Kosslyn, S. M. (1994). Encoding shape and spatial relations—the role of receptive field size in coordination complementary representations. Cognitive Science, 18(3), 361–368] in the mixtures of experts (ME) modeling paradigm, we provide a preliminary computational demonstration of how this theory accounts for the double dissociation between face and object processing. We present two feed-forward computational models of visual processing. In both models, the selection mechanism is a gating network that mediates a competition between modules attempting to classify input stimuli. In Model I, when the modules are simple unbiased classifiers, the competition is sufficient to achieve enough of a specialization that damaging one module impairs the model's face recognition more than its object recognition, and damaging the other module impairs the model's object recognition more than its face recognition. However, the model is not completely satisfactory because it requires a search of parameter space. With Model II, we explore biases that lead to more consistent specialization. We bias the modules by providing one with low spatial frequency information and the other with high spatial frequency information. In this case, when the model's task is subordinate classification of faces and superordinate classification of objects, the low spatial frequency network shows an even stronger specialization for faces. No other combination of tasks and inputs shows this strong specialization. We take these results as support for the idea that something resembling a face processing “module” could arise as a natural consequence of the infant's developmental environment without being innately specified.

An improved neural classification network for the two-group problem

In this paper we present the neural network model known as the mixture-of-experts (MOE) and determine its accuracy and its robustness. We do this by comparing the classification accuracy of MOE, backpropagation neural network (BPN), Fisher’s discriminant analysis, logistics regression, k nearest neighbor, and the kernel density on five real-world two-group data sets. Our results lead to three major conclusions: (1) the MOE network architecture is more accurate than BPN; (2) MOE tends to be more accurate than the parametric and non-parametric methods investigated; (3) MOE is a far more robust classifier than the other methods for the two-group problem. Scope and purpose The two-group classification problem is the assignment of objects to one of two predetermined groups. These classification decisions are routinely made in business, health care, and government. Classification errors result when an object is assigned to the wrong group. A healthy patient might be diagnosed with cancer, or a patient suffering from cancer might be diagnosed as healthy. A firm assigned the wrong bond rating will incur additional cost of capital of millions of dollars. Unfortunately, there is no classification method that is the “best” for all data sets. Some classification methods will do well for a given data set only to perform poorly on another. What is needed, therefore, is a robust methodology that classifies accurately across a wide range of two-group data sets. Researchers and practitioners have turned to neural classification models in a quest for a robust method. The purpose of this research is to investigate the accuracy and robust nature of the mixture-of-experts neural network (MOE). Our research demonstrates that MOE performs very well against back-propagation neural networks and four traditional statistical classification procedures based upon results from five diverse real world data sets. We conclude that the mixture-of-experts neural network may be the robust classification scheme being sought for use with the two-group problem.

正規化ガウス関数ネットワーク，Ｍｉｘｔｕｒｅ ｏｆ ｅｘｐｅｒｔｓとＥＭアルゴリズム

正規化ガウス関数ネットワーク，Ｍｉｘｔｕｒｅ ｏｆ ｅｘｐｅｒｔｓとＥＭアルゴリズム

Constructive incremental learning from only local information

We introduce a constructive, incremental learning system for regression problems that models data by means of spatially localized linear models. In contrast to other approaches, the size and shape of the receptive field of each locally linear model, as well as the parameters of the locally linear model itself, are learned independently, that is, without the need for competition or any other kind of communication. Independent learning is accomplished by incrementally minimizing a weighted local cross-validation error. As a result, we obtain a learning system that can allocate resources as needed while dealing with the bias-variance dilemma in a principled way. The spatial localization of the linear models increases robustness toward negative interference. Our learning system can be interpreted as a nonparametric adaptive bandwidth smoother, as a mixture of experts where the experts are trained in isolation, and as a learning system that profits from combining independent expert knowledge on the same problem. This article illustrates the potential learning capabilities of purely local learning and offers an interesting and powerful approach to learning with receptive fields.

RBF nets, mixture experts, and Bayesian Ying–Yang learning

The connections of the alternative model for mixture of experts (ME) to the normalized radial basis function (NRBF) nets and extended normalized RBF (ENRBF) nets are established, and the well-known expectation-maximization (EM) algorithm for maximum likelihood learning is suggested to the two types of RBF nets. This new learning technique determines the parameters of the input layer (including the covariance matrices or so-called receptive fields) and the parameters of the output layer in the RBF nets globally, instead of separately training the input layer by the K-means algorithm and the output layer by the least-squares learning as done in most of the existing RBF learning methods. In addition, coordinated competitive learning (CCL) and adaptive algorithms are proposed to approximate the EM algorithm for considerably speeding up the learning of the original and alternative ME models as well as the NRBF and ENRBF nets. Furthermore, the two ME models are linked to the recent proposed Bayesian Ying–Yang (BYY) learning system and theory such that not only the architecture of ME and RBF nets is shown to be more preferred than multilayer architecture, but also a new model selection criterion has been obtained to determine the number of experts and basis functions. A number of experiments are made on the prediction of foreign exchange rate and trading investment as well as piecewise nonlinear regression and piecewise line fitting. As shown in these experiments, the EM algorithm for NRBF nets and ENRBF nets obviously outperforms the conventional RBF learning technique, CCL speeds up the learning considerably with only a slight sacrifice on performance accuracy, the adaptive algorithm gives significant improvements on financial predication and trading investment, as well as that the proposed criterion can select the number of basis functions successfully. In addition, the ENRBF net and the alternative ME model are also shown to be able to implement curve fitting and detection.

Competitive principal component analysis for locally stationary time series

A new unsupervised algorithm is proposed that performs competitive principal component analysis (PCA) of a time series. A set of expert PCA networks compete, through the mixture of experts (MOE) formalism, on the basis of their ability to reconstruct the original signal. The resulting network finds an optimal projection of the input onto a reduced dimensional space as a function of the input and, hence, of time. As a byproduct, the time series is both segmented and identified according to stationary regions. Examples showing the performance of the algorithm are included.

Deterministic annealing for clustering, compression, classification, regression, and related optimization problems

The deterministic annealing approach to clustering and its extensions has demonstrated substantial performance improvement over standard supervised and unsupervised learning methods in a variety of important applications including compression, estimation, pattern recognition and classification, and statistical regression. The application-specific cost is minimized subject to a constraint on the randomness of the solution, which is gradually lowered. We emphasize the intuition gained from analogy to statistical physics. Alternatively the method is derived within rate-distortion theory, where the annealing process is equivalent to computation of Shannon's rate-distortion function, and the annealing temperature is inversely proportional to the slope of the curve. The basic algorithm is extended by incorporating structural constraints to allow optimization of numerous popular structures including vector quantizers, decision trees, multilayer perceptrons, radial basis functions, and mixtures of experts.

Now Comes the Time to Defuzzify Neuro-Fuzzy Models

Fuzzy models present a singular Janus-faced, On the one hand, they are knowledge-based software environments constructed from a collection of linguistic IF-THEN rules, and on the other hand, they realize nonlinear mappings which have interesting mathematical properties like ”low-order interpolation”, “smooth cooperation between local approximators” and”universal function approximation”. Within this second vision, fuzzy models can be taken as additional members in the large family of multi-expert networks which already count as members, Radial Basis Functions, GRNN, CMAC, B-splines network, Locally Weighted Learning or Regression, Kernel Regression Estimator, Jordan and Jacob’s mixture of experts, etc. Here we will focus on this second vision trying to point out what remains original in the fuzzy approach as compared with the other members, then describing some learning strategies of these fuzzy models and presenting comparative experimental results on a classical time series prediction benchmark.

Modular decomposition in visuomotor learning.

The principle of 'divide-and-conquer' the decomposition of a complex task into simpler subtasks each learned by a separate module, has been proposed as a computational strategy during learning. We explore the possibility that the human motor system uses such a modular decomposition strategy to learn the visuomotor map, the relationship between visual inputs and motor outputs. Using a virtual reality system, subjects were exposed to opposite prism-like visuomotor remappings-discrepancies between actual and visually perceived hand locations- for movements starting from two distinct locations. Despite this conflicting pairing between visual and motor space, subjects learned the two starting-point-dependent visuomotor mappings and the generalization of this learning to intermediate starting locations demonstrated an interpolation of the two learned maps. This interpolation was a weighted average of the two learned visuomotor mappings, with the weighting sigmoidally dependent on starting location, a prediction made by a computational model of modular learning known as the "mixture of experts". These results provide evidence that the brain may employ a modular decomposition strategy during learning.

Bias/variance analyses of mixtures-of-experts architectures.

This article investigates the bias and variance of mixtures-of-experts (ME) architectures. The variance of an ME architecture can be expressed as the sum of two terms: the first term is related to the variances of the expert networks that comprise the architecture and the second term is related to the expert networks' covariances. One goal of this article is to study and quantify a number of properties of ME architectures via the metrics of bias and variance. A second goal is to clarify the relationships between this class of systems and other systems that have recently been proposed. It is shown that in contrast to systems that produce unbiased experts whose estimation errors are uncorrelated, ME architectures produce biased experts whose estimates are negatively correlated.

Speaker identification using time-delay HMEs.

In this paper, we extend the Hierarchical Mixture of Experts (HME) to temporal processing and explore it for a substantial problem, that of text-dependent speaker identification. For a specific multiway classification, we propose a generalized Bernoulli density instead of the multinomial logit density to avoid the instability during training. Time-delay technique is applied for spatio-temporal processing in the HME and a combining scheme is presented for combining multiple time-delay HMEs in order to complete a multi-scale analysis for the temporal data. Using the time-delay HME along with the EM algorithm as well as the combination of multiple time-delay HMEs, the speaker identification system has a good performance and yields significantly fast training. We have also addressed some issues about the time-delay techniques in the HME.

A modified HME architecture for text-dependent speaker identification

A modified hierarchical mixtures of experts (HME) architecture is presented for text-dependent speaker identification. A new gating network is introduced to the original HME architecture for the use of instantaneous and transitional spectral information in text-dependent speaker identification. The statistical model underlying the proposed architecture is presented and learning is treated as a maximum likelihood problem; in particular, an expectation-maximization (EM) algorithm is also proposed for adjusting the parameters of the proposed architecture. An evaluation has been carried out using a database of isolated digit utterances by 10 male speakers. Experimental results demonstrate that the proposed architecture outperforms the original HME architecture in text-dependent speaker identification.

Reinforcement learning of multiple tasks using a hierarchical CMAC architecture

Abstract A reinforcement learning approach based on modular function approximation is presented. Cerebellar Model Articulation Controller (CMAC) networks are incorporated in the Hierarchical Mixtures of Experts (HME) architecture and the resulting architecture is referred to as HME-CMAC. A computationally efficient on-line learning algorithm based on the Expectation Maximization (EM) algorithm is proposed in order to achieve fast function approximation with the HME-CMAC architecture. The Compositional Q-Learning (CQ-L) framework establishes the relationship between the Q-values of composite tasks and those of elemental tasks in its decomposition. This framework is extended here to allow rewards in non-terminal states. An implementation of the extended CQ-L framework using the HME-CMAC architecture is used to perform task decomposition in a realistic simulation of a two-linked manipulator having non-linear dynamics. The context-dependent reinforcement learning achieved by adopting this approach has advantages over monolithic approaches in terms of speed of learning, storage requirements and the ability to cope with changing goals.

Information geometry of the EM and em algorithms for neural networks

To realize an input-output relation given by noise-contaminated examples, it is effective to use a stochastic model of neural networks. When the model network includes hidden units whose activation values are not specified nor observed, it is useful to estimate the hidden variables from the observed or specified input-output data based on the stochastic model. Two algorithms, the EM and em algorithms, have so far been proposed for this purpose. The EM algorithm is an iterative statistical technique of using the conditional expectation, and the em algorithm is a geometrical one given by information geometry. The em algorithm minimizes iteratively the Kullback-Leibler divergence in the manifold of neural networks. These two algorithms are equivalent in most cases. The present paper gives a unified information geometrical framework for studying stochastic models of neural networks, by focusing on the EM and em algorithms, and proves a condition that guarantees their equivalence. Examples include: (1) stochastic multilayer perceptron, (2) mixtures of experts, and (3) normal mixture model.

Statistical Physics, Mixtures of Distributions, and the EM Algorithm

We show that there are strong relationships between approaches to optmization and learning based on statistical physics or mixtures of experts. In particular, the EM algorithm can be interpreted as converging either to a local maximum of the mixtures model or to a saddle point solution to the statistical physics system. An advantage of the statistical physics approach is that it naturally gives rise to a heuristic continuation method, deterministic annealing, for finding good solutions.