Mixture Of Experts Architecture Research Articles

Accelerating Chemical Kinetics Calculations With Physics Informed Neural Networks

Abstract Detailed chemical kinetics calculations can be very computationally expensive, and so various approaches have been used to speed up combustion calculations. Deep neural networks (DNNs) are one promising approach that has seen significant development recently. Standard DNNs, however, do not necessarily follow physical constraints such as conservation of mass. Physics Informed Neural Networks (PINNs) are a class of neural networks that have physical laws embedded within the training process to create networks that follow those physical laws. A new PINN-based DNN approach to chemical kinetics modeling has been developed to make sure mass fraction predictions adhere to the conservation of atomic species. The approach also utilizes a mixture-of-experts (MOE) architecture where the data is distributed on multiple subnetworks followed by a softmax selective layer. The MOE architecture allows the different subnetworks to specialize in different thermochemical regimes, such as early stage ignition reactions or post-flame equilibrium chemistry, then the softmax layer smoothly transitions between the subnetwork predictions. This modeling approach was applied to the prediction of methane-air combustion using the GRI-Mech 3.0 as the reference mechanism. The training database was composed of data from 0D ignition delay simulations under initial conditions of 0.2–50 bar pressure, 500–2000 K temperature, an equivalence ratio between 0 and 2, and an N2-dilution percentage of up to 50%. A wide variety of network sizes and architectures of between 3 and 20 subnetworks and 6,600 to 77,000 neurons were tested. The resulting networks were able to predict 0D combustion simulations with similar accuracy and atomic mass conservation as standard kinetics solvers while having a 10-50× speedup in online evaluation time using CPUs, and on average over 200× when using a GPU.

Anatomical and functional plasticity in early blind individuals and the mixture of experts architecture.

As described elsewhere in this special issue, recent advances in neuroimaging over the last decade have led to a rapid expansion in our knowledge of anatomical and functional correlations within the normal and abnormal human brain. Here, we review how early blindness has been used as a model system for examining the role of visual experience in the development of anatomical connections and functional responses. We discuss how lack of power in group comparisons may provide a potential explanation for why extensive anatomical changes in cortico-cortical connectivity are not observed. Finally we suggest a framework—cortical specialization via hierarchical mixtures of experts—which offers some promise in reconciling a wide range of functional and anatomical data.

Asymptotic Convergence Properties of the EM Algorithm for Mixture of Experts

Mixture of experts (ME) is a modular neural network architecture for supervised classification. The double-loop expectation-maximization (EM) algorithm has been developed for learning the parameters of the ME architecture, and the iteratively reweighted least squares (IRLS) algorithm and the Newton-Raphson algorithm are two popular schemes for learning the parameters in the inner loop or gating network. In this letter, we investigate asymptotic convergence properties of the EM algorithm for ME using either the IRLS or Newton-Raphson approach. With the help of an overlap measure for the ME model, we obtain an upper bound of the asymptotic convergence rate of the EM algorithm in each case. Moreover, we find that for the Newton approach as a specific Newton-Raphson approach to learning the parameters in the inner loop, the upper bound of asymptotic convergence rate of the EM algorithm locally around the true solution Θ* is [Formula: see text], where ϵ>0 is an arbitrarily small number, o(x) means that it is a higher-order infinitesimal as x → 0, and e(Θ*) is a measure of the average overlap of the ME model. That is, as the average overlap of the true ME model with large sample tends to zero, the EM algorithm with the Newton approach to learning the parameters in the inner loop tends to be asymptotically superlinear. Finally, we substantiate our theoretical results by simulation experiments.

A multigradient algorithm using a mixture of experts architecture for land cover classification of multisensor images

An algorithm for supervised classification of multisensor images is proposed. The mixture of experts (ME) architecture with dynamic weight allocation is used for multiclass classification. Here the classification is treated as a maximum likelihood problem and the synaptic weights of the expert network and gating network are updated by a stochastic multigradient approach. Data from an optical sensor with four bands and a synthetic aperture radar (SAR) image of the same scene has been fused and classified. The algorithm is compared to some other advanced training algorithms in the literature for the same image data.

Bioprocess hybrid parametric/nonparametric modelling based on the concept of mixture of experts

This paper presents a novel method for bioprocess hybrid parametric/nonparametric modelling based on mixture of experts (ME) and the expectation maximisation (EM) algorithm. The bioreactor system is described by material balance equations whereas the cell population subsystem is described by an adjustable mixture of parametric/nonparametric sub-models inspired in the ME architecture. This idea was motivated by the fact that cellular metabolism has an inherent “modular” structure, organised in metabolic pathways, with complex interactions. This study was supported by simulations using models of different levels of complexity. The proposed method was compared with the conventional hybrid technique employing the multi-layer perceptron (MLP) and the radial basis function (RBF) networks. As main conclusions it can be stated that MEs trained with the EM algorithm are able to systematically detect metabolic shifts with the individual experts developing expertise in describing the individual pathways. The hybrid ME model with thin-plate spline RBF network as experts outperforms both the hybrid MLP model and the hybrid RBF model in its ability to describe metabolic switches.

Linear Dynamic Models With Mixture of Experts Architecture for Recognition of Speech Under Additive Noise Conditions

This letter presents a new approach to enhance speech feature estimation in the log spectral domain under noisy environments. A mixture of linear dynamic models with an architecture similar to the so-called mixture of experts (ME) is investigated to describe the clean speech feature distribution parametrically. Switching Kalman filters are adapted to the proposed model, and they estimate the clean speech components by means of a generalized pseudo-Bayesian (GPB) algorithm. Experimental results suggest that compared with previous methods, the proposed approach can be more powerful to compensate the noisy speech features for robust speech recognition

The VC dimension for mixtures of binary classifiers.

The mixtures-of-experts (ME) methodology provides a tool of classification when experts of logistic regression models or Bernoulli models are mixed according to a set of local weights. We show that the Vapnik-Chervonenkis dimension of the ME architecture is bounded below by the number of experts m and above by O(m4s2), where s is the dimension of the input. For mixtures of Bernoulli experts with a scalar input, we show that the lower bound m is attained, in which case we obtain the exact result that the VC dimension is equal to the number of experts.

Improved learning algorithms for mixture of experts in multiclass classification

Mixture of experts (ME) is a modular neural network architecture for supervised learning. A double-loop Expectation-Maximization (EM) algorithm has been introduced to the ME architecture for adjusting the parameters and the iteratively reweighted least squares (IRLS) algorithm is used to perform maximization in the inner loop [Jordan, M.I., Jacobs, R.A. (1994). Hierarchical mixture of experts and the EM algorithm, Neural Computation, 6(2), 181–214]. However, it is reported in literature that the IRLS algorithm is of instability and the ME architecture trained by the EM algorithm, where IRLS algorithm is used in the inner loop, often produces the poor performance in multiclass classification. In this paper, the reason of this instability is explored. We find out that due to an implicitly imposed incorrect assumption on parameter independence in multiclass classification, an incomplete Hessian matrix is used in that IRLS algorithm. Based on this finding, we apply the Newton–Raphson method to the inner loop of the EM algorithm in the case of multiclass classification, where the exact Hessian matrix is adopted. To tackle the expensive computation of the Hessian matrix and its inverse, we propose an approximation to the Newton–Raphson algorithm based on a so-called generalized Bernoulli density. The Newton–Raphson algorithm and its approximation have been applied to synthetic data, benchmark, and real-world multiclass classification tasks. For comparison, the IRLS algorithm and a quasi-Newton algorithm called BFGS have also been applied to the same tasks. Simulation results have shown that the use of the proposed learning algorithms avoids the instability problem and makes the ME architecture produce good performance in multiclass classification. In particular, our approximation algorithm leads to fast learning. In addition, the limitation of our approximation algorithm is also empirically investigated in this paper.

Bias/variance analyses of mixtures-of-experts architectures.

This article investigates the bias and variance of mixtures-of-experts (ME) architectures. The variance of an ME architecture can be expressed as the sum of two terms: the first term is related to the variances of the expert networks that comprise the architecture and the second term is related to the expert networks' covariances. One goal of this article is to study and quantify a number of properties of ME architectures via the metrics of bias and variance. A second goal is to clarify the relationships between this class of systems and other systems that have recently been proposed. It is shown that in contrast to systems that produce unbiased experts whose estimation errors are uncorrelated, ME architectures produce biased experts whose estimates are negatively correlated.

Competition and Multiple Cause Models

If different causes can interact on any occasion to generate a set of patterns, then systems modeling the generation have to model the interaction too. We discuss a way of combining multiple causes that is based on the Integrated Segmentation and Recognition architecture of Keeler et al. (1991). It is more cooperative than the scheme embodied in the mixture of experts architecture, which insists that just one cause generate each output, and more competitive than the noisy-or combination function, which was recently suggested by Saund (1994a,b). Simulations confirm its efficacy.