Mixture Of Experts Framework Research Articles

Towards overcoming data scarcity in materials science: unifying models and datasets with a mixture of experts framework

While machine learning has emerged in recent years as a useful tool for the rapid prediction of materials properties, generating sufficient data to reliably train models without overfitting is often impractical. Towards overcoming this limitation, we present a general framework for leveraging complementary information across different models and datasets for accurate prediction of data-scarce materials properties. Our approach, based on a machine learning paradigm called mixture of experts, outperforms pairwise transfer learning on 14 of 19 materials property regression tasks, performing comparably on four of the remaining five. The approach is interpretable, model-agnostic, and scalable to combining an arbitrary number of pre-trained models and datasets to any downstream property prediction task. We anticipate the performance of our framework will further improve as better model architectures, new pre-training tasks, and larger materials datasets are developed by the community.

Parsimonious mixture‐of‐experts based on mean mixture of multivariate normal distributions

The mixture‐of‐experts (MoE) paradigm attempts to learn complex models by combining several “experts” via probabilistic mixture models. Each expert in the MoE model handles a small area of the data space in which a gating function controls the data‐to‐expert assignment. The MoE framework has been used extensively in designing non‐linear models in machine learning and statistics to model the heterogeneity in data for the purpose of regression, classification and clustering. The existing MoE of multi‐target regression (MoE‐MTR) models for continuous data is based on multivariate normal distributions. However, in many practical situations, for a set of data, a group or groups of observations may exhibit asymmetric and heavy‐tailed behaviour, and inference based on symmetric distributions in such situations can unduly affect the fit of the regression model. We introduce here a novel robust multivariate non‐normal MoE model by the use of mean mixture of normal distributions. The proposed model can handle the issues of MoE‐MTR models regarding possibly skewed, heavy‐tailed and noisy data. Maximum likelihood estimates of model parameters are developed based on an expectation‐maximization (EM)‐type algorithm. Parsimony is also obtained by imposing suitable constraints on the expert dispersion matrices. The usefulness of the proposed methodology is illustrated using simulated and real data sets.

Mixture of Experts Approach for Piecewise Modeling and Linearization of RF Power Amplifiers

Piecewise behavioral models are commonly adopted for modeling and linearization of RF power amplifiers (PAs) that exhibit strong amplitude-dependent nonlinear distortion characteristics, as global polynomial approximations tend to underperform in such scenarios. In this article, we consider a new piecewise model for PAs based on the mixture of experts (ME) approach, which builds on a probabilistic model that allows the different submodels to cooperate—as opposed to operating in an independent fashion that is commonly the case in existing reference methods. We first introduce the ME framework theory while also extend it such that it can be applied to model complex baseband signals and nonlinearities. Then, we show how the ME model allows overcoming some of the intrinsic shortcomings that existing piecewise behavioral models commonly exhibit, which translates into improved modeling accuracy and improved linearization performance. Furthermore, the extension of the ME approach to a tree-structured regression model, referred to as the hierarchical ME model, is also introduced and shown to provide further performance improvements over the basic ME approach. The proposed solutions are validated with extensive RF measurements, covering both PA direct modeling and digital predistortion (DPD)-based linearization, on a gallium nitride (GaN) load-modulated balanced PA, on a GaN Doherty PA, and on a class AB GaN high electron mobility transistor PA, while being compared against several state-of-the-art piecewise methods. The results demonstrate that the ME framework-based models outperform the state of the art.

mvClaim : an R package for multivariate general insurance claims severity modelling

Abstract The mvClaim package in R provides flexible modelling frameworks for multivariate insurance claim severity modelling. The current version of the package implements a parsimonious mixture of experts (MoE) model family with bivariate gamma distributions, as introduced in Hu et al., and a finite mixture of copula regressions within the MoE framework as in Hu & O’Hagan. This paper presents the modelling approach theory briefly and the usage of the models in the package in detail. This package is hosted on GitHub at https://github.com/senhu/.

Efficient bifurcation and tabulation of multi-dimensional combustion manifolds using deep mixture of experts: An a priori study

This work describes and validates an approach for autonomously bifurcating turbulent combustion manifolds to divide regression tasks amongst specialized artificial neural networks (ANNs). This approach relies on the mixture of experts (MoE) framework, where each neural network is trained to be specialized in a given portion of the input space. The assignment of different input regions to the experts is determined by a gating network, which is a neural network classifier. In some previous studies [1–4], it has been demonstrated that bifurcation of a complex combustion manifold and fitting different ANNs for each part leads to better fits or faster inference speeds. However, the manner of bifurcation in these studies was based on heuristic approaches or clustering techniques. In contrast, the proposed technique enables automatic bifurcation using non-linear planes in high-dimensional turbulent combustion manifolds that are often associated with complex behavior due to different dominating physics in various zones. The proposed concept is validated using 4-dimensional (4D) and 5D flamelet tables, showing that the errors obtained with a given network size, or conversely the network size required to achieve a given accuracy, is considerably reduced. The effect of the number of experts on inference speed is also investigated, showing that by increasing the number of experts from 1 to 8, the inference time can be approximately reduced by a factor of two. Moreover, it is shown that the MoE approach divides the input manifold in a physically intuitive manner, suggesting that the MoE framework can elucidate high-dimensional datasets in a physically meaningful way.

A unified approach to universal prediction: generalized upper and lower bounds.

We study sequential prediction of real-valued, arbitrary, and unknown sequences under the squared error loss as well as the best parametric predictor out of a large, continuous class of predictors. Inspired by recent results from computational learning theory, we refrain from any statistical assumptions and define the performance with respect to the class of general parametric predictors. In particular, we present generic lower and upper bounds on this relative performance by transforming the prediction task into a parameter learning problem. We first introduce the lower bounds on this relative performance in the mixture of experts framework, where we show that for any sequential algorithm, there always exists a sequence for which the performance of the sequential algorithm is lower bounded by zero. We then introduce a sequential learning algorithm to predict such arbitrary and unknown sequences, and calculate upper bounds on its total squared prediction error for every bounded sequence. We further show that in some scenarios, we achieve matching lower and upper bounds, demonstrating that our algorithms are optimal in a strong minimax sense such that their performances cannot be improved further. As an interesting result, we also prove that for the worst case scenario, the performance of randomized output algorithms can be achieved by sequential algorithms so that randomized output algorithms do not improve the performance.

Modelling and understanding the hierarchy in a mixture of experts using multiple catchment descriptors

• Extends specification of a hierarchical mixture of experts (HME) catchment modelling framework. • Presents basis for choosing a HME gating function knowing only general catchment attributes. • Validates the HME gating function classification framework using data from 50 gauged catchments. Modelling with a mixture of experts represent a platform for probabilistic combination of responses from multiple hydrologic models, thereby better expressing the uncertainty associated with the use of a single stand-alone model structure. In hydrology, the mixture of experts framework has previously been applied successfully and demonstrated to address modelling uncertainty and improving the goodness-of-fit with respect to the observed runoff ( Jeremiah et al., 2013 , Marshall et al., 2006 , Marshall et al., 2007b ). The key to successful reduction in model uncertainty through the mixture of experts architecture lies in the specification of the associated gating function. The gating function models calculate the probability of selecting each component using a range of plausible predictor variables. The predictors used in this function include derived or modelled indicators of the catchment state that force the different component model forms to be used. A previous article by the authors assessed the advantages of identifying multiple predictor variables in the gating function. The present study takes this further, by attempting to relate the gating function and associated predictor variables to measurable catchment attributes. This is performed using data for over 50 catchments in Australia, the result being a classification of gating function complexity and formulation as a function of one or more catchment attributes. Formulation of the gating function using this classification enables users to specify the mixture of experts architecture over ungauged models, through a transposition of the model from gauged catchments via the classification proposed.

A mixture of experts latent position cluster model for social network data

Social network data represent the interactions between a group of social actors. Interactions between colleagues and friendship networks are typical examples of such data. The latent space model for social network data locates each actor in a network in a latent (social) space and models the probability of an interaction between two actors as a function of their locations. The latent position cluster model extends the latent space model to deal with network data in which clusters of actors exist — actor locations are drawn from a finite mixture model, each component of which represents a cluster of actors. A mixture of experts model builds on the structure of a mixture model by taking account of both observations and associated covariates when modeling a heterogeneous population. Herein, a mixture of experts extension of the latent position cluster model is developed. The mixture of experts framework allows covariates to enter the latent position cluster model in a number of ways, yielding different model interpretations. Estimates of the model parameters are derived in a Bayesian framework using a Markov Chain Monte Carlo algorithm. The algorithm is generally computationally expensive — surrogate proposal distributions which shadow the target distributions are derived, reducing the computational burden. The methodology is demonstrated through an illustrative example detailing relationships between a group of lawyers in the USA.

Product of Gaussians for speech recognition

Recently, there has been interest in the use of classifiers based on the product of experts (PoE) framework. PoEs offer an alternative to the standard mixture of experts (MoE) framework. It may be viewed as examining the intersection of a series of experts, rather than the union as in the MoE framework. This paper presents a particular implementation of PoEs, the normalised product of Gaussians (PoG). Here, each expert is a Gaussian mixture model. In this work, the PoG model is presented within a hidden Markov model framework. This allows the classification of variable length data, such as speech data. Training and initialisation procedures are described for this PoG system. The relationship of the PoG system with other schemes, including covariance modeling schemes, is also discussed. In addition the scheme is shown to be related to a standard speech recognition approach, multiple stream systems. The PoG system performance is examined on an automatic speech recognition task, Switchboard. The performance is compared to standard Gaussian mixture systems and multiple stream systems.

Structurally adaptive modular networks for nonstationary environments

This paper introduces a neural network capable of dynamically adapting its architecture to realize time variant nonlinear input-output maps. This network has its roots in the mixture of experts framework but uses a localized model for the gating network. Modules or experts are grown or pruned depending on the complexity of the modeling problem. The structural adaptation procedure addresses the model selection problem and typically leads to much better parameter estimation. Batch mode learning equations are extended to obtain on-line update rules enabling the network to model time varying environments. Simulation results are presented throughout the paper to support the proposed techniques.