Automatic Relevance Determination Research Articles

A novel prediction method of creep rupture life of 9–12% chromium ferritic steel based on abductive network

The creep rupture life of 9–12% chromium ferritic steel is predicted as a function of alloy composition, creep stress, and creep temperature. A database is made up from data in previous publications. A novel abductive network is constructed to predict creep rupture life. With a four-layer architecture, the network shows a precise prediction of creep rupture life of 9–12% chromium ferritic steel. Performance is examined and compared with backpropagation algorithm (BP) neural network. Results indicate that the proposed approach is more accurate than Larson–Miller parametric method and more efficient than that of BP neural network. Automatic relevance determination reveals that the influence of Cr and W on creep rupture life of 9–12% chromium ferritic steel are the greatest amongst the alloying elements.

Generalized spike-and-slab priors for Bayesian group feature selection using expectation propagation

We describe a Bayesian method for group feature selection in linear regression problems. The method is based on a generalized version of the standard spike-and-slab prior distribution which is often used for individual feature selection. Exact Bayesian inference under the prior considered is infeasible for typical regression problems. However, approximate inference can be carried out efficiently using Expectation Propagation (EP). A detailed analysis of the generalized spike-and-slab prior shows that it is well suited for regression problems that are sparse at the group level. Furthermore, this prior can be used to introduce prior knowledge about specific groups of features that are a priori believed to be more relevant. An experimental evaluation compares the performance of the proposed method with those of group LASSO, Bayesian group LASSO, automatic relevance determination and additional variants used for group feature selection. The results of these experiments show that a model based on the generalized spike-and-slab prior and the EP algorithm has state-of-the-art prediction performance in the problems analyzed. Furthermore, this model is also very useful to carry out sequential experimental design (also known as active learning), where the data instances that are most informative are iteratively included in the training set, reducing the number of instances needed to obtain a particular level of prediction accuracy.

Bayesian Sparse Estimation Using Double Lomax Priors

Sparsity-promoting prior along with Bayesian inference is an effective approach in solving sparse linear models (SLMs). In this paper, we first introduce a new sparsity-promoting prior coined as Double Lomax prior, which corresponds to a three-level hierarchical model, and then we derive a full variational Bayesian (VB) inference procedure. When noninformative hyperprior is assumed, we further show that the proposed method has one more latent variable than the canonical automatic relevance determination (ARD). This variable has a smoothing effect on the solution trajectories, thus providing improved convergence performance. The effectiveness of the proposed method is demonstrated by numerical simulations including autoregressive (AR) model identification and compressive sensing (CS) problems.

Source reconstruction accuracy of MEG and EEG Bayesian inversion approaches.

Electro- and magnetoencephalography allow for non-invasive investigation of human brain activation and corresponding networks with high temporal resolution. Still, no correct network detection is possible without reliable source localization. In this paper, we examine four different source localization schemes under a common Variational Bayesian framework. A Bayesian approach to the Minimum Norm Model (MNM), an Empirical Bayesian Beamformer (EBB) and two iterative Bayesian schemes (Automatic Relevance Determination (ARD) and Greedy Search (GS)) are quantitatively compared. While EBB and MNM each use a single empirical prior, ARD and GS employ a library of anatomical priors that define possible source configurations. The localization performance was investigated as a function of (i) the number of sources (one vs. two vs. three), (ii) the signal to noise ratio (SNR; 5 levels) and (iii) the temporal correlation of source time courses (for the cases of two or three sources). We also tested whether the use of additional bilateral priors specifying source covariance for ARD and GS algorithms improved performance. Our results show that MNM proves effective only with single source configurations. EBB shows a spatial accuracy of few millimeters with high SNRs and low correlation between sources. In contrast, ARD and GS are more robust to noise and less affected by temporal correlations between sources. However, the spatial accuracy of ARD and GS is generally limited to the order of one centimeter. We found that the use of correlated covariance priors made no difference to ARD/GS performance.

Evidence for Phenotypic Plasticity among Multihost Campylobacter jejuni and C. coli Lineages, Obtained Using Ribosomal Multilocus Sequence Typing and Raman Spectroscopy

Closely related bacterial isolates can display divergent phenotypes. This can limit the usefulness of phylogenetic studies for understanding bacterial ecology and evolution. Here, we compare phenotyping based on Raman spectrometric analysis of cellular composition to phylogenetic classification by ribosomal multilocus sequence typing (rMLST) in 108 isolates of the zoonotic pathogens Campylobacter jejuni and C. coli. Automatic relevance determination (ARD) was used to identify informative peaks in the Raman spectra that could be used to distinguish strains in taxonomic and host source groups (species, clade, clonal complex, and isolate source/host). Phenotypic characterization based on Raman spectra showed a degree of agreement with genotypic classification using rMLST, with segregation accuracy between species (83.95%), clade (in C. coli, 98.41%), and, to some extent, clonal complex (86.89% C. jejuni ST-21 and ST-45 complexes) being achieved. This confirmed the utility of Raman spectroscopy for lineage classification and the correlation between genotypic and phenotypic classification. In parallel analysis, relatively distantly related isolates (different clonal complexes) were assigned the correct host origin irrespective of the clonal origin (74.07 to 96.97% accuracy) based upon different Raman peaks. This suggests that the phenotypic characteristics, from which the phenotypic signal is derived, are not fixed by clonal descent but are influenced by the host environment and change as strains move between hosts.

PREDICTIVE MODELING WITH MISSING DATA USING AN AUTOMATIC RELEVANCE DETERMINATION ENSEMBLE: A COMPARATIVE STUDY

The objective of this article is to present an automatic relevance determination ensemble as an effective variable extraction method for insurance datasets with large numbers of variables. Automatic relevance determination is a method that uses a Bayesian neural network and the evidence framework to rank variables in the order of relevance to the target variable. The current approach uses a single Bayesian neural network that searches only for local minima or maxima. In large datasets with numerous variables, this is a concern because we cannot be certain that the outcome is an optimal one. The method used to address this issue in this study is an automatic relevance determination ensemble with various configurations (or structures) of the Bayesian neural networks. Each outcome in the ensemble is determined by using a confidence factor rather than by scrutinizing the most probable weights values or hyperparameters directly. The extraction method is used with the repeated incremental pruning to produce error reduction, logistic discriminant analysis, and k-nearest neighbor models to evaluate the performance. Furthermore, the datasets employed contain escalating missing data to measure the accuracy and resilience of the models when they are used with the proposed ensemble. The ensemble is compared with the principal component analysis method. The results show that with the automatic relevance determination ensemble, the models achieve higher accuracies in performance than when used with the principal component analysis. Furthermore, the resilience and strength of models is higher when using the ensemble, compared with the principal component analysis method.

Incremental Reformulated Automatic Relevance Determination

In this work, the relationship between the incremental version of sparse Bayesian learning (SBL) with automatic relevance determination (ARD)-a fast marginal likelihood maximization (FMLM) algorithm-and a recently proposed reformulated ARD scheme is established. The FMLM algorithm is an incremental approach to SBL with ARD, where the corresponding objective function-the marginal likelihood-is optimized with respect to the parameters of a single component provided that the other parameters are fixed; the corresponding maximizer is computed in closed form, which enables a very efficient SBL realization. Wipf and Nagarajan have recently proposed a reformulated ARD (R-ARD) approach, which optimizes the marginal likelihood using auxiliary upper bounding functions. The resulting algorithm is then shown to correspond to a series of reweighted l1-constrained convex optimization problems. This correspondence establishes and analyzes the relationship between the FMLM and R-ARD schemes. Specifically, it is demonstrated that the FMLM algorithm realizes an incremental approach to the optimization of the R-ARD objective function. This relationship allows deriving the R-ARD pruning conditions similar to those used in the FMLM scheme to analytically detect components that are to be removed from the model, thus regulating the estimated signal sparsity and accelerating the algorithm convergence.

Prediction of daily global solar irradiation data using Bayesian neural network: A comparative study

This paper presents a comparative study between Bayesian Neural Network (BNN), classical Neural Network (NN) and empirical models for estimating the daily global solar irradiation (DGSR). An experimental meteorological database from 1998 to 2002 at Al-Madinah (Saudi Arabia) has been used. Four input parameters have been employed: air temperature, relative humidity, sunshine duration and extraterrestrial irradiation. Automatic relevance determination (ARD) method has investigated in order to select the optimum input parameters of the NN. Results show that the BNN performs better that other NN structures and empirical models.

Hub-centered gene network reconstruction using automatic relevance determination.

Network inference deals with the reconstruction of biological networks from experimental data. A variety of different reverse engineering techniques are available; they differ in the underlying assumptions and mathematical models used. One common problem for all approaches stems from the complexity of the task, due to the combinatorial explosion of different network topologies for increasing network size. To handle this problem, constraints are frequently used, for example on the node degree, number of edges, or constraints on regulation functions between network components. We propose to exploit topological considerations in the inference of gene regulatory networks. Such systems are often controlled by a small number of hub genes, while most other genes have only limited influence on the network's dynamic. We model gene regulation using a Bayesian network with discrete, Boolean nodes. A hierarchical prior is employed to identify hub genes. The first layer of the prior is used to regularize weights on edges emanating from one specific node. A second prior on hyperparameters controls the magnitude of the former regularization for different nodes. The net effect is that central nodes tend to form in reconstructed networks. Network reconstruction is then performed by maximization of or sampling from the posterior distribution. We evaluate our approach on simulated and real experimental data, indicating that we can reconstruct main regulatory interactions from the data. We furthermore compare our approach to other state-of-the art methods, showing superior performance in identifying hubs. Using a large publicly available dataset of over 800 cell cycle regulated genes, we are able to identify several main hub genes. Our method may thus provide a valuable tool to identify interesting candidate genes for further study. Furthermore, the approach presented may stimulate further developments in regularization methods for network reconstruction from data.

Bayesian Support Vector Regression With Automatic Relevance Determination Kernel for Modeling of Antenna Input Characteristics

The modeling of microwave antennas and devices typically requires that non-linear input-output mappings be determined between a set of variable parameters (such as geometry dimensions and frequency), and the corresponding scattering parameter(s). Support vector regression (SVR) employing an isotropic Gaussian kernel has been widely used for such tasks; this kernel has one tunable hyperparameter that can be optimized (along with the penalty constant C) using a standard procedure that involves a parameter grid search combined with cross-validation. The isotropic kernel however suffers from limited expressiveness, and might provide inadequate predictive accuracy for nonlinear mappings that involve multiple tunable input variables. The present study shows that Bayesian support vector regression using the inherently more flexible Gaussian kernel with automatic relevance determination (ARD) is eminently suitable for highly non-linear modeling tasks, such as the input reflection coefficient magnitude |S11| of broadband and ultrawideband antennas. The Bayesian framework enables efficient training of the multiple kernel ARD hyperparameters-a task that would be computationally infeasible for the grid search/cross-validation approach of standard SVR.

Multi-objective learning of Relevance Vector Machine classifiers with multi-resolution kernels

The Relevance Vector Machine (RVM) is a sparse classifier in which complexity is controlled with the Automatic Relevance Determination prior. However, sparsity is dependent on kernel choice and severe over-fitting can occur.We describe multi-objective evolutionary algorithms (MOEAs) which optimise RVMs allowing selection of the best operating true and false positive rates and complexity from the Pareto set of optimal trade-offs. We introduce several cross-validation methods for use during evolutionary optimisation. Comparisons on benchmark datasets using multi-resolution kernels show that the MOEAs can locate markedly sparser RVMs than the standard, with comparable accuracies.

A sparse nonparametric hierarchical Bayesian approach towards inductive transfer for preference modeling

In this paper, we present a novel methodology for preference learning based on the concept of inductive transfer. Specifically, we introduce a nonparametric hierarchical Bayesian multitask learning approach, based on the notion that human subjects may cluster together forming groups of individuals with similar preference rationale (but not identical preferences). Our approach is facilitated by the utilization of a Dirichlet process prior, which allows for the automatic inference of the most appropriate number of subject groups (clusters), as well as the employment of the automatic relevance determination (ARD) mechanism, giving rise to a sparse nature for our model, which significantly enhances its computational efficiency. We explore the efficacy of our novel approach by applying it to both a synthetic experiment and a real-world music recommendation application. As we show, our approach offers a significant enhancement in the effectiveness of knowledge transfer in statistical preference learning applications, being capable of correctly inferring the actual number of human subject groups in a modeled dataset, and limiting knowledge transfer only to subjects belonging to the same group (wherein knowledge transferability is more likely).

PSO-based automatic relevance determination and feature selection system for hyperspectral image classification

Based on the particle swarm optimisation algorithm, an ARD-FS system integrating the automatic relevance determination (ARD) and the feature selection (FS) for a Gaussian process classifier (GPC) is proposed. The ARD-FS system aims to reduce the complexity of the GPC and simultaneously improve the classification accuracy. It is applied to the hyperspectral images and experimental results demonstrate its good performance for hyperspectral image classification even with very limited labelled samples but high-dimensional features.

Predicting the shear resistance of RC beams without shear reinforcement using a Bayesian neural network

Advances in neural computing have shown that a neural learning approach that uses Bayesian inference can essentially eliminate the problem of over fitting, which is common with conventional back-propagation neural networks. In addition, Bayesian neural network can provide the confidence (error) associated with its prediction. This paper presents the application of Bayesian learning to train a multilayer perceptron network to predict the shear resistance of reinforced concrete beams without shear reinforcement. The automatic relevance determination technique was employed to assess the relative importance of the different input variables considered in this study on the shear resistance of reinforced concrete beams. The performance of the Bayesian neural network is examined and discussed along with that of current shear design provisions.

Fast Variational Sparse Bayesian Learning With Automatic Relevance Determination for Superimposed Signals

In this work, a new fast variational sparse Bayesian learning (SBL) approach with automatic relevance determination (ARD) is proposed. The sparse Bayesian modeling, exemplified by the relevance vector machine (RVM), allows a sparse regression or classification function to be constructed as a linear combination of a few basis functions. It is demonstrated that, by computing the stationary points of the variational update expressions with noninformative (ARD) hyperpriors, a fast version of variational SBL can be constructed. Analysis of the computed stationary points indicates that SBL with Gaussian sparsity priors and noninformative hyperpriors corresponds to removing components with signal-to-noise ratio below a 0 dB threshold; this threshold can also be adjusted to significantly improve the convergence rate and sparsity of SBL. It is demonstrated that the pruning conditions derived for fast variational SBL coincide with those obtained for fast marginal likelihood maximization; moreover, the parameters that maximize the variational lower bound also maximize the marginal likelihood function. The effectiveness of fast variational SBL is demonstrated with synthetic as well as with real data.

Partial logistic relevance vector machines in survival analysis

The use of relevance vector machines to flexibly model hazard rate functions is explored. This technique is adapted to survival analysis problems through the partial logistic approach. The method exploits the Bayesian automatic relevance determination procedure to obtain sparse solutions and it incorporates the flexibility of kernel-based models. Example results are presented on literature data from a head-and-neck cancer survival study using Gaussian and spline kernels. Sensitivity analysis is conducted to assess the influence of hyperprior distribution parameters. The proposed method is then contrasted with other flexible hazard regression methods, in particular the HARE model proposed by Kooperberg et al. [16]. A simulation study is conducted to carry out the comparison. The model developed in this paper exhibited good performance in the prediction of hazard rate. The application of this sparse Bayesian technique to a real cancer data set demonstrated that the proposed method can potentially reveal characteristics of the hazards, associated with the dynamics of the studied diseases, which may be missed by existing modeling approaches based on different perspectives on the bias vs. variance balance.

Bayesian Nonparametric Inference of Switching Dynamic Linear Models

Many complex dynamical phenomena can be effectively modeled by a system that switches among a set of conditionally linear dynamical modes. We consider two such models: the switching linear dynamical system (SLDS) and the switching vector autoregressive (VAR) process. Our Bayesian nonparametric approach utilizes a hierarchical Dirichlet process prior to learn an unknown number of persistent, smooth dynamical modes. We additionally employ automatic relevance determination to infer a sparse set of dynamic dependencies allowing us to learn SLDS with varying state dimension or switching VAR processes with varying autoregressive order. We develop a sampling algorithm that combines a truncated approximation to the Dirichlet process with efficient joint sampling of the mode and state sequences. The utility and flexibility of our model are demonstrated on synthetic data, sequences of dancing honey bees, the IBOVESPA stock index, and a maneuvering target tracking application.

Post hoc Bayesian model selection

This note describes a Bayesian model selection or optimization procedure for post hoc inferences about reduced versions of a full model. The scheme provides the evidence (marginal likelihood) for any reduced model as a function of the posterior density over the parameters of the full model. It rests upon specifying models through priors on their parameters, under the assumption that the likelihood remains the same for all models considered. This provides a quick and efficient scheme for scoring arbitrarily large numbers of models, after inverting a single (full) model. In turn, this enables the selection among discrete models that are distinguished by the presence or absence of free parameters, where free parameters are effectively removed from the model using very precise shrinkage priors. An alternative application of this post hoc model selection considers continuous model spaces, defined in terms of hyperparameters (sufficient statistics) of the prior density over model parameters. In this instance, the prior (model) can be optimized with respect to its evidence. The expressions for model evidence become remarkably simple under the Laplace (Gaussian) approximation to the posterior density. Special cases of this scheme include Savage–Dickey density ratio tests for reduced models and automatic relevance determination in model optimization. We illustrate the approach using general linear models and a more complicated nonlinear state-space model.

Product Form Feature Selection for Mobile Phone Design Using LS-SVR and ARD

In the product design field, it is important to pin point critical product form features (PFFs) that influence consumers' affective responses (CARs) of a product design. In this paper, an approach based on least squares support vector regression (LS-SVR) and automatic relevance determination (ARD) is proposed to streamline the task of product form feature selection (PFFS) according to the CAR data. The representation of PFFs is determined by morphological analysis and pairwise adjectives are used to express CARs. In order to gather the CAR data, an experiment of semantic differential (SD) evaluation on collected product samples was conducted. The LS-SVR prediction model can be constructed using the PFFs as input data and the evaluated SD scores as output value. The optimal parameters of the LS-SVR model are tuned by using Bayesian inference. Finally, an ARD selection process is used to analyze the relative relevance of PFFs to obtain feature ranking.

Detection of Hidden Structures in Nonstationary Spike Trains

We propose an algorithm for simultaneously estimating state transitions among neural states and nonstationary firing rates using a switching state-space model (SSSM). This algorithm enables us to detect state transitions on the basis of not only discontinuous changes in mean firing rates but also discontinuous changes in the temporal profiles of firing rates (e.g., temporal correlation). We construct estimation and learning algorithms for a nongaussian SSSM, whose nongaussian property is caused by binary spike events. Local variational methods can transform the binary observation process into a quadratic form. The transformed observation process enables us to construct a variational Bayes algorithm that can determine the number of neural states based on automatic relevance determination. Additionally, our algorithm can estimate model parameters from single-trial data using a priori knowledge about state transitions and firing rates. Synthetic data analysis reveals that our algorithm has higher performance for estimating nonstationary firing rates than previous methods. The analysis also confirms that our algorithm can detect state transitions on the basis of discontinuous changes in temporal correlation, which are transitions that previous hidden Markov models could not detect. We also analyze neural data recorded from the medial temporal area. The statistically detected neural states probably coincide with transient and sustained states that have been detected heuristically. Estimated parameters suggest that our algorithm detects the state transitions on the basis of discontinuous changes in the temporal correlation of firing rates. These results suggest that our algorithm is advantageous in real-data analysis.