Automatic Relevance Determination Research Articles

Bayesian Learning Using Automatic Relevance Determination Prior with an Application to Earthquake Early Warning

A novel method of Bayesian learning with automatic relevance determination prior is presented that provides a powerful approach to problems of classification based on data features, for example, classifying soil liquefaction potential based on soil and seismic shaking parameters, automatically classifying the damage states of a structure after severe loading based on features of its dynamic response, and real-time classification of earthquakes based on seismic signals. After introduction of the theory, the method is illustrated by applying it to an earthquake record dataset from nine earthquakes to build an efficient real-time algorithm for near-source versus far-source classification of incoming seismic ground motion signals. This classification is needed in the development of early warning systems for large earthquakes. It is shown that the proposed methodology is promising since it provides a classifier with higher correct classification rates and better generalization performance than a previous Bayesian learning method with a fixed prior distribution that was applied to the same classification problem.

Subclassification and Individual Survival Time Prediction from Gene Expression Data of Neuroblastoma Patients by Using CASPAR

To predict individual survival times for neuroblastoma patients from gene expression data using the cancer survival prediction using automatic relevance determination (CASPAR) algorithm. A first set of oligonucleotide microarray gene expression profiles comprising 256 neuroblastoma patients was generated. Then, CASPAR was combined with a leave-one-out cross-validation to predict individual times for both the whole cohort and subgroups of patients with unfavorable markers, including stage 4 disease (n = 67), unfavorable genetic alterations, intermediate-risk or high-risk stratification by the German neuroblastoma trial, and patients predicted as unfavorable by a recently described gene expression classifier (n = 83). Prediction accuracy of individual survival times was assessed by Kaplan-Meier analyses and time-dependent receiver operator characteristics curve analyses. Subsequently, classification results were validated in an independent cohort (n = 120). CASPAR separated patients with divergent outcome in both the initial and the validation cohort [initial set, 5y-OS 0.94 +/- 0.04 (predicted long survival) versus 0.38 +/- 0.17 (predicted short survival), P < 0.0001; validation cohort, 5y-OS 0.94 +/- 0.07 (long) versus 0.40 +/- 0.13 (short), P < 0.0001]. Time-dependent receiver operator characteristics analyses showed that CASPAR-predicted individual survival times were highly accurate (initial set, mean area under the curve for first 10 years of overall survival prediction 0.92 +/- 0.04; validation set, 0.81 +/- 0.05). Furthermore, CASPAR significantly discriminated short (<5 years) from long survivors (>5 years) in subgroups of patients with unfavorable markers with the exception of MYCN-amplified patients (initial set). Confirmatory results with high significance were observed in the validation cohort [stage 4 disease (P = 0.0049), NB2004 intermediate-risk or high-risk stratification (P = 0.0017), and unfavorable gene expression prediction (P = 0.0017)]. CASPAR accurately forecasts individual survival times for neuroblastoma patients from gene expression data.

Engaging uncertainty in hydrologic data sets using principal component analysis: BaNPCA algorithm

Principal component analysis (PCA) is the most widely used method for dimensionality reduction, data reconstruction, feature extraction, and data visualization in geosciences. However, in its standard form, PCA makes no distinction between data points for which the associated measurement errors vary in both space and time. Using the backdrop of sea surface temperature (SST) data, a Bayesian variant of noisy principal component analysis (BaNPCA) was developed to incorporate observation uncertainty when performing PCA. The algorithm was first assessed using synthetic data sets. Comparison of BaNPCA results with current PCA techniques showed that BaNPCA has lower data reconstruction error; that is, for a given number of principal components, it explains more variance in SST data. Using the automatic relevance determination method, BaNPCA could correctly identify the appropriate number of principal components in the data. BaNPCA was shown to exhibit distinct advantages in filling missing values in the data when compared to existing methods. In addition, the extracted principal vectors from BaNPCA were found to be smoother and more representative of large‐scale signals like El Niño–Southern Oscillation and Pacific Decadal Oscillation. To classify extreme states of all India summer monsoon rainfall, we used robust optimization that utilizes the PCs along with computed uncertainty from BaNPCA algorithm as inputs, thus engaging uncertainty in data. Results from this study demonstrate the value of utilizing uncertainty information available with hydrologic data sets.

A COMPARATIVE STUDY OF FEATURE SELECTION FOR HIDDEN MARKOV MODEL-BASED MICRO-MILLING TOOL WEAR MONITORING

This paper presents a discriminant feature selection approach for hidden Markov model (HMM) modeling of micro-milling tool conditions. The approach is compared with other popular feature selection methods such as principal component analysis (PCA) and automatic relevance determination (ARD) according to their HMM classification rate. In tool condition monitoring (TCM), there are a lot of features that contain redundant information or less sensitive to tool state discrimination. These features are expected to be deleted for less computation and more robust modeling of tool conditions. Fisher's linear discriminant analysis (FDA) is modified for this purpose. The FDA is generally used for classification, and the features are mapped to another space and lose their physical meanings. In the modified discriminant feature selection, the features are selected in the original feature space by maximizing tool state separation and ranked by their separation ability between different tool states. Experimental results from both micro-milling of copper and steel under different working conditions indicate that the FDA is superior to both PCA and ARD for feature selection in HMM's classification. The reasons behind these differences are also discussed.

O.523 Long-span mandibular reconstruction with non-vascularized free graft

Understanding the sorption of pharmaceuticals to sewage sludge during waste water treatment processes is important for understanding their environmental fate and in risk assessments. The degree of sorption is defined by the sludge/water partition coefficient (Kd). Experimental Kd values (n = 297) for active pharmaceutical ingredients (n = 148) in primary and activated sludge were collected from literature. The compounds were classified by their charge at pH 7.4 (44 uncharged, 60 positively and 28 negatively charged, and 16 zwitterions). Univariate models relating log Kd to log Kow for each charge class showed weak correlations (maximum R2 = 0.51 for positively charged) with no overall correlation for the combined dataset (R2 = 0.04). Weaker correlations were found when relating log Kd to log Dow. Three sets of molecular descriptors (Molecular Operating Environment, VolSurf and ParaSurf) encoding a range of physico-chemical properties were used to derive multivariate models using stepwise regression, partial least squares and Bayesian artificial neural networks (ANN). The best predictive performance was obtained with ANN, with R2 = 0.62–0.69 for these descriptors using the complete dataset. Use of more complex Vsurf and ParaSurf descriptors showed little improvement over Molecular Operating Environment descriptors. The most influential descriptors in the ANN models, identified by automatic relevance determination, highlighted the importance of hydrophobicity, charge and molecular shape effects in these sorbate-sorbent interactions. The heterogeneous nature of the different sewage sludges used to measure Kd limited the predictability of sorption from physico-chemical properties of the pharmaceuticals alone. Standardization of test materials for the measurement of Kd would improve comparability of data from different studies, in the long-term leading to better quality environmental risk assessments.

Automatic Relevance Determination for Identifying Thalamic Regions Implicated in Schizophrenia

There have been many theories about and computational models of the schizophrenic disease state. Brain imaging techniques have suggested that abnormalities of the thalamus may contribute to the pathophysiology of schizophrenia. Several studies have found the thalamus to be altered in schizophrenia, and the thalamus has connections with other brain structures implicated in the disorder. This paper describes an experiment examining thalamic levels of the metabolite N-acetylaspartate (NAA), taken from schizophrenics and controls using in vivo proton magnetic resonance spectroscopic imaging. Automatic relevance determination was performed on neural networks trained on this data, identifying NAA group differences in the pulvinar and mediodorsal nucleus, underscoring the importance of examining thalamic subregions in schizophrenia.

Regularity selection for effective 3D object reconstruction from a single line drawing

Different regularities have been used in the reconstruction of a 3D object from a single-view line drawing. These regularities are not all equally informative in the reconstruction process: certain regularities may correspond mainly to noise, not information; some may overlap with each other or are not too relevant to the reconstruction. This paper studies these regularities comprehensively, so as to select the most effective set that can give robust and reliable 3D reconstruction. The selection is made through a method called automatic relevance determination (ARD), which employs the Bayesian framework and support vector regression estimation. The proposed method is able to identify the worst regularities according to their ARD parameters and eliminate them. The effectiveness of this pruning is evaluated by model validation. The regularity set so obtained is effective for general 3D reconstruction. The experimental results show that the regularity set selected can reduce the reconstruction complexity and produce satisfactory reconstruction performance.

A unified Bayesian framework for MEG/EEG source imaging

The ill-posed nature of the MEG (or related EEG) source localization problem requires the incorporation of prior assumptions when choosing an appropriate solution out of an infinite set of candidates. Bayesian approaches are useful in this capacity because they allow these assumptions to be explicitly quantified using postulated prior distributions. However, the means by which these priors are chosen, as well as the estimation and inference procedures that are subsequently adopted to affect localization, have led to a daunting array of algorithms with seemingly very different properties and assumptions. From the vantage point of a simple Gaussian scale mixture model with flexible covariance components, this paper analyzes and extends several broad categories of Bayesian inference directly applicable to source localization including empirical Bayesian approaches, standard MAP estimation, and multiple variational Bayesian (VB) approximations. Theoretical properties related to convergence, global and local minima, and localization bias are analyzed and fast algorithms are derived that improve upon existing methods. This perspective leads to explicit connections between many established algorithms and suggests natural extensions for handling unknown dipole orientations, extended source configurations, correlated sources, temporal smoothness, and computational expediency. Specific imaging methods elucidated under this paradigm include the weighted minimum ℓ2-norm, FOCUSS, minimum current estimation, VESTAL, sLORETA, restricted maximum likelihood, covariance component estimation, beamforming, variational Bayes, the Laplace approximation, and automatic relevance determination, as well as many others. Perhaps surprisingly, all of these methods can be formulated as particular cases of covariance component estimation using different concave regularization terms and optimization rules, making general theoretical analyses and algorithmic extensions/improvements particularly relevant.

An integrated framework for risk profiling of breast cancer patients following surgery

An integrated decision support framework is proposed for clinical oncologists making prognostic assessments of patients with operable breast cancer. The framework may be delivered over a web interface. It comprises a triangulation of prognostic modelling, visualisation of historical patient data and an explanatory facility to interpret risk group assignments using empirically derived Boolean rules expressed directly in clinical terms. The prognostic inferences in the interface are validated in a multicentre longitudinal cohort study by modelling retrospective data from 917 patients recruited at Christie Hospital, Wilmslow between 1983 and 1989 and predicting for 931 patients recruited in the same centre during 1990-1993. There were also 291 patients recruited between 1984 and 1998 at the Clatterbridge Centre for Oncology and the Linda McCartney Centre, Liverpool, UK. There are three novel contributions relating this paper to breast cancer cases. First, the widely used Nottingham prognostic index (NPI) is enhanced with additional clinical features from which prognostic assessments can be made more specific for patients in need of adjuvant treatment. This is shown with a cross matching of the NPI and a new prognostic index which also provides a two-dimensional visualisation of the complete patient database by risk of negative outcome. Second, a principled rule-extraction method, orthogonal search rule extraction, generates readily interpretable explanations of risk group allocations derived from a partial logistic artificial neural network with automatic relevance determination (PLANN-ARD). Third, 95% confidence intervals for individual predictions of survival are obtained by Monte Carlo sampling from the PLANN-ARD model.

Time-to-event analysis with artificial neural networks: An integrated analytical and rule-based study for breast cancer

This paper presents an analysis of censored survival data for breast cancer specific mortality and disease-free survival. There are three stages to the process, namely time-to-event modelling, risk stratification by predicted outcome and model interpretation using rule extraction. Model selection was carried out using the benchmark linear model, Cox regression but risk staging was derived with Cox regression and with Partial Logistic Regression Artificial Neural Networks regularised with Automatic Relevance Determination (PLANN-ARD). This analysis compares the two approaches showing the benefit of using the neural network framework especially for patients at high risk. The neural network model also has results in a smooth model of the hazard without the need for limiting assumptions of proportionality. The model predictions were verified using out-of-sample testing with the mortality model also compared with two other prognostic models called TNG and the NPI rule model. Further verification was carried out by comparing marginal estimates of the predicted and actual cumulative hazards. It was also observed that doctors seem to treat mortality and disease-free models as equivalent, so a further analysis was performed to observe if this was the case. The analysis was extended with automatic rule generation using Orthogonal Search Rule Extraction (OSRE). This methodology translates analytical risk scores into the language of the clinical domain, enabling direct validation of the operation of the Cox or neural network model. This paper extends the existing OSRE methodology to data sets that include continuous-valued variables.

Vbmp: Variational Bayesian Multinomial Probit Regression for multi-class classification in R

Vbmp is an R package for Gaussian Process classification of data over multiple classes. It features multinomial probit regression with Gaussian Process priors and estimates class posterior probabilities employing fast variational approximations to the full posterior. This software also incorporates feature weighting by means of Automatic Relevance Determination. Being equipped with only one main function and reasonable default values for optional parameters, vbmp combines flexibility with ease of usage as is demonstrated on a breast cancer microarray study. The R library vbmp implementing this method is part of Bioconductor and can be downloaded from http://www.dcs.gla.ac.uk/~girolami

Daily solar irradiation estimation over a mountainous area using artificial neural networks

In order to design both active and passive solar energy systems, radiation data are needed for the studied location. The implementation of such renewable energy systems is especially important in places like natural parks, where acoustic and fossil fuel derived contamination has to be completely avoided. Measure of solar radiation is usually accomplished by means of radiometric station nets with a low spatial resolution. To estimate the radiation in sites located away from the stations, different interpolation/extrapolation techniques may be used. These methods are valid on places where the spatial variability of radiation is not significant, but becomes less accurate if complex terrain areas are present in between the radiometric stations. As an alternative, artificial intelligence techniques have been used in this work, along with a 20 m resolution digital model of terrain. The inputs to the network have been selected using the automatic relevance determination methodology. The data set contains 3 years’ data of daily global radiation measured at 12 different stations located in the north face of the Sierra Nevada National Park in the surroundings of Huéneja (Granada), a town located in the South East of Spain. The stations altitude varies from 1000 to 1700 m. The goal of this work has been to estimate daily global irradiation on stations located in a complex terrain, and the values estimated by the neural network model have been compared with the measured ones leading to a root mean square error (RMSE) of 6.0% and a mean bias error (MBE) of 0.2%, both expressed as a percentage of the mean value. Performance achieved individually for each of the stations lies in the range [5.0–7.5]% for the RMSE and [−1.2 to +2.1]% for the MBE. Results point out artificial neural networks as an efficient and easy methodology for calculating solar radiation levels over complex mountain terrains from only one radiometric station data. In addition, this methodology can be applied to other areas with a complex topography.

A variational Bayesian approach to number of sources estimation for multichannel blind deconvolution

Most traditional multichannel blind deconvolution algorithms rely on some assumptions on the mixing model, e.g. the number of sources is known a priori; and the mixing environment is noise-free. Unfortunately, these assumptions are not necessarily true in practice. In this paper, we will relax the assumption placed on the number of sources by studying a state space mixing model where the number of sources is assumed to be unknown but not greater than the number of sensors. Based on this mixing model, we will formulate the estimation of the number of sources problem as a model order selection problem. Model comparison, as a common method of model order selection, usually involves the evaluation of multi-variable integrals which is computationally intractable. A variational Bayesian method is therefore used to overcome this multi-variable integral issue. The problem is solved by approximating the true, complicated posteriors with a set of independent, simple, tractable posteriors. To realize the objective of optimal approximation, we maximize an objective function called negative free energy. We will derive a variational Bayesian algorithm, in which the number of sources will be estimated through two approaches: automatic relevance determination and comparison of the optimized negative free energy. The proposed variational Bayesian algorithm will be evaluated on both artificially generated examples, and practical signals.

Using Bayesian neural networks with ARD input selection to detect malignant ovarian masses prior to surgery

In this paper, we applied Bayesian multi-layer perceptrons (MLP) using the evidence procedure to predict malignancy of ovarian masses in a large (n = 1,066) multi-centre data set. Automatic relevance determination (ARD) was used to select the most relevant inputs. Fivefold cross-validation (5CV) and repeated 5CV was used to select the optimal combination of input set and number of hidden neurons. Results indicate good performance of the models with area under the receiver operating characteristic curve values of 0.93–0.94 on independent data. Comparison with a linear benchmark model and a previously developed logistic regression model shows that the present problem is very well linearly separable. A resampling analysis further shows that the number of hidden neurons specified in the ARD analyses for input selection may influence model performance. This paper shows that Bayesian MLPs, although not frequently used, are a useful tool for detecting malignant ovarian tumours.

An Empirical Bayesian Strategy for Solving the Simultaneous Sparse Approximation Problem

Given a large overcomplete dictionary of basis vectors, the goal is to simultaneously represent L>1 signal vectors using coefficient expansions marked by a common sparsity profile. This generalizes the standard sparse representation problem to the case where multiple responses exist that were putatively generated by the same small subset of features. Ideally, the associated sparse generating weights should be recovered, which can have physical significance in many applications (e.g., source localization). The generic solution to this problem is intractable and, therefore, approximate procedures are sought. Based on the concept of automatic relevance determination, this paper uses an empirical Bayesian prior to estimate a convenient posterior distribution over candidate basis vectors. This particular approximation enforces a common sparsity profile and consistently places its prominent posterior mass on the appropriate region of weight-space necessary for simultaneous sparse recovery. The resultant algorithm is then compared with multiple response extensions of matching pursuit, basis pursuit, FOCUSS, and Jeffreys prior-based Bayesian methods, finding that it often outperforms the others. Additional motivation for this particular choice of cost function is also provided, including the analysis of global and local minima and a variational derivation that highlights the similarities and differences between the proposed algorithm and previous approaches.

A combined MRI and MRSI based multiclass system for brain tumour recognition using LS-SVMs with class probabilities and feature selection

This study investigates the use of automated pattern recognition methods on magnetic resonance data with the ultimate goal to assist clinicians in the diagnosis of brain tumours. Recently, the combined use of magnetic resonance imaging (MRI) and magnetic resonance spectroscopic imaging (MRSI) has demonstrated to improve the accuracy of classifiers. In this paper we extend previous work that only uses binary classifiers to assess the type and grade of a tumour to a multiclass classification system obtaining class probabilities. The important problem of input feature selection is also addressed. Least squares support vector machines (LS-SVMs) with radial basis function kernel are applied and compared with linear discriminant analysis (LDA). Both a Bayesian framework and cross-validation are used to infer the parameters of the LS-SVM classifiers. Four different techniques to obtain multiclass probabilities as a measure of accuracy are compared. Four variable selection methods are explored. MRI and MRSI data are selected from the INTERPRET project database. The results illustrate the significantly better performance of automatic relevance determination (ARD), in combination with LS-SVMs in a Bayesian framework and coupling of class probabilities, compared to classical LDA. It is demonstrated that binary LS-SVMs can be extended to a multiclass classifier system obtaining class probabilities by Bayesian techniques and pairwise coupling. Feature selection based on ARD further improves the results. This classifier system can be of great help in the diagnosis of brain tumours.

Automatic relevance determination based hierarchical Bayesian MEG inversion in practice

In recent simulation studies, a hierarchical Variational Bayesian (VB) method, which can be seen as a generalisation of the traditional minimum-norm estimate (MNE), was introduced for reconstructing distributed MEG sources. Here, we studied how nonlinearities in the estimation process and hyperparameter selection affect the inverse solutions, the feasibility of a full Bayesian treatment of the hyperparameters, and multimodality of the true posterior, in an empirical dataset wherein a male subject was presented with pure tone and checkerboard reversal stimuli, alone and in combination. An MRI-based cortical surface model was employed. Our results show, with a comparison to the basic MNE, that the hierarchical VB approach yields robust and physiologically plausible estimates of distributed sources underlying MEG measurements, in a rather automated fashion.

Probabilistic diffusion tractography with multiple fibre orientations: What can we gain?

We present a direct extension of probabilistic diffusion tractography to the case of multiple fibre orientations. Using automatic relevance determination, we are able to perform online selection of the number of fibre orientations supported by the data at each voxel, simplifying the problem of tracking in a multi-orientation field. We then apply the identical probabilistic algorithm to tractography in the multi- and single-fibre cases in a number of example systems which have previously been tracked successfully or unsuccessfully with single-fibre tractography. We show that multi-fibre tractography offers significant advantages in sensitivity when tracking non-dominant fibre populations, but does not dramatically change tractography results for the dominant pathways.

Variational free energy and the Laplace approximation

This note derives the variational free energy under the Laplace approximation, with a focus on accounting for additional model complexity induced by increasing the number of model parameters. This is relevant when using the free energy as an approximation to the log-evidence in Bayesian model averaging and selection. By setting restricted maximum likelihood (ReML) in the larger context of variational learning and expectation maximisation (EM), we show how the ReML objective function can be adjusted to provide an approximation to the log-evidence for a particular model. This means ReML can be used for model selection, specifically to select or compare models with different covariance components. This is useful in the context of hierarchical models because it enables a principled selection of priors that, under simple hyperpriors, can be used for automatic model selection and relevance determination (ARD). Deriving the ReML objective function, from basic variational principles, discloses the simple relationships among Variational Bayes, EM and ReML. Furthermore, we show that EM is formally identical to a full variational treatment when the precisions are linear in the hyperparameters. Finally, we also consider, briefly, dynamic models and how these inform the regularisation of free energy ascent schemes, like EM and ReML.

Determination of daily solar ultraviolet radiation using statistical models and artificial neural networks

Abstract. In this study, two different methodologies are used to develop two models for estimating daily solar UV radiation. The first is based on traditional statistical techniques whereas the second is based on artificial neural network methods. Both models use daily solar global broadband radiation as the only measured input. The statistical model is derived from a relationship between the daily UV and the global clearness indices but modulated by the relative optical air mass. The inputs to the neural network model were determined from a large number of radiometric and atmospheric parameters using the automatic relevance determination method, although only the daily solar global irradiation, daily global clearness index and relative optical air mass were shown to be the optimal input variables. Both statistical and neural network models were developed using data measured at Almería (Spain), a semiarid and coastal climate, and tested against data from Table Mountain (Golden, CO, USA), a mountainous and dry environment. Results show that the statistical model performs adequately in both sites for all weather conditions, especially when only snow-free days at Golden were considered (RMSE=4.6%, MBE= –0.1%). The neural network based model provides the best overall estimates in the site where it has been trained, but presents an inadequate performance for the Golden site when snow-covered days are included (RMSE=6.5%, MBE= –3.0%). This result confirms that the neural network model does not adequately respond on those ranges of the input parameters which were not used for its development.