Automatic Relevance Determination Research Articles

Bayesian neural network learning for repeat purchase modelling in direct marketing

We focus on purchase incidence modelling for a European direct mail company. Response models based on statistical and neural network techniques are contrasted. The evidence framework of MacKay is used as an example implementation of Bayesian neural network learning, a method that is fairly robust with respect to problems typically encountered when implementing neural networks. The automatic relevance determination (ARD) method, an integrated feature of this framework, allows us to assess the relative importance of the inputs. The basic response models use operationalisations of the traditionally discussed Recency, Frequency and Monetary (RFM) predictor categories. In a second experiment, the RFM response framework is enriched by the inclusion of other (non-RFM) customer profiling predictors. We contribute to the literature by providing experimental evidence that: (1) Bayesian neural networks offer a viable alternative for purchase incidence modelling; (2) a combined use of all three RFM predictor categories is advocated by the ARD method; (3) the inclusion of non-RFM variables allows to significantly augment the predictive power of the constructed RFM classifiers; (4) this rise is mainly attributed to the inclusion of customer/company interaction variables and a variable measuring whether a customer uses the credit facilities of the direct mailing company.

Compositional prediction of creep rupture life of single crystal Ni base superalloy by Bayesian neural network

The creep rupture lives of single crystal Ni base superalloys are predicted as a function of alloy composition, creep stress, and creep temperature. A database has been constructed from previous publications and the Rolls-Royce materials database. The Bayesian neural network technique was used for the modelling of creep rupture life. To obtain the posterior distribution and prediction which is of a high dimensional integral form, the Markov chain Monte Carlo method was applied . With a complex 13-31-1 architecture, the neural network shows a precise prediction of creep rupture life of superalloys. Automatic relevance determination reveals that the influences of Re and Cr on creep rupture life of single crystal superalloys are the greatest amongst the alloying elements.

Gaussian Process: An Efficient Technique to Solve Quantitative Structure-Property Relationship Problems

We introduce the Gaussian process (GP) model for the empirical modelling of the log P values of 44 1,2-dithiole-3-one molecules. A brief theoretical description of the method is given. Descriptive and predictive abilities of GP are evaluated and compared to multilinear regression results. Special attention is devoted to the automatic relevance determination (ARD) to reduce input variable numbers, which avoid the use of principal component analysis. The present approach was found to be an efficient method and a good alternative to more complicated using artificial neural network systems.

An adaptive neural-wavelet model for short term load forecasting

This paper proposed a novel model for short term load forecast in the competitive electricity market. The prior electricity demand data are treated as time series. The forecast model is based on wavelet multi-resolution decomposition by autocorrelation shell representation and neural networks (multilayer perceptrons, or MLPs) modeling of wavelet coefficients. To minimize the influence of noisy low level coefficients, we applied the practical Bayesian method Automatic Relevance Determination (ARD) model to choose the size of MLPs, which are then trained to provide forecasts. The individual wavelet domain forecasts are recombined to form the accurate overall forecast. The proposed method is tested using Queensland electricity demand data from the Australian National Electricity Market. (C) 2001 Elsevier Science B.V. All rights reserved.

Multiresolution forecasting for futures trading using wavelet decompositions

We investigate the effectiveness of a financial time-series forecasting strategy which exploits the multiresolution property of the wavelet transform. A financial series is decomposed into an over complete, shift invariant scale-related representation. In transform space, each individual wavelet series is modeled by a separate multilayer perceptron (MLP). We apply the Bayesian method of automatic relevance determination to choose short past windows (short-term history) for the inputs to the MLPs at lower scales and long past windows (long-term history) at higher scales. To form the overall forecast, the individual forecasts are then recombined by the linear reconstruction property of the inverse transform with the chosen autocorrelation shell representation, or by another perceptron which learns the weight of each scale in the prediction of the original time series. The forecast results are then passed to a money management system to generate trades.

Comparing Bayesian neural network algorithms for classifying segmented outdoor images

In this paper we investigate the Bayesian training of neural networks for region labelling of segmented outdoor scenes; the data are drawn from the Sowerby Image Database of British Aerospace. Neural networks are trained with two Bayesian methods, (i) the evidence framework of MacKay (1992a,b) and (ii) a Markov Chain Monte Carlo method due to Neal (1996). The performance of the two methods is compared to evaluating the empirical learning curves of neural networks trained with the two methods. We also investigate the use of the Automatic Relevance Determination method for input feature selection.

An Electronic Commerce Application of the Bayesian Framework for MLPs: The Effect of Marginalisation and ARD

Automatic Relevance Determination (ARD), with multiple regularisation terms. A real-world classification problem, concerning the discrimination of online purchasers and non-purchasers using Internet’s WWW users’ opinions, is the test-bed for this assessment.

Improved long-term temperature prediction by chaining of neural networks.

When an artificial neural network (ANN) is trained to predict signals p steps ahead, the quality of the prediction typically decreases for large values of p. In this paper, we compare two methods for prediction with ANNs: the classical recursion of one-step ahead predictors and a new kind of chain structure. When applying both techniques to the prediction of the temperature at the end of a blast furnace, we conclude that the chaining approach leads to an improved prediction of the temperature and avoidance of instabilities, since the chained networks gradually take the prediction of their predecessors in the chain as an extra input. It is observed that instabilities might occur in the iterative case, which does not happen with the chaining approach. To select relevant inputs and decrease the number of weights in this approach, Automatic Relevance Determination (ARD) for multilayer perceptrons is applied.

Use of automatic relevance determination in QSAR studies using Bayesian neural networks.

We describe the use of Bayesian regularized artificial neural networks (BRANNs) coupled with automatic relevance determination (ARD) in the development of quantitative structure-activity relationship (QSAR) models. These BRANN-ARD networks have the potential to solve a number of problems which arise in QSAR modeling such as the following: choice of model; robustness of model; choice of validation set; size of validation effort; and optimization of network architecture. The ARD method ensures that irrelevant or highly correlated indices used in the modeling are neglected as well as showing which are the most important variables in modeling the activity data. The application of the methods to QSAR of compounds active at the benzodiazepine and muscarinic receptors as well as some toxicological data of the effect of substituted benzenes on Tetetrahymena pyriformis is illustrated.

Bayesian neural networks for classification: how useful is the evidence framework?

This paper presents an empirical assessment of the Bayesian evidence framework for neural networks using four synthetic and four real-world classification problems. We focus on three issues; model selection, automatic relevance determination (ARD) and the use of committees. Model selection using the evidence criterion is only tenable if the number of training examples exceeds the number of network weights by a factor of five or ten. With this number of available examples, however, cross-validation is a viable alternative. The ARD feature selection scheme is only useful in networks with many hidden units and for data sets containing many irrelevant variables. ARD is also useful as a hard feature selection method. Results on applying the evidence framework to the real-world data sets showed that committees of Bayesian networks achieved classification accuracies similar to the best alternative methods. Importantly, this was achievable with a minimum of human intervention.

An empirical evaluation of Bayesian sampling with hybrid Monte Carlo for training neural network classifiers

This article gives a concise overview of Bayesian sampling for neural networks, and then presents an extensive evaluation on a set of various benchmark classification problems. The main objective is to study the sensitivity of this scheme to changes in the prior distribution of the parameters and hyperparameters, and to evaluate the efficiency of the so-called automatic relevance determination (ARD) method. The article concludes with a comparison of the achieved classification results with those obtained with (i) the evidence scheme and (ii) with non-Bayesian methods.

A review of Bayesian neural networks with an application to near infrared spectroscopy

MacKay's (1992) Bayesian framework for backpropagation is a practical and powerful means to improve the generalization ability of neural networks. It is based on a Gaussian approximation to the posterior weight distribution. The framework is extended, reviewed, and demonstrated in a pedagogical way. The notation is simplified using the ordinary weight decay parameter, and a detailed and explicit procedure for adjusting several weight decay parameters is given. Bayesian backprop is applied in the prediction of fat content in minced meat from near infrared spectra. It outperforms "early stopping" as well as quadratic regression. The evidence of a committee of differently trained networks is computed, and the corresponding improved generalization is verified. The error bars on the predictions of the fat content are computed. There are three contributors: The random noise, the uncertainty in the weights, and the deviation among the committee members. The Bayesian framework is compared to Moody's GPE (1992). Finally, MacKay and Neal's automatic relevance determination, in which the weight decay parameters depend on the input number, is applied to the data with improved results.