Bayesian Feature Selection Research Articles

Regional Wind Power Forecasting Based On Bayesian Feature Selection

Regional Wind Power Forecasting Based On Bayesian Feature Selection

News indices on country fundamentals

We propose a novel method to extract textual information about macro fundamentals. The method has two pillars, a set of pre-defined regular expressions and a Bayesian feature selection model. We apply our technique to a 2007–2022 Reuters news corpus from Factiva to create news indices of country fundamentals. Compared to several literature alternatives, we find our method to better identify and discriminate among fundamentals based on both (i) observed economic surprises (macro announcements compared to Bloomberg survey expectations) and (ii) labels on a manually classified test sample. In an application that investigates the determinants of sovereign credit spreads, we show that including our news indices next to traditional macro variables significantly raises the explanatory power attributed to fundamentals. We also show that part of the covariance between sovereign spreads and the VIX and US high yield indices is related to global fundamentals captured by our indices.

Bayesian feature selection for radiomics using reliability metrics.

Introduction: Imaging of tumors is a standard step in diagnosing cancer and making subsequent treatment decisions. The field of radiomics aims to develop imaging based biomarkers using methods rooted in artificial intelligence applied to medical imaging. However, a challenging aspect of developing predictive models for clinical use is that many quantitative features derived from image data exhibit instability or lack of reproducibility across different imaging systems or image-processing pipelines. Methods: To address this challenge, we propose a Bayesian sparse modeling approach for image classification based on radiomic features, where the inclusion of more reliable features is favored via a probit prior formulation. Results: We verify through simulation studies that this approach can improve feature selection and prediction given correct prior information. Finally, we illustrate the method with an application to the classification of head and neck cancer patients by human papillomavirus status, using as our prior information a reliability metric quantifying feature stability across different imaging systems.

Addictive brain-network identification by spatial attention recurrent network with feature selection

Addiction in the brain is associated with adaptive changes that reshape addiction-related brain regions and lead to functional abnormalities that cause a range of behavioral changes, and functional magnetic resonance imaging (fMRI) studies can reveal complex dynamic patterns of brain functional change. However, it is still a challenge to identify functional brain networks and discover region-level biomarkers between nicotine addiction (NA) and healthy control (HC) groups. To tackle it, we transform the fMRI of the rat brain into a network with biological attributes and propose a novel feature-selected framework to extract and select the features of addictive brain regions and identify these graph-level networks. In this framework, spatial attention recurrent network (SARN) is designed to capture the features with spatial and time-sequential information. And the Bayesian feature selection(BFS) strategy is adopted to optimize the model and improve classification tasks by restricting features. Our experiments on the addiction brain imaging dataset obtain superior identification performance and interpretable biomarkers associated with addiction-relevant brain regions.

Bayesian Feature Selection in Joint Quantile Time Series Analysis

Bayesian Feature Selection in Joint Quantile Time Series Analysis

A rapid feature selection method for catalyst design: Iterative Bayesian additive regression trees (iBART).

Feature selection (FS) methods often are used to develop data-driven descriptors (i.e., features) for rapidly predicting the functional properties of a physical or chemical system based on its composition and structure. FS algorithms identify descriptors from a candidate pool (i.e., feature space) built by feature engineering (FE) steps that construct complex features from the system's fundamental physical properties. Recursive FE, which involves repeated FE operations on the feature space, is necessary to build features with sufficient complexity to capture the physical behavior of a system. However, this approach creates a highly correlated feature space that contains millions or billions of candidate features. Such feature spaces are computationally demanding to process using traditional FS approaches that often struggle with strong collinearity. Herein, we address this shortcoming by developing a new method that interleaves the FE and FS steps to progressively build and select powerful descriptors with reduced computational demand. We call this method iterative Bayesian additive regression trees (iBART), as it iterates between FE with unary/binary operators and FS with Bayesian additive regression trees (BART). The capabilities of iBART are illustrated by extracting descriptors for predicting metal-support interactions in catalysis, which we compare to those predicted in our previous work using other state-of-the-art FS methods (i.e., least absolute shrinkage and selection operator + l0, sure independence screening and sparsifying operator, and Bayesian FS). iBART matches the performance of these methods yet uses a fraction of the computational resources because it generates a maximum feature space of size O(102), as opposed to O(106) generated by one-shot FE/FS methods.

Theory of Optimal Bayesian Feature Filtering

Optimal Bayesian feature filtering (OBF) is a supervised screening method designed for biomarker discovery. In this article, we prove two major theoretical properties of OBF. First, optimal Bayesian feature selection under a general family of Bayesian models reduces to filtering if and only if the underlying Bayesian model assumes all features are mutually independent. Therefore, OBF is optimal if and only if one assumes all features are mutually independent, and OBF is the only filter method that is optimal under at least one model in the general Bayesian framework. Second, OBF under independent Gaussian models is consistent under very mild conditions, including cases where the data is non-Gaussian with correlated features. This result provides conditions where OBF is guaranteed to identify the correct feature set given enough data, and it justifies the use of OBF in non-design settings where its assumptions are invalid.

Simultaneous Bayesian Clustering and Feature Selection Through Student's ${t}$ Mixtures Model.

In this paper, we proposed a generative model for feature selection under the unsupervised learning context. The model assumes that data are independently and identically sampled from a finite mixture of Student's distributions, which can reduce the sensitiveness to outliers. Latent random variables that represent the features' salience are included in the model for the indication of the relevance of features. As a result, the model is expected to simultaneously realize clustering, feature selection, and outlier detection. Inference is carried out by a tree-structured variational Bayes algorithm. Full Bayesian treatment is adopted in the model to realize automatic model selection. Controlled experimental studies showed that the developed model is capable of modeling the data set with outliers accurately. Furthermore, experiment results showed that the developed algorithm compares favorably against existing unsupervised probability model-based Bayesian feature selection algorithms on artificial and real data sets. Moreover, the application of the developed algorithm on real leukemia gene expression data indicated that it is able to identify the discriminating genes successfully.

Bayesian Feature Selection with Strongly Regularizing Priors Maps to the Ising Model.

Identifying small subsets of features that are relevant for prediction and classification tasks is a central problem in machine learning and statistics. The feature selection task is especially important, and computationally difficult, for modern data sets where the number of features can be comparable to or even exceed the number of samples. Here, we show that feature selection with Bayesian inference takes a universal form and reduces to calculating the magnetizations of an Ising model under some mild conditions. Our results exploit the observation that the evidence takes a universal form for strongly regularizing priors--priors that have a large effect on the posterior probability even in the infinite data limit. We derive explicit expressions for feature selection for generalized linear models, a large class of statistical techniques that includes linear and logistic regression. We illustrate the power of our approach by analyzing feature selection in a logistic regression-based classifier trained to distinguish between the letters B and D in the notMNIST data set.

A biological mechanism for Bayesian feature selection: Weight decay and raising the LASSO

Biological systems are capable of learning that certain stimuli are valuable while ignoring the many that are not, and thus perform feature selection. In machine learning, one effective feature selection approach is the least absolute shrinkage and selection operator (LASSO) form of regularization, which is equivalent to assuming a Laplacian prior distribution on the parameters. We review how such Bayesian priors can be implemented in gradient descent as a form of weight decay, which is a biologically plausible mechanism for Bayesian feature selection. In particular, we describe a new prior that offsets or “raises” the Laplacian prior distribution. We evaluate this alongside the Gaussian and Cauchy priors in gradient descent using a generic regression task where there are few relevant and many irrelevant features. We find that raising the Laplacian leads to less prediction error because it is a better model of the underlying distribution. We also consider two biologically relevant online learning tasks, one synthetic and one modeled after the perceptual expertise task of Krigolson et al. (2009). Here, raising the Laplacian prior avoids the fast erosion of relevant parameters over the period following training because it only allows small weights to decay. This better matches the limited loss of association seen between days in the human data of the perceptual expertise task. Raising the Laplacian prior thus results in a biologically plausible form of Bayesian feature selection that is effective in biologically relevant contexts.

Bayesian feature selection for high-dimensional linear regression via the Ising approximation with applications to genomics

Feature selection, identifying a subset of variables that are relevant for predicting a response, is an important and challenging component of many methods in statistics and machine learning. Feature selection is especially difficult and computationally intensive when the number of variables approaches or exceeds the number of samples, as is often the case for many genomic datasets. Here, we introduce a new approach--the Bayesian Ising Approximation (BIA)-to rapidly calculate posterior probabilities for feature relevance in L2 penalized linear regression. In the regime where the regression problem is strongly regularized by the prior, we show that computing the marginal posterior probabilities for features is equivalent to computing the magnetizations of an Ising model with weak couplings. Using a mean field approximation, we show it is possible to rapidly compute the feature selection path described by the posterior probabilities as a function of the L2 penalty. We present simulations and analytical results illustrating the accuracy of the BIA on some simple regression problems. Finally, we demonstrate the applicability of the BIA to high-dimensional regression by analyzing a gene expression dataset with nearly 30 000 features. These results also highlight the impact of correlations between features on Bayesian feature selection. An implementation of the BIA in C++, along with data for reproducing our gene expression analyses, are freely available at http://physics.bu.edu/∼pankajm/BIACode.

Feature ranking of type 1 diabetes susceptibility genes improves prediction of type 1 diabetes.

More than 40 regions of the human genome confer susceptibility for type 1 diabetes and could be used to establish population screening strategies. The aim of our study was to identify weighted sets of SNP combinations for type 1 diabetes prediction. We applied multivariable logistic regression and Bayesian feature selection to the Type 1 Diabetes Genetics Consortium (T1DGC) dataset with genotyping of HLA plus 40 SNPs within other type 1 diabetes-associated gene regions in 4,574 cases and 1,207 controls. We tested the weighted models in an independent validation set (765 cases, 423 controls), and assessed their performance in 1,772 prospectively followed children. The inclusion of 40 non-HLA gene SNPs significantly improved the prediction of type 1 diabetes over that provided by HLA alone (p = 3.1 × 10(-25)), with a receiver operating characteristic AUC of 0.87 in the T1DGC set, and 0.84 in the validation set. Feature selection identified HLA plus nine SNPs from the PTPN22, INS, IL2RA, ERBB3, ORMDL3, BACH2, IL27, GLIS3 and RNLS genes that could achieve similar prediction accuracy as the total SNP set. Application of this ten SNP model to prospectively followed children was able to improve risk stratification over that achieved by HLA genotype alone. We provided a weighted risk model with selected SNPs that could be considered for recruitment of infants into studies of early type 1 diabetes natural history or appropriately safe prevention.

Bayesian feature selection in high‐dimensional regression in presence of correlated noise

We consider the problem of feature selection in a high‐dimensional multiple predictors, multiple responses regression setting. Assuming that regression errors are i.i.d. when they are in fact dependent leads to inconsistent and inefficient feature estimates. We relax the i.i.d. assumption by allowing the errors to exhibit a tree‐structured dependence. This allows a Bayesian problem formulation with the error dependence structure treated as an auxiliary variable that can be integrated out analytically with the help of the matrix‐tree theorem. Mixing over trees results in a flexible technique for modelling the graphical structure for the regression errors. Furthermore, the analytic integration results in a collapsed Gibbs sampler for feature selection that is computationally efficient. Our approach offers significant performance gains over the competing methods in simulations, especially when the features themselves are correlated. In addition to comprehensive simulation studies, we apply our method to a high‐dimensional breast cancer data set to identify markers significantly associated with the disease. Copyright © 2014 John Wiley & Sons, Ltd.

Capturing the Crystal: Prediction of Enthalpy of Sublimation, Crystal Lattice Energy, and Melting Points of Organic Compounds

Accurate computational prediction of melting points and aqueous solubilities of organic compounds would be very useful but is notoriously difficult. Predicting the lattice energies of compounds is key to understanding and predicting their melting behavior and ultimately their solubility behavior. We report robust, predictive, quantitative structure-property relationship (QSPR) models for enthalpies of sublimation, crystal lattice energies, and melting points for a very large and structurally diverse set of small organic compounds. Sparse Bayesian feature selection and machine learning methods were employed to select the most relevant molecular descriptors for the model and to generate parsimonious quantitative models. The final enthalpy of sublimation model is a four-parameter multilinear equation that has an r(2) value of 0.96 and an average absolute error of 7.9 ± 0.3 kJ.mol(-1). The melting point model can predict this property with a standard error of 45° ± 1 K and r(2) value of 0.79. Given the size and diversity of the training data, these conceptually transparent and accurate models can be used to predict sublimation enthalpy, lattice energy, and melting points of organic compounds in general.

Abstract IA15: Predicting drug sensitivity from cancer cell lines

Abstract High-throughput molecular characterization technologies have enabled detailed genetic and genomic characterization of large panels of cancer cell lines, including profiling of gene expression, copy number variation (CNV), SNP, mutation, and RNA-seq. The same cell lines have also been profiled for sensitivity to potential anti-cancer compounds, allowing development of computational models used to predict genotype-specific drug treatments linked to tumor molecular subtypes. Inferring such models is challenging because the model inputs contain far more features than observations, known as the p>>n problem, precluding the use of classical statistical models such as least squares (a.k.a. multiple linear regression). Recently, generalized linear models or penalized linear models have been proposed and extended for feature selection and overcoming the p>>n. Bayesian extension of such methods have also been developed, using algorithms such as Metropolis-Hasting and Markov Chain Monte Carlo, and demonstrated to improve prediction accuracy at the expense of increased computing cost. A comprehensive evaluation of predictive modeling techniques is essential to leverage cell line studies to infer the most accurate genetic predictors of drug sensitivity, which can then be used to inform patient selection strategies in clinical trials and to identify functional genetic determinants of drug sensitivity or resistance. In this study, we evaluate state-of-the-art predictive models which utilize dimension reduction methods (e.g., partial least square, support vector machine, principal component regression), feature selection (e.g., LASSO, RIDGE, and elastic-net), and Bayesian feature selection (e.g., Bayesian LASSO and Bayesian RIDGE). We assess each method based on 1) the accuracy of sensitivity predictions in a cross validation setting and in an independent dataset, and 2) the biological coherence of inferred predictive features by comparison to publicly available pathway databases. Citation Format: Adam Arne Margolin, In Sock Jang, Stephen Friend. Predicting drug sensitivity from cancer cell lines. [abstract]. In: Proceedings of the AACR Special Conference on Chemical Systems Biology: Assembling and Interrogating Computational Models of the Cancer Cell by Chemical Perturbations; 2012 Jun 27-30; Boston, MA. Philadelphia (PA): AACR; Cancer Res 2012;72(13 Suppl):Abstract nr IA15.

Robust, quantitative tools for modelling ex-vivo expansion of haematopoietic stem cells and progenitors

Despite substantial research activity on bioreactor design and experiments, there are very few reports of modelling tools that can be used to generate predictive models describing how bioreactor parameters affect performance. New developments in mathematics, such as sparse Bayesian feature selection methods and nonlinear model-free modelling regression methods, offer considerable promise for modelling diverse types of data. The utility of these mathematical tools in stem cell biology are demonstrated by analysis of a large set of bioreactor data derived from the literature. In spite of the diversity of the data sources, and the inherent difficulty in representing bioreactor variables, these modelling methods were able to develop robust, quantitative, predictive models. These models relate bioreactor operational parameters to the degree of expansion of haematopoietic stem cells or their progenitors, and also identify the bioreactor variables that are most likely to affect performance across many experiments. These methods show substantial promise in assisting the design and optimisation of stem cell bioreactors.

A Bayesian feature selection paradigm for text classification

The automated classification of texts into predefined categories has witnessed a booming interest, due to the increased availability of documents in digital form and the ensuing need to organize them. An important problem for text classification is feature selection, whose goals are to improve classification effectiveness, computational efficiency, or both. Due to categorization unbalancedness and feature sparsity in social text collection, filter methods may work poorly. In this paper, we perform feature selection in the training process, automatically selecting the best feature subset by learning, from a set of preclassified documents, the characteristics of the categories. We propose a generative probabilistic model, describing categories by distributions, handling the feature selection problem by introducing a binary exclusion/inclusion latent vector, which is updated via an efficient Metropolis search. Real-life examples illustrate the effectiveness of the approach.

Bayesian feature selection for classification with possibly large number of classes

In what follows, we introduce two Bayesian models for feature selection in high-dimensional data, specifically designed for the purpose of classification. We use two approaches to the problem: one which discards the components which have “almost constant” values (Model 1) and another which retains the components for which variations in-between the groups are larger than those within the groups (Model 2). We assume that p ⪢ n , i.e. the number of components p is much larger than the number of samples n, and that only few of those p components are useful for subsequent classification. We show that particular cases of the above two models recover familiar variance or ANOVA-based component selection. When one has only two classes and features are a priori independent, Model 2 reduces to the Feature Annealed Independence Rule (FAIR) introduced by Fan and Fan (2008) and can be viewed as a natural generalization of FAIR to the case of L > 2 classes. The performance of the methodology is studies via simulations and using a biological dataset of animal communication signals comprising 43 groups of electric signals recorded from tropical South American electric knife fishes.

Modelling Inhalational Anaesthetics Using Bayesian Feature Selection and QSAR Modelling Methods

The development of robust and predictive QSAR models is highly dependent on the use of molecular descriptors that contain information relevant to the property being modelled. Selection of these relevant features from a large pool of possibilities is difficult to achieve effectively. Modern Bayesian methods provide substantial advantages over conventional feature selection methods for feature selection and QSAR modelling. We illustrate the importance of descriptor choice and the beneficial properties of Bayesian methods to select context-dependent relevant descriptors and build robust QSAR models, using data on anaesthetics. Our results show the effectiveness of Bayesian feature selection methods in choosing the best descriptors when these are mixed with less informative descriptors. They also demonstrate the efficacy of the Abraham descriptors and identify deficiencies in ParaSurf descriptors for modelling anaesthetic action.

Approximate Bayesian feature selection on a large meta-dataset offers novel insights on factors that effect siRNA potency

Motivation: Short interfering RNA (siRNA)-induced RNA interference is an endogenous pathway in sequence-specific gene silencing. The potency of different siRNAs to inhibit a common target varies greatly and features affecting inhibition are of high current interest. The limited success in predicting siRNA potency being reported so far could originate in the small number and the heterogeneity of available datasets in addition to the knowledge-driven, empirical basis on which features thought to be affecting siRNA potency are often chosen. We attempt to overcome these problems by first constructing a meta-dataset of 6483 publicly available siRNAs (targeting mammalian mRNA), the largest to date, and then applying a Bayesian analysis which accommodates feature set uncertainty. A stochastic logistic regression-based algorithm is designed to explore a vast model space of 497 compositional, structural and thermodynamic features, identifying associations with siRNA potency.Results: Our algorithm reveals a number of features associated with siRNA potency that are, to the best of our knowledge, either under reported in literature, such as anti-sense 5′ −3′ motif ‘UCU’, or not reported at all, such as the anti-sense 5′ -3′ motif ‘ACGA’. These findings should aid in improving future siRNA potency predictions and might offer further insights into the working of the RNA-induced silencing complex (RISC).Contact: cholmes@stats.ox.ac.ukSupplementary information: Supplementary data are available at Bioinformatics online.