Bayesian kernel machine models for testing genetic pathway effects in prostate cancer prognosis

Abstract

In this paper we propose a Bayesian semiparametric regression model to estimate and test the effect of a genetic pathway on prostate‐specific antigen (PSA) measurements for patients with prostate cancer. The underlying functional relationship between the genetic pathway and PSA is modeled using reproducing kernel Hilbert space (RKHS) theory. The RKHS formulation makes our model highly flexible, which can capture the complex multidimensional relationship between the genes in a genetic pathway and the response. Moreover, the higher order and nonlinear interactions among the genes in a pathway are also automatically modeled through our kernel‐based representation. We illustrate the connection between our semiparametric regression based on RKHS and a linear mixed model by choosing a special prior distribution on the model parameters. To test the significance of a genetic pathway toward the phenotypic response like PSA, we propose a Bayesian hypothesis testing scheme based on the Bayes factor. An efficient Markov chain Monte Carlo algorithm is designed to estimate the model parameters, Bayes factors, and the genetic pathway effect simultaneously. We illustrate the effectiveness of our model by five simulation studies and one real prostate cancer gene expression data analysis.