Finite Mixtures of Semiparametric Bayesian Survival Kernel Machine Regressions: Application to Breast Cancer Gene Pathway Subgroup Analysis

Abstract

Abstract A gene pathway is defined as a set of genes that functionally work together to regulate a certain biological process. Gene pathway expression data, which is a special case of highly correlated high-dimensional data, exhibits the ‘small n and large p’ problem. Pathway analysis can take into account the dependency structures among genes and the possibility that several moderately regulated genes may have significant impacts on the clinical outcomes. To test the significance of gene pathways in the presence of subgroups, we propose a finite mixture model of semiparametric Bayesian survival kernel machine regressions (fm-BKSurv). Within each hidden group, we model the unknown function of gene pathways via a Gaussian kernel machine. We demonstrate how fm-BKSurv excels in terms of true positive rate, false positive rate, accuracy, and precision in a simulation study, and further illustrate the outperformance of fm-BKSurv in detecting significant gene pathways using a gene pathway expression dataset of breast cancer patients.