Assessing reproducibility and veracity across machine learning techniques in biomedicine: A case study using TCGA data

Abstract

BackgroundMany studies that aim to identify gene biomarkers using statistical methods and translate them into FDA-approved drugs have faced challenges due to lack of clinical validity and methodological reproducibility. Since genomic data analysis relies heavily on these statistical learning tools more than before, it is vital to address the limitations of these computational techniques. MethodsOur study demonstrates these methodological gaps among most common statistical learning techniques used in gene expression analysis. To assess the classification ability and reproducibility of statistical learning tools for gene biomarker detection, six state-of-the-art machine learning models were trained on four different cancer data retrieved from The Cancer Genome Atlas (TCGA). Standard performance metrics including specificity, sensitivity, precision, and F1 score were evaluated to investigate the classification ability. For analysis of reproducibility, the identifiability of gene classifiers was examined by quantifying the consistency of the chosen classifier genes. ResultsAmong the six state-of-the-art machine learning methods, the random forest had the best classification ability overall. Very few genes were selected by multiple methods, which suggests poor identifiability and reproducibility of statistical learning methods for gene expression data. Our results demonstrated the challenges of reproducing discoveries from gene expression analysis due to the inherent differences that exist in statistical machine learning methods. ConclusionSince statistical machine learning models can have large variations in high-dimensional settings such as analysis of gene expression data, transparent analysis procedures including data preprocessing, model parameterization, and evaluation and choice of interpretable models are required for clinical validity and utility.