Prognostic analysis of histopathological images using pre-trained convolutional neural networks: application to hepatocellular carcinoma.

Liangqun Lu,Bernie J. Daigle

doi:10.7717/peerj.8668

Abstract

Histopathological images contain rich phenotypic descriptions of the molecular processes underlying disease progression. Convolutional neural networks, state-of-the-art image analysis techniques in computer vision, automatically learn representative features from such images which can be useful for disease diagnosis, prognosis, and subtyping. Hepatocellular carcinoma (HCC) is the sixth most common type of primary liver malignancy. Despite the high mortality rate of HCC, little previous work has made use of CNN models to explore the use of histopathological images for prognosis and clinical survival prediction of HCC. We applied three pre-trained CNN models—VGG 16, Inception V3 and ResNet 50—to extract features from HCC histopathological images. Sample visualization and classification analyses based on these features showed a very clear separation between cancer and normal samples. In a univariate Cox regression analysis, 21.4% and 16% of image features on average were significantly associated with overall survival (OS) and disease-free survival (DFS), respectively. We also observed significant correlations between these features and integrated biological pathways derived from gene expression and copy number variation. Using an elastic net regularized Cox Proportional Hazards model of OS constructed from Inception image features, we obtained a concordance index (C-index) of 0.789 and a significant log-rank test (p = 7.6E−18). We also performed unsupervised classification to identify HCC subgroups from image features. The optimal two subgroups discovered using Inception model image features showed significant differences in both overall (C-index = 0.628 and p = 7.39E−07) and DFS (C-index = 0.558 and p = 0.012). Our work demonstrates the utility of extracting image features using pre-trained models by using them to build accurate prognostic models of HCC as well as highlight significant correlations between these features, clinical survival, and relevant biological pathways. Image features extracted from HCC histopathological images using the pre-trained CNN models VGG 16, Inception V3 and ResNet 50 can accurately distinguish normal and cancer samples. Furthermore, these image features are significantly correlated with survival and relevant biological pathways.

Highlights

IntroductionConvolutional neural networks (CNNs), a state-of-the-art image analysis technique in computer vision, automatically learns representative features from such images which can be useful for disease diagnosis, prognosis, and subtyping
Histopathological images contain rich phenotypic descriptions of the molecular processes underlying disease progression
Image feature extraction and survival analysis Histopathology assessment is mandatory in hepatocellular carcinoma (HCC) diagnosis [24] and the characteristics tumor number, size, cell differentiation and grade, and presence of satellite nodules were reported to be prognostic biomarkers [25]

Summary

Introduction

Convolutional neural networks (CNNs), a state-of-the-art image analysis technique in computer vision, automatically learns representative features from such images which can be useful for disease diagnosis, prognosis, and subtyping. Histopathological images contain rich phenotypic descriptions of the molecular processes underlying disease progression and have been used for diagnosis, prognosis, and subtype discovery [1]. These images contain visual features including nuclear atypia, mitotic activity, cellular density, tissue architecture and higher-order patterns, which are examined by pathologists to diagnose and grade lesions. Convolutional neural networks (CNNs), a state-of-the-art image analysis technique in computer vision, automatically learns representative features from images and has been dominant since its astonishing results at the ImageNet Large Scale Visual Recognition Competition (ILSVRC) in 2012 [4]. Compared with traditional machine learning techniques, CNNs have witnessed significant advances in areas of image registration and localization, detection of anatomical and cellular structures, tissue segmentation, and computer-aided disease prognosis and diagnosis [11]

Methods

Results

Discussion

Conclusion