TransPath: Transformer-Based Self-supervised Learning for Histopathological Image Classification

Abstract

A large-scale labeled dataset is a key factor for the success of supervised deep learning in histopathological image analysis. However, exhaustive annotation requires a careful visual inspection by pathologists, which is extremely time-consuming and labor-intensive. Self-supervised learning (SSL) can alleviate this issue by pre-training models under the supervision of data itself, which generalizes well to various downstream tasks with limited annotations. In this work, we propose a hybrid model (TransPath) which is pre-trained in an SSL manner on massively unlabeled histopathological images to discover the inherent image property and capture domain-specific feature embedding. The TransPath can serve as a collaborative local-global feature extractor, which is designed by combining a convolutional neural network (CNN) and a modified transformer architecture. We propose a token-aggregating and excitation (TAE) module which is placed behind the self-attention of the transformer encoder for capturing more global information. We evaluate the performance of pre-trained TransPath by fine-tuning it on three downstream histopathological image classification tasks. Our experimental results indicate that TransPath outperforms state-of-the-art vision transformer networks, and the visual representations generated by SSL on domain-relevant histopathological images are more transferable than the supervised baseline on ImageNet. Our code and pre-trained models will be available at https://github.com/Xiyue-Wang/TransPath.