TransWS: Transformer-Based Weakly Supervised Histology Image Segmentation

Abstract

AbstractRecently, weakly supervised histology image segmentation has received increasingly more attentions. Most solutions utilize a convolutional neural network (CNN) as a classifier and treat the generated class activation map (CAM) as a pseudo annotation, based on which a segmentation network is trained in a supervised manner. This pipeline suffers from two disadvantages. First, the CNN classifier may fail to generate the high-quality CAM that highlights the exact and integral target, resulting in incomplete activation and blurred boundaries. Second, it splits the original problem into two, leading to a sub-optimal solution and low efficiency. To address both issues, we propose a Transformer-based weakly supervised segmentation (TransWS) method for histology images. TransWS is composed of a classification branch and a segmentation branch. The former learns semantic information from image-level annotations and uses CAM to generate pseudo pixel-level annotations. The latter performs the class-agnostic segmentation (CAS), i.e., binary segmentation, under the supervision of pseudo annotations. The semantic information and foreground region are combined to generate the final segmentation result. Comparing to CNN, Transformer is superior in modeling long-term dependencies and can generate more integral and accurate CAMs. More important, both branches in our TransWS can be jointly optimized in an end-to-end manner. We evaluated TransWS on the benchmark GlaS and Camelyon16-P512 datasets. Our results suggest that TransWS outperforms other weakly supervised segmentation competitors, setting a new state of the art.KeywordsWeakly supervised segmentationHistology imagesTransformerEnd-to-end optimization