Histopathology Image Segmentation Research Articles

Real-time segmentation and classification of whole-slide images for tumor biomarker scoring

Histopathology image segmentation and classification are essential for diagnosing and treating breast cancer. This study introduced a highly accurate segmentation and classification for histopathology images using a single architecture. We utilized the famous segmentation architectures, SegNet and U-Net, and modified the decoder to attach ResNet, VGG and DenseNet to perform classification tasks. These hybrid models are integrated with Stardist as the backbone, and implemented in a real-time pathologist workflow with a graphical user interface. These models were trained and tested offline using the ER-IHC-stained private and H&E-stained public datasets (MoNuSeg). For real-time evaluation, the proposed model was evaluated using PR-IHC-stained glass slides. It achieved the highest segmentation pixel-based F1-score of 0.902 and 0.903 for private and public datasets respectively, and a classification-based F1-score of 0.833 for private dataset. The experiment shows the robustness of our method where a model trained on ER-IHC dataset able to perform well on real-time microscopy of PR-IHC slides on both 20x and 40x magnification. This will help the pathologists with a quick decision-making process.

WSSS-CRAM: precise segmentation of histopathological images via class region activation mapping.

Fast, accurate, and automatic analysis of histopathological images using digital image processing and deep learning technology is a necessary task. Conventional histopathological image analysis algorithms require the manual design of features, while deep learning methods can achieve fast prediction and accurate analysis, but rely on the drive of a large amount of labeled data. In this work, we introduce WSSS-CRAM a weakly-supervised semantic segmentation that can obtain detailed pixel-level labels from image-level annotated data. Specifically, we use a discriminative activation strategy to generate category-specific image activation maps via class labels. The category-specific activation maps are then post-processed using conditional random fields to obtain reliable regions that are directly used as ground-truth labels for the segmentation branch. Critically, the two steps of the pseudo-label acquisition and training segmentation model are integrated into an end-to-end model for joint training in this method. Through quantitative evaluation and visualization results, we demonstrate that the framework can predict pixel-level labels from image-level labels, and also perform well when testing images without image-level annotations. Future, we consider extending the algorithm to different pathological datasets and types of tissue images to validate its generalization capability.

Comparing nnU-Net and deepflash2 for Histopathological Tumor Segmentation.

Machine Learning (ML) has evolved beyond being a specialized technique exclusively used by computer scientists. Besides the general ease of use, automated pipelines allow for training sophisticated ML models with minimal knowledge of computer science. In recent years, Automated ML (AutoML) frameworks have become serious competitors for specialized ML models and have even been able to outperform the latter for specific tasks. Moreover, this success is not limited to simple tasks but also complex ones, like tumor segmentation in histopathological tissue, a very time-consuming task requiring years of expertise by medical professionals. Regarding medical image segmentation, the leading AutoML frameworks are nnU-Net and deepflash2. In this work, we begin to compare those two frameworks in the area of histopathological image segmentation. This use case proves especially challenging, as tumor and healthy tissue are often not clearly distinguishable by hard borders but rather through heterogeneous transitions. A dataset of 103 whole-slide images from 56 glioblastoma patients was used for the evaluation. Training and evaluation were run on a notebook with consumer hardware, determining the suitability of the frameworks for their application in clinical scenarios rather than high-performance scenarios in research labs.

Global–local consistent semi-supervised segmentation of histopathological image with different perturbations

A histopathological image is a microscopic image applied to examine cellular and tissue structures and identify any abnormalities or disease processes. Histopathological image segmentation is a prerequisite step for analyzing histopathological images that can divide an image into meaningful regions or objects to accurately classify and analyze tissue structures, cellular regions, or particular histological entities. However, the existing deep learning based pathological image segmentation methods require huge annotation efforts from the pathologists, which is labor-intensive and time-consuming. In this scenario, it has become a hotspot to leverage abundantly available unlabeled data to help learn segmentation models given limited labeled data. In this paper, we propose a global–local consistent semi-supervised segmentation (GLCS) model that enforce the consistency of the segmentation results with weak and strong perturbations on unlabeled data. In GLCS, we firstly generate different weak perturbations for each unlabeled sample, and then add a regularization term to ensure the segmentation consistency among different weak perturbations. Next, different from the existing studies applying the regression methods to match the segmentation results among different perturbations, our methods are based on the generative adversarial learning that can keep the global structure consistency among unlabeled data with different strength of perturbations. Finally, we also add a patch-correlation based regularization term to preserve the local structure similarity among different perturbations images. We validate our GLCS on three datasets, i.e. Glas, Crag and MoNuSeg. The experimental results show that our method can achieve to the dice ratio of 90.35, 82.61 and 81.60 with 1:1 proportion of labeled data, which are significantly superior to the state-of-the-art semi-supervised histopathological image segmentation methods. Our code is public available at https://github.com/ISBELLAG/GLCS.

Application of graph-curvature features in computer-aided diagnosis for histopathological image identification of gastric cancer

BackgroundHistopathology diagnosis is often regarded as the final diagnostic method for malignant tumors; however, it has some drawbacks. This study explored a computer-aided diagnostic method that can be used to identify benign and malignant gastric cancer using histopathological images. MethodsThe most suitable process was selected through multiple experiments by comparing multiple methods and features for classification. First, the U-net was applied to segment the image. Next, the nucleus was extracted from the segmented image, and the minimum spanning tree (MST) diagram structure that can capture the topological information was drawn. The third step was to extract the graph-curvature features of the histopathological image according to the MST image. Finally, by inputting the graph-curvature features into the classifier, the recognition results for benign or malignant cancer can be obtained. ResultsDuring the experiment, we used various methods for comparison. In the image segmentation stage, U-net, watershed algorithm, and Otsu threshold segmentation methods were used. We found that the U-net method, combined with multiple indicators, was the most suitable for segmentation of histopathological images. In the feature extraction stage, in addition to extracting graph-edge and graph-curvature features, several basic image features were extracted, including the red, green and blue feature, gray-level co-occurrence matrix feature, histogram of oriented gradient feature, and local binary pattern feature. In the classifier design stage, we experimented with various methods, such as support vector machine (SVM), random forest, artificial neural network, K nearest neighbors, VGG-16, and inception-V3. Through comparison and analysis, it was found that classification results with an accuracy of 98.57% can be obtained by inputting the graph-curvature feature into the SVM classifier. ConclusionThis study created a unique feature, the graph-curvature feature, based on the MST to represent and analyze histopathological images. This graph-based feature could be used to identify benign and malignant cells in histopathological images and assist pathologists in diagnosis.

Self-supervised multi-magnification feature enhancement for segmentation of hepatocellular carcinoma region in pathological images

Accurate segmentation of hepatocellular carcinoma regions is a critical step in surgical evaluation of whole slide pathological images. Recently, emerging multi-magnification learning-based methods have shown promise with evaluation of whole slide images. However, traditional multi-magnification learning segmentation models focus on uniformity and fail to effectively exploit different magnification information. The purpose of this study is to develop a novel multi-magnification learning segmentation model by effectively utilizing diverse magnification information. Therefore, we developed a novel multi-magnification feature self-enhancement network to extract magnification-specific features from multi-magnification images via self-supervised learning without extra supervision. Specifically, the proposed network enhances feature information via a super-resolution module based on self-supervision to transfer the representative information into a segmentation encoder, which takes full advantage of the pyramid storage characteristic of whole slide images. Moreover, a multi-scale feature fusion module based on the attention mechanism is used to fuse pretrained multi-scale features through a gated unit block, which is designed to fuse features from generation task into and segmentation task. We evaluated this method using the hepatocellular carcinoma data set from The Cancer Genome Atlas with cross-validation and achieved a 0.829 Dice similarity coefficient, nearly a 3% improvement over state-of-the-art models used for segmentation. To the best of our knowledge, this is the first study of the application of super-resolution as a self-enhancement network for histopathological image segmentation via self-supervised pretext learning. The code will be available at https://github.com/SH-Diao123/Self-Supervised-Multi-magnification-Segmentation.

Colorectal cancer prediction via histopathology segmentation using DC-GAN and VAE-GAN

Colorectal cancer ranks as the third most common form of cancer in the United States. The Centres of Disease Control and Prevention report that males and individuals assigned male at birth (AMAB) have a slightly higher incidence of colon cancer than females and those assigned female at birth (AFAB) Black humans are more likely than other ethnic groups or races to develop colon cancer. Early detection of suspicious tissues can improve a person's life for 3-4 years. In this project, we use the EBHI-seg dataset. This study explores a technique called Generative Adversarial Networks (GAN) that can be utilized for data augmentation colorectal cancer histopathology Image Segmentation. Specifically, we compare the effectiveness of two GAN models, namely the deep convolutional GAN (DC-GAN) and the Variational autoencoder GAN (VAE-GAN), in generating realistic synthetic images for training a neural network model for cancer prediction. Our findings suggest that DC-GAN outperforms VAE-GAN in generating high-quality synthetic images and improving the neural network model. These results highlight the possibility of GAN-based data augmentation to enhance machine learning models’ performance in medical image analysis tasks. The result shows DC-GAN outperformed VAE-GAN.

An automatic histopathological image segmentation network based on global context-aware module and deep feature aggregation

Histopathological analysis is the gold standard for cancer diagnosis, but traditional manual identification methods are time-consuming, labor-intensive, and highly relied on the experience of pathologists. In recent years, deep learning methods can assist pathologists to achieve accurate and automated detection of lesion areas. However, lesion detection is a difficult task due to its various size, shape, and surrounding complex scenes. Existing methods ignore the global interaction between lesion area and normal area, so it is insufficient to extract important global features. In addition, most methods fail to effectively fuse deep and shallow features, resulting in the extracted features lacking discrimination, and these deficiencies lead to incomplete extraction and irregular boundaries of lesion. To address the above-mentioned issues, an end-to-end histopathological image segmentation network based on global context-aware module and deep feature aggregation (GCMDFA) is proposed for segmenting lesions in histopathological images (HIs). In the GCMDFA, a global context-aware module is applied to better explore the interactive relationship between any two pixels to directly capture global contextual information related to lesion tissue of HIs for achieving global features. To better decode the features of encoder, a deep feature aggregation structure is designed to effectively fuse deep and shallow layer features of HIs. Experimental results on three HI datasets (including public and external data) show that the proposed GCMDFA method is more competitive compared with some state-of-the-art (SOTA) methods. This research can assist pathologists to analyze HIs in clinical computer-aided diagnosis.

WHRIME: A weight-based recursive hierarchical RIME optimizer for breast cancer histopathology image segmentation

In medical image processing, multi-threshold image segmentation has been challenging, as selecting appropriate thresholds is crucial for distinguishing different structures within an image, especially when dealing with breast cancer images. Breast cancer images are complex with multiple tissue types, which pose challenges to precise diagnosis. A weight-based recursive hierarchical bootstrapping rime algorithm (WHRIME) is proposed to tackle this case effectively. The proposed WHRIME segregates the population into elite and non-elite individuals. The hierarchical bootstrapping strategy ensures comprehensive exploration of the solution space, with elite individuals guiding the position updates of non-elite individuals. A weighted method is introduced based on solution quality differences to maintain population diversity and enhance convergence accuracy. We apply WHRIME in multi-threshold image segmentation on breast cancer histopathology images. Experimental results on both the IEEE CEC 2017 benchmark suit and breast cancer histopathology images from the Databiox dataset validate the superiority of WHRIME over competing algorithms, affirming WHRIME’s capability in addressing complex breast cancer histopathology image segmentation problems.

Learning from crowds for automated histopathological image segmentation

Automated semantic segmentation of histopathological images is an essential task in Computational Pathology (CPATH). The main limitation of Deep Learning (DL) to address this task is the scarcity of expert annotations. Crowdsourcing (CR) has emerged as a promising solution to reduce the individual (expert) annotation cost by distributing the labeling effort among a group of (non-expert) annotators. Extracting knowledge in this scenario is challenging, as it involves noisy annotations. Jointly learning the underlying (expert) segmentation and the annotators’ expertise is currently a commonly used approach. Unfortunately, this approach is frequently carried out by learning a different neural network for each annotator, which scales poorly when the number of annotators grows. For this reason, this strategy cannot be easily applied to real-world CPATH segmentation. This paper proposes a new family of methods for CR segmentation of histopathological images. Our approach consists of two coupled networks: a segmentation network (for learning the expert segmentation) and an annotator network (for learning the annotators’ expertise). We propose to estimate the annotators’ behavior with only one network that receives the annotator ID as input, achieving scalability on the number of annotators. Our family is composed of three different models for the annotator network. Within this family, we propose a novel modeling of the annotator network in the CR segmentation literature, which considers the global features of the image. We validate our methods on a real-world dataset of Triple Negative Breast Cancer images labeled by several medical students. Our new CR modeling achieves a Dice coefficient of 0.7827, outperforming the well-known STAPLE (0.7039) and being competitive with the supervised method with expert labels (0.7723). The code is available at https://github.com/wizmik12/CRowd_Seg.

GenSelfDiff-HIS: Generative Self-Supervision Using Diffusion for Histopathological Image Segmentation.

Histopathological image segmentation is a laborious and time-intensive task, often requiring analysis from experienced pathologists for accurate examinations. To reduce this burden, supervised machine-learning approaches have been adopted using large-scale annotated datasets for histopathological image analysis. However, in several scenarios, the availability of large-scale annotated data is a bottleneck while training such models. Self-supervised learning (SSL) is an alternative paradigm that provides some respite by constructing models utilizing only the unannotated data which is often abundant. The basic idea of SSL is to train a network to perform one or many pseudo or pretext tasks on unannotated data and use it subsequently as the basis for a variety of downstream tasks. It is seen that the success of SSL depends critically on the considered pretext task. While there have been many efforts in designing pretext tasks for classification problems, there have not been many attempts on SSL for histopathological image segmentation. Motivated by this, we propose an SSL approach for segmenting histopathological images via generative diffusion models. Our method is based on the observation that diffusion models effectively solve an image-to-image translation task akin to a segmentation task. Hence, we propose generative diffusion as the pretext task for histopathological image segmentation. We also utilize a multi-loss function-based fine-tuning for the downstream task. We validate our method using several metrics on two publicly available datasets along with a newly proposed head and neck (HN) cancer dataset containing Hematoxylin and Eosin (H&E) stained images along with annotations.

AIR-UNet++: a deep learning framework for histopathology image segmentation and detection

AIR-UNet++: a deep learning framework for histopathology image segmentation and detection

Inter- and intra-uncertainty based feature aggregation model for semi-supervised histopathology image segmentation

Acquiring pixel-level annotations is often limited in applications such as histology studies that require domain expertise. Various semi-supervised learning approaches have been developed to work with limited ground truth annotations, such as the popular teacher-student models. However, hierarchical prediction uncertainty within the student model (intra-uncertainty) and image prediction uncertainty (inter-uncertainty) have not been fully utilized by existing methods. To address these issues, we first propose a novel inter- and intra-uncertainty regularization method to measure and constrain both inter- and intra-inconsistencies in the teacher-student architecture. We also propose a new two-stage network with pseudo-mask guided feature aggregation (PG-FANet) as the segmentation model. The two-stage structure complements with the uncertainty regularization strategy to avoid introducing extra modules in solving uncertainties and the aggregation mechanisms enable multi-scale and multi-stage feature integration. Comprehensive experimental results over the MoNuSeg and CRAG datasets show that our PG-FANet outperforms other state-of-the-art methods and our semi-supervised learning framework yields competitive performance with a limited amount of labeled data.

HAMIL: High-Resolution Activation Maps and Interleaved Learning for Weakly Supervised Segmentation of Histopathological Images.

Semantic segmentation of histopathological images is important for automatic cancer diagnosis, and it is challenged by time-consuming and labor-intensive annotation process that obtains pixel-level labels for training. To reduce annotation costs, Weakly Supervised Semantic Segmentation (WSSS) aims to segment objects by only using image or patch-level classification labels. Current WSSS methods are mostly based on Class Activation Map (CAM) that usually locates the most discriminative object part with limited segmentation accuracy. In this work, we propose a novel two-stage weakly supervised segmentation framework based on High-resolution Activation Maps and Interleaved Learning (HAMIL). First, we propose a simple yet effective Classification Network with High-resolution Activation Maps (HAM-Net) that exploits a lightweight classification head combined with Multiple Layer Fusion (MLF) of activation maps and Monte Carlo Augmentation (MCA) to obtain precise foreground regions. Second, we use dense pseudo labels generated by HAM-Net to train a better segmentation model, where three networks with the same structure are trained with interleaved learning: The agreement between two networks is used to highlight reliable pseudo labels for training the third network, and at the same time, the two networks serve as teachers for guiding the third network via knowledge distillation. Extensive experiments on two public histopathological image datasets of lung cancer demonstrated that our proposed HAMIL outperformed state-of-the-art weakly supervised and noisy label learning methods, respectively. The code is available at https://github.com/HiLab-git/HAMIL.

MLAGG-Net: Multi-level aggregation and global guidance network for pancreatic lesion segmentation in histopathological images

Pancreatic cancer is one of the most aggressive and lethal malignancies in the world. Automatic and accurate segmentation of lesion regions from histopathological images is essential for disease diagnosis and analysis. Nevertheless, it remains a challenging task due to the complicated pathological manifestations of the lesion, including the ambiguity of boundaries, the heterogeneity of textures, and the large variations in morphology, size and location. To address these problems, we propose a multi-level aggregation and global guidance network. Specifically, we utilize a multi-level feature fusion and distribution module that aims to alleviate the scale diversity and complex boundary problems in lesions by aggregating fine-grained information of low-level features and semantic information of high-level features. The cross-attention fusion module is proposed to perform the aggregation of hierarchical multi-level features, which can absorb effective information and suppress redundant information from them. This paper also introduces a global information fusion module to guide the network to locate the lesion areas with inconsistent distribution between the internal and boundary regions more accurately for more complete segmentation results. We conduct experiments on our PANC dataset and the public GlaS challenge dataset to verify the effectiveness of the proposed network. The experimental results show that the proposed method can achieve better segmentation performance with 90.02% Dice and 82.07% Jaccard compared to other state-of-the-art methods on our PANC dataset while achieving competitive results on the GlaS challenge dataset.

Valuing vicinity: Memory attention framework for context-based semantic segmentation in histopathology.

The segmentation of histopathological whole slide images into tumourous and non-tumourous types of tissue is a challenging task that requires the consideration of both local and global spatial contexts to classify tumourous regions precisely. The identification of subtypes of tumour tissue complicates the issue as the sharpness of separation decreases and the pathologist's reasoning is even more guided by spatial context. However, the identification of detailed tissue types is crucial for providing personalized cancer therapies. Due to the high resolution of whole slide images, existing semantic segmentation methods, restricted to isolated image sections, are incapable of processing context information beyond. To take a step towards better context comprehension, we propose a patch neighbour attention mechanism to query the neighbouring tissue context from a patch embedding memory bank and infuse context embeddings into bottleneck hidden feature maps. Our memory attention framework (MAF) mimics a pathologist's annotation procedure - zooming out and considering surrounding tissue context. The framework can be integrated into any encoder-decoder segmentation method. We evaluate the MAF on two public breast cancer and liver cancer data sets and an internal kidney cancer data set using famous segmentation models (U-Net, DeeplabV3) and demonstrate the superiority over other context-integrating algorithms - achieving a substantial improvement of up to 17% on Dice score. The code is publicly available at https://github.com/tio-ikim/valuing-vicinity.

Dense Dilated Multi-Scale Supervised Attention-Guided Network for histopathology image segmentation

Over the last couple of decades, the introduction and proliferation of whole-slide scanners led to increasing interest in the research of digital pathology. Although manual analysis of histopathological images is still the gold standard, the process is often tedious and time consuming. Furthermore, manual analysis also suffers from intra- and interobserver variability. Separating structures or grading morphological changes can be difficult due to architectural variability of these images. Deep learning techniques have shown great potential in histopathology image segmentation that drastically reduces the time needed for downstream tasks of analysis and providing accurate diagnosis. However, few algorithms have clinical implementations. In this paper, we propose a new deep learning model Dense Dilated Multiscale Supervised Attention-Guided (D2MSA) Network for histopathology image segmentation that makes use of deep supervision coupled with a hierarchical system of novel attention mechanisms. The proposed model surpasses state-of-the-art performance while using similar computational resources. The performance of the model has been evaluated for the tasks of gland segmentation and nuclei instance segmentation, both of which are clinically relevant tasks to assess the state and progress of malignancy. Here, we have used histopathology image datasets for three different types of cancer. We have also performed extensive ablation tests and hyperparameter tuning to ensure the validity and reproducibility of the model performance. The proposed model is available at www.github.com/shirshabose/D2MSA-Net.

Multilevel Colonoscopy Histopathology Image Segmentation Using Particle Swarm Optimization Techniques.

Histopathology image segmentation is a challenging task in medical image processing. This work aims to segment lesion regions from colonoscopy histopathology images. Initially, the images are preprocessed and then segmented using the multilevel image thresholding technique. Multilevel thresholding is considered an optimization problem. Particle swarm optimization (PSO) and its variants, darwinian particle swarm optimization (DPSO), and fractional order darwinian particle swarm optimization (FODPSO) are used to solve the optimization problem and they generate the threshold values. The threshold values obtained are used to segment the lesion regions from the images of the colonoscopy tissue data set. Segmented images containing the lesion regions are then postprocessed to remove unnecessary regions. Experimental results reveal that the FODPSO algorithm with Otsu's discriminant criterion as the objective function achieves the best accuracy, Dice and Jaccard values of 0.89, 0.68 and 0.52, respectively, for the colonoscopy data set. The FODPSO algorithm also outperforms other optimization methods such as artificial bee colony and the firefly algorithms in terms of the accuracy, Dice and Jaccard values.

Deep learning for multi-class semantic segmentation enables colorectal cancer detection and classification in digital pathology images

In colorectal cancer (CRC), artificial intelligence (AI) can alleviate the laborious task of characterization and reporting on resected biopsies, including polyps, the numbers of which are increasing as a result of CRC population screening programs ongoing in many countries all around the globe. Here, we present an approach to address two major challenges in the automated assessment of CRC histopathology whole-slide images. We present an AI-based method to segment multiple (n=14) tissue compartments in the H &E-stained whole-slide image, which provides a different, more perceptible picture of tissue morphology and composition. We test and compare a panel of state-of-the-art loss functions available for segmentation models, and provide indications about their use in histopathology image segmentation, based on the analysis of (a) a multi-centric cohort of CRC cases from five medical centers in the Netherlands and Germany, and (b) two publicly available datasets on segmentation in CRC. We used the best performing AI model as the basis for a computer-aided diagnosis system that classifies colon biopsies into four main categories that are relevant pathologically. We report the performance of this system on an independent cohort of more than 1000 patients. The results show that with a good segmentation network as a base, a tool can be developed which can support pathologists in the risk stratification of colorectal cancer patients, among other possible uses. We have made the segmentation model available for research use on https://grand-challenge.org/algorithms/colon-tissue-segmentation/.

Weakly supervised histopathology image segmentation with self-attention.

Accurate segmentation in histopathology images at pixel-level plays a critical role in the digital pathology workflow. The development of weakly supervised methods for histopathology image segmentation liberates pathologists from time-consuming and labor-intensive works, opening up possibilities of further automated quantitative analysis of whole-slide histopathology images. As an effective subgroup of weakly supervised methods, multiple instance learning (MIL) has achieved great success in histopathology images. In this paper, we specially treat pixels as instances so that the histopathology image segmentation task is transformed into an instance prediction task in MIL. However, the lack of relations between instances in MIL limits the further improvement of segmentation performance. Therefore, we propose a novel weakly supervised method called SA-MIL for pixel-level segmentation in histopathology images. SA-MIL introduces a self-attention mechanism into the MIL framework, which captures global correlation among all instances. In addition, we use deep supervision to make the best use of information from limited annotations in the weakly supervised method. Our approach makes up for the shortcoming that instances are independent of each other in MIL by aggregating global contextual information. We demonstrate state-of-the-art results compared to other weakly supervised methods on two histopathology image datasets. It is evident that our approach has generalization ability for the high performance on both tissue and cell histopathology datasets. There is potential in our approach for various applications in medical images.