Group affinity weakly supervised segmentation from prior selected tissue in colorectal histopathology images

Histopathological image classification (HIC) and content-based histopathological image retrieval (CBHIR) are two promising applications for the histopathological whole slide image (WSI) analysis. HIC can efficiently predict the type of lesion involved in a histopathological image. In general, HIC can aid pathologists in locating high-risk cancer regions from a WSI by providing a cancerous probability map for the WSI. In contrast, CBHIR was developed to allow searches for regions with similar content for a region of interest (ROI) from a database consisting of historical cases. Sets of cases with similar content are accessible to pathologists, which can provide more valuable references for diagnosis. A drawback of the recent CBHIR framework is that a query ROI needs to be manually selected from a WSI. An automatic CBHIR approach for a WSI-wise analysis needs to be developed. In this paper, we propose a novel aided-diagnosis framework of breast cancer using whole slide images, which shares the advantages of both HIC and CBHIR. In our framework, CBHIR is automatically processed throughout the WSI, based on which a probability map regarding the malignancy of breast tumors is calculated. Through the probability map, the malignant regions in WSIs can be easily recognized. Furthermore, the retrieval results corresponding to each sub-region of the WSIs are recorded during the automatic analysis and are available to pathologists during their diagnosis. Our method was validated on fully annotated WSI data sets of breast tumors. The experimental results certify the effectiveness of the proposed method.

Gliomas are primary brain tumors caused by glial cells. These cancers’ classification and grading are crucial for prognosis and treatment planning. Deep learning (DL) can potentially improve the digital pathology investigation of brain tumors. In this paper, we developed a technique for visualizing a predictive tumor grading model on histopathology pictures to help guide doctors by emphasizing characteristics and heterogeneity in forecasts. The proposed technique is a hybrid model based on YOLOv5 and ResNet50. The function of YOLOv5 is to localize and classify the tumor in large histopathological whole slide images (WSIs). The suggested technique incorporates ResNet into the feature extraction of the YOLOv5 framework, and the detection results show that our hybrid network is effective for identifying brain tumors from histopathological images. Next, we estimate the glioma grades using the extreme gradient boosting classifier. The high-dimensional characteristics and nonlinear interactions present in histopathology images are well-handled by this classifier. DL techniques have been used in previous computer-aided diagnosis systems for brain tumor diagnosis. However, by combining the YOLOv5 and ResNet50 architectures into a hybrid model specifically designed for accurate tumor localization and predictive grading within histopathological WSIs, our study presents a new approach that advances the field. By utilizing the advantages of both models, this creative integration goes beyond traditional techniques to produce improved tumor localization accuracy and thorough feature extraction. Additionally, our method ensures stable training dynamics and strong model performance by integrating ResNet50 into the YOLOv5 framework, addressing concerns about gradient explosion. The proposed technique is tested using the cancer genome atlas dataset. During the experiments, our model outperforms the other standard ways on the same dataset. Our results indicate that the proposed hybrid model substantially impacts tumor subtype discrimination between low-grade glioma (LGG) II and LGG III. With 97.2% of accuracy, 97.8% of precision, 98.6% of sensitivity, and the Dice similarity coefficient of 97%, the proposed model performs well in classifying four grades. These results outperform current approaches for identifying LGG from high-grade glioma and provide competitive performance in classifying four categories of glioma in the literature.

Automatic histopathological Whole Slide Image (WSI) analysis for cancer classification has been highlighted along with the advancements in microscopic imaging techniques, since manual examination and diagnosis with WSIs are time- and cost-consuming. Recently, deep convolutional neural networks have succeeded in histopathological image analysis. However, despite the success of the development, there are still opportunities for further enhancements. In this paper, we propose a novel cancer texture-based deep neural network (CAT-Net) that learns scalable morphological features from histopathological WSIs. The innovation of CAT-Net is twofold: (1) capturing invariant spatial patterns by dilated convolutional layers and (2) improving predictive performance while reducing model complexity. Moreover, CAT-Net can provide discriminative morphological (texture) patterns formed on cancerous regions of histopathological images comparing to normal regions. We elucidated how our proposed method, CAT-Net, captures morphological patterns of interest in hierarchical levels in the model. The proposed method out-performed the current state-of-the-art benchmark methods on accuracy, precision, recall, and F1 score.

Ductal carcinoma in situ (DCIS) is considered a pre-invasive breast cancer and sometimes it can develop into an invasive ductal carcinoma. The analysis of histopathological images to detect tumour border of DCIS could provide important information for better diagnosis of patients. We present a deep learning based system to automatically identify DCIS in histopathological images. Specifically, a convolutional neural network (CNN) is first trained to predict labels of small patches cropped out of a histopathological whole slide image. Next, a sliding window method is used to produce a probability map of DCIS. Finally, given the probability map, a tumor border of DCIS is produced and delineated with the method of Marching Cubes to facilitate pathologists’ review and assessment. Evaluation of cross validation demonstrates that the CNN model of GoogleNet performs well in histology image patch classification with an overall accuracy of (98.46±0.40)% and identifies the DCIS tissue patches with a F1-score of (97.40±1.18)% (mean±variance). Moreover, around 95.6% tumour tissue within the enclosed tumour regions can be identified by our developed method. Finally, the goal of tumor border detection can be well achieved with a few post-processing steps.

Encoding histopathology whole slide images with location-aware graphs for diagnostically relevant regions retrieval.

Colorectal cancer classification using weakly annotated whole slide images: Multiple instance learning optimization study.

In digital pathology, computer-assisted techniques are acquired to view, manage, and analyze images taken under a microscope, commonly referred to as whole slide images. Using artificial intelligence-based algorithms for automatic diagnosis, the field is rapidly evolving. An important subfield of automatic diagnosis is the identification of cells in mitosis in whole slide images. Mitotic score determines tumour aggressiveness by grading histopathological images. Mitosis score is a critical component of treatment decisions based on histopathological images. Mitosis counting manually is extremely tedious, but automated methods can eliminate inefficiencies and subjectivity. A mitosis detection algorithm based on an ensemble of segmentation and detection methods is presented in this thesis. Using deep learning methods, we implement an ensemble algorithm while overcoming inadequate and complex training data. The proposed deep learning pipeline comprises regions of interest locator, an adversarial network for the segmentation, a regional convolutional network method for detecting cells in mitosis, and an ensemble model to incorporate segmentation and detection models. Mitosis results from segmentation and detection parts are fused by merging all the predictions from multiple models using weighted box fusion method. In order to improve the performance and capabilities of the model, techniques such as augmentation, normalization, and sampling are considered. In addition, to have a pixel-based annotation dataset, a semi-supervised pseudo labelling method is considered. We achieved promising results, demonstrating the power of the proposed machine learning and data enhancement methods.

In digital pathology, computer-assisted techniques are acquired to view, manage, and analyze images taken under a microscope, commonly referred to as whole slide images. Using artificial intelligence-based algorithms for automatic diagnosis, the field is rapidly evolving. An important subfield of automatic diagnosis is the identification of cells in mitosis in whole slide images. Mitotic score determines tumour aggressiveness by grading histopathological images. Mitosis score is a critical component of treatment decisions based on histopathological images. Mitosis counting manually is extremely tedious, but automated methods can eliminate inefficiencies and subjectivity. A mitosis detection algorithm based on an ensemble of segmentation and detection methods is presented in this thesis. Using deep learning methods, we implement an ensemble algorithm while overcoming inadequate and complex training data. The proposed deep learning pipeline comprises regions of interest locator, an adversarial network for the segmentation, a regional convolutional network method for detecting cells in mitosis, and an ensemble model to incorporate segmentation and detection models. Mitosis results from segmentation and detection parts are fused by merging all the predictions from multiple models using weighted box fusion method. In order to improve the performance and capabilities of the model, techniques such as augmentation, normalization, and sampling are considered. In addition, to have a pixel-based annotation dataset, a semi-supervised pseudo labelling method is considered. We achieved promising results, demonstrating the power of the proposed machine learning and data enhancement methods.

A study about color normalization methods for histopathology images

BackgroundSince nuclei segmentation in histopathology images can provide key information for identifying the presence or stage of a disease, the images need to be assessed carefully. However, color variation in histopathology images, and various structures of nuclei are two major obstacles in accurately segmenting and analyzing histopathology images. Several machine learning methods heavily rely on hand-crafted features which have limitations due to manual thresholding.ResultsTo obtain robust results, deep learning based methods have been proposed. Deep convolutional neural networks (DCNN) used for automatically extracting features from raw image data have been proven to achieve great performance. Inspired by such achievements, we propose a nuclei segmentation method based on DCNNs. To normalize the color of histopathology images, we use a deep convolutional Gaussian mixture color normalization model which is able to cluster pixels while considering the structures of nuclei. To segment nuclei, we use Mask R-CNN which achieves state-of-the-art object segmentation performance in the field of computer vision. In addition, we perform multiple inference as a post-processing step to boost segmentation performance. We evaluate our segmentation method on two different datasets. The first dataset consists of histopathology images of various organ while the other consists histopathology images of the same organ. Performance of our segmentation method is measured in various experimental setups at the object-level and the pixel-level. In addition, we compare the performance of our method with that of existing state-of-the-art methods. The experimental results show that our nuclei segmentation method outperforms the existing methods.ConclusionsWe propose a nuclei segmentation method based on DCNNs for histopathology images. The proposed method which uses Mask R-CNN with color normalization and multiple inference post-processing provides robust nuclei segmentation results. Our method also can facilitate downstream nuclei morphological analyses as it provides high-quality features extracted from histopathology images.

The advent of whole slide imaging has brought advanced computer-aided diagnosis via medical imaging and artificial intelligence technologies in digital pathology. The examination of tissue samples through whole slide imaging is commonly used to diagnose cancerous diseases, but the analysis of histopathology images through a decision support system is not always accurate due to variations in color caused by different scanning equipment, staining methods, and tissue reactivity. These variabilities decrease the accuracy of computer-aided diagnosis and affect the diagnosis of pathologists. In this context, an effective stain normalization method has proved as a powerful tool to standardize different color appearances and minimize color variations in histopathology images. This study reviews different stain normalization methods highlighting the main methodologies, contributions, advantages, and limitations of correlated works. The state-of-the-art methods are grouped into four distinct categories. Next, we select ten representative methods from the groups and conduct an experimental comparison to investigate the strengths and weaknesses of different methods and rank them according to selected performance accuracy measures. The quality performances of selected methods are compared in terms of quaternion structure similarity index metric, structural similarity index metric, and Pearson correlation coefficient conducting experiments on three histopathological image datasets. Our findings conclude that the structure-preserving unified transformation-based methods consistently outperform the state-of-the-art methods by improving robustness against variability and reproducibility. The comparative analysis we conducted in this paper will serve as the basis for future research, which will help to refine existing techniques and develop new approaches to address the complexities of stain normalization in complex histopathology images.

Sub-scene segmentation using constraints based on Gestalt principles

Aerial images has different data characteristics when compared to other types of images. An aerial image usually contains small insignificant objects that can cause errors in the unsupervised segmentation method. K-means clustering, one of the widely used unsupervised image segmentation methods, is highly vulnerable to local optima. In this study, Adaptive Fireworks Algorithm (AFWA) is proposed as an alternative to the K-means algorithm in optimizing the clustering process in the cluster-based segmentation method. AFWA is then applied to perform aerial image segmentation and the results are compared with K-means. Based on the comparison using Probabilistic Rand Index (PRI) and Variation of Information (VI) evaluation metrics, AFWA produces an overall better segmentation quality.

Image segmentation by fusion of low level and domain specific information via Markov Random Fields

Analysis of high-resolution histopathology whole slide images (WSIs) is a vital step in diagnosis and treatment of many diseases, including placenta-mediated diseases (PMDs) and respiratory illnesses. Currently, the most trusted approach is manual/semi-automated analysis of histopathology WSIs by an expert pathologist. This is problematic because high-resolution histopathology WSIs usually have a very large size (e.g., 80,000×80,000 pixels) and a large number of complex biological structures. As such, applying manual/semi-automated approaches to assess histopathology WSIs can be inefficient, expensive, and subject inter- and intra-rater variability. An alternative approach to manual and semi-automated approaches is to implement machine learning and image processing techniques to develop automated histopathology image analysis (AHIA) pipelines. A fundamental step in generating accurate AHIA approaches is semantic segmentation of biological structures in high-resolution histopathology images. In this thesis, our main objective is to develop accurate AHIA pipelines for semantic segmentation of complex biological structures in histopathology WSIs. We specifically focus on two histopathology applications: 1) segmentation of villi in histopathology WSIs of human placenta and 2) segmentation of complex biological structures of mouse lung tissue. Initially, we investigate the rule-based methods using conventional machine learning and image processing methods for segmentation of histopathology WSIs, developing two separate pipelines to address the challenges associated with each of our applications. We demonstrate that rule-based AHIA approaches show promising performance for analysis of histopathology WSIs in each of applications in comparison to manual assessment by expert pathologists and can be considered as a potential replacement for manual/semi-automated approaches. Then, we investigate deep learning methods in semantic segmentation of histopathology WSIs to further improve our developed rule-based approaches in terms of segmentation performance, generalizability, and training and testing speed. One of the bottlenecks in developing deep learning methods is the large size of histopathology WSIs, which requires implementing a patch-based approach to feed the images to deep learning models. We demonstrate that this bottleneck may limit the performance of the deep learning models due to three-fold trade-off between field-of-view, computational efficiency, and spatial resolution. As such, we propose a multi-resolution semantic segmentation pipeline to address this trade-off in AHIA using deep learning.