Weakly-supervised Deep Learning Research Articles

Weakly-supervised deep learning models enable HER2-low prediction from H &E stained slides

BackgroundHuman epidermal growth factor receptor 2 (HER2)-low breast cancer has emerged as a new subtype of tumor, for which novel antibody–drug conjugates have shown beneficial effects. Assessment of HER2 requires several immunohistochemistry tests with an additional in situ hybridization test if a case is classified as HER2 2+. Therefore, novel cost-effective methods to speed up the HER2 assessment are highly desirable.MethodsWe used a self-supervised attention-based weakly supervised method to predict HER2-low directly from 1437 histopathological images from 1351 breast cancer patients. We built six distinct models to explore the ability of classifiers to distinguish between the HER2-negative, HER2-low, and HER2-high classes in different scenarios. The attention-based model was used to comprehend the decision-making process aimed at relevant tissue regions.ResultsOur results indicate that the effectiveness of classification models hinges on the consistency and dependability of assay-based tests for HER2, as the outcomes from these tests are utilized as the baseline truth for training our models. Through the use of explainable AI, we reveal histologic patterns associated with the HER2 subtypes.ConclusionOur findings offer a demonstration of how deep learning technologies can be applied to identify HER2 subgroup statuses, potentially enriching the toolkit available for clinical decision-making in oncology.

Enhancing Colorectal Cancer Tumor Bud Detection Using Deep Learning from Routine H&E-Stained Slides.

Tumor budding refers to a cluster of one to four tumor cells located at the tumor-invasive front. While tumor budding is a prognostic factor for colorectal cancer, counting and grading tumor budding are time consuming and not highly reproducible. There could be high inter- and intra-reader disagreement on H&E evaluation. This leads to the noisy training (imperfect ground truth) of deep learning algorithms, resulting in high variability and losing their ability to generalize on unseen datasets. Pan-cytokeratin staining is one of the potential solutions to enhance the agreement, but it is not routinely used to identify tumor buds and can lead to false positives. Therefore, we aim to develop a weakly-supervised deep learning method for tumor bud detection from routine H&E-stained images that does not require strict tissue-level annotations. We also propose Bayesian Multiple Instance Learning (BMIL) that combines multiple annotated regions during the training process to further enhance the generalizability and stability in tumor bud detection. Our dataset consists of 29 colorectal cancer H&E-stained images that contain 115 tumor buds per slide on average. In six-fold cross-validation, our method demonstrated an average precision and recall of 0.94, and 0.86 respectively. These results provide preliminary evidence of the feasibility of our approach in improving the generalizability in tumor budding detection using H&E images while avoiding the need for non-routine immunohistochemical staining methods.

Abstract 2313: Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images

Abstract Purpose: Cancer patients routinely undergo radiologic and pathologic evaluation for their diagnostic workup. These data modalities represent a valuable and readily available resource for developing new prognostic tools. Given their vast difference in spatial scales, effective methods to integrate the two modalities are currently lacking. Here, we aim to develop a multi-modal approach to integrate radiology and pathology images for predicting outcomes in cancer patients. Methods: We propose a multi-modal weakly-supervised deep learning framework to integrate radiology and pathology images for survival prediction. We first extract multi-scale features from whole-slide H&E-stained pathology images to characterize cellular and tissue phenotypes as well as spatial cellular organization. We then build a hierarchical co-attention transformer to effectively learn the multi-modal interactions between radiology and pathology image features. Finally, a multimodal risk score is derived by combining complementary information from two images modalities and clinical data for predicting outcome. We evaluate our approach in lung, gastric, and brain cancers with matched radiology and pathology images and clinical data available, each with separate training and external validation cohorts. Results: The multi-modal deep learning models achieved a reasonably high accuracy for predicting survival outcomes in the external validation cohorts (C-index range: 0.72-0.75 across three cancer types). The multi-modal prognostic models significantly improved upon single-modal approach based on radiology or pathology images or clinical data alone (C-index range: 0.53-0.71, P<0.01). The multi-modal deep learning models were significantly associated with disease-free survival and overall survival (hazard ratio range: 3.23-4.46, P<0.0001). In multivariable analyses, the models remained an independent prognostic factor (P<0.01) after adjusting for clinicopathological variables including cancer stage and tumor differentiation. Conclusions: The proposed multi-modal deep learning approach outperforms traditional methods for predicting survival outcomes by leveraging routinely available radiology and pathology images. With further independent validation, this may afford a promising approach to improve risk stratification and better inform treatment strategies for cancer patients. Citation Format: Zhe Li, Yuming Jiang, Ruijiang Li. Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 2313.

A weakly-supervised deep learning model for end-to-end detection of airfield pavement distress

Accurate and timely surveying of airfield pavement distress is crucial for cost-effective airport maintenance management. Deep learning (DL) approaches, leveraging advancements in computer science and image acquisition techniques, have become the mainstream for automated airfield pavement distress detection. However, fully-supervised DL methods require a large number of manually annotated ground truth labels to achieve high accuracy. To address the challenge of limited high-quality manual annotations, we propose a novel end-to-end distress detection model called WSDD-CAM (Class Activation Map informed Weakly-Supervised Distress Detection). Based on YOLOv5, WSDD-CAM consists of an efficient backbone, a classification branch, and a localization network. By utilizing CAM (Class Activation Map) information, our model significantly reduces the need for manual annotations, automatically generating pseudo bounding boxes with a 71% overlap with the ground truth. To evaluate WSDD-CAM, we tested it on a self-made dataset and compared it with other weakly-supervised and fully-supervised models. The results show that our model achieves 49.2% mean average precision (mAP), outperforming other weakly-supervised methods and even approaching state-of-the-art fully-supervised methods. Additionally, ablation experiments confirm the effectiveness of our architecture design. In conclusion, our WSDD-CAM model offers a promising solution for airfield pavement distress detection, reducing manual annotation time while maintaining high accuracy. This efficient and effective approach can significantly contribute to cost-effective airport maintenance management.

Artificial intelligence-assisted management of retinal detachment from ultra-widefield fundus images based on weakly-supervised approach.

Retinal detachment (RD) is a common sight-threatening condition in the emergency department. Early postural intervention based on detachment regions can improve visual prognosis. We developed a weakly supervised model with 24,208 ultra-widefield fundus images to localize and coarsely outline the anatomical RD regions. The customized preoperative postural guidance was generated for patients accordingly. The localization performance was then compared with the baseline model and an ophthalmologist according to the reference standard established by the retina experts. In the 48-partition lesion detection, our proposed model reached an 86.42% (95% confidence interval (CI): 85.81-87.01%) precision and an 83.27% (95%CI: 82.62-83.90%) recall with an average precision (PA) of 0.9132. In contrast, the baseline model achieved a 92.67% (95%CI: 92.11-93.19%) precision and limited recall of 68.07% (95%CI: 67.25-68.88%). Our holistic lesion localization performance was comparable to the ophthalmologist's 89.16% (95%CI: 88.75-89.55%) precision and 83.38% (95%CI: 82.91-83.84%) recall. As to the performance of four-zone anatomical localization, compared with the ground truth, the un-weighted Cohen's κ coefficients were 0.710(95%CI: 0.659-0.761) and 0.753(95%CI: 0.702-0.804) for the weakly-supervised model and the general ophthalmologist, respectively. The proposed weakly-supervised deep learning model showed outstanding performance comparable to that of the general ophthalmologist in localizing and outlining the RD regions. Hopefully, it would greatly facilitate managing RD patients, especially for medical referral and patient education.

Abstract B082: Early risk stratification of ER+/HER2– breast cancer patients using digital pathology and multi-task, weakly-supervised deep learning

Abstract Breast cancer prognostication guides treatment decisions, with transcriptomic assays like Prosigna and Oncotype DX providing valuable risk of recurrence (ROR) scores in ER+/HER2– patients. However, these tests are costly and require sufficient biopsy tissue for accuracy. Pathology images are routinely available and could expedite rapid risk stratification at lower cost. We propose a multi-modal, multi-task deep neural network that learns transcriptomic ROR using digital pathology images and hormone receptor expression status. Because the incorporation of both pathology images and clinical data has proven beneficial in other deep learning models, we hypothesized that incorporation of this complementary information would improve ROR predictions compared to using pathology images alone. We modified the “clustering-constrained-attention multiple-instance learning” (CLAM) deep learning method to perform multi-task regression and infer continuous risk scores (instead of discrete risk categories). We evaluated (1) PAM50 ROR-P, calculated using a multivariate model based on intrinsic subtype centroids and a proliferation score, and (2) a 21-gene assay recurrence score (RS) (research version of Oncotype DX), calculated from component scores relating to proliferation, ER, HER2, and invasion. Using data from 899 patients in the Carolina Breast Cancer Study, a population-based study of diverse patients including 50% Black women and 50% women under age 50, we constructed training (n=714), validation (n=93), and test (n=92) sets. Genomic data were assayed by Nanostring, and digital pathology was based on H&E-stained whole slide images (WSIs). Models were assessed using Pearson correlation between measured and predicted ROR-P or RS. The model with the highest correlations on the test set used H&E WSIs and ER/PR/HER2 expression status as inputs and generated 10 independent outputs: ROR-P, four PAM50 centroid correlations (i.e., Basal, HER2-enriched, Luminal A, and Luminal B), RS, and four RS component scores. Prediction was strong when considering all participants in the test set (ROR-P, 0.78; RS, 0.83, N, 92) but was reduced among more clinically homogeneous ER+/HER2– patients (ROR-P, 0.55; RS, 0.49; N, 49). For RS, correlations were higher in models that included ER/PR/HER2 (Overall, 0.83; ER+/HER2–, 0.50) than in those that relied on pathology alone (Overall, 0.60, ER+/HER2–, 0.34). Future models will consider strategies to enhance performance in ER+/HER2– patients, such as oversampling of ER+/HER2– patients and use of loss functions designed to focus learning on mis-predicted examples. In summary, multi-task, tissue-based regression deep learning models recapitulate transcription-based risk assays with high correlations and, with optimization (especially integration of additional clinicopathologic data), hold promise for personalized treatment decisions from early, routinely-collected biopsy images. Citation Format: Jakub R Kaczmarzyk, Luke A Torre-Healy, Richard A Moffitt, Rajarsi Gupta, Alina M Hamilton, Tahsin M Kurc, Katherine A Hoadley, Melissa A Troester, Joel H Saltz. Early risk stratification of ER+/HER2– breast cancer patients using digital pathology and multi-task, weakly-supervised deep learning [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Advances in Breast Cancer Research; 2023 Oct 19-22; San Diego, California. Philadelphia (PA): AACR; Cancer Res 2024;84(3 Suppl_1):Abstract nr B082.

A weakly supervised deep learning model integrating noncontrasted computed tomography images and clinical factors facilitates haemorrhagic transformation prediction after intravenous thrombolysis in acute ischaemic stroke patients

BackgroundHaemorrhage transformation (HT) is a serious complication of intravenous thrombolysis (IVT) in acute ischaemic stroke (AIS). Accurate and timely prediction of the risk of HT before IVT may change the treatment decision and improve clinical prognosis. We aimed to develop a deep learning method for predicting HT after IVT for AIS using noncontrast computed tomography (NCCT) images.MethodsWe retrospectively collected data from 828 AIS patients undergoing recombinant tissue plasminogen activator (rt-PA) treatment within a 4.5-h time window (n = 665) or of undergoing urokinase treatment within a 6-h time window (n = 163) and divided them into the HT group (n = 69) and non-HT group (n = 759). HT was defined based on the criteria of the European Cooperative Acute Stroke Study-II trial. To address the problems of indiscernible features and imbalanced data, a weakly supervised deep learning (WSDL) model for HT prediction was constructed based on multiple instance learning and active learning using admission NCCT images and clinical information in addition to conventional deep learning models. Threefold cross-validation and transfer learning were performed to confirm the robustness of the network. Of note, the predictive value of the commonly used scales in clinics associated with NCCT images (i.e., the HAT and SEDAN score) was also analysed and compared to measure the feasibility of our proposed DL algorithms.ResultsCompared to the conventional DL and ML models, the WSDL model had the highest AUC of 0.799 (95% CI 0.712–0.883). Significant differences were observed between the WSDL model and five ML models (P < 0.05). The prediction performance of the WSDL model outperforms the HAT and SEDAN scores at the optimal operating point (threshold = 1.5). Further subgroup analysis showed that the WSDL model performed better for symptomatic intracranial haemorrhage (AUC = 0.833, F1 score = 0.909).ConclusionsOur WSDL model based on NCCT images had relatively good performance for predicting HT in AIS and may be suitable for assisting in clinical treatment decision-making.

Weakly supervised deep learning-based concrete aggregates automatic segmentation for assessing separation degree

Vision-based automatic concrete aggregate segmentation helps to facilitate fast quantitative analysis of concrete aggregate shape, gradation, and uniformity. However, existing methods are mainly based on supervised deep learning. Since concrete surface images often contain numerous aggregates with irregular sizes and boundaries, this leads to a time-consuming and labor-intensive data annotation process. To reduce annotation costs, this study proposes a weakly supervised deep learning (WSDL) concrete aggregates segmentation method, which can train accurate pixel-level concrete aggregates segmentation models using only low-cost bounding box-level annotation. A concrete aggregate segmentation dataset containing 1400 images is created and experiments are conducted. The results show that the proposed method can achieve pixel-level concrete aggregate segmentation with an AP50 of 95.27 %. Compared with the supervised deep learning-based method, the annotation cost is reduced by 86.24 %. This study also verifies the feasibility of the proposed method for quantitative evaluation of concrete aggregate segregation degree. Expert systems for automatic analysis of concrete aggregate gradation and uniformity can be developed based on this method in the future.

AdvMIL: Adversarial multiple instance learning for the survival analysis on whole-slide images

The survival analysis on histological whole-slide images (WSIs) is one of the most important means to estimate patient prognosis. Although many weakly-supervised deep learning models have been developed for gigapixel WSIs, their potential is generally restricted by classical survival analysis rules and fully-supervised learning requirements. As a result, these models provide patients only with a completely-certain point estimation of time-to-event, and they could only learn from the labeled WSI data currently at a small scale. To tackle these problems, we propose a novel adversarial multiple instance learning (AdvMIL) framework. This framework is based on adversarial time-to-event modeling, and integrates the multiple instance learning (MIL) that is much necessary for WSI representation learning. It is a plug-and-play one, so that most existing MIL-based end-to-end methods can be easily upgraded by applying this framework, gaining the improved abilities of survival distribution estimation and semi-supervised learning. Our extensive experiments show that AdvMIL not only could often bring performance improvement to mainstream WSI survival analysis methods at a relatively low computational cost, but also enables these methods to effectively utilize unlabeled data via semi-supervised learning. Moreover, it is observed that AdvMIL could help improving the robustness of models against patch occlusion and two representative image noises. The proposed AdvMIL framework could promote the research of survival analysis in computational pathology with its novel adversarial MIL paradigm.

Large-scale investigation of weakly-supervised deep learning for the fine-grained semantic indexing of biomedical literature

Objective:Semantic indexing of biomedical literature is usually done at the level of MeSH descriptors with several related but distinct biomedical concepts often grouped together and treated as a single topic. This study proposes a new method for the automated refinement of subject annotations at the level of MeSH concepts. Methods:Lacking labelled data, we rely on weak supervision based on concept occurrence in the abstract of an article, which is also enhanced by dictionary-based heuristics. In addition, we investigate deep learning approaches, making design choices to tackle the particular challenges of this task. The new method is evaluated on a large-scale retrospective scenario, based on concepts that have been promoted to descriptors. Results:In our experiments concept occurrence was the strongest heuristic achieving a macro-F1 score of about 0.63 across several labels. The proposed method improved it further by more than 4pp. Conclusion: The results suggest that concept occurrence is a strong heuristic for refining the coarse-grained labels at the level of MeSH concepts and the proposed method improves it further.

Performance comparison between multi-center histopathology datasets of a weakly-supervised deep learning model for pancreatic ductal adenocarcinoma detection

BackgroundPancreatic ductal carcinoma patients have a really poor prognosis given its difficult early detection and the lack of early symptoms. Digital pathology is routinely used by pathologists to diagnose the disease. However, visually inspecting the tissue is a time-consuming task, which slows down the diagnostic procedure. With the advances occurred in the area of artificial intelligence, specifically with deep learning models, and the growing availability of public histology data, clinical decision support systems are being created. However, the generalization capabilities of these systems are not always tested, nor the integration of publicly available datasets for pancreatic ductal carcinoma detection (PDAC).MethodsIn this work, we explored the performace of two weakly-supervised deep learning models using the two more widely available datasets with pancreatic ductal carcinoma histology images, The Cancer Genome Atlas Project (TCGA) and the Clinical Proteomic Tumor Analysis Consortium (CPTAC). In order to have sufficient training data, the TCGA dataset was integrated with the Genotype-Tissue Expression (GTEx) project dataset, which contains healthy pancreatic samples.ResultsWe showed how the model trained on CPTAC generalizes better than the one trained on the integrated dataset, obtaining an inter-dataset accuracy of 90.62% ± 2.32 and an outer-dataset accuracy of 92.17% when evaluated on TCGA + GTEx. Furthermore, we tested the performance on another dataset formed by tissue micro-arrays, obtaining an accuracy of 98.59%. We showed how the features learned in an integrated dataset do not differentiate between the classes, but between the datasets, noticing that a stronger normalization might be needed when creating clinical decision support systems with datasets obtained from different sources. To mitigate this effect, we proposed to train on the three available datasets, improving the detection performance and generalization capabilities of a model trained only on TCGA + GTEx and achieving a similar performance to the model trained only on CPTAC.ConclusionsThe integration of datasets where both classes are present can mitigate the batch effect present when integrating datasets, improving the classification performance, and accurately detecting PDAC across different datasets.

Weakly supervised volumetric prostate registration for MRI-TRUS image driven by signed distance map

Image registration is a fundamental step for MRI-TRUS fusion targeted biopsy. Due to the inherent representational differences between these two image modalities, though, intensity-based similarity losses for registration tend to result in poor performance. To mitigate this, comparison of organ segmentations, functioning as a weak proxy measure of image similarity, has been proposed. Segmentations, though, are limited in their information encoding capabilities. Signed distance maps (SDMs), on the other hand, encode these segmentations into a higher dimensional space where shape and boundary information are implicitly captured, and which, in addition, yield high gradients even for slight mismatches, thus preventing vanishing gradients during deep-network training. Based on these advantages, this study proposes a weakly-supervised deep learning volumetric registration approach driven by a mixed loss that operates both on segmentations and their corresponding SDMs, and which is not only robust to outliers, but also encourages optimal global alignment. Our experimental results, performed on a public prostate MRI-TRUS biopsy dataset, demonstrate that our method outperforms other weakly-supervised registration approaches with a dice similarity coefficient (DSC), Hausdorff distance (HD) and mean surface distance (MSD) of 87.3 ± 11.3, 4.56 ± 1.95 mm, and 0.053 ± 0.026 mm, respectively. We also show that the proposed method effectively preserves the prostate gland's internal structure.

Cost-efficient information extraction from massive remote sensing data: When weakly supervised deep learning meets remote sensing big data

With many platforms and sensors continuously observing the earth surface, the large amount of remote sensing data presents a big data challenge. While remote sensing data acquisition capability can fully meet the requirements of many application domains, there is still a need to further explore how to efficiently mine the useful information from remote sensing big data (RSBD). Many researchers in the remote sensing community have introduced deep learning in the process of RSBD, and deep learning-based methods have achieved better performance compared with traditional methods. However, there are still substantial obstacles to the application of deep learning in remote sensing. One of the major challenges is the generation of pixel-level labels with high quality for training samples, which is essential to deep learning models. Weakly supervised deep learning (WSDL) is a promising solution to address this problem as WSDL can utilize greedily labeled datasets that are easy to collect but not ideal to train the deep networks. In this review, we summarize the achievements of WSDL-driven cost-efficient information extraction from RSBD. We first analyze the opportunities and challenges of information extraction from RSBD. Based on the analysis of the theoretical foundations of WSDL in the computer vision (CV) domain, we conduct a survey on the WSDL-based information extraction methods under the data characteristic and task demand of RSBD in four different tasks: (i) scene classification, (ii) object detection, (iii) semantic segmentation and (iv) change detection. Finally, potential research directions are outlined to guide researchers to further exploit WSDL-based information extraction from RSBD.

Predicting Lymph Node Metastasis Status from Primary Muscle-Invasive Bladder Cancer Histology Slides Using Deep Learning: A Retrospective Multicenter Study.

Accurate prediction of lymph node metastasis (LNM) status in patients with muscle-invasive bladder cancer (MIBC) before radical cystectomy can guide the use of neoadjuvant chemotherapy and the extent of pelvic lymph node dissection. We aimed to develop and validate a weakly-supervised deep learning model to predict LNM status from digitized histopathological slides in MIBC. We trained a multiple instance learning model with an attention mechanism (namely SBLNP) from a cohort of 323 patients in the TCGA cohort. In parallel, we collected corresponding clinical information to construct a logistic regression model. Subsequently, the score predicted by the SBLNP was incorporated into the logistic regression model. In total, 417 WSIs from 139 patients in the RHWU cohort and 230 WSIs from 78 patients in the PHHC cohort were used as independent external validation sets. In the TCGA cohort, the SBLNP achieved an AUROC of 0.811 (95% confidence interval [CI], 0.771-0.855), the clinical classifier achieved an AUROC of 0.697 (95% CI, 0.661-0.728) and the combined classifier yielded an improvement to 0.864 (95% CI, 0.827-0.906). Encouragingly, the SBLNP still maintained high performance in the RHWU cohort and PHHC cohort, with an AUROC of 0.762 (95% CI, 0.725-0.801) and 0.746 (95% CI, 0.687-0.799), respectively. Moreover, the interpretability of SBLNP identified stroma with lymphocytic inflammation as a key feature of predicting LNM presence. Our proposed weakly-supervised deep learning model can predict the LNM status of MIBC patients from routine WSIs, demonstrating decent generalization performance and holding promise for clinical implementation.

Few-shot weakly-supervised cybersecurity anomaly detection

With increased reliance on Internet based technologies, cyberattacks compromising users’ sensitive data are becoming more prevalent. The scale and frequency of these attacks are escalating rapidly, affecting systems and devices connected to the Internet. The traditional defense mechanisms may not be sufficiently equipped to handle the complex and ever-changing new threats. The significant breakthroughs in the machine learning methods including deep learning, had attracted interests from the cybersecurity research community for further enhancements in the existing anomaly detection methods. Unfortunately, collecting labelled anomaly data for all new evolving and sophisticated attacks is not practical. Training and tuning the machine learning model for anomaly detection using only a handful of labelled data samples is a pragmatic approach. Therefore, few-shot weakly supervised anomaly detection is an encouraging research direction. In this paper, we propose an enhancement to an existing few-shot weakly-supervised deep learning anomaly detection framework. This framework incorporates data augmentation, representation learning and ordinal regression. We then evaluated and showed the performance of our implemented framework on three benchmark datasets: NSL-KDD, CIC-IDS2018, and TON_IoT.

Weakly supervised deep learning to predict recurrence in low-grade endometrial cancer from multiplexed immunofluorescence images

Predicting recurrence in low-grade, early-stage endometrial cancer (EC) is both challenging and clinically relevant. We present a weakly-supervised deep learning framework, NaroNet, that can learn, without manual expert annotation, the complex tumor-immune interrelations at three levels: local phenotypes, cellular neighborhoods, and tissue areas. It uses multiplexed immunofluorescence for the simultaneous visualization and quantification of CD68 + macrophages, CD8 + T cells, FOXP3 + regulatory T cells, PD-L1/PD-1 protein expression, and tumor cells. We used 489 tumor cores from 250 patients to train a multilevel deep-learning model to predict tumor recurrence. Using a tenfold cross-validation strategy, our model achieved an area under the curve of 0.90 with a 95% confidence interval of 0.83–0.95. Our model predictions resulted in concordance for 96,8% of cases (κ = 0.88). This method could accurately assess the risk of recurrence in EC, outperforming current prognostic factors, including molecular subtyping.

Automatic diagnosis and grading of Prostate Cancer with weakly supervised learning on whole slide images

Background:The workflow of prostate cancer diagnosis and grading is cumbersome and the results suffer from substantial inter-observer variability. Recent trials have shown potential in using machine learning to develop automated systems to address this challenge. Most automated deep learning systems for prostate cancer Gleason grading focused on supervised learning requiring demanding fine-grained pixel-level annotations. Methods:A weakly-supervised deep learning model with slide-level labels is presented in this study for the diagnosis and grading of prostate cancer with whole slide image (WSI). WSIs are first cropped into small patches and then processed with a deep learning model to extract patch-level features. A graph convolution network (GCN) is used to aggregate the features for classifications. Throughout the training process, the noisy labels are progressively filtered out to reduce inter-observer variations in clinical reports. Finally, multi-center independent test cohorts with 6,174 slides are collected to evaluate the prostate cancer diagnosis and grading performance of our model. Results:The cancer diagnosis (2-level classification) results on two external test sets (n= 4,675, n= 844) show an area under the receiver operating characteristic curve (AUC) of 0.985 and 0.986. The Gleason grading (6-level classification) results reach 0.931 quadratic weighted kappa on the internal test set (n= 531). It generalizes well on the external test dataset (n= 844) with 0.801 quadratic weighted kappa with the reference standard set independently. The model enables pathological meaningful interpretability by visualizing the most attended lesions which are highly consistent with expert annotations. Conclusion:The proposed model incorporates a graph network in weakly supervised learning with only slide-level reports. A robust learning strategy is also employed to correct the label noise. It is highly accurate (>0.985 AUC for diagnosis) and also interpretable with intuitive heatmap visualization. It can be unified with a digital pathology pipeline to deliver prostate cancer metrics for a pathology report.

Land Use and Land Cover Mapping Using Deep Learning Based Segmentation Approaches and VHR Worldview-3 Images

Deep learning-based segmentation of very high-resolution (VHR) satellite images is a significant task providing valuable information for various geospatial applications, specifically for land use/land cover (LULC) mapping. The segmentation task becomes more challenging with the increasing number and complexity of LULC classes. In this research, we generated a new benchmark dataset from VHR Worldview-3 images for twelve distinct LULC classes of two different geographical locations. We evaluated the performance of different segmentation architectures and encoders to find the best design to create highly accurate LULC maps. Our results showed that the DeepLabv3+ architecture with an ResNeXt50 encoder achieved the best performance for different metric values with an IoU of 89.46%, an F-1 score of 94.35%, a precision of 94.25%, and a recall of 94.49%. This design could be used by other researchers for LULC mapping of similar classes from different satellite images or for different geographical regions. Moreover, our benchmark dataset can be used as a reference for implementing new segmentation models via supervised, semi- or weakly-supervised deep learning models. In addition, our model results can be used for transfer learning and generalizability of different methodologies.

Development and Evaluation of a Novel Deep-Learning-Based Framework for the Classification of Renal Histopathology Images.

Kidney cancer has several types, with renal cell carcinoma (RCC) being the most prevalent and severe type, accounting for more than 85% of adult patients. The manual analysis of whole slide images (WSI) of renal tissues is the primary tool for RCC diagnosis and prognosis. However, the manual identification of RCC is time-consuming and prone to inter-subject variability. In this paper, we aim to distinguish between benign tissue and malignant RCC tumors and identify the tumor subtypes to support medical therapy management. We propose a novel multiscale weakly-supervised deep learning approach for RCC subtyping. Our system starts by applying the RGB-histogram specification stain normalization on the whole slide images to eliminate the effect of the color variations on the system performance. Then, we follow the multiple instance learning approach by dividing the input data into multiple overlapping patches to maintain the tissue connectivity. Finally, we train three multiscale convolutional neural networks (CNNs) and apply decision fusion to their predicted results to obtain the final classification decision. Our dataset comprises four classes of renal tissues: non-RCC renal parenchyma, non-RCC fat tissues, clear cell RCC (ccRCC), and clear cell papillary RCC (ccpRCC). The developed system demonstrates a high classification accuracy and sensitivity on the RCC biopsy samples at the slide level. Following a leave-one-subject-out cross-validation approach, the developed RCC subtype classification system achieves an overall classification accuracy of 93.0% ± 4.9%, a sensitivity of 91.3% ± 10.7%, and a high classification specificity of 95.6% ± 5.2%, in distinguishing ccRCC from ccpRCC or non-RCC tissues. Furthermore, our method outperformed the state-of-the-art Resnet-50 model.

Benchmarking weakly-supervised deep learning pipelines for whole slide classification in computational pathology.

Artificial intelligence (AI) can extract visual information from histopathological slides and yield biological insight and clinical biomarkers. Whole slide images are cut into thousands of tiles and classification problems are often weakly-supervised: the ground truth is only known for the slide, not for every single tile. In classical weakly-supervised analysis pipelines, all tiles inherit the slide label while in multiple-instance learning (MIL), only bags of tiles inherit the label. However, it is still unclear how these widely used but markedly different approaches perform relative to each other. We implemented and systematically compared six methods in six clinically relevant end-to-end prediction tasks using data from N=2980 patients for training with rigorous external validation. We tested three classical weakly-supervised approaches with convolutional neural networks and vision transformers (ViT) and three MIL-based approaches with and without an additional attention module. Our results empirically demonstrate that histological tumor subtyping of renal cell carcinoma is an easy task in which all approaches achieve an area under the receiver operating curve (AUROC) of above 0.9. In contrast, we report significant performance differences for clinically relevant tasks of mutation prediction in colorectal, gastric, and bladder cancer. In these mutation prediction tasks, classical weakly-supervised workflows outperformed MIL-based weakly-supervised methods for mutation prediction, which is surprising given their simplicity. This shows that new end-to-end image analysis pipelines in computational pathology should be compared to classical weakly-supervised methods. Also, these findings motivate the development of new methods which combine the elegant assumptions of MIL with the empirically observed higher performance of classical weakly-supervised approaches. We make all source codes publicly available at https://github.com/KatherLab/HIA, allowing easy application of all methods to any similar task.