Image Tiles Research Articles

ScabyNet, a user-friendly application for detecting common scab in potato tubers using deep learning and morphological traits

Common scab (CS) is a major bacterial disease causing lesions on potato tubers, degrading their appearance and reducing their market value. To accurately grade scab-infected potato tubers, this study introduces “ScabyNet”, an image processing approach combining color-morphology analysis with deep learning techniques. ScabyNet estimates tuber quality traits and accurately detects and quantifies CS severity levels from color images. It is presented as a standalone application with a graphical user interface comprising two main modules. One module identifies and separates tubers on images and estimates quality-related morphological features. In addition, it enables the extraction of tubers as standard tiles for the deep-learning module. The deep-learning module detects and quantifies the scab infection into five severity classes related to the relative infected area. The analysis was performed on a dataset of 7154 images of individual tiles collected from field and glasshouse experiments. Combining the two modules yields essential parameters for quality and disease inspection. The first module simplifies imaging by replacing the region proposal step of instance segmentation networks. Furthermore, the approach is an operational tool for an affordable phenotyping system that selects scab-resistant genotypes while maintaining their market standards.

PathEX: Make good choice for whole slide image extraction.

The tile-based approach has been widely used for slide-level predictions in whole slide image (WSI) analysis. However, the irregular shapes and variable dimensions of tumor regions pose challenges for the process. To address this issue, we proposed PathEX, a framework that integrates intersection over tile (IoT) and background over tile (BoT) algorithms to extract tile images around boundaries of annotated regions while excluding the blank tile images within these regions. We developed PathEX, which incorporated IoT and BoT into tile extraction, for training a classification model in CAM (239 WSIs) and PAIP (40 WSIs) datasets. By adjusting the IoT and BoT parameters, we generated eight training sets and corresponding models for each dataset. The performance of PathEX was assessed on the testing set comprising 13,076 tile images from 48 WSIs of CAM dataset and 6,391 tile images from 10 WSIs of PAIP dataset. PathEX could extract tile images around boundaries of annotated region differently by adjusting the IoT parameter, while exclusion of blank tile images within annotated regions achieved by setting the BoT parameter. As adjusting IoT from 0.1 to 1.0, and 1-BoT from 0.0 to 0.5, we got 8 train sets. Experimentation revealed that set C demonstrates potential as the most optimal candidate. Nevertheless, a combination of IoT values ranging from 0.2 to 0.5 and 1-BoT values ranging from 0.2 to 0.5 also yielded favorable outcomes. In this study, we proposed PathEX, a framework that integrates IoT and BoT algorithms for tile image extraction at the boundaries of annotated regions while excluding blank tiles within these regions. Researchers can conveniently set the thresholds for IoT and BoT to facilitate tile image extraction in their own studies. The insights gained from this research provide valuable guidance for tile image extraction in digital pathology applications.

Neighborhood attention transformer multiple instance learning for whole slide image classification.

Pathologists rely on whole slide images (WSIs) to diagnose cancer by identifying tumor cells and subtypes. Deep learning models, particularly weakly supervised ones, classify WSIs using image tiles but may overlook false positives and negatives due to the heterogeneous nature of tumors. Both cancerous and healthy cells can proliferate in patterns that extend beyond individual tiles, leading to errors at the tile level that result in inaccurate tumor-level classifications. To address this limitation, we introduce NATMIL (Neighborhood Attention Transformer Multiple Instance Learning), which utilizes the Neighborhood Attention Transformer to incorporate contextual dependencies among WSI tiles. NATMIL enhances multiple instance learning by integrating a broader tissue context into the model. Our approach enhances the accuracy of tumor classification by considering the broader tissue context, thus reducing errors associated with isolated tile analysis. We conducted a quantitative analysis to evaluate NATMIL's performance against other weakly supervised algorithms. When applied to subtyping non-small cell lung cancer (NSCLC) and lymph node (LN) tumors, NATMIL demonstrated superior accuracy. Specifically, NATMIL achieved accuracy values of 89.6% on the Camelyon dataset and 88.1% on the TCGA-LUSC dataset, outperforming existing methods. These results underscore NATMIL's potential as a robust tool for improving the precision of cancer diagnosis using WSIs. Our findings demonstrate that NATMIL significantly improves tumor classification accuracy by reducing errors associated with isolated tile analysis. The integration of contextual dependencies enhances the precision of cancer diagnosis using WSIs, highlighting NATMILs´ potential as a robust tool in pathology.

Single-cell Heterogeneity-aware Transformer-guided Multiple Instance Learning for Cancer Aneuploidy Prediction from Whole Slide Histopathology Images.

Aneuploidy is a hallmark of aggressive malignancies associated with therapeutic resistance and poor survival. Measuring aneuploidy requires expensive specialized techniques that are not clinically applicable. Deep learning analysis of routine histopathology slides has revealed associations with genetic mutations. However, existing studies focus on image patches or tiles, and there is no prior work that predicts aneuploidy using single-cell analysis. Here, we present a single-cell heterogeneity-aware and transformer-guided deep learning framework to predict aneuploidy from whole slide histopathology images. First, we perform nuclei segmentation and classification to obtain individual cancer cells, which are clustered into multiple subtypes. The cell subtype distributions are computed to measure cancer cell heterogeneity. Additionally, morphological features of different cell subtypes are extracted. Further, we leverage a multiple instance learning module with Transformer, which encourages the network to focus on the most informative cancer cells. Lastly, a hybrid network is built to unify cell heterogeneity, morphology, and deep features for aneuploidy prediction. We train and validate our method on two public datasets from TCGA: lung adenocarcinoma (LUAD) and head and neck squamous cell carcinoma (HNSC), with 339 and 245 patients. Our model achieves promising performance with AUC of 0.818 (95% CI: 0.718-0.919) and 0.827 (95% CI: 0.704-0.949) on the LUAD and HNSC test sets, respectively. Through extensive ablation and comparison studies, we demonstrate the effectiveness of each component of the model and superior performance over alternative networks. In conclusion, we present a novel deep learning approach to predict aneuploidy from histopathology images, which could inform personalized cancer treatment.

Weakly Supervised Classification for Nasopharyngeal Carcinoma with Transformer in Whole Slide Images.

Pathological examination of nasopharyngeal carcinoma (NPC) is an indispensable factor for diagnosis, guiding clinical treatment and judging prognosis. Traditional and fully supervised NPC diagnosis algorithms require manual delineation of regions of interest on the gigapixel of whole slide images (WSIs), which however is laborious and often biased. In this paper, we propose a weakly supervised framework based on Tokens-to-Token Vision Transformer (WS-T2T-ViT) for accurate NPC classification with only a slide-level label. The label of tile images is inherited from their slide-level label. Specifically, WS-T2T-ViT is composed of the multi-resolution pyramid, T2T-ViT and multi-scale attention module. The multi-resolution pyramid is designed for imitating the coarse-to-fine process of manual pathological analysis to learn features from different magnification levels. The T2T module captures the local and global features to overcome the lack of global information. The multi-scale attention module improves classification performance by weighting the contributions of different granularity levels. Extensive experiments are performed on the 802-patient NPC and CAMELYON16 dataset. WS-T2T-ViT achieves an area under the receiver operating characteristic curve (AUC) of 0.989 for NPC classification on the NPC dataset. The experiment results of CAMELYON16 dataset demonstrate the robustness and generalizability of WS-T2T-ViT in WSI-level classification.

Classifier-guided multi-style tile image generation method

Image generative models for ceramic tile design lack style diversity and controllability of high-quality generated styles. It is difficult to find a series of ceramic tiles with the same texture but distinct styles, that makes it a challenging for users to select from a limited number of tiles with a single style. Although, Generative Adversarial Networks (GANs) can slightly increase the style diversity of tile images, the style controllability remains very weak. Additionally, concatenating generated tile image blocks to obtain a larger texture region can easily result in seams at the boundaries that decrease image quality. In this paper, we propose a style transfer method for ceramic tiles texture generation that combines a classifier-guided StyleGAN with AdaIN-GAN to overcome the above limitations. Firstly, we introduce a new conditional classifier-guided module into the StyleGAN. With the guidance of the input condition vector, the output image is made to have the tile style characteristics that match the vector. At the same time, the fusion of the condition vectors realizes the style gradient effect of the tile image to expand the style diversity. Secondly, we use the AdaIN-GAN to color the original texture in tile style. The style images generated by StyleGAN are then used as a dataset for model training to enhance the generalization ability of the model and achieve a style transfer effect with fixed texture features but significantly diverse styles. Finally, a linear weighted image stitching method is adopted, which uses an adaptive kernel linear weighted matrix to cover and splice arbitrary seams with image blocks, thereby successfully eliminating seams and enhancing image continuity. When this method is applied to high-resolution tile image generation, the method still maintains higher continuity and clearer image quality. Extensive experiments and human evaluation confirm the superior performance of the proposed method compared with other SOTA methods. The experimental results also verify that the new tile images generated by the proposed algorithm have diverse styles and meet the design requirements for tile style diversity.

Msemalign: a pipeline for serial section multibeam scanning electron microscopy volume alignment.

Serial section multibeam scanning electron microscopy (ssmSEM) is currently among the fastest technologies available for acquiring 3D anatomical data spanning relatively large neural tissue volumes, on the order of 1 mm3 or larger, at a resolution sufficient to resolve the fine detail of neuronal morphologies and synapses. These petabyte-scale volumes can be analyzed to create connectomes, datasets that contain detailed anatomical information including synaptic connectivity, neuronal morphologies and distributions of cellular organelles. The mSEM acquisition process creates hundreds of millions of individual image tiles for a single cubic-millimeter-sized dataset and these tiles must be aligned to create 3D volumes. Here we introduce msemalign, an alignment pipeline that strives for scalability and design simplicity. The pipeline can align petabyte-scale datasets such that they contain smooth transitions as the dataset is navigated in all directions, but critically that does so in a fashion that minimizes the overall magnitude of section distortions relative to the originally acquired micrographs.

Development of Automated Risk Stratification for Sporadic Odontogenic Keratocyst Whole Slide Images with an Attention-Based Image Sequence Analyzer

(1) Background: The categorization of recurrent and non-recurrent odontogenic keratocyst is complex and challenging for both clinicians and pathologists. What sets this cyst apart is its aggressive nature and high likelihood of recurrence. Despite identifying various predictive clinical/radiological/histopathological parameters, clinicians still face difficulties in therapeutic management due to its inherent aggressive nature. This research aims to build a pipeline system that accurately detects recurring and non-recurring OKC. (2) Objective: To automate the risk stratification of OKCs as recurring or non-recurring based on whole slide images (WSIs) using an attention-based image sequence analyzer (ABISA). (3) Materials and methods: The presented architecture combines transformer-based self-attention mechanisms with sequential modeling using LSTM (long short-term memory) to predict the class label. This architecture leverages self-attention to capture spatial dependencies in image patches and LSTM to capture sequential dependencies across patches or frames, making it suitable for this image analysis. These two powerful combinations were integrated and applied on a custom dataset of 48 labeled WSIs (508 tiled images) generated from the highest zoom level WSI. (4) Results: The proposed ABISA algorithm attained 0.98, 1.0, and 0.98 testing accuracy, recall, and area under the curve, respectively, whereas VGG16, VGG19, and Inception V3, standard vision transformer attained testing accuracies of 0.80, 0.73, 0.82, 0.91, respectively. ABISA used 58% fewer trainable parameters than the standard vision transformer. (5) Conclusions: The proposed novel ABISA algorithm was integrated into a risk stratification pipeline to automate the detection of recurring OKC significantly faster, thus allowing the pathologist to define risk stratification faster.

Assessment of the large-scale extraction of photovoltaic (PV) panels with a workflow based on artificial neural networks and algorithmic postprocessing of vectorization results

Having a complete and high-quality geospatial catalogue of existing large-scale photovoltaic (PV) panels is very important nowadays, due to the rapid increase in the use of this type of installations. This catalogue could be used to estimate, with a higher level of granularity, the energy that can be produced from solar radiation forecasts, to operate the electricity system efficiently by adjusting supply to demand, as well as for fiscal reasons. However, most existing works in the specialized literature focus on the extraction of PV panels in reduced, favourable scenes. In this study, a processing strategy to obtain PV panel arrays geometries from aerial orthoimages at a very large scale, is proposed. The processing strategy includes operations for PV panel array classification and semantic extraction, and algorithmic improvement and simplification of the vectorization results. The processing workflow was trained and tested on two datasets containing aerial images of 256 × 256 pixels with the studied polygonal feature at two different spatial resolutions (with more than 185,000 and 690,000 images for spatial resolutions of 0.5 m and 0.25 m, respectively). The image tiles were labelled at pixel level with PV panel information for both spatial resolutions and cover representative, extended urban and rural regions of Spain. The accuracies achieved in the PV panel arrays classification are superior to 0.999, while the semantic segmentation performance is superior to 0.90 in the Intersection over Union score. The assessment of the proposed procedure of fifteen new, unseen areas distributed over the Spanish territory (covering approximately 8125 km2) demonstrated its suitability (in the analysis, percentage of incompletely detected installations of only 11 % were achieved) and indicated that future works focused on the improvement of the initial predictions are should be carried out.

Interpretable Artificial Intelligence (AI) Differentiates Prefibrotic Primary Myelofibrosis (prePMF) from Essential Thrombocythemia (ET): A Multi-Center Study of a New Clinical Decision Support Tool

Introduction Overlapping clinical, molecular, and histopathological characteristics pose challenges in differentiating prePMF from ET. The median overall survival, however, significantly differs between prePMF and ET (11.9 vs 22.2 years, Jeryczynski, 2017). The difference in survival highlights the need to distinguish between these two myeloproliferative neoplasms (MPNs) to select disease-specific therapeutic options. This area of unmet need often requires expert assessment at high-volume academic institutions to render a definitive diagnosis. Our aim in this study is to develop and validate a biologically-motivated AI algorithm to rapidly, accurately, and inexpensively diagnose prePMF and ET directly from diagnostic bone marrow (BM) biopsy digital whole-slide images (WSI). Methods Patients with a clinical/histopathological diagnosis of prePMF or ET as determined by the International Consensus Classification of Myeloid Neoplasms were identified at the University of Florence, Italy (Florence) between 06/2007 and 05/2023 and Moffitt Cancer Center, Tampa, FL (Moffitt) between 01/2013 and 01/2022. Diagnostic H&E-stained BM biopsy slides were digitized using Aperio AT2 slide scanners (Leica Biosystems, Deer Park, IL) at each institution . The training cohort comprised of 200 (100 prePMF / 100 ET) patients from Florence, and the external test cohort entailed 26 (6 prePMF / 20 ET) patients from Moffitt. In total, the resultant model was trained on 32,226 patient-derived WSI. Our chosen pretrained neural network, RetCCL, was previously trained on 32,000 diagnostic WSIs to potentially represent a histologically-informed model (Wang, 2023). BM WSI were tessellated into representative image tiles extracted at 10x magnification (302 microns per image dimensions) for model training. Finally, a prediction upon each patient's WSI was calculated by attention-based multiple instance learning, which is a method that automatically assigns a numeric weight to an image portion representing its relative importance to the classification task. Model performance was assessed utilizing the area under the receiver operator curve (AUC). The cutoff threshold for diagnosis classification was determined by maximizing Youden's Index. For qualitative assessment, attention scores were plotted as a heatmap across the BM WSI and reviewed for morphological features by an expert hematopathologist. Custom scripts were written using our open-source AI framework, Slideflow (Dolezal, 2021). Model development was performed on the Minerva High Performance Computer at Mount Sinai Hospital. Evaluation time upon a single WSI was estimated using a consumer-grade computer with an NVIDIA RTX 3080 graphics processing unit. Results Within the training cohort, 5-fold cross validation resulted in a mean AUC of 0.90 and standard deviation of 0.04. A final locked model re-trained on the entire training cohort resulted in an AUC of 0.90 upon evaluation of the test cohort ( Figure 1). We optimized the classification threshold to balance sensitivity and specificity; the final diagnostic classification accuracy on the test cohort was 92.3% with a sensitivity and specificity for prePMF diagnosis of 66.6% and 100%, respectively. Upon review of the slides with highest prediction value per class, attention heatmaps highlighted the model's reliance on areas of cellular marrow without reliance on image artifacts or background ( Figure 2). Using affordable consumer-grade hardware, evaluation upon a previously unseen WSI was completed in approximately 6.1 seconds (4.9 for preprocessing and 1.2 for evaluation). Conclusion We developed a novel AI model with high accuracy for distinguishing between prePMF and ET in distinct clinical cohorts. To our knowledge, this study represents the largest image-based AI study within MPNs with external validation. Our proposed model may assist clinicians in appropriately identifying patient cohorts who would benefit from disease-specific therapies or enrollment onto clinical trials. We imagine that a potential high-speed, low-cost algorithm may reliably distinguish prePMF from ET patients with high specificity which can be democratized to the MPN clinical community in routine practice and drive clinical trial accrual for biologically rational novel therapeutics.

Automatic Characterization of Boulders on Planetary Surfaces From High‐Resolution Satellite Images

AbstractBoulders form from a variety of geological processes, which their size, shape, and orientation may help us better understand. Furthermore, they represent potential hazards to spacecraft landing that need to be characterized. However, mapping individual boulders across vast areas is extremely labor‐intensive, often limiting the extent over which they are characterized and the statistical robustness of obtained boulder morphometrics. To automate boulder characterization, we use an instance segmentation neural network, Mask R‐CNN, to detect and outline boulders in high‐resolution satellite and aerial images. Our neural network, BoulderNet, was trained from a data set of >33,000 boulders in >750 image tiles from Earth, the Moon, and Mars. BoulderNet not only correctly detects the majority of boulders in images but also identifies the outline of boulders with high fidelity, achieving average precision and recall values of 72% and 64% relative to manually digitized boulders from the test data set, when only detections with intersection‐over‐union ratios >50% are considered valid. These values are similar to those obtained from human mappers. On Earth, equivalent boulder diameters, aspect ratios, and orientations extracted from predictions were benchmarked against ground measurements and yield values within ±15%, ±0.20, and ±20° of their ground‐truth values, respectively. BoulderNet achieves better boulder detection and characterization performance relative to existing methods, providing a versatile open‐source tool to characterize entire boulder fields on planetary surfaces.

Development and validation of artificial intelligence-based prescreening of large-bowel biopsies taken in the UK and Portugal: a retrospective cohort study

Histopathological examination is a crucial step in the diagnosis and treatment of many major diseases. Aiming to facilitate diagnostic decision making and improve the workload of pathologists, we developed an artificial intelligence (AI)-based prescreening tool that analyses whole-slide images (WSIs) of large-bowel biopsies to identify typical, non-neoplastic, and neoplastic biopsies. This retrospective cohort study was conducted with an internal development cohort of slides acquired from a hospital in the UK and three external validation cohorts of WSIs acquired from two hospitals in the UK and one clinical laboratory in Portugal. To learn the differential histological patterns from digitised WSIs of large-bowel biopsy slides, our proposed weakly supervised deep-learning model (Colorectal AI Model for Abnormality Detection [CAIMAN]) used slide-level diagnostic labels and no detailed cell or region-level annotations. The method was developed with an internal development cohort of 5054 biopsy slides from 2080 patients that were labelled with corresponding diagnostic categories assigned by pathologists. The three external validation cohorts, with a total of 1536 slides, were used for independent validation of CAIMAN. Each WSI was classified into one of three classes (ie, typical, atypical non-neoplastic, and atypical neoplastic). Prediction scores of image tiles were aggregated into three prediction scores for the whole slide, one for its likelihood of being typical, one for its likelihood of being non-neoplastic, and one for its likelihood of being neoplastic. The assessment of the external validation cohorts was conducted by the trained and frozen CAIMAN model. To evaluate model performance, we calculated area under the convex hull of the receiver operating characteristic curve (AUROC), area under the precision-recall curve, and specificity compared with our previously published iterative draw and rank sampling (IDaRS) algorithm. We also generated heat maps and saliency maps to analyse and visualise the relationship between the WSI diagnostic labels and spatial features of the tissue microenvironment. The main outcome of this study was the ability of CAIMAN to accurately identify typical and atypical WSIs of colon biopsies, which could potentially facilitate automatic removing of typical biopsies from the diagnostic workload in clinics. A randomly selected subset of all large bowel biopsies was obtained between Jan 1, 2012, and Dec 31, 2017. The AI training, validation, and assessments were done between Jan 1, 2021, and Sept 30, 2022. WSIs with diagnostic labels were collected between Jan 1 and Sept 30, 2022. Our analysis showed no statistically significant differences across prediction scores from CAIMAN for typical and atypical classes based on anatomical sites of the biopsy. At 0·99 sensitivity, CAIMAN (specificity 0·5592) was more accurate than an IDaRS-based weakly supervised WSI-classification pipeline (0·4629) in identifying typical and atypical biopsies on cross-validation in the internal development cohort (p<0·0001). At 0·99 sensitivity, CAIMAN was also more accurate than IDaRS for two external validation cohorts (p<0·0001), but not for a third external validation cohort (p=0·10). CAIMAN provided higher specificity than IDaRS at some high-sensitivity thresholds (0·7763 vs 0·6222 for 0·95 sensitivity, 0·7126 vs 0·5407 for 0·97 sensitivity, and 0·5615 vs 0·3970 for 0·99 sensitivity on one of the external validation cohorts) and showed high classification performance in distinguishing between neoplastic biopsies (AUROC 0·9928, 95% CI 0·9927-0·9929), inflammatory biopsies (0·9658, 0·9655-0·9661), and atypical biopsies (0·9789, 0·9786-0·9792). On the three external validation cohorts, CAIMAN had AUROC values of 0·9431 (95% CI 0·9165-0·9697), 0·9576 (0·9568-0·9584), and 0·9636 (0·9615-0·9657) for the detection of atypical biopsies. Saliency maps supported the representation of disease heterogeneity in model predictions and its association with relevant histological features. CAIMAN, with its high sensitivity in detecting atypical large-bowel biopsies, might be a promising improvement in clinical workflow efficiency and diagnostic decision making in prescreening of typical colorectal biopsies. The Pathology Image Data Lake for Analytics, Knowledge and Education Centre of Excellence; the UK Government's Industrial Strategy Challenge Fund; and Innovate UK on behalf of UK Research and Innovation.

Manifold Explorer: Satellite Image Labelling and Clustering Tool with Using Deep Convolutional Autoencoders

We present a novel approach to providing greater insight into the characteristics of an unlabelled dataset, increasing the efficiency with which labelled datasets can be created. We leverage dimension-reduction techniques in combination with autoencoders to create an efficient feature representation for image tiles derived from remote sensing satellite imagery. The proposed methodology consists of two main stages. Firstly, an autoencoder network is utilised to reduce the high-dimensional image tile data into a compact and expressive latentfeature representation. Subsequently, features are further reduced to a two-dimensional embedding space using the manifold learning algorithm Uniform Manifold Approximation and Projection (UMAP) and t-distributed Stochastic Neighbour Embedding (t-SNE). This step enables the visualization of the image tile clusters in a 2D plot, providing an intuitive and interactive representation that can be used to aid rapid and geographically distributed image labelling. To facilitate the labelling process, our approach allows users to interact with the 2D visualization and label clusters based on their domain knowledge. In cases where certain classes are not effectively separated, users can re-apply dimension reduction to interactively refine subsets of clusters and achieve better class separation, enabling a comprehensively labelled dataset. We evaluate the proposed approach on real-world remote sensing satellite image datasets and demonstrate its effectiveness in achieving accurate and efficient image tile clustering and labelling. Users actively participate in the labelling process through our interactive approach, leading to enhanced relevance of the labelled data, by allowing domain experts to contribute their expertise and enrich the dataset for improved downstream analysis and applications.

A Federated Learning Approach to Tumor Detection in Colon Histology Images.

Federated learning (FL), a relatively new area of research in medical image analysis, enables collaborative learning of a federated deep learning model without sharing the data of participating clients. In this paper, we propose FedDropoutAvg, a new federated learning approach for detection of tumor in images of colon tissue slides. The proposed method leverages the power of dropout, a commonly employed scheme to avoid overfitting in neural networks, in both client selection and federated averaging processes. We examine FedDropoutAvg against other FL benchmark algorithms for two different image classification tasks using a publicly available multi-site histopathology image dataset. We train and test the proposed model on a large dataset consisting of 1.2 million image tiles from 21 different sites. For testing the generalization of all models, we select held-out test sets from sites that were not used during training. We show that the proposed approach outperforms other FL methods and reduces the performance gap (to less than 3% in terms of AUC on independent test sites) between FL and a central deep learning model that requires all data to be shared for centralized training, demonstrating the potential of the proposed FedDropoutAvg model to be more generalizable than other state-of-the-art federated models. To the best of our knowledge, ours is the first study to effectively utilize the dropout strategy in a federated setting for tumor detection in histology images.

Deep Learning–Enabled Diagnosis of Liver Adenocarcinoma

Diagnosis of adenocarcinoma in the liver is a frequent scenario in routine pathology and has a critical impact on clinical decision making. However, rendering a correct diagnosis can be challenging, and often requires the integration of clinical, radiologic, and immunohistochemical information. We present a deep learning model (HEPNET) to distinguish intrahepatic cholangiocarcinoma from colorectal liver metastasis, as the most frequent primary and secondary forms of liver adenocarcinoma, with clinical grade accuracy using H&E-stained whole-slide images. HEPNET was trained on 714,589 image tiles from 456 patients who were randomly selected in a stratified manner from a pool of 571 patients who underwent surgical resection or biopsy at Heidelberg University Hospital. Model performance was evaluated on a hold-out internal test set comprising 115 patients and externally validated on 159 patients recruited at Mainz University Hospital. On the hold-out internal test set, HEPNET achieved an area under the receiver operating characteristic curve of 0.994 (95% CI, 0.989-1.000) and an accuracy of 96.522% (95% CI, 94.521%-98.694%) at the patient level. Validation on the external test set yielded an area under the receiver operating characteristic curve of 0.997 (95% CI, 0.995-1.000), corresponding to an accuracy of 98.113% (95% CI, 96.907%-100.000%). HEPNET surpassed the performance of 6 pathology experts with different levels of experience in a reader study of 50 patients (P= .0005), boosted the performance of resident pathologists to the level of senior pathologists, and reduced potential downstream analyses. We provided a ready-to-use tool with clinical grade performance that may facilitate routine pathology by rendering a definitive diagnosis and guiding ancillary testing. The incorporation of HEPNET into pathology laboratories may optimize the diagnostic workflow, complemented by test-related labor and cost savings.

Synthetic whole-slide image tile generation with gene expression profile-infused deep generative models

In this work, we propose an approach to generate whole-slide image (WSI) tiles by using deep generative models infused with matched gene expression profiles. First, we train a variational autoencoder (VAE) that learns a latent, lower-dimensional representation of multi-tissue gene expression profiles. Then, we use this representation to infuse generative adversarial networks (GANs) that generate lung and brain cortex tissue tiles, resulting in a new model that we call RNA-GAN. Tiles generated by RNA-GAN were preferred by expert pathologists compared with tiles generated using traditional GANs, and in addition, RNA-GAN needs fewer training epochs to generate high-quality tiles. Finally, RNA-GAN was able to generalize to gene expression profiles outside of the training set, showing imputation capabilities. A web-based quiz is available for users to play a game distinguishing real and synthetic tiles: https://rna-gan.stanford.edu/, and the code for RNA-GAN is available here: https://github.com/gevaertlab/RNA-GAN.

Region of interest (ROI) selection using vision transformer for automatic analysis using whole slide images

Selecting regions of interest (ROI) is a common step in medical image analysis across all imaging modalities. An ROI is a subset of an image appropriate for the intended analysis and identified manually by experts. In modern pathology, the analysis involves processing multidimensional and high resolution whole slide image (WSI) tiles automatically with an overwhelming quantity of structural and functional information. Despite recent improvements in computing capacity, analyzing such a plethora of data is challenging but vital to accurate analysis. Automatic ROI detection can significantly reduce the number of pixels to be processed, speed the analysis, improve accuracy and reduce dependency on pathologists. In this paper, we present an ROI detection method for WSI and demonstrated it for human epidermal growth factor receptor 2 (HER2) grading for breast cancer patients. Existing HER2 grading relies on manual ROI selection, which is tedious, time-consuming and suffers from inter-observer and intra-observer variability. This study found that the HER2 grade changes with ROI selection. We proposed an ROI detection method using Vision Transformer and investigated the role of image magnification for ROI detection. This method yielded an accuracy of 99% using 20 × WSI and 97% using 10 × WSI for the ROI detection. In the demonstration, the proposed method increased the diagnostic agreement to 99.3% with the clinical scores and reduced the time to 15 seconds for automated HER2 grading.

Computational textural mapping harmonises sampling variation and reveals multidimensional histopathological fingerprints

BackgroundTechnical factors can bias H&E digital slides potentially compromising computational histopathology studies. Here, we hypothesised that sample quality and sampling variation can introduce even greater and undocumented technical fallacy.MethodsUsing The Cancer Genome Atlas (TCGA) clear-cell renal cell carcinoma (ccRCC) as a model disease, we annotated ~78,000 image tiles and trained deep learning models to detect histological textures and lymphocyte infiltration at the tumour core and its surrounding margin and correlated these with clinical, immunological, genomic, and transcriptomic profiles.ResultsThe models reached 95% validation accuracy for classifying textures and 95% for lymphocyte infiltration enabling reliable profiling of ccRCC samples. We validated the lymphocyte-per-texture distributions in the Helsinki dataset (n = 64). Texture analysis indicated constitutive sampling bias by TCGA clinical centres and technically suboptimal samples. We demonstrate how computational texture mapping (CTM) can abrogate these issues by normalising textural variance. CTM-harmonised histopathological architecture resonated with both expected associations and novel molecular fingerprints. For instance, tumour fibrosis associated with histological grade, epithelial-to-mesenchymal transition, low mutation burden and metastasis.ConclusionsThis study highlights texture-based standardisation to resolve technical bias in computational histopathology and understand the molecular basis of tissue architecture. All code, data and models are released as a community resource.

Inconsistency Detection in Cross-Layer Tile Maps with Super-Pixel Segmentation

The consistency of geospatial data is of great significance for the application and updating of geographic information in web maps. Due to the multiple data sources and different temporal versions, the tile web maps usually meet the inconsistency question across different layers. This study tries to develop a method to detect this kind of inconsistency utilizing a raster-based scaling approach. Compared with vector-based handling, this method can be directly available for multi-level tile images in a pixel representation form. The proposed cross-layer raster tile map rendering method (CRTMRM) consists of four primary aspects: geographic object separation, consistency rendering rules, data scaling and derivation with super-pixel segmentation, and inconsistency detection. The scale transformation strategy with the super-pixel attempts to obtain a simplified representation. Taking the scale lifespan variation and geometric consistency rules into account, the inconsistency detection of tile maps is conducted between temporal versions, multi-sources, and different scales through actual and derived data overlay analysis. The experiment focuses on features of cross-layer water or vegetation areas with Level 9 to Level 14 in Baidu Maps, Amap, and Google Maps. This method is able to serve as a basis for massive unstructured web map data inconsistency detection and support intelligent web map rendering.

Efficient Management and Scheduling of Massive Remote Sensing Image Datasets

The rapid development of remote sensing image sensor technology has led to exponential increases in available image data. The real-time scheduling of gigabyte-level images and the storage and management of massive image datasets are incredibly challenging for current hardware, networking and storage systems. This paper’s three novel strategies (ring caching, multi-threading and tile-prefetching mechanisms) are designed to comprehensively optimize the remote sensing image scheduling process from image retrieval, transmission and visualization perspectives. A novel remote sensing image management and scheduling system (RSIMSS) is designed using these three strategies as its core algorithm, the PostgreSQL database and HDFS distributed file system as its underlying storage system, and the multilayer Hilbert spatial index and image tile pyramid to organize massive remote sensing image datasets. Test results show that the RSIMSS provides efficient and stable image storage performance and allows real-time image scheduling and view roaming.