Image Tiles Research Articles

Mapping the landscape of histomorphological cancer phenotypes using self-supervised learning on unannotated pathology slides

Cancer diagnosis and management depend upon the extraction of complex information from microscopy images by pathologists, which requires time-consuming expert interpretation prone to human bias. Supervised deep learning approaches have proven powerful, but are inherently limited by the cost and quality of annotations used for training. Therefore, we present Histomorphological Phenotype Learning, a self-supervised methodology requiring no labels and operating via the automatic discovery of discriminatory features in image tiles. Tiles are grouped into morphologically similar clusters which constitute an atlas of histomorphological phenotypes (HP-Atlas), revealing trajectories from benign to malignant tissue via inflammatory and reactive phenotypes. These clusters have distinct features which can be identified using orthogonal methods, linking histologic, molecular and clinical phenotypes. Applied to lung cancer, we show that they align closely with patient survival, with histopathologically recognised tumor types and growth patterns, and with transcriptomic measures of immunophenotype. These properties are maintained in a multi-cancer study.

Fast and robust feature-based stitching algorithm for microscopic images

The limited field of view of high-resolution microscopic images hinders the study of biological samples in a single shot. Stitching of microscope images (tiles) captured by the whole-slide imaging (WSI) technique solves this problem. However, stitching is challenging due to the repetitive textures of tissues, the non-informative background part of the slide, and the large number of tiles that impact performance and computational time. To address these challenges, we proposed the Fast and Robust Microscopic Image Stitching (FRMIS) algorithm, which relies on pairwise and global alignment. The speeded up robust features (SURF) were extracted and matched within a small part of the overlapping region to compute the transformation and align two neighboring tiles. In cases where the transformation could not be computed due to an insufficient number of matched features, features were extracted from the entire overlapping region. This enhances the efficiency of the algorithm since most of the computational load is related to pairwise registration and reduces misalignment that may occur by matching duplicated features in tiles with repetitive textures. Then, global alignment was achieved by constructing a weighted graph where the weight of each edge is determined by the normalized inverse of the number of matched features between two tiles. FRMIS has been evaluated on experimental and synthetic datasets from different modalities with different numbers of tiles and overlaps, demonstrating faster stitching time compared to existing algorithms such as the Microscopy Image Stitching Tool (MIST) toolbox. FRMIS outperforms MIST by 481% for bright-field, 259% for phase-contrast, and 282% for fluorescence modalities, while also being robust to uneven illumination.

High-resolution, large field-of-view label-free imaging via aberration-corrected, closed-form complex field reconstruction

Computational imaging methods empower modern microscopes to produce high-resolution, large field-of-view, aberration-free images. Fourier ptychographic microscopy can increase the space-bandwidth product of conventional microscopy, but its iterative reconstruction methods are prone to parameter selection and tend to fail under excessive aberrations. Spatial Kramers–Kronig methods can analytically reconstruct complex fields, but is limited by aberration or providing extended resolution enhancement. Here, we present APIC, a closed-form method that weds the strengths of both methods while using only NA-matching and darkfield measurements. We establish an analytical phase retrieval framework which demonstrates the feasibility of analytically reconstructing the complex field associated with darkfield measurements. APIC can retrieve complex aberrations of an imaging system with no additional hardware and avoids iterative algorithms, requiring no human-designed convergence metrics while always obtaining a closed-form complex field solution. We experimentally demonstrate that APIC gives correct reconstruction results where Fourier ptychographic microscopy fails when constrained to the same number of measurements. APIC achieves 2.8 times faster computation using image tile size of 256 (length-wise), is robust against aberrations compared to Fourier ptychographic microscopy, and capable of addressing aberrations whose maximal phase difference exceeds 3.8π when using a NA 0.25 objective in experiment.

A whole-slide foundation model for digital pathology from real-world data

Digital pathology poses unique computational challenges, as a standard gigapixel slide may comprise tens of thousands of image tiles1–3. Prior models have often resorted to subsampling a small portion of tiles for each slide, thus missing the important slide-level context4. Here we present Prov-GigaPath, a whole-slide pathology foundation model pretrained on 1.3 billion 256 × 256 pathology image tiles in 171,189 whole slides from Providence, a large US health network comprising 28 cancer centres. The slides originated from more than 30,000 patients covering 31 major tissue types. To pretrain Prov-GigaPath, we propose GigaPath, a novel vision transformer architecture for pretraining gigapixel pathology slides. To scale GigaPath for slide-level learning with tens of thousands of image tiles, GigaPath adapts the newly developed LongNet5 method to digital pathology. To evaluate Prov-GigaPath, we construct a digital pathology benchmark comprising 9 cancer subtyping tasks and 17 pathomics tasks, using both Providence and TCGA data6. With large-scale pretraining and ultra-large-context modelling, Prov-GigaPath attains state-of-the-art performance on 25 out of 26 tasks, with significant improvement over the second-best method on 18 tasks. We further demonstrate the potential of Prov-GigaPath on vision–language pretraining for pathology7,8 by incorporating the pathology reports. In sum, Prov-GigaPath is an open-weight foundation model that achieves state-of-the-art performance on various digital pathology tasks, demonstrating the importance of real-world data and whole-slide modelling.

Deep learning and digital pathology powers prediction of HCC development in steatotic liver disease.

Identifying patients with steatotic liver disease who are at a high risk of developing HCC remains challenging. We present a deep learning (DL) model to predict HCC development using hematoxylin and eosin-stained whole-slide images of biopsy-proven steatotic liver disease. We included 639 patients who did not develop HCC for ≥7 years after biopsy (non-HCC class) and 46 patients who developed HCC <7 years after biopsy (HCC class). Paired cases of the HCC and non-HCC classes matched by biopsy date and institution were used for training, and the remaining nonpaired cases were used for validation. The DL model was trained using deep convolutional neural networks with 28,000 image tiles cropped from whole-slide images of the paired cases, with an accuracy of 81.0% and an AUC of 0.80 for predicting HCC development. Validation using the nonpaired cases also demonstrated a good accuracy of 82.3% and an AUC of 0.84. These results were comparable to the predictive ability of logistic regression model using fibrosis stage. Notably, the DL model also detected the cases of HCC development in patients with mild fibrosis. The saliency maps generated by the DL model highlighted various pathological features associated with HCC development, including nuclear atypia, hepatocytes with a high nuclear-cytoplasmic ratio, immune cell infiltration, fibrosis, and a lack of large fat droplets. The ability of the DL model to capture subtle pathological features beyond fibrosis suggests its potential for identifying early signs of hepatocarcinogenesis in patients with steatotic liver disease.

Abstract B004: Deep learning unveils molecular footprints in histology: predicting molecular subtypes from bladder cancer histology slides

Abstract Introduction: Transurethral resection followed by intravesical bacillus Calmette-Guerin (BCG) is the standard of care for high-risk non-muscle-invasive bladder cancer (HR-NMIBC) patients. However, ≈20% of HR-NMIBC patients progress to muscle-invasive or metastatic disease, which is associated with a high-risk of death. Early identification of patients who are unlikely to benefit from BCG treatment could benefit from an early radical cystectomy or bladder-sparing alternatives. Previously, we showed that molecular subtyping or HR-NMIBC identified BCG response subtypes (BRS 1/2/3). BRS3 tumors were associated with a poor response to BCG and a high-risk of progression. However, molecular subtyping relies on resource-intensive and costly sequencing methods such as next-generation sequencing, and microarray analysis. We employed deep learning (DL) to predict BRS3 vs BRS1/2 molecular subtypes from digitized histology slides. Methods: Haematoxylin and eosin (H&amp;E)-stained slides were digitized, and quality control was performed. The areas where the RNA was extracted were annotated pixel-wise by a trained pathology assistant, using QuPath. Image tiles of 512 × 512 pixels at 10X, 20X, and 40X magnifications were extracted with a 25% overlap. For classification, BRS1 and BRS2 images were grouped, while BRS3 images were categorized separately. After applying Macenko normalization, tiles were split based on individual patients into a 75/25 training/validation ratio via stratified 4-fold cross-validation. Four DL models (DenseNet, Inception, ShuffleNet, ResNet) were trained to classify tiles into BRS3 and BRS1/2, with tile labels based on the patient’s subtype. Patient-level classification was performed using majority voting from tile predictions. Model efficacy was gauged using the area under the ROC curve (AUC). Results: Out of 245 H&amp;E slides, 45 were discarded after quality control. From 200 patients – with a 70:30 distribution between BRS1&amp;2 and BRS3 The DenseNet model at 10x magnification was the best-performing model, achieving the highest AUC compared to all other models and magnifications. On the image tile level, the best DenseNet achieved an average AUC of 0.65 with a standard deviation of 0.03 across the four folds of the validation set. At the patient-level prediction, the best DenseNet reached an average AUC of 0.80 and a standard deviation of 0.05 on the validation set. Conclusion: Our study demonstrates that deep learning classifies BSR3 from BSR1 and 2 subtypes from H&amp;E slides. This method holds potential to identify BRS3 HR-NMIBC patients who may benefit from alternative treatments than BCG. With further refinement, this deep learning strategy can become a key tool in histology, aiding in swift, affordable, and efficient subtype identification. Hence, this method has the potential to bridge the gap between molecular techniques and clinical practice. Future research should prioritize evaluating the model on an external test cohort and consider predicting additional molecular markers to fine-tune prediction outcomes. Citation Format: Farbod Khoraminia, Flouros C de Jong, Farhan Akram, Geert Litjens, Maarten D.J. Jansen, Alberto Nakuama Gonzalez, Danique Lichtenburg, Andrew Stubbs, Nadieh Khalili, Tahlita C.M. Zuiverloon. Deep learning unveils molecular footprints in histology: predicting molecular subtypes from bladder cancer histology slides [abstract]. In: Proceedings of the AACR Special Conference on Bladder Cancer: Transforming the Field; 2024 May 17-20; Charlotte, NC. Philadelphia (PA): AACR; Clin Cancer Res 2024;30(10_Suppl):Abstract nr B004.

Addressing incomplete tile phenomena in image tiling: Introducing the grid six-intersection model

Abstract With the continuous development of Earth Observation technology, resolution of imagery and gridded data has significantly increased, leading to a rapid increase in data volume. To efficiently acquire and analyze these vast amounts of imagery and gridded data, image tiling technology has been developed to effectively access data of interested areas. Tiling technology divides large-scale image data into smaller tiles, providing fast, accurate, and efficient access support for imagery and gridded data. The spatial grid model, as the foundational framework of the new generation of geographic spatial information, plays a critical role in the retrieval, integration, services, and applications of imagery and gridded data resources. When tiling image data based on the spatial grid model, it always generates both complete and incomplete tiles. Particularly, when conducting image tile retrieval using the same rule-based grid in the retrieval area, incomplete tile phenomena along the boundary regions of the retrieved images often occur, resulting in gaps within the retrieval area. To tackle this issue, in this study, we present a new topological model called the Grid Six-Intersection Model (G-6IM), specifically designed for regular rectangular grids, to accurately represent boundary issues in image tiling. Through a practical case study, we demonstrate the effectiveness and practical application potential of the proposed G-6IM model, providing new insights and guidance for the improvement and optimization of imagery and gridded data tiling technology.

Deep neural network for the prediction of KRAS, NRAS, and BRAF genotypes in left-sided colorectal cancer based on histopathologic images

BackgroundThe KRAS, NRAS, and BRAF genotypes are critical for selecting targeted therapies for patients with metastatic colorectal cancer (mCRC). Here, we aimed to develop a deep learning model that utilizes pathologic whole-slide images (WSIs) to accurately predict the status of KRAS, NRAS, and BRAFV600E. Methods129 patients with left-sided colon cancer and rectal cancer from the Third Affiliated Hospital of Sun Yat-sen University were assigned to the training and testing cohorts. Utilizing three convolutional neural networks (ResNet18, ResNet50, and Inception v3), we extracted 206 pathological features from H&E-stained WSIs, serving as the foundation for constructing specific pathological models. A clinical feature model was then developed, with carcinoembryonic antigen (CEA) identified through comprehensive multiple regression analysis as the key biomarker. Subsequently, these two models were combined to create a clinical-pathological integrated model, resulting in a total of three genetic prediction models. Result103 patients were evaluated in the training cohort (1782,302 image tiles), while the remaining 26 patients were enrolled in the testing cohort (489,481 image tiles). Compared with the clinical model and the pathology model, the combined model which incorporated CEA levels and pathological signatures, showed increased predictive ability, with an area under the curve (AUC) of 0.96 in the training and an AUC of 0.83 in the testing cohort, accompanied by a high positive predictive value (PPV 0.92). ConclusionThe combined model demonstrated a considerable ability to accurately predict the status of KRAS, NRAS, and BRAFV600E in patients with left-sided colorectal cancer, with potential application to assist doctors in developing targeted treatment strategies for mCRC patients, and effectively identifying mutations and eliminating the need for confirmatory genetic testing.

Abstract PO1-26-01: Leveraging Clinicopathological Factors and Deep Learning-Based Morphometrics for PDX Engraftment Success Prediction in Breast Cancer

Abstract Background: Patient-derived xenografts (PDXs) are pivotal in cancer research. Despite histopathological insights into factors driving PDX success, the role of artificial intelligence (AI) in predicting PDX engraftment remains unexplored. We aimed to bridge this gap by integrating clinicopathological data and AI-based morphometric analysis to predict PDX success in breast cancer. Methods: PDXs were generated from tumor tissues derived from breast cancer patients who had undergone surgical intervention. Clinicopathological information including subtypes, pathological diagnosis, modified Bloom-Richardson system histologic grades, treatment with neoadjuvant chemotherapy (NAC), Miller Payne grade, residual caner burden score, invasive tumor size, lymphovascular invasion status, AJCC 8th T and N stages and the percentage of tumor infiltrating lymphocytes were collected and analyzed. For the image analysis component, whole-slide images (WSIs) of hematoxylin and eosin–stained tissue samples from 64 surgically resected breast cancer patients were used as a training set for an AI model under the supervision of 2 pathologists to extract morphometric features. The model transformed image tiles into patches of morphologically similar patterns, and categorized them into adipose tissue, background, necrosis, invasive carcinoma, carcinoma in situ, stroma, and terminal ductal lobular unit. This trained model was subsequently applied to the WSIs employed in the establishment of PDXs. The classified patches and their relative ratios within the invasive tumor boundary were compiled. The consolidated data from clinicopathological and image analyses were subjected to logistic regression to discern correlates of successful PDX engraftment. Results: Out of the 311 patient tumor samples used for generating PDXs, (131 post-chemotherapy, 180 chemo-naïve), 47 PDXs were successfully established (15.1%). Logistic regression revealed several factors for successful engraftment including NAC treatment with an odds ratio of 6.71 (28.2% vs 5.6%, 95% CI: 2.53 - 17.80, p &amp;lt; 0.001), higher histologic grades (25.3% vs 2.2%, OR = 5.80, 95% CI: 1.49 - 22.63, p = 0.01), triple negative breast cancer compared to hormone receptor-positive cancers (32.0% vs 3.3%, OR = 10.99, 95% CI: 1.26 - 95.89, p = 0.03), and tumors of larger size (OR = 1.34, 95% CI: 1.02 - 1.76, p = 0.03). Interestingly, the percentage of specific tissue patch types within tumor did not significantly impact the likelihood of successful PDX engraftment. However, in our logistic regression analysis based solely on morphometric features, presence of necrosis within the tumor notably enhanced PDX establishment. Specifically, each percent increase in necrosis within tumor boosted the odds of successful PDX creation by 0.01% (OR = 1.0001, 95% CI: 1.00003 - 1.00024, p = 0.01). Conclusions: PDXs are often successfully established from clinically aggressive breast cancers, particularly those with NAC treatment, higher histologic grades, TNBC subtype, and larger tumor size. While morphometric features contribute to the prediction of PDX engraftment success, their importance is surpassed by these clinicopathological factors. However, presence of necrosis emerged as a key morphometric predictor of successful PDX engraftment. Keywords: breast cancer; patient-derived xenograft (PDX); Breast Cancer Morphometrics; Cancer Predictive Modeling. Citation Format: Jongwon Lee, GeonHee Lee, Gyungyub Gong, Hee Jin Lee. Leveraging Clinicopathological Factors and Deep Learning-Based Morphometrics for PDX Engraftment Success Prediction in Breast Cancer [abstract]. In: Proceedings of the 2023 San Antonio Breast Cancer Symposium; 2023 Dec 5-9; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2024;84(9 Suppl):Abstract nr PO1-26-01.

A comparative analysis of pairwise image stitching techniques for microscopy images

Stitching of microscopic images is a technique used to combine multiple overlapping images (tiles) from biological samples with a limited field of view and high resolution to create a whole slide image. Image stitching involves two main steps: pairwise registration and global alignment. Most of the computational load and the accuracy of the stitching algorithm depend on the pairwise registration method. Therefore, choosing an efficient, accurate, robust, and fast pairwise registration method is crucial in the whole slide imaging technique. This paper presents a detailed comparative analysis of different pairwise registration techniques in terms of execution time and quality. These techniques included feature-based methods such as Harris, Shi-Thomasi, FAST, ORB, BRISK, SURF, SIFT, KAZE, MSER, and deep learning-based SuperPoint features. Additionally, region-based methods were analyzed, which were based on the normalized cross-correlation (NCC) and the combination of phase correlation and NCC. Investigations have been conducted on microscopy images from different modalities such as bright-field, phase-contrast, and fluorescence. The feature-based methods were highly robust to uneven illumination in tiles. Moreover, some features were found to be more accurate and faster than region-based methods, with the SURF features identified as the most effective technique. This study provides valuable insights into the selection of the most efficient and accurate pairwise registration method for creating whole slide images, which is essential for the advancement of computational pathology and biology.

Abstract 4298: Predicting tumor evolution from digital histology using AI

Abstract Diverse clinical presentations of clear cell renal cell carcinoma (ccRCC) confound clinical decision making, leading to over and undertreatment. Clonal evolution of ccRCC proceeds through distinct trajectories characterised by differing levels of genomic intratumoral heterogeneity (gITH) and chromosomal complexity (weighted genomic instability index, wGII). However, accurate evaluation of these indices requires multiregional profiling of fresh tumour; cost prohibitive and logistically challenging in the clinical setting. Clinical histopathology workflows routinely capture multiple tumour areas enabling the use artificial intelligence (AI) to predict tumour evolutionary features directly from clinical grade H&amp;E whole slide image (WSIs). ccRCC displays profound genetic and histological ITH but the link between these entities remains unclear. We leverage the TRACERx Renal cohort, comprising 1485 WSIs from 81 tumours to predict WGII and gITH and to gain insights into the relationship between genetic and histological ITH. Critically, each WSI is associated with a wGII and gITH label derived from a closely linked fresh tumour sample. For both prediction tasks, we extracted meaningful features for each WSI using self-supervised representation learning “MoCo”. Since high wGII confers poor prognosis we focussed on predicting binary stratification label of high wGII or low wGII (relative to the cohort median). First we predicted wGII as a continuous variable using a supervised multiple instance learning regression model trained on the MoCo features, and then classified the predicted wGII into “high” or “low” achieving 0.80 AUROC. To predict gITH we postulated that the degree of gITH would correlate with histological ITH. Using an unsupervised clustering of refined MoCo features we defined 24 histological clusters. The number of computationally derived histological clusters within a single tumour positively correlated with gITH (pearson’s 0.56). We used the number of clusters to classify WSIs into prognostic binary groups of high or low gITH (relative to the cohort median) achieving an AUROC of 0.80. To understand the biological relationship between histological and genetic ITH we pathologically characterised the histological clusters: a pathologist annotated WSIs with tumour architecture and cytomorphology. Image tiles were associated with the annotations using spatial coordinates, illuminating phenotypic traits of different evolutionary trajectories and providing an interpretability framework for our AI pipelines. Since the tumour evolutionary course dictates disease progression tempo, applying evolutionary classification in clinic can fundamentally improve patient care. Here, for the first time, we provide a framework to translate fundamental evolutionary principles underpinning tumour biology and clinical progression into a prognostic computational pathology biomarker possible to clinically implement. Citation Format: Charlotte E. Spencer, Axel Camara, Auriane Riou, Lewis Au, Jose I. Lopez, Zayd Tippu, Charles Maussion, Kenneth Ho, Amy Strange, Emma Nye, Veronique Birault, Lydwine Van-praet, Kim Edmonds, Eleanor Carlyle, Steve Hazell, Sarah Rudman, James Larkin, Samra Turajlic. Predicting tumor evolution from digital histology using AI [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 4298.

Abstract 4933: A workflow for cloud-based AI development of multiplex IF image analysis using the OMERO Plus platform

Abstract Background: The advancement of AI in digital pathology is essential for analyzing image data containing billions of cells. This is true for digital pathology workflows using bright-field images, but this is even more critical for multiplex immunofluorescence (mIF) whole-slides due to the increased number of fluorescent channels and bit-depth per pixel. These complexities necessitate improved data management, and efficient visualization for both the mIF image and AI-based analysis results. We present a workflow using OMERO Plus, AWS SageMaker, and Ultivue software for AI model development in mIF. Method: Images were uploaded to AWS S3 and imported to the data management system OMERO Plus, where they were organized into training and validation groups. Annotations: PathViewer was used to place rectangular fields of view (FOV) in locations selected by the pathologist where all positive cells for each target marker were manually annotated. Data Preparation: Using the OMERO Plus API, annotation coordinates, image tiles, and image metadata were extracted and formatted for model training. All training data is under a version control system using Gitlab and Data Version Control tools using AWS S3. Additionally, strict quality checks were performed to ensure data integrity. Training: For both semantic segmentation and cell object detection, the training data was used in a customized training workflow using AWS SageMaker, including modern deep learning architectures, data augmentation, training logging, evaluation, and deployment. All AI models were trained using Pytorch and deployed with the open model format ONNX. Results Visualization: AI segmentation results of cells and regions are visualized using PathViewer as overlays on the original images by converting them to the OME-NGFF label image format that uses Zarr multidimensional arrays. Results: We utilized 58 and 24 mIF images for training and validation, respectively, on 15 markers using ISP Ultivue technology with 1681 and 296 FOVs, encompassing over 60K cells. Using this data and methodology we trained 3 semantic segmentation models and over five object detection models suitable for all twenty-four OmniVUE markers. For instance, the CD3 positive cell detection model, trained with 390 FOVs, achieved highly accurate results: f1-score of 0.915 evaluated on 34 FOVs. Conclusions: The combination of OMERO Plus, PathViewer, AWS SageMaker, and internal tools can successfully address mIF Big Data data management complexities in a cloud environment for digital pathology. The architecture is highly scalable, cost effective, and reliable, enabling a very efficient AI workflow. Citation Format: Ruben Cardenes, Erin Diel, Douglas Wood, Je Lee, Angela Vasaturo, Martin Schulze, Lorenz Rognoni. A workflow for cloud-based AI development of multiplex IF image analysis using the OMERO Plus platform [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 4933.

Mapping proglacial headwater streams in High Mountain Asia using PlanetScope imagery

Headwater streams transport nutrients, sediment, and mineral-rich groundwater downstream. In High Mountain Asia (HMA), headwater streams also funnel glacier and snow melt to sustain continuous water supply for the downstream region. These channels remain poorly mapped because of their inaccessibility and because they are smaller than the resolution of Landsat (30 m) and Sentinel-2 (10 m). In this study, we assessed the ability of 3 m resolution PlanetScope imagery to detect the proglacial headwaters downstream of all high-altitude glaciers larger than 5 km2 in HMA. We created 3000 manually labeled image tiles to train and evaluate computer vision (CV) against techniques common in the hydrologic remote sensing literature, specifically normalized difference water index (NDWI) thresholding and random forests (RF). Results indicate that CV best detects the headwater streams with >0.60 F1-scores, nearly 0.20 points higher than RF and 0.45 points higher than thresholding. We also assessed how errors in CV propagate to derived hydrologic information, exemplified by the biogeochemically critical measurement of stream surface area. We found that CV classifications produced surface areas with 0.98 R2, 0.01 km2 MAE, and 0.02 km2 RMSE against manually labeled surface areas. We also observed the best CV performance during the spring season with 30% more skillful classification performance than in summer and fall. Our results prove the ability of PlanetScope imagery to detect and map headwater streams accurately and at scale, and that classification errors stemming from the imagery or the CV methods do not greatly impair our ability to quantify stream surface area meaningful for biogeochemical exchange and hydrology studies.

Generation of synthetic whole-slide image tiles of tumours from RNA-sequencing data via cascaded diffusion models.

Training machine-learning models with synthetically generated data can alleviate the problem of data scarcity when acquiring diverse and sufficiently large datasets is costly and challenging. Here we show that cascaded diffusion models can be used to synthesize realistic whole-slide image tiles from latent representations of RNA-sequencing data from human tumours. Alterations in gene expression affected the composition of cell types in the generated synthetic image tiles, which accurately preserved the distribution of cell types and maintained the cell fraction observed in bulk RNA-sequencing data, as we show for lung adenocarcinoma, kidney renal papillary cell carcinoma, cervical squamous cell carcinoma, colon adenocarcinoma and glioblastoma. Machine-learning models pretrained with the generated synthetic data performed better than models trained from scratch. Synthetic data may accelerate the development of machine-learning models in scarce-data settings and allow for the imputation of missing data modalities.

Automating Ground Control Point Detection in Drone Imagery: From Computer Vision to Deep Learning

Drone-based photogrammetry typically requires the task of georeferencing aerial images by detecting the center of Ground Control Points (GCPs) placed in the field. Since this is a very labor-intensive task, it could benefit greatly from automation. In this study, we explore the extent to which traditional computer vision approaches can be generalized to deal with variability in real-world drone data sets and focus on training different residual neural networks (ResNet) to improve generalization. The models were trained to detect single keypoints of fixed-sized image tiles with a historic collection of drone-based Red–Green–Blue (RGB) images with black and white GCP markers in which the center was manually labeled by experienced photogrammetry operators. Different depths of ResNets and various hyperparameters (learning rate, batch size) were tested. The best results reached sub-pixel accuracy with a mean absolute error of 0.586. The paper demonstrates that this approach to drone-based mapping is a promising and effective way to reduce the human workload required for georeferencing aerial images.

Latitude-Adaptive Integer Bit Allocation for Quantization of Omnidirectional Images

Omnidirectional images have gained significant popularity and drawn great attention nowadays, which poses challenges to omnidirectional image processing in solving the bottleneck of storage and transmission. Projecting onto a two-dimensional image plane is generally used to compress an omnidirectional image. However, the most commonly used projection format, the equirectangular projection (ERP), results in a significant amount of redundant samples in the polar areas, thus incurring extra bitrate and geometric distortion. We derive the optimal latitude-adaptive bit allocation for each image tile. Subsequently, we propose a greedy algorithm for non-negative integer bit allocation (NNIBA) for non-uniform quantization under an omnidirectional image quality metric WMSE. In our experiment, we design quantization tables based on JPEG and compare our approach with other sampling-related methods. Our method achieves an average bit saving of 7.9% compared with JPEG while outperforming other sampling-related methods. Besides, we compare our non-uniform quantization approach with two proposed bit allocation methods, achieving an average improvement of 0.35 dB and 2.66 dB under WS-PSNR, respectively. The visual quality assessment also confirms the superiority of our method.

A benchmark GaoFen-7 dataset for building extraction from satellite images

Accurate building extraction is crucial for urban understanding, but it often requires a substantial number of building samples. While some building datasets are available for model training, there remains a lack of high-quality building datasets covering urban and rural areas in China. To fill this gap, this study creates a high-resolution GaoFen-7 (GF-7) Building dataset utilizing the Chinese GF-7 imagery from six Chinese cities. The dataset comprises 5,175 pairs of 512 × 512 image tiles, covering 573.17 km2. It contains 170,015 buildings, with 84.8% of the buildings in urban areas and 15.2% in rural areas. The usability of the GF-7 Building dataset has been proved with seven convolutional neural networks, all achieving an overall accuracy (OA) exceeding 93%. Experiments have shown that the GF-7 building dataset can be used for building extraction in urban and rural scenarios. The proposed dataset boasts high quality and high diversity. It supplements existing building datasets and will contribute to promoting new algorithms for building extraction, as well as facilitating intelligent building interpretation in China.

Unraveling a Histopathological Needle-in-Haystack Problem: Exploring the Challenges of Detecting Tumor Budding in Colorectal Carcinoma Histology

Background: In this study focusing on colorectal carcinoma (CRC), we address the imperative task of predicting post-surgery treatment needs by identifying crucial tumor features within whole slide images of solid tumors, analogous to locating a needle in a histological haystack. We evaluate two approaches to address this challenge using a small CRC dataset. Methods: First, we explore a conventional tile-level training approach, testing various data augmentation methods to mitigate the memorization effect in a noisy label setting. Second, we examine a multi-instance learning (MIL) approach at the case level, adapting data augmentation techniques to prevent over-fitting in the limited data set context. Results: The tile-level approach proves ineffective due to the limited number of informative image tiles per case. Conversely, the MIL approach demonstrates success for the small dataset when coupled with post-feature vector creation data augmentation techniques. In this setting, the MIL model accurately predicts nodal status corresponding to expert-based budding scores for these cases. Conclusions: This study incorporates data augmentation techniques into a MIL approach, highlighting the effectiveness of the MIL method in detecting predictive factors such as tumor budding, despite the constraints of a limited dataset size.

Indoor surface classification for mobile robots.

The ability to recognize the surface type is crucial for both indoor and outdoor mobile robots. Knowing the surface type can help indoor mobile robots move more safely and adjust their movement accordingly. However, recognizing surface characteristics is challenging since similar planes can appear substantially different; for instance, carpets come in various types and colors. To address this inherent uncertainty in vision-based surface classification, this study first generates a new, unique data set composed of 2,081 surface images (carpet, tiles, and wood) captured in different indoor environments. Secondly, the pre-trained state-of-the-art deep learning models, namely InceptionV3, VGG16, VGG19, ResNet50, Xception, InceptionResNetV2, and MobileNetV2, were utilized to recognize the surface type. Additionally, a lightweight MobileNetV2-modified model was proposed for surface classification. The proposed model has approximately four times fewer total parameters than the original MobileNetV2 model, reducing the size of the trained model weights from 42 MB to 11 MB. Thus, the proposed model can be used in robotic systems with limited computational capacity and embedded systems. Lastly, several optimizers, such as SGD, RMSProp, Adam, Adadelta, Adamax, Adagrad, and Nadam, are applied to distinguish the most efficient network. Experimental results demonstrate that the proposed model outperforms all other applied methods and existing approaches in the literature by achieving 99.52% accuracy and an average score of 99.66% in precision, recall, and F1-score. In addition to this, the proposed lightweight model was tested in real-time on a mobile robot in 11 scenarios consisting of various indoor environments such as offices, hallways, and homes, resulting in an accuracy of 99.25%. Finally, each model was evaluated in terms of model loading time and processing time. The proposed model requires less loading and processing time than the other models.

ScabyNet, a user-friendly application for detecting common scab in potato tubers using deep learning and morphological traits

Common scab (CS) is a major bacterial disease causing lesions on potato tubers, degrading their appearance and reducing their market value. To accurately grade scab-infected potato tubers, this study introduces “ScabyNet”, an image processing approach combining color-morphology analysis with deep learning techniques. ScabyNet estimates tuber quality traits and accurately detects and quantifies CS severity levels from color images. It is presented as a standalone application with a graphical user interface comprising two main modules. One module identifies and separates tubers on images and estimates quality-related morphological features. In addition, it enables the extraction of tubers as standard tiles for the deep-learning module. The deep-learning module detects and quantifies the scab infection into five severity classes related to the relative infected area. The analysis was performed on a dataset of 7154 images of individual tiles collected from field and glasshouse experiments. Combining the two modules yields essential parameters for quality and disease inspection. The first module simplifies imaging by replacing the region proposal step of instance segmentation networks. Furthermore, the approach is an operational tool for an affordable phenotyping system that selects scab-resistant genotypes while maintaining their market standards.