Billions Of Pixels Research Articles

TDT-MIL: a framework with a dual-channel spatial positional encoder for weakly-supervised whole slide image classification.

The classic multiple instance learning (MIL) paradigm is harnessed for weakly-supervised whole slide image (WSI) classification. The spatial position relationship located between positive tissues is crucial for this task due to the small percentage of these tissues in billions of pixels, which has been overlooked by most studies. Therefore, we propose a framework called TDT-MIL. We first serially connect a convolutional neural network and transformer for basic feature extraction. Then, a novel dual-channel spatial positional encoder (DCSPE) module is designed to simultaneously capture the complementary local and global positional information between instances. To further supplement the spatial position relationship, we construct a convolutional triple-attention (CTA) module to attend to the inter-channel information. Thus, the spatial positional and inter-channel information is fully mined by our model to characterize the key pathological semantics in WSI. We evaluated TDT-MIL on two publicly available datasets, including CAMELYON16 and TCGA-NSCLC, with the corresponding classification accuracy and AUC up to 91.54%, 94.96%, and 90.21%, 94.36%, respectively, outperforming state-of-the-art baselines. More importantly, our model possesses a satisfactory capability in solving the imbalanced WSI classification task using an ingenious but interpretable structure.

Prognostic prediction of ovarian cancer based on hierarchical sampling & fine-grained recognition convolution neural network

Ovarian cancer ranks among the deadliest gynecological malignancies, with high-grade serous adenocarcinoma (HGSA) constituting 75 % of ovarian cancer cases and accounting for 80 %–90 % of ovarian cancer-related fatalities. Accurate prognosis prediction for ovarian HGSA is of critical clinical significance. However, existing prognostic analysis methods exhibit suboptimal performance in identifying prognostically relevant pathological markers, making it challenging, even for experienced pathologists, to forecast the prognosis of ovarian HGSA patients accurately. Deep learning holds promise in enhancing prognostic prediction accuracy. However, a significant challenge in this field arises from the impracticality of directly inputting histopathology whole-slide images with millions to billions of pixels into existing deep learning networks for training and inference. To address this issue, we propose a prognostic analysis network for ovarian cancer based on hierarchical sampling and fine-grained recognition. This network comprises a two-stage hierarchical sampling sub-network, a fine-grained image recognition sub-network, and a prognostic analysis sub-network. We assess the system's performance using a pathological dataset of 450 cases of ovarian HGSA diagnosed and treated at the Pathology Department of the West China Second University Hospital of Sichuan University. Results indicate that: (1) The proposed prognostic analysis network based on two-stage hierarchical sampling sub-network can effectively analyze the histopathology whole-slide image; (2) the use of fine-grained image recognition and the introduction of clinical information can improve the performance of HGSA prognosis analysis method, with an improvement range of 4.77–10.66%; and (3) the proposed method can be used for the model analysis of pathological datasets of HGSA. Moreover, this method can be used to explore effective characteristics from the multi-modal dataset through automatic learning with prediction recall, accuracy, and precision rates of 80.0%, 81.1%, and 81.8%, respectively, underscoring its clinical potential. This study reveals the reliability and effectiveness of the proposed prognosis evaluation method of HGCA. Conclusions can help clinicians precisely evaluate the recurrence risk of patients, take the initiative to master the diagnosis and treatment, and increase the long-term survival rate of patients.

Magnifying Networks for Histopathological Images with Billions of Pixels.

Amongst the other benefits conferred by the shift from traditional to digital pathology is the potential to use machine learning for diagnosis, prognosis, and personalization. A major challenge in the realization of this potential emerges from the extremely large size of digitized images, which are often in excess of 100,000 × 100,000 pixels. In this paper, we tackle this challenge head-on by diverging from the existing approaches in the literature-which rely on the splitting of the original images into small patches-and introducing magnifying networks (MagNets). By using an attention mechanism, MagNets identify the regions of the gigapixel image that benefit from an analysis on a finer scale. This process is repeated, resulting in an attention-driven coarse-to-fine analysis of only a small portion of the information contained in the original whole-slide images. Importantly, this is achieved using minimal ground truth annotation, namely, using only global, slide-level labels. The results from our tests on the publicly available Camelyon16 and Camelyon17 datasets demonstrate the effectiveness of MagNets-as well as the proposed optimization framework-in the task of whole-slide image classification. Importantly, MagNets process at least five times fewer patches from each whole-slide image than any of the existing end-to-end approaches.

FR-MIL: Distribution Re-calibration based Multiple Instance Learning with Transformer for Whole Slide Image Classification.

In digital pathology, whole slide images (WSI) are crucial for cancer prognostication and treatment planning. WSI classification is generally addressed using multiple instance learning (MIL), alleviating the challenge of processing billions of pixels and curating rich annotations. Though recent MIL approaches leverage variants of the attention mechanism to learn better representations, they scarcely study the properties of the data distribution itself i.e., different staining and acquisition protocols resulting in intra-patch and inter-slide variations. In this work, we first introduce a distribution re-calibration strategy to shift the feature distribution of a WSI bag (instances) using the statistics of the max-instance (critical) feature. Second, we enforce class (bag) separation via a metric loss assuming that positive bags exhibit larger magnitudes than negatives. We also introduce a generative process leveraging Vector Quantization (VQ) for improved instance discrimination i.e., VQ helps model bag latent factors for improved classification. To model spatial and context information, a position encoding module (PEM) is employed with transformer-based pooling by multi-head self-attention (PMSA). Evaluation of popular WSI benchmark datasets reveals our approach improves over state-of-the-art MIL methods. Further, we validate the general applicability of our method on classic MIL benchmark tasks and for point cloud classification with limited points https://github.com/PhilipChicco/FRMIL.

Multi-scale representation attention based deep multiple instance learning for gigapixel whole slide image analysis.

Recently, convolutional neural networks (CNNs) directly using whole slide images (WSIs) for tumor diagnosis and analysis have attracted considerable attention, because they only utilize the slide-level label for model training without any additional annotations. However, it is still a challenging task to directly handle gigapixel WSIs, due to the billions of pixels and intra-variations in each WSI. To overcome this problem, in this paper, we propose a novel end-to-end interpretable deep MIL framework for WSI analysis, by using a two-branch deep neural network and a multi-scale representation attention mechanism to directly extract features from all patches of each WSI. Specifically, we first divide each WSI into bag-, patch- and cell-level images, and then assign the slide-level label to its corresponding bag-level images, so that WSI classification becomes a MIL problem. Additionally, we design a novel multi-scale representation attention mechanism, and embed it into a two-branch deep network to simultaneously mine the bag with a correct label, the significant patches and their cell-level information. Extensive experiments demonstrate the superior performance of the proposed framework over recent state-of-the-art methods, in term of classification accuracy and model interpretability. All source codes are released at: https://github.com/xhangchen/MRAN/.

Anthropogenic Object Localization: Evaluation of Broad-Area High-Resolution Imagery Scans Using Deep Learning in Overhead Imagery.

Too often, the testing and evaluation of object detection, as well as the classification techniques for high-resolution remote sensing imagery, are confined to clean, discretely partitioned datasets, i.e., the closed-world model. In recent years, the performance on a number of benchmark datasets has exceeded 99% when evaluated using cross-validation techniques. However, real-world remote sensing data are truly big data, which often exceed billions of pixels. Therefore, one of the greatest challenges regarding the evaluation of machine learning models taken out of the clean laboratory setting and into the real world is the difficulty of measuring performance. It is necessary to evaluate these models on a grander scale, namely, tens of thousands of square kilometers, where it is intractable to the ground truth and the ever-changing anthropogenic surface of Earth. The ultimate goal of computer vision model development for automated analysis and broad area search and discovery is to augment and assist humans, specifically human-machine teaming for real-world tasks. In this research, various models have been trained using object classes from benchmark datasets such as UC Merced, PatternNet, RESISC-45, and MDSv2. We detail techniques to scan broad swaths of the Earth with deep convolutional neural networks. We present algorithms for localizing object detection results, as well as a methodology for the evaluation of the results of broad-area scans. Our research explores the challenges of transitioning these models out of the training-validation laboratory setting and into the real-world application domain. We show a scalable approach to leverage state-of-the-art deep convolutional neural networks for the search, detection, and annotation of objects within large swaths of imagery, with the ultimate goal of providing a methodology for evaluating object detection machine learning models in real-world scenarios.

Improving model parsimony and accuracy by modified greedy feature selection in digital soil mapping

In the context of increasing soil degradation worldwide, spatially explicit soil information is urgently needed to support decision-making for sustaining limited soil resources. Digital soil mapping (DSM) has been proven as an efficient way to deliver soil information from local to global scales. The number of environmental covariates used for DSM has rapidly increased due to the growing volume of remote sensing data, therefore variable selection is necessary to deal with multicollinearity and improve model parsimony. Compared with Boruta, recursive feature elimination (RFE), and variance inflation factor (VIF) analysis, we proposed the use of modified greedy feature selection (MGFS), for DSM regression. For this purpose, using quantile regression forest, 402 soil samples and 392 environmental covariates were used to map the spatial distribution of soil organic carbon density (SOCD) in Northeast and North China. The result showed that MGFS selected the most parsimonious model with only 9 covariates (e.g., brightness index, mean annual temperature), much lower than RFE (22 covariates), VIF (30 covariates), and Boruta (76 covariates). The repeated validation (50 times) showed that the MGFS derived model performed better (R2 of 0.60, LCCC of 0.74, RMSE of 13.80 t ha−1) than these using full covariates, Boruta, RFE and VIF (R2 of 0.48–0.57, LCCC of 0.64–0.72, RMSE of 14.24–15.79 t ha−1). Despite the similar performance of the uncertainty estimate (PICP), the model using MGFS and RFE had the lowest global uncertainty (0.86) as indicated by the uncertainty index. In addition, MGFS had the best computation efficiency when considering the steps of variable selection and map prediction. Given these advantages over Boruta, RFE and VIF, MGFS has a high potential in fine-resolution soil mapping practices, especially for these studies at a broad scale involving heavy computation on millions or billions of pixels.

HAT: a high-energy surface X-ray diffraction analysis toolkit.

This work introduces the high-energy surface X-ray diffraction analysis toolkit (HAT), an open-source cross-platform software package written in Python to allow the extraction and processing of high-energy surface X-ray diffraction (HESXRD) data sets. Thousands of large-area detector images are collected in a single HESXRD scan, corresponding to billions of pixels and hence reciprocal space positions. HAT is an optimized reciprocal space binner that implements a graphical user interface to allow the easy and interactive exploration of HESXRD data sets. Regions of reciprocal space can be selected with movable and resizable masks in multiple views and are projected onto different axes to allow the creation of reciprocal space maps and the extraction of crystal truncation rods. Current and future versions of HAT can be downloaded and used free of charge.

High performance w-stacking for imaging radio astronomy data: a parallel and accelerated solution

Abstract Current and upcoming radio-interferometers are expected to produce volumes of data of increasing size that need to be processed in order to generate the corresponding sky brightness distributions through imaging. This represents an outstanding computational challenge, especially when large fields of view and/or high-resolution observations are processed. We have investigated the adoption of modern high performance computing systems specifically addressing the gridding, fast Fourier transform, and w-correction of imaging, combining parallel and accelerated solutions. We have demonstrated that the code we have developed can support data set and images of any size compatible with the available hardware, efficiently scaling up to thousands of cores or hundreds of graphic processing units, keeping the time to solution <1 h even when images of the size of the order of billions or tens of billions of pixels are generated. In addition, portability has been targeted as a primary objective, both in terms of usability on different computing platforms and in terms of performance. The presented results have been obtained on two different state-of-the-art high performance computing architectures.

PS-BPC01-6: DEEP LEARNING IS USEFUL FOR THE PATHOLOGICAL ANALYSES OF HUMAN KIDNEY BIOPSY IMAGES

Objective: The kidney plays a pivotal role in the maintenance of blood pressure. Non-neoplastic kidney biopsy images yield a great deal of diagnostic information, however, several studies have shown that there is room for improvement in pathological diagnostic reproducibility. Recently, deep learning has made major breakthroughs in various tasks. Therefore, we tried to analyze human kidney biopsy images using deep learning. Design and Method: We analyzed 5002 non-neoplastic kidney biopsy whole slide images (WSIs) obtained from 24 facilities in Japan using deep learning. Images of periodic acid-Schiff stained kidney sections were analyzed. Results: WSIs have billions of pixels, and deep neural networks (NNs) cannot accept such a huge image as an input. Therefore, we prepared 88 μm square 6,125,990 patch images (PIs) from the WSIs, and trained NNs with these PIs. We manually selected 259,682 typical PIs from the whole set of PIs, and annotated them into 56 classes. Using these annotated PIs, we trained EfficientNets B0-B7, whose network depth and width are carefully balanced for various input resolutions (224 to 600 pixels), and found that EfficientNetB4 (hereafter referred to CNN_Man) achieved the best F1 score. Using histological parameters obtained from renal biopsy reports made by nephrologists and pathologists at each facility, we confirmed the reliability of CNN_Man-based prediction. We also finely visualized histological features of WSIs based on CNN_Man prediction (Figure). In addition to these supervised histological analyses, we tried to extract disease features in a loosely supervised manner. In this loosely supervised learning, we tried to extract disease-related rather than histology-related glomerular features by training EfficientNetB4 using final diagnosis as true labels. It is noteworthy that the final-diagnosis-based training is not typical supervised learning because each disease class contains patch images with various histological features to varying degrees. We extracted glomerulus-related PIs by using CNN_Man, and then labeled these PIs into 20 final-diagnosis categories. EfficientNetB4 (hereafter CNN_Diag) was trained with the 20 class images in a 5-fold cross-validation manner. Using UMAP, we visualized the final layer parameters of CNN_Diag corresponding to each PI. This approach divided PIs from IgA nephropathy into many clusters, and each cluster reflected blood pressure, renal function, age, etc., even though they were not explicitly given as teacher labels. Conclusions: The analysis of kidney biopsy images using deep learning may be useful for the classification of renal diseases based on new perspectives.

A Deep Learning Quantification Algorithm for HER2 Scoring of Gastric Cancer.

Gastric cancer is the third most common cause of cancer-related death in the world. Human epidermal growth factor receptor 2 (HER2) positive is an important subtype of gastric cancer, which can provide significant diagnostic information for gastric cancer pathologists. However, pathologists usually use a semi-quantitative assessment method to assign HER2 scores for gastric cancer by repeatedly comparing hematoxylin and eosin (H&E) whole slide images (WSIs) with their HER2 immunohistochemical WSIs one by one under the microscope. It is a repetitive, tedious, and highly subjective process. Additionally, WSIs have billions of pixels in an image, which poses computational challenges to Computer-Aided Diagnosis (CAD) systems. This study proposed a deep learning algorithm for HER2 quantification evaluation of gastric cancer. Different from other studies that use convolutional neural networks for extracting feature maps or pre-processing on WSIs, we proposed a novel automatic HER2 scoring framework in this study. In order to accelerate the computational process, we proposed to use the re-parameterization scheme to separate the training model from the deployment model, which significantly speedup the inference process. To the best of our knowledge, this is the first study to provide a deep learning quantification algorithm for HER2 scoring of gastric cancer to assist the pathologist's diagnosis. Experiment results have demonstrated the effectiveness of our proposed method with an accuracy of 0.94 for the HER2 scoring prediction.

MPMR: Multi-Scale Feature and Probability Map for Melanoma Recognition.

Malignant melanoma (MM) recognition in whole-slide images (WSIs) is challenging due to the huge image size of billions of pixels and complex visual characteristics. We propose a novel automatic melanoma recognition method based on the multi-scale features and probability map, named MPMR. First, we introduce the idea of breaking up the WSI into patches to overcome the difficult-to-calculate problem of WSIs with huge sizes. Second, to obtain and visualize the recognition result of MM tissues in WSIs, a probability mapping method is proposed to generate the mask based on predicted categories, confidence probabilities, and location information of patches. Third, considering that the pathological features related to melanoma are at different scales, such as tissue, cell, and nucleus, and to enhance the representation of multi-scale features is important for melanoma recognition, we construct a multi-scale feature fusion architecture by additional branch paths and shortcut connections, which extracts the enriched lesion features from low-level features containing more detail information and high-level features containing more semantic information. Fourth, to improve the extraction feature of the irregular-shaped lesion and focus on essential features, we reconstructed the residual blocks by a deformable convolution and channel attention mechanism, which further reduces information redundancy and noisy features. The experimental results demonstrate that the proposed method outperforms the compared algorithms, and it has a potential for practical applications in clinical diagnosis.

Accurate reconstruction of horizontal parallax-only holograms by angular spectrum and efficient zero-padding.

Accurate reconstruction of digital holograms that are large in the x direction and small in the y direction, known as horizontal parallax only digital hologram (HPO-DH), must be carried out by non-paraxial propagation approaches such as the classical angular spectrum (AS) method. However, the required space-bandwidth product (SBP) for reconstruction of HPO-DHs requires billions of pixels, which is computationally intensive. Moreover, application of zero-padding for removing aliasing components would generate an unbearable computational burden. In this work, a novel AS technique that reconstructs non-paraxial HPO-DHs with low SBP is proposed. The proposed technique first employs the multi-Fourier transform plane propagation method, which avoids the increase of size in the vertical direction of the HPO-DH to be processed. The second ingredient for field calculation is coherent superposition of vertical tiles formed from the multi-Fourier transform calculations. The described methodology enables reconstruction of HPO-DHs with the AS method and reduced SBP. Efficient managing of the SBP allows implementing zero-padding strategies in the x direction. It is shown that the padding strategies can be implemented in the frequency, space, and space-frequency domains. Hence, suppression of aliased components and increase of the spatial resolution is possible at the same time. Finally, the accuracy and utility of the developed technique is proved by both numerical simulations and experiments.

Breast Histopathology with High-Performance Computing and Deep Learning

The increasingly intensive collection of digitalized images of tumor tissue over the last decade made histopathology a demanding application in terms of computational and storage resources. With images containing billions of pixels, the need for optimizing and adapting histopathology to large-scale data analysis is compelling. This paper presents a modular pipeline with three independent layers for the detection of tumoros regions in digital specimens of breast lymph nodes with deep learning models. Our pipeline can be deployed either on local machines or high-performance computing resources with a containerized approach. The need for expertise in high-performance computing is removed by the self-sufficient structure of Docker containers, whereas a large possibility for customization is left in terms of deep learning models and hyperparameters optimization. We show that by deploying the software layers in different infrastructures we optimize both the data preprocessing and the network training times, further increasing the scalability of the application to datasets of approximatively 43 million images. The code is open source and available on Github.

Deep Learning for Whole Slide Image Analysis: An Overview.

The widespread adoption of whole slide imaging has increased the demand for effective and efficient gigapixel image analysis. Deep learning is at the forefront of computer vision, showcasing significant improvements over previous methodologies on visual understanding. However, whole slide images have billions of pixels and suffer from high morphological heterogeneity as well as from different types of artifacts. Collectively, these impede the conventional use of deep learning. For the clinical translation of deep learning solutions to become a reality, these challenges need to be addressed. In this paper, we review work on the interdisciplinary attempt of training deep neural networks using whole slide images, and highlight the different ideas underlying these methodologies.

The Use of Gigapixel Photogrammetry for the Understanding of Landslide Processes in Alpine Terrain

The work in this paper illustrates an experimental application for geosciences by coupling new and low cost photogrammetric techniques: Gigapixel and Structure-from-Motion (SfM). Gigapixel photography is a digital image composed of billions of pixels (≥1000 megapixels) obtained from a conventional Digital single-lens reflex camera (DSLR), whereas the SfM technique obtains three-dimensional (3D) information from two-dimensional (2D) image sequences. The field test was carried out at the Ingelsberg slope (Bad Hofgastein, Austria), which hosts one of the most dangerous landslides in the Salzburg Land. The stereographic analysis carried out on the preliminary 3D model, integrated with Ground Based Synthetic Aperture Radar Interferometry (GBInSAR) data, allowed us to obtain the main fractures and discontinuities of the unstable rock mass.

A practical trial of landslide detection from single-temporal Landsat8 images using contour-based proposals and random forest: a case study of national Nepal

Landslides are frequent all around the world, causing tremendous loss to human beings. Rapid access to the locations where landslides occur is crucial for emergency response. Most researches in landslide detection from remotely sensed images focus on small regions, which are handpicked. That makes it easy to distinguish landslides from background objects, but hard to apply in practical cases. The complicated non-landslide background pixels increase the difficulty to accurately detect landslides. In this study, we propose a technique framework to remove non-landslide background pixels for national Nepal using 12 Landsat8 images and digital elevation model (DEM). DEM is useful in removing flat areas, where landslides are less likely to occur. The framework consists of three sections: image enhancement, landslide proposal extraction, and detection model setup. Bare land, including landslides, is enhanced using vegetation index after haze/cloud re-movement. Later, calculate connective contours and propose them as potential regions that may contain landslides. For each proposal, calculate texture feature and build detection model using one of the Landsat8 images, which is further applied on other images to check its applicability and robustness. The assessment shows that the method is able to remove 99% of the background pixels in the scale of national Nepal, taking over billions of pixels. Even there is still much to do to achieve high accurate landslide detection results from large-scale images, the experiment validates a strong potential applicability for the proposed method in large-scale landslide-related analysis.

A resolution adaptive deep hierarchical (RADHicaL) learning scheme applied to nuclear segmentation of digital pathology images

Deep learning (DL) has recently been successfully applied to a number of image analysis problems. However, DL approaches tend to be inefficient for segmentation on large image data, such as high-resolution digital pathology slide images. For example, typical breast biopsy images scanned at 40 magnification contain billions of pixels, of which usually only a small percentage belong to the class of interest. For a typical naïve deep learning scheme, parsing through and interrogating all the image pixels would represent hundreds if not thousands of hours of compute time using high performance computing environments. In this paper, we present a resolution adaptive deep hierarchical (RADHicaL) learning scheme wherein DL networks at lower resolutions are leveraged to determine if higher levels of magnification, and thus computation, are necessary to provide precise results. We evaluate our approach on a nuclear segmentation task with a cohort of 141 ER+ breast cancer images and show we can reduce computation time on average by about 85%. Expert annotations of 12,000 nuclei across these 141 images were employed for quantitative evaluation of RADHicaL. A head-to-head comparison with a naïve DL approach, operating solely at the highest magnification, yielded the following performance metrics: .9407 vs .9854 Detection Rate, .8218 vs .8489 F-score, .8061 vs .8364 true positive rate and .8822 vs 0.8932 positive predictive value. Our performance indices compare favourably with state of the art nuclear segmentation approaches for digital pathology images.

An Interactive Learning Framework for Scalable Classification of Pathology Images.

Recent advances in microscopy imaging and genomics have created an explosion of patient data in the pathology domain. Whole-slide images (WSIs) of tissues can now capture disease processes as they unfold in high resolution, recording the visual cues that have been the basis of pathologic diagnosis for over a century. Each WSI contains billions of pixels and up to a million or more microanatomic objects whose appearances hold important prognostic information. Computational image analysis enables the mining of massive WSI datasets to extract quantitative morphologic features describing the visual qualities of patient tissues. When combined with genomic and clinical variables, this quantitative information provides scientists and clinicians with insights into disease biology and patient outcomes. To facilitate interaction with this rich resource, we have developed a web-based machine-learning framework that enables users to rapidly build classifiers using an intuitive active learning process that minimizes data labeling effort. In this paper we describe the architecture and design of this system, and demonstrate its effectiveness through quantification of glioma brain tumors.

An efficient and comprehensive method for drainage network extraction from DEM with billions of pixels using a size-balanced binary search tree

With the increasing resolution of digital elevation models (DEMs), computational efficiency problems have been encountered when extracting the drainage network of a large river basin at billion-pixel scales. The efficiency of the most time-consuming depression-filling pretreatment has been improved by using the O(NlogN) complexity least-cost path search method, but the complete extraction steps following this method have not been proposed and tested. In this paper, an improved O(NlogN) algorithm was proposed by introducing a size-balanced binary search tree (BST) to improve the efficiency of the depression-filling pretreatment further. The following extraction steps, including the flow direction determination and the upslope area accumulation, were also redesigned to benefit from this improvement. Therefore, an efficient and comprehensive method was developed. The method was tested to extract drainage networks of 31 river basins with areas greater than 500,000km2 from the 30-m-resolution ASTER GDEM and two sub-basins with areas of approximately 1000km2 from the 1-m-resolution airborne LiDAR DEM. Complete drainage networks with both vector features and topographic parameters were obtained with time consumptions in O(NlogN) complexity. The results indicate that the developed method can be used to extract entire drainage networks from DEMs with billions of pixels with high efficiency.