Spatial Image Features Research Articles

Full-scale semantic segmentation of hyperspectral imaging based on spatial spatial-spectral joint network

Abstract. Hyperspectral images contain dozens or even hundreds of spectral bands, which contain rich spectral information and help distinguish different ground objects. Hyperspectral images have a wide range of applications in urban planning, environmental monitoring, and other fields. The semantic segmentation of hyperspectral images is one of the current research hotspots. The difficulty lies in the rich spectral information and strong correlation of hyperspectral images. Traditional semantic segmentation methods cannot fully extract information, which affects the accuracy of classification. This article utilizes an encoding decoding structure to simultaneously extract deep and shallow features of images. A REGCS convolution module was constructed using the idea of group convolution to extract spectral and spatial features of images. We compared the Salinas Valley dataset and MUUFL dataset with various classification algorithms. The experimental results show that compared with other classification models, the RESSU model has achieved stable and excellent results in hyperspectral image classification experiments. Among them, in the classification experiment of the Salinas Valley dataset, the accuracy of single class classification reached over 92%. In the effectiveness analysis experiment, we calculated different model parameter quantities to verify the performance of our method, and ultimately achieved good results.

Semantic-aware normalizing flow with feature fusion for image anomaly detection

In the field of computer vision, anomaly detection is a binary classification task used to identify exceptional instances within image datasets. Typically, it can be divided into two aspects: texture defect detection and semantic anomaly detection. Existing methods often use pre-trained feature extractors to singly capture semantic or spatial features of images, and then employ different classifiers to handle these two types of anomaly detection tasks. However, these methods fail to fully utilize the synergistic relationship between these two types of features, resulting in algorithms that excel in one type of anomaly detection task but perform poorly in the other type. Therefore, we propose a novel approach that successfully combines these two types of features into a normalizing flow learning module to address both types of anomaly detection tasks. Specifically, we first adopt a pre-trained Vision Transformer (ViT) model to capture both texture and semantic features of input images. Subsequently, using the semantic features as input, we design a novel normalizing flow model to fit the semantic distribution of normal data. In addition, we introduce a feature fusion module based on attention mechanisms to integrate the most relevant texture and semantic information between these two types of features, significantly enhancing the model’s ability to simultaneously represent the spatial texture and semantic features of the input image. Finally, We conduct comprehensive experiments on well-known semantic and texture anomaly detection datasets, namely Cifar10 and MVTec, to evaluate the performance of our proposed method. The results demonstrate that our model achieves outstanding performance in both semantic and texture anomaly detection tasks, particularly achieving state-of-the-art results in semantic anomaly detection.

Sine tangent search algorithm enabled LeNet for cotton crop classification using satellite image

Economic growth of country largely depends on crop production quantity and quality. Among various crops, cotton is one of the major crops in India, where 23 percent of cotton gets exported to various other countries. To classify these cotton crops, farmers consume much time, and this remains inaccurate most probably. Hence, to eradicate this issue, cotton crops are classified using deep learning model, named LeNet in this research paper. Novelty of this paper lies in utilization of hybrid optimization algorithm, named proposed sine tangent search algorithm for training LeNet. Initially, hyperspectral image is pre-processed by anisotropic diffusion, and then allowed for further processing. Also, SegNet is deep learning model that is used for segmenting pre-processed image. For perfect and clear details of pre-processed image, feature extraction is carried out, wherein vegetation index and spectral spatial features of image are found accurately. Finally, cotton crop is classified from segmented image and features extracted, using LeNet that is trained by sine tangent search algorithm. Here, sine tangent search algorithm is formed by hybridization of sine cosine algorithm and tangent search algorithm. Then, performance of sine tangent search algorithm enabled LeNet is assessed with evaluation metrics along with Receiver Operating Characteristic (ROC) curve. These metrics showed that sine tangent search algorithm enabled LeNet is highly effective for cotton crop classification with superior values of accuracy of 91.7%, true negative rate of 92%, and true positive rate of 92%.

Combining Color and Spatial Image Features for Unsupervised Image Segmentation with Mixture Modelling and Spectral Clustering

The demand for accurate and reliable unsupervised image segmentation methods is high. Regardless of whether we are faced with a problem for which we do not have a usable training dataset, or whether it is not possible to obtain one, we still need to be able to extract the desired information from images. In such cases, we are usually gently pushed towards the best possible clustering method, as it is often more robust than simple traditional image processing methods. We investigate the usefulness of combining two clustering methods for unsupervised image segmentation. We use the mixture models to extract the color and spatial image features based on the obtained output segments. Then we construct a similarity matrix (adjacency matrix) based on these features to perform spectral clustering. In between, we propose a label noise correction using Markov random fields. We investigate the usefulness of our method on many hand-crafted images of different objects with different shapes, colorization, and noise. Compared to other clustering methods, our proposal performs better, with 10% higher accuracy. Compared to state-of-the-art supervised image segmentation methods based on deep convolutional neural networks, our proposal proves to be competitive.

Hyperspectral imaging combined with dual-channel deep learning feature fusion model for fast and non-destructive recognition of brew wheat varieties

In this study, a dual-channel deep learning feature fusion model (DLFM) was developed to process hyperspectral imaging data for the rapid and nondestructive identification of brewing wheat varieties. The DLFM model extracts spectral features using a one-dimensional convolution module and spatial image features from the RGB image using a two-dimensional convolution module. These features are then fused using a feature adaptive fusion module within the DLFM and input into the fully connected layer for variety recognition. Support vector machine (SVM), one-dimensional convolutional neural network (1DCNN), two-dimensional convolutional neural network (2DCNN), and DLFM were built, respectively. Among them, DLFM had the highest recognition accuracy, which was 99.18%, 97.30%, and 93.18% for the three-variety, four-variety, and five-variety wheat combinations, respectively. The average accuracies of all combinations were improved by 11.93%, 6.84%, 12.54%, and 2.39% for 1DCNN, 2DCNN, and 1DCNN of fused data, respectively, compared to SVM. The results show that hyperspectral imaging (HSI) combined with DLFM can realize fast and nondestructive identification of different brewing wheat varieties, providing a new method for variety identification of cereals.

Novel deep learning–based prognostic prediction model for colorectal cancer using hematoxylin and eosin–stained whole slide images.

3119 Background: Previous studies have shown that the presence or absence of genetic mutations is critical for colorectal cancer prognosis. However, genomic testing can be expensive and difficult to perform on all samples. In contrast, hematoxylin and eosin (H&E) staining is relatively inexpensive and can be performed on all tissue specimens. In this study, we designed a novel prognostic method using spatial image features extracted from H&E–stained whole slide images (WSIs) and genetic mutation prediction neural networks. Methods: We obtained H&E–stained WSIs and data on Microsatellite Instability ( MSI), BRAF, TTN and APC gene mutations from a clinical cohort of 548 patients with The Cancer Genome Atlas (TCGA) Colon adenocarcinoma and rectum adenocarcinoma. We divided them into training (n=361), validation (n=90), and test (n=115) groups. Classification models were trained to predict the presence or absence of MSI, BRAF, TTN, and APC mutations. The model input comprised features of the H&E–stained WSIs, as obtained via a deep learning–based feature extractor. All resultant models were incorporated into a prognostic model (overall survival: > 60 months (low risk)/< 60 months (high risk)). Our prognostic model’s performance was evaluated against TCGA colorectal dataset, and a survival analysis was performed on the model using the Kaplan–Meier method. Finally, we compared our model’s performance with the end–to–end prognostic prediction of a convolutional neural network (CNN) that also used H&E–stained WSIs as input and provided prognostic prediction as output. Results: Our deep learning–based prognostic prediction model achieved an AUC score of 0.834 with a 95% confidence interval (CI) of 0.734–1.000 alongside TCGA dataset; the survival analysis compared the survival distributions of low–risk and high–risk groups, as predicted by our model; a p–value < 0.01 was obtained. The model could classify low– and high–risk patients and accurately predict patient status as alive (low risk) or deceased (high risk) at 60 months. In contrast, the CNN–based model achieved an AUC score of only 0.502 (95% CI: 0.315–0.690) on the same TCGA dataset, and the p–value obtained for it under the Kaplan–Meier log–rank test was greater than 0.5. The CNN–based method was unable to distinguish between low– and high–risk patients, confirming that our method using spatial imaging features extracted from WSIs was a more effective approach. Conclusions: We developed a novel prognostic prediction method using spatial image features extracted from WSIs and genetic mutation prediction neural networks. Our results demonstrated the advantage of using image features over gene mutation data for prognostic prediction in colorectal cancer patients.

HIGF-Net: Hierarchical information-guided fusion network for polyp segmentation based on transformer and convolution feature learning

Polyp segmentation plays a role in image analysis during colonoscopy screening, thus improving the diagnostic efficiency of early colorectal cancer. However, due to the variable shape and size characteristics of polyps, small difference between lesion area and background, and interference of image acquisition conditions, existing segmentation methods have the phenomenon of missing polyp and rough boundary division. To overcome the above challenges, we propose a multi-level fusion network called HIGF-Net, which uses hierarchical guidance strategy to aggregate rich information to produce reliable segmentation results. Specifically, our HIGF-Net excavates deep global semantic information and shallow local spatial features of images together with Transformer encoder and CNN encoder. Then, Double-stream structure is used to transmit polyp shape properties between feature layers at different depths. The module calibrates the position and shape of polyps in different sizes to improve the model's efficient use of the rich polyp features. In addition, Separate Refinement module refines the polyp profile in the uncertain region to highlight the difference between the polyp and the background. Finally, in order to adapt to diverse collection environments, Hierarchical Pyramid Fusion module merges the features of multiple layers with different representational capabilities. We evaluate the learning and generalization abilities of HIGF-Net on five datasets using six evaluation metrics, including Kvasir-SEG, CVC-ClinicDB, ETIS, CVC-300, and CVC-ColonDB. Experimental results show that the proposed model is effective in polyp feature mining and lesion identification, and its segmentation performance is better than ten excellent models.

Latent Feature Detection Using LDA in Interior Image

Interior image recommendations have been paid attention to with the development of image retrieval technology. Although a classification model is often used to model user-preferred interiors, it is not suitable for explaining why users prefer them. On the other hand, LDA, which analyzes sentence features based on latent features, can better understand sentences by estimating latent features in sentences. Therefore, we propose a latent feature estimation method in interior images by incorporating spatial features of images and LDA to understand the structure of preferred interior images by users. Specifically, instead of sentences, we use interior images to extract spatial features using BoVW based on SURF features and estimate topics based on the histograms by LDA. We compared the proposed method with subject experiments in interior images with several styles. We verified that the proposed method could estimate similar topics compared to the subject’s subjective view.

Image segmentation of skin lesions based on dense atrous spatial pyramid pooling and attention mechanism

The skin is the largest organ of the human body, and many visceral diseases will be directly reflected on the skin, so it is of great clinical significance to accurately segment the skin lesion images. To address the characteristics of complex color, blurred boundaries, and uneven scale information, a skin lesion image segmentation method based on dense atrous spatial pyramid pooling (DenseASPP) and attention mechanism is proposed. The method is based on the U-shaped network (U-Net). Firstly, a new encoder is redesigned to replace the ordinary convolutional stacking with a large number of residual connections, which can effectively retain key features even after expanding the network depth. Secondly, channel attention is fused with spatial attention, and residual connections are added so that the network can adaptively learn channel and spatial features of images. Finally, the DenseASPP module is introduced and redesigned to expand the perceptual field size and obtain multi-scale feature information. The algorithm proposed in this paper has obtained satisfactory results in the official public dataset of the International Skin Imaging Collaboration (ISIC 2016). The mean Intersection over Union (mIOU), sensitivity (SE), precision (PC), accuracy (ACC), and Dice coefficient (Dice) are 0.901 8, 0.945 9, 0.948 7, 0.968 1, 0.947 3, respectively. The experimental results demonstrate that the method in this paper can improve the segmentation effect of skin lesion images, and is expected to provide an auxiliary diagnosis for professional dermatologists.

Research on Animated GIFs Emotion Recognition Based on ResNet-ConvGRU

Animated Graphics Interchange Format (GIF) images have become an important part of network information interaction, and are one of the main characteristics of analyzing social media emotions. At present, most of the research on GIF affection recognition fails to make full use of spatial-temporal characteristics of GIF images, which limits the performance of model recognition to a certain extent. A GIF emotion recognition algorithm based on ResNet-ConvGRU is proposed in this paper. First, GIF data is preprocessed, converting its image sequences to static image format for saving. Then, the spatial features of images and the temporal features of static image sequences are extracted with ResNet and ConvGRU networks, respectively. At last, the animated GIFs data features are synthesized and the seven emotional intensities of GIF data are calculated. The GIFGIF dataset is used to verify the experiment. From the experimental results, the proposed animated GIFs emotion recognition model based on ResNet-ConvGRU, compared with the classical emotion recognition algorithms such as VGGNet-ConvGRU, ResNet3D, CNN-LSTM, and C3D, has a stronger feature extraction ability, and sentiment classification performance. This method provides a finer-grained analysis for the study of public opinion trends and a new idea for affection recognition of GIF data in social media.

Prediction of regional vegetation cover using spatial image features and semantic segmentation

Vegetation is an essential part of our ecosystem; it also determines health of our planet. According to “The State of the World’s Forests 2020” only 31% of the global land area is covered with vegetation. This paper presents a superior way of representing vegetation cover in a region by utilizing Atmospherically Resistant Vegetation Index (ARVI) as vital features out of multispectral satellite images. These images comprise Green, Blue, Red and Near Infrared bands data which was further utilized by U-Net to efficiently segment satellite images. In remote sensing greenery of environment is traditionally determined using NDVI, one of the critical imperfections with this method is that it is liable to compute inaccurate values as a consequence of variations in soil, air moisture and shadowing affected by varying incidence angle of sunlight. On the other hand, ARVI is immune to such flaws and integration with deep learning provides a better solution for segmenting regional vegetation and predicting its coverage up to some extent. This method can be employed for predicting vegetation in any region along with assisting in events such as repopulating trees and urban planning while conserving beauty of our nature.

Combined Multi-Atlas and Multi-Layer Perception for Alzheimer's Disease Classification.

Alzheimer's disease (AD) is a progressive and irreversible neurodegenerative disease. To distinguish the stage of the disease, AD classification technology challenge has been proposed in Pattern Recognition and Computer Vision 2021 (PRCV 2021) which provides the gray volume and average cortical thickness data extracted in multiple atlases from magnetic resonance imaging (MRI). Traditional methods either train with convolutional neural network (CNN) by MRI data to adapt the spatial features of images or train with recurrent neural network (RNN) by temporal features to predict the next stage. However, the morphological features from the challenge have been extracted into discrete values. We present a multi-atlases multi-layer perceptron (MAMLP) approach to deal with the relationship between morphological features and the stage of the disease. The model consists of multiple multi-layer perceptron (MLP) modules, and morphological features extracted from different atlases will be classified by different MLP modules. The final vote of all classification results obtains the predicted disease stage. Firstly, to preserve the diversity of brain features, the most representative atlases are chosen from groups of similar atlases, and one atlas is selected in each group. Secondly, each atlas is fed into one MLP to fetch the score of the classification. Thirdly, to obtain more stable results, scores from different atlases are combined to vote the result of the classification. Based on this approach, we rank 10th among 373 teams in the challenge. The results of the experiment indicate as follows: (1) Group selection of atlas reduces the number of features required without reducing the accuracy of the model; (2) The MLP architecture achieves better performance than CNN and RNN networks in morphological features; and (3) Compared with other networks, the combination of multiple MLP networks has faster convergence of about 40% and makes the classification more stable.

Are low-level visual features of scenes associated with perceived restorative qualities?

Exposure to the natural environment has been associated with cognitive benefits. Attention Restoration Theory (ART) proposes that the physical environment, including the natural environment, can be cognitively restorative if the setting embodies a set of high-level qualities (e.g., fascination). However, these restorative qualities are typically measured subjectively. In this study, we examined whether a set of objectively measured low-level visual features predict subjective restorative qualities that are based on ART. Eighty-eight participants from Amazon Mechanical Turk rated 592 images on naturalness and four restorative qualities. Color and spatial features of images explained 5–20% of the variance in ratings of restorative qualities. We also found that visual features partially mediated the relationship between naturalness and restorative qualities. These results suggest that a small but significant portion of the perception of restorative qualities is grounded in objectively measured low-level visual features. Further research is needed to understand the potential for and limits of visual features as predictors of restorative qualities.

Regional Vegetation Cover Prediction Using Spatial Image Features and Semantic Segmentation

Regional Vegetation Cover Prediction Using Spatial Image Features and Semantic Segmentation

A Deep-ConvLSTM Collision Prediction Model for Manipulators in Dynamic Environment

Obstacle avoidance is one of the fundamental problems in human-robot collaboration (HRC) studies. The close proximity between robots and human usually leaves robots a short period of time to re-plan a safe motion, especially when facing non-static obstacles. Therefore, to identify collisions in advance and mitigate the computational efforts regarding robot motion re-planning, this paper proposes a network-based stop-go algorithm that uses only images capturing the states of the robot arm and a non-static obstacle without accessing any robot dynamics. In particular, a deep convolutional long-short-term memory (ConvLSTM) neural network is first developed to learn the spatial features of images, and predict both the robot arm and the non-static obstacle states five steps in advance. Next, the predictions are set back to the robot arm so that the robot arm would halt the current task when a potential future collision is identified. Eventually, the robot arm resumes the task after the non-static obstacle is clear. Extensive numerical studies have been conducted to validate the effectiveness of the proposed trajectory prediction scheme in presence of obstacles.

Densely connected convolutional networks with attention long short-term memory for estimating PM2.5 values from images

The PM2.5 index is a vitally important air pollution indicator that reflects the atmospheric concentration of particulate matter with diameters less than 2.5 μm. Recently, researchers have sought to address the inadequacies involved with the capture of environmental data from monitoring stations by developing deep learning methods for estimating PM2.5 concentrations in real time based on image data. However, the currently available methods neglect the effect of PM2.5 concentrations in earlier periods on the current air quality. The present work addresses this issue by proposing a novel hybrid model that first applies the DenseNet model to extract the local high-order spatial features of images captured over several hours or days at a fixed location, and then feeds the extracted features as a time series into an attention-based long short-term memory model. Ultimately, a fully connected layer is applied to establish a regression model to evaluate the PM2.5 concentrations reflected in the image data. The effectiveness of the proposed method for assessing PM2.5 concentrations is demonstrated by its application to two datasets composed of images captured in Shanghai and Beijing, China.

Forecasting of Typhoon-Induced Wind-Wave by Using Convolutional Deep Learning on Fused Data of Remote Sensing and Ground Measurements.

Taiwan is an island, and its economic activities are primarily dependent on maritime transport and international trade. However, Taiwan is also located in the region of typhoon development in the Northwestern Pacific Basin. Thus, it frequently receives strong winds and large waves brought by typhoons, which pose a considerable threat to port operations. To determine the real-time status of winds and waves brought by typhoons near the coasts of major ports in Taiwan, this study developed models for predicting the wind speed and wave height near the coasts of ports during typhoon periods. The forecasting horizons range from 1 to 6 h. In this study, the gated recurrent unit (GRU) neural networks and convolutional neural networks (CNNs) were combined and adopted to formulate the typhoon-induced wind and wave height prediction models. This work designed two wind speed prediction models (WIND-1 and WIND-2) and four wave height prediction models (WAVE-1 to WAVE-4), which are based on the WIND-1 and WIND-2 model outcomes. The Longdong and Liuqiu Buoys were the experiment locations. The observatory data from the ground stations and buoys, as well as radar reflectivity images, were adopted. The results indicated that, first, WIND-2 has a superior wind speed prediction performance to WIND-1, where WIND-2 can be used to identify the temporal and spatial changes in wind speeds using ground station data and reflectivity images. Second, WAVE-4 has the optimal wave height prediction performance, followed by WAVE-3, WAVE-2, and WAVE-1. The results of WAVE-4 revealed using the designed models with in-situ and reflectivity data directly yielded optimal predictions of the wind-based wave heights. Overall, the results indicated that the presented combination models were able to extract the spatial image features using multiple convolutional and pooling layers and provide useful information from time-series data using the GRU memory cell units. Overall, the presented models could exhibit promising results.

Abstract PO-030: Deep-learning image-based tumor, stroma, and lymphocytes spatial relationships and clinicopathological features that affect survival in pancreatic cancer patients

Abstract Background: Stromal and lymphocyte density have each been implicated in differential survival in pancreatic cancer (Torphy, 2018 & Orhan, 2020). In this study, we developed an automated deep-learning system to provide risk-assessment upon spatial relationships between tumor, stroma, and lymphocyte regions in pancreatic pathology images. Methods: Diagnostic H&E-stained pathology images from 82 pancreatic adenocarcinoma patients who underwent chemotherapy were acquired from TCGA sources. Thirty-two patients were held out for testing purposes. Tumor, stroma, and lymphocytes image masks were generated using pre-trained convolutional neural networks, and their co-distribution was summarized in nine numerical image-based features. Optimal thresholds in these image-based features were identified using 2-way Gaussian mixture models. This process found four spatial image features that significantly contributed to low-risk of early death: Low lymphocyte count, lymphocytes adjacent to tumor regions, and stromal adjacency to tumor regions. Ability to separate patients based on these features was evaluated using silhouette score, concordance index, and Cox proportional hazards ratios (HR). Results: Without using image-based features, exhaustive search of this cohort’s clinicopathological features found an optimal Cox proportional hazards model can yield a HR = 0.22 (p = 0.02) in 50 training examples and HR = 0.41 (p = 0.29) on 32 unseen test patients, ultimately utilizing just pathologically-determined T, and N information. The developed image-based risk predictor improved performance with HR = 0.51 (p = 0.06) on training data and HR = 0.52 (p = 0.09) on unseen test data. Combining the image-based risk models to selected clinicopathological features enhanced performance further to HR = 0.25 (p = 0.01) on the training set and HR = 0.37 (p = 0.07) on unseen test patients. Conclusions: Our interpretable image-based risk predictor shows high-risk pancreatic cancer patients have higher lymphocyte count overall but proportionally fewer tumor-infiltrating lymphocytes (TILs). In addition, this system shows high-risk patients have less stromal tissue within 100um from tumor compared to low risk patients. By aggregating both standard clinicopathological features with the proposed image-based risk assessment, superior separation in survival curves was achieved for both training and testing sets compared to either risk-model alone. Thus, our study demonstrates that image-based risk-associated features are independently prognostic of clinicopathological features. Despite the very limited sample-size of similarly-treated patients within the training dataset, these results trend towards significance and warrant further study within a larger cohort. Citation Format: Mustafa I. Jaber, Liudmila Beziaeva, Robert J. Torphy, Stephen C. Benz, Shahrooz Rabizadeh, Patrick Soon-Shiong, Christopher W Szeto. Deep-learning image-based tumor, stroma, and lymphocytes spatial relationships and clinicopathological features that affect survival in pancreatic cancer patients [abstract]. In: Proceedings of the AACR Virtual Special Conference on Pancreatic Cancer; 2020 Sep 29-30. Philadelphia (PA): AACR; Cancer Res 2020;80(22 Suppl):Abstract nr PO-030.

Spectral-Spatial Features Integrated Convolution Neural Network for Breast Cancer Classification.

Cancer identification and classification from histopathological images of the breast depends greatly on experts, and computer-aided diagnosis can play an important role in disagreement of experts. This automatic process has increased the accuracy of the classification at a reduced cost. The advancement in Convolution Neural Network (CNN) structure has outperformed the traditional approaches in biomedical imaging applications. One of the limiting factors of CNN is it uses spatial image features only for classification. The spectral features from the transform domain have equivalent importance in the complex image classification algorithm. This paper proposes a new CNN structure to classify the histopathological cancer images based on integrating the spectral features obtained using a multi-resolution wavelet transform with the spatial features of CNN. In addition, batch normalization process is used after every layer in the convolution network to improve the poor convergence problem of CNN and the deep layers of CNN are trained with spectral–spatial features. The proposed structure is tested on malignant histology images of the breast for both binary and multi-class classification of tissue using the BreaKHis Dataset and the Breast Cancer Classification Challenge 2015 Datasest. Experimental results show that the combination of spectral–spatial features improves classification accuracy of the CNN network and requires less training parameters in comparison with the well known models (i.e., VGG16 and ALEXNET). The proposed structure achieves an average accuracy of 97.58% and 97.45% with 7.6 million training parameters on both datasets, respectively.

Deep-Learning-Based Characterization of Tumor-Infiltrating Lymphocytes in Breast Cancers From Histopathology Images and Multiomics Data.

PURPOSETumor-infiltrating lymphocytes (TILs) and their spatial characterizations on whole-slide images (WSIs) of histopathology sections have become crucial in diagnosis, prognosis, and treatment response prediction for different cancers. However, fully automatic assessment of TILs on WSIs currently remains a great challenge because of the heterogeneity and large size of WSIs. We present an automatic pipeline based on a cascade-training U-net to generate high-resolution TIL maps on WSIs.METHODSWe present global cell-level TIL maps and 43 quantitative TIL spatial image features for 1,000 WSIs of The Cancer Genome Atlas patients with breast cancer. For more specific analysis, all the patients were divided into three subtypes, namely, estrogen receptor (ER)–positive, ER-negative, and triple-negative groups. The associations between TIL scores and gene expression and somatic mutation were examined separately in three breast cancer subtypes. Both univariate and multivariate survival analyses were performed on 43 TIL image features to examine the prognostic value of TIL spatial patterns in different breast cancer subtypes.RESULTSThe TIL score was in strong association with immune response pathway and genes (eg, programmed death-1 and CLTA4). Different breast cancer subtypes showed TIL score in association with mutations from different genes suggesting that different genetic alterations may lead to similar phenotypes. Spatial TIL features that represent density and distribution of TIL clusters were important indicators of the patient outcomes.CONCLUSIONOur pipeline can facilitate computational pathology-based discovery in cancer immunology and research on immunotherapy. Our analysis results are available for the research community to generate new hypotheses and insights on breast cancer immunology and development.