Convolutional Neural Networks For Segmentation Research Articles

Image copy-move forgery detection and localization based on super-BPD segmentation and DCNN

With the increasing importance of image information, image forgery seriously threatens the security of image content. Copy-move forgery detection (CMFD) is a greater challenge because its abnormality is smaller than other forgeries. To solve the problem that the detection results of the most image CMFD based on convolutional neural networks (CNN) have relatively low accuracy, an image copy-move forgery detection and localization based on super boundary-to-pixel direction (super-BPD) segmentation and deep CNN (DCNN) is proposed: SD-Net. Firstly, the segmentation technology is used to enhance the connection between the same or similar image blocks, improving the detection accuracy. Secondly, DCNN is used to extract image features, replacing conventional hand-crafted features with automatic learning features. The feature pyramid is used to improve the robustness to the scaling attack. Thirdly, the image BPD information is used to optimize the edges of rough detected image and obtain final detected image. The experiments proved that the SD-Net could detect and locate multiple, rotated, and scaling forgery well, especially large-level scaling forgery. Compared with other methods, the SD-Net is more accurately located and robust to various post-processing operations: brightness change, contrast adjustments, color reduction, image blurring, JPEG compression, and noise adding.

Volumetric Pancreas Segmentation on Computed Tomography: Accuracy and Efficiency of a Convolutional Neural Network Versus Manual Segmentation in 3D Slicer in the Context of Interreader Variability of Expert Radiologists.

This study aimed to compare accuracy and efficiency of a convolutional neural network (CNN)-enhanced workflow for pancreas segmentation versus radiologists in the context of interreader reliability. Volumetric pancreas segmentations on a data set of 294 portal venous computed tomographies were performed by 3 radiologists (R1, R2, and R3) and by a CNN. Convolutional neural network segmentations were reviewed and, if needed, corrected ("corrected CNN [c-CNN]" segmentations) by radiologists. Ground truth was obtained from radiologists' manual segmentations using simultaneous truth and performance level estimation algorithm. Interreader reliability and model's accuracy were evaluated with Dice-Sorenson coefficient (DSC) and Jaccard coefficient (JC). Equivalence was determined using a two 1-sided test. Convolutional neural network segmentations below the 25th percentile DSC were reviewed to evaluate segmentation errors. Time for manual segmentation and c-CNN was compared. Pancreas volumes from 3 sets of segmentations (manual, CNN, and c-CNN) were noninferior to simultaneous truth and performance level estimation-derived volumes [76.6 cm 3 (20.2 cm 3 ), P < 0.05]. Interreader reliability was high (mean [SD] DSC between R2-R1, 0.87 [0.04]; R3-R1, 0.90 [0.05]; R2-R3, 0.87 [0.04]). Convolutional neural network segmentations were highly accurate (DSC, 0.88 [0.05]; JC, 0.79 [0.07]) and required minimal-to-no corrections (c-CNN: DSC, 0.89 [0.04]; JC, 0.81 [0.06]; equivalence, P < 0.05). Undersegmentation (n = 47 [64%]) was common in the 73 CNN segmentations below 25th percentile DSC, but there were no major errors. Total inference time (minutes) for CNN was 1.2 (0.3). Average time (minutes) taken by radiologists for c-CNN (0.6 [0.97]) was substantially lower compared with manual segmentation (3.37 [1.47]; savings of 77.9%-87% [ P < 0.0001]). Convolutional neural network-enhanced workflow provides high accuracy and efficiency for volumetric pancreas segmentation on computed tomography.

Development of a Hybrid-Imaging-Based Prognostic Index for Metastasized-Melanoma Patients in Whole-Body 18F-FDG PET/CT and PET/MRI Data.

Besides tremendous treatment success in advanced melanoma patients, the rapid development of oncologic treatment options comes with increasingly high costs and can cause severe life-threatening side effects. For this purpose, predictive baseline biomarkers are becoming increasingly important for risk stratification and personalized treatment planning. Thus, the aim of this pilot study was the development of a prognostic tool for the risk stratification of the treatment response and mortality based on PET/MRI and PET/CT, including a convolutional neural network (CNN) for metastasized-melanoma patients before systemic-treatment initiation. The evaluation was based on 37 patients (19 f, 62 ± 13 y/o) with unresectable metastasized melanomas who underwent whole-body 18F-FDG PET/MRI and PET/CT scans on the same day before the initiation of therapy with checkpoint inhibitors and/or BRAF/MEK inhibitors. The overall survival (OS), therapy response, metastatically involved organs, number of lesions, total lesion glycolysis, total metabolic tumor volume (TMTV), peak standardized uptake value (SULpeak), diameter (Dmlesion) and mean apparent diffusion coefficient (ADCmean) were assessed. For each marker, a Kaplan–Meier analysis and the statistical significance (Wilcoxon test, paired t-test and Bonferroni correction) were assessed. Patients were divided into high- and low-risk groups depending on the OS and treatment response. The CNN segmentation and prediction utilized multimodality imaging data for a complementary in-depth risk analysis per patient. The following parameters correlated with longer OS: a TMTV < 50 mL; no metastases in the brain, bone, liver, spleen or pleura; ≤4 affected organ regions; no metastases; a Dmlesion > 37 mm or SULpeak < 1.3; a range of the ADCmean < 600 mm2/s. However, none of the parameters correlated significantly with the stratification of the patients into the high- or low-risk groups. For the CNN, the sensitivity, specificity, PPV and accuracy were 92%, 96%, 92% and 95%, respectively. Imaging biomarkers such as the metastatic involvement of specific organs, a high tumor burden, the presence of at least one large lesion or a high range of intermetastatic diffusivity were negative predictors for the OS, but the identification of high-risk patients was not feasible with the handcrafted parameters. In contrast, the proposed CNN supplied risk stratification with high specificity and sensitivity.

Identifying Urban Functional Regions from High-Resolution Satellite Images Using a Context-Aware Segmentation Network

The automatic identification of urban functional regions (UFRs) is crucial for urban planning and management. A key issue involved in URF classification is to properly determine the basic functional units, for which popular practices are usually based upon existing land use boundaries or road networks. Such practices suffer from the unavailability of existing datasets, leading to difficulty in large-scale mapping. To deal with this problem, this paper presents a method to automatically obtain functional units for URF classification using high-resolution remote sensing images. We develop a context-aware segmentation network to simultaneously extract buildings and road networks from remote sensing images. The extracted road networks are used for partitioning functional units, upon which five main building types are distinguished considering building height, morphology, and geometry. Finally, the UFRs are classified according to the distribution of building types. We conducted experiments using a GaoFen-2 satellite image with a spatial resolution of 0.8 m acquired in Fuzhou, China. Experimental results showed that the proposed segmentation network performed better than other convolutional neural network segmentation methods (i.e., PSPNet, Deeplabv3+, DANet, and JointNet), with an increase of F1-score up to 1.37% and 1.19% for road and building extraction, respectively. Results also showed that the residential regions, accounting for most of the urban areas, identified by the proposed method had a user accuracy of 94%, implying the promise of the proposed method for deriving the spatial units and the types of urban functional regions.

Convolutional neural networks for the automatic segmentation of lumbar paraspinal muscles in people with low back pain

The size, shape, and composition of paraspinal muscles have been widely reported in disorders of the cervical and lumbar spine. Measures of size, shape, and composition have required time-consuming and rater-dependent manual segmentation techniques. Convolutional neural networks (CNNs) provide alternate timesaving, state-of-the-art performance measures, which could realise clinical translation. Here we trained a CNN for the automatic segmentation of lumbar paraspinal muscles and determined the impact of CNN architecture and training choices on segmentation performance. T2-weighted MRI axial images from 76 participants (46 female; age (SD): 45.6 (12.8) years) with low back pain were used to train CNN models to segment the multifidus, erector spinae, and psoas major muscles (left and right segmented separately). Using cross-validation, we compared 2D and 3D CNNs with and without data augmentation. Segmentation accuracy was compared between the models using the Sørensen-Dice index as the primary outcome measure. The effect of increasing network depth on segmentation accuracy was also investigated. Each model showed high segmentation accuracy (Sørensen-Dice index ≥ 0.885) and excellent reliability (ICC2,1 ≥ 0.941). Overall, across all muscles, 2D models performed better than 3D models (p = 0.012), and training without data augmentation outperformed training with data augmentation (p < 0.001). The 2D model trained without data augmentation demonstrated the highest average segmentation accuracy. Increasing network depth did not improve accuracy (p = 0.771). All trained CNN models demonstrated high accuracy and excellent reliability for segmenting lumbar paraspinal muscles. CNNs can be used to efficiently and accurately extract measures of paraspinal muscle health from MRI.

Research on Segmentation of Brain Tumor in MRI Image Based on Convolutional Neural Network.

Brain tumors are the brain diseases with the highest mortality and prevalence, and magnetic resonance imaging has high-resolution and multiparameter. As the basis for realizing the quantitative analysis of brain tumors, automatic segmentation plays a vital role in diagnosis and treatment. A new network model is proposed to improve the accuracy of convolutional neural network segmentation of brain tumor regions and control the parameter space scale of the network model. The model first uses a convolutional layer composed of a series of 3D convolution filters to construct a backbone network for feature learning of input 3D MRI image blocks. Then, a pyramid structure constructed by a 3D convolutional layer is designed to extract and fuse features of tumor lesions and context information of different scales and then classify the fused feature at the voxel level to obtain segmentation results. Finally, a conditional random field is used to postprocess segmentation results for structured refinement. By designing massive ablation experiments to analyze the sensitivity of the essential modules of the comparison network, the results confirm that our method can better solve the problems faced by the traditional fully connected convolutional neural network.

Automatic detection of ischemic-stroke-lesion with CNN segmentation: a study

The vital organ in human physiology is the brain, and abnormality in the brain will reason for various behavioural problems. Ischemic-Stroke is a medical emergency, and early detection and action will help the patient recover quickly. This scheme aims to implement Convolutional-Neural-Network (CNN) segmentation method to extract and evaluate the infected portion from the MRI slice of the brain. In our study the pre-trained UNet scheme is adopted to extract the stroke region from the Flair modality MRI slice with axial-, coronal- and sagittal plane. In this work, the ISLES2015 database is used for the experimental investigation. The segmented portion is further evaluated to the ground-truth and the metrics such as Jaccard, Dice and Accuracy are computed. The experimental investigation is implemented using Python software. The experimental outcome of this research proves that the proposed CNN scheme aids to improve segmentation accuracy on axial-plane images compared with other images. The performance of the CNN segmentation scheme is then validated with other related results existing in the literature. The outcome of this study confirms that UNet supported technique helps extract the stroke lesion from the MRI slice with more accurate accuracy.

Automatic Inspection and Evaluation System for Pavement Distress

Pavement distress detection is of significance for road maintenance and traffic safety. Manual pavement distress detection suffers from high workloads, inefficiency, low accuracy, and high cost. To replace the manual operations in the pre-filling detection with the aim to improve efficiency and reduce cost, this paper proposes a three-stage automatic inspection and evaluation system for pavement distress based on improved deep convolutional neural networks (CNNs). First, the system integrates multi-level context information from the CNN classification model to construct discriminative super-features to determine whether there is distress in the pavement image and the type of the distress, so as to achieve rapid detection of pavement distress. Then, the pavement images with distress are fed into the CNN segmentation model to highlight the distress region with pixel-wise. In the segmentation model, a novel pyramid feature extraction module and a novel guidance attention mechanism are introduced. Finally, we evaluate the degree of pavement damage according to the segmentation results of the CNN segmentation model. In the experiments, we compare our classification model and segmentation model with other state-of-the-art methods on two pavement distress datasets, and the results demonstrate that the proposed models achieve out-performance on different evaluation metrics.

GlacierNet2: A hybrid Multi-Model learning architecture for alpine glacier mapping

In recent decades, climate change has significantly affected glacier dynamics, resulting in mass loss and an increased risk of glacier-related hazards including supraglacial and proglacial lake development, as well as catastrophic outburst flooding. Rapidly changing conditions dictate the need for continuous and detailed observations and analysis of climate-glacier dynamics. Thematic and quantitative information regarding glacier geometry is fundamental for understanding climate forcing and the sensitivity of glaciers to climate change, however, accurately mapping debris-cover glaciers (DCGs) is notoriously difficult based upon the use of spectral information and conventional machine-learning techniques. The objective of this research is to improve upon an earlier proposed deep-learning-based approach, GlacierNet, which was developed to exploit a convolutional neural-network segmentation model to accurately outline regional DCG ablation zones. Specifically, we developed an enhanced GlacierNet2 architecture thatincorporates multiple models, automatic post-processing, and basin-level hydrological flow techniques to improve the mapping of DCGs such that it includes both the ablation and accumulation zones. Experimental evaluations demonstrate that GlacierNet2 improves the estimation of the ablation zone and allows a high level of intersection over union (IOU: 0.8839) score. The proposed architecture provides complete glacier (both accumulation and ablation zone) outlines at regional scales, with an overall IOU score of 0.8619. This is a crucial first step in automating complete glacier mapping that can be used for accurate glacier modeling or mass-balance analysis.

3D deep convolutional neural network segmentation model for precipitate and porosity identification in synchrotron X-ray tomograms.

New developments at synchrotron beamlines and the ongoing upgrades of synchrotron facilities allow the possibility to study complex structures with a much better spatial and temporal resolution than ever before. However, the downside is that the data collected are also significantly larger (more than several terabytes) than ever before, and post-processing and analyzing these data is very challenging to perform manually. This issue can be solved by employing automated methods such as machine learning, which show significantly improved performance in data processing and image segmentation than manual methods. In this work, a 3D U-net deep convolutional neural network (DCNN) model with four layers and base-8 characteristic features has been developed to segment precipitates and porosities in synchrotron transmission X-ray micrograms. Transmission X-ray microscopy experiments were conducted on micropillars prepared from additively manufactured 316L steel to evaluate precipitate information. After training the 3D U-net DCNN model, it was used on unseen data and the prediction was compared with manual segmentation. A good agreement was found between both segmentations. An ablation study was performed and revealed that the proposed model showed better statistics than other models with lower numbers of layers and/or characteristic features. The proposed model is able to segment several hundreds of gigabytes of data in a few minutes and could be applied to other materials and tomography techniques. The code and the fitted weights are made available with this paper for any interested researcher to use for their needs (https://github.com/manasvupadhyay/erc-gamma-3D-DCNN).

Comparison of Human Intestinal Parasite Ova Segmentation Using Machine Learning and Deep Learning Techniques

Helminthiasis disease is one of the most serious health problems in the world and frequently occurs in children, especially in unhygienic conditions. The manual diagnosis method is time consuming and challenging, especially when there are a large number of samples. An automated system is acknowledged as a quick and easy technique to assess helminth sample images by offering direct visibility on the computer monitor without the requirement for examination under a microscope. Thus, this paper aims to compare the human intestinal parasite ova segmentation performance between machine learning segmentation and deep learning segmentation. Four types of helminth ova are tested, which are Ascaris Lumbricoides Ova (ALO), Enterobious Vermicularis Ova (EVO), Hookworm Ova (HWO), and Trichuris Trichiura Ova (TTO). In this paper, fuzzy c-Mean (FCM) segmentation technique is used in machine learning segmentation, while convolutional neural network (CNN) segmentation technique is used for deep learning. The performance of segmentation algorithms based on FCM and CNN segmentation techniques is investigated and compared to select the best segmentation procedure for helminth ova detection. The results reveal that the accuracy obtained for each helminth species is in the range of 97% to 100% for both techniques. However, IoU analysis showed that CNN based on ResNet technique performed better than FCM for ALO, EVO, and TTO with values of 75.80%, 55.48%, and 77.06%, respectively. Therefore, segmentation through deep learning is more suitable for segmenting the human intestinal parasite ova.

Recognition of fatigue and severe sliding wear particles using a CNN model with multi-scale feature extractor

PurposeThe purpose of this study is to provide a new convolutional neural network (CNN) model with multi-scale feature extractor to segment and recognize wear particles in complex ferrograph images, especially fatigue and severe sliding wear particles, which are similar in morphology while different in wear mechanism.Design/methodology/approachA CNN model named DWear is proposed to semantically segment fatigue, severe sliding particles and four other types of particles, that is, chain, spherical, cutting and oxide particles, which unifies segmentation and recognition together. DWear is constructed using four modules, namely, encoder, densely connected atrous spatial pyramid pooling, decoder and fully connected conditional random field. Different from the architectures of ordinary semantic segmentation CNN models, a multi-scale feature extractor using cascade connections and a coprime atrous rate group is incorporated into the DWear model to obtain multi-scale receptive fields and better extract features of wear particles. Moreover, fully connected conditional random field module is adopted for post-processing to smooth coarse prediction and obtain finer results.FindingsDWear is trained and verified on the ferrograph image data set, and experimental results show that the final Mean Pixel Accuracy is 95.6% and the Mean Intersection over Union is 92.2%, which means that the recognition and segmentation accuracy is higher than those of previous works.Originality/valueDWear provides a promising approach for wear particle analysis and can be further developed in equipment condition monitoring applications.

Convolutional Neural Networks for the Evaluation of Chronic and Inflammatory Lesions in Kidney Transplant Biopsies

In kidney transplant biopsies, both inflammation and chronic changes are important features that predict long-term graft survival. Quantitative scoring of these features is important for transplant diagnostics and kidney research. However, visual scoring is poorly reproducible and labor intensive. The goal of this study was to investigate the potential of convolutional neural networks (CNNs) to quantify inflammation and chronic features in kidney transplant biopsies. A structure segmentation CNN and a lymphocyte detection CNN were applied on 125 whole-slide image pairs of periodic acid-Schiff- and CD3-stained slides. The CNN results were used to quantify healthy and sclerotic glomeruli, interstitial fibrosis, tubular atrophy, and inflammation within both nonatrophic and atrophic tubuli, and in areas of interstitial fibrosis. The computed tissue features showed high correlation with Banff lesion scores of five pathologists (A.A., A.Dend., J.H.B., J.K., and T.N.). Analyses on a small subset showed a moderate correlation toward higher CD3+ cell density within scarred regions and higher CD3+ cell count inside atrophic tubuli correlated with long-term change of estimated glomerular filtration rate. The presented CNNs are valid tools to yield objective quantitative information on glomeruli number, fibrotic tissue, and inflammation within scarred and non-scarred kidney parenchyma in a reproducible manner. CNNs have the potential to improve kidney transplant diagnostics and will benefit the community as a novel method to generate surrogate end points for large-scale clinical studies.

Comparing CNNs and Random Forests for Landsat Image Segmentation Trained on a Large Proxy Land Cover Dataset

Land cover mapping from satellite images has progressed from visual and statistical approaches to Random Forests (RFs) and, more recently, advanced image recognition techniques such as convolutional neural networks (CNNs). CNNs have a conceptual benefit over RFs in recognising spatial feature context, but potentially at the cost of reduced spatial detail. We tested the use of CNNs for improved land cover mapping based on Landsat data, compared with RFs, for a study area of approximately 500 km × 500 km in southeastern Australia. Landsat 8 geomedian composite surface reflectances were available for 2018. Label data were a simple nine-member land cover classification derived from reference land use mapping (Catchment Scale Land Use of Australia—CLUM), and further enhanced by using custom forest extent mapping (Forests of Australia). Experiments were undertaken testing U-Net CNN for segmentation of Landsat 8 geomedian imagery to determine the optimal combination of input Landsat 8 bands. The results were compared with those from a simple autoencoder as well as an RF model. Segmentation test results for the best performing U-Net CNN models produced an overall accuracy of 79% and weighted-mean F1 score of 77% (9 band input) or 76% (6 band input) for a simple nine-member land cover classification, compared with 73% and 68% (6 band input), respectively, for the best RF model. We conclude that U-Net CNN models can generate annual land cover maps with good accuracy from proxy training data, and can also be used for quality control or improvement of existing land cover products.

3D Flow Reconstruction Using Conic Sections And Learning-Based Image Segmentation

Extending 3D Particle Streak Velocimetry (3D-PSV) with conic section matching constraints for volumetric flow reconstruction allows the measurement of flows in large volumes using lower frame rates than typically required for 3D-PTV, with fewer reconstruction ambiguities. Streak segmentation is tackled using a state-of-the-art instance segmentation Convolutional Neural Network (CNN), highlighting the potential of learning-based methods to solve vision-based tasks in experimental fluid dynamics. The curved streaks are processed with an optimization-based conic section reconstruction approach, which results in fewer reconstruction ambiguities compared to linear streaks or individual endpoint models and enables the direct and accurate reconstruction of the curved pathlines.

RBorderNet: Rider Border Collie Optimization-based Deep Convolutional Neural Network for road scene segmentation and road intersection classification

Intersection is an important component in urban traffic networks, as it connects pedestrian flows and vehicles between the network links. Intersection classification plays an active role in mitigating traffic congestion, increasing the level of road safety and traffic efficiency. Various intersection classification methods are adopted to classify the intersection point in the road segment, but detecting the accurate location results from a complex task in an automatic driving system. Hence, this paper proposes a Rider Border Collie Optimization-based Deep Convolutional Neural Network (RBorderNet) for road scene segmentation intersection classification. The RBCO model is modeled by combining the Rider Optimization Algorithm (ROA) with Border Collie Optimization (BCO). Here, Fresnel Transform (FrT) is employed to detect the keyframes from the video based on the angular distance. DCNN classifier is used for classifying the intersection of road segments by considering the optimal region extracted from the segmented frames. The training of the DCNN classifier is accomplished by the RBCO algorithm. The developed RBCO-based DCNN achieved higher performance with the metrics, like accuracy, training error, precision, and recall is 94.51%, 5.49%, 96.43%, and 95.29%, respectively, by considering vehicles and traffic scenarios.

Recognition and Segmentation of Individual Bone Fragments with a Deep Learning Approach in CT Scans of Complex Intertrochanteric Fractures: A Retrospective Study.

The characteristics of bone fragments are the main influencing factors for the choice of treatment in intertrochanteric fractures. This study aimed to develop a deep learning algorithm for recognizing and segmenting individual fragments in CT images of complex intertrochanteric fractures for orthopedic surgeons. This study was based on 160 hip CT scans (43,510 images) of complex fractures of three types based on the Evans-Jensen classification (40 cases of type 3 (IIA) fractures, 80 cases of type 4 (IIB)fractures, and 40 cases of type 5 (III)fractures) retrospectively. The images were randomly split into two groups to construct a training set of 120 CT scans (32,045 images) and a testing set of 40 CT scans (11,465 images). A deep learning model was built into a cascaded architecture composed by a convolutional neural network (CNN) for location of the fracture ROI and another CNN for recognition and segmentation of individual fragments within the ROI. The accuracy of object detection and dice coefficient of segmentation of individual fragments were used to evaluate model performance. The model yielded an average accuracy of 89.4% for individual fragment recognition and an average dice coefficient of 90.5% for segmentation in CT images. The results demonstrated the feasibility of recognition and segmentation of individual fragments in complex intertrochanteric fractures with a deep learning approach. Altogether, these promising results suggest the potential of our model to be applied to many clinical scenarios.

Abstract 469: A multi-scale analysis and visualization platform for cancer data - deriving tumor microenvironment behavior from pathology and transcriptomics

Abstract Heterogeneity is a hallmark of cancer and perhaps one of the most important features associated with resistance to therapies and likelihood of recurrence and/or metastasis. Genomic instability contributes to genetic diversity, which leads to high levels of intratumoural heterogeneity. While genetic diversity is one contributor to phenotypic heterogeneity in tumors, the spatial variation in the tumor microenvironment (TME) also drives emergent phenotypic heterogeneity and intrinsic variation in resistance to drugs and the immune system. Ultimately, heterogeneity leads to higher likelihood of metastasis. Traditionally, pathology allowed for the assessment of heterogeneity. However, this was limited to gross assessment of the TME, through measurement of mitotic levels and severity of cancer invasion. We present SimBioSys PhenoScope, a multi-scale analysis and visualization platform that integrates cancer data across scales to extract cross modality trends that drive cancer invasion. As a demonstrated use case for the platform, we present a vignette of using the platform to analyze pathology slides at three scales. Three convolutional neural networks (CNN) are developed and validated. The outputs of these networks were combined with 2D simulations of the metabolic behavior and growth of cells within the TME. Two CNNs were developed and one implemented: one that identifies cells undergoing mitosis, one that segments individual cells and classifies their type, and one that segments five tissues from pathology slides. Additionally, transcriptional data was used to generate patient specific metabolic models. Each CNN was developed using training images and validated on the test images. The mitosis detection CNN was found to have an accuracy of 76.2% (precision=83%, recall=76%) in the test set. The classification CNN was found to have a Dice Similarity Coefficient (DSC) of 0.821 for segmenting cells and an F1 score for classifying cells ranging from 0.559 (F1i) to 0.756 (F1d). The segmentation CNN was found to have an accuracy of 78.1% with DSC ranging from 0.66 and 0.86 depending on tissue. The segmentations were input into a proprietary simulation framework along with patient-specific metabolic models to predict the spatial gradients of nutrients, and spatial organization of growth and metabolism. Simulations show multiple behaviors such as regions of high lactate production or consumption by cancer, and regions that differ by lactate production and alanine uptake in cancers. These behaviors were correlated with local cell mitoses and invasion of Tumor Infiltrating Lymphocytes. Tools to examine cancers across scales and within the TME are currently lacking. We demonstrate a proof-of-principle approach of combining data across scales in a fashion that allows for novel predictions of TME behavior. Citation Format: Joseph R. Peterson, Matthew Shin, Patricia Carrigan, Michael J. Hallock, Snehal Patel, The SimBioSys Team. A multi-scale analysis and visualization platform for cancer data - deriving tumor microenvironment behavior from pathology and transcriptomics [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 469.

O-Net: A Novel Framework With Deep Fusion of CNN and Transformer for Simultaneous Segmentation and Classification.

The application of deep learning in the medical field has continuously made huge breakthroughs in recent years. Based on convolutional neural network (CNN), the U-Net framework has become the benchmark of the medical image segmentation task. However, this framework cannot fully learn global information and remote semantic information. The transformer structure has been demonstrated to capture global information relatively better than the U-Net, but the ability to learn local information is not as good as CNN. Therefore, we propose a novel network referred to as the O-Net, which combines the advantages of CNN and transformer to fully use both the global and the local information for improving medical image segmentation and classification. In the encoder part of our proposed O-Net framework, we combine the CNN and the Swin Transformer to acquire both global and local contextual features. In the decoder part, the results of the Swin Transformer and the CNN blocks are fused to get the final results. We have evaluated the proposed network on the synapse multi-organ CT dataset and the ISIC 2017 challenge dataset for the segmentation task. The classification network is simultaneously trained by using the encoder weights of the segmentation network. The experimental results show that our proposed O-Net achieves superior segmentation performance than state-of-the-art approaches, and the segmentation results are beneficial for improving the accuracy of the classification task. The codes and models of this study are available at https://github.com/ortonwang/O-Net.

Videomics of the Upper Aero-Digestive Tract Cancer: Deep Learning Applied to White Light and Narrow Band Imaging for Automatic Segmentation of Endoscopic Images.

IntroductionNarrow Band Imaging (NBI) is an endoscopic visualization technique useful for upper aero-digestive tract (UADT) cancer detection and margins evaluation. However, NBI analysis is strongly operator-dependent and requires high expertise, thus limiting its wider implementation. Recently, artificial intelligence (AI) has demonstrated potential for applications in UADT videoendoscopy. Among AI methods, deep learning algorithms, and especially convolutional neural networks (CNNs), are particularly suitable for delineating cancers on videoendoscopy. This study is aimed to develop a CNN for automatic semantic segmentation of UADT cancer on endoscopic images.Materials and MethodsA dataset of white light and NBI videoframes of laryngeal squamous cell carcinoma (LSCC) was collected and manually annotated. A novel DL segmentation model (SegMENT) was designed. SegMENT relies on DeepLabV3+ CNN architecture, modified using Xception as a backbone and incorporating ensemble features from other CNNs. The performance of SegMENT was compared to state-of-the-art CNNs (UNet, ResUNet, and DeepLabv3). SegMENT was then validated on two external datasets of NBI images of oropharyngeal (OPSCC) and oral cavity SCC (OSCC) obtained from a previously published study. The impact of in-domain transfer learning through an ensemble technique was evaluated on the external datasets.Results219 LSCC patients were retrospectively included in the study. A total of 683 videoframes composed the LSCC dataset, while the external validation cohorts of OPSCC and OCSCC contained 116 and 102 images. On the LSCC dataset, SegMENT outperformed the other DL models, obtaining the following median values: 0.68 intersection over union (IoU), 0.81 dice similarity coefficient (DSC), 0.95 recall, 0.78 precision, 0.97 accuracy. For the OCSCC and OPSCC datasets, results were superior compared to previously published data: the median performance metrics were, respectively, improved as follows: DSC=10.3% and 11.9%, recall=15.0% and 5.1%, precision=17.0% and 14.7%, accuracy=4.1% and 10.3%.ConclusionSegMENT achieved promising performances, showing that automatic tumor segmentation in endoscopic images is feasible even within the highly heterogeneous and complex UADT environment. SegMENT outperformed the previously published results on the external validation cohorts. The model demonstrated potential for improved detection of early tumors, more precise biopsies, and better selection of resection margins.