Patch-based Convolutional Neural Network Research Articles

Patch individual filter layers in CNNs to harness the spatial homogeneity of neuroimaging data

Convolutional neural networks (CNNs)—as a type of deep learning—have been specifically designed for highly heterogeneous data, such as natural images. Neuroimaging data, however, is comparably homogeneous due to (1) the uniform structure of the brain and (2) additional efforts to spatially normalize the data to a standard template using linear and non-linear transformations. To harness spatial homogeneity of neuroimaging data, we suggest here a new CNN architecture that combines the idea of hierarchical abstraction in CNNs with a prior on the spatial homogeneity of neuroimaging data. Whereas early layers are trained globally using standard convolutional layers, we introduce patch individual filters (PIF) for higher, more abstract layers. By learning filters in individual latent space patches without sharing weights, PIF layers can learn abstract features faster and specific to regions. We thoroughly evaluated PIF layers for three different tasks and data sets, namely sex classification on UK Biobank data, Alzheimer’s disease detection on ADNI data and multiple sclerosis detection on private hospital data, and compared it with two baseline models, a standard CNN and a patch-based CNN. We obtained two main results: First, CNNs using PIF layers converge consistently faster, measured in run time in seconds and number of iterations than both baseline models. Second, both the standard CNN and the PIF model outperformed the patch-based CNN in terms of balanced accuracy and receiver operating characteristic area under the curve (ROC AUC) with a maximal balanced accuracy (ROC AUC) of 94.21% (99.10%) for the sex classification task (PIF model), and 81.24% and 80.48% (88.89% and 87.35%) respectively for the Alzheimer’s disease and multiple sclerosis detection tasks (standard CNN model). In conclusion, we demonstrated that CNNs using PIF layers result in faster convergence while obtaining the same predictive performance as a standard CNN. To the best of our knowledge, this is the first study that introduces a prior in form of an inductive bias to harness spatial homogeneity of neuroimaging data.

Machine-assisted interpolation algorithm for semi-automated segmentation of highly deformable organs.

Accurate and robust auto-segmentation of highly deformable organs (HDOs), for example, stomach or bowel, remains an outstanding problem due to these organs' frequent and large anatomical variations. Yet, time-consuming manual segmentation of these organs presents a particular challenge to time-limited modern radiotherapy techniques such as on-line adaptive radiotherapy and high-dose-rate brachytherapy. We propose a machine-assisted interpolation (MAI) that uses prior information in the form of sparse manual delineations to facilitate rapid, accurate segmentation of the stomach from low field magnetic resonance images (MRI) and the bowel from computed tomography (CT) images. Stomach MR images from 116 patients undergoing 0.35T MRI-guided abdominal radiotherapy and bowel CT images from 120 patients undergoing high dose rate pelvic brachytherapy treatment were collected. For each patient volume, the manual delineation of the HDO was extracted from every 8th slice. These manually drawn contours were first interpolated to obtain an initial estimate of the HDO contour. A two-channel 64×64pixel patch-based convolutional neural network (CNN) was trained to localize the position of the organ's boundary on each slice within a five-pixel wide road using the image and interpolated contour estimate. This boundary prediction was then input, in conjunction with the image, to an organ closing CNN which output the final organ segmentation. A Dense-UNet architecture was used for both networks. The MAI algorithm was separately trained for the stomach segmentation and the bowel segmentation. Algorithm performance was compared against linear interpolation (LI) alone and against fully automated segmentation (FAS) using a Dense-UNet trained on the same datasets. The Dice Similarity Coefficient (DSC) and mean surface distance (MSD) metrics were used to compare the predictions from the three methods. Statistically significance was tested using Student's t test. For the stomach segmentation, the mean DSC from MAI (0.91 ± 0.02) was 5.0% and 10.0% higher as compared to LI and FAS, respectively. The average MSD from MAI (0.77 ± 0.25mm) was 0.54 and 3.19mm lower compared to the two other methods. Only 7% of MAI stomach predictions resulted in a DSC<0.8, as compared to 30% and 28% for LI and FAS, respectively. For the bowel segmentation, the mean DSC of MAI (0.90 ± 0.04) was 6% and 18% higher, and the average MSD of MAI (0.93 ± 0.48mm) was 0.42 and 4.9mm lower as compared to LI and FAS. Sixteen percent of the predicted contour from MAI resulted in a DSC<0.8, as compared to 46% and 60% for FAS and LI, respectively. All comparisons between MAI and the baseline methods were found to be statistically significant (p-value<0.001). The proposed MAI algorithm significantly outperformed LI in terms of accuracy and robustness for both stomach segmentation from low-field MRIs and bowel segmentation from CT images. At this time, FAS methods for HDOs still require significant manual editing. Therefore, we believe that the MAI algorithm has the potential to expedite the process of HDO delineation within the radiation therapy workflow.

Building damage assessment for rapid disaster response with a deep object-based semantic change detection framework: From natural disasters to man-made disasters

Sudden-onset natural and man-made disasters represent a threat to the safety of human life and property. Rapid and accurate building damage assessment using bitemporal high spatial resolution (HSR) remote sensing images can quickly and safely provide us with spatial distribution information and statistics of the damage degree to assist with humanitarian assistance and disaster response. For building damage assessment, strong feature representation and semantic consistency are the keys to obtaining a high accuracy. However, the conventional object-based image analysis (OBIA) framework using a patch-based convolutional neural network (CNN) can guarantee semantic consistency, but with weak feature representation, while the Siamese fully convolutional network approach has strong feature representation capabilities but is semantically inconsistent. In this paper, we propose a deep object-based semantic change detection framework, called ChangeOS, for building damage assessment. To seamlessly integrate OBIA and deep learning, we adopt a deep object localization network to generate accurate building objects, in place of the superpixel segmentation commonly used in the conventional OBIA framework. Furthermore, the deep object localization network and deep damage classification network are integrated into a unified semantic change detection network for end-to-end building damage assessment. This also provides deep object features that can supply an object prior to the deep damage classification network for more consistent semantic feature representation. Object-based post-processing is adopted to further guarantee the semantic consistency of each object. The experimental results obtained on a global scale dataset including 19 natural disaster events and two local scale datasets including the Beirut port explosion event and the Bata military barracks explosion event show that ChangeOS is superior to the currently published methods in speed and accuracy, and has a superior generalization ability for man-made disasters.

CocoNet: a hybrid machine learning framework for coconut farm identification and its cyclonic damage assessment on bitemporal SAR images

The coconut tree is an important cash crop in coastal tropical regions. These trees undergo substantial damage during annual cyclones. Early and effective assessment of damage is important to farmers for appropriate monetary compensation. However, the identification of coconut trees presents multiple challenges: (i) the trees are with other vegetation and (ii) cloud cover during monsoons and cyclones precludes using simple imaging techniques. Although automated approaches based on classical machine learning techniques have been attempted for other crops on remote sensed images, these are not adequate for identifying coconut trees from different land cover. We present a hybrid approach termed CocoNet that combines (i) unsupervised K-means for pixel-based coarse classification of land cover and (ii) a patch-based convolutional neural network for fine classification of vegetation to identify coconut farms. The challenge presented by cloud cover is addressed using synthetic aperture radar (SAR) images. Change detection is then performed on bitemporal Sentinel-1 SAR images taken before and after a cyclone to obtain the change map depicting the damage caused. Experimental results show that the proposed two-phase CocoNet model gives a classification accuracy of 92.5% and a change detection accuracy of 89.1%.

Ensemble of multi-stage deep convolutional neural networks for automated grading of diabetic retinopathy using image patches

Diabetic retinopathy (DR) is one of the most common retinal diseases that cause preventable blindness in diabetic patients. The timely screening and grading of retinal images minimize the possibility of vision loss. However, manual screening of retinal images, for detecting micro lesions in the early stages of DR, is time-consuming. This paper proposes an ensemble of deep convolutional neural network (CNN) models for accurate detection and grading of DR using fundus images. Each input image is divided into four patches at the first stage and passed on to pre-trained CNN models (InceptionV3, Xception) for training. The relevant features in the shallow-dense layers of CNN models are utilized as prior knowledge. The integration of shallow and dense layer features helps the model learn the significant information of DR images. At the second stage, a classifier based on artificial neural network is trained using the fused probability vectors of four patches. The results of individual CNN models are combined to generate the final decision in the third stage. This ensemble approach of multi-stage deep learning model improves the overall classification accuracy of diabetic retinopathy grading. Among the five different classification schemes presented in this paper, multistage patch-based deep CNN (MPDCNN), in which local patch-based and holistic details of fundus image are concatenated, provides the best classification accuracy. This ensemble classifier exhibits 96.2 % classification accuracy with fivefold cross-validation.

Fast and accurate land-cover classification on medium-resolution remote-sensing images using segmentation models

ABSTRACT Land-cover classification especially global mapping has become a new trend in recent years. Traditional convolutional neural network (CNN) methods for land-cover classification are usually patch based and have problems of high computation cost and low efficiency, which hinder their wide applications when timely and accurately mapping land covers. Fortunately, methods based on the fully convolutional network (FCN) have achieved state-of-the-art performance on the semantic segmentation task, which provides a new possibility for efficient land-cover classification. Many works have been done for land-cover classification but are almost focused on very high-resolution remote-sensing images and few research works are implemented on medium-resolution images. In this paper, six representative state-of-the-art segmentation models including ‘U’-shaped network (U-Net), fully convolutional DenseNet (FC-DenseNet), full-resolution residual network (FRRN), bilateral segmentation network (BiSeNet), DeepLab version 3 plus (DeepLabV3+), and pyramid scene parsing network (PSPNet) are selected to compare their performances on the land-cover classification of Land Remote-Sensing Satellite System)-5 satellite remote-sensing images. Based on the analysis of their performances, an improved model named atrous spatial pyramid pooling U-Net (ASPP-U-Net) is proposed for classification. Methods including support vector machine, patch-based CNN, and U-Net are also selected for comparison with the proposed model. Furthermore, to overcome the insufficiency of reference data when training deep models, an integration strategy based on two existing global land-cover products finer resolution observation and monitoring of global land cover of 2010 and global land-cover mapping at 30 m resolution is designed to produce reference data. Experimental results show that the encoder–decoder architecture especially U-Net is the most competitive network and is highly recommended for mapping land covers of medium-resolution images. The proposed ASPP-U-Net outperforms other compared methods not only in the classification accuracy but also in the inference time efficiency. In addition, it is advisable to use existing global land-cover products to produce reference data for segmentation models when the labelled datasets are insufficient.

River state classification combining patch-based processing and CNN.

This paper proposes a method for classifying the river state (a flood risk exists or not) from river surveillance camera images by combining patch-based processing and a convolutional neural network (CNN). Although CNN needs much training data, the number of river surveillance camera images is limited because flood does not frequently occur. Also, river surveillance camera images include objects that are irrelevant to the flood risk. Therefore, the direct use of CNN may not work well for the river state classification. To overcome this limitation, this paper develops patch-based processing for adjusting CNN to the river state classification. By increasing training data via the patch segmentation of an image and selecting patches that are relevant to the river state, the adjustment of general CNNs to the river state classification becomes feasible. The proposed patch-based processing and CNN are developed independently. This yields the practical merits that any CNN can be used according to each user’s purposes, and the maintenance and improvement of each component of the whole system can be easily performed. In the experiment, river state classification is defined as the following problems using two datasets, to verify the effectiveness of the proposed method. First, river images from the public dataset called Places are classified to images with Muddy labels and images with Clear labels. Second, images from the river surveillance camera in Nagaoka City, Japan are classified to images captured when the government announced heavy rain or flood warning and the other images.

GAMER MRI: Gated-attention mechanism ranking of multi-contrast MRI in brain pathology

IntroductionDuring the last decade, a multitude of novel quantitative and semiquantitative MRI techniques have provided new information about the pathophysiology of neurological diseases. Yet, selection of the most relevant contrasts for a given pathology remains challenging. In this work, we developed and validated a method, Gated-Attention MEchanism Ranking of multi-contrast MRI in brain pathology (GAMER MRI), to rank the relative importance of MR measures in the classification of well understood ischemic stroke lesions. Subsequently, we applied this method to the classification of multiple sclerosis (MS) lesions, where the relative importance of MR measures is less understood. MethodsGAMER MRI was developed based on the gated attention mechanism, which computes attention weights (AWs) as proxies of importance of hidden features in the classification. In the first two experiments, we used Trace-weighted (Trace), apparent diffusion coefficient (ADC), Fluid-Attenuated Inversion Recovery (FLAIR), and T1-weighted (T1w) images acquired in 904 acute/subacute ischemic stroke patients and in 6,230 healthy controls and patients with other brain pathologies to assess if GAMER MRI could produce clinically meaningful importance orders in two different classification scenarios. In the first experiment, GAMER MRI with a pretrained convolutional neural network (CNN) was used in conjunction with Trace, ADC, and FLAIR to distinguish patients with ischemic stroke from those with other pathologies and healthy controls. In the second experiment, GAMER MRI with a patch-based CNN used Trace, ADC and T1w to differentiate acute ischemic stroke lesions from healthy tissue. The last experiment explored the performance of patch-based CNN with GAMER MRI in ranking the importance of quantitative MRI measures to distinguish two groups of lesions with different pathological characteristics and unknown quantitative MR features. Specifically, GAMER MRI was applied to assess the relative importance of the myelin water fraction (MWF), quantitative susceptibility mapping (QSM), T1 relaxometry map (qT1), and neurite density index (NDI) in distinguishing 750 juxtacortical lesions from 242 periventricular lesions in 47 MS patients. Pair-wise permutation t-tests were used to evaluate the differences between the AWs obtained for each quantitative measure. ResultsIn the first experiment, we achieved a mean test AUC of 0.881 and the obtained AWs of FLAIR and the sum of AWs of Trace and ADC were 0.11 and 0.89, respectively, as expected based on previous knowledge. In the second experiment, we achieved a mean test F1 score of 0.895 and a mean AW of Trace = 0.49, of ADC = 0.28, and of T1w = 0.23, thereby confirming the findings of the first experiment. In the third experiment, MS lesion classification achieved test balanced accuracy = 0.777, sensitivity = 0.739, and specificity = 0.814. The mean AWs of T1map, MWF, NDI, and QSM were 0.29, 0.26, 0.24, and 0.22 (p < 0.001), respectively. ConclusionsThis work demonstrates that the proposed GAMER MRI might be a useful method to assess the relative importance of MRI measures in neurological diseases with focal pathology. Moreover, the obtained AWs may in fact help to choose the best combination of MR contrasts for a specific classification problem.

A deep learning framework for pancreas segmentation with multi-atlas registration and 3D level-set.

In this paper, we propose and validate a deep learning framework that incorporates both multi-atlas registration and level-set for segmenting pancreas from CT volume images. The proposed segmentation pipeline consists of three stages, namely coarse, fine, and refine stages. Firstly, a coarse segmentation is obtained through multi-atlas based 3D diffeomorphic registration and fusion. After that, to learn the connection feature, a 3D patch-based convolutional neural network (CNN) and three 2D slice-based CNNs are jointly used to predict a fine segmentation based on a bounding box determined from the coarse segmentation. Finally, a 3D level-set method is used, with the fine segmentation being one of its constraints, to integrate information of the original image and the CNN-derived probability map to achieve a refine segmentation. In other words, we jointly utilize global 3D location information (registration), contextual information (patch-based 3D CNN), shape information (slice-based 2.5D CNN) and edge information (3D level-set) in the proposed framework. These components form our cascaded coarse-fine-refine segmentation framework. We test the proposed framework on three different datasets with varying intensity ranges obtained from different resources, respectively containing 36, 82 and 281 CT volume images. In each dataset, we achieve an average Dice score over 82%, being superior or comparable to other existing state-of-the-art pancreas segmentation algorithms.

Exploring multiscale object-based convolutional neural network (multi-OCNN) for remote sensing image classification at high spatial resolution

Convolutional Neural Network (CNN) has been increasingly used for land cover mapping of remotely sensed imagery. However, large-area classification using traditional CNN is computationally expensive and produces coarse maps using a sliding window approach. To address this problem, object-based CNN (OCNN) becomes an alternative solution to improve classification performance. However, previous studies were mainly focused on urban areas or small scenes, and implementation of OCNN method is still needed for large-area classification over heterogeneous landscape. Additionally, the massive labeling of segmented objects requires a practical approach for less computation, including object analysis and multiple CNNs. This study presents a new multiscale OCNN (multi-OCNN) framework for large-scale land cover classification at 1-m resolution over 145,740 km2. Our approach consists of three main steps: (i) image segmentation, (ii) object analysis with skeleton-based algorithm, and (iii) application of multiple CNNs for final classification. Also, we developed a large benchmark dataset, called IowaNet, with 1 million labeled images and 10 classes. In our approach, multiscale CNNs were trained to capture the best contextual information during the semantic labeling of objects. Meanwhile, skeletonization algorithm provided morphological representation (“medial axis”) of objects to support the selection of convolutional locations for CNN predictions. In general, proposed multi-OCNN presented better classification accuracy (overall accuracy ~87.2%) compared to traditional patch-based CNN (81.6%) and fixed-input OCNN (82%). In addition, the results showed that this framework is 8.1 and 111.5 times faster than traditional pixel-wise CNN16 or CNN256, respectively. Multiple CNNs and object analysis have proved to be essential for accurate and fast classification. While multi-OCNN produced a high-level of spatial details in the land cover product, misclassification was observed for some classes, such as road versus buildings or shadow versus lake. Despite these minor drawbacks, our results also demonstrated the benefits of IowaNet training dataset in the model performance; overfitting process reduces as the number of samples increases. The limitations of multi-OCNN are partially explained by segmentation quality and limited number of spectral bands in the aerial data. With the advance of deep learning methods, this study supports the claim of multi-OCNN benefits for operational large-scale land cover product at 1-m resolution.

Path detection for autonomous traveling in orchards using patch-based CNN

This paper proposes a novel and efficient patch-based approach for autonomous path detection in semi-structured environments such as orchards. The proposed approach can segment a perspective path area in the frontal scene and is expected to be applicable to various types of area detection tasks in agriculture. In the proposed approach, a patch-based convolutional neural network (CNN) is used for image classification to achieve path area segmentation, which involves cropping image patches from an input image by using a sliding window, generating a path score map by using a four-layer CNN for tree and path classification, path area segmentation, and target path detection by using boundary line determination. Results show that the maximum intersection over union (IoU) is approximately 0.81 for path area localization and the average lateral and angular errors are 0.051 and 7.8°, respectively. The performance of patch-based path detection depends on the patch size of the sliding window. Hence, the path detection performance is evaluated in terms of the patch size, and a patch size of 96 × 96 shows the best performance, with a classification accuracy of 0.93, IoU of 0.75, and processing time of 0.111 s. In addition, the proposed approach was verified by applying the approach to various images including curved paths. The results indicate that the performance of the proposed patch-based approach for path detection is comparable to that of previous approaches. Moreover, an autonomous farm robot can be easily developed using the proposed technique with a simple hardware configuration.

Tumor detection for whole slide image of liver based on patch-based convolutional neural network

Liver cancer has a huge negative impact on the human survival. However, the traditional histopathological diagnostic methods have a large burden on the clinical diagnosis due to the large workload required. To this end, by combining the machine learning method and the whole slide images (WSIs), this paper proposes a novelty method to improve the efficiency of the clinical diagnosis. The core of our proposed method is to use a patch-based convolutional neural network to perform the category prediction (normal or tumor) on the patches extracted from the 60 liver tumor WSIs. Specifically, we design a total of four sets of the comparative experiment for the patch classification in order to screen the classifier with the best classification effect. The best classifier is then used to predict the category of patches in the testing set, and the results are combined to generate a probability heatmap to assist clinical diagnosis in an intuitive way. The experimental comparisons verify the validity of the proposed model which can achieve a classification accuracy rate of 99.94% in the patch classification task.

Investigating the challenges and generalizability of deep learning brain conductivity mapping

To investigate deep learning electrical properties tomography (EPT) for application on different simulated and in-vivo datasets, including pathologies for brain conductivity reconstructions, 3D patch-based convolutional neural networks were trained to predict conductivity maps from B1 transceive phase data. To compare the performance of DL-EPT networks on different datasets, three datasets were used throughout this work, one from simulations and two from in-vivo measurements from healthy volunteers and patients with brain lesions, respectively. At first, networks trained on simulations were tested on all datasets with different levels of homogeneous Gaussian noise introduced in training and testing. Secondly, to investigate potential robustness towards systematical differences between simulated and measured phase maps, in-vivo data with conductivity labels from conventional EPT were used for training. High quality conductivity reconstructions from networks trained on simulations with and without noise confirm the potential of deep learning for EPT. However, when this network is used for in-vivo reconstructions, measurement related artifacts affect the quality of conductivity maps. Training DL-EPT networks using conductivity labels from conventional EPT improves the quality of the results. Networks trained on realistic simulations yield reconstruction artifacts when applied to in-vivo data. Training with realistic phase data and conductivity labels from conventional EPT allows for reducing these artifacts.

Deep Learning COVID-19 Features on CXR Using Limited Training Data Sets.

Under the global pandemic of COVID-19, the use of artificial intelligence to analyze chest X-ray (CXR) image for COVID-19 diagnosis and patient triage is becoming important. Unfortunately, due to the emergent nature of the COVID-19 pandemic, a systematic collection of CXR data set for deep neural network training is difficult. To address this problem, here we propose a patch-based convolutional neural network approach with a relatively small number of trainable parameters for COVID-19 diagnosis. The proposed method is inspired by our statistical analysis of the potential imaging biomarkers of the CXR radiographs. Experimental results show that our method achieves state-of-the-art performance and provides clinically interpretable saliency maps, which are useful for COVID-19 diagnosis and patient triage.

Deep learning‐based reconstruction of in vivo pelvis conductivity with a 3D patch‐based convolutional neural network trained on simulated MR data

PurposeTo demonstrate that mapping pelvis conductivity at 3T with deep learning (DL) is feasible.Methods210 dielectric pelvic models were generated based on CT scans of 42 cervical cancer patients. For all dielectric models, electromagnetic and MR simulations with realistic accuracy and precision were performed to obtain B1+ and transceive phase (ϕ±). Simulated B1+ and ϕ± served as input to a 3D patch‐based convolutional neural network, which was trained in a supervised fashion to retrieve the conductivity. The same network architecture was retrained using only ϕ± in input. Both network configurations were tested on simulated MR data and their conductivity reconstruction accuracy and precision were assessed. Furthermore, both network configurations were used to reconstruct conductivity maps from a healthy volunteer and two cervical cancer patients. DL‐based conductivity was compared in vivo and in silico to Helmholtz‐based (H‐EPT) conductivity.ResultsConductivity maps obtained from both network configurations were comparable. Accuracy was assessed by mean error (ME) with respect to ground truth conductivity. On average, ME < 0.1 Sm−1 for all tissues. Maximum MEs were 0.2 Sm−1 for muscle and tumour, and 0.4 Sm−1 for bladder. Precision was indicated with the difference between 90th and 10th conductivity percentiles, and was below 0.1 Sm−1 for fat, bone and muscle, 0.2 Sm−1 for tumour and 0.3 Sm−1 for bladder. In vivo, DL‐based conductivity had median values in agreement with H‐EPT values, but a higher precision.ConclusionAnatomically detailed, noise‐robust 3D conductivity maps with good sensitivity to tissue conductivity variations were reconstructed in the pelvis with DL.

Segmentation of cervical intervertebral disks in videofluorography by CNN, multi-channelization and feature selection.

Dysphagia has a large impact on the society because it is a risk factor of malnutrition and aspiration pneumonia, and therefore, it is necessary to elucidate the entire mechanism of dysphagia. In this study, we propose a segmentation method of cervical intervertebral disks (CIDs) in videofluorography (VF) by use of patch-based convolutional neural network (CNN), our multi-channelization (MC) method and image feature selection. Twenty image filters are individually applied to a VF frame image to generate feature images. One color image, called a multi-channelized image, is generated by setting three selected feature images to its red, green and blue channels. Patch-based CNN is applied to the MC image, and the segmentation accuracy of CIDs is evaluated by the pixel-based F-measure. The combination of the three feature images is optimized by the simulated annealing method. The proposed method was applied to actual VF dataset consisting of 19 patients and 39 healthy participants. The segmentation accuracy was 59.3% in the F-measure when Sobel and morphological top-hat filters were selected in MC, whereas it was 56.2% when original frame images were used. The experimental results demonstrated that the proposed method was able to segment CIDs from actual VF and also that the MC method was able to increase the segmentation accuracy by approximately 3%. In this study, LeNet was used as CNN. One of our future tasks is to use other CNNs.

Different Spectral Domain Transformation for Land Cover Classification Using Convolutional Neural Networks with Multi-Temporal Satellite Imagery

This study compares some different types of spectral domain transformations for convolutional neural network (CNN)-based land cover classification. A novel approach was proposed, which transforms one-dimensional (1-D) spectral vectors into two-dimensional (2-D) features: Polygon graph images (CNN-Polygon) and 2-D matrices (CNN-Matrix). The motivations of this study are that (1) the shape of the converted 2-D images is more intuitive for human eyes to interpret when compared to 1-D spectral input; and (2) CNNs are highly specialized and may be able to similarly utilize this information for land cover classification. Four seasonal Landsat 8 images over three study areas—Lake Tapps, Washington, Concord, New Hampshire, USA, and Gwangju, Korea—were used to evaluate the proposed approach for nine land cover classes compared to several other methods: Random forest (RF), support vector machine (SVM), 1-D CNN, and patch-based CNN. Oversampling and undersampling approaches were conducted to examine the effect of the sample size on the model performance. The CNN-Polygon had better performance than the other methods, with overall accuracies of about 93%–95 % for both Concord and Lake Tapps and 80%–84% for Gwangju. The CNN-Polygon particularly performed well when the training sample size was small, less than 200 per class, while the CNN-Matrix resulted in similar or higher performance as sample sizes became larger. The contributing input variables to the models were carefully analyzed through sensitivity analysis based on occlusion maps and accuracy decreases. Our result showed that a more visually intuitive representation of input features for CNN-based classification models yielded higher performance, especially when the training sample size was small. This implies that the proposed graph-based CNNs would be useful for land cover classification where reference data are limited.

Hyperspectral Microscopic Imaging for Automatic Detection of Head and Neck Squamous Cell Carcinoma Using Histologic Image and Machine Learning.

The purpose of this study is to develop hyperspectral imaging (HSI) for automatic detection of head and neck cancer cells on histologic slides. A compact hyperspectral microscopic system is developed in this study. Histologic slides from 15 patients with squamous cell carcinoma (SCC) of the larynx and hypopharynx are imaged with the system. The proposed nuclei segmentation method based on principle component analysis (PCA) can extract most nuclei in the hyperspectral image without extracting other sub-cellular components. Both spectra-based support vector machine (SVM) and patch-based convolutional neural network (CNN) are used for nuclei classification. CNNs were trained with both hyperspectral images and pseudo RGB images of extracted nuclei, in order to evaluate the usefulness of extra information provided by hyperspectral imaging. The average accuracy of spectra-based SVM classification is 68%. The average AUC and average accuracy of the HSI patch-based CNN classification is 0.94 and 82.4%, respectively. The hyperspectral microscopic imaging and classification methods provide an automatic tool to aid pathologists in detecting SCC on histologic slides.

Improving Patch-Based Convolutional Neural Networks for MRI Brain Tumor Segmentation by Leveraging Location Information.

The manual brain tumor annotation process is time consuming and resource consuming, therefore, an automated and accurate brain tumor segmentation tool is greatly in demand. In this paper, we introduce a novel method to integrate location information with the state-of-the-art patch-based neural networks for brain tumor segmentation. This is motivated by the observation that lesions are not uniformly distributed across different brain parcellation regions and that a locality-sensitive segmentation is likely to obtain better segmentation accuracy. Toward this, we use an existing brain parcellation atlas in the Montreal Neurological Institute (MNI) space and map this atlas to the individual subject data. This mapped atlas in the subject data space is integrated with structural Magnetic Resonance (MR) imaging data, and patch-based neural networks, including 3D U-Net and DeepMedic, are trained to classify the different brain lesions. Multiple state-of-the-art neural networks are trained and integrated with XGBoost fusion in the proposed two-level ensemble method. The first level reduces the uncertainty of the same type of models with different seed initializations, and the second level leverages the advantages of different types of neural network models. The proposed location information fusion method improves the segmentation performance of state-of-the-art networks including 3D U-Net and DeepMedic. Our proposed ensemble also achieves better segmentation performance compared to the state-of-the-art networks in BraTS 2017 and rivals state-of-the-art networks in BraTS 2018. Detailed results are provided on the public multimodal brain tumor segmentation (BraTS) benchmarks.

Automated Segmentation of Cervical Intervertebral Disks from Videofluorography Using a Convolutional Neural Network and its Performance Evaluation

Dysphagia has become an important issue in many countries, and there is a strong need for elucidating the causes of dysphagia. One of the promising ways is to analyze the static and dynamic mechanisms of cervical structures, such as epiglottises, hyoid bones, and cervical vertebral bodies, based on medical images. In this study, we propose an automated segmentation method of cervical intervertebral disks (IDs) from videofluorography (VF) by use of a convolutional neural network (CNN). First, cervical masks are extracted from the frame images of VF, and then a patch-based CNN is applied to the cervical masks to obtain the probability images of ID regions. The sizes of patches are changed in a certain range, and pixel values in the patches are normalized. Morphological image filters are applied to the probability images to eliminate false-positive pixels. The proposed method is applied to VF of 58 participants, consisting of 39 healthy people and 19 patients. The segmentation results are compared with the ground truth as determined by a medical doctor and are evaluated with the pixel-wise F-measure. The F-measure is highest (0.880) when the patch size is 21 × 21 pixels and the both of the pixel value normalization (PVN) and the false positive elimination (FPE) are applied. On the other hand, the F-measure is lowest (0.443) when the patch size is 15 × 15 pixels and neither PVN nor FPE is applied.