Global Convolutional Neural Networks Research Articles

Deep Tutti Frutti: Exploring CNN architectures for dry matter prediction in fruit from multi-fruit near-infrared spectra

Convolutional Neural Networks (CNNs) have proven to be a valuable Deep Learning (DL) algorithm to model near-infrared spectral data in Chemometrics. However, optimizing CNN architectures and their associated hyperparameters for specific tasks is challenging. In this study, we explore the development of 1D-CNN architectures for the task of fruit dry matter (DM) estimation, testing various designs and optimization strategies to achieve a generic DL model that is robust against data fluctuations. The models are built using a multi-fruit data set (n = 2397) that includes NIR spectra of apples, kiwis, mangoes, and pears. The obtained CNN models are compared with PLS (taken as baseline), and to Locally Weighted PLS (LW-PLS) models. In general, the optimized CNN architectures obtained lower RMSEs (best RMSE = 0.605 %) than PLS (RMSE = 0.892 %) and LW-PLS (RMSE = 0.687 %) on a holdout test set. For this specific task, CNNs start outperforming PLS when the number of training samples is around 500. Furthermore, it is also shown how a global CNN model, trained on multi-fruit data, performs against PLS models of individual fruits in the sub-tasks of individual fruit DM prediction and generalization to an external mango data set. Overall, with proper architecture optimization, CNNs show strong performance and generalization for NIR-based dry matter estimation across diverse fruits.

Improving convolutional neural networks for fault diagnosis in chemical processes by incorporating global correlations

Fault diagnosis (FD) has received attention because of its importance in maintaining safe operations of industrial processes. Recently, modern data-driven FD approaches such as deep learning have shown encouraging performance. Particularly, convolutional neural networks (CNNs) offer an alluring capacity to deal with multivariate time-series data converted into images. Nonetheless, existing CNN techniques focus on capturing local correlations. However, global spatiotemporal correlations often prevail in multivariate time-series data from industrial processes. Hence, extracting global correlations using CNNs from such data requires deep architectures that incur many trainable parameters. This paper proposes a novel local–global scale CNN (LGS-CNN) that directly accounts for local and global correlations. Specifically, the proposed network incorporates local correlations through traditional square kernels and global correlations are collected utilizing spatially separated one-dimensional kernels in a unique arrangement. FD performance on the benchmark Tennessee Eastman process dataset validates the proposed LGS-CNN against CNNs, and other state-of-the-art data-driven FD approaches.

FedStack: Personalized activity monitoring using stacked federated learning

Recent advances in remote patient monitoring (RPM) systems can recognize various human activities to measure vital signs, including subtle motions from superficial vessels. There is a growing interest in applying artificial intelligence (AI) to this area of healthcare by addressing known limitations and challenges such as predicting and classifying vital signs and physical movements, which are considered crucial tasks. Federated learning is a relatively new AI technique designed to enhance data privacy by decentralizing traditional machine learning modeling. However, traditional federated learning requires identical architectural models to be trained across the local clients and global servers. This limits global model architecture due to the lack of local models’ heterogeneity. To overcome this, a novel federated learning architecture, FedStack, which supports ensembling heterogeneous architectural client models was proposed in this study. This work offers a protected privacy system for hospitalized in-patients in a decentralized approach and identifies optimum sensor placement. The proposed architecture was applied to a mobile health sensor benchmark dataset from 10 different subjects to classify 12 routine activities. Three AI models, artificial neural network (ANN), convolutional neural network (CNN), and bidirectional long short-term memory (Bi-LSTM) were trained on individual subject data. The federated learning architecture was applied to these models to build local and global models capable of state-of-the-art performances. The local CNN model outperformed ANN and Bi-LSTM models on each subject data. Our proposed work has demonstrated better performance for heterogeneous stacking of the local models compared to homogeneous stacking. Further analysis of the global heterogeneous CNN model determined that the optimum placement of the sensors on human limbs resulted in better activity recognition. This work sets the stage to build an enhanced RPM system that incorporates client privacy to assist with clinical observations for patients in an acute mental health facility and ultimately help to prevent unexpected death.

Data mining of urban soil spectral library for estimating organic carbon

Accurate quantification of urban soil organic carbon (SOC) is essential for understanding anthropogenic changes and further guiding effective city managements. Visible and near infrared (vis–NIR) spectroscopy can monitor the SOC content in a time- and cost-effective manner. However, processes and mechanisms dominating the relationships between SOC and spectral data in urban soils remain unknown. The main objective of this paper was to evaluate whether multiple stratification strategies (i.e., based on land-use/land-cover [LULC], pH, and spectral clustering) resulted in better predicted performance for SOC compared to the non-stratified (global) models. Results showed that regarding the non-stratified models, the convolutional neural network (CNN) model exhibited the best performance (validation R2 = 0.73), followed by Cubist (validation R2 = 0.66) and memory-based learning (validation R2 = 0.65). After LULC stratification, Cubist model achieved the best prediction (validation R2 = 0.76), improving the value of ratio of performance to interquartile distance by 0.11 compared to the global CNN model. Areas with high SOC values were mainly located in the city center. Stratification by LULC class is a promising strategy for addressing the impact of the soil-landscape diversity and complexity on vis–NIR spectral estimation of SOC in urban soil spectral library.

Directionally separable dilated CNN with hierarchical attention feature fusion for hyperspectral image classification

ABSTRACT In recent years, the convolutional neural network (CNN) plays a vital role in hyperspectral image classification and performs more competitively than many other methods. However, in order to pursue better performance, most of existing CNN-based methods just simply stack rather deep convolutional layers. Although they improve the classification accuracy to a certain extent, they result in plenty of network parameters. In this paper, a light-weighted directionally separable dilated CNN with hierarchical attention feature fusion (DSD-HAFF) is proposed to solve these problems. First, two global dense dilated CNN branches that focus on two spatial directions separately are constructed to extract and reuse spatial information as much as possible. Second, a hierarchical attention feature fusion branch that consists of several coordinate attention blocks (CABs) is constructed. Hierarchical features from two directionally separable dilated CNN branches are adopted as inputs of CABs. In this way, the structure can not only fully incorporate hierarchical features, but also significantly reduce the network parameters. Meanwhile, the hierarchical attention feature fusion branch incorporates features from high-level to low-level in the kernel-number pyramid strategy. Experimental results on three popular benchmark datasets demonstrate that the DSD-HAFF achieves better performance and has a much smaller number of network parameters than the other state-of-the-art methods.

A convolutional neural network for common coordinate registration of high-resolution histology images.

MotivationRegistration of histology images from multiple sources is a pressing problem in large-scale studies of spatial -omics data. Researchers often perform ‘common coordinate registration’, akin to segmentation, in which samples are partitioned based on tissue type to allow for quantitative comparison of similar regions across samples. Accuracy in such registration requires both high image resolution and global awareness, which mark a difficult balancing act for contemporary deep learning architectures.ResultsWe present a novel convolutional neural network (CNN) architecture that combines (i) a local classification CNN that extracts features from image patches sampled sparsely across the tissue surface and (ii) a global segmentation CNN that operates on these extracted features. This hybrid network can be trained in an end-to-end manner, and we demonstrate its relative merits over competing approaches on a reference histology dataset as well as two published spatial transcriptomics datasets. We believe that this paradigm will greatly enhance our ability to process spatial -omics data, and has general purpose applications for the processing of high-resolution histology images on commercially available GPUs.Availability and implementationAll code is publicly available at https://github.com/flatironinstitute/st_gridnet.Supplementary information Supplementary data are available at Bioinformatics online.

A very high-resolution scene classification model using transfer deep CNNs based on saliency features

Developing remote sensing technology enables the production of very high-resolution (VHR) images. Classification of the VHR imagery scene has become a challenging problem. In this paper, we propose a model for VHR scene classification. First, convolutional neural networks (CNNs) with pre-trained weights are used as a deep feature extractor to extract the global and local CNNs features from the original VHR images. Second, the spectral residual-based saliency detection algorithm is used to extract the saliency map. Then, saliency features from the saliency map are extracted using CNNs in order to extract robust features for the VHR imagery, especially for the image with salience object. Third, we use the feature fusion technique rather than the raw deep features to represent the final shape of the VHR image scenes. In feature fusion, discriminant correlation analysis (DCA) is used to fuse both the global and local CNNs features and saliency features. DCA is a more suitable and cost-effective fusion method than the traditional fusion techniques. Finally, we propose an enhanced multilayer perceptron to classify the image. Experiments are performed on four widely used datasets: UC-Merced, WHU-RS, Aerial Image, and NWPU-RESISC45. Results confirm that the proposed model performs better than state-of-the-art scene classification models.

ERF-IMCS: An Efficient and Robust Framework with Image-Based Monte Carlo Scheme for Indoor Topological Navigation

Conventional approaches to global localization and navigation mainly rely on metric maps to provide precise geometric coordinates, which may cause the problem of large-scale structural ambiguity and lack semantic information of the environment. This paper presents a scalable vision-based topological mapping and navigation method for a mobile robot to work robustly and flexibly in large-scale environment. In the vision-based topological navigation, an image-based Monte Carlo localization method is presented to realize global topological localization based on image retrieval, in which fine-tuned local region features from an object detection convolutional neural network (CNN) are adopted to perform image matching. The combination of image retrieval and Monte Carlo provide the robot with the ability to effectively avoid perceptual aliasing. Additionally, we propose an effective visual localization method, simultaneously employing the global and local CNN features of images to construct discriminative representation for environment, which makes the navigation system more robust to the interference of occlusion, translation, and illumination. Extensive experimental results demonstrate that ERF-IMCS exhibits great performance in the robustness and efficiency of navigation.

CNN Feature-Based Image Copy Detection with Contextual Hash Embedding

As one of the important techniques for protecting the copyrights of digital images, content-based image copy detection has attracted a lot of attention in the past few decades. The traditional content-based copy detection methods usually extract local hand-crafted features and then quantize these features to visual words by the bag-of-visual-words (BOW) model to build an inverted index file for rapid image matching. Recently, deep learning features, such as the features derived from convolutional neural networks (CNN), have been proven to outperform the hand-crafted features in many applications of computer vision. However, it is not feasible to directly apply the existing global CNN features for copy detection, since they are usually sensitive to partial content-discarded attacks, such as copping and occlusion. Thus, we propose a local CNN feature-based image copy detection method with contextual hash embedding. We first extract the local CNN features from images and then quantize them to visual words to construct an index file. Then, as the BOW quantization process decreases the discriminability of these features to some extent, a contextual hash sequence is captured from a relatively large region surrounding each CNN feature and then is embedded into the index file to improve the feature’s discriminability. Extensive experimental results demonstrate that the proposed method achieves a superior performance compared to the related works in the copy detection task.

Near-Duplicate Image Detection System Using Coarse-to-Fine Matching Scheme Based on Global and Local CNN Features

Due to the great success of convolutional neural networks (CNNs) in the area of computer vision, the existing methods tend to match the global or local CNN features between images for near-duplicate image detection. However, global CNN features are not robust enough to combat background clutter and partial occlusion, while local CNN features lead to high computational complexity in the step of feature matching. To achieve high efficiency while maintaining good accuracy, we propose a coarse-to-fine feature matching scheme using both global and local CNN features for real-time near-duplicate image detection. In the coarse matching stage, we implement the sum-pooling operation on convolutional feature maps (CFMs) to generate the global CNN features, and match these global CNN features between a given query image and database images to efficiently filter most of irrelevant images of the query. In the fine matching stage, the local CNN features are extracted by using maximum values of the CFMs and the saliency map generated by the graph-based visual saliency detection (GBVS) algorithm. These local CNN features are then matched between images to detect the near-duplicate versions of the query. Experimental results demonstrate that our proposed method not only achieves a real-time detection, but also provides higher accuracy than the state-of-the-art methods.

GP-CNN-DTEL: Global-Part CNN Model With Data-Transformed Ensemble Learning for Skin Lesion Classification.

Precise skin lesion classification is still challenging due to two problems, i.e., (1) inter-class similarity and intra-class variation of skin lesion images, and (2) the weak generalization ability of single Deep Convolutional Neural Network trained with limited data. Therefore, we propose a Global-Part Convolutional Neural Network (GP-CNN) model, which treats the fine-grained local information and global context information with equal importance. The Global-Part model consists of a Global Convolutional Neural Network (G-CNN) and a Part Convolutional Neural Network (P-CNN). Specifically, the G-CNN is trained with downscaled dermoscopy images, and is used to extract the global-scale information of dermoscopy images and produce the Classification Activation Map (CAM). While the P-CNN is trained with the CAM guided cropped image patches and is used to capture local-scale information of skin lesion regions. Additionally, we present a data-transformed ensemble learning strategy, which can further boost the classification performance by integrating the different discriminant information from GP-CNNs that are trained with original images, color constancy transformed images, and feature saliency transformed images, respectively. The proposed method is evaluated on the ISIC 2016 and ISIC 2017 Skin Lesion Challenge (SLC) classification datasets. Experimental results indicate that the proposed method can achieve the state-of-the-art skin lesion classification performance (i.e., an AP value of 0.718 on the ISIC 2016 SLC dataset and an Average Auc value of 0.926 on the ISIC 2017 SLC dataset) without any external data, compared with other current methods which need to use external data.

Near Duplicate Image Retrieval using Multilevel Local and Global Convolutional Neural Network Features

In this work, we present an approach based on multilevel local as well as global Convolutional Neural Network (CNN) feature matching to retrieve near duplicate images. CNN features are suitable for visual matching. The CNN features of entire image may not give accuracy in retrieval due to various image editing/capturing operations. Our retrieval task focuses on matching image pairs based on local and global levels. In local matching, an image is segmented into fixed size blocks followed by extracting patches by considering neighboring regions at different levels. Matching local image patches at different levels provides robustness to our retrieval model. In local patch extraction, we select blocks containing SURF feature points instead of selecting all blocks. CNN features are extracted and stored for each image patch and then followed by extraction of global CNN features. Finally, similarity between image pairs is computed by considering all extracted CNN features. Our similarity function is based on correlation and number of blocks found in matching. We implemented our proposed approach on benchmarking Holiday dataset. Retrieval results show remarkable improvement in mean average precision (mAP) on the dataset.

Research on image feature extraction and retrieval algorithms based on convolutional neural network

With the rapid development of mobile Internet and digital technology, people are more and more keen to share pictures on social networks, and online pictures have exploded. How to retrieve similar images from large-scale images has always been a hot issue in the field of image retrieval, and the selection of image features largely affects the performance of image retrieval. The Convolutional Neural Networks (CNN), which contains more hidden layers, has more complex network structure and stronger ability of feature learning and expression compared with traditional feature extraction methods. By analyzing the disadvantage that global CNN features cannot effectively describe local details when they act on image retrieval tasks, a strategy of aggregating low-level CNN feature maps to generate local features is proposed. The high-level features of CNN model pay more attention to semantic information, but the low-level features pay more attention to local details. Using the increasingly abstract characteristics of CNN model from low to high. This paper presents a probabilistic semantic retrieval algorithm, proposes a probabilistic semantic hash retrieval method based on CNN, and designs a new end-to-end supervised learning framework, which can simultaneously learn semantic features and hash features to achieve fast image retrieval. Using convolution network, the error rate is reduced to 14.41% in this test set. In three open image libraries, namely Oxford, Holidays and ImageNet, the performance of traditional SIFT-based retrieval algorithms and other CNN-based image retrieval algorithms in tasks are compared and analyzed. The experimental results show that the proposed algorithm is superior to other contrast algorithms in terms of comprehensive retrieval effect and retrieval time.

Using Machine Learning to Construct Velocity Fields from OH-PLIF Images

ABSTRACT This work utilizes data-driven methods to morph a series of time-resolved experimental OH-PLIF images into corresponding three-component planar PIV fields in the closed domain of a premixed swirl combustor. The task is carried out with a fully convolutional network, which is a type of convolutional neural network (CNN) used in many applications in machine learning, alongside an existing experimental dataset which consists of simultaneous OH-PLIF and PIV measurements in both attached and detached flame regimes. Two types of models are compared: 1) a global CNN which is trained using images from the entire domain, and 2) a set of local CNNs, which are trained only on individual sections of the domain. The locally trained models show improvement in creating mappings in the detached regime over the global models. A comparison between model performance in attached and detached regimes shows that the CNNs are much more accurate across the board in creating velocity fields for attached flames. Inclusion of time history in the PLIF input resulted in small noticeable improvement on average, which could imply a greater physical role of instantaneous spatial correlations in the decoding process over temporal dependencies from the perspective of the CNN. Additionally, the performance of local models trained to produce mappings in one section of the domain is tested on other, unexplored sections of the domain. Interestingly, local CNN performance on unseen domain regions revealed the models’ ability to utilize symmetry and antisymmetry in the velocity field. Ultimately, this work shows the powerful ability of the CNN to decode the three-dimensional PIV fields from input OH-PLIF images, providing a potential groundwork for a very useful tool for experimental configurations in which accessibility of forms of simultaneous measurements are limited.

A Deep Scene Representation for Aerial Scene Classification

As a fundamental problem in earth observation, aerial scene classification tries to assign a specific semantic label to an aerial image. In recent years, the deep convolutional neural networks (CNNs) have shown advanced performances in aerial scene classification. The successful pretrained CNNs can be transferable to aerial images. However, global CNN activations may lack geometric invariance and, therefore, limit the improvement of aerial scene classification. To address this problem, this paper proposes a deep scene representation to achieve the invariance of CNN features and further enhance the discriminative power. The proposed method: 1) extracts CNN activations from the last convolutional layer of pretrained CNN; 2) performs multiscale pooling (MSP) on these activations; and 3) builds a holistic representation by the Fisher vector method. MSP is a simple and effective multiscale strategy, which enriches multiscale spatial information in affordable computational time. The proposed representation is particularly suited at aerial scenes and consistently outperforms global CNN activations without requiring feature adaptation. Extensive experiments on five aerial scene data sets indicate that the proposed method, even with a simple linear classifier, can achieve the state-of-the-art performance.

Bidirectional adaptive feature fusion for remote sensing scene classification

Abstract Scene classification has become an effective way to interpret the High Spatial Resolution (HSR) remote sensing images. Recently, Convolutional Neural Networks (CNN) have been found to be excellent for scene classification. However, only using the deep models as feature extractor on the aerial image directly is not proper, because the extracted deep features can not capture spatial scale variability and rotation variability in HSR remote sensing images. To relieve this limitation, a bidirectional adaptive feature fusion strategy is investigated to deal with the remote sensing scene classification. The deep learning feature and the SIFT feature are fused together to get a discriminative image presentation. The fused feature can not only describe the scenes effectively by employing deep learning feature but also overcome the scale and rotation variability with the usage of the SIFT feature. By fusing both SIFT feature and global CNN feature, our method achieves state-of-the-art scene classification performances on the UCMerced, the Sydney and the AID datasets.

Pedestrian gender classification using combined global and local parts-based convolutional neural networks

The identification of a person’s gender plays an important role in various visual surveillance and monitoring applications which are growing more ubiquitously. This paper proposes a method for gender classification of pedestrians based on whole body images which, unlike facial-based methods, allows for observation from different viewpoints. We use a parts-based model that combines global and local information to make inference. Convolutional neural network (CNN) is leveraged for its superior feature learning and classification capability. Our method requires that only the gender label is available for the training images, without the need for any other expensive annotation such as the anatomical parts, key points or other attributes. We trained a CNN on the bounding box containing the whole body (global CNN) or a defined portion of the body (local CNN). Experimental results show that the upper half region of the body is the most discriminative for gender, in comparison with the middle or lower half. The best model is a jointly trained combination of a global CNN and a local upper body CNN, which achieves higher accuracy than previous works on publicly available datasets.

Predicting RNA-protein binding sites and motifs through combining local and global deep convolutional neural networks.

RNA-binding proteins (RBPs) take over 5-10% of the eukaryotic proteome and play key roles in many biological processes, e.g. gene regulation. Experimental detection of RBP binding sites is still time-intensive and high-costly. Instead, computational prediction of the RBP binding sites using patterns learned from existing annotation knowledge is a fast approach. From the biological point of view, the local structure context derived from local sequences will be recognized by specific RBPs. However, in computational modeling using deep learning, to our best knowledge, only global representations of entire RNA sequences are employed. So far, the local sequence information is ignored in the deep model construction process. In this study, we present a computational method iDeepE to predict RNA-protein binding sites from RNA sequences by combining global and local convolutional neural networks (CNNs). For the global CNN, we pad the RNA sequences into the same length. For the local CNN, we split a RNA sequence into multiple overlapping fixed-length subsequences, where each subsequence is a signal channel of the whole sequence. Next, we train deep CNNs for multiple subsequences and the padded sequences to learn high-level features, respectively. Finally, the outputs from local and global CNNs are combined to improve the prediction. iDeepE demonstrates a better performance over state-of-the-art methods on two large-scale datasets derived from CLIP-seq. We also find that the local CNN runs 1.8 times faster than the global CNN with comparable performance when using GPUs. Our results show that iDeepE has captured experimentally verified binding motifs. https://github.com/xypan1232/iDeepE. Supplementary data are available at Bioinformatics online.

Patient-specific and global convolutional neural networks for robust automatic liver tumor delineation in follow-up CT studies.

Radiological longitudinal follow-up of tumors in CT scans is essential for disease assessment and liver tumor therapy. Currently, most tumor size measurements follow the RECIST guidelines, which can be off by as much as 50%. True volumetric measurements are more accurate but require manual delineation, which is time-consuming and user-dependent. We present a convolutional neural networks (CNN) based method for robust automatic liver tumor delineation in longitudinal CT studies that uses both global and patient specific CNNs trained on a small database of delineated images. The inputs are the baseline scan and the tumor delineation, a follow-up scan, and a liver tumor global CNN voxel classifier built from radiologist-validated liver tumor delineations. The outputs are the tumor delineations in the follow-up CT scan. The baseline scan tumor delineation serves as a high-quality prior for the tumor characterization in the follow-up scans. It is used to evaluate the global CNN performance on the new case and to reliably predict failures of the global CNN on the follow-up scan. High-scoring cases are segmented with a global CNN; low-scoring cases, which are predicted to be failures of the global CNN, are segmented with a patient-specific CNN built from the baseline scan. Our experimental results on 222 tumors from 31 patients yield an average overlap error of 17% (std = 11.2) and surface distance of 2.1mm (std = 1.8), far better than stand-alone segmentation. Importantly, the robustness of our method improved from 67% for stand-alone global CNN segmentation to 100%. Unlike other medical imaging deep learning approaches, which require large annotated training datasets, our method exploits the follow-up framework to yield accurate tumor tracking and failure detection and correction with a small training dataset. Graphical abstract Flow diagram of the proposed method. In the offline mode (orange), a global CNN is trained as a voxel classifier to segment liver tumor as in [31]. The online mode (blue) is used for each new case. The input is baseline scan with delineation and the follow-up CT scan to be segmented. The main novelty is the ability to predict failures by trying the system on the baseline scan and the ability to correct them using the patient-specific CNN.

Three-stream CNNs for action recognition

Existing Convolutional Neural Networks (CNNs) based methods for action recognition are either spatial or temporally local while actions are 3D signals. In this paper, we propose a global spatial-temporal three-stream CNNs architecture, which is able to be used for action feature extraction. Specifically, the three-stream CNNs comprises of spatial, local temporal and global temporal streams generated respectively from deep learning single frame, optical flow and global accumulated motion features in the form of a new formulation named Motion Stacked Difference Image (MSDI). Moreover, a novel soft Vector of Locally Aggregated Descriptors (soft-VLAD) is developed to further represent the extracted features, combining the advantage of Gaussian Mixture Models (GMMs) and VLAD by encoding data according to their overall probability distribution and the corresponding difference with respect to clustered centers. To deal with the inadequacy of training samples during learning, we introduce a data augmentation scheme which is very efficient due to its origin at cropping across videos. We conduct our experiments on UCF101 and HMDB51 datasets, and the results demonstrate the effectiveness of our approach.