Representative Convolutional Neural Networks Research Articles

Reusing Convolutional Neural Network Models through Modularization and Composition

With the widespread success of deep learning technologies, many trained deep neural network (DNN) models are now publicly available. However, directly reusing the public DNN models for new tasks often fails due to mismatching functionality or performance. Inspired by the notion of modularization and composition in software reuse, we investigate the possibility of improving the reusability of DNN models in a more fine-grained manner. Specifically, we propose two modularization approaches named CNNSplitter and GradSplitter, which can decompose a trained convolutional neural network (CNN) model for N -class classification into N small reusable modules. Each module recognizes one of the N classes and contains a part of the convolution kernels of the trained CNN model. Then, the resulting modules can be reused to patch existing CNN models or build new CNN models through composition. The main difference between CNNSplitter and GradSplitter lies in their search methods: the former relies on a genetic algorithm to explore search space, while the latter utilizes a gradient-based search method. Our experiments with three representative CNNs on three widely used public datasets demonstrate the effectiveness of the proposed approaches. Compared with CNNSplitter, GradSplitter incurs less accuracy loss, produces much smaller modules (19.88% fewer kernels), and achieves better results on patching weak models. In particular, experiments on GradSplitter show that (1) by patching weak models, the average improvement in terms of precision, recall, and F1-score is 17.13%, 4.95%, and 11.47%, respectively, and (2) for a new task, compared with the models trained from scratch, reusing modules achieves similar accuracy (the average loss of accuracy is only 2.46%) without a costly training process. Our approaches provide a viable solution to the rapid development and improvement of CNN models.

Dataflow optimization with layer-wise design variables estimation method for enflame CNN accelerators

As convolution layers have been proved to be the most time-consuming operation in convolutional neural network (CNN) algorithms, many efficient CNN accelerators have been designed to boost the performance of convolution operations. Previous works on CNN acceleration usually use fixed design variables for diverse convolutional layers, which would lead to inefficient data movements and low utilization of computing resource. We tackle this issue by proposing a flexible dataflow optimization method with design variables estimation for different layers. The optimization method first narrows the design space by the priori constraints, and then enumerates all legal solutions to select the optimal design variables. We demonstrate the effectiveness of the proposed optimization method by implementing representative CNN models (VGG-16, ResNet-18 and MobileNet V1) on Enflame Technology's programmable CNN accelerator, General Computing Unit (GCU). The results indicate that our optimization can significantly enhance the throughput of the convolution layers in ResNet, VGG and MobileNet on GCU, with improvement of up to 1.84×. Furthermore, it achieves up to 2.08× of GCU utilization specifically for the convolution layers of ResNet on GCU.

Development and Preliminary Validation of a Novel Convolutional Neural Network Model for Predicting Treatment Response in Patients with Unresectable Hepatocellular Carcinoma Receiving Hepatic Arterial Infusion Chemotherapy.

The goal of this study was to evaluate the performance of a convolutional neural network (CNN) with preoperative MRI and clinical factors in predicting the treatment response of unresectable hepatocellular carcinoma (HCC) patients receiving hepatic arterial infusion chemotherapy (HAIC). A total of 191 patients with unresectable HCC who underwent HAIC in our hospital between May 2019 and March 2022 were retrospectively recruited. We selected InceptionV4 from three representative CNN models, AlexNet, ResNet, and InceptionV4, according to the cross-entropy loss (CEL). We subsequently developed InceptionV4 to fuse the information from qualified pretreatment MRI data and patient clinical factors. Radiomic information was evaluated based on several constant sequences, including enhanced T1-weighted sequences (with arterial, portal, and delayed phases), T2 FSE sequences, and dual-echo sequences. The performance of InceptionV4 was cross-validated in the training cohort (n = 127) and internally validated in an independent cohort (n = 64), with comparisons against single important clinical factors and radiologists in terms of receiver operating characteristic (ROC) curves. Class activation mapping was used to visualize the InceptionV4 model. The InceptionV4 model achieved an AUC of 0.871 (95% confidence interval [CI] 0.761-0.981) in the cross-validation cohort and an AUC of 0.826 (95% CI 0.682-0.970) in the internal validation cohort; these two models performed better than did the other methods (AUC ranges 0.783-0.873 and 0.708-0.806 for cross- and internal validations, respectively; P < 0.01). The present InceptionV4 model, which integrates radiomic information and clinical factors, helps predict the treatment response of unresectable HCC patients receiving HAIC treatment.

Transfer learning graph representations of molecules for pKa, 13C-NMR, and solubility

We explore transfer learning models from a pre-trained graph convolutional neural network representation of molecules, obtained from SchNet, to predict 13C-NMR, pKa, and log S solubility. SchNet learns a graph representation of a molecule by associating each atom with an “embedding vector” and interacts the atom-embeddings with each other by leveraging graph convolutional filters on their interatomic distances. We pre-trained SchNet on molecular energy and demonstrate that the pre-trained atomistic embeddings can then be used as a transferable representation for a wide array of properties. On the one hand, for atomic properties such as micro-pK1 and 13C-NMR, we investigate two models, one linear and one neural net, that input pre-trained atom-embeddings of a particular atom (e.g. carbon) and predict a local property (e.g., 13C-NMR). On the other hand, for molecular properties such as solubility, a size-extensive graph model is built using the embeddings of all atoms in the molecule as input. For all cases, qualitatively correct predictions are made with relatively little training data (&lt;1000 training points), showcasing the ease with which pre-trained embeddings pick up on important chemical patterns. The proposed models successfully capture well-understood trends of pK1 and solubility. This study advances our understanding of current neural net graph representations and their capacity for transfer learning applications in chemistry.

Global geometry of chemical graph neural network representations in terms of chemical moieties

The embedding vectors from a Graph Neural Network trained on quantum chemical data allow for a global geometric space with a Euclidean distance metric. Moieties that are close in chemical sense, are also close in Euclidean sense.

Learning image representations for anomaly detection: Application to discovery of histological alterations in drug development

We present a system for anomaly detection in histopathological images. In histology, normal samples are usually abundant, whereas anomalous (pathological) cases are scarce or not available. Under such settings, one-class classifiers trained on healthy data can detect out-of-distribution anomalous samples. Such approaches combined with pre-trained Convolutional Neural Network (CNN) representations of images were previously employed for anomaly detection (AD). However, pre-trained off-the-shelf CNN representations may not be sensitive to abnormal conditions in tissues, while natural variations of healthy tissue may result in distant representations. To adapt representations to relevant details in healthy tissue we propose training a CNN on an auxiliary task that discriminates healthy tissue of different species, organs, and staining reagents. Almost no additional labeling workload is required, since healthy samples come automatically with aforementioned labels. During training we enforce compact image representations with a center-loss term, which further improves representations for AD. The proposed system outperforms established AD methods on a published dataset of liver anomalies. Moreover, it provided comparable results to conventional methods specifically tailored for quantification of liver anomalies. We show that our approach can be used for toxicity assessment of candidate drugs at early development stages and thereby may reduce expensive late-stage drug attrition.

SCA-Former: transformer-like network based on stream-cross attention for medical image segmentation

Objective. Deep convolutional neural networks (CNNs) have been widely applied in medical image analysis and achieved satisfactory performances. While most CNN-based methods exhibit strong feature representation capabilities, they face challenges in encoding long-range interaction information due to the limited receptive fields. Recently, the Transformer has been proposed to alleviate this issue, but its cost is greatly enlarging the model size, which may inhibit its promotion. Approach. To take strong long-range interaction modeling ability and small model size into account simultaneously, we propose a Transformer-like block-based U-shaped network for medical image segmentation, dubbed as SCA-Former. Furthermore, we propose a novel stream-cross attention (SCA) module to enforce the network to focus on finding a balance between local and global representations by extracting multi-scale and interactive features along spatial and channel dimensions. And SCA can effectively extract channel, multi-scale spatial, and long-range information for a more comprehensive feature representation. Main results. Experimental results demonstrate that SCA-Former outperforms the current state-of-the-art (SOTA) methods on three public datasets, including GLAS, ISIC 2017 and LUNG. Significance. This work exhibits a promising method to enhance the feature representation of convolutional neural networks and improve segmentation performance.

On convolutional neural networks for selection inference: Revealing the effect of preprocessing on model learning and the capacity to discover novel patterns.

A central challenge in population genetics is the detection of genomic footprints of selection. As machine learning tools including convolutional neural networks (CNNs) have become more sophisticated and applied more broadly, these provide a logical next step for increasing our power to learn and detect such patterns; indeed, CNNs trained on simulated genome sequences have recently been shown to be highly effective at this task. Unlike previous approaches, which rely upon human-crafted summary statistics, these methods are able to be applied directly to raw genomic data, allowing them to potentially learn new signatures that, if well-understood, could improve the current theory surrounding selective sweeps. Towards this end, we examine a representative CNN from the literature, paring it down to the minimal complexity needed to maintain comparable performance; this low-complexity CNN allows us to directly interpret the learned evolutionary signatures. We then validate these patterns in more complex models using metrics that evaluate feature importance. Our findings reveal that preprocessing steps, which determine how the population genetic data is presented to the model, play a central role in the learned prediction method. This results in models that mimic previously-defined summary statistics; in one case, the summary statistic itself achieves similarly high accuracy. For evolutionary processes that are less well understood than selective sweeps, we hope this provides an initial framework for using CNNs in ways that go beyond simply achieving high classification performance. Instead, we propose that CNNs might be useful as tools for learning novel patterns that can translate to easy-to-implement summary statistics available to a wider community of researchers.

HAU-Net: Hybrid CNN-transformer for breast ultrasound image segmentation

Breast cancer is a significant health concern that remains one of the leading causes of mortality in women worldwide. Convolutional Neural Networks (CNNs) have been shown to be effective in ultrasound breast image segmentation. Yet, because of the lack of long-distance dependence, the segmentation performance of CNNs is limited in addressing challenges typical of segmentation of ultrasound breast lesions, such as similar intensity distributions, the presence of irregular objects, and blurred boundaries. In order to overcome these issues, several studies have combined transformers and CNNs, to compensate for the shortcomings of CNNs with the ability of transformers to exploit long-distance dependence. Most of these studies limited themselves to rigidly plug transformer blocks into the CNN, lacking consistency in the process of feature extraction and therefore leading to poor performances in segmenting challenging medical images. In this paper, we propose HAU-Net(hierarchical attention-guided U-Net), a hybrid CNN-transformer framework that benefits from both the long-range dependency of transformers and the local detail representation of CNNs. To incorporate global context information, we introduce a L-G transformer block nested into the skip connections of the U shape architecture network. In addition, to further improve the segmentation performance, we added a cross attention block (CAB) module on the decoder side to allow different layers to interact. Extensive experimental results on three public datasets indicate that the proposed HAU-Net can achieve better performance than other state-of-the-art methods for breast lesions segmentation, with Dice coefficient of 83.11% for BUSI, 88.73% for UDIAT, and 89.48% for BLUI respectively.

Continuous Cartesian Genetic Programming based representation for multi-objective neural architecture search

We propose a novel neural architecture search (NAS) approach for the challenge of designing convolutional neural networks (CNNs) that achieve a good tradeoff between complexity and accuracy. We rely on Cartesian genetic programming (CGP) and integrated real-based and block-chained CNN representation, for optimization using multi-objective evolutionary algorithms (MOEAs) in the continuous domain. We introduce two variants, CGP-NASV1 and CGP-NASV2, which differ in the granularity of their respective search spaces. To evaluate the proposed algorithms, we utilized the non-dominated sorting genetic algorithm II (NSGA-II) on the CIFAR-10, CIFAR-100,and SVHN datasets. Additionally, we extended the empirical analysis while maintaining the same solution representation to assess other searching techniques such as differential evolution (DE), the multi-objective evolutionary algorithm based on decomposition (MOEA/D), and the S metric selection evolutionary multi-objective algorithm (SMS-EMOA). The experimental results demonstrate that our approach exhibits competitive classification performance and model complexity compared to state-of-the-art methods.

A novel feature-scrambling approach reveals the capacity of convolutional neural networks to learn spatial relations

Convolutional neural networks (CNNs) are one of the most successful computer vision systems to solve object recognition. Furthermore, CNNs have major applications in understanding the nature of visual representations in the human brain. Yet it remains poorly understood how CNNs actually make their decisions, what the nature of their internal representations is, and how their recognition strategies differ from humans. Specifically, there is a major debate about the question of whether CNNs primarily rely on surface regularities of objects, or whether they are capable of exploiting the spatial arrangement of features, similar to humans. Here, we develop a novel feature-scrambling approach to explicitly test whether CNNs use the spatial arrangement of features (i.e. object parts) to classify objects. We combine this approach with a systematic manipulation of effective receptive field sizes of CNNs as well as minimal recognizable configurations (MIRCs) analysis. In contrast to much previous literature, we provide evidence that CNNs are in fact capable of using relatively long-range spatial relationships for object classification. Moreover, the extent to which CNNs use spatial relationships depends heavily on the dataset, e.g. texture vs. sketch. In fact, CNNs even use different strategies for different classes within heterogeneous datasets (ImageNet), suggesting CNNs have a continuous spectrum of classification strategies. Finally, we show that CNNs learn the spatial arrangement of features only up to an intermediate level of granularity, which suggests that intermediate rather than global shape features provide the optimal trade-off between sensitivity and specificity in object classification. These results provide novel insights into the nature of CNN representations and the extent to which they rely on the spatial arrangement of features for object classification.

Scene context is predictive of unconstrained object similarity judgments

What makes objects alike in the human mind? Computational approaches for characterizing object similarity have largely focused on the visual forms of objects or their linguistic associations. However, intuitive notions of object similarity may depend heavily on contextual reasoning—that is, objects may be grouped together in the mind if they occur in the context of similar scenes or events. Using large-scale analyses of natural scene statistics and human behavior, we found that a computational model of the associations between objects and their scene contexts is strongly predictive of how humans spontaneously group objects by similarity. Specifically, we learned contextual prototypes for a diverse set of object categories by taking the average response of a convolutional neural network (CNN) to the scene contexts in which the objects typically occurred. In behavioral experiments, we found that contextual prototypes were strongly predictive of human similarity judgments for a large set of objects and rivaled the performance of models based on CNN representations of the objects themselves or word embeddings for their names. Together, our findings reveal the remarkable degree to which the natural statistics of context predict commonsense notions of object similarity.

Understanding contributing neurons via attribution visualization

Understanding contributing neuron features is crucial to explaining convolutional neural network (CNN) decisions. The attribution research provides an effective way to detect contributing neuron features and numerically assign them attribution scores. However, a method to clearly and intuitively represent the implications hidden in neuron attributions is lacking. Attribution scores show the numerical importance of contributing neurons, but the meanings implied by these numerical scores are still not available. To mitigate this gap, we propose an optimization-based visualization method named attribution visualization, which enables an intuitive understanding of neuron attributions. Our approach is distinguished from existing visualization methods by its ability to produce noise-free result, i.e., the ability to remove irrelevant regions from visualizations. We achieve this by introducing an optimizable mask into the visualization process and designing an objective function that simultaneously optimizes the area-constrained mask and visualization. Furthermore, we propose the fractal noise pyramid with diverse and natural frequency spectra as our mask perturbation technique which is key to removing unrelated noise in visualization. We implement several comparisons and user studies with other visual explanations to demonstrate the unique properties of our attribution visualization. We also apply our attribution visualization on two representative CNNs, showcasing its ability to intuitively understand contributing neuron features.

Tree Species Classification in UAV Remote Sensing Images Based on Super-Resolution Reconstruction and Deep Learning

We studied the use of self-attention mechanism networks (SAN) and convolutional neural networks (CNNs) for forest tree species classification using unmanned aerial vehicle (UAV) remote sensing imagery in Dongtai Forest Farm, Jiangsu Province, China. We trained and validated representative CNN models, such as ResNet and ConvNeXt, as well as the SAN model, which incorporates Transformer models such as Swin Transformer and Vision Transformer (ViT). Our goal was to compare and evaluate the performance and accuracy of these networks when used in parallel. Due to various factors, such as noise, motion blur, and atmospheric scattering, the quality of low-altitude aerial images may be compromised, resulting in indistinct tree crown edges and deficient texture. To address these issues, we adopted Real-ESRGAN technology for image super-resolution reconstruction. Our results showed that the image dataset after reconstruction improved classification accuracy for both the CNN and Transformer models. The final classification accuracies, validated by ResNet, ConvNeXt, ViT, and Swin Transformer, were 96.71%, 98.70%, 97.88%, and 98.59%, respectively, with corresponding improvements of 1.39%, 1.53%, 0.47%, and 1.18%. Our study highlights the potential benefits of Transformer and CNN for forest tree species classification and the importance of addressing the image quality degradation issues in low-altitude aerial images.

FaceFormer: Aggregating Global and Local Representation for Face Hallucination

Recently, face hallucination methods either feed whole face image into convolutional neural networks (CNNs) or utilize extra facial priors (e.g., facial parsing maps and landmarks) to focus on global facial structure and constrain facial texture generation. However, the limited receptive fields of CNNs and inaccurate facial priors will reduce the naturalness and fidelity of restored face. In this paper, we propose a FaceFormer that aggregates global representation of Transformers and local representation of CNNs to maintain the consistency of facial structure while restoring local facial details. The reason for this design is that the Transformer can capture global facial information by exploiting the long-distance visual relation modeling, while the local modeling capability of CNNs can recover fine-grained facial details. Therefore, aggregating these two independent representations can help to maximize their merits and reconstruct high-quality and high-fidelity face images. Experimental results of face reconstruction and recognition verify that the proposed FaceFormer significantly outperforms current state-of-the-arts.

Evaluation and Comparison of Semantic Segmentation Networks for Rice Identification Based on Sentinel-2 Imagery

Efficient and accurate rice identification based on high spatial and temporal resolution remote sensing imagery is essential for achieving precision agriculture and ensuring food security. Semantic segmentation networks in deep learning are an effective solution for crop identification, and they are mainly based on two architectures: the commonly used convolutional neural network (CNN) architecture and the novel Vision Transformer architecture. Research on crop identification from remote sensing imagery using Vision Transformer has only emerged in recent times, mostly in sub-meter resolution or even higher resolution imagery. Sub-meter resolution images are not suitable for large scale crop identification as they are difficult to obtain. Therefore, studying and analyzing the differences between Vision Transformer and CNN in crop identification in the meter resolution images can validate the generalizability of Vision Transformer and provide new ideas for model selection in crop identification research at large scale. This paper compares the performance of two representative CNN networks (U-Net and DeepLab v3) and a novel Vision Transformer network (Swin Transformer) on rice identification in Sentinel-2 of 10 m resolution. The results show that the three networks have different characteristics: (1) Swin Transformer has the highest rice identification accuracy and good farmland boundary segmentation ability. Although Swin Transformer has the largest number of model parameters, the training time is shorter than DeepLab v3, indicating that Swin Transformer has good computational efficiency. (2) DeepLab v3 also has good accuracy in rice identification. However, the boundaries of the rice fields identified by DeepLab v3 tend to shift towards the upper left corner. (3) U-Net takes the shortest time for both training and prediction and is able to segment the farmland boundaries accurately for correctly identified rice fields. However, U-Net’s accuracy of rice identification is lowest, and rice is easily confused with soybean, corn, sweet potato and cotton in the prediction. The results reveal that the Vision Transformer network has great potential for identifying crops at the country or even global scale.

PiTLiD: Identification of Plant Disease From Leaf Images Based on Convolutional Neural Network.

With the development of plant phenomics, the identification of plant diseases from leaf images has become an effective and economic approach in plant disease science. Among the methods of plant diseases identification, the convolutional neural network (CNN) is the most popular one for its superior performance. However, CNN's representation power is still a challenge in dealing with small datasets, which greatly affects its popularization. In this work, we propose a new method, namely PiTLiD, based on pretrained Inception-V3 convolutional neural network and transfer learning to identify plant leaf diseases from phenotype data of plant leaf with small sample size. To evaluate the robustness of the proposed method, the experiments on several datasets with small-scale samples were implemented. The results show that PiTLiD performs better than compared methods. This study provides a plant disease identification tool based on a deep learning algorithm for plant phenomics. All the source data and code are accessible at https://github.com/zhanglab-wbgcas/PiTLiD.

Explainable Convolutional Neural Networks: A Taxonomy, Review, and Future Directions

Convolutional neural networks (CNNs) have shown promising results and have outperformed classical machine learning techniques in tasks such as image classification and object recognition. Their human-brain like structure enabled them to learn sophisticated features while passing images through their layers. However, their lack of explainability led to the demand for interpretations to justify their predictions. Research on Explainable AI or XAI has gained momentum to provide knowledge and insights into neural networks. This study summarizes the literature to gain more understanding of explainability in CNNs (i.e., Explainable Convolutional Neural Networks). We classify models that made efforts to improve the CNNs interpretation. We present and discuss taxonomies for XAI models that modify CNN architecture, simplify CNN representations, analyze feature relevance, and visualize interpretations. We review various metrics used to evaluate XAI interpretations. In addition, we discuss the applications and tasks of XAI models. This focused and extensive survey develops a perspective on this area by addressing suggestions for overcoming XAI interpretation challenges, like models’ generalization, unifying evaluation criteria, building robust models, and providing interpretations with semantic descriptions. Our taxonomy can be a reference to motivate future research in interpreting neural networks.

A Deep Multi-level Attentive Network for Multimodal Sentiment Analysis

Multimodal sentiment analysis has attracted increasing attention with broad application prospects. Most of the existing methods have focused on a single modality, which fails to handle social media data due to its multiple modalities. Moreover, in multimodal learning, most of the works have focused on simply combining the two modalities without exploring the complicated correlations between them. This resulted in dissatisfying performance for multimodal sentiment classification. Motivated by the status quo, we propose a Deep Multi-level Attentive network (DMLANet), which exploits the correlation between image and text modalities to improve multimodal learning. Specifically, we generate the bi-attentive visual map along the spatial and channel dimensions to magnify Convolutional neural network representation power. Then, we model the correlation between the image regions and semantics of the word by extracting the textual features related to the bi-attentive visual features by applying semantic attention. Finally, self-attention is employed to fetch the sentiment-rich multimodal features for the classification automatically. We conduct extensive evaluations on four real-world datasets, namely, MVSA-Single, MVSA-Multiple, Flickr, and Getty Images, which verify our method's superiority.

Fusion of Quality Evaluation Metrics and Convolutional Neural Network Representations for ROI Filtering in LC-MS.

Region of interest (ROI) extraction is a fundamental step in analyzing metabolomic datasets acquired by liquid chromatography-mass spectrometry (LC-MS). However, noises and backgrounds in LC-MS data often affect the quality of extracted ROIs. Therefore, developing effective ROI evaluation algorithms is necessary to eliminate false positives meanwhile keep the false-negative rate as low as possible. In this study, a deep fused filter of ROIs (dffROI) was proposed to improve the accuracy of ROI extraction by combining the handcrafted evaluation metrics with convolutional neural network (CNN)-learned representations. To evaluate the performance of dffROI, dffROI was compared with peakonly (CNN-learned representation) and five handcrafted metrics on three LC-MS datasets and a gas chromatography-mass spectrometry (GC-MS) dataset. Results show that dffROI can achieve higher accuracy, better true-positive rate, and lower false-positive rate. Its accuracy, true-positive rate, and false-positive rate are 0.9841, 0.9869, and 0.0186 on the test set, respectively. The classification error rate of dffROI (1.59%) is significantly reduced compared with peakonly (2.73%). The model-agnostic feature importance demonstrates the necessity of fusing handcrafted evaluation metrics with the convolutional neural network representations. dffROI is an automatic, robust, and universal method for ROI filtering by virtue of information fusion and end-to-end learning. It is implemented in Python programming language and open-sourced at https://github.com/zhanghailiangcsu/dffROI under BSD License. Furthermore, it has been integrated into the KPIC2 framework previously proposed by our group to facilitate real metabolomic LC-MS dataset analysis.