Intermediate Convolutional Layers Research Articles

A deep multi-branch attention model for histopathological breast cancer image classification

Since the impressive superior performance demonstrated by deep learning methods is widely used in histopathological image analysis and diagnosis, existing work cannot fully extract the information in the breast cancer images due to the limited high resolution of histopathological images. In this study, we construct a novel intermediate layer structure that fully extracts feature information and name it DMBANet, which can extract as much feature information as possible from the input image by up-dimensioning the intermediate convolutional layers to improve the performance of the network. Furthermore, we employ the depth-separable convolution method on the Spindle Structure by decoupling the intermediate convolutional layers and convolving them separately, to significantly reduce the number of parameters and computation of the Spindle Structure and improve the overall network operation speed. We also design the Spindle Structure as a multi-branch model and add different attention mechanisms to different branches. Spindle Structure can effectively improve the performance of the network, the branches with added attention can extract richer and more focused feature information, and the branch with residual connections can minimize the degradation phenomenon in our network and speed up network optimization. The comprehensive experiment shows the superior performance of DMBANet compared to the state-of-the-art method, achieving about 98% classification accuracy, which is better than existing methods. The code is available at https://github.com/Nagi-Dr/DMBANet-main.

Towards deep understanding of graph convolutional networks for relation extraction

Relation extraction aims at identifying semantic relations between pairs of named entities from unstructured texts and is considered an essential prerequisite for many downstream tasks in natural language processing (NLP). Owing to the ability in expressing complex relationships and interdependency, graph neural networks (GNNs) have been gradually used to solve the relation extraction problem and have achieved state-of-the-art results. However, the designs of GNN-based relation extraction methods are mostly based on empirical intuition, heuristic, and experimental trial-and-error. A clear understanding of why and how GNNs perform well in relation extraction tasks is lacking. In this study, we investigate three well-known GNN-based relation extraction models, CGCN, AGGCN, and SGCN, and aim to understand the underlying mechanisms of the extractions. In particular, we provide a visual analytic to reveal the dynamics of the models and provide insight into the function of intermediate convolutional layers. We determine that entities, particularly subjects and objects in them, are more important features than other words for relation extraction tasks. With various masking strategies, the significance of entity type to relation extraction is recognized. Then, from the perspective of the model architecture, we find that graph structure modeling and aggregation mechanisms in GCN do not significantly affect the performance improvement of GCN-based relation extraction models. The above findings are of great significance in promoting the development of GNNs. Based on these findings, an engineering oriented MLP-based GNN relation extraction model is proposed to achieve a comparable performance and greater efficiency.

Study on pruning optimization based on HRank pruning method

Convolutional neural network (CNN), as one of the most important pillars of deep learning technology, has been paid increased attention to by researchers. However, by the expansion of the application of convolutional neural network, it faces more and more complex problems and more and more diverse situations. In order to solve these problems more appropriately, the number of parameters of convolutional neural networks continuously increases. This limits its potential to be deployed to distal devices with relatively low computing power and memory space. To improve this situation, this paper studies the compression of convolutional neural network model with acceptable sacrifice of accuracy. Based on Pytorch, Resnet-56 on CIFAR-10 is pruned. By using HRank pruning method, convolution kernels of each convolution layers are ordered according to the size of their determinant rank. Through deleting low-rank, less importance convolution kernels and reserving high-rank, more important convolution kernels, the pruning purpose is achieved. By changing the default compression ratio of different convolution layer, more important ones are applied with lower compression rates and less important ones are applied with higher compression rates, the paper achieves higher compression rate with the acceptable loss of accuracy. At the same time, the paper tests the efficiency of pruning under different learning rates. Finally, the paper finds that when the preset compression rate of the intermediate convolutional layer is changed slightly, the accuracy maintains at the level of 92.720% but the compress rate of Params reaches to 44.7% with 0.47M left and the compress rate of Flops reaches 51.1% with 61.39M left. In addition, the paper finds that when the learning rate is 0.05, the learning efficiency reaches its optimum.

Encoding of speech in convolutional layers and the brain stem based on language experience

Comparing artificial neural networks with outputs of neuroimaging techniques has recently seen substantial advances in (computer) vision and text-based language models. Here, we propose a framework to compare biological and artificial neural computations of spoken language representations and propose several new challenges to this paradigm. The proposed technique is based on a similar principle that underlies electroencephalography (EEG): averaging of neural (artificial or biological) activity across neurons in the time domain, and allows to compare encoding of any acoustic property in the brain and in intermediate convolutional layers of an artificial neural network. Our approach allows a direct comparison of responses to a phonetic property in the brain and in deep neural networks that requires no linear transformations between the signals. We argue that the brain stem response (cABR) and the response in intermediate convolutional layers to the exact same stimulus are highly similar without applying any transformations, and we quantify this observation. The proposed technique not only reveals similarities, but also allows for analysis of the encoding of actual acoustic properties in the two signals: we compare peak latency (i) in cABR relative to the stimulus in the brain stem and in (ii) intermediate convolutional layers relative to the input/output in deep convolutional networks. We also examine and compare the effect of prior language exposure on the peak latency in cABR and in intermediate convolutional layers. Substantial similarities in peak latency encoding between the human brain and intermediate convolutional networks emerge based on results from eight trained networks (including a replication experiment). The proposed technique can be used to compare encoding between the human brain and intermediate convolutional layers for any acoustic property and for other neuroimaging techniques.

Dually attentive multiscale networks for health state recognition of rotating machinery

Recent advances in convolutional neural networks (CNN) have boosted the research on reliability monitoring of rotating machinery. In actual industry production, the mechanical equipment often operates under variable speed and strong noise conditions, so the discriminative fault-related features of the collected vibration signals are easily buried by interference information. Thus, it poses a huge challenge for the existing CNN models to achieve favorable diagnostic results. To address this issue, we put forward a dually attentive multiscale network (DAMN) for mechanical fault diagnosis. To begin with, a new hierarchical structure is built to make full use of the features from intermediate convolutional layers. Then, to explore abundant discriminative information from mechanical signals, a dually attentive multiscale module (DAMM) is introduced to guide the CNN model to extract multiscale and multilevel features. Further, a feature reinforcement module (FRM) is designed to expand receptive field and filter out unrelated interference information. Finally, embarking on the above improvements, an end-to-end CNN model named DAMN is built for intelligent fault diagnosis. Experimental results show that DAMN outperforms seven state-of-the-art methods for health recognition of rotating machinery.

Interpreting Intermediate Convolutional Layers of Generative CNNs Trained on Waveforms

This paper presents a technique to interpret and visualize intermediate layers in generative CNNs trained on raw speech data in an unsupervised manner. We argue that averaging over feature maps after ReLU activation in each transpose convolutional layer yields interpretable time-series data. This technique allows for acoustic analysis of intermediate layers that parallels the acoustic analysis of human speech data: we can extract F0, intensity, duration, formants, and other acoustic properties from intermediate layers in order to test where and how CNNs encode various types of information. We further combine this technique with linear interpolation of a model's latent space to show a causal relationship between individual variables in the latent space and activations in a model's intermediate convolutional layers. In particular, observing the causal effect between linear interpolation and the resulting changes in intermediate layers can reveal how individual latent variables get transformed into spikes in activation in intermediate layers. We train and probe internal representations of two models -- a bare WaveGAN architecture and a ciwGAN extension which forces the Generator to output informative data and results in the emergence of linguistically meaningful representations. Interpretation and visualization is performed for three basic acoustic properties of speech: periodic vibration (corresponding to vowels), aperiodic noise vibration (corresponding to fricatives), and silence (corresponding to stops). The proposal also allows testing of higher-level morphophonological alternations such as reduplication (copying). In short, using the proposed technique, we can analyze how linguistically meaningful units in speech get encoded in each convolutional layer of a generative neural network.

Malaria detection through digital microscopic imaging using Deep Greedy Network with transfer learning.

Purpose: In conventional diagnosis, the visual inspection of the malaria parasite Plasmodium falciparum in infected red blood cells under a microscope, is done manually by pathologists, which is both laborious and error-prone. Recent studies on automating this process have been conducted using artificial intelligence and feature selection of positional and morphological features from blood smear cell images using convolutional neural network (CNN). However, most deep CNN models do not perform well as per the expectation on small datasets. Approach: In this context, we propose a comprehensive computer-aided diagnosis scheme for automating the detection of malaria parasites in thin blood smear images using deep CNN, where transfer learning is used for optimizing the feature selection process. As an extra layer of security, layer embeddings are extracted from the intermediate convolutional layers using the feature matrix to cross-check the selection of features in the intermediate layers. The proposal includes the utilization of the ResNet 152 model integrated with the Deep Greedy Network for training, which produces an enhanced quality of prediction. Results: The performance of the proposed hybrid model has been evaluated concerning the evaluation metrics such as accuracy, precision, recall, specificity, and F1-score, which has been further compared with the pre-existing deep learning algorithms. Conclusions: The comparative analysis of the results reported based on the accuracy metrics demonstrates promising outcomes concerning the other models. Lastly, the embedding extraction from the intermediate hidden layers and their visual analysis also provides an opportunity for manual verification of the performance of the trained model.

Multilevel Capsule Weighted Aggregation Network Based on a Decoupled Dynamic Filter for Remote Sensing Scene Classification

The improvement of remote sensing scene classification(RSSC) by effectively extracting discriminant representations for complex and diverse scenes remains a challenging task. The capsule network(CapsNet) can encode the spatial relationship of features in an image, which exhibits encouraging performance. Nevertheless, the original CapsNet is unsuitable for RSSC with complex image background. In addition, conventional neural network methods only use the last features from their last convolutional layer and discount the intermediate features with complementary information. To search the additional information in intermediate convolutional layers and increase the performance of feature aggregation, this paper proposes a multilevel capsule weighted aggregation network (MCWANet) based on a decoupled dynamic filter(DDF), in which a new multilevel capsule encoding module and a new capsule sorting pooling (CSPool) method are implemented by combining the advantageous attributes of a residual DDF block, weighted capsule aggregation, and the new CSPool method. Extensive experiments on two challenging datasets, AID and NWPU-RESISC45, demonstrate that multilevel and multiscale features can be extracted and fused to extract semantically strong feature representation and that the proposed MCWANet performs competitively in RSSC.

AffectiveNet: Affective-Motion Feature Learning for Microexpression Recognition

Microexpressions are hard to spot due to fleeting and involuntary moments of facial muscles. Interpretation of microemotions from video clips is a challenging task. In this article, we propose an affective-motion imaging that cumulates rapid and short-lived variational information of microexpressions into a single response. Moreover, we have proposed an AffectiveNet: Affective-motion feature learning network that can perceive subtle changes and learns the most discriminative dynamic features to describe the emotion classes. The AffectiveNet holds two blocks: MICRoFeat and MFL block. MICRoFeat block conserves the scale-invariant features, which allows network to capture both coarse and tiny edge variations. Whereas, the MFL block learns microlevel dynamic variations from two different intermediate convolutional layers. Effectiveness of the proposed network is tested over four datasets by using two experimental setups: person independent and cross dataset validation. The experimental results of the proposed network outperform the state-of-the-art approaches with significant margin for MER approaches.

Visual complexity analysis using deep intermediate-layer features

In this paper, we focus on visual complexity, an image attribute that humans can subjectively evaluate based on the level of details in the image. We explore unsupervised information extraction from intermediate convolutional layers of deep neural networks to measure visual complexity. We derive an activation energy metric that combines convolutional layer activations to quantify visual complexity. To show the effectiveness of our proposed metric for various applications, we introduce Savoias, a visual complexity dataset that compromises of more than 1,400 images from seven diverse image categories (e.g., advertisement and interior design). We demonstrate high correlations of our deep neural network-based measure of visual complexity with human-curated ground-truth (GT) scores on various widely used network architectures, e.g., VGG16, ResNet-v2-152, and EfficientNet, and in networks trained on two classification tasks (object and scene classification). This result reveals that intermediate convolutional layers of deep neural networks carry information about the complexity of images that is meaningful to people. Furthermore, we show that our method of measuring visual complexity outperforms traditional methods on Savoias and two other state-of-the-art benchmark datasets. Moreover, we perform extensive analysis on the performance difference between our unsupervised method and a supervised method trained on the feature map, and show that by supervision, we can improve the prediction. Finally, we demonstrate that, within the context of a category, visually more complex images are also more memorable to human observers.

Event-Oriented 3D Convolutional Features Selection and Hash Codes Generation Using PCA for Video Retrieval

The extensive video surveillance networks gather an enormous amount of data exponentially on a daily basis and its management is a challenging task, requiring efficient and effective techniques for searching, indexing, and retrieval. The employed mainstream techniques are focusing on general category videos, where the important events in surveillance require fine-grained events retrieval. In this paper, we introduce an event-oriented feature selection mechanism by utilizing the intermediate convolutional layer of a pre-trained 3D-CNN model, that is selected after deep investigation of its weights and response to a particular event. The extracted exclusive features represent an event semantically and effectively eliminate those neurons which do not respond to an event. Furthermore, the event-oriented convolutional features are of very high-dimensions, requiring additional storage, and take more time in feature comparison for retrieval. Therefore, we generate compact binary codes from these features using principle component analysis (PCA) algorithm. This makes our system more efficient to retrieve videos from large scale database. We evaluated our approach on the challenging events of UCF101 and HMDB51 datasets for original features and generated compact codes to achieve reduced execution time and better precision and recall scores.

Abstract 3351: Overall survival prediction of glioblastoma patients combining clinical factors with texture features extracted from 3-D convolutional neural networks

Abstract Objective: We aim to predict overall survival (OS) in Glioblastoma treated with gross total resection (GTR) using pre-operative MRI images. Methods: A cohort of 87 GBM patients (59 patients for training and 28 patients for validation) who underwent GTR was analyzed using multi-institutional data from brain tumor segmentation (BraTS) challenge 2018 (Menze BH, et al. TMI 2015; Bakas S, et al. NSD 2017). Each patient consisted of a series of pre-surgical MR images including T1 pre contrast, T1-Contrast Enhanced, T2 and Flair images. A group of experienced radiologists delineated edema, tumor core and enhanced tumor for each testing patient using these image sequences. A 2D U-net was trained to segment these structures on the validation cohort. A 3D CNN model with orthogonalized random filters was used to learn images features from the three segmented subregions including texture, size, location, etc. Global maximum pooling was performed on intermediate convolutional layers to obtain representative image features for each patient. Since mid-term survivors (6-18 months) outnumbered short (<6 months) and long term (>18 months) survivors for a large margin, the MR images from both short and long term survivors were augmented with random rotations to balance the number of patients among three cohorts of patients. The extracted image features was then fed into an RBF-kernel based L-2 norm regression algorithm (Huang GB, IEEE SMC, 2012) to predict patient’s OS. Results: The average [standard deviation] of dice similarity coefficient (DSC) for the whole tumor, enhanced tumor, and tumor core contours were 0.882[0.080], 0.712 [0.294], and 0.769 [0.263], respectively for the validation cohort. Parameters of regression algorithm was optimized using leave one out cross validation. One convolutional layer was used in the CNN archietecture due to limited training samples. The model performance deteriorated when using deeper layers. The best architecture to classify patients into short, mid and long term survivors was one convolutional layer with 30 filters. The prediction accuracy was 64.3%, and the spearman’r sank correlation was 0.395. The model performance was slightly improved by including clinical factors such as age, tumor location, ratio of the whole tumor size to the entire brain etc. The Spearman’s rank correlation coefficient was increased to 0.432 while the accuracy maintained the same. Conclusions: We developed a 3D CNN model followd with kernel regression to extract image features from pre-operative MR images and predict OS of GBM patients after GTR. These signatures have shown potential values as biomarkers to predict OS. Citation Format: Weiwei Zong, Joon Lee, Chang Liu, James Snyder, Ning Wen. Overall survival prediction of glioblastoma patients combining clinical factors with texture features extracted from 3-D convolutional neural networks [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2019; 2019 Mar 29-Apr 3; Atlanta, GA. Philadelphia (PA): AACR; Cancer Res 2019;79(13 Suppl):Abstract nr 3351.

Bi-stream CNN Down Syndrome screening model based on genotyping array

BackgroundHuman Down syndrome (DS) is usually caused by genomic micro-duplications and dosage imbalances of human chromosome 21. It is associated with many genomic and phenotype abnormalities. Even though human DS occurs about 1 per 1,000 births worldwide, which is a very high rate, researchers haven’t found any effective method to cure DS. Currently, the most efficient ways of human DS prevention are screening and early detection.MethodsIn this study, we used deep learning techniques and analyzed a set of Illumina genotyping array data. We built a bi-stream convolutional neural networks model to screen/predict the occurrence of DS. Firstly, we built image input data by converting the intensities of each SNP site into chromosome SNP maps. Next, we proposed a bi-stream convolutional neural network (CNN) architecture with nine layers and two branch models. We further merged two CNN branch models into one model in the fourth convolutional layer, and output the prediction in the last layer.ResultsOur bi-stream CNN model achieved 99.3% average accuracies, and very low false-positive and false-negative rates, which was necessary for further applications in disease prediction and medical practice. We further visualized the feature maps and learned filters from intermediate convolutional layers, which showed the genomic patterns and correlated SNPs variations in human DS genomes. We also compared our methods with other CNN and traditional machine learning models. We further analyzed and discussed the characteristics and strengths of our bi-stream CNN model.ConclusionsOur bi-stream model used two branch CNN models to learn the local genome features and regional patterns among adjacent genes and SNP sites from two chromosomes simultaneously. It achieved the best performance in all evaluating metrics when compared with two single-stream CNN models and three traditional machine-learning algorithms. The visualized feature maps also provided opportunities to study the genomic markers and pathway components associated with Human DS, which provided insights for gene therapy and genomic medicine developments.

Weakly supervised semantic segmentation using distinct class specific saliency maps

Weakly supervised segmentation has drawn considerable attention, because of the high costs associated with the creation of pixel-wise annotated image datasets that are used for training fully supervised segmentation models. We propose a weakly supervised semantic segmentation method using CNN-based class-specific saliency maps and fully connected CRF. To obtain distinct class-specific saliency maps (DCSM) that can be used as unary potentials of CRF, we propose a novel method of estimating class saliency maps, which significantly improves the method proposed by Simonyan et al. (2014) through the following improvements: (1) using CNN derivatives with respect to feature maps of the intermediate convolutional layers with up-sampling instead of an input image; (2) subtracting the saliency maps of other classes from the saliency maps of the target class to differentiate target objects among other objects; (3) aggregating multiple-scale class saliency maps to compensate for the low resolution in feature maps. In addition, we propose the use of a novel algorithm for estimating segmentation “Easiness” combined with the proposed saliency-based method. Wei et al. (2016) recently demonstrated that a fully supervised segmentation model enhanced the performance of weakly supervised segmentation by training the model using the estimated initial masks in a weakly supervised setting. However, the initial estimated masks tend to include some noise, which sometimes produces erroneous results. Therefore, we focus on improving the quality of the initial estimated masks for training a fully supervised segmentation model. We propose a method for retrieving “good seeds” by predicting the segmentation “Easiness” of images based on the consistency of the outputs under different conditions. We illustrate that our proposed method can retrieve “good seeds”. Despite of the trade-off between training data quality and the number of training images, retrieved images can improve the accuracy of weakly supervised segmentation by combining data augmentation.

Fused feature encoding in convolutional neural network

Recently, deep hashing (DH) methods have been proposed to learn specific image representations and a series of hash functions. However, existing DH methods mainly use convolutional neural networks (CNN) to extract global features, losing some local information. What’s more, the pairwise or triplet wise model applied in DH methods increases computational complexity and storage requirements. In this paper, we propose a new DH method called fused feature encoding (FFE). In FFE, we introduce a bypass from the intermediate convolutional layer to extract images’ local information and unify local and global information into one neural network to explore richer semantic information within the image. In our model, the number of neurons in the global or local encoding layer corresponds to the number of global or local encoding bits respectively. We also apply a new method to update the weights in our network to improve the efficiency. Experimental results show the superiority of the proposed approach over the state-of-the-arts.

Vehicle color recognition using Multiple-Layer Feature Representations of lightweight convolutional neural network

In this paper, a vehicle color recognition method using lightweight convolutional neural network (CNN) is proposed. Firstly, a lightweight CNN network architecture is specifically designed for the recognition task, which contains five layers, i.e. three convolutional layers, a global pooling layer and a fully connected layer. Different from the existing CNN based methods that only use the features output from the final layer for recognition, in this paper, the feature maps of intermediate convolutional layers are all applied for recognition based on the fact that these convolutional features can provide hierarchical representations of the images. Spatial Pyramid Matching (SPM) strategy is adopted to divide the feature map, and each SPM sub-region is encoded to generate a feature representation vector. These feature representation vectors of convolutional layers and the output feature vector of the global pooling layer are normalized and cascaded as a whole feature vector, which is finally utilized to train Support Vector Machine classifier to obtain the recognition model. The experimental results show that, compared with the state-of-art methods, the proposed method can obtain more than 0.7% higher recognition accuracy, up to 95.41%, while the dimensionality of the feature vector is only 18% and the memory footprint is only 0.5%.

Content-based image retrieval with compact deep convolutional features

Convolutional neural networks (CNNs) with deep learning have recently achieved a remarkable success with a superior performance in computer vision applications. Most of CNN-based methods extract image features at the last layer using a single CNN architecture with orderless quantization approaches, which limits the utilization of intermediate convolutional layers for identifying image local patterns. As one of the first works in the context of content-based image retrieval (CBIR), this paper proposes a new bilinear CNN-based architecture using two parallel CNNs as feature extractors. The activations of convolutional layers are directly used to extract the image features at various image locations and scales. The network architecture is initialized by deep CNNs sufficiently pre-trained on a large generic image dataset then fine-tuned for the CBIR task. Additionally, an efficient bilinear root pooling is proposed and applied to the low-dimensional pooling layer to reduce the dimension of image features to compact but high discriminative image descriptors. Finally, an end-to-end training with backpropagation is performed to fine-tune the final architecture and to learn its parameters for the image retrieval task. The experimental results achieved on three standard benchmarking image datasets demonstrate the outstanding performance of the proposed architecture at extracting and learning complex features for the CBIR task without prior knowledge about the semantic meta-data of images. For instance, using a very compact image vector of 16-length, we achieve a retrieval accuracy of 95.7% (mAP) on Oxford 5K and 88.6% on Oxford 105K; which outperforms the best results reported by state-of-the-art approaches. Additionally, a noticeable reduction is attained in the required extraction time for image features and the memory size required for storage.

Energy-Efficient and Improved Image Recognition with Conditional Deep Learning

Deep-learning neural networks have proven to be very successful for a wide range of recognition tasks across modern computing platforms. However, the computational requirements associated with such deep nets can be quite high, and hence their energy-efficient implementation is of great interest. Although, traditionally, the entire network is utilized for the recognition of all inputs, we observe that the classification difficulty varies widely across inputs in real-world datasets; only a small fraction of inputs requires the full computational effort of a network, while a large majority can be classified correctly with very low effort. In this article, we propose Conditional Deep Learning (CDL), where the convolutional layer features are used to identify the variability in the difficulty of input instances and conditionally activate the deeper layers of the network. We achieve this by cascading a linear network of output neurons for each convolutional layer and monitoring the output of the linear network to decide whether classification can be terminated at the current stage or not. The proposed methodology thus enables the network to dynamically adjust the computational effort depending on the difficulty of the input data while maintaining competitive classification accuracy. The overall energy benefits for MNIST/CIFAR10/Tiny ImageNet datasets with state-of-the-art deep-learning architectures are 1.84 × /2.83 × /4.02 × , respectively. We further employ the conditional approach to train deep-learning networks from scratch with integrated supervision from the additional output neurons appended at the intermediate convolutional layers. Our proposed integrated CDL training leads to an improvement in the gradient convergence behavior giving substantial error rate reduction on MNIST/CIFAR-10, resulting in improved classification over state-of-the-art baseline networks.