Features Of Convolutional Layers Research Articles

Change detection with cross enhancement of high‐ and low‐level change‐related features

Change detection (CD) is a fundamental yet challenging problem, which aims at detecting changed object in two observations. Recent CD methods are designed based on the off-the-shelf semantic segmentation network architectures, which is not optimal for extracting and using change-related features. In this paper, a novel CD network architecture is proposed, including change-related feature extraction, cross feature enhancement, and multi-level supervision. Absolute difference of the features of different convolutional layers is first computed from a Unet-like network for two observations. The features are partitioned into high- and low-level features according to their functionalities. Then the high- and low-level features are recurrently refined by cross feature enhancement to increase the representational ability of the features. The network learns change-related features with multi-level supervisions. The final CD result can be obtained by fusing multiple predictions. Experimental results on three CD benchmark datasets indicate the superiority of the authors' method when compared with six state-of-the-art deep learning-based CD methods.

Facial Expression Recognition Algorithm Based on Fusion of Transformed Multilevel Features and Improved Weighted Voting SVM

In allusion to the shortcomings of traditional facial expression recognition (FER) that only uses a single feature and the recognition rate is not high, a FER method based on fusion of transformed multilevel features and improved weighted voting SVM (FTMS) is proposed. The algorithm combines the transformed traditional shallow features and convolutional neural network (CNN) deep semantic features and uses an improved weighted voting method to make a comprehensive decision on the results of the four trained SVM classifiers to obtain the final recognition result. The shallow features include local Gabor features, LBP features, and joint geometric features designed in this study, which are composed of distance and deformation characteristics. The deep feature of CNN is the multilayer feature fusion of CNN proposed in this study. This study also proposes to use a better performance SVM classifier with CNN to replace Softmax since the poor distinction between facial expressions. Experiments on the FERPlus database show that the recognition rate of this method is 17.2% higher than that of the traditional CNN, which proves the effectiveness of the fusion of the multilayer convolutional layer features and SVM. FTMS-based facial expression recognition experiments are carried out on the JAFFE and CK+ datasets. Experimental results show that, compared with the single feature, the proposed algorithm has higher recognition rate and robustness and makes full use of the advantages and characteristics of different features.

Research on Ocean Feature Mining Processing Based on Deep Convolution Optimization Algorithm

Yang, S.; Chong, X.; Yan, Z.; Sun, J., and Fu, F., 2020. Research on ocean feature mining processing based on deep convolution optimization algorithm. In: Yang, D.F. and Wang, H. (eds.), Recent Advances in Marine Geology and Environmental Oceanography. Journal of Coastal Research, Special Issue No. 108, pp. 58–62. Coconut Creek (Florida), ISSN 0749-0208.With the breakthrough progress of deep learning methods in ocean image classification and target detection, the sea-level tracking algorithms based on deep learning have also attracted widespread attention. Therefore, an anti-occlusion real-time target tracking algorithm using a deep convolution model to extract ocean image features is proposed in which offline pretraining is performed on a large-scale data set with a stack-type noise reduction auto-encoder to obtain general object representation capabilities Then, the particle filtering framework is introduced to input the labeled samples of the first frame in tracking data set for online fine-tuning so that the extracted features of different convolutional layers in the pretrained deep network can be combined with the relevant filtering framework. The algorithm in the paper cannot achieve only good tracking accuracy, but it also realized faster tracking speed under complex sea-level environments. The problem of target occlusion during tracking is solved as well.

Human action recognition using convolutional LSTM and fully-connected LSTM with different attentions

This paper aims to address the human action recognition issue by using convolutional long short-term memory networks (Conv-LSTM) and fully-connected LSTM (FC-LSTM) with different attentions. To this end, the spatial-temporal dual-attention network (STDAN), which is mainly composed of feature extraction, attention and fusion modules, is designed. Different from the features of high-level fully-connected layer mostly used in previous work, the features of convolution and fully-connected layers of convolutional neural network (CNN) are both extracted in STDAN, which can enrich the initial level of video representation. Besides, the Conv-LSTM and FC-LSTM are employed to handle the long-duration sequential features with different temporal context information. To reinforce the spatial-temporal attention ability, a temporal attention module (TAM) and a joint spatial-temporal attention module (JSTAM) are implemented. Through the principle components analysis (PCA) and features fusion, the potential of STDAN is effectively explored and weighted. Finally, the experimental results show that the proposed STDAN has better recognition performance than existing state-of-the-art methods.

Color image multi-scale guided depth image super-resolution reconstruction

In order to obtain better super-resolution reconstruction results of depth images, this paper constructs a multi-scale color image guidance depth image super-resolution reconstruction convolutional neural network. In this paper, the multi-scale fusion method is used to realize the guidance of high resolution (HR) color image features to low resolution (LR) depth image features, which is beneficial to the restoration of image details. In the process of extracting features from LR depth images, a multiple receptive field residual block (MRFRB) is constructed to extract and fuse the features of different receptive fields, and then connect and fuse the features of each MRFRB output to obtain global fusion features. Finally, the HR depth image is obtained through sub-pixel convolution layer and global fusion features. The experimental results show that the super-resolution image obtained by this method alleviates the edge distortion and artifact problems, and has better visual effects.

Gradient boosting in crowd ensembles for Q-learning using weight sharing

Reinforcement learning (RL) is a double-edged sword: it frees the human trainer from having to provide voluminous supervised training data or from even knowing a solution. On the other hand, a common complaint about RL is that learning is slow. Deep Q-learning (DQN), a somewhat recent development, has allowed practitioners and scientists to solve tasks previously thought unsolvable by a reinforcement learning approach. However DQN has resulted in an explosion in the number of model parameters which has further exasperated the computational needs of Q-learning during training. In this work, an ensemble approach which improves the training time, in terms of the number of interactions with the training environment, is proposed. In the presented experiments, it is shown that the proposed approach improves stability of during training, results in improved average performance, results in more reliable training, and faster learning of features in convolutional layers.

Ensemble of fine‐tuned convolutional neural networks for urine sediment microscopic image classification

In this study, an ensemble of fine‐tuned convolutional neural networks (CNNs) is proposed. As CNN training requires large annotated data, which are lacking in the field of urine sediment microscopic image processing, the authors first pre‐trained the CNNs, including ResNet50 and GoogLeNet, and developed AlexNet on an ImageNet dataset. Thereafter, some of the weights of the pre‐trained CNNs were transferred to the urine sediment microscopic image dataset. To guide fine‐tuning of the learning rate and cascading features, the hierarchical nature of features in different convolutional layers was investigated by visualising the CNN. Then, they combined three CNNs as an ensemble of CNNs to decrease the differences and impurity interference among features of urine sediment microscopic image. These fusion features were employed to train the fully connected neural network for classification. In this study, they improved the accuracy of each CNN by an average of 2.2% through fine‐tuning of the learning rate and cascading features. Moreover, the better experimental results were achieved compared with other state‐of‐the‐art methods and indicated that a 97% classification accuracy can be attained.

An Automatic Nucleus Segmentation and CNN Model based Classification Method of White Blood Cell

White blood cells (WBCs) play a remarkable role in the human immune system. To diagnose blood-related diseases, pathologists need to consider the characteristics of WBC. The characteristics of WBC can be defined based on the morphological properties of WBC nucleus. Therefore, nucleus segmentation plays a vital role to classify the WBC image and it is an important part of the medical diagnosis system. In this study, color space conversion and k-means algorithm based new WBC nucleus segmentation method is proposed. Then we localize the WBC based on the location of segmented nucleus to separate them from the entire blood smear image. To classify the localized WBC image, we propose a new convolutional neural network (CNN) model by combining the concept of fusing the features of first and last convolutional layers, and propagating the input image to the convolutional layer. We also use a dropout layer for preventing the model from overfitting problem. We show the effectiveness of our proposed nucleus segmentation method by evaluating with seven quality metrics and comparing with other methods on four public databases. We achieve average accuracy of 98.61% and more than 97% on each public database. We also evaluate our proposed CNN model by using nine classification metrics and achieve an overall accuracy of 96% on BCCD test database. To validate the generalization capability of our proposed CNN model, we show the training and testing accuracy and loss curves for random test set of BCCD database. Further, we compare the performance of our proposed CNN model with four state-of-the-art CNN models (biomedical image classifier) by measuring the value of evaluation metrics.

Cross-Modality Person Re-Identification via Modality-aware Collaborative Ensemble Learning.

Visible thermal person re-identification (VT-ReID) is a challenging cross-modality pedestrian retrieval problem due to the large intra-class variations and modality discrepancy across different cameras. Existing VT-ReID methods mainly focus on learning cross-modality sharable feature representations by handling the modality-discrepancy in feature level. However, the modality difference in classifier level has received much less attention, resulting in limited discriminability. In this paper, we propose a novel modality-aware collaborative ensemble (MACE) learning method with middle-level sharable two-stream network (MSTN) for VT-ReID, which handles the modality-discrepancy in both feature level and classifier level. In feature level, MSTN achieves much better performance than existing methods by capturing sharable discriminative middlelevel features in convolutional layers. In classifier level, we introduce both modality-specific and modality-sharable identity classifiers for two modalities to handle the modality discrepancy. To utilize the complementary information among different classifiers, we propose an ensemble learning scheme to incorporate the modality sharable classifier and the modality specific classifiers. In addition, we introduce a collaborative learning strategy, which regularizes modality-specific identity predictions and the ensemble outputs. Extensive experiments on two cross-modality datasets demonstrate that the proposed method outperforms current state-of-the-art by a large margin, achieving rank- 1/mAP accuracy 51.64%/50.11% on the SYSU-MM01 dataset, and 72.37%/69.09% on the RegDB dataset.

Efficient Feature Recombining Network Based on Refining Multi-Level Feature Maps for Semantic Segmentation

Modern approaches for semantic segmentation usually concatenate the feature map of the last convolutional layer and multi-scale features to form the final feature representation, which can achieve the more accurate classification of target pixels for the input image. However, the feature information of the last layer is not complete and refined so that there is a performance bottleneck in the concatenation between the final feature map and multi-scale feature representations. To solve this problem, we propose the Feature Recombining Network to get more refined and precise features for Semantic Segmentation. Our network is composed of Feature Recombining Module and Modified Pyramid Pooling Module. The two modules can extract more detailed and representative features through the feature recombination and acquire richer context information than the previous module respectively. Experiments show that our modules are effective to improve the segmentation precision and the Modified Pyramid Pooling Module is also superior to the previous module. Based on our proposed network, we achieve the performance of 51.9% mIoU on Pascal Context dataset and 44.75% mIoU on ADE20K dataset.

An Improved Image Style Transfer Algorithm Based on Deep Learning Network

Aiming at the problem of local style migration distortion in image stylization, an improved image style migration algorithm based on deep learning network is proposed. Firstly, the VGGNet-19 network is used to extract the convolution layer features of images. Then the characteristics of convolution layer are analyzed, and the combination of style and content features is studied. The Block3 layer with the smallest content loss and style loss is selected for feature fusion with conv1_1, conv2_1, conv3_1, conv4_1 and conv5_1 layers. Finally, under the optimal combination of features, Adam algorithm is used to optimize the image style migration. The experimental results show that the proposed algorithm can effectively improve the distortion of local style migration and provide theoretical support for the implementation of style migration technology.

Fully Dense Multiscale Fusion Network for Hyperspectral Image Classification

The convolutional neural network (CNN) can automatically extract hierarchical feature representations from raw data and has recently achieved great success in the classification of hyperspectral images (HSIs). However, most CNN based methods used in HSI classification neglect adequately utilizing the strong complementary yet correlated information from each convolutional layer and only employ the last convolutional layer features for classification. In this paper, we propose a novel fully dense multiscale fusion network (FDMFN) that takes full advantage of the hierarchical features from all the convolutional layers for HSI classification. In the proposed network, shortcut connections are introduced between any two layers in a feed-forward manner, enabling features learned by each layer to be accessed by all subsequent layers. This fully dense connectivity pattern achieves comprehensive feature reuse and enforces discriminative feature learning. In addition, various spectral-spatial features with multiple scales from all convolutional layers are fused to extract more discriminative features for HSI classification. Experimental results on three widely used hyperspectral scenes demonstrate that the proposed FDMFN can achieve better classification performance in comparison with several state-of-the-art approaches.

Multilayer feature descriptors fusion CNN models for fine‐grained visual recognition

AbstractFine‐grained image classification is a challenging topic in the field of computer vision. General models based on first‐order local features cannot achieve acceptable performance because the features are not so efficient in capturing fine‐grained difference. A bilinear convolutional neural network (CNN) model exhibits that a second‐order statistical feature is more efficient in capturing fine‐grained difference than a first‐order local feature. However, this framework only considers the extraction of a second‐order feature descriptor, using a single convolutional layer. The potential effective classification features of other convolutional layers are ignored, resulting in loss of recognition accuracy. In this paper, a multilayer feature descriptors fusion CNN model is proposed. It fully considers the second‐order feature descriptors and the first‐order local feature descriptor generated by different layers. Experimental verification was carried out on fine‐grained classification benchmark data sets, CUB‐200‐2011, Stanford Cars, and FGVC‐aircraft. Compared with the bilinear CNN model, the proposed method has improved accuracy by 0.8%, 1.1%, and 5.5%. Compared with the compact bilinear pooling model, there is an accuracy increase of 0.64%, 1.63%, and 1.45%, respectively. In addition, the proposed model effectively uses multiple 1×1 convolution kernels to reduce dimension. The experimental results show that the multilayer low‐dimensional second‐order feature descriptors fusion model has comparable recognition accuracy of the original model.

Iterative fusion convolutional neural networks for classification of optical coherence tomography images

Optical coherence tomography (OCT) can achieve the high-resolution 3D tomography imaging of the retina, which is crucial for the diagnosis of retinal diseases. Currently, the classification of retinal OCT images is mainly conducted by ophthalmologists, which is time consuming and subjective. In this paper, we propose an iterative fusion convolutional neural network (IFCNN) method for the automatic classification of retinal OCT image. In the convolutional neural network (CNN), different convolutional layers contain feature information from different scales. Therefore, the proposed network adopts an iterative fusion strategy, which iteratively combines features in current convolutional layer with those of all previous layers in the CNN network, and thus can jointly utilize the features of different convolutional layers to achieve accurate classification of OCT images. Experimental results on a real retinal OCT dataset and a musculoskeletal radiographs dataset demonstrate the superiority of the proposed method over the traditional CNN and several well-known OCT classification methods.

Detecting objects using Rolling Convolution and Recurrent Neural Network

Abstract — At present, most of the existing target detection algorithms use the method of region proposal to search for the target in the image. The most effective regional proposal method usually requires thousands of target prediction areas to achieve high recall rate. This lowers the detection efficiency. Even though recent region proposal network approach have yielded good results by using hundreds of proposals, it still faces the challenge when applied to small objects and precise locations. This is mainly because these approaches use coarse feature. Therefore, we propose a new method for extracting more efficient global features and multi-scale features to provide target detection performance. Given that feature maps under continuous convolution lose the resolution required to detect small objects when obtaining deeper semantic information; hence, we use rolling convolution (RC) to maintain the high resolution of low-level feature maps to explore objects in greater detail, even if there is no structure dedicated to combining the features of multiple convolutional layers. Furthermore, we use a recurrent neural network of multiple gated recurrent units (GRUs) at the top of the convolutional layer to highlight useful global context locations for assisting in the detection of objects. Through experiments in the benchmark data set, our proposed method achieved 78.2% mAP in P ASCAL VOC 2007 and 72.3% mAP in PASCAL VOC 2012 dataset. It has been verified through many experiments that this method has reached a more advanced level of detection.

Multiple Feature Reweight DenseNet for Image Classification

Recent network research has demonstrated that the performance of convolutional neural networks can be improved by introducing a learning block that captures spatial correlations. In this paper, we propose a novel multiple feature reweight DenseNet (MFR-DenseNet) architecture. The MFR-DenseNet improves the representation power of the DenseNet by adaptively recalibrating the channel-wise feature responses and explicitly modeling the interdependencies between the features of different convolutional layers. First, in order to perform dynamic channel-wise feature recalibration, we construct the channel feature reweight DenseNet (CFR-DenseNet) by introducing the squeeze-and-excitation module (SEM) to DenseNet. Then, to model the interdependencies between the features of different convolutional layers, we propose the double squeeze-and-excitation module (DSEM) and construct the inter-layer feature reweight DenseNet (ILFR-DenseNet). In the last step, we designed the MFR-DenseNet by combining the CFR-DenseNet and the ILFR-DenseNet with an ensemble learning approach. Our experiments demonstrate the effectiveness of CFR-DenseNet, ILFR-DenseNet, and MFR-DenseNet. More importantly, the MFR-DenseNet drops the error rate on CIFAR-10 and CIFAR-100 by a large margin with significantly fewer parameters. Our 100-layer MFR-DenseNet (with 7.1M parameters) model achieves competitive results on CIFAR-10 and CIFAR-100 data sets, with test errors of 3.57% and 18.27% respectively, achieving a 4.5% relative improvement on CIFAR-10 and a 5.09% relative improvement on CIFAR-100 over the best result of DenseNet (with 27.2M parameters).

Sequence recognition of Chinese license plates

Abstract The recognition of license plates is very important for intelligent transportation systems. Generally, the performance of an intelligent recognition algorithm is greatly affected by different shooting angles, illumination conditions and backgrounds of the license plate images. This paper presents a sequence recognition approach for intelligent recognition of Chinese license plates. Firstly, a spatial transformer network (STN) is employed to adjust the inclined and deformed license plates such that all the plates have a uniform orientation and thus are easier to be recognized. Then, an improved convolutional neural network (CNN) is designed to extract sequence features of the rectified license plates. The features of different convolutional layers are integrated as input to a bi-directional recurrent neural network (BRNN), where the character segmentation is not needed. Finally, the recognition is accomplished by the BRNN and connectionist temporal classification (CTC). Due to the lack of adequate Chinese license plates, an effective training method is presented in which the network is pre-trained by sufficiently enough synthetic license plates and is fine-tuned by our collected real Chinese license plates. The experimental results show that our model achieves better recognition accuracy and lower average edit distance than some existing methods.

Robust Visual Tracking via Hierarchical Convolutional Features.

Visual tracking is challenging as target objects often undergo significant appearance changes caused by deformation, abrupt motion, background clutter and occlusion. In this paper, we propose to exploit the rich hierarchical features of deep convolutional neural networks to improve the accuracy and robustness of visual tracking. Deep neural networks trained on object recognition datasets consist of multiple convolutional layers. These layers encode target appearance with different levels of abstraction. For example, the outputs of the last convolutional layers encode the semantic information of targets and such representations are invariant to significant appearance variations. However, their spatial resolutions are too coarse to precisely localize the target. In contrast, features from earlier convolutional layers provide more precise localization but are less invariant to appearance changes. We interpret the hierarchical features of convolutional layers as a nonlinear counterpart of an image pyramid representation and explicitly exploit these multiple levels of abstraction to represent target objects. Specifically, we learn adaptive correlation filters on the outputs from each convolutional layer to encode the target appearance. We infer the maximum response of each layer to locate targets in a coarse-to-fine manner. To further handle the issues with scale estimation and re-detecting target objects from tracking failures caused by heavy occlusion or out-of-the-view movement, we conservatively learn another correlation filter, that maintains a long-term memory of target appearance, as a discriminative classifier. We apply the classifier to two types of object proposals: (1) proposals with a small step size and tightly around the estimated location for scale estimation; and (2) proposals with large step size and across the whole image for target re-detection. Extensive experimental results on large-scale benchmark datasets show that the proposed algorithm performs favorably against the state-of-the-art tracking methods.

Using Convolutional Layer Features for Indoor Human Activity Recognition based on Spatial Location Information

Identifying human actions has great importance for various applications, especially in the smart home, fitness tracking and health monitoring domains. However, human activity recognition still remains a challenging task. This is mainly due to the broad range of human activities as well as the rich variation of a given activity can be performed. In this paper, we dealt with the problem by making use of spatial location information of three different parts of a human body, which are derived via three UWB (ultrawide band) tags and an Ubisense positioning system. In order to improve the accuracy, we proposed a recognition method: convolutional layer features plus SVM (Support Vector Machine). We pre-process the raw spatial location data and transfer them into motion feature, frequency feature and statistic feature. These features are input into the CNN (Convolutional Neural Network) to generate the convolutional layer features, and then we use SVM to classify these features. By comparing the experimental results, the best recognition rate of different experimenters is 89.75%, which shows its feasibility.

UniCNN: A Pipelined Accelerator Towards Uniformed Computing for CNNs

Convolutional neural networks (CNNs) have been widely applied for image recognition, face detection, and video analysis because of their ability to achieve accuracy close to or even better than human level perception. However, different features of convolution layers and fully connected layers have brought many challenges to the implementation of CNN on FPGA platforms, because different accelerator units must be designed to process the whole networks. In order to overcome this problem, this work proposes a pipelined accelerator towards uniformed computing for convolutional neural networks. For the convolution layer, the accelerator first repositions the input features into matrix on-the-fly when they are stored to FPGA on-chip buffers, thus the computation of convolution layer can be completed through matrix multiplication. For the fully connected layer, the batch-based method is used to reduce the required memory bandwidth, which also can be completed through matrix multiplication. Then a pipelined computation method for matrix multiplication is proposed to increase the throughput and also reduce the buffer requirement. The experiment results show that the proposed accelerator surpasses CPUs and GPUs platform in terms of energy efficiency. The proposed accelerator can achieve the throughput of 49.31 GFLOPS, which is done using only 198 DSP modules. Compared to the state-of-the-art implementatuion, our accelerator has better hardware utilization efficiency.