Multilayer Convolutional Features Research Articles

Design of Intelligent Interactive Teaching System for College English Based on Streamlined Bilinear Attention Networks

Abstract Intelligent methods such as deep learning provide an innovative optimization path for English teaching in colleges and universities. In this paper, the attention mechanism is based on traditional CNN. Embedded attention and multilayer convolutional feature fusion are used to complete the design of streamlined bilinear attention networks. The streamlined bilinear attention network is used to obtain the characteristics of students’ classroom behavior in the process of English teaching, combine the characteristics of students’ classroom behavior and computer programming language to design and implement the intelligent interactive teaching system of English in colleges and universities, and test and apply the analysis of the intelligent interactive teaching system of English. The results show that in the query performance test report, even though the number of requests issued in the test is 2,000, the error rate is always kept at 0, the throughput is 39.8, and the response time is kept at 95~1144s, which indicates that this system can meet the performance requirements with a moderate number of users. In the application analysis, the total score of the teaching method of English Intelligent Interactive Teaching System (mean=94.62) is higher than that of the traditional English teaching method (mean=89.65), which indicates that the teaching method of English Intelligent Interactive Teaching System is more capable of improving the interaction and communication ability of the students in English teaching. The system constructed in this study can meet the diverse needs of students and contribute to the quality improvement of English teaching in colleges and universities.

A Long Time Span-Specific Emitter Identification Method Based on Unsupervised Domain Adaptation

Specific emitter identification (SEI) is a professional technology to recognize different emitters by measuring the unique features of received signals. It has been widely used in both civilian and military fields. Recently, many SEI methods based on deep learning have been proposed, most of which assume that the training set and testing set have the same data distribution. However, in reality, the testing set is generally used later than the training set and lacks labels. The long time span may change the signal transmission environment and fingerprint features. These changes result in considerable differences in data distribution between the training and testing sets, thereby affecting the recognition and prediction abilities of the model. Therefore, the existing works cannot achieve satisfactory results for a long time span SEI. To address this challenge and obtain stable fingerprints, we transform the long time span SEI problem into a domain adaptive problem and propose an unsupervised domain adaptive method called LTS-SEI. Noteworthily, LTS-SEI uses a multilayer convolutional feature extractor to learn feature knowledge and confronts a domain discriminator to generate domain-invariant shallow fingerprints. The classifier of LTS-SEI applies feature matching to source domain samples and target domain samples to achieve the domain alignment of deep fingerprints. The classifier further reduces the intraclass diversity of deep features to alleviate the misclassification problem of edge samples in the target domain. To confirm the effectiveness and reliability of LTS-SEI, we collect multiple sets of real satellite navigation signals using two antennas with 13 m- and 40 m-large apertures, respectively, and construct two available datasets. Numerous experiments demonstrate that LTS–SEI can considerably increase the recognition accuracy of the long time span SEI and is superior to the other existing methods.

Implicit space pose consistent transfer network for deep face verification

Unconstrained face recognition has achieved state-of-the-art performance with the help of the powerful representation capabilities of deep convolutional neural networks. However, face recognition always suffers from other intra-class imbalances due to the influence of pose variations present in the real world, which significantly degrades the performance, especially for tilted face recognition under surveillance videos, which is rarely studied. Tilted face recognition is more challenging than frontal and profile face recognition due to the effect of self-obscuration. To this end, we consider promoting the frontalization of tilted faces in the implicit space, while preserving identity information. In this paper, we propose an implicit spatial pose consistent transfer network (PCTN) that exploits the prior guidance of pose information to enhance the model’s ability to perceive pose. Specifically, we first develop the multi-path attention block (MPAB), which is used as the main block for feature embedding in deep networks. Such a design is employed to enhance the relevance of multi-layer convolutional features and provide a powerful foundation for the deep asymptotic transfer of feature frontalization. Then, the pose consistent transfer module (PCTM) combines pose labels and deep embedding features to learn robust discriminative information. PCTM consists of two parts: the pose guidance block (PGB) explores the implicit nonlinear feature mapping from pose faces to frontal faces in feature space, and the attention compensation module (ACM) aims to exploit the complementarity of dense and sparse features, enhancing pose-related features and suppressing pose-unrelated features in terms of both spatial-wise and channel-wise. Furthermore, we investigate the performance comparison on datasets with multiple pose variations without performance degradation. Extensive experimental results on multiple benchmark datasets demonstrate the effectiveness of our proposed method.

Deep Feature Aggregation Framework Driven by Graph Convolutional Network for Scene Classification in Remote Sensing.

Scene classification of high spatial resolution (HSR) images can provide data support for many practical applications, such as land planning and utilization, and it has been a crucial research topic in the remote sensing (RS) community. Recently, deep learning methods driven by massive data show the impressive ability of feature learning in the field of HSR scene classification, especially convolutional neural networks (CNNs). Although traditional CNNs achieve good classification results, it is difficult for them to effectively capture potential context relationships. The graphs have powerful capacity to represent the relevance of data, and graph-based deep learning methods can spontaneously learn intrinsic attributes contained in RS images. Inspired by the abovementioned facts, we develop a deep feature aggregation framework driven by graph convolutional network (DFAGCN) for the HSR scene classification. First, the off-the-shelf CNN pretrained on ImageNet is employed to obtain multilayer features. Second, a graph convolutional network-based model is introduced to effectively reveal patch-to-patch correlations of convolutional feature maps, and more refined features can be harvested. Finally, a weighted concatenation method is adopted to integrate multiple features (i.e., multilayer convolutional features and fully connected features) by introducing three weighting coefficients, and then a linear classifier is employed to predict semantic classes of query images. Experimental results performed on the UCM, AID, RSSCN7, and NWPU-RESISC45 data sets demonstrate that the proposed DFAGCN framework obtains more competitive performance than some state-of-the-art methods of scene classification in terms of OAs.

Recognition and Error Correction Techniques for Piano Playing Music Based on Convolutional Cyclic Hashing Method

Music as a sound symbol can express what people think; music is both a form of social behavior and can promote people’s emotional communication; music is also a form of entertainment and can enrich people’s spiritual life. In this paper, we propose a new convolutional recurrent hashing method CRNNH, which uses multilayer RNN to learn to discriminate piano playing music using convolutional feature map sequences. Firstly, a convolutional feature map sequence learning preserving similarity hash function is designed consisting of multilayer convolutional feature maps extracted from multiple convolutional layers of a pretrained CNN; secondly, a new deep learning framework is proposed to generate hash codes using a multilayer RNN, which directly uses the convolutional feature maps as input to preserve the spatial structure of the feature maps; finally, a new loss function is proposed to preserve the semantic similarity and balance of the hash codes, while considering the quantization error generated when the hash layer outputs binary hash codes. The experimental results illustrate that the proposed CRNNH can obtain better performance compared to other hashing methods.

Multilayer Feature Fusion Network With Spatial Attention and Gated Mechanism for Remote Sensing Scene Classification

Remote sensing (RS) scene classification has attracted extensive attention due to its large number of applications. Recently, convolutional neural networks (CNNs) methods have shown impressive ability of feature learning in RS scene classification. However, the performance is still limited by large-scale variance and complex background. To address these problems, we present a multilayer feature fusion network with spatial attention and gated mechanism (MLF2Net_SAGM) for RS scene classification. At first, the backbone is employed to extract multilayer convolutional features. Then, a residual spatial attention module (RSAM) is proposed to enhance discriminative regions of the multilayer feature maps, and key areas can be harvested. Finally, the multilayer spatial calibration features are fused to form the final feature map, and a gated fusion module (GFM) is designed to eliminate feature redundancy and mutual exclusion (FRME). To verify the effectiveness of the proposed method, we conduct comparative experiments based on three widely used RS image scene classification benchmarks. The results show that the direct fusion of multilayer features via element-wise addition leads to FRME, whereas our method fuses multilayer features more effectively and improves the performance of scene classification.

A Cross-Layer Nonlocal Network for Remote Sensing Scene Classification

Remote sensing scene classification (RSSC) is a fundamental yet challenging task in the domain of remote sensing (RS). Currently, the methods based on deep features from convolutional neural networks (CNNs) have significantly improved the scene classification accuracy (ACC). However, the standard convolution operations have limited capacity to model the long-range correlations and cannot effectively obtain global contextual understanding ability. In this letter, we propose a novel scene classification framework, termed cross-layer nonlocal network (CL-NL-Net), consisting of a backbone network, a cross-layer nonlocal (CL-NL) module, and a classifier. Among them, the backbone network is used to obtain multilayer convolutional features. The CL-NL module is the core of the proposed method, which captures the long-range correlations between different layers, so as to achieve a better global scene understanding ability. To verify the effectiveness of the proposed CL-NL-Net, we conduct experiments on four benchmark datasets, and the results demonstrate that the proposed method achieves competitive classification ACC and outperforms some state-of-the-art methods.

HCDC-SRCF tracker: Learning an adaptively multi-feature fuse tracker in spatial regularized correlation filters framework

Integrating multi-feature based on multi-layer features from the convolutional network or based on multiple hand-crafted features has been proved to be an effective way for improving tracking performance. In this work, we investigate how to integrate multi-layer convolutional features with hand-crafted features. Specifically, an adaptive multi-feature fusion strategy is proposed based on convolutional features from ResNet-101 and hand-crafted features from HOG as well as Grayscale in spatial regularized correlation filter framework. We fully consider the complementary advantages of multi-layer convolutional features and hand-crafted features to construct a robust and reliable appearance representation of the target. Comprehensive experimental results on benchmark datasets demonstrate that our tracker has achieved significant performance improvements in various challenging environments. Compared to the trackers based only on multi-layer convolutional features or complete hand-crafted fusion features, the most important is that our proposed tracker obtains more competitive tracking performance. Our tracker is publicly available. You can find open-sourced code of our tracker at https://github.com/binger1225/HCDC-SRCF.

An Attention-Guided Multilayer Feature Aggregation Network for Remote Sensing Image Scene Classification

Remote sensing image scene classification (RSISC) has broad application prospects, but related challenges still exist and urgently need to be addressed. One of the most important challenges is how to learn a strong discriminative scene representation. Recently, convolutional neural networks (CNNs) have shown great potential in RSISC due to their powerful feature learning ability; however, their performance may be restricted by the complexity of remote sensing images, such as spatial layout, varying scales, complex backgrounds, category diversity, etc. In this paper, we propose an attention-guided multilayer feature aggregation network (AGMFA-Net) that attempts to improve the scene classification performance by effectively aggregating features from different layers. Specifically, to reduce the discrepancies between different layers, we employed the channel–spatial attention on multiple high-level convolutional feature maps to capture more accurately semantic regions that correspond to the content of the given scene. Then, we utilized the learned semantic regions as guidance to aggregate the valuable information from multilayer convolutional features, so as to achieve stronger scene features for classification. Experimental results on three remote sensing scene datasets indicated that our approach achieved competitive classification performance in comparison to the baselines and other state-of-the-art methods.

White blood cell type identification using multi-layer convolutional features with an extreme-learning machine

White blood cells (WBCs) are the main component of the immune system that have a major role in protecting the body against different types of infections arising due to viruses, bacteria, fungi, and so on. The WBCs are generally referred to as 5 main subtypes: lymphocytes, monocytes, neutrophils, eosinophils, and basophils. Recognizing and counting each type of WBC is important for diagnosing and treating various disorders, such as infectious diseases, autoimmune disorders, immune deficiencies, leukemia, etc. To this end, a fast and accurate WBC classification model is crucial. This study offers a new model that works with a deep neural network—namely, multi-layer (ML) convolutional features of the AlexNet architecture followed by a feature selection (FS) strategy (MLANet-FS) for WBC-type identification. The proposed model exploits multi-layer convolutional features from different layers of the AlexNet model to provide rich discriminative detail, because different convolutional layers contain different visual characteristics of WBCs, and thereafter, linear fusion of these features occurs automatically. FS strategy is used to select the most distinguishing features from the feature fusion pool. Next, an extreme-learning machine (ELM) is employed to learn a discriminative model of WBC type identification. The proposed MLANet-FS-ELM model was evaluated in extensive experiments on the WBC benchmark dataset. It achieved 99.99% training accuracy and 99.12% testing accuracy, demonstrating that the proposed model outperforms alternative methods in the literature developed for WBC identification.

Learning Local-Global Multiple Correlation Filters for Robust Visual Tracking with Kalman Filter Redetection.

Visual object tracking is a significant technology for camera-based sensor networks applications. Multilayer convolutional features comprehensively used in correlation filter (CF)-based tracking algorithms have achieved excellent performance. However, there are tracking failures in some challenging situations because ordinary features are not able to well represent the object appearance variations and the correlation filters are updated irrationally. In this paper, we propose a local–global multiple correlation filters (LGCF) tracking algorithm for edge computing systems capturing moving targets, such as vehicles and pedestrians. First, we construct a global correlation filter model with deep convolutional features, and choose horizontal or vertical division according to the aspect ratio to build two local filters with hand-crafted features. Then, we propose a local–global collaborative strategy to exchange information between local and global correlation filters. This strategy can avoid the wrong learning of the object appearance model. Finally, we propose a time-space peak to sidelobe ratio (TSPSR) to evaluate the stability of the current CF. When the estimated results of the current CF are not reliable, the Kalman filter redetection (KFR) model would be enabled to recapture the object. The experimental results show that our presented algorithm achieves better performances on OTB-2013 and OTB-2015 compared with the other latest 12 tracking algorithms. Moreover, our algorithm handles various challenges in object tracking well.

A Multi-Kernel Mode Using a Local Binary Pattern and Random Patch Convolution for Hyperspectral Image Classification

With the development of deep learning technology, more and more scholars have applied it to hyperspectral image (HSI) classification to improve classification accuracy. However, these deep-learning methods not only take a lot of time in the pre-training phase, but also have relatively limited classification performance when there are fewer labeled samples. In order to improve classification performance while reducing costs, this article proposes a multikernel method based on a local binary pattern and random patches (LBPRP-MK), which integrates a local binary pattern (LBP) and deep learning into a multiple-kernel framework. First, we use LBP and hierarchical convolutional neural networks to extract local textural features and multilayer convolutional features, respectively. The convolution kernel for the convolution operation is obtained from the original image using a random strategy without training. Then, we input local textural features, multilayer convolutional features, and spectral features obtained from the original image into the radial basis function to obtain three kernel functions. Finally, the three kernel functions are merged into a multikernel function according to their optimal weights under the composite kernel strategy. This multikernel function is used as the input for the support vector machine to obtain the classification result map. Experiments show that compared with other HSI classification methods, the proposed method achieves better classification performance on three HSI datasets.

Multi-layer convolutional features fusion for dual-band decision-level ship recognition

Multi-layer convolutional features fusion for dual-band decision-level ship recognition

Dual-Band Maritime Imagery Ship Classification Based on Multilayer Convolutional Feature Fusion

Addressing to the problems of few annotated samples and low-quality fused feature in visible and infrared dual-band maritime ship classification, this paper leverages hierarchical features of deep convolutional neural network to propose a dual-band maritime ship classification method based on multilayer convolutional feature fusion. Firstly, the VGGNet model pretrained on the ImageNet dataset is fine-tuned to capture semantic information of the specific dual-band ship dataset. Secondly, the pretrained and fine-tuned VGGNet models are used to extract low-level, middle-level, and high-level convolutional features of each band image, and a number of improved recursive neural networks with random weights are exploited to reduce feature dimension and learn feature representation. Thirdly, to improve the quality of feature fusion, multilevel and multilayer convolutional features of dual-band images are concatenated to fuse hierarchical information and spectral information. Finally, the fused feature vector is fed into a linear support vector machine for dual-band maritime ship category recognition. Experimental results on the public dual-band maritime ship dataset show that multilayer convolution feature fusion outperforms single-layer convolution feature by about 2% mean per-class classification accuracy for single-band image, dual-band images perform better than single-band image by about 2.3%, and the proposed method achieves the best accuracy of 89.4%, which is higher than the state-of-the-art method by 1.2%.

Intelligent Ship Detection in Remote Sensing Images Based on Multi-Layer Convolutional Feature Fusion

Intelligent detection and recognition of ships from high-resolution remote sensing images is an extraordinarily useful task in civil and military reconnaissance. It is difficult to detect ships with high precision because various disturbances are present in the sea such as clouds, mist, islands, coastlines, ripples, and so on. To solve this problem, we propose a novel ship detection network based on multi-layer convolutional feature fusion (CFF-SDN). Our ship detection network consists of three parts. Firstly, the convolutional feature extraction network is used to extract ship features of different levels. Residual connection is introduced so that the model can be designed very deeply, and it is easy to train and converge. Secondly, the proposed network fuses fine-grained features from shallow layers with semantic features from deep layers, which is beneficial for detecting ship targets with different sizes. At the same time, it is helpful to improve the localization accuracy and detection accuracy of small objects. Finally, multiple fused feature maps are used for classification and regression, which can adapt to ships of multiple scales. Since the CFF-SDN model uses a pruning strategy, the detection speed is greatly improved. In the experiment, we create a dataset for ship detection in remote sensing images (DSDR), including actual satellite images from Google Earth and aerial images from electro-optical pod. The DSDR dataset contains not only visible light images, but also infrared images. To improve the robustness to various sea scenes, images under different scales, perspectives and illumination are obtained through data augmentation or affine transformation methods. To reduce the influence of atmospheric absorption and scattering, a dark channel prior is adopted to solve atmospheric correction on the sea scenes. Moreover, soft non-maximum suppression (NMS) is introduced to increase the recall rate for densely arranged ships. In addition, better detection performance is observed in comparison with the existing models in terms of precision rate and recall rate. The experimental results show that the proposed detection model can achieve the superior performance of ship detection in optical remote sensing image.

SiamMN: Siamese modulation network for visual object tracking

Visual object tracking methods based on Siamese network are often difficult to distinguish objects with the same semantic or similar appearance as tracking target in tracking process due to the lack of discriminating strategies for the confusing objects. We propose a visual object tracking method based on Siamese modulation network. It takes the given bounding box in the target frame and the current frame as input, and fuses these multi-layer convolutional features to obtain more target appearance information of bounding box and the current frame. The feature modulator generates feature modulation vector based on the given bounding box to enhance visual appearance information of target instance in multi-layer feature of the current frame, so as to make target instance obtain higher score in response map of region proposal network, and thus realize target instance-specific tracking task. Experiments on two public benchmark datasets, OTB2015 and VOT2018, show that the proposed tracker has a competitive performance among other state-of-the art trackers.

Residual Dense Network Based on Channel-Spatial Attention for the Scene Classification of a High-Resolution Remote Sensing Image

The scene classification of a remote sensing image has been widely used in various fields as an important task of understanding the content of a remote sensing image. Specially, a high-resolution remote sensing scene contains rich information and complex content. Considering that the scene content in a remote sensing image is very tight to the spatial relationship characteristics, how to design an effective feature extraction network directly decides the quality of classification by fully mining the spatial information in a high-resolution remote sensing image. In recent years, convolutional neural networks (CNNs) have achieved excellent performance in remote sensing image classification, especially the residual dense network (RDN) as one of the representative networks of CNN, which shows a stronger feature learning ability as it fully utilizes all the convolutional layer information. Therefore, we design an RDN based on channel-spatial attention for scene classification of a high-resolution remote sensing image. First, multi-layer convolutional features are fused with residual dense blocks. Then, a channel-spatial attention module is added to obtain more effective feature representation. Finally, softmax classifier is applied to classify the scene after adopting data augmentation strategy for meeting the training requirements of the network parameters. Five experiments are conducted on the UC Merced Land-Use Dataset (UCM) and Aerial Image Dataset (AID), and the competitive results demonstrate that our method can extract more effective features and is more conducive to classifying a scene.

Multilayer Feature Fusion Network for Scene Classification in Remote Sensing

The scene classification of high spatial resolution (HSR) images is a challenging task in the remote sensing community. How to construct a discriminative representation of the HSR scene is a key step to improve classification performance. In this letter, we propose a novel feature extraction method termed multilayer feature fusion network (MF2Net) for scene classification. At first, the transferred VGGNet-16 model is employed as a feature extractor to acquire multilayer convolutional features. Then, several layers including pooling, transformation, and fusion layers are designed to process hierarchical features in four branches, and the prediction probability can be obtained for classification. Finally, the proposed model is optimized by fine-tuning techniques, where a novel data augmentation approach is explored to improve generalization ability. As a result, MF2Net effectively applies useful information from multilayers to improve the accuracy of scene classification. The experimental results on AID and NWPU-RESISC45 data sets exhibit that the MF2Net method obtains quite competitive classification results compared with many state-of-the-art methods.

基于自适应多层卷积特征决策融合的目标跟踪

基于自适应多层卷积特征决策融合的目标跟踪

Structured robust correlation filter with L2,1 norm for object tracking

Recently, the correlation filter (CF)-based methods have achieved great success in the field of object tracking. In most of these methods, the CF utilizes L 2 norm as the regularization, which does not pay attention to the stability and robustness of the feature. However, there may exist some unstable points in the image because the object in the video may have different appearance changes. We propose a tracking method based on a structured robust correlation filter (SRCF), which employs the L 2,1 norm as the regularization. The robust CF can not only retain the accuracy from the regression formulation but also take into account the stability of the image region to improve the robustness of the appearance model. The alternating direction method of multipliers algorithm is used to solve the L 2,1 optimization problem in SRCF. Moreover, the multilayer convolutional features are adopted to further improve the representation accuracy. The proposed method is evaluated in several benchmark datasets, and the results demonstrate that it can achieve comparable performance with respect to the state-of-the-art tracking methods.