Feature Fusion Model Research Articles

Arbitrary-oriented object detection via dense feature fusion and attention model for remote sensing super-resolution image

In this paper, we aim at developing a new arbitrary-oriented end-to-end object detection method to further push the frontier of object detection for remote sensing image. The proposed method comprehensively takes into account multiple strategies, such as attention mechanism, feature fusion, rotation region proposal as well as super-resolution pre-processing simultaneously to boost the performance in terms of localization and classification under the faster RCNN-like framework. Specifically, a channel attention network is integrated for selectively enhancing useful features and suppressing useless ones. Next, a dense feature fusion network is designed based on multi-scale detection framework, which fuses multiple layers of features to improve the sensitivity to small objects. In addition, considering the objects for detection are often densely arranged and appear in various orientations, we design a rotation anchor strategy to reduce the redundant detection regions. Extensive experiments on two remote sensing public datasets DOTA, NWPU VHR-10 and scene text dataset ICDAR2015 demonstrate that the proposed method can be competitive with or even superior to the state-of-the-art ones, like R2CNN and R2CNN++.

Multi-Level Joint Feature Learning for Person Re-Identification

In person re-identification, extracting image features is an important step when retrieving pedestrian images. Most of the current methods only extract global features or local features of pedestrian images. Some inconspicuous details are easily ignored when learning image features, which is not efficient or robust to for scenarios with large differences. In this paper, we propose a Multi-level Feature Fusion model that combines both global features and local features of images through deep learning networks to generate more discriminative pedestrian descriptors. Specifically, we extract local features from different depths of network by the Part-based Multi-level Net to fuse low-to-high level local features of pedestrian images. Global-Local Branches are used to extract the local features and global features at the highest level. The experiments have proved that our deep learning model based on multi-level feature fusion works well in person re-identification. The overall results outperform the state of the art with considerable margins on three widely-used datasets. For instance, we achieve 96% Rank-1 accuracy on the Market-1501 dataset and 76.1% mAP on the DukeMTMC-reID dataset, outperforming the existing works by a large margin (more than 6%).

Few-Shot Learning for Domain-Specific Fine-Grained Image Classification

Learning to recognize novel visual categories from a few examples is a challenging task for machines in real-world industrial applications. In contrast, humans have the ability to discriminate even similar objects with little supervision. This article attempts to address the few-shot fine-grained image classification problem. We propose a feature fusion model to explore discriminative features by focusing on key regions. The model utilizes the focus-area location mechanism to discover the perceptually similar regions among objects. High-order integration is employed to capture the interaction information among intraparts. We also design a center neighbor loss to form robust embedding space distributions. Furthermore, we build a typical fine-grained and few-shot learning dataset miniPPlankton from the real-world application in the area of marine ecological environments. Extensive experiments are carried out to validate the performance of our method. The results demonstrate that our model achieves competitive performance compared with state-of-the-art models. Our work is a valuable complement to the model domain-specific industrial applications.

HYPERSPECTRAL IMAGES CLASSIFICATION BASED ON FUSION FEATURES DERIVED FROM 1D AND 2D CONVOLUTIONAL NEURAL NETWORK

Abstract. In recent years, deep learning technology has been continuously developed and gradually transferred to various fields. Among them, Convolutional Neural Network (CNN), which has the ability to extract deep features of images due to its unique network structure, plays an increasingly important role in the realm of Hyperspectral images classification. This paper attempts to construct a features fusion model that combines the deep features derived from 1D-CNN and 2D-CNN, and explores the potential of features fusion model in the field of hyperspectral image classification. The experiment is based on the deep learning open source framework TensorFlow with Python3 as programming environment. Firstly, constructing multi-layer perceptron (MLP), 1D-CNN and 2DCNN models respectively, and then, using the pre-trained 1D-CNN and 2D-CNN models as feature extractors, finally, extracting features via constructing the features fusion model. The general open hyperspectral datasets (Pavia University) were selected as a test to compare classification accuracy and classification confidence among different models. The experimental results show that the features fusion model obtains higher overall accuracy (99.65%), Kappa coefficient (0.9953) and lower uncertainty for the boundary and unknown regions (3.43%) in the data set. Since features fusion model inherits the structural characteristics of 1D-CNN and 2DCNN, the complementary advantages between the models are achieved. The spectral and spatial features of hyperspectral images are fully exploited, thus getting state-of-the-art classification accuracy and generalization performance.

Deep multi-scale feature fusion for pancreas segmentation from CT images.

Pancreas segmentation from computed tomography (CT) images is an important step in surgical procedures such as cancer detection and radiation treatment. While manual segmentation is time-consuming and operator-dependent, current computer-assisted segmentation methods are facing challenges posed by varying shapes and sizes. To address these challenges, this paper presents a multi-scale feature fusion (MsFF) model for accurate pancreas segmentation from CT images. The proposed MsFF is built upon the well-recognized encoder-decoder framework. Firstly, in the encoder stage, the squeeze-and-excitation module is incorporated to enhance the learning of features by exploiting channel-wise independence. Secondly, a hierarchical fusion module is introduced to better utilize both low-level and high-level features to retain boundary information and make final predictions. The proposed MsFF is evaluated on the NIH pancreas dataset and outperforms the current state-of-the-art methods, by achieving a mean of 87.26% and 22.67% under the Dice Sorensen Coefficient and Volumetric Overlap Error, respectively. The experimental results confirm that the incorporation of squeeze-and-excitation and hierarchical fusion modules contributes to a net gain in the performance of our proposed MsFF.

Deep Convolutional Feature Fusion Model for Multispectral Maritime Imagery Ship Recognition

Combining both visible and infrared object information, multispectral data is a promising source data for automatic maritime ship recognition. In this paper, in order to take advantage of deep convolutional neural network and multispectral data, we model multispectral ship recognition task into a convolutional feature fusion problem, and propose a feature fusion architecture called Hybrid Fusion. We fine-tune the VGG-16 model pre-trained on ImageNet through three channels single spectral image and four channels multispectral images, and use existing regularization techniques to avoid over-fitting problem. Hybrid Fusion as well as the other three feature fusion architectures is investigated. Each fusion architecture consists of visible image and infrared image feature extraction branches, in which the pre-trained and fine-tuned VGG-16 models are taken as feature extractor. In each fusion architecture, image features of two branches are firstly extracted from the same layer or different layers of VGG-16 model. Subsequently, the features extracted from the two branches are flattened and concatenated to produce a multispectral feature vector, which is finally fed into a classifier to achieve ship recognition task. Furthermore, based on these fusion architectures, we also evaluate recognition performance of a feature vector normalization method and three combinations of feature extractors. Experimental results on the visible and infrared ship (VAIS) dataset show that the best Hybrid Fusion achieves 89.6% mean per-class recognition accuracy on daytime paired images and 64.9% on nighttime infrared images, and outperforms the state-of-the-art method by 1.4% and 3.9%, respectively.

Feature Fusion Using Stacked Denoising Auto-Encoder and GBDT for Wi-Fi Fingerprint-Based Indoor Positioning

It may be very difficult to receive the signals from satellite positioning systems due to the existing obstacles in indoor environment. Arising from the popularity of smart phones, Wi-Fi based indoor positioning technology has the advantages with convenient deployment and low hardware cost. In this study, we focused on indoor positioning using Wi-Fi fingerprint data that were collected in shopping malls. Due to the volatility of Wi-Fi signals and the high-dimensional sparseness of fingerprint data, we proposed a feature extraction algorithm, called joint multi-task stacked denoising auto-encoder (JMT-SDAE), aiming at reducing the dimensionality of the original fingerprint data and improving the indoor positioning performance in shopping malls. Furthermore, the features extracted by JMT-SDAE and gradient boosting decision tree (GBDT) were merged to construct a hybrid model, named as JMT-SDAE+GBDT. The experimental results based on 13 location datasets showed that the proposed feature fusion model had better positioning accuracy when compared with other existing positioners, and thus confirmed the effectiveness of our proposed feature extraction algorithm through multi-task learning.

An Enhanced Facial Expression Recognition Model Using Local Feature Fusion of Gabor Wavelets and Local Directionality Patterns

Facial expression analysis and recognition has gained popularity in the last few years for its challenging nature and broad area of applications like HCI, pain detection, operator fatigue detection, surveillance, etc. The key of real-time FER system is exploiting its variety of features extracted from the source image. In this article, three different features viz. local binary pattern, Gabor, and local directionality pattern were exploited to perform feature fusion and two classification algorithms viz. support vector machines and artificial neural networks were used to validate the proposed model on benchmark datasets. The classification accuracy has been improved in the proposed feature fusion of Gabor and LDP features with SVM classifier, recorded an average accuracy of 93.83% on JAFFE, 95.83% on CK and 96.50% on MMI. The recognition rates were compared with the existing studies in the literature and found that the proposed feature fusion model has improved the performance.

Long-distance deformation object recognition by integrating contour structure and scale-invariant heat kernel signature

The tough challenges of object recognition in long-distance scene involves contour shape deformation invariant features construction. In this work, an effective contour shape descriptor integrating critical points structure and Scale-invariant Heat Kernel Signature (SI-HKS) is proposed for long-distance object recognition. We firstly propose a general feature fusion model. Then, we capture the object contour structure feature with Critical-points Inner-distance Shape Context (CP-IDSC). Meanwhile, we pull-in the SI-HKS for capturing the local deformation-invariant properties of 2D shape. Based on the integration of the above two feature descriptors, the fusion descriptor is compacted by mapping into a low dimensional subspace using the bags-of-features, allowing for an efficient Bayesian classifier recognition. The extensive experiments on synthetic turbulence-degraded shapes and real-life infrared image show that the proposed method outperformed other compared approaches in terms of the recognition precision and robustness.

A study of a clothing image segmentation method in complex conditions using a features fusion model

According to a priori knowledge in complex conditions, this paper proposes an unsupervised image segmentation algorithm to be used for clothing images that combines colour and texture features. First, block truncation encoding is used to divide the traditional three-dimensional colour space into a six-dimensional colour space so that more fine colour features can be obtained. Then, a texture feature based on the improved local binary pattern (LBP) algorithm is designed and used to describe the clothing image with the colour features. After that, according to the statistical appearance law of the object region and background information in the clothing image, a bisection method is proposed for the segmentation operation. Since the image is divided into several subimage blocks, bisection image segmentation will be accomplished more efficiently. The experimental results show that the proposed algorithm can quickly and effectively extract effective clothing regions from complex circumstances without any artificial parameters. The proposed clothing image segmentation method will play an important role in computer vision, machine learning applications, pattern recognition and intelligent systems.

Bridge Crack Identification Based on Feature Fusion of Convolutional Neural Networks

Aiming at the problem of low image quality of bridge cracks and poor identification of bridge cracks by single convolutional neural network method, this paper proposes a bridge crack identification algorithm based on feature fusion of convolutional neural networks. The algorithm first augments the image to obtain enough training samples to simulate the real scene. Then, using super-resolution technology to reconstruct the image size to enhance the image details; then, using AlexNet and VGG11 to construct and train the bridge crack feature fusion model. Finally, the SoftMax classifier is used to classify and identify the merged features to obtain the crack detection results. The experimental results show that the proposed algorithm is better than the single AlexNet and VGG11 feature extraction, and the image recognition accuracy is increased by 0.32% and 0.41% respectively. The loss value is smaller under the same iteration number, and the improved algorithm is better.

Feature Fusion Text Classification Model Combining CNN and BiGRU with Multi-Attention Mechanism

Convolutional neural networks (CNN) and long short-term memory (LSTM) have gained wide recognition in the field of natural language processing. However, due to the pre- and post-dependence of natural language structure, relying solely on CNN to implement text categorization will ignore the contextual meaning of words and bidirectional long short-term memory (BiLSTM). The feature fusion model is divided into a multiple attention (MATT) CNN model and a bi-directional gated recurrent unit (BiGRU) model. The CNN model inputs the word vector (word vector attention, part of speech attention, position attention) that has been labeled by the attention mechanism into our multi-attention mechanism CNN model. Obtaining the influence intensity of the target keyword on the sentiment polarity of the sentence, and forming the first dimension of the sentiment classification, the BiGRU model replaces the original BiLSTM and extracts the global semantic features of the sentence level to form the second dimension of sentiment classification. Then, using PCA to reduce the dimension of the two-dimensional fusion vector, we finally obtain a classification result combining two dimensions of keywords and sentences. The experimental results show that the proposed MATT-CNN+BiGRU fusion model has 5.94% and 11.01% higher classification accuracy on the MRD and SemEval2016 datasets, respectively, than the mainstream CNN+BiLSTM method.

Postoperative glioma segmentation in CT image using deep feature fusion model guided by multi-sequence MRIs.

Computed tomography (CT) and magnetic resonance imaging (MRI) are the most commonly selected methods for imaging gliomas. Clinically, radiotherapists always delineate the CT glioma region with reference to multi-modal MR image information. On this basis, we develop a deep feature fusion model (DFFM) guided by multi-sequence MRIs for postoperative glioma segmentation in CT images. DFFM is a multi-sequence MRI-guided convolutional neural network (CNN) that iteratively learns the deep features from CT images and multi-sequence MR images simultaneously by utilizing a multi-channel CNN architecture, and then combines these two deep features together to produce the segmentation result. The whole network is optimized together via a standard back-propagation. A total of 59 CT and MRI datasets (T1/T2-weighted FLAIR, T1-weighted contrast-enhanced, T2-weighted) of postoperative gliomas as tumor grade II (n = 24), grade III (n = 18), or grade IV (n = 17) were included. Dice coefficient (DSC), precision, and recall were used to measure the overlap between automated segmentation results and manual segmentation. The Wilcoxon signed-rank test was used for statistical analysis. DFFM showed a significantly (p < 0.01) higher DSC of 0.836 than U-Net trained by single CT images and U-Net trained by stacking the CT and multi-sequence MR images, which yielded 0.713 DSC and 0.818 DSC, respectively. The precision values showed similar behavior as DSC. Moreover, DSC and precision values have no significant statistical difference (p > 0.01) with difference grades. DFFM enables the accurate automated segmentation of CT postoperative gliomas of profit guided by multi-sequence MR images and may thus improve and facilitate radiotherapy planning. • A fully automated deep learning method was developed to segment postoperative gliomas on CT images guided by multi-sequence MRIs. • CT and multi-sequence MR image integration allows for improvements in deep learning postoperative glioma segmentation method. • This deep feature fusion model produces reliable segmentation results and could be useful in delineating GTV in postoperative glioma radiotherapy planning.

Retracted: Multiscale fast correlation filtering tracking algorithm based on a feature fusion model

SummaryRetraction: Multiscale fast correlation filtering tracking algorithm based on a feature fusion model Yuantao Chen, Jin Wang, Songjie Liu, Xi Chen, Jie Xiong, Jingbo Xie, Kai Yang, 2021, 33 (15), (https://doi.org/10.1002/cpe.5533

Deep Feature Fusion for High-Resolution Aerial Scene Classification

The rapid development of remote sensing technology let us acquire a large collection of remote sensing scene images with high resolution. Aerial scene classification has become a crucial problem for understanding high-resolution remote sensing imagery. In this letter, we propose a novel framework for aerial scene classification. Unlike some traditional methods in which the features are produced by using handcrafted feature descriptors, our proposed method uses the raw RGB network stream and the saliency coded network stream to extract two different types of informative features. Then, we further propose a deep feature fusion model to fuse these two sets of features for final classification. The comprehensive performance evaluation of our proposed method is tested on two publicly available remote sensing scene classification benchmarks, i.e., the UC-Merced dataset and the AID dataset. Experimental results show that our proposed method achieves satisfactory results and outperforms the state-of-the-art approaches.

Multi-level feature fusion model-based real-time person re-identification for forensics

Person forensics aims to retrieve the specified person across non-overlapping cameras. It is difficult owing to the appearance variations caused by occlusion, human pose change, background clutter, illumination variation, etc. In this scenario, current models face great challenges in extracting effective features. Recent deep learning models mainly focus on extracting representative deep features to cope with appearance variations, while handcrafted features are not fully explored. In this paper, a multi-level feature fusion model (MFFM) is designed to combine both deep features and handcrafted features in real time. MFFM is first utilized to describe person appearance. Then, local binary pattern (LBP) and histogram of oriented gradient (HOG) are extracted to cope with geometric change and illumination variance. Using LBP and HOG, 11.89% on the CUHK03, 15.30% on the Market-1501 and 8.25% on the VIPeR top-1 recognition accuracy improvement for the proposed method are achieved with only 9.66%, 4.90%, and 7.59% extra processing time. Experimental results indicate MFFM can achieve the best performance compared to the state-of-the-art models on the Market1501, CUHK03, and VIPeR datasets.

Feature Fusion With Predictive Weighting for Spectral Image Classification and Segmentation

In this paper, we propose a spatial-spectral feature fusion model with a predictive feature weighting mechanism and demonstrate its applications to the problems of hyperspectral image classification and segmentation. To address these problems, we learn a set of 1-D convolutional local spectral filters and 2-D spatial-spectral filters that feed features into a fusion module, in an end-to-end fashion. We propose a lightweight predictive feature weighting component embedded in the fusion model and consider four design fusion options, i.e., by adding or concatenating features with equal or predicted weights. For the pixel classification task, the training input consists of image patches with labeled central pixels, whereas for the spatial segmentation task, it includes the label maps of image regions. The proposed networks have been evaluated on the Indian Pines, Pavia University, and Houston University for the classification problem and the SpaceNet data set for the spatial segmentation problem. The quantitative results favor the proposed approach over the state-of-the-art methods across all the four data sets.

Leveraging deep neural networks for anomaly‐based web application firewall

Web applications are the most common platforms for the exchange of information and services on the Internet. With the launch of web 2.0, information has flourished through social networking and business online. Therefore, websites are often attacked directly. As a result, the industry has paid more attention to the security of web applications in addition to security under computer networks. Intelligent systems based on machine learning have demonstrated excellent result on tasks such as anomaly detection in web requests. However, current methods based on traditional models cannot extract high-level features from huge data. In this study, the authors proposed methods based on deep-neural-network and parallel-feature-fusion that features engineering as an integral part of them and plays the most important role in their performance. The proposed methods use stacked autoencoder and deep belief network as feature learning methods, in which only normal data is used in the classification of training phase, then, one-class SVM, isolation forest, and elliptic envelope are applied as classifiers. The authors compared the proposed model with different strategies on CSIC 2010 and ECML/PKDD 2007 datasets. Results show that deep model and feature fusion model demonstrated as hierarchical feature learning which had better performance in terms of accuracy and generalisation in a reasonable time.

Saliency detection based on self-adaptive multiple feature fusion for remote sensing images

ABSTRACTSaliency detection has been revealed an effective and reliable approach to extract the region of interest (ROI) in remote sensing images. However, most existing saliency detection methods employing multiple saliency cues ignore the intrinsic relationship between different cues and do not distinguish the diverse contributions of different cues to the final saliency map. In this paper, we propose a novel self-adaptively multiple feature fusion model for saliency detection in remote sensing images to take advantage of this relationship to improve the accuracy of ROI extraction. First, we take multiple feature channels, namely colour, intensity, texture and global contrast into consideration to produce primary feature maps. Particularly, we design a novel method based on dual-tree complex wavelet transform for remote sensing images to generate texture feature pyramids. Then, we introduce a novel self-adaptive multiple feature fusion method based on low-rank matrix recovery, in which the significances of feature maps are ranked by the low rank constraint recovery, and subsequently multiple features’ contributions are allocated adaptively to produce the final saliency map. Experimental results demonstrate that our proposal outperforms the state-of-the-art methods.

An Efficient Character-Level and Word-Level Feature Fusion Method for Chinese Text Classification

In order to extract semantic feature information between texts more efficiently and reduce the effect of text representation on classification results, we propose a features fusion model C_BiGRU_ATT based on deep learning. The core task of our model is to extract the context information and local information of the text using Convolutional Neural Network(CNN) and Attention-based Bidirectional Gated Recurrent Unit(BiGRU) at character-level and word-level. Our experimental results show that the classification accuracies of C_BiGRU_ATT reach 95.55% and 95.60% on two Chinese datasets THUCNews and WangYi respectively. Meanwhile, compared with the single model based on character-level and word-level for CNN, the classification accuracies of C_BiGRU_ATT is increased by 1.6%, 2.7% on the THUCNews, and is increased by 0.6%, 5.2% on the WangYi. The results show that the proposed model C_BiGRU_ATT can extract text features more effectively.