Efficient Feature Fusion Research Articles

A Small-Sized Object Detection Oriented Multi-Scale Feature Fusion Approach With Application to Defect Detection

Object detection is a well-known task in the field of computer vision, especially the small target detection problem that has aroused great academic attention. In order to improve the detection performance of small objects, in this article, a novel enhanced multiscale feature fusion method is proposed, namely, the atrous spatial pyramid pooling-balanced-feature pyramid network (ABFPN). In particular, the atrous convolution operators with different dilation rates are employed to make full use of context information, where the skip connection is applied to achieve sufficient feature fusions. In addition, there is a balanced module to integrate and enhance features at different levels. The performance of the proposed ABFPN is evaluated on three public benchmark datasets, and experimental results demonstrate that it is a reliable and efficient feature fusion method. Furthermore, in order to validate the applicational potential in small objects, the developed ABFPN is utilized to detect surface tiny defects of the printed circuit board (PCB), which acts as the neck part of an improved PCB defect detection (IPDD) framework. While designing the IPDD, several powerful strategies are also employed to further improve the overall performance, which is evaluated via extensive ablation studies. Experiments on a public PCB defect detection database have demonstrated the superiority of the designed IPDD framework against the other seven state-of-the-art methods, which further validates the practicality of the proposed ABFPN.

A Fast Intrusion Detection Method for High-Speed Railway Clearance Based on Low-Cost Embedded GPUs.

The efficiency and the effectiveness of railway intrusion detection are crucial to the safety of railway transportation. Most current methods of railway intrusion detection or obstacle detection are inappropriate for large-scale applications due to their high cost or limited coverage. In this study, we present a fast and low-cost solution to intrusion detection of high-speed railways. As the solution to heavy computational burdens in the current convolutional-neural-network-based detection methods, the proposed method is mainly a novel neural network based on the SSD framework, which includes a feature extractor using an improved MobileNet and a lightweight and efficient feature fusion module. In addition, aiming to improve the detection accuracy of small objects, the feature map weights are introduced through convolution operation to fuse features at different scales. TensorRT is employed to optimize and deploy the proposed network in the low-cost embedded GPU platform, NVIDIA Jetson TX2, to enhance the efficiency. The experimental results show that the proposed methods achieved 89% mAP on the railway intrusion detection dataset, and the average processing time for a single frame was 38.6 ms on the Jetson TX2 module, which satisfies the need of real-time processing.

Gesture Detection and Recognition Based on Pyramid Frequency Feature Fusion Module and Multiscale Attention in Human-Computer Interaction

Aiming at the problem of the absence of detail texture and other high-frequency features in the feature extraction process of the deep network employing the upsampling operation, the accuracy of gesture recognition is seriously affected in complex scenes. This study integrates object detection and gesture recognition into one model and proposes a gesture detection and recognition based on the pyramid frequency feature fusion module and multiscale attention in human-computer interaction. Pyramid fusion module is used to perform efficient feature fusion and is proposed to obtain feature layers with rich details and semantic information, which is helpful to improve the efficiency and accuracy of gesture recognition. In addition, the multiscale attention module is further adopted to adaptively mine important and effective feature information from both temporal and spatial channels and embedded into the detection layer. Finally, our proposed network realizes the enhancement of the effective information and the suppression of the invalid information of the detection layer. Experimental results show that our proposed model makes full use of the high-low frequency feature fusion module without replacing the basic backbone network, which can greatly reduce the computational overhead while improving the detection accuracy.

Real-time traffic sign detection network using DS-DetNet and lite fusion FPN

Traffic sign detection (TSD) using convolutional neural networks (CNN) is promising and intriguing for autonomous driving. Especially, with sophisticated large-scale CNN models, TSD can be performed with high accuracy. However, the conventional CNN models suffer the drawbacks of being time-consuming and resource-hungry, which limit their application and deployments in various platforms of limited resources. In this paper, we propose a novel real-time traffic sign detection system with a lightweight backbone network named Depth Separable DetNet (DS-DetNet) and a lite fusion feature pyramid network (LFFPN) for efficient feature fusion. The new model can achieve a performance trade-off between speed and accuracy using a depthwise separable bottleneck block, a lite fusion module, and an improved SSD detection front-end. The testing results on the MS COCO and the GTSDB datasets reveal that 23.1% mAP with 6.39 M parameters and only 1.08B FLOPs on MSCOCO, 81.35% mAP with 5.78 M parameters on GTSDB. With our model, the run speed is 61 frames per second (fps) on GTX 1080ti, 12 fps on Nvidia Jetson Nano and 16 fps on Nvidia Jetson Xavier NX.

Fractional Gabor Convolutional Network for Multisource Remote Sensing Data Classification

Remote sensing using multisensor platforms has been systematically applied for monitoring and optimizing human activities. Several advanced techniques have been developed to enhance and extract the spatially and spectrally semantic information in the hyperspectral image (HSI) and light detection and ranging (LiDAR) data processing and analysis. However, an abundance of redundant information and sometimes a lack of discriminative features reduce the efficiency and effectiveness of multisource classification methods. This article proposes a fractional Gabor convolutional network (FGCN), focusing on efficient feature fusion and comprehensive feature extraction. First, the proposed FGCN uses Octave convolution layers to perform multisource information fusion and preserve discriminative information. Second, fractional Gabor convolutional (FGC) layers are proposed to extract multiscale, multidirectional, and semantic change features. The completeness and discrimination of the multisource features using different FGC kernels are improved, which yield robust feature extraction against semantic changes. Finally, the fractional Gabor feature and spectral feature are combined with two weighting factors which can be learned during the network training. Experimental results and comparisons with state-of-the-art multisource classification methods indicate the effectiveness of the proposed FGCN. With the FGCN, we can obtain an 89.90% overall accuracy on the challenging Muufl Gulfport (MUUFL) data set, with an improvement of 3% over state-of-the-art methods.

Building Load Forecasting Using Deep Neural Network with Efficient Feature Fusion

The energy consumption of buildings has risen steadily in recent years. It is vital for the managers and owners of the building to manage the electric energy demand of the buildings. Forecasting electric energy consumption of the buildings will bring great profits, which is influenced by many factors that make it very difficult to provide an advanced forecasting. Recently, deep learning techniques are widely adopted to solve this problem. Deep neural network offers an excellent capability in handling complex non-linear relationships and competence in exploring regular patterns and uncertainties of consumption behaviors at the building level. In this paper, we propose a deep convolutional neural network based on ResNet for hour-ahead building load forecasting. In addition, we design a branch that integrates the temperature per hour into the forecasting branch. To enhance the learning capability of the model, an innovative feature fusion is presented. At last, sufficient ablation studies are conducted on the point forecasting, probabilistic forecasting, fusion method, and computation efficiency. The results show that the proposed model has the state-of-the-art performance, which reflects a promising prospect in application of the electricity market.

An Efficient Feature Fusion of Graph Convolutional Networks and Its Application for Real-Time Traffic Control Gestures Recognition

Recently, skeleton-based gesture and action recognition have emerged thanks to the progress in human pose estimation. Gesture representation using skeletal data is robust since skeletal data are invariant to the individual’s appearance. Among different approaches proposed for skeleton-based action/gesture recognition, Graph Convolutional Network (GCN) and its variations have obtained great attention thanks to its ability to capture the graph essence of the skeletal data. In this paper, we aim to design an efficient scheme using relative joints of skeleton sequences adapted in a GCN framework. Both spatial features (i.e., joint positions) and temporal ones (i.e., the velocity of joints) are combined to form the input of Attention-enhanced Adaptive GCN (AAGCN). The proposed framework deals with limitations of the original AAGCN when it works on challenging datasets with incomplete and noisy skeletal data. Extensive experiments are carried out on three datasets CMDFALL, MICA-Action3D, NTU-RGBD. Experimental results show that the proposed method achieves superior performance compared with existing methods. Moreover, to illustrate the application of the proposed method in real-time traffic control gesture recognition for autonomous vehicles, we have evaluated the proposed method on the TCG dataset. The obtained results show that the proposed method offers real-time computation capability and good recognition results. These results suggest a promising solution to deploy a real-time and robust recognition technique for gesture-based traffic control in autonomous vehicles.

ITP-Pred: an interpretable method for predicting, therapeutic peptides with fused features low-dimension representation.

The peptide therapeutics market is providing new opportunities for the biotechnology and pharmaceutical industries. Therefore, identifying therapeutic peptides and exploring their properties are important. Although several studies have proposed different machine learning methods to predict peptides as being therapeutic peptides, most do not explain the decision factors of model in detail. In this work, an Interpretable Therapeutic Peptide Prediction (ITP-Pred) model based on efficient feature fusion was developed. First, we proposed three kinds of feature descriptors based on sequence and physicochemical property encoded, namely amino acid composition (AAC), group AAC and coding autocorrelation, and concatenated them to obtain the feature representation of therapeutic peptide. Then, we input it into the CNN-Bi-directional Long Short-Term Memory (BiLSTM) model to automatically learn recognition of therapeutic peptides. The cross-validation and independent verification experiments results indicated that ITP-Pred has a higher prediction performance on the benchmark dataset than other comparison methods. Finally, we analyzed the output of the model from two aspects: sequence order and physical and chemical properties, mining important features as guidance for the design of better models that can complement existing methods.

Learning a Dynamic Feature Fusion Tracker for Object Tracking

Object tracking is a key component of self-driving systems and has important meanings to alleviate traffic accidents. Therefore, it is meaningful to design a high performance and real-time tracker for improving the stability and safety of self-driving systems. In this paper, an effective and efficient feature fusion tracker, which dynamically fuses gradient and color features to model the appearance of the target object, is designed with the correlation filters framework for fast tracking. To be specific, two complementary correlation filters for gradient (e.g. HOG) and color (e.g. ColorNames) features are maintained during tracking, and the proposed feature fusion method adaptively adjusts the weights of them to deal with large appearance changes of the target object in challenging tracking scenes. The weights are decided by the consistency of the final tracking result and the predicted results obtained by two correlation filters. Moreover, a failure detection scheme is designed to alleviate the model drift issue caused by undesirable model updates to improve the tracking accuracy. If a tracking result is identified as a failed case, re-detection operations are performed to accurately localize the target object. The experimental results prove that the proposed tracker can achieve competitive tracking performance and a satisfactory tracking speed of 25.3 FPS in comparison with several state-of-the-art trackers on challenging tracking benchmarks.

Enhancing feature fusion for human pose estimation

Current human pose estimation methods mainly rely on designing efficient Convolutional Neural Networks (CNN) frameworks. These CNN architectures typically consist of high-to-low resolution sub-networks to learn semantic information, and then followed by low-to-high sub-networks to raise the resolution to locate the keypoints. Because low-level features have high resolution but less semantic information, while high-level features have rich semantic information but less high resolution details, so it is important to fuse different level features to improve the final performance. However, most existing models implement feature fusion by simply concatenate low-level and high-level features without considering the gap between spatial resolution and semantic levels. In this paper, we propose a new feature fusion method for human pose estimation. We introduce high level semantic information into low-level features to enhance feature fusion. Further, to keep both the high-level semantic information and high-resolution location details, we use Global Convolutional Network blocks to bridge the gap between low-level and high-level features. Experiments on MPII and LSP human pose estimation datasets demonstrate that efficient feature fusion can significantly improve the performance. The code is available at: https://github.com/tongjiangwei/FeatureFusion .

Single-Shot Object Detection with Split and Combine Blocks

Feature fusion is widely used in various neural network-based visual recognition tasks, such as object detection, to enhance the quality of feature representation. It is common practice for both the one-stage object detectors and the two-stage object detectors to implement feature fusion in feature pyramid networks (FPN) to enhance the capacity to detect objects of different scales. In this work, we propose a novel and efficient feature fusion unit, which is referred to as the Split and Combine (SC) Block, that splits the input feature maps into several parts, then processes these sub-feature maps with different emphasis, and finally gradually concatenates the outputs one-by-one. The SC block implicitly encourages the network to focus on features that are more important to the task, thus improving network efficiency and reducing inference computations. In order to prove our analysis and conclusions, a backbone network and an FPN employing this technique are assembled into a one-stage detector and evaluated on the MS COCO dataset. With the newly introduced SC block and other novel training tricks, our detector achieves a good speed-accuracy trade-off on COCO test-dev set, with 37.1% AP (average precision) at 51 FPS and 38.9% AP at 40 FPS.

TDFSSD: Top-Down Feature Fusion Single Shot MultiBox Detector

Object detection across different scales is challenging as the variances of object scales. Thus, a novel detection network, Top-Down Feature Fusion Single Shot MultiBox Detector (TDFSSD), is proposed. The proposed network is based on Single Shot MultiBox Detector (SSD) using VGG-16 as backbone with a novel, simple yet efficient feature fusion module, namely, the Top-Down Feature Fusion Module. The proposed module fuses features from higher-level features, containing semantic information, to lower-level features, containing boundary information, iteratively. Extensive experiments have been conducted on PASCAL VOC2007, PASCAL VOC2012, and MS COCO datasets to demonstrate the efficiency of the proposed method. The proposed TDFSSD network is trained end to end and outperforms the state-of-the-art methods across the three datasets. The TDFSSD network achieves 81.7% and 80.1% mAPs on VOC2007 and 2012 respectively, which outperforms the reported best results of both one-stage and two-stage frameworks. In the meantime, it achieves 33.4% mAP on MS COCO test-dev, especially 17.2% average precision (AP) on small objects. Thus all the results show the efficiency of the proposed method on object detection. Code and model are available at: https://github.com/dongfengxijian/TDFSSD.

HyCAD-OCT: A Hybrid Computer-Aided Diagnosis of Retinopathy by Optical Coherence Tomography Integrating Machine Learning and Feature Maps Localization

Optical Coherence Tomography (OCT) imaging has major advantages in effectively identifying the presence of various ocular pathologies and detecting a wide range of macular diseases. OCT examinations can aid in the detection of many retina disorders in early stages that could not be detected in traditional retina images. In this paper, a new hybrid computer-aided OCT diagnostic system (HyCAD) is proposed for classification of Diabetic Macular Edema (DME), Choroidal Neovascularization (CNV) and drusen disorders, while separating them from Normal OCT images. The proposed HyCAD hybrid learning system integrates the segmentation of Region of Interest (RoI), based on central serious chorioretinopathy (CSC) in Spectral Domain Optical Coherence Tomography (SD-OCT) images, with deep learning architectures for effective diagnosis of retinal disorders. The proposed system assimilates a range of techniques including RoI localization and feature extraction, followed by classification and diagnosis. An efficient feature fusion phase has been introduced for combining the OCT image features, extracted by Deep Convolutional Neural Network (CNN), with the features extracted from the RoI segmentation phase. This fused feature set is used to predict multiclass OCT retina disorders. The proposed segmentation phase of retinal RoI regions adds substantial contribution as it draws attention to the most significant areas that are candidate for diagnosis. A new modified deep learning architecture (Norm-VGG16) is introduced integrating a kernel regularizer. Norm-VGG16 is trained from scratch on a large benchmark dataset and used in RoI localization and segmentation. Various experiments have been carried out to illustrate the performance of the proposed system. Large Dataset of Labeled Optical Coherence Tomography (OCT) v3 benchmark is used to validate the efficiency of the model compared with others in literature. The experimental results show that the proposed model achieves relatively high-performance in terms of accuracy, sensitivity and specificity. An average accuracy, sensitivity and specificity of 98.8%, 99.4% and 98.2% is achieved, respectively. The remarkable performance achieved reflects that the fusion phase can effectively improve the identification ratio of the urgent patients’ diagnostic images and clinical data. In addition, an outstanding performance is achieved compared to others in literature.

A New Single-Image Super-Resolution Using Efficient Feature Fusion and Patch Similarity in Non-Euclidean Space

Efficient trade-off between the reconstruction qualities and the processing time of any single-image super-resolution reconstruction (SISRR) approach is critically influenced by two major aspects. These aspects are (i) appropriate representation of image patch in feature space and (ii) effective searching of candidate patches from the pool of training patches or learned dictionary. This paper proposes a neighbor embedding-based SISRR method. Novelties of our work include integration of (i) efficient feature mapping scheme which fuses multiple correlated features naturally, (ii) faster searching of candidate patches by measuring the patch correlation in non-Euclidean space and (iii) adaptive selection of neighborhood size using patch characteristic. Correlation among features is modeled via global covariance matrix, and the fusion process enables to preserve sufficient structural, spatial correlation among patches. Distance functions based on notion of generalized eigenvalue are used for measuring patch similarity which support faster searching of candidate patches. Performance analysis of the suggested method is compared with some of the competent state-of-the-art methodologies. From the simulated result analysis, proposed work is found to be outperforming in terms of sharpened image details with diminished effect of artifacts at a reasonable computational burden.

An Improved Deep Canonical Correlation Fusion Method for Underwater Multisource Data

In complex underwater environments, the single mode of a single sensor cannot meet the precision requirement of object identification, and multisource fusion is currently the mainstream research approach. Deep canonical correlation analysis is an efficient feature fusion method but suffers from problems such as not strong scalability and low efficiency. Therefore, an improved deep canonical correlation analysis fusion method is proposed for underwater multisource sensor data containing noise. First, a denoising autoencoder is used for denoising and to reduce the data dimension to extract new feature expressions of raw data. Second, given that underwater acoustic data can be characterized as 1-dimensional time series, a 1-dimensional convolutional neural network is used to improve the deep canonical correlation analysis model, and multilayer convolution and pooling are implemented to decrease the number of parameters and increase the efficiency. To improve the scalability and robustness of the model, a stochastic decorrelation loss function is used to optimize the objective function, which reduces the algorithm complexity from O(n 3 ) to O(n 2 ). The comparison experiment of the proposed algorithm and other typical algorithms on MNIST containing noise and underwater multisource data in different scenes shows that the proposed algorithm is superior to others regardless of the efficiency or precision of target classification.

3D object detection based on sparse convolution neural network and feature fusion for autonomous driving in smart cities

People in cities are suffering from traffic congestion and air pollution in daily life partly due to a great number of private cars, and always face the danger of accidents, so autonomous driving is developed by many institutes and companies especially in recent years. Autonomous driving will play an import role in the future smart cities, reduce the time and economic cost of the whole society, and be helpful for the sustainability of the city and society. A significant task for autonomous driving is to detect surrounding objects accurately in real-time, including car, pedestrian, cyclist, etc. In this paper, we propose one end-to-end three dimensional (3D) object detection method based on voxelization, sparse convolution, and feature fusion. The proposed method exploits only point cloud as input, and it has two key components—small voxels and efficient feature fusion. Instead of utilizing extra networks to transform voxels, we directly average the points within each voxel as their feature representation. To enrich features for prediction, we have designed a two-step feature fusion method called fusion of fusion network that can combine information of multiple scales and 3D space. We have submitted to the official test server of the 3D detection benchmark—KITTI, and achieved state-of-the-art performance especially in the Cyclist class. Besides, detection speed of our method achieves 0.05 s/frame with a 2–4 fold runtime improvement against state-of-the-art methods due to its simple and compact architecture.

Residual Attribute Attention Network for Face Image Super-Resolution

Facial prior knowledge based methods recently achieved great success on the task of face image super-resolution (SR). The combination of different type of facial knowledge could be leveraged for better super-resolving face images, e.g., facial attribute information with texture and shape information. In this paper, we present a novel deep end-to-end network for face super resolution, named Residual Attribute Attention Network (RAAN), which realizes the efficient feature fusion of various types of facial information. Specifically, we construct a multi-block cascaded structure network with dense connection. Each block has three branches: Texture Prediction Network (TPN), Shape Generation Network (SGN) and Attribute Analysis Network (AAN). We divide the task of face image reconstruction into three steps: extracting the pixel level representation information from the input very low resolution (LR) image via TPN and SGN, extracting the semantic level representation information by AAN from the input, and finally combining the pixel level and semantic level information to recover the high resolution (HR) image. Experiments on benchmark database illustrate that RAAN significantly outperforms state-of-the-arts for very low-resolution face SR problem, both quantitatively and qualitatively.

An efficient feature fusion in HSI image classification

In recent times, the fusion of spatial relaxation with spectral data has achieved remarkable success in target classification methods. Spatial relaxation is a scheme which exploits the neighbourhood relationship between the pixels of an image to minimize the spatio-spectral distortion. Application of spatial relaxation with spectral data can lead to reduce the noise effect and increase the class characterization. Such methods can also be applied to estimate the posteriors of the probabilistic classifier, to increase the classifier’s final accuracy. In this paper, we have introduced an edge based feature fusion method which helps in characterizing the class labels of hyperspectral image (HSI) in a better sense. It is an iterative method which exploits the spatial information from an image in such a manner that it assumes the feature preservation in vertical and horizontal directions for each pixel. With combining subspace regression based probabilistic method, the proposed method gives better accuracy for benchmark HSI datasets. Before this, we have implemented a fast Bayesian subspace regression method to achieve the posterior probabilities, for our edge feature relaxation method. Finally, we have compared the results with some recently proposed methods, and $$\alpha $$ expansion graph cut optimization method, which is an efficient technique to fuse the contextual knowledge in posterior probabilities.

An Efficient Object Detection Algorithm Based on Compressed Networks

For a long time, object detection has been a popular but difficult research problem in the field of pattern recognition. In recent years, object detection algorithms based on convolutional neural networks have achieved excellent results. However, neural networks are computationally intensive and parameter redundant, so they are difficult to deploy on resource-limited embedded devices. Especially for two-stage detectors, operations and parameters are mainly clustered on feature fusion of proposals after the region of interest (ROI) pooling layer, and they are enormous. In order to deal with these problems, we propose a subnetwork—efficient feature fusion module (EFFM) to reduce the number of operations and parameters for a two-stage detector. In addition, we propose a multi-scale dilation region proposal network (RPN) to further improve detection accuracy. Finally, our accuracy is higher than Faster RCNN based on VGG16, the number of operations is only half of the latter, and the number of parameters is only one third.

FEATURE FUSION TECHNIQUE FOR COLOUR TEXTURE CLASSIFICATION SYSTEM BASED ON GRAY LEVEL CO-OCCURRENCE MATRIX

In this study, an efficient feature fusion based technique for the classification of colour texture images in VisTex album is presented. Gray Level Co-occurrence Matrix (GLCM) and its associated texture features contrast, correlation, energy and homogeneity are used in the proposed approach. The proposed GLCM texture features are obtained from the original colour texture as well as the first non singleton dimension of the same image. These features are fused at feature level to classify the colour texture image using nearest neighbor classifier. The results demonstrate that the proposed fusion of difference image GLCM features is much more efficient than the original GLCM features.