Hierarchical Convolutional Features Research Articles

An Enhanced Visual Object Tracking Approach using dual tree complex wavelet transform and Hierarchical Convolutional Features

This paper study introduces a novel technique to enhance object tracking algorithms by utilizing dual tree complex wavelet transform for extracting hierarchical convolutional features from images. The method replaces the image template in the input of VGG-Net-19 with wavelet coefficients to generate hierarchical convolutional features using five layers. While the last convolutional layers provide robust semantic information of targets and are resistant to appearance variations, their spatial resolution is insufficient for precise localization. However, earlier convolutional layers offer more accurate localization but are vulnerable to appearance changes. The study considers the convolutional layer hierarchies as a nonlinear equivalent of an image pyramid representation and leverages these multiple levels of abstraction for visual tracking. The proposed method learns adaptive correlation filters on each convolutional layer to encode the target appearance and hierarchically infers the maximum response of each layer to locate targets. The study evaluates the approach with specific performance metrics and benchmark datasets, including OTB50, OTB100, TC128, and UAV20, and the results demonstrate that the proposed approach surpasses the performance of state-of-the-art tracking methods, indicating its efficacy in addressing the challenges encountered in visual object tracking.

Deep Convolutional Compressed Sensing-Based Adaptive 3D Reconstruction of Sparse LiDAR Data: A Case Study for Forests

LiDAR point clouds are characterized by high geometric and radiometric resolution and are therefore of great use for large-scale forest analysis. Although the analysis of 3D geometries and shapes has improved at different resolutions, processing large-scale 3D LiDAR point clouds is difficult due to their enormous volume. From the perspective of using LiDAR point clouds for forests, the challenge lies in learning local and global features, as the number of points in a typical 3D LiDAR point cloud is in the range of millions. In this research, we present a novel end-to-end deep learning framework called ADCoSNet, capable of adaptively reconstructing 3D LiDAR point clouds from a few sparse measurements. ADCoSNet uses empirical mode decomposition (EMD), a data-driven signal processing approach with Deep Learning, to decompose input signals into intrinsic mode functions (IMFs). These IMFs capture hierarchical implicit features in the form of decreasing spatial frequency. This research proposes using the last IMF (least varying component), also known as the Residual function, as a statistical prior for capturing local features, followed by fusing with the hierarchical convolutional features from the deep compressive sensing (CS) network. The central idea is that the Residue approximately represents the overall forest structure considering it is relatively homogenous due to the presence of vegetation. ADCoSNet utilizes this last IMF for generating sparse representation based on a set of CS measurement ratios. The research presents extensive experiments for reconstructing 3D LiDAR point clouds with high fidelity for various CS measurement ratios. Our approach achieves a maximum peak signal-to-noise ratio (PSNR) of 48.96 dB (approx. 8 dB better than reconstruction without data-dependent transforms) with reconstruction root mean square error (RMSE) of 7.21. It is envisaged that the proposed framework finds high potential as an end-to-end learning framework for generating adaptive and sparse representations to capture geometrical features for the 3D reconstruction of forests.

Reinforced Tracker Based on Hierarchical Convolutional Features

Reinforced Tracker Based on Hierarchical Convolutional Features

Improved Hierarchical Convolutional Features for Robust Visual Object Tracking

The target and background will change continuously in the long-term tracking process, which brings great challenges to the accurate prediction of targets. The correlation filter algorithm based on manual features is difficult to meet the actual needs due to its limited feature representation ability. Thus, to improve the tracking performance and robustness, an improved hierarchical convolutional features model is proposed into a correlation filter framework for visual object tracking. First, the objective function is designed by lasso regression modeling, and a sparse, time-series low-rank filter is learned to increase the interpretability of the model. Second, the features of the last layer and the second pool layer of the convolutional neural network are extracted to realize the target position prediction from coarse to fine. In addition, using the filters learned from the first frame and the current frame to calculate the response maps, respectively, the target position is obtained by finding the maximum response value in the response map. The filter model is updated only when these two maximum responses meet the threshold condition. The proposed tracker is evaluated by simulation analysis on TC-128/OTB2015 benchmarks including more than 100 video sequences. Extensive experiments demonstrate that the proposed tracker achieves competitive performance against state-of-the-art trackers. The distance precision rate and overlap success rate of the proposed algorithm on OTB2015 are 0.829 and 0.695, respectively. The proposed algorithm effectively solves the long-term object tracking problem in complex scenes.

Hierarchical Convolutional Features Fusion for Visual Tracking

Hierarchical convolutional features have different impact on the tracking performance, as the higher convolutional layers encode the semantic information of targets and earlier convolutional layers are more precise to localize targets. In this paper, we propose a novel scheme for hierarchical convolutional features fusion for visual tracking. In the proposed scheme, hierarchical convolutional features are first concatenated to form the cascading feature at the feature level, and then a convolutional layer is added to reduce the feature dimension. Discriminative correlation filter (DCF) is finally utilized to obtain the target location, which is treated as a differentiable layer in the neural network. The experimental results demonstrate that our proposed scheme achieves superior performances on the visual tracking benchmark.

Developing a feature decoder network with low-to-high hierarchies to improve edge detection

Low-to-high hierarchical convolutional features can significantly improve edge detection. This paper proposes a feature decoder-based algorithm that employs a Feature Decoder Network (FDN) to extract more information within limited Convolutional Neural Network (CNN) features. Previous studies applied convolutional elements by weight fusion, but we measure a feature decoder as a pyramid by qualifying convolutional layers. The feature decoder fuses CNN features of adjacent layers to judge the edge and non-edge pixels, which can learn the relationship and distinction between low-level edge hierarchies and high-level semantic hierarchies. Furthermore, we use Gaussian blur labels to train the network to optimize network convergence and training. From the experimental results, our proposed algorithm performs better on the BSDS500 (average accuracy (AP) of 0.865) and NYUD (OIS F-measure of 0.775) datasets compared to the state-of-the-art algorithms, including RCF.

Efficient object tracking using hierarchical convolutional features model and correlation filters

Visual object tracking is a very important task in computer vision. This paper develops a method based on the convolutional neural network (CNN) and correlation filters for visual object tracking. To implement a superior tracking method, we develop a multiple correlation tracker. This paper presents an effective method to track an object based on a combination of feature hierarchies of CNNs. We combine several feature hierarchies and compute the more discriminative map to track the object. Firstly, the correlation filters framework is selected to build the new tracker. Secondly, three feature maps from the CNN, which are inserted into the correlation filters framework, are adopted to evaluate the object location independently. Finally, a novel method of feature hierarchies integration based on Kullback–Leibler (KL) divergence is adopted. Experiments on the different sequences are carried out, and the outputs reveal that the proposed tracker has better results than those of the state-of-the-art methods, and it has the ability to handle various challenges.

A mosaic method for multi-temporal data registration by using convolutional neural networks for forestry remote sensing applications

Image registration is one of the most important processes for the generation of remote sensing image mosaics. This paper focuses on the special problems related to remote sensing data registration, and multi-temporal data mosaic applications in the domain of forestry. It proposes an image registration method based on hierarchical convolutional features, and applies it to improve the efficiency of large scale forestry image mosaic generation. This method uses a deep learning architecture to adaptively obtain image features from deep convolutional neural networks. The features derived from different images at different depth are sent to a correlation filter to compute the similarity between them; then the locations of the feature points are computed precisely. Based on this method, we study forestry image registration and the mosaic framework. We apply our approach to remote sensing images under different weather and seasonal conditions, and compare the results with those generated using the traditional SIFT image mosaic method. The experimental result shows that our method can detect and match the image feature points with significant spectral difference, and effectively extract feature points to generate accurate image registration and mosaic results. This demonstrates the effectiveness and robustness of the proposed approach.

Beyond feature integration: a coarse-to-fine framework for cascade correlation tracking

Discriminative correlation filters (DCF) have achieved enormous popularity in the tracking community. Recently, the performance advancement in DCF-based trackers is predominantly driven by the use of convolutional features. In pursuit of extreme tracking performance, state-of-the-art trackers (e.g., cascade correlation tracking [1] and HCF [2]) equip DCF with hierarchical convolutional features to capture both semantics and spatial details of the target appearance. While such methods have been shown to work well, multiple feature integration results in high model complexity which significantly increases the over-fitting risk and computational burden. In this paper, we present a coarse-to-fine framework for cascade correlation tracking (CCT). Instead of integrating hierarchical features, this framework decomposes a complicated tracker into two low-complexity modules, a coarse tracker $${\mathcal {C}}$$ and a refined tracker $${\mathcal {R}}$$ , working in a coarse-to-fine manner. The coarse tracker $${\mathcal {C}}$$ employs low-resolution semantic convolutional features extracted from a large search area to cope with large target displacement and appearance change between adjacent frames. By contrast, the refined tracker $${\mathcal {R}}$$ employs high-resolution handcraft features extracted from a small search area to further refine the coarse location of $${\mathcal {C}}$$ . Our CCT tracker enjoys the strong discriminative power of $${\mathcal {C}}$$ and the high efficiency of $${\mathcal {R}}$$ . Experiments on the OTB2013 and TC128 benchmarks show that CCT performs favorably against state-of-the-art trackers.

Context-aware long-term correlation tracking with hierarchical convolutional features

Visual tracking is of great significance in computer vision. Utilization of efficient correlation filters and discriminative convolutional features has gained much attention. In this paper, we propose a long–term tracking method (LHCAT) to deal with appearance variation caused by occlusion, deformation, out–of–view, etc. Firstly, we integrate context features into correlation filters by sampling surrounding patches around the target. Secondly, we reformulate the HCFT tracker with modified correlation filters. Then a HOG based scale correlation filter is applied to search target pyramids cropped around target position for optimal scale. In case of tracking failure and heavy occlusion, we train a SVM based module to re–detect the target. Additionally, for computational efficiency, proper strategy is proposed to determine when to activate the re–detector and update correlation filters. Experimental results on OTB–50 and OTB–100 show that our algorithm is of effectiveness and robustness in case of heavy appearance changes.

End-to-End Feature Integration for Correlation Filter Tracking With Channel Attention

Recently, the performance advancement of discriminative correlation filter (DCF) based trackers is predominantly driven by the use of deep convolutional features. As convolutional features from multiple layers capture different target information, existing works integrate hierarchical convolutional features to enhance target representation. However, these works separate feature integration from DCF learning and hardly benefit from end-to-end training. In this letter, we incorporates feature integration and DCF learning in a unified convolutional neural network. This network reformulates feature integration as a differential module that concatenates features from the shallow and deep layers. A channel attention mechanism is introduced to adaptively impose channel-wise weight on the integrated features. Experimental results on OTB100 and UAV123 demonstrate that our method achieves significant performance improvement while running in real-time.

Hierarchical convolutional features for visual tracking via two combined color spaces with SVM classifier

As the state-of-the-art object trackers majority, hierarchical convolutional features (HCF) cannot recover tracking processes from problems of drifting caused by several challenges, especially by heavy occlusion, scale variation, and illumination variation. In this paper, we present a new effective method with the aim of treating these challenges robustly based on two principal tasks. First, we infer the target location using multichannel correlation maps, resulting from the combination of five learned correlation filters with convolutional features. In order to handle the illumination variation and get more rich features, we exploit an HSV energy condition to control the use of two color spaces, RGB and HSV. Second, we use the histogram of gradient features to learn another correlation filter in order to estimate the scale variation. Furthermore, we exploit an online training SVM classifier for target re-detecting in failure cases. The extensive experiments on a commonly used tracking benchmark dataset justify that our tracker significantly improves HCF and outperforms the state-of-the-art methods.

MoE-SPNet: A mixture-of-experts scene parsing network

Scene parsing is an indispensable component in understanding the semantics within a scene. Traditional methods rely on handcrafted local features and probabilistic graphical models to incorporate local and global cues. Recently, methods based on fully convolutional neural networks have achieved new records on scene parsing. An important strategy common to these methods is the aggregation of hierarchical features yielded by a deep convolutional neural network. However, typical algorithms usually aggregate hierarchical convolutional features via concatenation or linear combination, which cannot sufficiently exploit the diversities of contextual information in multi-scale features and the spatial inhomogeneity of a scene. In this paper, we propose a mixture-of-experts scene parsing network (MoE-SPNet) that incorporates a convolutional mixture-of-experts layer to assess the importance of features from different levels and at different spatial locations. In addition, we propose a variant of mixture-of-experts called the adaptive hierarchical feature aggregation (AHFA) mechanism which can be incorporated into existing scene parsing networks that use skip-connections to fuse features layer-wisely. In the proposed networks, different levels of features at each spatial location are adaptively re-weighted according to the local structure and surrounding contextual information before aggregation. We demonstrate the effectiveness of the proposed methods on two scene parsing datasets including PASCAL VOC 2012 and SceneParse150 based on two kinds of baseline models FCN-8s and DeepLab-ASPP.

Hierarchical convolutional features for end-to-end representation-based visual tracking

Recently, deep learning is widely developed in computer vision applications. In this paper, a novel simple tracker with deep learning is proposed to complete the tracking task. A simple fully convolutional Siamese network is applied to capture the similarity between different frames. Nevertheless, the detailed information from lower layers, which is also important for locating the target object, is not considered into the tracking task. In this paper, the detailed information from two lower layers is considered into the response map to improve the performance and not to increase much time spent. This leads more significant improvement for feature representation and localization of the target object. The experimental results demonstrate that the proposed algorithm is efficient and robust compared with the baseline and the state-of-the-art trackers.

Robust Visual Tracking via Semiadaptive Weighted Convolutional Features

In recent years, hierarchical features extracted from convolutional neural network (CNN) for robust visual tracking have been developed by several methods. As features from different layers characterize different information of target and are set fixed weighted parameters, the performance of traditional visual tracking methods based on CNN can be further improved. In this letter, we propose a novel online visual tracking method by using hierarchical convolutional features with semiadaptive weights. The responses from different layers are assessed by a novel loss function. It considers the log likelihood and the entropy term of each response. The layer with the lower loss value is set to a higher weight parameter and the layer with the higher loss value is set to a lower one. We further develop a target appearance pyramid to deal with the scale change and an online classifier to redetect targets in case of tracking failures. Extensive experiments on challenging videos demonstrate that our method can achieve better tracking results in terms of lower center location error and higher overlap rate.

Hierarchical Recurrent Neural Hashing for Image Retrieval With Hierarchical Convolutional Features.

Hashing has been an important and effective technology in image retrieval due to its computational efficiency and fast search speed. The traditional hashing methods usually learn hash functions to obtain binary codes by exploiting hand-crafted features, which cannot optimally represent the information of the sample. Recently, deep learning methods can achieve better performance, since deep learning architectures can learn more effective image representation features. However, these methods only use semantic features to generate hash codes by shallow projection but ignore texture details. In this paper, we proposed a novel hashing method, namely hierarchical recurrent neural hashing (HRNH), to exploit hierarchical recurrent neural network to generate effective hash codes. There are three contributions of this paper. First, a deep hashing method is proposed to extensively exploit both spatial details and semantic information, in which, we leverage hierarchical convolutional features to construct image pyramid representation. Second, our proposed deep network can exploit directly convolutional feature maps as input to preserve the spatial structure of convolutional feature maps. Finally, we propose a new loss function that considers the quantization error of binarizing the continuous embeddings into the discrete binary codes, and simultaneously maintains the semantic similarity and balanceable property of hash codes. Experimental results on four widely used data sets demonstrate that the proposed HRNH can achieve superior performance over other state-of-the-art hashing methods.Hashing has been an important and effective technology in image retrieval due to its computational efficiency and fast search speed. The traditional hashing methods usually learn hash functions to obtain binary codes by exploiting hand-crafted features, which cannot optimally represent the information of the sample. Recently, deep learning methods can achieve better performance, since deep learning architectures can learn more effective image representation features. However, these methods only use semantic features to generate hash codes by shallow projection but ignore texture details. In this paper, we proposed a novel hashing method, namely hierarchical recurrent neural hashing (HRNH), to exploit hierarchical recurrent neural network to generate effective hash codes. There are three contributions of this paper. First, a deep hashing method is proposed to extensively exploit both spatial details and semantic information, in which, we leverage hierarchical convolutional features to construct image pyramid representation. Second, our proposed deep network can exploit directly convolutional feature maps as input to preserve the spatial structure of convolutional feature maps. Finally, we propose a new loss function that considers the quantization error of binarizing the continuous embeddings into the discrete binary codes, and simultaneously maintains the semantic similarity and balanceable property of hash codes. Experimental results on four widely used data sets demonstrate that the proposed HRNH can achieve superior performance over other state-of-the-art hashing methods.

Learning spatially correlation filters based on convolutional features via PSO algorithm and two combined color spaces for visual tracking

Last years, we have seen an emergence of wide methods in visual object tracking topic as convolutional neural network combined with correlation filter such as hierarchical features (HCF) (Ma et al. 20). However, upon the fact that some features may cause the tracking failures, the existing methods are still suffering of handling complex object appearance changes such as fast motion, significant deformation and occlusions. Further, they learn the correlation filter in frequency domain using Fourier transform, which cause unwanted boundary effects, which severely degrade the quality of the tracking model. Moreover, these methods are incapable of dealing with the illumination variation because they rely only on RGB base for color sequences. In this paper, we propose a novel method, which addresses the pre-cited problems. As first contribution, we learn adaptively three correlation filters in the spatial domain, with hierarchical convolutional features extracted from specific layers. Indeed, we apply the Particle Swarm Optimization algorithm to solve the update model equation of the correlation filters. Second, we propose that the switching between RGB and HSV color bases, give a soft manner to handle the illumination variation. For this aim, an HSV-energy condition is presented to choose the appropriate color base resorting to the energy of the second HSV component. Extensive experiments on a common benchmark dataset, justify that the proposed method outperforms the state-of-art methods.

Robust Scale Adaptive Kernel Correlation Filter Tracker With Hierarchical Convolutional Features

Visual object tracking is a challenging task due to object appearance changes caused by shape deformation, heavy occlusion, background clutters, illumination variation, and camera motion. In this letter, we propose a novel robust algorithm which decomposes the task of tracking into translation and scale estimation. We estimate the translation by using five correlation filters with hierarchical convolutional features which produced multilevel correlation response maps to collaboratively infer the target location. We also calculate the scale variation by another correlation filter with histogram of oriented gradient features at the same time. Extensive experimental results on a large-scale 50 challenging benchmark dataset show that the proposed algorithm achieved outstanding performance against state-of-the-art methods.