Pyramidal Feature Hierarchy Research Articles

Superframe-Based Temporal Proposals for Weakly Supervised Temporal Action Detection

The weakly supervised Temporal Action Detection (TAD) by using the video-level annotations can lighten the burden of labor consumption. However, the current methods for weakly supervised TAD do not take full advantages of the short-term consistency between consecutive frames and the long-term continuity inside an action, resulting in less accurate detecting boundaries of actions in untrimmed videos. In this paper, the SuperFrame-based Temporal Proposal (SFTP) is proposed, in which superframes are formed for representing a series of consecutive frames with high temporal consistency and their features are pooled from the features of frames through the integration function. Then, the temporal proposal is built based on the multiple consecutive superframes and the features of all proposals are generated from a pyramidal feature hierarchy. This hierarchy consists of the designed Structured Outer-Inner Context (SOIC) features formed from superframe features and is able to explicitly characterize the temporal continuity inside a proposal. Furthermore, a novel Scale-Wise Normalization Strategy (SWNS) is proposed to identify proposals, which can effectively detect multiple actions with different duration in one untrimmed video. Extensive experiments are conducted on two public datasets: THUMOS14 and ActivityNet1.2 for performance evaluation. Our experimental results have demonstrated that the proposed approach is able to detect the boundaries of actions more effectively and obtain competitive mAP (mean average precision) compared with other approaches.

OrtDet: An Orientation Robust Detector via Transformer for Object Detection in Aerial Images

The detection of arbitrarily rotated objects in aerial images is challenging due to the highly complex backgrounds and the multiple angles of objects. Existing detectors are not robust relative to the varying angle of objects because the CNNs do not explicitly model the orientation’s variation. In this paper, we propose an Orientation Robust Detector (OrtDet) to solve this problem, which aims to learn features that change accordingly with the object’s rotation (i.e., rotation-equivariant features). Specifically, we introduce a vision transformer as the backbone to capture its remote contextual associations via the degree of feature similarities. By capturing the features of each part of the object and their relative spatial distribution, OrtDet can learn features that have a complete response to any direction of the object. In addition, we use the tokens concatenation layer (TCL) strategy, which generates a pyramidal feature hierarchy for addressing vastly different scales of objects. To avoid the confusion of angle regression, we predict the relative gliding offsets of the vertices in each corresponding side of the horizontal bounding boxes (HBBs) to represent the oriented bounding boxes (OBBs). To intuitively reflect the robustness of the detector, a new metric, the mean rotation precision (mRP), is proposed to quantitatively measure the model’s learning ability for a rotation-equivariant feature. Experiments on the DOTA-v1.0, DOTA-v1.5, and HRSC2016 datasets show that our method improves the mAP by 0.5, 1.1, and 2.2 and reduces mRP detection fluctuations by 0.74, 0.56, and 0.52, respectively.

SAFFNet: Self-Attention-Based Feature Fusion Network for Remote Sensing Few-Shot Scene Classification

In real applications, it is necessary to classify new unseen classes that cannot be acquired in training datasets. To solve this problem, few-shot learning methods are usually adopted to recognize new categories with only a few (out-of-bag) labeled samples together with the known classes available in the (large-scale) training dataset. Unlike common scene classification images obtained by CCD (Charge-Coupled Device) cameras, remote sensing scene classification datasets tend to have plentiful texture features rather than shape features. Therefore, it is important to extract more valuable texture semantic features from a limited number of labeled input images. In this paper, a multi-scale feature fusion network for few-shot remote sensing scene classification is proposed by integrating a novel self-attention feature selection module, denoted as SAFFNet. Unlike a pyramidal feature hierarchy for object detection, the informative representations of the images with different receptive fields are automatically selected and re-weighted for feature fusion after refining network and global pooling operation for a few-shot remote sensing classification task. Here, the feature weighting value can be fine-tuned by the support set in the few-shot learning task. The proposed model is evaluated on three publicly available datasets for few shot remote sensing scene classification. Experimental results demonstrate the effectiveness of the proposed SAFFNet to improve the few-shot classification accuracy significantly compared to other few-shot methods and the typical multi-scale feature fusion network.

PSANet: Pyramid Splitting and Aggregation Network for 3D Object Detection in Point Cloud.

3D object detection in LiDAR point clouds has been extensively used in autonomous driving, intelligent robotics, and augmented reality. Although the one-stage 3D detector has satisfactory training and inference speed, there are still some performance problems due to insufficient utilization of bird’s eye view (BEV) information. In this paper, a new backbone network is proposed to complete the cross-layer fusion of multi-scale BEV feature maps, which makes full use of various information for detection. Specifically, our proposed backbone network can be divided into a coarse branch and a fine branch. In the coarse branch, we use the pyramidal feature hierarchy (PFH) to generate multi-scale BEV feature maps, which retain the advantages of different levels and serves as the input of the fine branch. In the fine branch, our proposed pyramid splitting and aggregation (PSA) module deeply integrates different levels of multi-scale feature maps, thereby improving the expressive ability of the final features. Extensive experiments on the challenging KITTI-3D benchmark show that our method has better performance in both 3D and BEV object detection compared with some previous state-of-the-art methods. Experimental results with average precision (AP) prove the effectiveness of our network.

Video saliency prediction using enhanced spatiotemporal alignment network

Due to a variety of motions across different frames, it is highly challenging to learn an effective spatiotemporal representation for accurate video saliency prediction (VSP). To address this issue, we develop an effective spatiotemporal feature alignment network tailored to VSP, mainly including two key sub-networks: a multi-scale deformable convolutional alignment network (MDAN) and a bidirectional convolutional Long Short-Term Memory (Bi-ConvLSTM) network. The MDAN learns to align the features of the neighboring frames to the reference one in a coarse-to-fine manner, which can well handle various motions. Specifically, the MDAN owns a pyramidal feature hierarchy structure that first leverages deformable convolution (Dconv) to align the lower-resolution features across frames, and then aggregates the aligned features to align the higher-resolution features, progressively enhancing the features from top to bottom. The output of MDAN is then fed into the Bi-ConvLSTM for further enhancement, which captures the useful long-time temporal information along forward and backward timing directions to effectively guide attention orientation shift prediction under complex scene transformation. Finally, the enhanced features are decoded to generate the predicted saliency map. The proposed model is trained end-to-end without any intricate post processing. Extensive evaluations on four VSP benchmark datasets demonstrate that the proposed method achieves favorable performance against state-of-the-art methods. The source codes and all the results will be released at https://github.com/cj4L/ESAN-VSP.

HyNet: Hyper-scale object detection network framework for multiple spatial resolution remote sensing imagery

Faced with the problem of the large scale variation, geospatial object detection in multiple spatial resolution (MSR) remote sensing imagery is a challenging task. To avoid the scale problem, the current convolutional neural network (CNN) based object detectors use multi-scale structures in the convolutional layer level to improve the detection performance by utilizing different receptive fields in the convolutional layers with different scales to capture objects with different scales. Examples of such methods are the image pyramid, pyramidal feature hierarchy, and the feature pyramid network. However, in MSR imagery, it is still difficult to model the large scale variation of geospatial objects for the existing multi-scale structures as their receptive fields are limited due to the fixed number of layers. In this paper, to solve the problem, a hyper-scale object detection framework for MSR imagery, namely HyNet, is proposed to alleviate the extreme scale-variation problem by learning hyper-scale feature representation. Differing from the previous multi-scale structure operation in the level of the convolutional layer, HyNet uses a hyper-scale block as the core structure, namely the HyBlock, in the sub-layer group level. In the HyBlock, each convolutional layer in the multi-scale structure is first divided into sub-layer groups with an equal size. In the sub-layer group level, hyper-scale features are obtained by a multi-scale sub-layer group operation with pyramidal receptive fields in the convolutional layers of each scale, which means that HyBlock is a fine-grained multi-scale structure. To effectively aggregate the hyper-scale features, group connection in the sub-layer level is used for intra-layer message passing. By promoting the intra-layer message passing to capture the scale-invariance of the hyper-scale features, the group connection can alleviate the scale-variation issue for object detection in MSR imagery. To better utilize the hyper-scale features, adaptive feature selection is proposed to select more effective hyper-scale features via adaptively weighting the different hyper-scale features. The experimental results obtained using three object detection datasets demonstrate that HyNet can learn a robust scale-invariant feature representation and can outperform the previous algorithms, and hence provides an effective new option for object detection in MSR remote sensing imagery.

CSE-HRNet: A Context and Semantic Enhanced High-Resolution Network for Semantic Segmentation of Aerial Imagery

Semantic segmentation of high-resolution aerial images is a concerning issue of remote sensing applications. To address the issues of intra-class heterogeneity and inter-class homogeneity, a novel end-to-end semantic segmentation network, namely Context and Semantic Enhanced High-Resolution Network (CSE-HRNet), is proposed in this paper. Two procedures are considered comprehensively, which are multi-scale contextual feature extractor and multi-level semantic feature producer. Nested Dilated Residual Block (NDRB) is designed firstly, which could enhance the representational power of multi-scale contexts and tackle the issue of intra-class heterogeneity. The pyramidal feature hierarchy is introduced secondly, by which multi-level feature fusions could be utilized to enlarge inter-class semantic differences. Experimental results verify that, based on the Potsdam and Vaihingen benchmarks, the proposed CSE-HRNet can achieve competitive performance compared with other state-of-the-art methods.

WALLACE: Weakly Supervised Learning of Deep Convolutional Neural Networks With Multiscale Evidence

This paper presents WALLACE, a new framework of deep convolutional neural networks, which perform ConvNet’s pyramidal feature hierarchy for weakly supervised learning. Most prior works rely on the image pyramid or network ensemble, which is both complicated and usually expensive. Instead, WALLACE is a more simple single-stage network that can predict objects present and location in an image without multiple rescale. Our model is trained efficiently using only global image-level labels, and it could generate meaningful multi-scale semantic feature maps by only one evaluation. Furthermore, a novel constrain-to-highlight loss is proposed to balances region selection among hierarchical feature maps, which additional improve model performance. Extensive experiments on object classification and weakly supervised pointwise object localization show that WALLACE achieves state-of-the-art results on the VOC 2007 and VOC 2012 benchmark without bells and whistles.

Temporally Identity-Aware SSD With Attentional LSTM

Temporal object detection has attracted significant attention, but most popular detection methods cannot leverage rich temporal information in videos. Very recently, many algorithms have been developed for video detection task, yet very few approaches can achieve real-time online object detection in videos. In this paper, based on the attention mechanism and convolutional long short-term memory (ConvLSTM), we propose a temporal single-shot detector (TSSD) for real-world detection. Distinct from the previous methods, we take aim at temporally integrating pyramidal feature hierarchy using ConvLSTM, and design a novel structure, including a low-level temporal unit as well as a high-level one for multiscale feature maps. Moreover, we develop a creative temporal analysis unit, namely, attentional ConvLSTM, in which a temporal attention mechanism is specially tailored for background suppression and scale suppression, while a ConvLSTM integrates attention-aware features across time. An association loss and a multistep training are designed for temporal coherence. Besides, an online tubelet analysis (OTA) is exploited for identification. Our framework is evaluated on ImageNet VID dataset and 2DMOT15 dataset. Extensive comparisons on the detection and tracking capability validate the superiority of the proposed approach. Consequently, the developed TSSD-OTA achieves a fast speed and an overall competitive performance in terms of detection and tracking. Finally, a real-world maneuver is conducted for underwater object grasping.

Scale Pyramid Attention for Single Shot MultiBox Detector

Feature pyramid mechanism has improved the performance of object detectors by a large margin, especially for the objects with small scale. As one of the first attempts to use pyramidal feature hierarchy, Single Shot MultiBox Detector (SSD) has largely accelerated the pipeline of detection with the competitive performance. Even if the feature pyramid mechanism is used for better detecting objects with small scale, the complex background information still misleads the network to focus on invalid areas. Image areas where small objects exist are more easily to be misjudged which influences the performance of lower-level layer in feature pyramid. In addition, a great deal of false positives may also be introduced into each pyramid layer because of the complex background. In this paper, we propose a novel method named scale pyramid attention hierarchy to better assist the SSD. The general feature pyramid detector can be guided to focus on the valid image areas as well as neglect the influence of complex background with the assist of this proposed module. Furthermore, the end-to-end training can highlight the foreground information which reduces the pressure of detectors, especially for detecting small objects. Flexible implementation makes it easily embedded in the feature pyramid mechanism. Experimental results on PASCAL VOC 2007, VOC 2012 and MS COCO confirm the effectiveness of the proposed module.