Multi-scale Feature Pyramid Research Articles

MLFFNet: Multilevel Feature Fusion Network for Object Detection in Sonar Images

Sonar image object detection is essential in underwater rescue and resource exploration. Although many convolution neural network (CNN)-based object detection algorithms have achieved great success in natural images. However, for underwater sonar images, problems, such as seabed reverberation noise interference, low proportion of foreground object region pixels, and poor imaging resolution, present considerable challenges to achieving accurate underwater object detection. To address these problems, we propose a novel sonar image object detector called the multilevel feature fusion network (MLFFNet). The detector consists of multiscale convolution module (MS-Conv), multilevel feature extraction module (ML-FEM), multilevel feature fusion module (ML-FFM), neighborhood channel attention mechanism (N-CAM), multiscale feature pyramid module (MS-FPN), and feature association module (FA). First, we use the MS-Conv to extract different scale feature information in the object region. Second, the ML-FEM and ML-FFM are used to obtain the local detail and global context features. Third, the N-CAM and MS-FPN are used to obtain the foreground objects’ semantic feature and position feature, and suppress the background region noise interference. Finally, we use the FA module to enhance the category and feature correlation of different objects. Extensive experiments are conducted on the real scene sonar image dataset. The experimental results demonstrate that MLFFNet performs better than other state-of-the-art object detection methods. Code and dataset are publicly at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/darkseid-arch/SonarMLFFNet</uri> .

A ViT-Based Multiscale Feature Fusion Approach for Remote Sensing Image Segmentation

Semantic segmentation plays an indispensable role in automatic analysis of remote sensing image data. However, the abundant semantic information and irregular shape patterns in remote sensing images are difficult to utilize, making it hard to segment remote sensing images only using convolution and single-scale feature maps. To achieve better segmentation performance, a multiscale feature pyramid decoder (MFPD) is proposed to fuse image features extracted by vision transformer (ViT). The decoder employs a novel 2-D-to-3-D transform method to obtain multiscale feature maps that contain rich context information and fuses the multiscale feature maps by channel concatenation. Furthermore, a dimension attention module (DAM) is designed to further aggregate the context information of the extracted remote sensing image features. This approach yields superior mean intersection over union (mIoU) on the Gaofen2-CZ dataset (60.42&#x0025;) and GID-5 dataset (68.21&#x0025;). Experimental results indicate that the comprehensive performance of our approach exceeds the compared segmentation methods based on convolutional neural network (CNN) and ViT.

CLT-Det: Correlation Learning Based on Transformer for Detecting Dense Objects in Remote Sensing Images

Challenges still exist in the task of object detection in remote sensing images with densely distributed objects due to large variation in scale and neglect of the relative position and correlation. To address these issues, a Correlation Learning Detector based on Transformer (CLT-Det) is proposed for detecting dense objects in remote sensing images. A Transformer Attention Module (TAM) is designed to improve the densely packed objects’ model representation ability by learning pixel-wise attention with Transformer. To alleviate the semantic gap caused by variations in scale, a Feature Refinement Module (FRM) is proposed by improving the multi-scale feature pyramid. A Correlation Transformer Module (CTM) is proposed to extract correlation information and encodes position information of dense objects’ features on the classification branch for fully utilizing the position information and correlation among objects. Extensive experiments compared with several state-of-art methods on two challenging remote sensing datasets, namely DOTA and HRSC2016, demonstrate that the proposed CLT-Det achieves promising and competitive performance.

Adaptive Pyramid Context Fusion for Point Cloud Perception

Deep learning for 3-D point cloud perception has been a very active research topic in recent years. A current trend is toward the combination of the semantically strong and the fine-grained information from different scales of intermediate representations to boost network generalization power and robustness against scale variation. One prominent challenge is how to effectively conduct the allocation of multiple scales of information. In this letter, we propose a module, named adaptive pyramid context fusion (APCF), to adaptively capture scales of contextual information from a multiscale feature pyramid for the point cloud. The APCF module reweights and aggregates the features from different levels in the feature pyramid via a softmax attention strategy. The allocation of information is adaptively conducted level by level from bottom to up first and then from top to bottom. To ensure both effectiveness and efficiency, we propose a multiscale context-aware network APCF-Net through applying our proposed APCF to the PointConv architecture. Experiments demonstrate that APCF-Net surpasses its vanilla counterpart by a large margin both in effectiveness and efficiency. Especially, APCF-Net outperforms state-of-the-art approaches on 3-D object classification and semantic segmentation task, with the overall accuracy of 93.3% on ModelNet40 and mIoU of 63.1% on ScanNet V2 online test.

MBANet: Multi-branch aware network for kidney ultrasound images segmentation

Due to the influence of kidney morphology, heterogeneous structure and image quality, segmenting kidney in ultrasound images is challenging. To alleviate this challenge, we proposed a novel deep neural network architecture, namely Multi-branch Aware Network (MBANet), to segment kidney accurately and robustly. MBANet mainly consists of multi-scale feature pyramid (MSFP), multi-branch encoders (MBE) and master decoder. The design of MSFP can make the network more accessible to different kinds of class details at different scales. The information exchange between MBE can reduce the loss of feature information and improve the segmentation accuracy of the network. In addition, we designed a multi-scale fusion block (MFBlock) in the MBE to further extract and fuse more refined multi-scale image information. In order to further improve the robustness of MBANet, this paper also designed a step-by-step training mechanism. We validated the proposed approach and compared to several state-of-the-art approaches on the same kidney ultrasound datasets using six quantitative metrics. The results of our method on the six indicators of pixel accuracy (PA), intersection over union (IoU), precision, recall, specificity and F1-score (F1) are 98.83%, 92.38%, 97.10%, 95.03%, 99.46% and 0.9601, respectively. Compared with the comparison method, the average values on the six indicators are improved by about 2%. The evaluation results and segmentation results demonstrate that the proposed approach achieves the best overall performance on kidney ultrasound images segmentation.

DDNet: 3D densely connected convolutional networks with feature pyramids for nasopharyngeal carcinoma segmentation

Radiation therapy is the standard treatment for early stage Nasopharyngeal cancer (NPC). Thus, accurate delineation of target volumes at risk in NPC is important. While manual delineation is time-consuming and labour-intensive process and also leads to significant inter- and intra-practitioner variability. Thus, computer-aided segmentation algorithm is required. However, segmentation task is not trivial due to large variations (e.g., shape and size) of nasopharynx structure across subjects. Moreover, extreme foreground and background class imbalance in NPC segmentation remains challenge. In this paper, we propose a threedimensional densely connected convolutional neural network with multi-scale feature pyramids for NPC segmentation. We adapt the densely connected convolutional block into a new structure via adding feature pyramids. The concatenated pyramid feature carries multi-scale and hierarchical semantic information which is effective for segmenting different size of tumors and perceiving hierarchical context information. To address the foreground and background imbalance problem, we propose an enhanced version of focal loss. It prevents the large number of negative voxels far from boundaries from overwhelming the segmentation algorithm. We validated the proposed method on 120 clinical subjects. Experimental results demonstrate that our approach out-performed state-of-the-art methods and human experts.

MFP‐Net: Multi‐scale feature pyramid network for crowd counting

Although deep learning has been widely used for dense crowd counting, it still faces two challenges. Firstly, the popular network models are sensitive to scale variance of human head, human occlusions, and complex background due to repeated utilization of vanilla convolution kernels. Secondly, the vanilla feature fusion often depends on summation or concatenation, which ignores the correlation of different features leading to information redundancy and low robustness to background noise. To address these issues, a multi-scale feature pyramid network (MFP-Net) for dense crowd counting is proposed in this paper. The proposed MFP-Net makes two contributions. Firstly, the feature pyramid fusion module is designed that adopts rich convolutions with different depths and scales, not only to expand the receptive field, but also to improve the inference speed of models by using parallel group convolution. Secondly, a feature attention-aware module is added in the feature fusion stage. The module can achieve local and global information fusion by capturing the importance of the spatial and channel domains to improve model robustness. The proposed MFP-Net is evaluated on five publicly available datasets, and experiments show that the MFP-Net not only provides better crowd counting results than comparative models, but also requires fewer parameters.

FAGNet: Multi-Scale Object Detection Method in Remote Sensing Images by Combining MAFPN and GVR

<p indent=0mm>Remote sensing images of large scenes are complex, and have the characteristics of many categories of objects, different scales and changeable directions, which lead to the problem of multi-class, multi-scale and multi-oriented of objects in remote sensing images. A remote sensing image object detection method based on multi-scale attention feature pyramid network (MAFPN) and gliding vertex regression (GVR) mechanism is proposed. Firstly, multi-layer feature maps are extracted from backbone network as input of MAFPN, which combines feature fusion and attention mechanism. On the basis of fusing feature maps of multi-scale, channel attention and spatial attention mechanisms are used to suppress noise, enhance effective feature reuse, and improve the network’s adaptability to object multi-scale features. The fusion feature map output by MAFPN is input to the region proposal network to generate the regions of interest, and then they will be sent to the classification regression network. The GVR mechanism is used in the object classification regression network and the four vertex offset ratio parameters and rotation factors are added on the basis of predicting the horizontal boxes, which converts the horizontal boxes into a rotating box to reduce the redundant area in the bounding boxes, makes the predicted rotating bounding boxes fit the object more closely. The experimental results on the DOTA public dataset, compared with many classical detection algorithms based on convolutional neural networks, show that the average detection accuracy of the proposed method is significantly improved, which can detect objects of multi-scales and multi-oriented more accurately, and achieve the robust detection of multi-scale objects.

Rotational multipyramid network with bounding‐box transformation for object detection

The study proposes a rotational multipyramid network (RoMP Net) with bounding-box transformation for object detection. The RoMP Net is a single-stage object detection neural network featuring three characteristics. First, the network uses a rotational bounding box to minimize the effect of background images when extracting features of objects. Bounding-box transformation was proposed to compensate for the limitation of the rotational bounding boxes, which have relatively low prediction accuracy for objects with a high aspect ratio. Second, the RoMP Net introduces a multi-scale and multi-level feature pyramid network to extract distinct and semantic features efficiently. This network architecture ensures high prediction accuracy and robustness regardless of the size and complexity of objects. Third, hyperparameters in the bounding boxes are automatically determined through an unsupervised clustering method. This optimization method is also critical in improving accuracy. The performance of the proposed network and preprocessing methods are validated through image-sets comprising critical components in power transmission facilities, which have a variety of sizes and aspect ratios. This case study demonstrates the effectiveness and robustness of the three key characteristics in the RoMP Net. Furthermore, the RoMP Net outperforms other state-of-the-art deep neural networks in prediction accuracy and robustness for object detection. Specifically, the mean average precision of the RoMP Net in the validation image-sets shows that it has the highest prediction accuracy, whereas its values in the test image-sets confirm the network's robustness. The fast yet accurate RoMP Net will expand the range of object detection through deep neural networks.

Multi-Scale Feature Pyramid Network: A Heavily Occluded Pedestrian Detection Network Based on ResNet.

The existing pedestrian detection algorithms cannot effectively extract features of heavily occluded targets which results in lower detection accuracy. To solve the heavy occlusion in crowds, we propose a multi-scale feature pyramid network based on ResNet (MFPN) to enhance the features of occluded targets and improve the detection accuracy. MFPN includes two modules, namely double feature pyramid network (FPN) integrated with ResNet (DFR) and repulsion loss of minimum (RLM). We propose the double FPN which improves the architecture to further enhance the semantic information and contours of occluded pedestrians, and provide a new way for feature extraction of occluded targets. The features extracted by our network can be more separated and clearer, especially those heavily occluded pedestrians. Repulsion loss is introduced to improve the loss function which can keep predicted boxes away from the ground truths of the unrelated targets. Experiments carried out on the public CrowdHuman dataset, we obtain 90.96% AP which yields the best performance, 5.16% AP gains compared to the FPN-ResNet50 baseline. Compared with the state-of-the-art works, the performance of the pedestrian detection system has been boosted with our method.

Local Enhancement and Bidirectional Feature Refinement Network for Single-Shot Detector

Benefit from multi-scale feature pyramid methods, recently single-stage object detectors have achieved promising accuracy and fast inference speed. However, the majority of existing feature pyramid detection techniques only simply describe complex contextual relationships from different scales. Not only are there no effective modules that adaptively extend appropriate semantic information from deeper layers, but the finer spatial localization cues from lower layers are often ignored. In this paper, we present a Local Enhancement and Bidirectional Feature Refinement Network (LFBFR), which includes two optimization methods to achieve remarkable improvements in detection accuracy. Firstly, to make the backbone more suitable for detection task, we modify the pre-trained classification backbone to mitigate the loss of details in small objects due to consecutive decrease of the image resolution. Then we propose a Bidirectional Feature Refinement Pyramid, which can effectively utilize the inter-channel relationship of higher-level features and fine appearance cues from lower-level features by using the attention residual refinement module and the feature reuse module. Ultimately, to assess the performance of the proposed LFBFR, we design a powerful end-to-end single-stage detector called LFBFR-SSD by embedding it into the framework of SSD. Extensive experiments on the PASCAL VOC and MS COCO verify that our LFBFR-SSD outperforms a lot of state-of-the-art detectors while maintaining a real-time speed.

Multi-Scale Attention Network for Diabetic Retinopathy Classification

Diabetic Retinopathy (DR) is a highly prevalent complication of diabetes mellitus, which causes lesions on the retina that affect vision which may lead to blindness if it is not detected and diagnosed early. Convolutional neural networks (CNN) are becoming the state-of-the-art approach for automatic detection of DR by using fundus images. The high-level features extracted by CNN are mostly utilised for the detection and classification of lesions on the retina. This high-level representation is capable of classifying different DR classes; however, more effective features for detecting the damages are needed. This paper proposes the multi-scale attention network (MSA-Net) for DR classification. The proposed approach applies the encoder network to embed the retina image in a high-level representational space, where the combination of mid and high-level features is used to enrich the representation. Then a multi-scale feature pyramid is included to describe the retinal structure in a different locality. Furthermore, to enhance the discriminative power of the feature representation a multi-scale attention mechanism is used on top of the high-level representation. The model is trained in a standard way using the cross-entropy loss to classify the DR severity level. In parallel as an auxiliary task, the model is trained using the weakly annotated data to detect healthy and non-healthy retina images. This surrogate task helps the model to enrich its discriminative power for distinguishing the non-healthy retina images. The proposed method when implemented has achieved outstanding results on two public datasets: EyePACS and APTOS.

An Improved Faster R-CNN for Pulmonary Embolism Detection From CTPA Images

Computer-aided detection of pulmonary embolism is an important technology method for diagnosing pulmonary embolism, which can help doctors diagnose quickly and save a lot of manpower. However, due to the small area of pulmonary embolism in the Computed Tomography Pulmonary Angiography (CTPA) slice images, some previous methods for detecting pulmonary embolism have a high number of false detection and missed detection. This study proposes a detection method of pulmonary embolism based on the improved faster region-based convolutional neural network (Faster R-CNN) named More Accurate Faster R-CNN (MA Faster R-CNN). A new feature fusion network named Multi-scale Fusion Feature Pyramid Network (MF-FPN) is proposed by extending and adding two bottom-up paths on the Feature Pyramid Network (FPN). It enhances the feature extraction capability of the entire network by transmitting low-level accurate location information, and makes up for the original information lost after multiple down-sampling, strengthens the use of detailed information, which is more helpful to the detection of small object. In the prediction module, the residual block is added before the fully-connected layer to deepen the network and enhance the classification accuracy, named residual prediction module (RPM). Compared with the original Faster R-CNN, the proposed MA Faster R-CNN which combines MF-FPN and RPM has a higher detection precision and solves the problems of false detection and missed detection of pulmonary embolism effectively. The average precision (AP) reached 85.88% on the CTPA pulmonary embolism dataset used in this article.

PA-MVSNet: Sparse-to-Dense Multi-View Stereo With Pyramid Attention

Multi-view based 3D reconstruction aims to obtain 3D structure information of objects in space through two-dimensional images. In this paper, we propose a new multi-view stereo network that can robustly reconstruct the scene. To enhance the feature representation ability of Point-MVSNet, a pyramid attention module is introduced. Specifically, we exploit the attention mechanism for the multi-scale feature pyramid to capture larger receptive fields and richer information. Instead of constructing a feature pyramid as the input, results of the pyramid attention module at different scales are directly used for the next layer. The network eventually generates a high-quality depth estimation for 3D reconstruction from sparse to dense by an iterative refinement schedule. Experiments have been performed to evaluate 3D reconstruction quality by comparison with existing state-of-the-art methods on the DTU dataset. The experimental results indicate our method performs the best in overall quality compared with previous methods, proving the effectiveness of our method. In the end, we use the data collected by mobile devices to implement 3D reconstruction with a combination of traditional and learning-based methods, providing ideas for the 3D reconstruction technology on mobile devices.

Scattering Enhanced Attention Pyramid Network for Aircraft Detection in SAR Images

Aircraft detection in synthetic aperture radar (SAR) images is a challenging task because of the discreteness, variability, and interference of aircraft scattering features. This article proposes a new hybrid approach of scattering information enhancement (SIE) and an attention pyramid network (APN). It first extracts strong scattering points (SSPs) of aircraft via an adapted Harris-Laplace detector. These SSPs are then clustered into candidate scattering regions by density-based spatial clustering of applications with noise (DBSCAN) and are then modeled with a Gaussian mixture model (GMM). Target scattering clusters are discriminated from background clutter by measuring the Kullback-Leibler divergence (KLD) to the known target templates. These target scattering clusters are enhanced in the preprocessing stage. All the SIE-preprocessed images are then fed into the APN for training and testing. It is composed of the multiscale feature pyramid network (FPN) and the modified convolutional block attention module (CBAM) to cope with the discreteness and variability of aircraft. In addition, focal loss (FL) is adopted to deal with the issue of unbalanced sample distribution and the interference from hard samples. Experiments conducted on the Gaofen-3 and TerraSAR-X data sets demonstrate the effectiveness of the proposed method with an average precision (AP) of 83.25%.

Scale-aware feature pyramid architecture for marine object detection

Marine object detection is an appealing but challengeable task in computer vision. Even though recent popular object detection algorithms perform well on common classes, they cannot acquire satisfied detection performance on marine objects because underwater images are affected by color cast and blur, and scales of the target in underwater images are usually small. These phenomena aggravate the difficulty of detection. Thus, it is urgent to design a proper structure to settle marine object detection issues. To this end, this paper proposes a novel scale-aware feature pyramid architecture named SA-FPN to extract abundant robust features on underwater images and improve the performance on marine object detection. Specifically, we design a special backbone subnetwork to improve the ability of feature extraction, which could provide richer fine-grained features for small object detection. What is more, this paper proposes a multi-scale feature pyramid to enrich the semantic features for prediction. Each feature map is enhanced by the higher level layer with context information through a top-down upsampling pathway. Through obtaining ample feature maps on underwater images, our algorithm could generate multiple bounding boxes for each target. To mitigate the reduplicative boxes and avoid miss suppression, we replace the non-maximum suppression method with soft non-maximum suppression. In this paper, we evaluate our algorithm on underwater image datasets and achieve 76.27% mAP. Meanwhile, we conduct experiments on PASCAL VOC datasets and smart unmanned vending machines datasets and get 79.13% mAP and 91.81% mAP, respectively. The experimental results reveal that our approach achieves best performance not only on marine object detection, but also on common classes.

Encoder- and Decoder-Based Networks Using Multiscale Feature Fusion and Nonlocal Block for Remote Sensing Image Semantic Segmentation

With the development of convolutional neural networks, the semantic segmentation of remote sensing images has been widely developed, but there are still some unsolved problems in this field due to the lack of multiscale information and the feature mismatch at the upsampling process. To solve these problems, we propose a network called multiscale feature fusion and alignment network (MFANet). MFANet is composed of an encoder and a decoder. The encoder contains a fully convolutional network, a multilevel feature fusion block (MLFFB), and a multiscale feature pyramid (MSFP). These subnetworks can obtain fine-grained feature maps that are full of multiscale and global features and improve segmentation results at multiple object scales. Moreover, MFANet uses a light convolution subnetwork, called decoder, to upsample the segmentation map stage by stage. Combining three scales of features, the decoder can promote the feature alignment at the upsampling stage. Along with the decoder, MFANet utilizes a multistage supervision loss to enhance the localization performance and boundary regression ability. Benefitting from the encoder and decoder structure and the innovative components inside encoder, MFANet is very powerful for the semantic segmentation of remote sensing images and can suit the complicated environment. We evaluate our MFANet on the Vaihingen and Potsdam data sets, and it outperforms the state-of-art methods both in the metric and visual effect.

Visual Object Tracking Based on Mutual Learning Between Cohort Multiscale Feature-Fusion Networks With Weighted Loss

The deep convolutional neural network (CNN) based tracking-by-detection framework recently has become one of the most popular trackers. However, these methods are either time consuming or have greatly reduced performance. This article aims to achieve nearly identical tracking accuracy with the state-of-the-art CNN tracking-by-detection algorithm with relatively faster speed. We study the existing excellent trackers under the CNN tracking-by-detection framework and introduce the following: a multiscale feature pyramid fusion neural network based on dilated convolutions is constructed to learn a scale-invariant discriminative representation for tracking small objects, a hard-threshold weighted cross-entropy loss function is proposed to decrease the gap between object classification and tracking, and a mutual learning-based training policy is used to fuse the information from the network trained by image patches with different contextual regions to further improve the tracking performance. We conduct comprehensive experiments on visual object tracking benchmarks that validate the achievement of competitive performance of the proposed tracker with relatively faster speed both on qualitative and quantitative criteria. Additionally, the experimental results reveal that the proposed mutual learning-based training policy can accelerate the convergence speed and achieve better generalization performance.

Automated mammographic mass detection using deformable convolution and multiscale features.

Designing computer-assisted diagnosis (CAD) systems that can precisely identify lesions from mammography images would be useful for clinicians. Considering the morphological variation in breast cancer, it is necessary to extract robust features from the mammogram. Here, we propose a mass detection CAD system that is based on Faster R-CNN. First, we applied a novel convolution network in the backbone of Faster R-CNN, namely deformable convolution network (DCN), which improves the detection of lesions with varying shapes and sizes. Second, the original Faster R-CNN uses the output of the last layer of the backbone as a single-scale feature map. To facilitate the detection of small lesions, we used a multiscale feature pyramid network of multiple cross-scale connections between the different output layers of the backbone, called the neural architecture search-feature pyramid network (NAS-FPN). Thus, we were able to integrate the best features into the model. We then evaluated our method by using the datasets the Curated Breast Imaging Subset of Digital Database for Screening Mammography (CBIS-DDSM) and INbreast, respectively. Our method yielded a true positive rate of 0.9345 at 2.2805 false positive per image on CBIS-DDSM and a true positive rate of 0.9554 at 0.3829 false positive per image on INbreast. Graphical abstract.

Rethinking the Bottom-Up Framework for Query-Based Video Localization

In this paper, we focus on the task query-based video localization, i.e., localizing a query in a long and untrimmed video. The prevailing solutions for this problem can be grouped into two categories: i) Top-down approach: It pre-cuts the video into a set of moment candidates, then it does classification and regression for each candidate; ii) Bottom-up approach: It injects the whole query content into each video frame, then it predicts the probabilities of each frame as a ground truth segment boundary (i.e., start or end). Both two frameworks have respective shortcomings: the top-down models suffer from heavy computations and they are sensitive to the heuristic rules, while the performance of bottom-up models is behind the performance of top-down counterpart thus far. However, we argue that the performance of bottom-up framework is severely underestimated by current unreasonable designs, including both the backbone and head network. To this end, we design a novel bottom-up model: Graph-FPN with Dense Predictions (GDP). For the backbone, GDP firstly generates a frame feature pyramid to capture multi-level semantics, then it utilizes graph convolution to encode the plentiful scene relationships, which incidentally mitigates the semantic gaps in the multi-scale feature pyramid. For the head network, GDP regards all frames falling in the ground truth segment as the foreground, and each foreground frame regresses the unique distances from its location to bi-directional boundaries. Extensive experiments on two challenging query-based video localization tasks (natural language video localization and video relocalization), involving four challenging benchmarks (TACoS, Charades-STA, ActivityNet Captions, and Activity-VRL), have shown that GDP surpasses the state-of-the-art top-down models.