Multi-scale Feature Pyramid Research Articles

Accurate detection of arbitrary ship directions using SAR based on RTMDet

ABSTRACT Ship detection, an important branch of Synthetic Aperture Radar (SAR) images interpretation, is crucial for various maritime reconnaissance and surveillance but remains challenging due to background interference, dense ship alignments and high aspect ratios. To address these challenges, this letter proposes an arbitrary direction detection method based on Real-Time Models for object Detection (RTMDet), which is one of the fastest and most accurate single-stage detection detectors. Firstly, an attention selection module is designed to replace traditional full convolution method to achieve more parameter reduction; Secondly, in order to extract more key ship feature information and accelerate the network extraction, we construct a lightweight multi-scale feature pyramid network; Thirdly, a novel loss function is introduced to enhance the detection of rotated bounding box by addressing the problem of bounding box discontinuities and lack of scale invariance. The proposed method achieves state-of-the-art detection accuracy on two the high-resolution SAR image datasets for arbitrary ship detection including HRSID (85.6% AP50) and RSDD (90.8% AP50), while demonstrating significant accuracy improvements in the complex inshore scenes.

Speech Emotion Recognition Using Multi-Scale Global–Local Representation Learning with Feature Pyramid Network

Speech emotion recognition (SER) is important in facilitating natural human–computer interactions. In speech sequence modeling, a vital challenge is to learn context-aware sentence expression and temporal dynamics of paralinguistic features to achieve unambiguous emotional semantic understanding. In previous studies, the SER method based on the single-scale cascade feature extraction module could not effectively preserve the temporal structure of speech signals in the deep layer, downgrading the sequence modeling performance. To address these challenges, this paper proposes a novel multi-scale feature pyramid network. The enhanced multi-scale convolutional neural networks (MSCNNs) significantly improve the ability to extract multi-granular emotional features. Experimental results on the IEMOCAP corpus demonstrate the effectiveness of the proposed approach, achieving a weighted accuracy (WA) of 71.79% and an unweighted accuracy (UA) of 73.39%. Furthermore, on the RAVDESS dataset, the model achieves an unweighted accuracy (UA) of 86.5%. These results validate the system’s performance and highlight its competitive advantage.

A Lightweight Deep Learning Network with an Optimized Attention Module for Aluminum Surface Defect Detection.

Aluminum is extensively utilized in the aerospace, aviation, automotive, and other industries. The presence of surface defects on aluminum has a significant impact on product quality. However, traditional detection methods fail to meet the efficiency and accuracy requirements of industrial practices. In this study, we propose an innovative aluminum surface defect detection method based on an optimized two-stage Faster R-CNN network. A 2D camera serves as the image sensor, capturing high-resolution images in real time. Optimized lighting and focus ensure that defect features are clearly visible. After preprocessing, the images are fed into a deep learning network incorporated with a multi-scale feature pyramid structure, which effectively enhances defect recognition accuracy by integrating high-level semantic information with location details. Additionally, we introduced an optimized Convolutional Block Attention Module (CBAM) to further enhance network efficiency. Furthermore, we employed the genetic K-means algorithm to optimize prior region selection, and a lightweight Ghost model to reduce network complexity by 14.3%, demonstrating the superior performance of the Ghost model in terms of loss function optimization during training and validation as well as in terms of detection accuracy, speed, and stability. The network was trained on a dataset of 3200 images captured by the image sensor, split in an 8:1:1 ratio for training, validation, and testing, respectively. The experimental results show a mean Average Precision (mAP) of 94.25%, with individual Average Precision (AP) values exceeding 80%, meeting industrial standards for defect detection.

Distinguishing Difficulty Imbalances in Strawberry Ripeness Instances in a Complex Farmland Environment

The existing strawberry ripeness detection algorithm has the problems of a low precision and a high missing rate in real complex scenes. Therefore, we propose a novel model based on a hybrid attention mechanism. Firstly, a partial convolution-based compact inverted block is developed, which significantly enhances the feature extraction capability of the model. Secondly, an efficient partial hybrid attention mechanism is established, which realizes the remote dependence and accurate localization of strawberry fruit. Meanwhile, a multi-scale progressive feature pyramid network is constructed, and the fine-grained features of strawberry targets of different sizes are accurately extracted. Finally, a Focaler-shape-IoU loss function is proposed to effectively solve the problem of the difficulty imbalance between strawberry samples and the influence of the shape and size of the bounding box on the regression. The experimental results show that the model’s precision and mAP0.5 reach 92.1% and 92.7%, respectively, which are 2.0% and 1.7% higher than the baseline model. Additionally, our model is better in detection performance than most models with fewer parameters and lower FLOPs. In summary, the model can accurately identify the maturity of strawberry fruit under complex farmland environments and provide certain technical guidance for automated strawberry-picking robots.

M-YOLOv8s: An improved small target detection algorithm for UAV aerial photography

The object of UAV target detection usually means small target with complicated backgrounds. In this paper, an object detection model M-YOLOv8s based on UAV aerial photography scene is proposed. Firstly, to solve the problem that the YOLOv8s model cannot adapt to small target detection, a small target detection head (STDH) module is introduced to fuse the location and appearance feature information of the shallow layers of the backbone network. Secondly, Inner-Wise intersection over union (Inner-WIoU) is designed as the boundary box regression loss, and auxiliary boundary calculation is used to accelerate the regression speed of the model. Thirdly, the structure of multi-scale feature pyramid network (MS-FPN) can effectively combine the shallow network information with the deep network information and improve the performance of the detection model. Furthermore, a multi-scale cross-spatial attention (MCSA) module is proposed to expand the feature space through multi-scale branch, and then achieves the aggregation of target features through cross-spatial interaction, which improves the ability of the model to extract target features. Finally, the experimental results show that our model does not only possess fewer parameters, but also the values of mAP0.5 are 6.6% and 5.4% higher than the baseline model on the Visdrone2019 validation dataset and test dataset, respectively. Then, as a conclusion, the M-YOLOv8s model achieves better detection performance than some existing ones, indicating that our proposed method can be more suitable for detecting the small targets.

EDASNet: efficient dynamic adaptive-scale network for infrared pedestrian detection

Abstract Infrared images are widely utilized due to their exceptional anti-interference capabilities. However, challenges such as low resolution and an absence of detailed texture can impede the effective recognition of multi-scale target information, particularly for small targets. To address these issues, we introduce a multi-scale detection framework named efficient dynamic adaptive-scale network (EDASNet), which focuses on enhancing the feature extraction of small objects while ensuring efficient detection of multi-scale. Firstly, we design a lightweight dynamic enhance network as the backbone for feature extraction. It mainly includes a lightweight adaptive-weight downsampling module and a dynamic enhancement convolution module. In addition, a multi-scale aggregation feature pyramid network is proposed, which improves the perception effect of small objects through a multi-scale convolution module. Then, the Repulsion Loss term was introduced based on CIOU to effectively solve the missed detection problem caused by target overlap. Finally, the dynamic head was used as the network detection head, and through the superposition of dynamic convolution and multiple attention, the network was able to accurately realize multi-scale object detection. Comprehensive experiments show that EDASNet outperforms existing efficient models and achieves a good trade-off between speed and accuracy.

AMFP-net: Adaptive multi-scale feature pyramid network for diagnosis of pneumoconiosis from chest X-ray images

Early detection of pneumoconiosis by routine health screening of workers in the mining industry is critical for preventing the progression of this incurable disease. Automated pneumoconiosis classification in chest X-ray images is challenging due to the low contrast of opacities, inter-class similarity, intra-class variation and the existence of artifacts. Compared to traditional methods, convolutional neural networks have shown significant improvement in pneumoconiosis classification tasks, however, accurate classification remains challenging due to mainly the inability to focus on semantically meaningful lesion opacities. Most existing networks focus on high level abstract information and ignore low level detailed object information. Different from natural images where an object occupies large space, the classification of pneumoconiosis depends on the density of small opacities inside the lung. To address this issue, we propose a novel two-stage adaptive multi-scale feature pyramid network called AMFP-Net for the diagnosis of pneumoconiosis from chest X-rays. The proposed model consists of 1) an adaptive multi-scale context block to extract rich contextual and discriminative information and 2) a weighted feature fusion module to effectively combine low level detailed and high level global semantic information. This two-stage network first segments the lungs to focus more on relevant regions by excluding irrelevant parts of the image, and then utilises the segmented lungs to classify pneumoconiosis into different categories. Extensive experiments on public and private datasets demonstrate that the proposed approach can outperform state-of-the-art methods for both segmentation and classification.

YOLO-RSA: A Multiscale Ship Detection Algorithm Based on Optical Remote Sensing Image

Currently, deep learning is extensively utilized for ship target detection; however, achieving accurate and real-time detection of multi-scale targets remains a significant challenge. Considering the diverse scenes, varied scales, and complex backgrounds of ships in optical remote sensing images, we introduce a network model named YOLO-RSA. The model consists of a backbone feature extraction network, a multi-scale feature pyramid, and a rotated detection head. We conduct thorough tests on the HRSC2016 and DOTA datasets to validate the proposed algorithm. Through ablation experiments, we assess the impact of each improvement component on the model. In comparative experiments, the proposed model surpasses other models in terms of Recall, Precision, and MAP on the HRSC2016 dataset. Finally, in generalization experiments, our proposed ship detection model exhibits excellent detection performance across various scenarios. The method can accurately detect multi-scale ships in the image and provide a basis for marine ship monitoring and port management.

RTHEN: Unsupervised deep homography estimation based on dynamic attention for repetitive texture image stitching

Homography estimation is regarded as one of the key challenges in image alignment, where the goal is to estimate the projective transformation between two images on the same plane. Unsupervised learning methods are gradually becoming popular due to their excellent performance and lack of need for labeled data. However, in regional scenes with repeated textures, there may be ambiguity in the correspondence between local features, affecting homography estimation accuracy. This paper proposes a new unsupervised deep homography method RTHEN to solve such problems. In order to effectively obtain repeated texture features, a multi-scale Feature pyramid Siamese network (FPSN) is designed. Specifically, we dynamically allocate the weights of recited texture features through a dynamic attention module and introduce a channel attention module to provide rich contextual information for repeated texture areas. We propose a hard triplet loss function based on overlap constraints to optimize the matching results. At the same time, we collected a repetitive texture image dataset (RTID) for homography estimation training and evaluation. Experimental results show that our method outperforms existing learning methods in repetitive texture scenes and offers competitive performance with state-of-the-art traditional methods.

Multi-Attention Pyramid Context Network for Infrared Small Ship Detection

In the realm of maritime target detection, infrared imaging technology has become the predominant modality. Detecting infrared small ships on the sea surface is crucial for national defense and maritime security. However, the challenge of detecting infrared small targets persists, especially in the complex scenes of the sea surface. As a response to this challenge, we propose MAPC-Net, an enhanced algorithm based on an existing network. Unlike conventional approaches, our method focuses on addressing the intricacies of sea surface scenes and the sparse pixel occupancy of small ships. MAPC-Net incorporates a scale attention mechanism into the original network’s multi-scale feature pyramid, enabling the learning of more effective scale feature maps. Additionally, a channel attention mechanism is introduced during the upsampling process to capture relationships between different channels, resulting in superior feature representations. Notably, our proposed Maritime-SIRST dataset, meticulously annotated for infrared small ship detection, is introduced to stimulate advancements in this research domain. Experimental evaluations on the Maritime-SIRST dataset demonstrate the superiority of our algorithm over existing methods. Compared to the original network, our approach achieves a 6.14% increase in mIOU and a 4.41% increase in F1, while maintaining nearly unchanged runtime.

PmcaNet: Pyramid multiscale channel attention network for electron microscopy image segmentation

Recent advances in high-throughput electron microscopy (EM) have revolutionized the examination of microstructures by enabling fast EM image generation. However, accurately segmenting EM images remains challenging due to inherent characteristics, including low contrast and subtle grayscale variations. Moreover, as manually annotated EM images are limited, it is usually impractical to utilize deep learning techniques for EM image segmentation. To address these challenges, the pyramid multiscale channel attention network (PmcaNet) is specifically designed. PmcaNet employs a convolutional neural network-based architecture and a multiscale feature pyramid to effectively capture global context information, enhancing its ability to comprehend the intricate structures within EM images. To enable the rapid extraction of channel-wise dependencies, a novel attention module is introduced to enhance the representation of intricate nonlinear features within the images. The performance of PmcaNet is evaluated on two general EM image segmentation datasets as well as a homemade dataset of superalloy materials, regarding pixel-wise accuracy and mean intersection over union (mIoU) as evaluation metrics. Extensive experiments demonstrate that PmcaNet outperforms other models on the ISBI 2012 dataset, achieving 87.85% pixel-wise accuracy and 73.11% mean intersection over union (mIoU), while also advancing results on the Kathuri and SEM-material datasets.

A Lightweight Arbitrarily Oriented Detector Based on Transformers and Deformable Features for Ship Detection in SAR Images

Lightweight ship detection is an important application of synthetic aperture radar (SAR). The prevailing trend in recent research involves employing a detection framework based on convolutional neural networks (CNNs) and horizontal bounding boxes (HBBs). However, CNNs with local receptive fields fall short in acquiring adequate contextual information and exhibit sensitivity to noise. Moreover, HBBs introduce significant interference from both the background and adjacent ships. To overcome these limitations, this paper proposes a lightweight transformer-based method for detecting arbitrarily oriented ships in SAR images, called LD-Det, which excels at promptly and accurately identifying rotating ship targets. First, light pyramid vision transformer (LightPVT) is introduced as a lightweight backbone network. Built upon PVT v2-B0-Li, it effectively captures the long-range dependencies of ships in SAR images. Subsequently, multi-scale deformable feature pyramid network (MDFPN) is constructed as a neck network, utilizing the multi-scale deformable convolution (MDC) module to adjust receptive field regions and extract ship features from SAR images more effectively. Lastly, shared deformable head (SDHead) is proposed as a head network, enhancing ship feature extraction with the combination of deformable convolution operations and a shared parameter structure design. Experimental evaluations on two publicly available datasets validate the efficacy of the proposed method. Notably, the proposed method achieves state-of-the-art detection performance when compared with other lightweight methods in detecting rotated targets.

Infrared Small Target Detection via Nested U-Structure With Attention and Multiscale Feature Pyramid

Infrared Small Target Detection via Nested U-Structure With Attention and Multiscale Feature Pyramid

Multi-Scale Feature Pyramid for Detection of Red Lesions in Fundus Images

Diabetic retinopathy (DR) is increasing rapidly around the world, but there is a shortage of experienced ophthalmologists. Therefore, computer-based diagnosis of the fundus images is essential to screening of referable DR. Automated detection of red lesions is very important for screening of DR. This paper deals with a novel method for automatic detection of red lesion. The main contribution is developing a deep learning based detection framework to handle severe class imbalance and imbalance in sizes of red lesions. The multi-scale features are extracted using the feature pyramid network. A pyramid of features is generated with strong semantics. The proposed network is end-to-end trainable in image level with several scales and works for a wide range of red lesions with acceptable performance. Sensitivity of the proposed method is 0.76 with six false-positive per image on test set of publicly available DIARECTDB1 database and outperforms state-of-the-art approaches. A potential benefit with deep learning based detection framework could be used in screening programs of referable DR.

Cross-scale collaborative network for single image super resolution

Convolution-based networks show impressive performance in Single Image Super-Resolution (SISR) task by making full use of local patterns. However, it requires deeper structure to capture the long-range dependencies which is inefficient. Hence, the Non-Local Attention (NLA) is introduced to search similar patterns globally and effectively exchange information globally. Nevertheless, the naïve NLA only exchanges information in original spatial scale and has quadratic computational complexity with spatial size of input image. Besides, the global fusion of all similar patterns may lead to blurring, which deviates from the goal of SISR to accurately recover image details. Hence, the naïve NLA is not suit for SISR. To solve this problem, we propose a novel Cross-Scale Collaborative Attention (CSCA) which is designed for SISR. Due to similar patterns tend to occur at different locations and scales, we explore self-similarity and cross-scale similarity simultaneously on multi-scale feature pyramid to utilize more rich details for improving the performance of recovery. To guarantee the recovered details are accurate, we aggregate features in each layer of pyramid into groups and only exchange information between the most relevant groups across multi-scale. Hence, CSCA is both efficient and can accurately recover details using the rich information contained in multi-scale. Furthermore, we propose a Cross-Scale Collaborative Network (CSCN) by inserting a few CSCA modules into a simple backbone network. Extensive experiments show the effectivity and efficiency of CSCA, and state-of-the-art performance of CSCN in SISR task.

Augmented weighted bidirectional feature pyramid network for marine object detection

Detecting marine object is an attractive but challenging task in ocean exploration and conservation. Although the current popular object detection algorithms perform well in general. Because underwater images are affected by color projections and the scale of the object is usually smaller, the detection performance for marine object is not ideal. Therefore, this paper proposes an augmented weighted bidirectional feature pyramid network (AWBiFPN) that reduces the weakening of underwater image features and improves the integration efficiency of multi-scale feature, improving marine object detection performance. Specifically, this paper designs a multi-scale feature pyramid for efficient integration of feature information at all levels by combining weighted bidirectional integration pathway with designed Consistent Supervision module. Using Residual Feature Augmentation module to enhance the extraction of unchanging proportional contextual information, reduce the loss of information at the highest level of the feature map in the pyramid network and provide richer particulate characteristics for small-target detection. To scale the feature map and effectively utilize the feature information of different channels in the same spatial location, this article proposes the AWBiFPN layer, which replaces the depthwise separable convolution in the original BiFPN layer with conventional convolution. Evaluated on underwater image dataset (UTDAC), only 12 epochs are trained to 81.63% of mAP; experimented on MS COCO benchmark, achieving 37.9 AP. Meanwhile, the accuracy of the Aquarium Life dataset (AMLD) and underwater detection dataset under natural light (RUOD) are 81.41% mAP and 83.62% mAP, respectively. The results show that the method performed better not only in marine object detection but also in general class detection.

Multi-Stage Multi-Scale Local Feature Fusion for Infrared Small Target Detection

The detection of small infrared targets with dense distributions and large-scale variations is an extremely challenging problem. This paper proposes a multi-stage, multi-scale local feature fusion method for infrared small target detection to address this problem. The method is based on multi-stage and multi-scale local feature fusion. Firstly, considering the significant variation in target sizes, ResNet-18 is utilized to extract image features at different stages. Then, for each stage, multi-scale feature pyramids are employed to obtain corresponding multi-scale local features. Secondly, to enhance the detection rate of densely distributed targets, the multi-stage and multi-scale features are progressively fused and concatenated to form the final fusion results. Finally, the fusion results are fed into the target detector for detection. The experimental results for the SIRST and MDFA demonstrate that the proposed method effectively improves the performance of infrared small target detection. The proposed method achieved mIoU values of 63.43% and 46.29% on two datasets, along with F-measure values of 77.62% and 63.28%, respectively.

An improved single-stage convolutional neural network for rail transit obstacle detection

Obstacles that intrude into the rail area can lead to serious rail accidents, so obstacle detection technology is an essential guarantee for the safe operation of fully automatic trains. To meet the high-performance requirements of onboard obstacle detection, an efficient feature-aware convolutional neural network (EFA-Net) is proposed in this paper. The multi-scale aware feature pyramid network (MA-FPN) is designed as feature fusion network to extract multi-scale context information. In the detection head, the dynamic awareness block is used to refine the features. A joint representation branch and the generalized focal loss function are introduced to optimize the training effect. The experiments are based on the dataset of real-world rail transit environment. The results show that EFA-Net can achieve a detection accuracy of 90.4% mAP at a detection speed of 20.4 frames per second, and the lightweight design significantly reduces the computational complexity of the proposed model. Compared with other classical detectors, EFA-Net has the best comprehensive performance.

OAMSFNet: Orientation-Aware and Multi-Scale Feature Fusion Network for shadow detection in remote sensing images via pseudo shadow

ABSTRACT Remote sensing images (RSI) of urban regions often exhibit shadows cast by buildings and other objects, which may result in imprecise analysis and interpretation. Therefore, shadow detection plays a significant role in RSI scene understanding. Current approaches have paid little attention to the presence of pseudo shadows and their confounding effects on detection results. We tackle these questions in the spatial context of orientation-awareness, effectively capturing the intricate relationships between shadows and ground objects. Here, we introduce a novel deep learning network named the Orientation-Aware and Multi-Scale Feature Fusion Network (OAMSFNet) due to noise reduction considerations. The proposed OAMSFNet comprises Shadow Aware Feature Encoder (SFE), Orientation-Aware Context Module (OCM), and Multi-Scale Feature Pyramid (MFP). Furthermore, a multi-scale feature fusion algorithm has been devised to enhance the detection and segmentation capabilities of the model in shadow regions. Finally, a comparative study was conducted on the Aerial Imagery dataset for Shadow Detection (AISD) in both quantitative and qualitative aspects. The experimental results show that our method outperforms state-of-the-art methods while maintaining a lightweight model design, indicating that our approach exhibits remarkable accuracy and stability in excellent agreement with predictions. OAMSFNet achieved an average F-score of 89.85%, surpassing the straightforward semantic segmentation model SegNet by 9.94%, and demonstrated remarkable efficiency enhancements, with FLOPs reduced to as low as 11.48 G, resulting in efficiency improvements of 66.7%-96.5% compared to other shadow detection models.

Pyramid Attention Network for Image Restoration

Self-similarity refers to the image prior widely used in image restoration algorithms that small but similar patterns tend to occur at different locations and scales. However, recent advanced deep convolutional neural network-based methods for image restoration do not take full advantage of self-similarities by relying on self-attention neural modules that only process information at the same scale. To solve this problem, we present a novel Pyramid Attention module for image restoration, which captures long-range feature correspondences from a multi-scale feature pyramid. Inspired by the fact that corruptions, such as noise or compression artifacts, drop drastically at coarser image scales, our attention module is designed to be able to borrow clean signals from their “clean” correspondences at the coarser levels. The proposed pyramid attention module is a generic building block that can be flexibly integrated into various neural architectures. Its effectiveness is validated through extensive experiments on multiple image restoration tasks: image denoising, demosaicing, compression artifact reduction, and super resolution. Without any bells and whistles, our PANet (pyramid attention module with simple network backbones) can produce state-of-the-art results with superior accuracy and visual quality. Our code is available at https://github.com/SHI-Labs/Pyramid-Attention-Networks