Pyramid Fusion Research Articles

Defect Detection Method of Phosphor in Glass Based on Improved YOLO5 Algorithm

Phosphor in Glass (PiG) is easy to be stirred unevenly during production and processing, and improper use of instruments and other factors lead to defective products. In this paper, we propose an improved YOLOv5 target detection algorithm. Firstly, the Coordinate Attention (CA) is introduced into the backbone network to enable the network to notice detect targets in a larger range. Secondly, the Bidirectional Feature Pyramid Network (BiFPN) is used to fuse different scale information in the neck part to obtain the output feature map with rich semantic information. At the same time, the weighted bidirectional feature fusion pyramid structure adjusts the contribution of different scale input feature maps to the output by introducing weights. This optimization enhances the feature fusion effect, reduces the loss of feature information in the convolution process, and improves detection accuracy. Then, the GIOU_Loss function is replaced with the EIOU_Loss function to speed up the convergence. Finally, the comparative experiment is carried out with the self-made PiG dataset. The experimental results show that the average accuracy mAP of this method is 12.35% higher than that of the original method (YOLOv5s), with a detection speed is 53.92 FPS, aligning with the actual needs of industrial detection.

Sfnet: Faster and Accurate Semantic Segmentation Via Semantic Flow

In this paper, we focus on exploring effective methods for faster and accurate semantic segmentation. A common practice to improve the performance is to attain high-resolution feature maps with strong semantic representation. Two strategies are widely used: atrous convolutions and feature pyramid fusion, while both are either computationally intensive or ineffective. Inspired by the Optical Flow for motion alignment between adjacent video frames, we propose a Flow Alignment Module (FAM) to learn Semantic Flow between feature maps of adjacent levels and broadcast high-level features to high-resolution features effectively and efficiently. Furthermore, integrating our FAM to a standard feature pyramid structure exhibits superior performance over other real-time methods, even on lightweight backbone networks, such as ResNet-18 and DFNet. Then to further speed up the inference procedure, we also present a novel Gated Dual Flow Alignment Module to directly align high-resolution feature maps and low-resolution feature maps where we term the improved version network as SFNet-Lite. Extensive experiments are conducted on several challenging datasets, where results show the effectiveness of both SFNet and SFNet-Lite. In particular, when using Cityscapes test set, the SFNet-Lite series achieve 80.1 mIoU while running at 60 FPS using ResNet-18 backbone and 78.8 mIoU while running at 120 FPS using STDC backbone on RTX-3090. Moreover, we unify four challenging driving datasets (i.e., Cityscapes, Mapillary, IDD, and BDD) into one large dataset, which we named Unified Driving Segmentation (UDS) dataset. It contains diverse domain and style information. We benchmark several representative works on UDS. Both SFNet and SFNet-Lite still achieve the best speed and accuracy trade-off on UDS, which serves as a strong baseline in such a challenging setting. The code and models are publicly available at https://github.com/lxtGH/SFSegNets.

A lightweight vehicle mounted multi-scale traffic sign detector using attention fusion pyramid

A lightweight vehicle mounted multi-scale traffic sign detector using attention fusion pyramid

Laplacian Pyramid Fusion Network With Hierarchical Guidance for Infrared and Visible Image Fusion

The fusion of infrared and visible images combines the information from two complementary imaging modalities for various computer vision tasks. Many existing techniques, however, fail to maintain a uniform overall style and keep salient details of individual modalities simultaneously. This paper presents an end-to-end Laplacian Pyramid Fusion Network with hierarchical guidance (HG-LPFN) that takes advantage of pixel-level saliency reservation of Laplacian Pyramid and global optimization capability of deep learning. The proposed scheme generates hierarchical saliency maps through Laplacian Pyramid decomposition and modal difference calculation. In the pyramid fusion mode, all sub-networks are connected in a bottom-up manner. The sub-network for low-frequency fusion focuses on extracting universal features to produce an opposite style while sub-networks for high-frequency fusion determine how much the details of each modality will be retained. Taking the style, details, and background into consideration, we design a set of novel loss functions to supervise both low-frequency images and full-resolution images under the guidance of saliency maps. Experimental results on public datasets demonstrate that the proposed HG-LPFN outperforms the state-of-the-art image fusion techniques.

Underwater image enhancement via multi-scale fusion and adaptive color-gamma correction in low-light conditions

In dark underwater areas, existing single-model underwater image enhancement methods have poor enhancement effects. We propose an underwater image enhancement method based on color correction and multi-scale fusion (CCMF). Specifically, we first design a color correction method with red channel compensation, which compensates for the red channel according to light attenuation and removes color bias. We propose a contrast enhancement method based on guided filtering to enhance edge texture details. The image is decomposed into a base layer and a detail layer in the logarithmic domain, with layered enhancement. Secondly, we propose an adaptive gamma correction method that dynamically adjusts correction parameters based on the gray image values. This approach prevents over-enhancement and effectively enhances the exposure in dark areas. We extract weight maps that represent different features from the input images and employ a multi-scale pyramid fusion technique to integrate the aforementioned feature information. This approach enables the mutual complementarity of various features and enhances the overall visual effect. Experimental results show that our method can effectively integrate the advantages of different enhancement methods, and the objective indicators of UCIQE, UIQM, and EG are better than other related state-of-the-art methods.

FPIseg: Iterative segmentation network based on feature pyramid for few‐shot segmentation

AbstractFew‐shot segmentation (FSS) enables rapid adaptation to the segmentation task of unseen‐classes object based on a few labelled support samples. Currently, the focal point of research in the FSS field is to align features between support and query images, aiming to improve the segmentation performance. However, most existing FSS methods implement such support/query alignment by solely leveraging middle‐level feature for generalization, ignoring the category semantic information contained in high‐level feature, while pooling operation inevitably lose spatial information of the feature. To alleviate these issues, the authors propose the Iterative Segmentation Network Based on Feature Pyramid (FPIseg), which mainly consists of three modules: Feature Pyramid Fusion Module (FPFM), Region Feature Enhancement Module (RFEM), and Iterative Optimization Segmentation Module (IOSM). Firstly, FPFM fully utilizes the foreground information from the support image to implement support/query alignment under multi‐scale, multi‐level semantic backgrounds. Secondly, RFEM enhances the foreground detail information of aligned feature to improve generalization ability. Finally, ISOM iteratively segments the query image to optimize the prediction result and improve segmentation performance. Extensive experiments on the PASCAL‐5i and COCO‐20i datasets show that FPIseg achieves considerable segmentation performance under both 1‐shot and 5‐shot settings.

Small Object Detection in Remote Sensing Images Based on Window Self-Attention Mechanism

For remorte sensing image object detection tasks in the small object feature, extraction ability is insufficient and difficult to locate, and other problems. This paper proposes an improved algorithm for small object detection in remote sensing images based on a window self-attention mechanism. On the basis of You Only Look Once (YOLO)v5s, a shallow feature extraction layer with four times downsampling is added to the feature fusion pyramid and the window self-attention mechanism is added to the Path Aggregation Network. Experiments show that the improved model obtained the Mean Average Precision (mAP) of 78.3% and 91.8% on the DIOR and Remote Sensing Object Detection public data sets with frames per second of 65 and 51, respectively. Compared with the basal YOLOv5s network, the mAP has improved by 5.8% and 3.3%, respectively. Compared with other object detection methods, the detection accuracy and real-time performance have been improved.

Underwater image enhancement via red channel maximum attenuation prior and multi-scale detail fusion.

The underwater environment poses great challenges, which have a negative impact on the capture and processing of underwater images. However, currently underwater imaging systems cannot adapt to various underwater environments to guarantee image quality. To address this problem, this paper designs an efficient underwater image enhancement approach that gradually adjusts colors, increases contrast, and enhances details. Based on the red channel maximum attenuation prior, we initially adjust the blue and green channels and correct the red channel from the blue and green channels. Subsequently, the maximum and minimum brightness blocks are estimated in multiple channels to globally stretch the image, which also includes our improved guided noise reduction filtering. Finally, in order to amplify local details without affecting the naturalness of the results, we use a pyramid fusion model to fuse local details extracted from two methods, taking into account the detail restoration effect of the optical model. The enhanced underwater image through our method has rich colors without distortion, effectively improved contrast and details. The objective and subjective evaluations indicate that our approach surpasses the state-of-the-art methods currently. Furthermore, our approach is versatile and can be applied to diverse underwater scenes, which facilitates subsequent applications.

OTFPF: Optimal transport based feature pyramid fusion network for brain age estimation

Deep neural networks have shown promise in predicting the chronological age of a healthy brain using T1-weighted magnetic resonance images (T1 MRIs). This predicted brain age has the potential to serve as a valuable biomarker for identifying development-related and aging-related disorders. In this study, we propose the Optimal Transport based Feature Pyramid Fusion (OTFPF) network for estimating brain age using T1 MRIs. The OTFPF network comprises three key modules: the Optimal Transport based Feature Pyramid Fusion (OTFPF) module, the 3D overlapped ConvNeXt (3D OL-ConvNeXt) module, and the fusion module. These modules enhance the OTFPF network's ability to comprehend the semi-multimodal and multi-level feature pyramid information of each brain, thereby improving its understanding of brain development and aging. Compared to recent state-of-the-art models, the proposed OTFPF network demonstrates faster convergence, superior performance, and enhanced interpretability. Experimental results utilizing a dataset of 12,909 MRIs from individuals aged 3–97 years demonstrate the accurate estimation of brain age by the OTFPF network, achieving a mean absolute error (MAE) of 1.846, Pearson's correlation coefficient (PCC) of 0.9941, and Spearman's rank correlation coefficient (SRCC) of 0.9802. Thorough parameter evaluations, quantitative comparison experiments, dataset-scale evaluations, cross-validations, and ablation studies convincingly demonstrate the stability, interpretability, and superiority of the OTFPF network. According to the OTFPF network, the age-related heatmaps of the brain explain the biological mechanisms underlying brain aging. Furthermore, the OTFPF network is applied to analyze datasets associated with brain disorders, effectively demonstrating its practical utility.

Refined feature fusion for in-field high-density and multi-scale rice panicle counting in UAV images

The yield of rice crops is strongly correlated to the number of panicles per unit area. Computer vision techniques have been utilized in previous studies to count the number of panicles automatically. However, when using images captured by Unmanned Aerial Vehicles (UAVs), the height and coverage area of the UAVs can cause shrinkage of the rice panicle objects, leading to errors in feature learning and reduced counting efficacy. In addition, there is a considerable relative size difference in rice ears, which can further affect counting accuracy. To address these issues, this paper proposes an algorithm named Refined Feature Fusion for Panicle Counting (RFF-PC). This algorithm utilizes a more refined scale division by extracting and fusing optimal features according to the object size distribution. Firstly, the number of rice ears with different sizes is quantified, and a fine division of the scale is made to calculate the receptive field size of different features that need to be fused. Secondly, multi-scale convolution generates ]features at each layer and feature pyramid fusion fuses more appropriate features at different layers to improve the ability to represent objects in complex multi-scale. Redundant information is removed through channel attention. Additionally, a refined Gaussian is used to generate ground truth close to the shape of real rice panicles. Over the UFPC2019 dataset captured at the height of 5 meters, the average counting accuracy of RFF-PC is 89.80%, and the counting accuracy only decreases from 94.33% to 90.58% over the images with an increased number of rice ears from 180 to 260, which outperforms several state-of-the-art algorithms, including MCNN, CSRNet, TasselNetV2+, DSNet, and DMCount.

SY-Net: A Rice Seed Instance Segmentation Method Based on a Six-Layer Feature Fusion Network and a Parallel Prediction Head Structure.

During the rice quality testing process, the precise segmentation and extraction of grain pixels is a key technique for accurately determining the quality of each seed. Due to the similar physical characteristics, small particles and dense distributions of rice seeds, properly analysing rice is a difficult problem in the field of target segmentation. In this paper, a network called SY-net, which consists of a feature extractor module, a feature pyramid fusion module, a prediction head module and a prototype mask generation module, is proposed for rice seed instance segmentation. In the feature extraction module, a transformer backbone is used to improve the ability of the network to learn rice seed features; in the pyramid fusion module and the prediction head module, a six-layer feature fusion network and a parallel prediction head structure are employed to enhance the utilization of feature information; and in the prototype mask generation module, a large feature map is used to generate high-quality masks. Training and testing were performed on two public datasets and one private rice seed dataset. The results showed that SY-net achieved a mean average precision (mAP) of 90.71% for the private rice seed dataset and an average precision (AP) of 16.5% with small targets in COCO2017. The network improved the efficiency of rice seed segmentation and showed excellent application prospects in performing rice seed quality testing.

Pixel-level road crack detection in UAV remote sensing images based on ARD-Unet

Crack is the main manifestation of road damage, and its further deterioration will affect road traffic. The timely detection of road cracks is of great significance for ensuring road safety. In this work, starting from UAV remote sensing images,a pixel-by-pixel crack detection method named ARD-Unet is proposed based on U-Net combined with Depth Separable Residual Block (DR-Block), Atrous Spatial Pyramid Fusion Attention Module (ASAM) and Receptive Field Block (RFB). We used UAV to construct a remote sensing road crack dataset containing 1046 high-quality images. The proposed method achieves 76.41 % MIOU and 74.24 % F1-Score on the self-made dataset. Finally, we combine ARD-Unet with UAV to build a road crack detection UAV IoT system, which has been tested in practical applications and achieved excellent performance. Experiments show that ARD-Unet is effective for road crack detection in remote sensing images.

PFFNET: A Fast Progressive Feature Fusion Network for Detecting Drones in Infrared Images

The rampant misuse of drones poses a serious threat to national security and human life. Currently, CNN (Convolutional Neural Networks) are widely used to detect drones. However, small drone targets often reduced amplitude or even lost features in infrared images which traditional CNN cannot overcome. This paper proposes a Progressive Feature Fusion Network (PFFNET) and designs a Pooling Pyramid Fusion (PFM) to provide more effective global contextual priors for the highest downsampling output. Then, the Feature Selection Model (FSM) is designed to improve the use of the output coding graph and enhance the feature representation of the target in the network. Finally, a lightweight segmentation head is designed to achieve progressive feature fusion with multi-layer outputs. Experimental results show that the proposed algorithm has good real-time performance and high accuracy in drone target detection. On the public dataset, the intersection over union (IOU) is improved by 2.5% and the detection time is reduced by 81%.

Underwater Target Detection Algorithm Based on Feature Fusion Enhancement

Underwater robots that use optical images for dynamic target detection often encounter image blurring, poor contrast, and indistinct target features. As a result, the underwater robots have poor detection performance with a high rate of missed detections. To overcome these issues, a feature-enhanced algorithm for underwater target detection has been proposed in this paper. Based on YOLOv7, a feature enhancement module utilizing a triple-attention mechanism is developed to improve the network’s feature extraction ability without increasing the computational or algorithmic parameter quantity. Moreover, comprehensively considering the impact of a redundant feature in the images on detection accuracy, the ASPPCSPC structure was built. A parallel spatial convolutional pooling structure based on the original feature pyramid fusion structure, SPPCSPC, is introduced. The GhostNet network was utilized to optimize its convolution module, which reduces the model’s parameter quantity and optimizes the feature map. Furthermore, a Cat-BiFPN structure was designed to address the problem of fine-grained information loss in YOLOv7 feature fusion by adopting a weighted nonlinear fusion strategy to enhance the algorithm’s adaptability. Using the UPRC offshore dataset for validation, the algorithm’s detection accuracy was increased by 2.9%, and the recall rate was improved by 2.3% compared to the original YOLOv7 algorithm. In addition, the model quantity is reduced by 11.2%, and the model size is compressed by 10.9%. The experimental results significantly establish the validity of the proposed algorithm.

EFPN: Effective medical image detection using feature pyramid fusion enhancement

Feature pyramid networks (FPNs) are widely used in the existing deep detection models to help them utilize multi-scale features. However, there exist two multi-scale feature fusion problems for the FPN-based deep detection models in medical image detection tasks: insufficient multi-scale feature fusion and the same importance for multi-scale features. Therefore, in this work, we propose a new enhanced backbone model, EFPNs, to overcome these problems and help the existing FPN-based detection models to achieve much better medical image detection performances. We first introduce an additional top-down pyramid to help the detection networks fuse deeper multi-scale information; then, a scale enhancement module is developed to use different sizes of kernels to generate more diverse multi-scale features. Finally, we propose a feature fusion attention module to estimate and assign different importance weights to features with different depths and scales. Extensive experiments are conducted on two public lesion detection datasets for different medical image modalities (X-ray and MRI). On the mAP and mR evaluation metrics, EFPN-based Faster R-CNNs improved 1.55% and 4.3% on the PenD (X-ray) dataset, and 2.74% and 3.1% on the BraTs (MRI) dataset, respectively. EFPN-based Faster R-CNNs achieve much better performances than the state-of-the-art baselines in medical image detection tasks. The proposed three improvements are all essential and effective for EFPNs to achieve superior performances; and besides Faster R-CNNs, EFPNs can be easily applied to other deep models to significantly enhance their performances in medical image detection tasks.

Pyramid feature fusion through shifted window self-attention for tobacco leaf classification

Classification of tobacco leaves by stalk position and color is a necessary prerequisite for tobacco accurate pricing and subsequent fine blending. The similarity of color, contour, and texture features limits the classification accuracy of current tobacco recognition technology, which is focusing on either a local feature or global feature extraction. To improve the feature fusion ability, we propose a tobacco classification model FSWPNet based on pyramid feature fusion through shifted window self-attention. Firstly, owing to the large proportion of tobacco targets in the original image, it is difficult for the standard pooling module to make full use of the overall appearance feature information. The Serial Maximum Spatial Pyramid Pooling (SMSPP) module is designed to aggregate multi-scale leaf apex features by concatenating multiple maximum pooling layers consecutively. Subsequently, to fusion, both local and global tobacco features, the Shifted Window Transformer Bottleneck (STRB) module is integrated into the feature pyramid fusion architecture. In the window encoder, window self-attention focuses on the low-level semantic local features. While shifted window self-attention is capable of long-distance modeling. It can obtain macroscopic tobacco features such as color, contour, and texture through a cross-window connection. The tobacco classification needs to be combined with the comprehensive identification of multiple feature information. The Spatial and Channel Mixed Multi-Layer Perceptron (SCMix MLP) is used to blend intra-patch positional feature and inter-patch spatial feature information. Finally, aiming at the hard samples with similar features, FSWPNet adopted DIoU Non-Maximum Suppression (DIoU-NMS) strategy for calculating the minimum outer rectangular diagonal distance to eliminate redundant regression boxes with low scores. The FSWPNet model is deployed and tested on the tobacco grading equipment. The experimental results show that FSWPNet achieves the best performance in terms of average classification accuracy 75.8%, inference time 12 ms and model size 16.8 MB. The proposed FSWPNet model can effectively classify 10 categories of tobacco agricultural products, which provides an important theoretical basis and premise for acquisition and fine processing.

Beyond Trade-Off: An Optimized Binocular Stereo Vision Based Depth Estimation Algorithm for Designing Harvesting Robot in Orchards

Depth estimation is one of the bottleneck parts for harvesting robots to determine whether the operation of grasping or picking succeeds or not directly. This paper proposed a novel disparity completion method combined with bilateral filtering and pyramid fusion to improve the issues of incorrect outputs due to the missed or wrong matching when achieving 3D position from 2D images in open-world environments. Briefly, our proposed method has two significant advantages in general. Firstly, occlusion between leaves, branches, and fruits is a universal phenomenon in unstructured orchard environments, which results in the most depth estimation algorithms facing great challenges to obtain accurate outputs in these occluded regions. To alleviate these issues, unlike other research efforts that already exist, we optimized the semi-global matching algorithm to obtain high accuracy sparse values as an initial disparity map; then, an improved bilateral filtering algorithm is proposed to eliminate holes and discontinuous regions caused by occlusion to obtain precise and density disparity outputs. Secondly, due to taking the practical high-efficiency requirements of the automated harvesting robot in its working status into consideration, we attempted to merge multiple low-resolution bilateral filtering results through the pyramid fusion model which goes beyond the trade-off mechanism to improve the performance of both accuracy and time cost. Finally, a prototype harvesting robot was designed to conduct experiments at three kinds of different distances (0.6~0.75 m, 1~1.2 m, and 1.6~1.9 m). Experiment results showed that our proposed method achieved density disparity maps and eliminated holes and discontinuous defects in the disparity map effectively. The average absolute error of our proposed method is 3.2 mm, and the average relative error is 1.79%. In addition, the time cost is greatly reduced more than 90%. Comprehensive experimental results demonstrate that our proposed algorithm provides a potential possibility for designing harvesting.

Crowd counting in smart city via lightweight Ghost Attention Pyramid Network

Crowd counting targets for determining the number of pedestrians in an image, which is of crucial importance for smart city construction. The problem of scale variation is an ingrained and drastic challenge in crowd counting, and it severely degrades the performance of counting. To address this problem, many powerful models with complex network structures and tricks are built, but the constrained resources of embedded systems prevent the direct deployment of these models into an edge device. Thus, it is on high demand to design favourable lightweight models that require fewer parameters and a fast inference speed, while maintaining competitive counting performance. To this aim, we devise a lightweight network, termed as Ghost Attention Pyramid Network (GAPNet). Specifically, a lightweight GhostNet is adopted as the backbone to encode low-level features. Subsequently, a zero-parameter channel attention module is designed to select the discriminative crowd region efficiently. In addition, an efficient pyramid fusion module is built with a four-branch architecture to obtain multiscale hierarchy representations while reducing the parameters. Finally, a decoder generates the prediction by exploiting a series of transposed convolution blocks. Extensive experiments on crowd counting benchmarks have proved the superiority of the GAPNet in both accuracy and efficiency.

HIGF-Net: Hierarchical information-guided fusion network for polyp segmentation based on transformer and convolution feature learning

Polyp segmentation plays a role in image analysis during colonoscopy screening, thus improving the diagnostic efficiency of early colorectal cancer. However, due to the variable shape and size characteristics of polyps, small difference between lesion area and background, and interference of image acquisition conditions, existing segmentation methods have the phenomenon of missing polyp and rough boundary division. To overcome the above challenges, we propose a multi-level fusion network called HIGF-Net, which uses hierarchical guidance strategy to aggregate rich information to produce reliable segmentation results. Specifically, our HIGF-Net excavates deep global semantic information and shallow local spatial features of images together with Transformer encoder and CNN encoder. Then, Double-stream structure is used to transmit polyp shape properties between feature layers at different depths. The module calibrates the position and shape of polyps in different sizes to improve the model's efficient use of the rich polyp features. In addition, Separate Refinement module refines the polyp profile in the uncertain region to highlight the difference between the polyp and the background. Finally, in order to adapt to diverse collection environments, Hierarchical Pyramid Fusion module merges the features of multiple layers with different representational capabilities. We evaluate the learning and generalization abilities of HIGF-Net on five datasets using six evaluation metrics, including Kvasir-SEG, CVC-ClinicDB, ETIS, CVC-300, and CVC-ColonDB. Experimental results show that the proposed model is effective in polyp feature mining and lesion identification, and its segmentation performance is better than ten excellent models.

Image Dehazing Based on Improved Color Channel Transfer and Multiexposure Fusion

Image dehazing is one of the problems that need to be solved urgently in the field of computer vision. In recent years, more and more algorithms have been applied to image dehazing and achieved good results. However, the image after dehazing still has color distortion, contrast and saturation disorder, and other challenges; in order to solve these problems, in this paper, an effective image dehazing method is proposed, which is based on improved color channel transfer and multiexposure image fusion to achieve image dehazing. First, the image is preprocessed using a color channel transfer method based on k-means. Second, gamma correction is introduced on the basis of guided filtering to obtain a series of multiexposure images, and the obtained multiexposure images are fused into a dehazed image through a Laplacian pyramid fusion scheme based on local similarity of adaptive weights. Finally, contrast and saturation corrections are performed on the dehazed image. Experimental verification is carried out on synthetic dehazed images and natural dehazed images, and it is verified that the method proposed is superior to existing dehazed algorithms from both subjective and objective aspects.