Multi-scale Fusion Research Articles

Ship Detection in Visible Remote Sensing Image Based on Saliency Extraction and Modified Channel Features

Ship detection in visible remote sensing (VRS) images has been widely used in the military and civil fields. However, the various backgrounds and the variable scale and orientation bring great difficulties to effective detection. In this paper, we propose a novel ship target detection scheme based on small training samples. The scheme contains two main stages: candidate region extraction and ship identification. In the first stage, we propose a visual saliency detection model based on the difference in covariance statistical characteristics to quickly locate potential ships. Moreover, the multi-scale fusion for the saliency model is designed to overcome the problem of scale variation. In the second stage, we propose a three-channel aggregate feature, which combines a rotation-invariant histogram of oriented gradient and the circular frequency feature. The feature can identify the ship target well by avoiding the impact of its rotation and shift. Finally, we propose the VRS ship dataset that contains more realistic scenes. The results on the VRS ship dataset demonstrate that the saliency model achieves the best AUC value with 0.9476, and the overall detection achieves a better performance of 65.37% in terms of AP@0.5:0.95, which basically meets the need of the detection tasks.

Coarse-to-fine multiscale fusion network for single image deraining

Cameras convert optical signals into electronic information to enable image acquisition. However, cameras are susceptible to environmental factors such as light and weather conditions. The information captured by cameras in rainy condition tends to degrade, resulting in reduced contrast and visibility of the images. Deraining aims to restore the image degraded by raindrops and rain accumulation in rainy weather. Most state-of-the-art, end-to-end neural network-based deraining methods have achieved satisfactory results. By considering the features from different scales, we proposed a further improved multiscale information exchange module to extract features from different scales. As a result, the structure achieves a competitive effect on rain mask detection. Moreover, experiments have found that the rain layer-based prediction models tend to leave rain streak residuals in specific image patches. Therefore, we further developed the Feature Attention Compensation module (FAC) to better utilize the deraining image and rain layer information obtained from the multiscale model, which is demonstrated to boost deraining performance. To summarize, we design a coarse-to-fine rain removal model, starting with a rain detection network to get a coarse rain-free image, which is further refined with the FAC module to eventually create a fine rain-free image. We conduct our experiments on both synthetic and real-world datasets. Quantitative and qualitative experimental results demonstrate that the proposed method outperforms the state-of-the-art deraining methods. Source code will be available at https://github.com/Jstar-s/CFMFN

TransMF: Transformer-Based Multi-Scale Fusion Model for Crack Detection

Cracks are widespread in infrastructure that are closely related to human activity. It is very popular to use artificial intelligence to detect cracks intelligently, which is known as crack detection. The noise in the background of crack images, discontinuity of cracks and other problems make the crack detection task a huge challenge. Although many approaches have been proposed, there are still two challenges: (1) cracks are long and complex in shape, making it difficult to capture long-range continuity; (2) most of the images in the crack dataset have noise, and it is difficult to detect only the cracks and ignore the noise. In this paper, we propose a novel method called Transformer-based Multi-scale Fusion Model (TransMF) for crack detection, including an Encoder Module (EM), Decoder Module (DM) and Fusion Module (FM). The Encoder Module uses a hybrid of convolution blocks and Swin Transformer block to model the long-range dependencies of different parts in a crack image from a local and global perspective. The Decoder Module is designed with symmetrical structure to the Encoder Module. In the Fusion Module, the output in each layer with unique scales of Encoder Module and Decoder Module are fused in the form of convolution, which can release the effect of background noise and strengthen the correlations between relevant context in order to enhance the crack detection. Finally, the output of each layer of the Fusion Module is concatenated to achieve the purpose of crack detection. Extensive experiments on three benchmark datasets (CrackLS315, CRKWH100 and DeepCrack) demonstrate that the proposed TransMF in this paper exceeds the best performance of present baselines.

Underwater Image Enhancement by Attenuated Color Channel Correction and Detail Preserved Contrast Enhancement

An underwater image often suffers from quality degradation issues, such as color deviations, low contrast, and blurred details, due to the absorption and scattering of light. In this article, we propose to address the aforementioned degradation issues via attenuated color channel correction and detail preserved contrast enhancement. Concretely, we first propose an underwater image color correction method. Considering the differences between superior and inferior color channels of an underwater image, the inferior color channels are compensated via especially designed attenuation matrices. We then employ a dual-histogram-based iterative threshold method and a limited histogram method with Rayleigh distribution to improve the global and local contrast of the color-corrected image, thus achieving a global contrast-enhanced version and a local contrast-enhanced version, respectively. To integrate the complementary merits between the global contrast-enhanced version and the local contrast-enhanced version, we adopt a multiscale fusion strategy to fuse them. Finally, we propose a multiscale unsharp masking strategy to further sharpen the fused image for better visual quality. Extensive experiments on four underwater image enhancement benchmark data sets demonstrate that our method effectively enhances underwater images qualitatively and quantitatively. Besides, our method also generalizes well to the enhancement of low-light images and hazy images.

Remaining Useful Life Prediction Model for Rolling Bearings Based on MFPE-MACNN.

Aiming to resolve the problem of redundant information concerning rolling bearing degradation characteristics and to tackle the difficulty faced by convolutional deep learning models in learning feature information in complex time series, a prediction model for remaining useful life based on multiscale fusion permutation entropy (MFPE) and a multiscale convolutional attention neural network (MACNN) is proposed. The original signal of the rolling bearing was extracted and decomposed by resonance sparse decomposition to obtain the high-resonance and low-resonance components. The multiscale permutation entropy of the low-resonance component was calculated. Moreover, the locally linear-embedding algorithm was used for dimensionality reduction to remove redundant information. The multiscale convolution module was constructed to learn the feature information at different time scales. The attention module was used to fuse the feature information and input it into the remaining useful life prediction module for evaluation. The appropriate network structure and parameter configuration were determined, and a multiscale convolutional attention neural network was designed to determine the remaining useful life prediction model. The results show that the method demonstrates effectiveness and superiority in degrading the feature information representation and improving the remaining useful life prediction accuracy compared with other models.

Learning from the Target: Dual Prototype Network for Few Shot Semantic Segmentation

Due to the scarcity of annotated samples, the diversity between support set and query set becomes the main obstacle for few shot semantic segmentation. Most existing prototype-based approaches only exploit the prototype from the support feature and ignore the information from the query sample, failing to remove this obstacle.In this paper, we proposes a dual prototype network (DPNet) to dispose of few shot semantic segmentation from a new perspective. Along with the prototype extracted from the support set, we propose to build the pseudo-prototype based on foreground features in the query image. To achieve this goal, the cycle comparison module is developed to select reliable foreground features and generate the pseudo-prototype with them. Then, a prototype interaction module is utilized to integrate the information of the prototype and the pseudo-prototype based on their underlying correlation. Finally, a multi-scale fusion module is introduced to capture contextual information during the dense comparison between prototype (pseudo-prototype) and query feature. Extensive experiments conducted on two benchmarks demonstrate that our method exceeds previous state-of-the-arts with a sizable margin, verifying the effectiveness of the proposed method.

SCIR-Net: Structured Color Image Representation Based 3D Object Detection Network from Point Clouds

3D object detection from point clouds data has become an indispensable part in autonomous driving. Previous works for processing point clouds lie in either projection or voxelization. However, projection-based methods suffer from information loss while voxelization-based methods bring huge computation. In this paper, we propose to encode point clouds into structured color image representation (SCIR) and utilize 2D CNN to fulfill the 3D detection task. Specifically, we use the structured color image encoding module to convert the irregular 3D point clouds into a squared 2D tensor image, where each point corresponds to a spatial point in the 3D space. Furthermore, in order to fit for the Euclidean structure, we apply feature normalization to parameterize the 2D tensor image onto a regular dense color image. Then, we conduct repeated multi-scale fusion with different levels so as to augment the initial features and learn scale-aware feature representations for box prediction. Extensive experiments on KITTI benchmark, Waymo Open Dataset and more challenging nuScenes dataset show that our proposed method yields decent results and demonstrate the effectiveness of such representations for point clouds.

Tiny‐Crack‐Net: A multiscale feature fusion network with attention mechanisms for segmentation of tiny cracks

AbstractConvolutional neural networks (CNNs) have gained growing interest in recent years for their advantages in detecting cracks on concrete bridge components. Class imbalance is a fundamental problem in crack segmentation, resulting in unsatisfactory segmentation for tiny cracks. Besides, limited by the local receptive field, CNNs often cannot integrate local features with global dependencies, thus significantly affecting the detection accuracy of tiny cracks across the entire image. To solve those problems in segmenting tiny cracks, a multiscale feature fusion network with attention mechanisms named “Tiny‐Crack‐Net” (TCN) is proposed. The modified residual network was used to capture the local features of tiny cracks. The dual attention module was then incorporated into the architecture to better separate the tiny cracks from the background. Also, a multiscale fusion operation was implemented to preserve the edge details of tiny cracks. Finally, a joint learning loss of the cross‐entropy and similarity was proposed to alleviate the poor convergence induced by the severe class imbalance of the pixels representing tiny cracks. The capability of the network in segmenting tiny cracks was remarkably enhanced by the aforementioned arrangements, and the “Tiny‐Crack‐Net” achieved a Dice similarity coefficient of 87.96% on an open‐source data set, which was at least 5.84% higher than those of the six cutting‐edge networks. The effectiveness and robustness of the “Tiny‐Crack‐Net” were validated with field test results, which showed that the intersection over union (IOU) for cracks with a width of 0.05 mm or wider reaches 91.44%.

Joint Transformer and Multi-scale CNN for DCE-MRI Breast Cancer Segmentation

Automatic segmentation of breast cancer lesions in dynamic contrast-enhanced magnetic resonance imaging is challenged by low accuracy of delineation of the infiltration area, variable structure and shapes, large intensity heterogeneity changes, and low boundary contrast. This study constructed a two-stage breast cancer image segmentation framework and proposes a novel breast cancer lesion segmentation model (TR-IMUnet). The benchmark U-Net network model enables a rough delineation of the breast area in the acquired images and eliminates the influence of unrelated tissues (chest muscle, fat, and heart) on breast tumor segmentation. Based on the extracted results of the region of interest, the rectified linear unit (ReLU) function of the encoding–decoding structure in the model was replaced by an improved ReLU function to reserve and adjust the data dynamically according to input information. The segmentation accuracy of breast cancer lesions was improved by embedding a multi-scale fusion block and a transformer module in the coding path of the model, thereby obtaining multi-scale and global attention information. The experimental results showed that the breast tumor segmentation indexes Dice coefficient (Dice), Intersection over Union (IoU), Sensitivity (SEN), and Positive Predictive Value (PPV) increased by 4.27, 5.21, 3.37, and 3.68%, respectively, relative to the U-Net reference model. The proposed model improves the segmentation results of breast cancer lesions and reduces small area mis-segmentation and calcification segmentation.

Smear character recognition method of side-end power meter based on PCA image enhancement

Abstract Since it is difficult for manual recording to track the rapid change of indication of the power meter, the power meter images are collected by the camera and automatically recognized and recorded to effectively overcome the disadvantages of manual recording. However, the complex scene lighting environment and smearing character shadows make it difficult to transfer captured images directly to convolutional neural networks for character recognition. A smear character recognition method of side-end power meter under complex lighting conditions is proposed in this article. First, the uneven illumination image enhancement algorithm is studied. Through the estimation of the illumination component of the image, the fusion weight is calculated by the principal component analysis for multiscale fusion, and the up-sampling and down-sampling are adopted to reduce the calculation of the algorithm and achieve the rapid image enhancement. A convolution neural network framework based on deep learning is proposed to realize the segmentation of smear characters, and the final segmented individual characters are fed into a network to identify the meter readings. The experimental results show that the proposed smear character recognition method has fast recognition speed, and the recognition rate of samples with the smear character and complex illumination reaches 99.8%, which meets the requirements of power meter character recognition and is better than other algorithms.

PPG-based blood pressure estimation can benefit from scalable multi-scale fusion neural networks and multi-task learning

The continuous measurement of blood pressure (BP) plays an important role in preventing cardiovascular diseases. However, common cuff-based devices with cumbersome operations are difficult to realize continuous BP monitoring. We propose a continuous BP estimation method based on the end-to-end multi-scale fusion and multi-task learning neural network (MSF-MTLNet). It can use the 8-second PPG segment and its derivatives as input and automatically learn their multi-scale features to improve BP accuracy. The proposed method implements feature extraction by adaptively fusing multiple convolution kernels with different sizes through the attention mechanism. Simultaneously, multi-task learning was introduced to discover task-relevant features. A total of 277,600 segments from 1825 patients in the University of California, Irvine database are used for experiments. They are divided into the independent training set, validation set, and test set according to the ratio of 8:1:1 in an inter-subject manner. The mean and standard deviation of estimation error for systolic blood pressure and diastolic blood pressure are 0.97 ± 8.87 mmHg and 0.55 ± 4.23 mmHg, respectively, which outperforms the current cutting-edge method. Our approach almost meets the Association for the Advancement of Medical Instrumentation standard. Considering the British Hypertension Society standards, it achieves an 'A' grade for DBP estimation and a 'B' grade for SBP estimation. Experimental results show that MSF-MTLNet can obtain promising precision in cuffless and continuous BP estimation.

Tracking and Counting of Tomato at Different Growth Period Using an Improving YOLO-Deepsort Network for Inspection Robot

To realize tomato growth period monitoring and yield prediction of tomato cultivation, our study proposes a visual object tracking network called YOLO-deepsort to identify and count tomatoes in different growth periods. Based on the YOLOv5s model, our model uses shufflenetv2, combined with the CBAM attention mechanism, to compress the model size from the algorithm level. In the neck part of the network, the BiFPN multi-scale fusion structure is used to improve the prediction accuracy of the network. When the target detection network completes the bounding box prediction of the target, the Kalman filter algorithm is used to predict the target’s location in the next frame, which is called the tracker in this paper. Finally, calculate the bounding box error between the predicted bounding box and the bounding box output by the object detection network to update the parameters of the Kalman filter and repeat the above steps to achieve the target tracking of tomato fruits and flowers. After getting the tracking results, we use OpenCV to create a virtual count line to count the targets. Our algorithm achieved a competitive result based on the above methods: The mean average precision of flower, green tomato, and red tomato was 93.1%, 96.4%, and 97.9%. Moreover, we demonstrate the tracking ability of the model and the counting process by counting tomato flowers. Overall, the YOLO-deepsort model could fulfill the actual requirements of tomato yield forecast in the greenhouse scene, which provide theoretical support for crop growth status detection and yield forecast.

Birds Detection in Natural Scenes Based on Improved Faster RCNN

To realize the accurate detection of small-scale birds in natural scenes, this paper proposes an improved Faster RCNN model to detect bird species. Firstly, the model uses a depth residual network to extract convolution features and performs multi-scale fusion for feature maps of different convolutional layers. Secondly, the K-means clustering algorithm is used to cluster the bounding boxes. We improve the anchoring according to the clustering results. The improved anchor frame tends toward the real bounding box of the dataset. Finally, the Soft Non-Maximum Suppression method is used to reduce the missed detection of overlapping birds. Compared with the original model, the improved model has faster effect and higher accuracy.

Le-SKT: Lightweight traffic density estimation method based on structured knowledge transfer

The traffic density estimation algorithm usually runs on an embedded platform, and its efficiency is very important to the implementation of applications. However, most of the existing methods rely on heavy backbone networks, require high-performance hardware platform support and take a long time to calculate, which are difficult prerequisites to apply in practice. In consideration of the aforementioned problems, this paper proposes a lightweight traffic density estimation method based on structured knowledge transfer (Le-SKT). This method consists of a teacher network and student network. In the teacher network, in order to solve the problem of variable vehicle scales in video images, a multiscale fusion structure is designed, which can fully fuse the low-level and high-level features of an image. To solve the problem of background interference in video images, a fusion module based on an attention mechanism is constructed to enhance the useful traffic flow feature information and suppress the background information. At the same time, the fused traffic flow feature information is dilated convoluted to expand the receptive field and obtain richer contextual vehicle feature information, which generates a trained teacher network. Finally, a lightweight student network is generated by using the structured knowledge of the teacher network, which solves the problems of long computing times and high hardware requirements. The MAE and GAME of this method are reduced to 3.92 and 5.69, respectively in the TRANCOS dataset and 3.74 and 5.57, respectively in the VisDrone2019 vehicle dataset, respectively; The parameter quantity of the student network is only 0.65% of that of the teacher network, which greatly reduces the scale of the neural network model and is suitable for an embedded computing platform with low performance.

HRNet Encoder and Dual-Branch Decoder Framework-Based Scene Text Recognition Model

Scene text recognition (STR) is designed to automatically recognize the text content in natural scenes. Different from regular document text, text in natural scenes has the characteristics of irregular shapes, complex background, and distorted and blurred contents, which makes STR challenging. To solve the problems of STR for distorted, blurred, and low-resolution texts in natural scenes, this paper proposes a HRNet encoder and dual-branch decoder framework-based STR model. The model mainly consists of an encoder module and a dual-branch decoder module composed of a super-resolution branch and a recognition branch in parallel. In the encoder module, the HRNet is adopted to realize the cross-parallel aggregation representation with multiple resolutions during feature extraction and then outputs four kinds of feature maps with different resolutions. Moreover, the supervised attention module is used to strengthen the learning of the important feature information. In the decoder module, the dual-branch structure is adopted, in which the super-resolution branch takes the feature maps with the highest resolution obtained in the encoder module as input and restores images by upsampling through transposed convolution. The four kinds of feature maps with different resolutions are fused through independent transposed convolution layers for multiscale fusion in the recognition branch and then inputted into the attention-based decoder for text recognition. To improve the accuracy of text recognition, the feature extraction effect of the encoder module is together supervised by the super-resolution branch loss and the recognition branch loss. In addition, the super-resolution branch is only used for training and is abandoned during testing to reduce the complexity of the model. The proposed model is trained on Synth90K and SynthText datasets and tested on seven natural scene datasets. Compared with classical models such as ASTER, TextSR, and SCGAN, the recognition accuracy of the proposed model is improved and better recognition results can be achieved on irregular and blurred datasets such as IC15, SVTP, and CUTE80.

Convolutional Neural Network with Multiscale Fusion and Attention Mechanism for Skin Diseases Assisted Diagnosis.

Melanoma segmentation based on a convolutional neural network (CNN) has recently attracted extensive attention. However, the features captured by CNN are always local that result in discontinuous feature extraction. To solve this problem, we propose a novel multiscale feature fusion network (MSFA-Net). MSFA-Net can extract feature information at different scales through a multiscale feature fusion structure (MSF) in the network and then calibrate and restore the extracted information to achieve the purpose of melanoma segmentation. Specifically, based on the popular encoder-decoder structure, we designed three functional modules, namely MSF, asymmetric skip connection structure (ASCS), and calibration decoder (Decoder). In addition, a weighted cross-entropy loss and two-stage learning rate optimization strategy are designed to train the network more effectively. Compared qualitatively and quantitatively with the representative neural network methods with encoder-decoder structure, such as U-Net, the proposed method can achieve advanced performance.

Underwater image enhancement based on color correction and complementary dual image multi-scale fusion.

Underwater images often suffer from color cast, poor contrast, and detail loss owing to the scattering and absorption of light in water. To solve these problems, we propose what we believe to be a novel underwater image enhancement method based on color correction and dual image multi-scale fusion. We first use the color correction method to solve the problem of color cast, and we compensate the other two-color channels with the highest mean value color channel; further, all the color channels are dynamically stretched. Next, a complementary dual image multi-scale fusion method is used to improve the contrast, pairs of complementary adaptive gamma correction with weighted distribution enhanced images are used as the two inputs of multi-scale fusion, and appropriate weight maps are selected. Then, a multi-scale detail-sharpening method is used to enhance the image details. Qualitative and quantitative evaluations prove that the proposed method can produce high-quality underwater images. Moreover, the proposed method has relatively high evaluator values compared to the state-of-the-art methods.

DOPE++: 6D pose estimation algorithm for weakly textured objects based on deep neural networks.

This paper focuses on 6D pose estimation for weakly textured targets from RGB-D images. A 6D pose estimation algorithm (DOPE++) based on a deep neural network for weakly textured objects is proposed to solve the poor real-time pose estimation and low recognition efficiency in the robot grasping process of parts with weak texture. More specifically, we first introduce the depthwise separable convolution operation to lighten the original deep object pose estimation (DOPE) network structure to improve the network operation speed. Second, an attention mechanism is introduced to improve network accuracy. In response to the low recognition efficiency of the original DOPE network for parts with occlusion relationships and the false recognition problem in recognizing parts with scales that are too large or too small, a random mask local processing method and a multiscale fusion pose estimation module are proposed. The results show that our proposed DOPE++ network improves the real-time performance of 6D pose estimation and enhances the recognition of parts at different scales without loss of accuracy. To address the problem of a single background representation of the part pose estimation dataset, a virtual dataset is constructed for data expansion to form a hybrid dataset.

Performance analysis of different DCNN models in remote sensing image object detection

In recent years, deep learning, especially deep convolutional neural networks (DCNN), has made great progress. Many researchers use different DCNN models to detect remote sensing targets. Different DCNN models have different advantages and disadvantages. In this paper, we use YoloV4 as the detector to “fine-tune” various mainstream deep convolutional neural networks on two large public remote sensing data sets−LEVIR data set and DOTA data set to compare the advantages of various networks. This paper analyzes the reasons why the effect of “fine-tuning” convolutional neural networks is sometimes not good, and points out the difficulties of object detection in optical remote sensing images. To improve the detection accuracy of optical remote sensing targets, in addition to “fine-tuning” convolutional neural network, we also provide a variety of adaptive multi-scale feature fusion methods to improve the detection accuracy. In addition, for the large number of parameters generated by deep convolutional neural network, we provide a method to save storage space.

DNL-Net: deformed non-local neural network for blood vessel segmentation

BackgroundThe non-local module has been primarily used in literature to capturing long-range dependencies. However, it suffers from prohibitive computational complexity and lacks the interactions among positions across the channels.MethodsWe present a deformed non-local neural network (DNL-Net) for medical image segmentation, which has two prominent components; deformed non-local module (DNL) and multi-scale feature fusion. The former optimizes the structure of the non-local block (NL), hence, reduces the problem of excessive computation and memory usage, significantly. The latter is derived from the attention mechanisms to fuse the features of different levels and improve the ability to exchange information across channels. In addition, we introduce a residual squeeze and excitation pyramid pooling (RSEP) module that is like spatial pyramid pooling to effectively resample the features at different scales and improve the network receptive field.ResultsThe proposed method achieved 96.63% and 92.93% for Dice coefficient and mean intersection over union, respectively, on the intracranial blood vessel dataset. Also, DNL-Net attained 86.64%, 96.10%, and 98.37% for sensitivity, accuracy and area under receiver operation characteristic curve, respectively, on the DRIVE dataset.ConclusionsThe overall performance of DNL-Net outperforms other current state-of-the-art vessel segmentation methods, which indicates that the proposed network is more suitable for blood vessel segmentation, and is of great clinical significance.