Multi-scale Module Research Articles

Collaborative Content-Dependent Modeling: A Return to the Roots of Salient Object Detection.

Salient object detection (SOD) aims to identify the most visually distinctive object(s) from each given image. Most recent progresses focus on either adding elaborative connections among different convolution blocks or introducing boundary-aware supervision to help achieve better segmentation, which is actually moving away from the essence of SOD, i.e., distinctiveness/salience. This paper goes back to the roots of SOD and investigates the principles of how to identify distinctive object(s) in a more effective and efficient way. Intuitively, the salience of one object should largely depend on its global context within the input image. Based on this, we devise a clean yet effective architecture for SOD, named Collaborative Content-Dependent Networks (CCD-Net). In detail, we propose a collaborative content-dependent head whose parameters are conditioned on the input image's global context information. Within the content-dependent head, a hand-crafted multi-scale (HMS) module and a self-induced (SI) module are carefully designed to collaboratively generate content-aware convolution kernels for prediction. Benefited from the content-dependent head, CCD-Net is capable of leveraging global context to detect distinctive object(s) while keeping a simple encoder-decoder design. Extensive experimental results demonstrate that our CCD-Net achieves state-of-the-art results on various benchmarks. Our architecture is simple and intuitive compared to previous solutions, resulting in competitive characteristics with respect to model complexity, operating efficiency, and segmentation accuracy.

Multiscale recurrent-guided denoising network for distributed acoustic sensing-vertical seismic profile background noise attenuation

In recent years, distributed optical fiber acoustic sensing (DAS) has emerged as a novel seismic acquisition technique. Compared with conventional hydrophones and geophones of microelectromechanical systems, DAS has an advantage in terms of acquisition geometry, such as low-cost and high-density observations. However, the collected DAS records always suffer from various types of noise, which poses challenges for subsequent processing. Thus, deep learning-based solutions have addressed the attenuation of seismic background noise. Denoising networks can provide excellent denoising results. Nonetheless, the architectures of these networks are relatively simple, which may result in degeneration when confronted with the complex DAS background noise. To effectively attenuate complex noise, a novel multiscale network called recurrent-guided self-enhanced attention network (RGSA-Net) is developed for complex seismic data processing. Specifically, the backbone of RGSA-Net uses a conventional feedforward neural network to preliminarily extract the potential features. Meanwhile, multiscale modules, inspired by the recurrent-guided scheme, are used to enhance the contour information. On this basis, a self-enhanced attention module is applied to fuse the multiscale features and further reinforce the effective information, thereby improving the noise attenuation capability. Synthetic and field experiments demonstrate that RGSA-Net indicates promise in complex noise attenuation and weak upgoing signal recovery.

DeepFake detection with multi-scale convolution and vision transformer

With the help of some modern image generative techniques, it is possible to generate or manipulate image or video contents without introducing any obvious visual artifacts. If these manipulated images/videos are abused, it probably has a huge negative impact on society and individuals. Thus, deepfake detection has attracted considerable attention in recent years. Although the existing methods can achieve good detection performance on high-quality datasets, they are still far from satisfactory for low-quality dataset and cross-dataset evaluation. In this paper, therefore, we propose a new CNN-based method via multi-scale convolution and vision transformer for deepfake detection. In the proposed model, we design a multi-scale module with dilation convolution and depthwise separable convolution to capture more face details and tampering artifacts at different scales. Unlike the traditional classification module, furthermore, we employ a vision transformer to further learn the global information of face features for classification. Extensive experiments demonstrate that in most cases the proposed method achieves better detection results on both high-quality and low-quality datasets compared with related modern methods, and the cross-dataset generalization capabilities of the proposed method are good. In addition, many ablation experiments are provided to verify the rationality of the proposed network.

A High-Quality Instance-Segmentation Network for Floating-Algae Detection Using RGB Images

Floating-algae detection plays an important role in marine-pollution monitoring. The surveillance cameras on ships and shores provide a powerful way of monitoring floating macroalgae. However, the previous methods cannot effectively solve the challenging problem of detecting Ulva prolifera and Sargassum, due to many factors, such as strong interference with the marine environment and the drastic change of scale. Recently, the instance-segmentation methods based on deep learning have been successfully applied to many image-recognition tasks. In this paper, a novel instance-segmentation network named AlgaeFiner is proposed for high-quality floating-algae detection using RGB images from surveillance cameras. For improving the robustness of the model in complex ocean scenes, the CA-ResNet is firstly proposed by integrating coordinate attention into the ResNet structure to model both the channel- and position-dependencies. Meanwhile, the Ms-BiFPN is proposed by embedding the multi-scale module into the architecture of BiFPN to strengthen the ability of feature fusion at different levels. To improve the quality of floating-algae segmentation, the Mask Transfiner network is introduced into the AlgaeFiner to obtain the high-quality segmentation results. Experimental results demonstrate that the AlgaeFiner can achieve better performance on floating-algae segmentation than other state-of-the-art instance-segmentation methods, and has high application-value in the field of floating-macroalgae monitoring.

Land cover classification of remote sensing images based on improved DeeplabV3+ network

For land survey and land planning, the precise segmentation of land cover based on remote sensing images is extremely important. Based on the high-resolution satellite remote sensing images and the corresponding land cover GIS vector data in Hangzhou, for the categorization and change detection of urban land cover, a unique convolutional neural network is suggested. The data source is from high-resolution satellite remote sensing images, the semantic segmentation model DeeplabV3+ is improved. The original backbone network Xception is replaced by MobileNetV2 with smaller parameters, and add the channel attention module after the multi-scale module. In addition, to deal with the effect of category imbalance on model results, the focus loss function is introduced for network training. The pixel accuracy and average intersection ratio of 77.63% and 67.46% respectively, which are better than the commonly used semantic segmentation models DeeplabV3 +, UNet, and PSPNet models, and the model size is only 20.1MB. The experimental results show that the model is effective in redecing the number of model paramenters and improving the segmentation effect.

Multiscale image dehazing network based on spatially weighted sparse representation

Due to the influence of hazy weather, the images captured often suffer from severe degradation, which will reduce the effectiveness of the devices for computer-related applications. To eliminate the degradation of hazy images, a multiscale image dehazing network based on spatially weighted sparse representation is proposed. The model is mainly divided into two modules: multiscale module (MSM) and detail enhancement module (DEM). The MSM can effectively obtain more feature information in more hazy images. Specifically, the degradation of hazy image is considered in terms of haze distribution and image details. A spatially weighted nonlocal sparse attention module was developed to focus on channel-level features adaptively and to pay more attention to the recovery of dense haze areas in response to the problem of uneven distribution of haze in images. In addition, an independent feature fusion module is proposed to capture better the contextual information, which can also effectively address the traditional fusion, leading to redundant feature extraction and hindering the reconstructive performance of the network. Finally, a DEM is employed by a combination of pyramid pooling and resblock group, which guides the adaptive fusion of latent features, where different details and plausible appearances are integrated. Experimental results on public datasets show that this method is effective and superior in the metrics of structure similarity (SSIM), peak-signal-to-noise ratio, multiscale-SSIM, and root-mean-square error.

Multi-Kernel Temporal and Spatial Convolution for EEG-Based Emotion Classification.

Deep learning using an end-to-end convolutional neural network (ConvNet) has been applied to several electroencephalography (EEG)-based brain-computer interface tasks to extract feature maps and classify the target output. However, the EEG analysis remains challenging since it requires consideration of various architectural design components that influence the representational ability of extracted features. This study proposes an EEG-based emotion classification model called the multi-kernel temporal and spatial convolution network (MultiT-S ConvNet). The multi-scale kernel is used in the model to learn various time resolutions, and separable convolutions are applied to find related spatial patterns. In addition, we enhanced both the temporal and spatial filters with a lightweight gating mechanism. To validate the performance and classification accuracy of MultiT-S ConvNet, we conduct subject-dependent and subject-independent experiments on EEG-based emotion datasets: DEAP and SEED. Compared with existing methods, MultiT-S ConvNet outperforms with higher accuracy results and a few trainable parameters. Moreover, the proposed multi-scale module in temporal filtering enables extracting a wide range of EEG representations, covering short- to long-wavelength components. This module could be further implemented in any model of EEG-based convolution networks, and its ability potentially improves the model's learning capacity.

A generative adversarial network with multi-scale convolution and dilated convolution res-network for OCT retinal image despeckling

Optical coherence tomography (OCT) has been widely adopted for imaging in various areas, yet it is largely affected by speckle noise generated from the coherent multiple-scattered photons. To alleviate the influences of speckle noise, a generative adversarial network with multi-scale convolution and dilated convolution res-network (MDR-GAN) is proposed in this study. Specifically, a cascade multi-scale module (CMSM) consisting of three convolution and dilated convolution res-network (CD-Rn) blocks is proposed to raise network learning capacity, while a new residual learning method is devised to link the input and output feature maps for feature reconstructions. Among them, CMSM has the characteristics of capturing multi-scale local features of images. Residual learning effectively avoids the degradation problem of the network. Extensive experiments with four retinal OCT datasets are conducted and results are compared with those of the state-of-the-art deep learning networks to verify the effectiveness of the proposed MDR-GAN. Results demonstrate that the denoising effect of MDR-GAN is better than those of the other denoising methods. The peak single-to-noise ratio (PSNR) of MDR-GAN is improved by 2 dB as compared that of Pix2pix, while its equivalent number of looks (ENL) is improved by at least 233.9% as compared with the-state-of-the-art existing methods. Our MDR-GAN code can be download at https://github.com/Austin-Lms/MDR-GAN.

Spatiotemporal image fusion using multiscale attention-aware two-stream convolutional neural networks

Spatiotemporal fusion (STF) is considered as a promising way to produce remote sensing images at fine scales in both space and time by blending two types of satellite images. The learning-based STF approaches with deep convolutional neural networks can provide the unified framework to address both gradual and abrupt changes. This paper intends to develop an enhanced learning-based STF using multiscale attention-aware two-stream convolutional neural networks (MACNN). With a coarse image at the prediction date and two pairs of coarse and fine images at other dates as inputs, it employs a multiscale module to characterize different sizes of objects and a spatial and channel attention module to emphasize important information in feature learning. Two experiments on real Landsat and MODIS images are conducted to demonstrate the effectiveness of the proposed MACNN and it outperforms four existing STF methods in both visual and quantitative.

An End-to-end Medical Image Segmentation Model based on Multi-scale Feature Extraction

With the development of technology, medical image examination methods are steadily diversified, including ultrasound, pathology, endoscopy, computed tomography, computed radiography, magnetic resonance imaging, radionuclide imaging, positron emission computed tomography, angiography, etc. More than 70% of clinical diagnoses depends on medical imaging. Owning to the fact that medical image data grows faster, the problem of lack of radiodiagnosis doctors are more serious. Although the accuracy of doctors’ diagnosis is high, the accuracy of judgment is questionable due to energy constraints, emotional fluctuations and other factors. To assist accurate and effective diagnosis and reduce the probability of misdiagnosis, in this paper, a novel end-to-end model with an encoder network and a decoder network (MS-UNet) is proposed. The encoder network extracts feature information from the input image, and the decoder network restores the encoded content to the input size. Among them, a multi-scale module (MSM) is built to help extract more accurate information that represents the feature maps of objective segmentation by fusing multiple different convolution information at different stages. The experimental results on two well-known data sets show that the proposed model outperforms other state-of-the-art algorithms, and the introduction of MSM can significantly improve performance.

Semantic-refined spatial pyramid network for crowd counting

• We design a novel semantic-refined spatial pyramid network (SSPNet). • The SSPNet can benefit the fusing of multi-scale information. • The SSPNet can enhance the representation ability of deep semantic features. • The multi-scale feature will be adaptively integrated by our method. • The texture features are obtained to enrich the multi-scale features. In this paper, we propose a novel encoder-decoder model called Semantic-refined Spatial Pyramid Network (SSPNet) for generating high-quality density maps, which aims to build a scale-aware counting network to estimate the number of crowds accurately. The SSPNet consists of the front-end based on VGG-16, spatial pyramid multi-scale module (SPMM), and semantic enhancement module (SEM). First, a series of convolutional neural layers are utilized as the front-end to get deeper features without the extra computational cost. Moreover, the SPMM, which has a spatial pyramid structure with multiple receptive fields, is employed to capture multi-scale features. Furthermore, the SEM is designed to refine the features captured by SPMM, which uses deep semantic information to better integrate multi-scale features. Finally, the shallow texture information is adopted to compensate for the detail of the feature map to enhance the quality of the density map. Extensive experiments and comparisons on three challenge datasets, including ShanghaiTech Part_A & Part_B, UCF_CC_50, and UCF-QNRF, illustrate the superiority of our method.

EEG Identity Authentication in Multi-Domain Features: A Multi-Scale 3D-CNN Approach.

Electroencephalogram (EEG) authentication has become a research hotspot in the field of information security due to its advantages of living, internal, and anti-stress. However, the performance of identity authentication system is limited by the inherent attributes of EEG, such as low SNR, low stability, and strong randomness. Researchers generally believe that the in-depth fusion of features can improve the performance of identity authentication and have explored among various feature domains. This experiment invited 70 subjects to participate in the EEG identity authentication task, and the experimental materials were visual stimuli of the self and non-self-names. This paper proposes an innovative EEG authentication framework, including efficient three-dimensional representation of EEG signals, multi-scale convolution structure, and the combination of multiple authentication strategies. In this work, individual EEG signals are converted into spatial–temporal–frequency domain three-dimensional forms to provide multi-angle mixed feature representation. Then, the individual identity features are extracted by the various convolution kernel of multi-scale vision, and the strategy of combining multiple convolution kernels is explored. The results show that the small-size and long-shape convolution kernel is suitable for ERP tasks, which can obtain better convergence and accuracy. The experimental results show that the classification performance of the proposed framework is excellent, and the multi-scale convolution method is effective to extract high-quality identity characteristics across feature domains. The results show that the branch number matches the EEG component number can obtain the excellent cost performance. In addition, this paper explores the network training performance for multi-scale module combination strategy and provides reference for deep network construction strategy of EEG signal processing.

A Multiscale CNN Framework for Wireless Technique Classification in Internet of Things

Wireless technique classification (WTC) is of crucial importance in Internet of Things for realizing efficient spectrum sharing and interference management. However, the existing deep-learning-based methods have low classification accuracy, especially at low signal-to-noise ratio levels. In this article, a multiscale convolutional neural network framework is proposed for WTC. A multiscale module is exploited to capture the higher abstraction features. Simulation results demonstrate that our proposed scheme can achieve a better classification performance and a higher convergence speed compared to the state-of-the-art schemes.

Dually attentive multiscale networks for health state recognition of rotating machinery

Recent advances in convolutional neural networks (CNN) have boosted the research on reliability monitoring of rotating machinery. In actual industry production, the mechanical equipment often operates under variable speed and strong noise conditions, so the discriminative fault-related features of the collected vibration signals are easily buried by interference information. Thus, it poses a huge challenge for the existing CNN models to achieve favorable diagnostic results. To address this issue, we put forward a dually attentive multiscale network (DAMN) for mechanical fault diagnosis. To begin with, a new hierarchical structure is built to make full use of the features from intermediate convolutional layers. Then, to explore abundant discriminative information from mechanical signals, a dually attentive multiscale module (DAMM) is introduced to guide the CNN model to extract multiscale and multilevel features. Further, a feature reinforcement module (FRM) is designed to expand receptive field and filter out unrelated interference information. Finally, embarking on the above improvements, an end-to-end CNN model named DAMN is built for intelligent fault diagnosis. Experimental results show that DAMN outperforms seven state-of-the-art methods for health recognition of rotating machinery.

A Multi-Scale Brain Tumor Segmentation Method based on U-Net Network

Brain tumor is a serious threat to human health. Because the size and shape of brain tumors can be uneven, irregular and unstructured. Automatic segmentation of tumors from magnetic resonance imaging (MRI) is a challenging task. Brain tumor segmentation using computer-aided diagnosis has important clinical significance for the prognosis and treatment of patients. The traditional U-Net network can not take full advantage of context information, which is easy to cause the loss of effective information of image. Therefore, we propose a multi-scale segmentation method for brain tumors based on U-Net network, in which a multi-scale module for feature extraction is added between down-sampling and up-sampling. The brain tumor public dataset BraTS2020 is used for testing. Dice coefficient was used as the evaluation index, and our experimental method made the average dice of the whole tumor area, core tumor area and enhanced tumor area reach 87.57%, 86.20% and 84.24%, respectively. The experimental results show that the proposed method has comparable precision with typical brain tumor segmentation methods.

A Surface Target Recognition Algorithm Based on Coordinate Attention and Double-Layer Cascade

As a branch of target recognition, surface target recognition plays an irreplaceable role in both military and civilian applications. However, the large target size variation, low image resolution, and high real-time requirements pose challenges to existing algorithms. To address the issues, we take YOLOv5 as a backbone and adopt coordinate attention and a double-layer cascade structure to enhance both the recognition performance and speed. Specifically, coordinate attention is introduced to guide the corresponding network to focus on discriminative features by capturing channel and location information. Meanwhile, the double-layer cascade structure is designed for finely extracting and aggregating semantic features and spatial features at different scales. We test the model on the COCO dataset, the VOC dataset, and self-built surface target dataset. Experimental results show that proposed coordinate attention module and multiscale module improve the recognition effect of multiscale surface targets and meet the requirement of real time.

Context-Aware Multi-Scale Aggregation Network for Congested Crowd Counting.

In this paper, we propose a context-aware multi-scale aggregation network named CMSNet for dense crowd counting, which effectively uses contextual information and multi-scale information to conduct crowd density estimation. To achieve this, a context-aware multi-scale aggregation module (CMSM) is designed. Specifically, CMSM consists of a multi-scale aggregation module (MSAM) and a context-aware module (CAM). The MSAM is used to obtain multi-scale crowd features. The CAM is used to enhance the extracted multi-scale crowd feature with more context information to efficiently recognize crowds. We conduct extensive experiments on three challenging datasets, i.e., ShanghaiTech, UCF_CC_50, and UCF-QNRF, and the results showed that our model yielded compelling performance against the other state-of-the-art methods, which demonstrate the effectiveness of our method for congested crowd counting.

Non-linear perceptual multi-scale network for single image super-resolution

Recently, deep convolutional neural networks (CNNs) have been widely explored in single image super-resolution (SISR) and achieved remarkable progress. However, most of the existing CNN-based SISR networks with a single-stream structure fail to make full use of the multi-scale features of low-resolution (LR) image. While those multi-scale SR models often integrate the information with different receptive fields by means of linear fusion, which leads to the redundant feature extraction and hinders the reconstruction performance of the network. To address both issues, in this paper, we propose a non-linear perceptual multi-scale network (NLPMSNet) to fuse the multi-scale image information in a non-linear manner. Specifically, a novel non-linear perceptual multi-scale module (NLPMSM) is developed to learn more discriminative multi-scale feature correlation by using high-order channel attention mechanism, so as to adaptively extract image features at different scales. Besides, we present a multi-cascade residual nested group (MC-RNG) structure, which uses a global multi-cascade mechanism to organize multiple local residual nested groups (LRNG) to capture sufficient non-local hierarchical context information for reconstructing high-frequency details. LRNG uses a local residual nesting mechanism to stack NLPMSMs, which aims to form a more effective residual learning mechanism and obtain more representative local features. Experimental results show that, compared with the state-of-the-art SISR methods, the proposed NLPMSNet performs well in both quantitative metrics and visual quality with a small number of parameters.

A multi-scale semantic attention representation for multi-label image recognition with graph networks

Multi-label image recognition is a basic and challenging task in computer vision and multimedia fields. Graph Convolutional Networks (GCNs) are often used to learn the multi-label semantic features and multi-label dependency. Although the label semantic features in GCNs can learn the global image visual representation well, they are rarely used on the local image regions. Therefore, we try to use GCNs to learn global and local features at the same time, and make a balance between them. In this paper, we give a multi-scale semantic attention model MS-SGA-GCN including three main modules (i.e., MS, SGA and GCN) for multi-label image recognition. The Multi-Scale module (MS) utilizes feature maps of different sizes to obtain global features and have strong generalization capabilities. Semantic Guide Attention module (SGA) applies the label embeddings learned by GCNs to guide the generation of the cross-modality class-specific attention maps, which can discover the locations of semantically related regions for each label. Experiments show that our model on two datasets MS-COCO and PASCAL VOC2007 separately achieves the classification accuracy by 83.4% and 94.2%, which has a competitive advantage over other mainstream models.

HyMM: hybrid method for disease-gene prediction by integrating multiscale module structure

Identifying disease-related genes is an important issue in computational biology. Module structure widely exists in biomolecule networks, and complex diseases are usually thought to be caused by perturbations of local neighborhoods in the networks, which can provide useful insights for the study of disease-related genes. However, the mining and effective utilization of the module structure is still challenging in such issues as a disease gene prediction. We propose a hybrid disease-gene prediction method integrating multiscale module structure (HyMM), which can utilize multiscale information from local to global structure to more effectively predict disease-related genes. HyMM extracts module partitions from local to global scales by multiscale modularity optimization with exponential sampling, and estimates the disease relatedness of genes in partitions by the abundance of disease-related genes within modules. Then, a probabilistic model for integration of gene rankings is designed in order to integrate multiple predictions derived from multiscale module partitions and network propagation, and a parameter estimation strategy based on functional information is proposed to further enhance HyMM's predictive power. By a series of experiments, we reveal the importance of module partitions at different scales, and verify the stable and good performance of HyMM compared with eight other state-of-the-arts and its further performance improvement derived from the parameter estimation. The results confirm that HyMM is an effective framework for integrating multiscale module structure to enhance the ability to predict disease-related genes, which may provide useful insights for the study of the multiscale module structure and its application in such issues as a disease-gene prediction.