Multi-scale Convolution Module Research Articles

MCGRAM: Linking Multi-Scale CNN With a Graph-Based Recurrent Attention Model for Subject-Independent ERP Detection

Electroencephalogram (EEG)-based event-related potential (ERP) detection is expected to play an important role in many applications. However, limited by the low signal-to-noise ratio and large inter-subject variability of EEG signals, it is still challenging to design a high-precision ERP detection method in subject-independent scenarios. In this brief, a novel graph-based multi-scale convolutional recurrent attention model (MCGRAM) is proposed to extract the underlying invariant features of EEG signals for subject-independent ERP detection. Specifically, considering the frequency factors of ERP signals, the multi-scale convolution module is designed for learning the frequency representations. Then, the graph convolution module is developed by constructing the links between each EEG node and designing trainable weights of the links to extract the discriminative spatial features. Besides, two-layer LSTM combined with a self-attention model is utilized to extract temporal representations of different EEG slices. Compared to the existing state-of-the-art methods, MCGRAM achieves the best performance in terms of AUC and F1-Score on a rapid serial visual presentation benchmark dataset. Lastly, the ablation studies demonstrate the superiority of each component in MCGRAM.

Tool Wear Prediction Based on a Multi-Scale Convolutional Neural Network with Attention Fusion

Compared with traditional machine learning algorithms, the convolutional neural network (CNN) has an excellent automatic feature learning ability and can complete the nonlinear representation from original data input to output by itself. However, the CNN does not sufficiently mine the tool wear information contained in the multi-sensor data due to disregard of the differences in the contribution of different features when extracting features. In this paper, a tool wear prediction method based on a multi-scale convolutional neural network with attention fusion is proposed, which fuses the tool wear degradation information collected by different types of sensors. In the multi-scale convolution module, convolution kernels with different sizes are used to extract the degradation information of different scales in the wear information, and then the attention fusion module is constructed to fuse the multi-scale feature information. Finally, the mapping between tool wear and multi-sensor data is realized through the feature information obtained by residual connection and full connection layer. By comparing the multi-scale convolutional neural network with different attention mechanisms, the experiments demonstrated the effectiveness and superiority of the proposed method.

Prediction of protein secondary structure based on an improved channel attention and multiscale convolution module.

Prediction of the protein secondary structure is a key issue in protein science. Protein secondary structure prediction (PSSP) aims to construct a function that can map the amino acid sequence into the secondary structure so that the protein secondary structure can be obtained according to the amino acid sequence. Driven by deep learning, the prediction accuracy of the protein secondary structure has been greatly improved in recent years. To explore a new technique of PSSP, this study introduces the concept of an adversarial game into the prediction of the secondary structure, and a conditional generative adversarial network (GAN)-based prediction model is proposed. We introduce a new multiscale convolution module and an improved channel attention (ICA) module into the generator to generate the secondary structure, and then a discriminator is designed to conflict with the generator to learn the complicated features of proteins. Then, we propose a PSSP method based on the proposed multiscale convolution module and ICA module. The experimental results indicate that the conditional GAN-based protein secondary structure prediction (CGAN-PSSP) model is workable and worthy of further study because of the strong feature-learning ability of adversarial learning.

Remaining Useful Life Prediction Model for Rolling Bearings Based on MFPE-MACNN.

Aiming to resolve the problem of redundant information concerning rolling bearing degradation characteristics and to tackle the difficulty faced by convolutional deep learning models in learning feature information in complex time series, a prediction model for remaining useful life based on multiscale fusion permutation entropy (MFPE) and a multiscale convolutional attention neural network (MACNN) is proposed. The original signal of the rolling bearing was extracted and decomposed by resonance sparse decomposition to obtain the high-resonance and low-resonance components. The multiscale permutation entropy of the low-resonance component was calculated. Moreover, the locally linear-embedding algorithm was used for dimensionality reduction to remove redundant information. The multiscale convolution module was constructed to learn the feature information at different time scales. The attention module was used to fuse the feature information and input it into the remaining useful life prediction module for evaluation. The appropriate network structure and parameter configuration were determined, and a multiscale convolutional attention neural network was designed to determine the remaining useful life prediction model. The results show that the method demonstrates effectiveness and superiority in degrading the feature information representation and improving the remaining useful life prediction accuracy compared with other models.

Multi-Scale Convolution-Capsule Network for Crop Insect Pest Recognition

Accurate crop insect pest identification in fields is useful to control pests and beneficial to agricultural yield and quality. However, it is a difficult and challenging problem due to the crop insect pests being small with various sizes, postures, shapes, and disorganized backgrounds. Multi-scale convolution-capsule network (MSCCN) is constructed for crop insect pest identification. It consists of a multi-scale convolution module, capsule network (CapsNet) module, and SoftMax classification module. Multi-scale convolution is used to extract the multi-scale discriminative features, CapsNet is employed to encode the hierarchical structure of the size-variant insect pests in the crop images, and Softmax is adopted for insect pest identification. MSCCN combines the advantages of convolutional neural network (CNN), CapsNet, and multi-scale CNN, and can learn multi-scale robust features from pest images of different shapes and sizes for pest recognition and identify various morphed pests. Experimental results on the crop pest image dataset show that this method has a good recognition rate of 91.4%.

GPR image denoising with NSST-UNET and an improved BM3D

To suppress random noise while preserving effective information in the edge areas of ground penetrating radar (GPR) images, this paper proposes a novel denoising method by making use of a deep neural network called NSST-UNET and an improved BM3D. At first, NSST-UNET is designed with a non-subsampled shearlet transform (NSST) coding layer and a skip connection based on a multi-scale convolution module and applied to identify the edge and smooth areas of noisy GPR images. Then, the denoising is accomplished with the improved BM3D in two steps. In the first step, a larger search range for similar blocks and a soft threshold are used to denoise the edge and smooth areas, respectively. In the second step, the Wiener filter optimized by mean square error and the Wiener filter optimized by structural similarity are utilized to denoise the smooth and unsmooth areas, respectively. Finally, the excellent denoising performance of the proposed method is verified by qualitative and quantitative analysis with simulation and field exploration data.

Identifying Oceanic Eddy With an Edge-Enhanced Multiscale Convolutional Network

Identifying oceanic eddy from remotely sensed sea surface height (SSH) data is challenging, mainly because of its large-size variations. This article proposes an automatic identification model upon convolutional neural networks for dealing with this issue. The proposed network is comprised of two branches: an eddy identification branch and an edge extraction branch. Both of them adopt encoder–decoder frameworks, and the encoder is shared with each other. The eddy identification branch simultaneously uses multiscale convolution modules in the encoder and skip-layer connections between the encoder and the decoder to learn multiscale features, thus effectively identifying eddies with different sizes. Differently, the edge extraction branch is designed to learn the edge information of eddies, which is not fully captured by the eddy identification branch. To sufficiently evaluate the identification performance of our proposed model, several experiments are conducted on a public eddy identification dataset named SCSE-Eddy, and the results indicate that the proposed model is capable of achieving higher performance than those of the existing identification models.

Simple Multiscale UNet for Change Detection With Heterogeneous Remote Sensing Images

Change detection with heterogeneous remote sensing images (HRSIs) is attractive for observing the Earth’s surface when homogeneous images are unavailable. However, HRSIs cannot be compared directly because the imaging mechanisms for bitemporal HRSIs are different, and detecting change with HRSIs is challenging. In this letter, a simple yet effective deep learning approach based on the classical UNet is proposed. First, a pair of image patches are concatenated together to learn a shared abstract feature in both image patch domains. Then, a multiscale convolution module is embedded in a UNet backbone to cover the various sizes and shapes of ground targets in an image scene. Finally, a combined loss function, which incorporates the focal and dice losses with an adjustable parameter, was incorporated to alleviate the effect of the imbalanced quantity of positive and negative samples in the training progress. By comparisons with five state-of-the-art methods in three pairs of real HRSIs, the experimental results achieved by our proposed approach have the best overall accuracy (OA), average accuracy (AA), recall (RC), and F-Score that are more than 95%, 79%, 60%, and 61%, respectively. The quantitative results and visual performance indicated the feasibility and superiority of the proposed approach for detecting land cover change with HRSIs.

Angle steel tower bolt defect detection based on YOLO-V3

The bolts in the angle steel tower are seriously affected by corrosion and loss. This paper proposes a novel detection system based on YOLO-V3 to avoid the danger of traditional manual detection method for the bolt fault detection of the angle steel tower. A multi-scale convolution module is used to replace the ordinary convolution of original YOLO-V3 so as to obtain the spatial characteristics information of different scales in the image, and enhance the detection accuracy. The experimental results show that mAP of the proposed YOLO-SKIP network is 0.91. Our YOLO-SKIP model has achieved the best detection performance on the defective angle steel tower bolt data.

MLFFNet: Multilevel Feature Fusion Network for Object Detection in Sonar Images

Sonar image object detection is essential in underwater rescue and resource exploration. Although many convolution neural network (CNN)-based object detection algorithms have achieved great success in natural images. However, for underwater sonar images, problems, such as seabed reverberation noise interference, low proportion of foreground object region pixels, and poor imaging resolution, present considerable challenges to achieving accurate underwater object detection. To address these problems, we propose a novel sonar image object detector called the multilevel feature fusion network (MLFFNet). The detector consists of multiscale convolution module (MS-Conv), multilevel feature extraction module (ML-FEM), multilevel feature fusion module (ML-FFM), neighborhood channel attention mechanism (N-CAM), multiscale feature pyramid module (MS-FPN), and feature association module (FA). First, we use the MS-Conv to extract different scale feature information in the object region. Second, the ML-FEM and ML-FFM are used to obtain the local detail and global context features. Third, the N-CAM and MS-FPN are used to obtain the foreground objects’ semantic feature and position feature, and suppress the background region noise interference. Finally, we use the FA module to enhance the category and feature correlation of different objects. Extensive experiments are conducted on the real scene sonar image dataset. The experimental results demonstrate that MLFFNet performs better than other state-of-the-art object detection methods. Code and dataset are publicly at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/darkseid-arch/SonarMLFFNet</uri> .

A Deep-Shallow Fusion Network With Multidetail Extractor and Spectral Attention for Hyperspectral Pansharpening

Hyperspectral (HS) pansharpening aims at fusing a low-resolution HS (LRHS) image with a high-resolution panchromatic (PAN) image to obtain a hyperspectral image with both higher spectral and spatial resolutions. However, existing HS pansharpening algorithms are mainly based on multispectral (MS) pansharpening approaches, which cannot perfectly restore much spectral information and more high-frequency spatial details in the continuous spectral bands and much broader spectral range, leading to spectral distortion and spatial blur. In this paper, we develop a new hyperspectral pansharpening network architecture (called Hyper-DSNet) to fully preserve latent spatial details and spectral fidelity via a deep-shallow fusion structure with multi-detail extractor and spectral attention. Specifically, the proposed architecture mainly consists of three parts. First, to solve the problem of spatial ambiguity and exploit the potential information, five types of high-pass filter templates are used to fully extract the spatial details of the PAN image, constructing a so-called multi-detail extractor (MDE). Then, after passing a multi-scale convolution module, a deep-shallow fusion (DSF) structure, which reduces parameters by decreasing the number of output channels as the network goes deeper, is utilized sequentially. In final, a spectral attention (SA) module is conducted to preserve the spectrum for a wealth of spectral information of HS images. Visual and quantitative experiments on three commonly used simulated datasets and one full-resolution dataset demonstrate the effectiveness and robustness of the proposed Hyper-DSNet against the recent state-of-the-art hyperspectral pansharpening techniques. Ablation studies and discussions further verify our contributions, <i>e</i>.<i>g</i>., better spectral preservation and spatial detail recovery. The source code is available at <uri>https://github.com/liangjiandeng/Hyper-DSNet</uri>, and the related datasets can be found from <uri>https://github.com/liangjiandeng/HyperPanCollection</uri>.

Research on the Identification and Evaluation of Aerobics Movements Based on Deep Learning

A deep learning approach is used in this study to provide insight into aerobics movement recognition, and the model is used for aerobics movement recognition. The model complexity is significantly reduced, while the multi-scale features of the target at the fine-grained level are extracted, significantly improving the characterization of the target, by embedding lightweight multi-scale convolution modules in 3D convolutional residual networks to increase the local perceptual field range in each layer of the network. Finally, using the channel attention mechanism, the key features are extracted from the multi-scale features. To create a dual-speed frame rate detection model, the fast-slow combination idea is fused into a 3D convolutional network. To obtain spatial semantic information and motion information in the video, the model uses different frame rates, and the two-channel information is fused with features using lateral concatenation. Following the acquisition of all features, the features are fed into a temporal detection network to identify temporal actions and to design a behavior recognition system for the network model to demonstrate the network model's applicability. The average scores of students in the experimental group were significantly higher than those in the control group in seven areas: set accuracy, movement amplitude, movement strength, body coordination, coordination of movement and music, movement expression, and aesthetics; the average scores of movement proficiency and body control in the experimental group were also significantly higher than those in the control group, but the differences were not significant. The differences between the eight indicators in the experimental group were not significant when compared to those in the preexperimental group, indicating that intensive rhythm training for students improves secondary school students' comprehension, proficiency, and presentation of aerobics sets.

Multi-scale deep coupling convolutional neural network with heterogeneous sensor data for intelligent fault diagnosis

With the innovation and development of detection technology, various types of sensors are installed to monitor the operating status of equipment in modern industry. Compared with the same type of sensors for monitoring, heterogeneous sensors can collect more comprehensive complementary fault information. Due to the large distribution differences and serious noise pollution of heterogeneous sensor data collected in industrial sites, this brings certain challenges to the development of heterogeneous data fusion strategies. In view of the large distribution difference in the feature spatial of heterogeneous data and the difficulty of effective fusion of fault information, this paper presents a multi-scale deep coupling convolutional neural network (MDCN), which is used to map the heterogeneous fault information from different feature spaces to the common spaces for full fusion. Specifically, a multi-scale convolution module (MSC) with multiple filters of different sizes is adopted to extract multi-scale fault features of heterogeneous sensor data. Then, the maximum mean discrepancy (MMD) is applied to measure the distance between different spatial features in the coupling layer, and the common failure information in the heterogeneous data is mined by minimizing MMD to fuse effectively in order to identify the failure state of the device. The validity of this method is verified by the data collected on a first-level parallel gearbox mixed fault experiment platform.

Incorporation of residual attention modules into two neural networks for low-dose CT denoising.

The low-dose computed tomography (CT) imaging can reduce the damage caused by x-ray radiation to the human body. However, low-dose CT images have a different degree of artifacts than conventional CT images, and their resolution is lower than that of conventional CT images, which can affect disease diagnosis by clinicians. Therefore, methods for noise-level reduction and resolution improvement in low-dose CT images have inevitably become a research hotspot in the field of low-dose CT imaging. In this paper, residual attention modules (RAMs) are incorporated into the residual encoder-decoder convolutional neural network (RED-CNN) and generative adversarial network with Wasserstein distance (WGAN) to learn features that are beneficial to improving the performances of denoising networks, and developed models are denoted as RED-CNN-RAM and WGAN-RAM, respectively. In detail, RAM is composed of a multi-scale convolution module and an attention module built on the residual network architecture, where the attention module consists of a channel attention module and a spatial attention module. The residual network architecture solves the problem of network degradation with increased network depth. The function of the attention module is to learn which features are beneficial to reduce the noise level of low-dose CT images to reduce the loss of detail in the final denoising images, which is also the key point of the proposed algorithms. To develop a robust network for low-dose CT image denoising, multidose-level torso phantom images provided by a cooperating equipment vendor are used to train the network, which can improve the network's adaptability to clinical application. In addition, a clinical dataset is used to test the network's migration capabilities and clinical applicability. The experimental results demonstrate that these proposed networks can effectively remove noise and artifacts from multidose CT scans. Subjective and objective analyses of multiple groups of comparison experiments show that the proposed networks achieve good noise suppression performance while preserving the image texture details. In this study, two deep learning network models are developed using multidose-level CT images acquired from a commercial spiral CT scanner. The two network models can reduce and even remove streaking artifacts, and noise from low-dose CT images confirms the effectiveness of the proposed algorithms.

An improved one-stage pedestrian detection method based on multi-scale attention feature extraction

In recent years, the performance of the convolutional neural network-based pedestrian detection method has improved significantly. However, an imbalance remains between detection accuracy and speed. In this paper, we employ a one-stage object detection framework and propose a pedestrian detection method based on the multi-scale attention mechanism of a convolutional neural network to improve the imbalance between accuracy and speed. First, a multi-scale convolution module is designed to extract corresponding features at different scales. Second, using the attention module, association information between features is mined from space and channel perspectives to strengthen the original features. Then, the enhanced features are passed through a classification and regression module to perform object positioning and bounding box regression. Finally, to learn more pedestrian location information, we improve the loss function to realise better network training. The proposed method achieved considerable results on the challenging CityPersons and Caltech pedestrian detection datasets.

Joint optic disc and cup segmentation based on residual multi-scale fully convolutional neural network

Glaucoma is the leading cause of irreversible blindness, but its early symptoms are not obvious and are easily overlooked, so early screening for glaucoma is particularly important. The cup to disc ratio is an important indicator for clinical glaucoma screening, and accurate segmentation of the optic cup and disc is the key to calculating the cup to disc ratio. In this paper, a full convolutional neural network with residual multi-scale convolution module was proposed for the optic cup and disc segmentation. First, the fundus image was contrast enhanced and polar transformation was introduced. Subsequently, W-Net was used as the backbone network, which replaced the standard convolution unit with the residual multi-scale full convolution module, the input port was added to the image pyramid to construct the multi-scale input, and the side output layer was used as the early classifier to generate the local prediction output. Finally, a new multi-tag loss function was proposed to guide network segmentation. The mean intersection over union of the optic cup and disc segmentation in the REFUGE dataset was 0.904 0 and 0.955 3 respectively, and the overlapping error was 0.178 0 and 0.066 5 respectively. The results show that this method not only realizes the joint segmentation of cup and disc, but also improves the segmentation accuracy effectively, which could be helpful for the promotion of large-scale early glaucoma screening.

Crowd counting method based on the self-attention residual network

Estimating the crowd density in surveillance videos is a hot issue in the field of computer vision and has become the basis of data processing and analysis of public transport services, commercial passenger flow analysis, public security protection and other industries. However, in terms of practical applications, due to the problems of pedestrian occlusion and scale changes, existing methods are inadequate with regard to the acquisition of the human head, which affects the accuracy of counting. To solve this problem, a crowd counting method based on a self-attention residual network is proposed. First, a multiscale convolution module composed of dilated convolution and deformation convolution is used. To avoid losing image resolution, some of the sampling positions are shifted to the occluded crowd by shifting the sampling points, which solves the problem of crowd occlusion. Then, a self-attention residual module is designed to score and classify the feature map, which allows all pixels in the feature map to be classified. The corresponding weight is generated, and the population scale is determined by the weight, which solves the problem of crowd scale changes. The algorithm is applied in ShanghaiTech and the UCF_CC_50 and WorldExpo’10 datasets are tested. The experimental results show that the mean absolute error (MAE) and mean square error (MSE) of this algorithm are significantly reduced compared with those of a comparative algorithm.

Progressive Back-Traced Dehazing Network Based on Multi-Resolution Recurrent Reconstruction

In order to alleviate adverse impacts of haze on high-level vision tasks, image dehazing attracts great attention from computer vision research field in recent years. Most of existing methods are grouped into physical prior based and non-physical data-driven based categories. However, image dehazing is a challenging ill-conditioned and inherently ambiguous problem. Due to random distribution and concentration of haze, color distortion and excessive brightness often happen in physical prior based methods. Defects on high-frequency details’ recovery are not solved well in non-physical data-driven methods. Therefore, to overcome these obstacles, in this paper, we have proposed an effective progressive back-traced dehazing network based on multi-resolution recurrent reconstruction strategies. A kind of irregular multi-scale convolution module is proposed to extract fine-grain local structures. And a kind of multi-resolution residual fusion module is proposed to progressively reconstruct intermediate haze-free images. We have compared our method with several popular state-of-the-art methods on public RESIDE and 2018 NTIRE Dehazing datasets. The experiment results demonstrate that our method could restore satisfactory high-frequency textures and high-fidelity colors. Related source code and parameters will be distributed on Github for further study.