Multi-scale Receptive Fields Research Articles

Multi-scale graph neural network for global stereo matching

Currently, deep learning-based stereo matching is solely based on local convolution networks, which lack enough global information for accurate disparity estimation. Motivated by the excellent global representation of the graph, a novel Multi-scale Graph Neural Network (MGNN) is proposed to essentially improve stereo matching from the global aspect. Firstly, we construct the multi-scale graph structure, where the multi-scale nodes with projected multi-scale image features can be directly linked by the inner-scale and cross-scale edges, instead of solely relying on local convolutions for deep learning-based stereo matching. To enhance the spatial position information at non-Euclidean multi-scale graph space, we further propose a multi-scale position embedding to embed the potential position features of Euclidean space into projected multi-scale image features. Secondly, we propose the multi-scale graph feature inference to extract global context information on multi-scale graph structure. Thus, the features not only be globally inferred on each scale, but also can be interactively inferred across different scales to comprehensively consider global context information with multi-scale receptive fields. Finally, MGNN is deployed into dense stereo matching and experiments demonstrate that our method achieves state-of-the-art performance on Scene Flow, KITTI 2012/2015, and Middlebury Stereo Evaluation v.3/2021.

A lightweight pose estimation network with multi-scale receptive field

A lightweight pose estimation network with multi-scale receptive field

Multi-scale information distillation network for efficient image super-resolution

Efficient image super-resolution (SR), being preferred in the resource-constrained scenarios, aims at not only higher super-resolving accuracy but also lower computational complexity. Taking the perception capability of deep networks into account, efficiently and effectively obtaining the large receptive field is a key principle for this task. Thus, in this paper, we integrate the multi-scale receptive field design with information distillation structure and attention mechanism, and develop a lightweight Multi-Scale Information Distillation (MSID) network. In detail, we design a multi-scale feature distillation (MSFD) block by employing multi-scale convolutions with different kernels into feature distillation connection, which effectively distills information from multiple receptive fields with low computational cost for better feature refinement. Moreover, we construct a scalable large kernel attention (SLKA) block via scaling attentive fields across network layers, that possesses large and scalable receptive field in attention to discriminatively enhance the distilled features. Extensive quantitative and qualitative evaluations on benchmark datasets validate the effectiveness of each proposed component and also demonstrate the superiority of our MSID network over state-of-the-art efficient SR methods. The code is available at https://github.com/YuanfeiHuang/MSID.

DMCT-Net: dual modules convolution transformer network for head and neck tumor segmentation in PET/CT

Objective. Accurate segmentation of head and neck (H&N) tumors is critical in radiotherapy. However, the existing methods lack effective strategies to integrate local and global information, strong semantic information and context information, and spatial and channel features, which are effective clues to improve the accuracy of tumor segmentation. In this paper, we propose a novel method called dual modules convolution transformer network (DMCT-Net) for H&N tumor segmentation in the fluorodeoxyglucose positron emission tomography/computed tomography (FDG-PET/CT) images. Approach. The DMCT-Net consists of the convolution transformer block (CTB), the squeeze and excitation (SE) pool module, and the multi-attention fusion (MAF) module. First, the CTB is designed to capture the remote dependency and local multi-scale receptive field information by using the standard convolution, the dilated convolution, and the transformer operation. Second, to extract feature information from different angles, we construct the SE pool module, which not only extracts strong semantic features and context features simultaneously but also uses the SE normalization to adaptively fuse features and adjust feature distribution. Third, the MAF module is proposed to combine the global context information, channel information, and voxel-wise local spatial information. Besides, we adopt the up-sampling auxiliary paths to supplement the multi-scale information. Main results. The experimental results show that the method has better or more competitive segmentation performance than several advanced methods on three datasets. The best segmentation metric scores are as follows: DSC of 0.781, HD95 of 3.044, precision of 0.798, and sensitivity of 0.857. Comparative experiments based on bimodal and single modal indicate that bimodal input provides more sufficient and effective information for improving tumor segmentation performance. Ablation experiments verify the effectiveness and significance of each module. Significance. We propose a new network for 3D H&N tumor segmentation in FDG-PET/CT images, which achieves high accuracy.

Data-driven time–frequency analysis of nonlinear Lamb waves for characterization of grain size distribution

Nondestructive characterization of grain size distribution is significant for revealing the mechanical properties of metal materials. Traditional linear ultrasonic methods have been applied to estimate mean grain size but are less sensitive to small grain size. Nonlinear ultrasonic technology shows high sensitivity to microstructures, while it unfortunately encounters underdetermined problem in the characterization of microstructures across multiple scales. Combining linear and nonlinear ultrasonic responses would be a quite attractive way to achieve comprehensive observation of microstructural evolution. This work presents a data-driven approach for characterizing grain size distribution using nonlinear Lamb waves. The short-time Fourier transform (STFT) of detected ultrasonic signals were input into a convolutional neural network (CNN) to comprehensively learn the implicit linear and nonlinear dynamics involved with grain size distribution. Multiple convolution kernels slide across the STFT images with multiscale receptive fields to collectively model hierarchical representations and capture local correlation of interesting from time-frequency domain. The trained model achieved 94.7% accuracy in predicting mean grain size, and also achieved 95.4% and 86.3% accuracy in predicting the expectation and standard deviation of the lognormal distribution of grain size, respectively. The visual activation maps demonstrate that local interesting features are successfully captured. In particular, the prediction accuracy was demonstrated to reduce greatly after removing the fundamental frequency and second harmonic from the STFT images, confirming that incorporating linear and nonlinear ultrasonic information is indispensable for accurate characterization of grain size distribution.

Lightweight Device-Free Wireless Sensing Using Information-for-Complexity Strategy

Device-free wireless sensing (DFWS) has drawn lots of attention due to its potential application in the fields of human–computer interaction and smart home. Deep networks-based DFWS technique has achieved excellent sensing performance. However, network complexity limits its deployment on resource-limited sensing devices. A feasible way is to implement a simple network to accomplish the DFWS task. However, the sensing performance will drop dramatically due to its limited learning ability. In this article, to realize lightweight DFWS with acceptable performance, we propose an information-for-complexity strategy to promote the learning ability of the simple network. We leverage knowledge distillation framework to explore external information to augment the extrinsic learning ability, and utilize multiscale receptive fields to explore the internal information to augment the intrinsic learning ability. Extensive experiments on a 77 GHz mmWave testbed show that the performance degradation of the lightweight DFWS system is within 3%, while the complexity decreases remarkably.

All-in-one aerial image enhancement network for forest scenes.

Drone monitoring plays an irreplaceable and significant role in forest firefighting due to its characteristics of wide-range observation and real-time messaging. However, aerial images are often susceptible to different degradation problems before performing high-level visual tasks including but not limited to smoke detection, fire classification, and regional localization. Recently, the majority of image enhancement methods are centered around particular types of degradation, necessitating the memory unit to accommodate different models for distinct scenarios in practical applications. Furthermore, such a paradigm requires wasted computational and storage resources to determine the type of degradation, making it difficult to meet the real-time and lightweight requirements of real-world scenarios. In this paper, we propose an All-in-one Image Enhancement Network (AIENet) that can restore various degraded images in one network. Specifically, we design a new multi-scale receptive field image enhancement block, which can better reconstruct high-resolution details of target regions of different sizes. In particular, this plug-and-play module enables it to be embedded in any learning-based model. And it has better flexibility and generalization in practical applications. This paper takes three challenging image enhancement tasks encountered in drone monitoring as examples, whereby we conduct task-specific and all-in-one image enhancement experiments on a synthetic forest dataset. The results show that the proposed AIENet outperforms the state-of-the-art image enhancement algorithms quantitatively and qualitatively. Furthermore, extra experiments on high-level vision detection also show the promising performance of our method compared with some recent baselines.

A Marine Organism Detection Framework Based on Dataset Augmentation and CNN-ViT Fusion

Underwater vision-based detection plays an important role in marine resources exploration, marine ecological protection and other fields. Due to the restricted carrier movement and the clustering effect of some marine organisms, the size of some marine organisms in the underwater image is very small, and the samples in the dataset are very unbalanced, which aggravate the difficulty of vision detection of marine organisms. To solve these problems, this study proposes a marine organism detection framework with a dataset augmentation strategy and Convolutional Neural Networks (CNN)-Vision Transformer (ViT) fusion model. The proposed framework adopts two data augmentation methods, namely, random expansion of small objects and non-overlapping filling of scarce samples, to significantly improve the data quality of the dataset. At the same time, the framework takes YOLOv5 as the baseline model, introduces ViT, deformable convolution and trident block in the feature extraction network, and extracts richer features of marine organisms through multi-scale receptive fields with the help of the fusion of CNN and ViT. The experimental results show that, compared with various one-stage detection models, the mean average precision (mAP) of the proposed framework can be improved by 27%. At the same time, it gives consideration to both performance and real-time, so as to achieve high-precision real-time detection of the marine organisms on the underwater mobile platform.

A Hyperspectral Image Classification Method Based on the Nonlocal Attention Mechanism of a Multiscale Convolutional Neural Network.

Recently, convolution neural networks have been widely used in hyperspectral image classification and have achieved excellent performance. However, the fixed convolution kernel receptive field often leads to incomplete feature extraction, and the high redundancy of spectral information leads to difficulties in spectral feature extraction. To solve these problems, we propose a nonlocal attention mechanism of a 2D-3D hybrid CNN (2-3D-NL CNN), which includes an inception block and a nonlocal attention module. The inception block uses convolution kernels of different sizes to equip the network with multiscale receptive fields to extract the multiscale spatial features of ground objects. The nonlocal attention module enables the network to obtain a more comprehensive receptive field in the spatial and spectral dimensions while suppressing the information redundancy of the spectral dimension, making the extraction of spectral features easier. Experiments on two hyperspectral datasets, Pavia University and Salians, validate the effectiveness of the inception block and the nonlocal attention module. The results show that our model achieves an overall classification accuracy of 99.81% and 99.42% on the two datasets, respectively, which is higher than the accuracy of the existing model.

Multi-scale receptive fields: Graph attention neural network for hyperspectral image classification

Hyperspectral image (HSI) classification has attracted wide attention in many fields. Applying Graph Neural Network (GNN) to HSI classification is one of the research frontiers, which has improved the HSI classification accuracy greatly. However, GNN-based methods have not been widely applied due to their time-consuming, inefficient information description as well as poor anti-noise robustness. To overcome the deficiencies, a novel multi-scale receptive fields graph attention neural network (MRGAT) is proposed for HSI classification in this paper. In this network, a superpixel segment method is adopted to abstract the original HSI local spatial features. A two-layer one-dimensional convolution neural network (1D CNN) spectral transformer mechanism, is designed to extract the spectral features of superpixels, with which the spectral features can be acquired automatically. Furthermore, graph edges are introduced into Graph Attention Network (GAT) to acquire the local semantic feature of the graph. Moreover, inspired by the transformer network, we design a novel multi-scale receptive field GAT to extract the local-global adjacent node-features and edges-features. Finally, a graph attention network and a softMax function are utilized for multi-receptive feature fusion and pixel-label predicting. On Pavia University, Salinas, and Houston 2013 datasets, the overall accuracies (OAs) of our MRGAT are 71.76%, 82.61%, and 63.82%, respectively. Moreover, the performances with limited labeled samples indicates that the MRGAT contains superior adaptability. Compared with the competitive classifiers, MRGAT achieves high classification efficiency verified by training time comparison experiment.

Multi-Scale Receptive Fields Convolutional Network for Action Recognition

Extracting good action representations from video frames is an intricate challenge due to the presence of moving objects of various sizes across current action recognition datasets. Most of the current action recognition methodologies have paid scant attention to this characteristic and have relied on deep learning models to automatically solve it. In this paper, we introduce a multi-scale receptive fields convolutional network (MSRFNet), which is fashioned after the pseudo-3D residual network architecture to mitigate the impact of scale variation in moving objects. The crux of MSRFNet is the integration of a multi-scale receptive fields block, which incorporates multiple dilated convolution layers that share identical convolutional parameters, but feature different receptive fields. MSRFNet leverages three scales of receptive fields to extract features from moving objects of diverse sizes, striving to produce scale-specific feature maps with a uniform representational power. Through visualization of the attention of MSRFNet, we analyze how the model re-allocates its attention to moving objects after implementing the multi-scale receptive fields approach. Experimental results on the benchmark dataset demonstrate that MSRFNet achieves improvement of 3.2% on UCF101, improvement of 5.8% on HMDB51, and improvement of 7.7% on Kinetics-400 compared with the baseline. Compared with state-of-the-art techniques, MSRFNet gets comparable or superior results, thereby affirming the effectiveness of the proposed approach.

Material-Aware Path Aggregation Network and Shape Decoupled SIoU for X-ray Contraband Detection

X-ray contraband detection plays an important role in the field of public safety. To solve the multi-scale and obscuration problem in X-ray contraband detection, we propose a material-aware path aggregation network to detect and classify contraband in X-ray baggage images. Based on YoloX, our network integrates two new modules: multi-scale smoothed atrous convolution (SCA) and material-aware coordinate attention modules (MCA). In SAC, an improved receptive field-enhanced network structure is proposed by combining smoothed atrous convolution, using separate shared convolution, with a parallel branching structure, which allows for the acquisition of multi-scale receptive fields while reducing grid effects. In the MCA, we incorporate a spatial coordinate separation material perception module with a coordinated attention mechanism. A material perception module can extract the material information features in X and Y dimensions, respectively, which alleviates the obscuring problem by focusing on the distinctive material characteristics. Finally, we design the shape-decoupled SIoU loss function (SD-SIoU) for the shape characteristics of the X-ray contraband. The category decoupling module and the long–short side decoupling module are integrated to the shape loss. It can effectively balance the effect of the long–short side. We evaluate our approach on the public X-ray contraband SIXray and OPIXray datasets, and the results show that our approach is competitive with other X-ray baggage inspection approaches.

Multiscale Deep Feature Learning for Human Activity Recognition Using Wearable Sensors

Deep convolutional neural networks (CNNs) achieve state-of-the-art performance in wearable human activity recognition (HAR), which has become a new research trend in ubiquitous computing scenario. Increasing network depth or width can further improve accuracy. However, in order to obtain the optimal HAR performance on mobile platform, it has to consider a reasonable tradeoff between recognition accuracy and resource consumption. Improving the performance of CNNs without increasing memory and computational burden is more beneficial for HAR. In this article, we first propose a new CNN that uses hierarchical-split (HS) idea for a large variety of HAR tasks, which is able to enhance multiscale feature representation ability via capturing a wider range of receptive fields of human activities within one feature layer. Experiments conducted on benchmarks demonstrate that the proposed HS module is an impressive alternative to baseline models with similar model complexity, and can achieve higher recognition performance (e.g., 97.28%, 93.75%, 99.02%, and 79.02% classification accuracies) on UCI-HAR, PAMAP2, WISDM, and UNIMIB-SHAR. Extensive ablation studies are performed to evaluate the effect of the variations of receptive fields on classification performance. Finally, we demonstrate that multiscale receptive fields can help to learn more discriminative features (achieving 94.10% SOTA accuracy) in weakly labeled HAR dataset.

Accurate unsupervised monocular depth estimation for ill-posed region

Unsupervised monocular depth estimation is challenging in ill-posed regions, such as weak texture scenes, projection occlusion, and redundant error of detail information, etc. In this paper, in order to tackle these problems, an improved unsupervised monocular depth estimation method for the ill-posed region is proposed through cascading training depth estimation network and pose estimation network by loss function. Firstly, for the depth estimation network, a feature extraction network using asymmetric convolution is designed instead of traditional convolution, which strengthens the extraction of the feature information and improves the accuracy of the weak texture scenes. Meanwhile, a feature extraction network integrating multi-scale receptive fields with the structure of different scale convolution and dilated convolution stack is designed to increase the underlying receptive field of the depth estimation network, which strengthens the fusion ability of the network for multi-scale detail information, and improves the integrity of the model output details. Secondly, a pose estimation network using an attention mechanism is presented to strengthen the pose detail information of keyframes and suppress redundant errors of the pose information of non-keyframes. Finally, a loss function with minimum reprojection error is adopted to alleviate the occlusion problem of the projection process between adjacent pixels and enhance the quality of the output depth images of the model. The experiments demonstrate that our method achieves state-of-the-art performance on KITTI monocular datasets.

Low-dose CT denoising with a high-level feature refinement and dynamic convolution network.

Since the potential health risks of the radiation generated by computer tomography (CT), concerns have been expressed on reducing the radiation dose. However, low-dose CT (LDCT) images contain complex noise and artifacts, bringing uncertainty to medical diagnosis. Existing deep learning (DL)-based denoising methods are difficult to fully exploit hierarchical features of different levels, limiting the effect of denoising. Moreover, the standard convolution kernel is parameter sharing and cannot be adjusted dynamically with input change. This paper proposes an LDCT denoising network using high-level feature refinement and multiscale dynamic convolution to mitigate theseproblems. The dual network structure proposed in this paper consists of the feature refinement network (FRN) and the dynamic perception network (DPN). The FDN extracts features of different levels through residual dense connections. The high-level hierarchical information is transmitted to DPN to improve the low-level representations. In DPN, the two networks' features are fused by local channel attention (LCA) to assign weights in different regions and handle CT images' delicate tissues better. Then, the dynamic dilated convolution (DDC) with multibranch and multiscale receptive fields is proposed to enhance the expression and processing ability of the denoising network. The experiments were trained and tested on the dataset "NIH-AAPM-Mayo Clinic Low-Dose CT Grand Challenge," consisting of 10 anonymous patients with normal-dose abdominal CT and LDCT at 25% dose. In addition, external validation was performed on the dataset "Low Dose CT Image and Projection Data," which included 300 chest CT images at 10% dose and 300 head CT images at 25%dose. Proposed method compared with seven mainstream LDCT denoising algorithms. On the Mayo dataset, achieved peak signal-to-noise ratio (PSNR): 46.3526 dB (95% CI: 46.0121-46.6931 dB) and structural similarity (SSIM): 0.9844 (95% CI: 0.9834-0.9854). Compared with LDCT, the average increase was 3.4159 dB and 0.0239, respectively. The results are relatively optimal and statistically significant compared with other methods. In external verification, our algorithm can cope well with ultra-low-dose chest CT images at 10% dose and obtain PSNR: 28.6130 (95% CI: 28.1680-29.0580 dB) and SSIM: 0.7201 (95% CI: 0.7101-0.7301). Compared with LDCT, PSNR/SSIM is increased by 3.6536dB and 0.2132, respectively. In addition, the quality of LDCT can also be improved in head CT denoising. This paper proposes a DL-based LDCT denoising algorithm, which utilizes high-level features and multiscale dynamic convolution to optimize the network's denoising effect. This method can realize speedy denoising and performs well in noise suppression and detail preservation, which can be helpful for the diagnosis ofLDCT.

MSRF-Net: Multiscale Receptive Field Network for Building Detection From Remote Sensing Images

Extracting buildings from remote sensing images plays an important role in urban development planning, disaster assessment and mapping. Convolutional neural network (CNN) has been widely applied to building extraction because of its powerful deep semantic feature extraction ability. However, existing CNN-based building extraction methods are difficult to accurately extract multiscale buildings with accurate edges because of the limitation of feature receptive fields and the loss of spatial detail information. For the above problems, this paper proposes a multiscale receptive field network (MSRF-Net) to accurately extract multiscale buildings from remote sensing images. MSRF-Net includes multiscale receptive field feature encoder (MRFF-Encoder) and multipath decoder. In the MRFF-Encoder, a multiscale attentional down (MSAD) module and asymmetric residual inception (ARI) module are proposed to capture multiscale receptive field features. In the multipath decoder, convolutions with different kernel size and dilation are used in three parallel paths to learn localization-preserved multiscale features with multiscale receptive field. What’s more, the features of different branches and MRFF-Encoder are fused by the proposed feature combination module, which contribute to capture context information of multiscale receptive field while recovering the resolution of feature space. The experimental results show that compared with the latest MAP-Net, MSRF-Net has achieved F1 score growth of 1.14%, 0.42%, 1.11% and IoU score growth of 1.68%, 0.76% and 1.64% respectively on Massachusetts data set, WHU data set and the Typical Cities Building data set.

Rethinking Lightweight Salient Object Detection via Network Depth-Width Tradeoff.

Existing salient object detection methods often adopt deeper and wider networks for better performance, resulting in heavy computational burden and slow inference speed. This inspires us to rethink saliency detection to achieve a favorable balance between efficiency and accuracy. To this end, we design a lightweight framework while maintaining satisfying competitive accuracy. Specifically, we propose a novel trilateral decoder framework by decoupling the U-shape structure into three complementary branches, which are devised to confront the dilution of semantic context, loss of spatial structure and absence of boundary detail, respectively. Along with the fusion of three branches, the coarse segmentation results are gradually refined in structure details and boundary quality. Without adding additional learnable parameters, we further propose Scale-Adaptive Pooling Module to obtain multi-scale receptive field. In particular, on the premise of inheriting this framework, we rethink the relationship among accuracy, parameters and speed via network depth-width tradeoff. With these insightful considerations, we comprehensively design shallower and narrower models to explore the maximum potential of lightweight SOD. Our models are proposed for different application environments: 1) a tiny version CTD-S (1.7M, 125FPS) for resource constrained devices, 2) a fast version CTD-M (12.6M, 158FPS) for speed-demanding scenarios, 3) a standard version CTD-L (26.5M, 84FPS) for high-performance platforms. Extensive experiments validate the superiority of our method, which achieves better efficiency-accuracy balance across five benchmarks.

Infrared and Visible Image Fusion via Multiscale Receptive Field Amplification Fusion Network

Infrared and visible image fusion, which highlights radiometric and detailed texture information and completely and accurately describes objects, is a long-standing and well-studied task in computer vision. Existing convolutional neural network-based approaches that leverage end-to-end networks to fuse infrared and visible images have made significant progress. However, most approaches typically extract the features in the encoder segment and use a coarse fusion strategy. Unlike these algorithms, this study proposes a multiscale receptive field amplification fusion network (MRANet) to effectively extract the local and global features from images. Particularly, we extract long-range information in the encoder segment using a convolutional residual structure as the main backbone and a simplified uniformer as an auxiliary backbone, both of which are ResNet-inspired. Additionally, we propose an effective multiscale fusion strategy based on an attention mechanism to integrate the two modalities. Extensive experiments demonstrate that MRANet performs efficiently on image fusion datasets.

Multi-Scale Receptive Field Feature Fusion Algorithm based on MobileNet

Multi-Scale Receptive Field Feature Fusion Algorithm based on MobileNet

AMS-PAN: Breast ultrasound image segmentation model combining attention mechanism and multi-scale features

Breast ultrasound medical images are characterized by poor imaging quality and irregular target edges. During the diagnosis process, it is difficult for physicians to segment tumors manually, and the segmentation accuracy required for diagnosis is high, so there is an urgent need for an automated method to improve the segmentation accuracy as a technical tool to assist diagnosis. This study designed an improved Pyramid Attention Network combining Attention mechanism and Multi-Scale features (AMS-PAN) for breast ultrasound image segmentation. On the encoding side, the model adopts the depthwise separable convolution strategy to achieve a multi-scale receptive field with cumulative small-size convolution, which performs multi-dimensional feature extraction and forms a feature pyramid. The model uses Global Attention Upsample (GAU) feature fusion on the decoding side. In order to further process the fused feature information, the proposed method uses a Spatial and Channel Attention (SCA) module to shift the model’s segmentation focus to the edge texture information. The good segmentation performance of our method is verified through experiments on BUSI and OASBUD. All the designed parts have contributed to the segmentation performance in practical applications. Compared with the traditional non-deep learning methods and the current mainstream deep learning methods, the improvement of the model in Dice and IoU metrics is pronounced. AMS-PAN has high computational efficiency, and its good performance has been proven to play a role in ultrasound detection tasks of breast tumors for physicians to specific auxiliary diagnostic roles to guide the subsequent diagnosis and treatment services for patients.