Multi-scale Contextual Information Research Articles

Multiscale Feature Filtering Network for Image Recognition System in Unmanned Aerial Vehicle

For unmanned aerial vehicle (UAV), object detection at different scales is an important component for the visual recognition. Recent advances in convolutional neural networks (CNNs) have demonstrated that attention mechanism remarkably enhances multiscale representation of CNNs. However, most existing multiscale feature representation methods simply employ several attention blocks in the attention mechanism to adaptively recalibrate the feature response, which overlooks the context information at a multiscale level. To solve this problem, a multiscale feature filtering network (MFFNet) is proposed in this paper for image recognition system in the UAV. A novel building block, namely, multiscale feature filtering (MFF) module, is proposed for ResNet-like backbones and it allows feature-selective learning for multiscale context information across multiparallel branches. These branches employ multiple atrous convolutions at different scales, respectively, and further adaptively generate channel-wise feature responses by emphasizing channel-wise dependencies. Experimental results on CIFAR100 and Tiny ImageNet datasets reflect that the MFFNet achieves very competitive results in comparison with previous baseline models. Further ablation experiments verify that the MFFNet can achieve consistent performance gains in image classification and object detection tasks.

Dense Encoder-Decoder–Based Architecture for Skin Lesion Segmentation

Melanoma is one kind of dangerous cancer that has been increasing rapidly in the world. Initial diagnosis is essential to survival, but often the disease is diagnosed in the fatal stage. The rapid growth of skin cancers raises a huge demand for accurate automatic skin lesion segmentation. While deep learning techniques, i.e., convolutional neural network (CNN), have been widely used for precise segmentation, the existing densely connected network (DenseNet) and residual network (ResNet)–based encoder-decoder architectures used non-biomedical features for skin lesion tasks. The complexity of tuned parameters, small information in the pre-trained features, and the lack of multi-scale information degrade the performance of skin lesion segmentation. To address these issues, we present encoder-decoder–based CNN for skin lesion segmentation, based on the widely used UNet architecture. We exploit the benefit of combining DenseNet and ResNet to improve the performance of skin lesion segmentation. In the encoder path, atrous spatial pyramid pooling (ASPP) is used to generate multi-scale features from different dilation rates. We used dense skip connection to combine the feature maps of both encoder and decoder paths. We evaluate our approach on ISIC 2018 dataset and achieve competitive performance as compared to other state-of-the-art approaches. Compared to the previous UNet approaches, our method gains a high Jaccard index, Dice, accuracy, and sensitivity. We think that this progress is mainly due to the combined architecture of DenseNet, ResNet, ASPP, and dense skip connection that preserve the contextual information in the encoder-decoder paths. We utilized the combined benefits of both recent DenseNet and ResNet architectures. We used ASPP to exploit multi-scale contextual information by adopting multiple dilation rates. We also implemented dense skip connections for better recovery of fine-grained information of target objects. In the future, we believe that this approach will be helpful to other medical image segmentation tasks.

DFFNet: An IoT-perceptive dual feature fusion network for general real-time semantic segmentation

Semantic segmentation is a valuable yet challenging research in the Internet of Things (IoT), especially for some real-time and resource-constrained applications. Recently we witness a strong tendency of fusing multi-level features or multi-scale context information to achieve promising segmentation performance. However, we find that existing literature has at least one of the following issues: 1) relying on more resource-consuming feature extraction operations, e.g., standard convolution with large kernels, for multiple information fusion; 2) seldom considering that how to narrow the semantic gap between multi-layer features, leading to sub-optimal performance. To tackle these issues, we propose a novel IoT-perceptive Dual Feature Fusion Network (DFFNet) for semantic segmentation, which aims to leverage multi-level features and multi-scale context information in an efficient yet effective manner. Specifically, the multi-level feature fusion module (MFFM), which enhances the semantic consistency between multi-level features by two attention refinement blocks, is proposed to exploit multi-layer features for jointly learning spatial and semantic information with small overheads. Moreover, a multi-scale component termed as lightweight semantic pyramid module (LSPM) is presented to improve the efficiency of context encoding by depthwise factorized convolution operations. Extensive experimental results on benchmarks have demonstrated that DFFNet achieves better performance than existing advanced methods.

Corn-Plant Counting Using Scare-Aware Feature and Channel Interdependence

Corn-plant counting is an important process for predicting corn yield and analyzing corn-plant phenotypes. In this letter, an effective corn-plant counting method is proposed, which is based on utilizing the scale-aware (SA) contextual feature and channel interdependence (CI). Given the Visual Geometry Group (VGG) Network features, the SA features are extracted by spatial pyramid pooling to derive multiscale context information. In order to utilize the channel interdependent information, the VGG features are integrated via a channel attention module. Moreover, an encoder–decoder structure is constructed to fuse the SA features and the CI-based features. Considering the sparsity of a corn plant, a hybrid loss function is adopted to train the network, by considering a density map loss function and an absolute count loss function. Experimental results demonstrate the effectiveness of the proposed method for corn-plant counting.

Automatic airway tree segmentation based on multi-scale context information.

Airway tree segmentation plays a pivotal role in chest computed tomography (CT) analysis tasks such as lesion localization, surgical planning, and intra-operative guidance. The remaining challenge is to identify small bronchi correctly, which facilitates further segmentation of the pulmonary anatomies. A three-dimensional (3D) multi-scale feature aggregation network (MFA-Net) is proposed against the scale difference of substructures in airway tree segmentation. In this model, the multi-scale feature aggregation (MFA) block is used to capture the multi-scale context information, which improves the sensitivity of the small bronchi segmentation and addresses the local discontinuities. Meanwhile, the concept of airway tree partition is introduced to evaluate the segmentation performance at a more granular level. Experiments were conducted on a dataset of 250 CT scans, which were annotated by experienced clinical radiologists. Through the airway partition, we evaluated the segmentation results of the small bronchi compared with the state-of-the-art methods. Experiments show that MFA-Net achieves the best performance in the Dice similarity coefficient (DSC) in the intra-lobar airway and improves the true positive rate (TPR) by 7.59% on average. Besides, in the entire airway, the proposed method achieves the best results in DSC and TPR scores of 86.18% and 79.31%, respectively, with the consequence of higher false positives. The MFA-Net is competitive with the state-of-the-art methods. The experiment results indicate that the MFA block improves the performance of the network by utilizing multi-scale context information. More accurate segmentation results will be more helpful in further clinical analysis.

Aerial-BiSeNet: A real-time semantic segmentation network for high resolution aerial imagery

The aircraft system has recently gained its reputation as a reliable and efficient tool for sensing and parsing aerial scenes. However, accurate and fast semantic segmentation of high-resolution aerial images for remote sensing applications is still facing three challenges: the requirements for limited processing resources and low-latency operations based on aerial platforms, the balance between high accuracy and real-time efficiency for model performance, and the confusing objects with large intra-class variations and small inter-class differences in high-resolution aerial images. To address these issues, a lightweight and dual-path deep convolutional architecture, namely Aerial Bilateral Segmentation Network (Aerial-BiSeNet), is proposed to perform real-time segmentation on high-resolution aerial images with favorable accuracy. Specifically, inspired by the receptive field concept in human visual systems, Receptive Field Module (RFM) is proposed to encode rich multi-scale contextual information. Based on channel attention mechanism, two novel modules, called Feature Attention Module (FAM) and Channel Attention based Feature Fusion Module (CAFFM) respectively, are proposed to refine and combine features effectively to boost the model performance. Aerial-BiSeNet is evaluated on the Potsdam and Vaihingen datasets, where leading performance is reported compared with other state-of-the-art models, in terms of both accuracy and efficiency.

Towards accurate coronary artery calcium segmentation with multi‐scale attention mechanism

Coronary artery calcium is a strong and independent marker of atherosclerosis and cardiovascular disease. Typically, the accurate segmentation of computed tomography images of the chest is an important prerequisite and basis for coronary artery calcium identification and analysis. However, this is very challenging in practice because the boundaries of coronary artery calcium, the small lesions with large shape variation, are very blurry, resulting in poor performance in existing studies. To tackle this challenge, we present a novel Attention-based Multi-Scale Network called AMSN, which can process information through both the main and boundary branches in parallel. Key to our AMSN is a new non-local multi-scale context encoder module, which is mainly composed of the multi-scale attention mechanism and local global long short-term memory module. By aggregating the multi-scale context information, i.e. high-resolution low-level and low-resolution high-level features, the model's feature representative capability and deployment ability are improved effectively. Besides, we introduce a new boundary preserving loss, which can consider the boundary information of all coronary artery calcium together and establish links for the segmentation of different coronary artery calcium simultaneously. Extensive experiments demonstrate our AMSN enables reliable accurate coronary artery calcium segmentation for assisted cardiovascular disease diagnosis clinically.

Lightweight Oriented Object Detection Using Multiscale Context and Enhanced Channel Attention in Remote Sensing Images

Object detection is a focal point in remote sensing applications. Remote sensing images typically contain a large number of small objects and a wide range of orientations across objects. This results in great challenges to small object detection approaches based on remote sensing images. Methods directly employ channel relations with equal weights to construct information features leads to inadequate feature representation in complex image small object detection tasks. Multiscale detection methods improve the speed and accuracy of detection, while small objects themselves contain limited information, and the features are easily lost following down-sampling. During the detection, the feature images are independent across scales, resulting in a discontinuity at the detection scale. In this article, we propose the multiscale context and enhanced channel attention (MSCCA) model. MSCCA employs PeleeNet as the backbone network. In particular, the feature image channel attention is enhanced and the multiscale context information is fused with multiscale detection methods to improve the characterization ability of the convolutional neural network. The proposed MSCCA method is evaluated on two real datasets. Results show that for 512 × 512 input images, MSCCA was able to achieve 80.4% and 94.4% mAP on the DOTA and NWPU VHR-10, respectively. Meanwhile, the model size of MSCCA is 21% smaller than that of its predecessor. MSCCA can be considered as a practical lightweight oriented object detection model in remote sensing images.

ClouDet: A Dilated Separable CNN-Based Cloud Detection Framework for Remote Sensing Imagery

Cloud detection is one of the essential procedures in optical remote sensing image processing because clouds are widely distributed in remote sensing images and cause a lot of challenges, such as climate research and object detection. In this article, a lightweight deep-learning-based framework is proposed to detect cloud in remote sensing imagery. First, a multiple features fusion strategy is designed to extract learnable manual features and convolution features from visible and near-infrared bands. Then, a lightweight fully convolutional neural network (ClouDet) with a microarchitecture named dilated separable convolutional module is used to extract the multiscale contextual information and gradually recovers segmentation results with the same size as input image, which is more effective for large-scale cloud detection with larger receptive field, less parameters, and lower compute complexity. Third, context pooling is designed to amend the possible misjudgments. Visual and quantitative comparison experiments are conducted on several public cloud detection datasets, which indicates that our proposed method can accurately detect clouds under different conditions, which is more effective and accurate than the compared state-of-the-art methods.

MH UNet: A Multi-Scale Hierarchical Based Architecture for Medical Image Segmentation

UNet and its variations achieve state-of-the-art performances in medical image segmentation. In end-to-end learning, the training with high-resolution medical images achieves higher accuracy for medical image segmentation. However, the network depth, a massive number of parameters, and low receptive fields are issues in developing deep architecture. Moreover, the lack of multi-scale contextual information degrades the segmentation performance due to the different sizes and shapes of regions of interest. The extraction and aggregation of multi-scale features play an important role in improving medical image segmentation performance. This paper introduces the MH UNet, a multi-scale hierarchical-based architecture for medical image segmentation that addresses the challenges of heterogeneous organ segmentation. To reduce the training parameters and increase efficient gradient flow, we implement densely connected blocks. Residual-Inception blocks are used to obtain full contextual information. A hierarchical block is introduced between the encoder-decoder for acquiring and merging features to extract multi-scale information in the proposed architecture. We implement and validate our proposed architecture on four challenging MICCAI datasets. Our proposed approach achieves state-of-the-art performance on the BraTS 2018, 2019, and 2020 Magnetic Resonance Imaging (MRI) validation datasets. Our approach is 14.05 times lighter than the best method of BraTS 2018. In the meantime, our proposed approach has 2.2 times fewer training parameters than the top 3D approach on the ISLES 2018 Computed Tomographic Perfusion (CTP) testing dataset. MH UNet is available at https://github.com/parvezamu/MHUnet .

Multi-Vision Network for Accurate and Real-Time Small Object Detection in Optical Remote Sensing Images

Accurate and real-time detection of airplanes, cars, and ships in remote sensing images is an important but challenging task that plays an important role in both military and civilian life. The most challenging issues posed by this task are the intensive and tiny size of objects and the complexity of application scenarios. In this letter, we propose a multi-vision small object detector that can rapidly and accurately detect airplanes, cars, and ships in remote sensing images. We make the following three contributions: a multiscale residual block (MRB) is proposed, whereby dilated convolution is employed in a cascade residual block to capture multiscale context information, thus improving the feature representation ability of convolutional neural networks; a multiscale receptive field enhancement module (MRFEM) is proposed that combines features obtained using dilated convolution at different dilation rates to further enhance the multiscale feature representation of the remote sensing targets; and a multi-vision network (MVNet) is presented that uses multiple low-level feature maps with multi-branch convolution to detect small objects. Experimental results show that the proposed method can achieve a significant mean average precision (mAP) of 94.70% in remote sensing images and can run at 24 FPS on a single NVIDIA 1080Ti GPU.

Context attention network for occluded pedestrian detection

Pedestrian detection in occluded scenes has always been a thorny problem in computer vision. In this case, due to the large difference in scale of occluded pedestrians and low visibility, it usually brings great challenges to detection. To solve this problem, this paper proposes a model structure for pedestrian occlusion detection, which improves the pedestrian detection method based on anchor-free. Specifically, we introduce a structure for extracting multi-scale context information to learn a better feature representation, and a channel attention module on each decoder layer to provide global context as a guidance of low-level features to select category localization details. Experimental results show that this method achieves 41.93% of MR-2 on the occlusion subset of Caltech pedestrian dataset, which is better than other contrast detectors.

Deep convolutional neural networks for multiplanar lung nodule detection: Improvement in small nodule identification.

PurposeEarly detection of lung cancer is of importance since it can increase patients’ chances of survival. To detect nodules accurately during screening, radiologists would commonly take the axial, coronal, and sagittal planes into account, rather than solely the axial plane in clinical evaluation. Inspired by clinical work, the paper aims to develop an accurate deep learning framework for nodule detection by a combination of multiple planes.MethodsThe nodule detection system is designed in two stages, multiplanar nodule candidate detection, multiscale false positive (FP) reduction. At the first stage, a deeply supervised encoder–decoder network is trained by axial, coronal, and sagittal slices for the candidate detection task. All possible nodule candidates from the three different planes are merged. To further refine results, a three‐dimensional multiscale dense convolutional neural network that extracts multiscale contextual information is applied to remove non‐nodules. In the public LIDC‐IDRI dataset, 888 computed tomography scans with 1186 nodules accepted by at least three of four radiologists are selected to train and evaluate our proposed system via a tenfold cross‐validation scheme. The free‐response receiver operating characteristic curve is used for performance assessment.ResultsThe proposed system achieves a sensitivity of 94.2% with 1.0 FP/scan and a sensitivity of 96.0% with 2.0 FPs/scan. Although it is difficult to detect small nodules (i.e., <6 mm), our designed CAD system reaches a sensitivity of 93.4% (95.0%) of these small nodules at an overall FP rate of 1.0 (2.0) FPs/scan. At the nodule candidate detection stage, results show that the system with a multiplanar method is capable to detect more nodules compared to using a single plane.ConclusionOur approach achieves good performance not only for small nodules but also for large lesions on this dataset. This demonstrates the effectiveness of our developed CAD system for lung nodule detection.

A CenterNet++ model for ship detection in SAR images

Ship detection in SAR images is a challenging task due to two difficulties. (1) Because of the long observation distance, ships in SAR images are small with low resolution, leading to high false negative. (2) Because of the complex onshore background, ships are easily confused with other objects with similar appearance. To solve these problems, we propose an effective and stable single-stage detector called CenterNet++. Our model mainly consists of three modules, i.e., feature refinement module, feature pyramids fusion module, and head enhancement module. Firstly, to address small objects detection problem, we design a feature refinement module for extracting multi-scale contextual information. Secondly, feature pyramids fusion module is developed for generating more powerful semantic information. Finally, to alleviate the impact of complex background, head enhancement module is proposed for a balance between foreground and background. To prove the effectiveness and robustness of the proposed method, we make extensive experiments on three popular SAR image datasets, i.e., AIR-SARShip, SSDD, SAR-Ship. The experimental results show that our CenterNet++ reaches state-of-the-art performance on all datasets. In addition, compared with the baseline CenterNet, the proposed method achieves a remarkable accuracy improvement with negligible increase in time cost.

Building Extraction from High Spatial Resolution Remote Sensing Images via Multiscale-Aware and Segmentation-Prior Conditional Random Fields

Building extraction is a binary classification task that separates the building area from the background in remote sensing images. The conditional random field (CRF) is directly modelled by the maximum posterior probability, which can make full use of the spatial neighbourhood information of both labelled and observed images. CRF is widely used in building footprint extraction. However, edge oversmoothing still exists when CRF is directly used to extract buildings from high spatial resolution (HSR) remote sensing images. Based on a computer vision multi-scale semantic segmentation network (D-LinkNet), a novel building extraction framework is proposed, named multiscale-aware and segmentation-prior conditional random fields (MSCRF). To solve the problem of losing building details in the downsampling process, D-LinkNet connecting the encoder and decoder is correspondingly used to generate the unary potential. By integrating multi-scale building features in the central module, D-LinkNet can integrate multiscale contextual information without loss of resolution. For the pairwise potential, the segmentation prior is fused to alleviate the influence of spectral diversity between the building and the background area. Moreover, the local class label cost term is introduced. The clear boundaries of the buildings are obtained by using the larger-scale context information. The experimental results demonstrate that the proposed MSCRF framework is superior to the state-of-the-art methods and performs well for building extraction of complex scenes.

A novel green apple segmentation algorithm based on ensemble U-Net under complex orchard environment

Fruit segmentation is a critical step for fruit recognition, and key to the efficiency of a robotic vision system for harvesting and accurate orchard yield estimation. Due to the unstructured characteristics of orchard environment, the segmentation performance of traditional segmentation algorithms is insufficient for handling green target fruits in a complex environment. Deep learning algorithms bring a fresh perspective to target segmentation. This study proposes an ensemble U-Net segmentation model suitable for small sample datasets. Edge structures are designed by integrating residual blocks and gated convolutions to obtain the boundary semantic information of the target image; atrous convolutions are applied to resolve the contradiction between the resolution of the feature map and the receiving field, retain more multiscale context information and achieve target fruit segmentation. An atrous spatial pyramid pooling (ASPP) structure is applied to merge the edge features and the high-level features of U-Net. The experimental results show that the proposed method effectively improves the segmentation accuracy of the target fruit and the generalization ability of the model. The proposed method extends the application scope of the harvesting robot and orchard yield measurements, thereby providing a theoretical reference for other fruit and vegetable target fruit segmentation efforts.

Multi-Scale Dense Attention Network for Stereo Matching

To improve the accuracy of stereo matching, the multi-scale dense attention network (MDA-Net) is proposed. The network introduces two novel modules in the feature extraction stage to achieve better exploit of context information: dual-path upsampling (DU) block and attention-guided context-aware pyramid feature extraction (ACPFE) block. The DU block is introduced to fuse different scale feature maps. It introduces sub-pixel convolution to compensate for the loss of information caused by the traditional interpolation upsampling method. The ACPFE block is proposed to extract multi-scale context information. Pyramid atrous convolution is adopted to exploit multi-scale features and the channel-attention is used to fuse the multi-scale features. The proposed network has been evaluated on several benchmark datasets. The three-pixel-error evaluated over all ground truth pixels is 2.10% on KITTI 2015 dataset. The experiment results prove that MDA-Net achieves state-of-the-art accuracy on KITTI 2012 and 2015 datasets.

LCP-Net: A local context-perception deep neural network for medical image segmentation

Automatic image segmentation is an indispensable step in medical image analysis, and it plays an important role in computer-assisted radiotherapy, disease diagnosis and treatment effect evaluation. The difficulty of medical image segmentation is greatly enhanced by the blurry nature of medical image, the complex shape of objects and the existence of noise. In recent years, segmentation methods based on deep learning, especially convolutional neural network, have made great progress in improving the accuracy of medical image segmentation. However, these methods also have poor ability to distinguish similar objects in different environments, because of insufficient use of the local context information of images during the process of feature extraction. To address this problem, this paper proposes a deep neural network (LCP-Net) that can perceive multi-scale context information of images. LCP-Net improves the utilization of context information of feature encoders by using Parallel Dilated Convolution (PDC) and Local Context Embedding (LCE), which are beneficial to get feature map rich in environmental information. In addition, to improve the segmentation accuracy of the model for small objects and alleviate the swing issue during training, we propose a novel improved cross-entropy loss (DDCLoss), which can adaptively adjust the weight of loss according to the certainty and deviation distance of the predicted pixel value and enable the model to focus on optimizing the sample points with low certainty and tend to be mislabeled. Experimental results on three different medical datasets demonstrate that compared with the state-of-the-art medical image segmentation models, our proposed LCP-Net can achieve better segmentation performance.

Empower Chinese event detection with improved atrous convolution neural networks

Event Detection (ED) is an essential and challenging task in Information Extraction (IE). Recent advances in neural networks make it possible to build reliable models without complicated feature engineering. Neural network-based models commonly regard event detection as a char-wise or word-wise labeling task, which suffers from the problems of long-distance information miss-capturing, discontinuous labeling errors, etc., between characters/words and event triggers, especially in languages without natural word delimiters such as Chinese. In this paper, we propose a novel multi-layer Residual and Gated-Based Atrous Convolution Neural Network (RG-ACNN) framework, which attempts to alleviate above-mentioned problems. Specifically, the ACNN is introduced in our model to expand the receptive field to obtain multi-scale context information to capture dependencies between long-distance information. While gated and residual mechanisms are both imported to ACNN to improve our networks’ capability of the information filtering and aggregation. Besides, RG-ACNN performs event detection in a char-wise paradigm, where a novel “head-tail dual-pointer” labeled strategy is used to overcome the incomplete continuous labeling problem. Experiments on the ACE2005-CN and several standard benchmark datasets show that RG-ACNN significantly outperforms state-of-the-art (SOTA) methods.

Atrous convolutional feature network for weakly supervised semantic segmentation

Weakly supervised semantic segmentation has been attracting increasing attention as it can alleviate the need for expensive pixel-level annotations through the use of image-level labels. Relevant methods mainly rely on the implicit object localization ability of convolutional neural networks (CNNs). However, generated object attention maps remain mostly small and incomplete. In this paper, we propose an Atrous Convolutional Feature Network (ACFN) to generate dense object attention maps. This is achieved by enhancing the context representation of image classification CNNs. More specifically, cascaded atrous convolutions are used in the middle layers to retain sufficient spatial details, and pyramidal atrous convolutions are used in the last convolutional layers to provide multi-scale context information for the extraction of object attention maps. Moreover, we propose an attentive fusion strategy to adaptively fuse the multi-scale features. Our method shows improvements over existing methods on both the PASCAL VOC 2012 and MS COCO datasets, achieving state-of-the-art performance.