Multi-scale Contextual Information Research Articles

Iterative Convolutional Encoder-Decoder Network with Multi-Scale Context Learning for Liver Segmentation

ABSTRACT Rapid and accurate extraction of liver tissue from abdominal computed tomography (CT) and magnetic resonance (MR) images has critical importance for diagnosis and treatment of hepatic diseases. Due to adjacent organs with similar intensities and anatomical variations between different subjects, the performance of segmentation approaches based on deep learning still has room for improvement. In this study, a novel convolutional encoder-decoder network incorporating multi-scale context information is proposed. The probabilistic map from previous classifier is iteratively fed into the encoder layers, which fuses high-level shape context with low-level appearance features in a multi-scale manner. The dense connectivity is adopted to aggregate feature maps of varying scales from the encoder and decoder. We evaluated the proposed method with 2D and 3D application on abdominal CT and MR images of three public datasets. The proposed method generated liver segmentation with significantly higher accuracy (p <0.05), in comparison to several competing methods. These promising results suggest that the novel model could offer high potential for clinical workflow.

MAGF-Net: A multiscale attention-guided fusion network for retinal vessel segmentation

Retinal fundus images contain plenty of morphological information, so it is particularly important to realize precise segmentation of the retinal vessels for clinical diagnosis. With the rapid development of deep convolutional neural networks (DCNNs), to replace earlier manual labeling methods and reduce the labor cost, DCNN-based automatic segmentation methods have been greatly developed. U-Net and its variant models have obtained superior performance, but segmentation tasks are still challenging for the following reasons: First, features from encoders and decoders are not sufficiently fused to retain more effective information. Second, the limited receptive field will also affect contextual information extraction. In addition, although the continuous pooling operations can speed up the segmentation network training efficiency, they also lose detailed information during the downsampling process. To address the above issues and precisely segment the vessel structures from fundus images, a multiscale attention-guided fusion network, called MAGF-Net, is presented for automatic retinal vessel segmentation. To capture multiscale contextual features, a multiscale attention (MSA) block is proposed to construct the backbone network. Furthermore, a feature enhancement (FE) block is also proposed and embedded in the bottleneck layer to acquire global multiscale contextual information. To take full advantage of the channel information from deep layers and the spatial information from shallow layers, an attention-guided fusion (AGF) block is designed to fuse features from different network layers. Moreover, a hybrid feature pooling (HFP) block is employed to preserve more information during the downsampling operation. To evaluate the segmentation performance of the proposed MAGF-Net, extensive segmentation experiments are conducted on three public datasets: the CHASE_DB1 set, the DRIVE set and the STARE set. The experimental results show that the proposed MAGF-Net can obtain remarkable segmentation performance compared with other advanced methods. In particular, the ability of the proposed MAGF-Net to segment thin blood vessels is significantly improved.

Deep hybrid model for single image dehazing and detail refinement

Deep learning technologies have been applied in Single Image Dehazing (SID) tasks successfully. However, most SID algorithms seldom consider to refine image details during dehazing. Therefore, there exist some detail-loss regions in dehazed results. To solve this issue, we design a deep hybrid network to improve dehazing performance and remedy the loss of details. Different from existing algorithms that usually ignore detail refinement and adopt a unified framework to remove haze, we propose to treat dehazing and detail refinement as two separate tasks, so that each task could be solved via different ways. Particularly, we design two sub-networks with a multi-term loss function. First, for removing haze effectively, we introduce the Squeeze-and-Excitation (SE) to design a haze residual attention sub-network, which is used to reconstruct the dehazed image. Second, as for remedying details, we take the previous dehazed image as the input to a detail refinement sub-network, where the image details can be enhanced via multi-scale contextual information aggregation. Through the joint training of two sub-network, the haze can be removed clearly and the image details can be preserved well. Moreover, the detail refinement sub-network can be detached into other existing dehazing methods to improve their model performances. Extensive experiments also verify the superiority of our proposed network against recently proposed state-of-the-arts.

MSCFNet: A Lightweight Network With Multi-Scale Context Fusion for Real-Time Semantic Segmentation

In recent years, how to strike a good trade-off between accuracy, inference speed, and model size has become the core issue for real-time semantic segmentation applications, which plays a vital role in real-world scenarios such as autonomous driving systems and drones. In this study, we devise a novel lightweight network using a multi-scale context fusion (MSCFNet) scheme, which explores an asymmetric encoder-decoder architecture to alleviate these problems. More specifically, the encoder adopts some developed efficient asymmetric residual (EAR) modules, which are composed of factorization depth-wise convolution and dilation convolution. Meanwhile, instead of complicated computation, simple deconvolution is applied in the decoder to further reduce the amount of parameters while still maintaining the high segmentation accuracy. Also, MSCFNet has branches with efficient attention modules from different stages of the network to well capture multi-scale contextual information. Then we combine them before the final classification to enhance the expression of the features and improve the segmentation efficiency. Comprehensive experiments on challenging datasets have demonstrated that the proposed MSCFNet, which contains only 1.15M parameters, achieves 71.9% Mean IoU on the Cityscapes testing dataset and can run at over 50 FPS on a single Titan XP GPU configuration.

An improved algorithm for small object detection based on YOLO v4 and multi-scale contextual information

In real life, object detection is widely applied and plays a significant part in the field of computer vision. However, when detecting small objects, the advanced You Only Look Once v4 (YOLO v4) model often misses or incorrectly detects them due to the limited resolution and unclear features of the small objects, which reduces the detection accuracy. A small object detection algorithm based on YOLO v4 and Multi-scale Contextual information and Soft-CIOU loss function, called MCS-YOLO v4, is proposed in this paper. MCS-YOLO v4 adds a scale detection to the existing three scales to obtain rich location information. To enhance the ability of network to locate and classify the object, MCS-YOLO v4 introduces an expanded field-of-perception block. This block obtains the object contextual features and integrates them with the convolutional features to obtain more robust and discriminative features. To reduce the influence of insignificant information in images on small object, the attention module is introduced in the neck part of YOLO v4. To further improve the detection accuracy of small objects, the Soft-CIOU loss function is proposed. The aspect ratio weight factor is introduced into the weight function of the CIOU (Complete-IOU) loss function, while the Euclidean distance is subjected to the open-root operation, which improves the contribution of small objects to the loss function and enhances the learning ability of the network for small objects. The experimental results on the publicly available small object datasets verify that the proposed model has better detection effect than other detection models in detecting small objects.

MICOS: Mixed supervised contrastive learning for multivariate time series classification

Multivariate time series (MTS) classification is an emerging field with increasing demand. Existing representation learning methods for MTS classification are generally based on self-supervised learning. This results in their inability to maximize the use of labels. In addition, the inherent complexity of MTS makes it difficult to learn latent features. To this end, we introduce a new Mixed Supervised Contrastive Loss (MSCL) for MTS representation learning. To effectively leverage labels, the MSCL is calculated by mixing self-supervised, intra-class and inter-class supervised contrastive learning approaches. Then, based on MSCL, we further propose a novel MIxed supervised COntrastive learning framework for MTS classification (MICOS). It uses the spatial and temporal channels to extract the complicated spatio-temporal features of MTS. Additionally, the MSCL is applied at the timestamp level to capture the multiscale contextual information. Experiments were carried out by performing supervised and self-supervised classification tasks on 30 public datasets from the UEA MTS archives. The results show the reliability and efficiency of MICOS compared with 13 competitive baselines.

Single image deraining using multi-scales context information and attention network

The existing deraining methods based on convolutional neural networks (CNNs) have made great success, but some remaining rain streaks can degrade images drastically. In this work, we proposed an end-to-end multi-scale context information and attention network, called MSCIANet. The proposed network consists of multi-scale feature extraction (MSFE) and multi-receptive fields feature extraction (MRFFE). Firstly, the MSFE can pick up features of rain streaks in different scales and propagate deep features of the two layers across stages by skip connections. Secondly, the MRFFE can refine details of the background by attention mechanism and the depthwise separable convolution of different receptive fields with different scales. Finally, the fusion of these outputs of two subnetworks can reconstruct the clean background image. Extensive experimental results have shown that the proposed network achieves a good effect on the deraining task on synthetic and real-world datasets. The demo can be available at https://github.com/CoderLi365/MSCIANet.

ThelR547v1-An Asymmetric Dilated Convolutional Neural Network for Real-time Semantic Segmentation of Horticultural Crops.

Robust and automated image segmentation in high-throughput image-based plant phenotyping has received considerable attention in the last decade. The possibility of this approach has not been well studied due to the time-consuming manual segmentation and lack of appropriate datasets. Segmenting images of greenhouse and open-field grown crops from the background is a challenging task linked to various factors such as complex background (presence of humans, equipment, devices, and machinery for crop management practices), environmental conditions (humidity, cloudy/sunny, fog, rain), occlusion, low-contrast and variability in crops and pose over time. This paper presents a new ubiquitous deep learning architecture ThelR547v1 (Thermal RGB 547 layers version 1) that segmented each pixel as crop or crop canopy from the background (non-crop) in real time by abstracting multi-scale contextual information with reduced memory cost. By evaluating over 37,328 augmented images (aug1: thermal RGB and RGB), our method achieves mean IoU of 0.94 and 0.87 for leaves and background and mean Bf scores of 0.93 and 0.86, respectively. ThelR547v1 has a training accuracy of 96.27%, a training loss of 0.09, a validation accuracy of 96.15%, and a validation loss of 0.10. Qualitative analysis further shows that despite the low resolution of training data, ThelR547v1 successfully distinguishes leaf/canopy pixels from complex and noisy background pixels, enabling it to be used for real-time semantic segmentation of horticultural crops.

BASeg: Boundary aware semantic segmentation for autonomous driving

Semantic segmentation is a critical component for street understanding task in autonomous driving field. Existing various methods either focus on constructing the object’s inner consistency by aggregating global or multi-scale context information, or simply combine semantic features with boundary features to refine object details. Despite impressive, most of them neglect the long-range dependences between the inner objects and boundaries. To this end, we present a Boundary Aware Network (BASeg) for semantic segmentation by exploiting boundary information as a significant cue to guide context aggregation. Specifically, a Boundary Refined Module (BRM) is proposed in the BASeg to refine coarse low-level boundary features from a Canny detector by high-level multi-scale semantic features from the backbone, and based on which, the Context Aggregation Module (CAM) is further proposed to capture long-range dependences between the boundary regions and the object inner pixels, achieving mutual gains and enhancing the intra-class consistency. Moreover, our method can be plugged into other CNN backbones for higher performance with a minor computation budget, and obtains 45.72%, 81.2%, and 77.3% of mIoU on the datasets ADE20K, Cityscapes, and CamVid, respectively. Compared with some state-of-the-art ResNet101-based segmentation methods, extensive experiments demonstrate the effectiveness of our method. Our code is available at https://github.com/Lature-Yang/BASeg.

A Multi-Scale Contextual Information Enhancement Network for Crack Segmentation

In recent years, convolutional neural-network-based crack segmentation methods have performed excellently. However, existing crack segmentation methods still suffer from background noise interference, such as dirt patches and pitting, as well as the imprecise segmentation of fine-grained spatial structures. This is mainly due to the fact that convolutional neural networks dilute low-level spatial information in the process of extracting deep semantic features, and the network cannot obtain accurate context awareness because of the limitation of the actual receptive field size. To address these problems, an encoder–decoder crack segmentation network based on multi-scale contextual information enhancement is proposed. First, a new architecture of skip connection is proposed, enabling the network to obtain refined crack segmentation results; then, a contextual feature enhancement module is designed to make the network more effective at distinguishing between cracks and background noise; finally, the deformable convolution is introduced into the encoder network to further enhance its ability to extract the diverse morphological features of cracks by adaptively adjusting the sampling area and the receptive field size. Experiments show that the proposed method is effective in crack segmentation and outperforms mainstream segmentation networks such as DeepLab V3+ and UNet++.

Multi-Branch Adaptive Hard Region Mining Network for Urban Scene Parsing of High-Resolution Remote-Sensing Images

Scene parsing of high-resolution remote-sensing images (HRRSIs) refers to parsing different semantic regions from the images, which is an important fundamental task in image understanding. However, due to the inherent complexity of urban scenes, HRRSIs contain numerous object classes. These objects present large-scale variation and irregular morphological structures. Furthermore, their spatial distribution is uneven and contains substantial spatial details. All these features make it difficult to parse urban scenes accurately. To deal with these dilemmas, in this paper, we propose a multi-branch adaptive hard region mining network (MBANet) for urban scene parsing of HRRSIs. MBANet consists of three branches, namely, a multi-scale semantic branch, an adaptive hard region mining (AHRM) branch, and an edge branch. First, the multi-scale semantic branch is constructed based on a feature pyramid network (FPN). To reduce the memory footprint, ResNet50 is chosen as the backbone, which, combined with the atrous spatial pyramid pooling module, can extract rich multi-scale contextual information effectively, thereby enhancing object representation at various scales. Second, an AHRM branch is proposed to enhance feature representation of hard regions with a complex distribution, which would be difficult to parse otherwise. Third, the edge-extraction branch is introduced to supervise boundary perception training so that the contours of objects can be better captured. In our experiments, the three branches complemented each other in feature extraction and demonstrated state-of-the-art performance for urban scene parsing of HRRSIs. We also performed ablation studies on two HRRSI datasets from ISPRS and compared them with other methods.

DPFL-Nets: Deep Pyramid Feature Learning Networks for Multiscale Change Detection.

Due to the complementary properties of different types of sensors, change detection between heterogeneous images receives increasing attention from researchers. However, change detection cannot be handled by directly comparing two heterogeneous images since they demonstrate different image appearances and statistics. In this article, we propose a deep pyramid feature learning network (DPFL-Net) for change detection, especially between heterogeneous images. DPFL-Net can learn a series of hierarchical features in an unsupervised fashion, containing both spatial details and multiscale contextual information. The learned pyramid features from two input images make unchanged pixels matched exactly and changed ones dissimilar and after transformed into the same space for each scale successively. We further propose fusion blocks to aggregate multiscale difference images (DIs), generating an enhanced DI with strong separability. Based on the enhanced DI, unchanged areas are predicted and used to train DPFL-Net in the next iteration. In this article, pyramid features and unchanged areas are updated alternately, leading to an unsupervised change detection method. In the feature transformation process, local consistency is introduced to constrain the learned pyramid features, modeling the correlations between the neighboring pixels and reducing the false alarms. Experimental results demonstrate that the proposed approach achieves superior or at least comparable results to the existing state-of-the-art change detection methods in both homogeneous and heterogeneous cases.

Relation and context augmentation network for facial expression recognition

Facial Expression Recognition (FER) is a challenging task due to the complex properties of human facial expression. Recently, convolutional neural networks (CNNs) have been widely adopted by most FER approaches. However, CNN-models extract features by using convolutional and pooling operations which ignore the relations between pixels and channels. The relations among spatial positions and channels provide crucial information which can be leveraged for facial expression classification. Another important aspect of FER is utilization of global and local contextual information to improve recognition performance. In this work, we present a deep network, the Relation and Context Augmentation Network (RCANet), for facial expression classification. RCANet consists of two relation modules and a context module. The relation modules compute global relations in spatial and channel dimensions. The context module is composed of cascaded context units to capture multi-scale contextual information. Extensive experiments are conducted on two popular in-the-wild FER datasets, including RAF-DB and AffectNet. Experimental results demonstrate that our proposed method achieves 90.15% and 65.65% accuracy rate on the RAF-DB and AffectNet datasets respectively. • We propose a relation and context augmentation network which constructs global relations for facial expression recognition. • The spatial relation module and the channel relation module encode image-level spatial and channel relations. • The cascaded context augmentation module facilitates feature reuse and generates feature maps with more spatial details.

MFPA-Net: An efficient deep learning network for automatic ground fissures extraction in UAV images of the coal mining area

Ground fissures caused by high-intensity underground coal mining activities will damage the ecological environment and endanger mine safety. The complicated surface conditions in coal mining areas make fissure detection labor-intensive work, and manual fissure extraction is inefficient, restricting the surface monitoring work. In this study, an efficient Deep Learning (DL) network named MFPA-Net (Multi-scale Feature Pyramids and Attention Network) is proposed to extract ground fissures in Unmanned Aerial Vehicle (UAV) images from the coal mining area automatically and accurately. In MFPA-Net, the Dilated Residual Networks (DRN) are used to extract diverse context information, the Dual Attention Mechanism (DAM) is introduced to integrate the dependence of pixels' spatial location and feature channels to generate high-level features, the Atrous Spatial Pyramid Pooling (ASPP) is utilized for mining multi-scale context information from high-level features, and the Multi-scale Feature Pyramid Network (MFPN) is designed to combine high-level and low-level features. Moreover, the Focal Tversky Loss Function is adopted to handle the unbalanced samples. To promote DL technologies application in the fissure monitoring of mining areas, the GFCMA (Ground Fissures of the Coal Mining Area) dataset is constructed. Experiments on GFCMA show that MFPA-Net can achieve high Precision (69.4%), Recall (70.7%), F1-Score (70.0%), and Mean Intersection over Union (MIoU) (75.1%) simultaneously, which significantly outperform traditional image processing methods and recently DL networks. Experiments on public pavement datasets Crack500 and DeepCrack prove MFPA-Net's high reliability and widespread applicability. The performance of the trained MFPA-Net on real large-scale scenarios demonstrates its practical value, strong robustness, and high efficiency. This study provides a solution for rapid monitoring of ground fissures under complicated surface conditions, which can serve the safe production and ecological restoration high-efficiently in mining areas.

GCCINet: Global feature capture and cross-layer information interaction network for building extraction from remote sensing imagery

• Our GCCINet can effectively solve the current building extraction problems. • GCCINet performs state-of-the-art compared with eleven methods in recent studies. • The sufficient comparative experiments and analysis are implemented. • An efficient plug and play CAFE module is proposed. The extraction of buildings from remote sensing images is a challenging task. However, existing methods are insufficiently accurate because of the diverse types of buildings, large-scale variability, and complex backgrounds in remote sensing images. There are many deficiencies of the extraction results, such as small building omission, building internal discontinuity, blurred boundary, and irregular building appearance extraction. To solve these problems, a global feature capture and cross-layer information interaction network (GCCINet) is proposed, in which the continuous atrous convolution feature enhancement module is designed to capture a larger range of multiscale building feature information by using continuous atrous convolution to generate different sizes of receptive fields, thus alleviating the problem of discontinuities and the overall appearance of irregular buildings. The global high-low feature cross-fusion module reduces the loss of local information to enhance the ability to identify small buildings through the effective cross-fusion of high-low features. The cross-layer refined fusion and boundary refinement module adopts a unique fusion method to form information fusion between different layers, obtain multiscale context information, and further refine boundaries, thus improving the capability of boundary extraction. The WHU Building Dataset and Inria Dataset are selected for validation. The results show that GCCINet performs state-of-the-art (SOTA) compared with other existing methods, and the performance of GCCINet is verified in terms of usability, interference resistance, robustness, and ablation study. Furthermore, a plug-and-play CAFE module is designed, which can introduce a few parameters to other models while improving their performance.

Feature-Enhanced CenterNet for Small Object Detection in Remote Sensing Images

Compared with anchor-based detectors, anchor-free detectors have the advantage of flexibility and a lower calculation complexity. However, in complex remote sensing scenes, the limited geometric size, weak features of objects, and widely distributed environmental elements similar to the characteristics of objects make small object detection a challenging task. To solve these issues, we propose an anchor-free detector named FE-CenterNet, which can accurately detect small objects such as vehicles in complicated remote sensing scenes. First, we designed a feature enhancement module (FEM) composed of a feature aggregation structure (FAS) and an attention generation structure (AGS). This module contributes to suppressing the interference of false alarms in the scene by mining multiscale contextual information and combining a coordinate attention mechanism, thus improving the perception of small objects. Meanwhile, to meet the high positioning accuracy requirements of small objects, we proposed a new loss function without extra calculation and time cost during the inference process. Finally, to verify the algorithm performance and provide a foundation for subsequent research, we established a dim and small vehicle dataset (DSVD) containing various objects and complex scenes. The experiment results demonstrate that the proposed method performs better than mainstream object detectors. Specifically, the average precision (AP) metric of our method is 7.2% higher than that of the original CenterNet with only a decrease of 1.3 FPS.

LARFNet: Lightweight asymmetric refining fusion network for real-time semantic segmentation

In this paper, we propose a lightweight asymmetric refining fusion network (LARFNet) for real-time semantic segmentation to solve the problem that some existing models cannot achieve good segmentation accuracy with real-time inference speed in mobile devices due to the huge computational overhead. Specifically, LARFNet adopts an asymmetric encoder–decoder structure. The depth-wise separable asymmetric interaction module (DSAI module) is designed in the encoding process, which effectively extracted local and surrounding information under different receptive fields with optimized convolution in the condition of ensuring communication between channels. In the decoder, we design the bilateral pyramid pooling attention module (BPPA module) and the multi-stage refinement fusion module (MRF Module). The BPPA module is used to integrate the high-level output multi-scale context information. Based on spatial and channel attention mechanisms, the MRF module is proposed to refine the feature maps of different resolutions and guide the feature fusion. Experimental results show that LARFNet achieves 69.2% mIoU and 65.6% mIoU on Cityscapes and Camvid datasets at 127 FPS and 222 FPS respectively, only using a single NVIDIA GeForce GTX2080Ti GPU and 0.72M parameters without any pre-training or pre-processing. Compared with most of the existing state-of-the-art models, the proposed method realizes the efficient use of network parameters at a faster speed, reduces the number of network parameters, and still achieves the accuracy of good segmentation.

VP-KLNet: efficient 6D object pose estimation with an enhanced vector-field prediction network and a keypoint localization network

6D object pose estimation is essential for many applications with high demands in accuracy and speed. Compared with end-to-end approaches, pixel-wise voting network (PVNet), a vector-field based two-stage approach, has shown the superiority in accuracy but the inferiority in speed because of the time-consuming RANSAC-based voting strategy. To resolve this problem, we propose an efficient deep architecture that consists of an enhanced vector-field prediction network (VPNet) and a keypoint localization network (KLNet) and call it VP-KLNet. Specifically, the KLNet replaces PVNet’s time-consuming voting scheme by directly regressing 2D keypoints from the vector fields, which significantly improves the inference speed. Furthermore, to capture multiscale contextual information, we embed the pyramid pooling module between the encoder and decoder in VPNet to obtain more accurate object segmentation and vector-field prediction, with negligible speed loss. Experiments demonstrate that our method has more than 50% improved in the running speed to the baseline method PVNet and achieves comparable accuracy with the state-of-the-art methods on the LINEMOD and occlusion LINEMOD datasets.

A novel MCF-Net: Multi-level context fusion network for 2D medical image segmentation

Medical image segmentation is a crucial step in the clinical applications for diagnosis and analysis of some diseases. U-Net-based convolution neural networks have achieved impressive performance in medical image segmentation tasks. However, the multi-level contextual information integration capability and the feature extraction ability are often insufficient. In this paper, we present a novel multi-level context fusion network (MCF-Net) to improve the performance of U-Net on various segmentation tasks by designing three modules, hybrid attention-based residual atrous convolution (HARA) module, multi-scale feature memory (MSFM) module, and multi-receptive field fusion (MRFF) module, to fuse multi-scale contextual information. HARA module was proposed to effectively extract multi-receptive field features by combing atrous spatial pyramid pooling and attention mechanism. We further design the MSFM and MRFF modules to fuse features of different levels and effectively extract contextual information. The proposed MCF-Net was evaluated on the ISIC 2018, DRIVE, BUSI, and Kvasir-SEG datasets, which have challenging images of many sizes and widely varying anatomy. The experimental results show that MCF-Net is very competitive with other U-Net models, and it offers tremendous potential as a general-purpose deep learning model for 2D medical image segmentation.

Person Re-Identification Method Based on the Construction of Graph Convolutional Network with Attribute Feature

To fully pay attention to identity-sensitive feature information and utilize the correlations of inter-attributes and attributes-body parts, this paper proposes a person re-identification (re-ID) method based on the construction of graph convolutional network (GCN) with crucial attribute feature and body parts. First, it establishes the multiscale context-aware network (MSCAN) using dilated convolution with different expansion ratios, which can learn multiscale context information and obtain diversified global features. Subsequently, the human parsing model is utilized to extract the body part features. According to the attribute importance degree, the paper constructs low-dimensional GCN integrating the vital attributes and body parts of person descriptions to obtain discriminative local features. Finally, based on attribute prediction, it reduces the range of the images to be matched with discriminating possible objects from query images, thereby simplifying retrieval process. The experimental results demonstrate that the novel designed method can effectively improve person re-ID performance and achieve competitive evaluation results on typical public testing datasets.